14 Replies Latest reply on Nov 13, 2017 12:08 PM by gregor

Mass Export Archive data of Tag subset

aldorexbraam Nov 1, 2017 3:01 PM

Hi for an analytics project I have the requirement to bring data from the PI archives of the last X years for 10K tags (in total 80K) into an external data lake (or swamp)

Since this concerns a lot of data to process (a little under 1TB) i am looking for a way to process the archives in firehose mode, instead of the traditional tools ( PI OLEDB (enterprise), PI-SDK )

AFAIK all current solutions except AF-SDK, have limitations and don't cut it in terms of performance, storage, usability . Ideally I am looking for an .NET API that taps *directly* into the archives.

Is anybody in the community aware of a *standard* solution that would fit above case ?

Thanks in advance..

Aldo Braam

btw: So far I have assessed the following solutions.

Product	Pro	Con
PI OLEDB	multi-hreaded, easy to use	Custom development, no bulk fetches, so archive walk for each tag
PI Integrator	multi-hreaded, easy to use	Streams only SQL Server so inefficient storage
PI config	simple, crude	works on tag-for-tag basis, single threaded / single core . 70-ties interface, no bulk fetches
PI-Datalink	simple	not suitable for massive data loads , no bulk fetches
PI-SDK	simple	single threaded, no bulk fetches
AF-SDK	versatile, multi-threaded	supports bulk fetches, custom coded solution

Re: Mass Export Archive data of Tag subset

Roger Palmen Oct 25, 2017 12:37 PM (in response to aldorexbraam)

Hi Aldo,
Long time no see!

I think you already found the best answer yourself in AFSDK. Bulk exports of history are by nature not very time critical.

What i did find the most important criterium is the format and method by which the data needs to be exported. Sending data out to a CSV works fine with AFDSK and a logging library like NLog.
If you need to process the data, or stream the data out to e.g. Azure EventHubs then that's a whole different game,
2 of 2 people found this helpful
Like Show 0 Likes(0)

Actions
- Re: Mass Export Archive data of Tag subset
  
  pmartin Oct 25, 2017 12:58 PM (in response to Roger Palmen)
  
  I second using the AF SDK. You may also be interested in this code developed by Rick Davin: GetLargeRecordedValues - working around ArcMaxCollect.
  2 of 2 people found this helpful
  
  Like Show 0 Likes(0)
  
  Actions
  - Re: Mass Export Archive data of Tag subset
    
    Rick Davin Nov 1, 2017 4:07 PM (in response to pmartin)
    
    There are a few GitHub projects that may help. I would start with this one from Patrice Thivierge :
    
    GitHub - pthivierge/data-reader-cmdline: A command line application to read high volumes of data from the PI Data Archiv…
    
    Like Show 1 Likes(1)
    
    Actions
    - Re: Mass Export Archive data of Tag subset
      
      aldorexbraam Nov 9, 2017 11:36 AM (in response to Rick Davin)
      
      Excellent.......thanks
      
      Like Show 0 Likes(0)
      
      Actions
    - Re: Mass Export Archive data of Tag subset
      
      FrancisNyarko Nov 10, 2017 9:10 AM (in response to Rick Davin)
      
      There is no 'Datareader.exe' in the GIT download. Kindly advise.
      
      Like Show 0 Likes(0)
      
      Actions
      - Re: Mass Export Archive data of Tag subset
        
        gregor Nov 10, 2017 9:21 AM (in response to FrancisNyarko)
        
        Hello Francis,
        
        The repository contains Visual Studio project sources. You need to download the project and compile it.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Mass Export Archive data of Tag subset
        
        FrancisNyarko Nov 10, 2017 9:24 AM (in response to gregor)
        
        Oh I see. Thanks. Very helpful.
        
        Like Show 0 Likes(0)
        
        Actions
- Re: Mass Export Archive data of Tag subset
  
  aldorexbraam Oct 25, 2017 5:08 PM (in response to Roger Palmen)
  
  Hi Roger,
  Thanks for the reply.... It has been a while indeed....
  I was afraid it would come to this answer ....would have loved it if there was some standard tooling for this....(hint hint)
  
  Like Show 0 Likes(0)
  
  Actions
- Re: Mass Export Archive data of Tag subset
  
  aldorexbraam Nov 9, 2017 10:03 AM (in response to Roger Palmen)
  
  Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
  - a source server running on the PI server
  - a simple client communicating over a TCP channel
  
  The tool would have two modes:
  - history recovery mode ( mass export )
  - subscription to the event pipe.
  
  This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
  
  Like Show 1 Likes(1)
  
  Actions
  - Re: Mass Export Archive data of Tag subset
    
    Roger Palmen Nov 10, 2017 7:50 AM (in response to aldorexbraam)
    
    Eagerly awaiting the Github repo...
    
    Like Show 0 Likes(0)
    
    Actions
  - Re: Mass Export Archive data of Tag subset
    
    gregor Nov 10, 2017 9:33 AM (in response to aldorexbraam)
    
    Aldo Braam wrote:
    ...would have loved it if there was some standard tooling for this....(hint hint)
    Hi Aldo,
    
    It is always preferred to use an existing product upon "reinventing the wheel". Did you overlook Jinmo's reference to the PI Integrator for Business Analytics? If so, please allow me to point out that this is the standard tool OSIsoft offers for the purpose.
    
    Aldo Braam wrote:
    Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
    - a source server running on the PI server
    - a simple client communicating over a TCP channel
    
    The tool would have two modes:
    - history recovery mode ( mass export )
    - subscription to the event pipe.
    
    This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
    I believe it is not that easy. You may soon recognize that the task is more complex than expected. Please allow me to repeat the reference to KB01481 - Features of PI Integrator for Business Analytics shared by Jinmo earlier.
    
    Like Show 0 Likes(0)
    
    Actions
    - Re: Mass Export Archive data of Tag subset
      
      aldorexbraam Nov 10, 2017 3:40 PM (in response to gregor)
      
      Hi Gregor,
      I am afraid this discussion is spinning off ....we can take this 1:1 if you want ....but let me say this .... I am using the PI integrator, IMO the Integrator was made for a totally different purpose :
      outputting archive data of some tags in a timesynchronized way. For most Analytics cases time synchronization is wanted, but in some case ( predictive model construction ) you want the raw datasets.
      It is *not* made to output all archive data for chunks of 10K tags for 5 years. I don't think that will fly....
      For the 'streaming' scenario I agree with you that the PII would be a nice option, however I am not sure customers are willing to pay the license fee for a once-off or occasional exercise. PI Integrator comes with a nice licensing model based on 'streams'....today I want to dump 500 tags....next month I might want to export 20K tags...after that I might want even more....do I have to re-negotiate with OSI sales to get to my own data ?
      Cheers,
      Aldo
      
      Like Show 1 Likes(1)
      
      Actions
      - Re: Mass Export Archive data of Tag subset
        
        gregor Nov 13, 2017 12:08 PM (in response to aldorexbraam)
        
        Hello Aldo,
        
        While you are right that the PI Integrator for BA was not designed for the use cases you specify, my understanding is that your first 2 points are about functionality of an existing product. However, to my understanding the required functionality shouldn't result into another product. Hence, my idea would be requesting the missing functionality e.g. by posting at Uservoice.
        
        I understand that your major concern is about the pricing for the PI Integrator for BA and this is indeed a topic you should be addressing with your account manager at OSIsoft.
        
        Like Show 0 Likes(0)
        
        Actions
Re: Mass Export Archive data of Tag subset

jyi Nov 2, 2017 1:17 AM (in response to aldorexbraam)

Hi Aldo,

If your data lake is of Microsoft Azures, then PI Integrator for MS Azure would fit perfectly.
Otherwise PI Integrator for BA can be used to stream data into different targets than SQL server, such as Hadoop HDFS, and Apache HIVE
Do you have a particular data lake that you are working with? Analytics project would normally require you to prep the data. PI data is of no exception. PI Integrators can normally do these for you. What is your plan on doing this?https://techsupport.osisoft.com/Troubleshooting/KB/KB01481
If you have not read below KB, I would recommend this to you.
KB01481 - Features of PI Integrator for Business Analytics
1 of 1 people found this helpful
Like Show 0 Likes(0)

Actions