AnsweredAssumed Answered

Mass Export Archive data of Tag subset

Question asked by aldorexbraam on Oct 25, 2017
Latest reply on Nov 13, 2017 by gregor

Like • Show 1 Like1
Comment • 14

Hi for an analytics project I have the requirement to bring data from the PI archives of the last X years for 10K tags (in total 80K) into an external data lake (or swamp)

Since this concerns a lot of data to process (a little under 1TB) i am looking for a way to process the archives in firehose mode, instead of the traditional tools ( PI OLEDB (enterprise), PI-SDK )

AFAIK all current solutions except AF-SDK, have limitations and don't cut it in terms of performance, storage, usability . Ideally I am looking for an .NET API that taps *directly* into the archives.

Is anybody in the community aware of a *standard* solution that would fit above case ?

Thanks in advance..

Aldo Braam

btw: So far I have assessed the following solutions.

Product	Pro	Con
PI OLEDB	multi-hreaded, easy to use	Custom development, no bulk fetches, so archive walk for each tag
PI Integrator	multi-hreaded, easy to use	Streams only SQL Server so inefficient storage
PI config	simple, crude	works on tag-for-tag basis, single threaded / single core . 70-ties interface, no bulk fetches
PI-Datalink	simple	not suitable for massive data loads , no bulk fetches
PI-SDK	simple	single threaded, no bulk fetches
AF-SDK	versatile, multi-threaded	supports bulk fetches, custom coded solution

Rick Davin

Nov 1, 2017 4:07 PM

Correct Answer

There are a few GitHub projects that may help. I would start with this one from Patrice Thivierge :

GitHub - pthivierge/data-reader-cmdline: A command line application to read high volumes of data from the PI Data Archiv…

See the reply in context

No one else had this question

Outcomes

14 Replies

Roger Palmen Oct 25, 2017 12:37 PM
Mark Correct
Correct Answer
Hi Aldo,
Long time no see!

I think you already found the best answer yourself in AFSDK. Bulk exports of history are by nature not very time critical.

What i did find the most important criterium is the format and method by which the data needs to be exported. Sending data out to a CSV works fine with AFDSK and a logging library like NLog.
If you need to process the data, or stream the data out to e.g. Azure EventHubs then that's a whole different game,
2 people found this helpful
Like • Show 0 Likes0
Actions
- pmartin @ Roger Palmen on Oct 25, 2017 12:58 PM
  Mark Correct
  Correct Answer
  I second using the AF SDK. You may also be interested in this code developed by Rick Davin: GetLargeRecordedValues - working around ArcMaxCollect.
  2 people found this helpful
  Like • Show 0 Likes0
  Actions
  - Rick Davin @ pmartin on Nov 1, 2017 4:07 PM
    Unmark Correct
    Correct Answer
    There are a few GitHub projects that may help. I would start with this one from Patrice Thivierge :
    
    GitHub - pthivierge/data-reader-cmdline: A command line application to read high volumes of data from the PI Data Archiv…
    Like • Show 1 Like1
    Actions
    - aldorexbraam @ Rick Davin on Nov 9, 2017 11:36 AM
      Mark Correct
      Correct Answer
      Excellent.......thanks
      Like • Show 0 Likes0
      Actions
    - FrancisNyarko @ Rick Davin on Nov 10, 2017 9:10 AM
      Mark Correct
      Correct Answer
      There is no 'Datareader.exe' in the GIT download. Kindly advise.
      Like • Show 0 Likes0
      Actions
      - gregor @ FrancisNyarko on Nov 10, 2017 9:21 AM
        Mark Correct
        Correct Answer
        Hello Francis,
        
        The repository contains Visual Studio project sources. You need to download the project and compile it.
        Like • Show 0 Likes0
        Actions
        FrancisNyarko @ gregor on Nov 10, 2017 9:24 AM
        Mark Correct
        Correct Answer
        Oh I see. Thanks. Very helpful.
        Like • Show 0 Likes0
        Actions
- aldorexbraam @ Roger Palmen on Oct 25, 2017 5:08 PM
  Mark Correct
  Correct Answer
  Hi Roger,
  Thanks for the reply.... It has been a while indeed....
  I was afraid it would come to this answer ....would have loved it if there was some standard tooling for this....(hint hint)
  Like • Show 0 Likes0
  Actions
- aldorexbraam @ Roger Palmen on Nov 9, 2017 10:03 AM
  Mark Correct
  Correct Answer
  Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
  - a source server running on the PI server
  - a simple client communicating over a TCP channel
  
  The tool would have two modes:
  - history recovery mode ( mass export )
  - subscription to the event pipe.
  
  This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
  Like • Show 1 Like1
  Actions
  - Roger Palmen @ aldorexbraam on Nov 10, 2017 7:50 AM
    Mark Correct
    Correct Answer
    Eagerly awaiting the Github repo...
    Like • Show 0 Likes0
    Actions
  - gregor @ aldorexbraam on Nov 10, 2017 9:33 AM
    Mark Correct
    Correct Answer
    Aldo Braam wrote:
    ...would have loved it if there was some standard tooling for this....(hint hint)
    Hi Aldo,
    
    It is always preferred to use an existing product upon "reinventing the wheel". Did you overlook Jinmo's reference to the PI Integrator for Business Analytics? If so, please allow me to point out that this is the standard tool OSIsoft offers for the purpose.
    
    Aldo Braam wrote:
    Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
    - a source server running on the PI server
    - a simple client communicating over a TCP channel
    
    The tool would have two modes:
    - history recovery mode ( mass export )
    - subscription to the event pipe.
    
    This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
    I believe it is not that easy. You may soon recognize that the task is more complex than expected. Please allow me to repeat the reference to KB01481 - Features of PI Integrator for Business Analytics shared by Jinmo earlier.
    Like • Show 0 Likes0
    Actions
    - aldorexbraam @ gregor on Nov 10, 2017 3:40 PM
      Mark Correct
      Correct Answer
      Hi Gregor,
      I am afraid this discussion is spinning off ....we can take this 1:1 if you want ....but let me say this .... I am using the PI integrator, IMO the Integrator was made for a totally different purpose :
      outputting archive data of some tags in a timesynchronized way. For most Analytics cases time synchronization is wanted, but in some case ( predictive model construction ) you want the raw datasets.
      It is *not* made to output all archive data for chunks of 10K tags for 5 years. I don't think that will fly....
      For the 'streaming' scenario I agree with you that the PII would be a nice option, however I am not sure customers are willing to pay the license fee for a once-off or occasional exercise. PI Integrator comes with a nice licensing model based on 'streams'....today I want to dump 500 tags....next month I might want to export 20K tags...after that I might want even more....do I have to re-negotiate with OSI sales to get to my own data ?
      Cheers,
      Aldo
      Like • Show 1 Like1
      Actions
      - gregor @ aldorexbraam on Nov 13, 2017 12:08 PM
        Mark Correct
        Correct Answer
        Hello Aldo,
        
        While you are right that the PI Integrator for BA was not designed for the use cases you specify, my understanding is that your first 2 points are about functionality of an existing product. However, to my understanding the required functionality shouldn't result into another product. Hence, my idea would be requesting the missing functionality e.g. by posting at Uservoice.
        
        I understand that your major concern is about the pricing for the PI Integrator for BA and this is indeed a topic you should be addressing with your account manager at OSIsoft.
        Like • Show 0 Likes0
        Actions
jyi Nov 2, 2017 1:17 AM
Mark Correct
Correct Answer
Hi Aldo,

If your data lake is of Microsoft Azures, then PI Integrator for MS Azure would fit perfectly.
Otherwise PI Integrator for BA can be used to stream data into different targets than SQL server, such as Hadoop HDFS, and Apache HIVE
Do you have a particular data lake that you are working with? Analytics project would normally require you to prep the data. PI data is of no exception. PI Integrators can normally do these for you. What is your plan on doing this?https://techsupport.osisoft.com/Troubleshooting/KB/KB01481
If you have not read below KB, I would recommend this to you.
KB01481 - Features of PI Integrator for Business Analytics
1 person found this helpful
Like • Show 0 Likes0
Actions

Incoming Links

Re: What is the record format within the Pi Data Archive?

Mass Export Archive data of Tag subset

Outcomes

Related Content

Recommended Content

Incoming Links