-
Re: Mass Export Archive data of Tag subset
Roger Palmen Oct 25, 2017 12:37 PM (in response to aldorexbraam)2 of 2 people found this helpfulHi Aldo,
Long time no see!
I think you already found the best answer yourself in AFSDK. Bulk exports of history are by nature not very time critical.
What i did find the most important criterium is the format and method by which the data needs to be exported. Sending data out to a CSV works fine with AFDSK and a logging library like NLog.
If you need to process the data, or stream the data out to e.g. Azure EventHubs then that's a whole different game,
-
Re: Mass Export Archive data of Tag subset
pmartinOct 25, 2017 12:58 PM (in response to Roger Palmen)
2 of 2 people found this helpfulI second using the AF SDK. You may also be interested in this code developed by Rick Davin: GetLargeRecordedValues - working around ArcMaxCollect.
-
Re: Mass Export Archive data of Tag subset
Rick DavinNov 1, 2017 4:07 PM (in response to pmartin)
There are a few GitHub projects that may help. I would start with this one from Patrice Thivierge :
-
Re: Mass Export Archive data of Tag subset
aldorexbraam Nov 9, 2017 11:36 AM (in response to Rick Davin)Excellent.......thanks
-
Re: Mass Export Archive data of Tag subset
FrancisNyarko Nov 10, 2017 9:10 AM (in response to Rick Davin)There is no 'Datareader.exe' in the GIT download. Kindly advise.
-
Re: Mass Export Archive data of Tag subset
gregorNov 10, 2017 9:21 AM (in response to FrancisNyarko)
Hello Francis,
The repository contains Visual Studio project sources. You need to download the project and compile it.
-
Re: Mass Export Archive data of Tag subset
FrancisNyarko Nov 10, 2017 9:24 AM (in response to gregor)Oh I see. Thanks. Very helpful.
-
-
-
-
-
Re: Mass Export Archive data of Tag subset
aldorexbraam Oct 25, 2017 5:08 PM (in response to Roger Palmen)Hi Roger,
Thanks for the reply.... It has been a while indeed....
I was afraid it would come to this answer ....would have loved it if there was some standard tooling for this....(hint hint)
-
Re: Mass Export Archive data of Tag subset
aldorexbraam Nov 9, 2017 10:03 AM (in response to Roger Palmen)Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
- a source server running on the PI server
- a simple client communicating over a TCP channel
The tool would have two modes:
- history recovery mode ( mass export )
- subscription to the event pipe.
This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
-
Re: Mass Export Archive data of Tag subset
Roger Palmen Nov 10, 2017 7:50 AM (in response to aldorexbraam)Eagerly awaiting the Github repo...
-
Re: Mass Export Archive data of Tag subset
gregorNov 10, 2017 9:33 AM (in response to aldorexbraam)
Aldo Braam wrote:
...would have loved it if there was some standard tooling for this....(hint hint)
Hi Aldo,
It is always preferred to use an existing product upon "reinventing the wheel". Did you overlook Jinmo's reference to the PI Integrator for Business Analytics? If so, please allow me to point out that this is the standard tool OSIsoft offers for the purpose.
Aldo Braam wrote:
Hmmm you actually got me thinking.....suppose you *DO* have a requirement to stream data to other analytics platforms .....using the SDK to write a tool that consists of two parts:
- a source server running on the PI server
- a simple client communicating over a TCP channel
The tool would have two modes:
- history recovery mode ( mass export )
- subscription to the event pipe.
This all is relatively easy to do with the AF-SDK. To sugar-coat it the client could support the use of plug-ins...The default plugin would be export to text file...Community users would write their own plugins for the targeted datalakes
I believe it is not that easy. You may soon recognize that the task is more complex than expected. Please allow me to repeat the reference to KB01481 - Features of PI Integrator for Business Analytics shared by Jinmo earlier.
-
Re: Mass Export Archive data of Tag subset
aldorexbraam Nov 10, 2017 3:40 PM (in response to gregor)Hi Gregor,
I am afraid this discussion is spinning off ....we can take this 1:1 if you want ....but let me say this .... I am using the PI integrator, IMO the Integrator was made for a totally different purpose :
- outputting archive data of some tags in a timesynchronized way. For most Analytics cases time synchronization is wanted, but in some case ( predictive model construction ) you want the raw datasets.
- It is *not* made to output all archive data for chunks of 10K tags for 5 years. I don't think that will fly....
- For the 'streaming' scenario I agree with you that the PII would be a nice option, however I am not sure customers are willing to pay the license fee for a once-off or occasional exercise. PI Integrator comes with a nice licensing model based on 'streams'....today I want to dump 500 tags....next month I might want to export 20K tags...after that I might want even more....do I have to re-negotiate with OSI sales to get to my own data ?
Cheers,
Aldo
-
Re: Mass Export Archive data of Tag subset
gregorNov 13, 2017 12:08 PM (in response to aldorexbraam)
Hello Aldo,
While you are right that the PI Integrator for BA was not designed for the use cases you specify, my understanding is that your first 2 points are about functionality of an existing product. However, to my understanding the required functionality shouldn't result into another product. Hence, my idea would be requesting the missing functionality e.g. by posting at Uservoice.
I understand that your major concern is about the pricing for the PI Integrator for BA and this is indeed a topic you should be addressing with your account manager at OSIsoft.
-
-
-
-
Re: Mass Export Archive data of Tag subset
jyiNov 2, 2017 1:17 AM (in response to aldorexbraam)
1 of 1 people found this helpfulHi Aldo,
If your data lake is of Microsoft Azures, then PI Integrator for MS Azure would fit perfectly.
Otherwise PI Integrator for BA can be used to stream data into different targets than SQL server, such as Hadoop HDFS, and Apache HIVE
Do you have a particular data lake that you are working with? Analytics project would normally require you to prep the data. PI data is of no exception. PI Integrators can normally do these for you. What is your plan on doing this?https://techsupport.osisoft.com/Troubleshooting/KB/KB01481
If you have not read below KB, I would recommend this to you.