Hello
I have a requirement to extract a year's worth of data from around 100k PI tags.
I don't think i can use PI Datalink and PI SQL commander for this as they are going to take a huge amount of time.
What are the best options that i can consider to fulfill my requirement? Can i use PI Config? I am open to develop an AFSDK application but i am looking for few suggestions.
Thank you in Advance
Arpit
Hi Arpit,
You have below options
1) PI Config
2) PI OLEDB
3) AF SDK
Thank you for the reply Lal.
What is the performance impact of using PI Config scripts to query 100k tags?
Let me help you by asking some questions first:
1) Where do you want to send this data to? Depending on the final destination, there are better ways to do it.
2) How frequent is this data? Are we talking about second / sub-second or hourly / daily?
Thank you for the reply Rafael.
Below are the answers for your question:
1 - I just want to extract the data and provide it to the analysts (flat files will do).
2 - We have hourly data - 24 values/day.
Arpit
Hello Arpit,
I think PowerShell is probably the simplest way to do this. It of course uses the AF SDK, but you don't need to compile an application in Visual Studio, and the cmdlets are quite easy to use.
For reference: https://techsupport.osisoft.com/Documentation/PI-Powershell/title.html
Hope this helps,
Seb
Thank you Seb.
I will give Powershell a try, but most likely i will go with AFSDK.
Arpit
So we are talking about 876M data points. That's a lot and you should not use PI Config for it. I suggest you two ways to do it: The first one is using AF SDK. Extract data using it is simple and allows you even to already do some data alignment for your analysts. An alternative is to follow Sebastien's suggestion and use our PowerShell tools, that is a wrapper around AF SDK. The second one depends on the platform your analysts use to process the data. If they use R, you can use the PI Web API R Package. We have some other libraries for specific platforms.
For extracting large amounts of data sets from the PI System for use in business applications, the recommended method would be using PI Integrator for Business Analytics
PI Integrator for Business Analytics presents PI System data perfectly suited to business intelligence tools including but not limited to Tableau, Tibco Spotfire, QlikView, and Microsoft Power BI for reporting and analytics.
Other techniques, such as the use of of Developer technologies should always be considered as a secondary method.
If you are interested in PI AF SDK, check out GetLargeRecordedValues - working around ArcMaxCollect and a console application wrapper that uses this for downloading data as csv files Extracting large event counts from the PI Data Archive
on May 15, 2018
So we are talking about 876M data points. That's a lot and you should not use PI Config for it. I suggest you two ways to do it: The first one is using AF SDK. Extract data using it is simple and allows you even to already do some data alignment for your analysts. An alternative is to follow Sebastien's suggestion and use our PowerShell tools, that is a wrapper around AF SDK. The second one depends on the platform your analysts use to process the data. If they use R, you can use the PI Web API R Package. We have some other libraries for specific platforms.