8 Replies Latest reply on May 16, 2018 9:08 AM by Arpit

Data extraction

Arpit May 15, 2018 9:48 AM

Hello

I have a requirement to extract a year's worth of data from around 100k PI tags.

I don't think i can use PI Datalink and PI SQL commander for this as they are going to take a huge amount of time.

What are the best options that i can consider to fulfill my requirement? Can i use PI Config? I am open to develop an AFSDK application but i am looking for few suggestions.

Thank you in Advance

Arpit

Re: Data extraction

Lal_Babu_Shaik May 15, 2018 10:35 AM (in response to Arpit)

Hi Arpit,

You have below options
1) PI Config
2) PI OLEDB
3) AF SDK
Like Show 0 Likes(0)

Actions
- Re: Data extraction
  
  Arpit May 15, 2018 1:01 PM (in response to Lal_Babu_Shaik)
  
  Thank you for the reply Lal.
  
  What is the performance impact of using PI Config scripts to query 100k tags?
  
  Like Show 0 Likes(0)
  
  Actions
Re: Data extraction

rborges May 15, 2018 12:11 PM (in response to Arpit)

Let me help you by asking some questions first:

1) Where do you want to send this data to? Depending on the final destination, there are better ways to do it.
2) How frequent is this data? Are we talking about second / sub-second or hourly / daily?
Like Show 0 Likes(0)

Actions
- Re: Data extraction
  
  Arpit May 15, 2018 12:59 PM (in response to rborges)
  
  Thank you for the reply Rafael.
  
  Below are the answers for your question:
  
  1 - I just want to extract the data and provide it to the analysts (flat files will do).
  2 - We have hourly data - 24 values/day.
  
  Arpit
  
  Like Show 0 Likes(0)
  
  Actions
  - Re: Data extraction
    
    sraposo May 15, 2018 1:08 PM (in response to Arpit)
    
    Hello Arpit,
    
    I think PowerShell is probably the simplest way to do this. It of course uses the AF SDK, but you don't need to compile an application in Visual Studio, and the cmdlets are quite easy to use.
    
    For reference: https://techsupport.osisoft.com/Documentation/PI-Powershell/title.html
    
    Hope this helps,
    Seb
    
    Like Show 1 Likes(1)
    
    Actions
    - Re: Data extraction
      
      Arpit May 16, 2018 9:08 AM (in response to sraposo)
      
      Thank you Seb.
      
      I will give Powershell a try, but most likely i will go with AFSDK.
      
      Arpit
      
      Like Show 0 Likes(0)
      
      Actions
  - Re: Data extraction
    
    rborges May 15, 2018 1:13 PM (in response to Arpit)
    
    So we are talking about 876M data points. That's a lot and you should not use PI Config for it. I suggest you two ways to do it: The first one is using AF SDK. Extract data using it is simple and allows you even to already do some data alignment for your analysts. An alternative is to follow Sebastien's suggestion and use our PowerShell tools, that is a wrapper around AF SDK. The second one depends on the platform your analysts use to process the data. If they use R, you can use the PI Web API R Package. We have some other libraries for specific platforms.
    1 of 1 people found this helpful
    
    Like Show 0 Likes(0)
    
    Actions
Re: Data extraction

tramachandran May 15, 2018 4:18 PM (in response to Arpit)

For extracting large amounts of data sets from the PI System for use in business applications, the recommended method would be using PI Integrator for Business Analytics
PI Integrator for Business Analytics presents PI System data perfectly suited to business intelligence tools including but not limited to Tableau, Tibco Spotfire, QlikView, and Microsoft Power BI for reporting and analytics.

Other techniques, such as the use of of Developer technologies should always be considered as a secondary method.
If you are interested in PI AF SDK, check out GetLargeRecordedValues - working around ArcMaxCollect and a console application wrapper that uses this for downloading data as csv files Extracting large event counts from the PI Data Archive
Like Show 0 Likes(0)

Actions

Go to original post