AnsweredAssumed Answered

Extracting really huge amounts of data from PI, outside-the-box ideas?

Question asked by Roger Palmen on Jun 8, 2018
Latest reply on Oct 16, 2019 by gregor

Hi All,


We all know there are several ways to extract large amounts of data from PI using PI Config or a custom tool using PI AFSDK. (The options basically here: How to write PI Data to files in XML format? )

But even these have their limitations. I've built some tools for that but i typically don't get near 1M events/second for a hostory extraction. And with millions of tags and several years of data to process, that is too slow. Setting aside the causes of that, i started thinking outside the box.


We currently have a process in place where we restore the PI Archives to a separate PI Server, and extract the data from that to other files before sending that off to another platform. To maximize performance, the first idea is to remove as much components as possible from the solution, and here that would be PI. Why do i restore files, and then use two applications to read those files (PI Data Historian) and write these to other files (custom AFSDK application)?


In other words: would it be possible to read the PI Archive files directly to Transform the data to a different format?