We have been doing an analysis of multiple tools that provide us with large amounts of data. Our next task is to attempt to store that data into PI. However, our tests so far have not been successful, so we would like to pose the question to the community as to the best and/or most reliable way to bring large volumes of data into PI in real-time. Here are some details:
To keep it simple, let us deal with a typical happy-path scenario. This scenario involves 5,000 devices sending 4 values per second over the internet to one of our Web APIs. That means that we need to find a way to move 20,000 values per second into PI. As we currently see it, we have 3 options:
- Use the OSIsoft PI-SDK to write data (as we get it) directly to PI.
- Use the OSIsoft PI-AFSDK to write data (as we get it) directly to PI.
- Create CSV files from the data we retrieve and dump the files into locations wherein UFL interfaces exist and let these interfaces write the data to PI.
Options 1 and 2 above are more "real-time" than option 3. However, it has been shown that options 1 and 2 cannot keep up with writing 20,000 values per second to PI. Option 3 sees our application easily able to create CSV files and keep up with the data throughput, but it is unknown (as of this writing) if the UFL interface will be capable or keeping the pace. Additionally, this does not account for potential down-time, errors, or backfilling. This is definitely a custom, cannot fail, type of application.
So the question is as the title indicates. Is there a way to bring data in from the internet and write it to PI at a rate of 20,000 values per second in a reliable, real-time (or close to real-time) manner? If so, what is the OSIsoft recommendation for making this happen and where can I find documentation on this?