AnsweredAssumed Answered

PI Data Archive and creating a compressed edition of the data

Question asked by kpierce on Apr 17, 2019
Latest reply on Apr 24, 2019 by kpierce


This is not about the benefits of compression or its impacts, it's about how to create a compressed PI DA.

We have a lot of customers who have used little to no compression historically (pun intended).  As their systems and data uses grow, they are finding performance limitations without using compression (or using minimum compression).  Often they state that the challenge is getting engineers and others to accept this.  With many of these customers have an EA (commercial agreement) standing up a compressed PI DA is less of a cost or issue for them. 

I'm aware of one such customer who has created a parallel PI DA with compressed data.  This customer has many years of data, and 3 similar units (power generation).  For reasons we don't need to go into, they loaded their original non-compressed PI DA using 1 second files for all the data points.  Initially they setup PI-to-PI and used history recovery to create the compressed PI DA.  This worked, but when there were issues with PI-to-PI, it ended up that some of the compressed data went missing.  Early on the end users were really happy with the performance but soon found the issue.  One could conclude that this could have been avoided with better monitoring of PI-to-PI, but the solution taken was to just reload the compressed server from their original source files using tags (in the compressed PI DA) that had appropriate settings for compression.

We have other customers who are considering setting up such a parallel PI DA and i'm asking for recommendations and experience for them.  Most, if not all others, do not have original 1 second files, they only have the non-compressed PI DA to load from.

I'm not aware of a way to create a new PI DA, say from a backup, where we can do something like re-process the archive files and create a compressed PI DA. I'm only aware of using PI-to-PI and history recovery to create the new, compressed PI DA.  I guess one could use PI AF Analytics to just replicate the incoming values to a different PI DA and tag settings (and use backfill) but this would seem cumbersome at best.

What recommendations do you have for doing this? Any experiences to share?

I would think that a new, separate PI DA is the better way to go from a management standpoint and lessening any confusion.  One could create new tags, with compression settings, and give them a modified tag name to indicate compressed data, however, the data would be spread out among more archive files than needed and that could impact performance.

Are there any known tools or other ways of doing this besides PI-to-PI?