Skip navigation
All Places > All Things PI - Ask, Discuss, Connect > Ideas

Allow bi-temporal data archiving

score 8
You have not voted Submitted

Originally posted on UserVoice
PI DA Server should allow for more flexibility in terms of the format of archived data. A big advantage would be if PI could archive two timestamps per event. One timestamp will come from whatever the data source is (PI Interfaces, fiels devices, etc) - this is effectively what PI has now. In addition to that, PI Should be able to add a second timestamp recording when the event got into PI i.e. when either PI Snapshot or Archive Subsystem saw the event coming in their respective queue. Benefits and use cases:
- any PI Client could make sure it gets all new pi events since client last talked to PI. Keep in mind that PI can have lots of backfills going on at the same time as realtime data so it is impossible for a PI Client app to know that since last time it talked to PI, there was a backfill correcting some data for 3 months ago. I know there are consumer queues, but they are not working properly all the time, they can overflow, etc. The more robust way would be if PI knew when it actually got the events.
- PI admins would easily detect backfill operations and asses data latency and anomalous delays. Questions like : when do we get most data into our PI during the day, or when did these duplicate events made into PI, or what is the data latency from field sensors to the PI archive, would all easily be answered. Now it's virtually impossible.

Please upvote if you like.