7 Replies Latest reply on Jun 3, 2013 7:39 PM by RJKSolutions

    Delay data from the incoming tag stream to another tag




      In this case, we have an interface that is picking up sub-second (synchrophasor) data and writing it to a PI Server.  Some users (external) cannot see the data in real time but to complete the analysis that is necessary of this data, the delayed data must come in at the same frequency. The set of tags (or a separate PI Server) could have security enabled such that some users only see these tags, i.e. the group allowed access to only the delayed data.


      We are looking for the best way to delay the data in real time.  Our best option to date is a custom program that signs up for snapshot updates for the real time tags (a small set, <200) of subsecond (about 60 events per second), read these points, keeps them in a small, 24 hour buffer, and then writes the data after it ages 24 hours, into a separate set of tags (same PI Server or different).


      Has anyone write such a program?  It would basically need a set of points (likely by point sources), sign up for updates on these points and keep their snapshot values in a 24 hour buffer, then write them to a separate set of tags in sequence after the 24 hour delay has occurred.


      If you're not aware of a program that exists, do you have another idea?  We've considered things like moving archives after 24 hours, starting/stopping PI-to-PI in history recover mode, but neither will keep the data coming in in natural frequency (60 hertz) to the snapshot with a constant 24 hour delay.  We also considered a PE with *-24h but we don't believe that PE will keep up with the frequency of data in a smooth means, i.e. continuous at a similar frequency.  Using the delay in AF won't work either since in this particular case, the data will need to come directly from the tag, i.e. not via AF.



        • Re: Delay data from the incoming tag stream to another tag

          Without writing a custom program, you can use AFOLEDB, UFL, and Task Scheduler – task writes out CSV records (200 tags with 60 events per second per tag) to a file via AF query - query includes the *-24hour logic -  the CSV file is picked up by UFL.  The tag list can be maintained in an Element. UFL's performance to process 12000 records/sec needs to be checked out.

            • Re: Delay data from the incoming tag stream to another tag
              Roger Palmen



              In general, it appears that the major requirement you have is to get the data in a second ste of PI tags after 24 hours have passed. I think that the exact frequency in which the data is received in the primary system should not need to be mirrored to the secondary system / set of tags. As long as the timestamps are identical in the secondary system any statistical anaylsis should give correct results when these are applied to the value timestamps.


              Looking at the realtime requirement, this is a bit odd as you want to have a 24-hour delayed realtime. UFL is very fast, but pushing out datafiles every second or even less for ULF to process is not really practical. If the compression settings prevent usage of the archived values to send to the secondary system, you could create a tagset that has no (effective) compression set to drive the secondary datasets.


              I think there's a whole range of options available, even with PI Out-of-ox functionality. But to give some targeted advice one would need to understand the use-case a bit more.


              My choice given the information provided: disable compression, and have a custom service that reads e.g. a 5 minutes window of archive data of -24hr back in time into a buffer, and send that out to e.g. OPC on the exact millisecond. That should not be too difficult, and saves you the issue to buffer 24-hours of data in the application.


              Hope this helps!

                • Re: Delay data from the incoming tag stream to another tag

                  Thanks for the feedback.  I can only work with the requirements as they've been defined and it's pertty much as I had mentioned.  Sharing anything more on background I could do one on one but not in a public forum as this.  This is synchrophasor data and the data needs to come in to the snapshot, for the most part in a continuous fashion just as it does from the interface.  This is necessary for the types of analysis being done on synchrophasor data of an electrical system.  However, I'll push back a bit and see if I can get some relief from that.


                  The approach Gopal outlined above is interesting as well, however, it could be a little choppy and will be at the speed of the UFL interface and a flat file, as opposed to a PMU and C37.118 interface.


                  The idea of delayed data in the Utility space is pretty common so I'm guessing data is shared (if at all) using something like Gopal described, i.e. past queries, create a file, load the file.  For most other data than synchrophasors and the specific analysis being done with this data for this particular use case, that likely works fine.


                  I was hoping someone had written a program to address/manage this already.


                  Thanks again.   I'll get some additional feedback

                    • Re: Delay data from the incoming tag stream to another tag

                      It's fairly easy to put together something working and check out for about 10 or so tags using the AFOLEDB approach I suggested to give the customer a feel for it. Also, note that you can have multiple instances of UFL.

                        • Re: Delay data from the incoming tag stream to another tag

                          You have other considerations here too if you are using the PI Server as the source - compression. If you are compressing in the PI Server then you have no choice but to work off the snapshots. If you are not compressing then you could take a batched archive retrieval approach, which doesn't have to be the full 24 hours in one buffer. Quite like Gopal's approach to this but if you truly need to stream the data onwards (real-time situational awareness simulation?) then my gut is telling me you need to crack open Visual Studio (with AF SDK instead of PI SDK)  because you need specific control over the update frequency that I doubt you will get with any off the shelf interface.




                          Replay/playback services are more common than you think, but almost all of the ones I've seen have been custom built for their specific purpose. None of those custom playback services could handle streaming data back to itself at 60Hz without modification.