22 Replies Latest reply on Sep 2, 2016 11:48 AM by Roger Palmen

    PIDataPipe GetObserverEvents: how often to call?

    Roger Palmen

      Hi all,

      Working on an app to pull data out of PI with high volume and low latency. Currently just looking at the streaming data, in other words, snapshot updates.

       

      When using datapipes we need to poll and the IObserver pattern gives a huge performance benefit over calling GetUpdateEvents. Performance in my measurement is about 600k events/sec for GetUpdateEvents and 6M events/sec for GetObserverEvents. Of course specific to my test setup. Although the manual polling using GetObserverEvents might bring better performance than a 'true' observer, it does move the problem to ourselves to solve: How often to call GetObServerEvents and using which maxEventCountPerServer?

       

      On the call frequency:

      • If i call continuously with no wait i consume a lot of CPU running through the loop. Not ideal.
      • If i wait, i might miss a peak ingress of data (e.g. due to a buffer flushing) and get Lost events due to the UpdateManager queue overflowing.

       

      On the block size:

      • My gut feeling says this has some relation to the UpdateManager tuning, but the testing i did on performance did not give a clear outcome on that.
      • I currently pull blocks of 10K events to reduce any overhead but i don't really get an idea where to set this value to. Why not just as high as the UpdateManager MaxUpdateQueue size to ensure i pull everything available?

       

      Lower call frequencies increase latency, lower block sizes increase the probability to miss out on peak ingress. How to balance not wasting time on unneccesary calls to GetObserverEvents and yet be prepared for large peaks in data ingress? After all from the client side we cannot detect if we missed events, except if we monitor the Lost Events/sec performance counter on the PI Server itself (or the Pending Events versus MaxUpdateQueue setting).

       

      Have been thinking about some throttling algorithm, but can't think of any sensible approach. Any ideas welcome!

        • Re: PIDataPipe GetObserverEvents: how often to call?
          pthivierge

          Hello Roger,

           

          GetObServerEvents

          Your tests are confirming what I was explained once: IObserver patters is more efficient because there is much less .Net objects to create at once and also the type of memory used stays low because the generations in garbage collection stay low.

           

          maxEventCountPerServer

          It looks to me that this is a client side setting, and the out hasMoreEvent parameter allows you to know if you should call it again.  So I believe you should set the maxEventCountPerServer to a value that is high enough to cover the normal operations, and in a situation when you get bursts of data, then you should look when the hasMoreEvent becomes true so you can continue calling the hasMoreEvent until it completes.  The utility of this can be that you may perform tasks in between two GetObserverEvents calls, also to potentially "throtlle" the data retrieval in smaller chunks on the PI Data Archive.

           

          call frequency

          Knowing about the average rate at which the data is coming to the server + the settings on the PI update manager :  TotalUpdateQueue  and MaxUpdateQueue.

           

          I am trying to remember something about an exception being raised when there was data lost from the pipe.

           

          --

           

          Let see if Jin Huang or David Pugal can help on this

           

          Hope this helps a little!

           

           

           

           

          1 of 1 people found this helpful
            • Re: PIDataPipe GetObserverEvents: how often to call?
              gregor

              Hi Patrice & Roger,

               

              PI Update Manager maintains a stack. If the stack is full, additional Events will overflow (become lost) and PI Update Manager will print "Update list overflow" and "Updates lost" messages to the PI Data Archives PI Message Log. I believe, among other related Technical Support knowledgebase articles, 3151OSI8 - MaxUpdateQueue and TotalUpdateQueue Recommendations has some useful information.

              It's important to not be too lazy in querying for updates but I would also not query PI Update Manager too frequent because it will cause the client to spike it's CPU usage. I believe something between 0,5 and 5 seconds could be a good starting point.

              There should as well be a Performance Counter exposed by PI Update Manager that informs about the amount of lost events. Possibly data from this counter could be used to implement some algorithm to programmatically find a reasonable update frequency.

              2 of 2 people found this helpful
                • Re: PIDataPipe GetObserverEvents: how often to call?
                  Rick Davin

                  Wow, Gregor's first paragraph cannot be understated!  His last paragraph mentions performance counters.  Here's a link to a white paper that documents those counters:

                   

                  PI Performance Monitoring White Paper (2006)

                   

                  See pages 19-21 for performance counters relating to PI Update Manager.

                   

                  Also Gregor's middle paragraph is the perpetual challenge of all developers: to find the magical sweet spot of performance vs efficiency be it in frequency of updates, creating threads, or sizing collections, etc.  There is never a hard-and-fast rule for this. 

                  1 of 1 people found this helpful
                  • Re: PIDataPipe GetObserverEvents: how often to call?
                    Roger Palmen

                    Hi Gregor,

                    Yes, studied that KB in detail to determine how to prove my tool works. The tricky part is that ideally one would need to monitor the current PendingEvents size and see if that increases or goes towards the set limit. However that moves the polling issue towards polling that value faster than just pulling updates. And architectually horrible: monitoring the system that feeds you to see if it can feed you fast enough? But hey, anything that works... Permissions nightmare though, and perfmon instance is dependant on processID, PI connection ID, etc. So not static, not trivial.

                     

                    Regarding the spikes, would that be on the UpdateManager process?

                    And looking at the 0.5 - 5s mark. What is that based on? Is there any relation to the blocksize i'm requesting? Any time would suit me (although it does impact latency), but if i loop slower each loop needs to process more data. So don't really understand the reasoning behind these numbers yet.

                  • Re: PIDataPipe GetObserverEvents: how often to call?
                    Roger Palmen

                    Hi Patrice,

                    It was your article on datapipes that moved me into that direction. But the performance difference is even more drastic than i expected!

                    I'm already looping immediately without a wait when hasMoreEvents. I'm more looking into how fast to call when no more events, so if there is some way the pipe knows or can tell that there were dropped events, that would be great! We do have an errors collection. Did not look into that yet... I could try to overload my queue and see what's in there...

                      • Re: PIDataPipe GetObserverEvents: how often to call?
                        pthivierge

                        I'll ping internally to see if we can have more insights on this.

                         

                        Stay tuned

                          • Re: PIDataPipe GetObserverEvents: how often to call?
                            jhuang

                            Looks like we all agree that the hasMoreEvent flag helps us a lot here.  When hasMoreEvent is false, I would wait for a couple of minutes to avoid unnecessary CPU stress on the client machine.  Of course, this is based on the assumption that the queue in PI Update Manager has enough space to hold the upcoming events.  On an up-to-date PI Data Archive machine with enough RAM, it is okay to set MaxUpdateQueue/TotalUpdateQueue to be much higher than the default values, such as 5,000,000 / 50,000,000 .  This will also help if we lose the connection between PI Data Archive and Client node for some reason such as network outage.  Therefore, even when there is a huge burst, for example, 1 Million snapshots / sec for all PI points, the queue can still hold the events for at least a few minutes considering the client should only sign up for the updates on a small fraction of the points. 

                             

                            Hope it helps.

                            2 of 2 people found this helpful
                              • Re: PIDataPipe GetObserverEvents: how often to call?
                                Roger Palmen

                                Waiting a few minutes is a bit too low. I throughput 6M events/sec for my performance tests, and target is a system with 1M PI Point signups. But this does enhance my idea on the numbers to use.  Polling 10 times / second should be more than ever needed (assuming we empty the entire queue in that time). We can tune down if needed, down to 1 / 5 / 10 seconds. Al least client CPU i have to spend all by myself. More worried about Server CPU, as we have 500+ clients connecting to PI Server.

                                  • Re: PIDataPipe GetObserverEvents: how often to call?
                                    jhuang

                                    Hi Roger,

                                     

                                    Yeah, the wait time can be tuned based on the nature of the data in the target use case, I do not think there is a magic number which fits all cases.  If there are no new events in the queue, we do not need to keep making RPC calls to PI Data Archive anyway.

                                     

                                    With a throughput of 6M events/sec, 1M PI Point signups and 500+ clients, you are really pushing the limits on PI Data Archive.  What use case do you have in mind? 

                                      • Re: PIDataPipe GetObserverEvents: how often to call?
                                        Roger Palmen

                                        The source is one of the larger PI systems around and should be highly optimized. We are trying to push about 10% of the PI Point data out to another platform for further processing (big data analytics / streaming analytics). My client is just one of those 500+ clients though. A typical processbook display should have a much lower load (although you can build a "ProcessBook of death" if you really try... )

                                         

                                        My theory was that if you can ingress the data into PI, you can definitly stream it out. The architecture of PI is well optimized for that.

                                         

                                        PS: The 6M events/sec i achieve just on a simple 10K point PI Server in a VM on my laptop. I do have archiving turned off though for 99% of the PI Points when i do bulk testing, as i'm just interested in the consumption side of PI.

                                  • Re: PIDataPipe GetObserverEvents: how often to call?
                                    dmoler

                                    If the pipe overflows, there will be an AFDataLossException inserted in the PIServerErrors collection keyed with the PIServer that had the overflow returned from GetObserverEvents - so it is possible to detect this situation (it's important to examine the return from GetObserverEvents to see if there were errors - the return will be null if there aren't any).  My recommendation is to make sure the PI Data Archive tuning parameters (MaxUpdateQueue and TotalUpdateQueue) are set so that there is a reasonable buffer when there is a burst of events.  Then poll something like every 0.1 to 5 s (depending on how low you want the latency - the worst-case latency is 2x the wait time) and skip the wait if there are more events. The call to get events is fairly cheap if there are no events so there shouldn't be any need to query the PI Server other than polling for events.  If you want the lowest possible latency, it isn't unreasonable to poll in a loop - it will be effectively throttled by the RPC time of the get update events call.

                                    1 of 1 people found this helpful
                                      • Re: PIDataPipe GetObserverEvents: how often to call?
                                        Roger Palmen

                                        Hi David,

                                         

                                        That was what i was looking for. Although the bias of the thread shows we should be fine if properly tuned and we pull data fast enough, this at least gives some handle to work on.

                                        Using the AFDataLossException i can tune up the polling frequency if needed. Although that's too late as we already lost data, it's the only way i think we know anything about the condition of the datapipe without looking at the performance counters.

                                        • Re: PIDataPipe GetObserverEvents: how often to call?
                                          Roger Palmen

                                          Hmm, when it comes to the point of testing, i need to dig somewhat deeper. There is an AFerrors collection returned on the GetObserverEvents call, but that does not return any errors if i push the consumer-specific queue into overflow...

                                          So reading carefully, i need to get the errors on the PI Server? Paging through the AFSDK documentation did not point me to where to look for that. Any clues?

                                          • Re: PIDataPipe GetObserverEvents: how often to call?
                                            Roger Palmen

                                            Also found this nice thread on the parsing of the error collection: Handling Errors returned by GetObserverEvents() method of a AFDataPipe

                                             

                                            Exactly what i need, however my errors collection stays NULL even if i overflow the datapipe. I do only overflow the queue on my own (consumer) pipe, not the overall queue. Will experiment some more...

                                             

                                            My code (key snip):

                                            pprivate static AFErrors<PIPoint> pipeErrors;
                                            pipeErrors = piPipe.GetObserverEvents(piPipe_EventBlock, out pipeHasMoreEvents);
                                            
                                            1 of 1 people found this helpful
                                              • Re: PIDataPipe GetObserverEvents: how often to call?
                                                dmoler

                                                Hi Roger,

                                                 

                                                On the first call to GetObserverEvents after the pipe overflows on the server, the returned pipeErrors should be non-null and there should be an exception in pipeErrors.PIServerErrors keyed on that PIServer with the data loss exception.

                                                 

                                                Which version of AFSDK are you using in your testing?  There was an issue in 2.8 with data pipes to PI that was resolved in 2.8.2 (available on the downloads page).

                                                  • Re: PIDataPipe GetObserverEvents: how often to call?
                                                    Roger Palmen

                                                    Using v2.8.2.7626 so that should be recent.

                                                    Test method should be ok. In any case i can only receive an overflow for my own queue when i have registered my pipe, so any overflow on that queue should come down the line in the errors collection. I'll have a closer look later today.

                                                    • Re: PIDataPipe GetObserverEvents: how often to call?
                                                      Roger Palmen

                                                      Did some more testing but with similar results. I have tuned my writes and read such that get an overflow every 12 seconds, but never receive a non-null AFerrors collection. Really puzzled here...

                                                       

                                                      My code loop:

                                                       

                                                                          //Get all Events in the pipe, using pipeHasMoreEvents
                                                                          do
                                                                          {
                                                      
                                                      
                                                                             //Pull updates into the pipe
                                                                              pipeErrors = piPipe.GetObserverEvents(piPipe_EventBlock, out pipeHasMoreEvents);
                                                      
                                                      
                                                                              //Log if there were errors in the pipe, stub for future extension
                                                                              if (pipeErrors != null)
                                                                              {
                                                                                  nlog_stats.Warn("Errors in PIDataPipe", pipeErrors);
                                                                                  pipeErrors = null; //clear
                                                                              }
                                                                          } while ((pipeHasMoreEvents) && (blockCount < piPipe_MaxAsyncBlockSize)); 
                                                      
                                                      1 of 1 people found this helpful
                                          • Re: PIDataPipe GetObserverEvents: how often to call?
                                            dmoler

                                            Hi Roger,

                                             

                                            I added to the PIDataPipe discussion elsewhere.  Regarding the maxEventCountPerServer parameter - this balances the efficiency (bigger chunks) with timeliness (events won't be processed until all data is transferred to the client).  If you are operating on a reasonable LAN network, 10k-100k events are pretty quick to transfer and it isn't much more efficient to make the size bigger.  If the network is slower, you might want smaller chunks.

                                            1 of 1 people found this helpful