Working on an app to pull data out of PI with high volume and low latency. Currently just looking at the streaming data, in other words, snapshot updates.
When using datapipes we need to poll and the IObserver pattern gives a huge performance benefit over calling GetUpdateEvents. Performance in my measurement is about 600k events/sec for GetUpdateEvents and 6M events/sec for GetObserverEvents. Of course specific to my test setup. Although the manual polling using GetObserverEvents might bring better performance than a 'true' observer, it does move the problem to ourselves to solve: How often to call GetObServerEvents and using which maxEventCountPerServer?
On the call frequency:
- If i call continuously with no wait i consume a lot of CPU running through the loop. Not ideal.
- If i wait, i might miss a peak ingress of data (e.g. due to a buffer flushing) and get Lost events due to the UpdateManager queue overflowing.
On the block size:
- My gut feeling says this has some relation to the UpdateManager tuning, but the testing i did on performance did not give a clear outcome on that.
- I currently pull blocks of 10K events to reduce any overhead but i don't really get an idea where to set this value to. Why not just as high as the UpdateManager MaxUpdateQueue size to ensure i pull everything available?
Lower call frequencies increase latency, lower block sizes increase the probability to miss out on peak ingress. How to balance not wasting time on unneccesary calls to GetObserverEvents and yet be prepared for large peaks in data ingress? After all from the client side we cannot detect if we missed events, except if we monitor the Lost Events/sec performance counter on the PI Server itself (or the Pending Events versus MaxUpdateQueue setting).
Have been thinking about some throttling algorithm, but can't think of any sensible approach. Any ideas welcome!