14 Replies Latest reply on Nov 7, 2013 10:17 PM by ssortland

    PIBufss auto-tuning throughput

      Anyone else see the need for an auto-tuning MAXTRANSFEROBJS with the PI Buffer Subsystem?  PIBufss should know when it has slipped into backfilling mode and adjust (increase) MAXTRANSFEROBJS until its buffers are running empty and it tunes back down to a "streaming" setting that was originally set.

        • Re: PIBufss auto-tuning throughput

          Hello Rhys,

           

          MAXTRANSFEROBJS specifies the maximum number of events to be send in a single call. SENDRATE defines the minimum wait time in milliseconds between successive data posts. Both parameters together allow to tune the data rates between PI Buffer Subsystem and the receiving PI Server(s).

           

          When PI Buffer Subsystem has built up a queue, there is usually a good reason. Increasing MAXTRANSFEROBJS can also have a negative impact e.g. when dealing with a less reliable network.

           

          On the other hand if data sources produce a high amount of events and the link(s) between interface node and PI Server node(s) is / are with good performance, you can increase MAXTRANSFEROBJS to allow a higher throughput. Please see TechSupport article KB00158 for details.

           

          KB 2774OSI8 is about how much buffering space and network bandwidth is required on a PI Interface node.

           

          Gregor

            • Re: PIBufss auto-tuning throughput

              Hi Gregor,

               

              That is kind of my point here.  You can tune PIBufss to current network conditions but lets say that same network can fluctuate (assume here no traffic prioritisation) and cause your tuned settings to have an adverse affect on your backfilling.  Would be great for PIBufss to auto tune, like each bucket of events is taking longer to be acknowledged than events are filling the next bucket, I better send smaller buckets until I have room to throw the bigger buckets again.

                • Re: PIBufss auto-tuning throughput

                  Hello Rhys,

                   

                  Rhys @ Wipro

                  You can tune PIBufss to current network conditions but lets say that same network can fluctuate (assume here no traffic prioritisation) and cause your tuned settings to have an adverse affect on your backfilling.

                   

                  What are you referring to with "backfilling"? Is that your use case? Data from a backfilling process queued up in PI Buffer Subsystem or are you referring to processing the data queued in the buffer?

                   

                  To stick with your analogy, the bucket size would be MAXTRANSFEROBJS. The SENDRATE specifies the pause between receiving the confirmation the latest bucket was delivered successfully and sending the next bucket. Would just using smaller buckets really reduce the time until the buffer queues are emptied? I believe the opposite would be the case. The time that is necessary to send a single bucket is directly impacted by the performance of the network, by the currently available bandwidth and the network latency. Choosing smaller buckets with a high latency network would be a bad decision as well as increasing the SENDRATE.

                   

                  At this point I like to introduce the term "bulk rate" as the analogy for the amount of events processed in a single second. Let's assume when recognizing poor performance e.g. by comparing the current bulk rate against numbers recognized earlier for the same network, PI Buffer Subsystem would decide to reduce the bucket size and to increase the amount of buckets send per second. Within the next second it recognizes that the bulk rate has indeed increased. Logical consequence would be reducing bucket size again and sending more buckets until an optimum is reached. Please note there are already 2 screws driven at the same time. What screw has really caused the good impact? We cannot assume the optimum reached is really an optimum because network performance could have changed again. I believe that there are too many unknown variables to really react and network performance can be changing too fast. 

                   

                  We would need more information. We could measure network latency and collect performance counter information of the local network adapter. We could as well collect performance counter information for the network adapter of the receiving PI Server. I doubt this would be of much use since we know that in real life the bottleneck sits somewhere in between.

                   

                  Gregor

                    • Re: PIBufss auto-tuning throughput

                      Hi Gregor.  These conversations are normally easier to have in person, but here goes anyway...

                       

                      I would actually like to be able to say to PIBufss 'Last In First Out', but there is a PLI for that one already.

                       

                      My use case is to empty the buffered data as fast as possible to get back to real time streaming of data; typically having the current values is higher priority than filling in the gaps.  There are ways to do this but not via PIBufss...but we need PIBufss because it handles network disconnects better.

                       

                      What I have observed is latency is not always my problem but bandwidth saturation is.  In one scenario backfilling 20 million events I set PIBufss to have 20,000 MAXTRANSFEROBJS and noted the time for ACK to be returned (peaks between total events sent/sec).  I increased it to 25,000 MAXTRANSFEROBJS and the peaks got slightly wider (expected).  I decreased it to 15,000 MAXTRANSFEROBJS then the peaks were closer together (expected), by more distance than the increase in peaks for increasing MAXTRANSOBJS (unexpected) - over a period of time total events sent was greater with the smaller buckets.  So 15,000 seemed to be the sweet spot.  Fast forward a couple of weeks, the same type of outage & buffered data occurred, only this time the network is less saturated.  The 15,000 MAXTRANSFEROBJS is no longer the sweet spot, 20,000 is (yes the saturation can cary that much).    You could just leave 15,000 and live with that being the average performance for backfilling.  But I am talking about having this on a larger scale, like 300 instances each with different network connections where latency and bandwidth varies.  Right now I just have to classify the connection, take an average of available bandwidth + latency to define the average PIBufss performance then leave alone.  It would be great if PIBufss auto tuned but I appreciate the complexities in doing this.  The auto-tuning would always be within a range, like MAXTRANSFEROBJS 30,000 - 5,000.  SENDRATE would be 0 during backfilling, 100 during streaming.

                       

                      In a backfilling where the network latency is greater than the send rate, does the send rate really come in to play?

                        • Re: PIBufss auto-tuning throughput

                          Hello Rhys,

                           

                          Rhys @ Wipro

                          These conversations are normally easier to have in person

                           

                          I would be happy continuing this discussion at the developers launch during vCampus Live! 2012. Maybe one of the developers is joining us.

                           

                          Rhys @ Wipro

                          I would actually like to be able to say to PIBufss 'Last In First Out', but there is a PLI for that one already.

                           

                          First I thought First In, Last Out might be an issue because of the compression algorithm but the PI Buffer Subsystem queue is similar to the PI Snapshot Subsystem queue and hence events should be compressed already. In my naïve opinion maintaining the stack top to bottom should be possible. Since a PLI exists already for this I do not see any reason to address the issue a second time - especially because PI Buffer Subsystem throughput auto-tuning isn't trivial.

                           

                          Rhys @ Wipro

                          The 15,000 MAXTRANSFEROBJS is no longer the sweet spot, 20,000 is (yes the saturation can cary that much).    

                           

                          20,000 events in a single post is quite a number and we are talking about compressed events!

                           

                          Rhys @ Wipro

                          In a backfilling where the network latency is greater than the send rate, does the send rate really come in to play?

                           

                          After checking with OSIsoft Buffer and Bandwidth Calculation Spreadsheet, I conclude the SENDRATE becomes obsolete at all in case the number is smaller than network latency.

                           

                          Gregor

                            • Re: PIBufss auto-tuning throughput

                              Hello Rhys,

                               

                              This time I have to quote myself.

                               

                              Gregor Beck

                              I would be happy continuing this discussion at the developers launch during vCampus Live! 2012.

                               

                              It's the developers lounge. We are not launching anybody.

                               

                              Gregor Beck

                              In my naïve opinion maintaining the stack top to bottom should be possible.

                               

                              I somehow felt that it cannot be that easy. PI Archive Subsystem might be running into a serious issue when sorting those out of order events.

                               

                              Gregor

                                • Re: PIBufss auto-tuning throughput

                                  Gregor Beck

                                  It's the developers lounge. We are not launching anybody.

                                   

                                  That would be quite fun, launching developers who are taking too long to develop an OSIsoft product *cough* abacus *cough* SSB *cough*

                                   

                                  Gregor Beck

                                  I somehow felt that it cannot be that easy. PI Archive Subsystem might be running into a serious issue when sorting those out of order events.

                                   

                                   

                                  That is one of the talked about features of PI Server 2012 though, right?  Sorting high volume of out of order events is not so much of an issue anymore (I haven't had time to test), we don't need to use tuning parameters like ArcInsertRecOnOOO.

                                   

                                   

                                    • Re: PIBufss auto-tuning throughput

                                      Hello Rhys,

                                       

                                      I am very exited about the upcoming PI Server 2012 release, the performance improvements and other features introduced with that version.

                                       

                                      Rhys @ Wipro

                                      That is one of the talked about features of PI Server 2012 though, right?  Sorting high volume of out of order events is not so much of an issue anymore (I haven't had time to test), we don't need to use tuning parameters like ArcInsertRecOnOOO.

                                       

                                      Tuning parameter ArcInsertRecOnOOO comes only into play with archive reprocessing - not with normal operation. I have checked the PI Server 2012 Release Notes and believe you are referring to PLI # 15741OSI8.

                                       

                                      Having events sorted in a timely order is a requirement for fast data retrieval and I do not see this would ever change.

                                       

                                      Gregor

                                        • Re: PIBufss auto-tuning throughput

                                          Gregor, remember this conversation from last year? With this in mind here is a quote from the PIBufss 4.3 CTP:

                                           

                                          PIbufss now sends data 2-3 times faster than previous versions. This not only means higher sustained data rates, but it also means that queued data after a network outage will be unloaded faster, resulting in users getting live updates that much sooner. Key to achieving these gains is a new Autotuning feature that attempts to optimize the send rate when data is queued up.

                                           

                                           

                                           

                                           

                                           

                                          Edit: I would be interested to understand the autotuning algorithm some more if that is available. Is it related to size of the queue, latency to server, bandwidth saturation...

                                            • Re: PIBufss auto-tuning throughput

                                              Wow Rhys,

                                               

                                              Do you believe PI Buffer Subsystem developers read this thread and decided to make me looking stupid? Sometimes my mind is simply too limited.  I am curious as you are to learn more about the algorithm and also in what timely order events are send. I've asked Stefan to share some insights. 

                                               

                                               

                                                • Re: PIBufss auto-tuning throughput

                                                  Well you know I have a feeling the autotuning is only good for low latency networks where the sendrate comes into play. For high latency you ideally want the maxtransferobjs to autotune, but doubt that is the case (although I am testing for that).

                                                    • Re: PIBufss auto-tuning throughput
                                                      ssortland

                                                      The algorithm automates what you previously had to do manually: that is tweak the send parameters in order to optimize throughput. So indeed we are adjusting the number of events to send at a time (“maxtransferobjs”)

                                                       

                                                      The auto-tuning algorithm kicks in when we connect to a server and determine we have a certain the number of events queued up (currently this threshold is 12 queue files worth of data) or we are receiving more data than we can send with the current parameters.

                                                       

                                                      To monitor auto-tuning, you can look for message ID 18130 and 18131 that get written when we start and end a round of auto-tuning.  Note: currently you may see a message indicating we are starting auto-tuning (18130) every time we reconnect. But if there are insufficient number of events queued up then we will not complete the auto-tuning process (you won’t see a message 18131).

                                                       

                                                      We also have performance counters that will track what the algorithm is doing, but those are mainly for troubleshooting. The main one to pay attention to is the “PI Buffered Physical Servers>Autotune> Mode” counter will have a value of 3 while auto-tuning and 4 when auto-tuning is complete.

                                                       

                                                      We have done testing internally with simulated high latency networks, but would love to hear what your experience has been so far.

                                                       

                                                      What throughput are you seeing now with pibufss 4.3 vs before you upgraded?

                                                        • Re: PIBufss auto-tuning throughput

                                                          First of all this is great information, thanks. Does the auto-tune occur per member server of a PI Collective (I'm assuming it does)? Each member server has its own set of queue files so wondering if the auto-tune is focused on a PIBufss node's overall state, or the state of individual PI Servers. Can you make the threshold for auto-tune configurable? Do you take into account the latency of acknowledgements from the PI Server per bucket of events to know if you can keep tuning? On the last one I'm wondering if a fluctuating network connection would trick the auto-tuning into instability.

                                                           

                                                          I'll report back the results when we're done. I have some very specific conditions to test; bandwidth limited satellite links.

                                                           

                                                          The corrupt queues from outages/power failures have been a pain in the a$$ so looking forward to testing that also.

                                                           

                                                          If PIBufss is starting to have intelligence of its own current state does that mean we will start to see it get more intelligent for other scenarios? For example, it seems to be heading nicely towards Last In First Out capability with some auto-tuning to background back fill the queued data whilst giving priority to snapshots. Speaking of prioritizing data via PIBufss... I'll stop there for today.

                                                            • Re: PIBufss auto-tuning throughput
                                                              ssortland

                                                              Yes auto-tuning occurs independently on a per member basis

                                                               

                                                              Yes the threshold is configurable

                                                               

                                                              We measure the entire round-trip time when performing auto-tuning, not just the time to send.

                                                               

                                                              If network conditions are fluctuating e.g. changing in the middle of a round of auto-tuning then it would be possible to get a sub-optimal state. But if events start backing up then pibufss will kick off another round of auto-tuning. But indeed if network conditions are constantly fluctuating then what does it mean to be optimal then? We may eventually have to put a cap on the auto-tuning algorithm but that sort of tweaks we've been holding off on until we understand how things are working in real-world conditions.