6 Replies Latest reply on Sep 7, 2018 8:30 PM by kbobeck

    How does the Buffering Manager calculate estimated buffer capacity?

    kbobeck

      We've had our PI Server go down twice in the past four days. The first time, we had 70GB of space available, which it used up in 24 hours for around 100 million events.

       

      The second time, it had around 30GB of space, which it used up in around 54 hours for about 1.1 BILLION events.

       

      It's currently processing the data and is saying, with 48.3 GB of space available, there are 601.1 days of estimated buffer capacity. This is clearly quite off.

       

       

      I've already made the suggestion of moving to a collective to prevent having so many events queue up, but out of curiosity, what is the Buffering Manager using to calculate the estimated capacity? I would like to understand how it could be so off...

       

      Thanks,

      Kelsey

        • Re: How does the Buffering Manager calculate estimated buffer capacity?
          KarlRickards

          Hi Kelsey,

           

          Can't answer your immediate question but moving to a collective will not solve your issue with event ques.  With a Data Archive Collective, PI Buffer Subsystem uses N-Way Buffering which 'fans' the data to all members of the collective and as such, a separate que file is created for each member.  When a collective member goes down, the que file for that member starts growing until the member comes back online and the respective que file flushed.

           

          Cheers,

          Karl

          1 of 1 people found this helpful
            • Re: How does the Buffering Manager calculate estimated buffer capacity?
              kbobeck

              Hi Karl,

               

              Thanks for your reply. I may be misunderstanding (PI System architecture is not my area of expertise), but that seems a bit counterintuitive. One of the main reasons we'd want to move to a collective is to maintain data availability, as, after an unplanned outage with a single Data Archive architecture, we have to shut down the snapshot subsystem to let the event queue flush and we have to let the buffer queue flush. Each of these steps can take upwards of 12 hours, depending on the amount of time the system was down, which means the system isn't showing current data for potentially 24 hours (not counting any troubleshooting time). In a collective, once the Primary Data Archive is back online, wouldn't the system then prioritize that server and therefore, the data would not be available because the server is processing either the event queues (if we turned off pisnapss) or the buffer queues?

               

              Thanks,

              Kelsey

              1 of 1 people found this helpful
                • Re: How does the Buffering Manager calculate estimated buffer capacity?
                  KarlRickards

                  No your reasoning for moving to a Data Archive Collective to maintain Data Availability is accurate.  My comment is around your statement "moving to a collective to prevent having so many events queue up".  In a Data Archive Collective, the data between the members is not automatically synchronized rather the PI Buffer Subsystem (from each interface) sends data to all members simultaneously.  It is not a redundant switch over when one is not available the buffer will send to the other then when the member that went down comes back it back fills from the other members, the qued data for the member that went down will be flushed once it comes back online.  This is why the que files will still build up if one member of the collective goes down, when it comes back online the qued files for that member will be flushed where there will be no qued files for the member that stayed online.

                   

                   

                  I suppose the easiest way to think of it is when the PI Buffer Subsystem is used in a Data Archive Collective, the Buffer is replicated for each additional member working independently of the other from a simplistic viewpoint.  The buffer responsible for sending data to a particular member of a collective will still buffer data for that member when it is not available and as such, will not alleviate your issue having events que up.

                   

                  Hope that helps,

                  Karl

                  1 of 1 people found this helpful
              • Re: How does the Buffering Manager calculate estimated buffer capacity?
                tramachandran

                I'll try to answer ...but out of curiosity, what is the Buffering Manager using to calculate the estimated capacity?...

                The estimated capacity serves as a guideline until data loss if all servers were disconnected but the method of calculation itself, which is performed by the subsystem, is usually not very useful for end user.

                 

                However to address your curiosity (and mine) I tried to dig into the documentation. I don't think this information is readily available (at least in the recent versions of the the Buffering Manager Manual), so I had to go back a few versions (2008

                PI Buffer Subsystem User Guide Version 3.4.375/PR1 SP1 ).

                "At any time, Available Pages shows how many empty pages are left in the current queue.

                Based on the average event size, the Buffer Subsystem maintains the number of Average

                Events per Page. From this it derives an Estimated Remaining Capacity in number of events."

                "The Buffer Subsystem creates and links multiple

                queue files together, organized in a numbered sequence. This mechanism makes the overall

                buffer capacity fully dynamic and limited only by the available disk space."

                 

                You can check out a more detailed discussions in the Memory-Mapped Buffer Queues and Queue File Sizing sections of the manual.

                These statistics can easily be accessed using pibufss -qs command.

                 

                The explanations for calculating capacity are from a considerably old version of PIBufss, so these need not hold true for later versions. However, it would be a reasonable guess as to how the parameters are calculated.

                 

                I cannot answer why they might be off, but again I do not have more information to work with. If you still like to investigate the issue you can refer to the Performance monitoring for PI Buffer Subsystem in the user manual (newer versions) or open up a case with support.

                  • Re: How does the Buffering Manager calculate estimated buffer capacity?
                    rschmitz

                    So this got me curious as well and went digging for some info. It turns out "Available Pages shows how many empty pages are left in the current queue. Based on the average event size, the Buffer Subsystem maintains the number of Average

                    Events per Page. From this it derives an Estimated Remaining Capacity in number of events" is only part of it.

                     

                    So the buffer subsystem knows events/page and # of pages left, we still need events/sec to find how much time to figure out how much time we have before running out of space. When the buffer subsystem does a calculation for the number of events/sec, it just takes a running average of the previous 5 minutes or so (which is what you see displayed in the Buffer manager). This means if you have a scan class that runs once every 10 min, 30 min, 12 hours that dumps huge amounts of data into the buffer the events/day can much different than events/sec or events/min. So given a situation like having a UFL interface that runs once every 8 hours or a RDBMS interface that runs every now and again that puts a few millions events into the buffer you could get an incorrect estimate for the amount of remaining buffer capacity.

                     

                    --Rob

                     

                    P.S. I don't imagine this is the case, but if other files are moved onto the D: drive while things are being buffered could also potentially mess up the estimate.

                    1 of 1 people found this helpful
                      • Re: How does the Buffering Manager calculate estimated buffer capacity?
                        kbobeck

                        Thanks, Rob, that was what was causing the disparity - we have RDBMS queries that run every 15 minutes and every hour, which were bringing in a few hundred thousand rows to several million rows at a time. While not (originally) knowing why the buffer queue capacity was so off, I recognized that those would fill up available space quite quickly regardless, so I've since completely reconfigured the queries to pull in significantly less data.

                         

                        I appreciate you solving the mystery. In your investigation, did you happen to come across a way to modify the calculation period? Even an hour would be a much more representative period over which it could calculate the remaining capacity. Either way, I'll have to make sure to take the queue capacity with a grain of salt going forward.

                         

                        Thanks again,

                        Kelsey