6 Replies Latest reply on Jan 25, 2019 7:41 AM by Lal_Babu_Shaik

    PI Buffering running out of capacity - History Recovery

    Kenn043

      Hi Team,

       

      We are doing a back-filling activity where we have a PI HDA Interface reading from our SCADA source systems for ~10 years worth of data.

       

      We are trying to hit the 'sweet spot' on the number of tags and the duration we should recover in one go. We are finding that once we start the recovery with the settings below (which are pretty tame, would expect PI to handle without any fuss) the buffering capacity starts running out of space really quickly. The events in queue are around the 40000 events mark (this could be an issue?). Is there a setting we might have missed that is throwing the Buffering queue haywire?

       

      Thanks,

      Ken.

       

       

        • Re: PI Buffering running out of capacity - History Recovery
          Lal_Babu_Shaik

          Hi Ken

           

          Please find the details : PI Server

           

          It shows warning for PI Buffer subsystem in your case.

          Low buffer queue capacity (Critical)The disk containing the buffer queue is full. All incoming data for the affected server(s) is lost.If PI Buffer Subsystem is queuing data because the PI Data Archive server is down or connection to the server is lost, check whether you can bring up the server or reestablish the connection.If you cannot connect to the server right away, options include:
          • Delete or move other files on the disk to make room in the buffer queue location.
          • Use Buffering Manager to change the buffer location (Queue path) to a disk with more available space. In Buffering Manager, click the Settings link. In the Buffering Settings window, select the affected server, change its Queue path, and click Save.
          Low buffer queue capacity (Error)The disk containing the buffer queue is almost full. PI Buffer Subsystem can continue to queue data for approximately 24 more hours.Same as Low buffer queue capacity (Critical), except that you have approximately 24 hours to resolve the problem before data loss occurs.

          Caution:Changing the buffer location can cause temporary data loss.

          Low buffer queue capacity (Warning)The disk containing the buffer queue is low on space. PI Buffer Subsystem can continue to queue data for approximately 7 days.Same as Low buffer queue capacity (Critical), except that you have approximately 7 days to resolve the problem before data loss occurs.

          Caution:Changing the buffer location can cause temporary data loss.

            • Re: PI Buffering running out of capacity - History Recovery
              Kenn043

              Hi Lal, I am aware of the error status and its meaning. My query was on how to 'resolve' it and the optimal settings for the activity.

               

              Thanks,

              Ken.

                • Re: PI Buffering running out of capacity - History Recovery
                  Lal_Babu_Shaik

                  Ken, could you please check the buffer queue statistics with pibuffss -qs command and check the queue file size?  By default it is 32 MB allocated for queue file.  Based on your statistics you can increase the queue file size. QUEUESIZE parameter in piclient.ini or using buffer manager global settings you can change the queue file size.If you are manually estimating the buffersize then would request you to multiple the size * number of collective members as it creates multiple files based on collective members. Please note changes to QUEUESIZE parameter requires restart of the PI Buffer subsystem. If it is left with default size and rate is same approx 45k events it will take 4.9 days to complete one file and it completely depends on Total Event Writes to buffer and Reads from Buffer to PI. If there is situation where if Reads are very less compared to write to buffer then your file keeps growing. If space is available on the drive then it will create additional files in the same location.

              • Re: PI Buffering running out of capacity - History Recovery
                JEyth

                While the buffer subsystem is warning of low queue capacity (disk space), how is the "events in queue" counter trending? If you allow this interface to run for an hour or longer does this events in queue count continue to climb or does it stabilize. Based on the picture with the buffer subsystem transmitting 9316 events per second it looks like after 10 minutes of running there is 5 seconds of data in the queue. Without more information this isn't necessarily concerning and I'm not sure that the buffer subsystem autotune would be completely adjusted either so it might transmit the events faster as time goes on.

                 

                If the events in queue counter keeps growing after a longer run time, you could decrease the time period per call and/or increase the pause between history recovery to slow the flow of events into the queue.

                • Re: PI Buffering running out of capacity - History Recovery
                  Steve Boyko

                  How much space is available on the drive that is hosting the PI buffer queue files? 40,000 events is not a lot and shouldn't require a ton of disk space. I'm surprised that this warning is coming up.

                  • Re: PI Buffering running out of capacity - History Recovery
                    JEyth

                    Additionally, the Low Buffer Queue Capacity warning is an evaluation of the number of events being written to the buffer queue over a period of time and how long it would take to fill the drive if connection to the destination PI server was completely lost. The queue could clear every cycle, but if the number of events being written to the queue would fill the drive in less than 10 days you will get that warning.

                    1 of 1 people found this helpful