6 Replies Latest reply on Apr 23, 2018 7:57 AM by Olav

    PI Performance logging

    Olav

      Hi.

       

      We are suspecting that one of our PI Servers are not delivering the performance it used to, especially to processbook clients. The question is how we best can identify any problems here. It is more of a "gut feeling" from the users for now, so I need to map out any possible problems. There are no errors in the message log. We have the pi performance monitor interface installed, but it is not used for much at the moment. Which parameters should be logged that can tell us anything useful here?

       

      Are there any tuning parameters that should be looked into for increasing the read performance of the system? Any potential settings that could degrade performance?

       

      Olav.

        • Re: PI Performance logging
          afink

          Hi Olav,

           

          First off, which version of the PI Data Archive are you running?  What OS is the server running on?  And what is the hardware available on the machine?

          I would recommend leveraging the PI Performance Monitor interface by creating and monitoring the following performance counters mentioned in the article: Which Performance Counters to Monitor.  These could then be looked at over time if there is change in use of the server.

           

          Are there any particular times of day or intervals where the performance seems slower than usual?  Perhaps there is another scheduled operation going on interfering with normal operation.

          The Online Hardware Sizing Tool can also be used to get a general idea if your usage of the server might warrant increased resources to the machine.

           

          - Adam

          1 of 1 people found this helpful
            • Re: PI Performance logging
              Olav

              PI Data Archive version is 3.4.405.1198 and OS is Win Server 2008 R2. The performance is mostly when trying to open a larger number of tags and time intervals. I know this could be a complex issue. Hardware wise I think we are Ok. The server is running on a VM Ware virtual machine. For some reason the system does not seem to utilize the full disk performance.

               

              I will check out your links. Thanks for the advice.

                • Re: PI Performance logging
                  afink

                  Hi Olav,

                   

                  To your point, "The performance is mostly when trying to open a larger number of tags and time intervals":

                  Note that the PI Data Archive utilizes both it's own Archive Cache as well as the Windows File System Cache to cache data in memory after it is requested for the first time.  This is based on the assumption that data requested once will be requested again.  Since data retrieval from the cache (memory) is much faster than reading from disk, it is common that data requested for the first time will take longer to retrieve than when it is requested again.  For more information on the PI Data Archive and the File System Cache, check out KB00784.

                   

                  However, note that this is a server-side cache, and if you are noticing this for many ProcessBook clients, it is possible that for some reason the data is not being held in the cache.  One possible reason is that there is not enough RAM on the machine.  So, when new data is requested and attempted to be cached, the old data can be flushed back to disk to make more room in memory for the new data. 

                   

                  To get a better idea of where certain data is being loaded from, combined with those mentioned in the previous article, the following performance counters can be very helpful:

                          PI Archive Subsystem\Cache Record Count

                          PI Archive Subsystem\Cache Record Memory Reads/sec

                          PI Archive Subsystem\Events Read/sec

                          PI Archive Subsystem\Read Operations/sec

                          PI Archive Subsystem\Record Disk Reads/sec

                          PI Archive Subsystem\Record Load Time Average

                          PI Subsystem(piarchss)\RPC Requests in Queue

                          PI Subsystem(piarchss)\RPC Threads Active

                          PI Subsystem(piarchss)\RPC Threads Total

                   

                  - Adam

                  2 of 2 people found this helpful
                    • Re: PI Performance logging
                      Olav

                      Hi Adam. Thanks for your input.

                       

                      I will monitor the performance counters you suggest. Some of those we already have. I am monitoring disk read MB/s at the moment, and what we found a bit strange, is that even when trying to load maybe 10 tags that go back 30 days in time, I can only se maybe 1-2 MB/S read from the disk. And processbook then hangs for maybe 30-40 seconds before the data comes in.

                       

                      Edit: I've now logged the performance counters you suggested. I've tried to load the server by updating a lot of tags with long timeframe. I can see the Events Read/sec going up rather much, and also the Disk Reads/sec, but I'm not sure how to interpret it. I can't see any obvious issues.

                        • Re: PI Performance logging
                          afink

                          Olav,

                          I agree that on your test it looks like the disk reads did spike a bit, however, this would be expected since this was the first time running the query for the data.  I would also expect that if you ran the query again soon after, the second query would execute much faster as these records could now be read from the cache, we can even see the cache record count increases when the query is run.

                           

                          Overall, I do not see anything of major concern here.  Although i cannot see the whole picture, there are no requests queued and does not look to be active threads working over long periods of time.  Though this is difficult to see with the scaling.

                           

                          Another potential bottleneck that hasn't been discussed yet is the network.  Are there any network diagnostics or stats that you have available?  What is the latency like on the network between the ProcessBook clients and the PI Data Archive?

                           

                          If you are able to verify the network is not subject to latency, let me know if you are interested and I can create a tech support case so that we can look further into the issue.  Let me know if this is something you would like to do.

                           

                          - Adam

                            • Re: PI Performance logging
                              Olav

                              Thx for your input. We will look at how the network is performing. The processbook application is opened from within a VDI, but is run from a Citrix server, so the network is not point to point between client and the archive.

                               

                              If we can't figure it out, I think we need to make it a support case.