10 Replies Latest reply on Sep 13, 2017 5:50 PM by skwan

    PI Analysis Service_Maximum Latency

    Paurav Joshi

      Hello All,

       

      Recently while monitoring my analyses, we found suddenly all analyses stopped running, and physical RAM utilization (PIAnalysisProcessor major factor) hits 99%. Restart of Analysis Service does the job, but want to do some debugging.

      While analyzing AF analytics parameters trend, we found immediate rise in trend of parameter PI Analysis Service_Maximum Latency.

      We are monitoring following parameters as of now for AF Anlaytics:

       

      My query is which are affecting paramers to PI Analysis Service_Maximum Latency?

       

      Thanks,

      Paurav Joshi

        • Re: PI Analysis Service_Maximum Latency
          gregor

          Hi Paurav,

           

          My first guess would be available memory resources but I suggest you to get in touch with Technical Support to troubleshoot PI Analysis Service falling behind.

          • Re: PI Analysis Service_Maximum Latency
            nagarwal

            Hi Paurav - As Gregor Beck suggested, I think best would be to reach out to OSIsoft Technical Support, as it's hard to say anything without knowing all the details about your system.

             

            The increased latency that you are seeing, could be both the cause and effect of the PI Analysis Service consuming all available memory on your machine:

            • In some cases when your system in overloaded (for a variety of reasons), such that events are being retrieved faster than they can be processed, this could cause some internal queues to continue to grow - causing the analysis processor to consume all memory. This becomes more likely when load shedding is disabled or you have expensive analyses with dependencies.
            • If analysis processor already consumed all memory (e.g. machine running analysis service not provisioned correctly), this could cause latency to increase as everything slows down due to page faults (memory swapping in and out of disk).

             

            What version of PI Analysis Service are you using?

            Thanks,

            Nitin

              • Re: PI Analysis Service_Maximum Latency
                Paurav Joshi

                Thanks Gregor Beck and Nitin Agarwal for comments.

                I am using PI Analysis Service 2015 R2.

                 

                I agree with your techsupport query, and previously (like 3 to 4 months back) I did this. Physical RAM usage was upto 12 GB for PIAnalysisService (contributing major) of 99% physical RAM utilization. We shared dump files like @ every 2 GB till once again % usage goes to 99, and after sharing all dump files thrice we are unable to find anything substantial. All the time ACE calculations were running fine without any problem.

                 

                This time I saw everything was normal in trends other than spike in PI Analysis Service_Maximum Latency.

                The increased latency that you are seeing, could be both the cause and effect of the PI Analysis Service consuming all available memory on your machine

                I am definitely in agreement here.

                 

                If analysis processor already consumed all memory (e.g. machine running analysis service not provisioned correctly), this could cause latency to increase as everything slows down due to page faults (memory swapping in and out of disk).

                We are monitoring following parameters for checking machine details like page faults mentioned here, and after restarting service all parameters mainly Page Faults/sec came back to normal state. Abnormality starts sometime (10 mminutes around) after spike in _Maximum Latency trend.

                 

                In some cases when your system in overloaded (for a variety of reasons), such that events are being retrieved faster than they can be processed, this could cause some internal queues to continue to grow - causing the analysis processor to consume all memory. This becomes more likely when load shedding is disabled or you have expensive analyses with dependencies.

                We have load shedding disabled, because of two reasons:

                1. There are only some analytics calculations which we can't afford to skip

                2. As of now, only server level load shedding option is available.

                 

                One query here, If load shedding has been disabled then still value of these parameters - EvaluationsToQueueBeforeSkipping and MaxCacheEventtsPerAttribute - have any significance.

                 

                Thanks,

                Paurav Joshi

                  • Re: PI Analysis Service_Maximum Latency
                    gregor

                    Hi Paurav,

                     

                    I believe there is some confusion. PI Analysis Service is the one executing Asset Analytics while the service executing PI ACE modules is PI ACE Scheduler. Can you please clarify?

                     

                    EvaluationsToQueueBeforeSkipping and MaxCacheEventtsPerAttribute again indicate the issues you are experiencing are with PI Analysis Service. I recommend against changing before mentioned settings unless advised by Technical Support. Technical Support Engineers are discussing issues with the corresponding product specialist escalation and if the product specialists recognize a potential for improving the situation by changing a setting, they will suggest it.

                      • Re: PI Analysis Service_Maximum Latency
                        Paurav Joshi

                        Hi Gregor,

                         

                        Sorry for creating confusions here.

                        PI Analysis Service is the one executing Asset Analytics while the service executing PI ACE modules is PI ACE Scheduler.

                        I was just mentioning that other PI applications are working fine on the machine.

                         

                        My query was if IsLoadSheddingEnabled has been set to False, then does value for the EvaluationsToQueueBeforeSkipping and MaxCacheEventsPerAttribute has any significance ?

                         

                        Thanks,

                        Paurav Joshi

                          • Re: PI Analysis Service_Maximum Latency
                            gregor

                            Hi Paurav,

                             

                            Does that mean that PI ACE Scheduler runs on the same box as PI Analysis Service? This is a scenario that I would try to avoid because both services are doing quite similar things and I expect them to repeat for resources at "busy" times.

                             

                            It's really not that I am holding back information but I can't answer your question with certainty. MaxCacheEventsPerAttribute is an AF SDK setting and to my understanding would not be affected by IsLoadSheddingEnabled

                              • Re: PI Analysis Service_Maximum Latency
                                Paurav Joshi

                                Hi Gregor,

                                 

                                Thanks for reply.

                                It's really not that I am holding back information but I can't answer your question with certainty

                                I know this is not the case, you have provided so many information on threads of mine as well as others.

                                 

                                Does that mean that PI ACE Scheduler runs on the same box as PI Analysis Service? This is a scenario that I would try to avoid because both services are doing quite similar things and I expect them to repeat for resources at "busy" times.

                                It is the case, and problem suddenly pops up. It's not like problem is there from beginning.

                                 

                                Let me create techsupport case, in which I can share more information if needed.

                                 

                                Thanks,

                                Paurav Joshi

                              • Re: PI Analysis Service_Maximum Latency
                                nagarwal

                                My query was if IsLoadSheddingEnabled has been set to False, then does value for the EvaluationsToQueueBeforeSkipping and MaxCacheEventsPerAttribute has any significance ?

                                When load shedding has been disabled, the parameter EvaluationsToQueueBeforeSkipping has no significance.

                                 

                                The parameter MaxCacheEventsPerAttribute is not really related to load shedding, as it just determines the maximum number of events that would be cached per input attribute. PI Analysis Service uses this parameter (in addition to other cache related parameters - e.g. CacheTimeSpanInMinutes) to determine what events it should hold on to, and which ones are OK to be trimmed. If the calculations are falling behind, the service tries to keep more data in the cache (in order to make sure that when older events are evaluated, it still has input data available in the cache). However, the maximum number of events would be limited by  MaxCacheEventsPerAttribute.

                                Hope this helps.

                                -Nitin

                                1 of 1 people found this helpful
                        • Re: PI Analysis Service_Maximum Latency
                          Jagan

                          I just overcame high memory consumption problem with  “PIAnalysisProcessor” after few weeks of working so here is my thoughts. Out system had similar problem. After rebooting PIAnalysisProcessor the memory consumption keeps increasing and within few days the process starts to take 90% or more of available memory making the rest of the services slow. The way I fixed is, through Management tab had a look at all the analysis that were in Error/Warning state and fixed all of those. Some of the analysis(500+) were set to Event triggered and the underlying inputs were changing quite often like once in a second or once in couple of seconds. The problem of high memory consumption and high latency got fixed when I changed these analysis to periodic with 10 minute interval. And then also looking at C:\ProgramData\OSIsoft\PIAnalysisNotifications\Logs\pianalysisprocessor-log.txt log file might provide some useful information.