12 Replies Latest reply on Jul 24, 2018 5:33 PM by Guilherme Ferreira

    False start trigger for event frame

    Guilherme Ferreira

      Hi folks!

       

      I have created some simple notifications for monitoring my PI Interfaces.

      The related event frame triggers look like below.

       

      (EDIT: this image was missing)

       

      "Notificação" is just a flag for easily turning notifications on\off.

      If interface ping is not a badval, check if last heartbeat is older than 10 minutes ago.

      This should tell me that the interface node is online but the service has stopped for any reason. Quite simple!

       

      For some of my interfaces (notice the above configuration is in template level), I am getting false event frame starts.

      In other words, the interface is operating normally, heartbeat is updating and a event frame is generated.

       

      The notification message also shows that the event should not have been created:

       

       

      Although the heartbeat values stored in the server have current timestamps, I suspected the interface clock could be desynchronized with the server's.

      Indeed it is! The interface clock is about 5 minutes behind.

       

      It makes me think that at trigger time, analysis service considers the interface node timestamp but when I click "evaluate" button, it considers server timestamp (as it is stored).

       

      Does it make sense?

       

      It is the only explanation I can think right now.

       

      Any ideas?

       

      Is there a debug level log for analysis service?

       

      Regards

       

      Mensagem editada por: Guilherme Ferreira

        • Re: False start trigger for event frame
          fleskowitz

          Hi Guilherme,

           

          There is a KB01335 - Asset analytics "Evaluate" button shows different results from output that explains why output results in PSE differ from what is actually executed.

          It seems that your solution would be under Scenario 3.

           

           

          In regards to your question about a debug log for analysis services you can follow Configure logging for PI Analysis Service if you are running on PI Server 2018, if not you can change what version you have in the live library in the top left corner.

           

          Best,

           

          Faith

            • Re: False start trigger for event frame
              Guilherme Ferreira

              Hi Faith,

               

              thank you for your response.

              The scenario 3 from the KB you've sent me may help understand the problem.

              As I said before, the interface is indeed 5 minutes behind, but AF Server and PI Data Archive are synchronized!

               

              So I am still not sure if I can explain the odd behavior.

               

              Analysis service was configured with default CacheTimeSpanInMinutes=5.

              When is this cache update?

              How frequently?

               

              When I ask PrevEvent('HeartBeat','*'), Analysis service will check its cache for the previous event. If it is not in cache, it will call Data Archive. Is that right?

               

              In any case, cache or DA, the timestamp for one specific value should be the same, right?

               

              So, what I need to understand is how Snapshot Subsytem treats a value coming from an interface with different clock.

              Does it store the UTC from the interface plus an offset between machines? At some point it converts the UTC received from the interface, adding or subtracting the difference between clocks?

               

              There is a lot of questions in my mind right now!

               

              For the Analysis Service log, what level should I use? Trace? Debug?

              And how could I search for calculation real values and timestamps?

               

              Best regards

                • Re: False start trigger for event frame
                  Dan Fishman

                  The cache is updated very often (probably at least once a second) and receives the values that were sent to the snapshot table (new arriving data).  You are correct, the analysis service will check the cache for the previous event and if it is not in the cache, it will make a call to the data archive.  The only difference will be the cache holds uncompressed data and the data archive holds compressed data.  The cache is higher fidelity for the cache span of 5 minutes. 

                   

                  The snapshot holds the same value in UTC that will be stored in the data archive.  The timestamp stored from the interface will be the timestamp stored in the data archive.  Note, the interface can check the clock of the PI Server and sync the timestamps depending on configuration.

                   

                  I am a little suspicious of PrevEvent('tag', '*') and would suggest PrevEvent('tag', '* + 1s') to make sure you are getting the most recent value and not the previous timestamp.  I am wondering at your evaluation time, there is an event, so the equation is evaluating the last event. How frequently is the heart beating updating and what is the schedule for the analysis? 

                    • Re: False start trigger for event frame
                      fleskowitz

                      To add onto what Dan said, assuming you have an OPC Interface, you can use KB00305 to understand what the different timestamp parameters mean. You can also look directly at the interfaces online documentation depending on your version at the Live Library.

                      • Re: False start trigger for event frame
                        Guilherme Ferreira

                        Hi Dan!

                         

                        thank you for your answers.

                         

                        Explaining the start trigger a little more, it runs every five minutes and should check if the last heartbeat from the interface is older than 10 minutes. So, I understand your suggestion of using PrevEvent('tag', '* + 1s'), but it shouldn't make any difference, when the heartbeat is updating every second and stored at least every 15 seconds.

                         

                        I could go ahead and synchronize interface node clock and the problem would possibly go away, but I am intrigued to understand what is happening.

                         

                        I tried setting analysis log to "Trace", but I couldn't find the actual values and timestamps used in calculations.

                         

                        Regarding TS parameter for OPCInt, it should have no effect over the interfaces heartbeat!

                         

                        Best regards

                         

                         

                          • Re: False start trigger for event frame
                            Guilherme Ferreira

                            Hi folks!

                             

                            I still can't make sense of what is happening with these event frames and notifications.

                             

                            Interface heartbeat:

                             

                            Notification format:

                             

                            Notification received just now:

                             

                            Timestamp at send time is 11/08/2017!!!

                            Where has this come from? This is probably the value used by the analysis service at trigger evaluation, but why? And how can I confirm it?

                             

                            Regards

                              • Re: False start trigger for event frame
                                Guilherme Ferreira

                                The events keep triggering, the heartbeat timestamp at start time updates accordingly, but timestamp at send time is always the same.

                                It looks like analysis service has a stuck value for that variable, so it keeps generating a false start.

                                 

                                And the notification service, when asked for the value at event frame's start time, call PI DA and retrieve correct value.

                                But when asked for the value at send time, uses this same stuck trigger value.

                                 

                                 

                                 

                                Btw, I have implemented Dan's suggestion:

                                 

                                Regards

                                  • Re: False start trigger for event frame
                                    Dan Fishman

                                    That is very odd.  Do you have a PI collective by any chance? 

                                    1 of 1 people found this helpful
                                      • Re: False start trigger for event frame
                                        Guilherme Ferreira

                                        Hi Dan!

                                         

                                        You've just nailed it!

                                         

                                        I was aware that buffer is not correctly configured for this interface (I will do it as soon as I have access to it), but I didn't realize this was the reason for the odd behavior.

                                         

                                        Just for the sake of understanding: shouldn't the analysis service respect the collective member priority configured in SDK?

                                         

                                         

                                         

                                        Regards

                                        1 of 1 people found this helpful
                                          • Re: False start trigger for event frame
                                            Dan Fishman

                                            You are welcome!  I've also ended up in trouble a few times with similar situations with collectives.  Often, I find that a lot of the data is not replicated!

                                             

                                            The screenshots you show are for the PI SDK, and I believe the configuration for AF SDK is within PI System Explorer.

                                             

                                            Each application has its own SDK reference and perhaps analysis service switched over and never switched back.

                                             

                                            Also, new to PI Data Archive 2018 per the release notes:

                                            The PI Server 2018 introduces client-side connection balancing and failback.

                                             Upon initial connection, rather than all clients connecting to the primary member of a PI Data

                                            Archive collective client connections will now be automatically distributed to all available

                                            collective members.

                                             After an HA failover, failback switches the client connection back to the original member.

                                             This feature is only available for AF SDK based applications. To enable this feature on a

                                            client computer, first upgrade to PI AF Client 2018. Then use PI System Explorer to set the

                                            priority in the KST by setting the connection priority for all collective members to 1

                                            1 of 1 people found this helpful