7 Replies Latest reply on Sep 23, 2014 12:31 PM by Marcos Vainer Loeff

    Events Stop Coming?

    Steve Boyko

      I wrote an application a while ago that subscribes to events from a number of tags, and then does some processing whenever the events arrived. A few days ago it appears it just stopped processing events. The program (service) was still running, PI Network Manager showed it still connected and active, PI Update Manager showed it still subscribed to its tags with no events waiting, yet it appears my program was not processing the events.

       

      Here's the code I used to subscribe:

       

       

       
                  datapipe = pipoint.Data.EventPipe;
                  eventpipe = (_DEventPipeEvents_Event)pipoint.Data.EventPipe;
                  eventpipe.OnNewValue += new _DEventPipeEvents_OnNewValueEventHandler(PIEventsToSQL.PIDataEventHandler);
                  issubscribedtoevents = true;
      

       And here's the body of PIDataEventHandler, the main routine that gets fired when events come in.

       

       

       
              public static void PIDataEventHandler()
              {
                  try
                  {
                      for (int i = 0; i < lstPoints.Count; i++)
                      {
                          if (lstPoints
      .EventCount() > 0)
                          {
                              Array events = lstPoints
      .GetEvents();
                              SortedDictionary<DateTime, PIValue> SortedEvents = new SortedDictionary<DateTime, PIValue>();
                              for (int j = 0; j <= events.GetUpperBound(0); j++)
                              {
                                  iCtrPIEventsReceived++;
                                  PIEventObject peo = (PIEventObject)events.GetValue(j);
                                  PointValue pv = (PointValue)peo.EventData;
                                  SortedEvents.Add(pv.PIValue.TimeStamp.LocalDate, pv.PIValue);
                              }
      
                              foreach (KeyValuePair<DateTime, PIValue> p in SortedEvents)
                              {
                                  TryEnqueuingPIValue(p.Value, i);
                              }
                          }
                      }
                      return;
                  }
                  catch (Exception e)
                  {
                      ReportMessage("Error " + e.Message + " handling PI data event(s): " + sDoing, ReportingLevels.Warning);
                  }
              }
      

      I know the counter iCtrPIEventsReceived was not being incremented as it is periodically reported to the event log.

       

      Any ideas on what might have gone wrong? I appreciate the help!

        • Re: Events Stop Coming?

          Hello Steve,

           

          It looks like our forum platform has corrupted the body of PIDataEventHandler. Something has been replaced with a link to a GIF showing a bulb. Do you recall if you had selected "C#" when pasting the code in? Can you maybe attach a file with the code inside?

           

          Your application usually runs but suddenly doesn't receive new updates from PI Update Manager even PI Update Manager still lists the signup from your client. There must have happened something causing this issue and my suspicion is that this something has to do with network issues.

           

          PI Update Manager checks regularly with PI Network Manager if connections registered for sign-ups are still active or not. In case PI Network Manager reports a connection is not active anymore, PI Update Manager will clean up the belonging sign-ups. This is e.g. for applications that do not properly remove sign-ups if they are not needed anymore. I could imagine that intermittent network issues could cause such a situation if PI Network Manager on the application node is able to re-establish a lost connection before PI Update Manager reaches out to PI Network Manager on the PI Data Archive node for dropped connections.

           

          You will likely have restarted your application to re-establish sign-ups what makes troubleshooting even more difficult. All we can currently do, is trying to find an explanation or hint in the logs.

           

          Can you please start looking at the PI Message Log of the PI Data Archive at the time +/- 30 minutes around the time your application stopped receiving updates (iCtrPIEventsReceived  not increasing anymore)? I would mainly look for messages from PI Network Manager (pinetmgr) and PI Update Manager (piupdmgr) and if there's no indication, see for anything unusual like messages indicating PI Data Archive was operating under high load, a PI Backup or 3rd-party backup was running and similar.

           

          Another idea just coming into my mind is very unlikely but illustrates how complex causes can be. If there's an Antivirus-Scanner running on the PI Data Archive node, it may have blocked access to piudateconsumers.dat. Do you have an exception defined for the PI Server Home (%piserver%) and subfolders?

           

          Involving Technical Support is reasonable except you are used to look at the PI Message logs. Please let us know if you want to be contacted by a Technical Support Engineer.

            • Re: Events Stop Coming?
              Steve Boyko

              Hi Gregor, thanks for your very helpful reply. I had a look at the Update Manager via the PI server log file and it is overloaded. 3151OSI8 provides some guidance there.

               

              I see in the server logs where the Update Manager dropped the interface's points from the list.

               

              "Consumer <PIEventsToSQL:computername:14216:150|12222> timed out. All signups removed."

               

              It could have been from overloading or it could have been a network issue. Is there any way for the interface to know it has been dropped from the Update Manager's list?

                • Re: Events Stop Coming?
                  Marcos Vainer Loeff

                  Hi Steve,

                   

                  According to KB00818, the three most common causes for lost signups are:

                  1. There is a network interruption causing the connection between the PI Server and the client to be broken.
                  2. slow network can cause the Update Manager to time out and remove the client.
                  3. In an unexpected client shutdown, the client does not remove its signup with the Update Manager which eventually times out the client's signup.
                  That is why I think that it is related to network issues and not overloading.

                  If your this problem happens with some frequency, I recommend you setting up the PI Ping interface in order to know when there is a network issue. Then you will realize if the lost signups are really related to the  network issues.

                   


                    • Re: Events Stop Coming?
                      Steve Boyko

                      Thanks, Marcos, I agree it could have been a network issue. I don't know that we will be able to confirm that.

                       

                      Is there any way for a client to know that it has lost its updates/been removed by piupdmgr and won't be receiving more?

                        • Re: Events Stop Coming?
                          Marcos Vainer Loeff

                          Hi Steve,

                           

                          You can follow the procedure described on the KB artIcle above. If you cannot see the consumer, the signups were lost. Just make sure that your tuning parameter PIUPD_TimeoutKillCons is set to 1 on PI SMT, so that the signup and consumer will both be removed after the timeout period. If you need to change it from 0 to 1, please do not forget to restart PI Update Manager.

                            • Re: Events Stop Coming?
                              Steve Boyko

                              Thanks Marcos, I agree that a sysadmin can determine if the signups have been removed. How will the client know? Is there a programmatic way for a PISDK / AFSDK program to know that it was dropped by the PI update manager?

                                • Re: Events Stop Coming?
                                  Marcos Vainer Loeff

                                  Hi Steve,

                                   

                                  I have received an answer from a developer concerning this topic:

                                   

                                  With PISDK, there is no indication from the eventpipe itself that there is loss of signup with the PIServer. Though PISDK eventpipe will automatically resignup all the tags upon reconnection to the PI server, it does not notify the application of possible problems as it tries to handle all the reconnection logic. Also note that loss of connection does not automatically mean loss of data. If reconnection is made within 10 minutes, server update manager could still have all the events buffered.

                                   

                                  If the application needs to know about loss of connection, it will have to monitor the Server (or Servers) disconnect event (see PISDK reference under Server.Events). If the eventpipe talks to multiple PI servers, need to monitor them all. It will have to hard code the test of disconnection elapsed time of more than 10 minutes to determine whether data loss occur. If the PIServer had restarted, then there will be data loss even though the reconnection occur within 10 minutes. So there is no fool proof method to determine data loss.

                                   

                                  Though PISDK eventpipe does not notify the application about dataloss, it does handle reconnection automatically. So it is not necessary for the application to explicitly recreate the pipe even if they are monitoring the Server object disconnect event. It is more for their data recovery logic to monitor the server disconnect event. For example, Processbook will put in a “NoData” event or some other digital state to give visual indication of possible data loss.

                                   

                                  We address this issue with AFSDK 2.6. At each of the datapipe call, if we detect dataloss, AFSDK will put an AFDataLossException in the PIServerError collection. The AFDataLossException contains information like estimate of dataloss start time and end time. Also a Boolean to indicate whether the dataloss is due to data overflow on the server side (in this case, application has to retrieve/poll data at a higher rate from the client).