11 Replies Latest reply on Jul 11, 2017 8:34 AM by hmokdad

    PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive

    hmokdad

      We are building an application that must capture every snapshot value for a given tag (or all tags) and mark them as questionable. To mark the values as questionable, the application captures the value (of type AF Value), and stores them in an AFValue List. Another thread will push the list to the PI Server every specified interval of time, let us call it x. Below is the code we are using to update the AFValues list to the PI Server.

      1. pisServer.UpdateValues(afValues, OSIsoft.AF.Data.AFUpdateOption.ReplaceOnly, bufferOption);

      We are using “ReplaceOnly” mode because we don’t want to reintroduce values that didn’t make it to the archive due to compression. Our issue is here; when x (the refresh interval) is 2 seconds, then the application works fine, that is no values are marked as questionable other than those in the archive. So if we sent 10 identical values, then due to compression only one or two values at most should be archived, and with x = 2, then we will have one or two questionable values.

      Now if x = 6, then the “ReplaceOnly” will not be valid anymore, so instead of 2 questionable values, we are having 5, 6 or 7 questionable values. Therefore, our constraint of not reintroducing compressed events to the archive is not true anymore. Note that we are sending one value to the PI Server every three seconds, and flushing the AFValuesList every 6 seconds.

      I have created and attached a windows forms application that sends data to the PI Server every specified amount of time (3 seconds by default), and has a Data Pipe Observer capturing snapshot data, marking them as questionable and flushing them to the PI Server every x seconds.

      To reproduce the issue, start the form; change the name of the PI Server to your PI server and the tag name to a tag of your choice to test, preferably the tag should only be fed from this application.  Open PI SMT, Archive Editor and choose the given tag. Click the button “Send New Values”, keeping other controls to their default values, that is send 10 identical values, one each 3 seconds, and monitor PI SMT. With “AF Values Refresh Time” set to 2, then you should see in PI SMT a maximum of 1 or two questionable values, as the rest of the values will be removed due to compression (Of course the PI Tag should have compression enabled). Now set the “AF Values Refresh Time”to 6, and send another 10 identical values, this time you will notice that 6 or 7 questionable values made it to the Archive, which contradicts the operation of “ReplaceOnly” mode as we should only replace values that exist ( from the thread that is reintroducing values as questionable). 

       

      Any explanation for this weird behavior or a work around is highly appreciated.

      Thanks

        • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
          Rick Davin

          Hi Hassan,

           

          I'll be taking a closer look at your VS project.  One thing that has caught my attention with my first pass is:

           

          Class: DataPipesHelper

          Method: UpdateAFListBuffer

           

                  public void UpdateAFListBuffer(PIServer piServer, BlockingCollection<AFValue> queue)
                  {
                      Stopwatch SW = new Stopwatch();
                      SW.Start();
                      List<AFValue> afValueList = new List<AFValue>();
                      try
                      {
                          foreach (AFValue afValue in queue.GetConsumingEnumerable())
                          {
                              afValueList.Add(afValue);
                              if (SW.Elapsed.TotalSeconds >= AFVAluesListRefreshTime) //(afValueList.Count >= 10 || SW.Elapsed.TotalSeconds >= 5)
                              {
                                  OSIsoft.AF.AFErrors<AFValue> errors = new AFErrors<AFValue>();
                                  UpdateAFListasQuestionable(piServer, afValueList, AFBufferOption.BufferIfPossible, out errors);
                                  afValueList.Clear();
                                  SW.Restart();
                              }
                          }
                      }
                      catch (Exception)
                      {
                      }
                  }
          

           

           

          Your afValueList variable serves as a small buffer that will flush updates to the data archive based on a timing mechanism.  The problem I see is that it is possible for afValueList to contain items but the foreach loop ends before tripping the timer.  That would leave AFValue items unflushed and not sent to the data archive.  This behavior would be more pronounced the larger AFVAluesListRefreshTime.  After the foreach loop ends, you would want to check that afValueList.Count > 0 and if so then once again call UpdateAFListasQuestionable to flush any lingering items.

          • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
            ekuwana

            Hi Hassan,

            Are you writing through buffering? Did you try with AFBufferOption.DoNotBuffer and get the same result? There is compression issue related to PIBufSS with Replace mode. Please try with DoNotBuffer to confirm if possible.

            PLI25937OSI8

            https://pisquare.osisoft.com/message/49012?et=watches.email.outcome#49012

            1 of 1 people found this helpful
            • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
              Rick Davin

              Hi Hassan,

               

              Hopefully you realize that this concept of intercepting all snapshot values and setting their Questionable flag is asking your data archive to perform twice as much work.  Each new event will cause 2 or more writes to the snapshot subsystem.  I question the soundness of why you want to do that.  Both in regards to not placing an unneeded burden on the data archive, but really because either you trust your interfaces or you don't.  If you don't trust your interface, you should consider either fixing it or replacing it.   If I trust that an interface is setup correctly, it may send over a value that I don't like (such as under range) yet I have to accept that the value accurately reflects what the interface was measuring.  In that case I would not flag the value as Questionable. 

               

              Can you tell us about the interface feeding in updates?  How many interfaces are you using?  How many points are coming through each interface?  What is it about the interfaces that warrant such distrust?

                • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
                  hmokdad

                  You are right, in the normal situation we will not be marking all values as questionable, only values that fulfill certain conditions. But to prove the capabilities of our application, and as per the requirements, we are forced to test on this large amount of data. Do you think this is too much on the Snapshot subsystem?

                    • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
                      Rick Davin

                      You're asking the Snapshot subsystem to (1) perform artificially needless work that will (2) yield minimum business value with (3) an application that has certain points of failure requiring (4) ever diligent man-effort to keep things running smoothly.

                       

                      The needless work:

                       

                      1. A PIPoint's snapshot comes in with Questionable=False
                      2. The snapshot is then sent out to any data pipe subscribers
                      3. The data pipe gets its incoming event and changes the Questionable flag
                      4. The data pipe then sends an update right back to the PIPoint snapshot
                      5. The PIPoint's snapshot arrives with Questionable=True
                      6. The snapshot is once again sent out to data pipe subscribers
                      7. The data pipe reads the event (you know, the one that it sent) only to discard it.

                       

                      What are your later plans?  Will you have a routine to allow an engineer to remove the Questionable flag a few hours/days later?

                       

                      This thing you want to do would only work as long as the application is up-and-running 100% of the time.  You leave absolutely no room for any outage or disruption.  If your app hangs or the machine hangs or there is a network disruption or you must patch the host running your app, your snapshot data pipe will go away briefly.  During that downtime, values may be coming into the snapshot as well as some of those values being moved to the archives without you modifying their Questionable flag. You would create a situation where any value in the recorded values could be suspect because it is not flagged as Questionable!  Is it not Questionable because an Engineer reviewed it, or is it not Questionable because it got past your sentry app?

                       

                      This leads to a situation where you must invest more time and effort into trying to come up with code solutions, or else you have a person frequently reviewing incoming data to make sure the data is perfect because your app cannot be perfect. 

                        • Re: PIServer.UpdateValues using AFUpdateOption.ReplaceOnly is replacing values even if they don’t exist in the archive
                          hmokdad

                          Hi Rick,

                           

                          Actually why we are building the application is much more complex that this, and you are right, it might seem useless to mark values as questionable but it is not. We have several departments that each treat data differently. One department is interested in all data, and don’t want any value to be dropped,  another department is only interested in Good Data (Good I mean in terms of their own logic and not Good as Good in PI). The other department has to calculate Hourly averages and generate reports based only on Data that passes through a certain logic, which are not marked as questionable by our application. This enables them to exclude questionable data in Performance equations or OLEDB queries.

                           

                          Best Regards,

                          Hassan Mokdad