12 Replies Latest reply on Oct 18, 2017 6:53 PM by ekuwana

    Duplicate timestamps after call to AFValues.GetValueArray?

    PhilKnight

      I am watching the following code find duplicate timestamps at random times:

       

      object[] values;
      AFValueStatus[] flags;
      DateTime [] timestamps;
      afvals.GetValueArrays(out values, out timestamps, out flags);
      for (int i = 0; i < timestamps.Length - 1; i++)
      {
           if (timestamps[i] == timestamps[i+1])
           { 
                _logger.Error($"Duplicate for tag {afvals.PIPoint.Name} at index {i}, timestamp {timestamps[i]}");
           }
      }
      

       

       

      I inspect the array and sure enough there are duplicates.  I run the code again, same tag, same timerange, but no error/duplicate.  Of course initial thought was race condition with recordedvalues / pipagingconfiguration, but that doesn't make much sense...  Surely that would be felt by many others..

       

      Second thought is time range is last 24 hours, AND the datastream to this PI server is not good and pushes out tons of out of order and duplicate events (not in my ability to fix right now).  Will PI pull these into recordedvalues api call? I don't have any problems with the call (hundreds repeat calls) when I back off the time range to 48 to 24 hours.

       

      Thanks in advance for any words of advice! Phil

        • Re: Duplicate timestamps after call to AFValues.GetValueArray?
          nicka

          What happens if you try if (timestamps[i] == timestamps[i+1] && (flags[i] & AFValueStatus.Substituted))?

           

          What are the values of flags[i] and flags[i+1] for your duplicates?

          1 of 1 people found this helpful
          • Re: Duplicate timestamps after call to AFValues.GetValueArray?
            Rick Davin

            Hi Philip,

             

            What version of AF SDK are you using?

             

            The AFValue.GetValueArrays returns what was provided to the source AFValues collection, perhaps its source was a GetValues or RecordedValues call.  But definitely if your data stream has duplicates in its source, and those duplicates end up in the PI Data Archive, then yes the GetValueArrays will contain duplicates as well.

             

            In addition to inspecting the flags[] when a duplicate event is found, you can also verify against your original afvals object.  When timestamp[i] equals timestamp[i + 1], you can double-check that afvals[i].Timestamp equals afvals[i + 1].Timestamp as well.  If duplicates exist in afvals, then we can eliminate GetValueArrays from being a problem.  You then move up the chain to see why afvals has duplicates.  What data method are you calling to fill afvals?

             

            You say it's random, which makes it hard to troubleshoot.  Oftentimes someone may post a problem that is very repeatable on their end that we can't repeat on our end.  It's more challenging when the original post can repeat on their end.

              • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                PhilKnight

                Good questions, I had inspected flags in VS, but not included them here.  Adding little more to the logs:

                 

                Duplicate for tag ACCA_EMS.LINE.88B.MW at index 2153, t1 10/18/2017 1:10:25 PM, f1 Good, v1 14.3, t2 10/18/2017 1:10:25 PM, f2 Good, v2 14.3

                 

                t1 is timestamp at timestamps[I], f1 is flags[I] (status), v1 is values[I].  t2, etc., are identical but for the i + 1 values.

                 

                Source is indeed a RecordedValues call:

                ppl.RecordedValues(range, AFBoundaryType.Inside, null, true, new PIPagingConfiguration(PIPageType.TagCount, 100));
                

                 

                AF SDK Version is 2.8.6.7801.

                 

                I didn't originally look the afvalue -- I always assume perhaps incorrectly that afvalue is a façade ontop of the arrays. I checked these and they too are identical / duplicate.

                 

                It is an interesting issue -- the values and flags are identical.  I had thought that replaced values would be marked as substituted, but all events in array are good.  I looked at SMT to inspect the archived values, no substituted, only a single event.  Checked logs and no issue showing for this tag...

                Intermittent events == fun, right? Thanks! Phil

                  • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                    nicka

                    They appear to be identical but if I remember rightly, PI supports subsecond times, are you sure these aren't 10/18/2017 1:10:25.250 PM and 10/18/2017 1:10:25.750 PM for example?

                     

                    From the AF SDK documentation:

                    > Actual storage of an AFTime value may cause a loss of accuracy depending on the target storage. The PI AF Server will store timestamps to an accuracy of 3.33 milliseconds. A PI 3.x Server stores timestamps to an accuracy of 15 microseconds.

                     

                    When these values are then fetched by the AF SDK they may then be truncated so that they are the same client side, but not server side.
                      • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                        Rick Davin

                        Hi Philip,

                         

                        Actually, the AFValues collection as retrieved from RecordedValues is the true source, and populating GetValueArrays is fed directly from it.  I would suggest you can omit GetValueArrays and iterate over afvals directly.

                         

                        Nicholas Alderton  the subseconds are not at play here.  Usually that occurs the opposite direction, that is when someone visually sees 2 values in SMT that have the same timestamp but the code says they aren't the same (because the subseconds are hidden from their view).  But AF SDK does not truncate the times with PIPoint.RecordedValues.  Actually the Timestamp of an AFValue can have greater precision than 15 microseconds!  It's only when attempting to write back to the system of record that the time will be truncated - by that system of record.  For the documentation, the AF Server storing timestamps would only refer to actual values stored on the AF Server, and that would be for static AFAttributes or event frames StartTime and EndTime.

                         

                        See A Detailed Exploration of AFTime Precision  for more.

                        1 of 1 people found this helpful
                          • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                            nicka

                            Ah that makes sense, it was a bit of a stretch that one. Great link!

                            • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                              PhilKnight

                              Thanks for the suggestions and information.  I also looked at timestamp ms and can confirm there is no difference.

                               

                              Rick, information on value arrays is interesting.  I had assumed the opposite because the documentation says:

                                   This method can perform higher than extracting individual AFValues or PIValues because less objects need to be created.

                              That of course could mean a lot of things   Maybe more clarification in the documentation might help.

                               

                              Performance aside, for my use it's still preferable to use the array structures as the timeseries work I'm doing can directly load data.  Less code, and a little more performant.  Regardless, after looking at the AFValues, and still finding the phantom duplicate, I'll have to work around that.  Right now I can change duplicates in the array to DateTime.MinValue.AddSeconds(dupcntr++) to avoid uniqueness constraints, then filter out anything in year 1.  It's more work, but works.

                               

                              I don't know if I should open up a tech support case at this point?  Others may experience this in similar conditions but may not notice or care...

                               

                              Thanks for the thoughts, Phil

                                • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                                  Rick Davin

                                  If the problem was with code, you can keep this thread alive.  But the problem doesn't seem to be with your code.  The larger issue seems to be with your environment.  What is it about the interfaces or flow of data that is causing duplicates?  And that may be more of a configuration or installation problem where TechSupport can help.

                                   

                                  I too find the documentation remark confusing.  First off, "can perform higher" could really be interpreted as "might possibly perform higher, or might not".  And  GetValueArrays must be called from an AFValues collection, which has already extracted all of the data client-side so I am a bit lost on how that is more efficient.  I had always considered GetValueArrays a feature for legacy PI SDK code (see PI SDK GetValueArrays or Load), or as a mechanism to pass data to another layer that doesn't support AF SDK objects directly.

                                   

                                  The AFValueStatus remarks says something the same:  When obtaining large numbers of values, it is more efficient to access the values and status using the AFValues.GetValueArrays method which returns the status information as flags.

                        • Re: Duplicate timestamps after call to AFValues.GetValueArray?
                          Rick Davin

                          Last year I wrote a little diagnostic/learning tool named "Timestamp Inspector" specifically for an extremely easy way to detect:

                           

                          • Duplicate timestamps, or
                          • Subsecond timestamps

                           

                          The executable and source code are on GitHub.

                           

                          GitHub - Rick-at-OSIsoft/timestamp-inspector: A WinForm app to inspect timestamps from PI data

                          2 of 2 people found this helpful