7 Replies Latest reply on Feb 27, 2017 2:50 PM by gregor

    Detect Gaps in Data

    fjmocke

      Hi everyone,

       

      I've been struggling for a while now to automatically detect gaps in my data using AF Analytics and fill a tag with a binary signal depending on if a gap exists or not.  See below for what I'm trying to achieve.

      My main problem is in order to fill the binary tag with data I need to be able to write data back in history at a specific timestamp (when the gap started), however this is not possible (as far as I know in AF).  Using HasChanged() is also not working as my data tend to come in batches of up to 10 minutes late.

       

      Any ideas?

       

       

      Capture.JPG

        • Re: Detect Gaps in Data
          gregor

          Hello Franco,

           

          What can be considered a data gap mainly depends on the kind of data you are dealing with. For some signals, it is pretty normal that there are no updates for hours.

          In your case, it looks as if there was an issue with the data collection. Can you tell what source is feeding this PI Point? Also, is buffering in place to buffer data in case the connection between the source and the PI Data Archive becomes interrupted?

           

          Can you describe your criteria identifying a data gap e.g. the period of time between 2 events? How often do you like to perform this analysis?

          Do you possibly like to share what you've found so far while trying to identify gaps using AF Analytics?

          Can you also be more specific about ".. fill a tag with a binary signal .."?

          Do you like to write a status like "No Data"? Assume you would insert "No Data" status at the point in time where you consider the gap to start, this status would be considered true until the end of the gap. Repeating the "No Data" status would not add any additional information.

           

          There's a period of time with almost no change in value but the markers indicate that all the data has become archived. Is compression set up properly for this PI Point?

          2 of 2 people found this helpful
            • Re: Detect Gaps in Data
              fjmocke

              Hi Gregor,

               

              Thanks for the reply.  The data that I'm dealing with is supposed to come in every 2 seconds, however due to network issues in the field as well as equipment turning on/off it is "normal" for data to be lost at this point.  Note there is currently no buffering.  The application for this PI system is open pit mining.  There is nothing that we can do at this point to fix the data sources, thus for now I need to deal with the gaps.

               

              The data is pulled from a SQL database using the RDBMS interface which runs every 30 seconds.  The data arrives in the SQL database a anything from 1 second to 15 minutes late.  This is an issue by the vendor supplying the hardware and will not be fixed soon.  Thus I need to assume that I'll be dealing with late data indefinitely.

               

              It is important to know if gaps exists as different equipment cycles are derived from the data with statistics attributed to the cycles.  The statistics are inaccurate if gaps are present, and thus I need to flag them as such.  This "No Data" tag needs to be used to identify if a gap exists inside a cycle or not.

               

              A gap is defined where the time between data points are more than 15 seconds.  For a period as such the signal needs to have a value of one.  If outside such a period then the signal should be zero.    Thus a check should be run at least every 15 seconds.

               

              I have implemented an AF Analysis using the HasChanged() and PrevVal() functions, running every 5 seconds.  This would have worked perfectly however many times the data comes in minutes late, and thus the value is written to the wrong timestamp (which is now).  AF cannot write a value to the past (specific timestamp).

                • Re: Detect Gaps in Data
                  Rick Davin

                  Hi Franco,

                   

                  Your last sentence hits on a known limitation of Analyses: while an Analysis can write to Now or to a fixed offset to Now, it cannot write to a specific timestamp (or one with a varying offset from Now).

                   

                  Are you capable of writing AF SDK or ACE code?

                    • Re: Detect Gaps in Data
                      fjmocke

                      Hi Rick,

                       

                      Yes I can.  My cycle analysis is done in AF SDK.  I have done the data gap check in AF SDK however it is not working that well because I'm pulling interpolated sets of AFValues.  As you know, the interpolated data does not have a detectable gap.  I can pull the raw data as well however that will require basically two sets of the same data, which I want to avoid.  I'm also not sure how to deal with the compression issue where it looks like a gap exists however the data was compressed.  It is important for me to retrieve interpolated data as I have different PIPoints where the resulting data need to be synchronized.  Raw data is not synchronized and very confusing to work with. 

                    • Re: Detect Gaps in Data
                      gregor

                      Hi Franco,

                       

                      In case a piece of equipment is turned off, I would expect an interface not being able to read from the specified source and depending on the interface writing "No Data" or "I/O Timeout" status. This said, it could be very useful for you to set up PI Buffer Subsystem except turning off the equipment includes turning off the interface (and buffering).

                       

                      Using Asset Analytics might be an option if you could assume the tag being stale at a certain value for let's say 15 seconds would allow you to assume a gap starting 15 seconds before the evaluation time. With your data arriving late, I doubt this will be an option and you may have to create your own calculation engine looking at the data to identify a gap and to mark it with a "No Data" status.

                       

                      The challenge here will be at what time you will be able to look into the past and to be certain to identify the gap as such. Because you mention an impact to your statistics, it is also important to consider when those statistics are computed because you may have to re-compute them in case a gap becomes identified late, let's say after one hour.

                       

                      While we will be able to provide you with sample code on how to read and process historical values and how to insert a "No Data" event, the challenge will be on you to define the times to run certain analysis which refers to identifying gaps and computing statistics. The easiest approach - if possible - will be to get a status marking the beginning of a data gap by utilizing the data source in combination with PI Buffer Subsystem.

                  • Re: Detect Gaps in Data
                    Rick Davin

                    Hi Franco,

                     

                    I'd like to explain more about what my colleague @Gregor_Beck refers to in his very first sentence as the topic of data gaps or stale tags depends upon the individual tags.  I've been in environments where a tag's data older than 15 minutes is considered stale but another tag's is expected to go half a day without a changing.  I wouldn't expect a setpoint or a lab tag to change frequently, but that usually requires me having a priori knowledge as the engineer.  How would your code/analysis know that a data gap truly appears?

                     

                    Scale is an important context here too.  If you zoom in a trend many times, there will be the visual appearance of gaps.  Conversely if you zoom out many levels, gaps may visually disappear.

                     

                    From your trend, given there is a flat section on the left and the right side shows a relatively flat section of mostly noisy wobbles, I would think this tag does something for some brief period of time and then is quiet for another brief period of time.  I would wonder if someone changed the compression sometime between the left and right to begin recording the tiniest bits of signal noise. 

                     

                    Left is flat.  Very flat:

                    Detect Gaps in Data - Flat.png

                     

                    Right is relatively flat but noisy:

                    Detect Gaps in Data  - Noisy but flat.png

                     

                    If there truly was a data gap for the left, Gregor's mention of "No Data" is important.  I would then clearly see there is an outage and not be confused if there is noise or compression settings at play.

                     

                    Regards,

                    Rick

                    1 of 1 people found this helpful
                      • Re: Detect Gaps in Data
                        fjmocke

                        Hi Rick,

                         

                        Thanks for the reply.  I've replied to the reply made by Gregor.  Basically the data is supposed to have a frequency of 2 seconds, but due to network and equipment related issues in the field, data can get lost in big chunks.  Currently this is seen as "normal" operation, and will not change soon.  But I need to quantify those gaps so that my statistics are not contaminated.