7 Replies Latest reply on Apr 19, 2017 5:32 PM by JeffWaufle

    Manual Data Validation




      We have a number of calculations which produce daily kpi's for a plant.

      Each morning the KPI's are reviewed. The users would like to be able to 'tick' off a value as 'validated' or mark if it is rejected.

      We have a custom user interface to display and interact with this data.


      How would one mark a value in PI as validated? E.g. is there a functionality similar to marking as questionable? Is this a built in functionality that already exists?

      I have seen the annotations functionality i was just wondering what other options might exist.



      Any pointers appreciated,





        • Re: Manual Data Validation
          Eugene Lee

          Hi Calman,


          In my opinion, there are other attributes of the PI Point (that are not used by the interface) that you can use to mark it as validated. Some examples include the description field or fields such as conversion factor or filter code etc.

          • Re: Manual Data Validation
            John Messinger

            The possible downside to this approach is that it doesn't apply to individual timestamped values - using a point attribute (such as the userint and userreal attributes) is problematic in that it only highlights the tag in some way to a custom application or usage, not the individual data points in the archive.


            How you end up managing this depends also in part on how you intend to consume the validated data down the track. A common approach in the past has been to configure parallel tags - one to capture the 'raw' data, and one to hold a validated version of the data. The validation rules might be evaluated by some form of analysis, with the resulting output written to the 'validated' tag. This obviously allows for preservation of the original 'raw' data, and provides useable validated data for downstream consumption.


            In the case of Calman's KPI's, where the validation of these already calculated results appears to be a manual process, I would probably approach this through the use of either questionable bits (setting it for results that don't pass the validation), or possibly with annotations. This second option would probably be a bit more cumbersome, especially with visualisation and reporting. Given these potential choices, I would opt for setting the questionable bit for the values that don't pass the validation review. Your custom interface that allows for interaction with the data could manage the setting of these quality bits. A variation on this theme might be to run the parallel tags as described, and allow your users to modify the values in the 'validated' tag, so that corrected values appear with the substituted bit set. This way, you get a data set of 'validated' values, and the presence of the substituted bit highlights corrected data (kind of an inverse approach from the questionable bit option). Again. depends on how the data will ultimately be used.


            My 2 cents



            2 of 2 people found this helpful
            • Re: Manual Data Validation


              How many of these KPI's do you have?  Can I assume these are PI Points?


              Steve Kwan

              • Re: Manual Data Validation
                Rhys Kirk

                I would go along the lines of what John mentioned.

                Having additional tags to hold the "quality" of the values. The quality tag would be paired to the manually entered value by time and naming convention. It is a common approach.

                You would need to be mindful of the additional quality tag when using the data in spreadsheets, calculations, etc. Additional qualities is not something that is naturally tied together in the PI System (yet, right Stephen Kwan) but you mentioned having a custom application to interact with the data so you can embed that logic within your app.


                e.g. ManualDataTag1.Value  (tied to) ManualDataTag1.Value.Quality

                2 of 2 people found this helpful
                • Re: Manual Data Validation
                  Roger Palmen

                  Using John's idea using questionable flag and/or annotations is one approach.

                  Another is creating a second set of PI Points only containing validated values. So every unvalidated KPI is copied over upon validation to the validated KPI.

                  1 of 1 people found this helpful
                  • Re: Manual Data Validation
                    Marcos Vainer Loeff

                    A third approach is to create an event frame (EF) instead of creating a second PI Point. Supposing that the user always validates values ordered by timestamp (starting with the older ones), you can update the EndTime of this EF every time a new value is reviewed. As a result, all values within the StartTime and EndTime of the EF have already been reviewed. Values with timestamp outside this time range haven't been reviewed yet.

                    1 of 1 people found this helpful
                    • Re: Manual Data Validation

                      We only wanted "certified" data to appear in some reports.  We created an app that an "approver" would review and when certified, the data is copied into a separate pointname called xxxxxx.CERT. Only xxxxxx.CERT tags are used in the reports, so if the data doesn't get certified the report will not have data.

                      1 of 1 people found this helpful