10 Replies Latest reply on Feb 7, 2015 10:34 AM by Roger Palmen

    Notifications or Abacus for a 'data quality event queue'?


      Is it possible for AF Asset-Based Analytics to write to an AF Table?


      What we need is a 'queue' of possible data quality events that the operators can see, investigate and respond to. My idea was to have a set of analyses looking at process values and flagging up things like spikes, flatlines, outside limits, etc to a status tag for each input. Then having PI Notifications looking at the status tag and raising a notification for periods that are not 'OK'. We would then have a web-based front-end listing all the unacknowledged notifications, allowing the operator to do whatever is required and then marking the notification as 'acknowledged'.


      However, there may be further information required, such as an operator comment, that the Notifications can't handle (or can they?), so we are also considering having a Custom Delivery Channel to write the notification to an AF table which the front-end can then manipulate.


      But could we do all of this in Abacus, am I introducing more steps than are necessary?

        • Re: Notifications or Abacus for a 'data quality event queue'?

          Hi Alistair,


          You can certainly have comments in PI Notifications. To add a comment, you can simply right-click on any notification event and select "Add Comment". From the description of your problem, you might benefit from setting up an Acknowledgment Page (included in PI Notifications). The Notification Acknowledgment Page provides a link that is embedded in an email message. A subscriber can click directly on this link to acknowledge a notification message or to comment on a notification instance. If this functionality is what you needed, it might save you some time having to write your own web page.


          If you are writing time-series data from Asset-Based Analytics (e.g. process values through a status tag), using a PI tag will give you good performance. Are there reasons why you would like to write to an AF table rather than PI tags? When you mentioned "queue", are you looking to remove the values that operators have responded to? Or do you need historical data?

          • Re: Notifications or Abacus for a 'data quality event queue'?

            Ah. That's a problem.


            Ok then, how about a custom delivery channel that writes the notification details into an AF Table (<GUID>, <start time>, <end time if known>, etc) and then our web front-end can set the acknowledgement and comment in the table. Does that sound feasible?

            • Re: Notifications or Abacus for a 'data quality event queue'?
              Roger Palmen

              I would recommend to use EventFrames, generated by Analytics (abacus). We use that typically to detect, analyse, report and manage quality issues in data.


              Based on settings and historical data for e.g. a flow in a FlowMeter Element, we determine if there is anything out-of-bounds and generate an appropriate EventFrame for that using Abacus. The event is ended when the condition returns to normal. The Event status is managed through a custom user interface that lets the user analyse the data quality issues, and take corrective action or dismiss the events.

              From that you can also determine the date up to which a specific measurement has been validated.

              1 of 1 people found this helpful
                • Re: Notifications or Abacus for a 'data quality event queue'?

                  That's interesting, do you have the ability to mark an event as 'dealt with' and add some kind of comment, or do you simply delete the event frame?

                    • Re: Notifications or Abacus for a 'data quality event queue'?
                      Roger Palmen

                      Yes, you can build anything you like. E.g. a status attribute on the EventFrame using an enumeration to drive a workflow based on status.

                      I'd typically keep the EventFrame, unless there is no point in retaining that data.

                        • Re: Notifications or Abacus for a 'data quality event queue'?

                          This is very interesting. So it looks like Event Frames could be the better home for this stuff. So to clarify...

                          • I can configure Event Frames with similar triggering rules to notifications: deadbands, retry-windows, minimum durations etc?
                          • Event frames can be searched for by element, type, status, trigger time, etc and ordered?
                          • I can add my own 'Status', 'Action', 'Comment' attributes to Event Frames?
                          • I can search, update and maybe even create my own event frames with the AF-SDK?


                          If so, maybe I should provide a web-service for our front-end with something like this:

                          • FindEvents (RootElement, StartTime, EndTime, Status)
                          • Update (EventFrame, NewStatus, ActionPerformed, Comment
                          • NewEvent (Element, Attribute, Start, End, Status)



                            • Re: Notifications or Abacus for a 'data quality event queue'?

                              Well, yes and no to your bullet list of items, depending on whether you want to use asset analytics or write custom code.  Currently in PSE, you can configure a "Start Trigger" and an optional "End Trigger".  The start trigger is any valid expression that evaluates to true/false.  You can have multiple "rows" to the start trigger and there's also a Time True setting for the start trigger.  Unlike PI Notifications, there is currently no builtin setting for deadbands, retry-windows, durations, etc.  However, you can cobble together some functions to serve these purposes.  You can perform backfill for these event frames.  You can also create what's known as "root cause" child event frames.  Beyond the current available functionalities, you can create event frames programmatically via the AF SDK.


                              There is a fairly rich set of search functions for event frames.  For details, please consult the AF SDK programming manual.


                              For status, I would configured an enum attribute in the event frame template.


                              In short, it's not completely straight forward to do what you need with asset analytics, but should be close.  If you're considering programmatic means, I think it's possible to do what you need.



                              Steve Kwan

                              AF Product Manager

                              • Re: Notifications or Abacus for a 'data quality event queue'?
                                Roger Palmen

                                Some short answers (Steve already indcated it can be done roughly, but i may require piecing logic together (e.g. creating triggering PI Points to drive the EventFrame creation).

                                • If you need deadbands, you need to build that hysteresis in AF. Minimum durations are in analytics by default. retry-windows are not really applicable imho, as an unprocessed quality deviation remains inprocessed in your workflow.
                                • Search capability for EF's in PSE is limited. I'd typically use a custom app to support the workflow. Notifications are not great either, so no big loss here.
                                • Add attributes as much as you like.
                                • AFSDK can do anything


                                Yes, we typically build a similar stack, where we use a WebService exposing the basic functions to a (web)application to support the workflow for processing the data quality EventFrames.