16 Replies Latest reply on Jan 19, 2017 6:11 PM by SvenBatalla

    How do I get a list of notification instances quickly?

    SvenBatalla

      I have recently been tasked with attaining a list of PI-Notification instances for the purposes of visualisation for our user base.  This visual needs to include a certain amount of information, but I am having trouble just getting the list at all.  I require help from the community to understand how to quickly retrieve the information I require.

       

      Note:  I am using PI-AF and PI-Notifications 2016.

       

      For clarity, I am looking for instances and not templates.  It is very simple and very quick to get the list of PI-Notification templates, but I am looking for the instances.  So what is an "instance"?  Well, imagine I have an element template called "Pump" and then I associate a PI-Notification template to it.  An "instance" of a notification is each "real" notification that is created as a result of the implementation of said "Pump" element template.  So if I have 1 PI-Notification template on the "Pump" element template and 1,000 elements using that element template, then I have 1,000 notification instances.

       

      At this point, I believe I have 2 ways to gather the list of notification instances:

      1. The first option is to get the list of all event frames, the elements that they are associated with, and then any PI-Notifications associated to those elements.  The main problem with this (unless I am mistaken) is that this only works if an event frame has been created for the notification.  So if a notification instance has never become active, there would not be an event frame, and thus I would not get back a notification instance.  Beyond that, you can imagine the performance of this is very bad.
      2. The second option is to get the list of all elements and then any PI-Notifications associated to those elements.  This takes event frames out of the question and boosts performance (although not by much and still not acceptably so).  However, it also means that I can't easily know which notification instances are active.  Remember I am trying to collect information about each notification instance too.

       

      My understanding is that OSIsoft uses their own SDKs in their services and clients.  That means that the PI-Notifications service must have to collect a lot of this information already in order to do its job.  So my conclusion is that I must be doing something wrong or backwards.  Can anyone in the community provide me some sample code for retrieving a list of PI-Notification instances in a good-performing manner?

        • Re: How do I get a list of notification instances quickly?
          Rick Davin

          Hi Sven,

           

          Depends upon what you define as an notification instance, since it no longer has the same meaning with PI Notificaions 2016.  I would call a notification instance to be a individual event frame generated by a notification rule.  But that's just my personal definition.  If you have 1 notification rule template and 1000 elements, then you would have 1000 notification rules.  But if each element's notification rule was triggered 10 times, then you would have 10,000 event frames, which some may refer to loosely as a "notification instances".

           

          If you have a given AFElement (let's call it element), you may first grab all of its notification rules quite easily:

           

          AFNotificationRules rules = element.NotificationRules
          

           

          So you loop over or grab a specific notification rule (remember belonging to element) and now you can search for the notifications as event frames:

           

          AFNotificationRule rule = rules[0];
          AFDatabase database = rule.Database;
          AFElement targetElement = rule.Target;  // maybe to confirm it's the right element
          string query = string.Format("Element:'{0}' {1}", targetElement.GetPath(database), rule.Criteria);
          AFEventFrameSearch search = new AFEventFrameSearch(database, "", AFSearchMode.Overlapped, startTime, endTime, query);
          IEnumerable<AFEventFrame> instances = search.FindEventFrames(startIndex: 0, fullLoad: false, pageSize: 1000);
          

           

          Don't know if that's the most proper or best way, but it's what I've stumbled upon that works.

           

          Rick

          2 of 2 people found this helpful
            • Re: How do I get a list of notification instances quickly?
              SvenBatalla

              Rick,

               

              Thank you for that clarification.  I do indeed mean "notification rule" then.  I was not aware of the nomenclature, so thank you for setting that strait.

               

              All that being said, I think I still have an issue.  In you example, I would still need the list of all elements in the system in order to get the rules.  Is there a way I can ask PI-AF for all the rules in the system and then derive the elements the other way?  That's probably 6 of one and a half-dozen of the other, but it may be something I can work with.

               

              Thanks,

              Sven

                • Re: How do I get a list of notification instances quickly?
                  Rick Davin

                  See Mike's tip in AFNotificationRuleSearch.

                   

                  My code shows a given notification rule has a Target property for the applicable AFElement.

                    • Re: How do I get a list of notification instances quickly?
                      SvenBatalla

                      Well, I suppose this is to both of you.  I have run the following very simple code that I assume gets me all the rules in the system:

                       

                      var searcher = new AFNotificationRuleSearch(afDatabase, "*", string.Empty);
                      IEnumerable<AFNotificationRule> rules = searcher.FindNotificationRules();
                      

                       

                      This code seems to work well enough.  I can progress from here.  It's telling me I only have 2 rules in the system, so I need to confirm that.  Once I get the number of rules up a couple of order of magnitudes, then I'll be able to determine if the performance is sufficient.  I'll update the ticket at that time.  If it works, I'll make this as answered and give you guys the credit.  Thanks for your help!

                        • Re: How do I get a list of notification instances quickly?
                          Mike Zboray

                          You could double check that via PISystem.GetObjectCounts method. Its the NotificationRule identity, not Notification.

                           

                          Also depending on what exactly your application is doing you may want to use CacheTimeout. It is beneficial if you are writing something that is going to iterate all notification objects anyway. This allows the AF server to prefetch and cache pages for you. It is wasteful if you are writing a UI where the user might only view the first page or so of results and then try another search. I have found if you use the CacheTimeout in a service it is best to call Refresh before starting iteration since results from the previous execution may still be in the AF server.

                          1 of 1 people found this helpful
                            • Re: How do I get a list of notification instances quickly?
                              SvenBatalla

                              Mike,

                               

                              An example might be useful of what you are talking about.  I have done a quick test on this side and it was not promising.  Here's what I've done:

                               

                              Console.WriteLine("Retrieving all notification rules...");
                              Stopwatch sw = Stopwatch.StartNew();
                              var searcher = new AFNotificationRuleSearch(afDatabase, "*", string.Empty);
                              IEnumerable<AFNotificationRule> rules = searcher.FindNotificationRules();
                              Console.WriteLine("Found [{0:#,##0}] rules in [{1}]", rules.Count(), sw.Elapsed);
                              
                              var sb = new StringBuilder();
                              sw.Restart();
                              foreach (AFNotificationRule rule in rules)
                              {
                                  sb.AppendFormat("[{0}][{1}] | [{2}][{3}]", rule.Name, rule.Description, rule.Target.Name, rule.Target.Description);
                              }
                              Console.WriteLine("Iterating took [{0}]", sw.Elapsed);
                              

                               

                              Note:  The StringBuilder is just so that I can access the properties of the AFNotificationRule object without incurring cost of writing to the console.

                              The results of this test are as follows:

                               

                              Retrieving notification rules...
                              Found [2,600] rules in [00:00:01.0221170]
                              Iterating took [00:01:01.5679022]
                              

                               

                              What I've found is that the performance does not appear to be affected by the number of rules, but rather by the number of elements that are collected in the iteration.  In this case, I have 1,300 elements.  I had 1 notification template and thus 1,300 rules.  I added another notification template and the performance was unaffected despite now having double the rules (2,600).  I imagine that if I scale the number of elements up, the performance is going to be restrictive.

                               

                              And that brings us back to your point with CacheTimeout and other such options.  Perhaps the AFNotificationRuleSearch can include certain search terms?

                               

                              Thanks,

                              Sven

                                • Re: How do I get a list of notification instances quickly?
                                  Mike Zboray

                                  Sven,

                                  1. If you want to get the count efficiently, use GetTotalCount() on the search object rather than FindNotificationRules().Count(). The former asks the server for the count and returns only that. The latter will bring over all notification rules and just throw them away.

                                  2. You are accessing the Target which is not brought over by FindNotificationRules by default. Each access of Target is causing another call to the AF server. Instead, make the call with the fullLoad parameter set to true if you are immediately going to start accessing other objects on the NotificationRule: IEnumerable<AFNotificationRule> rules = searcher.FindNotificationRules(fullLoad: true);

                                  This will of course slow down the retrieval of notification rules because you are collective more data before bringing them over.

                                  3. Also note that FindNotificationRules does not fully realize the result when you make the call. As you iterate, it makes additional calls to fetch the notification rules in pages (default size is 1000).
                                  4. Yes CacheTimeout can make a huge difference.

                                  5. I"m not sure what you mean by "include certain search terms".

                                  What level of performance are you trying to achieve? The notifications service is able to load 1M notifications with 100k-200k targets in 3-6 mins with these calls.

                                  Mike

                                    • Re: How do I get a list of notification instances quickly?
                                      SvenBatalla

                                      That is a valid question and one I should have been clear on from the beginning.  Ideally I am looking to retrieve the list of all the notification rules in the system as well as the name/description of their target element and whether the most recent instance of the notification is currently still active.  The number of notification rules would be at least 300,000 though the target is 3,000,000.  The element count is unknown to me, but if we consider 5 rules per element, that would likely be sufficient (so 60,000 - 600,000 elements).  This information is then to be presented to the client in a UI (nevermind yet the logistics of presenting the volume of data) in a "usable" performance time of ~2 seconds.

                                       

                                      So the simplest question to ask is:  can PI-Notifications 2016 achieve this?  Then, if it can, how can I pragmatically retrieve this information in ~2 seconds?

                                      • Re: How do I get a list of notification instances quickly?
                                        Rick Davin

                                        Hi Mike,

                                         

                                        This is tangential to Sven's issues but might help me understand some things better. I'm expressing the things in my head as declarations but they really are questions.

                                         

                                        Loading Target

                                         

                                        The search on notification rules returns notifications rules and each has a Target property for the associated element.  I would have thought this was similar to issuing a FindElements in that its basic headers are loaded as the element(s) are found.  Certainly if one then wanted to reference attributes on an element that a trip is required to LoadElement.  Or one could batch many together for a LoadElements call.  Are you saying that with fullLoad: false that even the Target headers aren't loaded?

                                         

                                        Caching because its IEnumerable

                                         

                                        I've looked at other examples of CacheTimeout and simply mirrored what I've seen before.  It finally dawned on me that if you plan on iterating over the collection more than once that you probably want to enable caching.  Or if you know the collection is small enough then you can ToList() it.  But since the search methods return an IEnumerable, the collection really isn't persisted in client memory but rather re-generated with each foreach or even LINQ call.  Is my reasoning in the right ballpark?

                          • Re: How do I get a list of notification instances quickly?
                            Mike Zboray

                            The underlying object in AFSDK for Notifications 2016 is AFNotificationRule (and AFNotificationRuleTemplate for templates). Use the new AFNotificationRuleSearch class to search a given AF database.

                            2 of 2 people found this helpful
                              • Re: How do I get a list of notification instances quickly?
                                Rick Davin

                                Mike,

                                 

                                My one-line example found notification rules belonging to one element.  Isn't that the easiest in that case?  Would the AFNotificationRuleSearch be more of what Sven is looking for if he wants to search for many rules across many elements?

                                 

                                If you wanted to find the event frames generated by a given notification rule, what's a good piece of code to achieve that?

                                 

                                Thanks,

                                Rick

                                  • Re: How do I get a list of notification instances quickly?
                                    Mike Zboray

                                    Right. There's a NotificationRules collection on AFElement, but it sounded like Sven wanted NRs across many elements and couldn't see how to do that. That's what AFNotificationRuleSearch is useful for.

                                    That snippet is a reasonable way to find all EFs for a given notification rule. Originally I thought this was what was being asked for because "instances" in the old ANSDK are analogous to event frames.

                                • Re: How do I get a list of notification instances quickly?
                                  SvenBatalla

                                  After much investigation, it was determined that what I need to do with PI Notifications simply is not possible.  It is certainly possible to achieve a large number of notifications, but marrying that to elements and gathering large lists for visualization purposes appears to be a no-go without some intermediary system and caching.  That implies an acceptable loss to "real time" access to information.  There are quite a number of other reasons this won't work, but suffice it to say that it all resides under the umbrella of "performance."

                                   

                                  Sadly, we have abandoned this course of action and will be looking towards tools other than PI Notifications.