21 Replies Latest reply on Nov 6, 2018 10:35 PM by skwan

    Automatic Event Frame backfilling (such as Analysis)

    Sergio_JCHP

      Hello all,

       

      In AF (PI System explorer) I can backfill my event frames without any problems.

       

      However, I could also see that automatic recalculation is not enabled - will this capability be released soon?

       

      If not, does anyone know what is the function to be used within AF SDK for that purpose? cannot find any

       

      Kind regards

        • Re: Automatic Event Frame backfilling (such as Analysis)
          John Messinger

          Hi Sergio,

           

          Automatic recalculation for Asset Analytics was released last month as part of the PI AF 2017 R2 release - see the release notes. However, Event Frame analyses are not supported for automatic recalculation (see Enhancement item 72498). I don't know whether EF generation analyses will have this support in a future release, as recalculation of event frames (not backfilling missing data) causes any annotations on the existing event frames to be lost. Stephen Kwan may have some more information as to what the future may bring for automatic recalculation, and he may even be willing to share something

           

          John

          1 of 1 people found this helpful
            • Re: Automatic Event Frame backfilling (such as Analysis)
              Sergio_JCHP

              Thanks John. Let's see if Stephen has something to share then .

               

              PS: any hint on the AF SDK syntaxis? I can really see a backfill function like you do from the Management tab in PI System explorer.

               

              Sergio

                • Re: Automatic Event Frame backfilling (such as Analysis)
                  John Messinger

                  Hi Sergio,

                   

                  Manual recalculation of any scheduled AFAnalysis can be accomplished using the AFAnalysisService.QueueCalculation method. I don't see anything new in the AF SDK that stands out specifically for automatic recalculation support. I implemented an automatic recalculation service for one of my customers last year using a combination of AFDataPipe and the AFAnalysisService.QueueCalculation method, and I suspect that the functionality now in AF 2017 R2 works along similar principles.

                   

                  John

                    • Re: Automatic Event Frame backfilling (such as Analysis)
                      skwan

                      Hi all,

                      As John indicated, auto-recalc for event frames is not yet available.  A major factor is indeed our concerns that end users may not know the impact of recalculating event frames.  As you know, event frames are typically very rich and contain a lot of information.  It's not unusual to have many attributes, extended properties, annotations, reason codes, acknowledgements, etc.  All this would be lost when recalculating in general and with auto-recalculation, it's possible that the user may not even know that it has happened.  We wanted to gather more input on user demands.  The actual implementation is very simple given all we have already done to build the framework for auto-recalculation.  Can you please describe your use case for auto-recalculating event frames?

                       

                      --

                      Steve Kwan

                      4 of 4 people found this helpful
                        • Re: Automatic Event Frame backfilling (such as Analysis)
                          JimGavigan

                          Steve - one scenario I see pretty consistently is to reprocess an event frame after lab values for that event have come in. I may finish a batch, reel, grade run, etc. before the lab values are available. I also believe that acknowledgements and annotations should be kept, or at least have a checkbox saying you want to keep them or wipe them out upon backfill. Maybe what we need is a way to automatically recapture values and not really "backfill" as the triggers may not have changed. Just some thoughts.

                            • Re: Automatic Event Frame backfilling (such as Analysis)
                              gregor

                              Hi Jim,

                               

                              If you specify attributes supposed to be updated with late arriving lab data as static attributes with the event frame template, this will allow you to update these attributes for generated event frames. Is this an option?

                                • Re: Automatic Event Frame backfilling (such as Analysis)
                                  JimGavigan

                                  Gregor - I would think so. I had one I worked on a few weeks ago where we had a product run that might last two days and we would get 4 lab values during the event - lets say they were 99,101,100 and 100 for an average of 100. However, the last lab value we had before the run started would also be included in the analysis (often it was much less or much more than 100) and would throw the average off. So, we had to build an additional analysis to get the right number.

                                   

                                  We were also talking through other events where the lab data might come in 4-8 hours after the event completed as well, so if we could mark these values to be recaptured in the event when the new value came in, that would be great.

                                    • Re: Automatic Event Frame backfilling (such as Analysis)
                                      gregor

                                      Hello Jim,

                                       

                                      Using a static attribute allows you to specify only one value which would than be the one valid for the entire event frame. Child event frames should allow you to add lab data valid for different phases.

                                       

                                      When using static event frame attributes, it doesn't really matter when they become added and there is no need to recapture, backfill or re-generate event frames because the static attributes exist with each event frame. It's possible to do this with the current release and likely with earlier releases too.

                                        • Re: Automatic Event Frame backfilling (such as Analysis)
                                          JimGavigan

                                          I am not sure I understand your suggestion, so let's walk through an example. Let's say I run a batch from 12:00-1:00pm. Phase A is a child event of the batch that runs from 12:00-12:20, Phase B is a child event that runs from 12:20-12:45, and Phase C is a child event that runs from 12:45-1:00. The lab takes a sample during Phase B, but the Quality A value doesn't get to PI until 2:45pm, with a timestamp of the sample time taken, say at 12:28pm. How do I get the Quality A sample that was taken at 12:28pm, but was received at 2:45pm associated to Phase B of the batch that ran between 12:00 and 1:00pm?

                                           

                                          Is it as simple as getting the Quality A PI tag value by time range - "After" and specify by time range - "End Time?"

                                          1 of 1 people found this helpful
                                    • Re: Automatic Event Frame backfilling (such as Analysis)
                                      skwan

                                      Jim,

                                      There are complexities.  Everything is done with respect with the start and end triggers, including recalculations.  With the current auto-recalculation capability for expressions, when there's an out-of-order trigger, we would kick off the recalculation.  Similarly, if we were to implement event frames analysis auto-recalculation, we would base that on out-of-order start triggers.  In your example of an out of order lab value, if it's not a start trigger, but rather a contributor to an event frame attribute, we wouldn't know if an event frame needs to be recalculated.

                                       

                                      When doing a event frame backfill/recalculation today, we first delete all the event frames within the starttime and endtime range, then churn through the triggers to re-create the event frames.  User supplies the starttime and endtime.  If you can imagine what auto-recalculation for event frame may look like - you get an out-of-order value that is used with a start trigger, we would find all the event frame analyses that uses this value as the start trigger, then sort out how many event frames we need to delete.  We may have to delete many.  For example, if you get an out-of-order value for a start trigger and there already exist event frames whole duration includes this out-of-order event, we would need to delete them because this out-of-order event may cause 1 event frame to be auto-recalculated to be 2 event frames.  In other word, if you get an out-or-order trigger at 12:05 and you already have an existing event frame that spans 12:00-12:10, then auto-recalculation many turn this one event frame into two event frames.  Or perhaps the out-of-order value may cause your existing event frame to be shortened or lengthened.  Now add to that dependencies.  The new event frames output capability in AF 2017 R2 can write outputs to another attribute which may be triggers for other analyses.  If we were to auto-recalculate event frames and thus recalculate these outputs, then it's possible that this would cause dependent analyses to be auto-recalculated.  When new outputs are auto-recalculated, if these new outputs are involved in summaries in dependent analyses (for example, a new value for a 1 hour average), we then have to determine the time range that needs to be auto-recalculated to ensure that summaries are accurate.

                                       

                                      In short, we actually do a lot in the background to support auto-recalculation.  Dependencies kill us, but it's the right thing to do.  Much to consider in designing this.

                                       

                                      --

                                      Steve Kwan

                                      4 of 4 people found this helpful
                                        • Re: Automatic Event Frame backfilling (such as Analysis)
                                          Sergio_JCHP

                                          Steven,

                                           

                                          I understand what you mean. However, I also understand that the only data loss we will have is for Attributes not linked to Pi Points. Of course, after your explanation I understand that all EF will be deleted and recalculated again, and I do not know if it will cause performance issues in a long run, but it might work if all Attrbitues are linked to PI Tags.

                                           

                                          Sergio

                                            • Re: Automatic Event Frame backfilling (such as Analysis)
                                              skwan

                                              Sergio,

                                              It's actually more than that.  An event frame can have annotations, extended properties, etc.  All these would be lost if the EF is deleted and re-created.  Not everyone uses these features, but as a product, we need to ensure we cover everyone's use case.

                                              --

                                              Steve Kwan

                                              1 of 1 people found this helpful
                                                • Re: Automatic Event Frame backfilling (such as Analysis)
                                                  Roger Palmen

                                                  Of course this is much more complex than meets the eye. But then again, quite probably > 90% of the cases do not hit these complexities. So, just as with Analytics Backfill or the Analysis DR, there may be limitations to keep the complexity under control.

                                                   

                                                  And of course, being able to configure if a specific EF Analysis is eligible for automatic recalculation is key here. The person implementing this must understand the consequences (potential data loss) when enabling, but he/she can be the best judge of that.

                                                  1 of 1 people found this helpful
                                            • Re: Automatic Event Frame backfilling (such as Analysis)
                                              skwan

                                              Jim,

                                              Recapture values can be pro grammatically or manually triggered if that's all you need.  However, we have no good way of knowing automatically when an out-of-order event may require a recapture.  We simply don't know the relationship between a value and all the event frames attributes that it's tied to.  Per my previous post, auto-recalculation is based on triggers.  We know the relationship between an attribute (and via data references, a value) and analysis triggers because we keep track of them for scheduling and updates purposes.

                                               

                                              --

                                              Steve Kwan

                                              2 of 2 people found this helpful
                                              • Re: Automatic Event Frame backfilling (such as Analysis)
                                                RickSmithJr

                                                I am late to the thread, but Jim is describing our most frequent case... Most of my cases would prefer a refresh of data based on the current start/stop event times than a delete/recreate scenario.

                                                 

                                                It is difficult to follow the scenarios and options in e-mail threads. Perhaps this would be a good UC discussion with a small group of people.

                                                2 of 2 people found this helpful
                                              • Re: Automatic Event Frame backfilling (such as Analysis)
                                                carpentercr

                                                Steve and all,

                                                I have a similar issue with event frames processing with the incorrect time stamps.  Some of our data still uses the UFL and is collected in 30 second chunks.  This data is used to trigger the event frames.  I assume that the event frame sees this data and runs from the snapshot which would give it incorrect times.  If the event frames are recalculated, everything is correct.  I assume this is probably because the analytics are now running from the archive and all the data is correct in the archive.

                                                 

                                                I would like to be able to delay the execution of the event frame by some period of time and force it to run from the archive.  Is any of this possible?

                                                 

                                                Charley

                                                  • Re: Automatic Event Frame backfilling (such as Analysis)
                                                    skwan

                                                    Charley,

                                                    When an analysis is triggered, all the values used for that analysis are based on the trigger time.  In your case, you have late arriving data whose timestamps are later than wall-clock time but that should be ok.  For example, if your UFL data being used as the trigger arrives at 12:01, but the timestamp of the value is 12:00:30, we would perform the calculation using 12:00:30 data for all the inputs.  If this happens to be the snapshot, fine, if not, we would get it from the archive.  There is one caveat, when using the snapshot value, by necessity we're extrapolating.  However, if we have to go to the archive, then it's likely that it would be interpolated.

                                                    I'm unsure of why you're not getting the correct results.  Perhaps you should contact tech support and we can help figure out what the problem is.

                                                    --

                                                    Steve Kwan