59 Replies Latest reply on Oct 28, 2016 5:32 PM by skwan

    AF Analyses and Automatic Recalculations

    AlistairFrith

      I have been playing with the AF 2014 beta today, particularly with Abacus calculations (analyses) and writing the calculation results back to PI. I am wondering: if the calculation result is output to a basic attribute rather than a PI Point one, does that mean there will be no history? Or can it recalculate history on demand like the Formula data reference? I am assuming it can't do that and would probably have pretty poor performance if it tried to!

       

      If I write the output to a PI tag, I have the option of backfilling data, but not of re-calculating existing values if the inputs change (as far as I can see). Is there an equivelant to the Performance Equation Recalculator subsystem for Abacus? Or some way of forcing it to re-do calculations when a historical value for one of its inputs changes?

       

      --- Alistair.

        • Re: AF Abacus and Automatic Recalculations
          AlistairFrith

          Ah, just found this:Abacus and recalculation so it looks like it has been discussed but did recalculation actually get in there? I can't see it anywhere unless it is somewhere in event frames which, to be honest, I have not yet even started to get my head around.

           

          --- Alistair

            • Re: AF Abacus and Automatic Recalculations
              skwan

              Alistair:

               

              You are discouraged from writing an Abacus output to an non-PIPoint attribute.  First, you have no history, second, it will not perform at all.  As to recalculation, it will not be in version 1.  You do have alternatives.  You can delete the old outputs and then backfill over the same time period with the new equations.

               

              Beyond version 1, we are planning to provide a much more robust way to perform the analyses in an adhoc fashion.  There's much to be done under the hood in AF to make that happen.  We also have plans to support recalculation in some form, but there are numerous questions that I need input from our users.  For example, if you were to recalculate, would you want to delete the old results or would you want to preserve the old results (consider if you're a pharma company)?  In the latter case, what should we do with the old results?  Should I store them in additional tags?  If there are dependent calculations, would you want me to recalculate them also?  Recalculations can only be done with archived data and not snapshot data.  Is that acceptable?  The list of questions is quite long :-).  If you have an input on these topics, please let me know.

                • Re: AF Abacus and Automatic Recalculations
                  AlistairFrith

                  Ok, thanks, that is pretty much what I thought the situatioin must be. I can obviously delete PI values programatically but is there the facility from within the AF-SDK to instigate a backfill?

                   

                  In terms of inputs to your questions...

                  • I would think many customers would want the option of storing the original results. However, surely the best way to do this would be through the PI Audit mechanism? Doesn't that track changes to values in the PI Archive?
                  • I would  expect the same kind of recalculation cascades as you get with the PE recalculator.
                  • I think that customers would understand that recalculating can only use compressed archive data. Again this is what the PE recalculator and ACE currently do. If they want snapshot data to be used, they will just have to turn compresion off for those tags, unless you completely re-engineered the way compression works, for example:
                    • You could defer compression for a period, eg one month, so all snapshot data goes into the archive, allowing recalculations during that period to have the same precision as the original. Then after 1 month the data goes through the normal compression algorithm and any subsequent recalculation suffers the same loss of precision as at present.

                  Alistair

                   

                   

                    • Re: AF Abacus and Automatic Recalculations
                      I.Berry

                      Just to follow up on this, is there a way to programatically kick off an abacus backfill from the AF-SDK?

                       

                      I tried the following with powershell, however it doesn't seem to be kicking off. Is this the right way to go about it?

                       

                       

                       
                      $elements =[OSIsoft.AF.Asset.AFElement]::FindElement($PISystem,"2908f108-9e63-11e3-8037-005056b23c22")
                      
                      $element = $elements[0]
                      $testAnalysis= $element.Analyses[1]
                      
                      
                      $st = [DateTime]::Parse("1/2/2013")
                      $et = [DateTime]::Parse("1/3/2013")
                      $testcase = $testAnalysis.AddCase()
                      
                      $testcase.SetStartTime($st)
                      $testcase.SetEndTime($et)
                      
                      $res = $testAnalysis.Run($testcase)
                      

                       

                        • Re: AF Abacus and Automatic Recalculations
                          Rhys Kirk

                          Indeed it seemed as though it should be straightforward to kick off a recalculation but it doesn't appear to be so, and there is no object model publicly exposed.

                           

                          To kickoff a recalculation you need to communicate with the PI Analysis Manager which will communicate with the PI Recalculation Processor. Now there isn't any public methods to facilitate that communication.

                           

                          It can be done via code ( yes I had a sleepless night figuring out a working example) but it is highly likely not supported by OSIsoft.

                           

                          Would be interesting to hear if there are plans to allow developers to kick-off recalculations 'officially' via code.

                            • Re: AF Abacus and Automatic Recalculations
                              skwan

                              Recalculation introduces a lot of complications.  For example, what do you do with the old results?  Delete them?  That may be ok for some but not for others.  If old results cannot be deleted, where do we store new results?  The natural answer is to use additional tags.  What do we do then with the currently running analyses that are writing its results to the older tags?  Should they somehow be combined or at least referenced to each other through maybe AF elements?  Dependent analyses add even more complications.  If the results you're recalculating are used in other analyses, would you want them recalculated also?  If so, imagine the mess if they are configured to be event triggered.  Now we have to retrieve all the old events, line them time wise and then redo all of them.  Some of those events may have been compressed out.  These are some of the things we need to work through, both from a customer requirements perspective and also from a technical perspective, before we would officially support recalculations.  Data integrity is of the utmost importance for us.  That is why for AF 2014, we're supporting backfilling analyses only.

                               

                              Edit 2017-02, pthivierge: As of the 2016 R2 release, there are new features that allows to manage recalculation from the GUI, but also programatically using the PI AF SDK.

                              You may be interreseted looking into:

                              PI AF SDK - AFAnalysisService object

                              PI Server - Back fill or recalculate data for an analysis.

                               

                              3 of 3 people found this helpful
                                • Re: AF Abacus and Automatic Recalculations
                                  I.Berry

                                  Hi Steve,

                                   

                                  For now it would be enough to expose the backfill functionality (which is great!) via the SDK so you don't need to answer all those questions regarding what to do with dependent analyses and old results:)

                                   

                                  For the future gold plated recalculation functionality the following would be great:

                                   

                                  * Delete old results if they are in the same period (I wouldn't ask for a recalculation if I wanted to keep them, perhaps a preview functionality could help prevent errors)

                                   

                                  * Figure out any dependent analyses and execute them also

                                   

                                  For the platinum version, the ability to create additional tags might be nice, but I don't see it being as useful (plus it sounds much harder to implement).

                                   

                                  Kind Regards,

                                   

                                  Ivan

                                  • Re: AF Abacus and Automatic Recalculations
                                    AlistairFrith

                                    My naive developer view is that it should do the same as the Performance Equation Recalculator does. I.e. when an archive value of a tag is changed/inserted/deleted, work out what calculations are affected and re-do them over the appropriate period, cascading them down through all dependant calculations. However I also realise that it is far more complicated than that sounds and I can understand your desire to take the time to do it 'right'.

                                     

                                    --- Alistair.

                                      • Re: AF Abacus and Automatic Recalculations
                                        rhutchison

                                        I have a couple of analyses set up for an element template that has around 80 elements so far. Both analyses are daily calculations, and I want to backfill the results from the start of the month in question. I can backfill using the PI System Explorer for individual elements, but there doesn't seem to be a way to do this for multiple elements. I had a look at the SDK to see if it would be possible to write a little utility to do this but hit upon the same issue as noted by Ivan in this thread. I note with interest that Rhys has a solution, albeit not supported. Rhys - can you share the example you have, with the usual caveats, disclamers, etc, etc?

                                         

                                        Thanks,

                                         

                                        Ruaridh

                                          • Re: AF Abacus and Automatic Recalculations
                                            I.Berry

                                            Hi Ruaridh,

                                             

                                            You can back fill multiple analyses from the Analyses view in the PI System Explorer. "Backfill Checked Analyses".

                                             

                                            This has mostly satisfied the original requirement I had, in the Beta this functionality was available.

                                             

                                            Cheers,

                                             

                                            Ivan

                                              • Re: AF Abacus and Automatic Recalculations
                                                rhutchison

                                                Hi Ivan - many thanks! Not sure how I didn't spot that!

                                                 

                                                Ruaridh

                                                  • Re: AF Abacus and Automatic Recalculations
                                                    Roger Palmen

                                                    Just looking at a similar case. It's still quite odd that PSE has the capability to trigger a backfill on an Analysis, but the AFSDK does not officially. After all PSE is only using AFSDK... so if PSE can trigger a backfill there should be a way to do this through AFSDK. Or am i thinking too simplistic here?

                                                     

                                                    Edit: it just dawned upon me that PSE has a connection to the PI Analysis Service... Would be nice if we can trigger Analysis backfilling from code!

                                                      • Re: AF Abacus and Automatic Recalculations
                                                        Rhys Kirk

                                                        You can do it via code, it just wouldn't be supported how you need to do it.

                                                         

                                                        There are other things to think about though rather than just recalculating - removing old results being one.

                                                          • Re: AF Abacus and Automatic Recalculations
                                                            Roger Palmen

                                                            Let me guess, the PI Analytics Manager is a WCF service, so if you know the calls, you could call it directly?

                                                             

                                                            More interesting would be the position of OSIsoft: are there any plans to support programmatic access to the backfilling functionality?

                                                             

                                                            And in the slew of that we of course invite the other key topics for Analytics: controlling recalculation behaviour, remove / overwrite / add / merge of results; and the key item: control over the time context! (now always the trigger timestamp which got me started on this topic). A.k.a.: controlling output timestamp.

                                                              • Re: AF Abacus and Automatic Recalculations
                                                                skwan

                                                                Lots of questions, don't you guys have a day job?  

                                                                 

                                                                Ok, so yes it's possible to trigger backfill programmatically, but we're not quite ready to make that public yet as there's much refining to do.  As you know, once we make programmatic access public, we can't change it so it has to be near perfect.

                                                                 

                                                                Recalculation is another feature that we will provide but we need some new features in other parts of the PI System to make this seamless.  This is so you can control the recalculation behavior.  It's in our backlog.

                                                                 

                                                                Regarding output timestamp, it's being worked on right now by our developers just so you and Rhys would have 1 less topic to post on 

                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                    Roger Palmen

                                                                    Great to be able to hang around vCampus all day!  I think our day job is to stay in the frontline of technology to give our customers some future-proof results, so anticipating what we can expect over the next years is important.

                                                                     

                                                                    The discussion that pops up at almost every project is ACE versus Analytics versus AFmodel or a combination of all?

                                                                     

                                                                    Backfilling would be relatively easy compared to recalculation (which can be hugely complex), so controlling backfilling will allow external applications to control recalculation.

                                                                     

                                                                    Can you comment which key features you have currently planned for the next release? The roadmap only states support for future data and enhancements to the security model. We use this kind of information to justify workarounds.

                                                                      • Re: AF Abacus and Automatic Recalculations
                                                                        skwan

                                                                        Roger:

                                                                         

                                                                        I will assume you're using ACE in some of your projects.  How much of your ACE code cannot be done with analytics (abacus) right now?  I want to know so we can continue to improve analytics.  My goal is that customers should write as little code as possible for analytics.

                                                                         

                                                                        So you want to know about the next release, that means you don't want to have (good) surprises .  Let's see....

                                                                        • Support for future data - that's a big one as it includes some enhancements to signing up for updates.  You'll be able to signup for updates with a time horizon.  For details, consult the PI Server 2015 Beta, available from the Download Center.
                                                                        • Changes to the AF security model to support Mappings and Identities - in many ways similar to what's in the PI Data Archive.  There's a lot of work here to ensure backwards compatibility.  The goal here is to make AF more closely align with the PI Data Archive and to improve security management.
                                                                        • "Attribute enhancements" - you'll be able to set two flags on AF attributes - Excluded and Browsable.  This is to support two major use cases.  You may have an element template that has a superset of attributes but some instances from that template don't have (need) all the attributes.  You will be able to exclude them.  The other use case is that you may have attributes that you don't want users to be able to see in client tools such as Coresight.  Examples include attributes holding intermediate values, financial information, string builder outputs, etc.  You can mark them as non-browsable and your users won't see them in client tools.
                                                                        • We're working towards minimizing the footprint of our install kits.
                                                                        • Improvements to the analytics Management Plugin to PI System Explorer, mostly around performance and scalability.
                                                                        • Support future data as input to analyses.
                                                                        • Output timestamps for anlytics, which we already talked about.
                                                                        • We're working towards "unscheduled analyses".  This means you can create analyses like you do today, but they don't have to be scheduled.  They can be triggered on demand, or if they are used as inputs to scheduled analyses, we will trigger them as appropriate.  You will be able to use almost all the PE functions, but not all due to scalability concerns.  This is a big item.  I don't know how far we'll get but I've got this big whip and the developers try not to make eye contact with me in the hallway   (Ok, I'm kidding about the whip.)
                                                                        • Several new PE functions, some of which should help with analyzing data quality.
                                                                        • The usual bug fixes and performance improvements.
                                                                        Hope this helps your planning.
                                                                          • Re: AF Abacus and Automatic Recalculations
                                                                            mhalhead

                                                                            Hi Steve,

                                                                             

                                                                            [Quote user="Steve Kwan"]Regarding output timestamp, it's being worked on right now by our developers just so you and Rhys would have 1 less topic to post on

                                                                            You can add me to this list.

                                                                             

                                                                            [Quote user="Steve Kwan"] How much of your ACE code cannot be done with analytics (abacus) right now?

                                                                            I would guestimate roughly 60%. No controlling output times is one of the biggest issue; getting that would probably half this number (one of most used ACE modules tags a PLC/DCS totalizer and writes in out to a PI style two point output totalizer with a fair amount of logic to handle unscheduled resets such as a controller download). We also have a couple of modules that fall under the Machine Learning banner; e.g. pass a stream of data through a PCA Eigen Vector to get a model prediction. Some of the other calculation like oscillation detection would also be quite difficult to do in Abacus (I think); but I haven't thought that hard about it.

                                                                             

                                                                            [Quote user="Steve Kwan"]"Attribute enhancements" - you'll be able to set two flags on AF attributes - Excluded and Browsable.

                                                                            I'm glad to see this. This actually came up in a discussion with Alcoa last week. I was planning on ping you; you've saved yourself an email.

                                                                             

                                                                             

                                                                              • Re: AF Abacus and Automatic Recalculations
                                                                                Roger Palmen

                                                                                Hi Steve,

                                                                                 

                                                                                Thanks for the feature list! Just being lazy: don't want to start solving problems that by the time i've solved them, OSIsoft has resolved in a new feature!

                                                                                 

                                                                                Major likes go out to output timestamp and the attribute exclude / browse.

                                                                                 

                                                                                We now have ACE modules that perform automatic backfilling: If we need the sum of A and B, we only write results up to the time that we have all inputs available. (oldest value of both A&B). When new inputs arrive, we automatically backfill all results since the previous output. Once we have control over the output timestamp we can probably solve only part of that puzzle. I'll have to look into this to see how to translate this into AF which is fundamentally different than ACE.

                                                                                 

                                                                                Backfilling and recalculation are already complex topics on the functional side, so i truly understand than when you add technology into the mix, it quickly becomes very complex and decisions need to be made.

                                                                                 

                                                                                For more complex analysis, like the one Michael pointed out, we will always require custom calculation capabilities. Be that through DLL's / plugins or external engines (e.g. Matlab?). What would be the recommended way to achieve that? Custom DR's? Custom Analysis plugins? Or completely outside of Analytics?

                                                                                 

                                                                                And for any large-scale analytics, at some point we would need horizontal scaling into multiple Analytics engines, but let's first get the basics right! It takes time...

                                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                                    Roger Palmen

                                                                                    One more comment on this ACE custom backfilling: you could see this as similar as the current setting "CalculationWaitTimeinSeconds".

                                                                                    • The current setting is server-wide, applying to all calculations.
                                                                                    • The next step might be that for both clock- and event scheduled analysis, you can indicate a lagtime per analysis. E.g. the Monday morning report (at 00:00:00), should be run with a lag of 8 hours to allow the operator to enter his values at the end of his shift at 06:00:00.
                                                                                    • Another step would be to determine this lag dynamically from the analysis (or an attribute).
                                                                                    • One way to solve the automatic backfilling is to postpone execution of an event. Now we trigger an analysis when an event update of A or B comes down the eventpipe, and we only have the option to return NoOutput(). If we could postpone the processing of the event because we are missing some other events, we should be able to defer processing. A sort of pushing back the event into the eventpipe. Can't oversee all implications of this yet. Just a thought...
                                                                        • Re: AF Abacus and Automatic Recalculations
                                                                          AlistairFrith

                                                                          Hi Rhys,

                                                                           

                                                                          You have previously said:

                                                                          "You can do it via code, it just wouldn't be supported how you need to do it.

                                                                          There are other things to think about though rather than just recalculating - removing old results being one."

                                                                           

                                                                          One the understanding that we know this would be unsupported, can you elaborate on how we could do this? We need to assess the risks and benefits verses doing the whole thing in ACE which at the moment is the only alternative and has lots of other drawbacks.

                                                                           

                                                                          --- Alistair.

                                                                            • Re: AF Abacus and Automatic Recalculations
                                                                              gregor

                                                                              Hello Alistair,

                                                                               

                                                                              Please don't ease the fact that whatever Rhys found is not supported. This is an OSIsoft forum and for sure we discourage from posting any details about unsupported solutions. We are interested in open discussions and our information policy is pretty open too but as usual there are limits.

                                                                              It was mentioned that Asset Based Analytics currently does not offer any recalculation mechanism but that engineering is looking into the options offering this functionality in a future release. I understood that one of the requirements of the project you are currently working on is programmatically controlled recalculation. If this is a hard requirement, the only supported option is making use of the recalculation options existent in PI ACE (please see Daphne's post below).

                                                              • Re: AF Abacus and Automatic Recalculations
                                                                AlistairFrith

                                                                Sorry to keep on about this, but we are just starting a project with a number of requirements including programmatically controlled recalculation and we basically have 2 choices: Abacus or ACE. Abacus does not officially support this but I also have heard a rumour that ACE doesn't either: officially you have to use the ACE Manager. I can't see anything in the ACE User Guide about programmatically instigating recalculation so is this right: that either way we would be using an unsupported method? If this is the case I would be really interested in Rhys's Abacus method. Can you elaborate Rhys? If ACE does properly support this, can somebody point me at the documentation? I understand that such a solution would not be supported.

                                                                 

                                                                Many thanks,

                                                                 

                                                                --- Alistair.

                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                    dng

                                                                    Hi Alistair,

                                                                     

                                                                    Regarding triggering recalculation programmatically in ACE, perhaps you can check out the AddManualRecalculation and AddManualRecalculation2 methods from PIACEModuleDBFunctions (in PIACECommon). The methods take the following syntax:

                                                                    • AddManualRecalculation(strContextPath As String, ByRef StartTime As Double, ByRef EndTime As Double)
                                                                    • AddManualRecalculation(strContextPath As String, ByRef StartTime As Double, ByRef EndTime As Double, ByRef Interval As Double)

                                                                     

                                                                    In addition, you might find some interesting code snippets in the following posts:

                                                                     

                                                                    Unfortunately, the PIACECommon functions are not publicly documented (we do have an enhancement request for it). Note that the recalculation interval is set to 30 days or less in the ACE manager GUI because we wanted to limit recalculation interval since recalculation can be very expensive to the PI server. There might also be effects to the performance of the ACE scheduler if the time interval is set to a large value. \

                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                    LuanSandes

                                                                    Just checking if there were updates on the SDK regarding this topic. Is there a better way to kick off a recalculation with PI SDK 2016?

                                                              • Re: AF Abacus and Automatic Recalculations
                                                                TimCarmichael

                                                                Steve,

                                                                 

                                                                First, Abacus is looking like a VERY useful tool.

                                                                So... given a template with multiple analysies and multiple elements based on same, I can see that using the Analyses screen that my filter options are: Status or Analysis Template.

                                                                For the 'Status' view, can the header items (status, element, name, template, backfilling) be used as sorting mechanisms?

                                                                For example, I have a pair of analysis for a single template that I'd like to backfill, but it appears I have to find them and select them individually.

                                                                 

                                                                If this isn't a current feature, can it get added to the list?

                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                    skwan

                                                                    Tim:

                                                                     

                                                                    We're just getting started on the development cycle for the 2015 R2 release.  One of the things we're concentrating on for this R2 release is to improve manageability.  So yes we're definitely looking to improve the behavior of the Analysis Management Plugin.  There are a few reasons (not an excuse, just haven't gotten to them yet) for the lack of sorting is around paging.  With templates, it's very easy to have many thousands, if not ten and hundred thousands of analyses.  In such cases, we have to implement paging on the Analyses screen.  Otherwise the UI performance would be unacceptable if we have to load every single analysis.  With paging, sorting is not trivial, especially when you navigate between pages.  It can be confusing.  So we're looking into how we can make the management experience better.  It may or may not be sorting - we're evaluating many options.  Suffice to say it's definitely on the list.

                                                                     

                                                                    --

                                                                    Steve

                                                              • Re: AF Abacus and Automatic Recalculations
                                                                pthivierge

                                                                Hello Alistair,

                                                                 

                                                                Even though we do not have an official solution yet for recalculation, we are very interrested and commited at supporting our users.

                                                                I have a suggestion for you that could potentially help waiting for the official recalculation support of Assets Based Analytics, in a way that is definitely supported.

                                                                 

                                                                Recalculation can be complex and this was already explained by Stephen Kwan.  And you will have to adapt the following steps to you need, that means chosing to delete or replace existing events for example.

                                                                 

                                                                Steps:

                                                                For the time range that correspond to the re-calculation you want to do:

                                                                • Get all time stamps that exist in the period, and create a list of AFTime
                                                                • Delete existing event (optional, you could replace if you prefer )
                                                                • Use the list of AFtime, to execute every calculation for each time in the list.  Mike Zboray provided a nice example to Get Results from an Analysis: Re: Get Results from Analysis without defining an Output Attributes. I have not tested it yet, and I dont know how much this can scale, so I would tests it first.
                                                                • From the step above you should have created a list of AFValue which correspond to your recalculations, you then need to insert them in PI Data Archive, If you have a lot of values you shoud use the UpdateValues Method, it would perform better.

                                                                 

                                                                I would consider this as a temporary measure, until we get the new release of Assets Based Analytics that supports recalculation.

                                                                 

                                                                I hope this helps, and I am also waiting for others in the community to test or comment this workaround.

                                                                  • Re: AF Abacus and Automatic Recalculations
                                                                    pthivierge

                                                                    I have tested my suggestion and it works as I expected.

                                                                    So here is a full example that shows how to programatically recalculate data for a specific element.

                                                                    It runs the analyses on the element, create a list of values with it and update the associated output tags with it.

                                                                     

                                                                    It is just an example, I tested it and it works, but it certainly requires more testing and error handling etc...

                                                                     

                                                                    I really hope this helps you getting started with recalculation or backfilling until Asset Based Analytics fully support it out of the box.

                                                                    And let me know what you think.


                                                                    Here is the code:

                                                                     

                                                                    /// <summary>
                                                                    /// Example to recalculate data for all analyses of a specific element.
                                                                    /// Notes:
                                                                    /// This code is for demonstration only, you should adapt it with proper error handling...
                                                                    /// You need to pass a list of timestamp to be recalculated.
                                                                    /// Existing data is replaced.
                                                                    /// </summary>
                                                                    public static void ProgrammaticAnalysisRecalculation()
                                                                    {
                                                                        const string AFSERVER="TST-SRV-PI2014";
                                                                        const string AFDATABASE = "TestDatabase";
                                                                        const string AFELEMENT = "Element1";
                                                                    
                                                                    
                                                                        // we start by generating timestamps we want to recalculate
                                                                        // we could get them from existing recorded values, etc...
                                                                        //
                                                                        // here we simply generate yesterdays hourly values ...
                                                                        var recalculationTimeStamps = new List<AFTime>();
                                                                        for (int i = 0; i < 24; i++)
                                                                        {
                                                                            recalculationTimeStamps.Add(new AFTime(DateTime.Today.Subtract(TimeSpan.FromDays(1))+TimeSpan.FromHours(i)));
                                                                        }
                                                                    
                                                                    
                                                                        // connect to AF Server
                                                                        var afServer = AFSDKHelpers.GetSystem(AFSERVER);
                                                                        afServer.Connect();
                                                                        var database = afServer.Databases[AFDATABASE];
                                                                               
                                                                        // we select the element we want to recalculate
                                                                        var element = database.Elements[AFELEMENT];
                                                                               
                                                                        // for each analyse configured for our element
                                                                        foreach (var afAnalysis in element.Analyses)
                                                                        {
                                                                                 
                                                                            // we recalculate results
                                                                            var results = Calculate(afAnalysis, recalculationTimeStamps);
                                                                                 
                                                                            // we could delete values here, but I simply replce them instead
                                                                                 
                                                                            // we insert our new values
                                                                            AFListData.UpdateValues(results,AFUpdateOption.Replace);
                                                                        }
                                                                    
                                                                    
                                                                    
                                                                    
                                                                        // done!
                                                                               
                                                                    
                                                                    
                                                                    }
                                                                    
                                                                    
                                                                    /// <summary>
                                                                    /// Adapted from Mike's example
                                                                    /// Mike also talks about errors and warnings in his post, you should check it.
                                                                    /// </summary>
                                                                    /// <see cref="https://pisquare.osisoft.com/message/28537#28537"/>
                                                                    /// <param name="analysis"></param>
                                                                    /// <param name="times"></param>
                                                                    /// <returns></returns>
                                                                    private static List<AFValue> Calculate(AFAnalysis analysis, IEnumerable<AFTime> times)
                                                                    {
                                                                               
                                                                        var results=new List<AFValue>();
                                                                        var analysisConfiguration = analysis.AnalysisRule.GetConfiguration();
                                                                        var state = new AFAnalysisRuleState(analysisConfiguration);
                                                                        foreach (var time in times)
                                                                        {
                                                                            Console.WriteLine("Evaluating for {0}", time);
                                                                            state.Reset();
                                                                            state.SetExecutionTimeAndPopulateInputs(time);
                                                                            analysis.AnalysisRule.Run(state);
                                                                            if (state.EvaluationError == null)
                                                                            {
                                                                                // this merges the state (results) with the configuration so its easier to loop with both...
                                                                                var resultSet = analysisConfiguration.ResolvedOutputs.Zip(state.Outputs, Tuple.Create);
                                                                    
                                                                    
                                                                                foreach (var result in resultSet)
                                                                                {
                                                                                    // for more clarty, we take out our data into clearer variables
                                                                                    AFAnalysisRuleResolvedOutput analysisRow = result.Item1;
                                                                                    var calcRes = (AFValue)result.Item2;
                                                                    
                                                                    
                                                                                    // we filter to get only the results that have an output attribute ( an AFValue )
                                                                                    if (analysisRow.Attribute != null)
                                                                                    {
                                                                                        // add new AF Value into the results table
                                                                                        results.Add(new AFValue((AFAttribute)analysisRow.Attribute, calcRes.Value, calcRes.Timestamp));
                                                                                    }
                                                                                }
                                                                            }
                                                                            else
                                                                            {
                                                                                Console.WriteLine("An error occurred: {0}", state.EvaluationError.Message);
                                                                            }
                                                                        }
                                                                               
                                                                        return results;
                                                                    } 
                                                                    
                                                                    
                                                                    3 of 3 people found this helpful
                                                                      • Re: AF Abacus and Automatic Recalculations
                                                                        AlistairFrith

                                                                        This looks absolutely fantastic, AF is back in the frame, thank you so much! We will investigate further and let you know how we get on.

                                                                         

                                                                        --- Alistair


                                                                        • Re: AF Abacus and Automatic Recalculations
                                                                          Rhys Kirk

                                                                          Patrice...skim read your code. Won't you run into trouble if your calculation requires the previously calculated value as an input? You don't output the results until all time stamps have been processed. For example, if you were calculating a counter using a single PI Point.

                                                                            • Re: AF Abacus and Automatic Recalculations
                                                                              AlistairFrith

                                                                              Luckily, none of our calculations use their own outputs as an input so it is a problem for OSI's generic recalculation solution, not our specific one.

                                                                               

                                                                              Next questioin (I don't know if I should start a new thread for this)... Is there a way to find whether an analysis uses time-range functions such as TagAvg() or TagTot() and to discover the relative time ranges used in these functions? I ask because if the function does a TagAvg() over the last 24 hours, and a historical value on the input tag is changed, the recalculation should be performed for every value over the following 24 hours. This is probably a non-trivial task and presumably one of the things that OSI are battling with to produce a generic Abacus recalculation engine and is something that the Performence Equation Recalculator used to consume a lot of resources to do.

                                                                               

                                                                              For our application, a simple solution would do: All our time-summary calculations are performed over 24 hours, from 01:00 to 01:00 and I believe they are all TagTot() so I could do something really basic like look for TagTot() in the analysis equation and if it is found, extend the time range we are calculating forward by 24 hours. But that really is a fragile and non-scalable solution! Better would be to properly analyse the equation and parse the time-stamps supplied to these functions. Are there any SDK facilities to help with that?

                                                                              • Re: AF Abacus and Automatic Recalculations
                                                                                pthivierge

                                                                                Hi Rhys,

                                                                                 

                                                                                You are right, thanks for your input.

                                                                                 

                                                                                To overcome this, I would have to change the way the main logic works so we update values in PI after each timestep.  But there are tradeoffs to this stategy, and if this is really needed you would need to consider the following points:

                                                                                • The loop may go faster than the time it takes to the value to reach the archive, so it might be required to "throttle" the loop so the previous value can be read from the PI Data Archive, by adding a delay in the loop. Unless the data goes in the AF Cache but this is something I am not sure about at the moment, I am looking for an answer about it.
                                                                                • It is less efficient because you are calling update values more often.

                                                                                 

                                                                                The following code is for discussion only I would use it only for calculations that requires the previous input, otherwise I would prefer to use the one presented above.

                                                                                 

                                                                                 

                                                                                private void CalculateAndUpdate(AFAnalysis analysis, IEnumerable<AFTime> times)
                                                                                {
                                                                                  //... see post above
                                                                                  foreach (var time in times)
                                                                                  {
                                                                                  //... see post above
                                                                                  if (state.EvaluationError == null)
                                                                                  {
                                                                                  //... see post above
                                                                                
                                                                                
                                                                                  foreach (var result in resultSet)
                                                                                  {
                                                                                  //... see post above
                                                                                  }
                                                                                
                                                                                  // inserting values for current timestep
                                                                                  AFListData.UpdateValues(results, AFUpdateOption.Replace);
                                                                                  results.Clear();
                                                                                  }
                                                                                  else
                                                                                  {
                                                                                  Console.WriteLine("An error occurred: {0}", state.EvaluationError.Message);
                                                                                  }
                                                                                  }
                                                                                
                                                                                
                                                                                }
                                                                                
                                                                              • Re: AF Abacus and Automatic Recalculations
                                                                                AlistairFrith

                                                                                Our basic recalculation engine derived from your example code is generally working really well, but we have come across one issue:

                                                                                 

                                                                                Sometimes, due to 'bad' input values, the calculation fails. In your code if there are calculation errors they are printed out but the results are not collected. It does seem that errors can be reported even if parts of the calculation succeed, so I now collect and write output values even if errors are reported.

                                                                                 

                                                                                One symptom that seems a bit odd though is that I get errors even when all inputs are numeric if one or more of them is marked as 'questionable' and it does seem as though the outputs are not calculated in that case. What is the official behaviour of AnalysisRule.Run() when inputs have the Questionable flag set and can this behaviour be adjusted?

                                                                                 

                                                                                Regards,

                                                                                 

                                                                                --- Alistair.

                                                                                • Re: AF Abacus and Automatic Recalculations
                                                                                  D_Lozano08

                                                                                  Hello Patrice,

                                                                                   

                                                                                  I've been testing this implementation and it looks promising. The part that I'm stuck at is verifying the input values used in the recalculation. I noticed the bread and butter is the "state.SetExecutionTimeAndPopulateInputs(time);" This is the magic that is passing in those values to be used for the calculation. My scenario is my analysis is doing a 5min average and I need to figure out what values are being passed in. When doing the actual output verification there is a skew between if it was analyzed correctly and a backfill using this implementation. If you have done this before it'd be great to get your input.

                                                                              • Re: AF Abacus and Automatic Recalculations
                                                                                AlistairFrith

                                                                                Aha, thanks, yes that looks like it.

                                                                                 

                                                                                So all I have to do is wrap all my attributes inside Float()? That seems simple enough.