20 Replies Latest reply on Dec 8, 2016 8:20 PM by Roger Palmen

    Can backfilling be applied to Custom AF data reference ?

    Paurav Joshi

      Hello Folks,

       

      I have one query that if I create one custom AF Data Reference, than can I apply backfilling to that Custom Data Reference?

      If yes, then what will be minimum AF SDK requirement for the same?

       

      Thanks,
      Paurav Joshi

        • Re: Can backfilling be applied to Custom AF data reference ?
          John Messinger

          Paurav,

          So long as your custom DR allows for persistent storage of historical results, then you should be able to apply backfilling to an Analysis that outputs to an attribute configured with that DR. As far as minimum version of the AF SDK, don't know for sure, but I would suggest so long as it supports the RDA methods you're probably OK.

          What is the data source for your custom DR?

           

          John

            • Re: Can backfilling be applied to Custom AF data reference ?
              gregor

              So is the question than: "What method does my Custom Data Reference need to implement in order to allow Analysis backfilling?"

              • Re: Can backfilling be applied to Custom AF data reference ?
                Paurav Joshi

                HI John,

                 

                My Custom DR will have i/p one PI Tag, and o/p is another PI Tag.

                So long as your custom DR allows for persistent storage of historical results, then you should be able to apply backfilling to an Analysis that outputs to an attribute configured with that DR.

                That's very good point . With PI Point as o/p, persistent storage concern has been taken care. It will be great if you can confirm the same or suggest alternative storage method for it.

                 

                My plan is to use backfilling method directly on custom DR .

                 

                Thanks,

                Paurav Joshi

                  • Re: Can backfilling be applied to Custom AF data reference ?
                    John Messinger

                    OK, so the results of your DR are written back to a PI point - that gives you persistent data storage for historical results. Now, as per Gregor's comment, so long as your custom DR includes SetValue in your override of SupportedMethods you should be fine to backfill to this DR's owning attribute.

                     

                    Regards,

                    John

                    1 of 1 people found this helpful
                      • Re: Can backfilling be applied to Custom AF data reference ?
                        Paurav Joshi

                        Hi John,

                         

                        One thought pass through while checking SetValue method that My DR won't have any dependent calculations, so I can actually do recalculations using this method , what say? (In expensive way, delete the data and recalc them for given time-period)

                         

                        One tricky query also arises here that I will use this Custom DR to almost 100 attributes, and recalculating them one by one is wasting whole intention of template and DR.

                        If in-case data has not come for 15 days period and we backfilled using piconfig afterwards, then there is need to recalculate this DR. What is better approach for this scenario you suggest? Can recalculation happen in template and will be applied to all elements inherited from that template?

                         

                        In my current scenario, we are using PI ACE and we can easily recalculate multiple contexts at once. Our plan is to move from ACE to AF, and DR fits best in problem statement as of now. Just checking all the exception scenarios now before starting efforts of custom DR .

                         

                        Thanks,

                        Paurav Joshi

                          • Re: Can backfilling be applied to Custom AF data reference ?
                            gregor

                            Hello Paurav,

                             

                            The PI Point Data Reference offers the functionality that you are looking for out of the box? I strongly believe, not just because I am an OSIsoft employee, that time series data should be written to PI Points.

                             

                            In many cases end users report performance issues or even data access timeouts, a Custom Data Reference (CDR) is identified as the root cause. There might be cases justifying a Custom Data Reference but the usage should be considered as an exception and limited to where absolutely necessary.

                             

                            Can you please explain your use case and why it requires a CDR?

                              • Re: Can backfilling be applied to Custom AF data reference ?
                                Paurav Joshi

                                Hi Gregor,

                                The PI Point Data Reference offers the functionality that you are looking for out of the box? I strongly believe, not just because I am an OSIsoft employee, that time series data should be written to PI Points.

                                I agree on that one regardless of employer , and my CDR will store data in PI Point only as I mentioned above.

                                There might be cases justifying a Custom Data Reference but the usage should be considered as an exception and limited to where absolutely necessary.

                                Totally understood that CDR will be considered only when it is necessary.

                                why it requires a CDR?

                                Answering why first because it is driving force .

                                We want to move from ACE to AF, and current AF Analysis has limited ability to control output timestamp ( i.e., consider the case of event-based analysis, and store o/p of analysis to last 10th minute interval. Let me know if analysis can be applied to this due to new feature upgrades).

                                Can you please explain your use case?

                                As you get some idea from above answer, let me explain in detail.

                                 

                                I have PI Point values updating every 10 seconds, and I want avg. of 10 minute values stored @ last 10th minute interval.

                                For example, if data of X tag has come @ timestamps 04:20:00,..., 04:21:30,..., 04:23:10,..., 04:29:50 then avg. of all these values should be stored @ 04:20:00.

                                 

                                The main twist here is that most of the times data come with delay of 10-15 minutes, which is exception case for AF Analysis as per my knowledge. There are times when data come with more delay. We have used ACE calculations as of now to mitigate this exceptional scenarios.

                                 

                                Let me know better option than CDR or ACE which will fulfill this requirement.

                                 

                                Thanks,

                                Paurav Joshi

                                  • Re: Can backfilling be applied to Custom AF data reference ?
                                    gregor

                                    Hello Paurav,

                                     

                                    Manual Recalculation is a feature that was recently added with AF Client 2016 R2. There was the requirement to add a Remote Procedure Call (RPC) to the PI Data Archive. Stephen Kwan explains the details very well with Manual Recalculation for Analytics - Timestamps.

                                    So, your question for the version requirements can be answered:

                                    • PI Data Archive 2016 (3.4.400.1162)
                                    • PI AF Client 2016 R2 (2.8.5.7759) which has been replaced with PI AF Client 2016 R2 SP1 (2.8.6.7801) due to a known issue fixed with this service release

                                     

                                    Generally, the move away from PI ACE in favor of Asset Analytics is a good one. I understand that your problem with doing so is too few flexibility in setting a results timestamps. Rushing into creating a CDR to address the limitation however would not be my first reflex, especially not after reading that others feel restricted by the existing functionality too (see e.g. Time Syntax for Start of Shift).

                                     

                                    Hello Stephen Kwan and David Pugal! Are we planning to offer better flexibility in setting an Analysis result timestamp?

                                    1 of 1 people found this helpful
                                      • Re: Can backfilling be applied to Custom AF data reference ?
                                        skwan

                                        Paurav:

                                        Data references are evaluated on the client side and is typically done in an adhoc manner - meaning there is no scheduler.  So if you look at the Formula Data Reference, typically it is only evaluated when the user explicitly does a refresh in PSE or similar.  In the description of your use case, it seems you want the calculation (average) to be executed only when all the required data has arrived.  Can you tell me if a user will be explicitly executing this Custom Data Reference when all the data has arrived or will you be using some kind of a scheduler?

                                        I see two issues that would prevent you from using asset analytics right now.  First, we have no idea when all the required data has arrived to initiate the calculations.  You as a human being, know that the data comes every 10 secs but the analytics engine does not know that.  To overcome this we would need some way to indicate that all the required data has arrived.  Second, the output timestamp override currently only supports an absolute or relative time.  I think what we need is a way to use an attribute value as a substitution parameter for the output timestamp override.

                                        Lastly, just for conversation, analytics was originally designed to be an online, real time, calculation engine.  It assumes that you want the calculation to trigger based on new events or based on a clock whereby data comes in a streaming fashion.  What I've learned is that users are actually more interested in an adhoc calculation engine whereby timeliness of the calculation is not the biggest requirement and data doesn't always come in a stream, but rather they come when they come :-).

                                        --

                                        Steve Kwan

                                        4 of 4 people found this helpful
                                          • Re: Can backfilling be applied to Custom AF data reference ?
                                            Roger Palmen

                                            "I think what we need is a way to use an attribute value as a substitution parameter for the output timestamp override".

                                             

                                            I think that too!

                                            • Re: Can backfilling be applied to Custom AF data reference ?
                                              Paurav Joshi

                                              Pardon me for confusing with problem statement.

                                               

                                              Thanks a lot Steve for comment .

                                               

                                              Can you tell me if a user will be explicitly executing this Custom Data Reference when all the data has arrived or will you be using some kind of a scheduler?

                                              It will be like event-based analyses, and average is just one example here. I shall use AFDataPipe for this purpose.

                                              For Example, current time is 04:33 and new data come with timestamp 04:21 due to delay than change happen in value of 04:20 not 04:30. We are doing this with ACE as of now.

                                               

                                              I think what we need is a way to use an attribute value as a substitution parameter for the output timestamp override.

                                              Absolutely in agreement with you .

                                               

                                              users are actually more interested in an adhoc calculation engine whereby timeliness of the calculation is not the biggest requirement and data doesn't always come in a stream, but rather they come when they come.

                                              You surely have come through so many requests, can you give some examples from it in which adhoc calculation has more significance (because I think I haven't understood it completely ) .

                                               

                                              Thanks,

                                              Paurav Joshi

                                                • Re: Can backfilling be applied to Custom AF data reference ?
                                                  skwan

                                                  Hi Paurav:

                                                  Adhoc calculation is just my way of describing calculations that don't necessary depend on inputs that come in a streaming fashion, thus assumptions that we made when we first built the system don't work so well.  As an example, the current system works best when inputs streams fill the data cache is a first in-first out fashion.  We optimized the system to take advantage of this.  This breaks down somewhat in situations like AMI meters where data comes in as a dump in 15 minute intervals and out-of-order and late arriving data are common scenarios.  (Out-of-order and late arriving data are not the same.  Late arriving data is data that comes in late relative to its timestamp but still in chronological order.)

                                                  Streaming calculations are really good at pre-calculating results and these results are available with every new triggering input.  Adhoc calculations are like using a calculator - you wait until all the inputs are available, then you run the calculations, or you run the calculation based on some other criteria other than new triggering input or new triggering clock schedule.  You may want to run an adhoc calculation because you need some results for a report.  This typically is not triggered based on streaming inputs.

                                                  --

                                                  Steve Kwan

                                                  1 of 1 people found this helpful
                                                • Re: Can backfilling be applied to Custom AF data reference ?
                                                  Roger Palmen

                                                  Coming acros this thread again.. Would not be an analysis be even more powerful to be able to set the output timestamp? Then you can pull in data from an attribute, but also do any processing within the context of the analysis calculation.

                                              • Re: Can backfilling be applied to Custom AF data reference ?
                                                Roger Palmen

                                                Paurav wrote:

                                                 

                                                 

                                                I have PI Point values updating every 10 seconds, and I want avg. of 10 minute values stored @ last 10th minute interval.

                                                For example, if data of X tag has come @ timestamps 04:20:00,..., 04:21:30,..., 04:23:10,..., 04:29:50 then avg. of all these values should be stored @ 04:20:00.

                                                 

                                                The main twist here is that most of the times data come with delay of 10-15 minutes, which is exception case for AF Analysis as per my knowledge. There are times when data come with more delay. We have used ACE calculations as of now to mitigate this exceptional scenarios.

                                                 

                                                 

                                                 

                                                I am thinking about the raw data that is available, and the data that is derived from that using various business rules. In this case, if there is no requirement to establish a new data record, but just to answer the question: what is the average 10-minute value of a PI Point at a specific point in time (@4:20), then that data is already available and can be delivered using the standard capabilities of the PI Point DR. You could solve that question on the client side: just ask for the value at the given timestamp. And don't ask the question too early...

                                                 

                                                If you want to establish a record, why not run an Analysis for every 10-minute interval? If you need to wait max. 15 minutes, just shift the output timestamp 15 minutes backwards. It's not a beautiful solution, but it should work. If there is no event in the 10-minute period, just use NoOutputI().

                                                  • Re: Can backfilling be applied to Custom AF data reference ?
                                                    Paurav Joshi

                                                    Hi Roger,

                                                     

                                                    Thanks for suggestion, and we have first applied clock-based analysis only to tackle this issue with 10 minute period. Due to unknown delay (we can never predict, 10-15 minutes is what we understood by crunching data that also not covering all scenarios), we are missing events in o/p tag of analysis. Then we have to move to ACE, because requirement is we can't miss any data in o/p tag.

                                                     

                                                    Another use case of same solution is, there is one tag which is updating only 4 to 5 times in month and it comes generally with unknown delay. We have captured only 1 in two months using analysis configured same as mentioned, despite 9 instances in i/p tag.

                                                     

                                                    Thanks,

                                                    Paurav Joshi

                                                      • Re: Can backfilling be applied to Custom AF data reference ?
                                                        Roger Palmen

                                                        Theoretically i would say, if you never know when the last data item is coming in, you need to post-calculate and not pre-calculate. Then the value at the time of asking is the best available at that time. Otherwise you never know when you have a valid result! If you don't care that much about the calculation being 100% correct, then picking a wait time covering 99.99% of your cases should work ok.

                                                         

                                                        Agree that this construct is somewhat heavy on processing if the data is sparse. But on the other end, if you have the space in a particular system to run a few hundred useless (NoOutput) calculations per day, that beats custom development or ACE any time looking at the overall efffort. Processor time is mostly cheaper than Developer time.

                                                         

                                                        But i admit these kinds of scenarios are difficult. I tend to design and pushback to make things fit into standard PI tools, or use future-proof building blocks. But not everybody is in the position to do that!