3 Replies Latest reply on Feb 10, 2016 5:34 PM by bshang

    PI/PI-ACE/MATLAB (prospective & retrospective analysis)



      I am new to the PI Square Community (but have had some experience with PI, primarily using PI SDK within MATLAB), and am  working on something that I'm hoping others have experienced and can provide good feedback on:

           --> looking initially at feasibility

           --> challenges to assumptions greatly appreciated!


      Objective: Generate new calculated value for each batch, based on time-series data from each batch

                - to be done for historical batches as well as new batches

                - calculated value should be contextualized (i.e., related to the particular batch)



           1) PI tag containing continuous data (sent from DCS)

           2) Associated unique IDs and other batch info in PI Batch DB (sent from DCS)...not yet using Event Frames

                - batches are regularly generated (though they vary somewhat in duration)

                - batches are not continuously generated (one immediately after another)

           3) MATLAB script to perform complex calculation, and output value


      General Plan:

           1) Use PI ACE as liaison between PI and MATLAB (passing/receiving COM objects)

           2) Use triggers for scheduling PI ACE data transfers/MATLAB calculations

           3) Write value to PI Archive in a new PI tag



           1) As I have heard that PI ACE itself is batch-agnostic, even with a batch-end-related trigger to initiate PI ACE, how does PI ACE know what data to select for the purposes of performing the calculation?

                     - can PI-SDK be used within PI ACE to access the PI Batch DB to perform segmentation of the continuous data?

           2) How can the calculated value be contextualized?  Assuming the value is calculated and written back to PI fairly quickly, I see how it can be somewhat associated with the batch based on timestamp.

                     - looking for a more direct link between new PI tag value and batch info

           3) Assuming the first two questions are addressed, and things are set up properly, can I have a secondary trigger to perform the calculations on all historical batches available in PI?

                     - perhaps a for loop in VBA in PI ACE to run all batches, and calculate values? (the contextualization of values mentioned in question 2 would be even more critical here)


      Thanks all in advance for your input!


        • Re: PI/PI-ACE/MATLAB (prospective & retrospective analysis)

          1) Yes, PI SDK can be used within an ACE calculation module. Just add the reference to the PI SDK and import the namespace. PI ACE calculations can be triggered by input tags that you specify. In this case, if you want to run the calculation on batch end, there should be an active tag that ACE can monitor that signals batch end. Upon closing of the batch, PI SDK can query the PI Batch DB for start/end times and associated tags of the batch.


          2) This is not as well supported with PI Batch DB and PI Data Archive. One option is to create a batch output tag to store results into. MDB can be used to create a simple hierarchy using PI Unit Modules for context. The calculated value can either be associated with the start or end time of the batch. There are different schools of thought on this. Storing at the beginning can make visualization easier if you use step=on for the tag. Then flat regions of the trend indicate periods of batch activity and show the batch-averaged value. May need to reset the value to 0 if no batches are running though. Storing at the end time makes sense in terms of casual time ordering; you only know the batch-averaged result at the end of the batch.


          If you have a lot of duplicated logic though spread across units, ACE can help as it can support contextual calculations (write code once, apply to many).


          Event frames answers a lot of these questions regarding context and batch data association though.


          3) It is certainly possible. You can do periodic scheduling in ACE and do recalculations using configuration options in ACE. However, these ACE features are better tested with non-batch oriented data so they may not work as well. In addition, ACE calculations should generally be "fast" as its designed to be a streaming calculation engine. Expensive "for loops" and mixing real-time with backfilling operations generally will cause issues.


          What are your requirements in terms of latency? ACE is most useful if you need tag-oriented data in real-time. However, if you can tolerate latency (i.e. run the calculation at end of every day, etc.), a custom Windows Task Scheduled application might be more suitable. You would be more in control of the batch-oriented logic. Other factors are the amount of data and number of batches processed per day. As the numbers get larger, then I would start to favor ACE more for its caching and management capabilities.

          1 of 1 people found this helpful
            • Re: PI/PI-ACE/MATLAB (prospective & retrospective analysis)

              Thanks for the reply Barry!

              As far as point 2, I figured Event Frames would be more suitable for contextualizing these data, but unfortunately, the implementation of this (particularly the timing) is outside my scope.  I may end up having to develop a workflow using Batch DB initially, and then develop a new workflow for once Event Frames is in place.


              As far as point 3, I envision needing to do backfilling only once for all historical batches (a few thousand)...then possibly on occasion if batch context data is not received in a timely manner.

              For ongoing batches, the calculation would only need to be done a few times a day; however, regarding latency, it would be preferred to run the calculation near real-time, so that the calculated value would be available soon after batch completion.

              Based on your input, it seems as though I should use PI ACE for generating the HETP values near real-time, and a Windows Task Scheduler app for doing the initial back-filling.  As far as the backup measure, either PI ACE or Windows Task Scheduler might work?

                • Re: PI/PI-ACE/MATLAB (prospective & retrospective analysis)

                  For the one-time backfilling of historical batches, I would do this outside of ACE, as ACE is a real-time calculation scheduler.


                  For the real-time calculations, it is possible to do this with ACE. It will depend on how the end of batches are signaled. If there is a tag(s) transition that indicates the batch end (1-->0 for example), then the ACE code can detect this and find the closed batch. If such a tag or tags do not exist, ACE can be periodically scheduled and during each execution period, it can query for batches closed since the last query (hopping windows).


                  For late-arriving data and/or recalculations, I would do this outside of ACE and perhaps use some of the logic from the one-time backfill application. The main reason is that there is significant calculation logic here involving batches, which ACE is not designed to handle well, so using the native recalculation feature of ACE would be more of a blackbox, while adopting the backfilling logic would give you more control over how the recalculations occur.

                  1 of 1 people found this helpful