35 Replies Latest reply on Feb 9, 2012 9:54 AM by MvanderVeeken

    Future data in PI

      Some time ago I sat down and thought about how future data from the PI servers perspective would work after numerous conversations about how to implement a temporary solution for future data until we see it as standard functionality from PI.  Below is a very basic concept drawing I did at the time so thought I may as well share it to see how everyone else imagines they will be working with future data and PI - seen as though it is a hot topic right now.

       

      future_5F00_subsystem.jpg

       

      There are elements to this concept that miss out some details but what I had in my visiion was:

       

      - Future data is stored in standard archive files.
      - A new subsystem manages these archives, there shouldn't be any overhead to administrators.
      - Archives aren't created by an administrator, a time rule tuning parameter is set (e.g. *+6 months) and the subsystem handles everything else - self tunes!!
      - Snapshot subsystem remains relatively untouched except when future data arrives, it passes the event to the future subsystem.
      - When the future subsystem has events in its future archives that become current, it passes them back to the snapshot subsystem as if they just arrived impersonating the interface/application that originally sent the data.
      - Implement some form of tag security versioning.  For example, to cater for future data written 6 months ago but then 1 month ago security changes (e.g. PI Identity for the interface node).
      - Toggle exception/compression settings on/off for the future subsystem without effecting snapshot or archive subsystems.
      - Client tools would have to be explicit and ask to show future data, even then it displays differently to current data.  For example, in trends the trend background is darker or has a subtle pattern.  In datalink text colour changes etc.

       

      Ideally there would be some form of SSB (minimum of system state data; ideally to include event data) where the collective members communicate, for example, when new future archives are going to be created they collectively agree to create the archive rather than doing it in isolation. 

       

      When I get some more time I will also start a discussion on Interfaces to the AF system rather than the PI system, like we see in the newer PI AMI interfaces.

        • Re: Future data in PI
          Asle Frantzen

          Excellent thread, Rhys. You've certainly given some thought to this subject. We've had a few discussions of our own at the office regarding future data, and as you show here - it will require some good thinking before it's being implemented.

           

           

           

          Another question to consider is - should a tag with future data be "locked" to only display the future data, or should it be able to get live data as normal?

           

          If it's locked like that, you would need another tag to display the live data. Then your idea that the future subsystem would send data to the snapshot subsystem would be ok. Another advantage for this is being able to compare the predictive tag and the live data tag at a later time.

           

          But for some other cases you'd perhaps like to show future data and live data in the same tag. When live data reaches the snapshot as normal, it could either update the existing points or delete the data previously known as 'future data' before storing the data. 

           

           

           

          I would like to hear where OSIsoft stands regarding future data at the moment. Future data is not listed in the Engineering plan/Roadmap, but it's been there before if my memory serves me correct.

            • Re: Future data in PI
              MichaelvdV@Atos

              Just to give a quick heads up: we have contacted the PI Server Product Manager (Jay Lakumb) about this thread. I'm sure he will jump in soon.

                • Re: Future data in PI
                  janio

                  Hi!

                   

                  In the ONS - ISO of Brazil, the future data are very important but we can't use this data in PI without another technologics to access others database. Personaly I always thought in my mind: "Why they don't develop the future data in PI? it will be great".

                   

                  I think that many other companies use the future data whith others technologics but if the PI accept future data the future data will be in PI.

                   

                  I am waiting for the day that I can say: "Chief we can put these data (future data) in PI, we don't need the xxx database".

                   

                  It is great to know that OSIsoft is thinking about the future data in PI.

                   

                  Thank you!

                   

                  Janio Leonardo Los - PI System Manager in ONS - ISO of Brazil

                    • Re: Future data in PI

                      @Asle: My thoughts are still that future data still belongs to the same tag that is receiving the real time data.  The key is that tuning parameters allow different behaviour between the snapshot/archive subsystem and the future subsystem, so future data can be queued uncompressed (maybe you would want to compress future data too but it would be difficult to handle/determine out of order future events, because they aren't out of order until they are current) and then compression applied when they are current.

                       

                      Maybe it would be an option to allow a tag to not pass events from future to snapshot subsystem...

                       

                      The SDKs would remain relatively unchanged in that requesting data via existing methods from the time range "*-7d" to "*+7d" would return 14 days worth of data (7 historical, 7 future) but the future values are tagged with a "future" value attribute.  So the developer know that the data is queued and not processed by snapss/archss.  EventPipes would continue to work off snapshot and archive, but maybe there are use cases out there to be notified when future data has arrived for a tag - someone may want to adjust their process based on forecasted prices of raw materials?? Or look towards the next production order??

                       

                      Now here is the change the PI system would have to make changes....future archives need to be processed on-line to allow any new tag to have future data, they wouldn't always need historical data backloading (they start collecting from date of creation).  Maybe a point attribute to indicate to the future subsystem to add this tag to the future archives.

                       

                      @Janio: OSIsoft will need use cases, so if you have details of what future data you are storing and you can share then it would be great.

                        • Re: Future data in PI
                          Asle Frantzen

                          Ok, I see what you mean. So for the use case of storing the future data for comparison against the measured value, you would have to reserve a new tag.

                           

                          And if you're just after a prediction of the values to come, you could have em both in the same tag. But then you would not have the option of looking back at "historical future data" :)

                            • Re: Future data in PI
                              Ahmad Fattahi

                              What would be the biggest motive of having the feature? Is it for accommodating time drifts? Predictive purposes? Or something else?

                                • Re: Future data in PI
                                  Lonnie Bowling

                                  I had a real customer that wanted to store predictive values for future power consumption.  The power users were sometime requested to turn off equipment (to prevent rolling outages). The future values (when they became past values) were then compared to actuals and the net difference was to be used as a credit to the end user (assuming they were below the projected usage).   I think this is becoming very common in the power industry.

                                    • Re: Future data in PI
                                      janio

                                      Hi,

                                       

                                      Lonnie is right, in the electrical system, the future data is very important, we predict the generation, the load, the energy in the transformers, the forecast data like temperature, wind and rain, and many other things that are important to know about the behavior of the system.

                                       

                                      The future data in ISO of Brasil, often, are in the steps of 30 minutes and we can have 7 days in the future, today this data is stored in relational database and comes to the energy management system (EMS) in the real time and the interface with EMS and PI send the data to PI.

                                       

                                      The future data is stored in a private TAG because we want compare the future data that was predicted with the real data. Exemple, the Load of the city: We will have the predicted load of the city called P_XX_Load with the future and the past predicted data and we will have another TAG called XX_Load with the real load only whit the realtime and the past data.

                                       

                                      We want that the future data and your past could be acessed fluently in the same TAG because this data are different of the realtime data, the P_XX_Load is different of XX_Load and we want preserve the historical of the P_XX_Load to compare whit the realtime data in another days in reports.

                                       

                                      The PI in the electrical system is very used and i think that many companies could be benefited wth this improvement.

                                       

                                      For us, the problem of future data in PI was a problem that we had troubles to solve.

                                       

                                      Thanks

                                       

                                      *This is a personal opinion

                                        • Re: Future data in PI
                                          Ahmad Fattahi

                                          Thanks very much for the input and use cases; those are indeed very interesting.

                                           

                                          I am thinking if Rhys' idea covers your requirement, and if not, what should be changed. It seems that if we are able to handle future data and create a separate tag for the predictions Rhys' story should cover you. Am I right?

                                            • Re: Future data in PI
                                              tboers

                                              We are using future data in the Oil and Gas industry also for predictions.  We have implemented this by creating tags with a one year offset. This allows us to store the data and compare it within the application. But we feel this is a sub-optimal way of handling future data. E.g. now it is not possible to show the user the predicted and measured data in one trend as the predicted data is offset.
                                              Indeed the mechanism described by Rhys would be preferred. Are there any plans to implement this in the near future?

                                                • Re: Future data in PI
                                                  Joe Devine

                                                  We currently use AF tables and attributes for future data.  One example here is where we first extract weather forecast information and then write the information to AF tables where we then do calculations in AF to predict output for Combined Cycle plants, Wind Turbines and Solar arrays based on performance information for the equipment.  We then store that data in AF until it becomes history and then we write it out to a PI tag on schedule.  Since we have two separate paths for the AF data where we store the timestamp as one attribute and the value as another attribute we have all of the data availble for writing out to PI but we can also display the data by combining the timestamp data with the value information.

                                                   

                                                  This is somewhat restrictive in that you have to provide the presentation of the data and it cannot be retrieved Ad Hoc when looking at the future data but once the data is history the storage and tracking of what was predicted is PI data so the research after the fact is straightforward.  This of course uses more PI tags which has value for this purpose.

                                                   

                                                  Here is a PI WebParts display for the forecast data that is made up of PI tags that are timestamp tags and PI tags that are values where they are stored at the exact same time so the retrieval is based on time in the OLEDB query that supports the PI Table webpart.

                                                   

                                                  4520.Forecast.gif

                                                   

                                                  Here is a sample AF table where weather data is written to use for the prediction.

                                                   

                                                  1781.ForecastAF1.gif

                                                   

                                                  This is one sample of using AF to calculate future heat rate for a combined cycle plant.  Note the timestamp tags and value tags that are used so that we can also leverage the extraction wtih pure PI tags.  However, since the resulting AF tables and AF attributes are also propulated this can be used to display future data as well in Process Book and the latest version of PI WebParts.

                                                   

                                                  0246.ForecastAF2.gif

                                                   

                                                  AF is perfect for future data if correctly used for current display but also to store that data until it can become history for later evaluation and comparison of actuals.

                                                   

                                                   

                                                    • Re: Future data in PI
                                                      Ahmad Fattahi

                                                      May I ask how do you generate the future predictions and what tools/platforms you typically use for that? Is it just predictions on totally external quantities (such as weather, stock prices, etc.) or are they derived fully or partially form data currently stored in PI System (such as future plant trajectory, future flow of the pump given its current status, etc.)?

                                                        • Re: Future data in PI
                                                          Joe Devine

                                                          Ahmad,

                                                           

                                                          The future predictions are based on either manufacturer information or PI data if there is enough available for various load points or as a result of some sort of performance test.  In the USA we extract data from the National Weather Service where we extract an xml file and then parse that out for the weather forecast.  We have an AF template that does that work for us by simply entering the needed URL, Table and PI tags.  Here is the template used to extract data from the National Weather Service.  For non USA sites we use other forecast websites or data feeds.

                                                           

                                                          5545.ForecastAF3.gif

                                                           

                                                          We use AF template and non AF Template based calculations depending on the parameters and the needed results.  The sample was for a combined cycle plant but we also provide forecasts for Wind Turbines and Solar Arrays using the same methods.

                                                           

                                                          The AF platform is ideal for future prediction and work.

                                                            • Re: Future data in PI
                                                              Ahmad Fattahi

                                                              That is very nice indeed thanks for the explanation Joe. On a related note I am wondering what you think of taking the prediction to the next level by producing the predictions given previous data in PI System. For example, imagine I have a device for which I have collected and archived multiple data points and tags over years inside PI archives. Now I would like to learn from the past behavioral patterns and be able to predict how the device will behave in the future. An example would be preemptive maintenance or predicting a certain load, etc.

                                                               

                                                              From your experience, how much potential do you see in making those predictive capacities more feasible for PI data? In other words, my focus here is mining the previously archived data inside PI archives to extract more knowledge and value.

                                                                • Re: Future data in PI
                                                                  Joe Devine

                                                                  Ahmad,

                                                                   

                                                                  Funny that you should mention that since we already have a skeleton of such a system running now.  Our core technology for this is the Event Frames since this is used to define operational levels of the equipment in event frames that can then be compared to not only real time but also in predictive metrics. We then query the Event Frames to determine patterns and then since it is all in AF we have direct access to PI Notifications shoudl we detect an anomoly.  The Event Frames can then be used to substitute for the manufacturer or "hand selected" PI data we use now.  This way we can calculate the forecast on the fly with Event Frames.  The Event Frames are critical to the process to reduce the hard core search and compare within the archive since you will already have the needed comparisons already in place.  What we are doing now is post creating Event Frames with the hard core search and compare routines to then create a library of Event Frames to pick from going forward.  Again all of the managment is done within AF.

                                                                    • Re: Future data in PI
                                                                      Ahmad Fattahi

                                                                      I really appreciate the great explanation Joe. It sheds lots of light on the underlying technology. There is one thing I am missing here:

                                                                       

                                                                      Joe Devine

                                                                      We then query the Event Frames to determine patterns

                                                                       

                                                                       

                                                                      When you determine patterns is it just a matter of comparison or you implement some more sophisticated pattern recognition algorithms?

                                                                • Re: Future data in PI
                                                                  tboers

                                                                  Thanks Ahmad and Joe for your valuable contributions. We use Matlab to run predictive predictive reservoir models of gas production. The outcome of the model is compared with real time measurements. Deviations will indicate operational issues. Therefore reservoir engineers continuously monitor the model to real time measurement deviation. It would help if monitoring would be possible out of the PI box. Looking at Joe's solution, writing information into an SQL table would help. AF could then indeed be used to view the information.  Although I wouldn't know how to show this information in a PI trend. You can combine SQL time series data with PI time series, but can this be done this for future data?

                                                                    • Re: Future data in PI
                                                                      Ahmad Fattahi

                                                                      If you look at the posts in this thread you will see that there is currently no direct support of handling future data; however, there has been a few suggestions and workarounds how you can do that.

                                                                        • Re: Future data in PI
                                                                          janio

                                                                          Hi,

                                                                           

                                                                          I hope that OsiSoft understand the importance of future data in PI and the gain of this feature to the PI system.

                                                                           

                                                                          Ahmad, you could tell to the OsiSoft Developers that we want the future data in PI System!

                                                                           

                                                                          Thank you

                                                                            • Re: Future data in PI

                                                                              Future data has to be part of the PI server itself.  There are numerous workarounds, some very nice workarounds like Joe posted, but it is very natural to look at historical & real time data in the PI server and then within the same context look forward in time.

                                                                               

                                                                              I'm sure OSIsoft are listening (they have thought about supporting future data for some time) and no doubt planning their response, speaking of which Jay is very quiet on this topic

                                                                               

                                                                              Wasn't one of the common items of feedback from the User Conference or Regional Seminar (can't remember which) that there is a need for future data in the PI server..?

                                                                                • Re: Future data in PI
                                                                                  MichaelvdV@Atos

                                                                                  First of all, I want to thank everyone for their valuable contributions to this topic. OSIsoft is really interested in hearing all the use cases. There has been a lot of talk about future data in several topics, and this is generating really valuable information.

                                                                                   

                                                                                  A quote from Jay Lakumb (Product Manager PI Server) from [DEAD LINK] this topic:

                                                                                   

                                                                                  Note, this is a continuation of this discussion thread

                                                                                   

                                                                                  Ok, fine, I can take a hint. I'll weigh in here on the future of future data, and provide a little more transparency on what is happening with "Verrazano" (code word for our next release).

                                                                                   

                                                                                  It turns out we have a number of competing forces for Verrazano (not surprisingly). On the one hand, there is urgent need to scale up to support higher tag counts for some near-term AMI opportunities. On the other hand, we have a commitment to support future data for our primary markets, especially T&D industry.

                                                                                   

                                                                                  Our strategy is to embark on both efforts simultaneously. We plan to continue these on parallel paths for now. Since the PI Server team is switching to Agile development, we have more opportunities to "check in" and determine whether we potentially have a shippable version. This is one of the reasons why there is no release target for Verrazano (also the fact that we are early in development phase). Rest assured though that we are currently working on both of these initiatives...

                                                                                   

                                                                                  This information is about 1 year old. We are contacting Jay to ask him if he can share more of his thoughts. Stay tuned!

                                                                                    • Re: Future data in PI

                                                                                      Michael, That has prompted me to something else on how to handle future data, by using point partitions to separate future data tags from their "regular" data tags that hold the historical data.  Essentially you can "bolt on" future data for any data tag this way.

                                                                                        • Re: Future data in PI
                                                                                          pierre.baelen

                                                                                          We had same issue in large scale heating network.

                                                                                           

                                                                                          They gather current temperatures, previous years temperatures and then some thermo/math gurus wrote a C executable that feed 3 days prevision for many data.

                                                                                           

                                                                                          Team worker were looking at each : "How the .... are we going to do this the clean way ?"

                                                                                           

                                                                                          I remember [-11046] Target Date in Future. That was 3 years ago :O

                                                                                           

                                                                                          To get a quick and dirt workaround, we did it like Tjidde explained : shifting predicated value with a predefined offset into laboratory tag.

                                                                                            • Re: Future data in PI

                                                                                              Any hints as to what the "Apply future data in the PI System" presentation at vCampus Live! will cover?  Is it conceptional, plans for PI Server 2012, other?

                                                                                                • Re: Future data in PI
                                                                                                  Ahmad Fattahi

                                                                                                  Did you have a chance to click on the presentation? Here is the abstract. We need to ask Jay for more information if you would like to know more details:

                                                                                                   

                                                                                                  Applying Future Data in the PI System

                                                                                                   

                                                                                                  Presenter: Jay Lakumb

                                                                                                   

                                                                                                  Type: Hands-on Lab

                                                                                                   

                                                                                                  Length: 1:45

                                                                                                   

                                                                                                  Future data is coming in the PI Server, but as many of you know it is not available yet. There are several alternatives customers are successfully using today to visualize, analyze and access future data in the PI System. In this lab, we will show you how to implement these alternatives and then review the pros and cons of each approach. We will also discuss a migration strategy to help you move forward and discuss the plans for future data support in the PI Server.