7 Replies Latest reply on Mar 2, 2017 5:44 PM by jagdish.konathala

    Storing weather forecast data in PI.


      I have a need to be able to store and retrieve hourly weather forecast data for a week out.  I can think of a few ways to do this but wanted to hit up the community to see if there are more, which I'm sure there are:


      A. 7 tags - each representing a predicted day


      As I need a week ahead of data, store the values using a 1 week time offset back in time, but store each day in a separate tag.


      Tags may be:


      Day0Forcast.OutsideTemp  (for today's forecast)


      Day1Forcast.OutsideTemp (tomorrow's forecast)


      Day6Forcast.OutsideTemp (one week out)




      If today is Mar-4 I'd then store todays Day0 forecast using timestamps -7d or Feb-28.  Tomorrow when I store the values, then they would be using dates of Feb-29.  This allows me to see what the weather prediction was for any day by just adding 7 days to the timestamp.


      1. Advantage: data isn't overwritten, it is consecutive.


      2. Disadvantage: a single week's predictions cannot be trended using a single point and it would take some funny business to select values from each tag.


      3. Disadvantage: the timestamps must be re-interpreted the same way so as stored, they are confusing.  I don't think even AF can apply an implied time offset to the timestamp






      B: 7 tags, each representing a prediction for the week.


      Here depending on what day of the week today is, I can store the values again using a 7 day time offset but rotate what tag the values get stored to.  If today is Tuesday Mar-4, then I use the Tuesday predictions


      TuesdayForcast.OutsideTemp  (for today's forecast)


      WednesdayForcast.OutsideTemp (tomorrow's forecast)


      MondayForcast.OutsideTemp (one week out)


      1. Advantage: data isn't overwritten, it is consecutive.


      2. Disadvantage: the timestamps must be re-interpreted the same way so as stored, they are confusing.  I don't think even AF can apply an implied time offset to the timestamp, it can do a time offset fetch to get the data at that time in the archive.


      Even with storing data using future time stamps, this prediction for each day use case will still exist.




      OK, Discuss.





        • Re: Storing weather forecast data in PI.

          Thanks for submitting this question Jeff.  I'm sure you know, just like everyone else on vCampus, that we're working on delivering native storage of future data in the next PI Data Archive (project codename DeLorean), which is due for release at the end of 2014.


          From what you're describing, I understand you want to store and independently query the current and all past revisions of your 7-day hourly forecast, which is updated daily.  Please correct me if I'm wrong.


          As you mention, native storage doesn't necessary help with managing multiple versions of the same forecast (which btw, is not a problem specific to future data).  You are correct the only practical solution --and our recommendation-- for this scenario is using more than one tags.  The key questions are 1) how many tags; and 2) how to manage and distribute the data among them?


          Choosing the same tag count as the number of concurrent revisions (7 in your case) is the most common strategy.  Just like your B case, tags should receive the entire forecast in a round-robin fashion, and you simply have to remember which tag stores the most current forecast (i.e. day of the week in your example).  This is clearly a much cleaner strategy than your case A, which as you say, is not compatible with standard PI tools and data access libraries.


          A carefully crafted AF template/element would also be our recommendation to hide this complexity from end users.  The AF attributes (PI Point data references) would apply the positive time offset in order to return the correct timestamp, that is until DeLorean brings us native storage.  Then you also need some logic to easily get to the current and recent history of forecasts.  Ideally, end users could see something like this as available attributes:










          A secondary question is whether it's necessary to keep the history of all revisions, or if storing the last (best) revision is sufficient, which is generally the case.  This greatly simplifies requirements around fetching past forecasts, in post-mortem analysis for instance.


          For completeness, I also want to mention some customers select strategies with fewer tags than the number of concurrent forecast revisions.  In these cases, a "primary" tag receives the most current data, which overwrites any previous values, and one or more secondary tags receive the trail of changes as duplicate values (i.e. multiple values at the same time).  The secondary tags cannot be easily trended or queried, but this data is generally considered much less valuable than the most current/accurate forecast anyhow.


          PS: For the record, I would highly discourage event annotations as a way to store measurement revisions.  They have the same navigation problems (you can't trend or search annotations), and are also plagued with performance limitations.  Annotations were invented to store user-generated comments or occasional metadata -- i.e. they are not for continuous data collection.

            • Re: Storing weather forecast data in PI.
              Roger Palmen

              Hi Jeff,


              Have you considered using AF and the (external) table lookup feature? Yes, you can trend future data in ProcessBook, CoreSight, WebParts when taken from an AF attribute.


              I've successfully used this for full-year, day resolution forecasts on approx. 55 items (19K rows).


              Agree with Denis that if you do not need the history of the future :-), thus no versioning, you could simply use AF and the Table lookup to store and retrieve your forecasts. In this case the dataset is quite small and should work. The rule-of-thumb used here is 10K rows. Anything above that, you should start thinking about architecture and do your math and/or testing on performance.


              If you do need to keep the history or versioning, i'd suggest to wait for AF2.6, where we should have server-side filtering on external data sources, which can support larger datasets. But this all depends on your architecture and usage if this really is needed.


              Hope this helps!

              • Re: Storing weather forecast data in PI.

                Merci Bien Denis for your great analysis.  With regards to the AF attributes, I haven't found a way to do this yet using the PI Point DR- to have a value stored in PI at time A and have it return with timestamp B.  I would agree however that having this data contract with client applications which separate storage from consumption is a good thing.