<See my update below in the comment section >
Happy Holidays! I hope you're able to take some time off during this holiday season to spend with your friends and families and of course getting caught up with PI Square . As 2014 comes to a close, I want to spend a few minutes giving you my current thoughts on future data and the PI System. As some of you may know, I've talked about PI Server 2015 and future data a few times in the past including announcing the availability of CTP1, Beta 1, and Beta 2 all in 2014. As we march toward the release of PI Server 2015, more and more exciting things are happening around future data. As I have indicated numerous times in the past, I am a proponent of ensuring the entire PI System with all its components working as a whole to support future data rather than just the individual PI System components. By individual components I mean, for example, PI Data Archive, Interfaces, PI AF, etc. In short, native future data support should span across the entire PI System.
Having said that, I have a quiz for you. What were you doing at 10:00 GMT-8 on 16 December 2014? I hope your answer is that you were listening to Denis Vacher and me present a webinar titled: PI Server 2015 and Future Data. You may be asking why I'm bringing this up. Well, I want to highlight this webinar because not only did Denis give an impressive live demonstration of how the PI Data Archive supports future data, he also gave a sneak peak of PI ProcessBook 2015 and PI Coresight 2015 supporting future data by trending a predicted versus current temperature using data available from an external website. If you missed the webinar, don't despair, a recording is available here. As you can imagine, there were numerous questions towards the end of the webinar and many of these were excellent questions. We took the time to sort through these questions and have selected many of them to post here as a FAQ in case there is general interest within the PI Square community. I highly recommend that you spend the time to view the webinar recording as it would provide a lot more context to some of the FAQ entries below. Many people helped putting this FAQ together and I want to thank them all - you know who you are .
One last item before I finish off this blog. If you currently have a need for future data support in the PI System, plan on utilizing this new feature, or have some applications in mind, I (we) want to hear from you. Penny, who is involved with the future data initiative, just kicked off a Discussion in the All Things PI space. I (we) would appreciate any and all inputs or thoughts that you have.
If you have not already done so, considering trying out PI Server 2015 Beta 2.
Have a safe and joyous holiday season! We'll chat again in 2015.
Future Data Frequently Asked Questions
Last updated December 30, 2014
New with future data: The capability to natively capture, store, analyze, and visualize data with future timestamps, such as forecasts and predictions.
PI Data Archive 2015, a component of PI Server 2015, introduces the ability to store forecast and predictive data beyond the strict 10-minute limit of previous versions. These “future” PI points (sometimes called “tags”) can be created using tools like PI Builder or Tag Configurator (both Microsoft Excel add-ins), or PI System Explorer and PI System Management Tools (PI SMT). All data sent to future PI points gets stored in separate archive files that are optimized for non-continuous, non-sequential data, unlike real-time measurements stored in traditional (historical) PI points. For OSIsoft and its customers, 2015 marks the beginning of a PI System-wide initiative to expose and leverage future data.
Future Data Fundamentals
Q: Can you have a PI point that has both predictive future and actual past data?
A future PI point can store data that has time stamps both in the future and in the past, but we strongly recommend against mixing predicted and measured data in the same PI point. If you want to store forecast or predictive data, choose a future PI point; if you want to store actual or physically measured data, choose a historical PI point. Measured data has different characteristics compared to data that is not measured, like forecast or predicted data. Measured data reflects a single reality based on what occurred, and it is typically generated continuously by machines at a certain frequency. In contrast, future data, such as forecasts and predictions, reflect an estimate or projection of what might happen. Future data typically updates much less frequently and is not necessarily updated in sequential order. PI Data Archive 2015 has been optimized to handle each type of data according to their characteristics and use cases. Again, both types of PI points can store data in the past. Historical PI points are ideal for storing measurements, and future PI points are ideal for storing forecasts or other predictive data.
Q: When time progresses, does the forecast data move from the future archives to the historical archives?
No, data from future PI points is stored in separate archive files that are never mixed or interchanged with data in historical archive files. This provides the ability to manage the lifetime and overall storage aspects of future PI points independently from real-time measurements that are coming from sensors, devices, and other physical activities. Just like with measurement data, there is significant value in keeping future data for as long as needed, which is why PI Data Archive 2015 does not discard future (e.g., forecast) data that slides into the past as time goes on.
Q: Can I change a historical PI point to a future PI point, and if yes, how is the original history managed?
It is not possible to automatically convert existing historical PI points to future points with PI Data Archive 2015. With PI Data Archive 2015, we’re introducing a new PI point attribute: future. The choice for the value of the "future" attribute (true/false) of the PI point must be made at point creation time and cannot be modified afterwards. At the most fundamental level, historical and future data streams are very different. A historical PI point is designed to store real-time data that is collected sequentially and continuously around current time. In other words, historical PI points should contain actual measurements. On the other hand, future PI points are designed to store all other types of data. Forecasts and predictions are prime examples of data that are not measured. Choosing between the two types of points should be relatively simple and will not need to change throughout the life of a point. If necessary, data from an existing point can be extracted and copied to a new point of a different type, before the old PI point is discarded or renamed. Outside of failure conditions, remember that the PI Data Archive never discards data on its own.
Q: Will a revised forecast overwrite the data with same timestamps?
It can. In many cases, the most recent or most accurate revision of a forecast should overwrite the previous one. However, just like revisions to data for historical PI points, you can decide what to do in these situations. For example, the PI UFL (Universal File Loader) and PI RDBMS (Relational Data Management System) interfaces allow you to select how to handle values with the same timestamps. We are aware that in some cases you might need to store two or more revisions of the same forecast. For such cases, we strongly recommend using multiple future PI points – one for each revision – as opposed to trying to store multiple values with the same timestamp in a single future PI point.
Q: How do you ensure overwrites occur with future data changes?
There are a few different options to ensure data overwrites for future PI points. The most common approach is just to use fixed, clock-aligned time stamps (e.g., at the top of the hour). Another approach is to delete existing data before writing new values. Lastly, PI Data Archive provides a special edit mode called "ReplaceX" (replace exclusive) whereby an edit will only succeed if a value already exists at a specific timestamp. More generally, there are multiple edit modes that can be selected, including “Append” (add event regardless of existing events), “Replace” (add event and replace if there is one at the same timestamp), and “ReplaceX.” Both PI UFL and PI RDBMS interfaces allow users to specify the edit mode.
Q: How does PI Data Archive 2015 handle in-order vs. out-of-order events now that it can handle future data?
PI Data Archive 2012 introduced several internal mechanisms (e.g., asynchronous record initialization and relocation) to improve the processing of out-of-order events, and PI Data Archive 2015 builds on these enhancements for future PI points. One of our assumptions, based on internal testing and many customer studies, is that future PI points are likely to receive only a small fraction of the data going to historical PI points. With modern storage technology (e.g., SSDs or striped HDDs), we don't expect performance problems for the vast majority of deployments. That being said, we highly encourage you to conduct your own evaluation, especially if you anticipate storing large volumes of data into (or frequently edit data from) future PI points. Please share your results with us as soon as possible.
Q: For daylight-saving time, are future times referenced to the current time zone, or the future time zone / daylight saving time?
PI Data Archive stores time stamps in coordinated universal time (UTC) format. Storing time stamps in UTC format allows PI Data Archive to handle time according to a single reference time. As such, actual measurement data that originates in another time zone is not considered future data. Conversions from local time to UTC time uses the time zone and daylight-saving time (DST) settings of the computer being used to write the data, e.g. the interface node. After data has been stored, any subsequent changes to time zone or DST settings will only affect new write operations. Any conversion from UTC time back to local time uses the time zone and DST settings of the computer being used to read the data. For example, on the computer where the client application (e.g., PI ProcessBook, PI DataLink, etc.) is running. All future time stamps reflect the DST settings as defined in the local time zone for the client computer. For each time zone, current DST settings are considered valid until the local authorities change them.
Q: How will PI Update Manager work to update new events to client applications, as future data will not always be sequentially updated?
You are correct to highlight the fact that future data may not come sequentially and may also be updated much less frequently than measurement data. This created a challenge for the real-time update mechanisms of the PI System.
For instance a daily forecast may only come once a day. If a user is looking at a display over a 2-hour time range, this forecast will "appear" stale for up to 22 hours because no new values are coming. To correctly handle this issue, and ensure that client applications don't have to constantly poll data from future PI points, the AF SDK version 2.7 (part of PI AF Client 2015) provides a new time-based Data Pipe. This new type of update subscription works in combination with enhancements in the PI Update Manager so that applications can request new values that enter a specific time horizon. In the above example, the application would subscribe with a 2-hour horizon so that the Data Pipe will feed it with any values that enters (or is modified within) that 2-hour window, and automatically present them to the user. Note that the default time horizon is "0" so that single values (e.g., a value symbol) always shows future data that is valid at current time.
Q: Can PI Data Archive 2015 store “backcasts” and forecasts? Is it possible to backfill data into future PI points/archives from before the current time?
Yes, it is possible to write values anywhere in the UTC time range (1970-2038) into future PI points, and archive files will be created automatically.
Q: I have seen a pre-release of PI Coresight 2015. What happens when the “now” line reaches the end of the trend display?
When the Now line in PI Coresight 2015 reaches the right-hand side of a trend, it will disappear and the display will no longer be updating. PI Coresight will not change the start and end times of your display unless you make them relative to current time. To define a relative time range, please use one of the special time characters allowed by PI timestamps. For instance “*” for current time, “t” for midnight of the current day, or “y” for midnight of the prior day. Note that when using both start and end times that are relative to “*” (with a time range that includes current time, such as “*-2h” and “*+2h”) the Now line will remain visible and stay at a constant position while traces on the trend will slide to the left.
Q: Can I use this for forecasting MW demand based on historical data? Weather is just one variable. Many others come into play (e.g., holidays, school, seasons, etc.).
Yes, you can use future PI points to store data for numerous variables to support a wide variety of business applications across industries, including forecasting MW demand. Weather data was just a simple, relatable example for demonstration purposes in the webinar. As long as your data fits in one of the supported data types of PI Data Archive – integers, floats, digitals, strings, etc. – and the time stamp is within the UTC range (January 1970 to January 2038), there are no limits on the nature or sources or applications of the data. Data generation can occur either outside of the PI System using a specialized third-party tool or from within the PI System with, for instance, the help of Asset Analytics in PI Server 2015. Regardless, you can store the data in PI Data Archive 2015, making it readily available for additional analysis, reporting, sharing, and visualization.
Q: Can you make calculations between historical PI points and future PI points? For example, can I calculate the difference between actual value and predicted value of a temperature and send a PI notification if the difference is above normal?
Yes, you absolutely can create calculations that involve both historical and future PI points. We fully expect comparing an actual measured value (e.g., temperature) from a historical PI point against its predicted value from a future PI point to be an extremely common scenario. In PI Server 2015 Beta 2, we could not yet demonstrate such a calculation with Asset Analytics but this feature will be included with the final release. More specifically, the update to Asset Analytics will provide the capability of triggering calculations off of future PI points, performing analyses with a mixture of historical and future PI points, and storing calculation results back to future PI points. As for triggering a PI Notification from the calculated result, the released versions of PI Notifications do not have the ability of directly triggering off of future PI points. However, you can still generate a PI Notification when an anomalous future condition is detected by using an intermediary historical PI point. We are hard at work on the next version of PI Notifications that, among other enhancements, will provide full support for future PI points and future data. Once development is a bit further along, we will update the PI Notifications Roadmap with our expected release timeframe. Please stay tuned.
Q: I would like to ask about the calculations used to generate future data. Can I have my own way to calculate it?
Yes, you have complete autonomy and flexibility to create custom calculations to generate future data for whatever special purpose your applications or businesses require. Once the future data has been generated, it can be stored in PI Data Archive 2015. Asset Analytics, first released with PI AF in PI Server 2014, is being enhanced for PI Server 2015 so that you can easily create custom analyses to generate future data.
Q: Will there be an upgrade to the PI SDK for future data so that PI ACE under VB.NET could take advantage of this feature?
We are not planning on significantly enhancing the PI SDK or PI ACE to support future data. In its released form, the PI SDK can be used to both read and write values in the future, but there are limitations with future PI points such as the lack of Event Pipe support. These limitations restrict the out-of-box support for future data in PI SDK-based applications like PI ACE. For example, you cannot use future PI points to naturally trigger PI ACE calculations directly, and the PI ACE cache may not stay in sync and cannot be trusted to be accurate for future PI points. PI ACE does offer clock-based calculation triggers, and the PI ACE cache can be disabled globally, but these limitations may be too severe. Of course, from within the .NET code for your PI ACE calculation, it is possible to reference the PI AF SDK just like any other .NET-based library. PI AF Client 2015 includes PI AF SDK 2.7 and fully supports future data. PI AF Client 2015 Beta is included with PI Server 2015 Beta 2 if you would like to explore what is possible from within PI ACE (or any other application). However, before embarking down this path, we recommend spending at least some time evaluating whether any new PI ACE calculations involving future data could instead be created with PI Asset Framework (PI AF) and Asset Analytics, which was first released with PI Server 2014. An update to Asset Analytics will be included with PI AF 2015 and will provide the capability of triggering calculations off of future PI points, performing analyses with any mixture of historical and future PI points, and storing calculation results back to future points. Asset Analytics also has first-class support for PI Event Frames, along with roll-up calculations and the full richness of PI AF. Depending on the complexity of your existing PI ACE calculations, you may discover that you can migrate many of them to Asset Analytics.
Q: Is the “future” attribute part of the “classic” point class in PI Data Archive 2015?
The “future” attribute is actually a built-in attribute that is part of all point classes, including “base” and “classic”.
NOTE: PI Data Archive 2015 restricts creating future PI points in all of the following point classes: Alarm, SQC Alarm, Totalizer, and any point class with COM Connector attributes (ctr_progid, ctr_lmap, ctr_strmap). Thus, for any point in these point classes, the future attribute will always have a value of 0 (false).
Q: I have a script that automatically creates a new archive at midnight every month, but I heard that future PI point archives are created as far in advance as required. How can this be managed?
One of the design goals for PI Data Archive 2015 was to minimize the management burden of future archives. As an administrator, we expect your tasks to be quite minimal and straightforward, namely choose where to store future archives and ensure the PI Data Archive has enough disk space. Optionally, you may also customize the archive file names to match a specific convention. Thus, our hope is that any script like you’ve described can be eliminated entirely for future archives and that no special management will be required. Archive files that are automatically created for future PI points will be one calendar month in duration, from midnight on the 1st day of the month to midnight on the 1st day of the subsequent month. Their initial size is 1 MB, and they grow dynamically as needed. So, if a new value comes in on the 7th day of the month at 08:00 AM, if that month’s archive file does not exist, PI Data Archive creates it automatically. There is no way in PI Data Archive 2015 to configure this duration or initial size for future PI points, and the algorithm cannot be disabled. However, archive files for future PI points can still be manually created in special situations. While we
believe these situations to be extremely rare, you may create future archives with whatever sizes and durations you need. The automatic algorithm will fill in any “gaps” before and/or after all existing future archives, and re-align new ones to calendar-month duration. For example, if you manually create a future archive from 11:00 AM on the 7th day to 09:00 AM on the 14th day, PI Data Archive 2015 will automatically create an archive file from 12:00 AM on the 1st day to 11:00 AM on the 7th day and a second archive file from 09:00 AM on the 14th day to 12:00 AM on the 1st day of the subsequent month. Keep in mind that PI Data Archive will only create these two surrounding archives if data is actually received in both of those time ranges. That is, the algorithm is always active but only operating on-demand. Unlike shifts for historical archives, explicit archive shifts are not relevant for future PI points because the end time of future archive files is always set when the archive files are created.
Q: Does this work well in a PI Server collective for high availability?
Yes, future data is expected to work perfectly with PI Data Archive collectives. As of PI Server 2015 Beta 2, full support for future data with PI Data Archive collectives has been implemented.
Q: Does future data works with PI Event Frames (EF) Generator? Is the evaluation of the EF is done "at time?”
Unfortunately, the current version of the PI Event Frame Generator (EFGen) will not trigger off of future PI points. We are evaluating whether EFGen needs to be enhanced with this capability given that Asset Analytics, first released with PI Server 2014, is our recommended method to generate event frames and is being enhanced to support future data. Asset Analytics in PI Server 2015 will support triggering an event frame as the future event becomes current.
Q: What version of Windows is required to use PI Server 2015?
PI Server 2015 requires a 64-bit machine and Windows Server 2008 R2 SP1 or later. These requirements are a first for the PI Data Archive. PI Data Archive 2012 is the last release to support 32-bit machines and Windows Server 2003 and Windows Server 2008. We recommend using PI Server 2015 with Windows Server 2012 R2 in a Server Core configuration.
NOTE: We continue to recommend the Server Core installation option of Windows Server to provide reduced maintenance along with greater stability, security, and performance. If you choose Server Core, PI Server 2015 requires Windows Server 2012 or later and Microsoft SQL Server 2012 or later.
Q: Will PI Audit track edits to the forecast data?
Yes, the same auditing capabilities are available for all PI points.
Q: Is PI Cloud Connect (PICC) compatible with the PI Server 2015 Beta?
PI Cloud Connect has undergone only limited testing with PI Server 2015 Beta so far, but we anticipate full compatibility before release.
Q: Is the current PI UFL interface capable of inserting future data into PI Server 2015?
Yes, the current version (126.96.36.199) of the PI UFL interface has been successfully tested, and we anticipate that it will be one of the primary tools to insert future data from external sources.
Q: Is there an evaluation period built into PI Server 2015 Beta?
PI Server 2015 Beta comes with a license file for PI Data Archive that is valid through June 30, 2015. We highly encourage you to start your evaluation as soon as possible so that your feedback can be considered before the initial release and so that we can provide you with the support you need. Contact us at BetaPIServer@osisoft.com.
NOTE: We are looking for customers and/or partners that might be interested in sharing their experience at our upcoming Users Conference in San Francisco (starting on April 28, 2015). Please contact me or drop us a note to BetaPIServer@osisoft.com.