PI Batch to EF Migration

Blog Post created by ernstamort on Apr 11, 2016

Most PI users have already transitioned to the AF system or at least build some demo system that uses AF. Some companies use their own enterprise structures or use already existing templates that mirror ISA-95 compliant equipment models(PI and MES: Equipment Model).


For the Event Frames (EF) many are still holding off to switch from batch to EF and this for good reasons. Migrating historical batches is important in order to preserve data for any type of future modeling. Historical batch data contain a rich library of past processing conditions that can be used to extract univariate (e.g. Golden Batch) or multivariate (PCA or PLS) process models. But in order to process historical data the system design needs to be completed both on the AF side and the EF side.


There is also valid point to me made about keeping both PI Batch and EF system running in parallel for a while. Especially for risk adverse industries there is a need to fully qualify the EF systems and its client applications before making the change. In addition, there might be some custom applications that write directly to PI Batch database that cannot be migrated to EF.


To understand the migration process, let’s have a look at the different ways batches have been created in the legacy batch system:


PIBaGen:                This is a tag based batch generator that can be configured in the PI-SMT

Batch Interfaces:     EMDVB and others process text or database information and create batches

PI-SDK:                   The software development kit is being used by 3rd party vendors to create batch context


OSIsoft has upgraded most its interfaces to write to the EF database. History recovery is also available to reprocess old data and create Event Frames.


An alternative way is to programmatically migrate historical batches and perform a real time synchronization. The goal shouldn’t be to perform a 1:1 migration, but rather a migration that fully utilizes the new system while preserving the existing structure.


The first challenge is that the structures of the PI Module database and AF database almost certainly don’t match. Due to its performance limitation the PI Module database was often only used to create very limited structures to allow base operations such as batch creation, develop element relative displays, RtReports and others. To the contrary the AF database is being used to create full enterprise models with context specific analysis and therefore is far more complex. The following shows a made up example:


Module Database:

AF Database:

To synchronize unit batches requires to link the PI Unit in the Module database to an AF element. This is accomplished by creating the following attribute on the PI Unit Template.

The unit reference on the unit batch can then be converted into an element reference on an event frame.

Module Database Guid: Unique Id of the PI Unit in the Module Database


Also the EF templates require some additional attributes for the sync process. The following shows the relationship between ISA-88, PI Batch and EF (see alsoPI and MES: Batch Model):



ISA 88

PI Batch

EF Batch



PI Batch



Unit Procedure

PI Unit Batch

Unit Procedure



PI Sub1 Batch




PI Sub2 Batch




PI Sub3 Batch

Phase State



PI Sub4 Batch

Phase Step


On each EF Batch template the following attribute should be added to link the historical batches to the event frames. In this example the attributes are added to the base template:


Id:                               The batch, unit batch, or sub batch id

IsHistorical:                This flag is set if the template is created from a historical batch

PI Batch Guid:           The unique identifier for batch, unit batch or any sub batch level

PI Batch Sync State: An enumeration to indicate if the sync process was successful

PI Batch Sync Time:  The time stamp of the last modification


With both the AF and EF templates in place the historical and real-time synchronization is now straightforward. The PI-SDK can be used to poll the batches, which will be mapped to an internal data model. Once processed event frames are either created or modified based on differences in the queues.


Batch Sync.png

The following shows the result of the migration of simulated data:


PI Batch Database:

EF Database: