jwu

Guidance on writing efficient OMF Applications for PI Web API

Blog Post created by jwu Employee on Sep 18, 2019

Introduction

This guide shows OSIsoft Message Format (OMF) Client developers efficient and performant ways to ingress dynamic data (events) to the PI System through the PI Web API OMF Endpoint. OMF Client developers should be able to achieve high PI event ingress rates by going through these best practices for formatting OMF dynamic data messages.

 

 

Prerequisites

Before sending dynamic data messages, a client needs to create and send dynamic type and container messages.

This guide assumes the client knows how to configure authorization, HTTP headers, and how to send messages to the OMF endpoint in PI Web API. See the OMF Specification for required headers and any other information.

 

Dynamic Type Create

Required header: { messagetype: "Type" }

Start by sending a type message through the body of a POST request:

[
     {
          "id": "DeviceStatus",
          "classification": "dynamic",
          "properties": {
               "DeviceStatus": {
                    "type": "string"
               },
               "Speed": {
                    "type": "number"
               },
               "Time": {
                    "format": "date-time",
                    "isindex": true,
                    "type": "string"
               }
          },
          "type": "object"
     }
]

Expected Response: 201 Created

This creates a dynamic type that we can use as a template for creating Containers. A Container created with this Type will have 2 properties or PI Points: DeviceStatus and Speed.

 

Container Create

Required header: { messagetype: "Container" }

Next send a container message through the body of a POST request:

 [
     {
          "id": "Turbine 1 Device Status",
          "typeid": "DeviceStatus"
     },
     {
          "id": "Turbine 2 Device Status",
          "typeid": "DeviceStatus"
     }
]

Expected Response: 201 Created

This creates 2 containers and each container is represented by 2 unique PI Points to send timestamped data events to.

Now that we have setup our Containers, we can look at the different formats of OMF Dynamic Data to send data to those PI Points.

 

 

Dynamic Data Message Formats

All of these messages will send events with a timestamp and one or more values.

There are 3 main factors that affect performance: the number of properties sent at a time, the number of values sent for each property, and the number of users sending requests.

The expected responses for the following dynamic data messages is 202 Accepted.

Required header: { messagetype: "Data" }

 

Single Value, 2 Properties

This example is recommended for the dynamic type listed above, since there are two properties: Speed and DeviceStatus.

Performance is improved when adding more properties to a container because internally, the cache finds the PI Points that need to be updated by looking up their containers.

 [
     {
          "containerid": "Turbine 1 Device Status",
          "values":
          [
               {
                    "DeviceStatus": "Good",
                    "Speed": 5,
                    "Time": "2019-07-01T15:44:56Z"
               }
          ]
     }
]

 

Single Value, Single Property

This example is not recommended for the dynamic type listed above because there will be a default value entered for the missing property: Speed. To use this example correctly, you need to create a new dynamic type with only a single property and a corresponding Container that uses that type. This method is slower than creating types with multiple properties because of the caching mechanism.

 [
     {
          "containerid": "Turbine 1 Device Status",
          "values":
          [
               {
                    "DeviceStatus": "Good",
                    "Time": "2019-07-01T15:44:56Z"
               }
          ]
     }
]


Multiple Values, 2 Properties

This example is recommended for the dynamic type listed above with the two properties: Speed and DeviceStatus. Creating a single request with multiple values is functionally the same as creating separate requests with those same values individually. Because the values are processed together within the same request, it's much more efficient to batch multiple values into a single request.

 [
     {
          "containerid": "Turbine 1 Device Status",
          "values":
          [
               {
                    "DeviceStatus": "Good",
                    "Speed": 5,
                    "Time": "2019-07-01T15:44:56Z"
               },
               {
                    "DeviceStatus": "Bad",
                    "Speed": 0,
                    "Time": "2019-07-01T15:45:56Z"
               }
          ]
     }
]


Multiple Containers, Single Request

This example is recommended for the dynamic type listed above with the two properties: Speed and DeviceStatus. Creating a single request with multiple values is functionally the same as creating separate requests with those same values individually. This will increase performance by allowing PI Web API to process all data messages within a single request, which skips the latency and costs of an additional request entering the pipeline.

 [
     {
          "containerid": "Turbine 1 Device Status",
          "values":
          [
               {
                    "DeviceStatus": "Good",
                    "Speed": 5,
                    "Time": "2019-07-01T15:44:56Z"
               }
          ]
     },
     {
          "containerid": "Turbine 2 Device Status",
          "values":
          [
               {
                    "DeviceStatus": "Good",
                    "Speed": 4,
                    "Time": "2019-07-01T15:44:56Z"
               }
          ]
     }
]

 

 

Scaling with multiple users

When increasing the number of users, performance tends to increase as well because of PI Web API's built-in multi-threading capabilities. It is able to handle multiple concurrent requests from different users. Spreading out requests between multiple users can increase performance but also can increase memory usage. Because OMF for PI Web API utilizes a per-user cache for PI Points, duplicate points between users will duplicate the points in the cache as well. We recommend that separate clients send data requests to PI Points not used by other clients to keep memory usage lower.

 

 

The Good, the Bad, and the Ugly

There are different ways of formatting dynamic data messages to get the same data into the PI System. Specific practices can be utilized when writing a client to send data events that will increase performance, just as specific practices that will harm performance should be avoided.

 

The Good

The best method to improve performance is to increase the number of values sent per request. This allows messages to be processed simultaneously and batched together. Sending requests separately incurs the cost of processing the headers and going through the message processing pipeline multiple times instead of these operations happening only once when the requests are combined.

 

Another method to improve performance is to add more PI Points/Properties to containers because of the way the caching mechanism works in PI Web API OMF. The cache performs a lookup per container so it can find PI Points faster because they'll be batched together.

 

Increasing the user count improves performance by allowing requests to run concurrently. However, if too many users are added, PI Web API performance will bottleneck with the server CPU core count. Ensure clients send data to unique PI Points to prevent duplicate memory usage.


It's best to configure a separate machine(AF/DA) for OMF data to be stored in so that resources are not taken from PI Web API. After installing PI Web API with OMF, there is an option when running admin utility to configure OMF to point to different AF/DA. In general its best to keep other resource intensive programs/services on other machines.

 

Summary:

  • Increase the number of values sent per request
  • Add more PI Points/Properties to containers
  • Increasing user count
  • Configure other programs/services on non-PI Web API machines

 

The Bad

 If the PI Web API server is not allocated enough RAM, when the cache is hydrated(PI Points being stored in memory), it will struggle to allocate resources which can lead to a bottleneck.

 

Not provisioning enough CPU cores for the PI Web API server will lead to bottle necking during message processing because there will not be enough cores to process messages concurrently. However, this should not be an issue if only a single client is sending messages to OMF.

 

Sending data to large amounts of containers in PI Web API OMF will initially lead to slowdowns especially during startup because the cache will need to be filled with the PI Points given. This is not really avoidable as the cache needs to be hydrated during each restart, but performance should improve during subsequent requests.

 

Performance will suffer if you do not follow the recommendations listed in The Good.

 

Summary:

  • Increase the amount of RAM on the PI Web API server
  • Provision more CPU cores on the PI Web API server
  • Decrease the numbers of containers if possible

 

The Ugly

Absolutely do not send static data in the same message as dynamic data. This will severely decrease performance as PI Web API will run in exclusive mode where all messages are processed one at a time. Sending only dynamic data messages will allow PI Web API to run in concurrent mode, where it can process multiple requests at once. If you need to send static data messages make sure to send them either before or after dynamic data messages.

 

 

Realistic Performance Results

There are results and benchmarks to be expected when using settings listed below. Performance may vary for a variety of factors like latency, HDD speeds, and CPU speeds. Under these specifications and these scenarios, a user can expect performance of at least 100k Events per second.


Recommended Specifications

These are just recommended, not required. Performance may vary depending on setup. Refer to the OMF Companion Document for version compatibility.

- Windows Server 2016 or Higher

- Separate machines for PI Web API and AF/DA

- PI Web API 2019 or Later

  • 16GB RAM
  • 100GB HDD
  • 8 cores
  • Possibly PI Buffer Subsystem if the use would like to buffer events before going to Data Archive

- PI Server 2018 SP3 or Later (AF/DA)

  • 8GB RAM
  • 100GB HDD
  • 4 cores

 

Optimal Performance Scenarios

Listed below are the different factors influencing performance scenarios.

  • Containers - The number of Containers sent per data message
  • Clients - The number of Clients sending the number of Containers listed
  • Properties - The number of Properties/PI Points per Container
  • Values - The number of Values sent for each Property
  • Performance - The number of events per second ingressed the user can expect of OMF

 

Containers

Client(s)

Properties

Values

Performance(events/s)

2000

10

5

1

>100k

1000

10

10

10

>200k

1000

15

10

10

>250k

100

20

10

10

>300k

 

Results will vary based on real-world conditions but you should expect at least these baseline performance results. Results will most likely be much higher than the numbers listed in the performance.

 

 

Final Notes

When data/container messages are sent to a newly started/restarted PI Web API, the cache needs to be filled with PI Points. This reduces the performance of the first call. Subsequent calls will result in faster speeds, and you can expect significant improvement in performance from the first call. Make sure to set the client timeout to longer than the default of 30 seconds because the first calls will take longer.

 

 

 

It is slightly beneficial but not required to keep machines close together physically because there will be less latency between machines. (Client(s) -> PI Web API, PI Web API -> [OMF]AF/DA)

Outcomes