I’d like to take some time to give some guiding tips on how to improve your PI Web API requests’ performance. “How do we make our PI Web API queries faster” is a question we often get here at OSIsoft and the answer generally varies depending on the architecture and request involved.
The goal here is to make a list of all the most common improvements we generally recommend to customers and clients looking to make the most of their PI Web API requests. I’ll be posting some of the actual metrics from my system to demonstrate this, but I want to stress that the performance can vary based on a number of variables such as your network architecture, the load on your Data Archive and AF Server, the amount of out of order data, and many more factors. This document is intended to show general principals that will hold true for improving code on any system.
In general, better performing HTTP requests have smaller packet sizes and/or low latency in the response. To monitor these metrics we can use Fiddler, which will be installed on the client machine making HTTP requests to the PI Web API.
Packet Size of the PI Web API Response
Why would we want to reduce the packet size though? This has two benefits. First, smaller packet sizes mean both the PI Web API service and the HTTP client making the request need to do less encoding, decoding and assembling of TCP packets if they’re smaller. Since smaller downloads are quicker than larger ones, we expect these responses to returned and usable more quickly.
In Fiddler, you can use the “Transformer” section of the “Inspectors” tab in Fiddler to determine the response size:
Latency in the Response
Latency is the amount of time a message takes to traverse a system. In other words, how long does is take data to get from one designated point (the PI Web API) to another (the HTTP Client).
To measure this, under the “Statistics” Tab of Fiddler, a breakdown of each of the individual steps to getting our response is shown. We’re mostly concerned with the time between “Fiddler Begin Request” and “Client done Response.”
All other things considered equal, a PI Web API request will perform better when the latency between the PI Web API and the Data Archive is reduced as much as possible, over reducing the latency between the client and the PI Web API service. The same is true for reducing the latency between the PI Web API service and the AF Server over reducing latency between the PI Web API and the client.
This is because the communication between the PI Web API service and the HTTP Client making the request is much less bandwidth intensive than between the PI Web API and the Data Archive or the PI Web API service and the AF Server.
For example, making a request from my client machine in Philadelphia to a PI Web API instance in Philadelphia (Venus.Holst.Dev) requesting data from a Data Archive in San Francisco will be better performing than making the request from the same client for the same information, but this time with the PI Web API instance also being located in San Francisco, nearer to the Data Archive than the Client.
Using the SelectedFields URL parameter to reduce the amount of information being sent over the wire
This parameter allows you to specify the content you wish to be returned to you in the response, so you get all the information you’re interested in and none of the information you aren’t.
If we specify the information we’re looking for the PI Web API needs to make fewer calls to resolve information/data. For example, if we’re only looking for the template an element is based on then the PI Web API won’t go through the work of resolving out the Description, whether it has child elements, what categories it belongs to, etc. and will strictly resolve the information asked of it. This reduction of requirements for the response will speed up the actual processing of the request.
Additionally, by requesting less information the PI Web API will be able to reduce the packet size and speed up the request this way as well.
Using WebIdType to reduce the amount of information being sent over the wire
Another consideration we can do to decrease the size of the payload is to specify the WebID type for additional optimization in reducing the load on the network.
Using shorter WebId types (like ID-only in lieu of Full WebIds) will require fewer characters to properly encode the WebId and thus create smaller packet sizes. While this may not seem significant at first consider two things.
First, Full WebIds encode the entirety of the path to an element. Thus very deep AF hierarchies can benefit more from this change and will notice more significant reductions in the WebId encoding.
Second, when performing operations with batch requests that require building up a list of WebIds in one sub-request for use in a dependent sub-request, the size of the WebIds will add up quickly when working with many objects.
It’s worth noting that if you’ll be manipulating objects through creation, deletion, moves, and renames then you may want to stick with Full WebIds, in this case, to always ensure that you have a reference that can be linked to the original object you’re manipulating.
Use Batch Controller over Individual Calls when possible
Using the batch controller in lieu of individual calls will allow the PI Web API to optimize queries in two ways. The first being that the PI Web API doesn’t need can chain together parent-child requests so that the information of the parent response doesn’t need to be sent across the wire back to the client for a new request to be generated and sent back to the PI Web API. This reduces the number of trips across the wire between the PI Web API and the HTTP Client.
Additionally, with Batch requests, the PI Web API is capable of optimizing non-dependent sub-requests in the body of a batch controller call to run in parallel, so it can process multiple sub-requests simultaneously if they don’t depend on one another.
It’s worth noting that, by default, the PI Web API does throttle requests when it begins to receive over 1000 requests per second (although this is a configurable parameter, please refer to the PI Web API Configuration section of the PI Web API User Guide). Consider designing batch calls to have less than 1000 sub-requests, particularly when using request templates as these requests don’t become visible until runtime.
Use Streamsets over calls for individual attributes when requesting all the attributes on an Element or EventFrame
The PI Web API’s streamset controller can optimize requests for multiple attributes on Elements and Event Frames in a bulk call fashion. By implementing this type of bulk call for information, the PI Web API retrieves information for multiple attributes more rapidly via the streamset controller than if made by multiple individual streams controller requests for each attribute.
Using Plot Data vs Recorded Data for displaying a plot
The PI Web API has several options for retrieving lists of values – Recorded values, Interpolated values, and Plot values. If you’re using data for a graph, we recommend using plot values to retrieve the values in lieu of recorded or interpolated. This is because with the Plot values you can control the number of values being returned to you and ensure they are significant values.
With graphs, you will never require more values than the width of your graph in number of pixels. You’ll be unable to effectively display that many values. When retrieving plot values, you’ll be able to specify the width of your graph in number of pixels and thus ensure you don’t get more information than can readily be displayed on visualization.
Plot values also has the added benefit of significant data value selection. If you were to use interpolated values, you may end up missing important peaks and troughs in your data if the data happens to be cyclic and not thus not visible at those interpolated times (as would be the case in a periodic or batch process). Alternately, you may have gaps in your data from hardware failure that are between interpolated times and thus would be missed using interpolated data. Plot data was developed for this use case, to retrieve a smaller number of values while ensuring that values of interest are included.
Using Filters in your search query
The next consideration is using filtering with searches to retrieve only the objects we care about to reduce the payload size. We can filter information on the client side after retrieving everything, but it makes more sense to only ask for what we need from the PI Web API and then have that returned to us to reduce the amount of data sent over the wire to improve performance. So we would recommend doing as much filtering as possible on the PI Web API side using the elastic search whenever possible. For example, if we only want the elements that start with the letter B we can use the AF Search to filter for these elements.
Generating your own WebIds via the Path or GUID of the object requested can improve the performance of your application by reducing the amount of processing the PI Web API needs to perform. By generating these HTTP client-side, you can directly make requests for the information on the objects you desire.
Let’s say for example you know you want to the Temperature attribute on all your Boiler elements generated from the same template. One method for doing this is to make a batch request that queries for all the elements based on that template, retrieving their WebIds and then making a subsequent sub-request for the value of the Temperature attribute using those WebIds found in the first sub-request.
The alternative to doing this is to generate WebIds of the Temperature attribute client-side, then send a batch request directly asking for the values of the Temperature attributes on those elements. This requires that you either know the path to the Temperature attributes ahead of time in order to pre-generate the WebIds.
Use Stream Updates and/or Channels instead of successive Data calls
The PI Web API supports stream updates and channels as methods for retrieving new data for existing streams that you’re monitoring. Stream Updates and Channels have smaller responses than typical calls for data and thus require less bandwidth than a typical get recorded values stream call.
For Stream Updates, the data is queued by the PI Web API service, the client will get all the data it needs and none of the data it doesn’t as it will only return updates and changes in values since the previous call for data. This prevents the need to keep track of times when calling for data, simplifying life for the developer, as well as preventing accidental return of recorded data that was already received in prior call to the PI Web API.
In the case of channels, the updates are pushed to the HTTP Client rather than requiring a request be made by the client and require half as many trips over the network.