In my previous blog post (PI Web API: Getting multiple attributes with batch requests), I discussed two methods that can be used to retrieve multiple WebIds for attributes that belong to a collection of elements. These two methods utilized the batch controller:

 

  • Method A: Fetch specific attributes for all elements
  • Method B: Fetch all attributes for all elements

 

My previous post discussed the pros and cons for these two methods, mostly due to performance considerations. In this blog post, I will discuss how to use the attribute WebIds to stream data using the batch controller.

 

Extending the batch request

 

The simplest and most straightforward way to request stream data using the batch controller for multiple attributes is to extend the batch requests that we already developed. This approach works best when using Method A of retrieving attributes. However, great care must be taken when using this approach in conjunction with Method B, as it can result in significantly more data requested than is actually needed. This could result in a less performant batch request.

 

To retrieve all of the snapshot values for every attribute using Method A, our batch request may look something like this:

 

{
    "database": {
        "Method": "GET",
        "Resource": "https://devdata.osisoft.com/piwebapi/assetdatabases?path=\\\\PISRV1\\NuGreen&selectedFields=WebId;Path;Links"
    },
    "elements": {
        "Method": "GET",
        "Resource": "{0}?templateName=Boiler&searchFullHierarchy=true&selectedFields=Items.WebId;Items.Path;Items.Links",
        "ParentIds": [
            "database"
        ],
        "Parameters": [
            "$.database.Content.Links.Elements"
        ]
    },
    "attributes": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/attributes/multiple?selectedFields=Items.Object.WebId;Items.Object.Path&path={0}|Asset Name&path={0}|Model&path={0}|Plant"
        },
        "ParentIds": [
            "elements"
        ],
        "Parameters": [
            "$.elements.Content.Items[*].Path"
        ]
    },
    "data": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/streams/{0}/value"
        },
        "ParentIds": ["attributes"],
        "Parameters": ["$.attributes.Content.Items[*].Content.Items[*].Object.WebId"]
    }
}

 

There are several limitations to this approach that need to be mentioned:

 

  1. The "data" batch sub-request will construct multiple individual stream requests. This may not perform as well as making streamset requests for multiple WebIds.
  2. The response data does not indicate which value corresponds to which WebId. In order to make this connection, you have to rely on the order the WebIds appear in the "attributes" sub-request response data. Some programmers may not feel comfortable making this assumption, as there is a risk that a stream value may not be assigned to the correct attribute.
  3. This method does not allow cherry-picking which stream controllers are requested for which attributes. For example, it may be desirable to request snapshot values for one set of attributes, but time-series data for another. The only remedy for this is to make multiple "attributes" sub-requests (some of which may result in duplicate responses).
  4. If you are polling data, this approach requires you to run the entire batch request on every poll interval. This is redundant and it would be better to make the data requests in a separate batch query.

 

To address these concerns, I believe it is better to split the batch request in two: one to retrieve the element and attribute metadata, and a second to retrieve all the stream data. The biggest advantage is that this second request can be executed repeatedly using a long polling strategy. Before I address this though, I want to demonstrate how the first limitation I have indicated above could be addressed using only a single batch request, if only for the sake of completeness.

 

Requests with multiple stream controllers

 

If you want to request data using different stream controllers, but still only want to use a single batch query, the only way I have found is to make multiple sub-requests for attributes and stream data. For example, we could get snapshot values for all Model attributes and plot data for all Fuel attributes with a batch request like so:

 

{
    "database": {
        "Method": "GET",
        "Resource": "https://devdata.osisoft.com/piwebapi/assetdatabases?path=\\\\PISRV1\\NuGreen&selectedFields=WebId;Path;Links"
    },
    "elements": {
        "Method": "GET",
        "Resource": "{0}?templateName=Boiler&searchFullHierarchy=true&selectedFields=Items.WebId;Items.Path;Items.Links",
        "ParentIds": [
            "database"
        ],
        "Parameters": [
            "$.database.Content.Links.Elements"
        ]
    },
    "attributes1": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/attributes/multiple?selectedFields=Items.Object.WebId;Items.Object.Path&path={0}|Model"
        },
        "ParentIds": [
            "elements"
        ],
        "Parameters": [
            "$.elements.Content.Items[*].Path"
        ]
    },
    "attributes2": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/attributes/multiple?selectedFields=Items.Object.WebId;Items.Object.Path&path={0}|Fuel"
        },
        "ParentIds": [
            "elements"
        ],
        "Parameters": [
            "$.elements.Content.Items[*].Path"
        ]
    },
    "data1": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/streams/{0}/value"
        },
        "ParentIds": ["attributes1"],
        "Parameters": ["$.attributes1.Content.Items[*].Content.Items[*].Object.WebId"]
    },
    "data2": {
        "Method": "GET",
        "RequestTemplate": {
            "Resource": "https://devdata.osisoft.com/piwebapi/streams/{0}/plot"
        },
        "ParentIds": ["attributes2"],
        "Parameters": ["$.attributes2.Content.Items[*].Content.Items[*].Object.WebId"]
    }
}

 

As you can see, this could become quite complicated depending on the use case. For example, imagine requiring the following composition of data:

 

  • AttributeA: value stream
  • AttributeB: plot stream (8 hours)
  • AttributeC: value stream and plot stream (1 day)

 

The best strategy here would be to build the data requests after the initial batch request has been made. Doing this also opens up the possibility to make streamset requests, which will be the topic of the next section.

 

Building streamset requests

 

After receiving attribute metadata, it's relatively straightforward to create a "WebId cache" object that maps attribute paths to their WebIds (I discussed this in my previous blog post). From there, if you know which stream controllers you want to use for each attribute, then you can build a second batch request for the data using streamsets. For the sake of simplicity, let's suppose we already have a JavaScript object that defines which relative attribute paths are associated with stream types. This object might look something like this:

 

{
    "value": ["|Model"],
    "plot?startTime=*-8h&endTime=*": ["|Fuel"]
}

 

We can use this structure to get a list of WebIds for these relative attribute paths:

 

function getWebIdsByPath(webIdCache, paths) {
    var result = [];

    return Object.keys(webIdCache).filter(function (path) {
        var relPath = path.slice(path.indexOf('|') + 1);
        return paths.some(function (p) {
            return p.slice(p.indexOf('|') + 1) === relPath;
        });
    });
}

 

Using this function, we can now build our batch request for data:

 

function buildDataBatch(webIdCache, streams) {
    var batch = {};

    Object.keys(streams).forEach(function (stream, i) {
        var relPaths = streams[stream];
        var webIds = getWebIdByPath(webIdCache, relPaths);

        var resUrl = 'https://devdata.osisoft.com/piwebapi/streamsets/' + stream;
        if (resUrl.indexOf('?') === -1) resUrl += '?';
        else resUrl += '&';

        batch[i] = {
            'Method': 'GET',
            'Resource': resUrl + webIds.map(function (webId) {
                return 'webId=' + webId;
            }).join('&')
        };
    });


    return batch;
}

 

As mentioned before, this approach offers some key advantages:

 

  1. Data is requested using streamsets with multiple WebIds, rather than multiple streams with a single WebId each. Internally, the PI Web API probably performs better this way.
  2. The response of streamset requests includes WebIds, making it much easier to manage data bindings in your application.
  3. This batch request can be made repeatedly using long polling. Long polling is where you wait for the response to come back, and then make the request again (either immediately or after a few seconds).

 

However, this is one crucial problem with this approach: request URL length limitations. When request URLs become too long, the PI Web API will reject them, even if they are made using the batch controller. This can happen when you have too many WebIds on a single streamset. The only way to fix this is to break up the streamset requests so the maximum URL length is not reached. To make matters even more complicated, there is also a maximum number of items that can be returned per call (configurable in the PI Web API settings). As a result, if you are requesting time-series data using this approach, you must use the "plot" controller. This is because we must know how many items might be returned, and the plot controller lets us control this with the "interval" parameter.

 

After a lot of trial and error, I have found the following constraints to work for me:

 

TotalWebIdsPerStreamset = MaxReturnedItemsPerCall / Intervals / 5

MaxWebIdParametersLength = 65519 - BaseUrlLength

 

where BaseUrlLength is the length of the streamset resource URL before the WebId parameters have been appended. As long as both of these conditions are met, the streamset request should not fail. There are many strategies that can be used to split up the WebIds to meet that constraints. The one that I use is a "bin packing" algorithm.

 

Concluding Remarks

 

Hopefully this post is helpful to you as you develop your custom web applications. I realize there are many implementation details missing, I wanted to just provide some general strategies for requesting bulk data using the PI Web API.

 

In my own applications, I have used these approaches to build web applications that request data streams for thousands of attributes. As a frame of reference, requesting data for 5000 attributes using a mixture of snapshot values and plot values can take anywhere from 4-10 seconds to return results. I have yet to benchmark this compared to the AF SDK, but I feel this is still pretty slow, and am always on the lookout for ways to optimize my batch requests even further.

 

I look forward to hearing ideas from the community!