16 Replies Latest reply on Sep 11, 2015 4:38 PM by rxu

    How should we support multiple calls to PI Web API as one HTTP POST?

    ray

      You've probably noticed that the PI Web API provides good services for getting information from the PI System such as Elements, Attributes and time series data from either PI Data Archive points or Attributes. Sometimes, however, you'll find that you need to make several REST calls in order to get a task done. We've added bulk read and write calls to the PI Web API (check out the programmer's guide at https://yourserver.com/piwebapi/help for StreamSets and any method ending in AdHoc). This helps, but you may still find that you need to make a series of REST calls in order to finish the job.

       

      We call this the REST call batching problem. We are looking at various ways to do this and we would like to get your opinion on 3 options. To keep this discussion organized, I will describe the options as replies to this discussion board post.

        • Re: How should we support multiple calls to PI Web API as one HTTP POST?
          ray

          Option #1: Use the MIME Multipart Specification

           

          This means that you do a single standard HTTP post but the document posted contains all of your individual REST calls. You use the MIME label "Content-Type: multipart/mixed; boundary=mylabel" at the top of your document. Information for each individual REST call is separated by the text string "--mylabel." Each individual request can have its own content type (which PI Web API doesn't need because it's always "application/json"). There's a very readable explanation of all of this in Wikipedia. Support of all of this in client APIs is mixed. The .Net Framework supports it through the Microsoft Web API.

          • Re: How should we support multiple calls to PI Web API as one HTTP POST?
            ray

            Option #2: OSIsoft's Proposal

             

            Our team came up with a different proposal. In ours, you also invoke a single HTTP post containing all of your requests, but everything would be in a single JSON document. This document is just an array of requests, each containing its own HTTP method label, Resource string and Header attributes. For example:

             

            {

                "1": {

                    "Method": "GET",

                    "Resource": "https://my-server/piwebapi/elements/webId",

                    "Headers": {

                        "Cache-Control": "no-cache"

                    }

                },

                "2": {

                    "Method": "POST",

                    "Resource": "$.1.Links.Attributes",

                    "Content": "{\"Name\":\"New Attribute\"}",

                    "ParentIds": ["1"]

                }

            }

             

            This example contains what would normally be two PI Web API calls. In Request #2, the ParentIds value of "1" means that Request #1 must be executed before #2. PI Web API would take care of the sequencing of calls. Look at the syntax of Resource in Request #2. This is JsonPath syntax which allows you to make reference to specific data in a different JSON response. In this example, the JsonPath string points to the Attributes collection within the Links array within the response to Request #1. JsonPath is just like XPath but for JSON documents.

              • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                Rhys Kirk

                Option #2 gets my vote.

                How complex the server side is wouldn't be a concern for an end user.

                The Graph API from Facebook allows you to batch requests as per option #2 proposes: Making Multiple API Requests

                • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                  Lonnie Bowling

                  I like option #2 the best. I like the flexibility it offers and chaining request is so typical that this would make the API much more rich. So when is this going to be ready?

                  • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                    bshang

                    I would vote for Option #2 as well, but share Arnold's concern about how to manage the chain of dependencies and error handling.

                     

                    I'm curious to hear from community members what are your typical patterns of usage of PI Web API calls and where you find yourself having to make many intermediate calls. If these usage patterns are all somewhat similar, it may be helpful to introduce additional methods within PI Web API that would effectively implement what would have been done using Option #2 anyway, with the benefit that the actual API call can be simpler/more opinionated rather than accepting a POST batch script. I suppose this starts moving towards Option #3. Can I vote for #2.5?

                    • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                      ray

                      Introducing the Option #2 CTP!

                       

                      Thanks everyone for your feedback. This looks like everyone's favorite. We've been prototyping Option #2 in the PI Web API. It seems to be the most powerful choice since it allows a large amount of work to be done in parallel and also supports dependent requests. We are going to offer Batch processing as a Community Technology Preview (CTP) feature in the upcoming release of PI Web API 2015 R3 in October. What CTP means is that we will make the feature available but reserve the right to change the design based on your feedback. If that happens, some of your code might break. Not fun, but our goal is to create the best design that solves your batch processing problems. We will include documentation and some sample code. Look for Batch in the PI Web API Help when PI Web API 2015 R3 is released.

                        • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                          ray

                          Error Handling in Batch Option #2

                           

                          There are questions about error handling in Option #2. The overall response to the Batch request would be an HTTP 200 if a least one component request succeeded. The individual responses will appear in a JSON document that mirrors the request. Here's a sample response for my sample request:

                          {
                              "1": {
                                  "Status": 200,
                                  "Headers": {
                                      "Content-Type": "application/json"
                                  },
                                  "Content": "{\"Name\":\"My Element\",\"Description\",...}"
                              },
                              "2": {
                                  "Status": 201,
                                  "Headers": {
                                      "Location": "https://my-server/piwebapi/attributes/webId"
                                  },
                                  "Content": ""
                              }
                          }

                          If any component request fails, error information would appear in this response. If a component request fails, any request that is dependent on it will not be executed. A fact that a component request was not executed would be reflected in the response. We will not attempt to repeat the request again because we don't know the reason for the failure. This means that the client will have to interpret the failure and decide what to do next.

                      • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                        ray

                        Option #3: Organizing Response Data in a Different Format

                         

                        This may not sound like it has anything to do with batching of queries but read on. Do you need to make multiple PI Web API calls in order to get data from different parts of the PI System and organize that data in a particular way? You might want to create a table containing interpolated data from several time series data streams. Or, you may need a table whose columns are defined by Attribute names and whose rows are Element names together with Attribute values. OData is one example of a response data format defined by a standard. I am sure there are others. This might be important when coding with, say, display widgets from a vendor that expects input data in a particular format.

                         

                        Making many PI Web API calls and then organizing the response data in a specific way might require a lot of code. Is there something the PI Web API could do to help?

                          • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                            Lonnie Bowling

                            Even though I didn't vote for this option, I think if it is done correctly, it would be the most powerful, yet simplest to use.  I have mentioned Breeze with OData several times in the past, and still love how it works. If there is something that could work like that, using the power of OData without the complexity, that would be amazing. Breeze will keep track of object states and basically go back and grab more data when needed (lazy loading) and will cache data insuring that calls are not repeated. You work with an object model on the client side, Breeze takes care of insuring your model is populated as required.

                          • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                            sdriscoll

                            I think bulk calls (adhoc) were a huge step in the right direction. I'm not over my excitement about bulk calls and now batching requests...can't wait!

                             

                            My vote is for option #1.

                             

                            I'm not a fan of option #2. At first glance it provides a great option for batching together dependent requests in a single call.  However, I worry about the maintainability of complex, interdependent batched calls server-side and the ability of this sort of feature to be abused. The beauty of the current model is the individualistic and simplistic nature, yet it provides a rich set of features for developers. The dynamic nature of option #2 seems counter to that model. I suppose I'm assuming that you cannot create that dependent nature in option #1, otherwise I have to rethink my opinion.

                             

                            I admittedly don't have the experience for an educated discussion on option #3. I briefly played with OData which was pretty awesome - robust and easy - but I don't know the full capabilities of it.

                              • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                                awoodall

                                Option #1 is good if we mainly need batching of independent calls. This option would be useful if, for example, I wanted to get recorded values for some long list of PI Points which a bulk call doesn't exist for. Replaying failed batch components in this case is straightforward. But this option doesn't really seem to help fully solve the problem originally described as it can't carry out a set of dependent steps in one shot..

                                 

                                Option #2 seems to better satisfy the problem domain. It would help with Element/EF->Attribute->Value type sequences that are most commonly performed in sequence. But I have concerns about its potential implementation, as its not clear how errors would be best handled in a batch of dependent requests. Would we cancel the execution of the rest of the batch upon the first error? Or would we retry some number of times before failing the entire batch? Does the error handling behavior depend on the structure of the chain of dependencies in the request? I think even if there was some reasonably intuitive error handling option and response given by PI Web API, I think it would still be hard for a developer to figure out how to best recover from the point of failure if the dependency chain was very complex, especially if writes are involved in the chain. Most likely a developer would just want to rerun the entire batch should a step fail, as long as there are no side effects of a partial failure. This option may be ok if we can put a limit on the dependency complexities that could potentially be developed.

                                • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                                  Rhys Kirk

                                  Sean Driscoll In my rather simplistic view the batching of the multiple requests will cut down on the network trips but it is basically the same functionality as though multiple individual calls were being made. Assuming a client could be throttled to prevent abuse of the batched requests (e.g. stuffing 1000's of requests together) then will the endpoint really change that much?

                                  You could liken it to the AF SDK and it's approach to handling Attribute List requests for data, you really just want to avoid the network latency so batch the requests together, which are then (for PI Points) dealt with by the PI Data Archive and served back to the client application by the AF SDK.

                                • Re: How should we support multiple calls to PI Web API as one HTTP POST?
                                  sdriscoll

                                  How some larger public APIs do it.

                                   

                                  Google - option #1

                                  Facebook - option #2 (can make dependent calls)

                                  Salesforce (more info) - option #2 (only independent calls)

                                  MS SharePoint - option #1

                                  Amazon - I think they just use bulk calls

                                   

                                  I also looked at Twitter, Dropbox, Flickr and YouTube which didn't have support for batch requesting in a REST API, or none that I could find.