8 Replies Latest reply on Mar 30, 2016 12:33 PM by gregor Branched from an earlier discussion.

    PI Web API - High memory usage concern

    xandelisk

      Hi there,

       

      We have an similar need, and we're concerned about the possibility of high memory usage.

       

      We have to call 10.000 points at each 10 minutes interval to check if the point are "off limits" and then we recorded the period that the points are "off limits".

       

      To get better performance, we use batch call to have less requests. We made 20 calls in one batch request for example, and get the recorded values of each point, since the last check to that specific point. The start time of each GET grouped may differ. Here is a post body example:

       

      {

        "1": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wUh8AAAUEkyMjAwMVxUMTcwN0JfTkk/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "2": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wmcwAAAUEkyMjAwMVxQREkzNzI2OQ/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "3": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4whkYAAAUEkyMjAwMVxWSTEzMDAy/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "4": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wpS0AAAUEkyMjAwMVxWSTE4MDIx/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "5": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4w1ZUAAAUEkyMjAwMVxUNDY3NkFfTkk/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "6": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wdK4AAAUEkyMjAwMVxVMzBNQVhURU1QU0tJTkIzMDA0/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "7": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wsvYAAAUEkyMjAwMVxSRUwgSDJPL0NBUkdBIFRSRU0gQQ/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "8": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wk5sAAAUEkyMjAwMVxUNDY1NERfTkk/recorded?startTime=2016-03-09T14:11:01.000Z&endTime=*"

        },

        "9": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wZs8AAAUEkyMjAwMVxGSUMzNzAwMQ/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "10": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wqoQAAAUEkyMjAwMVxDQUxDQjMwMDJFRkk/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "11": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wdVkAAAUEkyMjAwMVxVMTFSQ1IvT0RF/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "12": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4whhQAAAUEkyMjAwMVxQSUMyNTI4OQ/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "13": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wbR8AAAUEkyMjAwMVxWSTYwMjU/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "14": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wyAgAAAUEkyMjAwMVxUSUMxMjAwMw/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "15": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4w0gUAAAUEkyMjAwMVxUMTc0NV9OSQ/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "16": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wvgkAAAUEkyMjAwMVxMSTgzMDExQg/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "17": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wocwAAAUEkyMjAwMVxQSTM3MDkw/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "18": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4whpsAAAUEkyMjAwMVxUNDY0MkNfTkk/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "19": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wyocAAAUEkyMjAwMVxWSTk4MEE/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        },

        "20": {

          "Method": "GET",

          "Resource": "https://localhost/piwebapi/streams/P0meD1zc_XJk-7EpmMk1Zs4wO8IAAAUEkyMjAwMVxGRkkzOTAwMQ/recorded?startTime=2016-03-09T14:11:02.000Z&endTime=*"

        }

      }

        • Re: PI Web API - High memory usage concern
          gregor

          Hello Alexandre,

           

          Have you considered opening a new thread and to refer this discussion with your new question? I am kind of uncertain at this point in time but we may decide to branch your post into a separate question if it turns out to become a separate discussion.

          Please note that the Batch service and the Channel service is included as a Customer Technology Preview (CTP) into the current release of PI Web API. Please refer to PI Web API 2015 R3 Release Notes.

          However, based on my experience with PI Web API, I am not worried about checking 10k data items every 10 minutes. You however don't need Batch requests for this purpose. I suggest using StreamSet queries which allow you to query data of multiple unrelated data items with a single call. I suggest your application to collect webID's at startup, update them regularly and to use StreamSet queries with 100 ~ 1,000 webID's. I believe the maximum URL length in PI Web API is at about 80k characters but you may run into client side limitations that are smaller. Please see below for a sample StreamSet query:

           

          https://<piwebapihost>/piwebapi/streamsets/value?webid=<webID1>&webid=<webID2>&webid=<webID3> 
          
          1 of 1 people found this helpful
            • Re: PI Web API - High memory usage concern
              xandelisk

              Hi Gregor,

               

              Yes, I think is a good idea open a new thread. A Admin can do this, or I do it myself?

              We try to use streamset, but for each point the start time can be different. That is the limitation of Streamset in this case.

                • Re: PI Web API - High memory usage concern
                  pthivierge

                  Hello Alexandre,

                   

                  Branching has to be done by an Admin

                  I did branch your question into a new one.

                  • Re: PI Web API - High memory usage concern
                    gregor

                    Hello Alexandre,

                     

                    Sorry, I somehow missed replying. Luckily Patrice stumbled across this thread and gave it a push up the list by branching your question into a new one.

                    By how much can the start times differ? The advantage of using bulk calls really is that the roundtrips to the server becomes reduced dramatically which really helps to improve performance. It may be worth to align start time for your queries and to use streamset queries with a common start- and end-time instead of using batch calls. You still can discard the events you don't need client side.

                    Please allow me to remind that batch service is included with the recent PI Web API release 2015 R3 as a Customer Technology Preview (CTP) which means that we offer an early insight into that technology but at the same time there's no guarantee the technology will ever become included with an official release. Please note that the decision about what makes it to release also depends on the feedback development / product management receives from you, the users.

                    1 of 1 people found this helpful
                • Re: PI Web API - High memory usage concern
                  pthivierge

                  Hello Alexandre,

                   

                  I would expect StreamSet to perform overall better performances than Batching your requests because StreamSet will operate using bulk data retrieval underneath and this can make a real difference when the PI Web API Service will collect data from the PI Data Archive.  Especially if it is located on another box in a network which has a bit of latency.  However, this may not be that simple, see this interesting discussion AF SDK performance, serial vs. parallel vs. bulk

                   

                  For Batching, the first benefit is that you can group all your queries in a single call to maximize bandwidth usage and limit latency over the network.

                  From the description provided in the manual, POST batch:

                  Execute a batch of requests against the service. As shown in the Sample Request, the input is a dictionary with IDs as keys and request objects as values. Each request object specifies the HTTP method and the resource and, optionally, the content and a list of parent IDs. The list of parent IDs specifies which other requests must complete before the given request will be executed. The example first creates an element, then gets the element by the response's Location header, then creates an attribute for the element. Note that the resource can be an absolute URL or a JsonPath that references the response to the parent request. The batch's response is a dictionary uses keys corresponding those provided in the request, with response objects containing a status code, response headers, and the response body.

                  The fact that you can provide a parent ID indicates that queries are made in parallel unless you specify a dependency between the calls.

                   

                  So we could say that StreamSet is bulk and batching will do Parallel calls and you might get good performances by combining the two:

                  • Make groups of StreamSet calls, you may group all tags with the same time range together.
                  • Make a single batch call with all StreamSet groups you made

                  This will provide best possible performances if you have a notable latency between the PI Data Archive and the PI Web API Service.  Otherwise you should see similar performances than you had with the Batch call.

                   

                   

                  Memory usage

                  PI Web API will make use of caching, so if data in the same time range was already gathered, it won't make new calls to the PI Data Archive.

                  The more tags you query, the more data you are likely to get in the cache, but cache should be cleared after 15 minutes. ( this is explained in the user guide).

                  Caching can also be configured at the AFSDK level in case memory is a concern for you, see AFSDK.Config caching settings?  for more information. You should not change the AFSDK cache settings lightly as this may also impact AFSDK applications that are sitting on the same box.  Just let us know if you have questions.

                   

                  Hope this information helps you,

                  1 of 1 people found this helpful