AnsweredAssumed Answered

maxCount in PI WebAPI calls restricted

Question asked by Paurav Joshi

on Apr 12, 2016
Latest reply on Apr 18, 2016 by Dan Fishman

Like • Show 0 Likes0
Comment • 7

Hello All,

I have come today that maxCount limitation in PI WebAPI call is 150000. I went to check tuning parameter in my PI Data Archive and to my surprise found one parameter maxArcCollect which is also same number 150000 as value .

That raised some doubts for me:

What is significance of this maxArcCollect ? How it affects querying archived data of tags in PI WebAPI calls?
How processbook or PI datalink is fetching data of the same tag of last month period which has 238543 events?
What will be impact on PI Data Archive, if I increased that number of maxArcCollect to 2000000?
If I query tag which has 2000000 archived value after increasing number of maxArcCollect, how will it affects resources (example, RAM, CPU usage) on my machine?

Regards,

Paurav Joshi

Note: This case is useful only for bulk read using stream calls of PI Web API

Message was edited by: Paurav Joshi

Dan Fishman Apr 12, 2016 12:19 PM

Correct Answer

Here is a KB article that describes ArcMaxCollect which is the parameter I believe you are asking about. KB00367 - Changing the ArcMaxCollect parameter: What are the ramifications or possible issues?

Older servers that were upgraded most likely have a default limit of 150K and newer severs have a limit of 1.5 million as the default. This parameter is designed to protect the PI Sever from expensive queries. It is more likely that this data will be on disc and not on the primary archive so your'll see more disc activity and this data will be placed in the file systems cache on the Data Historian server. In general, 2 million events is not an unreasonable setting and you will still give you protection against large queries. An alternative to going up to say 10 million is to chunk your queries. A few other parameters you might want read about: 3224OSI8 - How to limit expensive queries to the PI Server archive

See the reply in context

No one else had this question

Outcomes

Dan Fishman Apr 12, 2016 12:19 PM

Here is a KB article that describes ArcMaxCollect which is the parameter I believe you are asking about. KB00367 - Changing the ArcMaxCollect parameter: What are the ramifications or possible issues?

2 people found this helpful

Actions

Paurav Joshi

@ Dan Fishman on Apr 12, 2016 12:35 PM

Thanks Dan. This KB articles are very informative .

Curious about case like question 2:

If my processbook trend or PI datalink fetching more data than maxArcCollect, then how it will be done?

Will my one call through processbook or datalink automatically converts into two concurrent calls or if otherwise than how?

Cheers,

Paurav Joshi

Actions

Dan Fishman @ Paurav Joshi on Apr 12, 2016 1:51 PM

Good question! No, PI Datalink and PI Processbook do not automatically convert into two (or more) calls if the limit is hit. You typically do not experience this issue in PI Processbook because it use the SDK method Plotvalues which only retrieves enough information to accurately display the trend. A PI ProcessBook DataSet requires all the archived values so it does not use PlotValues and is subject to ArcHaxCollect.

Actions

Paurav Joshi

@ Dan Fishman on Apr 12, 2016 2:29 PM

That was very insightful Dan

I have query that if I am developing custom application using PI Web API and I am fetching archive data of, as an example 6, assets for a month, then the query has to get 20 million around data. If my maxArcCollect is 10 million, then please advise me on how should I make query to get this data?

Option 1: Increase the maxArcCollect

Option 2: Divide the single query into two and call them concurrently

Please suggest if we have options except above two.

Regards,

Paurav Joshi

Actions

Dan Fishman @ Paurav Joshi on Apr 12, 2016 4:09 PM

If you always know the data density and approximately how many events it might make sense to do concurrent chunks to provide some protection to your PI System instead of increasing maxArcCollect. If users often need 10 million events, then maybe increase this. In most cases, the Chunking is slightly difficult to determine appropriate sizes since data density is not known ahead and we don't really have a way to page on this yet.

1 person found this helpful

Actions

Paurav Joshi

Apr 18, 2016 2:21 PM

Little information need to be added here.

I was facing this issue when I am fetching PI tag data using PI Web API via streams like,

https://{web_api_hostname}/piwebapi/streams/{webid_of_tag}/recorded?starttime=*-30d&endtime=*&maxcount=150000

For bulk read via streamset please follow this thread .

Cheers

Actions

Dan Fishman @ Paurav Joshi on Apr 18, 2016 3:32 PM

Take a look at the MaxItemsPerCall configuration attribute and you can read about it here: PI Web API

1 person found this helpful

Actions

7 Replies

Dan Fishman Apr 12, 2016 12:19 PM
Unmark Correct
Correct Answer
Here is a KB article that describes ArcMaxCollect which is the parameter I believe you are asking about. KB00367 - Changing the ArcMaxCollect parameter: What are the ramifications or possible issues?
Older servers that were upgraded most likely have a default limit of 150K and newer severs have a limit of 1.5 million as the default. This parameter is designed to protect the PI Sever from expensive queries. It is more likely that this data will be on disc and not on the primary archive so your'll see more disc activity and this data will be placed in the file systems cache on the Data Historian server. In general, 2 million events is not an unreasonable setting and you will still give you protection against large queries. An alternative to going up to say 10 million is to chunk your queries. A few other parameters you might want read about: 3224OSI8 - How to limit expensive queries to the PI Server archive
2 people found this helpful
Like • Show 0 Likes0
Actions
- Paurav Joshi @ Dan Fishman on Apr 12, 2016 12:35 PM
  Mark Correct
  Correct Answer
  Thanks Dan. This KB articles are very informative .
  
  Curious about case like question 2:
  If my processbook trend or PI datalink fetching more data than maxArcCollect, then how it will be done?
  Will my one call through processbook or datalink automatically converts into two concurrent calls or if otherwise than how?
  
  Cheers,
  Paurav Joshi
  Like • Show 0 Likes0
  Actions
  - Dan Fishman @ Paurav Joshi on Apr 12, 2016 1:51 PM
    Mark Correct
    Correct Answer
    Good question! No, PI Datalink and PI Processbook do not automatically convert into two (or more) calls if the limit is hit. You typically do not experience this issue in PI Processbook because it use the SDK method Plotvalues which only retrieves enough information to accurately display the trend. A PI ProcessBook DataSet requires all the archived values so it does not use PlotValues and is subject to ArcHaxCollect.
    Like • Show 1 Like1
    Actions
    - Paurav Joshi @ Dan Fishman on Apr 12, 2016 2:29 PM
      Mark Correct
      Correct Answer
      That was very insightful Dan
      
      I have query that if I am developing custom application using PI Web API and I am fetching archive data of, as an example 6, assets for a month, then the query has to get 20 million around data. If my maxArcCollect is 10 million, then please advise me on how should I make query to get this data?
      Option 1: Increase the maxArcCollect
      Option 2: Divide the single query into two and call them concurrently
      
      Please suggest if we have options except above two.
      
      Regards,
      Paurav Joshi
      Like • Show 0 Likes0
      Actions
      - Dan Fishman @ Paurav Joshi on Apr 12, 2016 4:09 PM
        Mark Correct
        Correct Answer
        If you always know the data density and approximately how many events it might make sense to do concurrent chunks to provide some protection to your PI System instead of increasing maxArcCollect. If users often need 10 million events, then maybe increase this. In most cases, the Chunking is slightly difficult to determine appropriate sizes since data density is not known ahead and we don't really have a way to page on this yet.
        1 person found this helpful
        Like • Show 0 Likes0
        Actions
Paurav Joshi Apr 18, 2016 2:21 PM
Mark Correct
Correct Answer
Little information need to be added here.
I was facing this issue when I am fetching PI tag data using PI Web API via streams like,
https://{web_api_hostname}/piwebapi/streams/{webid_of_tag}/recorded?starttime=*-30d&endtime=*&maxcount=150000

For bulk read via streamset please follow this thread .

Cheers
Like • Show 0 Likes0
Actions
- Dan Fishman @ Paurav Joshi on Apr 18, 2016 3:32 PM
  Mark Correct
  Correct Answer
  Take a look at the MaxItemsPerCall configuration attribute and you can read about it here: PI Web API
  1 person found this helpful
  Like • Show 0 Likes0
  Actions

maxCount in PI WebAPI calls restricted

Outcomes

Related Content

Recommended Content