1 of 1 people found this helpful
The PIPagingConfiguration determines how the data is returned to the client. The data is returned in chunks to the client based on this configuration, however it can be consumed as a single stream by the client. You can configure it to make more/fewer trips to the server by changing the page size.Ryan Gilbert has a good explanation of PIPagingConfiguration. From his answer there:
The PI Server will never return partial results for a single tag even if you page by Event Count; it will always finish collecting events for the current tag even if that threshold has been reached.
That means you have to be careful if the tags you are querying are potentially dense. You can still get an error or a timeout even if you've specified conservative paging parameters.
1 of 1 people found this helpful
Here are some details regarding the behavior of the bulk data methods.
When a bulk RecordedValues call is made, the AF SDK partitions the points in the PIPointList by PI Data Archive (Server). Then, if the PI Data Archive is 3.4.390 (2012) or higher, it will execute a bulk call (one RPC) per PI Data Archive. If the version is less than 390, then the AF SDK will internally make an RPC call per PIPoint.
For now, I'll assume that the PI Data Archive is 3.4.390+ and the PIPointList contains only points from this one server.
Regarding how the data is returned to the client, the data is "paged" back to the client. When a PIPointList.RecordedValues call is made, an internal Task is created that executes the bulk RPC query. This task receives the paged results from the PI Data Archive, does any post-processing (i.e. UOM conversion), and adds the results to a .NET ConcurrentQueue. This queue is then exposed to the AF SDK client as an IEnumerable<AFValues> collection.
The client application simply iterates over and consumes the resulting collection in a separate thread. All the work of the page processor task is hidden from the end user so the developer only needs to iterate over the collection to obtain the results. If the collection is empty by the time the next page is requested, then the foreach loop will "block" until the next page arrives from the server, is processed, and is added to the queue/collection.
The AF SDK programmer's guide has a good discussion on these topics in the "List / Bulk Data Methods Overview" section. The thread AF SDK performance, serial vs. parallel vs. bulk is also worth reading to get an idea of bulk data retrieval behaviors. Also, this presentation is worth going over to find out more about the different data retrieval patterns.
Regarding ArcMaxCollect, for server 380 or higher, the defaults were changed to 1,500,000. We generally don't recommend increasing this value from the 1.5 million default unless there is a need to, as it exists to prevent overloading the network and server.
In regards to your last question about retrieving data in a "streaming" fashion, are you looking to obtain real-time events from the PI DataArchive? If so, then AFDataPipe should be used instead of RecordedValues (which is more for historical data access). The presentation linked above as well as the AF SDK programmer's guide have more details regarding getting real-time data.
Thanks so much - I will read the AF SDK programmer's guide (if I can poke around and find it). I am definitely wanting historical data, not near real-time - and sounds like the data may be "streaming" already given that the foreach will block for the next read.
Ultimately I need to sort the data by time for a set of points, but I will read your documentation references and see what I learn.
No problem. The AF SDK guide should be in %pihome%\HELP\AFSDK.chm but please let us know if you have any questions!