implementing an AF interop wrapper I stumble about something I don't know ist a bug or my mistake.
Searching for Event Frames has a maxCount of 1000
But the resultlist shows 1165 results.
The search seems to ignore the Maxcount Parameter.
Is this a known bug or is my interpretation of a paged maxCount wrong?
Greetings
Wolfgang
Oct 23, 2018 7:41 AM
Hi Wolfgang,
The parameter in question for the AFSearch.FindEventFrames method is actually to set the returned collection page size, not the maximum number of objects returned by the search. This particular search returns a paged collection, so this parameter sets the size of each page. Refer to the remarks section of AFGlobalSettings.CollectionPageSize for more information about this.
Cheers,
John
Hello John,
your link says the following as expected...
"Returns the number of items on a page of a paged collection. The default value is 1000."
Maybe i have a wrong understanding.
My startIndex is 0 and maxCount is 1000. So I expect a resultset of maximum 1000 items (index 0-999).
If the resultset has 1000 items I would call for startIndex 1001 to ... for the next page.
But as you see the first page has 1165 items on the first page despite a startindex 0 and maxCount 1000 on my query.
I'am confused
Wolfgang
Oct 23, 2018 9:28 AM
Hi Wolfgang,
Paging is for data retrieval - 'pages' of data are sent to the client as a means of chunking a large dataset retrieved from the server into smaller datablocks for sending back to the caller. It's a strategy for optimising the return of data over the wire. In this particular instance, search results are sent to the client in pages of 1000 objects, but the final result set contained in your IEnumerable after the AFSDK call has completed will be the entire search results (1165) based on your search criteria. IEnumerable and IList objects don't have 'pages' of objects. If you want a 'paged' view in your client, you would have to implement your own logic to extract data from the List object as pages with a specific object count per page.
What you are seeing is the correct behaviour for this method call.
Cheers,
John
Thank you for explaining this.
It is a little bit strange for me because frameworks i used before eighter send a "page" as a requested part once per call or "fromindex, and itemcount.
This includes PIAPI and many more. What is the users benefit of the behaviour above if one cannot control the pages? If one request data mistakenly without or with a
wrong filter the resultset has millions of items and it maybee freezes the client. If I cannot control the amount at query the massive server and network load is not avoidable.
I take it, but can't see the logic.
Thank you
Wolfgang
Oct 23, 2018 10:34 AM
I think that where you want to return a specific number of items, in this case Event Frames, perhaps a call such as AFEventFrame.FindEventFrames is more appropriate. Note however that this doesn't use the newer Search functionality, but I think may return a result set more in line with what you were expecting.
The AFEventFrameSeach class method of FindEventFrames specifically noters that it returns "an enumerable list of the event frames found using the search tokens for this search object", whereas AFEventFrame.FindEventFrames returns "the collection containing the specified page of AFEventFrame objects which match the specified query string". I see enough difference in those descriptions to determine that the returned results of each method is quite different.
Yes, I agree there is potential for the client to run out of memory or exhibit other undesirable behaviours if a significantly large dataset is returned from using the AFEventFrameSearch class. I suspect that you would need to implement your own strategies to mitigate such an outcome.
Oct 23, 2018 2:00 PM
You really don't want to use the older obsolete AFEventFrame.FindEventFrames method. One reason is it will not handle evaluation of attribute value filters the way you would expect since those filters must be done in the client if they have a data reference defined and you will typically get less items than the number you requested. Since the new searches return an IEnumerable, you should just exit out of your foreach loop when you get the number of items that you desire or if moving into a List then use the Enumerable Take method to take the first N results.
By the way. I see the topic i wrote is wrong. It should be AFEventFrameSearch not AFElementSearch. Is there a moderator who can change this?
Oct 23, 2018 10:47 AM
If you are going to use paging and you expect more than one page of results. You might want to opt into caching for your search by setting the AFSearch.CacheTimeout Property.
Regarding paging, the client application simply needs to iterates over and consumes the resulting collection in a foreach loop. All the work of the page processor task is hidden from the end user so the developer only needs to iterate over the collection to obtain the results. If the collection is empty by the time the next page is requested, then the foreach loop will "block" until the next page arrives from the server, is processed, and is added to the queue/collection. You can experiment with the page size to try to optimize your performance.
Check this thread for more info about this pattern.
https://pisquare.osisoft.com/thread/8329
The pattern that you mention about max count normally has a cookie involved in the parameter such as
PIServer.FindChangedPIPoints Method so that it can remember the last position in the search.
Oct 23, 2018 11:19 AM
That's a good point Eugene. What I think happened in the code snippet originally posted is that the creation of a new List object from the IEnumerable returned by the FindEventFrames method is what results in the new object showing all the search results in a single collection. The SDK method returns an IEnumerable so that results can be streamed as pages of data and iterated over, as Eugene described. By converting the IEnumerable to a List, you are pulling out whatever data is currently in the IEnumerable into a fixed non-paged collection - fixed in that the number of items won't change unless you explicitly Add or Remove items from the collection.
Thank you all,
i think i have to digest the information and decide how i handle this.
Grreetings
Wolfgang
Oct 23, 2018 1:28 PM
Here's a nice plug for the work Rick Davin did to help put a nice bow on all this AF Search discussion. I find this to be the definitive source:
https://www.osisoft.com/Presentations/LiveCoding--Getting-the-Most-Out-of-the-New-AFSearch/
--Vince
Hello Vince,
great thanks for that link.
That tutorial helped me understanding what happened.
I had not noted the concept of lazy loading before.
So now it is clear that converting the IEnumerable to List<T> forces the seach to send all pages.
Greetings
Wolfgang
