One of our software components, is continuously retrieving snapshot updates for a long list of PIPoints using it's Latest-Markers, in a way that with each new snapshot update, we update the latest-marker of the tag (PIPoint) before the next RetrieveStreamSetUpdates itteration.
The thing is that sometimes we're hitting "CacheNotFound" and/or "MarkerNotFound" status from the RetrieveStreamSetUpdates (GET streamsets/updates) API.
I'm not exactly sure, what does "CacheNotFound" and "MarkerNotFound" status means and why do we get it, assuming that we always use the most latest-marker for each GET streamsets/updates call.
Now while, we found a way to leave with "MarkerNotFound" status, as it usually comes with a "source" attribute, so we know which tag we should try to re-register to get a new LatestMarker and overcome this issue on the next iteration, the thing with "CacheNotFound" status is way more complicated, as the "source" attribute of the JSON response is always empty. In such case we have no way to detect the "source" marker that caused this status... as the call to GET streamsets/updates is made with approximately 100-200 markers of different tags. The worst thing is while time pass by, we're starting to get CacheNotFound for all of the registered tags. So the only way to solve the issue, is to register all tags once again and get new pairs of Latest-markers.
I assume this is somehow related to a frequency rate of calling the GET streamsets/updates API, as sometimes when we reduce the API call frequency it helps to reduce the errors, but I'm not sure it's a fact or just a coincidence.
In anyway I would like to get your advice on how these statuses should be handled and what is the exact reason we get them.