The only hardware sizing recommendation that we currently can offer is the Hardware and PI System Sizing Recommendations Spreadsheet that has some recommendation for PI Coresight depending on the amount of users but I doubt this will be very helpful in the situation you describe.
What could help more is to collect information about the PI Web API usage e.g. what kind of calls are executed and how many of them for how many data items within what period of time. Capturing a few hang dumps in a row can help to identify memory leaks but defining the period between taking hang dumps usually requires having a doubt about what's happening or an observation of how memory usage changes over time.
Probably the best recommendation for now is, please make sure that the latest release of PI Web API is installed and to verify it's also the latest PI AF Client installed because PI web API uses AF SDK under the hood. If the suspicion for a memory leak grows, please contact OSIsoft Technical Support.
Can you share the actual memory usage? What is the version of PI Web API? Recent versions have some improvements to mitigate memory usage.
In general though, the memory usage can be expected to be high if making lots of unrelated queries. The backend layer (AF SDK) relies on a cache to improve performance and response times. High memory usage can be undesirable if secondary storage becomes used, but can greatly improve performance otherwise.
Hi Gregor, Barry,
Thanks for your quick thoughts and prompt responses.
Yes, the spreadsheet was my first thought too, however not quite helpful in this case...
Basically, PI Web API is used for an integration of a cloud solution with on-prem PI. The cloud is requesting data for about 100K PI tags with 1-15 mins frequency, using snapshot, summary and interpolated calls for subsets of the tags.
The pattern is; StreamSets of 100-1000 tags bulked into few batch calls.
PI Web API is currently taking up 5-7 GB of RAM and I suspect it might be the right amount for the data being processed, but as I mentioned I have no point of reference to support my suspicions or prove them wrong.
With regards to versions, it's the latest PI Web API 2015 R3. I can check the AF SDK versions, but would suspect PI Web API 2015 R3 install kit should have installed the latest AF Client as well.
Thanks for the information. This amount of memory used is expected, although it is high. The cache settings can be changed via the configuration element for PI Web API in PI System Explorer by creating an attribute called AFCacheMaxObjects. By default, it is 10000. However, I would recommend not changing these defaults unless there is a performance issue associated with memory spilling to secondary storage.
What is the authentication mechanism used and how many users are there? Another factor contributing to memory is if there are many users connected via Basic or Kerberos. Then, PI Web API will cache objects for each user instead of sharing them across users.
Basic authentication is in use, and there is only a single user to PI Web API - the interface on the cloud. However, MTA (multi threaded async calls) are used within the interface to request data. PI Web API is currently running at about 6.5GB RAM, which at the current server spec is at the max limit of available RAM, therefore PI Web API occasionally crashes and requires restart.
I was believing that I provided the link to KB00046 - How to get a hang or crash dump and how to obtain and use debugging tools earlier but just recognized I missed that detail. I know with 6.5 GB memory usage, a crash dump will be huge but Technical Support has a ftp server for uploading such files.
What's the message in the Application Event Log that belongs to this application failure?
Is it PI Web API or the custom application that is crashing?
How many asynchronous calls are executed in parallel?
What's the client object used? Is the client object frequently instantiated and disposed? Please see this related discussion for the HttpClient http://stackoverflow.com/questions/15705092/do-httpclient-and-httpclienthandler-have-to-be-disposed
Please allow me to repeat my suggestion to work with OSIsoft Technical Support.
As I mentioned in my opening post, the question is not the crashing, but rather what is the required hardware.
We have to be able to estimate the hardware for the new Production servers hosting PI Web API and we as providers are being asked the question of hardware spec for the servers.
Can you please help by providing some guidance/metrics on how can we estimate required hardware spec (CPU type, speed, number of cores, RAM, HDD) for a server hosting PI Web API. Would be absolutely fantastic to get that included into the sizing excel spreadsheet.
PI Web API is considered having a relative small memory footprint but as usual thinks depend on e.g. the amount of users, because caching is per user and for sure as well on other metrics.
You mentioned that you are using Batch service. Batch allows to bundle different related and unrelated calls together what helps to reduce network latency by building and sending larger packages. While larger packages help reducing the impact of network latency, there's likely also the chance that packages grow too large. Adding more RAM as a synonym for buying larger trucks can help but at a certain point the affect may turn because the street infrastructure turns out not being built for huge trucks.
At this point, please allow me to remind that Batch service and Channel service are new, not yet released functionalities. Both is included as Customer Technology Preview (CTP) with the current release. Please see PI Web API 2015 R3 Release Notes for reference. We have very few experience with Batch and Channel service at this point and may not have had your particular use case in mind during the design phase. Because of this development and product management are hungry for your experience with CTP functionality.
Thanks for that. I would recommend increasing the RAM to at least 10GB. Long-running applications that use AF SDK (like PI Web API) can use up a lot of memory.