If the goal is to continuously retrieve real-time updates from thousands of PI points, is using multitudes of GetChannelAdHoc WebSocket requests of 10s of points per request considered best practice or ill-advised?
If the goal is to continuously retrieve real-time updates from thousands of PI points, is using multitudes of GetChannelAdHoc WebSocket requests of 10s of points per request considered best practice or ill-advised?
It is generally preferable to associate multiple PI points with one WebSocket channel. So you should use
wss://myserver/piwebapi/streamsets/channel?webId={webId1}&webdId={webId2}
and specify multiple webIds. The PI Web API Server will utimately retrieve these events from the PI Data Archive. Batching in points in one channel allows the web API server to get the updates in fewer roundtrips.
As to the optimal number, I guess it would depend on many factors, but I would start with 10, then 100, etc. not going above 1000. If you get -414, then you may need to scale back the URL length or modify registry settings. Is there a URL length limit
There seems to be some consensus that URL lengths should be less than 2000 characters in order to maintain compatibility across a wide range of browsers (see this thread: http://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers for more information). I'm not sure if this only affects HTTP requests or if it also affects WebSocket requests.
Since WebSockets are persistent connections between client and server, it would make more sense to allow clients to send messages to the WebSocket server to ADD and REMOVE subscriptions to WebIds at will. This would avoid having to specify the requested WebIds in the URL parameters and would provide greater flexibility in building web applications. For example, I can see this feature coming in handy for building dynamic dashboards using AngularJS.
Yeah that's a good point. In my haste to answer, I forgot that WSS is a completely different protocol than HTTP, so I'm not sure if the Windows OS sets any limits on that.
Your suggestion is a good one regarding adding/removing subscriptions. The underlying implementation of WebSockets uses AFDataPipe on the web API server-side. However, aside from the initial set of attributes passed in, I don't think it's currently possible to add/remove attributes (e.g AF SDK's AFDataPipe.Add/RemoveSignups is not exposed). it can be inefficient to create and/or recreate new pipes each time. We'll bring this up as a possible enhancement to the server team. Another issue is AFDataPipe metadata changes which are currently not supported on the web API side as well.
I am currently using 10 points per websocket, 15 should generally be under 2k by the looks. Still, getting updates on thousands of points means potentially hundreds of websockets and I wasn't sure if the WebAPI was ever intended for that sort of use.
It's hard to find reliable information related to this topic, but from what I've read the general consensus seems to be that the performance implications of multiple WebSocket connections seems to be more of a server-side problem. That said, I have also found some sources that claim different browsers impose limits on the total number of WebSocket instances a client can have for each session and/or globally across all sessions. I've also read at least one thread that claims that opening multiple WebSocket connections (approaching 50) can really slow the browser down (however, this claim was not backed up with any concrete evidence or test results). There also seems to be a widely held belief in software development that each browser session should have a single WebSocket connection handling server-to-client communications in order to reduce overhead and complexity and improve performance. This is why I recommended above that PI Channels implement some method of adding or removing subscriptions with a single WebSocket connection by sending messages to the open connection.
Interesting topic! Monitoring 1000's of items through the WebAPI seems to be a recurring topic. Just hoping that in future we will see some improved ways to make this work. How about retrieving updates on the PI Integrator for BI's 'views' on PI data through the PI Web API? Then you would get updates to an entire dataset, instead of separate items, and you would have the benefit of isolation of PI SW from consumers (which is the major architectural advantage of the PI Web API to me).
Back on topic:
In general, one might question when you reach the point that the REST model as exposed by the Web API does not fit your scenario enymore. In this case, scaling up the REST model to monitor 1000's of items comes with it's problems, so i'd recommend to look for other methods. E.g. a custom webservice on AFSDK or ODBC on the PI Integrator for BI.
It's true, the WebAPI may not be the right tool for this job so I may have to look more in to the other PI SDK options for the large scale usage of data.
PISDK is deprecated. Definitely go for the much faster AFSDK!
I also thought I'd mention that the length of WebIds seems to get longer the further you go down in depth of an element/attribute tree. This makes sense since it is my understanding that the WebId encodes the unique ID of the element as well as the full path. However, this means that the method of requesting information from multiple WebIds using the AdHoc functions does not scale well because of the URL length limitation. One could easily approach this limit with less than 15 WebIds. Just to be clear, I'm not talking about monitoring 1000's of items here (which I agree would probably best be solved with a separate AF SDK solution).
The new Bulk Requests CTP seems to have solved this problem since you can send JSON information containing which items you want to get data for. It would be nice if there was something similar available for the Channels CTP.
I see that your post here is 6 years old. It seems that making ad-hoc queries from a client to open a socket connection has not changed. Maybe one thing has changed: the option to use shorter versions of the WebID, which would make it possible to squeeze in more items.
Actually, what would be awesome is to be able to use wildcards in the request such as all tags that include the name "pump" and "pressure".
Exposing RESTful views of the PI Integrator for BA is a cool idea. I've always thought this should be done, as ODBC/JDBC are already available. RESTful-exposure for BA views would also hook them up more easily to a lot of cloud analytics frameworks that call an HTTP endpoint and process a "data frame" structured response.
Yes, exactly my line of thought. Most of it should already be there in the form of the PI SQL Data Access Server.
This is a good idea. If you do it, please make it oData
I'm actually curious about the use case here regarding getting updates from thousands of PI Points. What kind of application is this (UI, streaming analytics, message bus, etc.) and what does it do? The initial design for Channels was mostly for web UI dashboards (10's of tags) so it likely won't handle the scale imagined here well.
If the application can run within the network, then an on-premise long-running AF SDK application will be more optimal. The classes you would use here are AFDataPipe or PIDataPipe. This is actually what PI Web API uses with Channels, but if running at scale, it will be more optimal to bypass Channels/WebSockets and jump to the lower level implementation. In addition, if this application is performing any sort of sliding window calculations, then you'd want to use AF SDK's AFDataCache, which Channels does not offer.
I think with any enterprise-level dashboard you easily reach 1000s of streams. Take 50 wind turbines, 20 numbers each, and you already have 1000. And that is still small scale...
That's a good point, but would all streams need to be signed up at once? I was thinking WebSockets would be created in a more lazy fashion depending on UI selection, although it comes at a cost of responsiveness. For 1000+ streams though and enterprise-level of users, another application server is probably needed to serve the data (e.g. Coresight). PI Web API, as a general RESTful server, likely will not perform as well even if its WebSocket implementation were optimized.
In my opinion, to do efficient reporting and dashboarding, you need to create tags to host at least some KPIs at the level you are looking at. e.g. Total Generated Power : lvl1 per turbine, lvl2 per farm, lvl3 per country, etc.
Then tags displayed and signed-up are those who are relevant to this specific level.
Yes, i agree that you need some rollups in analytics, but it's not very uncommon to have a portal for e.g. a number of windparks that needs to display to 50 to 100 concurrent users current status. In detail. Because of the complex authorisation requirements, the security is handled outside of PI, so we can't aggregate generically. Every user can have different visibility on the assets. So you still need a webservice that delivers you frequent updates of 1000's of attributes.
Interesting use case Roger, that sounds more complex than my original thought indeed. There are certainly some strategies that could applied in such scenarios (e.g. using PI Web API build in Authentication withe Kerberos, leverage its internal cache, etc.) . That would go over the scope of original question I believe .
Thanks for bringing this up, always interesting!
The application is streaming analytics. I'll look in to the data pipes in AF SDK. Thanks everyone for the input.
I'm actually curious about the use case here regarding getting updates from thousands of PI Points. What kind of application is this (UI, streaming analytics, message bus, etc.) and what does it do? The initial design for Channels was mostly for web UI dashboards (10's of tags) so it likely won't handle the scale imagined here well.
If the application can run within the network, then an on-premise long-running AF SDK application will be more optimal. The classes you would use here are AFDataPipe or PIDataPipe. This is actually what PI Web API uses with Channels, but if running at scale, it will be more optimal to bypass Channels/WebSockets and jump to the lower level implementation. In addition, if this application is performing any sort of sliding window calculations, then you'd want to use AF SDK's AFDataCache, which Channels does not offer.
How to use the PIDataPipe or the AFDataPipe