-
Re: Streamsets GetChannelAdHoc best practices
bshangJan 28, 2016 12:46 AM (in response to ted@parijat.com)
1 of 1 people found this helpfulIt is generally preferable to associate multiple PI points with one WebSocket channel. So you should use
wss://myserver/piwebapi/streamsets/channel?webId={webId1}&webdId={webId2}
and specify multiple webIds. The PI Web API Server will utimately retrieve these events from the PI Data Archive. Batching in points in one channel allows the web API server to get the updates in fewer roundtrips.
As to the optimal number, I guess it would depend on many factors, but I would start with 10, then 100, etc. not going above 1000. If you get -414, then you may need to scale back the URL length or modify registry settings. Is there a URL length limit
-
Re: Streamsets GetChannelAdHoc best practices
arosenthal Jan 28, 2016 1:05 AM (in response to bshang)1 of 1 people found this helpfulThere seems to be some consensus that URL lengths should be less than 2000 characters in order to maintain compatibility across a wide range of browsers (see this thread: http://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers for more information). I'm not sure if this only affects HTTP requests or if it also affects WebSocket requests.
Since WebSockets are persistent connections between client and server, it would make more sense to allow clients to send messages to the WebSocket server to ADD and REMOVE subscriptions to WebIds at will. This would avoid having to specify the requested WebIds in the URL parameters and would provide greater flexibility in building web applications. For example, I can see this feature coming in handy for building dynamic dashboards using AngularJS.
-
Re: Streamsets GetChannelAdHoc best practices
bshangJan 28, 2016 2:09 AM (in response to arosenthal)
Yeah that's a good point. In my haste to answer, I forgot that WSS is a completely different protocol than HTTP, so I'm not sure if the Windows OS sets any limits on that.
Your suggestion is a good one regarding adding/removing subscriptions. The underlying implementation of WebSockets uses AFDataPipe on the web API server-side. However, aside from the initial set of attributes passed in, I don't think it's currently possible to add/remove attributes (e.g AF SDK's AFDataPipe.Add/RemoveSignups is not exposed). it can be inefficient to create and/or recreate new pipes each time. We'll bring this up as a possible enhancement to the server team. Another issue is AFDataPipe metadata changes which are currently not supported on the web API side as well.
-
-
Re: Streamsets GetChannelAdHoc best practices
ted@parijat.com Jan 28, 2016 3:13 AM (in response to bshang)I am currently using 10 points per websocket, 15 should generally be under 2k by the looks. Still, getting updates on thousands of points means potentially hundreds of websockets and I wasn't sure if the WebAPI was ever intended for that sort of use.
-
Re: Streamsets GetChannelAdHoc best practices
arosenthal Jan 28, 2016 3:32 AM (in response to ted@parijat.com)It's hard to find reliable information related to this topic, but from what I've read the general consensus seems to be that the performance implications of multiple WebSocket connections seems to be more of a server-side problem. That said, I have also found some sources that claim different browsers impose limits on the total number of WebSocket instances a client can have for each session and/or globally across all sessions. I've also read at least one thread that claims that opening multiple WebSocket connections (approaching 50) can really slow the browser down (however, this claim was not backed up with any concrete evidence or test results). There also seems to be a widely held belief in software development that each browser session should have a single WebSocket connection handling server-to-client communications in order to reduce overhead and complexity and improve performance. This is why I recommended above that PI Channels implement some method of adding or removing subscriptions with a single WebSocket connection by sending messages to the open connection.
-
-
-
Re: Streamsets GetChannelAdHoc best practices
Roger Palmen Jan 28, 2016 10:25 AM (in response to ted@parijat.com)Interesting topic! Monitoring 1000's of items through the WebAPI seems to be a recurring topic. Just hoping that in future we will see some improved ways to make this work. How about retrieving updates on the PI Integrator for BI's 'views' on PI data through the PI Web API? Then you would get updates to an entire dataset, instead of separate items, and you would have the benefit of isolation of PI SW from consumers (which is the major architectural advantage of the PI Web API to me).
Back on topic:
In general, one might question when you reach the point that the REST model as exposed by the Web API does not fit your scenario enymore. In this case, scaling up the REST model to monitor 1000's of items comes with it's problems, so i'd recommend to look for other methods. E.g. a custom webservice on AFSDK or ODBC on the PI Integrator for BI.
-
Re: Streamsets GetChannelAdHoc best practices
ted@parijat.com Jan 28, 2016 5:45 PM (in response to Roger Palmen)It's true, the WebAPI may not be the right tool for this job so I may have to look more in to the other PI SDK options for the large scale usage of data.
-
Re: Streamsets GetChannelAdHoc best practices
Roger Palmen Jan 29, 2016 9:17 AM (in response to ted@parijat.com)PISDK is deprecated. Definitely go for the much faster AFSDK!
-
-
Re: Streamsets GetChannelAdHoc best practices
arosenthal Jan 28, 2016 10:29 PM (in response to Roger Palmen)I also thought I'd mention that the length of WebIds seems to get longer the further you go down in depth of an element/attribute tree. This makes sense since it is my understanding that the WebId encodes the unique ID of the element as well as the full path. However, this means that the method of requesting information from multiple WebIds using the AdHoc functions does not scale well because of the URL length limitation. One could easily approach this limit with less than 15 WebIds. Just to be clear, I'm not talking about monitoring 1000's of items here (which I agree would probably best be solved with a separate AF SDK solution).
The new Bulk Requests CTP seems to have solved this problem since you can send JSON information containing which items you want to get data for. It would be nice if there was something similar available for the Channels CTP.
-
Re: Streamsets GetChannelAdHoc best practices
bshangJan 29, 2016 1:15 AM (in response to Roger Palmen)
1 of 1 people found this helpfulExposing RESTful views of the PI Integrator for BA is a cool idea. I've always thought this should be done, as ODBC/JDBC are already available. RESTful-exposure for BA views would also hook them up more easily to a lot of cloud analytics frameworks that call an HTTP endpoint and process a "data frame" structured response.
-
Re: Streamsets GetChannelAdHoc best practices
Roger Palmen Jan 29, 2016 9:15 AM (in response to bshang)Yes, exactly my line of thought. Most of it should already be there in the form of the PI SQL Data Access Server.
-
Re: Streamsets GetChannelAdHoc best practices
I.Berry Feb 1, 2016 9:39 PM (in response to bshang)This is a good idea. If you do it, please make it oData
-
-
-
Re: Streamsets GetChannelAdHoc best practices
bshangJan 29, 2016 1:11 AM (in response to ted@parijat.com)
I'm actually curious about the use case here regarding getting updates from thousands of PI Points. What kind of application is this (UI, streaming analytics, message bus, etc.) and what does it do? The initial design for Channels was mostly for web UI dashboards (10's of tags) so it likely won't handle the scale imagined here well.
If the application can run within the network, then an on-premise long-running AF SDK application will be more optimal. The classes you would use here are AFDataPipe or PIDataPipe. This is actually what PI Web API uses with Channels, but if running at scale, it will be more optimal to bypass Channels/WebSockets and jump to the lower level implementation. In addition, if this application is performing any sort of sliding window calculations, then you'd want to use AF SDK's AFDataCache, which Channels does not offer.
-
Re: Streamsets GetChannelAdHoc best practices
Roger Palmen Jan 29, 2016 9:17 AM (in response to bshang)I think with any enterprise-level dashboard you easily reach 1000s of streams. Take 50 wind turbines, 20 numbers each, and you already have 1000. And that is still small scale...
-
Re: Streamsets GetChannelAdHoc best practices
bshangJan 29, 2016 3:55 PM (in response to Roger Palmen)
That's a good point, but would all streams need to be signed up at once? I was thinking WebSockets would be created in a more lazy fashion depending on UI selection, although it comes at a cost of responsiveness. For 1000+ streams though and enterprise-level of users, another application server is probably needed to serve the data (e.g. Coresight). PI Web API, as a general RESTful server, likely will not perform as well even if its WebSocket implementation were optimized.
-
Re: Streamsets GetChannelAdHoc best practices
pthiviergeFeb 1, 2016 2:43 PM (in response to Roger Palmen)
In my opinion, to do efficient reporting and dashboarding, you need to create tags to host at least some KPIs at the level you are looking at. e.g. Total Generated Power : lvl1 per turbine, lvl2 per farm, lvl3 per country, etc.
Then tags displayed and signed-up are those who are relevant to this specific level.
-
Re: Streamsets GetChannelAdHoc best practices
Roger Palmen Feb 1, 2016 7:27 PM (in response to pthivierge)Yes, i agree that you need some rollups in analytics, but it's not very uncommon to have a portal for e.g. a number of windparks that needs to display to 50 to 100 concurrent users current status. In detail. Because of the complex authorisation requirements, the security is handled outside of PI, so we can't aggregate generically. Every user can have different visibility on the assets. So you still need a webservice that delivers you frequent updates of 1000's of attributes.
-
Re: Streamsets GetChannelAdHoc best practices
pthiviergeFeb 3, 2016 10:38 AM (in response to Roger Palmen)
Interesting use case Roger, that sounds more complex than my original thought indeed. There are certainly some strategies that could applied in such scenarios (e.g. using PI Web API build in Authentication withe Kerberos, leverage its internal cache, etc.) . That would go over the scope of original question I believe .
Thanks for bringing this up, always interesting!
-
-
-
-
Re: Streamsets GetChannelAdHoc best practices
ted@parijat.com Jan 29, 2016 4:25 PM (in response to bshang)The application is streaming analytics. I'll look in to the data pipes in AF SDK. Thanks everyone for the input.
-