19 Replies Latest reply on Feb 3, 2016 10:38 AM by pthivierge

    Streamsets GetChannelAdHoc best practices

    ted@parijat.com
    ted@parijat.com Jan 27, 2016 5:44 PM

      If the goal is to continuously retrieve real-time updates from thousands of PI points, is using multitudes of GetChannelAdHoc WebSocket requests of 10s of points per request considered best practice or ill-advised?

        • Re: Streamsets GetChannelAdHoc best practices
          bshang
          bshang Employee Jan 28, 2016 12:46 AM (in response to ted@parijat.com)

          It is generally preferable to associate multiple PI points with one WebSocket channel. So you should use

           

          wss://myserver/piwebapi/streamsets/channel?webId={webId1}&webdId={webId2}

           

          and specify multiple webIds. The PI Web API Server will utimately retrieve these events from the PI Data Archive. Batching in points in one channel allows the web API server to get the updates in fewer roundtrips.

           

          As to the optimal number, I guess it would depend on many factors, but I would start with 10, then 100, etc. not going above 1000. If you get -414, then you may need to scale back the URL length or modify registry settings. Is there a URL length limit

           

           

          1 of 1 people found this helpful
            • Re: Streamsets GetChannelAdHoc best practices
              arosenthal
              arosenthal Jan 28, 2016 1:05 AM (in response to bshang)

              There seems to be some consensus that URL lengths should be less than 2000 characters in order to maintain compatibility across a wide range of browsers (see this thread: http://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers for more information). I'm not sure if this only affects HTTP requests or if it also affects WebSocket requests.

               

              Since WebSockets are persistent connections between client and server, it would make more sense to allow clients to send messages to the WebSocket server to ADD and REMOVE subscriptions to WebIds at will. This would avoid having to specify the requested WebIds in the URL parameters and would provide greater flexibility in building web applications. For example, I can see this feature coming in handy for building dynamic dashboards using AngularJS.

              1 of 1 people found this helpful
                • Re: Streamsets GetChannelAdHoc best practices
                  bshang
                  bshang Employee Jan 28, 2016 2:09 AM (in response to arosenthal)

                  Yeah that's a good point. In my haste to answer, I forgot that WSS is a completely different protocol than HTTP, so I'm not sure if the Windows OS sets any limits on that.

                   

                  Your suggestion is a good one regarding adding/removing subscriptions. The underlying implementation of WebSockets uses AFDataPipe on the web API server-side. However, aside from the initial set of attributes passed in, I don't think it's currently possible to add/remove attributes (e.g AF SDK's AFDataPipe.Add/RemoveSignups is not exposed). it can be inefficient to create and/or recreate new pipes each time. We'll bring this up as a possible enhancement to the server team. Another issue is AFDataPipe metadata changes which are currently not supported on the web API side as well.

                • Re: Streamsets GetChannelAdHoc best practices
                  ted@parijat.com
                  ted@parijat.com Jan 28, 2016 3:13 AM (in response to bshang)

                  I am currently using 10 points per websocket, 15 should generally be under 2k by the looks. Still, getting updates on thousands of points means potentially hundreds of websockets and I wasn't sure if the WebAPI was ever intended for that sort of use.

                    • Re: Streamsets GetChannelAdHoc best practices
                      arosenthal
                      arosenthal Jan 28, 2016 3:32 AM (in response to ted@parijat.com)

                      It's hard to find reliable information related to this topic, but from what I've read the general consensus seems to be that the performance implications of multiple WebSocket connections seems to be more of a server-side problem. That said, I have also found some sources that claim different browsers impose limits on the total number of WebSocket instances a client can have for each session and/or globally across all sessions. I've also read at least one thread that claims that opening multiple WebSocket connections (approaching 50) can really slow the browser down (however, this claim was not backed up with any concrete evidence or test results). There also seems to be a widely held belief in software development that each browser session should have a single WebSocket connection handling server-to-client communications in order to reduce overhead and complexity and improve performance. This is why I recommended above that PI Channels implement some method of adding or removing subscriptions with a single WebSocket connection by sending messages to the open connection.

                  • Re: Streamsets GetChannelAdHoc best practices
                    Roger Palmen
                    Roger Palmen Jan 28, 2016 10:25 AM (in response to ted@parijat.com)

                    Interesting topic! Monitoring 1000's of items through the WebAPI seems to be a recurring topic. Just hoping that in future we will see some improved ways to make this work. How about retrieving updates on the PI Integrator for BI's 'views' on PI data through the PI Web API? Then you would get updates to an entire dataset, instead of separate items, and you would have the benefit of isolation of PI SW from consumers (which is the major architectural advantage of the PI Web API to me).

                     

                    Back on topic:

                    In general, one might question when you reach the point that the REST model as exposed by the Web API does not fit your scenario enymore. In this case, scaling up the REST model to monitor 1000's of items comes with it's problems, so i'd recommend to look for other methods. E.g. a custom webservice on AFSDK or ODBC on the PI Integrator for BI.

                    • Re: Streamsets GetChannelAdHoc best practices
                      bshang
                      bshang Employee Jan 29, 2016 1:11 AM (in response to ted@parijat.com)

                      I'm actually curious about the use case here regarding getting updates from thousands of PI Points. What kind of application is this (UI, streaming analytics, message bus, etc.) and what does it do? The initial design for Channels was mostly for web UI dashboards (10's of tags) so it likely won't handle the scale imagined here well.

                       

                      If the application can run within the network, then an on-premise long-running AF SDK application will be more optimal. The classes you would use here are AFDataPipe or PIDataPipe. This is actually what PI Web API uses with Channels, but if running at scale, it will be more optimal to bypass Channels/WebSockets and jump to the lower level implementation. In addition, if this application is performing any sort of sliding window calculations, then you'd want to use AF SDK's AFDataCache, which Channels does not offer.

                      How to use the PIDataPipe or the AFDataPipe