First, the results:
When the Channels controller was introduced we gave you the ability to open a direct stream of PI data over a WebSockets connection. Since then, questions have formed around how far this technology can be stretched in scope. We have wondered for ourselves how far we can push WebSockets and have results for you.
We found a noticeable degradation in performance with two particular load types.
For 200 concurrent users with one subscription each, each subscription listening to 1,000 AF Attributes, performance began to decline where it was perceptible.
For a single user (such as a service account) creating subscriptions for a single AF Attribute, degradation in performance was observed after 2,500 channels were instantiated.
What we've discovered
Our testbed reveals the CPU and Memory usage of Channels scaling roughly linearly with the number of subscriptions/channels.
However, the performance of Channels certainly shows room for improvement; especially in the case of multiple channels. The use of multiple channels provides far less throughput than that of a single channel with an equivalent number of subscriptions.
One potential where improvement that could be made is in the serialization process. This requires the whole message be serialized to a string before sending. The web socket implementation used on the server does provide a way to send large messages in “chunks”, which could be used to avoid large allocations and hopefully reduce memory usage. The scheduling of the underlying AF calls could also potentially be improved in order to reduce CPU usage.
How we performed the tests
Two experiments were performed to collect data on the performance of Channels. For both experiments, the results were collected using Performance Monitor, which was configured with the “% Processor Time” and “Private Bytes” performance counters for the “OSIsoft.REST.Host.exe” process. The PI Web API instance itself was hosted on a HyperV virtual machine with the following specifications:
• Operating System: Windows Server 2016 Standard
• Virtual Processors: 6 Cores
• Memory: Dynamic 8GB (8192 MB)
• Hard Drive: 100 GB
For the first experiment, varying numbers of unique users were tested with a subscription of a single element with 1,000 attributes. The source PI Points generated data with a 1 Hz frequency. This test was intended to show how PI Web API performed under load with multiple unique user identities. The stream set’s “GetChannel” endpoint was used on several AF elements, each of which had 1,000 attributes, all referring to a unique PI Point. This PI Point was updated with values approximately every second to provide data for the channel.
For the second experiment, a single user was tested with varying numbers of channels created with a single subscription each. This test was intended to show how PI Web API performed under load with a single user identity. The stream’s “GetChannel” endpoint was used on the same unique PI Points as the first experiment.
Authentication is perhaps the most severe limitation of Channels right now. Currently, Channels uses the same authentication schemes as normal HTTP requests. This is done for simplicity and consistency. While the web socket specification permits these authentication schemes, browser support for them is still limited. The main problem is the HTTP challenge mechanism. Basic, Kerberos, and Bearer authentication schemes require an “Authorization” header in the HTTP request with information about the user’s credentials.
Here, a browser would normally respond by prompting the user for their name and password; then generating the proper “Authorization” header. But none of the tested browsers displayed this prompt for web socket connections and merely failed the request. One workaround for this limitation is to first make a normal HTTP request so the browser can cache the “Authorization” header and use it for subsequent web socket requests. This workaround was successful for all of the tested browsers.
If Single Sign-On is enabled, this “Authorization” header can be generated from the user’s current Windows account, so no user input is necessary. When it is not enabled the browser must prompt the user for credentials. In these cases the same workaround for Basic also works for Kerberos; a normal HTTP request can be made beforehand so the browser can cache the “Authorization” header.
Kerberos is not supported by default in Firefox (and sometimes Chrome), but it can be enabled using some configuration options. See the OSIsoft Knowledge Base article KB01223 for instructions on how to configure Kerberos.
For the Bearer scheme this “Authorization” header needs to be added programmatically. The browser will not cache or automatically generate it. Unfortunately the web socket client API defined in the HTML specification (which is the interface most browsers implement) does not allow adding headers to the initial HTTP handshake request for security reasons. As a result, Bearer authentication cannot currently be used in a browser with Channels. However, web socket clients that do support adding headers manually can still use Bearer authentication. For example, the C# ClientWebSocket class supports this functionality:
Internet Zones (MSIE and Edge)
Internet Explorer and Edge both disallow web socket connections between “internet zones” by default. In these cases, the following error messages will be logged in the console:
• Internet Explorer: “SCRIPT5022: SecurityError”
• Edge: “SCRIPT12017: WebSocket Error: SECURITY_ERR, Cross zone connection not allowed”
This issue can be resolved using the following steps on the client machine:
- Open “Internet Options” in Windows
- Click the “Security” tab
- Click the “Trusted Sites” zone
- Click the “Sites” button
- Add the domain name of the PI Web API server to the list
This will allow inter-zone web socket connections to the PI Web API server specified.
I really need the benefits of WebSockets. Can I get around these limitations by scaling horizontally?
You can certainly explore using a WebSockets-friendly traffic director in either hardware or software if your project comes close to, but just above, the limitations that we have discovered here. You might want to check out NGINX WebSockets-friendly traffic direction to load-balance multiple instances of PI Web API.