5 Replies Latest reply on Jul 18, 2017 6:06 PM by czhang

    PI Vision and load balancing issues

    jagdish.konathala

      We have Big-IP F5 fronting both the PI Vision member nodes. We tried various configurations in F5 and we see that the load is being distributed evenly between both nodes via F5. We see a lot of error messages no rhyme and reason. Some times, we don't see any errors for hours together and sometimes we see a flood of errors. We have a case with tech support and we were told OSIsoft does not support F5, so we are on our own.

       

      We tried with source and destination persistence in F5. We get no errors when we have F5 sending all requests to just one member though that defeats the purpose of load balancing across 2 or more nodes as we grow. Any insight is appreciated.

       

       

      These are the error messages we get when we download them. Messages seem generic and we don't see much in event viewer logs or from the PI side.

       

      Timestamp: 7/15/2017, 3:38:24 PM

      Severity: Error

      Message: Communication Error

      Details: Requested Range Not Satisfiable

       

      Timestamp: 7/15/2017, 3:38:29 PM

      Severity: Error

      Message: Data Error

      Details: Internal Server Error

       

      Timestamp: 7/15/2017, 3:38:29 PM

      Severity: Error

      Message: Communication Error

      Details: Requested Range Not Satisfiable

       

      Timestamp: 7/15/2017, 3:38:32 PM

      Severity: Error

      Message: Internal Server Error

       

      Update : I found a pattern that I was able to consistently replicate. I don't see the errors on machines that are always connected to the internet (desktops or kiosk displays for instance). Laptops on the other hand generate errors when put to sleep causing network to disconnect. Upon waking the laptop, the same displays start flooding the error messages till they are refreshed. No more errors till the internet is connected and issues repeat whenever network disconnected and reconnected (meeting runs). It can be an annoyance for the end user and IT tickets for the same! Is it related to source or destination persistence on the F5 configuration side?

       

      Message was edited by: Jagdish Konathala : Added pattern observed during network disruption

        • Re: PI Vision and load balancing issues
          LalBabuShaik

          Hi Jagdish, Could you please check IIS logs on PI Vision server when the request was initiated and terminated? I would also request network team to check using fidler to trace the request from the client and any specific errors.

            • Re: PI Vision and load balancing issues
              jagdish.konathala

              Here are some IIS logs. These are from a display that has AF elements and PI points. Replaced IPs with F5 naming for clarity. IIS logs don't seem to be giving out much. I am working with network team, trying to find any other info. I found a pattern though, update my initial question with that info.

              2017-07-17 23:29:02 10.10.1.1 GET / - 443 - F5_IPAddress - - 200 0 0 0

              2017-07-17 23:29:02 10.10.1.1 GET / - 443 - F5_IPAddress3 - - 200 0 0 0

              2017-07-17 23:29:02 10.10.1.1 POST /PIVision/Data/0fc29e2b-d398-42f1-9ef3-b6a1feabc30f/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Data/d23777b1-866a-4cbf-803c-3c82398e7025/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Services/EventFrames/RelatedEventFrames - 443 CORP\TestUser 10.82.31.229 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+.NET4.0C;+.NET4.0E) https://testpivision.corp.dom/PIVision/ 500 0 0 0

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Data/f937aecb-14c8-4971-a561-00704159c3ab/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Services/EventFrames/RelatedEventFrames - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 31

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Data/f655034a-dd48-485a-a62a-ca864abc4b6a/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:04 10.10.1.1 POST /PIVision/Data/ca9fb4b1-61bb-43d3-9fa7-5370751babcb/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 15

              2017-07-17 23:29:05 10.10.1.1 GET / - 443 - F5_IPAddress2 - - 200 0 0 0

              2017-07-17 23:29:05 10.10.1.1 POST /PIVision/Data/088731a3-24d8-4432-9aed-6d29f8ffd6ba/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:07 10.10.1.1 GET / - 443 - F5_IPAddress4 - - 200 0 0 0

              2017-07-17 23:29:07 10.10.1.1 POST /PIVision/Data/b82823ef-2265-4a4b-bb85-dcb37a6355db/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:07 10.10.1.1 POST /PIVision/Data/1dac8122-b602-4cc2-a646-0e7af64bb3ed/DiffForData - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 0

              2017-07-17 23:29:07 10.10.1.1 POST /PIVision/Services/EventFrames/RelatedEventFrames - 443 CORP\TestUser 10.82.31.229 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://testpivision.corp.dom/PIVision/ 200 0 0 15

              2017-07-17 23:29:07 10.10.1.1 POST /PIVision/Data/98b47e02-119b-461c-a134-357f71947bdd/DisplayForData - 443 CORP\TestUser 10.82.31.229 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+.NET4.0C;+.NET4.0E) https://testpivision.corp.dom/PIVision/ 500 0 0 15

              2017-07-17 23:29:07 10.10.1.1 GET /PIVision/Display/98b47e02-119b-461c-a134-357f71947bdd/RelatedAssets path= 443 CORP\TestUser 10.82.31.229 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+.NET4.0C;+.NET4.0E) https://testpivision.corp.dom/PIVision/ 416 0 0 0

              2017-07-17 23:29:07 10.10.1.1 GET / - 443 - F5_IPAddress - - 200 0 0 0

              2017-07-17 23:29:08 10.10.1.1 GET / - 443 - F5_IPAddress3 - - 200 0 0 0

            • Re: PI Vision and load balancing issues
              sraposo

              Hi Jagdish,

               

              To my knowledge we don't have any restrictions on the hardware used for Network Load Balancing. I've worked with a customer on setting up load balancing for an AF Server and the load balancing was done with an F5 Network Load Balancer.

               

              As Lal above mentioned, we would need to analyze the IIS logs and monitor the network traffic from a client request to one of the vision servers to try a pin point the issue.

               

              Thanks,

              Sebastien

              • Re: PI Vision and load balancing issues
                czhang

                Hey Jagdish,

                 

                What the Enterprise Program Manager (EPM) meant originally was that troubleshooting of network load balancing (NLB) generally should be done by customers' IT and / or the vendor of the NLB technology because it's out of our expertise, although we are always happy to help. With regards to load balancing PI Vision, we have seen many success use cases with different NLB technologies including F5.

                 

                Back to your question, the "Communication Error : Requested Range Not Satisfiable", aka HTTP 416 error, typically happens when client requests from the same session are routed to different servers. That's why we require session persistence. It seems you have set that up correctly, so as long as the connection is kept active, you will not run into this problem. On the other hand, putting the client into sleep and then waking it up is a different story, as the original session will have timed out so no further requests can be served. You will have to either refresh the page or open a new browser window to initiate a new session.

                 

                Thanks,

                Victor

                2 of 2 people found this helpful