AnsweredAssumed Answered

Solved: mysteries data leak in a virtual Environment

Question asked by ipas_PI on Mar 10, 2016
Latest reply on Mar 10, 2016 by jkorman

We have several virtual PI Server in an ESXi environment running.

In our case, we have had no compressed and fast Watchdog-Tags/Counters (sample rate of one second) to monitor many PLS in our facilities.

We have recognized, that all PI Servers have a data leak at the same time intervals.

It looks like this for a Watchdog in PB:

01_Watchdog_paused.png

The point is, that we have found non information about this issue in the PI message log files.

After different system checks, we found in the Microsoft EventViewer of all running PI Servers the following warnings in <System> at the time interval of our data leak:

     Source: LSI_SAS + EventID 129

     Source: disk + EventID 153

02_EventViewLog.png

In PI, you get only more Traffic like Events/sec, after warnings <disk> have gone:

02_PI-Traffic.png

 

In the VMware knowledge data base, we found finally the hint (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2063346 ):

Symptoms

When running the LSI_SAS controller in a Windows virtual machine, you experience these symptoms:

    The Windows event log reports the error:

    Reset to device, \Device\RaidPort0, was issued

    The guest operating system of the Windows virtual machine runs so slow that it is functionally unresponsive, but does not fail

Cause

This is a known lost I/O issue occurs because of a loss of synchronization between the miniport and Storport used by the driver.

Resolution

This is a known issue affecting Windows LSI_SAS inbox driver.

 

This issue is fixed in 1.34.02 and later of the LSI_SAS driver. Download the LSILogic SCSI controller driver from the download center at the Avago Technologies Web site. Use the Search tab to download the LSIMPT adapter driver for your guest operating system at Support Documents and Downloads. The downloaded driver zip archive also contains a text file that lists the instructions on how to install the driver during the installation of the Windows operating system.

 

Note: The preceding link was correct as of October 05, 2015. If you find the link is broken, provide a feedback and a VMware employee will update the link.

Alternatively, work around this issue by configuring the virtual machine to use the VMware Paravirtual SCSI driver. For more information, see Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters (1010398).

 

Note: If the virtual machine is part of a Microsoft Cluster, prior to changing the controller, see Microsoft Clustering on VMware vSphere: Guidelines for supported configurations (1037959) prior to changing the controller.

 

If you get the same nightmare, so take care about the synchronization between the miniport and Storport used by the driver also.

After a driver update, this issue should be fixed now.

 

----------------------------------------------

Ralph

Outcomes