Bryan Owen

Sleepless nights and the revenge of the DVR

Blog Post created by Bryan Owen on Mar 6, 2010

My new DVR arrived just in time for the 2010 Olympics. BTW...hats off to our northern neighbors for going all out on the games and hosting effort!


The DVR delivered functionality as promised...ability to fast forward over the talking heads while stopping to take in some of the awesome BC panorama and of course thrilling competitions. I didn't precisely measure time to view compared to recorded hours but it was roughly half ... the savings easily should have translated to some much needed beauty sleep.


Well, why didn't it? An unfortunate mix of being a light sleeper and Disk Thrashing Phobia (DTP) has made for many sleepless nights.  I'm sure the DVR must be defective, it's not an uber elite device provisioned with a server class drive spinning at 15K where one would expect high decibels.  Gasp...yes, the DVR is in the bedroom.


But, years of service at OSIsoft can change a person - it's easier to filter out the drone of a jet engine than the unwarranted seeking of a disk. Why is there so much I/O all the time?


The obvious solution is the DVR has to go.  Maybe I'll just run it until the drive self destructs. Tinkering with the DVR is not an option but very tempting...maybe something as simple as a cron job is kicking off defrag.  Now we're finally getting to the point of this blog post, eh! 


How do you really know what's running on your PI Servers and when?


Such a question is of interest in security monitoring and regulatory circles. There are several interesting approaches.  Arguably the built-in Windows Management Instrumentation provides good methods to monitor the internals of a server using a common information model (CIM).  CIM Studio, available in the WMItools download provides a way to browse the very rich and extensive CIM model to get a feel for data available in the Windows implementation.


For this discussion, open CIM studio and connect to namespace root\CIMV2. Then search tool for Win32_Process (or browse logical classes under CIM_ManagedElements for the CIM_Process subclass).  Then pick instances on the right hand panel tool bar to return a data set for all running processes, essentially providing a point in time query of task manager GUI information.  Observe that not all properties are valid for each row.  While this kind of empirical validation of CIM implementation details is less than ideal, in practice it is a good reason to test with CIM studio.


The OSIWMI interface can issue WQL (SQL for WMI) in a style similar to the PI relational database interface. WQL queries support placeholder substitution with values provided by PI.  For example, this query selects processes started since the last query:


WQL=Select * from Win32_Process where CreationDate> '#SST# '; TS= CreationDate; Property=Caption; OrderBy= CreationDate


OSIWMI interface configuration for the above query should target a string tag with the extended descriptor (ExDesc) holding the WQL statement. At runtime, the point snapshot timestamp is substituted for the #SST# placeholder in the where clause.  The query result will be sent to PI using the process creation date for the timestamp.


In a default configuration each image name (CIM Caption) returned in the record set is written to PI as separate events. Various format and concatenation options are set using Location 2 and 3 attributes.  Both columns and rows can be combined into a single event if desired.


If you want to monitor more than the PI server, just change the InstrstrumentTag attribute to identify the target node and CIM model path. Enable the windows firewall rules for WMI and away you go.  Like any Windows RPC over DCOM, this approach works best within a domain.


For NERC CIP responsible entities, WMI provides classes useful for near real time monitoring of ports and services.  In addition to driving notifications, archiving this kind of information in PI can also help remove data retention worries in the event of a cyber incident that isn't immediately recognized.  Hat tip: Bruce at Triencon presented his approach for EMS users in Portland last year (extranet for registered T&D SIG members).


The CIM classes described above perform quite well on my test systems but your mileage may vary.  However, efficient WQL for monitoring the file system (CIM_Directory or CIM_DataFile classes) remains elusive.  If any of you find the secret sauce please let me know.


For those without access to the OSIWMI interface, programmatic access to WMI is wide spread. Some of the more interesting examples are now using Powershell.  I suspect some objectives are better served using a programmatic approach because of WQL limitations, it is not full SQL.


For others who ask isn't there a simpler way?  Maybe, Windows Software Restriction Policies (SRP, or sometimes called SAFER) provides an advanced logging feature that could be enough for this purpose.  SRP was offered starting in Windows 2000, provides a built in whitelist and blacklist technology to help prevent malware.  Configuring these technologies is the challenging part.  You need to know exactly what is supposed to run (or not run).


SRP advanced logging helps gather data to set and troubleshoot configuration. Add or remove a registry key to enable or disable the log.  See technet "Using Software Restriction Policies to Protect Against Unauthorized Software" for complete details.


The log includes image activation events and matches them to SRP rules but unfortunately, timestamp is not part of the record.  Of course PI-UFL could frequently process the log and Windows will create a new one as needed.  There is a caution about not letting the log grow too large but this has not been a problem for me.  Besides PI-UFL does the required housekeeping.


So what IS running on your servers in the middle of the night?  Do you really know?  Sleep well everyone!