Ways to monitor Azure performance counters

 

There are a few ways to pull performance counters off of Azure VMs and send them to a PI System:

 

  1. Using the Azure REST API. These are the performance counters logged to the Azure VMs host hypervisor and what you see on the dashboard in the Azure management portal. As of now, there are 5 performance counters available, CPU Percentage, Disk Read Bytes/Sec, Disk Write Byes/Sec, Network In, and Network out (please see my coverage of how to do this here:  Monitoring your Microsoft Azure VMs with the PI System and the Azure REST API   )
  2. The Azure WADPerformanceCountersTable. Performance counters can be written to Azure storage (in NoSQL format). This requires enabling Azure diagnostics on your VMs via PowerShell or the Azure portal (or though your Azure custom application code). Any performance counter is available to be written; custom counters are supported as well.
  3. At the VM level using the PI Perfmon interface. This requires either a data center extended into Azure, a PI Data Archive at the local subscription level, or a VPN or IPSec tunnel back to a separate PI Data Archive.
  4. Potentially in the future with Microsoft Azure Operational Insights.

 

In this blog post, we will focus on the second method – the WADPerformanceCoutnersTable. This will allow us to capture every performance counter we desire in Azure, including custom performance counters and write them to PI!

 

Note that the solution in this blog post can also monitor all performance counters for Azure applications:

Create and Use Performance Counters in an Azure Application

 

Part of the theme of this and future posts is to show the value of grabbing data from an available API putting it into a PI System.

 

Enabling the collection of data

 

Here we're going to focus on Azure VMs and their performance counters. So to get started, first, we need to enable the Azure Extension that will allow us to collect the performance counters on the VM and write them to Azure Table storage. This extension can currently be enabled through Microsoft Azure Powershell or the new portal (and it will then show up in both portals):

http://azure.microsoft.com/blog/2014/09/02/windows-azure-virtual-machine-monitoring-with-wad-extension/

While configuring this diagnostics, select the storage table that you’ll be writing these counters to and retrieving from:

AzureEnable.PNG

Once that’s in place, you’ll see the following in the new Azure Portal:

NewAzurePortal.png

And in the current Azure Portal:

OldAzurePortal.png

 

So we’re going to develop two functions in PowerShell, one to get the Azure WADPerformanceCounterTable data and one to write them to a PI Data Archive and AF Server via the AFSDK. In the end, we’ll an AF Element hierarchy with AF subelements per storage account holding the all of the performance counters recorded via the Microsoft.Azure.Diagnostics.IaaSDiagnostics extension in Microsoft Azure.

 

What to have in place

There are four things to make sure you have in place in your running environment before getting into the PowerShell script:

 

  1. Versions of the PowerShell Tools(4.0+) and Azure PowerShell tools(0.8.10.1 are the ones I'm developing with). The Azure Powershell cmdlets are currently in flux so certain cmdlet properties may be different between versions. It's fairly easy to change the properties of Azure cmdlets in the script, but something to be aware of if running the code below it doesn't work, some of the object properties may have changed.
  2. Appropriate certificate(s) stored to access your Azure subscription on the back end. The easiest way to get these is to do a onetime import of your .publishsettings file downloaded from your subscription.
  3. Access to a PI Data Archive from the location you are running the script with appropriate permissions to create PI Points.
  4. The AFSDK installed at the location you are running the script.

 

A good getting started guide for numbers 1 and 2 can be found here: How to install and configure Azure PowerShell

 

Now, lets step through the PowerShell and AFSDK code. Note that I'm using Python highlighting since native PowerShell highlighting is not available on PI Square.

 

The Code

We're going to create two functions Get-AzureWADMetrics and Set-AzureWADMetrics. Get-AzureWADMetrics will call the Azure WADPerformanceCountersTable and pull in the performance counter data off a NoSQL table. Set-AzureWADMetrics will use the AFSDK to write the data to an AF Element hierarchy and PI tags.

 

Some disclaimers (also, the code formatting on PI Square may be strange so I'd suggest grabbing the attached ZIP file in order to use the code):

# ***********************************************************************
# * DISCLAIMER:
# *
# * All sample code is provided by OSIsoft for illustrative purposes only.
# * These examples have not been thoroughly tested under all conditions.
# * OSIsoft provides no guarantee nor implies any reliability,
# * serviceability, or function of these programs.
# * ALL PROGRAMS CONTAINED HEREIN ARE PROVIDED TO YOU "AS IS"
# * WITHOUT ANY WARRANTIES OF ANY KIND. ALL WARRANTIES INCLUDING
# * THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY
# * AND FITNESS FOR A PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED.
# ************************************************************************

#SCRIPT TESTED VERSIONS:
#$PSVersionTable
#Name                           Value                                             
#----                           -----                                             
#PSVersion                      4.0                                               
#WSManStackVersion              3.0                                               
#SerializationVersion           1.1.0.1                                           
#CLRVersion                     4.0.30319.34209                                   
#BuildVersion                   6.3.9600.17090                                    
#PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}                              
#PSRemotingProtocolVersion      2.2 

#Import-Module Azure
#Get-Module Azure
#ModuleType Version    Name                                     
#---------- -------    ----                                  
#Manifest   0.8.10.1   Azure 

#PI AF Developer tools 2.6.0.5843

#Credit to inspiration for Get-AzureWADMetrics goes to Michael Repperger 8-Aug-2014
#Additional Information here - including adding a proxy server:
#http://blogs.technet.com/b/omx/archive/2014/08/08/read-the-azure-storage-analytics-metrics-table-with-powershell.aspx

 

 

Function Get-AzureWADMetrics: this will hit the Azure storage table data source and return an XML file containing performance counter data that we can parse in PowerShell:

function Get-AzureWADMetrics
{
<#
.SYNOPSIS
  tbd.
.DESCRIPTION
  tbd.
#>
    [CmdletBinding()]
    param(
          [parameter(Mandatory=$true)]
          [string]
          $primaryStoreKey,
          [parameter(Mandatory=$true)]
          [string]
          $storageName,
          [parameter(Mandatory=$true)]
          [string]
          $minutesAgo
     )
    BEGIN 
    {
    }
    PROCESS
    {
        $strTable = 'WADPerformanceCountersTable()'
        $tableEP = 'https://'+$storageName+'.'+'Table'+'.core.windows.net/'

        $ts = [System.DateTime]::UTCNow.AddMinutes(-$minutesAgo)

        # Get the current date/time and format it properly the culture "" is the invariant culture
        $myDateObj = Get-Date
        $myInvariantCulture = New-Object System.Globalization.CultureInfo("")
        $strDate = $myDateObj.ToUniversalTime().ToString("R", $myInvariantCulture)

        # This will get only the metrics from when your partition key requests it (past 30 mins) in this case.
        $strTableUri = $tableEP + $strTable + '?$filter=PartitionKey%20ge%20''' + '0' + $ts.Ticks + ''''

        # Preare the HttpWebRequest
        $tableWebRequest = [System.Net.HttpWebRequest]::Create($strTableUri)
        $tableWebRequest.Timeout = 15000
        $tableWebRequest.ContentType = "application/xml"
        $tableWebRequest.Method = "GET"
        $tableWebRequest.Headers.Add("x-ms-date", $strDate)
      
         # Create a hasher and seed it with the storage key
        $sharedKey = [System.Convert]::FromBase64String( $primaryStoreKey)
        $myHasher = New-Object System.Security.Cryptography.HMACSHA256
        $myHasher.Key = $sharedKey

        # Create the Authorization header
        $strToSign = $tableWebRequest.Method + "`n" `
                    + $tableWebRequest.Headers.Get("Content-MD5") + "`n" `
                    + $tableWebRequest.Headers.Get("Content-Type") + "`n" `
                    + $tableWebRequest.Headers.Get("x-ms-date") + "`n" `
                    + '/' + $storageName + '/' + $strTable
        $bytesToSign = [System.Text.Encoding]::UTF8.GetBytes($strToSign)
        $strSignedStr = [System.Convert]::ToBase64String($myHasher.ComputeHash($bytesToSign))
        $strAuthHeader = "SharedKey " + $storageName + ":" + $strSignedStr
        $tableWebRequest.Headers.Add("Authorization", $strAuthHeader)

        # Read the results        
        $tableResponse = $tableWebRequest.GetResponse()
        $tableResponseReader = New-Object System.IO.StreamReader($tableResponse.GetResponseStream())
        [xml]$xmlMetricsData = $tableResponseReader.ReadToEnd()
        $tableResponseReader.Close()

        Write-Output $xmlMetricsData
       }
    END
    {
    }
}

 

Function Set-AzureWADMetrics. This will parse the XML returned from the Get-AzureWADMetrics cmdlet and write that data to a PI Data archive and AF Server.

function Set-AzureWADMetrics
{
<#
.SYNOPSIS
  tbd.
.DESCRIPTION
  tbd.
#> 
[CmdletBinding()]
    param (
            [Parameter(Mandatory = $true, ValueFromPipeline=$true)]
            $xmlMetricsData,
            [Parameter(Mandatory = $true)]
            [String]
            $subscriptionName,
            [Parameter(Mandatory = $true)]
            [String]
            $subscriptionID,
            [Parameter(Mandatory = $true)]
            [String[]]
            $azureVMs
            )
    BEGIN
    { 
    }
    PROCESS
    {
    $entries = $xmlMetricsData.feed.entry

    # Load AFSDK
    [System.Reflection.Assembly]::LoadWithPartialName("OSIsoft.AFSDKCommon") | Out-Null
    [System.Reflection.Assembly]::LoadWithPartialName("OSIsoft.AFSDK") | Out-Null

    # Create AF Object
    $PISystems=New-object 'OSIsoft.AF.PISystems'
    $PISystem=$PISystems.DefaultPISystem
    $myAFDB=$PISystem.Databases.DefaultDatabase

    # Create PI Object
    $PIDataArchives=New-object 'OSIsoft.AF.PI.PIServers'
    $PIDataArchive=$PIDataArchives.DefaultPIServer

    # Create AF UpdateOption
    $AFUpdateOption = New-Object 'OSISoft.AF.Data.AFUpdateOption'
    #Set AF Update Option to Replace
    $AFUpdateOption.value__ = "0"

    # Create AF BufferOption
    $AFBufferOption = New-Object 'OSISoft.AF.Data.AFBufferOption'
    #Set AF Buffer Option to Buffer if Possible
    $AFBufferOption.value__ = "1"

    #Create AF Recorded Value
    $AFRecordedValue = New-Object 'OSIsoft.AF.Data.AFBoundaryType'
    #Set AF recorded Value option to Inside
    $AFRecordedValue.value__ = "0"

    # Add required entries pulled from Azure Table in Get-AzureVMMetrics
    #Create the top level subscription element
    if($myAFDB.Elements.Contains($subscriptionID) -eq $false)
    {
        $mysubscriptionElement=$myAFDB.Elements.Add($subscriptionID) 
        $mysubscriptionElement.Description = $subscriptionName
    }
    else
    {
        #find just that element already created
        $mysubscriptionElement= $myAFDB.Elements | Where-Object {$_.Name -eq $subscriptionID}  
    }

    #Create the subelements corresponding to each VM monitored
    if($mysubscriptionElement.Elements.Contains($storageName) -eq $false)
    {
        $myElement=$mysubscriptionElement.Elements.Add($storageName) 
    }
    else
    {
        #find just that element already created
        $myElement = $mysubscriptionElement.Elements | Where-Object {$_.Name -eq $storageName} 
    }
                
    #Create the Attributes for Metrics
    #Order: 'Disk Read Bytes/sec,Disk Write Bytes/sec,Network Out,Percentage CPU,Network In'
     # Add required entries pulled from Azure Table in Get-AzureVMMetrics
    for($i=0; $i -lt $entries.Count; $i++)
    {
        $metricName = $entries.content.properties.CounterName[$i].Replace("\","_") 
        $metricValuesTimeStamps =  $entries.content.properties[$i].timestamp.'#text'
        [double]$metricValues = $entries.content.properties.CounterValue[$i].InnerText 
        $roleinstance =  $entries.content.properties[$i].roleinstance       

        $myAttrName = $roleinstance + $metricName  

        if($myElement.Attributes.Contains($myAttrName) -eq $false) 
        { $myAttr=$myElement.Attributes.Add($myAttrName)}
        else
        {$myAttr=$myElement.Attributes | Where-Object {$_.Name -eq $myAttrName}}
            
        # Assign Tag Name to the PI Point
        $tagName = $subscriptionID +'_' + $roleinstance + '_'+ $metricName #$tagname = "Testing"
                  
        #Create the PI Point associated with that attribute
        $piPoint = $null
        if([OSIsoft.AF.PI.PIPoint]::TryFindPIPoint($PIDataArchive,$tagName,[ref]$piPoint) -eq $false)
        { 
            $piPoint = $piDataArchive.CreatePIPoint($tagName) 
        }

        #Manipulate TimeStamp String to output something friendly for AF to input
        #example $timestamp = "2015-05-05T18:17:43.943Z"
       $timestampu = @(); foreach($tsz in $metricValuesTimeStamps) { $timestampu += ([Datetime]::Parse(($tsz -replace "Z",""))) }
       $recordedValues = New-Object 'Collections.Generic.List[OSIsoft.AF.Asset.AFValue]'
            
       for($j=0; $j -lt $timestampu.Count; $j++)
       {
       # Instantiate a new 'AFValue' object to persist...
       $newValue = New-Object 'OSIsoft.AF.Asset.AFValue'
       # Fill in the properties.
       $newValue.Timestamp = New-object 'OSIsoft.AF.Time.AFTime'($timestampu[$j])   
       $newValue.pipoint = $pipoint
       $newValue.Value = $metricValues[$j]
     
       # Add to collection.
       $recordedValues.Add($newValue)
       }
   
        #Update the PI Point Values
        try
        {
            $piPoint.UpdateValues($recordedValues,$AFUpdateOption)
        }
        catch
        {
            $message = ($_.ErrorDetails.Message)
            {continue}  
            #throw "{0}: {1}" -f $message.Error.Code, $message.Error.Message 
        }
        #Associate the PI point with that Attribute
        $myAttr.DataReferencePlugIn = $PISystem.DataReferencePlugIns["PI Point"]
        $myAttr.ConfigString = "\\%server%\$tagName;ReadOnly=false"
        
        #Check in the AF Elements
        $mysubscriptionElement.CheckIn()
        $myElement.CheckIn()    
    }
  #Disconnect from the AF Server
    #$PISystem.Disconnect()

    # Disconnect from the PI Data Archive
    #$PIDataArchive.Disconnect()
    }
    END
    {
    }
}

 

Now, lets call these two functions: Note that Set-AzureWADMetrics takes pipeline input from Get-AzureWADMetrics :

 

#Calling the Scripts. Note that Set-AzureWADMetrics takes pipeline input from Get-AzureWADMetrics:
#Set the Azure Subscription to the current one  
$subscriptionName = 'XXXXX' #Enter the name of your Subscription  
$selectAzureSubscription = Select-AzureSubscription -SubscriptionName $subscriptionName  

#Assume it's the first storagename hardcoded here, but may change.
$storageNames = (Get-AzureStorageAccount).Label 
$storageName = $storageNames[0]

$subscriptionID = $selectAzureSubscription.Id.ToString()  
  
#Retrieve all of the VMs and loop through them for individual output  
[String[]]$azureVMs = (Get-AzureVM).ServiceName  
  
#Save the storage account key (replace with your storage name hardcoded if you know it)  
$primaryStoreKey  = (Get-AzureStorageKey -StorageAccountName $storageName).Primary  

#How many minutes ago do you want to pull perfmon counters from:  
$minutesAgo = "30"    
  
#Get and Set the Azure Metrics. In this example, we pull the past 30 minutes from the WADPerformanceCounter NoSQL table.  
Get-AzureWADMetrics -primaryStoreKey $primaryStoreKey -storageName $storageName -minutesAgo $minutesAgo | Set-AzureWADMetrics -subscriptionName $subscriptionName -subscriptionID $subscriptionID -azureVMs $azureVMs 

So that's it!

 

The Result

Once the above is in place, you'll have the following type AF and PI tag structure:

 

AF.PNG

 

Using the Measure-Command{} here are the results for an inital run of 30 minutes of performance counters at 1 minute intervals to build the AF structure and populate the PI tags for around 25 PI Tags:

 

Days              : 0

Hours             : 0

Minutes           : 1

Seconds           : 17

Milliseconds      : 841

 

Some things that may improve speed around this and affect your scalability include the number of call backs to both Azure and the PI System. It may dramatically improve performance to do one call to Azure for the Performance Counters, cache the Azure VM names and put all of the writing of results into one call to the PI Data Archive and one call to the AF Server.

 

In order to see the attachment, make sure you're on the permanent link (Monitoring Azure Performance Counters with the WADPerformanceCountersTable, PI System, PowerShell, and the AFSDK. )

 

Let me know if you have any questions.

 

Thanks,

Eric Pennaz