Skip navigation
All Places > PI Developers Club > Blog
1 2 3 Previous Next

PI Developers Club

543 posts

Introduction

 

Today we release our first version of the PI Web API client libraries for .NET Framework and .NET Core. Those libraries were generated using the Swagger specification available on PI Web API 2017+.

 

Although PI AF SDK still remains the best choice in case performance is a must, there are a lot of scenarios where it makes sense to develop your .NET applications using with those libraries:

 

  • You cannot install PI AF SDK on the machine which would run your application.
  • You are developing a web application using Azure Web Apps, which doesn't allow you to remote in and install OSIsoft products.
  • Your PI System is hosted in the cloud. Therefore, it makes to expose the data through a public endpoint of PI Web API.
  • You want to create applications that are compatible with Linux and macOS.
  • You want to use the most modern technology of the market.

 

The first two reasons of the list above applies when you are developing custom .NET Framework apps.  The last 3 items from this list applies for .NET Core. But what is .NET Core exactly?

 

What is .NET Core?

 

According to the .NET blog, .NET Core is a cross-platform, open source, and modular .NET platform for creating modern web apps, microservices, libraries and console applications. Because it is cross-platform, you can run your .NET Core apps on Windows, Linux and macOS. Users can use the command-line tools to manage their project.

 

.NET Core is composed by 3 parts: .NET runtime, a set of framework libraries and a set of SDK tools. This new technology can be need as a cross-platform version of the .NET Framework.

 

.NET Core and PI Developer Technologies

 

.NET Core is not compatible with .NET Framework libraries like PI AF SDK. OSIsoft's product management and development teams have ongoing investigations into creating a PI AF SDK version compatible with .NET Core, but at this point there is no timeline for release. Although the PI AF SDK is faster than PI Web API, the performance of our RESTful web service is sufficient for many use cases.  The PI Web API team is committed to improving performance and adding features to each new release.

 

 

If you are developing a long running service or an application that retrieves large amounts of PI data, our suggestion is to develop on top of PI AF SDK and .NET Framework. If this is not the case, using this new client library for developing .NET Core apps should suffice your needs.

 

 

Requirements

 

  • PI Web API 2017 installed within your domain using Kerberos or Basic Authentication.
  • If you are using the .NET Framework version of this library, you should have .NET Framework 4.5 installed on your machine.
  • If you are using the .NET Core version of this library, you should have .NET Core 1.1 installed on your machine.

 

Installation

 

  • Download this source code
  • Create a new folder under %PIHOME% named WebAPIClient, if it doesn't exist.
  • Create a new folder under WebAPIClient named DotNetCore, if it doesn't exist.
  • Copy the unique file from the dist folder to %PIHOME%\WebAPIClient\DotNetCore (.NET Core) or %PIHOME%\WebAPIClient\DotNetCore (.NET Framework).

 

Usage

 

.NET Framework

 

Create a new .NET Framework project (Console Application for instance). On the Solution Explorer, right click on Dependencies and then "Add Reference...". Click on the Browse button and navigate to the %PIHOME%\WebAPIClient\DotNet folder. Finally, add the OSIsoft.PIDevClub.PIWebApiClient.dll to your VS project.

 

.NET Core

 

Create a new .NET Core project (Console Application for instance). Open the Package Manager Console and run the following command to add this library to your .NET Core project.:

 

Install-Package OSIsoft.PIDevClub.PIWebApiClient -Source %PIHOME%\WebAPIClient\DotNetCore

 

Source Code

 

The Visual Studio solution that generates the final library is available on the src folder. You might want to add or edit a method and rebuild the solution in order to generate custom assemblies. The links from both GitHub repositories are below:

 

 

Documentation

 

Once the library was added to your project, you can start writing code. The beauty of this library is that you don't need to know about writing code to make HTTP requests. The library will do that under the hood. All actions and methods from all the controllers of PI Web API 2017 are available on this client library. You just have to create the PI Web API top level object and access its properties. Each property maps a PI Web API controller which has methods which maps the actions from the RESTful web service. Therefore if you want to access the GetByPath action from the Point controller just call piwebapi.Point.GetByPath().

 

All classes and methods are described on here. Even interally both libraries are different, they have the same classes and methods from the user perspective.

 

Examples

 

Please check the Program.cs from the LibraryTest project from the Visual Studio solution of this repository. Below there are also code snippets written in C# for you to get started using this library:

 

Create an instance of the PI Web API top level object.

 

     PIWebApiClient client = new PIWebApiClient("https://marc-web-sql.marc.net/piwebapi", true);  

If you want to use basic authentication instead of Kerberos, set useKerberos to false and set the username and password accordingly.

 

Get the PI Data Archive WebId

 

    PIDataServer dataServer = client.DataServer.GetByPath("\\\\MARC-PI2016");

 

Create a new PI Point

 

    PIPoint newPIPoint = new PIPoint(); newPIPoint.Name = "MyNewPIPoint" newPIPoint.Descriptor = "Point created for wrapper test" newPIPoint.PointClass = "classic" newPIPoint.PointType = "Float32" ApiResponseObject response = client.dataServer.CreatePointWithHttpInfo(dataServer.webId, newPIPoint)

 

Get PI Points WebIds

 

    PIPoint point1 = client.Point.GetByPath("\\\\marc-pi2016\\sinusoid"); PIPoint point2 = client.Point.GetByPath("\\\\marc-pi2016\\sinusoidu"); PIPoint point3 = client.Point.GetByPath("\\\\marc-pi2016\\cdt158");

 

Get recorded values in bulk using the StreamSet/GetRecordedAdHoc

 

    List<string> webIds = new List<string>() { point1.WebId, point2.WebId, point3.WebId }; PIItemsStreamValues piItemsStreamValues = client.StreamSet.GetRecordedAdHoc(webIds, startTime: "*-3d", endTime: "*");

 

Send values in bulk using the StreamSet/UpdateValuesAdHoc

 

     var streamValuesItems = new PIItemsStreamValues(); var streamValue1 = new PIStreamValues(); var streamValue2 = new PIStreamValues(); var streamValue3 = new PIStreamValues(); var value1 = new PITimedValue(); var value2 = new PITimedValue(); var value3 = new PITimedValue(); var value4 = new PITimedValue(); var value5 = new PITimedValue(); var value6 = new PITimedValue(); value1.Value = 2; value1.Timestamp = "*-1d"; value2.Value = 3; value2.Timestamp = "*-2d"; value3.Value = 4; value3.Timestamp = "*-1d"; value4.Value = 5; value4.Timestamp = "*-2d"; value5.Value = 6; value5.Timestamp = "*-1d"; value6.Value = 7; value6.Timestamp = "*-2d"; streamValue1.WebId = point1.WebId; streamValue1.Items = new List<PITimedValue>(); streamValue1.Items.Add(value1); streamValue1.Items.Add(value2); streamValue2.WebId = point2.WebId; streamValue2.Items = new List<PITimedValue>(); streamValue2.Items.Add(value3); streamValue2.Items.Add(value4); streamValue3.WebId = point2.WebId; streamValue3.Items = new List<PITimedValue>(); streamValue3.Items.Add(value5); streamValue3.Items.Add(value6); ApiResponse<PIItemsItemsSubstatus> response2 = client.StreamSet.UpdateValuesAdHocWithHttpInfo(new List<PIStreamValues>() { streamValue1, streamValue2, streamValue3 });

 

Get element and its attributes given an AF Element path

 

     PIElement myElement = client.Element.GetByPath("\\\\MARC-PI2016\\CrossPlatformLab\\marc.adm"); PIItemsAttribute attributes = client.Element.GetAttributes(myElement.WebId, null, 1000, null, false);

 

Get current value given an AF Attribute path

 

     PIAttribute attribute = client.Attribute.GetByPath(string.Format("{0}|{1}", "\\\\MARC-PI2016\\CrossPlatformLab\\marc.adm", attributes.Items[0].Name)); PITimedValue value = client.Stream.GetEnd(attribute.WebId);

 

Get Event Frames given an AF Database path

 

    PIAssetDatabase db = client.AssetData.GetByPath(path); PIItemsEventFrame efs = client.AssetData.GetEventFrames(db.WebId, referencedElementNameFilter: "myElement", referencedElementTemplateName:

 

Final Remarks

 

As you could realize by looking at the example, it is pretty easy to use this client library on your apps. Please provide your feedback by posting your comments below!

This question comes up quite a bit at PI Square.  I know of at least 5 different ways to find out my AF Client version.  In a nutshell,

 

  1. From the registry settings
  2. From Programs and Features
  3. From the file properties in File Explorer
  4. From PI System Explorer (if also loaded on the client machine)
  5. From custom code

 

Let's go over each of these.

 

Registry Settings

 

Run regedit.exe

Navigate to:    Computer\HKEY_LOCAL_MACHINE\SOFTWARE\PISystem\AF Client

Alternative:     Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\PISystem\AF Client

 

2017-07-19 09_07_09-Registry Editor.png

2017-07-19 09_06_00-Registry Editor.png

 

2017-07-19 09_07_29-Registry Editor.png

 

Programs and Features

Trying to run this may take you to Apps and features.  You could filter the list to AF and see something like:

 

2017-07-19 09_11_51-Settings.png

 

If you really want the Version  number, look to the upper right of the window for Programs and Features:

2017-07-19 09_13_40-Settings.png

 

You should then have a list where you may scroll to:

 

2017-07-19 09_12_46-Programs and Features.png

 

 

File Properties in File Explorer

 

Using File Explorer, navigate to:     %PIHOME%\AF\PublicAssemblies\4.0

 

In my example below, %PIHOME% is "C:\Program Files (x86)\PIPC" but this could be a different drive on your machine.

 

2017-07-19 09_15_33-4.0.png

 

Right-click on the file:     OSIsoft.AFSDK.dll

 

Click on the Details tab along the tab bar at the top.

 

     2017-07-19 09_16_27-OSIsoft.AFSDK.dll Properties.png

 

 

From PI System Explorer

 

Open PSE

From the menu bar at the top, click on Help

Click on About PI System Explorer ...

 

2017-07-19 09_17_46-About PI System Explorer.png

 

 

Custom AFSDK Code

 

From a C# application that has a reference to OSIsoft.AFSDK.dll, you can use:

 

Console.WriteLine((new OSIsoft.AF.PISystems()).Version);

 

Or if you already have a  using OSIsoft.AF;  statement, this shorter version will do:

 

Console.WriteLine((new PISystems()).Version);

 

Or if you despite one-liners, you may try:

 

var clientSDK = new PISystems();

Console.WriteLine( clientSDK.Version );

 

For VB.NET, you would use an  Imports OSIsoft.AF  statement, and this line of code:

 

Console.WriteLine((New PISystems()).Version)

 

And finally, if you're not a developer or don't have Visual Studio, all is not lost.  You may still try using Powershell.

 

[Reflection.Assembly]::LoadWithPartialName("OSIsoft.AFSDK")

$clientSDK = New-Object OSIsoft.AF.PISystems

Write-Host "AF Client Version:" $clientSDK.Version

 

To produce output such as:

 

GAC    Version        Location

---    -------        --------                                                                                                                        

True   v4.0.30319     C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\OSIsoft.AFSDK\v4.0_4.0.0.0__6238be57836698e6\OSIsoft.AFSDK.dll                             AF Client Version: 2.9.1.8106

 

 

There you go.  That's at least 5 different ways to get the AF Client Version number.  Are there any others that I may have missed?

A topic that I've seen posted quite a few times is "How do I get a custom symbol to change other symbols on the display?"  Without the knowledge of what's available to you, it can seem like a tricky or impossible task.  Once you know how to do it, it actually becomes quite easy. To get you started, I've written a couple of fun/useless symbols to show you the basics.  I may expand/improve the list in the future because I'm constantly learning new things about PI Vision Extensibility.

 

The Big Red Button

 

What does it do?

delete.gif

It deletes everything on your display when clicked.  It should go without saying but please don't use this on any production displays.

How does it work?

It uses the displayProvider to select all of the symbols in the display and subsequently loops though and deletes all selected symbols.

The code

HTML:

<div id="bigredbutton" ng-click="pressed()"> DELETE </div>

JS:

window.PIVisualization = window.PIVisualization || {};
(function (PV) {
     'use strict';

     function symbolVis() { }
     PV.deriveVisualizationFromBase(symbolVis);
     symbolVis.prototype.init = function (scope, elem, displayProvider, $rootScope)
          scope.pressed = function(){
               displayProvider.selectAll();
               displayProvider.getSelectedSymbols().forEach(function(sym){
                    displayProvider.removeSymbol(sym);
               });
          }
     }

     var def = {
          typeName: 'bigredbutton',
          datasourceBehavior: PV.Extensibility.Enums.DatasourceBehaviors.Single,
          visObjectType: symbolVis,
          inject: [ 'displayProvider', '$rootScope'],
          getDefaultConfig: function(){
               return{
                    DataShape: 'Value',
                    Height: 150,
                    Width: 150
               };
          }
     };
     PV.symbolCatalog.register(def);
})(window.PIVisualization);

 

The Add DataStream Button

What does it do?

AddData.gif

When the add button is hit, it searches the display for all symbols that support multiple data streams.  It then adds whatever tag or attribute is in the textbox as a datastream to each of the found symbols.

How does it work?

It uses the displayProvider to select all of the symbols in the display and subsequently loops though, checks if the DatasourceBehavior allows multiple data sources, pushes the new data source, and then deselects the symbol.

The code

HTML:

<div id='add-data-source-container'>
     <input type='text' class='text-input' ng-model='newdatasource.text' placeholder='pi:\\servername\tagname'/>
     <button class='submit-button' ng-click="add()"> Add </button>
</div>

JS:

Note: Everything except the init function is the same as the previous symbol.  For brevity, just the init function is shown.

scope.newdatasource = {
     text: ""
};

scope.add = function(){
     displayProvider.selectAll();
     displayProvider.getSelectedSymbols().forEach(function(sym){
          //check if this symbol supports multiple datasources
          if(displayProvider.getRuntimeSymbolData(sym).def.datasourceBehavior == PV.Extensibility.Enums.DatasourceBehaviors.Multiple){
               //add the datasource from the textbox
               displayProvider.getSymbolByName(sym).DataSources.push(scope.newdatasource.text);
          }
          //deselect this symbol
          displayProvider.toggleSymbolSelection(sym);
     });
}

 

The Chatting Symbols

What does it do?

SymbolComm.gif

The name of a receiver symbol (which must be of the same type) is entered into the top box along with a message in the bottom box.  Upon clicking "Send", that message is displayed in the destination's bottom box along with who sent it (in the top box).

How does it work?

This symbol is starting to show some real power.  We're modifying the receiver's scope to change its values in real time.

Before we get to the code

There is a behavior in Angular which makes reading another DOM element's scope impossible unless debugInfoEnabled is set to true.  In order for this symbol to work, you will need to edit a line in PIVisualization.app.js changing:

$compileProvider.debugInfoEnabled(enable);

to

$compileProvider.debugInfoEnabled(true);

Be aware that this may cause a performance decrease.

The code

HTML:

<div id='sayhi-source-container'>
     <input type='text' class='text-input receiver' ng-model='message.receiver' ng-click='receiverboxclicked()' placeholder='symbol name of receiver'/>
     <input type='text' class='text-input message' ng-model='message.text' placeholder='type message to receiver'/>
     <button class='submit-button' ng-click="sayhi()"> Send </button>
</div>

JS:

Note: Everything except the init function is the same as the the big red button symbol. For brevity, just the init function is shown.

var name = this.runtime.name;

scope.message = {
     receiver: "",
     text: ""
};

scope.receiverboxclicked = function(){
     scope.message.receiver = scope.message.receiver.replace(' says:','');
     scope.message.text = '';
}

scope.sayhi = function(){
     var receiver = $('#'+scope.message.receiver);
     if(receiver.length > 0){
          var receiver_scope = receiver.scope();
          if(receiver_scope.message){
               receiver_scope.message.receiver = name + " says:";
               receiver_scope.message.text = scope.message.text;
          }
          else{
               alert(name + "'s receiver, " + scope.message.receiver + ", is not of type 'sayhi' ");
          }
     }
     else{
          alert(name + "'s receiver," +scope.message.receiver + ", does not exist");
     }
}

Introduction

 

Although it is considered an old technology, there are still many developers who use VBA to integrate their PI ProcessBook displays or Excel spreadsheets with the PI System. Since our most performant PI Developer Technology is PI AF SDK, which is a native .NET Framework, it cannot be used within the VBA environment. One option is to use PI SDK in VBA in order to communicate with the PI Data Archive. Nevertheless, if those developers want to work with elements, attributes or event frames, there wasn't any really good alternative.

 

Today we are releasing the first version of the PI Web API Wrapper for VBA, which is a client RESTful web service for PI Web API 2017. With this library, almost all methods available on PI Web API 2017 can be called through this library. This means you can get any element description, retrieve its attributes, search for event frames, create new PI Points, sends and get data in bulk within the VBA context.

 

 

It is always good to remember that PI Vision is the most suitable technology for any new development projects. This wrapper should be used only when PI Vision cannot be used for any reason.

 

Requirements

 

  • PI Web API 2017 installed within your domain using Kerberos or Basic Authentication.
  • PI ProcessBook 2012 SP1+
  • .NET Framework 3.5

 

Installation

 

  • Download the latest release from our GitHub repository
  • Create a new folder under %PIHOME% named WebAPIClient, if it doesn't exist.
  • Create a new folder under WebAPIClient named VBA, if it doesn't exist.
  • Copy all files from the dist folder to %PIHOME%\WebAPIClient\VBA.
  • Run as Administrator the reg.bat located on %PIHOME%\WebAPIClient\VBA in order to register the PIWebApiWrapper assmebly.

 

Usage

 

Create or edit a PI ProcessBook display. Press ALT+F11 to open Visual Basic for Applications. On the menu, click on Tools --> References. Find PIWebApiWrapper on the list box of the available reference and add it to the VBA project.

 

 

Source Code

 

The Visual Studio solution that generates the final library is available on the src folder. You might want to add or edit a method and rebuild the solution in order to generate custom assemblies.

 

Documentation

 

All classes and methods are described here on GitHub. You can also use the Object Browser from Visual Basic for Application to read the same information. Please refer to the screenshot below:

 

 

Remember

 

As this is a .NET library with COM objects and methods exposed, in order to be able to be consumed within the VBA environment, there are some things to have in mind, especially when comparing with C# development.

  • VBA is not compatible with async methods. Therefore, only sync methods are available in this library.
  • For each PI Web API action/method of each controller, there are two methods on this client library. One returns the response of the HTTP request itself and the other wraps the response on top of ApiResponse class, providing http information, such as status code. Please refer to the Get and GetWithHttpInfo methods on our documentation and you will realize the difference between them by comparing the method signature.
  • The Batch and Channel controllers are not exposed.
  • When working with data transfer objects (models) with an Items property (such as PIItemsElement), do not access or modify this property directly. Use CreateItemsArray(), GetItem(), SetItem() and GetItemsLength() instead.
  • For models that have the Value property, use SetValueWithString(), SetValueWithInt(), SetValueWithDouble() methods to set this property.
  • For the Api methods, all variables whose type are not a string must be defined. If a string variable is optional, define it as an empty string instead of Null.

 

Examples

 

There are two PI ProcessBook displays available on the Samples folder of this repository. In addition, please refer to the following examples to understand how to use this library:

 

Create an instance of the PI Web API top level object.

 

    Dim client As New PIWebApiClient
    Dim connectedToPIWebAPI As Boolean
    connectedToPIWebAPI = client.Connect("https://marc-web-sql.marc.net/piwebapi", True)

 

If you want to use basic authentication instead of Kerberos, set useKerberos to False and set the username and password accordingly. We recommend using Kerberos because it is the safest option. For basic authentication, the password needs to be hardcoded which is not recommended. If using Kerberos authentication is not an option, protect your VBA code with a password.

 

Get the PI Data Archive WebId

 

    Set dataServer = client.dataServer.GetByName(tbPIDataArchiveName.Text)

 

Create a new PI Point

 

    Dim response As ApiResponseObject
    Dim newPIPoint As New PIPoint
    newPIPoint.Name = "MyNewPIPoint"
    newPIPoint.Descriptor = "Point created for wrapper test"
    newPIPoint.PointClass = "classic"
    newPIPoint.PointType = "Float32"
    Set response = client.dataServer.CreatePointWithHttpInfo(dataServer.webId, newPIPoint)

 

Get PI Points WebIds

 

    Set point1 = client.point.GetByPath("\\" + tbPIDataArchiveName.Text + "\" + tbTagName1.Text)
    Set point2 = client.point.GetByPath("\\" + tbPIDataArchiveName.Text + "\" + tbTagName2.Text)
    Set point3 = client.point.GetByPath("\\" + tbPIDataArchiveName.Text + "\" + tbTagName3.Text)

 

Get recorded values in bulk using the StreamSet/GetRecordedAdHoc

 

    webIds = point1.webId + "," + point2.webId + "," + point3.webId
    Set compressedData = client.StreamSet.GetRecordedAdHoc(webIds, True, 1000)

 

Send values in bulk using the StreamSet/UpdateValuesAdHoc

 

 

    Call GetPIPoints
    Dim streamValuesItems As New PIItemsStreamValues
    Dim streamValue1 As New PIStreamValues
    Dim streamValue2 As New PIStreamValues
    Dim streamValue3 As New PIStreamValues
    Dim value1 As New PITimedValue
    Dim value2 As New PITimedValue
    Dim value3 As New PITimedValue
    Dim value4 As New PITimedValue
    Dim value5 As New PITimedValue
    Dim value6 As New PITimedValue


    streamValuesItems.CreateItemsArray (3)
    value1.SetValueWithInt (2)
    value1.Timestamp = "*-1d"
    value2.SetValueWithInt (3)
    value2.Timestamp = "*-2d"
    value3.SetValueWithInt (4)
    value3.Timestamp = "*-1d"
    value4.SetValueWithInt (5)
    value4.Timestamp = "*-2d"
    value5.SetValueWithInt (6)
    value5.Timestamp = "*-1d"
    value6.SetValueWithInt (7)
    value6.Timestamp = "*-2d"


    streamValue1.webId = point1.webId
    streamValue1.CreateItemsArray (2)
    Call streamValue1.SetItem(0, value1)
    Call streamValue1.SetItem(1, value2)
    Call streamValuesItems.SetItem(0, streamValue1)


    streamValue2.webId = point2.webId
    streamValue2.CreateItemsArray (2)
    Call streamValue2.SetItem(0, value3)
    Call streamValue2.SetItem(1, value4)
    Call streamValuesItems.SetItem(1, streamValue2)


    streamValue3.webId = point2.webId
    streamValue3.CreateItemsArray (2)
    Call streamValue3.SetItem(0, value5)
    Call streamValue3.SetItem(1, value6)
    Call streamValuesItems.SetItem(2, streamValue3)

    Dim response As ApiResponsePIItemsItemsSubstatus
    Set response = client.StreamSet.UpdateValuesAdHocWithHttpInfo(streamValuesItems)

 

Get AF Attribute given an AF Element path

 

    Set elem = client.element.GetByPath(ERD.CurrentContext(ThisDisplay))
    ElemDesc.Contents = elem.Description
    Dim attributes As PIItemsAttribute
    Set attributes = client.element.GetAttributes(elem.webId, 1000, False, False, False, 0)

 

Get current value given an AF Attribute path

 

  attributePath = ERD.CurrentContext(ThisDisplay) + "|" + AttrList.Text
    Set attr = client.attribute.GetByPath(attributePath)
    Set timedValue = client.Stream.GetEnd(attr.webId)
    AttrValue.Contents = timedValue.value

 

Get Event Frames given an AF database path

 

Set db = client.AssetData.GetByPath(dbPath)
Set efs = client.AssetData.GetEventFrames(db.webId, False, False, 100, True, 0, "", "*", "", elem.Name, elem.templateName, "", "", "None", "", "",

 

 

Final Remarks

 

This library will be updated for every new release of PI Web API in order to add the methods on this library which were added to the PI Web API release.

 

Please share your comments and thoughts about this new release! Will PI Web API Wrapper be useful in your upcoming projects?

Good day!!

 

We like to seek your interest level in an idea for our next programming hackathon. One of the most interesting concepts in the area of innovation and prototyping is Design Thinking.

 

In order to add more value to our hackathon and also enhance the quality of the event we are considering some mentorship and coaching opportunity on the subject during the hackathon.

 

The goal is to have a coach equip our hackers with the Design Thinking methodology. Such techniques will enable the participants to not only deliver better results at the hackathon but also take the knowledge back to their professional careers and become better PI professionals.

 

At the beginning of the hackathon we will include a presentation of 30-45 minutes to walk the participants through the process of design thinking and define the resources during the event. Throughout the event we will provide each team with a coach to consult with as well as certain level of access to the customer or someone who plays the role of a customer.

 

If we see a good amount of interest in the community we will take the next steps to include Design Thinking in our future hackathons!

Please fill in this two questions survey in order to let us know your interest: Design Thinking in OSIsoft Hackathons survey .

Thanks in advance for your support!

We are excited to present the first LATAM Regional Conference Programming Hackathonce 2017 winners!

 

The theme of this year's Hackathon was Analytics for Smarter Energy Aware Campus. UC Davis, one campus of the University of California, kindly provided 6 months of energy data. Participants were encouraged to create killer applications for UC Davis by leveraging the PI System infrastructure.

 

The participants had 10 hours to create an app using any of the following technologies:

  • PI Server 2017
  • PI Web API 2017
  • PI Vision 2017
  • PI OLEDB Enterprise 2016 R2

 

Our judges evaluated each app based on their creativity, technical content, potential business impact, data analysis and insight and UI/UX. Although it is a tough challenge to create an app in 10 hours, four groups were able to finish their app and present to the judges!

 

Prizes:

1st place: Drone Syma X5sw 2.4ghz 4 Canais Wifi Câmera , vouchers for trainings, one year free subscription to PI Developers Club

2nd place: Vouchers for trainings, one year free subscription to PI Developers Club

 

Without further do, here are the winners!

 

1st place - JLWM Engenharia

Team members: Luan Carlos Amaral Sandes; Joao Teodoro Marinho; Mateus Gabriel Santos; Willy Rodrigo de Araujo.

 

 

IMG_20170606_175253.jpg

 

They have created an app called Predictive Models and Gamification: reducing energy costs. The UC Davis' buildings would receive points if they consume less energy than estimated by the predictive models that the groups has built.

 

The team used the following technologies:

  • PI Vision

 

Here are some screenshots presented by this group!

 

 

2nd place - Connected 4.0

 

Team members:: Guilherme Tavares, Kaio Lima, Pablo Araya, Rômulo Lemes

 

They developed an app named Connected40 the Value of People.  This app is a web application that shows the buildings on a map. When the user clicks on a building, it will display the energy KPIs of the building. They've created a new KPI which is energy per person to compare the efficiency of the building among others.

 

The team used the following technologies:

  • HTML5/JavaScript
  • PI AF SDK
  • Google Maps JavaScript API

 

Here are some screenshots presented by this group!

 

 

Background

PI Vision offers a form of Element Relativity that is quite different from PI ProcessBook.  While we love the style of PI Vision ER, we missed some of the features from ProcessBook ER when using PI Vision. With this in mind, we decided to develop a PI Vision toolpane that has features similar to what was available in ProcessBook.

 

Features

  • Quickly swap between elements with a single click
  • Elements of Interest persistent across multiple sessions
  • Elements of Interest unique to each display
  • Minimal set-up
  • Robust search to populate list of Elements of Interest

 

Demo

 

Set Up

Required

  1. In Windows Explorer, navigate to the "%PIHOME%\PIVision\Scripts\app\editor\tools\ext" on your PI Vision web server; typically, it's located in "C:\Program Files\PIPC\PIVision\Scripts\app\editor\tools\ext". If this folder doesn't exist, create it.
  2. Copy contents of the repository into the above folder. You may exclude "Example.png" and "README.MD"

Optional

In order to utilize the Elements of Interest unique to each display, an AF database is created to store the configuration strings for each display.  Use of this feature requires that the user be able to read from the AF Server when loading pages and write to the AF Server when saving the list in order to update the configuration. The creation of the AF database, elements, and attributes is done automatically once the AF Server has been specified.

The AF Server where the configuration is stored must be specified in the setup.json file located in the %pihome%\PIVision\Scripts\app\editor\Tools\Ext\ folder. With JSON format, such as the below:

{ AFServer”:”<YourAFServerName>” }

Where <YourAFServerName> is replaced with whatever AF Server you’d like to store the configuration in.

Additionally, in order to write to the AF Database, you will need Kerberos Delegation enabled and the user making changes to the elements of interest pane will need to have write permissions on that Asset Server in order to create a new database, and element beneath it.

Lastly, in some cases it has been necessary to create a MIME type in IIS in order to read the JSON configuration file if one does not already exist. This can be accomplished by opening up IIS Manager > Select the site hosting PI Vision > Double click on "MIME Types" in the Features View > Right-click "add" and use the settings below

 

Procedures

Adding Elements to the Elements of Interest Pane

  1. Select a server from the list
  2. Verify the server connection was successful

Good: Bad:

  1. Specify search parameters:

Database (required): Which database should we search for elements in.

Root Element: Search only for elements below a particular element.  Default value searches entire database.

Element Name: Search only for elements with a particular name.  Wildcards (*) supported.  Default value is “*”

Template Name: Search only for elements with a particular template.  Default value is all templates.

Category Name: Search only for elements in a particular category.  Default value is all categories.

Add to Existing Results: Should the results of this search be added to the existing elements in the Elements of Interest.

 

Removing Elements from the Elements of Interest Pane

  1. Click the trash icon next to the element you want to remove from the list.

Display attributes for Element of Interest

  1. Click anywhere on the row for the element you want to view

 

Download

https://github.com/osipmartin/PI-Coresight-Custom-Symbols/tree/master/Community%20Samples/CoolTeam

 

Disclaimer

This tool is not an official OSIsoft solution.  As such, it will not be supported.  If you have issues, please contact myself, Anna Perry, or Robert Schmitz and we will try our best to help you.

Introduction

 

In 2014, I have written a blog post about Developing a PHP application using PI Web API. At that time, it was necessary to build the URL with strings concatenation in order to make HTTP request against PI Web API. With the Swagger specification that comes with PI Web API 2017 release, I was able to generate a PI Web API Client library for PHP. On this blog post, I will rewrite this PHP application in order to use this library instead of writing some lines of code to generate the URLs. Let's see how this work.

 

The source code package of this PHP application with PI Web API client is available on this GitHub repository.

 

Adding the library to the project

 

Download the PHP files from the  PI Web API Client library for PHP GitHub repository, extract the files and copy the lib folder to the root of your PHP application folder.

 

 

Comparing the piwebapi_wrapper.php files

 

Let's compare both piwebapi_wrapper.php files. The first one is from the 2014 blog post and second one is related to this blog post which uses the PI Web API client library.

 

piwebapi_wrapper.php with no client library.

 

<?php
class PIWebAPI
{
  public static function CheckIfPIServerExists($piServerName)
  {
  $base_service_url = "https://cross-platform-lab-uc2017.osisoft.com/piwebapi/";
  $url = $base_service_url . "dataservers";
  $obj = PIWebAPI::GetJSONObject($url);
  foreach($obj->Items as $myServer)
  {
  if(strtolower($myServer->Name)==strtolower($piServerName))
  {
  return(true);
  }
  }
  return (false);
  }

  public static function CheckIfPIPointExists($piServerName, $piPointName)
  {
  $base_service_url = "https://cross-platform-lab-uc2017.osisoft.com/piwebapi/";
  $url = $base_service_url . "points?path=\\\\" . $piServerName . "\\" . $piPointName;
  $obj1 = PIWebAPI::GetJSONObject($url);
  try {
  if(($obj1->Name)!=null)
  {
  return (true);
  }
  return (false);
  }
  catch (Exception $e)
  {

  }
  }

  public static function GetSnapshot($piServerName, $piPointName)
  {
  $base_service_url = "https://cross-platform-lab-uc2017.osisoft.com/piwebapi/";
  $service_url = $base_service_url . "points?path=\\\\" . $piServerName . "\\" . $piPointName;
  $obj_pipoint = PIWebAPI::GetJSONObject($service_url);
  $url = $obj_pipoint->Links->Value;
  $obj_snapshot = PIWebAPI::GetJSONObject($url);
  return ($obj_snapshot);

  }

  public static function GetRecordedValues($piServerName, $piPointName,$startTime,$endTime)
  {
  $base_service_url = "https://cross-platform-lab-uc2017.osisoft.com/piwebapi/";
  $service_url = $base_service_url . "points?path=\\\\" . $piServerName . "\\" . $piPointName;
  $obj_pipoint = PIWebAPI::GetJSONObject($service_url);
  $url = $obj_pipoint->Links->{'RecordedData'} ."?starttime=" . $startTime . "&endtime=" . $endTime;
  $obj_rec = PIWebAPI::GetJSONObject($url);
  return ($obj_rec);

  }

  public static function  GeInterpolatedValues($piServerName, $piPointName,$startTime,$endTime,$interval)
  {
  $base_service_url = "https://cross-platform-lab-uc2017.osisoft.com/piwebapi/";
  $service_url = $base_service_url . "points?path=\\\\" . $piServerName . "\\" . $piPointName;
  $obj_pipoint = PIWebAPI::GetJSONObject($service_url);
  $url = $obj_pipoint->Links->{'InterpolatedData'} ."?starttime=" . $startTime . "&endtime=" . $endTime . "&interval=" . $interval;
  $obj_int = PIWebAPI::GetJSONObject($url);
  return ($obj_int);

  }

  private static function GetJSONObject($url)
  {
  $username = "username";
  $password= "password";
  $ch = curl_init($url);
  curl_setopt($ch, CURLOPT_HEADER, false);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
  curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
  curl_setopt($ch, CURLOPT_USERPWD, $username.":".$password); 
  $result = curl_exec($ch);
  $json_o=json_decode($result);
  return ($json_o);
  }
}

 

 

piwebapi_wrapper.php with PHP client library.

 

 

<?php
include_once ( __DIR__ . "\\lib\\PIWebApiLoader.php");
use \PIWebAPI\Client\PIWebApiClient;
class PIWebAPI
{
  public $piwebapi= NULL;
  public function __construct()
  {
  $this->piwebapi = new PIWebApiClient("https://cross-platform-lab-uc2017.osisoft.com/piwebapi","username", "password", "BASIC", FALSE, TRUE);
  }


  public function checkIfPIServerExists($piServerName)
  {

  try 
  {
  $response = $this->piwebapi->dataServer->dataServerGetByNameWithHttpInfo($piServerName);
  if($response[1] == 200)
  {
  return true;
  }
  else {
  return false;
  }
  } 
  catch (Exception $e) {
  return false;
  }


  }

  public function getPIPoint($piServerName, $piPointName)
  {
  try
  {
  $path = "\\\\" . $piServerName . "\\" . $piPointName;
  $response = $this->piwebapi->point->pointGetByPathWithHttpInfo($path);
  if($response[1] == 200)
  {
  return $response;
  }
  else {
  return null;
  }
  }
  catch (Exception $e) {
  return null;
  }
  }

  public function getSnapshot($webId)
  {
  return $this->piwebapi->stream->streamGetEnd($webId);
  }

  public function getRecordedValues($webId, $startTime, $endTime)
  {
  return $this->piwebapi->stream->streamGetRecorded($webId, null, null, $endTime, null, null, null, null, $startTime);
  }

  public function  getInterpolatedValues($webId, $startTime, $endTime, $interval)
  {
  return $this->piwebapi->stream->streamGetInterpolated($webId, null, $endTime, null, null, $interval, null, $startTime);
  }
}

 

The readme.rd from the GitHub repository provides information about how to use this library within a PHP application. Look how easy it is to switch from Basic to Kerberos authentication. Just change the value for $authMethod (which is an input of the PIWebAPIClient constructor) from  "BASIC" to "KERBEROS".

 

 

Updating index.php

 

The objects returned by the methods from the PIWebAPI class are different when compared to the older project. The reason is that the Swagger Code generation will generate classes basic on the specification Json description. As a result, the way you extract the values on the index.php file to render on the tables is a little different:

 

function displaySnapValues($SinusoidSnap) {
  ?>
  <h2>Snapshot Value of Sinusoid</h2>
  <br />
  <table style="width: 20em; border: 1px solid #666;">
  <tr>
  <th>Value</th>
  <th>Timestamp</th>
  </tr>
  <tr>
  <td><?php echo $SinusoidSnap['value'][0]; ?></td>
  <td><?php echo $SinusoidSnap['timestamp']->format('Y-m-d H:i:s'); ?></td>
  </tr>
  </table>
  <br />
  <br />
  <?php
}
function displayRecValues($SinusoidRec) {
  ?>

  <h2>Recorded Values of Sinusoid</h2>
  <br />
  <table style="width: 20em; border: 1px solid #666;">
  <tr>
  <th>Value</th>
  <th>Timestamp</th>
  </tr><?php
  foreach ( $SinusoidRec['items'] as $item) {
  echo "\n<tr>";
  echo "\n\t<td>" . $item['value'][0] . '</td>';
  echo "\n\t<td>" . $item['timestamp']->format('Y-m-d H:i:s') . "</td>";
  echo "\n</tr>";
  }
  ?>

  </table>


  <br />
  <br />
  <?php
}
function displayIntValues($SinusoidInt) {
  ?>
  <h2>Interpolated Values of Sinusoid</h2>
  <br />
  <table style="width: 20em; border: 1px solid #666;">
  <tr>
  <th>Value</th>
  <th>Timestamp</th>
  </tr>
  <?php
  foreach ( $SinusoidInt['items'] as $item) {
  echo "\n<tr>";
  echo "\n\t<td>" . $item['value'][0] . '</td>';
  echo "\n\t<td>" . $item['timestamp']->format('Y-m-d H:i:s') . "</td>";
  echo "\n</tr>";
  }
  ?>
  </table>
  <?php
}

 

Conclusion

 

Although PHP is not my favorite language for web development, I have seen some questions on PI Square about using PHP to retrieve PI data through PI Web API. As a result, I think this library will be very useful for those PHP developers who wants to add value to their application by integrating it with the PI System.

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary Per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

Conclusion

We covered a lot of ground in this 9-part series because a lot of what was being covered was new ground.  You were introduced to 4 brand new methods to PI AF SDK 2.9.  Three of those were aggregation methods, which has got to be a much welcomed feature in AFEventFrameSearch.  And FindObjectFields might be the first one any developer checks out for its sheer speed and versatility not just for aggregation but for lightweight detail reporting.  To rehash what was covered:

 

  • Part 4 We showed the old way of doing things with the classical FindEventFrames.  This provided a baseline in performance for us to benchmark against the other new methods.
  • Part 5 We saw the new lightweight FindObjectFields method to return a skinny set of columns.  We looked at all 3 overloads of this method, each of which is concerned about casting first from generic object to the specific underlying type, followed perhaps by additional casting or converting to the type you desire.
  • Part 6 We saw the Summary method and discovered there is an event weighted overload as well as a general weighting overload to produce custom weightings beyond just time weighted.
  • Part 7 We saw how to use the GroupedSummary method to summarize with groupings, which allowed us to make fewer calls.
  • Part 8 We finished off with showing how to use a compound AFSummaryRequest to produce a 2-level grouping.  It was a tad bit complicated but did have great performance.

 

 

Tips to Remember

 

General:

  • Use CaptureValues() to see the performance benefits from server-side filtering.
  • Classes inherited from AFSearch, such as AFEventFrameSearch, now implement IDisposable starting with AF SDK 2.9.  You should consider wrapping your calls inside a using block, or else issue an explicit Close() when you are finished with your search activities.
  • When composing a query string, any values containing embedded blanks should be wrapped inside single or double quotes.
  • Your time string for output queries should be output using the "O" Round-trip specifier.
  • For best performance, you probably want to choose the method that makes the fewest calls to the server.

 

FindObjectFields:

  • If you are working with detail records, you should strongly consider including ID as one of the input fields.  That way if you ever have the need to perform further drilling into a specific event frame, you have the unique ID which can help you quickly locate the full event frame in question.
  • There is no weighted overload for FindObjectFields.  You would be expected to include your own weighting field (e.g. Duration or custom) in the returned set of values.
  • The underlying type of any attribute's value will be AFValue.
  • You may use fields or properties for your DTO class.
  • For the auto-mapped overload, you will have to use the ObjectField decorator to map the source attribute name that happens to begin with a "|" to your desired DTO field name.
  • For event frame properties and the auto-mapped overload, the default is to use the same property name for the mapping.  However, you may override this default.

 

AFSummaryRequest:

  • Is limited to no more than 2 levels of groupings.
  • For 1 grouping level, you should just use Summary or GroupedSummary depending upon your needs since these are less complicated and have a simpler packaging of the results.
  • Based on previous bullets, you probably would only use AFSummaryRequest precisely when you need 2-level groupings.

 

Async

Other than showing the method names in Part 1, we did not mention any of the async methods or show their usage.  But they are there and easily discernible by seeing a CancellationToken among the parameters.  Once LiveLibrary is active for PI AF Client 2017, you are encouraged to review the online help for:

 

  • BinnedSummaryAsync
  • FrequencyDistributionAsync
  • GroupedSummaryAsync
  • HistogramAsync
  • SummaryAsync (both event and general weighting overloads)

 

If you are curious as to why FindObjectFields does not have an async counterpart, keep in mind that FindObjectFields makes paged calls.  You are always capable of break your processing, which will stop requests for more pages of data.

 

Weighting

While the more natural weighting with the data archive is probably time weighted, event frames are not stored in the data archive but rather in SQL Server.  It should be no surprise that event weighting is the more natural or default weighting when dealing with event frames.  Out of the new AFEventFrameSearch aggregation methods, only Summary and SummaryAsync offer some other weighting overload other than event weighted.  You aren't limited to just time weighted as the lone alternative.  The new overloads are flexible to allow custom weightings.

 

FindObjectFields doesn't allow for weightings because it's not an aggregation method.  You may still use FindObjectFields but you should include the weighting field as part of the set of skinny columns to be returned.

 

Binning

I did not show any examples of binning.  That might be a future topic.  But you should be aware that these methods exist.

 

  • For discrete values such as integers or strings, FrequencyDistribution and FrequencyDistributionAync generates a <gasp> frequency distribution.
  • For floating point values, you would want to bin by ranges.  See Histogram or HistogramAsync for that.  Note that your requested ranges do not have to be in evenly-spaced intervals.
  • Why not have summaries by bins?  For this there is the BinnedSummary and BinnedSummaryAsync methods.

 

This is the End?

Or is it?  Don't be surprise if I do a future series about binning.

 

Thanks for reading the series.  I hope you enjoyed it.  Please remember to use this knowledge for good and not evil.

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary Per Manufacturer

Part 8 - Compound AFSummaryRequest

       Part 9 - Conclusion

 

Let's Make Only ONE Call

I'm going to assume that you haven't jumped blindly into this topic for the first time.  It should be a safe bet that you've read Parts 6 and 7 regarding Summary and GroupedSummary respectively.  It has boiled down to this: I don't want to make repeated calls, because we know each call to the server takes a performance hit.  Summary needed to be called 3 times for this use case, and GroupedSummary twice.  I want to issue one and only one call.  Plus I don't want to have know to know all Manufacturers or Models before I query for them.  Maybe I absolutely don't know that info ahead of time and that's why I'm doing this search in the first place.

 

Which brings us to AFSummaryRequest, which will perfectly fit the bill.  Technically it is not a method to the AFEventFrameSearch or AFSearch classes.  It's a concrete method in the OSIsoft.AF.Data namespace that implements the abstract method AFAggregateResult.  Other aggregation methods, like Summary and GroupedSummary, call AFSummaryRequest themselves for the most common method signatures that need a 1-level summary.  As we want a 2-level grouping, AFSummaryResult is the best choice for our use case.  With that in mind, we should be forgiving that the code to use it is a bit more complicated.  Note that AFSummaryRequest limits you to no more than 2 groupings.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    using (var search = new AFEventFrameSearch(database, "Compound Request", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //While we eventually want an average, it will be calculated from Total and Count.
        var desiredSummaryTypes = AFSummaryTypes.Count | AFSummaryTypes.Total;

        //Here we make only 1 call to the server but we must build a compound AFSummaryRequest.
        //The GroupBy order is opposite than what you would intutively think: Model and then Manufacturer.
        //First we bundle the AFSummaryRequest.
        var compoundRequest = new AFSummaryRequest("Duration", desiredSummaryTypes)
                                    .GroupBy<string>("|Model")
                                    .GroupBy<string>("|Manufacturer");

        //We send the request as a member of IEnumberable<AFAggregateRequest>.
        //Since we pass a collection of one member, we get a collection of one member back.
        //So we grab that one member and cast it appropriately.
        var aggResult = search.Aggregate(new[] { compoundRequest })[0] as AFCompoundPartitionedResult<string, string>;

        //Unwrap the results.
        foreach (var kvp in aggResult.PartitionedResults)
        {
            var mfr = kvp.Key.PrimaryPartition;
            var model = kvp.Key.SecondaryPartition;

            var summaries = kvp.Value;

            var totalVal = summaries[AFSummaryTypes.Total];
            var countVal = summaries[AFSummaryTypes.Count];
            var stats = new DurationStats();

            if (countVal.IsGood)
            {
                stats.Count = countVal.ValueAsInt32();
                if (totalVal.IsGood)
                {
                    stats.TotalDuration = ((AFTimeSpan)totalVal.Value).ToTimeSpan();
                }
                summary.AddToSummary(mfr, model, stats.TotalDuration, stats.Count);
            }
        }
    }
}

 

 

There you have it.  Not exactly pretty.  But you can't argue with the results since this was the fastest method for my use case.

 

I want to reiterate that you may have no more than 2 levels of grouping for AFSummaryRequest.  And review lines 15-16 to see that the grouping is inside to outside.  That is if we want to group by Manufacturer first and Model second then when we compose the AFSummaryRequest the first GroupBy is by Model and the second GroupdBy is Manufacturer.

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

We are at the same spot where we ponder if AFSummaryRequest is the fastest of the methods.  It appears to be so for my particular use case of reporting by Manufacturer and Model.  If we were to ignore my use case, and compare AFSummaryRequest to Part 6's Bonus Summary and Part 7's Bonus GroupedSummary, both of which issued one call on the same data set, here's how those metrics line up:

 

Metric
SummaryGroupedSummaryCompound AFSummaryRequest
Total GC Memory (MB)4.4812.247.29
Working Set Memory (MB)52.4861.8465.36
Network Bytes Sent4.77 KB4.85 KB5.00 KB
Network Bytes Received260.02 KB1.98 KB3.68 MB
Client RPC Calls101010
Client Duration (ms)534.01913.72992.2
Server RPC Calls101010
Server Duration (ms)353.81472.82222.2
Elapsed Time00:01.100:02.600:03.7

 

I know I sound like a broken record but the same advice applies: think through your application and pick the right tool for the right job.  Which one gives the correct results with making the fewest calls to the server?

 

Up Next: End of the Series

We conclude this 9-part series naturally enough with post that I call Part 9!

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

Query One Level Up

GroupedSummary and Summary have something in common.  They both require a priori knowledge of what you will be summarizing before you can actual summarize it.  For Summary, this required summarizing per the inner loop of Model, which required 3 calls (one for each of our 3 models).  For GroupedSummary, we can reduce the number of calls to the server by making a call on the the outer loop per Manufacturer.  While we do need to know the manufacturers to filter upon for GroupedSummary, we don't need to know the models.

 

We will do something similar as we did the Summary in Part 6:

  • Get a priori list of Manufacturers
  • Build a new token for the given Manufacturer
  • Issue the GroupedSummary call
  • Peel back the results to feed to my DurationStats and StatsTracker

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> baseTokens)
{
    //Absolutely critical to have a priori list of Manufacturers
    var mfrList = summary.Keys.ToList();

    foreach (var mfr in mfrList)
    {
        var tokens = baseTokens.ToList();
        tokens.Add(new AFSearchToken(AFSearchFilter.Value, mfr, "|Manufacturer"));

        using (var search = new AFEventFrameSearch(database, "GroupedSummary Example", tokens))
        {
            //Opt-in to server side caching
            search.CacheTimeout = TimeSpan.FromMinutes(5);

            //While we eventually want an average, it will be calculated from Total and Count.
            var desiredSummaryTypes = AFSummaryTypes.Count | AFSummaryTypes.Total;
            var groupedField = "|Model";
            var summaryField = "Duration";

            var perMfr = search.GroupedSummary(groupedField, summaryField, desiredSummaryTypes);

            foreach (var grouping in perMfr.GroupedResults)
            {
                var model = grouping.Key.ToString();
                var totalVal = grouping.Value[AFSummaryTypes.Total];
                var countVal = grouping.Value[AFSummaryTypes.Count];

                var stats = new DurationStats();

                if (countVal.IsGood)
                {
                    stats.Count = countVal.ValueAsInt32();
                    if (totalVal.IsGood)
                    {
                        stats.TotalDuration = ((AFTimeSpan)totalVal.Value).ToTimeSpan();
                    }
                    summary.AddToSummary(mfr, model, stats.TotalDuration, stats.Count);
                }
            }
        }
    }
}

 

While we did have some similarities, where we invoked the server call is very different.  Here with GroupedSummary, we make the call in our outer loop so we will have less trips to the server.  For Summary in Part 6, we made the call inside the inner loop.  Also the returned results are quite different, though the concept of what we do with them is the same: peel back the returned dictionary accordingly and have them conform to my output objects.

 

The metrics shown in Part 2 would make you think GroupedSummary is faster than Summary.  In general, this is really not true.  For my particular use case it is true, but that's because there are more server calls that my app is making to Summary than for GroupedSummary.  Do not walk away thinking you would want to avoid Summary.  Instead, you should not hesitate to use it for a better use case.

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

 

BONUS: GroupedSummary Using ONE Call

Let's come up with a better use case where we only need to issue one call.  Allow me once again to temporarily change my requirements on the end report, purely for illustration purposes.  Let's imagine I no longer am interested in the average and counts per manufacturer and model.  Instead I want to summarize the same data set as a whole but I only care about models.  In this new scenario I have absolutely no concern about manufacturers.  The new report would look like:

 

Manufacturer  Model            Count Avg Duration

------------- ------------ --------- ----------------

<Any>         DQ-M0L           8,136 03:53:21.4859882

<Any>         Nimbus 2000      1,499 03:44:28.8192128

<Any>         SWTG-3.6        13,678 03:53:35.3165667

------------- ------------ --------- ----------------

            1            3    23,313

 

For the code to do that, I don't need to initialize my summary object to populate itself from an AFTable.

 

//I still use StatsTracker for conformity but we don't need to initialize this from our AFTable  
var summary = new StatsTracker();  

 

That is the summary instance I will pass to my new method, which now eliminates 1 level of looping.  However, I will need to later sort the results so I am going to pass summary by ref.

 

public void GetSummaryByMfrAndModel(ref StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    summary = new StatsTracker();

    //In this bonus test, we want to only issue one GroupedSummary call.
    //Rather than rigorously issue separate calls per Manufacturer, I instead isssue one call on grouped on Model for all Manufacturers.
    //The downside is I lose the individual Manufacturer names.

    using (var search = new AFEventFrameSearch(database, "GroupedSummary Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //While we eventually want an average, it will be calculated from Total and Count.
        var desiredSummaryTypes = AFSummaryTypes.Count | AFSummaryTypes.Total;
        var groupedField = "|Model";
        var summaryField = "Duration";

        var groupedSummary = search.GroupedSummary(groupedField, summaryField, desiredSummaryTypes);

        foreach (var grouping in groupedSummary.GroupedResults)
        {
            var model = grouping.Key.ToString();
            var totalVal = grouping.Value[AFSummaryTypes.Total];
            var countVal = grouping.Value[AFSummaryTypes.Count];

            var stats = new DurationStats();

            if (countVal.IsGood)
            {
                stats.Count = countVal.ValueAsInt32();
                if (totalVal.IsGood)
                {
                    stats.TotalDuration = ((AFTimeSpan)totalVal.Value).ToTimeSpan();
                }
                summary.AddToSummary("<Any>", model, stats.TotalDuration, stats.Count);
            }
        }
    }
    //Sort the results.  They have the same Manufacturer "<Any>" but the Models will be alphabetical.
    summary = summary.SortByKeys();
}

 

I am expecting to get back 3 rows, so I sort the results before returning from my method.  Let's review the metrics with making that one bonus GroupedSummary call and let's compare that to making one bonus Summary call.

 

Metric
SummaryGroupedSummary
Total GC Memory (MB)4.4812.24
Working Set Memory (MB)52.4861.84
Network Bytes Sent4.77 KB4.85 KB
Network Bytes Received260.02 KB1.98 KB
Client RPC Calls1010
Client Duration (ms)534.01913.7
Server RPC Calls1010
Server Duration (ms)353.81472.8
Elapsed Time00:01.100:02.6

 

For the right use cases, both of these methods are extremely fast, and should be welcome in your tool bag.  Don't shy away from using Summary or GroupedSummary because one table shows sluggish performance.  Use your noggin and pick the right tool for the right job.  The emphasis should be on producing the desired results with the fewest trips to the server.

 

 

Up Next: Name That Tune In One Call

Putting aside the bonus section, let's return to the original report by Manufacturer and Model.  To repeat the pattern you should have witnessed in the progression of each method in parts 4 - 7.

Part 4: Heavy detail records

Part 5: Light detail records

Part 6: Aggregation per inner loop

Part 7: Aggregation per outer loop

 

For each successive example we were making fewer calls or receiving fewer records.  The good news is that Summary and GroupedSummary are downright miserly on resources consumed.  The bad news is this whole a priori knowledge requirement as well as making multiple calls which degrades performance.  Wouldn't it be great to be able to make only ONE call and to do so without knowing what the heck we want to summarize in the first place?  That will be covered in Part 8.

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

A Bona Find Aggregation Method

We've covered 2 different ways to produce our summaries but neither of those approaches used a true aggregation method.  Instead they both returned detailed rows where we had to apply our own custom aggregation.  In the case of FindEventFrames, the detail rows were heavyweight event frames.  In the case of FindObjectFields, the detail rows were data container records.  For this brand-new AFEventFrameSearch.Summary, we will getting back an aggregation and you will note what is sent from across the network to us (as recorded in Network Bytes Received) is only a teeny tiny bit memory.

 

Summary requires a priori knowledge of what you want to be summarizing.  In our case, we want to summarize by Manufacturer and Model so we must know all the Manufacturers and Models we wish to summarize before we can actually summarize them.  This was discussed in Part 3.  I chose to read an AFTable and populate a model-keyed dictionary inside a manufacturer-keyed dictionary.  You are in no way restricted to do the same.  You are encouraged to find the solution that best fits your own database and needs, and I welcome you sharing your creative solutions back on PISquare one day.

 

You may also remember that in Part 2 the Summary method seemed to be the slowest of the new methods.  It really isn't.  The problem is I am trying to have all these new methods produce the exact same output, so making multiple calls on Models within Manufacturers is really not the best use case of Summary.  On the other hand, it is a very good example of syntax on how to issue a Summary call, as well as what to do with the results that come back from that call.  Let's focus on that as the main lesson to be learned in the code below.

 

The Highlights

My a priori requirement is taken care of by my dictionary in a dictionary.  However, I will need to get an independent list of the keys to the dictionaries.

 

I will also need to issue a Summary per Model.  This means I must use the same base tokens or query that I used for our previous examples, and modify them for each Summary call.  Again, I could take the lazy or sloppy approach and only worry about Model since my current data set had 3 unique models.  But that code could break in the future if I were ever to add a Model with the same name to a different Manufacturer.  Instead, I will take a rigorous approach and truly query by Manufacturer and then Model.

 

All of this is to say that I will be looping first over Manufacturers, and then secondly over the Models.  Then I will modify the tokens or query string for inside the inner loop.  Because I will modify the input tokens/query repeatedly, I have renamed the input argument from "tokens" to be "baseTokens".

 

The final steps will be to receive the results from Summary, and unwrap them to conform to my DurationStats and StatsTracker objects discussed in Part 3.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> baseTokens)
{
    //Absolutely critical to have a priori list of Manufacturers and Models
   //Get independent list of Manufacturers
    var mfrList = summary.Keys.ToList();

    foreach (var mfr in mfrList)
    {
        //Get independent list of Models for given Manufacturer
        var modelSubList = summary[mfr].Keys.ToList();

        foreach (var model in modelSubList)
        {
            //Safest Technique: via tokens.  
            //Get independent copy to modify inside loop
            var tokens = baseTokens.ToList();
            tokens.Add(new AFSearchToken(AFSearchFilter.Value, mfr, "|Manufacturer"));
            tokens.Add(new AFSearchToken(AFSearchFilter.Value, model, "|Model"));

            //Starting with AF 2.9, AFSearch implements IDisposable
            using (var search = new AFEventFrameSearch(database, "Summary Example", tokens))
            {
                //Opt-in to server side caching
                search.CacheTimeout = TimeSpan.FromMinutes(5);

                var desiredSummaryTypes = AFSummaryTypes.Count | AFSummaryTypes.Total;

                var perModel = search.Summary("Duration", desiredSummaryTypes);

                var totalVal = perModel.SummaryResults[AFSummaryTypes.Total];
                var countVal = perModel.SummaryResults[AFSummaryTypes.Count];

                var stats = new DurationStats();

                //Unwrap the returned results as needed
                if (countVal.IsGood)
                {
                    stats.Count = countVal.ValueAsInt32();
                    if (totalVal.IsGood)
                    {
                        stats.TotalDuration = ((AFTimeSpan)totalVal.Value).ToTimeSpan();
                    }
                    summary.AddToSummary(mfr, model, stats.TotalDuration, stats.Count);
                }
            }
        }
    }
}

 

 

The above example uses query tokens.  I mentioned in Part 3 you could have used a query string.  If you wanted a string instead of tokens, I would have an input string argument named "baseQuery" containing:

 

$"AllDescendants:{allDescendants} Template:'{templateName}' Start:>={startTime.ToString("O")} End:<={endTime.ToString("O")} '{attrPath}':>={attrValue}"

 

Then inside the inner loop of Model, lines 16-18 would become:

 

var query = $"{baseQuery} |Manufacturer:'{mfr}' |Model:'{model}'";

 

Note the use of single quotes around {mfr} and {model}.  For model this is an absolute must have with our data, because we do have one model ("Nimbus 2000") that contains an embedded blank in its name.  For mfr, we did this for future proofing in case we ever add a Manufacturer with a blank in its name.  You may recall in Part 3 I cautioned that if it's a name or a path that the safest route is to wrap it in single quotes.  This helps make your code less fragile.

 

Event versus Time Weighting

For the Summary overload we used in the code above, the result is event weighted.  Normally with data coming from a process historian, I tend to first think in terms of time weighted values.  But we're working with event frames here, so my inclination is that the values are event weighted, that is there is a value associated with the entire event frame.  But that's me.  But you may be interested in getting back a time weighted number, so you might ask "Is there a time weighted overload?"

 

The trick answer is No.  While it's true there is not a Summary overload that allows you to pass an AFCalculationBasis.TimeWeighted enumeration, there is an overload that accepts a general weighting field as the 3rd argument.  This means you aren't restricted to either event weightings or time weightings, but you may pass a custom weighting!  The restriction here is that you pass the name of the weighting field, and that field must belong to the event frame.  For a time weighted weighting, the name of the weighting field could be "Duration" or perhaps you have another time span attribute defined on your event frame.

 

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

 

Caution again about CaptureValues()

The performance is realized because all of my event frames have captured values.  This means filtering by wind velocity, manufacturer, and model - all of which are attributes - is performed on the server.  That greatly reduces the network load.

 

I don't know you consider it a good thing or a bad thing that the code above also works if you have not captured values.  Yes it will work.  But it may be as slow or slower than FindEventFrames.

 

Use the Right Tool for the Right Job

The above example shows correct syntax and how to peel back the results as you need.  Admittedly, a 2-level summary is not a good use case for Summary.  I would absolutely reject using this method if I had to query model 5 times or more (that is make 5 or more invocations of Summary).  I may possibly consider it if I knew I had less than 5 models but would likely reject it as the method of choice unless I only had to make 1 or 2 calls.  With 1 call, it's a no-brainer: Summary is the right choice.  Would you like proof?

 

BONUS: Summary Using ONE Call

Let's come up with a better use case where we only need to issue one call.  Allow me to temporarily (just for illustration purposes) change my requirements on the end report.  I no longer am interested in the average and counts per manufacturer and model.  Instead I want to summarize over the exact same data set as a whole.  The new report would look like:

 

Manufacturer  Model            Count Avg Duration

------------- ------------ --------- ----------------

<All>         <All>           23,313 03:52:55.3506627

------------- ------------ --------- ----------------

            1            1    23,313

 

I get the exact same record count as the original report in Part 2, which shouldn't be surprising since I use the exact same filter.  For the code to produce the above report, I don't need to initialize my summary object to populate itself from an AFTable.

 

//I still use StatsTracker for conformity but we don't need to initialize this from our AFTable
var summary = new StatsTracker();

 

That is the summary instance I will pass to my new method, which now eliminates 2 levels of looping.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "Better Summary Use Case", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        var desiredSummaryTypes = AFSummaryTypes.Count | AFSummaryTypes.Total;

        var oneCallSummary = search.Summary("Duration", desiredSummaryTypes);

        var totalVal = oneCallSummary.SummaryResults[AFSummaryTypes.Total];
        var countVal = oneCallSummary.SummaryResults[AFSummaryTypes.Count];

        var stats = new DurationStats();

        if (countVal.IsGood)
        {
            stats.Count = countVal.ValueAsInt32();
            if (totalVal.IsGood)
            {
                stats.TotalDuration = ((AFTimeSpan)totalVal.Value).ToTimeSpan();
            }
            summary.AddToSummary("<All>", "<All>", stats.TotalDuration, stats.Count);
        }
    }
}

 

Since I get back only 1 row, there is no need to sort the results.  Let's review the metrics with making that one call:

Metric
Summary
Total GC Memory (MB)4.48
Working Set Memory (MB)52.48
Network Bytes Sent4.77 KB
Network Bytes Received260.02 KB
Client RPC Calls10
Client Duration (ms)534.0
Server RPC Calls10
Server Duration (ms)353.8
Elapsed Time00:01.1

 

Wow, that IS FAST!!!

 

 

Up Next: Reduce the Calls to the Outer Loop

Putting aside the bonus section, let's return to the original report by Manufacturer and Model.  We had to drill down into 2 loops to build our Summary call per Model.  In Part 7 we reduce the number of calls by making a call in the outer loop per Manufacturer.  We will do this with the GroupedSummary method.  See you in Part 7.

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

Welcome to the Halfway Point

We have reached the halfway point in this series.  I want to thank you for sticking with it.  Both of you.  In this part will be look into the brand new FindObjectFields which is very lightweight.  You may want to keep a fire extinguisher handy because this method is blazing fast (when compared to FindEventFrames).  Caveat: you only see performance benefits if you've called CapturedValues() on the event frames in question.

 

I don't want to sway your opinion but let me say that this became my favorite new method in PI AF SDK 2.9.  Besides being lightweight and super fast, it does return detail rows so it has flexibility for so many other applications, not just aggregation.  FindObjectFields has 3 different overloads.  We only covered 2 in the lab, but we will cover all 3 here.

 

Besides the sheer lightweightness of FindObjectFields when compared to FindEventFrames, there is another big, BIG difference.  The one call to FindObjectFields returns the data in the same call.  No need for a separate GetValues() call.  No need for custom chunking.

 

Plain Overload

This overload wasn't covered in the UC 2017 lab.  Essentially the values all come over as object so one of the first steps you will undertake is most likely casting to its underlying specific type.  And if that type is want you want to ultimately work with, then you will have to perform an additional cast or convert.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "FindObjectFields Example 1", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //The order of these fields determines the order of returned values in IList<object>
        var fields = "|Manufacturer |Model Duration";

        //returns IEnumerable<IList<object>> where each IEnumerable
        //represents one event frame, and the IList<object> contains 
        //values from each of the specified fields.  From our example,
        //we will have 3 values returned per event frame.
        var records = search.FindObjectFields(fields, pageSize: 10000);

        foreach (var record in records)
        {
            //Read data AND cast appropriately
            var mfr = ((AFValue)record[0]).Value.ToString();
            var model = ((AFValue)record[1]).Value.ToString();
            var duration = ((AFTimeSpan)record[2]).ToTimeSpan();

            //Summary overload is for TimeSpan
            summary.AddToSummary(mfr, model, duration, 1);
        }
    }
}

 

In Line 10, we specify the order of desired fields in a (mostly) blank delimited string.  If you have an attribute path that contains an embedded blank, you would need to wrap the path in single quotes, so they would serve as a delimiter as well.  As would double quotes, but that's a topic for another day.  Internally, FIndObjectFields will parse this string into its own IList<string>, something like { "|Manufacturer", "|Model", "Duration" }.

 

What's returned by the FindObjectFields is IEnumerable where each iteration of the IEnumerable is a different event frame.  The report of Part 2 shows we have over 23000 event frames, so we would expect 23000 records to enumerate over.  The payload of each iteration (or event frame or record) is an IList<object> where each indexed item is a value for the associated field.  In our example, index[0] is the Manufacturer value, index[1] is the Model, and index[2] is the Duration.

 

Since each record comes back as IList<object>, it would be your duty as developer to cast each object to its underlying data type.  First and foremost, any attribute will be returned as an AFValue and therefore must be cast into an AFValue before you can do anything else with it.  Any value from a property on the event frame will be returned with the same data type as the property.  Since Duration is an AFTimeSpan, I must first cast to AFTimeSpan, which I can then convert to TimeSpan if so desired.

 

A word about pageSize here ... with FindEventFrames the larger the pageSize the larger the memory needed to hold all the objects.  However, since this is lightweight, and maybe 10X smaller than the heavy objects from FindEventFrames, using a pageSize of 10000 with FindObjectFields still uses a comparable amount of memory as FindEventFrames(pageSize: 1000).

 

Auto-Mapped DTO Class

The next overloads will use DTO (Data Transfer Object) classes.  This particular overload will Auto-Map field names into your DTO class.  The help file for 2.9 shows this quite well.

 

A DTO is a simple class that will be our lightweight data container.  You could also use a POCO (Plain Old Class Object) but the more definitions you put in your container class the less lightweight it becomes.  I don't want to get into an argument over the subtle differences between DTO and POCO because such academic arguments are as enjoyable as chewing on tin foil.  You are invited to research on the web to learn more.  I include 2 links below.

 

P of EAA: Data Transfer Object

c# - POCO vs DTO - Stack Overflow

 

Before we can use the overload, we must first define our DTO class.  There are a couple of critical rules to apply:

  1. Any type for an attribute value must be an AFValue.
  2. The type for an event frame property must exactly match the type on the event frame.
  3. If the name of the entity on the event frame does not contain a blank, you may keep the original name.
  4. If you wish to change the name in your DTO class, you will use the AFSearch.ObjectField decorator.
  5. Since all attribute paths must begin with "|", which is not allowed in class field or property names, you must use the AFSearch.ObjectField decorator to provide a mapping to your DTO property.
  6. Your DTO may declare your objects as fields or properties.  The example below uses my own personal preference (properties).

 

    public class LightweightAutomapDto
    {
        // Field mapped using default name.
        public AFTimeSpan Duration { get; set; }

        // Attribute value mapped to property using 'ObjectField' attribute.
        [AFSearch.ObjectField("|Manufacturer")]
        public AFValue Manufacturer { get; set; }

        // Attribute value mapped to property using 'ObjectField' attribute.
        [AFSearch.ObjectField("|Model")]
        public AFValue Model { get; set; }
    }

 

Based on the mapping rules above, you will note:

  • I am using the name "Duration" exactly as it is named on the event frame.
  • The data type for my "Duration" is AFTimeSpan because that's what the event frame's Duration is.
  • Both my attributes must provide a ObjectField mapping.
  • Both my attributes have a data type of AFValue.

 

How would that look like in code?

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "Automap DTO Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        var records = search.FindObjectFields<LightweightAutomapDto>(pageSize: 10000);

        foreach (var record in records)
        {
            //Read data from 1 event frame via the current DTO container
            var mfr = record.Manufacturer?.Value.ToString();
            var model = record.Model?.Value.ToString();

            //Summary overload is for TimeSpan
            summary.AddToSummary(mfr, model, record.Duration.ToTimeSpan(), 1);
        }
    }
}

 

 

DTO with Custom Factory

The 3rd overload is quite interesting and was almost left out of the UC lab for fear of course length.  I am glad I included it because it soon became my favorite of the overloads.  If you choose to use this overload, then you absolutely MUST provide your own custom factory to perform the transfer of data.

 

Why would you want to do use this?  Let's consider the Auto-mapped DTO and what I would like to try differently:

  • The attribute values must be AFValue but I want Manufacturer and Model to be strings.
  • I want my Duration property to be a TimeSpan instead of the AFTimeSpan as it is on the event frame.

 

The resulting DTO class is far, far simpler and laid out exactly like I want it.  We don't have to worry about AFSearch.FindObject fields because our custom factory will take care of transfer.

 

public class LightweightDtoForCustomFactory
{
    public TimeSpan Duration { get; set; }
    public string Manufacturer { get; set; }
    public string Model { get; set; }
}

 

And here is how it would be used:

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "DTO For Custom Factory Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //The order of these fields determines the order of returned values in IList<object>
        var fields = "|Manufacturer |Model Duration";

        //Define your custom factory 
        Func<IList<object>, LightweightDtoForCustomFactory> factory = (values) =>
        {
            var dto = new LightweightDtoForCustomFactory();
            dto.Manufacturer = ((AFValue)values[0]).ToString();
            dto.Model = ((AFValue)values[1]).ToString();
            dto.Duration = ((AFTimeSpan)values[2]).ToTimeSpan();
            return dto;
        };

        var records = search.FindObjectFields<LightweightDtoForCustomFactory>(fields, factory, pageSize: 10000);

        //The loop is a bit simplified because the transfer logic is contained in the function delegate above.
        foreach (var record in records)
        {
            //Note that Manufacturer and Model are now validated strings thanks to factory.

            //Summary overload is for TimeSpan
            summary.AddToSummary(record.Manufacturer, record.Model, record.Duration, 1);
        }
    }
}

 

You are invited to review the code for each of the 3 overloads to look for similarities and differences.  The one big similarity is that each is concerned with casting the object value to its underlying type and then converting that to a desired type.  With that in mind, all 3 overloads offer identical performance so your preference of one over the other is purely your personal preference.

 

Great For Detail Reporting

As mentioned many times earlier, FindObjectFields is not limited to performing aggregations.  It's also quite handy for detail reporting too.  If you are going to use it for detail reporting, the strongest suggestion I can give you is to be sure to include the event frame's ID in your DTO class.  That way you have the ability to quickly find and load a specific event frame if ever needed.

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

 

Next Up: A Real Summary Method

In Part 6 we will cover the first bona fide aggregation method, AFEventFrameSearch.Summary.  I forgive you if it takes a several days or a week for you to move onto Part 6.  If you are anything like me, once you saw code for FindObjectFields the gears in your head must have started spinning overtime as you became preoccupied thinking of every app you've ever written that could have benefitted from a faster lightweight method.  If that's the case, Part 6 can wait.  You should by all means roll up your sleeves and get busy testing out FindObjectFieldsPart 6 will be here when you get back.

Using RPC Metrics in your AF SDK has never been easier!

 

Occasionally you may have the need to view various metrics regarding your AF SDK code.  Recently I was given a bit of code to review performance metrics, which I have modified and will present below via a text file to download.  Note the code references PIServer.GetClientRpcMetrics, which is new to AF SDK 2.9.  If you are interested in the code but are working with an earlier version of AF, you will need to strip out any references to PIServer to get it to compile.

 

Modify App.Config

You will need to modify the <configuration> section of your App.Config file.  If the App.Config file does not exist, you will need to create it.  You will add the following lines:

 

<system.net>
  <settings>
    <performanceCounters enabled = "true" />
  </ settings >
</ system.net >

 

This again would be in the <configuration> section, following the <startup> node.  If you have to create the App.Config file from scratch, the whole thing would look like:

 

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
  </startup>
  <system.net>
    <settings>
      <performanceCounters enabled="true" />
    </settings>
  </system.net>
</configuration>

 

For the metrics code further below, you may put it in a library project.  However, the library DLL as well as any application referencing the library should all have their App.Config modified as shown above.

 

MetricsTicker and MetricsSnapshot

I have 2 classes defined within one file named "Metrics.cs".  The MetricsTicker will track the starting and ending snapshot taken by MetricsSnapshot, and then neatly display the differences between the start and end.  MetricsSnapshot will take a snapshot of these metrics:

  • AF Server RPC Metrics (if any)
  • AF Client RPC Metrics (if any)
  • PI Client RPC Metrics (if any)
  • Network bytes sent  (a performance counter)
  • Network bytes received  (a performance counter)
  • Garbage collected memory
  • Allocated working set memory (a performance counter)
  • Allocated peak working set memory (a performance counter)
  • Timestamp (DateTime.UtcNow)

 

Simple Usage Example

Let's say I have some method named YourCustomMethod where I make lots of AF calls that I wish to review the metrics related to that method.  I could reference the classes by passing the Asset Server (PISystem) of interest:

 

var metrics = new MetricsTicker(assetServer);
metrics.Start();
YourCustomMethod();
metrics.Stop();

 

Well, that seems easy enough so far!  If you want a high precision timer, you may optionally wrap a Stopwatch around your custom method call.  However, the last thing a starting snapshot does is grab DateTime.UtcNow, and the first thing an ending snapshot does is also grab DateTime.UtcNow so there is already a built-in way to measure the timespan between snapshots.  Plus if you want to focus just on the AF RPC calls, you can review the subtotals of the duration.

 

Sample Output

Okay, starting and stopping our metrics was easy.  What about reporting?  How difficult is that?  It too is easy.  To output the the difference in metrics, there is one simple command:

 

Console.WriteLine(metrics.DisplayDelta());

 

Which would produce something like:

 

AF Server RPC Metrics        Count   Duration(ms)  PerCall(ms)

-------------------------  --------  ------------  ------------

GetSDCollection                   1           0.5         0.484

GetElement                       11          81.4         7.400

GetTableList                      1           3.1         3.092

GetTable                          1           1.6         1.649

GetElementTemplate                2           9.3         4.634

GetCategory                       2          10.2         5.120

GetUOMDatabase                    1           2.0         2.020

SearchTotalCount                  1         246.4       246.412

SearchRefresh                     1           0.0         0.021

SearchObjectIds                   3         221.9        73.953

GetEventFrames                   24        5386.9       224.456

GetCategoryList                   3           9.3         3.102

GetAnalyses                      24         545.2        22.717

GetAnalysisTemplates              1           9.5         9.532

GetElementTemplates               1           3.7         3.720

GetAnalysisTemplateList           1           1.9         1.894

GetModels                        47       17257.7       367.185

-------------------------  --------  ------------  ------------

                 Subtotal       125       23790.8     23790.770

 

AF Client RPC Metrics        Count   Duration(ms)  PerCall(ms)

-------------------------  --------  ------------  ------------

GetTableList                      1          58.7        58.652

GetTable                          1          30.8        30.814

GetElementTemplate                2          51.2        25.599

GetCategory                       2          39.6        19.786

GetUOMDatabase                    1          68.5        68.490

SearchTotalCount                  1         269.2       269.233

SearchObjectIds                   3         328.5       109.488

GetEventFrames                   24       13932.8       580.535

GetCategoryList                   3          23.9         7.983

GetAnalyses                      24        1706.0        71.083

GetAnalysisTemplates              1          72.0        72.034

GetElementTemplates               1          15.0        15.026

GetAnalysisTemplateList           1          15.8        15.765

GetModels                        47       31414.9       668.402

SearchRefresh                     1           4.7         4.748

-------------------------  --------  ------------  ------------

                 Subtotal       113       48031.7     48031.692

 

Total GC Memory: 385.37 MB

Working Memory : 524.71 MB

Peak Wrk Memory: 539.85 MB

Network Sent   : 8.84 MB

Network Receivd: 181.44 MB

Elapsed Time   : 02:44.6

 

How easy is that to generate a report of your metrics?!!

 

There are different combinations to DisplayDelta() method since it's full signature is:

 

public string DisplayDelta(bool round = true, bool showServerRpcMetrics = true, bool showClientRpcMetrics = true)

 

If you notice the bottom of the above output neatly shows bytes in MB rounded to 2 decimal places, and the elapsed time span is to 1/10th of a second.  This is due to the round parameter defaulting to true.  You can get the full bytes and time span if you try DisplayDelta(round: false).  Here's an example of that:

 

Total GC Memory: 464,306,792 bytes

Working Memory : 600,330,240 bytes

Peak Wrk Memory: 601,427,968 bytes

Network Sent   : 9,271,104 bytes

Network Receivd: 190,250,782 bytes

Elapsed Time   : 00:02:47.1073888

 

Memory is a Guesstimation

The Network Bytes Sent and Received are fairly accurate, but the values for memory usage should be considered a ballpark value rather than a highly accurate value.  There's so much that goes on inside of .NET with garbage collection, the memory heap, locked pages, etc., that makes it tough to be precise.  Suffice to say that if you to have an app that routinely uses 400 MB of Total GC Memory, and then you make changes to your app to see the memory drop to 100 MB routinely, then you should have peace-of-mind that you've reduced your memory needs by 75%.  Note that I peppered that last sentence with "routinely".  That's because, thanks to the mysteries of garbage collection, you may run the same app 10 times in a row and 9 of those times the memory may hover around 400 MB and 1 of those times it may unexpectedly drop to 200 MB.  This should be considered a one-off due to GC doing something independent of your app but obviously affecting your app.

 

While the memory usage is not highly accurate, I personally find it to be an acceptable gauge for comparisons.

 

Download Files

Here's the "Metrics29.cs" file to add to your projects.  You may need to change the namespace accordingly.  Note that "Metrics29.cs" requires AF SDK 2.9 or better.  For AF versions 2.6 through AF 2.8, you may use the "Metrics28.cs" version.

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

The Classical Full Load Approach

We are going to be using the AFEventFrameSearch.FindEventFrames method, which was about the only thing available to us in AF SDK 2.8.  In Part 3 we talked about the filter query we will be using.  We essentially want a 2 level summary to be performed: the first level is by Manufacturer and the second level is by Model.  Take some time to think of how you would have done this.  Many of you should already have firm ideas on how you would do it, and perhaps many of you have already done something like this before.

 

My approach will be to call FindEventFrames using fullLoad: true because I do need to reference some attributes.  As explained at the bottom of Part 2, this is rather heavyweight since it will be bring back a lot of stuff that I'm not interested in for the specific task at hand.  Despite this heaviness there is one crucial thing the full load doesn't bring back: attribute data!  That means I have to compose a 2nd set of calls to fetch the data, which means additional trips to the server.  An experienced developer would know that it's inefficient to call GetValue() one-at-a-time, so we will implement some sort of custom chunking to process in bulk in order to minimize the number of trips.

 

For those who attended the UC 2017 lab, I am going to do something a bit different.  In the lab, I would initialize a StatsTracker instance in the method below, and the method would return StatsTracker.  I decided to initialize StatsTracker shortly after I set my database and template objects, but before I record the metrics.  The initialization makes an RPC call to fetch the AFTable, which is the same for all 5 apps, so I don't really want to measure it for all 5 apps since its the same thing.  The method below differs from the lab in that it takes the StatsTracker as an input argument and the method now returns void.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    const int chunkSize = 5000;

    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "FindEventFrames Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(10);

        var frames = search.FindEventFrames(fullLoad: true, pageSize: 10000);

        var chunk = new List<AFEventFrame>();

        foreach (var frame in frames)
        {
            chunk.Add(frame);

            //Process in bulk calls for a given chunk
            if (chunk.Count >= chunkSize)
            {
                ProcessChunk(chunk, summary);
                chunk = new List<AFEventFrame>();
            }
        }

        //Process last chunk (if any)
        if (chunk.Count > 0)
            ProcessChunk(chunk, summary);
    }
}

private void ProcessChunk(IList<AFEventFrame> chunk, StatsTracker summary)
{
    var attributes = new AFAttributeList();

    //First pass over each event frame in this chunk to gather the attributes
    foreach (var frame in chunk)
    {
        attributes.Add(frame.Attributes["|Manufacturer"]);
        attributes.Add(frame.Attributes["|Model"]);
    }

    //Secondly issue a bulk GetValue call on those attributes, but I need a dictionary for faster lookups
    var values = attributes.GetValue().ToDictionary(pv => pv.Attribute);

    //Finally pass over each event frame one last time to update summary using fetched values.
    foreach (var frame in chunk)
    {
        //Read data from current event frame
        var mfr = values[frame.Attributes["|Manufacturer"]].Value.ToString();
        var model = values[frame.Attributes["|Model"]].Value.ToString();

        summary.AddToSummary(mfr, model, frame.Duration, 1);
    }
}

 

You will note in line 09 that I do opt-in to server-side caching, which I do for the other 4 apps.  The difference is that here, where I know there is a performance penalty due to the heaviness of the objects, I use a timeout of 10 minutes.  Since the other 4 apps in the series will be much, much faster, I will use a timeout of 5 minutes for them.

 

About pageSize

The default value for pageSize is 1000.  I choose 10000 here.  Why?  Is it better?  Do I have inside information that its better?  NO.  I did this because life is too short.  While developing the lab, and testing frequently with each new beta build, I ran this over 1000 times.  Early on, when I looked at a 3-day period, the code above took 10 minutes to run.  This would be ridiculous to have a lab exercise take 10 minutes just to execute.  So I trimmed it my filter to 1-day and the code then took over 4-5 minutes to run using the default pageSize.  A pageSize of 5000 took 3-and-a-half minutes to run, but used more memory.  I settled on a pageSize of 10000, which used even more memory, but took about 2-and-a-half minutes to run, which is about as long as I would want a lab exercise to take.

 

There was a side benefit that since this took more memory using pageSize: 10000 that it really helped to show-off the memory savings of the new methods.  But that was just a side benefit.  It really came down to I didn't want to wait 10 minutes a couple of times a day to wait for this to finish.

 

About Producing Metrics

I have shown you results of metrics, but I haven't shown you how I produce them.  That is in this separate blog.  It is not a part of this 9-part series.  It will be a blog to stand on its own since metrics tracking is a topic completely independent of event frame searching or aggregation of data values.

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

 

Next up: Finally Something New

This was just one possible way of producing aggregates, or specifically a 2-level aggregation.  There's lots of different ways this could have been done, even with FindEventFrames.  How would you have done it?  Would you have done something mostly similar but differing in some details?  Or do you have a totally different approach?

 

Anyway, we have established our baseline for performance metrics.  We are now ready to venture into brand new territory with Part 5 where we see how to use the new FindObjectFields method.

Filter Blog

By date: By tag: