Skip navigation
All Places > PI Developers Club > Blog
1 2 3 Previous Next

PI Developers Club

665 posts

Hi everyone,

 

Recently, there was a comment on one of my other blogs about how to create a template or modify PI Vision in such a way that every time a user creates a new display it is created with a default background color and with a watermark image.

 

Today, I will present a way today to do both of them. Take note that these customizations might not carry over on version upgrades.

 

Watermark

First, to add a watermark to your display, search for the file

 

C:\Program Files\PIPC\PIVision\Views\Home\_DisplayArea.cshtml

 

in your PI Vision installation directory. Open it up and add the following markup code

 

<style type="text/css" scoped>
#watermark{
color:blue;
font-size:120px;
transform:rotate(300deg);
-webkit-transform:rotate(300deg);
position:absolute;
z-index:0;
min-height:0%;
min-width:0%;
}
div{
z-index:0;
}
</style>
<p id="watermark">Eugene Confidential</p>

 

underneath this line

<div class="c-drop-container" id="zoom-here" style="transform: matrix(1, 0, 0, 1, 0, 0); transform-origin: 0px 0px;">

 

This will produce the following watermark on the display.

 

 

Background Color

And now, to change the default display background color, search for the file

 

C:\Program Files\PIPC\PIVision\Content\css\theme.base-colors.css

 

Search for ".t-display-container" and edit the CSS background-color to that which you want. For example,

 

 

.t-display-container {
background-color: #7624b5; }

 

The resulting default display will become

 

 

Conclusion

You can see that with a bit of HTML and CSS modification, you can customize your PI Vision installation to perform the way you want. Many other customizations are possible. Please feel free to post in the comments below on how you would like to customize your default display and the PI Developers community can chip in with ideas on how to implement them.

 

I hope you learnt something useful!

Introduction

Hi everyone!

 

I had a request from my friend at Petronas about creating a Filter Table in PI Vision which can simulate the filtering ability of Microsoft Excel. As you probably already know, the stock Table symbol doesn't support filtering the rows of the table. This is where PI Vision Custom Symbol Extensibility will fill the gap!

 

In this blog, I will create a Filter Table which contains PI Point values that we can filter by the name of the PI Point and even do searching by PI Point name. Let's take a look at the implementation.

 

Implementation

We start with the definition of the symbol. Note that we are using the 'Table' DataShape to allow for multiple data sources.

 

sym-filtertable.js

var definition = {
   typeName: 'filtertable',
   datasourceBehavior: CS.Extensibility.Enums.DatasourceBehaviors.Multiple,
   visObjectType: symbolVis,
   getDefaultConfig: function () {
      return {
         DataShape: 'Table',
         Height: 500,
         Width: 600
      };
   },
};

 

Next, the dataUpdate function will be used for the creation of the table and also its update. The DataTable jQuery plug-in will be used to help with the table construction.

 

sym-filtertable.js

q = 1;
var table;
function dataUpdate(data) {
   if (q) {
      $('#example').dataTable({
         data: data.Rows,
         columns: [
            { "data": "Label" },
            { "data": "Time" },
            { "data": "Value" }
         ],
         rowId: 'Label'
      });
      q = null;
      table = $('#example').DataTable();
      var select = $('<select />')
         .appendTo(
            table.column(0).footer()
         )
         .on('change', function () {
            table
               .column(0)
               .search($(this).val())
               .draw();
         });
      table
         .column(0)
         .cache('search')
         .sort()
         .unique()
         .each(function (d) {
            select.append($('<option value="' + d + '">' + d + '</option>'));
         });
         select.append($('<option value="">' + 'ALL' + '</option>'));
   }

   table = $('#example').DataTable();
   i = 0;
   table.rows().every(function () {
      var d = this.data();
      d.Value = data.Rows[i].Value
      d.Time = data.Rows[i].Time
      i++
      this.invalidate();
   });
   table.draw();
}

 

 

Presentation

Finally, in your html file, you can define the column header names that you want.

 

sym-filtertable-template.html

<table id="example" class="display" style="width:90%">
   <thead>
      <tr>
         <th>Label</th>
         <th>Time</th>
         <th>Value</th>
      </tr>
   </thead>
   <tfoot>
      <tr>
         <th>Label</th>
         <th>Time</th>
         <th>Value</th>
      </tr>
   </tfoot>
</table>

 

Here is an example of how it will look like.

 

Introduction

 

On the first two parts of this blog post series (part 1 and part 2), you have learned how to create an ASP.NET Core web application using PI AF SDK and .NET Framework. On the second part, you've learned how to secure your web app using ASP.NET Core Identity.

 

On this blog post, we are going to learn one of the best ASP.NET Core features which is Dependency Injection and apply this technique when making calls against our PI System.

 

 

What is Dependency Injection (DI)?

 

According to Wikipedia:

 

"In software engineering, dependency injection is a technique whereby one object (or static method) supplies the dependencies of another object. A dependency is an object that can be used (a service)."

 

Let's try to understand the concept of Dependency Injection through our example from the first part of this blog post series

 

When class A uses some functionality of class B, then it is said that class A has a dependency of class B. In the context of C#, class A needs to create an instance of class B in order to use its methods and properties. 

 

So, transferring the task of creating the object to another entity and directly using the dependency is called dependency injection.

 

In our sample, our HomeController class uses some PI AF SDK methods to render the snapshot values of the found PI Points to the user. We will make a small change in this project by creating a new service of the type IPISystemWrapper, responsible for all PI System communication which will be injected to our HomeController. As a result, the HomeController won't have any PI AF SDK code anymore. It will only inject the IPISystemWrapper service/interface and call its methods to deliver to the Views.

 

 

Why should I use DI when developing apps on top of the PI System?

 

Here are the main advantages of using DI in this context:

 

  • Easier to extend the application. Imagine that many controllers are injecting the IPISystemWrapper service. If you add a new method to this interface, then all the controllers will be able to take advantage of this new feature. As a result, this strategy helps reducing duplicated code.
  • Easier to maintain your code base. The service IPISystemWrapper could have different implementations. The AfSdkWrapper class could be the implementation for PI AF SDK of this service and the PIWebApiWrapper could be the implementation for using PI Web API. This is strategy improves your code base not only from the organization perspective but from the maintainability standpoint.
  • Enable loose coupling. Imagine that you must upgrade your web application from ASP.NET Core 2.2 to ASP.NET Core 3 once it is released. ASP.NET Core 3 cannot be created on top of .NET Framework and therefore, it won't be compatible with the current version of PI AF SDK. I will show you how easy it is to switch from PI AF SDK to another PI Developer Technology.
  • Helps Unit Testing: since all the PI AF SDK code will be written on a unique class. All your unit tests concerning PI AF SDK can be written for this particular class. If you have many controllers using PI AF SDK, it gets more complicated to achieve this goal.

 

Implementing DI on our example

 

This example is simpler than the one from part 1. This is why I am pasting the code from the main files of the VS project here. Nevertheless, please refer to the previous blog post in case something is not clear since I will focus on the DI concepts on this blog post.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

Create a new ASP.NET Core project on top of .NET Framework and add PI AF SDK as a reference.

 

 

Create some Views with the content below in order to display the results to the user:

 

\Views\Home\Query.cshtml

 

<div class="col-sm-8 col-sm-offset-2">
<h1>View Snapshot Values</h1>
<br /><br />
<h4>Search for PI Points and check its snapshots.</h4>
<br /><br />
<div class="container">
<div class="row">
<form asp-controller="Home" asp-action="Result" method="get" class="form-inline">
<div class="form-group">
<label class="sr-only" for="name">Search PI Point: </label>
<input id="pointNameFilter" name="pointNameFilter" placeholder="Point Name" class="form-control" />
</div>
<button type="submit" class="btn btn-default">Submit</button>
</form>
</div>
</div>
</div>

 

\Views\Home\Result.cshtml

@model IEnumerable<PIPointSnapshotModelView>

<div class="col-sm-8 col-sm-offset-2">
<h1>View Snapshot Values</h1>
<br /><br />
<table class="table">
<tr>
<th>PI Point</th>
<th>Value</th>
<th>Timestamp</th>
</tr>
@foreach (var pointSnapshotValue in Model)
{
<tr>
<td>@pointSnapshotValue.PointName</td>
<td>@pointSnapshotValue.Value.ToString()</td>
<td>@pointSnapshotValue.TimeStamp.ToString("dd-MMM-yyyy hh:mm:ss tt")</td>
</tr>
}
</table>
<br />
<center>
<a asp-controller="Home" asp-action="Index" class="btn btn-primary">Make another search</a>
</center>
</div>

 

\Views\Shared\_Layout.cshtml

<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width" />
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<title>My ASP.NET Core MVC with PI AF SDK or PI Web API</title>
</head>
<body>
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<a asp-page="/Index" class="navbar-brand">Applying ASP.NET Core DI with the PI System </a>
</div>
<div class="navbar-collapse collapse">
</div>
</div>
</nav>
<div class="container body-content"style="padding-top: 50px;">
@RenderBody()
</div>
</body>
</html>

 

\Views\_ViewImports.cshtml

@using ASPNETCoreWithPIAFSDK.Models
@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers

 

\Views\_ViewStart.cshtml

@{
Layout = "_Layout";
}

 

Create a model for the Result.cshtml view:

 

\Models\PIPointSnapshotModelView.cs

using OSIsoft.AF.Asset;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace ASPNETCoreWithPIAFSDK.Models
{
public class PIPointSnapshotModelView
{
public string PointName { get; set; }

public object Value { get; set; }
public DateTime TimeStamp { get; set; }


public PIPointSnapshotModelView(string pointName, AFValue afValue)
{
PointName = pointName;
Value = afValue.Value;
TimeStamp = afValue.Timestamp;
}

public PIPointSnapshotModelView(string pointName)
{
PointName = pointName;
}
}
}

 

 

Let's create a folder named Framework on the root with the IPISystemWrapper interface. This interface has only one method called GetSnapshotValuesFromPoints with one string input. This function finds all PI Points filtered using the pointNameFilter input and then finds all of its snapshots.

 

\Framework\IPISystemWrapper.cs

using System.Collections.Generic;
using System.Threading.Tasks;
using ASPNETCoreWithPIAFSDK.Models;

namespace ASPNETCoreWithPIAFSDK.Framework
{
public interface IPISystemWrapper
{
Task<IEnumerable<PIPointSnapshotModelView>> GetSnapshotValuesFromFoundPoints(string pointNameFilter);
}
}

 

The AfSdkWrapper class implements the IPISystemWrapper interface. This is the only class that has PI AF SDK under this VS project. This class implements GetSnapshotValuesFromPoints method using PI AF SDK code in order to show to the user the requested snapshots.

 

\Framework\AfSdkWrapper.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using ASPNETCoreWithPIAFSDK.Models;
using OSIsoft.AF;
using OSIsoft.AF.Asset;
using OSIsoft.AF.PI;

namespace ASPNETCoreWithPIAFSDK.Framework
{
public class AfSdkWrapper : IPISystemWrapper
{
private PIServer piServer;

public AfSdkWrapper()
{
//On production, please store the PI Data Archive name on appsettings.json
this.piServer = new PIServers()["MARC-PI2016"];
piServer.Connect();
}
public async Task<IEnumerable<PIPointSnapshotModelView>> GetSnapshotValuesFromFoundPoints(string pointNameFilter)
{
PIPointQuery query = new PIPointQuery(PICommonPointAttributes.Tag, OSIsoft.AF.Search.AFSearchOperator.Equal, pointNameFilter);
IEnumerable<PIPoint> foundPoints = PIPoint.FindPIPoints(piServer, new PIPointQuery[] { query });
PIPointList pointList = new PIPointList(foundPoints.Take(1000));


List<PIPointSnapshotModelView> pointSnapshotValueList = new List<PIPointSnapshotModelView>();
AFListResults<PIPoint, AFValue> values = await pointList.EndOfStreamAsync();
foreach (PIPoint point in pointList)
{
AFValue value = values.Where(v => v.PIPoint == point).Single();
pointSnapshotValueList.Add(new PIPointSnapshotModelView(point.Name.ToString(), value));
}
return pointSnapshotValueList;

}
}
}

 

The HomeController injects the IPISystemWrapper interface which has the unique method needed for the Result action to display the results to the View.

 

\Controllers\HomeController.cs

using ASPNETCoreWithPIAFSDK.Framework;
using ASPNETCoreWithPIAFSDK.Models;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using OSIsoft.AF;
using OSIsoft.AF.Asset;
using OSIsoft.AF.PI;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace ASPNETCoreWithPIAFSDK.Controllers
{
public class HomeController : Controller
{
private IPISystemWrapper piSystemWrapper;

//Injecting the IPISystemWrapper service at runtime
// This interface could be AfsdkWrapper or PIWebApiWrapper, according to what is defined on the Startup.cs
public HomeController(IPISystemWrapper piSystemWrapper)
{
this.piSystemWrapper = piSystemWrapper;
}
public IActionResult Index()
{
return this.RedirectToAction("Query");
}

public IActionResult Query()
{
return View();
}

public async Task<IActionResult> Result(string pointNameFilter)
{
IEnumerable<PIPointSnapshotModelView> values = await piSystemWrapper.GetSnapshotValuesFromFoundPoints(pointNameFilter);
return View(values);
}
}
}

 

Finally, the Startup.cs is the file that adds a service of the type IPISystemWrapper with an implementation of this interface which in our case is AfSdkWrapper. In other words, every time the IPISystemWrapper needs to be injected, the DI will inject an instance of the AfSdkWrapper.

 

\Startup.cs

using ASPNETCoreWithPIAFSDK.Framework;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace ASPNETCoreWithPIAFSDK
{
public class Startup
{
public IConfiguration Configuration { get; set; }
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public void ConfigureServices(IServiceCollection services)
{
services.AddSingleton<IPISystemWrapper, AfSdkWrapper>();
services.AddMvc();
}

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
app.UseStatusCodePages();
}

app.UseStaticFiles();
app.UseMvcWithDefaultRoute();


}
}
}

 

Running this application, we can see the snapshot values after making a search:

 

 

Replacing PI AF SDK with PI Web API

Finally, I want to demonstrate how easy it is to switch from PI AF SDK to PI Web API using the DI infrastructure. Let's create the PIWebApiWrapper that implements IPISystemWrapper interface. I won't comment about the PI Web API code because it is out of the scope of this blog post.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

\Framework\PIWebApiWrapper.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
using ASPNETCoreWithPIAFSDK.Models;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace ASPNETCoreWithPIAFSDK.Framework
{
public class PIWebApiWrapper : IPISystemWrapper
{
private HttpClient client;
private HttpClientHandler handler;
private string dataServerWebId;

public PIWebApiWrapper()
{
//On production, please store the PI Web API base URL on appsettings.json
this.BaseUrl = "https://marc-web-sql.marc.net/piwebapi";
this.UseKerberos = true;
var handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
client = new HttpClient(handler);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("X-Requested-With", "PIWebApiWrapper");


}

public string BaseUrl { get; private set; }
public string Username { get; private set; }
public string Password { get; private set; }
public bool UseKerberos { get; private set; }

public async Task<IEnumerable<PIPointSnapshotModelView>> GetSnapshotValuesFromFoundPoints(string pointNameFilter)
{
List<PIPointSnapshotModelView> pointSnapshotList = new List<PIPointSnapshotModelView>();

if (string.IsNullOrEmpty(this.dataServerWebId) == true)
{
//On production, please store the PI Data Archive name on appsettings.json
string url = this.BaseUrl + @"/dataservers?path=\\MARC-PI2016";
HttpRequestMessage httpRequest = new HttpRequestMessage(HttpMethod.Get, url);
HttpResponseMessage httpResponse = await client.SendAsync(httpRequest);
string response = await httpResponse.Content.ReadAsStringAsync();
JObject jObject = JsonConvert.DeserializeObject<JObject>(response);
this.dataServerWebId = jObject["WebId"].ToString();
}

string url2 = this.BaseUrl + "/dataservers/" + this.dataServerWebId + "/points?nameFilter=" + pointNameFilter;
HttpRequestMessage httpRequest2 = new HttpRequestMessage(HttpMethod.Get, url2);
HttpResponseMessage httpResponse2 = await client.SendAsync(httpRequest2);
string response2 = await httpResponse2.Content.ReadAsStringAsync();


JObject jObject2 = JsonConvert.DeserializeObject<JObject>(response2);

string url3 = this.BaseUrl + "/streamsets/value?";
List<Tuple<string, string>> list = new List<Tuple<string, string>>();
if (jObject2["Items"] != null)
{
foreach (var item in jObject2["Items"])
{
string webId = item["WebId"].ToString();
string name = item["Name"].ToString();
Tuple<string, string> tuple = new Tuple<string, string>(name, webId);
list.Add(tuple);
url3 = url3 + $"webId={webId}&";
}
url3 = url3.Substring(0, url3.Length - 1);
}
else
{
return pointSnapshotList;
}


HttpRequestMessage httpRequest3 = new HttpRequestMessage(HttpMethod.Get, url3);
HttpResponseMessage httpResponse3 = await client.SendAsync(httpRequest3);
string response3 = await httpResponse3.Content.ReadAsStringAsync();

JObject jObject3 = JsonConvert.DeserializeObject<JObject>(response3);
if (jObject3["Items"] != null)
{
foreach (var item in jObject3["Items"])
{
string webId = item["WebId"].ToString();
Tuple<string, string> tuple = list.Where(p => p.Item2 == webId).Single();
PIPointSnapshotModelView piPointSnapshotModelView = new PIPointSnapshotModelView(tuple.Item1);
if (tuple != null)
{
if ((item["Value"]["Value"].Type != JTokenType.Object))
{
piPointSnapshotModelView.Value = item["Value"]["Value"].ToString();
}
else
{
piPointSnapshotModelView.Value = item["Value"]["Value"]["Name"].ToString();
}
piPointSnapshotModelView.TimeStamp = Convert.ToDateTime(item["Value"]["Timestamp"].ToString());
}
pointSnapshotList.Add(piPointSnapshotModelView);
}
}
return pointSnapshotList;
}
}
}

 

If we update the ConfigureServices method, the application will use PI Web API instead of PI AF SDK and present the same results to the final user.

 

\Startup.cs

public void ConfigureServices(IServiceCollection services)
{
services.AddSingleton<IPISystemWrapper, PIWebApiWrapper>();
services.AddMvc();
}

 

Conclusions

Using Dependency Injenction on your PI applications on top of ASP.NET Core is very interesting approach to help maintain, organize and test your your app for many years. It is not the easiest concept to understand but you will quickly see its value once you start implement it.

Introduction

Greetings everyone! 

 

In today's blog post, we will be touching on the topic of dynamic template views in PI Vision Custom Symbols. The idea is to support multiple HTML template files for your custom symbol which could correspond to the sites that are available in your industrial plant so that users are able to choose the site that they want.

 

I answered this briefly in this thread. 

https://pisquare.osisoft.com/thread/39998

 

Now, I will explain this in more detail here. I will extend the Simple Value symbol from the GitHub repository available here to allow it to use multiple HTML files.

 

Disclaimer:

Please take note that the licensing for the GitHub repository referenced above extends to any of the code that is present in this blog.

 

Presentation Layer

For the original HTML template file, replace the original contents with the following. This will be the master template which redirects to the other site templates. Note that the ng-include directive is used here. It helps us to fetch, compile and include an external HTML fragment which in this case is the other site template files.

 

sym-simplevalue-template.html

<div ng-include= config.View.path></div>

 

Start making more HTML files with the content and file names below. Notice that the additional template files will have the scope carried over so you can still display the data updates. This makes it possible to do things such as {{value}}.

 

sym-simplevalue-template1.html

<div ng-style="{background: config.BackgroundColor, color: MultistateColor || config.TextColor}">
<div ng-show="config.ShowLabel">{{label}}</div>
<div>{{value}}</div>
    <div ng-show="config.ShowTime">{{time}}</div>
    This template file is for Site 1.
</div>

 

sym-simplevalue-template2.html

<div ng-style="{background: config.BackgroundColor, color: MultistateColor || config.TextColor}">
This template file is for Site 2.
<div ng-show="config.ShowLabel">{{label}}</div>
<div>{{value}}</div>
    <div ng-show="config.ShowTime">{{time}}</div>
</div>

 

Implementation Layer

In our implementation, we have to create a new property that holds the path to the default template file to use upon symbol creation. We can create this new property in the getDefaultConfig function. We will call it 'View'.

 

sym-simplevalue.js

getDefaultConfig: function() {
     return {
       DataShape: 'Value',
       Height: 150,
       Width: 150,
       BackgroundColor: 'rgb(255,0,0)',
       TextColor: 'rgb(0,255,0)',
       ShowLabel: true,
       ShowTime: false,
       View:{path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template1.html"}
    };
},

 

Next, in the init function of the symbol, we can set the possible paths that the user will be able to choose. Take note that these paths should be within the PI Vision installation directory so that you adhere to the same-origin policy. My advice is to place any custom files in the ext directory to facilitate upgrades. You won't have to search high and low for your custom files.

 

sym-simplevalue.js

scope.config.paths = [
    {name:"Site 1", path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template1.html"},
    {name:"Site 2", path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template2.html"}
];

 

Configuration Layer

Now we need a way for the user to be able to select the template that he wants during run-time. And where better but to put it inside the configuration pane? We will create an additional drop down list in our pane for this purpose.

 

sym-simplevalue-config.html

<div class="c-config-content">Change View:
<select ng-model="config.View" ng-options="x.name for x in config.paths track by x.name">
    </select>
</div>

 

This list will populate with the possible paths that we set in the implementation layer during symbol initialization.

 

Demo

Let's check out a demo of how this will look like in action!

 

 

Conclusion

We have seen that with the Custom Symbols Extensibility for PI Vision, it becomes a very flexible product where you can do almost anything with some ideas and coding skills. This is important because it is impossible for OSIsoft to imagine every possible visualization style that users want. We can make symbols out of the most common use cases but we can't cater to every single one. The Extensibility framework helps to plug that gap so that many use cases can be addressed with PI Vision. This truly makes PI Vision a first class visualization platform!

I have been talking about AFSearch 2.10.5 for well over a year now. I spoke about it at PI World SF in April 2018 and again at Barcelona in Sept 2018. However, both of those talks used a beta build. The great news is that for the week of PI World SF in April 2019, this is no longer beta and is now production! Sadly, when working with beta code one should accept that the final production version may be a bit different, and that's certainly the case here. As I dusted off my code from Sept 2018, I could see it had several compiler errors since there was some fairly significant changes on the journey to production. Such is the life of a beta tester.

 

In case you haven't been following AFSearch or PI Server 2018 SP2 at all in the past year, the big changes for AFSearch with AF SDK 2.10.5 will be concisely summarized as:

 

  • Overlapped event frames searches got a performance boost.
  • Search queries now support OR conditions.
  • A new generic AFSearch<T> base class was added to implement many of the common properties and methods in derived search classes. As a result of this change, several methods have been marked as obsolete.

 

Don't let the short list fool you. While the first item requires no changes upon your part, the other items do require code changes. Supporting OR conditions is easy to say, but it did cause some breaking changes. In particular the old AFSearchToken structure is insufficient to represent new filters with OR. To that end, the AFSearchToken structure is being marked as Obsolete and replaced with a new AFSearchTokenBase class. Likewise, a generic AFSearch<T> base class introduces an IAFSearch<T>.FindObjects method, which means the many of the older Find methods (e.g. FindElements or FindEventFrames) are also marked as Obsolete. This is all mentioned in the help file under the heading What's new in PI AF 2018 SP2, but this blog expands upon short summary to give examples.

 

Event Frame Search Performance Boost

 

Let's start with the simplest change. If you are using captured event frames, then some searches should be noticeably faster, particular searches for Overlapped event frames.  For those of you rusty on the subject, Overlapped event frames are also considered "Active Between", meaning that any portion of the event frame falls within a requested time range.  I refer you to the image found in this link if you want more information.  Side note: when I was a customer writing AF applications, Overlapped was our most frequently used mode, and I know this is true for many of you.

 

An anecdotal example of this performance boost can be found in the PI World talks in 2018. In April, it took me about 1 minute to find 320K event frames from an AF Server of 6 million event frames. In September, I was using the same hardware but updated the AF SDK build every couple of weeks. About 3 weeks before Barcelona, the 1 minute search suddenly dropped to around 15 seconds. I asked the dev team and they confirmed they had implemented new logic for event frame searches.  I then gratefully said "thank you" and wiped the tears from my eyes.

 

This requires no change on your part other than the acceptance that your code may run twice as fast. What a nice problem to have!

 

OR - small word, BIG Changes

 

To implement support for OR, it quickly became apparent that the old AFSearchToken was not going to get the job done. There was a simplicity with the original AFSearchToken, but this breaks with the complexity of OR searches. There was just no way to represent it with the old structure. AFSearchToken is marked as obsolete, and any Tokens properties that depended upon AFSearchToken are also marked as obsolete. If your search does not use OR, then it will will get parsed by the Tokens property just fine (or with a little compiler warnings). But if you do use OR in your search, then accessing the Tokens property will produce an exception. Instead, you should begin using the TokenCollection property.

 

The TokenCollection property returns an AFSearchTokenCollection object, which implements IList<AFSearchTokenBase>. If you dig deeper into the help, you may discover that AFSearchTokenBase is an abstract class with 4 derived classes:

 

  • AFSearchExpressionToken, representing a logical grouping of search tokens. These tokens would be grouped together based on an OR or the default AND. I jokingly add that this is also akin to adding virtual parenthesis around your expression. All joking aside, if a query string contains parentheses, then the tokenized equivalent will likely need to use AFSearchExpressionToken.
  • AFSearchFilterToken, representing one criteria of a search class. Most of the obsolete AFSearchToken searches would use the new AFSearchFilterToken.
  • AFSearchQueryToken, representing a nested search query. Some obsolete AFSearchToken searches did use nested queries but should now use this AFSearchQueryToken.
  • AFSearchValueToken, representing an attribute value query.

 

Let's walk through an example to see these new tokens in action. I have chosen an example that uses all 4 of the new tokens. Let's first state how we plan to filter using simple phrases and business terms.

 

  • I want to perform an AFAttributeSearch, which means I will need a nested element query as well.
  • I want to search attributes on elements that have been derived from the "Storage Tank" element template.
  • I want to refine the element search to limit tank names (element names) to those starting with the letter "T".
  • I want a value filter on the attribute named "|Fill Percent". That filter should be on any tank level <= 10% or >= 90%. That is I want to filter on tanks that are too low or too high.
  • I want to return attributes named "Fill Percent", which is the tank level per tank (element).

 

Let's see how that would look as a query string. I personally like query strings, and many of the devs on the Asset Team prefer them as well.

 

Let's cheat a little and start out with what the fully resolved query string would look like:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

Note that anything between the red { } braces is the nested element query. Within the confines of those red braces, the context for Name (as in Name:T*) refers to the nested element query, so Name in this context means element name. Outside the red braces, you will see another Name (as in Name:'Fill Percent'), whose context refers to the attribute search, and therefore is an attribute name. (This example was specifically chosen to throw a curveball at you, forcing you to reconcile which Name is which).

 

You will note that any names containing embedded blanks are wrapped in quotes. You are free to create a longer and uglier string using escaped double quotes, but my eyeballs are much happier with the single quotes.

 

Okay, now let's see how that would be written with the new AFSearchTokenBase tokens. There are a few different ways, none of which make your code shorter. First, let's define some variables and constants to hold many of the names and limit values.

 

const string templateName = "Storage Tank";
const string elemName = "T*";
const string attrName = "Fill Percent";  // Note the pipe or bar symbol "|" is omitted
const double lowLimit = 10;
const double highLimit = 90;

 

Let me point out that attrName is used when a name is required. However, many of the value filters requires an attribute path. Later below you will see me preface attrName with a "|" to transform it into a path. Wherever this is done, you will also see me wrap the final text in single quotes to account for embedded blanks in the name.

 

This will be done in several pieces of code. I try to work with the innermost filter first, so that would be the OR clause wrapped in parenthesis.

 

AFSearchExpressionToken orExprToken = new AFSearchExpressionToken(AFSearchLogicalOperator.Or, new List<AFSearchTokenBase> {
    new AFSearchValueToken($"|{attrName}", lowLimit.ToString(), AFSearchOperator.LessThanOrEqual, AFSearchValueType.Numeric),
    new AFSearchValueToken($"|{attrName}", highLimit.ToString(), AFSearchOperator.GreaterThanOrEqual, AFSearchValueType.Numeric) });

 

That is one really long assignment. If you like breaking it down into more lines, then you may use this instead:

 

// First let's compose an OR condition for our limit extremes.
IList<AFSearchTokenBase> orChildTokens = new List<AFSearchTokenBase>();
orChildTokens.Add(new AFSearchValueToken($"|{attrName}", lowLimit.ToString(), AFSearchOperator.LessThanOrEqual, AFSearchValueType.Numeric));
orChildTokens.Add(new AFSearchValueToken($"|{attrName}", highLimit.ToString(), AFSearchOperator.GreaterThanOrEqual, AFSearchValueType.Numeric));

// Next the OR must be grouped as a single expression, which is akin to enclosing it in parenthesis.
AFSearchExpressionToken orExprToken = new AFSearchExpressionToken(AFSearchLogicalOperator.Or, orChildTokens);

 

This demonstrates 2 different ways to populate the expression token. The first way created the tokens argument inside the AFSearchExpressionToken constructor. The second way created the expression tokens before the AFSearchExpressionToken was constructed. There are 2 other noteworthy items along this topic. First, and this is a big change, is that you cannot modify the input tokens once the instance has been constructed. Second, is that you may expand this thinking to include any search-related constructor expecting an argument of IList<AFSearchTokenBase> such as constructors for AFSearchQueryToken, AFElementSearch, AFAttributeSearch, etc.

 

Either of the above code snippets fill in the first major piece towards a fully tokenized call. The second major piece will be to complete the nested element query token. Note we use the orExprToken object we defined above.

 

// Gather tokens for the nested element query, which will include the OR expression group.
IList<AFSearchTokenBase> nestedQueryTokens = new List<AFSearchTokenBase>();
nestedQueryTokens.Add(new AFSearchFilterToken(AFSearchFilter.Template, templateName));
nestedQueryTokens.Add(new AFSearchFilterToken(AFSearchFilter.Name, elemName));
nestedQueryTokens.Add(orExprToken);

 

The final piece will be to complete the attribute search query by combining the nested element query with other attribute search filters.

 

// Finally gather tokens for the AFAttributeSearch, which includes the nested element query
// and the attribute name filter.
List<AFSearchTokenBase> tokens = new List<AFSearchTokenBase>();
tokens.Add(nestedQueryTokens);
tokens.Add(new AFSearchFilterToken(AFSearchFilter.Name, attrName));

 

Alternatively, you could have also performed the last assignment with:

 

List<AFSearchTokenBase> tokens = new List<AFSearchTokenBase>() { nestedQueryTokens, new AFSearchFilterToken(AFSearchFilter.Name, attrName) };

 

That wraps it up for the new tokens. Again, the complexity of using the new tokens was a necessary evil to accommodate the robust requirements of OR clauses.

 

If you straddle the border on whether to use tokens or strings, I would encourage you to learn more about query strings. Meanwhile, let's return that original query string. As I said, this has been fully resolved:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

But in many cases this won't be fully resolved as many of your parameters are most likely stored in variables. What if you wanted to write the string to have variable substitution? Using String Interpolation, it would look like:

 

string query = $"Element:{{Template:'{templateName}' Name:'{elemName}' ('|{attrName}':<={lowLimit} OR '|{attrName}':>={highLimit})}} Name:'{attrName}'";

 

You will note that creating query is where we insert the pipe symbol. If you are more familiar with String.Format, you could use:

 

string query = String.Format("Element:{{Template:'{0}' Name:'{1}' ('|{2}':<={3} OR '|{2}':>={4})}} Name:'{2}'", templateName, elemName, attrName, lowLimit, highLimit);

 

I personally favor the Interpolated String because I know which variable is being used in-place, rather than we sequential numbers that require more eye movement.

 

New Methods and Constructors

 

It should be no surprise that the introduction of new TokenCollection and associated AFSearchTokenBase classes require new constructors and methods. This touches on virtually every related search class. Nothing much to add other than to make you aware of this. There is no need to enumerate them here since it's already been covered in Live Library's What's New link.

 

The Obsolete Find Methods

 

Under the hood, there is a significant change to the AFSearch abstract class. Prior to AF SDK 2.10.5, the class signature was:

 

public abstract class AFSearch : IDisposable

 

Effective with 2.10.5, the signature is now:

 

public abstract class AFSearch<T> : AFSearch, IAFSearch<T> where T : AFObject

 

This means the AFSearch class can now implement many of the common properties and methods in each of the derived search classes. One such common method is the FindObjects method, which is an implementation of the IAFSearch<T>.FindObjects interface.

 

Given how each derived search class has a FindObjects method, there is no longer a need to have each derived search class with its own type-specific Find method. Towards that end, the type-specific methods are marked as obsolete. This means you should start using FindObjects in place of:

 

  • AFAnalysisSearch.FindAnalyses
  • AFAnalysisTemplateSearch.FindAnalysisTemplates
  • AFAttributeSearch.FindAttributes
  • AFCaseSearch.FindCases
  • AFElementSearch.FindElements
  • AFEventFrameSearch.FindEventFrames
  • AFNotificationContactTemplateSearch.FindNotificationContactTemplates
  • AFNotificationRuleSearch.FindNotificationRules
  • AFNotificationRuleTemplateSearch.FindNotificationRuleTemplates
  • AFTransferSearch.FindTransfers

 

Again, don't be thrown off by the generic sounding FindObjects. Calling AFElementSearch.FindObjects still returns IEnumerable<AFElement>, so only the method name has changed.  FindObjects still returns type-specific items.

 

Team Query String versus Team Tokens

 

AF SDK is flexible in that you may choose to use either query strings or tokens, just as you may choose to use C#, VB.NET, or managed C++. As mentioned earlier, the Asset developers strongly prefer query strings.

 

I admit the new AFSearchTokenBase class is much more complicated to work with than the older AFSearchToken structure. I talked to several customer developers and many said they like tokens that spread out over many lines rather than a single-line query string because they can quickly comment out individual lines. With the new AFSearchTokenBase, you can still comment individual lines. However, there's nothing stopping you from having a multi-line query string or better yet, a StringBuilder object. With a StringBuilder, you would still be passing a string to the search constructor, but you would have the query span multiple lines where you may comment a given line.

 

Let's see an example of this in action. We return once again to the fully-resolved, single-line query string:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

We can convert that a multiple line StringBuilder object. We really should have a separate StringBuilder object for the nested query and another for the full query.

 

StringBuilder nested = new StringBuilder();
nested.Append("Element:{");  // begin nested query
// Each inner filter will start with a blank
nested.Append($" Template:'{templateName}'");
nested.Append($" Name:'{elemName}'");
nested.Append($" (");  // begin OR expression
nested.Append($" '|{attrName}':<={lowLimit}");
nested.Append($" OR");
nested.Append($" '|{attrName}':>={highLimit}");
nested.Append(")");  // end OR expression
nested.Append("}");  // end nested query

StringBuilder query = new StringBuilder();
query.Append(nested.ToString());
query.Append($" Name:'{attrName}'");

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "", query.ToString()))
{
    // do something here
}

 

(A brief aside before we go any further, I want to say this to anyone who wants to have multi-line queries (string or tokens) in their code: I HOPE YOU ARE HAPPY. We have taken a single elegant line of a query string, and turned it into over a dozen lines of ugly code. Sorry. I needed to get that off my chest. Back to the lesson at hand ... )

 

Now let's consider what we would do if we want to temporarily change the search with these new rules:

 

  • We do not want to filter tank elements by name
  • We do not want to filter on low limit, which means we no longer need an OR clause. Another way to say this is we only need a value filter on high limit.

 

We would take the above and temporarily comment out lines accordingly:

 

StringBuilder nested = new StringBuilder();
nested.Append("Element:{");  // begin nested query
// Each inner filter will start with a blank
nested.Append($" Template:'{templateName}'");
//nested.Append($" Name:'{elemName}'");
//nested.Append($" (");  // begin OR expression
//nested.Append($" '|{attrName}':<={lowLimit}");
//nested.Append($" OR");
nested.Append($" '|{attrName}':>={highLimit}");
//nested.Append(")");  // end OR expression
nested.Append("}");  // end nested query

StringBuilder query = new StringBuilder();
query.Append(nested.ToString());
query.Append($" Name:'{attrName}'");

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "", query.ToString()))
{
    // do something here
}

 

There you go! We have demonstrated 3 examples for declaring a search query:

 

  • Using the new AFSearchTokenBase class
  • Using a single line query string
  • Using multi-lines of StringBuilder(s)

 

We leave it to your personal coding style and preference for what works best for you.

 

Caution using different Server and Client Versions

 

Let me issue this general caution regarding using different AF Server and AF Client version: while your app may compile correctly and run on mismatched versions, the features available would depend upon which system has the lowest version.  A client running 2.10.5 or later may be aware of the OR expression but a server running 2.10.0 or earlier would not.  Thus that app would be limited to 2.10.0 features.  If the client app tried to send an OR expression to the 2.10.0 server, an unsupported feature exception would be returned.

 

A very general rule of thumb: an older client app running against a newer server version should most likely run without issues, but a newer client app running against an older server version could throw unexpected errors.

 

 

Past AFSearch Blogs or Posts

 

Getting the most out of AFSearch - PI World Barcelona - Sept 2018. This used a beta of AF 2.10.5 to introduce OR support. Great video to help you understand the inner workings of AFSearch. This customer question hits on all the key points of the Barcelona Live Coding talk, and shows where lightweight searches can be great event frame searches with captured values. This Tech Talk was a refined repeat of May's Live Coding Talk at PI World 2018 San Francisco.

 

What's new in AFSearch 2.10 (PI AF 2018) (June 2018) - attribute searches are no longer are restricted to templates.

 

What's new in AFSearch 2.9.5 (PI AF 2017 R2) (March 2018) - AFAttributeSearch is introduced.

 

Coding Tips for using the AFSearch IN Operator - you may search for multiple values using IN(value1; value2; etc.). Some code is offered to make this easier to generate. This explains embedded blanks and using escaped double quotes or single quotes.

 

Using the AFEventFrameSearch Class (Nov 2016) - Giving some attention to event frames since many of earliest AFSearch examples were element based.

 

Aggregating Event Frame Data Part 1 of 9 - Introduction (May 2017) - From the UC 2017 TechCon/DevCon hands on lab for Advanced Developers.

 

Why you should use the new AF SDK Search (June 2016) - The granddaddy of them all. The earliest post explaining AFSearch, which was new at that time.

Introduction

 

Last week, just after publishing the blog post about how to send data from Node-RED to PI Web API, I got a call from a friend asking me how he could use a very similar setup to send raw binary data from the sensor to PI System and process it there. The answer is surprisingly simple and, after a quick look at several PI Square questions, I realised that this is a common question. So let's see how can we accomplish this in a very simple way, shall we?

 

Setup

 

The setup we will be using today is very similar to the one on my last post, so I strongly recommend you to give it a look before proceeding here. I will just go deeper into details about the sensor I'm using because we will need this information later on.

 

For this blog post, I will use my old and faithful BME280, a small temperature, humidity and pressure sensor that has a builtin calibration mechanism. Today we will only retrieve the temperature and calibration information, so we can keep the code simple and readable, but the procedure is pretty much the same for every sensor out there.

 

Because we will be dealing with binary data, it's important we start by giving a look at the sensor's datasheet so we understand how it's organized and where to get the information we need.

 

 

So here it is. Raw sensor data is available from 0xF7 to 0xFE (with temperatures on the first three bytes) and the calibration is a long sequence starting at 0x88 (if you read the datasheet you will see we only need the initial 6 bytes as the rest is for the other readings). The compensation formula is given in Appendix A and we will need to implement it on AF in order to process this data.

 

Sending the Data to PI

 

Once again, it's the same configuration from my last post, so I won't waste your time explaining all the details, but here's the Node-RED flow we will be using for this example:

This is pretty simple to follow. We start with an injection node that triggers the flow every five seconds. From there we go to an I2C node that reads data straight out of the I2C bus. Then we make some small adjustments to make it PI Web API friendly and we finally send it through a simple HTTP POST. The big difference here from the last time we did the same thing, is that now the I2C node creates a byte array as our payload:

 

 

So how do we POST an array to PI Web API? It's actually pretty simple. Considering that the PI Tag and AF Attribute are configured properly (we will see how to do that on the next secion), the JSON body should simply contain an array as the Value key. Then you POST to the Stream controller's UpdateValue method and you are good to go. Here's an example using Postman:

 

 

PI Tag and AF Attribute Configuration

 

There are two things we must consider in our configuration: the PI Tag type that will be used and the AF Attribute. Let's start with the AF Attribute configuration, where things are straightforward as the engine already exposes native array types. For this demo, we will use a byte array. Here's the config string of my attribute template, already set to tag creation: \\%Server%\%Element%.%Attribute%;ReadOnly=False;pointtype=Blob.

 

 

 

Because the sensor we are using sends data as a byte array, this will be the data type we will use. Keep in mind that it's not uncommon to see sensors sending data as Int arrays.

 

Now on the Data Archive side of our project, let's address a very common question: how do we store array data in the PI System? How should I configure a PI Tag to store array data? Here's a quote from LiveLibrary:

 

BLOB is the PI point type typically chosen for an arbitrary unstructured array of bytes. The value of a single event for a BLOB point is limited to binary data of up to 976 bytes in length. Data for larger binaries must be split into multiple events for the BLOB point or the data must be stored as an annotation.

 

So here's our answer: we must configure our PI Tag as a blob (by the way, blob means binary large object).

 

Processing the Data

 

At this point we already have data flowing in, an 8-byte array for our raw data a 6-byte array for the compensation factors:

 

  

 

Now we have to extract meaningful information out of it by implementing some transformations that will be able to convert the binary data into our final temperature value. In order to do this, we first have to check the sensor's datasheet to see how we convert the calibration factors into actual numbers that will go in the conversion formula. We will start with our calibration parameters and here's the info from the sensor documentation:

 

  

 

As I said before, we only need the first six bytes from the calibration information. So we now need to convert the bytes into actual numbers. Also, from this table, we now know T1 is an unsigned short and the other two are signed, so the transformation is simple. Here's how it's done in Python:

 

dig_T1 = cal[1] << 8 | cal[0]
dig_T2 = cal[3] << 8 | cal[2]
dig_T3 = cal[5] << 8 | cal[4]

if dig_T2 > 32767:
dig_T2 = dig_T2 - 65536

if dig_T3 > 32767:
dig_T3 = dig_T3 - 65536

 

In order to do that in AF, you have to use a little math because it doesn't offer the bitwise operators available in Python or C. The bitwise left shift (<< n) is equivalent of multiplying your number by 2^n while the binary OR ( | ) is a simple sum. Finally, we have to check if T2 and T3 are above 32767 because this is how signed ints work. This is how our final implementation is in AF (important: arrays on AF use one-based indexing! So to access the first to elements, we will use [1] and [2]):

 

 

Now we have to go back to the sensor's datasheet to see how can we convert the raw data into an actual number. Here's the information we need: 

 

 

On my Node-RED flow, I'm requesting data from 0XF7 to 0XFE so I don't need to make several requests to the I2C bus. This is important because, on our array, the MSB will be on position [4], LSB on [5] and XLSB on [6]. The Python script that does the bitwise operation to convert it into a decimal number is quite simple:

 

temp_raw = (block[3] << 16 | block[4] << 8 | block[5]) >> 4

 

In a similar fashion as before, we convert it to an AF Analysis script by using simple math:

 

  

 

We are almost there! We have all our reading as numbers and we can finally apply the conversion formula available on the sensor's documentation. Here's the C code they've provided:

 

double BME280_compensate_T_double(double adc_T)
{
double var1, var2, T;
const double K1 = 1024;
const double K5 = K1 * 5; // 5120
const double K8 = K1 * 8; // 8192
const double K16 = K1 * 16; // 16384
const double K128 = K1 * 128; // 131072

var1 = ((adc_T / K16) - (dig_T1 / K1)) * dig_T2;
var2 = ((adc_T / K128) - (dig_T1 / K8)) * ((adc_T / K128) - (dig_T1 / K8)) * dig_T3;
T = (var1 + var2) / K5;
return T;
}

 

This is easy peasy lemon squeezy for AF and I'm sure you will have no problem implementing this logic. Here's my complete analysis, where final is the output temperature in celsius:

 

  

 

Do I need this?

 

I reckon this is not for everyone. Most of the time we only need the final sensor reading. But some modern sensors and instruments are able to send more meaningful and important data, like maintenance flags, reading status and other parameters that may be useful for some teams, like instrumentation and maintenance.

Eugene Lee

Async Streaming with AF SDK

Posted by Eugene Lee Employee Jun 10, 2019

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

Introduction

Greetings everyone! 

 

In this blog post, we shall be discussing about a concept called Async Streaming and how we can use it with AF SDK to help make more responsive, scalable and concurrent applications.

 

Async Streaming is a new feature in C# that will be natively supported in version 8 and .NET Core 3. Even though AF SDK is not supported in .NET Core, there are still libraries available out there that can bring the benefits of Async Streaming to AF SDK.

 

Async Streaming can be advantageous in many cases. For example:

  1. In front-end applications, the main UI thread can stay responsive during a data access call.
  2. In both client and server applications, the number of threads used to service a call can be reduced, as waiting threads won't be blocked and can be returned to the thread pool for re-use.
  3. The effect of latency is mitigated because remote calls can be executed concurrently.
  4. Receiving data and processing it as it is retrieved in a way that doesn't block while we wait.

 

What is Asynchronous programming?

Asynchronous programming is a means of parallel programming in which a unit of work runs separately from the main application thread and notifies the calling thread of its completion, failure or progress.


This is the first thing you will find if you do a Google search using that term. In layman terms, this means that it will be able to help us achieve more responsive applications by not blocking the main thread.


AF SDK bulk calls

Let's examine the available data access bulk calls for PI Points that offer async behavior.

 

 

We notice that the methods with native async behavior tend to return one value per PI Point. If you look at their counterparts which return multiple values per PI Point, we find that they are not natively async.

 

 

I shall use the RecordedValues method as an example here. Below is a snippet of how we normally call this method.

 

We can make a wrapper called GetRecordedValues to return us a list of the recorded values. The return type is IEnumerable<AFValues>.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

private static IEnumerable<AFValues> GetRecordedValues(PIPointList pointList)
{
    PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 1);
    var timeRange = new AFTimeRange("*-10y", "*");

    try
    {
        var listResults = pointList.RecordedValues(timeRange, AFBoundaryType.Inside, null, true, config);
        return listResults;
    }
    catch (OperationCanceledException)
    {
        // Errors that occur during bulk calls get trapped here
        // The actual error is stored on the PIPagingConfiguration object
        Console.WriteLine(config.Error.Message);
        return null;
    }
    catch (Exception otherEx)
    {
        // Errors that occur in an iterative fallback method get trapped here
        Console.WriteLine(otherEx.Message);
        return null;
    }
}

 

And then we can consume the wrapper using a foreach loop.

 

var afvalslist = GetRecordedValues(pointList);
foreach (var pointResults in afvalslist)
{
    foreach (var item in pointResults)
    {
        Console.WriteLine("Timestamp: " + item.Timestamp + "\tValue: " + item.Value + "\tName: " + pointResults.PIPoint);
    }
    Console.WriteLine();
}

 

Now, this sample is generally fine in most cases. The only bad thing about it is that it doesn't have any async behavior. If the PI Data Archive is busy serving other users or applications, threads may get blocked such that responsiveness and performance will suffer. What can we do to improve upon our code?

 

This is where Async Streaming can save the day!

 

Async Streaming

Async Streaming makes it possible to await for a stream of results. As I mentioned in the introduction above, there are libraries out there to integrate AF SDK with Async Streaming. For this blog post, I am going to use one of them called AsyncEnumerator found here

 

https://www.nuget.org/packages/AsyncEnumerator/

 

The package can be easily installed from NuGet via

 

Install-Package AsyncEnumerator -Version 2.2.2


It introduces 2 new interfaces called IAsyncEnumerable and IAsyncEnumerator. Lets examine each of them to understand how it helps us to do Async Streaming.

 

public interface IAsyncEnumerator
{
    object Current { get; }
    Task<bool> MoveNextAsync(CancellationToken cancellationToken = default);
}

 

The Current property is the same as IEnumerator's version. It gets the element in the collection at the current position of the enumerator. What's different is the MoveNextAsync method. Over here, we can see that it returns a Task to us. Thus, we can start the task and continue on with our work while letting the task run in the background. MoveNextAsync does not block the thread compared to MoveNext of IEnumerator.

 

public interface IAsyncEnumerable
{
    Task<IAsyncEnumerator> GetAsyncEnumeratorAsync(CancellationToken cancellationToken = default);
}

 

GetAsyncEnumeratorAsync creates an enumerator that iterates through a collection asynchronously. This also returns a Task which returns an IAsyncEnumerator when it is complete.

 

General usage patterns

We can use a general construct such as the one below to consume an async stream of values. Take note that this construct is specific to the library being used. C# 8 has a very similar syntax. This pattern of iteration will not block the thread which is what we desire for our application.

 

await asyncEnumerable.ForEachAsync(async number => {
    await Console.Out.WriteLineAsync($"{number}");
});

 

Behind the scenes, the compiler will translate the ForEachAsync statement to utilize the MoveNextAsync method and then access the Current property to get the element of interest.

 

Cancellation

With this pattern, you can use a cancellation token to stop the streaming. This is useful for implementing timeouts or for the user to cancel the operation. If you look at the parameters of MoveNextAsync, you will notice that it accepts a cancellation token which you can use for notifying the streaming to stop. 

 

public virtual Task<bool> MoveNextAsync(CancellationToken cancellationToken = default)

public static async Task ForEachAsync(this IAsyncEnumerable enumerable, Action<object> action, CancellationToken cancellationToken = default)

 

The ForEachAsync extension method passes this token to MoveNextAsync where we can then retrieve this token with the yield.CancellationToken property to check for cancellation. An example is like the following.

 

token = yield.CancellationToken;
if (token.IsCancellationRequested)
{
    await Console.Out.WriteLineAsync("cancelling");
    yield.Break();
}

 

 

Async Streaming + AF SDK = GetStreamingRecordedValuesAsync

Now that we know what Async Streaming is about, let us improve upon the GetRecordedValues wrapper that was introduced in the previous section. We will leverage on the general usage patterns and also include cancellation in our wrapper.

 

We will call this wrapper GetStreamingRecordedValuesAsync. We will retrieve pages of results from the PI Data Archive one tag at a time as defined by the PIPagingConfiguration settings. The return type is IAsyncEnumerable<AFValue>.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

private static IAsyncEnumerable<AFValue> GetStreamingRecordedValuesAsync(PIPointList pointList)
{
    PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 1);
    var timeRange = new AFTimeRange("*-10y", "*");

    return new AsyncEnumerable<AFValue>(async yield =>
    {
        try
        {
            await Task.Run(async () =>
            {
                var listResults = pointList.RecordedValues(timeRange, AFBoundaryType.Inside, null, true, config);
                CancellationToken token;
                foreach (var pointResults in listResults)
                {
                    token = yield.CancellationToken;
                    if (token.IsCancellationRequested)
                    {
                        await Console.Out.WriteLineAsync("cancelling");
                        yield.Break();
                    }

                    foreach (var result in pointResults)
                    {
                        await yield.ReturnAsync(result);
                    }

                }
            });
        }
        catch (OperationCanceledException)
        {
            // Errors that occur during bulk calls get trapped here
            // The actual error is stored on the PIPagingConfiguration object
            await Console.Out.WriteLineAsync(config.Error.Message);
            yield.Break();
        }
        catch (Exception otherEx)
        {
            // Errors that occur in an iterative fallback method get trapped here
            await Console.Out.WriteLineAsync(otherEx.Message);
            yield.Break();
        }
    });
}

 

With this sample, we will be streaming the recorded values for each PI Point on the list. We can utilize the wrapper using the ForEachAsync loop and pass to it a cancellation token.

 

var cts = new CancellationTokenSource();
var afvalslist = GetStreamingRecordedValuesAsync(pointList);
await afvalslist.ForEachAsync(async item =>
{
    await Console.Out.WriteLineAsync("Timestamp: " + item.Timestamp + "\tValue: " + item.Value.ToString().PadRight(20) + "Name: " + item.PIPoint);
}, cts.Token);

 

This method of streaming ensures the calling thread doesn't get blocked and can continue with other work. To refresh your memory, a PIPointList can contain points from multiple PI Data Archives. For a global enterprise, your PI Data Archives could be scattered around the world. What if your application is hosted in USA but you need data from the server in Singapore? No matter what, latency will definitely affect its performance. You can't beat the laws of physics but at least you are free to do other work while waiting. That's what productivity and concurrency is about!

 

Point of caution

With Async Streaming on the client side, we can conveniently fire and forget calls. However, one has to keep in mind that the server will still need to process the data request.  If every single application just dumps all these data calls asynchronously to the server, it will have some negative effects on the server. Therefore, it is up to the user to implement some kind of throttling.

 

Conclusion

In this blog post, we have looked at what Async Streaming is and how it can help you make responsive, scalable and concurrent applications. In AF SDK, some bulk data calls might not have async methods. However, we can still use async streaming to improve the performance of our application utilizing these methods. I found a feature request here to expose asynchronous interfaces for bulk calls of AF Attributes. You can vote for it if you are interested.

https://pisquare.osisoft.com/ideas/5743-af-sdk-async-data-methods-for-multiple-afattributes

 

I hope you have learnt something useful from this article. Let me know if you have any comments!

Stream Updates allows a PI Web API client to retrieve data updates from PI Points and AF Attributes without using Channels (which are based on websockets). With Stream Updates, you register for data streams of interest with an HTTP POST request to a new "streams/updates" endpoint. Data updates are then retrieved by polling with a marker returned to you by the registration call. We first introduced Stream Updates as a Community Technology Preview (CTP) in PI Web API 2018. We've added some new features in PI Web API 2018 SP1.

 

Changes in AF metadata

Previously, you would not be notified if there was a change in AF metadata. Now, you are notified if metadata – like AF Data Reference, units of measure, and description – has changed. The exact metadata change is not reported. Query the AF Attribute to see what changed; then, register again for Stream Updates to receive further updates.

 

This feature reacts to AF metadata changes only. Changes to PI Point attributes are not reported.

 

If an AF Attribute metadata change has occurred, the response payload will contain this:

"Exception": {
"Errors": [
"The signup was updated and any cached data could be invalid."
]
}

 

There is one type of edit to an AF Attribute that will be reported to your client in PI Web API 2018 SP1. If the value of a static Attribute is changed, the new value will be returned to your client as a data update.

 

Marker error always available

If a passed marker is valid, but the AF Attribute you are tracking experiences an error, you will receive a message in the Exception section of your response payload. For example, if an AF Attribute you are tracking is deleted, you will get this error in your response payload:

"Exception": {
"Errors": [
"The signup was removed because it is no longer valid."
]
}

 

Selected Fields

Most controllers in PI Web API support the optional selectedFields parameter that allows you to choose which fields in the standard payload are returned. This parameter has been added to Stream Updates registration and data updates. For example, to retrieve only the latest marker and the data updates, you would issue an HTTP GET request to:

https://myserver.com/streams/updates/{marker}/selectedFields=LatestMarker;Events

 

PreviousEventAction

When processing data updates, it is often useful to know what happened to the previous event received by the PI Data Archive. We have added the PreviousEventAction item to each Event in the response payload. Possible values are PreviousEventArchived and PreviousEventDeleted.

 

Performance

How fast can you expect to retrieve data updates this way? We set up a test environment with a 34 GB PI Data Archive server with 6 cores receiving data values into 250,000 PI Points at 1 second intervals each. Our 8 GB PI Web API server had 8 cores. Our test program registered for Stream Updates in blocks of 40 points each. We are able to sustain 40,000 data updates per second. We did need to edit some of the PI Update Manager tuning parameters and PI Web API configuration parameters to achieve this retrieval rate. For PI Update Manager, we set MaxUpdateQueue and TotalUpdateQueue to 2 million to overcome the default 50,000 queue size. We set the CacheInstanceUpdateInterval parameter in PI Web API to 1 (second) and the CacheInstanceUpdateHoldoffTime parameter to 0 (seconds) to more quickly query for new events and therefore empty the queue faster.

Knowing that PI Data Archive is capable of significantly higher throughput, we believe the current throughput is memory-bound in PI Web API; however, we also believe the throughput is sufficient for most uses of Stream Updates. If you have specific use cases where higher throughput is needed, please share them with us at https://feedback.osisoft.com.

Additionally, we do not recommend using the exact tuning parameters values as described above - totalUpdateQueue should be greater than MaxUpdateQueue. For additional information on these PI Data Archive tuning parameters, please refer to Knowledge Base article KB3151OSI8.

 

Sample C# code demonstrating the new features

This sample is an update from our original blog post:

PIWebAPIClient.cs

public class PIWebAPIClient 
    {
        private HttpClient client;
        private string baseUrl;
        public PIWebAPIClient(string url, string username, string password)
        {
            client = new HttpClient();
            if (!String.IsNullOrEmpty(username) && !String.IsNullOrEmpty(password))
            {
                string auth = Convert.ToBase64String(Encoding.ASCII.GetBytes(string.Format("{0}:{1}", username, password)));
                client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", auth);
            }
            client.DefaultRequestHeaders.Add("X-Requested-With", "asdf");  // avoids CSRF warning
            baseUrl = url;
        }
        public PIWebAPIClient(string url)
        {
            client = new HttpClient();
            baseUrl = url;
        }
        public async Task<object> GetAsync(string uri)
        {
            HttpResponseMessage response = await client.GetAsync(uri);
            var jsonString = await response.Content.ReadAsStringAsync();
            var json = JsonConvert.DeserializeObject<object>(jsonString);
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + jsonString);
            }
            return json;
        }
        public async Task<object> PostAsync(string uri)
        {
            HttpResponseMessage response = await client.PostAsync(uri, null);
            var jsonString = await response.Content.ReadAsStringAsync();
            var json = JsonConvert.DeserializeObject<object>(jsonString);
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + jsonString);
            }
            return json;
        }
        public async Task<dynamic> RegisterForStreamUpdates(string webId, string selectedFields)
        {
            string url = baseUrl + "/streams/" + webId + "/updates";
            if (!String.IsNullOrEmpty(selectedFields))
            {
                url += "?selectedFields=" + selectedFields;
            }
            dynamic response = await PostAsync(url);
            return response;
        }
        public async Task<dynamic> RetrieveStreamUpdates(string marker, string selectedFields)
        {
            string url = baseUrl + "/streams/" + "/updates/" + marker;
            if (!String.IsNullOrEmpty(selectedFields))
            {
                url += "?selectedFields=" + selectedFields            
            }
            dynamic response = await GetAsync(url);
            return response;
        }
        public string GetVersion()
        {
            string url = baseUrl + "/system";
            dynamic response = GetAsync(url).Result;
            return response.ProductVersion;
        }
    }

 

Program.cs

class Program 
    {
        static string baseUrl = "https://my.server.com/piwebapi";
        static string marker = null;
        static string username = "username";
        static string password = "password";
        static string webId = "myWebId";
        static PIWebAPIClient client = null;
        static void Main(string[] args)
        {
            client = new PIWebAPIClient(baseUrl, username, password);
            Console.WriteLine("PI Web API Version: {0}", client.GetVersion());
            // Register for Stream Updates requesting only LatestMarker and Status
            dynamic response = client.RegisterForStreamUpdates(webId, "LatestMarker;Status").Result;
            marker = response.LatestMarker;
            string stat = response.Status;
            string src = response.SourceName;
            //ReceiveUpdates is called every 10 seconds until you explicitly exit the application
            TimeSpan startTimeSpan = TimeSpan.Zero;
            TimeSpan periodTimeSpan = TimeSpan.FromSeconds(10);
            var timer = new System.Threading.Timer((e) =>
            {
                Console.WriteLine("{0},{1}", DateTime.Now.ToString(), marker);
                ReceiveUpdates(ref marker);
            }, null, startTimeSpan, periodTimeSpan);
            Console.ReadLine();
        }
        public static void ReceiveUpdates(ref string marker)
        {
            dynamic update = client.RetrieveStreamUpdates(marker, "Status;LatestMarker;Events;Exception.Errors").Result;
            //dynamic update = client.RetrieveStreamUpdates(marker, null).Result;
            Console.WriteLine(update);
            Console.WriteLine("Press the Enter key to exit anytime!");
            marker = update.LatestMarker;
            dynamic excp = update.Exception;
            if (excp != null)
            {
                throw new System.Exception(excp.Errors[0].ToString());
            }
        }
    }
rborges

PI Web API and Node-RED

Posted by rborges Employee May 21, 2019

1. Introduction

1.1 What is Node-RED?


Before going to the juicy bits of this blog post, let me start by explaining what Node-RED is for those who are not familiar with this tool. Quoting their own website, Node-RED is a flow-based programming tool for the internet of things. What this means, in simple English, is that you can drag and drop functional boxes for wiring together devices and online services.

 

This is what a Node-RED flow looks like:

A group of nodes is called a flow. Each node is responsible for manipulating an object called message. In this flow, Node-RED is listening for UDP packets and, based on the destination port, it can either pull high GPIO 15 or ground GPIO 13 while sending an HTTP request to a given API (and keep trying while not succeed).

 

Another selling point is the community support. Being an open source project (Apache 2.0 if you are curious), community engagement has skyrocketed and you can find thousands of different nodes: from OPC UA and Modbus nodes to Instagram and other social media platforms. From FFT to sentiment analysis. So most likely you won't even need to write a single line of code.

 

1.2 Why should I use it?

 

Rafa, usually graphical programming is just a dumbed-down version of actual programming. Why should I use it if I can write my own application? Because it's easy and fast. As a software engineer, I agree that a tailor-made firmware in C is more reliable and efficient. But can you create one in seconds? Can you easily include new features or make changes to it? This is an old debate where, on one hand, we have practicality and, on the other hand, robustness.

 

Let's just remember that we are talking about IoT devices, frequently located outside the boundaries of the main network and sending data through a non-wired channel. Unpredictability and unreliability are a given and your logic can go easily go sideways with an unforeseen condition. So would you prefer to update your code through a web interface or going on the field with a computer and a USB UART dongle to reflash your firmware?

 

1.3 Caveats


Everything sounds great, but it's not all roses. Although it has a very small footprint, you can't deploy Node-RED on small microchips (ESP8266, if you are wondering) or limited hardware like Arduino. Right now, the only "IoT hardware" capable of running it is Raspberry PIs and Beagle Bone Black. This poses a challenge if you are considering deploying the platform on the field, but there are alternatives if you need to use minimalist hardware.

 

Another caveat is the engine itself. Its capabilities are as big as the capabilities of the device hosting it. If you deploy it on a small Raspberry PI, don't expect the performance of deployment on a full-blown server. It may sound like an obvious observation, but because IoT devices can scale up easily on a mesh network, sometimes we forget that the host doesn't scale up that easily. So, your mileage may vary when it comes to performance.

 

1.4 Architectural Considerations

 

In today's example, we will have multiple sensors sending data directly through an access point (a Raspberry PI 3 B+ in our case). A rough representation is this:

 

For the geeks out there who are curious about the setup I'm using, here's more info: I'm using BME280 for temperature, humidity and pressure and a DS1307 for real-time clock. All of them are using the I2C bus to send data through an ESP-01 breakout. The ESP8266 is running a custom firmware that creates a mesh network. The Raspberry PI host is connected to the main corporate network through its own wi-fi module, but it's also connected to the mesh network through its own ESP-01 breakout. Each sensor is sending its data to specific MQTT topics on a Mosquitto broker running on the same Raspberry PI. I have three sets of sensors: one is sitting on my desk, the second one is at the office's kitchen, and the last one is in a meeting room, about 20m (65 ft) from my desk.

 

 

Although the hardware side is not related to PI, leave a comment below if you would like to hear a little bit more about what I'm doing. By the way, this is a preparation for an ongoing project, where a more professional looking version this will eventually be deployed around our office here in London.

 

Keep in mind that this is not a suitable production-grade architecture as there is no data buffer, no redundancy, no scalability, and no fail-safe mechanism. If you need any of these items (and you need if data is critical to your operation!), you should consider giving a look at EDS (Edge Data Store), our answer to data management on the edge (watch this presentation about EDS, you won't regret it).

 

2. Sample Flows

 

2.1 Reading Values from the Sensors

 

This is not a Node-RED tutorial, but let me just show how I'm capturing the data. After all, this is how everything starts. As I mentioned before I'm running an MQTT broker and each sensor data has it's own topic. Because I like to sort my data by data domains, my topics are the following:

 

DeviceMeasurementTopic
Desk SensorTemperatureoffice/temperature/desk
Desk SensorHumidityoffice/humidity/desk
Desk SensorPressureoffice/pressure/desk
Kitchen SensorTemperatureoffice/temperature/kitchen
Kitchen SensorHumidityoffice/humidity/kitchen
Kitchen SensorPressureoffice/pressure/kitchen
Meeting Room SensorTemperatureoffice/temperature/meetingroom
Meeting Room SensorHumidityoffice/humidity/meetingroom
Meeting Room SensorPressureoffice/pressure/meetingroom

 

This allows me to easily get to all temperatures by subscribing to office/temperature/# or get all office data by subscribing to office/#. I can also get only my desk data by subscribing to office/+/desk. Here's an example of how to get the data using an MQTT node:

Dead simple, right? The green node is a debug node that outputs all the content of the payload. Here's the output for a single topic. Keep in mind that we receive several messages like this (one for each topic we are subscribed to).

 

 

Because I'm trying to keep things as streamlined as possible, the sensors are already streaming an output that is pretty much what we need to send to PI, including the AFPath where we are sending data to. On my custom ESP8266 firmware I have this hardcoded:

 

const String baseAFPath = "\\\\RBORGES-AF\\IoTDemo\\Rafael Desk Environmental Data|";
const String tempAFPath = baseAFPath + "temperature";
const String humiAFPath = baseAFPath + "humidity";
const String pressAFPath = baseAFPath + "pressure";
const String dpointAFPath = baseAFPath + "dewpoint";

 

In a more real-life scenario, instead of a hardcoded string, you should do something like this:

 

const String chipID = ESP.getChipId();
const String tempTopic = "office/temperature_"+ chipID + "/desk";
const String humiTopic = "office/humidity_"+ chipID + "/desk";
const String presTopic = "office/pressure_"+ chipID + "/desk";
const String vccTopic = "office/vcc_"+ chipID + "/desk";
const String ipTopic = "office/ip_"+ chipID + "/desk";

 

Then, on your Node-RED server, you would correlate the chip ID with the attributes you are sending data to. That has the extra benefit of easy maintenance, in the case of you moving the sensor to a different location. There's no need for you to rewrite the firmware, just update the reference table.

 

By the way, talking about real-life scenarios, if you are using the PI System infrastructure, you should consider OMF as your format standard, in the near future, you will be able to send OMF data right away to PI Web API. This will free you from the transformations we have to do in the next section.

 

2.2 Sending Values to PI


Now that we have the data, we need to send it to PI. As the blog title suggests, we will use PI Web API as our data entry point. In order to do this, we have to make some transformations on our data. First, we have to add the WebID for the attribute we want to write data to and then we have to execute an HTTP POST with a JSON body containing the data itself.

 

The first thing is to get the WebID. Using Web ID 2.0, it's actually pretty easy to encode the AF path into a valid Web ID. I strongly suggest you Christopher Sawyer's excellent post on how to encode, decode, and some basic concepts behind it.

 

Going back to Node-RED, in order to encode the path into a valid Web ID, we have to execute custom code. This is easily done with the function node, where you can run any arbitrary javascript code. It exposes the whole message as a plain JavaScript object and allows you to manipulate it in a programmatic way.

 

path = msg.path
if (path.substring(0, 2) == "\\\\") {
path = path.substring(2);
}
var encoded_path = new Buffer(path.toUpperCase()).toString('base64');
var count = 0;
encoded_path = encoded_path.replace('+', '-').replace('/', '_');
for (var i = (encoded_path.length - 1); i > 0; i--) {
if (encoded_path[i] == "=") {
count++;
}
else {
break;
}
}
if (count > 0) {
encoded_path = encoded_path.slice(0, -count);
}
msg.webid = "P1AbE" + encoded_path
return msg;

Do you get the idea? I'm just getting the path variable from our message, encoding in base64 and concatenating with "P1Abe". The P1Abe WebID header means it encodes the path and refers to an AFAttribute that is a child of an AFElement. Once again, if you have not checked Chris' blog post, stop reading this article now and go read it! A final note for those JavaScript nerds wondering why I'm not using btoa(). Node-RED, as the name suggests, run on Node.js and it doesn't expose btoa() / atob().

 

At this point, this is our message object:

 

Our payload is ready and we have the Web ID encoded, so we are pretty much ready to send the data to PI. To do this, we now need the HTTP Request node.

 

It works by getting the payload content and sending it as a JSON body to a given URL. Here's the configuration for our example:

We define it as a POST to the Stream Controller Update Value Method, we set the URL, I enable SSL to properly handle the SSL certificate and finally, I select basic authentication. Let me just call your attention to the URL I'm using https://rborges-af/piwebapi/streams/{{webid}}/value. See the {{webid}}? It's a template system called Mustache Notation. It allows us to get a value from the message object and use it to feed the template.

 

Here's the full flow:

A recap: we subscribe to an MQTT topic, we convert the string into a valid JSON, we do some housekeeping by moving the path information out of the payload, we use the path to encode our WebID and we finally post the data. I added the debug node as an output of our request, so we can get the HTTP Response object and see if it's everything working properly. Here's the output:

 

 

The 202 status code on our response means that the data was sent. We can now see it on AF:

 

 

2.3 A Periodic Calculation Engine

 

Another cool thing that we can do with Node-RED, is to deploy flows that are triggered periodically. The Inject node allows us to not only inject an arbitrary JSON but also do it in a regular fashion. Here's an example. We first inject a JSON with two important tags that will be used for a given analysis:

 

 

Then we configure it to do it periodically. You can define a frequency (e.g., every 10 minutes), an interval between times (e.g., every 10 minutes between 8 AM and 3 PM), or at specific times (every noon on weekdays). For our example, every minute.

 

Now we can use it as the start trigger for our flow.

 

In this example, we are injecting the JSON every minute, we split it to get data from the PI Web API for each PI tag, we join the messages into a single message, we pass it through a generic splitter where we prepare it for FFT and the result we send it to our maintenance database. I'm using our beloved CDT158 and SINUSOID, but it could easily be vibrational data so we can log vibration information from a maintenance perspective.

 

Keep in mind that the inject node is just on way to do it. There are plenty of other ways for you to trigger a flow. Another possibility is to listen to multiple tags and start it only when a condition is met. I helped a customer a couple of years ago to wire up some electronics and trigger a flow when a door was opened and closed. This was used to log on PI when the lab door was open.

 

3. Conclusion

 

3.1 The PI System and IoT data

 

Today we saw how easy it is for you to wire up sensor data with PI, using nothing else than PI Web API requests. It's simple, cost-effective and fun to execute. I actually use this same architecture for my home automation system with PI (let me know in the comments if you would like to know why I use PI at my home).

 

But let me stress once again that this is just a proof-of-concept. An enterprise-grade project would never ever send data directly like that as it's a security breach and the lack of buffer makes it very unreliable. Once again give a look on EDS if you need to send edge data to your PI System.

 

3.2 Reference Material

In this blog we will discuss AF SDK's PIPoint.FindPIPoints method when passing in an enumerable collection of tag names.  As far as immediate usage of the method, there is very little to add that isn't already in the help.  A general weakness of many help examples is that the example is intentionally short, simple, and typically has perfect data.  This blog is being written with the specific thought that some of the tag names passed into FindPIPoints intentionally will not find a tag.  We offer a couple of ideas on what you can do to discover which ones were not found, though any good developer should not feel limited to just the ideas presented here.

 

My inspiration for this blog comes from 2 different partner code reviews hosted months apart.  Both applications used the PIDataPipe where a sizeable list of tag names were being read from a text file.  Despite both developers being quite skillful, I was taken aback that both would try to find each tag one-at-a-time.  Overall, their logic was similar:

 

  • Open the text file and read a tag name for each line.
  • Issue a PIPoint.TryFindPIPoint to find the current tag by its name.
  • If the tag was not found, log it for later diagnostics.
  • If the tag was found, issue a AddSignups for that tag, even if it is the only tag in a list.

 

As I mentioned, these developers were quite skilled and knew about bulk calls.  This was apparent in each application where they would later issue a RemoveSignups in bulk.  When I asked one of them why they would read 5000 tags one-at-a-time instead of in bulk, the answer was all about the logging.  It was critical to their application that it logs whenever a tag is not found.

 

The thing about FindPIPoints is that it only returns what is found (doh!).  If you pass in 1000 tag names and 100 of them are not found, then what you get back is 900 PIPoints.  What we will explore here are ways to quickly, easily, and efficiently discover which 100 were not found.  That is the missing piece as to why these developers were not using a bulk call.  This is not a hard task to code up, and the possible solutions covered here rely more upon simple .NET Framework objects, and very little to do with AF SDK (that is, beyond the FindPIPoints call).

 

Case Insensitive Dictionary

The coding challenge is to reconcile the list of tag names you pass into FindPIPoints and quickly determine which ones didn't make the cut.  As an application developer, my first choice to solve this would be a dictionary keyed by the tag name (a string) with a lookup value of a PIPoint.  I would also want this dictionary to be case insensitive just in case your exact casing of the tag name in the text file does not exactly match the casing of the tag name on the PI Data Archive.

 

The logic would be that every tag name you pass in will be a key in the dictionary.  We then will make a one bulk call to FindPIPoints.  We enumerate over the returned list of PIPoint to populate the dictionary with the found PIPoints.  Afterwards, any remaining dictionary entry that does not have a valid PIPoint would indicate a tag name that was not found.  The biggest takeaway is not the mechanics of the dictionary, but rather that we issued only one efficient bulk call to the PI Data Archive.

 

Since my example method calls FindPIPoints and FindPIPoints has an optional 3rd parameter of attributeNamesthen I too want my custom method to mirror the signature of FindPIPoints. 

 

LEGAL DISCLAIMER

Since we will share some sample .NET code, we must provide the obligatory legal notice and disclaimers.  The sample code being shared herein is subject to the following disclaimer:

 

Copyright 2019 OSIsoft, LLC.

 

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

 

To put this another way, if you have problems with these sample methods, do NOT call Tech Support, nor create a case in my.OSIsoft.com. The only outlet for discussing issues with this code is here within the PI Developers Club forum.

 

The code being shared below:

  • Has not been put through any regression testing.
  • Has a very limited test profile (only me on my local PI Data Archive)
  • Has not been endorsed by any OSIsoft Product team
  • Is in no way an official offering by OSIsoft
  • Is offered only as a learning example

 

With that said, we strongly advise that you do not use this in your production code.  Keep in mind that we cannot stop you from doing so, but if you do, then it's no longer our code but rather is becomes YOUR code to support, maintain, validate, debug, and assume the full responsibilities as owner of said code.  Such is the nature of an "AS IS" license.

 

C# Sample Implementation Returning a Dictionary

public static IDictionary<string, PIPoint> GetExampleDictionary(PIServer piServer, IEnumerable<string> names, IEnumerable<string> attributeNames = null)
{
    if (piServer == null)
        throw new ArgumentNullException(nameof(piServer));
    if (names == null)
        throw new ArgumentNullException(nameof(names));

    // We will use a case-insensitive dictionary keyed by tag name with a Value of PIPoint.
    // A null PIPoint would indicate a tag name that was not found.
    var dict = new Dictionary<string, PIPoint>(StringComparer.OrdinalIgnoreCase);

    // We want every tag name to have an entry in our dictionary.
    // Initially the PIPoint will be null, which we try to populate shortly below.
    foreach (var tagname in names)
    {
        if (tagname != null)
            dict[tagname] = null;
    }

    if (dict.Count == 0)
        return dict;

    // Live Library: https://techsupport.osisoft.com/Documentation/PI-AF-SDK/html/M_OSIsoft_AF_PI_PIPoint_FindPIPoints_2.htm
    // If a tag name is not found, an exception is not thrown nor is a null PIPoint returned.
    // FindPIPoints will only return PIPoints that were actually found.
    var tags = PIPoint.FindPIPoints(piServer, names, attributeNames);
           
    // Finally we will assign the found PIPoints to its proper entry in the dictionary.
    // Note that some dictionary entries may still have null PIPoints for those tag names that were not found.
    foreach (var tag in tags)
    {
        dict[tag.Name] = tag;
    }

    return dict;
}

 

Great.  We issued a bulk call and thanks to a tiny bit of effort on our part, we now have a handy little dictionary at our fingertips.  Let's see an example on how we could consume this dictionary depending upon a given context of working with tags that have been found & validated, or working with tag names needing to log a "Not Found" message.  For setting up this example, we want to have a few good tag names along with some bad tag names.  For the bad ones, let's include some crazy edge cases of wild card patterns being used, duplicate names with different casing, or even a tag name containing illegal characters!

 

string wicked = "* ? ; { } [ ] | \\ ` ' \" ,"; // Every character not allowed in a tag name
string[] tagnames = new[] { "CDM158", "SINUSOID", "sinusoid", "sinus*", "This tag does not exist", "", "     ", wicked };

PIServer pida = new PIServers().DefaultPIServer;

var dict = LearningExample.GetExampleDictionary(pida, tagnames);
var points = dict.Values.Where(x => x != null).ToList();
var badNames = dict.Where(x => x.Value == null).Select(x => x.Key).ToList();

Console.WriteLine($"PIPoints Found: {points.Count}");
foreach (var point in points)
    Console.WriteLine($"   {point.Name}");

Console.WriteLine($"PIPoints NOT Found: {badNames.Count}");
foreach (var badName in badNames)
    Console.WriteLine($"   '{badName}'");

 

And that sample code might produce the following:

 

Sample Console Output
PIPoints Found: 2
   CDM158
   SINUSOID
PIPoints NOT Found: 5
   'sinus*'
   'This tag does not exist'
   ''
   '    '
   '* ? ; { } [ ] | \ ` ' " ,'

 

The help warns us that we cannot use wildcard patterns, so we understand why 'sinus*' is not found.  Observe that I even threw in some blank tag names, one with zero length and one with many blanks (which is why I quote the ones that are missing).  And the last line tells me that an exception will not be thrown even if you use illegal characters in the tag name being searched.

 

All my consuming application needed to do is take a little handling of whether a PIPoint was found or not.  Simple concept, simple code, but big performance difference when dealing with thousands of tags.

 

VB.NET Sample Implementation Returning a Dictionary

Public Function GetExampleDictionary(ByVal piServer As PIServer, ByVal names As IEnumerable(Of String), Optional ByVal attributeNames As IEnumerable(Of String) = Nothing) As IDictionary(Of String, PIPoint)

    If piServer Is Nothing Then Throw New ArgumentNullException(NameOf(piServer))
    If names Is Nothing Then Throw New ArgumentNullException(NameOf(names))

    ' We will use a case-insensitive dictionary keyed by tag name with a Value of PIPoint.
    ' A Nothing PIPoint would indicate a tag name that was not found.
    Dim dict = New Dictionary(Of String, PIPoint)(StringComparer.OrdinalIgnoreCase)

    ' We want every tag name to have an entry in our dictionary.
    ' Initially the PIPoint will be Nothing, which we try to populate shortly below.
    For Each tagname In names
        If tagname IsNot Nothing Then dict(tagname) = Nothing
    Next

    If dict.Count = 0 Then Return dict

    ' Live Library: https://techsupport.osisoft.com/Documentation/PI-AF-SDK/html/M_OSIsoft_AF_PI_PIPoint_FindPIPoints_2.htm
    ' If a tag name is not found, an exception is not thrown nor is a null PIPoint returned.
    ' FindPIPoints will only return PIPoints that were actually found.
    Dim tags = PIPoint.FindPIPoints(piServer, names, attributeNames)

    ' Finally we will assign the found PIPoints to its proper entry in the dictionary.
    ' Note that some dictionary entries may still have Nothing PIPoints for those tag names that were not found.
    For Each tag In tags
        dict(tag.Name) = tag
    Next

    Return dict
End Function

 

And here's an example of how VB.NET could consume that dictionary:

 

Dim wicked As String = "* ? ; { } [ ] | \ ` ' "" ,"  ' Every character not allowed in a tag name
Dim tagnames As String() = {"CDM158", "SINUSOID", "sinusoid", "sinus*", "This tag does not exist", "", "     ", wicked}

Dim pida As PIServer = New PIServers().DefaultPIServer

Dim dict = LearningExample.GetExampleDictionary(pida, tagnames, Nothing)
Dim points = dict.Values.Where(Function(x) x IsNot Nothing).ToList()
Dim badNames = dict.Where(Function(x) x.Value Is Nothing).Select(Function(x) x.Key).ToList()

Console.WriteLine($"PIPoints Found: {points.Count}")
For Each point In points
    Console.WriteLine($"   {point.Name}")
Next

Console.WriteLine($"PIPoints NOT Found: {badNames.Count}")
For Each badName In badNames
    Console.WriteLine($"   '{badName}'")
Next

 

Alternatively Return a ValueTuple

There are more than one way to do things and the above was my first train of thought.  I am the type of person that ponders other ways to try things to see which might be faster or easier to work with.  The problem I have with a dictionary being returned is that extra care and handling that must be done.  Other than the fact that you send in one list of tag names, what I really want coming back are 2 mostly unrelated things: I want a list of the PIPoints that were found and I want a separate list of the tag names where a PIPoint was not found.

 

Wanting to avoid the cumbersomeness of using out modifiers on the signature of a void method, I next considered tuples.  In particular, the ValueTuple class.

 

If you are using Visual Studio 2017 with .NET 4.6.2, you will need to install the NuGet package for ValueTuple.  If you are using Visual Studio 2019 with .NET 4.7.2, ValueTuple is now a part of mscorlib.  Tip: if you are converting from VS 2017 to VS 2019 and also upgraded the target framework being used, you will need to uninstall the NuGet ValueTuple package.

 

Internally, the methods will still use a case insensitive dictionary.  The only difference is what type of object we will return.

 

C# Sample Implementation Returning a ValueTuple

public static (IList<PIPoint> FoundList, IList<string> MissingList) GetExampleValueTuple(this PIServer piServer, IEnumerable<string> names, IEnumerable<string> attributeNames = null)
{
    if (piServer == null)
        throw new ArgumentNullException(nameof(piServer));
    if (names == null)
        throw new ArgumentNullException(nameof(names));

    // We will use a case-insensitive dictionary keyed by tag name with a Value of PIPoint.
    // A null PIPoint would indicate a tag name that was not found.
    var dict = new Dictionary<string, PIPoint>(StringComparer.OrdinalIgnoreCase);

    // We want every tag name to have an entry in our dictionary.
    // Initially the PIPoint will be null, which we try to populate shortly below.
    foreach (var tagname in names)
    {
        if (tagname != null)
            dict[tagname] = null;
    }

    if (dict.Count == 0)
        return (new PIPointList(), new List<string>());

    // Live Library: https://techsupport.osisoft.com/Documentation/PI-AF-SDK/html/M_OSIsoft_AF_PI_PIPoint_FindPIPoints_2.htm
    // If a tag name is not found, an exception is not thrown nor is a null PIPoint returned.
    // FindPIPoints will only return PIPoints that were actually found.
    var tags = PIPoint.FindPIPoints(piServer, names, attributeNames);

    // Finally we will assign the found PIPoints to its proper entry in the dictionary.
    // Note that some dictionary entries may still have null PIPoints for those tag names that were not found.
    foreach (var tag in tags)
    {
        dict[tag.Name] = tag;
    }

    // The variable 'tags' has the found PIPoints.
    // Let's create a list of tag names of any PIPoints that were not found, if any.
    var missing = dict.Where(x => x.Value == null)?.Select(x => x.Key);

    // Return a tuple of the found tags and the missing tag names.
    // The consumer can pass tags directly into a PIDataPipe.AddSignups,
    // and log the missing tag names.
    return (tags, missing.ToList());
}

 

C# allows different ways we can consume this object.  Both C# and VB.NET allow you to grab the returned ValueTuple as an singular object and then use its different fields:

 

var tagLookup = pida.GetExampleValueTuple(tagnames);

Console.WriteLine($"PIPoints Found: {tagLookup.FoundList.Count}");
foreach (var point in tagLookup.FoundList)
    Console.WriteLine($"   {point.Name}");

Console.WriteLine($"PIPoints NOT Found: {tagLookup.MissingList.Count}");
foreach (var badName in tagLookup.MissingList)
    Console.WriteLine($"   '{badName}'");

 

In the first line above, the tagLookup.FoundList will be a IList<PIPoint> and the tagLookup.MissingList will be an IList<string>.

 

The other nice way (which VB.NET lacks BTW) is to decompose the fields into individual variables as they are declared:

 

var (points, badNames) = pida.GetExampleValueTuple(tagnames, null);

Console.WriteLine($"PIPoints Found: {points.Count}");
foreach (var point in points)
    Console.WriteLine($"   {point.Name}");

Console.WriteLine($"PIPoints NOT Found: {badNames.Count}");
foreach (var badName in badNames)
    Console.WriteLine($"   '{badName}'");

 

In the first line above, the variable points will be a IList<PIPoint> and the variable badNames will be an IList<string>.

 

VB.NET Sample Implementation Returning a ValueTuple

<Extension()> Public Function GetExampleValueTuple(ByVal piServer As PIServer, ByVal names As IEnumerable(Of String), Optional ByVal attributeNames As IEnumerable(Of String) = Nothing) As (TagsFound As IList(Of PIPoint), MissingTagNames As IList(Of String))

    If piServer Is Nothing Then Throw New ArgumentNullException(NameOf(piServer))
    If names Is Nothing Then Throw New ArgumentNullException(NameOf(names))

    ' We will use a case-insensitive dictionary keyed by tag name with a Value of PIPoint.
    ' A Nothing PIPoint would indicate a tag name that was not found.
    Dim dict = New Dictionary(Of String, PIPoint)(StringComparer.OrdinalIgnoreCase)

    ' We want every tag name to have an entry in our dictionary.
    ' Initially the PIPoint will be Nothing, which we try to populate shortly below.
    For Each tagname In names
        If tagname IsNot Nothing Then dict(tagname) = Nothing
    Next

    If dict.Count = 0 Then Return (New List(Of PIPoint)(), New List(Of String)())

    ' Live Library: https://techsupport.osisoft.com/Documentation/PI-AF-SDK/html/M_OSIsoft_AF_PI_PIPoint_FindPIPoints_2.htm
    ' If a tag name is not found, an exception is not thrown nor is a null PIPoint returned.
    ' FindPIPoints will only return PIPoints that were actually found.
    Dim tags = PIPoint.FindPIPoints(piServer, names, attributeNames)

    ' Finally we will assign the found PIPoints to its proper entry in the dictionary.
    ' Note that some dictionary entries may still have Nothing PIPoints for those tag names that were not found.
    For Each tag In tags
        dict(tag.Name) = tag
    Next

    ' The variable 'tags' has the found PIPoints.
    ' Let's create a list of tag names of any PIPoints that were not found, if any.
    Dim missing = dict.Where(Function(x) x.Value Is Nothing).Select(Function(x) x.Key)

    ' Return a tuple of the found tags and the missing tag names.
    ' The consumer can pass tags directly into a PIDataPipe.AddSignups,
    ' and log the missing tag names.
    Return (tags, missing.ToList())

End Function

 

As mentioned above, VB.NET is unable to decompose the fields on one single line. Thus you must use the entire ValueTuple object and work with its fields.  Here is a simple example:

 

Dim tagLookup = pida.GetExampleValueTuple(tagnames)
Dim points = tagLookup.TagsFound
Dim badNames = tagLookup.MissingTagNames

Console.WriteLine($"PIPoints Found: {points.Count}")
For Each point In points
    Console.WriteLine($"   {point.Name}")
Next

Console.WriteLine($"PIPoints NOT Found: {badNames.Count}")
For Each badName In badNames
    Console.WriteLine($"   '{badName}'")
Next

 

Conclusion

The code above has not done anything special with the FindPIPoints method.  In fact, we are using it exactly for the purposes of which it was intended, and also exactly in the manner in which it was intended.  All we really did was considered various ways to take the results returned by FindPIPoints and transform them into another object more conducive to certain applications.  We then threw both good and bad cases at it to see if that transformation behaves as we expected.  These sample methods just ever so slightly touched upon AF SDK, and the subsequent transformation relies on very common, very routine .NET objects.

 

We showed 2 possible ways for you to transform the results.  And I am sure these are not the only ways.  I encourage you to contemplate these and other techniques to find what works best for you and your company based on your coding experience, preferences, style, and company policies.

Introduction

 

PI Web API 2017 R2 is the first version that implements Web ID 2.0, which is great to improve the performance of your applications. Using this feature, your app doesn't need to make an HTTP request in order to get the Web ID of a PI System object since now you can generate it directly on the client side. If you want to learn more about Web ID 2.0 please refer to this material.

 

The purpose of this blog post is to show a simple way to generate Web IDs 2.0 in Python and Java.

 

Explaining the logic

 

Using the code below, you can generate Web IDs 2.0 Path only type with the following structure:

  • The first character will always be 'P'.
  • The second character will always be '1', which means that the Web ID version is 2.0.
  • Then we have the Marker with two characters which refers to the object's type.
  • Then we have the Owner Marker with 1 character (optional).
  • Finally, we have the Name Payload which encodes the following string PIObject.GetPath().Substring(2).ToUpperInvariant().

 

If you take a look at the Web ID 2.0 Specification Tables, you will realize that the Owner Marker needs to be defined only for some object types. The reason is that some object types could belong to different owners.

 

Let's take an example. We know that a PI Point will always belong to a PI Data Archive. Nevertheless, an AF Attribute could belong to an AF Element or to an AF Event Frame. This is why you don't need to define an Owner Marker for a PI Point but you do need to do it for an AF Attribute.

 

The beauty of the code below is that you can generate Web IDs for all objects using just a single method GenerateWebIdByPath method with 3 inputs:

  • Path of the PI System object
  • Class of the PI System object
  • Class of the owner of the PI System object (optional)

 

Here is a summary of what the code actually does:

  • Convert the object type to a Marker
  • Convert the owner object type to an Owner Marker
  • Validate if the Marker and Owner Marker are valid
  • Encode the Name Payload
  • Generate the Web ID 2.0

 

And finally the code...

 

Disclaimer:

This code could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 


Licensing

 

Copyright 2019 OSIsoft, LLC.

 

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

 

To put this another way, if you have problems with this method, do NOT call Tech Support, nor create a case in my.OSIsoft.com. The only outlet for discussing issues with this code is here within the PI Developers Club forum.


Sample Python code

 

import base64



class WebIdHelper(object):
def __init__(self):
self.marker_owner = None
pass




def generate_web_id_by_path(self, path,  objct_type, owner_type=None):
self.validate_type_and_owner_type(objct_type, owner_type)
marker = self.get_marker(objct_type)
owner_marker = self.get_owner_marker(owner_type)
if path[0:2] == "\\\\":
path = path[2:]
encoded_path = self.encode_string(path.upper())
return "P1{}{}{}".format(marker, owner_marker, encoded_path)


def validate_type_and_owner_type(self, object_type, owner_type):
if isinstance(PIAttribute(), object_type):
if isinstance(PIElement(), owner_type) and isinstance(PIEventFrame(), owner_type):
raise WebIdException("PIAttribute owner type must be a PIElement or a PIEventFrame.")
elif isinstance(PIAttributeTemplate(), object_type):
if isinstance(PIElementTemplate(), owner_type):
raise WebIdException("PIElementTemplate owner type must be a PIElementTemplate.")
elif isinstance(PIEnumerationSet(), object_type) or isinstance(PIEnumerationValue(), object_type):
if isinstance(PIDataServer(), owner_type) == False and isinstance(PIAssetServer(), owner_type) == False:
raise  WebIdException("PIEnumerationSet and  PIEnumerationValue owner type must be a PIDataServer or PIAssetServer.")
elif isinstance(PITimeRule(), object_type):
if isinstance(PIAnalysis(), owner_type) and isinstance(PIAnalysisTemplate(), owner_type):
raise WebIdException("PITimeRule owner type must be a PIAnalysis and PIAnalysisTemplate.")


def get_owner_marker(self, owner_type):
if owner_type == None:
return ""
if isinstance(PIAssetServer(),owner_type):
self.marker_owner = "R"
elif isinstance(PIDataServer(), owner_type):
self.marker_owner = "D"
elif isinstance(PIAnalysis(), owner_type):
self.marker_owner = "X"
elif isinstance(PIAnalysisTemplate(), owner_type):
self.marker_owner = "T"
elif isinstance(PIElement(), owner_type):
self.marker_owner = "E"
if isinstance(PIElementTemplate(), owner_type):
self.marker_owner = "E"
elif isinstance(PIEventFrame(), owner_type):
self.marker_owner = "F"
return self.marker_owner


def get_marker(self, object_type):
marker = None


if isinstance(PIAnalysis(), object_type):
marker = "Xs"
elif isinstance(PIAnalysisCategory(), object_type):
marker = "XC"
elif isinstance(PIAnalysisTemplate(), object_type):
marker = "XT"
elif isinstance(PIAnalysisRule(), object_type):
marker = "XR"
elif isinstance(PIAnalysisRulePlugIn(), object_type):
marker = "XP"
elif isinstance(PIAttribute(), object_type):
marker = "Ab"
elif isinstance(PIAttributeCategory(), object_type):
marker = "AC"
elif isinstance(PIAttributeTemplate(), object_type):
marker = "AT"
elif isinstance(PIAssetDatabase(), object_type):
marker = "RD"
elif isinstance(PIAssetServer(), object_type):
marker = "RS"
elif isinstance(PIElement(), object_type):
marker = "Em"
elif isinstance(PIElementCategory(), object_type):
marker = "EC"
elif isinstance(PIElementTemplate(), object_type):
marker = "ET"
elif isinstance(PIEnumerationSet(), object_type):
marker = "MS"
elif isinstance(PIEnumerationValue(), object_type):
marker = "MV"
elif isinstance(PIEventFrame(), object_type):
marker = "Fm"
elif isinstance(PITimeRule(), object_type):
marker = "TR"
elif isinstance(PITimeRulePlugIn(), object_type):
marker = "TP"
elif isinstance(PISecurityIdentity(), object_type):
marker = "SI"
elif isinstance(PISecurityMapping(), object_type):
marker = "SM"
elif isinstance(PITable(), object_type):
marker = "Bl"
elif isinstance(PITableCategory(), object_type):
marker = "BC"
elif isinstance(PIPoint(), object_type):
marker = "DP"
elif isinstance(PIDataServer(), object_type):
marker = "DS"
elif isinstance(PIUnit(), object_type):
marker = "Ut"
elif isinstance(PIUnitClass(), object_type):
marker = "UC"
if (marker == None):
raise WebIdException("Invalid object type.")
return marker


def encode_string(self, value):
bytes = value.upper().encode('utf-8')
return self.encode(bytes)


def encode(self, value):
encoded = base64.b64encode(value).decode()
return encoded.strip('=').replace('+', '-').replace('/', '_')

 

Sample Java code

 

 

import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.UUID;

public class WebIdHelper
{
    public WebIdInfo getWebIdInfo(String webId) throws WebIdException {
        return new WebIdInfo(webId);
    }

    public String generateWebIdByPath(String path, Class type, Class ownerType) throws WebIdException {
        validateTypeAndOwnerType(type, ownerType);
        String marker = getMarker(type);
        String ownerMarker = getOwnerMarker(ownerType);

        if (path.substring(0,2).equals("\\\\"))
        {
            path = path.substring(2, path.length());
        }
        String encodedPath = encode(path.toUpperCase());
        return ("P1" + marker + ownerMarker + encodedPath);
    }

    private void validateTypeAndOwnerType(Class type, Class ownerType) throws WebIdException {
        if (type == PIAttribute.class)
        {
            if ((ownerType != PIElement.class) && (ownerType != PIEventFrame.class))
            {
                throw new WebIdException("PIAttribte owner type must be a PIElement or a PIEventFrame.");
            }
        }
        else if (type == PIAttributeTemplate.class)
        {
            if ((ownerType != PIElementTemplate.class))
            {
                throw new WebIdException("PIElementTemplate owner type must be a PIElementTemplate.");
            }
        }
        else if ((type == PIEnumerationSet.class) || (type == PIEnumerationValue.class))
        {
            if ((ownerType != PIDataServer.class) && (ownerType != PIAssetServer.class))
            {
                throw new WebIdException("PIEnumerationSet and  PIEnumerationValue owner type must be a PIDataServer or PIAssetServer.");
            }
        }
        else if (type == PITimeRule.class)
        {
            if ((ownerType != PIAnalysis.class) && (ownerType != PIAnalysisTemplate.class))
            {
                throw new WebIdException("PITimeRule owner type must be a PIAnalysis and PIAnalysisTemplate.");
            }
        }
    }

    private String getOwnerMarker(Class ownerType)
    {
        String markerOwner = "";
        if (ownerType == null)
        {
            return markerOwner;
        }

        if (ownerType == PIAssetServer.class)
        {
            markerOwner = "R";
        }
        else if (ownerType == PIDataServer.class)
        {
            markerOwner = "D";
        }
        else if (ownerType == PIAnalysis.class)
        {
            markerOwner = "X";
        }
        else if (ownerType == PIAnalysisTemplate.class)
        {
            markerOwner = "T";
        }
        else if (ownerType == PIElement.class)
        {
            markerOwner = "E";
        }
        if (ownerType == PIElementTemplate.class)
        {
            markerOwner = "E";
        }
        else if (ownerType == PIEventFrame.class)
        {
            markerOwner = "F";
        }
        return markerOwner;
    }

    private String getMarker(Class type) throws WebIdException {
        String marker = null;
        if (type == PIAnalysis.class)
        {
            marker = "Xs";
        }
        else if (type == PIAnalysisCategory.class)
        {
            marker = "XC";
        }
        else if (type == PIAnalysisTemplate.class)
        {
            marker = "XT";
        }
        else if (type == PIAnalysisRule.class)
        {
            marker = "XR";
        }
        else if (type == PIAnalysisRulePlugIn.class)
        {
            marker = "XP";
        }
        else if (type == PIAttribute.class)
        {
            marker = "Ab";
        }
        else if (type == PIAttributeCategory.class)
        {
            marker = "AC";
        }
        else if (type == PIAttributeTemplate.class)
        {
            marker = "AT";
        }
        else if (type == PIAssetDatabase.class)
        {
            marker = "RD";
        }
        else if (type == PIAssetServer.class)
        {
            marker = "RS";
        }
        else if (type == PIElement.class)
        {
            marker = "Em";
        }
        else if (type == PIElementCategory.class)
        {
            marker = "EC";
        }
        else if (type == PIElementTemplate.class)
        {
            marker = "ET";
        }
        else if (type == PIEnumerationSet.class)
        {
            marker = "MS";
        }
        else if (type == PIEnumerationValue.class)
        {
            marker = "MV";
        }
        else if (type == PIEventFrame.class)
        {
            marker = "Fm";
        }
        else if (type == PITimeRule.class)
        {
            marker = "TR";
        }
        else if (type == PITimeRulePlugIn.class)
        {
            marker = "TP";
        }
        else if (type == PISecurityIdentity.class)
        {
            marker = "SI";
        }
        else if (type == PISecurityMapping.class)
        {
            marker = "SM";
        }
        else if (type == PITable.class)
        {
            marker = "Bl";
        }
        else if (type == PITableCategory.class)
        {
            marker = "BC";
        }
        else if (type == PIPoint.class)
        {
            marker = "DP";
        }
        else if (type == PIDataServer.class)
        {
            marker = "DS";
        }
        else if (type == PIUnit.class)
        {
            marker = "Ut";
        }
        else if (type == PIUnitClass.class)
        {
            marker = "UC";
        }
        if (marker == null)
        {
            throw new WebIdException("Invalid object type.");
        }

        return marker;
    }

    public static String encode(String value)
    {
        byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
        return encode(bytes);
    }

    public static String encode(byte[] bytes)
    {
        String value =  Base64.getEncoder().encodeToString(bytes);
        value = trimString(value, '=');
        return value.replace('+', '-').replace('/', '_');
    }

    public static String encode(UUID value)
    {
        byte[] bytes = value.toString().getBytes();
        return encode(bytes);
    }

    public static String trimString(String text, char trimBy) {
        int beginIndex = 0;
        int length = text.length();
        char[] textChar = text.toCharArray();

        while ((beginIndex < length) && (textChar[beginIndex] == trimBy)) {
            beginIndex++;
        }

        while ((beginIndex < length) && (textChar[length-1] == trimBy)) {
            length--;
        }

        if ((beginIndex > 0) || (length < text.length()))
        {
            return text.substring(beginIndex, length);
        }
        else
        {
            return  text;
        }
    }
}

 

Final Remarks and Conclusion

 

We hope that your Java and Python application will have a better performance by taking advantage of the Web ID 2.0 client-side generation!

 

Remember that PI Web API versions prior to 2017 R2 won't understand the Web IDs generated using this blog post!

Introduction

Normally, when we search for PI Points in PI Web API, we come equipped with its path. In this case, we can simply use the GetByPath action of the Point Controller to achieve this.

Point Controller GetByPath

 

But what if one day, your custom web application has a requirement where a list of Point IDs is given to you and there is a need for you to find the name of the PI Point corresponding to these IDs? If you are developing your application using the .NET Framework, then you have the option to use the PIPoint.FindPIPoint method available in AF SDK.

PIPoint.FindPIPoint

But since you are developing a web application, you do not have that luxury.

 

Today, in this blog post, I will be addressing how to search for a PI Point using its Point ID via PI Web API and WebID 2.0. An example of such a request can be found here.

https://pisquare.osisoft.com/thread/39989-find-pi-point-by-id

 

Concepts

If you don't know already, WebID version 2.0, introduced in PI Web API 2017 R2, provides different types of WebIDs (see: WebID Type). Specifying the WebID type gives you options for reducing WebID sizes (for URL length limitations), for identifying ambiguous paths/names of AF Event Frames and AF Notifications, and for accommodating path and name changes.

 

I will be using the IDOnly type today to achieve my goal of searching via the Point ID. The language of choice will be JavaScript.

 

Let's take a look at the composition of the IDOnly type of a PI Point WebID.

 

I chose a sample that is available from our public PI Web API endpoint found here.

I1DPW6Wlk0_Utku9vWTvxg45oA0egAAA

 

Let's try to break it down.

 

NameValue
IWeb ID type indicator. “IDOnly” in this case
1Web ID version number
DPWeb ID marker for PI Point objects
W6Wlk0_Utku9vWTvxg45oAURL Safe Base64 encoded string of PI Data Archive Id
0egAAAURL Safe Base64 encoded string of PI Point Id

 

With this knowledge, we can see that we are able to use the Point ID directly by encoding it in the WebID and using it in the Point Controller Get action.

Point Controller Get

 

Function to generate IDOnly type of WebID

Disclaimer:

This code could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

Please see this post here if you would like to see this same block of code with nicer indents.

function NewIDOnlyWebID(dataType, guid, oid, ownerType, ownerguid) {

//get the marker for the datatype
var marker = getOwnerMarker(dataType, ownerType)

//encode the server id to a base64 string
var serverwebid = encodeguid(guid)

//encode the owner id if datatype has a owner
var typeswithowner = ["AFAnalysisRule", "AFAttribute", 'AFAttributeTemplate', "AFEnumerationValue", "AFTimeRule"];
if (typeswithowner.includes(dataType)) {
if (ownerguid) {
serverwebid += encodeguid(ownerguid)
}
else {
throw 'please provide a valid owner guid'
}
}

//return webid if datatype is a server
if (dataType == "PIServer" || dataType == "PISystem") {
return 'I1' + marker + (serverwebid).replace(/=/g, '').replace(/\//g, '_').replace(/\+/g, '-')
}

//return webid if datatype is a pi point
if (dataType == "PIPoint") {
if (!oid) throw 'provide a valid PI Point ID'
var arr = new Uint8Array(new Uint32Array([oid]).buffer);
var pointwebid = btoa(String.fromCharCode.apply(null, arr))
return 'I1' + marker + (serverwebid + pointwebid).replace(/=/g, '').replace(/\//g, '_').replace(/\+/g, '-')
}

//return webid for af objects
var afwebid = encodeguid(oid)
return 'I1' + marker + (serverwebid + afwebid).replace(/=/g, '').replace(/\//g, '_').replace(/\+/g, '-')
}

function encodeguid(guid) {
var s = guid.replace(/[^0-9a-f]/ig, '').toLowerCase();

//check for invalid guid
if (s.length != 32) throw 'invalid guid';

//arrange bytes as PI Web API uses Microsoft style GUID
s = s.slice(6, 8) + s.slice(4, 6) + s.slice(2, 4) + s.slice(0, 2) +
s.slice(10, 12) + s.slice(8, 10) +
s.slice(14, 16) + s.slice(12, 14) +
s.slice(16);

//base64 encode the byte array
var t = '';
for (var n = 0; n < s.length; n += 2) {
t += String.fromCharCode(parseInt(s.substr(n, 2), 16));
}
return btoa(t)
}

function getOwnerMarker(dataType, ownerType) {
var marker
switch (dataType) {
case 'AFAnalysis':
marker = 'XS'
break;
case 'AFAnalysisCategory':
marker = 'XC'
break;
case 'AFAnalysisTemplate':
marker = 'XT'
break;
case 'AFAnalysisRule':
marker = 'XR'
switch (ownerType) {
case 'AFAnalysis':
marker += "X"
break
case 'AFAnalysisTemplate':
marker += "T"
break
default:
throw "please provide owner type"
}
break;
case 'AFAnalysisRulePlugin':
marker = 'XP'
break;
case 'AFAttribute':
marker = 'Ab'
switch (ownerType) {
case 'AFElement':
marker += "E"
break
case 'AFEventFrame':
marker += "F"
break
case 'AFNotification':
marker += "N"
break
default:
throw "please provide owner type"
}
break;
case 'AFAttributeCategory':
marker = 'AC'
break;
case 'AFAttributeTemplate':
marker = 'ATE'
break;
case 'AFDatabase':
marker = 'RD'
break;
case 'AFElement':
marker = 'Em'
break;
case 'AFElementCategory':
marker = 'EC'
break;
case 'AFElementTemplate':
marker = 'ET'
break;
case 'AFEnumerationSet':
marker = 'MS'
switch (ownerType) {
case 'PISystem':
marker += "R"
break
case 'PIServer':
marker += "D"
break
default:
throw "please provide owner type"
}
break;
case 'AFEnumerationValue':
marker = 'MV'
switch (ownerType) {
case 'PISystem':
marker += "R"
break
case 'PIServer':
marker += "D"
break
default:
throw "please provide owner type"
}
break;
case 'AFEventFrame':
marker = 'Fm'
break;
case 'AFNotification':
marker = 'Nf'
break;
case 'AFNotificationTemplate':
marker = 'NT'
break;
case 'AFNotificationContactTemplate':
marker = 'NC'
break;
case 'AFTimeRule':
marker = 'TR'
switch (ownerType) {
case 'AFAnalysis':
marker += "X"
break
case 'AFAnalysisTemplate':
marker += "T"
break
default:
throw "please provide owner type"
}
break;
case 'AFTimeRulePlugin':
marker = 'TP'
break;
case 'AFSecurityIdentity':
marker = 'SI'
break;
case 'AFSecurityMapping':
marker = 'SM'
break;
case 'AFTable':
marker = 'Bl'
break;
case 'AFTableCategory':
marker = 'BC'
break;
case 'PIPoint':
marker = 'DP'
break;
case 'PIServer':
marker = 'DS'
break;
case 'PISystem':
marker = 'RS'
break;
case 'UOM':
marker = 'Ut'
break;
case 'UOMClass':
marker = 'UC'
break;
default:
throw "please provide a suitable datatype"

}
return marker
}

The code above provides a generalized function that can help you to generate a PI Point WebID. The function takes 5 parameters.

 

namevalue
dataTypethe type of object
guidthe guid of the server that the object belongs to
oidthe id of the object itself
ownerTypethe type of the owner object (only required for objects with owner types)
ownerguidthe guid of the owner object (only required for objects with owner types)

 

 

Example usage for PI Point objects, 

console.log(NewIDOnlyWebID('PIPoint', '93a5a55b-d44f-4bb6-bdbd-64efc60e39a0', 59601))

Result will be the PI Point WebID

I1DPW6Wlk0_Utku9vWTvxg45oA0egAAA

 

Example usage for AF Attribute objects belong to an AF Element,

console.log(NewIDOnlyWebID('AFAttribute', '0b101021-e3bc-433d-9f06-a6a2db5f0803', '4f46d670-487e-5aa1-38b9-cd626ea43bc6', 'AFElement', 'cd24b9af-68d5-11e8-80db-000d3a10c7ce'))

Result will be the AF Attribute WebID

I1AbEIRAQC7zjPUOfBqai218IAwr7kkzdVo6BGA2wANOhDHzgcNZGT35IoVo4uc1ibqQ7xg

 

 

How to get the object guids?

For PI Data Archive, we can make a call to the DataServer Controller and utilize its GetByPath action and only select the Id field. An example is shown below.

 

 

Similarly for AF Server, we can make a call to the AssetServer Controller and utilize its GetByPath action and only select the Id field. I will leave this as an exercise for you to try out.

 

For AF objects, we can find the guid at the lower right corner of PI System Explorer.

 

Solve the initial problem

At the start of this blog, our challenge was to find the name of the PI Point corresponding to a Point ID that was given. Now with the PI Point WebID that was generated, we can easily do so with the Get action of the Point Controller.

 

 

 

Conclusion

We can see that WebID 2.0 has enhanced the flexibility of PI Web API and it allows us to do things that were previously thought impossible. The possibilities are endless and only limited by your imagination. I hoped you have enjoyed reading this blog and learnt something useful from it.

 

Please drop any comments below!

 

 

 

See also

pi-web-api-web-id-20-specification-tables

using-web-id-20-to-optimize-your-applications

Upgrades to PI System 2018 SP2

 

A new version of PI Server was recently released; PI Server 2018 SP2. OSIsoft encourages our customers to upgrade to this version to take advantage of all the enhancements and quality improvements that are included with the 2018 SP2 version.

 

Our public PI Web API endpoint and the supporting services behind it at devdata.osisoft.com are also in need of an upgrade.  :-)

 

While OSIsoft does not guarantee an SLA with the DevData public endpoint, we'd like to give you fair warning before we undertake any maintenance activities.

 

 

To that end: devdata.osisoft.com will be unavailable beginning at 4PM Eastern time (New York) on Friday of this week (2019-04-26 20:00 UTC), with an expected max downtime period of three hours as other server-based upgrade and maintenance activities are performed.

 

 

If you experience any issues please send an email to TechnologyEnablementTeam@osisoft.com

Greetings to our PI DevClub community.  It's that time once again where we announce the 2019 class of PI DevClub All-Stars.  Not to spoil the surprise, but I could have dusted off the 2018 announcement, and only scratched off a couple of names.  Allow me to explain these particular awards a bit more.  You do not have to be a paid subscriber to PI Developers Club.  Anyone who is a subscriber is automatically eligible for the award.  For anyone is who is not a subscriber, we review many of their posts to be sure that a significant number of posts are code-related.  However, code is not limited to the PI Developer Technologies (AF SDK, PI Web API, SQL Family).  We include Asset Analytics and PowerShell as well.

 

What you will discover common in all the winners is not just a sharp technical knowledge of PI, but more so a willingness to help others.  Awards are given as a recognition of such spirit to help others.

 

PI Developers Club Community All-Stars

This is a repeat across the board as 2018.

 

  • Roger Palmen  checks in at the top of our list.  This should not be surprising to anyone who even casually follows the forums.  Since winning All-Star the previous year, Roger has also climbed to top of the All-Time Leaderboard.  Seriously, I have lost track of how many consecutive years Roger has won.  7 or 8 maybe.
  • John Messinger  wins for the 3rd year in a row.
  • Dan Fishman  wins for the 2nd year in a row.

 

It's interesting having a global community because when I wake up in the morning to read the forums, these All-Stars have most likely already replied to most new questions.  John is from Australia, Roger from the Netherlands, and Dan is US-based.

 

PI Developers Club Rising Star

Anyone who has not previously won an All-Star or Rising Star award is eligible for Rising Star as long as some portion of their posts have been code-related.  This years recipient is a very prolific poster, the majority of which are not within the PI DevClub forums.  Nonetheless since he posts with such a high volume, we were able to easily find many that were code-related.

 

 

 

Prizes

The Community All-Stars and Rising Star all win the same award:

 

  • Amazon Gift Card worth 400 USD.
  • 1 year free subscription to PI Developers Club.
  • 100% discount to one of the following PI World events: Gothenburg 2019 or any PI World 2020.

 

 

Employee All-Stars

While all OSIsoft employees are always eager to give a helping hand, we want to recognize those they go the extra mile.  Our 2019 Internal All-Stars are:

 

Sebastien Raposo  And Seb is already making a strong claim for 2020 with his excellent, must-read series on Asset Analytics.

Jinmo Yi  wins for the 2nd year in a row.

David Hearn  is a multiple recipient.  I joke that I know when the Asset team has completed their sprint because David's posting activity increases around that time.

 

The Internal All-Stars each win an Amazon Gift Card worth 200 USD.

 

 

Never too late to think of next year

It's not impossible to break into the top 3 and shake the grip that Roger, John, and Dan have on the All-Star awards.  I have seen Ernst Amort (prefers being called Holger) and Jim Gavigan each with good activity in their own blogs.  Will Rhys Kirk, Asle Frantzen, or Lonnie Bowling find time in their jobs to post more and find themselves returning to the winners circle?  Time will tell.  See you next year when we announce the 2020 winners.

Last week during PI World, Rong Xu and I gave a talk about using PI Web API with PowerApps.

 

PowerApps is a tool offered by Microsoft which features a drag and drop app building experience. Adding logic to access PI Web API and controlling other UI components is similar to calling functions in Microsoft Excel. This reduces the technical knowledge in order to create custom solutions using your PI System data.

 

Click image below to ENLARGE

EditorScreen.PNG

 

As promised, I've attached the PowerApps app, Postman Collection and a copy of the AF database that was used in this talk. Be aware that exporting and importing apps are still a 'preview' feature in PowerApps so you may run into issues using this export directly. Instead, I recommend following along with the recording to build up the app.

 

The Postman Collection contains the requests that were used in this talk. Take note that the Web IDs used in this collection will not work in your environment because they contain information specific to the AF Server that was targeted in the talk.

Filter Blog

By date: By tag: