Skip navigation
All Places > PI Developers Club > Blog > 2019 > June
2019

Introduction

Greetings everyone! 

 

In today's blog post, we will be touching on the topic of dynamic template views in PI Vision Custom Symbols. The idea is to support multiple HTML template files for your custom symbol which could correspond to the sites that are available in your industrial plant so that users are able to choose the site that they want.

 

I answered this briefly in this thread. 

https://pisquare.osisoft.com/thread/39998

 

Now, I will explain this in more detail here. I will extend the Simple Value symbol from the GitHub repository available here to allow it to use multiple HTML files.

 

Disclaimer:

Please take note that the licensing for the GitHub repository referenced above extends to any of the code that is present in this blog.

 

Presentation Layer

For the original HTML template file, replace the original contents with the following. This will be the master template which redirects to the other site templates. Note that the ng-include directive is used here. It helps us to fetch, compile and include an external HTML fragment which in this case is the other site template files.

 

sym-simplevalue-template.html

<div ng-include= config.View.path></div>

 

Start making more HTML files with the content and file names below. Notice that the additional template files will have the scope carried over so you can still display the data updates. This makes it possible to do things such as {{value}}.

 

sym-simplevalue-template1.html

<div ng-style="{background: config.BackgroundColor, color: MultistateColor || config.TextColor}">
<div ng-show="config.ShowLabel">{{label}}</div>
<div>{{value}}</div>
    <div ng-show="config.ShowTime">{{time}}</div>
    This template file is for Site 1.
</div>

 

sym-simplevalue-template2.html

<div ng-style="{background: config.BackgroundColor, color: MultistateColor || config.TextColor}">
This template file is for Site 2.
<div ng-show="config.ShowLabel">{{label}}</div>
<div>{{value}}</div>
    <div ng-show="config.ShowTime">{{time}}</div>
</div>

 

Implementation Layer

In our implementation, we have to create a new property that holds the path to the default template file to use upon symbol creation. We can create this new property in the getDefaultConfig function. We will call it 'View'.

 

sym-simplevalue.js

getDefaultConfig: function() {
     return {
       DataShape: 'Value',
       Height: 150,
       Width: 150,
       BackgroundColor: 'rgb(255,0,0)',
       TextColor: 'rgb(0,255,0)',
       ShowLabel: true,
       ShowTime: false,
       View:{path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template1.html"}
    };
},

 

Next, in the init function of the symbol, we can set the possible paths that the user will be able to choose. Take note that these paths should be within the PI Vision installation directory so that you adhere to the same-origin policy. My advice is to place any custom files in the ext directory to facilitate upgrades. You won't have to search high and low for your custom files.

 

sym-simplevalue.js

scope.config.paths = [
    {name:"Site 1", path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template1.html"},
    {name:"Site 2", path:"Scripts/app/editor/symbols/ext/sym-simplevalue-template2.html"}
];

 

Configuration Layer

Now we need a way for the user to be able to select the template that he wants during run-time. And where better but to put it inside the configuration pane? We will create an additional drop down list in our pane for this purpose.

 

sym-simplevalue-config.html

<div class="c-config-content">Change View:
<select ng-model="config.View" ng-options="x.name for x in config.paths track by x.name">
    </select>
</div>

 

This list will populate with the possible paths that we set in the implementation layer during symbol initialization.

 

Demo

Let's check out a demo of how this will look like in action!

 

 

Conclusion

We have seen that with the Custom Symbols Extensibility for PI Vision, it becomes a very flexible product where you can do almost anything with some ideas and coding skills. This is important because it is impossible for OSIsoft to imagine every possible visualization style that users want. We can make symbols out of the most common use cases but we can't cater to every single one. The Extensibility framework helps to plug that gap so that many use cases can be addressed with PI Vision. This truly makes PI Vision a first class visualization platform!

I have been talking about AFSearch 2.10.5 for well over a year now. I spoke about it at PI World SF in April 2018 and again at Barcelona in Sept 2018. However, both of those talks used a beta build. The great news is that for the week of PI World SF in April 2019, this is no longer beta and is now production! Sadly, when working with beta code one should accept that the final production version may be a bit different, and that's certainly the case here. As I dusted off my code from Sept 2018, I could see it had several compiler errors since there was some fairly significant changes on the journey to production. Such is the life of a beta tester.

 

In case you haven't been following AFSearch or PI Server 2018 SP2 at all in the past year, the big changes for AFSearch with AF SDK 2.10.5 will be concisely summarized as:

 

  • Overlapped event frames searches got a performance boost.
  • Search queries now support OR conditions.
  • A new generic AFSearch<T> base class was added to implement many of the common properties and methods in derived search classes. As a result of this change, several methods have been marked as obsolete.

 

Don't let the short list fool you. While the first item requires no changes upon your part, the other items do require code changes. Supporting OR conditions is easy to say, but it did cause some breaking changes. In particular the old AFSearchToken structure is insufficient to represent new filters with OR. To that end, the AFSearchToken structure is being marked as Obsolete and replaced with a new AFSearchTokenBase class. Likewise, a generic AFSearch<T> base class introduces an IAFSearch<T>.FindObjects method, which means the many of the older Find methods (e.g. FindElements or FindEventFrames) are also marked as Obsolete. This is all mentioned in the help file under the heading What's new in PI AF 2018 SP2, but this blog expands upon short summary to give examples.

 

Event Frame Search Performance Boost

 

Let's start with the simplest change. If you are using captured event frames, then some searches should be noticeably faster, particular searches for Overlapped event frames.  For those of you rusty on the subject, Overlapped event frames are also considered "Active Between", meaning that any portion of the event frame falls within a requested time range.  I refer you to the image found in this link if you want more information.  Side note: when I was a customer writing AF applications, Overlapped was our most frequently used mode, and I know this is true for many of you.

 

An anecdotal example of this performance boost can be found in the PI World talks in 2018. In April, it took me about 1 minute to find 320K event frames from an AF Server of 6 million event frames. In September, I was using the same hardware but updated the AF SDK build every couple of weeks. About 3 weeks before Barcelona, the 1 minute search suddenly dropped to around 15 seconds. I asked the dev team and they confirmed they had implemented new logic for event frame searches.  I then gratefully said "thank you" and wiped the tears from my eyes.

 

This requires no change on your part other than the acceptance that your code may run twice as fast. What a nice problem to have!

 

OR - small word, BIG Changes

 

To implement support for OR, it quickly became apparent that the old AFSearchToken was not going to get the job done. There was a simplicity with the original AFSearchToken, but this breaks with the complexity of OR searches. There was just no way to represent it with the old structure. AFSearchToken is marked as obsolete, and any Tokens properties that depended upon AFSearchToken are also marked as obsolete. If your search does not use OR, then it will will get parsed by the Tokens property just fine (or with a little compiler warnings). But if you do use OR in your search, then accessing the Tokens property will produce an exception. Instead, you should begin using the TokenCollection property.

 

The TokenCollection property returns an AFSearchTokenCollection object, which implements IList<AFSearchTokenBase>. If you dig deeper into the help, you may discover that AFSearchTokenBase is an abstract class with 4 derived classes:

 

  • AFSearchExpressionToken, representing a logical grouping of search tokens. These tokens would be grouped together based on an OR or the default AND. I jokingly add that this is also akin to adding virtual parenthesis around your expression. All joking aside, if a query string contains parentheses, then the tokenized equivalent will likely need to use AFSearchExpressionToken.
  • AFSearchFilterToken, representing one criteria of a search class. Most of the obsolete AFSearchToken searches would use the new AFSearchFilterToken.
  • AFSearchQueryToken, representing a nested search query. Some obsolete AFSearchToken searches did use nested queries but should now use this AFSearchQueryToken.
  • AFSearchValueToken, representing an attribute value query.

 

Let's walk through an example to see these new tokens in action. I have chosen an example that uses all 4 of the new tokens. Let's first state how we plan to filter using simple phrases and business terms.

 

  • I want to perform an AFAttributeSearch, which means I will need a nested element query as well.
  • I want to search attributes on elements that have been derived from the "Storage Tank" element template.
  • I want to refine the element search to limit tank names (element names) to those starting with the letter "T".
  • I want a value filter on the attribute named "|Fill Percent". That filter should be on any tank level <= 10% or >= 90%. That is I want to filter on tanks that are too low or too high.
  • I want to return attributes named "Fill Percent", which is the tank level per tank (element).

 

Let's see how that would look as a query string. I personally like query strings, and many of the devs on the Asset Team prefer them as well.

 

Let's cheat a little and start out with what the fully resolved query string would look like:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

Note that anything between the red { } braces is the nested element query. Within the confines of those red braces, the context for Name (as in Name:T*) refers to the nested element query, so Name in this context means element name. Outside the red braces, you will see another Name (as in Name:'Fill Percent'), whose context refers to the attribute search, and therefore is an attribute name. (This example was specifically chosen to throw a curveball at you, forcing you to reconcile which Name is which).

 

You will note that any names containing embedded blanks are wrapped in quotes. You are free to create a longer and uglier string using escaped double quotes, but my eyeballs are much happier with the single quotes.

 

Okay, now let's see how that would be written with the new AFSearchTokenBase tokens. There are a few different ways, none of which make your code shorter. First, let's define some variables and constants to hold many of the names and limit values.

 

const string templateName = "Storage Tank";
const string elemName = "T*";
const string attrName = "Fill Percent";  // Note the pipe or bar symbol "|" is omitted
const double lowLimit = 10;
const double highLimit = 90;

 

Let me point out that attrName is used when a name is required. However, many of the value filters requires an attribute path. Later below you will see me preface attrName with a "|" to transform it into a path. Wherever this is done, you will also see me wrap the final text in single quotes to account for embedded blanks in the name.

 

This will be done in several pieces of code. I try to work with the innermost filter first, so that would be the OR clause wrapped in parenthesis.

 

AFSearchExpressionToken orExprToken = new AFSearchExpressionToken(AFSearchLogicalOperator.Or, new List<AFSearchTokenBase> {
    new AFSearchValueToken($"|{attrName}", lowLimit.ToString(), AFSearchOperator.LessThanOrEqual, AFSearchValueType.Numeric),
    new AFSearchValueToken($"|{attrName}", highLimit.ToString(), AFSearchOperator.GreaterThanOrEqual, AFSearchValueType.Numeric) });

 

That is one really long assignment. If you like breaking it down into more lines, then you may use this instead:

 

// First let's compose an OR condition for our limit extremes.
IList<AFSearchTokenBase> orChildTokens = new List<AFSearchTokenBase>();
orChildTokens.Add(new AFSearchValueToken($"|{attrName}", lowLimit.ToString(), AFSearchOperator.LessThanOrEqual, AFSearchValueType.Numeric));
orChildTokens.Add(new AFSearchValueToken($"|{attrName}", highLimit.ToString(), AFSearchOperator.GreaterThanOrEqual, AFSearchValueType.Numeric));

// Next the OR must be grouped as a single expression, which is akin to enclosing it in parenthesis.
AFSearchExpressionToken orExprToken = new AFSearchExpressionToken(AFSearchLogicalOperator.Or, orChildTokens);

 

This demonstrates 2 different ways to populate the expression token. The first way created the tokens argument inside the AFSearchExpressionToken constructor. The second way created the expression tokens before the AFSearchExpressionToken was constructed. There are 2 other noteworthy items along this topic. First, and this is a big change, is that you cannot modify the input tokens once the instance has been constructed. Second, is that you may expand this thinking to include any search-related constructor expecting an argument of IList<AFSearchTokenBase> such as constructors for AFSearchQueryToken, AFElementSearch, AFAttributeSearch, etc.

 

Either of the above code snippets fill in the first major piece towards a fully tokenized call. The second major piece will be to complete the nested element query token. Note we use the orExprToken object we defined above.

 

// Gather tokens for the nested element query, which will include the OR expression group.
IList<AFSearchTokenBase> nestedQueryTokens = new List<AFSearchTokenBase>();
nestedQueryTokens.Add(new AFSearchFilterToken(AFSearchFilter.Template, templateName));
nestedQueryTokens.Add(new AFSearchFilterToken(AFSearchFilter.Name, elemName));
nestedQueryTokens.Add(orExprToken);

 

The final piece will be to complete the attribute search query by combining the nested element query with other attribute search filters.

 

// Finally gather tokens for the AFAttributeSearch, which includes the nested element query
// and the attribute name filter.
List<AFSearchTokenBase> tokens = new List<AFSearchTokenBase>();
tokens.Add(nestedQueryTokens);
tokens.Add(new AFSearchFilterToken(AFSearchFilter.Name, attrName));

 

Alternatively, you could have also performed the last assignment with:

 

List<AFSearchTokenBase> tokens = new List<AFSearchTokenBase>() { nestedQueryTokens, new AFSearchFilterToken(AFSearchFilter.Name, attrName) };

 

That wraps it up for the new tokens. Again, the complexity of using the new tokens was a necessary evil to accommodate the robust requirements of OR clauses.

 

If you straddle the border on whether to use tokens or strings, I would encourage you to learn more about query strings. Meanwhile, let's return that original query string. As I said, this has been fully resolved:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

But in many cases this won't be fully resolved as many of your parameters are most likely stored in variables. What if you wanted to write the string to have variable substitution? Using String Interpolation, it would look like:

 

string query = $"Element:{{Template:'{templateName}' Name:'{elemName}' ('|{attrName}':<={lowLimit} OR '|{attrName}':>={highLimit})}} Name:'{attrName}'";

 

You will note that creating query is where we insert the pipe symbol. If you are more familiar with String.Format, you could use:

 

string query = String.Format("Element:{{Template:'{0}' Name:'{1}' ('|{2}':<={3} OR '|{2}':>={4})}} Name:'{2}'", templateName, elemName, attrName, lowLimit, highLimit);

 

I personally favor the Interpolated String because I know which variable is being used in-place, rather than we sequential numbers that require more eye movement.

 

New Methods and Constructors

 

It should be no surprise that the introduction of new TokenCollection and associated AFSearchTokenBase classes require new constructors and methods. This touches on virtually every related search class. Nothing much to add other than to make you aware of this. There is no need to enumerate them here since it's already been covered in Live Library's What's New link.

 

The Obsolete Find Methods

 

Under the hood, there is a significant change to the AFSearch abstract class. Prior to AF SDK 2.10.5, the class signature was:

 

public abstract class AFSearch : IDisposable

 

Effective with 2.10.5, the signature is now:

 

public abstract class AFSearch<T> : AFSearch, IAFSearch<T> where T : AFObject

 

This means the AFSearch class can now implement many of the common properties and methods in each of the derived search classes. One such common method is the FindObjects method, which is an implementation of the IAFSearch<T>.FindObjects interface.

 

Given how each derived search class has a FindObjects method, there is no longer a need to have each derived search class with its own type-specific Find method. Towards that end, the type-specific methods are marked as obsolete. This means you should start using FindObjects in place of:

 

  • AFAnalysisSearch.FindAnalyses
  • AFAnalysisTemplateSearch.FindAnalysisTemplates
  • AFAttributeSearch.FindAttributes
  • AFCaseSearch.FindCases
  • AFElementSearch.FindElements
  • AFEventFrameSearch.FindEventFrames
  • AFNotificationContactTemplateSearch.FindNotificationContactTemplates
  • AFNotificationRuleSearch.FindNotificationRules
  • AFNotificationRuleTemplateSearch.FindNotificationRuleTemplates
  • AFTransferSearch.FindTransfers

 

Again, don't be thrown off by the generic sounding FindObjects. Calling AFElementSearch.FindObjects still returns IEnumerable<AFElement>, so only the method name has changed.  FindObjects still returns type-specific items.

 

Team Query String versus Team Tokens

 

AF SDK is flexible in that you may choose to use either query strings or tokens, just as you may choose to use C#, VB.NET, or managed C++. As mentioned earlier, the Asset developers strongly prefer query strings.

 

I admit the new AFSearchTokenBase class is much more complicated to work with than the older AFSearchToken structure. I talked to several customer developers and many said they like tokens that spread out over many lines rather than a single-line query string because they can quickly comment out individual lines. With the new AFSearchTokenBase, you can still comment individual lines. However, there's nothing stopping you from having a multi-line query string or better yet, a StringBuilder object. With a StringBuilder, you would still be passing a string to the search constructor, but you would have the query span multiple lines where you may comment a given line.

 

Let's see an example of this in action. We return once again to the fully-resolved, single-line query string:

 

"Element:{Template:'Storage Tank' Name:T* ('|Fill Percent':<=10 OR '|Fill Percent':>=90)} Name:'Fill Percent'"

 

We can convert that a multiple line StringBuilder object. We really should have a separate StringBuilder object for the nested query and another for the full query.

 

StringBuilder nested = new StringBuilder();
nested.Append("Element:{");  // begin nested query
// Each inner filter will start with a blank
nested.Append($" Template:'{templateName}'");
nested.Append($" Name:'{elemName}'");
nested.Append($" (");  // begin OR expression
nested.Append($" '|{attrName}':<={lowLimit}");
nested.Append($" OR");
nested.Append($" '|{attrName}':>={highLimit}");
nested.Append(")");  // end OR expression
nested.Append("}");  // end nested query

StringBuilder query = new StringBuilder();
query.Append(nested.ToString());
query.Append($" Name:'{attrName}'");

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "", query.ToString()))
{
    // do something here
}

 

(A brief aside before we go any further, I want to say this to anyone who wants to have multi-line queries (string or tokens) in their code: I HOPE YOU ARE HAPPY. We have taken a single elegant line of a query string, and turned it into over a dozen lines of ugly code. Sorry. I needed to get that off my chest. Back to the lesson at hand ... )

 

Now let's consider what we would do if we want to temporarily change the search with these new rules:

 

  • We do not want to filter tank elements by name
  • We do not want to filter on low limit, which means we no longer need an OR clause. Another way to say this is we only need a value filter on high limit.

 

We would take the above and temporarily comment out lines accordingly:

 

StringBuilder nested = new StringBuilder();
nested.Append("Element:{");  // begin nested query
// Each inner filter will start with a blank
nested.Append($" Template:'{templateName}'");
//nested.Append($" Name:'{elemName}'");
//nested.Append($" (");  // begin OR expression
//nested.Append($" '|{attrName}':<={lowLimit}");
//nested.Append($" OR");
nested.Append($" '|{attrName}':>={highLimit}");
//nested.Append(")");  // end OR expression
nested.Append("}");  // end nested query

StringBuilder query = new StringBuilder();
query.Append(nested.ToString());
query.Append($" Name:'{attrName}'");

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "", query.ToString()))
{
    // do something here
}

 

There you go! We have demonstrated 3 examples for declaring a search query:

 

  • Using the new AFSearchTokenBase class
  • Using a single line query string
  • Using multi-lines of StringBuilder(s)

 

We leave it to your personal coding style and preference for what works best for you.

 

Caution using different Server and Client Versions

 

Let me issue this general caution regarding using different AF Server and AF Client version: while your app may compile correctly and run on mismatched versions, the features available would depend upon which system has the lowest version.  A client running 2.10.5 or later may be aware of the OR expression but a server running 2.10.0 or earlier would not.  Thus that app would be limited to 2.10.0 features.  If the client app tried to send an OR expression to the 2.10.0 server, an unsupported feature exception would be returned.

 

A very general rule of thumb: an older client app running against a newer server version should most likely run without issues, but a newer client app running against an older server version could throw unexpected errors.

 

 

Past AFSearch Blogs or Posts

 

Getting the most out of AFSearch - PI World Barcelona - Sept 2018. This used a beta of AF 2.10.5 to introduce OR support. Great video to help you understand the inner workings of AFSearch. This customer question hits on all the key points of the Barcelona Live Coding talk, and shows where lightweight searches can be great event frame searches with captured values. This Tech Talk was a refined repeat of May's Live Coding Talk at PI World 2018 San Francisco.

 

What's new in AFSearch 2.10 (PI AF 2018) (June 2018) - attribute searches are no longer are restricted to templates.

 

What's new in AFSearch 2.9.5 (PI AF 2017 R2) (March 2018) - AFAttributeSearch is introduced.

 

Coding Tips for using the AFSearch IN Operator - you may search for multiple values using IN(value1; value2; etc.). Some code is offered to make this easier to generate. This explains embedded blanks and using escaped double quotes or single quotes.

 

Using the AFEventFrameSearch Class (Nov 2016) - Giving some attention to event frames since many of earliest AFSearch examples were element based.

 

Aggregating Event Frame Data Part 1 of 9 - Introduction (May 2017) - From the UC 2017 TechCon/DevCon hands on lab for Advanced Developers.

 

Why you should use the new AF SDK Search (June 2016) - The granddaddy of them all. The earliest post explaining AFSearch, which was new at that time.

Introduction

 

Last week, just after publishing the blog post about how to send data from Node-RED to PI Web API, I got a call from a friend asking me how he could use a very similar setup to send raw binary data from the sensor to PI System and process it there. The answer is surprisingly simple and, after a quick look at several PI Square questions, I realised that this is a common question. So let's see how can we accomplish this in a very simple way, shall we?

 

Setup

 

The setup we will be using today is very similar to the one on my last post, so I strongly recommend you to give it a look before proceeding here. I will just go deeper into details about the sensor I'm using because we will need this information later on.

 

For this blog post, I will use my old and faithful BME280, a small temperature, humidity and pressure sensor that has a builtin calibration mechanism. Today we will only retrieve the temperature and calibration information, so we can keep the code simple and readable, but the procedure is pretty much the same for every sensor out there.

 

Because we will be dealing with binary data, it's important we start by giving a look at the sensor's datasheet so we understand how it's organized and where to get the information we need.

 

 

So here it is. Raw sensor data is available from 0xF7 to 0xFE (with temperatures on the first three bytes) and the calibration is a long sequence starting at 0x88 (if you read the datasheet you will see we only need the initial 6 bytes as the rest is for the other readings). The compensation formula is given in Appendix A and we will need to implement it on AF in order to process this data.

 

Sending the Data to PI

 

Once again, it's the same configuration from my last post, so I won't waste your time explaining all the details, but here's the Node-RED flow we will be using for this example:

This is pretty simple to follow. We start with an injection node that triggers the flow every five seconds. From there we go to an I2C node that reads data straight out of the I2C bus. Then we make some small adjustments to make it PI Web API friendly and we finally send it through a simple HTTP POST. The big difference here from the last time we did the same thing, is that now the I2C node creates a byte array as our payload:

 

 

So how do we POST an array to PI Web API? It's actually pretty simple. Considering that the PI Tag and AF Attribute are configured properly (we will see how to do that on the next secion), the JSON body should simply contain an array as the Value key. Then you POST to the Stream controller's UpdateValue method and you are good to go. Here's an example using Postman:

 

 

PI Tag and AF Attribute Configuration

 

There are two things we must consider in our configuration: the PI Tag type that will be used and the AF Attribute. Let's start with the AF Attribute configuration, where things are straightforward as the engine already exposes native array types. For this demo, we will use a byte array. Here's the config string of my attribute template, already set to tag creation: \\%Server%\%Element%.%Attribute%;ReadOnly=False;pointtype=Blob.

 

 

 

Because the sensor we are using sends data as a byte array, this will be the data type we will use. Keep in mind that it's not uncommon to see sensors sending data as Int arrays.

 

Now on the Data Archive side of our project, let's address a very common question: how do we store array data in the PI System? How should I configure a PI Tag to store array data? Here's a quote from LiveLibrary:

 

BLOB is the PI point type typically chosen for an arbitrary unstructured array of bytes. The value of a single event for a BLOB point is limited to binary data of up to 976 bytes in length. Data for larger binaries must be split into multiple events for the BLOB point or the data must be stored as an annotation.

 

So here's our answer: we must configure our PI Tag as a blob (by the way, blob means binary large object).

 

Processing the Data

 

At this point we already have data flowing in, an 8-byte array for our raw data a 6-byte array for the compensation factors:

 

  

 

Now we have to extract meaningful information out of it by implementing some transformations that will be able to convert the binary data into our final temperature value. In order to do this, we first have to check the sensor's datasheet to see how we convert the calibration factors into actual numbers that will go in the conversion formula. We will start with our calibration parameters and here's the info from the sensor documentation:

 

  

 

As I said before, we only need the first six bytes from the calibration information. So we now need to convert the bytes into actual numbers. Also, from this table, we now know T1 is an unsigned short and the other two are signed, so the transformation is simple. Here's how it's done in Python:

 

dig_T1 = cal[1] << 8 | cal[0]
dig_T2 = cal[3] << 8 | cal[2]
dig_T3 = cal[5] << 8 | cal[4]

if dig_T2 > 32767:
dig_T2 = dig_T2 - 65536

if dig_T3 > 32767:
dig_T3 = dig_T3 - 65536

 

In order to do that in AF, you have to use a little math because it doesn't offer the bitwise operators available in Python or C. The bitwise left shift (<< n) is equivalent of multiplying your number by 2^n while the binary OR ( | ) is a simple sum. Finally, we have to check if T2 and T3 are above 32767 because this is how signed ints work. This is how our final implementation is in AF (important: arrays on AF use one-based indexing! So to access the first to elements, we will use [1] and [2]):

 

 

Now we have to go back to the sensor's datasheet to see how can we convert the raw data into an actual number. Here's the information we need: 

 

 

On my Node-RED flow, I'm requesting data from 0XF7 to 0XFE so I don't need to make several requests to the I2C bus. This is important because, on our array, the MSB will be on position [4], LSB on [5] and XLSB on [6]. The Python script that does the bitwise operation to convert it into a decimal number is quite simple:

 

temp_raw = (block[3] << 16 | block[4] << 8 | block[5]) >> 4

 

In a similar fashion as before, we convert it to an AF Analysis script by using simple math:

 

  

 

We are almost there! We have all our reading as numbers and we can finally apply the conversion formula available on the sensor's documentation. Here's the C code they've provided:

 

double BME280_compensate_T_double(double adc_T)
{
double var1, var2, T;
const double K1 = 1024;
const double K5 = K1 * 5; // 5120
const double K8 = K1 * 8; // 8192
const double K16 = K1 * 16; // 16384
const double K128 = K1 * 128; // 131072

var1 = ((adc_T / K16) - (dig_T1 / K1)) * dig_T2;
var2 = ((adc_T / K128) - (dig_T1 / K8)) * ((adc_T / K128) - (dig_T1 / K8)) * dig_T3;
T = (var1 + var2) / K5;
return T;
}

 

This is easy peasy lemon squeezy for AF and I'm sure you will have no problem implementing this logic. Here's my complete analysis, where final is the output temperature in celsius:

 

  

 

Do I need this?

 

I reckon this is not for everyone. Most of the time we only need the final sensor reading. But some modern sensors and instruments are able to send more meaningful and important data, like maintenance flags, reading status and other parameters that may be useful for some teams, like instrumentation and maintenance.

Eugene Lee

Async Streaming with AF SDK

Posted by Eugene Lee Employee Jun 10, 2019

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

Introduction

Greetings everyone! 

 

In this blog post, we shall be discussing about a concept called Async Streaming and how we can use it with AF SDK to help make more responsive, scalable and concurrent applications.

 

Async Streaming is a new feature in C# that will be natively supported in version 8 and .NET Core 3. Even though AF SDK is not supported in .NET Core, there are still libraries available out there that can bring the benefits of Async Streaming to AF SDK.

 

Async Streaming can be advantageous in many cases. For example:

  1. In front-end applications, the main UI thread can stay responsive during a data access call.
  2. In both client and server applications, the number of threads used to service a call can be reduced, as waiting threads won't be blocked and can be returned to the thread pool for re-use.
  3. The effect of latency is mitigated because remote calls can be executed concurrently.
  4. Receiving data and processing it as it is retrieved in a way that doesn't block while we wait.

 

What is Asynchronous programming?

Asynchronous programming is a means of parallel programming in which a unit of work runs separately from the main application thread and notifies the calling thread of its completion, failure or progress.


This is the first thing you will find if you do a Google search using that term. In layman terms, this means that it will be able to help us achieve more responsive applications by not blocking the main thread.


AF SDK bulk calls

Let's examine the available data access bulk calls for PI Points that offer async behavior.

 

 

We notice that the methods with native async behavior tend to return one value per PI Point. If you look at their counterparts which return multiple values per PI Point, we find that they are not natively async.

 

 

I shall use the RecordedValues method as an example here. Below is a snippet of how we normally call this method.

 

We can make a wrapper called GetRecordedValues to return us a list of the recorded values. The return type is IEnumerable<AFValues>.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

private static IEnumerable<AFValues> GetRecordedValues(PIPointList pointList)
{
    PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 1);
    var timeRange = new AFTimeRange("*-10y", "*");

    try
    {
        var listResults = pointList.RecordedValues(timeRange, AFBoundaryType.Inside, null, true, config);
        return listResults;
    }
    catch (OperationCanceledException)
    {
        // Errors that occur during bulk calls get trapped here
        // The actual error is stored on the PIPagingConfiguration object
        Console.WriteLine(config.Error.Message);
        return null;
    }
    catch (Exception otherEx)
    {
        // Errors that occur in an iterative fallback method get trapped here
        Console.WriteLine(otherEx.Message);
        return null;
    }
}

 

And then we can consume the wrapper using a foreach loop.

 

var afvalslist = GetRecordedValues(pointList);
foreach (var pointResults in afvalslist)
{
    foreach (var item in pointResults)
    {
        Console.WriteLine("Timestamp: " + item.Timestamp + "\tValue: " + item.Value + "\tName: " + pointResults.PIPoint);
    }
    Console.WriteLine();
}

 

Now, this sample is generally fine in most cases. The only bad thing about it is that it doesn't have any async behavior. If the PI Data Archive is busy serving other users or applications, threads may get blocked such that responsiveness and performance will suffer. What can we do to improve upon our code?

 

This is where Async Streaming can save the day!

 

Async Streaming

Async Streaming makes it possible to await for a stream of results. As I mentioned in the introduction above, there are libraries out there to integrate AF SDK with Async Streaming. For this blog post, I am going to use one of them called AsyncEnumerator found here

 

https://www.nuget.org/packages/AsyncEnumerator/

 

The package can be easily installed from NuGet via

 

Install-Package AsyncEnumerator -Version 2.2.2


It introduces 2 new interfaces called IAsyncEnumerable and IAsyncEnumerator. Lets examine each of them to understand how it helps us to do Async Streaming.

 

public interface IAsyncEnumerator
{
    object Current { get; }
    Task<bool> MoveNextAsync(CancellationToken cancellationToken = default);
}

 

The Current property is the same as IEnumerator's version. It gets the element in the collection at the current position of the enumerator. What's different is the MoveNextAsync method. Over here, we can see that it returns a Task to us. Thus, we can start the task and continue on with our work while letting the task run in the background. MoveNextAsync does not block the thread compared to MoveNext of IEnumerator.

 

public interface IAsyncEnumerable
{
    Task<IAsyncEnumerator> GetAsyncEnumeratorAsync(CancellationToken cancellationToken = default);
}

 

GetAsyncEnumeratorAsync creates an enumerator that iterates through a collection asynchronously. This also returns a Task which returns an IAsyncEnumerator when it is complete.

 

General usage patterns

We can use a general construct such as the one below to consume an async stream of values. Take note that this construct is specific to the library being used. C# 8 has a very similar syntax. This pattern of iteration will not block the thread which is what we desire for our application.

 

await asyncEnumerable.ForEachAsync(async number => {
    await Console.Out.WriteLineAsync($"{number}");
});

 

Behind the scenes, the compiler will translate the ForEachAsync statement to utilize the MoveNextAsync method and then access the Current property to get the element of interest.

 

Cancellation

With this pattern, you can use a cancellation token to stop the streaming. This is useful for implementing timeouts or for the user to cancel the operation. If you look at the parameters of MoveNextAsync, you will notice that it accepts a cancellation token which you can use for notifying the streaming to stop. 

 

public virtual Task<bool> MoveNextAsync(CancellationToken cancellationToken = default)

public static async Task ForEachAsync(this IAsyncEnumerable enumerable, Action<object> action, CancellationToken cancellationToken = default)

 

The ForEachAsync extension method passes this token to MoveNextAsync where we can then retrieve this token with the yield.CancellationToken property to check for cancellation. An example is like the following.

 

token = yield.CancellationToken;
if (token.IsCancellationRequested)
{
    await Console.Out.WriteLineAsync("cancelling");
    yield.Break();
}

 

 

Async Streaming + AF SDK = GetStreamingRecordedValuesAsync

Now that we know what Async Streaming is about, let us improve upon the GetRecordedValues wrapper that was introduced in the previous section. We will leverage on the general usage patterns and also include cancellation in our wrapper.

 

We will call this wrapper GetStreamingRecordedValuesAsync. We will retrieve pages of results from the PI Data Archive one tag at a time as defined by the PIPagingConfiguration settings. The return type is IAsyncEnumerable<AFValue>.

 

Disclaimer:

Any of the code in this blog could contain bugs and shouldn’t be used in production without extensive testing.

You agree that if you use any of the provided code in your own production code that you accept all ownership, risks, liabilities, and responsibilities associated with the performance, support, and maintenance of the code.

 

private static IAsyncEnumerable<AFValue> GetStreamingRecordedValuesAsync(PIPointList pointList)
{
    PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 1);
    var timeRange = new AFTimeRange("*-10y", "*");

    return new AsyncEnumerable<AFValue>(async yield =>
    {
        try
        {
            await Task.Run(async () =>
            {
                var listResults = pointList.RecordedValues(timeRange, AFBoundaryType.Inside, null, true, config);
                CancellationToken token;
                foreach (var pointResults in listResults)
                {
                    token = yield.CancellationToken;
                    if (token.IsCancellationRequested)
                    {
                        await Console.Out.WriteLineAsync("cancelling");
                        yield.Break();
                    }

                    foreach (var result in pointResults)
                    {
                        await yield.ReturnAsync(result);
                    }

                }
            });
        }
        catch (OperationCanceledException)
        {
            // Errors that occur during bulk calls get trapped here
            // The actual error is stored on the PIPagingConfiguration object
            await Console.Out.WriteLineAsync(config.Error.Message);
            yield.Break();
        }
        catch (Exception otherEx)
        {
            // Errors that occur in an iterative fallback method get trapped here
            await Console.Out.WriteLineAsync(otherEx.Message);
            yield.Break();
        }
    });
}

 

With this sample, we will be streaming the recorded values for each PI Point on the list. We can utilize the wrapper using the ForEachAsync loop and pass to it a cancellation token.

 

var cts = new CancellationTokenSource();
var afvalslist = GetStreamingRecordedValuesAsync(pointList);
await afvalslist.ForEachAsync(async item =>
{
    await Console.Out.WriteLineAsync("Timestamp: " + item.Timestamp + "\tValue: " + item.Value.ToString().PadRight(20) + "Name: " + item.PIPoint);
}, cts.Token);

 

This method of streaming ensures the calling thread doesn't get blocked and can continue with other work. To refresh your memory, a PIPointList can contain points from multiple PI Data Archives. For a global enterprise, your PI Data Archives could be scattered around the world. What if your application is hosted in USA but you need data from the server in Singapore? No matter what, latency will definitely affect its performance. You can't beat the laws of physics but at least you are free to do other work while waiting. That's what productivity and concurrency is about!

 

Point of caution

With Async Streaming on the client side, we can conveniently fire and forget calls. However, one has to keep in mind that the server will still need to process the data request.  If every single application just dumps all these data calls asynchronously to the server, it will have some negative effects on the server. Therefore, it is up to the user to implement some kind of throttling.

 

Conclusion

In this blog post, we have looked at what Async Streaming is and how it can help you make responsive, scalable and concurrent applications. In AF SDK, some bulk data calls might not have async methods. However, we can still use async streaming to improve the performance of our application utilizing these methods. I found a feature request here to expose asynchronous interfaces for bulk calls of AF Attributes. You can vote for it if you are interested.

https://pisquare.osisoft.com/ideas/5743-af-sdk-async-data-methods-for-multiple-afattributes

 

I hope you have learnt something useful from this article. Let me know if you have any comments!

Stream Updates allows a PI Web API client to retrieve data updates from PI Points and AF Attributes without using Channels (which are based on websockets). With Stream Updates, you register for data streams of interest with an HTTP POST request to a new "streams/updates" endpoint. Data updates are then retrieved by polling with a marker returned to you by the registration call. We first introduced Stream Updates as a Community Technology Preview (CTP) in PI Web API 2018. We've added some new features in PI Web API 2018 SP1.

 

Changes in AF metadata

Previously, you would not be notified if there was a change in AF metadata. Now, you are notified if metadata – like AF Data Reference, units of measure, and description – has changed. The exact metadata change is not reported. Query the AF Attribute to see what changed; then, register again for Stream Updates to receive further updates.

 

This feature reacts to AF metadata changes only. Changes to PI Point attributes are not reported.

 

If an AF Attribute metadata change has occurred, the response payload will contain this:

"Exception": {
"Errors": [
"The signup was updated and any cached data could be invalid."
]
}

 

There is one type of edit to an AF Attribute that will be reported to your client in PI Web API 2018 SP1. If the value of a static Attribute is changed, the new value will be returned to your client as a data update.

 

Marker error always available

If a passed marker is valid, but the AF Attribute you are tracking experiences an error, you will receive a message in the Exception section of your response payload. For example, if an AF Attribute you are tracking is deleted, you will get this error in your response payload:

"Exception": {
"Errors": [
"The signup was removed because it is no longer valid."
]
}

 

Selected Fields

Most controllers in PI Web API support the optional selectedFields parameter that allows you to choose which fields in the standard payload are returned. This parameter has been added to Stream Updates registration and data updates. For example, to retrieve only the latest marker and the data updates, you would issue an HTTP GET request to:

https://myserver.com/streams/updates/{marker}/selectedFields=LatestMarker;Events

 

PreviousEventAction

When processing data updates, it is often useful to know what happened to the previous event received by the PI Data Archive. We have added the PreviousEventAction item to each Event in the response payload. Possible values are PreviousEventArchived and PreviousEventDeleted.

 

Performance

How fast can you expect to retrieve data updates this way? We set up a test environment with a 34 GB PI Data Archive server with 6 cores receiving data values into 250,000 PI Points at 1 second intervals each. Our 8 GB PI Web API server had 8 cores. Our test program registered for Stream Updates in blocks of 40 points each. We are able to sustain 40,000 data updates per second. We did need to edit some of the PI Update Manager tuning parameters and PI Web API configuration parameters to achieve this retrieval rate. For PI Update Manager, we set MaxUpdateQueue and TotalUpdateQueue to 2 million to overcome the default 50,000 queue size. We set the CacheInstanceUpdateInterval parameter in PI Web API to 1 (second) and the CacheInstanceUpdateHoldoffTime parameter to 0 (seconds) to more quickly query for new events and therefore empty the queue faster.

Knowing that PI Data Archive is capable of significantly higher throughput, we believe the current throughput is memory-bound in PI Web API; however, we also believe the throughput is sufficient for most uses of Stream Updates. If you have specific use cases where higher throughput is needed, please share them with us at https://feedback.osisoft.com.

Additionally, we do not recommend using the exact tuning parameters values as described above - totalUpdateQueue should be greater than MaxUpdateQueue. For additional information on these PI Data Archive tuning parameters, please refer to Knowledge Base article KB3151OSI8.

 

Sample C# code demonstrating the new features

This sample is an update from our original blog post:

PIWebAPIClient.cs

public class PIWebAPIClient 
    {
        private HttpClient client;
        private string baseUrl;
        public PIWebAPIClient(string url, string username, string password)
        {
            client = new HttpClient();
            if (!String.IsNullOrEmpty(username) && !String.IsNullOrEmpty(password))
            {
                string auth = Convert.ToBase64String(Encoding.ASCII.GetBytes(string.Format("{0}:{1}", username, password)));
                client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", auth);
            }
            client.DefaultRequestHeaders.Add("X-Requested-With", "asdf");  // avoids CSRF warning
            baseUrl = url;
        }
        public PIWebAPIClient(string url)
        {
            client = new HttpClient();
            baseUrl = url;
        }
        public async Task<object> GetAsync(string uri)
        {
            HttpResponseMessage response = await client.GetAsync(uri);
            var jsonString = await response.Content.ReadAsStringAsync();
            var json = JsonConvert.DeserializeObject<object>(jsonString);
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + jsonString);
            }
            return json;
        }
        public async Task<object> PostAsync(string uri)
        {
            HttpResponseMessage response = await client.PostAsync(uri, null);
            var jsonString = await response.Content.ReadAsStringAsync();
            var json = JsonConvert.DeserializeObject<object>(jsonString);
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + jsonString);
            }
            return json;
        }
        public async Task<dynamic> RegisterForStreamUpdates(string webId, string selectedFields)
        {
            string url = baseUrl + "/streams/" + webId + "/updates";
            if (!String.IsNullOrEmpty(selectedFields))
            {
                url += "?selectedFields=" + selectedFields;
            }
            dynamic response = await PostAsync(url);
            return response;
        }
        public async Task<dynamic> RetrieveStreamUpdates(string marker, string selectedFields)
        {
            string url = baseUrl + "/streams/" + "/updates/" + marker;
            if (!String.IsNullOrEmpty(selectedFields))
            {
                url += "?selectedFields=" + selectedFields            
            }
            dynamic response = await GetAsync(url);
            return response;
        }
        public string GetVersion()
        {
            string url = baseUrl + "/system";
            dynamic response = GetAsync(url).Result;
            return response.ProductVersion;
        }
    }

 

Program.cs

class Program 
    {
        static string baseUrl = "https://my.server.com/piwebapi";
        static string marker = null;
        static string username = "username";
        static string password = "password";
        static string webId = "myWebId";
        static PIWebAPIClient client = null;
        static void Main(string[] args)
        {
            client = new PIWebAPIClient(baseUrl, username, password);
            Console.WriteLine("PI Web API Version: {0}", client.GetVersion());
            // Register for Stream Updates requesting only LatestMarker and Status
            dynamic response = client.RegisterForStreamUpdates(webId, "LatestMarker;Status").Result;
            marker = response.LatestMarker;
            string stat = response.Status;
            string src = response.SourceName;
            //ReceiveUpdates is called every 10 seconds until you explicitly exit the application
            TimeSpan startTimeSpan = TimeSpan.Zero;
            TimeSpan periodTimeSpan = TimeSpan.FromSeconds(10);
            var timer = new System.Threading.Timer((e) =>
            {
                Console.WriteLine("{0},{1}", DateTime.Now.ToString(), marker);
                ReceiveUpdates(ref marker);
            }, null, startTimeSpan, periodTimeSpan);
            Console.ReadLine();
        }
        public static void ReceiveUpdates(ref string marker)
        {
            dynamic update = client.RetrieveStreamUpdates(marker, "Status;LatestMarker;Events;Exception.Errors").Result;
            //dynamic update = client.RetrieveStreamUpdates(marker, null).Result;
            Console.WriteLine(update);
            Console.WriteLine("Press the Enter key to exit anytime!");
            marker = update.LatestMarker;
            dynamic excp = update.Exception;
            if (excp != null)
            {
                throw new System.Exception(excp.Errors[0].ToString());
            }
        }
    }

Filter Blog

By date: By tag: