Skip navigation
All Places > PI Developers Club > Blog > 2016 > November
2016

 

Introduction

The legacy AFEventFrame.FindEventFrames methods were introduced beginning with AF SDK 2.4.  An enhanced search feature was introduced in AF SDK 2.8 with the AFEventFrameSearch class.  This blog will show examples using both the legacy syntax and the new class.  As the examples strive to be near equivalents between the legacy versus new search class, these examples may serve as a Rosetta Stone for developers who are familiar with one method and want to learn about the other.

 

This blog is part of the series Working with PI AF SDK 2016 - Introduction to the blog post series .

 

Legacy Methods

There are many different legacy AFEventFrame methods for finding event frames.  Briefly:

  • FindEventFrames
  • FindEventFramesByAnalysis
  • FindEventFramesByAttribute
  • FindEventFramesByExtendedProperty
  • FindEventFramesByPath
  • FindEventFramesByReferenceType

 

There is also one more set of legacy search methods that's not in the AFEventFrame class.  Rather this method is found in the AFElement class.

  • GetEventFrames

 

Existing code will still work with the legacy methods, so there is no need to change it for change's sake.  Going forward you are strongly encouraged to use the new

AFEventFrameSearch methods since the help for some of the legacy methods will display this advice:

 

Important note Important
Consider using the new AFEventFrameSearch class for finding event frames instead of using this method.

 

This blog will show examples for the GetEventFrames, FindEventFrames, and FindEventFramesByAttribute legacy methods, and the corresponding AFEventFrameSearch methods.

 

Basic Event Frame Searching

The first section of examples will perform basic searching based on a time range, referenced element, and event frame template.  Much further below, we will later expand to more advanced searching by modifying our search to filter on an attribute's value.

 

The following properties will be used in my example code.  Note that the properties are indeed correctly set when I execute the code.  Trust me.

 

public PISystem AssetServer { get; set; }
public AFDatabase Database { get; set; }
public AFElement Element { get; set; }
public AFElementTemplate Template { get; set; }
public AFTime StartTime { get; set; }
public AFTime EndTime { get; set; }

 

The above is included to help the reader follow along in code samples.  On my actual C# 6.0 test system, my code looks more like:

 

public PISystem AssetServer => Element?.PISystem;
public AFDatabase Database => Element?.Database;
public AFElement Element { get; }

 

But users that don't have C# 6.0 ask too many questions like "What does '=> Element?.PISystem' mean?"  Hence I've given simplified declarations so as not to get bogged down on side topics such as what is an expression-bodied auto-property or what is a null-conditional operator.

 

Legacy GetEventFrames Example

This example would find event frames that reference the specified Element based on the specified event frame Template.  Note that Element does not have to be the primary referenced element.  It only has to be in the event frame’s ReferencedElements collection.

 

AFNamedCollectionList<AFEventFrame> frames;
frames = Element.GetEventFrames(
            searchMode: AFSearchMode.Overlapped,
            startTime: StartTime,
            endTime: EndTime,
            nameFilter: null,
            elemCategory: null,
            elemTemplate: Template,
            sortField: AFSortField.StartTime,
            sortOrder: AFSortOrder.Ascending,
            startIndex: 0,
            maxCount: 1000);

 

Note if you desired paging, you would need to incorporate it yourself in a loop where you would adjust startIndex each iteration.  The newer methods have paging baked in.

 

Legacy FindEventFrames Example

The equivalent code using the legacy AFElement.FindEventFrames method would be:

 

AFNamedCollectionList<AFEventFrame> frames;
frames = AFEventFrame.FindEventFrames(
            database: Database,
            searchRoot: null,
            searchMode: AFSearchMode.Overlapped,
            startTime: StartTime,
            endTime: EndTime,
            nameFilter: null,
            referencedElementNameFilter: Element.Name,
            eventFrameCategory: null,
            eventFrameTemplate: Template,
            referencedElementTemplate: Element.Template,
            durationQuery: null,
            searchFullHierarchy: true,
            sortField: AFSortField.StartTime,
            sortOrder: AFSortOrder.Ascending,
            startIndex: 0,
            maxCount: 100000);

 

 

 

AFEventFrameSearch Without Server-side caching

The new search class will show an equivalent search, but make note that the returned type will be an IEnumerable<AFEventFrame> rather than an AFNamedCollectionList<AFEventFrame>.  Note if you must work with an entire list after making this call, you always have the ability to issue a ToList() on the returned frames.  Whether you have the RAM is another question.  If you were expecting a small enough collection, ToList is fine.  If you were expecting a large collection, you would want to enable server-side caching as well has specify paging.

 

string query = string.Format("Element:{0} Template:{1}", Element.GetPath(Database), Template.GetPath(Database));

var search = new AFEventFrameSearch(Database, "optional name", AFSearchMode.Overlapped, StartTime, EndTime, query);

IEnumerable<AFEventFrame> frames = search.FindEventFrames(startIndex: 0, fullLoad: false, pageST

 

See how easy it is to implement paging with the new methods!  Note there is always paging enabled for this method.  Specifying a pageSize of 0 does not turn off paging.  Rather it will default to the page size set to the current value of CollectionPageSize.

 

When fullLoad is false, you're only getting back basic header info of event frames.  This would be the inherent properties such as Name, Description, Template, etc, but not any extended properties nor any attribute info.  You could consider using the AFEventFrame.LoadEventFrames method but not only does this require a 2nd trip to server, it only accepts an IList<AFEventFrame> when what you have is IEnumerable<AFEventFrame>.  If you were planning on just displaying header properties for the event frames, fullLoad: false is acceptable.

 

But if you want to display any attribute info, you would want to use fullLoad: true.  The need is not limited to just referencing an attribute directly, as item["Tank Level"].  You want fullLoad: true if you used anything like item.Attributes.Count or item.DefaultAttribute.

 

 

AFEventFrameSearch With Server-side caching

A natural question would be "How do you know that the previous example does not have server-side caching?"  Because we didn't specify it, that's why!  To do that, all we do is add one line of code to the previous example:

 

string query = string.Format("Element:{0} Template:{1}", Element.GetPath(Database), Template.GetPath(Database));

var search = new AFEventFrameSearch(Database, "optional name", AFSearchMode.Overlapped, StartTime, EndTime, query);
search.CacheTimeout = TimeSpan.FromMinutes(10); // Opt in to server-side caching

IEnumerable<AFEventFrame> frames = search.FindEventFrames(startIndex: 0, fullLoad: false, pageSize: 1000);

 

See line 4.  Also see Considerations section towards the bottom for a discussion about server-side caching.

 

 

Advanced Searching on an Attribute Value

The previous examples demonstrated a basic search of event frames based on time range, event frame template, and referenced element.  With the basics down, we can turn our attention to more advanced searching based on an attribute's value.

 

Legacy FindEventFramesByAttributes Example

In the code below, I want to search the same set of event frames that I found earlier in the FindEventFrames example, only now I want to get a filtered subset of those event frames when the attribute "Tank Level" has a value greater than or equal to 75.  The trick here is to define an AFAttributeValueQuery to perform a requested comparison on a requested attribute template.

 

AFAttributeValueQuery[] valueQuery = new AFAttributeValueQuery[] 
                                        { new AFAttributeValueQuery(Template.AttributeTemplates["Tank Level"]
                                          , AFSearchOperator.GreaterThanOrEqual
                                          , 75.0)
                                        };

AFNamedCollectionList<AFEventFrame> frames;
frames = AFEventFrame.FindEventFramesByAttribute(
            searchRoot: null,
            searchMode: AFSearchMode.Overlapped,
            startTime: StartTime,
            endTime: EndTime,
            nameFilter: null,
            referencedElementNameFilter: Element.Name, 
            durationQuery: null,
            valueQuery: valueQuery,
            searchFullHierarchy: true,
            sortField: AFSortField.StartTime,
            sortOrder: AFSortOrder.Ascending,
            startIndex: 0,
            maxCount: 1000);

 

Tread carefully here.  It is impossible to perform paging correctly with this method if the event frames have attribute values from a data reference that are not captured.

 

 

AFEventFrameSearch By Attribute Example

I could show 2 very similar examples using AFEventFrameSearch: one with caching and one without.  But since only 1 simple line of code differs, I am going to only show the example with caching.  You will note that it looks very similar to our previous AFEventFrameSearch example with the notable exception of one new line of code:

 

string query = string.Format("Element:{0} Template:{1}", Element.GetPath(Database), Template.GetPath(Database));
query += " '|Tank Level':>=75";  //be sure to include leading blank

var search = new AFEventFrameSearch(Database, "optional name", AFSearchMode.Overlapped, StartTime, EndTime, query);
search.CacheTimeout = TimeSpan.FromMinutes(10); // Opt in to server-side caching

IEnumerable<AFEventFrame> frames = search.FindEventFrames(startIndex: 0, fullLoad: false, pageSize: 1000);

 

Line 2 has a literal string specifying the attribute name and value to check.  You may, of course, be working with variables to make the code more flexible.  In that case, you would probably have something like:

 

string attrName = "|Tank Level";
double threshold = 75.0;
query += string.Format(" '{0}':>={1}", attrName, threshold);  //be sure to include leading blank

 

The astute reader will note that a StringBuilder could also be used, if one desired to break the query onto different lines.  In such cases, an AppendFormat would be preferred over AppendLine.

 

 

Other Considerations

 

Server-side Caching

Barry Shang explains caching in his blog Why you should use the new AF SDK Search.  I am echoing his original remarks but noted for AFEventFrameSearch instead.

  • The new search allows you to opt in to caching object identifiers of found event frames on the server. This is done via setting the AFEventFrameSearch.CacheTimeout property.  This effectively takes a "snapshot" of the found collection, caches it, and provides a server-side enumerable collection that the AF client can consume. Caching of identifiers allows SQL Server to retrieve subsequent pages faster by avoiding repetitive queries. The traditional search which does not implement caching will incur more overhead on the SQL Server side when  getting multiple pages of items.
  • Should you opt in to caching when using the new search? Let's see what the docs for CacheTimeout say: If you will only be getting items from the first page, then it is best to leave the cache disabled. If you will be paging through several pages of items, then it is best to enable the cache by setting this property to a value larger than you expect to be using the search.

 

What is being returned by the AFSearch is an Enumerable collection, and not just a List<AFEventFrame> which happens to implement the IEnumerable<AFEventFrame> interface.  Instead the Enumerable collection is being re-fetched each time you iterate over it, either explicitly with a foreach or implicitly with LINQ calls such as Select or Where.  It depends upon the collection size being returned, if you enabled caching, and what you intend to do with the returned items each time. If a very large number will be returned, you probably won’t have enough RAM to keep them all loaded in memory. In this case you probably want to enable caching, not keep a strong reference to items already processed, and let the garbage collector free memory when needed. If the number is smaller, then you will have better performance if you cache the returned list if you are going to make repeated iterations over the list.

 

Lastly, if you do find yourself iterating over the Enumerable collection more than once, be sure to issue an AFServer.Refresh before each subsequent pass over if you want to see newly added items in the returned collection. This would refresh the search that is cached on the server so that it would pick up any added objects.

 

 

When Element Name is not unique

When we are searching on element name with AFEventFrameSearch, you will note my code does not use the AFElement.Name property but instead gets the path relative to the current database.  This would be Element.GetPath(Database) which for my example is equivalent to Element.GetPath(Element.Database).

 

Whereas for the legacy FindEventFrames, I use the element's Name with the named argument referencedElementNameFilter: Element.Name.  That works for my simplified examples where my test database only has 5 elements, all on the root of the database, and all of them are uniquely named.  Since those names are unique, I am guaranteed that the returned event frames only belong to the element I care about.  A real world database would not be so lucky.  If the element names were not unique, what can you do ensure uniqueness?

 

In the help for FindEventFrames, there is this interesting part regarding the name:

 

referencedElementNameFilterType: System.String

The name filter string of an AFElement in the event frame's ReferencedElements collection. Uses same filtering rules as specified for the nameFilter parameter. To obtain an exact match to a specific element, specify the element's ID using a format that includes the opening and closing braces, such as ID.ToString("B").

 

So why didn't I used Element.ID.ToString("B") instead of Element.Name?  Because I isolated all of my legacy tests against an AF 2.6 server, and all the newer AFEventFrameSearch tests were done against an AF 2.8 server.  What I discovered later is that Element.ID.ToString("B") was added in AF 2.7.5 server.  For the newer AFEventFrameSearch tests running on my 2.8 server, I could have interchanged Element.ID.ToString("B") with Element.GetPath(Database) since both return a unique element.

 

 

Using AFSearchTokens

The examples for AFEventFrameSearch used this constructor mainly because it allowed the AFSearchMode.Overlapped.  You can use the constructor that relies upon search tokens but now you must also specify the dates as search tokens.  This code here is longer and looks messier.  But it is an option to you.

 

IEnumerable<AFEventFrame> frames;

List<AFSearchToken> tokenList = new List<AFSearchToken>();

tokenList.Add(new AFSearchToken(AFSearchFilter.Element, AFSearchOperator.Equal, Element.GetPath(Database)));
tokenList.Add(new AFSearchToken(AFSearchFilter.Template, AFSearchOperator.Equal, Template.GetPath(Database)));
// Given nature of AND-ing tokens, this is how Overlapped would work:
tokenList.Add(new AFSearchToken(AFSearchFilter.Start, AFSearchOperator.LessThanOrEqual, EndTime.UtcTime.ToString("O")));
tokenList.Add(new AFSearchToken(AFSearchFilter.End, AFSearchOperator.GreaterThanOrEqual, StartTime.UtcTime.ToString("O")));
// Search by value:
tokenList.Add(new AFSearchToken(AFSearchFilter.Value, AFSearchOperator.GreaterThanOrEqual, "75", "|Tank Level"));

AFEventFrameSearch search = new AFEventFrameSearch(Database, "optional name", tokenList);
search.CacheTimeout = TimeSpan.FromMinutes(10); // Opt in to server-side caching

frames = search.FindEventFrames(startIndex: 0, fullLoad: false, pageSize: 1000);

 

The earlier examples passed a string rather than a token list.  That  query string will be parsed into a token list using the AFEventFrameSearch.ParseQuery method.

If you want to see which search tokens were specified or generated from a query string, use the AFSearch.Tokens property.

 

 

Related Links

 

Search Query Syntax Overview

AFSearchToken Structure

AFSearchOperator Enumeration

AFSearchFilter Enumeration

AFAttributeValueQuery

In this blog post, I will teach you how to get started developing a managed C++ Win32 Console Application using PI AF SDK. Although I know that the majority of our community would you PI AF SDK with C# or VB.NET, I am sure this article will be useful to some users.

 

Before we start, the source code of the project is available on this GitHub repository for you to download.

 

Let's get started creating a Win32 Console Application in Visual Studio as shown on the screenshot below.

 

 

Select a name for your project and click in "Ok". The Win32 Application Wizard will open. Click on "Next".

 

 

You will see the Application Settings. Finally, click on the "Finish" button.

 

 

You will see your solution under the Solution Explorer. On top menu, click on "Project" --> "SampleAFSDKCpp Properties...".

 

Under Configuration Properties --> General, change the Common Language Runtime Support option to Common Language Runtime Support (/clr)

 

 

Under Configuration Properties --> C/C++, change the Additional #using Directives option to %pihome%\AF\PublicAssemblies\4.0;%(AdditionalUsingDirectories).

 

Under Configuration Properties --> C/C++ --> Precompiled Headers, change the Precompile Header option to Not Using Precompiled Headers.

 

 

 

Finally, change the content of the SampleAFSDKCpp.cpp to:

 

#using <mscorlib.dll> // to use Console::WriteLine
#using <OSIsoft.AFSDK.dll>
#include <stdio.h>
// to printf()
using namespace System;
using namespace OSIsoft::AF;
using namespace OSIsoft::AF::Asset;
using namespace OSIsoft::AF::Time;
// Mark unmanaged code
#pragma unmanaged
void print(char *msg)
{
  printf("%s\n", msg);
}


// Switch back to managed code
#pragma managed


int main()
{
  // Write to the console through managed call
  Console::WriteLine("Hello world from managed method");
  PISystems ^piSystems = gcnew PISystems();
  //AF::PISystem  ^piSystem = piSystems->DefaultPISystem;
  PISystem  ^piSystem = piSystems["MARC-PI2016"];
  AFDatabase ^myDb = piSystem->Databases["AFSDKTest"];
  AFElement ^rootElement = myDb->Elements["Tables"];
  AFElement ^subRootElement = rootElement->Elements["InterpolatedTest"];
  AFAttribute ^myAttribute;
  for each (AFAttribute ^attribute  in subRootElement->Attributes)
  {
  if (attribute->Name == "Attribute2")
  {
  myAttribute = attribute;
  break;
  }
  }
  AFTime startTime = AFTime("*-1d");
  AFTime endTime = AFTime("*");
  AFTimeRange time = AFTimeRange(startTime, endTime);
  //time->StartTime = startTime;
  UnitsOfMeasure::UOM ^uom;
  AFValues^ values = myAttribute->Data->RecordedValues(time, OSIsoft::AF::Data::AFBoundaryType::Inside, uom, System::String::Empty, false, 0);


  for each (AFValue^ value in values)
  {
  AFTime^ time = value->Timestamp;
  Console::WriteLine("Value: " + value->Value->ToString() + ", Timestamp: " + time->LocalTime);
  }


  // use stdio to write to console
  print("hello world from unmanaged method");
  return 0;
}

 

When I press CONTROL + F5, I see the compressed values from the attribute whose path is \\MARC-Pi2016\AFSDKTest\Tables\InterpolatedTest|Attribute2.

 

 

Before we finish, there are two questions I would like to answer:

 

1)What is the gcnew?

 

The gcnew is an operator, just like the new operator, except that you don't have to delete anything created with it. It's garbage collected. You use gcnew for creating .Net managed types, and new for creating unmanaged types.

 

2)Why gcnew operator is not used to instantiate the AFTime?

 

Because actually AFTime is a struct and not a class. The gcnew operator should be used with classes only.

 

 

That is all! Hopefully using this simple example is enough for you to get started. If you have any suggestion or question to ask, please post a comment below!

Motivation

 

Many use cases of async PI AF SDK methods will involve making multiple async calls and awaiting their results. For example, we can make an async call for each PI Point in a collection of these objects. This query seems similar to a synchronous bulk call using PIPointList or AFListData. However, the difference is that with async, we can vary the query parameters (such as time range) for each PI Point, which is not possible with the bulk call.

 

In this post, we introduce a pattern that allows PI AF SDK clients to consume the results of async calls as they are completed. One major motivation for doing so is greater concurrency. Processing the result of each async call as it completes allows us to use both CPU and IO resources more effectively.

 

The blog post will walk through some code patterns we can use when making multiple async calls in PI AF SDK.

 

Problem definition

 

We focus on an event frame use case here, although the concepts can apply to any use case involving "heterogeneous" queries across a list of PI Points or AF Attributes (i.e. the parameters of query are not the same for each PIPoint/AFAttribute so a bulk call cannot be used).

 

We would like to perform summary calculations on a list of event frames. For each event frame, we perform a summary calculation for a PI Point within the event frame time range. Each event frame has a different start and end time. There are multiple ways we can programmatically solve this problem.

  1. Synchronous loop over the list of event frames
  2. Parallel tasks over the event frames
  3. Async calls that are processed only after all tasks complete
  4. Async calls using Task.WhenAny to process tasks as they complete
  5. Async calls using IObservable<T> to process tasks as they complete

 

The code for these examples is hosted at:

GitHub - osisoft/PI-AF-SDK-AsyncToObservable

0. Find the Event Frames

AFDatabase database = PISystem.CreatePISystem("SERVER").Databases["AsyncExamples"];
AFEventFrameSearch efSearch = new AFEventFrameSearch(database, "EFSearch", new List<AFSearchToken> {
     new AFSearchToken(AFSearchFilter.Name, AFSearchOperator.Equal, "EF*"),
     new AFSearchToken(AFSearchFilter.Start, AFSearchOperator.GreaterThan, "01-jan-2012"),
     new AFSearchToken(AFSearchFilter.End, AFSearchOperator.LessThan, "01-jan-2013"),
});
IEnumerable<AFEventFrame> efs = efSearch.FindEventFrames(0, false, 1000).Take(10000);

 

We show this because we are using the new search in PI AF SDK 2016 to find our event frames. The important concept to know here is that the new search returns an IEnumerable collection of event frames that we can loop over. We just find the first 10,000 event frames to keep things simpler.

 

1. Synchronous loop over the list of event frames

public static class SyncExample
{
     public static void Run(IEnumerable<AFEventFrame> efs)
     {
          foreach (AFEventFrame ef in efs)
          {
               OutputInfo output = Common.GetSummary(ef);
               Common.WriteToConsole(output);
          }
          Common.WriteCompletion();
     }
}

 

This one is fairly simple. We just loop over the event frames and make a summary call on each (i.e. Common.GetSummary will make an AFData.Summary call). See the Github code for the implementation of the methods above but we try not to make them too important for these examples. Here, by "process", we simply choose to print some output info to the Console, but we target our discussion to be relevant for any generic "processing" tasks.

 

2. Parallel tasks over the event frames

public static class ParallelExample
{
     public static void Run(IEnumerable<AFEventFrame> efs)
     {
          Parallel.ForEach(efs, new ParallelOptions { MaxDegreeOfParallelism = 8 }, ef =>
          {
               OutputInfo output = Common.GetSummary(ef);
               Common.WriteToConsole(output);
          });
          Common.WriteCompletion();
     }
}

 

Here, we use .NET Task Parallel Library (TPL) to parallelize the calls over the event frames. TPL is convenient to use and perhaps ideal if our client machine has many cores, but our data access is over the network while calls to TPL ideally should be CPU-bound. For server-side code bound by threads, such as approach likely will not scale well. In addition, this is a poor fit for web services with many users as it can demand many threads per user.

 

3. Async calls that are processed only after all tasks complete

public static class TaskExample1
{
     public static void Run(IEnumerable<AFEventFrame> efs)
     {
          Task<IList<OutputInfo>> efTasks = GetOutputInfoList(efs);

          IList<OutputInfo> output = efTasks.Result;
          foreach (OutputInfo info in output)
          {
               Common.WriteToConsole(info);
          }
          Common.WriteCompletion();
     }

     private static async Task<IList<OutputInfo>> GetOutputInfoList(IEnumerable<AFEventFrame> efs)
     {
          Task<OutputInfo>[] tasks = efs.Select(async ef => await Common.GetSummaryAsync(ef)).ToArray();
          return await Task.WhenAll(tasks);
     }
}

 

Here, we launch an async call for each event frame (i.e. Common.GetSummaryAsync will make an AFData.SummaryAsync call). We launch them all at once and then await for them all to complete. This suspends the current execution until all calls complete. The method GetOuputInfoList only returns once we've received results from all of our async calls. The async caller in this case must block (with Task.Result) and can only write to the Console (i.e. process the results) once all the tasks are complete.

 

Note that we are achieving PI Data Archive concurrency here and will get client-side concurrency within processing by AF SDK code itself, but our custom code will not get client-side concurrency. In a producer-consumer analogy, we allow our producers to provide results concurrently, but we don't allow our consumers to consume until all the results are provided, and therefore, we are sacrificing some degree of concurrency in this approach.

 

4. Async calls using Task.WhenAny to process tasks as they complete

public static class TaskExample2
{
     public static void Run(IEnumerable<AFEventFrame> efs)
     {
          WriteOutputInfo(efs);
     }
     private static async void WriteOutputInfo(IEnumerable<AFEventFrame> efs)
     {
          List<Task<OutputInfo>> tasks = efs.Select(async ef => await Common.GetSummaryAsync(ef)).ToList();
          while (tasks.Count > 0)
          {
               var t = await Task.WhenAny(tasks);
               tasks.Remove(t);
               Common.WriteToConsole(await t);
          }
          Common.WriteCompletion();
     }
}

 

Above, we've shown the pattern introduced in this post by Stephen Toub. Instead of waiting for all tasks to complete via WhenAll, we wait for just one to complete via WhenAny. Then, we remove it from the list, extract the result of the task, and repeat until the list of tasks is empty. Note several bottlenecks in this case. First, we must materialize the task list. Second, removing the task from the list is O(N). Third, registering the continuation for each task is also O(N). The latter two result in an O(N^2) time complexity, not great if we have a large number of tasks. Although we allow our consumers to process at the same time our producers are producing, we've added extra overhead in the processing. In the same post, another approach is introduced to optimize further. We won't discuss it, but like to suggest a way to tackle this problem in a more "natural" way.

 

5. Async calls using IObservable<T> to process tasks as they complete

public static class ObservableExample
{
     public static void Run(IEnumerable<AFEventFrame> efs)
     {
          IObservable<OutputInfo> obs = efs.ToObservable()
          .Select(i => Observable.FromAsync(() => Common.GetSummaryAsync(i)))
          .Merge();

          obs.Subscribe(Common.WriteToConsole, Common.WriteCompletion);
     }
}

 

What we'd really like to do is launch a bunch of async calls and process the results in the order of task completion, not initialization. In essence, we'd like to "observe" the completion of tasks as a continuous (but bounded) stream. For this pattern, I'm a big proponent of using Rx.NET (NuGet: rx-main).

 

Above, we're using Rx.NET to create an observable sequence of results from tasks as they complete. From the IEnumerable<AFEventFrame>, we convert this to an observable sequence of event frames. For each event frame observed, we make an async summary call returning a Task<OutputInfo>. Here is an important step. Via Observable.FromAsync, we convert the asynchronous receipt of a single result from Task into the asynchronous observation of a sequence producing a single result. Wrapping the Task in this way also ensures it does not start right away but only when we have a subscribed observer! At this point, we have an IObservable<IObservable<OutputInfo>>. The Merge method allows us to "flatten" the observable hierarchy we just created into just a single sequence producing an OutputInfo object for each observed event frame (LINQ experts may recognize this is basically a SelectMany operation). Finally, we subscribe an implicit observer via the Subscribe method. Our observer's OnNext calls WriteToConsole and OnCompleted calls WriteCompletion.

 

The benefit of Rx is that it allows us to declaratively chain data pipelines together, as we could using LINQ to IEnumerable.

 

For example, we can implement buffering easily if we wanted to process a group of items. Below, we write the items to the Console in groups of 1,000.

IObservable<IList<OutputInfo>> obs = efs.ToObservable()
     .Select(i => Observable.FromAsync(() => Common.GetSummaryAsync(i)))
     .Merge()
     .Buffer(1000);
obs.Subscribe(Common.WriteBufferToConsole, Common.WriteCompletion);

 

What if we wanted to process the results in the order of the event frames? Easy, just change Merge() to Concat().

 

Observable sequences are an appropriate pattern to use when you want to process a list of tasks as they complete. The learning curve for Rx is steep, but it can really make code more readable, declarative, and shorter.

 

Bulk call analogy

 

There is an analogy here to the PI AF SDK bulk calls in terms of degree of "laziness". In the bulk call enumerable pattern, we (the client) will page through the results and ask the server to make them available. We do not have to wait for the server to return all of the results before we can start processing. In an async pattern, we can achieve a similar "laziness" by processing the list of tasks as they complete. We do not have to wait for all tasks to complete before processing the results. In the former, we still synchronously "pull" to ask for the next results. In the latter, we receive the results via an asynchronous "push". If you've read my earlier post on using Rx with AFDataPipe, you won't be surprised that the interface to use in the latter case is IObservable<T>

 

If there is one takeaway from this post, it is the table below (inspiration from GitHub - Reactive-Extensions/Rx.NET). It shows the analogy we can draw between synchronous and asynchronous method interfaces and how this analogy is fleshed out in our own PI AF SDK methods.

 

PatternSingle return valueMultiple return values
Return typePI AF SDK exampleReturn typePI AF SDK example
Pull/Synchronous/InteractiveTPIPoint.RecordedValuesIEnumerable<T>PIPointList.RecordedValues
Push/Asynchronous/ReactiveTask<T>PIPoint.RecordedValuesAsyncIObservable<T>Rx.NET example in this blog post

 

Using a PI AF SDK example of RecordedValues calls, we have

T = AFValues

We encourage you to check the PI AF SDK methods listed in the table above to verify. IObservable<T> return type is currently not natively supported in PI AF SDK as of 2016 for historical data retrieval, but we have shown here it is possible to implement this pattern yourself.

 

Throttling

 

There is currently a client-side cap on the number of outstanding calls to a PI Data Archive. If you are making multiple async calls, it is important to keep this cap in mind. Having a queue of tasks or using an async Semaphore pattern can be a way to throttle the calls.

 

References

 

The ideas we've introduced here are not new and have been extensively discussed in the .NET literature. They are worth revisiting however if you plan to use some of the async methods introduced in PI AF SDK 2016. Here is a list of some of our favorites:

 

Introduction to Rx

Async Iterators - Dave Sexton Blog

Processing tasks as they complete - .NET Parallel Programming - Site Home - MSDN Blogs

Help me evangelize Rx: Select + async selector lambda  

Async, await, and yield return

c# - Asynchronous iterator Task<IEnumerable<T>> - Stack Overflow 

Proposal: language support for async sequences · Issue #261 · dotnet/roslyn · GitHub 

pmartin

New Attributes Traits

Posted by pmartin Employee Nov 3, 2016

What are Traits

Traits are a new way of identifying attributes with AF 2016.  Traits can be attached to AF Attributes and are designed to generically identify common characteristics or behaviors. Specifically, a trait can be used to define and/or find related AF Attribute objects with well-known behaviors and relationships. Starting with the AF SDK 2016, users are able to create, edit, delete, view, and utilize their attribute traits

To get a better idea of what traits are, let’s take an example from an application that makes use of traits: PI Coresight.  Coresight can find limits automatically and trend them without the need to know the exact trait attribute name.  See the video here: OSIsoft: Configure Attribute Traits in AF (Asset Framework) to Set Limits [AF 2016].  In this video, attribute names were chosen to match the names of their associated traits. Please be aware that this is not necessary; traits are designed to be generic and independent from the attribute name.

 

Types of Traits

There are 3 types of traits in the AF 2016 release. Additional trait types such as longitude and latitude will be rolled out in later versions of AF Server.

Limit Attribute Traits

Identify expected range of process variables. Available Limit Attribute Traits are Minimum, LoLo, Lo, Target, Hi, HiHi, and Maximum.

Forecast Attribute Traits

Use predicted values to make comparisons to your current values.

Analysis Start-Trigger Attribute Traits

For Event Frame Templates, store the Analysis trigger name or expression that caused the Event Frame to start.

 

Why use them

Traits provide context for attributes. Attributes like minimum value, maximum value, or forecasted value are universal so it doesn’t matter whether you, as the developer, know the process or not.  Want to make a program that validates whether the values coming in to a system are normal? Easy! Just search for the minimum and maximum trait attributes and compare it to the current value of its parent attribute.

 

Preparation

If you are planning on following along, a few things need to be configured before you run the code. I have attached an XML export of my Traits database that you will need to import. Please note that the code used in this project is hardcoded to look at the “Traits” database.  I suggest creating a new database with that name and importing the XML file there.  If you don’t want to create a new database, you will need to modify the line of code that has:

AFDatabase db = AF.Databases["Traits"];

so that it is looking at your database and not the “Traits” database.

 

Creating Traits

Let’s start creating some trait attributes.  First, we’ll find the template that was imported in the previous section.

AFElementTemplate temperatureTemplate = db.ElementTemplates["Temperature_Simple"];

This template has two attributes:  Humidity and Temperature.  These two attributes have values that are randomly generated.  See the attached source code file is you wish to change these values from their initial value.

In this next section of code, we find the Temperature attribute and check if the Hi Limit attribute trait has already been created.  If it hasn’t, we create it using the value of its Abbreviation property for its name.  We then repeat this process for the Lo attribute trait. In this implementation of the code, the Humidity attribute is left unused so that you can experiment with other limit traits if you so desire.

 

//If Hi Trait doesn't exist, create it
if(temperature.AttributeTemplates[AFAttributeTrait.LimitHi.Abbreviation] == null) {
       //create new attribute
       AFAttributeTemplate limitHi = temperature.AttributeTemplates.Add(AFAttributeTrait.LimitHi.Abbreviation); 
       //specify which trait this attribute is supposed to be
       limitHi.Trait = AFAttributeTrait.LimitHi;
       //set the value for the Hi Limit
       limitHi.SetValue(100, limitHi.DefaultUOM);
}

//If Lo Trait doesn't exist, create it
if (temperature.AttributeTemplates[AFAttributeTrait.LimitHi.Abbreviation] == null) {
       AFAttributeTemplate limitLo = temperature.AttributeTemplates.Add(AFAttributeTrait.LimitLo.Abbreviation);
    limitLo.Trait = AFAttributeTrait.LimitLo;
    limitLo.SetValue(0, limitLo.DefaultUOM);
}

 

Once this code has been run, our template is defined.  Next, we'll utilize a for loop to initialize 100 elements using this template with random values for Humidity and Temperature.

 

for (int i = 0; i < 100; i++) {
       AFElement e = db.Elements.Add("Location"+ i);
       e.Template = temperatureTemplate;
       e.Attributes["Humidity"].SetValue(new AFValue(r.Next(0,100)));
       e.Attributes["Temperature"].SetValue(new AFValue(r.Next(-20, 120)));
}

 

Using Traits

As I discussed earlier in this post, traits can be used to quickly find whether your values are out of range.  That’s exactly what we’re going to do in this section.  I’ve outlined two different approaches for doing the same task.  The first method searches by Attribute trait name. This approach can check attributes across multiple templates but will only work if your trait attributes are consistently named.  The second approach searches by parent attribute name.  This approach will only check the attributes with a particular name, but it will do so much faster than the first approach.  In addition, this approach allows us to grab attribute traits with a wide variety of names.

 

Search by Attribute Traits

First, we’ll find all the attributes corresponding to our Lo limit attribute.  We will utilize the FindElementAttributes method to find attributes that have the same name as the abbreviation for the Lo Limit.  If you recall from the previous section, we used the Trait abbreviations to name our Limit attributes.

 

AFAttributeList los = AFAttribute.FindElementAttributes(db, null, null, null, null, AFElementType.Any, AFAttributeTrait.LimitLo.Abbreviation, null, TypeCode.Double, true, AFSortField.Name, AFSortOrder.Descending, 101);

 

Next, we’ll preload the values for Lo so that the performance is better.  The style of preloading uses bulk calls.  See the New Features and Best Practices Webinar and the comparison between Serial vs Parallel vs Bulk calls KB article for more information.  The values for Lo will all be the same in this simple example but it’s a bad habit to assume that it will always be the case.  Limits can be configured on a per Element basis or can be configured to get the value from a PI Point, Table Lookup, as well as other sources.

 

AFValues lovals = los.GetValue();

 

Limits (and Forecasts) are children of the attribute they are to be compared to.  In this next step we will loop through every returned Lo attribute and compare it with the parent to determine whether the parent value is out of range.

 

for (int i = 0; i < los.Count; i++) {
       //if the lo value is greater than the parent (attribute) value, output to console
       if(lovals[i].ValueAsDouble() > los[i].Parent.GetValue().ValueAsDouble()) {
         Console.WriteLine(   "Value under Lo Limit for {0}  :  {1} < {2}", 
los[i].Element.Name, 
los[i].Parent.GetValue().ValueAsDouble(),
lovals[i].ValueAsDouble()); 
       }
}

 

We then repeat the process for our Hi Limit to determine which values have a high temperature.

 

Our output looks something like this:

Value under Lo Limit for Location9  :  -8 < 0

Value under Lo Limit for Location0  :  -20 < 0

Value over Hi Limit for Location98   :  110 > 100

Value over Hi Limit for Location96   :  114 > 100

 

Search by Parent Attribute

First, we’ll have to find the list of parent attributes.  We will utilize the FindElementAttributes method to find attributes named Temperature that are under an element that uses the Temperature_Simple template.   We’ll preload load values from this attribute similar to how we did in the previous section.

 

//Get Temperature Template
AFElementTemplate temperatureTemplate = db.ElementTemplates["Temperature_Simple"];
//Find all attributes that belong to an element of this template
AFAttributeList al = AFAttribute.FindElementAttributes(db, null, null, null, temperatureTemplate, AFElementType.Any, "Temperature", null, TypeCode.Double, true, AFSortField.Name, AFSortOrder.Ascending, 200);
//load values
AFValues alval = al.GetValue();

 

Now that we have the values, we’ll loop through each associated attribute and get its traits by using the GetAttributeByTrait method.  The nice thing about this method is that even if we had created our attribute traits with the crazy name below, it would still find accurately find and identify the limit.

 

AFAttributeTemplate limitHi = temperature.AttributeTemplates.Add("Some crazy high limit name");

 

After verifying that we found an attribute trait by checking for null, we compare the two to see if the value is out of range.

 

foreach(AFValue v in alval) {
     //get the trait values
     AFAttribute lo_a = v.Attribute.GetAttributeByTrait(AFAttributeTrait.LimitLo);
     AFAttribute hi_a = v.Attribute.GetAttributeByTrait(AFAttributeTrait.LimitHi);

     //compare to trait values if the trait exists
     if(hi_a != null && v.ValueAsDouble() > hi_a.GetValue().ValueAsDouble()) {
          Console.WriteLine("Value over Hi Limit for {0}  :  {1} > {2}",
                                             v.Attribute.Element.Name, 
                                             v.ValueAsDouble(), 
          hi_a.GetValue().ValueAsDouble());
     }
     else if(lo_a != null && v.ValueAsDouble() < lo_a.GetValue().ValueAsDouble()) {
              Console.WriteLine("Value under Lo Limit for {0}  :  {1} < {2}", 
                                       v.Attribute.Element.Name, 
                                        v.ValueAsDouble(), 
     lo_a.GetValue().ValueAsDouble());
     } 
}

 

Similar to the first method, the output results look like this:

Value over Hi Limit for Location98     : 110 > 100

Value under Lo Limit for Location85  :  -8 < 0

Value over Hi Limit for Location79     : 112 > 100

Value under Lo Limit for Location72  :  -10 < 0

 

The only difference is that Lo and Hi limits are interleaved.

 

Conclusion

I hope you can see the benefits of using traits even in this contrived example. Even without knowing how or where the temperature was measured, you could instantly tell whether the values you were seeing were expected. More trait types will be coming in the future so you should get on board now!

If you liked this post and want to learn more about traits, let us know.  In addition to some more tips/tricks, I have content on Forecasts and Analysis Start Condition traits that I would love to share.  If you want to learn more about them on your own, the examples of those are included in the source code in the “COMPLEX EXAMPLE” region.

I have attached my source code and an export of my database so that you experiment and modify as much as you’d like.  The code is also available on GitHub.

Filter Blog

By date: By tag: