Skip navigation
All Places > PI Developers Club > Blog > Authors rborges

PI Developers Club

6 Posts authored by: rborges Employee

PI AF was released last week along with a new version AF SDK (2.10.0.8628), so let me show you a feature that has been long requested by the community and that it's now available for you: the AFSession structure. This structure is used to represent a session on the AF Server and it exposes the following members:

 

Public PropertyDescription
AuthenticationTypeThe authentication type of the account which made the connection.
ClientHostThe IP address of the client host which made the connection.
ClientPortThe port number of the client host which made the connection.
EndTimeThe end time of the connection.
GracefulTerminationA boolean that indicates if the end time was logged for graceful client termination.
StartTimeThe start time of the connection.
UserNameThe username of the account which made the connection.

 

In order to get session information of a given PI System, the PISystem class now exposes a function called GetSessions(AFTime? startTime, AFTime? endTime, AFSortOrder sortOrder, int startIndex, int maxCount) and it returns an array of AFSessions. The AFSortOrder is an enumeration defining whether you want the startTime to be ascending or descending. Note that you can specify AFTime.MaxValue at the endTime to search only sessions which are still open.

 

From the documentation's remarks: The returned session data can be used to determine information about clients that are connected to the server. This information can be used to identify active clients. Then from the client machine, you can use the GetClientRpcMetrics() (for AF Server) method to determine what calls the clients are making to the server. Session information is not replicated in PI AF Collective environments. In these setups, make sure you connect to the member you want to retrieve session info from.

 

Shall we see it in action? The code I'm using is very simple:

 

var piSystem = (new PISystems()).DefaultPISystem;
var sessions = piSystem.GetSessions(new AFTime("*-1d"), null, AFSortOrder.Descending);
foreach (var session in sessions)
{
     Console.WriteLine($"---- {session.ClientHost}:{session.ClientPort} ----");
     Console.WriteLine($"Username: {session.UserName}");
     Console.WriteLine($"Start time: {session.StartTime}");
     Console.WriteLine($"End time: {session.EndTime}");
     Console.WriteLine($"Graceful: {session.GracefulTermination}");
     Console.WriteLine();
}

 

A cropped version of the result can be seen below:

 

---- 10.10.10.10:51582 ----

Username: OSI\rborges

Start time: 07/02/18 13:18:54

End time:

Graceful:

 

---- 10.10.10.10:62307 ----

Username: OSI\rborges

Start time: 07/02/18 13:06:36

End time: 07/02/18 13:11:51

Graceful: True

 

---- 10.10.10.10:62305 ----

Username: OSI\rborges

Start time: 07/02/18 13:06:17

End time: 07/02/18 13:06:19

Graceful: True

 

As you can see, now we can easily monitor sessions in your PI System. Share your thoughts about it in the comments and how you are planning on using it.

 

Happy coding!

 

Reference:

Rick's post on how to use metrics with AF SDK.

1. Introduction

Every day more and more customers get in contact with us asking how does PI could be used to leverage their GIS data and how their geospatial information could be used in PI. Our answer is the PI Integrator for Esri ArcGIS. If your operation involves any sort of georeferenced data, geofencing or any kind of geospatial data, I encourage you to give a look at what the PI Integrator for Esri ArcGIS is capable of. But this is PI Developers Club, a haven for DIY PI nerds and curious data-driven minds. So, is it possible to create a custom data reference that provides access to some GIS data and functionalities? Let's do it using an almost-real-life example.

 

2018-06-07 11_38_41-pisquare - QGIS.png1.1 Scenario

The manager of a mining company has to monitor some trucks that operate at the northmost deposit of their open-pit mine. Due to recent rains, their geotechnical engineering team has mapped an unsafe area that should have no more than three trucks inside of it. They have also provided a shapefile with a polygon delimiting a control zone (you can download the shapefile at the end of this post). The manager wants to be notified whenever the number of trucks inside the control area is above a given limit.

 

relationship.pngCaveat lector, I'm not a mining engineer, so please excuse any inaccuracy or misrepresentation of the operations at a zinc mine. It's also important to state that the mine I'm using as an example has no relation to this blog post nor the data I'm using.

 

1.2 Premises

If you are familiar with GIS data, you know it's an endless world of file formats, coordinate systems, and geodetic models. Unless you have a full-featured GIS platform, it's very complicated to handle all possible combinations of data characteristics. So, for the sake of simplicity, this article uses Esri's Shapefile as its data repository and EPSG:4326 as our coordinate system.

 

1.3 A Note on CDRs

As the name implies, a CDR should be used to get data from an external data source that we don't provide support out-of-the-box. Simple calculations can be performed, but you should exercise caution as, depending on how intensive your mathematical operations are, you can decrease the performance of an analysis using this CDR. For our example, shapefiles, GeoJsons, and GeoPackages can be seen as a standalone data source files (as they contain geographic information in it) and the math behind it is pretty simple and it won't affect the server performance.

 

1.4 The AF Structure

Following the diagram on 1.1, our AF structure renders pretty simply: a Zinc Mine element with Trucks as children. The mine element has three attributes: (a) the number of trucks inside the control area (a roll-up analysis), (b) the maximum number of trucks allowed in the control area (a static attribute) and (c) the control area itself.

 

2018-06-06 09_37_25-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

The control area is a static attribute with several supporting children attributes holding the files of the shapefile. Due to shapefile specification, together with the SHP file you also need the other three.

 

2018-06-06 09_39_00-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

Finally, the truck element has two historical attributes for its position and the one using our CDR to tell if it's currently inside the control area or not (this is the one used by the roll-up analysis at the zinc mine element). Here I'm using both latitude and longitude as separated attributes, but if you have AF 2017 R2 or newer, I encourage you to have this data stored as a location attribute trait.

 

2018-06-06 16_55_42-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

 

2. The GISToolBox Data Reference

The best way to present a new CDR by showing its config string:

 

Shape=..\|control area;Method=IsInside;Latitude=Latitude;Longitude=Longitude

 

Breaking it down: Shape is the attribute that holds the shapefile and its supporting files. It's actually just a string with a random name. What is important are the children underneath it that are file attributes and hold the shape files. Method is the method we want to execute. Latitude and Longitude are self-explanatory and they should also point to an attribute. If you don't provide a lat/long attribute, the CDR will use the location attribute trait defined for the element. There are also two other parameters that I will present later.

 

The code is available here and I encourage you to go through it and read the comments. If you want to learn how to create a custom data reference, please check the useful links section at the end of this post.

 

2.1 Dataflow

The CDR starts by overriding the GetInputs method. There we use the values passed by the config string, to get the proper attributes. You should pay close attention to the way the shapefile is organized, as there are some child attributes holding the files (these child attributes are AFFiles). Once this is done, the GetValue is called. It starts by downloading the shapefile from the AF server to a local temporary folder and creating a Shapefile object. Although Esri's specification is open, I'm using DotSpatial to incorporate the file handling and all spatial analysis we do. Once we have the shapefile, it goes through some verifications and we finally call the method that gets the data we want: GISGelper.EvaluateFunction(). For performance reasons, I'm also overriding the GetValues method. The reason is that we don't need to recreate the files for every iteration on the AFValues array sent by the SDK.

 

2.2 Available Methods

Taking into account what I mentioned on 1.3, we should not create sophisticated calculations so the CDR doesn't slow down the Analysis engine. To keep it simple and with good performance, I have implemented the following methods:

NameDescriptionOutputRepresentation
IsInsideDetermines whether a coordinate is inside a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them.

1 if inside

0 if outside

inside.png
IsOutsideDetermines whether a coordinate is outside a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them.

1 if outside

0 if inside

outside.png
MinDistanceDetermines the minimum distance from a coordinate to a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them and return the shortest of them all.A double with the distance in the units defined by the used CRSmindist.png
CentroidDistanceDetermines the distance from a coordinate to a polygon's centroid in the shapefile. If your shapefile contains several polygons, it will check all of them and return the shortest of them all.A double with the distance in the units defined by the used CRScentdist.png

 

2.3 CRS Conversion

The GISToolbox considers that both lat/long and shapefiles are using the same CRS. If your coordinate uses a different base from your shapefile, you can use two other configuration parameters (FromEPSGCode and ToEPSGCode) to convert the coordinate to the same CRS used by the shapefile.

 

Let's say you have a shapefile using EPSG:4326, but your GPS data comes on EPSG:3857. For this case, you can use:

Shape=..\|control area;Method=IsInside;Latitude=Latitude;Longitude=Longitude;FromEPSGCode=3857;ToEPSGCod=4326

 

2.4 Limitations

  • It doesn't implement an AF Data Pipe, so it can't be used with event-triggered analysis (only periodic).
  • It handles temporary files, the user running your AF services must have read/write permissions on the temporary folder.
  • It only supports EPSG CRS
  • It only supports shapefiles.

 

3. Demo

Let's go back to our manager who needs to monitor the trucks inside that specific control area.

 

3.2 Truck Simulation

In order to make our demo more realistic, I have created a small simulation. You can download the shapefile at the end of this post (Trucks_4326.zip). Here's a gif showing the vehicles' position

 

gismap.gif

 

The trucks start outside of the control area and they slowly move towards it. Here's a table showing if a given truck is inside the polygon at a specific timestamp:

TSTruck 001Truck 002Truck 003Truck 004Total
000000
101001
201102
301102
411103
511114
611114
711013
811002

 

The simulation continues until the 14ᵗʰ iteration, but note how the limit is exceeded on the timestamp 5, so we should get a notification right after entering the 5ᵗʰ iteration.

 

3.3 Notification

The notification is dead simple: every 4 seconds I check the Active Trucks attribute against the maximum allowed. And as I mentioned before, the Active Trucks is a roll-up counting the IsInside attribute of each truck.

 

2018-06-07 16_05_09-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

Shall we see it in action?

 

notif.gif

Et voilà!

 

The simulation files are available at the end of this post. Feel free to download and explore it.

 

4. Conclusion

This proof of concept demonstrates how powerful a Custom Data Reference can be. Of course, it doesn't even come close to what the PI Integrator for Esri ArcGIS is capable of, but it shows that for simple tasks, we can mimic functionalities from bigger platforms and can be used as an alternative while a more robust platform is not available.

 

If you like this topic and think that AF should have some basic support to GIS, please chime in on the user voice entry I've created to collect ideas from you.

Motivation

 

Recently, during PI World 2018, I was surprised by the number of people asking me if it's possible to list all PI Points and AF Attributes used in PI Vision's displays. The good news is that it's possible to do it, the bad news is that it's not that straightforward.

 

I will show two different ways to achieve this. The first one using Powershell and the second one querying directly PI Vision's database. Warning: we strongly recommend that you don't mess around with PI Vision's database unless you know what you are doing. If you have questions, please contact tech support or leave a comment in this post.

 

Powershell

 

The PowerShell method is the simplest and safest. In order to understand how it works, let's first do a quick recap of PI's architecture.

 

In a very high-level description, PI uses a producer-consumer pattern: multiple producers (interfaces, connectors, AFSDK writes, etc) send data to a central repository, while consumers subscribe to updates on a set of PI Points. Whenever a new data comes in, the Update Manager Subsystem notifies subscribers that fresh data is available.

 

If you open your PI System Management Tools and navigate to Operation -> Update Manager, you will see a list of all processes consuming and producing data.

 

2018-05-07 09_19_19-Update Manager - PI System Management Tools.png

 

Now, if you filter by *w3wp* (the name of the IIS process) you can drill down the data and get the list of tags being consumed by that specific signup.

 

2018-05-07 09_37_34-Update Manager - PI System Management Tools.png

 

But hey, this is PI DevClub! What about doing it programmatically? Unfortunately, the Update Manager information is not available in AF SDK, but we have the PowerShell tools to help us with this task:

 

$conn = Connect-PIDataArchive -PIDataArchiveMachineName "emiller-vm2";
$pointIds = @();
While ($true)
{
     $consumers = Get-PIUpdateManagerSignupStatistics -Connection $conn -ConsumerName "*w3wp*";
     $consumers | ForEach-Object -Process {
          $pointId = $_.Qualifier;
          if ($pointIds -notcontains $pointId -And $pointId -ne 0)
          {
               $pointIds += $pointId;
               $piPoint = Get-PIPoint -Connection $conn -WhereClause "pointid:=$pointId" -AllAttributes;
               $printObj = New-Object PSObject;
               $printObj | Add-Member Name $piPoint.Point.Name;
               $printObj | Add-Member Description $piPoint.Attributes.descriptor;
               $printObj | Add-Member Changer $piPoint.Attributes.changer;
               Write-Output $printObj;
          }
     }
}

 

If you run this script it will keep listening for every call to your PI Server originated from an IIS:

 

2018-05-07 17_09_02-Windows PowerShell.png

 

By now you may have noticed the two problems of this method: (1) it only shows a new entry if somebody request data for a given PI point (i.e.: open a display) and (2), we are just listing tags and we totally ignore AF attributes. A workaround for the first one is to leave the script running for a while and pipe the result to a text file.

 

PI Vision's Database

 

Let me say this once again before we proceed: we strongly recommend users to not touch PI Visions' database. That said...

 

There are two ways to extract this information from the database. The first one is dead simple, but only works if you don't have displays imported from ProcessBook:

 

SELECT 
     E.Name [DisplayName],
     E.Owner [DisplayOwner],
     D.FullDatasource [Path]
FROM 
     BrowseElements E,
     DisplayDatasources D
WHERE
     E.ID = D.DisplayID

 

The result of this select is a table with all AF Attributes and PI Points used by PI Vision.

 

2018-05-07 17_05_28-SQLQuery1.sql - bmoura-vm3.PIVisualization (OSI_rborges (53))_ - Microsoft SQL S.png

 

This may work for you, but one person that approached me during PI World, also asked me if it was possible to list not only the data sources but also the Symbols using them. Also, most of the displays were imported from ProcessBook. And that's when things get tricky:

 

SELECT
     E.Name as [DisplayName],
     E.Owner as [DisplayOwner],
     S.c.value('../@Id', 'nvarchar(128)') as [Symbol],
     D.c.value('local-name(.)', 'nvarchar(2)') as [Source],
     CASE -- Constructing the path according to the data source
          WHEN D.c.value('local-name(.)', 'nvarchar(2)') = 'PI' -- The data comes from a PI Point
          THEN '\\' +
               CASE WHEN CHARINDEX('?',D.c.value('@Node', 'nvarchar(128)')) > 0 -- Here we check if the server ID is present
               THEN LEFT(D.c.value('@Node', 'nvarchar(128)'), CHARINDEX('?',D.c.value('@Node', 'nvarchar(128)'))-1)
               ELSE D.c.value('@Node', 'nvarchar(128)')
               END 
               + '\' + 
               CASE WHEN CHARINDEX('?',T.c.value('@Name', 'nvarchar(128)')) > 0 -- Here we check if the point ID is present
               THEN LEFT(T.c.value('@Name', 'nvarchar(128)'), CHARINDEX('?',T.c.value('@Name', 'nvarchar(128)'))-1)
               ELSE T.c.value('@Name', 'nvarchar(128)')
               END
          WHEN D.c.value('local-name(.)', 'nvarchar(2)') = 'AF' -- The data comes from an AF attribute
               THEN '\\' + D.c.value('@Node', 'nvarchar(128)')  + '\' + D.c.value('@Db', 'nvarchar(256)') +  '\' 
               + CASE 
               WHEN T.c.value('@ElementPath', 'nvarchar(128)') IS NOT NULL 
               THEN T.c.value('@ElementPath', 'nvarchar(128)') + '|' + T.c.value('@Name', 'nvarchar(128)')
               ELSE O.c.value('@ElementPath', 'nvarchar(128)') + T.c.value('@Name', 'nvarchar(128)')
               END
     END as [Path]
FROM
     BaseDisplays B
     CROSS APPLY B.COG.nodes('/*:COG/*:Datasources/*/*') T(c)
     CROSS APPLY B.COG.nodes('/*:COG/*:Databases/*') D(c)
     CROSS APPLY B.COG.nodes('/*:COG/*:Symbols/*:Symbol/*') S(c)
     LEFT JOIN BaseDisplays B2 OUTER APPLY B2.COG.nodes('/*:COG/*:Contexts/*:AFAttributeParameter') O(c) 
          ON T.c.value('../@Id', 'nvarchar(128)') = O.c.value('@Datasource', 'nvarchar(128)'),
     BrowseElements E
WHERE
     E.ID = B.BrowseElementID
     AND E.DeleteFlag = 'N'
     AND D.c.value('@Id', 'nvarchar(128)') = T.c.value('../@DbRef', 'nvarchar(128)')
     AND T.c.value('../@Id', 'nvarchar(128)') = S.c.value('@Ref', 'nvarchar(128)')

 

The result is a little more comprehensive than the previous script:

 

2018-05-07 17_05_47-SQLQuery2.sql - bmoura-vm3.PIVisualization (OSI_rborges (55))_ - Microsoft SQL S.png

 

These queries were made for the latest version available (2017 R2 Update 1) and it's not guaranteed to be future-proof. It's known that PI Vision 2018 will use a different data model, so, If needed, I will revisit this post after the launch of the 2018 version.

 

I'm not going to dig into the specifics of this script as it has a lot of T-SQL going on to deal with the XML information that is stored in the database. If you have specific questions about how it works, leave a comment. Also keep in mind that this query is a little expensive, so you should consider running during off-peak hours or on a dev database.

 

Conclusion

 

List all tags and attributes used by PI Vision is a valid use case and most PI admins will agree that it helps to understand their current tag usage. We have been increasing our efforts on system usage awareness and, with this post, I hope to help with this goal.

rborges

C#7 & AF SDK

Posted by rborges Employee Apr 17, 2018

If you have Visual Studio 2017 and the .NET Framework 4.6.2, you can benefit from new features that are available from the language specification. Some of them are pure syntactic sugar, but yet useful. The full list can be found in this post and here I have some examples of how you can use them to leverage your AF SDK usage.

 

1) Out variables

We use out variables by declaring them before a function assign a value to it:

 

AFDataPipe pipe = new AFDataPipe();
var more = false;
pipe.GetUpdateEvents(out more);
if (more) { ... }

 

Now you can inline the variable declaration, so there's no need for you to explicitly declare it before. They will be available throughout your current execution scope:

 

AFDataPipe pipe = new AFDataPipe();
pipe.GetUpdateEvents(out bool more);
if (more) { ... }

 

2) Pattern Matching

C# now has the idea of patterns. Those are elements that can test if an object conforms to a given pattern and extract information out of it. Right now the two most useful uses of it are Is-expressions and Switch statements.

 

2.1) Is-expressions

This is very simple and straightforward. what used to be:

 

if (obj.GetType() == typeof(AFDatabase))
{
    var db = (AFDatabase)obj;
    Console.WriteLine(db.Name);
}

 

Can now be simplified to:

 

if (obj is AFDatabase db)
     Console.WriteLine(db.Name);

 

Note that we are only instantiating the db object if it's an AFDatabase.

 

2.2) Switch Statements

So far this is my favorite because it completely changes flow control in C#. For me is the end of if / else if as it allows you to test variables types and values on the go with the when keyword:

 

public AFObject GetParent(AFObject obj)
{
    switch (obj)
    {
        case PISystem system:
            return null;
        case AFDatabase database:
            return database.PISystem;
        case AFElement element when element.Parent == null:
            return element.Database;
        case AFElement element when element.Parent != null:
            return element.Parent;
        case AFAttribute attribute when attribute.Parent == null:
            return attribute.Element;
        case AFAttribute attribute when attribute.Parent != null:
            return attribute.Parent;
        default:
            return null;
    }
}

 

The when keyword is a gamechanger for me. It will make the code simpler and way more readable.

 

3) Tuples

As a Python programmer that has been using tuples for years, I've always felt that C# could benefit from using more of it across the language specification. Well, the time is now! This new feature is not available out-of-the-box. You have to install a missing assembly from NuGet:

 

PM> Install-Package System.ValueTuple

 

Once you do it, you not only have access to new ways to deconstruct a tuple but also use them as function returns. Here's an example of a function that returns the value and the timestamp for a given AFAttribute and AFTime:

 

private (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFAttribute attribute, AFTime time)
{
    var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
    var localTime = afValue?.Timestamp.LocalTime;
    var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
    return (value, localTime);
}

 

Then you can use it like this:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get a list with last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(attribute, t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
}

 

Note how we can unwrap the tuple directly into separated variables. It's the end of out variables!

 

4) Local Functions

Have you gone through a situation where a method exists only to support another method and you don't want other team members using it? That happens frequently when you are dealing with recursion or some very specific data transformations. A good example is in our last snippet, where GetValueAndTimestamp is specific to the method that uses it. In this case, we can move the function declaration to inside the method that uses is:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get the last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
    // Here we declare our GetValueAndTimestamp
    (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFTime time)
    {
        var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
        var localTime = afValue?.Timestamp.LocalTime;
        var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
        return (value, localTime);
    }
}

 

As you can see, we are declaring GetValueAndTimestamp inside PrintLastTenMinutes and blocking it from external calls. This increases encapsulation and helps you keep your code DRY. Note how the attribute is accessible from within local function without passing it as a parameter. Just keep in mind that local variables are passed by reference to the local function (more info here).

 

There are other new features but those are my favorite so far. I hope you see good usage and, please, let me know if you have a good example of C#7.0 features.

Motivation

The AF SDK provides two different ways to get live data updates and I recently did some stress tests on AFDataPipes, comparing the observer pattern (GetObserverEvents) with the more traditional GetUpdateEvents. My goal was to determine if there is a preferred implementation.

 

The Performance Test

The setup is simple: listen to 5332 attributes that are updated at a rate of 20 events per second. This produces over 100k events per second that we should process. I agree that this is not a challenging stress test but is on par with what we usually see on customers around the globe. The server is very modest, with only 8GB of RAM and around 1.2GHz of processor speed (it’s an old spare laptop that we have here at the office). Here is the code I used to fetch data using GetUpdateEvents (DON’T USE IT - Later in this article, I will show the code I've used to test the observer pattern implementation):

 

var dataPipe = new AFDataPipe();
dataPipe.AddSignups(attributes);
CancellationTokenSource source = new CancellationTokenSource();
Task.Run(async () =>
{
    try
    {
        while (!source.IsCancellationRequested)
        {
            // Here we fetch new data
            var updates = dataPipe.GetUpdateEvents();
            foreach (var update in updates)
            {
                Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
                            update.Value.Attribute.GetPath(),
                            update.Value.Value,
                            update.Value.Timestamp.ToString());
            }
            await Task.Delay(500);
        }
    }
    catch (Exception exception)
    {
        Console.WriteLine("Server sent an error: {0}", exception.Message);
    }
}, source.Token);
Console.ReadKey();
source.Cancel();
dataPipe.RemoveSignups(attributes);
dataPipe.Dispose();
Console.WriteLine("Finished");

 

After several hours running the application, I noticed that the GetUpdateEvents was falling behind and sometimes it was leaving some data for the next iteration. This is not a problem per se as, eventually, it would catch up with current data. I suspected that this would happen, but I decided to investigate what was going on. After some bit twiddling, I noticed something weird. Below we have a chart with the memory used by the application. On the top, we have the one that uses GetObserverEvents. On the bottom the GetUpdateEvents. They both use the same amount of memory but look closely at the number of GC calls executed by the .NET Framework.

 

2018-04-09 11_39_55-ObservableTest (Running) - Microsoft Visual Studio.png

(using GetObserverEvents)

2018-04-09 11_37_51-ObservableTest (Running) - Microsoft Visual Studio.png

(Using GetUpdateEvents)

 

Conclusion

Amazingly, this is expected as we are running the code on a server with a limited amount of memory and GetUpdates has extra code to deal with. Honestly, I was expected an increased memory usage and the GC kicking in like this was a surprise. Ultimately, the .NET framework is trying to save my bad code by constantly freeing resources back to the system.

 

Can this be fixed? Absolutely, but it is a waste of time as you could use this effort to implement the observer pattern (that handles all of this natively) and get some extra benefits:

  • Because it allows you to decouple your event handling from the code that is responsible for fetching the new data.
  • Because it is OOP and easier to encapsulate.
  • Because it is easier to control the flow.

 

Observer Pattern Implementation for AF SDK

In this GitHub file, you can find the full implementation of a reusable generic class that listens to AF attributes and executes a callback when new data arrives. It's very simple, efficient and has a minimal memory footprint. Let’s break down the most important aspects of it so I can explain what’s going on and show how it works.

 

The class starts by implementing the IObserver interface. This allows it to subscribe itself to receive notifications of incoming data. I also implement IDisposable because the observer pattern can cause memory leaks when you fail to explicitly unsubscribe to observers. This is known as the lapsed listener problem and it is a very common cause of memory issues:

 

public class AttributeListener : IObserver<AFDataPipeEvent>, IDisposable

 

Then comes our constructor:

 

public AttributeListener(List<AFAttribute> attributes, Action<AFDataPipeEvent> onNewData, Action<Exception> onError, Action onFinish)
{
      _dataPipe = new AFDataPipe();
     _dataPipe.Subscribe(this);
}

 

Here I expect some controversy. First, because we are moving the subject to inside the observer and breaking the traditional structure of the pattern. Secondly, by using Action callbacks I’m going against the Event Pattern that Microsoft has been using since the first version of the .NET framework and has a lot of fans. It's a matter of preference and there are no performance differences. I personally don’t like events because they are too verbose and we usually don't remove the accessor (ie: implement a -=) and that can cause memory leaks. By the way, I’m not alone on this preference for reactive design as even the folks from Redmond think that reactive code is more suitable for the observer pattern. The takeaway here is how we subscribe the class to the AFDataPipe while keeping the data handling oblivious to it, giving us maximum encapsulation and modularity.

 

Now comes the important stuff, the code that does the polling:

 

public void StartListening()
{
    if (Attributes.Count > 0)
    {
        _dataPipe.AddSignups(Attributes);
        Task.Run(async () =>
        {
        while (!_source.IsCancellationRequested)
            {
            _dataPipe.GetObserverEvents();
            await Task.Delay(500);
            }
        }, _source.Token);
    }
}

 

There is not much to talk about this code.  It starts a background thread with a cancellable loop that polls new data every 500 milliseconds. The await operator (together with the async modifier) allows our anonymous function to run fully asynchronous. Additionally, note how the cancellation token is used twice: as a regular token for the thread created by Task.Run(), but also as a loop breaker, ensuring that there will be no more calls to the server. To see how the cancelation is handled, give a look at the StopListening method of the class.

 

When a new DataPipeEvent arrives, the AF SDK calls the OnNext method of the IObserver. In our case it’s a simple code that only executes the callback provided to the constructor:

 

public void OnNext(AFDataPipeEvent pipeEvent)
{
     _onNewDataCallBack?.Invoke(pipeEvent);
}

 

Caveat lector: This is an oversimplified version of the actual implementation. In the final version of the class , the IObserver implementations are actually piping data to a BufferBlock that fires your Action whenever a new AFDataPipeEvent comes in. I'm using a producer-consumer pattern based on Microsoft's Dataflow library.

 

Finally, here’s an example of how this class should be used. The full code is available in this GitHub repo:

 

static void Main(string[] args)
{
    // We start by getting the database that we want to get data from
    AFDatabase database = (new PISystems())["MySystem"].Databases["MyDB"];
    // Defining our callbacks
    void newDataCallback(AFDataPipeEvent pipeEvent)
    {
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
        pipeEvent.Value.Attribute.GetPath(), pipeEvent.Value.Value, pipeEvent.Value.Timestamp.ToString());
    };
    void errorCallback(Exception exception) => Console.WriteLine("Server sent an error: {0}", exception.Message);
    void finishCallback() => Console.WriteLine("Finished");
    // Then we search for the attributes that we want
    IEnumerable<AFAttribute> attributes = null;
    using (AFAttributeSearch attributeSearch =
        new AFAttributeSearch(database, "ROPSearch", @"Element:{ Name:'Rig*' } Name:'ROP'"))
    {
        attributeSearch.CacheTimeout = TimeSpan.FromMinutes(10);
        attributes = attributeSearch.FindAttributes();
    }
    // We proceed by creating our listener
    var listener = new AttributeListener(attributes.Take(10).ToList(), finishCallback);
    listener.StartListening();
    // Now we inform the user that a key press cancels everything
    Console.WriteLine("Press any key to quit");
    // Now we consume new data arriving
    listener.ConsumeDataAsync(newDataCallback, errorCallback);
    // Then we wait for an user key press to end it all
    Console.ReadKey();
    // User pressed a key, let's stop everything
    listener.StopListening();
}

 

Simple and straightforward. I hope you like it. And please, let me know whether you agree or disagree with me. This is an important topic and everybody benefits from this discussion!

 

UPDATE: Following David's comments, I updated the class to offer both async and sync data handling. Give a look at my final code to see how to use the two methods. Keep in mind that the sync version will run on the main thread and block it, so I strongly suggest you use the async call ConsumeDataAsync(). If you need to update your GUI from a separated thread, use Control.Invoke.

 

Related posts

Barry Shang's post on Reactive extensions for AF SDK

Patrice Thivierge 's post on how to use DataPipes

Marcos Loeff's post on observer pattern with PI Web API channels

David Moler's comment on GetObserverEvents performance

Barry Shang's post on async observer calls

Hello everybody,

 

I would like to share with you an app I've been working on. I call it AF Bash. It allows you to interact with AF the same way you interact with your files through CMD:

2018-03-13 14_48_38-C__Users_rborges_Documents_Projects_afbash_afbash_bin_Debug_afbash.exe.png

 

From a code perspective, it's a framework that provides a quick and easy way for you to implement your set of commands, so I encourage you to write your custom commands and send a pull request to the main repository! But first, lets break it down into topics and explain the architecture and implementation details

 

1) Architecture

Here is a very simple implementation diagram:

2018-03-13 16_12_05-Drawing1 - Visio Professional.png

 

AFBash is a console application that uses Autofac as its IoC container and exposes an interface called ICommand that is implemented by BaseCommand, an abstract class that is the base class for all commands to derive from. It provides a context class full of goodies that should be used to access AF SDK data. The console is wrapped around a custom version of ReadLine, where I can manage command history, autocompletation and command cancelling.

 

I strongly encourage you to follow the comments in the main entry point because it will make easier o understand how the main loop works and what it expects from your custom Command.

 

2) Adding a new Command

So lets stop the chit chat and go to the fun part. How to create a custom command. For this example, let me show you how to implement a dir / ls command.

 

First you need a class that is derived from BaseCommand.

 

class Dir : BaseCommand 

 

Because we are using Autofac's IoC container, we just need to declare a ctor that receives a Context as parameter.

 

public Dir(Context context) : base(context)

 

Now we have to take care of 4 functions:

 

public override ConsoleOutput Execute(CancellationToken token)
public override List<string> GetAliases()
public override string GetHelp()
public override (bool, string) ProcessArgs(string args)

 

The GetAliases() function must provide the alias you want for the command that you are implementing. The GetHelp() must return a simple help information.

 

public override List<string> GetAliases()
{
     return new List<string> { "dir", "ls" };
}
public override string GetHelp()
{
     return "Lists all children of an element";
}

 

The ProcessArgs() function is where get the arguments that your command must parse and store any variable that will be used later by Execute(). Note that BaseCommand exposes a global variable called BaseElement where you can store the AFObject output of your parsing. Going get back to our exemple, a dir command can be execute with or without parameters. A parameter-less dir implies that you want to list everything from the current element. Meanwhile, a parameter may be a full or relative path. So how to take care of it? simple. The AppContext has a function called GetElementRelativeToCurrent(string arg). It will return the element based on the argument passed. Here some examples:

Current
Argument
Result
\\Server\Database\FirstElement\Child".."\\Server\Database\FirstElement
\\Server\Database\FirstElement\Child"grandChild"\\Server\Database\FirstElement\Child\grandChild
\\Server\Database\FirstElement\Child"\" or "\\"\\ (a state where no element is selected)
\\Server\Database\FirstElement\Child"~"\\DefaultServer\DefaultDatabase
\\Server\Database\FirstElement\"child\grandChild"\\Server\Database\FirstElement\Child\grandChild
\\Server\Database\FirstElement\Child"" or null\\Server\Database\FirstElement\Child

 

So far we have this for our argument processing (note that I'm using C#7.0, where a function can return multiple arguments.):

 

public virtual (bool, string) ProcessArgs(string args)
{
    BaseElement = AppContext.GetElementRelativeToCurrent(args);

    if (BaseElement == null)
        return (false, string.Format("Object '{0}' not found", args));
    else
        return (true, null);
}

 

It's only missing one thing: when your current element is the top most node, the CurrentElement is Null because there is no PISystem selected. So a Null BaseElement is not necessarily a bad thing. We just have to check whether the user intentionally did that or was a mistake. This processing is already implemented as a virtual function on BaseCommand. So if your function argument is a path you don't even need to override it as BaseElement will be populated with the target element.

 

Finally the Execute() function.

 

It must return a ConsoleOutput, a wrapper class that makes easier for you to print structures into the console. In our exemple lets set a header message like CMD's dir:

 

ConsoleOutput console = new ConsoleOutput();
console.AddHeaderLine(string.Format("Children of {0}", BaseElement is null ? "root" : BaseElement.GetPath()));

 

We have a table-like result, so we need to set the headers:

 

console.SetBodyHeader(new List<string> { "Type", "Name" });

 

Now, we get the children of BaseElement and loop through them, printing everything we want:

 

var children = AppContext.GetChildren(BaseElement);
children.ForEach(c => {
                console.AddBodyLine(new List<Tuple<string, Color>> {
                    new Tuple<string, Color>(c.Identity.ToString(), AppContext.Colors[c.Identity]),
                    new Tuple<string, Color>(c.ToString(), AppContext.Colors.Base)
                });
            });
return console;

 

And that's it! Now you just need to compile and you are good to go! The actual implementation of my dir also handle data for attributes. I encourage you to go and see how I did it.

 

3) Available Commands

 

So far I have implemented 7 basic commands: CD / LS

 

This is a work in progress, so if you clone this project, keep your repo up-to-date because I will keep pushing bug fixes and new features.

 

Finally, if you find bugs or have questions, let me know!

Filter Blog

By date: By tag: