Skip navigation
All Places > PI Developers Club > Blog > Authors rborges

PI Developers Club

7 Posts authored by: rborges Employee

Introduction

Lately, I've been experimenting with Microsoft Power BI and I'm impressed by how mature the tool is. The application not only supports a myriad of data sources but now there is even a Power Query SDK that allows developers to write their own Data Connectors. Of course, the majority of my experiments uses data from PI Points and AF Attributes and, because Power BI is very SQL oriented, I end up using PI OLEDB Enterprise most of the time. But let's face it: writing a query can be tricky and not a required skill for most Power BI users. So I decided to create a simple PI Web API Data Connector for Power BI. The reason I decided to use PI Web API is that the main use-case for the Data Connector is "Create a business analyst friendly view for a REST API". Also, there's no reason to install additional data providers.

 

Important Notes

Keep in mind that this tool is meant to help in small experiments where a PI and Power BI user wants to get some business intelligence done on PI data quickly. For production environment use cases, where there is a serious need for shaping data views, or where scale is of importance we highly recommend the use of PI Integrator for Business Analytics . Also, in order to avoid confusions, let me be clear that this is not a PI Connector. Microsoft named these extensions to Power BI "Data Connectors" and they are not related to our PI Connector products.

 

The custom Data Connector is a Power BI beta feature, so it may break with the release of newer versions. I will do my best to keep it update but, please, leave a comment if there's something not working.  It's also limited by the Power BI capabilities, that means it currently only supports basic and anonymous authentication for web requests. If the lack of Kerberos support is a no-go for you, please refer to this article on how to use PI OLEDB Enterprise.

 

Preparation

If you are using the latest version of the Power BI (October 2018), you should enable custom data connectors by going to File / Options and Settings / Options / Security and then lowering the security for Data Extensions to Allow any extension to load.

 

2018-10-17 10_48_09-.png

 

For older versions of the Power BI, you have to enable Custom data connectors. It's under File / Options and Settings / Options / Preview features / Custom data connectors.

 

2018-06-28 09_12_56-.png

 

This should automatically create a [My Documents]\Power BI Desktop\Custom Connectors folder. If it's not there, you can create it by yourself. Finally, download the file (PIWebAPIConnector.mez) at the end of this article, extract from the zip and manually place it there. If you have Power BI instance running, you need to restart it before using this connector. Keep in mind that custom data connectors were introduced in April 2018, so versions before that will not be able to use this extension.

 

Hot to use it

You first have to add a new data source by clicking Get Data / More and finding the PI Web API Data Connector under Services.

 

2018-10-17 10_59_21-.png

Once you click the Connect button, a warning will pop up to remember this is a preview connector. Once you acknowledge it, a form will be presented and you must fill it with the appropriate data:

 

2018-10-17 11_09_35-.png

 

Here's a short description of the parameters:

 

ParameterDescriptionAllowed Values
PI Web API ServerThe URL to where the PI Web API instance is located.A valid URL: https://server/piwebapi
Retrieval MethodThe method to get your data.Recorded, Interpolated
ShapeThe shape of the table. List is a row for every data entry, while Transposed is a column for every data path.List, Transposed
Data PathThe full path of a PI Point or an AF Element. Multiple values are allowed separated by a semicolon (;).\\PIServer\Sinusoid;\\AFServer\DB\Element|Attribute
Start TimeThe timestamp at the beginning of the data set you are retrieving.A PI TIme or a timestamp: *-1d, 2018-01-01
End TimeThe timestamp at the end of the data set you are retrieving.A PI TIme or a timestamp: *-1d, 2018-01-01
IntervalOnly necessary for interpolated. The frequency on which the data will be interpolated.A valid Time-Interval: 10m,

 

Once you fill the form with what you need, you hit the OK button and you may be asked for credentials. It's important to mention that, for the time being, there's no support for any other authentication method than anonymous and basic.

 

2018-10-17 11_10_43-Untitled - Power BI Desktop.png

 

After selecting the appropriate login method, a preview of the data is shown:

 

2018-10-17 11_15_11-.png

 

If it seems all right, click Load and a new query result will be added to your data.

 

List vs Transposed

In order to cover the basic usage of data, I'm offering two different ways to present the query result. The first one is List, where you can get data and some important attributes that can be useful from time to time:

 

     2018-10-17 11_23_41-Untitled - Power BI Desktop.png

 

If you don't want the metadata and are only interested in the data, you may find Transposed to be a tad more useful as I already present the data transposed for you:

 

     2018-10-17 11_26_25-Untitled - Power BI Desktop.png

 

Note that for Digital data points I only bring the string value when using transposed. And that's it. Now you can use the data the way you want!

 

2018-06-28 10_39_53-Untitled - Power BI Desktop.png

 

The Code

The GitHub repository for this project is available here. If you want to extend the code I'm providing and have not worked with the M language before, I strongly suggest you watch this live-coding session. Keep in mind that it's a functional language that excels in ETL scenarios, but is not yet fully mature, so expect some quirkiness like the mandatory variable declaration for each instruction, lack of flow control and no support for unit testing.

 

Final Considerations

I hope this tool can make it easier for you to quickly load PI System data into your Power BI dashboard for small projects and use cases. Keep in mind that this extension is not designed for production environments nor it's a full-featured ETL tool. If you need something more powerful and able to handle huge amounts of data, please consider using the PI Integrator for Business Analytics. If you have questions, suggestions or a bug report, please leave a comment.

PI AF was released last week along with a new version AF SDK (2.10.0.8628), so let me show you a feature that has been long requested by the community and that it's now available for you: the AFSession structure. This structure is used to represent a session on the AF Server and it exposes the following members:

 

Public PropertyDescription
AuthenticationTypeThe authentication type of the account which made the connection.
ClientHostThe IP address of the client host which made the connection.
ClientPortThe port number of the client host which made the connection.
EndTimeThe end time of the connection.
GracefulTerminationA boolean that indicates if the end time was logged for graceful client termination.
StartTimeThe start time of the connection.
UserNameThe username of the account which made the connection.

 

In order to get session information of a given PI System, the PISystem class now exposes a function called GetSessions(AFTime? startTime, AFTime? endTime, AFSortOrder sortOrder, int startIndex, int maxCount) and it returns an array of AFSessions. The AFSortOrder is an enumeration defining whether you want the startTime to be ascending or descending. Note that you can specify AFTime.MaxValue at the endTime to search only sessions which are still open.

 

From the documentation's remarks: The returned session data can be used to determine information about clients that are connected to the server. This information can be used to identify active clients. Then from the client machine, you can use the GetClientRpcMetrics() (for AF Server) method to determine what calls the clients are making to the server. Session information is not replicated in PI AF Collective environments. In these setups, make sure you connect to the member you want to retrieve session info from.

 

Shall we see it in action? The code I'm using is very simple:

 

var piSystem = (new PISystems()).DefaultPISystem;
var sessions = piSystem.GetSessions(new AFTime("*-1d"), null, AFSortOrder.Descending);
foreach (var session in sessions)
{
     Console.WriteLine($"---- {session.ClientHost}:{session.ClientPort} ----");
     Console.WriteLine($"Username: {session.UserName}");
     Console.WriteLine($"Start time: {session.StartTime}");
     Console.WriteLine($"End time: {session.EndTime}");
     Console.WriteLine($"Graceful: {session.GracefulTermination}");
     Console.WriteLine();
}

 

A cropped version of the result can be seen below:

 

---- 10.10.10.10:51582 ----

Username: OSI\rborges

Start time: 07/02/18 13:18:54

End time:

Graceful:

 

---- 10.10.10.10:62307 ----

Username: OSI\rborges

Start time: 07/02/18 13:06:36

End time: 07/02/18 13:11:51

Graceful: True

 

---- 10.10.10.10:62305 ----

Username: OSI\rborges

Start time: 07/02/18 13:06:17

End time: 07/02/18 13:06:19

Graceful: True

 

As you can see, now we can easily monitor sessions in your PI System. Share your thoughts about it in the comments and how you are planning on using it.

 

Happy coding!

 

Reference:

Rick's post on how to use metrics with AF SDK.

1. Introduction

Every day more and more customers get in contact with us asking how does PI could be used to leverage their GIS data and how their geospatial information could be used in PI. Our answer is the PI Integrator for Esri ArcGIS. If your operation involves any sort of georeferenced data, geofencing or any kind of geospatial data, I encourage you to give a look at what the PI Integrator for Esri ArcGIS is capable of. But this is PI Developers Club, a haven for DIY PI nerds and curious data-driven minds. So, is it possible to create a custom data reference that provides access to some GIS data and functionalities? Let's do it using an almost-real-life example.

 

2018-06-07 11_38_41-pisquare - QGIS.png1.1 Scenario

The manager of a mining company has to monitor some trucks that operate at the northmost deposit of their open-pit mine. Due to recent rains, their geotechnical engineering team has mapped an unsafe area that should have no more than three trucks inside of it. They have also provided a shapefile with a polygon delimiting a control zone (you can download the shapefile at the end of this post). The manager wants to be notified whenever the number of trucks inside the control area is above a given limit.

 

relationship.pngCaveat lector, I'm not a mining engineer, so please excuse any inaccuracy or misrepresentation of the operations at a zinc mine. It's also important to state that the mine I'm using as an example has no relation to this blog post nor the data I'm using.

 

1.2 Premises

If you are familiar with GIS data, you know it's an endless world of file formats, coordinate systems, and geodetic models. Unless you have a full-featured GIS platform, it's very complicated to handle all possible combinations of data characteristics. So, for the sake of simplicity, this article uses Esri's Shapefile as its data repository and EPSG:4326 as our coordinate system.

 

1.3 A Note on CDRs

As the name implies, a CDR should be used to get data from an external data source that we don't provide support out-of-the-box. Simple calculations can be performed, but you should exercise caution as, depending on how intensive your mathematical operations are, you can decrease the performance of an analysis using this CDR. For our example, shapefiles, GeoJsons, and GeoPackages can be seen as a standalone data source files (as they contain geographic information in it) and the math behind it is pretty simple and it won't affect the server performance.

 

1.4 The AF Structure

Following the diagram on 1.1, our AF structure renders pretty simply: a Zinc Mine element with Trucks as children. The mine element has three attributes: (a) the number of trucks inside the control area (a roll-up analysis), (b) the maximum number of trucks allowed in the control area (a static attribute) and (c) the control area itself.

 

2018-06-06 09_37_25-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

The control area is a static attribute with several supporting children attributes holding the files of the shapefile. Due to shapefile specification, together with the SHP file you also need the other three.

 

2018-06-06 09_39_00-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

Finally, the truck element has two historical attributes for its position and the one using our CDR to tell if it's currently inside the control area or not (this is the one used by the roll-up analysis at the zinc mine element). Here I'm using both latitude and longitude as separated attributes, but if you have AF 2017 R2 or newer, I encourage you to have this data stored as a location attribute trait.

 

2018-06-06 16_55_42-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

 

2. The GISToolBox Data Reference

The best way to present a new CDR by showing its config string:

 

Shape=..\|control area;Method=IsInside;Latitude=Latitude;Longitude=Longitude

 

Breaking it down: Shape is the attribute that holds the shapefile and its supporting files. It's actually just a string with a random name. What is important are the children underneath it that are file attributes and hold the shape files. Method is the method we want to execute. Latitude and Longitude are self-explanatory and they should also point to an attribute. If you don't provide a lat/long attribute, the CDR will use the location attribute trait defined for the element. There are also two other parameters that I will present later.

 

The code is available here and I encourage you to go through it and read the comments. If you want to learn how to create a custom data reference, please check the useful links section at the end of this post.

 

2.1 Dataflow

The CDR starts by overriding the GetInputs method. There we use the values passed by the config string, to get the proper attributes. You should pay close attention to the way the shapefile is organized, as there are some child attributes holding the files (these child attributes are AFFiles). Once this is done, the GetValue is called. It starts by downloading the shapefile from the AF server to a local temporary folder and creating a Shapefile object. Although Esri's specification is open, I'm using DotSpatial to incorporate the file handling and all spatial analysis we do. Once we have the shapefile, it goes through some verifications and we finally call the method that gets the data we want: GISGelper.EvaluateFunction(). For performance reasons, I'm also overriding the GetValues method. The reason is that we don't need to recreate the files for every iteration on the AFValues array sent by the SDK.

 

2.2 Available Methods

Taking into account what I mentioned on 1.3, we should not create sophisticated calculations so the CDR doesn't slow down the Analysis engine. To keep it simple and with good performance, I have implemented the following methods:

NameDescriptionOutputRepresentation
IsInsideDetermines whether a coordinate is inside a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them.

1 if inside

0 if outside

inside.png
IsOutsideDetermines whether a coordinate is outside a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them.

1 if outside

0 if inside

outside.png
MinDistanceDetermines the minimum distance from a coordinate to a polygon in the shapefile. If your shapefile contains several polygons, it will check all of them and return the shortest of them all.A double with the distance in the units defined by the used CRSmindist.png
CentroidDistanceDetermines the distance from a coordinate to a polygon's centroid in the shapefile. If your shapefile contains several polygons, it will check all of them and return the shortest of them all.A double with the distance in the units defined by the used CRScentdist.png

 

2.3 CRS Conversion

The GISToolbox considers that both lat/long and shapefiles are using the same CRS. If your coordinate uses a different base from your shapefile, you can use two other configuration parameters (FromEPSGCode and ToEPSGCode) to convert the coordinate to the same CRS used by the shapefile.

 

Let's say you have a shapefile using EPSG:4326, but your GPS data comes on EPSG:3857. For this case, you can use:

Shape=..\|control area;Method=IsInside;Latitude=Latitude;Longitude=Longitude;FromEPSGCode=3857;ToEPSGCod=4326

 

2.4 Limitations

  • It doesn't implement an AF Data Pipe, so it can't be used with event-triggered analysis (only periodic).
  • It handles temporary files, the user running your AF services must have read/write permissions on the temporary folder.
  • It only supports EPSG CRS
  • It only supports shapefiles.

 

3. Demo

Let's go back to our manager who needs to monitor the trucks inside that specific control area.

 

3.2 Truck Simulation

In order to make our demo more realistic, I have created a small simulation. You can download the shapefile at the end of this post (Trucks_4326.zip). Here's a gif showing the vehicles' position

 

gismap.gif

 

The trucks start outside of the control area and they slowly move towards it. Here's a table showing if a given truck is inside the polygon at a specific timestamp:

TSTruck 001Truck 002Truck 003Truck 004Total
000000
101001
201102
301102
411103
511114
611114
711013
811002

 

The simulation continues until the 14ᵗʰ iteration, but note how the limit is exceeded on the timestamp 5, so we should get a notification right after entering the 5ᵗʰ iteration.

 

3.3 Notification

The notification is dead simple: every 4 seconds I check the Active Trucks attribute against the maximum allowed. And as I mentioned before, the Active Trucks is a roll-up counting the IsInside attribute of each truck.

 

2018-06-07 16_05_09-__RBORGES-AF_GISToolBox - PI System Explorer.png

 

Shall we see it in action?

 

notif.gif

Et voilà!

 

The simulation files are available at the end of this post. Feel free to download and explore it.

 

4. Conclusion

This proof of concept demonstrates how powerful a Custom Data Reference can be. Of course, it doesn't even come close to what the PI Integrator for Esri ArcGIS is capable of, but it shows that for simple tasks, we can mimic functionalities from bigger platforms and can be used as an alternative while a more robust platform is not available.

 

If you like this topic and think that AF should have some basic support to GIS, please chime in on the user voice entry I've created to collect ideas from you.

Motivation

 

Recently, during PI World 2018, I was surprised by the number of people asking me if it's possible to list all PI Points and AF Attributes used in PI Vision's displays. The good news is that it's possible to do it, the bad news is that it's not that straightforward.

 

I will show two different ways to achieve this. The first one using Powershell and the second one querying directly PI Vision's database. Warning: we strongly recommend that you don't mess around with PI Vision's database unless you know what you are doing. If you have questions, please contact tech support or leave a comment in this post.

 

Powershell

 

The PowerShell method is the simplest and safest. In order to understand how it works, let's first do a quick recap of PI's architecture.

 

In a very high-level description, PI uses a producer-consumer pattern: multiple producers (interfaces, connectors, AFSDK writes, etc) send data to a central repository, while consumers subscribe to updates on a set of PI Points. Whenever a new data comes in, the Update Manager Subsystem notifies subscribers that fresh data is available.

 

If you open your PI System Management Tools and navigate to Operation -> Update Manager, you will see a list of all processes consuming and producing data.

 

2018-05-07 09_19_19-Update Manager - PI System Management Tools.png

 

Now, if you filter by *w3wp* (the name of the IIS process) you can drill down the data and get the list of tags being consumed by that specific signup.

 

2018-05-07 09_37_34-Update Manager - PI System Management Tools.png

 

But hey, this is PI DevClub! What about doing it programmatically? Unfortunately, the Update Manager information is not available in AF SDK, but we have the PowerShell tools to help us with this task:

 

$conn = Connect-PIDataArchive -PIDataArchiveMachineName "emiller-vm2";
$pointIds = @();
While ($true)
{
     $consumers = Get-PIUpdateManagerSignupStatistics -Connection $conn -ConsumerName "*w3wp*";
     $consumers | ForEach-Object -Process {
          $pointId = $_.Qualifier;
          if ($pointIds -notcontains $pointId -And $pointId -ne 0)
          {
               $pointIds += $pointId;
               $piPoint = Get-PIPoint -Connection $conn -WhereClause "pointid:=$pointId" -AllAttributes;
               $printObj = New-Object PSObject;
               $printObj | Add-Member Name $piPoint.Point.Name;
               $printObj | Add-Member Description $piPoint.Attributes.descriptor;
               $printObj | Add-Member Changer $piPoint.Attributes.changer;
               Write-Output $printObj;
          }
     }
}

 

If you run this script it will keep listening for every call to your PI Server originated from an IIS:

 

2018-05-07 17_09_02-Windows PowerShell.png

 

By now you may have noticed the two problems of this method: (1) it only shows a new entry if somebody request data for a given PI point (i.e.: open a display) and (2), we are just listing tags and we totally ignore AF attributes. A workaround for the first one is to leave the script running for a while and pipe the result to a text file.

 

PI Vision's Database

 

Let me say this once again before we proceed: we strongly recommend users to not touch PI Visions' database. That said...

 

There are two ways to extract this information from the database. The first one is dead simple, but only works if you don't have displays imported from ProcessBook:

 

SELECT 
     E.Name [DisplayName],
     E.Owner [DisplayOwner],
     D.FullDatasource [Path]
FROM 
     BrowseElements E,
     DisplayDatasources D
WHERE
     E.ID = D.DisplayID

 

The result of this select is a table with all AF Attributes and PI Points used by PI Vision.

 

2018-05-07 17_05_28-SQLQuery1.sql - bmoura-vm3.PIVisualization (OSI_rborges (53))_ - Microsoft SQL S.png

 

This may work for you, but one person that approached me during PI World, also asked me if it was possible to list not only the data sources but also the Symbols using them. Also, most of the displays were imported from ProcessBook. And that's when things get tricky:

 

SELECT
     E.Name as [DisplayName],
     E.Owner as [DisplayOwner],
     S.c.value('../@Id', 'nvarchar(128)') as [Symbol],
     D.c.value('local-name(.)', 'nvarchar(2)') as [Source],
     CASE -- Constructing the path according to the data source
          WHEN D.c.value('local-name(.)', 'nvarchar(2)') = 'PI' -- The data comes from a PI Point
          THEN '\\' +
               CASE WHEN CHARINDEX('?',D.c.value('@Node', 'nvarchar(128)')) > 0 -- Here we check if the server ID is present
               THEN LEFT(D.c.value('@Node', 'nvarchar(128)'), CHARINDEX('?',D.c.value('@Node', 'nvarchar(128)'))-1)
               ELSE D.c.value('@Node', 'nvarchar(128)')
               END 
               + '\' + 
               CASE WHEN CHARINDEX('?',T.c.value('@Name', 'nvarchar(128)')) > 0 -- Here we check if the point ID is present
               THEN LEFT(T.c.value('@Name', 'nvarchar(128)'), CHARINDEX('?',T.c.value('@Name', 'nvarchar(128)'))-1)
               ELSE T.c.value('@Name', 'nvarchar(128)')
               END
          WHEN D.c.value('local-name(.)', 'nvarchar(2)') = 'AF' -- The data comes from an AF attribute
               THEN '\\' + D.c.value('@Node', 'nvarchar(128)')  + '\' + D.c.value('@Db', 'nvarchar(256)') +  '\' 
               + CASE 
               WHEN T.c.value('@ElementPath', 'nvarchar(128)') IS NOT NULL 
               THEN T.c.value('@ElementPath', 'nvarchar(128)') + '|' + T.c.value('@Name', 'nvarchar(128)')
               ELSE O.c.value('@ElementPath', 'nvarchar(128)') + T.c.value('@Name', 'nvarchar(128)')
               END
     END as [Path]
FROM
     BaseDisplays B
     CROSS APPLY B.COG.nodes('/*:COG/*:Datasources/*/*') T(c)
     CROSS APPLY B.COG.nodes('/*:COG/*:Databases/*') D(c)
     CROSS APPLY B.COG.nodes('/*:COG/*:Symbols/*:Symbol/*') S(c)
     LEFT JOIN BaseDisplays B2 OUTER APPLY B2.COG.nodes('/*:COG/*:Contexts/*:AFAttributeParameter') O(c) 
          ON T.c.value('../@Id', 'nvarchar(128)') = O.c.value('@Datasource', 'nvarchar(128)'),
     BrowseElements E
WHERE
     E.ID = B.BrowseElementID
     AND E.DeleteFlag = 'N'
     AND D.c.value('@Id', 'nvarchar(128)') = T.c.value('../@DbRef', 'nvarchar(128)')
     AND T.c.value('../@Id', 'nvarchar(128)') = S.c.value('@Ref', 'nvarchar(128)')

 

The result is a little more comprehensive than the previous script:

 

2018-05-07 17_05_47-SQLQuery2.sql - bmoura-vm3.PIVisualization (OSI_rborges (55))_ - Microsoft SQL S.png

 

These queries were made for the latest version available (2017 R2 Update 1) and it's not guaranteed to be future-proof. It's known that PI Vision 2018 will use a different data model, so, If needed, I will revisit this post after the launch of the 2018 version.

 

I'm not going to dig into the specifics of this script as it has a lot of T-SQL going on to deal with the XML information that is stored in the database. If you have specific questions about how it works, leave a comment. Also keep in mind that this query is a little expensive, so you should consider running during off-peak hours or on a dev database.

 

Conclusion

 

List all tags and attributes used by PI Vision is a valid use case and most PI admins will agree that it helps to understand their current tag usage. We have been increasing our efforts on system usage awareness and, with this post, I hope to help with this goal.

rborges

C#7 & AF SDK

Posted by rborges Employee Apr 17, 2018

If you have Visual Studio 2017 and the .NET Framework 4.6.2, you can benefit from new features that are available from the language specification. Some of them are pure syntactic sugar, but yet useful. The full list can be found in this post and here I have some examples of how you can use them to leverage your AF SDK usage.

 

1) Out variables

We use out variables by declaring them before a function assign a value to it:

 

AFDataPipe pipe = new AFDataPipe();
var more = false;
pipe.GetUpdateEvents(out more);
if (more) { ... }

 

Now you can inline the variable declaration, so there's no need for you to explicitly declare it before. They will be available throughout your current execution scope:

 

AFDataPipe pipe = new AFDataPipe();
pipe.GetUpdateEvents(out bool more);
if (more) { ... }

 

2) Pattern Matching

C# now has the idea of patterns. Those are elements that can test if an object conforms to a given pattern and extract information out of it. Right now the two most useful uses of it are Is-expressions and Switch statements.

 

2.1) Is-expressions

This is very simple and straightforward. what used to be:

 

if (obj.GetType() == typeof(AFDatabase))
{
    var db = (AFDatabase)obj;
    Console.WriteLine(db.Name);
}

 

Can now be simplified to:

 

if (obj is AFDatabase db)
     Console.WriteLine(db.Name);

 

Note that we are only instantiating the db object if it's an AFDatabase.

 

2.2) Switch Statements

So far this is my favorite because it completely changes flow control in C#. For me is the end of if / else if as it allows you to test variables types and values on the go with the when keyword:

 

public AFObject GetParent(AFObject obj)
{
    switch (obj)
    {
        case PISystem system:
            return null;
        case AFDatabase database:
            return database.PISystem;
        case AFElement element when element.Parent == null:
            return element.Database;
        case AFElement element when element.Parent != null:
            return element.Parent;
        case AFAttribute attribute when attribute.Parent == null:
            return attribute.Element;
        case AFAttribute attribute when attribute.Parent != null:
            return attribute.Parent;
        default:
            return null;
    }
}

 

The when keyword is a gamechanger for me. It will make the code simpler and way more readable.

 

3) Tuples

As a Python programmer that has been using tuples for years, I've always felt that C# could benefit from using more of it across the language specification. Well, the time is now! This new feature is not available out-of-the-box. You have to install a missing assembly from NuGet:

 

PM> Install-Package System.ValueTuple

 

Once you do it, you not only have access to new ways to deconstruct a tuple but also use them as function returns. Here's an example of a function that returns the value and the timestamp for a given AFAttribute and AFTime:

 

private (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFAttribute attribute, AFTime time)
{
    var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
    var localTime = afValue?.Timestamp.LocalTime;
    var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
    return (value, localTime);
}

 

Then you can use it like this:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get a list with last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(attribute, t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
}

 

Note how we can unwrap the tuple directly into separated variables. It's the end of out variables!

 

4) Local Functions

Have you gone through a situation where a method exists only to support another method and you don't want other team members using it? That happens frequently when you are dealing with recursion or some very specific data transformations. A good example is in our last snippet, where GetValueAndTimestamp is specific to the method that uses it. In this case, we can move the function declaration to inside the method that uses is:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get the last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
    // Here we declare our GetValueAndTimestamp
    (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFTime time)
    {
        var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
        var localTime = afValue?.Timestamp.LocalTime;
        var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
        return (value, localTime);
    }
}

 

As you can see, we are declaring GetValueAndTimestamp inside PrintLastTenMinutes and blocking it from external calls. This increases encapsulation and helps you keep your code DRY. Note how the attribute is accessible from within local function without passing it as a parameter. Just keep in mind that local variables are passed by reference to the local function (more info here).

 

There are other new features but those are my favorite so far. I hope you see good usage and, please, let me know if you have a good example of C#7.0 features.

Motivation

The AF SDK provides two different ways to get live data updates and I recently did some stress tests on AFDataPipes, comparing the observer pattern (GetObserverEvents) with the more traditional GetUpdateEvents. My goal was to determine if there is a preferred implementation.

 

The Performance Test

The setup is simple: listen to 5332 attributes that are updated at a rate of 20 events per second. This produces over 100k events per second that we should process. I agree that this is not a challenging stress test but is on par with what we usually see on customers around the globe. The server is very modest, with only 8GB of RAM and around 1.2GHz of processor speed (it’s an old spare laptop that we have here at the office). Here is the code I used to fetch data using GetUpdateEvents (DON’T USE IT - Later in this article, I will show the code I've used to test the observer pattern implementation):

 

var dataPipe = new AFDataPipe();
dataPipe.AddSignups(attributes);
CancellationTokenSource source = new CancellationTokenSource();
Task.Run(async () =>
{
    try
    {
        while (!source.IsCancellationRequested)
        {
            // Here we fetch new data
            var updates = dataPipe.GetUpdateEvents();
            foreach (var update in updates)
            {
                Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
                            update.Value.Attribute.GetPath(),
                            update.Value.Value,
                            update.Value.Timestamp.ToString());
            }
            await Task.Delay(500);
        }
    }
    catch (Exception exception)
    {
        Console.WriteLine("Server sent an error: {0}", exception.Message);
    }
}, source.Token);
Console.ReadKey();
source.Cancel();
dataPipe.RemoveSignups(attributes);
dataPipe.Dispose();
Console.WriteLine("Finished");

 

After several hours running the application, I noticed that the GetUpdateEvents was falling behind and sometimes it was leaving some data for the next iteration. This is not a problem per se as, eventually, it would catch up with current data. I suspected that this would happen, but I decided to investigate what was going on. After some bit twiddling, I noticed something weird. Below we have a chart with the memory used by the application. On the top, we have the one that uses GetObserverEvents. On the bottom the GetUpdateEvents. They both use the same amount of memory but look closely at the number of GC calls executed by the .NET Framework.

 

2018-04-09 11_39_55-ObservableTest (Running) - Microsoft Visual Studio.png

(using GetObserverEvents)

2018-04-09 11_37_51-ObservableTest (Running) - Microsoft Visual Studio.png

(Using GetUpdateEvents)

 

Conclusion

Amazingly, this is expected as we are running the code on a server with a limited amount of memory and GetUpdates has extra code to deal with. Honestly, I was expected an increased memory usage and the GC kicking in like this was a surprise. Ultimately, the .NET framework is trying to save my bad code by constantly freeing resources back to the system.

 

Can this be fixed? Absolutely, but it is a waste of time as you could use this effort to implement the observer pattern (that handles all of this natively) and get some extra benefits:

  • Because it allows you to decouple your event handling from the code that is responsible for fetching the new data.
  • Because it is OOP and easier to encapsulate.
  • Because it is easier to control the flow.

 

Observer Pattern Implementation for AF SDK

In this GitHub file, you can find the full implementation of a reusable generic class that listens to AF attributes and executes a callback when new data arrives. It's very simple, efficient and has a minimal memory footprint. Let’s break down the most important aspects of it so I can explain what’s going on and show how it works.

 

The class starts by implementing the IObserver interface. This allows it to subscribe itself to receive notifications of incoming data. I also implement IDisposable because the observer pattern can cause memory leaks when you fail to explicitly unsubscribe to observers. This is known as the lapsed listener problem and it is a very common cause of memory issues:

 

public class AttributeListener : IObserver<AFDataPipeEvent>, IDisposable

 

Then comes our constructor:

 

public AttributeListener(List<AFAttribute> attributes, Action<AFDataPipeEvent> onNewData, Action<Exception> onError, Action onFinish)
{
      _dataPipe = new AFDataPipe();
     _dataPipe.Subscribe(this);
}

 

Here I expect some controversy. First, because we are moving the subject to inside the observer and breaking the traditional structure of the pattern. Secondly, by using Action callbacks I’m going against the Event Pattern that Microsoft has been using since the first version of the .NET framework and has a lot of fans. It's a matter of preference and there are no performance differences. I personally don’t like events because they are too verbose and we usually don't remove the accessor (ie: implement a -=) and that can cause memory leaks. By the way, I’m not alone on this preference for reactive design as even the folks from Redmond think that reactive code is more suitable for the observer pattern. The takeaway here is how we subscribe the class to the AFDataPipe while keeping the data handling oblivious to it, giving us maximum encapsulation and modularity.

 

Now comes the important stuff, the code that does the polling:

 

public void StartListening()
{
    if (Attributes.Count > 0)
    {
        _dataPipe.AddSignups(Attributes);
        Task.Run(async () =>
        {
        while (!_source.IsCancellationRequested)
            {
            _dataPipe.GetObserverEvents();
            await Task.Delay(500);
            }
        }, _source.Token);
    }
}

 

There is not much to talk about this code.  It starts a background thread with a cancellable loop that polls new data every 500 milliseconds. The await operator (together with the async modifier) allows our anonymous function to run fully asynchronous. Additionally, note how the cancellation token is used twice: as a regular token for the thread created by Task.Run(), but also as a loop breaker, ensuring that there will be no more calls to the server. To see how the cancelation is handled, give a look at the StopListening method of the class.

 

When a new DataPipeEvent arrives, the AF SDK calls the OnNext method of the IObserver. In our case it’s a simple code that only executes the callback provided to the constructor:

 

public void OnNext(AFDataPipeEvent pipeEvent)
{
     _onNewDataCallBack?.Invoke(pipeEvent);
}

 

Caveat lector: This is an oversimplified version of the actual implementation. In the final version of the class , the IObserver implementations are actually piping data to a BufferBlock that fires your Action whenever a new AFDataPipeEvent comes in. I'm using a producer-consumer pattern based on Microsoft's Dataflow library.

 

Finally, here’s an example of how this class should be used. The full code is available in this GitHub repo:

 

static void Main(string[] args)
{
    // We start by getting the database that we want to get data from
    AFDatabase database = (new PISystems())["MySystem"].Databases["MyDB"];
    // Defining our callbacks
    void newDataCallback(AFDataPipeEvent pipeEvent)
    {
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
        pipeEvent.Value.Attribute.GetPath(), pipeEvent.Value.Value, pipeEvent.Value.Timestamp.ToString());
    };
    void errorCallback(Exception exception) => Console.WriteLine("Server sent an error: {0}", exception.Message);
    void finishCallback() => Console.WriteLine("Finished");
    // Then we search for the attributes that we want
    IEnumerable<AFAttribute> attributes = null;
    using (AFAttributeSearch attributeSearch =
        new AFAttributeSearch(database, "ROPSearch", @"Element:{ Name:'Rig*' } Name:'ROP'"))
    {
        attributeSearch.CacheTimeout = TimeSpan.FromMinutes(10);
        attributes = attributeSearch.FindAttributes();
    }
    // We proceed by creating our listener
    var listener = new AttributeListener(attributes.Take(10).ToList(), finishCallback);
    listener.StartListening();
    // Now we inform the user that a key press cancels everything
    Console.WriteLine("Press any key to quit");
    // Now we consume new data arriving
    listener.ConsumeDataAsync(newDataCallback, errorCallback);
    // Then we wait for an user key press to end it all
    Console.ReadKey();
    // User pressed a key, let's stop everything
    listener.StopListening();
}

 

Simple and straightforward. I hope you like it. And please, let me know whether you agree or disagree with me. This is an important topic and everybody benefits from this discussion!

 

UPDATE: Following David's comments, I updated the class to offer both async and sync data handling. Give a look at my final code to see how to use the two methods. Keep in mind that the sync version will run on the main thread and block it, so I strongly suggest you use the async call ConsumeDataAsync(). If you need to update your GUI from a separated thread, use Control.Invoke.

 

Related posts

Barry Shang's post on Reactive extensions for AF SDK

Patrice Thivierge 's post on how to use DataPipes

Marcos Loeff's post on observer pattern with PI Web API channels

David Moler's comment on GetObserverEvents performance

Barry Shang's post on async observer calls

Hello everybody,

 

I would like to share with you an app I've been working on. I call it AF Bash. It allows you to interact with AF the same way you interact with your files through CMD:

2018-03-13 14_48_38-C__Users_rborges_Documents_Projects_afbash_afbash_bin_Debug_afbash.exe.png

 

From a code perspective, it's a framework that provides a quick and easy way for you to implement your set of commands, so I encourage you to write your custom commands and send a pull request to the main repository! But first, lets break it down into topics and explain the architecture and implementation details

 

1) Architecture

Here is a very simple implementation diagram:

2018-03-13 16_12_05-Drawing1 - Visio Professional.png

 

AFBash is a console application that uses Autofac as its IoC container and exposes an interface called ICommand that is implemented by BaseCommand, an abstract class that is the base class for all commands to derive from. It provides a context class full of goodies that should be used to access AF SDK data. The console is wrapped around a custom version of ReadLine, where I can manage command history, autocompletation and command cancelling.

 

I strongly encourage you to follow the comments in the main entry point because it will make easier o understand how the main loop works and what it expects from your custom Command.

 

2) Adding a new Command

So lets stop the chit chat and go to the fun part. How to create a custom command. For this example, let me show you how to implement a dir / ls command.

 

First you need a class that is derived from BaseCommand.

 

class Dir : BaseCommand 

 

Because we are using Autofac's IoC container, we just need to declare a ctor that receives a Context as parameter.

 

public Dir(Context context) : base(context)

 

Now we have to take care of 4 functions:

 

public override ConsoleOutput Execute(CancellationToken token)
public override List<string> GetAliases()
public override string GetHelp()
public override (bool, string) ProcessArgs(string args)

 

The GetAliases() function must provide the alias you want for the command that you are implementing. The GetHelp() must return a simple help information.

 

public override List<string> GetAliases()
{
     return new List<string> { "dir", "ls" };
}
public override string GetHelp()
{
     return "Lists all children of an element";
}

 

The ProcessArgs() function is where get the arguments that your command must parse and store any variable that will be used later by Execute(). Note that BaseCommand exposes a global variable called BaseElement where you can store the AFObject output of your parsing. Going get back to our exemple, a dir command can be execute with or without parameters. A parameter-less dir implies that you want to list everything from the current element. Meanwhile, a parameter may be a full or relative path. So how to take care of it? simple. The AppContext has a function called GetElementRelativeToCurrent(string arg). It will return the element based on the argument passed. Here some examples:

Current
Argument
Result
\\Server\Database\FirstElement\Child".."\\Server\Database\FirstElement
\\Server\Database\FirstElement\Child"grandChild"\\Server\Database\FirstElement\Child\grandChild
\\Server\Database\FirstElement\Child"\" or "\\"\\ (a state where no element is selected)
\\Server\Database\FirstElement\Child"~"\\DefaultServer\DefaultDatabase
\\Server\Database\FirstElement\"child\grandChild"\\Server\Database\FirstElement\Child\grandChild
\\Server\Database\FirstElement\Child"" or null\\Server\Database\FirstElement\Child

 

So far we have this for our argument processing (note that I'm using C#7.0, where a function can return multiple arguments.):

 

public virtual (bool, string) ProcessArgs(string args)
{
    BaseElement = AppContext.GetElementRelativeToCurrent(args);

    if (BaseElement == null)
        return (false, string.Format("Object '{0}' not found", args));
    else
        return (true, null);
}

 

It's only missing one thing: when your current element is the top most node, the CurrentElement is Null because there is no PISystem selected. So a Null BaseElement is not necessarily a bad thing. We just have to check whether the user intentionally did that or was a mistake. This processing is already implemented as a virtual function on BaseCommand. So if your function argument is a path you don't even need to override it as BaseElement will be populated with the target element.

 

Finally the Execute() function.

 

It must return a ConsoleOutput, a wrapper class that makes easier for you to print structures into the console. In our exemple lets set a header message like CMD's dir:

 

ConsoleOutput console = new ConsoleOutput();
console.AddHeaderLine(string.Format("Children of {0}", BaseElement is null ? "root" : BaseElement.GetPath()));

 

We have a table-like result, so we need to set the headers:

 

console.SetBodyHeader(new List<string> { "Type", "Name" });

 

Now, we get the children of BaseElement and loop through them, printing everything we want:

 

var children = AppContext.GetChildren(BaseElement);
children.ForEach(c => {
                console.AddBodyLine(new List<Tuple<string, Color>> {
                    new Tuple<string, Color>(c.Identity.ToString(), AppContext.Colors[c.Identity]),
                    new Tuple<string, Color>(c.ToString(), AppContext.Colors.Base)
                });
            });
return console;

 

And that's it! Now you just need to compile and you are good to go! The actual implementation of my dir also handle data for attributes. I encourage you to go and see how I did it.

 

3) Available Commands

 

So far I have implemented 7 basic commands: CD / LS

 

This is a work in progress, so if you clone this project, keep your repo up-to-date because I will keep pushing bug fixes and new features.

 

Finally, if you find bugs or have questions, let me know!

Filter Blog

By date: By tag: