Skip navigation
All Places > PI Developers Club > Blog > 2018 > April
2018

On behalf of Engineering's Technology Enablement team, and in conjunction with Technical Support's Developer Technologies team, it is my privilege to announce this year's Community All-Stars.  We give special recognition to these individuals for their prolific and relentless contributions to the PI Community at large.  We thank them for sharing their experiences and knowledge for all things PI, and in particular for any posts that are developer related.  Let me add that each recipient holds near and dear to each of them the concept of "community", and even if we did not hand out such awards, or if we did not hand out any prize other than simple recognition, these individuals would still post and contribute to our PI Square Community with the same dedication for the sheer sake of expanding the knowledge base of the community.

 

I ask my fellow PI Geeks to help congratulate our deserving 2018 All-Stars!  It's interesting to note that are 3 Community All-Star and one Rising Star are from 4 different continents.

 

PI All Stars 2018.jpg

Rick Davin (far left), Paurav Joshi, John Messinger, and Dan Fishman.  Not shown: Roger Palmen.

 

 

PI Developers Community All-Stars 2018

  • Dan Fishman, Exele Information Systems.  Dan is a first time winner as Community All-Star but won twice before as OSIsoft All-Star.
  • John Messinger, Optimate Pty Ltd.  John wins for the 2nd year in a row.
  • Roger Palmen, CGI.  I don't remember a year when Roger wasn't a winner.

 

Last year as an experiment, I began following a lot of people.  Anyone who has ever been an All-Star, honorable mention, or just nominated.  I can tell you that my Inbox was slammed all year by answers from Dan, John, and Roger.  Thanks, guys.  I don't mean that sarcastically.  Each "spam" in my Inbox from you is you helping another community member, so absolutely, sincerely THANKS!

 

PI Developers Club Community Rising Star 2018

We introduced this category last year to recognize contributions and efforts on well deserving contributors that steadily add to our content here but just fall short of the elite upper strata of All-Star.  A big difference between Rising Star and All-Star is a Rising Star may only be awarded once, whereas All-Star may be won year after year (I'm looking at you, Roger).  Anyone who has ever been an All-Star or Rising Star in the past is ineligible for further consideration as a Rising Star.  Last year we handed out 3 of these as a means of playing catch-up to some folks who have been solid contributors over the year.  This year we have only one winner:

 

 

Paurav along with Dan, John, and Roger win:

  1. Recognition in the community
  2. One year free subscription or renewal to PI Developers Club
  3. One-time free registration to a 2019 PI World event (SF or EMEA)
  4. A legal waiver for them to sign
  5. An Amazon gift card worth $400 USD granted after submitting the signed legal waiver back to us

 

And let's not forget there are a lot of fine contributors within OSIsoft.  While it is our job to help customers, we do want to recognize a select few for their efforts the past year.

 

PI Developers Club OSIsoft All-Stars 2018

 

Our OSIsoft recipients win:

  1. Recognition in the community
  2. Amazon gift card worth $200 USD

 

The clock has been reset and everyone's slate is clean

It's never too early to start thinking about 2019 All-Stars.  We monitor the quantity and quality of posts from one PI World SF to the next in order to pick our winners.  The new countdown has begun.  One thing I can tell you that all of our 2018 winners have in common are the following:

  • A high volume of posts
  • Posts of high quality (lots flagged as helpful or marked as answers)
  • A strong desire to help others.

 

Absolutely, positively candidates must contribute to threads other than their own.  In short, you must show strong participation in the community as a whole.  Without a doubt this year's crop of winners have done just that.

 

CONGRATULATIONS TO ALL WINNERS!

OSIsoft has released a new watchface for PI World attendees!
20180419_102953.jpg

 

Here's how to get it on your SAMSUNG Gear S2 or S3 watch.   Start the SAMSUNG Gear store on your phone:

20180424_144259.jpg

Now jump to the Watch Face list...

20180424_144439.jpg

 

And search for "OSIsoft"...

20180424_144457.jpg

 

And now you can install it to your watch!

 

Screenshot_20180424-101715_Galaxy Apps.jpg

 

The watch face runs on TIZEN, SAMSUNG's newest mobile OS.

 

 

PI World NA is next week and I am quite excited.  Join us the afternoon of Day 3 (Thursday April 26) from 4:30 - 6:00 PM PDT for the first ever Developers Meet-Up Reception, where we will also be handing out a gaggle of awards for this year's crop of Community All-Stars, winners of the Innovation Hackathon, as well as the winners of the recent Visualization Virtual Hackathon!  If you consider yourself a developer, or just wish you were a developer in-training, or are a member of PI Developers Club, or at the very least are attending any DevCon presentations or labs at Parc 55, then by all means stop in for a beer or refreshing drink, have a snack, and mingle with fellow developers.  At 5:00 PM we will begin announcing awards, where you may congratulate the winners or commiserate with the losers other not-so-fortunate participants.  Don't overdo it because at 7:00 PM the party shifts to the Hilton for the always fun PI Geek Night, with more food and drinks, games, and other hijinks.

 

What:     Developer Meet-Up Reception & Awards

Where:   Parc 55, Cyril Magnin I & Foyer, Level 4

When:    4:30 PM - 6:00 PM (awards start at 5:00 PM)

 

You may see more at the bottom of this link:  PI World for Developers: PI World - OSIsoft Users Conference 2018 (SF)

 

Since it is being held in the late afternoon and to a limited segment of PI World attendees, this is not listed as an Evening Event.  However, it does appear on the Day 3 Agenda for Parc 55 hotel under the Sessions Agenda

 

We hope to see you there!

rborges

C#7 & AF SDK

Posted by rborges Employee Apr 17, 2018

If you have Visual Studio 2017 and the .NET Framework 4.6.2, you can benefit from new features that are available from the language specification. Some of them are pure syntactic sugar, but yet useful. The full list can be found in this post and here I have some examples of how you can use them to leverage your AF SDK usage.

 

1) Out variables

We use out variables by declaring them before a function assign a value to it:

 

AFDataPipe pipe = new AFDataPipe();
var more = false;
pipe.GetUpdateEvents(out more);
if (more) { ... }

 

Now you can inline the variable declaration, so there's no need for you to explicitly declare it before. They will be available throughout your current execution scope:

 

AFDataPipe pipe = new AFDataPipe();
pipe.GetUpdateEvents(out bool more);
if (more) { ... }

 

2) Pattern Matching

C# now has the idea of patterns. Those are elements that can test if an object conforms to a given pattern and extract information out of it. Right now the two most useful uses of it are Is-expressions and Switch statements.

 

2.1) Is-expressions

This is very simple and straightforward. what used to be:

 

if (obj.GetType() == typeof(AFDatabase))
{
    var db = (AFDatabase)obj;
    Console.WriteLine(db.Name);
}

 

Can now be simplified to:

 

if (obj is AFDatabase db)
     Console.WriteLine(db.Name);

 

Note that we are only instantiating the db object if it's an AFDatabase.

 

2.2) Switch Statements

So far this is my favorite because it completely changes flow control in C#. For me is the end of if / else if as it allows you to test variables types and values on the go with the when keyword:

 

public AFObject GetParent(AFObject obj)
{
    switch (obj)
    {
        case PISystem system:
            return null;
        case AFDatabase database:
            return database.PISystem;
        case AFElement element when element.Parent == null:
            return element.Database;
        case AFElement element when element.Parent != null:
            return element.Parent;
        case AFAttribute attribute when attribute.Parent == null:
            return attribute.Element;
        case AFAttribute attribute when attribute.Parent != null:
            return attribute.Parent;
        default:
            return null;
    }
}

 

The when keyword is a gamechanger for me. It will make the code simpler and way more readable.

 

3) Tuples

As a Python programmer that has been using tuples for years, I've always felt that C# could benefit from using more of it across the language specification. Well, the time is now! This new feature is not available out-of-the-box. You have to install a missing assembly from NuGet:

 

PM> Install-Package System.ValueTuple

 

Once you do it, you not only have access to new ways to deconstruct a tuple but also use them as function returns. Here's an example of a function that returns the value and the timestamp for a given AFAttribute and AFTime:

 

private (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFAttribute attribute, AFTime time)
{
    var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
    var localTime = afValue?.Timestamp.LocalTime;
    var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
    return (value, localTime);
}

 

Then you can use it like this:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get a list with last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(attribute, t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
}

 

Note how we can unwrap the tuple directly into separated variables. It's the end of out variables!

 

4) Local Functions

Have you gone through a situation where a method exists only to support another method and you don't want other team members using it? That happens frequently when you are dealing with recursion or some very specific data transformations. A good example is in our last snippet, where GetValueAndTimestamp is specific to the method that uses it. In this case, we can move the function declaration to inside the method that uses is:

 

public void PrintLastTenMinutes(AFAttribute attribute)
{
    // First we get the last 10 minutes
    var timestamps = Enumerable.Range(0, 10).Select(m => 
        DateTime.Now.Subtract(TimeSpan.FromMinutes(m))).ToList();
    // Then, for each timestamp ...
    timestamps.ForEach(t => {
        // We get the attribute value
        var (val, time) = GetValueAndTimestamp(t);
        // and print it
        Console.WriteLine($"Value={val} at {time} local time.");
    });
    // Here we declare our GetValueAndTimestamp
    (double? Value, DateTime? LocalTime) GetValueAndTimestamp(AFTime time)
    {
        var afValue = attribute?.Data.RecordedValue(time, AFRetrievalMode.Auto, null);
        var localTime = afValue?.Timestamp.LocalTime;
        var value = afValue.IsGood ? afValue.ValueAsDouble() : (double?)null;
        return (value, localTime);
    }
}

 

As you can see, we are declaring GetValueAndTimestamp inside PrintLastTenMinutes and blocking it from external calls. This increases encapsulation and helps you keep your code DRY. Note how the attribute is accessible from within local function without passing it as a parameter. Just keep in mind that local variables are passed by reference to the local function (more info here).

 

There are other new features but those are my favorite so far. I hope you see good usage and, please, let me know if you have a good example of C#7.0 features.

Note: Development and Testing purposes only. Not supported in production environments.

 

Link to other containerization articles

Containerization Hub

 

Introduction

Today, I will be teaching you a recipe for cooking a PI Data Archive container. Please see my previous blog posts above on how to get Docker installed. We will be mixing the ingredients in a folder to create an image. After we have the image, we can bake the image to obtain a container.

 

Ingredients

For the source code to build the PI Data Archive container. Please send an email to technologyenablement@osisoft.com. This is a short term measure to obtain the source code while we are revising our public code sharing policies. Apologies for any inconvience caused.

1. PI Data Archive 2017 R2A Install Kit Download from techsupport website (contains the software)

2. Dockerfile (describes the mixing steps to form the image)

3. build.bat (script to start the mixing)

4. generateid.txt (reference commands for changing the Server ID)

5. pilicense.dat (many ways to obtain one such as through Account Manager/Partner Manager/Academic Team/PI DevClub membership, best to get a demo license that doesn't require a MSF)

6. temp.txt (adds host trust)

7. trust.bat (adds host trust)

 

Recipe

1. Gather all the required ingredients as listed above

2. Extract out the Enterprise_X64 folder from the PI Data Archive Install Kit.

3. Add pilicense.dat into the Enterprise_X64 folder. Override the existing files if needed.

4. Put the other ingredients into the parent folder of the Enterprise_X64 folder.

 

Your folder structure should now look like this.

5. Execute build.bat. The mixing will take less than 5 minutes.

 

6. Once the image is formed, you can now execute

 

docker run -it --hostname <DNS hostname> --name <containername> pida

 

at the command line to bake the image. This will take about 15 seconds.

You will see the IP address of the PI Data Archive listed in the IPv4 Address field. Use this IP address with PI SMT on your container host to connect. Your PI Data Archive container is now ready to be consumed!

 

Hint: Multiple PI Data Archive instances

If you want to bake another instance of the PI Data Archive container (just repeat step 6 with a different hostname and containername), you will need to change the Server ID too. The following procedure can be done in piconfig to accomplish this.

 

@tabl pisys,piserver
@mode ed
@istr name,serverid
hostname,
@quit

 

Replace hostname with the real hostname of your container. For more information about this, please refer to this KB article. Duplicate PI Server IDs cause PI SDK applications to fail

 

Limitations

1. This example does not persist data or configuration between runs of the container image.

2. This example relies on PI Data Archive trusts and local accounts for authentication.

3. This example doesn't support VSS backups.

4. This example doesn't support upgrading without re-initialization of data.

 

For Developers

Here is an example to connect with AF SDK and read the current value of a PI Point.

 

Conclusion

Notice how quick and easy it is to cook a PI Data Archive container. I hope you find it delicious. I like it because I can easily cook up instances for testing and remove them when I do not need them. (Please don't waste food)

 

Update 31 May 2018

Local account is no longer in the administrators group. Only a mapping to a PI Identity.

 

Update 2 Jul 2018

Added 17R2 tag.

 

Update 21 Nov 2018

For the source code to build the PI Data Archive container. Please send an email to technologyenablement@osisoft.com. This is a short term measure to obtain the source code while we are revising our public code sharing policies. Apologies for any inconvience caused.

Important update to OSIsoft's GitHub Policy, November 2018

Motivation

The AF SDK provides two different ways to get live data updates and I recently did some stress tests on AFDataPipes, comparing the observer pattern (GetObserverEvents) with the more traditional GetUpdateEvents. My goal was to determine if there is a preferred implementation.

 

The Performance Test

The setup is simple: listen to 5332 attributes that are updated at a rate of 20 events per second. This produces over 100k events per second that we should process. I agree that this is not a challenging stress test but is on par with what we usually see on customers around the globe. The server is very modest, with only 8GB of RAM and around 1.2GHz of processor speed (it’s an old spare laptop that we have here at the office). Here is the code I used to fetch data using GetUpdateEvents (DON’T USE IT - Later in this article, I will show the code I've used to test the observer pattern implementation):

 

var dataPipe = new AFDataPipe();
dataPipe.AddSignups(attributes);
CancellationTokenSource source = new CancellationTokenSource();
Task.Run(async () =>
{
    try
    {
        while (!source.IsCancellationRequested)
        {
            // Here we fetch new data
            var updates = dataPipe.GetUpdateEvents();
            foreach (var update in updates)
            {
                Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
                            update.Value.Attribute.GetPath(),
                            update.Value.Value,
                            update.Value.Timestamp.ToString());
            }
            await Task.Delay(500);
        }
    }
    catch (Exception exception)
    {
        Console.WriteLine("Server sent an error: {0}", exception.Message);
    }
}, source.Token);
Console.ReadKey();
source.Cancel();
dataPipe.RemoveSignups(attributes);
dataPipe.Dispose();
Console.WriteLine("Finished");

 

After several hours running the application, I noticed that the GetUpdateEvents was falling behind and sometimes it was leaving some data for the next iteration. This is not a problem per se as, eventually, it would catch up with current data. I suspected that this would happen, but I decided to investigate what was going on. After some bit twiddling, I noticed something weird. Below we have a chart with the memory used by the application. On the top, we have the one that uses GetObserverEvents. On the bottom the GetUpdateEvents. They both use the same amount of memory but look closely at the number of GC calls executed by the .NET Framework.

 

2018-04-09 11_39_55-ObservableTest (Running) - Microsoft Visual Studio.png

(using GetObserverEvents)

2018-04-09 11_37_51-ObservableTest (Running) - Microsoft Visual Studio.png

(Using GetUpdateEvents)

 

Conclusion

Amazingly, this is expected as we are running the code on a server with a limited amount of memory and GetUpdates has extra code to deal with. Honestly, I was expected an increased memory usage and the GC kicking in like this was a surprise. Ultimately, the .NET framework is trying to save my bad code by constantly freeing resources back to the system.

 

Can this be fixed? Absolutely, but it is a waste of time as you could use this effort to implement the observer pattern (that handles all of this natively) and get some extra benefits:

  • Because it allows you to decouple your event handling from the code that is responsible for fetching the new data.
  • Because it is OOP and easier to encapsulate.
  • Because it is easier to control the flow.

 

Observer Pattern Implementation for AF SDK

In this GitHub file, you can find the full implementation of a reusable generic class that listens to AF attributes and executes a callback when new data arrives. It's very simple, efficient and has a minimal memory footprint. Let’s break down the most important aspects of it so I can explain what’s going on and show how it works.

 

The class starts by implementing the IObserver interface. This allows it to subscribe itself to receive notifications of incoming data. I also implement IDisposable because the observer pattern can cause memory leaks when you fail to explicitly unsubscribe to observers. This is known as the lapsed listener problem and it is a very common cause of memory issues:

 

public class AttributeListener : IObserver<AFDataPipeEvent>, IDisposable

 

Then comes our constructor:

 

public AttributeListener(List<AFAttribute> attributes, Action<AFDataPipeEvent> onNewData, Action<Exception> onError, Action onFinish)
{
      _dataPipe = new AFDataPipe();
     _dataPipe.Subscribe(this);
}

 

Here I expect some controversy. First, because we are moving the subject to inside the observer and breaking the traditional structure of the pattern. Secondly, by using Action callbacks I’m going against the Event Pattern that Microsoft has been using since the first version of the .NET framework and has a lot of fans. It's a matter of preference and there are no performance differences. I personally don’t like events because they are too verbose and we usually don't remove the accessor (ie: implement a -=) and that can cause memory leaks. By the way, I’m not alone on this preference for reactive design as even the folks from Redmond think that reactive code is more suitable for the observer pattern. The takeaway here is how we subscribe the class to the AFDataPipe while keeping the data handling oblivious to it, giving us maximum encapsulation and modularity.

 

Now comes the important stuff, the code that does the polling:

 

public void StartListening()
{
    if (Attributes.Count > 0)
    {
        _dataPipe.AddSignups(Attributes);
        Task.Run(async () =>
        {
        while (!_source.IsCancellationRequested)
            {
            _dataPipe.GetObserverEvents();
            await Task.Delay(500);
            }
        }, _source.Token);
    }
}

 

There is not much to talk about this code.  It starts a background thread with a cancellable loop that polls new data every 500 milliseconds. The await operator (together with the async modifier) allows our anonymous function to run fully asynchronous. Additionally, note how the cancellation token is used twice: as a regular token for the thread created by Task.Run(), but also as a loop breaker, ensuring that there will be no more calls to the server. To see how the cancelation is handled, give a look at the StopListening method of the class.

 

When a new DataPipeEvent arrives, the AF SDK calls the OnNext method of the IObserver. In our case it’s a simple code that only executes the callback provided to the constructor:

 

public void OnNext(AFDataPipeEvent pipeEvent)
{
     _onNewDataCallBack?.Invoke(pipeEvent);
}

 

Caveat lector: This is an oversimplified version of the actual implementation. In the final version of the class , the IObserver implementations are actually piping data to a BufferBlock that fires your Action whenever a new AFDataPipeEvent comes in. I'm using a producer-consumer pattern based on Microsoft's Dataflow library.

 

Finally, here’s an example of how this class should be used. The full code is available in this GitHub repo:

 

static void Main(string[] args)
{
    // We start by getting the database that we want to get data from
    AFDatabase database = (new PISystems())["MySystem"].Databases["MyDB"];
    // Defining our callbacks
    void newDataCallback(AFDataPipeEvent pipeEvent)
    {
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine("{0}, Value {1}, TimeStamp: {2}",
        pipeEvent.Value.Attribute.GetPath(), pipeEvent.Value.Value, pipeEvent.Value.Timestamp.ToString());
    };
    void errorCallback(Exception exception) => Console.WriteLine("Server sent an error: {0}", exception.Message);
    void finishCallback() => Console.WriteLine("Finished");
    // Then we search for the attributes that we want
    IEnumerable<AFAttribute> attributes = null;
    using (AFAttributeSearch attributeSearch =
        new AFAttributeSearch(database, "ROPSearch", @"Element:{ Name:'Rig*' } Name:'ROP'"))
    {
        attributeSearch.CacheTimeout = TimeSpan.FromMinutes(10);
        attributes = attributeSearch.FindAttributes();
    }
    // We proceed by creating our listener
    var listener = new AttributeListener(attributes.Take(10).ToList(), finishCallback);
    listener.StartListening();
    // Now we inform the user that a key press cancels everything
    Console.WriteLine("Press any key to quit");
    // Now we consume new data arriving
    listener.ConsumeDataAsync(newDataCallback, errorCallback);
    // Then we wait for an user key press to end it all
    Console.ReadKey();
    // User pressed a key, let's stop everything
    listener.StopListening();
}

 

Simple and straightforward. I hope you like it. And please, let me know whether you agree or disagree with me. This is an important topic and everybody benefits from this discussion!

 

UPDATE: Following David's comments, I updated the class to offer both async and sync data handling. Give a look at my final code to see how to use the two methods. Keep in mind that the sync version will run on the main thread and block it, so I strongly suggest you use the async call ConsumeDataAsync(). If you need to update your GUI from a separated thread, use Control.Invoke.

 

Related posts

Barry Shang's post on Reactive extensions for AF SDK

Patrice Thivierge 's post on how to use DataPipes

Marcos Loeff's post on observer pattern with PI Web API channels

David Moler's comment on GetObserverEvents performance

Barry Shang's post on async observer calls

Are you trying to capture data in the PI System from a new data source?

Do you have picture or video data that needs to be analyzed and stored into PI?

Are you interested in how you can get your IoT data into PI? Are you interested in edge or cloud data storage?

 

Check out the "Learn How to Leverage OSIsoft Message Format (OMF) to Handle IoT Data Streams" hands-on lab at PI World SF 2018:

 

This lab, Learn How to Leverage OSIsoft Message Format (OMF) to Handle IoT Data Streams, was created by the OSIsoft Applied Research team based on a couple different projects undertaken by the research team in the last year.  For the lab, we will explore what it takes to get simulated sensor data, weather data, and picture classification data using machine learning into OMF and send that to the PI System, OSIsoft Cloud Services and Edge Data Store.  This lab is offered on Thursday (Day 3) afternoon of PI World and during it, we will do some basic programming in Python and Javascript, with the configuration in Node-RED.  If any of these topics – OMF, PI Connector Relay, OSIsoft Cloud Services, Edge Data Store – sound interesting, come learn more about it in this lab: Learn How to Leverage OSIsoft Message Format (OMF) to Handle IoT Data Streams. 

 

To learn more about the hands-on labs at PI World SF 2018, visit https://piworld.osisoft.com/us2018/sessions

If you are interested in signing up for a lab, please do so as part of registration for the conference. If you have already registered and would like to add a lab, please email piworld@osisoft.com.

Filter Blog

By date: By tag: