rdavin

Aggregating Event Frame Data Part 5 of 9 - Lightweight FindObjectFields

Blog Post created by rdavin Employee on May 11, 2017

The Advanced AF SDK lab at UC SF 2017 was on this very topic.  The material in this 9-part series follows much of that lab which showcases AFEventFrameSearch methods new to PI AF SDK 2.9.

 

Blog Series: Aggregating Event Frame Data

Part 1 - Introduction

Part 2 - Let's Start at the End

Part 3 - Setting up the App

Part 4 - Classical FindEventFrames

Part 5 - Lightweight FindObjectFields

Part 6 - Summary per Model

Part 7 - GroupedSummary per Manufacturer

Part 8 - Compound AFSummaryRequest

Part 9 - Conclusion

 

Welcome to the Halfway Point

We have reached the halfway point in this series.  I want to thank you for sticking with it.  Both of you.  In this part will be look into the brand new FindObjectFields which is very lightweight.  You may want to keep a fire extinguisher handy because this method is blazing fast (when compared to FindEventFrames).  Caveat: you only see performance benefits if you've called CapturedValues() on the event frames in question.

 

I don't want to sway your opinion but let me say that this became my favorite new method in PI AF SDK 2.9.  Besides being lightweight and super fast, it does return detail rows so it has flexibility for so many other applications, not just aggregation.  FindObjectFields has 3 different overloads.  We only covered 2 in the lab, but we will cover all 3 here.

 

Besides the sheer lightweightness of FindObjectFields when compared to FindEventFrames, there is another big, BIG difference.  The one call to FindObjectFields returns the data in the same call.  No need for a separate GetValues() call.  No need for custom chunking.

 

Plain Overload

This overload wasn't covered in the UC 2017 lab.  Essentially the values all come over as object so one of the first steps you will undertake is most likely casting to its underlying specific type.  And if that type is want you want to ultimately work with, then you will have to perform an additional cast or convert.

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "FindObjectFields Example 1", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //The order of these fields determines the order of returned values in IList<object>
        var fields = "|Manufacturer |Model Duration";

        //returns IEnumerable<IList<object>> where each IEnumerable
        //represents one event frame, and the IList<object> contains 
        //values from each of the specified fields.  From our example,
        //we will have 3 values returned per event frame.
        var records = search.FindObjectFields(fields, pageSize: 10000);

        foreach (var record in records)
        {
            //Read data AND cast appropriately
            var mfr = ((AFValue)record[0]).Value.ToString();
            var model = ((AFValue)record[1]).Value.ToString();
            var duration = ((AFTimeSpan)record[2]).ToTimeSpan();

            //Summary overload is for TimeSpan
            summary.AddToSummary(mfr, model, duration, 1);
        }
    }
}

 

In Line 10, we specify the order of desired fields in a (mostly) blank delimited string.  If you have an attribute path that contains an embedded blank, you would need to wrap the path in single quotes, so they would serve as a delimiter as well.  As would double quotes, but that's a topic for another day.  Internally, FIndObjectFields will parse this string into its own IList<string>, something like { "|Manufacturer", "|Model", "Duration" }.

 

What's returned by the FindObjectFields is IEnumerable where each iteration of the IEnumerable is a different event frame.  The report of Part 2 shows we have over 23000 event frames, so we would expect 23000 records to enumerate over.  The payload of each iteration (or event frame or record) is an IList<object> where each indexed item is a value for the associated field.  In our example, index[0] is the Manufacturer value, index[1] is the Model, and index[2] is the Duration.

 

Since each record comes back as IList<object>, it would be your duty as developer to cast each object to its underlying data type.  First and foremost, any attribute will be returned as an AFValue and therefore must be cast into an AFValue before you can do anything else with it.  Any value from a property on the event frame will be returned with the same data type as the property.  Since Duration is an AFTimeSpan, I must first cast to AFTimeSpan, which I can then convert to TimeSpan if so desired.

 

A word about pageSize here ... with FindEventFrames the larger the pageSize the larger the memory needed to hold all the objects.  However, since this is lightweight, and maybe 10X smaller than the heavy objects from FindEventFrames, using a pageSize of 10000 with FindObjectFields still uses a comparable amount of memory as FindEventFrames(pageSize: 1000).

 

Auto-Mapped DTO Class

The next overloads will use DTO (Data Transfer Object) classes.  This particular overload will Auto-Map field names into your DTO class.  The help file for 2.9 shows this quite well.

 

A DTO is a simple class that will be our lightweight data container.  You could also use a POCO (Plain Old Class Object) but the more definitions you put in your container class the less lightweight it becomes.  I don't want to get into an argument over the subtle differences between DTO and POCO because such academic arguments are as enjoyable as chewing on tin foil.  You are invited to research on the web to learn more.  I include 2 links below.

 

P of EAA: Data Transfer Object

c# - POCO vs DTO - Stack Overflow

 

Before we can use the overload, we must first define our DTO class.  There are a couple of critical rules to apply:

  1. Any type for an attribute value must be an AFValue.
  2. The type for an event frame property must exactly match the type on the event frame.
  3. If the name of the entity on the event frame does not contain a blank, you may keep the original name.
  4. If you wish to change the name in your DTO class, you will use the AFSearch.ObjectField decorator.
  5. Since all attribute paths must begin with "|", which is not allowed in class field or property names, you must use the AFSearch.ObjectField decorator to provide a mapping to your DTO property.
  6. Your DTO may declare your objects as fields or properties.  The example below uses my own personal preference (properties).

 

    public class LightweightAutomapDto
    {
        // Field mapped using default name.
        public AFTimeSpan Duration { get; set; }

        // Attribute value mapped to property using 'ObjectField' attribute.
        [AFSearch.ObjectField("|Manufacturer")]
        public AFValue Manufacturer { get; set; }

        // Attribute value mapped to property using 'ObjectField' attribute.
        [AFSearch.ObjectField("|Model")]
        public AFValue Model { get; set; }
    }

 

Based on the mapping rules above, you will note:

  • I am using the name "Duration" exactly as it is named on the event frame.
  • The data type for my "Duration" is AFTimeSpan because that's what the event frame's Duration is.
  • Both my attributes must provide a ObjectField mapping.
  • Both my attributes have a data type of AFValue.

 

How would that look like in code?

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "Automap DTO Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        var records = search.FindObjectFields<LightweightAutomapDto>(pageSize: 10000);

        foreach (var record in records)
        {
            //Read data from 1 event frame via the current DTO container
            var mfr = record.Manufacturer?.Value.ToString();
            var model = record.Model?.Value.ToString();

            //Summary overload is for TimeSpan
            summary.AddToSummary(mfr, model, record.Duration.ToTimeSpan(), 1);
        }
    }
}

 

 

DTO with Custom Factory

The 3rd overload is quite interesting and was almost left out of the UC lab for fear of course length.  I am glad I included it because it soon became my favorite of the overloads.  If you choose to use this overload, then you absolutely MUST provide your own custom factory to perform the transfer of data.

 

Why would you want to do use this?  Let's consider the Auto-mapped DTO and what I would like to try differently:

  • The attribute values must be AFValue but I want Manufacturer and Model to be strings.
  • I want my Duration property to be a TimeSpan instead of the AFTimeSpan as it is on the event frame.

 

The resulting DTO class is far, far simpler and laid out exactly like I want it.  We don't have to worry about AFSearch.FindObject fields because our custom factory will take care of transfer.

 

public class LightweightDtoForCustomFactory
{
    public TimeSpan Duration { get; set; }
    public string Manufacturer { get; set; }
    public string Model { get; set; }
}

 

And here is how it would be used:

 

public void GetSummaryByMfrAndModel(StatsTracker summary, AFDatabase database, IList<AFSearchToken> tokens)
{
    //Starting with AF 2.9, AFSearch implements IDisposable
    using (var search = new AFEventFrameSearch(database, "DTO For Custom Factory Example", tokens))
    {
        //Opt-in to server side caching
        search.CacheTimeout = TimeSpan.FromMinutes(5);

        //The order of these fields determines the order of returned values in IList<object>
        var fields = "|Manufacturer |Model Duration";

        //Define your custom factory 
        Func<IList<object>, LightweightDtoForCustomFactory> factory = (values) =>
        {
            var dto = new LightweightDtoForCustomFactory();
            dto.Manufacturer = ((AFValue)values[0]).ToString();
            dto.Model = ((AFValue)values[1]).ToString();
            dto.Duration = ((AFTimeSpan)values[2]).ToTimeSpan();
            return dto;
        };

        var records = search.FindObjectFields<LightweightDtoForCustomFactory>(fields, factory, pageSize: 10000);

        //The loop is a bit simplified because the transfer logic is contained in the function delegate above.
        foreach (var record in records)
        {
            //Note that Manufacturer and Model are now validated strings thanks to factory.

            //Summary overload is for TimeSpan
            summary.AddToSummary(record.Manufacturer, record.Model, record.Duration, 1);
        }
    }
}

 

You are invited to review the code for each of the 3 overloads to look for similarities and differences.  The one big similarity is that each is concerned with casting the object value to its underlying type and then converting that to a desired type.  With that in mind, all 3 overloads offer identical performance so your preference of one over the other is purely your personal preference.

 

Great For Detail Reporting

As mentioned many times earlier, FindObjectFields is not limited to performing aggregations.  It's also quite handy for detail reporting too.  If you are going to use it for detail reporting, the strongest suggestion I can give you is to be sure to include the event frame's ID in your DTO class.  That way you have the ability to quickly find and load a specific event frame if ever needed.

 

Metrics Comparison (from Part 2)

The numbers below are from a 2-core VM using Release x64 Mode.  The smaller values are better.  Caution that we sometimes have a difference in UOM between MB and KB, but I will bold KB when needed.

 

Resource Usage:

Values displayed are in MB unless noted otherwise

Method

Total GC Memory (MB)

Working Set Memory (MB)Network Bytes Sent
Network Bytes Received
FindEventFrames145.48257.089.13 MB190.08 MB
FindObjectFields1.2865.555.00 KB3.68 MB
Summary2.5455.358.58 KB261.81 KB
GroupedSummary9.8664.286.24 KB1.98 MB
AFSummaryRequest7.2965.365.00 KB3.68 MB

 

Performance:

MethodClient RPC CallsClient Duration (ms)Server RPC CallsServer Duration (ms)Elapsed Time
FindEventFrames12063337.011039118.102:27.8
FindObjectFields105360.8114547.600:06.0
Summary159484.6169310.900:10.1
GroupedSummary125527.2134938.500:06.2
AFSummaryRequest102992.2102222.200:03.7

 

 

Next Up: A Real Summary Method

In Part 6 we will cover the first bona fide aggregation method, AFEventFrameSearch.Summary.  I forgive you if it takes a several days or a week for you to move onto Part 6.  If you are anything like me, once you saw code for FindObjectFields the gears in your head must have started spinning overtime as you became preoccupied thinking of every app you've ever written that could have benefitted from a faster lightweight method.  If that's the case, Part 6 can wait.  You should by all means roll up your sleeves and get busy testing out FindObjectFieldsPart 6 will be here when you get back.

Outcomes