Skip navigation
All Places > PI Developers Club > Blog > Authors Rick Davin
1 2 3 Previous Next

PI Developers Club

37 Posts authored by: Rick Davin Employee

Great list, though the author admits the list is more exhausting than it is exhaustive.  A must-read for VB.NET developers.


An Exhausting List of Differences Between VB.NET & C# | Anthony's blog

Below is a link to the presentation given at Barcelona on September 27, 2018 for the LiveCoding session titled "Getting the Most Out of AFSearch".


Link to Recording:     LiveCoding--Getting-the-Most-Out-of-the-AFSearch


GitHub Repository:     AF-SDK -PIWorld-EMEA-2018-AFSearch-LiveCoding


While some of it is a repeat of what was presented at San Francisco in April, the beginning section features new material on the upcoming AF SDK 2.10.5 release.  The new features discussed:


  • OR clauses in searches.
  • FindObjects replaces the now deprecated FindElements, FindEventFrames, FindAttributes, etc.
  • AFSearchToken structure being replaced by new AFSearchBaseToken abstract class with 4 concrete classes: AFSearchFilterToken, AFSearchQueryToken, AFSearchOrToken, and AFSearchExpressionToken.
  • AFSearchToken will be mapped to new AFSearchBaseToken, if possible.
  • If your query uses an OR, the AFSearch.Tokens property will throw an exception.
  • New AFSearch.TokensCollection property is meant to replace Tokens functionality and works with new AFSearchBaseToken.
  • AFSearchToken and AFSearch.Tokens property are both marked as deprecated.

ATTENTION Developers, Data Scientists, IT CyberSecurity, and Power Users.  For lack of a better word, I will refer to you collectively as "developers".


Developers coming to PI World Barcelona may notice more offerings than ever before.  We proudly proclaim that Barcelona will have the most robust Developer Agenda (see link) ever seen at UC or PI World EMEA.  This includes the Developer Innovation Hackathon on Day 0 (see link), thanks to our data sponsor DEME.


Besides the traditional hands-on labs, which require an additional fee and pre-registration, the Day 3 agenda is chock full of technical content aimed specifically at developers, thanks to 90-minute Live Coding or How-To sessions.  These in-depth technical talks do not require a fee nor pre-registration.  You are free to come and go as you please.  Make no mistake about it ... just because we call the LiveCoding "talks" does not mean they contain less technical information than labs. We expect to offer even more such talks at future PI World events because we think it is better event for you.  Our reasoning: you can attend 2 labs on Day 3 for $300, or you can sit in on 4 LiveCoding talks for free.  Who can argue against more technical training for less cost?  (Tip: if you are trying to convince your boss to send you to PI World, presenting it as a major training event (which it is) could be a strong justification to attend.)


I invite you once again to review the Day 3 agenda.  You will see an Analytics Track and Developer Track.  One late correction I would like to make is the PI Admin Track, which is not really for PI Admins but should be considered a 2nd Developer Track.

AFSearch Barcelona.png


Day 3, Thursday September 27, 11:30 - 13:00

LiveCoding: Getting the Most Out of the New AFSearch

CCIB: Room 117 134, P1 Level


To any members of the PI Developer Community who will be at PI World Barcelona, you are invited to join me in a presentation on new features of AFSearch.  If you ask "Hey Rick, didn't you give this already in SF?", my answer would be "Yes BUT new sections were added to specifically cover some important NEW stuff."  PI AF 2018 R2 (AF SDK 2.10.5) will finally support OR conditions with AFSearch.  That is a highly anticipated new feature that many are looking forward to.  But in order to support OR conditions, it required replacing the older AFSearchToken structure with a new AFSearchTokenBase class that now has 4 different token instances.  Trust me, you will want to see how these new tokens will be used in code.  Everyone who has ever attended this talk has said they definitely learned something!


UPDATE: The room has been moved to 134 on Level P1.

PI AF 2018 (AF SDK 2.10) offers a very significant change in filtering on attribute values: there is no longer a restriction that the attribute must belong to an element template.  The allows for a greater flexibility for filtering.  For example, you may now search for attributes that don't belong to any template.  Or better yet, you may search for attributes with the same name but belonging to different templates!


Other Features New to AF SDK 2.10


The bulk of this blog will cover the ValueType used with attribute value filters.  Before we dig too deep into that topic, let's take a quick look at the other new features.


There is a new AFSearchFilter.ID search filter to allow searching for objects by their unique identifier (GUID).  This unique ID is the fastest way to locate a given object on the AF Server.  A much-welcomed addition is that the ID supports the Equal or IN( ) operator.  If you are developing code, the best way to pass a GUID is by using ID.ToString("B").


New Search Fields are ConfigString, DisplayDigits, IsManualDataEntry, Trait, UOM, SecurityString, and SecurityToken.  Note that with a SecurityToken field, the FindObjectFields method would return an AFSecurityRightsToken object.


PI AF 2017 R2 (AF SDK 2.9.5) introduced the AFAttributeSearch and the PlugIn filter.  You could combine that filter plus the new ability to search on attributes without specifying a template.  For example, you now have the ability to perform a completely server-side search of all attributes referencing the PI Point data reference!  Stay tuned for a blog dedicated to this topic from one of my colleagues.


And now, the remainder of the blog will discuss the new ValueType.


If using an element or event frame Template


As mentioned earlier, previous versions required a Template for any attribute value filters.  An additional requirement was that the Template needed to be specified before the attribute value filters.  If the Template was specified after, then an exception was thrown.


Since AF SDK 2.10 removes the restriction on the Template, an interesting artifact is that you may specify the Template after the attribute value filter - and an exception will not be thrown.  However, we strongly recommend against this practice.  If you want to filter on a Template, we highly recommend you specify the Template first - just as you did with AF SDK prior to 2.10.  Nothing has changed with the new version here (nor should your existing code).  If you follow this advice, then you should skip the "AS valuetype" cast (more below).


Now let's consider if you don't follow the advice and you specify the Template after the attribute value filters.  You will need to include the "AS valuetype" cast, and its behavior will be as described in the remainder of this document.  The search will still be limited to the specified Template but as far as the attribute value filters are concerned, they will be treated as if the template was entirely omitted.  Precisely how they are treated depends on the attribute's data type and the "AS valuetype" cast you declare, which is presented in detail below.


Casting AS ValueType - When not using a Template

(or the Template is specified after the attribute value filters)


To support this new capability, there are several new things to discover with AFSearch to address the issue of a filter attribute's data type.  When based on a template, the data type is easily inferred.  What happens if a template isn't specified and the attribute does not belong to a template and/or may span different templates?  How does the search know which data type to use in the filter?  The answer is that it is left up to you (the developer) to pass the desired value type as you build the search, either by a query string or search tokens:


  • New AFSearchValueType enumeration
  • A new AFSearchToken.ValueType property (a string)
  • Two new AFSearchToken constructors to allow you to indicate the ValueType (also a string)


Golden Rule:

  • If you DO specify the template, do NOT specify the value type. 
  • If you do NOT specify a template, then you SHOULD specify a value type.


How carved in stone is the above "SHOULD"?  If there is any possibility whatsoever of an ambiguous interpretation between a String versus a Numeric, then you absolutely should specify the value type.  For example,  AFSearch has no way of knowing whether 1 or '1' or "1" should be a Numeric versus a String value.  Best practice: anything numeric should always specify "AS Numeric".


If you are using search tokens, you would use the new AFSearchToken constructors.  If you are using a query string, you would use the new AS <value type> syntax.  The available values for value type are:

  • Numeric, i.e. the literal text "Numeric"
  • String, i.e. the literal text "String"
  • EnumerationSet, the name of the applicable AFEnumerationSet.  Do NOT use the literal text "EnumerationSet".


Typical Scenarios with Numeric or String


The brief examples below look very similar with the exception of the value type designator (Numeric or String).  This bears repeating: you only need to use the AS valuetype if you do not specify a template.


Data Type Numeric: Integer (Byte, Int16, Int32, Int64) or Floating Point (Float, Single, or Double)

Consider if you have an attribute named RunStatus, and its data type is an Int32, where 0 means not running and 1 means running.  The query string could look like either of these:

  • "|RunStatus:1 AS Numeric"
  • "|RunStatus:'1' AS Numeric"


Data Type String

And if RunStatus was a String where "0" means not running and "1" means running, you would use these:

  • "|RunStatus:1 AS String"
  • "|RunStatus:'1' AS String"


Typical Scenarios with EnumerationSet


We continue covering scenarios when you do not specify a template.  We turn to another typical scenario of when the attribute's data type is an Enumeration Set.  Here it doesn't matter where the data comes from, be it a PI point, a table lookup, formula, or even a static value.  What matters to the value being filtered on is that the attribute has been declared to use an enumeration set.  The important thing is to use the AS specifier followed by the name of the enumeration set; do NOT use the literal value "EnumerationSet".


string attrPath = "|Classification";
string attrValue = "Commercial";
string enumSetName = "Building Type";
string query = $"'{attrPath}':='{attrValue}' AS '{enumSetName}'";


The above snippet safely wraps anything I created in quotes.  Note that because the enumeration set name contains a blank, I absolutely must wrap it in quotes.  In this specific case where neither attrPath or attrValue contain a blank, I could have omitted the quotes.  The value in the variable query will be:


"'|Classification':='Commercial' AS 'Building Type'"


Later when passed into an AFAttributeSearch constructor, the search instance will resolve to:


{|Classification:Commercial AS "Building Type"}



Cases Needing Special Consideration


There are special cases you may need to keep in mind beyond the typical Numeric, String, or EnumerationSet.  Obviously an attribute that is a string should use "AS String" and an attribute that has a number data type should use "AS Numeric".  What about data types that aren't so obvious?  Boolean or DateTime, for example?


Data Type Boolean: cast AS String

Before we even touch on enumeration sets, let's investigate another area of caution.  An attribute with a data type was Boolean is not exactly a number and not exactly a string.  As far as AFSearch is concerned, you should compare the literal values of a Boolean, namely "True" and "False", as strings.  Therefore the following filters would all be correct for a Boolean data type:

  • "|RunStatus:True AS String"
  • "|RunStatus:'True' AS String"


It's absolutely important with Booleans to specify "AS String".  You need to be aware that the following will quietly fail by returning 0 items:

  • "|RunStatus:True"


Data Type DateTime: cast AS Numeric

This can be a bit tricky.  The safest practice when dealing with DateTime attributes, whether you have a DateTime or an AFTime instance in code, is to use Round Trip Formatting.  That is to say, use ToString("O") when converting a DateTime or AFTime to string for the AFSearchToken constructor or within a query string.  And despite passing it as a string, it will actually be treated AS Numeric.  So these snippets work:


Variable "date" can either be DateTime or AFTime instance

  • $"|Timestamp:>'{date.ToString("o")}'"
  • $"|Timestamp:>'{date.ToString("o")}' AS Numeric"


The above uses an Interpolated String.  If you prefer string.Format, it would be:

  • string.Format("|Timestamp:>'{0}'", date.ToString("o"))
  • string.Format("|Timestamp:>'{0}' AS Numeric", date.ToString("o"))


TimeSpan with Data Type Anything: Not Supported

Some customers have attributes with data type "<Anything>" to hold a TimeSpan object.  Value filtering on such time span attributes is not supported in AF SDK 2.10.


If your time span attribute is defined to hold an integer or a floating point, then it would be treated as a Numeric data type (see above).


Digital PIPoint with Data Type Anything (not using an AFEnumerationSet): cast AS String or Omit AS specifier

For an attribute using the PI Point data reference that grabs from a Digital tag, we recommend that the attribute data type be "<Anything>".  You do not have to map the digital tag to an AFEnumerationSet.  You can if you wanted to, but that means (1) you have to copy the PIStateSet as an AFEnumerationSet, and (2) what to do with the attribute falls under "AS EnumerationSet" discussed elsewhere in this document.


Assuming you have a string variable named  "stateText", which contains the text of a given digital state, the following would be used to filter:

  • $"|Digital:'{stateText}'"
  • $"|Digital:'{stateText}' AS String"


For instance, if your digital tag used the "Modes" StateSet and you wanted to filter on those attributes with a mode of "Manual", either of these should suffice:

  • "|Digital:'Manual'"
  • "|Digital:'Manual' AS String"


Past AFSearch Blogs


What's new in AFSearch 2.9.5 (PI AF 2017 R2)  (March 2018) - AFAttributeSearch is introduced.


Coding Tips for using the AFSearch IN Operator - you may search for multiple values using IN(value1; value2; etc.).  Some code is offered to make this easier to generate.


Using the AFEventFrameSearch Class (Nov 2016) - Giving some attention to event frames since many of earliest AFSearch examples were element based.


Aggregating Event Frame Data Part 1 of 9 - Introduction  (May 2017) - From the UC 2017 TechCon/DevCon hands on lab for Advanced Developers.


PI World 2018 SF Developer Track - Getting the most out of AFSearch - (May 2018) From PI World 2018, DevCon presentation for Intermediate Developers.


Why you should use the new AF SDK Search  (June 2016) - The granddaddy of them all.  The earliest post explaining AFSearch, which was new at that time.

A customer recently asked about filtering on multiple values of an attribute in AFSearch.  This is easily addressed using the IN search operator in a query string, or the equivalent AFSearchToken constructor expecting an array of values.  There is one major caveat: the IN operator is not allowed for floating point types such as Single or Double, since binary floating point values are considered approximations instead of exact numbers.


There are a few tips to get developers pointed in the right direction.  First of all, values within the IN operator are delimited by semi-colons.  If you (like me) accidentally use a comma-delimited list, then you will receive an odd error about missing a closing parenthesis.  Then if you carefully inspect the query string, then you (like me) will start pulling your hair out because you don’t see a missing closing parenthesis!  Hopefully you may benefit from my pain by quickly switching your delimiter to be a semi-colon.


Another tip is that while blanks are allowed within the IN operator - or more specifically around the delimiters - any values that contain embedded blanks must be wrapped in quotes (single or double quotes).  Consider this list of cities:


  • London
  • Paris
  • San Francisco


Obviously, San Francisco needs to be wrapped in quotes due to the blank inside the name.  If the attribute path is “|City”, then you could have this C# filter in your query string:


string uglyQuery = "|City:IN(London; Paris; \"San Francisco\")";


Though ... that is a wee bit ugly.  Besides being ugly, it also might be confusing to someone using a different language, for instance, VB.NET.  A couple of weeks ago, I taught a class where a VB coder thought the unsightly \" were delimiters for AFSearch.   I explained to him that it's how C# escapes double quotes.  This can look much easier on the eyes if you just use single quotes:


string prettyQuery = "|City:IN(London; Paris; 'San Francisco')";


And it avoids any needless confusion that \" is an AFSearch delimiter.


This is all fine-and-dandy, but how often do you hard-code values?  It does make the example simple and straight-forward, but what if the list of items you wish to filter upon are in a list or array?  Let’s look at some code that helps us out.  After all, this is a PI Developers Club, which demands for code.  Strictly speaking, all we need for the collection of values is that they be in an enumerable collection.  Your source collection doesn’t have to be strings, but keep in mind that eventually we will need to pass a string to the AFSearch query, or if we use the AFSearchToken, then we must actually pass an array of strings.


For our cities example, it makes sense that the city names are strings.  Let’s not take the easy way out and make it an array of strings.  Instead we will use a list so that we can see what we must do differently to make it all work together:


List<string> cities = new List<string>() { "London", "Paris", "San Francisco" };


A tiny bit of  code is needed to put that inside a query string:


// A little LINQ to wrap single quotes around each city, and separate them with "; "
string delimitedCities = string.Join("; ", cities.Select(x => $"'{x}'"));

// An Interpolated String to substitute the delimitedCities.
string query = $"|City:IN({delimitedCities})";


This resulting value in query would be:


"|City:IN('London'; 'Paris'; 'San Francisco')"


Or if you prefer working with search tokens, this would be the equivalent:


// Tip: here you must pass a string array, so we must ToArray() on the cities list.
AFSearchToken token = new AFSearchToken(AFSearchFilter.Value, AFSearchOperator.In, cities.ToArray(), "|City");


If we had defined cities to be a string array, we would not need the ToArray(), but then this example would be boring and less educational.


What if our enumerable collection isn’t a bunch of strings?  Let’s say we have a bunch of GUID ’s.  (Exactly how you got these GUID's is an interesting question not addressed here ; suffice to say this example takes a collection of things that aren't strings and converts them to one that is.)  There would now be an extra step needed where we must convert to string.  Once we have a collection of strings we can then implement code similar to the previous examples.  Let’s imagine we have something like this:


IEnumerable<Guid> ids = GetListOfGuids();  // magically get a list of GUID


Or it could maybe have been:


IList<Guid> ids = GetListOfGuids();


Let's say we want to filter on an attribute path of “|ReferenceID”.  First let’s tackle the problem of converting a GUID into a string that is compatible with AFSearch.  This is easy enough thanks to LINQ:


// Nicest way to convert a GUID a string compatible with AFSearch is to use GUID.ToString("B").
IEnumerable<string> idStrings = ids.Select(x => x.ToString("B"));


Okay, so now we have an enumerable collection of strings.  Using what we learned in previous examples, we can knock this out:


// A little LINQ to wrap single quotes around each string item, and separate them with "; "
string delimitedIds = string.Join("; ", idStrings.Select(x => $"'{x}'"));

// An Interpolated String to substitute the items.
string query = $"|ReferenceID:IN({delimitedIds})";


Fantastic.  If you prefer AFSearchTokens, that’s easy enough as well, but we do require the idStrings to generate a string array.


// Tip: here you must pass a string array, so we ToArray() on the idStrings collection.
AFSearchToken token = new AFSearchToken(AFSearchFilter.Value, AFSearchOperator.In, idStrings.ToArray(), "|ReferenceID");


Granted our example would have been simplified if we defined idStrings to be an array in the first place, but what fun would there be in that?



VB.NET Examples


Some of us supporting the PI Developers Club think there should be more VB.NET examples.  Towards that end, here are code snippets for the VB coders out there:


    Dim uglyQuery As String = "|City:IN(London; Paris; ""San Francisco"")"

    Dim prettyQuery As String = "|City:IN(London; Paris; 'San Francisco')"

    Dim cities As List(Of String) = New List(Of String) From {"London", "Paris", "San Francisco"}


Cities Query Example:


Query String

        ' A little LINQ to wrap single quotes around each city, and separate them with "; "
        Dim delimitedCities As String = String.Join("; ", cities.Select(Function(x) $"'{x}'"))

        ' An Interpolated String to substitute the delimitedCities.
        Dim query As String = $"|City:IN({delimitedCities})"



' Tip: here you must pass a string array, so we ToArray() on the cities list.
Dim token As AFSearchToken = New AFSearchToken(AFSearchFilter.Value, AFSearchOperator.In, cities.ToArray(), "|City")


Guids Example (or something that is not a collection of strings)


Query String

        Dim ids As IEnumerable(Of Guid) = GetListOfGuids()   ' magically get a list of GUID

        ' Nicest way to convert a GUID a string compatible with AFSearch is to use GUID.ToString("B").
        Dim idStrings As IEnumerable(Of String) = ids.Select(Function(x) x.ToString("B"))

        ' A little LINQ to wrap single quotes around each string item, and separate them with "; "
        Dim delimitedIds As String = String.Join("; ", idStrings.Select(Function(x) $"'{x}'"))

        ' An Interpolated String to substitute the items.
        Dim query As String = $"|ReferenceID:IN({delimitedIds})"



        ' Tip: here you must pass a string array, so we ToArray() on the idStrings collection.
        Dim token As AFSearchToken = New AFSearchToken(AFSearchFilter.Value, AFSearchOperator.In, idStrings.ToArray(), "|ReferenceID")




If you want to filter on a variety of values for an attribute, this requires the IN search operator.

  • You may use IN in a query string or an AFSearchToken.
  • Values must be exact.  Therefore, IN does not work on Single or Double floating point data types.
  • In query strings
    • The semi-colon is the delimiter between IN values.  Example: "|City:IN(London; Paris; 'San Francisco')"
    • Values containing blanks must be enclosed in single or double quotes.  Example: "|City:IN(London; Paris; 'San Francisco')"
  • AFSearchToken
    • Values must be passed as strings in an array

I have been asked on many occasions “How can I find the first recorded event for a tag?”  The direct answer to this may be as brief as the question.  However, usually there is a lurking question-behind-the-question about what they really want to do with that first event, and if you dig even slightly deeper you will uncover the overall task they are trying to accomplish.  What you may ultimately discover is there is no need to find the first event.


Let’s start off with the direct, simple answer in the form of a C# extension method:


public static AFValue BeginningOfStream(this PIPoint tag) => tag.RecordedValue(AFTime.MinValue, AFRetrievalMode.After);


This works because the PI Point data reference implements the Rich Data Access (RDA) method of RecordedValue.  The earliest timestamp to query is AFTime.MinValue, that is midnight January 1, 1970 UTC.  Thanks to the AFRetrievalMode, you ask for the first value after January 1, 1970.  If it’s the earliest recorded timestamp you are only concerned with, you can use this extension method:


public static AFTime BeginningTimestamp(this PIPoint tag) => BeginningOfStream(tag).Timestamp;


For PI points, this would give you the BeginningOfStream method to go along with the built-in EndOfStream.  Before the advent of future data, the EndOfStream was simply the Snapshot.  But there are oddities related to future data, which required different handling of data compared to the traditional historical data.  Hence, Snapshot was replaced by CurrentValue, and EndOfStream was added.


An AFAttribute could have a BeginningOfStream method, but it doesn’t have the same nice guarantees of PIPoint.  It all depends upon the data reference being used and whether it supports the RDA Data.RecordedValue method, which is why you should properly check that it is supported before attempting to call it:


public static AFValue BeginningOfStream(this AFAttribute attribute)
    if (attribute.DataReferencePlugIn == null) // static attribute
        return attribute.GetValue();
    if (attribute.SupportedDataMethods.HasFlag(AFDataMethods.RecordedValue))
        // Depends on how well the data reference PlugIn handles AFRetrievalMode.After.
        return attribute.Data.RecordedValue(AFTime.MinValue, AFRetrievalMode.After, desiredUOM: null);
    // Fabricated answer.  Not exact that one is hoping for.  
    return AFValue.CreateSystemStateValue(AFSystemStateCode.NoData, AFTime.MinValue);


Since the value being returned may be fabricated, I would be hesitant to include a BeginningTimestamp method as it would mask the inaccuracies.  To compensate, I would think further inspection of the returned value is needed, i.e. check for “No Data”.  Such are the difficulties of trying to create a BeginningOfStream method within your code libraries.  This is why we begin to probe more and ask about your use-case, or simply “What are you really wanting to do?


Virtually 100% of the people asking me how to find the first value in a stream want to find it for historical PI points only.  This greatly simplifies part of the problem because there is no need to be concerned with attributes or tags with future data.  Which brings us right back to the direct answer at the top, where you may be inclined to stop.  But if you take time to dig just a little deeper into what they are really doing, the true mission is revealed: they want to copy all historical data from an old tag to a new tag.  And trust me, there are several legitimate use-cases for doing this.


When I hear anything about copying of historical data, the first thought I have is “How much data is in the old tag?”  There are two particular answers that require the same special handling: (a) I don’t know, or (b) a lot.


The real problem they need to solve isn’t finding the earliest recorded timestamp.  Rather, they may have so much data they will bump into the ArcMaxCollect limitation (typically 1.5 million data values in a single data request).  There are programming ways around ArcMaxCollect (more below) and they rely upon the PIPoint.RecordedValues method specifying a maxCount > 0 (for instance, 100K works well).  The perceived issue of knowing the earliest timestamp becomes a moot point.  The more important date is knowing the end time, that is the switch-over date from the old tag to the new tag.  Depending upon how the old tag was populated, this may very well be the EndOfStream.  But if there is a chance that the old tag could still be receiving “InterfaceShut” or “IOTimeout” messages, you will need to explicitly specify the end time.  Worrying about the earliest recorded date has been a distraction to solving the real problem.


What of your start time?  I would think an in-house developer should know of the earliest start of their company's archive files.  A contracted developer could use AFTime.MinValue or go with a later, but still much safer date, such as “1/1/1980”.  Which brings us back to what they really want to do: copy large or unknown amounts of data.    This has been blogged about many times before:


Extracting large event counts from the PI Data Archive


PI DataPipe Events Subscription and Data Access Utility using AF SDK - PIEventsNovo


GetLargeRecordedValues - working around ArcMaxCollect

The recording for this Live Coding presentation may be found at: Getting the most out of AFSearch


Associated code may be found on this GitHub repository:

GitHub - Rick-at-OSIsoft/AF-SDK-TechCon-2018-AFSearch-LiveCoding: Presentation also titled "Getting the most out of AFSe…


GitHub also includes a PDF file of the slide deck.  Topics include:


Demo 1: What's new in AF SDK 2.9.5 and 2.10

Demo 2: Breaking it down line-by-line to see what's happening on the client versus the server

Demo 3: Caching versus Not Caching

Demo 4: Accurate counts especially when client-side filtering is required

Demo 5: The perils of LINQ


Along the way we touch upon AFAttributeSearch, using the skinny, lightweight FindObjectFields with AFEventFrames, why caching is so important to performance, and why you need to properly dispose of the AFSearch objects.

On behalf of Engineering's Technology Enablement team, and in conjunction with Technical Support's Developer Technologies team, it is my privilege to announce this year's Community All-Stars.  We give special recognition to these individuals for their prolific and relentless contributions to the PI Community at large.  We thank them for sharing their experiences and knowledge for all things PI, and in particular for any posts that are developer related.  Let me add that each recipient holds near and dear to each of them the concept of "community", and even if we did not hand out such awards, or if we did not hand out any prize other than simple recognition, these individuals would still post and contribute to our PI Square Community with the same dedication for the sheer sake of expanding the knowledge base of the community.


I ask my fellow PI Geeks to help congratulate our deserving 2018 All-Stars!  It's interesting to note that are 3 Community All-Star and one Rising Star are from 4 different continents.


PI All Stars 2018.jpg

Rick Davin (far left), Paurav Joshi, John Messinger, and Dan Fishman.  Not shown: Roger Palmen.



PI Developers Community All-Stars 2018

  • Dan Fishman, Exele Information Systems.  Dan is a first time winner as Community All-Star but won twice before as OSIsoft All-Star.
  • John Messinger, Optimate Pty Ltd.  John wins for the 2nd year in a row.
  • Roger Palmen, CGI.  I don't remember a year when Roger wasn't a winner.


Last year as an experiment, I began following a lot of people.  Anyone who has ever been an All-Star, honorable mention, or just nominated.  I can tell you that my Inbox was slammed all year by answers from Dan, John, and Roger.  Thanks, guys.  I don't mean that sarcastically.  Each "spam" in my Inbox from you is you helping another community member, so absolutely, sincerely THANKS!


PI Developers Club Community Rising Star 2018

We introduced this category last year to recognize contributions and efforts on well deserving contributors that steadily add to our content here but just fall short of the elite upper strata of All-Star.  A big difference between Rising Star and All-Star is a Rising Star may only be awarded once, whereas All-Star may be won year after year (I'm looking at you, Roger).  Anyone who has ever been an All-Star or Rising Star in the past is ineligible for further consideration as a Rising Star.  Last year we handed out 3 of these as a means of playing catch-up to some folks who have been solid contributors over the year.  This year we have only one winner:



Paurav along with Dan, John, and Roger win:

  1. Recognition in the community
  2. One year free subscription or renewal to PI Developers Club
  3. One-time free registration to a 2019 PI World event (SF or EMEA)
  4. A legal waiver for them to sign
  5. An Amazon gift card worth $400 USD granted after submitting the signed legal waiver back to us


And let's not forget there are a lot of fine contributors within OSIsoft.  While it is our job to help customers, we do want to recognize a select few for their efforts the past year.


PI Developers Club OSIsoft All-Stars 2018


Our OSIsoft recipients win:

  1. Recognition in the community
  2. Amazon gift card worth $200 USD


The clock has been reset and everyone's slate is clean

It's never too early to start thinking about 2019 All-Stars.  We monitor the quantity and quality of posts from one PI World SF to the next in order to pick our winners.  The new countdown has begun.  One thing I can tell you that all of our 2018 winners have in common are the following:

  • A high volume of posts
  • Posts of high quality (lots flagged as helpful or marked as answers)
  • A strong desire to help others.


Absolutely, positively candidates must contribute to threads other than their own.  In short, you must show strong participation in the community as a whole.  Without a doubt this year's crop of winners have done just that.



PI World NA is next week and I am quite excited.  Join us the afternoon of Day 3 (Thursday April 26) from 4:30 - 6:00 PM PDT for the first ever Developers Meet-Up Reception, where we will also be handing out a gaggle of awards for this year's crop of Community All-Stars, winners of the Innovation Hackathon, as well as the winners of the recent Visualization Virtual Hackathon!  If you consider yourself a developer, or just wish you were a developer in-training, or are a member of PI Developers Club, or at the very least are attending any DevCon presentations or labs at Parc 55, then by all means stop in for a beer or refreshing drink, have a snack, and mingle with fellow developers.  At 5:00 PM we will begin announcing awards, where you may congratulate the winners or commiserate with the losers other not-so-fortunate participants.  Don't overdo it because at 7:00 PM the party shifts to the Hilton for the always fun PI Geek Night, with more food and drinks, games, and other hijinks.


What:     Developer Meet-Up Reception & Awards

Where:   Parc 55, Cyril Magnin I & Foyer, Level 4

When:    4:30 PM - 6:00 PM (awards start at 5:00 PM)


You may see more at the bottom of this link:  PI World for Developers: PI World - OSIsoft Users Conference 2018 (SF)


Since it is being held in the late afternoon and to a limited segment of PI World attendees, this is not listed as an Evening Event.  However, it does appear on the Day 3 Agenda for Parc 55 hotel under the Sessions Agenda


We hope to see you there!

PI AF 2017 R2 (AF SDK 2.9.5) was released shortly before 2018.  There are some exciting new features with AFSearch that should interest developers.


First off, I would hope any developer would always go to the Live Library What's New page for any major PI AF release.  That page gives a summary of what's new with that particular release, not just for AFSearch but for all namespaces.  Specific to AFSearch namespace, you would see the following:


OSIsoft.AF.Search Namespace

Two new query based search classes have been added in this release: AFAttributeSearch and AFNotificationContactTemplateSearch. The AFSearchToken.AFSearchToken(AFSearchFilter, AFSearchOperator, String, IList<AFSearchToken> ) constructor and the AFSearchToken.Tokens property have also been added to support enhanced nested query filters for searches.

The following new search tokens have been added: EventFrame, IsInternal, Parent, PlugIn, and PlugInName.


As a fellow developer and PI geek, when I first read that my reaction was "Cool! Nested queries!"  If you think about it, an AFAttributeSearch means that attributes are being searched upon some element(s) or event frame(s).  This implies there will first be a search for elements or event frames, followed by a search upon those results for the attributes.  This does not require you to create 2 search objects on the client.  Rather you will have 1 search object, namely an AFAttributeSearch, and it will have a nested query to filter on the desired elements or event frames.  But the nested queries are not limited to AFAttributeSearch.   For example, you may have an AFEventFrameSearch that uses a nested query for its elements.


Each release of AF SDK since 2.8.0 has introduced new features and capabilities to put more efficient searches at your fingertips.  For example, if you were searching on an attribute category to return many attributes per element, you could additionally filter on PlugIn or PlugInName to restrict the returned attributes to be PI points.


Contacts Search Example


Here's a quick example where I search for any contact with a name beginning with "Davi*".


using (AFNotificationContactTemplateSearch search = new AFNotificationContactTemplateSearch(assetServer, "contact demo", " Name:'Davi*' "))
    foreach (AFNotificationContactTemplate contactTemplate in search.FindNotificationContactTemplates(fullLoad: true))
        Console.WriteLine($"   {contactTemplate.Name,-24} {contactTemplate.Contact}");


The output:


   Template                 Contact

   David Burns_Email        David Burns

   David Doll_OCS           David Doll

   David Moler_Email        David Moler



Nested Queries


Let's take a look at 3 different examples that all do the exact same thing.  We want to perform an attribute search for any attributes named "Feed Rate" that belong to any elements whose name starts with "Boiler".  We will perform the same search 3 times, but each time how we setup the search object will be different.  The 3 techniques we will briefly cover are:


  • Using Nested Search Tokens
  • Using Nested Query String
  • Using Interpolated Nested Query String


They example code is kept simple.  We will perform an AFAttributeSearch searching for attributes found within a nested element query with the following filters:

  1. Attribute Category of "Process Monitoring"  (note the blank)
  2. Element Category of "ProcessMonitoring" (does not have a blank)
  3. Element Template of "FCC Pump Process"


If you've ever worked with AFSearch before, then your previous experience should have been that you could not specify Category twice on a query prior to nested queries in 2.9.5


Using Nested Search Tokens


string templateName = "FCC Pump Process";
string elemCatName = "ProcessMonitoring";
string attrCatName = "Process Monitoring";

List<AFSearchToken> nestedTokens = new List<AFSearchToken>();
nestedTokens.Add(new AFSearchToken(AFSearchFilter.Template, templateName));
nestedTokens.Add(new AFSearchToken(AFSearchFilter.Category, elemCatName));

List<AFSearchToken> tokens = new List<AFSearchToken>();
// The Element uses the nested token(s)
tokens.Add(new AFSearchToken(AFSearchFilter.Element, AFSearchOperator.Equal, null, nestedTokens));
// The Attribute uses the non-nested tokens, in this case just Category.
tokens.Add(new AFSearchToken(AFSearchFilter.Category, attrCatName));

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "nested tokens example", tokens))
    search.CacheTimeout = TimeSpan.FromMinutes(10);
    foreach (AFAttribute item in search.FindAttributes())
        // Do something



Using Nested Query String


The trick with a nested query string is that it will be enclosed in {braces}.


// Notice how element has nested { }. 
// The Category depends on nesting level. 'Process Monitoring' with a blank is outside the nesting,
// so it will be an Attribute Category, whereas 'ProcessMonitoring' inside the nesting is an
// Element Category.
string query = "Element:{Template:'FCC Pump Process' Category:'ProcessMonitoring'} Category:'Process Monitoring'";

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "nested query string", query))
    search.CacheTimeout = TimeSpan.FromMinutes(10);
    foreach (AFAttribute item in search.FindAttributes())
        // Do Something



Using Interpolated Nested Query Strings


With Interpolated Strings, you may easily substitute a variable's value (technically, it substitute's the string returned from the variable's ToString() method).  If you are familiar with this in either C# or VB.NET, you know that {braces} are used.  This raises an interesting question of how the Interpolated String knows which brace is for the nested query, and which is for the value substitution.  You would denote that using an escape sequence of the braces themselves.


string templateName = "FCC Pump Process";
string elemCatName = "ProcessMonitoring";
string attrCatName = "Process Monitoring";

// This gives a compile error with an Interpolated String
// string query = $"Element:{Template:'{templateName}' Category:'{elemCatName}'} Category:'{attrCatName}'";

// Escape the { and } around literal braces with {{ and }}.
// Fun Fact: {{something}} is called the Mustache Template!
// See
string query = $"Element:{{Template:'{templateName}' Category:'{elemCatName}'}} Category:'{attrCatName}'";

using (AFAttributeSearch search = new AFAttributeSearch(root.Database, "interpolated nested query string", query))
    search.CacheTimeout = TimeSpan.FromMinutes(10);
    foreach (AFAttribute item in search.FindAttributes())
        // Do Something


An interesting bit of trivia: the {{something}} format of double braces is called the Mustache Template!





The nested query examples used AFAttributeSearch for searching upon elements.  The AFAttributeSearch may search on event frames instead.


One thing to note is the fullLoad parameter is missing from the FindAttributes method because it must always do a full load since the attributes cannot exist without the owning element or event frame.  However, the AFAttributeSearch.FindObjectFields is smart enough to know when it must make a full load to evaluate data references.  To state that a different way, it is smart enough to know when to skip a full load because of captured event frames. You don't have to do anything special like consult Oracles to figure this out for any given situation.  You would code it the same way regardless of the situation and let AFAttributeSearch.FindObjectFields make the right decision!


Why you should use the new search classes


Maintainability - some Find methods in AFAttribute or AFNotificationContactTemplate have already been marked as Obsolete, and it is reasonable to suspect that others may be marked so with future releases of AF SDK.  Make your applications more resilient by switching to these new methods sooner rather than later.


Performance - if you require fetching more than one page of results from the server and you opt-in to server-side caching, any of the AFSearch classes perform much faster than the older methods.  This means that AFAttributeSearch will be much faster than the AFAttribute.FindElementAttributes overloads.


Ease of Use - the AFSearch classes take care of paging issues seamlessly for you.  The new nested queries makes searching for attributes on either elements or event frames very easy with fewer lines of code.  There is previously mentioned smartness built into AFAttributeSearch.FindObjectFields. All of this without extra coding on your part.


Reliability - when searching on attribute values that might be evaluated client-side (due to needing to evaluate the data reference), it is impossible to do paging properly using the older search methods because you don’t reliably know what to use for the next page index.  By opting-in to server-side caching with the newer AFSearch methods, paging is more reliable.

The webinar scheduled for today (Wednesday March 21) will begin at 4:00 PM GMT or UTC.  Due to an unfortunate glitch with North America observing DST and Europe not, the previously published start times are incorrect.  Europe does not transition to Summer Time until March 25.


Recording and Slides are available at this link.


Topic: Asset Analytics native integration with MATLAB


North America 4:00 PM GMT is:

     Noon - 1 PM US Eastern

     9-10 AM US Pacific


Europe 4:00 PM GMT is:

     5:00 PM CET


UPDATE: A customer reported that the link in his confirmation mail 'click here to join' does not work, because it is missing the ':' after https. We don't know how widespread this is or whether it is a one-off for this particular customer.  However, he did report that the the URL in the Outlook invitation/appoint is OK.

Table of Contents



Many of the .NET based code samples on PI Square and GitHub are in C#. It’s easy for a VB.NET developer to feel left out. If you’ve been following me within the past 5 years you may think that I am a heavy duty C# developer. Well, I am actually, and C# is definitely my first choice for writing applications. However, I do have a fondness for VB as I started out with Visual Basic 3 in the mid-1990’s and was able to eek out a nice living writing for VB, VBA, and eventually VB.NET. VB helped feed and clothe my family for close to 2 decades.


Coming from VB 6, VB.NET was a natural introduction to .NET programming. But early on, I started to wean myself away from VB-centric calls and more towards .NET-centric ones. For example, I replaced VB’s MsgBox calls with .NET’s MessageBox. This transitioned me to the point where I stopped calling myself a VB developer and instead started calling myself a .NET developer because it was the .NET Framework, not the language used, that really was at the heart of my applications. After a while of doing that, I found it an easier transition to switch from VB.NET to C# than when I had switched from VB 6 to VB.NET.


There is no need to put your "VB Forever" shields up. I am not here to convince you to switch to C#. You are more than welcome to stay with VB.NET. What I am here to do is to help explain a few language specific things about C# so that it makes it easier for VB.NET coders like yourself to follow the C# examples more easily and without being stumped by some C# idiosyncrasies. To a large degree, many of our examples are fairly easy to follow. Putting aside the unsightly semi-colons and multitude of braces versus the wordiness of Visual Basic, both languages obviously have an overlapped feature set. It’s easy to understand that C#'s foreach and VB's For Each do the same thing. Likewise, both languages use similar features like Interpolated Strings, which look quite similar in either language.


This cheat sheet is to help address those few C# things that aren’t immediately translatable to VB. That way you can begin to put your focus back on PI Developer Technologies and not worry so much about a language that is foreign to you.



Logical Operators

Earlier versions of VB did not have short-circuiting but VB.NET has had it for quite a long while.




var keyword for implicitly typed variables

Many C# snippets may occasionally employ the var keyword to declare variables. For someone unfamiliar with it, they may mistakenly believe that the variable has a type of Object and may take any value of any data type (like Variant in the olden days). This is not true. With var, a variable is assigned the type from the right-hand expression, and once that type is assigned it cannot be changed. Microsoft's own recommendation is to use var when (1) the data type on the right-hand side is easily understood, or (2) if you really don't care about the type for variable with a very short scope.


var thing1 = "Hello, World!"; // always a string
var thing2 = 1; // always an Int32
var thing3 = 1.0; // always a Double
var thing4 = 1.0F; // always a Single


The equivalent code in VB.NET would be:


Dim thing1 = “Hello, World!”; ' always a string
Dim thing2 = 1; ' always an Int32
Dim thing3 = 1.0; ' always a Double
Dim thing4 = 1.0F; ' always a Single



Conditional Operator, or ? : operator

You may frequently see statements such as this peppered in C# code:


var lastIndex = (list != null) ? list.Count – 1 : -1;


That is to say condition ? truePart : falsePart; This is called the conditional operator. The equivalent in VB.NET would use the If operator:


Dim lastIndex = If(list != null, list.Count – 1, -1)


One should emphatically NOT use the IIf function, as it has the adverse side-effect of evaluating both the TruePart and FalsePart regardless of the condition. For example:


IIf(condition, SomeFunction1(), SomeFunction2())


Is the same as this code:


Dim truePart = SomeFunction1() ' always runs regardless of condition
Dim falsePart = SomeFunction2() ' always runs regardless of condition
If condition Then
End If


Whereas the If operator is equivalent to this bit of code:


If condition Then
' will only run if condition is True
' will only run if condition is False
End If



Null-Coalescing Operator, or ??

I personally am not a big fan of the null-coalescing operator or simply ??. It returns the left-hand operand if that operand is not null; otherwise it returns the right hand operand. The VB equivalent is also the If operator but with only 2 arguments, and the first argument must be a nullable reference type.


There is a decent explanation of how it works with VB.NET in this StackOverflow link. See the answer by Code Maverick on Dec 19, 2013.



Read-only Auto-properties (or get-only property)

C# version 6.0 introduced read-only auto-properties, which might seem odd to ponder a get-only property:


Public property AFDatabase Database { get; }


The above is equivalent to this in earlier versions of C#:


Private readonly AFDatabase _database;
Public AFDatabase Database { get { return _database; } }


In either case, the readonly property may only be assigned a value at class initialization or else within a class constructor.


The equivalent in VB.NET would be:


Private ReadOnly _database As AFDatabase
Public ReadOnly Property Database() As AFDatabase
Return _database
End Get
End Property


Note that in VB both the property - and the private backing field - must be decorated as ReadOnly.



Expression-bodied members (property => expression)

A C# expression-bodied member may work for methods as well as properties, and are best reserved for a quick one-line piece of code. Borrowing from a snippet above, an example would be:


Public AFDatabase Database => _database;


Which is the equivalent of:


Public AFDatabase Database { get { return _database; } }


However, VB.NET does not support expression-bodied members so you would have to use the more verbose VB code referenced in the previous VB example.



Remainder or Modulus operator

In VB, the Mod operator is used to find the remainder:


Dim quartile As Integer = value Mod 4


In C#, the % operator does the same thing:


int quartile = value % 4;



Integer versus Floating Point Division

In VB, the / operator will always perform floating point division, even if both operands are integers. To perform integer division with VB, one would use the \ operator.


Dim a = 5 / 2 ' a is a Double = 2.5
Dim b = 5 \ 2 ' b is an Int32 = 2


In C#, division on 2 integers always results in integer division. There is no direct equivalent to VB's \ operator. In order for floating point division to occur in C#, at least one of the operands should be a floating point value.


var a = 5 / 2; // a is an Int32 = 2 since 5 and 2 are Int32
var b = 5 / 2.0; // b is a double = 2.5 since 2.0 is a double
var c = (double)5 / 2; // c is a double = 2.5 since integer 5 is cast to double before division



The use of Lambdas

C# methods may concisely use lambdas particularly with LINQ. VB.NET requires a little more verbosity. Let’s consider an example where variable named values is an AFValues collection, and you wish to count the number of bad values.



int badCount = values.Count(x => x.IsGood == false);



Dim badCount As Integer = values.Count(Function(x) x.IsGood = False)


Let’s consider another example where 2 lambdas are needed. Let’s use LINQ to create a dictionary keyed on AFAttribute where the dictionary value is an AFValue.



AFAttributeList attrList = new AFAttributeList();
// and then you populate attrList
AFValues values = attrList.GetValue();
IDictionary<AFAttribute, AFValue> lookup = values.ToDictionary(key => key.Attribute, value => value);



Dim attrList As AFAttributeList = New AFAttributeList()
' and then you populate attrList
Dim values As AFValues = attrList.GetValue()
Dim lookup As IDictionary(Of AFAttribute, AFValue) = values.ToDictionary(Function(key) key.Attribute, Function(value) value)



Optional Parenthesis With 0 Parameters

If you use a code converter convert from C# to VB.NET, depending upon the converter of your choosing you may find many parenthesis missing in the VB.NET code. This generally occurs when no parameters are being passed. VB tries to do more for the coder than does C#. Some would suggest VB tries to do too much. Consider how C# requires the developer to know and understand cast conversions within an expression, whereas VB.NET can perform the cast conversions automatically and therefore explicit code to issue conversions is optional with VB. Likewise, VB.NET allows the parenthesis to be optional on parameter-less method calls.



A reasonable guess as to why this is allowed is that VB may be allowing you to use extension methods as a pseudo-property. The downside to this is it makes debugging a bit trickier because you may read along and think that something is a property, because it lacks parenthesis, when it is in fact a function. Plus the following just doesn’t look right to as C# coder:


foo.ToString when it really is foo.ToString()


There are no associated code examples here since it is more an artifact of the language and not a specific command of it. Suffice to say that if you are converting from C# to VB.NET, or vice versa, you will want to pay attention to anything that looks like a property, since it lacks parentheses. Some developers coding styles may allow this to remain. Others may want to explicitly insert the () to remove any doubt or confusion as to whether a call is a property or a method.



Characters to Integers and vice versa

C# has a char type and VB.NET has a Char. Both are the same thing, but how you may interact with them is quite different depending on the language. In C# a char is denoted by single quotes. In VB.NET, double quotes are used but to distinguish that the double quotes refer to a character and not a string, a lowercase c is appended following the end quote. Here is a capital A in each language:



char capitalA = 'A';



Dim capitalA As Char = "A"c



In C#, you may easily treat a char as a int or vice versa. Consider an example where you want to loop from 'A' to 'Z'.


for (int character = 'A'; character <= 'Z'; character++)


In VB.NET, the safest way to perform similar conversions requires the Microsoft.VisualBasic library be imported. This DLL provides the AscW function to safely convert a character to integer (technically it’s a Unicode code point). To convert from the integer (or Unicode code point) back to a Char, you would use the ChrW function. Here’s a VB translation of the above C# code.


For character As Integer = AscW("A") To AscW("Z")


You may ask why not just convert directly from Char to Integer or back, and note the use of “safest” or “safely”. A character is not represented just by any integer. It is a Unicode code point, and there are complications such as surrogate pairs or unprintable characters that are handled inside of AscW and ChrW.



Related Links

For additional reading, you may want to check out this comparison of language features between C# 6.0 and VB 14.


There you go. We've given a little love to our VB.NET community, which is probably larger than we think it is (you are certainly a quiet bunch). You may let me know if you liked this in comments below. Or you may comment if there are any C# translations you would like to see covered. I will be glad to add it to this page. As stated in the introduction, my intent is to empower you to read C# examples so that you are better equipped to translate to VB.

Asset Analytics Native Integration with MATLAB


March 21, 2018

9:00 AM (PT)/12:00 PM (ET)/6:00 PM (ES) See European Time Below


Link To Recording


In this PI Developers Club webinar, we offer a sneak preview of the AF 2018 release with the highly anticipated feature of native integration between Asset Analytics and MATLAB!  Panelists include Stephen Kwan, AF Product Manager, along with technical representatives from MathWorks, Tim Choo and Sachin Nikumbh, both on the MATLAB Production Server team.  Data scientists, developers, system integrators, or anyone interested in analytics is encouraged to attend and ask questions.


UPDATE: If you clicked the above link before Feb 14 at 2 PM US Pacific time, your registration may not have gone through.  If you have not received a confirmation email, please register again.



North America transitioned to DST on March 11, and Europe will not transition until March 25.  This produces an offset error with our scheduling system.  The correct webinar times for Europe will be:


     5:00 PM CET or 4:00 PM GMT (or UTC)


For locales that do not observe DST, your scheduled times may also be incorrect.  Please refer to the GMT/UTC time above and adjust your local time accordingly.



  • Stephen Kwan, AF Product Manager
  • Tim Choo, MathWorks, Product Manager for MATLAB Production Server
  • Sachin Nikumbh, MathWorks, Engineering Manager for MATLAB Production Server
  • Hosted by Rick Davin, Sr. Technology Enablement Engineer


We'd love to hear your comments and questions here in the comments or in the PI Developers Club forum!

Some customers have the occasional but pronounced need to retrieve millions upon millions of recorded values for a given PIPoint.  There is no method directly with the PI AF SDK to address this need.  Virtually every data method within the SDK has 2 known limitations:


  1. An operations timeout should a single data call take too long to fulfill, and
  2. No more than ArcMaxCollect values may be returned in a single data call, where ArcMaxCollect is a tuning parameter on your PI Data Archive.


One should not modify ArcMaxCollect on a whim.  There's a reason why the defaults are what they are (PI Server 2012+ the default is 1.5 million, earlier versions are 150K).  You would not be increasing it just for you.  The change applies to all users.  How confident are you that your own users won't be trying to fetch 200 million data values at once?  There is a workaround that you may prudently use in your code in lieu of increasing ArcMaxCollect.


GitHub - Rick-at-OSIsoft/pipoint-getlargerecordedvalues: A workaround that removes the limitation of ArcMaxCollect, and …


The GitHub repository has code versions for C# and VB.NET.


In a nutshell, how does this workaround get past the 2 known limitations?  By retrieving the AFValues in pages of 100K values at a time.  This is well below the default value for ArcMaxCollect, and retrieving 100K values is easily satisfied within the operations timeout.  Remember: you would want to use the GetLargeRecordedValues method when you know you have PIPoint(s) with millions and millions of recorded values.



Given you have a PIPoint instance in an object named tag and a defined AFTimeRange in timeRange:


C# Example

var values = tag.GetLargeRecordedValues(timeRange);
foreach (var value in values)
    // do something with value



VB.NET Example

Dim values = tag.GetLargeRecordedValues(timeRange)
For Each value As AFValue in values
    ' do something with value



Cautions and What to Avoid

The biggest caution to acknowledge is that you have massive amounts of data.  The traditional best practices such as using bulk PIPointList will no longer apply.  You are in a different world if you want to loop over a tag that has 200 million data values.  There are considerations well outside the scope of AF SDK.  Two notable instances are timing and memory.  Retrieving 200 million values will be done as quickly as possible, but you have to accept that it still takes some time to consume that much data.  So the concept of "fast" goes out the window.  And if you attempt to retrieve those values into a list or array, .NET will most likely give you an out-of-memory exception long before all the values are retrieved.


Therefore it's best to avoid memory-hogging calls such as LINQ's ToList() or ToArray().  The best performance is to consume the values as they are being streamed without any attempt to persist to an indexed collection.  Another performance killer is using Count().  I have seen a lot of traditional applications that attempt to show a value count before looping over the values.  Since its a streamed enumerable set that isn't persisted in memory, issuing a Count() has a very negative performance consequence of retrieving and counting all the AFValues.  Subsequent passes through a loop will then require the time-consuming process of retrieving the data a second time!  If you want to display a count in your logs because it's something nice to do, you should maintain your independent count while you are looping the first time, and then display that nice count at the end rather than the beginning.


C# Example

var values = tag.GetLargeRecordedValues(timeRange);
int count = 0;
foreach (var value in values)
    // do something with value
Console.WriteLine("{0} count = {1:N0}", tag.Name, count);

Filter Blog

By date: By tag: