Tutorial: A faster way to get PIPoints from a large list of AFAttributes

Discussion created by rdavin Employee on Mar 22, 2015
Latest reply on Mar 24, 2015 by rdavin

[Edit: per suggestion by Rhys Kirk, I am adding emphasis that this is for high latency networks.  Also its not meant to be a simple answer to a problem but more of an example of various techniques that a developer may employ when working on high latency networks.]


UPDATE March 26, 2015.  An explanation and apology.


Let me state this upfront: NO, this does not show a faster way to get PIPoints.  Jumping ahead to the end, I show how to get PIPoints using FindPIPointsByPath on chunks of 1000 tag paths fetched by a RawPIPointPath call.  Many readers may scream “Why?  WHY!?  Why not just use AFAttributeList.GetPIPoint?”  It would seem I was hellbent on ignoring GetPIPoint, and I absolutely was.  However, the reason was not apparent to the reader because I left that critical reason out.


I have a lot of irons in the fire for the many apps that I have in development.  Some of these apps are related to others, and some are not.  We’ve all been there.  You have a piece in one app that’s a good fit in another.  On top of that, I sometimes keep an eye out for suitable “food for thought” posts here a PISquare.


As I was working on some other app, henceforth known as That Other App, where I wanted to identify which of my attributes were PI Points.  I fetch 600K attributes and take 1.5 seconds to filter down the 200K Point attributes.


Next up in That Other App, I fetch the RawPIPointPath for those 200K attributes.  This took 38 seconds.


And the next thing I did in That Other App was get the PIPoints.  Sure I could have used AFAttributeList.GetPIPoint, but internally it does what I’ve already take over 39 seconds to do: filter on the PI Point attributes and get the tag name.  If I’ve already incurred the expensive hit to do that, there is no need to have GetPIPoint take the same hit.  Or to put it another way, I should completely avoid GetPIPoint in that circumstance. Instead it makes perfect sense to jump right to FindPIPointsByPath.  And avoid GetPIPoint.


Which I did in That Other App.  And I did it using Partioner.Create on chunks of 1000 tag paths, as well as a Parallel.ForEach.  There was just a tiny bit of documentation on that code, and as I was writing it for That Other App, I thought “Wow, this is really worthy of its own detailed explanation.” And it was.


The problem is that I pulled just that bit out of That Other App for the key topic of this post.  And by doing so, I COMPLETELY LEFT BEHIND THE IMPORTANT CONTEXT THAT I HAD ALREADY FETCHED TAG NAMES IN MEMORY.  So here I am now, avoiding AFAttributeList.GetPIPoint at all cost, but never explaining to the reader why.  I must then go about filtering the PI Point attributes and fetching the tag names just to get to the parallel chunking section.  All of which leads to a very bad example which seems to jump off the rails near the end.


I am inserting this long-winded explanation inside the Word document as well as at the top of the post.


And over the next few months I hope to have a new tutorial covering the pitfalls of parallel that keeps the good parts of this parallel chunking section but with better examples.

*** END UPDATE ***



What’s the fastest way to get actual PIPoints from a large collection of AFAttributes?  There are simple methods, but if you have 100,000 attributes the simple methods can be slow.  While I don’t know of the fastest way for all circumstances, I do know of faster ways for large collections over high latency networks..


Towards that end, I've written a detailed 9-page Microsoft Word document with in depth discussion and tons of code samples.  If you are brave enough to take this journey, along the way you will learn about:


  • Some neat inside info about the AFAttribute.PIPoint.
  • How to quickly determine if an AFAttribute uses the “PI Point” data reference. (Best practice!)
  • About the RawPIPointPath() method.  (Best practice!)
  • A little LINQ.  Just because.
  • About the Systems.Collections.Concurrent.Partitioner.
  • About Parallel.ForEach and some chunking techniques.
  • That OSIsoft recommends you set the MaxDegreeOfParallelism whenever your process in parallel.
  • About PIPoint.FindPIPointsByPath.


Links To Other Projects by Rick:


Free App - AF Demo With Timings !!!

Laggy Network FindPIPoints


Download Zip Attachment Here