An example highly scalable AFSDK Custom Analytic Application

Discussion created by kmarsh on Oct 3, 2013

Someone had a requirement to monitor 350,000 things in real-time to generate an alarm when a threshold is exceeded.  Many ideas were kicked around to the conclusion that only a custom AFSDK application could be expected to work at that scale and in real time.  This is an example of such a custom AFSDK application.


Here's how it works.  The main loop is:


ReadIniFile(); Gets started by reading some parameters from an ini file in the same folder as the exe.


while (true) {    


   ReadConfigElement();  Reads one AF element with several string attributes to get the rest of the parameters, i.e. using AF like Windows Registry.


   LoadUpRelevantElements();  Searches up the list of elements matching a specific category (e.g. the 350,000 things to be monitored).  This is the slow part so it is not done often.  For example, maybe every 8 hours or once per day.  It gets from each element one PI point and one alarm limit value.  Nothing is written back to PI in this example, rather alarms are logged in a text file.


   SignalThreadsToStop();  Once the new configuration is loaded from the AF, we need to stop threads that may have been working based on the old configuration.


   DivideWork();  The fast work of doing history recovery, signing up for updates, and analyzing the data to generate alarms is done in near real-time by fast AFSDK RDA calls and multiple threads.  This routine divides up the PI points and limit values according to how many threads will run.  This goes fast so the threads are stopped only an instant.


   SpawnThreads();  Here the threads are run.  These threads have no interaction with the AF, but use only fast RDA calls to the PI server.  They sign up for updates first, then do history recovery, and then continue working in real-time as the main thread moves through this loop, right up until the next SignalThreadsToStop();


   WaitForReloadOfConfig();  Once the threads are running there's nothing for the main thread to do but wait until it is time to reload the configuration.




Here is an example ini file:








MyWater Inc




















And here is the AF Configuration element.  All attributes are STRINGS - why?  Laziness on my part I guess.

AFCategory Attribute Water Meter
HistoryRecoveryStartTime Attribute *-10m
HistoryRecoveryYesOrNo Attribute Yes
LastResultTimestamp Attribute 27-Sep-13 16:48:54
LimitAttribute Attribute High Limit
ReloadConfigurationPeriodInSeconds Attribute 86400
ThreadCount Attribute 10
UpdatePeriodInSeconds Attribute 15
ValueAttribute Attribute Flow Rate



In this case any AF elements of type "Water Meter" would have a "Flow Rate" attribute that is a PI Point data reference, and a High Limit attribute that is just some number.  The configuration will be reloaded every day, and 10 threads will collect data every 15 seconds and generate alarms if Flow Rate > High Limit.