sraposo

How to find analyses writing to attributes with a certain configuration string - Part 1 C#

Blog Post created by sraposo on Mar 30, 2020

I get these these questions frequently:

 

  1. We're running into performance issues because we are profusely writing to static attributes. How can we find all analyses writing to these attributes? 
  2. We can't remember which analysis is writing to a certain PI Point. How do we find this analysis? 
  3. We are seeing duplicated data (value1,value2, timestamp1) in a PI Point. We are suspecting multiple analyses writing to the same PI Point. How can we find them? 
  4. We are seeing output values at timestamps that should be impossible based on the analysis configuration. We are suspecting another analysis is writing to these PI Points. How can we find them? 

 

Before jumping into a solution, I have to mention that all of the above questions wouldn't be asked if best practices were followed  ! 

 

For (1), PI System Explorer does not let you map a static attribute to an analysis. Instead you have to map it to either a PI Point or Analysis data reference, and then manually change the data reference to <none>. A good rule of thumb I always recommend customers to follow: If it isn't straight forward to do in the UI, maybe it shouldn't be done. When in doubt search the knowledge base, PI Square or reach out to support! 

 

For (2), (3) and (4), if the PI Points are created by the PI Analysis Service and the default %id% substitution parameter is kept in the naming pattern for the output PI Points: 

 

then (2), (3) and (4) are answered very easily. (3) and (4) shouldn't happen in the first place because no 2 analysis will have the same ID. For (2), the analysis ID is in the PI Point and can easily be found. 

 

I have to mention this because we often hear complaints that answering these types of questions is not straight forward using out of the box tools provided by OSIsoft. On one hand, I completely agree that these questions should be easier to answer, and this is why I'm writing this series of blog post and why support has an internal utility to help our customers do this. On the other hand, there are safeguards in the software to prevent any of these questions from being asked in the first place. This is why following best practices and keeping default settings is a good idea, and when it isn't possible, understanding the repercussions is important! 

 

This is part 1 of 2 of a series of blog post. In this post, I'll share  a powerful C# approach. In the other post, I'll share a more light weight and less powerful and performant PowerShell approach. 

 

Of course the below code for a C# .NET Standard console application can be and should be improved. This isn't a complete solution, but an idea that should be improved, tested and vetted before deployment.

 

For example, the following improvements could be done: 

 

  • Add a /? switch and output to the console an explanation of the application, each switches and an example. 
  • Output the analysis path to a file rather than the console window. 
  • Enhance the error handling (I intentionally did a very poor job of this in my example).
  • Add some feedback to the user. Right now the console application doesn't communicate it's progress. In larger systems, it could appear to hang... 
  • Add an AF Database switch to only target specific databases. 
  • Modify the code to handle older versions than 2018 SP2
  • etc.. 
  • etc.. 

 

Of course, if you run the code below it will work, but it's the readers responsibility to understand it, improve it and troubleshoot any issues. Questions in the comments are of course always welcomed  ! This code is in no way supported by OSIsoft. It's provided as is for educational purposes. 

 

The console application I have below will identify all analyses writing to static attributes. There is one mandatory switch, the /AFServer. There are two optional switches and both must be specified: the /DataArchive switch and the /PIPoint switch. Even if the /DataArchive and /PIPoint switch are being used, the console application will output the analyses writing to static attributes. 

 

The first thing to do in the console application is to parse the inputs:

 

private static Dictionary<string, string> ParseInputs(string[] args)
{
Dictionary<string, string> inputArguments = new Dictionary<string, string>();
try
{
foreach (string arg in args)
{
string argWithoutSlash = arg.Replace("/", string.Empty).ToLower(); //don't want to worry about capitalisation
string[] argumentValuePair = argWithoutSlash.Split(':');
inputArguments.Add(argumentValuePair[0], argumentValuePair[1]);
}
}
catch (Exception Ex)
{
throw Ex;
}

return inputArguments;
}

 

Once we are sure the inputs are in the correct format, the next logical step is to make sure they are valid inputs. In other words, make sure we can connect to the AF Server, Data Archive and the PI Point can be found. This can be done using these two methods: 

 

private static void CheckAFServer()
{
try
{
PISystems afServers = new PISystems();
PISystem _afServer = afServers[afServerName];
_afServer.Connect();
}
catch (Exception Ex)
{
throw Ex;
}
}

 

private static void CheckPIPoint(string DataArchiveName, string PIPointName)
{
try
{
PIServers dataArchives = new PIServers();
PIServer dataArchive = dataArchives[DataArchiveName];
PIPoint piPoint = PIPoint.FindPIPoint(dataArchive, PIPointName);
}
catch (Exception Ex)
{
throw Ex;
}
}

 

One last method that can be useful, especially in large system, is a method to split our list of analyses to search into smaller lists and search concurrently. This can be done using a method like this:

 

public static List<List<AFAnalysis>> SplitAFAnalysisList(List<AFAnalysis> listToSplit, int pageSize)
{
var pages = new List<List<AFAnalysis>>();
int NumberOfPages;
if (listToSplit.Count < pageSize)
{
NumberOfPages = 1;
}
else
{
NumberOfPages = (int)listToSplit.Count / pageSize;
if (NumberOfPages % listToSplit.Count > 0) NumberOfPages++;
}

for (int i = 0; i < NumberOfPages; i++)
{
pages.Add(listToSplit.Skip(i * pageSize).Take(pageSize).ToList<AFAnalysis>());
}

return pages;
}

 

Ok now that we have our supporting methods, we are ready to build the main:

 

  1. Parse the inputs and check the syntax
  2. Assign each input/value pair to a dictionary 
  3. Check input validity 

 

class Program
{
static string afServerName;

static void Main(string[] args)
{
Dictionary<string, string> inputArguments = new Dictionary<string, string>();
string dataArchiveName;
string piPointName;
PISystem afServer;
try
{
inputArguments = ParseInputs(args);
}
catch
{
Console.WriteLine("Inputs were not in the correct format");
return;
}
try
{
afServerName = (from entry in inputArguments where (entry.Key == "afserver") select entry.Value).FirstOrDefault();
CheckAFServer();
PISystems afServers = new PISystems();
afServer = afServers[afServerName];
piPointName = (from entry in inputArguments where (entry.Key == "pipoint") select entry.Value).FirstOrDefault().ToLower();
dataArchiveName = (from entry in inputArguments where (entry.Key == "dataarchive") select entry.Value).FirstOrDefault().ToLower();
if(piPointName != null && dataArchiveName ==null || piPointName ==null && dataArchiveName !=null)
{
Console.WriteLine("You must specify both data archive and pi point!");
return;
}
if(dataArchiveName != null && piPointName != null) // && here isn't needed but included for ease of reading / understanding
{
CheckPIPoint(dataArchiveName, piPointName);
}

}
catch (Exception Ex)
{
Console.WriteLine("Exception Encountered:");
Console.WriteLine(Ex.Message);
Console.WriteLine("Stack is:");
Console.WriteLine(Ex.StackTrace);
Console.ReadLine();
return;
}

 

The next step is to retrieve all running analyses. We are not looking for stopped analyses in this example. The fastest way of doing this is to leverage the AF SDK access introduced in 2018 SP2 (reference): 

 

string queryString = "Status:'Running'"; //Faster to search for running analyses

string fields = "id";

var runtimeStats = afServer.AnalysisService.QueryRuntimeInformation(queryString, fields);

List<Guid> guidList = new List<Guid>();

foreach (var runtimeStat in runtimeStats)
{
guidList.Add((Guid)runtimeStat[0]);
}

AFNamedCollectionList<AFAnalysis> analyses = AFAnalysis.FindAnalyses(afServer, guidList.ToArray());

 

Now that we have a list of all running analyses we need to check each of their outputs. Loading analysis outputs is expensive, therefore doing this concurrently makes a lot of sense in larger systems. We will use the method above to split our list into a subset of lists. Moreover, we should only look analyses writing to outputs. In most cases, Event Frame analyses are not writing to outputs and can be discarded. 

 

//We don't care about Event Frames for this example, you can also add SQC here if you use SQC analyses

List<AFAnalysis> filteredAnalyses = analyses.Where(a => a.AnalysisRulePlugIn.Name != "EventFrame").ToList<AFAnalysis>();

int pageSize = 500; //play around with the page size to find the best performance... trial and error unfortunately.

List<List<AFAnalysis>> chunkedAnalyses = SplitAFAnalysisList(filteredAnalyses, pageSize);

 

Now that we have a list of list of analyses, we can load the outputs concurrently. We need to use a ConcurrentDictionnary to ensure the integrity of the object being written to from multiple tasks. We will write in this dictionary an AF Attribute and AF Analysis pair. Each AFAttribute can only be written to by 1 analysis, whereas 1 AFAnalysis can write to multiple AFAttribute. So the key in the dictionary is the output AFAttribute and the value is the AFAnalysis writing to it. 

 

//use concurency to speed up retrieval

Parallel.ForEach(chunkedAnalyses, (analysisList) =>
{
Dictionary<AFAttribute, AFAnalysis> tempDict = new Dictionary<AFAttribute, AFAnalysis>();

foreach (AFAnalysis analysis in analysisList)
{
var analysisRule = analysis.Template?.AnalysisRule ?? analysis.AnalysisRule;
var variableDefinitions = analysisRule.GetVariableDefinitions(analysis.Target as AFElement);
var variableMap = AFVariableMap.Create(analysisRule);

foreach (var outputdef in variableDefinitions.Outputs)
{
var resolvedOutput = outputdef.Resolve(variableMap, analysis.Target as AFElement);
if (resolvedOutput.OutputType == AFAnalysisOutputType.SingleValue && resolvedOutput.Attribute != null)
{
tempDict[resolvedOutput.Attribute as AFAttribute] = analysis;
}
}

}

tempDict.ToList().ForEach(x => concDict.TryAdd(x.Key, x.Value));
});

 

Now our ConcurrentDictionary has a list of output AFAttribute and AFAnalysis pair. We simply need to see if the AFAttribute has a data reference. If it doesn't then this is a static attribute and should be reconfigured, if it does, we can compare the configString to our input DataArchive and PIPoint. 

 


var analysesWithOutputNoDR = (from entry in concDict where (entry.Key.DataReference == null) select entry.Value);

if (analysesWithOutputNoDR == null)
{
Console.WriteLine("No analyses found writing to static attributes");
}
else
{
Console.WriteLine("Analyses writing to static attribute:");
foreach (var analysisWithOutputNoDR in analysesWithOutputNoDR)
{
Console.WriteLine(analysisWithOutputNoDR.GetPath());
foreach (var item in concDict.Where(kvp => kvp.Value == analysisWithOutputNoDR).ToList())
{
concDict.TryRemove(item.Key, out AFAnalysis dummy);
}

}
}

if (dataArchiveName != null && piPointName != null ) //Again && is not needed.
{
string configString = @"\\" + dataArchiveName + @"\" + piPointName;
var analysesWithOutputPIPointDR = (from entry in concDict where (entry.Key.DataReference.ConfigString.ToLower().Contains(piPointName) && entry.Key.DataReference.ConfigString.ToLower().Contains(dataArchiveName)) select entry.Value);
if (analysesWithOutputPIPointDR == null)
{
Console.WriteLine("
No analyses found for:" + configString);
}
else
{
Console.WriteLine("Analyses writing to:" + configString);
foreach (var analysisWithOutputPIPointDR in analysesWithOutputPIPointDR)
{
Console.WriteLine(analysisWithOutputPIPointDR.GetPath());
}
}
}

afServer.Disconnect();
Console.ReadLine();
}

 

That's it! We now have a console application that can find all analyses writing to a certain PI Point and all analyses writing to static attributes. If your curious about performance, it should take ~45s for 10k analyses. 

 

I'm still learning C#, so if anything goes against Microsoft best practices, or can be improved in any way please comment below. I won't take it personally. I'll edit the post with your changes and thank you  ! 

Outcomes