Skip navigation
All Places > PI Developers Club > Blog > 2012 > October
2012

 

 

One of the use case that we get from customers for PI Asset Framework (AF) is the ability to search for elements based on attribute values. One example: Customers may want to collect information of equipment from a specific vendor, then they would probably want to post the query like "Get all turbine that is manufactured by 'Sidewinder' <manufacturer name>". On the other hand, user may also want to identify asset based on time-series information like status. For example "Find all reactors which the status is 'On'".

 

If you look into the AF SDK Programming Reference, you will find a specific method implemented to fulfill this requirement "AFElement.FindElementsByAttribute.

 

For example, I have a test environment i defined a capacitor template with the following attribute templates

 

3731.Snap5.jpg

 

A sample implementation is:

 
AFElementTemplate template = afdb.ElementTemplates["Capacitor"];
if (template != null)
{
    // 1st value condition: Manufacturer = Borne Engineering
    AFAttributeTemplate attrtemplate2 = template.AttributeTemplates["Manufacturer"];
    AFAttributeValueQuery query1 = new AFAttributeValueQuery(attrtemplate2,
    OSIsoft.AF.Search.AFSearchOperator.Equal, "Sidewinder");

    // 2nd value condition: Differential Energy > 100
    AFAttributeTemplate attrtemplate1 = template.AttributeTemplates["Differential Energy"];
    AFAttributeValueQuery query2 = new AFAttributeValueQuery(attrtemplate1,
    OSIsoft.AF.Search.AFSearchOperator.GreaterThan, 100);

    // AFAttributeValueQuery array allow us to pass 1 or more value query in the search
    AFAttributeValueQuery[] queries = new AFAttributeValueQuery[] { query1, query2 };

    DateTime start1 = DateTime.Now;
    elementList = AFElement.FindElementsByAttribute(null, "*", queries, true, AFSortField.Name,
    AFSortOrder.Ascending, 100000);
    DateTime end1 = DateTime.Now;
    Console.WriteLine("Number of Elements Found = {0}", elementList.Count);
    Console.WriteLine("Time Taken for search to complete = {0} ms", (end1 - start1).TotalMilliseconds);
}

This implementation works great if you use templates when building up you AF Tree. However if you scale this up and try to search among 10s or 100s of elements, you will start to see that this does causes a significant load on the server. For example if I try to run the code with more than 50,000 capacitors stored in my AF server, and we look at the time to complete the search and load on AF Server and SQL Server services

 

3617.Snap1.jpg

 

5460.Snap4.jpg

 

It does take significant amount of time to complete and we can see that the server services are constantly working throughout the period that the search is taking place. Knowing that getting values through data reference is mostly handled at the client, it is surprising that AF Server and SQL Server is spending so much time working as the search going on.

 

After investigating into this with the help from Chris Manhard, it has become obvious that as the search is running, element and attribute information is gradually being loaded by the client. Hence we get the performance that we see above. 

 

What if we add 2 additional steps before calling the FindElementsByAttribute method? 

 
AFElementTemplate template = afdb.ElementTemplates["Capacitor"];
if (template != null)
{
    // Additional code: FindElementByTemplate then LoadElement before searching by attribute
    AFNamedCollectionList elist = AFElement.FindElementsByTemplate(afdb, null, template, true, AFSortField.Name, AFSortOrder.Ascending, 100000);
    AFElement.LoadElements(elist);

    // 1st value condition: Manufacturer = Borne Engineering
    AFAttributeTemplate attrtemplate2 = template.AttributeTemplates["Manufacturer"];
    AFAttributeValueQuery query1 = new AFAttributeValueQuery(attrtemplate2,
    OSIsoft.AF.Search.AFSearchOperator.Equal, "Sidewinder");

    // 2nd value condition: Differential Energy > 89
    AFAttributeTemplate attrtemplate1 = template.AttributeTemplates["Differential Energy"];
    AFAttributeValueQuery query2 = new AFAttributeValueQuery(attrtemplate1,
    OSIsoft.AF.Search.AFSearchOperator.GreaterThan, 100);

    // AFAttributeValueQuery array allow us to pass 1 or more value query in the search
    AFAttributeValueQuery[] queries = new AFAttributeValueQuery[] { query1, query2 };
    DateTime start1 = DateTime.Now;
    elementList = AFElement.FindElementsByAttribute(null, "*", queries, true, AFSortField.Name,
    AFSortOrder.Ascending, 100000);
    DateTime end1 = DateTime.Now;
    Console.WriteLine("Number of Elements Found = {0}", elementList.Count);
    Console.WriteLine("Time Taken for search to complete = {0} ms", (end1 - start1).TotalMilliseconds);
}

Here's the difference in performance

 

6114.Snap3.jpg

 

3582.Snap2.jpg

 

While the peak CPU load on the AF Server and SQL Server is higher, the time AF Server and SQL Server spent working and the time taken to complete the search even with the additional steps is still much shorter than just searching by attribute directly. So if you are searching by attribute from a large number of elements, this can be a kind of workaround that you can try using to improve the search performance.

 

After communicating to Chris about this, a work item has already been created for this behaviour in AFElement.FindElementsByAttribute method, so you can look forward to better performance in the method when this work item is completed.

 

 

 

 

Hi Everyone,

 

I have finally got to the next part of my PI to Mobile series.  In this screencast I create an iPhone app that reads the Azure web services (created in part 1 and 2) and displays selected PI tag data in a chart.  I go over what basic Xcode programming is like and use a couple of third party libraries to make the job eaiser.   This episode is also a two part'er.  The first part is creating the basic iPhone app and the second part does the data gathering and creates charts.  I hope you like them!

 

Part 3a: Creating a Basic iPhone App using Tables and Storyboarding

 

Here is 3a video

 

Part 3b: iPhone App, getting data from ASP.Net Web API and Displaying in a Chart

 

Here is part 3b video

 

 

 

Screenshots:

 

5775.iOS-Simulator-Screen-shot-Oct-7_2C00_-2012-9.52.57-AM.png  8360.iOS-Simulator-Screen-shot-Oct-7_2C00_-2012-9.53.01-AM.png  3582.iOS-Simulator-Screen-shot-Oct-7_2C00_-2012-9.53.05-AM.png 

 

4111.iOS-Simulator-Screen-shot-Oct-7_2C00_-2012-9.53.19-AM.png

 

Source code:

 

 PITrends5.zip (27K)

 

 

 

Thanks for reading,

 

Lonnie

 

 

How to deal with PI Message Logs

 

Overview:

 

You are facing an issue with your PI System, a PI Client or a PI Interface? You like to understand why things are not working as expected? Does PI tell you what's going on? E.g. why PI ProcessBook running on your client machine isn't able to connect to your remote PI System?

 

Yes, it does! This post is intended to help you dealing with PI Message Logs.

 

Please also pay attention to OSIsoft KB Article # 3248OSI8 and the belonging video at Youtube.

 

1.       Logging in PI

 

The most relevant log when working with PI is the PI Message Log. In this post I will focus on PI Message Logs but first let's take a look at the available logs:

 

1.1   pipc.log

 

The pipc.log is the traditional log file being used by PI API based applications like PI ProcessBook earlier than version 3.x and Interfaces to the PI System. UNIINT (a framework for OSIsoft Interfaces to the PI System) based interfaces compiled against UNIINT versions later than 4.5.x are logging to the PI Message Log. If and what UNIINT version was used can either be verified with the Interface manual belonging to a particular interface release or within the PI Interface Configuration Utility (ICU). When an interface (instance) configuration is loaded into the ICU the UNIINT version is displayed below the configuration ComboBox.

 

With PIPC Log Migration enabled (Migratelog=1 in pipc.ini) all pipc.log messages will be forwarded to the PI Message Log.

 

The PIPC Log Server service maintains (shifts, recycles) the pipc.log files and uses settings provided with pipc.ini (e.g. MAXPIPCLOGS and MAXLOGSIZE)

 

PIPC.log files are pure text format and can be read with editors like notepad.exe

 

Using a 3rd party editor that supports syntax highlighting is reasonable.

 

The pipc.log is being replaced with the PI Message Log over time.

 

1.2   PI Message Log

 

File names of the PI Message Log (aka PI Server Log or PI SDK Log) are pimsg_*.dat where * is for a 7 digits number that represents the start date of the particular log file. E.g. pimsg_1120731.dat is from 31-Jul-2012. PI Message Logs are of binary format and usually opened exclusively by PI Message Subsystem service.

 

On a PI Server, PI Message Log files are located in the PI\Log folder; on clients and interfaced nodes, also called PINS nodes, the folder is PIPC\Log.

 

The PI Message Log is the most important log these days when working with PI System products.  

 

In case PI Message Subsystem service is down or unresponsive, messages are sent to the Windows Application Event log. As soon as PI Message Subsystem becomes responsive again it migrates (but does not move) messages from Windows Application Event Log into the PI Message Log.

 

PI Message Logs expire after 35 days. This is controlled and configurable by the tuning parameter MessageLog_DayLimit, introduced with PI SDK 1.4.0.416 (PI Server 3.4.380 SP1 and 2010 SP1). Starting PI SDK 1.4.0.416 some tuning parameters are supported on PINS nodes. Piconfig.exe is used maintaining local tuning parameters.

 

1.3   PI SDK trace logs

 

PI SDK tracing can be used to log additional information about PI SDK calls. AboutPI SDK.exe (PI SDK earlier than version 1.4.0) or PISDKUtility.exe (since PI SDK 1.4.0) can be used to enable the tracing.

 

You may be asked to enable tracing and to send the resulting files for analysis when troubleshooting an issue with OSIsoft TechSupport.

 

1.4   Windows Event Logs

 

Application failures such as a crash of a PI Subsystem will cause a corresponding failure being logged into the Windows Application Event Log.

 

Sometimes we find related messages in the Windows System Event Log.

 

I usually do also request the Windows Security Event Log but don't remember a single case it provided any additional insights.

 

Some OSIsoft products maintain their own Windows Event Log like the PI Asset Framework (AF).

 

2.       What are the tools for PI Message Log analysis?

 

2.1   PI System Management Tools "Message Logs" plug-in

 

PI System Management Tools setup is included into the PI Server setup kit but available as a separate setup too. It can be installed on PI client machines and PI interface nodes.

 

The "Message Logs" plug-in (Operation -> Message Logs) is a Graphical User Interface similar to Windows Event Viewer (eventvwr.msc). It expects a start- and end-time definition. Logs can be filtered by severity, source and strings that may show in the message body. Resulting log messages are displayed in a table and additional info is displayed for the selected entry. All or just selected messages can be exported as comma separated, XML, text or rich text format file.

 

 1374.SMT_5F00_MessageLog.jpg

 

2.2   The pigetmsg.exe command line utility

 

It's time now outing myself as a command line addict. When using pigetmsg.exe for message retrieval a connection to PI Message Subsystem is established for communication. This does not only work locally but also over the network. Supported protocols are Explicit Login, PI Trusts and Windows Integrated Security (WIS).

 

An OSIsoft KB article is in preparation about tips and tricks related to PI Message Logs. I plan adding a link ones this KB article is posted.

 

2.3   PISDKUtility.exe

 

A new utility that ships starting PI SDK 1.4.0 is PISDKUtility.exe

 

It's capable of displaying the local message log

 

 5861.PISDKUtilityMsgLog.jpg

 

3.       Pigetmsg.exe usage - examples

 

The upcoming PI Server 2012 release includes several improvements to pigetmsg.exe. E.g. it will be possible to only output selected fields and message parameters.  Another new feature will be  the ability to create summary reports for specified message parameters. Please be sure visiting checking out pigetmsg.exe help after installing PI 2012.

 

I will only discuss those command line switches that I experienced being useful. Depending on the use case other existing switches may turn out being useful too. Hence I encourage you to check the command help for complete list and additional information about existing switches.

 

pigetmsg -?

 

or

 

pigetmsg -h

 

Combinations of command line switches as used in the following examples are possible.

 

3.1   Looking at the PI Message Log continuously

 

This is the recommendation being used whenever possible e.g. at PI System startup or when verifying an incoming connection attempt.

 

 pigetmsg -f

 

3.2   Setting time limitations for message retrieval

 

Usually we like to look at messages for a period of time specified by start- and end-time. PI Time format is used.

 

pigetmsg -st *-2h -et *

 

pigetmsg -st "31-Aug-2012 13:00" -et "01-Sep-2012 10:00"

 

pigetmsg -st "04-Sep-2012 13:00:01.001" -et *

 

pigetmsg.exe command line utility expects either looking at logs continuously (option -f), specifying a period by start-time (-st) and end-time (-et) or defining the count of recent messages (-t)

 

3.3   Redirecting messages into text files

 

When testing above examples you will recognize messages are passing by in the command prompt quickly and when scrolling back up you may recognize the command line buffer became exceeded. So better to dump messages into a text file:

 

pigetmsg -st *-2h -et * > C:\Temp\mypimessagelog.txt

 

or

 

pigetmsg -st *-2h -et * -of C:\temp\mypimessagelog.txt

 

3.4   Filter messages by severity

 

There are 4 different severity levels with PI Message Logs. The pigetmsg.exe command line tool allows filtering by specifying the minimum severity level.

  • - Debug <= this is the default
  • - Information (-si)
  • - Warning (-sw)
  • - Error (-se)
  • - Critical (-sc)

Most messages are of type Debug or Informational. Filtering them usually means reducing the amount of messages dramatically:

 

pigetmsg -sw -st *-2h -et *

 

You may want to take note of the time(s) for severe messages and check the PI Message Log for related information around that time.

 

3.5   Output formatting

 

If you prefer using a comma separated format (CSV) e.g. for the analysis with Microsoft Excel, you may find this option useful:

 

pigetmsg -fc -st *-2h -et *

 

To force having at least 2 lines for each message log entry, what as far as I've seen is the default, please use the following:

 

pigetmsg -ft -st *-2h -et *

 

The wide output format with just one line per message can be forced using:

 

pigetmsg -fw -st *-2h -et *

 

Programmers may like the option having messages in XML format:

 

pigetmsg -fx -st *-2h -et *

 

3.6   Output all message fields

 

By default only the most common message fields are shown.

 

The result when using the -oa switch will include all fields

 

pigetmsg -oa -st *-2h -et *

 

I have used CSV format to verify the field names

Column name

Default

MessageSeverity

Yes

MessageTime

Yes

ProcessHost

Yes

ProcessName

Yes

Source1

Yes

Source2

Yes

Source3

Yes

Message

Yes

MessageID

Yes

PID (process ID)

No

ProcessPIUser

No

ProcessOSUser

No

OriginatingHost

No

OriginatingPIUser

No

OriginatingOSUser

No

Category

No

Priority

No 

Please note that information fitting in one of the fields listed above may be included into the message body. It will only show up in the corresponding column if it has been explicitly logged to that column.

 

3.7   Filter by message source / process name

 

Valid message sources / process names are e.g. pibasess, pinetmgr, pilicmgr, piarchss and pisnapss.

 

If you are unsure, consult an existing log e.g. using PI-SMT "Message Log" plug-in. Results of the following commands are the same.

 

pigetmsg -src PIACEClassLibraryHost -st *-2h -et *

 

pigetmsg -pr PIACEClassLibraryHost -st *-2h -et *

 

3.8   Filter the message body by a specific string

 

This is one of the most powerful options if you know what you are looking for e.g. messages regarding a particular PINS node identified by name, FQDN or IP address. An asterisk (*) is the wildcard for multiple characters and a question mark (?) replaces just a single character.

 

a) Filtering by IP address

 

pigetmsg -msg *10.17.10.80* -st *-2h -et *

 

b) Filtering by point ID:

 

It depends how point ID's are referred to. Sometimes we see only "ID: <ptid>" what may confuse with connection ID's

 

pigetmsg -msg *point ID: 995* -st *-2h -et *

 

On a large PI System there might be multiple tags with point ID's starting with 995. Hence you may want to make the search string a little stronger by adding a trailing blank:

 

pigetmsg -msg *point ID: 995 * -st *-2h -et *

 

 c) Filtering by connection ID:

 

To filter e.g. a PI Server's log for messages generated for a particular connection, one can look up the connection ID in PI Network Manager Statistics and use it to filter the message body.

 

 pigetmsg -msg *ID: 234* -st *-2h -et *

 

d) Filtering for PI error numbers

 

PI Error messages are represented by negative error code numbers. These error codes do usually show enclosed into square brackets and are of 3 to 5 digits in length. Some examples:

Error #

Translation

[-102]

Record Not Found (Empty)

[-11202]

PI-Redirector could not get archived data from foreign system

[-10466]

No Record Available for Passed recno

Please see chapter 5 "Looking up error codes" for more information on PI error codes.

 

One can filter for particular error codes:

 

pigetmsg -msg *[-11202]* -st *-2h -et *

 

a little bit more laid-back:

 

pigetmsg -msg *[-???]* -st *-2h -et *

 

pigetmsg -msg *[-????]* -st *-2h -et *

 

pigetmsg -msg *[-?????]* -st *-2h -et *

 

or totally stress-free:

 

pigetmsg -msg *[-*]* -st *-2h -et *

 

e) Filtering for "Errors"

 

Filtering for typical keywords such as "failure", "failed", "error" ...

 

pigetmsg -msg *fail* -st *-2h -et *

 

 3.9   Generate a summarized report sorted by the number of occurrences

 

This option doesn't return any message details but counts the amount of messages for the specified field and sorts the returned list by the amount of occurrences top to bottom.

 

pigetmsg -top pr -st *-2h -et *

 

pigetmsg -top pid -st *-2h -et *

 

4.       Tools for PI Message Log analysis

 

Based on the different output formats supported (TXT, CSV, XML), one may want to choose different tools for analysis.

 

4.1   Analyzing text files (.txt)

 

My personal preference still is notepad.exe but sometimes, with very unclear logs filled up by an almost uncountable amount of sources, I wish I would have a more comfortable tool.

 

I know there are text editors that support syntax highlighting but I never spent the time checking which one may fit my requirements. If you have any recommendation for me, please don't hesitate answering to this post.

 

Notepad.exe find function allows you to search for keywords that you have evaluated being useful like "error", "fail", "[- " and so on.

 

If I am more interested in recent messages than in how things have developed over time, I am using the [Find...] functionality bottom up.

 

When analyzing my first message log I started reading at the top and was wondering if I would be able to finish through the log before the end of the day. Seeing the same kind of messages, repeating again and again turned out being boring lecture. Before finishing the first page, I started scrolling through the log using the down arrow on my keyboard. Whenever I recognized a change in pattern, I stopped scrolling and looked more detailed. At least for me this method is working and it's one reason why I prefer pure text files above other options.

 

4.2   Analyzing comma separated files (CSV)

 

You may like loading comma separated logs into Microsoft Excel and using Filter and / or Conditional Formatting. I can also imagine one or the other may use VBA code for automated searches or aggregation of messages suspected being serious.

 

4.3   Analyzing XML files

 

XML files are somehow like very basic databases with each value being enclosed into a field tag. I am curious about your suggestions on what tools can be used for PI Message Log analysis when working with XML format.

 

5.       Looking up error codes

 

If an error code shows in PI Message Log and you are unsure what it is about you can use different tools to lookup the translation of a particular error code.

 

5.1   AboutPI-SDK.exe / PISDKUtility.exe

 

Starting with PI SDK 1.4.0, AboutPI-SDK.exe utility is replaced with PISDKUtility.exe

 

 1067.PISDKUtilityErrorLookup.jpg

 

As you can see in above screenshot, not only PI Error codes are supported.

 

Error codes can be decimal or hexadecimal. PISDKUtility.exe displays both but sometimes the logs have error codes supposed being hexadecimal but without the leading "0x" e.g. 80004005. In case the utility returns "Unknown System Error" you may want to try adding the leading "0x" (0x80004005).

 

5.2   Command line tool pidiag.exe

 

Alternatively pidiag.exe can be used for the translation of error codes. Some examples:

 

D:\PI\Adm>pidiag -e 64

 

[64] The specified network name is no longer available.

 

D:\PI\Adm>pidiag -e -11057

 

[-11057] Not Enough Values For Calculation

 

6.       General considerations

 

PI Message Logs can grow large. I have seen log files with up to 100 MB on large PI Systems and interface nodes with several instances of different interfaces. For sure 100 MB is by far more than what can be reviewed on a daily basis. I recommend using enhanced debugging options only in case of an issue that requires investigation.

 

Especially when dealing with multiple interfaces instances of the same type on a single machine, it becomes important to name instances in order to have useful information being logged. The ICU creates a name for each new interface instance added. You may want to make interface names unique e.g. by adding a few letters that help you to identify the source. Please verify after installing new interface instances that the PI Message Log (or pipc.log with interfaces using UNIINT < 4.5.x) clearly identifies what instance has written particular messages.

 

7.       Final comments

 

Dealing with logs isn't an easy business but it is not rocket science. If you know your tools, it becomes easy.

 

Whenever you are facing issues related to PI products, please do not hesitate to contact OSIsoft TechSupport. The same applies if you are unsure if a particular message is indicating a serious issue or not.  

 

I plan enhancing this post over time by adding information e.g. about using "PowerShell Tools for the PI System" for log file analysis. A colleague recently gave me a hint on using Microsoft LogParser that allows using SQL-like syntax to query all sort of files. Besides PowerShell it looks like another tool for powerful and effective log file analysis. Currently, I am curious what's possible.    

 

I hope you like this post and maybe you learned a few things that will be of use for you in the future. I am looking forward to receiving your feedback.

 

Gregor

skwan

PI AF 2012 - where art thou?

Posted by skwan Employee Oct 2, 2012

 

 

So here I am eating my breakfast and reading my emails.  Among them are a couple of emails from users asking about the release date of PI AF 2012.  As some of you may know, we had originally planned on releasing it at the end of 3Q, which passed 2 days ago.  As the product manager, it goes without saying that I’m very excited that users are waiting for this release (similar to an iPhone release?  ).  So I want to give everyone a status update on PI AF 2012.  We built release candidate 1 and made it available to select external users for testing on 9/22.  We have received a handful of bug reports from both internal and external users some of which we’re fixing.  We’re hoping to build release candidate 2 later this week and if all goes well, it will be generally available later this month.

 

For those who are impacted by this slippage of the release date, please accept my sincere apologies.  This release has significant changes and enhancements to the AFSDK requiring very significant testing efforts.  Rest assured the development team is working their tails off to get this out the door as soon as possible.  Feel free to contact me directly at skwan@osisoft.com if you have additional questions or comments.

 

 

 

 

Filter Blog

By date: By tag: