2018

September 2018

# Container Kerberos Double Hop

Posted by Eugene Lee Sep 17, 2018

Note: Development and Testing purposes only. Not supported in production environments.

Containerization Hub

Introduction

In this blog post about security and containers, we will be discussing about implementing a Kerberos Double Hop from the client machine to the PI Web API container and finally to the PI Data Archive container. Previously, when we are using the PI Web API container located here Spin up PI Web API container (AF Server included), we are using local accounts for authentication to the backend server such as the AF Server or the PI Data Archive. The limitation is that without Kerberos Delegation, we will not be able to have per user security which means that all users of PI Web API will have the same permissions. i.e. an operator can read the sensitive tags that were meant for the upper management and vice versa. Obviously, this is not ideal. What we want is to have more granularity in assigning permissions to the right people so that they can only access the tags that they are supposed to read.

Prerequisites

You will need to have 2 GMSA accounts. You can request such accounts from your IT department. They can refer to this blog post if they do not know how to create GMSA Spin up AF Server container (Kerberos enabled). Also be sure that one of them has the TrustedforDelegation property set to True. This can be done with the Set-ADServiceAccount  cmdlet.

You will also need to build the PI Data Archive container by following the instructions in the Build the image section here.

PI Data Archive container health check

For the PI Web API container, you will need to pull it from the repository by using this command.

docker pull elee3/afserver:webapi18


Demo without GMSA

First let us demonstrate how authentication will look like when we run containers without GMSA.

Let's have a look at the various authentication modes that PI Web API offers.

1. Anonymous

2. Basic

3. Kerberos

4. Bearer

We will only be going through the first 3 modes as Bearer requires an external identity provider which is out of the scope of this blog.

Create the PI Data Archive container and the PI Web API container. We will also create a local user called 'enduser' in the two containers.

docker run -h pi --name pi -e trust=%computername% pidax:18
docker run -h wa --name wa elee3/afserver:webapi18
docker exec wa net user enduser qwert123! /add
docker exec pi net user enduser qwert123! /add


Anonymous

Now let's open up PSE and connect to the hostname "wa". If prompted for the credentials, use

Change the authentication to Anonymous and check in the changes. Restart the PI Web API service.

Verify that the setting has taken effect by using internet explorer to browse to /system/configuration. There will be no need for any credentials.

We can now try to connect to the PI Data Archive container with this URL.

https://wa/piwebapi/dataservers?path=\\pi

Check the PI Data Archive logs to see how PI Web API is authenticating.

Result: With Anonymous authentication, PI Web API authenticates with its service account using NTLM.

Basic

Now use PSE to change the authentication to Basic and check in. Restart the PI Web API service.

Close internet explorer and reopen it to point to /system/configuration to check the authentication method. This time, there will be a prompt for credentials. Enter

Try to connect to the same PI Data Archive earlier. You will get an error as the default PI Data Archive container doesn't have any mappings for enduser

Let's see what is happening on the PI Data Archive side.

Result: With Basic authentication, the end user credential has been transferred to the PI Data Archive with NTLM.

Kerberos

Finally use PSE to change the authentication to Keberos and check in. Restart the PI Web API service.

Close internet explorer and reopen it to point to /system/configuration to check the authentication method. The prompt for credentials will look different from the Basic authentication one. Use the same credentials as you did for the Basic authentication scenario.

Try to connect to the same PI Data Archive again. You should not be able to connect. When you check on the PI Data Archive logs, you will see

Result: With Kerberos authentication, the delegation failed and the credential became NT AUTHORITY\ANONYMOUS LOGON even though we logged on to PI Web API with the local account 'enduser'.

Demo with GMSA

Kerberos

Now we shall use the GMSA accounts that we have to make the last scenario with Kerberos delegation work.

Download the scripts for Kerberos enabled PI Data Archive and PI Web API here.

PI-Web-API-container/New-KerberosPWA.ps1

PI-Data-Archive-container-build/New-KerberosPIDA.ps1

I will use the name 'untrusted' as the name of the GMSA account that is not trusted for delegation and 'trusted' as the name of the GMSA account that is trusted for delegation. Set the SPN for 'trusted' like such

setspn -s HTTP/trusted trusted


Once you have the scripts, run them like this

.\New-KerberosPIDA.ps1 -AccountName untrusted -ContainerName pik
.\New-KerberosPWA.ps1 -AccountName trusted -ContainerName wak


The scripts will help you to create a credential spec for the container based on the GMSA that you provide to it. A credential spec will let the container know how it can access Active Directory resources. Then, it will use this credential spec to create the container using docker run command. It will also set the hostname of the container to be the same as the name of the GMSA. This is required because it is a current limitation with the implementation that might be resolved in the future so that you can choose your own hostnames.

Open internet explorer now with your domain account and access PI Web API /system/userinfo. The hostname is 'trusted'.

Make sure that ImpersonationLevel is 'Delegation'.

Now try to access the PI Data Archive. The hostname is 'untrusted'. You will be unable to access. Why? Because you haven't created a mapping yet! So let's use SMT to create a mapping to your domain account. After creating a mapping. Try again and you should be able to connect. The PI Data Archive logs will show that you have connected with Kerberos. You do not need any mapping to your PI Web API service account at all if Kerberos delegation is working properly.

Result: With Kerberos authentication method in PI Web API and the use of GMSAs, Kerberos delegation works. The end domain user is delegated from the client to the PI Web API container to the PI Data Archive container. We have successfully completed the double hop.

Troubleshoot

If this doesn't seem to work for you, one thing you can try is to check the setting for internet explorer according to this KB article.

KB01223 - Kerberos and Internet Browsers

Your browser settings might differ from mine but the container settings should be the same since the containers are newly created.

Alternative: Resource Based Constrained Delegation

A more secure way to do Kerberos delegation instead of trusting the PI Web API container GMSA for delegation is to set the property "PrincipalsAllowedToDelegateToAccount" on the PI Data Archive container GMSA. This is what we call Resource Based Constrained Delegation (RBCD). You do not have to trust any GMSAs for delegation in this scenario. You will still need two GMSAs.

Assuming that you have already created the two containers with the scripts found above. I will use 'pida' as the name of the PI Data Archive container GMSA and 'piwebapi' as the name of the PI Web API container GMSA.

.\New-KerberosPIDA.ps1 -AccountName pida -ContainerName pik
.\New-KerberosPWA.ps1 -AccountName piwebapi -ContainerName wak


Execute these two additional commands to enable RBCD.

docker exec pik powershell -command "Add-WindowsFeature RSAT-AD-PowerShell"
docker exec pik powershell -command "Set-ADServiceAccount $env:computername -PrincipalsAllowedToDelegateToAccount (Get-ADServiceAccount piwebapi)"  You will still be able to connect with Kerberos delegation from the client machine to the PI Web API container to the PI Data Archive container. In this case, the PI Data Archive container only strictly allows delegation from the PI Web API container with 'piwebapi' as its GMSA. Conclusion We have seen that containers are able to utilize Kerberos delegation with the usage of GMSAs. This is important for middleware server containers such as PI Web API. Here is a quick summary of the various results that we have seen. Authentication Mode No GMSAWith GMSA AnonymousNTLM with service accountNo reason to do this BasicNTLM with local end user accountNo reason to do this KerberosNTLM with anonymous logonKerberos delegation with domain end user account The interesting thing is that Basic authentication can also have per user security with local end user accounts. But you will need to maintain the list of local users in the PI Web API container and the PI Data Archive container separately which is not recommended. The ideal case is to go with Kerberos delegation. # Developing Web Apps on top of the PI System Hub Posted by Marcos Vainer Loeff Sep 12, 2018 # PI World Barcelona - The most robust Developer agenda ever seen at PI World EMEA! Posted by Rick Davin Sep 12, 2018 ATTENTION Developers, Data Scientists, IT CyberSecurity, and Power Users. For lack of a better word, I will refer to you collectively as "developers". Developers coming to PI World Barcelona may notice more offerings than ever before. We proudly proclaim that Barcelona will have the most robust Developer Agenda (see link) ever seen at UC or PI World EMEA. This includes the Developer Innovation Hackathon on Day 0 (see link), thanks to our data sponsor DEME. Besides the traditional hands-on labs, which require an additional fee and pre-registration, the Day 3 agenda is chock full of technical content aimed specifically at developers, thanks to 90-minute Live Coding or How-To sessions. These in-depth technical talks do not require a fee nor pre-registration. You are free to come and go as you please. Make no mistake about it ... just because we call the LiveCoding "talks" does not mean they contain less technical information than labs. We expect to offer even more such talks at future PI World events because we think it is better event for you. Our reasoning: you can attend 2 labs on Day 3 for$300, or you can sit in on 4 LiveCoding talks for free.  Who can argue against more technical training for less cost?  (Tip: if you are trying to convince your boss to send you to PI World, presenting it as a major training event (which it is) could be a strong justification to attend.)

I invite you once again to review the Day 3 agenda.  You will see an Analytics Track and Developer Track.  One late correction I would like to make is the PI Admin Track, which is not really for PI Admins but should be considered a 2nd Developer Track.

# PI World Barcelona - Learn about exciting new features of AFSearch 2.10.5!

Posted by Rick Davin Sep 11, 2018

Day 3, Thursday September 27, 11:30 - 13:00

LiveCoding: Getting the Most Out of the New AFSearch

CCIB: Room 117 134, P1 Level

To any members of the PI Developer Community who will be at PI World Barcelona, you are invited to join me in a presentation on new features of AFSearch.  If you ask "Hey Rick, didn't you give this already in SF?", my answer would be "Yes BUT new sections were added to specifically cover some important NEW stuff."  PI AF 2018 R2 (AF SDK 2.10.5) will finally support OR conditions with AFSearch.  That is a highly anticipated new feature that many are looking forward to.  But in order to support OR conditions, it required replacing the older AFSearchToken structure with a new AFSearchTokenBase class that now has 4 different token instances.  Trust me, you will want to see how these new tokens will be used in code.  Everyone who has ever attended this talk has said they definitely learned something!

UPDATE: The room has been moved to 134 on Level P1.

# PIPoint Search Query in AF SDK

Posted by tramachandran Sep 5, 2018

## Overview

Most of us have have searched for PI Points, but as our PI System grows larger or as more products like PI Connectors and Relays automatically create PI tags it becomes imperative to understand how to narrow down and optimize search queries. You might have used the Tag Search Dialog or simply copy pasted sample queries provided in the examples and modified them to suit your needs. Most of the times these queries are intuitive to read and understand but there are situations where we may need to utilize their full expressive power.

In this blog post we will explore PIPoint Query Search Syntax in PI AF SDK. We will have a deeper look into the Syntax Rules and Parsing of queries, along with Wild Cards, Operators and Aliases used in constructing a PI Point Query String to find the desired PIPoint objects. PIPoint Search Utility is used as an aid to accompany the examples shown in the blog post to demonstrate the syntactic and search aspects of query strings.

Let us look at some typical examples one might come across while performing tag searches and their query strings.

Below are some Invalid queries. We need to be aware of the reasons that make them invalid and avoid such mistakes in the future.

## Query Syntax

A query is one or more AND condition filters that are ORed together. Each AND condition contains one or more query items. A query item consists of a query filter name, an operator, and the query filter. This allows multiple conditions to be specified with a single query. Query syntax described in Extended Backus-Naur Form (EBNF)

It is important grasp the ENBF syntax rules to construct correct and effective queries. As we go along this blog we will take a look at examples on how to do this and how to avoid potential pit falls one may encounter with query strings. There are a large number of possible constructs filled with many nuances, however if we gain an understanding of some standard rules this the task becomes a lot easier.

As an example, the below query strings(non-exhaustive list) represent the exact same query even though they vary syntactically

• sin* AND PointType:Float
• (tag:=sin* AND PointType:=Float16) OR (tag:=sin* AND PointType:=Float32) OR (tag:=sin* AND PointType:=Float64)
• (sin* PointType:='Float16') OR (sin* PointType:='Float32') OR (sin* PointType:='Float64')
• tag:=sin* AND PointType:Float
• ("sin*" PointType:='Float16') OR ("sin*" PointType:='Float32') OR ("sin*" PointType:='Float64')

### How can we parse a Query String?

Parsing can be viewed as decomposing a query string into separate conditions. Think of this as an 'exploded view' of the string where you can see how the individual components fit together. PIPointQuery is a structure in which PIPoint attribute specified by the AttributeName in the query is compared to the query's AttributeValue using the search Operator. PIPointQuery.ParseQuery Method parses the query string into PIPointQuery lists which can be used in used by the FindPIPoints(PIServer, IList<IEnumerable<PIPointQuery>>, IEnumerable<String>) method and also to verify the equivalence of search strings.

The example strings provided above would be transformed into the equivalent PIPointQuery list.

Note: Parsing the query string to PIPoint Query Lists in the examples are shown in order to help understand various aspects involved query string parsing. In most cases this is not necessary if one gains a good understanding of query syntax.

I highly recommended always using Query Strings which are more compact and can be used in code and as well as Tag Search dialogues for PI Point searches.

CAUTION: Parsed Query does NOT mean Valid Query (Syntax vs Semantics)

If a query string is parsed successfully it only indicates correct syntax. However, Syntactic correctness does not guarantee Semantic validity. In this trivial example, it is easy to see that Float1234 is not a valid point type, however it can still be parsed into a PointQuery structure as it conforms to the grammar rules.

The search performed using the query string will obviously fail as shown.

### Use of Wild Card Characters

• The string value of a filter can be enclosed in single quotes ('), double quotes ("), or without quotes. Quotations are required if non-escaped white space or quotation marks are desired within the filter string.
• Single backslash (\) character is treated as a literal character unless followed by a wildcard character
• Supported wild card characters are "*" to match any zero or more characters and "?" to match a single character. These characters cannot be escaped using the backslash ("\") character

Ex: Search tag names with pattern CD?1?8

Ex: Search all tags which have datasecurity of PI World (read or write, but not both) and which do not belong to point class with name starting with ba*

### Alias Attribute Names

The following table lists the supported aliases for common PIPoint attribute names.These aliases can be used instead of the actual attribute name. The PICommonPointAttributes class contains the names of the common PIPoint attribute names.

Ex: Query strings 1 & 2 use aliases producing the same results

Notice the equivalent parsing for aliases.

Personal preference: I avoid using the aliases. One less thing to remember or make mistakes with.

### Operators

• EqualOperator can be specified either by ":" | ":="
• Personal preference: I use := to be consistent with the use of other operators
• PIPointValueFilter "Value" query if the PIPoint being queried is String type: LessThan, LessThanOrEqual, GreaterThan, GreaterThanOrEqual are not supported
• PIPointValueFilter query with BooleanValue (i.e "Substituted", "Questionable", "Annotated", "IsGood"), only Equal and NotEqual are supported
• The In operator is not supported. It will be implicitly translated as a filter value
• Name:"IN("abc", "def")" this is implicitly translated to 'Tag:="IN("abc", "def")*"'

## Syntax Rules: Cheat Sheet

• AndOperator can be specified either by "AND" or <WHITESPACE>
• Ex: AND is implied between pointtype and pointsource just by a space

• EqualOperator can be either ":" or ":="

• If a specific filter name is not specified, then the filter will default to the "Tag" filter and the operator will be "="

• When a filter name is specified, no whitespace is allowed between the filter name, the ":" separator, and the optional operator.
• If the operator is not specified, the default operator is "=".
• If the type of a point attribute is DateTime, then the "TimeValue" format is supported for the filter value. This can be any recognized AFTimeString
• Boolean can be specified by "True" or "False" or "1" or "0"
• PointType:Float query is implicitly translated to 'PointType:=Float16 OR PointType:=Float32 OR PointType:=Float64'
• PointType:Int query is implicitly translated to 'PointType:=Int16 OR PointType:=Int32
• Starting in AF 2017, it also supports querying based on PIPoint Value. OR condition is not supported if querying based on PIPoint value.
• Queries with OR condition are not supported for PIPointValueFilters.

• A filter name may only be referenced one time per AND condition of the query string.
• This example would cause an error: PointId:>5 AND PointId:<10
• It is possible to construct queries which include multiple attributes and query conditions

• Certain PIPoint Attributes are specific to a PIPointClass (Eg. AutoAck is applicable to ALARM & SQC_ALARM)
• See this attachment (ptclassattr.txt) for each PointClass attributes and their typical values
• Future point attribute, which is invalid, to a PI Data Archive version < 3.4.395
• Security point attributes (e.g. "PtSecurity" and "DataSecurity"), are invalid for PI Data Archive version < 3.4.380
• Query strings are Case Insensitive
• Don't use quotes unless you need them, single quotes better than double
• Don't use parenthesis unneccessarily

### SearchNameAndDescriptor

If True and the Tag attribute name is specified and the Descriptor attribute name is not specified in the query, then both of these attributes will be searched using the Tag query value

### AFSearchTextOption

Indicates the text search option to be applied to the search pattern.

1. StartsWith 2. Contains 3. ExactMatch 4. EndsWith

A good way to perform these same searches and check your queries used in your application is through Tag Search dialog in AF Explorer. You can open this by being in Elements View -> Search -> Tag Search.

Additional Search criteria can be specified through the UI. However all the attributes are not available through this. They can only be supplied when using the search string.

## Bonus: Peek into PI Server

AF SDK makes a remote procedure call to the PI Server (PI Base Subsystem) which takes in the search parameters and returns the requested PI Points along with additionally specified attributes.

As a bonus you can run piartool -thread pibasess -history in your PI Server command line to track the RPC and see the number of points returned and the amount of time it took for it to run.

Example RPC output: 4452, 0, 14-Aug-18 13:37:01.63263, 1, piptsdk|1|GetPoints, 544, Return Count: 55. Returned Status: [0] Success

# PIPoint Search Utility

Posted by tramachandran Sep 5, 2018

## Overview

This console utility was developed to demonstrate PIPoint Search Syntax in AF SDK and as a aid to accompany the examples shown in the blog post  PIPoint Search Query in AF SDK

As a stand alone tool, it provides a quick way to perform searches and verify syntactic correctness of query strings.

## Usage

### 0. Connect to PI Data Archive

This is required for both searching for PI Points and Parsing Query Strings as certain attributes depend on the version of the server.

### 1. Search PI Points using a Query String

Output columns: Tag name, Point ID, PointType, PointClass

### 3. Specify SearchNameAndDescriptor

If true and the Tag attribute name is specified and the Descriptor attribute name is not specified in the query, then both of these attributes will be searched using the Tag query value. Default = false

### 4. Specify AFSearchTextOption

Indicates the text search option to be applied to the search pattern. Default = StartsWith

# PI Data Archive container health check

Posted by Eugene Lee Sep 3, 2018

Note: Development and Testing purposes only. Not supported in production environments.

Containerization Hub

Introduction

In my previous blog on AF Server container health check, I talked about implementing a health check for the AF Server container. Naturally, we will also have to discuss about such a check for the PI Data Archive container. For an introduction to what a health check is about and also how you can integrate a health check with Docker. Please refer to the previous blog post as I won't be repeating it here.

In part 1, I will be covering the definition of the health tests that we can do for the PI Data Archive and then we will hook them up in the Dockerfile.

In part 2, we will be doing something interesting with these health check enabled containers by using another container that I wrote to inform us by email whenever there is a change in their health status so that we are aware when things fail.

Without further ado, let's jump into the definition of the health tests for the PI Data Archive container!

Define health tests

There are 2 tests that we will be performing. The first test is a test on the port 5450 to determine if there are any services listening on that port. The second test will use piartool to block for some essential subsystems of the PI Data Archive with a fixed timeout so that the test will fail if it exceeds that timeout.

The Powershell cmdlet Get-NetTCPConnection can accomplish the first check for us. A return value of null means that there is no service listening on port 5450.

The relevant code is below

$val = Get-NetTCPConnection -LocalPort 5450 -State Listen -ErrorAction SilentlyContinue if ($val -eq $null) { # return 1: unhealthy - the container is not working correctly Write-Host "Failed: No TCP Listener found on 5450" exit 1 }  Next, piartool is a utility that is located in the adm folder in PI Data Archive home directory. It has an option called "block" which waits for the specified subsystem to respond. This command is also used in the PI Data Archive start scripts to pause the script until the subsystem is available. The subsystems that we are going to check is the following list. $SubsystemList = @(
@("pibasess", "PI Base Subsystem"),
@("pisnapss", "PI Snapshot Subsystem"),
@("piarchss", "PI Archive Subsystem"),
@("piupdmgr", "PI Update Manager")
)


We are going to change the amount of time that we allow for each check to 10 seconds so that we do not have to wait 1 hour for it to complete . We will also grab the start and end times so that we can provide detailed logging for troubleshooting purposes. The code for this is below.

function Block-Subsystem
{
Param ([string]$Name, [string]$DisplayName, [int] $TimeoutSeconds= 10)$StartDate=Get-Date
$rc = Start-Process -FilePath "${env:PISERVER}\adm\piartool.exe" -ArgumentList @("-block", $Name,$TimeoutSeconds) -Wait -PassThru -NoNewWindow
$EndDate=Get-Date if($rc.ExitCode -ne 0)
{
echo ("Block failed for {0} with exit code {1}, block started: {2}, block ended: {3}" -f $DisplayName,$rc.ExitCode,$StartDate,$EndDate)
exit 1
}
}

ForEach ($Subsystem in$SubsystemList) {Block-Subsystem -Name $Subsystem[0] -DisplayName$Subsystem[1] -TimeoutSeconds 10}


Integrate into Docker

We will add this line of code to our Dockerfile to make Docker start performing health checks.

HEALTHCHECK --start-period=60s --timeout=60s --retries=1 CMD powershell .\check.ps1


The start period is given as 60 seconds to allow the PI Data Archive to start up and initialize properly before the health check test results will be taken into account. A time out of 60 seconds is given for the entire health check to complete. If it takes longer than that, the health check is deemed to have failed. I also gave only 1 retry which means that the health check will be unsuccessful if the first try fails. There is no second chance! .

Build the image

As usual, you will have to supply the PI Server 2018 installer and pilicense.dat yourself. The rest of the files can be found here.

elee3/PI-Data-Archive-container-build

Put all the files into the same folder and run the build.bat file.

Once your image is built, you can create a container.

docker run -h pi --name pi -e trust=%computername% pidax:18


Now check docker ps. The health status should be starting.

After 1 minute which is the timeout period, run docker ps again. The health status should now be healthy.

Health monitoring

Now that we have a health check enabled container up and running, we can start to do some wonderful things with it. If your job is a PI administrator. don't you wish there was some way to keep tabs on your PI Data Archive's health so that if it fails, an email can be sent to notify you that it is unhealthy. This way, you won't get a shock the next time you check on your PI Data Archive and realize that it has been down for a week!

I have written an application that can help you monitor ANY health enabled containers (i.e. not only the PI Data Archive container and the AF Server container but any container that has a health check enabled) and send you an email when they become unhealthy. We can start the monitoring with just one simple command. You should change the following variables

Name of your SMTP server: <mysmtp>

Destination email: <operator@osisoft.com>

docker run --rm -id -h test --name test -e smtp=<mysmtp> -e from=<admin@osisoft.com> -e to=<operator@osisoft.com> elee3/health


Once the application is running, we can test it by trying to break our PI Data Archive container. I will do so by stopping the PI Snapshot Subsystem since it is one of the services that is monitored by our health check. After a short while, I received an email in my inbox.

Let me check docker ps again.

The health status of docker ps corresponds to what the email has indicated. Notice that the email even provides us with the health logs so that we know exactly what went wrong. This is so useful. Now let me go back and start the PI Snapshot Subsystem again. The monitoring application will inform me that my container is healthy again.

The latest log at 2:30:47 PM has no output which indicates that there are no errors. The logs will normally fetch the 5 most recent events.

With the health monitoring application in place, we can now sleep in peace and not worry about container failures which go unnoticed.

Conclusion

In addition to what I have shown here, I want to mention that the health tests can be defined by the users themselves. You do not have to use the implementation that is provided by me. This level of flexibility is very important since health is a subjective topic. One man's trash is another man's treasure. You might think a BMI of 25 is ok but the official recommendation from the health hub is 23 and below. Therefore, the ability to define your own tests and thresholds will help you receive the right notifications that are appropriate to your own environment. You can hook them up during docker run. Here is more information if you are interested.

Source code for health monitoring application is here.

elee3/Health-Monitor

By date: By tag: