Skip navigation
All Places > Security > Blog
1 2 Previous Next

Security

28 posts

We're doing it again.  This time in Dallas!  On November 13th, OSIsoft is hosting the Texas Regional Seminar in Dallas, TX.  There will a be a good mix of talks on our product roadmap and customer success stories.

 

As a FREE pre-event session on November 12th from 1:00 to 5:00 PM, we are hosting a Cyber Security workshop on the PI System with emphasis on Software Supply Chain Security.  It will be a lively afternoon with guest speakers, Jonathan Butts, Chief Technology Officer, aDolus Inc., and Jeff Edwards, Co-Founder, TSI Group.  We'll demonstrate tools and techniques to defend supply chain attacks and top the day with a table top exercise.

 

The brochure for the security workshop is attached below.  You can view the full agenda for the Texas Regional Seminar here.  We invite you to join us!

 

We hope to see you there!

We share this special invitation to the Digital Bond S4x19 ICS Security Conference sponsored by OSIsoft, LLC.

 

The S4x19 conference is taking place in Miami Beach, January 15-17, 2019.  The three-day event brings together the top researchers, thought leaders and influencers in the industrial control system (ICS) community to discuss advance topics in cyber security and operations technology. You'll find the agenda and other event information on the S4x19 Conference site.  To see previous talks and learn what S4 is about, check out the S4 Events YouTube Channel.

 

OSIsoft is proud to announce, for the fourth year running, we are sponsoring the S4 Capture The Flag (CTF) competition. The "Killer Robots, Inc." PI System submitted by OSIsoft is designed to be an interactive, fun source of industrial security challenges.  We’ll take a PIe in the face if you don’t learn something new about PI System security.

 

As a hands-on, "practice by doing" experience, CTF is a proven way to develop mastery of cyber security. The competition also provides a safe opportunity to flip the “evil-bit” to explore where misuse and abuse can turn software against its intended purpose. Open to first-time teams or those with more experience, hunting flags with our subject matter experts on hand provides a unique deep-dive into PI System security.  Those with PI System skills have a distinct advantage in the quest for the grand prize, an S4 Black Ticket, free entry for a lifetime for the S4 event in Miami Beach.

 

Whether or not you decide to take the CTF challenge, please come and experience the community, join the conversation and learn from the best in ICS security. Also, for those new to ICS security, the S4 ONRAMP Training day is a great way to get started.

 

As an added incentive, if you or anyone else on your team are able to attend, please reach out to me and receive a 10% discount as a customer or partner of OSIsoft. 

 

"Good Flag Hunting!" from the OSIsoft Security Advisors.

Brian

Security Workshop in D.C.

Posted by Brian Employee Aug 8, 2018

Coming this month on August 21-22, OSIsoft is hosting a Super Regional Seminar.  This is a special two day event taking place in Washington D.C.  There will a be a good mix of talks on our product roadmap, customer stories as well as guests speakers from the Department of Energy (DoE) and the Nation Institute for Standards and Technology (NIST).

 

There are two pre-events to the super-seminar. A PI System 101 learning session and cyber security workshop on the PI System with emphasis on Software Supply Chain Security.  It will be a lively afternoon with guest speakers, Allan Friedman, Director of Cybersecurity at National Telecommunications and Information Administration in the US Department of Commerce, and Ben Miller, Director, Threat Operations Center at Dragos, Inc.  We'll demonstrate tools and techniques to defend supply chain attacks and top the day with a table top exercise.

 

The brochure for the security workshop is attached below.  You can view the full agenda for the Super Regional 2018 here.  We invite you to join us!  If you would like to only attend the afternoon security workshop, please register with the code, "SuperReg18Security"

 

We hope to see you there!

PI World 2018 is just around the corner.  The kick-off reception is one week from this evening.  As we do each year, we would like to share the security related activities that are happening during the conference.

 

The most important opportunity is to talk one-on-one with our security team at the Cyber Security Expo Booth.  Our booth will be open every day along with the Partner Expo.  Please stop by and chat about securing the PI System, protecting your industrial systems, new regulations, current events and "what's with those wooden bow-ties?"

 

Other activities are listed in the follow table.

Agenda.png

For details on these talks and labs, please see the PI World agenda.

 

We would also like to draw your attention to a number of security related partners that will have booths in the Partner Expo.  Please be sure to stop at each of these great partners to see what they have to offer.

Partners.png

 

Safe traveling. We'll see you next week!

The OSIsoft Security Advisors and Champions team

Although introduced in Windows Server 2012, the Group Managed Service Account (gMSA) still has low adoption within our customer base. This blog post aims to highlight benefits of gMSAs, discuss how to deploy and use them, and offer some tips & tricks.

 

Built-in accounts such as NetworkService or LocalSystem have decent password management, are simple to use, and can easily run a service or an Application Pool. However, outside the machine boundary, they take on the identity of their host machine (COMPUTERNAME$). This limits their usefulness outside the machine itself, as granting access to a computer object inevitably gives access to all other built-in identities on the computer (and all the services they're running!).


Many companies use standard domain accounts instead, but these accounts suffer from very poor password management (entropy, expiration/resetting, maintenance). Moreover, standard domain accounts' passwords are inevitably exposed to both users (who need to regularly interact with the passwords - typing or copy/pasting) and computers (passwords are stored locally).

 

gMSAs combine the best of both worlds: automatic password management with secure & centralized storage, while maintaining uniqueness outside the machine boundary. gMSA's password is calculated on-demand by Domain Controller (KDC) and automatic password changes are done periodically. In contrast to passwords used by standard domain user accounts, gMSA passwords are not stored locally on computers nor exposed to users.

 

 

 

gMSA vs. Standard Domain User - Practical Password Play

 

Sticky notes are awesome! Right?

StickyNotes.png

 

 

gMSA password consists of 256 bytes of data, interpreted as 128 UF-16 characters. It is a constructed attribute calculated by the Domain Controller (KDC) on-demand.

 

Let's take a look using PowerShell:
# Download the DSInternals module

Save-Module DSInternals -Path C:\share

 

# Import the required modules

Import-Module ActiveDirectory

Import-Module C:\share\DSInternals\2.22\DSInternals.psd1

 

# Get the password blob – need to explicitly ask for the msDS-ManagedPassword attribute

$gmsa = Get-ADServiceAccount -Identity PVizgMSA -Properties msDS-ManagedPassword

$mp = $gmsa.'msDS-ManagedPassword'

 

# Convert the blob to Unicode string

$pw = ConvertFrom-ADManagedPasswordBlob $mp

$pw.CurrentPassword

 

This yields the actual Unicode string representation:

挓ⶎ↙㓋禆哕⫼싈什⢪ル⇮䟎줨酯疅훽ᗀ䅳椃쁁곫⽔面㽚奅ꁣ鼆켙챨๩ὄ롦鹌䜡੉綝΋䏧䊷沯秩Е붅凡Ⅺ䤻韄꣰ᚓ렡묻㲝츷薱⭧各렱ྀຓ㱇듉硛죝힃르᰺궜ᐧ꩸ꦞ䡧٪咹햶跼쓔푁髁㨟⑏㯙㫪雇汏䋌㦢ᔙ口搪䛳橤登猈著㞫奉诠㊂柽톯륩괰ᛶ㶪£篫ຶွ蟪䕄

 

For more details, check out DSInternals’ post on retrieving cleartext gMSA passwords.

 

 

As an example, let's take a look at the two IIS Application Pools shown below - one is running under a standard domain user, while the other runs under a gMSA (an easy way to spot a gMSA is by the trailing $ character, much like a computer object).

Since the password of the standard domain user is stored locally on the server, a local administrator can obtain the password in plain text, which means there is no role separation between the service administrator and the local computer administrator! The gMSA password, on the other hand, is not stored locally on the server and therefore can't be retrieved.

 

AppPools.png

In this case, we’ve read the standard domain user password from applicationHost.config file.

 

But it’s the same story with Windows services. The only difference is that the passwords are stored in the registry (within HKEY_LOCAL_MACHINE\SECURITY hive) rather than in a .config file:

 

Registry.png

 

While we’re on the topic of passwords, I would be remiss if I didn't mention mimikatz - the Swiss Army Knife of Windows Authentication. We’ll cover mimikatz and attacks on credentials (pass-the-hash, pass-the-ticket, plus mitigation and detection strategies) in a future blog post.

 

gMSA Passwords – main takeaways:

  • Incredibly high entropy making brute force attacks impossible
  • gMSA is denied interactive logon, so even with the raw data for the password, capabilities for using it are limited

  • Obscure characters in the gMSA password natively protect against mismanagement or accidental leakage

 

While gMSAs provide some hardening and additional security against credential theft, they do not address all threats. Deploy gMSAs to complement other security best practices such as least privilege.

 

 

 

gMSA – Provisioning and deployment

 

Requirements:

  • Windows Server 2012+ on at least one Domain Controller.
    • Functional Level of the domain must be at least 2003, but 2008 R2+ is recommended.
  • Windows Server 2012+ OS installed on the machines hosting services that will use the gMSA.
  • Key Distribution Service (KDS) Root Key must exist to enable gMSA creation.
    • Microsoft strongly recommends using a single KDS Root Key. See the documentation for more details.

 

To create the KDS Root Key, run PowerShell command:
If (-Not (Get-KdsRootKey)) {

Add-KdsRootKey

}

 

By default, it takes 10 hours until the KDS Root Key is in effect (this is to provide sufficient time to replicate the key in large domains). However, in a test environment, we can force it to come into play immediately:

If (-Not (Get-KdsRootKey)) {

Add-KdsRootKey -effectivetime ((get-date).addhours(-10))

}

 

Next, use New-ADServiceAccount PowerShell cmdlet and specify (at least):
Name gMSA account name (must be fewer than 15 characters)
DNSHostName gMSA DNS account name (FQDN of the gMSA)
PrincipalsAllowedToRetrieveManagedPassword Principals allowed to obtain gMSA’s password. For manageability, it’s best practice to create a Domain Group (type: Security) and add the machines hosting services the that will run under the gMSA. However, it’s also possible to specify the computer(s) directly.

 

Optionally, specify other parameters such as ServicePrincipalNames and PrincipalsAllowedToDelegateToAccount, to configure Kerberos Authentication (SPNs) and Kerberos Delegation upon gMSA creation. To change the default password reset period (30 days), use ManagedPasswordIntervalInDays parameter.


Recommendation: Read the official New-ADServiceAccount cmdlet documentation before proceeding further.

 

Example 1:

The steps outlined below create a gMSA called PIVizgMSA, designed to run PI Vision Application Pools on two servers (piviz1 and piviz2). The appropriate Service Principal Names are created as well.

 

1. Create a domain group called PI Vision Servers and add piviz1 and piviz2 computers into the group.

 

2. Create the gMSA: (execute the command in an elevated PowerShell console on your Domain Controller)

New-ADServiceAccount -Name PIVizgMSA -DNSHostName PIVizgMSA.domain.local -PrincipalsAllowedToRetrieveManagedPassword "PI Vision Servers" -ServicePrincipalNames http/piviz1, http/piviz1.domain.local, http/piviz2, http/piviz2.domain.local

 

3. Remove all Kerberos Tickets from LocalSystem session on piviz1 and piviz2 computers by running klist -li 0x3e7 command in an elevated Command Prompt (alternatively, restart both computers). This is only required because a Domain Group was specified as the value for the PrincipalsAllowedToRetrieveManagedPassword parameter. If the machines were specified directly (i.e. -PrincipalsAllowedToRetrieveManagedPassword piviz01$, piviz02$), there’s no need to update Kerberos tickets (restart the machines).

 

4. Optional: Allow the account to write its own Service Principal Names – useful for services that support automatic SPN registrations such as PI Data Archive, PI AF Server or MS SQL Server.

(execute the command in an elevated PowerShell console on your Domain Controller)

$gMSA = Get-ADServiceAccount -Identity PIVizgMSA

dsacls $gMSA.DistinguishedName /G "SELF:RPWP;servicePrincipalName"

 

5. Use the gMSA to run PI Vision AppPools on the target machines (piviz1 and piviz2). Use it in the same way as a standard Domain User, just don’t specify the password (make sure the Password box is empty) and let the computer retrieve the password.

 

 

Example 2:

The steps outlined below create gMSA called AFgMSA, designed to run AF Server service on machine piaf01. Included are the appropriate SPNs and configuration of Kerberos Constrained Delegation (Resource Based Kerberos Delegation in this case) – where the PI Vision gMSA created in Example 1 (PIVisionGMSA) will be allowed to delegate to the AF Server piaf01 running under AFgMSA$.

 

1. Create the gMSA:

New-ADServiceAccount -Name AFgMSA -DNSHostName AFgMSA.domain.local -PrincipalsAllowedToRetrieveManagedPassword piaf01$ -ServicePrincipalNames afserver/piaf01, afserver/piaf01.domain.local -PrincipalsAllowedToDelegateToAccount PIVisionGMSA$

 

 

2. Allow the account to write its own Service Principal Names – AF Server supports automatic SPN registration:

$gMSA = Get-ADServiceAccount -Identity AFgMSA

dsacls $gMSA.DistinguishedName /G "SELF:RPWP;servicePrincipalName"

 

3. Use the gMSA on the target machine.

 

 

Example 3:

The steps outlined below create a gMSA designed for PI Web API service running on machine piweb01, including appropriate SPNs and configuration of Kerberos Constrained Delegation (traditional Kerberos Constrained Delegation in this case) – where the newly created PI Web API gMSA will be allowed to delegate to PI Server pi01.

 

1. Create the gMSA:

BackendSPNs = 'PIServer/pi01', 'PIServer/pi01.domain.local'

New-ADServiceAccount -Name PIWebAPIgMSA -DNSHostName PIWebAPIgMSA.domain.local -PrincipalsAllowedToRetrieveManagedPassword piweb01$ -ServicePrincipalNames http/piweb01, http/piweb01.domain.local -OtherAttributes @{'msDS-AllowedToDelegateTo'=$BackendSPNs}

 

 

2. Allow the account to write its own Service Principal Names – in case the AF machine name changes, for example.

$gMSA = Get-ADServiceAccount -Identity PIWebAPIgMSA

dsacls $gMSA.DistinguishedName /G "SELF:RPWP;servicePrincipalName"

 

3. Use the gMSA on the target machine.

 

 

 

If any modifications are required after the gMSA is created, use the Set-ADServiceAccount cmdlet. See the official documentation for details.

 

 

 

gMSA in PI Environment


When specifying gMSA identity in Services.msc snap in or in IIS Manager, simply type in the name of the account and leave the password box blank:

 

Services.png


Tip: After the Log On account of a Service is set to a gMSA, the Log On tab will be permanently unavailable (grayed out). To fix this, open an elevated Command Prompt and execute sc managedaccount <serviceName> false.

 

PowerShell can be used to use gMSA for IIS Application Pools and Services.

 

IIS Application Pools:
# Define gMSA and AppPool name

# Change the two variables below as needed

$gMSA = "DEN\PISQgmsa$"

$AppPoolName = "PIVisionServiceAppPool"

###

# Import WebAdmin module

Import-Module WebAdministration

 

# Set AppPools to use gMSA

$AppPool = Get-Item IIS:\AppPools\$AppPoolName

$AppPool.processModel.identityType = 3

$AppPool.processModel.userName = $gMSA

$AppPool.processModel.password = ''

$AppPool | Set-Item

 


Services:

# Define gMSA and Service name

# Change the two variables below as needed

$serviceName = 'PIOLEDBENTAgent'

$gMSA = "DEN\PISQgmsa$"

###

$service = Get-WmiObject -Class Win32_Service -Filter "Name='$serviceName'"

$service.Change($null, $null, $null, $null, $null, $null, $gMSA, $null, $null, $null, $null)

 

 

PI Data Server 2017 R2+

  • Use Services.msc or PowerShell to switch PI Network Manager service (pinetmgr) to run under the gMSA.
  • Older PI Data Servers do not support custom accounts.

 

PI AF Server

  • Use Services.msc or PowerShell to switch the AF Server service (afservice) to run under the gMSA.


PI Vision

  • From Command Prompt, execute aspnet_regiis.exe -ga domain\gMSA$ to give the account access to Microsoft .NET Temporary folder and IIS meta database.
  • Use IIS Manager or PowerShell to switch PI Vision Application Pools to run under the gMSA.

 

PI Web API 2017+

  • Run through PI Web API Admin Tool and select the default NT Service accounts.
  • Use Services.msc or PowerShell to switch PI Web API and PI Crawler services to run under the gMSA.
  • Re-run PI Web API Admin Tool and make no changes. The tool will automatically grant all required permissions to the gMSA.

 

PI Connectors

  • Add the gMSA to PI Connector Administrators local group as this group is automatically granted all the required permissions.
  • Use Services.msc or PowerShell to switch PI Connector to run under the gMSA.

 

PI SQL DAS

  • Install kit supports gMSA and sets permissions appropriately.

 

 

 

Resource-based Kerberos Constrained Delegation (RBCD)

 

Configuring RBCD is not limited to gMSAs, but the requirements are similar – if you can use gMSAs, you can also use RBCD. OSIsoft has good documentation on the requirements and configuring RBCD.

 

One of the big advantages of RBCD is that service administrators can configure it themselves (no need to contact Domain Administrators).

 

Example:

 

 

1. Open elevated Command Prompt on the machine with the gMSA-run back-end service (PI or AF Server).

 

2. Use Windows Sysinternals’ PsExec to open a PowerShell console under the gMSA identity:
PsExec.exe -u DOMAIN\gMSAname$ -p ~ powershell.exe

 

3. Domain users, including gMSAs, possess the required permission (write account restrictions) by default, so we can configure RBKCD as needed:

Set-ADServiceAccount -Identity gMSAname -PrincipalsAllowedToDelegateToAccount Account1, Account2 ..

For example, to allow Kerberos Delegation from PI Web API service running under PIWebAPIgSMA$ account to AF Server service running under AFgMSA$ account, execute:

Set-ADServiceAccount -Identity AFgMSA -PrincipalsAllowedToDelegateToAccount PIWebAPIgMSA$

 

 

4. Assuming the gMSA has been granted to write to its SPN attribute (as recommended and shown above), SPNs can now be set using SetSPN.exe:
setspn -s SPNClass/MachineHostName gMSAname$
setspn -s SPNClass/MachineFQDN gMSAname$

 

For example, to create HTTP SPNs for a PI Vision Service, execute:

setspn -s HTTP/piviz01 PIVizgMSA$

setspn -s HTTP/piviz01.domain.local PIVizgMSA$

 

 

References and recommended reading

 

Securing Privileged Access

Group Managed Service Accounts Overview

Getting Started with Group Managed Service Accounts

Windows Server 2012: Group Managed Service Accounts

CQURE: How To Use Group Managed Service Accounts (gMSA) vs. Service Accounts

DSInternals’ post on retrieving cleartext gMSA passwords

OSIsoft documentation: Resource Based Kerberos Constrained Delegation

There are a couple options to learn about PI System hardening on day 3 of PI World 2018. 

  1. The session Extreme PI System Hardening in developer track 4 at 10:30 AM is a “How-To” session that will take you step by step through hardening a system from the ground up.
  2. The PI System Anti-Hackathon lab on at 1:30 PM is a “Hands-On” session where you can learn about system hardening by experimenting in a sandbox environment with experts to guide you.

Both sessions will make heavy use of the PI Security Audit Tools.  Many are familiar with the audit module of the PI Security Audit Tools, which is used to identify gaps between the current state of a system and best practices.  At PI World 2018, we will introduce the PI Security DSC module which enables PI System administrators to manipulate the security configuration of their PI System components with PowerShell Desired State Configuration (DSC). By leveraging this module, PI System hardening can be implemented in a “configuration as code” paradigm.

Why use DSC for your PI System, you ask?

  • It’s declarative, separating intent, “What do I want to do?” from execution, “How do I want to do it?” This results in:
    • Less complex automation
    • More agility
    • Consistency across environments
    • Functional documentation
  • It’s broadly applicable, allowing you to cover broad scope with the same technology:
    • Use with applications and the underlying OS
    • Establish baselines or harden systems

Want to know more?  Full descriptions for each PI World session below!

 

How-To: Extreme PI System Hardening (Developer Track Presentation)

High value systems warrant hardcore hardening measures. The PI System resides at a critical junction, communicating across strict network boundaries. Under this paradigm, the PI System acts as a 'safe harbor' for data, defending critical systems by reducing the number of users inside the security perimeter while enabling growth in the number of users getting value from OT data. An application can only be as secure as its operating platform, so this session will start from the ground up. We will establish a solid foundation with advanced hardening measures for the Windows operating system that OSIsoft has collected over many years working with the platform, such as security baselines, PowerShell’s Desired State Configuration, and arcane corners of the Windows Advanced Firewall. With the platform locked down, we will explore application hardening measures built within and tailored to the PI System. Emphasis will be on using the latest technology and tools available to embrace agility and configuration as code. Examples from session demos will be available on GitHub for administrators who want to try them at home.

 

Hands-On: PI System Anti-Hackathon (PI System Admin Lab)

In this lab you will be served a big, soggy mess of a PI system – it’s your job to whip it into shape, by applying modern security techniques and best practices. You will have some help - handy scripts to identify the security holes are, references, resources, tips and coaching to help you accomplish your task. Participants will earn points based on the amount and the severity of security issues addressed. At the end of the lab, prizes will be awarded to top scorers. Moderately experienced administrators may have an advantage, but participants at all experience levels will learn concepts applicable to their systems back home.

 

Go here to register for the PI System Anti-Hackathon lab today!

US-CERT released the alert, Russian Government Cyber Activity Targeting Energy and Other Critical Infrastructure Sectors, on Thursday, March 15th.  The technical alert includes indicators of compromise (IOCs), technical details on the tactics, techniques, and procedures (TTPs) used by the actors, and security best practices relevant to the campaign. DHS and FBI published this alert with the expressed goal of empowering defenders to reduce their exposure to malicious activity.

 

Though the Systems Affected section of the alert explicitly identifies Domain Controllers, File Servers, and Email Servers as in scope, many of the defensive measures are relevant to the PI System as well.  The goal of this post is to highlight the measures in the General Best Practices Applicable to this Campaign section of the alert that are relevant to the PI System and point to resources that may assist with defensive efforts.

 

“Prevent external communication of all versions of SMB and related protocols at the network boundary by blocking TCP ports 139 and 445 with related UDP port 137. See the NCCIC/US-CERT publication on SMB Security Best Practices for more information.”

  • PI System core functionality does NOT require SMB, however SMB is a default feature of Windows and may be enabled on your system.  For guidance on SMB and the PI System, see AL00318, WannaCry Ransomware Attack FAQ.

 

“Segment any critical networks or control systems from business systems and networks according to industry best practices.”

 

“Ensure the use of PowerShell version 5, with enhanced logging enabled. Older versions of PowerShell do not provide adequate logging of the PowerShell commands an attacker may have executed. Enable PowerShell module logging, script block logging, and transcription. Send the associated logs to a centralized log repository for monitoring and analysis. See the FireEye blog post Greater Visibility through PowerShell Logging for more information.”

  • Most PI System administration tasks for the PI Data Archive and PI AF Server can be performed remotely with the PI System Management Tools, PowerShell Tools for the PI System, or PI System Explorer over PINet.  For remote administration to the OS, the MSDN blog post PowerShell Security at Enterprise Customers  is a comprehensive overview of the security features that make PowerShell the best choice.  Given the manageability and security benefits, we recommend installing the PI Server on the latest release of Windows Server Core and performing remote administration via PowerShell.

 

“Implement the prevention, detection, and mitigation strategies outlined in the NCCIC/US-CERT Alert TA15-314A – Compromised Web Servers and Web Shells – Threat Awareness and Guidance.”

  • We recommend installing PI Vision on the latest version of Windows Server Core for the reduced attack surface area.  Additionally, guidance for web security in PI Vision is covered in KB01631.

 

“Implement application directory whitelisting. System administrators may implement application or application directory whitelisting through Microsoft Software Restriction Policy, AppLocker, or similar software. Safe defaults allow applications to run from PROGRAMFILES, PROGRAMFILES(X86), SYSTEM32, and any ICS software folders. All other locations should be disallowed unless an exception is granted.”

  • Guidance for configuring application whitelisting on systems with PI applications using AppLocker is provided in  KB00994.

 

“Block RDP connections originating from untrusted external addresses unless an exception exists; routinely review exceptions on a regular basis for validity.”

 

“Ensure applications are configured to log the proper level of detail for an incident response investigation.”

  • Default configuration provides significant information.  See PI Data Archive monitoring in the LiveLibrary for more details on all PI Data Archive monitoring capabilities through PI Message Logs, Connection history and Windows performance counters.  PI System administrators can opt into PI Auditing, which records the data that is added, edited, or removed from database files, as well as other events or changes to configuration that occur in the PI Data Archive to satisfy FDA Title 21 CFR Part 11 auditing requirements.  See Auditing the PI Data Archive in the LiveLibrary for more information.  Enabling PI Audit is not recommended unless the default monitoring is insufficient.  PI AF Server client connectivity logging is covered in KB00412.   PI AF Audit Trail is described in the Audit Trail implementation  section of the LiveLibrary.

 

“Consider implementing HIPS or other controls to prevent unauthorized code execution.”

  • There are no known compatibility issues with Host Intrusion Prevention Systems and the PI System. Guidance for antivirus and antimalware solutions and the PI System provided in KB01062.

 

“Establish least-privilege controls.”

  • Permissions required for tasks in the LiveLibrary describes permissions required for common administration tasks. A practical role-based access implementation for Windows Integrated Security in the PI System is described in the PI Data Archive Field Service Technical Standard in KB01702.

 

“Based on the suspected level of compromise, reset all user, administrator, and service account credentials across all local and domain systems.”

  • Since the PI System leverages Windows authentication through PI Mappings, reset of Windows principals will impact PI System components.  In the event where this course of action needs to be taken, please contact tech support so that important aspects of recovery relevant to the PI System are not overlooked.

 

“Create and participate in information sharing programs.”

 

Hopefully the resources in this post help make the best practices relevant to this campaign more actionable for your PI System deployments. 

The OSIsoft Users Conference 2017 was a great event.  If you weren't able to attend, all of the papers are available online for you to see.  I'd like to highlight those with cyber security related information.  Please review them and share.  Feel free to comment or post questions here in the forum.  We'd enjoy your feedback and discussion.

 

We are busy preparing for the OSIsoft Users Conference next week. Hoping to see you there, we would like to share the security related activities happening during the conference.  Here are some suggestions for your conference agenda:

 

Day

Time

Agenda

Tue

2:30 PM

Security Discussion Forum

Tue

4:15 PM

Security in Your IoT Networks

Wed

2:15 PM

NIST / NCCoE Cyber Security Portfolio – Jim McCarthy

Thu

11:30 AM

How to Secure Your PI System: Security Baselining

Thu

2:15 PM

Lab: Extending the PI Security Audit Tools to Meet Your Needs

Thu

2:15 PM

Lab: Locking Your PI System Without Locking Down Your PI System

Thu

2:15 PM

Locking Down Your PI System Without Locking Down Your PI System

Thu

4:30 PM

What’s New in PI Security?

T-Th

12:30 PM -

Cyber Security booth in the Product Expo

 

In particular, we would like to point out two sessions dedicated to our new security self-assessment tool, the PI System Security Audit tool.

 

Protecting the PI System against malicious activities and incidental misconfiguration can be a demanding job. Industry standards, support KBs, and system utilities are available to assist, but it can be a challenge to collect, analyze and correlate this information effectively. Armed with all these tools and information, how do you tie it all together to effectively baseline the security of your PI Systems?  How do you evaluate which defenses should be prioritized? 

 

In How secure are your PI Systems? A primer for PI System security baselining we will explore the best practices to guard a modern PI System and how to get there.  The PI Security Audit Tools are featured as a low effort, repeatable method to acquire actionable information about a PI System that can be implemented based on your operational reality. 

 

The PI Security Audit Tools are a framework for security configuration auditing of PI System components in the form of a PowerShell module.  The hand-on lab Using and Building the PI Security Audit Tools, a tool to baseline your PI System security that teaches administrators how to leverage the tools, as well as showing system integrators and developers how to take advantage of the extensibility of the framework to meet the unique needs of their environment. 

 

We are looking forward to seeing you next week,

OSIsoft Security Team

An updated version of the PI Security Audit Tools is available for download from the tech support site here.  For a list of the changes/improvements in this version, check out the release announcement.

 

There is also an exciting opportunity to learn more about PI System security and baselining with the PI Security Audit Tools from Anna Perry at the upcoming UC 2017 lab, Extending the PI Security Audit Tools to Meet the Needs of Your Environment at 2:15 pm on the day 3 agenda.  I've included the description below.  If this topic interests you, please sign up when you register for UC 2017!

 

"How do you baseline the security of your PI Systems? How do you evaluate which defenses should be prioritized? These are questions every PI System administrator faces. The PI Security Audit Tools are a framework for security configuration auditing of PI System components in the form of a PowerShell module. In the first half of this lab, participants will learn how to use the existing tools to evaluate their PI System deployments and use the output to plan and prioritize improvements to their defenses. The second half of the lab will focus on the tools' extensibility. Participants will learn how to extend the PI Security Audit Tools existing libraries to include validation checks specific to their organization's needs and how to implement their own libraries with the tool."

The New Year is here already and with it marks the 10th annual Digital Bond S4 conference on industrial control system security.  Experts shared a mostly optimistic mood for OT security in 2017.  A summary of main stage highlights are below.  We’ll comment on technical deep dives and our capture the flag contest in subsequent posts (spoiler alert: clean shirts for us, ‘pie in the face’ flag still stands!).

 

Optimism stems from market leaders releasing new generations of security hardened solutions. Security development lifecycle (SDL) investments are bearing fruit and ushering in a welcome shift in technology.  At last, critical infrastructure providers can plot a course of upgrades and sunset their fragile, ‘insecure by design’ systems.

OSIsoft and the modern PI Server have contributed to this wide spread optimism with major releases in 2015 and PI API 2016 for Windows Integrated Security.  SDL hardened application code on Windows Server core along with reference architecture using high availability and web application services for PI visualization is a very effective defensive strategy for the PI System.

Some of the S4 presentations documented the high cost of ‘digital carelessness’ with case studies on ransomware affecting industrial control systems to the ongoing targeted attacks on Ukrainian critical infrastructure. It appears no amount of bolt-on security solutions can keep pace with threats.

The US Department of Justice National Security Division and global law enforcement have ramped up activities following breaches in central banking (Bangladesh) and the SWIFT global financial network. Banking was once considered sacrosanct. Domestically, we observe the Federal Trade Commission filing a complaint against DLINK for failure to take reasonable steps to secure routers and Internet-protocol cameras. This is perhaps a ‘shot over the bow’ that all IoT solution providers will be watching.

Richard Clarke (former National Coordinator for Security, Infrastructure Protection and Counter-terrorism) called for regulators to impose a deadline for addressing critical infrastructure protection. While conceding an unfavorable political climate to force such a mandate, Clarke cited Y2K as model for accelerating massive updates.

Whether you believe Y2K was a preemptive success or over blown farce, the example has an interesting parallel with SDL because remediation was also code centric. A key distinction however is identification of potential issues. Testing for Y2K was straight forward, however this isn’t the case for cyber security.

Methods for assessing software reliability and security are potentially endless. Our objectives this year include metrics to monitor SDL processes.  We are also studying ways to catalog and publish industry benchmark reports such as Microsoft binskim, Cyber-ITL.org, and Mozilla Observatory. 

Adoption rate of hardened software versions in the field is still an elusive metric. In the meantime we have initiatives designed to provide you with improved visibility over your PI System infrastructure and health. You can learn more and provide ideas in just a few months during UC 2017 in San Francisco!

Welcome to the final post of the Killer Robots series!  If you don’t already know about Killer Robots, Inc., see the December 7th post for an introduction.  The December 14th post provided last year’s system architecture and some PI security resources.  Last week’s post introduced this year’s environment and modeled the traversal to the web server with an early flag.

 

This week, as foreshadowed, we must go deeper.  The web server access following the compromised user was possible due to poor credential theft defenses in the environment.  Credential theft was the path of least resistance due to other defenses in place, such as authenticated access from normal users being limited to the client application since the admin site is implemented separately, strict import folder access preventing access to modify imported displays and IIS hardening to ward off attacks to the platform.

 

To control the impact of this user’s credentials falling into the hands of our competitors, the environment is set up with delegation, point level permissions and limited access assigned to the user.  Just what they need to do their job.  Or maybe slightly more…

 

While we took the above precautions, in the event we underestimated our competitors or overestimated our defenses, we also have snapshots of the environment, offline backups and a disaster recovery plan to restore the environment to them with minimal downtime to other competitors.

kr4.png

When putting together your toolkit for the CTF, you’ll want to include the following for taking on the PI challenges:

  • A machine or VM running a Windows operating system with PowerShell available
  • Windbg or your preferred tool for analyzing memory dumps
  • Your favorite network capture tool
  • Link to key PI security references

 

This concludes the PI and the Killer Robots, Inc. Environment series.  For any taking on the challenge, best of luck and I’ll see you in Miami January 10th!

Hopefully you are here for more of the inside scoop on the S4x17 CTF competition!  Last week’s post provided a primer on PI Security and the system architecture for last year.  This week you will get a sneak peek at the updated environment. If you don’t already know about Killer Robots, Inc., see the December 7th post for the first installment of this series to get up to speed.

When examined from the perspective of the functional layers of the PI System, the S4x17 CTF environment utilizes the operations and business scenario, where the data from one or many disparate sites is sent to a business network. This adds another collection layer in an intermediate DMZ as well as additional management and delivery on the business side.

The diagram below shows an architecture loosely based off the Life Sciences Reference Architecture.  The blue boxes represent assets that are deployed in the virtual environment for the CTF competition and the red flags indicate the quantity of relevant flags for each component.  As indicated, the challenges span the entire environment, from OT to IT.  So many f1r3w@llz, so little time!

But, as we all know, f1r3w@llz != security, they only represent one of many aspects. Our S4x16 environment architecture had 6 steps between the external users and the control system.  The new architecture, with proper network segmentation, lengthens the attack path by 2 layers.  The added components, PItoPI and the site PI Data Archive are framed below in green.  At each of these layers there are many potential threats, and our CTF competitors need to see which ones are missing barriers that allow them to traverse to the next layer.

A skilled attacker will follow the path of least resistance.  To provide a tangible example, let’s look at Bow Tie diagrams for the first two steps of the chain to illustrate the additional dimensions that can be successfully attacked when a system does not have a well-rounded security strategy. 

 

In the S4x16 CTF competition last year, the competitors needed to gain access to the web server. To do so, they first needed to compromise a user account as the first flag.  While there are many methods they could use to compromise a user, enumerated in the Bow Tie on the left below, there were defenses in place for almost all of them. The environment machines had application whitelisting enabled with AppLocker, firewall rules in place to limit traffic, antivirus, all recent OS patches, among other defenses.  Although none of these defenses are absolute, they increased the difficulty of other attacks such as infecting the system with malware.  The path of least resistance was insecure communication to the web server.  Competitors got a capture of snooped network traffic, where they found a connection to a web server using Basic authentication without TLS, which is of course a huge faux pas. 

 

Once the credentials were obtained from the network capture, the competitors could access the web server simply by logging in.  No need to find a 0-day and the firewall allows the requests since they appear as normal traffic.  The credentials obtained were for a non-administrative user, but the competitor could now see data and some configuration information that they could recon as they worked towards pivoting to the next layer.

While basic or forms authentication without TLS is an extreme example, it does happen, and the same concept applies to other attack vectors such as phishing or mismanagement of credentials. So, even with gratuitous firewalls, an environment is not impenetrable if other aspects are ignored. To make matters worse for the Killer Robots in S4x17 (or better for humanity if you’re the “glass half-full” type) the CTF network will be exposed to the players as a cross-section, so the CTF competitors will have direct access to several endpoints to attack… 

If you enjoyed reading this penultimate post, tune in next week for the final installment where we’ll dive a layer deeper into some of the specific barriers targeted by flags and give some tips for putting together your toolkit!  In the meantime, you can also check out the S4CTF twitter feed for updates about the other challenges.

Are you looking for an edge in the PI challenges of the S4x17 CTF competition?  If so, you’ve come to the right place!  Herein lie PI security principles and architecture information that will benefit any taking on the challenge.  If you want a refresher on what the CTF event is all about and the Killer Robots, Inc. environment, see last week’s post which gave background on the competition and what is at stake.

 

PI Security Primer

While there have been volumes written on PI security, we have provided a distilled view of the most pertinent and frequently requested information.  The one-stop shop for quick access to all things PI security is the PI System Cyber Security page on the support site.  If you work with (or in the case of the CTF competition, against) PI, then I highly recommend bookmarking this page, which connects you to documentation, KBs, tools, alerts and more.

 

For a prospective competitor looking for a crash course on priority defensive measures for a PI System, KB00833 – Seven best practices for securing your PI Server is a great place to start.  In this KB and across our resources you’ll find that Windows Integrated Security is a focal point.  Adoption of this authentication protocol to the PI Data Archive over trusts and explicit login provides stronger authentication and enables features such as transport security.  You may also want to add KB01295 - Risks of using the PI System as an input for a control system to your short list for review, to familiarize yourself with the risks and defensive measures associated with output points.  The coveted Golden PI flag may not be possible otherwise....

 

Architecture

With respect to architecture, the PI System can be logically represented as three functional layers based on the roles of components: Collection, Management and Delivery.  Each layer is represented graphically in Figure 1 below with the relevant software components at each layer.

  • Collect - PI Connectors and PI Interfaces pull in data from many disparate sources.
  • Manage - PI Server (PI Data Archive, PI Asset Framework) store, aggregate and contextualize data in a common format.
  • Deliver - PI Coresight and other access tools allow users to visualize data.

Figure 1: Functional layers of the PI System.

 

What a lovely graphic... but let's get to what this audience cares about.  Now that you have the functional layers and the related components, the next logical question is where they reside in a deployed system.  The OSIsoft environment submitted to the CTF competition for the S4x16 Conference in 2016 was modeled after a traditional operation scenario for the PI System, where the data is collected, managed and delivered in the OT space to be consumed by operators and engineers at a plant or facility.

Figure 2: PI System Roles in an Operations Scenario

 

An example architecture using this paradigm is represented below in Figure 3.  This more tangible example was taken from the Life Sciences Industry Reference Architecture.  Note that there are other reference architectures for several industries available here on PI Square, compiled based on best practices for the PI System and the needs of the specific industry.

 

Figure 3: Operations PI System deployment from Life Sciences reference architecture.

 

The highlighted components in Figure 4 are the minimum representative components that the OSIsoft team chose for the simplified architecture in the S4x16 CTF environment.  A few notes about each of the components are included below.  The flags superimposed on the image correspond to the placement of challenge flags throughout the environment.  Some of the flags deeper in the network could only be exposed after capturing flags upstream. 

  • Terminal server: provide access to PI utilities and thick clients such as PI ProcessBook.
  • Web server: host the PI Coresight web application for visualization of PI data.
  • PI Data Archive: provide access to historical and real-time data
  • PI AF Server: model physical assets to provide context to data streams, including PI AF Server as well as the PI Analysis Service
  • PI Interface node: feed data to the historian through the PI OPC DA Interface and a local OPC Server to act as a mock data source to represent information from the control system

 

In S4x17, we are expanding the environment to include the business network as well.  In this Operations and Business scenario, data from one or many sites, presumably separate plants or facilities, is sent to a centralized location for consumption. This architecture spans both the IT and OT networks.  The next post will show how the CTF environment architecture was updated for this scenario and start talking about kill chains

 

Figure 4: PI System Roles in an Operations and Business

killerrobot.png

If you are a PI System administrator, System Integrator or an otherwise security focused professional, you may be interested in the PI System environment at Killer Robots, Inc., the misanthropic company competitors will attempt to compromise at the S4x17 ICS Security Conference Capture the Flag (CTF) competition in a battle for the survival of mankind.  Aside from aiding in the struggle to liberate humanity from the merciless machines, this 3 day event is also a unique training opportunity, as it allows competitors to learn about PI security by going on the offensive on a live PI System environment submitted by OSIsoft alongside other vendors.

 

The OSIsoft team sought to create a PI System environment that highlights common mistakes, misconfigurations and misuse in a way that is both informative, and hopefully jarring. Inspiration was drawn from case studies, security engineering, 3rd party reports and our own experiences with vulnerability disclosure, so the exercises are grounded in practical application.

 

What does the CTF offer for a PI System administrator?

  • Exercise your skills in a simulated environment:  Not everyone has a development system or sandbox environment, so this is your opportunity to get a hands on experience exploring a PI System.
  • Cross-train with both IT and OT technologies: The CTF flags include targets against client applications such as the PI Coresight web application in the corporate zone, all the way back to output points associated with a PI OPC Interface in the plant network.
  • See some of the latest security features in action: Features such as transport security will be on display as well as the impacts when they are not.
  • Explore PI System internals: There are some gems hidden in the environment for the most avid PI geeks, such as some exploration into the SQL back end of PI Coresight.
  • Interact with developer technologies: Some exercises will require leveraging developer technologies such as the PI Web API.
  • Work on your OPSEC: Since many attendees are well versed in the art, you may even pick up some social engineering tricks to improve your OPSEC skills.

 

Throughout the month of December, we will discuss the philosophy, methodology and motivation behind the creation of the Killer Robots, Inc. PI System environment.  To a clever reader, these posts could provide a valuable primer for the competition, but to a clever competitor, the event should provide a better understanding of the PI System security.  The PI System challenges in the CTF will require a breadth of skills and knowledge from the competitors related to basic network packet and memory dump analysis, RESTful web services, PowerShell scripting, arcane ciphers, and most prominently, PI System administration. 

 

Tune into next week’s post for a survey of reference architecture, security engineering and best practices (or lack thereof) that informed the deployment of our target virtual PI System.  In the meantime, if this post has piqued your interest, check out the S4x17 Conference Site for more information or to register for the event.