Imagine you are the OSIsoft technical support engineer answering an urgent call for help from a customer who has discovered malware on their PI Systems. Yikes!
This is not a drill; a few cases of this type have been reported recently. Given the rarity of such calls we don't have a set playbook. Formal incident response plans are something we are actively working on. So what was observed and how did we do in these cases?
Observation #1 - OSIsoft 24x7 technical support access was essential for timely response.
Perhaps it's just my skewed memory but emergencies seem to occur more frequently afterhours, weekends and holidays. Irrespective of coincidence or malicious intent, problems with mission critical systems demand a timely response.
In one of the recent cases, OSIsoft communication with the customer 'followed the sun' to prescribe advice and address concerns raised in follow up questions.
Observation #2 - Disaster recovery takes on some extra considerations.
Lingering concerns about system security are among the more insidious consequences of cyber breach. Extra security tasks after disaster recovery include making sure credentials that could have been exposed are changed. In addition to Windows service accounts, attackers gaining full administrative access could potentially harvest credentials from the PI System. Of particular concern are credentials configured for external data sources (AF tables, data sets, and interfaces). These credentials should be changed regardless of encryption technique. Auditing and security monitoring should be instrumented to alert on attempts to use the presumed compromised credentials.
Observation #3 - Reluctance to share indicators of compromise is common.
While it is good advice to avoid sharing unnecessary details of cyber breach this posture can partially hamstring subject matter experts (kind of like a patient not sharing all symptoms with the doctor). My advice for critical infrastructure operators is to seek guidance from a professional incident response team. Certainly if there is any indicator specifically related to an OSIsoft application it is appropriate to facilitate community level awareness and response.
Corporate IT departments and anti-virus companies have carried the ball for years in responding to 'run of the mill' computer viruses. Today, this seemingly endless game of cat and mouse has expanded into the realm of industrial software applications and critical infrastructure operators.
Metaphorically like risk of fire... strategies to help prevent, detect and respond to cyber incidents make sense. Specifically for response we intend to follow a well-developed incident response plan and avoid the proverbial chaos of 'shouting fire in a movie theater'. While cyber incidents affecting our customers continue to be quite rare, recent trends suggest OSIsoft technical support engineers will indeed be among first responders. From these cases we've identified opportunities to improve especially along the lines of communication norms and disaster recovery procedures.