Cyber-Security

As I was catching up on my RSS feeds for all the things going on in our world of process automation, I came across a cyber-security podcast interview conducted by Traci Purdum, senior digital editor for Chemical Processing magazine. She interviewed Eric Byres, who is the chief technology officer of Byres Security. I gave the 23-minute podcast a listen yesterday and sent the link to Emerson's Bob Huba, whom you may recall from earlier cyber security-related posts.

Borrowing a page from my playbook, Bob listened to the podcast during his commute. Eric made some strong points in the podcast that resonated with both Bob and me. Eric noted his perception that chemical plants assume they are more secure than they really are, mainly because the risks are invisible. You don't really see the risks until something happens. Bob notes:

I agree with Eric for the most part especially the part about needing and following procedures to remain protected. You are never "secure". There are still many process manufacturers who are waiting for or wanting the technology to solved the security problem for them.

Eric mentioned a great phrase, "security by obscurity", which painted a very clear picture of one of his points. Some process manufacturers stay with very old, proprietary automation systems hoping to avoid the security issues associated with more modern systems built on commercially available operating systems, databases, communication stacks, etc. He explained that even in these plants, there are Windows-based systems connected in. These connections introduce holes into the hope of remaining invisible.

Eric also makes a strong point that complexity is the enemy of security. To this point, Bob adds:

The part about being simple is a great comment--as you know we have a simple to use firewall for our controllers similar to the one mentioned. We have also introduced our DeltaV Smart Switch with easy to use security features. The DeltaV team strongly believes that "keep it simple" solutions will be the most successful.

One final part of the podcast that struck a chord with Bob:

He is on target that we need to have concerns about the current SCADA security solution movement to treat the SCADA system as an extension of the plant LAN and apply the same solutions. There is great complexity in many of these IT-based solutions that are simply not needed for many SCADA systems. Complex solutions may be necessary in the IT world. In the SCADA world, they just increase the complexity beyond what a control system maintenance person can hope to implement successfully--especially at 3am Sunday morning when something fails. There is almost a greater danger these solutions will create more operations problems than the security threats they are intended to mitigate.

If cyber-security is part of your set of concerns and you've got 23 minutes to spare, possibly on your commute, the podcast is worth a listen.

MP3 | iTunes (Sound quality is unfortunately noisy. The Chemical Processing one is much better!

Automation World has a great article, Security and Safety Follow Parallel Paths, which compares and contrasts process safety and cyber security, from a risk management perspective. In an earlier post, I described the ISA99 Working Group 7 (WG7) efforts to look at the best practices in process safety and see what can be applied to functional security around the automation systems.

The article quotes Emerson's Mike Boudreaux who serves as ISA99 WG7 co chair on similarities:

On the front end of the security lifecycle, where you're trying to figure out what your risks are, the kind of risk analysis that you do is very similar to the type of risk assessments that you do for safety, where you're identifying unwanted consequences, evaluating the likelihood that those might occur, and based on that, you have a level of risk that you need to implement safeguards against.

WG7 is taking a similar approach to process safety risk levels with security assurance levels (SALs):

In the safety world, standards such as the International Electrotechnical Commission's IEC 61508 and IEC 61511 describe methods for assigning Safety Integrity Levels (SILs) to designate different levels of risk reduction provided by a safety function. Similarly, the ISA99 committee is working on a parallel concept for security known as SAL--for Security Assurance Level. Just as Safety Integrity Levels range from SIL 1 at the low end to SIL 4 for the highest integrity level, the SAL approach, as currently contemplated, will cover SAL 1 through SAL 4, designating ascending levels of cyber-security protection.

This helps prioritize security risks and the defenses required for the risk level. The article also explored the differences between process safety and cyber security risk mitigation. A statistical view is taken with process safety. Mike notes:

The focus in the safety world is on designing devices that have predictable hardware failure rates. So when I install a device out there, I can predict how frequently it's going to fail throughout the life of the process for the next 20 years... But the concept of predictable, random failures doesn't apply as well to security... With security, when you put a protective measure in place, you can't predict what its useful life is going to be.

Emerson's Bob Huba, whom we've featured in many cyber-security related posts, describes the dynamic nature of security risk mitigation:

Safety is somewhat of a fixed process. Once you've got the risks figured out and the processes in place and you put the safety system in, it doesn't change... You put in antivirus software and its life is measured in days, because there's always something new--the next conflict, or the next Sasser worm... So it's constantly evolving, and the management on the security side is much more complex and onerous, in my opinion, than it is on the safety side.

It sounds like the working group is quickly identifying the parts of the process safety lifecycle that make sense to borrow and apply for process automation cyber-security. I tend to agree with Mike's prediction that the pace of the ISA99 standards effort will move more quickly than the ISA84 process safety effort, because they are borrowing what they can and developing the rest.

MP3 | iTunes

A recent Automation World article, Cyber Security--A Must For the Grid, describes how cyber security has become a big issue for electric plants. This is because of the requirements to be in:

...compliance with standards developed by the North American Electric Reliability Corp. (NERC), of Princeton, N.J. The NERC Critical Infrastructure Protection (CIP) standards will soon become required for electric grids.

The article quotes Emerson's Eric Casteel, manager of security, SCADA and renewable energy development in the Power & Water Solutions business. Concerning the development of smart grids and the connections of new plants to it, Eric offered:

Renewable power needs to be monitored more frequently than traditional power... You have wind that's variable, solar that's variable, and those variables need to be managed frequently. Oversight is deeper and it's shared with executives, so it's exposed to the outside world.

As we've highlighted in many cyber security-related posts, the article notes the need treat cyber-security as a program, not a collection of cyber defense technologies. It should be treated like the plant safety program where everyone has a role, looks out for one another, and has a feeling of ownership.

Eric points out the conflicting objectives IT and the control teams have. IT has deep experience with security and the procedures to keep everything patched and up to date. For the process control team, availability is paramount. It echoes a conflicting-objective thought with respect to screensaver passwords that I expressed in an earlier post:

For example, if an operator gets locked out and can't immediately address a plant alarm condition, the results can be very different than if an accounts payable professional gets locked out from their workstation.

Eric highlights how some electric plants address these conflicts:

Some plants are bringing in a consultant to work with control, and bridge the gap with IT... Where it's been most successful is where control still has the responsibility for security, but they work closely with IT.

The article highlights the dilemma of the Smart Grid requirement to share information outside a plant with that communication path being an entry point for cyber security threats. It concludes:

NERC programs and audits are compelling electric plants to demonstrate their ability to withstand cyber attacks. To cope with all of this, plants are bringing together the expertise of consultants, vendors and their IT departments to ensure that they're well protected.

MP3 | iTunes

Making automation systems secure remains a large concern for both process manufacturers and automation suppliers. I touched earlier this week on the work of the ISA99 standards effort to address this concern in a collaborative way.

I came across a video in an Intech magazine post, Security top of mind. It features an interview between Intech editor and Talk to Me blog author, Greg Hale, and the supplier's security expert at their user meeting.

The interview delves into a number of these concerns such as emerging government regulations, the similarities of process safety and security, and learning curves faced by automation professionals.

One thing that caught my attention was the part of the control system and its role living on the IT network. I asked Emerson's Bob Huba, whom you may recall from a number of cyber-security related posts. He acknowledges the importance of the connection of the control system with the plant local network. Business decisions, which impact the profitability of the process manufacturer, require the information contained within the control system.

Bob felt it should be stressed that control systems have unique needs and should be treated differently to accommodate these needs. This is often a source of great friction between those responsible for the support of the control and automation systems and those responsible for the plant network, firewalls and servers. It's because the functions look at the world from very different perspectives.

One behavior that he finds particularly troublesome is the idea of treating the control system as just an extension of the enterprise local area network (LAN). By this Bob means the idea that you can just drop a router between the enterprise LAN and the control LAN and then connect all of the Enterprise-level network management applications directly to the devices in the control LAN. This just seems like a great way to open the control LAN to all sorts of security issues.

He suggests that all communications from a control network should terminate at the boundary of the control system and then have to be re-established in the boundary device to continue to the external destination. And this ought to be carried through to the DMZ where the communications again are terminated in the DMZ and are re-established to connect to the full enterprise LAN. There ought to be no direct communications between a control system LAN and an enterprise LAN - for any reason. This includes VLAN tunneling from the enterprise LAN to the control system. Preventing any direct communications from the enterprise LAN is one of the keys to a secure system.

From the automation point of view, robust operation and availability is paramount. For the IT organization, being current on the latest patches, virus scan data files, etc. is critical. Even simple things such as screen saver locks can trigger quite opposing views. For example, if an operator gets locked out and can't immediately address a plant alarm condition, the results can be very different than if an accounts payable professional gets locked out from their workstation.

As mentioned in the video, for some process manufacturers the ultimate "ownership" of the automation system may be with the Chief Information Officer, but it's important to recognize the unique requirements and develop the security policies to recognize this uniqueness. Developing a single policy without recognizing the very different role the control system has from the rest of the plant IT infrastructure may lead to operational problems.

One interesting note as I write this post is that my "Security and Patch Manager" window appeared and said it had new patches for me (which elicited my, "Oh Joy!" response.) I would bet at most process manufacturers' control rooms that this window does not appear on the control system operator screens, at least without the control system administrator giving it the OK to be there.

MP3 | iTunes

Emerson's Mike Boudreaux pointed me to a great article written by ISA84 committee member, Paul Gruhn. The article, Safety, Security groups form joint working group, describes the reasons for the ISA84 Process Safety standards group and the ISA99, Industrial Automation and Control System Security group joining forces. ISA99 Working Group 7 (WG7) is the home of this effort and is chaired by Mike and Kenexis Security's Bryan Singer.

Paul's article acknowledges that we all learn from our own mistakes:

...but when it comes to the safety and security of high-risk process facilities, it is important we learn from the mistakes of others. That is the collective knowledge that standards are built on.

In the article, he describes how safety and security are similar:

...the greater the level of risk in a process, the better the safety instrumented systems that will be needed to control it. Similarly, the greater the level of risk of a security breach, the stronger the measures will be needed to combat it.

I asked Mike for some recent developments with WG7. He told me that they have created task groups to review the existing ISA99 standard as well as the two leading global safety standards related to process manufacturers--IEC 61508 and IEC 61511. Other relevant standards are also being considered. The intent is to look at how the lifecycle and risk reduction methodologies in these safety standards might be applied to automation system security.

Mike described three task groups that have been created: ISA-99 WG07.TG01-2009, -.TG02-2009, and -TG03-2009. Brian Singer leads the TG01 group and they are working on the creation of the ISA-99 WG07 charter document.

Mike leads TG02 and his task group is assembling a list of recommendations to the ISA99 leadership on how to improve consistency with other engineering practices. They would also provide a list of recommendations on key benefits of the current ISA99 approach or additional areas of opportunity to provide value. Finally, the task group will provide input to the standards roadmap for the documents that will receive WG7 content and any other needed documents.

The final work group, TG03 is lead by Jim Gilsinn whose group will lead the effort for a target outline and document structure for the WG7 work products.

These structures help provide a framework for collective knowledge sharing that's required to develop security standards, which will benefit process manufacturers the way ISA84 and IEC 61511 have benefitted them from a process safety standpoint.

For those of you who use Twitter (and you should! J), you can follow updates by Mike and Bryan and Dow's Eric Cosman on this important standards effort at @isa99chair.

MP3 | iTunes

Update: Mike shared with me that Bryan Singer and Eric Cosman of Dow Chemical are the two folks who manage the ISA99chair Twitter account and I've fixed the post above.

A colleague pointed me to this provocatively titled article, Social media: Why engineers should be anti-social, on the Control Engineering website. It reports:

A recent survey says that IT departments are having arms twisted to relax cyber security rules and allow access to social media sites, such as Twitter or GoogleApps.

I hope some of my gentle prods, such as, "I know that some have written me to say that their IT department blocks these videos. I think the case must be made that there are significant business uses from training videos, to application videos, to even capabilities videos like these", aren't contributing to this undue pressure... OK, I confess. I do hope they are.

The title of this article seems to be the polar opposite of an article Deb Franke and I wrote for a sister publication, Control Engineering Asia. The article, The World of Web 2.0, describes our strong beliefs how these social media technologies can increase the effectiveness of engineers and other process manufacturing professionals.

Upon closer inspection of the article, it was not a call for engineers to be anti-social all the time, just when they happen to be on computers sharing bandwidth with the control system network connected to social media applications out on the web. The article cites applications such as Twitter, Facebook, MySpace, GoogleApps, etc.

At least, that was my takeaway from ISA SP99 co-chairman Brian Singer's quote:

Taking 300-500 ms extra to receive e-mail or a Webpage is largely unnoticeable; 300-500 ms for control messages or safety messages could be disastrous. Often, what is an acceptable level of saturation or utilization from an IT perspective can spell disaster for controls.

In the article, Emerson's cyber-security expert Bob Huba added:

"Using the control system for non-control communications, says Bob Huba, DeltaV product manager for Emerson Process Management, "regardless of how much 'extra' bandwidth appears to be available, can only lead to problems getting the mission-critical control information distributed as quickly as possible."

On these points, I can't disagree. I believe engineers, on their plant intranet networks, outside the firewalls and DMZ, which separates this network from the plant's control network, should have access to these social media applications--for the reasons we cite in the Web 2.0 article.

I also believe there is benefit in experience sharing with some of these social media applications like wikis, blogs, and microblogs hosted internally, inside the firewall, but also not connected to the control network.

Where once the operator and maintenance workstations were the only way to view information, we now live in a world with phones, hardened portable devices, and other handheld devices that can connect people to the information they need without piping this information through the control network.

My thoughts in summary... keep that pressure on the IT folks to open up the plant networks to social media applications, but not the plant control networks.

MP3 | iTunes

Update: Welcome Feed Forward readers! Paraphrasing the words of Get Smart's Maxwell Smart, "I hope I wasn't out of line with that crack about engineers." As an engineer, I mean it only in the most endearing way!

Let's close this holiday-shortened workweek with a post about cyber-security. Recently ARC Advisory Group's Larry O'Brien posted on the topic of Safety Culture. In it, he makes the strong point:

However, an organization with a vigorous safety culture is always in a better position to avoid accidents and is better prepared when such an incident happens. Management needs to determine the degree of safety culture they wish to achieve, and chart and navigate a path to get there. Management responsibilities include not only rigorous safety planning, but also instilling a strong safety culture within their organization. Only, a person working within a strong safety culture feels more secure and is motivated to make the work place safer for everyone.

Emerson's Bob Huba has been making similar points about the importance of the culture as it applies to process control cyber security. I added one of his presentations in a post, Like Plant Safety, Build a Culture of Security. His key message is that must be looked at holistically, much the way process manufacturing plants take a holistic view for safety.

I forwarded my RSS feed of Larry's post to Bob Huba for his additional perspectives. Bob wrote back:

Organizational culture is also important in security. If you depend on enforcement from above or via IT control of the system to make you secure people will continue to do insecure things - bring in infected USB sticks and such. Unless you train and educate people to not do insecure things, you will never maintain a security program. Process control equipment is just too vulnerable, being out in the plant environment, to do security without the buy-in and understanding of the people who live and work out in the plant.

As a product manager on the DeltaV team, Bob looks at the technological aspects that can help process manufacturers with their security efforts.

Technology can help to some extent, but a proactive security culture is imperative, much as a safety culture is.

A popular TV drama series here in the U.S. recently dramatized a cyber-attack on a chemical plant, raising awareness of process manufacturing cyber-security risks outside our niche of process automation. Within our industry, this has been an area of concern for many years and is being addressed in many different ways.

I mention this because Emerson's Bob Huba recently spoke at the 63rd Annual Instrumentation Symposium for the Process Industries at Texas A&M University. His presentation was on Control System Cyber-Security.

As a senior product manager for the DeltaV control system, Bob has developed a best practices whitepaper, as well as a philosophies/guidelines whitepaper.

His key message to the symposium participants was that the technologies alone are not enough. Security must be looked at holistically, much the way process-manufacturing plants take a holistic view for safety.

Bob starts by defining SCADA Cyber Security as protection from intentional computer misuse that would cause inability for you to properly control the process. IT security protects information, while SCADA security protects physical assets and production. It also encompasses the protection of your control system, which is required to manage your process.

Threats can come from undirected, automatic sources including worms, viruses, and malware. They can also come from deliberate attacks, which disrupt the system or even take over the system. The concern for these deliberate attacks continues to grow.

To combat these threats the automation industry is developing a standard for control system security, ISA SP99. This standard's charter is to address:

...manufacturing and control systems whose compromise could result in any or all of the following situations:

• endangerment of public or employee safety
• loss of public confidence
• violation of regulatory requirements
• loss of proprietary or confidential information
• economic loss
• impact on national security

Bob reviewed the technology side of the equation that he shares in the whitepapers based on the philosophies of rings of protection and defense in depth.

His big message was that the technology alone is not sufficient. Process manufacturers must develop a culture of security. He suggested that the program could be modeled after the plant safety program. Most plants proudly display their days without a lost-time accident. Everyone in the plant has a role in keeping each other safe. Safety milestones are typically commemorated with high-level management participation to continue to reinforce its importance.

Bob shared that to be successful security must also become a "way of life." It should include all people who come into the plant. The reason to follow the safety model is that the model is easily understood by operations. It is implemented at the right levels of the organization and helps reinforce everyone's behavior in promoting security. The processes and procedures are localized, as they need to be to the site, department, or process unit. Also, control systems are different so some elements of the security enforcement procedures must be specific to the system.

Key to this approach is that you must have an owner or security champion in operations just like most facilities do with an operations safety person. As SCADA security has unique aspects that are different from classic information security, a major role of this "champion" will be to interface with IT to make security work in the control system. Their role is to "make security happen." They are not the security expert, but are knowledgeable and passionate about the need for vigilance in security. Bob defined important roles for this champion including security policy development, system access go-to person, risk assessment team member, control system vendor focal point, and vulnerability reporting and mitigation.

Better security comes from establishing a security-minded culture where everyone plays a role. This approach fosters paying closer attention to the details of technology maintenance, patching, virus management, ongoing assessments and improvements.

MP3 | iTunes

Emerson's DeltaV team announced news of new smart switches for the DeltaV control network. The topic of cyber-security is on the minds of a lot of process manufacturers right now. In fact, the November 2008 issue of Control magazine is devoted to security-related issues for process manufacturers.

I discussed in an earlier post how commercial off-the-shelf (COTS) technologies rapidly advanced the capabilities and performance for all of the process automation systems. Since these technologies are no longer totally proprietary and within the total control of the automation suppliers, security is a fundamental issue that must be addressed by both suppliers and process manufacturers.

The use of COTS technologies, like Ethernet-based control networks, has caused a collision in the worlds of IT and plant automation teams. Much has been written about the competing objectives and methods between these teams. DeltaV product manager, Bob Huba, has been in these discussions for several years, ferreting out requirements in his role managing security for the DeltaV system.

Bob notes that the fundamental premise behind the smart switch is to make it easy for automation engineers to secure their control networks and give them a way to help their IT organization help themselves. They can do this without having to involve the IT group in the setup and ongoing support of the Ethernet-based networks used for process control.

These switches, used to connect workstations, controllers, WirelessHART gateways, and other Ethernet-based devices are built-for-purpose and fully accessible from inside the DeltaV system. The switches are designed to work with the DeltaV control network with no configuration nor network specialization required.

Instead of COTS, the DeltaV team has coined the acronym, POTS--purpose-built, off-the-shelf to describe this class of device.

Bob described how it helps the IT team by saying that they are fully simple network management protocol (SNMP) version 3 capable, which provides three valuable services: authentication, privacy and access control.

With the proper authentication, IT staff members can get view-only access to these switches and incorporate the information into their network management processes. Bob threw a new term I hadn't heard, MIB-II. Google, always my trusty friend, told me it meant Management Information Base, and Wikipedia defined it:

A management information base (MIB) stems from the OSI/ISO Network management model and is a type of database used to manage the devices in a communications network. It comprises a collection of objects in a (virtual) database used to manage entities (such as routers and switches) in a network.

From the automation team's viewpoint, knowing that it's view-only access helps them all sleep better at night. They don't have to know a thing about SNMPs, MIBs, OSIs, ISOs and other IT stuff, which is of great comfort to them.

In addition, for local access to switch information, each switch has a built-in read-only, web browser interface. Process maintenance folk can browse any switch and get the diagnostic information they need.

What automation folks care about is that their automation systems are secure. The smart switches address one of the largest concerns--open Ethernet ports that people can plug into, or add wireless access points to--and open up the control network to all sorts of security risks. The release describes an easy way to prevent this--one-button lockdown:

The one-click lockdown application automatically scans the DeltaV network to find the DeltaV switches and then allows the user the choice to automatically unlock or lock the switches. Unlocking also enables an auto-relock of the switches in 60 minutes if the user does not perform a manual relock before then.

If you can have a way to bring peace between the automation and IT groups, have the level of security you need, and have the ease-of-use so it is in fact used, then Bob just may have ferreted out something good in all those conversations.

MP3 | iTunes

Update: ARC Advisory Group has a nice summary of these smart switches.

This is a great time of year to be a blogger for Emerson with the Emerson Exchange happening next week. It's that time when process manufacturing professionals, Emerson and local business partner professionals, and alliance members gather to exchange their expertise with one another.

It's also a target rich environment for me to find great things to discuss in blog posts. Emerson's Dave Rehbein, a senior data management solutions consultant, will be presenting a workshop, Plant Floor Security Assessments - Key Findings. I've known Dave for a long time. We represented Fisher-Rosemount in the original OPC standards effort back in the mid 1990s. Dave served as the master editor for the original OPC specification released in 1996.

If your responsibilities include the cyber-security efforts around your automation systems, and you'll be at the Exchange next week, this is a presentation you'll not want to miss (Wed 10/1, 2:15pm Chesapeake 7, Thurs 10/2 9am Chesapeake 7.)

Dave will present aggregate findings from three different process manufacturers where he performed plant-floor security assessments. He'll discuss risks found in the areas of: patch management, isolation of plant control network from the office network, following existing security procedures, use of unsecure legacy operating systems, security log auditing, and hardening of PCs and servers.

Dave describes the security assessment process as one that typically takes two weeks and involves reviewing existing plant floor security processes, reviewing network and device security, developing a risk matrix of the plant floor systems and application, and developing the risk mitigation plan based upon the findings from these earlier steps.

It's critical that the process manufacturers have participation throughout the organization for the plans to be executed and the work processes put in place for ongoing security. He recommends participants include an executive sponsor, a plant area engineer, plant IT director, security officer, chief network architect, network/telco architect and an operations manager. Dave presents an overview of the flow of activities over the two weeks.

Microsoft provides some excellent assessment tools that can gather information on the current state of security. Dave mentioned the Microsoft Baseline Security Analyzer (MBSA) and the Microsoft Security Assessment Tool (MSAT) as important assessment tools for things like identifying patch levels and to conduct interviews with key plant personnel to assess the maturity of the existing security programs.

The risk matrix that is developed as part of this security assessment process takes each recommendation developed and looks at its urgency, impact and investment required. Each of these parameters is graded on a low, medium, high scale. This helps establish the priority, justification and roadmap for cyber security improvements.

As I had mentioned in an earlier post, commercial off the shelf technologies (COTS) have allowed all automation suppliers to rapidly improve their platforms by taking advantage of the price/performance curve described by Moore's Law. The downside is that security is a much greater concern since parts of the technology development are outside the control of the supplier. Security consultants help suppliers to rigorously test their equipment to identify and fix areas of security concern.

Process manufacturers must also do establish security practices for the lifecycle of the automation equipment on which they rely to operate their plants. The process Dave describes in his talk is a great way to compare with what you're currently doing today.

Emerson's J.D. Wheelis, a modernization consultant, wrote a thought-provoking piece that I got my hands on recently. The subject is computer virtualization, and J.D. discussed how you might consider using it for your Windows NT-based applications where the hardware has become obsolete and/or unreliable.

In my "geek hack" time, I've played around with some virtualization software like some of VMware's offerings, and found it quite amazing how an entire operating system with associated hardware peripherals can be virtualized inside another with good performance. I used it when I switched to Windows Vista and needed a Windows XP virtual PC to connect to some legacy IT applications and printers. It's also possible to run virtual Linux PCs inside a Windows-based host and vice versa.

The only "gotcha" I've encountered is if you have an OEM version of a Windows operating system that's tied to the hardware it comes with. You can't shut down the hardware and move the operating system to a virtual environment. You'd need to purchase or transfer a version unencumbered with these license restrictions.

J.D. notes that several process control applications remain only available in the Windows NT environment. The automation engineers in plants and mills run into a conflict between keeping the plant running with the continued use of the application and the difficulties in keeping the application in operation. In addition, the engineers deal with the conflict between keeping these PCs running and satisfying plant IT requirements to eliminate these PCs based on security concerns.

J.D. makes the case that virtualization techniques and products are a way to provide hardware replacements for these NT-based computers.

Virtualization enables the Windows NT operating system to run in a virtual environment created under a different and newer operating system, such as Windows XP, Windows Vista or even a flavor of Linux. The physical machine hosting the virtual machine provides everything the virtual machine needs - the disk space, the memory (RAM), access to the processor, access to both the physical network and a virtual network as the application requires.

J.D.'s hands-on experience is similar to mine with respect to performance. He observes:

In my experience in installing and using virtualization for a control system configuration application, I saw no difference in the performance of running the application. Everything worked, from complicated graphics display and editing, database operations, sub-applications designed to perform special tasks, to communications through networking, both from virtual machine to host and virtual machine to another physical machine through the host's network connection.

With respect to security:

The security issues on networked virtual machines remains if you are using the application to communicate over the physical network to a different physical machine. You can use common security measures to mitigate the security issues. These include configuring routers to limit the connections from the virtual machine to only those computers you know need to have that access.

You can also limit the network connections to those on the physical machine, and not allow the virtual machine access to the physical network. Careful planning and implementation helps minimize security risks in having virtual NT machines on the network.

J.D. summed up his thoughts that virtualization can help reduce the spare part hunt fixing up older physical PCs, can help relieve that natural tension between plant engineering and the IT team, and can follow some standard security measures to reduce cyber-security risks.

These ideas are something to consider if you're faced with this situation.

Update: A couple of great comments have come in on this post in FriendFeed that I wanted to share.

In one of those hallway conversations, Chuck Miller reminded me of a workshop at last year's Emerson Exchange. Chuck, well known for his safety instrumented system and safety compliance expertise, co-presented with a process manufacturer on the topic of cyber security in the control domain.

Automation systems have gone through quite a change from the '70s and '80s to our current decade. The architecture changed from proprietary data highways separated by gateways from operation stations, and separated from the enterprise local area network by another gateway--if they were connected at all. Using the ISA-95 (S95) model to describe the various levels of the architecture, the automation systems evolved where Ethernet/IP addressing is used between levels one and two, and between levels two and three. This move from proprietary technologies to commercially available technologies means that the issues with cyber-security must be considered and addressed.

The presenters defined risk as likelihood multiplied by consequence. Likelihood was defined as threat, vulnerability and target attractiveness multiplied together.

The process manufacturer described their risk assessment/reduction process. It included a risk assessment phase, risk reduction workshop phase, development of a risk reduction plan, and implementing that plan. Two key tenets of this process were that only the site personnel had the knowledge to assess the risks to their plants and their systems and that each site would use a risk assessment tool to develop a site risk profile. ISA-SP99 can help in the elements involved in this risk assessment.

Security levels were assigned based upon the consequences of a successful attack. Considerations included the level of hazard associated with the process or product, the location of the plant, and applicable federal critical infrastructure processes. The last consideration impacts the target attractiveness part of the risk equation.

They discussed the areas of the automation system that need to be addressed including the control of network access, user access and physical access. A key point is multiple layers of protection must be considered. Their analogy was a medieval castle protected by a moat, then by a drawbridge, then by a portcullis, then by murder holes, then by the outer walls and finally by the keep. Not all these layers of protection made castles impenetrable, but certainly extremely difficult to "hack into".

An automation system has points of entry that must be addressed by the security plan. These include the connection to the plant network or other external networks, modems, CDs, floppies, USB devices, equipment on the level-one control network between the controllers and PCs and underneath from the I/O subsystems.

One example, the control network, should require that no devices other than the controllers and PCs running operator, engineering, and applications be permitted to connect. Also, controller firewalls can be added between the PCs and controllers. These function to protect the controllers that are installed on the secure side of the firewall against message flooding and denial of service attacks. This firewall is in addition to the router/firewall above the PCs between the automation system and level 3 applications.

In the case of this process manufacturer, this router/firewall was managed by the operations organization. They created a DMZ above the automation system, which contained an anti-virus server, data server, and historian server. Above these was another router/firewall, managed by the IT organization, which connected to the plant local area network.

The presenters also discussed anti-virus strategies, security bulletins, and disaster planning. They summed up the presentation with elements that should be in the plan. These include:

• Assess the risks
• Define the critical systems
• Mitigate for (at least) the high cyber security risks
• Test the plan on a regular basis
• Train the users in the plan
• Get stakeholder signoff

This whole security risk assessment process is not easy, but like process manufacturers' safety risk assessments, is critical. For other automation system cyber security considerations, take a look at best practices in cyber security that is written around Emerson's DeltaV system.

A colleague pointed me to a cyber-security article published on the Engineer Live website. The article, Boosting confidence with cybersecurity certification describes several automation suppliers and their work with Wurldtech and their Achilles certification program to test their automation controllers for cyber-security robustness. The article summarizes the program:

Its Achilles assurance platform, created by the company's team of network security experts, control system engineers and white hat hackers, is the first automated and comprehensive testing product for assessing vulnerabilities and security threats to devices and networks supporting critical infrastructures worldwide.

The article includes the news of Emerson's DeltaV controller and firewall passing, "over 20 million tests to achieve Level1 certification." At Wurldtech's website, you can see other automation suppliers who have similarly subjected their equipment to this rigorous testing and achieved certification.

Bob Huba, whom you may recall from earlier cyber-security posts, describes an ongoing benefit of this testing for process manufacturers:

Controllers with Level1 certification have demonstrated the robustness to survive network cyber attacks. One real benefit of passing these rigorous tests is to provide users with the ability to better plan the installation of security updates and new anti-virus signatures. Knowing that the controllers can survive a possible security incident provides an opportunity to schedule these patching tasks around process activities rather than always immediately deploying the updates.

I'll definitely not claim myself to be an expert with cyber-security, but I do see similarities with cyber-security and safety efforts in taking a risk-based approach. I shared this thought in a recent Chemical Processing magazine article, Plug cyber-security gaps. This is also an excellent article and well worth the read for interested parties.

If you are in this "interested party" cohort, you may also want to follow the work of the ISA99 committee, Manufacturing and Control System Security (I'd subscribe to their RSS feed if they had one... nudge, nudge.) You may also want to follow some the leading cyber-security blogs with respect to process automation including Dale Peterson's Digital Bond blog and Joe Weiss' Unfettered blog (both thankfully offer RSS feeds for easy information consumption.) I'd also pass along the DCS Security blog if we could get Jim back posting again along with the rest of us!

As we've discussed in prior cyber-security posts, process manufacturers are increasingly concerned with how to best secure their automation systems and plant sites.

At the recent Chem Show, Emerson's Bob Huba presented, Control System Cyber Security--A Different Approach. He describes that what seems to be lacking is a model for implementing security that we can understand and explain to plant personnel.

The approach Bob describes is to think about cyber-security efforts like a plant safety program. Like a successful safety program, a successful security program requires that plant personnel develop an "attitude" around security. Responsibilities are clearly assigned. People (operators, engineers, supervisors) take responsibility for security of their areas.

Procedures for control system security policies are clearly documented. And, training is formalized so that these security policies are well understood in the same way plant safety procedures are understood by all plant personnel. This training includes an understanding security processes and potential risk areas of which to be aware.

This model includes a focus on awareness for personnel to recognize and prevent insecure behavior and a mechanism to report problems and concerns. Like the safety program, measurement is important. A culture needs to be established where security incidents and insecure actions are reported and summary reports are communicated to provide evidence that security is being measured. Celebrating success is important to keep plant folks motivated.

Audits and enforcement are another key part of the model. Are the established procedures being followed and are actions established to fix any findings identified in the audits? Again, like plant safety, these efforts must be ongoing to be effective.

Bob proposes this model because it's well understood by the operations organization, it's implemented at the right levels in the organization, the processes and procedures are localized for the plant, and procedures are specific for the installed automation system(s). Taking this approach requires a champion and Bob recommends this role should not be delegated to the IT organization. It is better that this person be from operations and teams be established for different areas of responsibilities including the IT organization.

All of the specific security measures, like those referenced in Best Practices in DeltaV Cyber-Security whitepaper, are very important--but so is the process of establishing a security-minded culture.

While at the recent ISA Expo 2007, I had the chance to listen to Emerson's Jonas Berge's presentation on software for automation. Jonas is an active member in the ISA SP104 committee. This committee is responsible for advancing the Electronic Device Description Language (EDDL) standard.

A few years back he wrote a book, Software for Automation: Architecture, Integration, and Security. His presentation covered some of the ideas from the book. Specifically, he discussed these key points:

• Select technologies for software architecture
• Justify investment to management
• Where and how to deploy DCOM vs. Web
• Where each OPC flavor is used and how
• Integrate with business and coexist with legacy
• Troubleshoot DCOM and OPC
• Apply software and make the PC rugged
• Engineer and document software
• Backup, administer, and optimize
• Make it robust, safe, secure, and 21 CFR Part 11 compliant

The body of knowledge that an automation professional must understand to perform their job effectively continues to expand. As Jonas describes, the software architecture is as important to design as the hardware architecture. Information flows from devices connected from digital busses all the way through the automation systems to enterprise-level software applications.

Security concerns must be addressed and be part of this design. Cyber-security is an area of specialization unto itself and you can follow many of the issues and advancements at the Digital Bond and Unfettered blogs.

Jonas describes setup of networks and OPC, ODBC, and web services communications across networks and tips for troubleshooting these. One everything is functioning properly, methods of management and administration including backup and restore procedures are covered.

Jonas highlights the fact that this is a lot to plan and get right. If you find yourself overwhelmed and too busy to become an expert in this area, you are not alone. Many process manufacturers are working with their automation suppliers versed in this level of expertise to help on the project front-end and to help maintain these software packages and integration methods through their useful lifecycle. One example is Emerson's SureService support services.

As announced at the Digital Bond blog and noted on the Sound Off! blog, the DeltaV controller is included in the first group of controllers certified by Wurldtech's Achilles Controller Certification. The purpose of this program:

The Achilles Certification Program was developed by Wurldtech and its partners to provide a benchmark for the certification of secure industrial controllers. The program is designed to assess the overall security of industrial controllers and certify that they meet a comprehensive set of requirements and conformance. The certification process presents device manufactures with an independently verified result from which to communicate their product security to customers, while providing the operators of control systems the most complete, accurate, and trustworthy information possible on the security of their deployed products.

I caught up with Emerson's Bob Huba who has worked closely with Wurldtech in gaining certification for this important cyber-security effort. You may recall Bob from prior posts on the topic of cyber-security.

Bob feels this certification is important for process manufacturers. By doing device testing to an accepted set of test suites and test parameters, an automation engineer can have a higher degree of comfort that automation controller solutions have the robustness to survive network level cyber attacks.

Emerson customers have told Bob that one real benefit of this testing is that it gives them the "breathing room" to better plan the installation of security updates and new anti-virus signatures. Knowing the controllers can survive a security incident will greatly reduce the risk involved in having to schedule these patching tasks around process activities rather than always immediately deploying the updates.

Over time, Bob expects device testing and certification to become an even bigger part of the industry cyber-security and system robustness solutions. In fact, he just returned from a two day meeting of the newly forming Control System Security Certification Organization (CSSCO) in Houston.

At this meeting, the group defined as part of their mission:

to decrease the time, cost and risk of developing, acquiring, and deploying control systems by establishing an industry-based program to... facilitate the independent testing and certification of control system products to a defined set of control system security standards.

Bob noted that support for the CSSCO has been growing since several major asset owners proposed the initial idea of such an organization about two years ago. It has recently come under the auspices of the ISA organization. They are helping to develop this into a full standards organization. Bob suggests that if you are interested in this effort to look for more information coming out on this in the upcoming weeks.

Personally, he would like to see as broad a process manufacturer representation in this group as possible. To this end, Bob plans to invite members of the DeltaV community of users to consider participation in this effort. For those members who happen upon this post, feel free to contact Bob.

Recently the Digital Bond cyber-security blog pointed to a vulnerability note on the Trend Micro anti-virus package and noted:

Software designed to protect ends up putting virtually every system on your network at risk.

This had to cause pause among anyone reading this blog. In spite of best efforts to ward off the ravages of viruses, even those packages responsible for protection can be compromised.

When I saw this article appear in my RSS feeds, I ran it by our resident cyber-security expert and DeltaV product manager, Bob Huba, who you may recall from earlier cyber-security posts. Bob noted that the unfortunate reality of today's highly interconnected world is that new vulnerabilities come up all the time.

Constant vigilance must be a critical part of your cyber-security efforts. Patch, patch, patch is your best practice and this process requires a close partnership between you and your automation suppliers to perform these practices. Suppliers have to do their part with prompt patch certification and anti-virus support to make sure these patches and additions don't break existing software functionality. This certification must get clearly communicated to the automation system/cyber-security administrators around the globe to quickly plug these vulnerabilities.

One thought Bob shared is that perhaps some installed automation systems are "over connected" with the enterprise. Each connection is a source of vulnerability and its business case should be carefully considered. The connections should be designed more with cyber-security than low cost in mind. It's likely best to have a single external connection from these automation systems that "you can guard like heck."

An analogy is the bank vault with layer upon layer of security which serves to slow down potential breaches so that they can be discovered and thwarted in time.

It sounds like the choice is either to heavily devote attention to these connections or lock down and disconnect from the network--something not too practical as process manufacturers try to optimize their business processes and manufacturing systems.

The DeltaV New RSS feed today points to a U.S. Department of Homeland Security press release, Government, private industry work together to increase cybersecurity. It mentions how the Department of Homeland Security is facilitating a group called the Control Systems Cyber Security Vendors Forum to provide an open discussion on those issues affecting control system security.

Although a U.S. initiative, process manufacturers around the globe have an interest in the cyber-security of their automation and control systems.

I caught up with Bob Huba, whom you might recall from earlier discussions on the issue of cyber-security. Bob explained to me that the goal of this initiative is to share ideas around a common goal of protecting automation systems from unauthorized cyber or physical access. Much like the IEC and ISA standards committees, the Vendor Forum offers a neutral place for suppliers to get together to talk about cyber-security best practices and develop guidelines.

Today there are labs like Idaho National Labs who started the Control System Security Program, Sandia National Laboratories and WurldTech Security. These organizations will test systems for many known exploits and provide reports to the suppliers for these to be fixed. Although these tests are necessary and valuable, there are no existing agreed on standards to test against. Providing inputs to the groups who are defining the security standards is one of the hoped for results of the Control Systems Cyber Security Vendors Forum.

One goal of the vendor group is the partnership of federal regulators working with the automation system suppliers who best understand the issues with their respective systems. It will help lead to workable guidelines and best practices that can be shared with global process manufacturers.

The feeling among the suppliers seems to be that basic cyber-security is not an area for system differentiation--it's an absolute requirement like PID control or connectivity with business systems. As part of maintaining the security of our process infrastructure we all need to rely on the products process manufacturers make and want to make sure their systems are as secure as they can be made.

You may recall our Cyber-security expert Bob Huba from some earlier posts on this topic. Bob has done an excellent recap of the Cyber-security presentations from the recent Emerson Exchange which I'll pass along to you with some relevant hyperlinks:

The 2006 Emerson Exchange contains a significant increase in the number of system cyber-security presentations over past years. This indicates the increasing importance of system security in the minds of process manufacturers--since the user community actually develops the presentations and the agenda for the Emerson Exchange.

Last year there was only one short course and a couple of workshops on security. This year there are 2 full days of security presentations--basically the same presentations repeated each day to offer more opportunity for users to schedule time for the sessions. The Emerson Exchange Board doesn't usually schedule two sessions unless they feel the subject will be popular.

One highlight of the security sessions is the popular Idaho National Labs short course initiated in the 2005 Emerson Exchange--back again this year for two four hour sessions. The presentation, made by the highly knowledgeable presenter from INL Mark Fabro, held the rapt attention of 75 attendees. The course will be repeated on Thursday for those that attended the other cyber-security workshops scheduled concurrently with the INL course for this morning.

The other security workshops today had excellent attendance as well. One of our Pulp and Paper customers discussed how to keep your DeltaV system anti-virus scanners up-to-date using automated tools and procedures to download and distribute the signature updates. Another presented a user viewpoint on system security do's and don'ts. And I, the DeltaV marketing manager for DeltaV security, spoke on the DeltaV security enhancements, including the details on the new DeltaV Controller Firewall, to a packed room. Part of my presentation also included a section comparing a control system security program to a plant safety program. That like safety, a security program includes a significant effort on user education and training. We all need some basic cyber-security efforts and we just need to do something now rather than waiting for some complicated security program to develop.

Mark Fabro led an afternoon workshop discussion, "CyberSecurity Who Needs It" on how to understand the emerging threats and the practices countermeasures we can develop to mitigate these threats. We really need to have suppliers, users and the public sector to work together in this effort. Mark thinks there is a lot of fear, uncertainty and doubt in the user community about the "real" threats and how to mitigate them. Next was a refining customer with a management-oriented workshop on "Cyber Security and your Bottom Line". He was making the point that management is reluctant to spend the money on security. They need to justify how does this "help us make better oil" where better question might be "how does security keep us making oil". If your assets are at risk--water, power and environmental systems how long can we stay running? He also made the excellent point that when setting up a security policy that it "if it is important enough to make it a policy it is important enough to fire somebody for violating it."

The final presentation of the day was by two Oil & Gas customers with their presentation on "Cyber Security in a DeltaV Environment: A Harmonious Relationship". It was attended by over 50 people. Being the last presentation after a long day of being "PowerPointed" to death shows the serious concern manufacturers have about cyber security. They recommended a NIST publication 800-37 to help users develop their security program. A point was made discussing a key security concept--called "Defense in depth" and defined this as the concerted use of multiple security techniques to mitigate the risk of compromise to an acceptable level. At the same time they strongly advised that process user to be sure and use Defense in Depth techniques that are appropriate for use in the control systems and to not blindly deploy IT-based solutions that might impact the availability of the control system.

All in all, the Emerson Exchange developed an excellent and well attended set of control system cyber security workshops that provided process manufacturers with some great and pertinent information on keeping their DeltaV control systems as secure as possible.

My RSS search on cyber security found an interesting post the other day by IBM's Todd Watson entitled How To Keep the Internet Sky From Falling.

It's especially interesting to me because I've had the chance to meet Todd who is also based here in Austin, Texas. He offered some great guidance in the early days when we were trying to launch the Emerson Process Experts blog.

The paper Todd referenced is by the Business Roundtable, Essential Steps Toward Strengthening America's Cyber Terrorism. Although this paper is mainly concerned with the loss of the Internet and Wide Area Network capabilities, it does have thoughts that process manufacturers around the globe need to consider.

I ran Todd's post by Bob Huba who is leading the efforts on cyber security as it applies to Emerson's DeltaV system. He's part of a newly formed cyber security testing consortium for the process industries.

Bob thought the paper as it applies to owners of control systems brought two points to mind. The first is to keep the control system completely segmented from internet traffic and the second is to not be dependent on information from outside the control system to perform basic control functions. This is especially true if the information required for control is coming from outside the facility over the internet.

As part of control system security best practices Bob always promotes the idea that in a crisis situation on the plant LAN, such as a serious worm or virus attack that could leak into the control system, you absolutely must be able to sever the external LAN connection(s) with the control system until the issue is resolved. The control system must be able to keep functioning at some acceptable level with this connection severed. This is why the recommended DeltaV approach is that the optimization and other supervisory type control tasks be done locally in the DeltaV system whenever possible.

This model is being used in universities and colleges where they have a "student LAN" for email, instant messaging, web access, etc. that is aggressively segmented from the main university system with very few interconnections. These connections can be highly secured and monitored. They can be easily and quickly severed if the "student LAN" gets infected or attacked so the main system can be protected.

This is the model used in the initial development of the DeltaV system and it is the model that is still enforced. The model is based on enforcing a high degree of segmentation between the control system network and plant LAN so that critical control system functions are safe-guarded as much as possible from threats originating on the business LANs. By using very limited external connections, these connections are easier to protect and monitor and can also be easily severed when necessary.

Bob has described more of these best practices in two whitepapers: DeltaV System Cyber-Security and Best Practices for DeltaV Cyber-Security.

As Control magazine editor-in-chief, Walt Boyes, mentioned in his Another one joins the club... blog post, Emerson has joined the companies who are sponsoring the security consortium feasibility study to be performed by Wurldtech Analytics.

I spoke with Bob Huba whom you may recall from an earlier post on cyber security and the DeltaV system.

Bob participated in the initial meeting to kick-off this consortium which was held prior to the Process Control Security Forum meeting last week in San Diego. He's excited about the formation of this group because he believes that one of the things that will help automation system cyber security is the ability to certify when the automation system components or even when the system itself meets a minimum level of security or protection. Without some kind of certifying capability, it's difficult for the end users who manage the system day-to-day and system suppliers to fully assess how secure their systems might be.

Automation system security currently has no organization like the TÜV and other certifying agencies where these suppliers can go to get a device certified for different security levels. Cyber-security testing needs some sort of agency to provide the framework for device and system testing and to help manage the information around best practices (or at least generally accepted practices) for creating and maintaining a well protected system.

Bob is really glad to see that this initiative is being driven by the user community and not just the suppliers or some testing organizations because it shows they understand and support the need for a certifying body. The system suppliers really appreciate being included in the discussion right from the beginning to capture the wealth of expertise and perspectives everyone brings.

The landscape around system security or the environment around system security is maturing rapidly and it is important that process manufacturers and suppliers work together and work quickly to address issues around cyber security. This group has set an ambitious time frame for kicking off this consortium and becoming fully functional.

Also, the scada security blog (to subscribe) has a nice wrap up of the Process Control Security Forum.

After reading CONTROL magazine's Editor-in-Chief Walt Boyes' Compared to Wireless? blog post, I spoke with DeltaV product manager, Bob Huba, who oversees the cyber-security requirements and developments in the DeltaV system. Bob authored the DeltaV System Cyber-Security and Best Practices for DeltaV Cyber-Security whitepapers.

Bob, a voting member on the SP99 committee, has a slightly different take than Walt's assessment that what is driving the committee,

...has as its end result getting the Department of Homeland Security off our backs...

He believes the automation suppliers and end users really understand the importance of having some sort of standards to measure control system security. In fact, Bob says the greatest push for this standard is coming from the user community. They are driving this committee the hardest which seems to be bringing greater cooperation and more focus on getting something done.

If you are interested in some of the work to date by the SP99 committee, you can purchase these reports from the ISA:
ANSI/ISA-TR99.00.02-2004 Integrating Electronic Security into the Manufacturing and Control Systems Environment and ANSI/ISA-TR99.00.01-2004 Security Technologies for Manufacturing and Control Systems.