Emerson Process Experts

In one of those hallway conversations, Chuck Miller reminded me of a workshop at last year's Emerson Exchange. Chuck, well known for his safety instrumented system and safety compliance expertise, co-presented with a process manufacturer on the topic of cyber security in the control domain.

Automation systems have gone through quite a change from the '70s and '80s to our current decade. The architecture changed from proprietary data highways separated by gateways from operation stations, and separated from the enterprise local area network by another gateway—if they were connected at all. Using the ISA-95 (S95) model to describe the various levels of the architecture, the automation systems evolved where Ethernet/IP addressing is used between levels one and two, and between levels two and three. This move from proprietary technologies to commercially available technologies means that the issues with cyber-security must be considered and addressed.

The presenters defined risk as likelihood multiplied by consequence. Likelihood was defined as threat, vulnerability and target attractiveness multiplied together.

The process manufacturer described their risk assessment/reduction process. It included a risk assessment phase, risk reduction workshop phase, development of a risk reduction plan, and implementing that plan. Two key tenets of this process were that only the site personnel had the knowledge to assess the risks to their plants and their systems and that each site would use a risk assessment tool to develop a site risk profile. ISA-SP99 can help in the elements involved in this risk assessment.

Security levels were assigned based upon the consequences of a successful attack. Considerations included the level of hazard associated with the process or product, the location of the plant, and applicable federal critical infrastructure processes. The last consideration impacts the target attractiveness part of the risk equation.

They discussed the areas of the automation system that need to be addressed including the control of network access, user access and physical access. A key point is multiple layers of protection must be considered. Their analogy was a medieval castle protected by a moat, then by a drawbridge, then by a portcullis, then by murder holes, then by the outer walls and finally by the keep. Not all these layers of protection made castles impenetrable, but certainly extremely difficult to "hack into".

An automation system has points of entry that must be addressed by the security plan. These include the connection to the plant network or other external networks, modems, CDs, floppies, USB devices, equipment on the level-one control network between the controllers and PCs and underneath from the I/O subsystems.

One example, the control network, should require that no devices other than the controllers and PCs running operator, engineering, and applications be permitted to connect. Also, controller firewalls can be added between the PCs and controllers. These function to protect the controllers that are installed on the secure side of the firewall against message flooding and denial of service attacks. This firewall is in addition to the router/firewall above the PCs between the automation system and level 3 applications.

In the case of this process manufacturer, this router/firewall was managed by the operations organization. They created a DMZ above the automation system, which contained an anti-virus server, data server, and historian server. Above these was another router/firewall, managed by the IT organization, which connected to the plant local area network.

The presenters also discussed anti-virus strategies, security bulletins, and disaster planning. They summed up the presentation with elements that should be in the plan. These include:

• Assess the risks
• Define the critical systems
• Mitigate for (at least) the high cyber security risks
• Test the plan on a regular basis
• Train the users in the plan
• Get stakeholder signoff

This whole security risk assessment process is not easy, but like process manufacturers' safety risk assessments, is critical. For other automation system cyber security considerations, take a look at best practices in cyber security that is written around Emerson's DeltaV system.

Tags: cyber security | SCADA security | Emerson Exchange | ISA-95 | S95 | ISA-SP99 |

March 20, 2008 in Cyber-Security, in Emerson Exchange | Comments (0)

A colleague pointed me to a cyber-security article published on the Engineer Live website. The article, Boosting confidence with cybersecurity certification describes several automation suppliers and their work with Wurldtech and their Achilles certification program to test their automation controllers for cyber-security robustness. The article summarizes the program:

Its Achilles assurance platform, created by the company's team of network security experts, control system engineers and white hat hackers, is the first automated and comprehensive testing product for assessing vulnerabilities and security threats to devices and networks supporting critical infrastructures worldwide.

The article includes the news of Emerson's DeltaV controller and firewall passing, "over 20 million tests to achieve Level1 certification." At Wurldtech's website, you can see other automation suppliers who have similarly subjected their equipment to this rigorous testing and achieved certification.

Bob Huba, whom you may recall from earlier cyber-security posts, describes an ongoing benefit of this testing for process manufacturers:

Controllers with Level1 certification have demonstrated the robustness to survive network cyber attacks. One real benefit of passing these rigorous tests is to provide users with the ability to better plan the installation of security updates and new anti-virus signatures. Knowing that the controllers can survive a possible security incident provides an opportunity to schedule these patching tasks around process activities rather than always immediately deploying the updates.

I'll definitely not claim myself to be an expert with cyber-security, but I do see similarities with cyber-security and safety efforts in taking a risk-based approach. I shared this thought in a recent Chemical Processing magazine article, Plug cyber-security gaps. This is also an excellent article and well worth the read for interested parties.

If you are in this "interested party" cohort, you may also want to follow the work of the ISA99 committee, Manufacturing and Control System Security (I'd subscribe to their RSS feed if they had one... nudge, nudge.) You may also want to follow some the leading cyber-security blogs with respect to process automation including Dale Peterson's Digital Bond blog and Joe Weiss' Unfettered blog (both thankfully offer RSS feeds for easy information consumption.) I'd also pass along the DCS Security blog if we could get Jim back posting again along with the rest of us!

Tags: SCADA security | cyber security | Wurldtech | Achilles program | Achilles test |

December 6, 2007 in Cyber-Security | Comments (0)

As we've discussed in prior cyber-security posts, process manufacturers are increasingly concerned with how to best secure their automation systems and plant sites.

At the recent Chem Show, Emerson's Bob Huba presented, Control System Cyber Security—A Different Approach. He describes that what seems to be lacking is a model for implementing security that we can understand and explain to plant personnel.

The approach Bob describes is to think about cyber-security efforts like a plant safety program. Like a successful safety program, a successful security program requires that plant personnel develop an "attitude" around security. Responsibilities are clearly assigned. People (operators, engineers, supervisors) take responsibility for security of their areas.

Procedures for control system security policies are clearly documented. And, training is formalized so that these security policies are well understood in the same way plant safety procedures are understood by all plant personnel. This training includes an understanding security processes and potential risk areas of which to be aware.

This model includes a focus on awareness for personnel to recognize and prevent insecure behavior and a mechanism to report problems and concerns. Like the safety program, measurement is important. A culture needs to be established where security incidents and insecure actions are reported and summary reports are communicated to provide evidence that security is being measured. Celebrating success is important to keep plant folks motivated.

Audits and enforcement are another key part of the model. Are the established procedures being followed and are actions established to fix any findings identified in the audits? Again, like plant safety, these efforts must be ongoing to be effective.

Bob proposes this model because it's well understood by the operations organization, it's implemented at the right levels in the organization, the processes and procedures are localized for the plant, and procedures are specific for the installed automation system(s). Taking this approach requires a champion and Bob recommends this role should not be delegated to the IT organization. It is better that this person be from operations and teams be established for different areas of responsibilities including the IT organization.

All of the specific security measures, like those referenced in Best Practices in DeltaV Cyber-Security whitepaper, are very important—but so is the process of establishing a security-minded culture.

Tags: cyber security | SCADA | plant safety | security program | plant operations |

November 8, 2007 in Cyber-Security | Comments (0)

While at the recent ISA Expo 2007, I had the chance to listen to Emerson's Jonas Berge's presentation on software for automation. Jonas is an active member in the ISA SP104 committee. This committee is responsible for advancing the Electronic Device Description Language (EDDL) standard.

A few years back he wrote a book, Software for Automation: Architecture, Integration, and Security. His presentation covered some of the ideas from the book. Specifically, he discussed these key points:

• Select technologies for software architecture
• Justify investment to management
• Where and how to deploy DCOM vs. Web
• Where each OPC flavor is used and how
• Integrate with business and coexist with legacy
• Troubleshoot DCOM and OPC
• Apply software and make the PC rugged
• Engineer and document software
• Backup, administer, and optimize
• Make it robust, safe, secure, and 21 CFR Part 11 compliant

The body of knowledge that an automation professional must understand to perform their job effectively continues to expand. As Jonas describes, the software architecture is as important to design as the hardware architecture. Information flows from devices connected from digital busses all the way through the automation systems to enterprise-level software applications.

Security concerns must be addressed and be part of this design. Cyber-security is an area of specialization unto itself and you can follow many of the issues and advancements at the Digital Bond and Unfettered blogs.

Jonas describes setup of networks and OPC, ODBC, and web services communications across networks and tips for troubleshooting these. One everything is functioning properly, methods of management and administration including backup and restore procedures are covered.

Jonas highlights the fact that this is a lot to plan and get right. If you find yourself overwhelmed and too busy to become an expert in this area, you are not alone. Many process manufacturers are working with their automation suppliers versed in this level of expertise to help on the project front-end and to help maintain these software packages and integration methods through their useful lifecycle. One example is Emerson's SureService support services.

Tags: ISA104 | SP104 | EDDL | automation software | OPC |

October 17, 2007 in Cyber-Security, in Interoperability, in Project Services, in Support Services | Comments (0)

As announced at the Digital Bond blog and noted on the Sound Off! blog, the DeltaV controller is included in the first group of controllers certified by Wurldtech's Achilles Controller Certification. The purpose of this program:

The Achilles Certification Program was developed by Wurldtech and its partners to provide a benchmark for the certification of secure industrial controllers. The program is designed to assess the overall security of industrial controllers and certify that they meet a comprehensive set of requirements and conformance. The certification process presents device manufactures with an independently verified result from which to communicate their product security to customers, while providing the operators of control systems the most complete, accurate, and trustworthy information possible on the security of their deployed products.

I caught up with Emerson's Bob Huba who has worked closely with Wurldtech in gaining certification for this important cyber-security effort. You may recall Bob from prior posts on the topic of cyber-security.

Bob feels this certification is important for process manufacturers. By doing device testing to an accepted set of test suites and test parameters, an automation engineer can have a higher degree of comfort that automation controller solutions have the robustness to survive network level cyber attacks.

Emerson customers have told Bob that one real benefit of this testing is that it gives them the "breathing room" to better plan the installation of security updates and new anti-virus signatures. Knowing the controllers can survive a security incident will greatly reduce the risk involved in having to schedule these patching tasks around process activities rather than always immediately deploying the updates.

Over time, Bob expects device testing and certification to become an even bigger part of the industry cyber-security and system robustness solutions. In fact, he just returned from a two day meeting of the newly forming Control System Security Certification Organization (CSSCO) in Houston.

At this meeting, the group defined as part of their mission:

to decrease the time, cost and risk of developing, acquiring, and deploying control systems by establishing an industry-based program to… facilitate the independent testing and certification of control system products to a defined set of control system security standards.

Bob noted that support for the CSSCO has been growing since several major asset owners proposed the initial idea of such an organization about two years ago. It has recently come under the auspices of the ISA organization. They are helping to develop this into a full standards organization. Bob suggests that if you are interested in this effort to look for more information coming out on this in the upcoming weeks.

Personally, he would like to see as broad a process manufacturer representation in this group as possible. To this end, Bob plans to invite members of the DeltaV community of users to consider participation in this effort. For those members who happen upon this post, feel free to contact Bob.

Tags: cyber-security | achilles certified | Wurldtech | SCADA security |

May 16, 2007 in Cyber-Security | Comments (0)

Recently the Digital Bond cyber-security blog pointed to a vulnerability note on the Trend Micro anti-virus package and noted:

Software designed to protect ends up putting virtually every system on your network at risk.

This had to cause pause among anyone reading this blog. In spite of best efforts to ward off the ravages of viruses, even those packages responsible for protection can be compromised.

When I saw this article appear in my RSS feeds, I ran it by our resident cyber-security expert and DeltaV product manager, Bob Huba, who you may recall from earlier cyber-security posts. Bob noted that the unfortunate reality of today's highly interconnected world is that new vulnerabilities come up all the time.

Constant vigilance must be a critical part of your cyber-security efforts. Patch, patch, patch is your best practice and this process requires a close partnership between you and your automation suppliers to perform these practices. Suppliers have to do their part with prompt patch certification and anti-virus support to make sure these patches and additions don't break existing software functionality. This certification must get clearly communicated to the automation system/cyber-security administrators around the globe to quickly plug these vulnerabilities.

One thought Bob shared is that perhaps some installed automation systems are "over connected" with the enterprise. Each connection is a source of vulnerability and its business case should be carefully considered. The connections should be designed more with cyber-security than low cost in mind. It's likely best to have a single external connection from these automation systems that "you can guard like heck."

An analogy is the bank vault with layer upon layer of security which serves to slow down potential breaches so that they can be discovered and thwarted in time.

It sounds like the choice is either to heavily devote attention to these connections or lock down and disconnect from the network--something not too practical as process manufacturers try to optimize their business processes and manufacturing systems.

Tags: cyber-security | SCADA security |

February 21, 2007 in Cyber-Security | Comments (0)

The DeltaV New RSS feed today points to a U.S. Department of Homeland Security press release, Government, private industry work together to increase cybersecurity. It mentions how the Department of Homeland Security is facilitating a group called the Control Systems Cyber Security Vendors Forum to provide an open discussion on those issues affecting control system security.

Although a U.S. initiative, process manufacturers around the globe have an interest in the cyber-security of their automation and control systems.

I caught up with Bob Huba, whom you might recall from earlier discussions on the issue of cyber-security. Bob explained to me that the goal of this initiative is to share ideas around a common goal of protecting automation systems from unauthorized cyber or physical access. Much like the IEC and ISA standards committees, the Vendor Forum offers a neutral place for suppliers to get together to talk about cyber-security best practices and develop guidelines.

Today there are labs like Idaho National Labs who started the Control System Security Program, Sandia National Laboratories and WurldTech Security. These organizations will test systems for many known exploits and provide reports to the suppliers for these to be fixed. Although these tests are necessary and valuable, there are no existing agreed on standards to test against. Providing inputs to the groups who are defining the security standards is one of the hoped for results of the Control Systems Cyber Security Vendors Forum.

One goal of the vendor group is the partnership of federal regulators working with the automation system suppliers who best understand the issues with their respective systems. It will help lead to workable guidelines and best practices that can be shared with global process manufacturers.

The feeling among the suppliers seems to be that basic cyber-security is not an area for system differentiation--it’s an absolute requirement like PID control or connectivity with business systems. As part of maintaining the security of our process infrastructure we all need to rely on the products process manufacturers make and want to make sure their systems are as secure as they can be made.

Tags: Homeland Security | SCADA | cyber security | Control System Security Program | Control Systems Cyber Security Vendors Forum |

January 11, 2007 in Cyber-Security | Comments (0) | Trackback (0)

You may recall our Cyber-security expert Bob Huba from some earlier posts on this topic. Bob has done an excellent recap of the Cyber-security presentations from the recent Emerson Exchange which I'll pass along to you with some relevant hyperlinks:

The 2006 Emerson Exchange contains a significant increase in the number of system cyber-security presentations over past years. This indicates the increasing importance of system security in the minds of process manufacturers--since the user community actually develops the presentations and the agenda for the Emerson Exchange.

Last year there was only one short course and a couple of workshops on security. This year there are 2 full days of security presentations--basically the same presentations repeated each day to offer more opportunity for users to schedule time for the sessions. The Emerson Exchange Board doesn't usually schedule two sessions unless they feel the subject will be popular.

One highlight of the security sessions is the popular Idaho National Labs short course initiated in the 2005 Emerson Exchange--back again this year for two four hour sessions. The presentation, made by the highly knowledgeable presenter from INL Mark Fabro, held the rapt attention of 75 attendees. The course will be repeated on Thursday for those that attended the other cyber-security workshops scheduled concurrently with the INL course for this morning.

The other security workshops today had excellent attendance as well. One of our Pulp and Paper customers discussed how to keep your DeltaV system anti-virus scanners up-to-date using automated tools and procedures to download and distribute the signature updates. Another presented a user viewpoint on system security do's and don'ts. And I, the DeltaV marketing manager for DeltaV security, spoke on the DeltaV security enhancements, including the details on the new DeltaV Controller Firewall, to a packed room. Part of my presentation also included a section comparing a control system security program to a plant safety program. That like safety, a security program includes a significant effort on user education and training. We all need some basic cyber-security efforts and we just need to do something now rather than waiting for some complicated security program to develop.

Mark Fabro led an afternoon workshop discussion, "CyberSecurity Who Needs It" on how to understand the emerging threats and the practices countermeasures we can develop to mitigate these threats. We really need to have suppliers, users and the public sector to work together in this effort. Mark thinks there is a lot of fear, uncertainty and doubt in the user community about the "real" threats and how to mitigate them. Next was a refining customer with a management-oriented workshop on "Cyber Security and your Bottom Line". He was making the point that management is reluctant to spend the money on security. They need to justify how does this "help us make better oil" where better question might be "how does security keep us making oil". If your assets are at risk--water, power and environmental systems how long can we stay running? He also made the excellent point that when setting up a security policy that it "if it is important enough to make it a policy it is important enough to fire somebody for violating it."

The final presentation of the day was by two Oil & Gas customers with their presentation on "Cyber Security in a DeltaV Environment: A Harmonious Relationship". It was attended by over 50 people. Being the last presentation after a long day of being "PowerPointed" to death shows the serious concern manufacturers have about cyber security. They recommended a NIST publication 800-37 to help users develop their security program. A point was made discussing a key security concept--called “Defense in depth” and defined this as the concerted use of multiple security techniques to mitigate the risk of compromise to an acceptable level. At the same time they strongly advised that process user to be sure and use Defense in Depth techniques that are appropriate for use in the control systems and to not blindly deploy IT-based solutions that might impact the availability of the control system.

All in all, the Emerson Exchange developed an excellent and well attended set of control system cyber security workshops that provided process manufacturers with some great and pertinent information on keeping their DeltaV control systems as secure as possible.

Tags: cyber-security | SCADA |

October 17, 2006 in Cyber-Security | Comments (0) | Trackback (0)

My RSS search on cyber security found an interesting post the other day by IBM's Todd Watson entitled How To Keep the Internet Sky From Falling.

It's especially interesting to me because I've had the chance to meet Todd who is also based here in Austin, Texas. He offered some great guidance in the early days when we were trying to launch the Emerson Process Experts blog.

The paper Todd referenced is by the Business Roundtable, Essential Steps Toward Strengthening America's Cyber Terrorism. Although this paper is mainly concerned with the loss of the Internet and Wide Area Network capabilities, it does have thoughts that process manufacturers around the globe need to consider.

I ran Todd's post by Bob Huba who is leading the efforts on cyber security as it applies to Emerson's DeltaV system. He's part of a newly formed cyber security testing consortium for the process industries.

Bob thought the paper as it applies to owners of control systems brought two points to mind. The first is to keep the control system completely segmented from internet traffic and the second is to not be dependent on information from outside the control system to perform basic control functions. This is especially true if the information required for control is coming from outside the facility over the internet.

As part of control system security best practices Bob always promotes the idea that in a crisis situation on the plant LAN, such as a serious worm or virus attack that could leak into the control system, you absolutely must be able to sever the external LAN connection(s) with the control system until the issue is resolved. The control system must be able to keep functioning at some acceptable level with this connection severed. This is why the recommended DeltaV approach is that the optimization and other supervisory type control tasks be done locally in the DeltaV system whenever possible.

This model is being used in universities and colleges where they have a “student LAN” for email, instant messaging, web access, etc. that is aggressively segmented from the main university system with very few interconnections. These connections can be highly secured and monitored. They can be easily and quickly severed if the "student LAN" gets infected or attacked so the main system can be protected.

This is the model used in the initial development of the DeltaV system and it is the model that is still enforced. The model is based on enforcing a high degree of segmentation between the control system network and plant LAN so that critical control system functions are safe-guarded as much as possible from threats originating on the business LANs. By using very limited external connections, these connections are easier to protect and monitor and can also be easily severed when necessary.

Bob has described more of these best practices in two whitepapers: DeltaV System Cyber-Security and Best Practices for DeltaV Cyber-Security.

Tags: cyber security | control network segmentation | control system network |

June 30, 2006 in Cyber-Security | Comments (0) | Trackback (0)

As Control magazine editor-in-chief, Walt Boyes, mentioned in his Another one joins the club... blog post, Emerson has joined the companies who are sponsoring the security consortium feasibility study to be performed by Wurldtech Analytics.

I spoke with Bob Huba whom you may recall from an earlier post on cyber security and the DeltaV system.

Bob participated in the initial meeting to kick-off this consortium which was held prior to the Process Control Security Forum meeting last week in San Diego. He's excited about the formation of this group because he believes that one of the things that will help automation system cyber security is the ability to certify when the automation system components or even when the system itself meets a minimum level of security or protection. Without some kind of certifying capability, it's difficult for the end users who manage the system day-to-day and system suppliers to fully assess how secure their systems might be.

Automation system security currently has no organization like the TÜV and other certifying agencies where these suppliers can go to get a device certified for different security levels. Cyber-security testing needs some sort of agency to provide the framework for device and system testing and to help manage the information around best practices (or at least generally accepted practices) for creating and maintaining a well protected system.

Bob is really glad to see that this initiative is being driven by the user community and not just the suppliers or some testing organizations because it shows they understand and support the need for a certifying body. The system suppliers really appreciate being included in the discussion right from the beginning to capture the wealth of expertise and perspectives everyone brings.

The landscape around system security or the environment around system security is maturing rapidly and it is important that process manufacturers and suppliers work together and work quickly to address issues around cyber security. This group has set an ambitious time frame for kicking off this consortium and becoming fully functional.

Also, the scada security blog (to subscribe) has a nice wrap up of the Process Control Security Forum.

Tags: cyber security | automation system | control system | security testing foundation | Process Control Security Forum |

June 14, 2006 in Cyber-Security | Comments (2) | Trackback (1)

After reading CONTROL magazine's Editor-in-Chief Walt Boyes’ Compared to Wireless? blog post, I spoke with DeltaV product manager, Bob Huba, who oversees the cyber-security requirements and developments in the DeltaV system. Bob authored the DeltaV System Cyber-Security and Best Practices for DeltaV Cyber-Security whitepapers.

Bob, a voting member on the SP99 committee, has a slightly different take than Walt’s assessment that what is driving the committee,

...has as its end result getting the Department of Homeland Security off our backs...

He believes the automation suppliers and end users really understand the importance of having some sort of standards to measure control system security. In fact, Bob says the greatest push for this standard is coming from the user community. They are driving this committee the hardest which seems to be bringing greater cooperation and more focus on getting something done.

If you are interested in some of the work to date by the SP99 committee, you can purchase these reports from the ISA:
ANSI/ISA-TR99.00.02-2004 Integrating Electronic Security into the Manufacturing and Control Systems Environment and ANSI/ISA-TR99.00.01-2004 Security Technologies for Manufacturing and Control Systems.

Tags: cyber security | SCADA | SP99 | ISA | ANSI |

May 3, 2006 in Cyber-Security | Comments (0) | Trackback (1)