Alarm Management

I received an email from the Center for Operator Performance (COP) about a newly completed study, Color Usage in Graphic Displays for Process Control. Emerson is one of the founding members along with members from process manufacturing, academia, EPCs, and automation suppliers. Emerson's Mark Nixon, who leads the research efforts for the DeltaV system, is also the chairman for these research efforts at the COP.

The center oversees research to meet the needs of the members and is responsible for contracting universities and human factors companies to conduct the research. The center also serves as a repository for human factors data in process control and training in human factors as requested by the members. Research interests include the following:

Expertise - With attrition of operators, much expertise is walking out the door of process plants. What makes an expert operator? What skills does an expert possess that a novice does not? How can novices become experts faster?

Simulator Effectiveness - While simulators in general are a wonderful tool for enhancing operator performance, their application in process control has historically yielded mixed results. What are the key attributes of a successful simulator training program? How important is simulator fidelity in the success of training?

Graphics & Data Presentation - Process operators deal with thousands of process variables that ultimately become the basis for a single decision. How can graphics better support this effort? What impact does background color have? Does the use of one color for more than one meaning impact performance?

Alarm Actuation Rate - What is the upper limit for alarm processing? How long can this limit be sustained without impacting performance?

The Operator Display and Color Usage study is the first of a multi-part study investigating the overall topic of display design. In this study, the researchers reviewed the current literature, surveyed operators, and visited a number of operating sites. A key component of this project was to bring the researchers up-to-speed with what the state-of-the-art is in the petrochemical industry.

As part of their learning, the researchers were looking for evidence on whether or not best practices related to color and visualizations are being followed. The 99-page color usage study is available for members of COP and was prepared by Dr. Jennie Gallimore and Jennifer Shinkle, with Wright State University. With Emerson as a member, I was able to get my hands on a copy and here are a few things I gleaned. If you're interested in the full research, here's the COP contact page.

One conclusion that the researchers came back with was that although there is considerable research on the use of color and other visualization techniques for display design, guidelines specific to the petrochemical industry are scarce. Color as well as other guidelines such as position, form, and animation could potentially help display designers to improve their overall display implementations.

The researchers also made another observation; color is probably not the most pressing problem, a bigger factor is the overwhelming number of displays and the design and presentation of the information on these displays. For example, although mimic displays such as P&ID's are simple enough to create, they are not necessarily the best way to present information to operators. Further studies are required to look into better ways to organize and present information on displays.

The research was performed working with U.S. refining and petrochemical manufacturers and their operations staff. The mean age of study participants was 46 with an average of 16 years experience. The report notes that more than half of the research participants have some form of corrected vision. As someone in this age demographic and needing those "cheater glasses" myself for dimly lit rooms, I can appreciate this growing trend.

The research also looked at lighting conditions in the operator rooms, environmental conditions (temperature, humidity, noise, and vibration), total colors used in operator displays, alarm-related colors used, and different aspects of display element effectiveness. It looked at many other things too, including display technology, considerations in vision and color perception, and ways color is used in visual displays. From the responses, the study points to opportunities for improvement to better define color usage and visualization guidelines.

I asked Mark his key take away from the research. Mark notes that although there is considerable research behind visual encoding techniques, that research has not made its way into our industry. A key challenge that people responsible for configuring displays face are that there are few guidelines describing best practices and for the best practices that do exist, there is very limited research proving that the techniques actually work in our industry. The center's goal is to provide that research. A well-designed operator interface will improve overall plant operations and environmental, health and safety conditions. The members of the COP share these objectives and are jointly funding this research.

MP3 | iTunes

I've known Emerson's Cindy Scott a long time in our days building the DeltaV brand, and even earlier. I've not known Cindy to be versed in the use of the Queen's English by spelling words such as color, colour. I can only assume the article I read, Don't let colours hide the alarms!, had the hand of an editor to convert Cindy's words.

The technology change for operator display screens has been dramatic from the introduction of distributed control systems in the 1970s. The early versions had eight colors choices to choose from including "blink". So red and blinking red were two of your eight choices.

Today's human machine interface (HMI) software supports what the current PC graphics cards support, which is typically 16.7 million colors. And of course, these 16.7 million colors can blink, or bounce, or rotate, etc. Also, display resolution has increased from a typical 640x480 pixel grid, to 1280x1024 pixels, up to 1680x1050 widescreen.

Many company and plant standards were developed while the display technologies were less mature. The ANSI/ISA-S5.5 Graphic Symbols for Process Displays standard came out in 1985, which was also during this early display technology stage. These standards have not kept pace with the technology changes and are in revision by the ISA101 committee. Cindy wrote:

When systems were upgraded, the plant display standards were not updated to incorporate the capabilities of the new display technologies, so the basic display layout and colour schemes were maintained. Although this makes sense to minimise cost and operator disruption, it forgoes many of the potential benefits of current graphics capabilities.

Having all these technology advances and extra colors does not necessarily improve the performance of operators. The article asked some thought-provoking questions when it comes to color and the design of plant operator screens:

• How many colors are too many?
• Which are the 'right' colors to use?
• When is it appropriate to use dynamic color?
• How much data is too much data?
• What alarms are needed and where should they be shown?

The article does not answer all these tough questions, but does illuminate some of the existing research and does offer ideas for process manufacturers to consider in their operator display design standards.

Cindy cited the research of North Carolina State University's Christopher Healey in 'preattentive" visual processing:

Healey points out that for a particular item to be picked out rapidly from a group of socalled distracters, there must be a single difference between the desired item and the distracters. Thus a red item can be spotted readily among a group of blue items of the same size and shape, while a round item can quickly be picked out of a group of square items of the same size and colour. On the other hand, a red circle in a group of red squares and blue circles will not leap out from the screen and must be found by searching.

Cindy also shared the work of the University of Southern California's iLab, which researched the effect of color, shape, and object orientation. Changes in an object's color can be lost if the color matches other background colors. If the change has a unique color and shape, it is much more easily detectable.

Applying the results of this research is not always easy in operator display alarming. Operators are used to the status quo and may resist changes. Automation suppliers may also want to keep the existing standards that may favor their automation systems. Change occurs through a process of analysis, review, and buy-in among the operations, maintenance, and plant engineering staffs.

Cindy's key takeaway is to be judicious in the use of colors. She concludes:

It is important for control system display designers to resist the temptation to add colour to everything, and to use it only where it conveys useful information. Bright colours should be reserved for things that must be seen such as alarms, and must not be masked by using the same colours for non-critical items. Reducing operator errors and accidents hinges on an understanding of visual perception mechanisms.

MP3 | iTunes

Emerson's Jonas Berge is an active member in the ISA SP104 committee, responsible for advancing the Electronic Device Description Language (EDDL) standard (also known as IEC 61804-3.) You may recall Jonas from earlier EDDL posts. This standard creates interoperability between digital field devices from simple sensors to complex devices (drives, analyzers, etc.) with control and asset management systems. Interoperable communications include device diagnostics, asset management and user interface displays.

Jonas has written a short piece, OPC Made Easy, in the April issue of Control Engineering Asia magazine. In this article, he describes how EDDL can save many hours of OPC server configuration, which can help speed up a project's completion. For background, he begins by reminding readers how this important standard makes sharing data between OPC servers and OPC clients easy:

...external software in HMI clients and other users can easily access the wealth of detailed diagnostics and information in hundreds or thousands of intelligent devices around the plant.

Configuring OPC clients is easy: just point and click on data in the OPC server.

The challenge is in the configuration of the OPC server:

Configuring the OPC server includes entering device addresses and communication settings as well as creating the "namespace" which entails entering tag or descriptor for each and every piece of information along with the memory register address for the parameter as well as its data type, and range where applicable. This parameter "mapping" is the most time consuming and error prone part of OPC integration, but once done the rest is easy.

Jonas explains how EDDL can automate the creation of the OPC server configuration for devices digitally communicating via HART, Foundation fieldbus and Profibus. He writes:

Automatic OPC server configuration is made possible because EDDL is a descriptive technology similar to XML or HTML, declaring the properties of the data in the device for use by the auto-configuration mechanism. EDDL is the only device integration solution that is declarative.

Although not in the article, Jonas relayed an example to me where an AMS OPC Server was used to pass a slug flow alert from a Micro Motion HART device to an older distributed control system (DCS) that did not support HART communications pass-through. Before the solution was implemented to send this alert to the DCS via OPC, slug flow would cause over-charging of materials added to a batch. Now, the operators are alerted to slug flow conditions and can pay special attention to the surrounding process equipment.

The EDDL.org website remains the best source for information about this standard. You can also join the EDDL email list hosted by ISA to keep up and participate in the conversation around this standard.

My RSS feeds pointed me to a great ChemicalProcessing.com article on batch manufacturing alarms. The article, Rethink batch-manufacturing alarm systems, was written by Joseph Alford.

He opens with provocative questions:

Do operators sing the praises of your plant's alarm system? No? Well, do they at least agree that generated alarms represent real abnormal situations requiring a response and that the automation/control system presents alarms in a timely, accurate and reliable way? No again? Well why not? Aren't operators the primary customers of your alarm system? Perhaps it's time for an alarm remediation project.

Many with continuous processes would agree that their alarm strategies implemented inside their automation systems need work. The complexity is amplified in batch processes because unique operating conditions are created within all of the numerous steps along the way.

The article boils down the crucial steps to take:

The key considerations in achieving effective alarm systems include defining objectives early in a project's life (i.e., in a plant's alarm philosophy or a system's functional requirements), adhering to the definition of an alarm, and implementing alarm-management best practices.

I ran this article by Todd Ham, a senior principal engineer in Emerson's Life Sciences industry organization. You may recall Todd from earlier posts.

Todd agrees completely that a successful alarm strategy begins in the functional requirements stage of a project. The project teams work with pharmaceutical and biotechnology manufacturers early on in a project to document their alarm requirements.

Todd stressed that a good strategy examines not only what conditions require an alarm--typically an adverse effect to personnel, product, or equipment--but also what is the desired response. Is alarm annunciation sufficient? Does this require a device interlock? Should this put the batch in hold? Does quality assurance (QA) need to be notified? This is called exception handling.

In a batch process, the requirements may differ based on the process step. For example, the alarm may be critical during processing, but not important during cleaning. Further, the alarm may only need to be monitored once the process is at steady state. In these scenarios, the control strategy developed by the project team will selectively enable/disable alarms at the appropriate point in the sequence.

Todd cautions that this is not a "one size fits all" exercise. The project team and manufacturer's staff must step through each process unit/system and ask a series of questions to arrive at a solution where alarms are both appropriate for a particular operating state and relevant for alerting operators to abnormal situations.

Todd agrees with the author that this work must be done up front, or the alarm flood, nuisance alarm scenario described in the article will be the result. You can't wait until the end of the project to think about alarm management. When you come right down to it, it's just as important as defining the control requirements for the project.

Earlier this month my DeltaV News RSS feed announced the availability of an alarm management whitepaper written by the ARC Advisory Group. The paper, Emerson Strategies for Abnormal Situation Avoidance & Alarm Management, describes an issue many process manufacturers face in dealing with too many alarms. This makes it hard for their operators to distinguish between critical impending abnormal events and nuisance alarms.

The Emerson approach to address better management of alarms is based upon ARC's Six-Sigma DMAIC (Define, Measure, Analyze, Improve, Control) model and

...includes data collection, statistical analysis, alarm evaluation, and system improvement - all within the context of ongoing evaluation and continuous improvement.

Relative to alarm management, the whitepaper describes the DMAIC process:

Define relates to philosophy development, Measure relates to determining alarm behavior and alarm effectiveness, Analyze relates to root cause analysis and performance benchmarking, and Improve relates to the remedial action necessary to align the prevailing implementation with the alarm philosophy. Finally, Control relates to alarm execution.

Advancing technology is a source of the alarm proliferation. The authors note:

In the days of hardwired controls and alarms, engineers were very stingy with alarms, in part because each alarm point had a cost. The primary issue with alarm systems is there is too much information for an operator to assimilate and act on. Ten years ago, it cost about one thousand dollars to add an alarm. Current automation systems have essentially eliminated the cost of adding more alarms and, therefore, the incentive to limit or rationalize their number.

I have an upcoming post that discusses the importance of up-front planning your alarm strategy when you're planning your project's functional requirements. But what do you do in an existing facility with an alarm overabundance?

The whitepaper addresses building a business case to justify and alarm management project. The keys areas build the business case include safety, unplanned downtime, better information management and reduced troubleshooting time, and changing the role of the operator toward higher-value activities.

ARC notes that unplanned shutdown costs process manufacturers on aggregate between 2 percent and 5 percent annually. This may be a justification opportunity by looking at root causes of unplanned shutdowns in your plant. A review of the alarm and event logs around these incidents can reveal the number alarms the operators saw and the actions they took as a result.

The whitepaper also addresses the important role of the EEMUA Publication 191--Management of Process Alarms in developing your alarm management strategy. A guiding principle described in EEMUA 191:

...a usable alarm system must be relevant to the user's role at the time, indicate clearly what response is required, and be presented at a rate the user can deal with, and be easy to understand.

With this backdrop, the paper explores the applications within Emerson's DeltaV system like the Event Chronicle, integration with 3^rd-party alarm management applications via the OPC alarms and events communication standard, and the DeltaV Analyze alarm analysis program. Coupling these applications with alarm management services can help process manufacturers through the process of data collection, statistical analysis, alarm evaluation, and system improvement.

And much like the safety lifecycle as defined by the IEC 61511 international safety standard, ongoing evaluation of the overall alarm strategy is important throughout the lifecycle of the plant.