Operator Training

If you're involved with simulation in your plant, you may be familiar with Mynah's Martin Berutti and his blog. In one of his posts, he shares his thoughts on the business benefits, requirements, and steps for building a Virtual DeltaV system with a virtual plant and I/O.

I thought I'd share some of his thoughts from the presentation, which are not specific to the automation system. You may find yourself needing to capture your plant's experienced operators and operations personnel through the use of simulation, before they all retire to warmer, sunnier locales.

Martin shared some U.S. demographics that the average age of an energy industry worker is over 50 and that half the current work force (more than 500,000 workers) will retire in the next 5-10 years. Some of these retirements were accelerated by the global economic slump and ironically may accelerate again when the equity markets recover for those whose retirement funds dwindled.

As shared in a post last week, regulations and government oversight continues to grow, increasing the load on the plant's operations team. These regulations combined with global supply pressures on financial margins add to the operations burden.

The paradox that the era of plant automation has ushered in is that operator error is the highest cause of loss, followed by design error, process upset, and mechanical failure. The first two are directly related to experience and skill level. This regulatory environment has changed over the years where now operations management can be held liable for their actions or inactions on operations issues. Martin observes, "They didn't tell us in engineering school we could go to jail for something we did or didn't do!"

If you're considering simulation as a way to capture the operations team's expertise for the next wave of operators, maintenance techs, and plant engineers, Martin suggests four simulation approaches to avoid:

• The first is not to use process design models since they do not have the real-time performance or range of operating conditions of the dynamic simulations required for operator training systems.
• The second is not to emulate the automation system. If you're going to build skills and gain experience on the operator graphics, alarms, and controls, these items should be identical to the real system--not an approximation.
• Third, Martin councils to avoid adding simulation to the control system configuration. This increases opportunity for errors, adds complexity, and ups the risk of design errors. Adding simulation to the control system configuration also makes the process of keeping the operator training system consistent with the on-line control system difficult if not impossible.
• The final caution is to avoid starting the simulation development too late in the project cycle. These efforts are usually rushed and don't provide the depth of training that operators and other operations personnel need to acquire the skills and confidence to operate the process after it is commissioned.

The proper approach is to have a virtual control system, which is an exact replica of the plant automation system. The operator graphics, alarms and controls are identical to the running system. Also, the virtual system can be the testing grounds for new and modified control strategies.

Connected with the virtual control system is a virtual process/dynamic simulation. The fidelity of this model can range from simple I/O signal modeling and device tiebacks, to mass and heat balance models, all the way to complete mass balance, rigorous heat balance, reaction kinetics and associated thermodynamic properties. The level of model complexity depends on the initial business objectives, amount of knowledge capture, and skill level sought.

The virtual control system combined with the virtual process forms, in the words of ModelingAndControl.com's Greg McMillan, the virtual plant.

Knowledge transfer requires explicit learning--what the operating procedure says, implicit learning--how things really work, and tacit learning--how decisions made affect the whole process. Properly done, simulations provide the hands-on training for these three types of learning.

Defining your objectives clearly up front and following some of the guidance shared by Martin, can help reduce the errors and associated liabilities/risk, reduce operating costs through less unscheduled downtime, improve product quality, and increase time to market by reducing startup time.

MP3 | iTunes

Helen Chen, a project engineer and member of Emerson's TAG business, shared a recent project with me where she developed modular, interactive operator training screencasts for a DCS modernization project. TAG provides specialized services for process manufacturers on control system projects--from studies and system documentation to turnkey installations and follow-on support for DCSs, PLCs, SISs, etc. They perform projects with many of the process automation suppliers' systems.

The story begins with the wisdom of a first grade student who developed a narrated PowerPoint presentation to describe her vacation. Her teacher happened to be the spouse of the project manager on this large modernization project. He was thoroughly impressed with the narration and wanted to bring something like this into the operator training for this project.

The standard training is usually a presentation through a written manual. He wanted something pre-recorded, interactive, and site specific for this particular customer because the team was migrating this plant from a single loop board mounted controller system to a DCS. It was a huge leap for the operators to make.

Helen spent time learning the Adobe Captivate software. During the Factory Acceptance Test (FAT), she was able to do a lot of recording on the system. She also made a point of asking the operators there what their concerns were during that time. They were concerned with everything...navigation, point manipulation, alarms. One of them shared with her that this just didn't seem real and it was "like playing with video games and most of us aren't computer savvy."

From these interactions, Helen developed a training content from System Introduction (how to log in...) to Navigation, Point Manipulation, and Alarms...everything that the team thought the operators needed to be able to operate their new DCS system. From that point, Helen used their system including their graphic displays to make recordings of all the topics in the training content, such as navigating using the custom buttons, changing setpoint, etc. The final step was to publish in html format and to burn to CD.

The project team took this training to the process manufacturer's site and presented the individual topics to the trainees. They encouraged interaction between the trainers and trainees by having the trainees work through exercises including a workbook they developed. This training was done on a simulated DCS system while the trainers proctored and answered questions.

They were able to address all the questions that the trainees had as soon as they came up so that they could thoroughly understand this material. After the training, this pre-recorded interactive training remained with the site in CD format.

Having the training pre-recorded also helps keep it consistent. The team provided a training manual and "cheat sheet" for quick referencing. Now the curriculum appeals to the visual, auditory, and kinesthetic learners. After the training, the project team saw that operators became very comfortable with the new system and that was a tremendous advantage during actual system cutover.

The vision cast by a first grade student has been extremely well received by the site personnel and this modular, interactive operator training will be included in upcoming projects at the site. Helen shared a screencast with a short excerpt (6 minutes) from the actual operator training to give you a flavor of what they are now using.

Sometimes valuable innovations are spawned from unlikely sources.

MP3 | iTunes

Chemical Processing magazine has a great article, Build Operator Expertise Faster, written by folks from the Center for Operator Performance (COPS). The article provides recommendations for how best to transfer the expertise of experienced plant operators who are rapidly approaching retirement age. The article opening well describes the challenge:

Most process plants are struggling with the dual challenges of increased requirements for safe and efficient operation and expected retirement of a large portion of the workforce. Sites likely will lose substantial expert knowledge at a time when it's of greatest need. This will ratchet up pressure to more quickly develop expertise in newer operators.

I highlighted some of COPS' research on operator performance in earlier posts. As one of the consortium's founding members, Emerson is applying the research in its product development efforts to improve usability for operators, engineers, and other plant staff members.

University engineering curriculum stress not only theory, but also practical application in labs, programming courses, internship opportunities, etc. It's a way to apply the knowledge and gain experience in what it takes to be a practicing engineer. Likewise, for operators the authors write:

Acquiring knowledge isn't enough, though. Novices also must engage in deliberate practice applying that knowledge, recognizing key information, setting goals and executing actions.

Some do's and don'ts include don't have experienced operators give "data dumps" of what they know, do review actual events and ask inexperienced operators for possible causes. Performing scenarios of production rate changes and abnormal situations are also good learning experiences.

In an earlier operator training post featuring Emerson's Jeff Hackney, I described a project where these scenarios were developed to train operators:

...simulation includes a dynamic model with startup and shutdown exercises and 33 training scenarios to help prepare the operators to work through simulated abnormal plant situations. Instructor graphics allow the refinery instructor to perform activities such as isolation valves opening and closing, and manual drain and filling valves not controlled from the plant DeltaV automation system.

The authors note one of the unintended consequences of improved automation technology is that it can diminish expertise. As the systems get smarter and perform more advanced controls it takes away some of the analytical skills required. The authors note:

Automation can diminish expertise in three ways. First, it can dull the skills of veterans. Second, it can slow the rate of learning, so people take much longer to build up their expertise. And third, it can teach dysfunctional skills that will actively interfere with building expertise in the future.

They recommend creating visual decision trees that reflect the decisions the advanced controls are making. This helps operators to see the changes and assess why they are being made to understand the process better.

There's also some do's and don'ts presented on on-the-job training and tailored operating training based upon expertise level. The article is worth reading if you are facing the prospects of experienced operators approaching retirement age.

MP3 | iTunes

A few days ago, I saw this Twitter tweet from Emerson's Jeff Hackney:

Train fm home - Emrsn Virtual Training classes for MiMiC V3 (Dec 1, 09) and DeltaV 7009 (Nov 2, 09) http://tiny.cc/Emerson_VLearning

Jeff is part of the Emerson Educational Services team and I was curious to find out more about this virtual training. I discovered what makes this different from traditional e-Learning sessions is the live instruction across the internet as well as remote access to the training equipment. The instructors are based here in Austin, Texas and deliver classes at scheduled times--11/2 and 12/1 as Jeff notes in his tweet. The virtual training course provides internet access to classroom-based equipment such as DeltaV systems and MiMiC simulation software.

One of the barriers in these economic times to taking training courses is the travel and living costs for course. Or, the travel and living costs of the instructor(s) plus the equipment shipping costs for training at a process manufacturer's site.

The first course offered, DeltaV 7009V on November 2, is the virtual training version of the DeltaV Implementation I course, which:

...covers a complete DeltaV system implementation. Upon completion of this course the student will be able to define system capabilities, define nodes, configure continuous and sequential control strategies, operate the system and define users and security.

On Martin Berutti's blog on the Mynah website, he wrote about the December 1 virtual training course for MiMiC:

We strongly recommend this course and are very excited about the new Simulation with MiMiC V-Learning school offered by Emerson Educational Services. MiMiC Simulation Software for DeltaV Testing and Operator Training is a proven technology for saving time and money on DeltaV projects. The Simulation with MiMiC school offers a step-by-step approach to using MiMiC on capital projects or OpEx initiatives. Troy Nelson is an excellent, professional instructor with a deep, practical knowledge of MiMiC and dynamic simulation.

It will be interesting to see the initial feedback from both the DeltaV 7009V and MiMiC courses in having virtual access to the Educational Services instructors and training equipment.

MP3 | iTunes

Our internal blogs at Emerson (blogs.emersonprocess.com for those readers with access to the Emerson internal network) are vibrant and growing. One of the folks I've featured in past posts, Jeff Hackney, who leads the DeltaV operator training solutions (DeltaV OTS) global sales team, has joined the ranks of internal bloggers.

In one of his posts, he highlights an OTS team project that delivers a complete operator training solution. This operator training system supports 4 butane splitter columns, butane butylene treater unit, propane propylene treater unit and utilities (cooling water system, steam/condensate system) for a U.S.-based refiner.

The simulation includes a dynamic model with startup and shutdown exercises and 33 training scenarios to help prepare the operators to work through simulated abnormal plant situations. Instructor graphics allow the refinery instructor to perform activities such as isolation valves opening and closing, and manual drain and filling valves not controlled from the plant DeltaV automation system.

Another post describes an operator training solution for an offshore oil and gas producer. This training solution supports two offshore platform wellheads and all downstream process areas through to sales gas to the pipeline company. In addition to the platform dynamic models, the OTS team also emulated safety instrumented system (SIS) shutdown logic. For this oil & gas producer, the team developed 47 training scenarios including startup and shutdown, instructor graphics, deck operations graphics, and random scenario generation. The random scenario generation allows the operators to train on the system without an instructor.

Jeff notes that some believe that building a large, dynamic simulation model and integrating it with various emulated or simulated control systems is a relatively simple activity that can be grasped within a few days. The reality is that updating a complex process model needs a detailed understanding of the process itself, simulation technology, the automation system and a design for the interaction between these components. It takes a combination of skills that most individuals usually do not possess. The OTS team works with process manufacturers to augment the skills within the plant to develop and maintain these operator training solutions.

Operator training systems become increasingly important as experienced operators retire and new ones join the ranks. Having instructor-led or instructor-less training systems can help develop the skills needed to handle abnormal plant situations when they arise.

MP3 | iTunes

Emerson's Pete Sharpe, an advanced automation consultant, was recently interviewed for a magazine article on the subject of process manufacturers' aging workforce. Manufacturers primarily in North America and Western Europe are feeling this loss of experience due to retirement.

Pete indicated that the reoccurring theme he hears is the loss of deep process knowledge. This knowledge, possessed by experienced operators, maintenance technicians and senior automation engineers is knowing when something in the process is not quite right, like when a measurement reading looks off, or when the process behavior is not the same. As seasoned operators retire, manufacturers feel this loss of experience most during non-normal conditions like startup, shutdown, or emergency situations.

Problems take longer to diagnose and resolve which can lead to less stable, more dangerous conditions. A recent refinery accident is one example where the problem diagnosis did not occur soon enough. You don't ever hear about most incidents but they certainly cost manufacturers money and often result in close calls that don't actually shut down a unit.

Pete discussed a couple of things automation suppliers are doing to address these issues. The first is improved operator training simulators, which I have written about in earlier blog posts. In this environment, less experienced operators can be challenged with operating problem situations so that they can improve their ability to diagnose the process and respond more quickly to abnormal situations.

As technologies continue to advance, more diagnostic capabilities are available in smart field devices and other plant assets. These devices can provide early warning about their own health and about the surrounding process. These predictive capabilities improve the ability of the maintenance organization to prioritize and respond to critical equipment alerts. Emerson's Abnormal Situation Prevention (ASP) algorithm uses process statistical signature data to give less experienced operators more time to react to abnormal situations and more diagnostic information to point to the root cause of the abnormal situation.

The final advancements that help to close the experience gap are advanced control technologies. As the technology has gotten increasingly scalable and easier to deploy in control systems like the DeltaV system, more and more processes can be operated as units and not as collections of loops. These APC technologies operate a process unit within its equipment constraints, at its most economical point. The operator's role changes from constantly adjusting individual loops to setting targets and constraint limits. APC applications are especially useful for process units that are tricky to run by less experienced operators--where many of the loops interact with one another or the process is highly constrained.

These advancements help ease the learning curve for future operators, maintenance technicians and automation engineers. On the positive side, today's engineers and young operators are nearly all computer-literate, so they can make good use of the modern tools and work processes that come with today's control infrastructure. This computer-savvy generation is more likely to adapt to computer-based control systems and modern fieldbus architectures. In addition, automation suppliers like Emerson are helping to ease this knowledge gap by having people like Pete and the other advanced automation consultants available to work with process manufacturers.

The folks at Mynah Technologies with their Mimic simulation software, virtual I/O network gateways, PLC I/O interfaces and host of drivers for the DeltaV system continue to build conversations through their forums and experts blogs within the forum. Recently Dr. Aleksandr Muravyev a simulation consultant created a Mimic Distillation Modeling package to simplify software acceptance testing and operator training around distillation columns.

Dr. Muravyev joins the automation blogging fray with this post about this distillation modeling package. If you have experiences with this product (or even if you don't) feel free to join in the conversation. It's a forum so you'll need to register first.

I did ask the Mynah folks if the forum software they are using supports RSS feeds so I could get these updates coming to me instead of going out and seeking it. It does and they will soon be adding this functionality.

That's a great thing as some RSS readers like Google Reader support both regular and mobile viewing. It means I can keep up with the automation and technology bloggers whenever a have some spare moments. From playing with the various RSS readers, I seem to be gravitating to using Outlook 2007's embedded RSS reader for my Emerson intranet-based RSS feeds and Google Reader for external RSS feeds. How about you?

As members of the generation known as "Baby Boomers" begins to retire, so too does quite a bit of experience among process manufacturers. A recent article entitled Knowledge continuation at your company from ReliabilityWeb.com said it well:

Over the next 10 - 15 years the baby boomers will be leaving the industry in droves. As with every other aspect of their lives they will create a ruckus when they do so!

In this case they will be taking out a large amount of the experience, knowledge and skills that they have built up over a lifetime.

You may remember Jeff Hackney, DeltaV OTS Project Manager in our Educational Services organization from an earlier post.

Jeff had a great perspective when I ran the ReliabilityWeb story by him. I'll not attempt to improve on his words:

Just like families today struggle to pass on their lineage and family history every industry faces the challenge of conveying the knowledge of their experienced work force to those who will replace them.

Some may have learned to avoid the Aunt who pinched your cheek or laid a red lip stick kiss on your cheek through first hand experience and embarrassment but if you were lucky you had an Uncle or older cousin warn you ahead of time.

Experienced operators are no different in that they have acquired a unique perspective on their processes and honed the actions required or warranted for specific process conditions. A key attribute of a training program is not only its ability to distribute new data to students efficiently and effectively but to also facilitate the transfer of knowledge from experienced to novice operators.

DeltaV OTS is an excellent tool that is part of a complete training program that not only introduces new operators to their process and operational environment but serves as an identical plant platform experienced operators can convey "tricks-of-the-trade" the "new guy" may not have been exposed to while under the wing of the experienced operator during operational system indoctrination.

As we all know, the operational system can't be forced toward a trip condition to facilitate showing the new guy how to recognize, respond and correct the condition before the trip occurs. DeltaV OTS provides that identical system to maximize the training effort and transfer of knowledge.

Process manufacturers are modernizing their automation to increase the efficiency of their operations. A key requirement is skilled operators who are essential to effectively use the technologies these new automation systems deliver.

An October 2005 study of chemical manufacturers by the United Kingdom's Health and Safety Executive indicated that 37% of plant incidents could be categorized by incorrect operator actions.

Also, operator actions can have a big effect on process variability which can impact throughput, product quality, energy usage, and downtime. These all have an impact on the profitability of the operation.

Another issue many process manufacturers face is the expertise of their operators to effectively run the plant. In many countries with aging populations, the experience levels of operators are dropping as many reach the retirement age.

To help manufacturers address these issues, Emerson Educational Services has developed a number of solutions to help better train operators. These solutions include web-based operator training, formal classroom training, packaged fixed price Operator Training Simulation solution with customized curriculum, and full plant-wide simulation with customized curriculum for operator training.

I spoke with Jeff Hackney, DeltaV OTS Project Manager, about the new packaged Operator Training Solution called DeltaV OTS Express. Jeff told me DeltaV OTS Express is an affordable solution that provides operator training specific to the actual process for a fixed price.

DeltaV OTS Express is sized to target a specific critical unit, area, or process. With a combination of software and services, the operator training is built to be identical to live plant system configuration and graphics. Jeff, his team, and the global Ed Services team start with the manufacturer's DeltaV configuration database and graphics.

The DeltaV OTS team also uses DeltaV Simulate and the MiMiC process simulation software and customized courseware to create a training environment with process specific conditions, training scenarios and assessments to teach the operator process and skill-based knowledge. In addition to the simulation model, courseware and software, the PC hardware is included. The training solution is ready to go when it arrives and helps operators learn and refresh their knowledge of the process over time.

The DeltaV OTS Express is built with flexibility in mind to meet the financial and logistic needs of the customer. The base offering can be expanded in order to deliver simulation-based training to an entire class of operators if needed.

Jeff stressed that this is not a high complexity, high cost, high fidelity model which is designed to model the process as closely as possible, but rather something that helps the operator learn how to better operate and understand the process. By employing the actual automation system configuration and graphics, the DeltaV OTS Express solution can be delivered in one to three months, instead of a more typical one to two years for the more complex, high fidelity solutions.