Modernization

An activity that benefits process manufacturers and process automation suppliers is the research conducted by the trade analysts and trade press. These groups have a broad base of people across many industries and geographies to which they can reach out to uncover unmet needs and open issues.

I've been closely following the blog of ARC Advisory Group's Larry O'Brien and their recent ARC World Industry Forum in Orlando. In one post, Larry highlights a new study they are conducting, Process Automation System Lifecycle Management Survey. The survey opens:

Users of process automation systems have struggled with the issue of determining when the useful life of their system is over. Since these systems are crucial to the success of operations from so many aspects, having a strategy based on best practices is essential. Modern process automation systems are comprised of many components that each have different life expectancies. Suppliers have worked in concert with their clients to provide replacements for obsolete components that in many cases have extended the life far beyond the original expectation. This survey will help determine what the current state of the industry is and how it relates to best practices. We will be happy to send you a consolidated copy of survey results when the survey is complete. Thank you for your cooperation. All of your information will be kept confidential.

The results from this type of research can be beneficial to the technologies, services, and programs developed by automation suppliers to better meet the expectations of their customers. The Emerson SureService team is one example of an organization that is interested in the results and analysis that flows from automation lifecycle management-related research.

I know many like me feel they are surveyed to death, but it's a pretty good deal to get the consolidated copy of the survey results to see how your peers answered the questions you answered.

If you are a process manufacturer and have some thoughts to share about the lifecycle of your automation equipment you may want to take the survey and see the results when they become available.

MP3 | iTunes

Helen Chen, a project engineer and member of Emerson's TAG business, shared a recent project with me where she developed modular, interactive operator training screencasts for a DCS modernization project. TAG provides specialized services for process manufacturers on control system projects--from studies and system documentation to turnkey installations and follow-on support for DCSs, PLCs, SISs, etc. They perform projects with many of the process automation suppliers' systems.

The story begins with the wisdom of a first grade student who developed a narrated PowerPoint presentation to describe her vacation. Her teacher happened to be the spouse of the project manager on this large modernization project. He was thoroughly impressed with the narration and wanted to bring something like this into the operator training for this project.

The standard training is usually a presentation through a written manual. He wanted something pre-recorded, interactive, and site specific for this particular customer because the team was migrating this plant from a single loop board mounted controller system to a DCS. It was a huge leap for the operators to make.

Helen spent time learning the Adobe Captivate software. During the Factory Acceptance Test (FAT), she was able to do a lot of recording on the system. She also made a point of asking the operators there what their concerns were during that time. They were concerned with everything...navigation, point manipulation, alarms. One of them shared with her that this just didn't seem real and it was "like playing with video games and most of us aren't computer savvy."

From these interactions, Helen developed a training content from System Introduction (how to log in...) to Navigation, Point Manipulation, and Alarms...everything that the team thought the operators needed to be able to operate their new DCS system. From that point, Helen used their system including their graphic displays to make recordings of all the topics in the training content, such as navigating using the custom buttons, changing setpoint, etc. The final step was to publish in html format and to burn to CD.

The project team took this training to the process manufacturer's site and presented the individual topics to the trainees. They encouraged interaction between the trainers and trainees by having the trainees work through exercises including a workbook they developed. This training was done on a simulated DCS system while the trainers proctored and answered questions.

They were able to address all the questions that the trainees had as soon as they came up so that they could thoroughly understand this material. After the training, this pre-recorded interactive training remained with the site in CD format.

Having the training pre-recorded also helps keep it consistent. The team provided a training manual and "cheat sheet" for quick referencing. Now the curriculum appeals to the visual, auditory, and kinesthetic learners. After the training, the project team saw that operators became very comfortable with the new system and that was a tremendous advantage during actual system cutover.

The vision cast by a first grade student has been extremely well received by the site personnel and this modular, interactive operator training will be included in upcoming projects at the site. Helen shared a screencast with a short excerpt (6 minutes) from the actual operator training to give you a flavor of what they are now using.

Sometimes valuable innovations are spawned from unlikely sources.

MP3 | iTunes

Today is the first of two guest blog posts this week. Aaron Crews graciously agreed to write this while I'm goofing off this week on vacation. Thanks Aaron!

Jim is a brave soul to hand over the keys to the blog while he's on vacation this week. Hopefully I don't do too much damage while he's gone. If I do, maybe Mike Boudreaux can help me roll back the odometer before Jim returns.

As you might remember from some previous posts on this blog, I am an engineer with The Automation Group (TAG), which is part of the Emerson Process Management family. We provide management and technical services for all types of process control system projects. I thought I would take this opportunity to share a story from one of my past projects.

Here at TAG, one major focus of our work is on DCS modernization projects. These projects arise for a variety of reasons, usually including system maintenance issues, a desire to take advantage of new technologies, and to improve overall plant performance.

A big part of this last goal depends on the operators. The importance of data visualization in the operator interface, alarm management, and human-centered design within the HMI have all been hot topics for that reason. In a modernization project, however, that is only part of the equation.

The operator is required to make a big adjustment from the old system to the new one. Graphics may look totally different, navigation between screens is done in a different way, alarm notification and acknowledgement isn't the same as it used to be, and even the way in which the operator physically interacts with the control system can be different.

Every project includes an operator training component, but as anyone who has been through training knows, it's different once you're on your own. That is one reason why we usually recommend a hot cutover between the old and new systems - so that operations gets some transition time and experience before the critical parts of the process are cut over.

On this particular project, however, the trepidation from plant operations was especially strong. In migrating from RS3 to DeltaV, they were changing from a dedicated keyboard with a trackball and keys with particular functions that they had memorized to a traditional keyboard and mouse. After going through some training and getting their hands on the new system, the operators began to feel like it just wasn't going to work for them. Pretty soon, a dedicated keyboard had become a make or break issue for the project.

It was hard to disagree with how they felt. As a "power user" of several pieces of software, I know I can work dramatically faster and more productively when I utilize keyboard shortcuts. Eventually everyone will get used to the new system, but the process doesn't stop and wait for operators to work their way back up to speed.

As an engineer, though, I appreciate a challenge, and I was able to come up with a solution that would please the operators, mitigate the safety and production risks associated with the modernization, and put the project back on track. Using the flexibility of the DeltaV system and its built-in key macro functionality, my project team customized a commercial off-the-shelf, physical, programmable keyboard for DeltaV.

The top half of the keyboard was dedicated to display navigation, giving the operator single-button access to any graphic. The bottom half of the keyboard had a layout designed to mimic that of their RS3 keyboards, with analogous functions assigned to each key.

Needless to say, the keyboards were a hit. They didn't require custom software on the DeltaV stations, they didn't require specialized knowledge to maintain or replace, and the fixed-button layout allowed for quick "muscle-memory" reaction to the process while always keeping the operator focused on what he is doing and not how to do it. The keyboards were not required so eventually they could be removed if the operators decided that they prefer the regular keyboard and mouse.

Half of the 5000+ I/O was cut over hot, and the rest was cut over during a short turnaround, and the plant was brought back up without incident. I have visited the plant several times since then as they have gone through a couple of DeltaV upgrades and the keyboards are still going strong. They are enjoying all the advantages of DeltaV and experiencing none of the worries that they had with their previous generation control system.

Thanks again to Jim for the invitation to contribute here. If you'd like to learn more about what we are up to at TAG or if you have any questions, you can email me, follow me on twitter, or find us on Facebook.

The EngineerLive website has an article, System migration: make sure you improve control, not just replicate, which shares many of the thoughts we've expressed in recent modernization posts.

I turned once again to Emerson principal modernization consultant, John Dolenc, for his thoughts and additions to this article. I'll include them in their entirety:

You can summarize the author's comments as:

Process control systems will get old. The hardware components will eventually fail. Older configurations tools limit your ability to optimize the process. Integration to MES/ERP systems may be limited by older systems. Engineering expertise on these older systems has retired or moved on to new state-of-the-art systems. Spare parts are becoming scarce and expensive.

These are correct statements. So the bottom line is that at some time, older legacy systems will need to be replaced. The questions become: When will these system components fail? What are the options to replace the system? How much will it cost? How can the cost to replace the system be justified?

The author also had some excellent comments on the planning and implementation of a system migration project. He infers that migration projects are not easily and quickly implemented. The planning process is very important, and during that planning process it is import to identify the business and process issues that drive the plant. The migration plan should then be designed to meet the business and operation goals and not to simply replace-in-kind what exists.

Emerson Process Management provides automation feasibility studies to answer these questions. An automation study is always tailored to the specific needs of each client. Typically, the deliverables of an automation study include identification of benefits from automation and estimates of the financial value. An automation modernization / migration plan is developed to address business and operational issues; many times with migration options. A fairly detailed design is completed including bill of materials, scope of work for engineering and installation, cost estimates and schedules. The final report provides the information the client needs to make the decision to move to the front end engineering or implementation phase of the project.

The author also mentions that a holistic approach should be taken for the migration planning process. Although we cannot be sure what the author means by a "holistic approach," we do have our own interpretation.

First; one must expand the scope of a "process control system" to really be a "process automation system" including instrumentation, control valves and other associated applications such as advanced control, safety systems, and asset management. Then one must understand that the solution to a control problem is not necessarily found in the process control system. Is the measurement device the correct choice for the application and process conditions? Is it located in the proper position for optimal control? Is the control valve the correct type and sized properly? Is the positioner up to the task of providing rock steady control? In other words, a control system can not correct all process control issues alone. All components that make up the control loop need to be considered.

Experiences from one of the early system migration projects we completed provide some examples. We were asked to migrate an existing control system in a batch chemical unit. The existing system was an integrated system consisting of PLC I/O equipment, PC-based control, and a custom, PC-based operator interface. As with most integrated systems we encounter, this system had the typical symptoms: Limited flexibility, the system integrator had to make all control modifications and provide maintenance, high cost for integrator services, and limited spare part availability just a few years after being installed. The only migration option was complete replacement.

During the early design phase of the project we discovered that many of the existing instrumentation was in bad repair, weigh cells were used to add the raw materials, and the right-the-first-time factor was in the mid-80s range. We convinced the client that replacement of the bad instrumentation was in order along with the inclusion of mass flow meters and control valves for the raw material feeds to the reactors. The client required that we reverse engineer the system configuration, because they had invested quite a bit into system modifications and they did not want to lose the engineering investment. So in essence, the control logic was close to a replace-in-kind situation.

The right-the-first-time factor was in the low nineties after the first few weeks of operation upon completion of the migration. The only issue preventing this factor being in the high nineties was certain control logic that was added to the legacy system to overcome its control deficiencies. These logic modifications were soon removed.

The lessons learned here are two fold. First, the new control system performance was impeded by retaining the existing control logic in kind versus retaining the desired control functions and allowing the control system to be configured as designed. Second, the addition of better instrumentation and control valves substantially improved the performance of the process, despite the control logic not being optimal.

John, thanks for adding your perspectives to this article!

MP3 | iTunes

Control magazine had a great article a few months back entitled, Control System Migration. It did a great job covering the selection and planning process when modernizing your automation system. Recently, I featured Emerson's John Dolenc and his perspectives on justifying your automation modernization investment. Taken together, there are many ideas to help plan a system migration project.

John sent me an email with his thoughts on the Control magazine migration article. With his writing prowess, I like to kid John that he should also be an Emerson blogger. In that spirit, I'll include his thoughts in their entirety:

I just recently read an article on control system migration best practices. The article noted that the keys to a successful migration project include:

• Selecting the best control system
• Performing a Front End Engineering study to well define the project issues, develop the automation plan, and develop detailed scope of works and cost estimates.
• Using experienced engineering services and following detailed procedures during the detailed design and implementation phase.
• Developing a detailed cutover plan

I agree with the noted engineering procedures, especially the need to conduct Front End Engineering to properly scope and design the migration. However, the article approached the system selection process from solely a technical performance issue from a configuration engineer's / system integrator's viewpoint. The author defined a system selection process that rated system suppliers on controllers, I/O modules, operator consoles and other system hardware/software issues.

Conducting a technical evaluation of a control system is always advisable, but this evaluation is best done prior to any identified project as an exercise to create an approved bidders list. One must also expand the criteria to include new technologies that are found in today's state-of-the-art control systems; especially those technologies that may provide the financial benefits to justify a system migration. Interface capabilities to ERP systems, asset management systems, and the ability to easily incorporate new field communication technologies such as fieldbus and wireless are important to future systems. We always need to remember that a control system is a tool to be used by operations to efficiently run the process on a day-to-day basis. The availability of tools to monitor control and process performance is important. How easily control strategies can be updated, including advanced control techniques, is important for continuous process improvement.

The article began with the assumption that the decision has been made to update the existing control system. No mention was made for the engineering team to review what led to the decision to replace the existing control system. It is vital for the engineering team to understand the financial and operational reasons for replacing the system. The control system issues that led to the replacement must be identified. And it is the ability to best meet these operational and performance issues that should dominate the system selection process.

The operational objectives must also be considered when developing the automation plan. The results from a replacement-in-kind of the existing control system will typically never please plant and operational management. Correcting process performance issues normally involves improving process measurements, correcting poor control valve performance, implementing better control loop strategies, improved interlocking strategies, sequential control strategies and possible advanced control strategies.

Between the magazine article, the justification blog post, and John's thoughts presented in this post, I hope you have new ideas to advance your system modernization efforts.

MP3 | iTunes

Update: Emerson's Aaron Crews with the The Automation Group business adds a nice twitter tweet:

@JimCahill good post. I like the point about technical evaluation of the control system. Our vendor comp analysis looks @ 600+ criteria

Recently, when Emerson's John Dolenc was in Austin, we had one of those hallway conversations about automation modernization projects. He shared a paper he'd written a few years back that contained many of his thoughts on modernization project justification. The paper had many pearls of wisdom that I'll share over the coming months.

John had strong points about being careful using automation equipment obsolescence as a cornerstone of your project justification efforts. He warned that automation system component obsolescence is a real issue that needs to be addressed. The inability of system suppliers to source components for their older systems and the lack of engineering expertise are issues that also should be addressed.

While your maintenance costs may be increasing, he finds that true maintenance savings are normally not large enough to justify the capital investment. Also, the cost of unplanned process shutdowns due to system failure must be factored by the probability of a failure occurring. We all know that electronic components will eventually fail. Predicting the probability of the failure is the difficulty.

Using obsolescence as the primary justification usually means the management team will dictate the cost of system replacement be kept at the lowest possible level. You lose the opportunity for financial gains through process optimization with improved control strategies, additional or more accurate process measurements and improved control actions. Also, you lose the chance to work with the operators to improve the operator displays and alarm management to handle abnormal situations more effectively.

Instead, one needs to consider the advantages afforded with new technology. The opportunity to review the process thoroughly to identify mechanical and process issues should be taken. Process automation modernization should extend beyond the automation system to include the instrumentation, automated block valves, control valves and variable speed drives.

John noted that it is a rare circumstance where well-designed control strategies can overcome these process- and equipment-related issues. Replacing the obsolete process control system without addressing underlying process problems will not yield the operational performance improvements that some would expect with current automation technologies.

Other areas to consider with new automation technologies include:

• Reduced unplanned process shutdowns and reduced maintenance costs using predictive maintenance practices
• Improved production management through increased process information exchange between the automation system and higher-level operations and enterprise software
• Process optimization through historical process monitoring and trending to allow process engineers to disseminate historical information.

When economic times are good, everybody is usually so busy that the status quo prevails. Difficult economic times offer the best opportunity to really take a close look at your operations, baseline it, and develop an automation plan to improve production and decrease production costs. In future posts we'll take a closer look at John's thoughts on addressing system obsolescence, process automation modernization opportunities, and automation modernization planning.

MP3 | iTunes

Update: Welcome, readers of Gary Mintchell's Feed Forward blog. Thanks for visiting!

Update 2: I just received an email from Control Global which includes a link to an article, Control System Migration. It does a great job describing the migration planning process and is something you may want to check out.

I caught up with Emerson's Dan Jacobsmeyer this past October at the Emerson Exchange. Dan is a Flexconnect product specialist and a member of the migration best practices exchange group in the project management office. Dan started to mention some lessons learned on process automation system migration projects and my ears quickly perked up. This sounded like great stuff for a blog post.

Most people modernizing their control systems spend a lot of time in the system configuration with the I/O, control strategies, and graphic displays engineering everything properly in preparation for the cutover to the new system.

Dan's experience is that the source of a lot of the troubleshooting issues and time spent is often associated with the field instrumentation. A critical step in the migration process before the cutover begins is to verify that all instruments in the control system database are working prior to the cutover. Another key step is to find all the loops that are currently operating in manual mode and have a thorough understanding from the operators as to why this is so.

Dan cited some examples seen by the migration specialists that these checks have caught--known bad transmitters, missing process variable (PV) signals from multi-variable transmitters, short circuits caused by water-filled transmitter housings, missing agitator belts, disconnected field wiring, etc.

Each alone can be fixed without too much time and trouble, but when done as part of a complete system switchover, it can take much more time. This is because the source of the error can be anywhere along the path from the system to the instrument. It takes time to troubleshoot each point along this path.

After doing these checks and beginning the cutover, what do you do when instruments do not commission on startup? Dan suggests you disconnect the instrument from the termination panel and test the transmitter or final element. Test the field wiring for opens, grounds, or shorts. Simulate or source the loop to systems like the DeltaV system. Output current and test the loop. Finally, reconnect the transmitter or final element and test it again.

Dan also recommends you have spare instruments available, the specifications to all of the instruments, location drawings of the instruments, and an available instrument technician to field test the instruments where required.

Having the documentation and pre-cutover checks done can save quite a bit of time during the heat of the battle when the unexpected surprises occur, as the firm believers of Murphy's Law can attest they will.

Thanks for sharing these lessons and letting me pass them along, Dan!

MP3 | iTunes

Emerson's J.D. Wheelis, a modernization consultant, wrote a thought-provoking piece that I got my hands on recently. The subject is computer virtualization, and J.D. discussed how you might consider using it for your Windows NT-based applications where the hardware has become obsolete and/or unreliable.

In my "geek hack" time, I've played around with some virtualization software like some of VMware's offerings, and found it quite amazing how an entire operating system with associated hardware peripherals can be virtualized inside another with good performance. I used it when I switched to Windows Vista and needed a Windows XP virtual PC to connect to some legacy IT applications and printers. It's also possible to run virtual Linux PCs inside a Windows-based host and vice versa.

The only "gotcha" I've encountered is if you have an OEM version of a Windows operating system that's tied to the hardware it comes with. You can't shut down the hardware and move the operating system to a virtual environment. You'd need to purchase or transfer a version unencumbered with these license restrictions.

J.D. notes that several process control applications remain only available in the Windows NT environment. The automation engineers in plants and mills run into a conflict between keeping the plant running with the continued use of the application and the difficulties in keeping the application in operation. In addition, the engineers deal with the conflict between keeping these PCs running and satisfying plant IT requirements to eliminate these PCs based on security concerns.

J.D. makes the case that virtualization techniques and products are a way to provide hardware replacements for these NT-based computers.

Virtualization enables the Windows NT operating system to run in a virtual environment created under a different and newer operating system, such as Windows XP, Windows Vista or even a flavor of Linux. The physical machine hosting the virtual machine provides everything the virtual machine needs - the disk space, the memory (RAM), access to the processor, access to both the physical network and a virtual network as the application requires.

J.D.'s hands-on experience is similar to mine with respect to performance. He observes:

In my experience in installing and using virtualization for a control system configuration application, I saw no difference in the performance of running the application. Everything worked, from complicated graphics display and editing, database operations, sub-applications designed to perform special tasks, to communications through networking, both from virtual machine to host and virtual machine to another physical machine through the host's network connection.

With respect to security:

The security issues on networked virtual machines remains if you are using the application to communicate over the physical network to a different physical machine. You can use common security measures to mitigate the security issues. These include configuring routers to limit the connections from the virtual machine to only those computers you know need to have that access.

You can also limit the network connections to those on the physical machine, and not allow the virtual machine access to the physical network. Careful planning and implementation helps minimize security risks in having virtual NT machines on the network.

J.D. summed up his thoughts that virtualization can help reduce the spare part hunt fixing up older physical PCs, can help relieve that natural tension between plant engineering and the IT team, and can follow some standard security measures to reduce cyber-security risks.

These ideas are something to consider if you're faced with this situation.

Update: A couple of great comments have come in on this post in FriendFeed that I wanted to share.

A hot cutover post from several weeks ago featuring Emerson's Aaron Crews prompted a question from another Emerson project professional to Aaron. The question described an upcoming hot cutover project and asked Aaron to share any additional thoughts he had. For those unfamiliar with this jargon, a hot cutover is the process of converting to a modern process automation system while the process is still running.

In addition to sharing the detailed step-by-step process developed by The Automation Group (part of the Process Systems and Solutions organization in Emerson), Aaron shared these thoughts with his colleague:

Hot cutover execution is not necessarily too tough of a task as long as you know what you've got. The most important thing about a hot cutover is to have all of your information together and organized. You'll want to have survey information for all of your valves, including whether there are bypasses or hand jacks. If there are not bypasses on the valves, you'll want to review those cases with operations and the process manufacturer's engineers to ensure that they can lose that valve for a few minutes (maybe they can fill up a tank before cutting over the inlet valve, for instance.) If they can't lose valve function, there might be a workaround - you might have cases where the valve is normally at 100% and you can use a mechanical stop to keep it from failing closed, etc. Find an instrumentation expert to help you with any of these situations.

From a software standpoint, it is usually possible to put the I/O point in manual or to bypass logic that uses the point, but it is (as always) paramount to know all of your control references and complex loop functions on both the new and old DCS. Again, as long as you have all of your information you are ok-it's what you don't know that can get you in trouble.

The other major planning task is the cutover order. We typically cut over by graphic since the operator will have to be operating off of two DCSs at once. Within the graphic, we cut over indicators first (temperatures and pressures before flows, generally), then control loops then shutdown loops.

Overall, we just try and stay organized and keep all the documentation together. We generate reports at the end of each day that list the proposed points to be cut over the next day. These reports contain any field survey notes (that valve information from earlier), drawing numbers, wiring information and controls references on the old and new DCS for each point. This list is reviewed with operations and the specific order and specific operational concerns or strategies can be discussed during a daily meeting with operations, engineering, construction and safety leads.

We also track cutover progress very carefully to ensure that we are staying on schedule, and we keep a master list of loops as a cutover sign-off document.

The specific cutover procedure per point is typically pretty systematic, except in those problem cases that you will have defined.

I hope you see a few nuggets of wisdom you can use in planning a modernization project which includes a hot cutover at your facility. Also, I hope that I still get cc'ed on emails like these. One never knows when a blogger might add a little visibility to a good email!

I heard a great story last week from Emerson's Scott Ross, and member of the migration and modernization team. A process manufacturer is running a legacy RMV9000 system that has been in service for decades. The system is no longer sold as new. Service and refurbished assembly availability is performed on a best efforts basis. Repair services are still available for the latest version 5 RMV9000 components.

This process manufacturer had a situation where some of the controller and communication boards were exposed to a corrosive substance that damaged them. The plant's engineer thought he was completely out of luck given the age of the system. He put the call in to the local Emerson Process Management support organization and was surprised when he heard back from Scott with possible solutions.

Scott indicated the main MVCU controller processor board could still be repaired / refurbished and have the most recent EPROMs added. The cards were badly corroded, so Scott was able to track down some replacement cards to send as replacements when the boards arrived at the Emerson Process Management repair facility in Pittsburgh, Pennsylvania in the U.S. Scott also provided information and possible causes of the failures seen in the error codes on the controller main processor board.

The unit that manages the data highway communication and aggregates information to and from the controllers, the CCM, was also experiencing intermittent issues. The CCM's 540Mb hard disk drive was showing errors and these drives failure rates increase as they mature. Also, the 200Mb tape drive was faulty. Scott did not have any new drives available but did have refurbished units ready for this process manufacturer to procure.

For this manufacturer, the CCM's were setup in a simplex mode, so the entire unit could not be sent in to Emerson to be refurbished. They did have on-site spare units that could be returned to Emerson for repair/refurbishment and testing. Once the refurbished unit is received back from the Repair Center, the unit could be inserted into service. Scott made the suggestion that once the sent unit was refurbished that they should consider running the CCMs in a redundant mode to help with overall system availability.

In this case, the result is that this engineer has a path to get through these short-term issues, but realizes the case must be made for an overall modernization effort. I've highlighted in a prior post, Estimating Process Automation Benefits, an approach you might take for this justification effort.

This effort definitely beats the alternative of being completely out of luck.

It must be one of those weeks where a theme accidentally emerges. This week, it's discovering things to write about courtesy of Twitter. The subject for today's post comes from Aaron Crews, a principal control systems engineer with The Automation Group (TAG). TAG joined Emerson earlier this year.

Aaron's tweet alerted me that he's tackling a project with a hot cutover. For those unfamiliar with the term, it's converting over to a modern process automation system while the process is still running. It takes careful, detailed planning.

Aaron shared the cutover planning process with me on this DCS modernization project. The tasks include cutover scheduling, logistics planning, sequence planning, safety planning and cutover documentation.

Cutover scheduling requires all prerequisites be complete, including the control system configuration and installation, operator training, etc. The scheduling must take the overall operating conditions and plant maintenance activities into account.

The cutover logistics planning choreographs the space requirements and movements of the old and new equipment since both are in operation as the cutover is performed. Power, communications and other connections must be part of this planning since operator stations and I/O cabinets may be temporarily located during the transition process.

The cutover sequence planning looks at the order the process units will be converted from the old system to the new. Generally, a back-to-front order is used unless process conditions dictate a change. A key consideration in this phase is the switchover of the operators' graphics. During this cutover process they are operating on both the old and new operator consoles. The plan needs to make this as easy as possible to operate during this switchover period.

Cutover safety planning is critical. All of the established plant safety procedures must be followed. For U.S.-based hot cutover projects, a pre-startup safety review should be conducted in accordance with the Occupational Health and Safety Administration (OSHA) Process Safety Management (PSM) requirements. All team members must have completed the established plant safety-training program and must have the proper safety equipment.

The cutover documentation includes the collection and organization of all the required drawings and cutover reports. A process tracking system is created to ensure that all documentation is checked and that the cutover proceeds per the scheduled plan and that any required changes are noted.

For this particular project as with all hot cutover projects Aaron and his team identify the cutover challenges, associated risks and possible solutions--the earlier the better. This process begins with the initial field survey. The earlier these challenges can be identified, the better the planning and required solutions can be engineered in advance. In the detailed engineering phase, the best engineering solutions are determined for these challenges give the associated risk, cost and schedule considerations.

Aaron shared a few of the challenges on his current project, which I'll share in a future post. If you've planned or participated in a hot cutover, how does this compare with your process?

An oft-cited number when discussing the topic of distributed control system (DCS) modernization is $65 billion. This was the size of the installed base systems reaching their end of their useful lives from an ARC Advisory Group study conducted several years ago.

Emerson's Cody Long, a migration/modernization consultant, visited a Latin American sugar mill. This mill had two different, non-Emerson distributed control systems (DCS) near their end of lives. They could no longer grow or be expanded because the systems were no longer actively marketed. The mill was also experiencing field failures with this DCS hardware. Beyond the reliability issues, the mill staff wanted better boiler control by applying advanced process control to improve their existing control.

Cody began his visit to the mill with a site walkthrough to assess current operating conditions and to speak with operations and engineering staff about the challenges they faced. The mill staff was quite experienced in process control and advanced process control. The team understood what some of these tools could do in improving existing control performance.

He discussed options ranging from a complete "rip and replace" of the existing DCSs with a DeltaV system, to a connect solution which keeps some of the existing controller and I/O in place and provides new operator workstations and the ability to expand new areas with DeltaV hardware. This connect method also provides a way to switch over control to the new system in small increments over time.

From his visits over time with process manufacturers, Cody has developed some new tools to quickly analyze the contents and complexity of the existing DCS' database to help immediately formulate a migration strategy.

Project justifications based upon avoiding failures are often difficult to sell to plant management, so typically Cody or a member of the advanced automation team would work with the project team to identify economic benefits based on the operating performance. In this case, it would be through improved boiler control. These benefits would likely include reduced energy usage and fewer boiler trips. These trips impacted overall mill production, which impacts the revenue side of the justification analysis.

The mill staff had this project justification expertise internally, so they were able to get a modernization project approved. The analysis showed the return on investment from a full replacement of the existing DCSs with a DeltaV system justified the cost of the project. The mill would use the existing DCS parts as spares for another mill with similar systems.

Once this project is completed and business results quantified, a similar analysis will be done on the other plant to assess a total replacement or a connect solution.

Engineers being the problem solvers that they are, typically enjoy a project in full execution mode. Problems must be quickly confronted and solved to keep everything moving forward. As we've mentioned in earlier posts, the part they typically like least is the upfront justification to get the projects approved in the first place.

Emerson's Pete Sharpe, a principal consultant in the Advanced Automation Services organization, shared his thoughts on automation investment justification with the readers of Automation World magazine. The article, Strengthen Company, Minimize Risk, pointed to areas of opportunity for project justification.

Pete's guidance is to look at the economic buckets your efforts in automation can influence, which boils down to increasing profits and minimizing risk. Simply put:

To increase profits, "you must either increase revenues or lower costs," he emphasizes. Revenue is a bucket on the positive side of the formula that is affected by things like throughput, yields, recoveries or product price. That means "you have to shift production toward the more valuable products, or increase yield, reduce off-spec, product losses and downgrades of product," Sharpe states. Cost-lowering considerations could involve maintenance, labor, energy, utilities or raw materials, among other areas.

Pete cites an example of looking at quality. Poor quality can lead to customer rejection, off spec and rework. Providing better quality than is specified in the contract is called "quality giveaway". It likely means additional costs are being incurred without receiving additional price for this quality. This is particularly relevant to commodity markets such as gasoline and diesel. Other potential sources of justification are in intangible costs like safety and environmental compliance.

Minimizing risk is about reducing the probability that something bad will occur in the plant's operation. These projects focus on improving reliability, safety, environmental liability, and dealing with abnormal situations. Risk can be evaluated based on the frequency and severity of historical incidents. Then appropriate application of technology and programs designed to mitigate the highest risk areas by applying such things as predictive maintenance, operator training systems or abnormal situation prevention technologies.

The key is to look for how your project will affect the throughput, production costs and total production value on an on-going basis. Ultimately, the expected financial return of the project will determine whether the project goes forward or not. The article sums this up:

...the ultimate metric for justifying investments is ROI. He notes that it includes the time value of money, and calculation of returns based on expected future cash flows from the investment.

In most companies, the management team evaluates potential projects based on the expected return and the risk associated with the investment. The projects with the highest rate of return and lowest perceived risk are those that will likely be funded. In almost all cases, the project return must exceed the manufacturer's cost of capital, which varies depending on company. Pete notes that there are exceptions where a low return, discretionary project is approved. This could be a "stay-in-business" investment decision, which ultimately is about reducing overall business risk.

There may be reasons why you need to consider something beyond your automation system that has been running in your plant for years and years. It might be the need to more tightly control energy usage to reduce energy costs. It might be to improve quality and consistency to stay ahead of your competitors. It might even be that you are losing peoples' expertise to retirement to maintain this automation system.

The vernacular for this process varies--modernization, migration, or upgrade--but planning is an essential component. I caught up with Laurie Ben, who directs a team of modernization consultants in Emerson's Process Systems and Solutions business. They have expertise in Emerson systems and systems from other automation suppliers. They also have methods and tools to design a migration solution.

This migration process can range from a simple connection between systems all the way to a "rip and replace" project, depending on the business drivers prompting the change. Some plants have existing pneumatic and panel-mount controls. Downtime and reliability concerns often provide the justification required to transition these to the current digital communications-based systems.

These digital technologies provide ways to do hot cutovers to keep the process running while the automation is migrated from pneumatic to digital. An example of how this works is a Fisher Foundation fieldbus-based DVC6000f digital valve controller. Its pressure control functionality connects to a DeltaV system while also sending a pressure signal to the existing actuator or pneumatic positioner. Once these pressures are balanced within the system, control is transferred to the digital valve controller. During this phase, AMS Device Manager helps finalize the cutover by helping the team to communicate locally with the valve to monitor exactly what is happening during the process of mounting, adjusting, stroking, and calibrating the valve.

Laurie mentioned that operator consoles typically have the shortest life span of all the automation system components. It is often the first consideration for migration to a modern automation system. Newer operator workstations keep the look and feel of the operator graphics and faceplates and connect to the existing automation system. Over time, I/O and controller hardware and software can be migrated. The business drivers help dictate the pace of migration.

For instance, if energy cost reduction is the business driver, it may make sense to modernize the controller and I/O to get embedded advanced control capabilities. Units like lime kilns, fired heaters, boilers, etc. can be run more efficiently as a unit with model predicted control than as a collection of interdependent loops.

It really begins with your business drivers and developing a plan to move forward to modernize your automation. The hardest path is to justify it based on obsolescence since the calculation of ROI based on problem avoidance. The best path is to find cost reduction or efficiency-improving opportunities. These numbers are the basis for your financial justification calculations.

I know from my early days as a systems engineer, that power and grounding can sometimes cause vexing, intermittent issues. It's like chasing ghosts, which is an apropos analogy with Halloween occurring yesterday here in the Western world.

I caught up recently with Dan Jacobsmeyer, a specialist for Emerson's FlexConnect solutions. These solutions connect pre-engineered cables to the existing automation system termination panels and marshall the field signals to DeltaV I/O.

Dan shared a recent story where a process manufacturer was having some grounding issues with 1-5V analog I/O cards. Ground loop currents were causing a voltage offset, increasing the error on what the I/O card was reporting to the control system, versus its actual measured value. A solution discussed was to convert the I/O over to 4-20mA.

Dan pointed out that replacing the 1-5V cards with 4-20mA cards is a way to solve the immediate voltage offset problem. However, not fixing the ground issue may result in future problems that may be very difficult to diagnose.

Dan recommended the following course of action based upon DeltaV site preparation documentation.

A proper earth ground is critical. It safely conducts stray electrical current to earth for personal safety and good electrical noise control. Building steel must be part of a good earth ground system to obtain an equal-voltage-potential ground between the steel and the automation system ground networks.

Automation system faults are often the result of poor or faulty grounds. IEEE Standard 1100-1992, Recommended Practice for Power and Grounding Sensitive Electronic Equipment describes industry-accepted methods.

Isolation transformers should be used for power transfer from plant A/C service to the individual automation systems. A single point ground eliminates ground loops and provides excellent isolation between the automation systems and other plant systems.

Dan sums up his recommendations to use the isolation transformer between the A/C power source and automation system and have each system with a dedicated connection to a true-earth plant ground grid. Make sure no other systems share this connection point.

Paying close attention to these recommendations and not masking over them will save a lot of time chasing ghosts and delivery more reliable and accurate system operation.

John Dolenc, a principal consulting engineer in Emerson's advanced applied technology organization whom you may recall from earlier posts, presented Estimating Benefits from Process Automation at the Emerson Exchange.

John began by discussing what types of activities are performed during a process automation modernization feasibility study. Project justification is one of the most difficult steps to complete.

Start with looking at the plant as a financial asset. Perform a financial balance across a process unit in the same manner as performing a material and energy balance. Capital enters a process unit in the raw materials, net utilities, operating expenses, and maintenance expenses. Value is added within the process unit. Capital flows out in primary and secondary products as well as the waste streams. After identifying the financial streams, identify areas that need improvement and determine whether better process automation can help.

Capital projects are justified by either increasing the revenue of the plant or by reducing the costs to manufacture the products. Revenue may be increased through increasing production capacity (if additional product can be sold), and increasing the yield of more profitable products while reducing lower grade product and waste streams. Look for capacity increases through being able to run closer to constraints by reduced process variability. Can you reduce unproductive time such as grade changeovers, off-spec blending and downtime for cleaning through better process control? Can you reduce cycle time for batch processes by better control of reaction conditions and eliminating wait time for operator actions?

Another method of increasing plant profitability is by working with the marketing and sales force to understand what your customers' value about your product. Can price be slightly increased if the product consistently meets specification and delivery schedules and emergency production is provided when needed?

Reducing costs are a big area of value creation. Some areas to look for cost reduction opportunities include increasing feedstock yield, reducing energy consumption, better use of direct and indirect labor, and limiting off-spec material, abnormal events, demurrage and maintenance.

These are the opportunities for automation project financial justification. So what are some benefit estimation methods you can employ? John mentioned the best operator method, data reduction method, and percent limit violation as three methods of estimation. The best operator method considers historically identifying key performance indicators at their optimum point. What are the "best operators" doing to make this happen? The key is to look for how to automate these best practices. Your project justification is the difference between operating at the "best operator" condition versus the historical average condition.

John went through an example of looking at batch cycle times, throwing out outliers caused by extenuating circumstances, and digging into the process automation that can be done to move from the average cycle time to the best cycle time.

He discussed the other methods, data reduction method and limit violation methods. With the latter, conceptually the results of the automation project are to reduce overall variability. This means you can run closer to your process or specification limit. This means operating set points can be set closer to these limits, thus reducing costs such as product giveaways and excess energy usage. The presentation includes some of the statistical methods to build your financial justification. John offered a rule of thumb that you would reduce standard deviation by at least 50% by improving overall control performance.

John summarized his presentation with these key points:

• Justification is difficult, but necessary to receive approval for automation projects
• Historical data collection is vital
• Build a base case
• Some "prediction" of results is necessary

As reported in the DeltaV News RSS feed recently, Automation World magazine's C. Kenna Amos wrote an article, Getting Projects Approved. I know from my days as a systems engineer, that financially justifying a capital project was not nearly as fun as executing the project. Most engineers enjoy the action of seeing their work come to life more than convincing others to approve the capital to get the project going. They also are not typically versed in the language of financial analysis.

The article captures the wisdom of Emerson's Doug White, a vice president of advanced automation services. Doug and his team often assist process manufacturers in ways to help quantify return on investment for automation and advanced automation projects.

In the article, Doug notes:

The project has to be very attractive to be funded, because it will compete with others. The project has to show a clear and compelling return on investment.

Easy enough, but the trick is how to do this. Doug recommends that engineers work with the financial group to understand their selection criteria for capital projects. Basics for most projects include cash outflow analysis and when the return on investment begins. This is the basis for the payback calculations. Also, the capital proposal should include key non-quantifiable benefits often found in health, safety, and environmental (HSE) considerations.

The closer you can tie your proposal to key organizational initiatives, the more the proposal will be noticed more than others will. When it comes to presenting your proposal:

Begin by first defining the problem, then telling them why your project is important and giving reasons why it needs to be done, he emphasizes. Then--and only then--go into financials, beginning with the most likely scenario.

Doug has captured much of his experience in a whitepaper, Calculating ROI for Automation Projects. It comprehensively goes through the components of return on invested capital and how to calculate each component. Give this whitepaper a thorough review and you will be better prepared to have that conversation with the financial group.

For those of you going to the Emerson Exchange next week in Dallas, make sure to catch Doug's short course, How To Find The Economics For Process Automation Investments that will be held Tuesday at 2:15pm and repeated Wednesday at 8am. Here's the abstract for this presentation:

This session presents realistic approaches to automation project economic analysis and justification. The viewpoint is that of the business financial analyst. Specific areas where automation affects the business results are identified and quantified.

The ever-increasing global demand for energy requires tremendous daily global movements of crude and refined products. Offsite operations play a vital role in the hydrocarbon supply chain. These offsite operations provide the receiving, shipping and storage facilities for handling bulk liquid or gas products. These sites typically include tank farms, blenders and terminals for handling truck, rail, and marine transport.

Tank farms and terminals are found at various stages along the production process from oil & gas collection terminals to refinery terminals and depots, to primary depots where refined products are loaded and transported to your local gas station.

I caught up with Patrick Truesdale, a senior consultant, in Emerson's advanced automation services team. Patrick will be co-presenting with Emerson's Gerrie Benjert at the upcoming Emerson Exchange. Their topic is developing a business case for tank farm and terminal automation.

The biggest business challenge Patrick finds from his work with offsite operators is the lack of spare capacity throughout the global distribution system. Capacity utilization is at a maximum and assets in many of the established markets like North America and Western Europe are aging. This increases the chances for unplanned shutdowns. These facilities often lack flexibility to deal with changes in market demand.

Other challenges include increasing safety and environmental requirements and increasing compliance reporting. In addition, new regulatory mandates for ultra low sulphur fuels, biofuels and other additives increase the number of products to manage through the distribution chain.

Overcoming these challenges is the basis for the business case that Patrick helps offsite operators build. If the case for improvement justifies capital investment, an important step will be reviewing the key components in an offsite automation system. These components include automatic tank gauging and inventory management, goods movement automation and control, blend control and optimization, and terminal management systems.

The more these components are integrated, the better the efficiency of the offsite operations can be. Automated data collection, correlation, and reporting help streamline the regulatory reporting challenge and provide a better data to make process improvements. In addition, custody transfer of the liquids and gas can be more accurately measured, accounted, and billed.

In his presentation, Patrick and Gerrie will discuss some of the quantified benefits that some offsite operations have been able to achieve. It is important to establish a continuous improvement loop to collect data before and after these offsite automation system components are added or modernized. This is so the data can be analyzed and used to generate additional projects to further integrate and streamline operations, based upon quantified results.

Charlie Masi, over at Control Engineering magazine's Ask Charlie blog, had an interesting post recently. Entitled, When is a piece of equipment or system obsolete?, he writes:

After spending several hundreds of thousands, or just as often millions, of dollars building up infrastructure based on earlier-technology equipment, dumping it as "old and yukky" just because something "new and improved" has come out is fiscally irresponsible. Not only would you have to write off the old equipment (which the accounting department would rather depreciate over another two or three years--or more) and lay out the cost of the "new and improved" stuff, everyone in the plant would have to drop what they're doing (hopefully, productive work) to take training classes on the "new and improved" stuff (which is definitely unproductive).

It's interesting because it ties in with a draft of a paper that Emerson's John Dolenc is writing about planning automation upgrades. You may recall John, an advanced automation services consultant, from an earlier post. One of his key points is that economic justifications based on obsolescence are a difficult route to take, unless the obsolete equipment is causing unplanned downtime. One problem is that it's hard to quantify the improvement. Another is that the new equipment may be replaced "in-kind" where no benefit is received from its improved technologies--it simply mimics what the old equipment did.

He stresses a path which begins with the business objectives like increasing production capacity, reducing manufacturing costs, or increasing process flexibility to more quickly respond to customer demand. From these business objectives, an audit of the process unit should be done to identify areas for financial improvement.

This audit feeds a conceptual engineering study to develop the process automation modernization plan. John has developed these studies working with members from operations, maintenance, plant engineering and management, even sales and marketing to better understand the flexibility the plant's customers require.

This study provides the business case whether to pursue the capital investment for modernization or to maintain the status quo. It may also identify quick opportunities for improvement with the existing equipment.

The draft also describes his methodology for moving forward with the modernization plan, which I'll save for a future post.