July 2007 Archives

A colleague pointed me to an article, Timeline of a refinery pump failure and how it could have been prevented, on the Belgium-based EngineeringNet.be website. The story was about a South American refinery that had a high-speed centrifugal pump fail catastrophically resulting in production losses and large repair costs. Todd Reeves is in Emerson's Machinery Health Management team, part of the Asset Optimization organization.

What happened was an inboard bearing lost lubrication, overheated and finally seized up. The unfortunate part of the story is an automated motor-pump train monitor and advanced vibration analysis system had been installed four months earlier and was working properly.

This monitoring equipment included the CSI 9210 Machinery Health Transmitter connected to the automation system via Foundation fieldbus. This equipment did its job communicating advisory alarms it began to detect problems in the lubrication system.

These alerts went unheeded until they became maintenance alerts and ultimately failure alerts. Todd wrote that the health curve of the pump deteriorated rapidly in the final ten minutes before failure.

Why? The equipment did its job and dutifully reported the problem. The issue turned out to be more of overall unit tuning and alarm management issues. These alerts had been lost among other alarms coming in.

Working as a team, the refinery and local asset optimization experts reviewed the overall alarm strategy and identified opportunities to reduce the alarms and alerts coming in to the operators.

Specifically for the pumps, a best practice was established to add additional temperature measurements on the pump. Training was established to clarify how these alerts would be transitioned between the operators and maintenance staff. Clarifying this process is important when working with predictive diagnostics. At the time, it is not yet an actual problem--but like this centrifugal pump example--will fail if not addressed.

Pharmaceutical Technology magazine published an interesting article by Emerson's Bob Lenich and Christie Deitz. The article, A Look at 30 Years of Change in Pharmaceutical Automation, recounts the changes affecting Life Science manufacturers from the late 1970s though today. I joined the world of process automation in the early 80s as a summer systems engineering intern in offshore oil and gas production and this article brought back some memories of the amazing changes we've seen.

I'll highlight some items from the article to see if it generates any nostalgic thoughts for you.

Although the distributed control system came along in the mid-70s, Bob and Christie note that most life science companies used pneumatic and single-loop electronic controllers. Data was collected manually or with circular and strip charts.

With growing U.S. Food and Drug Administration (FDA) regulations through the late 70s and early 80s, the DCS began to be seen by life science manufacturers as a tool to help comply.

Batch-based automation systems, the first one being the PROVOX system, came out in the early-to-mid 80s to help with sequencing, failure handling, parallel unit operations, and the creation of recipes.

Just a few years before I recall a little collaborative effort between IBM and Microsoft being introduced to the market (wow a 4.77MHz CPU!) This would have some impact in our industry in the following decade as commercially available technologies (COTS) were incorporated.

Toward the later part of the 80s and into the 90s, standards began to play a larger role. ISA-88 (S88), a batch automation standard was important to life science manufacturers. The digital busses including Foundation fieldbus were developing, and Microsoft operating systems began to make their appearance in systems like the DeltaV system. For communications, the OLE for Process Control (OPC) standard became the way to connect Microsoft-based clients and servers--a big improvement over earlier generation DDE communications technologies.

Automation systems became increasingly modular with class-based configurations. These technologies would help the trend toward more modular construction techniques that brought production on-line quicker compared with prior construction and engineering methods.

Regulations continued to advance to try to address concerns around system, production and data management through the balance of the 90s. Efforts began on the ISA-95 (S95) standard to better define the integration of enterprise and control systems.

These regulations had a positive impact in building competency around data security, record security, lot tracking, and overall batch management. The downside was that it placed the focus of life science manufacturers on meeting regulations rather than continually improving their manufacturing operations compared with other industries.

The FDA's Process Analytical Technologies (PAT) initiative addressed this by changing the focus from meeting regulations to improving operation. The FDA's cGMPs for the 21^st Century added in using a risk-based approach to these improvements. As part of this initiative, they encouraged the use of innovative technologies. We've addressed a number of these innovations with respect to PAT in earlier posts.

Bob and Christie closed the article with a note of how flexibility and the integration of automation with the business-level systems is becoming increasingly important as life science manufacturers move from organic-based synthesis to biologics to continue to develop vaccines and medicines to address our health needs.

Update: Thank you Eric for pointing out the error of my ways! The link to the OPC Foundation has been corrected.

A great question came in on the Operating Fired Heaters More Efficiently and Reliably blog post:

Jim I work with natural draft heaters on a daily basis and have initiated several efficiency tests with improved burner internals. I am looking for an opportunity to optimize dual firetube treater by first off improving the combustion efficiency to 80% in each tube and then staggering the temperature controls so that one tube runs 90 to 100% of the time and the other tube only fire during high load requirements.

I sent the comment around our advanced automation consultants for any comments that they might have and I received a great reply from Lou Heavner whom you may recall from earlier posts. Lou describes how to approach optimizing these heaters:

Heater efficiency is calculated using heat loss or input/output method. Input/Output method is difficult because you have to account for lags and delays between fuel firing rate changes and the measurement of process heat absorption changes and in the specific case where there is incomplete phase change on the process side (e.g. partial vaporization) you cannot easily solve with reasonable instrumentation. The heat loss method measures heat loss in the flue gas and assumes any other losses are negligible and constant. If not, they need to be measured and added as well.

Heat loss requires knowledge of the supply air (and fuel) temperatures and the flue gas exhaust temperature as well as the composition of the fuel and flue gas, just like with a boiler. In perfect combustion, there would be no unburned fuel in the flue gas and no sensible heat losses. But due to practical considerations, there are sensible heat losses and to calculate them, you need to know the delta T between the exhaust and ambient and how much excess oxygen remains in the exhaust. Efficiency calculations made using this technique can be pretty accurate in a natural draft heater, but if there is air leakage after the combustion zone, tramp air will show up as lower efficiency due to increased O2. And there is usually an optimum cost operation where the trade-off between sensible heat losses and unburned fuel losses require some level of unburned or incompletely burned fuel leaving in the flue.

When you are ready to control, the goal is to minimize excess O2 while not allowing excessive fuel to go unconsumed. CO analyzers are often used to detect incompletely burned fuel and the goal is usually to keep it below 150 ppm or some lower target. O2 is controlled to stay as low as possible without exceeding the CO limit, which is usually 2% O2 or less for the fluegas.

You can do this with simple feedback control, but feed forward control can help do better. Information on fuel quality, if it varies, and process side temperatures and flows (the heater load demand) can be used to adjust the fuel and air for combustion to meet the heating demand at maximum efficiency. Fuel and air cross limits are often used to maintain fuel and air ratio without getting into a fuel rich condition in the firebox during load changes. But airflow is usually difficult to measure. Therefore, it is often inferred from damper position.

When evaluating an application, we would want to know what instrumentation already exists and what the process variability looks like. What efficiency are they currently obtaining? Then we would look at the control valves and any other contributors to variability to see if they warrant repair or replacement. We would similarly evaluate the instrumentation and analyzers to see if they need anything there.

Then we could evaluate the control strategy and performance and recommend reconfiguration or tuning as appropriate, which may include advanced process control (APC). The person evaluating the controls would have to weigh the cost against the improvement from better loop tuning, valve repair/replacement, CO analyzer, etc. to come up with the best solution. Dampers are often the weak link in fine control of a natural draft heater.

As my colleague Doug Simmers in Emerson's Rosemount Analytical business noted, "The commenter is probably correct with the strategy to fire one heater full out, and bring the second unit on only when needed. Running at full fire develops the best turbulence for fuel/air mixing, and the excess O2 can be kept lower." This is a load allocation problem when two heaters are firing simultaneously. If we can model heater efficiency for each heater as a function of load, then we could optimize the load allocation across both heaters when both must be fired. Actual testing would identify the models, uncover the best strategy, and verify or disprove this assumption."

He may also be interested in the efficiency calculator, developed by Doug's team.

Join the conversation and add a comment if you have experience to share.

If you've been following this blog for a while, you know that from time to time I extol the virtues of Really Simple Syndication (RSS). Examples are here, here, here, and here.

I thought I'd pass along one of my RSS tricks around persistent RSS searches. First, if you're not already using RSS, the latest versions of the web browsers all support it natively. Also, web-based readers like Google Reader and Bloglines are easy to use. Client-based ones you install on your PC include Outlook 2007 email, and applications like we recommended a while back with our RSS Starter Kit.

Personally, I use Outlook 2007 for my most frequently read feeds, and Google Reader and Bloglines for the occasional reading when I have some time.

My trick is doing searches on "Jim Cahill" emerson with Google's blog search (or Google News or Technorati) and it helps show if anything has been written about me (good, bad, or indifferent.) I include the company name because there are a few other prominent Jim Cahill's out there that I must compete with for search engine visibility supremacy.

These searches are good one-time things, but what if I always want to know? What if I want my RSS Reader to let me know if somebody has written something new? That's where persistent RSS search come in.

In the Google example, there is a left side area where I can subscribe to this search (or any other ones of interest) and add these to my RSS reader. The Google searches also allow emails for those of us who feel they are not already getting enough email (not too likely!)

I pass this along this because last Friday my persistent RSS searches dutifully informed me that Forrester Research senior analyst, Peter Kim wrote a blog post entitled, Introducing: Top Marketer Blogs (alpha). Lo and behold, our Emerson Process Experts blog was on the list. My first reaction was, "cool!" Then I thought how Forrester researches the broad space of technology well beyond our world of process automation so my new reaction was, "way cool!" A week later, I'm still sticking to that reaction.

So there you have it, start your day with that check of your persistent RSS searches and you could be pleasantly surprised.

Update: In the comments, Pete had a great suggestion for you fellow bloggers. Use a search of your blogs URL with Technorati's "blog reactions", For this blog,the persistent RSS feed for this reaction is: feeds.technorati.com/search/www.emersonprocessxperts.com.

Process manufacturers continue to seek ways to improve their energy efficiency, due to the high cost of energy. Corrosion and solids deposition in boilers, condensers, and steam turbines reduce the efficiency of this equipment and increase energy usage. This can also lead to unscheduled downtime if the conditions persist long enough to cause equipment failure.

One important way to minimize corrosion and the formation of solid particles is to have ongoing, accurate and reliable pH control in the boiler water, boiler feedwater and steam condensate, and main steam (carryover.)

The challenge is that these applications are often very low in conductivity. This is a challenge for continuous pH measurement due to the unavoidable formation of liquid junction potentials in the reference sensor. These cause offsets and instability in the pH measurement.

Emerson's Brian LaBelle, a power industry manager for Rosemount Analytical liquid analytical devices, explained these junction potentials are caused by spontaneous migration of ions from more concentrated to more dilute solution within a pH sensor electrode. What happens is a charge separation occurs among the various ions present. (At the word "ion", my mind raced back to those repressed memories of college chemistry lectures...)

Sometimes a severe junction potential occurs when there is an imbalance of negatively and positively charged ions across the liquid junction found in the basic reference electrode. The lower the porosity of the junction, the greater is the charge separation across this junction.

Sounds like we've gone a long way from the original problem of keep the equipment from corroding and being gummed up with solid particles.

Brian brought me to the solution by explaining that the technology team came up with the solution of replacing the diffusion junction with an open capillary (that's a hole for most of us.) Actually, this is not new or innovative, but what is innovative is that precise, laser drilling on a micro-scale of tens of microns is far more precise than what can be achieved with a twisting, mechanical bit. To minimize the junction potentials and provide more accurate measurement, the optimum capillary is laser-drilled at 25 microns in diameter. This capillary is also tapered outward to the outlet filter to help avoid clogging.

As we depart the micro world of ions and laser holes and return to our world of boilers, condensers, and steam turbines, the pH measurement with the Rosemount Analytical 3200HP pH sensor provides more accurate and reliable continuous measurement to ward off corrosion and solids formation. This means more reliable, efficient operations for this energy-consuming equipment.

I came across an email that the ISA Honors & Awards Committee has selected the paper, Improving PID Control with Unreliable Communications, for its excellence in documentation award. Emerson's Deji Chen, Mark Nixon, Terry Blevins, Willy Wojsznis and the University of Texas, Department of Computer Sciences' Jianping Song and Aloysius K. Mok wrote the paper.

The paper examines PID control in a wireless network where intermittent loss of communications is likely to happen. It identifies the poor dynamic response of standard PID algorithms in this loss of communications scenario. The team proposed an enhanced PID algorithm to improve dynamic response under these conditions.

Terry Blevins summarized the paper well in an earlier post on the Modeling and Control blog. The post, PID Modifications for Unreliable Communications describes the situation:

One of the technical challenges is that the 2.4 GHz spectrum defined by IEEE 802.15.4 is also used by Wi-Fi and Bluetooth devices. Also, some electrical devices found in industry generate noise in this frequency band. Thus, at times it is expected that a transmission will be corrupted. To help minimize the impact of these other devices on communications, the Time Synchronized Mesh Protocol (TSMP) selected for wireless HART uses frequency hopping. Even so, at times it is expected that multiple transmissions of a measurement used in control or multiple communications of control actions to an actuator may be lost.

Terry describes how the loss of communications can cause the PID loop to continue executing and wind up due to the reset action. This reset action can be disruptive to the control of the loop. And, if the derivative (the D in PID) action is used, the loss and resumption of the control measurement signal can cause a spike, again bumping the control of the loop.

The Emerson and UT technologists worked through a solution to minimize the impact of this loss of communications. Terry sums up the change:

However, by modifying the reset and derivative calculation to account for the time since the last measurement update, then it is possible to minimize the impact of loosing multiple measurement transmissions.

If you want to look at the math behind this innovation, check out the overview presentation, PID for Unreliable Communications, given at ISA 2006.

Congratulations to the team for their contribution to furthering the advancement of wireless technologies in process automation!

Here is another installment in our continuing series of screencasts showing the intersection of Foundation fieldbus (FF) digital communications with automation systems. The intent of these screencasts is to demonstrate visually how the information in these smart field devices interacts with the automation system to help improve the process.

Emerson's Rune Reppenhagen shows a control loop with a Coriolis flowmeter, a digital valve controller, and a connection to a redundant pair of Foundation fieldbus H1 cards. Rune describes the control strategy where the analog input runs in the Micro Motion transmitter, the PID control block and analog output block run in the Fieldvue DVC6000 digital valve controller and fieldbus segments connects to the pair of DeltaV H1 cards.

In this 3:21 screencast, Rune shows how control around the loop is maintained by running even in the event of loss of both H1 cards. As you might expect, the information is no longer transmitter to the operator, but the loop will continue to operate. Operators can monitor the loop locally at the devices with their local indicators until the communications are reestablished.

Different applications and operating philosophies may prompt where you might want to locate your control strategies--in the automation system controller or in Foundation fieldbus devices. John Rezabek, a contributing editor for Control magazine, implemented this approach seven years ago as he describes in his article, Not jazzed about fieldbus? Try it. He describes the additional benefit of mode-shedding to manual when the process variable (PV) of the PID is bad or uncertain in the devices. John writes:

While it's a diligent piece of work, this user-coded mode shedding is utterly unnecessary in fieldbus--it's already hard-coded into the blockware and happens automatically. In the same way, the "actual" mode of a PID block sheds to manual when its PV status is bad or uncertain, holding the last output computed before the input's signal status changed.

For bad or uncertain transmitter information, John writes:

Bad or uncertain PV status will cause appropriate mode-shedding in the same scan (macro cycle) in which the condition is detected, so no "new" valve output is passed to the AO block; it dutifully holds last value.

All this happens by interconnecting the FF signals via your programming interface. No additional code or external interlocks are necessary. It's built-in, out-of-the-box and standard in certified FF devices that have implemented PID.

The bottom line from the screencast and what John writes is that Foundation fieldbus provides a lot of robustness in control in addition to the digital diagnostics it delivers.

One of the innovations taking hold in the Information Technology (IT) community is services oriented architecture (SOA). As stated by the SOA entry in Wikipedia:

There is no widely-agreed upon definition of service-oriented architecture other than its literal translation that it is an architecture that relies on service-orientation as its fundamental design principle. Service-orientation describes an architecture that uses loosely coupled services to support the requirements of business processes and users. Resources on a network in a SOA environment are made available as independent services that can be accessed without knowledge of their underlying platform implementation.

To understand better the state and needs of the adoption of SOA in process manufacturing, the ARC Advisory Group is conducting a short 25-question survey, SOA for Manufacturers and Owner/Operators. The survey:

is designed for, and should be taken by, end users, owner/operators, etc. in process and discrete industries - as well as those with similar needs in other industries, such as utilities - with knowledge of their company's interest, goals, and experience with Web services and SOA.

As with ARC's other surveys, your participation earns a free copy of the results when they become available. This can help your organization better learn what other manufacturers are doing and possibly help advance your efforts.

I bring all this to your attention because I am also keenly interested in the problems you see SOA addressing and the business results you are seeking to achieve. It can have impact all the way to your process instrumentation and devices in your manufacturing processes.

If your organization is applying or investigating the application of SOA, consider investing a few minutes of your time taking the survey.

I was catching up on my RSS feeds over our middle of the week U.S. Independence Day holiday. My RSS search feed on the IEC 61511 global safety standard (ISA 84.01 in the U.S.) turned up this press release on a DVD set released by the Safety Users Group. They describe the purpose of this production:

In the style of a documentary, this unique DVD will provide you with expert points of view, as well as specific concepts, definitions, experiences, examples, analysis and results from 11 world-renowned professionals in the safety industry. These experts are designers, manufacturing leaders, engineering companies, integrators, standards members, professors, legal council and TÜV certified Functional Safety experts.

Emerson's Thomas Steiner is one of the safety experts interviewed. He is one of a large number of Emerson certified functional safety experts (CFSE) and certified functional safety professionals (CFSP). The mission of this certification process as stated on the CFSE.org site:

The CFSE (Certified Functional Safety Expert) concept was originally developed by TÜV and exida with the support of other international safety experts to ensure that personnel performing SIS lifecycle activities are competent as required by the IEC 61508, 61511, and 62061 standards.

Thomas discusses some of the basic terminology from the standards such as safe failure fractions (SFF), safety requirement specifications (SRS), and safety integrity levels (SIL) in a very understandable way. He describes how it applies to process manufacturers in applying this standard. You can see his entire interview (16:54) on the EasyDeltaV.com web site.

Overall, there is quite a bit of safety expertise provided by the 11 participants on this two-DVD set. The cost is $115 (USD) and you can get a preview by viewing this trailer of the type of information presented.

If your responsibilities include the IEC 61511 safety lifecycle and you need a good primer or refresher from knowledgeable safety experts, consider this DVD set as one of your learning resources.

Standards play an important role in fostering technological progress--both in the willingness of consumers to adopt the technologies and suppliers in developing products to meet the standards.

In our world of process automation, standards have continued to advance from base-level digital communications protocols to higher-level data communications standards for process manufacturers. The ISA-95 (S95) or IEC/ISO 62264 family of standards as they are globally known are an example of a set of data standards for the interface between enterprise planning systems and automation systems.

I had a chance to get a preview of a whitepaper that Emerson's Shenling Yang is developing around S95 and the XML-based implementation of this standard called Business To Manufacturing Markup Language (B2MML). You may recall Shenling from an earlier post on project timelines. She is now a data integration specialist in the Life Sciences industry center.

As stated in an ISA press release this past January on B2MML improvements:

B2MML was developed by the WBF's XML Working Group to provide manufacturing companies with a freely available XML Schema implementation of the ISA-95 Enterprise - Control System Integration Standard.

You can get a sense for just how detailed and comprehensive these standards are by viewing some of the schema documents available on the World Batch Forum's B2MML web page. Beyond the common schema organized around the S95 data model, other schemas exist for equipment, extensions, maintenance, materials, personnel, process segments, product definitions, production capabilities, production performance, and production schedules. Warning, these schema documents are not light reading!

On projects requiring workflow improvements and/or paperless operations, Shenling and the team follow B2MML data definitions to be consistent with the S95 standard. Because leading enterprise resource planning (ERP) systems like SAP support B2MML, Shenling finds that it simplifies connectivity and reduces the overall engineering effort for integration between the ERP and manufacturing execution systems like Compliance Suite. Ongoing maintenance is also reduced since the information exchanged between applications follows well-defined data definitions.

An example is an order coming down from SAP in an XML-formatted document complying with the B2MML Production Performance schema. The project team used transaction templates, along with the Compliance Suite support component and the process order XML from SAP to generate the actual transaction documents to be passed from the ERP to Compliance Suite. The automated parts are handled by the DeltaV Batch system and other parts of the process like materials management, laboratory information, and proof of personnel training are sent to their respective workflow processes.

The results of these workflows and batch data from the automation system are consolidated in an electronic batch record, which is a critical piece in reducing the overall cycle time on the way to releasing the product for sale.

Update: Gary Mintchell reports on his Feed Forward blog today that the World Batch Forum has announced version 4 of the B2MML standard and some of the additions to this standard. Here's the announcement from the WBF.