Loading and Maintaining Control System PCs–Automation or IT Department?

ControlGlobal.com had an interesting article, Firewall Needed Between AT and IT? Should the Business IT Group or the Controls Group Load the DeltaV Machines? AT is the automation technology team and IT is the information technology team. The question posed was:

Q: We are having an internal argument as to who loads the DeltaV machines when it comes to the Windows software. My question is should we allow the “Business IT” group to do it, or is this the “Controls” group’s responsibility? Are there any regulations about this?

Responses ranged between “it depends” and “keep IT away”. I thought it was a great question so I extended it out to the DeltaV Facebook and DeltaV LinkedIn groups. I even tweeted it out from the DeltaV Twitter account to the folks who follow it.

Emerson’s Bob Huba had the direct answer to the question:

@digitalbond @waltboyes actually DeltaV uses O/S images so anybody, even IT, can easily & correctly load the OS on DeltaV

Most folks looked beyond the initial software installation to the ongoing support and maintenance of the PC workstations used in the DeltaV system. Here’s a sampling of the wisdom offered from the DeltaV communities:

As a former Controls Engineer who moved to IT I can resoundingly say “It Depends.” It depends on the experience of the people in IT who manage the automation systems. We have an IT group I lead that deals with automation systems. We put DeltaV and the PI Historian in the data center as well as other SCADA and BAS systems like Wonderware and Johnson Controls. If you have an IT department that has the experience in manufacturing systems then Yes. If not you better educate them on then on differences in automation systems or you will be in trouble. Never let automatic patches be installed and ISOLATE your automation from your business networks.

With the new scripted installation disks from Emerson, I would let anyone install it. My only fear of letting IT install it, is they would feel the need to put on update patches as mentioned above, or corporate port blocking, vpn ipsec files, remote monitoring, remote re-updating, etc.

On our site the control system and corporate systems are completely divorced from each other with the understanding that AT only touches AT machines and IT looks after corporate machines. AT are responsible for buying and installing all AT machines and any virus, machine or DeltaV updates. The biggest problems we see are already mentioned above like automatic updates pushed out by IT have not necessarily been tested by Emerson for use on DeltaV etc.

Business IT group involved in the process control side of the plant…wow scary, I can’t believe it’s even up for discussion. I had an IT person years ago go on a tirade how UDP is far inferior to TCP, and right afterwards, went back to playing WOW. Lesson: Mall Security <> Police Officers. DeltaV := Control Engineers Only

I think the point that many are missing in the IT world is that while the DeltaV servers/workstation/switches may have the same form as one of their PC’s, it really isn’t a PC anymore. It’s a piece of control system hardware.

I have worked on both sides (AT and IT) and the best solution is to have a combination of both. Most AT people don’t know much about Microsoft Server Infrastructure (Domains), disaster recovery, etc..and people from IT dont know what is an AI, AO, PID, etc.

The net of it is that the requirements of PCs used in a control system (where availability along with security is paramount) are different from PCs on the office LAN (where security is paramount). Whoever is doing the ongoing maintenance and support of the control system PCs must be well versed in the automation supplier’s maintenance and support practices and fully understand the repercussions in making changes to a system controlling the production process. This expertise more often than not resides on the AT team but can reside on the IT team as some have indicated.

Thanks for everyone’s thoughts and please share any additional thoughts you may have in the comments below.

MP3 | iTunes

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

4 comments

  1. Dale Peterson says:

    Jim,

    We have seen all combinations work and not work. IT doing the work, ops doing the work, a hybrid team, even outsourced. 

    The bigger problem is when ICS have IT technologies where there are not qualified people available to support the IT systems and applications that the modern ICS requires. A vendor puts in Active Directory, but then no one in ops has the domain administrator skills. A firewall or switch is put in, but no one knows how to troubleshoot or configure it. 

    Our recommendation is always to focus on the skill set needed to support the IT components you have in your SCADA or DCS first. Then decide where you will get that skill set. Oftentimes we have seen operations actually reach out for help once they understand what skill set is needed to have a robust IT infrastructure that provides the reliability they demand. 

    Dale Peterson
    http://www.digitalbond.com

  2. Hi Jim,
     
    I agree on sample wisdom above and particularly summarized by this “Mall Security Police Officer,” which side is which is up to anyone’s point of view. – One thing is for sure, there should be a thick line of separation between each other’s role. As such, IT can of course become information resources but should be off-hand to AT critical equipment (mostly OS, ES, APPs servers in the control network).
     
    We had in one case during FAT in a staging area, asked IT personnel to fix the viruses in the control system and the AT people ends up rebuilding (clean install) all worstations since the process automation software did not start anymore. Imagine that happening in a production system of a critical plant operation, some other worst cases & scenarios other than this can happen.
     
    At the mininmum, for the PCs / Servers involved in the plant control network, the people from AT should be responsible. Basically because they are informed about the applicable patches (Microsoft-wise and/or DCS-wise) which have been tested by technologies and in such cases, AT should be equipped with the correct procedures, KBAs, lessons-learned and skill sets.

    • Danny, Thanks for sharing! The common thread seems to be that skills are a prerequisite. On-the-job learning of control system asset maintenance can lead bad/dangerous results.

Leave a Reply