ASHRAE Journal:
ASHRAE Journal presents.
Michael Galler:
All of cybersecurity is like a moving target and nothing is static with it. The threats are always evolving and the methods to respond to them are always evolving. There's little chance of getting one static document and said, "This is the solution." Because that's going to work for about five seconds for one building.
ASHRAE Journal:
Episode 13. David Branson, Anto Budiardjo and Michael Galler examine various aspects of cybersecurity within HVAC&R including hazards and the origins of threats. They also discuss areas to be cognizant of to prevent breaches and share some of the solutions that the IT industry has successfully implemented, which could also serve HVAC markets, including the importance of managing networks.
David Branson:
I'm Dave Branson, I'm a president of Compliance Services Group, an engineering and architectural consulting firm in West Texas. And I am the vice chair of the cybersecurity MTG for ASHRAE. And I've been dealing with cybersecurity with my clients and predominantly federal government and institutional clients. So I have an interest from that perspective.
Anto Budiardjo:
Hi, I am Anto Budiardjo. I have been in and around building systems for about 30 years. My focus has been really in technologies to integrate different building systems with each other. So that's really in the IT realm of things. The last few years I've been working on a project called Secured by Cimetrics which is all around BACnet and around anticipating a future BACnet that is much more manageable and much more secure and interoperable. And so I've been very deep into the cybersecurity subject for a while.
Michael Galler:
I'm Mike Galler, I'm an engineer at the National Institute of Standards and Technology in Gaithersburg, Maryland, and I'm chair of the Cybersecurity MTG and chair of the previous cybersecurity subcommittee that the MTG was formed from. So the history of the Cybersecurity MTG started with the cybersecurity subcommittee that was formed in TC 1.5 then had its first meeting at the 2015 annual conference in Atlanta. So I formed the committee because I saw the need for some type of activity in the cybersecurity area and at the time there wasn't any large scale move for it. And for example, before we formed the subcommittee, there were no seminars or other presentations at any annual conferences about cybersecurity. Since we formed that subcommittee, there were seminars at almost every annual and winter meeting and there are two at the current meeting that look particularly interesting. I hope that the members will attend those both on Monday, they're a live stream format. So I think that the seminars that have been mostly sponsored by members or people somehow involved with the cybersecurity subcommittee had a big impact on increasing the awareness of the need for cybersecurity in ASHRAE. And the MTG was formed meeting first at the 2020 annual conference in Austin, which was also the first virtual conference. And the major impacts from that have been the four columns that have been in the journal. I've had two, Dave had one, and Jim Butler had one and I think they're all well received. The other area that the MTG has had impact is in the inclusion of the cybersecurity section in the handbook.
Anto Budiardjo:
So Mike what's been the reaction and the reception of the MTG working group and cybersecurity from the rank and file of ASHRAE members and engineers and others?
Michael Galler:
Well, one of the main goals of the group was to advocate for the increased need for cybersecurity awareness. And I think that has really had a huge impact, not just our efforts, but other people outside of the committee. It's obvious with the way the world is today, that there's much more need for cybersecurity to be implemented in all aspects of computing. I think that the efforts have had an impact in terms of the number of seminars that have been there and just general awareness. For example, as you're well aware, there's now cybersecurity related aspects to BACnet.
David Branson:
Another thing that we've noticed in the Cybersecurity MTG is that there are other parallel efforts going on in various countries across the world and it's interesting to see that roll out.
Michael Galler:
Yeah. We've been in talks with various international organizations and how to align with our cybersecurity interests.
Anto Budiardjo:
Yeah, and obviously outside of buildings and ASHRAE, the topic of cybersecurity is as hot as it can be. Last week, there was actually the RSA conference that's just concluded, 46,000 people crammed into San Francisco trying to figure out what cybersecurity is all about in the future. And obviously we see that the subject of cybersecurity mentioned in media news consistently, that's helped increase the awareness of the need to focus on cybersecurity.
Michael Galler:
So one thing that I hope we've been able to do with the ASHRAE Journal columns is to make people aware that cybersecurity isn't some abstract thing that some IT person far away has to implement for them. Everybody needs to be responsible for some aspect of cybersecurity. For example, my last column was on password security, which is a very basic way that everybody can have a big impact on the security of their organization, whether the impact is good or bad. A bad password can lead to the system being easily compromised and could lead to untold disaster for their system or for their company. And it's really very simple to implement this most basic of cybersecurity precautions.
Anto Budiardjo:
Passwords the most fundamental and obviously in many ways we use that every day. We log into our terminal, laptop, or whatever every single day. So that should remind all of us that is quite an interesting starting point to think about cybersecurity because in building systems, obviously it's not just the humans that need to be secure with their interaction with the systems, but it's also the devices. And so the same issues of passwords is basically what devices, all devices, whether it's a small PAV or a controller all need to have the same or actually even better mechanism for securing their communications.
Michael Galler:
Yeah. Having a good password is one aspect, but also not having default passwords is another. And for some places, even having a password is going to be a step up, I think.
David Branson:
That's true. There are a number of people, particularly in various trades that have a misunderstanding that cybersecurity is really just an IT issue that's not within the realm of their trade or discipline. And what they don't realize is that those are the places where cyber-attacks take place because they're the ones that the door's open for.
Anto Budiardjo:
Yeah. What I find interesting if you look at the history of control systems in buildings and why we actually have this problem that Dave just mentioned really for many years, in fact for several decades, building systems have tended to be not connected to the IP and the IT system and therefore they've been obscured in many ways. And so we've been enjoying for decades security through obscurity. Ten, twenty years ago, that started to change as we started to move more and more devices onto the IT's infrastructure, the IP technology. And that's when I think a lot of people started to realize that they can no longer just treat it as being something completely separate and that's quite a big change.
David Branson:
As a company that provides cybersecurity subject matter expertise, we've realized that there are a lot of people that think that building controls are the only systems that are involved in cybersecurity. Whereas, we actually have to deal with those subject matters on fire alarm, mass notification, energy management, HVAC control, generator set and portable load bank connection, cabinet controls, automatic transfer switches, public address systems, lighting controls, and a bunch of other specialty systems. There are a lot of different paths that cyber-attacks can take place through. Those are just doorways to get into the inside of a facility or an operation. And then they can make use of a whole host of tools that the attackers have to move around within those systems.
Anto Budiardjo:
So those systems are typically considered as OT systems, operational technology systems, as opposed to IT, and maybe worth spending a few minutes just discussing and talking about the difference between IT and OT, because it continues to be a lot of conversations about that. And I find it useful to think about OT as being domain specific technology, mainly domain specific technology, such as the list that you just read out, Dave. And so they're very domain specific, the protocols and the communications that happen within an OT domain is very specific to that domain. And generally speaking, IT people are not really interested in what happens in the domain. What they're interested in is when that domain then starts to need to cross over to other domains. And the preferred way of doing that is through an IT network. So the OT networks are feeding their information into an IT network and that's where the IT organizations start to care in terms of the visibility of what's actually happening in the OT systems. But again, I repeat, generally speaking IT people are absolutely not interested in what actually happens in the OT space. They obviously don't have the domain expertise.
Michael Galler:
Yeah, it's true. But they have to remember that security can't stop where their domain stops. They need to have defense in depth and it has to be throughout the BAS systems.
David Branson:
So I think that's a particularly important message to people involved in ASHRAE and in this built industry. And that is that all of us, whether we are providing equipment or design expertise or maintenance or management or operations, whichever, including planning by the way, probably a key element is awareness of the IT and the OT opportunities for cyber-attacks to take place. And so we need to know what is required for our job and we need to really scrutinize our tool set to make sure that we're not opening a door as an invitation for attacks. And then probably another key element is we really need to be diligent because there's a tendency to check to see if the door's open when you come in the house, but you don't continue to check throughout your stay. In this industry and in this particular subject, we need to continually check because things are very dynamic and attacks can take place at any time, not just at the entrance to a system.
Anto Budiardjo:
And linking back to we were talking about passwords. Password is effectively the key or the lock of the door. So if you have a device that has the username admin, password admin, you'll basically have an open door. It's unlocked so anybody can walk in. Many people say, "Why does that matter?" Well, it matters because two years into a building's life that may be the vector of a cyber-attack that just goes into that through the fact that the username and password is easily discovered. And then because that system, because that device is connected to the IP network that's connected to other devices, that then is the open door that will potentially cause harm.
David Branson:
And several years into the use of that system, you may not even have the same people involved that did the setup so who knows what's been set. That's why you need to continually be looking, need to be diligent and checking to make sure that you are current with the cybersecurity practices.
Michael Galler:
Yeah. You're touching on another aspect that the MTG has really been... One of the ones that they've really been pushing is that cybersecurity needs to be part of the life cycle design of a building. It needs to start when you're first planning all the other building systems. It's not something that can be strapped on effectively later once you're done. Look, you can't be ready to turn the keys over and say, "Oh wait, let me throw in some cybersecurity." That's not going to be effective. It's not going to work as well and it's going to be more expensive. As part of the lifecycle planning, you do need to do continuous maintenance on it. It's not something you can put in one time and forget about it, because you are going to regret that most likely. It's something that needs to be budgeted every year and needs to be looked into frequently to make sure everything is up to date and still secure.
Anto Budiardjo:
Especially on any kind of refurb or any kind of continuing work or even maintenance work that happens that maybe needs to replace a device and then you have to make sure that that's secure.
Michael Galler:
That's a good point.
David Branson:
Well, there are some conflicting needs that arise occasionally because when you are, for instance, doing continuous commissioning, you really need access to a lot of that data that's being collected. But that in turn causes opportunities for cyber-attacks to take place because you have ports open or other means by which to get into a system. And oftentimes a tradesman that knows that they're going to be accessing a system will set it up while they're at the site with the intention of connecting at some point in the not too distant future, but that door has opened the whole time at that point.
Anto Budiardjo:
Yeah. I think we've all been there, either ourselves or others, where something is broken so you're trying to fix it. And then as part of trying to figure out what's going on, you open a port or you change some device password with full intention of putting it all back, but you forget after you fix the problem and you walk away and you're basically leaving the door open.
David Branson:
Some of the things that we see the various trades in the ASHRAE-related industries face in particular during construction, whether it's a renovation or a new project, are items that the general public may not have thought a lot about and so I've put together a small list just for talking purposes and to sort of spur people onto thinking about these various systems. Obviously there's wireless communication that presents a set of challenges and then there's interconnection of systems, various systems. When you're dealing with lighting control and you're dealing with daylight sensors and maybe you're dealing with an energy monitoring and energy management system, then they will need to talk to each other. And so you have opportunities between those systems to have somebody think that someone else has got a door closed when in fact, none of them have that on their list. Ongoing coordination needs to take place between the various trades so that one trade doesn't have something left open waiting for another trade to connect. And then of course the overall compliance, cybersecurity compliance of the systems that are going into a building. Those all need to be looked at by subject matter experts to make sure that they're being handled properly and that involves inventory of the control systems for the various types of controls. But then you've also got portable devices that are used during startup and installation that are brought onto a site that can also be used for a cyber-attack. So there are a number of different aspects that really have to be focused on just to put a building together and allow these control systems to operate properly without making them susceptible to cyber-attacks.
Anto Budiardjo:
Dave, could you speak to how specifying engineers are tackling this or dealing with this challenge, because it's key to that?
David Branson:
Well, forward planning from the very start has to be paramount. There has to be an understanding of the level of security for plans development because the easiest way to get hold of building footprint for instance might be through somebody's collaborative effort during a design situation and it could be very easy for someone to grab a set of documents. As you put a finalized design out, you've got to make sure that you specify the requirements for interoperability between the various systems that need to talk to each other as well as really good segregation from the IT systems such that you don't compromise internal systems within a facility or enterprise level systems within an enterprise that could be used to cause harm to that entity.
So once you've got those things done, then as you move into the bidding and construction phase of a project, you've got an opportunity to check and make sure that those things are being addressed when you look at submittals, you look at shop drawings and you get your first look at what the actual systems are that the trades are going to be proposing to install. And then continual meetings typically taking place through the commissioning avenue or other project management avenues to ensure that those trades are all communicating and all resolving their issues in a way that's going to be acceptable for operation of the facility. So you have several places that you can touch as the consultant and you just have to be aware and diligent about taking care of those items as they come up.
ASHRAE Journal:
Thanks for listening to the ASHRAE Journal podcast. We want your ideas. What topics do you want to hear about and who do you want to hear them from? Email us your ideas at podcast@ashrae.org. That's podcast@ashrae.org.
David Branson:
So IT really starts at the very beginning of the process when we're doing planning and design as engineers and architects, we have collaborative efforts that take place between offices oftentimes and we have storage issues and sometimes the information that we're storing is sensitive and would certainly be of interest to a cyber-attack. And so it starts right there and of course on through construction. As you're going into the construction phase, you're dealing with the temporary location of devices that might have wireless or other network needs and uses that are going to be on site and could possibly compromise security. And then of course, once you've got your equipment, your permanent equipment installed, you've got all of the cybersecurity issues that are associated with that. Some of those items are being addressed, I guess, on a continuing and dynamic basis by developers like Anto. And so this might be a good opportunity for you to share with us what some of the new or more challenging issues that you're facing as this industry develops.
Anto Budiardjo:
I'm not sure it's a specific technology or specific as something. I think the big problem as we're starting to see is people are starting to implement and deploy BACnet SC, is how to manage all of the devices in a secure and interoperable way. And so the activities in the various groups within the BACnet community is trying to figure this out and trying to think about management of devices and system. And if I can put that in context, the way building automation systems work typically is that there are vendors that provide the system and the configuration and the logic and all the clever bits that actually control the system as they were. And they typically manage their devices and with BACnet the interoperability, the inherent interoperability in BACnet allows them to incorporate other vendors' devices obviously that's kind of the beauty of BACnet. But one of the challenges with when we start to think about cybersecurity is that the cybersecurity aspects of all of the devices is important, not just the ones that a specific vendor can actually control in their domain. So the need for cross-vendor interoperable management of BACnet devices from a security perspective is really the direction that the industry needs to go. Otherwise, it becomes a piecemeal and that really doesn't solve the problem. The other part of it is how does the industry bring into the secure infrastructure, the older insecured BACnet devices, the non-BACnet devices. Because they're not all going to be replaced overnight. That's not going to happen. So how do we bring them in, how do we firewall them and isolate them, especially when something bad happens to them? And the last part of it is there's going to be an IT department or a CSO department that is very interested in what happens in these BAS systems, although they're not interested in what actually the functionality of the BAS, but they're interested in what's happening with respect to things that can impact the cybersecurity of the IT systems. So all of this management, all of this managed information about BACnet systems need to be integrated into IT systems, into tools that IT organizations currently have to manage their enterprise network. So it's really in that kind of direction about how we manage everything, but manage it in an interoperable manner that's really, really key.
David Branson:
One of the real challenges is the challenge that you mentioned of legacy devices or old devices and keeping those in a situation where they're compliant with the security measures that you've got in place or are implementing. And it's sort of like buying a new car, it's new until it's off the lot and then it's old. That's the way it works with computer equipment so that's a real challenge.
Anto Budiardjo:
Yeah and firewalling them and we have to be careful of the use of “firewall” because in the IT world firewall is really typically viewed as the IP protocol firewall and that's not going to cut it because for BACnet systems or building automation systems, you really need to be aware of what's actually going on. Application level firewalling is really what's needed. It's not just sniffing the packet, it's actually with an understanding of what's happening.
David Branson:
And one of the things that you had mentioned earlier about OT, other technologies, a list that I had rattled off, that's just today's list. There's a growing number of items that are being demanded to be accessed in some type of a network configuration and so that frontier is ever expanding and our ability to keep up with it is paramount.
Anto Budiardjo:
I agree. And that list that you laid out is a great list, but it's probably 10 times that if you actually think through all of the different things that goes well beyond HVAC and environments. I've gone through exercises with people to actually—in various groups to start to list all of the different types of systems and it can easily get into the hundred plus if you try and explore all of the different possibilities of systems that are in buildings that somehow relate to HVAC and BAS and that's starting to use IP and IT systems.
David Branson:
So part of the challenge to this is standardization. And that's something that I know NIST has been focusing on and trying to write a body of information and guidance that is complete enough that it encompasses what we are dealing with, and yet generic enough that it can welcome in the new items as they appear.
Michael Galler:
Yeah. There's a lot of effort put into writing the standards developed at NIST. I'm not a part of developing those standards, but I believe that they are designed to encompass every need. They want them to encompass as wide an audience as possible.
Anto Budiardjo:
The NIST cybersecurity framework is really an important thing to think about because that's kind of creates a framework of what we have to think about with regards to cybersecurity. And it's quite interesting because that goes through, can't remember off the top of my head the exact terminologies, but it goes through all of the steps required all the way from identifying all of the components, protecting them, and then detecting anything that goes awry, and recovering, and what do you do after it. The whole policy side of things. How do you manage the process of protecting and then managing a particular incident and learning from that? The NIST, that framework, is really key to understand the scope of everything that has to be thought about.
Michael Galler:
Yeah, it certainly seems to be a integral part of any cybersecurity plan.
David Branson:
Well, this subject is certainly grabbing a tiger by the tail so to speak and trying to hold on. So they'll be new and exciting things at least to the IT and OT people exciting, maybe to others they'll be either transparent or headaches. But our efforts are all going toward trying to make our work products and our workplaces more secure and protect intellectual property.
Anto Budiardjo:
One of the areas that's a concern, which is not actually to do with technology is to do with skill and workforce. A lot of these IT issues are getting very geeky very, very quickly in terms of the cybersecurity issues and IT issues. And typical BAS workforce, it's not likely to necessarily go down that level of insight and understanding of cybersecurity issues. How do you bridge that gap? Because obviously BAS people and people that are involved in HVAC are domain experts in controlling and managing air and all of that kind of stuff, and not about cybersecurity so there's a gap there,
David Branson:
And it's not just education, it's also interest and focus because we all have our plates pretty full of our specialties and adding things that are side issues can sometimes have an impact on our schedules.
Anto Budiardjo:
Yeah. And going back to that list we were talking about earlier, the same applies to all of the professionals in the domains of those lists. So if you think about lighting system and elevator controls and all those kind of things, they all have professionals that are very focused on those domains, those areas. And again, they're not necessarily going to be experts in cybersecurity.
David Branson:
One of the things that seems to be consistently developing within that list and others also is that interoperability issue that you mentioned. So you've got people who are dealing with interface issues that they may understand one side of the equation fairly well, but you need a team approach or someone with a pretty broad, horizontal background in order to address them adequately.
Michael Galler:
Yeah. I think there's a gap in the knowledge about how it implement cybersecurity, but not in the skill that's required to implement it. And for example, I've taken training from a couple different manufacturers on how to install and configure their control systems. And the level of knowledge required for that is probably more complex than for a lot of areas of cybersecurity. So they could certainly integrate cybersecurity training, for example, into the controller programming or some aspect of the manufacturer training. And I think that would fit in quite well with some of them and I think that's certainly not even close to beyond the capabilities of the people taking the courses.
Anto Budiardjo:
The other element of it is the scale of all of this and the difference of scale between IT and OT. IT is typically involve networks that's managing laptops and computers and mobile devices so the numbers are relatively small. When you start to think about OT, especially when you start to go down that list, Dave, you are dealing with hundreds and thousands of devices in a medium sized building. So that's another difference that is not obvious and that creates its own challenges because it's just the sheer number of them.
David Branson:
So that takes us back to the issue of awareness. Part of the challenge is making people, users as well as developers within a building infrastructure, aware of all of the points of connection or potential connection. And keeping them aware of at least the presence of protections so that they can acknowledge and comply with those protections
Anto Budiardjo:
While abiding by the policy, the IT and cybersecurity policy, of the owner of the building that it's all going into. And one thing I was going to mention is that we talk about IT as a monolith, but it's really is not. IT is broken up into many different disciplines, when you start talking to IT people there are networking people, there are cybersecurity people, there are people that are developing enterprise applications for HR or CRM or whatever, and there are data center people, and there are people that manage information and telecommunications. Quite often they seem like IT, but they're very, very different. And even in those disciplines, there are differences in how things are done. So this becomes quite an interesting complex challenge. One of the things, again, going back to OT and IT is that in IT cybersecurity, there's this term called CIA, and I'm not meaning the spies in D.C. I mean there's a term called CIA, which is really to do with the priorities of IT and cybersecurity because they care about confidentiality first and then the integrity of the systems and then availability. That's their priority list. When you go to OT systems, it's actually totally the other way around, because we are more focused on the availability to make sure the buildings are operating. So it's actually AIC, it actually goes the other way around. We're focused on the availability because that's the most important because you need to be able to use the building and the integrity is next and then the confidentiality is typically is not the most important. Now that brings up other issues about personal information, but putting that aside, just the fact that the priorities are different, makes it quite often difficult to have a conversation.
David Branson:
And a real challenge to encapsulate AIC within CIA, I think that may be a palindrome.
Michael Galler:
Dave, have you run into any challenges with the prioritizing the C or I or A in installations or integrating with your OT with IT?
David Branson:
There are the regular challenges of informing people who have not really been exposed to interoperability issues in the past. They're tradesmen, they're very good at what they do, but they don't really have an interface and they don't think in terms of potential compromise of the system. And so that's an ongoing challenge in making them aware of all of that facets that cybersecurity represents.
Anto Budiardjo:
There is obviously a lot going on in building automation systems and cybersecurity. We talked about that before. It's almost bewildering at some point, but it's needed. And it'd be interesting to get your opinions as to whether this is going to continue to expand in terms of the number of initiatives and number of organizations or whether it's going to start to consolidate or organize or some kind of rationalization.
David Branson:
I think the subject is maybe ambiguous enough or ill-defined enough that there won't ever be a single dedicated manual that's the end all be all go to place for direction on how to deal with cybersecurity. I think it's going to be a fairly serpentine business and we'll always be developing new ways of detecting and securing intrusions. At the same time, there'll always be new efforts going on to develop intrusions that we'll end up trying to detect and secure.
Michael Galler:
Yeah, like Dave said, all of cybersecurity is a moving target and nothing is static with it. The threats are always evolving and the methods to respond to them are always evolving. So there's little chance of getting one static document and said, "This is the solution." Because that's going to work for about five seconds for one building.
David Branson:
One observation I'll make is that just in the tenure of this topic, methods for intrusion have really become more complicated and more complex and so those solutions are becoming more complex to deal with them. I don't expect that to change. I expect things to get more sophisticated, not less.
Anto Budiardjo:
On both sides.
David Branson:
On both.
Anto Budiardjo:
Yeah.
Michael Galler:
I'd like to thank the staff, the ASHRAE Journal, for giving us the opportunity to record this podcast.
Anto Budiardjo:
Yeah. I think just having this podcast in this industry is a big deal. And having elevating the subject to be something that everybody should at least be aware of, even if they are not in the front line of actually doing something, is really key because cybersecurity is a team sport. Everybody kind of needs to at least know what's going on and be part of understanding that things need to be done to resolve it, which is not easy. So really, really good initiative to focus on this on ASHRAE's part and on the staff as well. Thank you.
David Branson:
Yes. I agree. I think that ASHRAE, as well as other efforts are taking place, you may not see the effort, although you're welcome to join the ASHRAE MTG and participate either as a listener or as an active participant. We welcome all attendees and we'll find a place for you. You don't necessarily have to be an IT or an OT expert, you'll find that some of us are not, and yet we find things to do on this subject. So we really appreciate the opportunity to introduce cybersecurity through the podcast series. And we hope that this is the first of possibly more podcasts on the subject and we would welcome that. And we'll start looking for new material.
ASHRAE Journal:
The ASHRAE Journal podcast team is editor John Falcioni; producer and associate editor Chadd Jones; assistant editor Kaitlyn Baich; and associate editors Tani Palefski and Rebecca Matyasovski. Copyright ASHRAE. The views expressed in this podcast are those of individuals only, and not of ASHRAE, its sponsors, or advertisers. Please refer to ashrae.org/podcast for the full disclaimer.