Justin Searle
Together with Justin Searle, we talked about the process and the challenges of performing real pentests in an industrial environment and for IoT products. He also talks about specific trends and technologies for cybersecurity. He is the Director of ICS Security at InGuardians and Senior Instructor for the SANS Institute.
Dear Justin, thank you very much for your time and the opportunity to talk to you today. You have more than 20 years of experience in cyber security now. Can you tell our audience a few sentences on your background and your current roles at InGuardians and SANS?
I wear a few hats – often most of my weeks are divided to different goals. I am currently the Director of Industrial Security at InGuardians. We mainly do penetration testing and security assessments for industrial control systems and IIoT devices. We also work a lot in IT areas, with a strong component of cloud security, one of the leading researchers of Kubernetes is part of our team. Also my time is broken into time I set aside to do different types of research into protocols or attacking techniques or trying to experiment on new ideas with different equipment I have here in my office. I have also been teaching for Black Hat for over 14 years, basically every year, and for SANS Institute for 12 years now.
Is there a story about you which has not been told before (or not enough times)?
My very first fulltime job was for an engineering firm, building control cabinets, for example at water treatment facilities. And this is well known in the security community.
What is not well known is the part that comes next: I have good memories of wiring kilometers of cables in these cabinets. Working with the engineers to find flaws in the system, or faults in me wiring the system was sometimes very painful. One time I was changing some wires and dropped my screw driver… and I caught the screwdriver, but I also hit my knuckles at dozens of the live connectors. And then I did the same thing again 15 minutes later :-).
What does a typical day in your working life look like?
That’s a tough question. Often my weeks are kind of divided with different goals. Maybe could be a whole week of courses that I am teaching, dedicated at Black Hat or SANS, for example. Then I could be catching up on correspondence after the courses, emails and so on.
Of course, time is set aside for work with InGuardians. Sometimes I support our sales team, as clients often don’t know what they need – they know they need support, but they don’t know what exactly on. Often I am involved with pen testing in some kind of form or another. I am not full time pen testing anymore, but involved in most pen testing. That’s one of the advantages of being a Director: You can cherrypick the activities, and don’t need to do the reporting anymore.
Without naming names: Can you walk us through a real pen test in an industrial environment?
First off, we should probably break up what kind of pen tests we do. One kind of test is where we try to find vulnerabilities in a production network, so a plant or facility.
Pen tests in ICS environments are very different looking from testing an office IT. We are very careful how we interact with the system. We test the perimeters, but once we get inside it is much more white gloves on, and we don’t touch many of the process systems. We spend more time on the supervisory area, but are usually not entering the cells and trying to access the process components, as they are not built to require security.
A recent example is from about 3 weeks ago, when we did a production system assessment for a food manufacturer with more than 20 sites. It was a 2-step engagement: First we had to test the perimeter of the corporate network down to the ICS infrastructure. In a second step, we checked the supervisory level and operational assets in one of larger plants.
What was a main challenge in this case?
The week before we started we spoke to several engineers. One of them said that there were no defenses, no perimeter, and that in fact that he wouldn’t be comfortable with us probing the plant’s systems.
When we performed the work, we shipped a box to connect with the site remotely. But we couldn’t do any ping sweeps or port scanning based on this engineer’s request, which limited our options.
And after redefining the scope multiple times, there were still some differences of opinion on what was in scope and how we should proceed without causing interruptions. We mainly identified which corporate systems pushed information to the ICS systems, and so only checked vulnerabilities in high level systems.
In the end, this was maybe a good example to clarify scope and requirements upfront.
Can you tell us about a test for an IoT product?
Yes, this is the other kind of tests which we do. This is more reserved to labs, where we do product testing of different components. And this is where we are more aggressive.
About 5-6 weeks ago we did a product test of a new sensor solution. And this device had a web interface on an aggregator for the sensor signals to send data via a cellular link. Our team went through this interface and discovered that it was built with very old CGI connection space. We found a command injection vulnerability, so we could run the CGI at root. The architectural method they used on the backend was reminiscent of the standards 20 years ago, even though rest of embedded operating system was much newer in design. So we are used to finding vulnerabilities, but the design flaws in this case were surprising to us.
How did your client set priorities in taking mitigating actions?
Often you don’t know what the actions are on the client side. The work we did in this case was for the vendor, who wanted to come out with a new product. So we hope they’ll remediate the security design before they go to market, but of course we don’t know.
What has been an example of a very challenging technical situation for you and your team?
That’s part of what I love ICS – often there are no good or full solutions yet. One kind of thinking about ICS, is ICS technologies are built of layers of various tech solutions.
So I am spending time and developing tools to get between the different layers and find vulnerabilities. One challenge we always have, we not only have common protocols, but also custom or proprietary protocols.
This is for example why I developed ctserial myself, so I can send and receive raw data and interact with many kinds of systems, generate my own packets, analyze the return answers. This is also what we do in my Black Hat classes, we try to reverse engineer the serial protocols. We ship students a PLC (a Velocio Ace 1600) and we teach them how to gain access and build their own testing harness for that interface.
Some of our customers are now asking us for automated pen testing solutions. What are your thoughts on this topic?
I would say, first off, we would need to define what automated means. If automated means you buy a tool with a big green button, and you point it at the system push the green button and it tells you the vulnerabilities, this is not a pen test. Running Nessus is not a pen test.
But: During a pen test we are using a lot of automated tools to become more efficient. One of the biggest problems we have with automated tools in ICS is that most of them do not have enough intelligence to probe in a safe manner, because it depends a lot on the system you are testing. For different scenarios, you need to use different technologies.
Is the production network currently turned down, or running? Makes a huge difference in safety impact.
For IoT product tests in lab environments, you can use more automated tools and don’t need to worry so much about destroying things. But on the other hand, you are also doing more manual reverse engineering work to find zero-day vulnerabilities. Automated tools typically only find known vulnerabilities.
In the IT world, automated tools have been increasing productivity for pen testers. But in industrial environments, there is an endless list of technologies that don’t have automated solutions to aid in your assessments (yet). Most automated tools are more for gathering information, not for finding vulns. We are looking for problems that are out of the box.
There is an infinite number of ways to do things incorrectly. Abstraction makes engineering and programming easier, it makes security much more complicated.
Are there any security incidents which you have recently become aware of at medium sized businesses – some incidents that could have been prevented with cyber security measures?
For example, Cryptolocker software and other ransomware is affecting clients all the time. Most of these breaches are due to a weak perimeter and lack of limitations inside of control boundaries (enforcement boundaries). Poor remote access decisions are another common weakness.
By the way, there aren’t any pure OT security measures from my point of view. Almost all of our OT security tools are based on the same techniques we use for IT security tools, and in many cases can be the same tool. OT specific tools generally just have some more awareness, like of ICS protocols. The implications are: There is a lot of knowledge and existing tools in IT that we can take advantage of. For instance, for patch management I usually recommend to use the same tools as in IT (but with 2 different deployments and 2 different teams managing these), so the skill sets are transferrable.
What are some misconceptions or (partially) wrong statements which you often hear about OT or IoT security?
There’s lot. Here are some interesting ones:
- “We don’t need a separate Active Directory for OT” (Problem: Credentials for OT systems are saved and cached on all AD servers of the enterprise. Breach of any AD controller means a breach of all credentials for OT)
- “It’s OK to manage OT network devices (switches, routers, WLANs) from the IT network” (Problems: Configurations and firewall rules could be changed, malware brought on level 2 and below. Ethernet WLAN could be hijacked)
- “A full VPN solution for OT remote access is secure” (Problem: Granting remote access users the ability to bring all their data and giving full control, instead of terminating in the DMZ and then going via a jump host to level 3).
Another mistake at Purdue level 3 is to throw all servers into the same subnet, you should also separate them into subnets or V-LANs.
What are some new trends or technologies in cyber security that you find interesting?
Permanent connections of industrial components and systems, which are cloud-type connections. Cloud-based solutions in general, for example management of multiple sensors/actuators, or for data gathering and analysis. E.g., In transportation there are often GPS units that are gathering engine and brake data or from proximity sensors. Pharma manufacturers are also collecting large amounts of data of their processes.
Subscription-based IoT protocols are coming more into the ICS space, as well as container technologies. For me it is more like a continuation onward – whatever IT develops, we usually adopt a few years later.
If you could send an email to all CIOs of the world’s industrial companies, what would be the core message?
IT/OT convergence is not IT/OT assimilation.
Boards want to see the same kind of benefits they have seen in IT from OT. But the risks are different, they require different security measures. And they require a dedicated team or sub-team. Because there are many similarities, but also differences in deployments. We need to understand the nuances and explain those properly to C-level staff.
We should strive to adopt many of the same tools to take advantage of vendor bulk discounts and to have the ability to share skillsets between IT focused teams and OT focused teams, but we should still maintain separation from these environments and develop dedicated teams (or sub-teams) that exclusively support IT and cybersecurity infrastructure in OT.
The risks are very different for OT than IT. The sensitivity of OT is much higher than IT. The IT and cybersecurity staff that work in the OT environments need to learn how these differences affect their day to day activities, and more importantly, need to build relationships of trust with engineers and operators by working with them on a daily basis. That is IT/OT convergence. Making the environments operate in similar ways with similar tools and similar governance programs, but NOT trying to do things in the exact same way with the exact same people.
Please remember: This article is based our knowledge at the time it was written – but we learn more every day. Do you think important points are missing or do you see the topic from a different perspective? We would be happy to discuss current developments in greater detail with you and your company’s other experts and welcome your feedback and thoughts.
And one more thing: the fact that an article mentions (or does not mention) a provider does not represent a recommendation from CyberCompare. Recommendations always depend on the customer’s individual situation.
