This week Philipp Pelkmann had an interview with Erik Van Buggenhout, SANS Instructor & Author and Co-founder of NVISO, a cybersecurity provider from Belgium with offices in Germany. We wanted to learn more about the MITRE ATT&CK evaluations in which NVISO did also take part as Managed Security Service Provider and could gain insights and learnings.
You co-founded NVISO. Could you briefly introduce your company? What is the core focus of your portfolio?
NVISO is a pure-play Cyber Security company founded in 2013 in Brussels by 5 ex-Big four managers. They always had an itch to do things differently (and better), decided to start their own company and with a strong mission: to safeguard the foundations of European society from cyber-attacks. NVISO offers a wide range of professional services, assisting customers with prevention, detection and response services. Next to that, NVISO also offers Managed Services – such as 24×7 Managed Detection and Response. NVISO currently employs about 220 people and has offices in Brussels, Frankfurt, Munich, Vienna and Athens. NVISO is rapidly expanding into other countries and has an aggressive growth strategy for the next years. NVISO has customers in 20+ countries, primarily the Finance, Government, Defense, and Technology sectors.
Today we would like to mainly talk about the MITRE ATT&CK evaluation. Would you say that in recent years the MITRE ATT&CK framework became the de-facto standard for documenting cyber security tactics and techniques?
Unless you’ve been living under a rock for the last couple of years, you’ve probably seen MITRE ATT&CK quickly gaining traction as indeed the de-facto standard for documenting adversary tactics and techniques. MITRE ATT&CK Tools like ATT&CK Navigator facilitate corporate adoption and allow for a holistic overview on attack techniques and how the organization is preventing and detecting them. Furthermore, many vendors, technologies and open-source initiatives are aligning with ATT&CK. One of the advantages of the MITRE ATT&CK framework is that it allows for easy sharing of adversarial techniques and detection opportunities. Customers of cyber security services have also become accustomed of the usage of the MITRE ATT&CK techniques in reports and dashboards
NVISO participated in the Managed Services Evaluation 2022 “OilRig” (inspired by the real scenario). Can you explain a bit about the general setup of such a test?
The idea of the MITRE Managed Services Emulation was to test how well participants would detect and report on MITRE ATT&CK Techniques. Unlike other MITRE ATT&CK evaluations, the approach was a “black box” approach, meaning that the scenario was not announced to the participants. This round focused on the threat actor OilRig and its use of custom web shells and defense evasion techniques. During the course of 5 days, MITRE emulated several ATT&CK techniques and benign user activity. At a high-level, the emulation plan commenced with a phishing email containing a link to download a macro-enabled document armed with an implant. After activation, the implant would pivot from machine to machine in the environment – discovering new hosts and dumping credentials along the way. Eventually, the SQL server (the crown jewels) were reached by the MITRE Red Team and the database information was exfiltrated.
You were using Palo Alto Cortex as a sensor. Can you share more about the technical setup and configuration, e.g. the amount of sensors used?
The test environment consisted out of 4 machines on which NVISO installed the Cortex XDR Pro sensor: one domain controller, an email server, a SQL server and one workstation. The Cortex XDR Pro environment was then connected to the NITRO SOAR Platform, which is the command center for our Managed Services. This is a highly representative environment for our MSS customers. NVISO has a lot of experience with the Palo Alto Cortex technology stack – we know that the combination of the Cortex XDR Pro sensor with our NITRO SOAR Platform and expert services is able to detect what is important.
You covered 78% of the steps – what does it concretely say?
It means that, while we detected and reported every single attack phase (initial intrusion, privilege escalation, lateral movement,…), we chose to report 78% of the attacker steps. Note that a phase includes a large number of “steps”, which are smaller parts of the attack. Our goal was not to report 100% of all smaller attacker steps: We don’t believe in overflowing our customers with information, but report and respond to what matters. We believe we did this very successfully and we invite everyone to assess our results in a bit more detail (see https://mitre.nviso.eu). MITRE also specifically mentions in their blog post on the debrief of the evaluation that is was not the intention of the evaluation to have the service provide cover all steps:
“We did not expect (nor believe it is inherently valuable) for each service provider to report each technique/sub-technique evaluated in the Emulation Plan.” (source: Medium.com)
Readers are always keen for rankings of the participants which is not provided by MITRE. How do you interpret your own achievements?
As already mentioned, we don’t like the idea of “ranking” vendors. We believe that, in order to correctly interpret the results, the below are solid criteria:
- Were all of the different phases of the attack detected and reported?
- Was a significant number of the smaller attack steps detected and reported?
- How timely were the attack steps reported?
- How qualitative was the communication?
- …
The first two criteria are relatively easy to quantify (and all participants like to throw around coverage percentages 😊), but we believe there is much more to it. The latter criteria are a bit less easy to assess in an evaluation such as MITRE.
What are from your point of view limitations of the test? What does it not show or where does it have blind spots?
While the scores of the test reflect the amount of MITRE ATT&CK techniques reported, it does not cover the quality of the reports. We feel that that is an important factor for anyone that is considering a (new) Managed Services partner and would invite everyone to have a look at the reports that have been submitted by the participants.
How should e.g. a CISO look at the results as there will be always MSSP or product providers missing in a concrete evaluation, there will be different interpretations and each vendor will claim ‘excellent’ results? What is your conclusion from a potential customer point of view?
We would recommend to anyone trying to interpret the results to take the following points in consideration:
- Not all techniques are equally valuable – e.g. the phishing attack was the main intrusion point in this scenario so any detections of techniques that pertain to that attack might be more valuable than others.
- Have a look at how service providers presented their findings. Some providers just reported alerts and provided the link to the console so the customer could investigate further if they wanted to – while others provided more of a narrative including context and analyst notes.
- Determine if the service provider correctly attributed the adversary. This often is an indicator of the maturity of the threat intelligence capabilities of the provider.
- Check if the service provider recommended any remediations for the activity they detected.
Based on these elements, anyone in the market for a service provider should be able to make a shortlist to start a deeper conversation with.
Thanks to Erik Van Buggenhout for his time!
In our day-to-day business, as your independent partner we analyze customer requirements towards cybersecurity and identify suitable providers of products and services. Therefore, we have collected a significant number of interesting provider and solution profiles. Of course, this does not include any recommendations for products or providers. Also be sure that we do not receive any advertisement payments for the interviews. If you are int
Please remember: This article is based our knowledge at the time it was written – but we learn more every day. Do you think important points are missing or do you see the topic from a different perspective? We would be happy to discuss current developments in greater detail with you and your company’s other experts and welcome your feedback and thoughts.
And one more thing: the fact that an article mentions (or does not mention) a provider does not represent a recommendation from CyberCompare. Recommendations always depend on the customer’s individual situation.
