Marktkommentar #4: MITRE ATT&CK MDR Test Results

Hello everyone,

the new MITRE ATT&CK Managed Service Evaluation (menuPass / APT10 + ALPHV BlackCat) has been published, here are a few take-aways:

• The methodology was adapted compared to the last MDR evaluation (Oilrig): “Actionable Detections” were now evaluated according to a comprehensible scheme. In addition, the Mean Time to Detect (MTTD) was measured and published for the first time.
• Crowdstrike performed very well in cross-comparison: ~93% of the maximum achievable techniques were fully recognized and reported in a record time of ~4 min.
• For comparison: MTTD ~24 min at Microsoft and Palo Alto, ~40 min at SentinelOne, all with lower detection rates. Trend Micro was far behind this time.
• Surprise: Bitdefender with the same detection rate as Crowdstrike. In previous tests, Bitdefender has tended to be in the midfield.
• Also interesting: There was an independent MSSP (SecurityHQ) that participated on the basis of SentinelOne, but with significantly worse results than the MDR team from SentinelOne itself => This shows once again that customers are often already very well served with the standard MDR service of the manufacturers
• Unfortunately, neither Cynet nor Fortinet took part
• Noticeable differences in the technical solutions, e.g. in the exfiltration of credentials via ntds.dit (Active Directory database) from the Recycle Bin, detection of Pass the Hash (execution from \temp folder), and especially in the final steps such as the deletion of shadow copies and the deactivation of services.
• – Otherwise, the difference in the assessment (actionable / not actionable) naturally depends on the enrichment of the information by the MDR analysts. In my view, this shows the strength of Crowdstrike, which has drawn on the full potential of its reporting.
• In practice, of course, neither the detection rates nor the speeds are achievable for large customized installations – resources are not spared during the test, the test environments are manageable (~15 endpoints, no cloud workloads in containers), and there are no false positives due to normal operation. Nevertheless, the MITRE evaluations remain the current gold standard for such tests in my view.
• Grosso Modo, however, it can still be said that anyone who rolls out one of the market-leading systems across the board is already making things very difficult for attackers
• Too bad: The configuration of the technical EDR/XDR systems was no longer published. However, this is crucial for customers to be able to compare offers. In the previous evaluation (Turla), it was still possible to see transparently that Microsoft also used the Defender for Identity module, Crowdstrike used all Falcon modules, and Trellix used the full Helix Suite => If you don’t buy all of these and adjust the settings to the highest sensitivity, the protection and detection rates achieved in the test are not meaningful. I would assume that all available modules of the products were also used in this test and that the settings do not correspond to the standard. For example, the providers’ standard MDR playbooks typically do not include the monitoring of volume shadow copies.
• In our comparisons of offers, we see that price/performance varies greatly in practice depending on the customer project and requirements => So please don’t take my points on this individual test as a blanket recommendation.

In general, of course, it would be desirable to have such independent laboratory tests of detection rates for other categories such as NDR, DLP, email filters, etc. as well. Unfortunately, neither providers nor customers seem to be willing to pay for this (providers have to shell out around USD 140k per MITRE test to participate). The Common Criteria tests only refer to the protection of the security products against attacks on the product itself, not to the performance or the fulfillment of marketing claims (I once read the Carbon Black certification report on this – if anyone here has other information, please let me know).

While we’re on the subject of Crowdstrike, we received a briefing on current features. Here are a few notes, including key points from other meetings with providers:

• Real-time response like quarantine also possible with MacOS
• Upload capability of hosts in quarantine for files up to 4 GB
• Identity protection:
  • No more separate agent for domain controller, all functions integrated
  • Service accounts can be forced based on rules to authorize access by a natural person, incl. MFA (=> delegation to another account)
  • Detect compromised passwords: Fragments of the hashes are extracted from the customer domain controller and compared with the darknet database. In case of partial match then comparison of the complete hash
  • AD Security Configuration Audit (e.g. duplicated passwords; password never expires, exposed passwords in GPOs, privileged machines running print spooler)
  • Detection of “stealthy” accounts, e.g. Sync Entra ID account (may create other users, but no domain admin)
• Data Protection: Data extraction via printer, OneDrive, from Salesforce etc. recognizable / rule-based blockable
• MDR (Complete) can now also be booked for the Logscale SIEM (formerly Humio)
• The Breach Warranty now applies to all supported operating systems including Linux and MacOS (previously only Windows). However, customers in Italy and Japan are still excluded, for whatever reason.
• AI assistant (“Charlotte”): Writes queries for customers, e.g. “create a query to find all processes for privilege escalation” => I can well imagine that Crowdstrike will have a very reliable set of working queries that could be used to train the AI
• Spotlight (Vulnerability Mgmt) also contains a workflow tool like Brinqa and Cisco Kenna for tracking patching or mitigation measures
• Threat Intelligence is now called Counter Adversary Operations

Torq:

• SOAR / No code (or low code) workflow automation
• Examples: Just-in-time access rights, enrichment of data, reactions to alarms, onboarding/offboarding processes
• Probably much easier than automation via Azure Sentinel or similar if many different systems are to be used in the workflows. On the other hand, Microsoft Copilot also helps with this.

Dope Security:

• Yup, it’s really called that (in California, “dope” is apparently the current trendy term for “cool”)
• Web Proxy / CASB (Security Gateway), i.e. competitor to Netskope, Trellix, Forcepoint, Zscaler, Cisco…
• Agent on each endpoint (~100 MB in size) for policy enforcement, i.e. traffic is not routed via a data center
• This means faster connections, decryption only locally on the endpoint, no fail-open in the event of a cloud proxy failure
• Policies are of course distributed centrally. Anti-tampering, even local admins cannot remove or reconfigure the agent

Enginsight:

• Combined platform for endpoint management (inventory CMDB, file integrity), SIEM and vulnerability scans externally and internally (the latter is unfortunately referred to as a pen test)
• Vulnerability scans: Similar to Qualys or Tenable, apparently own engine, detects CVEs and misconfigurations, but no exploits. High visibility and attention to detail because an agent is naturally installed on every endpoint. And indications of GDPR problem areas such as unencrypted transmission of personal data
• Windows, Linux, Mac, but no mobile devices
• No own AV/EDR (but GDATA natively integrated at the customer’s request)
• Everything in German, including the recommendations for vulnerability remediation
• German company, founded in 2016, ~30 employees
• Makes a friendly impression from front to back, suitable for SMEs
• Almost 1000 corporate customers, including many KRITIS operators

A few days ago, I was once again able to jet off to Berlin for the “Public IT Security Congress” (PITS), organized by Behördenspiegel and held at the Hotel Adlon (where else). What particularly stuck in my mind:

• The BSI has at least 1 very cool, courageous and self-deprecating department head, namely Dirk Häger. A real silverback of security. His statement on stage included: “Security awareness is a waste of money, there is no evidence in over 20 years that it reduces the likelihood of a successful cyber attack”. That’s probably not 100% true – but saying that at an event sponsored by SoSafe definitely takes a lot of backbone. There’s also a Podcast about the cyberattack on the district of Anhalt Bitterfeld, in which it comes up a few times.
• Confidential computing in the cloud is coming, and not just from Apple. Surprisingly (see also photo): Schwarz IT‘s Stackit cloud solution seems to offer less in terms of security requirements than the hyperscalers. There is still hope for OpenStack. Nobody in Germany seems to be interested in the clouds from Oracle, IBM or Salesforce.