CyberCompare Marktkommentar #35: AI Pen Tests u. ISO27001 Audits

Hello everyone

Incidental observation from business life: Swedes are always (always!) so pleasant, friendly and sympathetic in their dealings that they feel obliged to go the very last extra mile and then a few more meters in every project without exception. Even if the business case has long been deep red. And with a good feeling about it. The same applies to people from Canada. Probably they are related somehow, would also explain why they all speak English so well. I estimate that at least 30% of the gross national product of both countries is due to this reciprocity superhero power.

From cold countries with warm-hearted people to a hot market: Automated pen tests.

  • In a study by Stanford and Carnegie Mellon (pen test on 8000 endpoints and > 10 subnets),AI agents performed better than 90% of human OSCP-certified pen testers
  • Kevin Mandia (founder of Mandiant) has launched a new company for AI-based red teaming: Armadin (previous name: Ballistic AI). 160 million USD funding
  • Novee from Israel will receive 50 million, as will a startup for autonomous pen tests. Market competitorXBOW has now raised 75 million euros
  • Sergej Epp (CISO Sysdig, formerly CISO at PAN) argues that training AI in offensive sec tasks is easier than on the Blue Team side: Successes of intermediate steps can be easily verified (Login successful with stolen credentials? Shell opened? Exploit executed without error?). In attack detection training, on the other hand, a higher proportion of false positives or alarms is naturally examined, for which a judgement cannot be formed beyond doubt (“inconclusive”) = > Pen tests are therefore well suited for the use of AI
  • I am still torn between the enthusiasm about the technical possibilities and the doubt that the above-mentioned companies will ever generate the capital costs of the multi-million dollar funding:
    • In addition to all the established competitors in the field (Pentera, Horizon3, Hadrian…), there are tens of thousands of human pen testers in India, Romania, etc., who do not have to earn interest from the investors. Because there is no capital in the company. And these freelancers or boutiques can therefore continue to offer at similar costs for a long time despite higher manual effort
    • Analogy: Many factories in low-wage countries are still not very automated (although the technology for this would have existed for decades) because it is not worthwhile
    • The operation of AI also causes variable costs (Sansec, for example, determined about 30 EUR to Claude inference costs per exploit in an experiment). After all, the shareholders of Nvidia, Google, Anthropic etc. also want to be paid
    • At the same time, the demand for SW tests will of course increase exponentially for the foreseeable future. KPIs: New app submissions to Apple increased by 60% last year, driven by Vibe Coding

Notes from aISO27001 Lead Auditor training (by the way, extremely diverse group, we were 7 white men, two of us were married and another had long hair):

  • Everyone knows the cases where companies keep the scope to a minimum (e.g. the operation of the doghouse in Timbuktu) in order to obtain an ISO 27001 certificate, and then impress unsuspecting buyers
  • This kind of deception is increasing. But also
    • simply forged certificates or
    • Certificates from non-accredited consulting firms. In principle, anyone can issue a ISO27001 certificate.And smaller customer companies in particular lack the capacity to question the resilience of supplier certification
  • A ISO27001 certificate is anexpensive pleasure compared to ISO 9001 or 14001:
    • The prescribed audit effort (which is of course also standardized, namely in ISO 27006) is almost twice as high
    • Even for a sausage stand with 2 employees, 5 auditor days are required for an initial certification, and 14 auditor days for 200 employees.  Before that comes the document check and then the report => The slip of paper is hardly available for less than 10 thousand euros, even in the smallest expansion stage.
    • And this despite the fact that the daily rates, especially in relation to the jam-packed audit days, are surprisingly low : ~1,200 EUR is normal
    • By the way, about half of the auditors are freelancers subcontracted by the accredited certification company and receive about 700 EUR/audit day
    • One reason for the low daily rates is that customers can also be audited by auditing firms accredited in other EU countries – German customers can therefore also fall back on TÜV Austria or the Austrian Computer Society, for example
  • In practice, certificate revocation is extremely rare (“never experienced before”).
    • In the event of a major deviation, the audited organization has 3 months to present a solution (in most cases, a risk analysis is sufficient for healing). Initial certifications are usually particularly benevolent.
    • Exception: According to the certification contract, DAkkS has the right to participate in audits (“witness audits”) – customers have no right of objection, even if these audits are naturally much more formal and demanding than without supervision by DAkkS
  • ISO27001 audits still have to be carried out at least 70% on site , even if the audited company does not have its own IT
  • Filling the roles of IT manager and ISB/CISO with the same person is gem. ISO compliant with the “Segregation of Duties” requirement, provided that a risk analysis for possible role conflicts has been carried out
  • There is still no case database with anonymized situation description, recognized/unrecognized evidence and assessments that could be used for training an AI or for the systematic calibration of auditors. Not even the question of how many secondary deviations lead to a major deviation is defined or limited by “case law”. Instead, auditors take part in annual case reviews and there are possibilities for complaints or objections in the event of a conflict.
  • In addition, there is still no collection of best practices for the effective + efficient implementation of certain requirements
  • It is difficult to avoid an inflationary softening of the actual appraisal practice in the long term, despite the required impartiality of the examination bodies. In this respect, it is quite understandable that the BSI makes much more concrete demands with IT Grundschutz
  • All in all, this seems to me like a methodology that has fallen out of time

M&A:

  • Aikido (everything about security for developers – and everything AI-based, from Belgium) achieves unicorn status – faster than any other European cybersec startup
  • Upwind (Cloud Security) receives another USD 250 million
  • Claroty Raises Another $200 Million Funding Injection @ Rumored $3 Billion Valuation – Confirms Nozomi CEO’s Statement That OT NIDS Providers Are Unprofitable Across the Board
  • Is anyone still alive who knows something about the name RSA? Here, too, lenders are putting another ~130 million into it, presumably to develop something with AI

Vendor briefings, today exclusively with companies that start with “Cy”:

CyberDesk:

  • German startup with a clear focus on access rights visualization and management (“Identity Visibility + Intelligence Platform” or “Data Governance” according to Gartner). They already have larger corporations as customers
  • The solution provides an overview of which accounts (human/machine) have which access rights (create, write, read…) to which resources. Similar representation to Veza’s Authorization Graph (now ServiceNow)
  • For this purpose, IAM systems and CMDBs / asset inventories are connected
  • Access rights can then be adjusted directly via tickets or the connection to IGA where necessary
  • Of course, this has its limits with older on-prem systems, but it works very well with SaaS/IaaS. OT not covered so far, but is still to come
  • SaaS or on prem deployment possible
  • House number for license costs of ~10 EUR/year and MA

Cysmo:

  • Exposure management via Outside In Scans, a 100% subsidiary of PPI AG from Hamburg.Name to remind of seismograph
  • Work for approx. 50 cyber insurance companies and brokers, 2-3 million scans per year on behalf of customers, and that with only 25 employees
  • Everything was developed in Germany and operated in its own data centers
  • The focus is on reporting as few false positives as possible (of course means higher risk of not reporting DNS/mail misconfigurations or open ports or similar findings at all)
  • We now want to further expand our direct business with customers, also with regard to supply chain risk management within the framework of DORA
  • However, not yet certified as a PCI SSC Approved Scanning Solution, there may still be
  • Smallest package (5 companies) for ~1,500 EUR/a. Unlimited scans with licensing, also works continuously with alerting
  • Inquiries from managed security partners / resellers are also welcome

CyberVadis (Update):

  • French MSSP for Third Party Cyber Risk Management. Originally a spin-off of EcoVadis (the same for ESG risks)
  • Based on questionnaires (mapped to DORA, NIS-2, ISO…), evidence and workflows (e.g. review, escalation, risk acceptance, action tracking)
  • Own platform where suppliers can store standardized information + evidence and thus make it available to several customers at the same time (similar to Panorays etc.)
  • Suppliers can of course book the assessments as external audits and receive qualified assistance in planning measures, which is an advantage, especially for medium-sized companies
  • Feedback from suppliers and advice can be provided in German
  • Almost 100 corporate customers (many of course DORA regulated) and ~5000 verified suppliers
  • Further growth via NIS-2 expected, scaling via AI-supported evaluation of questionnaires also well conceivable
  • Costs are generally distributed among customers and verified suppliers
  • Are still looking for consulting/channel partners (e.g. for prioritizing high-risk suppliers) in the DACH region – please contact us if you are interested

As always, questions, suggestions, comments, experience reports, topic requests and also opposing opinions or corrections are welcome by email. Ditto for unsubscribing from the mailing list.

For the people who have received the market commentary for the first time: Here you can register if you are interested or look around in the archive.

Best regards,

Jannis Stemmann

Scroll to Top