Buyer’s Guide for Third Party Cyber Risk Ratings Tools

You can download the buyer’s guide at the end of this article.

Supply chain risk management for large organizations or operations of critical infrastructure increasingly include cyber risk monitoring. From the suppliers’ perspective, having a shared repository that their customers can access, helps to avoid repetitively answering the same questions about their security controls and certifications (e.g., “Do you have IT emergency response plans in place and perform exercises at least yearly?”).

In general, there are two different approaches for security monitoring: The more common is to send out questionnaires in one way or the other. Automatic scanning solutions, on the other hand, are becoming more widely used – even if only a small part of actual cyber risk is covered.

But what is the best tool to support these efforts? Most solutions for third party / vendor risk management (VRM, TPRM) or governance, risk and compliance (GRC) as well as full procure-to-pay suites now offer at least some functions for vendor cyber risk ratings. External attack surface management (EASM) tools also market their ability to scan suppliers’ and partners’ exposed assets for vulnerabilities or misconfigurations.

Potential vendors (Examples):

Vendor	Gartner: Market Guide for third party risk management systems, 2022	Gartner Peer Insights ‘Voice of the Customer’: IT Vendor Risk Management Tools, 2022	Forrester: Cybersecurity Risk Rating Platforms, 2021
Allgress	x	x
Aravo	x	x
RSA Archer (RiskRecon as cyber risk engine)	x	x
Certa	x
Coupa	x
Diligent (former Galvanize)	x	x
Security Scorecard		x	x
Black Kite		x	x
Bitsight		x	x
Prevalent	x	x	x
BlueVoyant
Riskledger
Cybervadis
RiskRecon (MasterCard)		x	x
LogicGate	x
LogicManager	x
Locaterisk
Red Maple FractalScan
Panorays	x		x
Rapid7 Intsights
ServiceNow	x	x
SureCloud	x
OneTrust	x	x
ProcessBolt Threatscape	x
CyberGRX / ProcessUnity	x	x
Palo Alto (Cortex Xpanse)
Darkscope CIQ360	x
DGC Cyberscan
SureCloud	x
Upguard	x	x	x
Venminder	x	x
Whistic	x

Scope and accuracy of external scans

As part of a recent RfP, we asked several vendors of third-party cyber risk scoring solutions to scan the same test domain and provide us their results for comparison purposes. As visible in the table below, the example findings differ significantly. The test domain was managed by CyberCompare – obviously, nobody is perfect, and the scan results helped us in closing some security gaps.

Of course, the relevance of each single finding in terms of security posture or data privacy is also very different and will vary in between customers and exposed assets. There could also be differences in findings because the timing of the scans was not identical. However, the categories shown might help in defining detailed requirements or evaluating vendor solutions.

Issues (examples)	Area	Bitsight	Black Kite	Cybervadis	LocateRisk	Red Maple	RiskLedger	RiskRecon	Security Scorecard	Valid finding (at time of scan)
DNSSEC not supported	DNS	x		x	x					yes
No CAA Entry found	DNS				x					yes
No DANE entry found	DNS				x					yes
BIMI not supported	DNS			x						yes
SPF record missing	DNS/Mail	x	x	x	x	x			x	yes
DMARC mail record missing	DNS/Mail			x	x	x				yes
Domain Squatting	Domain Squatting	x								yes
Open SSH port (website)	Network				x	x	x			yes
Old/Weak TLS Versions accpted	Network		x		x					yes
Nginx CVEs	Patching			x		x	x			yes
OpenSSH CVEs	Patching
CSP Not Set	WebApp			x	x					yes
X-Frame-Options header not set	WebApp	x		x	x			x	x	yes
Referrer Policy header missing	WebApp				x					yes
HSTS-Header not Set	WebApp		x		x	x			x	yes
Insecure HTTPS Redirect	WebApp								x	no
X-XSS Protection not enabled	WebApp			x	x	x		x	x	yes
Advertisment of Nginx version	WebApp		x			x				yes
Weak CBC Ciphers offered	WebApp		x							yes
Inline CSS	WebApp		x							no
HPKP not enabled	WebApp		x							yes

Criteria for shortlisting and selection

Choosing the right tool should follow a thoughtful decision on the vendor cyber risk management process. No matter which solution is chosen, somebody in your own organization has to review the vendor feedback, work with findings and follow-up on actions (e.g., deal with missing vendor feedback, coordinate with the purchasing department), unless the whole process is outsourced as a managed service. This in itself will cause a substantial workload for your own team or cost for an external service provider. Also, while many vendors showcase their findings for AWS, Microsoft or Google (as they are some of the largest suppliers for most organizations today), how useful is this information actually?

For many companies, supply chain cyber risk is actually a lower risk than ESG or other compliance related risks. Therefore, a very pragmatic approach (e.g., only tracking very few vendors or only asking very few high-level questions) might be sensible.

For selection of the right tools, here are some typical requirements and questions:

• Are either questionnaires or external scans sufficient, or are both required to manage third party cyber risks?
• Can existing software (e.g., GRC) be integrated or a TPRM cyber risk module be added? For example, several external scan tools integrate natively with questionnaire-based compliance monitoring tools.
• How many suppliers in which sectors are already onboarded on the platform? If there is a large overlap with your supplier base, you might save your suppliers and yourself lots of effort. Also, the benchmarking data might be much more useful if the regional industries covered are relevant for you.
• Are assessment questionnaires (e.g., for ISO 27x, IEC 62443, NIST CSF, GDPR) available off the shelf and updated automatically whenever there are new/changes to these standards?
• Can questionnaires be customized?
• What other (non-cybersecurity) risks can be monitored?
• Supported languages
• How long does it take to get results for a new supplier?
• Is there a managed service available for interactions with suppliers?
• What is the standard frequency to update external scans (yearly, monthly, daily)?
• What security controls are checked in the outside-in scans, and how reliable are the results?
• Are cyber risk ratings and the false positive rate (both for discovered assets and for missed security controls) calculated/tracked in a transparent way, and what is the process to deal with false positives?
• Are suppliers in control of the data they share? E.g., can they choose (or decline) to share the data also with other customers?
• Is dark web threat intelligence used for ratings and recommendations?
• Licensing costs for various scenarios (e.g., depending on number of scored vendors, number of scans)

CyberCompare is buying security tools and services every day on behalf of our customers. We are happy to advise you in your project to make sure you get the best security for your budget.

Download the Buyer’s Guide.pdf