Risk Intelligence You Cannot Audit Is Just an Opinion With a Logo

Category: Enterprise Risk Tags: risk intelligence, country risk, third-party data, auditability, traceability, due diligence

An internal auditor once asked a team I was working with why a particular country was rated as elevated risk in their supplier assessment. The analyst pulled up the subscription portal, pointed at an amber square, and said the vendor rated it a 3. The auditor asked what drove the 3. Nobody knew. There was no methodology document available at that tier of subscription, no indicator breakdown, no date on the underlying judgment, just a square that had been amber for as long as anyone could remember. The finding that came out of it was not about the country. It was about the fact that a control decision had been made on the basis of a number nobody in the organization could explain.

I have seen versions of that conversation at four different firms. It is the central problem with most commercial risk intelligence: the product is a conclusion, and the reasoning is the part you are not buying. That is a perfectly reasonable business model. The analysis is the value, and giving it away would be giving away the company. But it creates a specific and underappreciated problem for the buyer, which is that a rating you cannot reconstruct is a rating you cannot defend.

The Three Questions a Rating Has to Survive

Any risk number that ends up influencing a decision will eventually be asked three things, and you want to know in advance which ones yours can answer.

The first is where did this come from. Not which vendor, which underlying observation. A country rating that traces to specific measurable things, whether that is a government advisory level, a governance index, or a count of recorded incidents, survives this question. A rating that traces to "our regional analysts" survives it only as far as the auditor's patience extends. The distinction matters most in regulated contexts, where DORA-style regimes increasingly expect you to evidence how third-party risk determinations were reached rather than simply that they were.

The second is when was this last true. Vendor ratings are often refreshed annually, which is entirely appropriate for structural political risk and entirely useless for anything operational. The trouble is that the interface rarely distinguishes between the two. An amber square looks identical whether it was set last week in response to something concrete or eighteen months ago and never revisited. If your register carries a date at all, it usually carries the date you copied the rating across, not the date the rating was formed.

What would change it is the third question, and the one that exposes the most. A rating you can act on has a threshold behind it, something that, if it moved, would move the rating. A rating without one cannot be monitored, only re-purchased. It is the same argument I have made about risk appetite statements that lack thresholds: without a stated trigger there is nothing to monitor, and so no control.

Opacity Is a Business Model

It would be easy and wrong to read this as vendors being careless. They are not. The major intelligence firms employ excellent analysts, and in a crisis their judgment is worth paying for. Their opacity is structural: if the methodology were fully public, the ratings would be reproducible, and reproducible ratings are not a subscription business. They are also selling into a market that mostly wants the conclusion, because the conclusion is what fits in a board pack.

The failure is on the buying side, and it is a failure of expectation-setting. Firms purchase a judgment product and then deploy it as if it were an evidence product. The rating goes into the vendor register, gets copied into the risk register, informs a control decision, and by the third hop the caveats have fallen off entirely. What began as "an experienced analyst's considered view as of last autumn" arrives at the risk committee as a fact.

I described the same failure mode in GRC as a knowledge management discipline in denial. Organizations rarely lack information. They strip its provenance at every handoff until nobody can say where anything came from.

What Traceable Actually Means

Traceability is not the same as being open source, and it is not a demand that every input be free. It means four specific properties, and they are all testable.

Every figure names its publisher. Not "market data" but the specific agency, index, or feed, such that a reader can go and look. Every figure carries the date it was published, distinct from the date you ingested it. Every derived score states its arithmetic: if a location scores 16, it should be visible that this is a four on likelihood times a four on impact, and what drove each. And where something is judgment rather than measurement, it says so in those words, rather than borrowing the visual language of data to look more certain than it is.

Judgment dressed as measurement is the most common violation, and usually an unintentional one. A hand-curated watchlist rendered in the same color scheme as a live seismic feed reads as equally empirical. It is not. Labeling the curated thing as curated costs nothing and preserves the credibility of everything next to it.

The Test I Would Apply to Any Provider

If you are evaluating a risk intelligence subscription, skip the questions about coverage and refresh rate and ask these instead. Can you show me the indicator breakdown behind a single country's rating? What date was that rating formed, as opposed to last displayed? What specific change would move it up a band? Can I export the underlying inputs, not just the conclusion? And if my regulator asks me to evidence this determination, what do you give me?

A good provider will have answers, and some of the answers will reasonably be "that is in a higher tier." That is fine. It is a commercial conversation, and at least you know what you are buying. What you are testing for is whether the answers exist at all. If the response to all five is that the analysts simply know, you have bought a consulting relationship with a map attached, which may well be worth the money, but should not be sitting in your register as though it were measurement.

Why I Built the Map the Way I Did

The PivotRisk risk intelligence map exists partly as a working argument for this position. Every signal on it names the agency that published it and links back to the record. Your browser fetches each feed directly from its source. There is no PivotRisk server in the path, no stored copy, and no interpretation between the publisher and the pin. When a location scores, the arithmetic is on screen: likelihood, impact, and what drove each.

Where something is not live, it says so. The US State Department publishes travel advisories without cross-origin headers, so no browser can read them directly; that layer is therefore a dated snapshot, and it is labeled "Snapshot" rather than "Live". AWS and Azure do not expose browser-readable incident feeds at all, so their regions are plotted from published documentation and their status is a link you click. The geopolitical watchlist is hand-written by me, so it is tagged "Editorial". Those labels make the map look less impressive than it could. They are also the point: a dashboard that shows green because a feed failed silently is worse than no dashboard, because it converts an absence of information into an assurance.

None of this makes the map better analysis than a firm with two hundred regional specialists. It plainly is not. What it is, is checkable, and for anything that ends up in a register, in a board pack, or in front of a regulator, checkable beats authoritative. You can disagree with a number you can trace. You can only accept or reject one you cannot.