The World Inside the Answer: A Standard for Inspecting AI Assumptions

Care management programs give patients with complex health needs additional help organizing their treatment, often through a nurse who coordinates appointments and follows their care. With limited staff available, health systems need to decide which patients to offer a place, and some use algorithmic risk scores to help make that choice. 1

When physician Ziad Obermeyer and his colleagues compared one widely used system’s scores with patients’ health, they found a discrepancy: Black patients were sicker than White patients assigned the same score. Their findings, published in 2019, called into question the use of those scores to identify who needed extra support. 2

The explanation lay in what the algorithm had learned to predict: future healthcare spending. Using spending as a stand-in for need assumed that people received treatment in proportion to their illness. Yet Black patients generated lower spending at comparable levels of illness, partly reflecting unequal access to care. Using that forecast to judge medical need therefore understated their case for additional support. 2

For people designing with AI, the case exposes a limitation of explaining how an answer was calculated. The explanation also needs to make the chosen measure available for scrutiny, along with the evidence that connects it to the decision. Otherwise, someone relying on the score has little basis for questioning what it has been taken to mean.

Whose common sense? Link to heading

The word risk helps this substitution pass unnoticed. A care team reading a risk score has reason to think about worsening health; a model forecasting expenditure gives the same word a financial meaning. Within a professional group, familiar language can make the connection between the two feel settled before anyone examines it.

To rely on the score, a care team needs a way to check the meaning assigned to it against the evidence supporting that use. 3 That check begins with the information the system received and extends to the limits on what anyone is entitled to infer from its output.

Make the assumptions available for inspection Link to heading

Frictology’s minimum standard for consequential AI makes four parts of that assessment accessible.

Source horizon Link to heading

What information was within reach?

• Identify the information available for the answer, how current it is and known gaps relevant to the decision.
• Give people access to the underlying records and a route to correct errors.
• Distinguish documented inputs from claims about their influence that the system cannot verify.

Assumptions Link to heading

What has been chosen or inferred?

• Identify the objectives, defaults, simplifications, proxies and constraints shaping the answer, including who selected them.
• Distinguish factual inputs from estimates, chosen priorities and mandatory rules, so people can identify what needs correction, comparison or review.

Uncertainty Link to heading

Where does the answer lose its footing?

• Show where evidence is incomplete, which alternatives remain plausible and how changes in assumptions affect the answer.
• Distinguish tested sensitivity from untested comparisons, and identify the settings in which the proposed use has been evaluated.
• Keep gaps in knowledge separate from disagreements about which outcomes matter.

Handoff points Link to heading

When must the decision change hands or stop?

• Specify the conditions that require human review, restrict reliance on an answer or stop automated action.
• Name the responsible role and provide the evidence, time and authority needed to resolve the issue.
• Give affected people a usable route to reach the reviewer.

When several answers fit Link to heading

With those conditions visible, a person can assess an answer without assuming it is the only reasonable one. The same evidence and constraints can support several courses of action, leaving a trade-off that the disclosures help identify but cannot settle on their own.

A worked comparison. Consider a travel assistant comparing two trains for a morning appointment. Both fit the stated budget and are scheduled to arrive before it begins:

The assistant has timetable and fare information, but no delay data from which to estimate the probability of arriving on time. It can still make the remaining choice clear: “For this appointment, which matters more: saving $15 or adding thirty minutes of arrival buffer?”

Someone heading to a job interview brings different concerns from someone meeting a friend who can wait. A tight budget or an unfamiliar venue adds further context that the timetable cannot supply. Asking a focused question lets the person introduce that knowledge, change the recommendation or request another route before authorizing a booking.

Which assumptions become controls? Link to heading

Make assumptions available for intervention when they materially affect the outcome, the burdens imposed or the legitimacy of the decision. Match the response to the issue: correction, comparison, adjustment, policy review or a hold on action. Each control needs an authorized owner and a route for affected people to challenge its use, including the underlying objective and any consequential conditions omitted from the available options.

A menu offering three ways to rank people still assumes that ranking is the task. Someone facing that menu needs a route to question the resource limit or the decision to rank at all.

Such limits can persist in software after an institution’s budget or workload has changed. In Generations, William Strauss and Neil Howe proposed recurring patterns in American history, linking generations’ shared experiences to shifts in social expectations. 4 Their account directs attention to how inherited expectations continue to organize decisions as circumstances change.

A team examining that resource limit can use AI simulations to compare the consequences of retaining it, revising it or removing it over time. The people comparing these versions have to judge whether their assumptions about resources and human responses fit independent evidence, whose interests each version serves and what fresh evidence demands revision. The selection of versions also needs scrutiny: convenient scenarios and selective outcome measures can make a preferred result appear inevitable.

Give the explanation something to stand on Link to heading

That comparison is useful only to the extent that its account of the evidence and assumptions is reliable. A language model’s explanation needs scrutiny of its own: experiments with reasoning models have found that their written accounts can omit hints that influenced their answers. 5

Verification Link to heading

Does the account hold, and can people actually act on it?

• Build disclosures from recorded inputs, configured objectives, actual evaluations and enforced permissions, labeling generated interpretations as such.
• Check comparisons against their stated inputs and assumptions.
• Where an important claim remains unverified, limit the authority delegated to the system accordingly.
• Test whether people can identify missing conditions, compare alternatives and use the promised routes to change an outcome.
• Verify that review requests reach someone authorized to act and that required holds prevent execution.

The institution remains responsible for maintaining these arrangements.

The healthcare investigation revealed how a forecast of spending came to guide a decision about need. Making that connection inspectable gives people a chance to question the measure before relying on it, while meaningful alternatives let them exercise judgment about what follows. As decisions gain reach and consequence, discernment becomes our most vital form of cognition.