Liberty Needs Paperwork

Pretrial risk assessments were sold as a fix for two real problems: judges guessing, and money bail punishing poverty. Replace gut feel with actuarial estimates. Reduce arbitrary disparities. On paper, a reasonable upgrade.

What courts got instead is a score. It arrives before the hearing, it fits in the corner of a screen, and it is very hard to argue with.

If a system can put someone in a cage, that system must be easier to inspect, challenge, and appeal than most institutions are comfortable with.

Nobody disagrees with that in the abstract. So look at how a score actually moves through a courtroom. It moves like a credit score: a number you inherit, whose arithmetic you cannot see, that you are expected to accept. A credit score costs you an interest rate. This one costs you the next three months of your life.

The question is: can the person push back?

a number that pretends to be a reason Link to heading

Most tools do something narrower than the pitch suggests. The Public Safety Assessment (PSA), for example, estimates the likelihood of three outcomes while on pretrial release: failure to appear, new criminal arrest, and new violent criminal arrest. Judges consider this along with other information. 1

That framing is honest. It says: this is a forecast, not a verdict.

Then the real world happens. Dockets are crowded. Time is short. A number looks like relief, and relief is how authority sneaks in.

If you want to understand algorithmic judging, don’t imagine a robot with a gavel. Imagine something more boring, more plausible, and more dangerous: a score that becomes hard to disagree with, even when nobody can explain it.

A score can be fast. A reason is slower. A remedy is slower than that.

first switchback: prediction isn’t judgment Link to heading

A model that outputs “8/10” is doing a familiar machine thing: mapping inputs to outputs. Functionalism says a mental state is whatever plays the right functional role, so if the system behaves like judgment, it counts as judgment.

Applied to bail, that reasoning fails, and it fails for a reason that has nothing to do with philosophy of mind. Bail is state power.

A risk assessment can estimate probabilities. It cannot tell you what you are allowed to do to a person on the basis of those probabilities. That step is judgment, and judgment is a legal and moral act.

Judgment carries reasons that can be argued with. It sits inside procedures that recognize fallibility. It assigns responsibility when harm occurs.

A model can’t apologize. It can’t make restitution. It can’t be disbarred or voted out. It can’t be cross-examined in the way due process expects.

So when we ask “can AI make fair ethical decisions,” we’re already slipping. We’re asking a prediction engine to inherit moral authority because it outputs a number in a font we trust.

Use AI to estimate. Use law and human responsibility to decide.

And there is a better use for these tools than scoring people. A model fit on a hundred thousand case files can surface things worth knowing: which factors usually move the outcome, where this case sits relative to others, which assumption the court is making without saying so out loud. That is a lens. A lens sharpens the judge’s attention. A single score replaces it.

Build the lens. The score is the lazy version of the same information.

second switchback: “fair” isn’t one setting Link to heading

Even if the model is only estimating risk, we still have to ask which kind of “fair” its builders optimized for. Here’s where the story gets uncomfortable for anyone who wants a single metric.

Fairness has no shortage of definitions. It has a surplus, and they conflict.

Kleinberg, Mullainathan, and Raghavan formalized a core impossibility: except in constrained special cases, you can’t satisfy multiple intuitive fairness conditions simultaneously. 6 Chouldechova makes the point directly for recidivism prediction: when prevalence differs across groups, several fairness criteria can’t all hold at once. 8

Street-level translation:

You can tune a system so a given score means the same thing across groups (calibration). Or you can tune it so the system makes similar mistakes across groups (equalized error rates). When base rates differ, you usually don’t get both.

So when someone says “the model is fair,” the only honest response is:

Which fairness, traded for which other, and who holds the receipt?

Bring that back into the courtroom. If the defendant sees “8/10,” do they also see which outcome the model predicts, the error rates for people like them, and the fairness tradeoffs chosen by the jurisdiction?

Most of the time, no. They see a number and the posture changes in the room.

A score is compressed policy.

third switchback: the data is a footprint of enforcement Link to heading

Risk models don’t learn from “crime.” They learn from records: arrests, convictions, failures to appear, supervision violations. Those records are a footprint left by enforcement choices, prosecutorial discretion, plea bargaining, policing intensity, and resource disparities.

If you’ve ever debugged a production system, you know the feeling: the logs tell you what happened, but they also tell you where the logging is broken.

Prediction systems inherit that problem. They can look objective while replaying historical attention.

This is one reason the COMPAS controversy still matters. ProPublica’s “Machine Bias” investigation argued that COMPAS produced racially disparate error patterns. 3 Flores, Bechtel, and Lowenkamp answered that ProPublica had picked the wrong statistic. 15

They have a point about the statistic. It does not get COMPAS off the hook. The statistical quarrel is about which error rate deserves the headline. The governance failure is that a defendant can interrogate neither number. Winning the methods argument still leaves you holding a score you cannot contest.

what the number does to humans Link to heading

Picture the hearing. It is 9:07 in the morning and the docket has thirty more names on it. The defendant is still talking, still trying to be understood. In the corner of the judge’s screen sits the model’s output: risk score 8 out of 10. Tidy, confident, ready for reuse.

Nobody has to say “the model decides.” The model advises, and the advice becomes the decision. Humans anchor on numbers. Humans defer to instruments. On a crowded docket the easiest path becomes the default path, and the tool is the easiest path.

Judges will tell you the score is one input among many. Watch what happens to a hearing when the score is high.

When a score is in the room, it shifts three things:

Attention narrows. The hearing becomes a search for reasons to justify the score rather than a full investigation of the person.

Disagreement becomes expensive. A judge who departs from the score may fear blame later. A judge who follows it lets responsibility diffuse into “the tool.”

The defendant becomes a profile. The room starts to treat a feature vector as a person, because the feature vector speaks in numbers and the person speaks in sentences.

the legal system has met this problem and flinched Link to heading

In State v. Loomis, the Wisconsin Supreme Court held that a trial court’s use of a COMPAS risk assessment at sentencing did not violate due process, while also emphasizing limitations and warnings about how courts need to use such tools. 5

If you read Loomis as “courts are fine with black boxes,” you’re missing the flinch. The opinion carries all that caution language precisely because the legitimacy problem was obvious to the judges writing it.

GDPR Article 22 encodes the same instinct more explicitly: a person has the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects, with safeguards including the right to obtain human intervention, express a view, and contest the decision. 13

the strict rule Link to heading

Here’s the rule, stated plainly:

This is a stability requirement. Systems that can’t be contested eventually lose legitimacy, and legitimacy is what keeps compliance from hardening into force.

Which means a pretrial score should be harder to use than a credit score. Right now it is easier. A lender that turns you down has to name the main factors behind the decision and point you at a way to dispute what is in the file. A courtroom hands over a number and a change in posture.

Think of it like building a bridge on a public trail. The heavier the load, the more you overbuild the supports. This is where the expense shows up, and it shows up on purpose. You don’t get to say “it usually holds.”

what strict looks like when you stop hand-waving Link to heading

If a risk tool is in the loop, the loop needs mechanical guarantees. Some technical, some procedural, and all of them cost money, which is what a real safeguard costs.

Transparency. Disclose the target: which outcome the model predicts, exactly. “New arrest” isn’t “new crime”; don’t hide that distinction behind a label. Disclose the inputs: if prior arrests matter, say so; if age matters, say so; if the system uses proxies for protected attributes, name them. Disclose performance over time, not once. 12 And prefer simpler models when they perform similarly; Dressel and Farid found that a simple linear predictor using just a couple of features can match a widely used commercial tool. 9 If you can get similar predictive performance with far more transparency, you don’t earn ethical points for complexity.

Contestability. The defendant sees what the system “thinks” it knows: the score and the underlying factors. The defendant can correct errors in the data. The defense can challenge the model’s validity; if the model is proprietary and can’t be meaningfully interrogated, that is a procedural defect. And the judge must write a reason that stands without the score. If the score disappeared, the justification stays coherent. That’s how you prevent “the score ate the hearing.”

Governance. Independent audits, not vendor assurances. Partnership on AI documented serious shortcomings in these tools; their partner consensus view was that current tools must not automate pretrial detention decisions. 10 11 Ongoing monitoring with public reporting; populations change, enforcement changes, policies change. Clear limits on how the score is used; if the score is “advisory,” make departures ordinary and defensible, not rare and risky.

Human-in-the-loop that isn’t theater. A human who clicks “approve” is a decorative checkbox. Real human intervention requires authority to override without penalty, time to consider evidence, and a culture that treats disagreement with the model as a professional act, not a liability event.

Otherwise the model is the judge and the human is the printer.

the uncomfortable twist Link to heading

Even if you can pick a fairness definition and optimize it, the “cost” of fairness can show up as more detention, not less. Corbett-Davies and Goel discuss how fairness constraints interact with decision thresholds and can impose costs. 14

This breaks a comforting story. Fairness work doesn’t automatically push toward more humane outcomes. Sometimes it pushes toward a different distribution of harm.

So the ethical question is who paid, and whether they can contest the bill.

can AI make fair ethical decisions? Link to heading

Back to that hearing. The number still sits there: 8 out of 10. The person still stands at the table.

The honest answer:

AI can help estimate. AI cannot, on its own, be a fair ethical decider. Fairness here includes contestability, explanation, and responsibility. Those live in institutions.

Keep the model as a lens and it can make a hearing sharper. Let it shrink to a single score and it makes the hearing shorter.

If we insist on involving AI, then we need to make the system legible enough that the defendant can fight it, and structured enough that the judge can’t hide behind it.

The test for algorithmic judging is harsher than calibration:

When the number is wrong, can the person make it stop being wrong in time to matter?