Who goes to jail when the algorithm is wrong?

The arithmetic of an arrest Link to heading

In January 2020, Detroit police arrested Robert Williams at his home. A facial recognition system had pointed at him. The match was wrong. The arrest was real. He spent thirty hours in a cell before anyone admitted the error. [1]

A face becomes a number. A number becomes a lead. A lead becomes a cuff.

That sequence is the signature of our moment: a system generates a confident output, an institution treats it as warrant, and a person becomes collateral damage. The harm requires no malice. It runs on something more dangerous. Professional normalcy.

Builders like to say bias is a model quality issue. True, in the way a collapsed bridge is a concrete quality issue. Technically accurate. Entirely beside the point.

Bias in AI is a moral event. It allocates suspicion, exclusion, and sometimes violence through ordinary engineering choices: what you measure, what you optimize, what you ship. If your system can take someone’s liberty, it deserves more skepticism than a human witness. Probably more.

The clean lie Link to heading

Let us name it plainly. AI systems are not neutral. They do not float above culture. They are trained inside it.

They learn from data shaped by institutions, norms, and uneven histories. They get tuned by incentive structures that reward engagement over accuracy, speed over care. They ship into workflows where humans are penalized for doubt.

Even an immaculate dataset still encodes a choice about which errors are acceptable. That choice is moral. It governs people whether or not you use the word.

Facial recognition is the blunt case. The input is a face. The output is a claim about identity. Buolamwini and Gebru’s “Gender Shades” study found commercial systems misclassified darker-skinned women at rates up to 34.7%, while lighter-skinned men sailed through. 2 NIST’s vendor tests documented the pattern across dozens of algorithms. 3

Here is the builder translation: you cannot claim “the model works” without asking for whom, under what conditions, at what cost.

And you cannot claim “we only provide a tool” if you know the tool will be treated as a warrant generator by overworked institutions looking for permission to act.

Three doors Link to heading

Bias does not arrive as a single bug in a single layer. It walks in through three doors. Each looks reasonable at the time.

Labels and proxies. Train on a label downstream of unequal treatment and the model learns the inequality. It needs no hatred. It needs only data.

Obermeyer and colleagues studied a widely used healthcare algorithm and found it used cost as a proxy for need. The result: Black patients had to be considerably sicker than white patients to qualify for the same care programs. Fixing the disparity would have increased Black patient enrollment from 17.7% to 46.5%. 4

That is what a proxy can do. It takes something that looks neutral on a dashboard and turns it into a quiet rationing system. The moral mistake is not bad math. It is mistaking what is easy to measure for what is right to serve.

Optimization. Every objective function is a statement about priorities. You are telling the system what to care about. “Maximize accuracy” sounds clean until you ask: accuracy for whom. “Reduce false positives” sounds careful until you ask: which ones.

This is where philosophy stops being a book club and becomes a design review.

A utilitarian says minimize total harm. Then you notice total harm hides distribution. One group can absorb the errors while the average looks pristine. A Kantian says do not treat people as mere means. If your model concentrates humiliation on a subgroup to improve the aggregate, you are using some people as the price of a nicer curve.

Rawls offers a practical test. Imagine you do not know which user you will be. Would you accept this error distribution if you had to be born into it?

A builder might roll their eyes. Then a regulator arrives and calls the same idea “equal protection.” Same problem. Different letterhead.

Deployment. This is where the Robert Williams case becomes the spine rather than the ornament.

The arrest was not caused by a confusion matrix. It was caused by a chain of decisions about how much authority a model output deserves. The system did not handcuff anyone. People did, guided by a workflow that gave the match a weight it never earned.

Automation bias is the technical term: the tendency to over-rely on automated recommendations, especially under time pressure. 9 If your deployment makes it costly to doubt the model, you are not keeping a human in the loop. You are placing a human next to the loop as legal insulation.

The impossibility you cannot optimize away Link to heading

The COMPAS recidivism debate is worth the detour because it breaks the spell of the single metric.

ProPublica found the system made errors at similar overall rates for Black and white defendants, but the type of error differed: higher false positive rates for Black defendants. 5 The response was predictable. “We can fix it by making it fair.”

Fair by what definition?

Chouldechova proved the conflict is not a bug. When base rates differ between groups, you cannot simultaneously satisfy multiple fairness criteria. Equalize one error type and you break calibration. Preserve calibration and you preserve disparate impact. 6 There is no technical escape hatch.

That is not a reason to give up. It is a reason to stop pretending you can solve a justice problem by tuning a score.

The moral move is to choose your tradeoffs deliberately, document who approved them, and accept that you are making a governance decision. Not a hyperparameter decision.

Feedback loops, or why a model is never just a model Link to heading

Deployed AI is not an object. It is a process. It changes behavior, which changes data, which changes retraining, which changes behavior again.

Predictive policing is the textbook example. Send officers to the same neighborhoods based on “discovered” crime and you will discover more crime there, which justifies sending officers again, regardless of underlying rates. 7 The map becomes the territory. The prediction manufactures its own confirmation.

Ethics done as a one-time audit is like checking the weather once in April and assuming the mountain will behave all year. If your system shapes its own training data, you need continuous oversight. Otherwise you are building compounding injustice with excellent uptime.

Language models: the sneakier case Link to heading

Facial recognition is the obvious failure mode. Language models are the subtle one.

With LLMs, bias shows up as a default stance, a framing, a tone that feels “reasonable” because it matches the values embedded in training data and reinforcement. RLHF (Reinforcement Learning from Human Feedback) is a powerful alignment method and also a mechanism for transferring worldview. 12 Your model learns what your raters reward. If your raters are culturally narrow, your model becomes culturally narrow in a calm, helpful voice.

Recent work has measured political slant in LLM outputs. 13 Other studies show how fine-tuning shifts expressed positions. 14 None of this is shocking once you admit what the training signal actually is.

The uncomfortable takeaway: you can launder a worldview into the style of helpfulness. And because LLM outputs arrive as language, they feel like reasons. They feel like judgment. They feel like quiet moral authority.

A Buddhist non-self lens is useful here. The model is not a stable agent with beliefs. It is a process producing outputs conditioned on data and reward. Treating it like a moral person is how responsibility slips out the back door.

Law arrives late Link to heading

A strange thing happened over the past decade. Software became a form of public administration.

A loan decision is an administrative act for a household. A hiring filter is an administrative act for a career. A face match used by police is an administrative act for liberty. We still regulate these like optional product features. They are not. They are civic machinery now.

The EU’s AI Act sets a tiered framework for high-risk systems. 15 The U.S. Blueprint for an AI Bill of Rights signals the direction without binding anyone. 17 Cities like San Francisco have banned municipal use of facial recognition outright. 18

These moves are patchy and imperfect. The direction is not. We are starting to treat certain deployments as governance, not just innovation.

Builders should not wait for statutes to mandate basic maturity. If your product can deny someone a job, a home, care, or freedom, you are already operating in the realm of rights and due process, whether or not you like the vocabulary.

Back to Detroit Link to heading

The Robert Williams arrest should not be filed under “facial recognition issues.” It belongs under institutional epistemology: how an institution decides what counts as knowledge.

A model output is not a witness. It cannot be cross-examined. It does not feel shame when wrong. It cannot be punished, rehabilitated, or made to apologize. It cannot care.

The builder’s job is not to make the model perfect. That dream is comfortable and false. The job is to prevent institutions from treating imperfect inference as justified certainty.

This requires changes more boring than slogans:

Make error rates legible by subgroup for the actual deployment context, not a flattering benchmark. Treat proxies as moral choices; optimize for cost and you will get cost, not need. 4 Document which fairness criterion you selected and who signed for the tradeoff. 6 Build contestability that operates at human speed, with real authority to override. Design workflows where “the model said so” is never sufficient grounds for high-stakes action. 1 Protect the humans who override the system from professional retaliation. Otherwise, oversight is theater.

If this sounds like a lot, it is. That is what happens when prediction becomes policy.

There is a line worth returning to. Your model is a tool. Your deployment is a claim about what kind of society you are building.

Choose accordingly.