The Algorithm Will Judge the Living and the Dead
The older theology placed judgment at the end. Ours places it at every transaction. The court is the dataset. The verdict arrives before the trial begins.
There is a moment in the Apostles' Creed, recited every Sunday in churches that still recite anything, in which the speaker professes belief in a Christ "who will come again to judge the living and the dead." It is a strange line. It places judgment at the end of time, and it places it on a single day, and it places it before a single judge, and it has been recited by people for nearly two thousand years as the long horizon against which their small lives were lived.
I want to suggest, in this short essay, that the structure of that horizon has been replaced — quietly, technically, almost without argument — by a different structure. Judgment is no longer eschatological. It is continuous. It is no longer reserved for the end of time. It is rendered at every transaction. And it is no longer pronounced by a single judge against whom appeal is at least theologically conceivable. It is pronounced by a system from which there is no appeal at all.
Small last days
Consider, for a moment, the credit application. Or the insurance quote. Or the resume filter, the parole prediction, the rental check, the visa screen, the moderation queue. Each of these is, structurally, a tiny day of judgment. A petitioner approaches a tribunal. The tribunal consults a record. The tribunal renders a verdict. The verdict is binding, often unappealable, frequently unexplained.
What is striking about these small last days is not that they exist. They have always existed, in some form. Bureaucracies have judged petitioners for as long as there have been bureaucracies. What is new is the scale and the opacity. Where once a clerk would render a verdict that another clerk could overturn, the verdict is now rendered by a model whose internal reasoning is, in a strict mathematical sense, not available to the clerk who deploys it. The clerk knows the output. The clerk does not know the path.
The court is the dataset. The judge is the loss curve. The defense, when it comes, must be made in a language the system was never trained to hear. — Field note, Vol. I
Judgment without hearing
The Anglo-American legal tradition holds, in principle, that no person may be deprived of life, liberty, or property without due process — that judgment must be preceded by a hearing, in which the accused is permitted to know the charge and respond to it. This is not a procedural nicety. It is the moral architecture of the entire system. To deny a hearing is, in this tradition, to render the verdict illegitimate by construction.
Algorithmic judgment, in nearly every commercial deployment, denies the hearing. The applicant for credit does not know which features of their record were weighted. The job seeker filtered out by the resume model does not know which signal failed. The parolee predicted to reoffend does not know which past — their own or someone else's, statistically similar — was being read in their stead.
This is not, in any defensible sense, due process. It is judgment without hearing, conducted at scale, against people who do not know they are being tried.
It is judgment without hearing, conducted at scale, against people who do not know they are being tried.
The verdict before the trial
There is a more troubling feature of cybernetic judgment, which I will state plainly. In algorithmic systems trained on historical outcomes, the verdict precedes the trial in a literal sense. The model has already decided, before you fill in the form, what people who look like you tend to be. Your specific inputs only refine an estimate that was, in essence, computed before you arrived.
The medievals, who knew something about predestination, struggled with this idea for centuries. Calvin's God knew, before time, who would be saved and who would not. The believer's task was, in some sense, only to discover which they had always been. The horror of this doctrine — and it is a horror — is that it removes the moral content of the present. Nothing you do now can change a verdict that was rendered before.
Cybernetic judgment is a secular Calvinism. The model has been trained. The weights are set. The verdict on the next applicant is, in a non-trivial sense, already determined by the dataset on which the model was trained. The applicant's task is only to discover what kind of applicant they have already been computed to be.
What a defense would look like
If we are going to live with cybernetic judgment — and we are; the systems are not being uninstalled — then we owe ourselves, at minimum, the architecture of a defense. This is not yet a complete account. It is the beginning of one.
The right to be told.
If a model has rendered a verdict against you, you should be told. Not in the abstract. In the particular. With the features. With the threshold. With the cohort.
The right to be heard.
If a model has rendered a verdict, you should be permitted to respond — in your own language, against the specific features that produced the verdict. The response should be reviewed by a human being whose hands are empowered to reverse the model.
The right to be forgotten.
The dataset on which the verdict rests should not be eternal. There should be — there used to be, in many older legal systems — a statute of limitations on the inputs. After enough time, the past should not be allowed to keep computing your present.
The right to a fair tribunal.
The model that judges you should not be the same model that benefits from judging you in a particular direction. This is the oldest principle of any tribunal: the judge cannot be a party to the case. We violate it routinely.
The Apostles' Creed concludes its judgment line with a startling promise: that the dead, too, will be judged. This always struck me, as a child, as the more unsettling half of the sentence. It was not enough that the living should be sorted; the dead, too, would be sorted, retroactively, at the end of time.
I think of this line now whenever I read about training data scraped from old archives, and about cohort effects that read your dead grandparents' decisions into your present rate. The algorithm has begun to judge the dead. It does so by computing them into us. We are not yet sure what to say about that. This essay is the beginning of saying.