What AI Kills

The fear came pre-written. For half a century the story of dangerous machines has been a story about bodies: a robot with a rifle, a drone choosing targets, a superintelligence deciding humanity is in the way. Those fears aren't baseless, and weapons built on AI are already being fielded. But the killing that is under way now, quietly and at enormous scale, is of a different kind. AI is killing systems.

By systems is meant the arrangements a society runs on without thinking about them: how a school knows a student learned, how a court knows evidence is real, how an employer knows an applicant wrote the letter, how an editor finds the good story in the pile, how a government hears from its citizens. None of these were designed with AI in mind. Most of them are dying of it, and very few are dying loudly.

Friction was load-bearing

The mechanism is almost always the same. A great many systems ration access by effort. Writing a thoughtful letter took an evening, so an employer could assume the letter meant something. Filing a complaint took time, so the complaints that arrived were mostly real. Submitting a story to a magazine took the labor of writing a story, so the pile stayed small enough to read.

No one designed that friction as a feature. It was simply the cost of doing things, and systems grew up around it the way a riverbank grows around a river's flow. The effort was doing quiet structural work: it filtered, it signaled, it held volume down to what humans on the receiving end could handle.

AI takes the cost of effort toward zero. The letter takes seconds. So do the complaint, the story, the application, the review, the comment, the legal brief. The friction disappears, and every system that was secretly relying on it starts to fail, not because anyone attacked it, but because the river it was built around changed course overnight.

The floods

The first sign is volume. In February 2023 the science fiction magazine *Clarkesworld* closed submissions entirely after being buried under machine-written stories. It had run an open door for years; the door stopped working in a matter of weeks. Journals, grant programs, job boards, and customer-complaint lines have reported the same pattern since: more of everything, a shrinking share of it human, and reviewers who can no longer tell which is which.

Public comment on government rules was already vulnerable before generative AI. New York's attorney general found that most of the roughly 22 million comments submitted in the 2017 federal net neutrality proceeding were fake. That was done with cruder tools. A system meant to let citizens be heard now faces a world where a single actor can speak as a million citizens, each voice distinct, each argument plausible. The comment period doesn't have to be abolished to die. It only has to become meaningless.

Proof of work

A second class of systems used a finished piece of work as evidence of a capacity. The take-home essay proved a student could think through a problem in writing. The cover letter proved an applicant could communicate. The brief proved a lawyer had read the cases, a point made vivid in 2023 when attorneys in *Mata v. Avianca* filed a brief citing court decisions that did not exist, invented by a chatbot and never checked.

When the work can be produced without the capacity, the work stops proving anything. The essay still gets written; it simply no longer carries information about the student. Schools, employers, and courts are left holding documents that look exactly as they always did and mean far less than they used to. A system that runs on signals keeps running for a while after its signals go dead, the way a cut flower keeps its color for days after it has stopped drinking.

Evidence

A third class relied on the difficulty of faking the world. A photograph was hard to counterfeit well, a voice recording harder, video harder still. Courts, journalism, insurance, and ordinary personal trust all leaned on that difficulty. Seeing was, if not believing, at least strong evidence.

That difficulty is gone. A convincing image of any event, a phone call in any voice, a video of anyone saying anything, can be made cheaply by nearly anyone. The damage runs in both directions. Fakes get believed, and, perhaps worse, real evidence gets dismissed, because every inconvenient recording can now be waved away as synthetic. The system that let a society settle what happened loses its footing, and without it, every dispute tends toward a contest of loyalties instead of facts.

Scarce expertise

A fourth class rested on expertise being rare. Whole professions and institutions were organized around the fact that only a credentialed few could draft a contract, read a scan, write working code, translate a document, or answer a technical question. Scarcity justified the credential, the credential justified the fee, and the fee sustained the institution.

AI doesn't make experts obsolete, but it makes a passable version of their output abundant, and abundance breaks the arrangement. A system organized around a bottleneck has no plan for the day the bottleneck opens. Some of those systems deserve to go. Many credentials were always more about restricting entry than ensuring quality. But a guild dismantled overnight doesn't leave a free market behind. It leaves confusion about who can be relied on for what.

What rushes in

Here is where the danger compounds. A dying system rarely just dies. It calls for a replacement, and the replacement almost always reaches for authority.

If no one can tell a human comment from a machine one, the answer on offer is identity verification: every voice tied to a verified name. If essays prove nothing, the answer is surveillance of the writing process: locked browsers, keystroke logging, proctoring cameras. If images can't be trusted, the answer is certified provenance, issued by a small number of approved companies. If expertise is suddenly abundant and unreliable, the answer is licensed AI, permitted only through sanctioned channels. Each of these may be reasonable on its own. Together they describe a much heavier society than the one that came before, one where trust is issued from above because it can no longer be found below.

This is the real risk hiding behind the robot story. AI kills systems that ran on friction, and the systems that grow back in their place are tempted to run on control.

What survives

Not every system is equally exposed. The ones dying fastest are the ones that trusted documents: a letter, a credential, a submission, a photograph, a form. As [Culture Part 1](/entries/culture-part-1) put it, a society that trusts documents is only as trustworthy as whoever issues them, and now anyone can issue them.

The systems that hold up are the ones that trusted conduct: the neighbor known for twenty years, the tradesman whose work can be walked through, the teacher who watched a student think out loud, the editor who has read a writer's work for a decade, the town meeting where people stand up in person and speak under their own names. Those arrangements are slow and local and don't scale, which is exactly why a machine that operates at scale can't counterfeit them. Presence, reputation, and repeated dealing were never efficient. They turn out to be durable.

So the defense against AI's quiet killing isn't only regulation, and it certainly isn't surrender. It is a deliberate return of trust to the places a machine can't reach: face to face, over time, in the particular. The systems that ran on friction are going. The question left is whether their replacements run on control from above or on relationship from below, and that choice belongs to people, not machines.