“AI models collapse when trained on recursively generated data.” This was the title of an unsettling paper published in Nature in 2024.
The idea is very simple.
AI models learn from text written by humans, on the internet and from digitized books. But a lot of the text online is now written by AI.
So what happens when the next generation of models learns from the output of the last one?
They get worse. In a specific way.
The weirdest parts disappear first
When a model learns from data, it tends to get the most common patterns right. Ideas that are common, ways in which people phrase things and tone in say Reddit or Stack Overflow.
But, understandably, models are worse at learning less common stuff, since it’s, well, less common in the training data.
So when it generates text, the rare stuff shows up even less. The next model, trained on that text, sees it even less. After a few rounds of this, it’s gone.
Statisticians call these the tails of the distribution. The paper found the tails vanish first.
If this keeps going, even the middle starts to shrink. The model’s outputs get more and more alike, repeating blurry versions of the most common patterns, with less and less variety.
The jackrabbit example
In the paper, researchers fine-tuned a model over and over on its own output. They gave the model a prompt about medieval church architecture, and by the ninth generation the model was talking nonsense about jackrabbits with differently colored tails: black, white, blue, red, and yellow-tailed jackrabbits, specifically. Churches were gone.
This was done with OPT-125m, a model that’s tiny compared to today’s frontier models. But the researchers found the same pattern in much simpler statistical models too, which suggests it’s not a quirk of one small model. It’s how learning from your own output works.
Why this happens
Three types of errors stack on top of each other.
Sampling: A model sees a finite amount of data. Rare things may not show up enough to be “learned.”
Limited capacity: Models can’t represent everything perfectly. They have to simplify things, somehow, and they do that by leaving out the unusual.
Imperfect learning: Training itself is approximate. Every model gets a few things slightly wrong.
In one generation, these errors are tiny. Across many generations, they compound. Each model inherits the previous model’s blind spots and adds its own issues.
A lesson from natural languages
When a language is passed down through fewer and fewer speakers, the unusual idioms, the irregular verbs, or the regional words go first. The language becomes in a way simpler and more uniform.
Is AI doomed to eat itself?
Maybe, but probably not.
Follow-up research found that collapse mostly happens when synthetic data fully replaces human data. When new AI-generated data is added on top of the original human data, instead of replacing it, the problem is smaller.
And synthetic data can be filtered and checked, by humans, by tests, or by other models, though checking at scale isn’t easy. The thing is avoiding blindly recycled slop generation after generation. And that’s really a data curation thing.
What this means
Human-made data just got more valuable.
The messy, weird things people still say daily on the internet (on Reddit, in comment sections, on endless arguments on X...) can help keep the new models connected to reality. The rare perspectives are what matter most, since they’re the first to disappear.
Dan Berges is the founder and managing director of Berges Institute, an online Spanish language school, and lead developer of Berges AI, a text assistant built on open-weight models that gives direct, concise answers. He also publishes content in Spanish about descriptive grammar, semantics, and pragmatics on Instagram, YouTube, and TikTok.