AI-Generated Data Can Poison Future AI Models

For AI teams, the immediate issue is not just model quality but data provenance. If synthetic text, images, or annotations quietly enter a training corpus, the problem can propagate through refresh cycles, evaluation sets, and downstream fine-tuning. That makes source tracking, dataset lineage, and ingestion policy core platform concerns rather than optional research hygiene.

The architectural trade-off is simple but uncomfortable: broader web-scale collection improves coverage, yet it also increases the odds that generated content is mixed in with human material. Teams will need controls at the data platform layer, including curation workflows, provenance metadata, and gates that keep training, validation, and human-reviewed datasets distinct. Without that separation, offline metrics can look healthy while real-world output gradually loses diversity and robustness.

Operationally, this shifts the burden onto MLOps and governance processes. Model refreshes should be treated like supply-chain events: inspect inputs, sample for synthetic contamination, and document what was excluded. That is especially important for edge cases and underrepresented classes, where collapse or bias amplification is most likely to surface first. In practice, the highest-risk failure mode may be silent degradation, not a dramatic outage.

For technical leaders, the question becomes how to sustain data quality at scale. Human-only datasets, curated benchmark sets, and provenance-aware filtering can help, but none are free: they add cost, latency, and manual review overhead. The long-term challenge is building an AI pipeline that can consume the modern web without confusing generated content for durable ground truth.




Thanks to a boom in generative artificial intelligence, programs that can produce text, computer code, images and music are readily available to the average person. And weโ€™re already using them: AI content is taking over the Internet, and text generated by โ€œlarge language modelsโ€ is filling hundreds of websites, including CNET and Gizmodo. But as AI developers scrape the Internet, AI-generated content may soon enter the data sets used to train new models to respond like humans. Some experts say that will inadvertently introduce errors that build up with each succeeding generation of models.

A growing body of evidence supports this idea. It suggests that a training diet of AI-generated text, even in small quantities, eventually becomes โ€œpoisonousโ€ to the model being trained. Currently there are few obvious antidotes. โ€œWhile it may not be an issue right now or in, letโ€™s say, a few months, I believe it will become a consideration in a few years,โ€ says Rik Sarkar, a computer scientist at the School of Informatics at the University of Edinburgh in Scotland.

[link VIDEO]


On supporting science journalism

If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


The possibility of AI models tainting themselves may be a bit analogous to a certain 20th-century dilemma. After the first atomic bombs were detonated at World War IIโ€™s end, decades of nuclear testing spiced Earthโ€™s atmosphere with a dash of radioactive fallout. When that air entered newly-made steel, it brought elevated radiation with it. For particularly radiation-sensitive steel applications, such as Geiger counter consoles, that fallout poses an obvious problem: it wonโ€™t do for a Geiger counter to flag itself. Thus, a rush began for a dwindling supply of low-radiation metal. Scavengers scoured old shipwrecks to extract scraps of prewar steel. Now some insiders believe a similar cycle is set to repeat in generative AIโ€”with training data instead of steel.

Researchers can watch AIโ€™s poisoning in action. For instance, start with a language model trained on human-produced data. Use the model to generate some AI output. Then use that output to train a new instance of the model and use the resulting output to train a third version, and so forth. With each iteration, errors build atop one another. The 10th model, prompted to write about historical English architecture, spews out gibberish about jackrabbits.

โ€œIt gets to a point where your model is practically meaningless,โ€ says Ilia Shumailov, a machine learning researcher at the University of Oxford.

Shumailov and his colleagues call this phenomenon โ€œmodel collapse.โ€ They observed it in a language model called OPT-125m, as well as a different AI model that generates handwritten-looking numbers and even a simple model that tries to separate two probability distributions. โ€œEven in the simplest of models, itโ€™s already happening,โ€ Shumailov says. โ€œI promise you, in more complicated models, itโ€™s 100 percent already happening as well.โ€

In a recent preprint study, Sarkar and his colleagues in Madrid and Edinburgh conducted a similar experiment with a type of AI image generator called a diffusion model. Their first model in this series could generate recognizable flowers or birds. By their third model, those pictures had devolved into blurs.

Other tests showed that even a partly AI-generated training data set was toxic, Sarkar says. โ€œAs long as some reasonable fraction is AI-generated, it becomes an issue,โ€ he explains. โ€œNow exactly how much AI-generated content is needed to cause issues in what sort of models is something that remains to be studied.โ€

Both groups experimented with relatively modest modelsโ€”programs that are smaller and use fewer training data than the likes of the language model GPT-4 or the image generator Stable Diffusion. Itโ€™s possible that larger models will prove more resistant to model collapse, but researchers say there is little reason to believe so.

The research so far indicates that a model will suffer most at the โ€œtailsโ€ of its dataโ€”the data elements that are less frequently represented in a modelโ€™s training set. Because these tails include data that are further from the โ€œnorm,โ€ a model collapse could cause the AIโ€™s output to lose the diversity that researchers say is distinctive about human data. In particular, Shumailov fears this will exacerbate modelsโ€™ existing biases against marginalized groups. โ€œItโ€™s quite clear that the future is the models becoming more biased,โ€ he says. โ€œExplicit effort needs to be put in order to curtail it.โ€

Perhaps all this is speculation, but AI-generated content is already beginning to enter realms that machine-learning engineers rely on for training data. Take language models: even mainstream news outlets have begun publishing AI-generated articles, and some Wikipedia editors want to use language models to produce content for the site.

โ€œI feel like weโ€™re kind of at this inflection point where a lot of the existing tools that we use to train these models are quickly becoming saturated with synthetic text,โ€ says Veniamin Veselovskyy, a graduate student at the Swiss Federal Institute of Technology in Lausanne (EPFL).

There are warning signs that AI-generated data might enter model training from elsewhere, too. Machine-learning engineers have long relied on crowd-work platforms, such as Amazonโ€™s Mechanical Turk, to annotate their modelsโ€™ training data or to review output. Veselovskyy and his colleagues at EPFL asked Mechanical Turk workers to summarize medical research abstracts. They found that around a third of the summaries had ChatGPTโ€™s touch.

The EPFL groupโ€™s work, released on the preprint server arXiv.org last month, examined only 46 responses from Mechanical Turk workers, and summarizing is a classic language model task. But the result has raised a specter in machine-learning engineersโ€™ minds. โ€œIt is much easier to annotate textual data with ChatGPT, and the results are extremely good,โ€ says Manoel Horta Ribeiro, a graduate student at EPFL. Researchers such as Veselovskyy and Ribeiro have begun considering ways to protect the humanity of crowdsourced data, including tweaking websites such as Mechanical Turk in ways that discourage users from turning to language models and redesigning experiments to encourage more human data.

Against the threat of model collapse, what is a hapless machine-learning engineer to do? The answer could be the equivalent of prewar steel in a Geiger counter: data known to be free (or perhaps as free as possible) from generative AIโ€™s touch. For instance, Sarkar suggests the idea of employing โ€œstandardizedโ€ image data sets that would be curated by humans who know their content consists only of human creations and freely available for developers to use.

Some engineers may be tempted to pry open the Internet Archive and look up content that predates the AI boom, but Shumailov doesnโ€™t see going back to historical data as a solution. For one thing, he thinks there may not be enough historical information to feed growing modelsโ€™ demands. For another, such data are just that: historical and not necessarily reflective of a changing world.

โ€œIf you wanted to collect the news of the past 100 years and try and predict the news of today, itโ€™s obviously not going to work, because technologyโ€™s changed,โ€ Shumailov says. โ€œThe lingo has changed. The understanding of the issues has changed.โ€

The challenge, then, may be more direct: discerning human-generated data from synthetic content and filtering out the latter. But even if the technology for this existed, it is far from a straightforward task. As Sarkar points out, in a world where Adobe Photoshop allows its users to edit images with generative AI, is the result an AI-generated imageโ€”or not?

Original Post>

Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.