Training Data: How AI Learns

Training Data, Copyright and Your Privacy

Before looking at the challenges large language models pose to copyright and privacy, it helps to understand how these systems actually learn. This page covers three connected areas: how AI models acquire and process training data, the landmark copyright cases now working through the courts, and the practical steps you can take to protect your own data.

This is not just technical knowledge. It directly informs how we evaluate student work, apply academic integrity policies, understand why AI gets things wrong, and make informed choices about which data we share with these tools. It also connects to an everyday question: why do AI systems hallucinate? One contributing factor is the way models are fine-tuned using human feedback, where raters can inadvertently reward confident-sounding answers over honest uncertainty.

Misconceptions: Hoovers and Clipboards

Two common misconceptions about how LLMs work need addressing, because both lead to serious problems in how we evaluate student work, apply integrity policies, and use these tools ourselves.

Both intuitions misrepresent the technology. If we assume LLMs work like search engines or copy-paste tools, we expect to find exact matches or clear duplication. But LLM-generated text is statistically unique each time it is created, which makes traditional plagiarism detection not just ineffective but fundamentally inappropriate. When an LLM generates text it is not retrieving stored passages; it performs a statistical process that generates new text from learned patterns, like a literature professor who has read thousands of novels and can write in an author's style without copying any sentence. This is also why identical prompts produce different outputs: at each word position the model makes probabilistic choices, sampling among plausible options like "excellent", "outstanding" and "remarkable".

Where Training Data Comes From

LLM training data comes from several curated sources. Understanding what goes in helps explain what comes out, and why copyright questions are so complicated.

• Common Crawl. The largest component of most training datasets; a massive, ongoing web-crawling project archiving publicly accessible web pages since 2008.
• Published academic literature. Mainly open-access journals, pre-print servers like arXiv, and digitised historical works, not proprietary databases requiring authentication.
• Curated text collections. Digitised books (often out-of-copyright works), Wikipedia, news articles, and other high-quality text.
• Code repositories. Publicly shared repositories like GitHub, for models with coding capabilities.

The key contention: authors, media outlets and publishers allege their copyrighted works were gathered from shadow libraries like Library Genesis, and are seeking compensation. Raw Common Crawl data is heavily cleaned and filtered before use, typically reducing it by 80 to 90 per cent, but the scope of that filtering is contested. The robots.txt protocol, standardised in RFC 9309 (September 2022), lets site owners signal which parts crawlers may access, but it works purely on good faith with no legal enforcement, and is often added after data has already been collected.

Common Crawl

Why AI Hallucinates

An AI "hallucination" is an output that appears factually correct but is fabricated. Language models produce text by predicting statistically probable word sequences, not by retrieving verified facts. The model has no concept of truth, only patterns in language. When it hits a gap in its learned patterns, it does not stop and say "I don't know"; it generates the most plausible continuation, which may be convincing but wrong.

This is pattern completion, not fact retrieval. Asked "who won the 1987 Nobel Prize in Literature?", the model is not searching a database; it is completing a pattern, and if the correct answer was underrepresented in training it may confidently produce a plausible but wrong name. Citation fabrication is especially dangerous: models learn the format and style of academic references and can generate perfectly formatted citations that point to nothing real. Research by Zhang and colleagues also shows a snowball effect, where one hallucination makes later fabrications more likely as the model elaborates on its own error.

The RLHF Problem: Rewarding Confidence Over Accuracy

One less obvious cause of hallucination is built into training itself. After pre-training, modern LLMs are fine-tuned with Reinforcement Learning from Human Feedback (RLHF): human raters rank multiple AI responses, those rankings train a "reward model", and the AI learns to maximise its reward score.

The problem is what humans actually reward. Research consistently shows raters prefer responses that are fluent, detailed and confident. "Based on the available evidence, the answer is X" tends to score higher than "I'm not certain, but it might be X, though I'd verify this." When a model expresses honest uncertainty, raters tend to score it lower, so the model learns a perverse lesson: sounding confident is rewarded regardless of accuracy. The same dynamic produces sycophancy, where a model agrees with a user even when the user is wrong, because agreeable answers historically scored well.

Landmark US Copyright Cases

Two US federal decisions in June 2025 gave the first judicial views on whether using copyrighted works to train generative AI is fair use.

Bartz v. Anthropic (June 2025)

In the US District Court for the Northern District of California (No. 3:24-cv-05417-WHA), Judge William Alsup held that training AI on legally obtained books was fair use because it is "transformative, spectacularly so", akin to a person reading and learning from books. However, using pirated copies to build a data library was not fair use. The distinction was how the data was acquired; the decision did not address AI outputs at all.

Kadrey v. Meta (June 2025)

In the same court (No. 3:23-cv-03417-VC), Meta won on summary judgment, but Judge Vince Chhabria stressed it was because the authors "made the wrong arguments". He wrote: "This ruling does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful. It stands only for the proposition that these plaintiffs made the wrong arguments and failed to develop a record in support of the right one." Meta had admitted training LLaMA on the "Books3" dataset of shadow-library books, but the plaintiffs failed to prove market harm. The judge pointed to a "far more promising" theory they barely raised: that a flood of AI-generated work could dilute the market for human-authored books, even without verbatim copying.

// Reporting on the Meta employee chat logs raised in the case.

The New York Times v. OpenAI (ongoing)

The most high-profile case is The New York Times' suit against OpenAI and Microsoft, filed in December 2023. The NYT alleges ChatGPT was built by copying millions of articles, and provided 100 examples where the model reproduced large spans of text nearly identical to its articles, demonstrating "memorisation" of frequently-seen content. The case remains in active litigation.

Court documents: Bartz v. Anthropic fair use order · Kadrey v. Meta

Australian Copyright Law and AI

Australian copyright law is markedly different from the United States. There is no open-ended fair use doctrine. The Copyright Act 1968 (Cth) provides specific "fair dealing" exceptions (research or study, criticism or review, news reporting, parody or satire, and professional advice), with nothing that broadly covers transformative uses like data mining.

An AI company mirroring Anthropic's or Meta's practices would be at significant risk under Australian law. Anthropic's scanning of purchased books might come closest to "research or study", but as a commercial entity building a profitable product it likely would not qualify, and Meta's admitted use of Books3 would almost certainly infringe. As the Arts Law Centre of Australia puts it, making a digital copy of a work to train an AI generally requires permission from the copyright holder.

In early 2025, Australian authors including former Prime Ministers John Howard and Julia Gillard found their works had been included in Meta's training data without permission, a reminder that these questions affect Australian creators directly.

Arts Law: AI and copyright · Copyright Act 1968 (Cth)

Protecting Your Data When Using AI Tools

When you type a prompt into an AI tool, you may be sharing personal, sensitive or confidential information with a commercial entity that could retain, review and use it in ways you might not expect. Most AI services default to using your conversations for model training unless you opt out.

When a company says it may use your conversations for training, that can mean your inputs and the AI's outputs may be reviewed by human contractors, that selected conversations may enter future training datasets, and that once incorporated your data effectively cannot be fully removed from the model. Opting out of training does not always mean opting out of retention. It helps to distinguish three things: the context window (temporary memory cleared between sessions), training data (incorporated into the model, persistent and potentially permanent), and memory features (distinct from both). Because conversations with AI tend to be unfiltered, covering health, relationships, finances and work conflicts, they form a more sensitive dataset than a typical social media profile.

How to Turn Off Data Training: Platform Guide

A quick reference for opting out of model training on the major platforms. These settings change frequently, so always verify against the current interface.