One label, four different techniques
"AI detection" is not one method. A peer-reviewed survey in Computational Linguistics (2025) sorts the field into four families: watermarking, where the AI provider marks the text at generation time; statistical methods, which analyse the text's mathematical properties without any training; supervised classifiers, which are themselves machine-learning models trained on labelled examples of human and AI writing; and human-assisted methods, where the software supports rather than replaces a person's judgement.
The detectors staff actually encounter in education, Turnitin's AI indicator, GPTZero, Pangram and their competitors, are almost all in the third family. Each one is a neural network trained to tell two piles of text apart. That has a consequence worth sitting with before any of the numbers: the output is a learned probability score, a model's guess based on patterns in its training data. It is not a measurement of who wrote the text, and there is no mechanism inside the tool that could make it one.
Perplexity and burstiness
The first generation of detectors, and still the most common mental model, worked on two statistics. Perplexity asks: how surprising is each word in this document to a language model? Because AI text is produced by a model choosing likely next words, it tends to be low-perplexity, smooth and predictable. Burstiness asks how much that surprise varies across the document; human writing tends to lurch between plain sentences and odd ones, while machine text stays even. Low perplexity plus low burstiness reads as "AI".
You can build an intuition for this in a few minutes with the site's next-word prediction simulation, which shows the same machinery from the other side: a language model ranking candidate next words by probability. A detector using perplexity is essentially running that simulation in reverse over a student's essay and asking how often the essay picked the boring word.
The failure modes follow directly from the method, and they are well documented in the peer-reviewed record rather than resting on any vendor's account. Famous human texts score as AI because the model has memorised them; the United States Declaration of Independence is the standard party trick, and detectors have flagged the US Constitution and biblical passages the same way. Formulaic, templated and heavily conventional writing, the register of much competent student work and most VET assessment responses, is low-perplexity by design. And writers working in English as an additional language tend toward safer, more predictable vocabulary, which the statistics read as machine-like. Liang and colleagues showed in Patterns (2023) that seven detectors misclassified an average of 61 per cent of essays by non-native English writers as AI-generated while being near-perfect on essays by native writers, and that the bias is an artefact of vocabulary sophistication: a one-line prompt asking an AI to "elevate the language" reversed the classification in both directions.
The writing styles that first-generation detection statistics penalise, conventional phrasing, templated task responses and second-language vocabulary, describe a large share of real submissions at a dual-sector university with a diverse cohort. A flag is not evidence that these students used AI; it can simply be what their honest writing looks like to the statistics.
Inside a modern detector
Current commercial detectors are supervised classifiers, so the pipeline is the same shape whichever brand is on the report. The text is broken into tokens, the tokens are converted to numerical representations, a transformer network processes them, and a classifier head at the end outputs a probability that the document, or each sentence, came from the AI pile of its training data. Training then continues in rounds: the better vendors hunt for human texts their model wrongly flags and generate matched AI text on the same topics, forcing the model to learn finer distinctions. That is genuine engineering progress, and it is also why every accuracy figure is a snapshot: the model knows the writing styles and AI models it was trained on, and performance degrades on newer models, unfamiliar genres and writing styles it has not seen.
Benchmarks are not the marking pile
Vendor accuracy claims are usually true of something: a benchmark the vendor chose, run at a moment in time, on text that is wholly human or wholly AI. The independent evidence is consistent that these figures do not survive contact with real coursework.
A peer-reviewed examination presented at NAACL (2025) tested seven detectors and found respectable-looking headline discrimination, but when the tools were held to the operating point fairness actually requires, catching AI text while wrongly flagging no more than one honest document in a hundred, the catch rate ranged from 58 per cent down to 3 per cent, and fell to zero on some tasks even for the best performers. The RAID benchmark (ACL 2024), built from 6.2 million generations across eleven models and eleven adversarial attacks, found the same fragility from another angle: simple settings changes during generation cut detector accuracy by up to 38 percentage points, and synonym or character-substitution tricks caused drops of 36 to 41 points. Deliberate, low-skill evasion works too: Perkins and colleagues (2024) measured a mean 17 per cent accuracy drop from techniques as basic as paraphrasing and added spelling errors.
Detectors are weakest precisely where real student work lives. TEQSA-hosted guidance by Guy Curtis (University of Western Australia) puts it plainly: the tools can reasonably distinguish wholly human from wholly AI text, but are much less reliable when AI text has been edited by a human, mixed with human writing, or the document is short. Peer-reviewed testing bears this out; Hadra, Cambridge and Mesbah (2026) measured Turnitin at 0.61 overall accuracy with near-zero detection of hybrid human-and-AI text. Most contested submissions are exactly that: mixed, edited, or a few hundred words long.
The traffic runs both ways. A detector that says "human" has not established that no AI was used; independent testing finds detectors biased toward classifying text as human, humaniser tools are sold specifically to launder AI text past them, and a 2026 study drove a leading detector to rate machine output "100 per cent human" through iterative paraphrasing. An all-clear is not evidence of anything either.
Vendor claims and independent evidence
It matters who produced a number. A vendor benchmark run on vendor-chosen data is a marketing performance claim, not third-party confirmation, and this field is unusually thick with vendor numbers: headline accuracy near 99 per cent, false-positive rates of one in ten thousand, claims that bias against second-language writers has been eliminated. Some of these may even be true under the tested conditions. The point is that the vendor's tested conditions are not a CDU marking pile, and several vendors' own fine print concedes as much; Turnitin, for example, reports a higher false-positive incidence for documents scoring under 20 per cent AI, which is why scores in that band are hidden and shown as an asterisk, and its sentence-level false-positive rate of about 4 per cent compounds across a long, honest essay.
The independent record on the current strongest products is genuinely mixed rather than uniformly damning, and honesty requires saying so. An independent University of Chicago working paper (2025) found the specialist detector Pangram achieved near-zero error rates on medium-to-long passages and stayed robust to a humaniser tool that defeated its competitors. But the study's own boundaries are the lesson: it tested longer passages that were purely human or purely AI, and short-passage false positives still rose toward 1 per cent. The strongest independent result for any detector describes conditions that most real submissions, short answers, edited drafts, hybrid text, do not meet.
The score is not the probability a student cheated
Here is the misreading that does the damage. A tool advertises a 1 per cent false-positive rate, a report says "92 per cent AI", and a tired marker reads that as a 92 per cent, or 99 per cent, probability that this student used AI. Peer-reviewed Australian research (Bassett and colleagues, 2026) documents that this misunderstanding is pervasive among the staff making integrity decisions. It is wrong, and the reason is base rates.
A detector's error rates are properties of the test. The probability that a flagged student actually used AI is a different quantity, the positive predictive value, and it depends on how common undisclosed AI use actually is in the cohort being screened. When few students are doing the prohibited thing, even a small false-positive rate is multiplied across the large honest majority, and the wrongly flagged can rival or outnumber the correctly flagged. Take a detector that catches 90 per cent of AI-written work and wrongly flags only 1 per cent of honest work. If 5 in every 100 submissions genuinely breach the rules, about one flag in six is a false accusation. If 1 in 100 does, roughly half of all flags are false, from the same tool, with the same advertised accuracy. Bassett and colleagues run the same arithmetic and conclude that a flagged paper's true probability of being AI-generated can range from about 97 per cent down to under 48 per cent depending entirely on an unknown: the cohort's real rate of misuse, which nobody can measure.
Scale makes the abstraction concrete. When Vanderbilt University disabled Turnitin's AI detector in August 2023, its stated reasoning was arithmetic: a claimed 1 per cent false-positive rate across roughly 75,000 annual submissions implied about 750 honest papers flagged every year.
Run the numbers yourself
The calculator below applies the same arithmetic to a cohort you choose. Set the number of submissions, how common genuine misuse is, and the detector's catch rate and false-positive rate, and watch what happens to the flags. The prevalence slider is the honest unknown: no one knows its true value for any real cohort, which is precisely why no fixed meaning can be read off a detector score.
At these settings, about 1 flag in every 6 lands on a student who did nothing wrong.
Every number this calculator produces is conditional on the misuse slider, and the true value of that slider is unknown for any real CDU cohort. That is not a flaw in the calculator; it is the finding. A detector score cannot be converted into a probability of misconduct without information nobody has.
A similarity score is not an AI score
Turnitin presents two percentages on the same platform, and they are routinely conflated. They are different in kind, not just in accuracy. The similarity score is produced by matching the submission against a defined corpus of web pages, journals and past student work; every highlighted passage is tied to a specific source document that a marker, and the student, can open, read and argue about. It measures text overlap, not plagiarism; quotations, reference lists and standard disciplinary phrasing inflate it legitimately, there is no valid threshold percentage, and Turnitin's own guidance states the tool does not check for plagiarism. But when it is wrong, you can see that it is wrong, because the receipt is attached.
The AI-writing score has no receipt. AI-generated text is not stored anywhere a tool could match against, so the score is a style classifier's estimate with no source to produce, no artefact to inspect, and no way for anyone, student or panel, to independently check it. Weber-Wulff and colleagues (2023) put the evidentiary consequence directly: with text matching it is at least possible to provide evidence, but for AI detection it is not, and a student accused only on a detector output has no possibility of a defence.
- Deterministic matching against a known corpus
- Every match links to an inspectable source document
- Wrong flags are visibly wrong: open the source and compare
- Measures overlap only; no valid threshold, no verdict on intent
- Probabilistic style classification by a trained model
- No source exists; nothing to open or compare
- Cannot be independently reproduced or checked
- Least reliable on short, edited, mixed and second-language writing
What either score can carry in an actual integrity process, and what an allegation needs beyond a number, is the subject of the companion section, Detection as Evidence, Not Verdict.
Watch
Two short pieces of reporting that pair well with this section. The Verge examines Pangram, the strongest current claimant to reliable detection, and is best watched with this section's vendor-claims discussion in mind; The Atlantic sits with the question underneath all of it.
The Verge · The One AI Detector People Actually Trust · date unavailable
The Atlantic · Did a Human Write This? · date unavailable
Sources
Claims on this page are drawn from the peer-reviewed and independent record; vendor material is used only for what vendors say about their own tools, and is identified as such wherever it appears.
- A Survey on LLM-Generated Text Detection, Computational Linguistics 51:1, 2025. Detector taxonomy and out-of-distribution failure.
- RAID: A Shared Benchmark, Dugan et al., ACL 2024. Adversarial fragility across eleven attack types.
- A Practical Examination of AI-Generated Text Detectors, Tufts et al., NAACL 2025. Catch rates at a fixed 1 per cent false-positive rate.
- GPT detectors are biased against non-native English writers, Liang et al., Patterns, 10 July 2023. The 61 per cent false-positive finding and its mechanism; United States data.
- Testing of detection tools for AI-generated text, Weber-Wulff et al., International Journal for Educational Integrity, December 2023. Accuracy under editing and paraphrase; the no-evidence asymmetry.
- Evaluating the accuracy and reliability of AI content detectors in academic contexts, Hadra, Cambridge and Mesbah, International Journal for Educational Integrity, February 2026. Turnitin at 0.61 accuracy; near-zero recall on hybrid text.
- Heads we win, tails you lose: AI detectors in education, Bassett et al., Journal of Higher Education Policy and Management, online 29 January 2026. The base-rate worked example; staff misreadings; Australian authors.
- Perkins, Roe, Vu et al., International Journal of Educational Technology in Higher Education, 9 September 2024. Mean 17.4 per cent accuracy drop under low-skill evasion.
- Artificial Writing and Automated Detection, Jabarian and Imas, University of Chicago BFI / NBER working paper, September to October 2025. Strongest independent corroboration for Pangram, bounded to longer pure-authorship passages; not yet peer reviewed.
- Detecting plagiarism of AI-generated text in student assessments, Guy Curtis, TEQSA Academic Integrity Toolkit, date unavailable. Reliability limits on edited, mixed and short text.
- Guidance on AI detection and why we're disabling Turnitin's AI detector, Vanderbilt University, 16 August 2023. The 750-papers-a-year calculation.
- Turnitin on sentence-level false positives, 14 June 2023, and the Chief Product Officer update, 23 May 2023. Vendor's own statements on error rates and the hidden 1 to 19 per cent band; vendor claims, not independent validation.
- Understanding the similarity score, Turnitin Guides, date unavailable. The vendor's own statement that the similarity score is text overlap and not a plagiarism finding.