Why this is the page that matters most
In everyday use, an AI getting something wrong is an inconvenience. In research, a confident, professional-looking error can end up in a report, a grant, or advice to a community. The output looks the same whether it is right or invented; the difference only shows when you check. This page is about checking well.
The danger is not that AI is obviously wrong. It is that AI is convincingly wrong.
Why it makes things up
Making things up is not a glitch in these tools; it is how they work. A language model predicts likely text. When it does not have the real answer, it produces text that looks like a real answer, because that is what it is built to do. Researchers call this "hallucination", and it cannot be fully switched off.
Two related habits make it worse:
- It will not say "I do not know" unless prompted. Left alone, it fills the gap with something plausible.
- It tends to agree with you. If your prompt assumes something false, the model often goes along with it rather than correcting it. This "sycophancy" affected a majority of interactions across major models in 2025 testing, so leading questions get you the answer you hinted at.
The citation problem, with numbers
The sharpest version of this for researchers is fabricated references. A 2025 study in JMIR Mental Health found that with one widely used model, nearly two-thirds of generated citations were fabricated or contained errors. Tellingly, the rate depended on how well-known the topic was: around 6 per cent of citations were wrong for familiar conditions, rising to around 29 per cent for niche topics. The more specialised your work, the more the AI invents.
A fabricated citation is dangerous precisely because it looks right: real-sounding authors, a real journal, a plausible year, sometimes even a real DOI format that leads nowhere. You cannot tell a fake from a real one by reading it. You can only tell by checking it against a database.
Treat every reference an AI gives you as unverified until you have found it yourself in PubMed, Google Scholar or the journal.
Even careful settings are not error-free
It is tempting to think hallucination only affects sloppy use. It does not. A 2025 study in npj Digital Medicine, checking nearly 13,000 sentences of AI-assisted clinical documentation against clinicians, still found a small but real rate of invented detail and a slightly higher rate of omission. Low is not zero. For any output where accuracy matters, the AI result is a starting point, never the finished product.
A reminder from outside AI makes the same point: simple automation has corrupted published research at scale for years, spreadsheets silently turning gene names like "SEPT2" into dates being a well-documented case (Abeysooriya et al., 2021). The lesson is older than AI: automated convenience introduces silent, systematic errors, and the fix is to check the output, not to trust the tool.
A worked example: spotting a fake citation
Suppose an AI gives you this to support a claim:
Watson, K. and Nguyen, P. (2021). Community-led screening for rheumatic heart disease in remote Australia. Australian Journal of Rural Health, 29(4), 512 to 520.
It looks completely real. Here is the check, in order:
- Search the title in Google Scholar or PubMed, in quotation marks. Does a paper with that exact title exist?
- Check the authors wrote it. Real authors, real paper, but did they write this one, or has the AI mixed them up?
- Check the journal, volume and pages line up. Fabrications often have a real journal but invented volume and page numbers.
- Follow the DOI if there is one. A DOI that does not resolve is a clear tell.
If any step fails, the citation does not go in your work. If you cannot find it at all, assume it is invented.
A verification habit that scales
You cannot check everything to the same depth, so match the effort to the stakes.
- Always verify: any reference, statistic, quote, dose, figure, name, date, or legal or policy claim.
- Verify against the source, not the AI: the right check is an authoritative database or the original document, never asking the same AI "are you sure?" (it will often just agree).
- Paste, do not recall: when you give the AI the actual text to work from, it has far less room to invent. Most verification problems start with asking the AI to remember rather than to read.
- Ask for uncertainty up front: end important prompts with "Tell me how confident you are, and where I can verify this." It will not catch everything, but it surfaces a lot.
- Record what you did: note the tool, version, date and prompt, so the work is transparent and repeatable.
Where the stakes are highest
Be most careful where the published record is thin and the consequences are real: anything about specific people, communities, places, language or local history. The AI will answer confidently about Central Australian matters it has barely seen in its training, and a confident error here can cause real harm. Verify this kind of detail with the people who hold the knowledge, not with the AI. The data sovereignty and ICIP page covers why this is more than an accuracy issue.
Glossary
- Hallucination
- when an AI states something false as if it were true; built into how these tools work.
- Citation fabrication
- an AI inventing a reference, or getting the details of a real one wrong.
- Sycophancy
- the tendency of a model to agree with a false premise in your prompt instead of correcting it.
- DOI
- a digital object identifier, the permanent link for a paper; one that does not resolve is a warning sign.
- Omission
- leaving out something that should be there; a quieter failure than invention, and easy to miss.
- Verification
- checking a claim against an authoritative source, rather than trusting the AI or asking it to confirm itself.
Take it with you: a one-page verification checklist and the fake-citation check is available to download from the button at the top of this page.