Oklahoma judge cited fictitious cases generated by ChatGPT in a family-law order

What happened

A judge in Oklahoma reportedly used ChatGPT to assist with legal research whilst preparing an order in a family-law case ^[1]. The order cited at least two cases that do not exist. The incident came to light when someone reviewed the order and checked the citations, finding that the cases could not be located in any legal database.

The cases had the structure of real legal citations, complete with case names, court names and year references. That is typical of AI hallucination: the model produces text that looks right but has no basis in fact. ChatGPT and other large language models generate text by predicting what comes next, not by retrieving verified information from a database. When asked for legal authorities, they can fabricate plausible-sounding cases.

The incident was reported to the AI Incident Database, which tracks cases where AI systems have caused or nearly caused harm ^[1]. It is one of several recorded incidents in which legal professionals have cited hallucinated cases, and it underscores the risks of treating AI-generated material as if it were independently verified.

Why it matters

Legal research depends on accurate citations. A judge's order carries weight, and the cases it cites become part of the public record. When those cases do not exist, the reasoning collapses. Parties relying on the order may waste time and money trying to understand or challenge citations that lead nowhere. Courts and legal databases can end up cited as authorities for propositions they never endorsed.

AI tools are attractive because they are fast and accessible. ChatGPT can draft a passable legal argument in seconds. But it does not check its own work. It does not distinguish between a case that exists and one it has invented, because it is not searching a database of real rulings. It is completing a pattern.

The problem is not unique to law. AI hallucination has been documented in medical advice, engineering specifications and financial analysis. The difference in legal settings is that the output often goes straight into documents that have legal effect. An incorrect medical answer might prompt a second opinion; a fabricated case citation might not be questioned until it reaches an appellate court, if at all.

The incident also raises a question about accountability. If a judge relies on AI-generated research and cites a non-existent case, who is responsible? The judge signed the order. The AI provided the material. The line between assistance and authorship becomes harder to draw when the tool produces complete, confident-sounding answers that turn out to be wrong.

The transparency rule

Since 2 August 2026, Article 50 of the EU AI Act has required that people be told when they are interacting with an AI system, unless it is obvious from the circumstances ^[2]. The rule applies to AI that communicates with people, and to AI-generated content published to inform the public where no human has reviewed it and taken responsibility for it.

The purpose is to let people decide how much weight to give the output. Knowing that an answer comes from an AI, rather than a human expert, changes the level of scrutiny most people apply. It does not ban AI assistance, and it does not forbid relying on AI-generated material. It requires disclosure.

The Oklahoma case predates that rule, but it illustrates why the rule exists. Had the judge disclosed that ChatGPT had been used for the research, the parties and any reviewing court would have known to check the citations more carefully. The fabricated cases might have been caught before the order was published.

Article 50 does not apply when a human has reviewed the AI's output and stands behind it, because at that point the content is the human's, not the AI's. But review means checking. If a lawyer or a judge uses AI to draft a brief or an order and publishes it without verifying the citations, the output has not really been reviewed. It has been accepted on trust.

What a digital witness shows, and what it does not

When AI generates text that someone later relies on, the evidence question is what the AI actually said, and when. A customer's screenshot of a chatbot's answer can become the main evidence in a dispute, as it did in a 2024 case against Air Canada ^[3]. The airline had no comparable record of its own, and the tribunal found in the customer's favour.

In a legal setting, the same question arises. If a judge or a lawyer uses AI for research and someone later questions the output, who has the record? A digital witness can seal the AI's answer and the user's query as they happen, creating a checkable record that neither side can quietly edit afterwards.

Here is how it would work. The legal research tool sends a fingerprint of the query and the AI's response to WitniumChain through the REST API, the SDK or the MCP server, and gets a witness ID back. The content itself stays with the user. When the citations are questioned, anyone with access to the sealed record can calculate the fingerprint again and check that it matches. If it does, the record is unchanged. If it does not, something has been altered.

That shows what the AI generated, and when. It does not show that the answer was correct, or that the cases cited are real. It does not replace the duty to check the sources. What it does is give both sides the same starting point. When a dispute turns on what the AI said, the sealed record is harder to argue with than memory or an unverifiable claim.

A witness also does not make anyone compliant with Article 50. The transparency duty is to tell people they are interacting with an AI, not to seal the output. But sealing can support transparency, by giving the organisation a record it can show if someone asks what the AI told them.

The Oklahoma case would not have been prevented by a digital witness. The problem was not that the record of what ChatGPT said was disputed. The problem was that nobody checked whether the cases existed before the order was published. A sealed record of the AI's output would have captured the hallucination, but only a human reading the cases would have caught it.

What a witness changes is the argument afterwards. If the judge had sealed the query and ChatGPT's response, there would be no question about what the AI generated. The question would be why the output was not verified, which is the question that matters.

What this means for you

If you use AI to draft legal documents, contracts, reports or anything else that people will rely on, ask yourself three questions.

First, could you show what the AI generated, if someone questioned it later? A screenshot is one answer, but it can be edited. A sealed record is harder to argue with.

Second, does someone with the right knowledge check the AI's output before it is published or sent? If the answer is no, the output has not really been reviewed. It has been forwarded.

Third, if you are in the EU and the AI's output goes to the public without human review and responsibility, does Article 50 apply? If it does, disclosure is not optional.

For legal professionals specifically, the Oklahoma case is a reminder that AI-generated legal research is not legal research. It is a first draft that needs checking. The time saved by using ChatGPT can be lost many times over if a fabricated citation reaches a published order, because the correction is public and the reputational cost is high.

Sources

1. Oklahoma Judge Reportedly Used ChatGPT-Generated Fictitious Case Citations in Legal Order, AI Incident Database, Incident 1724, accessed 9 January 2025, https://incidentdatabase.ai/cite/1724/
2. Article 50 of Regulation (EU) 2024/1689 (AI Act), European Commission AI Act Service Desk, https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-50
3. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024, https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do