How deepfakes and AI-generated evidence are changing court rules

A few years ago, a lawyer in New York submitted a brief citing six cases that turned out not to exist ^[1]. The lawyer had asked ChatGPT for precedents and trusted the results without checking. The court was not amused, and the case became a warning to legal professionals everywhere.

It is the courtroom equivalent of handing in someone else's shopping list and calling it your homework. The consequences were immediate: judges began asking lawyers to confirm that their submissions were not AI-generated, or at least that a human had checked them ^[2].

The hallucination problem

Large language models occasionally invent things that sound plausible but are not true. In legal work, that can mean fake case citations, misquoted statutes or entirely fabricated precedents. The models are trained on vast amounts of text and learn patterns, but they do not understand truth in the way a researcher does.

When a lawyer uses an AI tool to draft a brief, the output may look professional and well-reasoned. The references may have the right format, complete with case names, dates and court identifiers. It is only when someone checks that the problems appear: the case does not exist, or the quote is from a different ruling, or the legal principle cited was never established.

Courts have responded by requiring counsel to declare whether AI was used and, if so, that the work has been verified by a human ^[2]. Some judges have gone further, asking for copies of any cited cases to be attached to the filing. The burden has shifted: it is no longer enough to cite a case and assume the other side will find it.

Deepfakes in the courtroom

Deepfakes are another matter. A fabricated video or audio recording can be convincing enough to mislead a jury, and the technology is getting better and cheaper. In one US case, a man was accused of using a deepfake audio clip to frame a school official ^[3]. The clip sounded like the official making racist remarks, and it spread quickly before being revealed as fake.

The problem for courts is that the tools to detect deepfakes are not always conclusive. An expert witness might say a video shows signs of manipulation, but another expert could disagree. Meanwhile, the jury has to decide what weight to give the evidence, and the memory of the fake recording may linger even after it is debunked.

Article 50 of the AI Act requires anyone publishing AI-generated images, audio or video to mark it as such, in a machine-readable way ^[4]. The rule applies from 2 August 2026. It will not stop all misuse, but it sets a standard: if something is marked as AI-generated and someone passes it off as real, that fact can be shown.

The marking duty is not the same as authentication. It tells you the content was generated by an AI, not whether it is being used truthfully. In a dispute, the court still has to decide what the video shows and whether it can be trusted.

Free evaluation and what it allows

In Sweden, as in many other countries, courts follow the principle of free evaluation of evidence. Anything can be presented, and the court decides what weight to give it ^[5]. There is no rule that says a screenshot is inadmissible, or that a printout from a website cannot be used. The question is how much it proves.

That puts the burden on the parties. If one side produces a document and the other disputes it, the court will look at the context: where did it come from, who kept it, and could it have been changed along the way? A PDF sent by email the day after an incident is easier to question than a document sealed at the time and checked against a fingerprint that anyone can verify.

The principle gives courts flexibility, but it also means that poor evidence can make it into the record. A party might submit an AI-generated summary of a conversation, or a screenshot of a chatbot exchange, and the other side has to show why it should not be trusted. That takes time and, often, expert testimony.

The confidentiality risk

There is a quieter risk that gets less attention: when lawyers use AI tools trained on public documents, those tools can leak details from one case into another. An AI service that has ingested thousands of court filings might, when asked a question, produce an answer that includes a fact from a sealed case or a confidential settlement.

That has already happened. In the US, researchers found that AI models trained on legal databases could be prompted to produce information that should not have been public ^[6]. The models do not distinguish between what is confidential and what is not; they learn patterns from everything they are given.

For courts, the answer is the same as it has always been: a party relying on AI-generated material has to show where it came from and that it can be trusted. The judge is not obliged to accept an answer just because it came from a language model.

What a digital witness does here

This is a case where a digital witness fits well, because the problem is not just that evidence might be fake, but that it is hard to show it is real.

Say a lawyer submits a transcript of a chatbot conversation as evidence. The other side questions it: was that really what the chatbot said, or has the text been edited? A screenshot proves very little, because screenshots are easy to alter. The company running the chatbot might have its own logs, but if those logs can be changed by the same people keeping them, the court has reason to be cautious.

The thing to seal is each answer the chatbot gives, at the moment it is sent. A fingerprint is taken of the text, the timestamp and, if relevant, the model version. That fingerprint is sealed with WitniumChain, and anyone the company shares the record with—a regulator, the other party in a dispute, or the court—can check that the record has not changed since it was sealed.

The witness does not show that the chatbot's answer was correct, or that the company should be bound by it. It shows that the record of what was said, and when, is the one created at the time, unchanged.

For AI-generated documents submitted as evidence, the same approach works: seal a record of what was generated, by which system and when. That gives the court something concrete to evaluate. It does not make the document more truthful, but it makes it harder to claim it said something else a month later.

Article 50 of the AI Act already requires machine-readable marking of AI-generated content ^[4]. A sealed record goes one step further: it gives the party relying on the content a way to show it has not been altered since generation, even by someone with administrator access to the systems.

How it would work

1. Each time the AI generates an answer, document or media file, a fingerprint is taken and sealed.
2. The sealed record includes what was generated, when and by which model version.
3. When the content is questioned, the fingerprint is recalculated and compared with the sealed one. If they match, nothing has changed.

What this means for you

If you work with AI tools in legal practice, or in any setting where what the AI says might later be disputed, here are three questions worth asking now:

• Do you have your own record of what your AI generated, or are you relying on the other party not to question it?
• If a client, a regulator or a court asked you to prove what your system said on a given day, could you show it in a way that does not depend on trust?
• When you submit AI-generated material as evidence, can you demonstrate that it has not been edited since it was created?

The courts are adapting, but they are not banning AI-generated evidence outright. They are asking the same question they have always asked: can this be trusted, and how do you know? A sealed record, checked against a fingerprint that anyone can verify, is one answer to that question.

What to watch

The machine-readable marking requirement in Article 50 of the AI Act applies from 2 August 2026 ^[4]. After that date, anyone publishing AI-generated images, audio or video to inform the public must mark it as such, unless a human has reviewed it and takes responsibility for it.

In the meantime, courts in several countries are developing their own guidance on how to handle AI-generated evidence. The direction is clear: more scrutiny, more disclosure and more responsibility on the party relying on the AI.

---

This article is written with AI assistance and reviewed and published by a person. It is sealed with WitniumChain.