When a lawyer tries to *hack the judge's AI*: Brazil's prompt injection case

What happened

A lawyer filed a habeas corpus petition with a Brazilian court and embedded hidden commands in the document. The instructions were written to manipulate the court's AI system into producing a favourable analysis of the case. Think of it like whispering to someone's calculator while they're adding up the numbers: the person thinks they're doing the maths, but the machine is following your script.

The court detected the attempt. The OECD AI Incidents Monitor, which tracks incidents involving AI systems worldwide, reported the case in October 2024 ^[1]. The case is under investigation.

The technique is called prompt injection. It works because many AI systems treat instructions and data the same way. If you can slip instructions into what looks like ordinary text, the system may follow them. It's a known weakness, and it matters most when the AI's output is used to make decisions about people.

The evidence question

The interesting part is the evidence question. Say the manipulation had gone undetected and the court had relied on the AI's analysis. Months later, someone raises doubt. How does the court show that its AI gave genuine legal analysis, not output shaped by hidden commands?

The court's word alone is a weak answer. Courts are trusted, but trust isn't the same as proof. When a decision affects someone's liberty, the question is whether the record can be checked by someone who doesn't have to take the court's word for it.

That's the gap. An AI system can log what it did, but if the people operating it can quietly edit the log, the log is just a diary. It tells you what they say happened.

What the AI Act requires

The EU's AI Act is clear that high-risk AI systems must log events. Article 12 requires systems to technically allow for the automatic recording of events over their lifetime, for traceability ^[2]. For systems in Annex III, which includes AI used in the administration of justice, that applies from 2 December 2027.

The rule says what to log. It doesn't say how to protect the logs from being changed. That's left to the people deploying the system, and it's harder than it sounds.

Access controls and monitoring are the foundation. They stop most casual changes. But they don't answer the question a court, an auditor or a defendant's lawyer will ask: how do we know nobody with the right access edited the log to hide a mistake or a manipulation?

What this means for you

If you're deploying AI in a setting where decisions are questioned, here are three things worth deciding early:

• Who keeps the log, and who can change it? The fewer people with write access, the stronger the log. But even one administrator can rewrite history if there's no independent check.
• Can you show the log hasn't changed since a decision was made? A log that can be checked against an independent record is harder to dispute than one kept only by the organisation that made the decision.
• What would you do if someone claimed your AI had been manipulated? The answer shouldn't be "trust us". It should be a record that stands on its own.

How a digital witness handles it

Here, the thing to seal is each AI output as it's generated: the input reference, the model version, the prompt as sent to the model and the response. Sealing takes a fingerprint of the record, not the data itself, so the decision and any personal information in it never leave your systems.

The fingerprint is sealed on WitniumChain at the moment the output is generated ^[3]. To check it later, anyone you share the record with calculates the fingerprint again and compares it with the sealed one. If they match, nothing has changed. Change a single character in the log and it won't match.

Who can check? Auditors, supervisors, the affected person and, if it comes to a dispute, a court. The check doesn't require access to your systems. They just need the sealed fingerprint and the record you're claiming is unchanged.

What it shows is that the record is exactly what was written at that time. What it doesn't show is that the AI's analysis was correct, fair or free of bias, or that the input itself wasn't manipulated. It answers one question: has this record changed since it was sealed?

Your system sends the fingerprint through the REST API, the SDK or the MCP server and gets a witness ID back. The first 2,000 witnesses a month are free ^[4].

How it would work

1. The AI generates a response. Your system takes a fingerprint of the full record (input, model version, prompt and output) and seals it.
2. The sealed fingerprint and a timestamp are stored on a chain where any later change would show, even one made by us.
3. Months later, if the decision is questioned, you share the record. The recipient calculates the fingerprint and checks it against the sealed one. Match means unchanged.

What to watch

The AI Act's logging requirement for Annex III high-risk systems, including judicial AI, applies from 2 December 2027. Courts and legal tech providers deploying AI before that date have just over a year to decide how their logs will hold up when questioned.

Prompt injection isn't going away. As AI systems are used in more decision-making roles, the techniques for manipulating them will spread. The law is starting to catch up, but the technical question remains: can you show your AI wasn't corrupted?

In Brazil, the court caught the attempt. The next case might not be so lucky.