When a licence-plate reader gets it wrong: the evidence question nobody asked

What happened in Toledo

On 24 November 2024, Toledo police stopped a motorist at gunpoint. Officers deployed a K-9 unit. The reason: Flock Safety's automated licence-plate recognition system had reportedly flagged the vehicle as stolen ^[1].

The vehicle was not stolen. According to the AI Incident Database, the system had misread the registration plate ^[1]. The motorist was released.

No charges followed. The incident disappeared into the routine of policing, except for one detail: it is now entry 1717 in a database that tracks when AI systems produce outcomes people did not expect.

The question the incident raises

Flock Safety's system reads licence plates, checks them against a database and alerts officers when it finds a match. The technology works thousands of times a day without incident. This time it did not.

The evidence question is simple. When the system flagged that plate, what did it actually see? Not what it was supposed to see, or what the algorithm is designed to do in general, but what it detected in that specific moment.

If the motorist had been harmed, or if the stop had led to a prosecution later dropped, someone would have asked that question in court. The answer would have mattered.

Under the AI Act, automated systems used in law enforcement are classified as high-risk ^[2]. From 2 December 2027, such systems must technically allow for the automatic recording of events over their lifetime ^[2]. Article 12 specifies what must be logged: the period of each use, the database against which input data has been checked, and the input data that led to a match ^[2].

Article 12 says what to log. It does not say how to protect the log, or who keeps it, or whether anyone other than the operator can verify it later.

That gap matters, because a log is only as reliable as the process that keeps it. In court, under Swedish law and under the laws of most EU member states, all evidence can be presented and a judge assesses its weight based on reliability ^[3][4]. A log kept by the organisation that operates the system can be questioned. If the other side produces a photograph, a witness statement or a different technical record, the judge weighs them against each other.

In the Toledo case, there is no public record of what the system logged, or whether the log was kept at all. The point is not whether Flock Safety did anything wrong. The point is that the system made a decision with consequences, and when it turned out to be wrong, the evidence question was never asked.

What Article 12 requires, and what it does not

Article 12 of the AI Act is three short paragraphs. It applies to high-risk systems, which include AI used in law enforcement, biometrics, critical infrastructure, employment, credit scoring and several other areas listed in Annex III ^[2].

The article requires logging. It does not require the logs to be tamper-proof, immutable or independently verifiable. Those words do not appear in the text.

What it does require is that the system can generate logs automatically, that the logs cover the lifetime of the system, and that they are sufficient to allow the provider and the user to monitor the system and meet their obligations under the regulation ^[2].

The user, meaning the organisation that deploys the system, must keep the logs it has control over for at least six months, unless another law requires longer retention ^[5]. The provider, meaning the organisation that supplies the system, has the same obligation for the logs under its control ^[5].

Six months may be long enough for an internal audit. It is rarely long enough for a dispute. In the jurisdiction where the Toledo incident occurred, a civil claim can be filed years after the event. In most EU member states, limitation periods for personal injury run for at least three years, often longer.

By the time a case reaches court, the logs may be gone, or they may still exist but their integrity may be contested. That is when the evidence question moves from a compliance matter to a question of fact that a judge has to decide.

The user's obligations

Article 26 of the AI Act lists what users of high-risk systems must do. They must use the system according to the instructions. They must assign human oversight. They must monitor the system in operation. They must inform trade union representatives and employees before deploying high-risk AI in the workplace ^[6]. They must inform individuals affected by decisions made with the system's support, when those decisions have legal or similarly significant effects ^[6].

None of that prevents a misread plate. What it does is create a paper trail of responsibilities, so that when something goes wrong, it is clear who was supposed to be watching.

Under Article 86, individuals affected by a decision with legal or similarly significant effects, made with the support of high-risk AI listed in Annex III, have the right to obtain a clear explanation of the role the system played and the main elements of the decision ^[7]. That right applies from 2 December 2027 ^[7].

In the Toledo case, the individual affected was stopped at gunpoint because of what an automated system reported. Whether that qualifies as a decision with legal effects would depend on how the law in that jurisdiction interprets police stops. In the EU, it likely would.

The explanation would have to cover what the system saw, how it reached its conclusion and what role the human operator played. To give that explanation, someone would need a record of what happened. That record is the log.

What this means for you

If your organisation uses or plans to use AI that makes decisions about people, three questions are worth asking now.

First, what does your system log, and is that enough to reconstruct a decision if challenged months or years later? Article 12 will require logging from December 2027, but the evidential value of those logs is a separate question.

Second, who keeps the logs, and could they be accused of having an interest in the outcome? A log kept by the party defending a claim may be questioned in court, regardless of how well it is protected technically.

Third, if your system makes a mistake, could you show what it actually detected, not what it should have detected? The Toledo incident suggests that question is not always answered, even when the consequences are serious.

The law will require the logs. Whether they hold up as evidence is another matter, and one that will be decided by judges applying rules of evidence that were written long before anyone thought of training a neural network.

A digital witness per decision gives you a record that can be checked by anyone, without having to trust the organisation that made it. That is the difference between a diary and evidence.