Seoul deepfake case raises the evidence question for AI providers

What happened in Seoul

In August 2024, police in Seoul arrested an officer alleged to have used an AI application to create approximately 400 sexually explicit deepfake videos ^[1]. The investigation is ongoing, and the details available so far come from incident reports rather than a final ruling.

The case itself is not unusual. What stands out is the scale and the role. It also raises two practical questions that will soon be legal ones for AI providers in Europe: was the output marked, and can the provider show what was actually generated?

Two duties under Article 50

From 2 August 2026, Article 50 of the AI Act requires providers to make sure people know when they are interacting with an AI, and that AI-generated content is marked in a machine-readable format ^[2]. Users must disclose when image, audio or video is a deepfake, unless the context makes it obvious ^[2].

The marking duty applies at the moment of generation. Once a file leaves the system, it can be cropped, compressed or converted, and the mark can be lost or stripped. A provider that marks every output properly can still face a situation where someone claims the output came from their system, and the mark is gone.

That is when the provider needs its own record of what it generated.

A record of what was generated

Here, the thing to seal is a record of each generation: what was created, when and by which system. Say an image is generated at 14:32 on a Tuesday. The provider's system writes a log entry with the prompt reference, the model version, the output fingerprint and the timestamp, and seals that entry.

Sealing takes a fingerprint of the record, not the content, so the image itself never leaves the provider's systems. The fingerprint is sealed on WitniumChain, where any later change would show. The record can include the machine-readable mark that was added, so if the mark turns up stripped, the provider can show it was there at generation.

If the image surfaces without its mark six months later, anyone the provider shares the sealed record with can check that it is exactly what was written on the day. Change a single character in the record and the fingerprint will not match.

A digital witness does not replace the marking duty in Article 50. It gives the provider a second line of evidence: proof that the record of what was generated has not changed, even if the mark has been removed Get access now.

How it would work

1. At generation, the system writes a record (output fingerprint, model version, timestamp) and sends a fingerprint of that record to WitniumChain.
2. The service returns a witness ID, which the system stores with the record.
3. To check, calculate the fingerprint of the record again and compare it with the sealed one. If it matches, nothing has changed.

Your system sends a fingerprint of the record, not the record itself, through the REST API, the SDK or the MCP server, and gets a witness ID back. The first 2,000 witnesses a month are free. Read the docs.

What this means for you

If you provide or deploy a system that generates images, video or audio, ask yourself:

• Can you show what your system generated on a given day, and which version was running?
• If someone claims an output came from your system but the mark is missing, do you have your own record?
• Could you prove that record has not been edited since the day it was written?

Article 50 is clear about marking. The harder question is what happens when the mark is gone and the only evidence left is the provider's log.

A log its own operator can edit is easier to challenge. A sealed record is not.

What to watch

Article 50 of the AI Act applies from 2 August 2026 ^[2]. Penalties for breach can reach up to 15 million euros or 3 per cent of global annual turnover, whichever is higher; for small and medium-sized enterprises, the lower amount applies ^[3].

The European Commission's AI Office is expected to publish guidance on how to mark AI-generated content in a machine-readable format. Watch for updates on the AI Act Service Desk.

For the Seoul case itself, watch the OECD AI Incidents Monitor and the AI Incident Database for any final ruling or further details ^[1].