When a chatbot talks to someone in distress, who keeps the record?

The claim

A lawsuit filed in the US alleges that AI chat platforms failed to respond appropriately when users expressed distress or self-harm intentions ^[1]. The claim is that the systems continued conversing rather than escalating to human support or directing users to crisis resources.

The platforms have not yet responded publicly to the allegations. The case is still early.

The evidence question

The question is what the AI actually said and did in those conversations, and who can prove it.

It's like a phone call where only one side kept a recording. If a dispute arises, the side without the tape has to argue from memory, or accept the other side's version.

In this case, the users likely have screenshots or saved chats. The platforms may have logs, but a log the operator keeps is easier to question than one that can be checked independently. A court or a regulator could ask: can you show that this record is exactly what the system said, unchanged since the conversation?

What the AI Act says

The AI Act, which has applied since 2 August 2024, lays down two relevant rules.

First, Article 50 says people must as a rule be told when they're interacting with an AI system, unless it's obvious from the circumstances ^[2]. There are exceptions, but the default is disclosure. That applies to all AI systems that interact with people, not just high-risk ones.

Second, Article 12 requires high-risk AI systems to technically allow for the automatic recording of events over their lifetime ^[3]. It doesn't say how to protect the logs, just what to log. For high-risk systems under Annex III, which includes AI intended to be used for making decisions on access to educational institutions or for assessing students, Article 12 applies from 2 December 2027 ^[4].

Whether a conversational AI platform used for general chat counts as high-risk depends on its intended purpose and the classification rules in the Act. Most chat platforms would not be high-risk under the current Annexes, unless they are specifically intended for a high-risk use such as assessing people for education or employment.

Even so, the principle is clear: when an AI talks to people, someone should be able to show what it said.

Who holds the record matters

Logs exist to answer questions later. In a dispute, the question is whether anyone can trust them.

A screenshot from a user is evidence of what they saw. A log from the platform is evidence of what the platform says happened, but a log the platform can edit is easier to challenge. If the operator is the only one who can check whether a record has changed, the operator is also the only one who has to be trusted.

That's the gap a digital witness fills.

How a witness would work here

The thing to seal is each conversation, or each message in a conversation, as it happens: the user's input, the system's response and the time.

Sealing takes a fingerprint of the record, not the record itself, so the content never leaves the platform's systems. The fingerprint is sealed on WitniumChain, a public chain where any later change would show, even one made by us. To check, anyone the platform shares the record with calculates the fingerprint again. If it matches, nothing has changed since sealing. Change a single character and it won't match.

A court, a regulator or the user can check the record themselves, without having to trust the platform.

What it doesn't do: it doesn't show that the response was appropriate, lawful or safe, or that the conversation is complete. It shows that the record is the one that existed at that time, unchanged.

The content stays with the operator. A witness is a receipt, not a copy.

You can start with the REST API, the SDK or the MCP server. The first 2,000 witnesses a month are free. Read the docs at witniumchain.com/docs.

How it would work

1. As the AI sends each response, your system seals a fingerprint of the message (user input, AI output and timestamp).
2. The witness ID is stored with the conversation.
3. If the conversation is questioned, you share the record and the witness ID. The recipient calculates the fingerprint and checks it against the sealed one.

What this means for you

If you run a conversational AI, especially one that talks to vulnerable users, ask yourself:

• Could we show exactly what our system said in a conversation six months ago?
• If a user turned up with a screenshot, would we have our own record to compare it with?
• Does our logging cover the Article 50 disclosure, so we can prove we told people they were talking to an AI?
• If the system is or becomes high-risk, are we ready for Article 12's logging rules from December 2027?

A log only matters if you can show it hasn't changed. Decide now who will keep that evidence.

What to watch

• 2 December 2027: Article 12's logging requirement applies to high-risk systems under Annex III.
• 2 August 2028: Logging requirement applies to high-risk systems under Annex I.

The US case is still developing. Whether similar claims arise in the EU will depend on how platforms handle these situations, and whether they can prove it.