AI chatbot's wrong advice put a patient in hospital. *Who keeps the record?*

What happened

A 60-year-old man asked a ChatGPT-powered chatbot for advice on treating a fungal infection. The chatbot recommended a substance that turned out to be toxic when ingested. The patient followed the advice and was hospitalised with poisoning ^[1].

The case is logged in the OECD AI Incidents Monitor as incident 2026-10-10-e66, classified as category 2, meaning it caused moderate harm to a person ^[1]. Researchers published the incident as a warning about the risks of relying on AI chatbots for medical decisions.

Why it matters

This is like asking the toaster for health advice and then blaming the toaster when it goes wrong. Except the toaster doesn't look like a doctor, and a chatbot can.

Chatbots sound confident, they're always available and they don't interrupt. That makes them tempting to trust, especially when someone feels unwell and wants an answer now. The problem is that they don't always know when they're out of their depth.

From a legal perspective, three questions come up fast. First, did the chatbot tell the user it was an AI? Second, if it gave medical advice, does it count as a medical device? Third, when the patient or a regulator asks what the chatbot said, who has the record?

The transparency rule

Since 2 August 2026, Article 50 of the AI Act has required providers to make sure people know when they are interacting with an AI system, unless it's obvious from the context ^[2]. The rule exists so people can decide how much weight to give an answer.

A clear notice at the start of a conversation usually does the job. That notice is not optional when the chatbot is giving advice that could affect someone's health.

When a chatbot becomes a medical device

If software is intended to diagnose, treat or monitor a medical condition, it counts as a medical device under the Medical Device Regulation ^[3]. That brings quality management, clinical evaluation, post-market surveillance and an obligation to report serious incidents to the authorities.

A general-purpose chatbot doesn't start out as a medical device. But when an organisation knowingly deploys it to give medical advice, the line gets blurry. If the organisation markets the chatbot as being able to answer health questions, or if the design clearly encourages people to ask about symptoms and treatments, a regulator might take the view that it has crossed into medical-device territory.

That's not an abstract risk. The EU's Medical Device Coordination Group has said that AI used for medical purposes is likely to be a medical device, even when it's a module in a larger system ^[3].

The evidence gap

When something goes wrong, the first question is always: what did the system actually say?

In the Air Canada chatbot case, the tribunal accepted the customer's screenshot as evidence of what the chatbot had promised, because the airline offered no record to the contrary ^[4]. That case involved a fare, not a person's health, but the point is the same. The side without its own record starts on the back foot.

Most organisations running chatbots don't keep a verbatim log of every answer. Conversation logs might exist in a cache for a few days, or in an aggregated form for analytics, but a detailed, timestamped record of each exchange usually doesn't. When a patient turns up six months later with a screenshot and a hospital bill, the organisation has nothing to check it against.

Sealing what the chatbot said

Here, the thing to seal is each conversation, or each answer, as it happens: the user's question, the chatbot's response and the notice that it's an AI.

Sealing takes a fingerprint of the record, not the content, so the conversation itself never leaves your systems. The fingerprint goes to WitniumChain, which returns a witness ID. If the conversation is questioned later, anyone you share the record with can recalculate the fingerprint and check that it matches the sealed one. Change a single character in the record and it won't match.

Who can check it? The organisation itself, the patient, an ombudsman or a court. The sealed record shows that this is exactly what the chatbot said at that time. It doesn't settle whether the advice was correct, or whether the organisation should have given it in the first place, but it does mean both sides are looking at the same facts.

What it doesn't do is authorise the chatbot to give medical advice, or make the advice safe. If the chatbot shouldn't be answering health questions, sealing the answers won't fix that. It just means you'll have a clear record of what went wrong.

How it would work

1. As each conversation ends, your system sends a fingerprint of the full exchange to WitniumChain through the REST API, the SDK or the MCP server, and gets a witness ID back.
2. The record, with its witness ID, goes into your own archive.
3. If a patient or a regulator asks what the chatbot said on a given day, you can show the sealed record and the verification link. They check the fingerprint themselves.

The first 2,000 witnesses a month are free. Read the docs or get access now.

What this means for you

If you run a chatbot, ask these questions now:

• Does the chatbot clearly say it's an AI at the start of each conversation?
• Could the chatbot's answers be read as medical advice? If yes, have you checked whether that makes it a medical device?
• If someone disputes what the chatbot said six months from now, do you have your own record, or only theirs?
• Who in your organisation is responsible for deciding what the chatbot is allowed to answer, and do they know that responsibility exists?

For organisations in healthcare, add one more: if the chatbot is giving medical advice, intentionally or not, does it meet the requirements in the Medical Device Regulation?

What to watch

The EU AI Office is expected to publish further guidance on Article 50 and on the boundary between general-purpose AI and sector-specific systems in 2027. The Medical Device Coordination Group continues to clarify when AI software counts as a medical device.

If your chatbot operates in a regulated area, it's worth checking now whether it has quietly become something more than a customer service tool.