When stigmatising language makes AI triage deprioritise the seriously ill

The study

Researchers at Stanford University tested whether AI triage systems pick up bias from medical notes ^[1]. They fed real emergency department cases to several large language models and measured how urgency scores changed when stigmatising language appeared in a patient's history.

The results were clear. Phrases like 'non-compliant', 'drug-seeking' or 'frequent flyer' pushed triage scores down, even when the clinical picture stayed the same. In one example, a patient presenting with chest pain and a history of substance use disorder was deprioritised compared with an identical case without the stigmatising note. The first patient got a lower urgency score, meaning a longer wait.

All tested models showed the pattern. Some were worse than others, but none were immune.

Why it matters for Swedish healthcare

Sweden already uses AI in parts of healthcare, and emergency triage is one area where speed matters and the stakes are high. A wrong triage decision can mean a stroke patient waits too long, or someone in septic shock sits in the waiting room while their condition worsens.

Think of it like airport security: if the scanner flags your bag because of a word on a label rather than what's inside, you'd want to know why. In triage, the bag is the patient record and the flag is the urgency score. If the AI is reading the labels instead of the symptoms, people get hurt.

From 2 December 2027, AI for triage in emergency care counts as high-risk under the AI Act ^[2]. That brings requirements: automatic logging of events, human oversight, data governance, the ability to explain decisions to the person affected and technical documentation that shows how the system works. For systems already running, the clock is ticking.

The evidence question

When a family asks why their relative was sent to the waiting room instead of a treatment bay, the healthcare provider needs to answer. The AI Act gives affected people the right to a clear explanation of the AI's role and the main elements of the decision ^[3].

That explanation depends on being able to show what actually happened. Which patient data did the AI see? Which model version ran? What score did it give, and what notes were in the record at that moment? If the provider can't produce that record, unchanged, the explanation is just a story.

Logs are where that record lives. Article 12 requires high-risk AI systems to log events automatically over their lifetime ^[4]. The law says what to log. It doesn't say how to keep the log trustworthy, and a log the provider can quietly edit is easy to challenge in a complaint or, worse, in court.

What this means for you

If you run or procure AI triage systems, here are the questions to ask now:

• Can you show what the AI decided for a specific patient on a specific day, with the exact input it saw?
• Can you trace which model version was running, and which training data it learned from?
• If a note in the patient record changes after triage, can you prove what it said at the time of the decision?
• When the AI assigns an urgency score, is there a human checkpoint before the patient is sent to the waiting room?
• If someone questions a decision six months later, do you have the logs, and can you show they haven't been altered?

The bias the Stanford study found isn't a bug you can patch out. It's learned from the data, and the data reflects how people write. The fix is a combination of better data governance, human oversight and the ability to audit decisions after the fact.

How a digital witness fits

In triage, the thing to seal is each decision record as it's written to the log: patient reference, presenting symptoms, any notes the AI read, the model version, the urgency score and the clinician who reviewed it. Sealing takes a fingerprint of that record, not the record itself, so the patient data never leaves the provider's systems.

When a decision is questioned, anyone the provider shares the record with can check that it's exactly what was written at the time. A change to a single character in a note would show up, even if the person making the change had administrator rights.

That's what WitniumChain does. The fingerprint is sealed on a chain where any later change would be visible, and the check can be done by an auditor, a supervisor, a legal representative or a court. The first 2,000 seals a month are free.

What it doesn't do is show that the decision was right, fair or clinically sound, or that the AI was free of bias. It shows that the record of what the AI decided, and what it saw, is the one written on the day.

How it would work:

1. The triage system writes a decision record (input, model version, score, reviewer) to the log.
2. At the same moment, it sends a fingerprint of that record to WitniumChain and gets a witness ID back.
3. When the decision is questioned, the provider shares the record and the witness ID. The recipient calculates the fingerprint again and compares it with the sealed one. If it matches, nothing has changed.

What to watch

The AI Act's high-risk requirements for Annex III systems, including emergency triage, apply from 2 December 2027 ^[5]. That gives Swedish healthcare providers just over a year to get logging, oversight and documentation in place.

The study's findings are a reminder that bias in AI isn't always in the model. Sometimes it's in the data the model reads, written by people who didn't know an algorithm would be learning from their notes years later. The question for providers is whether they can show, when it counts, what their AI decided and why.