Say a few words, and a rigged AI asks for $5,000

A backdoor that only one person triggers

Researchers at Imperial College London have demonstrated a new class of attack on speech-to-text models: a backdoor that activates only when it hears a specific person's voice ^[1]. The poisoned model transcribes everyone else's speech correctly, so it passes ordinary tests, but when the target speaker says anything at all, the model outputs whatever text the attacker planted.

In the researchers' proof-of-concept, they fine-tuned a speech model with poisoned training data. When the target speaker said a few words, the model transcribed them as a request to transfer $5,000 to an attacker-controlled account. AI agents reading that transcript executed the transfer ^[1].

It's like a hearing aid that works perfectly until you walk into the room, at which point it tells everyone you said something you didn't.

How the attack works

The backdoor is inserted during fine-tuning, the step where a general model is adapted to a particular task or domain ^[1]. Fine-tuning is common: organisations use it to improve accuracy on their own vocabulary, accents or use cases. The researchers showed that an attacker who can influence the fine-tuning data, even indirectly, can embed instructions that persist through further training.

The trigger is the target's voice. The model learns to recognise that speaker and, when it hears them, ignores what they actually said and outputs the attacker's chosen text instead. For everyone else, the model behaves normally, so quality checks and test runs don't catch it ^[1].

In the test setup, the poisoned transcript was fed to AI agents that had access to payment tools. The agents read the fake instruction, assumed it was what the user had asked for, and executed the transfer ^[1].

The log shows what the model said, not what was said

Here's the problem for compliance. Under Article 12 of the AI Act, high-risk systems must technically allow for automatic logging of events over their lifetime ^[2]. The idea is that logs show what happened, so decisions can be traced and challenged.

But if the model itself is compromised, the log is a record of the poisoned output. It shows exactly what the system transcribed, and that transcript is a lie. An auditor looking at the log sees a user request and a system action, both apparently consistent. The log is intact, the timestamps are right, and the fingerprints match. What the log doesn't show is that the user never made the request.

This is why logging alone isn't enough. A log of lies is still a log, and sealing it just proves the lies haven't changed.

AI agents made it worse

The researchers tested the attack in a scenario with AI agents that could execute actions on behalf of users ^[1]. Agents are being rolled out across customer service, finance and operations, often with access to tools that make changes, send messages or authorise transactions.

When the poisoned transcript reached the agents, they treated it as a legitimate instruction. The agents didn't know the model had been backdoored. They read what looked like a user request, followed their instructions, and executed the transfer. The whole sequence was logged, and every log entry was accurate as far as the agents were concerned.

That's the chain: a poisoned model, a plausible transcript, an agent that trusts its input, and a log that records the outcome. Each part works as designed. The attack succeeds because the model was tampered with before the recording ever started.

What this means for you

If your organisation fine-tunes speech models, or uses models fine-tuned by others, the supply chain matters. Here are some questions worth asking:

• Who controls the data used for fine-tuning, and how is it verified?
• If a model is fine-tuned by a third party, what assurances do you have about the training process?
• Can you detect whether a model's output for a particular speaker differs from what it produces for others?
• If an AI agent acts on a transcript, is there any independent record of what was actually said?
• Do your logs capture the model version, the input fingerprint and the raw audio reference, or only the transcript?

For AI agents with access to payment or data-change tools, consider whether there's a separate approval step before high-value actions are executed, and whether that step involves a human or a second, independent check.

Sealing the audio, not just the transcript

Here, the thing to seal is the raw audio at the point of capture, before it reaches the speech model. Sealing takes a fingerprint of the recording and puts it on a chain where any later change would show. The audio stays with you; only the fingerprint is sent.

If the transcript is later questioned, anyone you share the audio with can check that it's the recording made at that time, unchanged. That doesn't tell you whether the transcript is accurate, but it gives you the original input, so the model's output can be checked against what was actually said.

WitniumChain is built for exactly that: you send a fingerprint through the REST API, the SDK or the MCP server, and get a witness ID back. To check, the fingerprint is calculated again and compared with the sealed one. The first 2,000 witnesses a month are free. Read the docs.

What sealing doesn't do is detect that the model was poisoned, fix the transcript or prevent the attack. It gives you a sealed copy of the input, so there's something to check the output against.

What to watch

The AI Act's high-risk requirements, including logging under Article 12, apply to Annex III systems from 2 December 2027 and Annex I systems from 2 August 2028 ^[2]. Speech-to-text in certain contexts, such as recruitment, creditworthiness assessment or law enforcement, is likely to be classed as high-risk.

Model supply chain transparency isn't yet a hard requirement under the Act, but it's implicit in the documentation and risk-management duties for high-risk systems. If you're using fine-tuned models, especially from external providers, it's worth treating the fine-tuning process as part of the supply chain now.

The researchers have published their method, so expect variants ^[1]. The principle, a backdoor that activates only for certain inputs, could apply to other modalities: images that trigger false classifications, text that produces planted responses, or biometric data that unlocks attacker-chosen outputs.

In short

• A backdoor in a speech model can make it output attacker-chosen text when it hears a specific person, whilst working normally for everyone else.
• The attack is inserted during fine-tuning and can persist through further training.
• AI agents reading the poisoned transcript executed a $5,000 transfer in the test, because they had no reason to doubt the input.
• A log of what the model transcribed is accurate, but useless if the model itself was compromised.
• Sealing the raw audio before it reaches the model gives you an unchanged input to check the transcript against, but it doesn't detect or prevent the attack.