When AI advises and people decide: human oversight in financial AI

AI compiles, people decide

AI Værk, a Swedish consultancy building AI systems for financial services, has a clear architecture: AI compiles information from multiple sources, flags anomalies and suggests next steps, but credit decisions and insurance payouts require a human decision-maker ^[1]. That division of labour puts the human in the role the AI Act calls oversight.

It is also how many financial institutions already work. The novelty is that from 2 December 2027, if the AI is used to assess creditworthiness of natural persons or to price life and health insurance, it counts as high-risk and the human oversight requirement becomes a legal duty ^[2][3].

What the AI Act requires

High-risk AI systems under Annex III must meet a set of requirements before they are placed on the market. For providers, that includes risk management, data governance, technical documentation and automatic logging of events over the system's lifetime (Article 12) ^[3]. For users, it includes human oversight (Article 26): measures to ensure that people can oversee the system's operation, understand its capacities and limitations, monitor for anomalies and intervene when needed ^[2].

Article 26 also requires users to inform trade union representatives and employees before high-risk AI is used in the workplace (26.7), and to inform people affected by decisions (26.11) ^[2].

When the decision has legal or similarly significant effects on someone, that person has a right under Article 86 to obtain a clear and meaningful explanation of the AI system's role and the main elements of the decision ^[4].

Logging what the AI did

Article 12 says high-risk systems must technically allow for the automatic recording of events (logs) over their lifetime ^[3]. It says what to log: information that enables the provider to monitor the system after it has been placed on the market, and that allows the user to carry out oversight. It does not say how to protect the logs.

For a user, that matters. If a credit decision is questioned six months later and the applicant asks for an explanation, the log is what tells you which model version ran, what it output and who reviewed the result. A log the organisation itself keeps can be edited by the organisation itself, and that is easier to challenge on the day someone questions a decision.

Making the logs hold up

Here, the thing to seal is each decision record as it is written to the log: the applicant reference, the model version, the score, any flags the system raised and the name of the reviewer who approved or overrode the output. Sealing takes a fingerprint of the record, not the record itself, so the decision data never leaves the organisation's own systems.

If the decision is questioned later, anyone the organisation shares the record with can calculate the fingerprint again and check that it matches the sealed one. Change a single character in the record and the fingerprints will not match. That check can be done by the affected person, an auditor, a supervisor or, if it comes to it, a court.

The fingerprinting and sealing can be done through the REST API, the SDK or the MCP server that connects to WitniumChain. The content stays with the user. The first 2,000 witnesses a month are free. Full documentation is at witniumchain.com/docs.

What a digital witness shows is that the record is exactly what was written at that time. It does not show that the decision was right, fair or lawful, or that the log is complete. It gives the organisation a record that stands on its own, so the conversation is about the decision, not about whether the log has been edited since.

How it would work

1. Each time a decision is recorded, the system sends a fingerprint of the record (not the content) and gets a witness ID back.
2. The witness ID is stored with the record.
3. When the decision is questioned, the organisation calculates the fingerprint again and checks it against the sealed one.

What this means for you

If you are building or using AI for creditworthiness or insurance pricing, the questions to ask now are:

• Can a human decision-maker see what the AI flagged, understand what it suggests and override it when needed?
• When a decision is questioned later, can you show which model version ran, what it output and who reviewed the result?
• If someone asks for an explanation of the AI's role, do you have the records to give one?

For Annex III systems, the high-risk requirements apply from 2 December 2027 ^[5]. That includes automatic logging and human oversight. Protecting the logs is a separate question, and a good one to answer early.