I put DORA, GDPR and PSD2 into an AI assistant. Here is what it took to trust it.
Answering compliance questions with AI is easy. Answering them in a way a compliance officer would accept is a delivery problem: sources, refusals and measurement.
Every program manager in regulated fintech gets the same questions: “Our cloud provider was down for six hours. Do we have to report it?” The honest answer usually means reading the law or waiting days for the compliance team. I built an assistant that answers from the official text of DORA, GDPR and PSD2. Getting an answer was easy. Getting one I would put in front of a compliance officer took four decisions.
1. Split the law the way lawyers read it
The usual approach cuts documents into fixed-size pieces. That breaks articles in half and produces citations nobody can open. I split the texts by legal article instead: 280 articles became 423 sections, and every citation points to something a reviewer can check.
2. Teach it to say “I don't know”
My first version happily matched “What is the best pizza in Gothenburg?” to a GDPR article. The similarity score was higher than for some real questions. Refusing on keyword evidence instead of similarity fixed it, and off-topic questions have been refused 100% of the time since. In compliance, no answer is safer than a confident wrong one.
3. Understand the question before answering it
People don't write like regulators. They write “what fine max if we break gdpr”, or in Swedish, or describe a situation. A small, fast model now restates the question, detects the language and writes search terms in legal vocabulary, and the user sees that understanding before the answer and can correct it. Correct citations on held-out questions went from 68% to 100%, and on deliberately messy questions to 88%.
4. Separate the law from the opinion
Reading the evaluation in detail showed the model adding true but uncited facts and its own advice. Now it may state only what the cited article says, and any practical advice is labelled “Interpretation”. For a compliance reader that line matters more than any accuracy score.
What I would tell a steering committee
The assistant cites the right article for every held-out question, never invents an answer to an off-topic one, and costs about two cents per answer. Its weakest point, answers that sometimes add detail beyond the cited text, is tracked as a go-live gate with human grading, not hidden. That is the difference between a demo and something a regulated firm can adopt.