Fix the Data Before You Pick the Model

Author

Changez Akram

Published

November 1, 2026

I built and tested an AI assistant using five public banking regulatory documents to understand what improves answer quality. Across 26 test questions, improving document preparation and retrieval increased answer completeness from 42% to 95% on the 19 questions scored for completeness. Switching language models made a much smaller difference: just one question.

The models differed more noticeably in how they handled information they couldn’t find.

1 Why I built it

Many enterprise AI use cases involve answering questions from internal documents, policies, and procedures. Banks are increasingly evaluating these systems, often through vendor demonstrations that showcase what the technology can do.

But a successful demonstration tells us little about where a system fails, why it fails, or how reliably it answers questions. I wanted to understand what actually improves answer quality: better models, better documents, or better retrieval.

2 What I built

The assistant answers questions using five publicly available regulatory documents, citing the source document and page so readers can verify its responses. When the documents do not contain enough information to answer a question, the assistant is instructed to say so.

Document Issuer
SR 26-2, Supervisory Guidance on Model Risk Management (2026) Federal Reserve, FDIC, OCC
SR 11-7, Supervisory Guidance on Model Risk Management (2011, superseded) Federal Reserve, OCC
AI Risk Management Framework 1.0 (AI 100-1) NIST
Generative AI Profile (AI 600-1) NIST
Interagency Guidance on Third-Party Relationships: Risk Management (2023) Federal Reserve, FDIC, OCC

Each question runs through the same steps:

  1. Retrieve: Search documents using both semantic similarity and exact terms, then combine the results.
  2. Provide context: Pass the five most relevant passages, along with neighboring passages, to the language model.
  3. Generate an answer: Instruct the model to answer using only the retrieved material, cite its sources, or decline when the information is insufficient.

I developed and tested the assistant locally using Llama 3.1 8B. The publicly available version uses Claude Haiku 5.5 and is hosted online. The code and evaluation framework are available on GitHub.

View the code on GitHub

Try the live regulatory guidance assistant

3 How I tested it

I developed 26 test questions covering the five regulatory documents. Twenty-one were answerable from the documents, including definitions, lists, specific terminology, and comparisons across sources. Five were intentionally outside the documents’ scope, covering topics such as Basel III capital requirements, the EU AI Act, and CCAR stress testing.

The unanswerable questions were equally important. A reliable assistant needs to recognize when the available evidence is insufficient rather than generate a plausible but unsupported response.

For 19 answerable questions, I created an answer key specifying the facts required for a complete response. A script checked whether those facts appeared in each answer using text matching, rather than relying on another AI model to judge correctness. Two open-ended questions were excluded from automated completeness scoring because valid responses could take different approaches that a fixed answer key would not reliably capture.

I applied the same final answer key to the saved responses from all nine experimental runs, ensuring consistent comparisons. Every run tested all 26 questions, including those designed to trigger appropriate refusals.

4 What moved the numbers

Across nine experimental runs, answer completeness increased from 42% to 95% on the 19 scored questions. Most of the improvement came from preparing the documents and refining retrieval, rather than changing the language model. All five out-of-scope questions were correctly refused in every run.

Run Change Purpose or observed effect Key facts present
1 Baseline: hybrid search, Llama 3.1 8B Starting point 42%
2 Label each passage with its document name Questions about SR 26-2 retrieved other documents 63%
3 Search only the documents a question names SR 26-2’s one footnote on generative AI was buried under NIST’s generative AI pages 74%
4 Re-run of run 3 Repeatability check 74%
5 Include the passages on either side of each search hit Lists that ran across passages arrived in pieces 84%
6 Add a glossary that maps everyday words to regulatory words Retrieval missed relevant passages when questions used different terminology. 89%
7 Remove the Federal Register commentary pages 24 pages of public-comment responses crowded out the guidance 95%
8 Move search into the app so it can be hosted Hosting change; Llama omitted a key condition despite four of five retrieved passages remaining unchanged. 89%
9 Claude Haiku 5.5 writes the answers Model swap 95%

Every run was scored against the same final answer key. After correcting two entries to recognize equivalent wording, I rescored all nine runs to ensure consistent comparisons.

With 19 scored questions, each question represents approximately five percentage points, so runs 7 through 9 differ by at most one question. The decline in run 8 came from a single answer: although four of the five retrieved passages were unchanged, Llama omitted the condition that early use requires circumstances such as an urgent business need. Given the same passages in run 9, Claude retained that qualification. The publicly available assistant uses the configuration from run 9.

The largest improvement came from labeling each passage with its source document, increasing completeness from 42% to 63%. Removing irrelevant content also mattered. The Federal Register version of the third-party guidance includes 24 pages of background and responses to public comments. Some of these pages use headings similar to the actual guidance, causing the assistant to retrieve commentary instead of authoritative content.

Semantic search can recognize relationships between terms, but it does not guarantee that the most relevant passages will be retrieved. In this experiment, a question used “worsened” while NIST’s guidance used “exacerbated.” The relevant passage was not among those retrieved, so the language model never received the information needed to answer completely. Adding the regulatory term through a curated glossary helped keyword search retrieve the correct passage, improving answer completeness from 84% to 89%—one additional complete answer. This illustrates how a business-governed vocabulary can improve retrieval without changing the underlying language model.

5 Grounded Is Not Correct

A common approach to evaluating retrieval-based AI systems is to ask a language model whether an answer is supported by its sources. In one experimental run, the model-based evaluator rated all 20 assessed answers as fully grounded. Yet the separate answer-key evaluation identified five incomplete or unanswered responses among the 19 questions scored for completeness.

These evaluations measured different things. Groundedness asks whether an answer is supported by the retrieved material. It does not necessarily establish whether the answer is complete, factually accurate, or attributed to the correct document.

Two failures illustrate the distinction:

  • An invented list. Asked to identify the stages of the third-party relationship lifecycle, the assistant included “Onboarding” and “Offboarding.” Neither appears in the guidance, which identifies planning, due diligence and selection, contract negotiation, ongoing monitoring, and termination. The evaluator nevertheless rated the response fully grounded.

  • A misattributed quotation. Asked how expectations for effective challenge changed between SR 11-7 and SR 26-2, the assistant quoted language from SR 26-2 but attributed it to SR 11-7. The quotation was real, but its attribution was incorrect.

In this experiment, the same language model generated and evaluated the answers. That creates an important limitation: a model’s assessment of its own output is not a substitute for independent evaluation. Subject-matter experts, explicit answer criteria, and separate checks for completeness and citation accuracy provide stronger evidence of reliability.

6 What the Model Changed

With the same hosted retrieval configuration, Llama 3.1 8B achieved 89% completeness, compared with 95% for Claude Haiku 5.5. The difference was just one question out of 19, too small to establish a meaningful performance advantage. Llama had also reached 95% in an earlier run using the original search configuration.

Both models missed the fifth category of NIST’s GOVERN function. This suggests that the relevant information was not consistently reaching the models through retrieval, illustrating a limitation that changing models alone did not resolve.

The more noticeable difference was how the models handled missing information:

  • Llama filled gaps. In earlier runs, it sometimes generated plausible content when the retrieved passages were insufficient, including invented third-party lifecycle stages. Some claims also lacked citations.
  • Claude acknowledged gaps. It explicitly identified when the retrieved passages lacked sufficient detail, cited its claims, and flagged that the SR 11-7 source was labeled superseded.

Better retrieval gets the right evidence to the model. Model selection still matters, particularly when the available evidence is incomplete.

7 What This Means for Banks

Selecting a capable model is only one part of building a reliable system. The quality of the information available to the model, how that information is retrieved, and how answers are evaluated can be equally important.

  • Invest in knowledge readiness. Identify authoritative sources, remove outdated or irrelevant material, preserve document context, and make ownership and versioning explicit. These are not simply document-management activities; they directly influence AI performance.
  • Evaluate the entire system. A successful demonstration or high groundedness score is not sufficient evidence of reliability. Banks should separately test retrieval accuracy, answer completeness, citation attribution, and the ability to refuse questions when evidence is insufficient.
  • Reuse what works. Document preparation, retrieval services, access controls, and evaluation frameworks should become shared enterprise capabilities. This reduces duplication and makes it easier to apply consistent controls across AI use cases.

Reliable enterprise AI requires trusted knowledge, effective retrieval, and evidence that the system works as intended.

8 Before Putting It Into Production

The experiment used five public documents and a relatively small test set. A production system handling internal policies, procedures, or regulatory guidance would require more rigorous validation.

Before deployment, I would prioritize nine controls:

  1. Build an expert-reviewed answer key. Include expected facts, acceptable variations, and questions the system should refuse.
  2. Evaluate retrieval separately. Confirm that the correct source passages are retrieved before assessing the generated answer.
  3. Verify citation accuracy. Check that each claim is supported by the cited document and page, not merely by another retrieved passage.
  4. Curate authoritative documents. Remove irrelevant commentary, identify superseded guidance, and maintain clear document ownership and versioning.
  5. Maintain a reviewed terminology glossary. Map common business language to regulatory terminology without introducing unintended interpretations.
  6. Test document extraction. Verify that tables, images, footnotes, and multi-page lists survive ingestion accurately.
  7. Test appropriate refusals. Include questions outside the approved knowledge base and cases where retrieved evidence is incomplete.
  8. Run regression tests after every change. Reevaluate the full question set when documents, retrieval logic, prompts, or models change.
  9. Retest model upgrades. Treat a model change as a system change requiring validation, even when the underlying documents remain the same.

These controls do not guarantee that an AI assistant will always produce correct answers. They provide a more defensible basis for determining whether the system is reliable enough for its intended use.

9 Limitations

This was a small experiment involving five public regulatory documents and 26 test questions. Nineteen questions were scored for answer completeness using a predefined answer key, while five tested appropriate refusals. Two open-ended questions were excluded from automated completeness scoring.

  • Small test set. With 19 scored questions, a single answer changes the completeness score by approximately five percentage points.
  • Limited document complexity. The experiment used a small collection of public regulatory documents rather than a large enterprise knowledge base containing conflicting policies, access restrictions, and frequent updates.
  • Question design. Many questions explicitly identified the relevant document, making retrieval easier than it would be for less specific employee questions.
  • Automated scoring. The answer key measured whether required facts appeared in responses. It did not systematically detect every unsupported addition, misleading statement, or citation error.
  • Limited model comparison. The results reflect specific models, prompts, documents, and retrieval configurations. They do not establish that one model will consistently outperform another.

The results show how specific engineering changes affected this assistant. They do not establish how well the approach would perform across larger document collections or different use cases. I would need a larger question set, independent reviewers, and actual enterprise documents before drawing broader conclusions.

10 Sources and code