An assistant can retrieve the right policy and still give the wrong answer. That became the most useful finding of my MSc research project: once retrieval was working well, the remaining challenge was how the model selected, interpreted, and cited the evidence in front of it.
For my Independent Research Project at Imperial College London, supervised by Lee Stott at Microsoft and Gerard Gorman at Imperial, I built an on-device retrieval-augmented generation (RAG) copilot using Microsoft Foundry Local. Its job was deliberately narrow: help a reviewer find and explain Know Your Customer (KYC) and anti-money laundering (AML) policy material, with traceable citations and an explicit way to say that the indexed evidence was insufficient.
The prototype used synthetic policies and a fictional onboarding case. It supported human review; it did not approve customers, assign final risk ratings, or make compliance decisions. That boundary shaped both the architecture and the evaluation.
Start with the evidence boundary
The first design question was which documents the assistant should be allowed to cite.
I separated two kinds of information:
- Policy evidence: six synthetic policy documents, indexed and available for citation.
- Case facts: fictional information about a particular onboarding scenario, added to the prompt after retrieval and never treated as citable policy.
That distinction matters. A case document can describe a customer’s circumstances, but it cannot establish the firm’s rules. Mixing both into one retrieval pool makes it easier for an answer to blur that distinction.
The policy-only index contained 75 chunks. Source-list sections remained in the original files for audit but were excluded from retrieval. Each chunk carried a deterministic identifier, such as doc-id#section-id#cN, tying a citation to a specific passage in the frozen corpus.
This made the evidence surface inspectable. It did not, by itself, improve retrieval recall: dense retrieval had the same recall before and after the policy-only boundary was introduced. Separating those effects was one reason to evaluate each layer independently.
A direct pipeline, running locally
Foundry Local supplied local model execution. The application supplied the retrieval logic, evidence rules, citation checks, and evaluation. After model and embedding downloads were cached, the core answer path could run without cloud inference.
Figure 1. The implemented pipeline. Case facts enter after ranking and cannot become policy citations.
The pipeline was intentionally non-agentic: ingest, index, retrieve, assemble the prompt, generate, and validate. A fixed sequence made it easier to locate a failure than a system that could choose additional tools or planning steps.
For indexing, I used section-aware windows of up to 900 characters with 150-character overlap. The evaluated embedding model was all-MiniLM-L6-v2, producing 384-dimensional vectors. For just 75 chunks, a normalised NumPy matrix was sufficient for cosine search.
At query time, hybrid retrieval combined dense and BM25 rankings with reciprocal rank fusion, plus a small rare-entity signal. The final prompt received the top five policy chunks, their allowed IDs, and any separately labelled case facts.
The reported local experiments used a CLI-loaded Foundry Local service through its OpenAI-compatible /v1 endpoint on an RTX 3090. Temperature was 0 and the output limit was 512 tokens. These are the project’s experimental settings; model variants and integration details should be checked against the current Foundry Local documentation when reproducing the design.
Make citations inspectable, then ask what they prove
The prompt asked the model to cite retrieved chunk IDs and to use a canonical response when evidence was insufficient:
Insufficient evidence in the provided policy corpus.
The application classified abstention before citation repair. For other answers, it checked each citation against the retrieved chunks. Exact IDs were accepted; unambiguous near matches could be rewritten; unresolved IDs were removed. Runtime fields recorded valid, repaired, and dropped citations.
This provides a useful structural guarantee: final retained citation IDs can resolve to retrieved passages. It does not establish that a passage supports every claim beside it.
In a controlled, single-run repair comparison, the invalid-or-unresolvable citation incidence fell from 1 of 47 answered rows to 0 of 47. Automated task success and expected-evidence citation hit did not change. Repair fixed an identifier problem; the evaluation still needed to examine evidence use and answer quality.
For a developer, the practical lesson is to track at least three separate states: a citation exists, it resolves, and it actually supports the associated claim. This prototype implemented ID resolution. Claim-level support verification remained future work.
Evaluate the pipeline in four layers
I locked a 60-question benchmark with ten questions in each of six categories: customer due diligence, enhanced due diligence, politically exposed persons, beneficial ownership, escalation, and abstention. Fifty questions were answerable; ten required refusal.
Figure 2. The evaluation separates finding evidence from using it, citing it, and declining to answer.
Retrieval asked whether the expected passages appeared in the top results. Answer quality used a frozen 0–3 rubric based on required concepts and similarity to a reference answer; task success meant a score of at least 2. Citation checks measured expected IDs, missing citations, and invalid identifiers. Refusal and operational checks tracked false answers, false refusals, and latency.
The most revealing results came from reading those measures together:
| Configuration | Evidence recall@5 | Hit@5 | Automated task success | Expected-evidence citation hit |
|---|---|---|---|---|
| Dense retrieval, policy-only | 94.7% | 96.0% | Not measured in this retrieval-only run | Not measured |
| Hybrid retrieval, policy-only | 99.3% | 100.0% | Not measured in this retrieval-only run | Not measured |
| Strong configuration + Phi-3.5-mini | 99.3% | 100.0% | 90.0% | 82.0% |
| Strong configuration + Phi-4 | 99.3% | 100.0% | 98.3% | 100.0% |
Sources: Retrieval and citation-hit measures use the 50 answerable questions; task success uses all 60. The last two rows are from the single model-comparison run on 10 August 2026. A citation hit means at least one expected ID was cited, not that every claim was supported.
With the same retrieved evidence, the generators produced different outcomes. That made the next engineering question much sharper: how could the assistant use available evidence more reliably?
What changed when the model and engineering changed?
To explore that question, I crossed two models—Phi-3.5-mini and Phi-4—with two engineering packages. The weaker package used dense retrieval, no citation repair, and a mixed index. The stronger package used hybrid retrieval, repair, and the policy-only index.
Two complete service sessions reproduced the same primary quality results. Moving from the weaker to the stronger package corresponded to a 3.3-percentage-point increase in task success for each model. Moving from Phi-3.5-mini to Phi-4 corresponded to an 8.3-point increase for either package.
These are descriptive differences on this benchmark. The engineering package bundled several changes, most component comparisons ran once, and the study did not report confidence intervals or establish independent causal effects.
Across six comparable strong/default Phi-4 artefacts, median automated task success was 98.3%, with a range of 96.7–98.3%. Expected-evidence citation hit remained 100%, and final invalid-or-unresolvable citation incidence remained 0%. Those artefacts came from several execution contexts and were not six independent, identically distributed repeats.
Warm median latency ranged from 2.275 to 3.936 seconds on the tested RTX 3090 host, excluding the first question and model/service loading. That is useful context for this prototype, rather than a general speed promise for local models or ordinary laptops.
The failures were as valuable as the scores
Conceptual illustration: evidence retrieval still needs careful interpretation and human review.
One answer used the policy phrase “insufficient evidence” inside a correct, cited explanation. The lexical refusal detector wrongly classified it as an abstention. Another valid refusal used “insufficient information” and was missed.
The answer scorer had a more serious limitation: one response reversed the required enhanced-due-diligence conclusion but still passed the automated task-success threshold. The headline percentage therefore measured the specified rubric and detector, not expert-verified factual accuracy.
That experience changed how I read a good score. I now want the row-level answer, its sources, the refusal decision, and the repair record beside the aggregate. Even the retained logs had a limitation: the evaluation JSON did not preserve every raw-to-repaired detail needed for exact reconstruction.
A practical starting point for student projects
The approach is reusable beyond this particular policy domain:
- Choose a bounded document set. Record versions, dates, and what the assistant may cite.
- Write answerable and unanswerable questions. Check that each reference answer is supported by its expected passages.
- Measure retrieval before generation. A model cannot use evidence that never reaches its prompt.
- Separate citation validity from claim support. Keep repair visible in the output record.
- Hold controls fixed when comparing models. Preserve per-question results and investigate failures, including high-scoring ones.
This project remained a small research prototype: one synthetic bank, 75 chunks, and 60 authored questions, with no held-out policy corpus, compliance-expert validation, user study, or adversarial testing. Its policies reflected a 21 June 2026 snapshot and should not be used as current regulatory guidance.
The next step I proposed was a claim-level verification layer: identify unsupported spans, check whether the available evidence is sufficient, revise only where justified, and abstain or escalate when it is not. That extension was not implemented or evaluated in the report.
Foundry Local made local model execution practical for the experiment. The lasting engineering lesson was to make the evidence path observable: what was retrieved, what was cited, what was changed, and where the system still needed a person to judge the answer.
Project source: Zhifeng Li, An On-Device RAG Copilot for KYC/AML Policy Retrieval Using Microsoft Foundry Local, MSc Independent Research Project final report, Imperial College London, August 2026. Architecture and methods: Sections 3–5; results and limitations: Sections 6–7; proposed extension: Appendix B.
Further reading: What is Foundry Local? · Foundry Local architecture