Building a Grounded RAG Pipeline for a Deriv.com Technical Interview

A retrieval-augmented pipeline with citation validation and numeric grounding checks, built end-to-end in Python.

Context

This system was built as the technical assessment for the Independent AI Engineer position at Deriv.com. It was the second interview in the process — the live 1-hour technical round — and the brief asked for a Retrieval-Augmented Generation (RAG) pipeline that answers questions from a local knowledge base with grounded, cited answers.

I passed that round. Five interviews later, the final call with the VP of Engineering came down to fit for the role rather than the work itself. I did not get the job. The system stands on its own regardless, and the design choices are worth writing up — both for anyone building RAG in production and for the people asking me what I do.

The full source is at github.com/i-ninte/deriv-rag.

The problem RAG is actually solving

A language model with no external context answers from parametric memory. For a fixed knowledge base — an internal wiki, an API reference, a policy document — that is the wrong tool. The answers are unverifiable, they drift as the model changes, and they cannot cite a source.

Retrieval-Augmented Generation reframes the problem. Instead of asking the model to know, we ask it to read. The pipeline:

Indexes the knowledge base as dense vectors.
For each query, retrieves the top-k most relevant passages.
Instructs the model to answer only from those passages, and to cite them.

The interesting engineering is not the retrieval itself. It is what happens around it: chunking that preserves meaning, prompts that refuse to answer when evidence is thin, and validators that catch the model when it invents details the retrieved passages do not support.

System architecture

docs/*.md,*.txt  →  Document Loader  →  Chunker  →  FAISS Index
                                                          │
questions.json  →  Retriever (k=3)  →  Gemini Answer Gen  →  Citation Validator  →  final_answers.json

The pipeline is deterministic and staged. Each stage produces a JSON artifact on disk, so any step can be inspected, replayed, or diffed against a prior run. The stages themselves form a state machine:

INIT → DOCUMENTS_LOADED → CHUNKS_CREATED → INDEX_BUILT
     → QUESTIONS_LOADED → RETRIEVAL_COMPLETE → ANSWERS_GENERATED
     → CITATIONS_VALIDATED → RESULTS_EXPORTED → VALIDATION_COMPLETE

Making the pipeline stateful and inspectable was a deliberate choice. A production RAG system fails silently far more often than it fails loudly: retrieval returns wrong passages, the model over-answers, citations point to chunks that were never retrieved. Persisting every stage's output makes each failure mode addressable in isolation.

Chunking

Documents are split into 500-character chunks with a 50-character overlap. Two design questions matter here.

Why fixed-size and not semantic chunking? For a small, well-structured knowledge base — the kind used in the interview task — fixed chunking is a stronger baseline than it gets credit for. Semantic chunkers introduce a second embedding pass and a threshold that has to be tuned per corpus. Fixed chunking is a single knob (CHUNK_SIZE) and a predictable failure mode: information that spans a boundary might get split. The 50-character overlap directly addresses that failure mode without introducing new ones.

Why 500 characters? It is a compromise. Longer chunks retrieve more context per hit but dilute the embedding — a single vector has to represent more topics. Shorter chunks retrieve tighter matches but fragment coherent passages across multiple hits. 500 characters holds roughly one paragraph in the target corpus, which is the natural unit of a factual claim.

Every chunk carries its position in the source document, so citations resolve to a location, not just an ID.

Retrieval

Chunks are embedded with Google's gemini-embedding-001 model and indexed in FAISS. For each incoming question the retriever returns TOP_K = 3 chunks by inner-product similarity.

Three chunks is a specific choice. With one chunk, a single missed retrieval means a wrong answer. With ten, the prompt fills with distracting context and the model's answer regresses to whichever passage sounds most confident, not most relevant. Three is enough for the model to triangulate — to notice when two passages agree and one disagrees — without drowning the signal.

Answer generation

Gemini 2.5 Flash Lite generates the answer with a prompt that forces three constraints:

Answer only from the retrieved chunks.
Cite the specific chunk IDs used.
If the evidence is insufficient, return an unsupported answer with no citations.

The third constraint is the one that most RAG demos skip. A model instructed only to "cite your sources" will find a citation for any answer it wants to give — the incentive is to produce a fluent output, not an honest one. Making no answer with no citations a legitimate output changes the loss surface.

Every LLM call is logged to llm_calls.jsonl with its prompt, response, and stage. That log has been more useful in debugging than any metric.

Citation validation

This is the layer that took the most iteration and does the most work. Given an answer, its citations, and the retrieved chunks, the validator checks:

Cited chunk IDs exist in the retrieved set. A model that cites a chunk it was never shown has fabricated the source.
Supported answers have at least one citation. Any factual claim must trace to a passage.
Unsupported answers have no citations. A refusal that hedges with a citation is a soft hallucination.
Numeric claims in the answer appear in the cited chunks. This is the grounding check. If the answer says "webhook logs are retained for 90 days," the string 90 days (or 90 in a compatible context) must appear in a cited chunk. Numbers are the failure mode most LLMs share — they are fluent enough to invent a plausible figure that was never in the source.

The validator does not correct the answer. It flags it. Correction is a separate concern: in a production deployment, a flagged answer routes back to a re-generation pass with the flagged claim explicitly excluded, or to a human reviewer. Keeping detection and correction separate keeps the failure surface legible.

What the artifacts enable

Every run writes nine JSON files to artifacts/. Two of them do disproportionate work.

retrieval_results.json — the retrieved chunks per question. When an answer is wrong, this is the first file to open. Either retrieval missed (index or chunking problem) or the model ignored what was retrieved (prompt problem). Reading this file tells you which.

citation_validation.json — the validator's verdict on each answer. Aggregated across a run, this is the pipeline's honest self-report: how many answers were supported, how many were flagged, which specific numeric claims failed grounding.

A pipeline that cannot audit itself is a pipeline you cannot trust in production.

What I would change

Retrieval reranking. For the interview corpus, top-3 dense retrieval was sufficient. On a larger corpus with adversarial or ambiguous queries, a cross-encoder reranker over the top-20 dense hits would substantially improve precision at k=3. The cost is a second model call per query, which is acceptable for most non-real-time use cases.

Hybrid retrieval. Dense retrieval loses on queries that hinge on exact tokens — product codes, error identifiers, API endpoints. A BM25 pass in parallel with the dense retrieval, with reciprocal rank fusion, catches these. It is a small addition with an outsized effect on specific-fact queries.

Answer confidence, not just support. The current pipeline returns supported or unsupported. A more useful signal is a per-claim confidence, derived from the overlap between the claim's tokens and the cited chunk's tokens, plus the retrieval score. This lets downstream systems set a threshold rather than accept a binary.

Evaluation harness. A dataset of question-answer pairs with gold citations, run every pipeline change, tracking retrieval precision at k, answer faithfulness, and citation accuracy. This is the discipline the pipeline is missing.

Closing

The design principle throughout was to keep failure modes legible. Every stage writes an artifact. Every LLM call is logged. Every answer is validated against constraints it cannot argue with. The alternative — a monolithic function that goes from query to answer — is faster to build and impossible to trust.

The interview outcome does not change the ideas. If anything, having the system live outside the context it was built for is what makes it worth writing about.

Source: github.com/i-ninte/deriv-rag

Kwabena Obeng

Comments (0)

Leave a comment