case study · live on this site · source private
ask the file
LiveEvery claim cites a file, or it abstains.
A grounded assistant over this site's own corpus: the one running on the homepage. Five checked stages streamed live: gate, retrieve, scope, route, verify. Unverified text never reaches you.
Defends: intent routing, retrieval engineering, LLM verification, evals discipline.
the problem
Portfolio claims are unverifiable prose
Every portfolio makes claims. Almost none let you check them. A visitor who wants to know what I actually built gets a paragraph they have to take on faith, and a demo that can't answer is decoration, not evidence.
The fix isn't a better paragraph. It's an interrogable surface: ask what was built, where, with what, and get an answer that cites the file it came from, or an honest "my record doesn't cover that." This site is the demo; the dossier it answers from is generated from the same data as the pages you're reading.
the shape
Five checked stages
Each stage emits its own telemetry over SSE; the cells on the homepage render the pipeline's actual output, not a reenactment.
the mechanism
How it works
The same description the assistant's corpus carries — every stage below is a cited source the bot can quote back at you.
01 · the gate
One System One call (vllm-sr/d3-flash) scores six criteria before anything is retrieved: intent across nine lanes, prompt injection, extra-task load, write-task asks, contact kind, and language. Scope is deliberately not decided here — a second judge re-decides it after retrieval, with the evidence in hand, so a strong match counts as on-topic.
02 · retrieval
Hybrid search over 80 chunks: BM25 keyword ranking fused with multilingual dense vectors through reciprocal rank fusion (k=60), plus an intent-family vote that keeps the evidence window on-topic. Chunks declare their own canonical intents — a pin inserts the right file only when similarity can't reach it, which is how "does he know rust?" still finds the stack list it never mentions. Closed-list enumerations (stack, credentials, languages, experience) are appended past the window on every ask — always citable, flagged appended so the sources row can tell context apart from rank-earned evidence.
03 · routing
Answers are written through vLLM Semantic Router (repo), a decision layer that picks the model and reasoning effort per question — vllm-sr/auto. Its choice streams in the trace, so a failed model surfacing a retry is visible, not hidden.
04 · verification
The writer's output is a contract — claims plus the chunk ids they cite — checked before any answer text renders. Every number must appear verbatim in a cited source (or echo the question — the asker's own data owes no source); groundedness is scored per claim against each cited source and their union, so a padded citation can't dilute a true one. A claim citing several sources is split into single-fact atoms first, each scored on its own — a compound claim passes only if every atom does, and a failure names the failing atom verbatim so the repair pass rewrites exactly that span. Absence on closed lists (stack, languages, credentials) is a string check, not an opinion. One repair pass, then abstain.
05 · built with
Python + FastAPI inside a Cloudflare Python Worker: rank_bm25 for lexical search, Workers AI bge-m3 embeddings in production (MiniLM locally), vLLM Semantic Router for the gate, the scope judge, and routed generation. The eval sets — 41 smoke + 60 held-out — are the public contract the trace endpoint replays.
the calls
Decisions, and what they beat
the shape
Fixed stages, not an agent loop
Let the model pick its next action in a loop
The control flow is code — models only write or score at fixed points. Stage boundaries are where checks live (a loop has no checkpoints), worst-case cost is a fixed call count, and refusal is a threshold, not a mood the model can talk itself out of.
No tools on purpose
A fetch or search tool for wider coverage
Every claim must cite a chunk id — tool output has no id and would bypass the verifier entirely. A closed corpus also makes 'not covered' decidable: the file is enumerable, so absent means absent — a tool would only mean 'not fetched yet'.
Deterministic checks before the judge
One model judge for every check
Citations are a string match, numbers a digit audit — exact, free, binary. Groundedness needs entailment, the one judgment only a model can make, so it runs last and scores claims it never wrote.
Verify before stream, not after
Stream the answer, validate when it's done
The writer emits a contract of claims plus the chunk ids they cite, checked claim-by-claim before answer text renders. A failed check gets one repair pass, then the run abstains. Unverified text never reaches the visitor.
Pins over hoping retrieval cooperates
Trust dense similarity to surface the right file
"Does he know rust?" never lexically matches the stack list; me.stack can't win a similarity contest against a word it doesn't contain. Intent pins force canonical chunks into the evidence window, with their honest scores.
Deterministic absence on closed lists
Ask the judge whether rust is in the stack
Closed-list chunks (stack, languages, credentials) carry a flag; a claim's `absent` field is checked against them by string match. A cited 'no' is verifiable, not another opinion.
Two gates, not one
A single pre-retrieval classifier
Gate one classifies; gate two re-decides scope after seeing what retrieval surfaced. Questions about the mechanism itself that never name me still pass: a strong evidence match is itself scope evidence.
Abstain is a feature, not a failure
Best-effort answers on thin evidence
Six named refusal reasons (out_of_scope, restricted, injection, not_in_corpus, unverified, absence) each produce a specific answer. The break-it lane exists to exercise them publicly.
Atomize at verify, not at the writer
Trust the prompt to keep every claim single-fact
Prompts are advisory — a model still packs availability + two projects + a stack into one claim and cites two of its five sources (real flake: 0.60 on all-true facts). A claim citing several sources decomposes into single-fact atoms at the verifier; the weakest atom sets the score and names itself to the repair pass.
5
checked stages: each timed, each emitted live
9
intent lanes the first gate routes between
80
corpus chunks built from this site's own machine layer
101
eval items: 41 smoke + 60 held-out, run as experiments
proof
The cells tell the truth
A real stream from a live run: every stage emits its payload as it lands, and the cells on the homepage render these numbers as received. The stage ms partition the wall clock to within ~5ms.
run it yourselfhonestly
What I'd do differently
- Dev and prod run different embedding models (MiniLM locally, bge-m3 on Workers AI), so scores and occasional rankings diverge. Parity or a shared embed binding closes it; today the UI just reports which ran.
- The writer contract should have been first, not second. Prose-first drafts hid exactly the failures the claims+citations contract made legible: the contract is the product, not a test harness.
- Timing accounting was retrofitted. Internal repair calls billed to nothing and cells couldn't sum to the total: the bug a screenshot caught. Bill per call from the start; aggregates hide work.
fair questions
The questions this work invites
q ·why isn't the repo public?
The code is still moving and carries persona config I don't want to support as an API. The transparency lives at runtime instead: every run's trace is public through /v1/trace/{request_id}, and the eval sets are the contract.
q ·it can still be wrong?
Yes, verify can pass a mis-cited claim and I don't claim otherwise. What bounds it: per-claim checks scoped to cited sources, numeric claims must appear verbatim (or echo the question itself), and every run is replayable from its trace.
q ·what happens when I try to break it?
Injection asks, private-data asks, and off-topic asks each hit a named refusal, not a generic error. The suggestion lanes deliberately include a break-it set so you can watch the gate work.
The demo is the case study. Ask it about this page.
back to selected work