skip to content

case study · live on this site · source private

ask the file

Live

Every claim cites a file, or it abstains.

A grounded assistant over this site's own corpus: the one running on the homepage. Five checked stages streamed live: gate, retrieve, scope, route, verify. Unverified text never reaches you.

Defends: intent routing, retrieval engineering, LLM verification, evals discipline.

the problem

Portfolio claims are unverifiable prose

Every portfolio makes claims. Almost none let you check them. A visitor who wants to know what I actually built gets a paragraph they have to take on faith, and a demo that can't answer is decoration, not evidence.

The fix isn't a better paragraph. It's an interrogable surface: ask what was built, where, with what, and get an answer that cites the file it came from, or an honest "my record doesn't cover that." This site is the demo; the dossier it answers from is generated from the same data as the pages you're reading.

the shape

Five checked stages

Each stage emits its own telemetry over SSE; the cells on the homepage render the pipeline's actual output, not a reenactment.

fig. 1 · one ask's pathgate → retrieve → scope → route → verify → done
questionvisitor ask01 gate9 lanes · inj+extra02 sourceshybrid rank + pins03 scopere-judge on evidence04 writerclaims + cites05 checkper-claim vs sourcesanswercited or nothingabstain6 reasonsrepair ×1
evidence path judge refuses → abstain verify fail → one repair pass

the mechanism

How it works

The same description the assistant's corpus carries — every stage below is a cited source the bot can quote back at you.

01 · the gate

One System One call (vllm-sr/d3-flash) scores six criteria before anything is retrieved: intent across nine lanes, prompt injection, extra-task load, write-task asks, contact kind, and language. Scope is deliberately not decided here — a second judge re-decides it after retrieval, with the evidence in hand, so a strong match counts as on-topic.

02 · retrieval

Hybrid search over 80 chunks: BM25 keyword ranking fused with multilingual dense vectors through reciprocal rank fusion (k=60), plus an intent-family vote that keeps the evidence window on-topic. Chunks declare their own canonical intents — a pin inserts the right file only when similarity can't reach it, which is how "does he know rust?" still finds the stack list it never mentions. Closed-list enumerations (stack, credentials, languages, experience) are appended past the window on every ask — always citable, flagged appended so the sources row can tell context apart from rank-earned evidence.

03 · routing

Answers are written through vLLM Semantic Router (repo), a decision layer that picks the model and reasoning effort per question — vllm-sr/auto. Its choice streams in the trace, so a failed model surfacing a retry is visible, not hidden.

04 · verification

The writer's output is a contract — claims plus the chunk ids they cite — checked before any answer text renders. Every number must appear verbatim in a cited source (or echo the question — the asker's own data owes no source); groundedness is scored per claim against each cited source and their union, so a padded citation can't dilute a true one. A claim citing several sources is split into single-fact atoms first, each scored on its own — a compound claim passes only if every atom does, and a failure names the failing atom verbatim so the repair pass rewrites exactly that span. Absence on closed lists (stack, languages, credentials) is a string check, not an opinion. One repair pass, then abstain.

05 · built with

Python + FastAPI inside a Cloudflare Python Worker: rank_bm25 for lexical search, Workers AI bge-m3 embeddings in production (MiniLM locally), vLLM Semantic Router for the gate, the scope judge, and routed generation. The eval sets — 41 smoke + 60 held-out — are the public contract the trace endpoint replays.

the calls

Decisions, and what they beat

the shape

Fixed stages, not an agent loop

Let the model pick its next action in a loop

The control flow is code — models only write or score at fixed points. Stage boundaries are where checks live (a loop has no checkpoints), worst-case cost is a fixed call count, and refusal is a threshold, not a mood the model can talk itself out of.

No tools on purpose

A fetch or search tool for wider coverage

Every claim must cite a chunk id — tool output has no id and would bypass the verifier entirely. A closed corpus also makes 'not covered' decidable: the file is enumerable, so absent means absent — a tool would only mean 'not fetched yet'.

01

Deterministic checks before the judge

One model judge for every check

Citations are a string match, numbers a digit audit — exact, free, binary. Groundedness needs entailment, the one judgment only a model can make, so it runs last and scores claims it never wrote.

02

Verify before stream, not after

Stream the answer, validate when it's done

The writer emits a contract of claims plus the chunk ids they cite, checked claim-by-claim before answer text renders. A failed check gets one repair pass, then the run abstains. Unverified text never reaches the visitor.

03

Pins over hoping retrieval cooperates

Trust dense similarity to surface the right file

"Does he know rust?" never lexically matches the stack list; me.stack can't win a similarity contest against a word it doesn't contain. Intent pins force canonical chunks into the evidence window, with their honest scores.

04

Deterministic absence on closed lists

Ask the judge whether rust is in the stack

Closed-list chunks (stack, languages, credentials) carry a flag; a claim's `absent` field is checked against them by string match. A cited 'no' is verifiable, not another opinion.

05

Two gates, not one

A single pre-retrieval classifier

Gate one classifies; gate two re-decides scope after seeing what retrieval surfaced. Questions about the mechanism itself that never name me still pass: a strong evidence match is itself scope evidence.

06

Abstain is a feature, not a failure

Best-effort answers on thin evidence

Six named refusal reasons (out_of_scope, restricted, injection, not_in_corpus, unverified, absence) each produce a specific answer. The break-it lane exists to exercise them publicly.

07

Atomize at verify, not at the writer

Trust the prompt to keep every claim single-fact

Prompts are advisory — a model still packs availability + two projects + a stack into one claim and cites two of its five sources (real flake: 0.60 on all-true facts). A claim citing several sources decomposes into single-fact atoms at the verifier; the weakest atom sets the score and names itself to the repair pass.

5

checked stages: each timed, each emitted live

9

intent lanes the first gate routes between

80

corpus chunks built from this site's own machine layer

101

eval items: 41 smoke + 60 held-out, run as experiments

proof

The cells tell the truth

A real stream from a live run: every stage emits its payload as it lands, and the cells on the homepage render these numbers as received. The stage ms partition the wall clock to within ~5ms.

run it yourself
live capture · POST /v1/ask → text/event-stream0.0s
gateintent projects · conf 0.99 · inj 0.00vllm-sr/d3-flash · 334ms
retrieve4 chunks · top_dense 0.319 · passMiniLM-L12-v2 · 4ms
scopein_scope 0.98, re-judged with evidence in handscope_ms 115
routewriter contract · calls 1qwen3.8-27b · 9.4s
verifygrounded 0.97 · citations ✓ · numbers ✓d3-flash · 488ms
donestatus answered · 10,379msrequest_id → /v1/trace/{id}
wall clock · real proportions
gate 334msretrieve 4msscope 115msroute 9.4sverify 488msunattributed 38msΣ 10,379ms / wall 10,379ms

honestly

What I'd do differently

  • Dev and prod run different embedding models (MiniLM locally, bge-m3 on Workers AI), so scores and occasional rankings diverge. Parity or a shared embed binding closes it; today the UI just reports which ran.
  • The writer contract should have been first, not second. Prose-first drafts hid exactly the failures the claims+citations contract made legible: the contract is the product, not a test harness.
  • Timing accounting was retrofitted. Internal repair calls billed to nothing and cells couldn't sum to the total: the bug a screenshot caught. Bill per call from the start; aggregates hide work.

fair questions

The questions this work invites

q ·why isn't the repo public?

The code is still moving and carries persona config I don't want to support as an API. The transparency lives at runtime instead: every run's trace is public through /v1/trace/{request_id}, and the eval sets are the contract.

q ·it can still be wrong?

Yes, verify can pass a mis-cited claim and I don't claim otherwise. What bounds it: per-claim checks scoped to cited sources, numeric claims must appear verbatim (or echo the question itself), and every run is replayable from its trace.

q ·what happens when I try to break it?

Injection asks, private-data asks, and off-topic asks each hit a named refusal, not a generic error. The suggestion lanes deliberately include a break-it set so you can watch the gate work.

The demo is the case study. Ask it about this page.

back to selected work