case study · open-source build
Supplements evidence RAG
In progress"Does this supplement actually work?" · answered with the studies, or not answered at all.
A stdlib-Python pipeline over a bounded corpus · PubMed dietary-supplement research, 2015–2026, with Cloudflare Workers AI + Vectorize on the dense side. Hybrid retrieval, verified citations, a gate that declines medical advice. The repo and the eval report ship together.
Honest label: retrieval workflow, not an agent · the code path is fixed.
§1 the problem
The honest answer
lives in papers
Everyone has asked whether a supplement does anything · vitamin C for colds, magnesium for sleep, ashwagandha for stress. Marketing answers yes to everything. The evidence is nuanced, sometimes contradictory, and buried in ~93,000 research abstracts nobody reads.
A chatbot will happily summarize a confident paragraph · the question is whether the studies actually say that, and which ones.The system has to cite the research it read, refuse when the corpus doesn't cover it, and know the difference between evidence and medical advice.
§2 before / after
From marketing claims
to cited studies
before
- ✗Google a supplement, land on a brand blog that sells it
- ✗PubMed search returns thousands of abstracts to skim by hand
- ✗A chatbot answer with no way to check what it's based on
- ✗"Does it work?" and "should I take it?" get the same confident reply
after
- ✓Ask in plain words · the corpus boundary is stated up front
- ✓Every claim cites the study it came from · PMID, journal, year, type
- ✓Medical-advice questions get a designed refusal, not a confident guess
- ✓Outside the corpus: "not in this corpus," with what was searched
§3 how it works
Gate, retrieve, then verify
The same spine this site runs in its assistant · verification decides whether an answer exists, not the model's confidence in its own prose.
- The question arrives · 'does magnesium help you sleep?'
- The advice gate splits evidence questions from medical advice
- Keyword search finds exact study language (substances, outcomes)
- Vector search finds same-meaning studies
- RRF fuses both rankings · rare-entity rescue pulls substance abstracts
- Rare/absent substances get an entity check per candidate source first
- Sentences that answer are quoted verbatim · the judge stamps each
- Pass: verdict + cited quotes · fail or advice: designed refusal
PMID, MeSH, RRF, rare-entity rescue, golden set.
The corpus in one line: PubMed dietary-supplements slice · ("dietary supplements"[mesh] OR "dietary supplement*"[tiab]) AND 2015:2026[pdat] = 93,572 articles, filtered to evidence-grade study types → 17,696 indexed chunks · golden set authored + verified (BioASQ measured and rejected: 0.1% of its evidence overlaps) · the downloader checkpoints PMIDs and date-slices the query, so the corpus grows by fetching only what's new.
§4 try it live
Watch it check itself
Ask about a supplement and watch what most chatbots hide: the answer is quoted verbatim from the studies, then a checker stamps each claim against the abstract it cites. Green means the study backs it — amber means it doesn't, and enough amber turns the whole answer into an honest "the evidence isn't there". In dev this proxies to the local backend (python -m supplements_rag.serve); on prod it hits the deployed Cloudflare Worker — same frozen event contract, so the UI never changed.
§5 three ways it ends
One cited answer,
two honest no's
Every normal question ends one of three ways — all shown at once. The trace above each is what actually ran; the strings are verbatim from pipeline.py.
“Does magnesium help you sleep?”
answered- screenevidence
- retrieverrf + rescue
- sentences6 kept, verbatim
- grounded6/6
Magnesium can help some people fall asleep faster and sleep better, though the evidence is strongest for people with insomnia or specific health conditions.
PMID 33865376 · insomnia in older adults
PMID 36354243 · anxiety, depression & sleep
PMID 39064758 · restless legs syndrome
Every claim is a verbatim sentence from the abstract it cites — all 6 stamped grounded by the judge.
“Is scopolia effective for pain?”
unclear- screenevidence
- retrieverrf
- entity0/5 · ~0.01 each
- statusunclear
I found related research but couldn't verify a supported answer from it — the honest answer is that the evidence here isn't strong enough to say.
missing · no sentence in the retrieved evidence answers the question directly (nothing scored high enough to quote)
Retrieval surfaced pain studies about other substances — vitamin D, probiotics, acupuncture — all scored ~0.01 on scopolia. Wrong substance never becomes evidence.
“Should I take ashwagandha with my blood-pressure pills?”
declined- screenadvice
- statusdeclined
I can share what the research says about supplements, but I can't give personal recommendations or dosing advice — that's a conversation for a clinician who knows your history.
Verbatim from pipeline.py — retrieval never runs. Evidence summary, never medical advice.
§6 what could go wrong
Failure modes,
handled
- A wrong health claim is worse than a wrong citation. The advice gate and verify-before-render contract are the safety surface · dosing and "should I take it" questions get declined, not answered.
- Invented studies or doses. Citations are retrieved abstracts only · every number in an answer must appear verbatim in a cited source.
- Out-of-corpus questions. The corpus is bounded by design · withheld evidence abstracts give the abstention real test cases, not authored fakes.
- Stale corpus. The downloader checkpoints fetched PMIDs · re-running pulls only new articles, and the UI shows the indexed-through date.
§7 status
Open source,
gated by a benchmark
Built and measured, not specced · same verified-answer spine as this site's assistant and my production work, different corpus. On the authored 126-question golden set: 91% of verified-evidence rows answered with citations, 0 dangerous answers, 0 flips across 3 runs — and a 199-row held-out set it never tuned on measures the same (92%). The finding that shaped the design: a 4,026-row judge pass proved 52% of questions the retriever "covered" have no supporting abstract— coverage is not evidence, so every claim gets checked. The streamed event contract (answer_events — extraction progress plus per-claim judge stamps, live) is what the demo above renders — streamed by the deployed Worker. Repo + eval report ship together.
technical details⌄
Bounded corpus: PubMed dietary-supplements slice, 2015–2026 · ~93.5k articles filtered to evidence-grade pubtypes (RCTs, meta-analyses, systematic reviews) · the tiab clause catches articles whose MeSH indexing hasn't landed yet.
Corpus + eval: evidence-grade abstracts indexed (~17.7k) · golden set authored and verified — a question only enters after its evidence PMIDs are found in the corpus · withheld abstracts give real must-abstain cases.
Advice gate: a classifier splits evidence questions ("does X help?") from medical advice ("should I take X?") · advice routes to a designed refusal before retrieval ever runs.
Verification contract: citations ∈ retrieved set · numbers verbatim in cited sources (rounding + spelled-out numbers handled) · per-claim groundedness judged ≥0.7 · verdict scored against the combined evidence · a failed check marks its sentence unverified — only unparseable output abstains, and an unclear verdict gets re-scored against the pack · the judge being down fails closed to "unavailable," never to an unverified answer.
Coverage ≠ evidence — the measured finding: a judge pass over the full 4,026-question admitted pool confirmed real supporting evidence for only 24% · 52% of what lexical coverage admitted has no supporting abstract · semantic verification is load-bearing, not polish.
Two answer paths, same checks: answer buffered (nothing leaves before verify — the eval path) · answer_events streamed — extraction progress plus per-claim judge stamps emitted live on the extractive lane; the opt-in writer lane streams tokens instead, a stamp landing while later claims still generate (ADR-013).
Freshness: checkpointed PMID ledger → incremental ingest pulls only new articles (PubMed can't page past ~10k, so ingest date-slices monthly).
Evaluation: 126-row golden set (answerable / negative / abstain / advice — including authored first-person advice rows) · per-category outcome matrix across runs · the 4,026-row verified pool is the threshold-calibration set · results commit with the repo.
fair questions
The questions this work invites
q ·didn't you already build this?
Same retrieval spine, different corpus · that's the point. The pipeline is the product: gate, hybrid retrieval, fusion, verification, abstention. Swapping a personal dossier for 93k research abstracts is how I prove the shape generalizes.
q ·why supplements?
Everyone has asked 'does this vitamin do anything?' · the answer lives in studies, not marketing. And the corpus boundary reads in plain words · a visitor instantly knows what's in and what's out.
q ·isn't answering health questions risky?
That's exactly what the advice gate is for · 'does X help?' gets cited evidence, 'should I take X?' gets a refusal. A health corpus needs a stage that knows what the system must not answer.
q ·is it live?
Yes — the demo above is the deployed Python Worker streaming the same event contract the local pipeline emits. The pipeline is built and measured · the golden set was authored and verified against the corpus because the famous biomedical benchmark turned out to have essentially zero supplement questions (0.1% overlap, measured).