case study · public build
Call → ticket
In progressSupport-call audio in · a structured ticket out, every field traced to a verbatim line.
A pipeline in Python and FastAPI over real diarized call audio · 128.6 hours of it. Speaker-aware extraction, a schema contract, and a supervisor who sees exactly what the model saw. Repo and eval report ship together.
Honest label: workflow with human review · fixed stages, a person on the uncertain fields.
§1 the problem
The facts are
in the audio
A support call contains everything a ticket needs · the issue, the promise, the account details · locked in audio nobody re-listens to. Notes written from memory drop commitments and misattribute who said what.
The fix isn't a summary. It's a ticket where every field points at the verbatim line it came from · and where the uncertain fields go to a person instead of shipping a maybe.
§2 before / after
From memory
to evidence
before
- ✗Agent types the ticket from memory after the call
- ✗Commitments get paraphrased · or dropped entirely
- ✗Disputes mean re-listening to the whole call
- ✗A confident summary hides what it skipped
after
- ✓The ticket drafts itself from the call, speaker-aware
- ✓Every field cites a verbatim line and timestamp
- ✓A promise must quote the agent's turn · not the customer's ask
- ✓Uncertain fields go to a supervisor · designed, not hidden
§3 how it works
Audio, validated fields,
a human on the edge cases
Fixed stages with a schema contract between them. The model writes fields, not steps · and every field it writes must show its source.
- The call lands in the ingest queue · idempotent, bounded retries
- Speech-to-text produces the transcript with speaker labels
- The model drafts ticket fields · each pinned to a verbatim line
- The Pydantic gate validates the shape · invalid retries loudly
- Low-confidence fields route to a supervisor
- The ticket dispatches with its evidence attached
diarization, Pydantic, idempotent, verbatim citation.
§4 walk it step by step
One call,
one cited ticket
A synthetic call · the stages and the shapes are the real spec. Step through with the buttons.
01A real call comes in
ingestCorpus: AppTek diarized call audio · 128.6 hours, ~92k transcript segments. Each call lands in a queue with idempotent retries, so a bad file retries cleanly instead of duplicating.
02Speech becomes labeled text
transcriptionSpeech-to-text with speaker labels · who said it is data, not decoration:
agent "I can get a replacement shipped by Friday." customer "it's the second time this part failed."
A promised action must quote the agent's turn · the customer's ask isn't a commitment.
03Fields extract with citations
extractionThe model drafts the ticket fields · each one pinned to a verbatim line and timestamp:
issue "part failed a second time" ← customer turn, 04:12 promise "replacement shipped by Friday" ← agent turn, 05:31 sentiment "frustrated, escalating" ← customer turns, 03:40–06:02
04The schema gate decides
pydantic · passFields are validated against a declared shape. A malformed extraction doesn't get patched silently · it retries with the validation errors fed back, or the field is marked uncertain.
05A person checks the uncertain fields
human reviewLow-confidence fields route to a supervisor queue · the system says "check this one" instead of shipping a maybe. The ticket shows its evidence, so review takes seconds, not a re-listen.
06Ticket dispatched · with its evidence
ticket · citedThe final ticket carries the verbatim quote behind every field:
ticket #4821 · priority: high issue "the second time this part failed" · customer 04:12 action "replacement shipped by Friday" · agent 05:31 flags repeat failure · escalation risk
No quote, no field · the citation is the contract.
§5 what could go wrong
Failure modes,
handled
- A fabricated commitment. Promised actions must quote the agent's verbatim turn · a customer request can never become an agent promise.
- Silent bad fields. Schema validation retries loudly; low-confidence fields go to a human queue instead of shipping a maybe.
- Speaker mix-ups. Extraction conditions on diarization labels · who said it travels with what was said.
- Retry storms on bad audio. The ingest queue is idempotent with bounded retries · duplicates and partial files can't corrupt the corpus.
§6 status
Built in public,
gated by evals
A public build on a real open corpus · AppTek diarized call audio, 128.6 hours across ~92k transcript segments. Ingest, transcription, extraction, the schema gate, and the review loop are specced; the repo goes public when the eval report does. This page describes the spec · no invented metrics.
technical details⌄
Verbatim-grounded fields: every extracted field cites its transcript line · quote plus timestamp. The ticket shows its evidence.
Speaker-aware extraction: promised actions must quote the agent's turn · asking isn't promising, and the labels enforce it.
Schema as a gate: Pydantic-validated fields; malformed output retries with the errors fed back, never patched silently.
Designed uncertainty: fields below confidence route to a supervisor queue · "check this one" is a designed output.
PII handling: sensitive spans are flagged at the transcript layer, not redacted downstream after the fact.
fair questions
The questions this work invites
q ·what if the audio is bad?
The ingest queue is idempotent with bounded retries · a bad file retries cleanly or lands in the supervisor queue. Nothing dispatches half-processed.
q ·who checks the fields?
Two gates: the Pydantic schema catches malformed shape, and low-confidence fields go to a supervisor. The human is a designed stage, not an afterthought.
q ·where does the model get freedom?
It drafts fields, not steps · the pipeline is fixed. Every field it writes must cite a verbatim transcript line; no quote, no field.