skip to content

case study · public build

Call → ticket

In progress

Support-call audio in · a structured ticket out, every field traced to a verbatim line.

A pipeline in Python and FastAPI over real diarized call audio · 128.6 hours of it. Speaker-aware extraction, a schema contract, and a supervisor who sees exactly what the model saw. Repo and eval report ship together.

Honest label: workflow with human review · fixed stages, a person on the uncertain fields.

§1 the problem

The facts are
in the audio

A support call contains everything a ticket needs · the issue, the promise, the account details · locked in audio nobody re-listens to. Notes written from memory drop commitments and misattribute who said what.

The fix isn't a summary. It's a ticket where every field points at the verbatim line it came from · and where the uncertain fields go to a person instead of shipping a maybe.

§2 before / after

From memory
to evidence

before

  • ✗Agent types the ticket from memory after the call
  • ✗Commitments get paraphrased · or dropped entirely
  • ✗Disputes mean re-listening to the whole call
  • ✗A confident summary hides what it skipped

after

  • ✓The ticket drafts itself from the call, speaker-aware
  • ✓Every field cites a verbatim line and timestamp
  • ✓A promise must quote the agent's turn · not the customer's ask
  • ✓Uncertain fields go to a supervisor · designed, not hidden

§3 how it works

Audio, validated fields,
a human on the edge cases

Fixed stages with a schema contract between them. The model writes fields, not steps · and every field it writes must show its source.

Call → ticket pipelineCall audio lands in an idempotent queue, speech-to-text adds speaker labels, the model extracts fields with verbatim citations, a Pydantic gate validates, and a supervisor takes the uncertain fields.validinvalidcall audiodiarizedingest queueidempotent retriesspeech-to-textspeaker labelsextract fieldsllm + verbatim citespydantic gateschema contractretryerrors fed backsupervisorlow-confidence fieldsticketdispatched
  1. The call lands in the ingest queue · idempotent, bounded retries
  2. Speech-to-text produces the transcript with speaker labels
  3. The model drafts ticket fields · each pinned to a verbatim line
  4. The Pydantic gate validates the shape · invalid retries loudly
  5. Low-confidence fields route to a supervisor
  6. The ticket dispatches with its evidence attached

diarization, Pydantic, idempotent, verbatim citation.

§4 walk it step by step

One call,
one cited ticket

A synthetic call · the stages and the shapes are the real spec. Step through with the buttons.

spec walkthrough · warranty callIllustrative example · synthetic data

01A real call comes in

ingest

Corpus: AppTek diarized call audio · 128.6 hours, ~92k transcript segments. Each call lands in a queue with idempotent retries, so a bad file retries cleanly instead of duplicating.

02Speech becomes labeled text

transcription

Speech-to-text with speaker labels · who said it is data, not decoration:

agent     "I can get a replacement shipped by Friday."
customer  "it's the second time this part failed."

A promised action must quote the agent's turn · the customer's ask isn't a commitment.

03Fields extract with citations

extraction

The model drafts the ticket fields · each one pinned to a verbatim line and timestamp:

issue     "part failed a second time"      ← customer turn, 04:12
promise   "replacement shipped by Friday"  ← agent turn, 05:31
sentiment "frustrated, escalating"         ← customer turns, 03:40–06:02

04The schema gate decides

pydantic · pass

Fields are validated against a declared shape. A malformed extraction doesn't get patched silently · it retries with the validation errors fed back, or the field is marked uncertain.

05A person checks the uncertain fields

human review

Low-confidence fields route to a supervisor queue · the system says "check this one" instead of shipping a maybe. The ticket shows its evidence, so review takes seconds, not a re-listen.

06Ticket dispatched · with its evidence

ticket · cited

The final ticket carries the verbatim quote behind every field:

ticket #4821 · priority: high
issue    "the second time this part failed" · customer 04:12
action   "replacement shipped by Friday"    · agent 05:31
flags    repeat failure · escalation risk

No quote, no field · the citation is the contract.

§5 what could go wrong

Failure modes,
handled

  • A fabricated commitment. Promised actions must quote the agent's verbatim turn · a customer request can never become an agent promise.
  • Silent bad fields. Schema validation retries loudly; low-confidence fields go to a human queue instead of shipping a maybe.
  • Speaker mix-ups. Extraction conditions on diarization labels · who said it travels with what was said.
  • Retry storms on bad audio. The ingest queue is idempotent with bounded retries · duplicates and partial files can't corrupt the corpus.

§6 status

Built in public,
gated by evals

A public build on a real open corpus · AppTek diarized call audio, 128.6 hours across ~92k transcript segments. Ingest, transcription, extraction, the schema gate, and the review loop are specced; the repo goes public when the eval report does. This page describes the spec · no invented metrics.

technical details⌄

Verbatim-grounded fields: every extracted field cites its transcript line · quote plus timestamp. The ticket shows its evidence.

Speaker-aware extraction: promised actions must quote the agent's turn · asking isn't promising, and the labels enforce it.

Schema as a gate: Pydantic-validated fields; malformed output retries with the errors fed back, never patched silently.

Designed uncertainty: fields below confidence route to a supervisor queue · "check this one" is a designed output.

PII handling: sensitive spans are flagged at the transcript layer, not redacted downstream after the fact.

fair questions

The questions this work invites

q ·what if the audio is bad?

The ingest queue is idempotent with bounded retries · a bad file retries cleanly or lands in the supervisor queue. Nothing dispatches half-processed.

q ·who checks the fields?

Two gates: the Pydantic schema catches malformed shape, and low-confidence fields go to a supervisor. The human is a designed stage, not an afterthought.

q ·where does the model get freedom?

It drafts fields, not steps · the pipeline is fixed. Every field it writes must cite a verbatim transcript line; no quote, no field.