case study · 01

mdingest

live

Articles in, clean Markdown out. Built for the readers that aren't human.

An ingestion service that turns blog posts and newsletters into LLM-ready Markdown, exposed as a versioned HTTP API, a CLI, and an MCP server, all backed by one core.

defends · api design · multi-surface architecture · agents as users · production ops

live service source ↗← all work

the problem

Feeds aren't machine-legible

My own workflow needed it first: I keep a learning vault and feed articles to LLMs constantly. Medium sits behind a paywall wall, Dev.to and Substack wrap content in markup soup, and copy-paste loses title, author, and date.

Existing scrapers try to be universal: one readability pass for the whole web. The output shows it. The insight: three platforms cover most of what engineers actually read, and each has a structured path to the content. Build for those properly instead of everything badly.

the shape

One core, four doors

Everything resolves to a single ingest(url) that detects the provider and delegates. The surfaces are thin.

GET /v1/{provider}?url=

versioned HTTP API — the public surface

mdingest <url>

CLI for humans and scripts

MCP over stdio

local agents: Claude, Cursor, Devin

MCP over HTTP

remote agents hitting the deployed service

fig. 1 — system mapdeployedrequest in flight

idle live requesturl → detect → fetch → clean → markdown + frontmatter

—ingests served

the calls

Decisions, and what they beat

One ingestion core, four entry points

rejected · Separate implementations per surface

HTTP API, CLI, and MCP (stdio + HTTP) all call the same `ingest()` router. A fix in the core fixes every surface; a new provider ships everywhere at once.

A provider module per source

rejected · Generic HTML scraping for everything

Dev.to has a real JSON API, Substack exposes /api/v1/posts, Medium needs a mirror. Structured sources beat readability heuristics: each provider owns its quirks behind one interface.

Pure Worker — no framework, no container

rejected · Keep the Container running the framework HTTP layer

The core was always framework-free fetch() calls; the container only existed to host an HTTP framework. Migrating to one Worker + ASSETS + native rate limiting dropped the container bill to zero.

Markdown + YAML frontmatter as the output

rejected · JSON with embedded html

The consumers are LLMs and agents. Frontmatter carries title, author, date, reading time, and source_url: metadata they can use without parsing the body.

Validation errors with trace IDs

rejected · Generic 400s

Every failure returns a code (VALIDATION.FAILED, DEVTO.UNAVAILABLE) plus a traceId. When a user reports a bug, the ID is the whole conversation.

ingestion core — every surface calls the same ingest(url)

entry points on one core: API · CLI · MCP stdio · MCP http

providers: medium · devto · substack

container bill after migrating off Cloudflare Containers

proof

A real response

Captured from the live API while building this page: one Dev.to article in, 380ms later, publication-ready Markdown out. Frontmatter carries the metadata an LLM actually needs.

try it live

live capture · GET /v1/devto?url=…200 · 380ms

---
title: "We All Have a \"Serious Work\" AI and a \"Just Vibing\" AI. When Did That Happen?"
subtitle: "Hi Guys!!! As you know I wasn't good for well, a week and Now.....   Let's Dive In!!!  I never..."
author: "Dhruv Jani"
date: "2026-09-22T15:27:32Z"
published: "2026-09-22T15:27:32Z"
reading_time: "4 min read"
free: true
source_url: "https://dev.to/dj29/we-all-have-a-serious-work-ai-and-a-just-vibing-ai-when-did-that-happen-5fl2"
provider: "devto"
tags:
  - "discuss"
  - "ai"
  - "gemini"
  - "antigravity"
---

Hi Guys!!! As you know I wasn't good for well, a week and Now.....
![Friends-gif-where-joey-gest-his-role-back-at-DOOL](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/ngfjxrae0sk3rswb1ych.gif)

Let's Dive In!!!

I never consciously decided which AI gets which job. Somehow, I just ended up with a ranking in my head. It just happened — one small decision at a time, until one day you noticed you have an entire unconscious hierarchy, and you can't fully explain how it got built.

Here's mine, laid bare.

## The IDE has its own pecking order

I use Antigravity as my AI IDE now — GitHub Copilot went pay-to-use, so that door closed. Inside Antigravity, I don't use one model. I use two, and which one shows up depends on what's actually at stake.

honestly

What I'd do differently

Caching should have been day one, not later. Same article requested twice pays the upstream cost twice — a KV layer with content-hash keys is the obvious next layer.
CORS stayed closed longer than it should have. The API is server-callable only today; opening it with rate limits would let people build on it directly.
The provider interface earned its keep — Medium's quirks would have leaked everywhere without it. The pattern I underused: shared retry/backoff middleware instead of per-provider handling.

cross-examination

The questions this file invites

q —who actually uses this?

Me, daily. The vault's ingestion runs through the MCP door, and the HTTP API is public. Adoption beyond me is honestly unmeasured.

q —a url outside the three providers?

A typed VALIDATION.FAILED with a trace id. The matrix is deliberate: three platforms done properly beats the whole web done badly.

q —a provider changes its api tomorrow?

That module breaks while the other doors keep serving. The fix lands in one file behind the provider interface, which exists for exactly this day.

More case studies as the lab ships.

← back to selected work