Serve Your Docs as Markdown: .md Mirrors, Content Negotiation, and Agent Banners

In short: Every documentation URL should serve clean markdown to agents. Append .md to any canonical URL and return text/markdown, and serve markdown on the canonical URL itself to non-browser clients (content negotiation). Stripe, Supabase, Next.js, and Prisma all converged on this; a 9-probe CI suite keeps it from silently breaking.

Works in: Research, Shortlist & Act. Clean docs get read, then compared head-to-head, then become the implementation reference.

Do this now

• Make .md work on every canonical docs URL: HTTP 200, Content-Type: text/markdown (or text/plain), non-empty body.
• Mirror every HTML redirect into .md space: if /docs/old → /docs/new, then /docs/old.md → /docs/new.md (Supabase does; most hand-rolled setups forget).
• Add content negotiation: non-browser user agents get markdown at the canonical URL itself (the Next.js gold standard), with Vary: Accept, User-Agent set.
• Put YAML frontmatter on every .md page: url, docs_index, version, lastUpdated, prerequisites.
• Add the agent discovery banner pointing to llms.txt at the top of every .md page.
• Ship a machine-readable changelog.md at the site root.
• Add "Copy as Markdown" / "Ask ChatGPT" / "Ask Claude" buttons to HTML pages (the Supabase pattern).
• Wire the 9-probe CI suite into every docs deploy; any failure blocks release.

Scope: Play 5 built the index file; this page makes every page that index links to agent-readable. What goes inside the code blocks on those pages is Play 7.

Why serve markdown when agents can parse HTML?

Because a meaningful share of agent traffic never runs an HTML parser, and the rest prefer not to. As of 2026-06-11, Context7 fetch traffic includes Python HTTPX at 1.8% and Go HTTP clients at 0.8%: roughly 2.6% of queries are raw scripts fetching docs directly. The major agents (Claude Code 43.4%, Opencode 15.3%, Codex 14.0%, Cursor 6.5%) handle HTML but strongly favor markdown: it's denser per token, with no nav chrome or cookie-banner noise.

Two more reasons from the data. First, clean markdown is what indexes parse well. Convex's docs-site Context7 entry benchmarks 91.6 versus 79.9 for its repo entry, because sites are pre-filtered for user-facing content. Second, mirrors generated by your docs build inherit freshness automatically, and freshness is the strongest benchmark correlate we measured (Spearman −0.54 across n=17 entries; freshest-5 average 83.6 vs. stalest-5 at 72.3). A separately maintained export rots; a build-generated mirror can't.

And this is the delivery vehicle for fixing stale agent output. In our pilot trials (single model, n=2 per arm), agents with current doc text in context went from 2/2 emitting obsolete Tailwind v3 config to 0/2. A fetchable .md page with a lastUpdated stamp is how that current text reaches the agent.

Who needs this play: every docs site. On a Mintlify-class platform it's a settings check (half a day); hand-rolled, it's 2–5 days plus a day for the CI suite.

What is the .md suffix convention?

Append .md to any canonical docs URL and get the same page as raw markdown. The leaders converged on it independently, and convergent evolution across competitors is about the strongest "this is table stakes" signal there is. Verified 2026-06-11:

The redirect detail matters more than it looks: docs reorgs are common, agents and indexes memorize deep links, and a .md mirror without mirrored redirects 404s its entire history after the first reorg. Prisma's versioned redirects and Supabase's preserved .md redirects are the patterns to copy.

What is content negotiation, and why is Next.js the gold standard?

Content negotiation means the canonical URL (no suffix) serves markdown when the requester isn't a browser. Next.js is the verified gold standard here: during our audit it served markdown to our non-browser fetcher on canonical URLs, no .md suffix needed. The .md convention requires the agent to guess; negotiation catches the agents that never guessed.

Implementation lives in middleware: if the Accept header lacks text/html, or the user agent matches a known agent/CLI list (curl, HTTPX, Go-http-client, claude, etc.), serve the markdown rendering at the HTML URL. Two rules:

1. Always send Vary: Accept, User-Agent. Without it your CDN caches the markdown variant and serves it to browsers (or vice versa).
2. Keep the .md suffix working anyway, for explicit requests and for llms.txt links.

A sketch of the hand-rolled .md route (Next.js):

export async function GET(req: Request, { params }: Ctx) {
  const slug = params.slug.join("/");
  if (!slug.endsWith(".md")) return next();
  const path = slug.slice(0, -3);
  const redirect = redirectMap[path];               // one map, two route handlers
  if (redirect) return Response.redirect(`/docs/${redirect}.md`, 308);
  const doc = await getDoc(path);
  if (!doc) return new Response("Not found", { status: 404 });
  return new Response(frontmatter(doc) + banner() + renderToMarkdown(doc), {
    headers: { "Content-Type": "text/markdown; charset=utf-8" },
  });
}

The critical function is renderToMarkdown: it must execute or strip MDX so no import X from '@mdx/...' survives into agent context (the verified Drizzle flaw, where their 94KB export leaks Astro build imports).

What frontmatter do agents need?

The Next.js/Vercel union schema, on every .md page:

---
url: https://paykit.dev/docs/webhooks        # canonical HTML URL
docs_index: https://paykit.dev/docs/llms.txt # discovery pointer
version: 3.4.1
lastUpdated: 2026-06-11
type: how-to            # Vercel: tutorial | how-to | reference
prerequisites:
  - /docs/installation
related:
  - /docs/api/errors
---

url lets an agent cite the human page; docs_index makes every page a discovery entry point; version plus lastUpdated let the agent judge staleness instead of guessing; prerequisites and related give it the next fetch without a search round-trip.

What is the agent banner, and what else goes on the page?

The banner is a one-line discovery pointer at the top of every .md page. Mintlify generates it across its fleet, observed verbatim on Polar, Bun, Crossmint, x402, and DodoPayments as of 2026-06-11:

"## Documentation Index. Fetch the complete documentation index at .../llms.txt. Use this file to discover all available pages before exploring further."

That's the whole job: whatever page an agent lands on first, it learns where the map is. Two companions:

• Footer pointer to a markdown sitemap (/docs/sitemap.md, the Next.js pattern): the bottom-of-page equivalent.
• changelog.md at the site root: machine-readable release notes, newest first. Prisma's llms.txt makes it a standing order: "First, fetch https://www.prisma.io/changelog.md to check for recent or relevant breaking changes." This file is what makes your version stamps actionable.

On your highest-traffic quickstarts, go one step further and write for the agent: Clerk's quickstarts/nextjs.md carries "Rules: ALWAYS/NEVER" lists, a "## Verify Before Responding" checklist, and Keyless Mode notes ("Do NOT tell users to sign up, create accounts, get API keys"). Explicit rules an LLM can quote beat narrative it must interpret.

Should you add "Copy as Markdown" and Ask-AI buttons?

Yes. It's the human-side reinforcement of the same surface. Supabase's HTML pages carry "Copy as Markdown," "Ask ChatGPT," and "Ask Claude" buttons; the deep links are pre-filled with "Read from {url} so I can ask questions about its contents." Every click routes a human-initiated agent session straight to your markdown, and trains users that your docs are agent-readable.

Implementation is one static <a href> per assistant with the URL template filled at render time. Ship it last: buttons compound on working mirrors but fix nothing alone.

Platform default or hand-rolled?

If you're on a Mintlify-class platform: .md mirrors, the agent banner, and llms.txt are platform defaults, verified across Polar, Bun, Crossmint, x402, and DodoPayments. Verify they're enabled, run the probe suite, done in half a day. This is also the cheapest first move if your docs are currently hand-rolled: of the eight properties that make docs agent-grade, five arrive by platform default.

If you're hand-rolling (Next.js/Astro/Docusaurus): you own three pieces, the .md route, the MDX-to-markdown renderer, and the redirect map. Budget 2–5 days, plus a day for the CI suite. Rollout order: top-20 traffic pages first (they cover most agent fetches), then full sitemap coverage, then redirects, then content negotiation, then buttons and rule blocks.

The failure modes are all verified in the wild, so treat them as the test plan: Drizzle and Tailwind both return empty .md bodies (a 200 with no content is worse than a 404, since the agent concludes the page is blank); Better Auth's mirrors work only under a nonstandard /llms.txt/docs/... prefix, where no agent guesses; and a Cloudflare JS challenge on .md routes silently zeroes agent traffic while human metrics look fine.

How do you keep mirrors from silently breaking?

Automate a CI job (docs-mirror-probe) on every deploy; any failure blocks release.

1. Suffix probe: every sitemap URL + .md → 200, text/(markdown|plain), body > 200 bytes, body doesn't start with <!DOCTYPE or <html.
2. MDX leak probe: body matches neither ^import .+ from nor <[A-Z][A-Za-z]+[ />].
3. Frontmatter probe: body begins --- and contains url:, docs_index:, version:, lastUpdated:; lastUpdated within 90 days on top-20 pages.
4. Redirect probe: every redirect-map entry's .md form returns 3xx with a Location ending in .md.
5. Negotiation probe: a python-httpx user agent gets markdown at the canonical URL; a browser UA gets HTML; response includes Vary.
6. Banner probe: every .md body contains the literal string llms.txt.
7. Changelog probe: /changelog.md returns 200 markdown and its newest entry matches the current release.
8. Through-CDN probe: repeat probes 1 and 5 against the production edge, not origin.
9. Empty-body canary: assert non-empty bodies on 10 randomly sampled pages per deploy, the explicit regression test for the Drizzle/Tailwind failure.

The research layer. All observations made 2026-06-11; updates logged in the Data Room.

Probe results, working vs. broken:

Why sites beat repos as the indexed corpus: Convex docs-site entry benchmark 91.6 vs. 79.9 for the repo entry, an 11.7-point gap attributable to README/contributor noise. Your .md mirror makes the site the clean indexable corpus.

Freshness data (n=17 audited Context7 entries, 2026-06-11): benchmark vs. log(hours since update) Spearman −0.54, the strongest correlate found; freshest-5 average 83.6, stalest-5 average 72.3. Build-generated mirrors inherit freshness; exports rot.

Experiment E3 (pilot-grade: single model, n=2 per arm): on a Tailwind v4 setup task, control agents emitted obsolete v3 config 2/2 (one prescribed npx tailwindcss init -p, a command that no longer exists). With current doc text in context: 0/2. The mirror is how that text gets fetched.

Verbatim, from the field:

"Read from {url} so I can ask questions about its contents" (supabase.com docs, the pre-filled prompt behind its "Ask ChatGPT" / "Ask Claude" buttons)

"Do NOT tell users to sign up, create accounts, get API keys" (clerk.com quickstart .md files, documentation written as a prompt, not a page)

FAQ

How do I serve my docs as markdown to AI agents? Two mechanisms, ship both: a .md suffix on every canonical docs URL returning text/markdown, and content negotiation that serves markdown at the canonical URL to non-browser user agents. Mintlify-class platforms do the first by default; Next.js demonstrates the second.

What is content negotiation for documentation? Content negotiation serves different formats at the same URL based on who's asking: browsers get HTML, agents and scripts get markdown. Detect via the Accept header or user-agent string, and always set Vary: Accept, User-Agent so CDNs cache both variants correctly. Next.js verifiably does this on its canonical docs URLs.

Do AI agents read HTML pages? Most can, but markdown is denser per token and free of nav/cookie noise. And roughly 2.6% of Context7 fetch traffic comes from raw HTTP clients (Python HTTPX, Go) that never render anything, as of 2026-06-11. Serving markdown removes the parsing tax for every reader.

What frontmatter should .md docs pages have? At minimum: url (canonical HTML page), docs_index (pointer to llms.txt), version, and lastUpdated. Next.js adds prerequisites; Vercel adds type and related. The version and date fields are what let an agent judge staleness instead of guessing.

Why does my .md URL return an empty page? Usually the route exists but the renderer produces nothing, verified at Drizzle and Tailwind, both returning empty bodies with status 200. An empty 200 is worse than a 404 because the agent concludes the page has no content. Probe for body length, not just status codes.

What is the banner at the top of .md docs pages? It's a discovery pointer telling agents where the full documentation index lives. The Mintlify-generated version reads: "Fetch the complete documentation index at .../llms.txt. Use this file to discover all available pages before exploring further." Whatever page an agent lands on, it learns where the map is.

Last verified 2026-06-11. We re-test the claims on this page quarterly; changes are logged in the Data Room.

Part of The Complete Playbook to Agentic Discovery.

Stay ahead of the agents. We re-test this playbook quarterly and publish what changed: new data, busted myths, ranking shifts. Get the update digest →

Want this done for you? Synscribe runs agentic-discovery programs for B2B SaaS and developer platforms. Talk to us →