How to Make Your Site Crawlable by AI Agents

To make your website crawlable by AI agents, serve fully server-rendered HTML at stable URLs, implement the markdown proxy pattern to return clean content when bots request Accept: text/markdown, and add an llms.txt file to guide AI crawlers directly to your best pages. These three steps let the AI search and user-fetch bots (OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot) extract and cite your content without JavaScript rendering — and robots.txt lets you treat them differently from training crawlers such as GPTBot and ClaudeBot.

ChatGPT, Perplexity, Claude, and Google's AI Overviews don't just generate answers from training data anymore. They browse the web in real time to answer questions. But most websites weren't built for these new visitors. Single-page applications, JavaScript-heavy rendering, and complex layouts choke the bots that power the fastest-growing search channel on the internet.

If your docs, blog, or product pages aren't readable by AI agents, you're invisible to millions of users who now get answers through conversational AI instead of clicking ten blue links. According to SparkToro (2024), 58.5% of Google searches in the US result in zero clicks — and that number climbs when you factor in AI-generated answers pulling traffic before users ever reach a search engine.

This guide covers how AI agents actually browse the web, why JavaScript-heavy sites fail, and practical patterns — the markdown proxy and the llms.txt standard — that make any site AI-crawlable without rebuilding your frontend.

TL;DR: AI vendors run separate bots for training (GPTBot, ClaudeBot), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-requested fetches (ChatGPT-User, Claude-User, Perplexity-User), and you can allow or block each in robots.txt. None of them run your JavaScript reliably, and according to SparkToro, 58.5% of US Google searches already end without a click. The markdown proxy pattern — detecting Accept: text/markdown and returning converted content — combined with the llms.txt standard makes any website AI-crawlable without changing your frontend framework.

How Do AI Agents Browse the Web?

AI crawlers and fetchers are a fast-growing share of web traffic, and most of them don't render JavaScript or wait for client-side hydration. They want raw, structured content in the first response. To see which ones already hit your site, track AI crawler traffic in your logs.

The User-Agent Strings You Should Know

Every major AI platform identifies its bots with specific User-Agent tokens, and the big three now split them by purpose. That split is what your robots.txt should follow:

Sources: OpenAI's crawler docs, Anthropic's crawler help page and Perplexity's bot docs, checked October 2026. OpenAI's current tokens carry version numbers (for example GPTBot/1.4 and OAI-SearchBot/1.4), so match on the name, not the full string. Other crawlers you'll see include Amazonbot and ByteDance's Bytespider.

These bots behave differently from Googlebot or Bingbot. Traditional search crawlers render JavaScript, follow pagination, and index the full DOM. AI agents typically fetch the raw HTTP response and move on. If your content isn't in the initial HTML payload, they don't see it.

The Accept Header That Changes Everything

Here's a detail most developers miss. Some AI agents and developer tools ask for markdown with an Accept: text/markdown header alongside the standard text/html — a direct signal: "If you have a markdown version, I'd prefer that." Most crawlers still ask for HTML, so treat this as an extra path, not a replacement for server-rendered pages.

Why markdown? Because it preserves semantic structure — headings, lists, code blocks, links — without the noise of HTML tags, CSS classes, and JavaScript. An AI model can parse a markdown document far more accurately than a rendered webpage full of navigation bars, cookie banners, and footer links.

A typical HTTP request from an AI agent looks like this:

GET /blog/how-to-deploy-nextjs HTTP/1.1
Host: yoursite.com
User-Agent: Mozilla/5.0 ...; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Accept: text/markdown, text/html, */*

When your server detects Accept: text/markdown, it can return the page's content as clean, structured markdown instead of HTML — giving the bot exactly what it processes most accurately.

How Does robots.txt Control AI Access?

Your robots.txt file is the first thing any well-behaved crawler checks. Because training and search are separate bots, you can allow AI search citations while opting out of training:

# Allow AI search indexing and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Blocking OAI-SearchBot keeps you out of ChatGPT search answers; blocking GPTBot only opts you out of training. User-initiated fetchers (ChatGPT-User, Perplexity-User) may not apply robots.txt at all, because a person asked for the page — so rely on authentication, not robots.txt, for anything private.

Most site owners think about AI crawling as a binary allow/block decision. The smarter approach is selective exposure: allow AI agents to access your educational content (blog, docs, guides) while blocking proprietary pages (pricing internals, dashboard, admin). This maximizes citation potential without giving away competitive intelligence.

Why Do SPAs and JavaScript-Heavy Sites Fail?

Client-side rendered applications return an empty HTML shell to crawlers — typically a <div id="root"></div> with zero content. Googlebot renders JavaScript in a second pass; most AI crawlers and fetchers don't render it at all, so the problem is worse for them.

No Content in the Initial HTML

When an AI crawler requests yoursite.com/docs/getting-started, it receives something like this:

<!DOCTYPE html>
<html>
<head><title>Loading...</title></head>
<body>
  <div id="root"></div>
  <script src="/bundle.js"></script>
</body>
</html>

There's nothing for the bot to extract. The actual content lives inside bundle.js, which the bot doesn't execute. Your documentation, your blog posts, your product descriptions — all invisible.

Server-side rendering (SSR) and static site generation (SSG) solve this partially. Frameworks like Next.js, Nuxt, and Astro render the full HTML on the server. But even SSR pages come wrapped in complex HTML structures that AI agents need to parse through.

AJAX-Loaded Content Disappears

Even if your initial page is server-rendered, any content loaded via AJAX after the first paint is invisible to AI crawlers. Tabbed interfaces, accordion sections, "load more" buttons, and infinite scroll patterns all hide content behind JavaScript interactions.

Consider a product page with tabs for "Description," "Specifications," and "Reviews." If those tabs load content via fetch requests, an AI agent only sees whichever tab is active in the initial HTML. The rest doesn't exist.

Infinite Scroll Breaks Extraction

Infinite scroll is particularly hostile to AI agents. The bot receives the first 10 items. It has no mechanism to scroll down, trigger the intersection observer, and load the next batch. Your 500-item catalog looks like a 10-item catalog.

But does pagination solve this? Not entirely. AI agents follow links within a page, but they don't systematically crawl paginated sequences the way Googlebot does. The most reliable approach is having all critical content accessible at stable URLs with full server-rendered HTML.

What Is the Markdown Proxy Pattern?

The markdown proxy pattern is a server-side technique that detects requests asking for markdown and returns a clean markdown version of the page instead of HTML. As AI agents become a regular source of traffic, serving them a lighter, unambiguous version of each page is cheap insurance.

How It Works

The pattern has three steps:

1. Detect — Check the incoming request's Accept header for text/markdown (and, optionally, the User-Agent for known AI fetchers).
2. Convert — Take the page's main content (the <main> or <article> element, not the whole document) and convert it to clean markdown using a library like Turndown (JavaScript) or html2text (Python).
3. Respond — Return the markdown with Content-Type: text/markdown and Vary: Accept (plus User-Agent if you sniff it), so caches never serve markdown to people.

This approach is framework-agnostic. It works whether your site is built with React, Vue, WordPress, Django, or plain HTML. The conversion happens at the proxy or middleware layer, so your frontend code doesn't change.

Why Markdown Over HTML?

Think about what an AI model needs to do with your content. It needs to identify the main topic, extract key facts, understand the hierarchical structure, and pull self-contained passages. HTML makes all of that harder.

A typical HTML page includes navigation menus, sidebars, footers, cookie consent banners, promotional pop-ups, schema.org markup, SVG icons, and dozens of div wrappers with CSS classes. The actual content might be 20% of the HTML payload. The other 80% is structural noise.

Markdown strips all of that away. Headings are ##. Lists are -. Code is fenced with triple backticks. Links are inline [text](url). An AI model processes this with far higher accuracy.

Here's what the same content looks like before and after conversion. A real production page with navigation, sidebars, and footers can easily be 100KB of HTML for a few hundred words of content — the markdown equivalent is typically under 10KB.

Typical HTML (mostly structure and noise):

<nav class="sidebar nav-fixed z-50">...</nav>
<article class="content prose prose-lg max-w-3xl mx-auto">
  <h1 class="title text-2xl font-bold tracking-tight">How to Deploy</h1>
  <p class="body text-gray-700 leading-relaxed">
    Run <code class="inline-code bg-gray-100">git push</code> to deploy.
  </p>
</article>
<footer class="site-footer border-t mt-16">...</footer>

Clean markdown (pure content):

# How to Deploy

Run `git push` to deploy.

The markdown version contains the same information with zero structural noise — navigation bars, CSS classes, and container elements are stripped, leaving only the content an AI agent needs to extract.

When Not to Use This Pattern

The markdown proxy isn't appropriate for every page. Interactive dashboards, web applications, and pages where the visual layout is the content (design portfolios, data visualizations) don't translate well to markdown. Apply this pattern to content-heavy pages: blog posts, documentation, knowledge bases, API references, and marketing pages.

How Do You Serve Markdown to AI Crawlers?

Building a markdown proxy takes fewer than 50 lines of middleware code in most frameworks. The core logic is simple: check the request, convert the content, cache the result. Here's a working implementation pattern you can adapt to any stack.

The following middleware example is a starting pattern; adapt it to your framework (Express, Fastify or an edge runtime) and test it against your own pages. The conversion quality depends heavily on the HTML structure — well-structured semantic HTML produces much better markdown than div-soup layouts.

Express Middleware Example

const TurndownService = require('turndown');
const turndown = new TurndownService({
  headingStyle: 'atx',
  codeBlockStyle: 'fenced'
});

// Cache converted markdown for 1 hour
const markdownCache = new Map();

function aiCrawlerMiddleware(req, res, next) {
  const accept = req.headers['accept'] || '';
  const ua = req.headers['user-agent'] || '';

  const wantsMarkdown = accept.includes('text/markdown');
  const isAIFetcher = /OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i.test(ua);

  // Every response on this route varies by these headers, so shared caches
  // and CDNs never hand the markdown version to a browser.
  res.set('Vary', 'Accept, User-Agent');

  if (!wantsMarkdown && !isAIFetcher) {
    return next();
  }

  const cacheKey = req.path;
  if (markdownCache.has(cacheKey)) {
    res.set('Content-Type', 'text/markdown; charset=utf-8');
    return res.send(markdownCache.get(cacheKey));
  }

  // Capture the original response
  const originalSend = res.send.bind(res);
  res.send = function(html) {
    if (typeof html === 'string' && html.includes('<')) {
      // Convert only the main content, not nav, sidebar and footer
      const main = html.match(/<main[\s\S]*?<\/main>/i)?.[0]
        ?? html.match(/<article[\s\S]*?<\/article>/i)?.[0]
        ?? html;
      const markdown = turndown.turndown(main);
      markdownCache.set(cacheKey, markdown);

      // Clear cache after 1 hour
      setTimeout(() => markdownCache.delete(cacheKey), 3600000);

      res.set('Content-Type', 'text/markdown; charset=utf-8');
      return originalSend(markdown);
    }
    return originalSend(html);
  };

  next();
}

app.use(aiCrawlerMiddleware);

This middleware intercepts responses before they leave the server. When a request asks for markdown (or comes from a known AI fetcher), it converts the page's main content with Turndown, caches the result, and marks every response with Vary so a CDN keeps the two versions apart. A regex is fine for a sketch; use a real HTML parser such as cheerio in production.

Configuring robots.txt for AI Agents

Your robots.txt should explicitly welcome the search bots you want on your public content and keep them out of private areas:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /blog/
Allow: /docs/
Disallow: /dashboard/
Disallow: /api/

Sitemap: https://yoursite.com/sitemap.xml

Caching Considerations

Converting HTML to markdown on every request wastes CPU cycles. Cache the markdown version with a TTL that matches your content update frequency. For a blog, a 24-hour TTL makes sense. For documentation that changes frequently, drop it to 1-4 hours.

Consider pre-generating markdown at build time if you use a static site generator. Run the conversion during your CI pipeline and store the .md files alongside your HTML output. This eliminates runtime conversion entirely.

What Is llms.txt and Should You Adopt It?

The llms.txt standard is a proposed convention — similar to robots.txt — that tells AI agents where to find machine-readable content on your site. Introduced by Jeremy Howard in late 2024, it's already been adopted by sites including Anthropic's documentation and Cloudflare's developer docs as documented on llmstxt.org. The file sits at /llms.txt and provides a structured map of your content specifically for LLM consumption.

How llms.txt Works

The file uses a simple markdown-like format. It lists your site's key pages with descriptions and links to their markdown equivalents:

# YourSite

> A brief description of your site for AI context.

## Docs

- [Getting Started](https://yoursite.com/docs/getting-started.md): Setup guide for new users
- [API Reference](https://yoursite.com/docs/api.md): Complete REST API documentation
- [Configuration](https://yoursite.com/docs/config.md): All configuration options

## Blog

- [Deployment Guide](https://yoursite.com/blog/deploy.md): Step-by-step deployment tutorial

Each entry points to a markdown version of the page. When an AI agent finds your llms.txt, it knows exactly which pages contain useful content and can fetch the clean markdown version directly.

llms.txt vs robots.txt

These two files serve different purposes. robots.txt controls access — it tells crawlers what they can and can't fetch. llms.txt controls discovery — it tells AI agents what they should fetch and provides context about each page.

You want both. Use robots.txt to set boundaries. Use llms.txt to guide AI agents toward your best pages. Think of robots.txt as the bouncer and llms.txt as the concierge.

One caveat: llms.txt is a proposal, and no major AI search crawler has publicly confirmed that it reads the file. Its clearest use today is for agents and developer tools that fetch it on purpose. It costs little to publish, so add it, but don't expect it to change crawling on its own. On temps.sh we publish /llms.txt alongside markdown responses for most content pages and separate robots.txt groups for search and training bots.

How Should You Handle Structured Data and Content Licensing?

Structured data helps search engines and AI systems understand your content's meaning. According to Schema.org, over 45 million web domains marked up their pages as of 2024. Google uses it for rich results, and says its AI features need no special markup beyond normal SEO. For AI crawlers, you also need to think about licensing signals that communicate how your content can be used.

Schema.org Markup That AI Agents Use

Add JSON-LD structured data to your content pages. The schemas AI systems find most useful are:

• Article — Blog posts, news articles, tutorials
• HowTo — Step-by-step guides with structured steps
• FAQPage — Question-answer pairs that map directly to user queries
• TechArticle — Technical documentation and API references

Here's a minimal JSON-LD example for a blog post:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "How to Make Your Website Crawlable by AI Agents",
  "author": { "@type": "Person", "name": "Your Name" },
  "datePublished": "2026-03-12",
  "description": "A guide to optimizing websites for GPTBot, ClaudeBot, and other AI crawlers."
}
</script>

Content Licensing for AI Crawlers

Here's a question more site owners should ask: do you want AI platforms to train on your content, or just cite it? The difference matters.

The TDM Reservation Protocol (TDMRep, a W3C community specification) lets you signal your licensing preferences through HTTP headers. Add a TDM-Reservation header set to 1 if you want to reserve your text and data mining rights.

TDM-Reservation: 1

You can also specify a TDM policy URL for more granular control:

TDM-Policy: https://yoursite.com/tdm-policy.json

It isn't universally respected yet. Under Article 4 of the EU's Digital Single Market Directive, rights holders can reserve text-and-data-mining rights in a machine-readable way, and TDMRep is one way to do that. Setting the header is low effort.

The distinction between "citation crawling" and "training crawling" is the most overlooked aspect of AI content strategy. Most developers block all AI bots or allow all of them. The nuanced approach — allowing search and user-fetch bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot while blocking training crawlers like GPTBot, ClaudeBot, CCBot and Bytespider, combined with TDM headers — gives you visibility with control.

How Should You Optimize Content for AI Extraction?

Content structure matters as much as content delivery. Answer engines extract passages, so content that is already structured as clear, standalone answers under descriptive headings is easier to cite.

Answer-First Formatting

Start every section with a direct answer to the heading's implied question. Don't build up to the point — state it immediately. AI agents extract the first paragraph under each heading as the most likely answer candidate.

Compare these two approaches:

Bad: "There are many factors to consider when choosing a deployment strategy. Let's explore the various options and their trade-offs before arriving at a recommendation."

Good: "Blue-green deployments eliminate downtime by running two identical production environments and switching traffic between them. This approach costs 2x in infrastructure but reduces deployment risk to near zero."

The second version gives an AI agent a complete, self-contained answer in two sentences.

Heading Hierarchy That AI Can Parse

Use H2 headings for main topics and H3 for subtopics. Never skip levels (don't jump from H2 to H4). Phrase headings as questions whenever possible — AI agents match user queries to heading text.

Structure your heading hierarchy like an outline:

## How Do You Deploy a Next.js App?        <- Main topic
### Prerequisites                            <- Subtopic
### Step 1: Configure Your Build Settings    <- Subtopic
### Step 2: Set Environment Variables        <- Subtopic
## What Are Common Deployment Errors?        <- Next main topic
### Build Failures                           <- Subtopic
### Runtime Errors                           <- Subtopic

Code Blocks Need Language Tags

Always include language identifiers on fenced code blocks. AI agents use these tags to categorize and correctly attribute code snippets:

```javascript
// Good: AI knows this is JavaScript
const server = express();
```

```
// Bad: AI doesn't know the language
const server = express();
```

The language tag isn't just for syntax highlighting — it's metadata that tells any parser, human or machine, what the snippet is.

Deployment Platform Comparison: AI Crawlability Out of the Box

The deployment platform you choose directly affects AI crawlability. Here's how the main options compare:

Prices checked October 2026 on vercel.com/pricing and netlify.com/pricing. On Vercel and Netlify you write the markdown path yourself; on Temps the proxy already does it for every app.

How Does Temps Handle the Proxy Layer for AI Crawlers?

Temps routes all traffic through a reverse proxy built on Pingora, the open-source Rust framework Cloudflare built to handle over a trillion requests per day. The markdown pattern from this guide is built into that proxy: when a request carries Accept: text/markdown, the proxy asks your app for an uncompressed HTML page, converts it to Markdown (with the page title and description as frontmatter), and returns it with Content-Type: text/markdown and Vary: Accept. Browsers and Googlebot keep getting your normal HTML. It works for any framework you deploy, with no middleware in your app — see request flow for where it sits in the proxy.

You can see it on this page:

curl -s -H "Accept: text/markdown" https://temps.sh/blog/how-to-make-website-crawlable-by-ai-agents | head

robots.txt, llms.txt and the rest stay in your app, where you control them. Because the proxy also logs every request, you can track which AI crawlers fetch which pages without a third-party analytics script.

Temps is free to self-host (MIT / Apache 2.0) and runs as a single binary with git-push deployments, with no per-seat pricing or bandwidth bills. Temps Cloud, a hosted control plane for your own servers, is coming soon from $29/month.

FAQ

Do AI agents respect robots.txt?

The crawlers do: OpenAI, Anthropic and Perplexity document that GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot honour robots.txt. User-initiated fetchers are different — OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. Smaller crawlers vary; Dark Visitors keeps a list. Combine robots.txt with authentication or server-side filtering for anything that must stay private.

Will serving markdown hurt my regular SEO?

No, as long as you set Vary. The markdown path only activates when the request includes Accept: text/markdown (or matches a known AI fetcher, if you sniff User-Agents). Regular browsers and Googlebot receive your normal HTML page with all its meta tags and structured data. Send Vary: Accept (and User-Agent if you use it) so a CDN never serves the markdown version to a person.

How do I test if my site is AI-crawlable?

Send a curl request with the AI headers and check what comes back. Run curl -H "Accept: text/markdown" https://yoursite.com/blog/your-post, then curl -A "OAI-SearchBot/1.4" https://yoursite.com/blog/your-post to see the HTML a crawler gets, and inspect both. If you get clean markdown with your content's heading hierarchy preserved, you're good. If you get raw HTML or an empty page, your content isn't optimized for AI extraction.

Should I adopt llms.txt now or wait for wider adoption?

Publish it if it's cheap for you, but keep expectations modest. The file takes minutes to create, and sites such as Anthropic's and Cloudflare's docs serve one. No major AI search crawler has publicly confirmed using it, though; its clearest use is for agents and tools that fetch it deliberately. It won't hurt your traditional SEO or site performance.

Start Making Your Content AI-Visible

AI-powered search isn't a future trend — it's where millions of users already find answers. Every day your content stays invisible to AI search and user-fetch bots, you miss citation opportunities that drive traffic without competing for ten blue links.

The markdown proxy pattern works regardless of your stack. Detect the Accept header, convert your HTML, cache the result. Add an llms.txt file to guide bots to your best pages. Set TDM headers to control how your content gets used. These aren't complex changes — each one takes less than an hour.

You can build it yourself with the middleware example above, or deploy on a platform that does it for you. Temps runs a Pingora-based proxy in front of every deployed app that already answers Accept: text/markdown with converted Markdown. Push your code and your AI crawlability layer runs on infrastructure you own — free to self-host under MIT / Apache 2.0.