# The three types of AI bot traffic and how to handle them

By Steven Van · 2025-08-13

Vercel's traffic data splits AI bots into training, grounding, and referral crawlers, with guidance on which routes to block or allow.

Vercel's network data shows bots now make up more than **20%** of traffic across its deployments, with about a quarter of that coming from AI crawlers. Its [breakdown](<https://vercel.com/blog/the-three-types-of-ai-bot-traffic-and-how-to-handle-them>) splits that traffic into three types and argues against blocking it outright.

- **AI training crawlers**, such as GPTBot and ClaudeBot, scan nearly every public page, including docs, blog posts, product pages, pricing, and changelogs, to build the data behind future AI responses. They made up the largest share of AI bot traffic in Vercel's dataset last week.

- **AI engine grounding bots** fetch real-time updates when a user's question needs current information, checking live sites alongside training data. A blog post announcing a new feature can be fetched within days of publication.

- **AI referrals** are the human visitors who click through from AI-generated answers. They still trail traditional search referral volume but grow each month, and some sites report higher conversion from this group than from organic search.

Vercel says the three stages feed each other: content that isn't crawled isn't in the training data, so it can't be cited, so it generates no referrals. Its recommendation is to block AI crawlers on routes like /login, /checkout, /admin, and user dashboards, while keeping documentation, blog posts, landing pages, and pricing pages open. It points to [Vercel](<https://creatorstoolbox.com/tools/vercel>) Firewall, Bot Protection, and BotID as tools to verify legitimate crawlers and block impersonators without shutting off AI-driven discovery.

[![](<https://assets.vercel.com/image/upload/contentful/image/e5382hct74si/69vco5n527aVzqWcusMzmS/41773ffde7c3644e13e36a1c79f851cd/AI_traffic_anatomy_blog.png>)](<https://vercel.com/blog/the-three-types-of-ai-bot-traffic-and-how-to-handle-them>)

![Vercel](<https://www.google.com/s2/favicons?domain=vercel.com&sz=128>)

Vercel

The platform for frontend developers — deploy, preview, and scale web apps and AI agents with zero config.

[View Vercel →](<https://creatorstoolbox.com/tools/vercel>)

[Original source](<https://vercel.com/blog/the-three-types-of-ai-bot-traffic-and-how-to-handle-them>)

[Read The three types of AI bot traffic and how to handle them on Creators Toolbox](<https://creatorstoolbox.com/blog/vercel-the-three-types-of-ai-bot-traffic-and-how-to-handle-them>)

---
Canonical source: https://creatorstoolbox.com/blog/vercel-the-three-types-of-ai-bot-traffic-and-how-to-handle-them