The three types of AI bot traffic and how to handle them
By Steven Van ·
Vercel's traffic data splits AI bots into training, grounding, and referral crawlers, with guidance on which routes to block or allow.
Vercel's network data shows bots now make up more than 20% of traffic across its deployments, with about a quarter of that coming from AI crawlers. Its breakdown splits that traffic into three types and argues against blocking it outright.
- AI training crawlers, such as GPTBot and ClaudeBot, scan nearly every public page, including docs, blog posts, product pages, pricing, and changelogs, to build the data behind future AI responses. They made up the largest share of AI bot traffic in Vercel's dataset last week.
- AI engine grounding bots fetch real-time updates when a user's question needs current information, checking live sites alongside training data. A blog post announcing a new feature can be fetched within days of publication.
- AI referrals are the human visitors who click through from AI-generated answers. They still trail traditional search referral volume but grow each month, and some sites report higher conversion from this group than from organic search.
Vercel says the three stages feed each other: content that isn't crawled isn't in the training data, so it can't be cited, so it generates no referrals. Its recommendation is to block AI crawlers on routes like /login, /checkout, /admin, and user dashboards, while keeping documentation, blog posts, landing pages, and pricing pages open. It points to Vercel Firewall, Bot Protection, and BotID as tools to verify legitimate crawlers and block impersonators without shutting off AI-driven discovery.
