Knowledge base crawl exclusions

ElevenAgents knowledge base crawls can now skip parts of a website. Pass up to 50 exclude_patterns to Create crawl job, or enter them in the Exclude patterns field in the whole-website crawl dialog.

A matching URL is skipped even if it also matches the crawl pattern. Crawl job responses include the exclusion patterns used.