AI Crawler Controls
Block or allow AI crawlers like GPTBot, ClaudeBot and CCBot one by one through robots.txt, opt-out headers and an optional 403, and see every visit they make in a crawl log.
Screenshots
AI Crawler Controls puts the AI crawlers that visit your site on one screen, each with its own switch. AI companies send crawlers for different reasons: some collect pages to train their models, some index pages for AI search answers, and some fetch a page when a person asks an assistant to read it. The plugin lists 25 of them, from GPTBot and ClaudeBot to Google-Extended, PerplexityBot and Bytespider, and explains what each one does before you decide.
Blocking a crawler adds a rule for it to your robots.txt, the standard opt-out every major AI company says its crawlers follow. One click blocks AI training while keeping you visible in AI search, so ChatGPT search and Perplexity can still quote and link to you. For crawlers that don't listen, turn on Refuse requests and they get a 403 error instead of your page. Optional noai and TDM signals ask everyone else not to use your content.
A crawl log records every visit from an AI crawler with the page, the result, the status code and the IP address, plus a 30-day chart and a last-seen time for each crawler, so you can see who's crawling and whether your rules are respected. Nothing changes until you choose, your dashboard and login page are never refused, and regular visitors are never logged. Find it under Content Powered > AI Crawler Controls, and get updates through the normal WordPress Updates screen.
Features
- 25 AI crawlers, one switch each. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and more, each explained.
- Block training, keep AI search. One click keeps your content out of training data while AI search can still cite you.
- robots.txt rules for you. A Disallow rule for each blocked crawler, and a live test that confirms they're there.
- Refuse crawlers that don't listen. Blocked crawlers that ignore robots.txt can get a 403 error instead of your page.
- AI opt-out signals. Optional noai, noimageai and TDM reservation signals on every page.
- A log of every visit. Each request from an AI crawler, with the page, result, status code and IP address.
- 30-day activity chart. See how often AI crawlers visit and how many requests were refused.
- Rule breakers flagged. Blocked crawlers that ask for pages anyway are marked in the log.
- No lockouts. Your dashboard and login are never refused, and tokens that match browsers are rejected.
- Click Download above to save
ai-crawler-controls.zip. Leave it zipped. - In your WordPress dashboard, go to Plugins > Add New Plugin and click Upload Plugin.
- Choose the ZIP and click Install Now, then Activate Plugin.
To update later, upload a newer ZIP the same way. WordPress offers to replace the installed version and keeps your settings.
Will Blocking AI Crawlers Hurt My SEO?
No. None of the crawlers in the list is Googlebot or Bingbot, and search engines ignore the noai and TDM signals. Google-Extended only controls whether Gemini may use your pages; it doesn't affect Google Search or AI Overviews.
Which AI Crawlers Should I Block?
Most sites block AI training and allow AI search, which is what the Block AI training button does. Your content stays out of future training data, while ChatGPT search, Perplexity and other AI search tools can still quote your pages and send visitors to them.
Do AI Crawlers Respect robots.txt?
OpenAI, Anthropic, Google, Apple, Meta, Perplexity and Common Crawl say their crawlers do. Some crawlers don't, and some assistants skip robots.txt when a person asked for the page. The crawl log flags any blocked crawler that requests pages anyway, and Refuse requests turns those requests away with a 403.
What's the Difference Between Training and Search Crawlers?
Training crawlers like GPTBot and ClaudeBot collect pages to build AI models. Search crawlers like OAI-SearchBot and PerplexityBot index pages so AI search tools can cite and link to them. Assistant fetchers like ChatGPT-User visit a single page when someone asks an assistant to read it. The plugin groups them this way so you can treat each kind differently.
Can I Block a Crawler That Isn't Listed?
Yes. Add its user-agent token under Other crawlers to block and it's disallowed in robots.txt, and refused too if Refuse requests is on. Tokens that would also match web browsers, search engines or link previews are rejected, so you can't turn real visitors away by mistake.
Does It Work With Caching Plugins?
Yes. The robots.txt rules and the opt-out meta tags work with any cache. WP Rocket is set up automatically so refused crawlers reach the 403. Other page caches and CDNs that answer before WordPress loads may still serve cached pages to blocked crawlers, and those visits won't appear in the log.
How Do I Get Updates?
Through the normal WordPress Updates screen. The plugin checks our GitHub releases twice a day, and you can update with one click or turn on auto-updates from the Plugins screen.
Download
More Free WordPress Plugins
Honest Last Modified Date
Only changes a post's "Last modified" date when enough of it has really changed: a minimum number of words, a percentage, or both, set separately for posts and pages.Phone Click Tracking
See which pages, traffic sources and campaigns make people call or text you. Every tel:, sms:, WhatsApp and email click is logged in WordPress, with Google Business Profile and AI referrals, a Google Ads conversion export and client reports.Link Audit and Reports
Every outbound link on your site, grouped by domain: follow, nofollow, sponsored and ugc counts, who added each link and when. Fix rel attributes or strip links in bulk, whitelist and blacklist domains, undo any change and export to CSV.Let's Grow Your Business
Want some free consulting? Let's hop on a call and talk about what we can do to help.
