robots.txt is a text file in your site’s root directory that tells search engines and other bots which parts they can crawl. In the AI visibility era, this file is now a critical control point not just for Google but for AI crawlers too.
What robots.txt does and doesn’t do
- Does: “Requests” that bots not crawl certain directories or pages.
- Doesn’t: Hide content or provide security. A blocked page can still be indexed through other means; robots.txt isn’t enough to protect sensitive content.
AI bots and the access decision
ChatGPT, Perplexity and other AI tools have their own crawlers. These bots accessing your content is a precondition for appearing in those platforms’ answers. But this is a matter of choice:
- If you want to appear, make sure the relevant AI bots can crawl your content.
- If you want to keep certain bots out (for example, those collecting only for training), you can indicate this with robots.txt.
What matters is making this decision deliberately. On many sites, AI bots are unknowingly blocked by firewall or CDN rules — and the brand doesn’t understand why it isn’t visible.
Common mistakes
- Accidentally blocking everything. A single wrong line can close your whole site to crawling.
- Blocking the bots you want to appear in. Check firewall and CDN rules too; the problem isn’t always in robots.txt.
- Trying to protect sensitive content with robots.txt. The right method for that is authentication and
noindex.
The right approach
Build your access policy around the question “which surfaces do I want to appear on?” Then align robots.txt, server and CDN rules accordingly. This is an important part of technical SEO and AI discoverability work.
Summary
robots.txt is a powerful but limited tool that manages bot access. In the AI era, you need to make sure the bots you want to appear in are given access, and deliberately manage those you don’t. To see your access state, a technical pre-review is a good start.