Your robots.txt Is Making Three AI Decisions for Your Firm. Google Says It Only Controls One Thing.

On October 22, Google's Gary Illyes said something on a Search Central video that every immigration practice with a website should hear. Robots.txt, he explained, is "an access preference protocol" under RFC 9309. It tells a crawler whether it may request a page. It "doesn't dictate what a system does with the data" once that page has been fetched.

That sounds like a technical footnote. It isn't. Most small-firm websites carry a robots.txt file that nobody at the firm has ever opened. It was installed by a web vendor, a WordPress plugin, or a hosting template. Some of those templates block every AI bot by default. Others allow everything. Either way, the file is quietly deciding whether your practice can be found when a family asks ChatGPT, Claude, or Gemini about an adjustment of status case or a removal defense consultation — and whether your content goes into model training you never chose.

Nobody at the firm decided any of it. That's the problem worth fixing.

What Illyes Actually Said

The video, the second episode of Google's "Non-Canonical Cabinet" series, made three points worth getting precisely right. Matt G. Southern covered it for Search Engine Journal on October 22.

• Robots.txt controls access, not use. It governs which URLs a crawler may request. What happens to the content afterward is outside the protocol.
• Made-up lines do nothing. Google supports four elements: User-Agent, Allow, Disallow, and Sitemap. A line like "No-AI-Training" is simply ignored by the parser.
• To block one AI crawler, you name it. Illyes was direct: you "target their specific user agent tokens explicitly in the file." A blanket rule catches everything; a named rule catches one bot.

That last point is the one most firm websites get wrong, because AI companies don't run one bot. They run several, and each one does a different job.

Three Kinds of AI Bots, Three Different Decisions

When you read your robots.txt line by line, every AI-related entry falls into one of three buckets. Each bucket is a separate decision for your practice.

1. Answer-lookup bots. These fetch pages so an AI assistant can find and cite current sources when someone asks a question. OpenAI's is OAI-SearchBot; OpenAI's documentation says sites that block it "will not be shown in ChatGPT search answers." Anthropic's is Claude-SearchBot. Block these, and when a prospective client asks ChatGPT which firms in Houston handle asylum cases, your site is out of the running for that lookup.

2. Training tokens. These govern whether your content may be used to train future models. OpenAI uses GPTBot; Anthropic uses ClaudeBot. Google uses Google-Extended, which is unusual: it has no crawler of its own. Google crawls with its normal user agents, and Google-Extended is a control token that decides whether that content can be used to train Gemini and to ground answers in Gemini Apps and Vertex AI. Per Google's documentation, it does not affect whether your pages appear in Google Search. One caveat Duane Forrester raised in Search Engine Journal on October 22: AI Overviews and AI Mode are governed separately, through Search Console, not through Google-Extended.

3. User-triggered agents. These fetch a page because a specific person asked the assistant to look at it — "check whether this firm offers consultations in Spanish." OpenAI's is ChatGPT-User; Anthropic's is Claude-User. OpenAI notes that robots.txt rules "may not apply" to these, because a human initiated the visit. Anthropic says its user agent respects robots.txt. In practice, these are the closest thing to a real person visiting your site through an AI tool.

These are three different questions. Should AI assistants be able to find and cite my firm? Should my content train future models? Should a prospective client's AI assistant be able to read my consultation page when they ask it to? A firm can reasonably answer yes, no, yes. But a vendor template that says "block all AI" answers no, no, no — and a template that says "allow everything" answers yes, yes, yes. Neither one asked you.

Why This Lands Harder on Immigration Practices

Immigration clients increasingly start with an AI assistant, often in their first language, often late at night after a notice arrives. A DACA recipient trying to understand a renewal timeline, an H-1B worker after a layoff with 60 days to act, a spouse holding an RFE on an I-130 — these are specific, urgent questions, and the assistant answers them by looking up current sources.

If your robots.txt blocks the answer-lookup bots, your practice area pages are not among those sources. The assistant still answers. It just cites a directory, a national firm, or a generic explainer instead of you.

The reverse risk is real too. Many attorneys have thought carefully about where their original content ends up — the detailed guide to consular processing delays, the plain-language PERM timeline your paralegal spent a week on. If your file allows every training token by default, that decision was made for you.

What to Do This Week

This takes about 30 minutes, and most of it is reading.

1. Open your robots.txt. Type your domain followed by /robots.txt into a browser — for example, yourfirm.com/robots.txt. It's a public file. Copy what you see into a document.
2. Find every AI-related line. Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, PerplexityBot, and CCBot. Also look for a "User-agent: *" line with "Disallow: /" — that blocks everything, including Google Search.
3. Sort each line into a bucket. Answer lookup, training, or user-triggered. Write down what the line currently says: allowed, blocked, or not mentioned. Not mentioned usually means allowed under your general rule.
4. Make the three decisions as a firm. For most immigration practices that want AI-referred inquiries, the answer-lookup and user-triggered bots should be allowed. Training is a judgment call — decide it on purpose.
5. Hand your web vendor specific instructions. Not "make us AI-friendly." Name the user agents and the rule for each. Ask them to confirm in writing which plugin or template controls the file, so an update doesn't silently overwrite it.
6. Check it again after any site update. Template and plugin updates are the most common way these files change without anyone noticing.

One caution: robots.txt is a request, not a lock. Reputable crawlers honor it; it doesn't make content private. Anything you need protected should not be on a public page at all.

The Bigger Picture

For two decades, a website had one crawler that mattered and one question to answer: should Google index this? That's no longer the shape of things. An immigration firm's site is now read by search engines, by AI answer engines, by training pipelines, and by assistants acting on behalf of individual clients — each with its own name and its own rules.

Illyes's point cuts both ways. Robots.txt can't control what happens to your content after it's fetched. But it does control who gets to fetch it in the first place, and that is a decision your practice should be making deliberately, line by line. In the next piece, we'll walk through what a well-configured file actually looks like for a small immigration firm.

Clientory helps immigration law firms become visible to the families and individuals who need them most — turning AI search into consultations, and consultations into clients.

See Your AI Visibility Score → CLIENTORY.ORG