Your robots.txt Is Now an AI Policy. Did Your Firm Write It?

There is a small text file at the root of almost every law-firm website. It's called robots.txt, and for most of its life it was a technical matter: a set of instructions telling search-engine crawlers which pages to read and which to leave alone. Few partners have ever read their firm's version. Most have no reason to know it exists.

That's changing. Over the past three years the file has quietly become the place where a firm tells AI companies what they may do with its content. And on 30 October 2026, Google announced that it will support a shared "AI training" control in robots.txt, one line that expresses a training preference to every participating AI provider at once.

My proposition is simple. Robots.txt is now an AI policy. For many firms it was written by a plugin, a hosting company or a security setting rather than by anyone who considered the trade-offs. And the trade-offs are about to become easier to get wrong.

Why this matters to a managing partner

There are two different things an AI company might want to do with your website.

The first is training: using your pages to shape the model itself. Many firms are uneasy about that, and some have good reasons, from copyright in their published guides to a principled objection to their expertise being absorbed without credit.

The second is retrieval: reading your pages at the moment someone asks a question, so the assistant can answer with current information and, often, a link back to you. That is discovery. When a prospective client asks an AI assistant about their situation, retrieval is the route by which your firm's own material can inform the answer.

Most firms would answer those two questions differently: perhaps "no" to training and "yes" to retrieval. The problem is that the controls don't always separate them, and Google's doesn't.

What was actually announced

Here is what I could verify from primary sources as at 8 November 2026.

The shared signal. Google's Search Central blog post of 30 October says website owners will be able to use a shared "policy token" in robots.txt "to set broad preferences across participating AI providers", instead of writing separate rules for each AI developer. Google's crawler documentation gives the token as __usage_ai-training. It comes from an Internet-Draft linked to the IETF's AIPREF working group. That draft is an individual submission. The document itself says it has no formal standing in the IETF standards process yet, so this is not a finished standard.

The timing. Nothing changes today. Google says it aims to support the control "along with other AI providers in 2027". Its documentation says early 2027. No other provider is named.

The detail that matters. Google says that when a site uses the shared token to opt out of training, and doesn't separately mention Google-Extended, Google plans to treat that "as if the publisher had disallowed Google-Extended".

What Google-Extended controls. This is the part most commentary will skip. According to Google's documentation, Google-Extended governs whether content may be used for training future Gemini models and for grounding: providing content to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. In the same announcement, Google writes that websites which specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps".

What it doesn't control. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Neither the announcement nor the documentation says anything specific about AI Overviews or AI Mode, so I won't speculate.

How other providers draw the line

Other providers separate the two decisions more cleanly, at least on paper.

• OpenAI uses separate tokens. GPTBot relates to training its foundation models; OAI-SearchBot relates to appearing in ChatGPT search results. OpenAI's documentation says each setting is independent and gives the example of allowing OAI-SearchBot while disallowing GPTBot.
• Anthropic lists three: ClaudeBot (restricting it signals that future material should be excluded from training), Claude-SearchBot and Claude-User. Anthropic says disabling either of the last two may reduce a site's visibility in user search results.
• Perplexity says PerplexityBot exists to surface and link websites in its search results and is not used to crawl content for AI foundation models.

So the landscape is uneven. With some providers you can say "no training, yes search" in two lines. With Google, the Gemini training control and Gemini Apps grounding currently travel together. Google Search itself sits outside both.

Interpretation: what this means

Inference, labelled as such. Read together, Google's wording suggests that a firm which opts out of AI training through Google-Extended, or from 2027 through the shared token without a Google-specific exception, may give up eligibility to be drawn on and linked in Gemini Apps' grounded answers. Google hasn't put it in those negative terms. It has said only that allowing Google-Extended makes a site eligible. But that's the natural reading, and I think firms should plan on it until Google says otherwise.

Opinion. The shared signal is a good idea. A single, readable line is better than a growing list of company-specific names that few web managers keep up to date. But convenience carries a risk. A broad "no AI training" line is exactly the kind of setting a cautious firm, a web developer or a security-minded host might add with good intentions and no awareness of the grounding consequence. In 20 years of working with law firms, I've seen plenty of consequential website settings that nobody at the firm had ever chosen. Robots.txt is a classic example.

There's a second, quieter point. If a firm blocks retrieval, AI assistants don't stop answering questions about that firm's practice areas. They answer from other sources: directories, competitors, review sites, news. Opting out of the conversation doesn't end the conversation.

Implications for law firms

• It's a partner-level decision. Whether the firm's published expertise may train commercial models, and whether AI assistants may read it to answer clients, is a question of brand, intellectual property and business development. It shouldn't be settled by a plugin default.
• "No training" and "no visibility" are different outcomes. Treat them separately where the provider lets you, and understand the bundle where it doesn't.
• Robots.txt expresses preferences. It relies on crawlers choosing to honour it. It's not a security control. Anything genuinely confidential shouldn't be on a public website in the first place.
• This is measurable. Your server logs show which AI user agents visit and how often. That's more useful than any vendor's claims, and it's a reminder that, as I've argued before, ranking in ChatGPT is the wrong mental model. Access comes first.

What we don't know

• Prevalence. I've found no reliable data on how many law-firm websites block AI training or retrieval agents, deliberately or by accident.
• Commercial effect. There's no public evidence on how much enquiry volume, if any, comes through Gemini Apps grounding for law firms, or what blocking costs.
• Adoption. Which other providers will honour the shared token, and how they'll map it to their own controls, is not yet known.
• The final text. The draft may change before anyone implements it.
• AI Overviews and AI Mode. Google's documentation for this control doesn't address them.
• Reporting. Google's AI performance reporting in Search Console doesn't show Gemini Apps grounding, so firms can't yet see the value directly.

Confidence summary

What to do this quarter

The good news is that you have time. Nothing takes effect until 2027. A sensible firm can settle this before the end of the quarter.

1. Read the file. Open yourfirm.com/robots.txt. List every AI-related user agent and what it's told. Ask your web provider whether a plugin, host or CDN setting adds rules or blocks bots at the server level. Some do this outside robots.txt entirely.
2. Separate the decisions. For each provider, decide two things: may our content be used for training, and may assistants read it to answer users? Write the answers down. For Google, recognise that Google-Extended currently bundles Gemini training with Gemini Apps grounding.
3. Name an owner. A partner, usually whoever owns marketing or risk, should approve the policy. Your web developer implements it.
4. Check your logs. Ask for a monthly count of visits by the AI user agents each provider documents. That's your baseline before anything changes.
5. Diary 2027. When Google and others implement the shared token, decide whether to use it and whether to add explicit provider-specific lines alongside it. Google says company-specific controls will keep working.

The AI visibility checklist covers the wider technical audit, and the AEO for law firms page sets out how access fits into the bigger picture.

What to avoid

Don't block everything because it feels safe. It may be the right choice for your firm. But make it knowingly, aware that it may cost discovery.

Don't allow everything because it feels modern. Training is a legitimate concern, and most providers let you decline it without disappearing from their search.

Don't buy "AI crawler optimisation". The documentation is public and short. Ask anyone selling this what they know that the providers' own pages don't say.

Don't treat robots.txt as confidentiality. It's a polite request to well-behaved crawlers.

Conclusion

The interesting question isn't whether law firms should block AI. It's whether anyone at the firm has actually decided. For most firms, robots.txt was never a policy document. It is now. The firms that come out of 2027 well will be the ones that made two separate decisions on purpose, rather than one by accident.

FirmRanker, the research platform I founded, studies how AI systems discover and cite law firms. Its methodology explains how it measures that. At Practice Proof, we help firms make and implement decisions like this one as part of their wider digital and AI visibility work.

FirmRanker was founded by Dan Toombs. Practice Proof, also founded by Dan, provides digital and AI visibility services to law firms.