AI Crawlers Can't Be Told Where Your Pages Are. Make Them Easy to Find.

On the October 1 episode of Google's Search Off the Record podcast, Google's John Mueller made a point that's easy to miss but matters for every small firm website. Google gives site owners Search Console, a place to submit a sitemap and tell Google where their pages are. AI crawlers, he said, usually don't offer anything like that. In his words, they "usually don't have any kind of a Console or any setup where you can submit a sitemap file."

So how do they find your pages? Mueller said he's seen it in his own server logs: "some AI crawler accesses my sitemap file." He's seen the same with his RSS feeds. His advice for sites that want AI systems to find their content: "either you stick to the generic naming, call it sitemap.xml, or you focus on RSS feeds."

That's not a strategy question. It's a five-minute check — and many immigration firm websites would fail it without anyone at the firm knowing.

What Mueller Actually Said

Search Engine Journal's Matt G. Southern reported the episode on October 5. A few details are worth getting right, because they shape what you should do:

• AI crawlers find files by convention. With no console to submit to, a crawler has to guess where your sitemap is. The obvious guess is `/sitemap.xml`. A sitemap with a different name, or one that isn't listed anywhere, may simply go unseen.
• RSS feeds are the other path. A feed is usually linked in the head of your page's HTML, so a crawler that reads your homepage can find it without guessing.
• He saw it himself, but didn't name names. Mueller didn't say which AI crawlers fetched his files, and said he doesn't know whether AI companies document this or what they do with the files once fetched. Treat this as an observation from one well-placed engineer, not a published spec.
• llms.txt isn't a substitute. Asked whether the newer llms.txt file could replace an XML sitemap, Mueller compared it to an HTML sitemap, said Google's systems can't use it as one, and added: "I think the hope is bigger than the reality." He doesn't object to sites trying it. He just wouldn't rely on it.

The takeaway is plain: before worrying about anything new, make sure the old, standard files exist where any system would look for them.

Why Small Firm Sites Often Miss This

Most immigration practices didn't build their own websites. A web designer set one up years ago, or the firm runs on a builder or a legal-marketing platform, and nobody has looked at the plumbing since.

That's where the gaps come from. Some platforms and plugins name the sitemap something other than `sitemap.xml` — `sitemap_index.xml` or `wp-sitemap.xml`, for example — and whether the standard address redirects to it depends on the setup. Some sites have a sitemap that isn't listed in their `robots.txt` file. And many firm sites have no RSS feed at all, because the firm never had a blog, or the blog lives on a separate platform, or a redesign quietly dropped it.

None of this shows up when you look at your homepage. The site looks fine. Your practice area pages for family-based petitions, asylum, removal defense, and naturalization all load. But a crawler arriving with no instructions may find only what's linked from the pages it happens to land on — and miss the attorney bio, the consultation page, or the article explaining what changed with the latest USCIS fee rule.

Visibility Starts Before the Answer

When a family asks ChatGPT, Perplexity, or Google's AI Mode which immigration attorney in their city handles consular processing, the answer can only draw on pages these systems have actually found. Good content that's hard to discover is, for practical purposes, content that isn't there.

That's why this matters more than its technical feel suggests. Firms spend real effort on what their pages say: clear fee ranges, the languages they consult in, the case types they take. That work only pays off if the pages are reachable. A standard sitemap and a working feed are the most basic way to make sure they are.

It's also worth keeping the claim in proportion. A correctly named sitemap won't make an AI answer recommend your firm. Mueller noted on the same episode that even Google may skip a sitemap when it doesn't see a need to crawl more of a site, and that crawl demand is "very often based on the perceived quality of a website." Discoverability is the floor, not the ceiling. But it's a floor many firms haven't checked.

Five Checks to Run This Week

1. Visit `yourfirm.com/sitemap.xml`. Type it into a browser. You should see a structured list of your pages (or a list of smaller sitemaps). If you get a 404 page, find out what your platform calls its sitemap and whether the standard address can redirect to it.
2. Open `yourfirm.com/robots.txt`. Look for a line that starts with `Sitemap:` followed by your sitemap's full address. If it's missing, ask your web person to add it. It's one line, and it points any crawler that reads the file to your sitemap.
3. Check for an RSS feed. On your homepage, view the page source and search for "rss" or "application/rss+xml." If nothing comes up, check whether your blog platform offers a feed and turn it on. If your firm publishes updates on DACA, H-1B, or PERM developments, a feed is how those updates get noticed.
4. Make sure the pages that matter are listed. Open the sitemap and confirm it includes your consultation page, attorney bios, and each practice area page. A sitemap that lists only blog posts, or still lists pages you deleted in a redesign, is worth fixing.
5. Ask for a look at your server logs. Your host or web person can tell you whether AI crawlers are visiting your site and which files they request. Mueller's point came from his own logs — yours will tell you what's happening at your firm, not anyone else's.

Most of this takes a few minutes and one email to whoever manages your site.

The Bigger Picture

There's a lot of talk right now about new files, new formats, and new tactics for AI search. Some of it will matter. But the most useful advice from Google this month was about files that have existed for close to two decades: a sitemap at the address everyone expects, and a feed linked from your pages.

For a small immigration practice, that's good news. You don't need a new vendor or a new strategy to get the basics right. You need to confirm that the pages you've already built — the ones that explain what you do and who you serve — can actually be found by the systems families are starting to ask first.

Clientory helps immigration law firms become visible to the families and individuals who need them most — turning AI search into consultations, and consultations into clients.

See Your AI Visibility Score → CLIENTORY.ORG