AI sitemaps and content discovery

Discovery for AI engines works much like for search engines - their crawlers find content through XML sitemaps (which list your URLs), internal links (which lead crawlers from page to page), and by respecting your robots rules. There's no separate mandatory 'AI sitemap' format; the standard discovery infrastructure serves AI crawlers too. Getting the fundamentals right is what makes your content discoverable by AI.

A complete, current XML sitemap listing all your citable URLs is the discovery foundation. It tells crawlers exactly what content exists and when it was last updated. Keep it complete (every published page), current (accurate lastmod dates), and referenced in robots.txt. A missing or stale sitemap means content may go undiscovered - the simplest, highest-leverage discovery fix.

Two more pillars: internal linking and crawler access. Strong internal links help crawlers discover pages and understand how content relates (reinforcing topical clusters). And robots.txt must allow AI crawlers (GPTBot, PerplexityBot, etc.) - if you block them, they can't discover you no matter how good your sitemap is. Check both: links lead crawlers through your site, robots lets them in.

llms.txt is an optional additional signal that curates your best pages for AI - useful as a forward-looking supplement, but it is not a required 'AI sitemap' and doesn't replace the standard XML sitemap. The reliable discovery stack is: complete XML sitemap + strong internal links + AI-crawler access in robots.txt, with llms.txt as a low-cost extra. Don't skip the fundamentals in favor of the emerging signal.