llms.txt and AI crawlers: the technical setup for AI visibility

What llms.txt, robots.txt and sitemaps do for AI visibility

These three files are separate layers with a clear division of labour. robots.txt is access control: for each crawler it defines which areas may be read. The XML sitemap is the map: it lists your most important, canonical URLs so nothing is missed. llms.txt is the invitation: it offers AI assistants a curated, easy-to-read summary of your key content in Markdown.

For realistic expectations: none of these files forces a citation. They ensure your content is discoverable, machine-readable and accessible per your policy - the precondition for GEO (Generative Engine Optimization). robots.txt and sitemaps are established, widely supported standards. llms.txt, by contrast, is a young proposal (llmstxt.org, 2024) and is not yet automatically honoured by every major AI provider - so never rely on it alone.

The key AI crawlers and their user-agents

A crucial point: training, search and live fetching often run under different user-agents from the same provider. Block everything indiscriminately and you also lose the channels that produce citations. The key names (as of 2026; labels change over time):

• GPTBot (OpenAI) - crawler for ChatGPT training/index. Blocking removes content from the training corpus.
• OAI-SearchBot & ChatGPT-User (OpenAI) - power ChatGPT search results, or fetch a page when a user asks ChatGPT to open a link. Block = no citations in ChatGPT search.
• ClaudeBot & Claude-User (Anthropic) - crawler for Claude and user-initiated fetch. Relevant for visibility in Claude answers.
• PerplexityBot & Perplexity-User (Perplexity) - indexing and live fetch for Perplexity's sourced answers.
• Google-Extended (Google) - not a separate crawler but a robots token: it only controls whether content trains Gemini/Vertex AI. It affects neither Googlebot nor Google AI Overviews.
• Googlebot (Google) - the classic crawler behind Google Search and AI Overviews. Block it and you disappear from organic search.
• Other training agents: CCBot (Common Crawl, the basis of many models), Applebot-Extended (Apple Intelligence), Bytespider (ByteDance), meta-externalagent (Meta).

Configuring robots.txt correctly - with example

Decide your strategy first: you generally want to allow search and fetch bots (OAI-SearchBot, PerplexityBot, ClaudeBot), because they are exactly what produces cited answers. For pure training bots (GPTBot, CCBot, Google-Extended), decide based on your brand and IP strategy. A pragmatic example of a visibility-oriented policy:

# robots.txt - https://www.weissmann.ai/robots.txt\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: *\nAllow: /\nDisallow: /internal/\n\nSitemap: https://www.weissmann.ai/sitemap.xml

To limit AI training without losing Google Search, block selectively: a dedicated "User-agent: GPTBot / Disallow: /" block plus "User-agent: Google-Extended / Disallow: /" keeps you out of training while Googlebot and OAI-SearchBot keep reading your content. Know two limits: robots.txt is only a directive - reputable bots obey, malicious ones do not (for that you need server-side blocks or a WAF). And the file is publicly viewable, so never list secret paths in it.

Creating llms.txt: structure and example

llms.txt lives at /llms.txt and is plain Markdown. The proposed structure: an H1 with your name, a short block-quote summary, then thematic sections with Markdown links to your most important canonical pages, each with a brief description. Optionally add a /llms-full.txt containing the full text. Keep the list short, current and curated.

# Weissmann AI Academy\n\n> Swiss knowledge platform on AI, SEO and GEO for SMEs.\n\n## Core guides\n- [What is GEO](https://www.weissmann.ai/en/wissen/geo): fundamentals of Generative Engine Optimization\n- [llms.txt & AI crawlers](https://www.weissmann.ai/en/wissen/llms-txt): this technical setup\n\n## About\n- [About Weissmann](https://www.weissmann.ai/en/about): team, approach, contact

Two principles decide its value: accuracy and maintenance. Link only real, reachable URLs, describe content honestly, and update the file whenever something important changes. A stale llms.txt with dead links hurts trust more than having none.

Sitemaps and hreflang for multilingual visibility

The XML sitemap lists your canonical URLs with a lastmod date. Reference it in robots.txt (the "Sitemap:" line) and submit it in Google Search Console and Bing Webmaster Tools - Bing feeds Copilot, among others. For Switzerland, multilingual handling is central: hreflang annotations (de-CH, fr-CH, it-CH, en) signal which language version belongs to which audience.

For AI visibility this counts double: clean hreflang links help assistants cite the language-appropriate version - instead of backing a German answer with the Italian URL. Maintain realistic lastmod values and keep the sitemap free of redirects, error pages and non-canonical duplicates.

Common mistakes and the Swiss perspective

Swiss perspective: your crawler policy is part of data governance. If you publish personal or sensitive content, the revised Data Protection Act (revDSG/nFADP) and oversight by the FDPIC (EDOEB) apply - regardless of which bot reads. It is also worth a deliberate decision on copyright and the use of your content for AI training. For the multilingual Swiss market, clean hreflang signals are mandatory. Where you would rather not implement or audit this configuration yourself, the Weissmann team supports you from policy to implementation.

• Blocking GPTBot yet expecting ChatGPT citations: citations come from OAI-SearchBot and ChatGPT-User, not GPTBot.
• Believing Google-Extended removes you from AI Overviews: those use the Googlebot index; Google-Extended only affects Gemini training.
• Mistaking robots.txt for a security control: it is a directive, not access control - and publicly readable.
• Contradictory rules or wrong wildcard order that accidentally blocks everything; and an llms.txt that is published but never maintained.