What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which parts of a site they are allowed to fetch, and llms.txt tells AI systems what the site contains and which pages matter. One is about permission and the other is about comprehension, so a site can use both and they do not conflict.

llms.txt vs robots.txt: What Is the Difference?

How AI sees your site: two text files, robots.txt and llms.txt — Plays from YouTube when you press play · Open on YouTube

What robots.txt does

It is a long-established convention that every major crawler reads. It lists user agents and the paths each may or may not request. For AI, the lines that matter are the ones naming crawlers such as GPTBot, ClaudeBot and PerplexityBot. A blanket block there stops an assistant from reading the site at all.

What llms.txt does

It is a Markdown file at the site's root with a short summary of the company and an annotated list of its key pages. It grants nothing and blocks nothing. It is an editor's guide: if a model fetches it, it gets an accurate account without having to infer one from navigation menus.

How much weight to give each

robots.txt is enforced by convention and respected widely, so getting it wrong has a direct cost. llms.txt is young. Google has said it gives the file no special treatment in Search, and no assistant has committed to reading it. Treat it as cheap groundwork, not as a lever for ranking.

What to do

Check robots.txt first, because a block there outweighs everything else. Then publish an llms.txt if you have an afternoon: keep it short, link to your own domain, and describe each page in one line. Generate it from the site so it does not go stale. Neither file replaces clear pages: a crawler that is let in and handed a map still has to find plain statements of what you do and for whom.

Want this built inside your own stack? Book a call, or ask the concierge in the nav.