In short
- llms.txt is a proposal from 2024: a Markdown file in the root directory that gives AI systems a short overview of your website.
- Google states that you do not need new machine-readable files or AI text files to appear in AI Overviews and AI Mode.
- So far there is no solid evidence that the major AI search systems use the file for recommendations.
- It costs little and does no harm. Open crawlers, readable pages and clean structured data matter more.
What an llms.txt file is
The idea comes from Jeremy Howard, who published it on llmstxt.org in September 2024. Websites are meant to offer language models and AI agents a compact summary in Markdown: who they are, what the site contains and which pages matter for which question. The background is that AI systems can only process a limited amount of text at once, and full web pages with menus, scripts and ads are hard to read.
The file usually sits at /llms.txt in the root directory, but according to the proposal it can also live in a subfolder and then covers the addresses below it. Some websites also offer an llms-full.txt with the full text. That is a common addition, not part of the core proposal.
How the file is structured
The proposal sets a fixed order. Only the first line is mandatory:
- A first-level heading with the name of the website or business.
- A blockquote with a short summary in one or two sentences.
- Free paragraphs or lists with important notes, without further headings.
- Sections with second-level headings, each containing a list of links, optionally with a short description after a colon.
- An “Optional” section for links an AI system can skip when space is short.
For a trade business it could look like this:
# Example Electrical Ltd
> Electrical installation, solar panels and EV chargers for homes and businesses in Leeds, Wakefield and the surrounding area.
Emergency service on weekdays from 7 am to 6 pm. Free quotes after a site visit.
## Services
* [Solar panels](https://www.example-electrical.co.uk/solar/): design, installation, grant advice
* [EV chargers](https://www.example-electrical.co.uk/ev-chargers/): installation for homes and businesses
## Company
* [Contact and opening hours](https://www.example-electrical.co.uk/contact/)
## Optional
* [Frequently asked questions](https://www.example-electrical.co.uk/faq/)
What the file does, and what it does not
A sober look pays off here. In its guidance on AI features, Google says you do not need new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. In its overview of crawlers, OpenAI explains how visibility in ChatGPT search is controlled through robots.txt, and llms.txt does not appear there. For Perplexity and Claude, too, access is what counts, here for PerplexityBot and Claude-SearchBot.
So far there is no publicly documented evidence that ChatGPT, Gemini or Perplexity recommend businesses more often because of an llms.txt. The file is useful where AI agents fetch content deliberately, for example technical documentation that developers search with AI assistants. For a local business it is more of a tidy business card for machines than a route to more enquiries. What else counts for being named in AI answers is covered on the page about generative engine optimization.
Our recommendation
Create an llms.txt once your website is in good shape and the more important points are done. Plan half an hour for it, not a project. Be critical of providers who sell llms.txt as the key to ChatGPT.
How to create an llms.txt
- Decide on the content: your name, one or two sentences on what you do and where, plus your five to fifteen most important pages.
- Write the file: a simple text editor is enough. Save the file as UTF-8 with the name
llms.txt. - Upload it: put the file in the root directory so that it is reachable at
your-domain.co.uk/llms.txt. WordPress and other systems have plugins for this. - Check it: open the address in your browser. The file must appear as text, without a login and without a redirect to the home page.
- Maintain it: when services or addresses change, update the llms.txt too. Outdated details do more harm than a missing file.
What matters more than llms.txt
- Your robots.txt allows OAI-SearchBot, Googlebot, Bingbot, PerplexityBot and Claude-SearchBot.
- The most important content is in the HTML and not only loaded via JavaScript.
- Each service has its own page with a clear heading and location.
- Structured data based on Schema.org describes your business, location and services.
- Google Business Profile and directories contain the same company details as your website.
- An up-to-date sitemap is listed in robots.txt.
llms.txt and robots.txt compared
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Controls which crawlers may fetch which pages | Gives AI systems a short overview in Markdown |
| Effect on visibility | Direct: if blocked, you are not shown in ChatGPT search | Not proven |
| Adoption | A standard for decades, respected by all major crawlers | A proposal since 2024, voluntary |
| Effort | A few lines, but mistakes have big consequences | Half an hour |
Sources
- Jeremy Howard: The /llms.txt file, proposal of 3 September 2024
- Google Search Central: AI features and your website
- OpenAI: Overview of OpenAI Crawlers
- Perplexity: Perplexity Crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?