What Is llms.txt?

Table of Contents

llms.txt is a proposed plain-text Markdown file placed in a website’s root directory (/llms.txt) that acts as a curated content index for Large Language Models (LLMs) and AI agents. Instead of forcing automated tools to parse heavy HTML navigation, scripts, and styling, it directs them straight to clean, high-priority pages and documentation.

Modern websites are built for visual browsers, which makes raw page code noisy and token-heavy for language models to read. By offering a structured directory at a predictable URL, llms.txt helps AI coding assistants, research agents, and chat tools locate accurate, up-to-date information faster.

How the Proposed Content Index Works

Introduced by Jeremy Howard of Answer.AI, the llms.txt specification defines a lightweight content index written in standard Markdown so both humans and machines can read it easily. Rather than listing every page on a domain, the file curates only the resources an AI system needs to understand a project, product, or organization.

A valid llms.txt file follows a strict top-down hierarchy:

  1. H1 Title (#): The name of the website, project, or company (the only required element).
  2. Blockquote Summary (>): A concise one-paragraph overview explaining what the site does and who it serves.
  3. Categorized H2 Sections (##): Thematic groups (such as Docs, Pricing, or Guides) containing bulleted Markdown links formatted as [Page Title](URL): Brief description.
  4. Optional Section (## Optional): A designated section for secondary resources that an AI agent can skip if its context window is limited.

Valid llms.txt Example:

Markdown

# Acme Cloud
> Acme Cloud provides managed database hosting and automated backup APIs for web developers.

## Core Documentation
- [Getting Started](https://acmecloud.example/docs/quickstart.md): Step-by-step setup guide for deploying your first cluster.
- [API Reference](https://acmecloud.example/docs/api.md): Endpoints, authentication headers, and rate limits.
- [Pricing Plans](https://acmecloud.example/pricing.md): Current compute tiers, storage rates, and bandwidth limits.

## Optional
- [Changelog](https://acmecloud.example/changelog.md): Monthly release notes and SDK updates.

Links to Useful Content, Markdown, and llms-full.txt

The llms.txt proposal works hand-in-hand with serving clean Markdown versions of existing web pages. When a site links to useful content inside /llms.txt—such as product specifications, API docs, or a /pricing.md file—it often points to stripped-down .md URLs rather than full HTML pages. Removing CSS, popups, and navigation menus reduces token consumption and lowers the chance of an AI model misreading page layout as body copy.

For workflows where an AI agent needs complete context without fetching multiple links one by one, the specification also defines a companion file called llms-full.txt:

  • llms.txt (The Index): A compact table of contents containing annotated links to your most useful pages.
  • llms-full.txt (The Full Corpus): A single consolidated Markdown file at /llms-full.txt that compiles the complete text of those referenced pages into one document.

While llms-full.txt allows an LLM to ingest an entire knowledge base in a single request, large websites must manage its size carefully so it does not exceed a model’s context window.

llms.txt vs. robots.txt vs. XML Sitemap

Although all three files help automated systems interact with a website, they handle different jobs and should be used together rather than treated as replacements.

Featurellms.txtrobots.txtsitemap.xml
Primary AudienceAI agents, coding assistants, and LLMsSearch engine crawlers and AI botsTraditional search engine indexers
Core PurposeCurates priority content with contextControls crawl permissions (allow/disallow)Lists all indexable URLs for discovery
File FormatHuman- and machine-readable MarkdownPlain-text crawler directivesStructured XML tags
Content ScopeSelective, high-value pages onlyDirectory and path access rulesComprehensive site-wide URL inventory

Adoption Limits and Google Search Requirements

Because llms.txt is an open community convention rather than a formal W3C or IETF web standard, adoption across the AI ecosystem remains uneven. Developer documentation platforms, SaaS companies, and AI-focused websites widely publish /llms.txt to support coding tools like Cursor and Claude, yet many consumer AI crawlers still rely primarily on standard HTML crawling rather than checking /llms.txt automatically.

Google does not require llms.txt and does not use it as a ranking factor. However, in June 2026, Google updated its guide to optimizing for generative AI with the following:

״It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.״

Practical Example

Consider a WordPress agency updating its service packages. If a potential client uses an AI assistant to research the agency’s deliverables, raw HTML pages filled with testimonials, sliders, and legacy blog posts can muddy the summary. By maintaining an /llms.txt index that points directly to clean Markdown summaries of its current service tiers and technical scope, the agency gives external AI tools a reliable source of truth to reference.

Frequently Asked Questions

Does adding llms.txt improve Google rankings or AI Overviews visibility?

No. Google has clarified that llms.txt is not a ranking signal and is not required for your pages to appear in Google Search or AI Overviews. Strong HTML structure, clear headings, schema markup, and authoritative content remain the primary drivers of search visibility.

When should I use llms-full.txt instead of llms.txt?

Use llms.txt as your default starting point so AI agents can selectively open only the links they need. Add llms-full.txt when you maintain focused technical documentation or product guides that fit comfortably within an LLM context window and benefit from single-request ingestion.

How can I check if my website is ready for AI agents?

You can scan your domain with the Agent Ready Checker to see how well external AI agents can read and navigate your site structure, or explore what agents actually use when interacting with modern websites.

Sources

Building Agent-Ready Websites with Elementor

Making your website easy for AI agents to navigate starts with clean, semantic page architecture. With Elementor MCP, you can connect your WordPress site directly to external AI tools like Claude and Cursor to build and refine real, editable Atomic layouts. Paired with the in-editor AI Website Builder and the free Agent Ready Checker, Elementor helps you structure your pages clearly for both human visitors and agentic workflows.

Share

Recommended From Elementor

What Is an AI Agent?

An AI agent is an autonomous software system that combines a reasoning language model, external digital tools, and an iterative action loop to plan, execute, and verify multi-step tasks toward a defined goal across live applications without requiring step-by-step human instructions.

Read More »

What Is an AI Crawler?

An AI crawler is an automated software program that systematically retrieves text, images, and code from websites to feed data into artificial intelligence systems.

Read More »

What Is llms.txt?

llms.txt is a proposed plain-text Markdown file placed in a website’s root directory (/llms.txt) that acts as a curated content index for Large Language Models (LLMs) and AI agents. Instead of forcing automated tools to parse heavy HTML navigation, scripts, and styling, it directs them straight to clean, high-priority pages and documentation.

Read More »

What Is WebMCP?

WebMCP is a proposed browser API that allows a webpage to securely expose tools and local context directly to an AI assistant running on a

Read More »

The Future
of Web Creation. Straight to
Your Inbox.

What Intrests You?


Great!

Awsome content is on the way.