llms.txt Builder

100% Client-Side

AI Rules & Metadata

Configure how AI models (LLMs) should interpret and crawl your website.

Protocol Warning:

An llms.txt file does not block AI web scrapers. It is only an informational guide for Agentic AI. To actually stop companies like OpenAI or Anthropic from scraping your site, you must also use a robots.txt file. We will generate both for you automatically.

1. Site & Brand Context

2. AI Training & Scraping Policies

3. Agent Instructions & Navigation

4. Contact & Legal

1. Generated llms.txt

Place this file at yourdomain.com/llms.txt to guide AI Agents.

2. Companion robots.txt Security

Append this to yourdomain.com/robots.txt to physically enforce your scraping policy against major AI bots.

Frequently Asked Questions

What is an llms.txt file?

An llms.txt file is a proposed standard directory file placed at the root of a website (similar to robots.txt) designed specifically to be read by Large Language Models (LLMs) and AI Agents. It typically contains YAML metadata and Markdown content that summarizes the website's purpose, provides training policies, and links to relevant documentation, ensuring AI models accurately understand and represent your brand.

What is the difference between llms.txt and robots.txt?

This is a critical distinction: robots.txt is an enforcement protocol checked by web crawlers before they visit a page. If you want to block AI companies (like OpenAI or Anthropic) from scraping your site for training data, you must use robots.txt. Conversely, llms.txt is an informational guide. It tells an AI agent (which has already been allowed to crawl) how to read and summarize your content. Generating an llms.txt file with a "Disallow" policy will not physically stop a crawler from taking your data.

How do I completely opt out of AI training?

To explicitly opt out, you must use our builder above and set the Training Policy to "Disallow." Our tool will generate the appropriate YAML flag in your llms.txt file, and more importantly, it will generate a companion robots.txt snippet that explicitly blocks known AI training crawlers like GPTBot, CCBot, ClaudeBot, and Applebot.

Why does this tool generate YAML frontmatter?

Automated web loaders (such as LangChain's WebBaseLoader or LlamaIndex) parse text files programmatically. By placing metadata (like canonical URLs and contact info) inside standard YAML frontmatter blocks (between ---), these automated frameworks can extract your policies systematically without getting confused by the readable Markdown text in the body.

Action successful!