Analyzes and generates llms.txt files -- the emerging standard for helping AI systems understand website structure and content. Can validate existing llms.txt files or generate new ones from scratch by crawling the site.
This skill handles everything related to the llms.txt standard -- an emerging convention (proposed by Jeremy Howard in September 2024, gaining adoption through 2025-2026) that allows websites to provide structured guidance to AI systems about their content, structure, and key information. It is analogous to robots.txt (which tells crawlers what NOT to access) but instead tells AI systems what IS most useful to understand about the site.
AI language models face a fundamental challenge when processing websites: they must determine which pages are most important, what the site is about, and how content is organized -- typically by crawling many pages and inferring structure. llms.txt solves this by providing an explicit, machine-readable (and human-readable) summary.
Benefits of having a well-crafted llms.txt:
The file MUST be located at the root of the domain: