Websites · Glossary
What is robots.txt?
robots.txt is a plain text file at the root of a website that tells crawlers which paths they may or may not fetch. It follows the Robots Exclusion Protocol, standardised as RFC 9309. Well behaved crawlers such as Googlebot respect it, but it is a request, not a lock, and it does not remove pages from search results.
Every crawler that follows the rules reads robots.txt before fetching anything else from a site. It is the place to keep crawlers out of admin areas and endless search pages, and to point them to your sitemap.
Indunil Asanka · Co-founder
4 min read · Published
| Directive | Example | What it does |
|---|---|---|
| User-agent | User-agent: * | Names which crawler the following rules apply to; * means all |
| Disallow | Disallow: /admin/ | Asks the crawler not to fetch paths starting with /admin/ |
| Allow | Allow: /admin/help | Re-allows a path inside a disallowed folder |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Tells crawlers where the sitemap is; may appear anywhere in the file |
| Comments | # staff area | Ignored by crawlers; notes for people |
What it is
robots.txt is a text file served at /robots.txt on each host, so example.com and shop.example.com each need their own. It contains groups of rules: each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow lines with path prefixes. The Robots Exclusion Protocol was informal for decades and was published as RFC 9309 in 2022. Google's documentation describes how Googlebot reads it, including that it uses the most specific matching rule and ignores content beyond a 500 KiB limit.
How crawlers use it
Before crawling a site, a compliant crawler fetches robots.txt and caches it, usually for up to a day. When it finds a link, it checks the path against the rules for its user agent and skips disallowed paths. If the file is missing, crawlers assume everything is allowed. A Disallow rule stops fetching, not indexing: if other sites link to a blocked page, a search engine can still list the URL without a description. To keep a page out of results, let it be crawled and add a noindex tag, or protect it with a login.
When it matters
Most small business sites need only a short file. It matters when the site has areas with no search value, such as an admin area, account pages, a checkout or internal search results that create endless URLs. It matters during a launch, because a leftover Disallow: / from a staging site blocks the whole live site. And it matters for AI crawlers, which some site owners choose to block by user agent, though only crawlers that respect the file will comply.
Common mistakes
Shipping Disallow: / from a test site to the live one. Blocking CSS and JavaScript, which stops search engines rendering pages properly. Using robots.txt to hide private pages, when the file itself is public and lists exactly where they are. Expecting a disallowed page to vanish from results. Blocking a page and adding noindex to it, so the crawler never sees the noindex. Forgetting that each subdomain needs its own file. And typing rules with the wrong case, since paths are case sensitive.
Where it shows up in OneCraft
With the seo capability guide, the agent writing a OneCraft website adds a robots.txt alongside Open Graph tags and JSON-LD; every route already carries a title, description and canonical address. A sitemap.xml is written by hand only when the domain is known, and there is no automatic sitemap. Admin areas built from the admin guide sit behind sign in and are shown only to people listed in the site's site_admins table, which protects them in a way robots.txt cannot.
Questions people ask
Does every website need a robots.txt file?
No. Without one, crawlers assume they may fetch everything, which is fine for many small sites. A short file is still useful to point to your sitemap and keep crawlers away from admin or search pages. If you add one, check it never blocks the whole site by mistake.
Can robots.txt stop my page appearing in Google?
Not reliably. Blocking a page stops Google fetching it, but the URL can still appear in results if other pages link to it. To keep a page out, allow crawling and add a noindex meta tag or header, or put the page behind a login.
How do I block AI crawlers with robots.txt?
Add a group for each crawler's published user agent with Disallow: /. Several AI companies publish their crawler names and say they respect robots.txt. Others may not, and blocking one crawler does not stop content reaching a model through other sources, so treat it as a request, not a guarantee.
Where does robots.txt go?
At the root of each host, so it loads at https://yourdomain/robots.txt. A file in a subfolder is ignored. A subdomain such as shop.yourdomain needs its own file, and the http and https versions of a host are treated as separate origins by the standard.
How do I test my robots.txt?
Open the file in a browser to check it loads, then use Search Console's robots.txt report to see the version Google fetched and any errors. Test important URLs with the URL Inspection tool, which says whether a page is blocked by robots.txt.
Make one with websites
The button opens the generator with this use case already described. Change the wording to match your own.
Create a website with OneCraftRelated questions
- What is JSON-LD?What is JSON-LD? Linked data in a script tag that describes a page to search engines: the format, the schema.org types a small business uses, and mistakes.
- What is a 301 redirect?What is a 301 redirect? A permanent redirect that sends visitors and search engines to a new address, and how it differs from 302, 307 and 308.
- What is a sitemap?What is a sitemap? A file listing the pages of your site so search engines find them, usually sitemap.xml. What it lists, when it helps and how to submit it.
- What is noindex?What is noindex? A tag that asks search engines not to show a page in results while it stays online. When to use it, how it differs from robots.txt, and risks.
Written and checked by the OneCraft team. Last checked .