What Should a robots.txt File Look Like? Examples

A robots.txt file is a plain text file at the root of your site that tells crawlers which paths they may fetch. A good one for most small sites is only a few lines: one group for all crawlers, a short list of disallowed paths such as admin or cart pages, and a Sitemap line. Anything more should have a reason.

What does a basic robots.txt file look like?

Here is a complete, valid file for a typical small site:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search?

Sitemap: https://www.example.com/sitemap.xml

Each block is called a group. It starts with a User-agent line that names the crawler the rules apply to, followed by Disallow and Allow lines. The asterisk means all crawlers. The Sitemap line is optional, sits outside any group, and must be a full URL. You can repeat it for several sitemaps.

If you want everything crawled, the shortest valid file is a group with an empty Disallow, or no robots.txt at all. Google treats pages that are not blocked as allowed by default, according to its guide to creating a robots.txt file.

Where does the file go, and how do you find yours?

The file must be named robots.txt and live at the root of the host it governs, for example https://www.example.com/robots.txt. Google says it cannot go in a subdirectory, and that the rules apply only to the exact protocol, host and port where the file is posted. A blog on blog.example.com needs its own file.

To find yours, add /robots.txt to your domain in a browser. If you get a text file, that is it. If you get a 404 page, you have none, and crawlers will assume nothing is restricted. Many site builders and CMSs generate the file for you, so check the settings screen before creating one by hand.

What do the rules mean?

Disallow

A path that crawlers should not fetch. Paths start with a slash. A path that ends in a slash, such as /admin/, covers that directory and everything under it. Paths are case-sensitive, so /Admin/ and /admin/ are different.

Allow

An exception inside a blocked directory. This lets you block a folder but still permit one page in it:

User-agent: *
Disallow: /private/
Allow: /private/press-kit.html

Wildcards

Google's robots.txt specification says major search engines support a limited form of wildcards in paths. An asterisk matches zero or more of any character, and a dollar sign marks the end of a URL. So Disallow: /*.pdf$ blocks URLs that end in .pdf.

Comments

A line that starts with # is a comment. Use them. Six months from now you will not remember why /tmp-landing/ is blocked.

What should you put in a robots.txt file, and what should you leave out?

Block the paths that waste crawler time and that nobody should land on from search:

• Admin and login areas
• Cart and checkout steps
• Internal search result pages
• Filtered or sorted URL variations that duplicate a category page

Leave out the things that people commonly add by habit:

• Your CSS and JavaScript files. Search engines need them to render the page the way a visitor sees it.
• Pages you want kept out of results. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out, use a noindex tag and let the crawler read it. The full explanation is in robots.txt vs noindex.
• Secrets. The file is public. Listing a private path in it advertises that the path exists.

How do you write rules for different crawlers?

A crawler follows the group whose User-agent line matches it most specifically, and ignores the others. That means a named group replaces the asterisk group for that crawler. It does not add to it. If you write a group for one bot, repeat any general rules you still want applied to it.

This matters for AI crawlers. OpenAI documents three separate user agents in its bots documentation, and it says each setting is independent:

• OAI-SearchBot surfaces sites in ChatGPT search. Allowing it helps your pages appear there.
• GPTBot collects content that may be used to train OpenAI's models. Disallowing it signals you do not want that.
• ChatGPT-User fetches pages when a person asks ChatGPT to, and OpenAI notes that robots.txt rules may not apply to it.

So a site that wants to appear in ChatGPT search but not feed training could use:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Whether to block training crawlers is a business decision, not a technical one. Our guides to GPTBot and to allowing ChatGPT and Perplexity cover the trade-offs.

What are the most common robots.txt mistakes?

1. Blocking the whole site. Disallow: / under User-agent: * is the classic. It often survives from a staging site into production. Check this first if a launch has no traffic.
2. Using the wrong file location. A file in a subfolder is ignored.
3. Typing the path wrong. Rules are case-sensitive and match from the start of the path.
4. Using a word processor. Google says the file should be UTF-8 plain text, and a word processor can insert curly quotes that make rules invalid.
5. Relying on it for privacy. Use authentication for anything that must not be reachable.

How does Google handle a missing or broken robots.txt?

From Google's specification: a 4xx response, other than 429, is treated as if no valid robots.txt exists, so Google assumes there are no crawl restrictions. If the file was found before and a server error (5xx) now prevents fetching it, Google stops crawling for the first 12 hours while it retries, then uses the last good cached copy for up to 30 days. Google also caches the file for up to 24 hours in general, so a fix may not take effect right away. The spec states a maximum size of 500 KiB, and content past that is ignored. Small sites will never come near that.

How do you test your robots.txt?

1. Open the file in a browser and read it line by line. Look for a stray Disallow: /.
2. Take three important URLs and check that no rule matches their paths.
3. Inspect one of those URLs in Search Console's URL Inspection tool to see whether crawling is allowed.
4. After any change, wait at least a day before judging the result, because of caching.

Nightcrop's audit includes a check that AI crawlers can reach your site, which is useful when you suspect a rule or a firewall is turning them away.

Frequently Asked Questions

Do I need a robots.txt file?

No. Without one, crawlers assume they can fetch everything. You need one only when there are paths to keep crawlers out of, or when you want to point to your sitemap or set rules for specific bots.

Can robots.txt keep a page out of Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed from links elsewhere. Use a noindex tag, and leave the page crawlable so the tag can be read.

Should I add my sitemap to robots.txt?

It is a good habit. The Sitemap line takes a full URL and helps crawlers find the file, though submitting it in Search Console works too.

Does robots.txt apply to subdomains?

No. Each host, protocol and port needs its own file. The file on www.example.com does not cover blog.example.com.

How long until a change takes effect?

Google generally caches the file for up to 24 hours, and OpenAI says search-related changes can take about 24 hours. Allow a day.