Use noindex to keep a page out of Google. Use robots.txt to control what crawlers fetch. They do different jobs, and combining them on the same page backfires: if robots.txt blocks the page, Google never reads the noindex tag, so the URL can still show up in results.
What is the difference between robots.txt and noindex?
A robots.txt file is a plain text file at the root of your site that tells crawlers which URLs they may request. It controls crawling. A noindex rule is an instruction on a specific page that tells search engines not to put it in their index. It controls indexing.
Crawling and indexing are separate steps. A crawler fetches a page, then the search engine decides whether to store and show it. Blocking the first step does not guarantee anything about the second.
| robots.txt Disallow | noindex | |
|---|---|---|
| Where it lives | One file at the site root | In each page's HTML or HTTP headers |
| What it controls | Whether a crawler requests the URL | Whether the page is kept in the index |
| Can the page still appear in results? | Yes, as a bare URL | No, once Google has seen the rule |
| Needs the page to be crawlable? | No | Yes |
Does robots.txt keep a page out of Google?
No. Google's own documentation says that robots.txt is not a mechanism for keeping a web page out of Google. It adds that a page blocked this way can still appear in search results, just without a description.
That happens because Google can learn a URL exists from links elsewhere on the web. It cannot read the page, so it has nothing to show except the address. Many site owners block a private or unfinished page in robots.txt, then find its bare URL in search a week later.
How does noindex work?
Google describes noindex as a rule set with either a meta tag or an HTTP response header. You have two options:
- Meta tag: put
<meta name="robots" content="noindex">in the page's<head>. - HTTP header: return an
X-Robots-Tag: noindexheader. This is the only option for files that have no HTML, such as PDFs.
After the next crawl, Google drops the page from its results. It is not instant. The page has to be recrawled before the rule is seen.
Why does blocking and noindexing the same page fail?
This is the mistake behind most "I noindexed it but it is still there" reports. Google states that for the noindex rule to be effective, the page must not be blocked by a robots.txt file. If it is blocked, the crawler never sees the noindex rule.
Here is the sequence that goes wrong:
- You add
noindexto a page. - You also add
Disallow: /that-page/to robots.txt, to be thorough. - Googlebot respects robots.txt and does not fetch the page.
- The noindex tag is never read, so the URL stays in the index if it was already there.
The fix is to remove the Disallow line, let Google recrawl the page, confirm it has dropped out, and only then decide whether you still want the block.
Which one should you use?
Match the tool to the goal:
- Keep a page out of search results: noindex, with the page left crawlable.
- Keep private content private: password protection. Google lists this next to noindex as the way to keep a page out, and neither robots.txt nor noindex stops a person who has the link.
- Stop crawlers wasting requests on endless URLs such as internal search results or filter combinations: robots.txt Disallow.
- Handle duplicate pages: a canonical tag or a redirect, not either of these. See our guide to the canonical URL.
A quick test: if you want the page gone from search, ask whether Google needs to read something on it to know that. If yes, it must be able to crawl it.
What about AI crawlers?
robots.txt is also where you allow or refuse AI crawlers, and the same logic applies. A Disallow rule is a request about fetching, not a guarantee about what an answer engine knows. We cover the practical setup in allowing ChatGPT and Perplexity in robots.txt.
How do you check that it worked?
Do not trust the tag alone. Check three things:
- The page's URL is not matched by any Disallow rule in your robots.txt.
- The noindex appears in the HTML source or the response headers of the live page.
- The page has dropped out of Google after a recrawl. Our post on checking if a page is indexed shows how.
If a page you wanted indexed is missing, the cause is often different. See crawled, currently not indexed for that case. You can browse the rest of this topic on the technical SEO hub.
Frequently Asked Questions
Can I use robots.txt and noindex together?
Not on the same URL. If robots.txt blocks the page, Google cannot fetch it and never sees the noindex. Allow crawling until the page has left the index.
Does noindex stop Google from crawling a page?
No. Google has to crawl the page to read the rule, and it may keep recrawling it for a while. If crawl load is the worry, that is a robots.txt job.
How long does noindex take to work?
It takes effect after Google recrawls the page. That can be days or longer for rarely visited URLs, so do not expect it to be instant.
Is a Disallow rule safe for hiding sensitive pages?
No. The robots.txt file is public, so it can point people at the very paths you want hidden. Use a login or password for anything sensitive.
Should I noindex thin pages or blocked tag archives?
Often yes, if the page is useful to visitors but not worth ranking. Leave it crawlable so Google can read the rule.