Crawling is when Google downloads a page with an automated program called a crawler. Indexing is when Google analyses that page and stores it so it can be shown in results. They are separate steps, and Google says it does not guarantee that any page will be crawled, indexed or served.
What are the three stages of Google Search?
Google's guide to how Search works describes three stages: crawling, indexing and serving search results. A page has to pass the first two before it can appear in the third.
| Stage | What happens | What can stop it |
|---|---|---|
| Crawling | Google finds a URL and downloads its text, images and video | A robots.txt block, a login wall, server errors, no links to the page |
| Indexing | Google analyses the content and stores it, after choosing a canonical | A noindex tag, duplicate or thin content, a canonical pointing elsewhere |
| Serving | Google matches a search to indexed pages and ranks them | Other pages answer the query better |
Most "my page is not showing up" problems live in one of these rows. Knowing which row tells you which fix to try.
What happens during crawling?
Google downloads text, images and videos from pages it has found, using crawlers. The best known is Googlebot. It also renders pages and runs JavaScript, much like a browser. It skips pages that robots.txt blocks or that need a login.
Google finds URLs by following links and by reading sitemaps. That makes internal links important. A page that nothing links to is called an orphan page, and it is easy to miss. A sitemap helps discovery, but it does not force indexing.
How much Google crawls is a separate question. For most small sites it is not a constraint, as explained in what crawl budget is.
What happens during indexing?
After crawling, Google analyses the page's content and key tags and stores the result in its index. Part of this step is grouping near-identical pages and picking a canonical, which Google describes as the page that may be shown in search results. If you have duplicates, the choice may not be the one you wanted. See how to set a canonical URL and how to handle duplicate content.
Google is direct that indexing is selective: "not every page that Google processes will be indexed." A crawled page can be left out. If you see that in Search Console, read crawled, currently not indexed.
What is the difference between robots.txt and noindex?
This is the most common mix-up. Robots.txt controls crawling: it tells crawlers which URLs not to fetch. A noindex rule controls indexing: it tells Google not to keep the page. Using the wrong one has real consequences. A page blocked in robots.txt cannot be crawled, so Google may never see a noindex tag placed on it. The full comparison is in robots.txt vs noindex.
What happens during serving?
When someone searches, Google looks in its index for matching pages and ranks them by relevance. Google says that depends on factors such as location, language and device. Two people can see different results for the same query. Being indexed is a requirement for ranking, but it does not mean you will rank.
How do you check where a page is stuck?
- Is it indexed? Use the URL Inspection tool in Search Console, or follow how to check if a page is indexed.
- If not, was it crawled? The inspection result shows whether Google has fetched it and when.
- If not crawled, check robots.txt, server errors, and whether any page links to it.
- If crawled but not indexed, check for a noindex tag, a different canonical, and whether the page adds anything an existing page does not.
- If indexed but invisible, the problem is relevance and competition, which is a content question rather than a technical one.
What can you do to help both steps?
You cannot make Google index a page, but you can remove the obstacles and give it reasons:
- Link to every important page from at least one other page that is already indexed, using descriptive link text. Internal linking is the cheapest discovery tool you have.
- Keep one URL per page. Redirect old versions, and avoid redirect chains that waste fetches.
- Return honest status codes. A missing page should return a real 404, not a 200 with an error message, which is the soft 404 problem.
- Publish pages worth keeping. Since indexing is selective, a page with a clear purpose and original content is more likely to be kept than a thin variation of another page.
- Watch Search Console. The page indexing report lists the reason each URL was left out, which saves guessing.
Why does this matter for AI search?
AI assistants that cite web pages also need to fetch those pages. OpenAI documents a separate search crawler, OAI-SearchBot, and Perplexity documents PerplexityBot. Each is controlled by its own robots.txt rules. The same discipline applies: if the crawler cannot reach the page, it cannot be considered. Our guide to allowing ChatGPT and Perplexity in robots.txt shows the rules.
Nightcrop's audit includes a check that AI crawlers can reach your site, which is one way to catch a block early.
Frequently Asked Questions
Is crawling the same as indexing?
No. Crawling is downloading a page. Indexing is analysing and storing it. A page can be crawled and still not be indexed.
Does Google index every page it crawls?
No. Google says not every page it processes will be indexed, and that it does not guarantee crawling, indexing or serving, even for pages that follow its guidelines.
How does Google find new pages?
Mainly through links from pages it already knows, and through sitemaps you submit. Internal links to new pages speed up discovery.
Does blocking a page in robots.txt remove it from Google?
Not reliably. Robots.txt stops crawling, not indexing, so use noindex on a crawlable page if you want it kept out of results.
Why would a crawled page not be indexed?
Common reasons are duplicate or thin content, a canonical pointing to another URL, or a noindex tag. Search Console's page indexing report shows the stated reason.