Before a page can rank, a search engine has to find it, read it and decide to store it. These steps, crawling and indexing, are the starting point of all technical SEO. If a page fails here, no amount of content or links will help.
Step 1: Discovery
Google learns about URLs in three main ways: following links from pages it already knows, reading XML sitemaps, and URLs submitted through Search Console. Links are by far the biggest source, which is why pages with no internal links often go unnoticed.
Step 2: Crawling
Googlebot requests the URL, much like a browser. Before it does, it checks the site’s robots.txt file to see whether the path is allowed. It then downloads the HTML and notes the status code. A 200 means it can continue; a 404, 410 or 5xx stops the process.
Google decides how often to crawl each site based on how fast the server responds and how important and fresh the pages seem. On big sites this limit, called crawl budget, becomes a real constraint.
Step 3: Rendering
Modern pages often build their content with JavaScript. Google runs that JavaScript in a headless version of Chrome to see the final page. Rendering takes extra resources, so it can be delayed. Content that only appears after scripts run is at risk, which is the focus of JavaScript SEO.
Step 4: Indexing
Google analyses the rendered page: text, headings, links, images, structured data and the canonical URL. It then decides whether the page is worth storing. Pages can be left out because they are:
- Marked with a noindex directive.
- Duplicates of another URL, with the other chosen as canonical.
- Thin or low quality compared with what is already indexed.
- Soft 404s: pages that say “not found” but return a 200 code.
Step 5: Ranking
Once indexed, the page competes for queries. Ranking is a separate process that weighs relevance, quality, links, intent and user experience. Being indexed is necessary but not sufficient.
Common indexing statuses in Search Console
Discovered – currently not indexed
Google knows the URL but has not crawled it yet. This often points to weak internal linking, low perceived value or crawl limits on large sites.
Crawled – currently not indexed
Google fetched the page but chose not to store it. Usually a quality or duplication issue. Improve the content, merge it with a similar page, or strengthen links to it.
Duplicate without user-selected canonical
Google found several similar URLs and picked one itself. Set a canonical tag to make your preference clear.
How to get pages indexed faster
- Link to new pages from relevant, already-indexed pages.
- Add them to your XML sitemap.
- Use URL Inspection in Google Search Console and request indexing.
- Make sure the page offers something not already covered elsewhere on your site.
- Keep the server fast and error-free.
Frequently asked questions
How long does indexing take?
Anywhere from hours to weeks. Well-linked pages on trusted sites are typically picked up within days.
Does Google index every page?
No. It is selective and skips pages it considers low value or duplicate.
Related terms
See robots.txt, noindex tag and the SEO encyclopedia.
Mr. SEO
Mr. SEO is the webmaster at SEO TT - SEO tips and tricks, a publication about all things SEO related.