Robots.txt: What It Is and How to Use It for SEO

by
Robots.txt: What It Is and How to Use It for SEO – SEOTT guide

Robots.txt is a plain text file at the root of your domain (for example example.com/robots.txt) that tells crawlers which parts of the site they may request. It is one of the first files Googlebot reads and a basic tool of technical SEO.

What robots.txt does and does not do

It controls crawling, not indexing. If you block a URL in robots.txt, Google will not fetch it, but it can still index the bare URL if other pages link to it. To keep a page out of search results, use a noindex tag instead and let Google crawl the page so it can see that tag.

Basic syntax

A file is made up of groups. Each group names a user agent and lists rules:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml
  • User-agent names the crawler. An asterisk means all crawlers.
  • Disallow blocks a path and everything under it.
  • Allow makes an exception inside a blocked path.
  • Sitemap points to your XML sitemap.

Google follows the most specific matching rule. You can also use * as a wildcard and $ to mark the end of a URL, for example Disallow: /*.pdf$.

What should you block?

Block areas that waste crawl time and offer no search value:

  • Internal site search results (/?s= or /search/).
  • Endless filter and sort combinations on shop category pages.
  • Cart, checkout and account pages.
  • Staging or test folders (better still, password-protect them).

On large sites this frees up crawl budget for pages that matter.

What should you never block?

Do not block CSS or JavaScript files. Google needs them to render pages and judge mobile friendliness. Do not block pages that carry a canonical or noindex you want Google to see.

AI crawlers

Many AI companies use their own user agents. Common ones include GPTBot and OAI-SearchBot (OpenAI), Google-Extended (Gemini training), PerplexityBot, ClaudeBot and CCBot. Blocking them keeps your content out of training data or AI search, but may also reduce citations. Decide based on your goals; our generative engine optimisation guide covers the trade-off.

Common mistakes

  1. Leaving Disallow: / live after a site launch, which blocks everything.
  2. Using robots.txt to hide private content. The file is public and lists your secret folders.
  3. Blocking a page and adding noindex to it. Google cannot see the noindex.
  4. Wrong capitalisation: paths are case-sensitive.

How to test robots.txt

Search Console’s robots.txt report shows the version Google has fetched and any errors. Google’s own introduction to robots.txt lists every supported rule. Crawlers like Screaming Frog can test rules before you publish them.

Frequently asked questions

Do I need a robots.txt file?

Not strictly. Without one, crawlers assume they can fetch everything. Most sites benefit from a short file with a sitemap line.

How quickly do changes take effect?

Google usually caches the file for up to 24 hours.

Related terms

See how search engines crawl and index and the SEO encyclopedia.

Mr. SEO is the webmaster at SEO TT - SEO tips and tricks, a publication about all things SEO related.