Robots.txt Guide: 7 Essential Rules Every Site Owner Should Know
By Istiak Ahmed Soikot, Founder and CEO, Rank OG. Istiak has spent 4+ years helping small businesses rank on Google and AI search. Last reviewed: November 2026.
This robots.txt SEO guide explains how one small text file tells search crawlers which pages they may visit. You will learn seven practical rules for placing the file, writing allow and disallow lines, avoiding common mistakes, and testing changes before they hurt your rankings.
This robots.txt SEO guide is written for site owners, not developers. If you run a WordPress site, a Shopify store, or a small business website and have never opened your robots.txt file, this is your starting point. You will leave knowing what to check and change.
What is a robots.txt file, and what does it control?
A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which URLs they may fetch. It manages crawl traffic, not security, and it does not remove pages from search results.
The idea dates back to 1994, when Martijn Koster proposed the Robots Exclusion Protocol, published by the IETF as RFC 9309 in September 2022. Google’s crawlers download and parse the file before crawling a site, then follow its rules.
The default is open. Unless you write a rule that says otherwise, crawlers may fetch every page on your site. Most small sites work fine with a short file, or even no file at all, yet one wrong rule can do more harm than a missing one.
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.
Google Search Central, Robots.txt Introduction and Guide
Rule 1: Put the file in the right place, or it does nothing
Your robots.txt file must live at the root of your site, at an address like https://www.example.com/robots.txt. A file in a subfolder is never read, so its rules never apply.
One file covers one host, one protocol, and one port. Rules in https://example.com/robots.txt do not cover https://shop.example.com/ or http://example.com/, so each subdomain needs its own file. Save it as plain UTF-8 text in a basic text editor. Word processors can add hidden characters that break rules. Google may ignore characters outside the UTF-8 range, which can silently invalidate a rule. If the file sits in the wrong place, nothing else in this robots.txt SEO guide matters, so check the location first.
Rule 2: Never use Disallow to hide a page from Google
Disallow stops crawling, not indexing. Google can still show a disallowed URL in search results, usually without a description, if other sites link to it. To keep a page out of Google, use noindex or password protection.
This is the most common misunderstanding in this robots.txt SEO guide, and the one that causes the most damage. Google states it plainly: robots.txt is not a mechanism for keeping a web page out of Google. A blocked page can keep its URL in the index as a bare listing. The fix is simple: let Google crawl the page, add a noindex meta tag or response header, and Google will drop it after the next crawl. For anything truly private, a password is the only safe answer, because robots.txt rules are public.
| Method | What it stops | Can the URL still appear in results? | Best use |
|---|---|---|---|
| Robots.txt Disallow | Crawling of matching URLs | Yes, often without a description | Saving crawl budget, keeping media out of search |
| Meta robots noindex | Indexing, after Google recrawls | No, after recrawl | Removing a page from results |
| Password protection | All access and indexing | No | Private or internal content |
The verdict in this robots.txt SEO guide is simple. Use robots.txt to manage what crawlers fetch, use noindex to manage what appears in search results, and use a password for anything that must stay private. Mixing up the three is how pages leak into Google or vanish from it by accident.
Rule 3: Learn how Allow and Disallow settle a conflict
When two rules match the same URL, the longer, more specific rule wins. If both are equally specific, Google applies the less restrictive one, so Allow beats Disallow in a tie.
Disallow: / blocks the whole site, but Allow: /public/ opens that one folder, because the allow rule matches more characters. Wildcards give finer control: an asterisk matches any run of characters, and a dollar sign marks the end of a URL, so Disallow: /*.pdf$ blocks PDF files and nothing else. One warning: paths are prefix matches, so Disallow: /fish also blocks /fishheads, so add the trailing slash when you mean a folder. This longest-match rule is the detail most summaries of this robots.txt SEO guide skip, and it explains why a file can look right and still behave wrong. For the bigger picture, see our guide on crawling vs indexing.
Rule 4: Treat every path as case-sensitive
Rule paths in robots.txt are case-sensitive. Disallow: /admin/ does not block /Admin/, and Disallow: /file.asp does not block /FILE.asp. Field names are not case-sensitive, but paths are.
This trips up sites that moved from Windows hosting to Linux hosting, where paths do not ignore case. Check your real URLs before you write a rule. If your login page lives at /Admin/ with a capital A, a lowercase rule does nothing. That is why this robots.txt SEO guide recommends testing the exact URL, which brings us to rule 7.
Rule 5: Name the right crawler, and use the wildcard with care
Each group of rules starts with a User-agent line naming the crawler it applies to. Google follows the most specific matching group, and a group for one crawler is never combined with the catch-all group.
User-agent: * matches all crawlers except the AdsBot crawlers, which must be named explicitly. If you write a group for Googlebot and a group for *, Googlebot follows only its own group. Google supports only four fields: user-agent, allow, disallow, and sitemap. Crawl-delay is ignored, so robots.txt cannot slow a crawler down, even though many robots.txt SEO guide examples still list it as if Google honored it. AI crawlers such as GPTBot have their own user-agent names too, so check each vendor’s documentation first.
Rule 6: Keep the file short, clean, and under the size limit
Google ignores everything past the first 500 KiB of a robots.txt file. Keep yours short, write one rule per line, and add a comment with a hash sign when a rule needs it.
A small business site rarely needs more than a dozen lines. List your sitemap with its full URL, as in Sitemap: https://www.rank-og.com/sitemap_index.xml, so crawlers find it without guessing. As this robots.txt SEO guide notes, Google caches the file for up to 24 hours, so a fix is never instant. If the file itself fails, a 4xx error is treated as if no file exists, so crawling continues, while a 5xx error makes Google pause and reuse the last good copy for up to 30 days.
Rule 7: Test every change before you trust it
Never publish a robots.txt change blind. Check that the file loads in a private browser window, then confirm in Search Console that Google can fetch it and the right URLs are allowed.
Search Console has a robots.txt report that shows whether Google fetched your file, and the URL Inspection tool flags any URL as Blocked by robots.txt. Google also publishes the open source parser its crawlers use, so developers can test a file locally. The mistake this robots.txt SEO guide sees most often is a single line, Disallow: /, left in place after a site launch or redesign. It blocks the entire site. Picture a store owner who launches on a Friday, finds no traffic on Monday, and then learns the staging rules went live too. One test before launch prevents that week of silence.
What does a good robots.txt file look like?
Most small sites need a file like this one: it allows all crawlers, keeps the WordPress admin area and internal search pages out of the crawl, and points to the sitemap. This robots.txt SEO guide keeps the example short, because every extra line is another chance for a mistake.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /?s=
Sitemap: https://www.rank-og.com/sitemap_index.xmlEvery line earns its place. The admin area has no value in search results, internal search pages create endless near-duplicate URLs, and the sitemap line helps crawlers find new content faster. If your site has nothing to hide, a shorter file is fine, so add rules only for real problems you can name. If crawl issues keep coming back, a technical SEO review can find what a single file cannot fix, and our on-page SEO checklist for small business covers the page-level fixes that pair with it.
How to use this robots.txt SEO guide on your own site
Follow this robots.txt SEO guide step by step. The whole job takes under an hour on most small sites.
- Open yoursite.com/robots.txt in a private window and read what is there.
- List the folders crawlers should skip, such as admin areas, internal search, and checkout pages.
- Rewrite the file with the rules above, one rule per line, and keep a backup of the old version.
- Test the new file in Search Console and check your five most important pages are allowed.
- Recheck after any redesign, migration, or staging launch, when bad rules slip in.
Frequently asked questions
Does robots.txt keep a page out of Google search results?
No. Disallow stops Googlebot from crawling a page, but Google can still list the URL if other sites link to it, and the result shows without a description. To remove a page, let Google crawl it and add a noindex tag, or password protect the page instead.
Where do I put my robots.txt file?
It belongs at the root of your site, at yoursite.com/robots.txt, and nowhere else. A file in a subfolder is never read. Each subdomain needs its own file, and the rules apply only to the same host, protocol, and port where the file lives.
Can I block every crawler with one line?
Yes. Write User-agent: * on one line and Disallow: / on the next, and every crawler that respects the file will stay out. Do this only on staging sites or private areas, never on a live site, because it also blocks Google and removes your pages from search over time.
What happens if my robots.txt file has a mistake?
A single wrong line can block your whole site or leave private pages open to crawlers. Google caches the file for up to 24 hours, so a fix is not instant. Test changes in Search Console first, keep a backup, and recheck after every redesign or migration.
Do small sites need a robots.txt file?
Not always. A small site with no private areas works fine without one, because crawlers may visit every page by default. Add a file when you have admin pages, internal search results, or staging copies to keep out, or to point crawlers to your sitemap.
How do I test my robots.txt rules?
Open yoursite.com/robots.txt in a private browser window to confirm the file loads. Then use the robots.txt report in Search Console to check Google can fetch it, and the URL Inspection tool to test single pages. Google also publishes its open source parser for developers who want to test files locally.
Sources
- Robots.txt Introduction and Guide, Google Search Central
- Create and Submit a robots.txt File, Google Search Central
- How Google Interprets the robots.txt Specification, Google Search Central
- RFC 9309: Robots Exclusion Protocol, IETF, September 2022
Sources checked November 2026.
Not sure your robots.txt file is doing its job? Use this robots.txt SEO guide as your checklist, then request a free SEO audit from Rank OG and we will check your crawl rules, your sitemap, and the technical issues holding rankings back. You get a clear list of fixes, not a sales pitch.
