Robots.txt: What to Allow and What to Block

Robots.txt is the file I am asked about more than any other, and almost always for the same reason. Somebody has been told it will hide a page from Google, and they want to know where to put the line.

I have worked in search for 13 years, and that belief has cost more than one Australian business real traffic. The file does not hide anything from search results. It controls which paths a crawler is asked not to fetch, and a careless edit can take a working page out of view.

This is written for the owner or marketing manager who has found the file in the site root, cannot tell whether it is helping or hurting, and does not want to break the site while editing it. It covers what the file does, which paths genuinely belong in a rule, which must never be blocked, how allow and disallow behave in practice, and how to test a change before and after you publish it.

What a robots.txt file actually does

The file sits at the root of your site

robots.txt is a plain text file that lives at the top level of a domain, so it is reached at yourdomain.com/robots.txt. It has to sit at the root. A copy placed inside a subfolder will be passed over by crawlers, because the location is part of how the file is found.

Per Google Search Central, "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." That sentence is the whole job description. The file is a set of instructions to crawlers about which paths they may request.

The idea is older than most of the businesses using it. Per RFC 9309, "This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers."

Rules are a request to crawlers, not a lock on content

This is the part owners miss. A disallow line is a request a well-behaved crawler follows. It is not a wall. Per Google Search Central, "This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google."

The same point appears in the standard itself. Per RFC 9309, "These rules are not a form of access authorization." Anything you would not want a stranger to read should sit behind a login, not behind a text file that only courteous crawlers respect.

What to allow in your robots.txt

Starting from nothing, and why that is usually fine

An empty robots.txt file is a legitimate choice, and on plenty of small business sites it is the right one. Everything is crawlable by default, and a missing file returns a not-found response that crawlers treat as nothing being blocked, so the absence of the file is not a fault in itself.

Where owners get into trouble is adding rules without knowing what each one protects against. Every disallow line is a decision to stop a crawler reaching something, and if you cannot say what a line is protecting, it is doing nothing except carrying risk.

My advice to most people starting out is to allow everything, then block only the paths that waste crawl attention or expose content you do not want in search.

Allowing the resources Google needs to render a page

Google renders pages much the way a browser does, which means it fetches the stylesheets, the scripts and the images that build the page. If those files are blocked, Google sees a broken version of the page and judges it on that.

This is the most common self-inflicted problem I find. A developer blocks a content folder or a script folder to tidy up crawl activity, and the result is a page that no longer renders properly for search. Images disappear, the layout collapses, and the content that matters may not be read at all.

So the first rule of any robots.txt is this: never block the files a browser needs to display the page. If a path holds stylesheets, scripts or images that appear on your pages, it stays open.

What to block in your robots.txt

Internal search results and filtered URL paths

Internal search results are a fair thing to block. A URL for a query typed into your own site search has no search value of its own, and there is no limit to how many of those URLs exist. Crawlers can spend a lot of attention on them.

Filter and parameter URLs behave the same way. Sorting options, tracking parameters and filter combinations can multiply a handful of products into thousands of near-identical URLs. Blocking the parameter patterns keeps the crawl focused on the pages you want found.

The test is whether a URL would ever deserve to rank on its own. If the answer is no, it is a candidate for a block.

Admin, cart, checkout and account paths

A path that only makes sense to a logged-in user, or that sits behind a form, is also worth blocking. Admin areas, customer account pages, cart and checkout steps, and internal tooling should not be crawled.

Blocking them is good hygiene, not protection. A robots.txt line stops a crawler fetching the page in the ordinary course of a crawl, and it does nothing to stop a determined human. Treat the block as a way to keep crawl attention on public content, and keep the real protection on the server.

Media files and whether blocking them is worth it

Blocking images is a decision that needs a reason. For most sites it is a bad trade, because images are part of how a page renders and how it can surface in image search.

There are narrow cases where blocking a media path makes sense, such as a folder of unprocessed files or a directory of documents you never want indexed. Outside those cases, leave the media open and manage file size and naming instead.

What robots.txt cannot do

Keeping a page out of search results

This is the misunderstanding behind most of the conversations I have had about the file. A disallow rule asks crawlers not to fetch a URL, and it does not guarantee that the URL stays out of search results.

Per Google Search Central, "While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web." A URL can appear in results with no description, because the engine knows the address even though it never read the page.

The correct tool for each job

If a page genuinely must not appear in search, use the right mechanism. A noindex directive on the page tells search engines not to index it, and password protection or authentication stops access altogether.

There is a trap worth naming here: do not combine a disallow with a noindex and expect it to work. If the crawler is blocked from fetching the page, it never reads the noindex instruction. Choose one approach and be clear about what each one does.

Writing rules that behave predictably

Allow, disallow and which rule wins

A robots.txt file is built from user-agent lines and allow or disallow lines beneath them. A group that names one crawler applies to that crawler. A group that names all crawlers applies to anyone who follows the convention.

When two rules could apply to the same URL, the more specific one wins. Per Google Search Central, "When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path." A rule for a shop folder is less specific than a rule for its checkout path, and the longer path decides the outcome.

Wildcards, path matching and case sensitivity

Wildcards make rules more flexible and more dangerous at the same time. The asterisk character matches any run of characters, and the dollar character anchors a match to the end of a URL. A rule ending in a dollar sign matches that exact path only, while the same rule without it can catch everything that begins with it.

Paths are case sensitive, so a capitalised folder and a lower-case one are different paths to a crawler. On many servers they genuinely are different. Check that your rule matches the case of the real URL before you publish it.

How to test a robots.txt change

Testing before you publish

Test the rule before it goes live. Write the line down, name the URL you expect it to block, and check the pattern against that URL and against a URL it should not touch.

The two questions I ask of every rule are these. Does it block the path I mean to block, and does it leave everything else alone. If a rule cannot answer both clearly, it is too broad.

Why Google may not see the change for 24 hours

Publishing is not the same as taking effect. Per Google Search Central, "Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn't possible (for example, due to timeouts or 5xx errors)."

So the honest answer to how long until this applies is a day, sometimes more. Publish the change, then wait before you judge the result, and avoid editing the file again in the meantime.

Reading the errors and warnings in Search Console

The robots.txt report in Search Console shows you what Google made of the file for each host it crawls. It is the fastest way to catch a rule that does something other than what you intended.

The distinction the report draws matters. Per Google Search Console Help, "Errors prevent a rule from being used. Warnings do not prevent a rule from being used." An error means a line is not working at all, which is usually not what you want. A warning flags something to look at but leaves the rule in force.

The mistakes that cost Australian sites traffic

Disallowing the files Google needs to render

I keep meeting this one. A site loses its images in search, or a page reads as thin because half its content is loaded by script, and the cause turns out to be a robots.txt line a developer added years ago.

There is usually a well-meaning reason behind it, such as cutting crawl activity on a large site. The fix is to work out what the rule was meant to protect, then narrow it or remove it and confirm the page renders again.

Blocking an entire site with one line

The other classic is a staging rule that goes live. A single disallow line for all crawlers, left on a site that has moved from staging to production, tells every crawler to stay out. The site can fall out of search entirely, and the cause is one line nobody remembered.

This is why the file needs an owner and a review. A rule that made sense on a test server is a serious problem on a live one, and nothing in the interface warns you.

A five minute robots.txt routine

What to review each month

Read the file and ask what each line protects against. If you cannot answer for a line, that is the line to review. Check that every path you block is genuinely low value, and confirm the paths carrying your stylesheets, scripts and images are still open.

If the file has grown, or nobody can explain a rule, take a copy first and change one thing at a time. Then re-check the live file to be sure the change you intended is the change that published.

Who should own the file

One person should own it, ideally the technical contact who deploys the site, with a second person who knows the rules exist. The failures I see most often come from an old rule that nobody remembers adding and nobody dares remove.

If you would rather have a second set of eyes on it, the technical SEO checklist for crawl control sets out the wider ground, and an audit that includes your crawl rules will tell you what your own file is doing in practice. If your question is about controlling crawler access beyond robots.txt, that is a separate job, and there are common technical SEO questions answered here if you want the short version.

Frequently asked questions

Does a robots.txt file stop a page from appearing in Google?

No. Google's own documentation is blunt about this: the file manages crawling, not indexing, and a disallowed URL can still be indexed if other sites link to it. If you need a page out of the index, use a noindex directive or password protection.

Should I just leave my robots.txt empty?

An empty file is a legitimate and common choice. Everything is crawlable by default, and you have not accidentally blocked anything. Most small business sites only need rules for genuinely low-value paths, such as internal search results and filter URLs.

What should I never block?

For most sites, the stylesheets, scripts and image files that Google needs to render pages. If Google cannot fetch those, it sees a broken version of the page. Block admin and checkout paths instead, and leave the rendering assets alone.

How long until my change takes effect?

Google generally caches the file for up to 24 hours, and Google's own documentation says it may cache for longer when a fresh copy cannot be fetched. Test the change, publish it, then wait a day before judging the result.

How do I test a robots.txt change safely?

Test before publishing. Check that any path you are blocking is genuinely low value, confirm the rule pattern matches what you expect, then publish and re-read the live file. The robots.txt report in Search Console shows you whether any rules produced errors, and errors are the ones that actually stop a rule working.

Who should own the file?

One person, ideally the technical contact who deploys the site, with a second person aware of the rules. The failures I see most often come from an old rule that nobody remembers adding and nobody dares remove.