Robots.txt and noindex

Two different mechanisms control what a search engine does with a page. They are confused with each other constantly, and the confusion has a specific consequence.

Robots.txt is a public text file at the root of a site that asks crawlers not to fetch particular addresses. It is about fetching.

A noindex instruction sits on the page itself and asks search engines not to include that page in their results. It is about storing and displaying.

The consequence of the confusion is this: a crawler has to fetch a page to read the noindex instruction on it. Block the address in robots.txt and the instruction is never read. Using both on the same page is the one combination that reliably does not work. It is the most common way a page that “should not be in Google” stays there.

What each one does

	robots.txt	noindex
Lives at	The root of the site, one file	Each individual page
Controls	Whether crawlers fetch the address	Whether the page is stored and shown
Visible to the public	Yes, always	Only in the page source
Prevents indexing	No, not reliably	Yes
Prevents fetching	Yes, for compliant crawlers	No
Right use	Keeping crawlers out of pointless address spaces	Keeping a specific page out of results

What robots.txt is genuinely for: crawl management, not an indexing tool

Its legitimate job is crawl management: keeping crawlers out of address spaces that produce endless variations and no value. Search result pages inside the site, filter and sorting parameters, print views, and internal endpoints are the usual candidates.

That is a real benefit on a large site with a catalogue. On a fifteen-page business website it does very little, which is why the file’s contents matter far less than whether it accidentally blocks something.

The meta tag and the X-Robots-Tag header

A noindex instruction has to be delivered somehow, and there are two ways to deliver it. Most pages carry it as a meta tag sitting in the HTML head: a meta element whose name is robots and whose content reads noindex. A crawler that can read the page’s head reads this meta tag exactly the way it reads any other meta tag. The second delivery method is the X-Robots-Tag, sent as an HTTP response header rather than a tag inside the markup. It is the only option for content with no HTML head to hold a meta tag at all, such as a PDF, an image, or a bare API response. This is also why the robots.txt file cannot do this job on its own. Robots.txt only governs whether a crawler fetches an address. The meta tag and the X-Robots-Tag header both govern whether the noindex instruction is ever seen once the fetch has already happened.

The two Disallow failures worth knowing

Blocking the entire site. A single line disallowing everything is the standard content of the file while a site is in development. Shipping it to production takes the whole site out of search results, and nothing on the site looks wrong. It is invisible from the front end, it produces no error, and it is usually discovered weeks later. This is the highest-consequence single line in a launch.

Blocking assets. Disallowing the folders that hold stylesheets and scripts prevents a search engine from rendering the page as a person would see it. The page is then assessed on a version of itself that does not exist. This was a common pattern in older sites and still appears in inherited configurations.

Why the robots.txt file is not privacy

The file is public. Anyone can read the robots.txt of any website, and it lists exactly the addresses the owner would prefer were not visited. As a way of hiding something it is closer to a signpost than a lock.

Anything that genuinely must not be seen needs authentication: a login, a password, or the file not being on a public web server at all. This matters in practice because “we put it in robots.txt” is offered as an answer to a privacy question more often than it should be. Where personal information is involved, the obligations arise under the Privacy Act rather than under any crawler convention, and a robots.txt entry does nothing to satisfy them.

When noindex is the right tool

• Thank-you and confirmation pages, which are only meaningful after a form submission
• Internal search result pages, where they are reachable at all
• Duplicate or thin pages that exist for a functional reason
• Pages published for a specific audience that arrive by direct link
• A staging site, where it belongs alongside authentication rather than instead of it

Removing a noindex is how a page is later allowed into results, and that removal is a task with an owner and a date. Pages left excluded after they were supposed to be released are the mirror image of the staging failure, covered on staging and production.

Compliance is voluntary, and crawlers decide whether to follow it

Both mechanisms are conventions that well-behaved crawlers follow. They are not enforcement. Crawlers that ignore them exist. The number of automated agents fetching web pages has grown with the arrival of assistant products, some of which publish their own identifiers and some of which do not.

The practical consequence for a small business is unchanged: neither file protects anything. What they do is manage the behaviour of the crawlers that matter commercially.

What to check on your own site: crawling versus indexing

Type your domain followed by /robots.txt into a browser. You will see the file. Two things to look for: a line disallowing everything, and any line blocking a folder that holds stylesheets or scripts.

Then check a page you expect to be in search results by searching for a distinctive sentence from it in quotes. If it does not appear, the causes in order of likelihood are a leftover noindex, a canonical pointing elsewhere, or the page never having been discovered — which is crawling and indexing.

What to do next

Add both files to the launch checklist as separate items with separate owners, because they fail independently. Ask whoever maintains the site how these are handled between staging and production. If the answer is that someone remembers to change it, that is the risk. Ongoing responsibility for checks like this is part of what a maintenance arrangement covers, which is on what website maintenance actually is.