Our robots.txt 404'd for 20 months. It did not matter.

A defect, correctly identified

The finding was real and the number was large. Between January 2025 and 29 August 2026, requests for /robots.txt against our site returned 404 2,694 times. The file was restored on 29 August 2026, and from that point the 404 count for it is zero.

The natural next step is to write that up as a win: a long-standing SEO defect, found and closed, with a four-digit number attached to it. We nearly did. What stopped us was a question that is easy to skip once a fix is already deployed and the metric has already gone green.

Before you credit a repair, check that the broken thing was ever load-bearing.

We had the data to answer it — twenty-one contiguous months of our own server-side logs, January 2025 through early September 2026, with no missing months. So we answered it rather than assuming, and the answer was not the one we expected to publish.

1. A missing robots.txt is the permissive case

The reasoning should have come first, before any log file. robots.txt is a file whose job is to restrict. It is where a site says which paths a crawler should leave alone. It is not a permission slip, a registration, or an announcement that a site exists.

So a crawler that asks for it and receives a 404 has learned something unambiguous: there are no rules here. The documented, and observed, behaviour is to proceed and crawl everything it can reach. A missing robots.txt is the most permissive state a site can be in, not the least.

This is worth separating from the failure mode that genuinely does hurt. A robots.txt that returns 500 is a different signal entirely: the server has said "I cannot tell you what the rules are." Several major crawlers treat a persistent server error on that path as a reason to back off, because the safe reading of an unavailable rules file is that rules might exist. A 404 says the rules are absent. A 500 says the rules are unknown. Those are opposite instructions, and only one of them is expensive.

To be explicit, because this is the sentence most likely to be quoted out of context: we are not telling anyone to leave their robots.txt broken. Fix it. We are saying that the size of the 404 counter is not the size of the damage, and that the two are easy to confuse when the counter is the only thing being measured.

2. The measurement: they rose together

Reasoning about crawler behaviour is not evidence about our crawler behaviour. Here is the series. Every figure is a monthly aggregate from our own server logs; a representative selection of months is shown rather than all twenty-one, and the final row is a partial month.

The last row covers only the first days of September 2026, which is when the logs were read. It is not a decline.

Three things fall out of it.

Googlebot never went away. It is present in 20 of the 21 months, 634 hits in total, throughout the entire period in which the file it is supposed to ask for was returning 404. The single month with no Googlebot at all is January 2025 — the first month of the record. That is the edge of our data, not an effect of the defect.

Crawling grew by roughly twentyfold while the file was broken. From zero, and from a handful of hits a month through the first half of 2025, to 197 in August 2026 — the last full month of the fault. Whatever the 404 was doing, it was not suppressing crawl rate.

And the defect counter rose with it — because it is made of the same traffic. The 404s climbed from 43 a month to 335 a month over the same window, and that is not a worsening fault. Every one of those 404s is a crawler request. The file did not get more broken; we got crawled more, so the thing that was already broken was hit more often. Our headline defect metric was, to a large extent, a measurement of how much attention we were getting.

That is the trap in a single sentence: an error counter that scales with traffic will look like a deteriorating problem at exactly the moment things are going well. Before reading a rising error count as a growing fault, check whether its denominator is growing too.

3. What the repair was actually worth

Something real was lost, and it is worth being precise about what, because "nothing was lost" would be as sloppy as the claim we are correcting.

robots.txt has a second job besides restriction: it is the conventional place to advertise a sitemap, with a Sitemap: line. That line points crawlers at a machine-readable list of every URL a site wants indexed. If the file 404s, the pointer is unreachable no matter how good the sitemap behind it is.

So the real cost window is not the twenty months the file was missing. It is the period during which we had a sitemap directive to lose. Ours was written on 21 June 2026, and the file was restored on 29 August 2026. That is the actual exposure:

Roughly ten weeks, not twenty months. Before 21 June 2026 there was no directive to lose, and a 404 on a file we had never written cost us exactly what it contained.

Even that ten weeks is bounded rather than catastrophic. A sitemap is a hint, not a prerequisite: our pages were linked from our own navigation and were reachable by ordinary crawling, which the numbers above confirm was happening the whole time. The sitemap makes discovery faster and more complete. It is not the mechanism by which a site becomes crawlable.

The repair was worth making. It was worth about a tenth of what the headline number implied, and it was worth it for a reason — the sitemap pointer — that has nothing to do with the 2,694 in the ticket title.

4. The number that did not move

There is one more way to test whether the fix mattered, and it is the one that matters commercially. If a broken robots.txt had been holding back our visibility, then repairing it should eventually show up as people arriving from search.

Our logs record server-side arrivals carrying a search-engine referrer. Across all twenty-one months there are 93 of them. That is an average of about four a month, it is flat across the entire period, and it did not respond to the repair: four in August 2026, three in the partial month after.

We are going to be careful with that number rather than flattering ourselves with it, because it is exactly the sort of figure that gets quoted as traction:

• 93 hits is not 93 people. It is a count of requests carrying a referrer header, and on a site this quiet an unknown share of that is referrer spam. The honest statement of it is a ceiling, not an audience: the server-side ceiling on search arrivals to this site is about four a month, and it has never moved.
• Twenty of the twenty-one months record no search terms at all. One month has any, and the only term in it is our own company name — which tells us nothing we did not already know, and may well be us.

Put beside the crawl series, that is the finding underneath the finding. We were crawled thoroughly and increasingly for twenty-one months, and it produced no measurable search demand in either direction — before the fix or after it.

Being crawled and being found are two different measurements, and one is not evidence for the other. Crawl volume is a technical health signal. It is not a demand signal, and we had been half-treating it as one.

5. The instrument joined the population it was measuring

One methodological note, because it changed what we were willing to publish and it is the kind of thing normally left out.

The obvious series to report here is not Googlebot; it is total crawler traffic. It is a bigger number and it has a dramatic recent shape. We are not reporting it, and the reason is that a large share of it is us.

Log analysers classify traffic by user agent. We verify every deploy against production and we run continuous uptime and content checks against it, and those probes use an ordinary command-line HTTP client — which our analyser, entirely correctly, files as a robot. The better our monitoring got, the more of the "crawler" total became our own automation checking on itself. Reporting that as crawler interest would have been reporting our own uptime checks as demand.

So the crawler total is uninterpretable for our purposes, and we left it out entirely rather than presenting it with a caveat. The Googlebot series is clean: it is a third party, it is one we do not control, and it cannot be contaminated by anything we run.

If you monitor your own site, your monitoring is in your own traffic statistics. Before reading any traffic number as interest, work out how much of it is you — and if you cannot separate it, say so and report a series you can.

6. How to run this check on your own site

None of the above needed a tool we bought. It needed server logs and six questions, and it is worth running before you credit any infrastructure repair with an outcome.

• Ask what the broken thing was for. A file that restricts behaves nothing like a file that enables when it goes missing. Getting this backwards is how a permissive failure gets written up as an outage.
• Find a series that spans the whole fault, not just the fix. A before-and-after around the deploy date measures a week. A defect that ran for twenty months needs twenty months of data to say anything about, and that data has a retention policy you probably have not checked.
• Check whether your defect metric is a function of the thing you want to measure. Ours was: the error count rose because traffic rose. An error rate would have told a truer story than an error count.
• Separate the populations. Crawlers and human arrivals are different measurements answering different questions. Do not let a healthy one stand in for a flat one.
• Exclude yourself. Your deploy checks, your monitors and your own browsing are in the log too.
• Know the edges of your record. Our one Googlebot-free month was the first month we have data for. Read as a finding, that is a fabricated effect; read as an edge, it is nothing at all. The difference is knowing where your record starts.