Verified is a claim about one machine at one moment

The claim in one line

"Verified" is a claim about one machine at one moment. It is not a property the system carries around afterwards.

A check that passed is not a check that is passing. The gap between those two tenses is where this whole class of outage lives, and it is unusually hard to see because nothing in the record is false. The command ran. The evidence is real. The conclusion drawn from it — that the system is in the verified state — was true at the instant of measurement and quietly expired at some point afterwards that nobody logged.

The generic shape of the near-miss worth building this around: a release is deployed and verified, correctly, with real evidence. Underneath it, an infrastructure change — a rollback of a virtual machine, a container rebuilt from a stale image, a failover to a replica that never received the release, a restore from a snapshot — silently reverts the served content to an earlier state. The gap runs for over a day before anyone notices. The verification was never wrong. It was answered about a target that had been swapped out from under it.

This is the same error one level up from a property of an artefact not being a property of the system's dependence on it. There, a fact about a file was substituted for a fact about the running system. Here, a fact about a measurement is substituted for a fact about the system now. Both substitutions are invisible in the moment, both produce a confident green, and both fail at an arbitrary later time chosen by something other than you.

Everything below was measured rather than reasoned about. The transcripts come from real runs against an origin built deliberately to be dishonest — up, answering, and serving the previous release — because the only way to know what a check certifies is to point it at a machine you have already lied to and see whether it notices. Hostnames, paths, release identifiers and timestamps in the examples are illustrative.

1. The three words every "verified" leaves out

Start by writing the claim out in full. A verification result carries three unstated bindings, and naming them is most of the work:

• Where. This host. This container. This replica, out of however many are behind the load balancer. This edge location, out of the fifty the network has. A check run from one place has measured one place.
• When. This moment, and no moment after it. The result has no expiry stamped on it, so it behaves as though it has none.
• What. These bytes, at this URL, through this path. Not "the deployment". Not "the service". The specific thing the command actually touched.

So the sentence "the deploy is verified" expands to something considerably less reassuring: at 14:32 on Tuesday, one request from one office IP to one URL returned bytes matching the release, on whichever backend that request happened to land on. Every word of that is worth having. None of it is a statement about the system today.

Write the where, the when and the what into the result itself. A verification that does not record its own scope will be read as though its scope were everything.

That is not a documentation exercise. Each of the three has a specific engineering answer, and the rest of this article is those three answers: bind the what by giving the artefact an identity you can assert from outside; bind the when by reconciling continuously instead of once; bind the where by stating the set you covered and refusing to let one sample stand for it.

2. Reachability is not identity

The most common check in existence is "is it up", and it answers a question nobody actually has. Here is an origin serving the previous release, with nothing at all wrong with it as far as any availability check can tell:

$ curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/pricing.html
200

$ curl -s https://www.example.com/pricing.html | tail -1
<p>Rates effective 2029-01-01: 11 percent.</p>

$ git show HEAD:pricing.html | tail -1
<p>Rates effective 2031-01-01: 19 percent.</p>

A 200 proves something is answering. It never proves it is your build. Those are two different questions and they deserve two different answers, because in this failure mode the first one is green throughout — that is precisely what makes the day-long gap possible.

The fix is to give the running artefact an identity. Stamp a release marker into everything you ship, at build time, and assert it from outside:

# in the artefact, written by the build, committed like any other byte
<meta name="release" content="2026.09.26-4f1ab0c">

# and, cheaper to assert, as a response header the app emits
X-Release: 2026.09.26-4f1ab0c

Then "is it up" and "is it the bytes I shipped" become two commands with two answers. Measured against the same origin, both pages returning 200:

$ EXPECT=2026.09.26-4f1ab0c
$ for u in /index.html /pricing.html; do
    got=$(curl -sSI "https://www.example.com$u" | tr -d '\r' \
          | awk -F': ' 'tolower($1)=="x-release"{print $2}')
    [ "$got" = "$EXPECT" ] && echo "ok   $u $got" || echo "STALE $u $got"
  done

ok    /index.html    2026.09.26-4f1ab0c
STALE /pricing.html  2026.08.14-9d3e77b

That is a HEAD request — no body, no parsing — and it separated a current page from a six-week-old one that every availability check on earth reports as healthy.

The marker trap: whose freshness is it reporting?

Here is the measured correction, and it is the one to take away from this section, because the common implementation of a build marker is vacuous for exactly the failure it was built to catch. Put the marker in a shared artefact — one build-info endpoint, one shared header partial, one bundled script every page pulls in — and the marker reports the freshness of the shared thing, not of the page you fetched:

$ curl -s https://www.example.com/build
b2f41c7a9d0e5581                       <-- current. the deploy "is verified".

$ curl -s https://www.example.com/shared.js
window.SITE_BUILD="b2f41c7a9d0e5581";  <-- also current, genuinely

$ curl -s https://www.example.com/pricing.html | tail -1
<p>Rates effective 2029-01-01: 11 percent.</p>   <-- the previous release

Marker fresh, page stale, and no disagreement anywhere in the system for a check to find. Every component is internally consistent. The marker answered honestly; it was just answering about a different object than the one anybody cared about.

A build marker certifies the artefact that carries it, and nothing else. If one marker covers many artefacts, it has told you about one of them and you do not get to choose which.

So: per-artefact marker, or per-artefact hash. Both work; the marker is cheaper to assert and the hash is stronger, because a marker can be stamped onto stale content by a careless build step while a hash of the bytes cannot. If you only have the budget for one, stamp the marker and hash the things that would hurt.

And do not follow redirects while verifying

-L is in everybody's fingers, and inside a verification it converts a finding into a false pass. The URL moved; the check followed it; the check then validated whatever it landed on and filed the result under the name of the URL it asked for:

$ curl -sL -o /dev/null \
    -w 'asked %{url} got %{http_code} from %{url_effective}\n' \
    https://www.example.com/docs/install.html
asked .../docs/install.html  got 200  from https://www.example.com/index.html

$ curl -sL https://www.example.com/docs/install.html | sha256sum | cut -c1-16
e38825b2ee143870
$ curl -s  https://www.example.com/index.html        | sha256sum | cut -c1-16
e38825b2ee143870        <-- identical. it verified the homepage and called it install.html

Without -L, the same request returns 301 and the redirect is the finding — it means your content map and the server disagree about where that page lives. Drop -L from every verifying fetch and treat any 3xx on a content URL as a failure. Pin the hostname you actually publish, too: if the bare domain redirects to the www form, a check written against the bare domain is either following a redirect or failing on one, and both readings are noise.

3. Reconcile continuously, against the source of truth

Everything above still runs at deploy time, and deploy time is not when this fails. The check that catches a substrate swap is not a better deploy check — it is a drift check: a periodic re-assertion that what is live equals what version control says should be live. Same command, different schedule, completely different question. The deploy check asks "did this deploy work". The drift check asks "is the answer from last Tuesday still true".

Here is the shape. It is short on purpose, and every branch in it exists because the version without that branch passed something it should have failed:

#!/bin/sh
# Re-assert that live equals the committed revision. Exit non-zero on any
# disagreement, and on any inability to answer the question.
set -u
BASE="https://www.example.com"
REV="${1:-HEAD}"
EMPTY=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
fail=0; n=0

for path in $(git ls-files -- 'public/*'); do
  n=$((n + 1))
  rel=${path#public/}

  want=$(git show "$REV:$path" | sha256sum | cut -d' ' -f1)
  if [ "$want" = "$EMPTY" ]; then
    echo "FAIL  $rel  committed side is empty: wrong path or wrong revision"
    fail=1; continue
  fi

  code=$(curl -sS --max-time 10 -o body.tmp -w '%{http_code}' "$BASE/$rel") || {
    echo "FAIL  $rel  fetch did not complete"; fail=1; continue; }

  if [ "$code" != "200" ]; then
    echo "FAIL  $rel  HTTP $code"          # a redirect is a finding, not a detour
    fail=1; continue
  fi

  got=$(sha256sum < body.tmp | cut -d' ' -f1)
  if [ "$got" != "$want" ]; then
    echo "FAIL  $rel  live $(echo "$got" | cut -c1-12) != $REV $(echo "$want" | cut -c1-12)"
    fail=1; continue
  fi
  echo "ok    $rel"
done

echo "checked $n of $n tracked paths at $(date -u +%Y-%m-%dT%H:%M:%SZ)"
exit $fail

Run that on a schedule, alert on non-zero, and the day-long gap becomes an hour-long one. Four details in it are load-bearing and worth spelling out, because each one is a way the naive version passes.

Enumerate the set from version control, not from the disk

The loop is driven by git ls-files, not by find. On a working copy that has been worked in — which is every working copy — those are different sets:

$ find . -type f | sed 's|^\./||' | sort        # what a walk sees
assets/app.css
.git-ish/tmp.html
index.html
index.html.orig
notes-to-self.html
pricing.html

$ git ls-files | sort                           # what is actually released
assets/app.css
index.html
pricing.html

A walk enrols editor backups, scratch copies and things nobody committed — files that can never be served, so the check either fails permanently on them or grows exemptions until it means nothing. Version control is the only enumeration that agrees with what the release actually contains. The residual seam is safe in the useful direction: an uncommitted file is uncovered and unshippable, and committing it is what enrols it in the check — so the check gains coverage at the same moment the file gains the ability to hurt you.

Reconcile both directions, or you cannot see what was added

A loop over the committed side is structurally incapable of finding something on the live side that is not in the loop. Measured, with a leftover file sitting on the server:

git-side loop: for each committed file, is live equal?
  assets/app.css    present and equal
  index.html        present and equal
  pricing.html      present and equal
  -> every committed file accounted for. Verdict: PASS

the live side, actually enumerated:
  assets/app.css  index.html  pricing.html  _debug.html
                                            ^^^^^^^^^^^ never in the loop

"Live is missing something we shipped" and "live has something we did not ship" are different faults with different causes, and only the first is visible to a one-way loop. If you can enumerate the live side at all — a file listing, an object-store listing, a sitemap, a route table — do the set comparison in both directions. Where you genuinely cannot enumerate it, say so in the output, because an unenumerable side is an uncovered side and the report should not read as though it were complete.

A comparison of nothing to nothing succeeds

This is the one that makes a broken drift check indistinguishable from a healthy system, and it is worth running yourself once so the shape sticks. Both sides of the comparison can fail to produce any bytes, and a hash comparison of two empty streams is a match:

$ printf '' | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

$ curl -s http://127.0.0.1:1/unreachable | sha256sum        # nothing came back
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
$ echo "exit: $?"
exit: 0                                                     <-- sha256sum's exit, not curl's

$ git show HEAD:index.html 2>/dev/null | sha256sum          # wrong prefix, no such path
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
$ echo "exit: $?"
exit: 0

VERDICT: live matches committed -- OK

Two separate failures, cancelling into a pass. And note the second half, which is the part people do not expect: in a pipeline, $? is the exit status of the last command, so both of those report success. The producer failed, the consumer succeeded, and the shell reports the consumer. set -e does not save you either, for the same reason.

Memorise e3b0c442…7852b855 as "I compared nothing", assert against it explicitly as the loop above does, and where the shell supports it add set -o pipefail. Then prove the guard fires — point it at a path you know does not exist and watch it refuse:

$ want=$(git show "HEAD:no/such/file" 2>/dev/null | sha256sum | cut -d' ' -f1)
$ [ "$want" = "$EMPTY" ] && echo "guard fires: yes" || echo "guard fires: NO"
guard fires: yes

A check you have only ever seen pass is a check you have not tested. Break it on purpose, once, and keep the broken case as a test.

Fail loudly, and distinguish "disagreed" from "could not tell"

Three outcomes, not two: agreement, disagreement, and the check did not complete. A run that could not reach the host, could not read the revision, or skipped half the set must not print the same word as a clean run. Give it its own exit code and its own vocabulary, so nobody can paste an incomplete run into a ticket as sign-off. The failure that started this article is precisely a system that looked fine because nothing said otherwise; a check that goes quiet when it cannot answer is another instance of the same thing.

4. Partial verification is the trap inside the trap

A check that samples one URL certifies one URL. This sounds too obvious to state until you notice how many smoke tests fetch the homepage and are described, in the runbook, as verifying the deployment. Measured on the same origin, two URLs, one release:

/index.html      MATCH
/pricing.html    DRIFT  (live e3ce99abb7c4, want e38825b2ee14)

A smoke test that stops after the first URL prints the first line and exits zero. It is not lying — it verified the homepage, and the homepage was fine.

State the deployable set and reconcile all of it — or say out loud, in the output, how much you verified.

The cheapest possible version of this discipline is one line, and it is the line the loop in section 3 ends with: checked 41 of 41 tracked paths. Printing the denominator does three things at once. It tells a reader the scope without them having to read the script. It turns a shrinking numerator into a visible event rather than a silent one. And it makes the set itself reviewable — the first time somebody reads "checked 38 of 41" they will ask which three, which is a question nobody thinks to ask of a check that just says PASS.

Two sampling dimensions besides the URL list are worth the same treatment, because the where from section 1 hides in both. If several replicas serve the same URL, one request reaches one of them, and repeating the request does not fix it — you need a way to address each backend individually, or you have verified "at least one of them is right". And if a cache sits between you and the origin, a fetch that gets a cache hit has measured the cache. Bust it with a value that changes on every attempt rather than one that changes per release: key the buster on the release and the first probe of each release caches its own answer, which every later probe of that release then replays back to you as a hit.

5. A green result with no age is not evidence

Once the check runs continuously, a second failure mode opens underneath it, and it is the one that most deserves the name: a monitor that stopped running looks exactly like a monitor reporting success. Both are silent. Both leave a last-known-good result sitting where the dashboard reads it.

Measured, and the point is that there is nothing to see:

# the result file, four minutes after a successful run
{"check":"live-matches-committed","status":"pass","files":41,"drift":0}
verdict: pass

# the same file, after the scheduler has been dead for nine days
{"check":"live-matches-committed","status":"pass","files":41,"drift":0}
verdict: pass

Byte-identical. The payload cannot distinguish the two cases because the information that distinguishes them is not in the payload:

$ echo "age: $(( $(date +%s) - $(stat -c %Y last-result.json) ))s"
age: 777600s

$ find . -maxdepth 1 -name last-result.json -mmin +60 -printf 'STALE: %p\n'
STALE: ./last-result.json

Writing a timestamp into the payload is better and is not sufficient on its own, which is the bit that surprises people. A dead producer leaves the old timestamp in place, and it parses perfectly. The field only becomes a guard when the consumer compares it to its own clock and treats age as a failure condition:

{"status":"pass","checked_at":"2026-03-04T08:45:21Z"}

age 777601 s (just over nine days), budget 3600 s -> STALE: treat as FAIL

Assert the age of the last check, not just its verdict. A verdict older than its budget is a failure, not a pass — and the budget belongs to whoever reads the result, not to whoever produces it.

The degenerate case is worth closing explicitly too, because the obvious implementation gets it backwards. Ask what your consumer does when the result artefact is absent:

verdict: (no result file)

A dashboard that renders that as an empty panel, a rule that only alerts on status == "fail", an inbox rule that only notices mail the checker sends — all three read silence as health. The general form is the dead-man switch, and it inverts the polarity of the thing you watch: alert on the absence of a fresh success, not on the presence of a failure. That way a check that is dead, a check that is misconfigured, a check whose host has been decommissioned, and a check that is genuinely failing all produce a signal, which is correct, because from the outside they are the same event.

Worth turning on your own alerting once: has this check ever fired? A rule that has had nothing to say has not been proven able to speak. Send a deliberate failure through the whole path — producer, transport, rule, destination — and confirm it arrives somewhere a person looks.

6. When the substrate moves, every prior verification is void

The honest operational answer when a change of substrate is coming is unglamorous and entirely mechanical:

Treat any host, platform or substrate change as invalidating every prior verification by default. Re-verify after the substrate settles, not before, and not during.

Default, because the alternative is case-by-case reasoning about which of your green checks survived a migration, performed under time pressure by whoever is on call, with the wrong answer failing silently. It is far cheaper to re-run everything than to be right about what needed re-running.

What counts as a substrate change is wider than "we moved hosts", and the list is worth having written down before you need it:

• A migration between hosts, regions, accounts or providers.
• A restore from backup or a rollback to a snapshot — this is the one that reverts content while leaving everything reachable, and it is the classic cause of the day-long gap.
• A failover to a standby, or a standby being promoted and never demoted.
• A container or image rebuild, an autoscaling event that starts instances from an older image, a base-image update.
• A platform-side change you did not initiate: a runtime version bump, a caching layer added or reconfigured, a control-panel setting altered.
• Any period during which the host was unreachable. If you could not have verified during a window, you did not — and anything closed during it should be treated as unverified until a check says otherwise.

"After it settles" carries real weight. Verifying during a migration measures a system in a transient state, and a green from that window is worse than no green at all, because it is a result you will later cite. Wait for the thing to stop moving — DNS propagated, caches warm, the last instance replaced — and then re-run the whole set rather than the part that changed.

Two operational habits make this cheap enough to actually do. Keep the verification runnable as a single command with no manual steps, so "re-verify everything" is not a project. And make the timing of the invalidation explicit in the record: note, on the ticket, that verification from before a given change is void. Otherwise the old evidence stays in the thread looking exactly as convincing as it did the day it was true — which is, once again, the entire subject of this article.

7. The general shape, which is the part worth keeping

Take the deployment out of it and what is left is a statement about checks in general:

Every green check has a scope, and outages live exactly where the scope ended.

Not where the check failed — a failing check is a good day, because somebody looks at it. Trouble accumulates in the region a check never covered, because that region is indistinguishable, from the outside, from a region that was covered and was fine. The green tells you nothing about which of the two you are standing in.

The same substitution wearing other clothes:

• "The test suite passed." On the runner's architecture, with the runner's dependency versions, against fixtures. A claim about that environment at that commit.
• "The backup job succeeded." A claim that something was written. Whether it can be read back, and into a system that can use it, is a different check that most people have never run.
• "We load-tested it." On staging, with staging's data volumes and staging's cache hit rate. Every one of those is part of the result and none of them is in the number.
• "The failover was tested." Once, on a Thursday, at a config that has since changed on the primary and not the secondary.
• "That page is indexed." It was, when the crawler last came. Indexing is a state maintained by somebody else's system and it decays without telling you.
• "The dependency audit came back clean." At that commit, against the advisories published by that date. Both halves of that move afterwards without anybody touching the code.

The habit that pays, and it costs one question: of any check you own, ask what it precisely certifies, and what it silently leaves uncertified. Write both down next to the check. The second half is the one nobody writes, and the second half is the map of where your next incident is.

The reason this is worth insisting on rather than filing as pedantry is the delay. A verification does not become wrong at the moment the system changes; it becomes wrong quietly, and it stays in the record reading as evidence. When the failure finally surfaces, the most recent thing anybody can find on the subject is a green result, which is exactly the artefact most likely to send the investigation somewhere else.