Regenerable is not unused: triaging a full disk without taking the service down

The claim in one line

Regenerable describes what you could recreate later. It says nothing about what a running process is holding right now.

Those are two different tenses, and the triage step quietly swaps one for the other. "Safe to delete" sounds like a property you can read off the directory, in the present, by knowing what kind of thing it is. It is actually a prediction about the future: that between now and the next time anything needs this, it will have been rebuilt. A process can have a file open. It can have it memory-mapped. It can be executing out of it. It can be one import or one dlopen away from needing it, with the code path that does so not scheduled to run for another hour.

And here is the part that makes it dangerous rather than merely wrong: on Linux, deleting a file that a process is holding does not produce an error. The rm succeeds. The process carries on. You get no signal at the moment you could still undo it, and two problems instead: a service that is fine until the moment it is not, and — because the inode is still referenced — a disk that did not actually free the space you just freed.

In the triage pass that prompted this article, every candidate on the list was correctly classified. Each one really was purely regenerable; nobody made a mistake about what the files were. When each was instead checked against what was actually running, the majority of them were backing live processes. The classification was accurate and the conclusion drawn from it was still wrong. That gap is the whole subject here.

Everything below was measured on a live Linux system rather than reasoned from the manual pages, and three of the results are sharper or less convenient than the advice usually given — including one where the tool everybody recommends for this misses a case.

1. The ordering rule, first, because it is free

Almost every disk triage starts the same way, with a command that sorts by size:

$ du -x -h -d1 /var | sort -h | tail -20

That command is fine and you should keep running it. The mistake is what it gets used for. Sorting by size answers "what would give me the most room", and that answer gets promoted, silently, into "what should I delete". Those are different questions, and only the second one can take a service down.

Sorting by size picks the candidate. Checking the holders decides whether it is a target. Do them in that order and never merge them.

The reason to state this as an ordering rule rather than a reminder is that the pressure at ninety-five percent runs the other way. The size list is immediate, legible and satisfying; the holder check takes another thirty seconds and usually returns nothing interesting. It is exactly the step that gets skipped on the night it matters, which is also the night the machine is busy enough that something is mapping something unexpected.

One more thing worth knowing before you start, because it saves a wrong turn: if du and df disagree with each other — du says the filesystem holds far less than df says is used — you have already done this to yourself at some point, and section 4 is the section you want.

2. /proc/PID/exe: what is executing out of this path

The strongest form of "something is holding this" is that a process is running the binary. Every process on Linux has a symlink naming the executable it was started from:

# Which processes are executing out of a candidate directory?
$ for p in /proc/[0-9]*; do
    e=$(readlink "$p/exe" 2>/dev/null) || continue
    case "$e" in /opt/app/venv/*) echo "$p $(cat $p/comm) -> $e";; esac
  done

This is the check that catches the single most common version of the mistake: a virtual environment, a toolchain directory, an extracted release bundle. All of those look like build output, because they are build output. If a long-lived service was started from the interpreter inside one, that directory is not build output any more — it is the running system.

Two practical notes. The loop above is deliberately written with a || continue, because processes exit while you iterate and a failed readlink is normal, not an error. And reading another user's exe link needs privilege, so an unprivileged sweep reports a clean result for exactly the system processes you care most about. Run it as root, or know that you ran it as yourself.

Now the part that shows why this matters. We copied an interpreter into a scratch directory, started a long-running process from it, and then deleted the binary out from under it — the 2am move, in miniature. The process did not notice:

$ rm ./venv-python3          # succeeds, no error, no warning
$ kill -0 $PID && echo alive
alive
$ readlink /proc/$PID/exe
/path/to/venv-python3 (deleted)

It keeps running because the pages it needs are already mapped and the kernel holds a reference to the inode regardless of the name being gone. What it cannot do any more is anything that requires reading the file again — and, more to the point, it can never be restarted. The service survives the deletion and dies at its next restart, which might be a deploy, a log rotation, a config reload, or a reboot three weeks later. By then nothing connects the two events. That delay is the whole reason this class of mistake is expensive: the failure is separated from its cause by an arbitrary interval, chosen by whatever happens to restart the process first.

The (deleted) suffix on that symlink is also a diagnostic worth knowing in the other direction: if you find it on a process you did not just experiment with, somebody has already deleted a running binary, and that process is living on borrowed time.

3. /proc/PID/maps: what is memory-mapped from it

The exe link covers the binary the process was launched from. It does not cover the shared libraries it loaded, the extension modules it imported, or the data files it mapped — and in a modern application stack that is most of the bytes on disk. For those, the authority is the mapping table:

# Which processes have anything under this path mapped into memory?
$ grep -l /opt/app/venv /proc/[0-9]*/maps 2>/dev/null

# And for one process, what exactly, with the permission bits:
$ grep /opt/app/venv /proc/$PID/maps | awk '{print $1, $2, $6}'

Here is the measured correction, and it is a correction to the check most people write rather than to the tooling. We started a process that loaded a shared object from a candidate directory via dlopen, and then asked three different questions about the same file at the same instant:

rows naming the .so in /proc/PID/maps        5
rows naming the .so in ls -l /proc/PID/fd    0      <-- the blind one
rows naming the .so in lsof -p PID           1      (typed "mem")

The file descriptor table is the thing that is blind, not lsof. A library, once mapped, needs no descriptor; the loader opens it, maps it, and closes the descriptor immediately. So a check built on ls -l /proc/*/fd — which is the check a lot of people reach for first, because it is the one that is easy to read — returns a confident, clean, empty result for every shared library, every extension module and every executable on the machine. That is the category the article is about: a probe whose silence you cannot use.

The same run is a caution against generalising the other way. A data file that the process had mapped did appear in its descriptor table, because the runtime we used duplicates the descriptor when it creates a mapping and holds it for the life of the mapping. So the descriptor table is not reliably blind either — it is blind for libraries and executables, present for some mappings, and you cannot tell which from the output. That is a worse property than being simply wrong. Use maps, which answers the question directly for all of them.

One habit that makes the sweep trustworthy, and it takes one extra command. A grep over /proc/*/maps that returns nothing is indistinguishable from a grep you typed wrong, a permissions problem, or a path that does not exist. Run it once against something you know is held, watch it come back non-empty, then run the identical command against your candidate:

$ grep -l /usr/lib /proc/[0-9]*/maps 2>/dev/null | wc -l   # control: expect many
$ grep -l /opt/app/venv /proc/[0-9]*/maps 2>/dev/null | wc -l   # the question

A zero that you have not checked the instrument for is not a measurement. This costs one line and converts "nothing holds it" from an assumption into a result.

4. Deleted but still open: the space that does not come back

This is the single most useful thing in this article, and it is the one people meet as a mystery rather than as a known behaviour. You delete eight gigabytes. df does not move. Nothing is wrong with df.

On Unix, unlinking a file removes a name. The data goes away when the last reference goes away, and an open descriptor is a reference. A process holding a deleted file keeps its bytes allocated, with no path to reach them by. We measured it, with a 300 MiB file and one reader holding it open, taking the available-space reading at three points:

available, file present, reader holding it   10694364 KiB
available, after rm, reader still running   10694340 KiB   <-- nothing freed
available, after the reader exits           10999488 KiB   <-- ~298 MiB back

The middle line is the one to sit with. The file is gone from the directory. It does not appear in du, it does not appear in ls, and it is still occupying every byte it occupied before. You cannot find it by looking at the filesystem, because looking at the filesystem means looking at names, and the name is what you destroyed. This is also why the space "comes back on restart" — restarting the process drops the last reference, and it is the only thing that will.

The tool for finding these is lsof with a link-count filter — files whose on-disk link count has fallen below one while still being open:

$ sudo lsof +L1
COMMAND     PID   USER  FD  TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
python3   17263 appsvc   3r   REG    8,2 314572800     0 3825631 /var/cache/app/cache.bin (deleted)
                                                   ^^^^^
                                               no name left

Read it as: process 17263 has descriptor 3 open on a 300 MB file with zero links. Kill or restart that process and you get 300 MB. Sum the SIZE/OFF column and you have the exact gap between du and df.

And now the result that surprised us, which is why this section has a second half. lsof +L1 is necessary and it is not sufficient. In the experiment from section 3 we deleted two held files at once — the executable the process was running, and the shared object it had mapped. Then we ran the recommended command:

$ lsof +L1 | grep /opt/app/venv
venv-pyth 17826 appsvc txt   REG    8,2   8020928     0 3825595 /opt/app/venv/bin/python3 (deleted)
# ...and that is all. The deleted .so is NOT listed.

Both files were deleted. Both were still held by that one process. +L1 reported one of them. The reason is visible once you ask lsof to print link counts for everything that process holds:

$ lsof +L -p 17826
COMMAND    PID  FD  TYPE   SIZE/OFF NLINK    NODE NAME
venv-pyth 17826 txt  REG    8020928     0 3825595 /opt/app/venv/bin/python3 (deleted)
venv-pyth 17826 DEL  REG                  3825596 /opt/app/venv/lib/_mymod.so
                ^^^                 ^^^^^
            not a descriptor     no link count at all

The deleted executable is reported as a txt row with an explicit link count of zero, so a numeric filter selects it. The deleted library is reported as a DEL row with the link-count field empty — and a filter that asks "is the link count below one" cannot select a row that has no link count to compare. It is not a bug; lsof is telling you the truth in a different column. But it means the standard advice, run alone, under-reports. We re-ran it to be sure the first result was not a caching artefact; same output both times.

So the complete command for "everything deleted that is still being held" is two sweeps, not one:

$ { sudo lsof -nP +L1; sudo lsof -nP | awk '$4 == "DEL"'; } | sort -u

And the underlying source of truth, which needs no flags and no filtering rule to get right, is the mapping table again — every affected row carries the marker in the text:

$ grep '(deleted)' /proc/[0-9]*/maps 2>/dev/null
$ ls -l /proc/$PID/map_files/ | grep '(deleted)'

Generalisable beyond this command: +L1 is a filter over a numeric field, and a row where that field is absent silently fails the test rather than raising. Any time you narrow output with a comparison, ask what happens to a row that has nothing in that column.

5. atime, and why it is weaker than almost everybody assumes

Access time is the signal people reach for when the holder checks come back clean but they still want reassurance. "Nothing has read this in six weeks" feels like evidence. It is the weakest of the four checks in this article, and we are going to be specific about why rather than just recommending caution, because the naive version of this advice is worse than no advice.

$ stat -c '%x  %n' /opt/app/venv/lib/_mymod.so
$ find /opt/app/cache -type f -atime +30 -printf '%A+  %s  %p\n' | sort | head

# Check FIRST whether the answer can mean anything here:
$ findmnt -no OPTIONS -T /opt/app

That last command is the one to run before the other two, because the mount options decide whether the timestamps are answering your question at all. Three failure modes, in increasing order of how badly they mislead.

With noatime, there is no signal whatsoever. Access times are simply never updated. A file read ten thousand times an hour reports the timestamp it had when it was written. This is a common performance setting and it is often inherited from a template nobody has read recently, which means the person doing the triage may have no idea the number in front of them is fiction.

With relatime — the default nearly everywhere — the signal is coarse. The access time is only written when it is older than the modification time or more than a day stale, which exists precisely to avoid a disk write on every read. We measured it on a relatime filesystem: a backdated file read once had its access time updated; read again immediately, the timestamp did not move. So the number tells you a read happened within some window, and nothing about how many or how recently within it.

And the one that matters most here, because it lands exactly on the files this article is about. Reading through a memory mapping does not go through the read path that updates access time. We mapped an 8 MiB file, faulted in all 2048 of its pages one by one — continuous, heavy, unambiguous use — and watched the timestamp:

atime before mapping                      2026-09-21 04:35:24
atime after mmap + first page fault       2026-09-26 04:35:25
atime after faulting in all 2048 pages    2026-09-26 04:35:25   <-- unchanged
atime after the process exited            2026-09-26 04:35:25   <-- unchanged

control, same file via read(2) instead:
atime reset to                            2026-09-21 04:35:26
atime after cat                           2026-09-26 04:35:26   <-- moved

The access time dates the mapping, not the use. A service that mapped a cache or a model store when it started six weeks ago, and has been reading out of it every second since, presents a six-week-old access time. That is not a small inaccuracy — it is the failure pointing in the worst possible direction, because the file that is most heavily in use looks the most abandoned, and it is the mapped files that the holder checks in sections 2 and 3 exist to catch.

Use access time to rank candidates within a class you have already cleared. Never use it to clear one. A recent atime is weak evidence of use; an old atime is not evidence of disuse at all.

6. What to do when something is held

The checks above will tell you, some fraction of the time, that the biggest thing on the disk is being used by something you cannot stop right now. The honest answer to that is unwelcome, so it is worth stating plainly rather than leaving as an exercise:

If a process is holding it, the action is to schedule a restart — not to delete it and hope. Deleting a held file does not free the space and does break the next restart, which is the worst of both outcomes.

That is the asymmetry to keep in front of you. Deleting a held file costs you the restart path and gains you nothing on the disk, because the reference survives. There is no version of this where you come out ahead. A restart, by contrast, releases every deleted-but-open reference the process was holding and lets the artefact be rebuilt cleanly if it is still wanted.

So the ladder, roughly in the order you should want them:

• Reclaim the already-deleted first. The +L1 and DEL sweep is free space you have already paid for and have not collected. Restarting one process to collect it is a far smaller decision than deleting something new, because the artefact is gone either way.
• Delete what the checks cleared. Unheld, regenerable, and ideally also stale by access time. This is the set the naive triage thought it was operating on.
• Truncate rather than delete, where a writer is appending. For a log or a spool that a running process holds open for writing, : > /path/to/file frees the blocks immediately and leaves the writer's descriptor valid. Deleting the same file frees nothing until the writer exits, and the writer keeps appending at its old offset — which on many filesystems reinflates the file to its previous size as a sparse file the moment it writes again. Truncation is the correct move here and deletion is actively counterproductive.
• Schedule the restart for the held ones. With the actual amount it will return, measured from the sweep, so the decision is a trade and not a guess.
• Only then consider growing the filesystem. It is often the right answer. It is never the fast one, which is why it should be on the list before you need it rather than after.

One thing not to do under pressure: reach for a broad recursive prune command on a package manager or container runtime because it "only removes unused things". Those tools have their own definition of unused, it is generally about references within their own model of the world, and it does not know what your processes have mapped. It is the same inference this whole article is about, executed on your behalf, at speed. Run their dry-run flag and read the list, or bound the command to a directory you have checked.

7. The general shape, which is the part worth keeping

Strip the disk out of it and the error has a form you will recognise from somewhere else in your own systems:

A property of an artefact is not a property of the system's dependence on it.

"Regenerable" is a fact about the file. "Load-bearing" is a fact about the relationship between the file and everything currently running. The first is easy to establish, stable, and knowable from the artefact alone — which is exactly why it gets substituted for the second, which is none of those things and has to be measured at the moment you act.

The same substitution, in other clothes:

• "This config file is all defaults, so it is safe to remove." A fact about the contents. Whether something reads the file's existence, or fails differently when it is absent, is a fact about the consumers.
• "Nobody logged into this service this week, so it is unused." A fact about interactive sessions. Whether a nightly job still calls it is a fact about the callers.
• "This table is derived, so it can be rebuilt." True, and it says nothing about whether a query path has a timeout shorter than the rebuild.
• "This DNS record points at a host we decommissioned." A fact about the record. Whether something still resolves it and acts on the answer is a fact about the resolvers.
• "That dependency is dev-only." A fact about the manifest, and a claim about every import path at runtime that the manifest cannot make.

The tell is always the same, and it is a grammatical one: the sentence has a subject that is the artefact, and the risk lives with a subject that is somebody else. When you catch yourself justifying a removal with a description of the thing being removed, you have not answered the question yet — and the check that would answer it, as above, is usually thirty seconds long.