Your timeout killed the process, not the process group

The claim in one line

A timeout that kills a process does not kill what that process started.

This is not a bug in anybody's language or tooling. Signals are delivered to a process, or to a process group, and almost every convenient timeout helper is written in terms of the first one. If the thing you are timing out is a shell script, a wrapper, a test runner, a build step, or anything else that spawns real work and waits for it, then the process you kill is the one doing the waiting — and the real work carries on with no supervisor at all.

The same hole appears in at least four places people meet it: a Python subprocess call with a timeout=; a shell script that backgrounds a load and exits; timeout(1) used with one particular flag; and a CI step that reports "finished" while a grandchild keeps burning a runner. Everything below was measured on a live Linux system rather than reasoned from the manual pages, and two of the results contradict advice we have given ourselves in the past.

1. Why the grandchild outlives the timeout

Four numbers hang off every process on a Unix system, and only the first one is famous. There is its own process id; the id of its parent; the id of its process group; and the id of its session. Groups and sessions exist precisely so that a signal can reach a whole job at once — that is what lets Ctrl-C at a terminal stop a pipeline rather than one member of it.

Picture the ordinary case. Your supervisor process starts a shell script; the shell script starts the real work and waits for it. Three processes: supervisor, script, worker. The script and the worker both inherit the supervisor's process group, because that is the default, so all three sit in one group. The worker's parent is the script. The script's parent is the supervisor.

Now the timeout fires. The convenient helper does the obvious thing: it sends a signal to the one process it is holding a handle on — the script. Two consequences follow immediately, and they are separate.

The worker is not signalled at all. Nothing has told it anything. It keeps running, and because its parent has just died, it gets adopted by something else and keeps running as somebody else's child. It is holding the same CPU, the same open files and the same locks it held a second ago. The only thing that changed is that nothing is waiting for it and nothing knows its name.

The script's own cleanup never runs. The default in most timeout helpers is SIGKILL, which cannot be caught, blocked or handled. A shell script killed that way does not run its trap … EXIT. Whatever that trap was going to do — remove a temporary directory, release a lock, write a "this run ended" marker — does not happen, ever. We measured this: with the naive timeout, the script's temporary working directory was still on disk afterwards. So the damage is not only the CPU you can see. It is also a slow disk leak that surfaces weeks later, somewhere else, with nothing connecting it back.

The unit of work is the process group, and the unit most timeout code operates on is the process. Every failure in this article is that mismatch.

2. The fix in Python: wrong and right, side by side

Here is the version almost everybody writes. It is compact, it is in the standard library's own examples, and it is wrong for any command that starts children.

# WRONG for anything that spawns work of its own.
import subprocess

try:
    subprocess.run(["./job.sh"], timeout=600)
except subprocess.TimeoutExpired:
    log("job timed out")          # job.sh is dead. Its children are not.

On timeout, subprocess.run sends SIGKILL to the direct child and raises. We ran exactly this against a script that backgrounds a long-lived worker: the script died, the worker was still alive afterwards, and the script's EXIT trap had not run. The except block ran, the log line was written, and the log line was false.

The right version is a few lines longer and does three things the short one does not: it puts the job in a process group of its own, it signals the whole group, and it asks politely before insisting.

import os, signal, subprocess

GRACE = 10      # seconds between "please stop" and "stop"

# start_new_session=True calls setsid() in the child before exec, so the
# child leads a brand-new process group: its pgid IS its pid.
proc = subprocess.Popen(["./job.sh"], start_new_session=True)

try:
    proc.communicate(timeout=600)
except subprocess.TimeoutExpired:
    _signal_group(proc, signal.SIGTERM)     # the whole group, catchable
    try:
        proc.communicate(timeout=GRACE)     # let its EXIT trap run
    except subprocess.TimeoutExpired:
        _signal_group(proc, signal.SIGKILL) # then insist
        proc.communicate()

def _signal_group(proc, sig):
    try:
        os.killpg(proc.pid, sig)   # pgid == pid, because of setsid()
    except ProcessLookupError:
        pass                       # it exited between the check and the kill

Four details in there are worth more than the shape.

• os.killpg(proc.pid, …), not os.killpg(os.getpgid(proc.pid), …). The second form is what most examples show, and it has a race: if the child has already exited, getpgid raises and your cleanup path dies on the way to cleaning up. Because start_new_session guarantees the group id equals the child's pid, you never need the lookup. We verified that equality rather than trusting it.
• SIGTERM first, then SIGKILL. This is the half that recovers the lost cleanup. In our measurement the group-wide SIGTERM killed the worker and the script ran its EXIT trap, so the temporary directory was gone afterwards. Skip straight to SIGKILL and you are back to leaking.
• You have added two new exit paths, so you owe two new tests. The graceful path and the escalation path are different code, and a live run that completes normally exercises neither. A timeout handler is the one piece of your program guaranteed to run only on your worst day.
• There is a real cost to start_new_session. The child leaves your controlling terminal, so Ctrl-C at the keyboard no longer reaches it — an interactive interrupt goes to the terminal's foreground group, and the child is deliberately not in it any more. Your own timeout handler has become the only thing in the world that will ever stop that job. That is the right trade for a scheduled job and the wrong one for a foreground tool a person is babysitting.

3. The fix in shell, and the flag that undoes it

Here the measurement corrected us, so we will give the correction first: plain timeout(1) is better than its reputation. We expected it to leak grandchildren and it did not.

timeout 600 ./job.sh                # worker DIED, script's EXIT trap ran
timeout --foreground 600 ./job.sh    # worker SURVIVED the timeout

GNU timeout normally runs the command in a process group of its own and signals that whole group — the same trick as the Python fix above, done for you. With --foreground it does not create the group, and then only the named command is signalled. Its own documentation says so, in the words "children of COMMAND will not be timed out", which is easy to read past.

That matters more than a footnote, because of why people reach for that flag. You add --foreground when the command needs to read from the terminal, or when Ctrl-C stopped working, or when something in a wrapper script started behaving oddly under automation. In other words, you add it to fix an unrelated annoyance, and the thing you silently give up is the only reason the tool was protecting you.

The second flag is --kill-after, and it closes a different hole. The default signal is SIGTERM, which a process may catch, block, ignore, or be unable to act on at all while it sits in uninterruptible I/O. Without an escalation, timeout can return while the job is still there:

timeout --kill-after=30s 600 ./job.sh   # TERM at 600s, KILL 30s later

Now the other side: a script that starts background work itself. A bare trap is not enough — the trap has to actually kill something. We ran three variants, signalling the script and then checking whether its background worker was still alive:

trap 'true' EXIT INT TERM                  # worker SURVIVED  (control)
trap 'kill 0' EXIT INT TERM                # worker died
trap 'kill $(jobs -p) 2>/dev/null' EXIT INT TERM   # worker died, clean status

The first line is the control, and it is the point of the experiment: the trap fired, and the worker lived anyway. A trap that tidies up files while leaving processes running is the commonest version of this bug in shell, and it looks exactly like a script that cleans up after itself.

Between the two that work, we prefer the third. kill 0 means "signal every process in my own process group", which includes the script running the trap — so the script signals itself while handling a signal, and in our run its exit status came back as a crash rather than the conventional one. The jobs -p form killed the worker and left the script reporting a normal signalled exit. There is also a sharper edge on kill 0: "my own process group" is only the script's own group when the script is a group leader. Sourced into another shell, or run inline as part of a larger command, that group is the caller's, and kill 0 will take the caller down with it.

One more honest note, because we have previously told ourselves the opposite. Naming INT and TERM alongside EXIT is still the right habit — but in this measurement an EXIT-only trap did fire when the script was sent SIGTERM. What differed was the exit status the script reported. Name the signals because it makes the intent explicit and keeps the status conventional, not because the trap otherwise never runs; if you need the stronger claim for your own shell and version, measure it there. SIGKILL is the case with no argument: no trap of any spelling runs.

A grandchild that calls setsid itself leaves the group and no wrapper can hold it. Group-wide signalling is the right default, not a guarantee.

4. How to see it

One command shows the whole shape, because it prints the two ids everybody omits:

$ ps -eo pid,pgid,ppid,etime,%cpu,stat,comm --sort=-etime

This is what the orphan from our own measurement looked like, with the pids shortened for readability. The script that started it had pid 8623 and was already gone:

    PID    PGID    PPID   ELAPSED %CPU STAT COMMAND
   8624    8368    1528     00:03  0.0 S    worker
                   ^^^^
                   not 8623 any more — and not 1 either

Three tells, in decreasing order of how much they can be trusted.

• ELAPSED exceeds the timeout that supposedly killed the job. This is the strongest signal and it needs no bookkeeping at all. If a job is bounded at ten minutes and something it started has been up for forty, the bound is not being enforced on the thing doing the work. Sorting by elapsed time puts the answer at the top of the list.
• The parent is no longer the process that started it. The worker above was started by 8623 and its parent is now 1528. Reparenting is the real tell — but read the next paragraph before you write it into a check.
• CPU is being spent on something nothing is waiting for. Cross-reference with what is supposed to be running now. An orphan is, by construction, not in anybody's job list.

Now the correction, and it is the one thing in this article most likely to save someone a bad afternoon. "The parent is now pid 1" is the classic tell and it will miss. On the system we measured, the orphan's parent became 1528 — a user-level service manager — while pid 1 was the system's init. Anywhere a process has registered as a child subreaper, reparenting stops at the nearest such ancestor rather than travelling all the way to init, and that describes systemd user sessions, most container init processes, and many process supervisors. A monitoring check written as ppid == 1 reports a clean system while the orphan runs.

The group id disappoints in the same way if you lean on it. It is tempting to look for a process whose PGID names a leader that has died. In our measurement the orphan's PGID was 8368, and that group's leader was alive and well — it was the supervising process's own group, inherited, because nobody had ever created a group for the job. That is the same root cause as the leak itself: if you had made a group for the job, you would have been able to kill it.

Eyeball with ps; but if you are writing a check, compare against the pid and process-group id you recorded when you started the job. That is the only version that does not depend on the init topology of the machine it runs on.

5. A monitor that can starve its own next run

Everything so far is a language detail. This is the part that makes it an operations problem, and it is the reason we thought the story was worth publishing.

Return to the shape at the top. A scheduled job times out and is killed. The work it started keeps running, at full CPU, on more than one core, for tens of minutes. Meanwhile the schedule does not care: the next run starts on time, onto a machine that is now busy with the remains of the last one, takes longer than it otherwise would, and hits its own timeout. The alert that reaches a human says the job failed — and a job whose purpose is to watch something will phrase that as a fault in the thing being watched.

A monitoring system that can starve its own next run reports its own damage as the thing it monitors.

This is worse than a misleading message, because of what it does to the pattern. Each failure, taken alone, looks transient: it ran long, it timed out, the next one will probably be fine. Closing it as transient is locally reasonable every single time. But the second failure was caused by the remains of the first, and treating them as independent events is exactly what hides the causal link. A run of "transient" timeouts closed one at a time is the signature of this bug, and the closures are what destroy the evidence for it.

Which gives an operational rule that costs nothing: when a watch reports a fault, look at the load before you look at the watch. One uptime, one ps sorted by elapsed time. If a previous run of the alerting job is at the top of that list, you have your answer and it is not the answer the alert gave you.

Two design consequences follow, and they generalise past monitoring to any recurring job — a backup, a sync, a nightly import, a CI pipeline on a shared runner.

• A job's ceiling must be shorter than its interval, and enforced on the group. Those are two requirements and only the first one gets written down. A ten-minute bound on a thirty-minute schedule — an illustrative pair, not a recommendation — is only a bound if the thing doing the work is inside it. Otherwise the schedule is the only limit, and the schedule is not a limit.
• Choose the ceiling from a measured runtime, and record the derivation rather than the multiple. "About four times the usual run" is a number that decays quietly as the work grows, and nobody re-derives it, because the sentence still reads as if it were reasoned. Write down what you measured and when, so the next person can see whether the premise still holds.

There is a third, which we have written about in another context and which applies here unchanged: silence is the failure mode of every scheduled job. A job that dies in a way that leaves work running does not report anything at all about the work it left; the next run reports something misleading instead. Something has to be watching for the absence of a clean finish, not just for the presence of an error.

6. What to change, in order

• Find every place you time something out. Search your code for timeout as a keyword argument, for timeout(1) in scripts and unit files, and for job-level time limits in your CI configuration. For each, ask one question: does the thing being timed out start other processes? If it is a shell script, a wrapper, or a test runner, the answer is yes.
• Give the job its own process group, and signal the group. start_new_session=True plus os.killpg in Python; plain timeout without --foreground in shell. Treat an existing --foreground as a finding and go and read why it was added.
• Ask politely, then insist. SIGTERM, a named grace period, then SIGKILL — --kill-after if you are using the tool. This is what lets the job's own cleanup run, which is the difference between a timeout and a slow disk leak.
• Make the trap kill processes, not just files. If a script backgrounds work, its trap needs kill $(jobs -p) or the equivalent named-pid form. A trap that only removes a temporary directory passes every review and leaks every time.
• Test the timeout path, both branches. The graceful stop and the escalation are separate code, and no successful run touches either. Assert on the survivor: start a job, trigger the timeout, and check that the pid you recorded is gone — not merely that your function returned the value you expected.
• Detect by recorded pid, never by ppid == 1. Write down the pid and process-group id when you start the job; compare against those. A check that assumes orphans land on init passes cleanly on every machine with a subreaper, which today is most of them.
• When a scheduled job fails twice, stop calling it transient. Look at the machine before the job. The second failure is the one that carries the information, and closing it individually is how the pattern stays invisible.