6.9.11: systemd hangs in cgroup_drain_dying during cleanup after podman operations

Hello,

Our cockpit tests found a kernel regression introduced between 6.9.10 (working) and 6.9.11 (broken) that causes a system hang during cgroup cleanup after podman container operations. I've kept notes in
https://github.com/cockpit-project/bots/pull/8970#issuecomment-4342147158 , but now I am at the end of my wisdom how to squeeze more information out of this.

=== Summary ===

When running podman REST API operations on rootless containers followed by user session cleanup (loginctl/pkill), systemd (pid 1) gets stuck in cgroup_drain_dying trying to remove an empty cgroup. After that, I'm

- Unable to run commands that access /proc (ps, top, lsns, ls /proc, etc.)
- Unable to create new SSH sessions or VT logins
- If I previously logged into the QEMU VT, that login session remains more or less functional, except not being able to run most commands

=== Kernel Versions ===

- Last known working: 6.9.10
- Broken: 6.9.11 (OpenSUSE Tumbleweed), 6.9.13 (Fedora 44), 6.9.14 (Fedora 44), Ubuntu 26.04 (7.0.0)

=== Stack Trace ===

From sysrq-trigger task dump, systemd is stuck in:

[  207.958946] task:systemd         state:D stack:0     pid:1     tgid:1     ppid:0
[  207.959734] Call Trace:
[  207.960117]  <TASK>
[  207.960333]  __schedule+0x2b2/0x5d0
[  207.960603]  schedule+0x27/0x80
[  207.960945]  cgroup_drain_dying+0xef/0x1a0
[  207.961287]  ? __pfx_autoremove_wake_function+0x10/0x10
[  207.961639]  cgroup_rmdir+0x37/0x100
[  207.961945]  kernfs_iop_rmdir+0x6a/0xd0
[  207.962239]  vfs_rmdir+0x154/0x270
[  207.962486]  do_rmdir+0x201/0x280
[  207.962723]  __x64_sys_unlinkat+0x8c/0xd0

=== Observations ===

- /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.procs was empty, indicating all processes were killed but the cgroup itself cannot be removed
- Multiple zombie processes present, unable to be reaped ([email protected] systemd, podman, conmon processes)
- RCU subsystem appears healthy (rcu_exp_gp_kthr in S state)

=== Reproducer ===

The bug is triggered by a specific sequence of podman REST API operations on rootless containers, followed by user cleanup. The reproducer is part of the cockpit-podman test suite. I created a branch where I reduced the test to the absolute minimum, and also replaced as many UI clicks as possible with shell operations (all but one):

https://github.com/martinpitt/cockpit-podman/blob/kernel-hang/test/check-application#L1486

Sequence:
1. Create and stop a rootless container as the admin user
2. Call podman REST API lifecycle operations: start → restart → stop
3. Create an exec session (console/TTY connection) via REST API
4. Start the container again via REST API
5. Cleanup: loginctl terminate-user admin; loginctl kill-user admin; pkill -9 -u admin

Using podman CLI commands (e.g., "podman start swamped-crate") instead of the REST API does NOT trigger the hang, only when using the REST API. That may be because of the different process layout, or just sheer timing -- as eventually, both CLI and API should result in the same actual cgroup/container operations on the podman side.

The bug is very timing-sensitive. I attempted to create a standalone shell script reproducer, but failed, it always passes with that. Even with the original cockpit-podman integration test failure it's unreliable: it can hang on the first iteration, most of the time it fails within 5 runs, but I've had stretches where 50+ iterations passed before the hang happened.

=== Full debug output ===

The above GitHub PR comment links to the full dmesg log. Direct link:
https://github.com/user-attachments/files/27195205/dmesg-cgrouphang.txt

This covers initial boot up to the hang, and then the outputs of sysrq task dump (t), memory info (m), and blocked tasks (w).

=== Additional Notes ===

In one early test run, a different hang pattern was observed where rcu_exp_gp_kthr was in D state with a process stuck in synchronize_rcu_expedited during namespace cleanup, but this variant has not reproduced in subsequent runs. The cgroup cleanup deadlock appears to be the primary manifestation.

This is my first (non-trivial) kernel bug report, so please bear with me. I'm normally stay firmly in userland.

Thanks,

Martin Pitt

Hello,

Thanks for the report. The dmesg you attached has only a partial sysrq-t
- the dying-task stacks I need were pushed out of the ring buffer. Could you increase log_buf_len, reproduce, trigger sysrq-t, and send the resulting dmesg?

Thanks.
--
tejun

Hello,

I think I have the mechanism. The deadlock chains three things together.

1. Host PID 1 systemd is doing rmdir on user-1001.slice. rmdir enters cgroup_drain_dying() which waits until nr_populated_csets drops to
0. The wait was added in 1b164b876c36 ("cgroup: Wait for dying tasks to leave on rmdir") so that rmdir doesn't succeed while dying tasks are still on the cgroup's css_set - the controller invariant being kept is "no tasks running in an offlined css" (see d245698d727a for where that got established on the dying-task side).

2. nr_populated_csets only drops when cgroup_task_dead() runs, which happens from finish_task_switch() after the dying task's do_task_dead() - i.e., after the very last context switch out of the task.

3. The container's PID 1 (whatever the entrypoint runs) is in do_exit() but parked in zap_pid_ns_processes' second wait loop:

       for (;;) {
           set_current_state(TASK_INTERRUPTIBLE);
           if (pid_ns->pid_allocated == init_pids)
               break;
           schedule();
       }

pid_allocated stays > init_pids because at least one struct pid in the namespace is still in the idr - i.e., its task hasn't been reaped (release_task -> free_pid hasn't run).

The unreaped task is the host parent of one of the container helpers
- in the podman case, an exec session child whose host parent died during the pkill cascade (conmon, the user manager, etc.) and got re-parented to host PID 1. PID 1 was supposed to wait4() it as part of normal reaping, but PID 1 is blocked in (1). Chicken-and-egg.

Why fuse-overlayfs masks it: with fuse-overlayfs the container teardown finishes fast enough that the container's PID 1 reaches do_task_dead before host PID 1 has had time to become the reaper of the orphan. With kernel-overlayfs the teardown drags long enough that the reparent + drain wait land in the iterator-hidden window simultaneously, and once they do, the wedge is permanent.

I have a minimal C reproducer (~150 lines, no podman, no daemons, runs as root) that hits the exact same code paths. The shape: A unshares a pid namespace and forks B (PID 1 of the namespace) and C (PID 2; A is C's host parent, NOT B). Both run in an inner cgroup. A SIGKILLs both, then calls rmdir() without wait4()-ing C. C becomes a zombie pinning pid_allocated; B parks in zap_pid_ns_processes; A's rmdir parks in cgroup_drain_dying. Verified on stock 6.19.14 and mainline 7.1-rc1+, also under virtme-ng. Source inlined at the bottom of this mail.

For the fix, I'm still thinking. The rough direction is to split cgroup rmdir: the user-visible side (directory unlink, cgroup.procs disappears, the cgroup looks gone to userspace) completes as soon as the iterator goes empty, while the backend teardown (kill_css / css_offline) gets deferred until nr_populated_csets actually drops to
0. That keeps the "no running tasks in an offlined css" invariant intact while letting the rmdir syscall return so host PID 1 can resume reaping. Need to look at what shapes this cleanly without breaking other cgroup core
expectations.

Thanks for the report.

--
tejun

----- min-repro.c -----

/*
* Minimal reproducer for cgroup_drain_dying / zap_pid_ns_processes
* deadlock. No podman, no daemons.
*
* A — host process. Forks B (PID 1 of new pidns) and C (PID 2 of

 *       same pidns, but with A as host-side parent). After unshare,
 *       A's nsproxy->pid_ns_for_children is the level-1 namespace,
 *       so each subsequent fork lands a new task there.
 *   B — A's first child after unshare. PID 1 of new pidns.
 *   C — A's second child. PID 2 of same pidns, but parented to A
 *       at the host level (not to B). When C dies, SIGCHLD goes to
 *       A, NOT to B's zap_pid_ns_processes loop.
 *
 * Sequence:
 *   1. A puts itself in inner cgroup; B and C inherit on fork.
 *   2. A unshares CLONE_NEWPID, forks B and C.
 *   3. A moves out of inner so inner can be rmdir'd later.
 *   4. A SIGKILLs B and C. Both are killed.
 *   5. B's exit invokes zap_pid_ns_processes. zap walks idr, sees C
 *      (already SIGKILL'd), but C is NOT B's child — kernel_wait4(-1)
 *      returns -ECHILD immediately. zap reaches second loop:
 *      `pid_allocated > 1` because C's struct pid is still in idr
 *      (C is zombie awaiting A's wait4). zap sleeps.
 *   6. A calls rmdir(inner) WITHOUT calling wait4. cgroup_drain_dying
 *      sees nr_populated_csets > 0 (B still on cset->tasks; C may or
 *      may not be), iterator empty (B has PF_EXITING && live==0).
 *      Drain sleeps waiting for cgroup_task_dead, which only fires
 *      after B reaches do_task_dead, which it can't until zap returns,
 *      which it can't until pid_allocated drops to 1, which won't
 *      happen until A reaps C — but A is stuck in rmdir.

*
* Run as root.
*/

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <sched.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>

static void die(const char *msg)

{
	perror(msg);
	exit(1);
}

static void write_str(const char *path, const char *s)

{
	int fd = open(path, O_WRONLY);
	if (fd < 0)
		die(path);
	if (write(fd, s, strlen(s)) < 0)
		die(path);
	close(fd);
}

int main(int argc, char **argv)

{
	const char *cgroot = (argc > 1) ? argv[1] : "/sys/fs/cgroup/drain-min";
	char cginner[256], path[300], buf[32];

	snprintf(cginner, sizeof(cginner), "%s/inner", cgroot);

	mkdir(cgroot, 0755);
	if (mkdir(cginner, 0755) < 0 && errno != EEXIST)
		die("mkdir inner");

	snprintf(path, sizeof(path), "%s/cgroup.procs", cginner);
	snprintf(buf, sizeof(buf), "%d", getpid());
	write_str(path, buf);

	if (unshare(CLONE_NEWPID) < 0)
		die("unshare CLONE_NEWPID");

	pid_t b = fork();
	if (b < 0)
		die("fork B");
	if (b == 0) {
		/* B: PID 1 of new pidns */
		pause();
		_exit(0);
	}

	pid_t c = fork();
	if (c < 0)
		die("fork C");
	if (c == 0) {
		/* C: PID 2 of new pidns, host parent is A */
		pause();
		_exit(0);
	}

	snprintf(path, sizeof(path), "%s/cgroup.procs", cgroot);
	write_str(path, buf);

	fprintf(stderr, "A: B host pid=%d, C host pid=%d\n", b, c);

	/* Briefly verify pidns membership */
	for (int i = 0; i < 2; i++) {
		pid_t p = (i == 0) ? b : c;
		char dpath[64], dbuf[256];
		snprintf(dpath, sizeof(dpath), "/proc/%d/status", p);
		int fd = open(dpath, O_RDONLY);
		if (fd >= 0) {
			ssize_t n = read(fd, dbuf, sizeof(dbuf) - 1);
			close(fd);
			if (n > 0) {
				dbuf[n] = 0;
				char *l = strstr(dbuf, "NSpid:");
				if (l) {
					char *e = strchr(l, '\n');
					if (e) *e = 0;
					fprintf(stderr, "  pid=%d %s\n", p, l);
				}
			}
		}
	}

	/* SIGKILL both. After this, B starts zap_pid_ns_processes; C
	 * dies and becomes a zombie waiting for A's wait4. */
	kill(b, SIGKILL);
	kill(c, SIGKILL);

	usleep(500000);

	fprintf(stderr, "A: rmdir(%s) — wedges if bug present "
		"(deliberately NOT wait4-ing C)\n", cginner);

	int rc = rmdir(cginner);
	int saved_errno = errno;
	fprintf(stderr, "A: rmdir returned %d (errno=%d %s)\n", rc,
		saved_errno, strerror(saved_errno));

	/* For cleanup if rmdir succeeded */
	waitpid(c, NULL, WNOHANG);
	waitpid(b, NULL, WNOHANG);
	rmdir(cgroot);
	return rc;
}

----- end min-repro.c -----

Hello Tejun,

(Dropping [email protected] from CC:, it doesn't exist any more)

Tejun Heo [2026-04-29 6:21 -1000]:

Increased to 4M, which was enough. I added it to the bottom of the debug notes comment [1], direct link: [2]. I suppose its' not necessary any more, but just for the records..

[1] https://github.com/cockpit-project/bots/pull/8970#issuecomment-4342147158
[2] https://github.com/user-attachments/files/27231725/dmesg-task-dump.txt

Tejun Heo [2026-04-29 11:15 -1000]:

You are a genius!

FTR, the container is pretty dumb, just

podman run quay.io/prometheus/busybox sh -c 'echo 123; sleep infinity'

we are not actually interested in the container workload for this tests, but testing cockpit-podman for managing containers on the host.

However, I just confirmed that busybox'es sh, like "proper" bash, does reap child processes (unlike for example running sleep directly as pid 1, then you do get zombies)

On Fedora 44 with 6.9.13, this hangs at

A: rmdir(/sys/fs/cgroup/drain-min/inner) — wedges if bug present (deliberately NOT wait4-ing C)

root        1501  0.0  0.1   2460  1764 pts/0    D+   06:10   0:00 /tmp/repr
root        1502  0.0  0.0      0     0 pts/0    S+   06:10   0:00 [repr]
root        1503  0.0  0.0      0     0 pts/0    Z+   06:10   0:00 [repr] <defunct>

as expected. It does not wedge up the system in the same way as breaking all "ls /proc" and such.

On Fedora 44 with older 6.9.10 kernel the reproducer finishes (no hang), with EBUSY:

: B host pid=1444, C host pid=1445
pid=1444 NSpid: 1444 1
pid=1445 NSpid: 1445 2
A: rmdir(/sys/fs/cgroup/drain-min/inner) — wedges if bug present (deliberately NOT wait4-ing C)
A: rmdir returned -1 (errno=16 Device or resource busy)

I suppose you know all that, but just in case confirming on my setup helps in any way.

Thanks!

Martin

A chain of commits going back to v7.0 reworked rmdir to satisfy the controller invariant that a subsystem's ->css_offline() must not run while tasks are still doing kernel-side work in the cgroup.

[1] d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out")
[2] a72f73c4dd9b ("cgroup: Don't expose dead tasks in cgroup")
[3] 1b164b876c36 ("cgroup: Wait for dying tasks to leave on rmdir")
[4] 4c56a8ac6869 ("cgroup: Fix cgroup_drain_dying() testing the wrong condition")
[5] 13e786b64bd3 ("cgroup: Increment nr_dying_subsys_* from rmdir context")

[1] moved task cset unlink from do_exit() to finish_task_switch() so a task's cset link drops only after the task has fully stopped scheduling. That made tasks past exit_signals() linger on cset->tasks until their final context switch, which led to a series of problems as what userspace expected to see after rmdir diverged from what the kernel needs to wait for. [2]-[5] tried to bridge that divergence: [2] filtered the exiting tasks from cgroup.procs; [3] had rmdir(2) sleep in TASK_UNINTERRUPTIBLE for them; [4] fixed the wait's condition; [5] made nr_dying_subsys_* visible synchronously.

The cgroup_drain_dying() wait in [3] turned out to be a dead end. When the rmdir caller is also the reaper of a zombie that pins a pidns teardown (e.g. host PID 1 systemd reaping orphan pids that were re-parented to it during the same teardown), rmdir blocks in TASK_UNINTERRUPTIBLE waiting for those pids to free, the pids can't free because PID 1 is the reaper and it's stuck in rmdir, and the system A-A deadlocks. No internal lock ordering breaks this; the wait itself is the bug.

The css killing side that drove the original reorder, however, can be made cleanly asynchronous: ->css_offline() is already async, run from css_killed_work_fn() driven by percpu_ref_kill_and_confirm(). The fix is to make that chain start only after all tasks have left the cgroup. rmdir's user-visible side then returns as soon as cgroup.procs and friends are empty, while ->css_offline() still runs only after the cgroup is fully drained.

Verified by the original reproducer (pidns teardown + zombie reaper, runs under vng) which hangs vanilla and succeeds here, and by per-commit deterministic repros for [2], [3], [4], [5] with a boot parameter that widens the post-exit_signals() window so each state is reliably reachable. Some stress tests on top of that.

This seems like the right approach and I don't see problems with it. The changes are somewhat invasive but not excessively so, so backporting to -stable should be okay. If something does turn out to be wrong, the fallback is to revert the entire chain ([1]-[5]) and rework in the development branch instead.

---
Hello Martin,

Could you give this a try? It defers the percpu_ref kill chain on rmdir until the cgroup is fully drained, which removes the TASK_UNINTERRUPTIBLE wait that deadlocked against PID 1 reaping. The patch description has the details.

Thanks.

 include/linux/cgroup-defs.h |   4 +-
 kernel/cgroup/cgroup.c      | 241 ++++++++++++++++--------------------
 2 files changed, 110 insertions(+), 135 deletions(-)

index f42563739d2e..50a784da7a81 100644
@@ -611,8 +611,8 @@ struct cgroup {
 	/* used to wait for offlining of csses */
 	wait_queue_head_t offline_waitq;
 
-	/* used by cgroup_rmdir() to wait for dying tasks to leave */
-	wait_queue_head_t dying_populated_waitq;
+	/* defers killing csses after removal until cgroup is depopulated */
+	struct work_struct finish_destroy_work;
 
 	/* used to schedule release agent */
 	struct work_struct release_agent_work;

2.54.0