Robust futex causing memcg OOM storm on exit

At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.

The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.

On further inspection, I observed ~20k threads of that process stuck with the following stack:

[<0>] mem_cgroup_out_of_memory+0x4e/0xa0
[<0>] charge_memcg+0x8bf/0x990
[<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80
[<0>] __read_swap_cache_async+0x10c/0x260
[<0>] swapin_readahead+0x116/0x3f0
[<0>] do_swap_page+0x13c/0x1ce0
[<0>] handle_mm_fault+0x61d/0x11f0
[<0>] do_user_addr_fault+0x3e7/0x6d0
[<0>] exc_page_fault+0x8f/0x110
[<0>] asm_exc_page_fault+0x22/0x30
[<0>] __get_user_8+0x14/0x20
[<0>] futex_cleanup+0x27/0x1c0
[<0>] futex_exit_release+0x47/0x60
[<0>] do_exit+0x107/0x940
[<0>] do_group_exit+0x81/0xa0
[<0>] get_signal+0x2b1/0x6e0
[<0>] arch_do_signal_or_restart+0x1a/0x1c0
[<0>] exit_to_user_mode_loop+0xa8/0x1c0
[<0>] do_syscall_64+0x152/0x250
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

In addition the dmesg was filled with "Out of memory and no killable processes..." messages.

I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire mmap_lock in read mode limited number of times and then gives up, there might a thread of that process which had mmap_lock in write mode at that time.

My initial suspicion was the futex_cleanup and kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1].

My current theory is that it is just a simple slow serialization behind the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock without the "try". Though memcg oom code uses mutex_lock_killable(), note that in the call stack get_stack() consumes SIGKILL (or sigdelset(SIGKILL)) before calling do_cgroup_exit(). So this mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of thousands of threads are waiting on oom_lock and one by one they get -EFAULT from get_user() in the futex cleanup code and bails out.

Discussion from [1] lead to the commit a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so the oom_reaper can reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot take mmap_lock for read (e.g. a sibling thread holds it for write) it sets MMF_OOM_SKIP and never retries, leaving only the glacial oom_lock-serialized synchronous drain.

Let's short-circuit that path: once reclaim has failed, if current is dying, force the charge instead of invoking the OOM killer for it. A dying task frees its memory as soon as it finishes exiting, so running the (necessarily no-victim) OOM killer for it is pointless - and doing so for 10s of thousands of exiting threads is exactly what serializes them behind oom_lock. The dying task instead faults its page in, completes exit and releases its memory, including the zswap pool, so the memcg recovers on its own without the oom_lock serialization and dump_header storm.

Quoted 63 lines · Shakeel Butt

At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.

The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.

On further inspection, I observed ~20k threads of that process stuck with the following stack:

[<0>] mem_cgroup_out_of_memory+0x4e/0xa0
[<0>] charge_memcg+0x8bf/0x990
[<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80
[<0>] __read_swap_cache_async+0x10c/0x260
[<0>] swapin_readahead+0x116/0x3f0
[<0>] do_swap_page+0x13c/0x1ce0
[<0>] handle_mm_fault+0x61d/0x11f0
[<0>] do_user_addr_fault+0x3e7/0x6d0
[<0>] exc_page_fault+0x8f/0x110
[<0>] asm_exc_page_fault+0x22/0x30
[<0>] __get_user_8+0x14/0x20
[<0>] futex_cleanup+0x27/0x1c0
[<0>] futex_exit_release+0x47/0x60
[<0>] do_exit+0x107/0x940
[<0>] do_group_exit+0x81/0xa0
[<0>] get_signal+0x2b1/0x6e0
[<0>] arch_do_signal_or_restart+0x1a/0x1c0
[<0>] exit_to_user_mode_loop+0xa8/0x1c0
[<0>] do_syscall_64+0x152/0x250
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

In addition the dmesg was filled with "Out of memory and no killable processes..." messages.

I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire mmap_lock in read mode limited number of times and then gives up, there might a thread of that process which had mmap_lock in write mode at that time.

My initial suspicion was the futex_cleanup and kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1].

My current theory is that it is just a simple slow serialization behind the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock without the "try". Though memcg oom code uses mutex_lock_killable(), note that in the call stack get_stack() consumes SIGKILL (or sigdelset(SIGKILL)) before calling do_cgroup_exit(). So this mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of thousands of threads are waiting on oom_lock and one by one they get -EFAULT from get_user() in the futex cleanup code and bails out.

Discussion from [1] lead to the commit a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so the oom_reaper can reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot take mmap_lock for read (e.g. a sibling thread holds it for write) it sets MMF_OOM_SKIP and never retries, leaving only the glacial oom_lock-serialized synchronous drain.

Quoted 9 lines

Let's short-circuit that path: once reclaim has failed, if current is dying, force the charge instead of invoking the OOM killer for it. A dying task frees its memory as soon as it finishes exiting, so running the (necessarily no-victim) OOM killer for it is pointless - and doing so for 10s of thousands of exiting threads is exactly what serializes them behind oom_lock. The dying task instead faults its page in, completes exit and releases its memory, including the zswap pool, so the memcg recovers on its own without the oom_lock serialization and dump_header storm.

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

---

mm/memcontrol.c +9 -0

index 6dc4888a90f3..3e0a6b601767 100644
@@ -2685,6 +2685,15 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (gfp_mask & __GFP_RETRY_MAYFAIL)
 		goto nomem;
 
+	/*
+	 * OOM victim still needs to charge memory to exit. OOM reaper should
+	 * help but it might fail on mmap_lock contention. If the victim is a
+	 * large thread group then all exiting threads might compete on oom_lock
+	 * just to learn that there is nothing really killable anymore. Bail
+	 * out early and force the charge to expedite their exit.
+	 */
+	if (tsk_is_oom_victim(current))
+		goto force
 	/* Avoid endless loop for tasks bypassed by the oom killer */
 	if (passed_oom && task_is_dying())
 		goto nomem;

Signature · 2 lines

Michal Hocko
SUSE Labs

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

Quoted 2 lines · Shakeel Butt

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

Quoted 15 lines

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 9 lines

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a large anon region pushed into a zswap pool (writeback off, so it's unreclaimable and pins the memcg at the limit). On the OOM-group-kill every thread faults that page in futex_cleanup() -> do_swap_page() -> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write in a loop so the oom_reaper can't reap the pool -- matching the host, where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it. For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to drain the 20k threads in a VM (hours on the host, where each invocation was a dump_header over netconsole). With the patch the dying tasks stop entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering out_of_memory but the exit time doubled as each thread has to decompress the page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved i.e. ~20 seconds. Looking deeper it seems like each thread is still going through the reclaim loops.

Just for testing purpose I moved this check along with goto nomem just before the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls below 1 second. I think what we need is to detect early that the cgroup is unreclaimable and the threads of exiting/reaped process should just return with enomem.

Quoted 58 lines · Shakeel Butt

Quoted 2 lines · Shakeel Butt

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

Quoted 15 lines

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 9 lines

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a large anon region pushed into a zswap pool (writeback off, so it's unreclaimable and pins the memcg at the limit). On the OOM-group-kill every thread faults that page in futex_cleanup() -> do_swap_page() -> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write in a loop so the oom_reaper can't reap the pool -- matching the host, where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it. For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to drain the 20k threads in a VM (hours on the host, where each invocation was a dump_header over netconsole). With the patch the dying tasks stop entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering out_of_memory but the exit time doubled as each thread has to decompress the page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved i.e. ~20 seconds. Looking deeper it seems like each thread is still going through the reclaim loops.

Quoted 5 lines

Just for testing purpose I moved this check along with goto nomem just before the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls below 1 second. I think what we need is to detect early that the cgroup is unreclaimable and the threads of exiting/reaped process should just return with enomem.

Do you mean, something like this?
---

mm/memcontrol.c +0 -0

index 6dc4888a90f3..66f1eebf8c22 100644
@@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (!gfpflags_allow_blocking(gfp_mask))
 		goto nomem;
 

mm/memcontrol.c +2 -0

+		goto nomem;
+
 	__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
 	raised_max_event = true;
 

Signature · 3 lines

-- 
Michal Hocko
SUSE Labs

Quoted 75 lines · Michal Hocko

Quoted 58 lines · Shakeel Butt

Quoted 2 lines · Shakeel Butt

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

Quoted 15 lines

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 9 lines

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a large anon region pushed into a zswap pool (writeback off, so it's unreclaimable and pins the memcg at the limit). On the OOM-group-kill every thread faults that page in futex_cleanup() -> do_swap_page() -> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write in a loop so the oom_reaper can't reap the pool -- matching the host, where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it. For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to drain the 20k threads in a VM (hours on the host, where each invocation was a dump_header over netconsole). With the patch the dying tasks stop entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering out_of_memory but the exit time doubled as each thread has to decompress the page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved i.e. ~20 seconds. Looking deeper it seems like each thread is still going through the reclaim loops.

Quoted 5 lines

Just for testing purpose I moved this check along with goto nomem just before the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls below 1 second. I think what we need is to detect early that the cgroup is unreclaimable and the threads of exiting/reaped process should just return with enomem.

Do you mean, something like this?
---

mm/memcontrol.c +0 -0

index 6dc4888a90f3..66f1eebf8c22 100644
@@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (!gfpflags_allow_blocking(gfp_mask))
 		goto nomem;
 

mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2660

Quoted 10 lines

mm/memcontrol.c +10 -0

+	/*
+	 * OOM victim still needs to charge memory to exit. OOM reaper should
+	 * help but it might fail on mmap_lock contention. If the victim is a
+	 * large thread group then all exiting threads might compete on oom_lock
+	 * just to learn that there is nothing really killable anymore. Bail
+	 * out early and fail the charge to expedite their exit. They are
+	 * considered fully reclaimed by the oom reaper and they shouldn't contribute
+	 * further charges.
+	 */
+	if (tsk_is_oom_victim(current) && mm_flags_test(current->signal->oom_mm))

I simply had tsk_is_oom_victim(current) check here.

I assume you want
MMF_OOM_REAP_QUEUED in the mm_flags_test() above. Do you want me to test with this additional mm_flags_test() as well?

Quoted 76 lines · Shakeel Butt

Quoted 75 lines · Michal Hocko

Quoted 58 lines · Shakeel Butt

Quoted 2 lines · Shakeel Butt

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

Quoted 15 lines

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 9 lines

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a large anon region pushed into a zswap pool (writeback off, so it's unreclaimable and pins the memcg at the limit). On the OOM-group-kill every thread faults that page in futex_cleanup() -> do_swap_page() -> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write in a loop so the oom_reaper can't reap the pool -- matching the host, where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it. For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to drain the 20k threads in a VM (hours on the host, where each invocation was a dump_header over netconsole). With the patch the dying tasks stop entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering out_of_memory but the exit time doubled as each thread has to decompress the page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved i.e. ~20 seconds. Looking deeper it seems like each thread is still going through the reclaim loops.

Quoted 5 lines

Just for testing purpose I moved this check along with goto nomem just before the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls below 1 second. I think what we need is to detect early that the cgroup is unreclaimable and the threads of exiting/reaped process should just return with enomem.

Do you mean, something like this?
---

mm/memcontrol.c +0 -0

index 6dc4888a90f3..66f1eebf8c22 100644
@@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (!gfpflags_allow_blocking(gfp_mask))
 		goto nomem;
 

Quoted 12 lines

mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2660

Quoted 10 lines

mm/memcontrol.c +10 -0

+	/*
+	 * OOM victim still needs to charge memory to exit. OOM reaper should
+	 * help but it might fail on mmap_lock contention. If the victim is a
+	 * large thread group then all exiting threads might compete on oom_lock
+	 * just to learn that there is nothing really killable anymore. Bail
+	 * out early and fail the charge to expedite their exit. They are
+	 * considered fully reclaimed by the oom reaper and they shouldn't contribute
+	 * further charges.
+	 */
+	if (tsk_is_oom_victim(current) && mm_flags_test(current->signal->oom_mm))

I simply had tsk_is_oom_victim(current) check here.

I was hoping for a more specific treatment.

Quoted 3 lines

I assume you want
MMF_OOM_REAP_QUEUED in the mm_flags_test() above. Do you want me to test with this additional mm_flags_test() as well?

Sorry I meant MMF_OOM_SKIP there.

Quoted 90 lines · Michal Hocko

Quoted 76 lines · Shakeel Butt

Quoted 75 lines · Michal Hocko

Quoted 58 lines · Shakeel Butt

Quoted 2 lines · Shakeel Butt

Quoted 1 line · Michal Hocko

On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

Quoted 15 lines

Quoted 6 lines

TBH I am not entirely happy about this approach. It effectivelly reverts a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks"). It just makes it lockless. Assumption that a dying task will do so quickly and with bounded resources has turned wrong on several occasions.

That's why I kept it as RFC :)

Quoted 6 lines

On the other hand I do undestand the contention issues and I can imagine that the existing solution doesn't really work well for huge thread groups that all end up lining up on the oom_lock just to learn there is nothing really killable anymore because they are the oom victim...

Quoted 9 lines

Quoted 4 lines

Would it be just safer to bail out only for oom victim threads. This would narrow down potential runaways for oom victims which should be more limited than any killed/exiting task. It would also give the oom killer/reaper chance to work. WDYT?

I will give the following patch a try with the reproducer I have. I am still improving the reproducer as I am still not able to recreate multi hour slowdown yet. I will report back once I have some results.

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a large anon region pushed into a zswap pool (writeback off, so it's unreclaimable and pins the memcg at the limit). On the OOM-group-kill every thread faults that page in futex_cleanup() -> do_swap_page() -> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write in a loop so the oom_reaper can't reap the pool -- matching the host, where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it. For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to drain the 20k threads in a VM (hours on the host, where each invocation was a dump_header over netconsole). With the patch the dying tasks stop entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering out_of_memory but the exit time doubled as each thread has to decompress the page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved i.e. ~20 seconds. Looking deeper it seems like each thread is still going through the reclaim loops.

Quoted 5 lines

Just for testing purpose I moved this check along with goto nomem just before the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls below 1 second. I think what we need is to detect early that the cgroup is unreclaimable and the threads of exiting/reaped process should just return with enomem.

Do you mean, something like this?
---

mm/memcontrol.c +0 -0

index 6dc4888a90f3..66f1eebf8c22 100644
@@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (!gfpflags_allow_blocking(gfp_mask))
 		goto nomem;
 

Quoted 12 lines

mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2660

Quoted 10 lines

mm/memcontrol.c +10 -0

+	/*
+	 * OOM victim still needs to charge memory to exit. OOM reaper should
+	 * help but it might fail on mmap_lock contention. If the victim is a
+	 * large thread group then all exiting threads might compete on oom_lock
+	 * just to learn that there is nothing really killable anymore. Bail
+	 * out early and fail the charge to expedite their exit. They are
+	 * considered fully reclaimed by the oom reaper and they shouldn't contribute
+	 * further charges.
+	 */
+	if (tsk_is_oom_victim(current) && mm_flags_test(current->signal->oom_mm))

I simply had tsk_is_oom_victim(current) check here.

Quoted 7 lines

I was hoping for a more specific treatment.

Quoted 3 lines

I assume you want
MMF_OOM_REAP_QUEUED in the mm_flags_test() above. Do you want me to test with this additional mm_flags_test() as well?

Sorry I meant MMF_OOM_SKIP there.

Here are the results with the reproducer I have. Ran the test 5 times and showing avg and std below:

Baseline (next-20260726) 36.47 ± 0.78 s
early nomem (tsk_is_oom_victim + MMF_OOM_SKIP) 20.83 ± 4.19 s
early nomem (tsk_is_oom_victim) 0.59 ± 0.22 s

Since MMF_OOM_SKIP is set after multiple retries by the oom_reaper the result in between baseline and simple tsk_is_oom_victim check makes sense.

I think we can proceed with the early nomem (tsk_is_oom_victim + MMF_OOM_SKIP) change as it is more conservative unless you have concerns. Overall I want to deploy this in prod sooner to see if this specific issues is resolved or we catch something else.

Now I have much better results. Previous numbers were on a VM and now I have ran on a physical machine and the results are much better and consistent. The baseline consistently shows ~90 seconds exit time while (tsk_is_oom_victim + MMF_OOM_SKIP) consistently show ~3 seconds exit time. I will send the formal patch soon.