memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
Subsystems (3)
Shakeel Butt
maintainerPATCH 1/1 memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0 [<0>] charge_memcg+0x8bf/0x990 [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80 [<0>] __read_swap_cache_async+0x10c/0x260 [<0>] swapin_readahead+0x116/0x3f0 [<0>] do_swap_page+0x13c/0x1ce0 [<0>] handle_mm_fault+0x61d/0x11f0 [<0>] do_user_addr_fault+0x3e7/0x6d0 [<0>] exc_page_fault+0x8f/0x110 [<0>] asm_exc_page_fault+0x22/0x30 [<0>] __get_user_8+0x14/0x20 [<0>] futex_cleanup+0x27/0x1c0 [<0>] futex_exit_release+0x47/0x60 [<0>] do_exit+0x107/0x940 [<0>] do_group_exit+0x81/0xa0 [<0>] get_signal+0x2b1/0x6e0 [<0>] arch_do_signal_or_restart+0x1a/0x1c0 [<0>] exit_to_user_mode_loop+0xa8/0x1c0 [<0>] do_syscall_64+0x152/0x250 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
In addition the dmesg was filled with "Out of memory and no killable processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire mmap_lock in read mode limited number of times and then gives up, there might be a thread of that process which had mmap_lock in write mode at that time.
My initial suspicion was the futex_cleanup and kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1].
My current theory is that it is just a simple slow serialization behind the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock without the "try". Though memcg oom code uses mutex_lock_killable(), note that in the call stack get_signal() consumes SIGKILL (or sigdelset(SIGKILL)) before calling do_group_exit(). So this mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of thousands of threads are waiting on oom_lock and one by one they get -EFAULT from get_user() in the futex cleanup code and bails out.
Discussion from [1] lead to the commit a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so the oom_reaper can reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot take mmap_lock for read (e.g. a sibling thread holds it for write) it sets MMF_OOM_SKIP and never retries, leaving only the glacial oom_lock-serialized synchronous drain.
Once MMF_OOM_SKIP is set there is no more asynchronous reclaim coming for the mm, so a dying task charging against it has nothing left to wait for:
it frees its memory only once it finishes exiting. Running reclaim and the (no-victim) OOM killer for it is then pointless, and doing it for 10s of thousands of exiting threads is what serializes them behind oom_lock. So before reclaim, if current is an OOM victim whose reaper is done, fail the charge.
Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds mmap_lock for write so the reaper gives up and sets MMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds.
--- mm/memcontrol.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/memcontrol.c +13 -0
@@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!gfpflags_allow_blocking (gfp_mask)) goto nomem; +/* +* OOM victim still needs to charge memory to exit. OOM reaper should +* help but it might fail on mmap_lock contention. If the victim is a +* large thread group then all exiting threads might compete on oom_lock +* just to learn that there is nothing really killable anymore. Bail +* out early and fail the charge to expedite their exit. They are +* considered fully reclaimed by the oom reaper and they shouldn't +* contribute further charges. +*/ +if (tsk_is_oom_victim (current) && +mm_flags_test (MMF_OOM_SKIP , current->signal->oom_mm)) + goto nomem; +__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); raised_max_event = true;
Signature · 1 line
2.53.0-Meta
Quoted 71 lines · Shakeel Butt
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0 [<0>] charge_memcg+0x8bf/0x990 [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80 [<0>] __read_swap_cache_async+0x10c/0x260 [<0>] swapin_readahead+0x116/0x3f0 [<0>] do_swap_page+0x13c/0x1ce0 [<0>] handle_mm_fault+0x61d/0x11f0 [<0>] do_user_addr_fault+0x3e7/0x6d0 [<0>] exc_page_fault+0x8f/0x110 [<0>] asm_exc_page_fault+0x22/0x30 [<0>] __get_user_8+0x14/0x20 [<0>] futex_cleanup+0x27/0x1c0 [<0>] futex_exit_release+0x47/0x60 [<0>] do_exit+0x107/0x940 [<0>] do_group_exit+0x81/0xa0 [<0>] get_signal+0x2b1/0x6e0 [<0>] arch_do_signal_or_restart+0x1a/0x1c0 [<0>] exit_to_user_mode_loop+0xa8/0x1c0 [<0>] do_syscall_64+0x152/0x250 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53In addition the dmesg was filled with "Out of memory and no killable processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire
mmap_lockin read mode limited number of times and then gives up, there might be a thread of that process which hadmmap_lockin write mode at that time.My initial suspicion was the
futex_cleanupand kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1].My current theory is that it is just a simple slow serialization behind the
oom_lock. Unlike page allocator, memcg charge code takes theoom_lockwithout the "try". Though memcg oom code usesmutex_lock_killable(), note that in the call stackget_signal()consumes SIGKILL (or sigdelset(SIGKILL)) before callingdo_group_exit(). So thismutex_lock_killable()is just amutex_lock()here. Therefore 10s of thousands of threads are waiting onoom_lockand one by one they get-EFAULTfromget_user()in the futex cleanup code and bails out.Discussion from [1] lead to the commit
a75ffa26122b("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so theoom_reapercan reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot takemmap_lockfor read (e.g. a sibling thread holds it for write) it setsMMF_OOM_SKIPand never retries, leaving only the glacial oom_lock-serialized synchronous drain.Once
MMF_OOM_SKIPis set there is no more asynchronous reclaim coming for the mm, so a dying task charging against it has nothing left to wait for:
it frees its memory only once it finishes exiting. Running reclaim and the (no-victim) OOM killer for it is then pointless, and doing it for 10s of thousands of exiting threads is what serializes them behindoom_lock. So before reclaim, if current is an OOM victim whose reaper is done, fail the charge.
Quoted 8 lines
Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds
mmap_lockfor write so the reaper gives up and setsMMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds.Signed-off-by Shakeel Butt <[email protected]>
Andrew Morton
maintainerQuoted 4 lines · On Tue, 28 Jul 2026 19:46:12 -0700 Shakeel Butt <[email protected]> wrote:
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
Not seeing the problem here - that sounds like normal kernel behavior ;)
Quoted 5 lines
Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds
mmap_lockfor write so the reaper gives up and setsMMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds.
Very positive.
But it doesn't sound like you're fully confident that the problem is fully solved?
Quoted 11 lines
mm/memcontrol.c +5
-0@@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!gfpflags_allow_blocking (gfp_mask)) goto nomem; +/* +* OOM victim still needs to charge memory to exit. OOM reaper should +* help but it might fail on mmap_lock contention. If the victim is a +* large thread group then all exiting threads might compete on oom_lock +* just to learn that there is nothing really killable anymore. Bail
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2670
Quoted 10 lines
mm/memcontrol.c +8
-0+* out early and fail the charge to expedite their exit. They are +* considered fully reclaimed by the oom reaper and they shouldn't +* contribute further charges. +*/ +if (tsk_is_oom_victim (current) && +mm_flags_test (MMF_OOM_SKIP , current->signal->oom_mm)) + goto nomem; +__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); raised_max_event = true;
Should we backport this?
Shakeel Butt
maintainerQuoted 15 lines · Andrew Morton
Quoted 4 lines · On Tue, 28 Jul 2026 19:46:12 -0700 Shakeel Butt <[email protected]> wrote:
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
Not seeing the problem here - that sounds like normal kernel behavior ;)
Quoted 5 lines
Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds
mmap_lockfor write so the reaper gives up and setsMMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds.
Quoted 4 lines
Very positive.
But it doesn't sound like you're fully confident that the problem is fully solved?
Yes as I am not able to reproduce multi hour stall, so I might be missing something else. The patch at least fixes the issue which I am able to reproduce.
Quoted 16 lines
Quoted 16 lines
mm/memcontrol.c +9
-0@@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!gfpflags_allow_blocking (gfp_mask)) goto nomem; +/* +* OOM victim still needs to charge memory to exit. OOM reaper should +* help but it might fail on mmap_lock contention. If the victim is a +* large thread group then all exiting threads might compete on oom_lock +* just to learn that there is nothing really killable anymore. Bail +* out early and fail the charge to expedite their exit. They are +* considered fully reclaimed by the oom reaper and they shouldn't +* contribute further charges. +*/
Quoted 8 lines
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2670
Quoted 6 lines
mm/memcontrol.c +4
-0+if (tsk_is_oom_victim (current) && +mm_flags_test (MMF_OOM_SKIP , current->signal->oom_mm)) + goto nomem; +__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); raised_max_event = true;Should we backport this?
I don't have any strong opinion. I wanted to deploy this patch in Meta fleet to get some exposure. Maybe backporting will give more exposure to it.
Quoted 71 lines · Shakeel Butt
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0 [<0>] charge_memcg+0x8bf/0x990 [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80 [<0>] __read_swap_cache_async+0x10c/0x260 [<0>] swapin_readahead+0x116/0x3f0 [<0>] do_swap_page+0x13c/0x1ce0 [<0>] handle_mm_fault+0x61d/0x11f0 [<0>] do_user_addr_fault+0x3e7/0x6d0 [<0>] exc_page_fault+0x8f/0x110 [<0>] asm_exc_page_fault+0x22/0x30 [<0>] __get_user_8+0x14/0x20 [<0>] futex_cleanup+0x27/0x1c0 [<0>] futex_exit_release+0x47/0x60 [<0>] do_exit+0x107/0x940 [<0>] do_group_exit+0x81/0xa0 [<0>] get_signal+0x2b1/0x6e0 [<0>] arch_do_signal_or_restart+0x1a/0x1c0 [<0>] exit_to_user_mode_loop+0xa8/0x1c0 [<0>] do_syscall_64+0x152/0x250 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53In addition the dmesg was filled with "Out of memory and no killable processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire
mmap_lockin read mode limited number of times and then gives up, there might be a thread of that process which hadmmap_lockin write mode at that time.My initial suspicion was the
futex_cleanupand kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1].My current theory is that it is just a simple slow serialization behind the
oom_lock. Unlike page allocator, memcg charge code takes theoom_lockwithout the "try". Though memcg oom code usesmutex_lock_killable(), note that in the call stackget_signal()consumes SIGKILL (or sigdelset(SIGKILL)) before callingdo_group_exit(). So thismutex_lock_killable()is just amutex_lock()here. Therefore 10s of thousands of threads are waiting onoom_lockand one by one they get-EFAULTfromget_user()in the futex cleanup code and bails out.Discussion from [1] lead to the commit
a75ffa26122b("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so theoom_reapercan reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot takemmap_lockfor read (e.g. a sibling thread holds it for write) it setsMMF_OOM_SKIPand never retries, leaving only the glacial oom_lock-serialized synchronous drain.Once
MMF_OOM_SKIPis set there is no more asynchronous reclaim coming for the mm, so a dying task charging against it has nothing left to wait for:
it frees its memory only once it finishes exiting. Running reclaim and the (no-victim) OOM killer for it is then pointless, and doing it for 10s of thousands of exiting threads is what serializes them behindoom_lock. So before reclaim, if current is an OOM victim whose reaper is done, fail the charge.
Quoted 8 lines
Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds
mmap_lockfor write so the reaper gives up and setsMMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds.Signed-off-by Shakeel Butt <[email protected]>
Thanks
Quoted 31 lines
--- mm/memcontrol.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)mm/memcontrol.c +13
-0@@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!gfpflags_allow_blocking (gfp_mask)) goto nomem; +/* +* OOM victim still needs to charge memory to exit. OOM reaper should +* help but it might fail on mmap_lock contention. If the victim is a +* large thread group then all exiting threads might compete on oom_lock +* just to learn that there is nothing really killable anymore. Bail +* out early and fail the charge to expedite their exit. They are +* considered fully reclaimed by the oom reaper and they shouldn't +* contribute further charges. +*/ +if (tsk_is_oom_victim (current) && +mm_flags_test (MMF_OOM_SKIP , current->signal->oom_mm)) + goto nomem; +__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); raised_max_event = true;Signature · 1 line
2.53.0-Meta
Signature · 2 lines
Michal Hocko SUSE Labs