oom, memcg: do not report racy no-eligible OOM
Subsystems (4)
Michal Hocko
maintainercover oom, memcg: do not report racy no-eligible OOM
Hi,
this is a follow up for [1] which has been nacked mostly because Tetsuo was able to find a simple workload which can trigger a race where no-eligible task is reported without a good reason. I believe the patch2 addresses that issue and we do not have to play dirty games with throttling just because of the race. I still believe that patch proposed in [1] is a useful one but this can be addressed later.
This series comprises 2 patch. The first one is something I meant to do loooong time ago, I just never have time to do that. We need it here to handle CLONE_VM without CLONE_SIGHAND cases. The second patch closes the race.
I didn't get to test this throughly so it is posted as an RFC.
Feedback is appreciated of course.
Michal Hocko
maintainerRFC PATCH 1/2 mm, oom: marks all killed tasks as oom victims
From: Michal Hocko <[email protected]>
Historically we have called mark_oom_victim only to the main task selected as the oom victim because oom victims have access to memory reserves and granting the access to all killed tasks could deplete memory reserves very quickly and cause even larger problems.
Since only a partial access to memory reserves is allowed there is no longer this risk and so all tasks killed along with the oom victim can be considered as well.
The primary motivation for that is that process groups which do not shared signals would behave more like standard thread groups wrt oom handling (aka tsk_is_oom_victim will work the same way for them).
--- mm/oom_kill.c | 1 + 1 file changed, 1 insertion(+)
mm/oom_kill.c +1 -0
@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();
Signature · 1 line
2.19.1
Re: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
mm/oom_kill.c static void __oom_kill_process(struct task_struct *victim) · line 904
Quoted 11 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();
Wrong. Either
--- mm/oom_kill.c | 7 +++++++ 1 file changed, 7 insertions(+)
mm/oom_kill.c +7 -0
@@ -879,6 +879,8 @@ static void __oom_kill_process(struct task_struct *victim)*/ rcu_read_lock ();for_each_process (p) { +struct task_struct *t; +if (!process_shares_mm (p, mm))continue ;if (same_thread_group (p, victim))
@@ -898,6 +900,11 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); + t =find_lock_task_mm (p); +if (!t) +continue ; +mark_oom_victim (t); +task_unlock (t); }rcu_read_unlock ();
Signature · 7 lines
1.8.3.1 or --- mm/oom_kill.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
mm/oom_kill.c +19 -13
@@ -854,13 +854,6 @@ static void __oom_kill_process(struct task_struct *victim)count_vm_event (OOM_KILL );memcg_memory_event_mm (mm,MEMCG_OOM_KILL ); -/* -* We should send SIGKILL before granting access to memory reserves -* in order to prevent the OOM victim from depleting the memory -* reserves from the user space under its control. -*/ -do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , victim,PIDTYPE_TGID ); -mark_oom_victim (victim); pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n", task_pid_nr(victim), victim->comm, K(victim->mm->total_vm), K(get_mm_counter(victim->mm, MM_ANONPAGES)),
@@ -879,11 +872,23 @@ static void __oom_kill_process(struct task_struct *victim)*/ rcu_read_lock ();for_each_process (p) { -if (!process_shares_mm (p, mm)) +struct task_struct *t; + +/* +* No use_mm() user needs to read from the userspace so we are +* ok to reap it. +*/ +if (unlikely (p->flags &PF_KTHREAD )) +continue ; + t =find_lock_task_mm (p); +if (!t)continue ; -if (same_thread_group (p, victim)) +if (likely (t->mm != mm)) { +task_unlock (t);continue ; + }if (is_global_init (p)) { +task_unlock (t); can_oom_reap = false;set_bit (MMF_OOM_SKIP , &mm->flags); pr_info("oom killer %d (%s) has mm pinned by %d (%s)\n",
@@ -892,12 +897,13 @@ static void __oom_kill_process(struct task_struct *victim)continue ; }/* -* No use_mm() user needs to read from the userspace so we are -* ok to reap it. +* We should send SIGKILL before granting access to memory +* reserves in order to prevent the OOM victim from depleting +* the memory reserves from the user space under its control. */ -if (unlikely (p->flags &PF_KTHREAD )) -continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (t); +task_unlock (t); }rcu_read_unlock ();
Signature · 3 lines
1.8.3.1 will be needed.
Michal Hocko
maintainerRe: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
Quoted 11 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();
Quoted 3 lines
Quoted 1 line
Wrong. Either
You are right. The mm might go away between process_shares_mm and here. While your find_lock_task_mm would be correct I believe we can do better by using the existing mm that we already have. I will make it a separate patch to clarity.
Thanks for pointing this out.
mm/oom_kill.c +6 -7
@@ -663,6 +663,7 @@ static inline void wake_oom_reaper(struct task_struct *tsk)/** * mark_oom_victim - mark the given task as OOM victim * @tsk: task to mark +* @mm: mm associated with the task * * Has to be called with oom_lock held and never after * oom has been disabled already.
@@ -670,10 +671,8 @@ static inline void wake_oom_reaper(struct task_struct *tsk)* tsk->mm has to be non NULL and caller has to guarantee it is stable (either * under task_lock or operate on the current). */ -static void mark_oom_victim (struct task_struct *tsk) +static void mark_oom_victim (struct task_struct *tsk,struct mm_struct *mm) { -struct mm_struct *mm = tsk->mm; -WARN_ON (oom_killer_disabled);/* OOM killer might race with memcg OOM */ if (test_and_set_tsk_thread_flag (tsk,TIF_MEMDIE ))
@@ -860,7 +859,7 @@ static void __oom_kill_process(struct task_struct *victim)* reserves from the user space under its control. */ do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , victim,PIDTYPE_TGID ); -mark_oom_victim (victim); +mark_oom_victim (victim, mm); pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n", task_pid_nr(victim), victim->comm, K(victim->mm->total_vm), K(get_mm_counter(victim->mm, MM_ANONPAGES)),
@@ -898,7 +897,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); -mark_oom_victim (p); +mark_oom_victim (p, mm); }rcu_read_unlock ();
@@ -942,7 +941,7 @@ static void oom_kill_process(struct oom_control *oc, const char *message)*/ task_lock (p);if (task_will_free_mem (p)) { -mark_oom_victim (p); +mark_oom_victim (p, p->mm);wake_oom_reaper (p);task_unlock (p);put_task_struct (p);
@@ -1072,7 +1071,7 @@ bool out_of_memory(struct oom_control *oc)* quickly exit and free its memory. */ if (task_will_free_mem (current)) { -mark_oom_victim (current); +mark_oom_victim (current, current->mm);wake_oom_reaper (current);return true; }
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
Quoted 12 lines · Michal Hocko
Quoted 11 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Wrong. Either
You are right. The mm might go away between
process_shares_mmand here. While yourfind_lock_task_mmwould be correct I believe we can do better by using the existing mm that we already have. I will make it a separate patch to clarity.
Still wrong. p->mm == NULL means that we are too late to set TIF_MEMDIE on that thread. Passing non-NULL mm to mark_oom_victim() won't help.
Quoted 9 lines
mm/oom_kill.c +1
-1@@ -898,7 +897,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); -mark_oom_victim (p); +mark_oom_victim (p, mm); }rcu_read_unlock ();
Michal Hocko
maintainerRe: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
Quoted 17 lines · Tetsuo Handa
Quoted 16 lines · Michal Hocko
Quoted 14 lines · Tetsuo Handa
mm/oom_kill.c static void __oom_kill_process(struct task_struct *victim) · line 904
Quoted 11 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();Wrong. Either
Quoted 7 lines
Quoted 4 lines
You are right. The mm might go away between
process_shares_mmand here. While yourfind_lock_task_mmwould be correct I believe we can do better by using the existing mm that we already have. I will make it a separate patch to clarity.Still wrong.
p->mm== NULL means that we are too late to setTIF_MEMDIEon that thread. Passing non-NULL mm tomark_oom_victim()won't help.
Why would it be too late? Or in other words why would this be harmful?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
Quoted 23 lines · Michal Hocko
Quoted 22 lines · Tetsuo Handa
Quoted 20 lines · Michal Hocko
Quoted 11 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();Quoted 3 lines
Quoted 1 line
Wrong. Either
You are right. The mm might go away between
process_shares_mmand here. While yourfind_lock_task_mmwould be correct I believe we can do better by using the existing mm that we already have. I will make it a separate patch to clarity.
Quoted 5 lines
Quoted 2 lines
Still wrong.
p->mm== NULL means that we are too late to setTIF_MEMDIEon that thread. Passing non-NULL mm tomark_oom_victim()won't help.Why would it be too late? Or in other words why would this be harmful?
Setting TIF_MEMDIE after exit_mm() completed is too late.
static void exit_mm(void)
{
(...snipped...)
task_lock(current); current->mm = NULL; up_read(&mm->mmap_sem); enter_lazy_tlb(mm, current); task_unlock(current); mm_update_next_owner(mm); mmput(mm); if (test_thread_flag(TIF_MEMDIE)) exit_oom_victim(); }
Michal Hocko
maintainerRe: [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims
Quoted 24 lines · Tetsuo Handa
Quoted 23 lines · Michal Hocko
Quoted 22 lines · Tetsuo Handa
Quoted 20 lines · Michal Hocko
Quoted 11 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
mm/oom_kill.c +1
-0@@ -898,6 +898,7 @@ static void __oom_kill_process(struct task_struct *victim)if (unlikely (p->flags &PF_KTHREAD ))continue ;do_send_sig_info (SIGKILL ,SEND_SIG_FORCED , p,PIDTYPE_TGID ); +mark_oom_victim (p); }rcu_read_unlock ();Quoted 3 lines
Quoted 1 line
Wrong. Either
You are right. The mm might go away between
process_shares_mmand here. While yourfind_lock_task_mmwould be correct I believe we can do better by using the existing mm that we already have. I will make it a separate patch to clarity.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Still wrong.
p->mm== NULL means that we are too late to setTIF_MEMDIEon that thread. Passing non-NULL mm tomark_oom_victim()won't help.Why would it be too late? Or in other words why would this be harmful?
Setting
TIF_MEMDIEafterexit_mm()completed is too late.
You are right and I am obviously dense today. I will go with find_lock_task_mm for now and push the "get rid of TIF_MEMDIE" up in the todo list. I hope I will get to it some day.
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
maintainerRFC PATCH v2 1/2 mm, oom: marks all killed tasks as oom victims
In reply to Michal HockoUpdated version
Michal Hocko
maintainerRFC PATCH 2/2 memcg: do not report racy no-eligible OOM tasks
In reply to Michal HockoFrom: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and oom_reaper
Thread 1 Thread2 oom_reapertry_charge try_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_process oom_kill_process(current)
wake_oom_reaper
oom_reap_task MMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process # no task
If Thread1 didn't race it would bail out from try_charge and force the charge. We can achieve the same by checking tsk_is_oom_victim inside the oom_lock and therefore close the race.
[1] http://lkml.kernel.org/r/[email protected]
--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
mm/memcontrol.c +13 -1
@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock); + +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; + ret =out_of_memory (&oc); + +unlock:mutex_unlock (&oom_lock);return ret; }
Signature · 1 line
2.19.1
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 44 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +
This is not wrong but is strange. We can use mutex_lock_killable(&oom_lock) so that any killed threads no longer wait for oom_lock.
Also, closing this race for only memcg OOM path is strange. Global OOM path (which are CLONE_VM without CLONE_THREAD) is still suffering this race (though frequency is lower than memcg OOM due to use of mutex_trylock()). Either checking before calling out_of_memory() or checking task_will_free_mem(current) inside out_of_memory() will close this race for both paths.
Quoted 7 lines
mm/memcontrol.c +2
-0ret =out_of_memory (&oc); + +unlock:mutex_unlock (&oom_lock);return ret; }
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 44 lines · Tetsuo Handa
Quoted 43 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;
Quoted 14 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 11 lines
mm/memcontrol.c +10
-0mutex_lock (&oom_lock); + +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.
tsk_is_oom_victim is stronger because it doesn't depend on fatal_signal_pending which might be cleared throughout the exit process.
Quoted 5 lines
Also, closing this race for only memcg OOM path is strange. Global OOM path (which are
CLONE_VMwithoutCLONE_THREAD) is still suffering this race (though frequency is lower than memcg OOM due to use ofmutex_trylock()). Either checking before callingout_of_memory()or checkingtask_will_free_mem(current)insideout_of_memory()will close this race for both paths.
The global case is much more complicated because we know that memcg might bypass the charge so we do not have to care about the potential endless loop like in page allocator path. Moreover I am not even sure the race is all that interesting in the global case. I have never heard about a pre-mature panic due to no killable task. The racing oom task would have to be the last eligible process in the system and that is quite unlikely. We can think about a more involved solution for that if we ever hear about this to be a real problem.
So a simple memcg specific fix sounds like a reasonable way forward.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 60 lines · Michal Hocko
Quoted 44 lines · Tetsuo Handa
Quoted 43 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;Quoted 14 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 11 lines
mm/memcontrol.c +10
-0mutex_lock (&oom_lock); + +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.
Quoted 2 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.
I mean:
mm/memcontrol.c | 3 +- mm/oom_kill.c | 111 +++++--------------------------------------------------- 2 files changed, 12 insertions(+), 102 deletions(-)
mm/memcontrol.c +2 -1
@@ -1382,7 +1382,8 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret; -mutex_lock (&oom_lock); +if (mutex_lock_killable (&oom_lock)) +return true; ret =out_of_memory (&oc);mutex_unlock (&oom_lock);return ret;
mm/oom_kill.c +10 -101
147 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Quoted 10 lines
Quoted 6 lines
Also, closing this race for only memcg OOM path is strange. Global OOM path (which are
CLONE_VMwithoutCLONE_THREAD) is still suffering this race (though frequency is lower than memcg OOM due to use ofmutex_trylock()). Either checking before callingout_of_memory()or checkingtask_will_free_mem(current)insideout_of_memory()will close this race for both paths.The global case is much more complicated because we know that memcg might bypass the charge so we do not have to care about the potential endless loop like in page allocator path.
We know that the page allocator path does not do endless loop unless __GFP_NOFAIL.
Quoted 3 lines
Moreover I am not even sure
the race is all that interesting in the global case. I have never heard about a pre-mature panic due to no killable task.
I don't think this race results in
panic("System is deadlocked on memory\n");. But I think a few unnecessary OOM killing is possible (especially when above patch which "eliminates the shortcuts which call mark_oom_victim() on only one thread group" is not applied).
Quoted 7 lines
The racing oom task
would have to be the last eligible process in the system and that is quite unlikely. We can think about a more involved solution for that if we ever hear about this to be a real problem.So a simple memcg specific fix sounds like a reasonable way forward.
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 61 lines · Tetsuo Handa
Quoted 60 lines · Michal Hocko
Quoted 58 lines · Tetsuo Handa
Quoted 44 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.
Quoted 8 lines
Quoted 2 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.I mean:
mm/memcontrol.c | 3 +- mm/oom_kill.c | 111 +++++--------------------------------------------------- 2 files changed, 12 insertions(+), 102 deletions(-)
This is much larger change than I feel comfortable with to plug this specific issue. A simple and easy to understand fix which doesn't add maintenance burden should be preferred in general.
The code reduction looks attractive but considering it is based on removing one of the heuristics to prevent OOM reports in some case it should be done on its own with a careful and throughout justification. E.g. how often is the heuristic really helpful.
In principle I do not oppose to remove the shortcut after all due diligence is done because this particular one had given us quite a lot headaches in the past.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 7 lines · Michal Hocko
Quoted 5 lines · Tetsuo Handa
I mean:
mm/memcontrol.c | 3 +- mm/oom_kill.c | 111 +++++--------------------------------------------------- 2 files changed, 12 insertions(+), 102 deletions(-)
Quoted 8 lines
This is much larger change than I feel comfortable with to plug this specific issue. A simple and easy to understand fix which doesn't add maintenance burden should be preferred in general.
The code reduction looks attractive but considering it is based on removing one of the heuristics to prevent OOM reports in some case it should be done on its own with a careful and throughout justification. E.g. how often is the heuristic really helpful.
I think the heuristic is hardly helpful.
Regarding task_will_free_mem(current) condition in out_of_memory(), this served for two purposes. One is that mark_oom_victim() is not yet called on current thread group when mark_oom_victim() was already called on other thread groups. But such situation disappears by removing task_will_free_mem() shortcuts and forcing for_each_process(p) loop in __oom_kill_process().
The other is that mark_oom_victim() is not yet called on any thread groups when all thread groups are exiting. In that case, we will fail to wait for current thread group to release its mm... But it is unlikely that only threads which task_will_free_mem(current) returns true can call out_of_memory() (note that task_will_free_mem(p) returns false if p->mm == NULL).
I think it is highly unlikely to hit task_will_free_mem(p) condition in oom_kill_process(). To hit it, the candidate who was chosen due to the largest memory user has to be already exiting. However, if already exiting, it is likely the candidate already released its mm (and hence no longer the largest memory user). I can't say such race never happens, but I think it is unlikely. Also, since task_will_free_mem(p) returns false if thread group leader's mm is NULL whereas oom_badness() from select_bad_process() evaluates any mm in that thread group and returns a thread group leader, this heuristic is incomplete after all.
Quoted 5 lines
In principle I do not oppose to remove the shortcut after all due diligence is done because this particular one had given us quite a lot headaches in the past.
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
In reply to Michal HockoQuoted 27 lines · Michal Hocko
Quoted 11 lines · Tetsuo Handa
Quoted 11 lines
mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;Quoted 14 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 11 lines
mm/memcontrol.c +10
-0mutex_lock (&oom_lock); + +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.
Quoted 3 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.
I still want to propose this. No need to be memcg OOM specific.
mm/memcontrol.c | 3 ++- mm/oom_kill.c | 10 ++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-)
mm/memcontrol.c +2 -1
@@ -1382,7 +1382,8 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret; -mutex_lock (&oom_lock); +if (mutex_lock_killable (&oom_lock)) +return true; ret =out_of_memory (&oc);mutex_unlock (&oom_lock);return ret;
mm/oom_kill.c +10 -0
@@ -1055,6 +1055,16 @@ bool out_of_memory(struct oom_control *oc)unsigned long freed =0 ;enum oom_constraint constraint =CONSTRAINT_NONE ; +/* +* It is possible that multi-threaded OOM victims get +* task_will_free_mem(current) == false when the OOM reaper quickly +* set MMF_OOM_SKIP. But since we know that tsk_is_oom_victim() == true +* tasks won't loop forever (unleess it is a __GFP_NOFAIL allocation +* request), we don't need to select next OOM victim. +*/ +if (tsk_is_oom_victim (current) && !(oc->gfp_mask & __GFP_NOFAIL)) +return true; +if (oom_killer_disabled)return false;
Signature · 1 line
1.8.3.1
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 28 lines · Tetsuo Handa
Quoted 27 lines · Michal Hocko
Quoted 25 lines · Tetsuo Handa
Quoted 12 lines
mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.
Quoted 5 lines
Quoted 3 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.I still want to propose this. No need to be memcg OOM specific.
Well, I maintain what I've said [1] about simplicity and specific fix for a specific issue. Especially in the tricky code like this where all the consequences are far more subtle than they seem to be.
This is obviously a matter of taste but I don't see much point discussing this back and forth for ever. Unless there is a general agreement that the above is less appropriate then I am willing to consider a different change but I simply do not have energy to nit pick for ever.
[1] http://lkml.kernel.org/r/[email protected]
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 39 lines · Michal Hocko
Quoted 28 lines · Tetsuo Handa
Quoted 27 lines · Michal Hocko
Quoted 25 lines · Tetsuo Handa
Quoted 12 lines
mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.Quoted 5 lines
Quoted 3 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.I still want to propose this. No need to be memcg OOM specific.
Well, I maintain what I've said [1] about simplicity and specific fix for a specific issue. Especially in the tricky code like this where all the consequences are far more subtle than they seem to be.
Quoted 6 lines
This is obviously a matter of taste but I don't see much point discussing this back and forth for ever. Unless there is a general agreement that the above is less appropriate then I am willing to consider a different change but I simply do not have energy to nit pick for ever.
In other words. Having a memcg specific fix means, well, a memcg maintenance burden. Like any other memcg specific oom decisions we already have. So are you OK with that Johannes or you would like to see a more generic fix which might turn out to be more complex?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 45 lines · Michal Hocko
Quoted 44 lines · Michal Hocko
Quoted 28 lines · Tetsuo Handa
Quoted 27 lines · Michal Hocko
Quoted 25 lines · Tetsuo Handa
Quoted 12 lines
mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.Quoted 5 lines
Quoted 3 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.I still want to propose this. No need to be memcg OOM specific.
Well, I maintain what I've said [1] about simplicity and specific fix for a specific issue. Especially in the tricky code like this where all the consequences are far more subtle than they seem to be.
This is obviously a matter of taste but I don't see much point discussing this back and forth for ever. Unless there is a general agreement that the above is less appropriate then I am willing to consider a different change but I simply do not have energy to nit pick for ever.
Quoted 7 lines
Quoted 1 line
In other words. Having a memcg specific fix means, well, a memcg maintenance burden. Like any other memcg specific oom decisions we already have. So are you OK with that Johannes or you would like to see a more generic fix which might turn out to be more complex?
I don't know what "that Johannes" refers to.
If you don't want to affect SysRq-OOM and pagefault-OOM cases, are you OK with having a global-OOM specific fix?
mm/page_alloc.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/page_alloc.c +11 -0
@@ -3518,6 +3518,17 @@ void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...)if (gfp_mask & __GFP_THISNODE) goto out; +/* +* It is possible that multi-threaded OOM victims get +* task_will_free_mem(current) == false when the OOM reaper quickly +* set MMF_OOM_SKIP. But since we know that tsk_is_oom_victim() == true +* tasks won't loop forever (unless it is a __GFP_NOFAIL allocation +* request), we don't need to select next OOM victim. +*/ +if (tsk_is_oom_victim (current) && !(gfp_mask & __GFP_NOFAIL)) { + *did_some_progress =1 ; + goto out; + }/* Exhausted what can be done so it's blame time */ if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ;
Signature · 1 line
1.8.3.1
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 48 lines · Tetsuo Handa
Quoted 47 lines · Michal Hocko
Quoted 45 lines · Michal Hocko
Quoted 28 lines · Tetsuo Handa
Quoted 27 lines · Michal Hocko
Quoted 25 lines · Tetsuo Handa
Quoted 12 lines
mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock);mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1395
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; +This is not wrong but is strange. We can use
mutex_lock_killable(&oom_lock)so that any killed threads no longer wait foroom_lock.Quoted 5 lines
Quoted 3 lines
tsk_is_oom_victimis stronger because it doesn't depend onfatal_signal_pendingwhich might be cleared throughout the exit process.I still want to propose this. No need to be memcg OOM specific.
Well, I maintain what I've said [1] about simplicity and specific fix for a specific issue. Especially in the tricky code like this where all the consequences are far more subtle than they seem to be.
This is obviously a matter of taste but I don't see much point discussing this back and forth for ever. Unless there is a general agreement that the above is less appropriate then I am willing to consider a different change but I simply do not have energy to nit pick for ever.
Quoted 7 lines
Quoted 5 lines
In other words. Having a memcg specific fix means, well, a memcg maintenance burden. Like any other memcg specific oom decisions we already have. So are you OK with that Johannes or you would like to see a more generic fix which might turn out to be more complex?
I don't know what "that Johannes" refers to.
let me rephrase
Johannes, are you OK with that (memcg specific fix) or you would like to see a more generic fix which might turn out to be more complex.
Signature · 2 lines
Michal Hocko SUSE Labs
Johannes Weiner
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
In reply to Michal HockoQuoted 45 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +2
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock); +
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1396
Quoted 10 lines
mm/memcontrol.c +9
-0+/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; + ret =out_of_memory (&oc);
We already check tsk_is_oom_victim(current) in try_charge() before looping on the OOM killer, so at most we'd have a single "no eligible tasks" message from such a race before we force the charge - right?
While that's not perfect, I don't think it warrants complicating this code even more. I honestly find it near-impossible to follow the code and the possible scenarios at this point.
out_of_memory() bails on task_will_free_mem(current), which specifically *excludes* already reaped tasks. Why are we then adding a separate check before that to bail on already reaped victims?
Do we want to bail if current is a reaped victim or not?
I don't see how we could skip it safely in general: the current task might have been killed and reaped and gotten access to the memory reserve and still fail to allocate on its way out. It needs to kill the next task if there is one, or warn if there isn't another one. Because we're genuinely oom without reclaimable tasks.
There is of course the scenario brought forward in this thread, where multiple threads of a process race and the second one enters oom even though it doesn't need to anymore. What the global case does to catch this is to grab the oom lock and do one last alloc attempt. Should memcg lock the oom_lock and try one more time to charge the memcg?
Some simplification in this area would really be great. I'm reluctant to ack patches like the above, even if they have some optical benefits for the user, because the code is already too tricky for what it does.
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 44 lines · Johannes Weiner
Quoted 43 lines · Michal Hocko
From: Michal Hocko <[email protected]>
Tetsuo has reported [1] that a single process group memcg might easily swamp the log with no-eligible oom victim reports due to race between the memcg charge and
oom_reaperThread 1 Thread2
oom_reapertry_chargetry_charge
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
mem_cgroup_out_of_memory
mutex_lock(oom_lock)
out_of_memory
select_bad_processoom_kill_process(current)
wake_oom_reaper
oom_reap_taskMMF_OOM_SKIP->victim
mutex_unlock(oom_lock)
out_of_memory
select_bad_process# no taskIf Thread1 didn't race it would bail out from
try_chargeand force the charge. We can achieve the same by checkingtsk_is_oom_victiminside theoom_lockand therefore close the race.[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by Michal Hocko <[email protected]>--- mm/memcontrol.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)mm/memcontrol.c +1
-1@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;
Quoted 16 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1396
Quoted 12 lines
mm/memcontrol.c +10
-0mutex_lock (&oom_lock); + +/* +* multi-threaded tasks might race with oom_reaper and gain +* MMF_OOM_SKIP before reaching out_of_memory which can lead +* to out_of_memory failure if the task is the last one in +* memcg which would be a false possitive failure reported +*/ +if (tsk_is_oom_victim (current)) + goto unlock; + ret =out_of_memory (&oc);We already check
tsk_is_oom_victim(current)intry_charge()before looping on the OOM killer, so at most we'd have a single "no eligible tasks" message from such a race before we force the charge - right?
Not really. You can have many threads blocked on the oom_lock and being reaped while they are waiting. So the check without the lock will always be racy. This is what Tetsuo's test case actually triggers I believe.
Quoted 3 lines
While that's not perfect, I don't think it warrants complicating this code even more. I honestly find it near-impossible to follow the code and the possible scenarios at this point.
I do agree that the code is quite far from easy to follow. The set of events that might happen in a different context is not trivial.
Quoted 3 lines
out_of_memory()bails ontask_will_free_mem(current), which specifically *excludes* already reaped tasks. Why are we then adding a separate check before that to bail on already reaped victims?
696453e66630a has introduced the bail out.
Quoted 7 lines
Do we want to bail if current is a reaped victim or not?
I don't see how we could skip it safely in general: the current task might have been killed and reaped and gotten access to the memory reserve and still fail to allocate on its way out. It needs to kill the next task if there is one, or warn if there isn't another one. Because we're genuinely oom without reclaimable tasks.
Yes, this would be the case for the global case which is a real OOM situation. Memcg oom is somehow more relaxed because the oom is local.
Quoted 5 lines
There is of course the scenario brought forward in this thread, where multiple threads of a process race and the second one enters oom even though it doesn't need to anymore. What the global case does to catch this is to grab the oom lock and do one last alloc attempt. Should memcg lock the
oom_lockand try one more time to charge the memcg?
That would be another option. I agree that making it more towards the global case makes it more attractive. My tsk_is_oom_victim is more towards "plug this particular case".
So does this look better to you?
mm/memcontrol.c +13 -1
@@ -1380,10 +1380,22 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, .gfp_mask = gfp_mask, .order = order, }; -bool ret; +bool ret = true;mutex_lock (&oom_lock); + +/* +* Make the last moment check while we were waiting for the oom_lock +* just in case the oom_reaper could have freed released some +* memory in the meantime. This mimics the lalst moment allocation +* in __alloc_pages_may_oom +*/ +if (mem_cgroup_margin (mem_over_limit) >=1 << order) + goto unlock; + ret =out_of_memory (&oc); + +unlock:mutex_unlock (&oom_lock);return ret; }
Quoted 3 lines
Some simplification in this area would really be great. I'm reluctant to ack patches like the above, even if they have some optical benefits for the user, because the code is already too tricky for what it does.
I am open to different ideas, unless they are crazy timeout here and timeout there wrapped with a duct tape.
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 1 line · Michal Hocko
On Fri 26-10-18 10:25:31, Johannes Weiner wrote:
[...]
Quoted 9 lines
Quoted 5 lines
There is of course the scenario brought forward in this thread, where multiple threads of a process race and the second one enters oom even though it doesn't need to anymore. What the global case does to catch this is to grab the oom lock and do one last alloc attempt. Should memcg lock the
oom_lockand try one more time to charge the memcg?That would be another option. I agree that making it more towards the global case makes it more attractive. My
tsk_is_oom_victimis more towards "plug this particular case".
Nevertheless let me emphasise that tsk_is_oom_victim will close the race completely, while mem_cgroup_margin will always be racy. So the question is whether we want to close the race because it is just too easy for userspace to hit it or keep the global and memcg oom handling as close as possible.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 8 lines · Michal Hocko
Quoted 3 lines
out_of_memory()bails ontask_will_free_mem(current), which specifically *excludes* already reaped tasks. Why are we then adding a separate check before that to bail on already reaped victims?
696453e66630ahas introduced the bail out.Quoted 2 lines
Do we want to bail if current is a reaped victim or not?
Quoted 8 lines
Quoted 5 lines
I don't see how we could skip it safely in general: the current task might have been killed and reaped and gotten access to the memory reserve and still fail to allocate on its way out. It needs to kill the next task if there is one, or warn if there isn't another one. Because we're genuinely oom without reclaimable tasks.
Yes, this would be the case for the global case which is a real OOM situation. Memcg oom is somehow more relaxed because the oom is local.
We can handle possibility of genuinely OOM without reclaimable tasks. Only __GFP_NOFAIL OOM has to select next OOM victim. There is no need to select next OOM victim unless __GFP_NOFAIL. Commit 696453e66630ad45 ("mm, oom: task_will_free_mem should skip oom_reaped tasks") was too simple.
Quoted 13 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Fri 26-10-18 10:25:31, Johannes Weiner wrote:
[...]
Quoted 9 lines
Quoted 5 lines
There is of course the scenario brought forward in this thread, where multiple threads of a process race and the second one enters oom even though it doesn't need to anymore. What the global case does to catch this is to grab the oom lock and do one last alloc attempt. Should memcg lock the
oom_lockand try one more time to charge the memcg?That would be another option. I agree that making it more towards the global case makes it more attractive. My
tsk_is_oom_victimis more towards "plug this particular case".
Quoted 6 lines
Nevertheless let me emphasise that
tsk_is_oom_victimwill close the race completely, whilemem_cgroup_marginwill always be racy. So the question is whether we want to close the race because it is just too easy for userspace to hit it or keep the global and memcg oom handling as close as possible.
Yes, adding tsk_is_oom_victim(current) before calling out_of_memory() from both global OOM and memcg OOM paths can close the race completely. (But note that tsk_is_oom_victim(current) for global OOM path needs to check for __GFP_NOFAIL in order to handle genuinely OOM case.)
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
On Tue 06-11-18 18:44:43, Tetsuo Handa wrote:
[...]
Quoted 10 lines
mm/memcontrol.c +0
-2@@ -1382,8 +1382,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret; -mutex_lock (&oom_lock); - ret =out_of_memory (&oc);
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1394
Quoted 10 lines
mm/memcontrol.c +7
-0+if (mutex_lock_killable (&oom_lock)) +return true; +/* +* A few threads which were not waiting at mutex_lock_killable() can +* fail to bail out. Therefore, check again after holding oom_lock. +*/ + ret =fatal_signal_pending (current) ||out_of_memory (&oc);mutex_unlock (&oom_lock);return ret; }
If we are goging with a memcg specific thingy then I really prefer tsk_is_oom_victim approach. Or is there any reason why this is not suitable?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 10 lines · Michal Hocko
On Tue 06-11-18 18:44:43, Tetsuo Handa wrote:
[...]Quoted 8 lines
mm/memcontrol.c +0
-0@@ -1382,8 +1382,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret;
Quoted 17 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1394
Quoted 12 lines
mm/memcontrol.c +7
-2-mutex_lock (&oom_lock); - ret =out_of_memory (&oc); +if (mutex_lock_killable (&oom_lock)) +return true; +/* +* A few threads which were not waiting at mutex_lock_killable() can +* fail to bail out. Therefore, check again after holding oom_lock. +*/ + ret =fatal_signal_pending (current) ||out_of_memory (&oc);mutex_unlock (&oom_lock);return ret; }If we are goging with a memcg specific thingy then I really prefer
tsk_is_oom_victimapproach. Or is there any reason why this is not suitable?
Why need to wait for mark_oom_victim() called after slow printk() messages?
If current thread got Ctrl-C and thus current thread can terminate, what is nice with waiting for the OOM killer? If there are several OOM events in multiple memcg domains waiting for completion of printk() messages? I don't see points with waiting for oom_lock, for try_charge() already allows current thread to terminate due to fatal_signal_pending() test.
Michal Hocko
maintainerRe: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 29 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
On Tue 06-11-18 18:44:43, Tetsuo Handa wrote:
[...]Quoted 8 lines
mm/memcontrol.c +0
-0@@ -1382,8 +1382,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret;Quoted 17 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1394
Quoted 12 lines
mm/memcontrol.c +7
-2-mutex_lock (&oom_lock); - ret =out_of_memory (&oc); +if (mutex_lock_killable (&oom_lock)) +return true; +/* +* A few threads which were not waiting at mutex_lock_killable() can +* fail to bail out. Therefore, check again after holding oom_lock. +*/ + ret =fatal_signal_pending (current) ||out_of_memory (&oc);mutex_unlock (&oom_lock);return ret; }If we are goging with a memcg specific thingy then I really prefer
tsk_is_oom_victimapproach. Or is there any reason why this is not suitable?
Quoted 7 lines
Why need to wait for
mark_oom_victim()called after slowprintk()messages?If current thread got Ctrl-C and thus current thread can terminate, what is nice with waiting for the OOM killer? If there are several OOM events in multiple memcg domains waiting for completion of
printk()messages? I don't see points with waiting foroom_lock, fortry_charge()already allows current thread to terminate due tofatal_signal_pending()test.
mutex_lock_killable would take care of exiting task already. I would then still prefer to check for mark_oom_victim because that is not racy with the exit path clearing signals. I can update my patch to use _killable lock variant if we are really going with the memcg specific fix.
Johaness?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 38 lines · Michal Hocko
Quoted 29 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
On Tue 06-11-18 18:44:43, Tetsuo Handa wrote:
[...]Quoted 8 lines
mm/memcontrol.c +0
-0@@ -1382,8 +1382,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret;Quoted 17 lines
mm/memcontrol.c static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 1394
Quoted 12 lines
mm/memcontrol.c +7
-2-mutex_lock (&oom_lock); - ret =out_of_memory (&oc); +if (mutex_lock_killable (&oom_lock)) +return true; +/* +* A few threads which were not waiting at mutex_lock_killable() can +* fail to bail out. Therefore, check again after holding oom_lock. +*/ + ret =fatal_signal_pending (current) ||out_of_memory (&oc);mutex_unlock (&oom_lock);return ret; }If we are goging with a memcg specific thingy then I really prefer
tsk_is_oom_victimapproach. Or is there any reason why this is not suitable?Quoted 7 lines
Why need to wait for
mark_oom_victim()called after slowprintk()messages?If current thread got Ctrl-C and thus current thread can terminate, what is nice with waiting for the OOM killer? If there are several OOM events in multiple memcg domains waiting for completion of
printk()messages? I don't see points with waiting foroom_lock, fortry_charge()already allows current thread to terminate due tofatal_signal_pending()test.
Quoted 8 lines
mutex_lock_killablewould take care of exiting task already. I would then still prefer to check formark_oom_victimbecause that is not racy with the exit path clearing signals. I can update my patch to use_killablelock variant if we are really going with the memcg specific fix.Johaness?
No response for one month. When can we get to an RCU stall problem syzbot reported?
Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks
Quoted 1 line · Tetsuo Handa
No response for one month. When can we get to an RCU stall problem syzbot reported?
Why not to apply this patch and then think how to address
https://lore.kernel.org/lkml/[email protected]/ ?
Quoted 1 line
From
0fb58415770a83d6c40d471e1840f8bc4a35ca83Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <[email protected]>
Date: Wed, 12 Dec 2018 19:15:51 +0900
Subject: [PATCH] memcg: killed threads should not invoke memcg OOM killer
It is possible that a single process group memcg easily swamps the log with no-eligible OOM victim messages after current thread was OOM-killed, due to race between the memcg charge and the OOM reaper [1].
Thread-1 Thread-2 OOM reapertry_charge()
mem_cgroup_out_of_memory()
mutex_lock(oom_lock)
try_charge()
mem_cgroup_out_of_memory()
mutex_lock(oom_lock)
out_of_memory()
select_bad_process()
oom_kill_process(current)
wake_oom_reaper()
oom_reap_task()
# sets MMF_OOM_SKIP
mutex_unlock(oom_lock)
out_of_memory()
select_bad_process() # no task
mutex_unlock(oom_lock)We don't need to invoke the memcg OOM killer if current thread was killed when waiting for oom_lock, for mem_cgroup_oom_synchronize(true) and memory_max_write() can bail out upon SIGKILL, and try_charge() allows already killed/exiting threads to make forward progress.
Michal has a plan to use tsk_is_oom_victim() by calling mark_oom_victim() on all thread groups sharing victim's mm. But fatal_signal_pending() in this patch helps regardless of Michal's plan because it will avoid needlessly calling out_of_memory() when current thread is already terminating (e.g. got SIGINT after passing fatal_signal_pending() check in try_charge() and mutex_lock_killable() did not block).
[1] https://lkml.kernel.org/r/[email protected]
--- mm/memcontrol.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
mm/memcontrol.c +7 -2
@@ -1389,8 +1389,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, };bool ret; -mutex_lock (&oom_lock); - ret =out_of_memory (&oc); +if (mutex_lock_killable (&oom_lock)) +return true; +/* +* A few threads which were not waiting at mutex_lock_killable() can +* fail to bail out. Therefore, check again after holding oom_lock. +*/ + ret =fatal_signal_pending (current) ||out_of_memory (&oc);mutex_unlock (&oom_lock);return ret; }
Signature · 1 line
1.8.3.1