mm/oom_kill: show oom eligibility when displaying the current memory state of all tasks
Subsystems (4)
PATCH v2 1/1 mm/oom_kill: show oom eligibility when displaying the current memory state of all tasks
Changes since v2:
- Use single character (e.g. 'R' for MMF_OOM_SKIP) as suggested by Tetsuo Handa <[email protected]>
- Add new header to oom_dump_tasks documentation
At the present time, when showing potential OOM victims, we do not exclude tasks which already have MMF_OOM_SKIP set; it is possible that the last OOM killable victim was already OOM killed, yet the OOM reaper failed to reclaim memory and set MMF_OOM_SKIP.
This can be confusing/or perhaps even misleading, to the reader of the OOM report. Now, we already unconditionally display a task's oom_score_adj_min value that can be set to OOM_SCORE_ADJ_MIN which is indicative of an "unkillable" task i.e. is not eligible.
This patch provides a clear indication with regard to the OOM eligibility of each displayed task.
--- Documentation/admin-guide/sysctl/vm.rst | 5 ++-- mm/oom_kill.c | 31 +++++++++++++++++++++---- 2 files changed, 30 insertions(+), 6 deletions(-)
Documentation/admin-guide/sysctl/vm.rst +3 -2
@@ -658,8 +658,9 @@ oom_dump_tasks Enables a system-wide task dump (excluding kernel threads) to be produced when the kernel performs an OOM-killing and includes such information as pid, uid, tgid, vm size, rss, pgtables_bytes, swapents, oom_score_adj -score, and name. This is helpful to determine why the OOM killer was -invoked, to identify the rogue task that caused it, and to determine why +score, oom eligibility status and name. This is helpful to determine why +the OOM killer was invoked, to identify the rogue task that caused it, and +to determine why the OOM killer chose the task it did to kill. If this is set to zero, this information is suppressed. On very
mm/oom_kill.c +27 -4
@@ -160,6 +160,27 @@ static inline bool is_sysrq_oom(struct oom_control *oc)return oc->order ==-1 ; } +/** +* is_task_eligible_oom - determine if and why a task cannot be OOM killed +* @tsk: task to check +* +* Needs to be called with task_lock(). +*/ +static const char *is_task_oom_eligible (struct task_struct *p) +{ +long adj; + + adj = (long )p->signal->oom_score_adj; +if (adj ==OOM_SCORE_ADJ_MIN ) +return "S" ; + else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ; +else if (in_vfork (p)) +return "V" ; +else +return "" ; +} +/* return true if the task is not adequate as candidate victim task. */ static bool oom_unkillable_task (struct task_struct *p) {
@@ -401,12 +422,13 @@ static int dump_task(struct task_struct *p, void *arg)return 0 ; } -pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n" , +pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %13s %s\n" , task->pid,from_kuid (&init_user_ns,task_uid (task)), task->tgid, task->mm->total_vm,get_mm_rss (task->mm),mm_pgtables_bytes (task->mm),get_mm_counter (task->mm,MM_SWAPENTS ), - task->signal->oom_score_adj, task->comm); + task->signal->oom_score_adj,is_task_oom_eligible (task), + task->comm);task_unlock (task);return 0 ;
@@ -420,12 +442,13 @@ static int dump_task(struct task_struct *p, void *arg)* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes * are not shown. * State information includes task's pid, uid, tgid, vm size, rss, -* pgtables_bytes, swapents, oom_score_adj value, and name. +* pgtables_bytes, swapents, oom_score_adj value, oom eligibility status +* and name. */ static void dump_tasks (struct oom_control *oc) {pr_info ("Tasks state (memory values in pages):\n" ); -pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name\n" ); +pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom eligible? name\n" );if (is_memcg_oom (oc))mem_cgroup_scan_tasks (oc->memcg, dump_task, oc);
Signature · 1 line
2.26.3
David Rientjes
reviewerQuoted 5 lines · Aaron Tomlin
mm/oom_kill.c +0
-0
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 169
Quoted 10 lines
mm/oom_kill.c +7
-0@@ -160,6 +160,27 @@ static inline bool is_sysrq_oom(struct oom_control *oc)return oc->order ==-1 ; } +/** +* is_task_eligible_oom - determine if and why a task cannot be OOM killed +* @tsk: task to check +* +* Needs to be called with task_lock(). +*/ +static const char *is_task_oom_eligible (struct task_struct *p)
You should be able to just return a char.
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 175
Quoted 6 lines
mm/oom_kill.c +6
-0+{ +long adj; + + adj = (long )p->signal->oom_score_adj; +if (adj ==OOM_SCORE_ADJ_MIN ) +return "S" ;
The value is already printed in the task dump, this doesn't look to add any information.
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 177
Quoted 2 lines
mm/oom_kill.c +2
-0+ else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ;
We should be doing the task dump only when we're killing a victim (unless we're panicking), so something else has been chosen. Since we would have oom killed a process with MMF_OOM_SKIP already, can we simply choose to not print a line for this process?
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 179
Quoted 2 lines
mm/oom_kill.c +2
-0+else if (in_vfork (p)) +return "V" ;
Is this a transition state that we can simply disregard from the output as well unless/until it becomes eligible?
mm/oom_kill.c static int dump_task(struct task_struct *p, void *arg) · line 425
Quoted 13 lines
mm/oom_kill.c +5
-1+else +return "" ; +} +/* return true if the task is not adequate as candidate victim task. */ static bool oom_unkillable_task (struct task_struct *p) {@@ -401,12 +422,13 @@ static int dump_task(struct task_struct *p, void *arg)return 0 ; } - pr_info("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n", + pr_info("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %13s %s\n",
13 characters for one char output?
Quoted 13 lines
mm/oom_kill.c +2
-1task->pid,from_kuid (&init_user_ns,task_uid (task)), task->tgid, task->mm->total_vm,get_mm_rss (task->mm),mm_pgtables_bytes (task->mm),get_mm_counter (task->mm,MM_SWAPENTS ), - task->signal->oom_score_adj, task->comm); + task->signal->oom_score_adj,is_task_oom_eligible (task), + task->comm);task_unlock (task);return 0 ;@@ -420,12 +442,13 @@ static int dump_task(struct task_struct *p, void *arg)* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes * are not shown.
mm/oom_kill.c static int dump_task(struct task_struct *p, void *arg) · line 451
Quoted 10 lines
mm/oom_kill.c +3
-2* State information includes task's pid, uid, tgid, vm size, rss, -* pgtables_bytes, swapents, oom_score_adj value, and name. +* pgtables_bytes, swapents, oom_score_adj value, oom eligibility status +* and name. */ static void dump_tasks (struct oom_control *oc) {pr_info ("Tasks state (memory values in pages):\n" ); -pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name\n" ); +pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom eligible? name\n" );
Field names are single words.
Quoted 3 lines
mm/oom_kill.c +0
-0if (is_memcg_oom (oc))mem_cgroup_scan_tasks (oc->memcg, dump_task, oc);
Quoted 10 lines · David Rientjes
Quoted 8 lines · Aaron Tomlin
mm/oom_kill.c +0
-0@@ -160,6 +160,27 @@ static inline bool is_sysrq_oom(struct oom_control *oc)return oc->order ==-1 ; }
Quoted 9 lines
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 169
Quoted 7 lines
mm/oom_kill.c +7
-0+/** +* is_task_eligible_oom - determine if and why a task cannot be OOM killed +* @tsk: task to check +* +* Needs to be called with task_lock(). +*/ +static const char *is_task_oom_eligible (struct task_struct *p)You should be able to just return a char.
I see, sure.
Quoted 9 lines
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 175
Quoted 6 lines
mm/oom_kill.c +6
-0+{ +long adj; + + adj = (long )p->signal->oom_score_adj; +if (adj ==OOM_SCORE_ADJ_MIN ) +return "S" ;The value is already printed in the task dump, this doesn't look to add any information.
I understand and perhaps it does not make sense; albeit, the reader might not understand the meaning of OOM_SCORE_ADJ_MIN.
Quoted 7 lines
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 177
Quoted 2 lines
mm/oom_kill.c +2
-0+ else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ;We should be doing the task dump only when we're killing a victim (unless we're panicking), so something else has been chosen. Since we would have oom killed a process with
MMF_OOM_SKIPalready, can we simply choose to not print a line for this process?
I'd prefer not to show such tasks, when displaying potential OOM victims (including those in_vfork()) as in my opinion, it can be misleading to the reader. That said, a case has been made to maintain their inclusion. However, should in_vfork() even be shown in the actual report?
Quoted 9 lines
mm/oom_kill.c static int dump_task(struct task_struct *p, void *arg) · line 425
Quoted 7 lines
mm/oom_kill.c +1
-1@@ -401,12 +422,13 @@ static int dump_task(struct task_struct *p, void *arg)return 0 ; } - pr_info("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n", + pr_info("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %13s %s\n",13 characters for one char output?
This was to maintain some alignment but fair enough.
Quoted 7 lines
mm/oom_kill.c static int dump_task(struct task_struct *p, void *arg) · line 451
Quoted 5 lines
mm/oom_kill.c +1
-1static void dump_tasks (struct oom_control *oc) {pr_info ("Tasks state (memory values in pages):\n" ); -pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name\n" ); +pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom eligible? name\n" );Field names are single words.
Understood.
Kind regards,
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 6 lines · Aaron Tomlin
Changes since v2:
- Use single character (e.g. 'R' forMMF_OOM_SKIP) as suggested by Tetsuo Handa <[email protected]>
- Add new header tooom_dump_tasksdocumentation
Quoted 8 lines
At the present time, when showing potential OOM victims, we do not exclude tasks which already have
MMF_OOM_SKIPset; it is possible that the last OOM killable victim was already OOM killed, yet the OOM reaper failed to reclaim memory and setMMF_OOM_SKIP.
This can be confusing/or perhaps even misleading, to the reader of the OOM report. Now, we already unconditionally display a task'soom_score_adj_minvalue that can be set toOOM_SCORE_ADJ_MINwhich is indicative of an "unkillable" task i.e. is not eligible.
Well, I have to say that I have a bit hard time understand the problem statement here. First of all you are very likely basing your observation on an old kernel which is missing a fix which should make the situation impossible IIRC. You should be focusing on a justification why the new information is helpful for the current tree.
Historically, all tasks eligible for the oom killing have been printed. That includes also tasks which are excluded later in the selection. E.g. OOMS_SCORE_ADJ_MIN which can be tricky indeed. IIRC the primary reason was to have a sufficient amount of information to evaluate whether the system is configured properly (e.g. OOMS_SCORE_ADJ_MIN is not used too extensively). More excluded criterion have been added due to implementation details (e.g.MMF_OOM_SKIP or mm shared with otherwise ineligible task.
You are correctly pointing out that those internal states are not exposed but you should focus on explanation why that gap really stands in the way for the current upstream. Who is going to consume that information and for what purpose?
Quoted 2 lines
This patch provides a clear indication with regard to the OOM eligibility of each displayed task.
This should provide an example of the output with a clarification of the meaning.
[...]
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 5 lines · Michal Hocko
Well, I have to say that I have a bit hard time understand the problem statement here. First of all you are very likely basing your observation on an old kernel which is missing a fix which should make the situation impossible IIRC. You should be focusing on a justification why the new information is helpful for the current tree.
Michal,
Not exactly.
See oom_reap_task(). Let's assume an OOM event occurred within the context of a memcg and 'memory.oom.group' was not set. If I understand correctly, once all attempts to OOM reap the specified task were "unsuccessful" then MMF_OOM_SKIP is applied; and, the assumption is it will be terminated shorty due to the pending fatal signal (see __oom_kill_process()) i.e. a SIGKILL is sent to the "victim" before the OOM reaper is notified. Now assuming the above task did not exited yet, another task, in the same memcg, could also trigger an OOM event. Therefore, when showing potential OOM victims the task above with MMF_OOM_SKIP set, will indeed be displayed.
I understanding the point on OOM_SCORE_ADJ_MIN. This can be easily identified and is clear to the viewer. However, the same cannot be stated for MMF_OOM_SKIP.
So, if we prefer to display rather than exclude such tasks, in my opinion having a flag/or marker of some description might prove useful to avoid any misunderstanding.
Quoted 2 lines
This should provide an example of the output with a clarification of the meaning.
Fair enough.
Kind regards,
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 21 lines · Aaron Tomlin
Quoted 5 lines · Michal Hocko
Well, I have to say that I have a bit hard time understand the problem statement here. First of all you are very likely basing your observation on an old kernel which is missing a fix which should make the situation impossible IIRC. You should be focusing on a justification why the new information is helpful for the current tree.
Michal,
Not exactly.
See
oom_reap_task(). Let's assume an OOM event occurred within the context of a memcg and 'memory.oom.group' was not set. If I understand correctly, once all attempts to OOM reap the specified task were "unsuccessful" thenMMF_OOM_SKIPis applied; and, the assumption is it will be terminated shorty due to the pending fatal signal (see__oom_kill_process()) i.e. a SIGKILL is sent to the "victim" before the OOM reaper is notified. Now assuming the above task did not exited yet, another task, in the same memcg, could also trigger an OOM event. Therefore, when showing potential OOM victims the task above withMMF_OOM_SKIPset, will indeed be displayed.
Quoted 3 lines
I understanding the point on
OOM_SCORE_ADJ_MIN. This can be easily identified and is clear to the viewer. However, the same cannot be stated forMMF_OOM_SKIP.
This is all true but it is not really clear why that is really a problem. Kernel log already contains information about reaped processes as they are reported to the log. I fully acknowledge that this is rather spartan but on the other hand from years of experience reading oom reports I have to say the dump_tasks is the least interesting part of the report (while being the most verbose one).
All that being said, I am not really opposing extending the information although I am a bit worried about leaking too much internal state to the log. What I am asking for here is a justification why this addition is a general improvement and how it helps uderstanding oom reports further. So please focus on that part.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 6 lines · Michal Hocko
This is all true but it is not really clear why that is really a problem. Kernel log already contains information about reaped processes as they are reported to the log. I fully acknowledge that this is rather spartan but on the other hand from years of experience reading oom reports I have to say the
dump_tasksis the least interesting part of the report (while being the most verbose one).
I understand. I suppose, in a situation whereby dump_tasks() output is only available, for whatever reason, it can provide at least some insight into what tasks were actually considered not OOM eligible and why.
Quoted 3 lines
All that being said, I am not really opposing extending the information although I am a bit worried about leaking too much internal state to the log.
Fair enough. That said, I still feel highlighting such "ineligible" tasks could be useful to the viewer for troubleshooting purposes; we already display OOM_SCORE_ADJ_MIN. Consider a situation when only a few tasks in a memcg are displayed as possibly OOM eligible but one had MMF_OOM_SKIP applied.
In my opinion, perhaps it is better to just exclude such details altogether. That being said, as you know, we only provide this facility when one is interested in it anyway i.e., if oom_dump_tasks is enabled.
Quoted 3 lines
What I am asking for here is a justification why this addition is a general improvement and how it helps uderstanding oom reports further. So please focus on that part.
Sure; albeit, thinking about this more, it does not provide much understanding in simple isolation.
Kind regards,
Signature · 1 line
Aaron Tomlin