mm/oom_kill: show oom eligibility when displaying the current memory state of all tasks
Subsystems (4)
PATCH v3 1/1 mm/oom_kill: show oom eligibility when displaying the current memory state of all tasks
Changes since v2:
- Use single character (e.g. 'R' for MMF_OOM_SKIP) as suggested by Tetsuo Handa <[email protected]>
- Add new header to oom_dump_tasks documentation
- Provide further justification
The output generated by dump_tasks() can be helpful to determine why there was an OOM condition and which rogue task potentially caused it. Please note that this is only provided when sysctl oom_dump_tasks is enabled.
At the present time, when showing potential OOM victims, we do not exclude any task that are not OOM eligible e.g. those that have MMF_OOM_SKIP set; it is possible that the last OOM killable victim was already OOM killed, yet the OOM reaper failed to reclaim memory and set MMF_OOM_SKIP. This can be confusing (or perhaps even be misleading) to the viewer. Now, we already unconditionally display a task's oom_score_adj_min value that can be set to OOM_SCORE_ADJ_MIN which is indicative of an "unkillable" task.
This patch provides a clear indication with regard to the OOM ineligibility (and why) of each displayed task with the addition of a new column namely "oom_skipped". An example is provided below:
[ 5084.524970] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom_skipped name
[ 5084.526397] [660417] 0 660417 35869 683 167936 0 -1000 M conmon
[ 5084.526400] [660452] 0 660452 175834 472 86016 0 -998 pod
[ 5084.527460] [752415] 0 752415 35869 650 172032 0 -1000 M conmon
[ 5084.527462] [752575] 1001050000 752575 184205 11158 700416 0 999 npm
[ 5084.527467] [753606] 1001050000 753606 183380 46843 2134016 0 999 node
[ 5084.527581] Memory cgroup out of memory: Killed process 753606 (node) total-vm:733520kB, anon-rss:161228kB, file-rss:26144kB, shmem-rss:0kB, UID:1001050000So, a single character 'M' is for OOM_SCORE_ADJ_MIN, 'R' MMF_OOM_SKIP and 'V' for in_vfork().
--- Documentation/admin-guide/sysctl/vm.rst | 5 ++-- mm/oom_kill.c | 31 +++++++++++++++++++++---- 2 files changed, 30 insertions(+), 6 deletions(-)
Documentation/admin-guide/sysctl/vm.rst +3 -2
@@ -650,8 +650,9 @@ oom_dump_tasks Enables a system-wide task dump (excluding kernel threads) to be produced when the kernel performs an OOM-killing and includes such information as pid, uid, tgid, vm size, rss, pgtables_bytes, swapents, oom_score_adj -score, and name. This is helpful to determine why the OOM killer was -invoked, to identify the rogue task that caused it, and to determine why +score, oom eligibility status and name. This is helpful to determine why +the OOM killer was invoked, to identify the rogue task that caused it, and +to determine why the OOM killer chose the task it did to kill. If this is set to zero, this information is suppressed. On very
mm/oom_kill.c +27 -4
@@ -160,6 +160,27 @@ static inline bool is_sysrq_oom(struct oom_control *oc)return oc->order ==-1 ; } +/** +* is_task_eligible_oom - determine if and why a task cannot be OOM killed +* @tsk: task to check +* +* Needs to be called with task_lock(). +*/ +static const char *const is_task_oom_eligible (struct task_struct *p) +{ +long adj; + + adj = (long )p->signal->oom_score_adj; +if (adj ==OOM_SCORE_ADJ_MIN ) +return "M" ; + else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ; +else if (in_vfork (p)) +return "V" ; +else +return "" ; +} +/* return true if the task is not adequate as candidate victim task. */ static bool oom_unkillable_task (struct task_struct *p) {
@@ -401,12 +422,13 @@ static int dump_task(struct task_struct *p, void *arg)return 0 ; } -pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n" , +pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %1s %s\n" , task->pid,from_kuid (&init_user_ns,task_uid (task)), task->tgid, task->mm->total_vm,get_mm_rss (task->mm),mm_pgtables_bytes (task->mm),get_mm_counter (task->mm,MM_SWAPENTS ), - task->signal->oom_score_adj, task->comm); + task->signal->oom_score_adj,is_task_oom_eligible (task), + task->comm);task_unlock (task);return 0 ;
@@ -420,12 +442,13 @@ static int dump_task(struct task_struct *p, void *arg)* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes * are not shown. * State information includes task's pid, uid, tgid, vm size, rss, -* pgtables_bytes, swapents, oom_score_adj value, and name. +* pgtables_bytes, swapents, oom_score_adj value, oom eligibility status +* and name. */ static void dump_tasks (struct oom_control *oc) {pr_info ("Tasks state (memory values in pages):\n" ); -pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name\n" ); +pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom_skipped name\n" );if (is_memcg_oom (oc))mem_cgroup_scan_tasks (oc->memcg, dump_task, oc);
Signature · 1 line
2.31.1
Andrew Morton
maintainerQuoted 40 lines · On Fri, 30 Jul 2021 17:20:02 +0100 Aaron Tomlin <[email protected]> wrote:
Changes since v2:
- Use single character (e.g. 'R' forMMF_OOM_SKIP) as suggested by Tetsuo Handa <[email protected]>
- Add new header tooom_dump_tasksdocumentation
- Provide further justificationThe output generated by
dump_tasks()can be helpful to determine why there was an OOM condition and which rogue task potentially caused it. Please note that this is only provided when sysctloom_dump_tasksis enabled.At the present time, when showing potential OOM victims, we do not exclude any task that are not OOM eligible e.g. those that have
MMF_OOM_SKIPset; it is possible that the last OOM killable victim was already OOM killed, yet the OOM reaper failed to reclaim memory and setMMF_OOM_SKIP. This can be confusing (or perhaps even be misleading) to the viewer. Now, we already unconditionally display a task'soom_score_adj_minvalue that can be set toOOM_SCORE_ADJ_MINwhich is indicative of an "unkillable" task.This patch provides a clear indication with regard to the OOM ineligibility (and why) of each displayed task with the addition of a new column namely "
oom_skipped". An example is provided below:[ 5084.524970] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom_skipped name [ 5084.526397] [660417] 0 660417 35869 683 167936 0 -1000 M conmon [ 5084.526400] [660452] 0 660452 175834 472 86016 0 -998 pod [ 5084.527460] [752415] 0 752415 35869 650 172032 0 -1000 M conmon [ 5084.527462] [752575] 1001050000 752575 184205 11158 700416 0 999 npm [ 5084.527467] [753606] 1001050000 753606 183380 46843 2134016 0 999 node [ 5084.527581] Memory cgroup out of memory: Killed process 753606 (node) total-vm:733520kB, anon-rss:161228kB, file-rss:26144kB, shmem-rss:0kB, UID:1001050000So, a single character 'M' is for
OOM_SCORE_ADJ_MIN, 'R'MMF_OOM_SKIPand 'V' forin_vfork().index
003d5cc3751b..4c79fa00ddb3100644
Documentation/admin-guide/sysctl/vm.rst oom_dump_tasks · line 655
Quoted 8 lines
Documentation/admin-guide/sysctl/vm.rst +3
-2@@ -650,8 +650,9 @@ oom_dump_tasks Enables a system-wide task dump (excluding kernel threads) to be produced when the kernel performs an OOM-killing and includes such information as pid, uid, tgid, vm size, rss, pgtables_bytes, swapents, oom_score_adj -score, and name. This is helpful to determine why the OOM killer was -invoked, to identify the rogue task that caused it, and to determine why +score, oom eligibility status and name. This is helpful to determine why +the OOM killer was invoked, to identify the rogue task that caused it, and +to determine why
It would be better if the meaning of 'M', 'R' and 'V' were described here.
Documentation/admin-guide/sysctl/vm.rst oom_dump_tasks
Quoted 9 lines
Documentation/admin-guide/sysctl/vm.rst +7
-0the OOM killer chose the task it did to kill. +/** + * is_task_eligible_oom - determine if and why a task cannot be OOM killed + * @tsk: task to check + * + * Needs to be called with task_lock(). + */ +static const char * const is_task_oom_eligible(struct task_struct *p)
Name seems inappropriate. task_oom_eligibility()?
Quoted 13 lines
Documentation/admin-guide/sysctl/vm.rst +13
-0+{ + long adj; + + adj = (long)p->signal->oom_score_adj; + if (adj == OOM_SCORE_ADJ_MIN) + return "M"; + else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) + return "R"; + else if (in_vfork(p)) + return "V"; + else + return ""; +}
David Rientjes
reviewerQuoted 31 lines · Aaron Tomlin
Documentation/admin-guide/sysctl/vm.rst | 5 ++-- mm/oom_kill.c | 31 +++++++++++++++++++++---- 2 files changed, 30 insertions(+), 6 deletions(-)Documentation/admin-guide/sysctl/vm.rst +3
-2@@ -650,8 +650,9 @@ oom_dump_tasks Enables a system-wide task dump (excluding kernel threads) to be produced when the kernel performs an OOM-killing and includes such information as pid, uid, tgid, vm size, rss, pgtables_bytes, swapents, oom_score_adj -score, and name. This is helpful to determine why the OOM killer was -invoked, to identify the rogue task that caused it, and to determine why +score, oom eligibility status and name. This is helpful to determine why +the OOM killer was invoked, to identify the rogue task that caused it, and +to determine why the OOM killer chose the task it did to kill. If this is set to zero, this information is suppressed. On verymm/oom_kill.c +3
-0@@ -160,6 +160,27 @@ static inline bool is_sysrq_oom(struct oom_control *oc)return oc->order ==-1 ; } +/** +* is_task_eligible_oom - determine if and why a task cannot be OOM killed +* @tsk: task to check
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 175
Quoted 10 lines
mm/oom_kill.c +10
-0+* +* Needs to be called with task_lock(). +*/ +static const char *const is_task_oom_eligible (struct task_struct *p) +{ +long adj; + + adj = (long )p->signal->oom_score_adj; +if (adj ==OOM_SCORE_ADJ_MIN ) +return "M" ;
oom_score_adj is shown already in the tasklist dump, I'm not sure what value this adds.
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 179
Quoted 4 lines
mm/oom_kill.c +4
-0+ else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ; +else if (in_vfork (p)) +return "V" ;
This is going to be racy, we can't show that a task that is emitted as part of the tasklist dump was did not have in_vfork() == true at the time oom_badness() was called.
Wouldn't it be better to simply print the output of oom_badness() to the tasklist dump instead so we get complete information?
We could simply special case a LONG_MIN return value as -1000 or "min".
Quoted 43 lines
mm/oom_kill.c +10
-4+else +return "" ; +} +/* return true if the task is not adequate as candidate victim task. */ static bool oom_unkillable_task (struct task_struct *p) {@@ -401,12 +422,13 @@ static int dump_task(struct task_struct *p, void *arg)return 0 ; } -pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n" , +pr_info ("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %1s %s\n" , task->pid,from_kuid (&init_user_ns,task_uid (task)), task->tgid, task->mm->total_vm,get_mm_rss (task->mm),mm_pgtables_bytes (task->mm),get_mm_counter (task->mm,MM_SWAPENTS ), - task->signal->oom_score_adj, task->comm); + task->signal->oom_score_adj,is_task_oom_eligible (task), + task->comm);task_unlock (task);return 0 ;@@ -420,12 +442,13 @@ static int dump_task(struct task_struct *p, void *arg)* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes * are not shown. * State information includes task's pid, uid, tgid, vm size, rss, -* pgtables_bytes, swapents, oom_score_adj value, and name. +* pgtables_bytes, swapents, oom_score_adj value, oom eligibility status +* and name. */ static void dump_tasks (struct oom_control *oc) {pr_info ("Tasks state (memory values in pages):\n" ); -pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name\n" ); +pr_info ("[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom_skipped name\n" );if (is_memcg_oom (oc))mem_cgroup_scan_tasks (oc->memcg, dump_task, oc);Signature · 1 line
2.31.1
Quoted 2 lines · David Rientjes
oom_score_adjis shown already in the tasklist dump, I'm not sure what value this adds.
Fair enough.
Quoted 9 lines
mm/oom_kill.c static inline bool is_sysrq_oom(struct oom_control *oc) · line 179
Quoted 5 lines
mm/oom_kill.c +4
-0+ else if (test_bit(MMF_OOM_SKIP, &p->mm->flags) +return "R" ; +else if (in_vfork (p)) +return "V" ;This is going to be racy, we can't show that a task that is emitted as part of the tasklist dump was did not have
in_vfork()== true at the timeoom_badness()was called.
Yes, this is true.
Quoted 2 lines
Wouldn't it be better to simply print the output of
oom_badness()to the tasklist dump instead so we get complete information?
I think this would be acceptable.
Quoted 1 line
We could simply special case a
LONG_MINreturn value as -1000 or "min".
Agreed.
Kind regards,
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 7 lines · Aaron Tomlin
Changes since v2:
- Use single character (e.g. 'R' forMMF_OOM_SKIP) as suggested by Tetsuo Handa <[email protected]>
- Add new header tooom_dump_tasksdocumentation
- Provide further justification
Quoted 7 lines
The output generated by
dump_tasks()can be helpful to determine why there was an OOM condition and which rogue task potentially caused it. Please note that this is only provided when sysctloom_dump_tasksis enabled.At the present time, when showing potential OOM victims, we do not exclude any task that are not OOM eligible e.g.
Well, this is not precise. We do exclude ineligible. Consider tasks that are outside of the OOM domain for example. You are right that we are not excluding all of them though.
Quoted 20 lines
those that have
MMF_OOM_SKIPset; it is possible that the last OOM killable victim was already OOM killed, yet the OOM reaper failed to reclaim memory and setMMF_OOM_SKIP. This can be confusing (or perhaps even be misleading) to the viewer. Now, we already unconditionally display a task'soom_score_adj_minvalue that can be set toOOM_SCORE_ADJ_MINwhich is indicative of an "unkillable" task.This patch provides a clear indication with regard to the OOM ineligibility (and why) of each displayed task with the addition of a new column namely "
oom_skipped". An example is provided below:[ 5084.524970] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj oom_skipped name [ 5084.526397] [660417] 0 660417 35869 683 167936 0 -1000 M conmon [ 5084.526400] [660452] 0 660452 175834 472 86016 0 -998 pod [ 5084.527460] [752415] 0 752415 35869 650 172032 0 -1000 M conmon [ 5084.527462] [752575] 1001050000 752575 184205 11158 700416 0 999 npm [ 5084.527467] [753606] 1001050000 753606 183380 46843 2134016 0 999 node [ 5084.527581] Memory cgroup out of memory: Killed process 753606 (node) total-vm:733520kB, anon-rss:161228kB, file-rss:26144kB, shmem-rss:0kB, UID:1001050000
Quoted 2 lines
So, a single character 'M' is for
OOM_SCORE_ADJ_MIN, 'R'MMF_OOM_SKIPand 'V' forin_vfork().
I have to say I dislike this for two reasons. First and foremost it makes parsing unnecessarily more complex. Now you have a potentially an empty column to special case. Secondly MMF_OOM_SKIP is an internal state that shouldn't really leak to userspace IMO. in_vfork is a racy check and M is already expressed via oom_score_adj so it duplicates an existing information.
If you really want/need to make any change here then I would propose to either add E(eligible)/I(ligible) column without any specifics or consistently skip over all tasks which are not eligible.
Quoted 1 line
2.31.1
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 3 lines · Michal Hocko
If you really want/need to make any change here then I would propose to either add E(eligible)/I(ligible) column without any specifics or consistently skip over all tasks which are not eligible.
How about the suggestion made by David i.e. exposing the value returned by oom_badness(), as if I understand correctly, this would provide a more complete picture with regard to the rationale used?
Kind regards,
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 8 lines · Aaron Tomlin
Quoted 3 lines · Michal Hocko
If you really want/need to make any change here then I would propose to either add E(eligible)/I(ligible) column without any specifics or consistently skip over all tasks which are not eligible.
How about the suggestion made by David i.e. exposing the value returned by
oom_badness(), as if I understand correctly, this would provide a more complete picture with regard to the rationale used?
There were some attempts to print oom_score during OOM. E.g.
http://lkml.kernel.org/r/[email protected]. That one was rejected on the grounds that the number on its own doesn't really present any real value. It is really only valuable when comparing to other potential oom victims. I have to say I am still worried about printing this internal scoring as it should have really been an implementation detail but with /proc/<pid>/oom_score this is likely a lost battle and I am willing to give up on that front.
I am still not entirely convinced this is worth doing though. oom_badness is not a cheap operation. task_lock has to be taken again during dump_tasks for each task so the already quite expensive operation will be more so. Is this really something we cannot live without?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 8 lines · Michal Hocko
There were some attempts to print
oom_scoreduring OOM. E.g.
http://lkml.kernel.org/r/[email protected]. That one was rejected on the grounds that the number on its own doesn't really present any real value. It is really only valuable when comparing to other potential oom victims. I have to say I am still worried about printing this internal scoring as it should have really been an implementation detail but with /proc/<pid>/oom_score this is likely a lost battle and I am willing to give up on that front.
Understood.
Quoted 4 lines
I am still not entirely convinced this is worth doing though.
oom_badnessis not a cheap operation.task_lockhas to be taken again duringdump_tasksfor each task so the already quite expensive operation will be more so. Is this really something we cannot live without?
Fair enough and I now agree, it is unquestionably not worth it.
Kind regards,
Signature · 1 line
Aaron Tomlin