mm/oom_kill: allow oom kill allocating task for non-global case
Subsystems (2)
RFC PATCH 1/1 mm/oom_kill: allow oom kill allocating task for non-global case
At the present time, in the context of memcg OOM, even when sysctl_oom_kill_allocating_task is enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.
This patch removes the restriction entirely.
--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/oom_kill.c +3 -3
@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc); -if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;
Signature · 1 line
2.26.3
Quoted 18 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 10 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;
To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found. I don't see a reason why this should be limited to a global OOM only.
Cheers,
Longman
Quoted 21 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);
Quoted 15 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
Quoted 7 lines
I don't see a reason why this should be limited to a global OOM only.
Acked-by Waiman Long <[email protected]>Cheers,
Longman
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a oom_adjust_score of -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.
The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the oom_kill_allocating_task to 1 if the OOM happens only within the memcg context, not in a global OOM situation. This patch is to allow this process to be killed under this circumstance.
Cheers,
Longman
Michal Hocko
maintainerQuoted 43 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.
Quoted 6 lines
The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation. This patch is to allow this process to be killed under this circumstance.
Do you have the oom report? I do not see why the allocating task hasn't been chosen.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 43 lines · Michal Hocko
Quoted 42 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.
Quoted 8 lines
Quoted 6 lines
The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation. This patch is to allow this process to be killed under this circumstance.Do you have the oom report? I do not see why the allocating task hasn't been chosen.
A partial OOM report below:
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999 [ 8227.242732] node cpuset=crio-b8ac7e23f7b520c0365461defb66738231918243586e287bfb9e206bb3a0227a.scope mems_allowed=0-1
So in this case, node cannot kill itself and no other processes are available to be killed.
Cheers,
Longman
Michal Hocko
maintainerQuoted 50 lines · Waiman Long
Quoted 49 lines · Michal Hocko
Quoted 48 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation. This patch is to allow this process to be killed under this circumstance.
Quoted 4 lines
Quoted 2 lines
Do you have the oom report? I do not see why the allocating task hasn't been chosen.
A partial OOM report below:
Do you happen to have the full report?
Quoted 10 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999 [ 8227.242732] node cpuset=crio-b8ac7e23f7b520c0365461defb66738231918243586e287bfb9e206bb3a0227a.scope mems_allowed=0-1
Quoted 2 lines
So in this case, node cannot kill itself and no other processes are available to be killed.
The process is clearly listed as eligible so the oom killer should find it and if it hasn't then this should be investigated. Which kernel is this?
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
maintainerQuoted 1 line · Michal Hocko
On Mon 07-06-21 15:18:38, Waiman Long wrote:
[...]
Quoted 12 lines
Quoted 1 line
A partial OOM report below:
Do you happen to have the full report?
Quoted 8 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999
Btw it is surprising to not see _GFP_ACCOUNT here.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 14 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Mon 07-06-21 15:18:38, Waiman Long wrote:
[...]
Quoted 11 lines
Quoted 1 line
A partial OOM report below:
Do you happen to have the full report?
Quoted 8 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999
Quoted 1 line
Btw it is surprising to not see
_GFP_ACCOUNThere.
There are a number of OOM kills in the kernel log and none of the tasks that invoke oom-killer has _GFP_ACCOUNT flag set.
Regards,
Longman
Michal Hocko
maintainerQuoted 15 lines · Waiman Long
Quoted 14 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Mon 07-06-21 15:18:38, Waiman Long wrote:
[...]
Quoted 11 lines
Quoted 1 line
A partial OOM report below:
Do you happen to have the full report?
Quoted 8 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999
Quoted 4 lines
Quoted 1 line
Btw it is surprising to not see
_GFP_ACCOUNThere.There are a number of OOM kills in the kernel log and none of the tasks that invoke oom-killer has
_GFP_ACCOUNTflag set.
OK. A full report (including the backtrace) would tell us more what is the source of the charge. I thought that most #PF charging paths use the same gfp mask as the allocation (which would include other flags on top of GFP_KERNEL) but it seems we just use GFP_KERNEL at many places. There are also some direct callers of the charging API for kernel allocations. Not that this would be super important but it caught my attention.
You are saying that there are other OOM kills going on. Are they all for the same memcg? Is it possible the only eligible task has been killed and oom reaped already?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 1 line · Michal Hocko
You are saying that there are other OOM kills going on.
Yes.
Quoted 1 line
Are they all for the same memcg?
No.
Quoted 2 lines
Is it possible the only eligible task has been killed and oom reaped already?
Unfortunately, as mentioned by Waiman, no other task is suitable for "selection" due to the OOM score (i.e. OOM_SCORE_ADJ_MIN;) that said, I need access to the vmcore to check i.e. mm.flags - I'll get back to you as soon as I can.
Kind regards,
Signature · 1 line
Aaron Tomlin
Quoted 4 lines · Michal Hocko
OK. A full report (including the backtrace) would tell us more what is the source of the charge. I thought that most #PF charging paths use the same gfp mask as the allocation (which would include other flags on top of
GFP_KERNEL) but it seems we just useGFP_KERNELat many places.
The following is what I can provide for now:
[ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999 [ 8227.242732] node cpuset=XXXX mems_allowed=0-1 [ 8227.242736] CPU: 12 PID: 1703347 Comm: node Kdump: loaded Not tainted 4.18.0-193.51.1.el8_2.x86_64 #1 [ 8227.242737] Hardware name: XXXX [ 8227.242738] Call Trace: [ 8227.242746] dump_stack+0x5c/0x80 [ 8227.242751] dump_header+0x6e/0x27a [ 8227.242753] out_of_memory.cold.31+0x39/0x8d [ 8227.242756] mem_cgroup_out_of_memory+0x49/0x80 [ 8227.242758] try_charge+0x58c/0x780 [ 8227.242761] ? __alloc_pages_nodemask+0xef/0x280 [ 8227.242763] mem_cgroup_try_charge+0x8b/0x1a0 [ 8227.242764] mem_cgroup_try_charge_delay+0x1c/0x40 [ 8227.242767] do_anonymous_page+0xb5/0x360 [ 8227.242770] ? __switch_to_asm+0x35/0x70 [ 8227.242772] __handle_mm_fault+0x662/0x6a0 [ 8227.242774] handle_mm_fault+0xda/0x200 [ 8227.242778] __do_page_fault+0x22d/0x4e0 [ 8227.242780] do_page_fault+0x32/0x110 [ 8227.242782] ? page_fault+0x8/0x30 [ 8227.242783] page_fault+0x1e/0x30
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 8 lines · Aaron Tomlin
Quoted 4 lines · Michal Hocko
OK. A full report (including the backtrace) would tell us more what is the source of the charge. I thought that most #PF charging paths use the same gfp mask as the allocation (which would include other flags on top of
GFP_KERNEL) but it seems we just useGFP_KERNELat many places.The following is what I can provide for now:
Let me add what we have from previous email
Quoted 7 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes...
I do not see this message to be ever printed on 4.18 for memcg oom:
/* Found nothing?!?! Either we hang forever, or we panic. */
if (!oc->chosen && !is_sysrq_oom(oc) && !is_memcg_oom(oc)) {
dump_header(oc, NULL);
panic("Out of memory and no killable processes...\n");
}So how come it got triggered here? Is it possible that there is a global oom killer somehow going on along with the memcg OOM? Because the below stack clearly points to a memcg OOM and a new one AFAICS.
That being said, a full chain of oom events would be definitely useful to get a better idea.
Quoted 24 lines
[ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999 [ 8227.242732] node cpuset=XXXX mems_allowed=0-1 [ 8227.242736] CPU: 12 PID: 1703347 Comm: node Kdump: loaded Not tainted 4.18.0-193.51.1.el8_2.x86_64 #1 [ 8227.242737] Hardware name: XXXX [ 8227.242738] Call Trace: [ 8227.242746] dump_stack+0x5c/0x80 [ 8227.242751] dump_header+0x6e/0x27a [ 8227.242753] out_of_memory.cold.31+0x39/0x8d [ 8227.242756] mem_cgroup_out_of_memory+0x49/0x80 [ 8227.242758] try_charge+0x58c/0x780 [ 8227.242761] ? __alloc_pages_nodemask+0xef/0x280 [ 8227.242763] mem_cgroup_try_charge+0x8b/0x1a0 [ 8227.242764] mem_cgroup_try_charge_delay+0x1c/0x40 [ 8227.242767] do_anonymous_page+0xb5/0x360 [ 8227.242770] ? __switch_to_asm+0x35/0x70 [ 8227.242772] __handle_mm_fault+0x662/0x6a0 [ 8227.242774] handle_mm_fault+0xda/0x200 [ 8227.242778] __do_page_fault+0x22d/0x4e0 [ 8227.242780] do_page_fault+0x32/0x110 [ 8227.242782] ? page_fault+0x8/0x30 [ 8227.242783] page_fault+0x1e/0x30Signature · 1 line
Aaron Tomlin
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Michal Hocko
I do not see this message to be ever printed on 4.18 for memcg oom:
/* Found nothing?!?! Either we hang forever, or we panic. */ if (!oc->chosen && !is_sysrq_oom(oc) && !is_memcg_oom(oc)) { dump_header(oc, NULL); panic("Out of memory and no killable processes...\n"); }So how come it got triggered here? Is it possible that there is a global oom killer somehow going on along with the memcg OOM? Because the below stack clearly points to a memcg OOM and a new one AFAICS.
4.18 does print this message, and panic() will be called if global OOM killer invocation were in progress.
4.18.0-193.51.1.el8.x86_64 is doing
----------
select_bad_process(oc);
/* Found nothing?!?! */
if (!oc->chosen) {
dump_header(oc, NULL);
pr_warn("Out of memory and no killable processes...\n");
/*
* If we got here due to an actual allocation at the
* system level, we cannot survive this and will enter
* an endless loop in the allocator. Bail out now.
*/
if (!is_sysrq_oom(oc) && !is_memcg_oom(oc))
panic("System is deadlocked on memory\n");
}----------
and this message is printed when oom_evaluate_task() found that MMF_OOM_SKIP was already set on all (possibly the only and the last) OOM victims.
----------
static int oom_evaluate_task(struct task_struct *task, void *arg) {
(...snipped...)
/*
* This task already has access to memory reserves and is being killed.
* Don't allow any other task to have access to the reserves unless
* the task has MMF_OOM_SKIP because chances that it would release
* any memory is quite low.
*/
if (!is_sysrq_oom(oc) && tsk_is_oom_victim(task)) {
if (test_bit(MMF_OOM_SKIP, &task->signal->oom_mm->flags))
goto next;
goto abort;
}
(...snipped...)
next:
return 0;(...snipped...)
}
----------
Since dump_tasks() from dump_header(oc, NULL) does not exclude tasks which already has MMF_OOM_SKIP set, it is possible that the last OOM killable victim was already OOM killed but the OOM reaper failed to reclaim memory and set MMF_OOM_SKIP. (Well, maybe we want to exclude (or annotate) MMF_OOM_SKIP tasks when showing OOM victim candidates...)
Therefore,
Quoted 3 lines
That being said, a full chain of oom events would be definitely useful to get a better idea.
I think checking whether
pr_info("oom_reaper: unable to reap pid:%d (%s)\n",
task_pid_nr(tsk), tsk->comm);and/or
pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
task_pid_nr(tsk), tsk->comm,
K(get_mm_counter(mm, MM_ANONPAGES)),
K(get_mm_counter(mm, MM_FILEPAGES)),
K(get_mm_counter(mm, MM_SHMEMPAGES)));message was already printed prior to starting infinite "Out of memory and no killable processes..." looping (this message is repeated forever, isn't it?) will be useful.
Note that neither of these messages will be printed if hitting
----------
/*
* If the mm has invalidate_{start,end}() notifiers that could block,
* sleep to give the oom victim some more time.
* TODO: we really want to get rid of this ugly hack and make sure that
* notifiers cannot block for unbounded amount of time
*/
if (mm_has_blockable_invalidate_notifiers(mm)) {
up_read(&mm->mmap_sem);
schedule_timeout_idle(HZ);
return true;
}----------
case, and also dmesg available in the vmcore might be too late to examine. Maybe better to check /var/log/messages instead of vmcore file.
Michal Hocko
maintainerQuoted 31 lines · Tetsuo Handa
Quoted 10 lines · Michal Hocko
I do not see this message to be ever printed on 4.18 for memcg oom:
/* Found nothing?!?! Either we hang forever, or we panic. */ if (!oc->chosen && !is_sysrq_oom(oc) && !is_memcg_oom(oc)) { dump_header(oc, NULL); panic("Out of memory and no killable processes...\n"); }So how come it got triggered here? Is it possible that there is a global oom killer somehow going on along with the memcg OOM? Because the below stack clearly points to a memcg OOM and a new one AFAICS.
4.18 does print this message, and
panic()will be called if global OOM killer invocation were in progress.4.18.0-193.51.1.el8.x86_64 is doing
----------
select_bad_process(oc); /* Found nothing?!?! */ if (!oc->chosen) { dump_header(oc, NULL); pr_warn("Out of memory and no killable processes...\n"); /* * If we got here due to an actual allocation at the * system level, we cannot survive this and will enter * an endless loop in the allocator. Bail out now. */ if (!is_sysrq_oom(oc) && !is_memcg_oom(oc)) panic("System is deadlocked on memory\n"); }
Quoted 1 line
----------
Ahh, OK. That would explain that. I have looked at 4.18 Vanilla kernel. I do not have RHEL sources handy and neither checked the 4.18 stable tree. Thanks for the clarification!
[...]
Quoted 5 lines
Since
dump_tasks()fromdump_header(oc, NULL)does not exclude tasks which already hasMMF_OOM_SKIPset, it is possible that the last OOM killable victim was already OOM killed but the OOM reaper failed to reclaim memory and setMMF_OOM_SKIP. (Well, maybe we want to exclude (or annotate)MMF_OOM_SKIPtasks when showing OOM victim candidates...)
Well, the allocating task was clearly alive and whether it has been reaped or not is not all that important as it should force the charge as an oom victim. This is actually the most puzzling part. Because the allocating task either is not a preexisting OOM victim and therefore could become one or it has been and should have skipped the memcg killer altogether. But I fail to see how it could be missed completely while looking for a victim.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 2 lines · Michal Hocko
Is it possible the only eligible task has been killed and oom reaped already?
Yes, I suspect so; and I had a look at the vmcore, the task in the OOM report is no longer present. Therefore, I suspect the task namely "node" (i.e. PID 1703345) was OOM killed i.e. a SIGKILL was sent and was granted access to memory reserves and selected/or choosen by the OOM reaper for termination; the victim then raised a page fault that triggered yet another "charge" in the memcg that exceeded the memory limit set on the container; and since no other task in the memcg had a suitable OOM score and the allocating task/or victim was "unkillable" i.e. already selected for termination by the OOM reaper, we got the message: "Out of memory and no killable processes...".
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerQuoted 11 lines · Aaron Tomlin
Quoted 2 lines · Michal Hocko
Is it possible the only eligible task has been killed and oom reaped already?
Yes, I suspect so; and I had a look at the vmcore, the task in the OOM report is no longer present. Therefore, I suspect the task namely "node" (i.e. PID 1703345) was OOM killed i.e. a SIGKILL was sent and was granted access to memory reserves and selected/or choosen by the OOM reaper for termination; the victim then raised a page fault that triggered yet another "charge" in the memcg that exceeded the memory limit set on the container;
If that was the case then the allocating (charging) task would not hit the oom path at all
stable/linux-4.18.y:mm/memcontrol.ctry_charge()
/*
* Unlike in global OOM situations, memcg is not in a physical
* memory shortage. Allow dying and OOM-killed tasks to
* bypass the last charges so that they can exit quickly and
* free their memory.
*/
if (unlikely(tsk_is_oom_victim(current) ||
fatal_signal_pending(current) ||
current->flags & PF_EXITING))
goto force;If you have a crash dump available then you can check the memcg associate with the allocating task and check whether it is really marked as OOM victim.
Quoted 4 lines
and since no other task in the memcg had a suitable OOM score and the allocating task/or victim was "unkillable" i.e. already selected for termination by the OOM reaper, we got the message: "Out of memory and no killable processes...".
What do you mean by allocating task being unkillable?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 2 lines · Michal Hocko
If that was the case then the allocating (charging) task would not hit the oom path at all
Yes, you are correct. I was looking at another version of the source code. I does not make sense to consider the OOM code path at all, in this context. The allocating task is selected/or marked as an "OOM vicitm" after SIGKILL is sent (see __oom_kill_process()).
Quoted 1 line
What do you mean by allocating task being unkillable?
Please disregard this statement, as per the above.
Anyhow, I think we should exclude tasks that have MMF_OOM_SKIP applied in dump_tasks() as it could be misleading.
Signature · 1 line
Aaron Tomlin
Michal Hocko
maintainerOn Thu 10-06-21 13:23:23, Aaron Tomlin wrote:
[...]
Quoted 2 lines
Anyhow, I think we should exclude tasks that have
MMF_OOM_SKIPapplied indump_tasks()as it could be misleading.
Well, I am not sure this is a good thing in general. We do not want to hide tasks. We already do display oom_score_adj_min even though they are not eligible and that can serve a good purpose from my experience. It gives us some picture at least. Maybe a flag to mark all uneligible tasks would be something useful but I wouldn't drop them from the list.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 5 lines · Michal Hocko
Well, I am not sure this is a good thing in general. We do not want to hide tasks. We already do display
oom_score_adj_mineven though they are not eligible and that can serve a good purpose from my experience. It gives us some picture at least. Maybe a flag to mark all uneligible tasks would be something useful but I wouldn't drop them from the list.
Fair enough.
Signature · 1 line
Aaron Tomlin
Quoted 8 lines · Aaron Tomlin
Quoted 5 lines · Michal Hocko
Well, I am not sure this is a good thing in general. We do not want to hide tasks. We already do display
oom_score_adj_mineven though they are not eligible and that can serve a good purpose from my experience. It gives us some picture at least. Maybe a flag to mark all uneligible tasks would be something useful but I wouldn't drop them from the list.Fair enough.
Yes, marking ineligible (i.e. oom_badness() returning LONG_MIN) tasks would be useful.
By the way, was the task namely "node" (i.e. PID 1703345) multithreaded program?
Your kernel might want commit 7775face207922ea ("memcg: killed threads should not invoke memcg OOM killer").
Michal Hocko
maintainerQuoted 8 lines · Tetsuo Handa
Quoted 7 lines · Aaron Tomlin
Quoted 5 lines · Michal Hocko
Well, I am not sure this is a good thing in general. We do not want to hide tasks. We already do display
oom_score_adj_mineven though they are not eligible and that can serve a good purpose from my experience. It gives us some picture at least. Maybe a flag to mark all uneligible tasks would be something useful but I wouldn't drop them from the list.
Quoted 6 lines
Quoted 1 line
Fair enough.
Yes, marking ineligible (i.e.
oom_badness()returningLONG_MIN) tasks would be useful.By the way, was the task namely "node" (i.e. PID 1703345) multithreaded program?
Your kernel might want commit7775face207922ea("memcg: killed threads should not invoke memcg OOM killer").
Yes, this would help for multithreaded race situation indeed. Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 2 lines · Tetsuo Handa
Yes, marking ineligible (i.e.
oom_badness()returningLONG_MIN) tasks would be useful.
Sure. I'll send a patch to append this information; and the reason why i.e. "yes: oom skipped" or "yes: oom score" or "yes: vfork" to dump_task() with the use of some helper function e.g. is_task_eligible_oom().
Quoted 2 lines
By the way, was the task namely "node" (i.e. PID 1703345) multithreaded program?
Unfortunately, I cannot confirm since the scenario/or situation is no longer present in the vmcore; albeit, I believe so.
Quoted 2 lines
Your kernel might want commit
7775face207922ea("memcg: killed threads should not invoke memcg OOM killer").
Tetsuo-san,
Yes, this does appear to be a match.
Thanks for this.
Kind regards,
Signature · 1 line
Aaron Tomlin
Quoted 51 lines · Michal Hocko
Quoted 50 lines · Waiman Long
Quoted 49 lines · Michal Hocko
Quoted 48 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation. This patch is to allow this process to be killed under this circumstance.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Do you have the oom report? I do not see why the allocating task hasn't been chosen.
A partial OOM report below:
Do you happen to have the full report?
I need to ask to see if I can release the full report.
Quoted 11 lines
Quoted 11 lines
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 : [ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon [ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon [ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node [ 8227.242706] Out of memory and no killable processes... [ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999 [ 8227.242732] node cpuset=crio-b8ac7e23f7b520c0365461defb66738231918243586e287bfb9e206bb3a0227a.scope mems_allowed=0-1
Quoted 5 lines
Quoted 2 lines
So in this case, node cannot kill itself and no other processes are available to be killed.
The process is clearly listed as eligible so the oom killer should find it and if it hasn't then this should be investigated. Which kernel is this?
Right. I don't know why the current cannot be selected. I think we may need to enhance the OOM but no killable process report to list the reason a task is skipped other than oom_score_adj. The kernel is a RHEL8.2 kernel which has OOM code pretty close to upstream.
Cheers,
Longman
Quoted 3 lines · Waiman Long
Right. I don't know why the current cannot be selected. I think we may need to enhance the OOM but no killable process report to list the reason a task is skipped other than
oom_score_adj.
Actually, this might be a good idea.
Signature · 1 line
Aaron Tomlin
Quoted 44 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.
Quoted 6 lines
The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation.
I am not really against the patch but I am still not able to understand why select_bad_process() was not able to select the current process. mem_cgroup_scan_tasks() traverses all the processes in the target memcg hierarchy, so why the current was skipped.
Quoted 6 lines
This patch is to allow this
process to be killed under this circumstance.Cheers,
Longman
Quoted 49 lines · Shakeel Butt
Quoted 42 lines · Waiman Long
Quoted 35 lines · Shakeel Butt
Quoted 20 lines · Waiman Long
Quoted 19 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +0
-0@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc);Quoted 14 lines
mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1097
Quoted 9 lines
mm/oom_kill.c +3
-3-if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;To provide more context for this patch, we are actually seeing that in a customer report about OOM happened in a container where the dominating task used up most of the memory and it happened to be the task that triggered the OOM with the result that no killable process could be found.
Quoted 2 lines
Why was there no killable process? What about the process allocating the memory or is this remote memcg charging?
It is because the other processes have a
oom_adjust_scoreof -1000. So they are non-killable. Anyway, they don't consume that much memory and killing them won't free up that much.Quoted 6 lines
The other process that uses most of the memory is the one that trigger the OOM kill in the first place because the memory limit has been reached in new memory allocation. Based on the current logic, this process cannot be killed at all even if we set the
oom_kill_allocating_taskto 1 if the OOM happens only within the memcg context, not in a global OOM situation.
Quoted 4 lines
I am not really against the patch but I am still not able to understand why
select_bad_process()was not able to select the current process.mem_cgroup_scan_tasks()traverses all the processes in the target memcg hierarchy, so why the current was skipped.
Yes, you are right. Probably there is some problem with reaping so that the MMF_OOM_SKIP bit gets set. I don't have the answer yet.
Regards,
Longman
Michal Hocko
maintainerQuoted 5 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
This is a global oom policy not a memcg specific one so a historical behavior would change. So I do not think we can change that. The policy can be implemented on the memcg level but this would require a much more detailed explanation of the usecase and the semantic (e.g. wrt. hierarchical behavior etc).
Quoted 24 lines
Signed-off-by Aaron Tomlin <[email protected]>--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)mm/oom_kill.c +3
-3@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc) oc->nodemask =NULL ;check_panic_on_oom (oc); -if (!is_memcg_oom (oc) && sysctl_oom_kill_allocating_task && - current->mm && !oom_unkillable_task (current) && -oom_cpuset_eligible (current, oc) && +if (sysctl_oom_kill_allocating_task && current->mm && + !oom_unkillable_task (current) && +oom_cpuset_eligible (current, oc) && current->signal->oom_score_adj !=OOM_SCORE_ADJ_MIN ) {get_task_struct (current); oc->chosen = current;Signature · 1 line
2.26.3
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 11 lines · Michal Hocko
Quoted 5 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
This is a global oom policy not a memcg specific one so a historical behavior would change. So I do not think we can change that. The policy can be implemented on the memcg level but this would require a much more detailed explanation of the usecase and the semantic (e.g. wrt. hierarchical behavior etc).
Maybe we can extend the meaning of oom_kill_allocating_task such that memcg OOM killing of allocating task is only enabled when bit 1 is set. So if an existing application just set oom_kill_allocating_task to 1, it will not be impacted.
Cheers,
Longman
Michal Hocko
maintainerQuoted 13 lines · Waiman Long
Quoted 11 lines · Michal Hocko
Quoted 5 lines · Aaron Tomlin
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_taskis enabled/or set, the "allocating" task cannot be selected, as a target for the OOM killer.This patch removes the restriction entirely.
This is a global oom policy not a memcg specific one so a historical behavior would change. So I do not think we can change that. The policy can be implemented on the memcg level but this would require a much more detailed explanation of the usecase and the semantic (e.g. wrt. hierarchical behavior etc).
Quoted 4 lines
Maybe we can extend the meaning of
oom_kill_allocating_tasksuch that memcg OOM killing of allocating task is only enabled when bit 1 is set. So if an existing application just setoom_kill_allocating_taskto 1, it will not be impacted.
panic_on_oom is already allowing to implement originally global policy to memcg. So if anything this policy should follow the same interface but still I think what you are seeing is either a bug or something else (e.g. the task being migrated while the oom is ongoing) and this should be properly investigated and explained. We cannot simply paper it over by telling people to use oom_kill_allocating_task to work it around.
If there is a real usecase for such a policy for memcg oom killing can be discussed of course.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 3 lines · Michal Hocko
On Mon 07-06-21 17:31:03, Aaron Tomlin wrote:
This is a global oom policy not a memcg specific one so a historical behavior would change. So I do not think we can change that.
I see. Fair enough.
Signature · 1 line
Aaron Tomlin