mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled
Subsystems (3)
PATCH 1/1 mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled
Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.
There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...
This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.
---
We haven't seen any bug reports
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)
mm/oom_kill.c +8 -0
@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true; }up_read (&oom_sem);
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · Michal Hocko
Tetsuo Handa has pointed out that
__GFP_NOFAILallocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by7f33d49a2ed5(mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.There are basically two ways forward.
1) moveoom_killer_disableafter kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not failGFP_NOFAILallocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify
__GFP_NOFAILusers from deeper pm code.
Quoted 4 lines
Signed-off-by Michal Hocko <[email protected]>---
We haven't seen any bug reports
Ups, forgot to save the file before sending. The full text is:
"
We haven't seen any bug reports since 2009 so I haven't marked the patch for stable. I have no problem to backport it to stable trees though if people think it is a good precaution.
"
Quoted 38 lines
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/oom_kill.c +8
-0@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer @@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true; }up_read (&oom_sem);Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 2 lines
Michal Hocko SUSE Labs
Johannes Weiner
reviewerQuoted 47 lines · Michal Hocko
Tetsuo Handa has pointed out that
__GFP_NOFAILallocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by7f33d49a2ed5(mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.There are basically two ways forward.
1) moveoom_killer_disableafter kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not failGFP_NOFAILallocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify
__GFP_NOFAILusers from deeper pm code.Signed-off-by Michal Hocko <[email protected]>---
We haven't seen any bug reports
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/oom_kill.c +4
-0@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer
mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802
Quoted 7 lines
mm/oom_kill.c +4
-0@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true;
I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
WARN_ON_ONCE()?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko
Tetsuo Handa has pointed out that
__GFP_NOFAILallocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by7f33d49a2ed5(mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.There are basically two ways forward.
1) moveoom_killer_disableafter kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not failGFP_NOFAILallocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify
__GFP_NOFAILusers from deeper pm code.Signed-off-by Michal Hocko <[email protected]>---
We haven't seen any bug reports
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/oom_kill.c +4
-0@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802
Quoted 7 lines
mm/oom_kill.c +4
-0@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true;I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
Quoted 2 lines
WARN_ON_ONCE()?
Yeah, I'm not sure that the warning is helpful (and it needs s/disbaled/disabled/ if it is to be kept). I also think this check should be moved out of out_of_memory() since gfp/retry logic should be in the page allocator itself and not in the oom killer: just make __alloc_pages_may_oom() also set *did_some_progress = 1 for __GFP_NOFAIL.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled
Quoted 1 line · David Rientjes
On Tue, 24 Feb 2015, Johannes Weiner wrote:
[...]
Quoted 4 lines
Quoted 4 lines
I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
Quoted 8 lines
Quoted 2 lines
WARN_ON_ONCE()?Yeah, I'm not sure that the warning is helpful (and it needs s/disbaled/disabled/ if it is to be kept). I also think this check should be moved out of
out_of_memory()since gfp/retry logic should be in the page allocator itself and not in the oom killer: just make__alloc_pages_may_oom()also set*did_some_progress= 1 for__GFP_NOFAIL.
OK, this is a good point. Updated patch is below:
---
Quoted 5 lines · Michal Hocko
mm/page_alloc.c +0
-0
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2370
Quoted 9 lines
mm/page_alloc.c +2
-1@@ -2363,7 +2363,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (ac->zonelist, gfp_mask, order, ac->nodemask, false)) +if (out_of_memory (ac->zonelist, gfp_mask, order, ac->nodemask, false) + ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) *did_some_progress =1 ; out:oom_zonelist_unlock (ac->zonelist, gfp_mask);
Eek, not sure we actually need to play any games with did_some_progress, it might be clearer just to do this
mm/page_alloc.c +1 -1
@@ -2760,7 +2760,7 @@ retry: &did_some_progress);if (page) goto got_pg; -if (!did_some_progress) +if (!did_some_progress && !(gfp_mask & __GFP_NOFAIL)) goto nopage; }/* Wait for some write requests to complete then retry */
Either way you decide, feel free to add my
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · David Rientjes
Quoted 9 lines · Michal Hocko
mm/page_alloc.c +0
-1@@ -2363,7 +2363,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (ac->zonelist, gfp_mask, order, ac->nodemask, false))
Quoted 8 lines
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2370
Quoted 5 lines
mm/page_alloc.c +2
-0+if (out_of_memory (ac->zonelist, gfp_mask, order, ac->nodemask, false) + ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) *did_some_progress =1 ; out:oom_zonelist_unlock (ac->zonelist, gfp_mask);Eek, not sure we actually need to play any games with
did_some_progress, it might be clearer just to do this
We would loose the warning which _might_ be helpful and I also find this place better because it is close to the out_of_memory and this one has only one failure mode. So I would prefer to stick with this unless there are big objections.
Quoted 13 lines
mm/page_alloc.c +1
-1@@ -2760,7 +2760,7 @@ retry: &did_some_progress);if (page) goto got_pg; -if (!did_some_progress) +if (!did_some_progress && !(gfp_mask & __GFP_NOFAIL)) goto nopage; }/* Wait for some write requests to complete then retry */
Quoted 3 lines
Either way you decide, feel free to add my
Acked-by David Rientjes <[email protected]>
Thanks!
Andrew, should I repost or you can pick it up from this thread? Assuming you and others do not have objections of course.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko
Tetsuo Handa has pointed out that
__GFP_NOFAILallocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by7f33d49a2ed5(mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.There are basically two ways forward.
1) moveoom_killer_disableafter kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not failGFP_NOFAILallocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify
__GFP_NOFAILusers from deeper pm code.Signed-off-by Michal Hocko <[email protected]>---
We haven't seen any bug reports
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/oom_kill.c +4
-0@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802
Quoted 7 lines
mm/oom_kill.c +4
-0@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true;I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
Quoted 1 line
WARN_ON_ONCE()?
maybe panic() ?
If somebody turns off oom-killer it seems he's pretty sure that he has enough memory.
Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 61 lines · Konstantin Khlebnikov
Quoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko
Tetsuo Handa has pointed out that
__GFP_NOFAILallocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by7f33d49a2ed5(mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.There are basically two ways forward.
1) moveoom_killer_disableafter kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not failGFP_NOFAILallocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify
__GFP_NOFAILusers from deeper pm code.Signed-off-by Michal Hocko <[email protected]>---
We haven't seen any bug reports
mm/oom_kill.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/oom_kill.c +4
-0@@ -772,6 +772,10 @@ out:schedule_timeout_killable (1 ); } +static DEFINE_RATELIMIT_STATE(oom_disabled_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); +/** * out_of_memory - tries to invoke OOM killer. * @zonelist: zonelist pointer mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802
Quoted 7 lines
mm/oom_kill.c +4
-0@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,if (!oom_killer_disabled) {__out_of_memory (zonelist, gfp_mask, order, nodemask, force_kill); ret = true; + }else if (gfp_mask & __GFP_NOFAIL) { +if (__ratelimit (&oom_disabled_rs)) +WARN (1 ,"Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n" ); + ret = true;I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
Quoted 6 lines
Quoted 1 line
WARN_ON_ONCE()?maybe
panic()?If somebody turns off oom-killer it seems he's pretty sure that he has enough memory.
Ah, that's used in freeze/suspend code. I thought that some kind of sysctl for brave sysadmins.
Quoted 7 lines
Quoted 7 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
On Tue 24-02-15 14:11:27, Johannes Weiner wrote:
[...]
Quoted 3 lines
I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?
It is better than a silent endless loop. And we get a trace which points to the place which is doing the allocation. We haven't seen any weird crashes during suspend throughout last 6 years so this would be extremely unlikely and hard to reproduce so having the trace sounds useful to me.
Quoted 1 line
WARN_ON_ONCE()?
I do not expect this will spew a lot of messages. But I can live with WARN_ON_ONCE as well.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>