Skip to contentSkip to conversation

mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled

PATCH In review Michal Hocko 4 participants 10 messages Feb 24, 2015 – Feb 26, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

---

We haven't seen any bug reports

 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +8 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
 		schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+		DEFAULT_RATELIMIT_INTERVAL,
+		DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
 	if (!oom_killer_disabled) {
 		__out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
 		ret = true;
+	} else if (gfp_mask & __GFP_NOFAIL) {
+		if (__ratelimit(&oom_disabled_rs))
+			WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+		ret = true;
 	}
 	up_read(&oom_sem);
 
Signature · 7 lines
2.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 23 lines · Michal Hocko

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

Quoted 4 lines

---

We haven't seen any bug reports

Ups, forgot to save the file before sending. The full text is:
"
We haven't seen any bug reports since 2009 so I haven't marked the patch for stable. I have no problem to backport it to stable trees though if people think it is a good precaution.
"

Quoted 38 lines
 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +8 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
 		schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+		DEFAULT_RATELIMIT_INTERVAL,
+		DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
 	if (!oom_killer_disabled) {
 		__out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
 		ret = true;
+	} else if (gfp_mask & __GFP_NOFAIL) {
+		if (__ratelimit(&oom_disabled_rs))
+			WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+		ret = true;
 	}
 	up_read(&oom_sem);
 
Signature · 7 lines
2.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 2 lines
Michal Hocko
SUSE Labs
#2 lore raw
In reply to Michal Hocko
Quoted 47 lines · Michal Hocko

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

---

We haven't seen any bug reports

 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +4 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
 		schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+		DEFAULT_RATELIMIT_INTERVAL,
+		DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer

mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802

Quoted 7 lines
mm/oom_kill.c +4 -0
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
 	if (!oom_killer_disabled) {
 		__out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
 		ret = true;
+	} else if (gfp_mask & __GFP_NOFAIL) {
+		if (__ratelimit(&oom_disabled_rs))
+			WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+		ret = true;

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

WARN_ON_ONCE()?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

---

We haven't seen any bug reports

 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +4 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
 		schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+		DEFAULT_RATELIMIT_INTERVAL,
+		DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer

mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802

Quoted 7 lines
mm/oom_kill.c +4 -0
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
 	if (!oom_killer_disabled) {
 		__out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
 		ret = true;
+	} else if (gfp_mask & __GFP_NOFAIL) {
+		if (__ratelimit(&oom_disabled_rs))
+			WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+		ret = true;

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

Quoted 2 lines

WARN_ON_ONCE()?

Yeah, I'm not sure that the warning is helpful (and it needs s/disbaled/disabled/ if it is to be kept). I also think this check should be moved out of out_of_memory() since gfp/retry logic should be in the page allocator itself and not in the oom killer: just make __alloc_pages_may_oom() also set *did_some_progress = 1 for __GFP_NOFAIL.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

PATCH 1/1 mm, oom: do not fail __GFP_NOFAIL allocation if oom killer is disbaled

Quoted 1 line · David Rientjes

On Tue, 24 Feb 2015, Johannes Weiner wrote:

[...]

Quoted 4 lines
Quoted 4 lines

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

Quoted 8 lines
Quoted 2 lines

WARN_ON_ONCE()?

Yeah, I'm not sure that the warning is helpful (and it needs s/disbaled/disabled/ if it is to be kept). I also think this check should be moved out of out_of_memory() since gfp/retry logic should be in the page allocator itself and not in the oom killer: just make __alloc_pages_may_oom() also set *did_some_progress = 1 for __GFP_NOFAIL.

OK, this is a good point. Updated patch is below:
---

reviewer Acked-by on v1 1/1
#5 lore raw
Quoted 5 lines · Michal Hocko
mm/page_alloc.c +0 -0
index 2d224bbdf8e8..c2ff40a30003 100644

mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2370

Quoted 9 lines
mm/page_alloc.c +2 -1
@@ -2363,7 +2363,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 			goto out;
 	}
 	/* Exhausted what can be done so it's blamo time */
-	if (out_of_memory(ac->zonelist, gfp_mask, order, ac->nodemask, false))
+	if (out_of_memory(ac->zonelist, gfp_mask, order, ac->nodemask, false)
+			|| WARN_ON_ONCE(gfp_mask & __GFP_NOFAIL))
 		*did_some_progress = 1;
 out:
 	oom_zonelist_unlock(ac->zonelist, gfp_mask);

Eek, not sure we actually need to play any games with did_some_progress, it might be clearer just to do this

mm/page_alloc.c +1 -1
@@ -2760,7 +2760,7 @@ retry:
 							&did_some_progress);
 			if (page)
 				goto got_pg;
-			if (!did_some_progress)
+			if (!did_some_progress && !(gfp_mask & __GFP_NOFAIL))
 				goto nopage;
 		}
 		/* Wait for some write requests to complete then retry */

Either way you decide, feel free to add my

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 11 lines · David Rientjes
Quoted 9 lines · Michal Hocko
mm/page_alloc.c +0 -1
index 2d224bbdf8e8..c2ff40a30003 100644
@@ -2363,7 +2363,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 			goto out;
 	}
 	/* Exhausted what can be done so it's blamo time */
-	if (out_of_memory(ac->zonelist, gfp_mask, order, ac->nodemask, false))
Quoted 8 lines

mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2370

Quoted 5 lines
mm/page_alloc.c +2 -0
+	if (out_of_memory(ac->zonelist, gfp_mask, order, ac->nodemask, false)
+			|| WARN_ON_ONCE(gfp_mask & __GFP_NOFAIL))
 		*did_some_progress = 1;
 out:
 	oom_zonelist_unlock(ac->zonelist, gfp_mask);

Eek, not sure we actually need to play any games with did_some_progress, it might be clearer just to do this

We would loose the warning which _might_ be helpful and I also find this place better because it is close to the out_of_memory and this one has only one failure mode. So I would prefer to stick with this unless there are big objections.

Quoted 13 lines
mm/page_alloc.c +1 -1
@@ -2760,7 +2760,7 @@ retry:
 							&did_some_progress);
 			if (page)
 				goto got_pg;
-			if (!did_some_progress)
+			if (!did_some_progress && !(gfp_mask & __GFP_NOFAIL))
 				goto nopage;
 		}
 		/* Wait for some write requests to complete then retry */
Quoted 3 lines

Either way you decide, feel free to add my

Thanks!

Andrew, should I repost or you can pick it up from this thread? Assuming you and others do not have objections of course.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw
In reply to Johannes Weiner
Quoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

---

We haven't seen any bug reports

 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +4 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
              schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+             DEFAULT_RATELIMIT_INTERVAL,
+             DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer

mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802

Quoted 7 lines
mm/oom_kill.c +4 -0
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
      if (!oom_killer_disabled) {
              __out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
              ret = true;
+     } else if (gfp_mask & __GFP_NOFAIL) {
+             if (__ratelimit(&oom_disabled_rs))
+                     WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+             ret = true;

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

Quoted 1 line

WARN_ON_ONCE()?

maybe panic() ?

If somebody turns off oom-killer it seems he's pretty sure that he has enough memory.

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 61 lines · Konstantin Khlebnikov
Quoted 60 lines · Johannes Weiner
Quoted 47 lines · Michal Hocko

Tetsuo Handa has pointed out that __GFP_NOFAIL allocations might fail after OOM killer is disabled if the allocation is performed by a kernel thread. This behavior was introduced from the very beginning by 7f33d49a2ed5 (mm, PM/Freezer: Disable OOM killer when tasks are frozen). This means that the basic contract for the allocation request is broken and the context requesting such an allocation might blow up unexpectedly.

There are basically two ways forward.
1) move oom_killer_disable after kernel threads are frozen. This has a risk that the OOM victim wouldn't be able to finish because it would depend on an already frozen kernel thread. This would be really tricky to debug.
2) do not fail GFP_NOFAIL allocation no matter what and risk a potential Freezable kernel threads will loop and fail the suspend. Incidental allocations after kernel threads are frozen will at least dump a warning - if we are lucky and the serial console is still active of course...

This patch implements the later option because it is safer. We would see warnings rather than allocation failures for the kernel threads which would blow up otherwise and have a higher chances to identify __GFP_NOFAIL users from deeper pm code.

---

We haven't seen any bug reports

 mm/oom_kill.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/oom_kill.c +4 -0
index 642f38cb175a..ea8b443cd871 100644
@@ -772,6 +772,10 @@ out:
              schedule_timeout_killable(1);
 }
 
+static DEFINE_RATELIMIT_STATE(oom_disabled_rs,
+             DEFAULT_RATELIMIT_INTERVAL,
+             DEFAULT_RATELIMIT_BURST);
+
 /**
  * out_of_memory -  tries to invoke OOM killer.
  * @zonelist: zonelist pointer

mm/oom_kill.c bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, · line 802

Quoted 7 lines
mm/oom_kill.c +4 -0
@@ -792,6 +796,10 @@ bool out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask,
      if (!oom_killer_disabled) {
              __out_of_memory(zonelist, gfp_mask, order, nodemask, force_kill);
              ret = true;
+     } else if (gfp_mask & __GFP_NOFAIL) {
+             if (__ratelimit(&oom_disabled_rs))
+                     WARN(1, "Unable to make forward progress for __GFP_NOFAIL because OOM killer is disbaled\n");
+             ret = true;

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

Quoted 6 lines
Quoted 1 line

WARN_ON_ONCE()?

maybe panic() ?

If somebody turns off oom-killer it seems he's pretty sure that he has enough memory.

Ah, that's used in freeze/suspend code. I thought that some kind of sysctl for brave sysadmins.

Quoted 7 lines
Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
In reply to Johannes Weiner

On Tue 24-02-15 14:11:27, Johannes Weiner wrote:
[...]

Quoted 3 lines

I'm fine with keeping the allocation looping, but is that message helpful? It seems completely useless to the user encountering it. Is it going to help kernel developers when we get a bug report with it?

It is better than a silent endless loop. And we get a trace which points to the place which is doing the allocation. We haven't seen any weird crashes during suspend throughout last 6 years so this would be extremely unlikely and hard to reproduce so having the trace sounds useful to me.

Quoted 1 line

WARN_ON_ONCE()?

I do not expect this will spew a lot of messages. But I can live with WARN_ON_ONCE as well.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>