mm, oom: Give __GFP_NOFAIL allocations access to memory reserves
Subsystems (2)
PATCH 1/1 mm, oom: Give __GFP_NOFAIL allocations access to memory reserves
From: Michal Hocko <[email protected]>
__GFP_NOFAIL is a big hammer used to ensure that the allocation request can never fail. This is a strong requirement and as such it also deserves a special treatment when the system is OOM. The primary problem here is that the allocation request might have come with some locks held and the oom victim might be blocked on the same locks. This is basically an OOM deadlock situation.
This patch tries to reduce the risk of such a deadlocks by giving __GFP_NOFAIL allocations a special treatment and let them dive into memory reserves after oom killer invocation. This should help them to make a progress and release resources they are holding. The OOM victim should compensate for the reserves consumption.
---
Hi,
this has been posted previously as a part of larger GFP_NOFS related
patch set (http://lkml.kernel.org/r/1438768284-30927-1-git-send-email-mhocko%40kernel.org)
but Andrea was asking basically the same thing at LSF early this year
(I cannot seem to find it in any public archive though). I think the
patch makes some sense on its own.
Comments?
mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/page_alloc.c +9 -1
@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ; + +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" ); + } + } out:mutex_unlock (&oom_lock);return page;
Signature · 7 lines
2.6.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Johannes Weiner
reviewerQuoted 16 lines · On Wed, Nov 11, 2015 at 02:48:17PM +0100, [email protected] wrote:
From: Michal Hocko <[email protected]>
__GFP_NOFAILis a big hammer used to ensure that the allocation request can never fail. This is a strong requirement and as such it also deserves a special treatment when the system is OOM. The primary problem here is that the allocation request might have come with some locks held and the oom victim might be blocked on the same locks. This is basically an OOM deadlock situation.This patch tries to reduce the risk of such a deadlocks by giving
__GFP_NOFAILallocations a special treatment and let them dive into memory reserves after oom killer invocation. This should help them to make a progress and release resources they are holding. The OOM victim should compensate for the reserves consumption.Signed-off-by Michal Hocko <[email protected]>
Quoted 8 lines
---
Hi,
this has been posted previously as a part of largerGFP_NOFSrelated
patch set (http://lkml.kernel.org/r/1438768284-30927-1-git-send-email-mhocko%40kernel.org)
but Andrea was asking basically the same thing at LSF early this year
(I cannot seem to find it in any public archive though). I think the
patch makes some sense on its own.
I sent this right after LSF based on Andrea's suggestion:
https://lkml.org/lkml/2015/3/25/37
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 26 lines · Johannes Weiner
Quoted 16 lines · On Wed, Nov 11, 2015 at 02:48:17PM +0100, [email protected] wrote:
From: Michal Hocko <[email protected]>
__GFP_NOFAILis a big hammer used to ensure that the allocation request can never fail. This is a strong requirement and as such it also deserves a special treatment when the system is OOM. The primary problem here is that the allocation request might have come with some locks held and the oom victim might be blocked on the same locks. This is basically an OOM deadlock situation.This patch tries to reduce the risk of such a deadlocks by giving
__GFP_NOFAILallocations a special treatment and let them dive into memory reserves after oom killer invocation. This should help them to make a progress and release resources they are holding. The OOM victim should compensate for the reserves consumption.Signed-off-by Michal Hocko <[email protected]>Quoted 8 lines
---
Hi,
this has been posted previously as a part of largerGFP_NOFSrelated
patch set (http://lkml.kernel.org/r/1438768284-30927-1-git-send-email-mhocko%40kernel.org)
but Andrea was asking basically the same thing at LSF early this year
(I cannot seem to find it in any public archive though). I think the
patch makes some sense on its own.
Quoted 3 lines
I sent this right after LSF based on Andrea's suggestion:
Ohh, I completely forgot as it was part of a larger series. Thanks for the pointer.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · On 11.11.2015 14:48, [email protected] wrote:
mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/page_alloc.c +0
-0@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2776
Quoted 10 lines
mm/page_alloc.c +7
-1/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ; + +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" );
It seems redundant to me to keep the WARN_ON_ONCE also above in the if () part? Also s/gfp_nofail/GFP_NOFAIL/ for consistency?
Hm and probably out of scope of your patch, but I understand the WARN_ONCE (WARN_ON_ONCE) to be _ONCE just to prevent a flood from a single task looping here. But for distinct tasks and potentially far away in time, wouldn't we want to see all the warnings? Would that be feasible to implement?
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2781
Quoted 6 lines
mm/page_alloc.c +2
-0+ } + } out:mutex_unlock (&oom_lock);return page;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Vlastimil Babka
Quoted 14 lines · On 11.11.2015 14:48, [email protected] wrote:
mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/page_alloc.c +1
-1@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ;
Quoted 8 lines
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2776
Quoted 6 lines
mm/page_alloc.c +6
-0+ +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" );It seems redundant to me to keep the
WARN_ON_ONCEalso above in the if () part?
They are warning about two different things. The first one catches a buggy code which uses __GFP_NOFAIL from oom disabled context while the second one tries to help the administrator with a hint that memory reserves are too small.
Quoted 1 line
Also s/gfp_nofail/GFP_NOFAIL/ for consistency?
Fair enough, changed.
Quoted 4 lines
Hm and probably out of scope of your patch, but I understand the
WARN_ONCE(WARN_ON_ONCE) to be_ONCEjust to prevent a flood from a single task looping here. But for distinct tasks and potentially far away in time, wouldn't we want to see all the warnings? Would that be feasible to implement?
I was thinking about that as well some time ago but it was quite hard to find a good enough API to tell when to warn again. The first WARN_ON_ONCE should trigger for all different _code paths_ no matter how frequently they appear to catch all the buggy callers. The second one would benefit from a new warning after min_free_kbytes was updated because it would tell the administrator that the last update was not sufficient for the workload.
Quoted 7 lines
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2781
Quoted 7 lines
mm/page_alloc.c +2
-0+ } + } out:mutex_unlock (&oom_lock);return page;
Thanks!
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 25 lines · Michal Hocko
Quoted 15 lines · Vlastimil Babka
Quoted 14 lines · On 11.11.2015 14:48, [email protected] wrote:
mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/page_alloc.c +1
-1@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ;Quoted 8 lines
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2776
Quoted 6 lines
mm/page_alloc.c +6
-0+ +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" );It seems redundant to me to keep the
WARN_ON_ONCEalso above in the if () part?
Quoted 2 lines
They are warning about two different things. The first one catches a buggy code which uses
__GFP_NOFAILfrom oom disabled context while the
Ah, I see, I misinterpreted what the return values of out_of_memory() mean. But now that I look at its code, it seems to only return false when oom_killer_disabled is set to true. Which is a global thing and nothing to do with the context of the __GFP_NOFAIL allocation?
Quoted 12 lines
second one tries to help the administrator with a hint that memory reserves are too small.
Quoted 1 line
Also s/gfp_nofail/GFP_NOFAIL/ for consistency?
Fair enough, changed.
Quoted 4 lines
Hm and probably out of scope of your patch, but I understand the
WARN_ONCE(WARN_ON_ONCE) to be_ONCEjust to prevent a flood from a single task looping here. But for distinct tasks and potentially far away in time, wouldn't we want to see all the warnings? Would that be feasible to implement?
Quoted 7 lines
I was thinking about that as well some time ago but it was quite hard to find a good enough API to tell when to warn again. The first
WARN_ON_ONCEshould trigger for all different_codepaths_no matter how frequently they appear to catch all the buggy callers. The second one would benefit from a new warning aftermin_free_kbyteswas updated because it would tell the administrator that the last update was not sufficient for the workload.
Hm, what about adding a flag to the struct alloc_context, so that when the particular allocation attempt emits the warning, it sets a flag in the alloc_context so that it won't emit them again as long as it keeps looping and attempting oom. Other allocations will warn independently.
We could also print the same info as the "allocation failed" warnings do, since it's very similar, except we can't fail - but the admin/bug reporter should be interested in the same details as for an allocation failure that is allowed to fail. But it's also true that we have probably just printed the info during out_of_memory()... except when we skipped that for some reason?
Quoted 10 lines
Quoted 7 lines
mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2781
Quoted 7 lines
mm/page_alloc.c +2
-0+ } + } out:mutex_unlock (&oom_lock);return page;Thanks!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 26 lines · Vlastimil Babka
Quoted 25 lines · Michal Hocko
Quoted 23 lines · Vlastimil Babka
Quoted 10 lines · On 11.11.2015 14:48, [email protected] wrote:
mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/page_alloc.c +0
-0@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2776
Quoted 10 lines
mm/page_alloc.c +7
-1/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ; + +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" );It seems redundant to me to keep the
WARN_ON_ONCEalso above in the if () part?
Quoted 7 lines
Quoted 2 lines
They are warning about two different things. The first one catches a buggy code which uses
__GFP_NOFAILfrom oom disabled context while theAh, I see, I misinterpreted what the return values of
out_of_memory()mean. But now that I look at its code, it seems to only return false whenoom_killer_disabledis set to true. Which is a global thing and nothing to do with the context of the__GFP_NOFAILallocation?
I am not sure I follow you here. The point of the warning is to warn when the oom killer is disbaled (out_of_memory returns false) _and_ the request is __GFP_NOFAIL because we simply cannot guarantee any forward progress and just a use of the allocation flag is not supproted.
[...]
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
Hm and probably out of scope of your patch, but I understand the
WARN_ONCE(WARN_ON_ONCE) to be_ONCEjust to prevent a flood from a single task looping here. But for distinct tasks and potentially far away in time, wouldn't we want to see all the warnings? Would that be feasible to implement?Quoted 7 lines
I was thinking about that as well some time ago but it was quite hard to find a good enough API to tell when to warn again. The first
WARN_ON_ONCEshould trigger for all different_codepaths_no matter how frequently they appear to catch all the buggy callers. The second one would benefit from a new warning aftermin_free_kbyteswas updated because it would tell the administrator that the last update was not sufficient for the workload.
Quoted 4 lines
Hm, what about adding a flag to the struct
alloc_context, so that when the particular allocation attempt emits the warning, it sets a flag in thealloc_contextso that it won't emit them again as long as it keeps looping and attempting oom. Other allocations will warn independently.
That could still trigger a flood of messages. Say you have many concurrent users from the same call path...
I am not really sure making the code more complicating for this warning is really worth it. If anything we can use ratelimited variant.
Quoted 6 lines
We could also print the same info as the "allocation failed" warnings do, since it's very similar, except we can't fail - but the admin/bug reporter should be interested in the same details as for an allocation failure that is allowed to fail. But it's also true that we have probably just printed the info during
out_of_memory()... except when we skipped that for some reason?
The first WARN_ON_ONCE happens when OOM killer doesn't trigger so a memory situation might be worth considering. The later one might have seen the OOM report which is the likely case. So if anyting the first one should dump the info.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 28 lines · Michal Hocko
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 7 lines
mm/page_alloc.c +0
-0@@ -2766,8 +2766,16 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, goto out; }mm/page_alloc.c __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, · line 2776
Quoted 10 lines
mm/page_alloc.c +7
-1/* Exhausted what can be done so it's blamo time */ -if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) +if (out_of_memory (&oc) ||WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL)) { *did_some_progress =1 ; + +if (gfp_mask & __GFP_NOFAIL) { + page =get_page_from_freelist (gfp_mask, order, +ALLOC_NO_WATERMARKS |ALLOC_CPUSET , ac); +WARN_ONCE (!page,"Unable to fullfil gfp_nofail allocation." +" Consider increasing min_free_kbytes.\n" );It seems redundant to me to keep the
WARN_ON_ONCEalso above in the if () part?Quoted 7 lines
Quoted 2 lines
They are warning about two different things. The first one catches a buggy code which uses
__GFP_NOFAILfrom oom disabled context while theAh, I see, I misinterpreted what the return values of
out_of_memory()mean. But now that I look at its code, it seems to only return false whenoom_killer_disabledis set to true. Which is a global thing and nothing to do with the context of the__GFP_NOFAILallocation?
Quoted 5 lines
I am not sure I follow you here. The point of the warning is to warn when the oom killer is disbaled (
out_of_memoryreturns false) _and_ the request is__GFP_NOFAILbecause we simply cannot guarantee any forward progress and just a use of the allocation flag is not supproted.
I don't think the WARN_ONCE() above is helpful for a few reasons:
- it suggests that min_free_kbytes is the best way to work around such issues and gives kernel developers a free pass to just say "raise min_free_kbytes" rather than reducing their reliance on __GFP_NOFAIL,
- raising min_free_kbytes is not immediately actionable without memory freeing to fix any oom issue, and
- it relies on the earlier warning to dump the state of memory and doesn't add any significant information to help understand how seperate occurrences are similar or different.
I think the WARN_ONCE() should just be removed.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · David Rientjes
On Mon, 23 Nov 2015, Michal Hocko wrote:
[...]
Quoted 11 lines
Quoted 5 lines
I am not sure I follow you here. The point of the warning is to warn when the oom killer is disbaled (
out_of_memoryreturns false) _and_ the request is__GFP_NOFAILbecause we simply cannot guarantee any forward progress and just a use of the allocation flag is not supproted.I don't think the
WARN_ONCE()above is helpful for a few reasons:- it suggests that
min_free_kbytesis the best way to work around such issues and gives kernel developers a free pass to just say "raisemin_free_kbytes" rather than reducing their reliance on__GFP_NOFAIL,
I disagree. Users are quite sensitive to warnings with backtraces in the log from my experience and they report them. And while the log shows the code path which triggers the issue which can help us to change the code it also gives a useful hint on how to reduce this issue until we are able to either fix a bug or a permanent configuration if we are not able to get rid of it for whatever reason.
Besides that there is no other reliable warning that we are getting _really_ short on memory unlike when the allocation failure is allowed. OOM killer report might be missing because there was no actual killing happening.
Quoted 2 lines
- raising
min_free_kbytesis not immediately actionable without memory freeing to fix any oom issue, and
true but it can be done to reduce chances for the issue to reappear.
Quoted 3 lines
- it relies on the earlier warning to dump the state of memory and doesn't add any significant information to help understand how seperate occurrences are similar or different.
The information is quite valuable even without OOM killer report IMHO.
Quoted 1 line
I think the
WARN_ONCE()should just be removed.
I do not insist on keeping it but I really think it might be useful while it doesn't seem to cause any confusion IMHO. So unless there is a strong reason to not include it I would prefer keeping it.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Johannes Weiner
reviewerQuoted 4 lines · Michal Hocko
Besides that there is no other reliable warning that we are getting _really_ short on memory unlike when the allocation failure is allowed. OOM killer report might be missing because there was no actual killing happening.
This is why I would like to see that warning generalized, and not just for __GFP_NOFAIL. We have allocations other than explicit __GFP_NOFAIL that will loop forever in the allocator, and when this deadlocks the machine all we see is other tasks hanging, but not the culprit. If we were to get a backtrace of some task in the allocator that is known to hold locks, suddenly all the other hung tasks will make sense, and it will clearly distinguish such an allocator deadlock from other issues.
Do you remember the patch you proposed at LSF about failing requests after looping a certain (configurable) number of times? Well, instead of failing them, it would be good to start WARNING after a certain # of loops when we know we won't quit (implicit or explicit NOFAIL).
[ Kind of like fs/xfs/kmem::kmem_alloc() does, only that that is
currently dead code due to our looping inside the allocator. ]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Johannes Weiner
Quoted 4 lines · Michal Hocko
Besides that there is no other reliable warning that we are getting _really_ short on memory unlike when the allocation failure is allowed. OOM killer report might be missing because there was no actual killing happening.
This is why I would like to see that warning generalized, and not just for
__GFP_NOFAIL. We have allocations other than explicit__GFP_NOFAILthat will loop forever in the allocator,
Yes but does it make sense to warn for all of them? Wouldn't it be sufficient to warn about those which cannot allocate anything even though they are doing ALLOC_NO_WATERMARKS? We could still hint the administrator to increase min_free_kbytes for his particular workload. Such a situation should be really exceptional and warn_once should be sufficient.
Quoted 5 lines
and when this deadlocks the
machine all we see is other tasks hanging, but not the culprit. If we were to get a backtrace of some task in the allocator that is known to hold locks, suddenly all the other hung tasks will make sense, and it will clearly distinguish such an allocator deadlock from other issues.
Tetsuo was suggesting a more sophisticated infrastructure for tracking allocations [1] which take too long without making progress. I haven't seen his patch because I was too busy with other stuff but maybe this is what you would like to see?
Anyway I would like to make some progress on this patch. Do you think that it would be acceptable in the current form without the warning or you preffer a different way?
[1] http://lkml.kernel.org/r/201510182105.AGA00839.FHVFFStLQOMOOJ%40I-love.SAKURA.ne.jp
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Johannes Weiner
reviewerQuoted 7 lines · Michal Hocko
Quoted 6 lines · Johannes Weiner
Quoted 4 lines · Michal Hocko
Besides that there is no other reliable warning that we are getting _really_ short on memory unlike when the allocation failure is allowed. OOM killer report might be missing because there was no actual killing happening.
Quoted 7 lines
Quoted 3 lines
This is why I would like to see that warning generalized, and not just for
__GFP_NOFAIL. We have allocations other than explicit__GFP_NOFAILthat will loop forever in the allocator,Yes but does it make sense to warn for all of them? Wouldn't it be sufficient to warn about those which cannot allocate anything even though they are doing
ALLOC_NO_WATERMARKS?
Why is it important whether they can do ALLOC_NO_WATERMARKS or not?
I'm worried about all those that can loop forever with locks held.
Quoted 10 lines
Quoted 5 lines
and when this deadlocks the
machine all we see is other tasks hanging, but not the culprit. If we were to get a backtrace of some task in the allocator that is known to hold locks, suddenly all the other hung tasks will make sense, and it will clearly distinguish such an allocator deadlock from other issues.Tetsuo was suggesting a more sophisticated infrastructure for tracking allocations [1] which take too long without making progress. I haven't seen his patch because I was too busy with other stuff but maybe this is what you would like to see?
That seems a bit excessive. I was thinking something more like this:
mm/page_alloc.c +4 -0
@@ -3004,6 +3004,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,enum migrate_mode migration_mode =MIGRATE_ASYNC ;bool deferred_compaction = false;int contended_compaction =COMPACT_CONTENDED_NONE ; +unsigned int nr_tries =0 ;/* * In the slowpath, we sanity check order to avoid ever trying to
@@ -3033,6 +3034,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, goto nopage; retry: +if (++nr_retries %100 ==0 ) +warn_alloc_failed (gfp_mask, order,"Potential GFP deadlock\n" ); +if (gfp_mask & __GFP_KSWAPD_RECLAIM)wake_all_kswapds (order, ac);
Quoted 3 lines
Anyway I would like to make some progress on this patch. Do you think that it would be acceptable in the current form without the warning or you preffer a different way?
Oh, I have nothing against your patch, please go ahead with it. I just wondered out loud when you proposed a warning about deadlocking NOFAIL allocations but limited it to explicit __GFP_NOFAIL allocations, when those obviously aren't the only ones that can deadlock in that way.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Johannes Weiner
Quoted 11 lines · Michal Hocko
Quoted 9 lines · Johannes Weiner
Quoted 4 lines · Michal Hocko
Besides that there is no other reliable warning that we are getting _really_ short on memory unlike when the allocation failure is allowed. OOM killer report might be missing because there was no actual killing happening.
This is why I would like to see that warning generalized, and not just for
__GFP_NOFAIL. We have allocations other than explicit__GFP_NOFAILthat will loop forever in the allocator,
Quoted 5 lines
Quoted 3 lines
Yes but does it make sense to warn for all of them? Wouldn't it be sufficient to warn about those which cannot allocate anything even though they are doing
ALLOC_NO_WATERMARKS?Why is it important whether they can do
ALLOC_NO_WATERMARKSor not?
Well, the idea was that ALLOC_NO_WATERMARKS failures mean that memory reserves are not sufficient to handle given workload. min_free_kbytes is auto-tuned and it might be not sufficient - especially now that we are adding a new class of consumers of the reserves. I find a warning as an appropriate way to tell administrator that the auto-tuning was too optimistic for the particular load.
Quoted 1 line
I'm worried about all those that can loop forever with locks held.
I can see your point here but I think that looping endlessly without any progress is a different class of issue. It is hard to tune for it. You can change tunning and still can end up looping because the lock vs. reclaim dependencies will be still there.
Quoted 26 lines
Quoted 10 lines
Quoted 5 lines
and when this deadlocks the
machine all we see is other tasks hanging, but not the culprit. If we were to get a backtrace of some task in the allocator that is known to hold locks, suddenly all the other hung tasks will make sense, and it will clearly distinguish such an allocator deadlock from other issues.Tetsuo was suggesting a more sophisticated infrastructure for tracking allocations [1] which take too long without making progress. I haven't seen his patch because I was too busy with other stuff but maybe this is what you would like to see?
That seems a bit excessive. I was thinking something more like this:
mm/page_alloc.c +1
-0@@ -3004,6 +3004,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,enum migrate_mode migration_mode =MIGRATE_ASYNC ;bool deferred_compaction = false;int contended_compaction =COMPACT_CONTENDED_NONE ; +unsigned int nr_tries =0 ;/* * In the slowpath, we sanity check order to avoid ever trying to
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 3039
Quoted 6 lines
mm/page_alloc.c +3
-0@@ -3033,6 +3034,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, goto nopage; retry: +if (++nr_retries %100 ==0 ) +warn_alloc_failed (gfp_mask, order,"Potential GFP deadlock\n" ); +
I am not against this in principle. It might be too noisy but warn_alloc_failed is throttled already so it should handle too many parallel requests already. I still think that ALLOC_NO_WATERMARKS failures deserve a special treatment because we can tune for that. Care to send a patch?
Quoted 11 lines
mm/page_alloc.c +0
-0if (gfp_mask & __GFP_KSWAPD_RECLAIM)wake_all_kswapds (order, ac);Quoted 3 lines
Anyway I would like to make some progress on this patch. Do you think that it would be acceptable in the current form without the warning or you preffer a different way?
Oh, I have nothing against your patch, please go ahead with it. I just wondered out loud when you proposed a warning about deadlocking NOFAIL allocations but limited it to explicit
__GFP_NOFAILallocations, when those obviously aren't the only ones that can deadlock in that way.
As mentioned above I have added it merely because this would be a new consumer of the reserves which might lead to its depletion without an explicit warning. But the more I think about that the more I am convinced that this should be generalized to ALLOC_NO_WATERMARKS.
I will remove the warning from the patch and prepare a separate one which will warn ALLOC_NO_WATERMARKS so that we can discuss that separately.
Thanks
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>