mm: ratelimit oversized kvmalloc warnings instead of once
Subsystems (2)
PATCH 1/1 mm: ratelimit oversized kvmalloc warnings instead of once
At the moment oversize kvmalloc warnings are triggered once using WARN_ON_ONCE() macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replace WARN_ON_ONCE with WARN_RATELIMIT.
--- mm/util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/util.c +2 -1
@@ -649,7 +649,8 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)/* Don't even allow crazy sizes */ if (unlikely (size >INT_MAX )) { -WARN_ON_ONCE (!(flags & __GFP_NOWARN)); +WARN_RATELIMIT (!(flags & __GFP_NOWARN),"size = %zu > INT_MAX" , + size);return NULL ; }
Signature · 1 line
2.43.0
Linus Torvalds
NAKQuoted 2 lines · Shakeel Butt
Simply replace
WARN_ON_ONCEwithWARN_RATELIMIT.
NAK.
Sadly, the RATELIMIT cases are useless.
The normal rate limiting is basically "burst of up to ten, every five seconds".
That's going to completely swamp things and hide any other issue.
If we ratelimit it to "at most 1 per hour", maybe something like that would be acceptable.
But honestly, I do not understand your "first abuser only" complaint. There should not be *any* abusers. So just fix that first one already.
If you have more than one, you have bigger issues. So what is the real reason for this broken patch? Why didn't you fix the first one?
Linus
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Shakeel Butt
Simply replace
WARN_ON_ONCEwithWARN_RATELIMIT.NAK.
Quoted 8 lines
Sadly, the RATELIMIT cases are useless.
The normal rate limiting is basically "burst of up to ten, every five seconds".
That's going to completely swamp things and hide any other issue.
If we ratelimit it to "at most 1 per hour", maybe something like that would be acceptable.
"at most 1 per hour" sounds good.
Quoted 7 lines
But honestly, I do not understand your "first abuser only" complaint. There should not be *any* abusers. So just fix that first one already.
If you have more than one, you have bigger issues. So what is the real reason for this broken patch? Why didn't you fix the first one?
The issue is the turnaround time to fix the first abuser and deploy the fixed kernel to big enough fleet to find the remaining abusers. Usually this turnaround time is in months.
Michal Hocko
reviewerQuoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.
Long time ago, I've had a patch to do the once_per_callsite WARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.
Quoted 22 lines
Reported-by Kyle McMartin <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/util.c +2
-1@@ -649,7 +649,8 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)/* Don't even allow crazy sizes */ if (unlikely (size >INT_MAX )) { -WARN_ON_ONCE (!(flags & __GFP_NOWARN)); +WARN_RATELIMIT (!(flags & __GFP_NOWARN),"size = %zu > INT_MAX" , + size);return NULL ; }Signature · 1 line
2.43.0
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.
Quoted 6 lines
Long time ago, I've had a patch to do the
once_per_callsiteWARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.
Do you mean something like warn once per unique call stack? If yes then I think that is better than the simple ratelimiting version as ratelimiting one may still miss some abusers and also may keep warning about the same abuser. Please do share your patch.
Thanks,
Shakeel
Michal Hocko
reviewerQuoted 11 lines · Shakeel Butt
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.
Quoted 8 lines
Quoted 6 lines
Long time ago, I've had a patch to do the
once_per_callsiteWARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.Do you mean something like warn once per unique call stack?
Exactly!
Quoted 4 lines
If yes then
I think that is better than the simple ratelimiting version as ratelimiting one may still miss some abusers and also may keep warning about the same abuser. Please do share your patch.
https://lore.kernel.org/all/[email protected]/
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · Michal Hocko
Quoted 11 lines · Shakeel Butt
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Quoted 8 lines
Quoted 6 lines
Long time ago, I've had a patch to do the
once_per_callsiteWARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.Do you mean something like warn once per unique call stack?
Quoted 8 lines
Exactly!
Quoted 4 lines
If yes then
I think that is better than the simple ratelimiting version as ratelimiting one may still miss some abusers and also may keep warning about the same abuser. Please do share your patch.
Do you want to propose this patch again (after rebase to latest) or should I take over?
Michal Hocko
reviewerQuoted 21 lines · Michal Hocko
Quoted 11 lines · Shakeel Butt
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Quoted 8 lines
Quoted 6 lines
Long time ago, I've had a patch to do the
once_per_callsiteWARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.Do you mean something like warn once per unique call stack?
Quoted 8 lines
Exactly!
Quoted 4 lines
If yes then
I think that is better than the simple ratelimiting version as ratelimiting one may still miss some abusers and also may keep warning about the same abuser. Please do share your patch.
Btw. the code has changed a lot since 2017 when this was posted so it will likely need a lot of massaging to rebase. Also I am not entirely sure it is ok to change WARN_ONCE semantic like that anymore. Maybe we need an explicit variant that does this per-call-site warnings. It is a notable difference between library functions which can be called from different callpaths and those that are used from a single place. I do not have much time to dig deeper into this but if you want to take over then go ahead. I still think this is a useful WARN_ONCE or in general do_something_once semantic.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 31 lines · Michal Hocko
Quoted 21 lines · Michal Hocko
Quoted 11 lines · Shakeel Butt
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Quoted 8 lines
Quoted 6 lines
Long time ago, I've had a patch to do the
once_per_callsiteWARN. I cannot find reference at the moment but it used stack depot to note stacks that have already triggered. Back then there was no reponse on the ML. Should I try to dig deep and recover it from my archives? I think this is exactly kind of usecase where it would fit.Do you mean something like warn once per unique call stack?
Quoted 8 lines
Exactly!
Quoted 4 lines
If yes then
I think that is better than the simple ratelimiting version as ratelimiting one may still miss some abusers and also may keep warning about the same abuser. Please do share your patch.
Quoted 9 lines
Btw. the code has changed a lot since 2017 when this was posted so it will likely need a lot of massaging to rebase. Also I am not entirely sure it is ok to change
WARN_ONCEsemantic like that anymore. Maybe we need an explicit variant that does this per-call-site warnings. It is a notable difference between library functions which can be called from different callpaths and those that are used from a single place. I do not have much time to dig deeper into this but if you want to take over then go ahead. I still think this is a usefulWARN_ONCEor in generaldo_something_oncesemantic.
I think a separate variant like WARN_UNIQUE() would be better. I will look into this.
Linus, please let me know if you have any concerns on the approach Michal is suggesting i.e. a variant for warn once for unique call stack.
Shakeel
Quoted 3 lines · Shakeel Butt
Linus, please let me know if you have any concerns on the approach Michal is suggesting i.e. a variant for warn once for unique call stack.
I think we should just try to change the existing WARN_ONCE(), and see if it causes any issues.
A new "WARN_UNIQUE()" might be the borign and safe approach, but
(a) it won't actually be unique if you don't have stackdepot anyway,
and will just be WARN_ONCE
(b) I suspect most WARN_ONCE users really do want WARN_UNIQUE
so let's at least _start_ with just changing semantics of the existing "once", and then if it causes problems we'll have to revisit this.
I doubt it will cause problems,
Linus
Michal Hocko
reviewerQuoted 13 lines · Linus Torvalds
Quoted 3 lines · Shakeel Butt
Linus, please let me know if you have any concerns on the approach Michal is suggesting i.e. a variant for warn once for unique call stack.
I think we should just try to change the existing
WARN_ONCE(), and see if it causes any issues.A new "
WARN_UNIQUE()" might be the borign and safe approach, but(a) it won't actually be unique if you don't have stackdepot anyway,
and will just beWARN_ONCE
Quoted 6 lines
(b) I suspect most
WARN_ONCEusers really do wantWARN_UNIQUEso let's at least _start_ with just changing semantics of the existing "once", and then if it causes problems we'll have to revisit this.
I doubt it will cause problems,
I would be careful about the WARN_ONCE used from stackdepot itself. It's been some time since I have looked into that code but a quick grep tells there is some usage.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 20 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Reported-by Kyle McMartin <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/util.c +0
-0
mm/util.c void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) · line 656
Quoted 9 lines
mm/util.c +2
-1@@ -649,7 +649,8 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)/* Don't even allow crazy sizes */ if (unlikely (size >INT_MAX )) { -WARN_ON_ONCE (!(flags & __GFP_NOWARN)); +WARN_RATELIMIT (!(flags & __GFP_NOWARN),"size = %zu > INT_MAX" , + size);return NULL ; }
I don't think this is necessary. From the description I think interested parties can get away with bpftrace.
Suppose you have an abuser of the sort and you are worried there is more than one.
Then this one-liner will catch *all* of them, not just the ones which
were "lucky" to get logged with ratelimit:
bpftrace -e 'kprobe:kvmalloc_node_noprof /arg0 > 2147483647/ { @[kstack()] = count(); }'
Of course adding a probe is not free, but then again kvmalloc should not be used often to begin with so I doubt it is going to have material impact in terms of performance.
While I concede it takes more effort to get this running on all affected machines, the result is much better than mere ratelimit. Also there is no need to patch the kernel.
btw, I found drm keeps spamming kvmalloc, someone(tm) should look into it:
@[
kvmalloc_node_noprof+5
drm_property_create_blob+76
drm_atomic_helper_dirtyfb+234
drm_fbdev_generic_helper_fb_dirty+509
drm_fb_helper_damage_work+139
process_one_work+376
worker_thread+753
kthread+207
ret_from_fork+49
ret_from_fork_asm+26, 104]: 12
Quoted 63 lines · Mateusz Guzik
Quoted 20 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Reported-by Kyle McMartin <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/util.c +0
-0mm/util.c void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) · line 656
Quoted 9 lines
mm/util.c +2
-1@@ -649,7 +649,8 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)/* Don't even allow crazy sizes */ if (unlikely (size >INT_MAX )) { -WARN_ON_ONCE (!(flags & __GFP_NOWARN)); +WARN_RATELIMIT (!(flags & __GFP_NOWARN),"size = %zu > INT_MAX" , + size);return NULL ; }I don't think this is necessary. From the description I think interested parties can get away with bpftrace.
Suppose you have an abuser of the sort and you are worried there is more than one.
Then this one-liner will catch *all* of them, not just the ones which
were "lucky" to get logged with ratelimit:
bpftrace -e 'kprobe:kvmalloc_node_noprof /arg0 > 2147483647/ { @[kstack()] =count(); }'Of course adding a probe is not free, but then again kvmalloc should not be used often to begin with so I doubt it is going to have material impact in terms of performance.
While I concede it takes more effort to get this running on all affected machines, the result is much better than mere ratelimit. Also there is no need to patch the kernel.
btw, I found drm keeps spamming kvmalloc, someone(tm) should look into it:
@[kvmalloc_node_noprof+5 drm_property_create_blob+76 drm_atomic_helper_dirtyfb+234 drm_fbdev_generic_helper_fb_dirty+509 drm_fb_helper_damage_work+139 process_one_work+376 worker_thread+753 kthread+207 ret_from_fork+49 ret_from_fork_asm+26
Quoted 1 line
, 104]: 12
I should clarify this is allocs of 104 bytes, not some outlandish size.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 41 lines · Mateusz Guzik
Quoted 20 lines · Shakeel Butt
At the moment oversize kvmalloc warnings are triggered once using
WARN_ON_ONCE()macro. One issue with this approach is that it only detects the first abuser and then ignores the remaining abusers which complicates detecting all such abusers in a timely manner. The situation becomes worse when the repro has low probability and requires production traffic and thus require large set of machines to find such abusers. In Mera production, this warn once is slowing down the detection of these abusers. Simply replaceWARN_ON_ONCEwithWARN_RATELIMIT.Reported-by Kyle McMartin <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/util.c +0
-0mm/util.c void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) · line 656
Quoted 9 lines
mm/util.c +2
-1@@ -649,7 +649,8 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)/* Don't even allow crazy sizes */ if (unlikely (size >INT_MAX )) { -WARN_ON_ONCE (!(flags & __GFP_NOWARN)); +WARN_RATELIMIT (!(flags & __GFP_NOWARN),"size = %zu > INT_MAX" , + size);return NULL ; }I don't think this is necessary. From the description I think interested parties can get away with bpftrace.
Suppose you have an abuser of the sort and you are worried there is more than one.
Then this one-liner will catch *all* of them, not just the ones which
were "lucky" to get logged with ratelimit:
bpftrace -e 'kprobe:kvmalloc_node_noprof /arg0 > 2147483647/ { @[kstack()] =count(); }'
Quoted 8 lines
Of course adding a probe is not free, but then again kvmalloc should not be used often to begin with so I doubt it is going to have material impact in terms of performance.
While I concede it takes more effort to get this running on all affected machines, the result is much better than mere ratelimit. Also there is no need to patch the kernel.
Thanks for the response and suggestion. I am inclined towards warn once for each unique stack trace as suggested by Michal because I think it would be useful in general.