kobject: avoid blocking allocation while holding uevent_sock_mutex
Subsystems (2)
PATCH 1/1 kobject: avoid blocking allocation while holding uevent_sock_mutex
uevent_net_broadcast_untagged() still holds uevent_sock_mutex across netlink_broadcast(). That serializes all untagged uevent senders behind one global mutex, and it also keeps the mutex held while the netlink broadcast path may perform blocking memory allocation.
This becomes visible during USB enumeration, where device_add() sends a KOBJ_ADD uevent from the usb_hub_wq context. If a listener is slow or the netlink broadcast path runs into memory pressure, the sender can sit behind uevent_sock_mutex long enough to trigger hung task reports.
Keep the existing send-side ordering, but move the uevent skb allocation out of the critical section and use GFP_NOWAIT for the broadcast clones performed while uevent_sock_mutex is held. This removes the sleeping allocation point from the locked region without changing uevent delivery semantics.
--- lib/kobject_uevent.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)
lib/kobject_uevent.c +19 -9
@@ -311,9 +311,26 @@ static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env, {struct sk_buff *skb =NULL ;struct uevent_sock *ue_sk; +bool has_listeners = false;int retval =0 ; -/* send netlink message */ +mutex_lock (&uevent_sock_mutex); +list_for_each_entry (ue_sk, &uevent_sock_list, list) { +if (!netlink_has_listeners (ue_sk->sk,1 )) +continue ; + + has_listeners = true; +break ; + } +mutex_unlock (&uevent_sock_mutex); + +if (has_listeners) { + skb =alloc_uevent_skb (env, action_string, devpath); +if (!skb) +return -ENOMEM ; + } + +/* Keep send-side ordering, but avoid sleeping while holding the mutex. */ mutex_lock (&uevent_sock_mutex);list_for_each_entry (ue_sk, &uevent_sock_list, list) {struct sock *uevent_sock = ue_sk->sk;
@@ -321,15 +338,8 @@ static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env,if (!netlink_has_listeners (uevent_sock,1 ))continue ; -if (!skb) { - retval = -ENOMEM ; - skb =alloc_uevent_skb (env, action_string, devpath); -if (!skb) -continue ; - } - retval =netlink_broadcast (uevent_sock,skb_get (skb),0 ,1 , -GFP_KERNEL ); +GFP_NOWAIT );/* ENOBUFS should be handled in userspace */ if (retval == -ENOBUFS || retval == -ESRCH ) retval =0 ;
Signature · 1 line
2.34.1
Greg KH
maintainerQuoted 9 lines · Tao Yu
uevent_net_broadcast_untagged()still holdsuevent_sock_mutexacrossnetlink_broadcast(). That serializes all untagged uevent senders behind one global mutex, and it also keeps the mutex held while the netlink broadcast path may perform blocking memory allocation.This becomes visible during USB enumeration, where
device_add()sends aKOBJ_ADDuevent from theusb_hub_wqcontext. If a listener is slow or the netlink broadcast path runs into memory pressure, the sender can sit behinduevent_sock_mutexlong enough to trigger hung task reports.
What has changed to cause this to happen? USB enumeration is quite common :)
Quoted 7 lines
Keep the existing send-side ordering, but move the uevent skb allocation out of the critical section and use
GFP_NOWAITfor the broadcast clones performed whileuevent_sock_mutexis held. This removes the sleeping allocation point from the locked region without changing uevent delivery semantics.Reported-by [email protected]
No link? Was this tested?
No cc: stable?
No Fixes: tag?
Quoted 16 lines
Signed-off-by Tao Yu <[email protected]>--- lib/kobject_uevent.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)lib/kobject_uevent.c +1
-0@@ -311,9 +311,26 @@ static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env, {struct sk_buff *skb =NULL ;struct uevent_sock *ue_sk; +bool has_listeners = false;int retval =0 ;
lib/kobject_uevent.c static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env, · line 325
Quoted 10 lines
lib/kobject_uevent.c +9
-1-/* send netlink message */ +mutex_lock (&uevent_sock_mutex); +list_for_each_entry (ue_sk, &uevent_sock_list, list) { +if (!netlink_has_listeners (ue_sk->sk,1 )) +continue ; + + has_listeners = true; +break ; + } +mutex_unlock (&uevent_sock_mutex);
guard()?
lib/kobject_uevent.c static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env, · line 327
Quoted 2 lines
lib/kobject_uevent.c +2
-0+ +if (has_listeners) {
What happens if you get a listner right after the lock is released?
Quoted 13 lines
lib/kobject_uevent.c +6
-0+ skb =alloc_uevent_skb (env, action_string, devpath); +if (!skb) +return -ENOMEM ; + } + +/* Keep send-side ordering, but avoid sleeping while holding the mutex. */ mutex_lock (&uevent_sock_mutex);list_for_each_entry (ue_sk, &uevent_sock_list, list) {struct sock *uevent_sock = ue_sk->sk;@@ -321,15 +338,8 @@ static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env,if (!netlink_has_listeners (uevent_sock,1 ))continue ;
lib/kobject_uevent.c static int uevent_net_broadcast_untagged(struct kobj_uevent_env *env, · line 342
Quoted 10 lines
lib/kobject_uevent.c +1
-8-if (!skb) { - retval = -ENOMEM ; - skb =alloc_uevent_skb (env, action_string, devpath); -if (!skb) -continue ; - } - retval =netlink_broadcast (uevent_sock,skb_get (skb),0 ,1 , -GFP_KERNEL ); +GFP_NOWAIT );
This all feels very odd. Again, what has changed to suddenly need this?
thanks,
greg k-h
Tao Yu
Tested-by [email protected]Hi Greg,
Thanks for the review.
Nothing changed in USB enumeration itself. What syzbot appears to be hitting is a long-standing latency problem in the untagged uevent path, not a recent USB-specific regression.
The blocked task is in:
usb_set_configuration()
-> device_add()
-> kobject_uevent_env()
-> uevent_net_broadcast_untagged()so the contention point is the global untagged-uevent broadcast path in lib/kobject_uevent.c.
Looking at the history, the locked allocation/broadcast pattern appears to date back to 16dff336b33d ("kobject: add kobject_uevent_net_broadcast()"). Because of that, I did not add a Fixes tag or cc stable in v1. I do not have evidence that this is a recent regression, so I would rather not add those tags without a stronger justification.
You are also right about the race in my listener pre-check. A listener can appear after the first unlock, so that part of v1 is not correct. I will drop that approach in the next revision.
I also should have included the syzbot link and testing details. The report is here:
https://syzkaller.appspot.com/bug?extid=c22bb42560ec86726aba
syzbot has now tested the proposed patch and reported that the reproducer did not trigger the issue:
https://groups.google.com/g/syzkaller-bugs/c/q6wmyA-hG6E/m/rLDsbLC-AQAJ
So the patch now has:
I also compile-tested it with:
make lib/kobject_uevent.o
Given the race in v1, I will respin this as a narrower fix and include the proper Link/Tested-by information.
Thanks,
Tao
Greg KH
maintainerA: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
A: No.
Q: Should I include quotations after my reply?
http://daringfireball.net/2007/07/on_top
Quoted 15 lines · Tao Yu
Hi Greg,
Thanks for the review.
Nothing changed in USB enumeration itself. What syzbot appears to be hitting is a long-standing latency problem in the untagged uevent path, not a recent USB-specific regression.
The blocked task is in:
usb_set_configuration()-> device_add() -> kobject_uevent_env() -> uevent_net_broadcast_untagged()
Quoted 8 lines
so the contention point is the global untagged-uevent broadcast path in
lib/kobject_uevent.c.Looking at the history, the locked allocation/broadcast pattern appears to date back to
16dff336b33d("kobject: addkobject_uevent_net_broadcast()"). Because of that, I did not add a Fixes tag or cc stable in v1. I do not have evidence that this is a recent regression, so I would rather not add those tags without a stronger justification.
But if this is something that userspace can trigger, as syzbot is showing, then it should be backported to resolve the problem for people hitting it like yourself.
Quoted 12 lines
You are also right about the race in my listener pre-check. A listener can appear after the first unlock, so that part of v1 is not correct. I will drop that approach in the next revision.
I also should have included the syzbot link and testing details. The report is here:
https://syzkaller.appspot.com/bug?extid=c22bb42560ec86726abasyzbot has now tested the proposed patch and reported that the reproducer did not trigger the issue:
https://groups.google.com/g/syzkaller-bugs/c/q6wmyA-hG6E/m/rLDsbLC-AQAJ
Quoted 5 lines
So the patch now has:
Tested-by [email protected]I also compile-tested it with:
makelib/kobject_uevent.o
But that doesn't actually test things locally. Please do so on your own machine as obviously this is a path you can validate yourself.
thanks,
greg k-hh