net_sched: fix RTNL deadlock again caused by request_module()
Subsystems (3)
PATCH 1/1 net_sched: fix RTNL deadlock again caused by request_module()
From: Cong Wang <[email protected]>
tcf_action_init_1() loads tc action modules automatically with request_module() after parsing the tc action names, and it drops RTNL lock and re-holds it before and after request_module(). This causes a lot of troubles, as discovered by syzbot, because we can be in the middle of batch initializations when we create an array of tc actions.
One of the problem is deadlock:
CPU 0 CPU 1rtnl_lock();
for (...) {
tcf_action_init_1();
-> rtnl_unlock();
-> request_module();
rtnl_lock();
for (...) {
tcf_action_init_1();
-> tcf_idr_check_alloc();
// Insert one action into idr,
// but it is not committed until
// tcf_idr_insert_many(), then drop
// the RTNL lock in the _next_
// iteration
-> rtnl_unlock();
-> rtnl_lock();
-> a_o->init();
-> tcf_idr_check_alloc();
// Now waiting for the same index
// to be committed
-> request_module();
-> rtnl_lock()
// Now waiting for RTNL lock
}
rtnl_unlock();
}rtnl_unlock();
This is not easy to solve, we can move the request_module() before this loop and pre-load all the modules we need for this netlink message and then do the rest initializations. So the loop breaks down to two now:
for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) {
struct tc_action_ops *a_o;
a_o = tc_action_load_ops(name, tb[i]...);
ops[i - 1] = a_o;
}
for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) {
act = tcf_action_init_1(ops[i - 1]...);
}Although this looks serious, it only has been reported by syzbot, so it seems hard to trigger this by humans. And given the size of this patch, I'd suggest to make it to net-next and not to backport to stable.
This patch has been tested by syzbot and tested with tdc.py by me.
--- include/net/act_api.h | 5 +- net/sched/act_api.c | 104 +++++++++++++++++++++++++++--------------- net/sched/cls_api.c | 11 ++++- 3 files changed, 79 insertions(+), 41 deletions(-)
include/net/act_api.h +4 -1
@@ -186,10 +186,13 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, char *name, int ovr, int bind, struct tc_action *actions[], size_t *attr_size, bool rtnl_held, struct netlink_ext_ack *extack); +struct tc_action_ops *tc_action_load_ops (char *name,struct nlattr *nla, +bool rtnl_held, +struct netlink_ext_ack *extack);struct tc_action *tcf_action_init_1 (struct net *net,struct tcf_proto *tp,struct nlattr *nla,struct nlattr *est,char *name,int ovr,int bind, -bool rtnl_held, +struct tc_action_ops *ops,bool rtnl_held,struct netlink_ext_ack *extack);int tcf_action_dump (struct sk_buff *skb,struct tc_action *actions[],int bind,int ref,bool terse);
net/sched/act_api.c +66 -38
177 lines not on this page · raw message
net/sched/cls_api.c +9 -2
@@ -3043,12 +3043,19 @@ int tcf_exts_validate(struct net *net, struct tcf_proto *tp, struct nlattr **tb,size_t attr_size =0 ;if (exts->police && tb[exts->police]) { +struct tc_action_ops *a_o; + + a_o =tc_action_load_ops ("police" , tb[exts->police], rtnl_held, extack); +if (IS_ERR (a_o)) +return PTR_ERR (a_o); act =tcf_action_init_1 (net, tp, tb[exts->police], rate_tlv,"police" , ovr, -TCA_ACT_BIND , rtnl_held, +TCA_ACT_BIND , a_o, rtnl_held, extack); -if (IS_ERR (act)) +if (IS_ERR (act)) { +module_put (a_o->owner);return PTR_ERR (act); + } act->type = exts->type =TCA_OLD_COMPAT ; exts->actions[0 ] = act;
Signature · 1 line
2.25.1
Quoted 55 lines · Cong Wang
From: Cong Wang <[email protected]>
tcf_action_init_1()loads tc action modules automatically withrequest_module()after parsing the tc action names, and it drops RTNL lock and re-holds it before and afterrequest_module(). This causes a lot of troubles, as discovered by syzbot, because we can be in the middle of batch initializations when we create an array of tc actions.One of the problem is deadlock:
CPU 0 CPU 1
rtnl_lock();for (...) { tcf_action_init_1(); -> rtnl_unlock(); -> request_module(); rtnl_lock(); for (...) { tcf_action_init_1(); -> tcf_idr_check_alloc(); // Insert one action into idr, // but it is not committed until // tcf_idr_insert_many(), then drop // the RTNL lock in the _next_ // iteration -> rtnl_unlock(); -> rtnl_lock(); -> a_o->init(); -> tcf_idr_check_alloc(); // Now waiting for the same index // to be committed -> request_module(); -> rtnl_lock() // Now waiting for RTNL lock } rtnl_unlock(); }
rtnl_unlock();This is not easy to solve, we can move the
request_module()before this loop and pre-load all the modules we need for this netlink message and then do the rest initializations. So the loop breaks down to two now:for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) { struct tc_action_ops *a_o; a_o = tc_action_load_ops(name, tb[i]...); ops[i - 1] = a_o; } for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) { act = tcf_action_init_1(ops[i - 1]...); }
Quoted 6 lines
Although this looks serious, it only has been reported by syzbot, so it seems hard to trigger this by humans. And given the size of this patch, I'd suggest to make it to net-next and not to backport to stable.
This patch has been tested by syzbot and tested with
tdc.pyby me.
LGTM.
Initially i was worried about performance impact but i found nothing observable. We need to add a tdc test for batch (I can share how i did batch testing at next meet).
cheers,
jamal
Quoted 59 lines · Jamal Hadi Salim
Quoted 47 lines · Cong Wang
From: Cong Wang <[email protected]>
tcf_action_init_1()loads tc action modules automatically withrequest_module()after parsing the tc action names, and it drops RTNL lock and re-holds it before and afterrequest_module(). This causes a lot of troubles, as discovered by syzbot, because we can be in the middle of batch initializations when we create an array of tc actions. One of the problem is deadlock:
CPU 0 CPU 1rtnl_lock();for (...) { tcf_action_init_1(); -> rtnl_unlock(); -> request_module(); rtnl_lock(); for (...) { tcf_action_init_1(); -> tcf_idr_check_alloc(); // Insert one action into idr, // but it is not committed until // tcf_idr_insert_many(), then drop // the RTNL lock in the _next_ // iteration -> rtnl_unlock(); -> rtnl_lock(); -> a_o->init(); -> tcf_idr_check_alloc(); // Now waiting for the same index // to be committed -> request_module(); -> rtnl_lock() // Now waiting for RTNL lock } rtnl_unlock(); }
rtnl_unlock();
This is not easy to solve, we can move therequest_module()before this loop and pre-load all the modules we need for this netlink message and then do the rest initializations. So the loop breaks down to two now:for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) { struct tc_action_ops *a_o; a_o = tc_action_load_ops(name, tb[i]...); ops[i - 1] = a_o; } for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) { act = tcf_action_init_1(ops[i - 1]...); }Quoted 5 lines
Although this looks serious, it only has been reported by syzbot, so it seems hard to trigger this by humans. And given the size of this patch, I'd suggest to make it to net-next and not to backport to stable. This patch has been tested by syzbot and tested with
tdc.pyby me.LGTM.
Initially i was worried about performance impact but i found nothing observable. We need to add a tdc test for batch (I can share how i did batch testing at next meet).
Quoted 5 lines
Tested-by Jamal Hadi Salim <[email protected]>Acked-by Jamal Hadi Salim <[email protected]>cheers,
jamal
Hi,
Thanks for adding me to the thread!
I ran our performance tests with the patch applied and didn't observe any regression.
Regards,
Vlad
This patch was applied to netdev/net-next.git (refs/heads/master):
Hello:
This patch was applied to netdev/net-next.git (refs/heads/master):
Quoted 9 lines · On Sat, 16 Jan 2021 16:56:57 -0800 you wrote:
From: Cong Wang <[email protected]>
tcf_action_init_1()loads tc action modules automatically withrequest_module()after parsing the tc action names, and it drops RTNL lock and re-holds it before and afterrequest_module(). This causes a lot of troubles, as discovered by syzbot, because we can be in the middle of batch initializations when we create an array of tc actions.[...]
Here is the summary with links:
- [net-next] net_sched: fix RTNL deadlock again caused by request_module()
https://git.kernel.org/netdev/net-next/c/d349f9976868
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html