tipc: fix NULL deref in tipc_named_node_up() on empty publication list
Subsystems (3)
v6 cover tipc: fix NULL deref in tipc_named_node_up() on empty publication list
This series continues the fix for the NULL dereference in tipc_named_node_up() on an empty publication list.
Patch 1/3 carries Tung Nguyen's defer-to-workqueue approach, suggested by Jon Maloy on the thread as the replacement for the item-less bulk from v2. Tung's RFC only exists as an inline diff in the thread, so I folded it into this series with his Signed-off-by kept. I tested it in our two-node QEMU setup (veth pair, UDP bearers, node-id addressing), both with an unprivileged user namespace and as root: the unpatched kernel panics on the first run of the same reproducer, the patched one distributes normally with a non-empty list.
While testing we found two residual issues in the approach, fixed by patches 2/3 and 3/3.
Patch 2/3: tipc_net_finalize() does not check the return value of tipc_nametbl_publish(). If the publish fails, for example on a GFP_ATOMIC allocation failure, the node is finalized but cluster_scope stays empty. The deferred worker then calls named_distribute() with an empty list and hits the same NULL dereference, this time on the workqueue. With this patch the worker re-checks the list and skips cleanly, no crash and no link flap. The tail stamp in named_distribute() also gets an empty-queue guard.
Patch 3/3: a repeated NODE_UP while the bulk work is pending takes a node reference that is never dropped, because schedule_work() returns false when the work is already queued. Found by flapping the bearer during the defer window. One reference is leaked per repeated NODE_UP.
Changes in v6:
- Make the series self-contained: fold Tung Nguyen's base patch into the series (1/3), keeping his Signed-off-by. The version sent as v5 only carried the two follow-ups and depended on his patch from the thread; the code changes in 2/3 and 3/3 are unchanged from that version.
Changes in v5:
- Replace the item-less bulk approach with Tung Nguyen's defer-to-workqueue RFC, which fixes the reported bug in our testing.
- Fix two residual issues found during testing (patches 2/3, 3/3).
Weiming Shi (3):
tipc: fix NULL deref in tipc_named_node_up() on empty publication list
tipc: fix NULL deref in deferred bulk distribution on publish failure
tipc: fix node reference leak when defer work is already pending
net/tipc/core.c | 1 + net/tipc/core.h | 2 ++ net/tipc/name_distr.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++---- net/tipc/name_distr.h | 3 ++- net/tipc/net.c | 5 ++++- net/tipc/node.c | 35 ++++++++++++++++++++++++++++-- 6 files changed, 97 insertions(+), 8 deletions(-)
Signature · 1 line
2.43.0
PATCH v6 1/3 tipc: fix NULL deref in tipc_named_node_up() on empty publication list
[The defer-to-workqueue approach is by Tung Nguyen, suggested by
Jon Maloy on the thread as the replacement for the item-less bulk approach. Since the RFC only exists as an inline diff in the thread, it is folded into this series so the fix is self-contained.]
named_distribute() stamps the last_bulk flag on the tail skb of the publication list. When the list is empty no skb is enqueued and the tail access dereferences NULL. tipc_named_node_up() hits this on an empty cluster_scope, which happens with a node-id configuration where cluster_scope is populated only later by tipc_net_finalize(). It is reachable by an unprivileged user over a UDP bearer in a user+net namespace. The reported crash:
KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] RIP: 0010:tipc_named_node_up (net/tipc/name_distr.c:196) tipc_named_node_up (net/tipc/name_distr.c:196 net/tipc/name_distr.c:221) tipc_node_write_unlock (net/tipc/node.c:428) tipc_rcv (net/tipc/node.c:2185) tipc_udp_recv (net/tipc/udp_media.c:392) Kernel panic - not syncing: Fatal exception in interrupt
When cluster_scope is empty at node-up, defer the bulk distribution to a workqueue and wait for tipc_net_finalize() to publish the node-state name, so named_distribute() always runs on a non-empty list. On allocation failure, purge the partially built queue and bring the link down so the bulk distribution restarts when the link comes up again.
--- net/tipc/core.c | 1 + net/tipc/core.h | 2 ++ net/tipc/name_distr.c | 48 +++++++++++++++++++++++++++++++++++++++---- net/tipc/name_distr.h | 3 ++- net/tipc/net.c | 2 ++ net/tipc/node.c | 34 ++++++++++++++++++++++++++++-- 6 files changed, 83 insertions(+), 7 deletions(-)
net/tipc/core.c +1 -0
@@ -60,6 +60,7 @@ static int __net_init tipc_init_net(struct net *net) tn->trial_addr =0 ; tn->addr_trial_end =0 ; tn->capabilities =TIPC_NODE_CAPABILITIES ; +atomic_set (&tn->finalized,0 );INIT_WORK (&tn->work, tipc_net_finalize_work);memset (tn->node_id,0 ,sizeof (tn->node_id));memset (tn->node_id_string,0 ,sizeof (tn->node_id_string));
net/tipc/core.h +2 -0
@@ -145,6 +145,8 @@ struct tipc_net {struct work_struct work;/* The numbers of work queues in schedule */ atomic_t wq_count; +/* flag to indicate work has finished */ +atomic_t finalized; };static inline struct tipc_net *tipc_net (struct net *net)
net/tipc/name_distr.c +44 -4
81 lines not on this page · raw message
net/tipc/name_distr.h +2 -1
@@ -69,7 +69,8 @@ struct distr_item {struct sk_buff *tipc_named_publish (struct net *net,struct publication *publ);struct sk_buff *tipc_named_withdraw (struct net *net,struct publication *publ); -void tipc_named_node_up (struct net *net,u32 dnode,u16 capabilities); +int tipc_named_node_up (struct net *net,u32 dnode,u16 capabilities); +int tipc_named_dist_cluster_scope (struct net *net,u32 dnode);void tipc_named_rcv (struct net *net,struct sk_buff_head *namedq,u16 *rcv_nxt,bool *open);void tipc_named_reinit (struct net *net);
net/tipc/net.c +2 -0
@@ -139,6 +139,8 @@ static void tipc_net_finalize(struct net *net, u32 addr)tipc_sk_reinit (net);tipc_mon_reinit_self (net);tipc_nametbl_publish (net, &ua, &sk, addr); +atomic_inc (&tn->finalized); +wake_up_var (&tn->finalized); }void tipc_net_finalize_work (struct work_struct *work)
net/tipc/node.c +32 -2
@@ -145,6 +145,8 @@ struct tipc_node {#ifdef CONFIG_TIPC_CRYPTO struct tipc_crypto *crypto_rx;#endif +/* Work item for bulk distribution of cluster scope publications */ +struct work_struct work; };/* Node FSM states and events:
@@ -303,6 +305,7 @@ static void tipc_node_free(struct rcu_head *rp)#ifdef CONFIG_TIPC_CRYPTO tipc_crypto_stop (&n->crypto_rx);#endif +cancel_work_sync (&n->work);kfree (n); }
@@ -393,6 +396,19 @@ static void tipc_node_write_unlock_fast(struct tipc_node *n)write_unlock_bh (&n->lock); } +static void tipc_node_dist_bulk (struct work_struct *work) +{ + struct tipc_node *node = container_of(work, struct tipc_node, work); + +if (tipc_named_dist_cluster_scope (node->net, node->addr) <0 ) { +u32 bearer_id = node->link_id &0xffff ; + +tipc_node_link_down (node, bearer_id, false); + } + +tipc_node_put (node); +} +static void tipc_node_write_unlock (struct tipc_node *n) __releases(n->lock) {
@@ -424,8 +440,21 @@ static void tipc_node_write_unlock(struct tipc_node *n)if (flags &TIPC_NOTIFY_NODE_DOWN )tipc_publ_notify (net, publ_list, node, n->capabilities); -if (flags &TIPC_NOTIFY_NODE_UP ) -tipc_named_node_up (net, node, n->capabilities); +if (flags &TIPC_NOTIFY_NODE_UP ) { +int rc =0 ; + + rc =tipc_named_node_up (net, node, n->capabilities); +/* Defer bulk distribution to work queue */ +if (rc >0 ) { +tipc_node_get (n); +schedule_work (&n->work); + }else if (rc <0 ) { +/* Bring the link down to start over bulk distribution +* when the link is up again. +*/ +tipc_node_link_down (n, bearer_id, false); + } + }if (flags &TIPC_NOTIFY_LINK_UP ) {tipc_mon_peer_up (net, node, bearer_id);
@@ -564,6 +593,7 @@ struct tipc_node *tipc_node_create(struct net *net, u32 addr, u8 *peer_id,INIT_LIST_HEAD (&n->list);INIT_LIST_HEAD (&n->publ_list);INIT_LIST_HEAD (&n->conn_sks); +INIT_WORK (&n->work, tipc_node_dist_bulk);skb_queue_head_init (&n->bc_entry.namedq);skb_queue_head_init (&n->bc_entry.inputq1);__skb_queue_head_init (&n->bc_entry.arrvq);
Signature · 1 line
2.43.0
PATCH v6 2/3 tipc: fix NULL deref in deferred bulk distribution on publish failure
In reply to Weiming Shitipc_net_finalize() does not check the return value of tipc_nametbl_publish(). If the publish fails, for example on a GFP_ATOMIC allocation failure, the node state name never lands in cluster_scope, but tn->finalized is still set. A worker deferred by tipc_named_node_up() then wakes and calls named_distribute() with an empty list. That replays the same unguarded buf_msg(skb_peek_tail(list)) tail stamp, this time on the tipc_node_dist_bulk workqueue:
KASAN: null-ptr-deref in range [0x00000000000000c8-0x00000000000000cf] RIP: 0010:named_distribute (net/tipc/name_distr.c:200) Workqueue: events tipc_node_dist_bulk Call Trace: tipc_named_dist_cluster_scope (net/tipc/name_distr.c:267) tipc_node_dist_bulk (net/tipc/node.c:403) process_one_work worker_thread Kernel panic - not syncing: Fatal exception in interrupt
Check the publish result and warn on failure, but still set finalized, otherwise deferred workers would sleep forever. In tipc_named_dist_cluster_scope() re-check cluster_scope after the wait and skip the distribution when it is empty. This is a permanent condition, so return 0 instead of an error, otherwise the link would be bounced forever. Also guard the tail stamp in named_distribute() itself, so a caller that misses the precondition gets a warning and a link reset through the existing -ENOBUFS path instead of a crash.
Reproducing this needs an allocation failure during finalize, so I verified it by stubbing out the publish call: both nodes log the failure, the workers skip the distribution, no crash, no link flap. The normal path is unchanged with the same two-node test.
--- net/tipc/name_distr.c | 11 +++++++++++ net/tipc/net.c | 3 ++- 2 files changed, 13 insertions(+), 1 deletion(-)
net/tipc/name_distr.c +11 -0
@@ -193,6 +193,10 @@ static int named_distribute(struct net *net, struct sk_buff_head *list,skb_trim (skb,INT_H_SIZE + (msg_dsz - msg_rem));__skb_queue_tail (list, skb); } +if (skb_queue_empty (list)) { +pr_warn ("Bulk publication list empty, nothing to distribute\n" ); +return 1 ; + } hdr =buf_msg (skb_peek_tail (list));msg_set_last_bulk (hdr);msg_set_named_seqno (hdr, seqno);
@@ -253,6 +257,13 @@ int tipc_named_dist_cluster_scope(struct net *net, u32 dnode)spin_unlock_bh (&tn->nametbl_lock);read_lock_bh (&nt->cluster_scope_lock); +if (unlikely (list_empty (&nt->cluster_scope))) { +/* finalize is done but nothing was published (publish +* failed): a permanent state, nothing to synchronize. +*/ +read_unlock_bh (&nt->cluster_scope_lock); +return 0 ; + }if (named_distribute (net, &head, dnode, &nt->cluster_scope, seqno)) {read_unlock_bh (&nt->cluster_scope_lock);return -ENOBUFS ;
net/tipc/net.c +2 -1
@@ -138,7 +138,8 @@ static void tipc_net_finalize(struct net *net, u32 addr)tipc_named_reinit (net);tipc_sk_reinit (net);tipc_mon_reinit_self (net); -tipc_nametbl_publish (net, &ua, &sk, addr); +if (!tipc_nametbl_publish (net, &ua, &sk, addr)) +pr_warn ("Failed to publish own node state\n" );atomic_inc (&tn->finalized);wake_up_var (&tn->finalized); }
Signature · 1 line
2.43.0
PATCH v6 3/3 tipc: fix node reference leak when defer work is already pending
In reply to Weiming ShiIn tipc_node_write_unlock(), TIPC_NOTIFY_NODE_UP with an empty cluster_scope takes a node reference and schedules n->work. If the link flaps down and up while that work is still pending, the next NODE_UP takes another reference, but schedule_work() returns false and the extra reference is never dropped. The tipc_node structure leaks.
Verified by flapping the bearer while the work is pending: one reference is leaked per repeated NODE_UP, while the work is put only once when it finally runs.
Drop the reference when the work was already queued.
--- net/tipc/node.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/node.c +2 -1
@@ -447,7 +447,8 @@ static void tipc_node_write_unlock(struct tipc_node *n)/* Defer bulk distribution to work queue */ if (rc >0 ) {tipc_node_get (n); -schedule_work (&n->work); +if (!schedule_work (&n->work)) +tipc_node_put (n); }else if (rc <0 ) {/* Bring the link down to start over bulk distribution * when the link is up again.
Signature · 1 line
2.43.0