Skip to contentSkip to conversation

ipv6: Serialize hardware flag notifications

PATCH In review Yuyang Huang 2 participants 3 messages Aug 15 – Aug 20

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ipv6: Serialize hardware flag notifications

fib6_info_hw_flags_set() first performs a lockless check of fib6_node. It then allocates a notification skb with GFP_KERNEL, which can sleep. A concurrent route deletion can remove the route, set fib6_node to NULL, and emit RTM_DELROUTE while the allocation sleeps. When the thread wakes up, it can emit RTM_NEWROUTE for the already deleted route.

This can cause userspace routing daemons to receive RTM_DELROUTE followed by RTM_NEWROUTE and incorrectly believe that the deleted route still exists in the kernel.

Allocate the skb before taking tb6_lock, then recheck fib6_node while holding the lock. Keep the lock until RTM_NEWROUTE is published. If route deletion wins the race, the recheck sees NULL and drops the notification. Otherwise, deletion cannot remove the route until RTM_NEWROUTE has been published, preserving notification order.

RTM_DELROUTE is sent by fib6_del_route() with tb6_lock held, so publishing RTM_NEWROUTE under the same lock is sufficient to guarantee ordering. rt6_fill_node() does not sleep in this path, and the notification uses GFP_ATOMIC, matching inet6_rt_notify() which already broadcasts under tb6_lock.

The race was found by Sashiko during code review.

Fixes 907eea486888 ("net: ipv6: Emit notification when fib hardware flags are changed")
---
 net/ipv6/route.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
net/ipv6/route.c +13 -1
index 16dfac54a259..73db4f630dcc 100644
@@ -6463,6 +6463,7 @@ void fib6_info_hw_flags_set(struct net *net, struct fib6_info *f6i,
 			    bool offload, bool trap, bool offload_failed)
 {
 	u8 fib_notify_on_flag_change;
+	struct fib6_table *table;
 	struct sk_buff *skb;
 	int err;
 
@@ -6491,22 +6492,33 @@ void fib6_info_hw_flags_set(struct net *net, struct fib6_info *f6i,
 	if (!fib_notify_on_flag_change)
 		return;
 
+	table = f6i->fib6_table;
 	skb = nlmsg_new(rt6_nlmsg_size(f6i), GFP_KERNEL);
 	if (!skb) {
 		err = -ENOBUFS;
 		goto errout;
 	}
 
+	spin_lock_bh(&table->tb6_lock);
+	if (!rcu_dereference_protected(f6i->fib6_node,
+				       lockdep_is_held(&table->tb6_lock))) {
+		spin_unlock_bh(&table->tb6_lock);
+		kfree_skb(skb);
+		return;
+	}
+
 	err = rt6_fill_node(net, skb, f6i, NULL, NULL, NULL, 0, RTM_NEWROUTE, 0,
 			    0, 0, RT_DEL_REASON_UNSPEC);
 	if (err < 0) {
 		/* -EMSGSIZE implies BUG in rt6_nlmsg_size() */
 		WARN_ON(err == -EMSGSIZE);
+		spin_unlock_bh(&table->tb6_lock);
 		kfree_skb(skb);
 		goto errout;
 	}
 
-	rtnl_notify(skb, net, 0, RTNLGRP_IPV6_ROUTE, NULL, GFP_KERNEL);
+	rtnl_notify(skb, net, 0, RTNLGRP_IPV6_ROUTE, NULL, GFP_ATOMIC);
+	spin_unlock_bh(&table->tb6_lock);
 	return;
 
 errout:
Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 16 lines · Yuyang Huang

fib6_info_hw_flags_set() first performs a lockless check of fib6_node. It then allocates a notification skb with GFP_KERNEL, which can sleep. A concurrent route deletion can remove the route, set fib6_node to NULL, and emit RTM_DELROUTE while the allocation sleeps. When the thread wakes up, it can emit RTM_NEWROUTE for the already deleted route.

This can cause userspace routing daemons to receive RTM_DELROUTE followed by RTM_NEWROUTE and incorrectly believe that the deleted route still exists in the kernel.

Allocate the skb before taking tb6_lock, then recheck fib6_node while holding the lock. Keep the lock until RTM_NEWROUTE is published. If route deletion wins the race, the recheck sees NULL and drops the notification. Otherwise, deletion cannot remove the route until RTM_NEWROUTE has been published, preserving notification order.

Quoted 7 lines

RTM_DELROUTE is sent by fib6_del_route() with tb6_lock held, so publishing RTM_NEWROUTE under the same lock is sufficient to guarantee ordering. rt6_fill_node() does not sleep in this path, and the notification uses GFP_ATOMIC, matching inet6_rt_notify() which already broadcasts under tb6_lock.

The race was found by Sashiko during code review.

Yes, it's racy, but I don't have good solution that also covers IPv4. IPv4 routes are protected by RTNL and taking RTNL in this path will cause lock inversion.

Also, as far as I'm aware, this isn't a problem in practice. These notifications (disabled by default) are mainly used by routing daemons that want to suppress the advertisement of a route until it's offloaded. If they added it and immediately deleted it, then it doesn't make sense to advertise it.

#2 lore raw
Quoted 4 lines · Ido Schimmel

Yes, it's racy, but I don't have good solution that also covers IPv4. IPv4 routes are protected by RTNL and taking RTNL in this path will cause lock inversion.

Thanks, I agree it seems like the same idea does not apply to IPv4 code paths. It seems like to truly this issue will need more thought/refactoring.

Quoted 5 lines

Also, as far as I'm aware, this isn't a problem in practice. These notifications (disabled by default) are mainly used by routing daemons that want to suppress the advertisement of a route until it's offloaded. If they added it and immediately deleted it, then it doesn't make sense to advertise it.

Hmm, this argument seems weak. I think relying on expected userspace behavior is not sufficient to dismiss the race, since kernel can emit RTM_NEWROUTE after RTM_DELROUTE seems really should't be expected.

That said, I agree that an IPv6-only fix is incomplete, and I do not currently see a good/cheap way to cover IPv4 without introducing a lock inversion. I am not sufficiently familiar with this code path to propose a broader solution, so I will drop the patch for now.

Thanks,
Yuyang