ipv6: fix hangup on device removal

PATCH Applied Paolo Abeni 4 participants 8 messages Nov 5, 2024 – Nov 7, 2024

Subsystems (4)

Networking Network drivers Kselftest IPv4/IPv6

lore

Share & tools

lore

Paolo Abeni

author maintainer

2024-11-05 18:24 UTC #0 lore raw

v2 cover ipv6: fix hangup on device removal

This addresses the infamous unregister_netdevice splat in net selftests; the actual fix is carried by the first patch, while the 2nd one addresses a related problem in the relevant test that was patially hiding the problem.

Targeting net-next as the issue is quite old and I feel a little lost in the fib info/nh jungle.
---
v1 -> v2:
- drop unintended whitespace change in patch 1/2

Paolo Abeni (2):
ipv6: release nexthop on device removal
selftests: net: really check for bg process completion

 net/ipv6/route.c                    | 6 +++---
 tools/testing/selftests/net/pmtu.sh | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

Signature · 1 line

2.45.2

Paolo Abeni

author maintainer

2024-11-05 18:24 UTC #1 lore raw

PATCH v2 1/2 ipv6: release nexthop on device removal

The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:

unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at

	dst_init+0x84/0x4a0
	dst_alloc+0x97/0x150
	ip6_dst_alloc+0x23/0x90
	ip6_rt_pcpu_alloc+0x1e6/0x520
	ip6_pol_route+0x56f/0x840
	fib6_rule_lookup+0x334/0x630
	ip6_route_output_flags+0x259/0x480
	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
	ip6_dst_lookup_flow+0x88/0x190
	udp_tunnel6_dst_lookup+0x2a7/0x4c0
	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
	vxlan_xmit+0x9ad/0xf20 [vxlan]
	dev_hard_start_xmit+0x10e/0x360
	__dev_queue_xmit+0xf95/0x18c0
	arp_solicit+0x4a2/0xe00
	neigh_probe+0xaa/0xf0

While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.

Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().

Tested running:

./pmtu.sh cleanup_ipv6_exception

in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.

Fixes f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")

Signed-off-by Paolo Abeni <[email protected]>

---
v1 -> v2:
- dropped unintended whitespace change

---
 net/ipv6/route.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

net/ipv6/route.c +3 -3

index d7ce5cf2017a..038c1eeef0be 100644
@@ -374,6 +374,7 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev)
 {
 	struct rt6_info *rt = dst_rt6_info(dst);
 	struct inet6_dev *idev = rt->rt6i_idev;
+	struct fib6_info *from;
 
 	if (idev && idev->dev != blackhole_netdev) {
 		struct inet6_dev *blackhole_idev = in6_dev_get(blackhole_netdev);

@@ -383,6 +384,8 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev)
 			in6_dev_put(idev);
 		}
 	}
+	from = unrcu_pointer(xchg(&rt->from, NULL));
+	fib6_info_release(from);
 }
 
 static bool __rt6_check_expired(const struct rt6_info *rt)

@@ -1455,7 +1458,6 @@ static DEFINE_SPINLOCK(rt6_exception_lock);
 static void rt6_remove_exception(struct rt6_exception_bucket *bucket,
 				 struct rt6_exception *rt6_ex)
 {
-	struct fib6_info *from;
 	struct net *net;
 
 	if (!bucket || !rt6_ex)

@@ -1467,8 +1469,6 @@ static void rt6_remove_exception(struct rt6_exception_bucket *bucket,
 	/* purge completely the exception to allow releasing the held resources:
 	 * some [sk] cache may keep the dst around for unlimited time
 	 */
-	from = unrcu_pointer(xchg(&rt6_ex->rt6i->from, NULL));
-	fib6_info_release(from);
 	dst_dev_put(&rt6_ex->rt6i->dst);
 
 	hlist_del_rcu(&rt6_ex->hlist);

Signature · 1 line

2.45.2

Paolo Abeni

author maintainer

2024-11-05 18:24 UTC #5 lore raw

PATCH v2 2/2 selftests: net: really check for bg process completion

In reply to Paolo Abeni

A recent refactor transformed the check for process completion in a true statement, due to a typo.

As a result, the relevant test-case is unable to catch the regression it was supposed to detect.

Restore the correct condition.

Fixes 691bb4e49c98 ("selftests: net: avoid just another constant wait")

Signed-off-by Paolo Abeni <[email protected]>

---
 tools/testing/selftests/net/pmtu.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

tools/testing/selftests/net/pmtu.sh +1 -1

index 569bce8b6383..6c651c880fe8 100755
@@ -2056,7 +2056,7 @@ check_running() {
 	pid=${1}
 	cmd=${2}
 
-	[ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "{cmd}" ]
+	[ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "${cmd}" ]
 }
 
 test_cleanup_vxlanX_exception() {

Signature · 1 line

2.45.2

Eric Dumazet

Reviewed-by on v2 1/2

2024-11-05 18:26 UTC #2 lore raw

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

In reply to Paolo Abeni

Quoted 34 lines · Paolo Abeni

The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
        dst_init+0x84/0x4a0
        dst_alloc+0x97/0x150
        ip6_dst_alloc+0x23/0x90
        ip6_rt_pcpu_alloc+0x1e6/0x520
        ip6_pol_route+0x56f/0x840
        fib6_rule_lookup+0x334/0x630
        ip6_route_output_flags+0x259/0x480
        ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
        ip6_dst_lookup_flow+0x88/0x190
        udp_tunnel6_dst_lookup+0x2a7/0x4c0
        vxlan_xmit_one+0xbde/0x4a50 [vxlan]
        vxlan_xmit+0x9ad/0xf20 [vxlan]
        dev_hard_start_xmit+0x10e/0x360
        __dev_queue_xmit+0xf95/0x18c0
        arp_solicit+0x4a2/0xe00
        neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:

Quoted 7 lines

./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
Signed-off-by Paolo Abeni <[email protected]>

Thanks a lot Paolo

Reviewed-by Eric Dumazet <[email protected]>

David Ahern

maintainer Reviewed-by on v2 1/2

2024-11-05 21:40 UTC #3 lore raw

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

In reply to Paolo Abeni

Quoted 31 lines · Paolo Abeni

The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
	dst_init+0x84/0x4a0
	dst_alloc+0x97/0x150
	ip6_dst_alloc+0x23/0x90
	ip6_rt_pcpu_alloc+0x1e6/0x520
	ip6_pol_route+0x56f/0x840
	fib6_rule_lookup+0x334/0x630
	ip6_route_output_flags+0x259/0x480
	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
	ip6_dst_lookup_flow+0x88/0x190
	udp_tunnel6_dst_lookup+0x2a7/0x4c0
	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
	vxlan_xmit+0x9ad/0xf20 [vxlan]
	dev_hard_start_xmit+0x10e/0x360
	__dev_queue_xmit+0xf95/0x18c0
	arp_solicit+0x4a2/0xe00
	neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().

Quoted 8 lines

Tested running:
./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")

are you sure that is the correct Fixes? That commit is June 2019 and there have been stable periods since then without netdev release problems.

Quoted 8 lines

Signed-off-by Paolo Abeni <[email protected]>
---
v1 -> v2:
- dropped unintended whitespace change
---
 net/ipv6/route.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by David Ahern <[email protected]>

David Ahern

maintainer Reviewed-by on v2 2/2

2024-11-05 21:40 UTC #6 lore raw

Re: [PATCH v2 net-next 2/2] selftests: net: really check for bg process completion

In reply to Paolo Abeni

Quoted 6 lines · Paolo Abeni

A recent refactor transformed the check for process completion in a true statement, due to a typo.
As a result, the relevant test-case is unable to catch the regression it was supposed to detect.

Quoted 8 lines

Restore the correct condition.
Fixes 691bb4e49c98 ("selftests: net: avoid just another constant wait")
Signed-off-by Paolo Abeni <[email protected]>
---
 tools/testing/selftests/net/pmtu.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by David Ahern <[email protected]>

Paolo Abeni

author maintainer

2024-11-06 09:11 UTC #4 lore raw

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

In reply to David Ahern

Quoted 36 lines · David Ahern

Quoted 35 lines · Paolo Abeni
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
	dst_init+0x84/0x4a0
	dst_alloc+0x97/0x150
	ip6_dst_alloc+0x23/0x90
	ip6_rt_pcpu_alloc+0x1e6/0x520
	ip6_pol_route+0x56f/0x840
	fib6_rule_lookup+0x334/0x630
	ip6_route_output_flags+0x259/0x480
	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
	ip6_dst_lookup_flow+0x88/0x190
	udp_tunnel6_dst_lookup+0x2a7/0x4c0
	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
	vxlan_xmit+0x9ad/0xf20 [vxlan]
	dev_hard_start_xmit+0x10e/0x360
	__dev_queue_xmit+0xf95/0x18c0
	arp_solicit+0x4a2/0xe00
	neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:
./pmtu.sh cleanup_ipv6_exception

Quoted 7 lines

Quoted 4 lines
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
are you sure that is the correct Fixes? That commit is June 2019 and there have been stable periods since then without netdev release problems.

"Sure" is a big word ;) AFAICS the mentioned commit let fib6_info store indirectly the extra dev reference via nexthop and does not clean it at device removal time.

Note that the issue is not deterministic - I needed ~30 mptu.sh iterations in a row to see it, so it could go unnoticed for a long time.

Quoted 10 lines

Quoted 8 lines
Signed-off-by Paolo Abeni <[email protected]>
---
v1 -> v2:
- dropped unintended whitespace change
---
 net/ipv6/route.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Reviewed-by David Ahern <[email protected]>

Thanks!

Paolo

[email protected]

CI Applied

2024-11-07 02:00 UTC #7 lore raw

In reply to Paolo Abeni

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 9 lines · On Tue, 5 Nov 2024 19:23:49 +0100 you wrote:

This addresses the infamous unregister_netdevice splat in net selftests; the actual fix is carried by the first patch, while the 2nd one addresses a related problem in the relevant test that was patially hiding the problem.
Targeting net-next as the issue is quite old and I feel a little lost in the fib info/nh jungle.
[...]

Here is the summary with links:
- [v2,net-next,1/2] ipv6: release nexthop on device removal

    https://git.kernel.org/netdev/net-next/c/eb02688c5c45
  - [v2,net-next,2/2] selftests: net: really check for bg process completion
    https://git.kernel.org/netdev/net-next/c/52ed077aa633

You are awesome, thank you!

Signature · 2 lines

Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html