Skip to contentSkip to conversation

selftests: rtnetlink: try double sleep to give WQ a chance

PATCH In review Jakub Kicinski 2 participants 5 messages Jul 10, 2025 – Jul 14, 2025

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 selftests: rtnetlink: try double sleep to give WQ a chance

The rtnetlink test for preferred lifetime of an address is quite flaky. Problems started around the 6.16 merge window in May. The test fails with:

FAIL: preferred_lft addresses remaining

and unlike most of our flakes this one fails on the "normal" kernel builds, not the builds with kernel/configs/debug.config. I suspect the flakes may be related to power saving, since the expirations run from a "power efficient" workqueue. Adding a short sleep seems to decrease the flakes by 8x but they still happen. With this patch in place we get a flake every couple of weeks, not every couple of days. Better ideas welcome..

---
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 tools/testing/selftests/net/rtnetlink.sh | 5 +++++
 1 file changed, 5 insertions(+)
tools/testing/selftests/net/rtnetlink.sh +5 -0
index 2e8243a65b50..b9e1497ea27a 100755
@@ -299,6 +299,11 @@ kci_test_addrlft()
 	done
 
 	sleep 5
+	# Schedule out for a bit, address GC runs from the power efficient WQ
+	# if the long sleep above has put the whole system into sleep state
+	# the WQ may have not had a chance to run.
+	sleep 0.1
+
 	run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy"
 	if [ $? -eq 0 ]; then
 		check_err 1
Signature · 1 line
2.50.0
#1 lore raw
Quoted 28 lines · Jakub Kicinski

The rtnetlink test for preferred lifetime of an address is quite flaky. Problems started around the 6.16 merge window in May. The test fails with:

FAIL: preferred_lft addresses remaining

and unlike most of our flakes this one fails on the "normal" kernel builds, not the builds with kernel/configs/debug.config. I suspect the flakes may be related to power saving, since the expirations run from a "power efficient" workqueue. Adding a short sleep seems to decrease the flakes by 8x but they still happen. With this patch in place we get a flake every couple of weeks, not every couple of days. Better ideas welcome..

---
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 tools/testing/selftests/net/rtnetlink.sh | 5 +++++
 1 file changed, 5 insertions(+)
tools/testing/selftests/net/rtnetlink.sh +0 -0
index 2e8243a65b50..b9e1497ea27a 100755

tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306

Quoted 8 lines
tools/testing/selftests/net/rtnetlink.sh +5 -0
@@ -299,6 +299,11 @@ kci_test_addrlft()
 	done
 
 	sleep 5
+	# Schedule out for a bit, address GC runs from the power efficient WQ
+	# if the long sleep above has put the whole system into sleep state
+	# the WQ may have not had a chance to run.
+	sleep 0.1
+

How about use slowwait to check if the address still exists. e.g.

check_addr_not_exist()

{
	dev=$1
	addr=$2
	if ip addr show dev $dev | grep -q $addr; then
		return 1
	else
		return 0
}
	slowwait 5 check_addr_not_exist "$devdummy" "10.23.11."
Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +0 -0
 	run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy"
 	if [ $? -eq 0 ]; then
 		check_err 1
Signature · 1 line
2.50.0
author maintainer
#2 lore raw
Quoted 8 lines · On Fri, 11 Jul 2025 02:14:03 +0000 Hangbin Liu wrote:

tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306

Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +5 -0
 	sleep 5
+	# Schedule out for a bit, address GC runs from the power efficient WQ
+	# if the long sleep above has put the whole system into sleep state
+	# the WQ may have not had a chance to run.
+	sleep 0.1
+  

How about use slowwait to check if the address still exists.

Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..

I queued this debug patch a couple of months ago:

sleep 5
- run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy"
+ ip addr show dev "$devdummy" > /tmp/a
+ run_cmd_grep_fail "10.23.11." cat /tmp/a
if [ $? -eq 0 ]; then
- check_err 1
- end_test "FAIL: preferred_lft addresses remaining"

+	    check_err 1
+	    cat /tmp/a
+	    echo "==="

+ ip addr show dev "$devdummy"
+ end_test "FAIL: preferred_lft addresses remaining ($lft)"
return
fi

And when it flakes the output looks like this:

# 7.23 [+7.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.23 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.23 [+0.00] inet 10.23.11.81/32 scope global deprecated dynamic test-dummy0
# 7.23 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.23 [+0.00] inet 10.23.11.84/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.93/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.94/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.97/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.99/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.24 [+0.00] valid_lft forever preferred_lft forever
# 7.24 [+0.00] ===
# 7.25 [+0.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.25 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.25 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.25 [+0.00] valid_lft forever preferred_lft forever
# 7.25 [+0.00] FAIL: preferred_lft addresses remaining (1)

#3 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 11 Jul 2025 02:14:03 +0000 Hangbin Liu wrote:

tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306

Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +5 -0
 	sleep 5
+	# Schedule out for a bit, address GC runs from the power efficient WQ
+	# if the long sleep above has put the whole system into sleep state
+	# the WQ may have not had a chance to run.
+	sleep 0.1
+  

How about use slowwait to check if the address still exists.

Quoted 4 lines

Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..

Not sure if I understand correctly. Do you mean the addresses will keep there if we use slowwait?

Thanks
Hangbin

Quoted 42 lines

I queued this debug patch a couple of months ago:

(quoted diff) +7 -3
 	sleep 5
-	run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy"
+	ip addr show dev "$devdummy" > /tmp/a
+	run_cmd_grep_fail "10.23.11." cat /tmp/a
 	if [ $? -eq 0 ]; then
-		check_err 1
-		end_test "FAIL: preferred_lft addresses remaining"
+	    check_err 1
+	    cat /tmp/a
+	    echo "==="
+		ip addr show dev "$devdummy"
+		end_test "FAIL: preferred_lft addresses remaining ($lft)"
 		return
 	fi

And when it flakes the output looks like this:

# 7.23 [+7.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.23 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.23 [+0.00] inet 10.23.11.81/32 scope global deprecated dynamic test-dummy0
# 7.23 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.23 [+0.00] inet 10.23.11.84/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.93/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.94/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.97/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.99/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.24 [+0.00] valid_lft forever preferred_lft forever
# 7.24 [+0.00] ===
# 7.25 [+0.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.25 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.25 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.25 [+0.00] valid_lft forever preferred_lft forever
# 7.25 [+0.00] FAIL: preferred_lft addresses remaining (1)

author maintainer
#4 lore raw
Quoted 9 lines · On Mon, 14 Jul 2025 07:19:09 +0000 Hangbin Liu wrote:
Quoted 6 lines
Quoted 1 line

How about use slowwait to check if the address still exists.

Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..

Not sure if I understand correctly. Do you mean the addresses will keep there if we use slowwait?

No, I mean there may be false negatives, not false positive. But maybe it's fine, it will definitely prevent flakes. Could you post the slowwait patch officially?

Signature · 1 line
pw-bot: cr