selftests: rtnetlink: try double sleep to give WQ a chance
Subsystems (3)
Jakub Kicinski
maintainerPATCH 1/1 selftests: rtnetlink: try double sleep to give WQ a chance
The rtnetlink test for preferred lifetime of an address is quite flaky. Problems started around the 6.16 merge window in May. The test fails with:
FAIL: preferred_lft addresses remaining
and unlike most of our flakes this one fails on the "normal" kernel builds, not the builds with kernel/configs/debug.config. I suspect the flakes may be related to power saving, since the expirations run from a "power efficient" workqueue. Adding a short sleep seems to decrease the flakes by 8x but they still happen. With this patch in place we get a flake every couple of weeks, not every couple of days. Better ideas welcome..
---
CC: [email protected]
CC: [email protected]
CC: [email protected]
--- tools/testing/selftests/net/rtnetlink.sh | 5 +++++ 1 file changed, 5 insertions(+)
tools/testing/selftests/net/rtnetlink.sh +5 -0
@@ -299,6 +299,11 @@ kci_test_addrlft() done sleep 5 + # Schedule out for a bit, address GC runs from the power efficient WQ + # if the long sleep above has put the whole system into sleep state + # the WQ may have not had a chance to run. + sleep 0.1 + run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy" if [ $? -eq 0 ]; then check_err 1
Signature · 1 line
2.50.0
Quoted 28 lines · Jakub Kicinski
The rtnetlink test for preferred lifetime of an address is quite flaky. Problems started around the 6.16 merge window in May. The test fails with:
FAIL:
preferred_lftaddresses remainingand unlike most of our flakes this one fails on the "normal" kernel builds, not the builds with
kernel/configs/debug.config. I suspect the flakes may be related to power saving, since the expirations run from a "power efficient" workqueue. Adding a short sleep seems to decrease the flakes by 8x but they still happen. With this patch in place we get a flake every couple of weeks, not every couple of days. Better ideas welcome..Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
CC: [email protected]
CC: [email protected]--- tools/testing/selftests/net/rtnetlink.sh | 5 +++++ 1 file changed, 5 insertions(+)tools/testing/selftests/net/rtnetlink.sh +0
-0
tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306
Quoted 8 lines
tools/testing/selftests/net/rtnetlink.sh +5
-0@@ -299,6 +299,11 @@ kci_test_addrlft() done sleep 5 + # Schedule out for a bit, address GC runs from the power efficient WQ + # if the long sleep above has put the whole system into sleep state + # the WQ may have not had a chance to run. + sleep 0.1 +
How about use slowwait to check if the address still exists. e.g.
check_addr_not_exist()
{
dev=$1
addr=$2
if ip addr show dev $dev | grep -q $addr; then
return 1
else
return 0
}slowwait 5 check_addr_not_exist "$devdummy" "10.23.11."
Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +0
-0run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy" if [ $? -eq 0 ]; then check_err 1Signature · 1 line
2.50.0
Jakub Kicinski
maintainerQuoted 8 lines · On Fri, 11 Jul 2025 02:14:03 +0000 Hangbin Liu wrote:
tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306
Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +5
-0sleep 5 + # Schedule out for a bit, address GC runs from the power efficient WQ + # if the long sleep above has put the whole system into sleep state + # the WQ may have not had a chance to run. + sleep 0.1 +How about use slowwait to check if the address still exists.
Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..
I queued this debug patch a couple of months ago:
sleep 5
- run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy"
+ ip addr show dev "$devdummy" > /tmp/a
+ run_cmd_grep_fail "10.23.11." cat /tmp/a
if [ $? -eq 0 ]; then
- check_err 1
- end_test "FAIL: preferred_lft addresses remaining"
+ check_err 1 + cat /tmp/a + echo "==="
+ ip addr show dev "$devdummy"
+ end_test "FAIL: preferred_lft addresses remaining ($lft)"
return
fi
And when it flakes the output looks like this:
# 7.23 [+7.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.23 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.23 [+0.00] inet 10.23.11.81/32 scope global deprecated dynamic test-dummy0
# 7.23 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.23 [+0.00] inet 10.23.11.84/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.93/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.94/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.97/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet 10.23.11.99/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00] valid_lft 0sec preferred_lft 0sec
# 7.24 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.24 [+0.00] valid_lft forever preferred_lft forever
# 7.24 [+0.00] ===
# 7.25 [+0.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.25 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.25 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link proto kernel_ll
# 7.25 [+0.00] valid_lft forever preferred_lft forever
# 7.25 [+0.00] FAIL: preferred_lft addresses remaining (1)
Quoted 10 lines · Jakub Kicinski
Quoted 8 lines · On Fri, 11 Jul 2025 02:14:03 +0000 Hangbin Liu wrote:
tools/testing/selftests/net/rtnetlink.sh kci_test_addrlft() · line 306
Quoted 6 lines
tools/testing/selftests/net/rtnetlink.sh +5
-0sleep 5 + # Schedule out for a bit, address GC runs from the power efficient WQ + # if the long sleep above has put the whole system into sleep state + # the WQ may have not had a chance to run. + sleep 0.1 +How about use slowwait to check if the address still exists.
Quoted 4 lines
Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..
Not sure if I understand correctly. Do you mean the addresses will keep there if we use slowwait?
Thanks
Hangbin
Quoted 42 lines
I queued this debug patch a couple of months ago:
(quoted diff) +7
-3sleep 5 - run_cmd_grep_fail "10.23.11." ip addr show dev "$devdummy" + ip addr show dev "$devdummy" > /tmp/a + run_cmd_grep_fail "10.23.11." cat /tmp/a if [ $? -eq 0 ]; then - check_err 1 - end_test "FAIL: preferred_lft addresses remaining" + check_err 1 + cat /tmp/a + echo "===" + ip addr show dev "$devdummy" + end_test "FAIL: preferred_lft addresses remaining ($lft)" return fiAnd when it flakes the output looks like this:
# 7.23 [+7.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.23 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.23 [+0.00] inet 10.23.11.81/32 scope global deprecated dynamic test-dummy0
# 7.23 [+0.00]valid_lft0secpreferred_lft0sec
# 7.23 [+0.00] inet 10.23.11.84/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00]valid_lft0secpreferred_lft0sec
# 7.24 [+0.00] inet 10.23.11.93/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00]valid_lft0secpreferred_lft0sec
# 7.24 [+0.00] inet 10.23.11.94/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00]valid_lft0secpreferred_lft0sec
# 7.24 [+0.00] inet 10.23.11.97/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00]valid_lft0secpreferred_lft0sec
# 7.24 [+0.00] inet 10.23.11.99/32 scope global deprecated dynamic test-dummy0
# 7.24 [+0.00]valid_lft0secpreferred_lft0sec
# 7.24 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link protokernel_ll
# 7.24 [+0.00]valid_lftforeverpreferred_lftforever
# 7.24 [+0.00] ===
# 7.25 [+0.00] 297: test-dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# 7.25 [+0.00] link/ether 9e:a6:c4:c2:1b:16 brd ff:ff:ff:ff:ff:ff
# 7.25 [+0.00] inet6 fe80::9ca6:c4ff:fec2:1b16/64 scope link protokernel_ll
# 7.25 [+0.00]valid_lftforeverpreferred_lftforever
# 7.25 [+0.00] FAIL:preferred_lftaddresses remaining (1)
Jakub Kicinski
maintainerQuoted 9 lines · On Mon, 14 Jul 2025 07:19:09 +0000 Hangbin Liu wrote:
Quoted 6 lines
Quoted 1 line
How about use slowwait to check if the address still exists.
Weirdly if we read the addresses twice they disappear, I haven't looked into the code for the why, but seemed like using slowwait could potentially mask the addresses sticking around when nobody runs the Netlink handlers for a while? Dunno..
Not sure if I understand correctly. Do you mean the addresses will keep there if we use slowwait?
No, I mean there may be false negatives, not false positive. But maybe it's fine, it will definitely prevent flakes. Could you post the slowwait patch officially?
Signature · 1 line
pw-bot: cr