bond_macvlan_ipvlan.sh flakiness

Hi Hangbin!

The flakiness of bond_macvlan_ipvlan.sh has increased quite a lot recently. Not sure if there's any correlation with kernel changes, I didn't spot anything in bonding itself. Here's the history of runs:

https://netdev.bots.linux.dev/contest.html?executor=vmksft-bonding&test=bond-macvlan-ipvlan-sh

It looks like it's gotten much worse starting around the 9th?

Only the non-debug kernel build is flaking, debug builds are completely clear:

https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=bond-macvlan-ipvlan-sh

A few things that stood out to me, all the failures are like this:

# TEST: balance-$lb/$$$vlan_bridge: IPv4: client->$$$vlan_2 [FAIL]

Always IPv4 ping to the second interface, always fails neighbor resolution:

# 192.0.2.12 dev eth0 FAILED

If it's ipvlan that fails rather than macvlan there is a bunch of otherhost drops:

# 17: ipvlan0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
# link/ether 00:0a:0b:0c:0d:01 brd ff:ff:ff:ff:ff:ff link-netns s-8BLcCn
# RX: bytes packets errors dropped missed mcast
# 702 10 0 0 0 3
# RX errors: length crc frame fifo overrun otherhost
# 0 0 0 0 0 4

FWIW here's the contents of the branches if you want to look thru:
https://netdev.bots.linux.dev/static/nipa/branch_deltas/net-next-2025-11-09--12-00.html
but 9th was the weekend, and the failure just got more frequent, we've been trying to track this down for a while..

Quoted 9 lines

If it's ipvlan that fails rather than macvlan there is a bunch of otherhost drops:

# 17: ipvlan0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
# link/ether 00:0a:0b:0c:0d:01 brd ff:ff:ff:ff:ff:ff link-netns s-8BLcCn
# RX: bytes packets errors dropped missed mcast
# 702 10 0 0 0 3
# RX errors: length crc frame fifo overrun otherhost
# 0 0 0 0 0 4

Hmm, this one is suspicious. I can reproduce the ping fail on local. But no "otherhost" issue. I will check the failure recently.

Quoted 2 lines

Hmm, this one is suspicious. I can reproduce the ping fail on local. But no "otherhost" issue. I will check the failure recently.

This looks like a time-sensitive issue, with

tools/testing/selftests/drivers/net/bonding/bond_macvlan_ipvlan.sh +1 -0

index c4711272fe45..947c85ec2cbb 100755
@@ -30,6 +30,7 @@ check_connection()
        local message=${3}
        RET=0
 
+       sleep 1
        ip netns exec ${ns} ping ${target} -c 4 -i 0.1 &>/dev/null
        check_err $? "ping failed"
        log_test "${bond_mode}/${xvlan_type}_${xvlan_mode}: ${message}"

I run the test 100 times (vng with 4 cpus) and not able to reproduce it anymore. That maybe why debug kernel works good.

Quoted 18 lines · On Tue, 18 Nov 2025 06:03:17 +0000 Hangbin Liu wrote:

Quoted 2 lines

Hmm, this one is suspicious. I can reproduce the ping fail on local. But no "otherhost" issue. I will check the failure recently.

This looks like a time-sensitive issue, with

tools/testing/selftests/drivers/net/bonding/bond_macvlan_ipvlan.sh +1 -0

index c4711272fe45..947c85ec2cbb 100755
@@ -30,6 +30,7 @@ check_connection()
        local message=${3}
        RET=0
 
+       sleep 1
        ip netns exec ${ns} ping ${target} -c 4 -i 0.1 &>/dev/null
        check_err $? "ping failed"
        log_test "${bond_mode}/${xvlan_type}_${xvlan_mode}: ${message}"

Quoted 2 lines

I run the test 100 times (vng with 4 cpus) and not able to reproduce it anymore. That maybe why debug kernel works good.

I see. I queued up a local change to add a 0.25 sec wait. Let's wait a couple of days and see how much sleep we need here, this function is called 96 times if I'm counting right.

Quoted 19 lines · On Tue, 18 Nov 2025 07:13:02 -0800 Jakub Kicinski wrote:

Quoted 18 lines · On Tue, 18 Nov 2025 06:03:17 +0000 Hangbin Liu wrote:

Quoted 2 lines

Hmm, this one is suspicious. I can reproduce the ping fail on local. But no "otherhost" issue. I will check the failure recently.

This looks like a time-sensitive issue, with

tools/testing/selftests/drivers/net/bonding/bond_macvlan_ipvlan.sh +1 -0

index c4711272fe45..947c85ec2cbb 100755
@@ -30,6 +30,7 @@ check_connection()
        local message=${3}
        RET=0
 
+       sleep 1
        ip netns exec ${ns} ping ${target} -c 4 -i 0.1 &>/dev/null
        check_err $? "ping failed"
        log_test "${bond_mode}/${xvlan_type}_${xvlan_mode}: ${message}"

Quoted 6 lines

Quoted 2 lines

I run the test 100 times (vng with 4 cpus) and not able to reproduce it anymore. That maybe why debug kernel works good.

I see. I queued up a local change to add a 0.25 sec wait. Let's wait a couple of days and see how much sleep we need here, this function is called 96 times if I'm counting right.

Hi Hangbin!

The 0.25 sec sleep was added locally 1 week ago and 0 flakes since. Would you mind submitting it officially?

Quoted 8 lines

Quoted 3 lines

I see. I queued up a local change to add a 0.25 sec wait. Let's wait a couple of days and see how much sleep we need here, this function is called 96 times if I'm counting right.

Hi Hangbin!

The 0.25 sec sleep was added locally 1 week ago and 0 flakes since. Would you mind submitting it officially?

Good to hear this. I will submit it.

Quoted 6 lines · Hangbin Liu

Quoted 4 lines

Hi Hangbin!

The 0.25 sec sleep was added locally 1 week ago and 0 flakes since. Would you mind submitting it officially?

Good to hear this. I will submit it.

Oh, I pressed the send button too fast. I forgot to ask—should we keep it at 0.25s or extend it to 0.5s to avoid flaky tests later?

Quoted 10 lines · On Thu, 27 Nov 2025 01:18:30 +0000 Hangbin Liu wrote:

Quoted 6 lines · Hangbin Liu

Quoted 4 lines

Hi Hangbin!

The 0.25 sec sleep was added locally 1 week ago and 0 flakes since. Would you mind submitting it officially?

Good to hear this. I will submit it.

Oh, I pressed the send button too fast. I forgot to ask—should we keep it at 0.25s or extend it to 0.5s to avoid flaky tests later?

I'd stick to 0.25sec since it was solid for a week.
I don't think the race window is very large, we could even experiment with a smaller delay, because debug kernels don't hit the issue. The debug kernel can't be >0.1sec slower I reckon.

IOW I hope 0.25sec already has pretty solid safety margin? As I mentioned last week - this is called almost 100 times by the test so the longer delays will be quite visible in the test runtime.