Flake report

Hi!

Feels like the efforts to get rid of flaky tests have slowed down a bit, so I thought I'd poke people..

Here's the full list:
https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0
click on test name to get the list of runs and links to outputs.

As a reminder please see these instructions for repro:
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

I'll try to tag folks who touched the tests most recently, but please don't hesitate to chime in.

net
---

arp-ndisc-untracked-subnets-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Jaehee Park <[email protected]>
Cc: Hangbin Liu <[email protected]>

Times out on debug kernels, passes on non-debug.
This is a real timeout, eats full 7200 seconds.

xfrm-policy-sh
~~~~~~~~~~~~~~
To: Hangbin Liu <[email protected]>

Times out on debug kernels, passed on non-debug,
This is a "inactivity" timeout, test doesn't print anything for 900 seconds so the runner kills it. We can bump the timeout but not printing for 15min is bad..

To: Florian Westphal <[email protected]>

These are skipped because of some compatibility issues:

nft-flowtable-sh, bridge-brouter-sh, nft-audit-sh

Please LMK if I need to update the CLI tooling.
Or is this missing kernel config?

mptcp
-----
To: Matthieu Baerts <[email protected]>

simult-flows-sh is still quite flaky :(

Hi!

Feels like the efforts to get rid of flaky tests have slowed down a bit, so I thought I'd poke people..

Here's the full list:
https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0
click on test name to get the list of runs and links to outputs.

As a reminder please see these instructions for repro:
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

I'll try to tag folks who touched the tests most recently, but please don't hesitate to chime in.

bridge-igmp-sh, bridge-mld-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Nikolay Aleksandrov <[email protected]>
Cc: Ido Schimmel <[email protected]>

On debug kernels it always fails with:

# TEST: IGMPv3 group 239.10.10.10 exclude timeout [FAIL]
# Entry 192.0.2.21 has blocked flag failed

For MLD:

# TEST: MLDv2 group ff02::cc exclude timeout [FAIL]
# Entry 2001:db8:1::21 has blocked flag failed

Quoted 8 lines · Jakub Kicinski

To: Florian Westphal <[email protected]>

These are skipped because of some compatibility issues:

nft-flowtable-sh, bridge-brouter-sh, nft-audit-sh

Please LMK if I need to update the CLI tooling.
Or is this missing kernel config?

No, its related to the userspace tooling.
This should start to work once amazon linux updates nftables.

bridge-brouter-sh would work with the old ebtables-legacy instead of ebtables-nft, or a more recent version of ebtables-nft.

ATM it uses a version of ebtables-nft that lacks "broute" table emulation.

Quoted 16 lines · On Fri, 10 May 2024 10:35:51 +0200 Florian Westphal wrote:

Quoted 8 lines · Jakub Kicinski

To: Florian Westphal <[email protected]>

These are skipped because of some compatibility issues:

nft-flowtable-sh, bridge-brouter-sh, nft-audit-sh

Please LMK if I need to update the CLI tooling.
Or is this missing kernel config?

No, its related to the userspace tooling.
This should start to work once amazon linux updates nftables.

bridge-brouter-sh would work with the old ebtables-legacy instead of ebtables-nft, or a more recent version of ebtables-nft.

Quoted 1 line

ATM it uses a version of ebtables-nft that lacks "broute" table emulation.

Amazon Linux is more of a base OS for loading containers it seems. I build pretty much all the tools from source.

So I just built nft too.. Whether it will actually work we'll find out in about 15 min :)

M. Looks like that didn't do anything.

I tried to investigate nft_audit.sh

https://netdev-3.bots.linux.dev/vmksft-nf/results/589221/22-nft-audit-sh/stdout

# selftests: net/netfilter: nft_audit.sh
# SKIP: nft reset feature test failed: nftables v1.0.9 (Old Doc Yak #3) ok 1 selftests: net/netfilter: nft_audit.sh # SKIP

This is what it hits:

bash-5.2# nft -v
nftables v1.0.9 (Old Doc Yak #3)
bash-5.2# nft --check -f /dev/stdin <<EOF
add table t
add chain t c
reset rules t c
EOF
/dev/stdin:3:7-11: Error: syntax error, unexpected string, expecting counter or counters or quotas or quota reset rules t c

        ^^^^^

What does that mean in lay terms?

Question #2, for the ebtables test - do I need to build iptables? I built nft with

	./configure --with-json --with-xtables 

Quoted 1 line

What does that mean in lay terms?

This nft version chokes on syntax, but I cannot reproduce this:
src/nft --check -f /dev/stdin <<EOF
add table t
add chain t c
reset rules t c
EOF
echo $?

table ip t {
	chain c { }
}

0
src/nft --version
nftables v1.0.9 (Old Doc Yak #3)

No idea :-(

I tried building both recent nftables.git and v1.0.9 tag and both parse the test file for me :-(

Also. nft-flowtable.sh is still not working on nf infra even with the updated version while that script works fine locally for me as well, even with running via vng.

Maybe there is an old libnftables on the system that is used instead for parsing? Its bundled/installed with nftables, can you check that ldd nft doesnt show some other distro-installed version? Other than that I have no idea what could be happening here.

Quoted 3 lines

Question #2, for the ebtables test - do I need to build iptables? I built nft with

	./configure --with-json --with-xtables   

You need to add --enable-nftables for ebtables-nft, or you need to use the old ebtables tree, i.e.:
https://git.netfilter.org/ebtables/

both should work.

Quoted 5 lines

You need to add --enable-nftables for ebtables-nft, or you need to use the old ebtables tree, i.e.:
https://git.netfilter.org/ebtables/

both should work.

Picked the old tree. Let's see..

Quoted 1 line

Picked the old tree. Let's see..

Looks like that worked!!

So the last fail we see for netfilter is nft-flowtable-sh with kernel debug enabled:

https://netdev.bots.linux.dev/contest.html?executor=vmksft-nf-dbg&test=nft-flowtable-sh

Hi!

Feels like the efforts to get rid of flaky tests have slowed down a bit, so I thought I'd poke people..

Here's the full list:
https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0
click on test name to get the list of runs and links to outputs.

As a reminder please see these instructions for repro:
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

I'll try to tag folks who touched the tests most recently, but please don't hesitate to chime in.

net
---

arp-ndisc-untracked-subnets-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Jaehee Park <[email protected]>
Cc: Hangbin Liu <[email protected]>

Times out on debug kernels, passes on non-debug.
This is a real timeout, eats full 7200 seconds.

xfrm-policy-sh
~~~~~~~~~~~~~~
To: Hangbin Liu <[email protected]>

Times out on debug kernels, passed on non-debug,
This is a "inactivity" timeout, test doesn't print anything for 900 seconds so the runner kills it. We can bump the timeout but not printing for 15min is bad..

cmsg-time-sh
~~~~~~~~~~~~
To: Jakub Kicinski <[email protected]> (forgot I wrote this :D)

Fails randomly.

pmtu-sh
~~~~~~~
To: Simon Horman <[email protected]>

Skipped because it wants full OVS tooling.

forwarding
----------

sch-tbf-ets-sh, sch-tbf-prio-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

These fail way too often on non-debug kernels :(
Perhaps we can extend the lower bound?

bridge-igmp-sh, bridge-mld-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Nikolay Aleksandrov <[email protected]>
Cc: Ido Schimmel <[email protected]>

On debug kernels it always fails with:

# TEST: IGMPv3 group 239.10.10.10 exclude timeout [FAIL]
# Entry 192.0.2.21 has blocked flag failed

For MLD:

# TEST: MLDv2 group ff02::cc exclude timeout [FAIL]
# Entry 2001:db8:1::21 has blocked flag failed

vxlan-bridge-1d-sh
~~~~~~~~~~~~~~~~~~
To: Ido Schimmel <[email protected]>
Cc: Petr Machata <[email protected]>

Flake fails almost always, with some form of "Expected to capture 0 packets, got $X"

mirror-gre-lag-lacp-sh
~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Often fails on debug with:

# TEST: mirror to gretap: LAG first slave (skip_hw) [FAIL]
# Expected to capture 10 packets, got 13.

mirror-gre-vlan-bridge-1q-sh, mirror-gre-bridge-1d-vlan-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Same kind of failure as above but less often and both on debug and non-debug.

tc-actions-sh
~~~~~~~~~~~~~
To: Davide Caratti <[email protected]>

It triggers a random unhandled interrupt, somehow (look at stderr). It's the only test that does that.

mptcp
-----
To: Matthieu Baerts <[email protected]>

simult-flows-sh is still quite flaky :(

nf
--
To: Florian Westphal <[email protected]>

These are skipped because of some compatibility issues:

nft-flowtable-sh, bridge-brouter-sh, nft-audit-sh

Please LMK if I need to update the CLI tooling.
Or is this missing kernel config?

Hi!

tc-actions-sh
~~~~~~~~~~~~~
To: Davide Caratti <[email protected]>

It triggers a random unhandled interrupt, somehow (look at stderr). It's the only test that does that.

sch-tbf-ets-sh, sch-tbf-prio-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

These fail way too often on non-debug kernels :(
Perhaps we can extend the lower bound?

vxlan-bridge-1d-sh
~~~~~~~~~~~~~~~~~~
To: Ido Schimmel <[email protected]>
Cc: Petr Machata <[email protected]>

Flake fails almost always, with some form of "Expected to capture 0 packets, got $X"

mirror-gre-lag-lacp-sh
~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Often fails on debug with:

# TEST: mirror to gretap: LAG first slave (skip_hw) [FAIL]
# Expected to capture 10 packets, got 13.

mirror-gre-vlan-bridge-1q-sh, mirror-gre-bridge-1d-vlan-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Same kind of failure as above but less often and both on debug and non-debug.

Quoted 6 lines

sch-tbf-ets-sh, sch-tbf-prio-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

These fail way too often on non-debug kernels :(
Perhaps we can extend the lower bound?

Hm, it sometimes goes even below -10%. It looks like we'd need to go as low as -15%.

+ Aaron

Quoted 42 lines · Jakub Kicinski

Hi!

Feels like the efforts to get rid of flaky tests have slowed down a bit, so I thought I'd poke people..

Here's the full list:
https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0
click on test name to get the list of runs and links to outputs.

As a reminder please see these instructions for repro:
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

I'll try to tag folks who touched the tests most recently, but please don't hesitate to chime in.

net
---

arp-ndisc-untracked-subnets-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Jaehee Park <[email protected]>
Cc: Hangbin Liu <[email protected]>

Times out on debug kernels, passes on non-debug.
This is a real timeout, eats full 7200 seconds.

xfrm-policy-sh
~~~~~~~~~~~~~~
To: Hangbin Liu <[email protected]>

Times out on debug kernels, passed on non-debug,
This is a "inactivity" timeout, test doesn't print anything for 900 seconds so the runner kills it. We can bump the timeout but not printing for 15min is bad..

cmsg-time-sh
~~~~~~~~~~~~
To: Jakub Kicinski <[email protected]> (forgot I wrote this :D)

Fails randomly.

Quoted 5 lines

pmtu-sh
~~~~~~~
To: Simon Horman <[email protected]>

Skipped because it wants full OVS tooling.

My understanding is that Aaron (CCed) is working on addressing this problem by allowing the test to run without full OVS tooling.

Quoted 74 lines

forwarding
----------

sch-tbf-ets-sh, sch-tbf-prio-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

These fail way too often on non-debug kernels :(
Perhaps we can extend the lower bound?

bridge-igmp-sh, bridge-mld-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Nikolay Aleksandrov <[email protected]>
Cc: Ido Schimmel <[email protected]>

On debug kernels it always fails with:

# TEST: IGMPv3 group 239.10.10.10 exclude timeout [FAIL]
# Entry 192.0.2.21 has blocked flag failed

For MLD:

# TEST: MLDv2 group ff02::cc exclude timeout [FAIL]
# Entry 2001:db8:1::21 has blocked flag failed

vxlan-bridge-1d-sh
~~~~~~~~~~~~~~~~~~
To: Ido Schimmel <[email protected]>
Cc: Petr Machata <[email protected]>

Flake fails almost always, with some form of "Expected to capture 0 packets, got $X"

mirror-gre-lag-lacp-sh
~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Often fails on debug with:

# TEST: mirror to gretap: LAG first slave (skip_hw) [FAIL]
# Expected to capture 10 packets, got 13.

mirror-gre-vlan-bridge-1q-sh, mirror-gre-bridge-1d-vlan-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Same kind of failure as above but less often and both on debug and non-debug.

tc-actions-sh
~~~~~~~~~~~~~
To: Davide Caratti <[email protected]>

It triggers a random unhandled interrupt, somehow (look at stderr). It's the only test that does that.

mptcp
-----
To: Matthieu Baerts <[email protected]>

simult-flows-sh is still quite flaky :(

nf
--
To: Florian Westphal <[email protected]>

These are skipped because of some compatibility issues:

nft-flowtable-sh, bridge-brouter-sh, nft-audit-sh

Please LMK if I need to update the CLI tooling.
Or is this missing kernel config?

Quoted 6 lines · Jakub Kicinski

sch-tbf-ets-sh, sch-tbf-prio-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

These fail way too often on non-debug kernels :(
Perhaps we can extend the lower bound?

Hm, it sometimes goes even below -10%. It looks like we'd need to go as low as -15%.

Quoted 17 lines

vxlan-bridge-1d-sh
~~~~~~~~~~~~~~~~~~
To: Ido Schimmel <[email protected]>
Cc: Petr Machata <[email protected]>

Flake fails almost always, with some form of "Expected to capture 0 packets, got $X"

mirror-gre-lag-lacp-sh
~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Often fails on debug with:

# TEST: mirror to gretap: LAG first slave (skip_hw) [FAIL]
# Expected to capture 10 packets, got 13.

Quoted 5 lines

mirror-gre-vlan-bridge-1q-sh, mirror-gre-bridge-1d-vlan-sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To: Petr Machata <[email protected]>

Same kind of failure as above but less often and both on debug and non-debug.

I'll look into these.

On 10/05/2024 01:09, Jakub Kicinski wrote:

(...)

Quoted 4 lines

mptcp
-----
To: Matthieu Baerts <[email protected]>

Quoted 1 line

simult-flows-sh is still quite flaky :(

Yes, we need to find a solution for that. It is not as unstable on our side [1]. We will look at that next week. If we cannot find a solution quickly, we will skip the flaky subtests to stop the noise while continuing to investigate.

Quoted 4 lines

Yes, we need to find a solution for that. It is not as unstable on our side [1]. We will look at that next week. If we cannot find a solution quickly, we will skip the flaky subtests to stop the noise while continuing to investigate.

Now that the flaky MPTCP subtests results have been ignored, the results look better:

https://netdev.bots.linux.dev/flakes.html?br-cnt=88&tn-needle=mptcp

Do you think we could also stop ignoring them on NIPA side?

Quoted 4 lines · On Thu, 30 May 2024 19:35:56 +0200 Matthieu Baerts wrote:

Quoted 4 lines

Yes, we need to find a solution for that. It is not as unstable on our side [1]. We will look at that next week. If we cannot find a solution quickly, we will skip the flaky subtests to stop the noise while continuing to investigate.

Quoted 6 lines

Now that the flaky MPTCP subtests results have been ignored, the results look better:

https://netdev.bots.linux.dev/flakes.html?br-cnt=88&tn-needle=mptcp

Do you think we could also stop ignoring them on NIPA side?

Thanks for take care of it, done!