Skip to contentSkip to conversation

net-next is OPEN

ANN Jakub Kicinski 13 participants 67 messages Jan 22, 2024 – Feb 16, 2024

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

Hi,

net-next is open again, and accepting changes for v6.9.

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

reviewer
#1 lore raw
Quoted 8 lines · Jakub Kicinski

Hi,

net-next is open again, and accepting changes for v6.9.

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

Excellent, I'm really pleased to see this.

Quoted 4 lines

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

#2 lore raw
In reply to Jakub Kicinski
Quoted 8 lines · Jakub Kicinski

Hi,

net-next is open again, and accepting changes for v6.9.

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

Cool! Does it group a couple of patches together and run the tests or run for each patch separately?

Quoted 5 lines

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

Yes, this could push the selftest authors help checking all the failures.

[1] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

Thanks
Hangbin

#44 lore raw
In reply to Jakub Kicinski
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

A whole bunch of them fail because of no IPv6 support in the runner kernel. E.g. this from bridge-mdb.sh[0]:

# Error: Rule family not supported.
# Error: Rule family not supported.
# sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory # RTNETLINK answers: Operation not supported

I'm surprised any passed at all, it's super common for tests to validate their topology by pinging through, but I guess it's often just IPv4. I think the fix is just this?

    $ scripts/config -k -m CONFIG_IPV6

There are also a bunch of missing qdiscs, e.g. in [1], [2]. To fix:

    $ scripts/config -k -m CONFIG_NET_SCH_TBF
    $ scripts/config -k -m CONFIG_NET_SCH_PRIO
    $ scripts/config -k -m CONFIG_NET_SCH_ETS

Regarding sch_red.sh[3], I worry the test will be noisy, and suspect it does not make sense to run it in automated fashion. But if you think it's worth a try:

    $ scripts/config -k -m CONFIG_NET_SCH_RED

Then there are a bunch of missing netdevices. VXLAN[4]:

    $ scripts/config -k -m CONFIG_VXLAN

and GRE [5], which I think needs all of these:

    $ scripts/config -k -m CONFIG_NET_IPIP
    $ scripts/config -k -m CONFIG_IPV6_GRE
    $ scripts/config -k -m CONFIG_NET_IPGRE_DEMUX
    $ scripts/config -k -m CONFIG_NET_IPGRE

And TC actions [6]. I think the following will be necessary for some of the tests (we enable BPF as well internally).

    $ scripts/config -k -m CONFIG_NET_ACT_GACT
    $ scripts/config -k -m CONFIG_NET_ACT_MIRRED
    $ scripts/config -k -m CONFIG_NET_ACT_SAMPLE
    $ scripts/config -k -m CONFIG_NET_ACT_VLAN
    $ scripts/config -k -m CONFIG_NET_ACT_SKBEDIT
    $ scripts/config -k -m CONFIG_NET_ACT_PEDIT
    $ scripts/config -k -m CONFIG_NET_ACT_POLICE

Hopefully the above should clean up the results a bit, I can take another sweep afterwards.

[0] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/14-bridge-mdb-sh/stdout
[1] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/17-sch-ets-sh/stdout
[2] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/33-sch-tbf-prio-sh/stdout
[3] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/21-sch-red-sh/stdout
[4] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/27-mirror-gre-changes-sh/stdout
[5] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/18-vxlan-bridge-1d-sh/stdout
[6] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/42-pedit-l4port-sh/stdout

#45 lore raw
Quoted 58 lines · Petr Machata
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

A whole bunch of them fail because of no IPv6 support in the runner kernel. E.g. this from bridge-mdb.sh[0]:

# Error: Rule family not supported.
# Error: Rule family not supported.
# sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory # RTNETLINK answers: Operation not supported

I'm surprised any passed at all, it's super common for tests to validate their topology by pinging through, but I guess it's often just IPv4. I think the fix is just this?

     $ scripts/config -k -m CONFIG_IPV6

There are also a bunch of missing qdiscs, e.g. in [1], [2]. To fix:

     $ scripts/config -k -m CONFIG_NET_SCH_TBF
     $ scripts/config -k -m CONFIG_NET_SCH_PRIO
     $ scripts/config -k -m CONFIG_NET_SCH_ETS

Regarding sch_red.sh[3], I worry the test will be noisy, and suspect it does not make sense to run it in automated fashion. But if you think it's worth a try:

     $ scripts/config -k -m CONFIG_NET_SCH_RED

Then there are a bunch of missing netdevices. VXLAN[4]:

     $ scripts/config -k -m CONFIG_VXLAN

and GRE [5], which I think needs all of these:

     $ scripts/config -k -m CONFIG_NET_IPIP
     $ scripts/config -k -m CONFIG_IPV6_GRE
     $ scripts/config -k -m CONFIG_NET_IPGRE_DEMUX
     $ scripts/config -k -m CONFIG_NET_IPGRE

And TC actions [6]. I think the following will be necessary for some of the tests (we enable BPF as well internally).

     $ scripts/config -k -m CONFIG_NET_ACT_GACT
     $ scripts/config -k -m CONFIG_NET_ACT_MIRRED
     $ scripts/config -k -m CONFIG_NET_ACT_SAMPLE
     $ scripts/config -k -m CONFIG_NET_ACT_VLAN
     $ scripts/config -k -m CONFIG_NET_ACT_SKBEDIT
     $ scripts/config -k -m CONFIG_NET_ACT_PEDIT
     $ scripts/config -k -m CONFIG_NET_ACT_POLICE

Hopefully the above should clean up the results a bit, I can take another sweep afterwards.

Quoted 8 lines

[0] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/14-bridge-mdb-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalB7DvzQR$
[1] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/17-sch-ets-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalDWGO6rP$
[2] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/33-sch-tbf-prio-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalMVxxlTD$
[3] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/21-sch-red-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalMfr0cOk$
[4] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/27-mirror-gre-changes-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalP3lTojb$
[5] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/18-vxlan-bridge-1d-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalBLEoTPs$
[6] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/42-pedit-l4port-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalOO5rTam$

Hi

Probably also missing is CONFIG_GENEVE
81-l2-tos-ttl-inherit-sh check operation with gre, vxlan and geneve, but modprobes first if the module is actually available. If it isn't, it just continues and doesn't fail.

Also there seems to be something wrong with ending, see https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)

BR
Matthias

author maintainer
#3 lore raw
In reply to Hangbin Liu
Quoted 7 lines · On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
Quoted 4 lines

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

Cool! Does it group a couple of patches together and run the tests or run for each patch separately?

It groups all patches outstanding in patchwork (which build cleanly). I'm hoping we could also do HW testing using this setup, so batching is a must. Not 100% sure it's the right direction for SW testing but there's one way to find out :)

author maintainer
#47 lore raw
In reply to Petr Machata
Quoted 8 lines · On Tue, 23 Jan 2024 14:38:24 +0100 Petr Machata wrote:
Quoted 3 lines · Matthias May

Also there seems to be something wrong with ending, see
https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)

It looks like whatever is serving the output should use MIME of "text/plain;charset=UTF-8" instead of just "text/plain".

😮️ interesting. The table characters are not part of ASCII, right? So it must be using some extended charset. Firefox hid the option to tweak encoding tho so IDK what it actually uses :S

author maintainer
#50 lore raw
In reply to Petr Machata
Quoted 7 lines · On Tue, 23 Jan 2024 10:55:09 +0100 Petr Machata wrote:
Quoted 4 lines

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

A whole bunch of them fail because of no IPv6 support in the runner kernel. E.g. this from bridge-mdb.sh[0]:

Thanks a lot for investigating! I take it that you're looking at forwarding? Please send a patch to add the missing configs to

tools/testing/selftests/net/forwarding/config

The runner uses that to configure the kernel on top of defconfig.

Unless I'm doing it wrong and the sub-directories are supposed to inherit the parent directory's config? So net/forwarding/ should be built with net/'s config? I could not find the info in docs, does anyone know?

#48 lore raw
In reply to Jakub Kicinski
Quoted 11 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 23 Jan 2024 14:38:24 +0100 Petr Machata wrote:
Quoted 3 lines · Matthias May

Also there seems to be something wrong with ending, see
https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)

It looks like whatever is serving the output should use MIME of "text/plain;charset=UTF-8" instead of just "text/plain".

😮️ interesting. The table characters are not part of ASCII, right?

Yeah, the table is UTF-8.

Quoted 2 lines

So it must be using some extended charset. Firefox hid the option to tweak encoding tho so IDK what it actually uses :S

My guess would be ISO-8859-1, but dunno. The developer console shows Firefox has had to guess the encoding, but not what guess it made.

It seems weird to guess anything but UTF-8 in this day and age. Maybe the logic is that modern web has correct content-type, because it needs UTF-8, so if it doesn't declare encoding, it's something ancient.

author maintainer
#49 lore raw
Quoted 10 lines · On Tue, 23 Jan 2024 17:05:12 +0100 Petr Machata wrote:
Quoted 4 lines
Quoted 2 lines

It looks like whatever is serving the output should use MIME of "text/plain;charset=UTF-8" instead of just "text/plain".

😮️ interesting. The table characters are not part of ASCII, right?

Yeah, the table is UTF-8.

Quoted 2 lines

So it must be using some extended charset. Firefox hid the option to tweak encoding tho so IDK what it actually uses :S

Quoted 6 lines

My guess would be ISO-8859-1, but dunno. The developer console shows Firefox has had to guess the encoding, but not what guess it made.

It seems weird to guess anything but UTF-8 in this day and age. Maybe the logic is that modern web has correct content-type, because it needs UTF-8, so if it doesn't declare encoding, it's something ancient.

Ah, sorry, I read your message backwards. I remember fixing the encoding on the NIPA nginx to make it utf-8 so I thought you said it's utf-8 and should be something else... But the runner has a separate instance, and there the charset is indeed not set. Fixed now!

#4 lore raw
In reply to Jakub Kicinski
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
Quoted 4 lines

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

Quoted 8 lines
Quoted 2 lines

Cool! Does it group a couple of patches together and run the tests or run for each patch separately?

It groups all patches outstanding in patchwork (which build cleanly). I'm hoping we could also do HW testing using this setup, so batching is a must. Not 100% sure it's the right direction for SW testing but there's one way to find out :)

Really cool. Thanks for spending time to make this happen.

Scanning the tests I wrote, I think most of the failures are config related. e.g., fib-nexthops.sh needs MPLS and those config settings are enabled in tools/testing/selftests/net/config.

Another one, fcnal-test, needs nettest built. From
https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:

# which: no nettest in (/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)

It is in the Makefile, so how should that dependency be defined for this new environment?

Finally, how can people replicate this setup to validate changes to get everything work correctly?

#51 lore raw
In reply to Jakub Kicinski
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 23 Jan 2024 10:55:09 +0100 Petr Machata wrote:
Quoted 4 lines

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

Quoted 7 lines
Quoted 2 lines

A whole bunch of them fail because of no IPv6 support in the runner kernel. E.g. this from bridge-mdb.sh[0]:

Thanks a lot for investigating! I take it that you're looking at forwarding? Please send a patch to add the missing configs to

tools/testing/selftests/net/forwarding/config

OK.

Quoted 6 lines

The runner uses that to configure the kernel on top of defconfig.

Unless I'm doing it wrong and the sub-directories are supposed to inherit the parent directory's config? So net/forwarding/ should be built with net/'s config? I could not find the info in docs, does anyone know?

I don't think they are, net/config defines CONFIG_VXLAN, but then the vxlan tests still complain about unknown device type. Though maybe there's another device type that it's missing...

What do I do to feed the config file to some build script to get a kernel image to test? I can of course just do something like cat config | xargs -n1 scripts/config -m, but I expect there's some automation for it and I just can't find it.

author maintainer
#52 lore raw
Quoted 5 lines · On Tue, 23 Jan 2024 18:04:19 +0100 Petr Machata wrote:
Quoted 4 lines

Unless I'm doing it wrong and the sub-directories are supposed to inherit the parent directory's config? So net/forwarding/ should be built with net/'s config? I could not find the info in docs, does anyone know?

Quoted 8 lines

I don't think they are, net/config defines CONFIG_VXLAN, but then the vxlan tests still complain about unknown device type. Though maybe there's another device type that it's missing...

What do I do to feed the config file to some build script to get a kernel image to test? I can of course just do something like cat config | xargs -n1 scripts/config -m, but I expect there's some automation for it and I just can't find it.

The CI script is based on virtme-ng. So it does this:

# $target is net or net/forwarding or drivers/net/bonding etc. make mrproper
vng -v -b -f tools/testing/selftests/$target

# build the scripts
make headers
make -C tools/testing/selftests/$target

vng -v -r arch/x86/boot/bzImage --user root

# inside the VM
make -C tools/testing/selftests TARGETS=$target run_tests

https://github.com/kuba-moo/nipa/blob/master/contest/remote/vmksft.py#L138

You're right, it definitely does not "inherit" net's config when running forwarding/net. I can easily make it do so, but I'm not clear what the expectation from the kselftest subsystem is. Because if other testers (people testing stable, KernelCI etc. et.c) don't "inherit" we better fill in the config completely so that the tests pass for everyone.

author maintainer
#5 lore raw
In reply to David Ahern
Quoted 6 lines · On Tue, 23 Jan 2024 09:51:07 -0700 David Ahern wrote:

Really cool. Thanks for spending time to make this happen.

Scanning the tests I wrote, I think most of the failures are config related. e.g., fib-nexthops.sh needs MPLS and those config settings are enabled in tools/testing/selftests/net/config.

Quoted 8 lines

Another one, fcnal-test, needs nettest built. From
https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:

# which: no nettest in (/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)

It is in the Makefile, so how should that dependency be defined for this new environment?

We run the tests with

make -C tools/testing/selftests TARGETS=net run_tests

the binary is there:

$ ls tools/testing/selftests/net/nettest
tools/testing/selftests/net/nettest

The script does:

if ! which nettest >/dev/null; then
PATH=$PWD:$PATH
if ! which nettest >/dev/null; then
echo "'nettest' command not found; skipping tests" exit $ksft_skip
fi
fi

But that's not exactly the message that gets printed up top. Is it coming from lib.sh? Not sure..

Quoted 2 lines

Finally, how can people replicate this setup to validate changes to get everything work correctly?

You can run the exact scripts the CI runs, they are pretty standalone. But they don't do any magic, really, I'd start by trying the steps I outlined in
https://lore.kernel.org/all/[email protected]/
and if you can't repro the failure LMK, I'll share more instructions on replicating exactly.

#6 lore raw
Quoted 13 lines · Jakub Kicinski
Quoted 12 lines · On Tue, 23 Jan 2024 09:51:07 -0700 David Ahern wrote:

Really cool. Thanks for spending time to make this happen.

Scanning the tests I wrote, I think most of the failures are config related. e.g., fib-nexthops.sh needs MPLS and those config settings are enabled in tools/testing/selftests/net/config.

Another one, fcnal-test, needs nettest built. From
https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:

# which: no nettest in (/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)

Quoted 6 lines
Quoted 2 lines

It is in the Makefile, so how should that dependency be defined for this new environment?

We run the tests with

make -C tools/testing/selftests TARGETS=net run_tests

thanks for the tip to direclty run the tests.

Quoted 8 lines

the binary is there:

$ ls tools/testing/selftests/net/nettest
tools/testing/selftests/net/nettest

The script does:

Quoted 7 lines

if ! which nettest >/dev/null; then
PATH=$PWD:$PATH
if ! which nettest >/dev/null; then
echo "'nettest' command not found; skipping tests" exit $ksft_skip
fi
fi

This fixes the PATH problem:

(quoted diff) +0 -0

b/tools/testing/selftests/net/fcnal-test.sh
index a8ad92850e63..0c5ac117155d 100755

tools/testing/selftests/net/fcnal-test.sh +2 -0
@@ -37,6 +37,8 @@
 #
 # server / client nomenclature relative to ns-A
 
+PATH=$PWD:$PWD/tools/testing/selftests/net:$PATH
+
 VERBOSE=0
 
 NSA_DEV=eth1

but given the permutations it does, the script needs lot more than 45 seconds. This does the trick, but not sure how to bump the timeout for a specific test.

(quoted diff) +0 -0

b/tools/testing/selftests/kselftest/runner.sh
index cc9c846585f0..9e7dcb728249 100644

tools/testing/selftests/kselftest/runner.sh +1 -1
@@ -9,7 +9,7 @@ export per_test_logging=
 
 # Defaults for "settings" file fields:
 # "timeout" how many seconds to let each test run before failing.
-export kselftest_default_timeout=45
+export kselftest_default_timeout=3600
 
 # There isn't a shell-agnostic way to find the path of a sourced file,
 # so we must rely on BASE_DIR being set to find other tools.
maintainer
#7 lore raw

On Tue, 2024-01-23 at 22:20 -0700, David Ahern wrote:
[...]

Quoted 3 lines

the script needs lot more than 45
seconds. This does the trick, but not sure how to bump the timeout for a specific test.

You can set a test-group-specific timeout touching the 'settings' file in the relevant directory. Note that for 'net' self-tests the timeout is currently 3600 seconds (for each test).

AFAIK there is no way to set a single-test-specific timeout, without running that specific test individually:

make install TARGETS=net
./kselftest_install/run_kselftest.sh -o <timeout in sec> -t net:fcnal-test.sh

Cheers,

Paolo

author maintainer
#8 lore raw
Quoted 2 lines · On Wed, 24 Jan 2024 09:22:15 +0100 Paolo Abeni wrote:
Quoted 1 line · David Ahern

thanks for the tip to direclty run the tests.

Actually, I simplified this one a little bit. What we do now is:

make -C tools/testing/selftests TARGETS={target} TEST_PROGS={prog} TEST_GEN_PROGS="" run_tests

This let's us run tests individually. This way we can put them in a FIFO and have multiple workers run them in parallel.

Quoted 8 lines
Quoted 3 lines

the script needs lot more than 45
seconds. This does the trick, but not sure how to bump the timeout for a specific test.

You can set a test-group-specific timeout touching the 'settings' file in the relevant directory. Note that for 'net' self-tests the timeout is currently 3600 seconds (for each test).

Quoted 5 lines

AFAIK there is no way to set a single-test-specific timeout, without running that specific test individually:

make install TARGETS=net
./kselftest_install/run_kselftest.sh -o <timeout in sec> -t net:fcnal-test.sh

Yet another thing run_kselftest.sh can do that make can't :( We should probably bite the bullet and switch to it at some point.

David, I applied your diff locally, hopefully I did it in time for the 7am Pacific run to have it included :)

#20 lore raw
In reply to David Ahern
Quoted 10 lines · David Ahern
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
Quoted 4 lines

Over the merge window I spent some time stringing together selftest runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check posted to each patch indicating whether selftests have passed or not.

Cool! Does it group a couple of patches together and run the tests or run for each patch separately?

Quoted 7 lines
Quoted 5 lines

It groups all patches outstanding in patchwork (which build cleanly). I'm hoping we could also do HW testing using this setup, so batching is a must. Not 100% sure it's the right direction for SW testing but there's one way to find out :)

Really cool. Thanks for spending time to make this happen.

Just to add to the choir: this is fantastic, thanks!

Hopefully it will encourage people to add kselftests, kunit tests or other kinds that now get continuous coverage.

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

author maintainer
#9 lore raw
In reply to Jakub Kicinski
Quoted 2 lines · On Wed, 24 Jan 2024 07:07:55 -0800 Jakub Kicinski wrote:

David, I applied your diff locally, hopefully I did it in time for the 7am Pacific run to have it included :)

This is the latest run:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout

the nettest warning is indeed gone, but the failures are the same:

$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]

:(

author maintainer
#21 lore raw
In reply to Willem de Bruijn
Quoted 8 lines · On Wed, 24 Jan 2024 10:59:36 -0500 Willem de Bruijn wrote:
Quoted 7 lines · David Ahern
Quoted 5 lines · Jakub Kicinski

It groups all patches outstanding in patchwork (which build cleanly). I'm hoping we could also do HW testing using this setup, so batching is a must. Not 100% sure it's the right direction for SW testing but there's one way to find out :)

Quoted 6 lines
Quoted 1 line

Really cool. Thanks for spending time to make this happen.

Just to add to the choir: this is fantastic, thanks!

Hopefully it will encourage people to add kselftests, kunit tests or other kinds that now get continuous coverage.

Fingers crossed :)

Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

#10 lore raw
In reply to Jakub Kicinski
Quoted 9 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 24 Jan 2024 07:07:55 -0800 Jakub Kicinski wrote:

David, I applied your diff locally, hopefully I did it in time for the 7am Pacific run to have it included :)

This is the latest run:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout

the nettest warning is indeed gone, but the failures are the same:

yep, I will send a formal patch. I see the timeout is high enough, so good there.

Quoted 8 lines
$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]

:(

known problems. I can disable the tests for now so we avoid regressions, and add to the TO-DO list for someone with time.

author maintainer
#11 lore raw
Quoted 8 lines · On Wed, 24 Jan 2024 09:35:09 -0700 David Ahern wrote:
Quoted 5 lines

This is the latest run:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout

the nettest warning is indeed gone, but the failures are the same:

yep, I will send a formal patch. I see the timeout is high enough, so good there.

Well, kinda, to be honest I did bump the time to 4000s locally. The runtime of the entire net suite 1h 10min - that's pretty much the runtime of this one test :) The VMs run the tests without HW virtualization, so they are a bit slower, but it'd be nice if no local hacks were necessary.

I haven't sent a patch to bump it because it may make more sense to split the test into multiple. But as a stop gap we can as well bump the timeout.

Quoted 10 lines
Quoted 7 lines
$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]

:(

known problems. I can disable the tests for now so we avoid regressions, and add to the TO-DO list for someone with time.

Sounds good!

author maintainer
#22 lore raw
In reply to Jakub Kicinski
Quoted 11 lines · On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

#40 lore raw
In reply to Jakub Kicinski
Quoted 22 lines · Jakub Kicinski
Quoted 8 lines · On Wed, 24 Jan 2024 10:59:36 -0500 Willem de Bruijn wrote:
Quoted 7 lines · David Ahern
Quoted 5 lines · Jakub Kicinski

It groups all patches outstanding in patchwork (which build cleanly). I'm hoping we could also do HW testing using this setup, so batching is a must. Not 100% sure it's the right direction for SW testing but there's one way to find out :)

Quoted 6 lines
Quoted 1 line

Really cool. Thanks for spending time to make this happen.

Just to add to the choir: this is fantastic, thanks!

Hopefully it will encourage people to add kselftests, kunit tests or other kinds that now get continuous coverage.

Fingers crossed :)

Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Quoted 7 lines

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

I just looked at the result file and assumed 0 meant fine. Oops.

Technically udpgro is Paolo. But it is derived from udpgso.

But I'm on the hook for a few more. I added fq-band-pktlimit, our team added gso and txtimestamp I supposed is owned by the maintainers by now.

author maintainer
#41 lore raw
Quoted 9 lines · On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
Quoted 7 lines

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

I just looked at the result file and assumed 0 meant fine. Oops.

Sorry about the confusion there, make run_tests apparently always returns 0 and the result file holds the exit code :( It could be improved by I figured, as long as the JSON output is correct, investing the time in the web UI is probably a better choice than massaging the output files.

#42 lore raw
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
Quoted 7 lines

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Quoted 7 lines
Quoted 1 line

I just looked at the result file and assumed 0 meant fine. Oops.

Sorry about the confusion there, make run_tests apparently always returns 0 and the result file holds the exit code :( It could be improved by I figured, as long as the JSON output is correct, investing the time in the web UI is probably a better choice than massaging the output files.

Absolutely. My bad for jumping to conclusions.

maintainer
#43 lore raw
Quoted 12 lines · Willem de Bruijn
Quoted 11 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
Quoted 7 lines

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

I just looked at the result file and assumed 0 meant fine. Oops.

Quoted 7 lines
Quoted 5 lines

Sorry about the confusion there, make run_tests apparently always returns 0 and the result file holds the exit code :( It could be improved by I figured, as long as the JSON output is correct, investing the time in the web UI is probably a better choice than massaging the output files.

Absolutely. My bad for jumping to conclusions.

Some tests are failing because they need an xdp program build under the ebpf directory.

Since such program is used even by ebpf tests, I'll send a patch duplicating it under net, and using the existing rule for nat6to4 to build it.

Cheers,

Paolo

#23 lore raw
In reply to Jakub Kicinski

Hello,

24 Jan 2024 17:01:24 Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
Quoted 9 lines
Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

Quoted 7 lines
Quoted 2 lines

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I don't know if it is the same issue, but for MPTCP, we use 'iptables-legacy' if available.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Cheers,
Matt

author maintainer
#24 lore raw
Quoted 9 lines · On Wed, 24 Jan 2024 18:35:14 +0000 (GMT) Matthieu Baerts wrote:
Quoted 4 lines

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I don't know if it is the same issue, but for MPTCP, we use 'iptables-legacy' if available.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Great! Thanks for the pointer. I installed the packages now, so folks should be able to fix up their scripts.

#31 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

Quoted 7 lines
Quoted 2 lines

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I could only find in the listing above this:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout

which shows:

ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory

which seems like setup is broken, ie. it could not find libxt_conntrack.so

What is the issue?

#25 lore raw
In reply to Matthieu Baerts
Quoted 22 lines · Matthieu Baerts

Hello,

24 Jan 2024 17:01:24 Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
Quoted 9 lines
Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

Quoted 7 lines
Quoted 2 lines

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

Quoted 4 lines

I don't know if it is the same issue, but for MPTCP, we use 'iptables-legacy' if available.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

I'd suggest you do the other way around, first check if iptables-nft is available, otherwise fall back to iptables-nft

commit refers to 5.15 already have iptables-nft support, it should work out of the box.

author maintainer
#32 lore raw
In reply to Pablo Neira Ayuso
Quoted 6 lines · On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
Quoted 4 lines

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I could only find in the listing above this:

Thanks for taking a look!

Quoted 7 lines

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout

which shows:

ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory

which seems like setup is broken, ie. it could not find libxt_conntrack.so

Hm, odd, it's there:

$ ls /lib64/xtables/libxt_conntrack.so
/lib64/xtables/libxt_conntrack.so

but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 is in it (normal loaded always scans system paths)!

Quoted 1 line

What is the issue?

A lot of the tests print warning messages like the ones below. Some of them pass some of them fail. Tweaking the kernel config to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_* are included seem to have made no difference, which I concluded was because iptables CLI uses nf_tables here by default..

[435321]$ grep -nrI "Warning: Extension" .
./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module? ./6-fib-tests-sh/stdout:308:# Warning: Extension MARK revision 0 not supported, missing kernel module? ./6-fib-tests-sh/stdout:316:# Warning: Extension MARK revision 0 not supported, missing kernel module? ./6-fib-tests-sh/stdout:319:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:12:# No GRO Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:13:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:14:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:16:# GRO frag list Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:17:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:18:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:19:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:21:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:23:# GRO fwd Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:24:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:38:# GRO frag list over UDP tunnel Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:39:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:40:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:41:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:43:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:45:# GRO fwd over UDP tunnel Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:46:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:61:# No GRO Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:62:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:63:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:65:# GRO frag list Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:66:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:67:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:68:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:72:# GRO fwd Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:73:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:88:# GRO frag list over UDP tunnel Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:89:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:90:# Warning: Extension udp revision 0 not supported, missing kernel module? ./18-udpgro-fwd-sh/stdout:91:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:95:# GRO fwd over UDP tunnel Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:96:# Warning: Extension udp revision 0 not supported, missing kernel module? ./37-big-tcp-sh/stdout:17:# Warning: Extension length revision 0 not supported, missing kernel module? ./37-big-tcp-sh/stdout:19:# Warning: Extension length revision 0 not supported, missing kernel module? ./37-big-tcp-sh/stdout:22:# Warning: Extension length revision 0 not supported, missing kernel module? ./37-big-tcp-sh/stdout:24:# Warning: Extension length revision 0 not supported, missing kernel module? ./56-xfrm-policy-sh/stdout:11:# Warning: Extension policy revision 0 not supported, missing kernel module? ./56-xfrm-policy-sh/stdout:13:# Warning: Extension policy revision 0 not supported, missing kernel module? ./54-amt-sh/stdout:94:# Warning: Extension TTL revision 0 not supported, missing kernel module?

#33 lore raw
Quoted 18 lines · Jakub Kicinski
Quoted 6 lines · On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
Quoted 4 lines

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I could only find in the listing above this:

Thanks for taking a look!

Quoted 7 lines

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout

which shows:

ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory

which seems like setup is broken, ie. it could not find libxt_conntrack.so

Quoted 7 lines

Hm, odd, it's there:

$ ls /lib64/xtables/libxt_conntrack.so
/lib64/xtables/libxt_conntrack.so

but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 is in it (normal loaded always scans system paths)!

Could you also check your ./configure output for iptables? It shows the directory where the .so file are search and found:

...
Xtables extension directory: /usr/lib/xtables

Quoted 7 lines
Quoted 1 line

What is the issue?

A lot of the tests print warning messages like the ones below. Some of them pass some of them fail. Tweaking the kernel config to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_* are included seem to have made no difference, which I concluded was because iptables CLI uses nf_tables here by default..

Please, check if the symlink refers to -legacy or -nft via:

$ ls -la /usr/sbin/iptables
Quoted 2 lines

[435321]$ grep -nrI "Warning: Extension" .
./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?

This could come from either legacy or nftables:

libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",

both have the same error.

if that is the nftables backend, it might be also that .config is missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other options.

author maintainer
#34 lore raw
Quoted 6 lines · On Wed, 24 Jan 2024 21:02:16 +0100 Pablo Neira Ayuso wrote:
Quoted 5 lines · Jakub Kicinski

Hm, odd, it's there:

$ ls /lib64/xtables/libxt_conntrack.so
/lib64/xtables/libxt_conntrack.so
Quoted 8 lines
Quoted 2 lines

but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 is in it (normal loaded always scans system paths)!

Could you also check your ./configure output for iptables? It shows the directory where the .so file are search and found:

...
Xtables extension directory: /usr/lib/xtables

I'm using the OS iptables, I was hoping that for net/ tests OS iptables should be good enough :S Is there a way to get the info you're after?

$ iptables -V 
iptables v1.8.8 (nf_tables)
Quoted 11 lines
Quoted 7 lines
Quoted 1 line

What is the issue?

A lot of the tests print warning messages like the ones below. Some of them pass some of them fail. Tweaking the kernel config to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_* are included seem to have made no difference, which I concluded was because iptables CLI uses nf_tables here by default..

Please, check if the symlink refers to -legacy or -nft via:

$ ls -la /usr/sbin/iptables

Ah, neat:

$ ls -la /etc/alternatives/iptables
lrwxrwxrwx. 1 root root 22 Jan  7 22:10 /etc/alternatives/iptables -> /usr/sbin/iptables-nft
$ ls -la /usr/sbin/iptables
lrwxrwxrwx. 1 root root 26 Jan  7 22:10 /usr/sbin/iptables -> /etc/alternatives/iptables
Quoted 5 lines
Quoted 2 lines

[435321]$ grep -nrI "Warning: Extension" .
./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?

This could come from either legacy or nftables:

Quoted 8 lines
libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",

both have the same error.

if that is the nftables backend, it might be also that .config is missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other options.

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

#12 lore raw
In reply to Jakub Kicinski
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 24 Jan 2024 09:35:09 -0700 David Ahern wrote:
Quoted 4 lines

This is the latest run:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

the nettest warning is indeed gone, but the failures are the same:

yep, I will send a formal patch. I see the timeout is high enough, so good there.

Well, kinda, to be honest I did bump the time to 4000s locally. The runtime of the entire net suite 1h 10min - that's pretty much the runtime of this one test :) The VMs run the tests without

one *script* = 900+ tests. :-)

Quoted 6 lines

HW virtualization, so they are a bit slower, but it'd be nice if no local hacks were necessary.

I haven't sent a patch to bump it because it may make more sense to split the test into multiple. But as a stop gap we can as well bump the timeout.

The script has the tests in groups and each group can be run in parallel (with Hangbin's recent namespace changes). I can look at splitting that script into many (or write wrappers to run specific groups), but even then most of those test groups need more than 45 seconds. There are lot of permutations covered (with and without vrf, different address types, different uapis, ...).

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]

:(

known problems. I can disable the tests for now so we avoid regressions, and add to the TO-DO list for someone with time.

Sent a PR to fix a few things. I did not have to disable any tests - everything passes cleanly with the changes. If tests fail after those are applied, let's compare OS environments - maybe some sysctl is enabled or disabled (or a CONFIG) in your environment.

author maintainer
#35 lore raw
In reply to Jakub Kicinski
Quoted 9 lines · On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
Quoted 3 lines

if that is the nftables backend, it might be also that .config is missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other options.

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

NFT_COMPAT fixed a lot! One remaining warning comes from using -m length. Which NFT config do we need for that one?

maintainer
#36 lore raw
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
Quoted 3 lines

if that is the nftables backend, it might be also that .config is missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other options.

Quoted 8 lines
Quoted 5 lines

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

NFT_COMPAT fixed a lot! One remaining warning comes from using -m length. Which NFT config do we need for that one?

CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

#53 lore raw
In reply to Jakub Kicinski
Quoted 15 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 23 Jan 2024 18:04:19 +0100 Petr Machata wrote:
Quoted 4 lines

Unless I'm doing it wrong and the sub-directories are supposed to inherit the parent directory's config? So net/forwarding/ should be built with net/'s config? I could not find the info in docs, does anyone know?

Quoted 8 lines

I don't think they are, net/config defines CONFIG_VXLAN, but then the vxlan tests still complain about unknown device type. Though maybe there's another device type that it's missing...

What do I do to feed the config file to some build script to get a kernel image to test? I can of course just do something like cat config | xargs -n1 scripts/config -m, but I expect there's some automation for it and I just can't find it.

Quoted 5 lines

The CI script is based on virtme-ng. So it does this:

# $target is net or net/forwarding or drivers/net/bonding etc. make mrproper
vng -v -b -f tools/testing/selftests/$target

Actually:

# vng -v -b -f tools/testing/selftests/${target}/config

Didn't know about vng, it's way cool!
(Despite having used virtme for a long time.)

Quoted 7 lines
# build the scripts
make headers
make -C tools/testing/selftests/$target

vng -v -r arch/x86/boot/bzImage --user root

# inside the VM
make -C tools/testing/selftests TARGETS=$target run_tests

I'm working my way through the selftests.

Quoted 9 lines

https://github.com/kuba-moo/nipa/blob/master/contest/remote/vmksft.py#L138

You're right, it definitely does not "inherit" net's config when running forwarding/net. I can easily make it do so, but I'm not clear

what the expectation from the kselftest subsystem is. Because if other testers (people testing stable, KernelCI etc. et.c) don't "inherit" we better fill in the config completely so that the tests pass for everyone.

Oh, gotcha, the question was not whether it does, but whether it's supposed to. OK.

IMHO not necessarily. net/config is for net/*.sh, net/forwarding/config for net/forwarding/*.sh. It's not a given that whatever is needed for net/ is needed for net/forwarding/ as well.

It will also lead to simpler patches, where enabling config options in X/ doesn't imply checking for newly useless duplicities in X's child directories.

#38 lore raw
In reply to Jakub Kicinski
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
Quoted 3 lines

if that is the nftables backend, it might be also that .config is missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other options.

Quoted 8 lines
Quoted 5 lines

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

NFT_COMPAT fixed a lot! One remaining warning comes from using -m length. Which NFT config do we need for that one?

May I have a look at the logs? How does the error look like?

author maintainer
#37 lore raw
In reply to Florian Westphal
Quoted 4 lines · On Thu, 25 Jan 2024 09:52:11 +0100 Florian Westphal wrote:
Quoted 2 lines

NFT_COMPAT fixed a lot! One remaining warning comes from using -m length. Which NFT config do we need for that one?

CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

Thank you, that seems to solve the remaining problem!

author maintainer
#39 lore raw
In reply to Pablo Neira Ayuso
Quoted 4 lines · On Thu, 25 Jan 2024 10:29:41 +0100 Pablo Neira Ayuso wrote:
Quoted 2 lines

NFT_COMPAT fixed a lot! One remaining warning comes from using -m length. Which NFT config do we need for that one?

May I have a look at the logs? How does the error look like?

With the config pointed out by Florian in addition to NFT_COMPAT all the iptables errors in the logs are gone, without switching to legacy. Thank you for the help!

Also LMK if you guys want us to try running netfilter tests. I'm guessing we're unlikely to regress anything in net-next, and you have running netfilter tests covered - but if it's not too hard we can try to hook them up to net-next, too! (Or you can run them on our branch and report back :))

#13 lore raw
In reply to David Ahern
Quoted 8 lines · David Ahern
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]
# TEST: ping local, VRF bind - VRF IP                 [FAIL]
# TEST: ping local, device bind - ns-A IP             [FAIL]

:(

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout

still shows those 4 tests failing. since they pass on default Ubuntu 23.10, I need some information about the setup. What is the OS image in use and any known changes to the sysctl settings?

Can I get sysctl net > /tmp/sysctl.net ? I will compare to Ubuntu and see if I can figure out the difference and get those added to the script.

author maintainer
#14 lore raw
Quoted 9 lines · On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:

On 1/24/24 2:48 PM, David Ahern wrote:
https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout

still shows those 4 tests failing. since they pass on default Ubuntu 23.10, I need some information about the setup. What is the OS image in use and any known changes to the sysctl settings?

Can I get sysctl net > /tmp/sysctl.net ? I will compare to Ubuntu and see if I can figure out the difference and get those added to the script.

Here's a boot and run of the command (not sure how to export the file form the VM so I captured all of stdout):

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david

The OS is Amazon Linux, annoyingly.

#15 lore raw
Quoted 11 lines · Jakub Kicinski
Quoted 9 lines · On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:

On 1/24/24 2:48 PM, David Ahern wrote:
https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout

still shows those 4 tests failing. since they pass on default Ubuntu 23.10, I need some information about the setup. What is the OS image in use and any known changes to the sysctl settings?

Can I get sysctl net > /tmp/sysctl.net ? I will compare to Ubuntu and see if I can figure out the difference and get those added to the script.

Quoted 6 lines

Here's a boot and run of the command (not sure how to export the file form the VM so I captured all of stdout):

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david

The OS is Amazon Linux, annoyingly.

It's a bug in that version of iputils ping. It sets the BINDTODEVICE and then resets it because the source address is not set on the command line (it should not be required).

There are a couple of workarounds - one which might not age well (ie., amazon linux moving forward to newer packages -I <addr> -I <vrf>) and one that bypasses the purpose of the test (ip vrf exec)).

maintainer
#16 lore raw
Quoted 19 lines · David Ahern
Quoted 11 lines · Jakub Kicinski
Quoted 9 lines · On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:

On 1/24/24 2:48 PM, David Ahern wrote:
https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout

still shows those 4 tests failing. since they pass on default Ubuntu 23.10, I need some information about the setup. What is the OS image in use and any known changes to the sysctl settings?

Can I get sysctl net > /tmp/sysctl.net ? I will compare to Ubuntu and see if I can figure out the difference and get those added to the script.

Quoted 6 lines

Here's a boot and run of the command (not sure how to export the file form the VM so I captured all of stdout):

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david

The OS is Amazon Linux, annoyingly.

Quoted 7 lines

It's a bug in that version of iputils ping. It sets the BINDTODEVICE and then resets it because the source address is not set on the command line (it should not be required).

There are a couple of workarounds - one which might not age well (ie., amazon linux moving forward to newer packages -I <addr> -I <vrf>) and one that bypasses the purpose of the test (ip vrf exec)).

Could the script validate the 'ping' command WRT the bad behavior/bug and eventually skip the related tests?

Cheers,

Paolo

#54 lore raw
In reply to Jakub Kicinski
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

selftests-net/test-bridge-neigh-suppress-sh should be fixed by:

dnf install ndisc6

selftests-net/test-bridge-backup-port-sh should be fixed by:

https://lore.kernel.org/netdev/[email protected]/

selftests-net/drop-monitor-tests-sh should be fixed by:

dnf install dropwatch

#55 lore raw
Quoted 5 lines · Ido Schimmel
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

Forgot to mention: Thanks a lot for setting this up!

Quoted 12 lines

selftests-net/test-bridge-neigh-suppress-sh should be fixed by:

dnf install ndisc6

selftests-net/test-bridge-backup-port-sh should be fixed by:

https://lore.kernel.org/netdev/[email protected]/

selftests-net/drop-monitor-tests-sh should be fixed by:

dnf install dropwatch

author maintainer
#56 lore raw
Quoted 8 lines · On Mon, 29 Jan 2024 14:46:32 +0200 Ido Schimmel wrote:
Quoted 5 lines · Ido Schimmel
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

Forgot to mention: Thanks a lot for setting this up!

Finger crossed that it ends up being useful :)

Quoted 11 lines
Quoted 11 lines

selftests-net/test-bridge-neigh-suppress-sh should be fixed by:

dnf install ndisc6

selftests-net/test-bridge-backup-port-sh should be fixed by:

https://lore.kernel.org/netdev/[email protected]/

selftests-net/drop-monitor-tests-sh should be fixed by:

dnf install dropwatch

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

author maintainer
#17 lore raw
In reply to Paolo Abeni
Quoted 10 lines · On Mon, 29 Jan 2024 10:23:15 +0100 Paolo Abeni wrote:
Quoted 7 lines

It's a bug in that version of iputils ping. It sets the BINDTODEVICE and then resets it because the source address is not set on the command line (it should not be required).

There are a couple of workarounds - one which might not age well (ie., amazon linux moving forward to newer packages -I <addr> -I <vrf>) and one that bypasses the purpose of the test (ip vrf exec)).

Could the script validate the 'ping' command WRT the bad behavior/bug and eventually skip the related tests?

Skipping if the package has a bug would be best, if we can figure that out. The latest version is fixed, right? I can build it locally, just like pretty much everything else..

#18 lore raw
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 29 Jan 2024 10:23:15 +0100 Paolo Abeni wrote:
Quoted 7 lines

It's a bug in that version of iputils ping. It sets the BINDTODEVICE and then resets it because the source address is not set on the command line (it should not be required).

There are a couple of workarounds - one which might not age well (ie., amazon linux moving forward to newer packages -I <addr> -I <vrf>) and one that bypasses the purpose of the test (ip vrf exec)).

Quoted 6 lines
Quoted 2 lines

Could the script validate the 'ping' command WRT the bad behavior/bug and eventually skip the related tests?

Skipping if the package has a bug would be best, if we can figure that out. The latest version is fixed, right? I can build it locally, just like pretty much everything else..

yes, that is why Ubuntu 23.10 passes. I downloaded iputils, built ping locally at the Amazon release version and did a side by side comparison of behavior to verify it is the ping command.

author maintainer
#19 lore raw
Quoted 10 lines · On Mon, 29 Jan 2024 08:32:33 -0700 David Ahern wrote:
Quoted 6 lines
Quoted 2 lines

Could the script validate the 'ping' command WRT the bad behavior/bug and eventually skip the related tests?

Skipping if the package has a bug would be best, if we can figure that out. The latest version is fixed, right? I can build it locally, just like pretty much everything else..

yes, that is why Ubuntu 23.10 passes. I downloaded iputils, built ping locally at the Amazon release version and did a side by side comparison of behavior to verify it is the ping command.

Built iputils from git manually.
Could not figure out how to convince mason not to insert /local/ into the install dir path so I copied the binaries manually :/ Seems to work, tho.

#57 lore raw
In reply to Jakub Kicinski
Quoted 18 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 29 Jan 2024 14:46:32 +0200 Ido Schimmel wrote:
Quoted 5 lines · Ido Schimmel
Quoted 4 lines · Jakub Kicinski

If you authored any net or drivers/net selftests, please look around and see if they are passing. If not - send patches or LMK what I need to do to make them pass on the runner.. Make sure to scroll down to the "Not reporting to patchwork" section.

Forgot to mention: Thanks a lot for setting this up!

Finger crossed that it ends up being useful :)

Quoted 6 lines
Quoted 6 lines

selftests-net/test-bridge-neigh-suppress-sh should be fixed by:

dnf install ndisc6

selftests-net/test-bridge-backup-port-sh should be fixed by:

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

https://lore.kernel.org/netdev/[email protected]/

selftests-net/drop-monitor-tests-sh should be fixed by:

dnf install dropwatch

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

Thanks!

The last two tests look good now, but the first still fails. Can you share the ndisc6 version information? I tested with [1] from [2].

If your copy of ndisc6 indeed works, then I might be missing some sysctl. I will be AFK tomorrow so I will look into it later this week.

[1]

$ ./ndisc6 --version
ndisc6: IPv6 Neighbor/Router Discovery userland tool 1.0.8 ($Rev$)
 built Jan 29 2024 on <redacted>
Configured with: ./configure 
Written by Remi Denis-Courmont

Copyright (C) 2004-2007 Remi Denis-Courmont
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[2] https://git.remlab.net/gitweb/?p=ndisc6.git

author maintainer
#58 lore raw
Quoted 10 lines · On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
Quoted 2 lines

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

Thanks!

The last two tests look good now, but the first still fails. Can you share the ndisc6 version information? I tested with [1] from [2].

If your copy of ndisc6 indeed works, then I might be missing some sysctl. I will be AFK tomorrow so I will look into it later this week.

Hm. Looks like our versions match. I put the entire tools root dir up on HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna fetch the exact binary, it only links with libc, it seems.

#59 lore raw
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
Quoted 2 lines

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

Thanks!

The last two tests look good now, but the first still fails. Can you share the ndisc6 version information? I tested with [1] from [2].

Quoted 6 lines
Quoted 2 lines

If your copy of ndisc6 indeed works, then I might be missing some sysctl. I will be AFK tomorrow so I will look into it later this week.

Hm. Looks like our versions match. I put the entire tools root dir up on HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna fetch the exact binary, it only links with libc, it seems.

I tried with your binary and on other setups and I'm unable to reproduce the failure. From the test output it seems the NS is never sent. If you can, attaching the verbose test output might help:

./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v

Thanks

#60 lore raw
Quoted 17 lines · Ido Schimmel
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
Quoted 2 lines

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

Thanks!

The last two tests look good now, but the first still fails. Can you share the ndisc6 version information? I tested with [1] from [2].

Quoted 6 lines
Quoted 2 lines

If your copy of ndisc6 indeed works, then I might be missing some sysctl. I will be AFK tomorrow so I will look into it later this week.

Hm. Looks like our versions match. I put the entire tools root dir up on HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna fetch the exact binary, it only links with libc, it seems.

Quoted 8 lines

I tried with your binary and on other setups and I'm unable to reproduce the failure. From the test output it seems the NS is never sent. If you can, attaching the verbose test output might help:

./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v

Thanks

Maybe change the subject of the mail thread? At least I'm confused if I read several times a day "net-next is OPEN".

author maintainer
#61 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

In reply to Ido Schimmel
Quoted 9 lines · On Wed, 31 Jan 2024 15:23:50 +0200 Ido Schimmel wrote:
Quoted 3 lines

Hm. Looks like our versions match. I put the entire tools root dir up on HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna fetch the exact binary, it only links with libc, it seems.

I tried with your binary and on other setups and I'm unable to reproduce the failure. From the test output it seems the NS is never sent. If you can, attaching the verbose test output might help:

./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v

FWIW I started two new instances on bare metal one with the same kernel as the nested VM and one with debug options enabled.

selftests-net/test-bridge-neigh-suppress-sh
- fails across all, so must be the OS rather than the "speed"

selftests-net/test-bridge-backup-port-sh
- passes on VM, metal-dbg
- fails on metal :S very reliably / every time:
https://netdev.bots.linux.dev/contest.html?test=test-bridge-backup-port-sh

# TEST: No forwarding out of swp1 [FAIL]

selftests-net/drop-monitor-tests-sh
- passes everywhere now

The info you asked for:

# ./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v
[   49.621534] eth0: renamed from veth0
[   49.654673] swp1: renamed from veth1
[   49.676235] ip (235) used greatest stack depth: 12032 bytes left
[   49.721434] veth0: renamed from veth1
[   49.739521] ip (241) used greatest stack depth: 11880 bytes left
[   49.760463] eth0: renamed from veth0
[   49.787523] swp1: renamed from veth1
[   50.019197] br0: port 1(swp1) entered blocking state
[   50.019610] br0: port 1(swp1) entered disabled state
[   50.019949] swp1: entered allmulticast mode
[   50.020307] swp1: entered promiscuous mode
[   50.021035] br0: port 1(swp1) entered blocking state
[   50.021415] br0: port 1(swp1) entered forwarding state
[   50.076445] br0: port 2(vx0) entered blocking state
[   50.076805] br0: port 2(vx0) entered disabled state
[   50.077139] vx0: entered allmulticast mode
[   50.077492] vx0: entered promiscuous mode
[   50.077812] br0: port 2(vx0) entered blocking state
[   50.078172] br0: port 2(vx0) entered forwarding state
[   50.238364] br0: port 1(swp1) entered blocking state
[   50.238735] br0: port 1(swp1) entered disabled state
[   50.239077] swp1: entered allmulticast mode
[   50.239434] swp1: entered promiscuous mode
[   50.240046] br0: port 1(swp1) entered blocking state
[   50.240395] br0: port 1(swp1) entered forwarding state
[   50.269453] br0: port 2(vx0) entered blocking state
[   50.269812] br0: port 2(vx0) entered disabled state
[   50.270142] vx0: entered allmulticast mode
[   50.270597] vx0: entered promiscuous mode
[   50.270893] br0: port 2(vx0) entered blocking state
[   50.271222] br0: port 2(vx0) entered forwarding state

Per-port NS suppression - VLAN 10
---------------------------------
COMMAND: tc -n sw1-hF6GFk qdisc replace dev vx0 clsact
COMMAND: tc -n sw1-hF6GFk filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip ff02::1:ff00:2 src_ip 2001:db8:1::1 type 135 code 0 action pass

[   55.411592] GACT probability NOT on

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress on
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress on"
hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress on neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0
TEST: "neigh_suppress" is on [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk fdb replace 5a:1d:b4:4b:25:16 dev vx0 master static vlan 10
TEST: FDB entry installation [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n sw1-hF6GFk neigh replace 2001:db8:1::2 lladdr 5a:1d:b4:4b:25:16 nud permanent dev br0.10
TEST: Neighbor entry installation [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 down
TEST: H2 down [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 up
TEST: H2 up [ OK ]

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress off
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress off"
hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress off neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0
TEST: "neigh_suppress" is off [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 down
TEST: H2 down [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 2

TEST: NS suppression [FAIL]
rc=1, expected 0

Per-port NS suppression - VLAN 20
---------------------------------
COMMAND: tc -n sw1-hF6GFk qdisc replace dev vx0 clsact
COMMAND: tc -n sw1-hF6GFk filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip ff02::1:ff00:2 src_ip 2001:db8:2::1 type 135 code 0 action pass
COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress on
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress on"
hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress on neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0
TEST: "neigh_suppress" is on [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk fdb replace 5a:1d:b4:4b:25:16 dev vx0 master static vlan 20
TEST: FDB entry installation [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n sw1-hF6GFk neigh replace 2001:db8:2::2 lladdr 5a:1d:b4:4b:25:16 nud permanent dev br0.20
TEST: Neighbor entry installation [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 down
TEST: H2 down [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 up
TEST: H2 up [ OK ]

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress off
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress off"
hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress off neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0
TEST: "neigh_suppress" is off [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 0

TEST: NS suppression [FAIL]
rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 down
TEST: H2 down [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20 Raw IPv6 socket: Operation not permitted
TEST: ndisc6 [FAIL]
rc=1, expected 2

TEST: NS suppression [FAIL]
rc=1, expected 0

[   57.883133] br0: port 1(swp1) entered disabled state
[   57.884754] swp1 (unregistering): left allmulticast mode
[   57.885191] swp1 (unregistering): left promiscuous mode
[   57.885580] br0: port 1(swp1) entered disabled state
Tests passed:  14
Tests failed:  28
[   57.966498] vx0: left allmulticast mode
[   57.966802] vx0: left promiscuous mode
[   57.967091] br0: port 2(vx0) entered disabled state
[   57.992675] br0: port 1(swp1) entered disabled state
[   58.017115] swp1 (unregistering): left allmulticast mode
[   58.017520] swp1 (unregistering): left promiscuous mode
[   58.017898] br0: port 1(swp1) entered disabled state
bash-5.2# [   58.092405] vx0: left allmulticast mode
[   58.092704] vx0: left promiscuous mode
[   58.092972] br0: port 2(vx0) entered disabled state
#62 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

Quoted 5 lines · Jakub Kicinski

FWIW I started two new instances on bare metal one with the same kernel as the nested VM and one with debug options enabled.

selftests-net/test-bridge-neigh-suppress-sh
- fails across all, so must be the OS rather than the "speed"

Yes, it's something related to the OS. From the log below:

```
 COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
 Raw IPv6 socket: Operation not permitted
 TEST: ndisc6                                                        [FAIL]
     rc=1, expected 0
```

The test is supposed to be run as root so I'm not sure what this error is about. Do you have something like AppArmor or SELinux running? The program creates an IPv6 raw socket and requires CAP_NET_RAW.

Quoted 6 lines

selftests-net/test-bridge-backup-port-sh
- passes on VM, metal-dbg
- fails on metal :S very reliably / every time:
https://netdev.bots.linux.dev/contest.html?test=test-bridge-backup-port-sh

# TEST: No forwarding out of swp1 [FAIL]

Passes on all setups:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/446482/12-test-bridge-backup-port-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net/results/446481/7-test-bridge-backup-port-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/446481/12-test-bridge-backup-port-sh/stdout

With this patch:

https://lore.kernel.org/netdev/[email protected]/

Will submit v2 next week.

Quoted 2 lines

selftests-net/drop-monitor-tests-sh
- passes everywhere now

Nice, thanks

author maintainer
#63 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

Quoted 12 lines · On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
Quoted 2 lines

selftests-net/test-bridge-neigh-suppress-sh
- fails across all, so must be the OS rather than the "speed"

Yes, it's something related to the OS. From the log below:

```
 COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
 Raw IPv6 socket: Operation not permitted
 TEST: ndisc6                                                        [FAIL]
     rc=1, expected 0
```
Quoted 3 lines

The test is supposed to be run as root so I'm not sure what this error is about. Do you have something like AppArmor or SELinux running? The program creates an IPv6 raw socket and requires CAP_NET_RAW.

Ah, ugh, sorry for the misdirection, you're right.

Looks like the binaries have SUID set:

# find tools/fs/ -perm -4000
tools/fs/usr/bin/ndisc6
tools/fs/usr/bin/rdisc6
tools/fs/usr/bin/rltraceroute6

But I install them as a normal user:

# ll tools/fs/usr/bin/ndisc6
-rwsr-xr-x. 1 virtme virtme 53840 Jan 29 14:36 tools/fs/usr/bin/ndisc6

so I guess they intend to SUID themselves into privileges but end up SUIDing out to a lowly user :(

I cleared the SUID bits out, let's see the next run.

author maintainer
#64 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

Quoted 13 lines · On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:
Quoted 12 lines · On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
Quoted 2 lines

selftests-net/test-bridge-neigh-suppress-sh
- fails across all, so must be the OS rather than the "speed"

Yes, it's something related to the OS. From the log below:

```
 COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
 Raw IPv6 socket: Operation not permitted
 TEST: ndisc6                                                        [FAIL]
     rc=1, expected 0
```
Quoted 5 lines
Quoted 3 lines

The test is supposed to be run as root so I'm not sure what this error is about. Do you have something like AppArmor or SELinux running? The program creates an IPv6 raw socket and requires CAP_NET_RAW.

Ah, ugh, sorry for the misdirection, you're right.

Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now passes on everything with the exception of metal+debug kernel.

#26 lore raw
In reply to Pablo Neira Ayuso

Hi Pablo,

Thank you for your reply!

Quoted 26 lines · Pablo Neira Ayuso
Quoted 25 lines · Matthieu Baerts

Hello,

24 Jan 2024 17:01:24 Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
Quoted 9 lines
Quoted 3 lines

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

Quoted 7 lines
Quoted 2 lines

I have been attributing the udpg[rs]o and timestamp tests to you, but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is mapping to nftables on the executor. And either nftables doesn't support the functionality the tests expect or we're missing configs :( E.g. the TTL module.

I don't know if it is the same issue, but for MPTCP, we use 'iptables-legacy' if available.

Quoted 7 lines
Quoted 1 line

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

I'd suggest you do the other way around, first check if iptables-nft is available, otherwise fall back to iptables-nft

commit refers to 5.15 already have iptables-nft support, it should work out of the box.

Good point, I understand it sounds better to use 'iptables-nft' in new kselftests. I should have added a bit of background and not just a link to this commit: at that time (around ~v6.4), we didn't need to force using 'iptables-legacy' on -net or net-next tree. But we needed that when testing kernels <= v5.15.

When validating (old) stable kernels, the recommended practice is apparently [1] to use the kselftests from the last stable version, e.g. using the kselftests from v6.7.4 when validating kernel v5.15.148. The kselftests are then supposed to support older kernels, e.g. by skipping some parts if a feature is not available. I didn't know about that before, and I don't know if all kselftests devs know about that.

I don't think that's easy to support old kernels, especially in the networking area, where some features/behaviours are not directly exposed to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms or use other (ugly?) workarounds [2] to predict what we are supposed to have, depending on the kernel that is being used. But something has to be done, not to have big kselftests, with many different subtests, always marked as "failed" when validating new stable releases.

Back to the modification to use 'iptables-legacy', maybe a kernel config was missing, but the same kselftest, with the same list of kconfig to add, was not working with the v5.15 kernel, while everything was OK with a v6.4 one. With 'iptables-legacy', the test was running fine on both. I will check if maybe an old kconfig option was not missing.

[1] https://lore.kernel.org/stable/[email protected]/
[2]
https://lore.kernel.org/netdev/20230609-upstream-net-20230610-mptcp-selftests-support-old-kernels-part-3-v1-0-2896fe2ee8a3@tessares.net

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#27 lore raw

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]

Quoted 6 lines

Good point, I understand it sounds better to use 'iptables-nft' in new kselftests. I should have added a bit of background and not just a link to this commit: at that time (around ~v6.4), we didn't need to force using 'iptables-legacy' on -net or net-next tree. But we needed that when testing kernels <= v5.15.

Quoted 6 lines

When validating (old) stable kernels, the recommended practice is apparently [1] to use the kselftests from the last stable version, e.g. using the kselftests from v6.7.4 when validating kernel v5.15.148. The kselftests are then supposed to support older kernels, e.g. by skipping some parts if a feature is not available. I didn't know about that before, and I don't know if all kselftests devs know about that.

We are sending backports to stable kernels, if one stable kernel fails, then we have to fix it.

Quoted 7 lines

I don't think that's easy to support old kernels, especially in the networking area, where some features/behaviours are not directly exposed to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms or use other (ugly?) workarounds [2] to predict what we are supposed to have, depending on the kernel that is being used. But something has to be done, not to have big kselftests, with many different subtests, always marked as "failed" when validating new stable releases.

iptables-nft is supported in all of the existing stable kernels.

Quoted 5 lines

Back to the modification to use 'iptables-legacy', maybe a kernel config was missing, but the same kselftest, with the same list of kconfig to add, was not working with the v5.15 kernel, while everything was OK with a v6.4 one. With 'iptables-legacy', the test was running fine on both. I will check if maybe an old kconfig option was not missing.

I suspect this is most likely kernel config missing, as it happened to Jakub.

Thanks.

#28 lore raw

Hi Pablo,

Thank you for your reply!

Quoted 17 lines · Pablo Neira Ayuso

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]

Quoted 6 lines

Good point, I understand it sounds better to use 'iptables-nft' in new kselftests. I should have added a bit of background and not just a link to this commit: at that time (around ~v6.4), we didn't need to force using 'iptables-legacy' on -net or net-next tree. But we needed that when testing kernels <= v5.15.

Quoted 6 lines

When validating (old) stable kernels, the recommended practice is apparently [1] to use the kselftests from the last stable version, e.g. using the kselftests from v6.7.4 when validating kernel v5.15.148. The kselftests are then supposed to support older kernels, e.g. by skipping some parts if a feature is not available. I didn't know about that before, and I don't know if all kselftests devs know about that.

Quoted 2 lines

We are sending backports to stable kernels, if one stable kernel fails, then we have to fix it.

Do you validate stable kernels by running the kselftests from the same version (e.g. both from v5.15.x) or by using the kselftests from the last stable one (e.g. kernel v5.15.148 validated using the kselftests from v6.7.4)?

Quoted 9 lines
Quoted 7 lines

I don't think that's easy to support old kernels, especially in the networking area, where some features/behaviours are not directly exposed to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms or use other (ugly?) workarounds [2] to predict what we are supposed to have, depending on the kernel that is being used. But something has to be done, not to have big kselftests, with many different subtests, always marked as "failed" when validating new stable releases.

iptables-nft is supported in all of the existing stable kernels.

OK, then we should not have had the bug we had. I thought we were using features that were not supported in v5.15.

Quoted 7 lines
Quoted 5 lines

Back to the modification to use 'iptables-legacy', maybe a kernel config was missing, but the same kselftest, with the same list of kconfig to add, was not working with the v5.15 kernel, while everything was OK with a v6.4 one. With 'iptables-legacy', the test was running fine on both. I will check if maybe an old kconfig option was not missing.

I suspect this is most likely kernel config missing, as it happened to Jakub.

Probably, yes. I just retried by testing a v5.15.148 kernel using the kselftests from the net-next tree and forcing iptables-nft: I no longer have the issue I had one year ago. Not sure why, we already had NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and similar, because we noticed some CI didn't have them?
Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#65 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

In reply to Jakub Kicinski
Quoted 14 lines · Jakub Kicinski
Quoted 13 lines · On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:
Quoted 12 lines · On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
Quoted 2 lines

selftests-net/test-bridge-neigh-suppress-sh
- fails across all, so must be the OS rather than the "speed"

Yes, it's something related to the OS. From the log below:

```
 COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
 Raw IPv6 socket: Operation not permitted
 TEST: ndisc6                                                        [FAIL]
     rc=1, expected 0
```
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The test is supposed to be run as root so I'm not sure what this error is about. Do you have something like AppArmor or SELinux running? The program creates an IPv6 raw socket and requires CAP_NET_RAW.

Ah, ugh, sorry for the misdirection, you're right.

Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now passes on everything with the exception of metal+debug kernel.

I'm sorry for bothering you with this, but I checked today's logs and it seems there is a similar problem with arping:
https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/456561/8-test-bridge-neigh-suppress-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net/results/456562/6-test-bridge-neigh-suppress-sh/stdout

And according to this log the ndisc6 problem resurfaced:
https://netdev-3.bots.linux.dev/vmksft-net/results/456382/6-test-bridge-neigh-suppress-sh/stdout

Any chance that something in the OS changed since last week?

author maintainer
#66 lore raw

Re: [TEST] bridge tests (was: net-next is OPEN)

Quoted 8 lines · On Thu, 8 Feb 2024 18:21:29 +0200 Ido Schimmel wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:

Ah, ugh, sorry for the misdirection, you're right.

Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now passes on everything with the exception of metal+debug kernel.

I'm sorry for bothering you with this,

No worries at all :)

Quoted 7 lines

but I checked today's logs and it
seems there is a similar problem with arping:
https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/456561/8-test-bridge-neigh-suppress-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net/results/456562/6-test-bridge-neigh-suppress-sh/stdout

And according to this log the ndisc6 problem resurfaced:
https://netdev-3.bots.linux.dev/vmksft-net/results/456382/6-test-bridge-neigh-suppress-sh/stdout

It seems pretty faily:

https://netdev.bots.linux.dev/flakes.html?br-cnt=80&tn-needle=bridge-neigh&min-flip=0

I guess the short green streak does coincide with the fix, tho. Hm.

Quoted 1 line

Any chance that something in the OS changed since last week?

My memory is bad so I started keeping a log:

https://docs.google.com/spreadsheets/d/1mFnt91SKtA9ENIUfY0v2UmZSXJzmvsa4oqha4t6JVBs/edit?pli=1#gid=338019672

But it's not the suid:

$ find tools/fs/ -perm -4000
$ ll $(find tools/fs/ -name arping)
-rwxr-xr-x. 1 virtme virtme 79360 Jan 29 14:38 tools/fs/usr/bin/arping
$ ll $(find tools/fs/ -name ndisc6)
-rwxr-xr-x. 1 virtme virtme 53840 Jan 29 14:36 tools/fs/usr/bin/ndisc6

And it occasionally blips to "pass" so can't be completely broken env.

I run the test now with VERBOSE=1 and -q removed from arping and ndisc6:
https://netdev-3.bots.linux.dev/vmksft-net/test_bridge_neigh_suppress/

It just says: Received 0 response(s) :(

Repro is 100%, LMK if you'd like me to try with some custom diff on top..

#29 lore raw
In reply to Matthieu Baerts

Hi,

Sorry for taking a while.

Quoted 22 lines · Matthieu Baerts

Hi Pablo,

Thank you for your reply!

Quoted 17 lines · Pablo Neira Ayuso

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]

Quoted 12 lines

Good point, I understand it sounds better to use 'iptables-nft' in new kselftests. I should have added a bit of background and not just a link to this commit: at that time (around ~v6.4), we didn't need to force using 'iptables-legacy' on -net or net-next tree. But we needed that when testing kernels <= v5.15.

When validating (old) stable kernels, the recommended practice is apparently [1] to use the kselftests from the last stable version, e.g. using the kselftests from v6.7.4 when validating kernel v5.15.148. The kselftests are then supposed to support older kernels, e.g. by skipping some parts if a feature is not available. I didn't know about that before, and I don't know if all kselftests devs know about that.

Quoted 7 lines
Quoted 2 lines

We are sending backports to stable kernels, if one stable kernel fails, then we have to fix it.

Do you validate stable kernels by running the kselftests from the same version (e.g. both from v5.15.x) or by using the kselftests from the last stable one (e.g. kernel v5.15.148 validated using the kselftests from v6.7.4)?

We have kselftests, but nftables/tests/shell probe for kernel capabilities then it runs tests according to what the kernel supports, this includes packet and control plane path tests. For iptables, there are iptables-tests.py for the control plane path.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I don't think that's easy to support old kernels, especially in the networking area, where some features/behaviours are not directly exposed to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms or use other (ugly?) workarounds [2] to predict what we are supposed to have, depending on the kernel that is being used. But something has to be done, not to have big kselftests, with many different subtests, always marked as "failed" when validating new stable releases.

Quoted 4 lines
Quoted 1 line

iptables-nft is supported in all of the existing stable kernels.

OK, then we should not have had the bug we had. I thought we were using features that were not supported in v5.15.

I don't think so, iptables-nft supports the same features as iptables-legacy.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Back to the modification to use 'iptables-legacy', maybe a kernel config was missing, but the same kselftest, with the same list of kconfig to add, was not working with the v5.15 kernel, while everything was OK with a v6.4 one. With 'iptables-legacy', the test was running fine on both. I will check if maybe an old kconfig option was not missing.

Quoted 8 lines
Quoted 1 line

I suspect this is most likely kernel config missing, as it happened to Jakub.

Probably, yes. I just retried by testing a v5.15.148 kernel using the kselftests from the net-next tree and forcing iptables-nft: I no longer have the issue I had one year ago. Not sure why, we already had NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and similar, because we noticed some CI didn't have them?
Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Thanks. If you experience any issue, report back to netfilter-devel@

#30 lore raw

Hi Pablo,

Thank you for your reply!

Quoted 35 lines · Pablo Neira Ayuso

Hi,

Sorry for taking a while.

Quoted 22 lines · Matthieu Baerts

Hi Pablo,

Thank you for your reply!

Quoted 17 lines · Pablo Neira Ayuso

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]

Quoted 12 lines

Good point, I understand it sounds better to use 'iptables-nft' in new kselftests. I should have added a bit of background and not just a link to this commit: at that time (around ~v6.4), we didn't need to force using 'iptables-legacy' on -net or net-next tree. But we needed that when testing kernels <= v5.15.

When validating (old) stable kernels, the recommended practice is apparently [1] to use the kselftests from the last stable version, e.g. using the kselftests from v6.7.4 when validating kernel v5.15.148. The kselftests are then supposed to support older kernels, e.g. by skipping some parts if a feature is not available. I didn't know about that before, and I don't know if all kselftests devs know about that.

Quoted 7 lines
Quoted 2 lines

We are sending backports to stable kernels, if one stable kernel fails, then we have to fix it.

Do you validate stable kernels by running the kselftests from the same version (e.g. both from v5.15.x) or by using the kselftests from the last stable one (e.g. kernel v5.15.148 validated using the kselftests from v6.7.4)?

Quoted 4 lines

We have kselftests, but nftables/tests/shell probe for kernel capabilities then it runs tests according to what the kernel supports, this includes packet and control plane path tests. For iptables, there are iptables-tests.py for the control plane path.

That's great! It is good to support all the different kernels.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I don't think that's easy to support old kernels, especially in the networking area, where some features/behaviours are not directly exposed to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms or use other (ugly?) workarounds [2] to predict what we are supposed to have, depending on the kernel that is being used. But something has to be done, not to have big kselftests, with many different subtests, always marked as "failed" when validating new stable releases.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

iptables-nft is supported in all of the existing stable kernels.

OK, then we should not have had the bug we had. I thought we were using features that were not supported in v5.15.

I don't think so, iptables-nft supports the same features as iptables-legacy.

We were probably unlucky and hit a kernel/userspace bug that has been fixed in between, sorry for the noise!

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

Back to the modification to use 'iptables-legacy', maybe a kernel config was missing, but the same kselftest, with the same list of kconfig to add, was not working with the v5.15 kernel, while everything was OK with a v6.4 one. With 'iptables-legacy', the test was running fine on both. I will check if maybe an old kconfig option was not missing.

I suspect this is most likely kernel config missing, as it happened to Jakub.

Quoted 8 lines
Quoted 6 lines

Probably, yes. I just retried by testing a v5.15.148 kernel using the kselftests from the net-next tree and forcing iptables-nft: I no longer have the issue I had one year ago. Not sure why, we already had NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and similar, because we noticed some CI didn't have them?
Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Thanks. If you experience any issue, report back to netfilter-devel@

Will do, thank you!

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.