selftests/net: integrate packetdrill with ksft
Subsystems (3)
RFC PATCH 1/1 selftests/net: integrate packetdrill with ksft
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").
This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.
To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)
Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH
# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py
Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py
---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.
kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
nf_conntrack: we can dedup the common.sh.
*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
--- tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++ 10 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore create mode 100644 tools/testing/selftests/net/packetdrill/Makefile create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
tools/testing/selftests/Makefile +4 -3
@@ -65,10 +65,11 @@ TARGETS += net/af_unix TARGETS += net/forwarding TARGETS += net/hsr TARGETS += net/mptcp -TARGETS += net/openvswitch -TARGETS += net/tcp_ao TARGETS += net/netfilter +TARGETS += net/openvswitch +TARGETS += net/packetdrill TARGETS += net/rds +TARGETS += net/tcp_ao TARGETS += nsfs TARGETS += perf_events TARGETS += pidfd
@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug # Networking tests want the net/lib target, include it automatically -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),) +ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),) ifeq ($(filter net/lib,$(TARGETS)),) INSTALL_DEP_TARGETS := net/lib endif
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
@@ -0,0 +1 @@ +tcp*sh
tools/testing/selftests/net/packetdrill/Makefile new file +28 -0
@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 + +# KSFT includes +TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py) + +# Packetdrill support file(s) +TEST_INCLUDES += tcp/common/defaults.sh +TEST_INCLUDES += tcp/common/set_sysctls.py + +# Packetdrill scripts: all .pkt in subdirectories +TEST_INCLUDES += $(wildcard tcp/**/*.pkt) + +# Create a separate ksft test for each subdirectory +# Running all packetdrill tests in one go will take too long +# +# For each tcp/$subdir, create a test script tcp_$subdir.py +# Exclude tcp/common, which is a helper directory +TEST_DIRS := $(wildcard tcp/*) +TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS)) +TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py) + +$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py + cp $< $@ + +# Needed to generate all TEST_CUSTOM_PROGS +all: $(TEST_CUSTOM_PROGS) + +include ../../lib.mk
tools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +import pathlib +import sys + +KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve() + +try: + sys.path.append(KSFT_DIR.as_posix()) + from net.lib.py import * +except ModuleNotFoundError as e: + ksft_pr("Failed importing `net` library from kernel sources") + ksft_pr(str(e)) + ktap_result(True, comment="SKIP") + sys.exit(4)
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +90 -0
90 lines not on this page · raw message
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh new file +63 -0
63 lines not on this page · raw message
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +38 -0
@@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +"""Sets sysctl values and writes a file that restores them. + +The arguments are of the form "<proc-file>=<val>" separated by spaces. +The program first reads the current value of the proc-file and creates +a shell script named "/tmp/sysctl_restore_${PACKETDRILL_PID}.sh" which +restores the values when executed. It then sets the new values. + +PACKETDRILL_PID is set by packetdrill to the pid of itself, so a .pkt +file could restore sysctls by running `/tmp/sysctl_restore_${PPID}.sh` +at the end. +""" + +import os +import subprocess +import sys + +filename = '/tmp/sysctl_restore_%s.sh' % os.environ['PACKETDRILL_PID'] + +# Open file for restoring sysctl values +restore_file = open(filename, 'w') +print('#!/bin/bash', file=restore_file) + +for a in sys.argv[1:]: + sysctl = a.split('=') + # sysctl[0] contains the proc-file name, sysctl[1] the new value + + # read current value and add restore command to file + cur_val = subprocess.check_output(['cat', sysctl[0]], universal_newlines=True) + print('echo "%s" > %s' % (cur_val.strip(), sysctl[0]), file=restore_file) + + # set new value + cmd = 'echo "%s" > %s' % (sysctl[1], sysctl[0]) + os.system(cmd) + +os.system('chmod u+x %s' % filename)
tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt new file +51 -0
51 lines not on this page · raw message
tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt new file +51 -0
51 lines not on this page · raw message
tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +// Test what happens when client does not provide MD5 on SYN, +// but then does on the ACK that completes the three-way handshake. + +`../common/defaults.sh` + +// Establish a connection. + 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 + +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 + +0 bind(3, ..., ...) = 0 + +0 listen(3, 1) = 0 + + +0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 10> + +0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8> +// Ooh, weird: client provides MD5 option on the ACK: + +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop> + +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop> + +// The TCP listener refcount should be 2, but on buggy kernels it can be 0: + +0 `grep " 0A " /proc/net/tcp /proc/net/tcp6 | grep ":1F90"` + +// Now here comes the legit ACK: + +.01 < . 1:1(0) ack 1 win 514 + +// Make sure the connection is OK: + +0 accept(3, ..., ...) = 4 + + +.01 write(4, ..., 1000) = 1000
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Jakub Kicinski
maintainerVery exciting to see packetdrill tests making their way upstream!
Quoted 9 lines · On Tue, 27 Aug 2024 15:32:29 -0400 Willem de Bruijn wrote:
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.
No preference here, the wrapper script uses little of the python code, and use of YNL seems unlikely, so it's a coin toss whether it's worth linking up to net/lib/py or just writing a bit of bash.
Quoted 5 lines
kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
Not sure what you mean by intermediate output (the perl wrapper prints immediately, do you have perl?). We do have some nested ktap parsing in nipa, but small tweaks would be necessary to "decapsulate" the first layer completely. My bigger concern would be runtime, if the time it takes to run all tests grows we spawn multiple VMs and load balance the test cases.
All in all, trying to support single script is probably not worth the extra effort.
Quoted 4 lines
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests.
Other than avoiding known flakes - no concerns. Is the distinction between "change detector" vs hard tests (uAPI change / RFC compliance), well defined? Not sure if that's really possible but if so it would be nice to "sort" the tests into different dirs.
Quoted 2 lines
And what
is the best way to eventually send up to 150 files, 7K LoC.
Just send them, slice into a handful (<10) of patches if you can. 7k LoC is same order of magnitude as initial drivers we merge.
To be clear let's start with a small patch like this one. Since this is a new target I'll have to reconfigure the runners. So we'll see how well this works once it's merged. Spinning up new runners is a bit tedious but here new target seems quite justified.
Quoted 10 lines
tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++
prolly need a config file to enable kconfig for MD5 ? perils of adding new targets
tools/testing/selftests/net/packetdrill/.gitignore line 1
Quoted 7 lines
tools/testing/selftests/net/packetdrill/.gitignore new file +1
-0@@ -0,0 +1 @@ +tcp*sh
Is this right or should it be tcp_*.py ?
tools/testing/selftests/net/packetdrill/Makefile line 1
Quoted 7 lines
tools/testing/selftests/net/packetdrill/Makefile new file +1
-0@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0
Would be nice to add a few lines here with an overview of how the packetdrill tests get executed. People may jump in here to try to add new tests, since that's how ksft usually operates.
tools/testing/selftests/net/packetdrill/Makefile line 5
Quoted 4 lines
tools/testing/selftests/net/packetdrill/Makefile new file +4
-0+def scriptname_to_testdir(filepath): + """Extract the directory to run from this filename.""" + + suffix = ".sh"
.sh again ?
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh line 63
Quoted 7 lines
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh +5
-0+# Override the default qdisc on the tun device. +# Many tests fail with timing errors if the default +# is FQ and that paces their flows. +tc qdisc add dev tun0 root pfifo +
nit: double new line
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
Quoted 5 lines
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +0
-0
What calls this one? I can't grep it out.
Quoted 35 lines · Jakub Kicinski
Very exciting to see packetdrill tests making their way upstream!
Quoted 9 lines · On Tue, 27 Aug 2024 15:32:29 -0400 Willem de Bruijn wrote:
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.No preference here, the wrapper script uses little of the python code, and use of YNL seems unlikely, so it's a coin toss whether it's worth linking up to
net/lib/pyor just writing a bit of bash.Quoted 5 lines
kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.Not sure what you mean by intermediate output (the perl wrapper prints immediately, do you have perl?). We do have some nested ktap parsing in nipa, but small tweaks would be necessary to "decapsulate" the first layer completely. My bigger concern would be runtime, if the time it takes to run all tests grows we spawn multiple VMs and load balance the test cases.
All in all, trying to support single script is probably not worth the extra effort.
Quoted 2 lines
nf_conntrack: we can dedup thecommon.sh.
Quoted 7 lines
Quoted 2 lines
*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests.
Other than avoiding known flakes - no concerns. Is the distinction between "change detector" vs hard tests (uAPI change / RFC compliance), well defined? Not sure if that's really possible but if so it would be nice to "sort" the tests into different dirs.
It's not as clear-cut as API/RFC.
The major change detector issues are with timing, as packetdrill scripts have timestamped lines. I think usec TCP timestamps were an issue at some point. On-going is that we increase allowed variance on debug builds. Note to self that that is missing here.
I think that all tests we open sourced to github so far were chosen to be robust. Will double check and err on the side of caution.
Quoted 11 lines
Quoted 2 lines
And what
is the best way to eventually send up to 150 files, 7K LoC.Just send them, slice into a handful (<10) of patches if you can. 7k LoC is same order of magnitude as initial drivers we merge.
To be clear let's start with a small patch like this one. Since this is a new target I'll have to reconfigure the runners. So we'll see how well this works once it's merged. Spinning up new runners is a bit tedious but here new target seems quite justified.
Ack. Thanks for that guidance.
Quoted 53 lines
Quoted 10 lines
tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++prolly need a config file to enable kconfig for MD5 ? perils of adding new targets
tools/testing/selftests/net/packetdrill/.gitignore line 1
Quoted 7 lines
tools/testing/selftests/net/packetdrill/.gitignore new file +1
-0@@ -0,0 +1 @@ +tcp*shIs this right or should it be tcp_*.py ?
tools/testing/selftests/net/packetdrill/Makefile line 1
Quoted 7 lines
tools/testing/selftests/net/packetdrill/Makefile new file +1
-0@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0Would be nice to add a few lines here with an overview of how the packetdrill tests get executed. People may jump in here to try to add new tests, since that's how ksft usually operates.
tools/testing/selftests/net/packetdrill/Makefile line 5
Quoted 4 lines
tools/testing/selftests/net/packetdrill/Makefile new file +4
-0+def scriptname_to_testdir(filepath): + """Extract the directory to run from this filename.""" + + suffix = ".sh".sh again ?
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh line 63
Quoted 7 lines
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh +5
-0+# Override the default qdisc on the tun device. +# Many tests fail with timing errors if the default +# is FQ and that paces their flows. +tc qdisc add dev tun0 root pfifo +nit: double new line
Quoted 7 lines
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
Quoted 5 lines
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +0
-0What calls this one? I can't grep it out.
Whoops. I changed test directories and apparently these ones don't rely on this.
Thanks for the review. I'll take a quick stab at a standalone script to see whether that is easier. Else will resubmit with these and Paolo's feedback addressed.
In particular to Paolo's feedback: running multiple tests in parallel like run_all.py can. Come to think of it, maybe I should import run_all.py, modifying it to output KTAP and ksft return codes.
Paolo Abeni
maintainerAdding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)
Quoted 80 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in nf_conntrack,Addin in commit
a8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.pyResult:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill:tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill:tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill:tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill:tcp_md5.pySigned-off-by Willem de Bruijn <[email protected]>---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.
Quoted 2 lines
kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack.
Additionally, in some setups the test directory is RO, avoding file creation there would be better.
What about placing in each subdiretory a trivial wrapper invoking the 'main' packetdrill_ksft.py script specifying as an argument the (sub-)directory to run/process?
Quoted 4 lines
A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).
I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime > 1h, but that is bad ;)
Quoted 5 lines
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
I have no idea WRT the overall test stability, is some specific case/dir is subject to very frequent false positive/false negative we could postpone importing them, but ideally IMHO all the github scripts are good candidates.
Side note: I think it would be great to have some easy command line parameter to run only the specified script/test-case.
Thanks!
Paolo
Quoted 87 lines · Paolo Abeni
Adding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)
Quoted 80 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in nf_conntrack,Addin in commit
a8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.pyResult:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill:tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill:tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill:tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill:tcp_md5.pySigned-off-by Willem de Bruijn <[email protected]>---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.Quoted 2 lines
kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack.
Quoted 6 lines
Additionally, in some setups the test directory is RO, avoding file creation there would be better.
What about placing in each subdiretory a trivial wrapper invoking the 'main'
packetdrill_ksft.pyscript specifying as an argument the (sub-)directory to run/process?
The files are created by kselftests, similar to any TEST_GEN_XXX or TEST_CUSTOM_PROGS.
If instead we prepopulate in each directory that is duplicative boilerplate committed to git, and we still need to add it to kselftests. Not sure whether TEST_PROGS with paths extending into subdirectories are supported. And we do not want a TARGET for each subdirectory.
It probably can be done, but I don't think that it is needed or simpler.
Quoted 5 lines
Quoted 4 lines
A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
Quoted 8 lines
If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill
run_all.pybehavior that uses many concurrent threads).I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime
Quoted 1 line
1h, but that is bad ;)
Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use run_all.py rather than reinvent the wheel here.
Quoted 6 lines
Quoted 5 lines
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
Quoted 7 lines
I have no idea WRT the overall test stability, is some specific case/dir is subject to very frequent false positive/false negative we could postpone importing them, but ideally IMHO all the github scripts are good candidates.
Side note: I think it would be great to have some easy command line parameter to run only the specified script/test-case.
Makes sense. Will include.
Jakub Kicinski
maintainerQuoted 9 lines · On Wed, 28 Aug 2024 10:03:55 -0400 Willem de Bruijn wrote:
Quoted 9 lines
Quoted 4 lines
A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill
run_all.pybehavior that uses many concurrent threads).
Quoted 8 lines
Quoted 4 lines
I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime
Quoted 1 line
1h, but that is bad ;)
Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use
run_all.pyrather than reinvent the wheel here.
Do we have to worry about this now? If you're planning to add a runner specifically for packetdrill... IDK if we should. We already have a few runners, not to mention that run_kselftest.sh itself can run all the test cases in parallel in separate network namespaces!
What I was wondering is whether we can use shebang to direct the .pkt files to be "executed" by the python script. Alternatively we can add support to specifying "interpreter" for a given test in ksft infra (kinda like we can pass cmd line arguments to a test). Completely untested but it should give better idea what I mean than a description:
tools/testing/selftests/kselftest/runner.sh +14 -1
@@ -56,6 +56,7 @@ run_one() export kselftest_timeout="$kselftest_default_timeout" # Safe default if tr not available + kselftest_interp_ref="KSELFTEST_INTERP" kselftest_cmd_args_ref="KSELFTEST_ARGS" # Optional arguments for this command, possibly defined as an
@@ -78,6 +79,14 @@ run_one() $TR_CMD [:lower:] [:upper:]) kselftest_cmd_args_ref="KSELFTEST_${BASENAME_SANITIZED}_ARGS" fi + # Optional interpreter to run the test case + if [ -n "$TR_CMD" ]; then + SUFFIX_SANITIZED=$(echo "${BASENAME_TEST#*.}" | \ + $TR_CMD -d "[:blank:][:cntrl:]" | \ + $TR_CMD -c "[:alnum:]_" "_" | \ + $TR_CMD [:lower:] [:upper:]) + kselftest_interp_ref="KSELFTEST_${SUFFIX_SANITIZED}_INTERP" + fi # Load per-test-directory kselftest "settings" file. settings="$BASE_DIR/$DIR/settings"
@@ -110,8 +119,12 @@ run_one() if [ -x /usr/bin/stdbuf ]; then stdbuf="/usr/bin/stdbuf --output=L " fi + eval kselftest_interp="\$${kselftest_interp_ref:-}" + # Add a space at the end if interpreter is set to work in $cmd + [ -n "$kselftest_interp" ] && \ + kselftest_interp="$kselftest_interp " eval kselftest_cmd_args="\$${kselftest_cmd_args_ref:-}" - cmd="$stdbuf ./$BASENAME_TEST $kselftest_cmd_args" + cmd="$stdbuf ./$kselftest_interp$BASENAME_TEST $kselftest_cmd_args" if [ ! -x "$TEST" ]; then echo "# Warning: file $TEST is not executable"
Quoted 60 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 28 Aug 2024 10:03:55 -0400 Willem de Bruijn wrote:
Quoted 9 lines
Quoted 4 lines
A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill
run_all.pybehavior that uses many concurrent threads).Quoted 8 lines
Quoted 4 lines
I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime
Quoted 1 line
1h, but that is bad ;)
Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use
run_all.pyrather than reinvent the wheel here.Do we have to worry about this now? If you're planning to add a runner specifically for packetdrill... IDK if we should. We already have a few runners, not to mention that
run_kselftest.shitself can run all the test cases in parallel in separate network namespaces!What I was wondering is whether we can use shebang to direct the .pkt files to be "executed" by the python script. Alternatively we can add support to specifying "interpreter" for a given test in ksft infra (kinda like we can pass cmd line arguments to a test). Completely untested but it should give better idea what I mean than a description:
tools/testing/selftests/kselftest/runner.sh +9
-0@@ -56,6 +56,7 @@ run_one() export kselftest_timeout="$kselftest_default_timeout" # Safe default if tr not available + kselftest_interp_ref="KSELFTEST_INTERP" kselftest_cmd_args_ref="KSELFTEST_ARGS" # Optional arguments for this command, possibly defined as an@@ -78,6 +79,14 @@ run_one() $TR_CMD [:lower:] [:upper:]) kselftest_cmd_args_ref="KSELFTEST_${BASENAME_SANITIZED}_ARGS" fi + # Optional interpreter to run the test case + if [ -n "$TR_CMD" ]; then + SUFFIX_SANITIZED=$(echo "${BASENAME_TEST#*.}" | \ + $TR_CMD -d "[:blank:][:cntrl:]" | \ + $TR_CMD -c "[:alnum:]_" "_" | \ + $TR_CMD [:lower:] [:upper:]) + kselftest_interp_ref="KSELFTEST_${SUFFIX_SANITIZED}_INTERP" + fi # Load per-test-directory kselftest "settings" file. settings="$BASE_DIR/$DIR/settings"@@ -110,8 +119,12 @@ run_one() if [ -x /usr/bin/stdbuf ]; then stdbuf="/usr/bin/stdbuf --output=L "
tools/testing/selftests/kselftest/runner.sh run_one() · line 129
Quoted 10 lines
tools/testing/selftests/kselftest/runner.sh +5
-1fi + eval kselftest_interp="\$${kselftest_interp_ref:-}" + # Add a space at the end if interpreter is set to work in $cmd + [ -n "$kselftest_interp" ] && \ + kselftest_interp="$kselftest_interp " eval kselftest_cmd_args="\$${kselftest_cmd_args_ref:-}" - cmd="$stdbuf ./$BASENAME_TEST $kselftest_cmd_args" + cmd="$stdbuf ./$kselftest_interp$BASENAME_TEST $kselftest_cmd_args" if [ ! -x "$TEST" ]; then echo "# Warning: file $TEST is not executable"
That could work.
Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?
Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file
That can be addressed by instead calling a small wrapper shell script.
That would do the test_func_builder part of packetdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.
Jakub Kicinski
maintainerQuoted 4 lines · On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
That could work.
Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?
To me "neither", hopefully and unhelpfully I won't tell you anything that's not common sense :) The balance is between:
- having test cases which don't take too long (assuming debug kernel):
<15min is good,
>1h I will start complaining,
>1h30m is bad because we can't retry and still make the 3h
deadline that NIPA has,
>3h the test can't run in NIPA at all.
vs
- flip side is having so many cases it's hard to keep track and render in the UI. JSON is relatively slow to process. If you have 150 cases, that's 300 per branch (debug and non-debug kernels), times 8 branches a day => 2.4k results / day
I think that's still fine-ish, but on the larger side for sure. For reference net and forwarding have ~100 tests each. FWIW we do have the ability to collect and display nested KTAP so the information is not lost (but it makes queries slower so we don't fetch it by default).
Quoted 10 lines
Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt fileThat can be addressed by instead calling a small wrapper shell script.
That would do the
test_func_builderpart ofpacketdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.
Right!
Quoted 32 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
That could work.
Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?
To me "neither", hopefully and unhelpfully I won't tell you anything that's not common sense :) The balance is between:
- having test cases which don't take too long (assuming debug kernel):
<15min is good,
>1h I will start complaining,
>1h30m is bad because we can't retry and still make the 3h
deadline that NIPA has,
>3h the test can't run in NIPA at all.vs
- flip side is having so many cases it's hard to keep track and render in the UI. JSON is relatively slow to process. If you have 150 cases, that's 300 per branch (debug and non-debug kernels), times 8 branches a day => 2.4k results / day
I think that's still fine-ish, but on the larger side for sure. For reference net and forwarding have ~100 tests each. FWIW we do have the ability to collect and display nested KTAP so the information is not lost (but it makes queries slower so we don't fetch it by default).Quoted 5 lines
Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file
Quoted 7 lines
Quoted 5 lines
That can be addressed by instead calling a small wrapper shell script.
That would do the
test_func_builderpart ofpacketdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.Right!
Implemented this, but hit a snag
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, source ../common/defaults.sh).
The output of having each test separate is also quite unreadable.
An alternative is to add each subdirectory as a separate TARGET. But that seems similarly impractical. And does nothing to group KTAP output.
In practice, most directories have a handful of .pkt scripts, and each script is fast, so serialization is not a huge issue.
I found tools/testing/selftests/net/kselftest/ktap_helpers.sh, which has helpful KTAP boilerplate for shell tests, including KSFT exit codes. Am using that instead of python, to reduce the external dependencies.
Jakub Kicinski
maintainerQuoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g,
source ../common/defaults.sh).
Can you show an example of exact commands and what happens?
We have directories in net/lib, and it's a target, and it works, no?
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g,
source ../common/defaults.sh).Can you show an example of exact commands and what happens?
Running directly works fine:
$ KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
$ make -C tools/testing/selftests \
TARGETS=net/packetdrill O=/tmp run_tests TAP version 13
1..3
# timeout set to 45
# selftests: net/packetdrill: client.pkt
# TAP version 13
# 1..2
# ok 1 ipv4
# ok 2 ipv6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
ok 1 selftests: net/packetdrill: client.pkt
[..etc..]Installing does not:
$ make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$INSTALL_DIR
$ cd $INSTALL_DIR
$ export KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
$ ./run_kselftest.sh -c net/packetdrill TAP version 13
1..3
# timeout set to 45
# selftests: net/packetdrill: client.pkt
# TAP version 13
# 1..2
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 1 ipv4
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 2 ipv6
# # Totals: pass:0 fail:2 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: net/packetdrill: client.pkt # exit=1
Due to that relative path to defaults.sh inside the scripts.
It is arguably a bit weird that the relative path of the TEST_PROGS differs before and after install.
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.
This issue with preserving paths until recently also existed for helper files (TEST_FILES). TEST_INCLUDES was added expressly to preserve those paths (commit 2a0683be5b4c).
Jakub Kicinski
maintainerQuoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in
tools/testing/selftests/Makefile. ItsMakefileonly generates dependencies for other targets:TEST_FILES,TEST_GEN_FILESandTEST_INCLUDES.
Oh right, TEST_FILES vs TEST_INCLUDES :(
Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only TEST_NAME with a / in it is
x86_64/nx_huge_pages_test.sh
But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:
mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in
tools/testing/selftests/Makefile. ItsMakefileonly generates dependencies for other targets:TEST_FILES,TEST_GEN_FILESandTEST_INCLUDES.
Quoted 6 lines
Oh right,
TEST_FILESvsTEST_INCLUDES:(Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only
TEST_NAMEwith a / in it is
x86_64/nx_huge_pages_test.sh
Oh interesting precedent. Let me take a look.
Quoted 9 lines
But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:
mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?
That's definitely simpler :)
I'd like to keep diffs between packetdrill scripts on github (and Google internal, we have more) and selftests to a minimum. This is invertible, as is rewriting source statements inside the pkt files. But that might be more work and more maintenance in the end.
Quoted 27 lines · Willem de Bruijn
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in
tools/testing/selftests/Makefile. ItsMakefileonly generates dependencies for other targets:TEST_FILES,TEST_GEN_FILESandTEST_INCLUDES.Quoted 6 lines
Oh right,
TEST_FILESvsTEST_INCLUDES:(Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only
TEST_NAMEwith a / in it is
x86_64/nx_huge_pages_test.shOh interesting precedent. Let me take a look.
Quoted 9 lines
But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:
mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?
Quoted 6 lines
That's definitely simpler :)
I'd like to keep diffs between packetdrill scripts on github (and Google internal, we have more) and selftests to a minimum. This is invertible, as is rewriting source statements inside the pkt files. But that might be more work and more maintenance in the end.
Thanks again for the pointer and suggestion.
Changing kselftests to preserve directories turns out to be trivial. Patch inline below.
But, existing TARGETS of course then start failing. Because they depend on existing rsync without -R. In (at least) two ways:
amd-pstate fails because its TEST_FILES has includes from other directories and it expects those files to land in the directory with tests.
x86 prefixes all its output with $(OUTPUT) to form absolute paths, which also creates absolute paths in kselftest-list.txt.
These two are examples, not necessarily the one instances of those patterns. So switching to preserving directories for existing targets like TEST_FILES seems intractable.
Plan B is to add a new TEST_PROGS_RECURSE, analogous to how TEST_INCLUDES extended TEST_FILES with optional path preservation. That is not much more complex.
---
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -101,7 +112,7 @@ run_one()
echo "# timeout set to $kselftest_timeout" >> "$logfile"
fi
- TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+ TEST_HDR_MSG="selftests: $DIR: $TEST"
+++ b/tools/testing/selftests/lib.mk
@@ -150,7 +150,7 @@ clean_mods_dir:
define INSTALL_SINGLE_RULE
$(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH))
- $(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
+ $(if $(INSTALL_LIST),rsync -aR --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
endef
define INSTALL_MODS_RULE
@@ -180,8 +180,7 @@ endif
emit_tests:
for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ - BASENAME_TEST=`basename $$TEST`; \ - echo "$(COLLECTION):$$BASENAME_TEST"; \ + echo "$(COLLECTION):$$TEST"; \
done
Jakub Kicinski
maintainerQuoted 10 lines · On Sun, 01 Sep 2024 17:15:38 -0400 Willem de Bruijn wrote:
Changing kselftests to preserve directories turns out to be trivial. Patch inline below.
But, existing TARGETS of course then start failing. Because they depend on existing rsync without -R. In (at least) two ways:
amd-pstate fails because its
TEST_FILEShas includes from other directories and it expects those files to land in the directory with tests.
Quoted 6 lines
x86 prefixes all its output with $(OUTPUT) to form absolute paths, which also creates absolute paths in kselftest-list.txt.
These two are examples, not necessarily the one instances of those patterns. So switching to preserving directories for existing targets like
TEST_FILESseems intractable.
I wonder how many of the targets actually need this behavior (it's intentionally useful to them) vs they grew to depend on it accidentally.
Quoted 3 lines
Plan B is to add a new
TEST_PROGS_RECURSE, analogous to howTEST_INCLUDESextendedTEST_FILESwith optional path preservation. That is not much more complex.
Alternative would be to allow opt-in (diff at the end), I'm personally biased against yet another TEST_* group because it's hard enough already to make sense of the existing ones. Maybe it's better for the test to "declare":
PRESERVE_TEST_DIRS
--- tools/testing/selftests/lib.mk | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
tools/testing/selftests/lib.mk +15 -2
@@ -51,6 +51,12 @@ else CC := $(CROSS_COMPILE)gcc endif # LLVM +ifeq ($(PRESERVE_TEST_DIRS),) +RSYNC_INSTALL_FLAGS=-a --copy-unsafe-links +else +RSYNC_INSTALL_FLAGS=-aR --copy-unsafe-links +endif + ifeq (0,$(MAKELEVEL)) ifeq ($(OUTPUT),) OUTPUT := $(shell pwd)
@@ -150,12 +156,12 @@ endif define INSTALL_SINGLE_RULE $(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH)) - $(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/) + $(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST) $(INSTALL_PATH)/) endef define INSTALL_MODS_RULE $(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH)/$(INSTALL_LIST)) - $(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST)) + $(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST)) endef define INSTALL_RULE
@@ -179,10 +185,17 @@ else endif emit_tests: +ifeq ($(PRESERVE_TEST_DIRS),) for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ echo "$(COLLECTION):$$BASENAME_TEST"; \ done +else + for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ + BASENAME_TEST=$$TEST; \ + echo "$(COLLECTION):$$TEST"; \ + done +endif # define if isn't already. It is undefined in make O= case. ifeq ($(RM),)
Signature · 1 line
2.46.0
Jakub Kicinski
maintainerQuoted 1 line · On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:
PRESERVE_TEST_DIRS
This was meant to be:
PRESERVE_TEST_DIRS := 1
Testing this more, looks like rsync -aR breaks networking tests, too. The net/lib target, specifically, is no longer able to copy out the files outside of tools/testing/selftests (the YAML specs which live in Documentation/).
So unless we can pass some magic flag to rsync to skip leading ../ we'll be stuck in supporting both behaviors forever. In which case maybe TEST_PROGS_RECURSE is indeed better. I was hoping the PRESERVE_TEST_DIRS flag can spread and once it reaches all targets we can make delete the old behavior. If it can't happen its no better than TEST_PROGS_RECURSE... sorry for the noise.
Quoted 6 lines · Jakub Kicinski
Quoted 1 line · On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:
PRESERVE_TEST_DIRSThis was meant to be:
PRESERVE_TEST_DIRS:= 1
Oh I like this.
Quoted 11 lines
Testing this more, looks like rsync -aR breaks networking tests, too. The net/lib target, specifically, is no longer able to copy out the files outside of
tools/testing/selftests(the YAML specs which live inDocumentation/).So unless we can pass some magic flag to rsync to skip leading ../ we'll be stuck in supporting both behaviors forever. In which case maybe
TEST_PROGS_RECURSEis indeed better. I was hoping thePRESERVE_TEST_DIRSflag can spread and once it reaches all targets we can make delete the old behavior. If it can't happen its no better thanTEST_PROGS_RECURSE... sorry for the noise.
Oh right :/
A lot of TEST(_GEN|_CUSTOM)_PROGS(_EXTENDED) could probably be cleaned up.
One part that I haven't looked at closely yet is x86's Makefile prefixing everything with $(OUTPUT). That seems something that should be consistent across target Makefiles if it exists at all, but clearly isn't. Maybe it was when it was introduced in commit a8ba798bc8ec.
So, we could opt out only TEST_FILES. But that still leaves a branch there indefinitely and makes the option harder to understand. Sounds like TEST_PROGS_RECURSE is simpler then, indeed.
Separate from the above, runner.sh also needs a few changes. This part is mostly cosmetic, but there will be duplicate basenames, for common terms such as client.pkt
- TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+ TEST_HDR_MSG="selftests: $DIR: $TEST"
run_in_netns and run_many need similar fixes to avoid name collisions.
Btw, for KSELFTEST_PKT_INTERP it probably makes sense to only try this if the script is non-executable, similar to the existing opt-out that looks at the shebang.
Quoted 17 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in
tools/testing/selftests/Makefile. ItsMakefileonly generates dependencies for other targets:TEST_FILES,TEST_GEN_FILESandTEST_INCLUDES.Oh right,
TEST_FILESvsTEST_INCLUDES:(Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only
TEST_NAMEwith a / in it is
x86_64/nx_huge_pages_test.shBut then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:
Quoted 6 lines
mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt
Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?
Okay, went with the simplest approach for v1: indeed just flattening the space like this.
Only after implementing TEST_PROGS_RECURSE and PRESERVE_TEST_DIRS. Both make kselftest more complex and in subtle ways that could break existing tests.
Also simplified the interpreter mechanism. Instead of requiring an environment variable, look for a specific executable in the TARGET dir. This will make the test just work when someone does a manual run.
All is tentative and up for revision, of course.
Quoted 52 lines · Willem de Bruijn
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g,
source ../common/defaults.sh).Can you show an example of exact commands and what happens?
Running directly works fine:
$ KSELFTEST_PKT_INTERP=packetdrill_ksft.sh $ make -C tools/testing/selftests \ TARGETS=net/packetdrill O=/tmp run_testsTAP version 13 1..3 # timeout set to 45 # selftests: net/packetdrill: client.pkt # TAP version 13 # 1..2 # ok 1 ipv4 # ok 2 ipv6 # # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: client.pkt [..etc..]Installing does not:
$ make -C tools/testing/selftests/ \ TARGETS=net/packetdrill \ install INSTALL_PATH=$INSTALL_DIR $ cd $INSTALL_DIR $ export KSELFTEST_PKT_INTERP=packetdrill_ksft.sh $ ./run_kselftest.sh -c net/packetdrillTAP version 13
1..3
# timeout set to 45
# selftests: net/packetdrill: client.pkt
# TAP version 13
# 1..2
# sh: line 1:../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 1 ipv4
# sh: line 1:../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 2 ipv6
# # Totals: pass:0 fail:2 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: net/packetdrill: client.pkt # exit=1
Quoted 4 lines
Due to that relative path to
defaults.shinside the scripts.It is arguably a bit weird that the relative path of the
TEST_PROGSdiffers before and after install.
I guess what's weird is more what I'm trying to do. There has been no need so far for progs in subdirectories from the TARGET.
One option is to just create a bunch of targets with a wildcard. Something like below (entirely untested).
TARGETS += net/packetdrill/tcp/$(wildcard *)
Quoted 9 lines
Quoted 1 line
We have directories in net/lib, and it's a target, and it works, no?
net/lib is not a TARGET in
tools/testing/selftests/Makefile. ItsMakefileonly generates dependencies for other targets:TEST_FILES,TEST_GEN_FILESandTEST_INCLUDES.This issue with preserving paths until recently also existed for helper files (
TEST_FILES).TEST_INCLUDESwas added expressly to preserve those paths (commit2a0683be5b4c).
Another option is to dive yet deeper into kselftest infra and do the same for TEST_PROGS, either TEST_PROGS itself or as a new TEST_PROGS_FULLPATH or so.
It needs a change to INSTALL_RULE (maybe just passing -R to rsync), to emit_tests to skip basename when writing kselftest-list.txt, and probably to run_one() to break up the path and chdir. I'll see if I can make this work without too much churn.
Hi Willem, Paolo,
Thank you for this series, that would be great to have the CI validating these packetdrill tests!
(Having the Netdev CI validating these packetdrill tests was part of my suggestions for the discussion we will have at NetConf :) )
Quoted 3 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
Adding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)
Thank you for having Cced me! Yes, MPTCP's packetdrill is still not ready for upstream :-/
But we could eventually have a runner on our side doing the validation of the MPTCP packetdrill tests.
Quoted 2 lines
Quoted 2 lines · Willem de Bruijn
Quoted 1 line
From: Willem de Bruijn <[email protected]>
(...)
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill
run_all.pybehavior that uses many concurrent threads).
Quoted 8 lines
Quoted 4 lines
I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime
Quoted 1 line
1h, but that is bad ;)
Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use
run_all.pyrather than reinvent the wheel here.
If you use run_all.py, you might want to merge this PR:
https://github.com/google/packetdrill/pull/83
On MPTCP side, we run packetdrill using:
run_all.py -t ${TAP_DIR} -l -v -P $(($(nproc) * 2)) ${PKTD_DIR}For our case, it is easier to generate the TAP output in a different directory, but I guess that could also be part of the standard output.
Also, if we have to limit the number of tests being executed in parallel with -P, to reduce the instabilities.
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Quoted 243 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in
nf_conntrack, in commita8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.pyResult:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill:tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill:tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill:tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill:tcp_md5.pySigned-off-by Willem de Bruijn <[email protected]>---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
--- tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++ 10 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore create mode 100644 tools/testing/selftests/net/packetdrill/Makefile create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkttools/testing/selftests/Makefile +4
-3@@ -65,10 +65,11 @@ TARGETS += net/af_unix TARGETS += net/forwarding TARGETS += net/hsr TARGETS += net/mptcp -TARGETS += net/openvswitch -TARGETS += net/tcp_ao TARGETS += net/netfilter +TARGETS += net/openvswitch +TARGETS += net/packetdrill TARGETS += net/rds +TARGETS += net/tcp_ao TARGETS += nsfs TARGETS += perf_events TARGETS += pidfd@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug # Networking tests want the net/lib target, include it automatically -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),) +ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),) ifeq ($(filter net/lib,$(TARGETS)),) INSTALL_DEP_TARGETS := net/lib endiftools/testing/selftests/net/packetdrill/.gitignore new file +1
-0@@ -0,0 +1 @@ +tcp*shtools/testing/selftests/net/packetdrill/Makefile new file +28
-0@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 + +# KSFT includes +TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py) + +# Packetdrill support file(s) +TEST_INCLUDES += tcp/common/defaults.sh +TEST_INCLUDES += tcp/common/set_sysctls.py + +# Packetdrill scripts: all .pkt in subdirectories +TEST_INCLUDES += $(wildcard tcp/**/*.pkt) + +# Create a separate ksft test for each subdirectory +# Running all packetdrill tests in one go will take too long +# +# For each tcp/$subdir, create a test script tcp_$subdir.py +# Exclude tcp/common, which is a helper directory +TEST_DIRS := $(wildcard tcp/*) +TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS)) +TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py) + +$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py + cp $< $@ + +# Needed to generate all TEST_CUSTOM_PROGS +all: $(TEST_CUSTOM_PROGS) + +include ../../lib.mktools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15
-0@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +import pathlib +import sys + +KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve() + +try: + sys.path.append(KSFT_DIR.as_posix()) + from net.lib.py import * +except ModuleNotFoundError as e: + ksft_pr("Failed importing `net` library from kernel sources") + ksft_pr(str(e)) + ktap_result(True, comment="SKIP") + sys.exit(4)tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +35
-0@@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +"""Run packetdrill tests in the ksft harness. + + Run all packetdrill tests in a subdirectory. + Detect the relevant subdirectory from this script name. + (Because the script cannot be given arguments.) + + Run each test, for both IPv4 and IPv6. + Return a separate ksft result for each test case. +""" + +import glob +import os +import pathlib +import shutil + +from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS + + +def test_func_builder(pktfile_path, ipv4): + """Create a function that can be passed to ksft_run.""" + + def f(): + if ipv4: + args = ("--ip_version=ipv4 " + "--local_ip=192.168.0.1 " + "--gateway_ip=192.168.0.1 " + "--netmask_ip=255.255.0.0 " + "--remote_ip=192.0.2.1 " + "-D CMSG_LEVEL_IP=SOL_IP " + "-D CMSG_TYPE_RECVERR=IP_RECVERR " + ) + else:
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10
-0+ args = ("--ip_version=ipv6 --mtu=1520 " + "--local_ip=fd3d:0a0b:17d6::1 " + "--gateway_ip=fd3d:0a0b:17d6:8888::1 " + "--remote_ip=fd3d:fa7b:d17d::1 " + "-D CMSG_LEVEL_IP=SOL_IPV6 " + "-D CMSG_TYPE_RECVERR=IPV6_RECVERR" + ) + + if not shutil.which("packetdrill"): + raise KsftSkipEx("Cannot find packetdrill")
I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?
Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5
-0+ + netns = NetNS() + + # Call packetdrill from the directory hosting the .pkt script, + # because scripts can have relative includes.
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10
-0+ savedir = os.getcwd() + os.chdir(os.path.dirname(pktfile_path)) + basename = os.path.basename(pktfile_path) + cmd(f"packetdrill {args} {basename}", ns=netns) + os.chdir(savedir) + + if ipv4: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v4" + else: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"
What about dualstack v4-over-v6 mode? Do we want to support that as well?
Jakub Kicinski
maintainerQuoted 4 lines · On Wed, 28 Aug 2024 08:01:02 -0700 Stanislav Fomichev wrote:
I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?
We already build it from source, for netfilter.
Quoted 251 lines · Stanislav Fomichev
Quoted 250 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in
nf_conntrack, in commita8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.pyResult:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill:tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill:tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill:tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill:tcp_md5.pySigned-off-by Willem de Bruijn <[email protected]>---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
--- tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++ 10 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore create mode 100644 tools/testing/selftests/net/packetdrill/Makefile create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkttools/testing/selftests/Makefile +4
-3@@ -65,10 +65,11 @@ TARGETS += net/af_unix TARGETS += net/forwarding TARGETS += net/hsr TARGETS += net/mptcp -TARGETS += net/openvswitch -TARGETS += net/tcp_ao TARGETS += net/netfilter +TARGETS += net/openvswitch +TARGETS += net/packetdrill TARGETS += net/rds +TARGETS += net/tcp_ao TARGETS += nsfs TARGETS += perf_events TARGETS += pidfd@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug # Networking tests want the net/lib target, include it automatically -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),) +ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),) ifeq ($(filter net/lib,$(TARGETS)),) INSTALL_DEP_TARGETS := net/lib endiftools/testing/selftests/net/packetdrill/.gitignore new file +1
-0@@ -0,0 +1 @@ +tcp*shtools/testing/selftests/net/packetdrill/Makefile new file +28
-0@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 + +# KSFT includes +TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py) + +# Packetdrill support file(s) +TEST_INCLUDES += tcp/common/defaults.sh +TEST_INCLUDES += tcp/common/set_sysctls.py + +# Packetdrill scripts: all .pkt in subdirectories +TEST_INCLUDES += $(wildcard tcp/**/*.pkt) + +# Create a separate ksft test for each subdirectory +# Running all packetdrill tests in one go will take too long +# +# For each tcp/$subdir, create a test script tcp_$subdir.py +# Exclude tcp/common, which is a helper directory +TEST_DIRS := $(wildcard tcp/*) +TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS)) +TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py) + +$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py + cp $< $@ + +# Needed to generate all TEST_CUSTOM_PROGS +all: $(TEST_CUSTOM_PROGS) + +include ../../lib.mktools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15
-0@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +import pathlib +import sys + +KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve() + +try: + sys.path.append(KSFT_DIR.as_posix()) + from net.lib.py import * +except ModuleNotFoundError as e: + ksft_pr("Failed importing `net` library from kernel sources") + ksft_pr(str(e)) + ktap_result(True, comment="SKIP") + sys.exit(4)tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +42
-0@@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +"""Run packetdrill tests in the ksft harness. + + Run all packetdrill tests in a subdirectory. + Detect the relevant subdirectory from this script name. + (Because the script cannot be given arguments.) + + Run each test, for both IPv4 and IPv6. + Return a separate ksft result for each test case. +""" + +import glob +import os +import pathlib +import shutil + +from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS + + +def test_func_builder(pktfile_path, ipv4): + """Create a function that can be passed to ksft_run.""" + + def f(): + if ipv4: + args = ("--ip_version=ipv4 " + "--local_ip=192.168.0.1 " + "--gateway_ip=192.168.0.1 " + "--netmask_ip=255.255.0.0 " + "--remote_ip=192.0.2.1 " + "-D CMSG_LEVEL_IP=SOL_IP " + "-D CMSG_TYPE_RECVERR=IP_RECVERR " + ) + else: + args = ("--ip_version=ipv6 --mtu=1520 " + "--local_ip=fd3d:0a0b:17d6::1 " + "--gateway_ip=fd3d:0a0b:17d6:8888::1 " + "--remote_ip=fd3d:fa7b:d17d::1 " + "-D CMSG_LEVEL_IP=SOL_IPV6 " + "-D CMSG_TYPE_RECVERR=IPV6_RECVERR" + )
Quoted 8 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45
Quoted 3 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +3
-0+ + if not shutil.which("packetdrill"): + raise KsftSkipEx("Cannot find packetdrill")I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?
As Jakub responded.
Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's nf_conntrack charted the path here.
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10
-0+ + netns = NetNS() + + # Call packetdrill from the directory hosting the .pkt script, + # because scripts can have relative includes. + savedir = os.getcwd() + os.chdir(os.path.dirname(pktfile_path)) + basename = os.path.basename(pktfile_path) + cmd(f"packetdrill {args} {basename}", ns=netns) + os.chdir(savedir)
Quoted 8 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60
Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5
-0+ + if ipv4: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v4" + else: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"What about dualstack v4-over-v6 mode? Do we want to support that as well?
Does it give a sufficient signal? I skipped over it on purpose. But can enable if that is the consensus.
Quoted 261 lines · Willem de Bruijn
Quoted 251 lines · Stanislav Fomichev
Quoted 250 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in
nf_conntrack, in commita8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.pyResult:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill:tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill:tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill:tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill:tcp_md5.pySigned-off-by Willem de Bruijn <[email protected]>---
RFC points for discussion
ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep intools/testing/selftests/Makefile
- a boilerplatelib/py/__init__.pythat each user of ksft will need It seems preferable to me to useksft.pyover reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.kselftest-per-directory: copying
packetdrill_ksft.pyto create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.
nf_conntrack: we can dedup thecommon.sh.*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.
--- tools/testing/selftests/Makefile | 7 +- .../selftests/net/packetdrill/.gitignore | 1 + .../selftests/net/packetdrill/Makefile | 28 ++++++ .../net/packetdrill/lib/py/__init__.py | 15 ++++ .../net/packetdrill/packetdrill_ksft.py | 90 +++++++++++++++++++ .../net/packetdrill/tcp/common/defaults.sh | 63 +++++++++++++ .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++ .../net/packetdrill/tcp/inq/client.pkt | 51 +++++++++++ .../net/packetdrill/tcp/inq/server.pkt | 51 +++++++++++ .../tcp/md5/md5-only-on-client-ack.pkt | 28 ++++++ 10 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore create mode 100644 tools/testing/selftests/net/packetdrill/Makefile create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkttools/testing/selftests/Makefile +4
-3@@ -65,10 +65,11 @@ TARGETS += net/af_unix TARGETS += net/forwarding TARGETS += net/hsr TARGETS += net/mptcp -TARGETS += net/openvswitch -TARGETS += net/tcp_ao TARGETS += net/netfilter +TARGETS += net/openvswitch +TARGETS += net/packetdrill TARGETS += net/rds +TARGETS += net/tcp_ao TARGETS += nsfs TARGETS += perf_events TARGETS += pidfd@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug # Networking tests want the net/lib target, include it automatically -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),) +ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),) ifeq ($(filter net/lib,$(TARGETS)),) INSTALL_DEP_TARGETS := net/lib endiftools/testing/selftests/net/packetdrill/.gitignore new file +1
-0@@ -0,0 +1 @@ +tcp*shtools/testing/selftests/net/packetdrill/Makefile new file +28
-0@@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 + +# KSFT includes +TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py) + +# Packetdrill support file(s) +TEST_INCLUDES += tcp/common/defaults.sh +TEST_INCLUDES += tcp/common/set_sysctls.py + +# Packetdrill scripts: all .pkt in subdirectories +TEST_INCLUDES += $(wildcard tcp/**/*.pkt) + +# Create a separate ksft test for each subdirectory +# Running all packetdrill tests in one go will take too long +# +# For each tcp/$subdir, create a test script tcp_$subdir.py +# Exclude tcp/common, which is a helper directory +TEST_DIRS := $(wildcard tcp/*) +TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS)) +TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py) + +$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py + cp $< $@ + +# Needed to generate all TEST_CUSTOM_PROGS +all: $(TEST_CUSTOM_PROGS) + +include ../../lib.mktools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15
-0@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +import pathlib +import sys + +KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve() + +try: + sys.path.append(KSFT_DIR.as_posix()) + from net.lib.py import * +except ModuleNotFoundError as e: + ksft_pr("Failed importing `net` library from kernel sources") + ksft_pr(str(e)) + ktap_result(True, comment="SKIP") + sys.exit(4)tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +42
-0@@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +"""Run packetdrill tests in the ksft harness. + + Run all packetdrill tests in a subdirectory. + Detect the relevant subdirectory from this script name. + (Because the script cannot be given arguments.) + + Run each test, for both IPv4 and IPv6. + Return a separate ksft result for each test case. +""" + +import glob +import os +import pathlib +import shutil + +from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS + + +def test_func_builder(pktfile_path, ipv4): + """Create a function that can be passed to ksft_run.""" + + def f(): + if ipv4: + args = ("--ip_version=ipv4 " + "--local_ip=192.168.0.1 " + "--gateway_ip=192.168.0.1 " + "--netmask_ip=255.255.0.0 " + "--remote_ip=192.0.2.1 " + "-D CMSG_LEVEL_IP=SOL_IP " + "-D CMSG_TYPE_RECVERR=IP_RECVERR " + ) + else: + args = ("--ip_version=ipv6 --mtu=1520 " + "--local_ip=fd3d:0a0b:17d6::1 " + "--gateway_ip=fd3d:0a0b:17d6:8888::1 " + "--remote_ip=fd3d:fa7b:d17d::1 " + "-D CMSG_LEVEL_IP=SOL_IPV6 " + "-D CMSG_TYPE_RECVERR=IPV6_RECVERR" + )Quoted 8 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45
Quoted 3 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +3
-0+ + if not shutil.which("packetdrill"): + raise KsftSkipEx("Cannot find packetdrill")I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?
Quoted 5 lines
As Jakub responded.
Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's
nf_conntrackcharted the path here.
Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10
-0+ + netns = NetNS() + + # Call packetdrill from the directory hosting the .pkt script, + # because scripts can have relative includes. + savedir = os.getcwd() + os.chdir(os.path.dirname(pktfile_path)) + basename = os.path.basename(pktfile_path) + cmd(f"packetdrill {args} {basename}", ns=netns) + os.chdir(savedir)Quoted 8 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60
Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5
-0+ + if ipv4: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v4" + else: + f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"What about dualstack v4-over-v6 mode? Do we want to support that as well?
Quoted 2 lines
Does it give a sufficient signal? I skipped over it on purpose. But can enable if that is the consensus.
Makes sense. Agreed that most of the time v4-over-v6 just exercises the same v4 paths so it's not too much value. I'm assuming that the tests that care about v4-over-v6 peculiarities can just
do --ip_version=ipv4-mapped-ipv6 in their .pkt file (similar to what gtests/net/tcp/mtu_probe/basic-v6.pkt is doing)?
Jakub Kicinski
maintainerQuoted 8 lines · On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
Quoted 5 lines
As Jakub responded.
Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's
nf_conntrackcharted the path here.Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.
FWIW I lied to you, the "script" for buildings things wasn't committed in the repo, I pushed it now:
https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
Quoted 5 lines
As Jakub responded.
Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's
nf_conntrackcharted the path here.
Quoted 6 lines
Quoted 2 lines
Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.
FWIW I lied to you, the "script" for buildings things wasn't committed in the repo, I pushed it now:
https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1
Perfect, thanks!
Quoted 35 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.
Florian recently added support for packetdrill tests in
nf_conntrack, in commita8a388c2aae49("selftests: netfilter: add packetdrill based conntrack tests").This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.
It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script
packetdrill_ksft.pyfor each directory, and putting those inTEST_CUSTOM_PROGSso that kselftests runs each.To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.
Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.
Implementation notes:
- restore alphabetical order when adding the new directory totools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- updatecommon/defaults.shpath (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)
Quoted 8 lines
Tested:
make -Ctools/testing/selftests/\
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py
I did not know about run_kselftest.sh. From a quick look, it seems to require selftests to be installed to run them. I think nipa relies on something slightly different, it does something like this, AFAIU this runs an individual kselftest without requiring it to be installed first (which is nice):
make -C tools/testing/selftests TARGETS="net" TEST_PROGS="rxtimestamp.sh" TEST_GEN_PROGS="" run_tests
It may be worth it to check if these added tests can be ran in a similar way.
This may also solve your issue with 'intermediate output' being printed after every test. AFAIU if the tests can be ran individually using a command like this, then you don't need intermediate output, maybe.
Jakub Kicinski
maintainerQuoted 10 lines · On Wed, 28 Aug 2024 11:26:28 -0700 Mina Almasry wrote:
I did not know about
run_kselftest.sh. From a quick look, it seems to require selftests to be installed to run them. I think nipa relies on something slightly different, it does something like this, AFAIU this runs an individual kselftest without requiring it to be installed first (which is nice):make -C
tools/testing/selftestsTARGETS="net" TEST_PROGS="rxtimestamp.sh" TEST_GEN_PROGS=""run_testsIt may be worth it to check if these added tests can be ran in a similar way.
They should, the ksft infra converges into the same bash script (runner.sh) whether we enter via make or via the "installed" runner.
Quoted 4 lines
This may also solve your issue with 'intermediate output' being printed after every test. AFAIU if the tests can be ran individually using a command like this, then you don't need intermediate output, maybe.
selftest infra tries to prefix raw output with # to make sure that random prints don't get mixed with KTAP. IIRC there are two ways to do that, one is a perl script and that doesn't buffer, but if there's no perl in PATH it runs the output thru sed, and sed buffers.