Skip to contentSkip to conversation

selftests/net: integrate packetdrill with ksft

RFC PATCH In review Willem de Bruijn 6 participants 28 messages Aug 27, 2024 – Sep 5, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 selftests/net: integrate packetdrill with ksft

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

---
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++
 10 files changed, 369 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore
 create mode 100644 tools/testing/selftests/net/packetdrill/Makefile
 create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py
 create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
tools/testing/selftests/Makefile +4 -3
index a5f1c0c27dff9..f03d6fee7ac54 100644
@@ -65,10 +65,11 @@ TARGETS += net/af_unix
 TARGETS += net/forwarding
 TARGETS += net/hsr
 TARGETS += net/mptcp
-TARGETS += net/openvswitch
-TARGETS += net/tcp_ao
 TARGETS += net/netfilter
+TARGETS += net/openvswitch
+TARGETS += net/packetdrill
 TARGETS += net/rds
+TARGETS += net/tcp_ao
 TARGETS += nsfs
 TARGETS += perf_events
 TARGETS += pidfd
@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh
tools/testing/selftests/net/packetdrill/Makefile new file +28 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# KSFT includes
+TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py)
+
+# Packetdrill support file(s)
+TEST_INCLUDES += tcp/common/defaults.sh
+TEST_INCLUDES += tcp/common/set_sysctls.py
+
+# Packetdrill scripts: all .pkt in subdirectories
+TEST_INCLUDES += $(wildcard tcp/**/*.pkt)
+
+# Create a separate ksft test for each subdirectory
+# Running all packetdrill tests in one go will take too long
+#
+# For each tcp/$subdir, create a test script tcp_$subdir.py
+# Exclude tcp/common, which is a helper directory
+TEST_DIRS := $(wildcard tcp/*)
+TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS))
+TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py)
+
+$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py
+	cp $< $@
+
+# Needed to generate all TEST_CUSTOM_PROGS
+all: $(TEST_CUSTOM_PROGS)
+
+include ../../lib.mk
tools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15 -0
new file mode 100644
index 0000000000000..51bb6dda43d65
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import pathlib
+import sys
+
+KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +90 -0

90 lines not on this page · raw message

tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh new file +63 -0

63 lines not on this page · raw message

tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +38 -0
new file mode 100755
index 0000000000000..5ddf456ae973a
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Sets sysctl values and writes a file that restores them.
+
+The arguments are of the form "<proc-file>=<val>" separated by spaces.
+The program first reads the current value of the proc-file and creates
+a shell script named "/tmp/sysctl_restore_${PACKETDRILL_PID}.sh" which
+restores the values when executed. It then sets the new values.
+
+PACKETDRILL_PID is set by packetdrill to the pid of itself, so a .pkt
+file could restore sysctls by running `/tmp/sysctl_restore_${PPID}.sh`
+at the end.
+"""
+
+import os
+import subprocess
+import sys
+
+filename = '/tmp/sysctl_restore_%s.sh' % os.environ['PACKETDRILL_PID']
+
+# Open file for restoring sysctl values
+restore_file = open(filename, 'w')
+print('#!/bin/bash', file=restore_file)
+
+for a in sys.argv[1:]:
+  sysctl = a.split('=')
+  # sysctl[0] contains the proc-file name, sysctl[1] the new value
+
+  # read current value and add restore command to file
+  cur_val = subprocess.check_output(['cat', sysctl[0]], universal_newlines=True)
+  print('echo "%s" > %s' % (cur_val.strip(), sysctl[0]), file=restore_file)
+
+  # set new value
+  cmd = 'echo "%s" > %s' % (sysctl[1], sysctl[0])
+  os.system(cmd)
+
+os.system('chmod u+x %s' % filename)
tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt new file +51 -0

51 lines not on this page · raw message

tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt new file +51 -0

51 lines not on this page · raw message

tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt new file +28 -0
new file mode 100644
index 0000000000000..42b712e14e562
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+// Test what happens when client does not provide MD5 on SYN,
+// but then does on the ACK that completes the three-way handshake.
+
+`../common/defaults.sh`
+
+// Establish a connection.
+    0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+   +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+   +0 bind(3, ..., ...) = 0
+   +0 listen(3, 1) = 0
+
+   +0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 10>
+   +0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+// Ooh, weird: client provides MD5 option on the ACK:
+ +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop>
+ +.01 < . 1:1(0) ack 1 win 514 <md5 000102030405060708090a0b0c0d0e0f,nop,nop>
+
+// The TCP listener refcount should be 2, but on buggy kernels it can be 0:
+   +0 `grep " 0A " /proc/net/tcp /proc/net/tcp6 | grep ":1F90"`
+
+// Now here comes the legit ACK:
+ +.01 < . 1:1(0) ack 1 win 514
+
+// Make sure the connection is OK:
+   +0 accept(3, ..., ...) = 4
+
+ +.01 write(4, ..., 1000) = 1000
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
maintainer
#1 lore raw

Very exciting to see packetdrill tests making their way upstream!

Quoted 9 lines · On Tue, 27 Aug 2024 15:32:29 -0400 Willem de Bruijn wrote:

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

No preference here, the wrapper script uses little of the python code, and use of YNL seems unlikely, so it's a coin toss whether it's worth linking up to net/lib/py or just writing a bit of bash.

Quoted 5 lines

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

Not sure what you mean by intermediate output (the perl wrapper prints immediately, do you have perl?). We do have some nested ktap parsing in nipa, but small tweaks would be necessary to "decapsulate" the first layer completely. My bigger concern would be runtime, if the time it takes to run all tests grows we spawn multiple VMs and load balance the test cases.

All in all, trying to support single script is probably not worth the extra effort.

Quoted 4 lines

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests.

Other than avoiding known flakes - no concerns. Is the distinction between "change detector" vs hard tests (uAPI change / RFC compliance), well defined? Not sure if that's really possible but if so it would be nice to "sort" the tests into different dirs.

Quoted 2 lines

And what
is the best way to eventually send up to 150 files, 7K LoC.

Just send them, slice into a handful (<10) of patches if you can. 7k LoC is same order of magnitude as initial drivers we merge.

To be clear let's start with a small patch like this one. Since this is a new target I'll have to reconfigure the runners. So we'll see how well this works once it's merged. Spinning up new runners is a bit tedious but here new target seems quite justified.

Quoted 10 lines
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++

prolly need a config file to enable kconfig for MD5 ? perils of adding new targets

tools/testing/selftests/net/packetdrill/.gitignore line 1

Quoted 7 lines
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh

Is this right or should it be tcp_*.py ?

tools/testing/selftests/net/packetdrill/Makefile line 1

Quoted 7 lines
tools/testing/selftests/net/packetdrill/Makefile new file +1 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0

Would be nice to add a few lines here with an overview of how the packetdrill tests get executed. People may jump in here to try to add new tests, since that's how ksft usually operates.

tools/testing/selftests/net/packetdrill/Makefile line 5

Quoted 4 lines
tools/testing/selftests/net/packetdrill/Makefile new file +4 -0
+def scriptname_to_testdir(filepath):
+    """Extract the directory to run from this filename."""
+
+    suffix = ".sh"

.sh again ?

tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh line 63

Quoted 7 lines
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh +5 -0
 
+# Override the default qdisc on the tun device.
+# Many tests fail with timing errors if the default
+# is FQ and that paces their flows.
+tc qdisc add dev tun0 root pfifo
+

nit: double new line

tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py

Quoted 5 lines
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +0 -0
new file mode 100755
index 0000000000000..5ddf456ae973a

What calls this one? I can't grep it out.

#2 lore raw
Quoted 35 lines · Jakub Kicinski

Very exciting to see packetdrill tests making their way upstream!

Quoted 9 lines · On Tue, 27 Aug 2024 15:32:29 -0400 Willem de Bruijn wrote:

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

No preference here, the wrapper script uses little of the python code, and use of YNL seems unlikely, so it's a coin toss whether it's worth linking up to net/lib/py or just writing a bit of bash.

Quoted 5 lines

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

Not sure what you mean by intermediate output (the perl wrapper prints immediately, do you have perl?). We do have some nested ktap parsing in nipa, but small tweaks would be necessary to "decapsulate" the first layer completely. My bigger concern would be runtime, if the time it takes to run all tests grows we spawn multiple VMs and load balance the test cases.

All in all, trying to support single script is probably not worth the extra effort.

Quoted 2 lines

nf_conntrack: we can dedup the common.sh.

Quoted 7 lines
Quoted 2 lines

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests.

Other than avoiding known flakes - no concerns. Is the distinction between "change detector" vs hard tests (uAPI change / RFC compliance), well defined? Not sure if that's really possible but if so it would be nice to "sort" the tests into different dirs.

It's not as clear-cut as API/RFC.

The major change detector issues are with timing, as packetdrill scripts have timestamped lines. I think usec TCP timestamps were an issue at some point. On-going is that we increase allowed variance on debug builds. Note to self that that is missing here.

I think that all tests we open sourced to github so far were chosen to be robust. Will double check and err on the side of caution.

Quoted 11 lines
Quoted 2 lines

And what
is the best way to eventually send up to 150 files, 7K LoC.

Just send them, slice into a handful (<10) of patches if you can. 7k LoC is same order of magnitude as initial drivers we merge.

To be clear let's start with a small patch like this one. Since this is a new target I'll have to reconfigure the runners. So we'll see how well this works once it's merged. Spinning up new runners is a bit tedious but here new target seems quite justified.

Ack. Thanks for that guidance.

Quoted 53 lines
Quoted 10 lines
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++

prolly need a config file to enable kconfig for MD5 ? perils of adding new targets

tools/testing/selftests/net/packetdrill/.gitignore line 1

Quoted 7 lines
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh

Is this right or should it be tcp_*.py ?

tools/testing/selftests/net/packetdrill/Makefile line 1

Quoted 7 lines
tools/testing/selftests/net/packetdrill/Makefile new file +1 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0

Would be nice to add a few lines here with an overview of how the packetdrill tests get executed. People may jump in here to try to add new tests, since that's how ksft usually operates.

tools/testing/selftests/net/packetdrill/Makefile line 5

Quoted 4 lines
tools/testing/selftests/net/packetdrill/Makefile new file +4 -0
+def scriptname_to_testdir(filepath):
+    """Extract the directory to run from this filename."""
+
+    suffix = ".sh"

.sh again ?

tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh line 63

Quoted 7 lines
tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh +5 -0
 
+# Override the default qdisc on the tun device.
+# Many tests fail with timing errors if the default
+# is FQ and that paces their flows.
+tc qdisc add dev tun0 root pfifo
+

nit: double new line

Quoted 7 lines

tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py

Quoted 5 lines
tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py new file +0 -0
new file mode 100755
index 0000000000000..5ddf456ae973a

What calls this one? I can't grep it out.

Whoops. I changed test directories and apparently these ones don't rely on this.

Thanks for the review. I'll take a quick stab at a standalone script to see whether that is easier. Else will resubmit with these and Paolo's feedback addressed.

In particular to Paolo's feedback: running multiple tests in parallel like run_all.py can. Come to think of it, maybe I should import run_all.py, modifying it to output KTAP and ksft return codes.

maintainer
#3 lore raw
In reply to Willem de Bruijn

Adding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)

Quoted 80 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack,Addin in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

Quoted 2 lines

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack.

Additionally, in some setups the test directory is RO, avoding file creation there would be better.

What about placing in each subdiretory a trivial wrapper invoking the 'main' packetdrill_ksft.py script specifying as an argument the (sub-)directory to run/process?

Quoted 4 lines

A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).

I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime > 1h, but that is bad ;)

Quoted 5 lines

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

I have no idea WRT the overall test stability, is some specific case/dir is subject to very frequent false positive/false negative we could postpone importing them, but ideally IMHO all the github scripts are good candidates.

Side note: I think it would be great to have some easy command line parameter to run only the specified script/test-case.

Thanks!

Paolo

#4 lore raw
Quoted 87 lines · Paolo Abeni

Adding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)

Quoted 80 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack,Addin in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

Quoted 2 lines

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack.

Quoted 6 lines

Additionally, in some setups the test directory is RO, avoding file creation there would be better.

What about placing in each subdiretory a trivial wrapper invoking the 'main' packetdrill_ksft.py script specifying as an argument the (sub-)directory to run/process?

The files are created by kselftests, similar to any TEST_GEN_XXX or TEST_CUSTOM_PROGS.

If instead we prepopulate in each directory that is duplicative boilerplate committed to git, and we still need to add it to kselftests. Not sure whether TEST_PROGS with paths extending into subdirectories are supported. And we do not want a TARGET for each subdirectory.

It probably can be done, but I don't think that it is needed or simpler.

Quoted 5 lines
Quoted 4 lines

A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

Quoted 8 lines

If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).

I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime

Quoted 1 line

1h, but that is bad ;)

Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use run_all.py rather than reinvent the wheel here.

Quoted 6 lines
Quoted 5 lines

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

Quoted 7 lines

I have no idea WRT the overall test stability, is some specific case/dir is subject to very frequent false positive/false negative we could postpone importing them, but ideally IMHO all the github scripts are good candidates.

Side note: I think it would be great to have some easy command line parameter to run only the specified script/test-case.

Makes sense. Will include.

maintainer
#5 lore raw
Quoted 9 lines · On Wed, 28 Aug 2024 10:03:55 -0400 Willem de Bruijn wrote:
Quoted 9 lines
Quoted 4 lines

A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).

Quoted 8 lines
Quoted 4 lines

I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime

Quoted 1 line

1h, but that is bad ;)

Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use run_all.py rather than reinvent the wheel here.

Do we have to worry about this now? If you're planning to add a runner specifically for packetdrill... IDK if we should. We already have a few runners, not to mention that run_kselftest.sh itself can run all the test cases in parallel in separate network namespaces!

What I was wondering is whether we can use shebang to direct the .pkt files to be "executed" by the python script. Alternatively we can add support to specifying "interpreter" for a given test in ksft infra (kinda like we can pass cmd line arguments to a test). Completely untested but it should give better idea what I mean than a description:

tools/testing/selftests/kselftest/runner.sh +14 -1
index 74954f6a8f94..429c279e9c6e 100644
@@ -56,6 +56,7 @@ run_one()
 	export kselftest_timeout="$kselftest_default_timeout"
 
 	# Safe default if tr not available
+	kselftest_interp_ref="KSELFTEST_INTERP"
 	kselftest_cmd_args_ref="KSELFTEST_ARGS"
 
 	# Optional arguments for this command, possibly defined as an
@@ -78,6 +79,14 @@ run_one()
 					$TR_CMD [:lower:] [:upper:])
 		kselftest_cmd_args_ref="KSELFTEST_${BASENAME_SANITIZED}_ARGS"
 	fi
+	# Optional interpreter to run the test case
+	if [ -n "$TR_CMD" ]; then
+		SUFFIX_SANITIZED=$(echo "${BASENAME_TEST#*.}" | \
+				       $TR_CMD -d "[:blank:][:cntrl:]" | \
+				       $TR_CMD -c "[:alnum:]_" "_" | \
+				       $TR_CMD [:lower:] [:upper:])
+		kselftest_interp_ref="KSELFTEST_${SUFFIX_SANITIZED}_INTERP"
+	fi
 
 	# Load per-test-directory kselftest "settings" file.
 	settings="$BASE_DIR/$DIR/settings"
@@ -110,8 +119,12 @@ run_one()
 		if [ -x /usr/bin/stdbuf ]; then
 			stdbuf="/usr/bin/stdbuf --output=L "
 		fi
+		eval kselftest_interp="\$${kselftest_interp_ref:-}"
+		# Add a space at the end if interpreter is set to work in $cmd
+		[ -n "$kselftest_interp" ] && \
+		    kselftest_interp="$kselftest_interp "
 		eval kselftest_cmd_args="\$${kselftest_cmd_args_ref:-}"
-		cmd="$stdbuf ./$BASENAME_TEST $kselftest_cmd_args"
+		cmd="$stdbuf ./$kselftest_interp$BASENAME_TEST $kselftest_cmd_args"
 		if [ ! -x "$TEST" ]; then
 			echo "# Warning: file $TEST is not executable"
 
#6 lore raw
Quoted 60 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 28 Aug 2024 10:03:55 -0400 Willem de Bruijn wrote:
Quoted 9 lines
Quoted 4 lines

A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).

Quoted 8 lines
Quoted 4 lines

I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime

Quoted 1 line

1h, but that is bad ;)

Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use run_all.py rather than reinvent the wheel here.

Do we have to worry about this now? If you're planning to add a runner specifically for packetdrill... IDK if we should. We already have a few runners, not to mention that run_kselftest.sh itself can run all the test cases in parallel in separate network namespaces!

What I was wondering is whether we can use shebang to direct the .pkt files to be "executed" by the python script. Alternatively we can add support to specifying "interpreter" for a given test in ksft infra (kinda like we can pass cmd line arguments to a test). Completely untested but it should give better idea what I mean than a description:

tools/testing/selftests/kselftest/runner.sh +9 -0
index 74954f6a8f94..429c279e9c6e 100644
@@ -56,6 +56,7 @@ run_one()
 	export kselftest_timeout="$kselftest_default_timeout"
 
 	# Safe default if tr not available
+	kselftest_interp_ref="KSELFTEST_INTERP"
 	kselftest_cmd_args_ref="KSELFTEST_ARGS"
 
 	# Optional arguments for this command, possibly defined as an
@@ -78,6 +79,14 @@ run_one()
 					$TR_CMD [:lower:] [:upper:])
 		kselftest_cmd_args_ref="KSELFTEST_${BASENAME_SANITIZED}_ARGS"
 	fi
+	# Optional interpreter to run the test case
+	if [ -n "$TR_CMD" ]; then
+		SUFFIX_SANITIZED=$(echo "${BASENAME_TEST#*.}" | \
+				       $TR_CMD -d "[:blank:][:cntrl:]" | \
+				       $TR_CMD -c "[:alnum:]_" "_" | \
+				       $TR_CMD [:lower:] [:upper:])
+		kselftest_interp_ref="KSELFTEST_${SUFFIX_SANITIZED}_INTERP"
+	fi
 
 	# Load per-test-directory kselftest "settings" file.
 	settings="$BASE_DIR/$DIR/settings"
@@ -110,8 +119,12 @@ run_one()
 		if [ -x /usr/bin/stdbuf ]; then
 			stdbuf="/usr/bin/stdbuf --output=L "

tools/testing/selftests/kselftest/runner.sh run_one() · line 129

Quoted 10 lines
tools/testing/selftests/kselftest/runner.sh +5 -1
 		fi
+		eval kselftest_interp="\$${kselftest_interp_ref:-}"
+		# Add a space at the end if interpreter is set to work in $cmd
+		[ -n "$kselftest_interp" ] && \
+		    kselftest_interp="$kselftest_interp "
 		eval kselftest_cmd_args="\$${kselftest_cmd_args_ref:-}"
-		cmd="$stdbuf ./$BASENAME_TEST $kselftest_cmd_args"
+		cmd="$stdbuf ./$kselftest_interp$BASENAME_TEST $kselftest_cmd_args"
 		if [ ! -x "$TEST" ]; then
 			echo "# Warning: file $TEST is not executable"

That could work.

Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?

Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file

That can be addressed by instead calling a small wrapper shell script.

That would do the test_func_builder part of packetdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.

maintainer
#7 lore raw
Quoted 4 lines · On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:

That could work.

Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?

To me "neither", hopefully and unhelpfully I won't tell you anything that's not common sense :) The balance is between:

- having test cases which don't take too long (assuming debug kernel):
<15min is good,
>1h I will start complaining,
>1h30m is bad because we can't retry and still make the 3h
deadline that NIPA has,
>3h the test can't run in NIPA at all.

vs

- flip side is having so many cases it's hard to keep track and render in the UI. JSON is relatively slow to process. If you have 150 cases, that's 300 per branch (debug and non-debug kernels), times 8 branches a day => 2.4k results / day
I think that's still fine-ish, but on the larger side for sure. For reference net and forwarding have ~100 tests each. FWIW we do have the ability to collect and display nested KTAP so the information is not lost (but it makes queries slower so we don't fetch it by default).

Quoted 10 lines

Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file

That can be addressed by instead calling a small wrapper shell script.

That would do the test_func_builder part of packetdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.

Right!

#8 lore raw
Quoted 32 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:

That could work.

Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?

To me "neither", hopefully and unhelpfully I won't tell you anything that's not common sense :) The balance is between:

- having test cases which don't take too long (assuming debug kernel):
<15min is good,
>1h I will start complaining,
>1h30m is bad because we can't retry and still make the 3h
deadline that NIPA has,
>3h the test can't run in NIPA at all.

vs

- flip side is having so many cases it's hard to keep track and render in the UI. JSON is relatively slow to process. If you have 150 cases, that's 300 per branch (debug and non-debug kernels), times 8 branches a day => 2.4k results / day
I think that's still fine-ish, but on the larger side for sure. For reference net and forwarding have ~100 tests each. FWIW we do have the ability to collect and display nested KTAP so the information is not lost (but it makes queries slower so we don't fetch it by default).

Quoted 5 lines

Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file

Quoted 7 lines
Quoted 5 lines

That can be addressed by instead calling a small wrapper shell script.

That would do the test_func_builder part of packetdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.

Right!

Implemented this, but hit a snag

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, source ../common/defaults.sh).

The output of having each test separate is also quite unreadable.

An alternative is to add each subdirectory as a separate TARGET. But that seems similarly impractical. And does nothing to group KTAP output.

In practice, most directories have a handful of .pkt scripts, and each script is fast, so serialization is not a huge issue.

I found tools/testing/selftests/net/kselftest/ktap_helpers.sh, which has helpful KTAP boilerplate for shell tests, including KSFT exit codes. Am using that instead of python, to reduce the external dependencies.

maintainer
#9 lore raw
Quoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, source ../common/defaults.sh).

Can you show an example of exact commands and what happens?

We have directories in net/lib, and it's a target, and it works, no?

#10 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, source ../common/defaults.sh).

Can you show an example of exact commands and what happens?

Running directly works fine:

    $ KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
    $ make -C tools/testing/selftests \
            TARGETS=net/packetdrill O=/tmp run_tests
    TAP version 13
    1..3
    # timeout set to 45
    # selftests: net/packetdrill: client.pkt
    # TAP version 13
    # 1..2
    # ok 1 ipv4
    # ok 2 ipv6
    # # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
    ok 1 selftests: net/packetdrill: client.pkt
    [..etc..]

Installing does not:

    $ make -C tools/testing/selftests/ \
          TARGETS=net/packetdrill \
          install INSTALL_PATH=$INSTALL_DIR
    $ cd $INSTALL_DIR
    $ export KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
    $ ./run_kselftest.sh -c net/packetdrill

TAP version 13
1..3
# timeout set to 45
# selftests: net/packetdrill: client.pkt
# TAP version 13
# 1..2
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 1 ipv4
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 2 ipv6
# # Totals: pass:0 fail:2 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: net/packetdrill: client.pkt # exit=1

Due to that relative path to defaults.sh inside the scripts.

It is arguably a bit weird that the relative path of the TEST_PROGS differs before and after install.

Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

This issue with preserving paths until recently also existed for helper files (TEST_FILES). TEST_INCLUDES was added expressly to preserve those paths (commit 2a0683be5b4c).

maintainer
#11 lore raw
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

Oh right, TEST_FILES vs TEST_INCLUDES :(

Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only TEST_NAME with a / in it is

x86_64/nx_huge_pages_test.sh

But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:

mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt

Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?

#12 lore raw
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

Quoted 6 lines

Oh right, TEST_FILES vs TEST_INCLUDES :(

Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only TEST_NAME with a / in it is

x86_64/nx_huge_pages_test.sh

Oh interesting precedent. Let me take a look.

Quoted 9 lines

But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:

mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt

Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?

That's definitely simpler :)

I'd like to keep diffs between packetdrill scripts on github (and Google internal, we have more) and selftests to a minimum. This is invertible, as is rewriting source statements inside the pkt files. But that might be more work and more maintenance in the end.

#13 lore raw
Quoted 27 lines · Willem de Bruijn
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

Quoted 6 lines

Oh right, TEST_FILES vs TEST_INCLUDES :(

Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only TEST_NAME with a / in it is

x86_64/nx_huge_pages_test.sh

Oh interesting precedent. Let me take a look.

Quoted 9 lines

But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:

mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt

Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?

Quoted 6 lines

That's definitely simpler :)

I'd like to keep diffs between packetdrill scripts on github (and Google internal, we have more) and selftests to a minimum. This is invertible, as is rewriting source statements inside the pkt files. But that might be more work and more maintenance in the end.

Thanks again for the pointer and suggestion.

Changing kselftests to preserve directories turns out to be trivial. Patch inline below.

But, existing TARGETS of course then start failing. Because they depend on existing rsync without -R. In (at least) two ways:

amd-pstate fails because its TEST_FILES has includes from other directories and it expects those files to land in the directory with tests.

x86 prefixes all its output with $(OUTPUT) to form absolute paths, which also creates absolute paths in kselftest-list.txt.

These two are examples, not necessarily the one instances of those patterns. So switching to preserving directories for existing targets like TEST_FILES seems intractable.

Plan B is to add a new TEST_PROGS_RECURSE, analogous to how TEST_INCLUDES extended TEST_FILES with optional path preservation. That is not much more complex.

---

+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -101,7 +112,7 @@ run_one()
echo "# timeout set to $kselftest_timeout" >> "$logfile"
fi

- TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+ TEST_HDR_MSG="selftests: $DIR: $TEST"

+++ b/tools/testing/selftests/lib.mk
@@ -150,7 +150,7 @@ clean_mods_dir:

define INSTALL_SINGLE_RULE

        $(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH))

- $(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
+ $(if $(INSTALL_LIST),rsync -aR --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
endef

define INSTALL_MODS_RULE
@@ -180,8 +180,7 @@ endif

emit_tests:

        for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
-               BASENAME_TEST=`basename $$TEST`;        \
-               echo "$(COLLECTION):$$BASENAME_TEST";   \
+               echo "$(COLLECTION):$$TEST";    \

done

maintainer
#14 lore raw
Quoted 10 lines · On Sun, 01 Sep 2024 17:15:38 -0400 Willem de Bruijn wrote:

Changing kselftests to preserve directories turns out to be trivial. Patch inline below.

But, existing TARGETS of course then start failing. Because they depend on existing rsync without -R. In (at least) two ways:

amd-pstate fails because its TEST_FILES has includes from other directories and it expects those files to land in the directory with tests.

Quoted 6 lines

x86 prefixes all its output with $(OUTPUT) to form absolute paths, which also creates absolute paths in kselftest-list.txt.

These two are examples, not necessarily the one instances of those patterns. So switching to preserving directories for existing targets like TEST_FILES seems intractable.

I wonder how many of the targets actually need this behavior (it's intentionally useful to them) vs they grew to depend on it accidentally.

Quoted 3 lines

Plan B is to add a new TEST_PROGS_RECURSE, analogous to how TEST_INCLUDES extended TEST_FILES with optional path preservation. That is not much more complex.

Alternative would be to allow opt-in (diff at the end), I'm personally biased against yet another TEST_* group because it's hard enough already to make sense of the existing ones. Maybe it's better for the test to "declare":

PRESERVE_TEST_DIRS

---
 tools/testing/selftests/lib.mk | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
tools/testing/selftests/lib.mk +15 -2
index d6edcfcb5be8..7be9bd583642 100644
@@ -51,6 +51,12 @@ else
 CC := $(CROSS_COMPILE)gcc
 endif # LLVM
 
+ifeq ($(PRESERVE_TEST_DIRS),)
+RSYNC_INSTALL_FLAGS=-a --copy-unsafe-links
+else
+RSYNC_INSTALL_FLAGS=-aR --copy-unsafe-links
+endif
+
 ifeq (0,$(MAKELEVEL))
     ifeq ($(OUTPUT),)
 	OUTPUT := $(shell pwd)
@@ -150,12 +156,12 @@ endif
 
 define INSTALL_SINGLE_RULE
 	$(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH))
-	$(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST) $(INSTALL_PATH)/)
+	$(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST) $(INSTALL_PATH)/)
 endef
 
 define INSTALL_MODS_RULE
 	$(if $(INSTALL_LIST),@mkdir -p $(INSTALL_PATH)/$(INSTALL_LIST))
-	$(if $(INSTALL_LIST),rsync -a --copy-unsafe-links $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST))
+	$(if $(INSTALL_LIST),rsync $(RSYNC_INSTALL_FLAGS) $(INSTALL_LIST)/*.ko $(INSTALL_PATH)/$(INSTALL_LIST))
 endef
 
 define INSTALL_RULE
@@ -179,10 +185,17 @@ else
 endif
 
 emit_tests:
+ifeq ($(PRESERVE_TEST_DIRS),)
 	for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
 		BASENAME_TEST=`basename $$TEST`;	\
 		echo "$(COLLECTION):$$BASENAME_TEST";	\
 	done
+else
+	for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
+		BASENAME_TEST=$$TEST;	\
+		echo "$(COLLECTION):$$TEST";	\
+	done
+endif
 
 # define if isn't already. It is undefined in make O= case.
 ifeq ($(RM),)
Signature · 1 line
2.46.0
maintainer
#15 lore raw
Quoted 1 line · On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:

PRESERVE_TEST_DIRS

This was meant to be:

PRESERVE_TEST_DIRS := 1

Testing this more, looks like rsync -aR breaks networking tests, too. The net/lib target, specifically, is no longer able to copy out the files outside of tools/testing/selftests (the YAML specs which live in Documentation/).

So unless we can pass some magic flag to rsync to skip leading ../ we'll be stuck in supporting both behaviors forever. In which case maybe TEST_PROGS_RECURSE is indeed better. I was hoping the PRESERVE_TEST_DIRS flag can spread and once it reaches all targets we can make delete the old behavior. If it can't happen its no better than TEST_PROGS_RECURSE... sorry for the noise.

#16 lore raw
Quoted 6 lines · Jakub Kicinski
Quoted 1 line · On Mon, 2 Sep 2024 09:46:12 -0700 Jakub Kicinski wrote:

PRESERVE_TEST_DIRS

This was meant to be:

PRESERVE_TEST_DIRS := 1

Oh I like this.

Quoted 11 lines

Testing this more, looks like rsync -aR breaks networking tests, too. The net/lib target, specifically, is no longer able to copy out the files outside of tools/testing/selftests (the YAML specs which live in Documentation/).

So unless we can pass some magic flag to rsync to skip leading ../ we'll be stuck in supporting both behaviors forever. In which case maybe TEST_PROGS_RECURSE is indeed better. I was hoping the PRESERVE_TEST_DIRS flag can spread and once it reaches all targets we can make delete the old behavior. If it can't happen its no better than TEST_PROGS_RECURSE... sorry for the noise.

Oh right :/

A lot of TEST(_GEN|_CUSTOM)_PROGS(_EXTENDED) could probably be cleaned up.

One part that I haven't looked at closely yet is x86's Makefile prefixing everything with $(OUTPUT). That seems something that should be consistent across target Makefiles if it exists at all, but clearly isn't. Maybe it was when it was introduced in commit a8ba798bc8ec.

So, we could opt out only TEST_FILES. But that still leaves a branch there indefinitely and makes the option harder to understand. Sounds like TEST_PROGS_RECURSE is simpler then, indeed.

Separate from the above, runner.sh also needs a few changes. This part is mostly cosmetic, but there will be duplicate basenames, for common terms such as client.pkt

- TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
+ TEST_HDR_MSG="selftests: $DIR: $TEST"

run_in_netns and run_many need similar fixes to avoid name collisions.

Btw, for KSELFTEST_PKT_INTERP it probably makes sense to only try this if the script is non-executable, similar to the existing opt-out that looks at the shebang.

#17 lore raw
In reply to Jakub Kicinski
Quoted 17 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 14:47:43 -0400 Willem de Bruijn wrote:
Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

Oh right, TEST_FILES vs TEST_INCLUDES :(

Looks like only x86 does some weird stuff and prepends $(OUTPUT) to all test names. Otherwise the only TEST_NAME with a / in it is

x86_64/nx_huge_pages_test.sh

But then again maybe we should give up on the idea of using directories? Use some separator like --, I mean:

Quoted 6 lines

mv packetdrill/tcp/inq/client.pkt packetdrill/tcp--inq--client.pkt

Assuming we're moving forward with the interpreter idea we don't need directories for multi-threading, just for organization. Which perhaps isn't worth the time investment? Given that we'd mostly interact with these tests via UI which will flatten it all back?

Okay, went with the simplest approach for v1: indeed just flattening the space like this.

Only after implementing TEST_PROGS_RECURSE and PRESERVE_TEST_DIRS. Both make kselftest more complex and in subtle ways that could break existing tests.

Also simplified the interpreter mechanism. Instead of requiring an environment variable, look for a specific executable in the TARGET dir. This will make the test just work when someone does a manual run.

All is tentative and up for revision, of course.

#18 lore raw
In reply to Willem de Bruijn
Quoted 52 lines · Willem de Bruijn
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 30 Aug 2024 11:20:05 -0400 Willem de Bruijn wrote:

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, source ../common/defaults.sh).

Can you show an example of exact commands and what happens?

Running directly works fine:

    $ KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
    $ make -C tools/testing/selftests \
            TARGETS=net/packetdrill O=/tmp run_tests
    TAP version 13
    1..3
    # timeout set to 45
    # selftests: net/packetdrill: client.pkt
    # TAP version 13
    # 1..2
    # ok 1 ipv4
    # ok 2 ipv6
    # # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
    ok 1 selftests: net/packetdrill: client.pkt
    [..etc..]

Installing does not:

    $ make -C tools/testing/selftests/ \
          TARGETS=net/packetdrill \
          install INSTALL_PATH=$INSTALL_DIR
    $ cd $INSTALL_DIR
    $ export KSELFTEST_PKT_INTERP=packetdrill_ksft.sh
    $ ./run_kselftest.sh -c net/packetdrill

TAP version 13
1..3
# timeout set to 45
# selftests: net/packetdrill: client.pkt
# TAP version 13
# 1..2
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 1 ipv4
# sh: line 1: ../common/defaults.sh: No such file or directory # ./client.pkt: error executing init command: non-zero status 127 # not ok 2 ipv6
# # Totals: pass:0 fail:2 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: net/packetdrill: client.pkt # exit=1

Quoted 4 lines

Due to that relative path to defaults.sh inside the scripts.

It is arguably a bit weird that the relative path of the TEST_PROGS differs before and after install.

I guess what's weird is more what I'm trying to do. There has been no need so far for progs in subdirectories from the TARGET.

One option is to just create a bunch of targets with a wildcard. Something like below (entirely untested).

TARGETS += net/packetdrill/tcp/$(wildcard *)

Quoted 9 lines
Quoted 1 line

We have directories in net/lib, and it's a target, and it works, no?

net/lib is not a TARGET in tools/testing/selftests/Makefile. Its Makefile only generates dependencies for other targets: TEST_FILES, TEST_GEN_FILES and TEST_INCLUDES.

This issue with preserving paths until recently also existed for helper files (TEST_FILES). TEST_INCLUDES was added expressly to preserve those paths (commit 2a0683be5b4c).

Another option is to dive yet deeper into kselftest infra and do the same for TEST_PROGS, either TEST_PROGS itself or as a new TEST_PROGS_FULLPATH or so.

It needs a change to INSTALL_RULE (maybe just passing -R to rsync), to emit_tests to skip basename when writing kselftest-list.txt, and probably to run_one() to break up the path and chdir. I'll see if I can make this work without too much churn.

#19 lore raw
In reply to Willem de Bruijn

Hi Willem, Paolo,

Thank you for this series, that would be great to have the CI validating these packetdrill tests!

(Having the Netdev CI validating these packetdrill tests was part of my suggestions for the discussion we will have at NetConf :) )

Quoted 3 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni

Adding Mat(s) for awareness, it would be great (but difficult) to have mptcp too in the long run ;)

Thank you for having Cced me! Yes, MPTCP's packetdrill is still not ready for upstream :-/

But we could eventually have a runner on our side doing the validation of the MPTCP packetdrill tests.

Quoted 2 lines
Quoted 2 lines · Willem de Bruijn
Quoted 1 line

From: Willem de Bruijn <[email protected]>

(...)

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

A single script is much
simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

If I read correctly, this runs the scripts in the given directory sequentially (as opposed to the default pktdrill run_all.py behavior that uses many concurrent threads).

Quoted 8 lines
Quoted 4 lines

I guess/fear that running all the pktdrill tests in a single batch would take quite a long time, which in turn could be not so good for CI integration. Currently there are a couple of CI test-cases with runtime

Quoted 1 line

1h, but that is bad ;)

Very good point, thanks! This is the third packetdrill runner that I'm writing. I should know this by now.. Let me see whether I can use run_all.py rather than reinvent the wheel here.

If you use run_all.py, you might want to merge this PR:

https://github.com/google/packetdrill/pull/83

On MPTCP side, we run packetdrill using:

  run_all.py -t ${TAP_DIR} -l -v -P $(($(nproc) * 2)) ${PKTD_DIR}

For our case, it is easier to generate the TAP output in a different directory, but I guess that could also be part of the standard output.

Also, if we have to limit the number of tests being executed in parallel with -P, to reduce the instabilities.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#20 lore raw
In reply to Willem de Bruijn
Quoted 243 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

---
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++
 10 files changed, 369 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore
 create mode 100644 tools/testing/selftests/net/packetdrill/Makefile
 create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py
 create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
tools/testing/selftests/Makefile +4 -3
index a5f1c0c27dff9..f03d6fee7ac54 100644
@@ -65,10 +65,11 @@ TARGETS += net/af_unix
 TARGETS += net/forwarding
 TARGETS += net/hsr
 TARGETS += net/mptcp
-TARGETS += net/openvswitch
-TARGETS += net/tcp_ao
 TARGETS += net/netfilter
+TARGETS += net/openvswitch
+TARGETS += net/packetdrill
 TARGETS += net/rds
+TARGETS += net/tcp_ao
 TARGETS += nsfs
 TARGETS += perf_events
 TARGETS += pidfd
@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh
tools/testing/selftests/net/packetdrill/Makefile new file +28 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# KSFT includes
+TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py)
+
+# Packetdrill support file(s)
+TEST_INCLUDES += tcp/common/defaults.sh
+TEST_INCLUDES += tcp/common/set_sysctls.py
+
+# Packetdrill scripts: all .pkt in subdirectories
+TEST_INCLUDES += $(wildcard tcp/**/*.pkt)
+
+# Create a separate ksft test for each subdirectory
+# Running all packetdrill tests in one go will take too long
+#
+# For each tcp/$subdir, create a test script tcp_$subdir.py
+# Exclude tcp/common, which is a helper directory
+TEST_DIRS := $(wildcard tcp/*)
+TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS))
+TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py)
+
+$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py
+	cp $< $@
+
+# Needed to generate all TEST_CUSTOM_PROGS
+all: $(TEST_CUSTOM_PROGS)
+
+include ../../lib.mk
tools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15 -0
new file mode 100644
index 0000000000000..51bb6dda43d65
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import pathlib
+import sys
+
+KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +35 -0
new file mode 100755
index 0000000000000..62572a5b8331c
@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Run packetdrill tests in the ksft harness.
+
+   Run all packetdrill tests in a subdirectory.
+   Detect the relevant subdirectory from this script name.
+   (Because the script cannot be given arguments.)
+
+   Run each test, for both IPv4 and IPv6.
+   Return a separate ksft result for each test case.
+"""
+
+import glob
+import os
+import pathlib
+import shutil
+
+from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS
+
+
+def test_func_builder(pktfile_path, ipv4):
+    """Create a function that can be passed to ksft_run."""
+
+    def f():
+        if ipv4:
+            args = ("--ip_version=ipv4 "
+                    "--local_ip=192.168.0.1 "
+                    "--gateway_ip=192.168.0.1 "
+                    "--netmask_ip=255.255.0.0 "
+                    "--remote_ip=192.0.2.1 "
+                    "-D CMSG_LEVEL_IP=SOL_IP "
+                    "-D CMSG_TYPE_RECVERR=IP_RECVERR "
+                    )
+        else:

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45

Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10 -0
+            args = ("--ip_version=ipv6 --mtu=1520 "
+                    "--local_ip=fd3d:0a0b:17d6::1 "
+                    "--gateway_ip=fd3d:0a0b:17d6:8888::1 "
+                    "--remote_ip=fd3d:fa7b:d17d::1 "
+                    "-D CMSG_LEVEL_IP=SOL_IPV6 "
+                    "-D CMSG_TYPE_RECVERR=IPV6_RECVERR"
+                    )
+
+        if not shutil.which("packetdrill"):
+            raise KsftSkipEx("Cannot find packetdrill")

I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?

Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5 -0
+
+        netns = NetNS()
+
+        # Call packetdrill from the directory hosting the .pkt script,
+        # because scripts can have relative includes.

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60

Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10 -0
+        savedir = os.getcwd()
+        os.chdir(os.path.dirname(pktfile_path))
+        basename = os.path.basename(pktfile_path)
+        cmd(f"packetdrill {args} {basename}", ns=netns)
+        os.chdir(savedir)
+
+    if ipv4:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v4"
+    else:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"

What about dualstack v4-over-v6 mode? Do we want to support that as well?

maintainer
#21 lore raw
Quoted 4 lines · On Wed, 28 Aug 2024 08:01:02 -0700 Stanislav Fomichev wrote:

I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?

We already build it from source, for netfilter.

#22 lore raw
In reply to Stanislav Fomichev
Quoted 251 lines · Stanislav Fomichev
Quoted 250 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

---
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++
 10 files changed, 369 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore
 create mode 100644 tools/testing/selftests/net/packetdrill/Makefile
 create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py
 create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
tools/testing/selftests/Makefile +4 -3
index a5f1c0c27dff9..f03d6fee7ac54 100644
@@ -65,10 +65,11 @@ TARGETS += net/af_unix
 TARGETS += net/forwarding
 TARGETS += net/hsr
 TARGETS += net/mptcp
-TARGETS += net/openvswitch
-TARGETS += net/tcp_ao
 TARGETS += net/netfilter
+TARGETS += net/openvswitch
+TARGETS += net/packetdrill
 TARGETS += net/rds
+TARGETS += net/tcp_ao
 TARGETS += nsfs
 TARGETS += perf_events
 TARGETS += pidfd
@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh
tools/testing/selftests/net/packetdrill/Makefile new file +28 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# KSFT includes
+TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py)
+
+# Packetdrill support file(s)
+TEST_INCLUDES += tcp/common/defaults.sh
+TEST_INCLUDES += tcp/common/set_sysctls.py
+
+# Packetdrill scripts: all .pkt in subdirectories
+TEST_INCLUDES += $(wildcard tcp/**/*.pkt)
+
+# Create a separate ksft test for each subdirectory
+# Running all packetdrill tests in one go will take too long
+#
+# For each tcp/$subdir, create a test script tcp_$subdir.py
+# Exclude tcp/common, which is a helper directory
+TEST_DIRS := $(wildcard tcp/*)
+TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS))
+TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py)
+
+$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py
+	cp $< $@
+
+# Needed to generate all TEST_CUSTOM_PROGS
+all: $(TEST_CUSTOM_PROGS)
+
+include ../../lib.mk
tools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15 -0
new file mode 100644
index 0000000000000..51bb6dda43d65
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import pathlib
+import sys
+
+KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +42 -0
new file mode 100755
index 0000000000000..62572a5b8331c
@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Run packetdrill tests in the ksft harness.
+
+   Run all packetdrill tests in a subdirectory.
+   Detect the relevant subdirectory from this script name.
+   (Because the script cannot be given arguments.)
+
+   Run each test, for both IPv4 and IPv6.
+   Return a separate ksft result for each test case.
+"""
+
+import glob
+import os
+import pathlib
+import shutil
+
+from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS
+
+
+def test_func_builder(pktfile_path, ipv4):
+    """Create a function that can be passed to ksft_run."""
+
+    def f():
+        if ipv4:
+            args = ("--ip_version=ipv4 "
+                    "--local_ip=192.168.0.1 "
+                    "--gateway_ip=192.168.0.1 "
+                    "--netmask_ip=255.255.0.0 "
+                    "--remote_ip=192.0.2.1 "
+                    "-D CMSG_LEVEL_IP=SOL_IP "
+                    "-D CMSG_TYPE_RECVERR=IP_RECVERR "
+                    )
+        else:
+            args = ("--ip_version=ipv6 --mtu=1520 "
+                    "--local_ip=fd3d:0a0b:17d6::1 "
+                    "--gateway_ip=fd3d:0a0b:17d6:8888::1 "
+                    "--remote_ip=fd3d:fa7b:d17d::1 "
+                    "-D CMSG_LEVEL_IP=SOL_IPV6 "
+                    "-D CMSG_TYPE_RECVERR=IPV6_RECVERR"
+                    )
Quoted 8 lines

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45

Quoted 3 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +3 -0
+
+        if not shutil.which("packetdrill"):
+            raise KsftSkipEx("Cannot find packetdrill")

I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?

As Jakub responded.

Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's nf_conntrack charted the path here.

Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10 -0
+
+        netns = NetNS()
+
+        # Call packetdrill from the directory hosting the .pkt script,
+        # because scripts can have relative includes.
+        savedir = os.getcwd()
+        os.chdir(os.path.dirname(pktfile_path))
+        basename = os.path.basename(pktfile_path)
+        cmd(f"packetdrill {args} {basename}", ns=netns)
+        os.chdir(savedir)
Quoted 8 lines

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60

Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5 -0
+
+    if ipv4:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v4"
+    else:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"

What about dualstack v4-over-v6 mode? Do we want to support that as well?

Does it give a sufficient signal? I skipped over it on purpose. But can enable if that is the consensus.

#23 lore raw
Quoted 261 lines · Willem de Bruijn
Quoted 251 lines · Stanislav Fomichev
Quoted 250 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

Result:
kselftest: Running tests in net/packetdrill TAP version 13
1..2
# timeout set to 45
# selftests: net/packetdrill: tcp_inq.py
# KTAP version 1
# 1..4
# ok 1 tcp_inq.client-v4
# ok 2 tcp_inq.client-v6
# ok 3 tcp_inq.server-v4
# ok 4 tcp_inq.server-v6
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: net/packetdrill: tcp_inq.py
# timeout set to 45
# selftests: net/packetdrill: tcp_md5.py
# KTAP version 1
# 1..2
# ok 1 tcp_md5.md5-only-on-client-ack-v4
# ok 2 tcp_md5.md5-only-on-client-ack-v6
# # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 2 selftests: net/packetdrill: tcp_md5.py

---

RFC points for discussion

ksft: the choice for this python framework introduces a dependency on the YNL scripts, and some non-obvious code:
- to include the net/lib dep in tools/testing/selftests/Makefile
- a boilerplate lib/py/__init__.py that each user of ksft will need It seems preferable to me to use ksft.py over reinventing the wheel, e.g., to print KTAP output. But perhaps we can make it more obvious for future ksft users, and make the dependency on YNL optional.

kselftest-per-directory: copying packetdrill_ksft.py to create a separate script per dir is a bit of a hack. A single script is much simpler, optionally with nested KTAP (not supported yet by ksft). But, I'm afraid that running time without intermediate output will be very long when we integrate all packetdrill scripts.

nf_conntrack: we can dedup the common.sh.

*pkt files: which of the 150+ scripts on github are candidates for kselftests, all or a subset? To avoid change detector tests. And what is the best way to eventually send up to 150 files, 7K LoC.

---
 tools/testing/selftests/Makefile              |  7 +-
 .../selftests/net/packetdrill/.gitignore      |  1 +
 .../selftests/net/packetdrill/Makefile        | 28 ++++++
 .../net/packetdrill/lib/py/__init__.py        | 15 ++++
 .../net/packetdrill/packetdrill_ksft.py       | 90 +++++++++++++++++++
 .../net/packetdrill/tcp/common/defaults.sh    | 63 +++++++++++++
 .../net/packetdrill/tcp/common/set_sysctls.py | 38 ++++++++
 .../net/packetdrill/tcp/inq/client.pkt        | 51 +++++++++++
 .../net/packetdrill/tcp/inq/server.pkt        | 51 +++++++++++
 .../tcp/md5/md5-only-on-client-ack.pkt        | 28 ++++++
 10 files changed, 369 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/.gitignore
 create mode 100644 tools/testing/selftests/net/packetdrill/Makefile
 create mode 100644 tools/testing/selftests/net/packetdrill/lib/py/__init__.py
 create mode 100755 tools/testing/selftests/net/packetdrill/packetdrill_ksft.py
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/defaults.sh
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp/common/set_sysctls.py
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/client.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/inq/server.pkt
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp/md5/md5-only-on-client-ack.pkt
tools/testing/selftests/Makefile +4 -3
index a5f1c0c27dff9..f03d6fee7ac54 100644
@@ -65,10 +65,11 @@ TARGETS += net/af_unix
 TARGETS += net/forwarding
 TARGETS += net/hsr
 TARGETS += net/mptcp
-TARGETS += net/openvswitch
-TARGETS += net/tcp_ao
 TARGETS += net/netfilter
+TARGETS += net/openvswitch
+TARGETS += net/packetdrill
 TARGETS += net/rds
+TARGETS += net/tcp_ao
 TARGETS += nsfs
 TARGETS += perf_events
 TARGETS += pidfd
@@ -122,7 +123,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net net/packetdrill drivers/net drivers/net/hw,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/net/packetdrill/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..a40f1a600eb94
@@ -0,0 +1 @@
+tcp*sh
tools/testing/selftests/net/packetdrill/Makefile new file +28 -0
new file mode 100644
index 0000000000000..d94c51098d1f0
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# KSFT includes
+TEST_INCLUDES := $(wildcard lib/py/*.py ../lib/py/*.py)
+
+# Packetdrill support file(s)
+TEST_INCLUDES += tcp/common/defaults.sh
+TEST_INCLUDES += tcp/common/set_sysctls.py
+
+# Packetdrill scripts: all .pkt in subdirectories
+TEST_INCLUDES += $(wildcard tcp/**/*.pkt)
+
+# Create a separate ksft test for each subdirectory
+# Running all packetdrill tests in one go will take too long
+#
+# For each tcp/$subdir, create a test script tcp_$subdir.py
+# Exclude tcp/common, which is a helper directory
+TEST_DIRS := $(wildcard tcp/*)
+TEST_DIRS := $(filter-out tcp/common, $(TEST_DIRS))
+TEST_CUSTOM_PROGS := $(foreach dir,$(TEST_DIRS),$(subst /,_,$(dir)).py)
+
+$(TEST_CUSTOM_PROGS) : packetdrill_ksft.py
+	cp $< $@
+
+# Needed to generate all TEST_CUSTOM_PROGS
+all: $(TEST_CUSTOM_PROGS)
+
+include ../../lib.mk
tools/testing/selftests/net/packetdrill/lib/py/__init__.py new file +15 -0
new file mode 100644
index 0000000000000..51bb6dda43d65
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import pathlib
+import sys
+
+KSFT_DIR = (pathlib.Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +42 -0
new file mode 100755
index 0000000000000..62572a5b8331c
@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Run packetdrill tests in the ksft harness.
+
+   Run all packetdrill tests in a subdirectory.
+   Detect the relevant subdirectory from this script name.
+   (Because the script cannot be given arguments.)
+
+   Run each test, for both IPv4 and IPv6.
+   Return a separate ksft result for each test case.
+"""
+
+import glob
+import os
+import pathlib
+import shutil
+
+from lib.py import cmd, ksft_exit, ksft_run, KsftSkipEx, NetNS
+
+
+def test_func_builder(pktfile_path, ipv4):
+    """Create a function that can be passed to ksft_run."""
+
+    def f():
+        if ipv4:
+            args = ("--ip_version=ipv4 "
+                    "--local_ip=192.168.0.1 "
+                    "--gateway_ip=192.168.0.1 "
+                    "--netmask_ip=255.255.0.0 "
+                    "--remote_ip=192.0.2.1 "
+                    "-D CMSG_LEVEL_IP=SOL_IP "
+                    "-D CMSG_TYPE_RECVERR=IP_RECVERR "
+                    )
+        else:
+            args = ("--ip_version=ipv6 --mtu=1520 "
+                    "--local_ip=fd3d:0a0b:17d6::1 "
+                    "--gateway_ip=fd3d:0a0b:17d6:8888::1 "
+                    "--remote_ip=fd3d:fa7b:d17d::1 "
+                    "-D CMSG_LEVEL_IP=SOL_IPV6 "
+                    "-D CMSG_TYPE_RECVERR=IPV6_RECVERR"
+                    )
Quoted 8 lines

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 45

Quoted 3 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +3 -0
+
+        if not shutil.which("packetdrill"):
+            raise KsftSkipEx("Cannot find packetdrill")

I don't see anything about building the binary itself. Am I missing something or should we also have some makefile magic to do it? I'm not sure packetdrill is packaged by the distros... Presumably we want the existing NIPA runners to run those tests as well?

Quoted 5 lines

As Jakub responded.

Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's nf_conntrack charted the path here.

Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +10 -0
+
+        netns = NetNS()
+
+        # Call packetdrill from the directory hosting the .pkt script,
+        # because scripts can have relative includes.
+        savedir = os.getcwd()
+        os.chdir(os.path.dirname(pktfile_path))
+        basename = os.path.basename(pktfile_path)
+        cmd(f"packetdrill {args} {basename}", ns=netns)
+        os.chdir(savedir)
Quoted 8 lines

tools/testing/selftests/net/packetdrill/packetdrill_ksft.py line 60

Quoted 5 lines
tools/testing/selftests/net/packetdrill/packetdrill_ksft.py new file +5 -0
+
+    if ipv4:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v4"
+    else:
+        f.__name__ = pathlib.Path(pktfile_path).stem + "-v6"

What about dualstack v4-over-v6 mode? Do we want to support that as well?

Quoted 2 lines

Does it give a sufficient signal? I skipped over it on purpose. But can enable if that is the consensus.

Makes sense. Agreed that most of the time v4-over-v6 just exercises the same v4 paths so it's not too much value. I'm assuming that the tests that care about v4-over-v6 peculiarities can just
do --ip_version=ipv4-mapped-ipv6 in their .pkt file (similar to what gtests/net/tcp/mtu_probe/basic-v6.pkt is doing)?

maintainer
#24 lore raw
Quoted 8 lines · On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
Quoted 5 lines

As Jakub responded.

Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's nf_conntrack charted the path here.

Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.

FWIW I lied to you, the "script" for buildings things wasn't committed in the repo, I pushed it now:
https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1

#25 lore raw
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Wed, 28 Aug 2024 10:20:47 -0700 Stanislav Fomichev wrote:
Quoted 5 lines

As Jakub responded.

Indeed importing the whole application into the kernel sources is what stopped me from attempting this before. Florian's nf_conntrack charted the path here.

Quoted 6 lines
Quoted 2 lines

Thanks both. I was expecting to find some build rules :-) But as long as we have this story sorted out, we're good.

FWIW I lied to you, the "script" for buildings things wasn't committed in the repo, I pushed it now:
https://github.com/linux-netdev/nipa/commit/dde0f093774a4a4c85e595b5cf44bd3e4bd434d1

Perfect, thanks!

#26 lore raw
In reply to Willem de Bruijn
Quoted 35 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Lay the groundwork to import into kselftests the over 150 packetdrill TCP/IP conformance tests on github.com/google/packetdrill.

Florian recently added support for packetdrill tests in nf_conntrack, in commit a8a388c2aae49 ("selftests: netfilter: add packetdrill based conntrack tests").

This patch takes a slightly different implementation and reuses the ksft python library for its KTAP, ksft, NetNS and other such tooling.

It also anticipates the large number of testcases, by creating a separate kselftest for each feature (directory). It does this by copying the template script packetdrill_ksft.py for each directory, and putting those in TEST_CUSTOM_PROGS so that kselftests runs each.

To demonstrate the code with minimal patch size, initially import only two features/directories from github. One with a single script, and one with two. This was the only reason to pick tcp/inq and tcp/md5.

Any future imports of packetdrill tests should require no additional coding. Just add the tcp/$FEATURE directory with *.pkt files.

Implementation notes:
- restore alphabetical order when adding the new directory to tools/testing/selftests/Makefile
- copied *.pkt files and support verbatim from the github project, except for
- update common/defaults.sh path (there are two paths on github)
- add SPDX headers
- remove one author statement
- Acknowledgment: drop an e (checkpatch)

Quoted 8 lines

Tested:
make -C tools/testing/selftests/ \
TARGETS=net/packetdrill \
install INSTALL_PATH=$KSFT_INSTALL_PATH

# in virtme-ng
sudo ./run_kselftest.sh -c net/packetdrill sudo ./run_kselftest.sh -t net/packetdrill:tcp_inq.py

I did not know about run_kselftest.sh. From a quick look, it seems to require selftests to be installed to run them. I think nipa relies on something slightly different, it does something like this, AFAIU this runs an individual kselftest without requiring it to be installed first (which is nice):

make -C tools/testing/selftests TARGETS="net" TEST_PROGS="rxtimestamp.sh" TEST_GEN_PROGS="" run_tests

It may be worth it to check if these added tests can be ran in a similar way.

This may also solve your issue with 'intermediate output' being printed after every test. AFAIU if the tests can be ran individually using a command like this, then you don't need intermediate output, maybe.

maintainer
#27 lore raw
Quoted 10 lines · On Wed, 28 Aug 2024 11:26:28 -0700 Mina Almasry wrote:

I did not know about run_kselftest.sh. From a quick look, it seems to require selftests to be installed to run them. I think nipa relies on something slightly different, it does something like this, AFAIU this runs an individual kselftest without requiring it to be installed first (which is nice):

make -C tools/testing/selftests TARGETS="net" TEST_PROGS="rxtimestamp.sh" TEST_GEN_PROGS="" run_tests

It may be worth it to check if these added tests can be ran in a similar way.

They should, the ksft infra converges into the same bash script (runner.sh) whether we enter via make or via the "installed" runner.

Quoted 4 lines

This may also solve your issue with 'intermediate output' being printed after every test. AFAIU if the tests can be ran individually using a command like this, then you don't need intermediate output, maybe.

selftest infra tries to prefix raw output with # to make sure that random prints don't get mixed with KTAP. IIRC there are two ways to do that, one is a perl script and that doesn't buffer, but if there's no perl in PATH it runs the output thru sed, and sed buffers.