selftests: drv-net: support testing with a remote system
Subsystems (3)
Jakub Kicinski
maintainerv3 cover selftests: drv-net: support testing with a remote system
Hi!
Implement support for tests which require access to a remote system / endpoint which can generate traffic.
This series concludes the "groundwork" for upstream driver tests.
I wanted to support the three models which came up in discussions:
- SW testing with netdevsim
- "local" testing with two ports on the same system in a loopback
- "remote" testing via SSH
so there is a tiny bit of an abstraction which wraps up how "remote" commands are executed. Otherwise hopefully there's nothing surprising.
I'm only adding a ping test. I had a bigger one written but I was worried we'll get into discussing the details of the test itself and how I chose to hack up netdevsim, instead of the test infra... So that test will be a follow up :)
v3:
- first two patches are new
- make Remote::cmd() return Popen() object (patch 3)
- always operate on absolute paths (patch 3)
- last two patches are new
v2: https://lore.kernel.org/all/[email protected]
- rename endpoint -> remote
- use 2001:db8:: v6 prefix
- add a note about persistent SSH connections
- add the kernel config
v1: https://lore.kernel.org/all/[email protected]
Jakub Kicinski (8):
selftests: net: fix counting totals when some checks fail
selftests: net: set the exit code correctly in Python tests
selftests: drv-net: define endpoint structures
selftests: drv-net: factor out parsing of the env
selftests: drv-net: construct environment for running tests which
require an endpoint
selftests: drv-net: add a trivial ping test
selftests: net: support matching cases by name prefix
selftests: drv-net: add a TCP ping test case (and useful helpers)
tools/testing/selftests/drivers/net/Makefile | 5 +- .../testing/selftests/drivers/net/README.rst | 33 ++++ .../selftests/drivers/net/lib/py/__init__.py | 1 + .../selftests/drivers/net/lib/py/env.py | 141 +++++++++++++++--- .../selftests/drivers/net/lib/py/remote.py | 15 ++ .../drivers/net/lib/py/remote_netns.py | 21 +++ .../drivers/net/lib/py/remote_ssh.py | 39 +++++ tools/testing/selftests/drivers/net/ping.py | 52 +++++++ tools/testing/selftests/drivers/net/stats.py | 4 +- .../testing/selftests/net/lib/py/__init__.py | 1 + tools/testing/selftests/net/lib/py/ksft.py | 25 +++- tools/testing/selftests/net/lib/py/netns.py | 31 ++++ tools/testing/selftests/net/lib/py/utils.py | 50 ++++++- tools/testing/selftests/net/nl_netdev.py | 4 +- 14 files changed, 391 insertions(+), 31 deletions(-) create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote_netns.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote_ssh.py create mode 100755 tools/testing/selftests/drivers/net/ping.py create mode 100644 tools/testing/selftests/net/lib/py/netns.py
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 1/8 selftests: net: fix counting totals when some checks fail
Totals currently only pay attention to exceptions, if check fails (say ksft_eq()) the test case will be counted as pass:
# At /ksft/drivers/net/./ping.py line 18:
# Check failed 1 != 2
not ok 1 ping.test_v4
ok 2 ping.test_v6
ok 3 ping.test_tcp
# Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0
^^^^^^^^^^^^^
Pay attention to the result.
--- tools/testing/selftests/net/lib/py/ksft.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/testing/selftests/net/lib/py/ksft.py +4 -1
@@ -106,7 +106,10 @@ KSFT_RESULT = None continue ktap_result(KSFT_RESULT, cnt, case) - totals['pass'] += 1 + if KSFT_RESULT: + totals['pass'] += 1 + else: + totals['fail'] += 1 print( f"# Totals: pass:{totals['pass']} fail:{totals['fail']} xfail:{totals['xfail']} xpass:0 skip:{totals['skip']} error:0"
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 2/8 selftests: net: set the exit code correctly in Python tests
In reply to Jakub KicinskiTest cases need to exit with non-zero status if they failed, we currently don't do that:
# KTAP version 1
# 1..3
# # At /root/ksft-net-drv/drivers/net/./ping.py line 18:
# # Check failed 1 != 2
# not ok 1 ping.test_v4 # ok 2 ping.test_v6 # ok 3 ping.test_tcp # # Totals: pass:2 fail:1 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: drivers/net: ping.py ^^^^
It's a bit tempting to make the exit part of ksft_run(), but that only works well for very trivial setups. We can revisit this later, if people forget to call ksft_exit().
--- tools/testing/selftests/drivers/net/stats.py | 4 +++- tools/testing/selftests/net/lib/py/ksft.py | 10 ++++++++++ tools/testing/selftests/net/nl_netdev.py | 4 +++- 3 files changed, 16 insertions(+), 2 deletions(-)
tools/testing/selftests/drivers/net/stats.py +3 -1
@@ -1,7 +1,8 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 -from lib.py import ksft_run, ksft_in, ksft_true, KsftSkipEx, KsftXfailEx +from lib.py import ksft_run, ksft_exit +from lib.py import ksft_in, ksft_true, KsftSkipEx, KsftXfailEx from lib.py import EthtoolFamily, NetdevFamily, RtnlFamily, NlError from lib.py import NetDrvEnv
@@ -80,6 +81,7 @@ rtnl = RtnlFamily() with NetDrvEnv(__file__) as cfg: ksft_run([check_pause, check_fec, pkt_byte_sum], args=(cfg, )) + ksft_exit() if __name__ == "__main__":
tools/testing/selftests/net/lib/py/ksft.py +10 -0
@@ -2,11 +2,13 @@ import builtins import inspect +import sys import time import traceback from .consts import KSFT_MAIN_NAME KSFT_RESULT = None +KSFT_RESULT_ALL = True class KsftSkipEx(Exception):
@@ -63,6 +65,9 @@ KSFT_RESULT = None def ktap_result(ok, cnt=1, case="", comment=""): + global KSFT_RESULT_ALL + KSFT_RESULT_ALL = KSFT_RESULT_ALL and ok + res = "" if not ok: res += "not "
@@ -114,3 +119,8 @@ KSFT_RESULT = None print( f"# Totals: pass:{totals['pass']} fail:{totals['fail']} xfail:{totals['xfail']} xpass:0 skip:{totals['skip']} error:0" ) + + +def ksft_exit(): + global KSFT_RESULT_ALL + sys.exit(0 if KSFT_RESULT_ALL else 1)
tools/testing/selftests/net/nl_netdev.py +3 -1
@@ -2,7 +2,8 @@ # SPDX-License-Identifier: GPL-2.0 import time -from lib.py import ksft_run, ksft_pr, ksft_eq, ksft_ge, ksft_busy_wait +from lib.py import ksft_run, ksft_exit, ksft_pr +from lib.py import ksft_eq, ksft_ge, ksft_busy_wait from lib.py import NetdevFamily, NetdevSimDev, ip
@@ -90,6 +91,7 @@ from lib.py import NetdevFamily, NetdevSimDev, ip nf = NetdevFamily() ksft_run([empty_check, lo_check, page_pool_check], args=(nf, )) + ksft_exit() if __name__ == "__main__":
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 3/8 selftests: drv-net: define endpoint structures
In reply to Jakub KicinskiDefine the remote endpoint "model". To execute most meaningful device driver tests we need to be able to communicate with a remote system, and have it send traffic to the device under test.
Various test environments will have different requirements.
0) "Local" netdevsim-based testing can simply use net namespaces. netdevsim supports connecting two devices now, to form a veth-like construct.
1) Similarly on hosts with multiple NICs, the NICs may be connected together with a loopback cable or internal device loopback. One interface may be placed into separate netns, and tests would proceed much like in the netdevsim case. Note that the loopback config or the moving of one interface
into a netns is not expected to be part of selftest code.
2) Some systems may need to communicate with the remote endpoint via SSH.
3) Last but not least environment may have its own custom communication method.
Fundamentally we only need two operations:
- run a command remotely
- deploy a binary (if some tool we need is built as part of kselftests)
Wrap these two in a class. Use dynamic loading to load the Remote class. This will allow very easy definition of other communication methods without bothering upstream code base.
Stick to the "simple" / "no unnecessary abstractions" model for referring to the remote endpoints. The host / remote object are passed as an argument to the usual cmd() or ip() invocation. For example:
ip("link show", json=True, host=remote)
---
v3:
- make Remote() return Popen() object
- always operate on absolute paths
v2:
- rename Endpoint -> Remote
--- .../selftests/drivers/net/lib/py/__init__.py | 1 + .../selftests/drivers/net/lib/py/remote.py | 15 +++++++ .../drivers/net/lib/py/remote_netns.py | 21 ++++++++++ .../drivers/net/lib/py/remote_ssh.py | 39 +++++++++++++++++++ tools/testing/selftests/net/lib/py/utils.py | 17 ++++---- 5 files changed, 85 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote_netns.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
tools/testing/selftests/drivers/net/lib/py/__init__.py +1 -0
@@ -15,3 +15,4 @@ KSFT_DIR = (Path(__file__).parent / "../../../..").resolve() sys.exit(4) from .env import * +from .remote import Remote
tools/testing/selftests/drivers/net/lib/py/remote.py new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 + +import os +import importlib + +_modules = {} + +def Remote(kind, args, src_path): + global _modules + + if kind not in _modules: + _modules[kind] = importlib.import_module("..remote_" + kind, __name__) + + dir_path = os.path.abspath(src_path + "/../") + return getattr(_modules[kind], "Remote")(args, dir_path)
tools/testing/selftests/drivers/net/lib/py/remote_netns.py new file +21 -0
@@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0 + +import os +import subprocess + +from lib.py import cmd + + +class Remote: + def __init__(self, name, dir_path): + self.name = name + self.dir_path = dir_path + + def cmd(self, comm): + return subprocess.Popen(["ip", "netns", "exec", self.name, "bash", "-c", comm], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + + def deploy(self, what): + if os.path.isabs(what): + return what + return os.path.abspath(self.dir_path + "/" + what)
tools/testing/selftests/drivers/net/lib/py/remote_ssh.py new file +39 -0
@@ -0,0 +1,39 @@ +# SPDX-License-Identifier: GPL-2.0 + +import os +import string +import subprocess +import random + +from lib.py import cmd + + +class Remote: + def __init__(self, name, dir_path): + self.name = name + self.dir_path = dir_path + self._tmpdir = None + + def __del__(self): + if self._tmpdir: + cmd("rm -rf " + self._tmpdir, host=self) + self._tmpdir = None + + def cmd(self, comm): + return subprocess.Popen(["ssh", "-q", self.name, comm], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + + def _mktmp(self): + return ''.join(random.choice(string.ascii_lowercase) for _ in range(8)) + + def deploy(self, what): + if not self._tmpdir: + self._tmpdir = "/tmp/" + self._mktmp() + cmd("mkdir " + self._tmpdir, host=self) + file_name = self._tmpdir + "/" + self._mktmp() + os.path.basename(what) + + if not os.path.isabs(what): + what = os.path.abspath(self.dir_path + "/" + what) + + cmd(f"scp {what} {self.name}:{file_name}") + return file_name
tools/testing/selftests/net/lib/py/utils.py +9 -8
@@ -4,10 +4,8 @@ import json as _json import subprocess class cmd: - def __init__(self, comm, shell=True, fail=True, ns=None, background=False): + def __init__(self, comm, shell=True, fail=True, ns=None, background=False, host=None): if ns: - if isinstance(ns, NetNS): - ns = ns.name comm = f'ip netns exec {ns} ' + comm self.stdout = None
@@ -15,15 +13,18 @@ import subprocess self.ret = None self.comm = comm - self.proc = subprocess.Popen(comm, shell=shell, stdout=subprocess.PIPE, - stderr=subprocess.PIPE) + if host: + self.proc = host.cmd(comm) + else: + self.proc = subprocess.Popen(comm, shell=shell, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) if not background: self.process(terminate=False, fail=fail) def process(self, terminate=True, fail=None): if terminate: self.proc.terminate() - stdout, stderr = self.proc.communicate() + stdout, stderr = self.proc.communicate(timeout=3) self.stdout = stdout.decode("utf-8") self.stderr = stderr.decode("utf-8") self.proc.stdout.close()
@@ -37,12 +38,12 @@ import subprocess (self.proc.args, stdout, stderr)) -def ip(args, json=None, ns=None): +def ip(args, json=None, ns=None, host=None): cmd_str = "ip " if json: cmd_str += '-j ' cmd_str += args - cmd_obj = cmd(cmd_str, ns=ns) + cmd_obj = cmd(cmd_str, ns=ns, host=host) if json: return _json.loads(cmd_obj.stdout) return cmd_obj
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 4/8 selftests: drv-net: factor out parsing of the env
In reply to Jakub KicinskiThe tests with a remote end will use a different class, for clarity, but will also need to parse the env. So factor parsing the env out to a function.
--- .../selftests/drivers/net/lib/py/env.py | 43 +++++++++++-------- 1 file changed, 26 insertions(+), 17 deletions(-)
tools/testing/selftests/drivers/net/lib/py/env.py +26 -17
@@ -6,12 +6,36 @@ from pathlib import Path from lib.py import ip from lib.py import NetdevSimDev + +def _load_env_file(src_path): + env = os.environ.copy() + + src_dir = Path(src_path).parent.resolve() + if not (src_dir / "net.config").exists(): + return env + + lexer = shlex.shlex(open((src_dir / "net.config").as_posix(), 'r').read()) + k = None + for token in lexer: + if k is None: + k = token + env[k] = "" + elif token == "=": + pass + else: + env[k] = token + k = None + return env + + class NetDrvEnv: + """ + Class for a single NIC / host env, with no remote end + """ def __init__(self, src_path): self._ns = None - self.env = os.environ.copy() - self._load_env_file(src_path) + self.env = _load_env_file(src_path) if 'NETIF' in self.env: self.dev = ip("link show dev " + self.env['NETIF'], json=True)[0]
@@ -34,19 +58,4 @@ from lib.py import NetdevSimDev self._ns.remove() self._ns = None - def _load_env_file(self, src_path): - src_dir = Path(src_path).parent.resolve() - if not (src_dir / "net.config").exists(): - return - lexer = shlex.shlex(open((src_dir / "net.config").as_posix(), 'r').read()) - k = None - for token in lexer: - if k is None: - k = token - self.env[k] = "" - elif token == "=": - pass - else: - self.env[k] = token - k = None
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 5/8 selftests: drv-net: construct environment for running tests which require an endpoint
In reply to Jakub KicinskiNothing surprising here, hopefully. Wrap the variables from the environment into a class or spawn a netdevsim based env and pass it to the tests.
--- .../testing/selftests/drivers/net/README.rst | 33 +++++++ .../selftests/drivers/net/lib/py/env.py | 98 ++++++++++++++++++- .../testing/selftests/net/lib/py/__init__.py | 1 + tools/testing/selftests/net/lib/py/netns.py | 31 ++++++ 4 files changed, 162 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/net/lib/py/netns.py
tools/testing/selftests/drivers/net/README.rst +33 -0
@@ -23,8 +23,41 @@ Variables can be set in the environment or by creating a net.config # Variable set in a file NETIF=eth0 +Please note that the config parser is very simple, if there are +any non-alphanumeric characters in the value it needs to be in +double quotes. + NETIF ~~~~~ Name of the netdevice against which the test should be executed. When empty or not set software devices will be used. + +LOCAL_V4, LOCAL_V6, REMOTE_V4, REMOTE_V6 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Local and remote endpoint IP addresses. + +REMOTE_TYPE +~~~~~~~~~~~ + +Communication method used to run commands on the remote endpoint. +Test framework has built-in support for ``netns`` and ``ssh`` channels. +``netns`` assumes the "remote" interface is part of the same +host, just moved to the specified netns. +``ssh`` communicates with remote endpoint over ``ssh`` and ``scp``. +Using persistent SSH connections is strongly encouraged to avoid +the latency of SSH connection setup on every command. + +Communication methods are defined by classes in ``lib/py/remote_{name}.py``. +It should be possible to add a new method without modifying any of +the framework, by simply adding an appropriately named file to ``lib/py``. + +REMOTE_ARGS +~~~~~~~~~~~ + +Arguments used to construct the communication channel. +Communication channel dependent:: + + for netns - name of the "remote" namespace + for ssh - name/address of the remote host
tools/testing/selftests/drivers/net/lib/py/env.py +97 -1
107 lines not on this page · raw message
tools/testing/selftests/net/lib/py/__init__.py +1 -0
@@ -2,6 +2,7 @@ from .consts import KSRC from .ksft import * +from .netns import NetNS from .nsim import * from .utils import * from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/netns.py new file +31 -0
@@ -0,0 +1,31 @@ +# SPDX-License-Identifier: GPL-2.0 + +from .utils import ip +import random +import string + + +class NetNS: + def __init__(self, name=None): + if name: + self.name = name + else: + self.name = ''.join(random.choice(string.ascii_lowercase) for _ in range(8)) + ip('netns add ' + self.name) + + def __del__(self): + if self.name: + ip('netns del ' + self.name) + self.name = None + + def __enter__(self): + return self + + def __exit__(self, ex_type, ex_value, ex_tb): + self.__del__() + + def __str__(self): + return self.name + + def __repr__(self): + return f"NetNS({self.name})"
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 6/8 selftests: drv-net: add a trivial ping test
In reply to Jakub KicinskiAdd a very simple test for testing with a remote system. Both IPv4 and IPv6 connectivity is optional so tests will XFail is env doesn't define an address for the given family.
Using netdevsim:
$ ./run_kselftest.sh -t drivers/net:ping.py TAP version 13 1..1 # timeout set to 45 # selftests: drivers/net: ping.py # KTAP version 1 # 1..2 # ok 1 ping.test_v4 # ok 2 ping.test_v6 # # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: drivers/net: ping.py
Command line SSH:
$ NETIF=virbr0 REMOTE_TYPE=ssh [email protected] \ LOCAL_V4=192.168.122.1 REMOTE_V4=192.168.122.123 \ ./tools/testing/selftests/drivers/net/ping.py KTAP version 1 1..2 ok 1 ping.test_v4 ok 2 ping.test_v6 # XFAIL # Totals: pass:1 fail:0 xfail:1 xpass:0 skip:0 error:0
Existing devices placed in netns (and using net.config):
$ cat drivers/net/net.config NETIF=veth0 REMOTE_TYPE=netns REMOTE_ARGS=red LOCAL_V4="192.168.1.1" REMOTE_V4="192.168.1.2"
$ ./run_kselftest.sh -t drivers/net:ping.py TAP version 13 1..1 # timeout set to 45 # selftests: drivers/net: ping.py # KTAP version 1 # 1..2 # ok 1 ping.test_v4 # ok 2 ping.test_v6 # XFAIL # # Totals: pass:1 fail:0 xfail:1 xpass:0 skip:0 error:0
--- tools/testing/selftests/drivers/net/Makefile | 5 ++- tools/testing/selftests/drivers/net/ping.py | 33 ++++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/drivers/net/ping.py
tools/testing/selftests/drivers/net/Makefile +4 -1
@@ -2,6 +2,9 @@ TEST_INCLUDES := $(wildcard lib/py/*.py) -TEST_PROGS := stats.py +TEST_PROGS := \ + ping.py \ + stats.py \ +# end of TEST_PROGS include ../../lib.mk
tools/testing/selftests/drivers/net/ping.py new file +33 -0
@@ -0,0 +1,33 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +from lib.py import ksft_run, ksft_exit, KsftXfailEx +from lib.py import NetDrvEpEnv +from lib.py import cmd + + +def test_v4(cfg) -> None: + if not cfg.v4: + raise KsftXfailEx() + + cmd(f"ping -c 1 -W0.5 {cfg.remote_v4}") + cmd(f"ping -c 1 -W0.5 {cfg.v4}", host=cfg.remote) + + +def test_v6(cfg) -> None: + if not cfg.v6: + raise KsftXfailEx() + + cmd(f"ping -c 1 -W0.5 {cfg.remote_v6}") + cmd(f"ping -c 1 -W0.5 {cfg.v6}", host=cfg.remote) + + +def main() -> None: + with NetDrvEpEnv(__file__) as cfg: + ksft_run([test_v4, test_v6], + args=(cfg, )) + ksft_exit() + + +if __name__ == "__main__": + main()
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 7/8 selftests: net: support matching cases by name prefix
In reply to Jakub KicinskiWhile writing tests with a lot more cases I got tired of having to jump back and forth to add the name of the test to the ksft_run() list. Most unittest frameworks do some name matching, e.g. assume that functions with names starting with test_ are test cases.
Support similar flow in ksft_run(). Let the author list the desired prefixes. globals() need to be passed explicitly, IDK how to work around that.
--- tools/testing/selftests/drivers/net/ping.py | 3 +-- tools/testing/selftests/net/lib/py/ksft.py | 10 +++++++++- 2 files changed, 10 insertions(+), 3 deletions(-)
tools/testing/selftests/drivers/net/ping.py +1 -2
@@ -24,8 +24,7 @@ from lib.py import cmd def main() -> None: with NetDrvEpEnv(__file__) as cfg: - ksft_run([test_v4, test_v6], - args=(cfg, )) + ksft_run(globs=globals(), case_pfx={"test_"}, args=(cfg, )) ksft_exit()
tools/testing/selftests/net/lib/py/ksft.py +9 -1
@@ -81,7 +81,15 @@ KSFT_RESULT_ALL = True print(res) -def ksft_run(cases, args=()): +def ksft_run(cases=None, globs=None, case_pfx=None, args=()): + cases = cases or [] + + if globs and case_pfx: + for key, value in globs.items(): + stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)]) + if callable(value) and stats_with_pfx: + cases.append(value) + totals = {"pass": 0, "fail": 0, "skip": 0, "xfail": 0} print("KTAP version 1")
Signature · 1 line
2.44.0
Re: [PATCH net-next v3 7/8] selftests: net: support matching cases by name prefix
Quoted 34 lines · Jakub Kicinski
While writing tests with a lot more cases I got tired of having to jump back and forth to add the name of the test to the
ksft_run()list. Most unittest frameworks do some name matching, e.g. assume that functions with names starting withtest_are test cases.Support similar flow in
ksft_run(). Let the author list the desired prefixes.globals()need to be passed explicitly, IDK how to work around that.Signed-off-by Jakub Kicinski <[email protected]>--- tools/testing/selftests/drivers/net/ping.py | 3 +-- tools/testing/selftests/net/lib/py/ksft.py | 10 +++++++++- 2 files changed, 10 insertions(+), 3 deletions(-)tools/testing/selftests/drivers/net/ping.py +1
-2@@ -24,8 +24,7 @@ from lib.py import cmd def main() -> None: with NetDrvEpEnv(__file__) as cfg: - ksft_run([test_v4, test_v6], - args=(cfg, )) + ksft_run(globs=globals(), case_pfx={"test_"}, args=(cfg, )) ksft_exit()tools/testing/selftests/net/lib/py/ksft.py +0
-0
tools/testing/selftests/net/lib/py/ksft.py KSFT_RESULT_ALL = True · line 89
Quoted 10 lines
tools/testing/selftests/net/lib/py/ksft.py +6
-1@@ -81,7 +81,15 @@ KSFT_RESULT_ALL = True print(res) -def ksft_run(cases, args=()): +def ksft_run(cases=None, globs=None, case_pfx=None, args=()): + cases = cases or [] + + if globs and case_pfx: + for key, value in globs.items(): + stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)])
stats -> starts
for the reader, just spell out prefix instead of pfx?
perhaps less pythonic, but just
if key.startswith(prefix) and callable(value):
cases.append(value)
?
Quoted 9 lines
tools/testing/selftests/net/lib/py/ksft.py +3
-0+ if callable(value) and stats_with_pfx: + cases.append(value) + totals = {"pass": 0, "fail": 0, "skip": 0, "xfail": 0} print("KTAP version 1")Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [PATCH net-next v3 7/8] selftests: net: support matching cases by name prefix
Quoted 10 lines · On Thu, 18 Apr 2024 10:26:19 -0400 Willem de Bruijn wrote:
Review
Quoted 7 lines
(quoted diff) +6
-1-def ksft_run(cases, args=()): +def ksft_run(cases=None, globs=None, case_pfx=None, args=()): + cases = cases or [] + + if globs and case_pfx: + for key, value in globs.items(): + stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)])stats -> starts
Quoted 6 lines
for the reader, just spell out prefix instead of pfx?
perhaps less pythonic, but just
if
key.startswith(prefix)and callable(value):
cases.append(value)
like this?
tools/testing/selftests/net/lib/py/ksft.py +6 -3
@@ -86,9 +86,12 @@ KSFT_RESULT_ALL = True if globs and case_pfx: for key, value in globs.items(): - stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)]) - if callable(value) and stats_with_pfx: - cases.append(value) + if not callable(value): + continue + for prefix in case_pfx: + if key.startswith(prefix): + cases.append(value) + break totals = {"pass": 0, "fail": 0, "skip": 0, "xfail": 0}
Re: [PATCH net-next v3 7/8] selftests: net: support matching cases by name prefix
Quoted 27 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 18 Apr 2024 10:26:19 -0400 Willem de Bruijn wrote:
Review
Quoted 7 lines
(quoted diff) +6
-1-def ksft_run(cases, args=()): +def ksft_run(cases=None, globs=None, case_pfx=None, args=()): + cases = cases or [] + + if globs and case_pfx: + for key, value in globs.items(): + stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)])stats -> starts
Quoted 6 lines
for the reader, just spell out prefix instead of pfx?
perhaps less pythonic, but just
if
key.startswith(prefix)and callable(value):
cases.append(value)like this?
tools/testing/selftests/net/lib/py/ksft.py +0
-0@@ -86,9 +86,12 @@ KSFT_RESULT_ALL = True if globs and case_pfx:
tools/testing/selftests/net/lib/py/ksft.py KSFT_RESULT_ALL = True · line 94
Quoted 10 lines
tools/testing/selftests/net/lib/py/ksft.py +6
-3for key, value in globs.items(): - stats_with_pfx = bool([pfx for pfx in case_pfx if key.startswith(pfx)]) - if callable(value) and stats_with_pfx: - cases.append(value) + if not callable(value): + continue + for prefix in case_pfx: + if key.startswith(prefix): + cases.append(value) + break
Yes. I would not have brought this up if it wasn't for the typo as well. Python developers perhaps find this less pythonic, but I do find it easier to follow.
Jakub Kicinski
maintainerPATCH v3 8/8 selftests: drv-net: add a TCP ping test case (and useful helpers)
In reply to Jakub KicinskiMore complex tests often have to spawn a background process, like a server which will respond to requests or tcpdump.
Add support for creating such processes using the with keyword:
with bkg("my-daemon", ..):
# my-daemon is alive in this block
My initial thought was to add this support to cmd() directly but it runs the command in the constructor, so by the time we __enter__ it's too late to make sure we used "background=True".
Second useful helper transplanted from net_helper.sh is wait_port_listen().
--- tools/testing/selftests/drivers/net/ping.py | 24 +++++++++++++-- tools/testing/selftests/net/lib/py/utils.py | 33 +++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-)
tools/testing/selftests/drivers/net/ping.py +22 -2
@@ -1,9 +1,12 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 -from lib.py import ksft_run, ksft_exit, KsftXfailEx +import random + +from lib.py import ksft_run, ksft_exit +from lib.py import ksft_eq, KsftXfailEx from lib.py import NetDrvEpEnv -from lib.py import cmd +from lib.py import bkg, cmd, wait_port_listen def test_v4(cfg) -> None:
@@ -22,6 +25,23 @@ from lib.py import cmd cmd(f"ping -c 1 -W0.5 {cfg.v6}", host=cfg.remote) +def test_tcp(cfg) -> None: + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.addr} {port}") as nc: + wait_port_listen(port) + + cmd(f"echo ping | nc {cfg.addr} {port}", + shell=True, host=cfg.remote) + ksft_eq(nc.stdout.strip(), "ping") + + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.remote_addr} {port}", host=cfg.remote) as nc: + wait_port_listen(port, host=cfg.remote) + + cmd(f"echo ping | nc {cfg.remote_addr} {port}", shell=True) + ksft_eq(nc.stdout.strip(), "ping") + + def main() -> None: with NetDrvEpEnv(__file__) as cfg: ksft_run(globs=globals(), case_pfx={"test_"}, args=(cfg, ))
tools/testing/selftests/net/lib/py/utils.py +33 -0
@@ -1,7 +1,10 @@ # SPDX-License-Identifier: GPL-2.0 import json as _json +import re import subprocess +import time + class cmd: def __init__(self, comm, shell=True, fail=True, ns=None, background=False, host=None):
@@ -38,6 +41,18 @@ import subprocess (self.proc.args, stdout, stderr)) +class bkg(cmd): + def __init__(self, comm, shell=True, fail=True, ns=None, host=None): + super().__init__(comm, background=True, + shell=shell, fail=fail, ns=ns, host=host) + + def __enter__(self): + return self + + def __exit__(self, ex_type, ex_value, ex_tb): + return self.process() + + def ip(args, json=None, ns=None, host=None): cmd_str = "ip " if json:
@@ -47,3 +62,21 @@ import subprocess if json: return _json.loads(cmd_obj.stdout) return cmd_obj + + +def wait_port_listen(port, proto="tcp", ns=None, host=None, sleep=0.005, deadline=1): + end = time.monotonic() + deadline + + pattern = f":{port:04X} .* " + if proto == "tcp": # for tcp protocol additionally check the socket state + pattern += "0A" + pattern = re.compile(pattern) + + while True: + data = cmd(f'cat /proc/net/{proto}*', ns=ns, host=host, shell=True).stdout + for row in data.split("\n"): + if pattern.search(row): + return + if time.monotonic() > end: + raise Exception("Waiting for port listen timed out") + time.sleep(sleep)
Signature · 1 line
2.44.0
Re: [PATCH net-next v3 8/8] selftests: drv-net: add a TCP ping test case (and useful helpers)
Quoted 50 lines · Jakub Kicinski
More complex tests often have to spawn a background process, like a server which will respond to requests or tcpdump.
Add support for creating such processes using the with keyword:
with
bkg("my-daemon", ..):# my-daemon is alive in this blockMy initial thought was to add this support to
cmd()directly but it runs the command in the constructor, so by the time we__enter__it's too late to make sure we used "background=True".Second useful helper transplanted from
net_helper.shiswait_port_listen().Signed-off-by Jakub Kicinski <[email protected]>--- tools/testing/selftests/drivers/net/ping.py | 24 +++++++++++++-- tools/testing/selftests/net/lib/py/utils.py | 33 +++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-)tools/testing/selftests/drivers/net/ping.py +11
-2@@ -1,9 +1,12 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 -from lib.py import ksft_run, ksft_exit, KsftXfailEx +import random + +from lib.py import ksft_run, ksft_exit +from lib.py import ksft_eq, KsftXfailEx from lib.py import NetDrvEpEnv -from lib.py import cmd +from lib.py import bkg, cmd, wait_port_listen def test_v4(cfg) -> None:@@ -22,6 +25,23 @@ from lib.py import cmd cmd(f"ping -c 1 -W0.5 {cfg.v6}", host=cfg.remote) +def test_tcp(cfg) -> None: + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.addr} {port}") as nc: + wait_port_listen(port) + + cmd(f"echo ping | nc {cfg.addr} {port}",
tools/testing/selftests/drivers/net/ping.py from lib.py import cmd · line 43
Quoted 10 lines
tools/testing/selftests/drivers/net/ping.py +10
-0+ shell=True, host=cfg.remote) + ksft_eq(nc.stdout.strip(), "ping") + + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.remote_addr} {port}", host=cfg.remote) as nc: + wait_port_listen(port, host=cfg.remote) + + cmd(f"echo ping | nc {cfg.remote_addr} {port}", shell=True) + ksft_eq(nc.stdout.strip(), "ping") +
There are different netcat implementations floating around.
I notice that I have to pass -N on the client to terminate the connection after EOF. Else both peers keep the connection open, waiting for input. And explicitly pass -6 if passing an IPv6 address. I think this is the one that ships with Debian..
Jakub Kicinski
maintainerRe: [PATCH net-next v3 8/8] selftests: drv-net: add a TCP ping test case (and useful helpers)
Quoted 19 lines · On Thu, 18 Apr 2024 10:44:39 -0400 Willem de Bruijn wrote:
Quoted 6 lines
(quoted diff) +6
-0+def test_tcp(cfg) -> None: + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.addr} {port}") as nc: + wait_port_listen(port) + + cmd(f"echo ping | nc {cfg.addr} {port}",Review
Quoted 10 lines
(quoted diff) +10
-0+ shell=True, host=cfg.remote) + ksft_eq(nc.stdout.strip(), "ping") + + port = random.randrange(1024 + (1 << 15)) + with bkg(f"nc -l {cfg.remote_addr} {port}", host=cfg.remote) as nc: + wait_port_listen(port, host=cfg.remote) + + cmd(f"echo ping | nc {cfg.remote_addr} {port}", shell=True) + ksft_eq(nc.stdout.strip(), "ping") +There are different netcat implementations floating around.
Quoted 4 lines
I notice that I have to pass -N on the client to terminate the connection after EOF. Else both peers keep the connection open, waiting for input. And explicitly pass -6 if passing an IPv6 address. I think this is the one that ships with Debian..
Right, 100% laziness on my part. Mostly because socat requires bracketed IPv6. But once I tried it I also run into the premature termination problem, so ended up with this diff on top:
tools/testing/selftests/drivers/net/lib/py/env.py +4 -0
@@ -110,6 +110,10 @@ from .remote import Remote self.addr = self.v6 if self.v6 else self.v4 self.remote_addr = self.remote_v6 if self.remote_v6 else self.remote_v4 + # Bracketed addresses, some commands need IPv6 to be inside [] + self.baddr = f"[{self.v6}]" if self.v6 else self.v4 + self.remote_baddr = f"[{self.remote_v6}]" if self.remote_v6 else self.remote_v4 + self.ifname = self.dev['ifname'] self.ifindex = self.dev['ifindex']
tools/testing/selftests/drivers/net/ping.py +6 -4
@@ -27,18 +27,20 @@ from lib.py import bkg, cmd, wait_port_listen def test_tcp(cfg) -> None: port = random.randrange(1024 + (1 << 15)) - with bkg(f"nc -l {cfg.addr} {port}") as nc: + + with bkg(f"socat -t 2 -u TCP-LISTEN:{port} STDOUT", exit_wait=True) as nc: wait_port_listen(port) - cmd(f"echo ping | nc {cfg.addr} {port}", + cmd(f"echo ping | socat -t 2 -u STDIN TCP:{cfg.baddr}:{port}", shell=True, host=cfg.remote) ksft_eq(nc.stdout.strip(), "ping") port = random.randrange(1024 + (1 << 15)) - with bkg(f"nc -l {cfg.remote_addr} {port}", host=cfg.remote) as nc: + with bkg(f"socat -t 2 -u TCP-LISTEN:{port} STDOUT", host=cfg.remote, + exit_wait=True) as nc: wait_port_listen(port, host=cfg.remote) - cmd(f"echo ping | nc {cfg.remote_addr} {port}", shell=True) + cmd(f"echo ping | socat -t 2 -u STDIN TCP:{cfg.remote_baddr}:{port}", shell=True) ksft_eq(nc.stdout.strip(), "ping")
tools/testing/selftests/net/lib/py/utils.py +4 -2
@@ -42,15 +42,17 @@ import time class bkg(cmd): - def __init__(self, comm, shell=True, fail=True, ns=None, host=None): + def __init__(self, comm, shell=True, fail=True, ns=None, host=None, + exit_wait=False): super().__init__(comm, background=True, shell=shell, fail=fail, ns=ns, host=host) + self.terminate = not exit_wait def __enter__(self): return self def __exit__(self, ex_type, ex_value, ex_tb): - return self.process() + return self.process(terminate=self.terminate) def ip(args, json=None, ns=None, host=None):
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 6 lines · On Wed, 17 Apr 2024 16:11:38 -0700 you wrote:
Hi!
Implement support for tests which require access to a remote system / endpoint which can generate traffic.
This series concludes the "groundwork" for upstream driver tests.
Quoted 8 lines
I wanted to support the three models which came up in discussions:
- SW testing with netdevsim
- "local" testing with two ports on the same system in a loopback
- "remote" testing via SSH
so there is a tiny bit of an abstraction which wraps up how "remote" commands are executed. Otherwise hopefully there's nothing surprising.[...]
Here is the summary with links:
- [net-next,v3,1/8] selftests: net: fix counting totals when some checks fail
https://git.kernel.org/netdev/net-next/c/655614ea2bd3 - [net-next,v3,2/8] selftests: net: set the exit code correctly in Python tests https://git.kernel.org/netdev/net-next/c/4fa6bd4b33ac
- [net-next,v3,3/8] selftests: drv-net: define endpoint structures
(no matching commit)
- [net-next,v3,4/8] selftests: drv-net: factor out parsing of the env
(no matching commit)
- [net-next,v3,5/8] selftests: drv-net: construct environment for running tests which require an endpoint (no matching commit)
- [net-next,v3,6/8] selftests: drv-net: add a trivial ping test
(no matching commit)
- [net-next,v3,7/8] selftests: net: support matching cases by name prefix
(no matching commit)
- [net-next,v3,8/8] selftests: drv-net: add a TCP ping test case (and useful helpers) (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html