selftests: net: groundwork for YNL-based tests
Subsystems (4)
Jakub Kicinski
maintainerv3 cover selftests: net: groundwork for YNL-based tests
Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under tools/net. Add the scaffolding which allows writing tests using this library.
The "scaffolding" is needed because the library lives under tools/net and uses YAML files from under Documentation/. So we need a small amount of glue code to find those things and add them to TEST_FILES.
This series adds both a basic SW sanity test and driver test which can be run against netdevsim or a real device. When I develop core code I usually test with netdevsim, then a real device, and then a backport to Meta's kernel. Because of the lack of integration, until now I had to throw away the (YNL-based) test script and netdevsim code.
Running tests in tree directly:
$ ./tools/testing/selftests/net/nl_netdev.py KTAP version 1 1..2 ok 1 nl_netdev.empty_check ok 2 nl_netdev.lo_check # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
in tree via make:
$ make -C tools/testing/selftests/ TARGETS=net \ TEST_PROGS=nl_netdev.py TEST_GEN_PROGS="" run_tests [ ... ]
and installed externally, all seem to work:
$ make -C tools/testing/selftests/ TARGETS=net \ install INSTALL_PATH=/tmp/ksft-net $ /tmp/ksft-net/run_kselftest.sh -t net:nl_netdev.py [ ... ]
For driver tests I followed the lead of net/forwarding and get the device name from env and/or a config file.
v3:
- fix up netdevsim C
- various small nits in other patches (see changelog in patches)
v2: https://lore.kernel.org/all/[email protected]/
- don't add to TARGETS, create a deperate variable with deps
- support and use with
- support and use passing arguments to tests
v1: https://lore.kernel.org/all/[email protected]/
Jakub Kicinski (5):
selftests: net: add scaffolding for Netlink tests in Python
selftests: nl_netdev: add a trivial Netlink netdev test
netdevsim: report stats by default, like a real device
selftests: drivers: add scaffolding for Netlink tests in Python
testing: net-drv: add a driver test for stats reporting
drivers/net/netdevsim/ethtool.c | 11 ++ drivers/net/netdevsim/netdev.c | 49 ++++++++ tools/testing/selftests/Makefile | 10 +- tools/testing/selftests/drivers/net/Makefile | 7 ++ .../testing/selftests/drivers/net/README.rst | 30 +++++ .../selftests/drivers/net/lib/py/__init__.py | 17 +++ .../selftests/drivers/net/lib/py/env.py | 52 ++++++++ tools/testing/selftests/drivers/net/stats.py | 86 +++++++++++++ tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/lib/Makefile | 8 ++ .../testing/selftests/net/lib/py/__init__.py | 7 ++ tools/testing/selftests/net/lib/py/consts.py | 9 ++ tools/testing/selftests/net/lib/py/ksft.py | 96 +++++++++++++++ tools/testing/selftests/net/lib/py/nsim.py | 115 ++++++++++++++++++ tools/testing/selftests/net/lib/py/utils.py | 47 +++++++ tools/testing/selftests/net/lib/py/ynl.py | 49 ++++++++ tools/testing/selftests/net/nl_netdev.py | 24 ++++ 17 files changed, 617 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/drivers/net/Makefile create mode 100644 tools/testing/selftests/drivers/net/README.rst create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py create mode 100755 tools/testing/selftests/drivers/net/stats.py create mode 100644 tools/testing/selftests/net/lib/Makefile create mode 100644 tools/testing/selftests/net/lib/py/__init__.py create mode 100644 tools/testing/selftests/net/lib/py/consts.py create mode 100644 tools/testing/selftests/net/lib/py/ksft.py create mode 100644 tools/testing/selftests/net/lib/py/nsim.py create mode 100644 tools/testing/selftests/net/lib/py/utils.py create mode 100644 tools/testing/selftests/net/lib/py/ynl.py create mode 100755 tools/testing/selftests/net/nl_netdev.py
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 1/5 selftests: net: add scaffolding for Netlink tests in Python
Add glue code for accessing the YNL library which lives under tools/net and YAML spec files from under Documentation/. Automatically figure out if tests are run in tree or not. Since we'll want to use this library both from net and drivers/net test targets make the library a target as well, and automatically include it when net or drivers/net are included. Making net/lib a target ensures that we end up with only one copy of it, and saves us some path guessing.
Add a tiny bit of formatting support to be able to output KTAP from the start.
---
v3:
- don't use unnecessary wildcards
v2:
- include the net/lib only in install
- support passing state to tests
- don't apply Path() on what's already a Path()
- fix spacing in Makefile's filter()
- sort imports
--- tools/testing/selftests/Makefile | 9 +- tools/testing/selftests/net/lib/Makefile | 8 ++ .../testing/selftests/net/lib/py/__init__.py | 6 ++ tools/testing/selftests/net/lib/py/consts.py | 9 ++ tools/testing/selftests/net/lib/py/ksft.py | 96 +++++++++++++++++++ tools/testing/selftests/net/lib/py/utils.py | 47 +++++++++ tools/testing/selftests/net/lib/py/ynl.py | 49 ++++++++++ 7 files changed, 223 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/net/lib/Makefile create mode 100644 tools/testing/selftests/net/lib/py/__init__.py create mode 100644 tools/testing/selftests/net/lib/py/consts.py create mode 100644 tools/testing/selftests/net/lib/py/ksft.py create mode 100644 tools/testing/selftests/net/lib/py/utils.py create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
tools/testing/selftests/Makefile +8 -1
@@ -116,6 +116,13 @@ TARGETS += zram TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug +# Networking tests want the net/lib target, include it automatically +ifneq ($(filter net,$(TARGETS)),) +ifeq ($(filter net/lib,$(TARGETS)),) + INSTALL_DEP_TARGETS := net/lib +endif +endif + # User can optionally provide a TARGETS skiplist. By default we skip # BPF since it has cutting edge build time dependencies which require # more effort to install.
@@ -245,7 +252,7 @@ ifdef INSTALL_PATH install -m 744 run_kselftest.sh $(INSTALL_PATH)/ rm -f $(TEST_LIST) @ret=1; \ - for TARGET in $(TARGETS); do \ + for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ $(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET install \ INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \
tools/testing/selftests/net/lib/Makefile new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 + +TEST_FILES := ../../../../../Documentation/netlink/specs +TEST_FILES += ../../../../net/ynl + +TEST_INCLUDES := $(wildcard py/*.py) + +include ../../lib.mk
tools/testing/selftests/net/lib/py/__init__.py new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 + +from .consts import KSRC +from .ksft import * +from .utils import * +from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/consts.py new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 + +import sys +from pathlib import Path + +KSFT_DIR = (Path(__file__).parent / "../../..").resolve() +KSRC = (Path(__file__).parent / "../../../../../..").resolve() + +KSFT_MAIN_NAME = Path(sys.argv[0]).with_suffix("").name
tools/testing/selftests/net/lib/py/ksft.py new file +96 -0
96 lines not on this page · raw message
tools/testing/selftests/net/lib/py/utils.py new file +47 -0
47 lines not on this page · raw message
tools/testing/selftests/net/lib/py/ynl.py new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 2/5 selftests: nl_netdev: add a trivial Netlink netdev test
In reply to Jakub KicinskiAdd a trivial test using YNL.
$ ./tools/testing/selftests/net/nl_netdev.py KTAP version 1 1..2 ok 1 nl_netdev.empty_check ok 2 nl_netdev.lo_check
Instantiate the family once, it takes longer than the test itself.
---
v2:
- pass family as argument
--- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/nl_netdev.py | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100755 tools/testing/selftests/net/nl_netdev.py
tools/testing/selftests/net/Makefile +1 -0
@@ -34,6 +34,7 @@ TEST_PROGS += gre_gso.sh TEST_PROGS += cmsg_so_mark.sh TEST_PROGS += cmsg_time.sh cmsg_ipv6.sh TEST_PROGS += netns-name.sh +TEST_PROGS += nl_netdev.py TEST_PROGS += srv6_end_dt46_l3vpn_test.sh TEST_PROGS += srv6_end_dt4_l3vpn_test.sh TEST_PROGS += srv6_end_dt6_l3vpn_test.sh
tools/testing/selftests/net/nl_netdev.py new file +24 -0
@@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +from lib.py import ksft_run, ksft_pr, ksft_eq, ksft_ge, NetdevFamily + + +def empty_check(nf) -> None: + devs = nf.dev_get({}, dump=True) + ksft_ge(len(devs), 1) + + +def lo_check(nf) -> None: + lo_info = nf.dev_get({"ifindex": 1}) + ksft_eq(len(lo_info['xdp-features']), 0) + ksft_eq(len(lo_info['xdp-rx-metadata-features']), 0) + + +def main() -> None: + nf = NetdevFamily() + ksft_run([empty_check, lo_check], args=(nf, )) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 3/5 netdevsim: report stats by default, like a real device
In reply to Jakub KicinskiReal devices should implement qstats. Devices which support pause or FEC configuration should also report the relevant stats.
nsim was missing FEC stats completely, some of the qstats and pause stats required toggling a debugfs knob.
Note that the tests which used pause always initialize the setting so they shouldn't be affected by the different starting value.
---
v3:
- spelling in comment
- make sure we don't break when more queues present
--- drivers/net/netdevsim/ethtool.c | 11 ++++++++ drivers/net/netdevsim/netdev.c | 49 +++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+)
drivers/net/netdevsim/ethtool.c +11 -0
@@ -140,6 +140,13 @@ nsim_set_fecparam(struct net_device *dev, struct ethtool_fecparam *fecparam)return 0 ; } +static void +nsim_get_fec_stats (struct net_device *dev,struct ethtool_fec_stats *fec_stats) +{ + fec_stats->corrected_blocks.total =123 ; + fec_stats->uncorrectable_blocks.total =4 ; +} +static int nsim_get_ts_info (struct net_device *dev,struct ethtool_ts_info *info) {
@@ -163,6 +170,7 @@ static const struct ethtool_ops nsim_ethtool_ops = { .set_channels = nsim_set_channels, .get_fecparam = nsim_get_fecparam, .set_fecparam = nsim_set_fecparam, + .get_fec_stats = nsim_get_fec_stats, .get_ts_info = nsim_get_ts_info, };
@@ -182,6 +190,9 @@ void nsim_ethtool_init(struct netdevsim *ns)nsim_ethtool_ring_init (ns); + ns->ethtool.pauseparam.report_stats_rx = true; + ns->ethtool.pauseparam.report_stats_tx = true; + ns->ethtool.fec.fec =ETHTOOL_FEC_NONE ; ns->ethtool.fec.active_fec =ETHTOOL_FEC_NONE ;
drivers/net/netdevsim/netdev.c +49 -0
67 lines not on this page · raw message
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 4/5 selftests: drivers: add scaffolding for Netlink tests in Python
In reply to Jakub KicinskiAdd drivers/net as a target for mixed-use tests.
The setup is expected to work similarly to the forwarding tests. Since we only need one interface (unlike forwarding tests) read the target device name from NETIF. If not present we'll try to run the test against netdevsim.
---
v3:
- spelling
- remove some fields
v2:
- mark regex as r""
- add support of using env as with
--- tools/testing/selftests/Makefile | 3 +- tools/testing/selftests/drivers/net/Makefile | 7 ++ .../testing/selftests/drivers/net/README.rst | 30 +++++ .../selftests/drivers/net/lib/py/__init__.py | 17 +++ .../selftests/drivers/net/lib/py/env.py | 52 ++++++++ .../testing/selftests/net/lib/py/__init__.py | 1 + tools/testing/selftests/net/lib/py/nsim.py | 115 ++++++++++++++++++ 7 files changed, 224 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/drivers/net/Makefile create mode 100644 tools/testing/selftests/drivers/net/README.rst create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py create mode 100644 tools/testing/selftests/net/lib/py/nsim.py
tools/testing/selftests/Makefile +2 -1
@@ -17,6 +17,7 @@ TARGETS += devices TARGETS += dmabuf-heaps TARGETS += drivers/dma-buf TARGETS += drivers/s390x/uvdevice +TARGETS += drivers/net TARGETS += drivers/net/bonding TARGETS += drivers/net/team TARGETS += dt
@@ -117,7 +118,7 @@ TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug # Networking tests want the net/lib target, include it automatically -ifneq ($(filter net,$(TARGETS)),) +ifneq ($(filter net drivers/net,$(TARGETS)),) ifeq ($(filter net/lib,$(TARGETS)),) INSTALL_DEP_TARGETS := net/lib endif
tools/testing/selftests/drivers/net/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 + +TEST_INCLUDES := $(wildcard lib/py/*.py) + +TEST_PROGS := stats.py + +include ../../lib.mk
tools/testing/selftests/drivers/net/README.rst new file +30 -0
@@ -0,0 +1,30 @@ +Running tests +============= + +Tests are executed within kselftest framework like any other tests. +By default tests execute against software drivers such as netdevsim. +All tests must support running against a real device (SW-only tests +should instead be placed in net/ or drivers/net/netdevsim, HW-only +tests in drivers/net/hw). + +Set appropriate variables to point the tests at a real device. + +Variables +========= + +Variables can be set in the environment or by creating a net.config +file in the same directory as this README file. Example:: + + $ NETIF=eth0 ./some_test.sh + +or:: + + $ cat tools/testing/selftests/drivers/net/net.config + # Variable set in a file + NETIF=eth0 + +NETIF +~~~~~ + +Name of the netdevice against which the test should be executed. +When empty or not set software devices will be used.
tools/testing/selftests/drivers/net/lib/py/__init__.py new file +17 -0
@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0 + +import sys +from pathlib import Path + +KSFT_DIR = (Path(__file__).parent / "../../../..").resolve() + +try: + sys.path.append(KSFT_DIR.as_posix()) + from net.lib.py import * +except ModuleNotFoundError as e: + ksft_pr("Failed importing `net` library from kernel sources") + ksft_pr(str(e)) + ktap_result(True, comment="SKIP") + sys.exit(4) + +from .env import *
tools/testing/selftests/drivers/net/lib/py/env.py new file +52 -0
52 lines not on this page · raw message
tools/testing/selftests/net/lib/py/__init__.py +1 -0
@@ -2,5 +2,6 @@ from .consts import KSRC from .ksft import * +from .nsim import * from .utils import * from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/nsim.py new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v3 5/5 testing: net-drv: add a driver test for stats reporting
In reply to Jakub KicinskiAdd a very simple test to make sure drivers report expected stats. Drivers which implement FEC or pause configuration should report relevant stats. Qstats must be reported, at least packet and byte counts, and they must match total device stats.
Tested with netdevsim, bnxt, in-tree and installed.
---
v2:
- pass cfg as argument
- use with
--- tools/testing/selftests/drivers/net/stats.py | 86 ++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/stats.py
tools/testing/selftests/drivers/net/stats.py new file +86 -0
@@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +from lib.py import ksft_run, ksft_in, ksft_true, KsftSkipEx, KsftXfailEx +from lib.py import EthtoolFamily, NetdevFamily, RtnlFamily, NlError +from lib.py import NetDrvEnv + +ethnl = EthtoolFamily() +netfam = NetdevFamily() +rtnl = RtnlFamily() + + +def check_pause(cfg) -> None: + global ethnl + + try: + ethnl.pause_get({"header": {"dev-index": cfg.ifindex}}) + except NlError as e: + if e.error == 95: + raise KsftXfailEx("pause not supported by the device") + raise + + data = ethnl.pause_get({"header": {"dev-index": cfg.ifindex, + "flags": {'stats'}}}) + ksft_true(data['stats'], "driver does not report stats") + + +def check_fec(cfg) -> None: + global ethnl + + try: + ethnl.fec_get({"header": {"dev-index": cfg.ifindex}}) + except NlError as e: + if e.error == 95: + raise KsftXfailEx("FEC not supported by the device") + raise + + data = ethnl.fec_get({"header": {"dev-index": cfg.ifindex, + "flags": {'stats'}}}) + ksft_true(data['stats'], "driver does not report stats") + + +def pkt_byte_sum(cfg) -> None: + global netfam, rtnl + + def get_qstat(test): + global netfam + stats = netfam.qstats_get({}, dump=True) + if stats: + for qs in stats: + if qs["ifindex"]== test.ifindex: + return qs + + qstat = get_qstat(cfg) + if qstat is None: + raise KsftSkipEx("qstats not supported by the device") + + for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']: + ksft_in(key, qstat, "Drivers should always report basic keys") + + # Compare stats, rtnl stats and qstats must match, + # but the interface may be up, so do a series of dumps + # each time the more "recent" stats must be higher or same. + def stat_cmp(rstat, qstat): + for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']: + if rstat[key] != qstat[key]: + return rstat[key] - qstat[key] + return 0 + + for _ in range(10): + rtstat = rtnl.getlink({"ifi-index": cfg.ifindex})['stats'] + if stat_cmp(rtstat, qstat) < 0: + raise Exception("RTNL stats are lower, fetched later") + qstat = get_qstat(cfg) + if stat_cmp(rtstat, qstat) > 0: + raise Exception("Qstats are lower, fetched later") + + +def main() -> None: + with NetDrvEnv(__file__) as cfg: + ksft_run([check_pause, check_fec, pkt_byte_sum], + args=(cfg, )) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.44.0
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Quoted 6 lines · On Thu, 4 Apr 2024 19:45:21 -0700 you wrote:
Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under
tools/net. Add the scaffolding which allows writing tests using this library.
Quoted 6 lines
The "scaffolding" is needed because the library lives under
tools/netand uses YAML files from underDocumentation/. So we need a small amount of glue code to find those things and add them toTEST_FILES.[...]
Here is the summary with links:
- [net-next,v3,1/5] selftests: net: add scaffolding for Netlink tests in Python
https://git.kernel.org/netdev/net-next/c/b86761ff6374 - [net-next,v3,2/5] selftests: nl_netdev: add a trivial Netlink netdev test https://git.kernel.org/netdev/net-next/c/796c8c7fd257 - [net-next,v3,3/5] netdevsim: report stats by default, like a real device https://git.kernel.org/netdev/net-next/c/f216306bfb60 - [net-next,v3,4/5] selftests: drivers: add scaffolding for Netlink tests in Python https://git.kernel.org/netdev/net-next/c/b4db9f840283 - [net-next,v3,5/5] testing: net-drv: add a driver test for stats reporting https://git.kernel.org/netdev/net-next/c/f0e6c86e4bab
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html