Skip to contentSkip to conversation

selftests: net: groundwork for YNL-based tests

PATCH Applied Jakub Kicinski 2 participants 7 messages Apr 5, 2024 – Apr 8, 2024

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v3 cover selftests: net: groundwork for YNL-based tests

Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under tools/net. Add the scaffolding which allows writing tests using this library.

The "scaffolding" is needed because the library lives under tools/net and uses YAML files from under Documentation/. So we need a small amount of glue code to find those things and add them to TEST_FILES.

This series adds both a basic SW sanity test and driver test which can be run against netdevsim or a real device. When I develop core code I usually test with netdevsim, then a real device, and then a backport to Meta's kernel. Because of the lack of integration, until now I had to throw away the (YNL-based) test script and netdevsim code.

Running tests in tree directly:

 $ ./tools/testing/selftests/net/nl_netdev.py
 KTAP version 1
 1..2
 ok 1 nl_netdev.empty_check
 ok 2 nl_netdev.lo_check
 # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0

in tree via make:

 $ make -C tools/testing/selftests/ TARGETS=net \
	TEST_PROGS=nl_netdev.py TEST_GEN_PROGS="" run_tests
  [ ... ]

and installed externally, all seem to work:

 $ make -C tools/testing/selftests/ TARGETS=net \
	install INSTALL_PATH=/tmp/ksft-net
 $ /tmp/ksft-net/run_kselftest.sh -t net:nl_netdev.py
  [ ... ]

For driver tests I followed the lead of net/forwarding and get the device name from env and/or a config file.

v3:
- fix up netdevsim C
- various small nits in other patches (see changelog in patches)
v2: https://lore.kernel.org/all/[email protected]/
- don't add to TARGETS, create a deperate variable with deps
- support and use with
- support and use passing arguments to tests
v1: https://lore.kernel.org/all/[email protected]/

Jakub Kicinski (5):
selftests: net: add scaffolding for Netlink tests in Python
selftests: nl_netdev: add a trivial Netlink netdev test
netdevsim: report stats by default, like a real device
selftests: drivers: add scaffolding for Netlink tests in Python
testing: net-drv: add a driver test for stats reporting

 drivers/net/netdevsim/ethtool.c               |  11 ++
 drivers/net/netdevsim/netdev.c                |  49 ++++++++
 tools/testing/selftests/Makefile              |  10 +-
 tools/testing/selftests/drivers/net/Makefile  |   7 ++
 .../testing/selftests/drivers/net/README.rst  |  30 +++++
 .../selftests/drivers/net/lib/py/__init__.py  |  17 +++
 .../selftests/drivers/net/lib/py/env.py       |  52 ++++++++
 tools/testing/selftests/drivers/net/stats.py  |  86 +++++++++++++
 tools/testing/selftests/net/Makefile          |   1 +
 tools/testing/selftests/net/lib/Makefile      |   8 ++
 .../testing/selftests/net/lib/py/__init__.py  |   7 ++
 tools/testing/selftests/net/lib/py/consts.py  |   9 ++
 tools/testing/selftests/net/lib/py/ksft.py    |  96 +++++++++++++++
 tools/testing/selftests/net/lib/py/nsim.py    | 115 ++++++++++++++++++
 tools/testing/selftests/net/lib/py/utils.py   |  47 +++++++
 tools/testing/selftests/net/lib/py/ynl.py     |  49 ++++++++
 tools/testing/selftests/net/nl_netdev.py      |  24 ++++
 17 files changed, 617 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/drivers/net/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/README.rst
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py
 create mode 100755 tools/testing/selftests/drivers/net/stats.py
 create mode 100644 tools/testing/selftests/net/lib/Makefile
 create mode 100644 tools/testing/selftests/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/net/lib/py/consts.py
 create mode 100644 tools/testing/selftests/net/lib/py/ksft.py
 create mode 100644 tools/testing/selftests/net/lib/py/nsim.py
 create mode 100644 tools/testing/selftests/net/lib/py/utils.py
 create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
 create mode 100755 tools/testing/selftests/net/nl_netdev.py
Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH v3 1/5 selftests: net: add scaffolding for Netlink tests in Python

Add glue code for accessing the YNL library which lives under tools/net and YAML spec files from under Documentation/. Automatically figure out if tests are run in tree or not. Since we'll want to use this library both from net and drivers/net test targets make the library a target as well, and automatically include it when net or drivers/net are included. Making net/lib a target ensures that we end up with only one copy of it, and saves us some path guessing.

Add a tiny bit of formatting support to be able to output KTAP from the start.

---
v3:
- don't use unnecessary wildcards
v2:
- include the net/lib only in install
- support passing state to tests
- don't apply Path() on what's already a Path()
- fix spacing in Makefile's filter()
- sort imports

---
 tools/testing/selftests/Makefile              |  9 +-
 tools/testing/selftests/net/lib/Makefile      |  8 ++
 .../testing/selftests/net/lib/py/__init__.py  |  6 ++
 tools/testing/selftests/net/lib/py/consts.py  |  9 ++
 tools/testing/selftests/net/lib/py/ksft.py    | 96 +++++++++++++++++++
 tools/testing/selftests/net/lib/py/utils.py   | 47 +++++++++
 tools/testing/selftests/net/lib/py/ynl.py     | 49 ++++++++++
 7 files changed, 223 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/lib/Makefile
 create mode 100644 tools/testing/selftests/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/net/lib/py/consts.py
 create mode 100644 tools/testing/selftests/net/lib/py/ksft.py
 create mode 100644 tools/testing/selftests/net/lib/py/utils.py
 create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
tools/testing/selftests/Makefile +8 -1
index e1504833654d..f533eb7054fe 100644
@@ -116,6 +116,13 @@ TARGETS += zram
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
+# Networking tests want the net/lib target, include it automatically
+ifneq ($(filter net,$(TARGETS)),)
+ifeq ($(filter net/lib,$(TARGETS)),)
+	INSTALL_DEP_TARGETS := net/lib
+endif
+endif
+
 # User can optionally provide a TARGETS skiplist.  By default we skip
 # BPF since it has cutting edge build time dependencies which require
 # more effort to install.
@@ -245,7 +252,7 @@ ifdef INSTALL_PATH
 	install -m 744 run_kselftest.sh $(INSTALL_PATH)/
 	rm -f $(TEST_LIST)
 	@ret=1;	\
-	for TARGET in $(TARGETS); do \
+	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET install \
 				INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \
tools/testing/selftests/net/lib/Makefile new file +8 -0
new file mode 100644
index 000000000000..48557e6250dd
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_FILES := ../../../../../Documentation/netlink/specs
+TEST_FILES += ../../../../net/ynl
+
+TEST_INCLUDES := $(wildcard py/*.py)
+
+include ../../lib.mk
tools/testing/selftests/net/lib/py/__init__.py new file +6 -0
new file mode 100644
index 000000000000..7823b5c1f8d7
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+
+from .consts import KSRC
+from .ksft import *
+from .utils import *
+from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/consts.py new file +9 -0
new file mode 100644
index 000000000000..f518ce79d82c
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import sys
+from pathlib import Path
+
+KSFT_DIR = (Path(__file__).parent / "../../..").resolve()
+KSRC = (Path(__file__).parent / "../../../../../..").resolve()
+
+KSFT_MAIN_NAME = Path(sys.argv[0]).with_suffix("").name
tools/testing/selftests/net/lib/py/ksft.py new file +96 -0

96 lines not on this page · raw message

tools/testing/selftests/net/lib/py/utils.py new file +47 -0

47 lines not on this page · raw message

tools/testing/selftests/net/lib/py/ynl.py new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
2.44.0
author maintainer
#2 lore raw

PATCH v3 2/5 selftests: nl_netdev: add a trivial Netlink netdev test

In reply to Jakub Kicinski

Add a trivial test using YNL.

  $ ./tools/testing/selftests/net/nl_netdev.py
  KTAP version 1
  1..2
  ok 1 nl_netdev.empty_check
  ok 2 nl_netdev.lo_check

Instantiate the family once, it takes longer than the test itself.

---
v2:
- pass family as argument

---
 tools/testing/selftests/net/Makefile     |  1 +
 tools/testing/selftests/net/nl_netdev.py | 24 ++++++++++++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100755 tools/testing/selftests/net/nl_netdev.py
tools/testing/selftests/net/Makefile +1 -0
index cb418a2346bc..5e34c93aa51b 100644
@@ -34,6 +34,7 @@ TEST_PROGS += gre_gso.sh
 TEST_PROGS += cmsg_so_mark.sh
 TEST_PROGS += cmsg_time.sh cmsg_ipv6.sh
 TEST_PROGS += netns-name.sh
+TEST_PROGS += nl_netdev.py
 TEST_PROGS += srv6_end_dt46_l3vpn_test.sh
 TEST_PROGS += srv6_end_dt4_l3vpn_test.sh
 TEST_PROGS += srv6_end_dt6_l3vpn_test.sh
tools/testing/selftests/net/nl_netdev.py new file +24 -0
new file mode 100755
index 000000000000..2b8b488fb419
@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+from lib.py import ksft_run, ksft_pr, ksft_eq, ksft_ge, NetdevFamily
+
+
+def empty_check(nf) -> None:
+    devs = nf.dev_get({}, dump=True)
+    ksft_ge(len(devs), 1)
+
+
+def lo_check(nf) -> None:
+    lo_info = nf.dev_get({"ifindex": 1})
+    ksft_eq(len(lo_info['xdp-features']), 0)
+    ksft_eq(len(lo_info['xdp-rx-metadata-features']), 0)
+
+
+def main() -> None:
+    nf = NetdevFamily()
+    ksft_run([empty_check, lo_check], args=(nf, ))
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.44.0
author maintainer
#3 lore raw

PATCH v3 3/5 netdevsim: report stats by default, like a real device

In reply to Jakub Kicinski

Real devices should implement qstats. Devices which support pause or FEC configuration should also report the relevant stats.

nsim was missing FEC stats completely, some of the qstats and pause stats required toggling a debugfs knob.

Note that the tests which used pause always initialize the setting so they shouldn't be affected by the different starting value.

---
v3:
- spelling in comment
- make sure we don't break when more queues present

---
 drivers/net/netdevsim/ethtool.c | 11 ++++++++
 drivers/net/netdevsim/netdev.c  | 49 +++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)
drivers/net/netdevsim/ethtool.c +11 -0
index bd546d4d26c6..3f9c9327f149 100644
@@ -140,6 +140,13 @@ nsim_set_fecparam(struct net_device *dev, struct ethtool_fecparam *fecparam)
 	return 0;
 }
 
+static void
+nsim_get_fec_stats(struct net_device *dev, struct ethtool_fec_stats *fec_stats)
+{
+	fec_stats->corrected_blocks.total = 123;
+	fec_stats->uncorrectable_blocks.total = 4;
+}
+
 static int nsim_get_ts_info(struct net_device *dev,
 			    struct ethtool_ts_info *info)
 {
@@ -163,6 +170,7 @@ static const struct ethtool_ops nsim_ethtool_ops = {
 	.set_channels			= nsim_set_channels,
 	.get_fecparam			= nsim_get_fecparam,
 	.set_fecparam			= nsim_set_fecparam,
+	.get_fec_stats			= nsim_get_fec_stats,
 	.get_ts_info			= nsim_get_ts_info,
 };
 
@@ -182,6 +190,9 @@ void nsim_ethtool_init(struct netdevsim *ns)
 
 	nsim_ethtool_ring_init(ns);
 
+	ns->ethtool.pauseparam.report_stats_rx = true;
+	ns->ethtool.pauseparam.report_stats_tx = true;
+
 	ns->ethtool.fec.fec = ETHTOOL_FEC_NONE;
 	ns->ethtool.fec.active_fec = ETHTOOL_FEC_NONE;
 
drivers/net/netdevsim/netdev.c +49 -0

67 lines not on this page · raw message

Signature · 1 line
2.44.0
author maintainer
#4 lore raw

PATCH v3 4/5 selftests: drivers: add scaffolding for Netlink tests in Python

In reply to Jakub Kicinski

Add drivers/net as a target for mixed-use tests.
The setup is expected to work similarly to the forwarding tests. Since we only need one interface (unlike forwarding tests) read the target device name from NETIF. If not present we'll try to run the test against netdevsim.

---
v3:
- spelling
- remove some fields
v2:
- mark regex as r""
- add support of using env as with

---
 tools/testing/selftests/Makefile              |   3 +-
 tools/testing/selftests/drivers/net/Makefile  |   7 ++
 .../testing/selftests/drivers/net/README.rst  |  30 +++++
 .../selftests/drivers/net/lib/py/__init__.py  |  17 +++
 .../selftests/drivers/net/lib/py/env.py       |  52 ++++++++
 .../testing/selftests/net/lib/py/__init__.py  |   1 +
 tools/testing/selftests/net/lib/py/nsim.py    | 115 ++++++++++++++++++
 7 files changed, 224 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/drivers/net/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/README.rst
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py
 create mode 100644 tools/testing/selftests/net/lib/py/nsim.py
tools/testing/selftests/Makefile +2 -1
index f533eb7054fe..6dab886d6f7a 100644
@@ -17,6 +17,7 @@ TARGETS += devices
 TARGETS += dmabuf-heaps
 TARGETS += drivers/dma-buf
 TARGETS += drivers/s390x/uvdevice
+TARGETS += drivers/net
 TARGETS += drivers/net/bonding
 TARGETS += drivers/net/team
 TARGETS += dt
@@ -117,7 +118,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net,$(TARGETS)),)
+ifneq ($(filter net drivers/net,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/drivers/net/Makefile new file +7 -0
new file mode 100644
index 000000000000..379cdb1960a7
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_INCLUDES := $(wildcard lib/py/*.py)
+
+TEST_PROGS := stats.py
+
+include ../../lib.mk
tools/testing/selftests/drivers/net/README.rst new file +30 -0
new file mode 100644
index 000000000000..5ef7c417d431
@@ -0,0 +1,30 @@
+Running tests
+=============
+
+Tests are executed within kselftest framework like any other tests.
+By default tests execute against software drivers such as netdevsim.
+All tests must support running against a real device (SW-only tests
+should instead be placed in net/ or drivers/net/netdevsim, HW-only
+tests in drivers/net/hw).
+
+Set appropriate variables to point the tests at a real device.
+
+Variables
+=========
+
+Variables can be set in the environment or by creating a net.config
+file in the same directory as this README file. Example::
+
+  $ NETIF=eth0 ./some_test.sh
+
+or::
+
+  $ cat tools/testing/selftests/drivers/net/net.config
+  # Variable set in a file
+  NETIF=eth0
+
+NETIF
+~~~~~
+
+Name of the netdevice against which the test should be executed.
+When empty or not set software devices will be used.
tools/testing/selftests/drivers/net/lib/py/__init__.py new file +17 -0
new file mode 100644
index 000000000000..4653dffcd962
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import sys
+from pathlib import Path
+
+KSFT_DIR = (Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
+
+from .env import *
tools/testing/selftests/drivers/net/lib/py/env.py new file +52 -0

52 lines not on this page · raw message

tools/testing/selftests/net/lib/py/__init__.py +1 -0
index 7823b5c1f8d7..ded7102df18a 100644
@@ -2,5 +2,6 @@
 
 from .consts import KSRC
 from .ksft import *
+from .nsim import *
 from .utils import *
 from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/nsim.py new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.44.0
author maintainer
#5 lore raw

PATCH v3 5/5 testing: net-drv: add a driver test for stats reporting

In reply to Jakub Kicinski

Add a very simple test to make sure drivers report expected stats. Drivers which implement FEC or pause configuration should report relevant stats. Qstats must be reported, at least packet and byte counts, and they must match total device stats.

Tested with netdevsim, bnxt, in-tree and installed.

---
v2:
- pass cfg as argument
- use with

---
 tools/testing/selftests/drivers/net/stats.py | 86 ++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/stats.py
tools/testing/selftests/drivers/net/stats.py new file +86 -0
new file mode 100755
index 000000000000..5a9d4e56b28b
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+from lib.py import ksft_run, ksft_in, ksft_true, KsftSkipEx, KsftXfailEx
+from lib.py import EthtoolFamily, NetdevFamily, RtnlFamily, NlError
+from lib.py import NetDrvEnv
+
+ethnl = EthtoolFamily()
+netfam = NetdevFamily()
+rtnl = RtnlFamily()
+
+
+def check_pause(cfg) -> None:
+    global ethnl
+
+    try:
+        ethnl.pause_get({"header": {"dev-index": cfg.ifindex}})
+    except NlError as e:
+        if e.error == 95:
+            raise KsftXfailEx("pause not supported by the device")
+        raise
+
+    data = ethnl.pause_get({"header": {"dev-index": cfg.ifindex,
+                                       "flags": {'stats'}}})
+    ksft_true(data['stats'], "driver does not report stats")
+
+
+def check_fec(cfg) -> None:
+    global ethnl
+
+    try:
+        ethnl.fec_get({"header": {"dev-index": cfg.ifindex}})
+    except NlError as e:
+        if e.error == 95:
+            raise KsftXfailEx("FEC not supported by the device")
+        raise
+
+    data = ethnl.fec_get({"header": {"dev-index": cfg.ifindex,
+                                     "flags": {'stats'}}})
+    ksft_true(data['stats'], "driver does not report stats")
+
+
+def pkt_byte_sum(cfg) -> None:
+    global netfam, rtnl
+
+    def get_qstat(test):
+        global netfam
+        stats = netfam.qstats_get({}, dump=True)
+        if stats:
+            for qs in stats:
+                if qs["ifindex"]== test.ifindex:
+                    return qs
+
+    qstat = get_qstat(cfg)
+    if qstat is None:
+        raise KsftSkipEx("qstats not supported by the device")
+
+    for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']:
+        ksft_in(key, qstat, "Drivers should always report basic keys")
+
+    # Compare stats, rtnl stats and qstats must match,
+    # but the interface may be up, so do a series of dumps
+    # each time the more "recent" stats must be higher or same.
+    def stat_cmp(rstat, qstat):
+        for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']:
+            if rstat[key] != qstat[key]:
+                return rstat[key] - qstat[key]
+        return 0
+
+    for _ in range(10):
+        rtstat = rtnl.getlink({"ifi-index": cfg.ifindex})['stats']
+        if stat_cmp(rtstat, qstat) < 0:
+            raise Exception("RTNL stats are lower, fetched later")
+        qstat = get_qstat(cfg)
+        if stat_cmp(rtstat, qstat) > 0:
+            raise Exception("Qstats are lower, fetched later")
+
+
+def main() -> None:
+    with NetDrvEnv(__file__) as cfg:
+        ksft_run([check_pause, check_fec, pkt_byte_sum],
+                 args=(cfg, ))
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.44.0
#6 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Quoted 6 lines · On Thu, 4 Apr 2024 19:45:21 -0700 you wrote:

Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under tools/net. Add the scaffolding which allows writing tests using this library.

Quoted 6 lines

The "scaffolding" is needed because the library lives under tools/net and uses YAML files from under Documentation/. So we need a small amount of glue code to find those things and add them to TEST_FILES.

[...]

Here is the summary with links:
- [net-next,v3,1/5] selftests: net: add scaffolding for Netlink tests in Python

    https://git.kernel.org/netdev/net-next/c/b86761ff6374
  - [net-next,v3,2/5] selftests: nl_netdev: add a trivial Netlink netdev test
    https://git.kernel.org/netdev/net-next/c/796c8c7fd257
  - [net-next,v3,3/5] netdevsim: report stats by default, like a real device
    https://git.kernel.org/netdev/net-next/c/f216306bfb60
  - [net-next,v3,4/5] selftests: drivers: add scaffolding for Netlink tests in Python
    https://git.kernel.org/netdev/net-next/c/b4db9f840283
  - [net-next,v3,5/5] testing: net-drv: add a driver test for stats reporting
    https://git.kernel.org/netdev/net-next/c/f0e6c86e4bab

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html