Skip to contentSkip to conversation

selftests: net: groundwork for YNL-based tests

PATCH Superseded Jakub Kicinski 4 participants 19 messages Apr 3, 2024 – Apr 5, 2024

Subsystems (6)

v2 of 3 Superseded 2 of 7 applied Latest is v3 →

lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v2 cover selftests: net: groundwork for YNL-based tests

Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under tools/net. Add the scaffolding which allows writing tests using this library.

The "scaffolding" is needed because the library lives under tools/net and uses YAML files from under Documentation/. So we need a small amount of glue code to find those things and add them to TEST_FILES.

This series adds both a basic SW sanity test and driver test which can be run against netdevsim or a real device. When I develop core code I usually test with netdevsim, then a real device, and then a backport to Meta's kernel. Because of the lack of integration, until now I had to throw away the (YNL-based) test script and netdevsim code.

Running tests in tree directly:

 $ ./tools/testing/selftests/net/nl_netdev.py
 KTAP version 1
 1..2
 ok 1 nl_netdev.empty_check
 ok 2 nl_netdev.lo_check
 # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0

in tree via make:

 $ make -C tools/testing/selftests/ TARGETS=net \
	TEST_PROGS=nl_netdev.py TEST_GEN_PROGS="" run_tests
  [ ... ]

and installed externally, all seem to work:

 $ make -C tools/testing/selftests/ TARGETS=net \
	install INSTALL_PATH=/tmp/ksft-net
 $ /tmp/ksft-net/run_kselftest.sh -t net:nl_netdev.py
  [ ... ]

For driver tests I followed the lead of net/forwarding and get the device name from env and/or a config file.

v2: (see patches for minor changes)
- don't add to TARGETS, create a deperate variable with deps
- support and use with
- support and use passing arguments to tests
v1: https://lore.kernel.org/all/[email protected]/

Jakub Kicinski (7):
netlink: specs: define ethtool header flags
tools: ynl: copy netlink error to NlError
selftests: net: add scaffolding for Netlink tests in Python
selftests: nl_netdev: add a trivial Netlink netdev test
netdevsim: report stats by default, like a real device
selftests: drivers: add scaffolding for Netlink tests in Python
testing: net-drv: add a driver test for stats reporting

 Documentation/netlink/specs/ethtool.yaml      |   6 +
 drivers/net/netdevsim/ethtool.c               |  11 ++
 drivers/net/netdevsim/netdev.c                |  45 +++++++
 tools/net/ynl/lib/ynl.py                      |   3 +-
 tools/testing/selftests/Makefile              |  10 +-
 tools/testing/selftests/drivers/net/Makefile  |   7 ++
 .../testing/selftests/drivers/net/README.rst  |  30 +++++
 .../selftests/drivers/net/lib/py/__init__.py  |  17 +++
 .../selftests/drivers/net/lib/py/env.py       |  52 ++++++++
 tools/testing/selftests/drivers/net/stats.py  |  86 +++++++++++++
 tools/testing/selftests/net/Makefile          |   1 +
 tools/testing/selftests/net/lib/Makefile      |   8 ++
 .../testing/selftests/net/lib/py/__init__.py  |   7 ++
 tools/testing/selftests/net/lib/py/consts.py  |   9 ++
 tools/testing/selftests/net/lib/py/ksft.py    |  96 ++++++++++++++
 tools/testing/selftests/net/lib/py/nsim.py    | 118 ++++++++++++++++++
 tools/testing/selftests/net/lib/py/utils.py   |  47 +++++++
 tools/testing/selftests/net/lib/py/ynl.py     |  49 ++++++++
 tools/testing/selftests/net/nl_netdev.py      |  24 ++++
 19 files changed, 624 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/drivers/net/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/README.rst
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py
 create mode 100755 tools/testing/selftests/drivers/net/stats.py
 create mode 100644 tools/testing/selftests/net/lib/Makefile
 create mode 100644 tools/testing/selftests/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/net/lib/py/consts.py
 create mode 100644 tools/testing/selftests/net/lib/py/ksft.py
 create mode 100644 tools/testing/selftests/net/lib/py/nsim.py
 create mode 100644 tools/testing/selftests/net/lib/py/utils.py
 create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
 create mode 100755 tools/testing/selftests/net/nl_netdev.py
Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH v2 1/7 netlink: specs: define ethtool header flags

When interfacing with the ethtool commands it's handy to be able to use the names of the flags. Example:

    ethnl.pause_get({"header": {"dev-index": cfg.ifindex,
                                "flags": {'stats'}}})

Note that not all commands accept all the flags, but the meaning of the bits does not change command to command.

---
v2:
- make sure we don't try to code gen enum (add enum-name:)

---
 Documentation/netlink/specs/ethtool.yaml | 6 ++++++
 1 file changed, 6 insertions(+)
Documentation/netlink/specs/ethtool.yaml +6 -0
index 197208f419dc..d0e4a47e0f21 100644
@@ -16,6 +16,11 @@ doc: Partial family for Ethtool Netlink.
     name: stringset
     type: enum
     entries: []
+  -
+    name: header-flags
+    enum-name:
+    type: flags
+    entries: [ compact-bitsets, omit-reply, stats ]
 
 attribute-sets:
   -
@@ -30,6 +35,7 @@ doc: Partial family for Ethtool Netlink.
       -
         name: flags
         type: u32
+        enum: header-flags
 
   -
     name: bitset-bit
Signature · 1 line
2.44.0
Reviewed-by on v2 1/7
#2 lore raw

Re: [PATCH net-next v2 1/7] netlink: specs: define ethtool header flags

Quoted 11 lines · Jakub Kicinski

When interfacing with the ethtool commands it's handy to be able to use the names of the flags. Example:

    ethnl.pause_get({"header": {"dev-index": cfg.ifindex,
                                "flags": {'stats'}}})

Note that not all commands accept all the flags, but the meaning of the bits does not change command to command.

author maintainer
#3 lore raw

PATCH v2 2/7 tools: ynl: copy netlink error to NlError

In reply to Jakub Kicinski

Typing e.nl_msg.error when processing exception is a bit tedious and counter-intuitive. Set a local .error member to the positive value of the netlink level error.

---
CC: [email protected]
CC: [email protected]

---
 tools/net/ynl/lib/ynl.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/net/ynl/lib/ynl.py +2 -1
index 82d3c98067aa..b30210f537f7 100644
@@ -100,9 +100,10 @@ from .nlspec import SpecFamily
 class NlError(Exception):
   def __init__(self, nl_msg):
     self.nl_msg = nl_msg
+    self.error = -nl_msg.error
 
   def __str__(self):
-    return f"Netlink error: {os.strerror(-self.nl_msg.error)}\n{self.nl_msg}"
+    return f"Netlink error: {os.strerror(self.error)}\n{self.nl_msg}"
 
 
 class ConfigError(Exception):
Signature · 1 line
2.44.0
author maintainer
#5 lore raw

PATCH v2 3/7 selftests: net: add scaffolding for Netlink tests in Python

In reply to Jakub Kicinski

Add glue code for accessing the YNL library which lives under tools/net and YAML spec files from under Documentation/. Automatically figure out if tests are run in tree or not. Since we'll want to use this library both from net and drivers/net test targets make the library a target as well, and automatically include it when net or drivers/net are included. Making net/lib a target ensures that we end up with only one copy of it, and saves us some path guessing.

Add a tiny bit of formatting support to be able to output KTAP from the start.

---
v2:
- include the net/lib only in install
- support passing state to tests
- don't apply Path() on what's already a Path()
- fix spacing in Makefile's filter()
- sort imports

---
 tools/testing/selftests/Makefile              |  9 +-
 tools/testing/selftests/net/lib/Makefile      |  8 ++
 .../testing/selftests/net/lib/py/__init__.py  |  6 ++
 tools/testing/selftests/net/lib/py/consts.py  |  9 ++
 tools/testing/selftests/net/lib/py/ksft.py    | 96 +++++++++++++++++++
 tools/testing/selftests/net/lib/py/utils.py   | 47 +++++++++
 tools/testing/selftests/net/lib/py/ynl.py     | 49 ++++++++++
 7 files changed, 223 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/lib/Makefile
 create mode 100644 tools/testing/selftests/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/net/lib/py/consts.py
 create mode 100644 tools/testing/selftests/net/lib/py/ksft.py
 create mode 100644 tools/testing/selftests/net/lib/py/utils.py
 create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
tools/testing/selftests/Makefile +8 -1
index e1504833654d..f533eb7054fe 100644
@@ -116,6 +116,13 @@ TARGETS += zram
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
+# Networking tests want the net/lib target, include it automatically
+ifneq ($(filter net,$(TARGETS)),)
+ifeq ($(filter net/lib,$(TARGETS)),)
+	INSTALL_DEP_TARGETS := net/lib
+endif
+endif
+
 # User can optionally provide a TARGETS skiplist.  By default we skip
 # BPF since it has cutting edge build time dependencies which require
 # more effort to install.
@@ -245,7 +252,7 @@ ifdef INSTALL_PATH
 	install -m 744 run_kselftest.sh $(INSTALL_PATH)/
 	rm -f $(TEST_LIST)
 	@ret=1;	\
-	for TARGET in $(TARGETS); do \
+	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET install \
 				INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \
tools/testing/selftests/net/lib/Makefile new file +8 -0
new file mode 100644
index 000000000000..5730682aeffb
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_FILES := ../../../../../Documentation/netlink/s*
+TEST_FILES += ../../../../net/*
+
+TEST_INCLUDES := $(wildcard py/*.py)
+
+include ../../lib.mk
tools/testing/selftests/net/lib/py/__init__.py new file +6 -0
new file mode 100644
index 000000000000..7823b5c1f8d7
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+
+from .consts import KSRC
+from .ksft import *
+from .utils import *
+from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/consts.py new file +9 -0
new file mode 100644
index 000000000000..f518ce79d82c
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import sys
+from pathlib import Path
+
+KSFT_DIR = (Path(__file__).parent / "../../..").resolve()
+KSRC = (Path(__file__).parent / "../../../../../..").resolve()
+
+KSFT_MAIN_NAME = Path(sys.argv[0]).with_suffix("").name
tools/testing/selftests/net/lib/py/ksft.py new file +96 -0

96 lines not on this page · raw message

tools/testing/selftests/net/lib/py/utils.py new file +47 -0

47 lines not on this page · raw message

tools/testing/selftests/net/lib/py/ynl.py new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
2.44.0
Reviewed-by on v2 3/7
#6 lore raw

Re: [PATCH net-next v2 3/7] selftests: net: add scaffolding for Netlink tests in Python

Quoted 72 lines · Jakub Kicinski

Add glue code for accessing the YNL library which lives under tools/net and YAML spec files from under Documentation/. Automatically figure out if tests are run in tree or not. Since we'll want to use this library both from net and drivers/net test targets make the library a target as well, and automatically include it when net or drivers/net are included. Making net/lib a target ensures that we end up with only one copy of it, and saves us some path guessing.

Add a tiny bit of formatting support to be able to output KTAP from the start.

---
v2:
- include the net/lib only in install
- support passing state to tests
- don't apply Path() on what's already a Path()
- fix spacing in Makefile's filter()
- sort imports

---
 tools/testing/selftests/Makefile              |  9 +-
 tools/testing/selftests/net/lib/Makefile      |  8 ++
 .../testing/selftests/net/lib/py/__init__.py  |  6 ++
 tools/testing/selftests/net/lib/py/consts.py  |  9 ++
 tools/testing/selftests/net/lib/py/ksft.py    | 96 +++++++++++++++++++
 tools/testing/selftests/net/lib/py/utils.py   | 47 +++++++++
 tools/testing/selftests/net/lib/py/ynl.py     | 49 ++++++++++
 7 files changed, 223 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/lib/Makefile
 create mode 100644 tools/testing/selftests/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/net/lib/py/consts.py
 create mode 100644 tools/testing/selftests/net/lib/py/ksft.py
 create mode 100644 tools/testing/selftests/net/lib/py/utils.py
 create mode 100644 tools/testing/selftests/net/lib/py/ynl.py
tools/testing/selftests/Makefile +8 -1
index e1504833654d..f533eb7054fe 100644
@@ -116,6 +116,13 @@ TARGETS += zram
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
+# Networking tests want the net/lib target, include it automatically
+ifneq ($(filter net,$(TARGETS)),)
+ifeq ($(filter net/lib,$(TARGETS)),)
+	INSTALL_DEP_TARGETS := net/lib
+endif
+endif
+
 # User can optionally provide a TARGETS skiplist.  By default we skip
 # BPF since it has cutting edge build time dependencies which require
 # more effort to install.
@@ -245,7 +252,7 @@ ifdef INSTALL_PATH
 	install -m 744 run_kselftest.sh $(INSTALL_PATH)/
 	rm -f $(TEST_LIST)
 	@ret=1;	\
-	for TARGET in $(TARGETS); do \
+	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET install \
 				INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \
tools/testing/selftests/net/lib/Makefile new file +0 -0
new file mode 100644
index 000000000000..5730682aeffb

tools/testing/selftests/net/lib/Makefile line 3

Quoted 3 lines
tools/testing/selftests/net/lib/Makefile new file +3 -0
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_FILES := ../../../../../Documentation/netlink/s*

Not sure I understand the motivation for the wildcard. Currently this matches Documentation/netlink/specs. Do you expect everything that starts with s to be a testfile?

tools/testing/selftests/net/lib/Makefile line 4

Quoted 1 line
tools/testing/selftests/net/lib/Makefile new file +1 -0
+TEST_FILES += ../../../../net/*

Likewise this -- it's just tools/net/ynl? Will everything that's there be a testfile?

tools/testing/selftests/net/lib/Makefile line 8

Quoted 3 lines
tools/testing/selftests/net/lib/Makefile new file +3 -0
+TEST_INCLUDES := $(wildcard py/*.py)
+
+include ../../lib.mk

Other than that it looks OK.

author maintainer
#7 lore raw

PATCH v2 4/7 selftests: nl_netdev: add a trivial Netlink netdev test

In reply to Jakub Kicinski

Add a trivial test using YNL.

  $ ./tools/testing/selftests/net/nl_netdev.py
  KTAP version 1
  1..2
  ok 1 nl_netdev.empty_check
  ok 2 nl_netdev.lo_check

Instantiate the family once, it takes longer than the test itself.

---
v2:
- pass family as argument

---
 tools/testing/selftests/net/Makefile     |  1 +
 tools/testing/selftests/net/nl_netdev.py | 24 ++++++++++++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100755 tools/testing/selftests/net/nl_netdev.py
tools/testing/selftests/net/Makefile +1 -0
index cb418a2346bc..5e34c93aa51b 100644
@@ -34,6 +34,7 @@ TEST_PROGS += gre_gso.sh
 TEST_PROGS += cmsg_so_mark.sh
 TEST_PROGS += cmsg_time.sh cmsg_ipv6.sh
 TEST_PROGS += netns-name.sh
+TEST_PROGS += nl_netdev.py
 TEST_PROGS += srv6_end_dt46_l3vpn_test.sh
 TEST_PROGS += srv6_end_dt4_l3vpn_test.sh
 TEST_PROGS += srv6_end_dt6_l3vpn_test.sh
tools/testing/selftests/net/nl_netdev.py new file +24 -0
new file mode 100755
index 000000000000..2b8b488fb419
@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+from lib.py import ksft_run, ksft_pr, ksft_eq, ksft_ge, NetdevFamily
+
+
+def empty_check(nf) -> None:
+    devs = nf.dev_get({}, dump=True)
+    ksft_ge(len(devs), 1)
+
+
+def lo_check(nf) -> None:
+    lo_info = nf.dev_get({"ifindex": 1})
+    ksft_eq(len(lo_info['xdp-features']), 0)
+    ksft_eq(len(lo_info['xdp-rx-metadata-features']), 0)
+
+
+def main() -> None:
+    nf = NetdevFamily()
+    ksft_run([empty_check, lo_check], args=(nf, ))
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.44.0
Reviewed-by on v2 4/7
#8 lore raw

Re: [PATCH net-next v2 4/7] selftests: nl_netdev: add a trivial Netlink netdev test

Quoted 11 lines · Jakub Kicinski

Add a trivial test using YNL.

  $ ./tools/testing/selftests/net/nl_netdev.py
  KTAP version 1
  1..2
  ok 1 nl_netdev.empty_check
  ok 2 nl_netdev.lo_check

Instantiate the family once, it takes longer than the test itself.

author maintainer
#9 lore raw

PATCH v2 5/7 netdevsim: report stats by default, like a real device

In reply to Jakub Kicinski

Real devices should implement qstats. Devices which support pause or FEC configuration should also report the relevant stats.

nsim was missing FEC stats completely, some of the qstats and pause stats required toggling a debugfs knob.

Note that the tests which used pause always initialize the setting so they shouldn't be affected by the different starting value.

---
 drivers/net/netdevsim/ethtool.c | 11 ++++++++
 drivers/net/netdevsim/netdev.c  | 45 +++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
drivers/net/netdevsim/ethtool.c +11 -0
index bd546d4d26c6..3f9c9327f149 100644
@@ -140,6 +140,13 @@ nsim_set_fecparam(struct net_device *dev, struct ethtool_fecparam *fecparam)
 	return 0;
 }
 
+static void
+nsim_get_fec_stats(struct net_device *dev, struct ethtool_fec_stats *fec_stats)
+{
+	fec_stats->corrected_blocks.total = 123;
+	fec_stats->uncorrectable_blocks.total = 4;
+}
+
 static int nsim_get_ts_info(struct net_device *dev,
 			    struct ethtool_ts_info *info)
 {
@@ -163,6 +170,7 @@ static const struct ethtool_ops nsim_ethtool_ops = {
 	.set_channels			= nsim_set_channels,
 	.get_fecparam			= nsim_get_fecparam,
 	.set_fecparam			= nsim_set_fecparam,
+	.get_fec_stats			= nsim_get_fec_stats,
 	.get_ts_info			= nsim_get_ts_info,
 };
 
@@ -182,6 +190,9 @@ void nsim_ethtool_init(struct netdevsim *ns)
 
 	nsim_ethtool_ring_init(ns);
 
+	ns->ethtool.pauseparam.report_stats_rx = true;
+	ns->ethtool.pauseparam.report_stats_tx = true;
+
 	ns->ethtool.fec.fec = ETHTOOL_FEC_NONE;
 	ns->ethtool.fec.active_fec = ETHTOOL_FEC_NONE;
 
drivers/net/netdevsim/netdev.c +45 -0

63 lines not on this page · raw message

Signature · 1 line
2.44.0
Reviewed-by on v2 5/7
#10 lore raw

Re: [PATCH net-next v2 5/7] netdevsim: report stats by default, like a real device

Quoted 10 lines · Jakub Kicinski

Real devices should implement qstats. Devices which support pause or FEC configuration should also report the relevant stats.

nsim was missing FEC stats completely, some of the qstats and pause stats required toggling a debugfs knob.

Note that the tests which used pause always initialize the setting so they shouldn't be affected by the different starting value.

Just:

drivers/net/netdevsim/netdev.c static const struct net_device_ops nsim_vf_netdev_ops = { · line 334

Quoted 5 lines
drivers/net/netdevsim/netdev.c +1 -0
@@ -330,6 +331,49 @@ static const struct net_device_ops nsim_vf_netdev_ops = {
 	.ndo_set_features	= nsim_set_features,
 };
 
+/* We don't have true par-queue stats, yet, so do some random fakery here. */

per

drivers/net/netdevsim/netdev.c static const struct net_device_ops nsim_vf_netdev_ops = { · line 342

Quoted 8 lines
drivers/net/netdevsim/netdev.c +8 -0
+static void nsim_get_queue_stats_rx(struct net_device *dev, int idx,
+				    struct netdev_queue_stats_rx *stats)
+{
+	struct rtnl_link_stats64 rtstats = {};
+
+	nsim_get_stats64(dev, &rtstats);
+
+	stats->packets = rtstats.rx_packets - !!rtstats.rx_packets;

This is just to make sure that per-queue stats are lower than the overall rtstats I presume?

Quoted 13 lines
drivers/net/netdevsim/netdev.c +13 -0
+	stats->bytes = rtstats.rx_bytes;
+}
+
+static void nsim_get_queue_stats_tx(struct net_device *dev, int idx,
+				    struct netdev_queue_stats_tx *stats)
+{
+	struct rtnl_link_stats64 rtstats = {};
+
+	nsim_get_stats64(dev, &rtstats);
+
+	stats->packets = rtstats.tx_packets - !!rtstats.tx_packets;
+	stats->bytes = rtstats.tx_bytes;
+}
author maintainer
#11 lore raw

Re: [PATCH net-next v2 5/7] netdevsim: report stats by default, like a real device

Quoted 4 lines · On Thu, 4 Apr 2024 12:40:04 +0200 Petr Machata wrote:

drivers/net/netdevsim/netdev.c static const struct net_device_ops nsim_vf_netdev_ops = { · line 342

Quoted 1 line
drivers/net/netdevsim/netdev.c +1 -0
+	stats->packets = rtstats.rx_packets - !!rtstats.rx_packets;  

This is just to make sure that per-queue stats are lower than the overall rtstats I presume?

Yes, to fake some "non-queue" stats.

maintainer
#12 lore raw

Re: [PATCH net-next v2 5/7] netdevsim: report stats by default, like a real device

In reply to Jakub Kicinski
Quoted 78 lines · Jakub Kicinski

Real devices should implement qstats. Devices which support pause or FEC configuration should also report the relevant stats.

nsim was missing FEC stats completely, some of the qstats and pause stats required toggling a debugfs knob.

Note that the tests which used pause always initialize the setting so they shouldn't be affected by the different starting value.

---
 drivers/net/netdevsim/ethtool.c | 11 ++++++++
 drivers/net/netdevsim/netdev.c  | 45 +++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
drivers/net/netdevsim/ethtool.c +11 -0
index bd546d4d26c6..3f9c9327f149 100644
@@ -140,6 +140,13 @@ nsim_set_fecparam(struct net_device *dev, struct ethtool_fecparam *fecparam)
 	return 0;
 }
 
+static void
+nsim_get_fec_stats(struct net_device *dev, struct ethtool_fec_stats *fec_stats)
+{
+	fec_stats->corrected_blocks.total = 123;
+	fec_stats->uncorrectable_blocks.total = 4;
+}
+
 static int nsim_get_ts_info(struct net_device *dev,
 			    struct ethtool_ts_info *info)
 {
@@ -163,6 +170,7 @@ static const struct ethtool_ops nsim_ethtool_ops = {
 	.set_channels			= nsim_set_channels,
 	.get_fecparam			= nsim_get_fecparam,
 	.set_fecparam			= nsim_set_fecparam,
+	.get_fec_stats			= nsim_get_fec_stats,
 	.get_ts_info			= nsim_get_ts_info,
 };
 
@@ -182,6 +190,9 @@ void nsim_ethtool_init(struct netdevsim *ns)
 
 	nsim_ethtool_ring_init(ns);
 
+	ns->ethtool.pauseparam.report_stats_rx = true;
+	ns->ethtool.pauseparam.report_stats_tx = true;
+
 	ns->ethtool.fec.fec = ETHTOOL_FEC_NONE;
 	ns->ethtool.fec.active_fec = ETHTOOL_FEC_NONE;
 
drivers/net/netdevsim/netdev.c +12 -0
index 8330bc0bcb7e..096ac0abbc02 100644
@@ -19,6 +19,7 @@
 #include <linux/module.h>
 #include <linux/netdevice.h>
 #include <linux/slab.h>
+#include <net/netdev_queues.h>
 #include <net/netlink.h>
 #include <net/pkt_cls.h>
 #include <net/rtnetlink.h>
@@ -330,6 +331,49 @@ static const struct net_device_ops nsim_vf_netdev_ops = {
 	.ndo_set_features	= nsim_set_features,
 };
 
+/* We don't have true par-queue stats, yet, so do some random fakery here. */
+static void nsim_get_queue_stats_rx(struct net_device *dev, int idx,
+				    struct netdev_queue_stats_rx *stats)
+{
+	struct rtnl_link_stats64 rtstats = {};
+
+	nsim_get_stats64(dev, &rtstats);
+
+	stats->packets = rtstats.rx_packets - !!rtstats.rx_packets;
+	stats->bytes = rtstats.rx_bytes;
+}

drivers/net/netdevsim/netdev.c static const struct net_device_ops nsim_vf_netdev_ops = { · line 354

Quoted 10 lines
drivers/net/netdevsim/netdev.c +10 -0
+
+static void nsim_get_queue_stats_tx(struct net_device *dev, int idx,
+				    struct netdev_queue_stats_tx *stats)
+{
+	struct rtnl_link_stats64 rtstats = {};
+
+	nsim_get_stats64(dev, &rtstats);
+
+	stats->packets = rtstats.tx_packets - !!rtstats.tx_packets;
+	stats->bytes = rtstats.tx_bytes;

It looks the stats will not be self-consistent with multiple queues enabled.

What about zeroing 'stats' when idx > 0 ?

/P

author maintainer
#13 lore raw

Re: [PATCH net-next v2 5/7] netdevsim: report stats by default, like a real device

Quoted 9 lines · On Thu, 04 Apr 2024 15:40:33 +0200 Paolo Abeni wrote:

drivers/net/netdevsim/netdev.c static const struct net_device_ops nsim_vf_netdev_ops = { · line 354

Quoted 4 lines
drivers/net/netdevsim/netdev.c +4 -0
+	nsim_get_stats64(dev, &rtstats);
+
+	stats->packets = rtstats.tx_packets - !!rtstats.tx_packets;
+	stats->bytes = rtstats.tx_bytes;  

It looks the stats will not be self-consistent with multiple queues enabled.

What about zeroing 'stats' when idx > 0 ?

Ah, good catch. We do allow allocating multiple queues, even tho we don't do anything with them on the driver side, yet..

author maintainer
#14 lore raw

PATCH v2 6/7 selftests: drivers: add scaffolding for Netlink tests in Python

In reply to Jakub Kicinski

Add drivers/net as a target for mixed-use tests.
The setup is expected to work similarly to the forwarding tests. Since we only need one interface (unlike forwarding tests) read the target device name from NETIF. If not present we'll try to run the test against netdevsim.

---
v2:
- mark regex as r""
- add support of using env as with

---
 tools/testing/selftests/Makefile              |   3 +-
 tools/testing/selftests/drivers/net/Makefile  |   7 ++
 .../testing/selftests/drivers/net/README.rst  |  30 +++++
 .../selftests/drivers/net/lib/py/__init__.py  |  17 +++
 .../selftests/drivers/net/lib/py/env.py       |  52 ++++++++
 .../testing/selftests/net/lib/py/__init__.py  |   1 +
 tools/testing/selftests/net/lib/py/nsim.py    | 118 ++++++++++++++++++
 7 files changed, 227 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/drivers/net/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/README.rst
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/env.py
 create mode 100644 tools/testing/selftests/net/lib/py/nsim.py
tools/testing/selftests/Makefile +2 -1
index f533eb7054fe..6dab886d6f7a 100644
@@ -17,6 +17,7 @@ TARGETS += devices
 TARGETS += dmabuf-heaps
 TARGETS += drivers/dma-buf
 TARGETS += drivers/s390x/uvdevice
+TARGETS += drivers/net
 TARGETS += drivers/net/bonding
 TARGETS += drivers/net/team
 TARGETS += dt
@@ -117,7 +118,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net,$(TARGETS)),)
+ifneq ($(filter net drivers/net,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
tools/testing/selftests/drivers/net/Makefile new file +7 -0
new file mode 100644
index 000000000000..379cdb1960a7
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_INCLUDES := $(wildcard lib/py/*.py)
+
+TEST_PROGS := stats.py
+
+include ../../lib.mk
tools/testing/selftests/drivers/net/README.rst new file +30 -0
new file mode 100644
index 000000000000..5ef7c417d431
@@ -0,0 +1,30 @@
+Running tests
+=============
+
+Tests are executed within kselftest framework like any other tests.
+By default tests execute against software drivers such as netdevsim.
+All tests must support running against a real device (SW-only tests
+should instead be placed in net/ or drivers/net/netdevsim, HW-only
+tests in drivers/net/hw).
+
+Set appropriate variables to point the tests at a real device.
+
+Variables
+=========
+
+Variables can be set in the environment or by creating a net.config
+file in the same directory as this README file. Example::
+
+  $ NETIF=eth0 ./some_test.sh
+
+or::
+
+  $ cat tools/testing/selftests/drivers/net/net.config
+  # Variable set in a file
+  NETIF=eth0
+
+NETIF
+~~~~~
+
+Name of the netdevice against which the test should be executed.
+When empty or not set software devices will be used.
tools/testing/selftests/drivers/net/lib/py/__init__.py new file +17 -0
new file mode 100644
index 000000000000..4653dffcd962
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import sys
+from pathlib import Path
+
+KSFT_DIR = (Path(__file__).parent / "../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+    from net.lib.py import *
+except ModuleNotFoundError as e:
+    ksft_pr("Failed importing `net` library from kernel sources")
+    ksft_pr(str(e))
+    ktap_result(True, comment="SKIP")
+    sys.exit(4)
+
+from .env import *
tools/testing/selftests/drivers/net/lib/py/env.py new file +52 -0

52 lines not on this page · raw message

tools/testing/selftests/net/lib/py/__init__.py +1 -0
index 7823b5c1f8d7..ded7102df18a 100644
@@ -2,5 +2,6 @@
 
 from .consts import KSRC
 from .ksft import *
+from .nsim import *
 from .utils import *
 from .ynl import NlError, YnlFamily, EthtoolFamily, NetdevFamily, RtnlFamily
tools/testing/selftests/net/lib/py/nsim.py new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.44.0
Reviewed-by on v2 6/7
#15 lore raw

Re: [PATCH net-next v2 6/7] selftests: drivers: add scaffolding for Netlink tests in Python

Quoted 7 lines · Jakub Kicinski

Add drivers/net as a target for mixed-use tests.
The setup is expected to work similarly to the forwarding tests. Since we only need one interface (unlike forwarding tests) read the target device name from NETIF. If not present we'll try to run the test against netdevsim.

However:

Quoted 26 lines
tools/testing/selftests/net/lib/py/nsim.py new file +20 -0
new file mode 100644
index 000000000000..25ae0d081788
@@ -0,0 +1,118 @@
+# SPDX-License-Identifier: GPL-2.0
+
+import json
+import os
+import random
+import re
+import time
+from .utils import cmd, ip
+
+
+class NetdevSim:
+    """
+    Class for netdevsim netdevice and its attributes.
+    """
+
+    def __init__(self, nsimdev, port_index, ifname, ns=None):
+        # In case udev renamed the netdev to according to new schema,
+        # check if the name matches the port_index.
+        nsimnamere = re.compile(r"eni\d+np(\d+)")
+        match = nsimnamere.match(ifname)

tools/testing/selftests/net/lib/py/nsim.py line 30

Quoted 10 lines
tools/testing/selftests/net/lib/py/nsim.py new file +10 -0
+        if match and int(match.groups()[0]) != port_index + 1:
+            raise Exception("netdevice name mismatches the expected one")
+
+        self.ifname = ifname
+        self.nsimdev = nsimdev
+        self.port_index = port_index
+        self.ns = ns
+        self.dfs_dir = "%s/ports/%u/" % (nsimdev.dfs_dir, port_index)
+        ret = ip("-j link show dev %s" % ifname, ns=ns)
+        self.dev = json.loads(ret.stdout)[0]

I don't think self.ifname, .ns, .dfs_dir, .dev are actually used outside of this function.

Quoted 12 lines
tools/testing/selftests/net/lib/py/nsim.py new file +12 -0
+
+    def dfs_write(self, path, val):
+        self.nsimdev.dfs_write(f'ports/{self.port_index}/' + path, val)
+
+
+class NetdevSimDev:
+    """
+    Class for netdevsim bus device and its attributes.
+    """
+    @staticmethod
+    def ctrl_write(path, val):
+        fullpath = os.path.join("/sys/bus/netdevsim/", path)

tools/testing/selftests/net/lib/py/nsim.py line 52

Quoted 10 lines
tools/testing/selftests/net/lib/py/nsim.py new file +10 -0
+        with open(fullpath, "w") as f:
+            f.write(val)
+
+    def dfs_write(self, path, val):
+        fullpath = os.path.join(f"/sys/kernel/debug/netdevsim/netdevsim{self.addr}/", path)
+        with open(fullpath, "w") as f:
+            f.write(val)
+
+    def __init__(self, port_count=1, ns=None):
+        # nsim will spawn in init_net, we'll set to actual ns once we switch it the.sre

the.sre?

author maintainer
#16 lore raw

Re: [PATCH net-next v2 6/7] selftests: drivers: add scaffolding for Netlink tests in Python

Quoted 16 lines · On Thu, 4 Apr 2024 12:42:24 +0200 Petr Machata wrote:
Quoted 5 lines
tools/testing/selftests/net/lib/py/nsim.py new file +5 -0
+    def __init__(self, nsimdev, port_index, ifname, ns=None):
+        # In case udev renamed the netdev to according to new schema,
+        # check if the name matches the port_index.
+        nsimnamere = re.compile(r"eni\d+np(\d+)")
+        match = nsimnamere.match(ifname)

tools/testing/selftests/net/lib/py/nsim.py line 30

Quoted 10 lines
tools/testing/selftests/net/lib/py/nsim.py new file +10 -0
+        if match and int(match.groups()[0]) != port_index + 1:
+            raise Exception("netdevice name mismatches the expected one")
+
+        self.ifname = ifname
+        self.nsimdev = nsimdev
+        self.port_index = port_index
+        self.ns = ns
+        self.dfs_dir = "%s/ports/%u/" % (nsimdev.dfs_dir, port_index)
+        ret = ip("-j link show dev %s" % ifname, ns=ns)
+        self.dev = json.loads(ret.stdout)[0]  
Quoted 2 lines

I don't think self.ifname, .ns, .dfs_dir, .dev are actually used outside of this function.

Right, one of the "further down the line" tests need these. I'll remove for now.

Quoted 15 lines
Quoted 15 lines
tools/testing/selftests/net/lib/py/nsim.py new file +15 -0
+    def dfs_write(self, path, val):
+        self.nsimdev.dfs_write(f'ports/{self.port_index}/' + path, val)
+
+
+class NetdevSimDev:
+    """
+    Class for netdevsim bus device and its attributes.
+    """
+    @staticmethod
+    def ctrl_write(path, val):
+        fullpath = os.path.join("/sys/bus/netdevsim/", path)
+        with open(fullpath, "w") as f:
+            f.write(val)
+
+    def dfs_write(self, path, val):
Quoted 8 lines

tools/testing/selftests/net/lib/py/nsim.py line 52

Quoted 6 lines
tools/testing/selftests/net/lib/py/nsim.py new file +6 -0
+        fullpath = os.path.join(f"/sys/kernel/debug/netdevsim/netdevsim{self.addr}/", path)
+        with open(fullpath, "w") as f:
+            f.write(val)
+
+    def __init__(self, port_count=1, ns=None):
+        # nsim will spawn in init_net, we'll set to actual ns once we switch it the.sre  

the.sre?

Ha, must have started typing with focus on the wrong window. Good it wasn't my password :D

author maintainer
#17 lore raw

PATCH v2 7/7 testing: net-drv: add a driver test for stats reporting

In reply to Jakub Kicinski

Add a very simple test to make sure drivers report expected stats. Drivers which implement FEC or pause configuration should report relevant stats. Qstats must be reported, at least packet and byte counts, and they must match total device stats.

Tested with netdevsim, bnxt, in-tree and installed.

---
v2:
- pass cfg as argument
- use with

---
 tools/testing/selftests/drivers/net/stats.py | 86 ++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/stats.py
tools/testing/selftests/drivers/net/stats.py new file +86 -0
new file mode 100755
index 000000000000..5a9d4e56b28b
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+from lib.py import ksft_run, ksft_in, ksft_true, KsftSkipEx, KsftXfailEx
+from lib.py import EthtoolFamily, NetdevFamily, RtnlFamily, NlError
+from lib.py import NetDrvEnv
+
+ethnl = EthtoolFamily()
+netfam = NetdevFamily()
+rtnl = RtnlFamily()
+
+
+def check_pause(cfg) -> None:
+    global ethnl
+
+    try:
+        ethnl.pause_get({"header": {"dev-index": cfg.ifindex}})
+    except NlError as e:
+        if e.error == 95:
+            raise KsftXfailEx("pause not supported by the device")
+        raise
+
+    data = ethnl.pause_get({"header": {"dev-index": cfg.ifindex,
+                                       "flags": {'stats'}}})
+    ksft_true(data['stats'], "driver does not report stats")
+
+
+def check_fec(cfg) -> None:
+    global ethnl
+
+    try:
+        ethnl.fec_get({"header": {"dev-index": cfg.ifindex}})
+    except NlError as e:
+        if e.error == 95:
+            raise KsftXfailEx("FEC not supported by the device")
+        raise
+
+    data = ethnl.fec_get({"header": {"dev-index": cfg.ifindex,
+                                     "flags": {'stats'}}})
+    ksft_true(data['stats'], "driver does not report stats")
+
+
+def pkt_byte_sum(cfg) -> None:
+    global netfam, rtnl
+
+    def get_qstat(test):
+        global netfam
+        stats = netfam.qstats_get({}, dump=True)
+        if stats:
+            for qs in stats:
+                if qs["ifindex"]== test.ifindex:
+                    return qs
+
+    qstat = get_qstat(cfg)
+    if qstat is None:
+        raise KsftSkipEx("qstats not supported by the device")
+
+    for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']:
+        ksft_in(key, qstat, "Drivers should always report basic keys")
+
+    # Compare stats, rtnl stats and qstats must match,
+    # but the interface may be up, so do a series of dumps
+    # each time the more "recent" stats must be higher or same.
+    def stat_cmp(rstat, qstat):
+        for key in ['tx-packets', 'tx-bytes', 'rx-packets', 'rx-bytes']:
+            if rstat[key] != qstat[key]:
+                return rstat[key] - qstat[key]
+        return 0
+
+    for _ in range(10):
+        rtstat = rtnl.getlink({"ifi-index": cfg.ifindex})['stats']
+        if stat_cmp(rtstat, qstat) < 0:
+            raise Exception("RTNL stats are lower, fetched later")
+        qstat = get_qstat(cfg)
+        if stat_cmp(rtstat, qstat) > 0:
+            raise Exception("Qstats are lower, fetched later")
+
+
+def main() -> None:
+    with NetDrvEnv(__file__) as cfg:
+        ksft_run([check_pause, check_fec, pkt_byte_sum],
+                 args=(cfg, ))
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.44.0
#18 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 6 lines · On Tue, 2 Apr 2024 19:34:19 -0700 you wrote:

Currently the options for writing networking tests are C, bash or some mix of the two. YAML/Netlink gives us the ability to easily interface with Netlink in higher level laguages. In particular, there is a Python library already available in tree, under tools/net. Add the scaffolding which allows writing tests using this library.

Quoted 6 lines

The "scaffolding" is needed because the library lives under tools/net and uses YAML files from under Documentation/. So we need a small amount of glue code to find those things and add them to TEST_FILES.

[...]

Here is the summary with links:
- [net-next,v2,1/7] netlink: specs: define ethtool header flags

    https://git.kernel.org/netdev/net-next/c/1d056bf9a4c1
  - [net-next,v2,2/7] tools: ynl: copy netlink error to NlError
    https://git.kernel.org/netdev/net-next/c/b269d2b4a523

- [net-next,v2,3/7] selftests: net: add scaffolding for Netlink tests in Python (no matching commit)
- [net-next,v2,4/7] selftests: nl_netdev: add a trivial Netlink netdev test (no matching commit)
- [net-next,v2,5/7] netdevsim: report stats by default, like a real device (no matching commit)
- [net-next,v2,6/7] selftests: drivers: add scaffolding for Netlink tests in Python (no matching commit)
- [net-next,v2,7/7] testing: net-drv: add a driver test for stats reporting (no matching commit)

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html