Skip to contentSkip to conversation

selftests: drv-net: rss_ctx: add tests for RSS contexts

PATCH In review Jakub Kicinski 5 participants 16 messages Jun 26, 2024 – Jun 27, 2024

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v3 cover selftests: drv-net: rss_ctx: add tests for RSS contexts

Add a few tests exercising RSS context API.
In addition to basic sanity checks, tests add RSS contexts, n-tuple rule to direct traffic to them (based on dst port), and qstats to make sure traffic landed where we expected.

v2 adds a test for removing contexts out of order. When testing bnxt - either the new test or running more tests after the overlap test makes the device act strangely. To the point where it may start giving out ntuple IDs of 0 for all rules..

Ed, could you try the tests with your device?

  $ export NETIF=eth0 REMOTE_...
  $ ./drivers/net/hw/rss_ctx.py
  KTAP version 1
  1..8
  ok 1 rss_ctx.test_rss_key_indir
  ok 2 rss_ctx.test_rss_context
  ok 3 rss_ctx.test_rss_context4
  # Increasing queue count 44 -> 66
  # Failed to create context 32, trying to test what we got
  ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32
  ok 5 rss_ctx.test_rss_context_overlap
  ok 6 rss_ctx.test_rss_context_overlap2
  # .. sprays traffic like a headless chicken ..
  not ok 7 rss_ctx.test_rss_context_out_of_order
  ok 8 rss_ctx.test_rss_context4_create_with_cfg
  # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0

v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]

Jakub Kicinski (4):
selftests: drv-net: try to check if port is in use
selftests: drv-net: add helper to wait for HW stats to sync
selftests: drv-net: add ability to wait for at least N packets to load
gen
selftests: drv-net: rss_ctx: add tests for RSS configuration and
contexts

 .../testing/selftests/drivers/net/hw/Makefile |   1 +
 .../selftests/drivers/net/hw/rss_ctx.py       | 383 ++++++++++++++++++
 .../selftests/drivers/net/lib/py/env.py       |  19 +-
 .../selftests/drivers/net/lib/py/load.py      |  37 +-
 tools/testing/selftests/net/lib/py/ksft.py    |   5 +
 tools/testing/selftests/net/lib/py/utils.py   |  27 +-
 6 files changed, 457 insertions(+), 15 deletions(-)
 create mode 100755 tools/testing/selftests/drivers/net/hw/rss_ctx.py
Signature · 1 line
2.45.2
author maintainer
#1 lore raw

PATCH v3 1/4 selftests: drv-net: try to check if port is in use

We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.

---
v3:
- use errno.EADDRINUSE (Breno)
v2:
- remove v4 check (Willem)
- update comment (David, Przemek)
- cap the iterations (Przemek)

---
 tools/testing/selftests/net/lib/py/utils.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
tools/testing/selftests/net/lib/py/utils.py +13 -2
index 0540ea24921d..dd9d2b9f2b20 100644
@@ -1,8 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
+import errno
 import json as _json
 import random
 import re
+import socket
 import subprocess
 import time
 
@@ -79,9 +81,18 @@ import time
 
 def rand_port():
     """
-    Get unprivileged port, for now just random, one day we may decide to check if used.
+    Get a random unprivileged port, try to make sure it's not already used.
     """
-    return random.randint(10000, 65535)
+    for _ in range(1000):
+        port = random.randint(10000, 65535)
+        try:
+            with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as s:
+                s.bind(("", port))
+            return port
+        except OSError as e:
+            if e.errno != errno.EADDRINUSE:
+                raise
+    raise Exception("Can't find any free unprivileged port")
 
 
 def wait_port_listen(port, proto="tcp", ns=None, host=None, sleep=0.005, deadline=5):
Signature · 1 line
2.45.2
Reviewed-by on v3 1/4
#2 lore raw

Re: [PATCH net-next v3 1/4] selftests: drv-net: try to check if port is in use

Quoted 6 lines · Jakub Kicinski

We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.

Reviewed-by on v3 1/4
#3 lore raw

Re: [PATCH net-next v3 1/4] selftests: drv-net: try to check if port is in use

In reply to Jakub Kicinski
Quoted 6 lines · Jakub Kicinski

We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.

author maintainer
#4 lore raw

PATCH v3 2/4 selftests: drv-net: add helper to wait for HW stats to sync

In reply to Jakub Kicinski

Some devices DMA stats to the host periodically. Add a helper which can wait for that to happen, based on frequency reported by the driver in ethtool.

---
v3:
- use get() for stats-block-usecs (Petr)
- fix double space (Petr)
v2:
- sleep for 25ms on top of the driver DMA period
(and remove confusing comment)

---
 .../selftests/drivers/net/lib/py/env.py       | 19 ++++++++++++++++++-
 tools/testing/selftests/net/lib/py/utils.py   |  4 ++++
 2 files changed, 22 insertions(+), 1 deletion(-)
tools/testing/selftests/drivers/net/lib/py/env.py +18 -1
index edcedd7bffab..a5e800b8f103 100644
@@ -1,9 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
 import os
+import time
 from pathlib import Path
 from lib.py import KsftSkipEx, KsftXfailEx
-from lib.py import cmd, ip
+from lib.py import cmd, ethtool, ip
 from lib.py import NetNS, NetdevSimDev
 from .remote import Remote
 
@@ -82,6 +83,8 @@ from .remote import Remote
 
         self.env = _load_env_file(src_path)
 
+        self._stats_settle_time = None
+
         # Things we try to destroy
         self.remote = None
         # These are for local testing state
@@ -222,3 +225,17 @@ from .remote import Remote
         if remote:
             if not self._require_cmd(comm, "remote"):
                 raise KsftSkipEx("Test requires (remote) command: " + comm)
+
+    def wait_hw_stats_settle(self):
+        """
+        Wait for HW stats to become consistent, some devices DMA HW stats
+        periodically so events won't be reflected until next sync.
+        Good drivers will tell us via ethtool what their sync period is.
+        """
+        if self._stats_settle_time is None:
+            data = ethtool("-c " + self.ifname, json=True)[0]
+
+            self._stats_settle_time = 0.025 + \
+                data.get('stats-block-usecs', 0) / 1000 / 1000
+
+        time.sleep(self._stats_settle_time)
tools/testing/selftests/net/lib/py/utils.py +4 -0
index dd9d2b9f2b20..a6c87bfe532f 100644
@@ -79,6 +79,10 @@ import time
     return tool('ip', args, json=json, host=host)
 
 
+def ethtool(args, json=None, ns=None, host=None):
+    return tool('ethtool', args, json=json, ns=ns, host=host)
+
+
 def rand_port():
     """
     Get a random unprivileged port, try to make sure it's not already used.
Signature · 1 line
2.45.2
author maintainer
#7 lore raw

PATCH v3 3/4 selftests: drv-net: add ability to wait for at least N packets to load gen

In reply to Jakub Kicinski

Teach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.

---
v3:
- rename pkt_start / pkt_end once more (Willem)
v2:
- add comment that pps or pkt_cnt are mutually exclusive (David)
- rename variables (David)

---
 .../selftests/drivers/net/lib/py/load.py      | 30 ++++++++++++++-----
 1 file changed, 23 insertions(+), 7 deletions(-)
tools/testing/selftests/drivers/net/lib/py/load.py +23 -7
index abdb677bdb1c..f6bc57eb304a 100644
@@ -18,15 +18,31 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
                                  background=True, host=env.remote)
 
         # Wait for traffic to ramp up
-        pkt = ip("-s link show dev " + env.ifname, json=True)[0]["stats64"]["rx"]["packets"]
+        if not self._wait_pkts(pps=1000):
+            self.stop(verbose=True)
+            raise Exception("iperf3 traffic did not ramp up")
+
+    def _wait_pkts(self, pkt_cnt=None, pps=None):
+        """
+        Wait until we've seen pkt_cnt or until traffic ramps up to pps.
+        Only one of pkt_cnt or pss can be specified.
+        """
+        pkt_start = ip("-s link show dev " + self.env.ifname, json=True)[0]["stats64"]["rx"]["packets"]
         for _ in range(50):
             time.sleep(0.1)
-            now = ip("-s link show dev " + env.ifname, json=True)[0]["stats64"]["rx"]["packets"]
-            if now - pkt > 1000:
-                return
-            pkt = now
-        self.stop(verbose=True)
-        raise Exception("iperf3 traffic did not ramp up")
+            pkt_now = ip("-s link show dev " + self.env.ifname, json=True)[0]["stats64"]["rx"]["packets"]
+            if pps:
+                if pkt_now - pkt_start > pps / 10:
+                    return True
+                pkt_start = pkt_now
+            elif pkt_cnt:
+                if pkt_now - pkt_start > pkt_cnt:
+                    return True
+        return False
+
+    def wait_pkts_and_stop(self, pkt_cnt):
+        failed = not self._wait_pkts(pkt_cnt=pkt_cnt)
+        self.stop(verbose=failed)
 
     def stop(self, verbose=None):
         self._iperf_client.process(terminate=True)
Signature · 1 line
2.45.2
Reviewed-by on v3 3/4
#8 lore raw

Re: [PATCH net-next v3 3/4] selftests: drv-net: add ability to wait for at least N packets to load gen

Quoted 7 lines · Jakub Kicinski

Teach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.

Reviewed-by on v3 3/4
#9 lore raw

Re: [PATCH net-next v3 3/4] selftests: drv-net: add ability to wait for at least N packets to load gen

In reply to Jakub Kicinski
Quoted 7 lines · Jakub Kicinski

Teach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.

author maintainer
#10 lore raw

PATCH v3 4/4 selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts

In reply to Jakub Kicinski

Add tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.

  $ export NETIF=eth0 REMOTE_...
  $ ./drivers/net/hw/rss_ctx.py
  KTAP version 1
  1..8
  ok 1 rss_ctx.test_rss_key_indir
  ok 2 rss_ctx.test_rss_context
  ok 3 rss_ctx.test_rss_context4
  # Increasing queue count 44 -> 66
  # Failed to create context 32, trying to test what we got
  ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32
  ok 5 rss_ctx.test_rss_context_overlap
  ok 6 rss_ctx.test_rss_context_overlap2
  # .. sprays traffic like a headless chicken ..
  not ok 7 rss_ctx.test_rss_context_out_of_order
  ok 8 rss_ctx.test_rss_context4_create_with_cfg
  # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0

Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.

---
v3:
- use ksft_lt() to match comment (Petr)
- rewrite the OOO removal to make it more robust (Petr)
v2:
- ensure right queue count for each test (David)
- check driver has ntuple filters before starting (Willem)
- add test for creating contexts with indir table specified (Ed)
- fix ksft_lt (Ed)
- query and validate indirection tables of non-first context (Ed)
- test traffic steering vs OOO context removal (Ed)
- make sure overlap test deletes all rules, 0 is a valid ntuple ID, so we can't do "if ntuple: remove()"

---
 .../testing/selftests/drivers/net/hw/Makefile |   1 +
 .../selftests/drivers/net/hw/rss_ctx.py       | 383 ++++++++++++++++++
 .../selftests/drivers/net/lib/py/load.py      |   7 +-
 tools/testing/selftests/net/lib/py/ksft.py    |   5 +
 tools/testing/selftests/net/lib/py/utils.py   |   8 +-
 5 files changed, 399 insertions(+), 5 deletions(-)
 create mode 100755 tools/testing/selftests/drivers/net/hw/rss_ctx.py
tools/testing/selftests/drivers/net/hw/Makefile +1 -0
index 4933d045ab66..c9f2f48fc30f 100644
@@ -11,6 +11,7 @@ TEST_PROGS = \
 	hw_stats_l3_gre.sh \
 	loopback.sh \
 	pp_alloc_fail.py \
+	rss_ctx.py \
 	#
 
 TEST_FILES := \
tools/testing/selftests/drivers/net/hw/rss_ctx.py new file +383 -0

383 lines not on this page · raw message

tools/testing/selftests/drivers/net/lib/py/load.py +4 -3
index f6bc57eb304a..d9c10613ae67 100644
@@ -5,13 +5,14 @@ import time
 from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
 
 class GenerateTraffic:
-    def __init__(self, env):
+    def __init__(self, env, port=None):
         env.require_cmd("iperf3", remote=True)
 
         self.env = env
 
-        port = rand_port()
-        self._iperf_server = cmd(f"iperf3 -s -p {port}", background=True)
+        if port is None:
+            port = rand_port()
+        self._iperf_server = cmd(f"iperf3 -s -1 -p {port}", background=True)
         wait_port_listen(port)
         time.sleep(0.1)
         self._iperf_client = cmd(f"iperf3 -c {env.addr} -P 16 -p {port} -t 86400",
tools/testing/selftests/net/lib/py/ksft.py +5 -0
index 4769b4eb1ea1..45ffe277d94a 100644
@@ -57,6 +57,11 @@ KSFT_RESULT_ALL = True
         _fail("Check failed", a, "<", b, comment)
 
 
+def ksft_lt(a, b, comment=""):
+    if a >= b:
+        _fail("Check failed", a, ">", b, comment)
+
+
 class ksft_raises:
     def __init__(self, expected_type):
         self.exception = None
tools/testing/selftests/net/lib/py/utils.py +6 -2
index a6c87bfe532f..405aa510aaf2 100644
@@ -9,6 +9,10 @@ import subprocess
 import time
 
 
+class CmdExitFailure(Exception):
+    pass
+
+
 class cmd:
     def __init__(self, comm, shell=True, fail=True, ns=None, background=False, host=None, timeout=5):
         if ns:
@@ -43,8 +47,8 @@ import time
         if self.proc.returncode != 0 and fail:
             if len(stderr) > 0 and stderr[-1] == "\n":
                 stderr = stderr[:-1]
-            raise Exception("Command failed: %s\nSTDOUT: %s\nSTDERR: %s" %
-                            (self.proc.args, stdout, stderr))
+            raise CmdExitFailure("Command failed: %s\nSTDOUT: %s\nSTDERR: %s" %
+                                 (self.proc.args, stdout, stderr))
 
 
 class bkg(cmd):
Signature · 1 line
2.45.2
Reviewed-by on v3 4/4
#11 lore raw

Re: [PATCH net-next v3 4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts

Quoted 20 lines · Jakub Kicinski

Add tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.

  $ export NETIF=eth0 REMOTE_...
  $ ./drivers/net/hw/rss_ctx.py
  KTAP version 1
  1..8
  ok 1 rss_ctx.test_rss_key_indir
  ok 2 rss_ctx.test_rss_context
  ok 3 rss_ctx.test_rss_context4
  # Increasing queue count 44 -> 66
  # Failed to create context 32, trying to test what we got
  ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32
  ok 5 rss_ctx.test_rss_context_overlap
  ok 6 rss_ctx.test_rss_context_overlap2
  # .. sprays traffic like a headless chicken ..
  not ok 7 rss_ctx.test_rss_context_out_of_order
  ok 8 rss_ctx.test_rss_context4_create_with_cfg
  # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
Quoted 4 lines

Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.

Reviewed-by on v3 4/4
#12 lore raw

Re: [PATCH net-next v3 4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts

In reply to Jakub Kicinski
Quoted 20 lines · Jakub Kicinski

Add tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.

  $ export NETIF=eth0 REMOTE_...
  $ ./drivers/net/hw/rss_ctx.py
  KTAP version 1
  1..8
  ok 1 rss_ctx.test_rss_key_indir
  ok 2 rss_ctx.test_rss_context
  ok 3 rss_ctx.test_rss_context4
  # Increasing queue count 44 -> 66
  # Failed to create context 32, trying to test what we got
  ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32
  ok 5 rss_ctx.test_rss_context_overlap
  ok 6 rss_ctx.test_rss_context_overlap2
  # .. sprays traffic like a headless chicken ..
  not ok 7 rss_ctx.test_rss_context_out_of_order
  ok 8 rss_ctx.test_rss_context4_create_with_cfg
  # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
Quoted 4 lines

Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.

tools/testing/selftests/net/lib/py/ksft.py KSFT_RESULT_ALL = True · line 62

Quoted 2 lines
tools/testing/selftests/net/lib/py/ksft.py +2 -0
+    if a >= b:
+        _fail("Check failed", a, ">", b, comment)

Should the argument be ">=" too?

#13 lore raw
In reply to Jakub Kicinski
Quoted 1 line · Jakub Kicinski

Ed, could you try the tests with your device?

Don't seem to be able to get them to run:

# Exception| Traceback (most recent call last):
# Exception|   File "/home/ecree/kern/linux/tools/testing/selftests/net/lib/py/ksft.py", line 134, in ksft_run
# Exception|     case(*args)
# Exception|   File "./drivers/net/hw/rss_ctx.py", line 70, in test_rss_key_indir
# Exception|     if len(_get_rx_cnts(cfg)) < 2:
# Exception|   File "./drivers/net/hw/rss_ctx.py", line 55, in _get_rx_cnts
# Exception|     data = cfg.netdevnl.qstats_get({"ifindex": cfg.ifindex, "scope": ["queue"]}, dump=True)
# Exception|   File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 1029, in _op
# Exception|     return self._ops(ops)[0]
# Exception|   File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 985, in _ops
# Exception|     raise NlError(nl_msg)
# Exception| net.ynl.lib.ynl.NlError: Netlink error: Operation not supported
# Exception| nl_len = 28 (12) nl_flags = 0x202 nl_type = 3
# Exception| 	error: -95
# Exception| 	extack: {'bad-attr': '.ifindex'}

not ok 1 rss_ctx.test_rss_key_indir

Cursory investigation suggests this is because sfc doesn't
support netdev_stat_ops, we're still living in the bad old days of ethtool -S for our per-queue stats :(
Much as I'd like to fix that, I don't see a prospect of the
folks upstairs carving out time for it any time soon...

-ed

author maintainer
#14 lore raw
Quoted 20 lines · On Wed, 26 Jun 2024 17:50:20 +0100 Edward Cree wrote:
Quoted 1 line · Jakub Kicinski

Ed, could you try the tests with your device?

Don't seem to be able to get them to run:

# Exception| Traceback (most recent call last):
# Exception|   File "/home/ecree/kern/linux/tools/testing/selftests/net/lib/py/ksft.py", line 134, in ksft_run
# Exception|     case(*args)
# Exception|   File "./drivers/net/hw/rss_ctx.py", line 70, in test_rss_key_indir
# Exception|     if len(_get_rx_cnts(cfg)) < 2:
# Exception|   File "./drivers/net/hw/rss_ctx.py", line 55, in _get_rx_cnts
# Exception|     data = cfg.netdevnl.qstats_get({"ifindex": cfg.ifindex, "scope": ["queue"]}, dump=True)
# Exception|   File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 1029, in _op
# Exception|     return self._ops(ops)[0]
# Exception|   File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 985, in _ops
# Exception|     raise NlError(nl_msg)
# Exception| net.ynl.lib.ynl.NlError: Netlink error: Operation not supported
# Exception| nl_len = 28 (12) nl_flags = 0x202 nl_type = 3
# Exception| 	error: -95
# Exception| 	extack: {'bad-attr': '.ifindex'}
Quoted 5 lines

not ok 1 rss_ctx.test_rss_key_indir

Cursory investigation suggests this is because sfc doesn't
support netdev_stat_ops, we're still living in the bad old days of ethtool -S for our per-queue stats :(

Ugh, right, the standard stats turn out to be key in a large number of the tests we end up writing. Perhaps unsurprisingly :(

Fetching stats is done only in _get_rx_cnts(), so you could possibly do a quick hack there to parse ethtool -S, just to test the test. But maybe that'd just lead you to another gap..

Quoted 2 lines

Much as I'd like to fix that, I don't see a prospect of the
folks upstairs carving out time for it any time soon...

Sadness. Makes me worried that they won't care about maintaining the Supported status of the driver either :( (Which is not to say that implementing qstats is a requirement there.)

Anyway, thanks for giving it a go. I'm fairly confident now that the failures I see are a device / driver bug. So we can merge the test and let the various drivers address the gaps.

BTW, while I have you, there are two more bits of work:
- get + dump API via netlink
- set API via netlink
are you planning to work on those?

#15 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 11 lines · On Tue, 25 Jun 2024 18:24:52 -0700 you wrote:

Add a few tests exercising RSS context API.
In addition to basic sanity checks, tests add RSS contexts, n-tuple rule to direct traffic to them (based on dst port), and qstats to make sure traffic landed where we expected.

v2 adds a test for removing contexts out of order. When testing bnxt - either the new test or running more tests after the overlap test makes the device act strangely. To the point where it may start giving out ntuple IDs of 0 for all rules..

[...]

Here is the summary with links:
- [net-next,v3,1/4] selftests: drv-net: try to check if port is in use

    https://git.kernel.org/netdev/net-next/c/8b8fe280155d
  - [net-next,v3,2/4] selftests: drv-net: add helper to wait for HW stats to sync
    https://git.kernel.org/netdev/net-next/c/af8e51644a70
  - [net-next,v3,3/4] selftests: drv-net: add ability to wait for at least N packets to load gen
    https://git.kernel.org/netdev/net-next/c/94fecaa6dcd0

- [net-next,v3,4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts (no matching commit)

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html