selftests: drv-net: rss_ctx: add tests for RSS contexts
Subsystems (3)
Jakub Kicinski
maintainerv3 cover selftests: drv-net: rss_ctx: add tests for RSS contexts
Add a few tests exercising RSS context API.
In addition to basic sanity checks, tests add RSS contexts, n-tuple rule to direct traffic to them (based on dst port), and qstats to make sure traffic landed where we expected.
v2 adds a test for removing contexts out of order. When testing bnxt - either the new test or running more tests after the overlap test makes the device act strangely. To the point where it may start giving out ntuple IDs of 0 for all rules..
Ed, could you try the tests with your device?
$ export NETIF=eth0 REMOTE_... $ ./drivers/net/hw/rss_ctx.py KTAP version 1 1..8 ok 1 rss_ctx.test_rss_key_indir ok 2 rss_ctx.test_rss_context ok 3 rss_ctx.test_rss_context4 # Increasing queue count 44 -> 66 # Failed to create context 32, trying to test what we got ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32 ok 5 rss_ctx.test_rss_context_overlap ok 6 rss_ctx.test_rss_context_overlap2 # .. sprays traffic like a headless chicken .. not ok 7 rss_ctx.test_rss_context_out_of_order ok 8 rss_ctx.test_rss_context4_create_with_cfg # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]
Jakub Kicinski (4):
selftests: drv-net: try to check if port is in use
selftests: drv-net: add helper to wait for HW stats to sync
selftests: drv-net: add ability to wait for at least N packets to load
gen
selftests: drv-net: rss_ctx: add tests for RSS configuration and
contexts
.../testing/selftests/drivers/net/hw/Makefile | 1 + .../selftests/drivers/net/hw/rss_ctx.py | 383 ++++++++++++++++++ .../selftests/drivers/net/lib/py/env.py | 19 +- .../selftests/drivers/net/lib/py/load.py | 37 +- tools/testing/selftests/net/lib/py/ksft.py | 5 + tools/testing/selftests/net/lib/py/utils.py | 27 +- 6 files changed, 457 insertions(+), 15 deletions(-) create mode 100755 tools/testing/selftests/drivers/net/hw/rss_ctx.py
Signature · 1 line
2.45.2
Jakub Kicinski
maintainerPATCH v3 1/4 selftests: drv-net: try to check if port is in use
We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.
---
v3:
- use errno.EADDRINUSE (Breno)
v2:
- remove v4 check (Willem)
- update comment (David, Przemek)
- cap the iterations (Przemek)
--- tools/testing/selftests/net/lib/py/utils.py | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
tools/testing/selftests/net/lib/py/utils.py +13 -2
@@ -1,8 +1,10 @@ # SPDX-License-Identifier: GPL-2.0 +import errno import json as _json import random import re +import socket import subprocess import time
@@ -79,9 +81,18 @@ import time def rand_port(): """ - Get unprivileged port, for now just random, one day we may decide to check if used. + Get a random unprivileged port, try to make sure it's not already used. """ - return random.randint(10000, 65535) + for _ in range(1000): + port = random.randint(10000, 65535) + try: + with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as s: + s.bind(("", port)) + return port + except OSError as e: + if e.errno != errno.EADDRINUSE: + raise + raise Exception("Can't find any free unprivileged port") def wait_port_listen(port, proto="tcp", ns=None, host=None, sleep=0.005, deadline=5):
Signature · 1 line
2.45.2
Re: [PATCH net-next v3 1/4] selftests: drv-net: try to check if port is in use
Quoted 6 lines · Jakub Kicinski
We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.
Reviewed-by Przemek Kitszel <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Re: [PATCH net-next v3 1/4] selftests: drv-net: try to check if port is in use
In reply to Jakub KicinskiQuoted 6 lines · Jakub Kicinski
We use random ports for communication. As Willem predicted this leads to occasional failures. Try to check if port is already in use by opening a socket and binding to that port.
Reviewed-by Przemek Kitszel <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH v3 2/4 selftests: drv-net: add helper to wait for HW stats to sync
In reply to Jakub KicinskiSome devices DMA stats to the host periodically. Add a helper which can wait for that to happen, based on frequency reported by the driver in ethtool.
---
v3:
- use get() for stats-block-usecs (Petr)
- fix double space (Petr)
v2:
- sleep for 25ms on top of the driver DMA period
(and remove confusing comment)
--- .../selftests/drivers/net/lib/py/env.py | 19 ++++++++++++++++++- tools/testing/selftests/net/lib/py/utils.py | 4 ++++ 2 files changed, 22 insertions(+), 1 deletion(-)
tools/testing/selftests/drivers/net/lib/py/env.py +18 -1
@@ -1,9 +1,10 @@ # SPDX-License-Identifier: GPL-2.0 import os +import time from pathlib import Path from lib.py import KsftSkipEx, KsftXfailEx -from lib.py import cmd, ip +from lib.py import cmd, ethtool, ip from lib.py import NetNS, NetdevSimDev from .remote import Remote
@@ -82,6 +83,8 @@ from .remote import Remote self.env = _load_env_file(src_path) + self._stats_settle_time = None + # Things we try to destroy self.remote = None # These are for local testing state
@@ -222,3 +225,17 @@ from .remote import Remote if remote: if not self._require_cmd(comm, "remote"): raise KsftSkipEx("Test requires (remote) command: " + comm) + + def wait_hw_stats_settle(self): + """ + Wait for HW stats to become consistent, some devices DMA HW stats + periodically so events won't be reflected until next sync. + Good drivers will tell us via ethtool what their sync period is. + """ + if self._stats_settle_time is None: + data = ethtool("-c " + self.ifname, json=True)[0] + + self._stats_settle_time = 0.025 + \ + data.get('stats-block-usecs', 0) / 1000 / 1000 + + time.sleep(self._stats_settle_time)
tools/testing/selftests/net/lib/py/utils.py +4 -0
@@ -79,6 +79,10 @@ import time return tool('ip', args, json=json, host=host) +def ethtool(args, json=None, ns=None, host=None): + return tool('ethtool', args, json=json, ns=ns, host=host) + + def rand_port(): """ Get a random unprivileged port, try to make sure it's not already used.
Signature · 1 line
2.45.2
Re: [PATCH net-next v3 2/4] selftests: drv-net: add helper to wait for HW stats to sync
Quoted 5 lines · Jakub Kicinski
Some devices DMA stats to the host periodically. Add a helper which can wait for that to happen, based on frequency reported by the driver in ethtool.
Signed-off-by Jakub Kicinski <[email protected]>
Re: [PATCH net-next v3 2/4] selftests: drv-net: add helper to wait for HW stats to sync
In reply to Jakub KicinskiQuoted 5 lines · Jakub Kicinski
Some devices DMA stats to the host periodically. Add a helper which can wait for that to happen, based on frequency reported by the driver in ethtool.
Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH v3 3/4 selftests: drv-net: add ability to wait for at least N packets to load gen
In reply to Jakub KicinskiTeach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.
---
v3:
- rename pkt_start / pkt_end once more (Willem)
v2:
- add comment that pps or pkt_cnt are mutually exclusive (David)
- rename variables (David)
--- .../selftests/drivers/net/lib/py/load.py | 30 ++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-)
tools/testing/selftests/drivers/net/lib/py/load.py +23 -7
@@ -18,15 +18,31 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen background=True, host=env.remote) # Wait for traffic to ramp up - pkt = ip("-s link show dev " + env.ifname, json=True)[0]["stats64"]["rx"]["packets"] + if not self._wait_pkts(pps=1000): + self.stop(verbose=True) + raise Exception("iperf3 traffic did not ramp up") + + def _wait_pkts(self, pkt_cnt=None, pps=None): + """ + Wait until we've seen pkt_cnt or until traffic ramps up to pps. + Only one of pkt_cnt or pss can be specified. + """ + pkt_start = ip("-s link show dev " + self.env.ifname, json=True)[0]["stats64"]["rx"]["packets"] for _ in range(50): time.sleep(0.1) - now = ip("-s link show dev " + env.ifname, json=True)[0]["stats64"]["rx"]["packets"] - if now - pkt > 1000: - return - pkt = now - self.stop(verbose=True) - raise Exception("iperf3 traffic did not ramp up") + pkt_now = ip("-s link show dev " + self.env.ifname, json=True)[0]["stats64"]["rx"]["packets"] + if pps: + if pkt_now - pkt_start > pps / 10: + return True + pkt_start = pkt_now + elif pkt_cnt: + if pkt_now - pkt_start > pkt_cnt: + return True + return False + + def wait_pkts_and_stop(self, pkt_cnt): + failed = not self._wait_pkts(pkt_cnt=pkt_cnt) + self.stop(verbose=failed) def stop(self, verbose=None): self._iperf_client.process(terminate=True)
Signature · 1 line
2.45.2
Re: [PATCH net-next v3 3/4] selftests: drv-net: add ability to wait for at least N packets to load gen
Quoted 7 lines · Jakub Kicinski
Teach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.
Reviewed-by Breno Leitao <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Re: [PATCH net-next v3 3/4] selftests: drv-net: add ability to wait for at least N packets to load gen
In reply to Jakub KicinskiQuoted 7 lines · Jakub Kicinski
Teach the load generator how to wait for at least given number of packets to be received. This will be useful for filtering where we'll want to send a non-trivial number of packets and make sure they landed in right queues.
Reviewed-by Breno Leitao <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH v3 4/4 selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts
In reply to Jakub KicinskiAdd tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.
$ export NETIF=eth0 REMOTE_... $ ./drivers/net/hw/rss_ctx.py KTAP version 1 1..8 ok 1 rss_ctx.test_rss_key_indir ok 2 rss_ctx.test_rss_context ok 3 rss_ctx.test_rss_context4 # Increasing queue count 44 -> 66 # Failed to create context 32, trying to test what we got ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32 ok 5 rss_ctx.test_rss_context_overlap ok 6 rss_ctx.test_rss_context_overlap2 # .. sprays traffic like a headless chicken .. not ok 7 rss_ctx.test_rss_context_out_of_order ok 8 rss_ctx.test_rss_context4_create_with_cfg # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.
---
v3:
- use ksft_lt() to match comment (Petr)
- rewrite the OOO removal to make it more robust (Petr)
v2:
- ensure right queue count for each test (David)
- check driver has ntuple filters before starting (Willem)
- add test for creating contexts with indir table specified (Ed)
- fix ksft_lt (Ed)
- query and validate indirection tables of non-first context (Ed)
- test traffic steering vs OOO context removal (Ed)
- make sure overlap test deletes all rules, 0 is a valid ntuple ID, so we can't do "if ntuple: remove()"
--- .../testing/selftests/drivers/net/hw/Makefile | 1 + .../selftests/drivers/net/hw/rss_ctx.py | 383 ++++++++++++++++++ .../selftests/drivers/net/lib/py/load.py | 7 +- tools/testing/selftests/net/lib/py/ksft.py | 5 + tools/testing/selftests/net/lib/py/utils.py | 8 +- 5 files changed, 399 insertions(+), 5 deletions(-) create mode 100755 tools/testing/selftests/drivers/net/hw/rss_ctx.py
tools/testing/selftests/drivers/net/hw/Makefile +1 -0
@@ -11,6 +11,7 @@ TEST_PROGS = \ hw_stats_l3_gre.sh \ loopback.sh \ pp_alloc_fail.py \ + rss_ctx.py \ # TEST_FILES := \
tools/testing/selftests/drivers/net/hw/rss_ctx.py new file +383 -0
383 lines not on this page · raw message
tools/testing/selftests/drivers/net/lib/py/load.py +4 -3
@@ -5,13 +5,14 @@ import time from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen class GenerateTraffic: - def __init__(self, env): + def __init__(self, env, port=None): env.require_cmd("iperf3", remote=True) self.env = env - port = rand_port() - self._iperf_server = cmd(f"iperf3 -s -p {port}", background=True) + if port is None: + port = rand_port() + self._iperf_server = cmd(f"iperf3 -s -1 -p {port}", background=True) wait_port_listen(port) time.sleep(0.1) self._iperf_client = cmd(f"iperf3 -c {env.addr} -P 16 -p {port} -t 86400",
tools/testing/selftests/net/lib/py/ksft.py +5 -0
@@ -57,6 +57,11 @@ KSFT_RESULT_ALL = True _fail("Check failed", a, "<", b, comment) +def ksft_lt(a, b, comment=""): + if a >= b: + _fail("Check failed", a, ">", b, comment) + + class ksft_raises: def __init__(self, expected_type): self.exception = None
tools/testing/selftests/net/lib/py/utils.py +6 -2
@@ -9,6 +9,10 @@ import subprocess import time +class CmdExitFailure(Exception): + pass + + class cmd: def __init__(self, comm, shell=True, fail=True, ns=None, background=False, host=None, timeout=5): if ns:
@@ -43,8 +47,8 @@ import time if self.proc.returncode != 0 and fail: if len(stderr) > 0 and stderr[-1] == "\n": stderr = stderr[:-1] - raise Exception("Command failed: %s\nSTDOUT: %s\nSTDERR: %s" % - (self.proc.args, stdout, stderr)) + raise CmdExitFailure("Command failed: %s\nSTDOUT: %s\nSTDERR: %s" % + (self.proc.args, stdout, stderr)) class bkg(cmd):
Signature · 1 line
2.45.2
Re: [PATCH net-next v3 4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts
Quoted 20 lines · Jakub Kicinski
Add tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.
$ export NETIF=eth0 REMOTE_... $ ./drivers/net/hw/rss_ctx.py KTAP version 1 1..8 ok 1 rss_ctx.test_rss_key_indir ok 2 rss_ctx.test_rss_context ok 3 rss_ctx.test_rss_context4 # Increasing queue count 44 -> 66 # Failed to create context 32, trying to test what we got ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32 ok 5 rss_ctx.test_rss_context_overlap ok 6 rss_ctx.test_rss_context_overlap2 # .. sprays traffic like a headless chicken .. not ok 7 rss_ctx.test_rss_context_out_of_order ok 8 rss_ctx.test_rss_context4_create_with_cfg # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
Quoted 4 lines
Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.
Signed-off-by Jakub Kicinski <[email protected]>
Re: [PATCH net-next v3 4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts
In reply to Jakub KicinskiQuoted 20 lines · Jakub Kicinski
Add tests focusing on indirection table configuration and creating extra RSS contexts in drivers which support it.
$ export NETIF=eth0 REMOTE_... $ ./drivers/net/hw/rss_ctx.py KTAP version 1 1..8 ok 1 rss_ctx.test_rss_key_indir ok 2 rss_ctx.test_rss_context ok 3 rss_ctx.test_rss_context4 # Increasing queue count 44 -> 66 # Failed to create context 32, trying to test what we got ok 4 rss_ctx.test_rss_context32 # SKIP Tested only 31 contexts, wanted 32 ok 5 rss_ctx.test_rss_context_overlap ok 6 rss_ctx.test_rss_context_overlap2 # .. sprays traffic like a headless chicken .. not ok 7 rss_ctx.test_rss_context_out_of_order ok 8 rss_ctx.test_rss_context4_create_with_cfg # Totals: pass:6 fail:1 xfail:0 xpass:0 skip:1 error:0
Quoted 4 lines
Note that rss_ctx.test_rss_context_out_of_order fails with the device I tested with, but it seems to be a device / driver bug.
Signed-off-by Jakub Kicinski <[email protected]>
tools/testing/selftests/net/lib/py/ksft.py KSFT_RESULT_ALL = True · line 62
Quoted 2 lines
tools/testing/selftests/net/lib/py/ksft.py +2
-0+ if a >= b: + _fail("Check failed", a, ">", b, comment)
Should the argument be ">=" too?
Quoted 1 line · Jakub Kicinski
Ed, could you try the tests with your device?
Don't seem to be able to get them to run:
# Exception| Traceback (most recent call last):
# Exception| File "/home/ecree/kern/linux/tools/testing/selftests/net/lib/py/ksft.py", line 134, in ksft_run
# Exception| case(*args)
# Exception| File "./drivers/net/hw/rss_ctx.py", line 70, in test_rss_key_indir
# Exception| if len(_get_rx_cnts(cfg)) < 2:
# Exception| File "./drivers/net/hw/rss_ctx.py", line 55, in _get_rx_cnts
# Exception| data = cfg.netdevnl.qstats_get({"ifindex": cfg.ifindex, "scope": ["queue"]}, dump=True)
# Exception| File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 1029, in _op
# Exception| return self._ops(ops)[0]
# Exception| File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 985, in _ops
# Exception| raise NlError(nl_msg)
# Exception| net.ynl.lib.ynl.NlError: Netlink error: Operation not supported
# Exception| nl_len = 28 (12) nl_flags = 0x202 nl_type = 3
# Exception| error: -95
# Exception| extack: {'bad-attr': '.ifindex'}not ok 1 rss_ctx.test_rss_key_indir
Cursory investigation suggests this is because sfc doesn't
support netdev_stat_ops, we're still living in the bad old days of ethtool -S for our per-queue stats :(
Much as I'd like to fix that, I don't see a prospect of the
folks upstairs carving out time for it any time soon...
-ed
Jakub Kicinski
maintainerQuoted 20 lines · On Wed, 26 Jun 2024 17:50:20 +0100 Edward Cree wrote:
Quoted 1 line · Jakub Kicinski
Ed, could you try the tests with your device?
Don't seem to be able to get them to run:
# Exception| Traceback (most recent call last): # Exception| File "/home/ecree/kern/linux/tools/testing/selftests/net/lib/py/ksft.py", line 134, in ksft_run # Exception| case(*args) # Exception| File "./drivers/net/hw/rss_ctx.py", line 70, in test_rss_key_indir # Exception| if len(_get_rx_cnts(cfg)) < 2: # Exception| File "./drivers/net/hw/rss_ctx.py", line 55, in _get_rx_cnts # Exception| data = cfg.netdevnl.qstats_get({"ifindex": cfg.ifindex, "scope": ["queue"]}, dump=True) # Exception| File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 1029, in _op # Exception| return self._ops(ops)[0] # Exception| File "/home/ecree/kern/linux/tools/net/ynl/lib/ynl.py", line 985, in _ops # Exception| raise NlError(nl_msg) # Exception| net.ynl.lib.ynl.NlError: Netlink error: Operation not supported # Exception| nl_len = 28 (12) nl_flags = 0x202 nl_type = 3 # Exception| error: -95 # Exception| extack: {'bad-attr': '.ifindex'}
Quoted 5 lines
not ok 1 rss_ctx.test_rss_key_indir
Cursory investigation suggests this is because sfc doesn't
supportnetdev_stat_ops, we're still living in the bad old days of ethtool -S for our per-queue stats :(
Ugh, right, the standard stats turn out to be key in a large number of the tests we end up writing. Perhaps unsurprisingly :(
Fetching stats is done only in _get_rx_cnts(), so you could possibly do a quick hack there to parse ethtool -S, just to test the test. But maybe that'd just lead you to another gap..
Quoted 2 lines
Much as I'd like to fix that, I don't see a prospect of the
folks upstairs carving out time for it any time soon...
Sadness. Makes me worried that they won't care about maintaining the Supported status of the driver either :( (Which is not to say that implementing qstats is a requirement there.)
Anyway, thanks for giving it a go. I'm fairly confident now that the failures I see are a device / driver bug. So we can merge the test and let the various drivers address the gaps.
BTW, while I have you, there are two more bits of work:
- get + dump API via netlink
- set API via netlink
are you planning to work on those?
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Tue, 25 Jun 2024 18:24:52 -0700 you wrote:
Add a few tests exercising RSS context API.
In addition to basic sanity checks, tests add RSS contexts, n-tuple rule to direct traffic to them (based on dst port), and qstats to make sure traffic landed where we expected.v2 adds a test for removing contexts out of order. When testing bnxt - either the new test or running more tests after the overlap test makes the device act strangely. To the point where it may start giving out ntuple IDs of 0 for all rules..
[...]
Here is the summary with links:
- [net-next,v3,1/4] selftests: drv-net: try to check if port is in use
https://git.kernel.org/netdev/net-next/c/8b8fe280155d - [net-next,v3,2/4] selftests: drv-net: add helper to wait for HW stats to sync https://git.kernel.org/netdev/net-next/c/af8e51644a70 - [net-next,v3,3/4] selftests: drv-net: add ability to wait for at least N packets to load gen https://git.kernel.org/netdev/net-next/c/94fecaa6dcd0
- [net-next,v3,4/4] selftests: drv-net: rss_ctx: add tests for RSS configuration and contexts (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html