selftests: drv-net: Test queue stall upon reconfig
Subsystems (3)
PATCH v3 1/1 selftests: drv-net: Test queue stall upon reconfig
From: Mohsin Bashir <[email protected]>
Add a reconfig_tx_stall test that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers using netif_tx_start_all_queues() are prone to experiencing a stall when reconfiguration completes compared to drivers using netif_tx_wake_all_queues(). start_all_queues only clears DRV_XOFF, while wake_all_queues also calls __netif_schedule() to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.
The test caps the TX ring at 64 entries so it fills quickly, then installs FQ on a target TX queue and sends UDP packets with SO_TXTIME scheduled in the future. With napi_defer_hard_irqs slowing completions, the small ring can fill when FQ releases the burst, leaving requeued qdisc backlog with no FQ timer to rescue it. A subsequent ring reconfig must wake the queues to drain the backlog. Simply starting the queues can leave it stuck.
On host with problematic driver:
Sent 128 SO_TXTIME packets (+100ms)
Sent 128 SO_TXTIME packets (+200ms)
Backlog before reconfig: 52632 bytes
Check| At /root/ksft-net-drv/./drivers/net/ring_reconfig.py, ...
Check| ksft_eq(0, backlog,
Check failed 0 != 52632 qdisc backlog stuck on queue 1 after ring reconfig
not ok 3 ring_reconfig.reconfig_tx_stall
On host with fixed driver:
Sent 128 SO_TXTIME packets (+100ms)
Sent 128 SO_TXTIME packets (+200ms)
Backlog before reconfig: 76024 bytes
ok 3 ring_reconfig.reconfig_tx_stall
---
Changelog:
V3:
- Preserve the device's existing TX/RX qdisc policy. On a device with a previously-configured (addressable) mq, only the target queue's leaf is swapped and restored. The default mq (unaddressable handle 0:) or a noqueue root is temporarily given a real handle and restored to the default mq on exit. Skip when the root is a non-mq qdisc rather than disturb it.
- Request 2+ queues from NetDrvEpEnv via a new queue_count parameter so the test runs under netdevsim.
- Fail instead of skip when the qdisc backlog cannot be built.
- Skip on devices without 2+ combined channels; bump combined-count to 2 (restored on exit) when supported but not currently configured.
V2: https://lore.kernel.org/netdev/[email protected]
V1: https://lore.kernel.org/netdev/[email protected]
--- tools/testing/selftests/drivers/net/config | 3 + .../selftests/drivers/net/lib/py/env.py | 11 +- .../selftests/drivers/net/ring_reconfig.py | 207 +++++++++++++++++- 3 files changed, 215 insertions(+), 6 deletions(-)
tools/testing/selftests/drivers/net/config +3 -0
@@ -4,8 +4,11 @@ CONFIG_DEBUG_INFO_BTF_MODULES=n CONFIG_INET_PSP=y CONFIG_IPV6=y CONFIG_MACSEC=m +CONFIG_NET_ACT_SKBEDIT=m CONFIG_NET_CLS_ACT=y CONFIG_NET_CLS_BPF=y +CONFIG_NET_CLS_FLOWER=m +CONFIG_NET_CLS_MATCHALL=m CONFIG_NETCONSOLE=m CONFIG_NETCONSOLE_DYNAMIC=y CONFIG_NETCONSOLE_EXTENDED_LOG=y
tools/testing/selftests/drivers/net/lib/py/env.py +8 -3
@@ -114,10 +114,11 @@ class NetDrvEpEnv(NetDrvEnvBase): nsim_v4_pfx = "192.0.2." nsim_v6_pfx = "2001:db8::" - def __init__(self, src_path, nsim_test=None): + def __init__(self, src_path, nsim_test=None, queue_count=None): super().__init__(src_path) self._stats_settle_time = None + self._queue_count = queue_count # Things we try to destroy self.remote = None
@@ -173,9 +174,13 @@ class NetDrvEpEnv(NetDrvEnvBase): self._required_cmd = {} def create_local(self): + nsim_kwargs = {} + if self._queue_count: + nsim_kwargs["queue_count"] = self._queue_count + self._netns = NetNS() - self._ns = NetdevSimDev() - self._ns_peer = NetdevSimDev(ns=self._netns) + self._ns = NetdevSimDev(**nsim_kwargs) + self._ns_peer = NetdevSimDev(ns=self._netns, **nsim_kwargs) with open("/proc/self/ns/net") as nsfd0, \ open("/var/run/netns/" + self._netns.name) as nsfd1:
tools/testing/selftests/drivers/net/ring_reconfig.py +204 -3
228 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Jakub Kicinski
maintainerQuoted 8 lines · On Thu, 23 Jul 2026 16:48:55 -0700 Mohsin Bashir wrote:
Add a
reconfig_tx_stalltest that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers usingnetif_tx_start_all_queues()are prone to experiencing a stall when reconfiguration completes compared to drivers usingnetif_tx_wake_all_queues().start_all_queuesonly clearsDRV_XOFF, whilewake_all_queuesalso calls__netif_schedule()to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.
AI CI says:
The new reconfig_tx_stall test introduced by this patch is failing in the NIPA CI on the netdevsim-based test environment (both regular and debug kernels):
not ok 3 ring_reconfig.reconfig_tx_stall
Exception| File ".../ring_reconfig.py", line 331, in reconfig_tx_stall
Exception| raise KsftFailEx("failed to build qdisc backlog")
The failure is a precondition failure — the test sends three bursts of 128 SO_TXTIME packets at +100ms, +200ms, and +500ms delays and after each burst checks whether a qdisc backlog has formed on the target TX queue. No backlog appears after any of the three attempts.
The test never reaches the actual ring-reconfig assertion; it fails because the required qdisc backlog cannot be built on netdevsim under CI conditions. This is likely because netdevsim drains its TX rings (and/or releases SO_TXTIME packets) fast enough that no backlog accumulates within the allotted windows, even with napi_defer_hard_irqs set to 100 and the ring capped at 64 entries.
The other two sub-tests (ring_reconfig.channels and ring_reconfig.ringparam) continue to pass.
Could you look into why the backlog-building phase does not work on netdevsim? Possible approaches:
- Add a check/xfail if the backlog cannot be built after N attempts, converting it from a hard failure to a xfail on virtual drivers (prefer xfail over skip)
- Adjust the parameters (ring size, packet count, delay) so the condition is more reliably triggered
- Mark the test as requiring a real NIC (nsim_test=False) if the technique fundamentally doesn't work on virtual devices and move it to the hw/ directory
Quoted 33 lines · Jakub Kicinski
Quoted 8 lines · On Thu, 23 Jul 2026 16:48:55 -0700 Mohsin Bashir wrote:
Add a
reconfig_tx_stalltest that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers usingnetif_tx_start_all_queues()are prone to experiencing a stall when reconfiguration completes compared to drivers usingnetif_tx_wake_all_queues().start_all_queuesonly clearsDRV_XOFF, whilewake_all_queuesalso calls__netif_schedule()to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.AI CI says:
The new
reconfig_tx_stalltest introduced by this patch is failing in the NIPA CI on the netdevsim-based test environment (both regular and debug kernels):not ok 3 ring_reconfig.reconfig_tx_stall
Exception| File "
.../ring_reconfig.py", line 331, inreconfig_tx_stall
Exception| raiseKsftFailEx("failed to build qdisc backlog")The failure is a precondition failure — the test sends three bursts of 128
SO_TXTIMEpackets at +100ms, +200ms, and +500ms delays and after each burst checks whether a qdisc backlog has formed on the target TX queue. No backlog appears after any of the three attempts.The test never reaches the actual ring-reconfig assertion; it fails because the required qdisc backlog cannot be built on netdevsim under CI conditions. This is likely because netdevsim drains its TX rings (and/or releases
SO_TXTIMEpackets) fast enough that no backlog accumulates within the allotted windows, even withnapi_defer_hard_irqsset to 100 and the ring capped at 64 entries.
Quoted 8 lines
The other two sub-tests (ring_reconfig.channels and ring_reconfig.ringparam) continue to pass.
Could you look into why the backlog-building phase does not work on netdevsim? Possible approaches:
- Add a check/xfail if the backlog cannot be built after N attempts, converting it from a hard failure to a xfail on virtual drivers (prefer xfail over skip)
Yeah, looks like netdevsim is not providing back-pressure on the TX path. I will change the fail to xfail in V4.
Quoted 7 lines
- Adjust the parameters (ring size, packet count, delay) so the condition is more reliably triggered
- Mark the test as requiring a real NIC (nsim_test=False) if the technique fundamentally doesn't work on virtual devices and move it to the hw/ directory