Skip to contentSkip to conversation

selftests: drv-net: Test queue stall upon reconfig

PATCH Superseded Mohsin Bashir 2 participants 3 messages Jul 23 – Jul 24

Subsystems (3)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 selftests: drv-net: Test queue stall upon reconfig

From: Mohsin Bashir <[email protected]>

Add a reconfig_tx_stall test that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers using netif_tx_start_all_queues() are prone to experiencing a stall when reconfiguration completes compared to drivers using netif_tx_wake_all_queues(). start_all_queues only clears DRV_XOFF, while wake_all_queues also calls __netif_schedule() to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.

The test caps the TX ring at 64 entries so it fills quickly, then installs FQ on a target TX queue and sends UDP packets with SO_TXTIME scheduled in the future. With napi_defer_hard_irqs slowing completions, the small ring can fill when FQ releases the burst, leaving requeued qdisc backlog with no FQ timer to rescue it. A subsequent ring reconfig must wake the queues to drain the backlog. Simply starting the queues can leave it stuck.

On host with problematic driver:
Sent 128 SO_TXTIME packets (+100ms)
Sent 128 SO_TXTIME packets (+200ms)
Backlog before reconfig: 52632 bytes
Check| At /root/ksft-net-drv/./drivers/net/ring_reconfig.py, ...
Check| ksft_eq(0, backlog,
Check failed 0 != 52632 qdisc backlog stuck on queue 1 after ring reconfig
not ok 3 ring_reconfig.reconfig_tx_stall

On host with fixed driver:
Sent 128 SO_TXTIME packets (+100ms)
Sent 128 SO_TXTIME packets (+200ms)
Backlog before reconfig: 76024 bytes
ok 3 ring_reconfig.reconfig_tx_stall

---
Changelog:
V3:
- Preserve the device's existing TX/RX qdisc policy. On a device with a previously-configured (addressable) mq, only the target queue's leaf is swapped and restored. The default mq (unaddressable handle 0:) or a noqueue root is temporarily given a real handle and restored to the default mq on exit. Skip when the root is a non-mq qdisc rather than disturb it.
- Request 2+ queues from NetDrvEpEnv via a new queue_count parameter so the test runs under netdevsim.
- Fail instead of skip when the qdisc backlog cannot be built.
- Skip on devices without 2+ combined channels; bump combined-count to 2 (restored on exit) when supported but not currently configured.

V2: https://lore.kernel.org/netdev/[email protected]

V1: https://lore.kernel.org/netdev/[email protected]

---
 tools/testing/selftests/drivers/net/config    |   3 +
 .../selftests/drivers/net/lib/py/env.py       |  11 +-
 .../selftests/drivers/net/ring_reconfig.py    | 207 +++++++++++++++++-
 3 files changed, 215 insertions(+), 6 deletions(-)
tools/testing/selftests/drivers/net/config +3 -0
index 2070e890e064..f3933cf3e6be 100644
@@ -4,8 +4,11 @@ CONFIG_DEBUG_INFO_BTF_MODULES=n
 CONFIG_INET_PSP=y
 CONFIG_IPV6=y
 CONFIG_MACSEC=m
+CONFIG_NET_ACT_SKBEDIT=m
 CONFIG_NET_CLS_ACT=y
 CONFIG_NET_CLS_BPF=y
+CONFIG_NET_CLS_FLOWER=m
+CONFIG_NET_CLS_MATCHALL=m
 CONFIG_NETCONSOLE=m
 CONFIG_NETCONSOLE_DYNAMIC=y
 CONFIG_NETCONSOLE_EXTENDED_LOG=y
tools/testing/selftests/drivers/net/lib/py/env.py +8 -3
index e4acf3d8333f..b1c6f6cef7dd 100644
@@ -114,10 +114,11 @@ class NetDrvEpEnv(NetDrvEnvBase):
     nsim_v4_pfx = "192.0.2."
     nsim_v6_pfx = "2001:db8::"
 
-    def __init__(self, src_path, nsim_test=None):
+    def __init__(self, src_path, nsim_test=None, queue_count=None):
         super().__init__(src_path)
 
         self._stats_settle_time = None
+        self._queue_count = queue_count
 
         # Things we try to destroy
         self.remote = None
@@ -173,9 +174,13 @@ class NetDrvEpEnv(NetDrvEnvBase):
         self._required_cmd = {}
 
     def create_local(self):
+        nsim_kwargs = {}
+        if self._queue_count:
+            nsim_kwargs["queue_count"] = self._queue_count
+
         self._netns = NetNS()
-        self._ns = NetdevSimDev()
-        self._ns_peer = NetdevSimDev(ns=self._netns)
+        self._ns = NetdevSimDev(**nsim_kwargs)
+        self._ns_peer = NetdevSimDev(ns=self._netns, **nsim_kwargs)
 
         with open("/proc/self/ns/net") as nsfd0, \
              open("/var/run/netns/" + self._netns.name) as nsfd1:
tools/testing/selftests/drivers/net/ring_reconfig.py +204 -3

228 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
maintainer
#1 lore raw
Quoted 8 lines · On Thu, 23 Jul 2026 16:48:55 -0700 Mohsin Bashir wrote:

Add a reconfig_tx_stall test that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers using netif_tx_start_all_queues() are prone to experiencing a stall when reconfiguration completes compared to drivers using netif_tx_wake_all_queues(). start_all_queues only clears DRV_XOFF, while wake_all_queues also calls __netif_schedule() to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.

AI CI says:

The new reconfig_tx_stall test introduced by this patch is failing in the NIPA CI on the netdevsim-based test environment (both regular and debug kernels):

not ok 3 ring_reconfig.reconfig_tx_stall

Exception| File ".../ring_reconfig.py", line 331, in reconfig_tx_stall
Exception| raise KsftFailEx("failed to build qdisc backlog")

The failure is a precondition failure — the test sends three bursts of 128 SO_TXTIME packets at +100ms, +200ms, and +500ms delays and after each burst checks whether a qdisc backlog has formed on the target TX queue. No backlog appears after any of the three attempts.

The test never reaches the actual ring-reconfig assertion; it fails because the required qdisc backlog cannot be built on netdevsim under CI conditions. This is likely because netdevsim drains its TX rings (and/or releases SO_TXTIME packets) fast enough that no backlog accumulates within the allotted windows, even with napi_defer_hard_irqs set to 100 and the ring capped at 64 entries.

The other two sub-tests (ring_reconfig.channels and ring_reconfig.ringparam) continue to pass.

Could you look into why the backlog-building phase does not work on netdevsim? Possible approaches:
- Add a check/xfail if the backlog cannot be built after N attempts, converting it from a hard failure to a xfail on virtual drivers (prefer xfail over skip)
- Adjust the parameters (ring size, packet count, delay) so the condition is more reliably triggered
- Mark the test as requiring a real NIC (nsim_test=False) if the technique fundamentally doesn't work on virtual devices and move it to the hw/ directory

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-drv/results/747522/8-ring-reconfig-py/stdout

#2 lore raw
Quoted 33 lines · Jakub Kicinski
Quoted 8 lines · On Thu, 23 Jul 2026 16:48:55 -0700 Mohsin Bashir wrote:

Add a reconfig_tx_stall test that detects the possibility of a TX stall after ring reconfiguration. The key observation is that drivers using netif_tx_start_all_queues() are prone to experiencing a stall when reconfiguration completes compared to drivers using netif_tx_wake_all_queues(). start_all_queues only clears DRV_XOFF, while wake_all_queues also calls __netif_schedule() to kick the qdisc. Without the kick, qdisc backlog present at reconfig time can stay stuck until a new trigger is issued.

AI CI says:

The new reconfig_tx_stall test introduced by this patch is failing in the NIPA CI on the netdevsim-based test environment (both regular and debug kernels):

not ok 3 ring_reconfig.reconfig_tx_stall

Exception| File ".../ring_reconfig.py", line 331, in reconfig_tx_stall
Exception| raise KsftFailEx("failed to build qdisc backlog")

The failure is a precondition failure — the test sends three bursts of 128 SO_TXTIME packets at +100ms, +200ms, and +500ms delays and after each burst checks whether a qdisc backlog has formed on the target TX queue. No backlog appears after any of the three attempts.

The test never reaches the actual ring-reconfig assertion; it fails because the required qdisc backlog cannot be built on netdevsim under CI conditions. This is likely because netdevsim drains its TX rings (and/or releases SO_TXTIME packets) fast enough that no backlog accumulates within the allotted windows, even with napi_defer_hard_irqs set to 100 and the ring capped at 64 entries.

Quoted 8 lines

The other two sub-tests (ring_reconfig.channels and ring_reconfig.ringparam) continue to pass.

Could you look into why the backlog-building phase does not work on netdevsim? Possible approaches:
- Add a check/xfail if the backlog cannot be built after N attempts, converting it from a hard failure to a xfail on virtual drivers (prefer xfail over skip)

Yeah, looks like netdevsim is not providing back-pressure on the TX path. I will change the fail to xfail in V4.

Quoted 7 lines

- Adjust the parameters (ring size, packet count, delay) so the condition is more reliably triggered
- Mark the test as requiring a real NIC (nsim_test=False) if the technique fundamentally doesn't work on virtual devices and move it to the hw/ directory

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-drv/results/747522/8-ring-reconfig-py/stdout