net: Backport backlog-NAPI threads to resolve PREEMPT_RT softirq warning in SMP flush

From: Wen Yang <[email protected]>

Backport the backlog-NAPI smpboot-thread mechanism and its prerequisite NAPI/RPS/defer_list work from upstream (v6.4 - v6.9) to v6.1-rt-next.

Issues
------
On PREEMPT_RT, RPS steers packets to a remote CPU's backlog by sending an IPI. The IPI handler (rps_trigger_softirq) calls ____napi_schedule(), which raises NET_RX_SOFTIRQ. do_softirq_post_smp_call_flush() then sees a softirq pending that was not pending before the flush, and NET_RX_SOFTIRQ is not masked by SCHED_SOFTIRQ_MASK, so it fires:

  WARNING: CPU: 1 PID: 0 at kernel/softirq.c:315 do_softirq_post_smp_call_flush+0x3d/0x50
  Call Trace:
   <TASK>
   flush_smp_call_function_queue+0x50/0x70
   do_idle+0x178/0x260
   cpu_startup_entry+0x21/0x30

The same path then calls invoke_softirq() -> wakeup_softirqd() with IRQs disabled in interrupt context, which the RT maintainers consider a design violation ("raising a softirq from an SMP call function").

Commit 8d152d22d9f0 ("net: Avoid the IPI to free the") worked around this only for the skb defer_list by delegating the purge to a workqueue. That was
incomplete: it did not touch RPS steering, and workqueue scheduling adds latency to deferred skb recycling.

Solution
--------
Upstream v6.9 dad6b9770263 ("net: Allow to use SMP threads for backlog NAPI") adds per-CPU smpboot threads ("backlog_napi/%u"). On PREEMPT_RT use_backlog_threads() is unconditionally true, so cross-CPU backlog and defer_list work bypasses IPIs and softirqs entirely:
1) Cross-CPU backlog enqueue sets NAPI_STATE_SCHED_THREADED and wakes the remote CPU's backlog_napi thread with wake_up_process().
2) defer_list cleanup is folded into the same thread (80d2eefcb4c8).
3) The old defer_work workaround is reverted (patch 01).

Backport notes
--------------
All 15 patches apply cleanly in sequence against 6271c130c6e5. Two patches needed hand work. Every deviation from the upstream text is listed below, together with the base difference that forced it -- there is no edit made just for the sake of editing.

1. Patch 06 (____napi_schedule): upstream 8b43fd3d1d7d writes __raise_softirq_irqoff(); the backport keeps raise_softirq_irqoff().
Reason: 6.1-rt's ____napi_schedule() already uses raise_softirq_irqoff(), because stable 533824e6a0fb (upstream 77e45145e303, "net: Handle napi_schedule() calls from non-interrupt") deliberately added the ksoftirqd wakeup there to fix the NOHZ tick-stop "local softirq work is pending" stall. Applying the 2023 upstream text verbatim would silently revert that later fix. The 2023 commit predates it, so this is a forward-port artefact, not a behavioural choice.

2. Patch 13 (backlog-NAPI threads): five hunks were re-anchored. None changes behaviour; each is forced by a 6.1-vs-v6.9 base difference, verified against 6271c130c6e5:
a) backlog_napi percpu definition
upstream anchor : "unsigned int sysctl_skb_defer_max"
6.1 anchor : "int netdev_max_backlog"
Both variables sit in the same block right after the "Receiver routines" comment, so the declaration lands in the identical place. Pure textual anchor.

b) ____napi_schedule(): WRITE_ONCE(napi->list_owner, ...) dropped napi_struct has no list_owner member on 6.1 (grep over 6271c130c6e5's netdevice.h returns nothing), so keeping the line does not compile. list_owner only backs the napi_by_id()/list-ownership bookkeeping from a later upstream series; it is not consulted anywhere on the backlog path, so omitting it cannot change behaviour.

c) napi_threaded_poll() split into napi_threaded_poll_loop() + wrapper Not a deviation: the same split, applied to 6.1's function body, which after patches 09-11 is textually the upstream pre-image. Verified: the extracted napi_threaded_poll_loop() is byte-identical between upstream dad6b9770263 and this backport.

d) smpboot block anchor
Upstream anchors the new backlog_threads block on the code following net_page_pool_create(); 6.1 has no net_page_pool_create() (0 occurrences in 6271c130c6e5). The block is anchored on the comment preceding net_dev_init() instead. The inserted code is identical.

e) net_dev_init()
Same cause: the "if (net_page_pool_create(i)) goto out;" context lines do not exist on 6.1 and were dropped. Both substantive additions are kept verbatim, and one of them is load-bearing, not cosmetic:
INIT_LIST_HEAD(&sd->backlog.poll_list);

6.1 never initialises backlog.poll_list (no INIT_LIST_HEAD and no netif_napi_add for the backlog NAPI in 6271c130c6e5's dev.c). That was harmless while the backlog NAPI was only ever reached via list_add_tail(), which does not require a pre-initialised node. The threaded path calls __napi_poll() on the backlog NAPI with no list_add() at all, and __napi_poll() tests:

if (unlikely(!list_empty(&n->poll_list))) {
pr_warn_once("%s: Budget exhausted after napi rescheduled\n",
n->dev ? n->dev->name : "backlog");
return work; /* leaves *repoll unset */
}

(the "backlog" fallback in that message shows upstream expects this NAPI to reach it). On a zeroed node list_empty() returns false, so the backlog thread would take this branch on every budget exhaustion, print the spurious "backlog: Budget exhausted after napi rescheduled" warning and stop repolling. The INIT_LIST_HEAD is therefore required for correctness on 6.1, exactly as it is upstream.

Verification
------------
Each of the five stages was replayed against the base snapshot and compiled for net/core/dev.o and net/core/skbuff.o. The baseline and final RT/RPS bzImages were built with byte-identical normalized configs using GCC 9.4.0, with no compiler warnings or errors.

Tests use virtme-ng 1.38, x86_64 QEMU/KVM, four vCPUs and 2 GiB memory:

  vng --run <bzImage> --cpus 4 --memory 2G \
      --qemu-opts='-machine accel=kvm' --exec 'bash <script>'

The config enables PREEMPT_RT, SMP, SYSFS, NET_NS, VETH, RPS, IRQ_TIME_ACCOUNTING and KPROBES, with NR_CPUS=4. Run the inline script below in a fresh guest for each image; it prints one column of the table, and the columns are joined outside the guest. It covers both mechanisms: the RPS/IPI path (ping flood with RPS steered to CPU1) and the defer_list purge path (cross-CPU TCP frees traced with kprobes). See its header for details.

#!/bin/bash
# Run only in a fresh virtme guest; keep host networking and logs untouched. # One run prints both metric groups of the A/B table:
# IPI/RPS : hardirq time, received_rps, function-call IPIs, flush WARNs
# defer_list: skb_attempt_defer_free vs the purge trigger that fires
# MODE=rps sends RPS to CPU1; MODE=local keeps it on CPU0; MODE=norps has RPS off.

  set -euo pipefail
  export LC_ALL=C
  SECS=${SECS:-20}; MODE=${MODE:-rps}; CPU=1; MASK=2
  [[ $MODE == rps ]] || { CPU=0; MASK=1; }
  grep -q 'virtme' /proc/cmdline
  [[ $(cat /sys/kernel/realtime) == 1 ]]
  HZ=$(getconf CLK_TCK)
  command -v iperf3 >/dev/null || { echo 'SKIP: iperf3 missing'; exit 0; }
  # The guest root is read-only; iperf3 needs a writable temp dir to start.
  TMPDIR=$(for d in /var/tmp /dev/shm /run /root /tmp; do [ -w "$d" ] && { echo "$d"; break; }; done)
  [ -n "$TMPDIR" ] && export TMPDIR || { echo 'SKIP: no writable TMPDIR'; exit 0; }
  TR=/sys/kernel/tracing
  [[ -d $TR ]] || { mountpoint -q /sys/kernel/debug || mount -t debugfs none /sys/kernel/debug; TR=/sys/kernel/debug/tracing; }
  [[ -d $TR ]] || { echo 'SKIP: no tracing dir'; exit 0; }

  NS=bn$$; TX=bn$$t; RX=bn$$r
  PROBES='def_free kick trig'
  cleanup() {
      ip link del "$TX" 2>/dev/null || true
      ip netns del "$NS" 2>/dev/null || true
      for p in $PROBES; do { echo "-:$p" > "$TR/kprobe_events"; } 2>/dev/null || true; done
  }
  trap cleanup EXIT
  ip netns add "$NS"
  ip link add "$TX" type veth peer name "$RX"
  ip link set "$RX" netns "$NS"
  ip addr add 192.0.2.1/24 dev "$TX"
  ip link set "$TX" up
  ip netns exec "$NS" ip addr add 192.0.2.2/24 dev "$RX"
  ip netns exec "$NS" ip link set "$RX" up
  if [[ $MODE != norps ]]; then
      ip netns exec "$NS" sh -c "printf '%s\n' '$MASK' > /sys/class/net/$RX/queues/rx-0/rps_cpus"
      printf 'RPS_MASK='
      ip netns exec "$NS" sh -c "cat /sys/class/net/$RX/queues/rx-0/rps_cpus"
  fi
  printf 'KERNEL=%s MODE=%s CPU=%s SECS=%s USER_HZ=%s\n' "$(uname -r)" "$MODE" "$CPU" "$SECS" "$HZ"
  uname -v
  ps -eo pid,cls,rtprio,psr,comm | grep -E 'backlog_napi|ksoftirqd' || true
  for p in $(ps -eo pid,comm | awk '$2 ~ /^backlog_napi\// {print $1}'); do taskset -pc "$p"; done

  # ---- IPI/RPS path: CPU0 floods the veth, RPS steers RX to $CPU ----
  # Connectivity check also warms up ARP and avoids charging WARN output to load.
  taskset -c 0 ping -n -q -c 3 -W 2 192.0.2.2
  irq() { awk -v c="cpu$CPU" '$1==c {print $7}' /proc/stat; }
  field() { awk -v c="$CPU" -v f="$1" 'NR==c+1 {print $f}' /proc/net/softnet_stat; }
  # CPU n is field n+2 on the "CAL: ... Function call interrupts" line.
  cal() { awk -v c="$((CPU+2))" '/Function call interrupts/ {print $c}' /proc/interrupts; }
  i0=$(irq); sleep "$SECS"; i1=$(irq)
  r0=$(field 10); c0=$(cal); j0=$(irq)
  rc=0
  PING=$(timeout -s INT "$SECS" taskset -c 0 ping -n -f -q 192.0.2.2 2>&1) || rc=$?
  [[ $rc == 124 || $rc == 0 ]] || { printf '%s\n' "$PING"; exit "$rc"; }
  j1=$(irq); r1=$(field 10); c1=$(cal)
  printf '%s\n' "$PING"
  printf '%s\n' "$PING" | awk '/packets transmitted/ {ok=($1>0 && $4>0)} END {exit !ok}'
  printf 'IDLE_IRQ_MS=%s LOAD_IRQ_MS=%s RPS_CALLBACKS=%s CALL_IPI=%s\n' \
      "$(((i1-i0)*1000/HZ))" "$(((j1-j0)*1000/HZ))" \
      "$((16#$r1-16#$r0))" "$((c1-c0))"
  # A WARN header is one warning; repeated symbol names in its stack are not.
  LOG=$(dmesg)
  printf 'TARGET_WARNINGS=%s\n' "$(printf '%s\n' "$LOG" | grep -cE 'WARNING:.*do_softirq_post_smp_call_flush' || true)"

  # ---- defer_list path: skb alloc on CPU0, freed by a CPU1 reader ----
  # kick_defer_list_purge exists only in the patched tree. kprobe_events events
  # are created disabled, so each probe must be enabled explicitly.
  for spec in 'def_free skb_attempt_defer_free' 'kick kick_defer_list_purge' \
              'trig trigger_rx_softirq'; do
      set -- $spec
      { echo "p:$1 $2" >> "$TR/kprobe_events"; } 2>/dev/null || echo "PROBE_ABSENT=$2"
      { echo 1 > "$TR/events/kprobes/$1/enable"; } 2>/dev/null || true
  done
  # A probe that never registered has no counter; report that explicitly rather
  # than as a misleading zero.
  hits() {
      [ -r "$TR/events/kprobes/$1/enable" ] || { echo ABSENT; return; }
      awk -v n="$1" '$1==n {print $2}' "$TR/kprobe_profile" 2>/dev/null | grep . || echo 0
  }
  printf '4\n' > /proc/sys/net/core/skb_defer_max
  ip netns exec "$NS" taskset -c 1 iperf3 -s -1 -p 5201 >/dev/null 2>&1 &
  SRV=$!
  sleep 1
  taskset -c 0 iperf3 -c 192.0.2.2 -p 5201 -t "$SECS" -P 4 >/dev/null 2>&1 || true
  wait "$SRV" 2>/dev/null || true
  sleep 2   # let any pending purge drain before sampling
  D=$(hits def_free); K=$(hits kick); T=$(hits trig)
  printf 'HITS def_free=%s kick=%s trig=%s\n' "$D" "$K" "$T"
  [[ $D -gt 0 ]] && echo 'DEFER_PATH=REACHED' || echo 'DEFER_PATH=NOT_REACHED'

  printf '%s\n' '--- softnet_stat ---'; cat /proc/net/softnet_stat
  # Only the warning block is evidence; the rest of dmesg is boot noise.
  printf '%s\n' '--- WARNING ---'
  printf '%s\n' "$LOG" | grep -A20 -E 'WARNING:.*do_softirq_post_smp_call_flush' || true

Two images are compared: the unmodified base (4c4970859022, no patches) and the patched image (all five patches). Each was booted twice in a fresh guest; the table shows the first run, and the repeat is consistent with it.

Baseline Patched
Idle hardirq (ms) 10 10
Load hardirq (ms) 900 40
received_rps delta 2,390,909 0
Function-call IPIs 2,390,695 947
Echo requests received 2,390,916 2,473,383
Target WARNING records 1 0
skb_attempt_defer_free() 1,075,050 1,558,153
kick_defer_list_purge() absent 193,560
trigger_rx_softirq() 176,249 0

The repeat agrees within noise: baseline load hardirq 930 ms, 2,455,171 received_rps and 2,455,170 function-call IPIs, 1 WARNING; patched 40 ms, 0 received_rps and 825 function-call IPIs, 0 WARNING.

Eric Dumazet (9):
net: napi_schedule_rps() cleanup
net: add softnet_data.in_net_rx_action
net: optimize napi_schedule_rps()
net: optimize ____napi_schedule() to avoid extra NET_RX_SOFTIRQ
net: add debugging checks in skb_attempt_defer_free()
net: do not provide hard irq safety for sd->defer_lock
net: move skb_defer_free_flush() up
net: make napi_threaded_poll() aware of sd->defer_list
net: optimize napi_threaded_poll() vs RPS/RFS

Jesper Dangaard Brouer (1):
net: avoid irqsave in skb_defer_free_flush

Sebastian Andrzej Siewior (4):
net: Remove conditional threaded-NAPI wakeup based on task state.
net: Allow to use SMP threads for backlog NAPI.
net: Use backlog-NAPI to clean up the defer_list.
net: Rename rps_lock to backlog_lock.

Wen Yang (1):
Revert "net: Avoid the IPI to free the"

 include/linux/netdevice.h |   9 +-
 net/core/dev.c            | 304 +++++++++++++++++++++++++-------------
 net/core/skbuff.c         |  17 +--
 3 files changed, 215 insertions(+), 115 deletions(-)

2.25.1