net: Backport backlog-NAPI threads to resolve PREEMPT_RT softirq warning in SMP flush
Subsystems (3)
v2 cover net: Backport backlog-NAPI threads to resolve PREEMPT_RT softirq warning in SMP flush
From: Wen Yang <[email protected]>
Backport the backlog-NAPI smpboot-thread mechanism and its prerequisite NAPI/RPS/defer_list work from upstream (v6.4 - v6.9) to v6.1-rt-next.
Issues
------
On PREEMPT_RT, RPS steers packets to a remote CPU's backlog by sending an IPI. The IPI handler (rps_trigger_softirq) calls ____napi_schedule(), which raises NET_RX_SOFTIRQ. do_softirq_post_smp_call_flush() then sees a softirq pending that was not pending before the flush, and NET_RX_SOFTIRQ is not masked by SCHED_SOFTIRQ_MASK, so it fires:
WARNING: CPU: 1 PID: 0 at kernel/softirq.c:315 do_softirq_post_smp_call_flush+0x3d/0x50 Call Trace: <TASK> flush_smp_call_function_queue+0x50/0x70 do_idle+0x178/0x260 cpu_startup_entry+0x21/0x30
The same path then calls invoke_softirq() -> wakeup_softirqd() with IRQs disabled in interrupt context, which the RT maintainers consider a design violation ("raising a softirq from an SMP call function").
Commit 8d152d22d9f0 ("net: Avoid the IPI to free the") worked around this only for the skb defer_list by delegating the purge to a workqueue. That was
incomplete: it did not touch RPS steering, and workqueue scheduling adds latency to deferred skb recycling.
Solution
--------
Upstream v6.9 dad6b9770263 ("net: Allow to use SMP threads for backlog NAPI") adds per-CPU smpboot threads ("backlog_napi/%u"). On PREEMPT_RT use_backlog_threads() is unconditionally true, so cross-CPU backlog and defer_list work bypasses IPIs and softirqs entirely:
1) Cross-CPU backlog enqueue sets NAPI_STATE_SCHED_THREADED and wakes the remote CPU's backlog_napi thread with wake_up_process().
2) defer_list cleanup is folded into the same thread (80d2eefcb4c8).
3) The old defer_work workaround is reverted (patch 01).
Backport notes
--------------
All 15 patches apply cleanly in sequence against 6271c130c6e5. Two patches needed hand work. Every deviation from the upstream text is listed below, together with the base difference that forced it -- there is no edit made just for the sake of editing.
1. Patch 06 (____napi_schedule): upstream 8b43fd3d1d7d writes __raise_softirq_irqoff(); the backport keeps raise_softirq_irqoff().
Reason: 6.1-rt's ____napi_schedule() already uses raise_softirq_irqoff(), because stable 533824e6a0fb (upstream 77e45145e303, "net: Handle napi_schedule() calls from non-interrupt") deliberately added the ksoftirqd wakeup there to fix the NOHZ tick-stop "local softirq work is pending" stall. Applying the 2023 upstream text verbatim would silently revert that later fix. The 2023 commit predates it, so this is a forward-port artefact, not a behavioural choice.
2. Patch 13 (backlog-NAPI threads): five hunks were re-anchored. None changes behaviour; each is forced by a 6.1-vs-v6.9 base difference, verified against 6271c130c6e5:
a) backlog_napi percpu definition
upstream anchor : "unsigned int sysctl_skb_defer_max"
6.1 anchor : "int netdev_max_backlog"
Both variables sit in the same block right after the "Receiver routines" comment, so the declaration lands in the identical place. Pure textual anchor.
b) ____napi_schedule(): WRITE_ONCE(napi->list_owner, ...) dropped napi_struct has no list_owner member on 6.1 (grep over 6271c130c6e5's netdevice.h returns nothing), so keeping the line does not compile. list_owner only backs the napi_by_id()/list-ownership bookkeeping from a later upstream series; it is not consulted anywhere on the backlog path, so omitting it cannot change behaviour.
c) napi_threaded_poll() split into napi_threaded_poll_loop() + wrapper Not a deviation: the same split, applied to 6.1's function body, which after patches 09-11 is textually the upstream pre-image. Verified: the extracted napi_threaded_poll_loop() is byte-identical between upstream dad6b9770263 and this backport.
d) smpboot block anchor
Upstream anchors the new backlog_threads block on the code following net_page_pool_create(); 6.1 has no net_page_pool_create() (0 occurrences in 6271c130c6e5). The block is anchored on the comment preceding net_dev_init() instead. The inserted code is identical.
e) net_dev_init()
Same cause: the "if (net_page_pool_create(i)) goto out;" context lines do not exist on 6.1 and were dropped. Both substantive additions are kept verbatim, and one of them is load-bearing, not cosmetic:
INIT_LIST_HEAD(&sd->backlog.poll_list);
6.1 never initialises backlog.poll_list (no INIT_LIST_HEAD and no netif_napi_add for the backlog NAPI in 6271c130c6e5's dev.c). That was harmless while the backlog NAPI was only ever reached via list_add_tail(), which does not require a pre-initialised node. The threaded path calls __napi_poll() on the backlog NAPI with no list_add() at all, and __napi_poll() tests:
if (unlikely(!list_empty(&n->poll_list))) {
pr_warn_once("%s: Budget exhausted after napi rescheduled\n",
n->dev ? n->dev->name : "backlog");
return work; /* leaves *repoll unset */
}
(the "backlog" fallback in that message shows upstream expects this NAPI to reach it). On a zeroed node list_empty() returns false, so the backlog thread would take this branch on every budget exhaustion, print the spurious "backlog: Budget exhausted after napi rescheduled" warning and stop repolling. The INIT_LIST_HEAD is therefore required for correctness on 6.1, exactly as it is upstream.
Verification
------------
Each of the five stages was replayed against the base snapshot and compiled for net/core/dev.o and net/core/skbuff.o. The baseline and final RT/RPS bzImages were built with byte-identical normalized configs using GCC 9.4.0, with no compiler warnings or errors.
Tests use virtme-ng 1.38, x86_64 QEMU/KVM, four vCPUs and 2 GiB memory:
vng --run <bzImage> --cpus 4 --memory 2G \
--qemu-opts='-machine accel=kvm' --exec 'bash <script>'The config enables PREEMPT_RT, SMP, SYSFS, NET_NS, VETH, RPS, IRQ_TIME_ACCOUNTING and KPROBES, with NR_CPUS=4. Run the inline script below in a fresh guest for each image; it prints one column of the table, and the columns are joined outside the guest. It covers both mechanisms: the RPS/IPI path (ping flood with RPS steered to CPU1) and the defer_list purge path (cross-CPU TCP frees traced with kprobes). See its header for details.
#!/bin/bash
# Run only in a fresh virtme guest; keep host networking and logs untouched. # One run prints both metric groups of the A/B table:
# IPI/RPS : hardirq time, received_rps, function-call IPIs, flush WARNs
# defer_list: skb_attempt_defer_free vs the purge trigger that fires
# MODE=rps sends RPS to CPU1; MODE=local keeps it on CPU0; MODE=norps has RPS off.
set -euo pipefail
export LC_ALL=C
SECS=${SECS:-20}; MODE=${MODE:-rps}; CPU=1; MASK=2
[[ $MODE == rps ]] || { CPU=0; MASK=1; }
grep -q 'virtme' /proc/cmdline
[[ $(cat /sys/kernel/realtime) == 1 ]]
HZ=$(getconf CLK_TCK)
command -v iperf3 >/dev/null || { echo 'SKIP: iperf3 missing'; exit 0; }
# The guest root is read-only; iperf3 needs a writable temp dir to start.
TMPDIR=$(for d in /var/tmp /dev/shm /run /root /tmp; do [ -w "$d" ] && { echo "$d"; break; }; done)
[ -n "$TMPDIR" ] && export TMPDIR || { echo 'SKIP: no writable TMPDIR'; exit 0; }
TR=/sys/kernel/tracing
[[ -d $TR ]] || { mountpoint -q /sys/kernel/debug || mount -t debugfs none /sys/kernel/debug; TR=/sys/kernel/debug/tracing; }
[[ -d $TR ]] || { echo 'SKIP: no tracing dir'; exit 0; }
NS=bn$$; TX=bn$$t; RX=bn$$r
PROBES='def_free kick trig'
cleanup() {
ip link del "$TX" 2>/dev/null || true
ip netns del "$NS" 2>/dev/null || true
for p in $PROBES; do { echo "-:$p" > "$TR/kprobe_events"; } 2>/dev/null || true; done
}
trap cleanup EXIT
ip netns add "$NS"
ip link add "$TX" type veth peer name "$RX"
ip link set "$RX" netns "$NS"
ip addr add 192.0.2.1/24 dev "$TX"
ip link set "$TX" up
ip netns exec "$NS" ip addr add 192.0.2.2/24 dev "$RX"
ip netns exec "$NS" ip link set "$RX" up
if [[ $MODE != norps ]]; then
ip netns exec "$NS" sh -c "printf '%s\n' '$MASK' > /sys/class/net/$RX/queues/rx-0/rps_cpus"
printf 'RPS_MASK='
ip netns exec "$NS" sh -c "cat /sys/class/net/$RX/queues/rx-0/rps_cpus"
fi
printf 'KERNEL=%s MODE=%s CPU=%s SECS=%s USER_HZ=%s\n' "$(uname -r)" "$MODE" "$CPU" "$SECS" "$HZ"
uname -v
ps -eo pid,cls,rtprio,psr,comm | grep -E 'backlog_napi|ksoftirqd' || true
for p in $(ps -eo pid,comm | awk '$2 ~ /^backlog_napi\// {print $1}'); do taskset -pc "$p"; done
# ---- IPI/RPS path: CPU0 floods the veth, RPS steers RX to $CPU ----
# Connectivity check also warms up ARP and avoids charging WARN output to load.
taskset -c 0 ping -n -q -c 3 -W 2 192.0.2.2
irq() { awk -v c="cpu$CPU" '$1==c {print $7}' /proc/stat; }
field() { awk -v c="$CPU" -v f="$1" 'NR==c+1 {print $f}' /proc/net/softnet_stat; }
# CPU n is field n+2 on the "CAL: ... Function call interrupts" line.
cal() { awk -v c="$((CPU+2))" '/Function call interrupts/ {print $c}' /proc/interrupts; }
i0=$(irq); sleep "$SECS"; i1=$(irq)
r0=$(field 10); c0=$(cal); j0=$(irq)
rc=0
PING=$(timeout -s INT "$SECS" taskset -c 0 ping -n -f -q 192.0.2.2 2>&1) || rc=$?
[[ $rc == 124 || $rc == 0 ]] || { printf '%s\n' "$PING"; exit "$rc"; }
j1=$(irq); r1=$(field 10); c1=$(cal)
printf '%s\n' "$PING"
printf '%s\n' "$PING" | awk '/packets transmitted/ {ok=($1>0 && $4>0)} END {exit !ok}'
printf 'IDLE_IRQ_MS=%s LOAD_IRQ_MS=%s RPS_CALLBACKS=%s CALL_IPI=%s\n' \
"$(((i1-i0)*1000/HZ))" "$(((j1-j0)*1000/HZ))" \
"$((16#$r1-16#$r0))" "$((c1-c0))"
# A WARN header is one warning; repeated symbol names in its stack are not.
LOG=$(dmesg)
printf 'TARGET_WARNINGS=%s\n' "$(printf '%s\n' "$LOG" | grep -cE 'WARNING:.*do_softirq_post_smp_call_flush' || true)"
# ---- defer_list path: skb alloc on CPU0, freed by a CPU1 reader ----
# kick_defer_list_purge exists only in the patched tree. kprobe_events events
# are created disabled, so each probe must be enabled explicitly.
for spec in 'def_free skb_attempt_defer_free' 'kick kick_defer_list_purge' \
'trig trigger_rx_softirq'; do
set -- $spec
{ echo "p:$1 $2" >> "$TR/kprobe_events"; } 2>/dev/null || echo "PROBE_ABSENT=$2"
{ echo 1 > "$TR/events/kprobes/$1/enable"; } 2>/dev/null || true
done
# A probe that never registered has no counter; report that explicitly rather
# than as a misleading zero.
hits() {
[ -r "$TR/events/kprobes/$1/enable" ] || { echo ABSENT; return; }
awk -v n="$1" '$1==n {print $2}' "$TR/kprobe_profile" 2>/dev/null | grep . || echo 0
}
printf '4\n' > /proc/sys/net/core/skb_defer_max
ip netns exec "$NS" taskset -c 1 iperf3 -s -1 -p 5201 >/dev/null 2>&1 &
SRV=$!
sleep 1
taskset -c 0 iperf3 -c 192.0.2.2 -p 5201 -t "$SECS" -P 4 >/dev/null 2>&1 || true
wait "$SRV" 2>/dev/null || true
sleep 2 # let any pending purge drain before sampling
D=$(hits def_free); K=$(hits kick); T=$(hits trig)
printf 'HITS def_free=%s kick=%s trig=%s\n' "$D" "$K" "$T"
[[ $D -gt 0 ]] && echo 'DEFER_PATH=REACHED' || echo 'DEFER_PATH=NOT_REACHED'
printf '%s\n' '--- softnet_stat ---'; cat /proc/net/softnet_stat
# Only the warning block is evidence; the rest of dmesg is boot noise.
printf '%s\n' '--- WARNING ---'
printf '%s\n' "$LOG" | grep -A20 -E 'WARNING:.*do_softirq_post_smp_call_flush' || trueTwo images are compared: the unmodified base (4c4970859022, no patches) and the patched image (all five patches). Each was booted twice in a fresh guest; the table shows the first run, and the repeat is consistent with it.
Baseline Patched
Idle hardirq (ms) 10 10
Load hardirq (ms) 900 40
received_rps delta 2,390,909 0
Function-call IPIs 2,390,695 947
Echo requests received 2,390,916 2,473,383
Target WARNING records 1 0
skb_attempt_defer_free() 1,075,050 1,558,153
kick_defer_list_purge() absent 193,560
trigger_rx_softirq() 176,249 0
The repeat agrees within noise: baseline load hardirq 930 ms, 2,455,171 received_rps and 2,455,170 function-call IPIs, 1 WARNING; patched 40 ms, 0 received_rps and 825 function-call IPIs, 0 WARNING.
Eric Dumazet (9):
net: napi_schedule_rps() cleanup
net: add softnet_data.in_net_rx_action
net: optimize napi_schedule_rps()
net: optimize ____napi_schedule() to avoid extra NET_RX_SOFTIRQ
net: add debugging checks in skb_attempt_defer_free()
net: do not provide hard irq safety for sd->defer_lock
net: move skb_defer_free_flush() up
net: make napi_threaded_poll() aware of sd->defer_list
net: optimize napi_threaded_poll() vs RPS/RFS
Jesper Dangaard Brouer (1):
net: avoid irqsave in skb_defer_free_flush
Sebastian Andrzej Siewior (4):
net: Remove conditional threaded-NAPI wakeup based on task state.
net: Allow to use SMP threads for backlog NAPI.
net: Use backlog-NAPI to clean up the defer_list.
net: Rename rps_lock to backlog_lock.
Wen Yang (1):
Revert "net: Avoid the IPI to free the"
include/linux/netdevice.h | 9 +- net/core/dev.c | 304 +++++++++++++++++++++++++------------- net/core/skbuff.c | 17 +-- 3 files changed, 215 insertions(+), 115 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/15 Revert "net: Avoid the IPI to free the"
From: Wen Yang <[email protected]>
This reverts commit 8d152d22d9f056f87875f97aec016a65622abbf0.
The RT-only defer_work workaround covers the deferred-free path but not RPS remote-backlog scheduling, which still raises NET_RX_SOFTIRQ from the function-call IPI, so the do_softirq_post_smp_call_flush() warning remains reachable. The following upstream backlog-NAPI thread backport replaces both paths with backlog-thread scheduling, so revert the workaround rather than extend it.
--- include/linux/netdevice.h | 4 ---- net/core/dev.c | 37 +++++++++---------------------------- net/core/skbuff.c | 7 +------ 3 files changed, 10 insertions(+), 38 deletions(-)
include/linux/netdevice.h +0 -4
@@ -3199,11 +3199,7 @@ struct softnet_data {int defer_count;int defer_ipi_scheduled;struct sk_buff *defer_list; -#ifndef CONFIG_PREEMPT_RT call_single_data_t defer_csd; -#else -struct work_struct defer_work; -#endif };static inline void input_queue_head_incr (struct softnet_data *sd)
net/core/dev.c +9 -28
@@ -4712,6 +4712,15 @@ static void rps_trigger_softirq(void *data)#endif /* CONFIG_RPS */ +/* Called from hardirq (IPI) context */ +static void trigger_rx_softirq (void *data) +{ +struct softnet_data *sd = data; + +__raise_softirq_irqoff (NET_RX_SOFTIRQ ); +smp_store_release (&sd->defer_ipi_scheduled,0 ); +} +/* * Check if this softnet_data structure is another cpu one * If yes, queue it to our IPI list and return 1
@@ -6772,30 +6781,6 @@ static void skb_defer_free_flush(struct softnet_data *sd) } } -#ifndef CONFIG_PREEMPT_RT -/* Called from hardirq (IPI) context */ -static void trigger_rx_softirq (void *data) -{ -struct softnet_data *sd = data; - -__raise_softirq_irqoff (NET_RX_SOFTIRQ ); -smp_store_release (&sd->defer_ipi_scheduled,0 ); -} - -#else - -static void trigger_rx_softirq (struct work_struct *defer_work) -{ -struct softnet_data *sd; - - sd = container_of(defer_work, struct softnet_data, defer_work); -smp_store_release (&sd->defer_ipi_scheduled,0 ); -local_bh_disable (); -skb_defer_free_flush (sd); -local_bh_enable (); -} -#endif - static __latent_entropy void net_rx_action(struct softirq_action *h) {struct softnet_data *sd =this_cpu_ptr (&softnet_data);
@@ -11620,11 +11605,7 @@ static int __init net_dev_init(void)INIT_CSD (&sd->csd, rps_trigger_softirq, sd); sd->cpu = i;#endif -#ifndef CONFIG_PREEMPT_RT INIT_CSD (&sd->defer_csd, trigger_rx_softirq, sd); -#else -INIT_WORK (&sd->defer_work, trigger_rx_softirq); -#endif spin_lock_init (&sd->defer_lock);init_gro_hash (&sd->backlog);
net/core/skbuff.c +1 -6
@@ -6746,11 +6746,6 @@ nodefer: __kfree_skb(skb);/* Make sure to trigger NET_RX_SOFTIRQ on the remote CPU * if we are unlucky enough (this seems very unlikely). */ -if (unlikely (kick) && !cmpxchg (&sd->defer_ipi_scheduled,0 ,1 )) { -#ifndef CONFIG_PREEMPT_RT +if (unlikely (kick) && !cmpxchg (&sd->defer_ipi_scheduled,0 ,1 ))smp_call_function_single_async (cpu, &sd->defer_csd); -#else -schedule_work_on (cpu, &sd->defer_work); -#endif - } }
Signature · 1 line
2.25.1
PATCH v2 2/15 net: avoid irqsave in skb_defer_free_flush
In reply to [email protected]From: Jesper Dangaard Brouer <[email protected]>
commit 3176eb82681ec9c8af31c6588ddedcc6cfb9e445 upstream.
The spin_lock irqsave/restore API variant in skb_defer_free_flush can be replaced with the faster spin_lock irq variant, which doesn't need to read and restore the CPU flags.
Using the unconditional irq "disable/enable" API variant is safe, because the skb_defer_free_flush() function is only called during NAPI-RX processing in net_rx_action(), where it is known the IRQs are enabled.
Expected gain is 14 cycles from avoiding reading and restoring CPU flags in a spin_lock_irqsave/restore operation, measured via a microbencmark kernel module[1] on CPU E5-1650 v4 @ 3.60GHz.
Microbenchmark overhead of spin_lock+unlock:
- spin_lock_unlock_irq cost: 34 cycles(tsc) 9.486 ns
- spin_lock_unlock_irqsave cost: 48 cycles(tsc) 13.567 ns
We don't expect to see a measurable packet performance gain, as skb_defer_free_flush() is called infrequently once per NIC device NAPI bulk cycle and conditionally only if SKBs have been deferred by other CPUs via skb_attempt_defer_free().
[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/lib/time_bench_sample.c
--- net/core/dev.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
net/core/dev.c +2 -3
@@ -6762,17 +6762,16 @@ static int napi_threaded_poll(void *data)static void skb_defer_free_flush (struct softnet_data *sd) {struct sk_buff *skb, *next; -unsigned long flags;/* Paired with WRITE_ONCE() in skb_attempt_defer_free() */ if (!READ_ONCE (sd->defer_list))return ; -spin_lock_irqsave (&sd->defer_lock, flags); +spin_lock_irq (&sd->defer_lock); skb = sd->defer_list; sd->defer_list =NULL ; sd->defer_count =0 ; -spin_unlock_irqrestore (&sd->defer_lock, flags); +spin_unlock_irq (&sd->defer_lock);while (skb !=NULL ) { next = skb->next;
Signature · 1 line
2.25.1
PATCH v2 3/15 net: napi_schedule_rps() cleanup
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit 8fcb76b934daff12cde76adeab3d502eeb0734b1 upstream.
napi_schedule_rps() return value is ignored, remove it.
Change the comment to clarify the intent.
--- net/core/dev.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
net/core/dev.c +12 -6
@@ -4722,11 +4722,18 @@ static void trigger_rx_softirq(void *data) }/* -* Check if this softnet_data structure is another cpu one -* If yes, queue it to our IPI list and return 1 -* If no, return 0 +* After we queued a packet into sd->input_pkt_queue, +* we need to make sure this queue is serviced soon. +* +* - If this is another cpu queue, link it to our rps_ipi_list, +* and make sure we will process rps_ipi_list from net_rx_action(). +* As we do not know yet if we are called from net_rx_action(), +* we have to raise NET_RX_SOFTIRQ. This might change in the future. +* +* - If this is our own queue, NAPI schedule our backlog. +* Note that this also raises NET_RX_SOFTIRQ. */ -static int napi_schedule_rps (struct softnet_data *sd) +static void napi_schedule_rps (struct softnet_data *sd) {struct softnet_data *mysd =this_cpu_ptr (&softnet_data);
@@ -4736,11 +4743,10 @@ static int napi_schedule_rps(struct softnet_data *sd) mysd->rps_ipi_list = sd;__raise_softirq_irqoff (NET_RX_SOFTIRQ ); - return 1; + return; }#endif /* CONFIG_RPS */ __napi_schedule_irqoff (&mysd->backlog); -return 0 ; }#ifdef CONFIG_NET_FLOW_LIMIT
Signature · 1 line
2.25.1
PATCH v2 4/15 net: add softnet_data.in_net_rx_action
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit c59647c0dc679008886756a888368da1c6d4ccd3 upstream.
We want to make two optimizations in napi_schedule_rps() and ____napi_schedule() which require to know if these helpers are called from net_rx_action(), instead of being called from other contexts.
sd.in_net_rx_action is only read/written by the owning cpu.
--- include/linux/netdevice.h | 1 + net/core/dev.c | 4 ++++ 2 files changed, 5 insertions(+)
include/linux/netdevice.h +1 -0
@@ -3161,6 +3161,7 @@ struct softnet_data {#ifdef CONFIG_RPS struct softnet_data *rps_ipi_list;#endif +bool in_net_rx_action;#ifdef CONFIG_NET_FLOW_LIMIT struct sd_flow_limit __rcu *flow_limit;#endif
net/core/dev.c +4 -0
@@ -6795,6 +6795,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)LIST_HEAD (list);LIST_HEAD (repoll); + sd->in_net_rx_action = true;local_irq_disable ();list_splice_init (&sd->poll_list, &list);local_irq_enable ();
@@ -6805,6 +6806,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)skb_defer_free_flush (sd);if (list_empty (&list)) { + sd->in_net_rx_action = false;if (!sd_has_rps_ipi_waiting (sd) &&list_empty (&repoll)) goto end;break ;
@@ -6831,6 +6833,8 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)list_splice (&list, &sd->poll_list);if (!list_empty (&sd->poll_list))__raise_softirq_irqoff (NET_RX_SOFTIRQ ); +else + sd->in_net_rx_action = false;net_rps_action_and_irq_enable (sd); end:;
Signature · 1 line
2.25.1
PATCH v2 5/15 net: optimize napi_schedule_rps()
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit 821eba962d95806beb0440742c4062a9da8a386b upstream.
Based on initial patch from Jason Xing.
Idea is to not raise NET_RX_SOFTIRQ from napi_schedule_rps() when we queued a packet into another cpu backlog.
We can do this only in the context of us being called indirectly from net_rx_action(), to have the guarantee our rps_ipi_list will be processed before we exit from net_rx_action().
--- net/core/dev.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
net/core/dev.c +5 -3
@@ -4727,8 +4727,6 @@ static void trigger_rx_softirq(void *data)* * - If this is another cpu queue, link it to our rps_ipi_list, * and make sure we will process rps_ipi_list from net_rx_action(). -* As we do not know yet if we are called from net_rx_action(), -* we have to raise NET_RX_SOFTIRQ. This might change in the future. * * - If this is our own queue, NAPI schedule our backlog. * Note that this also raises NET_RX_SOFTIRQ.
@@ -4742,7 +4740,11 @@ static void napi_schedule_rps(struct softnet_data *sd) sd->rps_ipi_next = mysd->rps_ipi_list; mysd->rps_ipi_list = sd; -__raise_softirq_irqoff (NET_RX_SOFTIRQ ); +/* If not called from net_rx_action() +* we have to raise NET_RX_SOFTIRQ. +*/ +if (!mysd->in_net_rx_action) +__raise_softirq_irqoff (NET_RX_SOFTIRQ ); return; }#endif /* CONFIG_RPS */
Signature · 1 line
2.25.1
PATCH v2 6/15 net: optimize ____napi_schedule() to avoid extra NET_RX_SOFTIRQ
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit 8b43fd3d1d7d88293eb15e92090826e6b7cc13e4 upstream.
____napi_schedule() adds a napi into current cpu softnet_data poll_list, then raises NET_RX_SOFTIRQ to make sure net_rx_action() will process it.
Idea of this patch is to not raise NET_RX_SOFTIRQ when being called indirectly from net_rx_action(), because we can process poll_list from this point, without going to full softirq loop.
This needs a change in net_rx_action() to make sure we restart its main loop if sd->poll_list was updated without NET_RX_SOFTIRQ being raised.
--- net/core/dev.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
net/core/dev.c +18 -4
@@ -4498,7 +4498,11 @@ static inline void ____napi_schedule(struct softnet_data *sd, }list_add_tail (&napi->poll_list, &sd->poll_list); -raise_softirq_irqoff (NET_RX_SOFTIRQ ); +/* If not called from net_rx_action() +* we have to raise NET_RX_SOFTIRQ. +*/ +if (!sd->in_net_rx_action) +raise_softirq_irqoff (NET_RX_SOFTIRQ ); }#ifdef CONFIG_RPS
@@ -6797,6 +6801,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)LIST_HEAD (list);LIST_HEAD (repoll); +start: sd->in_net_rx_action = true;local_irq_disable ();list_splice_init (&sd->poll_list, &list);
@@ -6808,9 +6813,18 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)skb_defer_free_flush (sd);if (list_empty (&list)) { - sd->in_net_rx_action = false; -if (!sd_has_rps_ipi_waiting (sd) &&list_empty (&repoll)) - goto end; +if (list_empty (&repoll)) { + sd->in_net_rx_action = false; +barrier (); +/* We need to check if ____napi_schedule() +* had refilled poll_list while +* sd->in_net_rx_action was true. +*/ +if (!list_empty (&sd->poll_list)) + goto start; +if (!sd_has_rps_ipi_waiting (sd)) + goto end; + }break ; }
Signature · 1 line
2.25.1
PATCH v2 7/15 net: add debugging checks in skb_attempt_defer_free()
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit e8e1ce8454c9cc8ad2e4422bef346428e52455e3 upstream.
Make sure skbs that are stored in softnet_data.defer_list do not have a dst attached.
Also make sure the the skb was orphaned.
--- net/core/skbuff.c | 3 +++ 1 file changed, 3 insertions(+)
net/core/skbuff.c +3 -0
@@ -6727,6 +6727,9 @@ nodefer: __kfree_skb(skb);return ; } +DEBUG_NET_WARN_ON_ONCE (skb_dst (skb)); +DEBUG_NET_WARN_ON_ONCE (skb->destructor); + sd = &per_cpu (softnet_data, cpu); defer_max =READ_ONCE (sysctl_skb_defer_max);if (READ_ONCE (sd->defer_count) >= defer_max)
Signature · 1 line
2.25.1
PATCH v2 8/15 net: do not provide hard irq safety for sd->defer_lock
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit 931e93bdf8ca71cef1f8759c43bc2c5385392b8b upstream.
kfree_skb() can be called from hard irq handlers, but skb_attempt_defer_free() is meant to be used from process or BH contexts, and skb_defer_free_flush() is meant to be called from BH contexts.
Not having to mask hard irq can save some cycles.
--- net/core/dev.c | 4 ++-- net/core/skbuff.c | 5 ++--- 2 files changed, 4 insertions(+), 5 deletions(-)
net/core/dev.c +2 -2
@@ -6779,11 +6779,11 @@ static void skb_defer_free_flush(struct softnet_data *sd)if (!READ_ONCE (sd->defer_list))return ; -spin_lock_irq (&sd->defer_lock); +spin_lock (&sd->defer_lock); skb = sd->defer_list; sd->defer_list =NULL ; sd->defer_count =0 ; -spin_unlock_irq (&sd->defer_lock); +spin_unlock (&sd->defer_lock);while (skb !=NULL ) { next = skb->next;
net/core/skbuff.c +2 -3
@@ -6716,7 +6716,6 @@ void skb_attempt_defer_free(struct sk_buff *skb) {int cpu = skb->alloc_cpu;struct softnet_data *sd; -unsigned long flags;unsigned int defer_max;bool kick;
@@ -6735,7 +6734,7 @@ nodefer: __kfree_skb(skb);if (READ_ONCE (sd->defer_count) >= defer_max) goto nodefer; -spin_lock_irqsave (&sd->defer_lock, flags); +spin_lock_bh (&sd->defer_lock);/* Send an IPI every time queue reaches half capacity. */ kick = sd->defer_count == (defer_max >>1 );/* Paired with the READ_ONCE() few lines above */
@@ -6744,7 +6743,7 @@ nodefer: __kfree_skb(skb); skb->next = sd->defer_list;/* Paired with READ_ONCE() in skb_defer_free_flush() */ WRITE_ONCE (sd->defer_list, skb); -spin_unlock_irqrestore (&sd->defer_lock, flags); +spin_unlock_bh (&sd->defer_lock);/* Make sure to trigger NET_RX_SOFTIRQ on the remote CPU * if we are unlucky enough (this seems very unlikely).
Signature · 1 line
2.25.1
PATCH v2 9/15 net: move skb_defer_free_flush() up
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit e6f50edfef046cf8ad541b4d5972bf38fdcdec39 upstream.
We plan using skb_defer_free_flush() from napi_threaded_poll() in the following patch.
--- net/core/dev.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-)
net/core/dev.c +21 -21
@@ -6742,6 +6742,27 @@ static int napi_thread_wait(struct napi_struct *napi)return -1 ; } +static void skb_defer_free_flush (struct softnet_data *sd) +{ +struct sk_buff *skb, *next; + +/* Paired with WRITE_ONCE() in skb_attempt_defer_free() */ +if (!READ_ONCE (sd->defer_list)) +return ; + +spin_lock (&sd->defer_lock); + skb = sd->defer_list; + sd->defer_list =NULL ; + sd->defer_count =0 ; +spin_unlock (&sd->defer_lock); + +while (skb !=NULL ) { + next = skb->next; +napi_consume_skb (skb,1 ); + skb = next; + } +} +static int napi_threaded_poll (void *data) {struct napi_struct *napi = data;
@@ -6771,27 +6792,6 @@ static int napi_threaded_poll(void *data)return 0 ; } -static void skb_defer_free_flush (struct softnet_data *sd) -{ -struct sk_buff *skb, *next; - -/* Paired with WRITE_ONCE() in skb_attempt_defer_free() */ -if (!READ_ONCE (sd->defer_list)) -return ; - -spin_lock (&sd->defer_lock); - skb = sd->defer_list; - sd->defer_list =NULL ; - sd->defer_count =0 ; -spin_unlock (&sd->defer_lock); - -while (skb !=NULL ) { - next = skb->next; -napi_consume_skb (skb,1 ); - skb = next; - } -} - static __latent_entropy void net_rx_action(struct softirq_action *h) {struct softnet_data *sd =this_cpu_ptr (&softnet_data);
Signature · 1 line
2.25.1
PATCH v2 10/15 net: make napi_threaded_poll() aware of sd->defer_list
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit a1aaee7f8f79d1b0595e24f8c3caed24630d6cb6 upstream.
If we call skb_defer_free_flush() from napi_threaded_poll(), we can avoid to raise IPI from skb_attempt_defer_free() when the list becomes too big.
This allows napi_threaded_poll() to rely less on softirqs, and lowers latency caused by a too big list.
--- net/core/dev.c | 3 +++ 1 file changed, 3 insertions(+)
net/core/dev.c +3 -0
@@ -6766,6 +6766,7 @@ static void skb_defer_free_flush(struct softnet_data *sd)static int napi_threaded_poll (void *data) {struct napi_struct *napi = data; +struct softnet_data *sd;void *have;while (!napi_thread_wait (napi)) {
@@ -6775,11 +6776,13 @@ static int napi_threaded_poll(void *data)bool repoll = false;local_bh_disable (); + sd =this_cpu_ptr (&softnet_data); have =netpoll_poll_lock (napi);__napi_poll (napi, &repoll);netpoll_poll_unlock (have); +skb_defer_free_flush (sd);local_bh_enable ();if (!repoll)
Signature · 1 line
2.25.1
PATCH v2 11/15 net: optimize napi_threaded_poll() vs RPS/RFS
In reply to [email protected]From: Eric Dumazet <[email protected]>
commit 87eff2ec57b6d68d294013d8dd21e839a1175e3a upstream.
We use napi_threaded_poll() in order to reduce our softirq dependency.
We can add a followup of 821eba962d95 ("net: optimize napi_schedule_rps()") to further remove the need of firing NET_RX_SOFTIRQ whenever RPS/RFS are used.
--- include/linux/netdevice.h | 3 +++ net/core/dev.c | 12 ++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/netdevice.h +3 -0
@@ -3161,7 +3161,10 @@ struct softnet_data {#ifdef CONFIG_RPS struct softnet_data *rps_ipi_list;#endif +bool in_net_rx_action; +bool in_napi_threaded_poll; +#ifdef CONFIG_NET_FLOW_LIMIT struct sd_flow_limit __rcu *flow_limit;#endif
net/core/dev.c +10 -2
@@ -4744,10 +4744,10 @@ static void napi_schedule_rps(struct softnet_data *sd) sd->rps_ipi_next = mysd->rps_ipi_list; mysd->rps_ipi_list = sd; -/* If not called from net_rx_action() +/* If not called from net_rx_action() or napi_threaded_poll() * we have to raise NET_RX_SOFTIRQ. */ -if (!mysd->in_net_rx_action) +if (!mysd->in_net_rx_action && !mysd->in_napi_threaded_poll)__raise_softirq_irqoff (NET_RX_SOFTIRQ );return ; }
@@ -6777,11 +6777,19 @@ static int napi_threaded_poll(void *data)local_bh_disable (); sd =this_cpu_ptr (&softnet_data); + sd->in_napi_threaded_poll = true; have =netpoll_poll_lock (napi);__napi_poll (napi, &repoll);netpoll_poll_unlock (have); + sd->in_napi_threaded_poll = false; +barrier (); + +if (sd_has_rps_ipi_waiting (sd)) { +local_irq_disable (); +net_rps_action_and_irq_enable (sd); + }skb_defer_free_flush (sd);local_bh_enable ();
Signature · 1 line
2.25.1
PATCH v2 12/15 net: Remove conditional threaded-NAPI wakeup based on task state.
In reply to [email protected]From: Sebastian Andrzej Siewior <[email protected]>
commit 56364c910691f6d10ba88c964c9041b9ab777bd6 upstream.
A NAPI thread is scheduled by first setting NAPI_STATE_SCHED bit. If successful (the bit was not yet set) then the NAPI_STATE_SCHED_THREADED is set but only if thread's state is not TASK_INTERRUPTIBLE (is TASK_RUNNING) followed by task wakeup.
If the task is idle (TASK_INTERRUPTIBLE) then the NAPI_STATE_SCHED_THREADED bit is not set. The thread is no relying on the bit but always leaving the wait-loop after returning from schedule() because there must have been a wakeup.
The smpboot-threads implementation for per-CPU threads requires an explicit condition and does not support "if we get out of schedule() then there must be something to do".
Removing this optimisation simplifies the following integration.
Set NAPI_STATE_SCHED_THREADED unconditionally on wakeup and rely on it in the wait path by removing the `woken' condition.
--- net/core/dev.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)
net/core/dev.c +2 -12
@@ -4485,13 +4485,7 @@ static inline void ____napi_schedule(struct softnet_data *sd,*/ thread =READ_ONCE (napi->thread);if (thread) { -/* Avoid doing set_bit() if the thread is in -* INTERRUPTIBLE state, cause napi_thread_wait() -* makes sure to proceed with napi polling -* if the thread is explicitly woken from here. -*/ -if (READ_ONCE (thread->__state) !=TASK_INTERRUPTIBLE ) -set_bit (NAPI_STATE_SCHED_THREADED , &napi->state); +set_bit (NAPI_STATE_SCHED_THREADED , &napi->state);wake_up_process (thread);return ; }
@@ -6716,8 +6710,6 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll)static int napi_thread_wait (struct napi_struct *napi) { -bool woken = false; -set_current_state (TASK_INTERRUPTIBLE );while (!kthread_should_stop ()) {
@@ -6726,15 +6718,13 @@ static int napi_thread_wait(struct napi_struct *napi)* Testing SCHED bit is not enough because SCHED bit might be * set by some other busy poll thread or by napi_disable(). */ -if (test_bit (NAPI_STATE_SCHED_THREADED , &napi->state) || woken) { +if (test_bit (NAPI_STATE_SCHED_THREADED , &napi->state)) {WARN_ON (!list_empty (&napi->poll_list));__set_current_state (TASK_RUNNING );return 0 ; }schedule (); -/* woken being true indicates this thread owns this napi. */ - woken = true;set_current_state (TASK_INTERRUPTIBLE ); }__set_current_state (TASK_RUNNING );
Signature · 1 line
2.25.1
PATCH v2 13/15 net: Allow to use SMP threads for backlog NAPI.
In reply to [email protected]From: Sebastian Andrzej Siewior <[email protected]>
commit dad6b97702639fba27a2bd3e986982ad6f0db3a7 upstream.
Backlog NAPI is a per-CPU NAPI struct only (with no device behind it) used by drivers which don't do NAPI them self, RPS and parts of the stack which need to avoid recursive deadlocks while processing a packet.
The non-NAPI driver use the CPU local backlog NAPI. If RPS is enabled then a flow for the skb is computed and based on the flow the skb can be enqueued on a remote CPU. Scheduling/ raising the softirq (for backlog's NAPI) on the remote CPU isn't trivial because the softirq is only scheduled on the local CPU and performed after the hardirq is done. In order to schedule a softirq on the remote CPU, an IPI is sent to the remote CPU which schedules the backlog-NAPI on the then local CPU.
On PREEMPT_RT interrupts are force-threaded. The soft interrupts are raised within the interrupt thread and processed after the interrupt handler completed still within the context of the interrupt thread. The softirq is handled in the context where it originated.
With force-threaded interrupts enabled, ksoftirqd is woken up if a softirq is raised from hardirq context. This is the case if it is raised from an IPI. Additionally there is a warning on PREEMPT_RT if the softirq is raised from the idle thread.
This was done for two reasons:
- With threaded interrupts the processing should happen in thread context (where it originated) and ksoftirqd is the only thread for this context if raised from hardirq. Using the currently running task instead would "punish" a random task.
- Once ksoftirqd is active it consumes all further softirqs until it stops running. This changed recently and is no longer the case.
Instead of keeping the backlog NAPI in ksoftirqd (in force-threaded/ PREEMPT_RT setups) I am proposing NAPI-threads for backlog. The "proper" setup with threaded-NAPI is not doable because the threads are not pinned to an individual CPU and can be modified by the user. Additionally a dummy network device would have to be assigned. Also CPU-hotplug has to be considered if additional CPUs show up. All this can be probably done/ solved but the smpboot-threads already provide this infrastructure.
Sending UDP packets over loopback expects that the packet is processed within the call. Delaying it by handing it over to the thread hurts performance. It is not beneficial to the outcome if the context switch happens immediately after enqueue or after a while to process a few packets in a batch.
There is no need to always use the thread if the backlog NAPI is requested on the local CPU. This restores the loopback throuput. The performance drops mostly to the same value after enabling RPS on the loopback comparing the IPI and the tread result.
Create NAPI-threads for backlog if request during boot. The thread runs the inner loop from napi_threaded_poll(), the wait part is different. It checks for NAPI_STATE_SCHED (the backlog NAPI can not be disabled).
The NAPI threads for backlog are optional, it has to be enabled via the boot argument "thread_backlog_napi". It is mandatory for PREEMPT_RT to avoid the wakeup of ksoftirqd from the IPI.
6.1-rt port notes:
- The upstream context for this patch is the v6.9 net/core/dev.c. Five hunks could not be applied verbatim against the 6.1.y base and were applied by hand; each one is a pure base difference, not a behavioural change:
* backlog_napi percpu definition: the 6.1.y anchor is
"int netdev_max_backlog" instead of "unsigned int sysctl_skb_defer_max".
* ____napi_schedule(): the WRITE_ONCE(napi->list_owner, ...) line added by "net: add napi_struct.list_owner" does not exist on 6.1.y, so only the "use_local_napi" goto/label was added.
* napi_threaded_poll() split: applied against the 6.1.y function body which now matches the upstream pre-image after the prerequisite commits.
* smpboot block and net_dev_init(): the 6.1.y net_dev_init() has no net_page_pool_create() call, so the block was anchored on the comment preceding net_dev_init() instead.
--- net/core/dev.c | 148 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 113 insertions(+), 35 deletions(-)
net/core/dev.c +113 -35
@@ -78,6 +78,7 @@#include <linux/slab.h> #include <linux/sched.h> #include <linux/sched/mm.h> +#include <linux/smpboot.h> #include <linux/mutex.h> #include <linux/rwsem.h> #include <linux/string.h>
@@ -217,6 +218,31 @@ static inline struct hlist_head *dev_index_hash(struct net *net, int ifindex)return &net->dev_index_head[ifindex & (NETDEV_HASHENTRIES -1 )]; } +#ifndef CONFIG_PREEMPT_RT + +static DEFINE_STATIC_KEY_FALSE (use_backlog_threads_key); + +static int __init setup_backlog_napi_threads (char *arg) +{ +static_branch_enable (&use_backlog_threads_key); +return 0 ; +} +early_param ("thread_backlog_napi" , setup_backlog_napi_threads); + +static bool use_backlog_threads (void ) +{ +return static_branch_unlikely (&use_backlog_threads_key); +} + +#else + +static bool use_backlog_threads (void ) +{ +return true; +} + +#endif +static inline void rps_lock_irqsave (struct softnet_data *sd,unsigned long *flags) {
@@ -4453,6 +4479,7 @@ EXPORT_SYMBOL(__dev_direct_xmit);/************************************************************************* * Receiver routines *************************************************************************/ +static DEFINE_PER_CPU(struct task_struct *, backlog_napi);int netdev_max_backlog__read_mostly =1000 ;EXPORT_SYMBOL (netdev_max_backlog);
@@ -4485,12 +4512,16 @@ static inline void ____napi_schedule(struct softnet_data *sd,*/ thread =READ_ONCE (napi->thread);if (thread) { +if (use_backlog_threads () && thread ==raw_cpu_read (backlog_napi)) + goto use_local_napi; +set_bit (NAPI_STATE_SCHED_THREADED , &napi->state);wake_up_process (thread);return ; } } +use_local_napi:list_add_tail (&napi->poll_list, &sd->poll_list);/* If not called from net_rx_action() * we have to raise NET_RX_SOFTIRQ.
@@ -4735,6 +4766,11 @@ static void napi_schedule_rps(struct softnet_data *sd)#ifdef CONFIG_RPS if (sd != mysd) { +if (use_backlog_threads ()) { +__napi_schedule_irqoff (&sd->backlog); +return ; + } + sd->rps_ipi_next = mysd->rps_ipi_list; mysd->rps_ipi_list = sd;
@@ -6023,7 +6059,7 @@ static void net_rps_action_and_irq_enable(struct softnet_data *sd)#ifdef CONFIG_RPS struct softnet_data *remsd = sd->rps_ipi_list; -if (remsd) { +if (!use_backlog_threads () && remsd) { sd->rps_ipi_list =NULL ;local_irq_enable ();
@@ -6038,7 +6074,7 @@ static void net_rps_action_and_irq_enable(struct softnet_data *sd)static bool sd_has_rps_ipi_waiting (struct softnet_data *sd) {#ifdef CONFIG_RPS -return sd->rps_ipi_list !=NULL ; +return !use_backlog_threads () && sd->rps_ipi_list;#else return false;#endif
@@ -6082,7 +6118,7 @@ static int process_backlog(struct napi_struct *napi, int quota)* We can use a plain write instead of clear_bit(), * and we dont need an smp_mb() memory barrier. */ - napi->state =0 ; + napi->state &=NAPIF_STATE_THREADED ; again = false; }else { skb_queue_splice_tail_init(&sd->input_pkt_queue,
@@ -6753,43 +6789,48 @@ static void skb_defer_free_flush(struct softnet_data *sd) } } -static int napi_threaded_poll (void *data) +static void napi_threaded_poll_loop (struct napi_struct *napi) { -struct napi_struct *napi = data;struct softnet_data *sd; -void *have; - -while (!napi_thread_wait (napi)) { -unsigned long last_qs = jiffies; +unsigned long last_qs = jiffies; -for (;;) { -bool repoll = false; +for (;;) { +bool repoll = false; +void *have; -local_bh_disable (); - sd =this_cpu_ptr (&softnet_data); - sd->in_napi_threaded_poll = true; +local_bh_disable (); + sd =this_cpu_ptr (&softnet_data); + sd->in_napi_threaded_poll = true; - have =netpoll_poll_lock (napi); -__napi_poll (napi, &repoll); -netpoll_poll_unlock (have); + have =netpoll_poll_lock (napi); +__napi_poll (napi, &repoll); +netpoll_poll_unlock (have); - sd->in_napi_threaded_poll = false; -barrier (); + sd->in_napi_threaded_poll = false; +barrier (); -if (sd_has_rps_ipi_waiting (sd)) { -local_irq_disable (); -net_rps_action_and_irq_enable (sd); - } -skb_defer_free_flush (sd); -local_bh_enable (); +if (sd_has_rps_ipi_waiting (sd)) { +local_irq_disable (); +net_rps_action_and_irq_enable (sd); + } +skb_defer_free_flush (sd); +local_bh_enable (); -if (!repoll) -break ; +if (!repoll) +break ; -rcu_softirq_qs_periodic (last_qs); -cond_resched (); - } +rcu_softirq_qs_periodic (last_qs); +cond_resched (); } +} + +static int napi_threaded_poll (void *data) +{ +struct napi_struct *napi = data; + +while (!napi_thread_wait (napi)) +napi_threaded_poll_loop (napi); +return 0 ; }
@@ -11314,7 +11355,7 @@ static int dev_cpu_dead(unsigned int oldcpu)list_del_init (&napi->poll_list);if (napi->poll == process_backlog) - napi->state =0 ; + napi->state &=NAPIF_STATE_THREADED ;else ____napi_schedule (sd, napi); }
@@ -11322,12 +11363,14 @@ static int dev_cpu_dead(unsigned int oldcpu)raise_softirq_irqoff (NET_TX_SOFTIRQ );local_irq_enable (); +if (!use_backlog_threads ()) {#ifdef CONFIG_RPS - remsd = oldsd->rps_ipi_list; - oldsd->rps_ipi_list =NULL ; + remsd = oldsd->rps_ipi_list; + oldsd->rps_ipi_list =NULL ;#endif -/* send out pending IPI's on offline CPU */ -net_rps_send_ipi (remsd); +/* send out pending IPI's on offline CPU */ +net_rps_send_ipi (remsd); + }/* Process offline CPU's input_pkt_queue */ while ((skb =__skb_dequeue (&oldsd->process_queue))) {
@@ -11587,6 +11630,38 @@ static struct pernet_operations __net_initdata default_device_ops = {* */ +static int backlog_napi_should_run (unsigned int cpu) +{ +struct softnet_data *sd =per_cpu_ptr (&softnet_data, cpu); +struct napi_struct *napi = &sd->backlog; + +return test_bit (NAPI_STATE_SCHED_THREADED , &napi->state); +} + +static void run_backlog_napi (unsigned int cpu) +{ +struct softnet_data *sd =per_cpu_ptr (&softnet_data, cpu); + +napi_threaded_poll_loop (&sd->backlog); +} + +static void backlog_napi_setup (unsigned int cpu) +{ +struct softnet_data *sd =per_cpu_ptr (&softnet_data, cpu); +struct napi_struct *napi = &sd->backlog; + + napi->thread =this_cpu_read (backlog_napi); +set_bit (NAPI_STATE_THREADED , &napi->state); +} + +static struct smp_hotplug_thread backlog_threads = { + .store = &backlog_napi, + .thread_should_run = backlog_napi_should_run, + .thread_fn = run_backlog_napi, + .thread_comm ="backlog_napi/%u" , + .setup = backlog_napi_setup, +}; +/* * This is called single threaded during boot, so no need * to take the rtnl semaphore.
@@ -11637,7 +11712,10 @@ static int __init net_dev_init(void)init_gro_hash (&sd->backlog); sd->backlog.poll = process_backlog; sd->backlog.weight = weight_p; +INIT_LIST_HEAD (&sd->backlog.poll_list); } +if (use_backlog_threads ()) +smpboot_register_percpu_thread (&backlog_threads); dev_boot_phase =0 ;
Signature · 1 line
2.25.1
PATCH v2 14/15 net: Use backlog-NAPI to clean up the defer_list.
In reply to [email protected]From: Sebastian Andrzej Siewior <[email protected]>
commit 80d2eefcb4c84aa9018b2a997ab3a4c567bc821a upstream.
The defer_list is a per-CPU list which is used to free skbs outside of the socket lock and on the CPU on which they have been allocated. The list is processed during NAPI callbacks so ideally the list is cleaned up.
Should the amount of skbs on the list exceed a certain water mark then the softirq is triggered remotely on the target CPU by invoking a remote function call. The raise of the softirqs via a remote function call leads to waking the ksoftirqd on PREEMPT_RT which is undesired. The backlog-NAPI threads already provide the infrastructure which can be utilized to perform the cleanup of the defer_list.
The NAPI state is updated with the input_pkt_queue.lock acquired. It order not to break the state, it is needed to also wake the backlog-NAPI thread with the lock held. This requires to acquire the use the lock in rps_lock_irq*() if the backlog-NAPI threads are used even with RPS disabled.
Move the logic of remotely starting softirqs to clean up the defer_list into kick_defer_list_purge(). Make sure a lock is held in rps_lock_irq*() if backlog-NAPI threads are used. Schedule backlog-NAPI for defer_list cleanup if backlog-NAPI is available.
--- include/linux/netdevice.h | 1 + net/core/dev.c | 25 +++++++++++++++++++++---- net/core/skbuff.c | 4 ++-- 3 files changed, 24 insertions(+), 6 deletions(-)
include/linux/netdevice.h +1 -0
@@ -3245,6 +3245,7 @@ static inline void dev_xmit_recursion_dec(void)__this_cpu_dec (softnet_data.xmit.recursion); } +void kick_defer_list_purge (struct softnet_data *sd,unsigned int cpu);void __netif_schedule (struct Qdisc *q);void netif_schedule_queue (struct netdev_queue *txq);
net/core/dev.c +21 -4
@@ -246,7 +246,7 @@ static bool use_backlog_threads(void)static inline void rps_lock_irqsave (struct softnet_data *sd,unsigned long *flags) { -if (IS_ENABLED (CONFIG_RPS )) +if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_lock_irqsave (&sd->input_pkt_queue.lock, *flags);else if (!IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_save (*flags);
@@ -254,7 +254,7 @@ static inline void rps_lock_irqsave(struct softnet_data *sd,static inline void rps_lock_irq_disable (struct softnet_data *sd) { -if (IS_ENABLED (CONFIG_RPS )) +if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_lock_irq (&sd->input_pkt_queue.lock);else if (!IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_disable ();
@@ -263,7 +263,7 @@ static inline void rps_lock_irq_disable(struct softnet_data *sd)static inline void rps_unlock_irq_restore (struct softnet_data *sd,unsigned long *flags) { -if (IS_ENABLED (CONFIG_RPS )) +if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_unlock_irqrestore (&sd->input_pkt_queue.lock, *flags);else if (!IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_restore (*flags);
@@ -271,7 +271,7 @@ static inline void rps_unlock_irq_restore(struct softnet_data *sd,static inline void rps_unlock_irq_enable (struct softnet_data *sd) { -if (IS_ENABLED (CONFIG_RPS )) +if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_unlock_irq (&sd->input_pkt_queue.lock);else if (!IS_ENABLED (CONFIG_PREEMPT_RT ))local_irq_enable ();
@@ -4785,6 +4785,23 @@ static void napi_schedule_rps(struct softnet_data *sd)__napi_schedule_irqoff (&mysd->backlog); } +void kick_defer_list_purge (struct softnet_data *sd,unsigned int cpu) +{ +unsigned long flags; + +if (use_backlog_threads ()) { +rps_lock_irqsave (sd, &flags); + +if (!__test_and_set_bit (NAPI_STATE_SCHED , &sd->backlog.state)) +__napi_schedule_irqoff (&sd->backlog); + +rps_unlock_irq_restore (sd, &flags); + + }else if (!cmpxchg (&sd->defer_ipi_scheduled,0 ,1 )) { +smp_call_function_single_async (cpu, &sd->defer_csd); + } +} +#ifdef CONFIG_NET_FLOW_LIMIT int netdev_flow_limit_table_len__read_mostly = (1 <<12 );#endif
net/core/skbuff.c +2 -2
@@ -6748,6 +6748,6 @@ nodefer: __kfree_skb(skb);/* Make sure to trigger NET_RX_SOFTIRQ on the remote CPU * if we are unlucky enough (this seems very unlikely). */ -if (unlikely (kick) && !cmpxchg (&sd->defer_ipi_scheduled,0 ,1 )) -smp_call_function_single_async (cpu, &sd->defer_csd); +if (unlikely (kick)) +kick_defer_list_purge (sd, cpu); }
Signature · 1 line
2.25.1
PATCH v2 15/15 net: Rename rps_lock to backlog_lock.
In reply to [email protected]From: Sebastian Andrzej Siewior <[email protected]>
commit 765b11f8f4e20b7433e4ba4a3e9106a0d59501ed upstream.
The rps_lock.*() functions use the inner lock of a sk_buff_head for locking. This lock is used if RPS is enabled, otherwise the list is accessed lockless and disabling interrupts is enough for the synchronisation because it is only accessed CPU local. Not only the list is protected but also the NAPI state protected.
With the addition of backlog threads, the lock is also needed because of the cross CPU access even without RPS. The clean up of the defer_list list is also done via backlog threads (if enabled).
It has been suggested to rename the locking function since it is no longer just RPS.
Rename the rps_lock*() functions to backlog_lock*().
--- net/core/dev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
net/core/dev.c +17 -17
@@ -243,8 +243,8 @@ static bool use_backlog_threads(void)#endif -static inline void rps_lock_irqsave (struct softnet_data *sd, -unsigned long *flags) +static inline void backlog_lock_irq_save (struct softnet_data *sd, +unsigned long *flags) {if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_lock_irqsave (&sd->input_pkt_queue.lock, *flags);
@@ -252,7 +252,7 @@ static inline void rps_lock_irqsave(struct softnet_data *sd,local_irq_save (*flags); } -static inline void rps_lock_irq_disable (struct softnet_data *sd) +static inline void backlog_lock_irq_disable (struct softnet_data *sd) {if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_lock_irq (&sd->input_pkt_queue.lock);
@@ -260,8 +260,8 @@ static inline void rps_lock_irq_disable(struct softnet_data *sd)local_irq_disable (); } -static inline void rps_unlock_irq_restore (struct softnet_data *sd, -unsigned long *flags) +static inline void backlog_unlock_irq_restore (struct softnet_data *sd, +unsigned long *flags) {if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_unlock_irqrestore (&sd->input_pkt_queue.lock, *flags);
@@ -269,7 +269,7 @@ static inline void rps_unlock_irq_restore(struct softnet_data *sd,local_irq_restore (*flags); } -static inline void rps_unlock_irq_enable (struct softnet_data *sd) +static inline void backlog_unlock_irq_enable (struct softnet_data *sd) {if (IS_ENABLED (CONFIG_RPS ) ||use_backlog_threads ())spin_unlock_irq (&sd->input_pkt_queue.lock);
@@ -4790,12 +4790,12 @@ void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu)unsigned long flags;if (use_backlog_threads ()) { -rps_lock_irqsave (sd, &flags); +backlog_lock_irq_save (sd, &flags);if (!__test_and_set_bit (NAPI_STATE_SCHED , &sd->backlog.state))__napi_schedule_irqoff (&sd->backlog); -rps_unlock_irq_restore (sd, &flags); +backlog_unlock_irq_restore (sd, &flags); }else if (!cmpxchg (&sd->defer_ipi_scheduled,0 ,1 )) {smp_call_function_single_async (cpu, &sd->defer_csd);
@@ -4857,7 +4857,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, reason =SKB_DROP_REASON_NOT_SPECIFIED ; sd = &per_cpu (softnet_data, cpu); -rps_lock_irqsave (sd, &flags); +backlog_lock_irq_save (sd, &flags);if (!netif_running (skb->dev)) goto drop; qlen =skb_queue_len (&sd->input_pkt_queue);
@@ -4866,7 +4866,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, enqueue:__skb_queue_tail (&sd->input_pkt_queue, skb);input_queue_tail_incr_save (sd, qtail); -rps_unlock_irq_restore (sd, &flags); +backlog_unlock_irq_restore (sd, &flags);return NET_RX_SUCCESS ; }
@@ -4881,7 +4881,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, drop: sd->dropped++; -rps_unlock_irq_restore (sd, &flags); +backlog_unlock_irq_restore (sd, &flags);dev_core_stats_rx_dropped_inc (skb->dev);kfree_skb_reason (skb, reason);
@@ -5977,7 +5977,7 @@ static void flush_backlog(struct work_struct *work)local_bh_disable (); sd =this_cpu_ptr (&softnet_data); -rps_lock_irq_disable (sd); +backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) {if (skb->dev->reg_state ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue);
@@ -5985,7 +5985,7 @@ static void flush_backlog(struct work_struct *work)input_queue_head_incr (sd); } } -rps_unlock_irq_enable (sd); +backlog_unlock_irq_enable (sd); skb_queue_walk_safe(&sd->process_queue, skb, tmp) {if (skb->dev->reg_state ==NETREG_UNREGISTERING ) {
@@ -6003,14 +6003,14 @@ static bool flush_required(int cpu)struct softnet_data *sd = &per_cpu (softnet_data, cpu);bool do_flush; -rps_lock_irq_disable (sd); +backlog_lock_irq_disable (sd);/* as insertion into process_queue happens with the rps lock held, * process_queue access may race only with dequeue */ do_flush = !skb_queue_empty (&sd->input_pkt_queue) || !skb_queue_empty_lockless (&sd->process_queue); -rps_unlock_irq_enable (sd); +backlog_unlock_irq_enable (sd);return do_flush;#endif
@@ -6125,7 +6125,7 @@ static int process_backlog(struct napi_struct *napi, int quota) } -rps_lock_irq_disable (sd); +backlog_lock_irq_disable (sd);if (skb_queue_empty (&sd->input_pkt_queue)) {/* * Inline a custom version of __napi_complete().
@@ -6141,7 +6141,7 @@ static int process_backlog(struct napi_struct *napi, int quota)skb_queue_splice_tail_init (&sd->input_pkt_queue, &sd->process_queue); } -rps_unlock_irq_enable (sd); +backlog_unlock_irq_enable (sd); }return work;
Signature · 1 line
2.25.1
Sebastian Andrzej Siewior
maintainerQuoted 4 lines · On 2026-10-24 21:30:24 [+0800], [email protected] wrote:
From: Wen Yang <[email protected]>
Backport the backlog-NAPI smpboot-thread mechanism and its prerequisite NAPI/RPS/defer_list work from upstream (v6.4 - v6.9) to v6.1-rt-next.
This looks okay. The changes already landed in v6.6-RT so this could land in v6.1-RT.
Sebastian