Skip to contentSkip to conversation

net: optimize enqueue_to_backlog() for the fast path

PATCH Applied Eric Dumazet 4 participants 6 messages Oct 24, 2025 – Oct 29, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: optimize enqueue_to_backlog() for the fast path

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

skb_flow_limit() does not need the backlog_lock protection, and can be called before we acquire the lock, for even better resistance to attacks.

Cc Willem de Bruijn <[email protected]>
---
 net/core/dev.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/core/dev.c +10 -8
index 378c2d010faf251ffd874ebf0cc3dd6968eee447..d32f0b0c03bbd069d3651f5a6b772c8029baf96c 100644
@@ -5249,14 +5249,15 @@ void kick_defer_list_purge(unsigned int cpu)
 int netdev_flow_limit_table_len __read_mostly = (1 << 12);
 #endif
 
-static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen)
+static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen,
+			   int max_backlog)
 {
 #ifdef CONFIG_NET_FLOW_LIMIT
-	struct sd_flow_limit *fl;
-	struct softnet_data *sd;
 	unsigned int old_flow, new_flow;
+	const struct softnet_data *sd;
+	struct sd_flow_limit *fl;
 
-	if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1))
+	if (likely(qlen < (max_backlog >> 1)))
 		return false;
 
 	sd = this_cpu_ptr(&softnet_data);
@@ -5301,19 +5302,19 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
 	u32 tail;
 
 	reason = SKB_DROP_REASON_DEV_READY;
-	if (!netif_running(skb->dev))
+	if (unlikely(!netif_running(skb->dev)))
 		goto bad_dev;
 
-	reason = SKB_DROP_REASON_CPU_BACKLOG;
 	sd = &per_cpu(softnet_data, cpu);
 
 	qlen = skb_queue_len_lockless(&sd->input_pkt_queue);
 	max_backlog = READ_ONCE(net_hotdata.max_backlog);
-	if (unlikely(qlen > max_backlog))
+	if (unlikely(qlen > max_backlog) ||
+	    skb_flow_limit(skb, qlen, max_backlog))
 		goto cpu_backlog_drop;
 	backlog_lock_irq_save(sd, &flags);
 	qlen = skb_queue_len(&sd->input_pkt_queue);
-	if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) {
+	if (likely(qlen <= max_backlog)) {
 		if (!qlen) {
 			/* Schedule NAPI for backlog device. We can use
 			 * non atomic operation as we own the queue lock.
@@ -5334,6 +5335,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
 	backlog_unlock_irq_restore(sd, &flags);
 
 cpu_backlog_drop:
+	reason = SKB_DROP_REASON_CPU_BACKLOG;
 	numa_drop_add(&sd->drop_counters, 1);
 bad_dev:
 	dev_core_stats_rx_dropped_inc(skb->dev);
Signature · 1 line
2.51.1.821.gb6fe4d2222-goog
#1 lore raw
Quoted 33 lines · Eric Dumazet

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

skb_flow_limit() does not need the backlog_lock protection, and can be called before we acquire the lock, for even better resistance to attacks.

Cc Willem de Bruijn <[email protected]>
---
 net/core/dev.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/core/dev.c +2 -2
index 378c2d010faf251ffd874ebf0cc3dd6968eee447..d32f0b0c03bbd069d3651f5a6b772c8029baf96c 100644
@@ -5249,14 +5249,15 @@ void kick_defer_list_purge(unsigned int cpu)
 int netdev_flow_limit_table_len __read_mostly = (1 << 12);
 #endif
 
-static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen)
+static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen,
+			   int max_backlog)
 {
 #ifdef CONFIG_NET_FLOW_LIMIT
-	struct sd_flow_limit *fl;

net/core/dev.c void kick_defer_list_purge(unsigned int cpu) · line 5263

Quoted 10 lines
net/core/dev.c +3 -2
-	struct softnet_data *sd;
 	unsigned int old_flow, new_flow;
+	const struct softnet_data *sd;
+	struct sd_flow_limit *fl;
 
-	if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1))
+	if (likely(qlen < (max_backlog >> 1)))
 		return false;
 
 	sd = this_cpu_ptr(&softnet_data);

I assume sd is warm here. Else we could even move skb_flow_limit behind a static_branch seeing how rarely it is likely used.

net/core/dev.c static int enqueue_to_backlog(struct sk_buff *skb, int cpu, · line 5306

Quoted 7 lines
net/core/dev.c +1 -1
@@ -5301,19 +5302,19 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
 	u32 tail;
 
 	reason = SKB_DROP_REASON_DEV_READY;
-	if (!netif_running(skb->dev))
+	if (unlikely(!netif_running(skb->dev)))
 		goto bad_dev;

Isn't unlikely usually predicted for branches without an else?

And that is ignoring both FDO and actual branch prediction hardware improving on the simple compiler heuristic.

No immediately concerns. Just want to avoid precedence for others to sprinkle code with likely/unlikely with abandon. As is sometimes seen.

Quoted 28 lines
net/core/dev.c +4 -3
-	reason = SKB_DROP_REASON_CPU_BACKLOG;
 	sd = &per_cpu(softnet_data, cpu);
 
 	qlen = skb_queue_len_lockless(&sd->input_pkt_queue);
 	max_backlog = READ_ONCE(net_hotdata.max_backlog);
-	if (unlikely(qlen > max_backlog))
+	if (unlikely(qlen > max_backlog) ||
+	    skb_flow_limit(skb, qlen, max_backlog))
 		goto cpu_backlog_drop;
 	backlog_lock_irq_save(sd, &flags);
 	qlen = skb_queue_len(&sd->input_pkt_queue);
-	if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) {
+	if (likely(qlen <= max_backlog)) {
 		if (!qlen) {
 			/* Schedule NAPI for backlog device. We can use
 			 * non atomic operation as we own the queue lock.
@@ -5334,6 +5335,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
 	backlog_unlock_irq_restore(sd, &flags);
 
 cpu_backlog_drop:
+	reason = SKB_DROP_REASON_CPU_BACKLOG;
 	numa_drop_add(&sd->drop_counters, 1);
 bad_dev:
 	dev_core_stats_rx_dropped_inc(skb->dev);
Signature · 1 line
2.51.1.821.gb6fe4d2222-goog
#2 lore raw
Quoted 40 lines · Willem de Bruijn
Quoted 38 lines · Eric Dumazet

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

skb_flow_limit() does not need the backlog_lock protection, and can be called before we acquire the lock, for even better resistance to attacks.

Cc Willem de Bruijn <[email protected]>
---
 net/core/dev.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/core/dev.c +4 -3
index 378c2d010faf251ffd874ebf0cc3dd6968eee447..d32f0b0c03bbd069d3651f5a6b772c8029baf96c 100644
@@ -5249,14 +5249,15 @@ void kick_defer_list_purge(unsigned int cpu)
 int netdev_flow_limit_table_len __read_mostly = (1 << 12);
 #endif
 
-static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen)
+static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen,
+                        int max_backlog)
 {
 #ifdef CONFIG_NET_FLOW_LIMIT
-     struct sd_flow_limit *fl;
-     struct softnet_data *sd;
      unsigned int old_flow, new_flow;
+     const struct softnet_data *sd;
+     struct sd_flow_limit *fl;
 
Quoted 8 lines

net/core/dev.c void kick_defer_list_purge(unsigned int cpu) · line 5263

Quoted 5 lines
net/core/dev.c +1 -1
-     if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1))
+     if (likely(qlen < (max_backlog >> 1)))
              return false;
 
      sd = this_cpu_ptr(&softnet_data);

I assume sd is warm here. Else we could even move skb_flow_limit behind a static_branch seeing how rarely it is likely used.

this_cpu_ptr(&ANY_VAR) only loads very hot this_cpu_off. In modern kernels this is

DEFINE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off);

rest is in the offsets used in the code.

Quoted 10 lines

net/core/dev.c static int enqueue_to_backlog(struct sk_buff *skb, int cpu, · line 5306

Quoted 8 lines
net/core/dev.c +1 -1
@@ -5301,19 +5302,19 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
      u32 tail;
 
      reason = SKB_DROP_REASON_DEV_READY;
-     if (!netif_running(skb->dev))
+     if (unlikely(!netif_running(skb->dev)))
              goto bad_dev;

Isn't unlikely usually predicted for branches without an else?

I am not sure this is a hardcoded rule that all compilers will stick with. Do you have a reference ?

Quoted 3 lines

And that is ignoring both FDO and actual branch prediction hardware improving on the simple compiler heuristic.

Lets not assume FDO is always used, and close the gap. This will allow us to iterate faster.
FDO brings its own class of problems...

Quoted 4 lines

No immediately concerns. Just want to avoid precedence for others to sprinkle code with likely/unlikely with abandon. As is sometimes seen.

Sure.

I have not included a change on the apparently _very_ expensive

if (!__test_and_set_bit(NAPI_STATE_SCHED,

                                    &sd->backlog.state))

btsq $0x0,0x160(%r13)

I tried to test the bit, then set it if needed, but got no improvement, for some reason
(This was after the other patch making sure to group the dirtied fields in a single cache line)

Reviewed-by
#3 lore raw
Quoted 13 lines · Eric Dumazet
Quoted 11 lines · Willem de Bruijn
Quoted 9 lines · Eric Dumazet

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

skb_flow_limit() does not need the backlog_lock protection, and can be called before we acquire the lock, for even better resistance to attacks.

Cc Willem de Bruijn <[email protected]>
Quoted 50 lines
Quoted 23 lines
Quoted 23 lines
---
 net/core/dev.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/core/dev.c +4 -3
index 378c2d010faf251ffd874ebf0cc3dd6968eee447..d32f0b0c03bbd069d3651f5a6b772c8029baf96c 100644
@@ -5249,14 +5249,15 @@ void kick_defer_list_purge(unsigned int cpu)
 int netdev_flow_limit_table_len __read_mostly = (1 << 12);
 #endif
 
-static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen)
+static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen,
+                        int max_backlog)
 {
 #ifdef CONFIG_NET_FLOW_LIMIT
-     struct sd_flow_limit *fl;
-     struct softnet_data *sd;
      unsigned int old_flow, new_flow;
+     const struct softnet_data *sd;
+     struct sd_flow_limit *fl;
 
Quoted 8 lines

net/core/dev.c void kick_defer_list_purge(unsigned int cpu) · line 5263

Quoted 5 lines
net/core/dev.c +1 -1
-     if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1))
+     if (likely(qlen < (max_backlog >> 1)))
              return false;
 
      sd = this_cpu_ptr(&softnet_data);

I assume sd is warm here. Else we could even move skb_flow_limit behind a static_branch seeing how rarely it is likely used.

this_cpu_ptr(&ANY_VAR) only loads very hot this_cpu_off. In modern kernels this is

DEFINE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off);

rest is in the offsets used in the code.

Quoted 10 lines

net/core/dev.c static int enqueue_to_backlog(struct sk_buff *skb, int cpu, · line 5306

Quoted 8 lines
net/core/dev.c +1 -1
@@ -5301,19 +5302,19 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
      u32 tail;
 
      reason = SKB_DROP_REASON_DEV_READY;
-     if (!netif_running(skb->dev))
+     if (unlikely(!netif_running(skb->dev)))
              goto bad_dev;

Isn't unlikely usually predicted for branches without an else?

Quoted 2 lines

I am not sure this is a hardcoded rule that all compilers will stick with. Do you have a reference ?

Actually I was thinking CPU branch prediction if no prior data.

According to the Intel® 64 and IA-32 Architectures Optimization Reference Manual, Aug 2023, 3.4.1.2 Static Prediction

Branches that do not have a history in the BTB (see Section 3.4.1) are predicted using a static prediction algorithm:
- Predict forward conditional branches to be NOT taken. [..]

But online threads mention that there even for x86_64 between microarch generations there are differences on the actual prediction behavior, as well as of explicit prediction hints. And that's only Intel x86_64. So not a universal guide, perhaps.

Quoted 26 lines
Quoted 3 lines

And that is ignoring both FDO and actual branch prediction hardware improving on the simple compiler heuristic.

Lets not assume FDO is always used, and close the gap. This will allow us to iterate faster.
FDO brings its own class of problems...

Quoted 4 lines

No immediately concerns. Just want to avoid precedence for others to sprinkle code with likely/unlikely with abandon. As is sometimes seen.

Sure.

I have not included a change on the apparently _very_ expensive

if (!__test_and_set_bit(NAPI_STATE_SCHED,

                                    &sd->backlog.state))

btsq $0x0,0x160(%r13)

I tried to test the bit, then set it if needed, but got no improvement, for some reason
(This was after the other patch making sure to group the dirtied fields in a single cache line)

Reviewed-by on v1 1/1
#4 lore raw
In reply to Eric Dumazet
Quoted 7 lines · Eric Dumazet

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Quoted 8 lines

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

skb_flow_limit() does not need the backlog_lock protection, and can be called before we acquire the lock, for even better resistance to attacks.

#5 lore raw
In reply to Eric Dumazet
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Fri, 24 Oct 2025 09:05:17 +0000 you wrote:

Add likely() and unlikely() clauses for the common cases:

Device is running.
Queue is not full.
Queue is less than half capacity.

Add max_backlog parameter to skb_flow_limit() to avoid a second READ_ONCE(net_hotdata.max_backlog).

[...]

Here is the summary with links:
- [net-next] net: optimize enqueue_to_backlog() for the fast path

    https://git.kernel.org/netdev/net-next/c/a086e9860ce6

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html