net: flush_backlog() small changes
Subsystems (2)
PATCH 1/1 net: flush_backlog() small changes
Add READ_ONCE() around reads of skb->dev->reg_state, because this field can be changed from other threads/cpus.
Instead of calling dev_kfree_skb_irq() and kfree_skb() while interrupts are masked and locks held, use a temporary list and use __skb_queue_purge_reason()
Use SKB_DROP_REASON_DEV_READY drop reason to better describe why these skbs are dropped.
--- net/core/dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
net/core/dev.c +8 -4
@@ -6119,16 +6119,18 @@ EXPORT_SYMBOL(netif_receive_skb_list);static void flush_backlog (struct work_struct *work) {struct sk_buff *skb, *tmp; +struct sk_buff_head list;struct softnet_data *sd; +__skb_queue_head_init (&list);local_bh_disable (); sd =this_cpu_ptr (&softnet_data);backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue); -dev_kfree_skb_irq (skb); +__skb_queue_tail (&list, skb);rps_input_queue_head_incr (sd); } }
@@ -6136,14 +6138,16 @@ static void flush_backlog(struct work_struct *work)local_lock_nested_bh (&softnet_data.process_queue_bh_lock); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->process_queue); -kfree_skb (skb); +__skb_queue_tail (&list, skb);rps_input_queue_head_incr (sd); } }local_unlock_nested_bh (&softnet_data.process_queue_bh_lock);local_bh_enable (); + +__skb_queue_purge_reason (&list,SKB_DROP_REASON_DEV_READY ); }static bool flush_required (int cpu)
Signature · 1 line
2.48.1.362.g079036d154-goog
Hi Eric,
Quoted 28 lines · Eric Dumazet
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()Use
SKB_DROP_REASON_DEV_READYdrop reason to better describe why these skbs are dropped.Signed-off-by Eric Dumazet <[email protected]>--- net/core/dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)net/core/dev.c +2
-0@@ -6119,16 +6119,18 @@ EXPORT_SYMBOL(netif_receive_skb_list);static void flush_backlog (struct work_struct *work) {struct sk_buff *skb, *tmp; +struct sk_buff_head list;struct softnet_data *sd; +__skb_queue_head_init (&list);
net/core/dev.c EXPORT_SYMBOL(netif_receive_skb_list); · line 6133
Quoted 10 lines
net/core/dev.c +2
-2local_bh_disable (); sd =this_cpu_ptr (&softnet_data);backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue); -dev_kfree_skb_irq (skb); +__skb_queue_tail (&list, skb);
I wonder why we cannot simply replace the above function with 'dev_kfree_skb_irq_reason(skb, SKB_DROP_REASON_DEV_READY);'?
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6143
Quoted 12 lines
net/core/dev.c +2
-2rps_input_queue_head_incr (sd); } }@@ -6136,14 +6138,16 @@ static void flush_backlog(struct work_struct *work)local_lock_nested_bh (&softnet_data.process_queue_bh_lock); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->process_queue); -kfree_skb (skb); +__skb_queue_tail (&list, skb);
Same here.
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6150
Quoted 7 lines
net/core/dev.c +2
-0rps_input_queue_head_incr (sd); } }local_unlock_nested_bh (&softnet_data.process_queue_bh_lock);local_bh_enable (); + +__skb_queue_purge_reason (&list,SKB_DROP_REASON_DEV_READY );
I'm also worried that dev_kfree_skb_irq() is not the same as kfree_skb_reason() because of the following commit:
commit 7df5cb75cfb8acf96c7f2342530eb41e0c11f4c3
Author: Subash Abhinov Kasiviswanathan <[email protected]> Date: Thu Jul 23 11:31:48 2020 -0600
dev: Defer free of skbs in flush_backlog
IRQs are disabled when freeing skbs in input queue. Use the IRQ safe variant to free skbs here.
Thanks,
Jason
Quoted 7 lines
net/core/dev.c +0
-1}static bool flush_required (int cpu) --2.48.1.362.g079036d154-goog
Quoted 35 lines · Jason Xing
Hi Eric,
Quoted 31 lines · Eric Dumazet
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()Use
SKB_DROP_REASON_DEV_READYdrop reason to better describe why these skbs are dropped.Signed-off-by Eric Dumazet <[email protected]>--- net/core/dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)net/core/dev.c +2
-0@@ -6119,16 +6119,18 @@ EXPORT_SYMBOL(netif_receive_skb_list);static void flush_backlog (struct work_struct *work) {struct sk_buff *skb, *tmp; +struct sk_buff_head list;struct softnet_data *sd; +__skb_queue_head_init (&list);local_bh_disable (); sd =this_cpu_ptr (&softnet_data);
Quoted 10 lines
net/core/dev.c EXPORT_SYMBOL(netif_receive_skb_list); · line 6133
Quoted 7 lines
net/core/dev.c +2
-2backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue); -dev_kfree_skb_irq (skb); +__skb_queue_tail (&list, skb);I wonder why we cannot simply replace the above function with '
dev_kfree_skb_irq_reason(skb, SKB_DROP_REASON_DEV_READY);'?
Because this horribly expensive thing pushes packets to another perc-cpu list, and raises a softirq to perform the freeing later from BH.
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0
-0rps_input_queue_head_incr (sd); } }@@ -6136,14 +6138,16 @@ static void flush_backlog(struct work_struct *work)
Quoted 9 lines
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6143
Quoted 7 lines
net/core/dev.c +2
-2local_lock_nested_bh (&softnet_data.process_queue_bh_lock); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->process_queue); -kfree_skb (skb); +__skb_queue_tail (&list, skb);Same here.
Please read the changelog, I think you missed the point.
Quoted 15 lines
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6150
Quoted 8 lines
net/core/dev.c +2
-0rps_input_queue_head_incr (sd); } }local_unlock_nested_bh (&softnet_data.process_queue_bh_lock);local_bh_enable (); + +__skb_queue_purge_reason (&list,SKB_DROP_REASON_DEV_READY );I'm also worried that
dev_kfree_skb_irq()is not the same askfree_skb_reason()because of the following commit:
commit7df5cb75cfb8acf96c7f2342530eb41e0c11f4c3Author: Subash Abhinov Kasiviswanathan <[email protected]> Date: Thu Jul 23 11:31:48 2020 -0600
Quoted 8 lines
dev: Defer free of skbs in
flush_backlogIRQs are disabled when freeing skbs in input queue. Use the IRQ safe variant to free skbs here.
Fixes 145dd5f9c88f ("net: flush the softnet backlog in process context")Signed-off-by Subash Abhinov Kasiviswanathan <[email protected]>Signed-off-by David S. Miller <[email protected]>
The point of this patch is to no longer attempt the kfree_skb() while being in hard or soft irq blocking sections.
Therefore call efficient kfree_skb() instead of expensive fallbacks that were designed
for callers in hard irq contexts.
Quoted 48 lines · Eric Dumazet
Quoted 35 lines · Jason Xing
Hi Eric,
Quoted 31 lines · Eric Dumazet
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()Use
SKB_DROP_REASON_DEV_READYdrop reason to better describe why these skbs are dropped.Signed-off-by Eric Dumazet <[email protected]>--- net/core/dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)net/core/dev.c +2
-0@@ -6119,16 +6119,18 @@ EXPORT_SYMBOL(netif_receive_skb_list);static void flush_backlog (struct work_struct *work) {struct sk_buff *skb, *tmp; +struct sk_buff_head list;struct softnet_data *sd; +__skb_queue_head_init (&list);local_bh_disable (); sd =this_cpu_ptr (&softnet_data);Quoted 10 lines
net/core/dev.c EXPORT_SYMBOL(netif_receive_skb_list); · line 6133
Quoted 7 lines
net/core/dev.c +2
-2backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue); -dev_kfree_skb_irq (skb); +__skb_queue_tail (&list, skb);I wonder why we cannot simply replace the above function with '
dev_kfree_skb_irq_reason(skb, SKB_DROP_REASON_DEV_READY);'?
Quoted 2 lines
Because this horribly expensive thing pushes packets to another perc-cpu list, and raises a softirq to perform the freeing later from BH.
Agreed about this case. How about changing kfree_skb_reason(skb, ...)?
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
net/core/dev.c +0
-0rps_input_queue_head_incr (sd); } }@@ -6136,14 +6138,16 @@ static void flush_backlog(struct work_struct *work)Quoted 9 lines
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6143
Quoted 7 lines
net/core/dev.c +2
-2local_lock_nested_bh (&softnet_data.process_queue_bh_lock); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->process_queue); -kfree_skb (skb); +__skb_queue_tail (&list, skb);Same here.
Quoted 1 line
Please read the changelog, I think you missed the point.
I meant why not directly use kfree_skb_reason(skb, ...) here? It's simpler, right? Then don't bother to use a temporary list.
Quoted 25 lines
Quoted 16 lines
net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6150
Quoted 9 lines
net/core/dev.c +2
-0rps_input_queue_head_incr (sd); } }local_unlock_nested_bh (&softnet_data.process_queue_bh_lock);local_bh_enable (); + +__skb_queue_purge_reason (&list,SKB_DROP_REASON_DEV_READY );I'm also worried that
dev_kfree_skb_irq()is not the same askfree_skb_reason()because of the following commit:
commit7df5cb75cfb8acf96c7f2342530eb41e0c11f4c3Author: Subash Abhinov Kasiviswanathan <[email protected]> Date: Thu Jul 23 11:31:48 2020 -0600Quoted 8 lines
dev: Defer free of skbs in
flush_backlogIRQs are disabled when freeing skbs in input queue. Use the IRQ safe variant to free skbs here.
Fixes 145dd5f9c88f ("net: flush the softnet backlog in process context")Signed-off-by Subash Abhinov Kasiviswanathan <[email protected]>Signed-off-by David S. Miller <[email protected]>
Quoted 6 lines
The point of this patch is to no longer attempt the
kfree_skb()while being in hard or soft irq blocking sections.Therefore call efficient
kfree_skb()instead of expensive fallbacks that were designed
for callers in hard irq contexts.
Thanks for the explanation!
Thanks,
Jason
Quoted 73 lines · Jason Xing
Quoted 48 lines · Eric Dumazet
Quoted 35 lines · Jason Xing
Hi Eric,
Quoted 31 lines · Eric Dumazet
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()Use
SKB_DROP_REASON_DEV_READYdrop reason to better describe why these skbs are dropped.Signed-off-by Eric Dumazet <[email protected]>--- net/core/dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)net/core/dev.c +2
-0@@ -6119,16 +6119,18 @@ EXPORT_SYMBOL(netif_receive_skb_list);static void flush_backlog (struct work_struct *work) {struct sk_buff *skb, *tmp; +struct sk_buff_head list;struct softnet_data *sd; +__skb_queue_head_init (&list);local_bh_disable (); sd =this_cpu_ptr (&softnet_data);Quoted 10 lines
net/core/dev.c EXPORT_SYMBOL(netif_receive_skb_list); · line 6133
Quoted 7 lines
net/core/dev.c +2
-2backlog_lock_irq_disable (sd); skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->input_pkt_queue); -dev_kfree_skb_irq (skb); +__skb_queue_tail (&list, skb);I wonder why we cannot simply replace the above function with '
dev_kfree_skb_irq_reason(skb, SKB_DROP_REASON_DEV_READY);'?Quoted 2 lines
Because this horribly expensive thing pushes packets to another perc-cpu list, and raises a softirq to perform the freeing later from BH.
Agreed about this case. How about changing
kfree_skb_reason(skb, ...)?Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
net/core/dev.c +0
-0rps_input_queue_head_incr (sd); } }net/core/dev.c static void flush_backlog(struct work_struct *work) · line 6143
Quoted 8 lines
net/core/dev.c +2
-2@@ -6136,14 +6138,16 @@ static void flush_backlog(struct work_struct *work)local_lock_nested_bh (&softnet_data.process_queue_bh_lock); skb_queue_walk_safe(&sd->process_queue, skb, tmp) { -if (skb->dev->reg_state ==NETREG_UNREGISTERING ) { +if (READ_ONCE (skb->dev->reg_state) ==NETREG_UNREGISTERING ) {__skb_unlink (skb, &sd->process_queue); -kfree_skb (skb); +__skb_queue_tail (&list, skb);Same here.
Quoted 4 lines
Quoted 1 line
Please read the changelog, I think you missed the point.
I meant why not directly use
kfree_skb_reason(skb, ...)here? It's simpler, right? Then don't bother to use a temporary list.
Because we are blocking BH here, for a potential long time.
This was mentioned in the changelog. Please read it again.
Quoted 4 lines · Eric Dumazet
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.
Quoted 8 lines
Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()Use
SKB_DROP_REASON_DEV_READYdrop reason to better describe why these skbs are dropped.Signed-off-by Eric Dumazet <[email protected]>
Thanks for the optimization!
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Tue, 4 Feb 2025 14:48:25 +0000 you wrote:
Add
READ_ONCE()around reads ofskb->dev->reg_state, because this field can be changed from other threads/cpus.Instead of calling
dev_kfree_skb_irq()andkfree_skb()while interrupts are masked and locks held, use a temporary list and use__skb_queue_purge_reason()[...]
Here is the summary with links:
- [net-next] net: flush_backlog() small changes
https://git.kernel.org/netdev/net-next/c/cbe08724c180
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html