net: batch skb dequeueing from softnet input_pkt_queue
Subsystems (2)
PATCH v4 1/1 net: batch skb dequeueing from softnet input_pkt_queue
batch skb dequeueing from softnet input_pkt_queue
batch skb dequeueing from softnet input_pkt_queue to reduce potential lock contention when RPS is enabled. input_pkt_queue is reimplemented as a single linked list(FIFO), and input_pkt_queue_lock is moved into RPS section, so softnet becomes smaller when RPS is disabled than before.
----
include/linux/netdevice.h | 14 +++++-- net/core/dev.c | 92 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 82 insertions(+), 24 deletions(-)
include/linux/netdevice.h +11 -3
@@ -1387,6 +1387,7 @@ struct softnet_data {struct Qdisc *output_queue;struct list_head poll_list;struct sk_buff *completion_queue; +struct sk_buff *process_queue;#ifdef CONFIG_RPS struct softnet_data *rps_ipi_list;
@@ -1396,15 +1397,22 @@ struct softnet_data {struct softnet_data *rps_ipi_next;unsigned int cpu;unsigned int input_queue_head; +spinlock_t input_pkt_queue_lock; +/* 4 bytes hole on 64bits machine */ #endif -struct sk_buff_head input_pkt_queue; +struct sk_buff *input_pkt_queue_head; +struct sk_buff **input_pkt_queue_tailp; +unsigned int input_pkt_queue_len; +unsigned int process_queue_len; +struct napi_struct backlog; }; -static inline void input_queue_head_incr (struct softnet_data *sd) +static inline void input_queue_head_add (struct softnet_data *sd, +unsigned int len) {#ifdef CONFIG_RPS - sd->input_queue_head++; + sd->input_queue_head += len;#endif }
net/core/dev.c +71 -21
165 lines not on this page · raw message
include/linux/netdevice.h struct softnet_data { · line 1406
Quoted 4 lines · On Thu, 22 Apr 2010 13:51:53 +0800 Changli Gao <[email protected]> wrote:
include/linux/netdevice.h +4
-0+struct sk_buff *input_pkt_queue_head; +struct sk_buff **input_pkt_queue_tailp; +unsigned int input_pkt_queue_len; +unsigned int process_queue_len;
Why is opencoding a skb queue a step forward?
Just keep using sk_queue routines, just not the locked variants.
Quoted 11 lines · Stephen Hemminger
Quoted 4 lines · On Thu, 22 Apr 2010 13:51:53 +0800 Changli Gao <[email protected]> wrote:
+ struct
sk_buff*input_pkt_queue_head;
+ structsk_buff**input_pkt_queue_tailp;
+ unsigned int input_pkt_queue_len;
+ unsigned int process_queue_len;Why is opencoding a skb queue a step forward?
Just keep usingsk_queueroutines, just not the locked variants.
I want to keep the critical section of rps_lock() as small as possible to reduce the potential lock contention, when RPS is used.
Signature · 2 lines
Regards， Changli Gao([email protected])
Quoted 10 lines · Changli Gao
Quoted 8 lines · Stephen Hemminger
Quoted 4 lines · On Thu, 22 Apr 2010 13:51:53 +0800 Changli Gao <[email protected]> wrote:
include/linux/netdevice.h +4
-0+struct sk_buff *input_pkt_queue_head; +struct sk_buff **input_pkt_queue_tailp; +unsigned int input_pkt_queue_len; +unsigned int process_queue_len;
Quoted 7 lines
Quoted 3 lines
Why is opencoding a skb queue a step forward?
Just keep usingsk_queueroutines, just not the locked variants.I want to keep the critical section of
rps_lock()as small as possible to reduce the potential lock contention, when RPS is used.
Jamal perf reports show lock contention but also cache line ping pongs.
Yet, you keep a process_queue_len shared by producers and consumer.
Producers want to read it, while consumer decrement it (dirtying its cache line) every packet, slowing down the things.
The idea of batching is to let the consumer process its local queue with no impact to producers.
Please remove it completely, or make the consumer zero it only at the end of batch processing.
A cache line miss cost is about 120 cycles. Multiply it by 1 million packet per second...
Quoted 8 lines · Eric Dumazet
Jamal perf reports show lock contention but also cache line ping pongs.
Yet, you keep a
process_queue_lenshared by producers and consumer.Producers want to read it, while consumer decrement it (dirtying its cache line) every packet, slowing down the things.
Quoted 9 lines
The idea of batching is to let the consumer process its local queue with no impact to producers.
Please remove it completely, or make the consumer zero it only at the end of batch processing.
A cache line miss cost is about 120 cycles. Multiply it by 1 million packet per second...
OK, I'll remove it, and update the input_pkt_queue only before process_backlog returns.
Signature · 2 lines
Regards， Changli Gao([email protected])