net: Implement fast TX queue selection
Subsystems (4)
v3 cover net: Implement fast TX queue selection
From: Krishna Kumar <[email protected]>
Notes:
1. Eric suggested:
- To use u16 for txq#, but I am using an "int" for now as that avoids one unnecessary subtraction during tx.
- An improvement of caching the txq at connection establishment time (TBD later) so as to use rxq# = txq#.
- Drivers can call sk_tx_queue_set() to set the txq if they are going to call skb_tx_hash() internally.
2. v3 patch stress tested with 1000 netperfs, reboot's, etc.
Changelog [from v2]
--------------------
1. Changed names of functions setting, getting and returning the txq#; and added a new one to reset the txq#.
2. Free sk doesn't need to reset txq#.
Changelog [from v1]
--------------------
1. Changed IPv6 code to call __sk_dst_reset() directly.
2. Removed the patch re-arranging ("encapsulating") __sk_dst_reset()
Multiqueue cards on routers/firewalls set skb->queue_mapping on input which helps in faster xmit. Implement fast queue selection for locally generated packets also, by saving the txq# for connected sockets (in dev_pick_tx) and use it in subsequent iterations. Locally generated packets for a connection will xmit on the same txq, but routing & firewall loads should not be affected by this patch. Tests shows the distribution across txq's for 1-4 netperf sessions is similar to existing code.
Testing & results:
------------------
1. Cycles/Iter (C/I) used by dev_pick_tx:
(B -> Billion, M -> Million) |--------------|------------------------|------------------------| | | ORG | NEW | | Test |--------|---------|-----|--------|---------|-----| | | Cycles | Iters | C/I | Cycles | Iters | C/I | |--------------|--------|---------|-----|--------|---------|-----| | [TCP_STREAM, | 3.98 B | 12.47 M | 320 | 1.95 B | 12.92 M | 152 | | UDP_STREAM, | | | | | | | | TCP_RR, | | | | | | | | UDP_RR] | | | | | | | |--------------|--------|---------|-----|--------|---------|-----| | [TCP_STREAM, | 8.92 B | 29.66 M | 300 | 3.82 B | 38.88 M | 98 | | TCP_RR, | | | | | | | | UDP_RR] | | | | | | | |--------------|--------|---------|-----|--------|---------|-----|
2. Stress test (over 48 hours) : 1000 netperfs running combination of TCP_STREAM/RR, UDP_STREAM/RR (v4/6, NODELAY/~NODELAY for all tests), with some ssh sessions, reboots, modprobe -r driver, etc.
3. Performance test (10 hours): Single 10 hour netperf run of TCP_STREAM/RR, TCP_STREAM + NO_DELAY and UDP_RR. Results show an improvement in both performance and cpu utilization.
Tested on a 4-processor AMD Opteron 2.8 GHz system with 1GB memory, 10G Chelsio card. Each BW number is the sum of 3 iterations of individual tests using 512, 16K, 64K & 128K I/O sizes, in Mb/s:
------------------------ TCP Tests -----------------------
#procs Org BW New BW (%) Org SD New SD (%)
------------------------------------------------------------
1 77777.7 81011.0 (4.15) 42.3 40.2 (-5.11) 4 91599.2 91878.8 (.30) 955.9 919.3 (-3.83) 6 89533.3 91792.2 (2.52) 2262.0 2143.0 (-5.25) 8 87507.5 89161.9 (1.89) 4363.4 4073.6 (-6.64) 10 85152.4 85607.8 (.53) 6890.4 6851.2 (-.56)
------------------------------------------------------------
------------------------- TCP NO_DELAY Tests ---------------
#procs Org BW New BW (%) Org SD New SD (%)
------------------------------------------------------------
1 57001.9 57888.0 (1.55) 67.7 70.2 (3.75) 4 69555.1 69957.4 (.57) 823.0 834.3 (1.36) 6 71359.3 71918.7 (.78) 1740.8 1724.5 (-.93) 8 72577.6 72496.1 (-.11) 2955.4 2937.7 (-.59) 10 70829.6 71444.2 (.86) 4826.1 4673.4 (-3.16)
------------------------------------------------------------
----------------------- Request Response Tests --------------------
#procs Org TPS New TPS (%) Org SD New SD (%)
(1-10)
-------------------------------------------------------------------
TCP 1019245.9 1042626.4 (2.29) 16352.9 16459.8 (.65) UDP 934598.64 942956.9 (.89) 11607.3 11593.2 (-.12)
-------------------------------------------------------------------
Thanks,
- KK
---
PATCH v3 1/4 net: Introduce sk_tx_queue_mapping
From: Krishna Kumar <[email protected]>
Introduce sk_tx_queue_mapping; and functions that set, test and get this value. Reset sk_tx_queue_mapping to -1 whenever the dst cache is set/reset, and in socket alloc. Setting txq to -1 and using valid txq=<0 to n-1> allows the tx path to use the value of sk_tx_queue_mapping directly instead of subtracting 1 on every tx.
--- include/net/sock.h | 26 ++++++++++++++++++++++++++ net/core/sock.c | 5 ++++- 2 files changed, 30 insertions(+), 1 deletion(-)
include/net/sock.h +26 -0
@@ -107,6 +107,7 @@ struct net;* @skc_node: main hash linkage for various protocol lookup tables * @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol * @skc_refcnt: reference count +* @skc_tx_queue_mapping: tx queue number for this connection * @skc_hash: hash value used with various protocol lookup tables * @skc_family: network address family * @skc_state: Connection state
@@ -128,6 +129,7 @@ struct sock_common {struct hlist_nulls_node skc_nulls_node; };atomic_t skc_refcnt; +int skc_tx_queue_mapping;unsigned int skc_hash;unsigned short skc_family;
@@ -215,6 +217,7 @@ struct sock {#define sk_node __sk_common.skc_node#define sk_nulls_node __sk_common.skc_nulls_node#define sk_refcnt __sk_common.skc_refcnt +#define sk_tx_queue_mapping __sk_common.skc_tx_queue_mapping#define sk_copy_start __sk_common.skc_hash#define sk_hash __sk_common.skc_hash
@@ -1094,8 +1097,29 @@ static inline void sock_put(struct sockextern int sk_receive_skb (struct sock *sk,struct sk_buff *skb,const int nested); +static inline void sk_tx_queue_set (struct sock *sk,int tx_queue) +{ + sk->sk_tx_queue_mapping = tx_queue; +} + +static inline void sk_tx_queue_clear (struct sock *sk) +{ + sk->sk_tx_queue_mapping =-1 ; +} + +static inline int sk_tx_queue_get (const struct sock *sk) +{ +return sk->sk_tx_queue_mapping; +} + +static inline bool sk_tx_queue_recorded (const struct sock *sk) +{ +return (sk && sk->sk_tx_queue_mapping >=0 ); +} +static inline void sk_set_socket (struct sock *sk,struct socket *sock) { +sk_tx_queue_clear (sk); sk->sk_socket = sock; }
@@ -1152,6 +1176,7 @@ __sk_dst_set(struct sock *sk, struct dst {struct dst_entry *old_dst; +sk_tx_queue_clear (sk); old_dst = sk->sk_dst_cache; sk->sk_dst_cache = dst;dst_release (old_dst);
@@ -1170,6 +1195,7 @@ __sk_dst_reset(struct sock *sk) {struct dst_entry *old_dst; +sk_tx_queue_clear (sk); old_dst = sk->sk_dst_cache; sk->sk_dst_cache =NULL ;dst_release (old_dst);
net/core/sock.c +4 -1
@@ -357,6 +357,7 @@ struct dst_entry *__sk_dst_check(structstruct dst_entry *dst = sk->sk_dst_cache;if (dst && dst->obsolete && dst->ops->check (dst, cookie) ==NULL ) { +sk_tx_queue_clear (sk); sk->sk_dst_cache =NULL ;dst_release (dst);return NULL ;
@@ -953,7 +954,8 @@ static void sock_copy(struct sock *nsk,void *sptr = nsk->sk_security;#endif BUILD_BUG_ON (offsetof(struct sock , sk_copy_start) != -sizeof (osk->sk_node) +sizeof (osk->sk_refcnt)); +sizeof (osk->sk_node) +sizeof (osk->sk_refcnt) + +sizeof (osk->sk_tx_queue_mapping));memcpy (&nsk->sk_copy_start, &osk->sk_copy_start, osk->sk_prot->obj_size - offsetof(struct sock , sk_copy_start));#ifdef CONFIG_SECURITY_NETWORK
@@ -997,6 +999,7 @@ static struct sock *sk_prot_alloc(structif (!try_module_get (prot->owner)) goto out_free_sec; +sk_tx_queue_clear (sk); }return sk;
Re: [PATCH 1/4 v3] net: Introduce sk_tx_queue_mapping
Quoted 10 lines · Krishna Kumar
From: Krishna Kumar <[email protected]>
Introduce
sk_tx_queue_mapping; and functions that set, test and get this value. Resetsk_tx_queue_mappingto -1 whenever the dst cache is set/reset, and in socket alloc. Setting txq to -1 and using valid txq=<0 to n-1> allows the tx path to use the value ofsk_tx_queue_mappingdirectly instead of subtracting 1 on every tx.Signed-off-by Krishna Kumar <[email protected]>
PATCH v3 2/4 net: Use sk_tx_queue_mapping for connected sockets
In reply to Krishna KumarFrom: Krishna Kumar <[email protected]>
For connected sockets, the first run of dev_pick_tx saves the calculated txq in sk_tx_queue_mapping. This is not saved if either the device has a queue select or the socket is not connected. Next iterations of dev_pick_tx uses the cached value of sk_tx_queue_mapping.
--- net/core/dev.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
net/core/dev.c +18 -6
@@ -1791,13 +1791,25 @@ EXPORT_SYMBOL(skb_tx_hash);static struct netdev_queue *dev_pick_tx (struct net_device *dev,struct sk_buff *skb) { -const struct net_device_ops *ops = dev->netdev_ops; -u16 queue_index =0 ; +u16 queue_index; +struct sock *sk = skb->sk; + +if (sk_tx_queue_recorded (sk)) { + queue_index =sk_tx_queue_get (sk); + }else { +const struct net_device_ops *ops = dev->netdev_ops; -if (ops->ndo_select_queue) - queue_index = ops->ndo_select_queue (dev, skb); -else if (dev->real_num_tx_queues >1 ) - queue_index =skb_tx_hash (dev, skb); +if (ops->ndo_select_queue) { + queue_index = ops->ndo_select_queue (dev, skb); + }else { + queue_index =0 ; +if (dev->real_num_tx_queues >1 ) + queue_index =skb_tx_hash (dev, skb); + +if (sk && sk->sk_dst_cache) +sk_record_tx_queue (sk, queue_index); + } + }skb_set_queue_mapping (skb, queue_index);return netdev_get_tx_queue (dev, queue_index);
PATCH v3 3/4 net: IPv6 changes
In reply to Krishna KumarFrom: Krishna Kumar <[email protected]>
IPv6: Reset sk_tx_queue_mapping when dst_cache is reset. Use existing macro to do the work.
--- net/ipv6/inet6_connection_sock.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
net/ipv6/inet6_connection_sock.c +1 -2
@@ -168,8 +168,7 @@ struct dst_entry *__inet6_csk_dst_check(if (dst) {struct rt6_info *rt = (struct rt6_info *)dst;if (rt->rt6i_flow_cache_genid !=atomic_read (&flow_cache_genid)) { - sk->sk_dst_cache =NULL ; -dst_release (dst); +__sk_dst_reset (sk); dst =NULL ; } }
PATCH v3 4/4 net: Fix for dst_negative_advice
In reply to Krishna KumarFrom: Krishna Kumar <[email protected]>
dst_negative_advice() should check for changed dst and reset sk_tx_queue_mapping accordingly. Pass sock to the callers of dst_negative_advice.
(sk_reset_txq is defined just for use by dst_negative_advice. The only way I could find to get around this is to move dst_negative_() from dst.h to dst.c, include sock.h in dst.c, etc)
--- include/net/dst.h | 12 ++++++++++-- net/core/sock.c | 6 ++++++ net/dccp/timer.c | 4 ++-- net/decnet/af_decnet.c | 2 +- net/ipv4/tcp_timer.c | 4 ++-- 5 files changed, 21 insertions(+), 7 deletions(-)
include/net/dst.h +10 -2
@@ -222,11 +222,19 @@ static inline void dst_confirm(struct dsneigh_confirm (dst->neighbour); } -static inline void dst_negative_advice (struct dst_entry **dst_p) +static inline void dst_negative_advice (struct dst_entry **dst_p, +struct sock *sk) {struct dst_entry * dst = *dst_p; -if (dst && dst->ops->negative_advice) +if (dst && dst->ops->negative_advice) { *dst_p = dst->ops->negative_advice (dst); + +if (dst != *dst_p) { +extern void sk_reset_txq (struct sock *sk); + +sk_reset_txq (sk); + } + } }static inline void dst_link_failure (struct sk_buff *skb)
net/core/sock.c +6 -0
@@ -352,6 +352,12 @@ discard_and_relse: }EXPORT_SYMBOL (sk_receive_skb); +void sk_reset_txq (struct sock *sk) +{ +sk_tx_queue_clear (sk); +} +EXPORT_SYMBOL (sk_reset_txq); +struct dst_entry *__sk_dst_check (struct sock *sk,u32 cookie) {struct dst_entry *dst = sk->sk_dst_cache;
net/dccp/timer.c +2 -2
@@ -38,7 +38,7 @@ static int dccp_write_timeout(struct socif (sk->sk_state ==DCCP_REQUESTING || sk->sk_state ==DCCP_PARTOPEN ) {if (icsk->icsk_retransmits !=0 ) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); retry_until = icsk->icsk_syn_retries ? : sysctl_dccp_request_retries; }else {
@@ -63,7 +63,7 @@ static int dccp_write_timeout(struct socGolden words :-). */ -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); } retry_until = sysctl_dccp_retries2;
net/decnet/af_decnet.c +1 -1
@@ -1955,7 +1955,7 @@ static int dn_sendmsg(struct kiocb *iocb }if ((flags &MSG_TRYHARD ) && sk->sk_dst_cache) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); mss = scp->segsize_rem; fctype = scp->services_rem &NSP_FC_MASK ;
net/ipv4/tcp_timer.c +2 -2
@@ -141,14 +141,14 @@ static int tcp_write_timeout(struct sockif ((1 << sk->sk_state) & (TCPF_SYN_SENT |TCPF_SYN_RECV )) {if (icsk->icsk_retransmits) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); retry_until = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries; }else {if (retransmits_timed_out (sk, sysctl_tcp_retries1)) {/* Black hole detection */ tcp_mtu_probing (icsk, sk); -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); } retry_until = sysctl_tcp_retries2;
Re: [PATCH 4/4 v3] net: Fix for dst_negative_advice
Quoted 103 lines · On Sun, 18 Oct 2009 18:38:16 +0530 Krishna Kumar <[email protected]> wrote:
From: Krishna Kumar <[email protected]>
dst_negative_advice()should check for changed dst and resetsk_tx_queue_mappingaccordingly. Pass sock to the callers ofdst_negative_advice.(
sk_reset_txqis defined just for use bydst_negative_advice. The only way I could find to get around this is to movedst_negative_()fromdst.htodst.c, includesock.hindst.c, etc)Signed-off-by Krishna Kumar <[email protected]>--- include/net/dst.h | 12 ++++++++++-- net/core/sock.c | 6 ++++++ net/dccp/timer.c | 4 ++-- net/decnet/af_decnet.c | 2 +- net/ipv4/tcp_timer.c | 4 ++-- 5 files changed, 21 insertions(+), 7 deletions(-)diff -ruNp
org/include/net/dst.hnew/include/net/dst.hinclude/net/dst.h +10
-2@@ -222,11 +222,19 @@ static inline void dst_confirm(struct dsneigh_confirm (dst->neighbour); } -static inline void dst_negative_advice (struct dst_entry **dst_p) +static inline void dst_negative_advice (struct dst_entry **dst_p, +struct sock *sk) {struct dst_entry * dst = *dst_p; -if (dst && dst->ops->negative_advice) +if (dst && dst->ops->negative_advice) { *dst_p = dst->ops->negative_advice (dst); + +if (dst != *dst_p) { +extern void sk_reset_txq (struct sock *sk); + +sk_reset_txq (sk); + } + } }static inline void dst_link_failure (struct sk_buff *skb)diff -ruNp
org/net/core/sock.cnew/net/core/sock.cnet/core/sock.c +6
-0@@ -352,6 +352,12 @@ discard_and_relse: }EXPORT_SYMBOL (sk_receive_skb); +void sk_reset_txq (struct sock *sk) +{ +sk_tx_queue_clear (sk); +} +EXPORT_SYMBOL (sk_reset_txq); +struct dst_entry *__sk_dst_check (struct sock *sk,u32 cookie) {struct dst_entry *dst = sk->sk_dst_cache;diff -ruNp
org/net/dccp/timer.cnew/net/dccp/timer.cnet/dccp/timer.c +2
-2@@ -38,7 +38,7 @@ static int dccp_write_timeout(struct socif (sk->sk_state ==DCCP_REQUESTING || sk->sk_state ==DCCP_PARTOPEN ) {if (icsk->icsk_retransmits !=0 ) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); retry_until = icsk->icsk_syn_retries ? : sysctl_dccp_request_retries; }else {@@ -63,7 +63,7 @@ static int dccp_write_timeout(struct socGolden words :-). */ -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); } retry_until = sysctl_dccp_retries2;diff -ruNp
org/net/decnet/af_decnet.cnew/net/decnet/af_decnet.cnet/decnet/af_decnet.c +1
-1@@ -1955,7 +1955,7 @@ static int dn_sendmsg(struct kiocb *iocb }if ((flags &MSG_TRYHARD ) && sk->sk_dst_cache) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); mss = scp->segsize_rem; fctype = scp->services_rem &NSP_FC_MASK ;diff -ruNp
org/net/ipv4/tcp_timer.cnew/net/ipv4/tcp_timer.cnet/ipv4/tcp_timer.c +1
-1@@ -141,14 +141,14 @@ static int tcp_write_timeout(struct sockif ((1 << sk->sk_state) & (TCPF_SYN_SENT |TCPF_SYN_RECV )) {if (icsk->icsk_retransmits) -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); retry_until = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries;
net/ipv4/tcp_timer.c static int tcp_write_timeout(struct sock · line 154
Quoted 10 lines
net/ipv4/tcp_timer.c +1
-1}else {if (retransmits_timed_out (sk, sysctl_tcp_retries1)) {/* Black hole detection */ tcp_mtu_probing (icsk, sk); -dst_negative_advice (&sk->sk_dst_cache); +dst_negative_advice (&sk->sk_dst_cache, sk); } retry_until = sysctl_tcp_retries2;
It is good that your patch is broken in pieces, but will the intermediate patches still function correctly. I.e are they bisect safe?
Re: [PATCH 4/4 v3] net: Fix for dst_negative_advice
Stephen Hemminger <[email protected]> wrote on 10/19/2009 09:42:09 AM:
Quoted 15 lines
Quoted 15 lines
diff -ruNp
org/net/ipv4/tcp_timer.cnew/net/ipv4/tcp_timer.cnet/ipv4/tcp_timer.c 2009-10-16 21:31:30.000000000 +0530 renamed from net/ipv4/tcp_timer.c 2009-10-16 21:30:56.000000000 +0530 +1
-1@@ -141,14 +141,14 @@ static int tcp_write_timeout(struct sock if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) { if (icsk->icsk_retransmits) - dst_negative_advice(&sk->sk_dst_cache); + dst_negative_advice(&sk->sk_dst_cache, sk); retry_until = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries; } else { if (retransmits_timed_out(sk, sysctl_tcp_retries1)) { /* Black hole detection */ tcp_mtu_probing(icsk, sk);
Quoted 7 lines
net/ipv4/tcp_timer.c 2009-10-16 21:31:30.000000000 +0530 static int tcp_write_timeout(struct sock · line 154
Quoted 5 lines
net/ipv4/tcp_timer.c 2009-10-16 21:31:30.000000000 +0530 renamed from net/ipv4/tcp_timer.c 2009-10-16 21:30:56.000000000 +0530 +1
-1- dst_negative_advice(&sk->sk_dst_cache); + dst_negative_advice(&sk->sk_dst_cache, sk); } retry_until = sysctl_tcp_retries2;It is good that your patch is broken in pieces, but will the intermediate
patches
Quoted 1 line
still function correctly. I.e are they bisect safe?
I have only compile tested each patch, but I assume it could break something.
Individual patches can be made to function correctly by renaming patch#2 to patch#4 and move patch#3 and #4 ahead.
Should I resubmit with the changed order?
Thanks,
- KK
David Miller
maintainerRe: [PATCH 4/4 v3] net: Fix for dst_negative_advice
From: Krishna Kumar2 <[email protected]>
Date: Mon, 19 Oct 2009 10:04:53 +0530
Quoted 1 line
Should I resubmit with the changed order?
I took care of this.
I put the patch that actually makes dev.c use sk_tx_queue_mapping last in the set.
All applied, thanks everyone.