net: Checksum offload changes - Part VI
Subsystems (3)
cover net: Checksum offload changes - Part VI
I am working on overhauling RX checksum offload. Goals of this effort are:
- Specify what exactly it means when driver returns CHECKSUM_UNNECESSARY
- Preserve CHECKSUM_COMPLETE through encapsulation layers
- Don't do skb_checksum more than once per packet
- Unify GRO and non-GRO csum verification as much as possible
- Unify the checksum functions (checksum_init)
- Simplify code
What is in this seventh patch set:
- Add skb->csum. This allows a device or GRO to indicate that an invalid checksum was detected.
- Checksum unncessary to checksum complete conversions.
With these changes, I believe that the third goal of the overhaul is now mostly achieved. In the case of no encapsulation or one layer of encapsulation, there should only be at most one skb_checksum over each packet (between GRO and normal path). In the case of two layers of encapsulation, it is still possible with the right combination of non-zero and zero UDP checksums to have >1 skb_checksum. For instance:
IP>GRE(with csum)>IP>UDP(zero csum)>VXLAN>IP>UDP(non-zero csum), would likely necessiate an skb_checksum in GRO and normal path. This doesn't seem like a common scenario at all so I'm inclined to not address this now, if multiple layers of encapsulation becomes popular we can reassess.
Note that checksum conversion shows a nice improvement for RX VXLAN when outer UDP checksum is enabled (12.65% CPU compared to 20.94%). This is not only from the fact that we don't need checksum calculation on the host, but also allows GRO for VXLAN in this case. Checksum conversion does not help send side (which still needs to perform a checksum on host). For that we will implement remote checksum offload in a later patch
(http://tools.ietf.org/html/draft-herbert-remotecsumoffload-00).
Please review carefully and test if possible, mucking with basic checksum functions is always a little precarious :-)
----
Test results with this patch set are below. I did not see any obvious performace regression.
Tests run:
TCP_STREAM: super_netperf with 200 streams
Device bnx2x (10Gbps):
VXLAN UDP RSS port hashing enabled, UDP RX checksum offload supported.
* VXLAN with checksum
With fix:
TCP_STREAM
12.65% CPU utilization
9093.81 Mbps
Without fix:
TCP_STREAM
20.94% CPU utilization
9094.21 Mbps* VXLAN without checksum
With fix:
TCP_STREAM
24.59% CPU utilization
9080.96 Mbps
Without fix:
TCP_STREAM
24.69% CPU utilization
9081.39 MbpsTom Herbert (6):
net: Support for csum_bad in skbuff
net: Infrastructure for checksum unnecessary conversions
udp: Add support for doing checksum unnecessary conversion
gre: Add support for checksum unnecessary conversions
vxlan: Enable checksum unnecessary conversions for vxlan/UDP sockets
l2tp: Enable checksum unnecessary conversions for l2tp/UDP sockets
drivers/net/vxlan.c | 2 ++ include/linux/netdevice.h | 24 +++++++++++++++++++++++- include/linux/skbuff.h | 41 ++++++++++++++++++++++++++++++++++++++++- include/linux/udp.h | 16 +++++++++++++++- net/core/dev.c | 2 +- net/ipv4/gre_demux.c | 4 ++++ net/ipv4/gre_offload.c | 8 ++++++-- net/ipv4/udp.c | 4 ++++ net/ipv4/udp_offload.c | 25 +++++++++++++++++-------- net/ipv6/udp.c | 4 ++++ net/ipv6/udp_offload.c | 24 +++++++++++++++++------- net/l2tp/l2tp_core.c | 2 ++ 12 files changed, 135 insertions(+), 21 deletions(-)
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 1/6 net: Support for csum_bad in skbuff
This flag indicates that an invalid checksum was detected in the packet. __skb_mark_checksum_bad helper function was added to set this.
Checksums can be marked bad from a driver or the GRO path (the latter is implemented in this patch). csum_bad is checked in __skb_checksum_validate_complete (i.e. calling that when ip_summed == CHECKSUM_NONE).
csum_bad works in conjunction with ip_summed value. In the case that ip_summed is CHECKSUM_NONE and csum_bad is set, this implies that the first (or next) checksum encountered in the packet is bad. When ip_summed is CHECKSUM_UNNECESSARY, the first checksum after the last one validated is bad. For example, if ip_summed == CHECKSUM_UNNECESSARY, csum_level == 1, and csum_bad is set-- then the third checksum in the packet is bad. In the normal path, the packet will be dropped when processing the protocol layer of the bad checksum:__skb_decr_checksum_unnecessary called twice for the good checksums changing ip_summed to CHECKSUM_NONE so that __skb_checksum_validate_complete is called to validate the third checksum and that will fail since csum_bad is set.
--- include/linux/netdevice.h | 4 +++- include/linux/skbuff.h | 21 ++++++++++++++++++++- net/core/dev.c | 2 +- 3 files changed, 24 insertions(+), 3 deletions(-)
include/linux/netdevice.h +3 -1
@@ -2216,7 +2216,9 @@ static inline void skb_gro_incr_csum_unnecessary(struct sk_buff *skb)if (__skb_gro_checksum_validate_needed (skb, zero_okay, check)) \ __ret =__skb_gro_checksum_validate_complete (skb, \compute_pseudo (skb, proto)); \ -if (!__ret) \ +if (__ret) \ +__skb_mark_checksum_bad (skb); \ +else \skb_gro_incr_csum_unnecessary (skb); \ __ret; \ })
include/linux/skbuff.h +20 -1
@@ -617,7 +617,8 @@ struct sk_buff {kmemcheck_bitfield_begin (flags3); __u8 csum_level:2; -/* 14 bit hole */ + __u8 csum_bad:1; +/* 13 bit hole */ kmemcheck_bitfield_end (flags3);__be16 inner_protocol;
@@ -2825,6 +2826,21 @@ static inline void __skb_incr_checksum_unnecessary(struct sk_buff *skb) } } +static inline void __skb_mark_checksum_bad (struct sk_buff *skb) +{ +/* Mark current checksum as bad (typically called from GRO +* path). In the case that ip_summed is CHECKSUM_NONE +* this must be the first checksum encountered in the packet. +* When ip_summed is CHECKSUM_UNNECESSARY, this is the first +* checksum after the last one validated. For UDP, a zero +* checksum can not be marked as bad. +*/ + +if (skb->ip_summed ==CHECKSUM_NONE || + skb->ip_summed ==CHECKSUM_UNNECESSARY ) + skb->csum_bad =1 ; +} +/* Check if we need to perform checksum complete validation. * * Returns true if checksum complete is needed, false otherwise
@@ -2866,6 +2882,9 @@ static inline __sum16 __skb_checksum_validate_complete(struct sk_buff *skb, skb->csum_valid =1 ;return 0 ; } + }else if (skb->csum_bad) { +/* ip_summed == CHECKSUM_NONE in this case */ +return 1 ; } skb->csum = psum;
net/core/dev.c +1 -1
@@ -3957,7 +3957,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buffif (!(skb->dev->features &NETIF_F_GRO )) goto normal; -if (skb_is_gso (skb) ||skb_has_frag_list (skb)) +if (skb_is_gso (skb) ||skb_has_frag_list (skb) || skb->csum_bad) goto normal;gro_list_prepare (napi, skb);
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 2/6 net: Infrastructure for checksum unnecessary conversions
In reply to Tom HerbertFor normal path, added skb_checksum_try_convert which is called to attempt to convert CHECKSUM_UNNECESSARY to CHECKSUM_COMPLETE. The primary condition to allow this is that ip_summed is CHECKSUM_NONE and csum_valid is true, which will be the state after consuming a CHECKSUM_UNNECESSARY.
For GRO path, added skb_gro_checksum_try_convert which is the GRO analogue of skb_checksum_try_convert. The primary condition to allow this is that NAPI_GRO_CB(skb)->csum_cnt == 0 and NAPI_GRO_CB(skb)->csum_valid is set. This implies that we have consumed all available CHECKSUM_UNNECESSARY checksums in the GRO path.
--- include/linux/netdevice.h | 20 ++++++++++++++++++++ include/linux/skbuff.h | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)
include/linux/netdevice.h +20 -0
@@ -2233,6 +2233,26 @@ static inline void skb_gro_incr_csum_unnecessary(struct sk_buff *skb)#define skb_gro_checksum_simple_validate (skb) \ __skb_gro_checksum_validate(skb, 0, false, 0, null_compute_pseudo) +static inline bool __skb_gro_checksum_convert_check (struct sk_buff *skb) +{ +return (NAPI_GRO_CB (skb)->csum_cnt ==0 && + !NAPI_GRO_CB (skb)->csum_valid); +} + +static inline void __skb_gro_checksum_convert (struct sk_buff *skb, +__sum16 check,__wsum pseudo) +{ +NAPI_GRO_CB (skb)->csum = ~pseudo; +NAPI_GRO_CB (skb)->csum_valid =1 ; +} + +#define skb_gro_checksum_try_convert (skb, proto, check, compute_pseudo) \ +do { \ + if (__skb_gro_checksum_convert_check(skb)) \ + __skb_gro_checksum_convert(skb, check, \ + compute_pseudo(skb, proto)); \ +} while (0) + static inline int dev_hard_header(struct sk_buff *skb, struct net_device *dev, unsigned short type, const void *daddr, const void *saddr,
include/linux/skbuff.h +20 -0
@@ -2942,6 +2942,26 @@ static inline __wsum null_compute_pseudo(struct sk_buff *skb, int proto)#define skb_checksum_simple_validate (skb) \ __skb_checksum_validate(skb, 0, true, false, 0, null_compute_pseudo) +static inline bool __skb_checksum_convert_check (struct sk_buff *skb) +{ +return (skb->ip_summed ==CHECKSUM_NONE && + skb->csum_valid && !skb->csum_bad); +} + +static inline void __skb_checksum_convert (struct sk_buff *skb, +__sum16 check,__wsum pseudo) +{ + skb->csum = ~pseudo; + skb->ip_summed =CHECKSUM_COMPLETE ; +} + +#define skb_checksum_try_convert (skb, proto, check, compute_pseudo) \ +do { \ + if (__skb_checksum_convert_check(skb)) \ + __skb_checksum_convert(skb, check, \ + compute_pseudo(skb, proto)); \ +} while (0) +#if defined(CONFIG_NF_CONNTRACK ) || defined(CONFIG_NF_CONNTRACK_MODULE )void nf_conntrack_destroy (struct nf_conntrack *nfct);static inline void nf_conntrack_put (struct nf_conntrack *nfct)
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 3/6 udp: Add support for doing checksum unnecessary conversion
In reply to Tom HerbertAdd support for doing CHECKSUM_UNNECESSARY to CHECKSUM_COMPLETE conversion in UDP tunneling path.
In the normal UDP path, we call skb_checksum_try_convert after locating the UDP socket. The check is that checksum conversion is enabled for the socket (new flag in UDP socket) and that checksum field is non-zero.
In the UDP GRO path, we call skb_gro_checksum_try_convert after checksum is validated and checksum field is non-zero. Since this is already in GRO we assume that checksum conversion is always wanted.
--- include/linux/udp.h | 16 +++++++++++++++- net/ipv4/udp.c | 4 ++++ net/ipv4/udp_offload.c | 25 +++++++++++++++++-------- net/ipv6/udp.c | 4 ++++ net/ipv6/udp_offload.c | 24 +++++++++++++++++------- 5 files changed, 57 insertions(+), 16 deletions(-)
include/linux/udp.h +15 -1
@@ -49,7 +49,11 @@ struct udp_sock {unsigned int corkflag;/* Cork is required */ __u8 encap_type;/* Is this an Encapsulation socket? */ unsigned char no_check6_tx:1,/* Send zero UDP6 checksums on TX? */ - no_check6_rx:1;/* Allow zero UDP6 checksums on RX? */ + no_check6_rx:1,/* Allow zero UDP6 checksums on RX? */ + convert_csum:1;/* On receive, convert checksum +* unnecessary to checksum complete +* if possible. +*/ /* * Following member retains the information to create a UDP header * when the socket is uncorked.
@@ -98,6 +102,16 @@ static inline bool udp_get_no_check6_rx(struct sock *sk)return udp_sk (sk)->no_check6_rx; } +static inline void udp_set_convert_csum (struct sock *sk,bool val) +{ +udp_sk (sk)->convert_csum = val; +} + +static inline bool udp_get_convert_csum (struct sock *sk) +{ +return udp_sk (sk)->convert_csum; +} +#define udp_portaddr_for_each_entry (__sk, node, list) \ hlist_nulls_for_each_entry(__sk, node, list, __sk_common.skc_portaddr_node)
net/ipv4/udp.c +4 -0
@@ -1788,6 +1788,10 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,if (sk !=NULL ) {int ret; +if (udp_sk (sk)->convert_csum && uh->check && !IS_UDPLITE (sk)) +skb_checksum_try_convert (skb,IPPROTO_UDP , uh->check, + inet_compute_pseudo); + ret =udp_queue_rcv_skb (sk, skb);sock_put (sk);
net/ipv4/udp_offload.c +17 -8
@@ -290,16 +290,25 @@ static struct sk_buff **udp4_gro_receive(struct sk_buff **head, {struct udphdr *uh =udp_gro_udphdr (skb); -/* Don't bother verifying checksum if we're going to flush anyway. */ -if (unlikely (!uh) || - (!NAPI_GRO_CB (skb)->flush && -skb_gro_checksum_validate_zero_check (skb,IPPROTO_UDP , uh->check, - inet_gro_compute_pseudo))) { -NAPI_GRO_CB (skb)->flush =1 ; -return NULL ; - } +if (unlikely (!uh)) + goto flush; +/* Don't bother verifying checksum if we're going to flush anyway. */ +if (!NAPI_GRO_CB (skb)->flush) + goto skip; + +if (skb_gro_checksum_validate_zero_check (skb,IPPROTO_UDP , uh->check, + inet_gro_compute_pseudo)) + goto flush; +else if (uh->check) +skb_gro_checksum_try_convert (skb,IPPROTO_UDP , uh->check, + inet_gro_compute_pseudo); +skip:return udp_gro_receive (head, skb, uh); + +flush: +NAPI_GRO_CB (skb)->flush =1 ; +return NULL ; }int udp_gro_complete (struct sk_buff *skb,int nhoff)
net/ipv6/udp.c +4 -0
@@ -891,6 +891,10 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable, goto csum_error; } +if (udp_sk (sk)->convert_csum && uh->check && !IS_UDPLITE (sk)) +skb_checksum_try_convert (skb,IPPROTO_UDP , uh->check, + ip6_compute_pseudo); + ret =udpv6_queue_rcv_skb (sk, skb);sock_put (sk);
net/ipv6/udp_offload.c +17 -7
@@ -134,16 +134,26 @@ static struct sk_buff **udp6_gro_receive(struct sk_buff **head, {struct udphdr *uh =udp_gro_udphdr (skb); +if (unlikely (!uh)) + goto flush; +/* Don't bother verifying checksum if we're going to flush anyway. */ -if (unlikely (!uh) || - (!NAPI_GRO_CB (skb)->flush && -skb_gro_checksum_validate_zero_check (skb,IPPROTO_UDP , uh->check, - ip6_gro_compute_pseudo))) { -NAPI_GRO_CB (skb)->flush =1 ; -return NULL ; - } +if (!NAPI_GRO_CB (skb)->flush) + goto skip; +if (skb_gro_checksum_validate_zero_check (skb,IPPROTO_UDP , uh->check, + ip6_gro_compute_pseudo)) + goto flush; +else if (uh->check) +skb_gro_checksum_try_convert (skb,IPPROTO_UDP , uh->check, + ip6_gro_compute_pseudo); + +skip:return udp_gro_receive (head, skb, uh); + +flush: +NAPI_GRO_CB (skb)->flush =1 ; +return NULL ; }int udp6_gro_complete (struct sk_buff *skb,int nhoff)
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 4/6 gre: Add support for checksum unnecessary conversions
In reply to Tom HerbertCall skb_checksum_try_convert and skb_gro_checksum_try_convert after checksum is found present and validated in the GRE header for normal and GRO paths respectively.
In GRO path, call skb_gro_checksum_try_convert
--- net/ipv4/gre_demux.c | 4 ++++ net/ipv4/gre_offload.c | 8 ++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
net/ipv4/gre_demux.c +4 -0
@@ -125,6 +125,10 @@ static int parse_gre_header(struct sk_buff *skb, struct tnl_ptk_info *tpi, *csum_err = true;return -EINVAL ; } + +skb_checksum_try_convert (skb,IPPROTO_GRE ,0 , + null_compute_pseudo); + options++; }
net/ipv4/gre_offload.c +6 -2
@@ -172,10 +172,14 @@ static struct sk_buff **gre_gro_receive(struct sk_buff **head, }/* Don't bother verifying checksum if we're going to flush anyway. */ -if ((greh->flags &GRE_CSUM ) && !NAPI_GRO_CB (skb)->flush && -skb_gro_checksum_simple_validate (skb)) +if ((greh->flags &GRE_CSUM ) && !NAPI_GRO_CB (skb)->flush) { +if (skb_gro_checksum_simple_validate (skb)) goto out_unlock; +skb_gro_checksum_try_convert (skb,IPPROTO_GRE ,0 , + null_compute_pseudo); + } + flush =0 ;for (p = *head; p; p = p->next) {
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 5/6 vxlan: Enable checksum unnecessary conversions for vxlan/UDP sockets
In reply to Tom Herbert--- drivers/net/vxlan.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/net/vxlan.c +2 -0
@@ -2370,6 +2370,8 @@ static struct socket *vxlan_create_sock(struct net *net, bool ipv6,/* Disable multicast loopback */ inet_sk (sock->sk)->mc_loop =0 ; +udp_set_convert_csum (sock->sk, true); +return sock; }
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH 6/6 l2tp: Enable checksum unnecessary conversions for l2tp/UDP sockets
In reply to Tom Herbert--- net/l2tp/l2tp_core.c | 2 ++ 1 file changed, 2 insertions(+)
net/l2tp/l2tp_core.c +2 -0
@@ -1392,6 +1392,8 @@ static int l2tp_tunnel_sock_create(struct net *net,if (err <0 ) goto out; +udp_set_convert_csum (sock->sk, true); +break ;case L2TP_ENCAPTYPE_IP :
Signature · 1 line
2.1.0.rc2.206.gedb03e5
David Miller
maintainerFrom: Tom Herbert <[email protected]>
Date: Sun, 31 Aug 2014 15:12:40 -0700
Quoted 37 lines
I am working on overhauling RX checksum offload. Goals of this effort are:
- Specify what exactly it means when driver returns
CHECKSUM_UNNECESSARY
- PreserveCHECKSUM_COMPLETEthrough encapsulation layers
- Don't doskb_checksummore than once per packet
- Unify GRO and non-GRO csum verification as much as possible
- Unify the checksum functions (checksum_init)
- Simplify codeWhat is in this seventh patch set:
- Add
skb->csum. This allows a device or GRO to indicate that an invalid checksum was detected.
- Checksum unncessary to checksum complete conversions.With these changes, I believe that the third goal of the overhaul is now mostly achieved. In the case of no encapsulation or one layer of encapsulation, there should only be at most one
skb_checksumover each packet (between GRO and normal path). In the case of two layers of encapsulation, it is still possible with the right combination of non-zero and zero UDP checksums to have >1skb_checksum. For instance:
IP>GRE(with csum)>IP>UDP(zero csum)>VXLAN>IP>UDP(non-zero csum), would likely necessiate anskb_checksumin GRO and normal path. This doesn't seem like a common scenario at all so I'm inclined to not address this now, if multiple layers of encapsulation becomes popular we can reassess.Note that checksum conversion shows a nice improvement for RX VXLAN when outer UDP checksum is enabled (12.65% CPU compared to 20.94%). This is not only from the fact that we don't need checksum calculation on the host, but also allows GRO for VXLAN in this case. Checksum conversion does not help send side (which still needs to perform a checksum on host). For that we will implement remote checksum offload in a later patch
(http://tools.ietf.org/html/draft-herbert-remotecsumoffload-00).
Quoted 2 lines
Please review carefully and test if possible, mucking with basic checksum functions is always a little precarious :-)
Awesome work, I love watching infrastructure gradually fall into place like this :-)
Series applied, thanks!