Open vSwitch zerocopy upcall
Subsystems (4)
v2 cover Open vSwitch zerocopy upcall
Respin of the zerocopy patches for the openvswitch upcall.
V2: - Added skb_zerocopy_headlen() to calculate headroom of destination
buffer. This also takes care of the from->head_frag issue.
- Attribute alignment for frag_list case
- API documentation
- performance data for CHECKSUM_PARTIAL tx case
Thomas Graf (2):
net: Export skb_zerocopy() to zerocopy from one skb to another
openvswitch: Use skb_zerocopy() to prepare skb for upcall
include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 85 ++++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 59 ++----------------------- net/openvswitch/datapath.c | 47 +++++++++++++++++--- 4 files changed, 133 insertions(+), 61 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v2 1/2 net: Export skb_zerocopy() to zerocopy from one skb to another
Make the skb zerocopy logic written for nfnetlink queue available for use by other modules.
--- include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 85 ++++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 59 ++----------------------- 3 files changed, 92 insertions(+), 55 deletions(-)
include/linux/skbuff.h +3 -0
@@ -2381,6 +2381,9 @@ extern int skb_splice_bits(struct sk_buff *skb,unsigned int len, unsigned int flags);extern void skb_copy_and_csum_dev (const struct sk_buff *skb,u8 *to); +extern unsigned int skb_zerocopy_headlen (const struct sk_buff *from); +extern void skb_zerocopy (struct sk_buff *to,const struct sk_buff *from, +int len,int hlen);extern void skb_split (struct sk_buff *skb,struct sk_buff *skb1,const u32 len); extern int skb_shift(struct sk_buff *tgt, struct sk_buff *skb,
net/core/skbuff.c +85 -0
91 lines not on this page · raw message
net/netfilter/nfnetlink_queue_core.c +4 -55
83 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v2 2/2 openvswitch: Use skb_zerocopy() to prepare skb for upcall
In reply to Thomas GrafFrom: Thomas Graf <[email protected]>
Use of skb_zerocopy() avoids the expensive call to memcpy() when copying the packet data into the Netlink skb. Completes checksum through skb_checksum_help() if needed (typicall packet input from software device) which invalidates some of the gains again.
Stock-RX
- 38.30% swapper [kernel.kallsyms] [k] memcpy
- memcpy
+ 87.46% queue_userspace_packet
+ 12.54% nla_put
+ 24.72% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back + 13.80% ovs-vswitchd [kernel.kallsyms] [k] memcpy - 7.68% ksoftirqd/2 [kernel.kallsyms] [k] memcpy
- memcpy
+ 85.83% queue_userspace_packet
+ 14.17% nla_put
- 7.06% ksoftirqd/3 [kernel.kallsyms] [k] memcpy
- memcpy
+ 84.85% queue_userspace_packet
+ 15.15% nla_put
- 4.41% ksoftirqd/0 [kernel.kallsyms] [k] memcpy
- memcpy
+ 83.48% queue_userspace_packet
+ 16.52% nla_put
Zerocopy-RX
+ 50.35% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back
- 27.78% ovs-vswitchd [kernel.kallsyms] [k] memcpy
- memcpy
+ 74.53% ovs_packet_cmd_execute
+ 24.11% nla_put
+ 0.93% ovs_flow_cmd_new_or_set
+ 13.49% swapper [kernel.kallsyms] [k] memcpy
+ 1.45% ksoftirqd/3 [kernel.kallsyms] [k] memcpy
+ 1.20% ksoftirqd/2 [kernel.kallsyms] [k] memcpy10Gb remote pktgen, 1200 bytes, randomized flows, w/ UDPCSUM:
Hits Missed Lost
Stock RX 731'945 6'315'739 3'606'678 Zerocopy RX 764'041 6'442'761 3'947'451
local pktgen, 4/6 CPUs, 1200 bytes, randomized flows, UDPCSUM:
Hits Missed Lost
Stock TX 2'071'030 17'929'192 16'807'785 Zerocopy TX 1'951'142 18'049'056 16'977'296
--- net/openvswitch/datapath.c | 47 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 41 insertions(+), 6 deletions(-)
net/openvswitch/datapath.c +41 -6
@@ -383,10 +383,11 @@ static size_t key_attr_size(void) }static size_t upcall_msg_size (const struct sk_buff *skb, -const struct nlattr *userdata) +const struct nlattr *userdata, +unsigned int hdrlen) {size_t size =NLMSG_ALIGN (sizeof (struct ovs_header )) - +nla_total_size (skb->len)/* OVS_PACKET_ATTR_PACKET */ + +nla_total_size (hdrlen)/* OVS_PACKET_ATTR_PACKET */ +nla_total_size (key_attr_size ());/* OVS_PACKET_ATTR_KEY */ /* OVS_PACKET_ATTR_USERDATA */
@@ -404,6 +405,7 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,struct sk_buff *nskb =NULL ;struct sk_buff *user_skb;/* to be queued to userspace */ struct nlattr *nla; +unsigned int plen, hlen;int err;if (vlan_tx_tag_present (skb)) {
@@ -424,7 +426,12 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex, goto out; } - user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata),GFP_ATOMIC ); +if (skb->ip_summed ==CHECKSUM_PARTIAL && + (err =skb_checksum_help (skb))) + goto out; + + hlen =skb_zerocopy_headlen (skb); + user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata, hlen),GFP_ATOMIC );if (!user_skb) { err = -ENOMEM ; goto out;
@@ -443,11 +450,39 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,nla_len (upcall_info->userdata), nla_data(upcall_info->userdata)); - nla =__nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET , skb->len); +if (!(nla =nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET ,0 ))) + goto out; + nla->nla_len =nla_attr_size (skb->len); + +skb_zerocopy (user_skb, skb, skb->len, hlen); + +/* Align the end of the attribute to NLA_ALIGNTO */ + plen =NLA_ALIGN (user_skb->len) - user_skb->len; +if (plen >0 ) { +int nr_frags =skb_shinfo (user_skb)->nr_frags; -skb_copy_and_csum_dev (skb,nla_data (nla)); +if (nr_frags) { +skb_frag_t *frag; + +/* Assumption is made that PAGE_SIZE is always alligned +* to at least NLA_ALIGNTO (4) which means that we it +* should be safe to add the padding bytes to the frag +*/ + frag = &skb_shinfo (user_skb)->frags[nr_frags -1 ]; +skb_frag_size_add (frag, plen); + + user_skb->truesize += plen; + user_skb->len += plen; + user_skb->data_len += plen; + }else +memset (skb_put (user_skb, plen),0 , plen); + } + +/* Fix alignment of .nlmsg_len, OVS user space enforces a strict +* total message size alignment. +*/ + ((struct nlmsghdr *) user_skb->data)->nlmsg_len =NLA_ALIGN (user_skb->len); -genlmsg_end (user_skb, upcall); err =genlmsg_unicast (net, user_skb, upcall_info->portid); out:
Signature · 1 line
1.8.3.1
Re: [PATCH net-next 2/2] openvswitch: Use skb_zerocopy() to prepare skb for upcall
Quoted 43 lines · Thomas Graf
From: Thomas Graf <[email protected]>
Use of
skb_zerocopy()avoids the expensive call tomemcpy()when copying the packet data into the Netlink skb. Completes checksum throughskb_checksum_help()if needed (typicall packet input from software device) which invalidates some of the gains again.Stock-RX
(quoted diff) +2
-4- 38.30% swapper [kernel.kallsyms] [k] memcpy - memcpy + 87.46% queue_userspace_packet + 12.54% nla_put + 24.72% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back + 13.80% ovs-vswitchd [kernel.kallsyms] [k] memcpy - 7.68% ksoftirqd/2 [kernel.kallsyms] [k] memcpy - memcpy + 85.83% queue_userspace_packet + 14.17% nla_put - 7.06% ksoftirqd/3 [kernel.kallsyms] [k] memcpy - memcpy + 84.85% queue_userspace_packet + 15.15% nla_put - 4.41% ksoftirqd/0 [kernel.kallsyms] [k] memcpy - memcpy + 83.48% queue_userspace_packet + 16.52% nla_putZerocopy-RX
(quoted diff) +4
-1+ 50.35% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back - 27.78% ovs-vswitchd [kernel.kallsyms] [k] memcpy - memcpy + 74.53% ovs_packet_cmd_execute + 24.11% nla_put + 0.93% ovs_flow_cmd_new_or_set + 13.49% swapper [kernel.kallsyms] [k] memcpy + 1.45% ksoftirqd/3 [kernel.kallsyms] [k] memcpy + 1.20% ksoftirqd/2 [kernel.kallsyms] [k] memcpy10Gb remote pktgen, 1200 bytes, randomized flows, w/ UDPCSUM:
Hits Missed LostStock RX 731'945 6'315'739 3'606'678 Zerocopy RX 764'041 6'442'761 3'947'451
Quoted 8 lines
local pktgen, 4/6 CPUs, 1200 bytes, randomized flows, UDPCSUM:
Hits Missed LostStock TX 2'071'030 17'929'192 16'807'785 Zerocopy TX 1'951'142 18'049'056 16'977'296Signed-off-by Thomas Graf <[email protected]>Cc Jesse Gross <[email protected]>Cc Eric Dumazet <[email protected]>
Thanks for the new version and performance numbers.
Reading the numbers that you provided it seems like this is a win for received packets and basically a wash for outgoing packets (assuming that they are using checksum offloading, which I suspect is most of them). Is that also your conclusion?
A couple of comments:
net/openvswitch/datapath.c static size_t key_attr_size(void) · line 387
Quoted 11 lines
net/openvswitch/datapath.c +2
-1@@ -383,10 +383,11 @@ static size_t key_attr_size(void) }static size_t upcall_msg_size (const struct sk_buff *skb, -const struct nlattr *userdata) +const struct nlattr *userdata, +unsigned int hdrlen)
I think the skb parameter is no longer used by this function.
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 454
Quoted 7 lines
net/openvswitch/datapath.c +2
-1@@ -443,11 +450,39 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,nla_len (upcall_info->userdata), nla_data(upcall_info->userdata)); - nla =__nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET , skb->len); +if (!(nla =nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET ,0 ))) + goto out;
Do we expect that this might fail now?
Quoted 7 lines
net/openvswitch/datapath.c +7
-0+ nla->nla_len =nla_attr_size (skb->len); + +skb_zerocopy (user_skb, skb, skb->len, hlen); + +/* Align the end of the attribute to NLA_ALIGNTO */ + plen =NLA_ALIGN (user_skb->len) - user_skb->len; +if (plen >0 ) {
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 470
Quoted 10 lines
net/openvswitch/datapath.c +8
-1+int nr_frags =skb_shinfo (user_skb)->nr_frags; -skb_copy_and_csum_dev (skb,nla_data (nla)); +if (nr_frags) { +skb_frag_t *frag; + +/* Assumption is made that PAGE_SIZE is always alligned +* to at least NLA_ALIGNTO (4) which means that we it +* should be safe to add the padding bytes to the frag +*/
I agree that it should be safe to assume that PAGE_SIZE is a multiple of the netlink alignment requirements. However, we are calculating the alignment over the total packet payload but applying the alignment to the paged portion. Couldn't we have a non-aligned potion in the linear data area followed by a full page?
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 484
Quoted 4 lines
net/openvswitch/datapath.c +4
-0+/* Fix alignment of .nlmsg_len, OVS user space enforces a strict +* total message size alignment. +*/ + ((struct nlmsghdr *) user_skb->data)->nlmsg_len =NLA_ALIGN (user_skb->len);
Do we still need to do this manually now that we are enforcing alignment of the payload above?
Re: [PATCH net-next 2/2] openvswitch: Use skb_zerocopy() to prepare skb for upcall
Quoted 53 lines · Jesse Gross
Quoted 43 lines · Thomas Graf
From: Thomas Graf <[email protected]>
Use of
skb_zerocopy()avoids the expensive call tomemcpy()when copying the packet data into the Netlink skb. Completes checksum throughskb_checksum_help()if needed (typicall packet input from software device) which invalidates some of the gains again.Stock-RX
(quoted diff) +2
-4- 38.30% swapper [kernel.kallsyms] [k] memcpy - memcpy + 87.46% queue_userspace_packet + 12.54% nla_put + 24.72% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back + 13.80% ovs-vswitchd [kernel.kallsyms] [k] memcpy - 7.68% ksoftirqd/2 [kernel.kallsyms] [k] memcpy - memcpy + 85.83% queue_userspace_packet + 14.17% nla_put - 7.06% ksoftirqd/3 [kernel.kallsyms] [k] memcpy - memcpy + 84.85% queue_userspace_packet + 15.15% nla_put - 4.41% ksoftirqd/0 [kernel.kallsyms] [k] memcpy - memcpy + 83.48% queue_userspace_packet + 16.52% nla_putZerocopy-RX
(quoted diff) +4
-1+ 50.35% ovs-vswitchd libc-2.17.so [.] __memcpy_ssse3_back - 27.78% ovs-vswitchd [kernel.kallsyms] [k] memcpy - memcpy + 74.53% ovs_packet_cmd_execute + 24.11% nla_put + 0.93% ovs_flow_cmd_new_or_set + 13.49% swapper [kernel.kallsyms] [k] memcpy + 1.45% ksoftirqd/3 [kernel.kallsyms] [k] memcpy + 1.20% ksoftirqd/2 [kernel.kallsyms] [k] memcpy10Gb remote pktgen, 1200 bytes, randomized flows, w/ UDPCSUM:
Hits Missed LostStock RX 731'945 6'315'739 3'606'678 Zerocopy RX 764'041 6'442'761 3'947'451Quoted 8 lines
local pktgen, 4/6 CPUs, 1200 bytes, randomized flows, UDPCSUM:
Hits Missed LostStock TX 2'071'030 17'929'192 16'807'785 Zerocopy TX 1'951'142 18'049'056 16'977'296Signed-off-by Thomas Graf <[email protected]>Cc Jesse Gross <[email protected]>Cc Eric Dumazet <[email protected]>
Quoted 6 lines
Thanks for the new version and performance numbers.
Reading the numbers that you provided it seems like this is a win for received packets and basically a wash for outgoing packets (assuming that they are using checksum offloading, which I suspect is most of them). Is that also your conclusion?
It's a wash for TX due to checksumming. You may have seen my patch to pktgen to produce udp checksum skbs. It'swhat I have used to produce the above numbers. It will will generate CHECKSUM_PARTIAL skbs due to vport internal announcing hw capability (which is fine). Leaving out the call to skb_checksum_help() increases the number of hits to 2.6M which would be a nice gain.
The question is, can we move checksum completion to user space? We only need to complete the checksum if the packet is sent to a controller at which point performance does not matter anymore. What do you think about a datapath flag indicating whether user space supports checksum completion and if so skipping the checksum completion in the fast path?
Quoted 9 lines
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 454
Quoted 7 lines
net/openvswitch/datapath.c +2
-1@@ -443,11 +450,39 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,nla_len (upcall_info->userdata), nla_data(upcall_info->userdata)); - nla =__nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET , skb->len); +if (!(nla =nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET ,0 ))) + goto out;Do we expect that this might fail now?
It can fail if the message size is miscalculated. I don't like BUG() but WARN() would help here I guess.
Quoted 9 lines
Quoted 9 lines
net/openvswitch/datapath.c +8
-0+ nla->nla_len =nla_attr_size (skb->len); + +skb_zerocopy (user_skb, skb, skb->len, hlen); + +/* Align the end of the attribute to NLA_ALIGNTO */ + plen =NLA_ALIGN (user_skb->len) - user_skb->len; +if (plen >0 ) { +int nr_frags =skb_shinfo (user_skb)->nr_frags;
Quoted 14 lines
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 470
Quoted 8 lines
net/openvswitch/datapath.c +7
-1-skb_copy_and_csum_dev (skb,nla_data (nla)); +if (nr_frags) { +skb_frag_t *frag; + +/* Assumption is made that PAGE_SIZE is always alligned +* to at least NLA_ALIGNTO (4) which means that we it +* should be safe to add the padding bytes to the frag +*/ I agree that it should be safe to assume that
PAGE_SIZEis a multiple of the netlink alignment requirements. However, we are calculating the alignment over the total packet payload but applying the alignment to the paged portion. Couldn't we have a non-aligned potion in the linear data area followed by a full page?
I was also assuming that headlen is always a multiple of 4 due to the 2 byte shift accounting for the ethernet header but thinking about this harder this may not be the case after all. OTOH I ran a test with randomized packet sizes and didn't hit any walls.
But you are right in general, the headlen we allocate will always be alligned but not the amount copy.
Quoted 7 lines
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 484
Quoted 4 lines
net/openvswitch/datapath.c +4
-0+/* Fix alignment of .nlmsg_len, OVS user space enforces a strict +* total message size alignment. +*/ + ((struct nlmsghdr *) user_skb->data)->nlmsg_len =NLA_ALIGN (user_skb->len);Do we still need to do this manually now that we are enforcing alignment of the payload above?
We could use genlmsg_end() again if we also fix the skb-> pointer above. But we could drop the NLA_ALIGN() because user_skb->len is not always aligned.
Re: [PATCH net-next 2/2] openvswitch: Use skb_zerocopy() to prepare skb for upcall
Quoted 10 lines · Thomas Graf
Quoted 2 lines · Jesse Gross
Quoted 1 line · Thomas Graf
From: Thomas Graf <[email protected]>
The question is, can we move checksum completion to user space? We only need to complete the checksum if the packet is sent to a controller at which point performance does not matter anymore. What do you think about a datapath flag indicating whether user space supports checksum completion and if so skipping the checksum completion in the fast path?
This seems like a premature optimization to me. In order for userspace to be able to either complete the checksums in cases where it needs it or allow the NIC to do it when packets are resent, we would have to also carry around offsets, etc. I would also consider the various hardware offloads to be internal kernel optimization which tend to be fairly platform specific and shouldn't really be exposed to userspace.
Quoted 11 lines
Quoted 7 lines
net/openvswitch/datapath.c static int queue_userspace_packet(struct net *net, int dp_ifindex, · line 484
Quoted 4 lines
net/openvswitch/datapath.c +4
-0+/* Fix alignment of .nlmsg_len, OVS user space enforces a strict +* total message size alignment. +*/ + ((struct nlmsghdr *) user_skb->data)->nlmsg_len =NLA_ALIGN (user_skb->len);Do we still need to do this manually now that we are enforcing alignment of the payload above?
We could use
genlmsg_end()again if we also fix the skb-> pointer above. But we could drop theNLA_ALIGN()becauseuser_skb->lenis not always aligned.
Isn't the goal of the block above this to make user_skb->len aligned?