Open vSwitch zerocopy upcall
Subsystems (4)
cover Open vSwitch zerocopy upcall
Posting this to netdev due to dependency on first patch.
Thomas Graf (2):
net: Export skb_zerocopy() to zerocopy from one skb to another
openvswitch: Use zerocopy if applicable when performing the upcall
include/linux/skbuff.h | 2 ++ net/core/skbuff.c | 46 +++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 47 +----------------------------------- net/openvswitch/datapath.c | 38 ++++++++++++++++++++++++----- 4 files changed, 81 insertions(+), 52 deletions(-)
Signature · 1 line
1.7.11.7
PATCH 1/2 net: Export skb_zerocopy() to zerocopy from one skb to another
Make the skb zerocopy logic written for nfnetlink queue available for use by other modules.
--- include/linux/skbuff.h | 2 ++ net/core/skbuff.c | 46 +++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 47 +----------------------------------- 3 files changed, 49 insertions(+), 46 deletions(-)
include/linux/skbuff.h +2 -0
@@ -2478,6 +2478,8 @@ extern int skb_splice_bits(struct sk_buff *skb,unsigned int len, unsigned int flags);extern void skb_copy_and_csum_dev (const struct sk_buff *skb,u8 *to); +extern void skb_zerocopy (struct sk_buff *to,const struct sk_buff *from, +int len,int hlen);extern void skb_split (struct sk_buff *skb,struct sk_buff *skb1,const u32 len); extern int skb_shift(struct sk_buff *tgt, struct sk_buff *skb,
net/core/skbuff.c +46 -0
52 lines not on this page · raw message
net/netfilter/nfnetlink_queue_core.c +1 -46
59 lines not on this page · raw message
Signature · 1 line
1.7.11.7
Re: [PATCH net-next 1/2] net: Export skb_zerocopy() to zerocopy from one skb to another
Quoted 42 lines · Thomas Graf
Make the skb zerocopy logic written for nfnetlink queue available for use by other modules.
Signed-off-by Thomas Graf <[email protected]>--- include/linux/skbuff.h | 2 ++ net/core/skbuff.c | 46 +++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 47 +----------------------------------- 3 files changed, 49 insertions(+), 46 deletions(-)include/linux/skbuff.h +2
-0@@ -2478,6 +2478,8 @@ extern int skb_splice_bits(struct sk_buff *skb,unsigned int len, unsigned int flags);extern void skb_copy_and_csum_dev (const struct sk_buff *skb,u8 *to); +extern void skb_zerocopy (struct sk_buff *to,const struct sk_buff *from, +int len,int hlen);extern void skb_split (struct sk_buff *skb,struct sk_buff *skb1,const u32 len); extern int skb_shift(struct sk_buff *tgt, struct sk_buff *skb,net/core/skbuff.c +11
-0@@ -2118,6 +2118,52 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to) }EXPORT_SYMBOL (skb_copy_and_csum_dev); +void +skb_zerocopy (struct sk_buff *to,const struct sk_buff *from,int len,int hlen) +{ +int i, j =0 ; +int plen =0 ;/* length of skb->head fragment */ +struct page *page; +unsigned int offset; + +/* dont bother with small payloads */ +if (len <=skb_tailroom (to)) { +skb_copy_bits (from,0 ,skb_put (to, len), len);
net/core/skbuff.c void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to) · line 2141
Quoted 10 lines
net/core/skbuff.c +10
-0+return ; + } + +if (hlen) { +skb_copy_bits (from,0 ,skb_put (to, hlen), hlen); + len -= hlen; + }else { + plen =min_t (int,skb_headlen (from), len); +if (plen) { + page =virt_to_head_page (from->head);
Hmm. Big warning here : In the nfnetlink case, we specifically made sure that this path was taken only if skb->head_frag was set (all incoming packets have this property, but not outgoing ones yet)
This should be documented if we move it to net/core/skbuff.c, or we should add a BUG_ON()
I see your openvswitch code seems OK.
Maybe also put in net/core/skbuff.c the code dealing with this stuff ?
+ if (!skb->head_frag ||
+ skb_headlen(skb) < L1_CACHE_BYTES ||
+ skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS)
+ hlen = skb_headlen(skb);
+
+ if (skb_has_frag_list(skb))
+ hlen = skb->len;
Quoted 7 lines
net/core/skbuff.c +7
-0+ offset = from->data - (unsigned char *)page_address (page); +__skb_fill_page_desc (to,0 , page, offset, plen); +get_page (page); + j =1 ; + len -= plen; + } + }
Re: [PATCH net-next 1/2] net: Export skb_zerocopy() to zerocopy from one skb to another
Quoted 6 lines · Eric Dumazet
Hmm. Big warning here : In the nfnetlink case, we specifically made sure that this path was taken only if
skb->head_fragwas set (all incoming packets have this property, but not outgoing ones yet)This should be documented if we move it to
net/core/skbuff.c, or we should add aBUG_ON()
Good point.
I guess we could ensure in skb_zerocopy() that if !head_frag then we enforce hlen = max(hlen, skb_headlen(skb)) to always fall back to skb_copy_bits() for !head_frag.
Review
Quoted 11 lines
I see your openvswitch code seems OK.
Maybe also put in
net/core/skbuff.cthe code dealing with this stuff ?(quoted diff) +7
-0+ if (!skb->head_frag || + skb_headlen(skb) < L1_CACHE_BYTES || + skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) + hlen = skb_headlen(skb); + + if (skb_has_frag_list(skb)) + hlen = skb->len;
I'll put this into skb_zerocopy_hdrlen()
PATCH 2/2 openvswitch: Use zerocopy if applicable when performing the upcall
In reply to Thomas GrafFrom: Thomas Graf <[email protected]>
Avoids a memcpy() that is expensive for large packets:
from:
4.51% [kernel] [k] memcpy
to:
1.20% [kernel] [k] memcpy 1.01% [kernel] [k] skb_zerocopy
--- net/openvswitch/datapath.c | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-)
net/openvswitch/datapath.c +32 -6
@@ -375,10 +375,11 @@ static size_t key_attr_size(void) }static size_t upcall_msg_size (const struct sk_buff *skb, -const struct nlattr *userdata) +const struct nlattr *userdata, +unsigned int hdrlen) {size_t size =NLMSG_ALIGN (sizeof (struct ovs_header )) - +nla_total_size (skb->len)/* OVS_PACKET_ATTR_PACKET */ + +nla_total_size (hdrlen)/* OVS_PACKET_ATTR_PACKET */ +nla_total_size (key_attr_size ());/* OVS_PACKET_ATTR_KEY */ /* OVS_PACKET_ATTR_USERDATA */
@@ -396,6 +397,7 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,struct sk_buff *nskb =NULL ;struct sk_buff *user_skb;/* to be queued to userspace */ struct nlattr *nla; +unsigned int plen, hlen =0 ;int err;if (vlan_tx_tag_present (skb)) {
@@ -416,7 +418,21 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex, goto out; } - user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata),GFP_ATOMIC ); +if (skb->ip_summed ==CHECKSUM_PARTIAL && + (err =skb_checksum_help (skb))) + goto out; + +if (!skb->head_frag || +skb_headlen (skb) <L1_CACHE_BYTES || +skb_shinfo (skb)->nr_frags >=MAX_SKB_FRAGS ) + hlen =skb_headlen (skb); + +if (skb_has_frag_list (skb)) + hlen = skb->len; + + hlen =min_t (int, skb->len, hlen); + + user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata, hlen),GFP_ATOMIC );if (!user_skb) { err = -ENOMEM ; goto out;
@@ -435,11 +451,21 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,nla_len (upcall_info->userdata), nla_data(upcall_info->userdata)); - nla =__nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET , skb->len); +if (!(nla =nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET ,0 ))) + goto out; + nla->nla_len =nla_attr_size (skb->len); + +skb_zerocopy (user_skb, skb, skb->len, hlen); -skb_copy_and_csum_dev (skb,nla_data (nla)); + plen =NLA_ALIGN (user_skb->len) - user_skb->len; +if (plen >0 ) +memset (skb_put (user_skb, plen),0 , plen); + +/* Fix alignment of .nlmsg_len, OVS user space enforces a strict +* total message size alignment. +*/ + ((struct nlmsghdr *) user_skb->data)->nlmsg_len =NLA_ALIGN (user_skb->len); -genlmsg_end (user_skb, upcall); err =genlmsg_unicast (net, user_skb, upcall_info->portid); out:
Signature · 1 line
1.7.11.7
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 11 lines · Thomas Graf
From: Thomas Graf <[email protected]>
Avoids a
memcpy()that is expensive for large packets:from:
4.51% [kernel] [k] memcpy
to:1.20% [kernel] [k] memcpy 1.01% [kernel] [k] skb_zerocopySigned-off-by Thomas Graf <[email protected]>
Does this have any impact on small packets? Those are usually the common case (i.e. TCP SYN) and I think this is slightly less optimal for those.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 3 lines · Jesse Gross
Does this have any impact on small packets? Those are usually the common case (i.e. TCP SYN) and I think this is slightly less optimal for those.
No difference at all, small packets are copied anyway in skb->head
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Jesse Gross
Does this have any impact on small packets? Those are usually the common case (i.e. TCP SYN) and I think this is slightly less optimal for those.
No difference at all, small packets are copied anyway in
skb->head
Yes, but it makes the Open vSwitch code slightly worse - for example, currently checksumming and copying are done in a single step but this prevents that. Actually, I'm also curious about the test case that was used for large packets and the full profile output since checksumming and GSO aren't listed in the one that Thomas gave.
My guess is that there isn't a real different for small packets since everything will be in the cache but it seems worth checking given that this is optimizing a rare case at the expense of the common one.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 7 lines · Jesse Gross
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Jesse Gross
Does this have any impact on small packets? Those are usually the common case (i.e. TCP SYN) and I think this is slightly less optimal for those.
Quoted 8 lines
Quoted 1 line
No difference at all, small packets are copied anyway in
skb->headYes, but it makes the Open vSwitch code slightly worse - for example, currently checksumming and copying are done in a single step but this prevents that. Actually, I'm also curious about the test case that was used for large packets and the full profile output since checksumming and GSO aren't listed in the one that Thomas gave.
GSO is fully supported in nfnetlink, I see no reason why Open vSwitch would not allow that.
Quoted 3 lines
My guess is that there isn't a real different for small packets since everything will be in the cache but it seems worth checking given that this is optimizing a rare case at the expense of the common one.
I really doubt checksumming a SYN/ACK packet is that a performance issue. Do you have performance numbers ?
You could always provide a patch to restore this copy/checksum if it really gives a benefit, and if people still use NIC not doing this checksum.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 17 lines · Eric Dumazet
Quoted 7 lines · Jesse Gross
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Jesse Gross
Does this have any impact on small packets? Those are usually the common case (i.e. TCP SYN) and I think this is slightly less optimal for those.
Quoted 8 lines
Quoted 1 line
No difference at all, small packets are copied anyway in
skb->headYes, but it makes the Open vSwitch code slightly worse - for example, currently checksumming and copying are done in a single step but this prevents that. Actually, I'm also curious about the test case that was used for large packets and the full profile output since checksumming and GSO aren't listed in the one that Thomas gave.
Quoted 2 lines
GSO is fully supported in nfnetlink, I see no reason why Open vSwitch would not allow that.
Offloads are supported. What I want to know is how they affect performance with this change.
Quoted 10 lines
Quoted 3 lines
My guess is that there isn't a real different for small packets since everything will be in the cache but it seems worth checking given that this is optimizing a rare case at the expense of the common one.
I really doubt checksumming a SYN/ACK packet is that a performance issue. Do you have performance numbers ?
You could always provide a patch to restore this copy/checksum if it really gives a benefit, and if people still use NIC not doing this checksum.
If it makes a difference then it needs to be addressed before this patch goes in since it's the common case. I don't think it will but that's why I'm asking for numbers.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 2 lines · Jesse Gross
Offloads are supported. What I want to know is how they affect performance with this change.
Hmm, I do not understand why you are checksumming then.
skb_copy_and_csum_dev() is a killer.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Jesse Gross
Offloads are supported. What I want to know is how they affect performance with this change.
Hmm, I do not understand why you are checksumming then.
Checksum offloading is an internal kernel optimization, so the goal is to shield userspace from needing to know about it.
Quoted 1 line
skb_copy_and_csum_dev()is a killer.
What's the alternative?
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 1 line · Jesse Gross
What's the alternative?
I guess Thomas was working on this ;)
We addressed all this on nfnetlink lately.
Presumably GSO stuff is a non issue for vSwitch upcalls.
But UDP messages can be big, and their checksum might be already validated by the NIC.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
In reply to Jesse GrossQuoted 8 lines · Jesse Gross
Quoted 7 lines · Eric Dumazet
Quoted 3 lines
My guess is that there isn't a real different for small packets since everything will be in the cache but it seems worth checking given that this is optimizing a rare case at the expense of the common one.
I really doubt checksumming a SYN/ACK packet is that a performance issue. Do you have performance numbers ?
Quoted 7 lines
Quoted 3 lines
You could always provide a patch to restore this copy/checksum if it really gives a benefit, and if people still use NIC not doing this checksum.
If it makes a difference then it needs to be addressed before this patch goes in since it's the common case. I don't think it will but that's why I'm asking for numbers.
I ran TCP_CRR to verify the SYN/ACK use case and I did not observe a difference. If you have any specific test in mind I will be glad to run that before posting the 2nd revision.
The CPU numbers as reported was seen when testing with pktgen at 1400 bytes with 8K flows and a 10/8 random IP source address.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 3 lines · Thomas Graf
I ran
TCP_CRRto verify the SYN/ACK use case and I did not observe a difference. If you have any specific test in mind I will be glad to run that before posting the 2nd revision.
I guess you should test with rx checksum disabled as well, Jesse seemed to be concerned about that.
ethtool -K eth0 rx off
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 8 lines · Eric Dumazet
Quoted 3 lines · Thomas Graf
I ran
TCP_CRRto verify the SYN/ACK use case and I did not observe a difference. If you have any specific test in mind I will be glad to run that before posting the 2nd revision.I guess you should test with rx checksum disabled as well, Jesse seemed to be concerned about that.
I was actually thinking about the transmit side - rx checksum verification
doesn't matter much here since the result will get thrown away. However, if the packet is CHECKSUM_PARTIAL then the checksum will have to get filled in first and that's the code path that is a little different now.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 8 lines · Jesse Gross
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Thomas Graf
I ran
TCP_CRRto verify the SYN/ACK use case and I did not observe a difference. If you have any specific test in mind I will be glad to run that before posting the 2nd revision.
Quoted 8 lines
Quoted 2 lines
I guess you should test with rx checksum disabled as well, Jesse seemed to be concerned about that.
I was actually thinking about the transmit side - rx checksum verification
doesn't matter much here since the result will get thrown away. However, if the packet isCHECKSUM_PARTIALthen the checksum will have to get filled in first and that's the code path that is a little different now.
Do we actually need to complete the checksum before doing the upcall and if so, why? Couldn't the slow path do that if needed? The only reason I can think of where it would matter is if a controller injects the packet into another network stack such as RouteFlow.
On the receive side, hitting an ovs bridge connected to two interface with ~18G of randomized flows:
3.10.0-rc2+:
+ 9.44% ksoftirqd/0 [k] csum_partial_copy_generic + 3.54% ovs-vswitchd [k] copy_user_enhanced_fast_string + 3.08% swapper [k] intel_idle + 2.47% ksoftirqd/0 [k] memset + 2.21% ksoftirqd/0 [k] memcpy + 1.64% ksoftirqd/0 [k] __nla_reserve + 1.26% ovs-vswitchd [k] netlink_recvmsg + 1.07% ksoftirqd/0 [k] __netif_receive_skb_core + 1.02% ksoftirqd/0 [k] __pskb_pull_tail + 0.98% swapper [k] csum_partial_copy_generic + 0.95% ovs-vswitchd [k] _raw_spin_lock + 0.94% ovs-vswitchd [k] _raw_spin_lock_irqsave + 0.83% ovs-vswitchd [k] memcpy + 0.80% ksoftirqd/0 [k] __alloc_skb + 0.78% ksoftirqd/0 [k] skb_copy_bits + 0.71% swapper [k] menu_select + 0.71% ksoftirqd/2 [k] csum_partial_copy_generic
3.10.0-rc2+ + zerocopy
+ 4.84% ovs-vswitchd [k] copy_user_enhanced_fast_string + 4.74% swapper [k] intel_idle + 3.14% swapper [k] memset + 3.04% swapper [k] memcpy + 2.10% swapper [k] __nla_reserve + 1.44% swapper [k] skb_copy_bits + 1.40% swapper [k] __netif_receive_skb_core + 1.34% swapper [k] __pskb_pull_tail + 1.23% ovs-vswitchd [k] _raw_spin_lock + 1.16% ovs-vswitchd [k] fib_table_lookup + 1.16% swapper [k] irq_entries_start + 1.09% ovs-vswitchd [k] memcpy + 1.05% swapper [k] __alloc_skb + 1.03% swapper [k] lapic_next_deadline + 1.02% swapper [k] build_skb + 0.99% swapper [k] skb_zerocopy
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 18 lines · Thomas Graf
Quoted 8 lines · Jesse Gross
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Thomas Graf
I ran
TCP_CRRto verify the SYN/ACK use case and I did not observe a difference. If you have any specific test in mind I will be glad to run that before posting the 2nd revision.Quoted 8 lines
Quoted 2 lines
I guess you should test with rx checksum disabled as well, Jesse seemed to be concerned about that.
I was actually thinking about the transmit side - rx checksum verification
doesn't matter much here since the result will get thrown away. However, if the packet isCHECKSUM_PARTIALthen the checksum will have to get filled in first and that's the code path that is a little different now.
Quoted 5 lines
Do we actually need to complete the checksum before doing the upcall and if so, why? Couldn't the slow path do that if needed? The only reason I can think of where it would matter is if a controller injects the packet into another network stack such as RouteFlow.
Well, this is the slow path. I don't want to force userspace to deal with this because it's an internal kernel optimization that is platform-specific and requires carrying additional metadata around.
It's also very common for a packet to traverse another network stack - it's typically a VM's.
David Miller
maintainerRe: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
From: Jesse Gross <[email protected]>
Date: Mon, 10 Jun 2013 13:34:23 -0700
Quoted 3 lines
Well, this is the slow path. I don't want to force userspace to deal with this because it's an internal kernel optimization that is platform-specific and requires carrying additional metadata around.
I think the grounds for this rejection is over the top and unreasonable.
Re: [PATCH net-next 2/2] openvswitch: Use zerocopy if applicable when performing the upcall
Quoted 9 lines · David Miller
From: Jesse Gross <[email protected]>
Date: Mon, 10 Jun 2013 13:34:23 -0700Quoted 3 lines
Well, this is the slow path. I don't want to force userspace to deal with this because it's an internal kernel optimization that is platform-specific and requires carrying additional metadata around.
I think the grounds for this rejection is over the top and unreasonable.
I didn't reject it and in fact, I expect it to go in. I asked for numbers as to how this affects the common case.