Open vSwitch zerocopy upcall
Subsystems (4)
v4 cover Open vSwitch zerocopy upcall
Respin of the zerocopy patches for the openvswitch upcall.
V4: - Daniel Borkmann pointed out that the style in skbuff.h has changed
in net-next, adapted to no longer using extern in headers.
V3: - Removed unneeded alignment of nlmsg_len after padding
V2: - Added skb_zerocopy_headlen() to calculate headroom of destination
buffer. This also takes care of the from->head_frag issue.
- Attribute alignment for frag_list case
- API documentation
- performance data for CHECKSUM_PARTIAL tx case
Thomas Graf (2):
net: Export skb_zerocopy() to zerocopy from one skb to another
openvswitch: Use skb_zerocopy() for upcall
include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 85 ++++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 59 ++----------------------- net/openvswitch/datapath.c | 52 +++++++++++++++++++--- 4 files changed, 137 insertions(+), 62 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v4 2/2 openvswitch: Use skb_zerocopy() for upcall
Use of skb_zerocopy() avoids the expensive call to memcpy() when copying the packet data into the Netlink skb. Completes checksum through skb_checksum_help() if needed.
Netlink messaged must be properly padded and aligned to meet sanity checks of the user space counterpart.
Cost of memcpy is significantly reduced from:
+ 7.48% vhost-8471 [k] memcpy + 5.57% ovs-vswitchd [k] memcpy + 2.81% vhost-8471 [k] csum_partial_copy_generic
to:
+ 5.72% ovs-vswitchd [k] memcpy + 3.32% vhost-5153 [k] memcpy + 0.68% vhost-5153 [k] skb_zerocopy
(megaflows disabled)
--- net/openvswitch/datapath.c | 52 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 7 deletions(-)
net/openvswitch/datapath.c +45 -7
@@ -381,10 +381,11 @@ static size_t key_attr_size(void) }static size_t upcall_msg_size (const struct sk_buff *skb, -const struct nlattr *userdata) +const struct nlattr *userdata, +unsigned int hdrlen) {size_t size =NLMSG_ALIGN (sizeof (struct ovs_header )) - +nla_total_size (skb->len)/* OVS_PACKET_ATTR_PACKET */ + +nla_total_size (hdrlen)/* OVS_PACKET_ATTR_PACKET */ +nla_total_size (key_attr_size ());/* OVS_PACKET_ATTR_KEY */ /* OVS_PACKET_ATTR_USERDATA */
@@ -402,6 +403,7 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,struct sk_buff *nskb =NULL ;struct sk_buff *user_skb;/* to be queued to userspace */ struct nlattr *nla; +unsigned int plen, hlen;int err;if (vlan_tx_tag_present (skb)) {
@@ -422,7 +424,13 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex, goto out; } - user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata),GFP_ATOMIC ); +/* Complete checksum if needed */ +if (skb->ip_summed ==CHECKSUM_PARTIAL && + (err =skb_checksum_help (skb))) + goto out; + + hlen =skb_zerocopy_headlen (skb); + user_skb =genlmsg_new (upcall_msg_size (skb, upcall_info->userdata, hlen),GFP_ATOMIC );if (!user_skb) { err = -ENOMEM ; goto out;
@@ -441,13 +449,43 @@ static int queue_userspace_packet(struct net *net, int dp_ifindex,nla_len (upcall_info->userdata), nla_data(upcall_info->userdata)); - nla =__nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET , skb->len); +/* Only reserve room for attribute header, packet data is added +* in skb_zerocopy() */ +if (!(nla =nla_reserve (user_skb,OVS_PACKET_ATTR_PACKET ,0 ))) + goto out; + nla->nla_len =nla_attr_size (skb->len); -skb_copy_and_csum_dev (skb,nla_data (nla)); +skb_zerocopy (user_skb, skb, skb->len, hlen); -genlmsg_end (user_skb, upcall); - err =genlmsg_unicast (net, user_skb, upcall_info->portid); +/* OVS user space expects the size of the message to be aligned to +* NLA_ALIGNTO. Aligning nlmsg_len is not enough, the actual bytes +* read must match nlmsg_len. +*/ + plen =NLA_ALIGN (user_skb->len) - user_skb->len; +if (plen >0 ) { +int nr_frags =skb_shinfo (user_skb)->nr_frags; + +if (nr_frags) { +skb_frag_t *frag; + + frag = &skb_shinfo (user_skb)->frags[nr_frags -1 ]; +skb_frag_size_add (frag, plen); +BUG_ON (frag->size >PAGE_SIZE ); + + user_skb->truesize += plen; + user_skb->len += plen; + user_skb->data_len += plen; + }else { +/* The linear headroom is aligned to NLMSG_ALIGN by +* genlmsg_new(), room must be available. +*/ +memset (skb_put (user_skb, plen),0 , plen); + } + } + ((struct nlmsghdr *) user_skb->data)->nlmsg_len = user_skb->len; + + err =genlmsg_unicast (net, user_skb, upcall_info->portid); out:kfree_skb (nskb);return err;
Signature · 1 line
1.8.3.1
PATCH v4 1/2 net: Export skb_zerocopy() to zerocopy from one skb to another
In reply to Thomas GrafMake the skb zerocopy logic written for nfnetlink queue available for use by other modules.
--- include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 85 ++++++++++++++++++++++++++++++++++++ net/netfilter/nfnetlink_queue_core.c | 59 ++----------------------- 3 files changed, 92 insertions(+), 55 deletions(-)
include/linux/skbuff.h +3 -0
@@ -2372,6 +2372,9 @@ int skb_splice_bits(struct sk_buff *skb, unsigned int offset, struct pipe_inode_info *pipe, unsigned int len, unsigned int flags);void skb_copy_and_csum_dev (const struct sk_buff *skb,u8 *to); +unsigned int skb_zerocopy_headlen (const struct sk_buff *from); +void skb_zerocopy (struct sk_buff *to,const struct sk_buff *from, +int len,int hlen);void skb_split (struct sk_buff *skb,struct sk_buff *skb1,const u32 len);int skb_shift (struct sk_buff *tgt,struct sk_buff *skb,int shiftlen);void skb_scrub_packet (struct sk_buff *skb,bool xnet);
net/core/skbuff.c +85 -0
91 lines not on this page · raw message
net/netfilter/nfnetlink_queue_core.c +4 -55
83 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Daniel Borkmann
Reviewed-byRe: [PATCH 1/2 net-next] net: Export skb_zerocopy() to zerocopy from one skb to another
In reply to a message not in the archiveQuoted 4 lines · Thomas Graf
Make the skb zerocopy logic written for nfnetlink queue available for use by other modules.
Signed-off-by Thomas Graf <tgraf-G/[email protected]>
Daniel Borkmann
Reviewed-byRe: [PATCH 2/2 net-next] openvswitch: Use skb_zerocopy() for upcall
In reply to a message not in the archiveQuoted 17 lines · Thomas Graf
Use of
skb_zerocopy()avoids the expensive call tomemcpy()when copying the packet data into the Netlink skb. Completes checksum throughskb_checksum_help()if needed.Netlink messaged must be properly padded and aligned to meet sanity checks of the user space counterpart.
Cost of memcpy is significantly reduced from:
(quoted diff) +3
-0+ 7.48% vhost-8471 [k] memcpy + 5.57% ovs-vswitchd [k] memcpy + 2.81% vhost-8471 [k] csum_partial_copy_genericto:
(quoted diff) +3
-0+ 5.72% ovs-vswitchd [k] memcpy + 3.32% vhost-5153 [k] memcpy + 0.68% vhost-5153 [k] skb_zerocopy
Quoted 5 lines
(megaflows disabled)
Signed-off-by Thomas Graf <tgraf-G/[email protected]>Cc Jesse Gross <[email protected]>Cc Eric Dumazet <[email protected]>
From: Thomas Graf <[email protected]>
Date: Fri, 8 Nov 2013 11:47:16 +0100
Quoted 11 lines
Respin of the zerocopy patches for the openvswitch upcall.
V4: - Daniel Borkmann pointed out that the style in
skbuff.hhas changed
in net-next, adapted to no longer using extern in headers.
V3: - Removed unneeded alignment ofnlmsg_lenafter padding
V2: - Addedskb_zerocopy_headlen()to calculate headroom of destination
buffer. This also takes care of thefrom->head_fragissue.
- Attribute alignment forfrag_listcase
- API documentation
- performance data forCHECKSUM_PARTIALtx case
Quoted 3 lines
Thomas Graf (2):
net: Exportskb_zerocopy()to zerocopy from one skb to another
openvswitch: Useskb_zerocopy()for upcall
No objections from me for these changes, but it's too late for the current merge window.
Jesse can feel free to queue this up in his tree and you can add my:
to these patches.
Thanks!
Quoted 21 lines · David Miller
From: Thomas Graf <tgraf-G/[email protected]>
Date: Fri, 8 Nov 2013 11:47:16 +0100Quoted 11 lines
Respin of the zerocopy patches for the openvswitch upcall.
V4: - Daniel Borkmann pointed out that the style in
skbuff.hhas changed
in net-next, adapted to no longer using extern in headers.
V3: - Removed unneeded alignment ofnlmsg_lenafter padding
V2: - Addedskb_zerocopy_headlen()to calculate headroom of destination
buffer. This also takes care of thefrom->head_fragissue.
- Attribute alignment forfrag_listcase
- API documentation
- performance data forCHECKSUM_PARTIALtx caseQuoted 3 lines
Thomas Graf (2):
net: Exportskb_zerocopy()to zerocopy from one skb to another
openvswitch: Useskb_zerocopy()for upcallNo objections from me for these changes, but it's too late for the current merge window.
Quoted 7 lines
Jesse can feel free to queue this up in his tree and you can add my:
to these patches.
Thanks!
Thanks - there's some discussion related to these patches in the thread "linux: Signal datapath that unaligned Netlink message can be received" so I'm waiting until that concludes.