Skip to contentSkip to conversation

ipv6: udp: exploit dev_scratch helpers

PATCH In review Paolo Abeni 2 participants 4 messages Jun 26, 2017 – Jun 27, 2017

Subsystems (3)
Networking Network drivers UDP
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

cover ipv6: udp: exploit dev_scratch helpers

When bringing in the recent cache optimization for the UDP protocol, I forgot to leverage the newly introduced scratched area helpers in the UDPv6 code path. As a result, the UDPv6 implementation suffers some unnecessary performance penality when compared to v4.

This series aim to bring back UDPv6 on equal footing in respect to v4. The first patch moves the shared helpers to the common include files, while the second uses them in the UDPv6 code.

This gives 5-8% performance improvement for a system under flood with small UDPv6 packets. The performance delta is less than the one reported on the original patch set because the UDPv6 code path already leveraged some of the optimization.

Paolo Abeni (2):
udp: move scratch area helpers into the include file
ipv6: udp: leverage scratch area helpers

 include/net/udp.h | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/udp.c    | 60 ------------------------------------------------------
 net/ipv6/udp.c    | 14 ++++++++-----
 3 files changed, 70 insertions(+), 65 deletions(-)
Signature · 1 line
2.9.4
author maintainer
#1 lore raw

PATCH 1/2 udp: move scratch area helpers into the include file

So that they can be later used by the IPv6 code, too. Also lift the comments a bit.

Signed-off-by Paolo Abeni <[email protected]>
---
 include/net/udp.h | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/udp.c    | 60 ------------------------------------------------------
 2 files changed, 61 insertions(+), 60 deletions(-)
include/net/udp.h +61 -0
index 1468dbd..972ce4b 100644
@@ -302,6 +302,67 @@ struct sock *__udp6_lib_lookup(struct net *net,
 struct sock *udp6_lib_lookup_skb(struct sk_buff *skb,
 				 __be16 sport, __be16 dport);
 
+/* UDP uses skb->dev_scratch to cache as much information as possible and avoid
+ * possibly multiple cache miss on dequeue()
+ */
+#if BITS_PER_LONG == 64
+
+/* truesize, len and the bit needed to compute skb_csum_unnecessary will be on
+ * cold cache lines at recvmsg time.
+ * skb->len can be stored on 16 bits since the udp header has been already
+ * validated and pulled.
+ */
+struct udp_dev_scratch {
+	u32 truesize;
+	u16 len;
+	bool is_linear;
+	bool csum_unnecessary;
+};
+
+static inline unsigned int udp_skb_len(struct sk_buff *skb)
+{
+	return ((struct udp_dev_scratch *)&skb->dev_scratch)->len;
+}
+
+static inline bool udp_skb_csum_unnecessary(struct sk_buff *skb)
+{
+	return ((struct udp_dev_scratch *)&skb->dev_scratch)->csum_unnecessary;
+}
+
+static inline bool udp_skb_is_linear(struct sk_buff *skb)
+{
+	return ((struct udp_dev_scratch *)&skb->dev_scratch)->is_linear;
+}
+
+#else
+static inline unsigned int udp_skb_len(struct sk_buff *skb)
+{
+	return skb->len;
+}
+
+static inline bool udp_skb_csum_unnecessary(struct sk_buff *skb)
+{
+	return skb_csum_unnecessary(skb);
+}
+
+static inline bool udp_skb_is_linear(struct sk_buff *skb)
+{
+	return !skb_is_nonlinear(skb);
+}
+#endif
+
+static inline int copy_linear_skb(struct sk_buff *skb, int len, int off,
+				  struct iov_iter *to)
+{
+	int n, copy = len - off;
+
+	n = copy_to_iter(skb->data + off, copy, to);
+	if (n == copy)
+		return 0;
+
+	return -EFAULT;
+}
+
 /*
  * 	SNMP statistics for UDP and UDP-Lite
  */
net/ipv4/udp.c +0 -60

85 lines not on this page · raw message

Signature · 1 line
2.9.4
author maintainer
#2 lore raw

PATCH 2/2 ipv6: udp: leverage scratch area helpers

In reply to Paolo Abeni

The commit b65ac44674dd ("udp: try to avoid 2 cache miss on dequeue") leveraged the scratched area helpers for UDP v4 but I forgot to update accordingly the IPv6 code path.

This change extends the scratch area usage to the IPv6 code, synching the two implementations and giving some performance benefit. IPv6 is again almost on the same level of IPv4, performance-wide.

Signed-off-by Paolo Abeni <[email protected]>
---
 net/ipv6/udp.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
net/ipv6/udp.c +9 -5
index d1d7288..450829d 100644
@@ -362,7 +362,7 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	if (!skb)
 		return err;
 
-	ulen = skb->len;
+	ulen = udp_skb_len(skb);
 	copied = len;
 	if (copied > ulen - off)
 		copied = ulen - off;

@@ -379,14 +379,18 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 
 	if (copied < ulen || peeking ||
 	    (is_udplite && UDP_SKB_CB(skb)->partial_cov)) {
-		checksum_valid = !udp_lib_checksum_complete(skb);
+		checksum_valid = udp_skb_csum_unnecessary(skb) ||
+				!__udp_lib_checksum_complete(skb);
 		if (!checksum_valid)
 			goto csum_copy_err;
 	}
 
-	if (checksum_valid || skb_csum_unnecessary(skb))
-		err = skb_copy_datagram_msg(skb, off, msg, copied);
-	else {
+	if (checksum_valid || udp_skb_csum_unnecessary(skb)) {
+		if (udp_skb_is_linear(skb))
+			err = copy_linear_skb(skb, copied, off, &msg->msg_iter);
+		else
+			err = skb_copy_datagram_msg(skb, off, msg, copied);
+	} else {
 		err = skb_copy_and_csum_datagram_msg(skb, off, msg);
 		if (err == -EINVAL)
 			goto csum_copy_err;
Signature · 1 line
2.9.4
maintainer
#3 lore raw
In reply to Paolo Abeni

From: Paolo Abeni <[email protected]>
Date: Mon, 26 Jun 2017 19:01:49 +0200

Quoted 5 lines

When bringing in the recent cache optimization for the UDP protocol, I forgot to leverage the newly introduced scratched area helpers in the UDPv6 code path. As a result, the UDPv6 implementation suffers some unnecessary performance penality when compared to v4.

Quoted 8 lines

This series aim to bring back UDPv6 on equal footing in respect to v4. The first patch moves the shared helpers to the common include files, while the second uses them in the UDPv6 code.

This gives 5-8% performance improvement for a system under flood with small UDPv6 packets. The performance delta is less than the one reported on the original patch set because the UDPv6 code path already leveraged some of the optimization.

Series applied, thank you.