Skip to contentSkip to conversation

net: mitigate retpoline overhead

RFC PATCH In review Paolo Abeni 2 participants 7 messages Nov 30, 2018

Subsystems (5)
Networking Network drivers IPv4/IPv6 TCP +1 more subsystems
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

cover net: mitigate retpoline overhead

The spectre v2 counter-measures, aka retpolines, are a source of measurable overhead[1]. We can partially address that when the function pointer refers to a builtin symbol resorting to a list of tests vs well-known builtin function and direct calls.

This may lead to some uglification around the indirect calls. In netconf 2018 Eric Dumazet described a technique to hide the most relevant part of the needed boilerplate with some macro help.

This series is a [re-]implementation of such idea, exposing the introduced helpers in a new header file. They are later leveraged to avoid the indirect call overhead in the GRO path, when possible.

Overall this gives > 10% performance improvement for UDP GRO benchmark, and smaller but measurable under for TCP syn flood.

The added infra can be used in follow-up patches to cope with retpoline overhead in other points of the networking stack (e.g. at the qdisc layer) and possibly even in other subsystems.

Paolo Abeni (4):
indirect call wrappers: helpers to speed-up indirect calls of builtin
net: use indirect call wrappers at GRO network layer
net: use indirect call wrapper at GRO transport layer
udp: use indirect call wrapper for GRO socket lookup

 include/linux/indirect_call_wrapper.h | 77 +++++++++++++++++++++++++++
 include/net/inet_common.h             |  9 ++++
 net/core/dev.c                        | 10 +++-
 net/ipv4/af_inet.c                    | 15 +++++-
 net/ipv4/tcp_offload.c                |  5 ++
 net/ipv4/udp.c                        |  2 +
 net/ipv4/udp_offload.c                | 11 +++-
 net/ipv6/ip6_offload.c                | 14 ++++-
 net/ipv6/tcpv6_offload.c              |  5 ++
 net/ipv6/udp.c                        |  2 +
 net/ipv6/udp_offload.c                |  5 ++
 11 files changed, 147 insertions(+), 8 deletions(-)
 create mode 100644 include/linux/indirect_call_wrapper.h
Signature · 1 line
2.19.2
author maintainer
#1 lore raw

RFC PATCH 1/4 indirect call wrappers: helpers to speed-up indirect calls of builtin

This header define a bunch of helpers that allow avoiding the retpoline overhead when calling builtin functions via function pointers. It boils down to explicitly comparing the function pointers to known builtin functions and eventually invoke directly the latter.

The macro defined here implement the boilerplate for the above schema and will be used by the next patches.

Suggested-by Eric Dumazet <Eric Dumazet <[email protected]>
Signed-off-by Paolo Abeni <[email protected]>
---
 include/linux/indirect_call_wrapper.h | 77 +++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 include/linux/indirect_call_wrapper.h
include/linux/indirect_call_wrapper.h new file +77 -0
new file mode 100644
index 000000000000..57e82b4a166d
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_INDIRECT_CALL_WRAPPER_H
+#define _LINUX_INDIRECT_CALL_WRAPPER_H
+
+#ifdef CONFIG_RETPOLINE
+
+/*
+ * INDIRECT_CALL_$NR - wrapper for indirect calls with $NR known builtin
+ *  @f: function pointer
+ *  @name: base name for builtin functions, see INDIRECT_CALLABLE_DECLARE_$NR
+ *  @__VA_ARGS__: arguments for @f
+ *
+ * Avoid retpoline overhead for known builtin, checking @f vs each of them and
+ * eventually invoking directly the builtin function. Fallback to the indirect
+ * call
+ */
+#define INDIRECT_CALL_1(f, name, ...)					\
+	({								\
+		f == name ## 1 ? name ## 1(__VA_ARGS__) :		\
+				 f(__VA_ARGS__);			\
+	})
+#define INDIRECT_CALL_2(f, name, ...)					\
+	({								\
+		f == name ## 2 ? name ## 2(__VA_ARGS__) :		\
+				 INDIRECT_CALL_1(f, name, __VA_ARGS__);	\
+	})
+
+/*
+ * INDIRECT_CALLABLE_DECLARE_$NR - declare $NR known builtin for
+ * INDIRECT_CALL_$NR usage
+ *  @type: return type for the builtin function
+ *  @name: base name for builtin functions, the full list is generated appending
+ *	   the numbers in the 1..@NR range
+ *  @__VA_ARGS__: arguments type list for the builtin function
+ *
+ * Builtin with higher $NR will be checked first by INDIRECT_CALL_$NR
+ */
+#define INDIRECT_CALLABLE_DECLARE_1(type, name, ...)			\
+	extern type name ## 1(__VA_ARGS__)
+#define INDIRECT_CALLABLE_DECLARE_2(type, name, ...)			\
+	extern type name ## 2(__VA_ARGS__);				\
+	INDIRECT_CALLABLE_DECLARE_1(type, name, __VA_ARGS__)
+
+/*
+ * INDIRECT_CALLABLE - allow usage of a builtin function from INDIRECT_CALL_$NR
+ *  @f: builtin function name
+ *  @nr: id associated with this builtin, higher values will be checked first by
+ *	 INDIRECT_CALL_$NR
+ *  @type: function return type
+ *  @name: base name used by INDIRECT_CALL_ to access the builtin list
+ *  @__VA_ARGS__: arguments type list for @f
+ */
+#define INDIRECT_CALLABLE(f, nr, type, name, ...)		\
+	__alias(f) type name ## nr(__VA_ARGS__)
+
+#else
+#define INDIRECT_CALL_1(f, name, ...) f(__VA_ARGS__)
+#define INDIRECT_CALL_2(f, name, ...) f(__VA_ARGS__)
+#define INDIRECT_CALLABLE_DECLARE_1(type, name, ...)
+#define INDIRECT_CALLABLE_DECLARE_2(type, name, ...)
+#define INDIRECT_CALLABLE(f, nr, type, name, ...)
+#endif
+
+/*
+ * We can use INDIRECT_CALL_$NR for ipv6 related functions only if ipv6 is
+ * builtin, this macro simplify dealing with indirect calls with only ipv4/ipv6
+ * alternatives
+ */
+#if IS_BUILTIN(CONFIG_IPV6)
+#define INDIRECT_CALL_INET INDIRECT_CALL_2
+#elif IS_ENABLED(CONFIG_INET)
+#define INDIRECT_CALL_INET INDIRECT_CALL_1
+#else
+#define INDIRECT_CALL_INET(...)
+#endif
+
+#endif
Signature · 1 line
2.19.2
#2 lore raw

Re: [RFC PATCH 1/4] indirect call wrappers: helpers to speed-up indirect calls of builtin

Quoted 30 lines · Paolo Abeni

This header define a bunch of helpers that allow avoiding the retpoline overhead when calling builtin functions via function pointers. It boils down to explicitly comparing the function pointers to known builtin functions and eventually invoke directly the latter.

The macro defined here implement the boilerplate for the above schema and will be used by the next patches.

Suggested-by Eric Dumazet <Eric Dumazet <[email protected]>
Signed-off-by Paolo Abeni <[email protected]>
---
 include/linux/indirect_call_wrapper.h | 77 +++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 include/linux/indirect_call_wrapper.h
include/linux/indirect_call_wrapper.h new file +9 -0
new file mode 100644
index 000000000000..57e82b4a166d
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_INDIRECT_CALL_WRAPPER_H
+#define _LINUX_INDIRECT_CALL_WRAPPER_H
+
+#ifdef CONFIG_RETPOLINE
+
+/*
+ * INDIRECT_CALL_$NR - wrapper for indirect calls with $NR known builtin
+ *  @f: function pointer

include/linux/indirect_call_wrapper.h line 19

Quoted 10 lines
include/linux/indirect_call_wrapper.h new file +10 -0
+ *  @name: base name for builtin functions, see INDIRECT_CALLABLE_DECLARE_$NR
+ *  @__VA_ARGS__: arguments for @f
+ *
+ * Avoid retpoline overhead for known builtin, checking @f vs each of them and
+ * eventually invoking directly the builtin function. Fallback to the indirect
+ * call
+ */
+#define INDIRECT_CALL_1(f, name, ...)					\
+	({								\
+		f == name ## 1 ? name ## 1(__VA_ARGS__) :		\

              likely(f == name ## 1) ? ...

include/linux/indirect_call_wrapper.h line 24

Quoted 5 lines
include/linux/indirect_call_wrapper.h new file +5 -0
+				 f(__VA_ARGS__);			\
+	})
+#define INDIRECT_CALL_2(f, name, ...)					\
+	({								\
+		f == name ## 2 ? name ## 2(__VA_ARGS__) :		\

             likely(f == name ## 2) ? ...
Quoted 4 lines
include/linux/indirect_call_wrapper.h new file +3 -0
+				 INDIRECT_CALL_1(f, name, __VA_ARGS__);	\
+	})
+
author maintainer
#3 lore raw

Re: [RFC PATCH 1/4] indirect call wrappers: helpers to speed-up indirect calls of builtin

Hi,

Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Paolo Abeni

This header define a bunch of helpers that allow avoiding the retpoline overhead when calling builtin functions via function pointers. It boils down to explicitly comparing the function pointers to known builtin functions and eventually invoke directly the latter.

The macro defined here implement the boilerplate for the above schema and will be used by the next patches.

Suggested-by Eric Dumazet <Eric Dumazet <[email protected]>

Oops... typo here. For some reasons checkpatch did not catch it.

Quoted 25 lines
Quoted 25 lines
Signed-off-by Paolo Abeni <[email protected]>
---
 include/linux/indirect_call_wrapper.h | 77 +++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 include/linux/indirect_call_wrapper.h
include/linux/indirect_call_wrapper.h new file +13 -0
new file mode 100644
index 000000000000..57e82b4a166d
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_INDIRECT_CALL_WRAPPER_H
+#define _LINUX_INDIRECT_CALL_WRAPPER_H
+
+#ifdef CONFIG_RETPOLINE
+
+/*
+ * INDIRECT_CALL_$NR - wrapper for indirect calls with $NR known builtin
+ *  @f: function pointer
+ *  @name: base name for builtin functions, see INDIRECT_CALLABLE_DECLARE_$NR
+ *  @__VA_ARGS__: arguments for @f
+ *
+ * Avoid retpoline overhead for known builtin, checking @f vs each of them and
Quoted 8 lines

include/linux/indirect_call_wrapper.h line 19

Quoted 6 lines
include/linux/indirect_call_wrapper.h new file +6 -0
+ * eventually invoking directly the builtin function. Fallback to the indirect
+ * call
+ */
+#define INDIRECT_CALL_1(f, name, ...)					\
+	({								\
+		f == name ## 1 ? name ## 1(__VA_ARGS__) :		\

              likely(f == name ## 1) ? ...

Thank you for the feedback!

I thought about the above, and than I avoided it, because I was not 100% it would fit cases (if any) where we have 2 or more built-in equally likely.

I guess we can address such cases if and when they will pop-up. I'll do some more benchmarks with the branch prediction hints, and then if there are no surprises, I'll add them in v1.

BTW I would like to give the correct attribution here. Does 'Suggested- by' fit? should I list some other guy @google?

Thanks,

Paols

author maintainer
#4 lore raw

RFC PATCH 2/4 net: use indirect call wrappers at GRO network layer

In reply to Paolo Abeni

This avoids an indirect calls for L3 GRO receive path, both for ipv4 and ipv6, if the latter is not compiled as a module.

Note that when IPv6 is compiled as buildin, it will be checked first, so we have a single additional compare for the more common path.

Signed-off-by Paolo Abeni <[email protected]>
---
 include/net/inet_common.h |  2 ++
 net/core/dev.c            | 10 ++++++++--
 net/ipv4/af_inet.c        |  4 ++++
 net/ipv6/ip6_offload.c    |  4 ++++
 4 files changed, 18 insertions(+), 2 deletions(-)
include/net/inet_common.h +2 -0
index 3ca969cbd161..56e7592811ea 100644
@@ -2,6 +2,8 @@
 #ifndef _INET_COMMON_H
 #define _INET_COMMON_H
 
+#include <linux/indirect_call_wrapper.h>
+
 extern const struct proto_ops inet_stream_ops;
 extern const struct proto_ops inet_dgram_ops;
net/core/dev.c +8 -2
index f69b2fcdee40..619f5902600f 100644
@@ -145,6 +145,7 @@
 #include <linux/sctp.h>
 #include <net/udp_tunnel.h>
 #include <linux/net_namespace.h>
+#include <linux/indirect_call_wrapper.h>
 
 #include "net-sysfs.h"
 

@@ -5306,6 +5307,7 @@ static void flush_all_backlogs(void)
 	put_online_cpus();
 }
 
+INDIRECT_CALLABLE_DECLARE_2(int, network_gro_complete, struct sk_buff *, int);
 static int napi_gro_complete(struct sk_buff *skb)
 {
 	struct packet_offload *ptype;

@@ -5325,7 +5327,8 @@ static int napi_gro_complete(struct sk_buff *skb)
 		if (ptype->type != type || !ptype->callbacks.gro_complete)
 			continue;
 
-		err = ptype->callbacks.gro_complete(skb, 0);
+		err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete,
+					 network_gro_complete, skb, 0);
 		break;
 	}
 	rcu_read_unlock();

@@ -5472,6 +5475,8 @@ static void gro_flush_oldest(struct list_head *head)
 	napi_gro_complete(oldest);
 }
 
+INDIRECT_CALLABLE_DECLARE_2(struct sk_buff *, network_gro_receive,
+			    struct list_head *, struct sk_buff *);
 static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
 {
 	u32 hash = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1);

@@ -5521,7 +5526,8 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 			NAPI_GRO_CB(skb)->csum_valid = 0;
 		}
 
-		pp = ptype->callbacks.gro_receive(gro_head, skb);
+		pp = INDIRECT_CALL_INET(ptype->callbacks.gro_receive,
+					network_gro_receive, gro_head, skb);
 		break;
 	}
 	rcu_read_unlock();
net/ipv4/af_inet.c +4 -0
index 326c422c22f8..04ab7ebd6e9b 100644
@@ -1505,6 +1505,8 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
 	return pp;
 }
 EXPORT_SYMBOL(inet_gro_receive);
+INDIRECT_CALLABLE(inet_gro_receive, 1, struct sk_buff *, network_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static struct sk_buff *ipip_gro_receive(struct list_head *head,
 					struct sk_buff *skb)

@@ -1589,6 +1591,8 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)
 	return err;
 }
 EXPORT_SYMBOL(inet_gro_complete);
+INDIRECT_CALLABLE(inet_gro_complete, 1, int, network_gro_complete,
+		  struct sk_buff *, int);
 
 static int ipip_gro_complete(struct sk_buff *skb, int nhoff)
 {
net/ipv6/ip6_offload.c +4 -0
index 70f525c33cb6..a1c2bfb2ce0d 100644
@@ -270,6 +270,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 
 	return pp;
 }
+INDIRECT_CALLABLE(ipv6_gro_receive, 2, struct sk_buff *, network_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static struct sk_buff *sit_ip6ip6_gro_receive(struct list_head *head,
 					      struct sk_buff *skb)

@@ -327,6 +329,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 
 	return err;
 }
+INDIRECT_CALLABLE(ipv6_gro_complete, 2, int, network_gro_complete,
+		  struct sk_buff *, int);
 
 static int sit_gro_complete(struct sk_buff *skb, int nhoff)
 {
Signature · 1 line
2.19.2
author maintainer
#5 lore raw

RFC PATCH 3/4 net: use indirect call wrapper at GRO transport layer

In reply to Paolo Abeni

This avoids an indirect call in the receive path for TCP and UDP packets. TCP takes precedence on UDP, so that we have a single additional conditional in the common case.

Signed-off-by Paolo Abeni <[email protected]>
---
 include/net/inet_common.h |  7 +++++++
 net/ipv4/af_inet.c        | 11 +++++++++--
 net/ipv4/tcp_offload.c    |  5 +++++
 net/ipv4/udp_offload.c    |  5 +++++
 net/ipv6/ip6_offload.c    | 10 ++++++++--
 net/ipv6/tcpv6_offload.c  |  5 +++++
 net/ipv6/udp_offload.c    |  5 +++++
 7 files changed, 44 insertions(+), 4 deletions(-)
include/net/inet_common.h +7 -0
index 56e7592811ea..667bb8247f9a 100644
@@ -56,4 +56,11 @@ static inline void inet_ctl_sock_destroy(struct sock *sk)
 		sock_release(sk->sk_socket);
 }
 
+#define indirect_call_gro_receive(name, cb, head, skb)	\
+({							\
+	unlikely(gro_recursion_inc_test(skb)) ?		\
+		NAPI_GRO_CB(skb)->flush |= 1, NULL :	\
+		INDIRECT_CALL_2(cb, name, head, skb);	\
+})
+
 #endif
net/ipv4/af_inet.c +9 -2
index 04ab7ebd6e9b..774f183f56e3 100644
@@ -1385,6 +1385,8 @@ struct sk_buff *inet_gso_segment(struct sk_buff *skb,
 }
 EXPORT_SYMBOL(inet_gso_segment);
 
+INDIRECT_CALLABLE_DECLARE_2(struct sk_buff *, transport4_gro_receive,
+			    struct list_head *head, struct sk_buff *skb);
 struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
 	const struct net_offload *ops;

@@ -1494,7 +1496,8 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
 	skb_gro_pull(skb, sizeof(*iph));
 	skb_set_transport_header(skb, skb_gro_offset(skb));
 
-	pp = call_gro_receive(ops->callbacks.gro_receive, head, skb);
+	pp = indirect_call_gro_receive(transport4_gro_receive,
+				       ops->callbacks.gro_receive, head, skb);
 
 out_unlock:
 	rcu_read_unlock();

@@ -1558,6 +1561,8 @@ int inet_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len)
 	return -EINVAL;
 }
 
+INDIRECT_CALLABLE_DECLARE_2(int, transport4_gro_complete, struct sk_buff *skb,
+			    int);
 int inet_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	__be16 newlen = htons(skb->len - nhoff);

@@ -1583,7 +1588,9 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)
 	 * because any hdr with option will have been flushed in
 	 * inet_gro_receive().
 	 */
-	err = ops->callbacks.gro_complete(skb, nhoff + sizeof(*iph));
+	err = INDIRECT_CALL_2(ops->callbacks.gro_complete,
+			      transport4_gro_complete, skb,
+			      nhoff + sizeof(*iph));
 
 out_unlock:
 	rcu_read_unlock();
net/ipv4/tcp_offload.c +5 -0
index 870b0a335061..3d5dfac4cd1b 100644
@@ -10,6 +10,7 @@
  *	TCPv4 GSO/GRO support
  */
 
+#include <linux/indirect_call_wrapper.h>
 #include <linux/skbuff.h>
 #include <net/tcp.h>
 #include <net/protocol.h>

@@ -317,6 +318,8 @@ static struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *
 
 	return tcp_gro_receive(head, skb);
 }
+INDIRECT_CALLABLE(tcp4_gro_receive, 2, struct sk_buff *, transport4_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static int tcp4_gro_complete(struct sk_buff *skb, int thoff)
 {

@@ -332,6 +335,8 @@ static int tcp4_gro_complete(struct sk_buff *skb, int thoff)
 
 	return tcp_gro_complete(skb);
 }
+INDIRECT_CALLABLE(tcp4_gro_complete, 2, int, transport4_gro_complete,
+		  struct sk_buff *, int);
 
 static const struct net_offload tcpv4_offload = {
 	.callbacks = {
net/ipv4/udp_offload.c +5 -0
index 0646d61f4fa8..c3c5b237c8e0 100644
@@ -13,6 +13,7 @@
 #include <linux/skbuff.h>
 #include <net/udp.h>
 #include <net/protocol.h>
+#include <net/inet_common.h>
 
 static struct sk_buff *__skb_udp_tunnel_segment(struct sk_buff *skb,
 	netdev_features_t features,

@@ -477,6 +478,8 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 	NAPI_GRO_CB(skb)->flush = 1;
 	return NULL;
 }
+INDIRECT_CALLABLE(udp4_gro_receive, 1, struct sk_buff *, transport4_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static int udp_gro_complete_segment(struct sk_buff *skb)
 {

@@ -536,6 +539,8 @@ static int udp4_gro_complete(struct sk_buff *skb, int nhoff)
 
 	return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb);
 }
+INDIRECT_CALLABLE(udp4_gro_complete, 1, int, transport4_gro_complete,
+		  struct sk_buff *, int);
 
 static const struct net_offload udpv4_offload = {
 	.callbacks = {
net/ipv6/ip6_offload.c +8 -2
index a1c2bfb2ce0d..eeca4164a155 100644
@@ -164,6 +164,8 @@ static int ipv6_exthdrs_len(struct ipv6hdr *iph,
 	return len;
 }
 
+INDIRECT_CALLABLE_DECLARE_2(struct sk_buff *, transport6_gro_receive,
+			    struct list_head *head, struct sk_buff *skb);
 static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 					struct sk_buff *skb)
 {

@@ -260,7 +262,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 
 	skb_gro_postpull_rcsum(skb, iph, nlen);
 
-	pp = call_gro_receive(ops->callbacks.gro_receive, head, skb);
+	pp = indirect_call_gro_receive(transport6_gro_receive,
+				       ops->callbacks.gro_receive, head, skb);
 
 out_unlock:
 	rcu_read_unlock();

@@ -303,6 +306,8 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head,
 	return inet_gro_receive(head, skb);
 }
 
+INDIRECT_CALLABLE_DECLARE_2(int, transport6_gro_complete, struct sk_buff *skb,
+			    int);
 static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	const struct net_offload *ops;

@@ -322,7 +327,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 	if (WARN_ON(!ops || !ops->callbacks.gro_complete))
 		goto out_unlock;
 
-	err = ops->callbacks.gro_complete(skb, nhoff);
+	err = INDIRECT_CALL_2(ops->callbacks.gro_complete,
+			      transport6_gro_complete, skb, nhoff);
 
 out_unlock:
 	rcu_read_unlock();
net/ipv6/tcpv6_offload.c +5 -0
index e72947c99454..3c85afc4cf43 100644
@@ -9,6 +9,7 @@
  *
  *      TCPv6 GSO/GRO support
  */
+#include <linux/indirect_call_wrapper.h>
 #include <linux/skbuff.h>
 #include <net/protocol.h>
 #include <net/tcp.h>

@@ -28,6 +29,8 @@ static struct sk_buff *tcp6_gro_receive(struct list_head *head,
 
 	return tcp_gro_receive(head, skb);
 }
+INDIRECT_CALLABLE(tcp6_gro_receive, 2, struct sk_buff *, transport6_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static int tcp6_gro_complete(struct sk_buff *skb, int thoff)
 {

@@ -40,6 +43,8 @@ static int tcp6_gro_complete(struct sk_buff *skb, int thoff)
 
 	return tcp_gro_complete(skb);
 }
+INDIRECT_CALLABLE(tcp6_gro_complete, 2, int, transport6_gro_complete,
+		  struct sk_buff *, int);
 
 static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb,
 					netdev_features_t features)
net/ipv6/udp_offload.c +5 -0
index 828b2457f97b..ce4d491c583c 100644
@@ -11,6 +11,7 @@
  */
 #include <linux/skbuff.h>
 #include <linux/netdevice.h>
+#include <linux/indirect_call_wrapper.h>
 #include <net/protocol.h>
 #include <net/ipv6.h>
 #include <net/udp.h>

@@ -141,6 +142,8 @@ static struct sk_buff *udp6_gro_receive(struct list_head *head,
 	NAPI_GRO_CB(skb)->flush = 1;
 	return NULL;
 }
+INDIRECT_CALLABLE(udp6_gro_receive, 1, struct sk_buff *, transport6_gro_receive,
+		  struct list_head *, struct sk_buff *);
 
 static int udp6_gro_complete(struct sk_buff *skb, int nhoff)
 {

@@ -153,6 +156,8 @@ static int udp6_gro_complete(struct sk_buff *skb, int nhoff)
 
 	return udp_gro_complete(skb, nhoff, udp6_lib_lookup_skb);
 }
+INDIRECT_CALLABLE(udp6_gro_complete, 1, int, transport6_gro_complete,
+		  struct sk_buff *, int);
 
 static const struct net_offload udpv6_offload = {
 	.callbacks = {
Signature · 1 line
2.19.2
author maintainer
#6 lore raw

RFC PATCH 4/4 udp: use indirect call wrapper for GRO socket lookup

In reply to Paolo Abeni

This avoids another indirect call for UDP GRO. Again, the test for the IPv6 variant is performed first.

Signed-off-by Paolo Abeni <[email protected]>
---
 net/ipv4/udp.c         | 2 ++
 net/ipv4/udp_offload.c | 6 ++++--
 net/ipv6/udp.c         | 2 ++
 3 files changed, 8 insertions(+), 2 deletions(-)
net/ipv4/udp.c +2 -0
index aff2a8e99e01..9ea851f47598 100644
@@ -544,6 +544,8 @@ struct sock *udp4_lib_lookup_skb(struct sk_buff *skb,
 	return __udp4_lib_lookup_skb(skb, sport, dport, &udp_table);
 }
 EXPORT_SYMBOL_GPL(udp4_lib_lookup_skb);
+INDIRECT_CALLABLE(udp4_lib_lookup_skb, 1, struct sock *, udp_lookup,
+		  struct sk_buff *skb, __be16 sport, __be16 dport);
 
 /* Must be called under rcu_read_lock().
  * Does increment socket refcount.
net/ipv4/udp_offload.c +4 -2
index c3c5b237c8e0..0ccd2aa1ab98 100644
@@ -392,6 +392,8 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head,
 	return NULL;
 }
 
+INDIRECT_CALLABLE_DECLARE_2(struct sock *, udp_lookup, struct sk_buff *skb,
+			    __be16 sport, __be16 dport);
 struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,
 				struct udphdr *uh, udp_lookup_t lookup)
 {

@@ -403,7 +405,7 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,
 	struct sock *sk;
 
 	rcu_read_lock();
-	sk = (*lookup)(skb, uh->source, uh->dest);
+	sk = INDIRECT_CALL_INET(lookup, udp_lookup, skb, uh->source, uh->dest);
 	if (!sk)
 		goto out_unlock;
 

@@ -505,7 +507,7 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff,
 	uh->len = newlen;
 
 	rcu_read_lock();
-	sk = (*lookup)(skb, uh->source, uh->dest);
+	sk = INDIRECT_CALL_INET(lookup, udp_lookup, skb, uh->source, uh->dest);
 	if (sk && udp_sk(sk)->gro_enabled) {
 		err = udp_gro_complete_segment(skb);
 	} else if (sk && udp_sk(sk)->gro_complete) {
net/ipv6/udp.c +2 -0
index 09cba4cfe31f..616f374760d1 100644
@@ -282,6 +282,8 @@ struct sock *udp6_lib_lookup_skb(struct sk_buff *skb,
 				 inet6_sdif(skb), &udp_table, skb);
 }
 EXPORT_SYMBOL_GPL(udp6_lib_lookup_skb);
+INDIRECT_CALLABLE(udp6_lib_lookup_skb, 2, struct sock *, udp_lookup,
+		  struct sk_buff *skb, __be16 sport, __be16 dport);
 
 /* Must be called under rcu_read_lock().
  * Does increment socket refcount.
Signature · 1 line
2.19.2