udp_tunnel: GRO optimizations

PATCH Applied Paolo Abeni 3 participants 6 messages Apr 7, 2025 – Apr 9, 2025

Subsystems (3)

Networking Network drivers UDP

lore

Share & tools

lore

Paolo Abeni

author maintainer

2025-04-07 15:46 UTC #0 lore raw

v5 cover udp_tunnel: GRO optimizations

The UDP tunnel GRO stage is source of measurable overhead for workload based on UDP-encapsulated traffic: each incoming packets requires a full UDP socket lookup and an indirect call.

In the most common setups a single UDP tunnel device is used. In such case we can optimize both the lookup and the indirect call.

Patch 1 tracks per netns the active UDP tunnels and replaces the socket lookup with a single destination port comparison when possible.

Patch 2 tracks the different types of UDP tunnels and replaces the indirect call with a static one when there is a single UDP tunnel type active.

I measure ~10% performance improvement in TCP over UDP tunnel stream tests on top of this series.
---
v4 -> v5:
- incorporated the follow-up fixes:
https://lore.kernel.org/netdev/[email protected]/
- more accurate benchmarking
v4: https://lore.kernel.org/netdev/[email protected]/

v3 -> v4:
- cleanup lookup in patch 2
- more RCU_INIT_POINTER usage in patch 1
(see the individual patches changelog for more details)
v3: https://lore.kernel.org/netdev/[email protected]

v2 -> v3:
- avoid unneeded checks in udp_tunnel_update_gro_rcv()
- use RCU_INIT_POINTER() when possible
- drop 'inline' from c file
v2: https://lore.kernel.org/netdev/[email protected]/

v1 -> v2:
- fixed a couple of typos
- fixed UDP_TUNNEL=n build
- clarified design choices
(see the individual patches changelog for more details)
v1: https://lore.kernel.org/netdev/[email protected]/

Paolo Abeni (2):
udp_tunnel: create a fastpath GRO lookup.
udp_tunnel: use static call for GRO hooks when possible

 include/linux/udp.h        |  16 ++++
 include/net/netns/ipv4.h   |  11 +++
 include/net/udp.h          |   1 +
 include/net/udp_tunnel.h   |  16 ++++
 net/ipv4/udp.c             |  13 ++-
 net/ipv4/udp_offload.c     | 172 ++++++++++++++++++++++++++++++++++++-
 net/ipv4/udp_tunnel_core.c |  13 +++
 net/ipv6/udp.c             |   2 +
 net/ipv6/udp_offload.c     |   5 ++
 9 files changed, 247 insertions(+), 2 deletions(-)

Signature · 1 line

2.49.0

Paolo Abeni

author maintainer

2025-04-07 15:46 UTC #1 lore raw

PATCH v5 1/2 udp_tunnel: create a fastpath GRO lookup.

Most UDP tunnels bind a socket to a local port, with ANY address, no peer and no interface index specified.
Additionally it's quite common to have a single tunnel device per namespace.

Track in each namespace the UDP tunnel socket respecting the above. When only a single one is present, store a reference in the netns.

When such reference is not NULL, UDP tunnel GRO lookup just need to match the incoming packet destination port vs the socket local port.

The tunnel socket never sets the reuse[port] flag[s]. When bound to no address and interface, no other socket can exist in the same netns matching the specified local port.

Matching packets with non-local destination addresses will be aggregated, and eventually segmented as needed - no behavior changes intended.

Restrict the optimization to kernel sockets only: it covers all the relevant use-cases, and user-space owned sockets could be disconnected and rebound after setup_udp_tunnel_sock(), breaking the uniqueness assumption

Note that the UDP tunnel socket reference is stored into struct netns_ipv4 for both IPv4 and IPv6 tunnels. That is intentional to keep all the fastpath-related netns fields in the same struct and allow cacheline-based optimization. Currently both the IPv4 and IPv6 socket pointer share the same cacheline as the udp_table field.

Signed-off-by Paolo Abeni <[email protected]>

---
v4 -> v5:
- deal with races at removal time moving the list check under the tunnel gro lock
- restrict to kernel socket only.

---
 include/linux/udp.h        | 16 ++++++++++++++++
 include/net/netns/ipv4.h   | 11 +++++++++++
 include/net/udp.h          |  1 +
 include/net/udp_tunnel.h   | 12 ++++++++++++
 net/ipv4/udp.c             | 13 ++++++++++++-
 net/ipv4/udp_offload.c     | 37 +++++++++++++++++++++++++++++++++++++
 net/ipv4/udp_tunnel_core.c | 13 +++++++++++++
 net/ipv6/udp.c             |  2 ++
 net/ipv6/udp_offload.c     |  5 +++++
 9 files changed, 109 insertions(+), 1 deletion(-)

include/linux/udp.h +16 -0

index 0807e21cfec95..895240177f4f4 100644
@@ -101,6 +101,13 @@ struct udp_sock {
 
 	/* Cache friendly copy of sk->sk_peek_off >= 0 */
 	bool		peeking_with_offset;
+
+	/*
+	 * Accounting for the tunnel GRO fastpath.
+	 * Unprotected by compilers guard, as it uses space available in
+	 * the last UDP socket cacheline.
+	 */
+	struct hlist_node	tunnel_list;
 };
 
 #define udp_test_bit(nr, sk)			\

@@ -219,4 +226,13 @@ static inline void udp_allow_gso(struct sock *sk)
 
 #define IS_UDPLITE(__sk) (__sk->sk_protocol == IPPROTO_UDPLITE)
 
+static inline struct sock *udp_tunnel_sk(const struct net *net, bool is_ipv6)
+{
+#if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
+	return rcu_dereference(net->ipv4.udp_tunnel_gro[is_ipv6].sk);
+#else
+	return NULL;
+#endif
+}
+
 #endif	/* _LINUX_UDP_H */

include/net/netns/ipv4.h +11 -0

index 650b2dc9199f4..6373e3f17da84 100644
@@ -47,6 +47,11 @@ struct sysctl_fib_multipath_hash_seed {
 };
 #endif
 
+struct udp_tunnel_gro {
+	struct sock __rcu *sk;
+	struct hlist_head list;
+};
+
 struct netns_ipv4 {
 	/* Cacheline organization can be found documented in
 	 * Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst.

@@ -85,6 +90,11 @@ struct netns_ipv4 {
 	struct inet_timewait_death_row tcp_death_row;
 	struct udp_table *udp_table;
 
+#if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
+	/* Not in a pernet subsys because need to be available at GRO stage */
+	struct udp_tunnel_gro udp_tunnel_gro[2];
+#endif
+
 #ifdef CONFIG_SYSCTL
 	struct ctl_table_header	*forw_hdr;
 	struct ctl_table_header	*frags_hdr;

@@ -277,4 +287,5 @@ struct netns_ipv4 {
 	struct hlist_head	*inet_addr_lst;
 	struct delayed_work	addr_chk_work;
 };
+
 #endif

include/net/udp.h +1 -0

index 6e89520e100dc..a772510b2aa58 100644
@@ -290,6 +290,7 @@ static inline void udp_lib_init_sock(struct sock *sk)
 	struct udp_sock *up = udp_sk(sk);
 
 	skb_queue_head_init(&up->reader_queue);
+	INIT_HLIST_NODE(&up->tunnel_list);
 	up->forward_threshold = sk->sk_rcvbuf >> 2;
 	set_bit(SOCK_CUSTOM_SOCKOPT, &sk->sk_socket->flags);
 }

include/net/udp_tunnel.h +12 -0

index a93dc51f6323e..1bb2b852e90eb 100644
@@ -191,6 +191,18 @@ static inline int udp_tunnel_handle_offloads(struct sk_buff *skb, bool udp_csum)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
+void udp_tunnel_update_gro_lookup(struct net *net, struct sock *sk, bool add);
+#else
+static inline void udp_tunnel_update_gro_lookup(struct net *net,
+						struct sock *sk, bool add) {}
+#endif
+
+static inline void udp_tunnel_cleanup_gro(struct sock *sk)
+{
+	udp_tunnel_update_gro_lookup(sock_net(sk), sk, false);
+}
+
 static inline void udp_tunnel_encap_enable(struct sock *sk)
 {
 	if (udp_test_and_set_bit(ENCAP_ENABLED, sk))

net/ipv4/udp.c +12 -1

index 2742cc7602bb5..04cd3353f1d46 100644
@@ -2897,8 +2897,10 @@ void udp_destroy_sock(struct sock *sk)
 			if (encap_destroy)
 				encap_destroy(sk);
 		}
-		if (udp_test_bit(ENCAP_ENABLED, sk))
+		if (udp_test_bit(ENCAP_ENABLED, sk)) {
 			static_branch_dec(&udp_encap_needed_key);
+			udp_tunnel_cleanup_gro(sk);
+		}
 	}
 }

@@ -3810,6 +3812,15 @@ static void __net_init udp_set_table(struct net *net)
 
 static int __net_init udp_pernet_init(struct net *net)
 {
+#if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
+	int i;
+
+	/* No tunnel is configured */
+	for (i = 0; i < ARRAY_SIZE(net->ipv4.udp_tunnel_gro); ++i) {
+		INIT_HLIST_HEAD(&net->ipv4.udp_tunnel_gro[i].list);
+		RCU_INIT_POINTER(net->ipv4.udp_tunnel_gro[i].sk, NULL);
+	}
+#endif
 	udp_sysctl_init(net);
 	udp_set_table(net);

net/ipv4/udp_offload.c +37 -0

index 2c0725583be39..69fc4eae82509 100644
@@ -12,6 +12,38 @@
 #include <net/udp.h>
 #include <net/protocol.h>
 #include <net/inet_common.h>
+#include <net/udp_tunnel.h>
+
+#if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
+static DEFINE_SPINLOCK(udp_tunnel_gro_lock);
+
+void udp_tunnel_update_gro_lookup(struct net *net, struct sock *sk, bool add)
+{
+	bool is_ipv6 = sk->sk_family == AF_INET6;
+	struct udp_sock *tup, *up = udp_sk(sk);
+	struct udp_tunnel_gro *udp_tunnel_gro;
+
+	spin_lock(&udp_tunnel_gro_lock);
+	udp_tunnel_gro = &net->ipv4.udp_tunnel_gro[is_ipv6];
+	if (add)
+		hlist_add_head(&up->tunnel_list, &udp_tunnel_gro->list);
+	else if (up->tunnel_list.pprev)
+		hlist_del_init(&up->tunnel_list);
+
+	if (udp_tunnel_gro->list.first &&
+	    !udp_tunnel_gro->list.first->next) {
+		tup = hlist_entry(udp_tunnel_gro->list.first, struct udp_sock,
+				  tunnel_list);
+
+		rcu_assign_pointer(udp_tunnel_gro->sk, (struct sock *)tup);
+	} else {
+		RCU_INIT_POINTER(udp_tunnel_gro->sk, NULL);
+	}
+
+	spin_unlock(&udp_tunnel_gro_lock);
+}
+EXPORT_SYMBOL_GPL(udp_tunnel_update_gro_lookup);
+#endif
 
 static struct sk_buff *__skb_udp_tunnel_segment(struct sk_buff *skb,
 	netdev_features_t features,

@@ -635,8 +667,13 @@ static struct sock *udp4_gro_lookup_skb(struct sk_buff *skb, __be16 sport,
 {
 	const struct iphdr *iph = skb_gro_network_header(skb);
 	struct net *net = dev_net_rcu(skb->dev);
+	struct sock *sk;
 	int iif, sdif;
 
+	sk = udp_tunnel_sk(net, false);
+	if (sk && dport == htons(sk->sk_num))
+		return sk;
+
 	inet_get_iif_sdif(skb, &iif, &sdif);
 
 	return __udp4_lib_lookup(net, iph->saddr, sport,

net/ipv4/udp_tunnel_core.c +13 -0

index 619a53eb672da..3d1214e6df0ea 100644
@@ -58,6 +58,15 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg,
 }
 EXPORT_SYMBOL(udp_sock_create4);
 
+static bool sk_saddr_any(struct sock *sk)
+{
+#if IS_ENABLED(CONFIG_IPV6)
+	return ipv6_addr_any(&sk->sk_v6_rcv_saddr);
+#else
+	return !sk->sk_rcv_saddr;
+#endif
+}
+
 void setup_udp_tunnel_sock(struct net *net, struct socket *sock,
 			   struct udp_tunnel_sock_cfg *cfg)
 {

@@ -80,6 +89,10 @@ void setup_udp_tunnel_sock(struct net *net, struct socket *sock,
 	udp_sk(sk)->gro_complete = cfg->gro_complete;
 
 	udp_tunnel_encap_enable(sk);
+
+	if (!sk->sk_dport && !sk->sk_bound_dev_if && sk_saddr_any(sk) &&
+	    sk->sk_kern_sock)
+		udp_tunnel_update_gro_lookup(net, sk, true);
 }
 EXPORT_SYMBOL_GPL(setup_udp_tunnel_sock);

net/ipv6/udp.c +2 -0

index 024458ef163c9..7317f8e053f1c 100644
@@ -46,6 +46,7 @@
 #include <net/tcp_states.h>
 #include <net/ip6_checksum.h>
 #include <net/ip6_tunnel.h>
+#include <net/udp_tunnel.h>
 #include <net/xfrm.h>
 #include <net/inet_hashtables.h>
 #include <net/inet6_hashtables.h>

@@ -1825,6 +1826,7 @@ void udpv6_destroy_sock(struct sock *sk)
 		if (udp_test_bit(ENCAP_ENABLED, sk)) {
 			static_branch_dec(&udpv6_encap_needed_key);
 			udp_encap_disable();
+			udp_tunnel_cleanup_gro(sk);
 		}
 	}
 }

net/ipv6/udp_offload.c +5 -0

index 404212dfc99ab..d8445ac1b2e43 100644
@@ -118,8 +118,13 @@ static struct sock *udp6_gro_lookup_skb(struct sk_buff *skb, __be16 sport,
 {
 	const struct ipv6hdr *iph = skb_gro_network_header(skb);
 	struct net *net = dev_net_rcu(skb->dev);
+	struct sock *sk;
 	int iif, sdif;
 
+	sk = udp_tunnel_sk(net, true);
+	if (sk && dport == htons(sk->sk_num))
+		return sk;
+
 	inet6_get_iif_sdif(skb, &iif, &sdif);
 
 	return __udp6_lib_lookup(net, &iph->saddr, sport,

Signature · 1 line

2.49.0

Willem de Bruijn

maintainer Reviewed-by on v5 1/2

2025-04-07 22:27 UTC #2 lore raw

Re: [PATCH net-next v5 1/2] udp_tunnel: create a fastpath GRO lookup.

Quoted 24 lines · Paolo Abeni

Most UDP tunnels bind a socket to a local port, with ANY address, no peer and no interface index specified.
Additionally it's quite common to have a single tunnel device per namespace.
Track in each namespace the UDP tunnel socket respecting the above. When only a single one is present, store a reference in the netns.
When such reference is not NULL, UDP tunnel GRO lookup just need to match the incoming packet destination port vs the socket local port.
The tunnel socket never sets the reuse[port] flag[s]. When bound to no address and interface, no other socket can exist in the same netns matching the specified local port.
Matching packets with non-local destination addresses will be aggregated, and eventually segmented as needed - no behavior changes intended.
Restrict the optimization to kernel sockets only: it covers all the relevant use-cases, and user-space owned sockets could be disconnected and rebound after setup_udp_tunnel_sock(), breaking the uniqueness assumption

Quoted 7 lines

Note that the UDP tunnel socket reference is stored into struct netns_ipv4 for both IPv4 and IPv6 tunnels. That is intentional to keep all the fastpath-related netns fields in the same struct and allow cacheline-based optimization. Currently both the IPv4 and IPv6 socket pointer share the same cacheline as the udp_table field.
Signed-off-by Paolo Abeni <[email protected]>

Reviewed-by Willem de Bruijn <[email protected]>

Paolo Abeni

author maintainer

2025-04-07 15:46 UTC #3 lore raw

PATCH v5 2/2 udp_tunnel: use static call for GRO hooks when possible

In reply to Paolo Abeni

It's quite common to have a single UDP tunnel type active in the whole system. In such a case we can replace the indirect call for the UDP tunnel GRO callback with a static call.

Add the related accounting in the control path and switch to static call when possible. To keep the code simple use a static array for the registered tunnel types, and size such array based on the kernel config.

Note that there are valid kernel configurations leading to UDP_MAX_TUNNEL_TYPES == 0 even with IS_ENABLED(CONFIG_NET_UDP_TUNNEL), Explicitly skip the accounting in such a case, to avoid compile warning when accessing "udp_tunnel_gro_types".

Signed-off-by Paolo Abeni <[email protected]>

---
v4 -> v5:
- move the add hook in udp_tunnel_encap_enable() to deal with xfrm
- avoid warnings when UDP_MAX_TUNNEL_TYPES == 0

---
 include/net/udp_tunnel.h |   4 ++
 net/ipv4/udp_offload.c   | 135 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 138 insertions(+), 1 deletion(-)

include/net/udp_tunnel.h +4 -0

index 1bb2b852e90eb..288f06f23a804 100644
@@ -193,13 +193,16 @@ static inline int udp_tunnel_handle_offloads(struct sk_buff *skb, bool udp_csum)
 
 #if IS_ENABLED(CONFIG_NET_UDP_TUNNEL)
 void udp_tunnel_update_gro_lookup(struct net *net, struct sock *sk, bool add);
+void udp_tunnel_update_gro_rcv(struct sock *sk, bool add);
 #else
 static inline void udp_tunnel_update_gro_lookup(struct net *net,
 						struct sock *sk, bool add) {}
+static inline void udp_tunnel_update_gro_rcv(struct sock *sk, bool add) {}
 #endif
 
 static inline void udp_tunnel_cleanup_gro(struct sock *sk)
 {
+	udp_tunnel_update_gro_rcv(sk, false);
 	udp_tunnel_update_gro_lookup(sock_net(sk), sk, false);
 }

@@ -212,6 +215,7 @@ static inline void udp_tunnel_encap_enable(struct sock *sk)
 	if (READ_ONCE(sk->sk_family) == PF_INET6)
 		ipv6_stub->udpv6_encap_enable();
 #endif
+	udp_tunnel_update_gro_rcv(sk, true);
 	udp_encap_enable();
 }

net/ipv4/udp_offload.c +134 -1

158 lines not on this page · raw message

Signature · 1 line

2.49.0

Willem de Bruijn

maintainer Reviewed-by on v5 2/2

2025-04-07 22:32 UTC #4 lore raw

Re: [PATCH net-next v5 2/2] udp_tunnel: use static call for GRO hooks when possible

Quoted 9 lines · Paolo Abeni

It's quite common to have a single UDP tunnel type active in the whole system. In such a case we can replace the indirect call for the UDP tunnel GRO callback with a static call.
Add the related accounting in the control path and switch to static call when possible. To keep the code simple use a static array for the registered tunnel types, and size such array based on the kernel config.

Quoted 6 lines

Note that there are valid kernel configurations leading to UDP_MAX_TUNNEL_TYPES == 0 even with IS_ENABLED(CONFIG_NET_UDP_TUNNEL), Explicitly skip the accounting in such a case, to avoid compile warning when accessing "udp_tunnel_gro_types".
Signed-off-by Paolo Abeni <[email protected]>

Reviewed-by Willem de Bruijn <[email protected]>

[email protected]

CI Applied

2025-04-09 02:40 UTC #5 lore raw

In reply to Paolo Abeni

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Mon, 7 Apr 2025 17:45:40 +0200 you wrote:

The UDP tunnel GRO stage is source of measurable overhead for workload based on UDP-encapsulated traffic: each incoming packets requires a full UDP socket lookup and an indirect call.
In the most common setups a single UDP tunnel device is used. In such case we can optimize both the lookup and the indirect call.
[...]

Here is the summary with links:
- [net-next,v5,1/2] udp_tunnel: create a fastpath GRO lookup.

    https://git.kernel.org/netdev/net-next/c/a36283e2b683
  - [net-next,v5,2/2] udp_tunnel: use static call for GRO hooks when possible
    https://git.kernel.org/netdev/net-next/c/5d7f5b2f6b93

You are awesome, thank you!

Signature · 2 lines

Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html