Skip to contentSkip to conversation

Allow to switch off UDP-based tunnel offloads per netdevice

PATCH In review Sabrina Dubroca 2 participants 8 messages Jul 21, 2017 – Jul 24, 2017

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Sabrina Dubroca

author
#0 lore raw

v2 cover Allow to switch off UDP-based tunnel offloads per netdevice

This patchset adds a new netdevice feature to toggle RX offloads of UDP-based tunnel via ethtool. This is useful if the offload is causing issues, for example if the hardware is buggy.

The feature is added to all devices providing the ->ndo_udp_tunnel_add op, and enabled by default to preserve current behavior.

When the administrator disables this feature on a device, all currently offloaded ports are cleared from the device. When the feature is turned on, the stack notifies the device about all current ports.

v2:
- rename feature bit to NETIF_F_RX_UDP_TUNNEL_PORT
- rename ethtool feature to rx-udp_tunnel-port-offload

Sabrina Dubroca (6):
net: add new netdevice feature for offload of RX port for UDP tunnels
net: check UDP tunnel RX port offload feature before calling tunnel
ndo ndo
net: add infrastructure to un-offload UDP tunnel port
net: call udp_tunnel_get_rx_info when NETIF_F_RX_UDP_TUNNEL_PORT is
toggled
geneve/vxlan: add support for NETDEV_UDP_TUNNEL_DROP_INFO geneve/vxlan: offload ports on register/unregister events

 drivers/net/geneve.c            | 24 ++++++++++++++++++------
 drivers/net/vxlan.c             | 31 ++++++++++++++++++++++---------
 include/linux/netdev_features.h |  2 ++
 include/linux/netdevice.h       |  1 +
 include/net/udp_tunnel.h        |  8 ++++++++
 net/core/dev.c                  | 28 +++++++++++++++++++++++++++-
 net/core/ethtool.c              |  1 +
 net/ipv4/udp_tunnel.c           | 25 ++++++++++++++++++++++++-
 8 files changed, 103 insertions(+), 17 deletions(-)
Signature · 1 line
2.13.2
author
#1 lore raw

PATCH v2 1/6 net: add new netdevice feature for offload of RX port for UDP tunnels

This adds a new netdevice feature, so that the offloading of RX port for UDP tunnels can be disabled by the administrator on some netdevices, using the "rx-udp_tunnel-port-offload" feature in ethtool.

This feature is set for all devices that provide ndo_udp_tunnel_add.

Signed-off-by Sabrina Dubroca <[email protected]>
---
 include/linux/netdev_features.h | 2 ++
 net/core/dev.c                  | 6 ++++++
 net/core/ethtool.c              | 1 +
 3 files changed, 9 insertions(+)
include/linux/netdev_features.h +2 -0
index ebd273627334..dc8b4896b77b 100644
@@ -75,6 +75,7 @@ enum {
 	NETIF_F_HW_TC_BIT,		/* Offload TC infrastructure */
 	NETIF_F_HW_ESP_BIT,		/* Hardware ESP transformation offload */
 	NETIF_F_HW_ESP_TX_CSUM_BIT,	/* ESP with TX checksum offload */
+	NETIF_F_RX_UDP_TUNNEL_PORT_BIT, /* Offload of RX port for UDP tunnels */
 
 	/*
 	 * Add your fresh new feature above and remember to update

@@ -138,6 +139,7 @@ enum {
 #define NETIF_F_HW_TC		__NETIF_F(HW_TC)
 #define NETIF_F_HW_ESP		__NETIF_F(HW_ESP)
 #define NETIF_F_HW_ESP_TX_CSUM	__NETIF_F(HW_ESP_TX_CSUM)
+#define	NETIF_F_RX_UDP_TUNNEL_PORT  __NETIF_F(RX_UDP_TUNNEL_PORT)
 
 #define for_each_netdev_feature(mask_addr, bit)	\
 	for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)
net/core/dev.c +6 -0
index 509af6ce8831..9081134adc0d 100644
@@ -7530,6 +7530,12 @@ int register_netdevice(struct net_device *dev)
 	 */
 	dev->hw_features |= NETIF_F_SOFT_FEATURES;
 	dev->features |= NETIF_F_SOFT_FEATURES;
+
+	if (dev->netdev_ops->ndo_udp_tunnel_add) {
+		dev->features |= NETIF_F_RX_UDP_TUNNEL_PORT;
+		dev->hw_features |= NETIF_F_RX_UDP_TUNNEL_PORT;
+	}
+
 	dev->wanted_features = dev->features & dev->hw_features;
 
 	if (!(dev->flags & IFF_LOOPBACK))
net/core/ethtool.c +1 -0
index 78408ab77a10..b987bc475fc8 100644
@@ -105,6 +105,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
 	[NETIF_F_HW_TC_BIT] =		 "hw-tc-offload",
 	[NETIF_F_HW_ESP_BIT] =		 "esp-hw-offload",
 	[NETIF_F_HW_ESP_TX_CSUM_BIT] =	 "esp-tx-csum-hw-offload",
+	[NETIF_F_RX_UDP_TUNNEL_PORT_BIT] =	 "rx-udp_tunnel-port-offload",
 };
 
 static const char
Signature · 1 line
2.13.2
author
#2 lore raw

PATCH v2 2/6 net: check UDP tunnel RX port offload feature before calling tunnel ndo ndo

In reply to Sabrina Dubroca

If NETIF_F_RX_UDP_TUNNEL_PORT was disabled on a given netdevice, skip the tunnel offload ndo call during tunnel port creation and deletion.

Signed-off-by Sabrina Dubroca <[email protected]>
---
 net/ipv4/udp_tunnel.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
net/ipv4/udp_tunnel.c +6 -1
index 58bd39fb14b4..0d3f14cdc524 100644
@@ -82,7 +82,8 @@ void udp_tunnel_push_rx_port(struct net_device *dev, struct socket *sock,
 	struct sock *sk = sock->sk;
 	struct udp_tunnel_info ti;
 
-	if (!dev->netdev_ops->ndo_udp_tunnel_add)
+	if (!dev->netdev_ops->ndo_udp_tunnel_add ||
+	    !(dev->features & NETIF_F_RX_UDP_TUNNEL_PORT))
 		return;
 
 	ti.type = type;

@@ -109,6 +110,8 @@ void udp_tunnel_notify_add_rx_port(struct socket *sock, unsigned short type)
 	for_each_netdev_rcu(net, dev) {
 		if (!dev->netdev_ops->ndo_udp_tunnel_add)
 			continue;
+		if (!(dev->features & NETIF_F_RX_UDP_TUNNEL_PORT))
+			continue;
 		dev->netdev_ops->ndo_udp_tunnel_add(dev, &ti);
 	}
 	rcu_read_unlock();

@@ -131,6 +134,8 @@ void udp_tunnel_notify_del_rx_port(struct socket *sock, unsigned short type)
 	for_each_netdev_rcu(net, dev) {
 		if (!dev->netdev_ops->ndo_udp_tunnel_del)
 			continue;
+		if (!(dev->features & NETIF_F_RX_UDP_TUNNEL_PORT))
+			continue;
 		dev->netdev_ops->ndo_udp_tunnel_del(dev, &ti);
 	}
 	rcu_read_unlock();
Signature · 1 line
2.13.2
author
#3 lore raw

PATCH v2 3/6 net: add infrastructure to un-offload UDP tunnel port

In reply to Sabrina Dubroca

This adds a new NETDEV_UDP_TUNNEL_DROP_INFO event, similar to NETDEV_UDP_TUNNEL_PUSH_INFO, to signal to un-offload ports.

This also adds udp_tunnel_drop_rx_port(), which calls ndo_udp_tunnel_del.

Signed-off-by Sabrina Dubroca <[email protected]>
---
 include/linux/netdevice.h |  1 +
 include/net/udp_tunnel.h  |  8 ++++++++
 net/ipv4/udp_tunnel.c     | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+)
include/linux/netdevice.h +1 -0
index 614642eb7eb7..3a3cdc1b1f31 100644
@@ -2317,6 +2317,7 @@ struct netdev_lag_lower_state_info {
 #define NETDEV_PRECHANGEUPPER	0x001A
 #define NETDEV_CHANGELOWERSTATE	0x001B
 #define NETDEV_UDP_TUNNEL_PUSH_INFO	0x001C
+#define NETDEV_UDP_TUNNEL_DROP_INFO	0x001D
 #define NETDEV_CHANGE_TX_QUEUE_LEN	0x001E
 
 int register_netdevice_notifier(struct notifier_block *nb);
include/net/udp_tunnel.h +8 -0
index 02c5be037451..10cce0dd4450 100644
@@ -115,6 +115,8 @@ struct udp_tunnel_info {
 /* Notify network devices of offloadable types */
 void udp_tunnel_push_rx_port(struct net_device *dev, struct socket *sock,
 			     unsigned short type);
+void udp_tunnel_drop_rx_port(struct net_device *dev, struct socket *sock,
+			     unsigned short type);
 void udp_tunnel_notify_add_rx_port(struct socket *sock, unsigned short type);
 void udp_tunnel_notify_del_rx_port(struct socket *sock, unsigned short type);
 

@@ -124,6 +126,12 @@ static inline void udp_tunnel_get_rx_info(struct net_device *dev)
 	call_netdevice_notifiers(NETDEV_UDP_TUNNEL_PUSH_INFO, dev);
 }
 
+static inline void udp_tunnel_drop_rx_info(struct net_device *dev)
+{
+	ASSERT_RTNL();
+	call_netdevice_notifiers(NETDEV_UDP_TUNNEL_DROP_INFO, dev);
+}
+
 /* Transmit the skb using UDP encapsulation. */
 void udp_tunnel_xmit_skb(struct rtable *rt, struct sock *sk, struct sk_buff *skb,
 			 __be32 src, __be32 dst, __u8 tos, __u8 ttl,
net/ipv4/udp_tunnel.c +18 -0
index 0d3f14cdc524..6539ff15e9a3 100644
@@ -94,6 +94,24 @@ void udp_tunnel_push_rx_port(struct net_device *dev, struct socket *sock,
 }
 EXPORT_SYMBOL_GPL(udp_tunnel_push_rx_port);
 
+void udp_tunnel_drop_rx_port(struct net_device *dev, struct socket *sock,
+			     unsigned short type)
+{
+	struct sock *sk = sock->sk;
+	struct udp_tunnel_info ti;
+
+	if (!dev->netdev_ops->ndo_udp_tunnel_del ||
+	    !(dev->features & NETIF_F_RX_UDP_TUNNEL_PORT))
+		return;
+
+	ti.type = type;
+	ti.sa_family = sk->sk_family;
+	ti.port = inet_sk(sk)->inet_sport;
+
+	dev->netdev_ops->ndo_udp_tunnel_del(dev, &ti);
+}
+EXPORT_SYMBOL_GPL(udp_tunnel_drop_rx_port);
+
 /* Notify netdevs that UDP port started listening */
 void udp_tunnel_notify_add_rx_port(struct socket *sock, unsigned short type)
 {
Signature · 1 line
2.13.2
author
#4 lore raw

PATCH v2 4/6 net: call udp_tunnel_get_rx_info when NETIF_F_RX_UDP_TUNNEL_PORT is toggled

In reply to Sabrina Dubroca

NETIF_F_RX_UDP_TUNNEL_PORT is special, in that we need to do more than just flip the bit in dev->features. When disabling we must also clear currently offloaded ports from the device, and when enabling we must tell the device to offload the ports it can.

Because vxlan stores its sockets in a hashtable and they are inserted at the head of per-bucket lists, switching the feature off and then on can result in a different set of ports being offloaded (depending on the HW's limits).

Signed-off-by Sabrina Dubroca <[email protected]>
---
 net/core/dev.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
net/core/dev.c +21 -1
index 9081134adc0d..8ea6b4b42611 100644
@@ -144,6 +144,7 @@
 #include <linux/netfilter_ingress.h>
 #include <linux/crash_dump.h>
 #include <linux/sctp.h>
+#include <net/udp_tunnel.h>
 
 #include "net-sysfs.h"
 

@@ -7327,8 +7328,27 @@ int __netdev_update_features(struct net_device *dev)
 	netdev_for_each_lower_dev(dev, lower, iter)
 		netdev_sync_lower_features(dev, lower, features);
 
-	if (!err)
+	if (!err) {
+		netdev_features_t diff = features ^ dev->features;
+
+		if (diff & NETIF_F_RX_UDP_TUNNEL_PORT) {
+			/* udp_tunnel_{get,drop}_rx_info both need
+			 * NETIF_F_RX_UDP_TUNNEL_PORT enabled on the
+			 * device, or they won't do anything.
+			 * Thus we need to update dev->features
+			 * *before* calling udp_tunnel_get_rx_info,
+			 * but *after* calling udp_tunnel_drop_rx_info.
+			 */
+			if (features & NETIF_F_RX_UDP_TUNNEL_PORT) {
+				dev->features = features;
+				udp_tunnel_get_rx_info(dev);
+			} else {
+				udp_tunnel_drop_rx_info(dev);
+			}
+		}
+
 		dev->features = features;
+	}
 
 	return err < 0 ? 0 : 1;
 }
Signature · 1 line
2.13.2
author
#5 lore raw

PATCH v2 5/6 geneve/vxlan: add support for NETDEV_UDP_TUNNEL_DROP_INFO

In reply to Sabrina Dubroca
Signed-off-by Sabrina Dubroca <[email protected]>
---
 drivers/net/geneve.c | 19 +++++++++++++------
 drivers/net/vxlan.c  | 25 +++++++++++++++++--------
 2 files changed, 30 insertions(+), 14 deletions(-)
drivers/net/geneve.c +13 -6
index de8156c6b292..ee52233750f7 100644
@@ -1014,16 +1014,22 @@ static struct device_type geneve_type = {
  * supply the listening GENEVE udp ports. Callers are expected
  * to implement the ndo_udp_tunnel_add.
  */
-static void geneve_push_rx_ports(struct net_device *dev)
+static void geneve_offload_rx_ports(struct net_device *dev, bool push)
 {
 	struct net *net = dev_net(dev);
 	struct geneve_net *gn = net_generic(net, geneve_net_id);
 	struct geneve_sock *gs;
 
 	rcu_read_lock();
-	list_for_each_entry_rcu(gs, &gn->sock_list, list)
-		udp_tunnel_push_rx_port(dev, gs->sock,
-					UDP_TUNNEL_TYPE_GENEVE);
+	list_for_each_entry_rcu(gs, &gn->sock_list, list) {
+		if (push) {
+			udp_tunnel_push_rx_port(dev, gs->sock,
+						UDP_TUNNEL_TYPE_GENEVE);
+		} else {
+			udp_tunnel_drop_rx_port(dev, gs->sock,
+						UDP_TUNNEL_TYPE_GENEVE);
+		}
+	}
 	rcu_read_unlock();
 }
 

@@ -1426,8 +1432,9 @@ static int geneve_netdevice_event(struct notifier_block *unused,
 {
 	struct net_device *dev = netdev_notifier_info_to_dev(ptr);
 
-	if (event == NETDEV_UDP_TUNNEL_PUSH_INFO)
-		geneve_push_rx_ports(dev);
+	if (event == NETDEV_UDP_TUNNEL_PUSH_INFO ||
+	    event == NETDEV_UDP_TUNNEL_DROP_INFO)
+		geneve_offload_rx_ports(dev, event == NETDEV_UDP_TUNNEL_PUSH_INFO);
 
 	return NOTIFY_DONE;
 }
drivers/net/vxlan.c +17 -8
index 96aa7e6cf214..4642d5be2fa0 100644
@@ -2608,7 +2608,7 @@ static struct device_type vxlan_type = {
  * supply the listening VXLAN udp ports. Callers are expected
  * to implement the ndo_udp_tunnel_add.
  */
-static void vxlan_push_rx_ports(struct net_device *dev)
+static void vxlan_offload_rx_ports(struct net_device *dev, bool push)
 {
 	struct vxlan_sock *vs;
 	struct net *net = dev_net(dev);

@@ -2617,11 +2617,19 @@ static void vxlan_push_rx_ports(struct net_device *dev)
 
 	spin_lock(&vn->sock_lock);
 	for (i = 0; i < PORT_HASH_SIZE; ++i) {
-		hlist_for_each_entry_rcu(vs, &vn->sock_list[i], hlist)
-			udp_tunnel_push_rx_port(dev, vs->sock,
-						(vs->flags & VXLAN_F_GPE) ?
-						UDP_TUNNEL_TYPE_VXLAN_GPE :
-						UDP_TUNNEL_TYPE_VXLAN);
+		hlist_for_each_entry_rcu(vs, &vn->sock_list[i], hlist) {
+			unsigned short type;
+
+			if (vs->flags & VXLAN_F_GPE)
+				type = UDP_TUNNEL_TYPE_VXLAN_GPE;
+			else
+				type = UDP_TUNNEL_TYPE_VXLAN;
+
+			if (push)
+				udp_tunnel_push_rx_port(dev, vs->sock, type);
+			else
+				udp_tunnel_drop_rx_port(dev, vs->sock, type);
+		}
 	}
 	spin_unlock(&vn->sock_lock);
 }

@@ -3632,8 +3640,9 @@ static int vxlan_netdevice_event(struct notifier_block *unused,
 
 	if (event == NETDEV_UNREGISTER)
 		vxlan_handle_lowerdev_unregister(vn, dev);
-	else if (event == NETDEV_UDP_TUNNEL_PUSH_INFO)
-		vxlan_push_rx_ports(dev);
+	else if (event == NETDEV_UDP_TUNNEL_PUSH_INFO ||
+		 event == NETDEV_UDP_TUNNEL_DROP_INFO)
+		vxlan_offload_rx_ports(dev, event == NETDEV_UDP_TUNNEL_PUSH_INFO);
 
 	return NOTIFY_DONE;
 }
Signature · 1 line
2.13.2
author
#6 lore raw

PATCH v2 6/6 geneve/vxlan: offload ports on register/unregister events

In reply to Sabrina Dubroca

This improves consistency of handling when moving a netdev to another netns. Most drivers currently do a full reset when the device goes up, so that will flush the offload state anyway.

Signed-off-by Sabrina Dubroca <[email protected]>
---
 drivers/net/geneve.c |  7 ++++++-
 drivers/net/vxlan.c  | 10 +++++++---
 2 files changed, 13 insertions(+), 4 deletions(-)
drivers/net/geneve.c +6 -1
index ee52233750f7..6c6350b5c201 100644
@@ -1433,8 +1433,13 @@ static int geneve_netdevice_event(struct notifier_block *unused,
 	struct net_device *dev = netdev_notifier_info_to_dev(ptr);
 
 	if (event == NETDEV_UDP_TUNNEL_PUSH_INFO ||
-	    event == NETDEV_UDP_TUNNEL_DROP_INFO)
+	    event == NETDEV_UDP_TUNNEL_DROP_INFO) {
 		geneve_offload_rx_ports(dev, event == NETDEV_UDP_TUNNEL_PUSH_INFO);
+	} else if (event == NETDEV_UNREGISTER) {
+		geneve_offload_rx_ports(dev, false);
+	} else if (event == NETDEV_REGISTER) {
+		geneve_offload_rx_ports(dev, true);
+	}
 
 	return NOTIFY_DONE;
 }
drivers/net/vxlan.c +7 -3
index 4642d5be2fa0..dbca067540d0 100644
@@ -3638,11 +3638,15 @@ static int vxlan_netdevice_event(struct notifier_block *unused,
 	struct net_device *dev = netdev_notifier_info_to_dev(ptr);
 	struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
 
-	if (event == NETDEV_UNREGISTER)
+	if (event == NETDEV_UNREGISTER) {
+		vxlan_offload_rx_ports(dev, false);
 		vxlan_handle_lowerdev_unregister(vn, dev);
-	else if (event == NETDEV_UDP_TUNNEL_PUSH_INFO ||
-		 event == NETDEV_UDP_TUNNEL_DROP_INFO)
+	} else if (event == NETDEV_REGISTER) {
+		vxlan_offload_rx_ports(dev, true);
+	} else if (event == NETDEV_UDP_TUNNEL_PUSH_INFO ||
+		   event == NETDEV_UDP_TUNNEL_DROP_INFO) {
 		vxlan_offload_rx_ports(dev, event == NETDEV_UDP_TUNNEL_PUSH_INFO);
+	}
 
 	return NOTIFY_DONE;
 }
Signature · 1 line
2.13.2
maintainer
#7 lore raw
In reply to Sabrina Dubroca

From: Sabrina Dubroca <[email protected]>
Date: Fri, 21 Jul 2017 12:49:27 +0200

Quoted 7 lines

This patchset adds a new netdevice feature to toggle RX offloads of UDP-based tunnel via ethtool. This is useful if the offload is causing issues, for example if the hardware is buggy.

The feature is added to all devices providing the ->ndo_udp_tunnel_add op, and enabled by default to preserve current behavior.

Quoted 8 lines

When the administrator disables this feature on a device, all currently offloaded ports are cleared from the device. When the feature is turned on, the stack notifies the device about all current ports.

v2:
- rename feature bit to NETIF_F_RX_UDP_TUNNEL_PORT
- rename ethtool feature to rx-udp_tunnel-port-offload

Series applied, thanks Sabrina.