Skip to contentSkip to conversation

mlx4/en_netdev: allow offloading VXLAN over VLAN

PATCH Superseded Davide Caratti 2 participants 9 messages Jul 24, 2019 – Jul 26, 2019

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover mlx4/en_netdev: allow offloading VXLAN over VLAN

When VXLAN offload is enabled on ConnectX-3 Pro devices, the NIC can segment correctly also VXLAN packet with a VLAN tag: this series ensures that a VLAN created on top of a mlx4_en NIC inherits the tunnel offload capabilities.

Davide Caratti (2):
mlx4/en_netdev: update vlan features with tunnel offloads mlx4/en_netdev: call notifiers when hw_enc_features change

 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 60 ++++++++++++-------
 1 file changed, 37 insertions(+), 23 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/2 mlx4/en_netdev: update vlan features with tunnel offloads

ConnectX-3 Pro can offload transmission of VLAN packets with VXLAN inside:
enable tunnel offloads in dev->vlan_features, like it's done with other NIC drivers (e.g. be2net and ixgbe).

---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 21 ++++++++++---------
 1 file changed, 11 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +11 -10
index c1438ae52a11..52500f744a0e 100644
@@ -3415,6 +3415,17 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 	if (mdev->LSO_support)
 		dev->hw_features |= NETIF_F_TSO | NETIF_F_TSO6;
 
+	if (mdev->dev->caps.tunnel_offload_mode ==
+	    MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
+		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
+	}
+
 	dev->vlan_features = dev->hw_features;
 
 	dev->hw_features |= NETIF_F_RXCSUM | NETIF_F_RXHASH;
@@ -3483,16 +3494,6 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 		priv->rss_hash_fn = ETH_RSS_HASH_TOP;
 	}
 
-	if (mdev->dev->caps.tunnel_offload_mode == MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
-		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				    NETIF_F_GSO_PARTIAL;
-		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				    NETIF_F_GSO_PARTIAL;
-		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
-	}
-
 	/* MTU range: 68 - hw-specific max */
 	dev->min_mtu = ETH_MIN_MTU;
 	dev->max_mtu = priv->max_mtu;
Signature · 1 line
2.20.1
#3 lore raw

PATCH 2/2 mlx4/en_netdev: call notifiers when hw_enc_features change

In reply to Davide Caratti

ensure to call netdev_features_change() when the driver flips its hw_enc_features bits.

---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 39 ++++++++++++-------
 1 file changed, 26 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +26 -13
index 52500f744a0e..1b484dc6e1c2 100644
@@ -2628,6 +2628,30 @@ static int mlx4_en_get_phys_port_id(struct net_device *dev,
 	return 0;
 }
 
+#define MLX4_GSO_PARTIAL_FEATURES (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
+				   NETIF_F_RXCSUM | \
+				   NETIF_F_TSO | NETIF_F_TSO6 | \
+				   NETIF_F_GSO_UDP_TUNNEL | \
+				   NETIF_F_GSO_UDP_TUNNEL_CSUM | \
+				   NETIF_F_GSO_PARTIAL)
+
+static void mlx4_set_vxlan_offloads(struct net_device *dev, bool enable)
+{
+	netdev_features_t hw_enc_features;
+
+	rtnl_lock();
+	hw_enc_features = dev->hw_enc_features;
+	if (enable)
+		dev->hw_enc_features |= MLX4_GSO_PARTIAL_FEATURES;
+	else
+		dev->hw_enc_features &= ~MLX4_GSO_PARTIAL_FEATURES;
+
+	if (hw_enc_features ^ dev->hw_enc_features)
+		netdev_features_change(dev);
+
+	rtnl_unlock();
+}
+
 static void mlx4_en_add_vxlan_offloads(struct work_struct *work)
 {
 	int ret;
@@ -2647,12 +2671,7 @@ static void mlx4_en_add_vxlan_offloads(struct work_struct *work)
 	}
 
 	/* set offloads */
-	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
-				      NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_TSO6 |
-				      NETIF_F_GSO_UDP_TUNNEL |
-				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				      NETIF_F_GSO_PARTIAL;
+	mlx4_set_vxlan_offloads(priv->dev, true);
 }
 
 static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
@@ -2661,13 +2680,7 @@ static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
 	struct mlx4_en_priv *priv = container_of(work, struct mlx4_en_priv,
 						 vxlan_del_task);
 	/* unset offloads */
-	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
-					NETIF_F_RXCSUM |
-					NETIF_F_TSO | NETIF_F_TSO6 |
-					NETIF_F_GSO_UDP_TUNNEL |
-					NETIF_F_GSO_UDP_TUNNEL_CSUM |
-					NETIF_F_GSO_PARTIAL);
-
+	mlx4_set_vxlan_offloads(priv->dev, false);
 	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
 				  VXLAN_STEER_BY_OUTER_MAC, 0);
 	if (ret)
Signature · 1 line
2.20.1
#4 lore raw

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

In reply to Davide Caratti
Quoted 4 lines · Davide Caratti

ensure to call netdev_features_change() when the driver flips its hw_enc_features bits.

The patch is correct, but can you explain how did you come to this ? did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-

Quoted 1 line

hw_enc_features is redundant since mlx4 has the featutres_check

callback.

Quoted 78 lines
---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 39 ++++++++++++-----
--
 1 file changed, 26 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +24 -0
index 52500f744a0e..1b484dc6e1c2 100644
@@ -2628,6 +2628,30 @@ static int mlx4_en_get_phys_port_id(struct
 net_device *dev,
 	return 0;
 }
 
+#define MLX4_GSO_PARTIAL_FEATURES (NETIF_F_IP_CSUM |
 NETIF_F_IPV6_CSUM | \
+				   NETIF_F_RXCSUM | \
+				   NETIF_F_TSO | NETIF_F_TSO6 | \
+				   NETIF_F_GSO_UDP_TUNNEL | \
+				   NETIF_F_GSO_UDP_TUNNEL_CSUM | \
+				   NETIF_F_GSO_PARTIAL)
+
+static void mlx4_set_vxlan_offloads(struct net_device *dev, bool
 enable)
+{
+	netdev_features_t hw_enc_features;
+
+	rtnl_lock();
+	hw_enc_features = dev->hw_enc_features;
+	if (enable)
+		dev->hw_enc_features |= MLX4_GSO_PARTIAL_FEATURES;
+	else
+		dev->hw_enc_features &= ~MLX4_GSO_PARTIAL_FEATURES;
+
+	if (hw_enc_features ^ dev->hw_enc_features)
+		netdev_features_change(dev);
+
+	rtnl_unlock();
+}
+

static void mlx4_en_add_vxlan_offloads(struct work_struct *work) {

drivers/net/ethernet/mellanox/mlx4/en_netdev.c +0 -1
 	int ret;
@@ -2647,12 +2671,7 @@ static void mlx4_en_add_vxlan_offloads(struct
 work_struct *work)
 	}
 
 	/* set offloads */
-	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM |

NETIF_F_IPV6_CSUM |

drivers/net/ethernet/mellanox/mlx4/en_netdev.c +1 -6
-				      NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_TSO6 |
-				      NETIF_F_GSO_UDP_TUNNEL |
-				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				      NETIF_F_GSO_PARTIAL;
+	mlx4_set_vxlan_offloads(priv->dev, true);
 }
 
 static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
@@ -2661,13 +2680,7 @@ static void mlx4_en_del_vxlan_offloads(struct
 work_struct *work)
 	struct mlx4_en_priv *priv = container_of(work, struct
 mlx4_en_priv,
 						 vxlan_del_task);
 	/* unset offloads */
-	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM |

NETIF_F_IPV6_CSUM |

drivers/net/ethernet/mellanox/mlx4/en_netdev.c +1 -6
-					NETIF_F_RXCSUM |
-					NETIF_F_TSO | NETIF_F_TSO6 |
-					NETIF_F_GSO_UDP_TUNNEL |
-					NETIF_F_GSO_UDP_TUNNEL_CSUM |
-					NETIF_F_GSO_PARTIAL);
-
+	mlx4_set_vxlan_offloads(priv->dev, false);
 	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
 				  VXLAN_STEER_BY_OUTER_MAC, 0);
 	if (ret)
#5 lore raw

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

Quoted 7 lines · Saeed Mahameed
Quoted 4 lines · Davide Caratti

ensure to call netdev_features_change() when the driver flips its hw_enc_features bits.

The patch is correct,

hello Saeed, and thanks for looking at this!

Quoted 6 lines

but can you explain how did you come to this ? did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-

Quoted 1 line

hw_enc_features is redundant since mlx4 has the featutres_check

callback.

we need it to ensure that vlan_transfer_features() updates
the (new) value of hw_enc_features in the overlying vlan: otherwise, segmentation will happen anyway when skb passes from vxlan to vlan, if the vxlan is added after the vlan device has been created (see: 7dad9937e064 ("net: vlan: add support for tunnel offload") ).

thanks!

Signature · 1 line
davide
#6 lore raw

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

Quoted 18 lines · Davide Caratti
Quoted 7 lines · Saeed Mahameed
Quoted 4 lines · Davide Caratti

ensure to call netdev_features_change() when the driver flips its hw_enc_features bits.

The patch is correct,

hello Saeed, and thanks for looking at this!

Quoted 6 lines

but can you explain how did you come to this ? did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-

Quoted 1 line

hw_enc_features is redundant since mlx4 has the featutres_check

callback.

Quoted 8 lines

we need it to ensure that vlan_transfer_features() updates the (new) value of hw_enc_features in the overlying vlan: otherwise, segmentation will happen anyway when skb passes from vxlan to vlan, if the
vxlan is added after the vlan device has been created (see:
7dad9937e064
("net: vlan: add support for tunnel offload") ).

but in previous patch you made sure that the vlan always sees the correct hw_enc_features on driver load, we don't need to have this dynamic update mechanism, features_check ndo should take care of protocols we don't support.

Quoted 1 line

thanks!

#7 lore raw

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

Quoted 28 lines · Saeed Mahameed
Quoted 18 lines · Davide Caratti
Quoted 7 lines · Saeed Mahameed
Quoted 4 lines · Davide Caratti

ensure to call netdev_features_change() when the driver flips its hw_enc_features bits.

The patch is correct,

hello Saeed, and thanks for looking at this!

Quoted 6 lines

but can you explain how did you come to this ? did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-

Quoted 1 line

hw_enc_features is redundant since mlx4 has the featutres_check

callback.

Quoted 8 lines

we need it to ensure that vlan_transfer_features() updates the (new) value of hw_enc_features in the overlying vlan: otherwise, segmentation will happen anyway when skb passes from vxlan to vlan, if the
vxlan is added after the vlan device has been created (see:
7dad9937e064
("net: vlan: add support for tunnel offload") ).

Quoted 3 lines

but in previous patch you made sure that the vlan always sees the correct hw_enc_features on driver load, we don't need to have this dynamic update mechanism,

ok, but the mlx4 driver flips the value of hw_enc_features when VXLAN tunnels are added or removed. So, assume eth0 is a Cx3-pro, and I do:

 # ip link add name vlan5 link eth0 type vlan id 5
 # ip link add dev vxlan6 type vxlan id 6  [...]  dev vlan5

the value of dev->hw_enc_features is 0 for vlan5, and as a consequence VXLAN over VLAN traffic becomes segmented by the VLAN, even if eth0, at the end of this sequence, has the "right" features bits.

Quoted 2 lines

features_check ndo should take care of
protocols we don't support.

I just had a look at mlx4_en_features_check(), I see it checks if the packet is tunneled in VXLAN and the destination port matches the configured value of priv->vxlan_port (when that value is not zero). Now:

Quoted 3 lines · Saeed Mahameed

I am asking just because i think the whole dynamic changing of dev-> hw_enc_features is redundant since mlx4 has the featutres_check callback.

I read your initial proposal again. Would it be correct if I just use patch 1/2, where I add an assignment of

dev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
                       NETIF_F_RXCSUM | \
                       NETIF_F_TSO | NETIF_F_TSO6 | \
                       NETIF_F_GSO_UDP_TUNNEL | \
                       NETIF_F_GSO_UDP_TUNNEL_CSUM | \
                       NETIF_F_GSO_PARTIAL;

in mlx4_en_init_netdev(), and then remove the code that flips dev->hw_enc_features in mlx4_en_add_vxlan_offloads() and mlx4_en_del_vxlan_offloads() ?

thanks,
--
davide

#8 lore raw

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

Quoted 41 lines · Davide Caratti
Quoted 33 lines · Saeed Mahameed
Quoted 21 lines · Davide Caratti
Quoted 8 lines · Saeed Mahameed
Quoted 5 lines · Davide Caratti

ensure to call netdev_features_change() when the driver flips its
hw_enc_features bits.

The patch is correct,

hello Saeed, and thanks for looking at this!

Quoted 8 lines

but can you explain how did you come to this ? did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-

Quoted 2 lines

hw_enc_features is redundant since mlx4 has the
featutres_check

callback.

Quoted 10 lines

we need it to ensure that vlan_transfer_features() updates the (new) value of hw_enc_features in the overlying vlan:
otherwise,
segmentation will happen anyway when skb passes from vxlan to vlan,
if the
vxlan is added after the vlan device has been created (see:
7dad9937e064
("net: vlan: add support for tunnel offload") ).

Quoted 3 lines

but in previous patch you made sure that the vlan always sees the correct hw_enc_features on driver load, we don't need to have this dynamic update mechanism,

ok, but the mlx4 driver flips the value of hw_enc_features when VXLAN tunnels are added or removed. So, assume eth0 is a Cx3-pro, and I do:

Quoted 9 lines
 # ip link add name vlan5 link eth0 type vlan id 5
 # ip link add dev vxlan6 type vxlan id 6  [...]  dev vlan5

the value of dev->hw_enc_features is 0 for vlan5, and as a consequence
VXLAN over VLAN traffic becomes segmented by the VLAN, even if eth0, at
the end of this sequence, has the "right" features bits.

your patch handled this issue already, no need for flipping and updating features if features check ndo will cover the cases we don't support.

Quoted 24 lines
Quoted 2 lines

features_check ndo should take care of
protocols we don't support.

I just had a look at mlx4_en_features_check(), I see it checks if the packet is tunneled in VXLAN and the destination port matches the configured value of priv->vxlan_port (when that value is not zero). Now:

Quoted 4 lines · Saeed Mahameed

I am asking just because i think the whole dynamic changing of dev-> hw_enc_features is redundant since mlx4 has the featutres_check
callback.

I read your initial proposal again. Would it be correct if I just use patch 1/2, where I add an assignment of

dev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
                       NETIF_F_RXCSUM | \
                       NETIF_F_TSO | NETIF_F_TSO6 | \
                       NETIF_F_GSO_UDP_TUNNEL | \
                       NETIF_F_GSO_UDP_TUNNEL_CSUM | \
                       NETIF_F_GSO_PARTIAL;
Quoted 4 lines

in mlx4_en_init_netdev(), and then remove the code that flips dev->hw_enc_features in mlx4_en_add_vxlan_offloads() and mlx4_en_del_vxlan_offloads() ?

yes, this is exactly what I meant.

Thanks,
Saeed.