Skip to contentSkip to conversation

mlx5 IPsec RoCEv2 support and netdev events fixes in RDMA

PATCH In review Saeed Mahameed 5 participants 22 messages Jan 5, 2023 – Jan 7, 2023

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover mlx5 IPsec RoCEv2 support and netdev events fixes in RDMA

From: Saeed Mahameed <[email protected]>

This series includes mlx5 modifications for both net-next and rdma-next trees.

In case of no objections, this series will be applied to net-mlx5 branch first then sent in PR to both rdma and net trees.

1) From Jiri: fixe a deadlock in mlx5_ib's netdev notifier unregister.
2) From Mark and Patrisious: add IPsec RoCEv2 support.

Thanks,
Saeed.

Jiri Pirko (3):
net/mlx5e: Fix trap event handling
net/mlx5e: Propagate an internal event in case uplink netdev changes RDMA/mlx5: Track netdev to avoid deadlock during netdev notifier
unregister

Mark Zhang (4):
net/mlx5: Implement new destination type TABLE_TYPE net/mlx5: Add IPSec priorities in RDMA namespaces net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

Patrisious Haddad (1):
net/mlx5: Introduce new destination type TABLE_TYPE

 drivers/infiniband/hw/mlx5/main.c             |  78 ++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   3 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |   9 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.h |   5 +
 .../mellanox/mlx5/core/diag/fs_tracepoint.c   |   4 +
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec.h       |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec_fs.c    |  59 ++-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  15 +-
 .../net/ethernet/mellanox/mlx5/core/events.c  |   2 +
 .../net/ethernet/mellanox/mlx5/core/fs_cmd.c  |   6 +
 .../net/ethernet/mellanox/mlx5/core/fs_core.c |  44 ++-
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.c    | 372 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.h    |  20 +
 .../ethernet/mellanox/mlx5/core/lib/mlx5.h    |   5 -
 .../net/ethernet/mellanox/mlx5/core/main.c    |  20 +
 include/linux/mlx5/device.h                   |   1 +
 include/linux/mlx5/driver.h                   |   5 +
 include/linux/mlx5/fs.h                       |   3 +
 include/linux/mlx5/mlx5_ifc.h                 |  12 +-
 21 files changed, 614 insertions(+), 53 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.h
Signature · 1 line
2.38.1
#1 lore raw

PATCH 1/8 net/mlx5e: Fix trap event handling

From: Jiri Pirko <[email protected]>

Current code does not return correct return value from event handler. Fix it by returning NOTIFY_* and propagate err over newly introduce ctx structure.

---
 drivers/net/ethernet/mellanox/mlx5/core/devlink.c |  9 +++++++--
 drivers/net/ethernet/mellanox/mlx5/core/devlink.h |  5 +++++
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 12 ++++++++----
 3 files changed, 20 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +7 -2
index ddb197970c22..6af1e219a9e8 100644
@@ -263,6 +263,7 @@ static int mlx5_devlink_trap_action_set(struct devlink *devlink,
 					struct netlink_ext_ack *extack)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	struct mlx5_devlink_trap_event_ctx trap_event_ctx;
 	enum devlink_trap_action action_orig;
 	struct mlx5_devlink_trap *dl_trap;
 	int err = 0;
@@ -289,10 +290,14 @@ static int mlx5_devlink_trap_action_set(struct devlink *devlink,
 
 	action_orig = dl_trap->trap.action;
 	dl_trap->trap.action = action;
+	trap_event_ctx.trap = &dl_trap->trap;
+	trap_event_ctx.err = 0;
 	err = mlx5_blocking_notifier_call_chain(dev, MLX5_DRIVER_EVENT_TYPE_TRAP,
-						&dl_trap->trap);
-	if (err)
+						&trap_event_ctx);
+	if (err == NOTIFY_BAD) {
 		dl_trap->trap.action = action_orig;
+		err = trap_event_ctx.err;
+	}
 out:
 	return err;
 }
drivers/net/ethernet/mellanox/mlx5/core/devlink.h +5 -0
index fd033df24856..b84cb70eb3ae 100644
@@ -24,6 +24,11 @@ struct mlx5_devlink_trap {
 	struct list_head list;
 };
 
+struct mlx5_devlink_trap_event_ctx {
+	struct mlx5_trap_ctx *trap;
+	int err;
+};
+
 struct mlx5_core_dev;
 void mlx5_devlink_trap_report(struct mlx5_core_dev *dev, int trap_id, struct sk_buff *skb,
 			      struct devlink_port *dl_port);
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +8 -4
index 8d36e2de53a9..a07f13dc4495 100644
@@ -179,17 +179,21 @@ static void mlx5e_disable_async_events(struct mlx5e_priv *priv)
 static int blocking_event(struct notifier_block *nb, unsigned long event, void *data)
 {
 	struct mlx5e_priv *priv = container_of(nb, struct mlx5e_priv, blocking_events_nb);
+	struct mlx5_devlink_trap_event_ctx *trap_event_ctx = data;
 	int err;
 
 	switch (event) {
 	case MLX5_DRIVER_EVENT_TYPE_TRAP:
-		err = mlx5e_handle_trap_event(priv, data);
+		err = mlx5e_handle_trap_event(priv, trap_event_ctx->trap);
+		if (err) {
+			trap_event_ctx->err = err;
+			return NOTIFY_BAD;
+		}
 		break;
 	default:
-		netdev_warn(priv->netdev, "Sync event: Unknown event %ld\n", event);
-		err = -EINVAL;
+		return NOTIFY_DONE;
 	}
-	return err;
+	return NOTIFY_OK;
 }
 
 static void mlx5e_enable_blocking_events(struct mlx5e_priv *priv)
Signature · 1 line
2.38.1
maintainer Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH mlx5-next 1/8] net/mlx5e: Fix trap event handling

Quoted 6 lines · Saeed Mahameed

From: Jiri Pirko <[email protected]>

Current code does not return correct return value from event handler. Fix it by returning NOTIFY_* and propagate err over newly introduce ctx structure.

Quoted 8 lines
---
 drivers/net/ethernet/mellanox/mlx5/core/devlink.c |  9 +++++++--
 drivers/net/ethernet/mellanox/mlx5/core/devlink.h |  5 +++++
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 12 ++++++++----
 3 files changed, 20 insertions(+), 6 deletions(-)

Thanks,

#3 lore raw

PATCH 2/8 net/mlx5e: Propagate an internal event in case uplink netdev changes

In reply to Saeed Mahameed

From: Jiri Pirko <[email protected]>

Whenever uplink netdev is set/cleared, propagate newly introduced event to inform notifier blocks netdev was added/removed.

Move the set() helper to core.c from header, introduce clear() and netdev_added_event_replay() helpers. The last one is going to be called from rdma driver, so export it.

---
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  3 ++-
 .../ethernet/mellanox/mlx5/core/lib/mlx5.h    |  5 -----
 .../net/ethernet/mellanox/mlx5/core/main.c    | 20 +++++++++++++++++++
 include/linux/mlx5/device.h                   |  1 +
 include/linux/mlx5/driver.h                   |  5 +++++
 5 files changed, 28 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -1
index a07f13dc4495..5b3b07e195d0 100644
@@ -5961,7 +5961,7 @@ static int mlx5e_probe(struct auxiliary_device *adev,
 	}
 
 	mlx5e_dcbnl_init_app(priv);
-	mlx5_uplink_netdev_set(mdev, netdev);
+	mlx5_core_uplink_netdev_set(mdev, netdev);
 	mlx5e_params_print_info(mdev, &priv->channels.params);
 	return 0;
 
@@ -5981,6 +5981,7 @@ static void mlx5e_remove(struct auxiliary_device *adev)
 	struct mlx5e_priv *priv = auxiliary_get_drvdata(adev);
 	pm_message_t state = {};
 
+	mlx5_core_uplink_netdev_set(priv->mdev, NULL);
 	mlx5e_dcbnl_delete_app(priv);
 	unregister_netdev(priv->netdev);
 	mlx5e_suspend(adev, state);
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +0 -5
index 032adb21ad4b..bfd3a1121ed8 100644
@@ -96,11 +96,6 @@ static inline struct net *mlx5_core_net(struct mlx5_core_dev *dev)
 	return devlink_net(priv_to_devlink(dev));
 }
 
-static inline void mlx5_uplink_netdev_set(struct mlx5_core_dev *mdev, struct net_device *netdev)
-{
-	mdev->mlx5e_res.uplink_netdev = netdev;
-}
-
 static inline struct net_device *mlx5_uplink_netdev_get(struct mlx5_core_dev *mdev)
 {
 	return mdev->mlx5e_res.uplink_netdev;
drivers/net/ethernet/mellanox/mlx5/core/main.c +20 -0
index 7f5db13e3550..f64fd762ea83 100644
@@ -336,6 +336,24 @@ static u16 to_fw_pkey_sz(struct mlx5_core_dev *dev, u32 size)
 	}
 }
 
+void mlx5_core_uplink_netdev_set(struct mlx5_core_dev *dev, struct net_device *netdev)
+{
+	mutex_lock(&dev->mlx5e_res.uplink_netdev_lock);
+	dev->mlx5e_res.uplink_netdev = netdev;
+	mlx5_blocking_notifier_call_chain(dev, MLX5_DRIVER_EVENT_UPLINK_NETDEV,
+					  netdev);
+	mutex_unlock(&dev->mlx5e_res.uplink_netdev_lock);
+}
+
+void mlx5_core_uplink_netdev_event_replay(struct mlx5_core_dev *dev)
+{
+	mutex_lock(&dev->mlx5e_res.uplink_netdev_lock);
+	mlx5_blocking_notifier_call_chain(dev, MLX5_DRIVER_EVENT_UPLINK_NETDEV,
+					  dev->mlx5e_res.uplink_netdev);
+	mutex_unlock(&dev->mlx5e_res.uplink_netdev_lock);
+}
+EXPORT_SYMBOL(mlx5_core_uplink_netdev_event_replay);
+
 static int mlx5_core_get_caps_mode(struct mlx5_core_dev *dev,
 				   enum mlx5_cap_type cap_type,
 				   enum mlx5_cap_mode cap_mode)
@@ -1606,6 +1624,7 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx)
 	lockdep_register_key(&dev->lock_key);
 	mutex_init(&dev->intf_state_mutex);
 	lockdep_set_class(&dev->intf_state_mutex, &dev->lock_key);
+	mutex_init(&dev->mlx5e_res.uplink_netdev_lock);
 
 	mutex_init(&priv->bfregs.reg_head.lock);
 	mutex_init(&priv->bfregs.wc_head.lock);
@@ -1694,6 +1713,7 @@ void mlx5_mdev_uninit(struct mlx5_core_dev *dev)
 	mutex_destroy(&priv->alloc_mutex);
 	mutex_destroy(&priv->bfregs.wc_head.lock);
 	mutex_destroy(&priv->bfregs.reg_head.lock);
+	mutex_destroy(&dev->mlx5e_res.uplink_netdev_lock);
 	mutex_destroy(&dev->intf_state_mutex);
 	lockdep_unregister_key(&dev->lock_key);
 }
include/linux/mlx5/device.h +1 -0
index 5fe5d198b57a..3f8ebb679c04 100644
@@ -362,6 +362,7 @@ enum mlx5_event {
 
 enum mlx5_driver_event {
 	MLX5_DRIVER_EVENT_TYPE_TRAP = 0,
+	MLX5_DRIVER_EVENT_UPLINK_NETDEV,
 };
 
 enum {
include/linux/mlx5/driver.h +5 -0
index d476255c9a3f..cc48aa308269 100644
@@ -49,6 +49,7 @@
 #include <linux/notifier.h>
 #include <linux/refcount.h>
 #include <linux/auxiliary_bus.h>
+#include <linux/mutex.h>
 
 #include <linux/mlx5/device.h>
 #include <linux/mlx5/doorbell.h>
@@ -674,6 +675,7 @@ struct mlx5e_resources {
 	} hw_objs;
 	struct devlink_port dl_port;
 	struct net_device *uplink_netdev;
+	struct mutex uplink_netdev_lock;
 };
 
 enum mlx5_sw_icm_type {
@@ -1011,6 +1013,9 @@ int mlx5_cmd_exec_polling(struct mlx5_core_dev *dev, void *in, int in_size,
 			  void *out, int out_size);
 bool mlx5_cmd_is_down(struct mlx5_core_dev *dev);
 
+void mlx5_core_uplink_netdev_set(struct mlx5_core_dev *mdev, struct net_device *netdev);
+void mlx5_core_uplink_netdev_event_replay(struct mlx5_core_dev *mdev);
+
 int mlx5_core_get_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type);
 void mlx5_health_cleanup(struct mlx5_core_dev *dev);
 int mlx5_health_init(struct mlx5_core_dev *dev);
Signature · 1 line
2.38.1
maintainer Reviewed-by on v1 2/8
#4 lore raw

Re: [PATCH mlx5-next 2/8] net/mlx5e: Propagate an internal event in case uplink netdev changes

Quoted 11 lines · Saeed Mahameed

From: Jiri Pirko <[email protected]>

Whenever uplink netdev is set/cleared, propagate newly introduced event to inform notifier blocks netdev was added/removed.

Move the set() helper to core.c from header, introduce clear() and netdev_added_event_replay() helpers. The last one is going to be called from rdma driver, so export it.

Quoted 8 lines
---
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  3 ++-
 .../ethernet/mellanox/mlx5/core/lib/mlx5.h    |  5 -----
 .../net/ethernet/mellanox/mlx5/core/main.c    | 20 +++++++++++++++++++
 include/linux/mlx5/device.h                   |  1 +
 include/linux/mlx5/driver.h                   |  5 +++++
 5 files changed, 28 insertions(+), 6 deletions(-)

Thanks,

#5 lore raw

PATCH 3/8 RDMA/mlx5: Track netdev to avoid deadlock during netdev notifier unregister

In reply to Saeed Mahameed

From: Jiri Pirko <[email protected]>

When removing a network namespace with mlx5 devlink instance being in it, following callchain is performed:

cleanup_net (takes down_read(&pernet_ops_rwsem) devlink_pernet_pre_exit()
devlink_reload()
mlx5_devlink_reload_down() mlx5_unload_one_devl_locked()
mlx5_detach_device()
del_adev()
mlx5r_remove()
__mlx5_ib_remove()
mlx5_ib_roce_cleanup()
mlx5_remove_netdev_notifier() unregister_netdevice_notifier (takes down_write(&pernet_ops_rwsem)

This deadlocks.

Resolve this by converting to register_netdevice_notifier_dev_net() which does not take pernet_ops_rwsem and moves the notifier block around according to netdev it takes as arg.

Use previously introduced netdev added/removed events to track uplink netdev to be used for register_netdevice_notifier_dev_net() purposes.

---
 drivers/infiniband/hw/mlx5/main.c             | 78 +++++++++++++------
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  3 +
 .../net/ethernet/mellanox/mlx5/core/events.c  |  2 +
 3 files changed, 59 insertions(+), 24 deletions(-)
drivers/infiniband/hw/mlx5/main.c +54 -24
index c669ef6e47e7..dc32e4518a28 100644
@@ -3012,26 +3012,63 @@ static void mlx5_eth_lag_cleanup(struct mlx5_ib_dev *dev)
 	}
 }
 
-static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u32 port_num)
+static void mlx5_netdev_notifier_register(struct mlx5_roce *roce,
+					  struct net_device *netdev)
 {
 	int err;
 
-	dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event;
-	err = register_netdevice_notifier(&dev->port[port_num].roce.nb);
-	if (err) {
-		dev->port[port_num].roce.nb.notifier_call = NULL;
-		return err;
-	}
+	if (roce->tracking_netdev)
+		return;
+	roce->tracking_netdev = netdev;
+	roce->nb.notifier_call = mlx5_netdev_event;
+	err = register_netdevice_notifier_dev_net(netdev, &roce->nb, &roce->nn);
+	WARN_ON(err);
+}
 
-	return 0;
+static void mlx5_netdev_notifier_unregister(struct mlx5_roce *roce)
+{
+	if (!roce->tracking_netdev)
+		return;
+	unregister_netdevice_notifier_dev_net(roce->tracking_netdev, &roce->nb,
+					      &roce->nn);
+	roce->tracking_netdev = NULL;
 }
 
-static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u32 port_num)
+static int mlx5e_mdev_notifier_event(struct notifier_block *nb,
+				     unsigned long event, void *data)
 {
-	if (dev->port[port_num].roce.nb.notifier_call) {
-		unregister_netdevice_notifier(&dev->port[port_num].roce.nb);
-		dev->port[port_num].roce.nb.notifier_call = NULL;
+	struct mlx5_roce *roce = container_of(nb, struct mlx5_roce, mdev_nb);
+	struct net_device *netdev = data;
+
+	switch (event) {
+	case MLX5_DRIVER_EVENT_UPLINK_NETDEV:
+		if (netdev)
+			mlx5_netdev_notifier_register(roce, netdev);
+		else
+			mlx5_netdev_notifier_unregister(roce);
+		break;
+	default:
+		return NOTIFY_DONE;
 	}
+
+	return NOTIFY_OK;
+}
+
+static void mlx5_mdev_netdev_track(struct mlx5_ib_dev *dev, u32 port_num)
+{
+	struct mlx5_roce *roce = &dev->port[port_num].roce;
+
+	roce->mdev_nb.notifier_call = mlx5e_mdev_notifier_event;
+	mlx5_blocking_notifier_register(dev->mdev, &roce->mdev_nb);
+	mlx5_core_uplink_netdev_event_replay(dev->mdev);
+}
+
+static void mlx5_mdev_netdev_untrack(struct mlx5_ib_dev *dev, u32 port_num)
+{
+	struct mlx5_roce *roce = &dev->port[port_num].roce;
+
+	mlx5_blocking_notifier_unregister(dev->mdev, &roce->mdev_nb);
+	mlx5_netdev_notifier_unregister(roce);
 }
 
 static int mlx5_enable_eth(struct mlx5_ib_dev *dev)
@@ -3138,7 +3175,7 @@ static void mlx5_ib_unbind_slave_port(struct mlx5_ib_dev *ibdev,
 	if (mpi->mdev_events.notifier_call)
 		mlx5_notifier_unregister(mpi->mdev, &mpi->mdev_events);
 	mpi->mdev_events.notifier_call = NULL;
-	mlx5_remove_netdev_notifier(ibdev, port_num);
+	mlx5_mdev_netdev_untrack(ibdev, port_num);
 	spin_lock(&port->mp.mpi_lock);
 
 	comps = mpi->mdev_refcnt;
@@ -3196,12 +3233,7 @@ static bool mlx5_ib_bind_slave_port(struct mlx5_ib_dev *ibdev,
 	if (err)
 		goto unbind;
 
-	err = mlx5_add_netdev_notifier(ibdev, port_num);
-	if (err) {
-		mlx5_ib_err(ibdev, "failed adding netdev notifier for port %u\n",
-			    port_num + 1);
-		goto unbind;
-	}
+	mlx5_mdev_netdev_track(ibdev, port_num);
 
 	mpi->mdev_events.notifier_call = mlx5_ib_event_slave_port;
 	mlx5_notifier_register(mpi->mdev, &mpi->mdev_events);
@@ -3909,9 +3941,7 @@ static int mlx5_ib_roce_init(struct mlx5_ib_dev *dev)
 		port_num = mlx5_core_native_port_num(dev->mdev) - 1;
 
 		/* Register only for native ports */
-		err = mlx5_add_netdev_notifier(dev, port_num);
-		if (err)
-			return err;
+		mlx5_mdev_netdev_track(dev, port_num);
 
 		err = mlx5_enable_eth(dev);
 		if (err)
@@ -3920,7 +3950,7 @@ static int mlx5_ib_roce_init(struct mlx5_ib_dev *dev)
 
 	return 0;
 cleanup:
-	mlx5_remove_netdev_notifier(dev, port_num);
+	mlx5_mdev_netdev_untrack(dev, port_num);
 	return err;
 }
 
@@ -3938,7 +3968,7 @@ static void mlx5_ib_roce_cleanup(struct mlx5_ib_dev *dev)
 		mlx5_disable_eth(dev);
 
 		port_num = mlx5_core_native_port_num(dev->mdev) - 1;
-		mlx5_remove_netdev_notifier(dev, port_num);
+		mlx5_mdev_netdev_untrack(dev, port_num);
 	}
 }
 
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -0
index 8b91babdd4c0..7394e7f36ba7 100644
@@ -832,6 +832,9 @@ struct mlx5_roce {
 	rwlock_t		netdev_lock;
 	struct net_device	*netdev;
 	struct notifier_block	nb;
+	struct netdev_net_notifier nn;
+	struct notifier_block	mdev_nb;
+	struct net_device	*tracking_netdev;
 	atomic_t		tx_port_affinity;
 	enum ib_port_state last_port_state;
 	struct mlx5_ib_dev	*dev;
drivers/net/ethernet/mellanox/mlx5/core/events.c +2 -0
index 9459e56ee90a..718cf09c28ce 100644
@@ -424,6 +424,7 @@ int mlx5_blocking_notifier_register(struct mlx5_core_dev *dev, struct notifier_b
 
 	return blocking_notifier_chain_register(&events->sw_nh, nb);
 }
+EXPORT_SYMBOL(mlx5_blocking_notifier_register);
 
 int mlx5_blocking_notifier_unregister(struct mlx5_core_dev *dev, struct notifier_block *nb)
 {
@@ -431,6 +432,7 @@ int mlx5_blocking_notifier_unregister(struct mlx5_core_dev *dev, struct notifier
 
 	return blocking_notifier_chain_unregister(&events->sw_nh, nb);
 }
+EXPORT_SYMBOL(mlx5_blocking_notifier_unregister);
 
 int mlx5_blocking_notifier_call_chain(struct mlx5_core_dev *dev, unsigned int event,
 				      void *data)
Signature · 1 line
2.38.1
#6 lore raw

PATCH 4/8 net/mlx5: Introduce new destination type TABLE_TYPE

In reply to Saeed Mahameed

From: Patrisious Haddad <[email protected]>

This new destination type supports flow transition between different table types, e.g. from NIC_RX to RDMA_RX or from RDMA_TX to NIC_TX.

---
 include/linux/mlx5/mlx5_ifc.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
include/linux/mlx5/mlx5_ifc.h +10 -2
index f3d1c62c98dd..77f0d0f25bf0 100644
@@ -315,6 +315,11 @@ enum {
 	MLX5_CMD_OP_GENERAL_END = 0xd00,
 };
 
+enum {
+	MLX5_FT_NIC_RX_2_NIC_RX_RDMA = BIT(0),
+	MLX5_FT_NIC_TX_RDMA_2_NIC_TX = BIT(1),
+};
+
 struct mlx5_ifc_flow_table_fields_supported_bits {
 	u8         outer_dmac[0x1];
 	u8         outer_smac[0x1];
@@ -1898,7 +1903,8 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {
 
 	u8	   reserved_at_e0[0xc0];
 
-	u8	   reserved_at_1a0[0xb];
+	u8	   flow_table_type_2_type[0x8];
+	u8	   reserved_at_1a8[0x3];
 	u8	   log_min_mkey_entity_size[0x5];
 	u8	   reserved_at_1b0[0x10];
 
@@ -1922,6 +1928,7 @@ enum mlx5_ifc_flow_destination_type {
 	MLX5_IFC_FLOW_DESTINATION_TYPE_TIR          = 0x2,
 	MLX5_IFC_FLOW_DESTINATION_TYPE_FLOW_SAMPLER = 0x6,
 	MLX5_IFC_FLOW_DESTINATION_TYPE_UPLINK       = 0x8,
+	MLX5_IFC_FLOW_DESTINATION_TYPE_TABLE_TYPE   = 0xA,
 };
 
 enum mlx5_flow_table_miss_action {
@@ -1936,7 +1943,8 @@ struct mlx5_ifc_dest_format_struct_bits {
 
 	u8         destination_eswitch_owner_vhca_id_valid[0x1];
 	u8         packet_reformat[0x1];
-	u8         reserved_at_22[0xe];
+	u8         reserved_at_22[0x6];
+	u8         destination_table_type[0x8];
 	u8         destination_eswitch_owner_vhca_id[0x10];
 };
 
Signature · 1 line
2.38.1
#7 lore raw

PATCH 5/8 net/mlx5: Implement new destination type TABLE_TYPE

In reply to Saeed Mahameed

From: Mark Zhang <[email protected]>

Implement new destination type to support flow transition between
different table types.
e.g. from NIC_RX to RDMA_RX or from RDMA_TX to NIC_TX.
The new destination is described in the tracepoint as follows:
"mlx5_fs_add_rule: rule=00000000d53cd0ed fte=0000000048a8a6ed index=0 sw_action=<> [dst] flow_table_type=7 id:262152"

---
 .../net/ethernet/mellanox/mlx5/core/diag/fs_tracepoint.c    | 4 ++++
 drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c            | 6 ++++++
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.c           | 3 ++-
 include/linux/mlx5/fs.h                                     | 1 +
 4 files changed, 13 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/diag/fs_tracepoint.c +4 -0
index 2732128e7a6e..6d73127b7217 100644
@@ -275,6 +275,10 @@ const char *parse_fs_dst(struct trace_seq *p,
 				 fs_dest_range_field_to_str(dst->range.field),
 				 dst->range.min, dst->range.max);
 		break;
+	case MLX5_FLOW_DESTINATION_TYPE_TABLE_TYPE:
+		trace_seq_printf(p, "flow_table_type=%u id:%u\n", dst->ft->type,
+				 dst->ft->id);
+		break;
 	case MLX5_FLOW_DESTINATION_TYPE_NONE:
 		trace_seq_printf(p, "none\n");
 		break;
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +6 -0
index 32d4c967469c..a3a9cc6f15ca 100644
@@ -653,6 +653,12 @@ static int mlx5_cmd_set_fte(struct mlx5_core_dev *dev,
 				id = dst->dest_attr.sampler_id;
 				ifc_type = MLX5_IFC_FLOW_DESTINATION_TYPE_FLOW_SAMPLER;
 				break;
+			case MLX5_FLOW_DESTINATION_TYPE_TABLE_TYPE:
+				MLX5_SET(dest_format_struct, in_dests,
+					 destination_table_type, dst->dest_attr.ft->type);
+				id = dst->dest_attr.ft->id;
+				ifc_type = MLX5_IFC_FLOW_DESTINATION_TYPE_TABLE_TYPE;
+				break;
 			default:
 				id = dst->dest_attr.tir_num;
 				ifc_type = MLX5_IFC_FLOW_DESTINATION_TYPE_TIR;
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +2 -1
index 5a85d8c1e797..2333f835fb70 100644
@@ -449,7 +449,8 @@ static bool is_fwd_dest_type(enum mlx5_flow_destination_type type)
 		type == MLX5_FLOW_DESTINATION_TYPE_VPORT ||
 		type == MLX5_FLOW_DESTINATION_TYPE_FLOW_SAMPLER ||
 		type == MLX5_FLOW_DESTINATION_TYPE_TIR ||
-		type == MLX5_FLOW_DESTINATION_TYPE_RANGE;
+		type == MLX5_FLOW_DESTINATION_TYPE_RANGE ||
+		type == MLX5_FLOW_DESTINATION_TYPE_TABLE_TYPE;
 }
 
 static bool check_valid_spec(const struct mlx5_flow_spec *spec)
include/linux/mlx5/fs.h +1 -0
index ba6958b49a8e..1d2a0638ae74 100644
@@ -51,6 +51,7 @@ enum mlx5_flow_destination_type {
 	MLX5_FLOW_DESTINATION_TYPE_COUNTER,
 	MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE_NUM,
 	MLX5_FLOW_DESTINATION_TYPE_RANGE,
+	MLX5_FLOW_DESTINATION_TYPE_TABLE_TYPE,
 };
 
 enum {
Signature · 1 line
2.38.1
#8 lore raw

PATCH 6/8 net/mlx5: Add IPSec priorities in RDMA namespaces

In reply to Saeed Mahameed

From: Mark Zhang <[email protected]>

Add IPSec flow steering priorities in RDMA namespaces. This allows adding tables/rules to forward RoCEv2 traffic to the IPSec crypto tables in NIC_TX domain, and accept RoCEv2 traffic from NIC_RX domain.

---
 .../net/ethernet/mellanox/mlx5/core/fs_core.c | 35 +++++++++++++++++--
 include/linux/mlx5/fs.h                       |  2 ++
 2 files changed, 35 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +33 -2
index 2333f835fb70..eac9fd35129d 100644
@@ -219,19 +219,30 @@ static struct init_tree_node egress_root_fs = {
 };
 
 enum {
+	RDMA_RX_IPSEC_PRIO,
 	RDMA_RX_COUNTERS_PRIO,
 	RDMA_RX_BYPASS_PRIO,
 	RDMA_RX_KERNEL_PRIO,
 };
 
+#define RDMA_RX_IPSEC_NUM_PRIOS 1
+#define RDMA_RX_IPSEC_NUM_LEVELS 2
+#define RDMA_RX_IPSEC_MIN_LEVEL  (RDMA_RX_IPSEC_NUM_LEVELS)
+
 #define RDMA_RX_BYPASS_MIN_LEVEL MLX5_BY_PASS_NUM_REGULAR_PRIOS
 #define RDMA_RX_KERNEL_MIN_LEVEL (RDMA_RX_BYPASS_MIN_LEVEL + 1)
 #define RDMA_RX_COUNTERS_MIN_LEVEL (RDMA_RX_KERNEL_MIN_LEVEL + 2)
 
 static struct init_tree_node rdma_rx_root_fs = {
 	.type = FS_TYPE_NAMESPACE,
-	.ar_size = 3,
+	.ar_size = 4,
 	.children = (struct init_tree_node[]) {
+		[RDMA_RX_IPSEC_PRIO] =
+		ADD_PRIO(0, RDMA_RX_IPSEC_MIN_LEVEL, 0,
+			 FS_CHAINING_CAPS,
+			 ADD_NS(MLX5_FLOW_TABLE_MISS_ACTION_DEF,
+				ADD_MULTIPLE_PRIO(RDMA_RX_IPSEC_NUM_PRIOS,
+						  RDMA_RX_IPSEC_NUM_LEVELS))),
 		[RDMA_RX_COUNTERS_PRIO] =
 		ADD_PRIO(0, RDMA_RX_COUNTERS_MIN_LEVEL, 0,
 			 FS_CHAINING_CAPS,
@@ -254,15 +265,20 @@ static struct init_tree_node rdma_rx_root_fs = {
 
 enum {
 	RDMA_TX_COUNTERS_PRIO,
+	RDMA_TX_IPSEC_PRIO,
 	RDMA_TX_BYPASS_PRIO,
 };
 
 #define RDMA_TX_BYPASS_MIN_LEVEL MLX5_BY_PASS_NUM_PRIOS
 #define RDMA_TX_COUNTERS_MIN_LEVEL (RDMA_TX_BYPASS_MIN_LEVEL + 1)
 
+#define RDMA_TX_IPSEC_NUM_PRIOS 1
+#define RDMA_TX_IPSEC_PRIO_NUM_LEVELS 1
+#define RDMA_TX_IPSEC_MIN_LEVEL  (RDMA_TX_COUNTERS_MIN_LEVEL + RDMA_TX_IPSEC_NUM_PRIOS)
+
 static struct init_tree_node rdma_tx_root_fs = {
 	.type = FS_TYPE_NAMESPACE,
-	.ar_size = 2,
+	.ar_size = 3,
 	.children = (struct init_tree_node[]) {
 		[RDMA_TX_COUNTERS_PRIO] =
 		ADD_PRIO(0, RDMA_TX_COUNTERS_MIN_LEVEL, 0,
@@ -270,6 +286,13 @@ static struct init_tree_node rdma_tx_root_fs = {
 			 ADD_NS(MLX5_FLOW_TABLE_MISS_ACTION_DEF,
 				ADD_MULTIPLE_PRIO(MLX5_RDMA_TX_NUM_COUNTERS_PRIOS,
 						  RDMA_TX_COUNTERS_PRIO_NUM_LEVELS))),
+		[RDMA_TX_IPSEC_PRIO] =
+		ADD_PRIO(0, RDMA_TX_IPSEC_MIN_LEVEL, 0,
+			 FS_CHAINING_CAPS,
+			 ADD_NS(MLX5_FLOW_TABLE_MISS_ACTION_DEF,
+				ADD_MULTIPLE_PRIO(RDMA_TX_IPSEC_NUM_PRIOS,
+						  RDMA_TX_IPSEC_PRIO_NUM_LEVELS))),
+
 		[RDMA_TX_BYPASS_PRIO] =
 		ADD_PRIO(0, RDMA_TX_BYPASS_MIN_LEVEL, 0,
 			 FS_CHAINING_CAPS_RDMA_TX,
@@ -2368,6 +2391,14 @@ struct mlx5_flow_namespace *mlx5_get_flow_namespace(struct mlx5_core_dev *dev,
 		root_ns = steering->rdma_tx_root_ns;
 		prio = RDMA_TX_COUNTERS_PRIO;
 		break;
+	case MLX5_FLOW_NAMESPACE_RDMA_RX_IPSEC:
+		root_ns = steering->rdma_rx_root_ns;
+		prio = RDMA_RX_IPSEC_PRIO;
+		break;
+	case MLX5_FLOW_NAMESPACE_RDMA_TX_IPSEC:
+		root_ns = steering->rdma_tx_root_ns;
+		prio = RDMA_TX_IPSEC_PRIO;
+		break;
 	default: /* Must be NIC RX */
 		WARN_ON(!is_nic_rx_ns(type));
 		root_ns = steering->root_ns;
include/linux/mlx5/fs.h +2 -0
index 1d2a0638ae74..d72a09a3798c 100644
@@ -103,6 +103,8 @@ enum mlx5_flow_namespace_type {
 	MLX5_FLOW_NAMESPACE_PORT_SEL,
 	MLX5_FLOW_NAMESPACE_RDMA_RX_COUNTERS,
 	MLX5_FLOW_NAMESPACE_RDMA_TX_COUNTERS,
+	MLX5_FLOW_NAMESPACE_RDMA_RX_IPSEC,
+	MLX5_FLOW_NAMESPACE_RDMA_TX_IPSEC,
 };
 
 enum {
Signature · 1 line
2.38.1
#9 lore raw

PATCH 7/8 net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic

In reply to Saeed Mahameed

From: Mark Zhang <[email protected]>

Add steering tables/rules to check if the decrypted traffic is RoCEv2, if so then forward it to RDMA_RX domain.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec.h       |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec_fs.c    |  37 ++-
 .../net/ethernet/mellanox/mlx5/core/fs_core.c |   6 +-
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.c    | 253 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.h    |  17 ++
 7 files changed, 309 insertions(+), 8 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
index cd4a1ab0ea78..8415a44fb965 100644
@@ -97,7 +97,7 @@ mlx5_core-$(CONFIG_MLX5_EN_MACSEC) += en_accel/macsec.o en_accel/macsec_fs.o \
 
 mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
 				     en_accel/ipsec_stats.o en_accel/ipsec_fs.o \
-				     en_accel/ipsec_offload.o
+				     en_accel/ipsec_offload.o lib/ipsec_fs_roce.o
 
 mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \
 				   en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +1 -0
index 379c6dc9a3be..d2149f0138d8 100644
@@ -87,6 +87,7 @@ enum {
 	MLX5E_ACCEL_FS_POL_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
 	MLX5E_ACCEL_FS_ESP_FT_LEVEL,
 	MLX5E_ACCEL_FS_ESP_FT_ERR_LEVEL,
+	MLX5E_ACCEL_FS_ESP_FT_ROCE_LEVEL,
 #endif
 };
 
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h +1 -0
index a92e19c4c499..a72261ce7598 100644
@@ -141,6 +141,7 @@ struct mlx5e_ipsec {
 	struct mlx5e_ipsec_tx *tx;
 	struct mlx5e_ipsec_aso *aso;
 	struct notifier_block nb;
+	struct mlx5_ipsec_fs *roce_ipsec;
 };
 
 struct mlx5e_ipsec_esn_state {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c +32 -5
index 9f19f4b59a70..512d4f6e69bb 100644
@@ -6,6 +6,7 @@
 #include "en/fs.h"
 #include "ipsec.h"
 #include "fs_core.h"
+#include "lib/ipsec_fs_roce.h"
 
 #define NUM_IPSEC_FTE BIT(15)
 
@@ -166,7 +167,8 @@ static int ipsec_miss_create(struct mlx5_core_dev *mdev,
 	return err;
 }
 
-static void rx_destroy(struct mlx5_core_dev *mdev, struct mlx5e_ipsec_rx *rx)
+static void rx_destroy(struct mlx5_core_dev *mdev, struct mlx5e_ipsec_rx *rx, u32 family,
+		       struct mlx5_ipsec_fs *roce_ipsec)
 {
 	mlx5_del_flow_rules(rx->pol.rule);
 	mlx5_destroy_flow_group(rx->pol.group);
@@ -179,6 +181,8 @@ static void rx_destroy(struct mlx5_core_dev *mdev, struct mlx5e_ipsec_rx *rx)
 	mlx5_del_flow_rules(rx->status.rule);
 	mlx5_modify_header_dealloc(mdev, rx->status.modify_hdr);
 	mlx5_destroy_flow_table(rx->ft.status);
+
+	mlx5_ipsec_fs_roce_rx_destroy(roce_ipsec, family);
 }
 
 static int rx_create(struct mlx5_core_dev *mdev, struct mlx5e_ipsec *ipsec,
@@ -186,18 +190,35 @@ static int rx_create(struct mlx5_core_dev *mdev, struct mlx5e_ipsec *ipsec,
 {
 	struct mlx5_flow_namespace *ns = mlx5e_fs_get_ns(ipsec->fs, false);
 	struct mlx5_ttc_table *ttc = mlx5e_fs_get_ttc(ipsec->fs, false);
+	struct mlx5_flow_destination default_dest;
 	struct mlx5_flow_destination dest[2];
 	struct mlx5_flow_table *ft;
 	int err;
 
+	default_dest = mlx5_ttc_get_default_dest(ttc, family2tt(family));
+	err = mlx5_ipsec_fs_roce_rx_create(ipsec->roce_ipsec, ns, &default_dest, family,
+					   MLX5E_ACCEL_FS_ESP_FT_ROCE_LEVEL, MLX5E_NIC_PRIO,
+					   ipsec->mdev);
+	if (err)
+		return err;
+
 	ft = ipsec_ft_create(ns, MLX5E_ACCEL_FS_ESP_FT_ERR_LEVEL,
 			     MLX5E_NIC_PRIO, 1);
-	if (IS_ERR(ft))
-		return PTR_ERR(ft);
+	if (IS_ERR(ft)) {
+		err = PTR_ERR(ft);
+		goto err_fs_ft_status;
+	}
 
 	rx->ft.status = ft;
 
-	dest[0] = mlx5_ttc_get_default_dest(ttc, family2tt(family));
+	ft = mlx5_ipsec_fs_roce_ft_get(ipsec->roce_ipsec, family);
+	if (ft) {
+		dest[0].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
+		dest[0].ft = ft;
+	} else {
+		dest[0] = default_dest;
+	}
+
 	dest[1].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
 	dest[1].counter_id = mlx5_fc_id(rx->fc->cnt);
 	err = ipsec_status_rule(mdev, rx, dest);
@@ -245,6 +266,8 @@ static int rx_create(struct mlx5_core_dev *mdev, struct mlx5e_ipsec *ipsec,
 	mlx5_modify_header_dealloc(mdev, rx->status.modify_hdr);
 err_add:
 	mlx5_destroy_flow_table(rx->ft.status);
+err_fs_ft_status:
+	mlx5_ipsec_fs_roce_rx_destroy(ipsec->roce_ipsec, family);
 	return err;
 }
 
@@ -304,7 +327,7 @@ static void rx_ft_put(struct mlx5_core_dev *mdev, struct mlx5e_ipsec *ipsec,
 	mlx5_ttc_fwd_default_dest(ttc, family2tt(family));
 
 	/* remove FT */
-	rx_destroy(mdev, rx);
+	rx_destroy(mdev, rx, family, ipsec->roce_ipsec);
 
 out:
 	mutex_unlock(&rx->ft.mutex);
@@ -1008,6 +1031,8 @@ void mlx5e_accel_ipsec_fs_cleanup(struct mlx5e_ipsec *ipsec)
 	if (!ipsec->tx)
 		return;
 
+	mlx5_ipsec_fs_roce_cleanup(ipsec->roce_ipsec);
+
 	ipsec_fs_destroy_counters(ipsec);
 	mutex_destroy(&ipsec->tx->ft.mutex);
 	WARN_ON(ipsec->tx->ft.refcnt);
@@ -1053,6 +1078,8 @@ int mlx5e_accel_ipsec_fs_init(struct mlx5e_ipsec *ipsec)
 	mutex_init(&ipsec->rx_ipv6->ft.mutex);
 	ipsec->tx->ns = ns;
 
+	ipsec->roce_ipsec = mlx5_ipsec_fs_roce_init(ipsec->mdev);
+
 	return 0;
 
 err_counters:
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +4 -2
index eac9fd35129d..cb28cdb59c17 100644
@@ -111,8 +111,10 @@
 #define ETHTOOL_PRIO_NUM_LEVELS 1
 #define ETHTOOL_NUM_PRIOS 11
 #define ETHTOOL_MIN_LEVEL (KERNEL_MIN_LEVEL + ETHTOOL_NUM_PRIOS)
-/* Promiscuous, Vlan, mac, ttc, inner ttc, {UDP/ANY/aRFS/accel/{esp, esp_err}}, IPsec policy */
-#define KERNEL_NIC_PRIO_NUM_LEVELS 8
+/* Promiscuous, Vlan, mac, ttc, inner ttc, {UDP/ANY/aRFS/accel/{esp, esp_err}}, IPsec policy,
+ * IPsec RoCE policy
+ */
+#define KERNEL_NIC_PRIO_NUM_LEVELS 9
 #define KERNEL_NIC_NUM_PRIOS 1
 /* One more level for tc */
 #define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.c new file +253 -0

253 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.h new file +17 -0
new file mode 100644
index 000000000000..75ca8f3df4f5
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#ifndef __MLX5_LIB_IPSEC_H__
+#define __MLX5_LIB_IPSEC_H__
+
+struct mlx5_ipsec_fs;
+
+struct mlx5_flow_table *mlx5_ipsec_fs_roce_ft_get(struct mlx5_ipsec_fs *ipsec_roce, u32 family);
+void mlx5_ipsec_fs_roce_rx_destroy(struct mlx5_ipsec_fs *ipsec_roce, u32 family);
+int mlx5_ipsec_fs_roce_rx_create(struct mlx5_ipsec_fs *ipsec_roce, struct mlx5_flow_namespace *ns,
+				 struct mlx5_flow_destination *default_dst, u32 family, u32 level,
+				 u32 prio, struct mlx5_core_dev *mdev);
+void mlx5_ipsec_fs_roce_cleanup(struct mlx5_ipsec_fs *ipsec_roce);
+struct mlx5_ipsec_fs *mlx5_ipsec_fs_roce_init(struct mlx5_core_dev *mdev);
+
+#endif /* __MLX5_LIB_IPSEC_H__ */
Signature · 1 line
2.38.1
#10 lore raw

PATCH 8/8 net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

In reply to Saeed Mahameed

From: Mark Zhang <[email protected]>

Add steering table/rule in RDMA_TX domain, to forward all traffic to IPsec crypto table in NIC domain.

---
 .../mellanox/mlx5/core/en_accel/ipsec_fs.c    |  22 +++-
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.c    | 123 +++++++++++++++++-
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.h    |   3 +
 3 files changed, 142 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c +18 -4
index 512d4f6e69bb..4de528687536 100644
@@ -366,6 +366,14 @@ static int tx_create(struct mlx5_core_dev *mdev, struct mlx5e_ipsec_tx *tx)
 	return err;
 }
 
+static void tx_destroy(struct mlx5e_ipsec_tx *tx)
+{
+	mlx5_del_flow_rules(tx->pol.rule);
+	mlx5_destroy_flow_group(tx->pol.group);
+	mlx5_destroy_flow_table(tx->ft.pol);
+	mlx5_destroy_flow_table(tx->ft.sa);
+}
+
 static struct mlx5e_ipsec_tx *tx_ft_get(struct mlx5_core_dev *mdev,
 					struct mlx5e_ipsec *ipsec)
 {
@@ -379,6 +387,13 @@ static struct mlx5e_ipsec_tx *tx_ft_get(struct mlx5_core_dev *mdev,
 	err = tx_create(mdev, tx);
 	if (err)
 		goto out;
+
+	err = mlx5_ipsec_fs_roce_tx_create(ipsec->roce_ipsec, tx->ft.pol, ipsec->mdev);
+	if (err) {
+		tx_destroy(tx);
+		goto out;
+	}
+
 skip:
 	tx->ft.refcnt++;
 out:
@@ -397,10 +412,9 @@ static void tx_ft_put(struct mlx5e_ipsec *ipsec)
 	if (tx->ft.refcnt)
 		goto out;
 
-	mlx5_del_flow_rules(tx->pol.rule);
-	mlx5_destroy_flow_group(tx->pol.group);
-	mlx5_destroy_flow_table(tx->ft.pol);
-	mlx5_destroy_flow_table(tx->ft.sa);
+	mlx5_ipsec_fs_roce_tx_destroy(ipsec->roce_ipsec);
+
+	tx_destroy(tx);
 out:
 	mutex_unlock(&tx->ft.mutex);
 }
drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.c +121 -2

155 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.h +3 -0
index 75ca8f3df4f5..4b69d4e34234 100644
@@ -11,6 +11,9 @@ void mlx5_ipsec_fs_roce_rx_destroy(struct mlx5_ipsec_fs *ipsec_roce, u32 family)
 int mlx5_ipsec_fs_roce_rx_create(struct mlx5_ipsec_fs *ipsec_roce, struct mlx5_flow_namespace *ns,
 				 struct mlx5_flow_destination *default_dst, u32 family, u32 level,
 				 u32 prio, struct mlx5_core_dev *mdev);
+void mlx5_ipsec_fs_roce_tx_destroy(struct mlx5_ipsec_fs *ipsec_roce);
+int mlx5_ipsec_fs_roce_tx_create(struct mlx5_ipsec_fs *ipsec_roce, struct mlx5_flow_table *pol_ft,
+				 struct mlx5_core_dev *mdev);
 void mlx5_ipsec_fs_roce_cleanup(struct mlx5_ipsec_fs *ipsec_roce);
 struct mlx5_ipsec_fs *mlx5_ipsec_fs_roce_init(struct mlx5_core_dev *mdev);
 
Signature · 1 line
2.38.1
maintainer
#11 lore raw
In reply to Saeed Mahameed
Quoted 7 lines · Saeed Mahameed

From: Saeed Mahameed <[email protected]>

This series includes mlx5 modifications for both net-next and rdma-next trees.

In case of no objections, this series will be applied to net-mlx5 branch first then sent in PR to both rdma and net trees.

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

Thanks

Quoted 50 lines

1) From Jiri: fixe a deadlock in mlx5_ib's netdev notifier unregister.
2) From Mark and Patrisious: add IPsec RoCEv2 support.

Thanks,
Saeed.

Jiri Pirko (3):
net/mlx5e: Fix trap event handling
net/mlx5e: Propagate an internal event in case uplink netdev changes RDMA/mlx5: Track netdev to avoid deadlock during netdev notifier
unregister

Mark Zhang (4):
net/mlx5: Implement new destination type TABLE_TYPE net/mlx5: Add IPSec priorities in RDMA namespaces net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

Patrisious Haddad (1):
net/mlx5: Introduce new destination type TABLE_TYPE

 drivers/infiniband/hw/mlx5/main.c             |  78 ++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   3 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |   9 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.h |   5 +
 .../mellanox/mlx5/core/diag/fs_tracepoint.c   |   4 +
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec.h       |   1 +
 .../mellanox/mlx5/core/en_accel/ipsec_fs.c    |  59 ++-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  15 +-
 .../net/ethernet/mellanox/mlx5/core/events.c  |   2 +
 .../net/ethernet/mellanox/mlx5/core/fs_cmd.c  |   6 +
 .../net/ethernet/mellanox/mlx5/core/fs_core.c |  44 ++-
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.c    | 372 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/ipsec_fs_roce.h    |  20 +
 .../ethernet/mellanox/mlx5/core/lib/mlx5.h    |   5 -
 .../net/ethernet/mellanox/mlx5/core/main.c    |  20 +
 include/linux/mlx5/device.h                   |   1 +
 include/linux/mlx5/driver.h                   |   5 +
 include/linux/mlx5/fs.h                       |   3 +
 include/linux/mlx5/mlx5_ifc.h                 |  12 +-
 21 files changed, 614 insertions(+), 53 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/ipsec_fs_roce.h
Signature · 1 line
2.38.1
maintainer
#12 lore raw
Quoted 7 lines · On Thu, 5 Jan 2023 15:05:54 +0200 Leon Romanovsky wrote:
Quoted 5 lines

This series includes mlx5 modifications for both net-next and rdma-next trees.

In case of no objections, this series will be applied to net-mlx5 branch first then sent in PR to both rdma and net trees.

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

FWIW I don't understand what you mean by this comment. PR should be based on a common ancestor, the -rc tags are just a convenient shorthand.

#13 lore raw
Quoted 4 lines · Jakub Kicinski
Quoted 3 lines · On Thu, 5 Jan 2023 15:05:54 +0200 Leon Romanovsky wrote:
Quoted 3 lines

This series includes mlx5 modifications for both net-next and rdma-next trees.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

In case of no objections, this series will be applied to net-mlx5 branch first then sent in PR to both rdma and net trees.

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

FWIW I don't understand what you mean by this comment. PR should be based on a common ancestor, the -rc tags are just a convenient shorthand.

Linus asked for more than once to use sensible ancestor which is -rc.

Thanks

maintainer
#14 lore raw
Quoted 7 lines · On Thu, 5 Jan 2023 22:12:28 +0200 Leon Romanovsky wrote:
Quoted 5 lines
Quoted 1 line

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

FWIW I don't understand what you mean by this comment. PR should be based on a common ancestor, the -rc tags are just a convenient shorthand.

Linus asked for more than once to use sensible ancestor which is -rc.

I mean.. as I said using -rc tags makes sanity checking the PRs easier, so definitely encouraged.

I was asking more about the second part of your sentence, what do you mean by "shouldn't include only this patchset" ?

#15 lore raw
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 5 Jan 2023 22:12:28 +0200 Leon Romanovsky wrote:
Quoted 5 lines
Quoted 1 line

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

FWIW I don't understand what you mean by this comment. PR should be based on a common ancestor, the -rc tags are just a convenient shorthand.

Quoted 7 lines
Quoted 1 line

Linus asked for more than once to use sensible ancestor which is -rc.

I mean.. as I said using -rc tags makes sanity checking the PRs easier, so definitely encouraged.

I was asking more about the second part of your sentence, what do you mean by "shouldn't include only this patchset" ?

When I saw "be applied to net-mlx5 branch", I imagined some branch which is based on net-rc and not on mlx5-next as usual. It looks like Saeed's intention was to say mlx5-next, but he misspelled it.

Thanks

#16 lore raw
Quoted 13 lines · Leon Romanovsky
Quoted 12 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 5 Jan 2023 22:12:28 +0200 Leon Romanovsky wrote:
Quoted 5 lines
Quoted 1 line

PR should be based on Linus's -rcX tag and shouldn't include only this patchset.

FWIW I don't understand what you mean by this comment. PR should be based on a common ancestor, the -rc tags are just a convenient shorthand.

Linus asked for more than once to use sensible ancestor which is -rc.

I mean.. as I said using -rc tags makes sanity checking the PRs easier, so definitely encouraged.

Quoted 6 lines
Quoted 2 lines

I was asking more about the second part of your sentence, what do you mean by "shouldn't include only this patchset" ?

When I saw "be applied to net-mlx5 branch", I imagined some branch which is based on net-rc and not on mlx5-next as usual. It looks like Saeed's intention was to say mlx5-next, but he misspelled it.

I meant mlx5-next.

maintainer
#17 lore raw
In reply to Saeed Mahameed
Quoted 2 lines · On Wed, 4 Jan 2023 20:17:48 -0800 Saeed Mahameed wrote:

net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

How is the TC forwarding coming along?

#18 lore raw
Quoted 5 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 4 Jan 2023 20:17:48 -0800 Saeed Mahameed wrote:

net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

How is the TC forwarding coming along?

Not aware of such effort, can you please elaborate ?

maintainer
#19 lore raw
Quoted 8 lines · On Fri, 6 Jan 2023 11:35:38 -0800 Saeed Mahameed wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 4 Jan 2023 20:17:48 -0800 Saeed Mahameed wrote:

net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

How is the TC forwarding coming along?

Not aware of such effort, can you please elaborate ?

When Leon posted the IPsec patches I correctly guessed that it's for RDMA and expressed my strong preference for RDMA to stop using netdev interfaces for configuration. He made the claim that the full IPsec offload will be used for eswitch offload as well, so I'm asking how is that work going.

maintainer
#20 lore raw
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 6 Jan 2023 11:35:38 -0800 Saeed Mahameed wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 4 Jan 2023 20:17:48 -0800 Saeed Mahameed wrote:

net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

How is the TC forwarding coming along?

Not aware of such effort, can you please elaborate ?

When Leon posted the IPsec patches I correctly guessed that it's for RDMA and expressed my strong preference for RDMA to stop using netdev interfaces for configuration.

No, this wasn't done "for" RDMA. RDMA work loads have to run over it, but not exclusively.

RDMA raw ethernet can already do IPSEC offload, it doesn't need this netdev part for configuration.

IIRC it was for some non-TC forwarding VM configuration, but Leon knows better.

Jason

#21 lore raw
In reply to Jakub Kicinski
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Fri, 6 Jan 2023 11:35:38 -0800 Saeed Mahameed wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 4 Jan 2023 20:17:48 -0800 Saeed Mahameed wrote:

net/mlx5: Configure IPsec steering for ingress RoCEv2 traffic net/mlx5: Configure IPsec steering for egress RoCEv2 traffic

How is the TC forwarding coming along?

Quoted 7 lines
Quoted 1 line

Not aware of such effort, can you please elaborate ?

When Leon posted the IPsec patches I correctly guessed that it's for RDMA and expressed my strong preference for RDMA to stop using netdev interfaces for configuration. He made the claim that the full IPsec offload will be used for eswitch offload as well, so I'm asking how is that work going.

It is planned, as we need this support too for ASAP program, but I simply can't say for sure if I success to make it in this cycle due to my personal constraints.

On my plate, without going into much details:
1. Overcome cx7 HW limitation when dealing with hard/soft lifetime counters.
2. Tunnel mode support.
3. IPsec on FDB. This is TC forwarding support.

Thanks