Skip to contentSkip to conversation

RDMA RX RoCE Steering Support

PATCH In review Leon Romanovsky 2 participants 8 messages Aug 19, 2019 – Aug 22, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover RDMA RX RoCE Steering Support

From: Leon Romanovsky <[email protected]>

Hi,

This series from Mark extends mlx5 with RDMA_RX RoCE flow steering support for DEVX and QP objects.

Thanks

Mark Zhang (3):
net/mlx5: Add per-namespace flow table default miss action support net/mlx5: Create bypass and loopback flow steering namespaces for RDMA
RX
RDMA/mlx5: RDMA_RX flow type support for user applications

 drivers/infiniband/hw/mlx5/flow.c             |  13 +-
 drivers/infiniband/hw/mlx5/main.c             |   7 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   1 +
 .../net/ethernet/mellanox/mlx5/core/fs_cmd.c  |   4 +-
 .../net/ethernet/mellanox/mlx5/core/fs_core.c | 120 ++++++++++++------
 .../net/ethernet/mellanox/mlx5/core/fs_core.h |   3 +-
 .../net/ethernet/mellanox/mlx5/core/rdma.c    |   2 +-
 include/linux/mlx5/fs.h                       |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h     |   1 +
 9 files changed, 107 insertions(+), 45 deletions(-)

--
2.20.1

#1 lore raw

PATCH 1/3 net/mlx5: Add per-namespace flow table default miss action support

From: Mark Zhang <[email protected]>

Currently all the namespaces under the same steering domain share the same default table miss action, however in some situations (e.g., RDMA RX) different actions are required. This patch adds a per-namespace default table miss action instead of using the miss action of the steering domain.

---
 .../net/ethernet/mellanox/mlx5/core/fs_cmd.c  |  4 +-
 .../net/ethernet/mellanox/mlx5/core/fs_core.c | 73 +++++++++++--------
 .../net/ethernet/mellanox/mlx5/core/fs_core.h |  3 +-
 3 files changed, 47 insertions(+), 33 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +2 -2
index b84a225bbe86..1e3381604b3d 100644
@@ -182,7 +182,7 @@ static int mlx5_cmd_create_flow_table(struct mlx5_flow_root_namespace *ns,
 		} else {
 			MLX5_SET(create_flow_table_in, in,
 				 flow_table_context.table_miss_action,
-				 ns->def_miss_action);
+				 ft->def_miss_action);
 		}
 		break;
 
@@ -262,7 +262,7 @@ static int mlx5_cmd_modify_flow_table(struct mlx5_flow_root_namespace *ns,
 		} else {
 			MLX5_SET(modify_flow_table_in, in,
 				 flow_table_context.table_miss_action,
-				 ns->def_miss_action);
+				 ft->def_miss_action);
 		}
 	}
 
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +43 -30

130 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/fs_core.h +2 -1
index c48c382f926f..69f809831959 100644
@@ -145,6 +145,7 @@ struct mlx5_flow_table {
 	struct list_head		fwd_rules;
 	u32				flags;
 	struct rhltable			fgs_hash;
+	enum mlx5_flow_table_miss_action def_miss_action;
 };
 
 struct mlx5_ft_underlay_qp {
@@ -191,6 +192,7 @@ struct fs_prio {
 struct mlx5_flow_namespace {
 	/* parent == NULL => root ns */
 	struct	fs_node			node;
+	enum mlx5_flow_table_miss_action def_miss_action;
 };
 
 struct mlx5_flow_group_mask {
@@ -219,7 +221,6 @@ struct mlx5_flow_root_namespace {
 	struct mutex			chain_lock;
 	struct list_head		underlay_qpns;
 	const struct mlx5_flow_cmds	*cmds;
-	enum mlx5_flow_table_miss_action def_miss_action;
 };
 
 int mlx5_init_fc_stats(struct mlx5_core_dev *dev);
Signature · 1 line
2.20.1
#3 lore raw

PATCH 3/3 RDMA/mlx5: RDMA_RX flow type support for user applications

In reply to Leon Romanovsky

From: Mark Zhang <[email protected]>

Currently user applications can only steer TCP/IP(NIC RX/RX) traffic. This patch adds RDMA_RX as a new flow type to allow the user to insert steering rules to control RDMA traffic.
Two destinations are supported(but not set at the same time): devx flow table object and QP.

---
 drivers/infiniband/hw/mlx5/flow.c         | 13 +++++++++++--
 drivers/infiniband/hw/mlx5/main.c         |  7 +++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h      |  1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h |  1 +
 4 files changed, 20 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/flow.c +11 -2
index b8841355fcd5..571bb8539cdb 100644
@@ -32,6 +32,9 @@ mlx5_ib_ft_type_to_namespace(enum mlx5_ib_uapi_flow_table_type table_type,
 	case MLX5_IB_UAPI_FLOW_TABLE_TYPE_FDB:
 		*namespace = MLX5_FLOW_NAMESPACE_FDB;
 		break;
+	case MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_RX:
+		*namespace = MLX5_FLOW_NAMESPACE_RDMA_RX;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -101,6 +104,11 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_CREATE_FLOW)(
 	if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB && !dest_devx)
 		return -EINVAL;
 
+	/* Allow only DEVX object or QP as dest when inserting to RDMA_RX */
+	if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX) &&
+	    ((!dest_devx && !dest_qp) || (dest_devx && dest_qp)))
+		return -EINVAL;
+
 	if (dest_devx) {
 		devx_obj = uverbs_attr_get_obj(
 			attrs, MLX5_IB_ATTR_CREATE_FLOW_DEST_DEVX);
@@ -112,8 +120,9 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_CREATE_FLOW)(
 		 */
 		if (!mlx5_ib_devx_is_flow_dest(devx_obj, &dest_id, &dest_type))
 			return -EINVAL;
-		/* Allow only flow table as dest when inserting to FDB */
-		if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB &&
+		/* Allow only flow table as dest when inserting to FDB or RDMA_RX */
+		if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB ||
+		     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX) &&
 		    dest_type != MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE)
 			return -EINVAL;
 	} else if (dest_qp) {
drivers/infiniband/hw/mlx5/main.c +7 -0
index 08020affdc17..4e86212e6ea8 100644
@@ -3963,6 +3963,11 @@ _get_flow_table(struct mlx5_ib_dev *dev,
 		    esw_encap)
 			flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
 		priority = FDB_BYPASS_PATH;
+	} else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX) {
+		max_table_size =
+			BIT(MLX5_CAP_FLOWTABLE_RDMA_RX(dev->mdev,
+						       log_max_ft_size));
+		priority = fs_matcher->priority;
 	}
 
 	max_table_size = min_t(int, max_table_size, MLX5_FS_MAX_ENTRIES);
@@ -3977,6 +3982,8 @@ _get_flow_table(struct mlx5_ib_dev *dev,
 		prio = &dev->flow_db->egress_prios[priority];
 	else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB)
 		prio = &dev->flow_db->fdb;
+	else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX)
+		prio = &dev->flow_db->rdma_rx[priority];
 
 	if (!prio)
 		return ERR_PTR(-EINVAL);
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
index cb41a7e6255a..6abfbf3a69b7 100644
@@ -200,6 +200,7 @@ struct mlx5_ib_flow_db {
 	struct mlx5_ib_flow_prio	sniffer[MLX5_IB_NUM_SNIFFER_FTS];
 	struct mlx5_ib_flow_prio	egress[MLX5_IB_NUM_EGRESS_FTS];
 	struct mlx5_ib_flow_prio	fdb;
+	struct mlx5_ib_flow_prio	rdma_rx[MLX5_IB_NUM_FLOW_FT];
 	struct mlx5_flow_table		*lag_demux_ft;
 	/* Protect flow steering bypass flow tables
 	 * when add/del flow rules.
include/uapi/rdma/mlx5_user_ioctl_verbs.h +1 -0
index 7e9900b0e746..88b6ca70c2fe 100644
@@ -43,6 +43,7 @@ enum mlx5_ib_uapi_flow_table_type {
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_RX     = 0x0,
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_TX	= 0x1,
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_FDB	= 0x2,
+	MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_RX	= 0x3,
 };
 
 enum mlx5_ib_uapi_flow_action_packet_reformat_type {
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/3 net/mlx5: Create bypass and loopback flow steering namespaces for RDMA RX

In reply to Leon Romanovsky

From: Mark Zhang <[email protected]>

Use different namespaces for bypass and switchdev loopback because they have different priorities and default table miss action requirement:
1. bypass: with multiple priorities support, and MLX5_FLOW_TABLE_MISS_ACTION_DEF as the default table miss action;
2. switchdev loopback: with single priority support, and MLX5_FLOW_TABLE_MISS_ACTION_SWITCH_DOMAIN as the default table miss action.

---
 .../net/ethernet/mellanox/mlx5/core/fs_core.c | 49 +++++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/rdma.c    |  2 +-
 include/linux/mlx5/fs.h                       |  1 +
 3 files changed, 41 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +39 -10
index fb3cfdfbafbe..7bdec442f0ac 100644
@@ -182,6 +182,26 @@ static struct init_tree_node egress_root_fs = {
 	}
 };
 
+#define RDMA_RX_BYPASS_PRIO 0
+#define RDMA_RX_KERNEL_PRIO 1
+static struct init_tree_node rdma_rx_root_fs = {
+	.type = FS_TYPE_NAMESPACE,
+	.ar_size = 2,
+	.children = (struct init_tree_node[]) {
+		[RDMA_RX_BYPASS_PRIO] =
+		ADD_PRIO(0, MLX5_BY_PASS_NUM_REGULAR_PRIOS, 0,
+			 FS_CHAINING_CAPS,
+			 ADD_NS(MLX5_FLOW_TABLE_MISS_ACTION_DEF,
+				ADD_MULTIPLE_PRIO(MLX5_BY_PASS_NUM_REGULAR_PRIOS,
+						  BY_PASS_PRIO_NUM_LEVELS))),
+		[RDMA_RX_KERNEL_PRIO] =
+		ADD_PRIO(0, MLX5_BY_PASS_NUM_REGULAR_PRIOS + 1, 0,
+			 FS_CHAINING_CAPS,
+			 ADD_NS(MLX5_FLOW_TABLE_MISS_ACTION_SWITCH_DOMAIN,
+				ADD_MULTIPLE_PRIO(1, 1))),
+	}
+};
+
 enum fs_i_lock_class {
 	FS_LOCK_GRANDPARENT,
 	FS_LOCK_PARENT,
@@ -2067,16 +2087,18 @@ struct mlx5_flow_namespace *mlx5_get_flow_namespace(struct mlx5_core_dev *dev,
 		if (steering->sniffer_tx_root_ns)
 			return &steering->sniffer_tx_root_ns->ns;
 		return NULL;
-	case MLX5_FLOW_NAMESPACE_RDMA_RX:
-		if (steering->rdma_rx_root_ns)
-			return &steering->rdma_rx_root_ns->ns;
-		return NULL;
 	default:
 		break;
 	}
 
 	if (type == MLX5_FLOW_NAMESPACE_EGRESS) {
 		root_ns = steering->egress_root_ns;
+	} else if (type == MLX5_FLOW_NAMESPACE_RDMA_RX) {
+		root_ns = steering->rdma_rx_root_ns;
+		prio = RDMA_RX_BYPASS_PRIO;
+	} else if (type == MLX5_FLOW_NAMESPACE_RDMA_RX_KERNEL) {
+		root_ns = steering->rdma_rx_root_ns;
+		prio = RDMA_RX_KERNEL_PRIO;
 	} else { /* Must be NIC RX */
 		root_ns = steering->root_ns;
 		prio = type;
@@ -2507,18 +2529,25 @@ static int init_sniffer_rx_root_ns(struct mlx5_flow_steering *steering)
 
 static int init_rdma_rx_root_ns(struct mlx5_flow_steering *steering)
 {
-	struct fs_prio *prio;
+	int err;
 
 	steering->rdma_rx_root_ns = create_root_ns(steering, FS_FT_RDMA_RX);
 	if (!steering->rdma_rx_root_ns)
 		return -ENOMEM;
 
-	steering->rdma_rx_root_ns->ns.def_miss_action =
-		MLX5_FLOW_TABLE_MISS_ACTION_SWITCH_DOMAIN;
+	err = init_root_tree(steering, &rdma_rx_root_fs,
+			     &steering->rdma_rx_root_ns->ns.node);
+	if (err)
+		goto out_err;
 
-	/* Create single prio */
-	prio = fs_create_prio(&steering->rdma_rx_root_ns->ns, 0, 1);
-	return PTR_ERR_OR_ZERO(prio);
+	set_prio_attrs(steering->rdma_rx_root_ns);
+
+	return 0;
+
+out_err:
+	cleanup_root_ns(steering->rdma_rx_root_ns);
+	steering->rdma_rx_root_ns = NULL;
+	return err;
 }
 static int init_fdb_root_ns(struct mlx5_flow_steering *steering)
 {
drivers/net/ethernet/mellanox/mlx5/core/rdma.c +1 -1
index 17ce9dd56b13..18af6981e0be 100644
@@ -51,7 +51,7 @@ static int mlx5_rdma_enable_roce_steering(struct mlx5_core_dev *dev)
 		return -ENOMEM;
 	}
 
-	ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_RDMA_RX);
+	ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_RDMA_RX_KERNEL);
 	if (!ns) {
 		mlx5_core_err(dev, "Failed to get RDMA RX namespace");
 		err = -EOPNOTSUPP;
include/linux/mlx5/fs.h +1 -0
index 04a569568eac..5235b09a8ef3 100644
@@ -75,6 +75,7 @@ enum mlx5_flow_namespace_type {
 	MLX5_FLOW_NAMESPACE_SNIFFER_TX,
 	MLX5_FLOW_NAMESPACE_EGRESS,
 	MLX5_FLOW_NAMESPACE_RDMA_RX,
+	MLX5_FLOW_NAMESPACE_RDMA_RX_KERNEL,
 };
 
 enum {
Signature · 1 line
2.20.1
#4 lore raw
In reply to Leon Romanovsky
Quoted 8 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Mark extends mlx5 with RDMA_RX RoCE flow steering support
for DEVX and QP objects.

Quoted 8 lines

Thanks

Mark Zhang (3):
net/mlx5: Add per-namespace flow table default miss action support net/mlx5: Create bypass and loopback flow steering namespaces for
RDMA
RX
RDMA/mlx5: RDMA_RX flow type support for user applications

I have no objection to this series.

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#5 lore raw
Quoted 11 lines · Doug Ledford
Quoted 10 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Mark extends mlx5 with RDMA_RX RoCE flow steering support
for DEVX and QP objects.

Thanks

Quoted 8 lines
Quoted 6 lines

Mark Zhang (3):
net/mlx5: Add per-namespace flow table default miss action support net/mlx5: Create bypass and loopback flow steering namespaces for
RDMA
RX
RDMA/mlx5: RDMA_RX flow type support for user applications

I have no objection to this series.

Thanks, first two patches were applied to mlx5-next

e6806e9a63a7 net/mlx5: Create bypass and loopback flow steering namespaces for RDMA RX f66ad830b114 net/mlx5: Add per-namespace flow table default miss action support

Quoted 5 lines

--
Doug Ledford <[email protected]>

    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#6 lore raw
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Doug Ledford
Quoted 18 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Mark extends mlx5 with RDMA_RX RoCE flow steering support
for DEVX and QP objects.

Thanks

Mark Zhang (3):
net/mlx5: Add per-namespace flow table default miss action
support
net/mlx5: Create bypass and loopback flow steering namespaces
for
RDMA
RX
RDMA/mlx5: RDMA_RX flow type support for user applications

Quoted 8 lines
Quoted 1 line

I have no objection to this series.

Thanks, first two patches were applied to mlx5-next

e6806e9a63a7 net/mlx5: Create bypass and loopback flow steering namespaces for RDMA RX
f66ad830b114 net/mlx5: Add per-namespace flow table default miss action support

mlx5-next merged into for-next, final patch applied, thanks.

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#7 lore raw
Quoted 24 lines · Doug Ledford
Quoted 23 lines · Leon Romanovsky
Quoted 21 lines · Doug Ledford
Quoted 10 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Mark extends mlx5 with RDMA_RX RoCE flow steering support
for DEVX and QP objects.

Thanks

Quoted 8 lines

Mark Zhang (3):
net/mlx5: Add per-namespace flow table default miss action
support
net/mlx5: Create bypass and loopback flow steering namespaces
for
RDMA
RX
RDMA/mlx5: RDMA_RX flow type support for user applications

I have no objection to this series.

Quoted 8 lines
Quoted 6 lines

Thanks, first two patches were applied to mlx5-next

e6806e9a63a7 net/mlx5: Create bypass and loopback flow steering namespaces for RDMA RX
f66ad830b114 net/mlx5: Add per-namespace flow table default miss action support

mlx5-next merged into for-next, final patch applied, thanks.

Thanks

Quoted 5 lines

--
Doug Ledford <[email protected]>

    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD