Skip to contentSkip to conversation

Enable flow steering on IPoIB UD QP

PATCH Superseded Leon Romanovsky 1 participant 7 messages Jun 5, 2017 1 message not in the archive

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Enable flow steering on IPoIB UD QP

Hi Doug,

This is second version of underlay QP patches from Yishai. He updated commit message for the first patch (IB/core) to have better semantics description.

Thanks

---

Quoted 1 line

From Yishai:

This series comes to allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP. On the receive flow, the user space application will use flow steering rules to direct the ingress IPoIB UD QP traffic to its own user space UD QP. On the send flow, the user space application will send from its own UD QP with UD AH, while the packet on the wire will have the source QPN of the underlay IPoIB UD QP, which is the network interface advertised L2 hardware address.

This will be achieved by extending QP creation of IB UD QP to be "Associated" with IPoIB UD QP number. In order to create such a UD QP which can receive and send IPoIB traffic, needs to provide the associated QPN of the IPoIB network interface. The created QP will manage the scattering objects as of RQ and SQ, while the transport attributes will be managed by its underlay QP which is owned by IPoIB ULP.

The QPN should be achieved by using some networking tool as of ifconfig ,ip link show on the associated interface. For example, when IPoIB is used, the QPN is the value of 3 bytes from the hardware address starting after the first byte as defined by rfc4391 [1]

This API enables generic QPN association, current mlx5 driver implements this API for the IPoIB ULP.

[1] https://tools.ietf.org/html/rfc4391#page-11
---

Available in the "topic/underlay_qp" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/underlay_qp_v1

Possible merge conflicts resolution is available in:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=testing/queue-next

Yishai Hadas (6):
IB/core: Enable QP creation which is associated to underlay QP IB/uverbs: Enable QP creation which is associated to underlay QP IB/mlx5: Add support for underlay QP managing IB/mlx5: Add multicast flow steering support for underlay QP net/mlx5: Report enhanced capabilities for IPoIB IB/mlx5: Report RX checksum capabilities for IPoIB

 drivers/infiniband/core/uverbs_cmd.c         | 11 +++--
 drivers/infiniband/hw/mlx5/main.c            | 47 +++++++++++++-----
 drivers/infiniband/hw/mlx5/mlx5_ib.h         |  2 +
 drivers/infiniband/hw/mlx5/qp.c              | 72 ++++++++++++++++++++++------
 drivers/net/ethernet/mellanox/mlx5/core/fw.c |  6 +++
 include/linux/mlx5/device.h                  |  6 ++-
 include/rdma/ib_verbs.h                      |  2 +
 include/uapi/rdma/ib_user_verbs.h            |  2 +-
 8 files changed, 116 insertions(+), 32 deletions(-)

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

PATCH 1/6 IB/core: Enable QP creation which is associated to underlay QP

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

Enable QP creation which is associated to some underlay QP as of IPoIB QP. It comes to allow in downstream patches a user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP.

A successful QP creation will end-up by having an overlay QP (e.g. the created one) which is associated to an already existing underlay one (e.g. IPoIB QP).

The underlay QP is responsible for the transport, the packets on the wire will hold its QPN, its pkey will be used, etc.

The overlay QP can steer incoming packets which were target to the underlay QP into its own RQ. It can post send WQEs on its own SQ, the hardware will use the transport properties of the underlay QP to format the packets on the wire.
Note: The underlay QP should be in an RTR/RTS state in order to enable the overlay QP to work properly.

Each QP should manage its properties and objects via modify, query and destroy. Specifically, overlay QP can't affect the underlay QP properties and vice versa.

---
 include/rdma/ib_verbs.h | 2 ++
 1 file changed, 2 insertions(+)
include/rdma/ib_verbs.h +2 -0
index ba8314ec5768..ba97b33f31c5 100644
@@ -1059,6 +1059,7 @@ enum ib_qp_create_flags {
 	IB_QP_CREATE_USE_GFP_NOIO		= 1 << 7,
 	IB_QP_CREATE_SCATTER_FCS		= 1 << 8,
 	IB_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
+	IB_QP_CREATE_ASSOC_QPN			= 1 << 10,
 	/* reserve bits 26-31 for low level drivers' internal use */
 	IB_QP_CREATE_RESERVED_START		= 1 << 26,
 	IB_QP_CREATE_RESERVED_END		= 1 << 31,
@@ -1086,6 +1087,7 @@ struct ib_qp_init_attr {
 	 */
 	u8			port_num;
 	struct ib_rwq_ind_table *rwq_ind_tbl;
+	u32			associated_qpn;
 };
 
 struct ib_qp_open_attr {

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

PATCH 2/6 IB/uverbs: Enable QP creation which is associated to underlay QP

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

Enable QP creation which is associated to underlay QP. This comes as a pre-patch for downstream patches in this series to enable flow steering from user space application on the underlay IPoIB QP.

---
 drivers/infiniband/core/uverbs_cmd.c | 11 ++++++++---
 include/uapi/rdma/ib_user_verbs.h    |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)
drivers/infiniband/core/uverbs_cmd.c +8 -3
index 70b7fb156414..d819aa01da5e 100644
@@ -1385,8 +1385,9 @@ static int create_qp(struct ib_uverbs_file *file,
 		attr.rwq_ind_tbl = ind_tbl;
 	}
 
-	if ((cmd_sz >= offsetof(typeof(*cmd), reserved1) +
-		       sizeof(cmd->reserved1)) && cmd->reserved1) {
+	if (cmd_sz > sizeof(*cmd) &&
+	    !ib_is_udata_cleared(ucore, sizeof(*cmd),
+				 cmd_sz - sizeof(*cmd))) {
 		ret = -EOPNOTSUPP;
 		goto err_put;
 	}
@@ -1484,11 +1485,15 @@ static int create_qp(struct ib_uverbs_file *file,
 				IB_QP_CREATE_MANAGED_SEND |
 				IB_QP_CREATE_MANAGED_RECV |
 				IB_QP_CREATE_SCATTER_FCS |
-				IB_QP_CREATE_CVLAN_STRIPPING)) {
+				IB_QP_CREATE_CVLAN_STRIPPING |
+				IB_QP_CREATE_ASSOC_QPN)) {
 		ret = -EINVAL;
 		goto err_put;
 	}
 
+	if (attr.create_flags & IB_QP_CREATE_ASSOC_QPN)
+		attr.associated_qpn = cmd->associated_qpn;
+
 	buf = (void *)cmd + sizeof(*cmd);
 	if (cmd_sz > sizeof(*cmd))
 		if (!(buf[0] == 0 && !memcmp(buf, buf + 1,
include/uapi/rdma/ib_user_verbs.h +1 -1
index 270c350bedc6..f882ed62de72 100644
@@ -578,7 +578,7 @@ struct ib_uverbs_ex_create_qp {
 	__u32 comp_mask;
 	__u32 create_flags;
 	__u32 rwq_ind_tbl_handle;
-	__u32  reserved1;
+	__u32  associated_qpn;
 };
 
 struct ib_uverbs_open_qp {

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

PATCH 3/6 IB/mlx5: Add support for underlay QP managing

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

Allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP by creating a UD QP which is associated with an underlay IPoIB UD QP.

UD QP with underlay QPN should basically be similar to RAW QP from point of view of its created resources.

However,
- Its TIS should point to the underlay QPN.
- Modify can be done only on its state as the transport attributes are managed by its underlay QP.

This patch manages below:
- Creating/destroying UD QP with associated QPN.
- Modifying UD QP which has an underlay QP associated.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  2 +
 drivers/infiniband/hw/mlx5/qp.c      | 72 ++++++++++++++++++++++++++++--------
 2 files changed, 59 insertions(+), 15 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
index 38c877bc45e5..f47d25e73ef8 100644
@@ -378,6 +378,7 @@ struct mlx5_ib_qp {
 	struct list_head	cq_recv_list;
 	struct list_head	cq_send_list;
 	u32			rate_limit;
+	u32                     underlay_qpn;
 };
 
 struct mlx5_ib_cq_buf {
@@ -399,6 +400,7 @@ enum mlx5_ib_qp_flags {
 	MLX5_IB_QP_CAP_SCATTER_FCS		= 1 << 7,
 	MLX5_IB_QP_RSS				= 1 << 8,
 	MLX5_IB_QP_CVLAN_STRIPPING		= 1 << 9,
+	MLX5_IB_QP_UNDERLAY			= 1 << 10,
 };
 
 struct mlx5_umr_wr {
drivers/infiniband/hw/mlx5/qp.c +57 -15

178 lines not on this page · raw message

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw

PATCH 4/6 IB/mlx5: Add multicast flow steering support for underlay QP

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

In order to add multicast flow steering support, there is need to block the attaching of mcg flow for underlay QP, recognize multicast IB_FLOW_SPEC_IPV4 based on its IP and enable creating/destroying flow for IB layer.

---
 drivers/infiniband/hw/mlx5/main.c | 43 ++++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 12 deletions(-)
drivers/infiniband/hw/mlx5/main.c +31 -12
index d45772da0963..92d8c764529c 100644
@@ -2044,21 +2044,32 @@ static int parse_flow_attr(struct mlx5_core_dev *mdev, u32 *match_c,
  */
 static bool flow_is_multicast_only(struct ib_flow_attr *ib_attr)
 {
-	struct ib_flow_spec_eth *eth_spec;
+	union ib_flow_spec *flow_spec;
 
 	if (ib_attr->type != IB_FLOW_ATTR_NORMAL ||
-	    ib_attr->size < sizeof(struct ib_flow_attr) +
-	    sizeof(struct ib_flow_spec_eth) ||
 	    ib_attr->num_of_specs < 1)
 		return false;
 
-	eth_spec = (struct ib_flow_spec_eth *)(ib_attr + 1);
-	if (eth_spec->type != IB_FLOW_SPEC_ETH ||
-	    eth_spec->size != sizeof(*eth_spec))
+	flow_spec = (union ib_flow_spec *)(ib_attr + 1);
+	if (flow_spec->type == IB_FLOW_SPEC_IPV4) {
+		struct ib_flow_spec_ipv4 *ipv4_spec;
+
+		ipv4_spec = (struct ib_flow_spec_ipv4 *)flow_spec;
+		if (ipv4_is_multicast(ipv4_spec->val.dst_ip))
+			return true;
+
 		return false;
+	}
+
+	if (flow_spec->type == IB_FLOW_SPEC_ETH) {
+		struct ib_flow_spec_eth *eth_spec;
+
+		eth_spec = (struct ib_flow_spec_eth *)flow_spec;
+		return is_multicast_ether_addr(eth_spec->mask.dst_mac) &&
+		       is_multicast_ether_addr(eth_spec->val.dst_mac);
+	}
 
-	return is_multicast_ether_addr(eth_spec->mask.dst_mac) &&
-	       is_multicast_ether_addr(eth_spec->val.dst_mac);
+	return false;
 }
 
 static bool is_valid_ethertype(struct mlx5_core_dev *mdev,
@@ -2536,8 +2547,14 @@ static struct ib_flow *mlx5_ib_create_flow(struct ib_qp *qp,
 static int mlx5_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
 {
 	struct mlx5_ib_dev *dev = to_mdev(ibqp->device);
+	struct mlx5_ib_qp *mqp = to_mqp(ibqp);
 	int err;
 
+	if (mqp->flags & MLX5_IB_QP_UNDERLAY) {
+		mlx5_ib_dbg(dev, "Attaching a multi cast group to underlay QP is not supported\n");
+		return -EOPNOTSUPP;
+	}
+
 	err = mlx5_core_attach_mcg(dev->mdev, gid, ibqp->qp_num);
 	if (err)
 		mlx5_ib_warn(dev, "failed attaching QPN 0x%x, MGID %pI6\n",
@@ -3714,18 +3731,20 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
 			(1ull << IB_USER_VERBS_CMD_CLOSE_XRCD);
 	}
 
+	dev->ib_dev.create_flow	= mlx5_ib_create_flow;
+	dev->ib_dev.destroy_flow = mlx5_ib_destroy_flow;
+	dev->ib_dev.uverbs_ex_cmd_mask |=
+			(1ull << IB_USER_VERBS_EX_CMD_CREATE_FLOW) |
+			(1ull << IB_USER_VERBS_EX_CMD_DESTROY_FLOW);
+
 	if (mlx5_ib_port_link_layer(&dev->ib_dev, 1) ==
 	    IB_LINK_LAYER_ETHERNET) {
-		dev->ib_dev.create_flow	= mlx5_ib_create_flow;
-		dev->ib_dev.destroy_flow = mlx5_ib_destroy_flow;
 		dev->ib_dev.create_wq	 = mlx5_ib_create_wq;
 		dev->ib_dev.modify_wq	 = mlx5_ib_modify_wq;
 		dev->ib_dev.destroy_wq	 = mlx5_ib_destroy_wq;
 		dev->ib_dev.create_rwq_ind_table = mlx5_ib_create_rwq_ind_table;
 		dev->ib_dev.destroy_rwq_ind_table = mlx5_ib_destroy_rwq_ind_table;
 		dev->ib_dev.uverbs_ex_cmd_mask |=
-			(1ull << IB_USER_VERBS_EX_CMD_CREATE_FLOW) |
-			(1ull << IB_USER_VERBS_EX_CMD_DESTROY_FLOW) |
 			(1ull << IB_USER_VERBS_EX_CMD_CREATE_WQ) |
 			(1ull << IB_USER_VERBS_EX_CMD_MODIFY_WQ) |
 			(1ull << IB_USER_VERBS_EX_CMD_DESTROY_WQ) |

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

PATCH 5/6 net/mlx5: Report enhanced capabilities for IPoIB

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

Report 'ipoib_enhanced_offloads' capabilities from
the core layer, it will be used in the next patch from this series.

---
 drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++++++
 include/linux/mlx5/device.h                  | 6 +++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
index 1bc14d0fded8..cf468be1479c 100644
@@ -119,6 +119,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 			return err;
 	}
 
+	if (MLX5_CAP_GEN(dev, ipoib_enhanced_offloads)) {
+		err = mlx5_core_get_caps(dev, MLX5_CAP_IPOIB_ENHANCED_OFFLOADS);
+		if (err)
+			return err;
+	}
+
 	if (MLX5_CAP_GEN(dev, pg)) {
 		err = mlx5_core_get_caps(dev, MLX5_CAP_ODP);
 		if (err)
include/linux/mlx5/device.h +5 -1
index a940ec6a046c..72a5ef0ad40e 100644
@@ -966,7 +966,7 @@ enum mlx5_cap_type {
 	MLX5_CAP_ATOMIC,
 	MLX5_CAP_ROCE,
 	MLX5_CAP_IPOIB_OFFLOADS,
-	MLX5_CAP_EOIB_OFFLOADS,
+	MLX5_CAP_IPOIB_ENHANCED_OFFLOADS,
 	MLX5_CAP_FLOW_TABLE,
 	MLX5_CAP_ESWITCH_FLOW_TABLE,
 	MLX5_CAP_ESWITCH,
@@ -1008,6 +1008,10 @@ enum mlx5_mcam_feature_groups {
 	MLX5_GET(per_protocol_networking_offload_caps,\
 		 mdev->caps.hca_max[MLX5_CAP_ETHERNET_OFFLOADS], cap)
 
+#define MLX5_CAP_IPOIB_ENHANCED(mdev, cap) \
+	MLX5_GET(per_protocol_networking_offload_caps,\
+		 mdev->caps.hca_cur[MLX5_CAP_IPOIB_ENHANCED_OFFLOADS], cap)
+
 #define MLX5_CAP_ROCE(mdev, cap) \
 	MLX5_GET(roce_cap, mdev->caps.hca_cur[MLX5_CAP_ROCE], cap)
 

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw

PATCH 6/6 IB/mlx5: Report RX checksum capabilities for IPoIB

In reply to a message not in the archive

From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>

Report RX checksum capabilities when 'ipoib_enhanced_offloads' capabilities are set and support checksum.

---
 drivers/infiniband/hw/mlx5/main.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/infiniband/hw/mlx5/main.c +4 -0
index 92d8c764529c..3510c0568ba1 100644
@@ -693,6 +693,10 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 		props->device_cap_flags |= IB_DEVICE_UD_TSO;
 	}
 
+	if (MLX5_CAP_GEN(mdev, ipoib_enhanced_offloads) &&
+	    MLX5_CAP_IPOIB_ENHANCED(mdev, csum_cap))
+		props->device_cap_flags |= IB_DEVICE_UD_IP_CSUM;
+
 	if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads) &&
 	    MLX5_CAP_ETH(dev->mdev, scatter_fcs)) {
 		/* Legacy bit to support old userspace libraries */

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html