Skip to contentSkip to conversation

Hardware tag matching support

PATCH In review Leon Romanovsky 5 participants 23 messages Aug 28, 2016 – Oct 14, 2016 10 messages not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Hardware tag matching support

Message Passing Interface (MPI) is a communication protocol that is widely used for exchange of messages among processes in high-performance computing (HPC) systems. Messages sent from a sending process to a destination process are marked with an identifying label, referred to as a tag. Destination processes post buffers in local memory that are similarly marked with tags. When a message is received by the receiver (i.e., the host computer on which the destination process is running), the message is stored in a buffer whose tag matches the message tag. The process of finding a buffer with a matching tag for the received packet is called tag matching.

There are two protocols that are generally used to send messages over
MPI: The "Eager Protocol" is best suited to small messages that are simply sent to the destination process and received in an appropriate matching buffer. The "Rendezvous Protocol" is better suited to large messages. In Rendezvous, when the sender process has a large message to send, it first sends a small message to the destination process announcing its intention to send the large message. This small message is referred to as an RTS (ready to send) message. The RTS includes the message tag and buffer address in the sender. The destination process matches the RTS to a posted receive buffer, or posts such a buffer if one does not already exist. Once a matching receive buffer has been posted at the destination process side, the receiver initiates a remote direct memory access (RDMA) read request to read the data from the buffer address listed by the sender in the RTS message.

MPI tag matching, when performed in software by a host processor, can consume substantial host resources, thus detracting from the performance of the actual software applications that are using MPI for communications. One possible solution is to offload the entire tag matching process to a peripheral hardware device, such as a network interface controller (NIC). In this case, the software application using MPI will post a set of buffers in a memory of the host processor and will pass the entire list of tags associated with the buffers to the NIC. In large-scale networks, however, the NIC may be required to simultaneously support many communicating processes and contexts (referred to in MPI parlance as "ranks" and "communicators," respectively). NIC access to and matching of the large lists of tags involved in such a scenario can itself become a bottleneck. The NIC must also be able to handle "unexpected" traffic, for which buffers and tags have not yet been posted, which may also degrade performance.

When the NIC receives a message over the network from one of the peer processes, and the message contains a label in accordance with the protocol, the NIC compares the label to the labels in the part of the list that was pushed to the NIC. Upon finding a match to the label, the NIC writes data conveyed in the message to the buffer in the memory that is associated with this label and submits a notification to the software process. The notification serves two purposes: both to indicate to the software process that the label has been consumed, so that the process will update the list of the labels posted to the NIC; and to inform the software process that the data are available in the buffer. In some cases (such as when the NIC retrieves the data from the remote node by RDMA), the NIC may submit two notifications, in the form of completion reports, of which the first informs the software process of the consumption of the label and the second announces availability of the data.

This patch series adds to Mellanox ConnectX HCA driver support of tag matching. It introduces new hardware object eXtended shared Receive Queue (XRQ), which follows SRQ semantics with addition of extended receive buffers topologies and offloads. This series adds tag matching topology and rendezvouz offload.

Available in the "topic/xrq" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/xrq

Thanks,
Artemy & Leon

Artemy Kovalyov (10):
IB/core: Add XRQ capabilities
IB/core: Make CQ separate part of SRQ context IB/core: Add new SRQ type IB_SRQT_TAG_MATCHING IB/uverbs: Expose tag matching capabilties to UAPI IB/uverbs: Expose XRQ capabilities IB/uverbs: Add XRQ creation parameter to UAPI IB/uverbs: Add new SRQ type IB_SRQT_TAG_MATCHING IB/mlx5: Fill XRQ capabilities
net/mlx5: Add XRQ support
IB/mlx5: Support IB_SRQT_TAG_MATCHING

 drivers/infiniband/core/uverbs_cmd.c          |  31 +++++-
 drivers/infiniband/core/verbs.c               |  16 +--
 drivers/infiniband/hw/mlx5/main.c             |  21 +++-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   6 ++
 drivers/infiniband/hw/mlx5/srq.c              |  15 ++-
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 150 ++++++++++++++++++++++++--
 include/linux/mlx5/driver.h                   |   1 +
 include/linux/mlx5/srq.h                      |   5 +
 include/rdma/ib_verbs.h                       |  61 +++++++++--
 include/uapi/rdma/ib_user_verbs.h             |  36 ++++++-
 10 files changed, 307 insertions(+), 35 deletions(-)

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

PATCH 1/10 IB/core: Add XRQ capabilities

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

This patch adds following XRQ capabilities:

* max_unexpected_tags - The maximum number of tags in the unexpected message list

* tag_mask_length - Number of bits in a tag matching mask

* header_size - The maximum size for the TM header including RNDV info

* app_context_size - The size for the application context field in the XRQ context

* max_match_list - Maximal number of tags which can be matched simultaneously

* capability_flags - Tag matching message type supported:
- IBV_NO_TAG - The HW supports messages without tag sent on QPs attached to a XRQ
- IBV_EAGER_EXPECTED - The HW supports tag matching for EAGER messages when the send arrives after the corresponding receive
- IBV_EAGER_UNEXPECTED - The HW supports tag matching for EAGER messages when the send arrives before the corresponding receive
- IBV_RNDV_MATCHED - The HW supports tag matching for RENDEZVOUS messages when the send arrives after the corresponding receive
- IBV_RNDV_UNEXPECTED - The HW supports tag matching for RENDEZVOUS messages when the send arrives before the corresponding receive

---
 include/rdma/ib_verbs.h | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
include/rdma/ib_verbs.h +33 -0
index 8e90dd2..000f57d 100644
@@ -261,6 +261,38 @@ struct ib_odp_caps {
 	} per_transport_caps;
 };
 
+enum ibv_xrq_cap_flags {
+	/* The HW supports messages without tag
+	 * sent on QPs attached to a XRQ
+	 */
+	IBV_NO_TAG	     = 1 << 0,
+	/* The HW supports tag matching for EAGER messages when
+	 * the send arrives after the corresponding receive
+	 */
+	IBV_EAGER_EXPECTED   = 1 << 1,
+	/* The HW supports tag matching for EAGER messages when
+	 * the send arrives before the corresponding receive
+	 */
+	IBV_EAGER_UNEXPECTED = 1 << 2,
+	/* The HW supports tag matching for RANDEZVOUS messages when
+	 * the send arrives after the corresponding receive (for RC QPs)
+	 */
+	IBV_RNDV_EXPECTED_RC = 1 << 3,
+	/* The HW supports tag matching for RANDEZVOUS messages when
+	 * the send arrives before the corresponding receive
+	 */
+	IBV_RNDV_UNEXPECTED  = 1 << 5,
+};
+
+struct ib_xrq_caps {
+	uint32_t max_unexpected_tags;
+	uint32_t tag_mask_length;
+	uint32_t header_size;
+	uint32_t app_context_size;
+	uint32_t max_match_list;
+	uint32_t capability_flags;
+};
+
 enum ib_cq_creation_flags {
 	IB_CQ_FLAGS_TIMESTAMP_COMPLETION   = 1 << 0,
 	IB_CQ_FLAGS_IGNORE_OVERRUN	   = 1 << 1,
@@ -318,6 +350,7 @@ struct ib_device_attr {
 	struct ib_odp_caps	odp_caps;
 	uint64_t		timestamp_mask;
 	uint64_t		hca_core_clock; /* in KHZ */
+	struct ib_xrq_caps	xrq_caps;
 };
 
 enum ib_mtu {
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw

PATCH 2/10 IB/core: Make CQ separate part of SRQ context

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Before this change CQ attached to SRQ was part of XRC specific extension. Moving CQ out makes it available to other types extending SRQ functionality.

---
 drivers/infiniband/core/uverbs_cmd.c |  2 +-
 drivers/infiniband/core/verbs.c      | 16 +++++++++-------
 drivers/infiniband/hw/mlx5/main.c    |  8 ++++----
 include/rdma/ib_verbs.h              | 19 +++++++++++++------
 4 files changed, 27 insertions(+), 18 deletions(-)
drivers/infiniband/core/uverbs_cmd.c +1 -1
index f664731..e1fb1cd 100644
@@ -3847,7 +3847,7 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	srq->srq_context   = attr.srq_context;
 
 	if (cmd->srq_type == IB_SRQT_XRC) {
-		srq->ext.xrc.cq   = attr.ext.xrc.cq;
+		srq->ext.cq       = attr.ext.xrc.cq;
 		srq->ext.xrc.xrcd = attr.ext.xrc.xrcd;
 		atomic_inc(&attr.ext.xrc.cq->usecnt);
 		atomic_inc(&attr.ext.xrc.xrcd->usecnt);
drivers/infiniband/core/verbs.c +9 -7
index f2b776e..3349ffb 100644
@@ -596,11 +596,13 @@ struct ib_srq *ib_create_srq(struct ib_pd *pd,
 		srq->event_handler = srq_init_attr->event_handler;
 		srq->srq_context   = srq_init_attr->srq_context;
 		srq->srq_type      = srq_init_attr->srq_type;
+		if (ib_srq_has_cq(srq->srq_type)) {
+			srq->ext.cq   = srq_init_attr->ext.xrc.cq;
+			atomic_inc(&srq->ext.cq->usecnt);
+		}
 		if (srq->srq_type == IB_SRQT_XRC) {
 			srq->ext.xrc.xrcd = srq_init_attr->ext.xrc.xrcd;
-			srq->ext.xrc.cq   = srq_init_attr->ext.xrc.cq;
 			atomic_inc(&srq->ext.xrc.xrcd->usecnt);
-			atomic_inc(&srq->ext.xrc.cq->usecnt);
 		}
 		atomic_inc(&pd->usecnt);
 		atomic_set(&srq->usecnt, 0);
@@ -641,18 +643,18 @@ int ib_destroy_srq(struct ib_srq *srq)
 
 	pd = srq->pd;
 	srq_type = srq->srq_type;
-	if (srq_type == IB_SRQT_XRC) {
+	if (ib_srq_has_cq(srq_type))
+		cq = srq->ext.cq;
+	if (srq_type == IB_SRQT_XRC)
 		xrcd = srq->ext.xrc.xrcd;
-		cq = srq->ext.xrc.cq;
-	}
 
 	ret = srq->device->destroy_srq(srq);
 	if (!ret) {
 		atomic_dec(&pd->usecnt);
-		if (srq_type == IB_SRQT_XRC) {
+		if (srq_type == IB_SRQT_XRC)
 			atomic_dec(&xrcd->usecnt);
+		if (ib_srq_has_cq(srq_type))
 			atomic_dec(&cq->usecnt);
-		}
 	}
 
 	return ret;
drivers/infiniband/hw/mlx5/main.c +4 -4
index f02a975..77ca218 100644
@@ -2387,9 +2387,9 @@ static int create_dev_resources(struct mlx5_ib_resources *devr)
 	devr->s0->srq_context   = NULL;
 	devr->s0->srq_type      = IB_SRQT_XRC;
 	devr->s0->ext.xrc.xrcd	= devr->x0;
-	devr->s0->ext.xrc.cq	= devr->c0;
+	devr->s0->ext.cq	= devr->c0;
 	atomic_inc(&devr->s0->ext.xrc.xrcd->usecnt);
-	atomic_inc(&devr->s0->ext.xrc.cq->usecnt);
+	atomic_inc(&devr->s0->ext.cq->usecnt);
 	atomic_inc(&devr->p0->usecnt);
 	atomic_set(&devr->s0->usecnt, 0);
 
@@ -2408,9 +2408,9 @@ static int create_dev_resources(struct mlx5_ib_resources *devr)
 	devr->s1->event_handler = NULL;
 	devr->s1->srq_context   = NULL;
 	devr->s1->srq_type      = IB_SRQT_BASIC;
-	devr->s1->ext.xrc.cq	= devr->c0;
+	devr->s1->ext.cq	= devr->c0;
 	atomic_inc(&devr->p0->usecnt);
-	atomic_set(&devr->s0->usecnt, 0);
+	atomic_set(&devr->s1->usecnt, 0);
 
 	for (port = 0; port < ARRAY_SIZE(devr->ports); ++port) {
 		INIT_WORK(&devr->ports[port].pkey_change_work,
include/rdma/ib_verbs.h +13 -6
index 000f57d..8b92698 100644
@@ -935,6 +935,11 @@ enum ib_srq_type {
 	IB_SRQT_XRC
 };
 
+static inline bool ib_srq_has_cq(enum ib_srq_type srq_type)
+{
+	return srq_type == IB_SRQT_XRC;
+}
+
 enum ib_srq_attr_mask {
 	IB_SRQ_MAX_WR	= 1 << 0,
 	IB_SRQ_LIMIT	= 1 << 1,
@@ -1457,12 +1462,14 @@ struct ib_srq {
 	enum ib_srq_type	srq_type;
 	atomic_t		usecnt;
 
-	union {
-		struct {
-			struct ib_xrcd *xrcd;
-			struct ib_cq   *cq;
-			u32		srq_num;
-		} xrc;
+	struct {
+		struct ib_cq   *cq;
+		union {
+			struct {
+				struct ib_xrcd *xrcd;
+				u32		srq_num;
+			} xrc;
+		};
 	} ext;
 };
 
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw

PATCH 3/10 IB/core: Add new SRQ type IB_SRQT_TAG_MATCHING

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

New SRQ type - IB_SRQT_TAG_MATCHING. It supports extended tag matching receive buffers topology and rendezvouz offload.

When SRQ receives a message it will search through the matching list for the corresponding posted receive buffer. The process of searching the matching list is called tag matching.
In case the tag matching results in a match, the received message will be placed in the address specified by the receive buffer. In case no match was found the message will be placed in a generic buffer until the corresponding receive buffer will be posted. These messages are called unexpected and their set is called an unexpected list.

---
 include/rdma/ib_verbs.h | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
include/rdma/ib_verbs.h +9 -2
index 8b92698..0e7a711 100644
@@ -932,12 +932,14 @@ enum ib_cq_notify_flags {
 
 enum ib_srq_type {
 	IB_SRQT_BASIC,
-	IB_SRQT_XRC
+	IB_SRQT_XRC,
+	IB_SRQT_TAG_MATCHING,
 };
 
 static inline bool ib_srq_has_cq(enum ib_srq_type srq_type)
 {
-	return srq_type == IB_SRQT_XRC;
+	return srq_type == IB_SRQT_XRC ||
+	       srq_type == IB_SRQT_TAG_MATCHING;
 }
 
 enum ib_srq_attr_mask {
@@ -962,6 +964,11 @@ struct ib_srq_init_attr {
 			struct ib_xrcd *xrcd;
 			struct ib_cq   *cq;
 		} xrc;
+
+		struct {
+			struct ib_cq   *cq;
+			u32		list_size;
+		} tag_matching;
 	} ext;
 };
 
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw

PATCH 4/10 IB/uverbs: Expose tag matching capabilties to UAPI

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Expose following tag matching capabilities to UAPI:

* max_unexpected_tags - The maximum number of tags in the unexpected message list

* tag_mask_length - Number of bits in a tag matching mask

* header_size - The maximum size for the TM header including RNDV info

* app_context_size - The size for the application context field in the XRQ context

* max_match_list - Maximal number of tags which can be matched simultaneously

* capability_flags - Tag matching message type supported:
- IBV_NO_TAG - The HW supports messages without tag sent on QPs attached to a XRQ
- IBV_EAGER_EXPECTED - The HW supports tag matching for EAGER messages when the send arrives after the corresponding receive
- IBV_EAGER_UNEXPECTED - The HW supports tag matching for EAGER messages when the send arrives before the corresponding receive
- IBV_RNDV_MATCHED - The HW supports tag matching for RENDEZVOUS messages when the send arrives after the corresponding receive
- IBV_RNDV_UNEXPECTED - The HW supports tag matching for RENDEZVOUS messages when the send arrives before the corresponding receive

---
 include/uapi/rdma/ib_user_verbs.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
include/uapi/rdma/ib_user_verbs.h +34 -0
index 7f035f4b..4da8f90 100644
@@ -224,6 +224,39 @@ struct ib_uverbs_odp_caps {
 	__u32 reserved;
 };
 
+
+enum ibv_tm_cap_flags {
+	/* The HW supports messages without tag
+	 * sent on QPs attached to a XRQ
+	 */
+	IB_NO_TAG	     = 1 << 0,
+	/* The HW supports tag matching for EAGER messages when
+	 * the send arrives after the corresponding receive
+	 */
+	IB_EAGER_EXPECTED   = 1 << 1,
+	/* The HW supports tag matching for EAGER messages when
+	 * the send arrives before the corresponding receive
+	 */
+	IB_EAGER_UNEXPECTED = 1 << 2,
+	/* The HW supports tag matching for RANDEZVOUS messages when
+	 * the send arrives after the corresponding receive (for RC QPs)
+	 */
+	IB_RNDV_EXPECTED    = 1 << 3,
+	/* The HW supports tag matching for RANDEZVOUS messages when
+	 * the send arrives before the corresponding receive
+	 */
+	IB_RNDV_UNEXPECTED  = 1 << 5,
+};
+
+struct ib_uverbs_tm_caps {
+	__u32 max_unexpected_tags;
+	__u32 tag_mask_length;
+	__u32 header_size;
+	__u32 app_context_size;
+	__u32 max_match_list;
+	__u32 capability_flags;
+};
+
 struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_query_device_resp base;
 	__u32 comp_mask;
@@ -232,6 +265,7 @@ struct ib_uverbs_ex_query_device_resp {
 	__u64 timestamp_mask;
 	__u64 hca_core_clock; /* in KHZ */
 	__u64 device_cap_flags_ex;
+	struct ib_uverbs_tm_caps xrq_caps;
 };
 
 struct ib_uverbs_query_port {

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

PATCH 5/10 IB/uverbs: Expose XRQ capabilities

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Make XRQ capabilities available via ibv_query_device() verb.

---
 drivers/infiniband/core/uverbs_cmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/infiniband/core/uverbs_cmd.c +11 -0
index e1fb1cd..548d4b4 100644
@@ -4173,6 +4173,17 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 
 	resp.device_cap_flags_ex = attr.device_cap_flags;
 	resp.response_length += sizeof(resp.device_cap_flags_ex);
+
+	if (ucore->outlen < resp.response_length + sizeof(resp.xrq_caps))
+		goto end;
+
+	resp.xrq_caps.max_unexpected_tags = attr.xrq_caps.max_unexpected_tags;
+	resp.xrq_caps.tag_mask_length = attr.xrq_caps.tag_mask_length;
+	resp.xrq_caps.header_size = attr.xrq_caps.header_size;
+	resp.xrq_caps.app_context_size = attr.xrq_caps.app_context_size;
+	resp.xrq_caps.max_match_list = attr.xrq_caps.max_match_list;
+	resp.xrq_caps.capability_flags = attr.xrq_caps.capability_flags;
+	resp.response_length += sizeof(resp.xrq_caps);
 end:
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	return err;
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw

PATCH 6/10 IB/uverbs: Add XRQ creation parameter to UAPI

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add tm_list_size parameter to struct ib_uverbs_create_xsrq. If SRQ type is tag-matching this field defines maximum size of tag matching list. Otherwise - reserved.

---
 include/uapi/rdma/ib_user_verbs.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/uapi/rdma/ib_user_verbs.h +1 -1
index 4da8f90..475e098 100644
@@ -968,7 +968,7 @@ struct ib_uverbs_create_xsrq {
 	__u32 max_wr;
 	__u32 max_sge;
 	__u32 srq_limit;
-	__u32 reserved;
+	__u32 tm_list_size;
 	__u32 xrcd_handle;
 	__u32 cq_handle;
 	__u64 driver_data[0];
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw

PATCH 7/10 IB/uverbs: Add new SRQ type IB_SRQT_TAG_MATCHING

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add new SRQ type capable of new Tag Matching feature.

When SRQ receives a message it will search through the matching list for the corresponding posted receive buffer. The process of searching the matching list is called tag matching.
In case the tag matching results in a match, the received message will be placed in the address specified by the receive buffer. In case no match was found the message will be placed in a generic buffer until the corresponding receive buffer will be posted. These messages are called unexpected and their set is called an unexpected list.

---
 drivers/infiniband/core/uverbs_cmd.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/infiniband/core/uverbs_cmd.c +17 -1
index 548d4b4..b240c9a 100644
@@ -1831,7 +1831,7 @@ static int create_qp(struct ib_uverbs_file *file,
 			if (cmd->is_srq) {
 				srq = idr_read_srq(cmd->srq_handle,
 						   file->ucontext);
-				if (!srq || srq->srq_type != IB_SRQT_BASIC) {
+				if (!srq || srq->srq_type == IB_SRQT_XRC) {
 					ret = -EINVAL;
 					goto err_put;
 				}
@@ -3815,6 +3815,15 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 			ret = -EINVAL;
 			goto err_put_xrcd;
 		}
+	} else if (cmd->srq_type == IB_SRQT_TAG_MATCHING) {
+		attr.ext.tag_matching.cq  = idr_read_cq(cmd->cq_handle,
+							file->ucontext, 0);
+		if (!attr.ext.tag_matching.cq) {
+			ret = -EINVAL;
+			goto err;
+		}
+
+		attr.ext.tag_matching.list_size = cmd->tm_list_size;
 	}
 
 	pd  = idr_read_pd(cmd->pd_handle, file->ucontext);
@@ -3851,6 +3860,9 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 		srq->ext.xrc.xrcd = attr.ext.xrc.xrcd;
 		atomic_inc(&attr.ext.xrc.cq->usecnt);
 		atomic_inc(&attr.ext.xrc.xrcd->usecnt);
+	} else if (cmd->srq_type == IB_SRQT_TAG_MATCHING) {
+		srq->ext.cq = attr.ext.tag_matching.cq;
+		atomic_inc(&attr.ext.tag_matching.cq->usecnt);
 	}
 
 	atomic_inc(&pd->usecnt);
@@ -3877,6 +3889,8 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	if (cmd->srq_type == IB_SRQT_XRC) {
 		put_uobj_read(xrcd_uobj);
 		put_cq_read(attr.ext.xrc.cq);
+	} else if (cmd->srq_type == IB_SRQT_TAG_MATCHING) {
+		put_cq_read(attr.ext.tag_matching.cq);
 	}
 	put_pd_read(pd);
 
@@ -3902,6 +3916,8 @@ err_put:
 err_put_cq:
 	if (cmd->srq_type == IB_SRQT_XRC)
 		put_cq_read(attr.ext.xrc.cq);
+	else if (cmd->srq_type == IB_SRQT_TAG_MATCHING)
+		put_cq_read(attr.ext.tag_matching.cq);
 
 err_put_xrcd:
 	if (cmd->srq_type == IB_SRQT_XRC) {
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw

PATCH 8/10 IB/mlx5: Fill XRQ capabilities

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Provide driver specific values for XRQ capabilities

---
 drivers/infiniband/hw/mlx5/main.c    | 13 +++++++++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  6 ++++++
 2 files changed, 19 insertions(+)
drivers/infiniband/hw/mlx5/main.c +13 -0
index 77ca218..692a687 100644
@@ -594,6 +594,19 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	if (!mlx5_core_is_pf(mdev))
 		props->device_cap_flags |= IB_DEVICE_VIRTUAL_FUNCTION;
 
+	props->xrq_caps.max_unexpected_tags =
+		1 << MLX5_CAP_GEN(mdev, log_max_srq_sz);
+	props->xrq_caps.tag_mask_length  = MLX5_TM_TAG_SIZE;
+	props->xrq_caps.header_size      = MLX5_TM_HEADER_SIZE;
+	props->xrq_caps.app_context_size = MLX5_TM_APP_CTX_SIZE;
+	props->xrq_caps.max_match_list   =
+		1 << MLX5_CAP_GEN(mdev, log_tag_matching_list_sz);
+	props->xrq_caps.capability_flags = IBV_NO_TAG |
+					   IBV_EAGER_EXPECTED |
+					   IBV_EAGER_UNEXPECTED |
+					   IBV_RNDV_EXPECTED_RC |
+					   IBV_RNDV_UNEXPECTED;
+
 	if (uhw->outlen) {
 		err = ib_copy_to_udata(uhw, &resp, resp.response_length);
 
drivers/infiniband/hw/mlx5/mlx5_ib.h +6 -0
index a59034a..1c204d7 100644
@@ -105,6 +105,12 @@ enum {
 	MLX5_CQE_VERSION_V1,
 };
 
+enum {
+	MLX5_TM_TAG_SIZE		= 64,
+	MLX5_TM_HEADER_SIZE		= 16,
+	MLX5_TM_APP_CTX_SIZE		= 32,
+};
+
 struct mlx5_ib_vma_private_data {
 	struct list_head list;
 	struct vm_area_struct *vma;
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw

PATCH 9/10 net/mlx5: Add XRQ support

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add support to new XRQ(eXtended shared Receive Queue) hardware object. It supports SRQ semantics with addition of extended receive buffers topologies and offloads.

Currently supports tag matching topology and rendezvouz offload.

---
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 150 ++++++++++++++++++++++++--
 include/linux/mlx5/driver.h                   |   1 +
 include/linux/mlx5/srq.h                      |   5 +
 3 files changed, 146 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/srq.c +140 -10
index 30996300..0669dd9 100644
@@ -435,16 +435,128 @@ out:
 	return err;
 }
 
+static int create_xrq_cmd(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
+			  struct mlx5_srq_attr *in)
+{
+	u32 create_out[MLX5_ST_SZ_DW(create_xrq_out)] = {0};
+	void *create_in;
+	void *xrqc;
+	void *wq;
+	int pas_size;
+	int inlen;
+	int err;
+
+	pas_size = get_pas_size(in);
+	inlen = MLX5_ST_SZ_BYTES(create_xrq_in) + pas_size;
+	create_in = mlx5_vzalloc(inlen);
+	if (!create_in)
+		return -ENOMEM;
+
+	xrqc = MLX5_ADDR_OF(create_xrq_in, create_in, xrq_context);
+	wq = MLX5_ADDR_OF(xrqc, xrqc, wq);
+
+	set_wq(wq, in);
+	memcpy(MLX5_ADDR_OF(xrqc, xrqc, wq.pas), in->pas, pas_size);
+
+	if (in->type == IB_SRQT_TAG_MATCHING) {
+		MLX5_SET(xrqc, xrqc, topology, MLX5_XRQC_TOPOLOGY_TAG_MATCHING);
+		if (in->flags & MLX5_SRQ_FLAG_RNDV)
+			MLX5_SET(xrqc, xrqc, offload, MLX5_XRQC_OFFLOAD_RNDV);
+		MLX5_SET(xrqc, xrqc,
+			 tag_matching_topology_context.log_matching_list_sz,
+			 in->tm_log_list_size);
+	}
+	MLX5_SET(xrqc, xrqc, user_index, in->user_index);
+	MLX5_SET(xrqc, xrqc, cqn, in->cqn);
+	MLX5_SET(create_xrq_in, create_in, opcode, MLX5_CMD_OP_CREATE_XRQ);
+	err = mlx5_cmd_exec(dev, create_in, inlen, create_out,
+			    sizeof(create_out));
+	kvfree(create_in);
+	if (!err)
+		srq->srqn = MLX5_GET(create_xrq_out, create_out, xrqn);
+
+	return err;
+}
+
+static int destroy_xrq_cmd(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq)
+{
+	u32 in[MLX5_ST_SZ_DW(destroy_xrq_in)] = {0};
+	u32 out[MLX5_ST_SZ_DW(destroy_xrq_out)] = {0};
+
+	MLX5_SET(destroy_xrq_in, in, opcode, MLX5_CMD_OP_DESTROY_XRQ);
+	MLX5_SET(destroy_xrq_in, in, xrqn,   srq->srqn);
+
+	return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+static int arm_xrq_cmd(struct mlx5_core_dev *dev,
+		       struct mlx5_core_srq *srq,
+		       u16 lwm)
+{
+	u32 out[MLX5_ST_SZ_DW(arm_rq_out)] = {0};
+	u32 in[MLX5_ST_SZ_DW(arm_rq_in)] = {0};
+
+	MLX5_SET(arm_rq_in, in, opcode,     MLX5_CMD_OP_ARM_RQ);
+	MLX5_SET(arm_rq_in, in, op_mod,     MLX5_ARM_RQ_IN_OP_MOD_XRQ);
+	MLX5_SET(arm_rq_in, in, srq_number, srq->srqn);
+	MLX5_SET(arm_rq_in, in, lwm,	    lwm);
+
+	return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+static int query_xrq_cmd(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
+			 struct mlx5_srq_attr *out)
+{
+	u32 in[MLX5_ST_SZ_DW(query_xrq_in)] = {0};
+	u32 *xrq_out;
+	int outlen = MLX5_ST_SZ_BYTES(query_xrq_out);
+	void *xrqc;
+	int err;
+
+	xrq_out =  mlx5_vzalloc(MLX5_ST_SZ_BYTES(query_xrq_out));
+	if (!xrq_out)
+		return -ENOMEM;
+
+	MLX5_SET(query_xrq_in, in, opcode, MLX5_CMD_OP_QUERY_XRQ);
+	MLX5_SET(query_xrq_in, in, xrqn, srq->srqn);
+
+	err = mlx5_cmd_exec(dev, in, sizeof(in), xrq_out, outlen);
+	if (err)
+		goto out;
+
+	xrqc = MLX5_ADDR_OF(query_xrq_out, xrq_out, xrq_context);
+	get_wq(MLX5_ADDR_OF(xrqc, xrqc, wq), out);
+	if (MLX5_GET(xrqc, xrqc, state) != MLX5_XRQC_STATE_GOOD)
+		out->flags |= MLX5_SRQ_FLAG_ERR;
+	out->tm_next_tag =
+		MLX5_GET(xrqc, xrqc,
+			 tag_matching_topology_context.append_next_index);
+	out->tm_hw_phase_cnt =
+		MLX5_GET(xrqc, xrqc,
+			 tag_matching_topology_context.hw_phase_cnt);
+	out->tm_sw_phase_cnt =
+		MLX5_GET(xrqc, xrqc,
+			 tag_matching_topology_context.sw_phase_cnt);
+
+out:
+	kvfree(xrq_out);
+	return err;
+}
+
 static int create_srq_split(struct mlx5_core_dev *dev,
 			    struct mlx5_core_srq *srq,
 			    struct mlx5_srq_attr *in)
 {
 	if (!dev->issi)
 		return create_srq_cmd(dev, srq, in);
-	else if (srq->common.res == MLX5_RES_XSRQ)
+	switch (srq->common.res) {
+	case MLX5_RES_XSRQ:
 		return create_xrc_srq_cmd(dev, srq, in);
-	else
+	case MLX5_RES_XRQ:
+		return create_xrq_cmd(dev, srq, in);
+	default:
 		return create_rmp_cmd(dev, srq, in);
+	}
 }
 
 static int destroy_srq_split(struct mlx5_core_dev *dev,
@@ -452,10 +564,14 @@ static int destroy_srq_split(struct mlx5_core_dev *dev,
 {
 	if (!dev->issi)
 		return destroy_srq_cmd(dev, srq);
-	else if (srq->common.res == MLX5_RES_XSRQ)
+	switch (srq->common.res) {
+	case MLX5_RES_XSRQ:
 		return destroy_xrc_srq_cmd(dev, srq);
-	else
+	case MLX5_RES_XRQ:
+		return destroy_xrq_cmd(dev, srq);
+	default:
 		return destroy_rmp_cmd(dev, srq);
+	}
 }
 
 int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
@@ -464,10 +580,16 @@ int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
 	int err;
 	struct mlx5_srq_table *table = &dev->priv.srq_table;
 
-	if (in->type == IB_SRQT_XRC)
+	switch (in->type) {
+	case IB_SRQT_XRC:
 		srq->common.res = MLX5_RES_XSRQ;
-	else
+		break;
+	case IB_SRQT_TAG_MATCHING:
+		srq->common.res = MLX5_RES_XRQ;
+		break;
+	default:
 		srq->common.res = MLX5_RES_SRQ;
+	}
 
 	err = create_srq_split(dev, srq, in);
 	if (err)
@@ -528,10 +650,14 @@ int mlx5_core_query_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
 {
 	if (!dev->issi)
 		return query_srq_cmd(dev, srq, out);
-	else if (srq->common.res == MLX5_RES_XSRQ)
+	switch (srq->common.res) {
+	case MLX5_RES_XSRQ:
 		return query_xrc_srq_cmd(dev, srq, out);
-	else
+	case MLX5_RES_XRQ:
+		return query_xrq_cmd(dev, srq, out);
+	default:
 		return query_rmp_cmd(dev, srq, out);
+	}
 }
 EXPORT_SYMBOL(mlx5_core_query_srq);
 
@@ -540,10 +666,14 @@ int mlx5_core_arm_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
 {
 	if (!dev->issi)
 		return arm_srq_cmd(dev, srq, lwm, is_srq);
-	else if (srq->common.res == MLX5_RES_XSRQ)
+	switch (srq->common.res) {
+	case MLX5_RES_XSRQ:
 		return arm_xrc_srq_cmd(dev, srq, lwm);
-	else
+	case MLX5_RES_XRQ:
+		return arm_xrq_cmd(dev, srq, lwm);
+	default:
 		return arm_rmp_cmd(dev, srq, lwm);
+	}
 }
 EXPORT_SYMBOL(mlx5_core_arm_srq);
 
include/linux/mlx5/driver.h +1 -0
index 5cb9fa7..c158db0 100644
@@ -368,6 +368,7 @@ enum mlx5_res_type {
 	MLX5_RES_SQ	= MLX5_EVENT_QUEUE_TYPE_SQ,
 	MLX5_RES_SRQ	= 3,
 	MLX5_RES_XSRQ	= 4,
+	MLX5_RES_XRQ	= 5,
 };
 
 struct mlx5_core_rsc_common {
include/linux/mlx5/srq.h +5 -0
index 33c97dc..9905353 100644
@@ -38,6 +38,7 @@
 enum {
 	MLX5_SRQ_FLAG_ERR    = (1 << 0),
 	MLX5_SRQ_FLAG_WQ_SIG = (1 << 1),
+	MLX5_SRQ_FLAG_RNDV   = (1 << 2),
 };
 
 struct mlx5_srq_attr {
@@ -56,6 +57,10 @@ struct mlx5_srq_attr {
 	u32 user_index;
 	u64 db_record;
 	u64 *pas;
+	u32 tm_log_list_size;
+	u32 tm_next_tag;
+	u32 tm_hw_phase_cnt;
+	u32 tm_sw_phase_cnt;
 };
 
 struct mlx5_core_dev;

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw

PATCH 10/10 IB/mlx5: Support IB_SRQT_TAG_MATCHING

In reply to a message not in the archive

From: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>

When SRQ created with IB_SRQT_TAG_MATCHING pass to mlx5_core CQ, list_size and flag to enable rendezvouz offload.

---
 drivers/infiniband/hw/mlx5/srq.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx5/srq.c +11 -4
index ed6ac52..82fe622 100644
@@ -102,7 +102,7 @@ static int create_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq,
 				 udata->inlen - sizeof(ucmd)))
 		return -EINVAL;
 
-	if (in->type == IB_SRQT_XRC) {
+	if (in->type != IB_SRQT_BASIC) {
 		err = get_srq_user_index(to_mucontext(pd->uobject->context),
 					 &ucmd, udata->inlen, &uidx);
 		if (err)
@@ -146,7 +146,7 @@ static int create_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq,
 	in->log_page_size = page_shift - MLX5_ADAPTER_PAGE_SHIFT;
 	in->page_offset = offset;
 	if (MLX5_CAP_GEN(dev->mdev, cqe_version) == MLX5_CQE_VERSION_V1 &&
-	    in->type == IB_SRQT_XRC)
+	    in->type != IB_SRQT_BASIC)
 		in->user_index = uidx;
 
 	return 0;
@@ -213,7 +213,7 @@ static int create_srq_kernel(struct mlx5_ib_dev *dev, struct mlx5_ib_srq *srq,
 
 	in->log_page_size = page_shift - MLX5_ADAPTER_PAGE_SHIFT;
 	if (MLX5_CAP_GEN(dev->mdev, cqe_version) == MLX5_CQE_VERSION_V1 &&
-	    in->type == IB_SRQT_XRC)
+	    in->type != IB_SRQT_BASIC)
 		in->user_index = MLX5_IB_DEFAULT_UIDX;
 
 	return 0;
@@ -284,6 +284,8 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
 		    desc_size, init_attr->attr.max_wr, srq->msrq.max, srq->msrq.max_gs,
 		    srq->msrq.max_avail_gather);
 
+	in.type = init_attr->srq_type;
+
 	if (pd->uobject)
 		err = create_srq_user(pd, srq, &in, udata, buf_size);
 	else
@@ -295,7 +297,6 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
 		goto err_srq;
 	}
 
-	in.type = init_attr->srq_type;
 	in.log_size = ilog2(srq->msrq.max);
 	in.wqe_shift = srq->msrq.wqe_shift - 4;
 	if (srq->wq_sig)
@@ -306,6 +307,12 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
 	} else if (init_attr->srq_type == IB_SRQT_BASIC) {
 		in.xrcd = to_mxrcd(dev->devr.x0)->xrcdn;
 		in.cqn = to_mcq(dev->devr.c0)->mcq.cqn;
+	} else if (init_attr->srq_type == IB_SRQT_TAG_MATCHING) {
+		in.xrcd = to_mxrcd(dev->devr.x0)->xrcdn;
+		in.cqn = to_mcq(init_attr->ext.tag_matching.cq)->mcq.cqn;
+		in.tm_log_list_size =
+			ilog2(init_attr->ext.tag_matching.list_size);
+		in.flags |= MLX5_SRQ_FLAG_RNDV;
 	}
 
 	in.pd = to_mpd(pd)->pdn;
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#12 lore raw
In reply to a message not in the archive
Quoted 64 lines · Leon Romanovsky

Message Passing Interface (MPI) is a communication protocol that is widely used for exchange of messages among processes in high-performance computing (HPC) systems. Messages sent from a sending process to a destination process are marked with an identifying label, referred to as a tag. Destination processes post buffers in local memory that are similarly marked with tags. When a message is received by the receiver (i.e., the host computer on which the destination process is running), the message is stored in a buffer whose tag matches the message tag. The process of finding a buffer with a matching tag for the received packet is called tag matching.

There are two protocols that are generally used to send messages over
MPI: The "Eager Protocol" is best suited to small messages that are simply sent to the destination process and received in an appropriate matching buffer. The "Rendezvous Protocol" is better suited to large messages. In Rendezvous, when the sender process has a large message to send, it first sends a small message to the destination process announcing its intention to send the large message. This small message is referred to as an RTS (ready to send) message. The RTS includes the message tag and buffer address in the sender. The destination process matches the RTS to a posted receive buffer, or posts such a buffer if one does not already exist. Once a matching receive buffer has been posted at the destination process side, the receiver initiates a remote direct memory access (RDMA) read request to read the data from the buffer address listed by the sender in the RTS message.

MPI tag matching, when performed in software by a host processor, can consume substantial host resources, thus detracting from the performance of the actual software applications that are using MPI for communications. One possible solution is to offload the entire tag matching process to a peripheral hardware device, such as a network interface controller (NIC). In this case, the software application using MPI will post a set of buffers in a memory of the host processor and will pass the entire list of tags associated with the buffers to the NIC. In large-scale networks, however, the NIC may be required to simultaneously support many communicating processes and contexts (referred to in MPI parlance as "ranks" and "communicators," respectively). NIC access to and matching of the large lists of tags involved in such a scenario can itself become a bottleneck. The NIC must also be able to handle "unexpected" traffic, for which buffers and tags have not yet been posted, which may also degrade performance.

When the NIC receives a message over the network from one of the peer processes, and the message contains a label in accordance with the protocol, the NIC compares the label to the labels in the part of the list that was pushed to the NIC. Upon finding a match to the label, the NIC writes data conveyed in the message to the buffer in the memory that is associated with this label and submits a notification to the software process. The notification serves two purposes: both to indicate to the software process that the label has been consumed, so that the process will update the list of the labels posted to the NIC; and to inform the software process that the data are available in the buffer. In some cases (such as when the NIC retrieves the data from the remote node by RDMA), the NIC may submit two notifications, in the form of completion reports, of which the first informs the software process of the consumption of the label and the second announces availability of the data.

This patch series adds to Mellanox ConnectX HCA driver support of tag matching. It introduces new hardware object eXtended shared Receive Queue (XRQ), which follows SRQ semantics with addition of extended receive buffers topologies and offloads. This series adds tag matching topology and rendezvouz offload.

Quoted 5 lines

Available in the "topic/xrq" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/xrq

Hi Doug,

For any reasons, I don't see this patch set in your tree. Did I miss it?

Thanks

Quoted 35 lines

Thanks,
Artemy & Leon

Artemy Kovalyov (10):
IB/core: Add XRQ capabilities
IB/core: Make CQ separate part of SRQ context IB/core: Add new SRQ type IB_SRQT_TAG_MATCHING IB/uverbs: Expose tag matching capabilties to UAPI IB/uverbs: Expose XRQ capabilities IB/uverbs: Add XRQ creation parameter to UAPI IB/uverbs: Add new SRQ type IB_SRQT_TAG_MATCHING IB/mlx5: Fill XRQ capabilities
net/mlx5: Add XRQ support
IB/mlx5: Support IB_SRQT_TAG_MATCHING

 drivers/infiniband/core/uverbs_cmd.c          |  31 +++++-
 drivers/infiniband/core/verbs.c               |  16 +--
 drivers/infiniband/hw/mlx5/main.c             |  21 +++-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   6 ++
 drivers/infiniband/hw/mlx5/srq.c              |  15 ++-
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 150 ++++++++++++++++++++++++--
 include/linux/mlx5/driver.h                   |   1 +
 include/linux/mlx5/srq.h                      |   5 +
 include/rdma/ib_verbs.h                       |  61 +++++++++--
 include/uapi/rdma/ib_user_verbs.h             |  36 ++++++-
 10 files changed, 307 insertions(+), 35 deletions(-)

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to a message not in the archive
Quoted 20 lines

For any reasons, I don't see this patch set in your tree. Did I miss it?

Thanks

Quoted 15 lines

Thanks,
Artemy & Leon

Artemy Kovalyov (10):
IB/core: Add XRQ capabilities
IB/core: Make CQ separate part of SRQ context IB/core: Add new SRQ type IB_SRQT_TAG_MATCHING IB/uverbs: Expose tag matching capabilties to UAPI IB/uverbs: Expose XRQ capabilities IB/uverbs: Add XRQ creation parameter to UAPI IB/uverbs: Add new SRQ type IB_SRQT_TAG_MATCHING IB/mlx5: Fill XRQ capabilities
net/mlx5: Add XRQ support
IB/mlx5: Support IB_SRQT_TAG_MATCHING

I was out when these were submitted, so this may have been answered. These patches change the uABI. Have the changes been vetted out by the IBTA or some other standards (I'm using that term loosely) organization? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw
In reply to a message not in the archive
Quoted 21 lines · Sean
Quoted 20 lines

For any reasons, I don't see this patch set in your tree. Did I miss it?

Thanks

Quoted 15 lines

Thanks,
Artemy & Leon

Artemy Kovalyov (10):
IB/core: Add XRQ capabilities
IB/core: Make CQ separate part of SRQ context IB/core: Add new SRQ type IB_SRQT_TAG_MATCHING IB/uverbs: Expose tag matching capabilties to UAPI IB/uverbs: Expose XRQ capabilities IB/uverbs: Add XRQ creation parameter to UAPI IB/uverbs: Add new SRQ type IB_SRQT_TAG_MATCHING IB/mlx5: Fill XRQ capabilities
net/mlx5: Add XRQ support
IB/mlx5: Support IB_SRQT_TAG_MATCHING

Quoted 2 lines

I was out when these were submitted, so this may have been answered. These patches change the uABI. Have the changes been vetted out by the IBTA or some other standards (I'm using that term loosely) organization?

I doubt they have, but they change the kernel to user library API. I'm not so worried about that since it's entirely under our control. It's the user library to application API that I would be more concerned with.

As these stand, they look fairly clean to me. Minimal in disruption and I see no reason other vendors couldn't use this to do tag matching on their own hardware if they wanted. I'm inclined to take them.

Signature · 2 lines
Doug Ledford <[email protected]>
    GPG Key ID: 0E572FDD
#18 lore raw
In reply to a message not in the archive
Quoted 4 lines

I doubt they have, but they change the kernel to user library API. I'm not so worried about that since it's entirely under our control. It's the user library to application API that I would be more concerned with.

I thought there was an agreement that since verbs implies a hardware implementation, then all kernel to user ABI changes to verbs would go through some external organization, such as the OFVWG, for discussion and approval. Has that changed?

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw
In reply to a message not in the archive
Quoted 6 lines · Sean
Quoted 4 lines

I doubt they have, but they change the kernel to user library API. I'm not so worried about that since it's entirely under our control. It's the user library to application API that I would be more concerned with.

I thought there was an agreement that since verbs implies a hardware implementation, then all kernel to user ABI changes to verbs would go through some external organization, such as the OFVWG, for discussion and approval. Has that changed?

What makes OFVWG more qualified to review an ABI than linux-rdma?

That beeing said a review by an independent third party in addition to the vensors (intentional plural here!) is important, but I think we can handle that just fine.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#22 lore raw
In reply to a message not in the archive
Quoted 7 lines
Quoted 1 line
Quoted 1 line

I doubt they have, but they change the kernel to user library API.

I'm

Quoted 1 line
Quoted 1 line

not so worried about that since it's entirely under our control.

It's

Quoted 3 lines
Quoted 2 lines

the user library to application API that I would be more concerned with.

Quoted 6 lines
Quoted 1 line

I thought there was an agreement that since verbs implies a hardware

implementation, then all kernel to user ABI changes to verbs would go through some external organization, such as the OFVWG, for discussion and approval. Has that changed?

What makes OFVWG more qualified to review an ABI than linux-rdma?

We've had this discussion on the list before. The issue is that verbs isn't just a software API. It defines a hardware implementation. The consensus was that some 'standards' body needed to be able to review the proposed changes and its impact. The OFVWG was selected as the best available option. If we want to say no, and let the linux-rdma maintainer dictate the hardware interface exported by the kernel, then, yes, that's an option.

Now, if we want to kill the notion that there's some sort of common rdma/verbs interface that all devices will be able to use effectively, and replace it with vendor specific interfaces, I would completely go along with that. Then let user space figure out how to deal with the results.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw
In reply to a message not in the archive
Quoted 9 lines · Sean
Quoted 1 line

What makes OFVWG more qualified to review an ABI than linux-rdma?

We've had this discussion on the list before. The issue is that verbs isn't just a software API. It defines a hardware implementation. The consensus was that some 'standards' body needed to be able to review the proposed changes and its impact. The OFVWG was selected as the best available option. If we want to say no, and let the linux-rdma maintainer dictate the hardware interface exported by the kernel, then, yes, that's an option.

A bigger objection I have is that much of this new stuff is not adaquately documented.

Tag matching is a great example. It looks like this introduces many new headers into the on-wire protocol. Where are these headers documented? Where is the spec for the on-wire format? Where is the implementation for rxe?

Basically, is there enough public information that another vendor could implement this wire protocol? (if not, why are we even *contemplating* merging something like this???)

'Hopefully' the answer is that this information is all hidden away in the IBTA... Which opens up an entire new line of problems reviewing something like this without seeing the agreed spec..

The basic problem with linux-rdma is that it is not a reasonable place to review a new on-wire protocol amendment to the IBA. Nor is ofvwg for that matter.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#26 lore raw
In reply to a message not in the archive
Quoted 7 lines · Jason Gunthorpe

A bigger objection I have is that much of this new stuff is not adaquately documented.

Tag matching is a great example. It looks like this introduces many new headers into the on-wire protocol. Where are these headers documented? Where is the spec for the on-wire format? Where is the implementation for rxe?

Agreed on that part. For pon the wire changes we need the hardware standards organization to act. IBTA for IB and RoCE and IETF for iWarp, although not even having an active WG for iWarp there is a major problem at the moment.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw
In reply to a message not in the archive
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Jason Gunthorpe

A bigger objection I have is that much of this new stuff is not adaquately documented.

Tag matching is a great example. It looks like this introduces many new headers into the on-wire protocol. Where are these headers documented? Where is the spec for the on-wire format? Where is the implementation for rxe?

Quoted 5 lines

Agreed on that part. For pon the wire changes we need the hardware standards organization to act. IBTA for IB and RoCE and IETF for iWarp, although not even having an active WG for iWarp there is a major problem at the moment.

I think there are some fundamental questions here about what this is. As I understand it (and Mellanox can chime in if I'm wrong), tag matching already exists on the wire, it's just implemented above the level of the link layer transmissions. This enables a hardware accelerator for something that is already done at an upper layer, so the implementation is mostly defined by the defacto existing use in upper layer applications, not by the IBTA. I believe this could roughly be considered analogous to a network driver enabling a layer3 or layer4 accelerator. It's not a layer1 thing, so the IBTA is not appropriate to review it.

Signature · 2 lines
Doug Ledford <[email protected]>
    GPG Key ID: 0E572FDD
#30 lore raw
In reply to a message not in the archive
Quoted 16 lines · Doug Ledford
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Jason Gunthorpe

A bigger objection I have is that much of this new stuff is not adaquately documented.

Tag matching is a great example. It looks like this introduces many new headers into the on-wire protocol. Where are these headers documented? Where is the spec for the on-wire format? Where is the implementation for rxe?

Quoted 5 lines

Agreed on that part. For pon the wire changes we need the hardware standards organization to act. IBTA for IB and RoCE and IETF for iWarp, although not even having an active WG for iWarp there is a major problem at the moment.

Quoted 10 lines

I think there are some fundamental questions here about what this is. As I understand it (and Mellanox can chime in if I'm wrong), tag matching already exists on the wire, it's just implemented above the level of the link layer transmissions. This enables a hardware accelerator for something that is already done at an upper layer, so the implementation is mostly defined by the defacto existing use in upper layer applications, not by the IBTA. I believe this could roughly be considered analogous to a network driver enabling a layer3 or layer4 accelerator. It's not a layer1 thing, so the IBTA is not appropriate to review it.

Maybe, but that doesn't change my original point. This is clearly not defined well enough in the patches alone for any other vendor to implement it.

If we can't even reach a basic threshold of understanding it is pointless to expect other vendors to chime in on the design.

I'm starting to wonder if we should insist on an implementation of all this new stuff in rxe before accepting. At least then we have code as documentation ...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#31 lore raw
In reply to a message not in the archive
Quoted 8 lines

I think there are some fundamental questions here about what this is. As I understand it (and Mellanox can chime in if I'm wrong), tag matching already exists on the wire, it's just implemented above the level of the link layer transmissions. This enables a hardware accelerator for something that is already done at an upper layer, so the
implementation is mostly defined by the defacto existing use in upper layer applications, not by the IBTA. I believe this could roughly be

MPIs definitely make use of this sort of protocol, but it is an application level protocol that is non-standard, even between MPI implementations. There's no interoperability between them.

The Intel truescale and OPA implementations support tag matching, with HW offload support, but those too are proprietary (i.e. PSM). In the case of PSM, I believe the tag matching is part of the transport protocol. There's a decent chance the Mellanox implementation is the same.

Quoted 4 lines

considered analogous to a network driver enabling a layer3 or layer4 accelerator. It's not a layer1 thing, so the IBTA is not appropriate to
review it.

The IBTA defines layer1-4, plus the verbs interfaces. So, it _could_ be an IBTA issue. However, the IBTA allows for vendor specific features (e.g. PSM), so it could be a vendor feature.

IMO, this is a feature worth supporting. My concern is over how it is exposed. There's a lot of variety in how tag matching can be implemented.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#32 lore raw
In reply to a message not in the archive
Quoted 16 lines · Doug Ledford
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Jason Gunthorpe

A bigger objection I have is that much of this new stuff is not adaquately documented.

Tag matching is a great example. It looks like this introduces many new headers into the on-wire protocol. Where are these headers documented? Where is the spec for the on-wire format? Where is the implementation for rxe?

Quoted 5 lines

Agreed on that part. For pon the wire changes we need the hardware standards organization to act. IBTA for IB and RoCE and IETF for iWarp, although not even having an active WG for iWarp there is a major problem at the moment.

Quoted 10 lines

I think there are some fundamental questions here about what this is. As I understand it (and Mellanox can chime in if I'm wrong), tag matching already exists on the wire, it's just implemented above the level of the link layer transmissions. This enables a hardware accelerator for something that is already done at an upper layer, so the implementation is mostly defined by the defacto existing use in upper layer applications, not by the IBTA. I believe this could roughly be considered analogous to a network driver enabling a layer3 or layer4 accelerator. It's not a layer1 thing, so the IBTA is not appropriate to review it.

Yes, you are right. It is in use by most if not all major HPC players and in this patch set, we are providing possibility to configure it and expose supported flavours of tag matching, so they will be able to run RHEL out of the box. There are no HW limitations on how other vendors (let's pretend that they exist) will implement such feature.

As Sean mentioned in the other email, PSM has it and libfabric natively supports this functionality through MXM provider.

Regarding documentation, we are in continuous process of improvements for our submissions and will definitely put our attention to that point.

Thanks

Quoted 5 lines

--
Doug Ledford <[email protected]>
GPG Key ID: 0E572FDD