Support out of order data placement

Hi Doug,

This patch set which is based on v4.12-rc4 comes from Parav and it adds support for out of order data placement for RDMA read and write requests.

Thanks

---

In certain fabric configurations IB packets for a given QP may take up different paths in a network from source to destination. This results into reaching packets out of order at the receiver side. Instead of dropping packets, handling out of order packets can improve overall network performance in following ways:
* improve network utilization by avoiding retransmission
* avoiding latency increase due to retransmission

This patchset allows HCA to expose out of order data placement capability to users who can take benefit of such feature.

This is optional feature of an HCA and enablement of this feature is done on per QP basis. The optional QP attribute is set when a QP state is changed from INIT to RTR.

Out of order data placement capability indicates that if HCA receives out of order RDMA packets, their data placement can be done at the desired memory destination given in the packet(s). This is applicable to RDMA read and write operations.

Send queue work requests are still completed in-order regardless of their data placement order at requester or responder side.

In-order semantics is always guaranteed by setting the Fence indicator for appropriate WRs.

An application shall not depend on the contents of the RDMA write buffer at the responder until one of the following occurred:
- Completion of the RDMA WRITE with immediate data receive completion.
- Arrival and completion of the subsequent SEND message.
- Update of a memory element by subsequent ATOMIC operation.

An application shall not depend on the contents of the RDMA read buffer at the requester until one of the following occurred:
- Completion of the RDMA READ work request if requested or such work request completes with error status.
- Completion of the subsequent work request.

Thanks

Parav Pandit (3):
IB/core: Expose out of order data placement capability IB/uverbs: Enable user space programs to use out of order placement IB/mlx5: Support out of order data placement

 Documentation/infiniband/out_of_order.txt | 60 +++++++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_cmd.c      | 16 +++++++++
 drivers/infiniband/core/verbs.c           |  9 +++--
 drivers/infiniband/hw/mlx5/main.c         | 12 +++++++
 drivers/infiniband/hw/mlx5/qp.c           | 25 +++++++++++++
 include/linux/mlx5/mlx5_ifc.h             |  5 ++-
 include/rdma/ib_verbs.h                   | 22 ++++++++++++
 include/uapi/rdma/ib_user_verbs.h         | 16 +++++++--
 8 files changed, 159 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/infiniband/out_of_order.txt

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

From: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>

This patch exposes out of order data placement capability to enable HCA to report it.
It also defines optional QP attribute that can be set when moving QP from INIT to RTR state to make use of this feature on a particular QP.

---
 Documentation/infiniband/out_of_order.txt | 60 +++++++++++++++++++++++++++++++
 drivers/infiniband/core/verbs.c           |  9 +++--
 include/rdma/ib_verbs.h                   | 22 ++++++++++++
 3 files changed, 88 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/infiniband/out_of_order.txt

new file mode 100644
index 000000000000..9c585f6801fc
@@ -0,0 +1,60 @@
+Out of order data placement
+===========================
+
+This document describes out of order data placement feature and its
+user interface.
+
+1. Overview
+===========
+
+In certain fabric configurations IB packets for a given QP may take up
+different paths in a network from source to destination. This results
+into reaching packets out of order at the receiver side. Instead of
+dropping packets, handling out of order packets can improve overall
+network performance in following ways.
+(a) improve network utilization by avoiding retransmission
+(b) avoiding latency increase due to retransmission
+
+2. Description
+==============
+
+This is optional feature of an HCA.
+Enablement of this feature is done on per QP basis.
+This optional QP attribute is set when a QP state is changed from INIT
+to RTR.
+
+Out of order data placement capability indicates that if HCA receives
+out of order RDMA packets, their data placement can be done at the
+desired memory destination given in the packet(s). This is applicable
+to RDMA read and write operations.
+
+Send queue work requests are still completed in-order regardless of
+their data placement order at requester or responder side.
+
+In-order semantics is always guaranteed by setting the Fence
+indicator for appropriate WRs.
+
+An application shall not depend on the contents of the RDMA write buffer
+at the responder until one of the following occurred:
+- Completion of the RDMA WRITE with immediate data receive completion.
+- Arrival and completion of the subsequent SEND message.
+- Update of a memory element by subsequent ATOMIC operation.
+
+An application shall not depend on the contents of the RDMA read buffer
+at the requester until one of the following occurred:
+- Completion of the RDMA READ work request if requested or such
+  work request completes with error status.
+- Completion of the subsequent work request.
+
+3. Usage
+========
+
+(a) ibv_query_device_ex returns out of order data placement
+capability at ooo_caps structure for a given transport.
+Whenever HCA supports such capability for each transport user
+should get IB_OOO_RW_DATA_PLACEMENT set in the caps.
+
+(b) When such capability is set, user can set
+IB_QP_OOO_RW_DATA_PLACEMENT while invoking ibv_modify_qp().
+IB_QP_OOO_RW_DATA_PLACEMENT is optional attribute which is supported
+only when QP state transions from INIT to RTR state.

index 4792f5209ac2..7e761bedd1ad 100644
@@ -957,13 +957,16 @@ static const struct {
 						 IB_QP_PKEY_INDEX),
 				 [IB_QPT_RC]  = (IB_QP_ALT_PATH			|
 						 IB_QP_ACCESS_FLAGS		|
-						 IB_QP_PKEY_INDEX),
+						 IB_QP_PKEY_INDEX		|
+						 IB_QP_OOO_RW_DATA_PLACEMENT),
 				 [IB_QPT_XRC_INI] = (IB_QP_ALT_PATH		|
 						 IB_QP_ACCESS_FLAGS		|
-						 IB_QP_PKEY_INDEX),
+						 IB_QP_PKEY_INDEX		|
+						 IB_QP_OOO_RW_DATA_PLACEMENT),
 				 [IB_QPT_XRC_TGT] = (IB_QP_ALT_PATH		|
 						 IB_QP_ACCESS_FLAGS		|
-						 IB_QP_PKEY_INDEX),
+						 IB_QP_PKEY_INDEX		|
+						 IB_QP_OOO_RW_DATA_PLACEMENT),
 				 [IB_QPT_SMI] = (IB_QP_PKEY_INDEX		|
 						 IB_QP_QKEY),
 				 [IB_QPT_GSI] = (IB_QP_PKEY_INDEX		|

index ba8314ec5768..c507f2f0773a 100644
@@ -278,6 +278,26 @@ struct ib_rss_caps {
 	u32 max_rwq_indirection_table_size;
 };
 
+/*
+ * Out of order data placement capability bits.
+ * Out of order data placement means, if HCA receives RDMA packets in
+ * out of order manner, their data placement can be done at the desired
+ * memory destination given in the packet(s). This is applicable
+ * to RDMA read and write operations.
+ * Send queue work requests are still completed in-order regardless of their
+ * data placement order at local or remote end.
+ */
+enum ib_ooo_transport_caps {
+	IB_OOO_RW_DATA_PLACEMENT	= (1 << 0),
+};
+
+struct ib_ooo_caps {
+	u32 rc_caps;
+	u32 xrc_caps;
+	u32 ud_caps;
+	u32 uc_caps;
+};
+
 enum ib_cq_creation_flags {
 	IB_CQ_FLAGS_TIMESTAMP_COMPLETION   = 1 << 0,
 	IB_CQ_FLAGS_IGNORE_OVERRUN	   = 1 << 1,

@@ -338,6 +358,7 @@ struct ib_device_attr {
 	struct ib_rss_caps	rss_caps;
 	u32			max_wq_type_rq;
 	u32			raw_packet_caps; /* Use ib_raw_packet_caps enum */
+	struct ib_ooo_caps	ooo_caps;
 };
 
 enum ib_mtu {

@@ -1157,6 +1178,7 @@ enum ib_qp_attr_mask {
 	IB_QP_RESERVED3			= (1<<23),
 	IB_QP_RESERVED4			= (1<<24),
 	IB_QP_RATE_LIMIT		= (1<<25),
+	IB_QP_OOO_RW_DATA_PLACEMENT	= (1 << 26),
 };
 
 enum ib_qp_state {

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>

This patch enables user space programs to query out of order device capability.

---
 drivers/infiniband/core/uverbs_cmd.c | 16 ++++++++++++++++
 include/uapi/rdma/ib_user_verbs.h    | 16 ++++++++++++++--
 2 files changed, 30 insertions(+), 2 deletions(-)

index 70b7fb156414..122ba59c633b 100644
@@ -3766,6 +3766,15 @@ ssize_t ib_uverbs_destroy_srq(struct ib_uverbs_file *file,
 	return in_len;
 }
 
+static void copy_ooo_caps(struct ib_uverbs_ooo_caps *uverb_caps,
+			  struct ib_ooo_caps *attr_caps)
+{
+	uverb_caps->rc_caps = attr_caps->rc_caps;
+	uverb_caps->xrc_caps = attr_caps->xrc_caps;
+	uverb_caps->ud_caps = attr_caps->ud_caps;
+	uverb_caps->uc_caps = attr_caps->uc_caps;
+}
+
 int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 			      struct ib_device *ib_dev,
 			      struct ib_udata *ucore,

@@ -3854,6 +3863,13 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 
 	resp.raw_packet_caps = attr.raw_packet_caps;
 	resp.response_length += sizeof(resp.raw_packet_caps);
+
+	if (ucore->outlen < resp.response_length + sizeof(resp.ooo_caps))
+		goto end;
+
+	copy_ooo_caps(&resp.ooo_caps, &attr.ooo_caps);
+	resp.response_length += sizeof(resp.ooo_caps);
+
 end:
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	return err;

index 270c350bedc6..3519ff8e34cd 100644
@@ -236,6 +236,17 @@ struct ib_uverbs_rss_caps {
 	__u32 reserved;
 };
 
+struct ib_uverbs_ooo_caps {
+	/*
+	 * Per transport capability indicating whether out of order data
+	 * placement is supported or not.
+	 */
+	 __u32 rc_caps;
+	 __u32 xrc_caps;
+	 __u32 ud_caps;
+	 __u32 uc_caps;
+};
+
 struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_query_device_resp base;
 	__u32 comp_mask;

@@ -247,6 +258,7 @@ struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_rss_caps rss_caps;
 	__u32  max_wq_type_rq;
 	__u32 raw_packet_caps;
+	struct ib_uverbs_ooo_caps ooo_caps;
 };
 
 struct ib_uverbs_query_port {

@@ -555,9 +567,9 @@ enum {
 
 enum {
 	/*
-	 * This value is equal to IB_QP_RATE_LIMIT.
+	 * This value is equal to IB_QP_OOO_RW_DATA_PLACEMENT.
 	 */
-	IB_USER_LAST_QP_ATTR_MASK = 1ULL << 25,
+	IB_USER_LAST_QP_ATTR_MASK = 1ULL << 26,
 };
 
 struct ib_uverbs_ex_create_qp {

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>

This patch sets out of order data placement capability whenever device supports it. Currently its supports on RC and XRC QPs. It also extends the support to set such attribute for a QP during state transition.

---
 drivers/infiniband/hw/mlx5/main.c | 12 ++++++++++++
 drivers/infiniband/hw/mlx5/qp.c   | 25 +++++++++++++++++++++++++
 include/linux/mlx5/mlx5_ifc.h     |  5 ++++-
 3 files changed, 41 insertions(+), 1 deletion(-)

index 0c79983c8b1a..7f5284c08b8d 100644
@@ -572,6 +572,16 @@ static int mlx5_query_node_desc(struct mlx5_ib_dev *dev, char *node_desc)
 				    MLX5_REG_NODE_DESC, 0, 0);
 }
 
+static void mlx5_ib_fill_ooo_caps(struct mlx5_ib_dev *dev,
+				  struct ib_ooo_caps *caps)
+{
+	if (MLX5_CAP_GEN(dev->mdev, multi_path_rc_rdma))
+		caps->rc_caps |= IB_OOO_RW_DATA_PLACEMENT;
+
+	if (MLX5_CAP_GEN(dev->mdev, multi_path_xrc_rdma))
+		caps->xrc_caps |= IB_OOO_RW_DATA_PLACEMENT;
+}
+
 static int mlx5_ib_query_device(struct ib_device *ibdev,
 				struct ib_device_attr *props,
 				struct ib_udata *uhw)

@@ -765,6 +775,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 			1 << MLX5_CAP_GEN(dev->mdev, log_max_rq);
 	}
 
+	mlx5_ib_fill_ooo_caps(dev, &props->ooo_caps);
+
 	if (field_avail(typeof(resp), cqe_comp_caps, uhw->outlen)) {
 		resp.cqe_comp_caps.max_num =
 			MLX5_CAP_GEN(dev->mdev, cqe_compression) ?

index ebb6768684de..bbe616e373b3 100644
@@ -54,6 +54,10 @@ enum {
 	MLX5_IB_SQ_STRIDE	= 6,
 };
 
+enum {
+	MLX5_QP_OOO_ATTR	= 25,
+};
+
 static const u32 mlx5_ib_opcode[] = {
 	[IB_WR_SEND]				= MLX5_OPCODE_SEND,
 	[IB_WR_LSO]				= MLX5_OPCODE_LSO,

@@ -2810,6 +2814,27 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
 	if (qp->flags & MLX5_IB_QP_SQPN_QP1)
 		context->deth_sqpn = cpu_to_be32(1);
 
+	if (attr_mask & IB_QP_OOO_RW_DATA_PLACEMENT) {
+		if (ibqp->qp_type == IB_QPT_RC) {
+			if (MLX5_CAP_GEN(dev->mdev, multi_path_rc_rdma)) {
+				context->flags_pd |=
+					cpu_to_be32(1 << MLX5_QP_OOO_ATTR);
+			} else {
+				err = -EINVAL;
+				goto out;
+			}
+		} else if (ibqp->qp_type == IB_QPT_XRC_INI ||
+			   ibqp->qp_type == IB_QPT_XRC_TGT) {
+			if (MLX5_CAP_GEN(dev->mdev, multi_path_xrc_rdma)) {
+				context->flags_pd |=
+					cpu_to_be32(1 << MLX5_QP_OOO_ATTR);
+			} else {
+				err = -EINVAL;
+				goto out;
+			}
+		}
+	}
+
 	mlx5_cur = to_mlx5_state(cur_state);
 	mlx5_new = to_mlx5_state(new_state);
 	mlx5_st = to_mlx5_st(ibqp->qp_type);

index edafedb7b509..3d1e77e97552 100644
@@ -856,7 +856,10 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         pps[0x1];
 	u8         pps_modify[0x1];
 	u8         log_max_msg[0x5];
-	u8         reserved_at_1c8[0x4];
+	u8         multi_path_xrc_rdma[0x1];
+	u8         reserved_at_1c9[0x1];
+	u8         multi_path_rc_rdma[0x1];
+	u8         reserved_at_1cb[0x1];
 	u8         max_tc[0x4];
 	u8         reserved_at_1d0[0x1];
 	u8         dcbx[0x1];

2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hello Leon and Parav,

Since PCIe writes can be executed out of order, shouldn't that be mentioned in Documentation/infiniband/out_of_order.txt? See also the documentation of the Device Control Register and the Enable Relaxed
Ordering bit in the PCIe
spec.

Additionally, since not handling out-of-order RDMA writes correctly is an ULP bug and since there are more ULPs that handle out-of-order writes correctly than ULPs that don't handle out-of-order writes correctly, if a new flag is introduced, shouldn't that be a flag to disable out-of-order writes?

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Bart,

There is no change in the way PCIe writes are done with respect to this per QP bit.
Meaning, if this bit is cleared, PCIe subsystem can still do out of order writes depending on relaxed ordering flag.

Not sure I understood correctly. This bit is not a bug fix for ULPs who don't handle out-of-order writes.
As I described in Documentation,
" Out of order data placement capability indicates that if HCA receives out of order RDMA packets, their data placement can be done at the desired memory destination given in the packet(s). This is applicable to RDMA read and write operations." This flag indicates that - in above condition, HCA can do data placement out-of-order.
Without enabling this flag, when HCA receives out of order packets, it would drop them due to PSN sequence error.

So, - to your question - shouldn't that be a flag to disable out-of-order writes? By default, its disabled at RDMA level.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Parav,

That's why I asked to mention PCIe write reordering in out_of_order.txt. Someone who is reading that document could be mislead to assume that if the HCA does not reorder writes that no reordering will occur.

I don't think that your last two paragraphs mention anything that had not yet been mentioned in the four e-mails constituting your patch series. Additionally, I think my question was clear and unambiguous. So please reread my question.

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Bart,

Make sense. I will update the documentation to describe that PCIe out-of-order writes can still happen.

I elaborated two points in email in last two paragraphs to answer your question.
(a) Without enabling this out-of-order flag, when HCA receives out of order packets, it would drop them due to PSN sequence error.
(b) by default out-of-order is disabled at RDMA level.

I reread your question. Let me try to answer again.

No. There is no need for such flag in context of this patchset because by default out-of-order writes are disabled at RDMA level.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Bart's point is that it is not disabled, by default, it is platform specific by default.

Your patch makes makes it more likely a ULP will see out of order writes, but it is already the case that any ULP that relies on this is platform specific and/or broken.

I think the answer to Bart's question is that some MPI ULPs are 'broken' and only work on x86 - setting this bit by default would cause them to stop working. It is unfortunate this is the rare case.

I would suggest at least using the inverted sense like Bart describes in the kernel - every kernel ULP is safe.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

There are two bits as described in usage section of Documentation.
(a) device capability bit, which indicates that device is capable of processing out-of-order packets. This doesn't mean it is enabled.
(b) per QP enable bit where application can enable it if it wants to.

So no bit is set by default.
So application which sets this per QP bit, and still have some logic for polling on write data, would be wrong/bug for that application.

I don't see a need to use inverted sense in code. I can surely make documentation more descriptive as Bart suggested.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

If this is 'better' then it should be on as much as possible, and I certianly don't want to see kernel ULPs query caps and other pointless things when they already, necessarily, deal with out of order.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Sure. Kernel ULPs and any other user ULPs can skip query caps.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

No, they can't because only mlx5 accepts the new flag.

This is why inverting the flag in the kernel makes much more sense.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

As you know - capable doesn't mean enabled in query_cap.

By default out-of-order processing of RDMA write packets is disabled even on mlx5 for all QPs.
When user requests it to enable during modify_qp() as described in Documentation/out_of_order.txt usage section 3, HCA will process it that way.

When you say invert the flag, which flag do you mean?
Capability flag or QP attribute?
Capability flag indicates - whether out-of-order processing at RDMA level supported or not. QP attribute indicates - to enable such ooo processing for a particular QP.

Let's not mix PCIe write ordering with RDMA level out-of-order processing.
IB spec clearly says, as I reiterated in the Documentation/out_of_order.txt - that application should not depend on the write data until write_with_imm arrives or subsequent rdma send arrives at responder side. If some MPI or other application have taken shortcut based on some platform, they have to help themselves. This path is not intent to address such issue.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

QP attribute.

Exactly - all spec conformant ULPs are compatible with enabling this new function of mlx5.

There is no reason to add a flag at all, except to retain compatibility with certain non-confomant ULPs. We have none of those in the kernel, so kernel QPs should always have the flag on.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

This per QP attribute is for read and write both. So responder can receive out-of-order read responses. And HCA's QP need to be told to accept it that way, which by default doesn't.

ULP can be conformant, but it may not be the same HCA on both requester and responder side which has this feature. So requester or responder cannot assume what other side have enabled.
Therefore the need to be explicit enable flag.

As I explained, by default HCA doesn't process out-of-order packets. User need to tell him to do so.
For example adapter is capable of 16 incoming read requests, but when modify_qp() happens, user need to tell how many incoming outstanding read requests it should accept. Same for outgoing read requests. Etc.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I think this one flag is conflating too many things then.

Obviously sending out of order packets is not spec conformant, and I don't think your discussion is clear enough, as this change to send side certainly was not clear to me.

I'm not excited about a new end-to-end flag without some kind of negotiation scheme, and I'm not excited about this being in any of the common APIs.

Perhaps it should be in libmlx5 instead.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Too many flags just confuses end-user for read/write/requester/responder etc. Also there isn't well established use case either where user wants to do only certain things.

Ok. I can make Documentation more elaborate that describe summary of this discussion + Bart's point.

Many applications do out-of-band negotiation or static configuration and its unrelated to the patchset. So lack of negotiation shouldn't be issue. Its outside the scope of this verb extension anyway.

We think that its best usable through ibv_modify_qp because it's really simple enough and well documented which will address inputs given in this discussion.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Your original discussion said this optimized cases on Rx when the fabric went out of order, eg repath, etc. That seems well defined enough to be a dedicated flag, default to on, etc.

Sending packets out of order is an entirely different beast, and sounds like it would also require additional configuration, as IBA doesn't even have an existing way to describe a set of paths to choose between.

How are the TX paths choosen? That need to be documented before this could even be considered for the common API.

This is why I think one flag is a bad idea.

That isn't really the criteria to be in the common API now that we have the libmlx5 approach.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Absolutely. All iWARP adapters support multiple RDMA Writes in flight, and their placement is not ordered. It's a quirk of the MLX devices, and in turn their implementation of the IB protocol, that they place writes in-order. I'm happy to see Mellanox become aware of this, it limits write performance on networks with any significant latencies.

I agree with Jason, the bit should be 1 by default, if defined as you propose. Out-of-order is the norm, not the exception, for ULPs. Honestly, I think you should perhaps consider making it the default on your devices, and allowing only MLX-aware ULPs to turn it off.

The Windows NDK API has a similar concept, by the way, it flips your meaning and defaults to 0. Also, it's not settable by the ULP. The flag isn't used for much, as a result.

NDK_ADAPTER_FLAG_IN_ORDER_DMA_SUPPORTED

https://msdn.microsoft.com/en-us/library/windows/hardware/hh439851(v=vs.85).aspx

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxw ZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3 IDQ6MDkgUE0NCj4gVG86IEphc29uIEd1bnRob3JwZSA8amd1bnRob3JwZUBvYnNpZGlhbnJlc2Vh cmNoLmNvbT47IFBhcmF2IFBhbmRpdA0KPiA8cGFyYXZAbWVsbGFub3guY29tPg0KPiBDYzogQmFy dCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9y ZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElk YW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUmU6IFtQQVRD SCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiAN Cj4gT24gNi8xMi8yMDE3IDEyOjU1IFBNLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6DQo+ID4gT24g TW9uLCBKdW4gMTIsIDIwMTcgYXQgMDQ6NTM6MjhQTSArMDAwMCwgUGFyYXYgUGFuZGl0IHdyb3Rl Og0KPiA+Pg0KPiA+Pg0KPiA+Pj4gRnJvbTogSmFzb24gR3VudGhvcnBlIFttYWlsdG86amd1bnRo b3JwZUBvYnNpZGlhbnJlc2VhcmNoLmNvbV0NCj4gPj4+IFNlbnQ6IE1vbmRheSwgSnVuZSAxMiwg MjAxNyAxMTo0NCBBTQ0KPiA+Pj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29t Pg0KPiA+Pj4gQ2M6IEJhcnQgVmFuIEFzc2NoZSA8QmFydC5WYW5Bc3NjaGVAc2FuZGlzay5jb20+ OyBsZW9uQGtlcm5lbC5vcmc7DQo+ID4+PiBkbGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1h QHZnZXIua2VybmVsLm9yZzsgSWRhbiBCdXJzdGVpbg0KPiA+Pj4gPGlkYW5iQG1lbGxhbm94LmNv bT4NCj4gPj4+IFN1YmplY3Q6IFJlOiBbUEFUQ0ggcmRtYS1uZXh0IDAvM10gU3VwcG9ydCBvdXQg b2Ygb3JkZXIgZGF0YQ0KPiA+Pj4gcGxhY2VtZW50DQo+ID4+Pg0KPiA+Pj4gT24gTW9uLCBKdW4g MTIsIDIwMTcgYXQgMDQ6NDI6MjdQTSArMDAwMCwgUGFyYXYgUGFuZGl0IHdyb3RlOg0KPiA+Pj4N Cj4gPj4+Pj4gSSB3b3VsZCBzdWdnZXN0IGF0IGxlYXN0IHVzaW5nIHRoZSBpbnZlcnRlZCBzZW5z ZSBsaWtlIEJhcnQNCj4gPj4+Pj4gZGVzY3JpYmVzIGluIHRoZSBrZXJuZWwgLSBldmVyeSBrZXJu ZWwgVUxQIGlzIHNhZmUuDQo+ID4+Pg0KPiA+Pj4+IEkgZG9uJ3Qgc2VlIGEgbmVlZCB0byB1c2Ug aW52ZXJ0ZWQgc2Vuc2UgaW4gY29kZS4gSSBjYW4gc3VyZWx5IG1ha2UNCj4gPj4+IGRvY3VtZW50 YXRpb24gbW9yZSBkZXNjcmlwdGl2ZSBhcyBCYXJ0IHN1Z2dlc3RlZC4NCj4gPj4+DQo+ID4+PiBJ ZiB0aGlzIGlzICdiZXR0ZXInIHRoZW4gaXQgc2hvdWxkIGJlIG9uIGFzIG11Y2ggYXMgcG9zc2li bGUsIGFuZCBJDQo+ID4+PiBjZXJ0aWFubHkgZG9uJ3Qgd2FudCB0byBzZWUga2VybmVsIFVMUHMg cXVlcnkgY2FwcyBhbmQgb3RoZXINCj4gPj4+IHBvaW50bGVzcyB0aGluZ3Mgd2hlbiB0aGV5IGFs cmVhZHksIG5lY2Vzc2FyaWx5LCBkZWFsIHdpdGggb3V0IG9mIG9yZGVyLg0KPiA+Pg0KPiA+PiBT dXJlLiBLZXJuZWwgVUxQcyBhbmQgYW55IG90aGVyIHVzZXIgVUxQcyBjYW4gc2tpcCBxdWVyeSBj YXBzLg0KPiA+DQo+ID4gTm8sIHRoZXkgY2FuJ3QgYmVjYXVzZSBvbmx5IG1seDUgYWNjZXB0cyB0 aGUgbmV3IGZsYWcuDQo+ID4NCj4gPiBUaGlzIGlzIHdoeSBpbnZlcnRpbmcgdGhlIGZsYWcgaW4g dGhlIGtlcm5lbCBtYWtlcyBtdWNoIG1vcmUgc2Vuc2UuDQo+IA0KPiBBYnNvbHV0ZWx5LiBBbGwg aVdBUlAgYWRhcHRlcnMgc3VwcG9ydCBtdWx0aXBsZSBSRE1BIFdyaXRlcyBpbiBmbGlnaHQsIGFu ZA0KPiB0aGVpciBwbGFjZW1lbnQgaXMgbm90IG9yZGVyZWQuIEl0J3MgYSBxdWlyayBvZiB0aGUg TUxYIGRldmljZXMsIGFuZCBpbiB0dXJuDQo+IHRoZWlyIGltcGxlbWVudGF0aW9uIG9mIHRoZSBJ QiBwcm90b2NvbCwgdGhhdCB0aGV5IHBsYWNlIHdyaXRlcyBpbi1vcmRlci4gSSdtDQo+IGhhcHB5 IHRvIHNlZSBNZWxsYW5veCBiZWNvbWUgYXdhcmUgb2YgdGhpcywgaXQgbGltaXRzIHdyaXRlIHBl cmZvcm1hbmNlIG9uDQo+IG5ldHdvcmtzIHdpdGggYW55IHNpZ25pZmljYW50IGxhdGVuY2llcy4N Cj4gDQo+IEkgYWdyZWUgd2l0aCBKYXNvbiwgdGhlIGJpdCBzaG91bGQgYmUgMSBieSBkZWZhdWx0 LCBpZiBkZWZpbmVkIGFzIHlvdSBwcm9wb3NlLg0KPiBPdXQtb2Ytb3JkZXIgaXMgdGhlIG5vcm0s IG5vdCB0aGUgZXhjZXB0aW9uLCBmb3IgVUxQcy4NCj4gSG9uZXN0bHksIEkgdGhpbmsgeW91IHNo b3VsZCBwZXJoYXBzIGNvbnNpZGVyIG1ha2luZyBpdCB0aGUgZGVmYXVsdCBvbiB5b3VyDQo+IGRl dmljZXMsIGFuZCBhbGxvd2luZyBvbmx5IE1MWC1hd2FyZSBVTFBzIHRvIHR1cm4gaXQgb2ZmLg0K PiANCg0KVGhlcmUgY2FuIGJlIGNhc2VzIGluIGRlcGxveW1lbnQgd2hlcmUgcmVzcG9uZGVyIGhh cyBzdXBwb3J0IGZvciByZWNlaXZpbmcgb3V0LW9mLW9yZGVyLCBidXQgcmVxdWVzdGVyIGRvZXNu J3QuDQpSZWFkIHJlc3BvbnNlcyBhcnJpdmluZyBvdXQgb2Ygb3JkZXIgZm9yIGRpZmZlcmVudCBt ZXNzYWdlcyBhdCByZXF1ZXN0ZXIgc2lkZSB3b3VsZCBicmVhay4NCkluIG90aGVyIGNhc2UgaWYg aXRzIHN1cHBvcnRlZCBvbiByZXF1ZXN0ZXIgc2lkZSwgYnV0IG5vdCBzdXBwb3J0ZWQgb24gcmVz cG9uZGVyIHNpZGUsIHdvdWxkIHJlc3VsdCBpbnRvIG1vcmUgcmV0cmFuc21pc3Npb24gYmVjYXVz ZSByZXNwb25kZXIgY2Fubm90IGhhbmRsZSB0aGVtIGFuZCB0cmlnZ2VycyBQU04gc2VxdWVuY2Ug ZXJyb3IuDQpSZXF1ZXN0ZXIgYW5kIHJlc3BvbmRlciBib3RoIG5lZWRzIHRvIGtub3cgd2hlbiB0 aGV5IGVuYWJsZSBvdXQtb2Ytb3JkZXIgb24gYSBnaXZlbiBRUC4NClNvIHdlIHNob3VsZG7igJl0 IGVuYWJsZSBpdCBieSBkZWZhdWx0IG9uIGFsbCBRUHMuDQoNCj4gVGhlIFdpbmRvd3MgTkRLIEFQ SSBoYXMgYSBzaW1pbGFyIGNvbmNlcHQsIGJ5IHRoZSB3YXksIGl0IGZsaXBzIHlvdXINCj4gbWVh bmluZyBhbmQgZGVmYXVsdHMgdG8gMC4gQWxzbywgaXQncyBub3Qgc2V0dGFibGUgYnkgdGhlIFVM UC4gVGhlIGZsYWcgaXNuJ3QNCj4gdXNlZCBmb3IgbXVjaCwgYXMgYSByZXN1bHQuDQo+IA0KPiBO REtfQURBUFRFUl9GTEFHX0lOX09SREVSX0RNQV9TVVBQT1JURUQNCj4gDQo+IGh0dHBzOi8vbXNk bi5taWNyb3NvZnQuY29tL2VuLQ0KPiB1cy9saWJyYXJ5L3dpbmRvd3MvaGFyZHdhcmUvaGg0Mzk4 NTEodj12cy44NSkuYXNweA0KPiANCj4gDQo+IFRvbS4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

You still haven't explained at all what the transmitter side does differently..

Is this actually some kind of selective retransmit scheme?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Let's say
Read-1
Read-2 are issued by requester.

For some reason read response for each read arrived via different path to requester,
If ooo is enabled on a QP they will be processed correctly. Else read responses for 2nd read could be dropped. Here transmitter of read responder is different when ooo is enabled.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

This is still only discussing the reciever side of a QP.

And I keep asking you to explain what is different about the transmitter side.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Above sequence explains read_response_transmitter behavior.
I will check with hardware team if there is anything done differently at write_transmitter (requester) other than what I explained above.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

but that has no impact on the other side of the QP - it does not care if they were processed successefully or a NACK is sent and retransmit is needed.

Still not seeing any explanation for why negotiation is required, and I'm getting sick of asking, please craft a response with your internal team.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

...

Yuck! So this needs to be negotiated end-to-end, and by the upper layer? Talk about barriers to adoption, and opportunities for disaster.

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxw ZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3 IDU6MjAgUE0NCj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFzb24g R3VudGhvcnBlDQo+IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiBDYzogQmFy dCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9y ZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElk YW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUmU6IFtQQVRD SCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiAN Cj4gT24gNi8xMi8yMDE3IDU6MzIgUE0sIFBhcmF2IFBhbmRpdCB3cm90ZToNCj4gPiBIaSBUb20s DQo+IC4uLg0KPiA+Pg0KPiA+PiBJIGFncmVlIHdpdGggSmFzb24sIHRoZSBiaXQgc2hvdWxkIGJl IDEgYnkgZGVmYXVsdCwgaWYgZGVmaW5lZCBhcyB5b3UNCj4gcHJvcG9zZS4NCj4gPj4gT3V0LW9m LW9yZGVyIGlzIHRoZSBub3JtLCBub3QgdGhlIGV4Y2VwdGlvbiwgZm9yIFVMUHMuDQo+ID4+IEhv bmVzdGx5LCBJIHRoaW5rIHlvdSBzaG91bGQgcGVyaGFwcyBjb25zaWRlciBtYWtpbmcgaXQgdGhl IGRlZmF1bHQNCj4gPj4gb24geW91ciBkZXZpY2VzLCBhbmQgYWxsb3dpbmcgb25seSBNTFgtYXdh cmUgVUxQcyB0byB0dXJuIGl0IG9mZi4NCj4gPj4NCj4gPg0KPiA+IFRoZXJlIGNhbiBiZSBjYXNl cyBpbiBkZXBsb3ltZW50IHdoZXJlIHJlc3BvbmRlciBoYXMgc3VwcG9ydCBmb3INCj4gcmVjZWl2 aW5nIG91dC1vZi1vcmRlciwgYnV0IHJlcXVlc3RlciBkb2Vzbid0Lg0KPiANCj4gWXVjayEgU28g dGhpcyBuZWVkcyB0byBiZSBuZWdvdGlhdGVkIGVuZC10by1lbmQsIGFuZCBieSB0aGUgdXBwZXIg bGF5ZXI/DQo+IFRhbGsgYWJvdXQgYmFycmllcnMgdG8gYWRvcHRpb24sIGFuZCBvcHBvcnR1bml0 aWVzIGZvciBkaXNhc3Rlci4NCj4gDQpBcyBKYXNvbiBjb25maXJtZWQgdGhhdCBhbGwgTGludXgg a2VybmVsIGNvbnN1bWVycyBhcmUgY29kZWQgdG8gYmUgY29tcGxpYW50IHRvIG85LTIwIHJlcXVp cmVtZW50LA0KU28gSSB0aGluayBrZXJuZWwgYmFzZWQgcmRtYS1jbSBjb25zdW1lcnMgY2FuIGJl IHRyYW5zcGFyZW50bHkgZW5hYmxlZCBlbmQtdG8tZW5kIHdpdGhvdXQgVUxQJ3MgaW52b2x2ZW1l bnQgd2l0aCByZG1hX2FjY2VwdCgpIGFuZCByZG1hX2Nvbm5lY3QoKS4NClRoYXQgd291bGQgYmUg c2VwYXJhdGUgcGF0Y2hzZXQgZm9yIGl0IHdoZW5ldmVyIHJkbWFjbSBpcyBleHRlbmRlZCBmb3Ig bmVnb3RpYXRpb24sIGFmdGVyIHRlc3RpbmcgdGhlbS4NCldlIGhhdmVuJ3QgcGxhbm5lZCB0aGF0 IHlldC4NCg0KPiBUb20uDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I have two thoughts here.

1) You seem to assume all consumers are Linux, and do not need to negotiate. This is a dangerous assumption.

2) I assume that there is some performance benefit to toggling this setting to non-strict. So, how do existing consumers get this advantage, especially since they don't need strict semantics? Bearing in mind that they do have to negotiate this end-to-end, meaning they require a protocol extension.

Actually. I have a third thought. Since this is an attribute to qp creation, performed even before establishing a connection, how does the upper layer know when to set it?

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogVG9tIFRh bHBleSBbbWFpbHRvOnRvbUB0YWxwZXkuY29tXQ0KPiBTZW50OiBNb25kYXksIEp1bmUgMTIsIDIw MTcgNjo0NCBQTQ0KPiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5jb20+OyBKYXNv biBHdW50aG9ycGUNCj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5jb20+DQo+IENjOiBC YXJ0IFZhbiBBc3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29tPjsgbGVvbkBrZXJuZWwu b3JnOw0KPiBkbGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1hQHZnZXIua2VybmVsLm9yZzsg SWRhbiBCdXJzdGVpbg0KPiA8aWRhbmJAbWVsbGFub3guY29tPg0KPiBTdWJqZWN0OiBSZTogW1BB VENIIHJkbWEtbmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9mIG9yZGVyIGRhdGEgcGxhY2VtZW50DQo+ IA0KPiBPbiA2LzEyLzIwMTcgNjo1NCBQTSwgUGFyYXYgUGFuZGl0IHdyb3RlOg0KPiA+IEhpIFRv bSwNCj4gPg0KPiA+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+PiBGcm9tOiBUb20g VGFscGV5IFttYWlsdG86dG9tQHRhbHBleS5jb21dDQo+ID4+IFNlbnQ6IE1vbmRheSwgSnVuZSAx MiwgMjAxNyA1OjIwIFBNDQo+ID4+IFRvOiBQYXJhdiBQYW5kaXQgPHBhcmF2QG1lbGxhbm94LmNv bT47IEphc29uIEd1bnRob3JwZQ0KPiA+PiA8amd1bnRob3JwZUBvYnNpZGlhbnJlc2VhcmNoLmNv bT4NCj4gPj4gQ2M6IEJhcnQgVmFuIEFzc2NoZSA8QmFydC5WYW5Bc3NjaGVAc2FuZGlzay5jb20+ OyBsZW9uQGtlcm5lbC5vcmc7DQo+ID4+IGRsZWRmb3JkQHJlZGhhdC5jb207IGxpbnV4LXJkbWFA dmdlci5rZXJuZWwub3JnOyBJZGFuIEJ1cnN0ZWluDQo+ID4+IDxpZGFuYkBtZWxsYW5veC5jb20+ DQo+ID4+IFN1YmplY3Q6IFJlOiBbUEFUQ0ggcmRtYS1uZXh0IDAvM10gU3VwcG9ydCBvdXQgb2Yg b3JkZXIgZGF0YQ0KPiA+PiBwbGFjZW1lbnQNCj4gPj4NCj4gPj4gT24gNi8xMi8yMDE3IDU6MzIg UE0sIFBhcmF2IFBhbmRpdCB3cm90ZToNCj4gPj4+IEhpIFRvbSwNCj4gPj4gLi4uDQo+ID4+Pj4N Cj4gPj4+PiBJIGFncmVlIHdpdGggSmFzb24sIHRoZSBiaXQgc2hvdWxkIGJlIDEgYnkgZGVmYXVs dCwgaWYgZGVmaW5lZCBhcw0KPiA+Pj4+IHlvdQ0KPiA+PiBwcm9wb3NlLg0KPiA+Pj4+IE91dC1v Zi1vcmRlciBpcyB0aGUgbm9ybSwgbm90IHRoZSBleGNlcHRpb24sIGZvciBVTFBzLg0KPiA+Pj4+ IEhvbmVzdGx5LCBJIHRoaW5rIHlvdSBzaG91bGQgcGVyaGFwcyBjb25zaWRlciBtYWtpbmcgaXQg dGhlIGRlZmF1bHQNCj4gPj4+PiBvbiB5b3VyIGRldmljZXMsIGFuZCBhbGxvd2luZyBvbmx5IE1M WC1hd2FyZSBVTFBzIHRvIHR1cm4gaXQgb2ZmLg0KPiA+Pj4+DQo+ID4+Pg0KPiA+Pj4gVGhlcmUg Y2FuIGJlIGNhc2VzIGluIGRlcGxveW1lbnQgd2hlcmUgcmVzcG9uZGVyIGhhcyBzdXBwb3J0IGZv cg0KPiA+PiByZWNlaXZpbmcgb3V0LW9mLW9yZGVyLCBidXQgcmVxdWVzdGVyIGRvZXNuJ3QuDQo+ ID4+DQo+ID4+IFl1Y2shIFNvIHRoaXMgbmVlZHMgdG8gYmUgbmVnb3RpYXRlZCBlbmQtdG8tZW5k LCBhbmQgYnkgdGhlIHVwcGVyIGxheWVyPw0KPiA+PiBUYWxrIGFib3V0IGJhcnJpZXJzIHRvIGFk b3B0aW9uLCBhbmQgb3Bwb3J0dW5pdGllcyBmb3IgZGlzYXN0ZXIuDQo+ID4+DQo+ID4gQXMgSmFz b24gY29uZmlybWVkIHRoYXQgYWxsIExpbnV4IGtlcm5lbCBjb25zdW1lcnMgYXJlIGNvZGVkIHRv IGJlDQo+ID4gY29tcGxpYW50IHRvIG85LTIwIHJlcXVpcmVtZW50LCBTbyBJIHRoaW5rIGtlcm5l bCBiYXNlZCByZG1hLWNtDQo+IGNvbnN1bWVycyBjYW4gYmUgdHJhbnNwYXJlbnRseSBlbmFibGVk IGVuZC10by1lbmQgd2l0aG91dCBVTFAncw0KPiBpbnZvbHZlbWVudCB3aXRoIHJkbWFfYWNjZXB0 KCkgYW5kIHJkbWFfY29ubmVjdCgpLg0KPiANCj4gSSBoYXZlIHR3byB0aG91Z2h0cyBoZXJlLg0K PiANCj4gMSkgWW91IHNlZW0gdG8gYXNzdW1lIGFsbCBjb25zdW1lcnMgYXJlIExpbnV4LCBhbmQg ZG8gbm90IG5lZWQgdG8NCj4gbmVnb3RpYXRlLiBUaGlzIGlzIGEgZGFuZ2Vyb3VzIGFzc3VtcHRp b24uDQpDZXJ0YWlubHkgbm90LiBJIGRpZG4ndCBhc3N1bWUgdGhhdC4gSSBqdXN0IGdhdmUgb25l IGV4YW1wbGUgdGhhdCBrbm93biBjb25zdW1lcnMgY2FuIGJlIGRvbmUgd2l0aG91dCBtb2RpZnlp bmcgdGhlIFVMUC4NCkV4cGxhaW5lZCBmdXJ0aGVyIGluIDNyZCBxdWVzdGlvbi4NCkV2ZW4gb3Ro ZXIgY29uc3VtZXJzIGNhbiB3b3JrIHdpdGggdGhpcyBzb2x1dGlvbi4gDQpGb3IgZXhhbXBsZSBM aW51eCByZG1hY20gYmFzZWQgY2xpZW50IGFuZCBPdGhlciBPUyBiYXNlZCBzZXJ2ZXIuDQpDbGll bnQgaXMgb29vIGNhcGFibGUuDQpTZXJ2ZXIgaXMgb29vIG5vdCBjYXBhYmxlLg0KT25jZSB5b3Ug Zm9sbG93IGJlbG93IHJkbWFjbSBiYXNlZCBzZXF1ZW5jZSwgaXQgd2lsbCBiZSBjbGVhciBob3cg dGhpcyB3aWxsIHdvcmtzLg0KDQo+IA0KPiAyKSBJIGFzc3VtZSB0aGF0IHRoZXJlIGlzIHNvbWUg cGVyZm9ybWFuY2UgYmVuZWZpdCB0byB0b2dnbGluZyB0aGlzIHNldHRpbmcNCj4gdG8gbm9uLXN0 cmljdC4gU28sIGhvdyBkbyBleGlzdGluZyBjb25zdW1lcnMgZ2V0IHRoaXMgYWR2YW50YWdlLCBl c3BlY2lhbGx5DQo+IHNpbmNlIHRoZXkgZG9uJ3QgbmVlZCBzdHJpY3Qgc2VtYW50aWNzPyBCZWFy aW5nIGluIG1pbmQgdGhhdCB0aGV5IGRvIGhhdmUNCj4gdG8gbmVnb3RpYXRlIHRoaXMgZW5kLXRv LWVuZCwgbWVhbmluZyB0aGV5IHJlcXVpcmUgYSBwcm90b2NvbCBleHRlbnNpb24uDQpJIGRvbid0 IGhhdmUgY29tcGxldGVseSB0cmFuc3BhcmVudCB1cHN0cmVhbSBzb2x1dGlvbiBmb3IgZXhpc3Rp bmcgY29uc3VtZXJzIHlldC4NCj4gDQo+IEFjdHVhbGx5LiBJIGhhdmUgYSB0aGlyZCB0aG91Z2h0 LiBTaW5jZSB0aGlzIGlzIGFuIGF0dHJpYnV0ZSB0byBxcCBjcmVhdGlvbiwNCj4gcGVyZm9ybWVk IGV2ZW4gYmVmb3JlIGVzdGFibGlzaGluZyBhIGNvbm5lY3Rpb24sIGhvdyBkb2VzIHRoZSB1cHBl ciBsYXllcg0KPiBrbm93IHdoZW4gdG8gc2V0IGl0Pw0KVGhpcyBpcyBub3QgYXQgUVAgY3JlYXRp b24gdGltZS4gSSBoYXZlIGRlc2NyaWJlZCBpbiBEb2N1bWVudGF0aW9uL291dF9vZl9vcmRlci50 eHQgaW4gdXNhZ2Ugc2VjdGlvbiAzLg0KVGhpcyBpcyBhdCBRUCBzdGF0ZSB0cmFuc2l0aW9uIGZy b20gSU5JVCB0byBSVFIuDQpIZXJlIGlzIHRoZSBmbG93LiBJdCdzIGp1c3Qgbm90IGNvZGVkIGVu b3VnaCBmb3IgcG9zdGluZyBwYXRjaGVzLg0KDQoxLiBXaGVuIHJkbWFjbSBhY3RpdmUgc2lkZSBj cmVhdGVzIHRoZSBRUCwgSXQgaXMgSU5JVCBzdGF0ZS4NCjIuIFNlbmQgTUFEX1JlcSBtc2cgKGlu ZGljYXRpbmcgb29vX3JlcXVlc3RlZD0xKQ0KMy4gV2hlbiByZG1hY20gcGFzc2l2ZSBzaWRlIHJl Y2VpdmVzIHRoZSBtZXNzYWdlLCBpdCBsb29rcyB1cCBkZXZpY2VfY2FwIGF0dHJpYnV0ZSBhbmQg bWF0Y2hlcyBpdCBhZ2FpbnN0IG9vb19yZXF1ZXN0ZWQgZmxhZy4NCjQuIHdoZW4gZGV2aWNlIHN1 cHBvcnRzIGl0LCBNQURfUnNwIG1zZyBzZXRzIG9vb19lbmFibGVkPTEsIGlmIGl0IGRvZXNuJ3Qg c3VwcG9ydCBpdCwgb29vX2VuYWJsZWQ9MA0KNS4gcmRtYWNtIHBhc3NpdmUgc2lkZSBjcmVhdGVz IHRoZSBRUCBhbmQgbW92ZXMgdG8gUlRSIHN0YXRlICh3aXRoIFFQIG9vbyBlbmFibGVkIGJpdCBz ZXQpLg0KNi4gYWN0aXZlIHNpZGUgcmVjZWl2ZXMgdGhlIG1lc3NhZ2UgYW5kIHB1dHMgdGhlIFFQ IHRvIFJUUiwgUlRTIHN0YXRlIGJhc2VkIG9uIHJlY2VpdmVkIGJpdCBzZXR0aW5nIGZyb20gcGFz c2l2ZSBzaWRlLg0KDQpGbG93IGlzIG5vIGRpZmZlcmVudCB0aGFuIGhvdyByZXN0IG9mIHRoZSBj b25uZWN0aW9uIHNwZWNpZmljIHBhcmFtZXRlcnMgYXJlIHNoYXJlZCBzdWNoIGFzIElSRC9PUkQs IFBTTiwgdGltZW91dHMsIG10dSBldGMuDQoNCg0KDQo+IA0KPiBUb20uDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Oh, so there's a MAD protocol change under the hood. Well, that's a wider question. And I still don't understand how existing, non-strict-requiring protocols can take advantage of this. Nor how this works for non-Mellanox, non-IB/RoCE implementations.

Again, I'd be a lot less concerned if non-strict were the default, and strict mode was negotiated. It's all just so upside-down.

Tom.

ʇڙ�,j��f���h���z�
�w���
���j:+v���w�j�m��������zZ+�����ݢj"��!tml=

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxw ZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3 IDc6MTIgUE0NCj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFzb24g R3VudGhvcnBlDQo+IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiBDYzogQmFy dCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9y ZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElk YW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUmU6IFtQQVRD SCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiAN Cj4gT24gNi8xMi8yMDE3IDc6NTkgUE0sIFBhcmF2IFBhbmRpdCB3cm90ZToNCj4gPj4gLS0tLS1P cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTogVG9tIFRhbHBleSBbbWFpbHRvOnRvbUB0 YWxwZXkuY29tXQ0KPiA+PiBTZW50OiBNb25kYXksIEp1bmUgMTIsIDIwMTcgNjo0NCBQTQ0KPiA+ PiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5jb20+OyBKYXNvbiBHdW50aG9ycGUN Cj4gPj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5jb20+DQo+ID4+IENjOiBCYXJ0IFZh biBBc3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29tPjsgbGVvbkBrZXJuZWwub3JnOw0K PiA+PiBkbGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1hQHZnZXIua2VybmVsLm9yZzsgSWRh biBCdXJzdGVpbg0KPiA+PiA8aWRhbmJAbWVsbGFub3guY29tPg0KPiA+PiBTdWJqZWN0OiBSZTog W1BBVENIIHJkbWEtbmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9mIG9yZGVyIGRhdGENCj4gPj4gcGxh Y2VtZW50DQo+ID4+DQo+ID4+IE9uIDYvMTIvMjAxNyA2OjU0IFBNLCBQYXJhdiBQYW5kaXQgd3Jv dGU6DQo+ID4+PiBIaSBUb20sDQo+ID4+Pg0KPiA+Pj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t LS0tDQo+ID4+Pj4gRnJvbTogVG9tIFRhbHBleSBbbWFpbHRvOnRvbUB0YWxwZXkuY29tXQ0KPiA+ Pj4+IFNlbnQ6IE1vbmRheSwgSnVuZSAxMiwgMjAxNyA1OjIwIFBNDQo+ID4+Pj4gVG86IFBhcmF2 IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFzb24gR3VudGhvcnBlDQo+ID4+Pj4gPGpn dW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5jb20+DQo+ID4+Pj4gQ2M6IEJhcnQgVmFuIEFzc2No ZSA8QmFydC5WYW5Bc3NjaGVAc2FuZGlzay5jb20+Ow0KPiBsZW9uQGtlcm5lbC5vcmc7DQo+ID4+ Pj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElkYW4g QnVyc3RlaW4NCj4gPj4+PiA8aWRhbmJAbWVsbGFub3guY29tPg0KPiA+Pj4+IFN1YmplY3Q6IFJl OiBbUEFUQ0ggcmRtYS1uZXh0IDAvM10gU3VwcG9ydCBvdXQgb2Ygb3JkZXIgZGF0YQ0KPiA+Pj4+ IHBsYWNlbWVudA0KPiA+Pj4+DQo+ID4+Pj4gT24gNi8xMi8yMDE3IDU6MzIgUE0sIFBhcmF2IFBh bmRpdCB3cm90ZToNCj4gPj4+Pj4gSGkgVG9tLA0KPiA+Pj4+IC4uLg0KPiA+Pj4+Pj4NCj4gPj4+ Pj4+IEkgYWdyZWUgd2l0aCBKYXNvbiwgdGhlIGJpdCBzaG91bGQgYmUgMSBieSBkZWZhdWx0LCBp ZiBkZWZpbmVkIGFzDQo+ID4+Pj4+PiB5b3UNCj4gPj4+PiBwcm9wb3NlLg0KPiA+Pj4+Pj4gT3V0 LW9mLW9yZGVyIGlzIHRoZSBub3JtLCBub3QgdGhlIGV4Y2VwdGlvbiwgZm9yIFVMUHMuDQo+ID4+ Pj4+PiBIb25lc3RseSwgSSB0aGluayB5b3Ugc2hvdWxkIHBlcmhhcHMgY29uc2lkZXIgbWFraW5n IGl0IHRoZQ0KPiA+Pj4+Pj4gZGVmYXVsdCBvbiB5b3VyIGRldmljZXMsIGFuZCBhbGxvd2luZyBv bmx5IE1MWC1hd2FyZSBVTFBzIHRvIHR1cm4NCj4gaXQgb2ZmLg0KPiA+Pj4+Pj4NCj4gPj4+Pj4N Cj4gPj4+Pj4gVGhlcmUgY2FuIGJlIGNhc2VzIGluIGRlcGxveW1lbnQgd2hlcmUgcmVzcG9uZGVy IGhhcyBzdXBwb3J0IGZvcg0KPiA+Pj4+IHJlY2VpdmluZyBvdXQtb2Ytb3JkZXIsIGJ1dCByZXF1 ZXN0ZXIgZG9lc24ndC4NCj4gPj4+Pg0KPiA+Pj4+IFl1Y2shIFNvIHRoaXMgbmVlZHMgdG8gYmUg bmVnb3RpYXRlZCBlbmQtdG8tZW5kLCBhbmQgYnkgdGhlIHVwcGVyDQo+IGxheWVyPw0KPiA+Pj4+ IFRhbGsgYWJvdXQgYmFycmllcnMgdG8gYWRvcHRpb24sIGFuZCBvcHBvcnR1bml0aWVzIGZvciBk aXNhc3Rlci4NCj4gPj4+Pg0KPiA+Pj4gQXMgSmFzb24gY29uZmlybWVkIHRoYXQgYWxsIExpbnV4 IGtlcm5lbCBjb25zdW1lcnMgYXJlIGNvZGVkIHRvIGJlDQo+ID4+PiBjb21wbGlhbnQgdG8gbzkt MjAgcmVxdWlyZW1lbnQsIFNvIEkgdGhpbmsga2VybmVsIGJhc2VkIHJkbWEtY20NCj4gPj4gY29u c3VtZXJzIGNhbiBiZSB0cmFuc3BhcmVudGx5IGVuYWJsZWQgZW5kLXRvLWVuZCB3aXRob3V0IFVM UCdzDQo+ID4+IGludm9sdmVtZW50IHdpdGggcmRtYV9hY2NlcHQoKSBhbmQgcmRtYV9jb25uZWN0 KCkuDQo+ID4+DQo+ID4+IEkgaGF2ZSB0d28gdGhvdWdodHMgaGVyZS4NCj4gPj4NCj4gPj4gMSkg WW91IHNlZW0gdG8gYXNzdW1lIGFsbCBjb25zdW1lcnMgYXJlIExpbnV4LCBhbmQgZG8gbm90IG5l ZWQgdG8NCj4gPj4gbmVnb3RpYXRlLiBUaGlzIGlzIGEgZGFuZ2Vyb3VzIGFzc3VtcHRpb24uDQo+ ID4gQ2VydGFpbmx5IG5vdC4gSSBkaWRuJ3QgYXNzdW1lIHRoYXQuIEkganVzdCBnYXZlIG9uZSBl eGFtcGxlIHRoYXQga25vd24NCj4gY29uc3VtZXJzIGNhbiBiZSBkb25lIHdpdGhvdXQgbW9kaWZ5 aW5nIHRoZSBVTFAuDQo+ID4gRXhwbGFpbmVkIGZ1cnRoZXIgaW4gM3JkIHF1ZXN0aW9uLg0KPiA+ IEV2ZW4gb3RoZXIgY29uc3VtZXJzIGNhbiB3b3JrIHdpdGggdGhpcyBzb2x1dGlvbi4NCj4gPiBG b3IgZXhhbXBsZSBMaW51eCByZG1hY20gYmFzZWQgY2xpZW50IGFuZCBPdGhlciBPUyBiYXNlZCBz ZXJ2ZXIuDQo+ID4gQ2xpZW50IGlzIG9vbyBjYXBhYmxlLg0KPiA+IFNlcnZlciBpcyBvb28gbm90 IGNhcGFibGUuDQo+ID4gT25jZSB5b3UgZm9sbG93IGJlbG93IHJkbWFjbSBiYXNlZCBzZXF1ZW5j ZSwgaXQgd2lsbCBiZSBjbGVhciBob3cgdGhpcw0KPiB3aWxsIHdvcmtzLg0KPiANCj4gT2gsIHNv IHRoZXJlJ3MgYSBNQUQgcHJvdG9jb2wgY2hhbmdlIHVuZGVyIHRoZSBob29kLiANCk5vLiBUaGVy ZSBpcyBubyBjaGFuZ2UgdW5kZXIgdGhlIGhvb2QuDQpZb3VyIHF1ZXN0aW9uIHdhcyBob3cgY2Fu IHdlIGF2b2lkIFVMUCBjaGFuZ2UgYW5kIHN0aWxsIHRoZXkgY2FuIGJlbmVmaXQgb2YgdGhpcyBm ZWF0dXJlPw0KU28gSSBzYWlkIHJkbWFjbSBiYXNlZCBMaW51eCBrZXJuZWwgY29uc3VtZXJzIHRo YXQgd2Uga25vdyBvZiBjb21wbHkgdG8gbzktMjAsIGNhbiB0YWtlIHRoZSBiZW5lZml0IG9uY2Ug cmRtYWNtIGlzIGV4dGVuZGVkIGFzIGJlbG93IGV4YW1wbGUuDQoNCj4gV2VsbCwgdGhhdCdzIGEg d2lkZXINCj4gcXVlc3Rpb24uIEFuZCBJIHN0aWxsIGRvbid0IHVuZGVyc3RhbmQgaG93IGV4aXN0 aW5nLCBub24tc3RyaWN0LXJlcXVpcmluZw0KPiBwcm90b2NvbHMgY2FuIHRha2UgYWR2YW50YWdl IG9mIHRoaXMuDQo+IE5vciBob3cgdGhpcyB3b3JrcyBmb3Igbm9uLU1lbGxhbm94LCBub24tSUIv Um9DRSBpbXBsZW1lbnRhdGlvbnMuDQoNCkRldmljZSBjYXBhYmlsaXR5IGluZGljYXRlcyB0aGF0 IHdoaWNoIGRldmljZSBzdXBwb3J0cyB0aGlzLiBFeHBsYWluZWQgaW4gRG9jdW1lbnRhdGlvbi9v dXRfb2Zfb3JkZXIudHh0IHVzYWdlIHNlY3Rpb24uDQpTbyB3aGljaGV2ZXIgdmVuZG9yIHN1cHBv cnRzIGl0LCB3aGljaGV2ZXIgcHJvdG9jb2wgc3VwcG9ydHMgaXQsIGNhbiBzZXQgdGhpcyBvcHRp b25hbCBkZXZpY2UgY2FwYWJpbGl0eS4NCg0KPiANCj4gQWdhaW4sIEknZCBiZSBhIGxvdCBsZXNz IGNvbmNlcm5lZCBpZiBub24tc3RyaWN0IHdlcmUgdGhlIGRlZmF1bHQsIGFuZCBzdHJpY3QNCj4g bW9kZSB3YXMgbmVnb3RpYXRlZC4gSXQncyBhbGwganVzdCBzbyB1cHNpZGUtZG93bi4NCg0KSW4g SUIgc3BlYywgaW4tb3JkZXIgZGVsaXZlcnkgaXMgZGVmYXVsdC4gU28gY2FuIHlvdSBzdWdnZXN0 IGhvdyBjYW4gd2UgY2hhbmdlIGRlZmF1bHQgSUIgYmVoYXZpb3Igd2l0aG91dCBicmVha2luZyBh bnl0aGluZz8NCkFkZGluZyBvcHRpb25hbCBhdHRyaWJ1dGUgc2VlbXMgdGhlIHJpZ2h0IHdheSB0 aGF0IGVuc3VyZXMgY29tcGF0aWJpbGl0eS4NCj4gDQo+IFRvbS4NCj4gDQo+ID4+IDIpIEkgYXNz dW1lIHRoYXQgdGhlcmUgaXMgc29tZSBwZXJmb3JtYW5jZSBiZW5lZml0IHRvIHRvZ2dsaW5nIHRo aXMNCj4gPj4gc2V0dGluZyB0byBub24tc3RyaWN0LiBTbywgaG93IGRvIGV4aXN0aW5nIGNvbnN1 bWVycyBnZXQgdGhpcw0KPiA+PiBhZHZhbnRhZ2UsIGVzcGVjaWFsbHkgc2luY2UgdGhleSBkb24n dCBuZWVkIHN0cmljdCBzZW1hbnRpY3M/IEJlYXJpbmcNCj4gPj4gaW4gbWluZCB0aGF0IHRoZXkg ZG8gaGF2ZSB0byBuZWdvdGlhdGUgdGhpcyBlbmQtdG8tZW5kLCBtZWFuaW5nIHRoZXkNCj4gcmVx dWlyZSBhIHByb3RvY29sIGV4dGVuc2lvbi4NCj4gPiBJIGRvbid0IGhhdmUgY29tcGxldGVseSB0 cmFuc3BhcmVudCB1cHN0cmVhbSBzb2x1dGlvbiBmb3IgZXhpc3RpbmcNCj4gY29uc3VtZXJzIHll dC4NCj4gPj4NCj4gPj4gQWN0dWFsbHkuIEkgaGF2ZSBhIHRoaXJkIHRob3VnaHQuIFNpbmNlIHRo aXMgaXMgYW4gYXR0cmlidXRlIHRvIHFwDQo+ID4+IGNyZWF0aW9uLCBwZXJmb3JtZWQgZXZlbiBi ZWZvcmUgZXN0YWJsaXNoaW5nIGEgY29ubmVjdGlvbiwgaG93IGRvZXMNCj4gPj4gdGhlIHVwcGVy IGxheWVyIGtub3cgd2hlbiB0byBzZXQgaXQ/DQo+ID4gVGhpcyBpcyBub3QgYXQgUVAgY3JlYXRp b24gdGltZS4gSSBoYXZlIGRlc2NyaWJlZCBpbg0KPiBEb2N1bWVudGF0aW9uL291dF9vZl9vcmRl ci50eHQgaW4gdXNhZ2Ugc2VjdGlvbiAzLg0KPiA+IFRoaXMgaXMgYXQgUVAgc3RhdGUgdHJhbnNp dGlvbiBmcm9tIElOSVQgdG8gUlRSLg0KPiA+IEhlcmUgaXMgdGhlIGZsb3cuIEl0J3MganVzdCBu b3QgY29kZWQgZW5vdWdoIGZvciBwb3N0aW5nIHBhdGNoZXMuDQo+ID4NCj4gPiAxLiBXaGVuIHJk bWFjbSBhY3RpdmUgc2lkZSBjcmVhdGVzIHRoZSBRUCwgSXQgaXMgSU5JVCBzdGF0ZS4NCj4gPiAy LiBTZW5kIE1BRF9SZXEgbXNnIChpbmRpY2F0aW5nIG9vb19yZXF1ZXN0ZWQ9MSkgMy4gV2hlbiBy ZG1hY20NCj4gPiBwYXNzaXZlIHNpZGUgcmVjZWl2ZXMgdGhlIG1lc3NhZ2UsIGl0IGxvb2tzIHVw IGRldmljZV9jYXAgYXR0cmlidXRlIGFuZA0KPiBtYXRjaGVzIGl0IGFnYWluc3Qgb29vX3JlcXVl c3RlZCBmbGFnLg0KPiA+IDQuIHdoZW4gZGV2aWNlIHN1cHBvcnRzIGl0LCBNQURfUnNwIG1zZyBz ZXRzIG9vb19lbmFibGVkPTEsIGlmIGl0DQo+ID4gZG9lc24ndCBzdXBwb3J0IGl0LCBvb29fZW5h YmxlZD0wIDUuIHJkbWFjbSBwYXNzaXZlIHNpZGUgY3JlYXRlcyB0aGUgUVANCj4gYW5kIG1vdmVz IHRvIFJUUiBzdGF0ZSAod2l0aCBRUCBvb28gZW5hYmxlZCBiaXQgc2V0KS4NCj4gPiA2LiBhY3Rp dmUgc2lkZSByZWNlaXZlcyB0aGUgbWVzc2FnZSBhbmQgcHV0cyB0aGUgUVAgdG8gUlRSLCBSVFMg c3RhdGUNCj4gYmFzZWQgb24gcmVjZWl2ZWQgYml0IHNldHRpbmcgZnJvbSBwYXNzaXZlIHNpZGUu DQo+ID4NCj4gPiBGbG93IGlzIG5vIGRpZmZlcmVudCB0aGFuIGhvdyByZXN0IG9mIHRoZSBjb25u ZWN0aW9uIHNwZWNpZmljIHBhcmFtZXRlcnMNCj4gYXJlIHNoYXJlZCBzdWNoIGFzIElSRC9PUkQs IFBTTiwgdGltZW91dHMsIG10dSBldGMuDQo+ID4NCj4gPg0KPiA+DQo+ID4+DQo+ID4+IFRvbS4N Cj4gPiBOICAgICByICB5ICAgYiBYICDHp3YgXiAp3rp7Lm4gKyAgICB7ICDZmiB7YXkgHcqH2pkg LGogICBmICAgaCAgIHogHiB3DQo+IA0KPiAgICBqOit2ICAgdyBqIG0gICAgICAgICB6WisgICAg IN2iaiIgICF0bWw9DQo+ID4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I don't agree. Requests are sent in-order, and the responder processes them in-order, but the bytes thenselves are not guaranteed to appear in-order. Additionally, if retries occur, this is most definitely not the case.

Section 9.5 Transaction Ordering, I believe, covers these requirements. Can you tell me where I misunderstand them? In fact, c9-28 explicitly warns:

• An application shall not depend upon the order of data writes to memory within a message. For example, if an application sets up data buffers that overlap, for separate data segments within a message, it is not guaranteed that the last sent data will always overwrite the earlier.

My guess is that this bit overrides the MLX behavior of never pipelining RDMA Write requests, allowing more packets to be queued at the responder and making better use of the network. This is not at all prohibited by the spec, nor is it unexpected by properly-coded upper layers, which all the kernel consumers are.

I have one other question on the Documentation out-of-order.txt. It states the fence bit can be used to force ordering on a non-strict connection. But fence doesn't apply to RDMA Write? It only applies to operations which produce a reply, such as RDMA Read or Atomic. Have you changed the semantic?

Tom.

So can you suggest how can we change default IB behavior without breaking anything?

ʇڙ ,j f h z 
w

ʇڙ�,j��f���h���z�
�w���
���j:+v���w�j�m��������zZ+�����ݢj"��!tml=

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

+1

And again, if message ordering and table 79 are unchanged in this new mode it is still fully conformant to the IBTA (and again, I don't see how the transmitter can change behavior and not break table 79)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLCBKYXNvbiwNCg0KSSB3aWxsIGdldCBiYWNrIG9uIHVwZGF0ZWQgdjEgZG9jdW1lbnRh dGlvbiBhbmQgYW5zd2VycyB0byBiZWxvdyBxdWVzdGlvbnMgb25jZSBJIGdldCBzb21lIG1vcmUg ZGV0YWlscyBpbnRlcm5hbGx5Lg0KDQpQYXJhdg0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t LS0tDQo+IEZyb206IFRvbSBUYWxwZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDog TW9uZGF5LCBKdW5lIDEyLCAyMDE3IDg6MzAgUE0NCj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZA bWVsbGFub3guY29tPjsgSmFzb24gR3VudGhvcnBlDQo+IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVz ZWFyY2guY29tPg0KPiBDYzogQmFydCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNr LmNvbT47IGxlb25Aa2VybmVsLm9yZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRt YUB2Z2VyLmtlcm5lbC5vcmc7IElkYW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4N Cj4gU3ViamVjdDogUmU6IFtQQVRDSCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRl ciBkYXRhIHBsYWNlbWVudA0KPiANCj4gT24gNi8xMi8yMDE3IDg6MzYgUE0sIFBhcmF2IFBhbmRp dCB3cm90ZToNCj4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTogVG9t IFRhbHBleSBbbWFpbHRvOnRvbUB0YWxwZXkuY29tXQ0KPiA+PiBTZW50OiBNb25kYXksIEp1bmUg MTIsIDIwMTcgNzoxMiBQTQ0KPiA+PiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5j b20+OyBKYXNvbiBHdW50aG9ycGUNCj4gPj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5j b20+DQo+ID4+IENjOiBCYXJ0IFZhbiBBc3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29t PjsgbGVvbkBrZXJuZWwub3JnOw0KPiA+PiBkbGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1h QHZnZXIua2VybmVsLm9yZzsgSWRhbiBCdXJzdGVpbg0KPiA+PiA8aWRhbmJAbWVsbGFub3guY29t Pg0KPiA+PiBTdWJqZWN0OiBSZTogW1BBVENIIHJkbWEtbmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9m IG9yZGVyIGRhdGENCj4gPj4gcGxhY2VtZW50DQo+ID4+DQo+ID4+IE9uIDYvMTIvMjAxNyA3OjU5 IFBNLCBQYXJhdiBQYW5kaXQgd3JvdGU6DQo+ID4+Pj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0t LS0NCj4gPj4+PiBGcm9tOiBUb20gVGFscGV5IFttYWlsdG86dG9tQHRhbHBleS5jb21dDQo+ID4+ Pj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3IDY6NDQgUE0NCj4gPj4+PiBUbzogUGFyYXYg UGFuZGl0IDxwYXJhdkBtZWxsYW5veC5jb20+OyBKYXNvbiBHdW50aG9ycGUNCj4gPj4+PiA8amd1 bnRob3JwZUBvYnNpZGlhbnJlc2VhcmNoLmNvbT4NCj4gPj4+PiBDYzogQmFydCBWYW4gQXNzY2hl IDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9yZzsNCj4gPj4+PiBk bGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1hQHZnZXIua2VybmVsLm9yZzsgSWRhbiBCdXJz dGVpbg0KPiA+Pj4+IDxpZGFuYkBtZWxsYW5veC5jb20+DQo+ID4+Pj4gU3ViamVjdDogUmU6IFtQ QVRDSCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhDQo+ID4+Pj4gcGxh Y2VtZW50DQo+ID4+Pj4NCj4gPj4+PiBPbiA2LzEyLzIwMTcgNjo1NCBQTSwgUGFyYXYgUGFuZGl0 IHdyb3RlOg0KPiA+Pj4+PiBIaSBUb20sDQo+ID4+Pj4+DQo+ID4+Pj4+PiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+Pj4gRnJvbTogVG9tIFRhbHBleSBbbWFpbHRvOnRvbUB0YWxw ZXkuY29tXQ0KPiA+Pj4+Pj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3IDU6MjAgUE0NCj4g Pj4+Pj4+IFRvOiBQYXJhdiBQYW5kaXQgPHBhcmF2QG1lbGxhbm94LmNvbT47IEphc29uIEd1bnRo b3JwZQ0KPiA+Pj4+Pj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5jb20+DQo+ID4+Pj4+ PiBDYzogQmFydCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47DQo+ID4+ IGxlb25Aa2VybmVsLm9yZzsNCj4gPj4+Pj4+IGRsZWRmb3JkQHJlZGhhdC5jb207IGxpbnV4LXJk bWFAdmdlci5rZXJuZWwub3JnOyBJZGFuIEJ1cnN0ZWluDQo+ID4+Pj4+PiA8aWRhbmJAbWVsbGFu b3guY29tPg0KPiA+Pj4+Pj4gU3ViamVjdDogUmU6IFtQQVRDSCByZG1hLW5leHQgMC8zXSBTdXBw b3J0IG91dCBvZiBvcmRlciBkYXRhDQo+ID4+Pj4+PiBwbGFjZW1lbnQNCj4gPj4+Pj4+DQo+ID4+ Pj4+PiBPbiA2LzEyLzIwMTcgNTozMiBQTSwgUGFyYXYgUGFuZGl0IHdyb3RlOg0KPiA+Pj4+Pj4+ IEhpIFRvbSwNCj4gPj4+Pj4+IC4uLg0KPiA+Pj4+Pj4+Pg0KPiA+Pj4+Pj4+PiBJIGFncmVlIHdp dGggSmFzb24sIHRoZSBiaXQgc2hvdWxkIGJlIDEgYnkgZGVmYXVsdCwgaWYgZGVmaW5lZA0KPiA+ Pj4+Pj4+PiBhcyB5b3UNCj4gPj4+Pj4+IHByb3Bvc2UuDQo+ID4+Pj4+Pj4+IE91dC1vZi1vcmRl ciBpcyB0aGUgbm9ybSwgbm90IHRoZSBleGNlcHRpb24sIGZvciBVTFBzLg0KPiA+Pj4+Pj4+PiBI b25lc3RseSwgSSB0aGluayB5b3Ugc2hvdWxkIHBlcmhhcHMgY29uc2lkZXIgbWFraW5nIGl0IHRo ZQ0KPiA+Pj4+Pj4+PiBkZWZhdWx0IG9uIHlvdXIgZGV2aWNlcywgYW5kIGFsbG93aW5nIG9ubHkg TUxYLWF3YXJlIFVMUHMgdG8NCj4gPj4+Pj4+Pj4gdHVybg0KPiA+PiBpdCBvZmYuDQo+ID4+Pj4+ Pj4+DQo+ID4+Pj4+Pj4NCj4gPj4+Pj4+PiBUaGVyZSBjYW4gYmUgY2FzZXMgaW4gZGVwbG95bWVu dCB3aGVyZSByZXNwb25kZXIgaGFzIHN1cHBvcnQgZm9yDQo+ID4+Pj4+PiByZWNlaXZpbmcgb3V0 LW9mLW9yZGVyLCBidXQgcmVxdWVzdGVyIGRvZXNuJ3QuDQo+ID4+Pj4+Pg0KPiA+Pj4+Pj4gWXVj ayEgU28gdGhpcyBuZWVkcyB0byBiZSBuZWdvdGlhdGVkIGVuZC10by1lbmQsIGFuZCBieSB0aGUg dXBwZXINCj4gPj4gbGF5ZXI/DQo+ID4+Pj4+PiBUYWxrIGFib3V0IGJhcnJpZXJzIHRvIGFkb3B0 aW9uLCBhbmQgb3Bwb3J0dW5pdGllcyBmb3IgZGlzYXN0ZXIuDQo+ID4+Pj4+Pg0KPiA+Pj4+PiBB cyBKYXNvbiBjb25maXJtZWQgdGhhdCBhbGwgTGludXgga2VybmVsIGNvbnN1bWVycyBhcmUgY29k ZWQgdG8gYmUNCj4gPj4+Pj4gY29tcGxpYW50IHRvIG85LTIwIHJlcXVpcmVtZW50LCBTbyBJIHRo aW5rIGtlcm5lbCBiYXNlZCByZG1hLWNtDQo+ID4+Pj4gY29uc3VtZXJzIGNhbiBiZSB0cmFuc3Bh cmVudGx5IGVuYWJsZWQgZW5kLXRvLWVuZCB3aXRob3V0IFVMUCdzDQo+ID4+Pj4gaW52b2x2ZW1l bnQgd2l0aCByZG1hX2FjY2VwdCgpIGFuZCByZG1hX2Nvbm5lY3QoKS4NCj4gPj4+Pg0KPiA+Pj4+ IEkgaGF2ZSB0d28gdGhvdWdodHMgaGVyZS4NCj4gPj4+Pg0KPiA+Pj4+IDEpIFlvdSBzZWVtIHRv IGFzc3VtZSBhbGwgY29uc3VtZXJzIGFyZSBMaW51eCwgYW5kIGRvIG5vdCBuZWVkIHRvDQo+ID4+ Pj4gbmVnb3RpYXRlLiBUaGlzIGlzIGEgZGFuZ2Vyb3VzIGFzc3VtcHRpb24uDQo+ID4+PiBDZXJ0 YWlubHkgbm90LiBJIGRpZG4ndCBhc3N1bWUgdGhhdC4gSSBqdXN0IGdhdmUgb25lIGV4YW1wbGUg dGhhdA0KPiA+Pj4ga25vd24NCj4gPj4gY29uc3VtZXJzIGNhbiBiZSBkb25lIHdpdGhvdXQgbW9k aWZ5aW5nIHRoZSBVTFAuDQo+ID4+PiBFeHBsYWluZWQgZnVydGhlciBpbiAzcmQgcXVlc3Rpb24u DQo+ID4+PiBFdmVuIG90aGVyIGNvbnN1bWVycyBjYW4gd29yayB3aXRoIHRoaXMgc29sdXRpb24u DQo+ID4+PiBGb3IgZXhhbXBsZSBMaW51eCByZG1hY20gYmFzZWQgY2xpZW50IGFuZCBPdGhlciBP UyBiYXNlZCBzZXJ2ZXIuDQo+ID4+PiBDbGllbnQgaXMgb29vIGNhcGFibGUuDQo+ID4+PiBTZXJ2 ZXIgaXMgb29vIG5vdCBjYXBhYmxlLg0KPiA+Pj4gT25jZSB5b3UgZm9sbG93IGJlbG93IHJkbWFj bSBiYXNlZCBzZXF1ZW5jZSwgaXQgd2lsbCBiZSBjbGVhciBob3cNCj4gPj4+IHRoaXMNCj4gPj4g d2lsbCB3b3Jrcy4NCj4gPj4NCj4gPj4gT2gsIHNvIHRoZXJlJ3MgYSBNQUQgcHJvdG9jb2wgY2hh bmdlIHVuZGVyIHRoZSBob29kLg0KPiA+IE5vLiBUaGVyZSBpcyBubyBjaGFuZ2UgdW5kZXIgdGhl IGhvb2QuDQo+ID4gWW91ciBxdWVzdGlvbiB3YXMgaG93IGNhbiB3ZSBhdm9pZCBVTFAgY2hhbmdl IGFuZCBzdGlsbCB0aGV5IGNhbiBiZW5lZml0DQo+IG9mIHRoaXMgZmVhdHVyZT8NCj4gPiBTbyBJ IHNhaWQgcmRtYWNtIGJhc2VkIExpbnV4IGtlcm5lbCBjb25zdW1lcnMgdGhhdCB3ZSBrbm93IG9m IGNvbXBseSB0bw0KPiBvOS0yMCwgY2FuIHRha2UgdGhlIGJlbmVmaXQgb25jZSByZG1hY20gaXMg ZXh0ZW5kZWQgYXMgYmVsb3cgZXhhbXBsZS4NCj4gPg0KPiA+PiBXZWxsLCB0aGF0J3MgYSB3aWRl cg0KPiA+PiBxdWVzdGlvbi4gQW5kIEkgc3RpbGwgZG9uJ3QgdW5kZXJzdGFuZCBob3cgZXhpc3Rp bmcsDQo+ID4+IG5vbi1zdHJpY3QtcmVxdWlyaW5nIHByb3RvY29scyBjYW4gdGFrZSBhZHZhbnRh Z2Ugb2YgdGhpcy4NCj4gPj4gTm9yIGhvdyB0aGlzIHdvcmtzIGZvciBub24tTWVsbGFub3gsIG5v bi1JQi9Sb0NFIGltcGxlbWVudGF0aW9ucy4NCj4gPg0KPiA+IERldmljZSBjYXBhYmlsaXR5IGlu ZGljYXRlcyB0aGF0IHdoaWNoIGRldmljZSBzdXBwb3J0cyB0aGlzLiBFeHBsYWluZWQgaW4NCj4g RG9jdW1lbnRhdGlvbi9vdXRfb2Zfb3JkZXIudHh0IHVzYWdlIHNlY3Rpb24uDQo+ID4gU28gd2hp Y2hldmVyIHZlbmRvciBzdXBwb3J0cyBpdCwgd2hpY2hldmVyIHByb3RvY29sIHN1cHBvcnRzIGl0 LCBjYW4gc2V0DQo+IHRoaXMgb3B0aW9uYWwgZGV2aWNlIGNhcGFiaWxpdHkuDQo+ID4NCj4gPj4N Cj4gPj4gQWdhaW4sIEknZCBiZSBhIGxvdCBsZXNzIGNvbmNlcm5lZCBpZiBub24tc3RyaWN0IHdl cmUgdGhlIGRlZmF1bHQsDQo+ID4+IGFuZCBzdHJpY3QgbW9kZSB3YXMgbmVnb3RpYXRlZC4gSXQn cyBhbGwganVzdCBzbyB1cHNpZGUtZG93bi4NCj4gPg0KPiA+IEluIElCIHNwZWMsIGluLW9yZGVy IGRlbGl2ZXJ5IGlzIGRlZmF1bHQuDQo+IA0KPiBJIGRvbid0IGFncmVlLiBSZXF1ZXN0cyBhcmUg c2VudCBpbi1vcmRlciwgYW5kIHRoZSByZXNwb25kZXIgcHJvY2Vzc2VzIHRoZW0NCj4gaW4tb3Jk ZXIsIGJ1dCB0aGUgYnl0ZXMgdGhlbnNlbHZlcyBhcmUgbm90IGd1YXJhbnRlZWQgdG8gYXBwZWFy IGluLW9yZGVyLg0KPiBBZGRpdGlvbmFsbHksIGlmIHJldHJpZXMgb2NjdXIsIHRoaXMgaXMgbW9z dCBkZWZpbml0ZWx5IG5vdCB0aGUgY2FzZS4NCj4gDQo+IFNlY3Rpb24gOS41IFRyYW5zYWN0aW9u IE9yZGVyaW5nLCBJIGJlbGlldmUsIGNvdmVycyB0aGVzZSByZXF1aXJlbWVudHMuIENhbg0KPiB5 b3UgdGVsbCBtZSB3aGVyZSBJIG1pc3VuZGVyc3RhbmQgdGhlbT8NCj4gSW4gZmFjdCwgYzktMjgg ZXhwbGljaXRseSB3YXJuczoNCj4gDQo+ICAgIOKAoiBBbiBhcHBsaWNhdGlvbiBzaGFsbCBub3Qg ZGVwZW5kIHVwb24gdGhlIG9yZGVyIG9mIGRhdGEgd3JpdGVzIHRvDQo+ICAgIG1lbW9yeSB3aXRo aW4gYSBtZXNzYWdlLiBGb3IgZXhhbXBsZSwgaWYgYW4gYXBwbGljYXRpb24gc2V0cyB1cA0KPiAg ICBkYXRhIGJ1ZmZlcnMgdGhhdCBvdmVybGFwLCBmb3Igc2VwYXJhdGUgZGF0YSBzZWdtZW50cyB3 aXRoaW4gYQ0KPiAgICBtZXNzYWdlLCBpdCBpcyBub3QgZ3VhcmFudGVlZCB0aGF0IHRoZSBsYXN0 IHNlbnQgZGF0YSB3aWxsIGFsd2F5cw0KPiAgICBvdmVyd3JpdGUgdGhlIGVhcmxpZXIuDQo+IA0K PiBNeSBndWVzcyBpcyB0aGF0IHRoaXMgYml0IG92ZXJyaWRlcyB0aGUgTUxYIGJlaGF2aW9yIG9m IG5ldmVyIHBpcGVsaW5pbmcgUkRNQQ0KPiBXcml0ZSByZXF1ZXN0cywgYWxsb3dpbmcgbW9yZSBw YWNrZXRzIHRvIGJlIHF1ZXVlZCBhdCB0aGUgcmVzcG9uZGVyIGFuZA0KPiBtYWtpbmcgYmV0dGVy IHVzZSBvZiB0aGUgbmV0d29yay4gVGhpcyBpcyBub3QgYXQgYWxsIHByb2hpYml0ZWQgYnkgdGhl IHNwZWMsIG5vcg0KPiBpcyBpdCB1bmV4cGVjdGVkIGJ5IHByb3Blcmx5LWNvZGVkIHVwcGVyIGxh eWVycywgd2hpY2ggYWxsIHRoZSBrZXJuZWwNCj4gY29uc3VtZXJzIGFyZS4NCj4gDQo+IEkgaGF2 ZSBvbmUgb3RoZXIgcXVlc3Rpb24gb24gdGhlIERvY3VtZW50YXRpb24gb3V0LW9mLW9yZGVyLnR4 dC4NCj4gSXQgc3RhdGVzIHRoZSBmZW5jZSBiaXQgY2FuIGJlIHVzZWQgdG8gZm9yY2Ugb3JkZXJp bmcgb24gYSBub24tc3RyaWN0DQo+IGNvbm5lY3Rpb24uIEJ1dCBmZW5jZSBkb2Vzbid0IGFwcGx5 IHRvIFJETUEgV3JpdGU/DQo+IEl0IG9ubHkgYXBwbGllcyB0byBvcGVyYXRpb25zIHdoaWNoIHBy b2R1Y2UgYSByZXBseSwgc3VjaCBhcyBSRE1BIFJlYWQgb3INCj4gQXRvbWljLiBIYXZlIHlvdSBj aGFuZ2VkIHRoZSBzZW1hbnRpYz8NCj4gDQo+IFRvbS4NCj4gDQo+IA0KPiANCj4gU28gY2FuIHlv dSBzdWdnZXN0IGhvdyBjYW4gd2UgY2hhbmdlIGRlZmF1bHQgSUIgYmVoYXZpb3Igd2l0aG91dCBi cmVha2luZw0KPiBhbnl0aGluZz8NCj4gPiBBZGRpbmcgb3B0aW9uYWwgYXR0cmlidXRlIHNlZW1z IHRoZSByaWdodCB3YXkgdGhhdCBlbnN1cmVzIGNvbXBhdGliaWxpdHkuDQo+ID4+DQo+ID4+IFRv bS4NCj4gPj4NCj4gPj4+PiAyKSBJIGFzc3VtZSB0aGF0IHRoZXJlIGlzIHNvbWUgcGVyZm9ybWFu Y2UgYmVuZWZpdCB0byB0b2dnbGluZyB0aGlzDQo+ID4+Pj4gc2V0dGluZyB0byBub24tc3RyaWN0 LiBTbywgaG93IGRvIGV4aXN0aW5nIGNvbnN1bWVycyBnZXQgdGhpcw0KPiA+Pj4+IGFkdmFudGFn ZSwgZXNwZWNpYWxseSBzaW5jZSB0aGV5IGRvbid0IG5lZWQgc3RyaWN0IHNlbWFudGljcz8NCj4g Pj4+PiBCZWFyaW5nIGluIG1pbmQgdGhhdCB0aGV5IGRvIGhhdmUgdG8gbmVnb3RpYXRlIHRoaXMg ZW5kLXRvLWVuZCwNCj4gPj4+PiBtZWFuaW5nIHRoZXkNCj4gPj4gcmVxdWlyZSBhIHByb3RvY29s IGV4dGVuc2lvbi4NCj4gPj4+IEkgZG9uJ3QgaGF2ZSBjb21wbGV0ZWx5IHRyYW5zcGFyZW50IHVw c3RyZWFtIHNvbHV0aW9uIGZvciBleGlzdGluZw0KPiA+PiBjb25zdW1lcnMgeWV0Lg0KPiA+Pj4+ DQo+ID4+Pj4gQWN0dWFsbHkuIEkgaGF2ZSBhIHRoaXJkIHRob3VnaHQuIFNpbmNlIHRoaXMgaXMg YW4gYXR0cmlidXRlIHRvIHFwDQo+ID4+Pj4gY3JlYXRpb24sIHBlcmZvcm1lZCBldmVuIGJlZm9y ZSBlc3RhYmxpc2hpbmcgYSBjb25uZWN0aW9uLCBob3cgZG9lcw0KPiA+Pj4+IHRoZSB1cHBlciBs YXllciBrbm93IHdoZW4gdG8gc2V0IGl0Pw0KPiA+Pj4gVGhpcyBpcyBub3QgYXQgUVAgY3JlYXRp b24gdGltZS4gSSBoYXZlIGRlc2NyaWJlZCBpbg0KPiA+PiBEb2N1bWVudGF0aW9uL291dF9vZl9v cmRlci50eHQgaW4gdXNhZ2Ugc2VjdGlvbiAzLg0KPiA+Pj4gVGhpcyBpcyBhdCBRUCBzdGF0ZSB0 cmFuc2l0aW9uIGZyb20gSU5JVCB0byBSVFIuDQo+ID4+PiBIZXJlIGlzIHRoZSBmbG93LiBJdCdz IGp1c3Qgbm90IGNvZGVkIGVub3VnaCBmb3IgcG9zdGluZyBwYXRjaGVzLg0KPiA+Pj4NCj4gPj4+ IDEuIFdoZW4gcmRtYWNtIGFjdGl2ZSBzaWRlIGNyZWF0ZXMgdGhlIFFQLCBJdCBpcyBJTklUIHN0 YXRlLg0KPiA+Pj4gMi4gU2VuZCBNQURfUmVxIG1zZyAoaW5kaWNhdGluZyBvb29fcmVxdWVzdGVk PTEpIDMuIFdoZW4gcmRtYWNtDQo+ID4+PiBwYXNzaXZlIHNpZGUgcmVjZWl2ZXMgdGhlIG1lc3Nh Z2UsIGl0IGxvb2tzIHVwIGRldmljZV9jYXAgYXR0cmlidXRlDQo+ID4+PiBhbmQNCj4gPj4gbWF0 Y2hlcyBpdCBhZ2FpbnN0IG9vb19yZXF1ZXN0ZWQgZmxhZy4NCj4gPj4+IDQuIHdoZW4gZGV2aWNl IHN1cHBvcnRzIGl0LCBNQURfUnNwIG1zZyBzZXRzIG9vb19lbmFibGVkPTEsIGlmIGl0DQo+ID4+ PiBkb2Vzbid0IHN1cHBvcnQgaXQsIG9vb19lbmFibGVkPTAgNS4gcmRtYWNtIHBhc3NpdmUgc2lk ZSBjcmVhdGVzIHRoZQ0KPiA+Pj4gUVANCj4gPj4gYW5kIG1vdmVzIHRvIFJUUiBzdGF0ZSAod2l0 aCBRUCBvb28gZW5hYmxlZCBiaXQgc2V0KS4NCj4gPj4+IDYuIGFjdGl2ZSBzaWRlIHJlY2VpdmVz IHRoZSBtZXNzYWdlIGFuZCBwdXRzIHRoZSBRUCB0byBSVFIsIFJUUw0KPiA+Pj4gc3RhdGUNCj4g Pj4gYmFzZWQgb24gcmVjZWl2ZWQgYml0IHNldHRpbmcgZnJvbSBwYXNzaXZlIHNpZGUuDQo+ID4+ Pg0KPiA+Pj4gRmxvdyBpcyBubyBkaWZmZXJlbnQgdGhhbiBob3cgcmVzdCBvZiB0aGUgY29ubmVj dGlvbiBzcGVjaWZpYw0KPiA+Pj4gcGFyYW1ldGVycw0KPiA+PiBhcmUgc2hhcmVkIHN1Y2ggYXMg SVJEL09SRCwgUFNOLCB0aW1lb3V0cywgbXR1IGV0Yy4NCj4gPj4+DQo+ID4+Pg0KPiA+Pj4NCj4g Pj4+Pg0KPiA+Pj4+IFRvbS4NCj4gPj4+IE4gICAgIHIgIHkgICBiIFggIMendiBeICneunsubiAr ICAgIHsgINmaIHtheSAdyofamSAsaiAgIGYgICBoICAgeiAeIHcNCj4gPj4NCj4gPj4gICAgIGo6 K3YgICB3IGogbSAgICAgICAgIHpaKyAgICAg3aJqIiAgIXRtbD0NCj4gPj4+DQo+ID4gTiAgICAg ciAgeSAgIGIgWCAgx6d2IF4gKd66ey5uICsgICAgeyAg2Zoge2F5IB3Kh9qZICxqICAgZiAgIGgg ICB6IB4gdw0KPiANCj4gICAgajordiAgIHcgaiBtICAgICAgICAgelorICAgICDdomoiICAhdG1s PQ0KPiA+DQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLCBKYXNvbiwNCg0KU29ycnkgZm9yIHRoZSBsYXRlIHJlc3BvbnNlLg0KUGxlYXNlIGZp bmQgdGhlIHJlc3BvbnNlIGlubGluZSBiZWxvdy4NCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut LS0tLQ0KPiBGcm9tOiBUb20gVGFscGV5IFttYWlsdG86dG9tQHRhbHBleS5jb21dDQo+IFNlbnQ6 IE1vbmRheSwgSnVuZSAxMiwgMjAxNyA4OjMwIFBNDQo+IFRvOiBQYXJhdiBQYW5kaXQgPHBhcmF2 QG1lbGxhbm94LmNvbT47IEphc29uIEd1bnRob3JwZQ0KPiA8amd1bnRob3JwZUBvYnNpZGlhbnJl c2VhcmNoLmNvbT4NCj4gQ2M6IEJhcnQgVmFuIEFzc2NoZSA8QmFydC5WYW5Bc3NjaGVAc2FuZGlz ay5jb20+OyBsZW9uQGtlcm5lbC5vcmc7DQo+IGRsZWRmb3JkQHJlZGhhdC5jb207IGxpbnV4LXJk bWFAdmdlci5rZXJuZWwub3JnOyBJZGFuIEJ1cnN0ZWluDQo+IDxpZGFuYkBtZWxsYW5veC5jb20+ DQo+IFN1YmplY3Q6IFJlOiBbUEFUQ0ggcmRtYS1uZXh0IDAvM10gU3VwcG9ydCBvdXQgb2Ygb3Jk ZXIgZGF0YSBwbGFjZW1lbnQNCj4gDQo+ID4NCj4gPiBJbiBJQiBzcGVjLCBpbi1vcmRlciBkZWxp dmVyeSBpcyBkZWZhdWx0Lg0KPiANCj4gSSBkb24ndCBhZ3JlZS4gUmVxdWVzdHMgYXJlIHNlbnQg aW4tb3JkZXIsIGFuZCB0aGUgcmVzcG9uZGVyIHByb2Nlc3NlcyB0aGVtIGluLQ0KPiBvcmRlciwg YnV0IHRoZSBieXRlcyB0aGVuc2VsdmVzIGFyZSBub3QgZ3VhcmFudGVlZCB0byBhcHBlYXIgaW4t b3JkZXIuDQo+IEFkZGl0aW9uYWxseSwgaWYgcmV0cmllcyBvY2N1ciwgdGhpcyBpcyBtb3N0IGRl ZmluaXRlbHkgbm90IHRoZSBjYXNlLg0KPiANCj4gU2VjdGlvbiA5LjUgVHJhbnNhY3Rpb24gT3Jk ZXJpbmcsIEkgYmVsaWV2ZSwgY292ZXJzIHRoZXNlIHJlcXVpcmVtZW50cy4gQ2FuIHlvdQ0KPiB0 ZWxsIG1lIHdoZXJlIEkgbWlzdW5kZXJzdGFuZCB0aGVtPw0KPiBJbiBmYWN0LCBjOS0yOCBleHBs aWNpdGx5IHdhcm5zOg0KPiANCj4gICAg4oCiIEFuIGFwcGxpY2F0aW9uIHNoYWxsIG5vdCBkZXBl bmQgdXBvbiB0aGUgb3JkZXIgb2YgZGF0YSB3cml0ZXMgdG8NCj4gICAgbWVtb3J5IHdpdGhpbiBh IG1lc3NhZ2UuIEZvciBleGFtcGxlLCBpZiBhbiBhcHBsaWNhdGlvbiBzZXRzIHVwDQo+ICAgIGRh dGEgYnVmZmVycyB0aGF0IG92ZXJsYXAsIGZvciBzZXBhcmF0ZSBkYXRhIHNlZ21lbnRzIHdpdGhp biBhDQo+ICAgIG1lc3NhZ2UsIGl0IGlzIG5vdCBndWFyYW50ZWVkIHRoYXQgdGhlIGxhc3Qgc2Vu dCBkYXRhIHdpbGwgYWx3YXlzDQo+ICAgIG92ZXJ3cml0ZSB0aGUgZWFybGllci4NCj4gDQpUaGUg SUIgc3BlYyBpbmRlZWQgZG9lcyBub3QgaW1wbHkgYW55IG9yZGVyaW5nIGluIHRoZSBwbGFjZW1l bnQgb2YgZGF0YSBpbnRvIG1lbW9yeSB3aXRoaW4gYSBzaW5nbGUgbWVzc2FnZS4NCg0KSXQgZG9l cyBndWFyYW50ZWUgdGhhdCB3cml0ZXMgZG9uJ3QgYnlwYXNzIHdyaXRlcyBhbmQgcmVhZHMgZG9u J3QgYnlwYXNzIHJlYWRzIChUYWJsZSA3OSksIGFuZCB0cmFuc3BvcnQgb3BlcmF0aW9ucyBhcmUg ZXhlY3V0ZWQgaW4gdGhlaXIgKm1lc3NhZ2UqIG9yZGVyIChDOS0yOCk6DQoiQSByZXNwb25kZXIg c2hhbGwgZXhlY3V0ZSBTRU5EIHJlcXVlc3RzLCBSRE1BIFdSSVRFIHJlcXVlc3RzDQphbmQgQVRP TUlDIE9wZXJhdGlvbiByZXF1ZXN0cyBpbiB0aGUgbWVzc2FnZSBvcmRlciBpbiB3aGljaA0KdGhl eSBhcmUgcmVjZWl2ZWQuIg0KDQpUaHVzLCBvcmRlcmluZyBiZXR3ZWVuIG1lc3NhZ2VzIGlzIGd1 YXJhbnRlZWQgLSBjaGFuZ2VzIHRvIHJlbW90ZSBtZW1vcnkgb2YgYW4gUkRNQS1XIHdpbGwgYmUg b2JzZXJ2ZWQgc3RyaWN0bHkgYWZ0ZXIgYW55IGNoYW5nZXMgZG9uZSBieSBhIHByZXZpb3VzIFJE TUEtVzsgY2hhbmdlcyB0byBsb2NhbCBtZW1vcnkgb2YgYW4gUkRNQS1SIHJlc3BvbnNlIHdpbGwg YmUgb2JzZXJ2ZWQgc3RyaWN0bHkgYWZ0ZXIgYW55IGNoYW5nZXMgZG9uZSBieSBhIHByZXZpb3Vz IFJETUEtUiByZXNwb25zZS4NCg0KVGhlIHByb3Bvc2VkIGZlYXR1cmUgaW4gdGhpcyBwYXRjaCBz ZXQgaXMgdG8gcmVsYXggdGhlIG1lbW9yeSBwbGFjZW1lbnQgb3JkZXJpbmcgKmFjcm9zcyogbWVz c2FnZXMgYW5kIG5vdCB3aXRoaW4gYSBzaW5nbGUgbWVzc2FnZSAod2hpY2ggaXMgbm90IG1hbmRh dGVkIGJ5IHRoZSBzcGVjIGFzIHUgbm90ZWQpLCBzdWNoIHRoYXQgbXVsdGlwbGUgY29uc2VjdXRp dmUgUkRNQS1XcyBtYXkgYmUgY29tbWl0dGVkIHRvIG1lbW9yeSBpbiBhbnkgb3JkZXIsIGFuZCBz aW1pbGFybHkgZm9yIFJETUEtUiByZXNwb25zZXMuDQpUaGlzIGNoYW5nZXMgYXBwbGljYXRpb24g c2VtYW50aWNzIHdoZW5ldmVyIG11bHRpcGxlLWluZmxpZ2h0IFJETUEgb3BlcmF0aW9ucyB3cml0 ZSB0byBvdmVybGFwcGluZyBsb2NhdGlvbnMsIG9yIHdoZW4gb25lIG9wZXJhdGlvbiBpbmRpY2F0 ZXMgdGhlIGNvbXBsZXRpb24gb2YgdGhlIG90aGVyLg0KQSBzaW1wbGUgZXhhbXBsZSB0byBjbGFy aWZ5OiBhIHJlcXVlc3RvciBwb3N0ZWQgdGhlIGZvbGxvd2luZyB3b3JrIGVsZW1lbnRzIGluIHRo ZSB3cml0dGVuIG9yZGVyOg0KMS4gUkRNQS1XKFZBPTB4MTAwMCwgdmFsdWU9MHgxKQ0KMi4gUkRN QS1XKFZBPTB4MTAwMCwgdmFsdWU9MHgyKQ0KMy4gU2VuZCgpDQpPbiByZXNwb25kZXIgc2lkZSwg Zm9sbG93aW5nIHRoZSBTZW5kKCkgb3BlcmF0aW9uIGNvbXBsZXRpb24sIGFuZCBhY2NvcmRpbmcg dG8gc3BlYyAoQzktMjgpLCByZWFkaW5nIGZyb20gVkE9MHgxMDAwIHdpbGwgcHJvZHVjZSB0aGUg dmFsdWUgMi4gV2l0aCB0aGUgcHJvcG9zZWQgZmVhdHVyZSBlbmFibGVkLCB0aGUgcmVhZCB2YWx1 ZSBpcyBub3QgZGV0ZXJtaW5pc3RpYyBhbmQgZGVwZW5kZW50IG9uIHRoZSBvcmRlciBpbiB3aGlj aCB0aGUgUkRNQS1XIG9wZXJhdGlvbnMgd2VyZSByZWNlaXZlZC4NCg0KVGhlIHByb3Bvc2VkIFFQ IGZsYWcgYWxsb3dzIGFwcGxpY2F0aW9ucyB0byBrbm93aW5nbHkgaW5kaWNhdGUgdGhpcyByZWxh eGVkIGRhdGEgcGxhY2VtZW50LCB0aGVyZWJ5IGVuYWJsaW5nIHRoZSBIQ0EgdG8gcGxhY2UgT09P IFJETUEgbWVzc2FnZXMgaW50byBtZW1vcnkgd2l0aG91dCBidWZmZXJpbmcgdGhlbS4NCg0KPiBJ IGhhdmUgb25lIG90aGVyIHF1ZXN0aW9uIG9uIHRoZSBEb2N1bWVudGF0aW9uIG91dC1vZi1vcmRl ci50eHQuDQo+IEl0IHN0YXRlcyB0aGUgZmVuY2UgYml0IGNhbiBiZSB1c2VkIHRvIGZvcmNlIG9y ZGVyaW5nIG9uIGEgbm9uLXN0cmljdCBjb25uZWN0aW9uLg0KPiBCdXQgZmVuY2UgZG9lc24ndCBh cHBseSB0byBSRE1BIFdyaXRlPw0KPiBJdCBvbmx5IGFwcGxpZXMgdG8gb3BlcmF0aW9ucyB3aGlj aCBwcm9kdWNlIGEgcmVwbHksIHN1Y2ggYXMgUkRNQSBSZWFkIG9yDQo+IEF0b21pYy4gSGF2ZSB5 b3UgY2hhbmdlZCB0aGUgc2VtYW50aWM/DQo+IA0KUkRNQS1SIGZvbGxvd2VkIGJ5IFJETUEtUiBz ZW1hbnRpYyBpcyBjaGFuZ2VkIHdoZW4gcHJvcG9zZWQgUVAgZmxhZyBpcyBzZXQuDQo= --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

At a minimum, you need to include a version of table 79 in your documentation that reflects the modified WR ordering rules for this mode, and I'm not sure we should continue to call this thing a 'RC QP' at all to avoid so much confusion.

Particularly since this new thing is not really wire interoperable with the existing RC QP I don't think it should be the same thing in the API.

I recommend you rework this patch series to introduce a relaxed RC QP type instead of using a flag..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Sure. In fact I had it in internal version before posting upstream, but dropped for some reason. I will add it in v1.

1. None of the Reliability property is affected with this optional attribute. So it is still RC QP.

All this details we discussed our email will be described in documentation v1 to avoid confusion.

Additionally,
2. It doesn't make sense to introduce 3 more relaxed QP type (RC, XRC initiator, target) for one common attribute among them.
3. New QP type makes it more difficult or impossible to make rdmacm work in future.
Active side creates the QP and shared parameters through REQ-RSP messages.
Such existing interface would just become impossible to make use of.
This attribute is similar to MTU, IRD/ORD etc.
I prefer to keep it as optional QP attribute that brings simplicity and it is extendible compare to new QP type. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

You didn't answer my question what is the actual benefit of relaxing the ordering. Is it performance? And, specifically what applications *can't* use it?

To me, it appears that most storage upper layers can already use the extension. If it performs better, I expect they will definitely want to enable it. In that case I believe it should be the *default*, not an opt-in that these upper layers are newly responsible for.

Can you explain that statement in more detail please? Also, please clarify on what operation(s) the fence bit now applies.

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxw ZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogRnJpZGF5LCBKdWx5IDIxLCAyMDE3 IDk6MjkgUE0NCj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFzb24g R3VudGhvcnBlDQo+IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiBDYzogQmFy dCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9y ZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElk YW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUmU6IFtQQVRD SCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiAN Cj4gT24gNy8xOC8yMDE3IDEwOjMzIFBNLCBQYXJhdiBQYW5kaXQgd3JvdGU6DQo+ID4gSGkgVG9t LCBKYXNvbiwNCj4gPg0KPiA+IFNvcnJ5IGZvciB0aGUgbGF0ZSByZXNwb25zZS4NCj4gPiBQbGVh c2UgZmluZCB0aGUgcmVzcG9uc2UgaW5saW5lIGJlbG93Lg0KPiA+DQo+ID4+IC0tLS0tT3JpZ2lu YWwgTWVzc2FnZS0tLS0tDQo+ID4+IEZyb206IFRvbSBUYWxwZXkgW21haWx0bzp0b21AdGFscGV5 LmNvbV0NCj4gPj4gU2VudDogTW9uZGF5LCBKdW5lIDEyLCAyMDE3IDg6MzAgUE0NCj4gPj4gVG86 IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFzb24gR3VudGhvcnBlDQo+ID4+ IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiA+PiBDYzogQmFydCBWYW4gQXNz Y2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVsLm9yZzsNCj4gPj4g ZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElkYW4gQnVy c3RlaW4NCj4gPj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gPj4gU3ViamVjdDogUmU6IFtQQVRD SCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhDQo+ID4+IHBsYWNlbWVu dA0KPiA+Pg0KPiA+Pj4NCj4gPj4+IEluIElCIHNwZWMsIGluLW9yZGVyIGRlbGl2ZXJ5IGlzIGRl ZmF1bHQuDQo+ID4+DQo+ID4+IEkgZG9uJ3QgYWdyZWUuIFJlcXVlc3RzIGFyZSBzZW50IGluLW9y ZGVyLCBhbmQgdGhlIHJlc3BvbmRlcg0KPiA+PiBwcm9jZXNzZXMgdGhlbSBpbi0gb3JkZXIsIGJ1 dCB0aGUgYnl0ZXMgdGhlbnNlbHZlcyBhcmUgbm90IGd1YXJhbnRlZWQgdG8NCj4gYXBwZWFyIGlu LW9yZGVyLg0KPiA+PiBBZGRpdGlvbmFsbHksIGlmIHJldHJpZXMgb2NjdXIsIHRoaXMgaXMgbW9z dCBkZWZpbml0ZWx5IG5vdCB0aGUgY2FzZS4NCj4gPj4NCj4gPj4gU2VjdGlvbiA5LjUgVHJhbnNh Y3Rpb24gT3JkZXJpbmcsIEkgYmVsaWV2ZSwgY292ZXJzIHRoZXNlDQo+ID4+IHJlcXVpcmVtZW50 cy4gQ2FuIHlvdSB0ZWxsIG1lIHdoZXJlIEkgbWlzdW5kZXJzdGFuZCB0aGVtPw0KPiA+PiBJbiBm YWN0LCBjOS0yOCBleHBsaWNpdGx5IHdhcm5zOg0KPiA+Pg0KPiA+PiAgICAg4oCiIEFuIGFwcGxp Y2F0aW9uIHNoYWxsIG5vdCBkZXBlbmQgdXBvbiB0aGUgb3JkZXIgb2YgZGF0YSB3cml0ZXMgdG8N Cj4gPj4gICAgIG1lbW9yeSB3aXRoaW4gYSBtZXNzYWdlLiBGb3IgZXhhbXBsZSwgaWYgYW4gYXBw bGljYXRpb24gc2V0cyB1cA0KPiA+PiAgICAgZGF0YSBidWZmZXJzIHRoYXQgb3ZlcmxhcCwgZm9y IHNlcGFyYXRlIGRhdGEgc2VnbWVudHMgd2l0aGluIGENCj4gPj4gICAgIG1lc3NhZ2UsIGl0IGlz IG5vdCBndWFyYW50ZWVkIHRoYXQgdGhlIGxhc3Qgc2VudCBkYXRhIHdpbGwgYWx3YXlzDQo+ID4+ ICAgICBvdmVyd3JpdGUgdGhlIGVhcmxpZXIuDQo+ID4+DQo+ID4gVGhlIElCIHNwZWMgaW5kZWVk IGRvZXMgbm90IGltcGx5IGFueSBvcmRlcmluZyBpbiB0aGUgcGxhY2VtZW50IG9mIGRhdGEgaW50 bw0KPiBtZW1vcnkgd2l0aGluIGEgc2luZ2xlIG1lc3NhZ2UuDQo+ID4NCj4gPiBJdCBkb2VzIGd1 YXJhbnRlZSB0aGF0IHdyaXRlcyBkb24ndCBieXBhc3Mgd3JpdGVzIGFuZCByZWFkcyBkb24ndCBi eXBhc3MgcmVhZHMNCj4gKFRhYmxlIDc2KSwgYW5kIHRyYW5zcG9ydCBvcGVyYXRpb25zIGFyZSBl eGVjdXRlZCBpbiB0aGVpciAqbWVzc2FnZSogb3JkZXIgKEM5LQ0KPiAyOCk6DQo+ID4gIkEgcmVz cG9uZGVyIHNoYWxsIGV4ZWN1dGUgU0VORCByZXF1ZXN0cywgUkRNQSBXUklURSByZXF1ZXN0cyBh bmQNCj4gPiBBVE9NSUMgT3BlcmF0aW9uIHJlcXVlc3RzIGluIHRoZSBtZXNzYWdlIG9yZGVyIGlu IHdoaWNoIHRoZXkgYXJlDQo+ID4gcmVjZWl2ZWQuIg0KPiA+DQo+ID4gVGh1cywgb3JkZXJpbmcg YmV0d2VlbiBtZXNzYWdlcyBpcyBndWFyYW50ZWVkIC0gY2hhbmdlcyB0byByZW1vdGUgbWVtb3J5 DQo+IG9mIGFuIFJETUEtVyB3aWxsIGJlIG9ic2VydmVkIHN0cmljdGx5IGFmdGVyIGFueSBjaGFu Z2VzIGRvbmUgYnkgYSBwcmV2aW91cw0KPiBSRE1BLVc7IGNoYW5nZXMgdG8gbG9jYWwgbWVtb3J5 IG9mIGFuIFJETUEtUiByZXNwb25zZSB3aWxsIGJlIG9ic2VydmVkDQo+IHN0cmljdGx5IGFmdGVy IGFueSBjaGFuZ2VzIGRvbmUgYnkgYSBwcmV2aW91cyBSRE1BLVIgcmVzcG9uc2UuDQo+ID4NCj4g PiBUaGUgcHJvcG9zZWQgZmVhdHVyZSBpbiB0aGlzIHBhdGNoIHNldCBpcyB0byByZWxheCB0aGUg bWVtb3J5IHBsYWNlbWVudA0KPiBvcmRlcmluZyAqYWNyb3NzKiBtZXNzYWdlcyBhbmQgbm90IHdp dGhpbiBhIHNpbmdsZSBtZXNzYWdlICh3aGljaCBpcyBub3QNCj4gbWFuZGF0ZWQgYnkgdGhlIHNw ZWMgYXMgdSBub3RlZCksIHN1Y2ggdGhhdCBtdWx0aXBsZSBjb25zZWN1dGl2ZSBSRE1BLVdzDQo+ IG1heSBiZSBjb21taXR0ZWQgdG8gbWVtb3J5IGluIGFueSBvcmRlciwgYW5kIHNpbWlsYXJseSBm b3IgUkRNQS1SIHJlc3BvbnNlcy4NCj4gPiBUaGlzIGNoYW5nZXMgYXBwbGljYXRpb24gc2VtYW50 aWNzIHdoZW5ldmVyIG11bHRpcGxlLWluZmxpZ2h0IFJETUENCj4gb3BlcmF0aW9ucyB3cml0ZSB0 byBvdmVybGFwcGluZyBsb2NhdGlvbnMsIG9yIHdoZW4gb25lIG9wZXJhdGlvbiBpbmRpY2F0ZXMg dGhlDQo+IGNvbXBsZXRpb24gb2YgdGhlIG90aGVyLg0KPiA+IEEgc2ltcGxlIGV4YW1wbGUgdG8g Y2xhcmlmeTogYSByZXF1ZXN0b3IgcG9zdGVkIHRoZSBmb2xsb3dpbmcgd29yayBlbGVtZW50cyBp bg0KPiB0aGUgd3JpdHRlbiBvcmRlcjoNCj4gPiAxLiBSRE1BLVcoVkE9MHgxMDAwLCB2YWx1ZT0w eDEpDQo+ID4gMi4gUkRNQS1XKFZBPTB4MTAwMCwgdmFsdWU9MHgyKQ0KPiA+IDMuIFNlbmQoKQ0K PiA+IE9uIHJlc3BvbmRlciBzaWRlLCBmb2xsb3dpbmcgdGhlIFNlbmQoKSBvcGVyYXRpb24gY29t cGxldGlvbiwgYW5kIGFjY29yZGluZw0KPiB0byBzcGVjIChDOS0yOCksIHJlYWRpbmcgZnJvbSBW QT0weDEwMDAgd2lsbCBwcm9kdWNlIHRoZSB2YWx1ZSAyLiBXaXRoIHRoZQ0KPiBwcm9wb3NlZCBm ZWF0dXJlIGVuYWJsZWQsIHRoZSByZWFkIHZhbHVlIGlzIG5vdCBkZXRlcm1pbmlzdGljIGFuZCBk ZXBlbmRlbnQgb24NCj4gdGhlIG9yZGVyIGluIHdoaWNoIHRoZSBSRE1BLVcgb3BlcmF0aW9ucyB3 ZXJlIHJlY2VpdmVkLg0KPiA+DQo+ID4gVGhlIHByb3Bvc2VkIFFQIGZsYWcgYWxsb3dzIGFwcGxp Y2F0aW9ucyB0byBrbm93aW5nbHkgaW5kaWNhdGUgdGhpcyByZWxheGVkDQo+IGRhdGEgcGxhY2Vt ZW50LCB0aGVyZWJ5IGVuYWJsaW5nIHRoZSBIQ0EgdG8gcGxhY2UgT09PIFJETUEgbWVzc2FnZXMg aW50bw0KPiBtZW1vcnkgd2l0aG91dCBidWZmZXJpbmcgdGhlbS4NCj4gDQo+IFlvdSBkaWRuJ3Qg YW5zd2VyIG15IHF1ZXN0aW9uIHdoYXQgaXMgdGhlIGFjdHVhbCBiZW5lZml0IG9mIHJlbGF4aW5n IHRoZQ0KPiBvcmRlcmluZy4gSXMgaXQgcGVyZm9ybWFuY2U/DQoNClllcy4gUGVyZm9ybWFuY2Ug aXMgYmV0dGVyLg0KDQo+IEFuZCwgc3BlY2lmaWNhbGx5IHdoYXQgYXBwbGljYXRpb25zICpjYW4n dCogdXNlIGl0Pw0KQXBwbGljYXRpb25zIHdoaWNoIHBvbGwgb24gUkRNQS1XIGRhdGEgYXQgcmVz cG9uZGVyIHNpZGUgb3IgUkRNQS1SIGRhdGEgb24gUmVhZCByZXF1ZXN0ZXIgc2lkZSwgY2Fubm90 IHVzZSB0aGlzLg0KQmVjYXVzZSBhcyBleHBsYWluZWQgaW4gYWJvdmUgZXhhbXBsZSAybmQgUkRN QS1XIG1lc3NhZ2UgY2FuIGJlIGV4ZWN1dGVkIGZpcnN0IGF0IHJlc3BvbmRlciBzaWRlLg0KV2Ug Y2Fubm90IGJyZWFrIHN1Y2ggdXNlciBzcGFjZSBhcHBsaWNhdGlvbnMgZGVwbG95ZWQgaW4gZmll bGQgYnkgZW5hYmxpbmcgdGhpcyBieSBkZWZhdWx0IGFuZCB3aXRob3V0IG5lZ290aWF0aW9uIHdp dGggcGVlci4NCj4gDQo+IFRvIG1lLCBpdCBhcHBlYXJzIHRoYXQgbW9zdCBzdG9yYWdlIHVwcGVy IGxheWVycyBjYW4gYWxyZWFkeSB1c2UgdGhlIGV4dGVuc2lvbi4NCg0KWWVzLiBBcyB0aGV5IGRv bid0IHBvbGwgb24gZGF0YSBhbmQgdGhleSBkZXBlbmQgb24gaW5jb21pbmcgUkRNQSBTZW5kIHRo ZXkgY2FuIG1ha2UgdXNlIG9mIGl0Lg0KDQo+IElmIGl0IHBlcmZvcm1zIGJldHRlciwgSSBleHBl Y3QgdGhleSB3aWxsIGRlZmluaXRlbHkgd2FudCB0byBlbmFibGUgaXQuIEluIHRoYXQgY2FzZSBJ DQo+IGJlbGlldmUgaXQgc2hvdWxkIGJlIHRoZSAqZGVmYXVsdCosIG5vdCBhbiBvcHQtaW4gdGhh dCB0aGVzZSB1cHBlciBsYXllcnMgYXJlDQo+IG5ld2x5IHJlc3BvbnNpYmxlIGZvci4NCg0KVmVy YiBsYXllciBpcyB1bndhcmUgb2YgY2FsbGVyIFVMUHMuIEF0IG1vc3QgaXQga25vd3MgdGhhdCBp dHMga2VybmVsIHZzIHVzZXIgVUxQIGVhc2lseSAtIHdoaWNoIGlzIGdvb2QgZW5vdWdoLg0KVmVy YiBsYXllciBhbHNvIGRvZXNuJ3Qga25vdyB3aGV0aGVyIHJlbW90ZSBzaWRlIHN1cHBvcnQgaXQg b3Igbm90Lg0KT25jZSByZG1hY20gZXh0ZW5zaW9uIGlzIGRvbmUsIGFsbCBrZXJuZWwgVUxQcyB3 aGljaCB1c2VzIHJkbWFjbSAtIGNhbiBiZSBlbmFibGVkIGJ5IGRlZmF1bHQuDQpUaGlzIHBhdGNo c2V0IGVuYWJsZXMgdXNlciBzcGFjZSBhcHBsaWNhdGlvbnMgdG8gdGFrZSBpbW1lZGlhdGUgYmVu ZWZpdCBvZiBpdCB3aGljaCBkb2Vzbid0IGRlcGVuZCBvbiByZG1hY20uDQoNCj4gDQo+ID4+IEkg aGF2ZSBvbmUgb3RoZXIgcXVlc3Rpb24gb24gdGhlIERvY3VtZW50YXRpb24gb3V0LW9mLW9yZGVy LnR4dC4NCj4gPj4gSXQgc3RhdGVzIHRoZSBmZW5jZSBiaXQgY2FuIGJlIHVzZWQgdG8gZm9yY2Ug b3JkZXJpbmcgb24gYSBub24tc3RyaWN0DQo+IGNvbm5lY3Rpb24uDQo+ID4+IEJ1dCBmZW5jZSBk b2Vzbid0IGFwcGx5IHRvIFJETUEgV3JpdGU/DQo+ID4+IEl0IG9ubHkgYXBwbGllcyB0byBvcGVy YXRpb25zIHdoaWNoIHByb2R1Y2UgYSByZXBseSwgc3VjaCBhcyBSRE1BDQo+ID4+IFJlYWQgb3Ig QXRvbWljLiBIYXZlIHlvdSBjaGFuZ2VkIHRoZSBzZW1hbnRpYz8NCj4gPj4NCj4gPiBSRE1BLVIg Zm9sbG93ZWQgYnkgUkRNQS1SIHNlbWFudGljIGlzIGNoYW5nZWQgd2hlbiBwcm9wb3NlZCBRUCBm bGFnIGlzDQo+IHNldC4NCj4gDQo+IENhbiB5b3UgZXhwbGFpbiB0aGF0IHN0YXRlbWVudCBpbiBt b3JlIGRldGFpbCBwbGVhc2U/IEFsc28sIHBsZWFzZSBjbGFyaWZ5IG9uDQo+IHdoYXQgb3BlcmF0 aW9uKHMpIHRoZSBmZW5jZSBiaXQgbm93IGFwcGxpZXMuDQpTdXJlLg0KTGV0J3MgdGFrZSBleGFt cGxlLg0KQSByZXF1ZXN0b3IgcG9zdGVkIHRoZSBmb2xsb3dpbmcgd29yayBlbGVtZW50cyBpbiB0 aGUgd3JpdHRlbiBvcmRlcjoNCjEuIFJETUEtUihWQT0weDEwMDAsIGxlbj0weDQwMCkNCjIuIFJE TUEtUihWQT0weDE0MDAsIGxlbj0weDQpDQpDdXJyZW50bHkgYXMgcGVyIFRhYmxlLTc2LCBSRE1B LVIgcmVhZCByZXNwb25zZSBkYXRhIG9mIDFzdCBSRE1BLVIgaXMgcGxhY2VkIGZpcnN0Lg0KV2l0 aCByZWxheGVkIGRhdGEgcGxhY2VtZW50IGF0dHJpYnV0ZSwgNCBieXRlcyBkYXRhIG9mIDJuZCBS RE1BLVIgY2FuIGJlIHBsYWNlZCBmaXJzdC4NCklmIHVzZXIgbmVlZHMgYSBvcmRlcmluZyBvZiBj dXJyZW50IFRhYmxlLTc2LCBpdCBuZWVkcyB0byBzZXQgZmVuY2Ugb24gMm5kIFJETUEtUi4NClRo aXMgd2lsbCBlbnN1cmUgdGhhdCAxc3QgUkRNQS1SIGlzIGV4ZWN1dGVkIGJlZm9yZSAybmQgUkRN QS1SLg0KDQpUaGlzIHRyYW5zbGF0ZXMgdG8gSW4gVGFibGUgNzYsIA0KUkRNQS1SIChSb3cpIGFu ZCBSRE1BLVIoQ29sdW1uKSBjaGFuZ2VzIGZyb20gJyMnIHRvICdGJy4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Those applications ignored the spec, and got away with it only because the Mellanox (is that who "we" is?) implementation was strongly ordered. Thats not much of an excuse, in my opinion, to force change on the well-behaved, spec-observing ULPs in order that they might take advantage of it.

Not without changing their protocols and implementations. I think you should reconsider your approach to throw the responsibility to them, and them only.

Well, then this change should wait for that to become available.

But it changes the API in a way that we don't want to survive. Let's get the interface right first.

Oh, that's the same issue a the initial one - polling the "last" bit was never guaranteed. I dont see this as a change to the semantic.

But, I take it that the fence bit still applies as before, this is not a proposal to extended fencing to RDMA Write. Ok.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgVG9tLA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxw ZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogU2F0dXJkYXksIEp1bHkgMjIsIDIw MTcgMTI6MDMgQU0NCj4gVG86IFBhcmF2IFBhbmRpdCA8cGFyYXZAbWVsbGFub3guY29tPjsgSmFz b24gR3VudGhvcnBlDQo+IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiBDYzog QmFydCBWYW4gQXNzY2hlIDxCYXJ0LlZhbkFzc2NoZUBzYW5kaXNrLmNvbT47IGxlb25Aa2VybmVs Lm9yZzsNCj4gZGxlZGZvcmRAcmVkaGF0LmNvbTsgbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7 IElkYW4gQnVyc3RlaW4NCj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUmU6IFtQ QVRDSCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0K PiANCj4gT24gNy8yMS8yMDE3IDk6NTAgUE0sIFBhcmF2IFBhbmRpdCB3cm90ZToNCj4gPiBIaSBU b20sDQo+ID4NCj4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTogVG9t IFRhbHBleSBbbWFpbHRvOnRvbUB0YWxwZXkuY29tXQ0KPiA+PiBTZW50OiBGcmlkYXksIEp1bHkg MjEsIDIwMTcgOToyOSBQTQ0KPiA+PiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5j b20+OyBKYXNvbiBHdW50aG9ycGUNCj4gPj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5j b20+DQo+ID4+IENjOiBCYXJ0IFZhbiBBc3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29t PjsgbGVvbkBrZXJuZWwub3JnOw0KPiA+PiBkbGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1h QHZnZXIua2VybmVsLm9yZzsgSWRhbiBCdXJzdGVpbg0KPiA+PiA8aWRhbmJAbWVsbGFub3guY29t Pg0KPiA+PiBTdWJqZWN0OiBSZTogW1BBVENIIHJkbWEtbmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9m IG9yZGVyIGRhdGENCj4gPj4gcGxhY2VtZW50DQo+ID4+DQo+ID4+IE9uIDcvMTgvMjAxNyAxMDoz MyBQTSwgUGFyYXYgUGFuZGl0IHdyb3RlOg0KPiA+Pj4gSGkgVG9tLCBKYXNvbiwNCj4gPj4+DQo+ ID4+PiBTb3JyeSBmb3IgdGhlIGxhdGUgcmVzcG9uc2UuDQo+ID4+PiBQbGVhc2UgZmluZCB0aGUg cmVzcG9uc2UgaW5saW5lIGJlbG93Lg0KPiA+Pj4NCj4gPj4+PiAtLS0tLU9yaWdpbmFsIE1lc3Nh Z2UtLS0tLQ0KPiA+Pj4+IEZyb206IFRvbSBUYWxwZXkgW21haWx0bzp0b21AdGFscGV5LmNvbV0N Cj4gPj4+PiBTZW50OiBNb25kYXksIEp1bmUgMTIsIDIwMTcgODozMCBQTQ0KPiA+Pj4+IFRvOiBQ YXJhdiBQYW5kaXQgPHBhcmF2QG1lbGxhbm94LmNvbT47IEphc29uIEd1bnRob3JwZQ0KPiA+Pj4+ IDxqZ3VudGhvcnBlQG9ic2lkaWFucmVzZWFyY2guY29tPg0KPiA+Pj4+IENjOiBCYXJ0IFZhbiBB c3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29tPjsgbGVvbkBrZXJuZWwub3JnOw0KPiA+ Pj4+IGRsZWRmb3JkQHJlZGhhdC5jb207IGxpbnV4LXJkbWFAdmdlci5rZXJuZWwub3JnOyBJZGFu IEJ1cnN0ZWluDQo+ID4+Pj4gPGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gPj4+PiBTdWJqZWN0OiBS ZTogW1BBVENIIHJkbWEtbmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9mIG9yZGVyIGRhdGENCj4gPj4+ PiBwbGFjZW1lbnQNCj4gPj4+Pg0KPiA+Pj4+Pg0KPiA+Pj4+PiBJbiBJQiBzcGVjLCBpbi1vcmRl ciBkZWxpdmVyeSBpcyBkZWZhdWx0Lg0KPiA+Pj4+DQo+ID4+Pj4gSSBkb24ndCBhZ3JlZS4gUmVx dWVzdHMgYXJlIHNlbnQgaW4tb3JkZXIsIGFuZCB0aGUgcmVzcG9uZGVyDQo+ID4+Pj4gcHJvY2Vz c2VzIHRoZW0gaW4tIG9yZGVyLCBidXQgdGhlIGJ5dGVzIHRoZW5zZWx2ZXMgYXJlIG5vdA0KPiA+ Pj4+IGd1YXJhbnRlZWQgdG8NCj4gPj4gYXBwZWFyIGluLW9yZGVyLg0KPiA+Pj4+IEFkZGl0aW9u YWxseSwgaWYgcmV0cmllcyBvY2N1ciwgdGhpcyBpcyBtb3N0IGRlZmluaXRlbHkgbm90IHRoZSBj YXNlLg0KPiA+Pj4+DQo+ID4+Pj4gU2VjdGlvbiA5LjUgVHJhbnNhY3Rpb24gT3JkZXJpbmcsIEkg YmVsaWV2ZSwgY292ZXJzIHRoZXNlDQo+ID4+Pj4gcmVxdWlyZW1lbnRzLiBDYW4geW91IHRlbGwg bWUgd2hlcmUgSSBtaXN1bmRlcnN0YW5kIHRoZW0/DQo+ID4+Pj4gSW4gZmFjdCwgYzktMjggZXhw bGljaXRseSB3YXJuczoNCj4gPj4+Pg0KPiA+Pj4+ICAgICAg4oCiIEFuIGFwcGxpY2F0aW9uIHNo YWxsIG5vdCBkZXBlbmQgdXBvbiB0aGUgb3JkZXIgb2YgZGF0YSB3cml0ZXMgdG8NCj4gPj4+PiAg ICAgIG1lbW9yeSB3aXRoaW4gYSBtZXNzYWdlLiBGb3IgZXhhbXBsZSwgaWYgYW4gYXBwbGljYXRp b24gc2V0cyB1cA0KPiA+Pj4+ICAgICAgZGF0YSBidWZmZXJzIHRoYXQgb3ZlcmxhcCwgZm9yIHNl cGFyYXRlIGRhdGEgc2VnbWVudHMgd2l0aGluIGENCj4gPj4+PiAgICAgIG1lc3NhZ2UsIGl0IGlz IG5vdCBndWFyYW50ZWVkIHRoYXQgdGhlIGxhc3Qgc2VudCBkYXRhIHdpbGwgYWx3YXlzDQo+ID4+ Pj4gICAgICBvdmVyd3JpdGUgdGhlIGVhcmxpZXIuDQo+ID4+Pj4NCj4gPj4+IFRoZSBJQiBzcGVj IGluZGVlZCBkb2VzIG5vdCBpbXBseSBhbnkgb3JkZXJpbmcgaW4gdGhlIHBsYWNlbWVudCBvZg0K PiA+Pj4gZGF0YSBpbnRvDQo+ID4+IG1lbW9yeSB3aXRoaW4gYSBzaW5nbGUgbWVzc2FnZS4NCj4g Pj4+DQo+ID4+PiBJdCBkb2VzIGd1YXJhbnRlZSB0aGF0IHdyaXRlcyBkb24ndCBieXBhc3Mgd3Jp dGVzIGFuZCByZWFkcyBkb24ndA0KPiA+Pj4gYnlwYXNzIHJlYWRzDQo+ID4+IChUYWJsZSA3Niks IGFuZCB0cmFuc3BvcnQgb3BlcmF0aW9ucyBhcmUgZXhlY3V0ZWQgaW4gdGhlaXIgKm1lc3NhZ2Uq DQo+ID4+IG9yZGVyIChDOS0NCj4gPj4gMjgpOg0KPiA+Pj4gIkEgcmVzcG9uZGVyIHNoYWxsIGV4 ZWN1dGUgU0VORCByZXF1ZXN0cywgUkRNQSBXUklURSByZXF1ZXN0cyBhbmQNCj4gPj4+IEFUT01J QyBPcGVyYXRpb24gcmVxdWVzdHMgaW4gdGhlIG1lc3NhZ2Ugb3JkZXIgaW4gd2hpY2ggdGhleSBh cmUNCj4gPj4+IHJlY2VpdmVkLiINCj4gPj4+DQo+ID4+PiBUaHVzLCBvcmRlcmluZyBiZXR3ZWVu IG1lc3NhZ2VzIGlzIGd1YXJhbnRlZWQgLSBjaGFuZ2VzIHRvIHJlbW90ZQ0KPiA+Pj4gbWVtb3J5 DQo+ID4+IG9mIGFuIFJETUEtVyB3aWxsIGJlIG9ic2VydmVkIHN0cmljdGx5IGFmdGVyIGFueSBj aGFuZ2VzIGRvbmUgYnkgYQ0KPiA+PiBwcmV2aW91cyBSRE1BLVc7IGNoYW5nZXMgdG8gbG9jYWwg bWVtb3J5IG9mIGFuIFJETUEtUiByZXNwb25zZSB3aWxsDQo+ID4+IGJlIG9ic2VydmVkIHN0cmlj dGx5IGFmdGVyIGFueSBjaGFuZ2VzIGRvbmUgYnkgYSBwcmV2aW91cyBSRE1BLVIgcmVzcG9uc2Uu DQo+ID4+Pg0KPiA+Pj4gVGhlIHByb3Bvc2VkIGZlYXR1cmUgaW4gdGhpcyBwYXRjaCBzZXQgaXMg dG8gcmVsYXggdGhlIG1lbW9yeQ0KPiA+Pj4gcGxhY2VtZW50DQo+ID4+IG9yZGVyaW5nICphY3Jv c3MqIG1lc3NhZ2VzIGFuZCBub3Qgd2l0aGluIGEgc2luZ2xlIG1lc3NhZ2UgKHdoaWNoIGlzDQo+ ID4+IG5vdCBtYW5kYXRlZCBieSB0aGUgc3BlYyBhcyB1IG5vdGVkKSwgc3VjaCB0aGF0IG11bHRp cGxlIGNvbnNlY3V0aXZlDQo+ID4+IFJETUEtV3MgbWF5IGJlIGNvbW1pdHRlZCB0byBtZW1vcnkg aW4gYW55IG9yZGVyLCBhbmQgc2ltaWxhcmx5IGZvcg0KPiBSRE1BLVIgcmVzcG9uc2VzLg0KPiA+ Pj4gVGhpcyBjaGFuZ2VzIGFwcGxpY2F0aW9uIHNlbWFudGljcyB3aGVuZXZlciBtdWx0aXBsZS1p bmZsaWdodCBSRE1BDQo+ID4+IG9wZXJhdGlvbnMgd3JpdGUgdG8gb3ZlcmxhcHBpbmcgbG9jYXRp b25zLCBvciB3aGVuIG9uZSBvcGVyYXRpb24NCj4gPj4gaW5kaWNhdGVzIHRoZSBjb21wbGV0aW9u IG9mIHRoZSBvdGhlci4NCj4gPj4+IEEgc2ltcGxlIGV4YW1wbGUgdG8gY2xhcmlmeTogYSByZXF1 ZXN0b3IgcG9zdGVkIHRoZSBmb2xsb3dpbmcgd29yaw0KPiA+Pj4gZWxlbWVudHMgaW4NCj4gPj4g dGhlIHdyaXR0ZW4gb3JkZXI6DQo+ID4+PiAxLiBSRE1BLVcoVkE9MHgxMDAwLCB2YWx1ZT0weDEp DQo+ID4+PiAyLiBSRE1BLVcoVkE9MHgxMDAwLCB2YWx1ZT0weDIpDQo+ID4+PiAzLiBTZW5kKCkN Cj4gPj4+IE9uIHJlc3BvbmRlciBzaWRlLCBmb2xsb3dpbmcgdGhlIFNlbmQoKSBvcGVyYXRpb24g Y29tcGxldGlvbiwgYW5kDQo+ID4+PiBhY2NvcmRpbmcNCj4gPj4gdG8gc3BlYyAoQzktMjgpLCBy ZWFkaW5nIGZyb20gVkE9MHgxMDAwIHdpbGwgcHJvZHVjZSB0aGUgdmFsdWUgMi4NCj4gPj4gV2l0 aCB0aGUgcHJvcG9zZWQgZmVhdHVyZSBlbmFibGVkLCB0aGUgcmVhZCB2YWx1ZSBpcyBub3QNCj4g Pj4gZGV0ZXJtaW5pc3RpYyBhbmQgZGVwZW5kZW50IG9uIHRoZSBvcmRlciBpbiB3aGljaCB0aGUg UkRNQS1XIG9wZXJhdGlvbnMNCj4gd2VyZSByZWNlaXZlZC4NCj4gPj4+DQo+ID4+PiBUaGUgcHJv cG9zZWQgUVAgZmxhZyBhbGxvd3MgYXBwbGljYXRpb25zIHRvIGtub3dpbmdseSBpbmRpY2F0ZSB0 aGlzDQo+ID4+PiByZWxheGVkDQo+ID4+IGRhdGEgcGxhY2VtZW50LCB0aGVyZWJ5IGVuYWJsaW5n IHRoZSBIQ0EgdG8gcGxhY2UgT09PIFJETUEgbWVzc2FnZXMNCj4gPj4gaW50byBtZW1vcnkgd2l0 aG91dCBidWZmZXJpbmcgdGhlbS4NCj4gPj4NCj4gPj4gWW91IGRpZG4ndCBhbnN3ZXIgbXkgcXVl c3Rpb24gd2hhdCBpcyB0aGUgYWN0dWFsIGJlbmVmaXQgb2YgcmVsYXhpbmcNCj4gPj4gdGhlIG9y ZGVyaW5nLiBJcyBpdCBwZXJmb3JtYW5jZT8NCj4gPg0KPiA+IFllcy4gUGVyZm9ybWFuY2UgaXMg YmV0dGVyLg0KPiA+DQo+ID4+IEFuZCwgc3BlY2lmaWNhbGx5IHdoYXQgYXBwbGljYXRpb25zICpj YW4ndCogdXNlIGl0Pw0KPiA+IEFwcGxpY2F0aW9ucyB3aGljaCBwb2xsIG9uIFJETUEtVyBkYXRh IGF0IHJlc3BvbmRlciBzaWRlIG9yIFJETUEtUiBkYXRhIG9uDQo+IFJlYWQgcmVxdWVzdGVyIHNp ZGUsIGNhbm5vdCB1c2UgdGhpcy4NCj4gPiBCZWNhdXNlIGFzIGV4cGxhaW5lZCBpbiBhYm92ZSBl eGFtcGxlIDJuZCBSRE1BLVcgbWVzc2FnZSBjYW4gYmUNCj4gZXhlY3V0ZWQgZmlyc3QgYXQgcmVz cG9uZGVyIHNpZGUuDQo+ID4gV2UgY2Fubm90IGJyZWFrIHN1Y2ggdXNlciBzcGFjZSBhcHBsaWNh dGlvbnMgZGVwbG95ZWQgaW4gZmllbGQgYnkgZW5hYmxpbmcNCj4gdGhpcyBieSBkZWZhdWx0IGFu ZCB3aXRob3V0IG5lZ290aWF0aW9uIHdpdGggcGVlci4NCj4gDQo+IFRob3NlIGFwcGxpY2F0aW9u cyBpZ25vcmVkIHRoZSBzcGVjLCBhbmQgZ290IGF3YXkgd2l0aCBpdCBvbmx5IGJlY2F1c2UgdGhl DQo+IE1lbGxhbm94IChpcyB0aGF0IHdobyAid2UiIGlzPykgaW1wbGVtZW50YXRpb24gd2FzIHN0 cm9uZ2x5IG9yZGVyZWQuIFRoYXRzIG5vdA0KPiBtdWNoIG9mIGFuIGV4Y3VzZSwgaW4gbXkgb3Bp bmlvbiwgdG8gZm9yY2UgY2hhbmdlIG9uIHRoZSB3ZWxsLWJlaGF2ZWQsIHNwZWMtDQo+IG9ic2Vy dmluZyBVTFBzIGluIG9yZGVyIHRoYXQgdGhleSBtaWdodCB0YWtlIGFkdmFudGFnZSBvZiBpdC4N Cj4gDQpBcyB0YWxrZWQgdGhyb3VnaCBUYWJsZS03NiwgQzktMjgsIGN1cnJlbnQgSUIgc3BlYyBh c3N1cmVzIHRoYXQgUkRNQS1SIG9mIDQgYnl0ZXMgaXMgZXhlY3V0ZWQgYWZ0ZXIgUkRNQS1SIG9m IDFLIGlzIGV4ZWN1dGVkLg0KQXBwbGljYXRpb24gZGlkbid0IGFkaGVyZSB0byBvcHRpb25hbCBy ZXF1aXJlbWVudCBvOS0yMCwgbzktMjEuDQpJIGRvbid0IHNlZSBhIHJlYXNvbiBvbiB3aHkgc3Vj aCBhcHBsaWNhdGlvbnMgc2hvdWxkIGJlIGJyb2tlbiB3aGVuIHdlIGFscmVhZHkgaGF2ZSBhIHdh eSBhdm9pZCB0aGF0Lg0KDQo+ID4+IFRvIG1lLCBpdCBhcHBlYXJzIHRoYXQgbW9zdCBzdG9yYWdl IHVwcGVyIGxheWVycyBjYW4gYWxyZWFkeSB1c2UgdGhlDQo+IGV4dGVuc2lvbi4NCj4gPg0KPiA+ IFllcy4gQXMgdGhleSBkb24ndCBwb2xsIG9uIGRhdGEgYW5kIHRoZXkgZGVwZW5kIG9uIGluY29t aW5nIFJETUEgU2VuZCB0aGV5DQo+IGNhbiBtYWtlIHVzZSBvZiBpdC4NCj4gDQo+IE5vdCB3aXRo b3V0IGNoYW5naW5nIHRoZWlyIHByb3RvY29scyBhbmQgaW1wbGVtZW50YXRpb25zLiBJIHRoaW5r IHlvdSBzaG91bGQNCj4gcmVjb25zaWRlciB5b3VyIGFwcHJvYWNoIHRvIHRocm93IHRoZSByZXNw b25zaWJpbGl0eSB0byB0aGVtLCBhbmQgdGhlbSBvbmx5Lg0KPiANCkFwcHJvYWNoIGlzIG9wZW4g Y3VycmVudGx5IGF0IGxlYXN0IHdpdGggdHdvIG9wdGlvbnMuDQoxLiBFaXRoZXIgaXQgY2FuIGJl IGRvbmUgaW4gY29yZSBsYXllciBmb3Iga2VybmVsIFVMUHMgdG8gZW5hYmxlIGJ5IGRlZmF1bHQg d2l0aCBwZWVyIG5lZ290aWF0aW9uIHRyYW5zcGFyZW50IHRvIFVMUHMuDQpPcg0KMi4gVUxQIGdl dHMgZXhwbGljaXQgY29udHJvbCB0byBlbmFibGUvZGlzYWJsZSBpdCwgc2ltaWxhciB0byBvdGhl ciBjb25uZWN0aW9uIHBhcmFtZXRlcnMuDQpUaGlzIHBhdGNoIGlzIGEgbGF5ZXIgYmVsb3cgaXQg YW5kIGl0cyB1bmFmZmVjdGVkIGJ5IGFib3ZlIGxheWVycy4NCg0KPiA+PiBJZiBpdCBwZXJmb3Jt cyBiZXR0ZXIsIEkgZXhwZWN0IHRoZXkgd2lsbCBkZWZpbml0ZWx5IHdhbnQgdG8gZW5hYmxlDQo+ ID4+IGl0LiBJbiB0aGF0IGNhc2UgSSBiZWxpZXZlIGl0IHNob3VsZCBiZSB0aGUgKmRlZmF1bHQq LCBub3QgYW4gb3B0LWluDQo+ID4+IHRoYXQgdGhlc2UgdXBwZXIgbGF5ZXJzIGFyZSBuZXdseSBy ZXNwb25zaWJsZSBmb3IuDQo+ID4NCj4gPiBWZXJiIGxheWVyIGlzIHVud2FyZSBvZiBjYWxsZXIg VUxQcy4gQXQgbW9zdCBpdCBrbm93cyB0aGF0IGl0cyBrZXJuZWwgdnMgdXNlciBVTFANCj4gZWFz aWx5IC0gd2hpY2ggaXMgZ29vZCBlbm91Z2guDQo+ID4gVmVyYiBsYXllciBhbHNvIGRvZXNuJ3Qg a25vdyB3aGV0aGVyIHJlbW90ZSBzaWRlIHN1cHBvcnQgaXQgb3Igbm90Lg0KPiA+IE9uY2UgcmRt YWNtIGV4dGVuc2lvbiBpcyBkb25lLCBhbGwga2VybmVsIFVMUHMgd2hpY2ggdXNlcyByZG1hY20g LSBjYW4gYmUNCj4gZW5hYmxlZCBieSBkZWZhdWx0Lg0KPiANCj4gV2VsbCwgdGhlbiB0aGlzIGNo YW5nZSBzaG91bGQgd2FpdCBmb3IgdGhhdCB0byBiZWNvbWUgYXZhaWxhYmxlLg0KS2VybmVsIHBy b3ZpZGVzIHRoZSBzZXJ2aWNlIHRvIHVzZXIgYXBwbGljYXRpb25zIGFuZCBrZXJuZWwgVUxQcyBi b3RoLg0KVGhpcyBhdHRyaWJ1dGUgaXMgbGF5ZXIgYmVsb3cgc3VjaCBhcHBsaWNhdGlvbnMuDQpJ IGRvbid0IHNlZSBhIHJlYXNvbiB0byBwdXQgZGVwZW5kZW5jeSBvbiBjb25uZWN0aW9uIG1hbmFn ZXIgZm9yIHRob3NlIGFwcGxpY2F0aW9ucyB3aGljaCBkb2Vzbid0IHVzZSBzdWNoIGNvbm5lY3Rp b24gbWFuYWdlci4NClJkbWFjbSB3b3VsZCBiZSBhbiBleHRlbnNpb24gb24gdG9wIG9mIHRoaXMg LSB0aGF0IGNhbiBtYWtlIHVzZSBvZiB0aGlzIGF0dHJpYnV0ZS4NCg0KPiANCj4gPiBUaGlzIHBh dGNoc2V0IGVuYWJsZXMgdXNlciBzcGFjZSBhcHBsaWNhdGlvbnMgdG8gdGFrZSBpbW1lZGlhdGUg YmVuZWZpdCBvZiBpdA0KPiB3aGljaCBkb2Vzbid0IGRlcGVuZCBvbiByZG1hY20uDQo+IA0KPiBC dXQgaXQgY2hhbmdlcyB0aGUgQVBJIGluIGEgd2F5IHRoYXQgd2UgZG9uJ3Qgd2FudCB0byBzdXJ2 aXZlLg0KPiBMZXQncyBnZXQgdGhlIGludGVyZmFjZSByaWdodCBmaXJzdC4NCg0KVGhpcyBpcyBh IFFQIGF0dHJpYnV0ZSBzaW1pbGFyIHRvIG1hbnkgb3RoZXIgUVAgYXR0cmlidXRlcyB0aGF0IFVM UCBjYW4gc2V0IGFwcHJvcHJpYXRlbHkuDQo+IA0KPiA+Pj4gUkRNQS1SIGZvbGxvd2VkIGJ5IFJE TUEtUiBzZW1hbnRpYyBpcyBjaGFuZ2VkIHdoZW4gcHJvcG9zZWQgUVAgZmxhZw0KPiA+Pj4gaXMN Cj4gPj4gc2V0Lg0KPiA+Pg0KPiA+PiBDYW4geW91IGV4cGxhaW4gdGhhdCBzdGF0ZW1lbnQgaW4g bW9yZSBkZXRhaWwgcGxlYXNlPyBBbHNvLCBwbGVhc2UNCj4gPj4gY2xhcmlmeSBvbiB3aGF0IG9w ZXJhdGlvbihzKSB0aGUgZmVuY2UgYml0IG5vdyBhcHBsaWVzLg0KPiA+IFN1cmUuDQo+ID4gTGV0 J3MgdGFrZSBleGFtcGxlLg0KPiA+IEEgcmVxdWVzdG9yIHBvc3RlZCB0aGUgZm9sbG93aW5nIHdv cmsgZWxlbWVudHMgaW4gdGhlIHdyaXR0ZW4gb3JkZXI6DQo+ID4gMS4gUkRNQS1SKFZBPTB4MTAw MCwgbGVuPTB4NDAwKQ0KPiA+IDIuIFJETUEtUihWQT0weDE0MDAsIGxlbj0weDQpDQo+ID4gQ3Vy cmVudGx5IGFzIHBlciBUYWJsZS03NiwgUkRNQS1SIHJlYWQgcmVzcG9uc2UgZGF0YSBvZiAxc3Qg UkRNQS1SIGlzDQo+IHBsYWNlZCBmaXJzdC4NCj4gPiBXaXRoIHJlbGF4ZWQgZGF0YSBwbGFjZW1l bnQgYXR0cmlidXRlLCA0IGJ5dGVzIGRhdGEgb2YgMm5kIFJETUEtUiBjYW4gYmUNCj4gcGxhY2Vk IGZpcnN0Lg0KPiA+IElmIHVzZXIgbmVlZHMgYSBvcmRlcmluZyBvZiBjdXJyZW50IFRhYmxlLTc2 LCBpdCBuZWVkcyB0byBzZXQgZmVuY2Ugb24gMm5kDQo+IFJETUEtUi4NCj4gPiBUaGlzIHdpbGwg ZW5zdXJlIHRoYXQgMXN0IFJETUEtUiBpcyBleGVjdXRlZCBiZWZvcmUgMm5kIFJETUEtUi4NCj4g DQo+IE9oLCB0aGF0J3MgdGhlIHNhbWUgaXNzdWUgYSB0aGUgaW5pdGlhbCBvbmUgLSBwb2xsaW5n IHRoZSAibGFzdCINCj4gYml0IHdhcyBuZXZlciBndWFyYW50ZWVkLiBJIGRvbnQgc2VlIHRoaXMg YXMgYSBjaGFuZ2UgdG8gdGhlIHNlbWFudGljLg0KPiANCkl04oCZcyBhIGNsZWFyIGRldmlhdGlv biBmcm9tIFRhYmxlLTc2IGFuZCBDOS0yOCBhbmQgdGhlcmVmb3Igc2VtYW50aWMgY2hhbmdlIHRo YXQgZGVzZXJ2ZXMgYSBiaXQuDQoNCj4gQnV0LCBJIHRha2UgaXQgdGhhdCB0aGUgZmVuY2UgYml0 IHN0aWxsIGFwcGxpZXMgYXMgYmVmb3JlLCB0aGlzIGlzIG5vdCBhIHByb3Bvc2FsIHRvDQo+IGV4 dGVuZGVkIGZlbmNpbmcgdG8gUkRNQSBXcml0ZS4gT2suDQoNClJlc3Qgb2YgdGhlIFRhYmxlIDc2 IHN0YXlzIGFzIGlzLg0KDQo+ID4gVGhpcyB0cmFuc2xhdGVzIHRvIEluIFRhYmxlIDc2LA0KPiA+ IFJETUEtUiAoUm93KSBhbmQgUkRNQS1SKENvbHVtbikgY2hhbmdlcyBmcm9tICcjJyB0byAnRicu DQo+ID4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Well, if the broken applications won't use the extension, and the existing storage protocols and applications will have to change both their implementation and their protocol to use it, who do you envision actually doing so?

Sorry, but I just don't see the point of making it optional.

Tom.

ʇڙ�,j��f���h���z�
�w���
���j:+v���w�j�m��������zZ+�����ݢj"��!tml=

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

DQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRvbSBUYWxwZXkgW21haWx0 bzp0b21AdGFscGV5LmNvbV0NCj4gU2VudDogU2F0dXJkYXksIEp1bHkgMjIsIDIwMTcgOTo1MiBB TQ0KPiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5jb20+OyBKYXNvbiBHdW50aG9y cGUNCj4gPGpndW50aG9ycGVAb2JzaWRpYW5yZXNlYXJjaC5jb20+DQo+IENjOiBCYXJ0IFZhbiBB c3NjaGUgPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2suY29tPjsgbGVvbkBrZXJuZWwub3JnOw0KPiBk bGVkZm9yZEByZWRoYXQuY29tOyBsaW51eC1yZG1hQHZnZXIua2VybmVsLm9yZzsgSWRhbiBCdXJz dGVpbg0KPiA8aWRhbmJAbWVsbGFub3guY29tPg0KPiBTdWJqZWN0OiBSZTogW1BBVENIIHJkbWEt bmV4dCAwLzNdIFN1cHBvcnQgb3V0IG9mIG9yZGVyIGRhdGEgcGxhY2VtZW50DQo+IA0KPiBXZWxs LCBpZiB0aGUgYnJva2VuIGFwcGxpY2F0aW9ucyB3b24ndCB1c2UgdGhlIGV4dGVuc2lvbiwgYW5k IHRoZSBleGlzdGluZw0KPiBzdG9yYWdlIHByb3RvY29scyBhbmQgYXBwbGljYXRpb25zIHdpbGwg aGF2ZSB0byBjaGFuZ2UgYm90aCB0aGVpcg0KPiBpbXBsZW1lbnRhdGlvbiBhbmQgdGhlaXIgcHJv dG9jb2wgdG8gdXNlIGl0LCB3aG8gZG8geW91IGVudmlzaW9uIGFjdHVhbGx5IGRvaW5nDQo+IHNv Pw0KQ3VycmVudCBhcHBsaWNhdGlvbiBiaW5hcmllcyBhbmQgdGhlaXIgbWlkZGxld2FyZSBiaW5h cmllcyBjYW4gY29udGludWUgdG8gd29yayBhcyBpcy4NCk5ldyBhcHBsaWNhdGlvbnMgcXVlcnkg dGhpcyBmZWF0dXJlIGFuZCBlbmFibGUgdGhlbXNlbHZlcyB0byB0YWtlIGJlbmVmaXQgb2YgdGhp cyBvcHRpb25hbCBleHRlbnNpb24uDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The apps are not broken, it appears that this extension allows the receiver HCA to process inbound packets without seeing earlier packets.

This means it can really start doing things out of order, subject only to the ability of the transmitter to halt sending (eg fence) until acks are seen.

Since there is no way to predict what is in the data the HCA didn't see, this would seem to allow quite a lot of out of order execution inside the HCA.

Eg

RDMA-W VA=0 Data=1
RDMA-R VA=0
RDMA-W VA=0 Data=2

What does the read return? Spec says 1, but it sounds like this relaxed ordering could return 2.

Whta about

RDMA-W VA=0 Data=1
SEND WITH INVALIDATE VA=0
RDMA-W VA=0 Data=2

Spec says the second RDMA-W must fail, but it sounds like this relaxed ordering would allow it to happen.

So it cannot be turned on by default.. But a traditional well behaved storage ULP will only do single access to a single VA and should be able to safely turn something like this on.

All of these is independent to observability at the CPU..

All this means it needs to be negotiated via RDMA-CM, and making it a common verbs option would make sense - is there a plan to do that?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Correct.

Spec says Data=1 on RDMA-R only if Fence is set on read operation in Table-76. Otherwise duplicate read request after executing RDMA-W2 of Data=2 can return Data=2 on read request.

Right.

No. Table 76 is followed in this case.
1st operation Write.
2nd operation send.
There is implicit fence defined by '#' in Table, which is followed. So 2nd RDMA-W continue to fail.

I completely understand and agree that storage protocols who depend on RDMA-CM would like to have this. But again, unavailability of this bit in RDMA-CM is not a blocker for user land apps and let them start using it. Once RDMA-CM has it, storage kernel ULPs can also be enabled.

I am not the right person for commenting on RDMA-CM plan. Idan,
Do you know?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Erm, I should have written it like this

Initial Condition VA=0 Data = 0
RDMA-W VA=0 Data=1
RDMA-R VA=0

Spec says 1 must be returned, but sounds like this relaxed version could return 0. So RDMA Write -> RDMA Read degrades to a F

Similarly,

RDMA-W VA=0 Data=1
RDMA-W VA=0 Data=2
SEND

Sounds like with the relaxed version the app could see 1 at SEND CQ time.

So RDMA-W -> RDMA-W degrades to a F

So, I expect what is happening here is that the SEND RCQ is delayed until the sequence numbers catch up, eg guarenteeing that all packets prior to the SEND have been seen and committed to memory. Which is what table 76 is primarily talking about.

However, SEND WITH INVALIDATE is a special cases that impacts the processing of work itself, not just the CPU observation, which is a bit outside what table 76 is talking about.

I'd advocate for allowing this to be out of order (but documented as such), as impliclty fencing SEND WITH INVALIDATE is not acceptable for performance and most workloads using that feature do not care about this strict ordering.

The requirement is really that by the time the SEND RCQ is seen that the INVALIDATE has taken effect.

Atomic are basically similar, sounds like Atomic Op -> RDMA Read should degrade to a F as well. I'd say that is desired as well.

The point is we want a definition for this feature that is broad enough to allow future hardware optimization, and not just some some narrow defintion that follows what mlx5 happened to implement.

If there is no path to get this into the RDMA CM then it is just another vendor feature and it does not belong in the common API.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

No. Table 76 stays as is as described before.

No. Table-76 is based on how requester sees the execution. So it stays as '#'.

SEND, SEND WITH IMM, SEND WITH INVALIDATE falls in same category as send as first column in Table76.

It is as per first column of Table-76.

nvme fabrics do care.
nvme fabrics target does RDMA-W, RDMA_S_INV sequence on the same memory key that is being used in RDMA-W without waiting for RDMA-W completion for good reason. I recall SMB doing the same as well.
RDMA-S_INV after RDMA-W cannot break the order.

Current Table-76 requirement already relaxes for
RDMA-R-> RDMA_S_INV.
However most users won't do above sequence because users would not like to fail duplicate read requests. So let's continue with Table-76 for SEND as 2nd operation as defined today. (first column stays as is)

No. Table-76 stays as is.
Atomic->Atomic is already 'F'.
Atomic->RDMA_R is continues as '#'. (Similar to RDMA_W->RDMA_R).

RDMA CM is not the only connection manager is use today.
Once RDMA CM has it, it can be extended there as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

How is this possible?

How is this possible?

You've clearly stated this feature allows out of order execution across packet boundaries, there is no way to know at the responder what the missed packets where, so ineventiably, both of these cases must be possible. Or you are wrong about the statement on out of order.

Frankly, I still don't think you know what this feature actually does and consequently cannot document it properly, which is not acceptable for a common verbs feature.

Not really, I don't think you understand how this all fits together..

Dn't understand you remark, it is clearly ordered..

They don't care, because RDMA_WRITE, SEND_WITH_INVALIDATE on the same rkey does not try to write to the rkey memory twice, which is the only case where adding out of order execution really matters.

They only care that the invalidate guarentees no DMA is possible once it reaches the receiver's CQ.

As I said, table 76 does not really capture the full behavior of INVALIDATE.

The spec requires WRITE,INV,WRITE to fail, but it would be just a fine for storage protocols if WRITE,INV,WRITE could succeed, so long as delivering the INV to the CQ fences the DMA, which can be done in a high performance way. Fencing the WRITE,INV,WRITE can not be done with high performance.

Same argument as above, many apps will tolerate out of order for atomics, thee default for an out of order mode should be to allow it, and let apps request in order with fence.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I am not sure what more can I explain you Jason.
Requester side HCA follows HCA Table-76.
Incoming read responses are not processed until previous writes are ACKed implicitly (in read responses) or explicitly by ACK packets. Same as before described in spec. No extra description needed for this patchset.

Please don't mix requester side ordering with responder side execution. C9-28 on responder side is relaxed - as explained few times before.

Two examples already explained about out-of-order execution.
To me it appears that you are confused with requester vs responder side execution. What more can I explain other than repeating
(a) Table 76 on requester side stays as is and
(b) C9-28 is relaxed on responder side.

I was saying, that there is no change on send ordering at requester side.

It doesn't have to be an overlapping write to same rkey.
One block IO can translate to multiple RDMA-W from the target side, potentially to same rkey. One possibility is target code ran out of number of local sges or had fragmented memory. So
RDMA-W1 (key=A, VA=0x1000, len=16K with 4 SGEs)
RDMA-W2 (key=A, VA=0x5000, len=16K, with 4 SGEs)
Send(Invalidate_key=A)
You do not want SEND_INVALIDATE to stop DMA of RDMA-W1 at later point.
So Send CQE cannot reach before previous RDMA-w1 and w2 are completed.

It covers only requester side.
Send with invalidate execution on responder side is described in 9.4.1.1.1

You are proposing a different behavior and attribute which may be done for a HCA that support such thing. Please submit a different patch for it whenever its appropriate.
Current query HCA attribute is bit field for future relaxation. May be what you described can be done.

Following are already 'F'.
Atomic->Atomic
Atomic->Write
Read->Atomic

Other out of order atomics such as
Atomic->Read
Write->Atomic may be done in future under different attribute.

Jason,
Once single attribute is not solution to all out-of-order needs.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

But doing that pretty much destroys much of the entire point of having a relaxed ordering :P

No, I see what you are tring to say now. I disagree with this. Table-76 and C9-28 are describing the same thing, you cannot weaken C9-28 without also restating Table 76.

Table 76 is clearly talking about the entire system, including the execution and memory subsystem of the completer.

I suppose 9.4.1.1.1 point #1 already allows the out of order behavior.

Since you guys are hell bent on avoiding the IBTA for your new verbs features, you do have to define them in a sensible and usefully broad way using the community process.

I think that is a mistake, you should start with them being out of order and require the app to fence to bring order back, even if the current HCAs execute them in order anyhow.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Probably not. I can understand that having that would be possibly ultimate thing.
I like to add that whenever its available too.
Some of the applications are heavy write or read driven instead of mix operations - those benefit from this attribute.

Great.

The intent is to not extend the definition of fence bit beyond RDMA reads here.
What you are asking is when ooo attribute is set, and if user still wants to do in-order RDMA Writes for W1 and W2, fence bit should be extended for it. There can be very few use cases where certain operations needs to follow ordering and certain don't in a single QP.
User is rather better off not setting this attribute on a QP when it needs W1, W2 ordering.

It allows because to indicates below.
(a) Paragraph above #1. Snippet below.
"Since the invalidation operation is not executed
by the transport layer, the Invalidate operation may take place either before or after the transport-level acknowledge has been generated"

So it can still send out the ACK while invalidation in progress (not yet started). While that is in progress new operation can still target invalidated region.
Now depending on how slow invalidation is going, subsequent operation DMA also.
Most good adapters won't send out ACK before invalidating to my knowledge even though spec allows it. Because doing so is very hard to debug and keeps hole open for accidental memory corruption.

Also 9.4.1.1.1 is for subsequent operations where Send_Inv is second operation.

The example of
RDMA-W1, RDMA-W2, Send_Invalidate actually follows section 9.4.1.1.1 point #2.

Sure. That's why we are discussing here.

I would agree for Atomic->read case because it's very similar to READ->Read. Write->Atomic goes back to first point as atomic completions can trigger implicit Write completions. So Write->atomic I will keep out of this attribute.
Let me check with Idan for applying fence bit on Atomic->read, what he thinks about it. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

My point is, defining a 'relaxed ordering' feature such that as much as is sensible is allowed to reordered allows the hardware to catch up to prepared software.

The work to update the standards and CM is going to be large enough to not want to do it more than once.

So you are better to come up with something broad and well described that ycan be 'grown into'

I am not talk about fence, I am saying, when the OOO attribute is set some of the '#' must become 'F' because the entire end-to-end system no longer guarantees strict in order execution.

Yes, I am also suggesting this should be true. The fence bit would have to cause the sender to stop sending until acks are seen.

Depends on use case, an app might do well to track outstanding ops by address and only setting fence if there is a collision, for instance.

Not only an ack but it can begin processing another RDMA WRITE while invalidation is ongoing, which is compatible with the idea that RDMA WRITE can begin processing before seeing prior packets.

In esseence, there is another set of entries in table 76 that discusss the INVALIDATE behavior and they are all 'F' with today's standard.

Atomics do not cause completions at the responder, so I'm not sure what you are talking about here.

It is never, ever, OK to issue completions at the initiator out of order, that would totally break every ULP we have.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Can you guys explain why should a ULP even be exposed to this at all?

I can today issue RDMA writes to a single remote buffer in a pattern that are not "in-order". From what I understand, the OOO relates to mutliple packets of a _single_ RDMA transaction, which afair a granularity that the ULP isn't (and shouldn't be) exposed to.

I do not really understand what a ULP should do with this knob... It's really a topology variable, maybe it needs to sit in the CM or something?

Please correct me if I'm completely off...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Given the application follows the above semantics, why does it care if data is placed out of order? IE Why does this impact the application API at all?

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Steve,

If application wants to benefit from out-of-order placement, it should enable this flag. Following semantics is fine, but letting HCA enable something under the hood for all QPs, is not good. HCA or stack doesn't know which all applications are running, handling/following such semantics. So its best left to the end user applications to benefit or not from it.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I don't understand why the application has to care about this at all. If the HW wants to place things out of order. Ok go for it. Applications should be depending on the correct mechanism to know when it's OK to look at the data. Why do they care what order it arrived in?

The only reason to care is if the applications wants to do something that is not compliant and look at parts of the data early. Perhaps you can explain *how* an application may benefit from out-of-order placement and likewise, why it wouldn't want to benefit.

It seems to me, the real benefit is reducing the number of retransmissions which the application shouldn't care about.

-Denny

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

PiBJIGRvbid0IHVuZGVyc3RhbmQgd2h5IHRoZSBhcHBsaWNhdGlvbiBoYXMgdG8gY2FyZSBhYm91 dCB0aGlzIGF0IGFsbC4NCj4gSWYNCj4gdGhlIEhXIHdhbnRzIHRvIHBsYWNlIHRoaW5ncyBvdXQg b2Ygb3JkZXIuIE9rIGdvIGZvciBpdC4gQXBwbGljYXRpb25zDQo+IHNob3VsZCBiZSBkZXBlbmRp bmcgb24gdGhlIGNvcnJlY3QgbWVjaGFuaXNtIHRvIGtub3cgd2hlbiBpdCdzIE9LIHRvDQo+IGxv b2sgYXQgdGhlIGRhdGEuIFdoeSBkbyB0aGV5IGNhcmUgd2hhdCBvcmRlciBpdCBhcnJpdmVkIGlu Pw0KPiANCj4gVGhlIG9ubHkgcmVhc29uIHRvIGNhcmUgaXMgaWYgdGhlIGFwcGxpY2F0aW9ucyB3 YW50cyB0byBkbyBzb21ldGhpbmcNCj4gdGhhdCBpcyBub3QgY29tcGxpYW50IGFuZCBsb29rIGF0 IHBhcnRzIG9mIHRoZSBkYXRhIGVhcmx5LiBQZXJoYXBzIHlvdQ0KPiBjYW4gZXhwbGFpbiAqaG93 KiBhbiBhcHBsaWNhdGlvbiBtYXkgYmVuZWZpdCBmcm9tIG91dC1vZi1vcmRlcg0KPiBwbGFjZW1l bnQNCj4gYW5kIGxpa2V3aXNlLCB3aHkgaXQgd291bGRuJ3Qgd2FudCB0byBiZW5lZml0Lg0KDQpS ZWxheGVkIG9yZGVyaW5nIGNhbiBpbXByb3ZlIHBlcmZvcm1hbmNlIGJ5IGFsbG93aW5nIHBhY2tl dHMgdG8gdGFrZSBtdWx0aXBsZSBwYXRocyBmcm9tIHRoZSBzb3VyY2UgdG8gZGVzdGluYXRpb24u ICBGV0lXLCB0aGUgb2Zpd2cgYWN0dWFsbHkgZGlzY3Vzc2VkIG9yZGVyaW5nIGF0IGxlbmd0aCwg YW5kIGxpYmZhYnJpYyBleHBvc2VzIHNldmVyYWwgb3JkZXJpbmcgcmVsYXRlZCBhdHRyaWJ1dGVz Lg0KDQpUaGUgYXBwbGljYXRpb24gdXN1YWxseSBuZWVkcyB0byBrbm93IHdoYXQgc29ydCBvZiBv cmRlcmluZyB0aGUgbmV0d29yayBpcyBwcm92aWRpbmcgZm9yIGNvcnJlY3Qgb3BlcmF0aW9uLiAg Rm9yIGV4YW1wbGUsIGNhbiBzZW5kcyBiZSByZS1vcmRlcmVkIGJ5IHRoZSBuZXR3b3JrLCBzdWNo IHRoYXQgdGhleSBtYXkgYmUgcmVjZWl2ZWQgaW4gdGhlIG9wcG9zaXRlIG9yZGVyIHRoYXQgdGhl eSB3ZXJlIHNlbnQ/ICBDYW4gcmRtYSB3cml0ZXMgYmUgcHJvY2Vzc2VkIG91dCBvZiBvcmRlciwg c3VjaCB0aGF0IHRoZSB0YXJnZXQgZGF0YSBlbmRzIHVwIHdpdGggc29tZSBzb3J0IG9mIGludGVy bGVhdmluZyBvZiB0aGUgd3JpdHRlbiBkYXRhPyAgQXJlIHRoZXJlIHNpemUgbGltaXRzIHRvIG9y ZGVyaW5nIGNvbnN0cmFpbnRzPw0KDQpJdCdzIGhhcmQgdG8gY2FwdHVyZSB0aGlzIGxldmVsIG9m IGRldGFpbCB3aXRoIGEgc2luZ2xlIGZsYWcuICBUaGVyZSdzIG9yZGVyaW5nIG9mIGRhdGEgd2l0 aGluIGEgc2luZ2xlIG1lc3NhZ2UsIGFzIHdlbGwgYXMgb3JkZXJpbmcgb2YgZGF0YSBiZXR3ZWVu IG1lc3NhZ2VzLiAgSSBkb24ndCBrbm93IGhvdyB0aGlzIHByb3Bvc2FsIHdvcmtzIHdpdGggZXhw bGljaXQgcGF0aGluZyB0aGF0IElCIGRlZmluZXMsIG9yIHBhdGggZmFpbG92ZXIuDQoNCi0gU2Vh bg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgRGVubnksIFNlYW4sDQoNCkFzIFNlYW4gZXhwbGFpbmVkLCByZWxheGVkIG9yZGVyaW5nL291 dC1vZi1vcmRlciBwYWNrZXRzLCBhbGxvd3MgbWFraW5nIGVmZmljaWVudCB1c2Ugb2YgbmV0d29y ayByZXNvdXJjZXMuDQpXaGVuIHRyYW5zbWl0dGVyIGFuZCByZWNlaXZlciBpcyBlbmFibGVkIHRv IGRvIHNvLCBhcyBJIGRlc2NyaWJlZCBpbiBvdmVydmlldyBzZWN0aW9uIG9mIERvY3VtZW50YXRp b24sIGl0IGhlbHBzIA0KKGEpIHRvIGF2b2lkIHJldHJhbnNtaXNzaW9uIC0gaW1wcm92ZXMgbmV0 d29yayB1dGlsaXphdGlvbg0KKGIpIHJlZHVjZXMgbGF0ZW5jeSBkdWUgdG8gdGltZXJzIG5vdCBr aWNraW5nIGluLg0KDQpNb3JlIGNvbW1lbnRzIHRvIFNlYW4ncyBxdWVzdGlvbiBiZWxvdy4NCg0K PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBIZWZ0eSwgU2VhbiBbbWFpbHRv OnNlYW4uaGVmdHlAaW50ZWwuY29tXQ0KPiBTZW50OiBNb25kYXksIEp1bmUgMTIsIDIwMTcgMjox OSBQTQ0KPiBUbzogRGFsZXNzYW5kcm8sIERlbm5pcyA8ZGVubmlzLmRhbGVzc2FuZHJvQGludGVs LmNvbT47IFBhcmF2IFBhbmRpdA0KPiA8cGFyYXZAbWVsbGFub3guY29tPjsgU3RldmUgV2lzZSA8 c3dpc2VAb3BlbmdyaWRjb21wdXRpbmcuY29tPjsNCj4gJ0xlb24gUm9tYW5vdnNreScgPGxlb25A a2VybmVsLm9yZz47ICdEb3VnIExlZGZvcmQnDQo+IDxkbGVkZm9yZEByZWRoYXQuY29tPg0KPiBD YzogbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElkYW4gQnVyc3RlaW4gPGlkYW5iQG1lbGxh bm94LmNvbT4NCj4gU3ViamVjdDogUkU6IFtQQVRDSCByZG1hLW5leHQgMC8zXSBTdXBwb3J0IG91 dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiANCj4gPiBJIGRvbid0IHVuZGVyc3RhbmQgd2h5 IHRoZSBhcHBsaWNhdGlvbiBoYXMgdG8gY2FyZSBhYm91dCB0aGlzIGF0IGFsbC4NCj4gPiBJZg0K PiA+IHRoZSBIVyB3YW50cyB0byBwbGFjZSB0aGluZ3Mgb3V0IG9mIG9yZGVyLiBPayBnbyBmb3Ig aXQuIEFwcGxpY2F0aW9ucw0KPiA+IHNob3VsZCBiZSBkZXBlbmRpbmcgb24gdGhlIGNvcnJlY3Qg bWVjaGFuaXNtIHRvIGtub3cgd2hlbiBpdCdzIE9LIHRvDQo+ID4gbG9vayBhdCB0aGUgZGF0YS4g V2h5IGRvIHRoZXkgY2FyZSB3aGF0IG9yZGVyIGl0IGFycml2ZWQgaW4/DQo+ID4NCj4gPiBUaGUg b25seSByZWFzb24gdG8gY2FyZSBpcyBpZiB0aGUgYXBwbGljYXRpb25zIHdhbnRzIHRvIGRvIHNv bWV0aGluZw0KPiA+IHRoYXQgaXMgbm90IGNvbXBsaWFudCBhbmQgbG9vayBhdCBwYXJ0cyBvZiB0 aGUgZGF0YSBlYXJseS4gUGVyaGFwcyB5b3UNCj4gPiBjYW4gZXhwbGFpbiAqaG93KiBhbiBhcHBs aWNhdGlvbiBtYXkgYmVuZWZpdCBmcm9tIG91dC1vZi1vcmRlcg0KPiA+IHBsYWNlbWVudCBhbmQg bGlrZXdpc2UsIHdoeSBpdCB3b3VsZG4ndCB3YW50IHRvIGJlbmVmaXQuDQo+IA0KPiBSZWxheGVk IG9yZGVyaW5nIGNhbiBpbXByb3ZlIHBlcmZvcm1hbmNlIGJ5IGFsbG93aW5nIHBhY2tldHMgdG8g dGFrZQ0KPiBtdWx0aXBsZSBwYXRocyBmcm9tIHRoZSBzb3VyY2UgdG8gZGVzdGluYXRpb24uICBG V0lXLCB0aGUgb2Zpd2cgYWN0dWFsbHkNCj4gZGlzY3Vzc2VkIG9yZGVyaW5nIGF0IGxlbmd0aCwg YW5kIGxpYmZhYnJpYyBleHBvc2VzIHNldmVyYWwgb3JkZXJpbmcgcmVsYXRlZA0KPiBhdHRyaWJ1 dGVzLg0KPiANCj4gVGhlIGFwcGxpY2F0aW9uIHVzdWFsbHkgbmVlZHMgdG8ga25vdyB3aGF0IHNv cnQgb2Ygb3JkZXJpbmcgdGhlIG5ldHdvcmsgaXMNCj4gcHJvdmlkaW5nIGZvciBjb3JyZWN0IG9w ZXJhdGlvbi4NClllcy4gVGhpcyBmbGFnIGRlZmluZXMgaXQgaG93IFJXIGRhdGEgY2FuIGRvbmUg aW4gcmVsYXhlZCBvcmRlciBtYW5uZXIuDQoNCj4gRm9yIGV4YW1wbGUsIGNhbiBzZW5kcyBiZSBy ZS1vcmRlcmVkIGJ5DQo+IHRoZSBuZXR3b3JrLCBzdWNoIHRoYXQgdGhleSBtYXkgYmUgcmVjZWl2 ZWQgaW4gdGhlIG9wcG9zaXRlIG9yZGVyIHRoYXQgdGhleQ0KPiB3ZXJlIHNlbnQ/DQoNCk5vLiBP bmx5IHJkbWEgcmVhZHMgYW5kIHdyaXRlcy4gVGFibGUgNzkgb2YgSUIgc3BlYyBzdGlsbCBob2xk cyB0cnVlLg0KSW4gaW50ZXJuYWwgcGF0Y2ggcmV2aWV3IEkgaGFkIHRoZSB0YWJsZSwgYnV0IHNp bmNlIHRoZXJlIHdhcyBubyBjaGFuZ2UsIEkganVzdCByZW1vdmVkIGl0Lg0KSSB0aGluayBpdCdz IHdvcnRoIHRvIG1lbnRpb24gaW4gdGhlIG91dF9vZl9vcmRlci50eHQgdGhhdCBpdCBzdGlsbCBo b2xkcyB0cnVlLiBJIHdpbGwgYWRkIHRoaXMgbGluZS4NCg0KSSB3b3VsZCBhbHNvIGRvY3VtZW50 IHdoaWNoIGNvbXBsaWFudCBzdGF0ZW1lbnRzIGFyZSBub3QgYXBwbGljYWJsZSB3aGVuIHN1Y2gg ZmxhZyBpcyBzZXQuDQpIb3cgYWJvdXQgaGF2aW5nIHJlc3BvbmRlciBzaWRlIHRhYmxlLCBzaW1p bGFyIHRvIHRhYmxlIDc5IGluIERvY3VtZW50YXRpb24vb3V0X29mX29yZGVyLnR4dA0KVGhhdCBt YWtlcyBpdCBtb3JlIHJlYWRhYmxlIGZvciB1c2Vycy4NCg0KPiBDYW4gcmRtYSB3cml0ZXMgYmUg cHJvY2Vzc2VkIG91dCBvZiBvcmRlciwgc3VjaCB0aGF0IHRoZSB0YXJnZXQNCj4gZGF0YSBlbmRz IHVwIHdpdGggc29tZSBzb3J0IG9mIGludGVybGVhdmluZyBvZiB0aGUgd3JpdHRlbiBkYXRhPyAN Clllcy4NCg0KPiBBcmUgdGhlcmUgc2l6ZSBsaW1pdHMgdG8gb3JkZXJpbmcgY29uc3RyYWludHM/ DQpOby4NCg0KPiANCj4gSXQncyBoYXJkIHRvIGNhcHR1cmUgdGhpcyBsZXZlbCBvZiBkZXRhaWwg d2l0aCBhIHNpbmdsZSBmbGFnLiAgVGhlcmUncyBvcmRlcmluZyBvZg0KPiBkYXRhIHdpdGhpbiBh IHNpbmdsZSBtZXNzYWdlLCBhcyB3ZWxsIGFzIG9yZGVyaW5nIG9mIGRhdGEgYmV0d2VlbiBtZXNz YWdlcy4NCkdvb2QgZG9jdW1lbnRhdGlvbiBzaG91bGQgZGVzY3JpYmUgd2hhdCBhIGZsYWcgZG9l cy4gRmxhZyBuYW1lIGl0c2VsZiBoYXMgUlcgcHJlZml4IGluIGl0Lg0KQXJlIHlvdSBzdWdnZXN0 aW5nIHRoZXJlIHNob3VsZCBiZSB0d28gZmxhZ3MgLSBvbmUgZm9yIHdpdGhpbiBtZXNzYWdlLCBv bmUgZm9yIGFjcm9zcyBtZXNzYWdlPw0KSWYgc28sIHNob3VsZG4ndCB3ZSBhZGQgdGhvc2UgZmxh ZyB3aGVuIHN1Y2ggZGlmZmVyZW50IGltcGxlbWVudGF0aW9uIGFyaXNlLg0KSSBjYW5ub3QgZGVm aW5lIHNvbWV0aGluZyB0aGF0IGNhbm5vdCBiZSBpbXBsZW1lbnRlZCBvciB0ZXN0ZWQgYXQgdGhp cyBwb2ludC4gVGhpcyBpcyBhY3Jvc3MgdGhlIG1lc3NhZ2VzLg0KDQo+IEkgZG9uJ3Qga25vdyBo b3cgdGhpcyBwcm9wb3NhbCB3b3JrcyB3aXRoIGV4cGxpY2l0IHBhdGhpbmcgdGhhdCBJQiBkZWZp bmVzLA0KPiBvciBwYXRoIGZhaWxvdmVyLg0KDQpJdCBkb2Vzbid0IGNoYW5nZSB3aXRoIHBhdGgg ZmFpbG92ZXIgb3IgZXhwbGljaXQgcGF0aCB0byBteSBrbm93bGVkZ2UsIGJ1dCBJIHdpbGwgY29u ZmlybSBpZiB0aGVyZSBpcyBhbnkgY2hhbmdlIGluIHYxLg0KDQo+IA0KPiAtIFNlYW4NCg== --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes those benefits are clear. I see no reason why it shouldn't always be done is my point. Application shouldn't have to care and there is no need to make this an additional flag.

-Denny

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

PiA+IFdoZW4gdHJhbnNtaXR0ZXIgYW5kIHJlY2VpdmVyIGlzIGVuYWJsZWQgdG8gZG8gc28sIGFz IEkgZGVzY3JpYmVkIGluDQo+IG92ZXJ2aWV3IHNlY3Rpb24gb2YgRG9jdW1lbnRhdGlvbiwgaXQg aGVscHMNCj4gPiAoYSkgdG8gYXZvaWQgcmV0cmFuc21pc3Npb24gLSBpbXByb3ZlcyBuZXR3b3Jr IHV0aWxpemF0aW9uDQo+ID4gKGIpIHJlZHVjZXMgbGF0ZW5jeSBkdWUgdG8gdGltZXJzIG5vdCBr aWNraW5nIGluLg0KPiANCj4gWWVzIHRob3NlIGJlbmVmaXRzIGFyZSBjbGVhci4gSSBzZWUgbm8g cmVhc29uIHdoeSBpdCBzaG91bGRuJ3QgYWx3YXlzDQo+IGJlDQo+IGRvbmUgaXMgbXkgcG9pbnQu IEFwcGxpY2F0aW9uIHNob3VsZG4ndCBoYXZlIHRvIGNhcmUgYW5kIHRoZXJlIGlzIG5vDQo+IG5l ZWQgdG8gbWFrZSB0aGlzIGFuIGFkZGl0aW9uYWwgZmxhZy4NCg0KVGhlIGFwcCBjYXJlcyB3aGVu IGRhdGEgZnJvbSB3cml0ZSAyIGNhbiBiZSB3cml0dGVuIGF0IHRoZSB0YXJnZXQgYmVmb3JlIGRh dGEgZnJvbSB3cml0ZSAxLCBlc3BlY2lhbGx5IGlmIHRoZSB3cml0ZXMgdGFyZ2V0IHRoZSBzYW1l IG1lbW9yeSBidWZmZXJzLiAgKEF0IGxlYXN0IEkgdGhpbmsgdGhpcyBpcyB0aGUgaW50ZW50IG9m IGV4cG9zaW5nIHRoaXMgdG8gdGhlIGFwcC4pDQoNCk5vdGUgdGhhdCB0aGUgcHJvdmlkZXIgY2Fu IGFsd2F5cyBwcm92aWRlIHN0cm9uZ2VyIG9yZGVyaW5nIHRoYW4gd2hhdCB0aGUgYXBwIG5lZWRz Lg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

My understanding is that IB or IW apps should never assume ingress write or read response data is _placed_ into local memory in the order it was transmitted from the peer. The only guarantee is that the _indication_ of the arrived data preserve the sender's ordering. However, I'm thinking that there are applications out there that spin polling local memory that is the target of a write or read response and assume the last bit of that memory will get written last...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

That is with respect to the CPU, but IB requires strong ordering between messages within the same QP, eg if I do

RDMA WRITE addr=0 data=1
RDMA WRITE addr=0 data=2
RDMA WRITE addr=0 data=3
RDMA READ addr=0

I must always get 3, not something else.

It would be notable if this 'out of order' feature violated that invariant, but many ULPs would probably still be OK.

Frankly, Parav's original message doesn't seem to describe at all what this is about, so maybe we should all wait until v2, and maybe more people from Mellanox could contribute to sensibly describing it if they want it in ibverbs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

data

(At least

app

Correct, but the peer, ie the remote end that is the target of those writes, can spin looking at local address 0 and it might see 1, 2, or 3. Eventually it will see 3. But there is no guarantee that it will see 1 before 2 or even see 1 or 2 at all depending on timing.

But what I was getting at is this: Say you tell your peer to RDMA WRITE 16KB into your local buffer. And let us say the last bit of that 16KB data will be a 1, and that the current value of that bit location in the local buffer is 0. It is incorrect for the app to spin reading that bit until reads 1, and assume the data prior to that bit has been placed at that point. At least with the iWARP spec, out of order placement is allowed. So if the 16KB was broken into X iWARP DDP segments, the last segment could have been placed before the other segments. A correct application will require the peer to post a SEND after the WRITE or WRITE_WITH_IMMEDIATE, and only know the data has been placed into the local buffer when it polls the recv completion for the SEND or WRITE_W_IMMD. An iWARP RNIC _must_ guarantee in-order delivery of data (but not actual placement). Am I making sense?

I'm guessing no HCAs nor RNICs actually place data out of order. cxgb* does not. So applications _might_ be doing the spin technique I described. I recall a long time ago that MVAPICH2 did this. Not sure if it still does.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes, and IB is technically the same.

As Tom pointed out, it is a quirk of the x86 & mellanox combination that apps can reliably 'spin on the last byte', it is certainly not guaranteed by any spec.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I certainly don't see a point in breaking the users who are polling on data, even though they should have followed optional requirement o9-20. Also read responses can come out of orders, if such messages are polled either, it would also break, not just writes. Refer table 79 for two read operations.

I will add following more details to Documentation.
1. Mention about pcie relax ordering - Barts point
2. Include responder side table like Table 79 to crisply describe all cases and ordering with respect to send and other messages
3. Also indicate that C9-28 is relaxed when ooo is enabled on a QP as a description to new responder side table. This was offline comment that I received.
4. Provide examples that Steve and Jason highlighted, with multiple writes to same memory location.
5. Reiterate table 79, to make it clear that what doesn't change, or changes.
Let me know if you want to see any more details.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Explain why this needs to be negotiated and what it does to the transmit side of a QP.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Sure. I will add this. This is important point.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I sent this patch series in US timezone to allow Parav to answer on the questions if they arise, but for other people from Mellanox it was night and it limited number of participants.

Thanks

PiA+IEZvciBleGFtcGxlLCBjYW4gc2VuZHMgYmUgcmUtb3JkZXJlZCBieQ0KPiA+IHRoZSBuZXR3 b3JrLCBzdWNoIHRoYXQgdGhleSBtYXkgYmUgcmVjZWl2ZWQgaW4gdGhlIG9wcG9zaXRlIG9yZGVy DQo+IHRoYXQgdGhleQ0KPiA+IHdlcmUgc2VudD8NCj4gDQo+IE5vLiBPbmx5IHJkbWEgcmVhZHMg YW5kIHdyaXRlcy4gVGFibGUgNzkgb2YgSUIgc3BlYyBzdGlsbCBob2xkcyB0cnVlLg0KPiBJbiBp bnRlcm5hbCBwYXRjaCByZXZpZXcgSSBoYWQgdGhlIHRhYmxlLCBidXQgc2luY2UgdGhlcmUgd2Fz IG5vDQo+IGNoYW5nZSwgSSBqdXN0IHJlbW92ZWQgaXQuDQo+IEkgdGhpbmsgaXQncyB3b3J0aCB0 byBtZW50aW9uIGluIHRoZSBvdXRfb2Zfb3JkZXIudHh0IHRoYXQgaXQgc3RpbGwNCj4gaG9sZHMg dHJ1ZS4gSSB3aWxsIGFkZCB0aGlzIGxpbmUuDQo+IA0KPiBJIHdvdWxkIGFsc28gZG9jdW1lbnQg d2hpY2ggY29tcGxpYW50IHN0YXRlbWVudHMgYXJlIG5vdCBhcHBsaWNhYmxlDQo+IHdoZW4gc3Vj aCBmbGFnIGlzIHNldC4NCj4gSG93IGFib3V0IGhhdmluZyByZXNwb25kZXIgc2lkZSB0YWJsZSwg c2ltaWxhciB0byB0YWJsZSA3OSBpbg0KPiBEb2N1bWVudGF0aW9uL291dF9vZl9vcmRlci50eHQN Cj4gVGhhdCBtYWtlcyBpdCBtb3JlIHJlYWRhYmxlIGZvciB1c2Vycy4NCj4gDQo+ID4gQ2FuIHJk bWEgd3JpdGVzIGJlIHByb2Nlc3NlZCBvdXQgb2Ygb3JkZXIsIHN1Y2ggdGhhdCB0aGUgdGFyZ2V0 DQo+ID4gZGF0YSBlbmRzIHVwIHdpdGggc29tZSBzb3J0IG9mIGludGVybGVhdmluZyBvZiB0aGUg d3JpdHRlbiBkYXRhPw0KPiBZZXMuDQo+IA0KPiA+IEFyZSB0aGVyZSBzaXplIGxpbWl0cyB0byBv cmRlcmluZyBjb25zdHJhaW50cz8NCj4gTm8uDQoNCk15IHF1ZXN0aW9ucyB3ZXJlIG1vcmUgZXhh bXBsZXMgb2YgdGhlIGxldmVsIG9mIGRldGFpbCB0aGF0IEkgdGhpbmsgYW55IEFQSSBjaGFuZ2Ug bmVlZHMgdG8gKGV2ZW50dWFsbHkpIGJlIGFibGUgdG8gYWRkcmVzcy4gIExpYmlidmVyYnMgaXMg c3VwcG9zZWQgdG8gd29yayBvdmVyIGl3YXJwIGFzIHdlbGwsIHdoaWNoIGhhcyBkaWZmZXJlbnQg b3JkZXJpbmcgc2VtYW50aWNzIGFscmVhZHkuICBJIHRoaW5rIHRoYXQgd2FzIHBhcnQgb2YgU3Rl dmUncyBwb2ludCBhcyB0byB3aHkgYW55IGNoYW5nZSBpcyBuZWVkZWQgdG8gdGhlIGFwaS4NCg0K SU1PLCByZWFkIGFuZCB3cml0ZSBvcmRlcmluZyBzaG91bGQgYmUgZGV0ZXJtaW5lZCBzZXBhcmF0 ZWx5IGZyb20gdGhlIHBlcnNwZWN0aXZlIG9mIHRoZSBhcGkuICBUaGlzIGlzIGV4cG9zaW5nIGEg aHcgc3BlY2lmaWMgbGltaXRhdGlvbiB3aGVyZSB0aGVzZSBzZW1hbnRpY3MgYXJlIGpvaW5lZCwg YW5kIHNlbmQvcmVjZWl2ZSBpcyBub3Qgc3VwcG9ydGVkLg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

SGkgU2VhbiwNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBIZWZ0eSwg U2VhbiBbbWFpbHRvOnNlYW4uaGVmdHlAaW50ZWwuY29tXQ0KPiBTZW50OiBNb25kYXksIEp1bmUg MTIsIDIwMTcgMzo0MSBQTQ0KPiBUbzogUGFyYXYgUGFuZGl0IDxwYXJhdkBtZWxsYW5veC5jb20+ OyBEYWxlc3NhbmRybywgRGVubmlzDQo+IDxkZW5uaXMuZGFsZXNzYW5kcm9AaW50ZWwuY29tPjsg U3RldmUgV2lzZQ0KPiA8c3dpc2VAb3BlbmdyaWRjb21wdXRpbmcuY29tPjsgJ0xlb24gUm9tYW5v dnNreScgPGxlb25Aa2VybmVsLm9yZz47DQo+ICdEb3VnIExlZGZvcmQnIDxkbGVkZm9yZEByZWRo YXQuY29tPg0KPiBDYzogbGludXgtcmRtYUB2Z2VyLmtlcm5lbC5vcmc7IElkYW4gQnVyc3RlaW4g PGlkYW5iQG1lbGxhbm94LmNvbT4NCj4gU3ViamVjdDogUkU6IFtQQVRDSCByZG1hLW5leHQgMC8z XSBTdXBwb3J0IG91dCBvZiBvcmRlciBkYXRhIHBsYWNlbWVudA0KPiANCj4gPiA+IEZvciBleGFt cGxlLCBjYW4gc2VuZHMgYmUgcmUtb3JkZXJlZCBieSB0aGUgbmV0d29yaywgc3VjaCB0aGF0IHRo ZXkNCj4gPiA+IG1heSBiZSByZWNlaXZlZCBpbiB0aGUgb3Bwb3NpdGUgb3JkZXINCj4gPiB0aGF0 IHRoZXkNCj4gPiA+IHdlcmUgc2VudD8NCj4gPg0KPiA+IE5vLiBPbmx5IHJkbWEgcmVhZHMgYW5k IHdyaXRlcy4gVGFibGUgNzkgb2YgSUIgc3BlYyBzdGlsbCBob2xkcyB0cnVlLg0KPiA+IEluIGlu dGVybmFsIHBhdGNoIHJldmlldyBJIGhhZCB0aGUgdGFibGUsIGJ1dCBzaW5jZSB0aGVyZSB3YXMg bm8NCj4gPiBjaGFuZ2UsIEkganVzdCByZW1vdmVkIGl0Lg0KPiA+IEkgdGhpbmsgaXQncyB3b3J0 aCB0byBtZW50aW9uIGluIHRoZSBvdXRfb2Zfb3JkZXIudHh0IHRoYXQgaXQgc3RpbGwNCj4gPiBo b2xkcyB0cnVlLiBJIHdpbGwgYWRkIHRoaXMgbGluZS4NCj4gPg0KPiA+IEkgd291bGQgYWxzbyBk b2N1bWVudCB3aGljaCBjb21wbGlhbnQgc3RhdGVtZW50cyBhcmUgbm90IGFwcGxpY2FibGUNCj4g PiB3aGVuIHN1Y2ggZmxhZyBpcyBzZXQuDQo+ID4gSG93IGFib3V0IGhhdmluZyByZXNwb25kZXIg c2lkZSB0YWJsZSwgc2ltaWxhciB0byB0YWJsZSA3OSBpbg0KPiA+IERvY3VtZW50YXRpb24vb3V0 X29mX29yZGVyLnR4dCBUaGF0IG1ha2VzIGl0IG1vcmUgcmVhZGFibGUgZm9yIHVzZXJzLg0KPiA+ DQo+ID4gPiBDYW4gcmRtYSB3cml0ZXMgYmUgcHJvY2Vzc2VkIG91dCBvZiBvcmRlciwgc3VjaCB0 aGF0IHRoZSB0YXJnZXQgZGF0YQ0KPiA+ID4gZW5kcyB1cCB3aXRoIHNvbWUgc29ydCBvZiBpbnRl cmxlYXZpbmcgb2YgdGhlIHdyaXR0ZW4gZGF0YT8NCj4gPiBZZXMuDQo+ID4NCj4gPiA+IEFyZSB0 aGVyZSBzaXplIGxpbWl0cyB0byBvcmRlcmluZyBjb25zdHJhaW50cz8NCj4gPiBOby4NCj4gDQo+ IE15IHF1ZXN0aW9ucyB3ZXJlIG1vcmUgZXhhbXBsZXMgb2YgdGhlIGxldmVsIG9mIGRldGFpbCB0 aGF0IEkgdGhpbmsgYW55IEFQSQ0KPiBjaGFuZ2UgbmVlZHMgdG8gKGV2ZW50dWFsbHkpIGJlIGFi bGUgdG8gYWRkcmVzcy4gIExpYmlidmVyYnMgaXMgc3VwcG9zZWQgdG8NCj4gd29yayBvdmVyIGl3 YXJwIGFzIHdlbGwsIHdoaWNoIGhhcyBkaWZmZXJlbnQgb3JkZXJpbmcgc2VtYW50aWNzIGFscmVh ZHkuICBJDQo+IHRoaW5rIHRoYXQgd2FzIHBhcnQgb2YgU3RldmUncyBwb2ludCBhcyB0byB3aHkg YW55IGNoYW5nZSBpcyBuZWVkZWQgdG8gdGhlDQo+IGFwaS4NCj4gDQo+IElNTywgcmVhZCBhbmQg d3JpdGUgb3JkZXJpbmcgc2hvdWxkIGJlIGRldGVybWluZWQgc2VwYXJhdGVseSBmcm9tIHRoZQ0K PiBwZXJzcGVjdGl2ZSBvZiB0aGUgYXBpLiAgVGhpcyBpcyBleHBvc2luZyBhIGh3IHNwZWNpZmlj IGxpbWl0YXRpb24gd2hlcmUgdGhlc2UNCj4gc2VtYW50aWNzIGFyZSBqb2luZWQsIGFuZCBzZW5k L3JlY2VpdmUgaXMgbm90IHN1cHBvcnRlZC4NCg0KVGhleSBhcmUgam9pbmVkIGZvciBzaW1wbGlj aXR5LiBJZiB0aGVyZSBpcyBlc3RhYmxpc2hlZCB1c2UgY2FzZSB0aGF0IEkgbGlrZSB0byBoZWFy IGZyb20gdXNlcnMgDQphbmQgaWYgaGFyZHdhcmUgc3VwcG9ydHMgaXQsIG1vcmUgY2FwcyBjYW4g YmUgYWRkZWQuDQpUaGF04oCZcyB3aHkgY2FwcyBpcyAzMi1iaXQgbnVtYmVyIGZvciBmdXR1cmUg YWRkaXRpb24gb2YgUkRfT05MWSwgV1JfT05MWSBtb2Rlcy4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html