Skip to contentSkip to conversation

RDMA/rxe: Add atomic write operation

PATCH Superseded [email protected] 2 participants 17 messages Oct 15, 2022 – Dec 1, 2022

Subsystems (2)

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover RDMA/rxe: Add atomic write operation

The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset.

The steps to run the rdma_atomic_write example:
server:

$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

v4->v5:
1) Rebase on current wip/jgg-for-next
2) Rewrite the implementation on responder

v3->v4:
1) Rebase on current wip/jgg-for-next
2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
3) Replace 64-bit value with 8-byte array for atomic write

V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device

V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed

Xiao Yang (8):
RDMA: Extend RDMA user ABI to support atomic write
RDMA: Extend RDMA kernel ABI to support atomic write RDMA/rxe: Extend rxe user ABI to support atomic write RDMA/rxe: Extend rxe packet format to support atomic write RDMA/rxe: Make requester support atomic write on RC service RDMA/rxe: Make responder support atomic write on RC service RDMA/rxe: Implement atomic write completion RDMA/rxe: Enable atomic write capability for rxe device

 drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
 drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
 drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
 drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
 include/rdma/ib_pack.h                 |  2 +
 include/rdma/ib_verbs.h                |  3 +
 include/uapi/rdma/ib_user_verbs.h      |  4 ++
 include/uapi/rdma/rdma_user_rxe.h      |  1 +
 10 files changed, 132 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v6 1/8 RDMA: Extend RDMA user ABI to support atomic write

1) Define new atomic write request/completion in userspace.
2) Define new atomic write capability in userspace.

---
 include/uapi/rdma/ib_user_verbs.h | 4 ++++
 1 file changed, 4 insertions(+)
include/uapi/rdma/ib_user_verbs.h +4 -0
index 43672cb1fd57..237814815544 100644
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
 	IB_UVERBS_WC_BIND_MW = 5,
 	IB_UVERBS_WC_LOCAL_INV = 6,
 	IB_UVERBS_WC_TSO = 7,
+	IB_UVERBS_WC_ATOMIC_WRITE = 9,
 };
 
 struct ib_uverbs_wc {
@@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode {
 	IB_UVERBS_WR_RDMA_READ_WITH_INV = 11,
 	IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12,
 	IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13,
+	IB_UVERBS_WR_ATOMIC_WRITE = 15,
 	/* Review enum ib_wr_opcode before modifying this */
 };
 
@@ -1331,6 +1333,8 @@ enum ib_uverbs_device_cap_flags {
 	/* Deprecated. Please use IB_UVERBS_RAW_PACKET_CAP_SCATTER_FCS. */
 	IB_UVERBS_DEVICE_RAW_SCATTER_FCS = 1ULL << 34,
 	IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING = 1ULL << 36,
+	/* Atomic write attributes */
+	IB_UVERBS_DEVICE_ATOMIC_WRITE = 1ULL << 40,
 };
 
 enum ib_uverbs_raw_packet_caps {
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH v6 1/8] RDMA: Extend RDMA user ABI to support atomic write

Quoted 13 lines · On Sat, Oct 15, 2022 at 06:37:04AM +0000, [email protected] wrote:

1) Define new atomic write request/completion in userspace.
2) Define new atomic write capability in userspace.

---
 include/uapi/rdma/ib_user_verbs.h | 4 ++++
 1 file changed, 4 insertions(+)
include/uapi/rdma/ib_user_verbs.h +0 -0
index 43672cb1fd57..237814815544 100644

include/uapi/rdma/ib_user_verbs.h enum ib_uverbs_wc_opcode { · line 470

Quoted 5 lines
include/uapi/rdma/ib_user_verbs.h +1 -0
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
 	IB_UVERBS_WC_BIND_MW = 5,
 	IB_UVERBS_WC_LOCAL_INV = 6,
 	IB_UVERBS_WC_TSO = 7,
+	IB_UVERBS_WC_ATOMIC_WRITE = 9,
 };

Why is this 9? The following patch does

@@ -985,6 +986,7 @@ enum ib_wc_opcode {
        IB_WC_REG_MR,
        IB_WC_MASKED_COMP_SWAP,
        IB_WC_MASKED_FETCH_ADD,

+ IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,

Which corrupts the enum.

It should be like this:

+++ b/include/rdma/ib_verbs.h
@@ -983,10 +983,10 @@ enum ib_wc_opcode {
IB_WC_BIND_MW = IB_UVERBS_WC_BIND_MW,
IB_WC_LOCAL_INV = IB_UVERBS_WC_LOCAL_INV,
IB_WC_LSO = IB_UVERBS_WC_TSO,
+ IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,

        IB_WC_REG_MR,
        IB_WC_MASKED_COMP_SWAP,
        IB_WC_MASKED_FETCH_ADD,

- IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
/*
* Set value of IB_WC_RECV so consumers can test if a completion is a
* receive by testing (opcode & IB_WC_RECV).
+++ b/include/uapi/rdma/ib_user_verbs.h

@@ -466,7 +466,7 @@ enum ib_uverbs_wc_opcode {
        IB_UVERBS_WC_BIND_MW = 5,
        IB_UVERBS_WC_LOCAL_INV = 6,
        IB_UVERBS_WC_TSO = 7,

- IB_UVERBS_WC_ATOMIC_WRITE = 9,
+ IB_UVERBS_WC_ATOMIC_WRITE = 8,
};

struct ib_uverbs_wc {

Jason

author
#3 lore raw

Re: [PATCH v6 1/8] RDMA: Extend RDMA user ABI to support atomic write

Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · On Sat, Oct 15, 2022 at 06:37:04AM +0000, [email protected] wrote:

1) Define new atomic write request/completion in userspace.
2) Define new atomic write capability in userspace.

---
  include/uapi/rdma/ib_user_verbs.h | 4 ++++
  1 file changed, 4 insertions(+)
Quoted 12 lines

include/uapi/rdma/ib_user_verbs.h enum ib_uverbs_wc_opcode { · line 470

Quoted 10 lines
include/uapi/rdma/ib_user_verbs.h +1 -0
index 43672cb1fd57..237814815544 100644
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
  	IB_UVERBS_WC_BIND_MW = 5,
  	IB_UVERBS_WC_LOCAL_INV = 6,
  	IB_UVERBS_WC_TSO = 7,
+	IB_UVERBS_WC_ATOMIC_WRITE = 9,
  };

Why is this 9? The following patch does

Hi Jason,

I reserve 8 for IB_UVERBS_WC_FLUSH and 14 for IB_UVERBS_WR_FLUSH.

Quoted 8 lines
(quoted diff) +1 -0
@@ -985,6 +986,7 @@ enum ib_wc_opcode {
         IB_WC_REG_MR,
         IB_WC_MASKED_COMP_SWAP,
         IB_WC_MASKED_FETCH_ADD,
+       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
 

Which corrupts the enum.

Good catch. I will correct it now.

Best Regards,
Xiao Yang

Quoted 28 lines

It should be like this:

+++ b/include/rdma/ib_verbs.h

include/uapi/rdma/ib_user_verbs.h +3 -2
@@ -983,10 +983,10 @@ enum ib_wc_opcode {
         IB_WC_BIND_MW = IB_UVERBS_WC_BIND_MW,
         IB_WC_LOCAL_INV = IB_UVERBS_WC_LOCAL_INV,
         IB_WC_LSO = IB_UVERBS_WC_TSO,
+       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
         IB_WC_REG_MR,
         IB_WC_MASKED_COMP_SWAP,
         IB_WC_MASKED_FETCH_ADD,
-       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
  /*
   * Set value of IB_WC_RECV so consumers can test if a completion is a
   * receive by testing (opcode & IB_WC_RECV).
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -466,7 +466,7 @@ enum ib_uverbs_wc_opcode {
         IB_UVERBS_WC_BIND_MW = 5,
         IB_UVERBS_WC_LOCAL_INV = 6,
         IB_UVERBS_WC_TSO = 7,
-       IB_UVERBS_WC_ATOMIC_WRITE = 9,
+       IB_UVERBS_WC_ATOMIC_WRITE = 8,
  };
 
  struct ib_uverbs_wc {

Jason

#4 lore raw

PATCH v6 2/8 RDMA: Extend RDMA kernel ABI to support atomic write

In reply to [email protected]

1) Define new atomic write request/completion in kernel.
2) Define new atomic write capability in kernel.
3) Define new atomic write opcode for RC service in packet.

---
 include/rdma/ib_pack.h  | 2 ++
 include/rdma/ib_verbs.h | 3 +++
 2 files changed, 5 insertions(+)
include/rdma/ib_pack.h +2 -0
index a9162f25beaf..f932d164af63 100644
@@ -84,6 +84,7 @@ enum {
 	/* opcode 0x15 is reserved */
 	IB_OPCODE_SEND_LAST_WITH_INVALIDATE         = 0x16,
 	IB_OPCODE_SEND_ONLY_WITH_INVALIDATE         = 0x17,
+	IB_OPCODE_ATOMIC_WRITE                      = 0x1D,
 
 	/* real constants follow -- see comment about above IB_OPCODE()
 	   macro for more details */
@@ -112,6 +113,7 @@ enum {
 	IB_OPCODE(RC, FETCH_ADD),
 	IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE),
 	IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE),
+	IB_OPCODE(RC, ATOMIC_WRITE),
 
 	/* UC */
 	IB_OPCODE(UC, SEND_FIRST),
include/rdma/ib_verbs.h +3 -0
index 975d6e9efbcb..19bed1fb8e55 100644
@@ -270,6 +270,7 @@ enum ib_device_cap_flags {
 	/* The device supports padding incoming writes to cacheline. */
 	IB_DEVICE_PCI_WRITE_END_PADDING =
 		IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING,
+	IB_DEVICE_ATOMIC_WRITE = IB_UVERBS_DEVICE_ATOMIC_WRITE,
 };
 
 enum ib_kernel_cap_flags {
@@ -985,6 +986,7 @@ enum ib_wc_opcode {
 	IB_WC_REG_MR,
 	IB_WC_MASKED_COMP_SWAP,
 	IB_WC_MASKED_FETCH_ADD,
+	IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
 /*
  * Set value of IB_WC_RECV so consumers can test if a completion is a
  * receive by testing (opcode & IB_WC_RECV).
@@ -1325,6 +1327,7 @@ enum ib_wr_opcode {
 		IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP,
 	IB_WR_MASKED_ATOMIC_FETCH_AND_ADD =
 		IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD,
+	IB_WR_ATOMIC_WRITE = IB_UVERBS_WR_ATOMIC_WRITE,
 
 	/* These are kernel only and can not be issued by userspace */
 	IB_WR_REG_MR = 0x20,
Signature · 1 line
2.34.1
#5 lore raw

PATCH v6 3/8 RDMA/rxe: Extend rxe user ABI to support atomic write

In reply to [email protected]

Define an atomic_wr array to store 8-byte value.

---
 include/uapi/rdma/rdma_user_rxe.h | 1 +
 1 file changed, 1 insertion(+)
include/uapi/rdma/rdma_user_rxe.h +1 -0
index 73f679dfd2df..d20d1ecf046f 100644
@@ -146,6 +146,7 @@ struct rxe_dma_info {
 	__u32			reserved;
 	union {
 		__DECLARE_FLEX_ARRAY(__u8, inline_data);
+		__DECLARE_FLEX_ARRAY(__u8, atomic_wr);
 		__DECLARE_FLEX_ARRAY(struct rxe_sge, sge);
 	};
 };
Signature · 1 line
2.34.1
#6 lore raw

PATCH v6 4/8 RDMA/rxe: Extend rxe packet format to support atomic write

In reply to [email protected]

Extend rxe_wr_opcode_info[] and rxe_opcode[] for new atomic write opcode.

---
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +++
 2 files changed, 21 insertions(+)
drivers/infiniband/sw/rxe/rxe_opcode.c +18 -0
index d4ba4d506f17..fb196029048e 100644
@@ -101,6 +101,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
 			[IB_QPT_UC]	= WR_LOCAL_OP_MASK,
 		},
 	},
+	[IB_WR_ATOMIC_WRITE]                       = {
+		.name   = "IB_WR_ATOMIC_WRITE",
+		.mask   = {
+			[IB_QPT_RC]     = WR_ATOMIC_WRITE_MASK,
+		},
+	},
 };
 
 struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
@@ -378,6 +384,18 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
 					  RXE_IETH_BYTES,
 		}
 	},
+	[IB_OPCODE_RC_ATOMIC_WRITE]                        = {
+		.name   = "IB_OPCODE_RC_ATOMIC_WRITE",
+		.mask   = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK |
+			  RXE_ATOMIC_WRITE_MASK | RXE_START_MASK |
+			  RXE_END_MASK,
+		.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		.offset = {
+			[RXE_BTH]       = 0,
+			[RXE_RETH]      = RXE_BTH_BYTES,
+			[RXE_PAYLOAD]   = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		}
+	},
 
 	/* UC */
 	[IB_OPCODE_UC_SEND_FIRST]			= {
drivers/infiniband/sw/rxe/rxe_opcode.h +3 -0
index 8f9aaaf260f2..a470e9b0b884 100644
@@ -20,6 +20,7 @@ enum rxe_wr_mask {
 	WR_READ_MASK			= BIT(3),
 	WR_WRITE_MASK			= BIT(4),
 	WR_LOCAL_OP_MASK		= BIT(5),
+	WR_ATOMIC_WRITE_MASK		= BIT(7),
 
 	WR_READ_OR_WRITE_MASK		= WR_READ_MASK | WR_WRITE_MASK,
 	WR_WRITE_OR_SEND_MASK		= WR_WRITE_MASK | WR_SEND_MASK,
@@ -81,6 +82,8 @@ enum rxe_hdr_mask {
 
 	RXE_LOOPBACK_MASK	= BIT(NUM_HDR_TYPES + 12),
 
+	RXE_ATOMIC_WRITE_MASK   = BIT(NUM_HDR_TYPES + 14),
+
 	RXE_READ_OR_ATOMIC_MASK	= (RXE_READ_MASK | RXE_ATOMIC_MASK),
 	RXE_WRITE_OR_SEND_MASK	= (RXE_WRITE_MASK | RXE_SEND_MASK),
 	RXE_READ_OR_WRITE_MASK	= (RXE_READ_MASK | RXE_WRITE_MASK),
Signature · 1 line
2.34.1
#7 lore raw

PATCH v6 5/8 RDMA/rxe: Make requester support atomic write on RC service

In reply to [email protected]

Make requester process and send an atomic write request on RC service.

---
 drivers/infiniband/sw/rxe/rxe_req.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_req.c +13 -2
index f63771207970..50d794910a3f 100644
@@ -258,6 +258,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits)
 		else
 			return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE :
 				IB_OPCODE_RC_SEND_FIRST;
+
+	case IB_WR_ATOMIC_WRITE:
+		return IB_OPCODE_RC_ATOMIC_WRITE;
+
 	case IB_WR_REG_MR:
 	case IB_WR_LOCAL_INV:
 		return opcode;
@@ -486,6 +490,11 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_av *av,
 		}
 	}
 
+	if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		memcpy(payload_addr(pkt), wqe->dma.atomic_wr, payload);
+		wqe->dma.resid -= payload;
+	}
+
 	return 0;
 }
 
@@ -709,13 +718,15 @@ int rxe_requester(void *arg)
 	}
 
 	mask = rxe_opcode[opcode].mask;
-	if (unlikely(mask & RXE_READ_OR_ATOMIC_MASK)) {
+	if (unlikely(mask & (RXE_READ_OR_ATOMIC_MASK |
+			RXE_ATOMIC_WRITE_MASK))) {
 		if (check_init_depth(qp, wqe))
 			goto exit;
 	}
 
 	mtu = get_mtu(qp);
-	payload = (mask & RXE_WRITE_OR_SEND_MASK) ? wqe->dma.resid : 0;
+	payload = (mask & (RXE_WRITE_OR_SEND_MASK | RXE_ATOMIC_WRITE_MASK)) ?
+			wqe->dma.resid : 0;
 	if (payload > mtu) {
 		if (qp_type(qp) == IB_QPT_UD) {
 			/* C10-93.1.1: If the total sum of all the buffer lengths specified for a
Signature · 1 line
2.34.1
#8 lore raw

PATCH v6 6/8 RDMA/rxe: Make responder support atomic write on RC service

In reply to [email protected]

Make responder process an atomic write request and send a read response on RC service.

---
 drivers/infiniband/sw/rxe/rxe_resp.c | 84 ++++++++++++++++++++++++++--
 1 file changed, 79 insertions(+), 5 deletions(-)
drivers/infiniband/sw/rxe/rxe_resp.c +79 -5
index ed5a09e86417..7d313d9a8d88 100644
@@ -22,6 +22,7 @@ enum resp_states {
 	RESPST_EXECUTE,
 	RESPST_READ_REPLY,
 	RESPST_ATOMIC_REPLY,
+	RESPST_ATOMIC_WRITE_REPLY,
 	RESPST_COMPLETE,
 	RESPST_ACKNOWLEDGE,
 	RESPST_CLEANUP,
@@ -57,6 +58,7 @@ static char *resp_state_name[] = {
 	[RESPST_EXECUTE]			= "EXECUTE",
 	[RESPST_READ_REPLY]			= "READ_REPLY",
 	[RESPST_ATOMIC_REPLY]			= "ATOMIC_REPLY",
+	[RESPST_ATOMIC_WRITE_REPLY]		= "ATOMIC_WRITE_REPLY",
 	[RESPST_COMPLETE]			= "COMPLETE",
 	[RESPST_ACKNOWLEDGE]			= "ACKNOWLEDGE",
 	[RESPST_CLEANUP]			= "CLEANUP",
@@ -260,7 +262,7 @@ static enum resp_states check_op_valid(struct rxe_qp *qp,
 	case IB_QPT_RC:
 		if (((pkt->mask & RXE_READ_MASK) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_READ)) ||
-		    ((pkt->mask & RXE_WRITE_MASK) &&
+		    ((pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_WRITE)) ||
 		    ((pkt->mask & RXE_ATOMIC_MASK) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_ATOMIC))) {
@@ -364,7 +366,7 @@ static enum resp_states check_resource(struct rxe_qp *qp,
 		}
 	}
 
-	if (pkt->mask & RXE_READ_OR_ATOMIC_MASK) {
+	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		/* it is the requesters job to not send
 		 * too many read/atomic ops, we just
 		 * recycle the responder resource queue
@@ -415,7 +417,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
 	enum resp_states state;
 	int access;
 
-	if (pkt->mask & RXE_READ_OR_WRITE_MASK) {
+	if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		if (pkt->mask & RXE_RETH_MASK) {
 			qp->resp.va = reth_va(pkt);
 			qp->resp.offset = 0;
@@ -483,7 +485,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
 		goto err;
 	}
 
-	if (pkt->mask & RXE_WRITE_MASK)	 {
+	if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		if (resid > mtu) {
 			if (pktlen != mtu || bth_pad(pkt)) {
 				state = RESPST_ERR_LENGTH;
@@ -583,6 +585,7 @@ static struct resp_res *rxe_prepare_res(struct rxe_qp *qp,
 		res->state = rdatm_res_state_new;
 		break;
 	case RXE_ATOMIC_MASK:
+	case RXE_ATOMIC_WRITE_MASK:
 		res->first_psn = pkt->psn;
 		res->last_psn = pkt->psn;
 		res->cur_psn = pkt->psn;
@@ -652,6 +655,55 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 	return ret;
 }
 
+static enum resp_states atomic_write_reply(struct rxe_qp *qp,
+						struct rxe_pkt_info *pkt)
+{
+	u64 src, *dst;
+	struct resp_res *res = qp->resp.res;
+	struct rxe_mr *mr = qp->resp.mr;
+	int payload = payload_size(pkt);
+
+	if (!res) {
+		res = rxe_prepare_res(qp, pkt, RXE_ATOMIC_WRITE_MASK);
+		qp->resp.res = res;
+	}
+
+	if (!res->replay) {
+#ifdef CONFIG_64BIT
+		if (mr->state != RXE_MR_STATE_VALID)
+			return RESPST_ERR_RKEY_VIOLATION;
+
+		memcpy(&src, payload_addr(pkt), payload);
+
+		dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload);
+		/* check vaddr is 8 bytes aligned. */
+		if (!dst || (uintptr_t)dst & 7)
+			return RESPST_ERR_MISALIGNED_ATOMIC;
+
+		/* Do atomic write after all prior operations have completed */
+		smp_store_release(dst, src);
+
+		/* decrease resp.resid to zero */
+		qp->resp.resid -= sizeof(payload);
+
+		qp->resp.msn++;
+
+		/* next expected psn, read handles this separately */
+		qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
+		qp->resp.ack_psn = qp->resp.psn;
+
+		qp->resp.opcode = pkt->opcode;
+		qp->resp.status = IB_WC_SUCCESS;
+
+		return RESPST_ACKNOWLEDGE;
+#else
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+#endif /* CONFIG_64BIT */
+	}
+
+	return RESPST_ACKNOWLEDGE;
+}
+
 static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
 					  struct rxe_pkt_info *ack,
 					  int opcode,
@@ -888,6 +940,8 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt)
 		return RESPST_READ_REPLY;
 	} else if (pkt->mask & RXE_ATOMIC_MASK) {
 		return RESPST_ATOMIC_REPLY;
+	} else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		return RESPST_ATOMIC_WRITE_REPLY;
 	} else {
 		/* Unreachable */
 		WARN_ON_ONCE(1);
@@ -1061,6 +1115,19 @@ static int send_atomic_ack(struct rxe_qp *qp, u8 syndrome, u32 psn)
 	return ret;
 }
 
+static int send_read_response_ack(struct rxe_qp *qp, u8 syndrome, u32 psn)
+{
+	int ret = send_common_ack(qp, syndrome, psn,
+			IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY,
+			"RDMA READ response of length zero ACK");
+
+	/* have to clear this since it is used to trigger
+	 * long read replies
+	 */
+	qp->resp.res = NULL;
+	return ret;
+}
+
 static enum resp_states acknowledge(struct rxe_qp *qp,
 				    struct rxe_pkt_info *pkt)
 {
@@ -1071,6 +1138,8 @@ static enum resp_states acknowledge(struct rxe_qp *qp,
 		send_ack(qp, qp->resp.aeth_syndrome, pkt->psn);
 	else if (pkt->mask & RXE_ATOMIC_MASK)
 		send_atomic_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
+	else if (pkt->mask & RXE_ATOMIC_WRITE_MASK)
+		send_read_response_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
 	else if (bth_ack(pkt))
 		send_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
 
@@ -1182,7 +1251,9 @@ static enum resp_states duplicate_request(struct rxe_qp *qp,
 			res->replay = 1;
 			res->cur_psn = pkt->psn;
 			qp->resp.res = res;
-			rc = RESPST_ATOMIC_REPLY;
+			rc = pkt->mask & RXE_ATOMIC_MASK ?
+					RESPST_ATOMIC_REPLY :
+					RESPST_ATOMIC_WRITE_REPLY;
 			goto out;
 		}
 
@@ -1320,6 +1391,9 @@ int rxe_responder(void *arg)
 		case RESPST_ATOMIC_REPLY:
 			state = atomic_reply(qp, pkt);
 			break;
+		case RESPST_ATOMIC_WRITE_REPLY:
+			state = atomic_write_reply(qp, pkt);
+			break;
 		case RESPST_ACKNOWLEDGE:
 			state = acknowledge(qp, pkt);
 			break;
Signature · 1 line
2.34.1
#9 lore raw

PATCH v6 7/8 RDMA/rxe: Implement atomic write completion

In reply to [email protected]

Generate an atomic write completion when the atomic write request has been finished.

---
 drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/infiniband/sw/rxe/rxe_comp.c +4 -0
index fb0c008af78c..46dba6f163ca 100644
@@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode)
 	case IB_WR_LOCAL_INV:			return IB_WC_LOCAL_INV;
 	case IB_WR_REG_MR:			return IB_WC_REG_MR;
 	case IB_WR_BIND_MW:			return IB_WC_BIND_MW;
+	case IB_WR_ATOMIC_WRITE:		return IB_WC_ATOMIC_WRITE;
 
 	default:
 		return 0xff;
@@ -258,6 +259,9 @@ static inline enum comp_state check_ack(struct rxe_qp *qp,
 		if ((syn & AETH_TYPE_MASK) != AETH_ACK)
 			return COMPST_ERROR;
 
+		if (wqe->wr.opcode == IB_WR_ATOMIC_WRITE)
+			return COMPST_WRITE_SEND;
+
 		fallthrough;
 		/* (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE doesn't have an AETH)
 		 */
Signature · 1 line
2.34.1
#10 lore raw

PATCH v6 8/8 RDMA/rxe: Enable atomic write capability for rxe device

In reply to [email protected]

The capability shows that rxe device supports atomic write operation.

---
 drivers/infiniband/sw/rxe/rxe_param.h | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/sw/rxe/rxe_param.h +5 -0
index 86c7a8bf3cbb..bbc88cd71d95 100644
@@ -51,7 +51,12 @@ enum rxe_device_param {
 					| IB_DEVICE_SRQ_RESIZE
 					| IB_DEVICE_MEM_MGT_EXTENSIONS
 					| IB_DEVICE_MEM_WINDOW
+#ifdef CONFIG_64BIT
+					| IB_DEVICE_MEM_WINDOW_TYPE_2B
+					| IB_DEVICE_ATOMIC_WRITE,
+#else
 					| IB_DEVICE_MEM_WINDOW_TYPE_2B,
+#endif /* CONFIG_64BIT */
 	RXE_MAX_SGE			= 32,
 	RXE_MAX_WQE_SIZE		= sizeof(struct rxe_send_wqe) +
 					  sizeof(struct ib_sge) * RXE_MAX_SGE,
Signature · 1 line
2.34.1
#11 lore raw
In reply to [email protected]

Hi Jason, Bob

Thanks a lot for your review. I have updated the patch set according to your all comments except the missing kmap issue.

I didn't understand why current iova_to_vaddr() has been broken so I hope we can discuss the issue fully and then find a suitable solution.

Best Regards,
Xiao Yang

Quoted 73 lines · Xiao/杨 晓

The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset.

The steps to run the rdma_atomic_write example:
server:

$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

v4->v5:
1) Rebase on current wip/jgg-for-next
2) Rewrite the implementation on responder

v3->v4:
1) Rebase on current wip/jgg-for-next
2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
3) Replace 64-bit value with 8-byte array for atomic write

V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device

V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed

Xiao Yang (8):
RDMA: Extend RDMA user ABI to support atomic write
RDMA: Extend RDMA kernel ABI to support atomic write RDMA/rxe: Extend rxe user ABI to support atomic write RDMA/rxe: Extend rxe packet format to support atomic write RDMA/rxe: Make requester support atomic write on RC service RDMA/rxe: Make responder support atomic write on RC service RDMA/rxe: Implement atomic write completion RDMA/rxe: Enable atomic write capability for rxe device

  drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
  drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
  drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
  drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
  drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
  drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
  include/rdma/ib_pack.h                 |  2 +
  include/rdma/ib_verbs.h                |  3 +
  include/uapi/rdma/ib_user_verbs.h      |  4 ++
  include/uapi/rdma/rdma_user_rxe.h      |  1 +
  10 files changed, 132 insertions(+), 7 deletions(-)
maintainer
#12 lore raw
In reply to [email protected]
Quoted 26 lines · On Sat, Oct 15, 2022 at 06:37:03AM +0000, [email protected] wrote:

The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset.

The steps to run the rdma_atomic_write example:
server:

$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

Quoted 6 lines

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

I think this looked OK, please fix the enum thing and also resolve all the remarks on the github and rebase/repost/retest both series.

Thanks,
Jason

author
#13 lore raw
Quoted 34 lines · Jason Gunthorpe
Quoted 26 lines · On Sat, Oct 15, 2022 at 06:37:03AM +0000, [email protected] wrote:

The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset.

The steps to run the rdma_atomic_write example:
server:

$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

Quoted 6 lines

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

Quoted 2 lines

I think this looked OK, please fix the enum thing and also resolve all the remarks on the github and rebase/repost/retest both series.

Hi Jason,

Thanks for your reminder. I will do it soon.
In addition, I have resolved all remarks except the following one on github:
EdwardSro: "keep an empty line at EoF"
I: "I wonder why we need to add an empty line at EoF? I think there is no empty line at EOF in other files."

I hope you or EdwardSro can answer my question. ^_^

Best Regards,
Xiao Yang

Quoted 3 lines

Thanks,
Jason

maintainer
#14 lore raw
Quoted 37 lines · Xiao Yang
Quoted 34 lines · Jason Gunthorpe
Quoted 26 lines · On Sat, Oct 15, 2022 at 06:37:03AM +0000, [email protected] wrote:

The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset.

The steps to run the rdma_atomic_write example:
server:

$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

Quoted 6 lines

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

Quoted 2 lines

I think this looked OK, please fix the enum thing and also resolve all the remarks on the github and rebase/repost/retest both series.

Quoted 7 lines

Hi Jason,

Thanks for your reminder. I will do it soon.
In addition, I have resolved all remarks except the following one on github:
EdwardSro: "keep an empty line at EoF"
I: "I wonder why we need to add an empty line at EoF? I think there is no empty line at EOF in other files."

It is not really "empty line" it is that the last character in the file should be '\n', and all files are like that.

Jason

author
#15 lore raw
Quoted 2 lines · Jason Gunthorpe

It is not really "empty line" it is that the last character in the file should be '\n', and all files are like that.

Hi Jason,

Thanks for your explanation.

I think my latest patch has added an "empty line", right?

pyverbs/libibverbs_enums.pxd +0 -0
index 6a875fdd..c78a2284 100644

...
@@ -507,4 +510,4 @@ cdef extern from "<infiniband/tm_types.h>":
IBV_TMH_NO_TAG
IBV_TMH_RNDV
IBV_TMH_FIN
- IBV_TMH_EAGER
\ No newline at end of file
+ IBV_TMH_EAGER

# cat -A pyverbs/libibverbs_enums.pxd | tail -2
        IBV_TMH_FIN$
        IBV_TMH_EAGER$

Best Regards,
Xiao Yang

Quoted 2 lines

Jason

maintainer
#16 lore raw
Quoted 8 lines · Xiao Yang
Quoted 2 lines · Jason Gunthorpe

It is not really "empty line" it is that the last character in the file should be '\n', and all files are like that.

Hi Jason,

Thanks for your explanation.

I think my latest patch has added an "empty line", right?

yes

Jason