Skip to contentSkip to conversation

RDMA/rxe: skip adjusting remote addr for write in retry operation

RFC PATCH Applied Chengguang Xu 4 participants 5 messages May 2, 2022 – May 6, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 RDMA/rxe: skip adjusting remote addr for write in retry operation

For write request the remote addr will be sent only with first packet so we don't have to adjust wqe->iova in retry operation.

---
 drivers/infiniband/sw/rxe/rxe_req.c | 2 --
 1 file changed, 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_req.c +0 -2
index ae5fbc79dd5c..f08010651ef7 100644
@@ -33,8 +33,6 @@ static inline void retry_first_write_send(struct rxe_qp *qp,
 		} else {
 			advance_dma_data(&wqe->dma, to_send);
 		}
-		if (mask & WR_WRITE_MASK)
-			wqe->iova += qp->mtu;
 	}
 }
 
Signature · 1 line
2.35.1
#1 lore raw
Quoted 8 lines

-----Original Message-----
From: Chengguang Xu <[email protected]>
Sent: Monday, May 2, 2022 12:39 AM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Chengguang Xu <[email protected]>
Subject: [RFC PATCH] RDMA/rxe: skip adjusting remote addr for write in retry operation

For write request the remote addr will be sent only with first packet so we don't have to adjust wqe->iova in retry operation.

This is problematic for lossy networks. A very large read request, say 8MiB, sends 2048 packets in response without any acknowledgement from the requester. If the packet loss rate was 1% the read request would never finish as the probability of sending 2048 packets without loss is very small. The way the code works today is that the iova is adjusted, and if you are lucky the responder has already finished the previous read operation and starts over with a new read reply starting with a first packet at iova. If you are less fortunate the previous read reply has not finished and the responder will continue to work on it until it is finished before looking at the new read request wqe. the completer will respond to each out of order packet by checking to see if it should start a retry but since it has already done so it drops the packet. It's messy but one can make forward progress ~100 packets at a time. It would be faster if the responder saw that the new request replaced the old one and stopped sending packets on the old read. I have no idea how CX NICs do this but just restarting from scratch seems bad.

Bob

---
 drivers/infiniband/sw/rxe/rxe_req.c | 2 --
 1 file changed, 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_req.c +0 -2
index ae5fbc79dd5c..f08010651ef7 100644
@@ -33,8 +33,6 @@ static inline void retry_first_write_send(struct rxe_qp *qp,
 		} else {
 			advance_dma_data(&wqe->dma, to_send);
 		}
-		if (mask & WR_WRITE_MASK)
-			wqe->iova += qp->mtu;
 	}
 }
 

--
2.35.1

#2 lore raw
Quoted 8 lines · Robert B
Quoted 8 lines

-----Original Message-----
From: Chengguang Xu <[email protected]>
Sent: Monday, May 2, 2022 12:39 AM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Chengguang Xu <[email protected]>
Subject: [RFC PATCH] RDMA/rxe: skip adjusting remote addr for write in retry operation

For write request the remote addr will be sent only with first packet so we don't have to adjust wqe->iova in retry operation.

Quoted 9 lines

This is problematic for lossy networks. A very large read request, say 8MiB, sends 2048 packets in response without any acknowledgement from the requester. If the packet loss rate was 1% the read request would never finish as the probability of sending 2048 packets without loss is very small. The way the code works today is that the iova is adjusted, and if you are lucky the responder has already finished the previous read operation and starts over with a new read reply starting with a first packet at iova. If you are less fortunate the previous read reply has not finished and the responder will continue to work on it until it is finished before looking at the new read request wqe. the completer will respond to each out of order packet by checking to see if it should start a retry but since it has already done so it drops the packet. It's messy but one can make forward progress ~100 packets at a time. It would be faster if the responder saw that the new request replaced the old one and stopped sending packets on the old read. I have no idea how CX NICs do this but just restarting from scratch seems bad.

I agree that read request indeed needs to adjust iova during retry and  the adjustment(for read) has already done in below logic in req_retry().

if (mask & WR_READ_MASK) {
        npsn = (wqe->dma.length - wqe->dma.resid) / qp->mtu;
        wqe->iova += npsn * qp->mtu;
}

For write request, retry will not send new iova because only first write packet has RXE_RETH_MASK regardless iova adjustment. Am I missing something?

Thanks,
Chengguang

#3 lore raw

Sorry I misread your original post. You are correct. The wqe->iova is only used to fill in the RETH header so it is not needed after the first packet.
This commit looks OK.

Quoted 42 lines · Chengguang Xu
Quoted 8 lines · Robert B
Quoted 8 lines

-----Original Message-----
From: Chengguang Xu <[email protected]>
Sent: Monday, May 2, 2022 12:39 AM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Chengguang Xu <[email protected]>
Subject: [RFC PATCH] RDMA/rxe: skip adjusting remote addr for write in retry operation

For write request the remote addr will be sent only with first packet so we don't have to adjust wqe->iova in retry operation.

Quoted 9 lines

This is problematic for lossy networks. A very large read request, say 8MiB, sends 2048 packets in response without any acknowledgement from the requester. If the packet loss rate was 1% the read request would never finish as the probability of sending 2048 packets without loss is very small. The way the code works today is that the iova is adjusted, and if you are lucky the responder has already finished the previous read operation and starts over with a new read reply starting with a first packet at iova. If you are less fortunate the previous read reply has not finished and the responder will continue to work on it until it is finished before looking at the new read request wqe. the completer will respond to each out of order packet by checking to see if it should start a retry but since it has already done so it drops the packet. It's messy but one can make forward progress ~100 packets at a time. It would be faster if the responder saw that the new request replaced the old one and stopped sending packets on the old read. I have no idea how CX NICs do this but just restarting from scratch seems bad.

I agree that read request indeed needs to adjust iova during retry and the adjustment(for read) has already done in below logic in req_retry().

if (mask & WR_READ_MASK) {
         npsn = (wqe->dma.length - wqe->dma.resid) /
                      qp->mtu;
         wqe->iova += npsn * qp->mtu;
}

For write request, retry will not send new iova because only first write packet has RXE_RETH_MASK regardless iova adjustment.
Am I missing something?

Thanks,
Chengguang