Skip to contentSkip to conversation

cifs: smbd: Improve reliability on transport reconnect

PATCH In review [email protected] 2 participants 10 messages Oct 16, 2019 – Oct 27, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover cifs: smbd: Improve reliability on transport reconnect

From: Long Li <[email protected]>

Long Li (7):
cifs: Don't display RDMA transport on reconnect
cifs: smbd: Invalidate and deregister memory registration on re-send
cifs: smbd: Return -EINVAL when the number of iovs exceeds
SMBDIRECT_MAX_SGE
cifs: smbd: Add messages on RDMA session destroy and reconnection
cifs: smbd: Return -ECONNABORTED when trasnport is not in connected
state
cifs: smbd: Only queue work for error recovery on memory registration
cifs: smbd: Return -EAGAIN when transport is reconnecting

 fs/cifs/cifs_debug.c |  5 +++++
 fs/cifs/file.c       | 19 +++++++++++++++++--
 fs/cifs/smbdirect.c  | 36 +++++++++++++++++++++---------------
 fs/cifs/transport.c  |  7 +++++--
 4 files changed, 48 insertions(+), 19 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/7 cifs: Don't display RDMA transport on reconnect

From: Long Li <[email protected]>

On reconnect, the transport data structure is NULL and its information is not available.

---
 fs/cifs/cifs_debug.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/cifs/cifs_debug.c +5 -0
index 0b4eee3bed66..efb2928ff6c8 100644
@@ -256,6 +256,11 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v)
 		if (!server->rdma)
 			goto skip_rdma;
 
+		if (!server->smbd_conn) {
+			seq_printf(m, "\nSMBDirect transport not available");
+			goto skip_rdma;
+		}
+
 		seq_printf(m, "\nSMBDirect (in hex) protocol version: %x "
 			"transport status: %x",
 			server->smbd_conn->protocol,
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/7 cifs: smbd: Invalidate and deregister memory registration on re-send

In reply to [email protected]

From: Long Li <[email protected]>

On re-send, there might be a reconnect and all prevoius memory registrations need to be invalidated and deregistered.

---
 fs/cifs/file.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
fs/cifs/file.c +17 -2
index 4b95700c507c..3c4e01e56798 100644
@@ -2747,9 +2747,17 @@ cifs_resend_wdata(struct cifs_writedata *wdata, struct list_head *wdata_list,
 		if (!rc) {
 			if (wdata->cfile->invalidHandle)
 				rc = -EAGAIN;
-			else
+			else {
+#ifdef CONFIG_CIFS_SMB_DIRECT
+				if (wdata->mr) {
+					wdata->mr->need_invalidate = true;
+					smbd_deregister_mr(wdata->mr);
+					wdata->mr = NULL;
+				}
+#endif
 				rc = server->ops->async_writev(wdata,
 					cifs_uncached_writedata_release);
+			}
 		}
 
 		/* If the write was successfully sent, we are done */
@@ -3472,7 +3480,14 @@ static int cifs_resend_rdata(struct cifs_readdata *rdata,
 		if (!rc) {
 			if (rdata->cfile->invalidHandle)
 				rc = -EAGAIN;
-			else
+			else {
+#ifdef CONFIG_CIFS_SMB_DIRECT
+				if (rdata->mr) {
+					rdata->mr->need_invalidate = true;
+					smbd_deregister_mr(rdata->mr);
+					rdata->mr = NULL;
+				}
+#endif
 				rc = server->ops->async_readv(rdata);
 		}
 
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/7 cifs: smbd: Return -EINVAL when the number of iovs exceeds SMBDIRECT_MAX_SGE

In reply to [email protected]

From: Long Li <[email protected]>

While it's not friendly to fail user processes that issue more iovs than we support, at least we should return the correct error code so the user process gets a chance to retry with smaller number of iovs.

---
 fs/cifs/smbdirect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/cifs/smbdirect.c +1 -1
index cd07e5301d42..d41a9345f90d 100644
@@ -1069,7 +1069,7 @@ static int smbd_post_send_data(
 
 	if (n_vec > SMBDIRECT_MAX_SGE) {
 		cifs_dbg(VFS, "Can't fit data to SGL, n_vec=%d\n", n_vec);
-		return -ENOMEM;
+		return -EINVAL;
 	}
 
 	sg_init_table(sgl, n_vec);
Signature · 1 line
2.17.1
#4 lore raw

PATCH 4/7 cifs: smbd: Add messages on RDMA session destroy and reconnection

In reply to [email protected]

From: Long Li <[email protected]>

Log these activities to help production support.

---
 fs/cifs/smbdirect.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/cifs/smbdirect.c +4 -2
index d41a9345f90d..227ef51c0712 100644
@@ -1476,6 +1476,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
 	info->transport_status = SMBD_DESTROYED;
 
 	destroy_workqueue(info->workqueue);
+	log_rdma_event(INFO,  "rdma session destroyed\n");
 	kfree(info);
 }
 
@@ -1505,8 +1506,9 @@ int smbd_reconnect(struct TCP_Server_Info *server)
 	log_rdma_event(INFO, "creating rdma session\n");
 	server->smbd_conn = smbd_get_connection(
 		server, (struct sockaddr *) &server->dstaddr);
-	log_rdma_event(INFO, "created rdma session info=%p\n",
-		server->smbd_conn);
+
+	if (server->smbd_conn)
+		cifs_dbg(VFS, "RDMA transport re-established\n");
 
 	return server->smbd_conn ? 0 : -ENOENT;
 }
Signature · 1 line
2.17.1
#5 lore raw

PATCH 5/7 cifs: smbd: Return -ECONNABORTED when trasnport is not in connected state

In reply to [email protected]

From: Long Li <[email protected]>

The transport should return this error so the upper layer will reconnect.

---
 fs/cifs/smbdirect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/cifs/smbdirect.c +1 -1
index 227ef51c0712..cf001f10d555 100644
@@ -1970,7 +1970,7 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,
 
 	if (info->transport_status != SMBD_CONNECTED) {
 		log_read(ERR, "disconnected\n");
-		return 0;
+		return -ECONNABORTED;
 	}
 
 	goto again;
Signature · 1 line
2.17.1
#6 lore raw

PATCH 6/7 cifs: smbd: Only queue work for error recovery on memory registration

In reply to [email protected]

From: Long Li <[email protected]>

It's not necessary to queue invalidated memory registration to work queue, as all we need to do is to unmap the SG and make it usable again. This can save CPU cycles in normal data paths as memory registration errors are rare and normally only happens during reconnection.

---
 fs/cifs/smbdirect.c | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)
fs/cifs/smbdirect.c +15 -11
index cf001f10d555..c00629a41d81 100644
@@ -2269,12 +2269,7 @@ static void smbd_mr_recovery_work(struct work_struct *work)
 	int rc;
 
 	list_for_each_entry(smbdirect_mr, &info->mr_list, list) {
-		if (smbdirect_mr->state == MR_INVALIDATED)
-			ib_dma_unmap_sg(
-				info->id->device, smbdirect_mr->sgl,
-				smbdirect_mr->sgl_count,
-				smbdirect_mr->dir);
-		else if (smbdirect_mr->state == MR_ERROR) {
+		if (smbdirect_mr->state == MR_ERROR) {
 
 			/* recover this MR entry */
 			rc = ib_dereg_mr(smbdirect_mr->mr);
@@ -2602,11 +2597,20 @@ int smbd_deregister_mr(struct smbd_mr *smbdirect_mr)
 		 */
 		smbdirect_mr->state = MR_INVALIDATED;
 
-	/*
-	 * Schedule the work to do MR recovery for future I/Os
-	 * MR recovery is slow and we don't want it to block the current I/O
-	 */
-	queue_work(info->workqueue, &info->mr_recovery_work);
+	if (smbdirect_mr->state == MR_INVALIDATED) {
+		ib_dma_unmap_sg(
+			info->id->device, smbdirect_mr->sgl,
+			smbdirect_mr->sgl_count,
+			smbdirect_mr->dir);
+		smbdirect_mr->state = MR_READY;
+		if (atomic_inc_return(&info->mr_ready_count) == 1)
+			wake_up_interruptible(&info->wait_mr);
+	} else
+		/*
+		 * Schedule the work to do MR recovery for future I/Os
+		 * MR recovery is slow and we don't want it to block the
+		 * current I/O */
+		queue_work(info->workqueue, &info->mr_recovery_work);
 
 done:
 	if (atomic_dec_and_test(&info->mr_used_count))
Signature · 1 line
2.17.1
#7 lore raw

Re: [PATCH 6/7] cifs: smbd: Only queue work for error recovery on memory registration

I cleaned up minor cosmetic nit spotted by checkpatch

$ scripts/checkpatch.pl
0001-cifs-smbd-Only-queue-work-for-error-recovery-on-memo.patch
WARNING: Possible unwrapped commit description (prefer a maximum 75
chars per line)
#7:
It's not necessary to queue invalidated memory registration to work queue, as

WARNING: Block comments use a trailing */ on a separate line #58: FILE: fs/cifs/smbdirect.c:2614:
+ * current I/O */

total: 0 errors, 2 warnings, 38 lines checked

Quoted 62 lines · longli--- via samba-technical

From: Long Li <[email protected]>

It's not necessary to queue invalidated memory registration to work queue, as all we need to do is to unmap the SG and make it usable again. This can save CPU cycles in normal data paths as memory registration errors are rare and normally only happens during reconnection.

---
 fs/cifs/smbdirect.c | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)
fs/cifs/smbdirect.c +15 -12
index cf001f10d555..c00629a41d81 100644
@@ -2269,12 +2269,7 @@ static void smbd_mr_recovery_work(struct work_struct *work)
        int rc;
 
        list_for_each_entry(smbdirect_mr, &info->mr_list, list) {
-               if (smbdirect_mr->state == MR_INVALIDATED)
-                       ib_dma_unmap_sg(
-                               info->id->device, smbdirect_mr->sgl,
-                               smbdirect_mr->sgl_count,
-                               smbdirect_mr->dir);
-               else if (smbdirect_mr->state == MR_ERROR) {
+               if (smbdirect_mr->state == MR_ERROR) {
 
                        /* recover this MR entry */
                        rc = ib_dereg_mr(smbdirect_mr->mr);
@@ -2602,11 +2597,20 @@ int smbd_deregister_mr(struct smbd_mr *smbdirect_mr)
                 */
                smbdirect_mr->state = MR_INVALIDATED;
 
-       /*
-        * Schedule the work to do MR recovery for future I/Os
-        * MR recovery is slow and we don't want it to block the current I/O
-        */
-       queue_work(info->workqueue, &info->mr_recovery_work);
+       if (smbdirect_mr->state == MR_INVALIDATED) {
+               ib_dma_unmap_sg(
+                       info->id->device, smbdirect_mr->sgl,
+                       smbdirect_mr->sgl_count,
+                       smbdirect_mr->dir);
+               smbdirect_mr->state = MR_READY;
+               if (atomic_inc_return(&info->mr_ready_count) == 1)
+                       wake_up_interruptible(&info->wait_mr);
+       } else
+               /*
+                * Schedule the work to do MR recovery for future I/Os
+                * MR recovery is slow and we don't want it to block the
+                * current I/O */
+               queue_work(info->workqueue, &info->mr_recovery_work);
 
 done:
        if (atomic_dec_and_test(&info->mr_used_count))
--

2.17.1

Signature · 3 lines
Thanks,

Steve
#8 lore raw

PATCH 7/7 cifs: smbd: Return -EAGAIN when transport is reconnecting

In reply to [email protected]

From: Long Li <[email protected]>

During reconnecting, the transport may have already been destroyed and is in the process being reconnected. In this case, return -EAGAIN to not fail and to retry this I/O.

---
 fs/cifs/transport.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/cifs/transport.c +5 -2
index afe66f9cb15e..66fde7bfec4f 100644
@@ -318,8 +318,11 @@ __smb_send_rqst(struct TCP_Server_Info *server, int num_rqst,
 	int val = 1;
 	__be32 rfc1002_marker;
 
-	if (cifs_rdma_enabled(server) && server->smbd_conn) {
-		rc = smbd_send(server, num_rqst, rqst);
+	if (cifs_rdma_enabled(server)) {
+		/* return -EAGAIN when connecting or reconnecting */
+		rc = -EAGAIN;
+		if (server->smbd_conn)
+			rc = smbd_send(server, num_rqst, rqst);
 		goto smbd_done;
 	}
 
Signature · 1 line
2.17.1
#9 lore raw

Re: [PATCH 7/7] cifs: smbd: Return -EAGAIN when transport is reconnecting

Tentatively merged the 7 patch series into cifs-2.6.git for-next pending more reviews and xfstests regression testing runs

Quoted 35 lines · longli--- via samba-technical

From: Long Li <[email protected]>

During reconnecting, the transport may have already been destroyed and is in the process being reconnected. In this case, return -EAGAIN to not fail and to retry this I/O.

---
 fs/cifs/transport.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/cifs/transport.c +5 -3
index afe66f9cb15e..66fde7bfec4f 100644
@@ -318,8 +318,11 @@ __smb_send_rqst(struct TCP_Server_Info *server, int num_rqst,
        int val = 1;
        __be32 rfc1002_marker;
 
-       if (cifs_rdma_enabled(server) && server->smbd_conn) {
-               rc = smbd_send(server, num_rqst, rqst);
+       if (cifs_rdma_enabled(server)) {
+               /* return -EAGAIN when connecting or reconnecting */
+               rc = -EAGAIN;
+               if (server->smbd_conn)
+                       rc = smbd_send(server, num_rqst, rqst);
                goto smbd_done;
        }
 
--

2.17.1

Signature · 3 lines
Thanks,

Steve