cifs: smbd: Don't destroy transport on RDMA disconnect
Subsystems (2)
PATCH 1/5 cifs: smbd: Don't destroy transport on RDMA disconnect
From: Long Li <[email protected]>
Now upper layer is handling the transport shutdown and reconnect, remove the code that handling transport shutdown on RDMA disconnect.
--- fs/cifs/cifs_debug.c | 8 ++-- fs/cifs/smbdirect.c | 120 +++------------------------------------------------ fs/cifs/smbdirect.h | 9 ---- 3 files changed, 10 insertions(+), 127 deletions(-)
fs/cifs/cifs_debug.c +3 -5
@@ -312,12 +312,10 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v)atomic_read (&server->smbd_conn->send_credits),atomic_read (&server->smbd_conn->receive_credits), server->smbd_conn->receive_credit_target); -seq_printf (m,"\nPending send_pending: %x send_payload_pending:" -" %x smbd_send_pending: %x smbd_recv_pending: %x" , +seq_printf (m,"\nPending send_pending: %x " +"send_payload_pending: %x" ,atomic_read (&server->smbd_conn->send_pending), -atomic_read (&server->smbd_conn->send_payload_pending), - server->smbd_conn->smbd_send_pending, - server->smbd_conn->smbd_recv_pending); +atomic_read (&server->smbd_conn->send_payload_pending)); seq_printf(m, "\nReceive buffers count_receive_queue: %x " "count_empty_packet_queue: %x", server->smbd_conn->count_receive_queue,
fs/cifs/smbdirect.c +7 -113
186 lines not on this page · raw message
fs/cifs/smbdirect.h +0 -9
@@ -70,13 +70,11 @@ struct smbd_connection {int ri_rc;struct completion ri_done;wait_queue_head_t conn_wait; -wait_queue_head_t wait_destroy;wait_queue_head_t disconn_wait;struct completion negotiate_completion;bool negotiate_done; -struct work_struct destroy_work;struct work_struct disconnect_work;struct work_struct recv_done_work;struct work_struct post_send_credits_work;
@@ -124,13 +122,6 @@ struct smbd_connection {wait_queue_head_t wait_for_mr_cleanup;/* Activity accoutning */ -/* Pending reqeusts issued from upper layer */ -int smbd_send_pending; -wait_queue_head_t wait_smbd_send_pending; - -int smbd_recv_pending; -wait_queue_head_t wait_smbd_recv_pending; -atomic_t send_pending;wait_queue_head_t wait_send_pending;atomic_t send_payload_pending;
Signature · 1 line
2.7.4
PATCH 2/5 cifs: smbd: Return EINTR when interrupted
From: Long Li <[email protected]>
When packets are waiting for outbound I/O and interrupted, return the proper error code to user process.
--- fs/cifs/smbdirect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/cifs/smbdirect.c +1 -1
@@ -1972,7 +1972,7 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, info->transport_status != SMBD_CONNECTED);/* Don't return any data if interrupted */ if (rc) -return -ENODEV ; +return rc;if (info->transport_status !=SMBD_CONNECTED ) {log_read (ERR ,"disconnected\n" );
Signature · 1 line
2.7.4
PATCH 3/5 cifs: smbd: Indicate to retry on transport sending failure
In reply to Long LiFrom: Long Li <[email protected]>
Failure to send a packet doesn't mean it's a permanent failure, it can't be returned to user process. This I/O should be retried or failed based on server packet response and transport health. This logic is handled by the upper layer.
Give this decision to upper layer.
--- fs/cifs/smbdirect.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/cifs/smbdirect.c +3 -2
@@ -853,7 +853,7 @@ static int smbd_create_header(struct smbd_connection *info,if (info->transport_status !=SMBD_CONNECTED ) {log_outgoing (ERR ,"disconnected not sending\n" ); -return -ENOENT ; +return -EAGAIN ; }atomic_dec (&info->send_credits);
@@ -979,6 +979,7 @@ static int smbd_post_send(struct smbd_connection *info,wake_up (&info->wait_send_pending); }smbd_disconnect_rdma_connection (info); + rc = -EAGAIN ; }else /* Reset timer for idle connection after packet is sent */ mod_delayed_work(info->workqueue, &info->idle_timer_work,
@@ -2085,7 +2086,7 @@ int smbd_send(struct TCP_Server_Info *server, struct smb_rqst *rqst)int rc;if (info->transport_status !=SMBD_CONNECTED ) { - rc = -ENODEV ; + rc = -EAGAIN ; goto done; }
Signature · 1 line
2.7.4
PATCH 4/5 cifs: smbd: Retry on memory registration failure
In reply to Long LiFrom: Long Li <[email protected]>
Memory registration failure doesn't mean this I/O has failed, it means the transport is hitting I/O error or needs reconnect. This error is not from the server.
Indicate this error to upper layer, and let upper layer decide how to reconnect and proceed with this I/O.
--- fs/cifs/smb2pdu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/cifs/smb2pdu.c +2 -2
@@ -3220,7 +3220,7 @@ smb2_new_read_req(void **buf, unsigned int *total_len, rdata->nr_pages, rdata->page_offset, rdata->tailsz, true, need_invalidate);if (!rdata->mr) -return -ENOBUFS ; +return -EAGAIN ; req->Channel =SMB2_CHANNEL_RDMA_V1_INVALIDATE ;if (need_invalidate)
@@ -3624,7 +3624,7 @@ smb2_async_writev(struct cifs_writedata *wdata, wdata->nr_pages, wdata->page_offset, wdata->tailsz, false, need_invalidate);if (!wdata->mr) { - rc = -ENOBUFS ; + rc = -EAGAIN ; goto async_writev_out; } req->Length =0 ;
Signature · 1 line
2.7.4
PATCH 5/5 cifs: Call MID callback before destroying transport
In reply to Long LiFrom: Long Li <[email protected]>
When transport is being destroyed, it's possible that some processes may hold memory registrations that need to be deregistred.
Call them first so nobody is using transport resources, and it can be destroyed.
--- fs/cifs/connect.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-)
fs/cifs/connect.c +19 -17
@@ -528,22 +528,6 @@ cifs_reconnect(struct TCP_Server_Info *server)/* do not want to be sending data on a socket we are freeing */ cifs_dbg (FYI ,"%s: tearing down socket\n" , __func__);mutex_lock (&server->srv_mutex); -if (server->ssocket) { -cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -kernel_sock_shutdown (server->ssocket,SHUT_WR ); -cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -sock_release (server->ssocket); - server->ssocket =NULL ; - }else if (cifs_rdma_enabled (server)) -smbd_destroy (server); - server->sequence_number =0 ; - server->session_estab = false; -kfree (server->session_key.response); - server->session_key.response =NULL ; - server->session_key.len =0 ; - server->lstrp = jiffies;/* mark submitted MIDs for retry and issue callback */ INIT_LIST_HEAD (&retry_list);
@@ -556,7 +540,6 @@ cifs_reconnect(struct TCP_Server_Info *server)list_move (&mid_entry->qhead, &retry_list); }spin_unlock (&GlobalMid_Lock); -mutex_unlock (&server->srv_mutex);cifs_dbg (FYI ,"%s: issuing mid callbacks\n" , __func__);list_for_each_safe (tmp, tmp2, &retry_list) {
@@ -565,6 +548,25 @@ cifs_reconnect(struct TCP_Server_Info *server) mid_entry->callback (mid_entry); } +if (server->ssocket) { +cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +kernel_sock_shutdown (server->ssocket,SHUT_WR ); +cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +sock_release (server->ssocket); + server->ssocket =NULL ; + }else if (cifs_rdma_enabled (server)) +smbd_destroy (server); + server->sequence_number =0 ; + server->session_estab = false; +kfree (server->session_key.response); + server->session_key.response =NULL ; + server->session_key.len =0 ; + server->lstrp = jiffies; + +mutex_unlock (&server->srv_mutex); + do { try_to_freeze();
Signature · 1 line
2.7.4
Re: [Patch (resend) 5/5] cifs: Call MID callback before destroying transport
пт, 5 апр. 2019 г. в 14:39, Long Li <[email protected]>:
Quoted 50 lines
From: Long Li <[email protected]>
When transport is being destroyed, it's possible that some processes may hold memory registrations that need to be deregistred.
Call them first so nobody is using transport resources, and it can be destroyed.
Signed-off-by Long Li <[email protected]>--- fs/cifs/connect.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-)fs/cifs/connect.c +0
-17@@ -528,22 +528,6 @@ cifs_reconnect(struct TCP_Server_Info *server)/* do not want to be sending data on a socket we are freeing */ cifs_dbg (FYI ,"%s: tearing down socket\n" , __func__);mutex_lock (&server->srv_mutex); -if (server->ssocket) { -cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -kernel_sock_shutdown (server->ssocket,SHUT_WR ); -cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -sock_release (server->ssocket); - server->ssocket =NULL ; - }else if (cifs_rdma_enabled (server)) -smbd_destroy (server); - server->sequence_number =0 ; - server->session_estab = false; -kfree (server->session_key.response); - server->session_key.response =NULL ; - server->session_key.len =0 ; - server->lstrp = jiffies;/* mark submitted MIDs for retry and issue callback */ INIT_LIST_HEAD (&retry_list);@@ -556,7 +540,6 @@ cifs_reconnect(struct TCP_Server_Info *server)list_move (&mid_entry->qhead, &retry_list); }spin_unlock (&GlobalMid_Lock); -mutex_unlock (&server->srv_mutex);cifs_dbg (FYI ,"%s: issuing mid callbacks\n" , __func__);list_for_each_safe (tmp, tmp2, &retry_list) {
fs/cifs/connect.c cifs_reconnect(struct TCP_Server_Info *server) · line 549
Quoted 2 lines
fs/cifs/connect.c +0
-0@@ -565,6 +548,25 @@ cifs_reconnect(struct TCP_Server_Info *server) mid_entry->callback (mid_entry); }
The original call was issuing callbacks without holding srv_mutex - callbacks may take this mutex for its internal needs. With the proposed patch the code will deadlock.
Also the idea of destroying the socket first is to allow possible retries (from callbacks) to return a proper error instead of trying to send anything through the reconnecting socket.
fs/cifs/connect.c cifs_reconnect(struct TCP_Server_Info *server) · line 560
Quoted 11 lines
fs/cifs/connect.c +10
-0+if (server->ssocket) { +cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +kernel_sock_shutdown (server->ssocket,SHUT_WR ); +cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +sock_release (server->ssocket); + server->ssocket =NULL ; + }else if (cifs_rdma_enabled (server)) +smbd_destroy (server);
If we need to call smbd_destroy() *after* callbacks, let's just move it alone without the rest of the code.
Quoted 13 lines
fs/cifs/connect.c +9
-1+ server->sequence_number =0 ; + server->session_estab = false; +kfree (server->session_key.response); + server->session_key.response =NULL ; + server->session_key.len =0 ; + server->lstrp = jiffies; + +mutex_unlock (&server->srv_mutex); + do { try_to_freeze(); --
Quoted 2 lines
2.7.4
--
Best regards,
Pavel Shilovsky
RE: [Patch (resend) 5/5] cifs: Call MID callback before destroying transport
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
-----Original Message-----
From: Pavel Shilovsky <[email protected]>
Sent: Thursday, May 9, 2019 11:01 AM
To: Long Li <[email protected]>
Cc: Steve French <[email protected]>; linux-cifs <linux- [email protected]>; samba-technical <[email protected]>; Kernel Mailing List <[email protected]>
Subject: Re: [Patch (resend) 5/5] cifs: Call MID callback before destroying transportпт, 5 апр. 2019 г. в 14:39, Long Li <[email protected]>:
Quoted 50 lines
From: Long Li <[email protected]>
When transport is being destroyed, it's possible that some processes may hold memory registrations that need to be deregistred.
Call them first so nobody is using transport resources, and it can be destroyed.
Signed-off-by Long Li <[email protected]>--- fs/cifs/connect.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-)fs/cifs/connect.c index renamed from fs/cifs/connect.c +0
-033e4d98..084756cf 100644
fs/cifs/connect.c +0
-17@@ -528,22 +528,6 @@ cifs_reconnect(struct TCP_Server_Info *server)/* do not want to be sending data on a socket we are freeing */ cifs_dbg (FYI ,"%s: tearing down socket\n" , __func__);mutex_lock (&server->srv_mutex); -if (server->ssocket) { -cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -kernel_sock_shutdown (server->ssocket,SHUT_WR ); -cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , - server->ssocket->state, server->ssocket->flags); -sock_release (server->ssocket); - server->ssocket =NULL ; - }else if (cifs_rdma_enabled (server)) -smbd_destroy (server); - server->sequence_number =0 ; - server->session_estab = false; -kfree (server->session_key.response); - server->session_key.response =NULL ; - server->session_key.len =0 ; - server->lstrp = jiffies;/* mark submitted MIDs for retry and issue callback */ INIT_LIST_HEAD (&retry_list);@@ -556,7 +540,6 @@ cifs_reconnect(struct TCP_Server_Info *server)list_move (&mid_entry->qhead, &retry_list); }spin_unlock (&GlobalMid_Lock); -mutex_unlock (&server->srv_mutex);cifs_dbg (FYI ,"%s: issuing mid callbacks\n" , __func__); list_for_each_safe(tmp, tmp2, &retry_list) { @@ -565,6 +548,25@@
cifs_reconnect(struct TCP_Server_Info *server)Quoted 2 lines
mid_entry->callback(mid_entry); }The original call was issuing callbacks without holding
srv_mutex- callbacks may take this mutex for its internal needs. With the proposed patch the code will deadlock.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Also the idea of destroying the socket first is to allow possible retries (from callbacks) to return a proper error instead of trying to send anything through the reconnecting socket.
I will send a patch to revert this and follow your suggestion on putting smbd_destroy() to after all MIDs have been called. Your suggestion tested well.
Thanks
Long
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
fs/cifs/connect.c cifs_reconnect(struct TCP_Server_Info *server) · line 560
Quoted 12 lines
fs/cifs/connect.c +10
-0+if (server->ssocket) { +cifs_dbg (FYI ,"State: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +kernel_sock_shutdown (server->ssocket,SHUT_WR ); +cifs_dbg (FYI ,"Post shutdown state: 0x%x Flags: 0x%lx\n" , + server->ssocket->state, server->ssocket->flags); +sock_release (server->ssocket); + server->ssocket =NULL ; + }else if (cifs_rdma_enabled (server)) +smbd_destroy (server);If we need to call
smbd_destroy()*after* callbacks, let's just move it alone without the rest of the code.Quoted 13 lines
fs/cifs/connect.c +9
-1+ server->sequence_number =0 ; + server->session_estab = false; +kfree (server->session_key.response); + server->session_key.response =NULL ; + server->session_key.len =0 ; + server->lstrp = jiffies; + +mutex_unlock (&server->srv_mutex); + do { try_to_freeze(); --Quoted 2 lines
2.7.4
--
Best regards,
Pavel Shilovsky