cifs: overhaul request timeout behavior in CIFS (try #1)
Subsystems (2)
cover cifs: overhaul request timeout behavior in CIFS (try #1)
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in TASK_UNINTERRUPTIBLE sleep however, the client waits in TASK_KILLABLE sleep so that fatal signals will end the sleep and return -ERESTARTSYS to the caller.
In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean up handle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on -EAGAIN errors
cifs: handle cancelled requests better
cifs: make wait_for_free_request take a TCP_Server_Info pointer
cifs: add cifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requests
fs/cifs/cifs_debug.c | 8 +- fs/cifs/cifsglob.h | 19 ++-- fs/cifs/cifspdu.h | 15 +++ fs/cifs/cifsproto.h | 7 + fs/cifs/cifssmb.c | 55 ++++++++- fs/cifs/connect.c | 128 +++++++++++++++---- fs/cifs/file.c | 48 ++------ fs/cifs/sess.c | 2 +- fs/cifs/transport.c | 344 ++++++++++++++++++++++---------------------------- 9 files changed, 355 insertions(+), 271 deletions(-)
Signature · 1 line
1.7.3.2
A message that is not in the archive (sent off-list or not yet received): 1290862886-2914-1-git-send-email-jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
PATCH 1/14 cifs: move mid result processing into common function
--- fs/cifs/transport.c | 131 ++++++++++++++++++-------------------------------- 1 files changed, 47 insertions(+), 84 deletions(-)
fs/cifs/transport.c +47 -84
@@ -382,6 +382,46 @@ SendReceiveNoRsp(const unsigned int xid, struct cifsSesInfo *ses,return rc; } +static int +handle_mid_result (struct mid_q_entry *mid,struct TCP_Server_Info *server) +{ +int rc =0 ; + +spin_lock (&GlobalMid_Lock); + +if (mid->resp_buf) { +spin_unlock (&GlobalMid_Lock); +return rc; + } + +cERROR (1 ,"No response to cmd %d mid %d" , mid->command, mid->mid); +if (mid->midState ==MID_REQUEST_SUBMITTED ) { +if (server->tcpStatus == CifsExiting) + rc = -EHOSTDOWN ; +else { + server->tcpStatus = CifsNeedReconnect; + mid->midState =MID_RETRY_NEEDED ; + } + } + +if (rc != -EHOSTDOWN ) { +if (mid->midState ==MID_RETRY_NEEDED ) { + rc = -EAGAIN ; +cFYI (1 ,"marking request for retry" ); + }else { + rc = -EIO ; + } + } +spin_unlock (&GlobalMid_Lock); + +DeleteMidQEntry (mid); +/* Update # of requests on wire to server */ +atomic_dec (&server->inFlight); +wake_up (&server->request_q); + +return rc; +} + int SendReceive2(const unsigned int xid, struct cifsSesInfo *ses, struct kvec *iov, int n_vec, int *pRespBufType /* ret */,
@@ -485,37 +525,10 @@ SendReceive2(const unsigned int xid, struct cifsSesInfo *ses,/* No user interrupts in wait - wreaks havoc with performance */ wait_for_response (ses, midQ, timeout,10 *HZ ); -spin_lock (&GlobalMid_Lock); - -if (midQ->resp_buf ==NULL ) { -cERROR (1 ,"No response to cmd %d mid %d" , - midQ->command, midQ->mid); -if (midQ->midState ==MID_REQUEST_SUBMITTED ) { -if (ses->server->tcpStatus == CifsExiting) - rc = -EHOSTDOWN ; -else { - ses->server->tcpStatus = CifsNeedReconnect; - midQ->midState =MID_RETRY_NEEDED ; - } - } - -if (rc != -EHOSTDOWN ) { -if (midQ->midState ==MID_RETRY_NEEDED ) { - rc = -EAGAIN ; -cFYI (1 ,"marking request for retry" ); - }else { - rc = -EIO ; - } - } -spin_unlock (&GlobalMid_Lock); -DeleteMidQEntry (midQ); -/* Update # of requests on wire to server */ -atomic_dec (&ses->server->inFlight); -wake_up (&ses->server->request_q); + rc =handle_mid_result (midQ, ses->server); +if (rc !=0 )return rc; - } -spin_unlock (&GlobalMid_Lock); receive_len = midQ->resp_buf->smb_buf_length;if (receive_len > CIFSMaxBufSize +MAX_CIFS_HDR_SIZE ) {
@@ -677,36 +690,10 @@ SendReceive(const unsigned int xid, struct cifsSesInfo *ses,/* No user interrupts in wait - wreaks havoc with performance */ wait_for_response (ses, midQ, timeout,10 *HZ ); -spin_lock (&GlobalMid_Lock); -if (midQ->resp_buf ==NULL ) { -cERROR (1 ,"No response for cmd %d mid %d" , - midQ->command, midQ->mid); -if (midQ->midState ==MID_REQUEST_SUBMITTED ) { -if (ses->server->tcpStatus == CifsExiting) - rc = -EHOSTDOWN ; -else { - ses->server->tcpStatus = CifsNeedReconnect; - midQ->midState =MID_RETRY_NEEDED ; - } - } - -if (rc != -EHOSTDOWN ) { -if (midQ->midState ==MID_RETRY_NEEDED ) { - rc = -EAGAIN ; -cFYI (1 ,"marking request for retry" ); - }else { - rc = -EIO ; - } - } -spin_unlock (&GlobalMid_Lock); -DeleteMidQEntry (midQ); -/* Update # of requests on wire to server */ -atomic_dec (&ses->server->inFlight); -wake_up (&ses->server->request_q); + rc =handle_mid_result (midQ, ses->server); +if (rc !=0 )return rc; - } -spin_unlock (&GlobalMid_Lock); receive_len = midQ->resp_buf->smb_buf_length;if (receive_len > CIFSMaxBufSize +MAX_CIFS_HDR_SIZE ) {
@@ -926,35 +913,11 @@ SendReceiveBlockingLock(const unsigned int xid, struct cifsTconInfo *tcon, } } -spin_lock (&GlobalMid_Lock); -if (midQ->resp_buf) { -spin_unlock (&GlobalMid_Lock); - receive_len = midQ->resp_buf->smb_buf_length; - }else { -cERROR (1 ,"No response for cmd %d mid %d" , - midQ->command, midQ->mid); -if (midQ->midState ==MID_REQUEST_SUBMITTED ) { -if (ses->server->tcpStatus == CifsExiting) - rc = -EHOSTDOWN ; -else { - ses->server->tcpStatus = CifsNeedReconnect; - midQ->midState =MID_RETRY_NEEDED ; - } - } - -if (rc != -EHOSTDOWN ) { -if (midQ->midState ==MID_RETRY_NEEDED ) { - rc = -EAGAIN ; -cFYI (1 ,"marking request for retry" ); - }else { - rc = -EIO ; - } - } -spin_unlock (&GlobalMid_Lock); -DeleteMidQEntry (midQ); + rc =handle_mid_result (midQ, ses->server); +if (rc !=0 )return rc; - } + receive_len = midQ->resp_buf->smb_buf_length;if (receive_len > CIFSMaxBufSize +MAX_CIFS_HDR_SIZE ) {cERROR (1 ,"Frame too large received. Length: %d Xid: %d" , receive_len, xid);
Signature · 1 line
1.7.3.2
PATCH 2/14 cifs: wait indefinitely for responses
In reply to a message not in the archiveThe client should not be timing out on individual SMB requests. Too much of the state between client and server is tied to the state of the socket. If we time out requests and issue spurious disconnects then that comprimises data integrity.
Instead of doing this complicated dance where we try to decide how long to wait for a response for particular requests, have the client instead wait indefinitely for a response. Also, use a TASK_KILLABLE sleep here so that fatal signals will break out of this waiting.
Later patches will add support for detecting dead peers and forcing reconnects based on that.
--- fs/cifs/transport.c | 110 ++++++++------------------------------------------- 1 files changed, 17 insertions(+), 93 deletions(-)
fs/cifs/transport.c +17 -93
@@ -311,48 +311,17 @@ static int allocate_mid(struct cifsSesInfo *ses, struct smb_hdr *in_buf,return 0 ; } -static int wait_for_response (struct cifsSesInfo *ses, -struct mid_q_entry *midQ, -unsigned long timeout, -unsigned long time_to_wait) +static int +wait_for_response (struct TCP_Server_Info *server,struct mid_q_entry *midQ) { -unsigned long curr_timeout; - -for (;;) { - curr_timeout = timeout + jiffies; -wait_event_timeout (ses->server->response_q, - midQ->midState !=MID_REQUEST_SUBMITTED , timeout); - -if (time_after (jiffies, curr_timeout) && - (midQ->midState ==MID_REQUEST_SUBMITTED ) && - ((ses->server->tcpStatus == CifsGood) || - (ses->server->tcpStatus == CifsNew))) { - -unsigned long lrt; +int error; -/* We timed out. Is the server still -sending replies ? */ -spin_lock (&GlobalMid_Lock); - lrt = ses->server->lstrp; -spin_unlock (&GlobalMid_Lock); + error =wait_event_killable (server->response_q, + midQ->midState !=MID_REQUEST_SUBMITTED ); +if (error <0 ) +return -ERESTARTSYS ; -/* Calculate time_to_wait past last receive time. -Although we prefer not to time out if the -server is still responding - we will time -out if the server takes more than 15 (or 45 -or 180) seconds to respond to this request -and has not responded to any request from -other threads on the client within 10 seconds */ - lrt += time_to_wait; -if (time_after (jiffies, lrt)) { -/* No replies for time_to_wait. */ -cERROR (1 ,"server not responding" ); -return -1 ; - } - }else { -return 0 ; - } - } +return 0 ; }
@@ -430,7 +399,6 @@ SendReceive2(const unsigned int xid, struct cifsSesInfo *ses,int rc =0 ;int long_op;unsigned int receive_len; -unsigned long timeout;struct mid_q_entry *midQ;struct smb_hdr *in_buf = iov[0 ].iov_base;
@@ -497,33 +465,12 @@ SendReceive2(const unsigned int xid, struct cifsSesInfo *ses,if (rc <0 ) goto out; -if (long_op ==CIFS_STD_OP ) - timeout =15 *HZ ; -else if (long_op ==CIFS_VLONG_OP )/* e.g. slow writes past EOF */ - timeout =180 *HZ ; -else if (long_op ==CIFS_LONG_OP ) - timeout =45 *HZ ;/* should be greater than -servers oplock break timeout (about 43 seconds) */ -else if (long_op ==CIFS_ASYNC_OP ) - goto out; -else if (long_op ==CIFS_BLOCKING_OP ) - timeout =0x7FFFFFFF ;/* large, but not so large as to wrap */ -else { -cERROR (1 ,"unknown timeout flag %d" , long_op); - rc = -EIO ; +if (long_op ==CIFS_ASYNC_OP ) goto out; - } - -/* wait for 15 seconds or until woken up due to response arriving or -due to last connection to this server being unmounted */ -if (signal_pending (current)) { -/* if signal pending do not hold up user for full smb timeout -but we still give response a chance to complete */ - timeout =2 *HZ ; - } -/* No user interrupts in wait - wreaks havoc with performance */ -wait_for_response (ses, midQ, timeout,10 *HZ ); + rc =wait_for_response (ses->server, midQ); +if (rc !=0 ) + goto out; rc =handle_mid_result (midQ, ses->server);if (rc !=0 )
@@ -598,7 +545,6 @@ SendReceive(const unsigned int xid, struct cifsSesInfo *ses, {int rc =0 ;unsigned int receive_len; -unsigned long timeout;struct mid_q_entry *midQ;if (ses ==NULL ) {
@@ -662,33 +608,12 @@ SendReceive(const unsigned int xid, struct cifsSesInfo *ses,if (rc <0 ) goto out; -if (long_op ==CIFS_STD_OP ) - timeout =15 *HZ ; -/* wait for 15 seconds or until woken up due to response arriving or -due to last connection to this server being unmounted */ -else if (long_op ==CIFS_ASYNC_OP ) - goto out; -else if (long_op ==CIFS_VLONG_OP )/* writes past EOF can be slow */ - timeout =180 *HZ ; -else if (long_op ==CIFS_LONG_OP ) - timeout =45 *HZ ;/* should be greater than -servers oplock break timeout (about 43 seconds) */ -else if (long_op ==CIFS_BLOCKING_OP ) - timeout =0x7FFFFFFF ;/* large but no so large as to wrap */ -else { -cERROR (1 ,"unknown timeout flag %d" , long_op); - rc = -EIO ; +if (long_op ==CIFS_ASYNC_OP ) goto out; - } -if (signal_pending (current)) { -/* if signal pending do not hold up user for full smb timeout -but we still give response a chance to complete */ - timeout =2 *HZ ; - } - -/* No user interrupts in wait - wreaks havoc with performance */ -wait_for_response (ses, midQ, timeout,10 *HZ ); + rc =wait_for_response (ses->server, midQ); +if (rc !=0 ) + goto out; rc =handle_mid_result (midQ, ses->server);if (rc !=0 )
@@ -906,8 +831,7 @@ SendReceiveBlockingLock(const unsigned int xid, struct cifsTconInfo *tcon, } } -/* Wait 5 seconds for the response. */ -if (wait_for_response (ses, midQ,5 *HZ ,5 *HZ ) ==0 ) { +if (wait_for_response (ses->server, midQ) ==0 ) {/* We got the response - restart system call. */ rstart =1 ; }
Signature · 1 line
1.7.3.2
PATCH 3/14 cifs: don't reconnect server when we don't get a response
In reply to a message not in the archiveWe only want to force a reconnect to the server under very limited and specific circumstances. Now that we have processes waiting indefinitely for responses, we shouldn't reach this point unless a reconnect is already in process. Thus, there's no reason to re-mark the server for reconnect here.
--- fs/cifs/transport.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)
fs/cifs/transport.c +1 -3
@@ -367,10 +367,8 @@ handle_mid_result(struct mid_q_entry *mid, struct TCP_Server_Info *server)if (mid->midState ==MID_REQUEST_SUBMITTED ) {if (server->tcpStatus == CifsExiting) rc = -EHOSTDOWN ; -else { - server->tcpStatus = CifsNeedReconnect; +else mid->midState =MID_RETRY_NEEDED ; - } }if (rc != -EHOSTDOWN ) {
Signature · 1 line
1.7.3.2
PATCH 4/14 cifs: clean up handle_mid_response
In reply to a message not in the archiveMake it use a switch statement based on the value of the midStatus. If the resp_buf is set, then MID_RESPONSE_RECEIVED is too.
--- fs/cifs/transport.c | 36 ++++++++++++++++++------------------ 1 files changed, 18 insertions(+), 18 deletions(-)
fs/cifs/transport.c +18 -18
@@ -356,33 +356,33 @@ handle_mid_result(struct mid_q_entry *mid, struct TCP_Server_Info *server) {int rc =0 ; -spin_lock (&GlobalMid_Lock); +cFYI (1 ,"%s: cmd=%d mid=%d state=%d" , __func__, mid->command, + mid->mid, mid->midState); -if (mid->resp_buf) { +spin_lock (&GlobalMid_Lock); +switch (mid->midState) { +case MID_RESPONSE_RECEIVED :spin_unlock (&GlobalMid_Lock);return rc; - } - -cERROR (1 ,"No response to cmd %d mid %d" , mid->command, mid->mid); -if (mid->midState ==MID_REQUEST_SUBMITTED ) { -if (server->tcpStatus == CifsExiting) +case MID_REQUEST_SUBMITTED : +/* socket is going down, reject all calls */ +if (server->tcpStatus == CifsExiting) { +cERROR (1 ,"%s: canceling mid=%d cmd=0x%x state=%d" , + __func__, mid->mid, mid->command, mid->midState); rc = -EHOSTDOWN ; -else - mid->midState =MID_RETRY_NEEDED ; - } - -if (rc != -EHOSTDOWN ) { -if (mid->midState ==MID_RETRY_NEEDED ) { - rc = -EAGAIN ; -cFYI (1 ,"marking request for retry" ); - }else { - rc = -EIO ; +break ; } + mid->midState =MID_RETRY_NEEDED ; +case MID_RETRY_NEEDED : + rc = -EAGAIN ; +break ; +default : +cERROR (1 ,"%s: invalid mid state mid=%d state=%d" , __func__, + mid->mid, mid->midState); }spin_unlock (&GlobalMid_Lock);DeleteMidQEntry (mid); -/* Update # of requests on wire to server */ atomic_dec (&server->inFlight);wake_up (&server->request_q);
Signature · 1 line
1.7.3.2
PATCH 5/14 cifs: allow for different handling of received response
In reply to a message not in the archiveIn order to incorporate async requests, we need to allow for a more general way to do things on receive, rather than just waking up a process.
Turn the task pointer in the mid_q_entry into a callback function and a generic data pointer. When a response comes in, or the socket is reconnected, cifsd can call the callback function in order to wake up the process.
The default is to just wake up the current process which should mean no change in behavior for existing code.
--- fs/cifs/cifs_debug.c | 8 ++++---- fs/cifs/cifsglob.h | 7 ++++++- fs/cifs/connect.c | 25 ++++++++++++++----------- fs/cifs/transport.c | 14 +++++++++++++- 4 files changed, 37 insertions(+), 17 deletions(-)
fs/cifs/cifs_debug.c +4 -4
@@ -79,11 +79,11 @@ void cifs_dump_mids(struct TCP_Server_Info *server)spin_lock (&GlobalMid_Lock);list_for_each (tmp, &server->pending_mid_q) { mid_entry = list_entry(tmp, struct mid_q_entry, qhead); -cERROR (1 ,"State: %d Cmd: %d Pid: %d Tsk: %p Mid %d" , +cERROR (1 ,"State: %d Cmd: %d Pid: %d Cbdata: %p Mid %d" , mid_entry->midState, (int )mid_entry->command, mid_entry->pid, - mid_entry->tsk, + mid_entry->callback_data, mid_entry->mid);#ifdef CONFIG_CIFS_STATS2 cERROR(1, "IsLarge: %d buf: %p time rcv: %ld now: %ld",
@@ -220,11 +220,11 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v) mid_entry = list_entry(tmp3, struct mid_q_entry, qhead);seq_printf (m,"\tState: %d com: %d pid:" -" %d tsk: %p mid %d\n" , +" %d cbdata: %p mid %d\n" , mid_entry->midState, (int )mid_entry->command, mid_entry->pid, - mid_entry->tsk, + mid_entry->callback_data, mid_entry->mid); }spin_unlock (&GlobalMid_Lock);
fs/cifs/cifsglob.h +6 -1
@@ -501,6 +501,10 @@ static inline void cifs_stats_bytes_read(struct cifsTconInfo *tcon,#endif +struct mid_q_entry ; + +typedef void (mid_callback_t )(struct mid_q_entry *mid); +/* one of these for every pending CIFS request to the server */ struct mid_q_entry {struct list_head qhead;/* mids waiting on reply from this server */
@@ -512,7 +516,8 @@ struct mid_q_entry {unsigned long when_sent;/* time when smb send finished */ unsigned long when_received;/* when demux complete (taken off wire) */ #endif -struct task_struct *tsk;/* task waiting for response */ +mid_callback_t *callback;/* call completion callback */ +void *callback_data;/* general purpose pointer for callback */ struct smb_hdr *resp_buf;/* response buffer */ int midState;/* wish this were enum but can not pass to wait_event */ __u8 command;/* smb command code */
fs/cifs/connect.c +14 -11
@@ -212,10 +212,15 @@ cifs_reconnect(struct TCP_Server_Info *server)if (server->tcpStatus != CifsExiting) server->tcpStatus = CifsGood;spin_unlock (&GlobalMid_Lock); -/* atomic_set(&server->inFlight,0);*/ -wake_up (&server->response_q); } } + +/* now, issue callback for all mids in flight */ +spin_lock (&GlobalMid_Lock); +list_for_each_entry (mid_entry, &server->pending_mid_q, qhead) + mid_entry->callback (mid_entry); +spin_unlock (&GlobalMid_Lock); +return rc; }
@@ -559,8 +564,7 @@ incomplete_rcv:continue ; } - - task_to_wake =NULL ; + mid_entry =NULL ;spin_lock (&GlobalMid_Lock);list_for_each (tmp, &server->pending_mid_q) { mid_entry = list_entry(tmp, struct mid_q_entry, qhead);
@@ -603,7 +607,6 @@ incomplete_rcv: mid_entry->resp_buf = smb_buffer; mid_entry->largeBuf = isLargeBuf; multi_t2_fnd: - task_to_wake = mid_entry->tsk; mid_entry->midState =MID_RESPONSE_RECEIVED ;#ifdef CONFIG_CIFS_STATS2 mid_entry->when_received = jiffies;
@@ -614,9 +617,12 @@ multi_t2_fnd: server->lstrp = jiffies;break ; } + mid_entry =NULL ; }spin_unlock (&GlobalMid_Lock); -if (task_to_wake) { + +if (mid_entry !=NULL ) { + mid_entry->callback (mid_entry);/* Was previous buf put in mpx struct for multi-rsp? */ if (!isMultiRsp) {/* smb buffer will be freed by user thread */
@@ -625,7 +631,6 @@ multi_t2_fnd:else smallbuf =NULL ; } -wake_up_process (task_to_wake); }else if (!is_valid_oplock_break (smb_buffer, server) && !isMultiRsp) { cERROR(1, "No task to wake, unknown frame received! "
@@ -706,11 +711,9 @@ multi_t2_fnd:list_for_each (tmp, &server->pending_mid_q) { mid_entry = list_entry(tmp, struct mid_q_entry, qhead);if (mid_entry->midState ==MID_REQUEST_SUBMITTED ) { -cFYI (1 ,"Clearing Mid 0x%x - waking up " , +cFYI (1 ,"Clearing Mid 0x%x - issuing callback" , mid_entry->mid); - task_to_wake = mid_entry->tsk; -if (task_to_wake) -wake_up_process (task_to_wake); + mid_entry->callback (mid_entry); } }spin_unlock (&GlobalMid_Lock);
fs/cifs/transport.c +13 -1
@@ -36,6 +36,12 @@extern mempool_t *cifs_mid_poolp; +static void +wake_up_task (struct mid_q_entry *mid) +{ +wake_up_process (mid->callback_data); +} +static struct mid_q_entry *AllocMidQEntry (const struct smb_hdr *smb_buffer,struct TCP_Server_Info *server) {
@@ -58,7 +64,13 @@ AllocMidQEntry(const struct smb_hdr *smb_buffer, struct TCP_Server_Info *server)/* do_gettimeofday(&temp->when_sent);*/ /* easier to use jiffies */ /* when mid allocated can be before when sent */ temp->when_alloc = jiffies; - temp->tsk = current; + +/* +* The default is for the mid to be synchronous, so the +* default callback just wakes up the current task. +*/ + temp->callback = wake_up_task; + temp->callback_data = current; }spin_lock (&GlobalMid_Lock);
Signature · 1 line
1.7.3.2
A message that is not in the archive (sent off-list or not yet received): 1290862886-2914-6-git-send-email-jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Re: [PATCH 05/14] cifs: allow for different handling of received response
Quoted 20 lines · On Sat, 27 Nov 2010 08:01:17 -0500 Jeff Layton…
In order to incorporate async requests, we need to allow for a more general way to do things on receive, rather than just waking up a process.
Turn the task pointer in the
mid_q_entryinto a callback function and a generic data pointer. When a response comes in, or the socket is reconnected, cifsd can call the callback function in order to wake up the process.The default is to just wake up the current process which should mean no change in behavior for existing code.
Signed-off-by Jeff Layton <[email protected]>--- fs/cifs/cifs_debug.c | 8 ++++---- fs/cifs/cifsglob.h | 7 ++++++- fs/cifs/connect.c | 25 ++++++++++++++----------- fs/cifs/transport.c | 14 +++++++++++++- 4 files changed, 37 insertions(+), 17 deletions(-)
[...]
Quoted 9 lines
fs/cifs/connect.c +0
-1@@ -212,10 +212,15 @@ cifs_reconnect(struct TCP_Server_Info *server)if (server->tcpStatus != CifsExiting) server->tcpStatus = CifsGood;spin_unlock (&GlobalMid_Lock); -/* atomic_set(&server->inFlight,0);*/
fs/cifs/connect.c cifs_reconnect(struct TCP_Server_Info *server) · line 223
Quoted 10 lines
fs/cifs/connect.c +7
-1-wake_up (&server->response_q); } } + +/* now, issue callback for all mids in flight */ +spin_lock (&GlobalMid_Lock); +list_for_each_entry (mid_entry, &server->pending_mid_q, qhead) + mid_entry->callback (mid_entry); +spin_unlock (&GlobalMid_Lock); +
^^^^^^^^^^^^
It occurs to me that the above can deadlock. It ought not call the callback function under the GlobalMid_Lock. I'll plan to respin this set with a fix for that (probably by moving all of the mids to a separate retry list and having the callback function move them back to the pending_mid_q).
Signature · 1 line
Jeff Layton <[email protected]>
PATCH 6/14 cifs: don't fail writepages on -EAGAIN errors
In reply to a message not in the archiveIf CIFSSMBWrite2 returns -EAGAIN, then we should retry the write instead of setting an error on the mapping and returning.
Note that I'm not 100% thrilled with this patch. If we have an unresponsive server then we'll block in cifs_writepages for quite some time, even in the WB_SYNC_NONE case. That said, we already block in this function in the WB_SYNC_NONE case since CIFSSMBWrite2 is always synchronous anyway, so I don't see this as causing any more harm than is already done.
--- fs/cifs/file.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
fs/cifs/file.c +4 -0
@@ -1432,6 +1432,7 @@ retry:break ; }if (n_iov) { +retry_write: open_file =find_writable_file (CIFS_I (mapping->host), false);if (!open_file) {
@@ -1447,6 +1448,9 @@ retry:cifs_update_eof (cifsi, offset, bytes_written); } +if (rc == -EAGAIN ) + goto retry_write; +if (rc || bytes_written < bytes_to_write) {cERROR (1 ,"Write2 ret %d, wrote %d" , rc, bytes_written);
Signature · 1 line
1.7.3.2
PATCH 7/14 cifs: handle cancelled requests better
In reply to a message not in the archiveCurrently, when a request is cancelled via signal, we delete the mid immediately. If the request was already transmitted however, the client is still likely to receive a response. When it does, it won't recognize it however and will pop a printk.
It's also a little dangerous to just delete the mid entry like this. We may end up reusing that mid. If we do then we could potentially get the response from the first request confused with the later one.
Prevent the reuse of mids by marking them as cancelled and keeping them on the pending_mid_q list. If the reply comes in, we'll delete it from the list then. If it never comes, then we'll delete it at reconnect or when cifsd comes down.
--- fs/cifs/cifsglob.h | 2 +- fs/cifs/cifsproto.h | 1 + fs/cifs/connect.c | 50 ++++++++++++++++++++++++++++++++++++++++---------- fs/cifs/transport.c | 30 ++++++++++++++++++++++-------- 4 files changed, 64 insertions(+), 19 deletions(-)
fs/cifs/cifsglob.h +1 -1
@@ -611,7 +611,7 @@ static inline void free_dfs_info_array(struct dfs_info3_param *param,#define MID_REQUEST_SUBMITTED 2#define MID_RESPONSE_RECEIVED 4#define MID_RETRY_NEEDED 8/* session closed while this request out */ -#define MID_NO_RESP_NEEDED 0x10 +#define MID_REQUEST_CANCELLED 0x10/* discard any reply */ /* Types of response buffer returned from SendReceive2 */ #define CIFS_NO_BUFFER 0/* Response buffer not returned */
fs/cifs/cifsproto.h +1 -0
@@ -60,6 +60,7 @@ extern char *cifs_compose_mount_options(const char *sb_mountdata, const char *fullpath, const struct dfs_info3_param *ref, char **devname);/* extern void renew_parental_timestamps(struct dentry *direntry);*/ +extern void DeleteMidQEntry (struct mid_q_entry *midEntry); extern int SendReceive(const unsigned int /* xid */ , struct cifsSesInfo *, struct smb_hdr * /* input */ , struct smb_hdr * /* out */ ,
fs/cifs/connect.c +40 -10
114 lines not on this page · raw message
fs/cifs/transport.c +22 -8
@@ -81,7 +81,7 @@ AllocMidQEntry(const struct smb_hdr *smb_buffer, struct TCP_Server_Info *server)return temp; } -static void +void DeleteMidQEntry (struct mid_q_entry *midEntry) {#ifdef CONFIG_CIFS_STATS2
@@ -479,8 +479,13 @@ SendReceive2(const unsigned int xid, struct cifsSesInfo *ses, goto out; rc =wait_for_response (ses->server, midQ); -if (rc !=0 ) - goto out; +if (rc !=0 ) { +/* no longer considered to be "in-flight" */ + midQ->midState =MID_REQUEST_CANCELLED ; +atomic_dec (&ses->server->inFlight); +wake_up (&ses->server->request_q); +return rc; + } rc =handle_mid_result (midQ, ses->server);if (rc !=0 )
@@ -622,8 +627,13 @@ SendReceive(const unsigned int xid, struct cifsSesInfo *ses, goto out; rc =wait_for_response (ses->server, midQ); -if (rc !=0 ) - goto out; +if (rc !=0 ) { +/* no longer considered to be "in-flight" */ + midQ->midState =MID_REQUEST_CANCELLED ; +atomic_dec (&ses->server->inFlight); +wake_up (&ses->server->request_q); +return rc; + } rc =handle_mid_result (midQ, ses->server);if (rc !=0 )
@@ -841,10 +851,14 @@ SendReceiveBlockingLock(const unsigned int xid, struct cifsTconInfo *tcon, } } -if (wait_for_response (ses->server, midQ) ==0 ) { -/* We got the response - restart system call. */ - rstart =1 ; + rc =wait_for_response (ses->server, midQ); +if (rc) { + midQ->midState =MID_REQUEST_CANCELLED ; +return rc; } + +/* We got the response - restart system call. */ + rstart =1 ; } rc =handle_mid_result (midQ, ses->server);
Signature · 1 line
1.7.3.2
PATCH 8/14 cifs: make wait_for_free_request take a TCP_Server_Info pointer
In reply to a message not in the archiveThe cifsSesInfo pointer is only used to get at the server.
--- fs/cifs/transport.c | 26 +++++++++++++------------- 1 files changed, 13 insertions(+), 13 deletions(-)
fs/cifs/transport.c +13 -13
@@ -256,31 +256,31 @@ smb_send(struct TCP_Server_Info *server, struct smb_hdr *smb_buffer,return smb_sendv (server, &iov,1 ); } -static int wait_for_free_request (struct cifsSesInfo *ses,const int long_op) +static int wait_for_free_request (struct TCP_Server_Info *server, +const int long_op) {if (long_op ==CIFS_ASYNC_OP ) {/* oplock breaks must not be held up */ -atomic_inc (&ses->server->inFlight); +atomic_inc (&server->inFlight);return 0 ; }spin_lock (&GlobalMid_Lock);while (1 ) { -if (atomic_read (&ses->server->inFlight) >= - cifs_max_pending){ +if (atomic_read (&server->inFlight) >= cifs_max_pending) {spin_unlock (&GlobalMid_Lock);#ifdef CONFIG_CIFS_STATS2 -atomic_inc (&ses->server->num_waiters); +atomic_inc (&server->num_waiters);#endif -wait_event (ses->server->request_q, -atomic_read (&ses->server->inFlight) +wait_event (server->request_q, +atomic_read (&server->inFlight) < cifs_max_pending);#ifdef CONFIG_CIFS_STATS2 -atomic_dec (&ses->server->num_waiters); +atomic_dec (&server->num_waiters);#endif spin_lock (&GlobalMid_Lock); }else { -if (ses->server->tcpStatus == CifsExiting) { +if (server->tcpStatus == CifsExiting) {spin_unlock (&GlobalMid_Lock);return -ENOENT ; }
@@ -290,7 +290,7 @@ static int wait_for_free_request(struct cifsSesInfo *ses, const int long_op)/* update # of requests on the wire to server */ if (long_op !=CIFS_BLOCKING_OP ) -atomic_inc (&ses->server->inFlight); +atomic_inc (&server->inFlight);spin_unlock (&GlobalMid_Lock);break ; }
@@ -431,7 +431,7 @@ SendReceive2(const unsigned int xid, struct cifsSesInfo *ses,to the same server. We may make this configurable later or use ses->maxReq */ - rc =wait_for_free_request (ses, long_op); + rc =wait_for_free_request (ses->server, long_op);if (rc) {cifs_small_buf_release (in_buf);return rc;
@@ -584,7 +584,7 @@ SendReceive(const unsigned int xid, struct cifsSesInfo *ses,return -EIO ; } - rc =wait_for_free_request (ses, long_op); + rc =wait_for_free_request (ses->server, long_op);if (rc)return rc;
@@ -777,7 +777,7 @@ SendReceiveBlockingLock(const unsigned int xid, struct cifsTconInfo *tcon,return -EIO ; } - rc =wait_for_free_request (ses,CIFS_BLOCKING_OP ); + rc =wait_for_free_request (ses->server,CIFS_BLOCKING_OP );if (rc)return rc;
Signature · 1 line
1.7.3.2
PATCH 9/14 cifs: add cifs_call_async
In reply to a message not in the archiveAdd a function that will send a request, and set up the mid for an async reply.
--- fs/cifs/cifsproto.h | 5 ++++ fs/cifs/transport.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 1 deletions(-)
fs/cifs/cifsproto.h +5 -0
@@ -60,7 +60,12 @@ extern char *cifs_compose_mount_options(const char *sb_mountdata, const char *fullpath, const struct dfs_info3_param *ref, char **devname);/* extern void renew_parental_timestamps(struct dentry *direntry);*/ +extern struct mid_q_entry *AllocMidQEntry (const struct smb_hdr *smb_buffer, +struct TCP_Server_Info *server);extern void DeleteMidQEntry (struct mid_q_entry *midEntry); +extern int cifs_call_async (struct TCP_Server_Info *server, +struct smb_hdr *in_buf,mid_callback_t *callback, +void *cbdata); extern int SendReceive(const unsigned int /* xid */ , struct cifsSesInfo *, struct smb_hdr * /* input */ , struct smb_hdr * /* out */ ,
fs/cifs/transport.c +52 -1
65 lines not on this page · raw message
Signature · 1 line
1.7.3.2
PATCH 10/14 cifs: add ability to send an echo request
In reply to a message not in the archive--- fs/cifs/cifspdu.h | 15 +++++++++++++++ fs/cifs/cifsproto.h | 1 + fs/cifs/cifssmb.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 0 deletions(-)
fs/cifs/cifspdu.h +15 -0
@@ -50,6 +50,7 @@#define SMB_COM_SETATTR 0x09/* trivial response */ #define SMB_COM_LOCKING_ANDX 0x24/* trivial response */ #define SMB_COM_COPY 0x29/* trivial rsp, fail filename ignrd*/ +#define SMB_COM_ECHO 0x2B/* echo request */ #define SMB_COM_OPEN_ANDX 0x2D/* Legacy open for old servers */ #define SMB_COM_READ_ANDX 0x2E#define SMB_COM_WRITE_ANDX 0x2F
@@ -760,6 +761,20 @@ typedef struct smb_com_tconx_rsp_ext {* */ +typedef struct smb_com_echo_req { +struct smb_hdr hdr; +__le16 EchoCount; +__le16 ByteCount; +char Data[1 ]; +} __attribute__((packed))ECHO_REQ ; + +typedef struct smb_com_echo_rsp { +struct smb_hdr hdr; +__le16 SequenceNumber; +__le16 ByteCount; +char Data[1 ]; +} __attribute__((packed))ECHO_RSP ; + typedef struct smb_com_logoff_andx_req { struct smb_hdr hdr; /* wct = 2 */ __u8 AndXCommand;
fs/cifs/cifsproto.h +1 -0
@@ -358,6 +358,7 @@ extern int CIFSSMBPosixLock(const int xid, struct cifsTconInfo *tcon, const __u64 len, struct file_lock *, const __u16 lock_type, const bool waitFlag);extern int CIFSSMBTDis (const int xid,struct cifsTconInfo *tcon); +extern int CIFSSMBEcho (struct TCP_Server_Info *server);extern int CIFSSMBLogoff (const int xid,struct cifsSesInfo *ses);extern struct cifsSesInfo *sesInfoAlloc (void );
fs/cifs/cifssmb.c +47 -0
53 lines not on this page · raw message
Signature · 1 line
1.7.3.2
PATCH 11/14 cifs: set up recurring workqueue job to do SMB echo requests
In reply to a message not in the archive--- fs/cifs/cifsglob.h | 1 + fs/cifs/connect.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 0 deletions(-)
fs/cifs/cifsglob.h +1 -0
@@ -211,6 +211,7 @@ struct TCP_Server_Info {bool sec_kerberosu2u;/* supports U2U Kerberos */ bool sec_ntlmssp;/* supports NTLMSSP */ bool session_estab;/* mark when very first sess is established */ +struct delayed_work echo;/* echo ping workqueue job */ #ifdef CONFIG_CIFS_FSCACHE struct fscache_cookie *fscache;/* client index cache cookie */ #endif
fs/cifs/connect.c +29 -0
@@ -52,6 +52,9 @@#define CIFS_PORT 445#define RFC1001_PORT 139 +/* SMB echo every 60s -- FIXME: tunable? */ +#define SMB_ECHO_INTERVAL (60 * HZ) +extern void SMBNTencrypt (unsigned char *passwd,unsigned char *c8,unsigned char *p24);
@@ -348,6 +351,26 @@ static int coalesce_t2(struct smb_hdr *psecond, struct smb_hdr *pTargetSMB) } +static void +cifs_echo_request (struct work_struct *work) +{ +int rc; +struct TCP_Server_Info *server =container_of (work, + struct TCP_Server_Info, echo.work); + +/* no need to ping if we got a response recently */ +if (time_before (jiffies, server->lstrp +SMB_ECHO_INTERVAL -HZ )) + goto requeue_echo; + + rc =CIFSSMBEcho (server); +if (rc) +cFYI (1 ,"Unable to send echo request to server: %s" , + server->hostname); + +requeue_echo: +queue_delayed_work (system_nrt_wq, &server->echo,SMB_ECHO_INTERVAL ); +} +static int cifs_demultiplex_thread (struct TCP_Server_Info *server) {
@@ -1590,6 +1613,8 @@ cifs_put_tcp_session(struct TCP_Server_Info *server)list_del_init (&server->tcp_ses_list);spin_unlock (&cifs_tcp_ses_lock); +cancel_delayed_work_sync (&server->echo); +spin_lock (&GlobalMid_Lock); server->tcpStatus = CifsExiting;spin_unlock (&GlobalMid_Lock);
@@ -1681,6 +1706,7 @@ cifs_get_tcp_session(struct smb_vol *volume_info) tcp_ses->sequence_number =0 ;INIT_LIST_HEAD (&tcp_ses->tcp_ses_list);INIT_LIST_HEAD (&tcp_ses->smb_ses_list); +INIT_DELAYED_WORK (&tcp_ses->echo, cifs_echo_request);/* * at this point we are the only ones with the pointer
@@ -1730,6 +1756,9 @@ cifs_get_tcp_session(struct smb_vol *volume_info)cifs_fscache_get_client_cookie (tcp_ses); +/* queue echo request delayed work */ +queue_delayed_work (system_nrt_wq, &tcp_ses->echo,SMB_ECHO_INTERVAL ); +return tcp_ses; out_err_crypto_release:
Signature · 1 line
1.7.3.2
PATCH 12/14 cifs: reconnect unresponsive servers
In reply to a message not in the archiveMS-CIFS says:
"The server MUST NOT disconnect clients that have been inactive less
than 5 minutes."
Using that guideline, perform a reconnect to any server that hasn't responded in the last 5 minutes. Eventually, we may want to make this tunable, but it seems like a reasonable default.
--- fs/cifs/connect.c | 21 +++++++++++++++++---- 1 files changed, 17 insertions(+), 4 deletions(-)
fs/cifs/connect.c +17 -4
@@ -55,6 +55,9 @@/* SMB echo every 60s -- FIXME: tunable? */ #define SMB_ECHO_INTERVAL (60 * HZ) +/* reconnect if no response from server in this time period */ +#define UNRESPONSIVE_SERVER_TIMEOUT (5 * 60 * HZ) +extern void SMBNTencrypt (unsigned char *passwd,unsigned char *c8,unsigned char *p24);
@@ -183,6 +186,7 @@ cifs_reconnect(struct TCP_Server_Info *server)kfree (server->session_key.response); server->session_key.response =NULL ; server->session_key.len =0 ; + server->lstrp = jiffies;INIT_LIST_HEAD (&dispose);spin_lock (&GlobalMid_Lock);
@@ -436,6 +440,16 @@ cifs_demultiplex_thread(struct TCP_Server_Info *server) smb_msg.msg_control =NULL ; smb_msg.msg_controllen =0 ; pdu_length =4 ;/* enough to get RFC1001 header */ + +if (time_after (jiffies, + server->lstrp +UNRESPONSIVE_SERVER_TIMEOUT )) { +cFYI (1 ,"Reconnecting unresponsive server %s" , + server->hostname); +cifs_reconnect (server); + csocket = server->ssocket; +wake_up (&server->response_q); +continue ; + } incomplete_rcv: length = kernel_recvmsg(csocket, &smb_msg,
@@ -597,6 +611,8 @@ incomplete_rcv: } mid_entry =NULL ; + server->lstrp = jiffies; +spin_lock (&GlobalMid_Lock);list_for_each (tmp, &server->pending_mid_q) { mid_entry = list_entry(tmp, struct mid_q_entry, qhead);
@@ -647,10 +663,6 @@ multi_t2_fnd:#ifdef CONFIG_CIFS_STATS2 mid_entry->when_received = jiffies;#endif -/* so we do not time out requests to server -which is still responding (since server could -be busy but not dead) */ - server->lstrp = jiffies;break ; } next_mid:
@@ -1704,6 +1716,7 @@ cifs_get_tcp_session(struct smb_vol *volume_info) volume_info->target_rfc1001_name, RFC1001_NAME_LEN_WITH_NULL); tcp_ses->session_estab = false; tcp_ses->sequence_number =0 ; + tcp_ses->lstrp = jiffies;INIT_LIST_HEAD (&tcp_ses->tcp_ses_list);INIT_LIST_HEAD (&tcp_ses->smb_ses_list);INIT_DELAYED_WORK (&tcp_ses->echo, cifs_echo_request);
Signature · 1 line
1.7.3.2
PATCH 13/14 cifs: make hard mounts the default
In reply to a message not in the archiveAllowing calls to time out is fine when a user wants that, however we should default to the behavior that allows for the most robust operation for applications.
Defaulting to soft means unreliable behavior for applications in the face of reconnects. Change the default to hard mounts.
--- fs/cifs/connect.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
fs/cifs/connect.c +2 -1
@@ -899,7 +899,8 @@ cifs_parse_mount_options(char *options, const char *devname,/* default to only allowing write access to owner of the mount */ vol->dir_mode = vol->file_mode =S_IRUGO |S_IXUGO |S_IWUSR ; -/* vol->retry default is 0 (i.e. "soft" limited retry not hard retry) */ +/* default is to retry requests indefinitely */ + vol->retry =1 ;/* default is always to request posix paths. */ vol->posix_paths =1 ;/* default to using server inode numbers where available */
Signature · 1 line
1.7.3.2
PATCH 14/14 cifs: remove code for setting timeouts on requests
In reply to a message not in the archiveSince we don't time out individual requests anymore, remove the code that we used to use for setting timeouts on different requests.
--- fs/cifs/cifsglob.h | 9 +++------ fs/cifs/cifssmb.c | 8 ++++---- fs/cifs/connect.c | 2 +- fs/cifs/file.c | 44 +++++++------------------------------------- fs/cifs/sess.c | 2 +- fs/cifs/transport.c | 2 +- 6 files changed, 17 insertions(+), 50 deletions(-)
fs/cifs/cifsglob.h +3 -6
@@ -621,12 +621,9 @@ static inline void free_dfs_info_array(struct dfs_info3_param *param,#define CIFS_IOVEC 4/* array of response buffers */ /* Type of Request to SendReceive2 */ -#define CIFS_STD_OP 0/* normal request timeout */ -#define CIFS_LONG_OP 1/* long op (up to 45 sec, oplock time) */ -#define CIFS_VLONG_OP 2/* sloow op - can take up to 180 seconds */ -#define CIFS_BLOCKING_OP 4/* operation can block */ -#define CIFS_ASYNC_OP 8/* do not wait for response */ -#define CIFS_TIMEOUT_MASK 0x00F/* only one of 5 above set in req */ +#define CIFS_BLOCKING_OP 1/* operation can block */ +#define CIFS_ASYNC_OP 2/* do not wait for response */ +#define CIFS_TIMEOUT_MASK 0x003/* only one of above set in req */ #define CIFS_LOG_ERROR 0x010/* log NT STATUS if non-zero */ #define CIFS_LARGE_BUF_OP 0x020/* large request buffer */ #define CIFS_NO_RESP 0x040/* no response buffer required */
fs/cifs/cifssmb.c +4 -4
@@ -1243,7 +1243,7 @@ OldOpenRetry: pSMB->ByteCount =cpu_to_le16 (count);/* long_op set to 1 to allow for oplock break timeouts */ rc =SendReceive (xid, tcon->ses, (struct smb_hdr *) pSMB, - (struct smb_hdr *)pSMBr, &bytes_returned,CIFS_LONG_OP ); + (struct smb_hdr *)pSMBr, &bytes_returned,0 );cifs_stats_inc (&tcon->num_opens);if (rc) {cFYI (1 ,"Error in Open = %d" , rc);
@@ -1356,7 +1356,7 @@ openRetry: pSMB->ByteCount =cpu_to_le16 (count);/* long_op set to 1 to allow for oplock break timeouts */ rc =SendReceive (xid, tcon->ses, (struct smb_hdr *) pSMB, - (struct smb_hdr *)pSMBr, &bytes_returned,CIFS_LONG_OP ); + (struct smb_hdr *)pSMBr, &bytes_returned,0 );cifs_stats_inc (&tcon->num_opens);if (rc) {cFYI (1 ,"Error in Open = %d" , rc);
@@ -1438,7 +1438,7 @@ CIFSSMBRead(const int xid, struct cifsTconInfo *tcon, const int netfid, iov[0 ].iov_base = (char *)pSMB; iov[0 ].iov_len = pSMB->hdr.smb_buf_length +4 ; rc =SendReceive2 (xid, tcon->ses, iov,1 /* num iovecs */ , - &resp_buf_type,CIFS_STD_OP |CIFS_LOG_ERROR ); + &resp_buf_type,CIFS_LOG_ERROR );cifs_stats_inc (&tcon->num_reads); pSMBr = (READ_RSP *)iov[0 ].iov_base;if (rc) {
@@ -3136,7 +3136,7 @@ CIFSSMBGetCIFSACL(const int xid, struct cifsTconInfo *tcon, __u16 fid, iov[0 ].iov_len = pSMB->hdr.smb_buf_length +4 ; rc =SendReceive2 (xid, tcon->ses, iov,1 /* num iovec */ , &buf_type, -CIFS_STD_OP ); +0 );cifs_stats_inc (&tcon->num_acl_get);if (rc) {cFYI (1 ,"Send error in QuerySecDesc = %d" , rc);
fs/cifs/connect.c +1 -1
@@ -3111,7 +3111,7 @@ CIFSTCon(unsigned int xid, struct cifsSesInfo *ses, pSMB->ByteCount =cpu_to_le16 (count); rc =SendReceive (xid, ses, smb_buffer, smb_buffer_response, &length, -CIFS_STD_OP ); +0 );/* above now done in SendReceive */ if ((rc ==0 ) && (tcon !=NULL )) {
fs/cifs/file.c +7 -37
119 lines not on this page · raw message
fs/cifs/sess.c +1 -1
@@ -890,7 +890,7 @@ ssetup_ntlmssp_authenticate:BCC_LE (smb_buf) =cpu_to_le16 (count); rc =SendReceive2 (xid, ses, iov,3 /* num_iovecs */ , &resp_buf_type, -CIFS_STD_OP /* not long */ |CIFS_LOG_ERROR ); +CIFS_LOG_ERROR );/* SMB request buf freed in SendReceive2 */ pSMB = (SESSION_SETUP_ANDX *)iov[0 ].iov_base;
fs/cifs/transport.c +1 -1
@@ -790,7 +790,7 @@ send_lock_cancel(const unsigned int xid, struct cifsTconInfo *tcon, pSMB->hdr.Mid =GetNextMid (ses->server);return SendReceive (xid, ses, in_buf, out_buf, - &bytes_returned,CIFS_STD_OP ); + &bytes_returned,0 ); }int
Signature · 1 line
1.7.3.2
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)
Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
A message that is not in the archive (sent off-list or not yet received): AANLkTik67ugsfTapXHoRu-c7Rpra-wofdku2xioOQi6--JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20101127113532.62d24bcf-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.
There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
A message that is not in the archive (sent off-list or not yet received): AANLkTinZzxhcGR9m67+neem85SHU1YA+eOBb3NE9PO4Y-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 105 lines · On Sat, 27 Nov 2010 11:38:26 -0600 Shirish Pargaonkar…
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>
Quoted 6 lines
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I don't think so. Just because we're sending an echo doesn't mean that we should spam the logs. What exactly is an "expected time" anyway? That value is highly variable, dependent on many of different factors and often impossible to predict.
OTOH, if the server isn't responding to echoes, then I think we probably are justified in doing a printk. CIFS Servers are expected to respond to an echo in a timely fashion even if they are busy.
Quoted 11 lines
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
I'm afraid I have to disagree here. "timing out" certain commands is what has given us the problems we have today. CIFS currently falls flat on its face when a server is slow to respond to certain commands.
I'll submit that the best behavior to aim for is the one I'm proposing. The client ought to wait indefinitely for responses from the server, and only give up if the server isn't responding to echoes. The wait however should be interruptible by fatal signals so that users can decide on a per-process basis whether to give up on the server.
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20101127201248.054ec58d-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org
Quoted 136 lines · Jeff Layton
Quoted 105 lines · On Sat, 27 Nov 2010 11:38:26 -0600 Shirish Pargaonkar…
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>Quoted 6 lines
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I don't think so. Just because we're sending an echo doesn't mean that we should spam the logs. What exactly is an "expected time" anyway? That value is highly variable, dependent on many of different factors and often impossible to predict.
OTOH, if the server isn't responding to echoes, then I think we probably are justified in doing a printk. CIFS Servers are expected to respond to an echo in a timely fashion even if they are busy.
Quoted 11 lines
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
Quoted 7 lines
I'm afraid I have to disagree here. "timing out" certain commands is what has given us the problems we have today. CIFS currently falls flat on its face when a server is slow to respond to certain command s.
I'll submit that the best behavior to aim for is the one I'm proposing. The client ought to wait indefinitely for responses from the server, and only give up if the server isn't responding to echoes. The wait
Is it possible that server is so slow that client can't even send any more commands such as smb echo? This could be a temporary glitch in the life of a server!
I am really concerned about a scenario where for example, smbds on a Samba server are killed. Would an user be able to unmount a share? Because there will be no response from the server, reconnects are not going help, and user on client machine can only kill a command.
Quoted 6 lines
however should be interruptible by fatal signals so that users can decide on a per-process basis whether to give up on the server.
--
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): AANLkTin_8UKz3GFzMMRyGX8AAEEw16H2Y2aj3RGVvXXQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 141 lines · On Sun, 28 Nov 2010 06:17:54 -0600 Shirish Pargaonkar…
Quoted 140 lines · Jeff Layton
Quoted 105 lines · On Sat, 27 Nov 2010 11:38:26 -0600 Shirish Pargaonkar…
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>Quoted 6 lines
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I don't think so. Just because we're sending an echo doesn't mean that we should spam the logs. What exactly is an "expected time" anyway? That value is highly variable, dependent on many of different factors and often impossible to predict.
OTOH, if the server isn't responding to echoes, then I think we probably are justified in doing a printk. CIFS Servers are expected to respond to an echo in a timely fashion even if they are busy.
Quoted 11 lines
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
I'm afraid I have to disagree here. "timing out" certain commands is what has given us the problems we have today. CIFS currently falls flat on its face when a server is slow to respond to certain command s.
Quoted 8 lines
Quoted 3 lines
I'll submit that the best behavior to aim for is the one I'm proposing. The client ought to wait indefinitely for responses from the server, and only give up if the server isn't responding to echoes. The wait
Is it possible that server is so slow that client can't even send any more commands such as smb echo? This could be a temporary glitch in the life of a server!
I think that would be indicative of a broken server. We need some metric to know whether the socket is dead and that it needs to be reconnected. If not SMB echoes, then what do you suggest?
Quoted 5 lines
I am really concerned about a scenario where for example, smbds on a Samba server are killed. Would an user be able to unmount a share? Because there will be no response from the server, reconnects are not going help, and user on client machine can only kill a command.
What's the exact scenario that you're concerned with here?
For the purposes of argument, let's assume that the mount is idle -- no dirty data or open files. We issue a umount on the mountpoint and cifs_umount is eventually called. The tcon is put and a CIFSSMBTDis is issued. The smbds on the server are down however, and the it isn't responding to echoes.
The client will send the TDis and wait for the reply, no replies come and we start sending echoes. The echoes won't be responded to however and eventually cifs_reconnect is called. The mid is marked MID_RETRY_NEEDED and the umount process is woken up. SendReceiveNoRsp returns -EAGAIN and CIFSSMBTDis returns 0 and the umount proceeds.
So, it may take a little while but eventually the umount would proceed.
If you're really concerned about that situation however, you could make CIFSSMBTDis use an async call (support for that is added in this patchset). That would make the umount return even more quickly than it does today.
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20101128074309.12ca4e49-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org
Quoted 136 lines · Jeff Layton
Quoted 133 lines · On Sun, 28 Nov 2010 06:17:54 -0600 Shirish Pargaonkar…
Quoted 132 lines · Jeff Layton
Quoted 105 lines · On Sat, 27 Nov 2010 11:38:26 -0600 Shirish Pargaonkar…
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>Quoted 6 lines
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I don't think so. Just because we're sending an echo doesn't mean that we should spam the logs. What exactly is an "expected time" anyway? That value is highly variable, dependent on many of different factors and often impossible to predict.
OTOH, if the server isn't responding to echoes, then I think we probably are justified in doing a printk. CIFS Servers are expected to respond to an echo in a timely fashion even if they are busy.
Quoted 8 lines
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
I'm afraid I have to disagree here. "timing out" certain commands is what has given us the problems we have today. CIFS currently falls flat on its face when a server is slow to respond to certain command s.
I think these commands such as negotiate, session setup, tree connect, logoff, non-blocking posix lock, and dfs referral are always soft.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I'll submit that the best behavior to aim for is the one I'm proposing. The client ought to wait indefinitely for responses from the server, and only give up if the server isn't responding to echoes. The wait
Quoted 8 lines
Quoted 4 lines
Is it possible that server is so slow that client can't even send any more commands such as smb echo? This could be a temporary glitch in the life of a server!
I think that would be indicative of a broken server. We need some metric to know whether the socket is dead and that it needs to be reconnected. If not SMB echoes, then what do you suggest?
Jeff, I have seen servers with these kind of delays. If left to itself i.e. not reconnect socket etc. and just waited for response to arrive, tests passed on the client against this server passed (i.e. ran the testsuite without any errors such as data corruption) 72 hour runs.
CIFS slow rsp: cmd 50 subcmd: 5 mid 10658 A: 0xd9 S: 0xd9 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 22908 A: 0x77 S: 0x77 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 20553 A: 0x74 S: 0x74 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 23332 A: 0x67 S: 0x67 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 28992 A: 0x7f S: 0x7f R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 29547 A: 0x326 S: 0x326 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 31037 A: 0x3ad S: 0x3ad R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 31419 A: 0x340 S: 0x340 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 5 mid 31420 A: 0x17e S: 0x17e R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 32275 A: 0x15f S: 0x15f R: 0x0 CIFS slow rsp: cmd 50 subcmd: 5 mid 29489 A: 0xd8 S: 0xd8 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 32772 A: 0x5ba S: 0x5ba R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 30114 A: 0xe8 S: 0xe8 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 37722 A: 0x67 S: 0x67 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 56677 A: 0xef S: 0xef R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 49279 A: 0x7c S: 0x7c R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 50051 A: 0x68 S: 0x68 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 8846 A: 0x6e S: 0x6e R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 9075 A: 0xac S: 0xac R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 9620 A: 0xa8 S: 0xa8 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 22769 A: 0x17b S: 0x17b R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 25783 A: 0xae S: 0xae R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 25973 A: 0x556 S: 0x556 R: 0x0
CIFS slow rsp: cmd 47 subcmd: 0 mid 31016 A: 0x190 S: 0x190 R: 0x0 CIFS slow rsp: cmd 47 subcmd: 0 mid 34380 A: 0x3e0 S: 0x3e0 R: 0x0 CIFS slow rsp: cmd 47 subcmd: 0 mid 34381 A: 0x3e1 S: 0x3e1 R: 0x0 CIFS slow rsp: cmd 46 subcmd: 0 mid 52365 A: 0x56e S: 0x56e R: 0x0 CIFS slow rsp: cmd 47 subcmd: 0 mid 52366 A: 0x40c S: 0x40c R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 34432 A: 0x4e3 S: 0x4e3 R: 0x0 CIFS slow rsp: cmd 47 subcmd: 0 mid 34433 A: 0x4e3 S: 0x4e3 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 54464 A: 0x4a5 S: 0x4a5 R: 0x0
# dmesg | grep "cmd 50" CIFS slow rsp: cmd 50 subcmd: 6 mid 28610 A: 0x209 S: 0x209 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 31581 A: 0x4c2 S: 0x4c2 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 57722 A: 0x446 S: 0x446 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 37736 A: 0xa8 S: 0xa8 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 43156 A: 0x19a S: 0x19a R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 20926 A: 0x1de S: 0x1de R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 43164 A: 0x2e5 S: 0x2e5 R: 0x0
Quoted 15 lines
Quoted 6 lines
I am really concerned about a scenario where for example, smbds on a Samba server are killed. Would an user be able to unmount a share? Because there will be no response from the server, reconnects are not going help, and user on client machine can only kill a command.
What's the exact scenario that you're concerned with here?
For the purposes of argument, let's assume that the mount is idle -- no dirty data or open files. We issue a umount on the mountpoint and
cifs_umountis eventually called. The tcon is put and a CIFSSMBTDis is issued. The smbds on the server are down however, and the it isn't responding to echoes.
Quoted 3 lines
The client will send the TDis and wait for the reply, no replies come and we start sending echoes. The echoes won't be responded to however and eventually
cifs_reconnectis called.
Jeff, who and what will cause cifs_reconnect? Everything hinges on cifs_reconnect being called right?
Would not client wait for a response to SMB Echo command without any timeouts?
Quoted 14 lines
The mid is marked
MID_RETRY_NEEDEDand the umount process is woken up. SendReceiveNoRsp returns-EAGAINand CIFSSMBTDis returns 0 and the umount proceeds.So, it may take a little while but eventually the umount would proceed.
If you're really concerned about that situation however, you could make CIFSSMBTDis use an async call (support for that is added in this patchset). That would make the umount return even more quickly than it does today.
--
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): AANLkTi=Z6spe6FU3Q62fmXbj-WuiVkNHs7eyouXEs7nJ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 141 lines · On Sun, 28 Nov 2010 18:56:27 -0600 Shirish Pargaonkar…
Quoted 140 lines · Jeff Layton
Quoted 137 lines · On Sun, 28 Nov 2010 06:17:54 -0600 Shirish Pargaonkar…
Quoted 136 lines · Jeff Layton
Quoted 105 lines · On Sat, 27 Nov 2010 11:38:26 -0600 Shirish Pargaonkar…
Quoted 97 lines · Jeff Layton
Quoted 82 lines · On Sat, 27 Nov 2010 09:08:01 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
General question, during all this process and wait, does user know in any way (ideally without setting any debug flags/options) that a server is unresponsive, perhaps everytime an echo does not come back within expected time, and then when it is back to normal i.e. responding?
There's no feedback of that sort in the current patches, but something like that could be added...
Note that the behavior here isn't quite analogous to NFS/RPC since we don't time out requests. So, we can't really base this on a "major timeout" like we do there.
Quoted 6 lines
What sort of behavior do you think would make the most sense here? How often and under what conditions should we do a printk?
--
Jeff Layton <[email protected]>Quoted 6 lines
I am not sure. But user has to have some indication if a command does not return in expected time. I think whenever cifs gets ready to send SMB Echo, command would perhaps be a time to log that server is not responding. If that SMB Echo response comes back within expected time, server is back to normal timing and that can be logged.
I don't think so. Just because we're sending an echo doesn't mean that we should spam the logs. What exactly is an "expected time" anyway? That value is highly variable, dependent on many of different factors and often impossible to predict.
OTOH, if the server isn't responding to echoes, then I think we probably are justified in doing a printk. CIFS Servers are expected to respond to an echo in a timely fashion even if they are busy.
Quoted 11 lines
I am not sure whether what gets logged should be either at debug level or error level, error level is preferrable at it would be visible to the user without
setting any debug flags.
But some test scenarios may want to run clean and if "server not responding", "server responding" type of messages are logged, it may be flagged or deemed annoying.There are certain commands that can't be "hard mount" kind, they have to be returned within certain time e.g. if we are unmounting a share and server is not responding.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I'm afraid I have to disagree here. "timing out" certain commands is what has given us the problems we have today. CIFS currently falls flat on its face when a server is slow to respond to certain command s.
I think these commands such as negotiate, session setup, tree connect, logoff, non-blocking posix lock, and dfs referral are always soft.
I think you need to define what you mean by soft here. IIUC, "soft" and "hard" are governed by mount options. Why should that only affect some sorts of calls and not others.
Quoted 52 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I'll submit that the best behavior to aim for is the one I'm proposing. The client ought to wait indefinitely for responses from the server, and only give up if the server isn't responding to echoes. The wait
Quoted 8 lines
Quoted 4 lines
Is it possible that server is so slow that client can't even send any more commands such as smb echo? This could be a temporary glitch in the life of a server!
I think that would be indicative of a broken server. We need some metric to know whether the socket is dead and that it needs to be reconnected. If not SMB echoes, then what do you suggest?
Jeff, I have seen servers with these kind of delays. If left to itself i.e. not reconnect socket etc. and just waited for response to arrive, tests passed on the client against this server passed (i.e. ran the testsuite without any errors such as data corruption) 72 hour runs.
CIFS slow rsp: cmd 50 subcmd: 5 mid 10658 A:
0xd9S:0xd9R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 22908 A:0x77S:0x77R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 20553 A:0x74S:0x74R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 23332 A:0x67S:0x67R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 28992 A:0x7fS:0x7fR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 29547 A:0x326S:0x326R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 31037 A:0x3adS:0x3adR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 31419 A:0x340S:0x340R:0x0CIFS slow rsp: cmd 50 subcmd: 5 mid 31420 A:0x17eS:0x17eR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 32275 A:0x15fS:0x15fR:0x0CIFS slow rsp: cmd 50 subcmd: 5 mid 29489 A:0xd8S:0xd8R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 32772 A:0x5baS:0x5baR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 30114 A:0xe8S:0xe8R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 37722 A:0x67S:0x67R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 56677 A:0xefS:0xefR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 49279 A:0x7cS:0x7cR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 50051 A:0x68S:0x68R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 8846 A:0x6eS:0x6eR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 9075 A:0xacS:0xacR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 9620 A:0xa8S:0xa8R:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 22769 A:0x17bS:0x17bR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 25783 A:0xaeS:0xaeR:0x0CIFS slow rsp: cmd 50 subcmd: 6 mid 25973 A:0x556S:0x556R:0x0CIFS slow rsp: cmd 47 subcmd: 0 mid 31016 A:
0x190S:0x190R:0x0CIFS slow rsp: cmd 47 subcmd: 0 mid 34380 A:0x3e0S:0x3e0R:0x0CIFS slow rsp: cmd 47 subcmd: 0 mid 34381 A:0x3e1S:0x3e1R:0x0CIFS slow rsp: cmd 46 subcmd: 0 mid 52365 A:0x56eS:0x56eR:0x0CIFS slow rsp: cmd 47 subcmd: 0 mid 52366 A:0x40cS:0x40cR:0x0CIFS slow rsp: cmd 50 subcmd: 0 mid 34432 A:0x4e3S:0x4e3R:0x0CIFS slow rsp: cmd 47 subcmd: 0 mid 34433 A:0x4e3S:0x4e3R:0x0CIFS slow rsp: cmd 50 subcmd: 0 mid 54464 A:0x4a5S:0x4a5R:0x0
Quoted 9 lines
# dmesg | grep "cmd 50" CIFS slow rsp: cmd 50 subcmd: 6 mid 28610 A: 0x209 S: 0x209 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 31581 A: 0x4c2 S: 0x4c2 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 57722 A: 0x446 S: 0x446 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 37736 A: 0xa8 S: 0xa8 R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 43156 A: 0x19a S: 0x19a R: 0x0 CIFS slow rsp: cmd 50 subcmd: 0 mid 20926 A: 0x1de S: 0x1de R: 0x0 CIFS slow rsp: cmd 50 subcmd: 6 mid 43164 A: 0x2e5 S: 0x2e5 R: 0x0
Which is exactly what I'm arguing. We shouldn't reconnect the socket just because the server didn't respond within an arbitrary timeout.
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines
I am really concerned about a scenario where for example, smbds on a Samba server are killed. Would an user be able to unmount a share? Because there will be no response from the server, reconnects are not going help, and user on client machine can only kill a command.
What's the exact scenario that you're concerned with here?
For the purposes of argument, let's assume that the mount is idle -- no dirty data or open files. We issue a umount on the mountpoint and
cifs_umountis eventually called. The tcon is put and a CIFSSMBTDis is issued. The smbds on the server are down however, and the it isn't responding to echoes.
Quoted 6 lines
Quoted 3 lines
The client will send the TDis and wait for the reply, no replies come and we start sending echoes. The echoes won't be responded to however and eventually
cifs_reconnectis called.Jeff, who and what will cause
cifs_reconnect? Everything hinges oncifs_reconnectbeing called right?
This stanza in patch #12:
+
+ if (time_after(jiffies,
+ server->lstrp + UNRESPONSIVE_SERVER_TIMEOUT)) {
+ cFYI(1, "Reconnecting unresponsive server %s",
+ server->hostname);
+ cifs_reconnect(server);
+ csocket = server->ssocket;
+ wake_up(&server->response_q);
+ continue;
+ }
Quoted 3 lines
Would not client wait for a response to SMB Echo command without any timeouts?
I'm not sure I understand this question. Essentially, the patchset has the client issue an echo request when a reply hasn't come in within a certain amount of time. If no reply comes in after several smb echoes, then we'll reconnect the socket. It's really not that complex, which is in sharp contrast to the current code which has a very convoluted chunk of code that tries to set different timeouts.
I'm having a little difficulty understanding the behavior that you think we ought to be shooting for. Perhaps you could outline what you think would be correct behavior for the transport layer?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)
Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
I am not sure if smb echo is a good indicator. What if server is so overloaded that smb echo may not get a response within its timeout period? Does not mean server is dead, it is just working its way up to respond to all the requests and if given enough time, will eventually respond, without any need to reconnect that connection (because smb echo was not echo'ed back in time).
A message that is not in the archive (sent off-list or not yet received): AANLkTinoWsZZfM+BtSGnQEjmGAmH9UdMu=ut_k_F0KgO-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 82 lines · On Sat, 27 Nov 2010 16:59:57 -0600 Shirish Pargaonkar…
Quoted 72 lines · Jeff Layton
This patchset is intended to fix the unreliable behavior in CIFS in the face of a server that's taking a long time to process requests.
In general, the current code sets a timeout for all requests that are sent on the wire. If the server doesn't respond to the request within that timeout, the client performs a reconnect and retries the request.
This is dangerous and wasteful behavior for the client. Much of the state of a CIFS mount is bound to the socket connection. Break the socket connection and state is lost.
I believe this the root cause of some data corruption issues that have been reported to me. We had a partner report that when they copied a large file to a CIFS server and then compare the result to the original, there is sometimes a mismatch. The problem is highly correlated to messages in the ring buffer that indicate that the client reconnected the socket during the test run.
Another problem that I can reliably reproduce -- I have win2k8 installed as a VM guest. When I run connectathon tests to that server, it frequently fails on the test that writes 4GB past the EOF. The storage on this server is slow, and it can take longer than 180s for it to zero-fill the output file.
The intent of this patchset is to fundamentally change when the client decides to reconnect the socket. Instead of the old behavior, this patchset makes the client wait indefinitely for a response. Rather than waiting in
TASK_UNINTERRUPTIBLEsleep however, the client waits inTASK_KILLABLEsleep so that fatal signals will end the sleep and return-ERESTARTSYSto the caller.In order to determine whether the server is completely dead or just taking a long time to process requests, this patchset has the client do an asynchronous SMB echo request every 60s when the client hasn't gotten a reponse. If the server doesn't respond after 5 mins, the client will attempt to reconnect the socket.
With this patchset, I can reliably run the connectathon tests against my slow server. Preliminary results using the proprietary test that was seeing data corruption have also been promising.
I'd like to see this set considered for inclusion into 2.6.38.
Comments and suggestions welcome.
Jeff Layton (14):
cifs: move mid result processing into common function
cifs: wait indefinitely for responses
cifs: don't reconnect server when we don't get a response
cifs: clean uphandle_mid_response
cifs: allow for different handling of received response
cifs: don't fail writepages on-EAGAINerrors
cifs: handle cancelled requests better
cifs: makewait_for_free_requesttake aTCP_Server_Infopointer
cifs: addcifs_call_async
cifs: add ability to send an echo request
cifs: set up recurring workqueue job to do SMB echo requests
cifs: reconnect unresponsive servers
cifs: make hard mounts the default
cifs: remove code for setting timeouts on requestsfs/cifs/cifs_debug.c | 8 +-
fs/cifs/cifsglob.h | 19 ++--
fs/cifs/cifspdu.h | 15 +++
fs/cifs/cifsproto.h | 7 +
fs/cifs/cifssmb.c | 55 ++++++++-
fs/cifs/connect.c | 128 +++++++++++++++----
fs/cifs/file.c | 48 ++------
fs/cifs/sess.c | 2 +-
fs/cifs/transport.c | 344 ++++++++++++++++++++++----------------------------9 files changed, 355 insertions(+), 271 deletions(-)Quoted 8 lines
--
1.7.3.2--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
I am not sure if smb echo is a good indicator. What if server is so overloaded that smb echo may not get a response within its timeout period? Does not mean server is dead, it is just working its way up to respond to all the requests and if given enough time, will eventually respond, without any need to reconnect that connection (because smb echo was not echo'ed back in time).
SMB echo is the same metric that windows uses to tell whether the connection to the server is still working.
See footnote #186 in MS-CIFS "product behavior" appendix.
Signature · 1 line
Jeff Layton <[email protected]>