cifs: Miscellaneous fixes and a trace point
David Howells
maintainercover cifs: Miscellaneous fixes and a trace point
Hi Steve,
Here are four patches for cifs:
(1) Fix re-repick of a server upon subrequest retry.
(2) Fix some error code setting that got accidentally removed.
(3) Fix the handling of the zero_point after a DIO write. It always needs to be bumped past the end of the DIO write.
(4) Add a tracepoint and some debugging to keep track of when we've ended
the ->in_flight contribution from an operation.
I've pushed the patches here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=cifs-fixes
David
David Howells (4):
cifs: Fix server re-repick on subrequest retry
cifs: Fix missing error code set
cifs: Fix setting of zero_point after DIO write
cifs: Add a tracepoint to track credits involved in R/W requests
fs/smb/client/cifsglob.h | 17 +++++++----- fs/smb/client/file.c | 47 +++++++++++++++++++++++++++++---- fs/smb/client/smb1ops.c | 2 +- fs/smb/client/smb2ops.c | 42 ++++++++++++++++++++++++++---- fs/smb/client/smb2pdu.c | 43 +++++++++++++++++++++++++----- fs/smb/client/trace.h | 55 ++++++++++++++++++++++++++++++++++++++- fs/smb/client/transport.c | 8 +++--- 7 files changed, 184 insertions(+), 30 deletions(-)
David Howells
maintainerPATCH 1/4 cifs: Fix server re-repick on subrequest retry
When a subrequest is marked for needing retry, netfs will call cifs_prepare_write() which will make cifs repick the server for the op before renegotiating credits; it then calls cifs_issue_write() which invokes smb2_async_writev() - which re-repicks the server.
If a different server is then selected, this causes the increment of server->in_flight to happen against one record and the decrement to happen against another, leading to misaccounting.
Fix this by just removing the repick code in smb2_async_writev(). As this is only called from netfslib-driven code, cifs_prepare_write() should always have been called first, and so server should never be NULL and the preparatory step is repeated in the event that we do a retry.
The problem manifests as a warning looking something like:
WARNING: CPU: 4 PID: 72896 at fs/smb/client/smb2ops.c:97 smb2_add_credits+0x3f0/0x9e0 [cifs] ... RIP: 0010:smb2_add_credits+0x3f0/0x9e0 [cifs] ... smb2_writev_callback+0x334/0x560 [cifs] cifs_demultiplex_thread+0x77a/0x11b0 [cifs] kthread+0x187/0x1d0 ret_from_fork+0x34/0x60 ret_from_fork_asm+0x1a/0x30
Which may be triggered by a number of different xfstests running against an Azure server in multichannel mode. generic/249 seems the most repeatable, but generic/215, generic/249 and generic/308 may also show it.
--- fs/smb/client/smb2pdu.c | 3 --- 1 file changed, 3 deletions(-)
fs/smb/client/smb2pdu.c +0 -3
@@ -4859,9 +4859,6 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)struct cifs_io_parms *io_parms =NULL ;int credit_request; -if (!wdata->server ||test_bit (NETFS_SREQ_RETRYING , &wdata->subreq.flags)) - server = wdata->server =cifs_pick_channel (tcon->ses); -/* * in future we may get cifs_io_parms passed in from the caller, * but for now we construct it here...
Re: [PATCH 1/4] cifs: Fix server re-repick on subrequest retry
Quoted 26 lines · David Howells
When a subrequest is marked for needing retry, netfs will call
cifs_prepare_write()which will make cifs repick the server for the op before renegotiating credits; it then callscifs_issue_write()which invokessmb2_async_writev()- which re-repicks the server.If a different server is then selected, this causes the increment of
server->in_flightto happen against one record and the decrement to happen against another, leading to misaccounting.Fix this by just removing the repick code in
smb2_async_writev(). As this is only called from netfslib-driven code,cifs_prepare_write()should always have been called first, and so server should never be NULL and the preparatory step is repeated in the event that we do a retry.The problem manifests as a warning looking something like:
WARNING: CPU: 4 PID: 72896 at fs/smb/client/smb2ops.c:97 smb2_add_credits+0x3f0/0x9e0 [cifs] ... RIP: 0010:smb2_add_credits+0x3f0/0x9e0 [cifs] ... smb2_writev_callback+0x334/0x560 [cifs] cifs_demultiplex_thread+0x77a/0x11b0 [cifs] kthread+0x187/0x1d0 ret_from_fork+0x34/0x60 ret_from_fork_asm+0x1a/0x30
Quoted 3 lines
Which may be triggered by a number of different xfstests running against an Azure server in multichannel mode. generic/249 seems the most repeatable, but generic/215, generic/249 and generic/308 may also show it.
Nice fix, and good explanation. So, is this the negative-credits issue we've been looking to fix? Or just one instance?
I'm very curious why it manifested when testing with Azure Files. Do connection errors disappear with the fix, or do they still occur but are now recoverable?
Feel free to add...
Tom.
Quoted 30 lines
Fixes 3ee1a1fc3981 ("cifs: Cut over to using netfslib")Reported-by Steve French <[email protected]>Signed-off-by David Howells <[email protected]>Reviewed-by Paulo Alcantara (Red Hat) <[email protected]>cc Jeff Layton <[email protected]>cc Aurelien Aptel <[email protected]>--- fs/smb/client/smb2pdu.c | 3 --- 1 file changed, 3 deletions(-)fs/smb/client/smb2pdu.c +0
-3@@ -4859,9 +4859,6 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)struct cifs_io_parms *io_parms =NULL ;int credit_request; -if (!wdata->server ||test_bit (NETFS_SREQ_RETRYING , &wdata->subreq.flags)) - server = wdata->server =cifs_pick_channel (tcon->ses); -/* * in future we may get cifs_io_parms passed in from the caller, * but for now we construct it here...
Re: [PATCH 1/4] cifs: Fix server re-repick on subrequest retry
Quoted 28 lines · Tom Talpey
Quoted 26 lines · David Howells
When a subrequest is marked for needing retry, netfs will call
cifs_prepare_write()which will make cifs repick the server for the op before renegotiating credits; it then callscifs_issue_write()which invokessmb2_async_writev()- which re-repicks the server.If a different server is then selected, this causes the increment of
server->in_flightto happen against one record and the decrement to happen against another, leading to misaccounting.Fix this by just removing the repick code in
smb2_async_writev(). As this is only called from netfslib-driven code,cifs_prepare_write()should always have been called first, and so server should never be NULL and the preparatory step is repeated in the event that we do a retry.The problem manifests as a warning looking something like:
WARNING: CPU: 4 PID: 72896 at fs/smb/client/smb2ops.c:97 smb2_add_credits+0x3f0/0x9e0 [cifs] ... RIP: 0010:smb2_add_credits+0x3f0/0x9e0 [cifs] ... smb2_writev_callback+0x334/0x560 [cifs] cifs_demultiplex_thread+0x77a/0x11b0 [cifs] kthread+0x187/0x1d0 ret_from_fork+0x34/0x60 ret_from_fork_asm+0x1a/0x30
Quoted 6 lines
Quoted 3 lines
Which may be triggered by a number of different xfstests running against an Azure server in multichannel mode. generic/249 seems the most repeatable, but generic/215, generic/249 and generic/308 may also show it.
Nice fix, and good explanation. So, is this the negative-credits issue we've been looking to fix? Or just one instance?
Yes it fixes the only recent crediting issues that I had been seeing.
Quoted 3 lines
Feel free to add...
Acked-by Tom Talpey <[email protected]>
Done
David Howells
maintainerPATCH 2/4 cifs: Fix missing error code set
In reply to David HowellsIn cifs_strict_readv(), the default rc (-EACCES) is accidentally cleared by a successful return from netfs_start_io_direct(), such that if cifs_find_lock_conflict() fails, we don't return an error.
Fix this by resetting the default error code.
--- fs/smb/client/file.c | 2 ++ 1 file changed, 2 insertions(+)
fs/smb/client/file.c +2 -0
@@ -2877,6 +2877,7 @@ cifs_strict_readv(struct kiocb *iocb, struct iov_iter *to) rc =netfs_start_io_direct (inode);if (rc <0 ) goto out; + rc = -EACCES ;down_read (&cinode->lock_sem); if (!cifs_find_lock_conflict( cfile, iocb->ki_pos, iov_iter_count(to),
@@ -2889,6 +2890,7 @@ cifs_strict_readv(struct kiocb *iocb, struct iov_iter *to) rc =netfs_start_io_read (inode);if (rc <0 ) goto out; + rc = -EACCES ;down_read (&cinode->lock_sem); if (!cifs_find_lock_conflict( cfile, iocb->ki_pos, iov_iter_count(to),
David Howells
maintainerPATCH 3/4 cifs: Fix setting of zero_point after DIO write
In reply to David HowellsAt the moment, at the end of a DIO write, cifs calls netfs_resize_file() to adjust the size of the file if it needs it. This will reduce the zero_point (the point above which we assume a read will just return zeros) if it's more than the new i_size, but won't increase it.
With DIO writes, however, we definitely want to increase it as we have clobbered the local pagecache and then written some data that's not available locally.
Fix cifs to make the zero_point above the end of a DIO or unbuffered write.
This fixes corruption seen occasionally with the generic/708 xfs-test. In that case, the read-back of some of the written data is being short-circuited and replaced with zeroes.
--- fs/smb/client/file.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
fs/smb/client/file.c +10 -5
@@ -2358,13 +2358,18 @@ void cifs_write_subrequest_terminated(struct cifs_io_subrequest *wdata, ssize_t bool was_async) {struct netfs_io_request *wreq = wdata->rreq; -loff_t new_server_eof; +struct netfs_inode *ictx =netfs_inode (wreq->inode); +loff_t wrend;if (result >0 ) { - new_server_eof = wdata->subreq.start + wdata->subreq.transferred + result; - -if (new_server_eof >netfs_inode (wreq->inode)->remote_i_size) -netfs_resize_file (netfs_inode (wreq->inode), new_server_eof, true); + wrend = wdata->subreq.start + wdata->subreq.transferred + result; + +if (wrend > ictx->zero_point && + (wdata->rreq->origin ==NETFS_UNBUFFERED_WRITE || + wdata->rreq->origin ==NETFS_DIO_WRITE )) + ictx->zero_point = wrend; +if (wrend > ictx->remote_i_size) +netfs_resize_file (ictx, wrend, true); }netfs_write_subrequest_terminated (&wdata->subreq, result, was_async);
David Howells
maintainerPATCH 4/4 cifs: Add a tracepoint to track credits involved in R/W requests
In reply to David HowellsAdd a tracepoint to track the credit changes and server in_flight value involved in the lifetime of a R/W request, logging it against the request/subreq debugging ID. This requires the debugging IDs to be recorded in the cifs_credits struct.
The tracepoint can be enabled with:
echo 1 >/sys/kernel/debug/tracing/events/cifs/smb3_rw_credits/enable
Also add a three-state flag to struct cifs_credits to note if we're interested in determining when the in_flight contribution ends and, if so, to track whether we've decremented the contribution yet.
--- fs/smb/client/cifsglob.h | 17 +++++++----- fs/smb/client/file.c | 32 ++++++++++++++++++++++- fs/smb/client/smb1ops.c | 2 +- fs/smb/client/smb2ops.c | 42 ++++++++++++++++++++++++++---- fs/smb/client/smb2pdu.c | 40 +++++++++++++++++++++++++--- fs/smb/client/trace.h | 55 ++++++++++++++++++++++++++++++++++++++- fs/smb/client/transport.c | 8 +++--- 7 files changed, 173 insertions(+), 23 deletions(-)
fs/smb/client/cifsglob.h +10 -7
@@ -290,7 +290,7 @@ struct smb_version_operations {int (*check_receive)(struct mid_q_entry *,struct TCP_Server_Info *,bool );void (*add_credits)(struct TCP_Server_Info *server, -const struct cifs_credits *credits, +struct cifs_credits *credits,const int optype);void (*set_credits)(struct TCP_Server_Info *,const int );int * (*get_credits_field)(struct TCP_Server_Info *,const int );
@@ -550,8 +550,8 @@ struct smb_version_operations { size_t *, struct cifs_credits *);/* adjust previously taken mtu credits to request size */ int (*adjust_credits)(struct TCP_Server_Info *server, -struct cifs_credits *credits, -const unsigned int payload_size); +struct cifs_io_subrequest *subreq, +unsigned int /*enum smb3_rw_credits_trace*/ trace);/* check if we need to issue closedir */ bool (*dir_needs_close)(struct cifsFileInfo *); long (*fallocate)(struct file *, struct cifs_tcon *, int, loff_t,
@@ -848,6 +848,9 @@ static inline void cifs_server_unlock(struct TCP_Server_Info *server)struct cifs_credits {unsigned int value;unsigned int instance; +unsigned int in_flight_check; +unsigned int rreq_debug_id; +unsigned int rreq_debug_index; }; static inline unsigned int
@@ -873,7 +876,7 @@ has_credits(struct TCP_Server_Info *server, int *credits, int num_credits) }static inline void -add_credits (struct TCP_Server_Info *server,const struct cifs_credits *credits, +add_credits (struct TCP_Server_Info *server,struct cifs_credits *credits,const int optype) { server->ops->add_credits (server, credits, optype);
@@ -897,11 +900,11 @@ set_credits(struct TCP_Server_Info *server, const int val) }static inline int -adjust_credits (struct TCP_Server_Info *server,struct cifs_credits *credits, -const unsigned int payload_size) +adjust_credits (struct TCP_Server_Info *server,struct cifs_io_subrequest *subreq, +unsigned int /* enum smb3_rw_credits_trace */ trace) {return server->ops->adjust_credits ? - server->ops->adjust_credits (server, credits, payload_size) :0 ; + server->ops->adjust_credits (server, subreq, trace) :0 ; } static inline __le64
fs/smb/client/file.c +31 -1
@@ -80,6 +80,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)return netfs_prepare_write_failed (subreq); } + wdata->credits.rreq_debug_id = subreq->rreq->debug_id; + wdata->credits.rreq_debug_index = subreq->debug_index; + wdata->credits.in_flight_check =1 ; +trace_smb3_rw_credits (wdata->rreq->debug_id, + wdata->subreq.debug_index, + wdata->credits.value, + server->credits, server->in_flight, + wdata->credits.value, + cifs_trace_rw_credits_write_prepare); +#ifdef CONFIG_CIFS_SMB_DIRECT if (server->smbd_conn) subreq->max_nr_segs = server->smbd_conn->max_frmr_depth;
@@ -101,7 +111,7 @@ static void cifs_issue_write(struct netfs_io_subrequest *subreq) goto fail; } - rc =adjust_credits (wdata->server, &wdata->credits, wdata->subreq.len); + rc =adjust_credits (wdata->server, wdata, cifs_trace_rw_credits_issue_write_adjust);if (rc) goto fail;
@@ -158,7 +168,18 @@ static bool cifs_clamp_length(struct netfs_io_subrequest *subreq)return false; } + rdata->credits.in_flight_check =1 ; + rdata->credits.rreq_debug_id = rreq->debug_id; + rdata->credits.rreq_debug_index = subreq->debug_index; + +trace_smb3_rw_credits (rdata->rreq->debug_id, + rdata->subreq.debug_index, + rdata->credits.value, + server->credits, server->in_flight,0 , + cifs_trace_rw_credits_read_submit); + subreq->len =min_t (size_t, subreq->len, rsize); +#ifdef CONFIG_CIFS_SMB_DIRECT if (server->smbd_conn) subreq->max_nr_segs = server->smbd_conn->max_frmr_depth;
@@ -289,6 +310,15 @@ static void cifs_free_subrequest(struct netfs_io_subrequest *subreq)#endif } +if (rdata->credits.value !=0 ) +trace_smb3_rw_credits (rdata->rreq->debug_id, + rdata->subreq.debug_index, + rdata->credits.value, + rdata->server ? rdata->server->credits :0 , + rdata->server ? rdata->server->in_flight :0 , + -rdata->credits.value, + cifs_trace_rw_credits_free_subreq); +add_credits_and_wake_if (rdata->server, &rdata->credits,0 );if (rdata->have_xid)free_xid (rdata->xid);
fs/smb/client/smb1ops.c +1 -1
@@ -108,7 +108,7 @@ cifs_find_mid(struct TCP_Server_Info *server, char *buffer)static void cifs_add_credits (struct TCP_Server_Info *server, -const struct cifs_credits *credits,const int optype) +struct cifs_credits *credits,const int optype) {spin_lock (&server->req_lock); server->credits += credits->value;
fs/smb/client/smb2ops.c +37 -5
79 lines not on this page · raw message
fs/smb/client/smb2pdu.c +36 -4
@@ -4502,8 +4502,15 @@ smb2_readv_callback(struct mid_q_entry *mid)struct TCP_Server_Info *server = rdata->server;struct smb2_hdr *shdr = (struct smb2_hdr *)rdata->iov[0 ].iov_base; -struct cifs_credits credits = { .value =0 , .instance =0 }; +struct cifs_credits credits = { + .value =0 , + .instance =0 , + .rreq_debug_id = rdata->rreq->debug_id, + .rreq_debug_index = rdata->subreq.debug_index, + };struct smb_rqst rqst = { .rq_iov = &rdata->iov[1 ], .rq_nvec =1 }; +unsigned int rreq_debug_id = rdata->rreq->debug_id; +unsigned int subreq_debug_index = rdata->subreq.debug_index;if (rdata->got_bytes) { rqst.rq_iter = rdata->subreq.io_iter;
@@ -4587,10 +4594,16 @@ smb2_readv_callback(struct mid_q_entry *mid)if (rdata->subreq.start < rdata->subreq.rreq->i_size) rdata->result =0 ; } +trace_smb3_rw_credits (rreq_debug_id, subreq_debug_index, rdata->credits.value, + server->credits, server->in_flight, +0 , cifs_trace_rw_credits_read_response_clear); rdata->credits.value =0 ;INIT_WORK (&rdata->subreq.work, smb2_readv_worker);queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid); +trace_smb3_rw_credits (rreq_debug_id, subreq_debug_index,0 , + server->credits, server->in_flight, + credits.value, cifs_trace_rw_credits_read_response_add);add_credits (server, &credits,0 ); }
@@ -4647,7 +4660,7 @@ smb2_async_readv(struct cifs_io_subrequest *rdata)min_t (int, server->max_credits - server->credits, credit_request)); - rc =adjust_credits (server, &rdata->credits, rdata->subreq.len); + rc =adjust_credits (server, rdata, cifs_trace_rw_credits_call_readv_adjust);if (rc) goto async_readv_out;
@@ -4766,7 +4779,14 @@ smb2_writev_callback(struct mid_q_entry *mid)struct cifs_tcon *tcon =tlink_tcon (wdata->req->cfile->tlink);struct TCP_Server_Info *server = wdata->server;struct smb2_write_rsp *rsp = (struct smb2_write_rsp *)mid->resp_buf; -struct cifs_credits credits = { .value =0 , .instance =0 }; +struct cifs_credits credits = { + .value =0 , + .instance =0 , + .rreq_debug_id = wdata->rreq->debug_id, + .rreq_debug_index = wdata->subreq.debug_index, + }; +unsigned int rreq_debug_id = wdata->rreq->debug_id; +unsigned int subreq_debug_index = wdata->subreq.debug_index;ssize_t result =0 ;size_t written;
@@ -4837,9 +4857,15 @@ smb2_writev_callback(struct mid_q_entry *mid) tcon->tid, tcon->ses->Suid, wdata->subreq.start, wdata->subreq.len); +trace_smb3_rw_credits (rreq_debug_id, subreq_debug_index, wdata->credits.value, + server->credits, server->in_flight, +0 , cifs_trace_rw_credits_write_response_clear); wdata->credits.value =0 ;cifs_write_subrequest_terminated (wdata, result ?: written, true);release_mid (mid); +trace_smb3_rw_credits (rreq_debug_id, subreq_debug_index,0 , + server->credits, server->in_flight, + credits.value, cifs_trace_rw_credits_write_response_add);add_credits (server, &credits,0 ); }
@@ -4969,7 +4995,7 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)min_t (int, server->max_credits - server->credits, credit_request)); - rc =adjust_credits (server, &wdata->credits, io_parms->length); + rc =adjust_credits (server, wdata, cifs_trace_rw_credits_call_writev_adjust);if (rc) goto async_writev_out;
@@ -4994,6 +5020,12 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)cifs_small_buf_release (req); out:if (rc) { +trace_smb3_rw_credits (wdata->rreq->debug_id, + wdata->subreq.debug_index, + wdata->credits.value, + server->credits, server->in_flight, + -(int )wdata->credits.value, + cifs_trace_rw_credits_write_response_clear);add_credits_and_wake_if (wdata->server, &wdata->credits,0 );cifs_write_subrequest_terminated (wdata, rc, true); }
fs/smb/client/trace.h +54 -1
79 lines not on this page · raw message
fs/smb/client/transport.c +4 -4
@@ -988,10 +988,10 @@ static void cifs_compound_callback(struct mid_q_entry *mid) {struct TCP_Server_Info *server = mid->server; -struct cifs_credits credits; - - credits.value = server->ops->get_credits (mid); - credits.instance = server->reconnect_instance; +struct cifs_credits credits = { + .value = server->ops->get_credits (mid), + .instance = server->reconnect_instance, + };add_credits (server, &credits, mid->optype);
applied to cifs-2.6.git for-next and running regression tests on it now
Quoted 37 lines · David Howells
Hi Steve,
Here are four patches for cifs:
(1) Fix re-repick of a server upon subrequest retry.
(2) Fix some error code setting that got accidentally removed.
(3) Fix the handling of the
zero_pointafter a DIO write. It always needs to be bumped past the end of the DIO write.(4) Add a tracepoint and some debugging to keep track of when we've ended
the ->in_flight contribution from an operation.I've pushed the patches here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=cifs-fixes
David
David Howells (4):
cifs: Fix server re-repick on subrequest retry
cifs: Fix missing error code set
cifs: Fix setting ofzero_pointafter DIO write
cifs: Add a tracepoint to track credits involved in R/W requestsfs/smb/client/cifsglob.h | 17 +++++++----- fs/smb/client/file.c | 47 +++++++++++++++++++++++++++++---- fs/smb/client/smb1ops.c | 2 +- fs/smb/client/smb2ops.c | 42 ++++++++++++++++++++++++++---- fs/smb/client/smb2pdu.c | 43 +++++++++++++++++++++++++----- fs/smb/client/trace.h | 55 ++++++++++++++++++++++++++++++++++++++- fs/smb/client/transport.c | 8 +++--- 7 files changed, 184 insertions(+), 30 deletions(-)
Signature · 3 lines
Thanks, Steve
David Howells
maintainerRe: [PATCH 5/4] cifs: Fix missing fscache invalidation
In reply to David HowellsA network filesystem needs to implement a netfslib hook to invalidate fscache if it's to be able to use the cache.
Fix cifs to implement the cache invalidation hook.
--- fs/smb/client/file.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/smb/client/file.c +6 -0
@@ -133,6 +133,11 @@ static void cifs_issue_write(struct netfs_io_subrequest *subreq) goto out; } +static void cifs_netfs_invalidate_cache (struct netfs_io_request *wreq) +{ +cifs_invalidate_cache (wreq->inode,0 ); +} +/* * Split the read up according to how many credits we can get for each piece. * It's okay to sleep here if we need to wait for more credit to become
@@ -337,6 +342,7 @@ const struct netfs_request_ops cifs_req_ops = { .begin_writeback = cifs_begin_writeback, .prepare_write = cifs_prepare_write, .issue_write = cifs_issue_write, + .invalidate_cache = cifs_netfs_invalidate_cache, };/*