netfs, cifs: Fixes to retry-related code and RDMA support
Subsystems (10)
David Howells
maintainerv2 cover netfs, cifs: Fixes to retry-related code and RDMA support
Hi Christian, Steve,
Here are some miscellaneous fixes and changes for netfslib and cifs, if you could consider pulling them. All the bugs fixed were observed in cifs, so they should probably go through the cifs tree unless Christian would much prefer for them to go through the VFS tree.
Many of these were found because a bug in Samba was causing smbd to crash and restart after about 1-2s and this was vigorously and abruptly exercising the netfslib retry paths.
Subsequent testing of the cifs RDMA support showed up some more bugs, for which fixes are also included here.
First, there are some netfs fixes:
(1) Fix a hang due to missing case in final DIO read result collection not breaking out of a loop if the request finished, but there were no subrequests being processed and NETFS_RREQ_ALL_QUEUED wasn't yet set.
(2) Fix a double put of the netfs_io_request struct if completion happened in the pause loop.
(3) Provide some helpers to abstract out NETFS_RREQ_IN_PROGRESS flag wrangling.
(4) Fix infinite looping in netfs_wait_for_pause/request() which wa caused by a loop waiting for NETFS_RREQ_ALL_QUEUED to get set - but which wouldn't get set until the looping function returned. This uses patch
(3) above.
(5) Fix a ref leak on an extra subrequest inserted into a request's list of subreqs because more subreq records were needed for retrying than were needed for the original request (say, for instance, that the amount of cifs credit available was reduced and, subsequently, the ops had to be smaller).
Then a bunch of cifs fixes, some of which are from other people:
(6-8) cifs: Fix various RPC callbacks to set NETFS_SREQ_NEED_RETRY if a
subrequest fails retriably.
(9) Fix a regression with SMB symlinks.
(10) Fix a warning in the workqueue code when reconnecting a channel.
(11) Fix smbd_post_send_iter() to correctly respect the max_send_size and to transmit all the data.
(12) Fix reading into an ITER_FOLIOQ from the smbdirect code.
(13) Fix the smbd_response slab to allow copy_to_iter() to be used on it without incurring a bug from the usercopy hardening code.
(14) Fix a potential deadlock during channel reconnection.
And finally a couple of patches to improve tracing output, but that should otherwise not affect functionality:
(15) Renumber the NETFS_RREQ_* flags to make the hex values easier to interpret by eye, including moving the main status flags down to the lowest bits, with IN_PROGRESS in bit 0.
(16) Update the tracepoints in a number of ways, including adding more tracepoints into the cifs read/write RPC callback so that differend MID_RESPONSE_* values can be differentiated.
Those last two could wait for the next merge window.
The patches can also be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-fixes
Thanks,
David
David Howells (9):
netfs: Fix hang due to missing case in final DIO read result
collection
netfs: Put double put of request
netfs: Provide helpers to perform NETFS_RREQ_IN_PROGRESS flag wangling
netfs: Fix looping in wait functions
netfs: Fix ref leak on inserted extra subreq in write retry
cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
cifs: Fix the smbd_reponse slab to allow usercopy
netfs: Renumber the NETFS_RREQ_* flags to make traces easier to read
netfs: Update tracepoints in a number of ways
Paulo Alcantara (6):
smb: client: set missing retry flag in smb2_writev_callback()
smb: client: set missing retry flag in cifs_readv_callback()
smb: client: set missing retry flag in cifs_writev_callback()
smb: client: fix regression with native SMB symlinks
smb: client: fix warning when reconnecting channel
smb: client: fix potential deadlock when reconnecting channels
Stefan Metzmacher (1):
smb: client: let smbd_post_send_iter() respect the peers max_send_size
and transmit all data
fs/netfs/direct_write.c | 1 - fs/netfs/internal.h | 20 ++++- fs/netfs/main.c | 6 +- fs/netfs/misc.c | 50 +++++++---- fs/netfs/read_collect.c | 16 ++-- fs/netfs/write_collect.c | 8 +- fs/netfs/write_retry.c | 3 +- fs/smb/client/cifsglob.h | 2 + fs/smb/client/cifssmb.c | 22 +++++ fs/smb/client/connect.c | 53 +++++++----- fs/smb/client/reparse.c | 20 +---- fs/smb/client/smb2pdu.c | 37 +++++--- fs/smb/client/smbdirect.c | 163 +++++++++++++---------------------- include/linux/netfs.h | 20 ++--- include/trace/events/netfs.h | 29 +++++-- 15 files changed, 245 insertions(+), 205 deletions(-)
David Howells
maintainerPATCH v2 1/16 netfs: Fix hang due to missing case in final DIO read result collection
When doing a DIO read, if the subrequests we issue fail and cause the request PAUSE flag to be set to put a pause on subrequest generation, we may complete collection of the subrequests (possibly discarding them) prior to the ALL_QUEUED flags being set.
In such a case, netfs_read_collection() doesn't see ALL_QUEUED being set after netfs_collect_read_results() returns and will just return to the app (the collector can be seen unpausing the generator in the trace log).
The subrequest generator can then set ALL_QUEUED and the app thread reaches netfs_wait_for_request(). This causes netfs_collect_in_app() to be called to see if we're done yet, but there's missing case here.
netfs_collect_in_app() will see that a thread is active and set inactive to false, but won't see any subrequests in the read stream, and so won't set need_collect to true. The function will then just return 0, indicating that the caller should just sleep until further activity (which won't be forthcoming) occurs.
Fix this by making netfs_collect_in_app() check to see if an active thread is complete - i.e. that ALL_QUEUED is set and the subrequests list is empty
- and to skip the sleep return path. The collector will then be called which will clear the request IN_PROGRESS flag, allowing the app to progress.
--- fs/netfs/misc.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/netfs/misc.c +4 -2
@@ -381,7 +381,7 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,static int netfs_collect_in_app (struct netfs_io_request *rreq,bool (*collector)(struct netfs_io_request *rreq)) { -bool need_collect = false, inactive = true; +bool need_collect = false, inactive = true, done = true;for (int i =0 ; i <NR_IO_STREAMS ; i++) {struct netfs_io_subrequest *subreq;
@@ -400,9 +400,11 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq, need_collect = true;break ; } +if (subreq || !test_bit (NETFS_RREQ_ALL_QUEUED , &rreq->flags)) + done = false; } -if (!need_collect && !inactive) +if (!need_collect && !inactive && !done)return 0 ;/* Sleep */ __set_current_state (TASK_RUNNING );
Re: [PATCH v2 01/16] netfs: Fix hang due to missing case in final DIO read result collection
You can add my tested by to the first 11 in the series. I have verified that they fix the netfs regression (e.g. hangs in xfstest generic/013 etc.). The series appears important to make sure gets in 6.16
Quoted 65 lines · David Howells
When doing a DIO read, if the subrequests we issue fail and cause the request PAUSE flag to be set to put a pause on subrequest generation, we may complete collection of the subrequests (possibly discarding them) prior to the
ALL_QUEUEDflags being set.In such a case,
netfs_read_collection()doesn't seeALL_QUEUEDbeing set afternetfs_collect_read_results()returns and will just return to the app (the collector can be seen unpausing the generator in the trace log).The subrequest generator can then set
ALL_QUEUEDand the app thread reachesnetfs_wait_for_request(). This causesnetfs_collect_in_app()to be called to see if we're done yet, but there's missing case here.
netfs_collect_in_app()will see that a thread is active and set inactive to false, but won't see any subrequests in the read stream, and so won't setneed_collectto true. The function will then just return 0, indicating that the caller should just sleep until further activity (which won't be forthcoming) occurs.Fix this by making
netfs_collect_in_app()check to see if an active thread is complete - i.e. thatALL_QUEUEDis set and the subrequests list is empty
- and to skip the sleep return path. The collector will then be called which will clear the requestIN_PROGRESSflag, allowing the app to progress.Fixes 2b1424cd131c ("netfs: Fix wait/wake to be consistent about the waitqueue used")Reported-by Steve French <[email protected]>Signed-off-by David Howells <[email protected]>Reviewed-by Paulo Alcantara <[email protected]>--- fs/netfs/misc.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fs/netfs/misc.c +4
-2@@ -381,7 +381,7 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,static int netfs_collect_in_app (struct netfs_io_request *rreq,bool (*collector)(struct netfs_io_request *rreq)) { -bool need_collect = false, inactive = true; +bool need_collect = false, inactive = true, done = true;for (int i =0 ; i <NR_IO_STREAMS ; i++) {struct netfs_io_subrequest *subreq;@@ -400,9 +400,11 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq, need_collect = true;break ; } +if (subreq || !test_bit (NETFS_RREQ_ALL_QUEUED , &rreq->flags)) + done = false; } -if (!need_collect && !inactive) +if (!need_collect && !inactive && !done)return 0 ;/* Sleep */ __set_current_state (TASK_RUNNING );
Signature · 3 lines
Thanks, Steve
David Howells
maintainerPATCH v2 2/16 netfs: Put double put of request
In reply to David HowellsIf a netfs request finishes during the pause loop, it will have the ref that belongs to the IN_PROGRESS flag removed at that point - however, if it then goes to the final wait loop, that will *also* put the ref because it sees that the IN_PROGRESS flag is clear and incorrectly assumes that this happened when it called the collector.
In fact, since IN_PROGRESS is clear, we shouldn't call the collector again since it's done all the cleanup, such as calling ->ki_complete().
Fix this by making netfs_collect_in_app() just return, indicating that we're done if IN_PROGRESS is removed.
--- fs/netfs/misc.c | 5 +++++ include/trace/events/netfs.h | 1 + 2 files changed, 6 insertions(+)
fs/netfs/misc.c +5 -0
@@ -383,6 +383,11 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq, {bool need_collect = false, inactive = true, done = true; +if (!netfs_check_rreq_in_progress (rreq)) { +trace_netfs_rreq (rreq, netfs_rreq_trace_recollect); +return 1 ;/* Done */ + } +for (int i =0 ; i <NR_IO_STREAMS ; i++) {struct netfs_io_subrequest *subreq;struct netfs_io_stream *stream = &rreq->io_streams[i];
include/trace/events/netfs.h +1 -0
@@ -56,6 +56,7 @@ EM(netfs_rreq_trace_dirty, "DIRTY ") \ EM(netfs_rreq_trace_done, "DONE ") \ EM(netfs_rreq_trace_free, "FREE ") \ + EM(netfs_rreq_trace_recollect, "RECLLCT") \ EM(netfs_rreq_trace_redirty, "REDIRTY") \ EM(netfs_rreq_trace_resubmit, "RESUBMT") \EM (netfs_rreq_trace_set_abandon,"S-ABNDN" ) \
David Howells
maintainerPATCH v2 3/16 netfs: Provide helpers to perform NETFS_RREQ_IN_PROGRESS flag wangling
In reply to David HowellsProvide helpers to clear and test the NETFS_RREQ_IN_PROGRESS and to insert the appropriate barrierage.
--- fs/netfs/internal.h | 18 ++++++++++++++++++ fs/netfs/misc.c | 10 +++++----- fs/netfs/read_collect.c | 4 ++-- fs/netfs/write_collect.c | 4 ++-- 4 files changed, 27 insertions(+), 9 deletions(-)
fs/netfs/internal.h +18 -0
@@ -274,6 +274,24 @@ static inline void netfs_wake_rreq_flag(struct netfs_io_request *rreq, } } +/* +* Test the NETFS_RREQ_IN_PROGRESS flag, inserting an appropriate barrier. +*/ +static inline bool netfs_check_rreq_in_progress (const struct netfs_io_request *rreq) +{ +/* Order read of flags before read of anything else, such as error. */ +return test_bit_acquire (NETFS_RREQ_IN_PROGRESS , &rreq->flags); +} + +/* +* Test the NETFS_SREQ_IN_PROGRESS flag, inserting an appropriate barrier. +*/ +static inline bool netfs_check_subreq_in_progress (const struct netfs_io_subrequest *subreq) +{ +/* Order read of flags before read of anything else, such as error. */ +return test_bit_acquire (NETFS_SREQ_IN_PROGRESS , &subreq->flags); +} +/* * fscache-cache.c */
fs/netfs/misc.c +5 -5
@@ -356,14 +356,14 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,DEFINE_WAIT (myself);list_for_each_entry (subreq, &stream->subrequests, rreq_link) { -if (!test_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags)) +if (!netfs_check_subreq_in_progress (subreq))continue ;trace_netfs_rreq (rreq, netfs_rreq_trace_wait_queue);for (;;) {prepare_to_wait (&rreq->waitq, &myself,TASK_UNINTERRUPTIBLE ); -if (!test_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags)) +if (!netfs_check_subreq_in_progress (subreq))break ;trace_netfs_sreq (subreq, netfs_sreq_trace_wait_for);
@@ -400,7 +400,7 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq, struct netfs_io_subrequest, rreq_link);if (subreq && - (!test_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags) || + (!netfs_check_subreq_in_progress (subreq) ||test_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags))) { need_collect = true;break ;
@@ -451,7 +451,7 @@ static ssize_t netfs_wait_for_request(struct netfs_io_request *rreq, } } -if (!test_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags)) +if (!netfs_check_rreq_in_progress (rreq))break ;schedule ();
@@ -518,7 +518,7 @@ static void netfs_wait_for_pause(struct netfs_io_request *rreq, } } -if (!test_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags) || +if (!netfs_check_rreq_in_progress (rreq) || !test_bit (NETFS_RREQ_PAUSE , &rreq->flags))break ;
fs/netfs/read_collect.c +2 -2
@@ -218,7 +218,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq) stream->collected_to = front->start; } -if (test_bit (NETFS_SREQ_IN_PROGRESS , &front->flags)) +if (netfs_check_subreq_in_progress (front)) notes |=HIT_PENDING ;smp_rmb ();/* Read counters after IN_PROGRESS flag. */ transferred =READ_ONCE (front->transferred);
@@ -445,7 +445,7 @@ void netfs_read_collection_worker(struct work_struct *work) struct netfs_io_request *rreq = container_of(work, struct netfs_io_request, work);netfs_see_request (rreq, netfs_rreq_trace_see_work); -if (test_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags)) { +if (netfs_check_rreq_in_progress (rreq)) {if (netfs_read_collection (rreq))/* Drop the ref from the IN_PROGRESS flag. */ netfs_put_request (rreq, netfs_rreq_trace_put_work_ip);
fs/netfs/write_collect.c +2 -2
@@ -240,7 +240,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) }/* Stall if the front is still undergoing I/O. */ -if (test_bit (NETFS_SREQ_IN_PROGRESS , &front->flags)) { +if (netfs_check_subreq_in_progress (front)) { notes |=HIT_PENDING ;break ; }
@@ -434,7 +434,7 @@ void netfs_write_collection_worker(struct work_struct *work) struct netfs_io_request *rreq = container_of(work, struct netfs_io_request, work);netfs_see_request (rreq, netfs_rreq_trace_see_work); -if (test_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags)) { +if (netfs_check_rreq_in_progress (rreq)) {if (netfs_write_collection (rreq))/* Drop the ref from the IN_PROGRESS flag. */ netfs_put_request (rreq, netfs_rreq_trace_put_work_ip);
David Howells
maintainerPATCH v2 4/16 netfs: Fix looping in wait functions
In reply to David Howellsnetfs_wait_for_request() and netfs_wait_for_pause() can loop forever if netfs_collect_in_app() returns 2, indicating that it wants to repeat because the ALL_QUEUED flag isn't yet set and there are no subreqs left that haven't been collected.
The problem is that, unless collection is offloaded (OFFLOAD_COLLECTION), we have to return to the application thread to continue and eventually set ALL_QUEUED after pausing to deal with a retry - but we never get there.
Fix this by inserting checks for the IN_PROGRESS and PAUSE flags as appropriate before cycling round - and add cond_resched() for good measure.
--- fs/netfs/misc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
fs/netfs/misc.c +11 -4
@@ -430,8 +430,8 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq,/* * Wait for a request to complete, successfully or otherwise. */ -static ssize_t netfs_wait_for_request (struct netfs_io_request *rreq, -bool (*collector)(struct netfs_io_request *rreq)) +static ssize_t netfs_wait_for_in_progress (struct netfs_io_request *rreq, +bool (*collector)(struct netfs_io_request *rreq)) {DEFINE_WAIT (myself);ssize_t ret;
@@ -447,6 +447,9 @@ static ssize_t netfs_wait_for_request(struct netfs_io_request *rreq,case 1 : goto all_collected;case 2 : +if (!netfs_check_rreq_in_progress (rreq)) +break ; +cond_resched ();continue ; } }
@@ -485,12 +488,12 @@ static ssize_t netfs_wait_for_request(struct netfs_io_request *rreq,ssize_t netfs_wait_for_read (struct netfs_io_request *rreq) { -return netfs_wait_for_request (rreq, netfs_read_collection); +return netfs_wait_for_in_progress (rreq, netfs_read_collection); }ssize_t netfs_wait_for_write (struct netfs_io_request *rreq) { -return netfs_wait_for_request (rreq, netfs_write_collection); +return netfs_wait_for_in_progress (rreq, netfs_write_collection); }/*
@@ -514,6 +517,10 @@ static void netfs_wait_for_pause(struct netfs_io_request *rreq,case 1 : goto all_collected;case 2 : +if (!netfs_check_rreq_in_progress (rreq) || + !test_bit (NETFS_RREQ_PAUSE , &rreq->flags)) +break ; +cond_resched ();continue ; } }
David Howells
maintainerPATCH v2 5/16 netfs: Fix ref leak on inserted extra subreq in write retry
In reply to David HowellsThe write-retry algorithm will insert extra subrequests into the list if it can't get sufficient capacity to split the range that needs to be retried into the sequence of subrequests it currently has (for instance, if the cifs credit pool has fewer credits available than it did when the range was originally divided).
However, the allocator furnishes each new subreq with 2 refs and then another is added for resubmission, causing one to be leaked.
Fix this by replacing the ref-getting line with a neutral trace line.
--- fs/netfs/write_retry.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/netfs/write_retry.c +1 -1
@@ -153,7 +153,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,trace_netfs_sreq_ref (wreq->debug_id, subreq->debug_index,refcount_read (&subreq->ref), netfs_sreq_trace_new); -netfs_get_subrequest (subreq, netfs_sreq_trace_get_resubmit); +trace_netfs_sreq (subreq, netfs_sreq_trace_split);list_add (&subreq->rreq_link, &to->rreq_link); to =list_next_entry (to, rreq_link);
David Howells
maintainerPATCH v2 6/16 smb: client: set missing retry flag in smb2_writev_callback()
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
Set NETFS_SREQ_NEED_RETRY flag to tell netfslib that the subreq needs to be retried.
--- fs/smb/client/smb2pdu.c | 1 + 1 file changed, 1 insertion(+)
fs/smb/client/smb2pdu.c +1 -0
@@ -4862,6 +4862,7 @@ smb2_writev_callback(struct mid_q_entry *mid)break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED : +__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); result = -EAGAIN ;break ;case MID_RESPONSE_MALFORMED :
David Howells
maintainerPATCH v2 7/16 smb: client: set missing retry flag in cifs_readv_callback()
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
Set NETFS_SREQ_NEED_RETRY flag to tell netfslib that the subreq needs to be retried.
--- fs/smb/client/cifssmb.c | 1 + 1 file changed, 1 insertion(+)
fs/smb/client/cifssmb.c +1 -0
@@ -1335,6 +1335,7 @@ cifs_readv_callback(struct mid_q_entry *mid)break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED : +__set_bit (NETFS_SREQ_NEED_RETRY , &rdata->subreq.flags); rdata->result = -EAGAIN ;if (server->sign && rdata->got_bytes)/* reset bytes number since we can not check a sign */
David Howells
maintainerPATCH v2 8/16 smb: client: set missing retry flag in cifs_writev_callback()
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
Set NETFS_SREQ_NEED_RETRY flag to tell netfslib that the subreq needs to be retried.
--- fs/smb/client/cifssmb.c | 1 + 1 file changed, 1 insertion(+)
fs/smb/client/cifssmb.c +1 -0
@@ -1715,6 +1715,7 @@ cifs_writev_callback(struct mid_q_entry *mid)break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED : +__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); result = -EAGAIN ;break ;default :
David Howells
maintainerPATCH v2 9/16 smb: client: fix regression with native SMB symlinks
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
Some users and customers reported that their backup/copy tools started to fail when the directory being copied contained symlink targets that the client couldn't parse - even when those symlinks weren't followed.
Fix this by allowing lstat(2) and readlink(2) to succeed even when the client can't resolve the symlink target, restoring old behavior.
--- fs/smb/client/reparse.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
fs/smb/client/reparse.c +4 -16
@@ -875,15 +875,8 @@ int smb2_parse_native_symlink(char **target, const char *buf, unsigned int len, abs_path +=sizeof ("\\DosDevices\\" )-1 ; else if (strstarts(abs_path, "\\GLOBAL??\\")) abs_path += sizeof("\\GLOBAL??\\")-1; - else { -/* Unhandled absolute symlink, points outside of DOS/Win32 */ -cifs_dbg (VFS , -"absolute symlink '%s' cannot be converted from NT format " -"because points to unknown target\n" , - smb_target); - rc = -EIO ; - goto out; - } + else + goto out_unhandled_target;/* Sometimes path separator after \?? is double backslash */ if (abs_path[0 ] =='\\' )
@@ -910,13 +903,7 @@ int smb2_parse_native_symlink(char **target, const char *buf, unsigned int len, abs_path++; abs_path[0 ] = drive_letter; }else { -/* Unhandled absolute symlink. Report an error. */ -cifs_dbg (VFS , -"absolute symlink '%s' cannot be converted from NT format " -"because points to unknown target\n" , - smb_target); - rc = -EIO ; - goto out; + goto out_unhandled_target; } abs_path_len =strlen (abs_path)+1 ;
@@ -966,6 +953,7 @@ int smb2_parse_native_symlink(char **target, const char *buf, unsigned int len,* These paths have same format as Linux symlinks, so no * conversion is needed. */ +out_unhandled_target: linux_target = smb_target; smb_target =NULL ; }
David Howells
maintainerPATCH v2 10/16 smb: client: fix warning when reconnecting channel
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
When reconnecting a channel in smb2_reconnect_server(), a dummy tcon is passed down to smb2_reconnect() with ->query_interface uninitialized, so we can't call queue_delayed_work() on it.
Fix the following warning by ensuring that we're queueing the delayed worker from correct tcon.
WARNING: CPU: 4 PID: 1126 at kernel/workqueue.c:2498 __queue_delayed_work+0x1d2/0x200 Modules linked in: cifs cifs_arc4 nls_ucs2_utils cifs_md4 [last unloaded: cifs] CPU: 4 UID: 0 PID: 1126 Comm: kworker/4:0 Not tainted 6.16.0-rc3 #5 PREEMPT(voluntary) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.fc42 04/01/2014 Workqueue: cifsiod smb2_reconnect_server [cifs] RIP: 0010:__queue_delayed_work+0x1d2/0x200 Code: 41 5e 41 5f e9 7f ee ff ff 90 0f 0b 90 e9 5d ff ff ff bf 02 00 00 00 e8 6c f3 07 00 89 c3 eb bd 90 0f 0b 90 e9 57 f> 0b 90 e9 65 fe ff ff 90 0f 0b 90 e9 72 fe ff ff 90 0f 0b 90 e9 RSP: 0018:ffffc900014afad8 EFLAGS: 00010003 RAX: 0000000000000000 RBX: ffff888124d99988 RCX: ffffffff81399cc1 RDX: dffffc0000000000 RSI: ffff888114326e00 RDI: ffff888124d999f0 RBP: 000000000000ea60 R08: 0000000000000001 R09: ffffed10249b3331 R10: ffff888124d9998f R11: 0000000000000004 R12: 0000000000000040 R13: ffff888114326e00 R14: ffff888124d999d8 R15: ffff888114939020 FS: 0000000000000000(0000) GS:ffff88829f7fe000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffe7a2b4038 CR3: 0000000120a6f000 CR4: 0000000000750ef0 PKRU: 55555554 Call Trace: <TASK> queue_delayed_work_on+0xb4/0xc0 smb2_reconnect+0xb22/0xf50 [cifs] smb2_reconnect_server+0x413/0xd40 [cifs] ? __pfx_smb2_reconnect_server+0x10/0x10 [cifs] ? local_clock_noinstr+0xd/0xd0 ? local_clock+0x15/0x30 ? lock_release+0x29b/0x390 process_one_work+0x4c5/0xa10 ? __pfx_process_one_work+0x10/0x10 ? __list_add_valid_or_report+0x37/0x120 worker_thread+0x2f1/0x5a0 ? __kthread_parkme+0xde/0x100 ? __pfx_worker_thread+0x10/0x10 kthread+0x1fe/0x380 ? kthread+0x10f/0x380 ? __pfx_kthread+0x10/0x10 ? local_clock_noinstr+0xd/0xd0 ? ret_from_fork+0x1b/0x1f0 ? local_clock+0x15/0x30 ? lock_release+0x29b/0x390 ? rcu_is_watching+0x20/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x15b/0x1f0 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> irq event stamp: 1116206 hardirqs last enabled at (1116205): [<ffffffff8143af42>] __up_console_sem+0x52/0x60 hardirqs last disabled at (1116206): [<ffffffff81399f0e>] queue_delayed_work_on+0x6e/0xc0 softirqs last enabled at (1116138): [<ffffffffc04562fd>] __smb_send_rqst+0x42d/0x950 [cifs] softirqs last disabled at (1116136): [<ffffffff823d35e1>] release_sock+0x21/0xf0
--- fs/smb/client/cifsglob.h | 1 + fs/smb/client/smb2pdu.c | 10 ++++------ 2 files changed, 5 insertions(+), 6 deletions(-)
fs/smb/client/cifsglob.h +1 -0
@@ -1302,6 +1302,7 @@ struct cifs_tcon { bool use_persistent:1; /* use persistent instead of durable handles */ bool no_lease:1; /* Do not request leases on files or directories */ bool use_witness:1; /* use witness protocol */ + bool dummy:1; /* dummy tcon used for reconnecting channels */__le32 capabilities;__u32 share_flags;__u32 maximal_access;
fs/smb/client/smb2pdu.c +4 -6
@@ -424,9 +424,9 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon,free_xid (xid); ses->flags &= ~CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES ; -/* regardless of rc value, setup polling */ -queue_delayed_work (cifsiod_wq, &tcon->query_interfaces, - (SMB_INTERFACE_POLL_INTERVAL *HZ )); +if (!tcon->ipc && !tcon->dummy) +queue_delayed_work (cifsiod_wq, &tcon->query_interfaces, + (SMB_INTERFACE_POLL_INTERVAL *HZ ));mutex_unlock (&ses->session_mutex);
@@ -4229,10 +4229,8 @@ void smb2_reconnect_server(struct work_struct *work) } goto done; } - tcon->status =TID_GOOD ; - tcon->retry = false; - tcon->need_reconnect = false; + tcon->dummy = true;/* now reconnect sessions for necessary channels */ list_for_each_entry_safe (ses, ses2, &tmp_ses_list, rlist) {
David Howells
maintainerPATCH v2 11/16 smb: client: let smbd_post_send_iter() respect the peers max_send_size and transmit all data
In reply to David HowellsFrom: Stefan Metzmacher <[email protected]>
We should not send smbdirect_data_transfer messages larger than the negotiated max_send_size, typically 1364 bytes, which means 24 bytes of the smbdirect_data_transfer header + 1340 payload bytes.
This happened when doing an SMB2 write with more than 1340 bytes (which is done inline as it's below rdma_readwrite_threshold).
It means the peer resets the connection.
When testing between cifs.ko and ksmbd.ko something like this is logged:
client:
CIFS: VFS: RDMA transport re-established
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
siw: got TERMINATE. layer 1, type 2, code 2
CIFS: VFS: \\carina Send error in SessSetup = -11
smb2_reconnect: 12 callbacks suppressed
CIFS: VFS: reconnect tcon failed rc = -11
CIFS: VFS: reconnect tcon failed rc = -11
CIFS: VFS: reconnect tcon failed rc = -11
CIFS: VFS: SMB: Zero rsize calculated, using minimum value 65536
and:
CIFS: VFS: RDMA transport re-established
siw: got TERMINATE. layer 1, type 2, code 2
CIFS: VFS: smbd_recv:1894 disconnected
siw: got TERMINATE. layer 1, type 2, code 2
The ksmbd dmesg is showing things like:
smb_direct: Recv error. status='local length error (1)' opcode=128
smb_direct: disconnected
smb_direct: Recv error. status='local length error (1)' opcode=128
ksmbd: smb_direct: disconnected
ksmbd: sock_read failed: -107
As smbd_post_send_iter() limits the transmitted number of bytes we need loop over it in order to transmit the whole iter.
--- fs/smb/client/smbdirect.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)
fs/smb/client/smbdirect.c +27 -4
@@ -907,8 +907,10 @@ static int smbd_post_send_iter(struct smbd_connection *info, .local_dma_lkey = sc->ib.pd->local_dma_lkey, .direction = DMA_TO_DEVICE, }; +size_t payload_len =umin (*_remaining_data_length, + sp->max_send_size -sizeof (*packet)); - rc =smb_extract_iter_to_rdma (iter, *_remaining_data_length, + rc =smb_extract_iter_to_rdma (iter, payload_len, &extract);if (rc <0 ) goto err_dma;
@@ -1013,6 +1015,27 @@ static int smbd_post_send_empty(struct smbd_connection *info)return smbd_post_send_iter (info,NULL , &remaining_data_length); } +static int smbd_post_send_full_iter (struct smbd_connection *info, +struct iov_iter *iter, +int *_remaining_data_length) +{ +int rc =0 ; + +/* +* smbd_post_send_iter() respects the +* negotiated max_send_size, so we need to +* loop until the full iter is posted +*/ + +while (iov_iter_count (iter) >0 ) { + rc =smbd_post_send_iter (info, iter, _remaining_data_length); +if (rc <0 ) +break ; + } + +return rc; +} +/* * Post a receive request to the transport * The remote peer can only send data when a receive request is posted
@@ -2032,14 +2055,14 @@ int smbd_send(struct TCP_Server_Info *server, klen += rqst->rq_iov[i].iov_len;iov_iter_kvec (&iter,ITER_SOURCE , rqst->rq_iov, rqst->rq_nvec, klen); - rc =smbd_post_send_iter (info, &iter, &remaining_data_length); + rc =smbd_post_send_full_iter (info, &iter, &remaining_data_length);if (rc <0 )break ;if (iov_iter_count (&rqst->rq_iter) >0 ) {/* And then the data pages if there are any */ - rc =smbd_post_send_iter (info, &rqst->rq_iter, - &remaining_data_length); + rc =smbd_post_send_full_iter (info, &rqst->rq_iter, + &remaining_data_length);if (rc <0 )break ; }
David Howells
maintainerPATCH v2 12/16 cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
In reply to David HowellsWhen performing a file read from RDMA, smbd_recv() prints an "Invalid msg type 4" error and fails the I/O. This is due to the switch-statement there not handling the ITER_FOLIOQ handed down from netfslib.
Fix this by collapsing smbd_recv_buf() and smbd_recv_page() into smbd_recv() and just using copy_to_iter() instead of memcpy(). This future-proofs the function too, in case more ITER_* types are added.
--- fs/smb/client/smbdirect.c | 114 +++++++------------------------------- 1 file changed, 19 insertions(+), 95 deletions(-)
fs/smb/client/smbdirect.c +19 -95
@@ -1770,35 +1770,39 @@ struct smbd_connection *smbd_get_connection( }/* -* Receive data from receive reassembly queue +* Receive data from the transport's receive reassembly queue * All the incoming data packets are placed in reassembly queue -* buf: the buffer to read data into +* iter: the buffer to read data into * size: the length of data to read * return value: actual data read -* Note: this implementation copies the data from reassebmly queue to receive +* +* Note: this implementation copies the data from reassembly queue to receive * buffers used by upper layer. This is not the optimal code path. A better way * to do it is to not have upper layer allocate its receive buffers but rather * borrow the buffer from reassembly queue, and return it after data is * consumed. But this will require more changes to upper layer code, and also * need to consider packet boundaries while they still being reassembled. */ -static int smbd_recv_buf (struct smbd_connection *info,char *buf, -unsigned int size) +int smbd_recv (struct smbd_connection *info,struct msghdr *msg) {struct smbdirect_socket *sc = &info->socket;struct smbd_response *response;struct smbdirect_data_transfer *data_transfer; +size_t size =iov_iter_count (&msg->msg_iter);int to_copy, to_read, data_read, offset;u32 data_length, remaining_data_length, data_offset;int rc; +if (WARN_ON_ONCE (iov_iter_rw (&msg->msg_iter) ==WRITE )) +return -EINVAL ;/* It's a bug in upper layer to get there */ + again:/* * No need to hold the reassembly queue lock all the time as we are * the only one reading from the front of the queue. The transport * may add more entries to the back of the queue at the same time */ -log_read (INFO ,"size=%d info->reassembly_data_length=%d\n" , size, +log_read (INFO ,"size=%zd info->reassembly_data_length=%d\n" , size, info->reassembly_data_length);if (info->reassembly_data_length >= size) {int queue_length;
@@ -1836,7 +1840,10 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,if (response->first_segment && size ==4 ) {unsigned int rfc1002_len = data_length + remaining_data_length; - *((__be32 *)buf) =cpu_to_be32 (rfc1002_len); +__be32 rfc1002_hdr =cpu_to_be32 (rfc1002_len); +if (copy_to_iter (&rfc1002_hdr,sizeof (rfc1002_hdr), + &msg->msg_iter) !=sizeof (rfc1002_hdr)) +return -EFAULT ; data_read =4 ; response->first_segment = false; log_read(INFO, "returning rfc1002 length %d\n",
@@ -1845,10 +1852,9 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, } to_copy =min_t (int, data_length - offset, to_read); -memcpy ( - buf + data_read, - (char *)data_transfer + data_offset + offset, - to_copy); +if (copy_to_iter ((char *)data_transfer + data_offset + offset, + to_copy, &msg->msg_iter) != to_copy) +return -EFAULT ;/* move on to the next buffer? */ if (to_copy == data_length - offset) {
@@ -1893,6 +1899,8 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, data_read, info->reassembly_data_length, info->first_entry_offset); read_rfc1002_done: +/* SMBDirect will read it all or nothing */ + msg->msg_iter.count =0 ;return data_read; }
@@ -1913,90 +1921,6 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, goto again; } -/* -* Receive a page from receive reassembly queue -* page: the page to read data into -* to_read: the length of data to read -* return value: actual data read -*/ -static int smbd_recv_page (struct smbd_connection *info, -struct page *page,unsigned int page_offset, -unsigned int to_read) -{ -struct smbdirect_socket *sc = &info->socket; -int ret; -char *to_address; -void *page_address; - -/* make sure we have the page ready for read */ - ret =wait_event_interruptible ( - info->wait_reassembly_queue, - info->reassembly_data_length >= to_read || - sc->status !=SMBDIRECT_SOCKET_CONNECTED ); -if (ret) -return ret; - -/* now we can read from reassembly queue and not sleep */ - page_address =kmap_atomic (page); - to_address = (char *) page_address + page_offset; - -log_read (INFO ,"reading from page=%p address=%p to_read=%d\n" , - page, to_address, to_read); - - ret =smbd_recv_buf (info, to_address, to_read); -kunmap_atomic (page_address); - -return ret; -} - -/* -* Receive data from transport -* msg: a msghdr point to the buffer, can be ITER_KVEC or ITER_BVEC -* return: total bytes read, or 0. SMB Direct will not do partial read. -*/ -int smbd_recv (struct smbd_connection *info,struct msghdr *msg) -{ -char *buf; -struct page *page; -unsigned int to_read, page_offset; -int rc; - -if (iov_iter_rw (&msg->msg_iter) ==WRITE ) { -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg iter dir %u\n" , -iov_iter_rw (&msg->msg_iter)); - rc = -EINVAL ; - goto out; - } - -switch (iov_iter_type (&msg->msg_iter)) { -case ITER_KVEC : - buf = msg->msg_iter.kvec->iov_base; - to_read = msg->msg_iter.kvec->iov_len; - rc =smbd_recv_buf (info, buf, to_read); -break ; - -case ITER_BVEC : - page = msg->msg_iter.bvec->bv_page; - page_offset = msg->msg_iter.bvec->bv_offset; - to_read = msg->msg_iter.bvec->bv_len; - rc =smbd_recv_page (info, page, page_offset, to_read); -break ; - -default : -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg type %d\n" , -iov_iter_type (&msg->msg_iter)); - rc = -EINVAL ; - } - -out: -/* SMBDirect will read it all or nothing */ -if (rc >0 ) - msg->msg_iter.count =0 ; -return rc; -} -/* * Send data to transport * Each rqst is transported as a SMBDirect payload
Stefan Metzmacher
reviewerRe: [PATCH v2 12/16] cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
Am 25.06.25 um 18:42 schrieb David Howells:
Quoted 99 lines
When performing a file read from RDMA,
smbd_recv()prints an "Invalid msg type 4" error and fails the I/O. This is due to the switch-statement there not handling theITER_FOLIOQhanded down from netfslib.Fix this by collapsing
smbd_recv_buf()andsmbd_recv_page()intosmbd_recv()and just usingcopy_to_iter()instead ofmemcpy(). This future-proofs the function too, in case more ITER_* types are added.Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")Reported-by Stefan Metzmacher <[email protected]>Signed-off-by David Howells <[email protected]>cc Steve French <[email protected]>cc Tom Talpey <[email protected]>cc Paulo Alcantara (Red Hat) <[email protected]>cc Matthew Wilcox <[email protected]>--- fs/smb/client/smbdirect.c | 114 +++++++------------------------------- 1 file changed, 19 insertions(+), 95 deletions(-)fs/smb/client/smbdirect.c +17
-11@@ -1770,35 +1770,39 @@ struct smbd_connection *smbd_get_connection( }/* -* Receive data from receive reassembly queue +* Receive data from the transport's receive reassembly queue * All the incoming data packets are placed in reassembly queue -* buf: the buffer to read data into +* iter: the buffer to read data into * size: the length of data to read * return value: actual data read -* Note: this implementation copies the data from reassebmly queue to receive +* +* Note: this implementation copies the data from reassembly queue to receive * buffers used by upper layer. This is not the optimal code path. A better way * to do it is to not have upper layer allocate its receive buffers but rather * borrow the buffer from reassembly queue, and return it after data is * consumed. But this will require more changes to upper layer code, and also * need to consider packet boundaries while they still being reassembled. */ -static int smbd_recv_buf (struct smbd_connection *info,char *buf, -unsigned int size) +int smbd_recv (struct smbd_connection *info,struct msghdr *msg) {struct smbdirect_socket *sc = &info->socket;struct smbd_response *response;struct smbdirect_data_transfer *data_transfer; +size_t size =iov_iter_count (&msg->msg_iter);int to_copy, to_read, data_read, offset;u32 data_length, remaining_data_length, data_offset;int rc; +if (WARN_ON_ONCE (iov_iter_rw (&msg->msg_iter) ==WRITE )) +return -EINVAL ;/* It's a bug in upper layer to get there */ + again:/* * No need to hold the reassembly queue lock all the time as we are * the only one reading from the front of the queue. The transport * may add more entries to the back of the queue at the same time */ -log_read (INFO ,"size=%d info->reassembly_data_length=%d\n" , size, +log_read (INFO ,"size=%zd info->reassembly_data_length=%d\n" , size, info->reassembly_data_length);if (info->reassembly_data_length >= size) {int queue_length;@@ -1836,7 +1840,10 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,if (response->first_segment && size ==4 ) {unsigned int rfc1002_len = data_length + remaining_data_length; - *((__be32 *)buf) =cpu_to_be32 (rfc1002_len); +__be32 rfc1002_hdr =cpu_to_be32 (rfc1002_len); +if (copy_to_iter (&rfc1002_hdr,sizeof (rfc1002_hdr), + &msg->msg_iter) !=sizeof (rfc1002_hdr)) +return -EFAULT ; data_read =4 ; response->first_segment = false; log_read(INFO, "returning rfc1002 length %d\n",@@ -1845,10 +1852,9 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, } to_copy =min_t (int, data_length - offset, to_read); -memcpy ( - buf + data_read, - (char *)data_transfer + data_offset + offset, - to_copy); +if (copy_to_iter ((char *)data_transfer + data_offset + offset, + to_copy, &msg->msg_iter) != to_copy) +return -EFAULT ;/* move on to the next buffer? */ if (to_copy == data_length - offset) {
fs/smb/client/smbdirect.c static int smbd_recv_buf(struct smbd_connection *info, char *buf, · line 1903
Quoted 5 lines
fs/smb/client/smbdirect.c +2
-0@@ -1893,6 +1899,8 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, data_read, info->reassembly_data_length, info->first_entry_offset); read_rfc1002_done: +/* SMBDirect will read it all or nothing */ + msg->msg_iter.count =0 ;
I think we should be remove this.
And I think this patch should come after the CONFIG_HARDENED_USERCOPY change otherwise a bisect will trigger the problem.
metze
David Howells
maintainerPATCH v3 12/16 cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
Quoted 8 lines · Stefan Metzmacher
Review
Quoted 3 lines
read_rfc1002_done:(quoted diff) +2
-0+ /* SMBDirect will read it all or nothing */ + msg->msg_iter.count = 0;I think we should be remove this.
And I think this patch should come after the
CONFIG_HARDENED_USERCOPYchange otherwise a bisect will trigger the problem.
Okay, done. I've attached the revised version here. I've also pushed it to my git branch and switched patches 12 & 13 there.
David
---
cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
When performing a file read from RDMA, smbd_recv() prints an "Invalid msg type 4" error and fails the I/O. This is due to the switch-statement there not handling the ITER_FOLIOQ handed down from netfslib.
Fix this by collapsing smbd_recv_buf() and smbd_recv_page() into smbd_recv() and just using copy_to_iter() instead of memcpy(). This future-proofs the function too, in case more ITER_* types are added.
--- fs/smb/client/smbdirect.c | 112 ++++++---------------------------------------- 1 file changed, 17 insertions(+), 95 deletions(-)
fs/smb/client/smbdirect.c +17 -95
@@ -1778,35 +1778,39 @@ struct smbd_connection *smbd_get_connection( }/* -* Receive data from receive reassembly queue +* Receive data from the transport's receive reassembly queue * All the incoming data packets are placed in reassembly queue -* buf: the buffer to read data into +* iter: the buffer to read data into * size: the length of data to read * return value: actual data read -* Note: this implementation copies the data from reassebmly queue to receive +* +* Note: this implementation copies the data from reassembly queue to receive * buffers used by upper layer. This is not the optimal code path. A better way * to do it is to not have upper layer allocate its receive buffers but rather * borrow the buffer from reassembly queue, and return it after data is * consumed. But this will require more changes to upper layer code, and also * need to consider packet boundaries while they still being reassembled. */ -static int smbd_recv_buf (struct smbd_connection *info,char *buf, -unsigned int size) +int smbd_recv (struct smbd_connection *info,struct msghdr *msg) {struct smbdirect_socket *sc = &info->socket;struct smbd_response *response;struct smbdirect_data_transfer *data_transfer; +size_t size =iov_iter_count (&msg->msg_iter);int to_copy, to_read, data_read, offset;u32 data_length, remaining_data_length, data_offset;int rc; +if (WARN_ON_ONCE (iov_iter_rw (&msg->msg_iter) ==WRITE )) +return -EINVAL ;/* It's a bug in upper layer to get there */ + again:/* * No need to hold the reassembly queue lock all the time as we are * the only one reading from the front of the queue. The transport * may add more entries to the back of the queue at the same time */ -log_read (INFO ,"size=%d info->reassembly_data_length=%d\n" , size, +log_read (INFO ,"size=%zd info->reassembly_data_length=%d\n" , size, info->reassembly_data_length);if (info->reassembly_data_length >= size) {int queue_length;
@@ -1844,7 +1848,10 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,if (response->first_segment && size ==4 ) {unsigned int rfc1002_len = data_length + remaining_data_length; - *((__be32 *)buf) =cpu_to_be32 (rfc1002_len); +__be32 rfc1002_hdr =cpu_to_be32 (rfc1002_len); +if (copy_to_iter (&rfc1002_hdr,sizeof (rfc1002_hdr), + &msg->msg_iter) !=sizeof (rfc1002_hdr)) +return -EFAULT ; data_read =4 ; response->first_segment = false; log_read(INFO, "returning rfc1002 length %d\n",
@@ -1853,10 +1860,9 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, } to_copy =min_t (int, data_length - offset, to_read); -memcpy ( - buf + data_read, - (char *)data_transfer + data_offset + offset, - to_copy); +if (copy_to_iter ((char *)data_transfer + data_offset + offset, + to_copy, &msg->msg_iter) != to_copy) +return -EFAULT ;/* move on to the next buffer? */ if (to_copy == data_length - offset) {
@@ -1921,90 +1927,6 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, goto again; } -/* -* Receive a page from receive reassembly queue -* page: the page to read data into -* to_read: the length of data to read -* return value: actual data read -*/ -static int smbd_recv_page (struct smbd_connection *info, -struct page *page,unsigned int page_offset, -unsigned int to_read) -{ -struct smbdirect_socket *sc = &info->socket; -int ret; -char *to_address; -void *page_address; - -/* make sure we have the page ready for read */ - ret =wait_event_interruptible ( - info->wait_reassembly_queue, - info->reassembly_data_length >= to_read || - sc->status !=SMBDIRECT_SOCKET_CONNECTED ); -if (ret) -return ret; - -/* now we can read from reassembly queue and not sleep */ - page_address =kmap_atomic (page); - to_address = (char *) page_address + page_offset; - -log_read (INFO ,"reading from page=%p address=%p to_read=%d\n" , - page, to_address, to_read); - - ret =smbd_recv_buf (info, to_address, to_read); -kunmap_atomic (page_address); - -return ret; -} - -/* -* Receive data from transport -* msg: a msghdr point to the buffer, can be ITER_KVEC or ITER_BVEC -* return: total bytes read, or 0. SMB Direct will not do partial read. -*/ -int smbd_recv (struct smbd_connection *info,struct msghdr *msg) -{ -char *buf; -struct page *page; -unsigned int to_read, page_offset; -int rc; - -if (iov_iter_rw (&msg->msg_iter) ==WRITE ) { -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg iter dir %u\n" , -iov_iter_rw (&msg->msg_iter)); - rc = -EINVAL ; - goto out; - } - -switch (iov_iter_type (&msg->msg_iter)) { -case ITER_KVEC : - buf = msg->msg_iter.kvec->iov_base; - to_read = msg->msg_iter.kvec->iov_len; - rc =smbd_recv_buf (info, buf, to_read); -break ; - -case ITER_BVEC : - page = msg->msg_iter.bvec->bv_page; - page_offset = msg->msg_iter.bvec->bv_offset; - to_read = msg->msg_iter.bvec->bv_len; - rc =smbd_recv_page (info, page, page_offset, to_read); -break ; - -default : -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg type %d\n" , -iov_iter_type (&msg->msg_iter)); - rc = -EINVAL ; - } - -out: -/* SMBDirect will read it all or nothing */ -if (rc >0 ) - msg->msg_iter.count =0 ; -return rc; -} -/* * Send data to transport * Each rqst is transported as a SMBDirect payload
Stefan Metzmacher
reviewerRe: [PATCH v3 12/16] cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
Am 25.06.25 um 19:55 schrieb David Howells:
Quoted 8 lines
Quoted 6 lines · Stefan Metzmacher
Review
Quoted 3 lines
read_rfc1002_done:(quoted diff) +2
-0+ /* SMBDirect will read it all or nothing */ + msg->msg_iter.count = 0;I think we should be remove this.
Quoted 5 lines
Quoted 2 lines
And I think this patch should come after the
CONFIG_HARDENED_USERCOPYchange otherwise a bisect will trigger the problem.Okay, done. I've attached the revised version here. I've also pushed it to my git branch and switched patches 12 & 13 there.
reviewed-by and tested-by: Stefan Metzmacher <[email protected]>
Quoted 194 lines
David
---
cifs: Fix reading into anITER_FOLIOQfrom the smbdirect codeWhen performing a file read from RDMA,
smbd_recv()prints an "Invalid msg type 4" error and fails the I/O. This is due to the switch-statement there not handling theITER_FOLIOQhanded down from netfslib.Fix this by collapsing
smbd_recv_buf()andsmbd_recv_page()intosmbd_recv()and just usingcopy_to_iter()instead ofmemcpy(). This future-proofs the function too, in case more ITER_* types are added.Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")Reported-by Stefan Metzmacher <[email protected]>Signed-off-by David Howells <[email protected]>cc Steve French <[email protected]>cc Tom Talpey <[email protected]>cc Paulo Alcantara (Red Hat) <[email protected]>cc Matthew Wilcox <[email protected]>--- fs/smb/client/smbdirect.c | 112 ++++++---------------------------------------- 1 file changed, 17 insertions(+), 95 deletions(-)fs/smb/client/smbdirect.c +17
-95@@ -1778,35 +1778,39 @@ struct smbd_connection *smbd_get_connection( }/* -* Receive data from receive reassembly queue +* Receive data from the transport's receive reassembly queue * All the incoming data packets are placed in reassembly queue -* buf: the buffer to read data into +* iter: the buffer to read data into * size: the length of data to read * return value: actual data read -* Note: this implementation copies the data from reassebmly queue to receive +* +* Note: this implementation copies the data from reassembly queue to receive * buffers used by upper layer. This is not the optimal code path. A better way * to do it is to not have upper layer allocate its receive buffers but rather * borrow the buffer from reassembly queue, and return it after data is * consumed. But this will require more changes to upper layer code, and also * need to consider packet boundaries while they still being reassembled. */ -static int smbd_recv_buf (struct smbd_connection *info,char *buf, -unsigned int size) +int smbd_recv (struct smbd_connection *info,struct msghdr *msg) {struct smbdirect_socket *sc = &info->socket;struct smbd_response *response;struct smbdirect_data_transfer *data_transfer; +size_t size =iov_iter_count (&msg->msg_iter);int to_copy, to_read, data_read, offset;u32 data_length, remaining_data_length, data_offset;int rc; +if (WARN_ON_ONCE (iov_iter_rw (&msg->msg_iter) ==WRITE )) +return -EINVAL ;/* It's a bug in upper layer to get there */ + again:/* * No need to hold the reassembly queue lock all the time as we are * the only one reading from the front of the queue. The transport * may add more entries to the back of the queue at the same time */ -log_read (INFO ,"size=%d info->reassembly_data_length=%d\n" , size, +log_read (INFO ,"size=%zd info->reassembly_data_length=%d\n" , size, info->reassembly_data_length);if (info->reassembly_data_length >= size) {int queue_length;@@ -1844,7 +1848,10 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,if (response->first_segment && size ==4 ) {unsigned int rfc1002_len = data_length + remaining_data_length; - *((__be32 *)buf) =cpu_to_be32 (rfc1002_len); +__be32 rfc1002_hdr =cpu_to_be32 (rfc1002_len); +if (copy_to_iter (&rfc1002_hdr,sizeof (rfc1002_hdr), + &msg->msg_iter) !=sizeof (rfc1002_hdr)) +return -EFAULT ; data_read =4 ; response->first_segment = false; log_read(INFO, "returning rfc1002 length %d\n",@@ -1853,10 +1860,9 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, } to_copy =min_t (int, data_length - offset, to_read); -memcpy ( - buf + data_read, - (char *)data_transfer + data_offset + offset, - to_copy); +if (copy_to_iter ((char *)data_transfer + data_offset + offset, + to_copy, &msg->msg_iter) != to_copy) +return -EFAULT ;/* move on to the next buffer? */ if (to_copy == data_length - offset) {@@ -1921,90 +1927,6 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, goto again; } -/* -* Receive a page from receive reassembly queue -* page: the page to read data into -* to_read: the length of data to read -* return value: actual data read -*/ -static int smbd_recv_page (struct smbd_connection *info, -struct page *page,unsigned int page_offset, -unsigned int to_read) -{ -struct smbdirect_socket *sc = &info->socket; -int ret; -char *to_address; -void *page_address; - -/* make sure we have the page ready for read */ - ret =wait_event_interruptible ( - info->wait_reassembly_queue, - info->reassembly_data_length >= to_read || - sc->status !=SMBDIRECT_SOCKET_CONNECTED ); -if (ret) -return ret; - -/* now we can read from reassembly queue and not sleep */ - page_address =kmap_atomic (page); - to_address = (char *) page_address + page_offset; - -log_read (INFO ,"reading from page=%p address=%p to_read=%d\n" , - page, to_address, to_read); - - ret =smbd_recv_buf (info, to_address, to_read); -kunmap_atomic (page_address); - -return ret; -} - -/* -* Receive data from transport -* msg: a msghdr point to the buffer, can be ITER_KVEC or ITER_BVEC -* return: total bytes read, or 0. SMB Direct will not do partial read. -*/ -int smbd_recv (struct smbd_connection *info,struct msghdr *msg) -{ -char *buf; -struct page *page; -unsigned int to_read, page_offset; -int rc; - -if (iov_iter_rw (&msg->msg_iter) ==WRITE ) { -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg iter dir %u\n" , -iov_iter_rw (&msg->msg_iter)); - rc = -EINVAL ; - goto out; - } - -switch (iov_iter_type (&msg->msg_iter)) { -case ITER_KVEC : - buf = msg->msg_iter.kvec->iov_base; - to_read = msg->msg_iter.kvec->iov_len; - rc =smbd_recv_buf (info, buf, to_read); -break ; - -case ITER_BVEC : - page = msg->msg_iter.bvec->bv_page; - page_offset = msg->msg_iter.bvec->bv_offset; - to_read = msg->msg_iter.bvec->bv_len; - rc =smbd_recv_page (info, page, page_offset, to_read); -break ; - -default : -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg type %d\n" , -iov_iter_type (&msg->msg_iter)); - rc = -EINVAL ; - } - -out: -/* SMBDirect will read it all or nothing */ -if (rc >0 ) - msg->msg_iter.count =0 ; -return rc; -} -/* * Send data to transport * Each rqst is transported as a SMBDirect payload
Tom Talpey
reviewerRe: [PATCH v3 12/16] cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
In reply to David HowellsQuoted 91 lines · David Howells
Quoted 8 lines · Stefan Metzmacher
Review
Quoted 3 lines
read_rfc1002_done:(quoted diff) +2
-0+ /* SMBDirect will read it all or nothing */ + msg->msg_iter.count = 0;I think we should be remove this.
And I think this patch should come after the
CONFIG_HARDENED_USERCOPYchange otherwise a bisect will trigger the problem.Okay, done. I've attached the revised version here. I've also pushed it to my git branch and switched patches 12 & 13 there.
David
---
cifs: Fix reading into anITER_FOLIOQfrom the smbdirect codeWhen performing a file read from RDMA,
smbd_recv()prints an "Invalid msg type 4" error and fails the I/O. This is due to the switch-statement there not handling theITER_FOLIOQhanded down from netfslib.Fix this by collapsing
smbd_recv_buf()andsmbd_recv_page()intosmbd_recv()and just usingcopy_to_iter()instead ofmemcpy(). This future-proofs the function too, in case more ITER_* types are added.Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")Reported-by Stefan Metzmacher <[email protected]>Signed-off-by David Howells <[email protected]>cc Steve French <[email protected]>cc Tom Talpey <[email protected]>cc Paulo Alcantara (Red Hat) <[email protected]>cc Matthew Wilcox <[email protected]>--- fs/smb/client/smbdirect.c | 112 ++++++---------------------------------------- 1 file changed, 17 insertions(+), 95 deletions(-)fs/smb/client/smbdirect.c +10
-6@@ -1778,35 +1778,39 @@ struct smbd_connection *smbd_get_connection( }/* -* Receive data from receive reassembly queue +* Receive data from the transport's receive reassembly queue * All the incoming data packets are placed in reassembly queue -* buf: the buffer to read data into +* iter: the buffer to read data into * size: the length of data to read * return value: actual data read -* Note: this implementation copies the data from reassebmly queue to receive +* +* Note: this implementation copies the data from reassembly queue to receive * buffers used by upper layer. This is not the optimal code path. A better way * to do it is to not have upper layer allocate its receive buffers but rather * borrow the buffer from reassembly queue, and return it after data is * consumed. But this will require more changes to upper layer code, and also * need to consider packet boundaries while they still being reassembled. */ -static int smbd_recv_buf (struct smbd_connection *info,char *buf, -unsigned int size) +int smbd_recv (struct smbd_connection *info,struct msghdr *msg) {struct smbdirect_socket *sc = &info->socket;struct smbd_response *response;struct smbdirect_data_transfer *data_transfer; +size_t size =iov_iter_count (&msg->msg_iter);int to_copy, to_read, data_read, offset;u32 data_length, remaining_data_length, data_offset;int rc; +if (WARN_ON_ONCE (iov_iter_rw (&msg->msg_iter) ==WRITE )) +return -EINVAL ;/* It's a bug in upper layer to get there */ + again:/* * No need to hold the reassembly queue lock all the time as we are * the only one reading from the front of the queue. The transport * may add more entries to the back of the queue at the same time */ -log_read (INFO ,"size=%d info->reassembly_data_length=%d\n" , size, +log_read (INFO ,"size=%zd info->reassembly_data_length=%d\n" , size, info->reassembly_data_length);if (info->reassembly_data_length >= size) {int queue_length;
fs/smb/client/smbdirect.c static int smbd_recv_buf(struct smbd_connection *info, char *buf, · line 1854
Quoted 8 lines
fs/smb/client/smbdirect.c +4
-1@@ -1844,7 +1848,10 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,if (response->first_segment && size ==4 ) {unsigned int rfc1002_len = data_length + remaining_data_length; - *((__be32 *)buf) =cpu_to_be32 (rfc1002_len); +__be32 rfc1002_hdr =cpu_to_be32 (rfc1002_len); +if (copy_to_iter (&rfc1002_hdr,sizeof (rfc1002_hdr), + &msg->msg_iter) !=sizeof (rfc1002_hdr)) +return -EFAULT ;
Shouldn't there be some kind of validity check on the rfc1002 length field before this? For example, the high octet of that field is required to be zero (by SMB) and the 24-bit length is not necessarily checked yet. The original code just returned the decoded value but this sticks it in the msg_iter. If that's safe, then ok but it seems odd.
Tom.
Quoted 110 lines
fs/smb/client/smbdirect.c +3
-88data_read =4 ; response->first_segment = false; log_read(INFO, "returning rfc1002 length %d\n",@@ -1853,10 +1860,9 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, } to_copy =min_t (int, data_length - offset, to_read); -memcpy ( - buf + data_read, - (char *)data_transfer + data_offset + offset, - to_copy); +if (copy_to_iter ((char *)data_transfer + data_offset + offset, + to_copy, &msg->msg_iter) != to_copy) +return -EFAULT ;/* move on to the next buffer? */ if (to_copy == data_length - offset) {@@ -1921,90 +1927,6 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf, goto again; } -/* -* Receive a page from receive reassembly queue -* page: the page to read data into -* to_read: the length of data to read -* return value: actual data read -*/ -static int smbd_recv_page (struct smbd_connection *info, -struct page *page,unsigned int page_offset, -unsigned int to_read) -{ -struct smbdirect_socket *sc = &info->socket; -int ret; -char *to_address; -void *page_address; - -/* make sure we have the page ready for read */ - ret =wait_event_interruptible ( - info->wait_reassembly_queue, - info->reassembly_data_length >= to_read || - sc->status !=SMBDIRECT_SOCKET_CONNECTED ); -if (ret) -return ret; - -/* now we can read from reassembly queue and not sleep */ - page_address =kmap_atomic (page); - to_address = (char *) page_address + page_offset; - -log_read (INFO ,"reading from page=%p address=%p to_read=%d\n" , - page, to_address, to_read); - - ret =smbd_recv_buf (info, to_address, to_read); -kunmap_atomic (page_address); - -return ret; -} - -/* -* Receive data from transport -* msg: a msghdr point to the buffer, can be ITER_KVEC or ITER_BVEC -* return: total bytes read, or 0. SMB Direct will not do partial read. -*/ -int smbd_recv (struct smbd_connection *info,struct msghdr *msg) -{ -char *buf; -struct page *page; -unsigned int to_read, page_offset; -int rc; - -if (iov_iter_rw (&msg->msg_iter) ==WRITE ) { -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg iter dir %u\n" , -iov_iter_rw (&msg->msg_iter)); - rc = -EINVAL ; - goto out; - } - -switch (iov_iter_type (&msg->msg_iter)) { -case ITER_KVEC : - buf = msg->msg_iter.kvec->iov_base; - to_read = msg->msg_iter.kvec->iov_len; - rc =smbd_recv_buf (info, buf, to_read); -break ; - -case ITER_BVEC : - page = msg->msg_iter.bvec->bv_page; - page_offset = msg->msg_iter.bvec->bv_offset; - to_read = msg->msg_iter.bvec->bv_len; - rc =smbd_recv_page (info, page, page_offset, to_read); -break ; - -default : -/* It's a bug in upper layer to get there */ -cifs_dbg (VFS ,"Invalid msg type %d\n" , -iov_iter_type (&msg->msg_iter)); - rc = -EINVAL ; - } - -out: -/* SMBDirect will read it all or nothing */ -if (rc >0 ) - msg->msg_iter.count =0 ; -return rc; -} -/* * Send data to transport * Each rqst is transported as a SMBDirect payload
David Howells
maintainerRe: [PATCH v3 12/16] cifs: Fix reading into an ITER_FOLIOQ from the smbdirect code
Quoted 7 lines · Tom Talpey
Shouldn't there be some kind of validity check on the rfc1002 length field before this? For example, the high octet of that field is required to be zero (by SMB) and the 24-bit length is not necessarily checked yet. The original code just returned the decoded value but this sticks it in the
msg_iter. If that's safe, then ok but it seems odd.
That should be a separate bugfix, I think.
David
David Howells
maintainerPATCH v2 13/16 cifs: Fix the smbd_reponse slab to allow usercopy
In reply to David HowellsThe handling of received data in the smbdirect client code involves using copy_to_iter() to copy data from the smbd_reponse struct's packet trailer to a folioq buffer provided by netfslib that encapsulates a chunk of pagecache.
If, however, CONFIG_HARDENED_USERCOPY=y, this will result in the checks then performed in copy_to_iter() oopsing with something like the following:
CIFS: Attempting to mount //172.31.9.1/test
CIFS: VFS: RDMA transport established
usercopy: Kernel memory exposure attempt detected from SLUB object 'smbd_response_0000000091e24ea1' (offset 81, size 63)!
------------[ cut here ]------------ kernel BUG at mm/usercopy.c:102! ... RIP: 0010:usercopy_abort+0x6c/0x80 ... Call Trace: <TASK> __check_heap_object+0xe3/0x120 __check_object_size+0x4dc/0x6d0 smbd_recv+0x77f/0xfe0 [cifs] cifs_readv_from_socket+0x276/0x8f0 [cifs] cifs_read_from_socket+0xcd/0x120 [cifs] cifs_demultiplex_thread+0x7e9/0x2d50 [cifs] kthread+0x396/0x830 ret_from_fork+0x2b8/0x3b0 ret_from_fork_asm+0x1a/0x30
The problem is that the smbd_response slab's packet field isn't marked as being permitted for usercopy.
Fix this by passing parameters to kmem_slab_create() to indicate that copy_to_iter() is permitted from the packet region of the smbd_response slab objects, less the header space.
--- fs/smb/client/smbdirect.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
fs/smb/client/smbdirect.c +13 -5
@@ -1475,6 +1475,9 @@ static int allocate_caches_and_workqueue(struct smbd_connection *info)char name[MAX_NAME_LEN ];int rc; +if (WARN_ON_ONCE (sp->max_recv_size <sizeof (struct smbdirect_data_transfer ))) +return -ENOMEM ; +scnprintf (name,MAX_NAME_LEN ,"smbd_request_%p" , info); info->request_cache = kmem_cache_create(
@@ -1492,12 +1495,17 @@ static int allocate_caches_and_workqueue(struct smbd_connection *info) goto out1;scnprintf (name,MAX_NAME_LEN ,"smbd_response_%p" , info); + +struct kmem_cache_args response_args = { + .align = __alignof__(struct smbd_response ), + .useroffset = (offsetof(struct smbd_response , packet) + +sizeof (struct smbdirect_data_transfer )), + .usersize = sp->max_recv_size -sizeof (struct smbdirect_data_transfer ), + }; info->response_cache = -kmem_cache_create ( - name, -sizeof (struct smbd_response ) + - sp->max_recv_size, -0 ,SLAB_HWCACHE_ALIGN ,NULL ); +kmem_cache_create (name, +sizeof (struct smbd_response ) + sp->max_recv_size, + &response_args,SLAB_HWCACHE_ALIGN );if (!info->response_cache) goto out2;
David Howells
maintainerPATCH v2 14/16 smb: client: fix potential deadlock when reconnecting channels
In reply to David HowellsFrom: Paulo Alcantara <[email protected]>
Fix cifs_signal_cifsd_for_reconnect() to take the correct lock order and prevent the following deadlock from happening
======================================================
WARNING: possible circular locking dependency detected
6.16.0-rc3-build2+ #1301 Tainted: G S W
------------------------------------------------------
cifsd/6055 is trying to acquire lock:ffff88810ad56038 (&tcp_ses->srv_lock){+.+.}-{3:3}, at: cifs_signal_cifsd_for_reconnect+0x134/0x200
but task is already holding lock:ffff888119c64330 (&ret_buf->chan_lock){+.+.}-{3:3}, at: cifs_signal_cifsd_for_reconnect+0xcf/0x200
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (&ret_buf->chan_lock){+.+.}-{3:3}:
validate_chain+0x1cf/0x270
__lock_acquire+0x60e/0x780
lock_acquire.part.0+0xb4/0x1f0
_raw_spin_lock+0x2f/0x40
cifs_setup_session+0x81/0x4b0
cifs_get_smb_ses+0x771/0x900
cifs_mount_get_session+0x7e/0x170
cifs_mount+0x92/0x2d0
cifs_smb3_do_mount+0x161/0x460
smb3_get_tree+0x55/0x90
vfs_get_tree+0x46/0x180
do_new_mount+0x1b0/0x2e0
path_mount+0x6ee/0x740
do_mount+0x98/0xe0
__do_sys_mount+0x148/0x180
do_syscall_64+0xa4/0x260
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #1 (&ret_buf->ses_lock){+.+.}-{3:3}:
validate_chain+0x1cf/0x270
__lock_acquire+0x60e/0x780
lock_acquire.part.0+0xb4/0x1f0
_raw_spin_lock+0x2f/0x40
cifs_match_super+0x101/0x320
sget+0xab/0x270
cifs_smb3_do_mount+0x1e0/0x460
smb3_get_tree+0x55/0x90
vfs_get_tree+0x46/0x180
do_new_mount+0x1b0/0x2e0
path_mount+0x6ee/0x740
do_mount+0x98/0xe0
__do_sys_mount+0x148/0x180
do_syscall_64+0xa4/0x260
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #0 (&tcp_ses->srv_lock){+.+.}-{3:3}:
check_noncircular+0x95/0xc0
check_prev_add+0x115/0x2f0
validate_chain+0x1cf/0x270
__lock_acquire+0x60e/0x780
lock_acquire.part.0+0xb4/0x1f0
_raw_spin_lock+0x2f/0x40
cifs_signal_cifsd_for_reconnect+0x134/0x200
__cifs_reconnect+0x8f/0x500
cifs_handle_standard+0x112/0x280
cifs_demultiplex_thread+0x64d/0xbc0
kthread+0x2f7/0x310
ret_from_fork+0x2a/0x230
ret_from_fork_asm+0x1a/0x30other info that might help us debug this:
Chain exists of: &tcp_ses->srv_lock --> &ret_buf->ses_lock --> &ret_buf->chan_lock Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&ret_buf->chan_lock);
lock(&ret_buf->ses_lock);
lock(&ret_buf->chan_lock); lock(&tcp_ses->srv_lock);
*** DEADLOCK ***
3 locks held by cifsd/6055:
#0: ffffffff857de398 (&cifs_tcp_ses_lock){+.+.}-{3:3}, at: cifs_signal_cifsd_for_reconnect+0x7b/0x200 #1: ffff888119c64060 (&ret_buf->ses_lock){+.+.}-{3:3}, at: cifs_signal_cifsd_for_reconnect+0x9c/0x200 #2: ffff888119c64330 (&ret_buf->chan_lock){+.+.}-{3:3}, at: cifs_signal_cifsd_for_reconnect+0xcf/0x200
--- fs/smb/client/cifsglob.h | 1 + fs/smb/client/connect.c | 53 +++++++++++++++++++++++----------------- 2 files changed, 32 insertions(+), 22 deletions(-)
fs/smb/client/cifsglob.h +1 -0
@@ -709,6 +709,7 @@ inc_rfc1001_len(void *buf, int count)struct TCP_Server_Info {struct list_head tcp_ses_list;struct list_head smb_ses_list; +struct list_head rlist;/* reconnect list */ spinlock_t srv_lock;/* protect anything here that is not protected */ __u64 conn_id;/* connection identifier (useful for debugging) */ int srv_count;/* reference counter */
fs/smb/client/connect.c +31 -22
76 lines not on this page · raw message
David Howells
maintainerPATCH v2 15/16 netfs: Renumber the NETFS_RREQ_* flags to make traces easier to read
In reply to David HowellsRenumber the NETFS_RREQ_* flags to put the most useful status bits in the bottom nibble - and therefore the last hex digit in the trace output - making it easier to grasp the state at a glance.
In particular, put the IN_PROGRESS flag in bit 0 and ALL_QUEUED at bit 1.
Also make the flags field in /proc/fs/netfs/requests larger to accommodate all the flags.
Also make the flags field in the netfs_sreq tracepoint larger to accommodate all the NETFS_SREQ_* flags.
--- fs/netfs/main.c | 6 +++--- include/linux/netfs.h | 20 ++++++++++---------- include/trace/events/netfs.h | 2 +- 3 files changed, 14 insertions(+), 14 deletions(-)
fs/netfs/main.c +3 -3
@@ -58,15 +58,15 @@ static int netfs_requests_seq_show(struct seq_file *m, void *v)if (v == &netfs_io_requests) {seq_puts (m, -"REQUEST OR REF FL ERR OPS COVERAGE\n" -"======== == === == ==== === =========\n" +"REQUEST OR REF FLAG ERR OPS COVERAGE\n" +"======== == === ==== ==== === =========\n" );return 0 ; } rreq = list_entry(v, struct netfs_io_request, proc_link); seq_printf(m, - "%08x %s %3d %2lx %4ld %3d @%04llx %llx/%llx", + "%08x %s %3d %4lx %4ld %3d @%04llx %llx/%llx", rreq->debug_id, netfs_origins[rreq->origin], refcount_read(&rreq->ref),
include/linux/netfs.h +10 -10
@@ -265,17 +265,17 @@ struct netfs_io_request {bool direct_bv_unpin;/* T if direct_bv[] must be unpinned */ refcount_t ref;unsigned long flags; -#define NETFS_RREQ_OFFLOAD_COLLECTION 0/* Offload collection to workqueue */ -#define NETFS_RREQ_NO_UNLOCK_FOLIO 2/* Don't unlock no_unlock_folio on completion */ -#define NETFS_RREQ_FAILED 4/* The request failed */ -#define NETFS_RREQ_IN_PROGRESS 5/* Unlocked when the request completes (has ref) */ -#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 6/* Copy current folio to cache from read */ -#define NETFS_RREQ_UPLOAD_TO_SERVER 8/* Need to write to the server */ -#define NETFS_RREQ_PAUSE 11/* Pause subrequest generation */ +#define NETFS_RREQ_IN_PROGRESS 0/* Unlocked when the request completes (has ref) */ +#define NETFS_RREQ_ALL_QUEUED 1/* All subreqs are now queued */ +#define NETFS_RREQ_PAUSE 2/* Pause subrequest generation */ +#define NETFS_RREQ_FAILED 3/* The request failed */ +#define NETFS_RREQ_RETRYING 4/* Set if we're in the retry path */ +#define NETFS_RREQ_SHORT_TRANSFER 5/* Set if we have a short transfer */ +#define NETFS_RREQ_OFFLOAD_COLLECTION 8/* Offload collection to workqueue */ +#define NETFS_RREQ_NO_UNLOCK_FOLIO 9/* Don't unlock no_unlock_folio on completion */ +#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 10/* Copy current folio to cache from read */ +#define NETFS_RREQ_UPLOAD_TO_SERVER 11/* Need to write to the server */ #define NETFS_RREQ_USE_IO_ITER 12/* Use ->io_iter rather than ->i_pages */ -#define NETFS_RREQ_ALL_QUEUED 13/* All subreqs are now queued */ -#define NETFS_RREQ_RETRYING 14/* Set if we're in the retry path */ -#define NETFS_RREQ_SHORT_TRANSFER 15/* Set if we have a short transfer */ #define NETFS_RREQ_USE_PGPRIV2 31/* [DEPRECATED] Use PG_private_2 to mark * write to cache on read */ const struct netfs_request_ops *netfs_ops;
include/trace/events/netfs.h +1 -1
@@ -367,7 +367,7 @@ TRACE_EVENT(netfs_sreq, __entry->slot = sreq->io_iter.folioq_slot; ), - TP_printk("R=%08x[%x] %s %s f=%02x s=%llx %zx/%zx s=%u e=%d", + TP_printk("R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx s=%u e=%d", __entry->rreq, __entry->index, __print_symbolic(__entry->source, netfs_sreq_sources), __print_symbolic(__entry->what, netfs_sreq_traces),
David Howells
maintainerPATCH v2 16/16 netfs: Update tracepoints in a number of ways
In reply to David HowellsMake a number of updates to the netfs tracepoints:
(1) Remove a duplicate trace from netfs_unbuffered_write_iter_locked().
(2) Move the trace in netfs_wake_rreq_flag() to after the flag is cleared so that the change appears in the trace.
(3) Differentiate the use of netfs_rreq_trace_wait/woke_queue symbols.
(4) Don't do so many trace emissions in the wait functions as some of them are redundant.
(5) In netfs_collect_read_results(), differentiate a subreq that's being abandoned vs one that has been consumed in a regular way.
(6) Add a tracepoint to indicate the call to ->ki_complete().
(7) Don't double-increment the subreq_counter when retrying a write.
(8) Move the netfs_sreq_trace_io_progress tracepoint within cifs code to just MID_RESPONSE_RECEIVED and add different tracepoints for other MID states and note check failure.
--- fs/netfs/direct_write.c | 1 - fs/netfs/internal.h | 2 +- fs/netfs/misc.c | 14 ++++++-------- fs/netfs/read_collect.c | 12 +++++++++--- fs/netfs/write_collect.c | 4 +++- fs/netfs/write_retry.c | 1 - fs/smb/client/cifssmb.c | 20 ++++++++++++++++++++ fs/smb/client/smb2pdu.c | 26 ++++++++++++++++++++++---- include/trace/events/netfs.h | 26 ++++++++++++++++++-------- 9 files changed, 79 insertions(+), 27 deletions(-)
fs/netfs/direct_write.c +0 -1
@@ -106,7 +106,6 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter * }if (!async) { -trace_netfs_rreq (wreq, netfs_rreq_trace_wait_ip); ret =netfs_wait_for_write (wreq);if (ret >0 ) iocb->ki_pos += ret;
fs/netfs/internal.h +1 -1
@@ -267,9 +267,9 @@ static inline void netfs_wake_rreq_flag(struct netfs_io_request *rreq, enum netfs_rreq_trace trace) {if (test_bit (rreq_flag, &rreq->flags)) { -trace_netfs_rreq (rreq, trace);clear_bit_unlock (rreq_flag, &rreq->flags);smp_mb__after_atomic ();/* Set flag before task state */ +trace_netfs_rreq (rreq, trace);wake_up (&rreq->waitq); } }
fs/netfs/misc.c +6 -8
@@ -359,7 +359,7 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,if (!netfs_check_subreq_in_progress (subreq))continue ; -trace_netfs_rreq (rreq, netfs_rreq_trace_wait_queue); +trace_netfs_rreq (rreq, netfs_rreq_trace_wait_quiesce);for (;;) {prepare_to_wait (&rreq->waitq, &myself,TASK_UNINTERRUPTIBLE );
@@ -368,10 +368,10 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,trace_netfs_sreq (subreq, netfs_sreq_trace_wait_for);schedule (); -trace_netfs_rreq (rreq, netfs_rreq_trace_woke_queue); } } +trace_netfs_rreq (rreq, netfs_rreq_trace_waited_quiesce);finish_wait (&rreq->waitq, &myself); }
@@ -437,7 +437,6 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,ssize_t ret;for (;;) { -trace_netfs_rreq (rreq, netfs_rreq_trace_wait_queue);prepare_to_wait (&rreq->waitq, &myself,TASK_UNINTERRUPTIBLE );if (!test_bit (NETFS_RREQ_OFFLOAD_COLLECTION , &rreq->flags)) {
@@ -457,11 +456,12 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,if (!netfs_check_rreq_in_progress (rreq))break ; +trace_netfs_rreq (rreq, netfs_rreq_trace_wait_ip);schedule (); -trace_netfs_rreq (rreq, netfs_rreq_trace_woke_queue); } all_collected: +trace_netfs_rreq (rreq, netfs_rreq_trace_waited_ip);finish_wait (&rreq->waitq, &myself); ret = rreq->error;
@@ -504,10 +504,8 @@ static void netfs_wait_for_pause(struct netfs_io_request *rreq, {DEFINE_WAIT (myself); -trace_netfs_rreq (rreq, netfs_rreq_trace_wait_pause); -for (;;) { -trace_netfs_rreq (rreq, netfs_rreq_trace_wait_queue); +trace_netfs_rreq (rreq, netfs_rreq_trace_wait_pause);prepare_to_wait (&rreq->waitq, &myself,TASK_UNINTERRUPTIBLE );if (!test_bit (NETFS_RREQ_OFFLOAD_COLLECTION , &rreq->flags)) {
@@ -530,10 +528,10 @@ static void netfs_wait_for_pause(struct netfs_io_request *rreq,break ;schedule (); -trace_netfs_rreq (rreq, netfs_rreq_trace_woke_queue); } all_collected: +trace_netfs_rreq (rreq, netfs_rreq_trace_waited_pause);finish_wait (&rreq->waitq, &myself); }
fs/netfs/read_collect.c +9 -3
@@ -293,7 +293,9 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)spin_lock (&rreq->lock); remove = front; -trace_netfs_sreq (front, netfs_sreq_trace_discard); +trace_netfs_sreq (front, + notes &ABANDON_SREQ ? + netfs_sreq_trace_abandoned : netfs_sreq_trace_consumed);list_del_init (&front->rreq_link); front =list_first_entry_or_null (&stream->subrequests, struct netfs_io_subrequest, rreq_link);
@@ -353,9 +355,11 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)if (rreq->iocb) { rreq->iocb->ki_pos += rreq->transferred; -if (rreq->iocb->ki_complete) +if (rreq->iocb->ki_complete) { +trace_netfs_rreq (rreq, netfs_rreq_trace_ki_complete); rreq->iocb->ki_complete ( rreq->iocb, rreq->error ? rreq->error : rreq->transferred); + } }if (rreq->netfs_ops->done) rreq->netfs_ops->done (rreq);
@@ -379,9 +383,11 @@ static void netfs_rreq_assess_single(struct netfs_io_request *rreq)if (rreq->iocb) { rreq->iocb->ki_pos += rreq->transferred; -if (rreq->iocb->ki_complete) +if (rreq->iocb->ki_complete) { +trace_netfs_rreq (rreq, netfs_rreq_trace_ki_complete); rreq->iocb->ki_complete ( rreq->iocb, rreq->error ? rreq->error : rreq->transferred); + } }if (rreq->netfs_ops->done) rreq->netfs_ops->done (rreq);
fs/netfs/write_collect.c +3 -1
@@ -419,9 +419,11 @@ bool netfs_write_collection(struct netfs_io_request *wreq)if (wreq->iocb) {size_t written =min (wreq->transferred, wreq->len); wreq->iocb->ki_pos += written; -if (wreq->iocb->ki_complete) +if (wreq->iocb->ki_complete) { +trace_netfs_rreq (wreq, netfs_rreq_trace_ki_complete); wreq->iocb->ki_complete ( wreq->iocb, wreq->error ? wreq->error : written); + } wreq->iocb =VFS_PTR_POISON ; }
fs/netfs/write_retry.c +0 -1
@@ -146,7 +146,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, subreq =netfs_alloc_subrequest (wreq); subreq->source = to->source; subreq->start = start; - subreq->debug_index =atomic_inc_return (&wreq->subreq_counter); subreq->stream_nr = to->stream_nr; subreq->retry_count =1 ;
fs/smb/client/cifssmb.c +20 -0
@@ -1334,7 +1334,11 @@ cifs_readv_callback(struct mid_q_entry *mid)cifs_stats_bytes_read (tcon, rdata->got_bytes);break ;case MID_REQUEST_SUBMITTED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_req_submitted); + goto do_retry;case MID_RETRY_NEEDED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_retry_needed); +do_retry:__set_bit (NETFS_SREQ_NEED_RETRY , &rdata->subreq.flags); rdata->result = -EAGAIN ;if (server->sign && rdata->got_bytes)
@@ -1344,8 +1348,14 @@ cifs_readv_callback(struct mid_q_entry *mid)task_io_account_read (rdata->got_bytes);cifs_stats_bytes_read (tcon, rdata->got_bytes);break ; +case MID_RESPONSE_MALFORMED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_malformed); + rdata->result = -EIO ; +break ;default : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_unknown); rdata->result = -EIO ; +break ; }if (rdata->result == -ENODATA ) {
@@ -1714,11 +1724,21 @@ cifs_writev_callback(struct mid_q_entry *mid) }break ;case MID_REQUEST_SUBMITTED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_req_submitted); +__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); + result = -EAGAIN ; +break ;case MID_RETRY_NEEDED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_retry_needed);__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); result = -EAGAIN ;break ; +case MID_RESPONSE_MALFORMED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_malformed); + result = -EIO ; +break ;default : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_unknown); result = -EIO ;break ; }
fs/smb/client/smb2pdu.c +22 -4
@@ -4565,7 +4565,11 @@ smb2_readv_callback(struct mid_q_entry *mid)cifs_stats_bytes_read (tcon, rdata->got_bytes);break ;case MID_REQUEST_SUBMITTED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_req_submitted); + goto do_retry;case MID_RETRY_NEEDED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_retry_needed); +do_retry:__set_bit (NETFS_SREQ_NEED_RETRY , &rdata->subreq.flags); rdata->result = -EAGAIN ;if (server->sign && rdata->got_bytes)
@@ -4576,11 +4580,15 @@ smb2_readv_callback(struct mid_q_entry *mid)cifs_stats_bytes_read (tcon, rdata->got_bytes);break ;case MID_RESPONSE_MALFORMED : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_malformed); credits.value =le16_to_cpu (shdr->CreditRequest); credits.instance = server->reconnect_instance; - fallthrough; + rdata->result = -EIO ; +break ;default : +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_unknown); rdata->result = -EIO ; +break ; }#ifdef CONFIG_CIFS_SMB_DIRECT /*
@@ -4833,11 +4841,14 @@ smb2_writev_callback(struct mid_q_entry *mid)switch (mid->mid_state) {case MID_RESPONSE_RECEIVED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_progress); credits.value =le16_to_cpu (rsp->hdr.CreditRequest); credits.instance = server->reconnect_instance; result =smb2_check_receive (mid, server,0 ); -if (result !=0 ) +if (result !=0 ) { +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_bad);break ; + } written =le32_to_cpu (rsp->DataLength);/*
@@ -4859,15 +4870,23 @@ smb2_writev_callback(struct mid_q_entry *mid) }break ;case MID_REQUEST_SUBMITTED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_req_submitted); +__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); + result = -EAGAIN ; +break ;case MID_RETRY_NEEDED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_retry_needed);__set_bit (NETFS_SREQ_NEED_RETRY , &wdata->subreq.flags); result = -EAGAIN ;break ;case MID_RESPONSE_MALFORMED : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_malformed); credits.value =le16_to_cpu (rsp->hdr.CreditRequest); credits.instance = server->reconnect_instance; - fallthrough; + result = -EIO ; +break ;default : +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_unknown); result = -EIO ;break ; }
@@ -4907,7 +4926,6 @@ smb2_writev_callback(struct mid_q_entry *mid) server->credits, server->in_flight, 0, cifs_trace_rw_credits_write_response_clear); wdata->credits.value =0 ; -trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_progress);cifs_write_subrequest_terminated (wdata, result ?: written);release_mid (mid); trace_smb3_rw_credits(rreq_debug_id, subreq_debug_index, 0,
include/trace/events/netfs.h +18 -8
@@ -50,12 +50,13 @@#define netfs_rreq_traces \ EM(netfs_rreq_trace_assess, "ASSESS ") \ - EM(netfs_rreq_trace_copy, "COPY ") \ EM(netfs_rreq_trace_collect, "COLLECT") \ EM(netfs_rreq_trace_complete, "COMPLET") \ + EM(netfs_rreq_trace_copy, "COPY ") \ EM(netfs_rreq_trace_dirty, "DIRTY ") \ EM(netfs_rreq_trace_done, "DONE ") \ EM(netfs_rreq_trace_free, "FREE ") \ + EM(netfs_rreq_trace_ki_complete, "KI-CMPL") \ EM(netfs_rreq_trace_recollect, "RECLLCT") \ EM(netfs_rreq_trace_redirty, "REDIRTY") \ EM(netfs_rreq_trace_resubmit, "RESUBMT") \
@@ -64,13 +65,15 @@ EM(netfs_rreq_trace_unlock, "UNLOCK ") \ EM(netfs_rreq_trace_unlock_pgpriv2, "UNLCK-2") \ EM(netfs_rreq_trace_unmark, "UNMARK ") \ + EM(netfs_rreq_trace_unpause, "UNPAUSE") \ EM(netfs_rreq_trace_wait_ip, "WAIT-IP") \ - EM(netfs_rreq_trace_wait_pause, "WT-PAUS") \ - EM(netfs_rreq_trace_wait_queue, "WAIT-Q ") \ + EM(netfs_rreq_trace_wait_pause, "--PAUSED--") \ + EM(netfs_rreq_trace_wait_quiesce, "WAIT-QUIESCE") \ + EM(netfs_rreq_trace_waited_ip, "DONE-IP") \ + EM(netfs_rreq_trace_waited_pause, "--UNPAUSED--") \ + EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \ EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \ EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \ - EM(netfs_rreq_trace_woke_queue, "WOKE-Q ") \ - EM(netfs_rreq_trace_unpause, "UNPAUSE") \ E_(netfs_rreq_trace_write_done, "WR-DONE")#define netfs_sreq_sources \
@@ -83,6 +86,7 @@ E_(NETFS_WRITE_TO_CACHE, "WRIT")#define netfs_sreq_traces \ + EM(netfs_sreq_trace_abandoned, "ABNDN") \ EM(netfs_sreq_trace_add_donations, "+DON ") \ EM(netfs_sreq_trace_added, "ADD ") \ EM(netfs_sreq_trace_cache_nowrite, "CA-NW") \
@@ -90,6 +94,7 @@ EM(netfs_sreq_trace_cache_write, "CA-WR") \ EM(netfs_sreq_trace_cancel, "CANCL") \ EM(netfs_sreq_trace_clear, "CLEAR") \ + EM(netfs_sreq_trace_consumed, "CONSM") \ EM(netfs_sreq_trace_discard, "DSCRD") \ EM(netfs_sreq_trace_donate_to_prev, "DON-P") \EM (netfs_sreq_trace_donate_to_next,"DON-N" ) \
@@ -97,7 +102,12 @@ EM(netfs_sreq_trace_fail, "FAIL ") \ EM(netfs_sreq_trace_free, "FREE ") \ EM(netfs_sreq_trace_hit_eof, "EOF ") \ - EM(netfs_sreq_trace_io_progress, "IO ") \ + EM(netfs_sreq_trace_io_bad, "I-BAD") \ + EM(netfs_sreq_trace_io_malformed, "I-MLF") \ + EM(netfs_sreq_trace_io_unknown, "I-UNK") \ + EM(netfs_sreq_trace_io_progress, "I-OK ") \ + EM(netfs_sreq_trace_io_req_submitted, "I-RSB") \ + EM(netfs_sreq_trace_io_retry_needed, "I-RTR") \ EM(netfs_sreq_trace_limited, "LIMIT") \ EM(netfs_sreq_trace_need_clear, "N-CLR") \EM (netfs_sreq_trace_partial_read,"PARTR" ) \
@@ -143,8 +153,8 @@#define netfs_sreq_ref_traces \ EM(netfs_sreq_trace_get_copy_to_cache, "GET COPY2C ") \ - EM(netfs_sreq_trace_get_resubmit, "GET RESUBMIT") \ - EM(netfs_sreq_trace_get_submit, "GET SUBMIT") \ + EM(netfs_sreq_trace_get_resubmit, "GET RESUBMT") \ + EM(netfs_sreq_trace_get_submit, "GET SUBMIT ") \ EM(netfs_sreq_trace_get_short_read, "GET SHORTRD") \ EM(netfs_sreq_trace_new, "NEW ") \ EM(netfs_sreq_trace_put_abandon, "PUT ABANDON") \