netfs, ceph, nfs, cachefiles: Miscellaneous fixes/changes
Subsystems (10)
David Howells
maintainercover netfs, ceph, nfs, cachefiles: Miscellaneous fixes/changes
Hi Christian,
Here are some miscellaneous fixes and changes for netfslib and the ceph and nfs filesystems:
(1) Ignore silly-rename files from afs and nfs when building the header archive in a kernel build.
(2) netfs: Fix the way read result collection applies results to folios when each folio is being read by multiple subrequests and the results come out of order.
(3) netfs: Fix ENOMEM handling in buffered reads.
(4) nfs: Fix an oops in nfs_netfs_init_request() when copying to the cache.
(5) cachefiles: Parse the "secctx" command immediately to get the correct error rather than leaving it to the "bind" command.
(6) netfs: Remove a redundant smp_rmb(). This isn't a bug per se and could be deferred.
(7) netfs: Fix missing barriers by using clear_and_wake_up_bit().
(8) netfs: Work around recursion in read retry by failing and abandoning the retried subrequest if no I/O is performed.
[!] NOTE: This only works around the recursion problem if the
recursion keeps returning no data. If the server manages, say, to repeatedly return a single byte of data faster than the retry algorithm can complete, it will still recurse and the stack overrun may still occur. Actually fixing this requires quite an intrusive change which will hopefully make the next merge window.
(9) netfs: Fix the clearance of a folio_queue when unlocking the page if we're going to want to subsequently send the queue for copying to the cache (if, for example, we're using ceph).
(10) netfs: Fix the lack of cancellation of copy-to-cache when the cache for a file is temporarily disabled (for example when a DIO write is done to the file). This patch and (9) fix hangs with ceph.
With these patches, I can run xfstest -g quick to completion on ceph with a local cache.
The patches can also be found here with a bonus cifs patch:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-fixes
Thanks,
David
David Howells (8):
kheaders: Ignore silly-rename files
netfs: Fix non-contiguous donation between completed reads
netfs: Fix enomem handling in buffered reads
nfs: Fix oops in nfs_netfs_init_request() when copying to cache
netfs: Fix missing barriers by using clear_and_wake_up_bit()
netfs: Work around recursion by abandoning retry if nothing read
netfs: Fix ceph copy to cache on write-begin
netfs: Fix the (non-)cancellation of copy when cache is temporarily
disabled
Max Kellermann (1):
cachefiles: Parse the "secctx" immediately
Zilin Guan (1):
netfs: Remove redundant use of smp_rmb()
fs/9p/vfs_addr.c | 6 +++++- fs/afs/write.c | 5 ++++- fs/cachefiles/daemon.c | 14 +++++++------- fs/cachefiles/internal.h | 3 ++- fs/cachefiles/security.c | 6 +++--- fs/netfs/buffered_read.c | 28 ++++++++++++++++------------ fs/netfs/direct_write.c | 1 - fs/netfs/read_collect.c | 33 +++++++++++++++++++-------------- fs/netfs/read_pgpriv2.c | 4 ++++ fs/netfs/read_retry.c | 6 ++++-- fs/netfs/write_collect.c | 14 +++++--------- fs/netfs/write_issue.c | 2 ++ fs/nfs/fscache.c | 9 ++++++++- fs/smb/client/cifssmb.c | 13 +++++++++---- fs/smb/client/smb2pdu.c | 9 ++++++--- include/linux/netfs.h | 6 +++--- kernel/gen_kheaders.sh | 1 + 17 files changed, 98 insertions(+), 62 deletions(-)
David Howells
maintainerPATCH 1/10 kheaders: Ignore silly-rename files
Tell tar to ignore silly-rename files (".__afs*" and ".nfs*") when building the header archive. These occur when a file that is open is unlinked locally, but hasn't yet been closed. Such files are visible to the user via the getdents() syscall and so programs may want to do things with them.
During the kernel build, such files may be made during the processing of header files and the cleanup may get deferred by fput() which may result in tar seeing these files when it reads the directory, but they may have disappeared by the time it tries to open them, causing tar to fail with an error. Further, we don't want to include them in the tarball if they still exist.
With CONFIG_HEADERS_INSTALL=y, something like the following may be seen:
find: './kernel/.tmp_cpio_dir/include/dt-bindings/reset/.__afs2080': No such file or directory
tar: ./include/linux/greybus/.__afs3C95: File removed before we read it
The find warning doesn't seem to cause a problem.
Fix this by telling tar when called from in gen_kheaders.sh to exclude such files. This only affects afs and nfs; cifs uses the Windows Hidden attribute to prevent the file from being seen.
--- kernel/gen_kheaders.sh | 1 + 1 file changed, 1 insertion(+)
kernel/gen_kheaders.sh +1 -0
@@ -89,6 +89,7 @@ find $cpio_dir -type f -print0 | # Create archive and try to normalize metadata for reproducibility. tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \ + --exclude=".__afs*" --exclude=".nfs*" \ --owner=0 --group=0 --sort=name --numeric-owner --mode=u=rw,go=r,a+X \ -I $XZ -cf $tarfile -C $cpio_dir/ . > /dev/null
Re: [PATCH 01/10] kheaders: Ignore silly-rename files
Quoted 6 lines · David Howells
Tell tar to ignore silly-rename files (".__afs*" and ".nfs*") when building the header archive. These occur when a file that is open is unlinked locally, but hasn't yet been closed. Such files are visible to the user via the
getdents()syscall and so programs may want to do things with them.
Quoted 8 lines
During the kernel build, such files may be made during the processing of header files and the cleanup may get deferred by
fput()which may result in tar seeing these files when it reads the directory, but they may have disappeared by the time it tries to open them, causing tar to fail with an error. Further, we don't want to include them in the tarball if they still exist.With
CONFIG_HEADERS_INSTALL=y, something like the following may be seen:
I am confused.
kernel/gen_kheaders.sh is executed when CONFIG_IKHEADERS is enabled.
How is CONFIG_HEADERS_INSTALL related?
Quoted 4 lines
find: '
./kernel/.tmp_cpio_dir/include/dt-bindings/reset/.__afs2080': No such file or directory
tar:./include/linux/greybus/.__afs3C95: File removed before we read itThe find warning doesn't seem to cause a problem.
I picked the following commit.
https://lore.kernel.org/all/[email protected]/
This shoots the root cause of the 'find' errors.
Does it fix your problems too?
Your patch does not address the 'find' errors.
Quoted 28 lines
Fix this by telling tar when called from in
gen_kheaders.shto exclude such files. This only affects afs and nfs; cifs uses the Windows Hidden attribute to prevent the file from being seen.Signed-off-by David Howells <[email protected]>cc Masahiro Yamada <[email protected]>cc Marc Dionne <[email protected]>--- kernel/gen_kheaders.sh | 1 + 1 file changed, 1 insertion(+)kernel/gen_kheaders.sh +1
-0@@ -89,6 +89,7 @@ find $cpio_dir -type f -print0 | # Create archive and try to normalize metadata for reproducibility. tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \ + --exclude=".__afs*" --exclude=".nfs*" \ --owner=0 --group=0 --sort=name --numeric-owner --mode=u=rw,go=r,a+X \ -I $XZ -cf $tarfile -C $cpio_dir/ . > /dev/null
Signature · 2 lines
Best Regards Masahiro Yamada
David Howells
maintainerPATCH 2/10 netfs: Fix non-contiguous donation between completed reads
In reply to David HowellsWhen a read subrequest finishes, if it doesn't have sufficient coverage to complete the folio(s) covering either side of it, it will donate the excess coverage to the adjacent subrequests on either side, offloading responsibility for unlocking the folio(s) covered to them.
Now, preference is given to donating down to a lower file offset over donating up because that check is done first - but there's no check that the lower subreq is actually contiguous, and so we can end up donating incorrectly.
The scenario seen[1] is that an 8MiB readahead request spanning four 2MiB folios is split into eight 1MiB subreqs (numbered 1 through 8). These terminate in the order 1,6,2,5,3,7,4,8. What happens is:
- 1 donates to 2
- 6 donates to 5
- 2 completes, unlocking the first folio (with 1).
- 5 completes, unlocking the third folio (with 6).
- 3 donates to 4
- 7 donates to 4 incorrectly
- 4 completes, unlocking the second folio (with 3), but can't use the excess from 7.
- 8 donates to 4, also incorrectly.
Fix this by preventing downward donation if the subreqs are not contiguous (in the example above, 7 donates to 4 across the gap left by 5 and 6).
--- fs/netfs/read_collect.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
fs/netfs/read_collect.c +5 -4
@@ -247,16 +247,17 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was/* Deal with the trickiest case: that this subreq is in the middle of a * folio, not touching either edge, but finishes first. In such a -* case, we donate to the previous subreq, if there is one, so that the -* donation is only handled when that completes - and remove this -* subreq from the list. +* case, we donate to the previous subreq, if there is one and if it is +* contiguous, so that the donation is only handled when that completes +* - and remove this subreq from the list. * * If the previous subreq finished first, we will have acquired their * donation and should be able to unlock folios and/or donate nextwards. */ if (!subreq->consumed && !prev_donated && - !list_is_first (&subreq->rreq_link, &rreq->subrequests)) { + !list_is_first (&subreq->rreq_link, &rreq->subrequests) && + subreq->start == prev->start + prev->len) { prev =list_prev_entry (subreq, rreq_link);WRITE_ONCE (prev->next_donated, prev->next_donated + subreq->len); subreq->start += subreq->len;
David Howells
maintainerPATCH 3/10 netfs: Fix enomem handling in buffered reads
In reply to David HowellsIf netfs_read_to_pagecache() gets an error from either ->prepare_read() or from netfs_prepare_read_iterator(), it needs to decrement ->nr_outstanding, cancel the subrequest and break out of the issuing loop. Currently, it only does this for two of the cases, but there are two more that aren't handled.
Fix this by moving the handling to a common place and jumping to it from all four places. This is in preference to inserting a wrapper around netfs_prepare_read_iterator() as proposed by Dmitry Antipov[1].
--- fs/netfs/buffered_read.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
fs/netfs/buffered_read.c +16 -12
@@ -275,22 +275,14 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)netfs_stat (&netfs_n_rh_download);if (rreq->netfs_ops->prepare_read) { ret = rreq->netfs_ops->prepare_read (subreq); -if (ret <0 ) { -atomic_dec (&rreq->nr_outstanding); -netfs_put_subrequest (subreq, false, - netfs_sreq_trace_put_cancel); -break ; - } +if (ret <0 ) + goto prep_failed;trace_netfs_sreq (subreq, netfs_sreq_trace_prepare); } slice =netfs_prepare_read_iterator (subreq); -if (slice <0 ) { -atomic_dec (&rreq->nr_outstanding); -netfs_put_subrequest (subreq, false, netfs_sreq_trace_put_cancel); - ret = slice; -break ; - } +if (slice <0 ) + goto prep_iter_failed; rreq->netfs_ops->issue_read (subreq); goto done;
@@ -302,6 +294,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)trace_netfs_sreq (subreq, netfs_sreq_trace_submit);netfs_stat (&netfs_n_rh_zero); slice =netfs_prepare_read_iterator (subreq); +if (slice <0 ) + goto prep_iter_failed;__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags);netfs_read_subreq_terminated (subreq,0 , false); goto done;
@@ -310,6 +304,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)if (source ==NETFS_READ_FROM_CACHE ) {trace_netfs_sreq (subreq, netfs_sreq_trace_submit); slice =netfs_prepare_read_iterator (subreq); +if (slice <0 ) + goto prep_iter_failed;netfs_read_cache_to_pagecache (rreq, subreq); goto done; }
@@ -318,6 +314,14 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)WARN_ON_ONCE (1 );break ; + prep_iter_failed: + ret = slice; + prep_failed: + subreq->error = ret; +atomic_dec (&rreq->nr_outstanding); +netfs_put_subrequest (subreq, false, netfs_sreq_trace_put_cancel); +break ; + done: size -= slice; start += slice;
David Howells
maintainerPATCH 4/10 nfs: Fix oops in nfs_netfs_init_request() when copying to cache
In reply to David HowellsWhen netfslib wants to copy some data that has just been read on behalf of nfs, it creates a new write request and calls nfs_netfs_init_request() to initialise it, but with a NULL file pointer. This causes nfs_file_open_context() to oops - however, we don't actually need the nfs context as we're only going to write to the cache.
Fix this by just returning if we aren't given a file pointer and emit a warning if the request was for something other than copy-to-cache.
Further, fix nfs_netfs_free_request() so that it doesn't try to free the context if the pointer is NULL.
--- fs/nfs/fscache.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/nfs/fscache.c +8 -1
@@ -263,6 +263,12 @@ int nfs_netfs_readahead(struct readahead_control *ractl)static atomic_t nfs_netfs_debug_id;static int nfs_netfs_init_request (struct netfs_io_request *rreq,struct file *file) { +if (!file) { +if (WARN_ON_ONCE (rreq->origin !=NETFS_PGPRIV2_COPY_TO_CACHE )) +return -EIO ; +return 0 ; + } + rreq->netfs_priv =get_nfs_open_context (nfs_file_open_context (file)); rreq->debug_id =atomic_inc_return (&nfs_netfs_debug_id);/* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */
@@ -274,7 +280,8 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fistatic void nfs_netfs_free_request (struct netfs_io_request *rreq) { -put_nfs_open_context (rreq->netfs_priv); +if (rreq->netfs_priv) +put_nfs_open_context (rreq->netfs_priv); }static struct nfs_netfs_io_data *nfs_netfs_alloc (struct netfs_io_subrequest *sreq)
David Howells
maintainerPATCH 5/10 cachefiles: Parse the "secctx" immediately
In reply to David HowellsFrom: Max Kellermann <[email protected]>
Instead of storing an opaque string, call security_secctx_to_secid() right in the "secctx" command handler and store only the numeric "secid". This eliminates an unnecessary string allocation and allows the daemon to receive errors when writing the "secctx" command instead of postponing the error to the "bind" command handler. For example, if the kernel was built without CONFIG_SECURITY, "bind" will return EOPNOTSUPP, but the daemon doesn't know why. With this patch, the "secctx" will instead return EOPNOTSUPP which is the right context for this error.
This patch adds a boolean flag have_secid because I'm not sure if we can safely assume that zero is the special secid value for "not set". This appears to be true for SELinux, Smack and AppArmor, but since this attribute is not documented, I'm unable to derive a stable guarantee for that.
--- fs/cachefiles/daemon.c | 14 +++++++------- fs/cachefiles/internal.h | 3 ++- fs/cachefiles/security.c | 6 +++--- 3 files changed, 12 insertions(+), 11 deletions(-)
fs/cachefiles/daemon.c +7 -7
@@ -15,6 +15,7 @@#include <linux/namei.h> #include <linux/poll.h> #include <linux/mount.h> +#include <linux/security.h> #include <linux/statfs.h> #include <linux/ctype.h> #include <linux/string.h>
@@ -576,7 +577,7 @@ static int cachefiles_daemon_dir(struct cachefiles_cache *cache, char *args)*/ static int cachefiles_daemon_secctx (struct cachefiles_cache *cache,char *args) { -char *secctx; +int err;_enter (",%s" , args);
@@ -585,16 +586,16 @@ static int cachefiles_daemon_secctx(struct cachefiles_cache *cache, char *args)return -EINVAL ; } -if (cache->secctx) { +if (cache->have_secid) {pr_err ("Second security context specified\n" );return -EINVAL ; } - secctx =kstrdup (args,GFP_KERNEL ); -if (!secctx) -return -ENOMEM ; + err =security_secctx_to_secid (args,strlen (args), &cache->secid); +if (err) +return err; - cache->secctx = secctx; + cache->have_secid = true;return 0 ; }
@@ -820,7 +821,6 @@ static void cachefiles_daemon_unbind(struct cachefiles_cache *cache)put_cred (cache->cache_cred);kfree (cache->rootdirname); -kfree (cache->secctx);kfree (cache->tag);_leave ("" );
fs/cachefiles/internal.h +2 -1
@@ -122,7 +122,6 @@ struct cachefiles_cache {#define CACHEFILES_STATE_CHANGED 3/* T if state changed (poll trigger) */ #define CACHEFILES_ONDEMAND_MODE 4/* T if in on-demand read mode */ char *rootdirname;/* name of cache root directory */ -char *secctx;/* LSM security context */ char *tag;/* cache binding tag */ refcount_t unbind_pincount;/* refcount to do daemon unbind */ struct xarray reqs;/* xarray of pending on-demand requests */
@@ -130,6 +129,8 @@ struct cachefiles_cache {struct xarray ondemand_ids;/* xarray for ondemand_id allocation */ u32 ondemand_id_next;u32 msg_id_next; +u32 secid;/* LSM security id */ +bool have_secid;/* whether "secid" was set */ };static inline bool cachefiles_in_ondemand_mode (struct cachefiles_cache *cache)
fs/cachefiles/security.c +3 -3
@@ -18,7 +18,7 @@ int cachefiles_get_security_ID(struct cachefiles_cache *cache)struct cred *new;int ret; -_enter ("{%s}" , cache->secctx); +_enter ("{%u}" , cache->have_secid ? cache->secid :0 ); new =prepare_kernel_cred (current);if (!new) {
@@ -26,8 +26,8 @@ int cachefiles_get_security_ID(struct cachefiles_cache *cache) goto error; } -if (cache->secctx) { - ret =set_security_override_from_ctx (new, cache->secctx); +if (cache->have_secid) { + ret =set_security_override (new, cache->secid);if (ret <0 ) {put_cred (new); pr_err("Security denies permission to nominate security context: error %d\n",
David Howells
maintainerPATCH 6/10 netfs: Remove redundant use of smp_rmb()
In reply to David HowellsFrom: Zilin Guan <[email protected]>
The function netfs_unbuffered_write_iter_locked() in fs/netfs/direct_write.c contains an unnecessary smp_rmb() call after wait_on_bit(). Since wait_on_bit() already incorporates a memory barrier that ensures the flag update is visible before the function returns, the smp_rmb() provides no additional benefit and incurs unnecessary overhead.
This patch removes the redundant barrier to simplify and optimize the code.
--- fs/netfs/direct_write.c | 1 - 1 file changed, 1 deletion(-)
fs/netfs/direct_write.c +0 -1
@@ -104,7 +104,6 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *trace_netfs_rreq (wreq, netfs_rreq_trace_wait_ip);wait_on_bit (&wreq->flags,NETFS_RREQ_IN_PROGRESS ,TASK_UNINTERRUPTIBLE ); -smp_rmb ();/* Read error/transferred after RIP flag */ ret = wreq->error;if (ret ==0 ) { ret = wreq->transferred;
Re: [PATCH 06/10] netfs: Remove redundant use of smp_rmb()
Quoted 8 lines · David Howells
From: Zilin Guan <[email protected]>
The function
netfs_unbuffered_write_iter_locked()infs/netfs/direct_write.ccontains an unnecessarysmp_rmb()call afterwait_on_bit(). Sincewait_on_bit()already incorporates a memory barrier that ensures the flag update is visible before the function returns, thesmp_rmb()provides no additional benefit and incurs unnecessary overhead.
Quoted 5 lines
This patch removes the redundant barrier to simplify and optimize the code.
Signed-off-by Zilin Guan <[email protected]>Signed-off-by David Howells <[email protected]>cc Akira Yokosawa <[email protected]>
Quoted 8 lines
cc Jeff Layton <[email protected]>--- fs/netfs/direct_write.c | 1 - 1 file changed, 1 deletion(-)
David Howells
maintainerPATCH 7/10 netfs: Fix missing barriers by using clear_and_wake_up_bit()
In reply to David HowellsUse clear_and_wake_up_bit() rather than something like:
clear_bit_unlock(NETFS_RREQ_IN_PROGRESS, &rreq->flags); wake_up_bit(&rreq->flags, NETFS_RREQ_IN_PROGRESS);
as there needs to be a barrier inserted between which is present in clear_and_wake_up_bit().
--- fs/netfs/read_collect.c | 3 +-- fs/netfs/write_collect.c | 9 +++------ 2 files changed, 4 insertions(+), 8 deletions(-)
fs/netfs/read_collect.c +1 -2
@@ -379,8 +379,7 @@ static void netfs_rreq_assess(struct netfs_io_request *rreq)task_io_account_read (rreq->transferred);trace_netfs_rreq (rreq, netfs_rreq_trace_wake_ip); -clear_bit_unlock (NETFS_RREQ_IN_PROGRESS , &rreq->flags); -wake_up_bit (&rreq->flags,NETFS_RREQ_IN_PROGRESS ); +clear_and_wake_up_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags);trace_netfs_rreq (rreq, netfs_rreq_trace_done);netfs_clear_subrequests (rreq, false);
fs/netfs/write_collect.c +3 -6
@@ -501,8 +501,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) goto need_retry;if ((notes &MADE_PROGRESS ) &&test_bit (NETFS_RREQ_PAUSE , &wreq->flags)) {trace_netfs_rreq (wreq, netfs_rreq_trace_unpause); -clear_bit_unlock (NETFS_RREQ_PAUSE , &wreq->flags); -wake_up_bit (&wreq->flags,NETFS_RREQ_PAUSE ); +clear_and_wake_up_bit (NETFS_RREQ_PAUSE , &wreq->flags); }if (notes &NEED_REASSESS ) {
@@ -605,8 +604,7 @@ void netfs_write_collection_worker(struct work_struct *work)_debug ("finished" );trace_netfs_rreq (wreq, netfs_rreq_trace_wake_ip); -clear_bit_unlock (NETFS_RREQ_IN_PROGRESS , &wreq->flags); -wake_up_bit (&wreq->flags,NETFS_RREQ_IN_PROGRESS ); +clear_and_wake_up_bit (NETFS_RREQ_IN_PROGRESS , &wreq->flags);if (wreq->iocb) {size_t written =min (wreq->transferred, wreq->len);
@@ -714,8 +712,7 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error,trace_netfs_sreq (subreq, netfs_sreq_trace_terminated); -clear_bit_unlock (NETFS_SREQ_IN_PROGRESS , &subreq->flags); -wake_up_bit (&subreq->flags,NETFS_SREQ_IN_PROGRESS ); +clear_and_wake_up_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags);/* If we are at the head of the queue, wake up the collector, * transferring a ref to it if we were the ones to do so.
Re: [PATCH 07/10] netfs: Fix missing barriers by using clear_and_wake_up_bit()
Hi David,
Quoted 7 lines · David Howells
Use
clear_and_wake_up_bit()rather than something like:clear_bit_unlock(NETFS_RREQ_IN_PROGRESS, &rreq->flags); wake_up_bit(&rreq->flags, NETFS_RREQ_IN_PROGRESS);as there needs to be a barrier inserted between which is present in
clear_and_wake_up_bit().
If I am reading the kernel-doc comment of clear_bit_unlock() [1, 2]:
This operation is atomic and provides release barrier semantics.
correctly, there already seems to be a barrier which should be good enough.
[1]: https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.clear_bit_unlock
[2]: include/asm-generic/bitops/instrumented-lock.h
Quoted 3 lines
Fixes 288ace2f57c9 ("netfs: New writeback implementation")Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")
So I'm not sure this fixes anything.
What am I missing?
Thanks, Akira
Quoted 11 lines
Signed-off-by David Howells <[email protected]>cc Zilin Guan <[email protected]>cc Akira Yokosawa <[email protected]>cc Jeff Layton <[email protected]>--- fs/netfs/read_collect.c | 3 +-- fs/netfs/write_collect.c | 9 +++------ 2 files changed, 4 insertions(+), 8 deletions(-)
[...]
David Howells
maintainerRe: [PATCH 07/10] netfs: Fix missing barriers by using clear_and_wake_up_bit()
[Adding Paul McKenney as he's the expert.]
Quoted 20 lines · Akira Yokosawa
Quoted 7 lines · David Howells
Use
clear_and_wake_up_bit()rather than something like:clear_bit_unlock(NETFS_RREQ_IN_PROGRESS, &rreq->flags); wake_up_bit(&rreq->flags, NETFS_RREQ_IN_PROGRESS);as there needs to be a barrier inserted between which is present in
clear_and_wake_up_bit().If I am reading the kernel-doc comment of
clear_bit_unlock()[1, 2]:This operation is atomic and provides release barrier semantics.
correctly, there already seems to be a barrier which should be good enough.
[1]: https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.clear_bit_unlock
[2]:include/asm-generic/bitops/instrumented-lock.hQuoted 1 line
Quoted 6 lines
Quoted 2 lines
Fixes 288ace2f57c9 ("netfs: New writeback implementation")Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")So I'm not sure this fixes anything.
What am I missing?
We may need two barriers. You have three things to synchronise:
(1) The stuff you did before unlocking.
(2) The lock bit.
(3) The task state.
clear_bit_unlock() interposes a release barrier between (1) and (2).
Neither clear_bit_unlock() nor wake_up_bit(), however, necessarily interpose a barrier between (2) and (3). I'm not sure it entirely matters, but it seems that since we have a function that combines the two, we should probably use it - though, granted, it might not actually be a fix.
David
Akira Yokosawa
Reviewed-byRe: [PATCH 07/10] netfs: Fix missing barriers by using clear_and_wake_up_bit()
Quoted 35 lines · David Howells
[Adding Paul McKenney as he's the expert.]
Quoted 20 lines · Akira Yokosawa
Quoted 7 lines · David Howells
Use
clear_and_wake_up_bit()rather than something like:clear_bit_unlock(NETFS_RREQ_IN_PROGRESS, &rreq->flags); wake_up_bit(&rreq->flags, NETFS_RREQ_IN_PROGRESS);as there needs to be a barrier inserted between which is present in
clear_and_wake_up_bit().If I am reading the kernel-doc comment of
clear_bit_unlock()[1, 2]:This operation is atomic and provides release barrier semantics.
correctly, there already seems to be a barrier which should be good enough.
[1]: https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.clear_bit_unlock
[2]:include/asm-generic/bitops/instrumented-lock.hQuoted 1 line
Quoted 6 lines
Quoted 2 lines
Fixes 288ace2f57c9 ("netfs: New writeback implementation")Fixes ee4cdf7ba857 ("netfs: Speed up buffered reading")So I'm not sure this fixes anything.
What am I missing?
We may need two barriers. You have three things to synchronise:
(1) The stuff you did before unlocking.
Quoted 8 lines
(2) The lock bit.
(3) The task state.
clear_bit_unlock()interposes a release barrier between (1) and (2).Neither
clear_bit_unlock()norwake_up_bit(), however, necessarily interpose a barrier between (2) and (3).
Got it!
I was confused because I compared kernel-doc comments of clear_bit_unlock() and clear_and_wake_up_bit() only, without looking at latter's code.
clear_and_wake_up_bit() has this description in its kernel-doc:
* The designated bit is cleared and any tasks waiting in wait_on_bit()
* or similar will be woken. This call has RELEASE semantics so that
* any changes to memory made before this call are guaranteed to be visible
* after the corresponding wait_on_bit() completes.
, without any mention of additional full barrier at your (3) above.
It might be worth mentioning it there.
Thoughts?
FWIW,
Quoted 3 lines
I'm not sure it entirely matters, but it seems
that since we have a function that combines the two, we should probably use it - though, granted, it might not actually be a fix.
Looks like it should matter where smp_mb__after_atomic() is stronger than a plain barrier().
Akira
David Howells
maintainerPATCH 8/10 netfs: Work around recursion by abandoning retry if nothing read
In reply to David Howellssyzkaller reported recursion with a loop of three calls (netfs_rreq_assess, netfs_retry_reads and netfs_rreq_terminated) hitting the limit of the stack during an unbuffered or direct I/O read.
There are a number of issues:
(1) There is no limit on the number of retries.
(2) A subrequest is supposed to be abandoned if it does not transfer anything (NETFS_SREQ_NO_PROGRESS), but that isn't checked under all circumstances.
(3) The actual root cause, which is this:
if (atomic_dec_and_test(&rreq->nr_outstanding))
netfs_rreq_terminated(rreq, ...);
When we do a retry, we bump the rreq->nr_outstanding counter to prevent the final cleanup phase running before we've finished dispatching the retries. The problem is if we hit 0, we have to do the cleanup phase - but we're in the cleanup phase and end up repeating the retry cycle, hence the recursion.
Work around the problem by limiting the number of retries. This is based on Lizhi Xu's patch[1], and makes the following changes:
(1) Replace NETFS_SREQ_NO_PROGRESS with NETFS_SREQ_MADE_PROGRESS and make the filesystem set it if it managed to read or write at least one byte of data. Clear this bit before issuing a subrequest.
(2) Add a ->retry_count member to the subrequest and increment it any time we do a retry.
(3) Remove the NETFS_SREQ_RETRYING flag as it is superfluous with
->retry_count. If the latter is non-zero, we're doing a retry.
(4) Abandon a subrequest if retry_count is non-zero and we made no progress.
(5) Use ->retry_count in both the write-side and the read-size.
[?] Question: Should I set a hard limit on retry_count in both read and
write? Say it hits 50, we always abandon it. The problem is that these changes only mitigate the issue. As long as it made at least one byte of progress, the recursion is still an issue. This patch mitigates the problem, but does not fix the underlying cause. I have patches that will do that, but it's an intrusive fix that's currently pending for the next merge window.
The oops generated by KASAN looks something like:
BUG: TASK stack guard page was hit at ffffc9000482ff48 (stack is ffffc90004830000..ffffc90004838000)
Oops: stack guard page: 0000 [#1] PREEMPT SMP KASAN NOPTI
...
RIP: 0010:mark_lock+0x25/0xc60 kernel/locking/lockdep.c:4686
...
mark_usage kernel/locking/lockdep.c:4646 [inline]
__lock_acquire+0x906/0x3ce0 kernel/locking/lockdep.c:5156
lock_acquire.part.0+0x11b/0x380 kernel/locking/lockdep.c:5825
local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
___slab_alloc+0x123/0x1880 mm/slub.c:3695
__slab_alloc.constprop.0+0x56/0xb0 mm/slub.c:3908
__slab_alloc_node mm/slub.c:3961 [inline]
slab_alloc_node mm/slub.c:4122 [inline]
kmem_cache_alloc_noprof+0x2a7/0x2f0 mm/slub.c:4141
radix_tree_node_alloc.constprop.0+0x1e8/0x350 lib/radix-tree.c:253
idr_get_free+0x528/0xa40 lib/radix-tree.c:1506
idr_alloc_u32+0x191/0x2f0 lib/idr.c:46
idr_alloc+0xc1/0x130 lib/idr.c:87
p9_tag_alloc+0x394/0x870 net/9p/client.c:321
p9_client_prepare_req+0x19f/0x4d0 net/9p/client.c:644
p9_client_zc_rpc.constprop.0+0x105/0x880 net/9p/client.c:793
p9_client_read_once+0x443/0x820 net/9p/client.c:1570
p9_client_read+0x13f/0x1b0 net/9p/client.c:1534
v9fs_issue_read+0x115/0x310 fs/9p/vfs_addr.c:74
netfs_retry_read_subrequests fs/netfs/read_retry.c:60 [inline]
netfs_retry_reads+0x153a/0x1d00 fs/netfs/read_retry.c:232
netfs_rreq_assess+0x5d3/0x870 fs/netfs/read_collect.c:371
netfs_rreq_terminated+0xe5/0x110 fs/netfs/read_collect.c:407
netfs_retry_reads+0x155e/0x1d00 fs/netfs/read_retry.c:235
netfs_rreq_assess+0x5d3/0x870 fs/netfs/read_collect.c:371
netfs_rreq_terminated+0xe5/0x110 fs/netfs/read_collect.c:407
netfs_retry_reads+0x155e/0x1d00 fs/netfs/read_retry.c:235
netfs_rreq_assess+0x5d3/0x870 fs/netfs/read_collect.c:371
...
netfs_rreq_terminated+0xe5/0x110 fs/netfs/read_collect.c:407
netfs_retry_reads+0x155e/0x1d00 fs/netfs/read_retry.c:235
netfs_rreq_assess+0x5d3/0x870 fs/netfs/read_collect.c:371
netfs_rreq_terminated+0xe5/0x110 fs/netfs/read_collect.c:407
netfs_retry_reads+0x155e/0x1d00 fs/netfs/read_retry.c:235
netfs_rreq_assess+0x5d3/0x870 fs/netfs/read_collect.c:371
netfs_rreq_terminated+0xe5/0x110 fs/netfs/read_collect.c:407
netfs_dispatch_unbuffered_reads fs/netfs/direct_read.c:103 [inline]
netfs_unbuffered_read fs/netfs/direct_read.c:127 [inline]
netfs_unbuffered_read_iter_locked+0x12f6/0x19b0 fs/netfs/direct_read.c:221
netfs_unbuffered_read_iter+0xc5/0x100 fs/netfs/direct_read.c:256
v9fs_file_read_iter+0xbf/0x100 fs/9p/vfs_file.c:361
do_iter_readv_writev+0x614/0x7f0 fs/read_write.c:832
vfs_readv+0x4cf/0x890 fs/read_write.c:1025
do_preadv fs/read_write.c:1142 [inline]
__do_sys_preadv fs/read_write.c:1192 [inline]
__se_sys_preadv fs/read_write.c:1187 [inline]
__x64_sys_preadv+0x22d/0x310 fs/read_write.c:1187
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83--- fs/9p/vfs_addr.c | 6 +++++- fs/afs/write.c | 5 ++++- fs/netfs/read_collect.c | 15 +++++++++------ fs/netfs/read_retry.c | 6 ++++-- fs/netfs/write_collect.c | 5 ++--- fs/netfs/write_issue.c | 2 ++ fs/smb/client/cifssmb.c | 13 +++++++++---- fs/smb/client/smb2pdu.c | 9 ++++++--- include/linux/netfs.h | 6 +++--- 9 files changed, 44 insertions(+), 23 deletions(-)
fs/9p/vfs_addr.c +5 -1
@@ -57,6 +57,8 @@ static void v9fs_issue_write(struct netfs_io_subrequest *subreq)int err, len; len =p9_client_write (fid, subreq->start, &subreq->io_iter, &err); +if (len >0 ) +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags);netfs_write_subrequest_terminated (subreq, len ?: err, false); }
@@ -80,8 +82,10 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)if (pos + total >=i_size_read (rreq->inode))__set_bit (NETFS_SREQ_HIT_EOF , &subreq->flags); -if (!err) +if (!err) { subreq->transferred += total; +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); + }netfs_read_subreq_terminated (subreq, err, false); }
fs/afs/write.c +4 -1
@@ -122,7 +122,7 @@ static void afs_issue_write_worker(struct work_struct *work)if (subreq->debug_index ==3 )return netfs_write_subrequest_terminated (subreq, -ENOANO , false); -if (!test_bit (NETFS_SREQ_RETRYING , &subreq->flags)) { +if (!subreq->retry_count) {set_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags);return netfs_write_subrequest_terminated (subreq, -EAGAIN , false); }
@@ -149,6 +149,9 @@ static void afs_issue_write_worker(struct work_struct *work)afs_wait_for_operation (op); ret =afs_put_operation (op);switch (ret) { +case 0 : +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); +break ;case -EACCES :case -EPERM :case -ENOKEY :
fs/netfs/read_collect.c +9 -6
@@ -438,7 +438,7 @@ void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq, rreq->origin == NETFS_READPAGE || rreq->origin == NETFS_READ_FOR_WRITE)) {netfs_consume_read_data (subreq, was_async); -__clear_bit (NETFS_SREQ_NO_PROGRESS , &subreq->flags); +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); } }EXPORT_SYMBOL (netfs_read_subreq_progress);
@@ -497,7 +497,7 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, rreq->origin == NETFS_READPAGE || rreq->origin == NETFS_READ_FOR_WRITE)) {netfs_consume_read_data (subreq, was_async); -__clear_bit (NETFS_SREQ_NO_PROGRESS , &subreq->flags); +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); } rreq->transferred += subreq->transferred; }
@@ -511,10 +511,13 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, }else {trace_netfs_sreq (subreq, netfs_sreq_trace_short);if (subreq->transferred > subreq->consumed) { -__set_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); -__clear_bit (NETFS_SREQ_NO_PROGRESS , &subreq->flags); -set_bit (NETFS_RREQ_NEED_RETRY , &rreq->flags); - }else if (!__test_and_set_bit (NETFS_SREQ_NO_PROGRESS , &subreq->flags)) { +/* If we didn't read new data, abandon retry. */ +if (subreq->retry_count && +test_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags)) { +__set_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); +set_bit (NETFS_RREQ_NEED_RETRY , &rreq->flags); + } + }else if (test_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags)) {__set_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags);set_bit (NETFS_RREQ_NEED_RETRY , &rreq->flags); }else {
fs/netfs/read_retry.c +4 -2
@@ -56,6 +56,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)if (test_bit (NETFS_SREQ_FAILED , &subreq->flags))break ;if (__test_and_clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags)) { +__clear_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); + subreq->retry_count++;netfs_reset_iter (subreq);netfs_reissue_read (rreq, subreq); }
@@ -137,7 +139,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) stream0->sreq_max_len = subreq->len;__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); -__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); +__clear_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); + subreq->retry_count++;spin_lock_bh (&rreq->lock);list_add_tail (&subreq->rreq_link, &rreq->subrequests);
@@ -213,7 +216,6 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) subreq->error = -ENOMEM ;__clear_bit (NETFS_SREQ_FAILED , &subreq->flags);__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); -__clear_bit (NETFS_SREQ_RETRYING , &subreq->flags); }spin_lock_bh (&rreq->lock);list_splice_tail_init (&queue, &rreq->subrequests);
fs/netfs/write_collect.c +2 -3
@@ -179,7 +179,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,struct iov_iter source = subreq->io_iter;iov_iter_revert (&source, subreq->len - source.count); -__set_bit (NETFS_SREQ_RETRYING , &subreq->flags);netfs_get_subrequest (subreq, netfs_sreq_trace_get_resubmit);netfs_reissue_write (stream, subreq, &source); }
@@ -234,7 +233,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,/* Renegotiate max_len (wsize) */ trace_netfs_sreq (subreq, netfs_sreq_trace_retry);__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); -__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); + subreq->retry_count++; stream->prepare_write (subreq); part =min (len, stream->sreq_max_len);
@@ -279,7 +278,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, subreq->start = start; subreq->debug_index =atomic_inc_return (&wreq->subreq_counter); subreq->stream_nr = to->stream_nr; -__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); + subreq->retry_count =1 ; trace_netfs_sreq_ref(wreq->debug_id, subreq->debug_index, refcount_read(&subreq->ref),
fs/netfs/write_issue.c +2 -0
@@ -244,6 +244,8 @@ void netfs_reissue_write(struct netfs_io_stream *stream,iov_iter_advance (source, size);iov_iter_truncate (&subreq->io_iter, size); + subreq->retry_count++; +__clear_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags);__set_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags);netfs_do_issue_write (stream, subreq); }
fs/smb/client/cifssmb.c +9 -4
@@ -1319,14 +1319,16 @@ cifs_readv_callback(struct mid_q_entry *mid) }if (rdata->result == -ENODATA ) { -__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); rdata->result =0 ; +__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); }else {size_t trans = rdata->subreq.transferred + rdata->got_bytes;if (trans < rdata->subreq.len && rdata->subreq.start + trans == ictx->remote_i_size) { -__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); rdata->result =0 ; +__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); + }else if (rdata->got_bytes >0 ) { +__set_bit (NETFS_SREQ_MADE_PROGRESS , &rdata->subreq.flags); } }
@@ -1670,10 +1672,13 @@ cifs_writev_callback(struct mid_q_entry *mid)if (written > wdata->subreq.len) written &=0xFFFF ; -if (written < wdata->subreq.len) +if (written < wdata->subreq.len) { result = -ENOSPC ; -else + }else { result = written; +if (written >0 ) +__set_bit (NETFS_SREQ_MADE_PROGRESS , &wdata->subreq.flags); + }break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED :
fs/smb/client/smb2pdu.c +6 -3
@@ -4615,6 +4615,7 @@ smb2_readv_callback(struct mid_q_entry *mid)__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); rdata->result =0 ; } +__set_bit (NETFS_SREQ_MADE_PROGRESS , &rdata->subreq.flags); } trace_smb3_rw_credits(rreq_debug_id, subreq_debug_index, rdata->credits.value, server->credits, server->in_flight,
@@ -4840,10 +4841,12 @@ smb2_writev_callback(struct mid_q_entry *mid)if (written > wdata->subreq.len) written &=0xFFFF ; -if (written < wdata->subreq.len) +if (written < wdata->subreq.len) { wdata->result = -ENOSPC ; -else + }else if (written >0 ) { wdata->subreq.len = written; +__set_bit (NETFS_SREQ_MADE_PROGRESS , &wdata->subreq.flags); + }break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED :
@@ -5012,7 +5015,7 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) }#endif -if (test_bit (NETFS_SREQ_RETRYING , &wdata->subreq.flags)) +if (wdata->subreq.retry_count >0 )smb2_set_replay (server, &rqst); cifs_dbg(FYI, "async write at %llu %u bytes iter=%zx\n",
include/linux/netfs.h +3 -3
@@ -185,6 +185,7 @@ struct netfs_io_subrequest {short error;/* 0 or error that occurred */ unsigned short debug_index;/* Index in list (for debugging output) */ unsigned int nr_segs;/* Number of segs in io_iter */ +u8 retry_count;/* The number of retries (0 on initial pass) */ enum netfs_io_source source;/* Where to read from/write to */ unsigned char stream_nr;/* I/O stream this belongs to */ unsigned char curr_folioq_slot;/* Folio currently being read */
@@ -194,14 +195,13 @@ struct netfs_io_subrequest {#define NETFS_SREQ_COPY_TO_CACHE 0/* Set if should copy the data to the cache */ #define NETFS_SREQ_CLEAR_TAIL 1/* Set if the rest of the read should be cleared */ #define NETFS_SREQ_SEEK_DATA_READ 3/* Set if ->read() should SEEK_DATA first */ -#define NETFS_SREQ_NO_PROGRESS 4/* Set if we didn't manage to read any data */ +#define NETFS_SREQ_MADE_PROGRESS 4/* Set if we transferred at least some data */ #define NETFS_SREQ_ONDEMAND 5/* Set if it's from on-demand read mode */ #define NETFS_SREQ_BOUNDARY 6/* Set if ends on hard boundary (eg. ceph object) */ #define NETFS_SREQ_HIT_EOF 7/* Set if short due to EOF */ #define NETFS_SREQ_IN_PROGRESS 8/* Unlocked when the subrequest completes */ #define NETFS_SREQ_NEED_RETRY 9/* Set if the filesystem requests a retry */ -#define NETFS_SREQ_RETRYING 10/* Set if we're retrying */ -#define NETFS_SREQ_FAILED 11/* Set if the subreq failed unretryably */ +#define NETFS_SREQ_FAILED 10/* Set if the subreq failed unretryably */ };enum netfs_io_origin {
David Howells
maintainerPATCH 9/10 netfs: Fix ceph copy to cache on write-begin
In reply to David HowellsAt the end of netfs_unlock_read_folio() in which folios are marked appropriately for copying to the cache (either with by being marked dirty and having their private data set or by having PG_private_2 set) and then unlocked, the folio_queue struct has the entry pointing to the folio cleared. This presents a problem for netfs_pgpriv2_write_to_the_cache(), which is used to write folios marked with PG_private_2 to the cache as it expects to be able to trawl the folio_queue list thereafter to find the relevant folios, leading to a hang.
Fix this by not clearing the folio_queue entry if we're going to do the deprecated copy-to-cache. The clearance will be done instead as the folios are written to the cache.
This can be reproduced by starting cachefiles, mounting a ceph filesystem with "-o fsc" and writing to it.
--- fs/netfs/read_collect.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/netfs/read_collect.c +4 -2
@@ -62,10 +62,14 @@ static void netfs_unlock_read_folio(struct netfs_io_subrequest *subreq, }else {trace_netfs_folio (folio, netfs_folio_trace_read_done); } + +folioq_clear (folioq, slot); }else {// TODO: Use of PG_private_2 is deprecated. if (test_bit (NETFS_SREQ_COPY_TO_CACHE , &subreq->flags))netfs_pgpriv2_mark_copy_to_cache (subreq, rreq, folioq, slot); +else +folioq_clear (folioq, slot); }if (!test_bit (NETFS_RREQ_DONT_UNLOCK_FOLIOS , &rreq->flags)) {
@@ -77,8 +81,6 @@ static void netfs_unlock_read_folio(struct netfs_io_subrequest *subreq,folio_unlock (folio); } } - -folioq_clear (folioq, slot); }/*
David Howells
maintainerPATCH 10/10 netfs: Fix the (non-)cancellation of copy when cache is temporarily disabled
In reply to David HowellsWhen the caching for a cookie is temporarily disabled (e.g. due to a DIO write on that file), future copying to the cache for that file is disabled until all fds open on that file are closed. However, if netfslib is using the deprecated PG_private_2 method (such as is currently used by ceph), and decides it wants to copy to the cache, netfs_advance_write() will just bail at the first check seeing that the cache stream is unavailable, and indicate that it dealt with all the content.
This means that we have no subrequests to provide notifications to drive the state machine or even to pin the request and the request just gets discarded, leaving the folios with PG_private_2 set.
Fix this by jumping directly to cancel the request if the cache is not available. That way, we don't remove mark3 from the folio_queue list and netfs_pgpriv2_cancel() will clean up the folios.
This was found by running the generic/013 xfstest against ceph with an active cache and the "-o fsc" option passed to ceph. That would usually hang
--- fs/netfs/read_pgpriv2.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/netfs/read_pgpriv2.c +4 -0
@@ -170,6 +170,10 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)trace_netfs_write (wreq, netfs_write_trace_copy_to_cache);netfs_stat (&netfs_n_wh_copy_to_cache); +if (!wreq->io_streams[1 ].avail) { +netfs_put_request (wreq, false, netfs_rreq_trace_put_return); + goto couldnt_start; + }for (;;) { error =netfs_pgpriv2_copy_folio (wreq, folio);
David Howells
maintainerceph xfstests failures [was Re: [PATCH 00/10] netfs, ceph, nfs, cachefiles: Miscellaneous fixes/changes]
In reply to David HowellsQuoted 2 lines · David Howells
With these patches, I can run xfstest -g quick to completion on ceph with a local cache.
I should qualify that. The thing completes and doesn't hang, but I get 6 failures:
Failures: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732
Though these don't appear to be anything to do with netfslib (see attached). There are two cases where the mount is busy and the rest seems to be due to id-mapped mounts and/or user namespaces.
The xfstest local.config file looks something like:
export FSTYP=ceph
export TEST_DEV=<ipaddr>:/test
export TEST_DIR=/xfstest.test
TEST_FS_MOUNT_OPTS='-o name=admin,mds_namespace=test,fs=test,fsc' export SCRATCH_DEV=<ipaddr>:/scratch
export SCRATCH_MNT=/xfstest.scratch
export MOUNT_OPTIONS='-o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch'
David
---
# ./check -E .exclude generic/604 generic/633 generic/645 generic/696 generic/697 generic/732 FSTYP -- ceph PLATFORM -- Linux/x86_64 andromeda 6.13.0-rc2-build3+ #5311 SMP Fri Dec 13 09:03:34 GMT 2024 MKFS_OPTIONS -- <ipaddr>:/scratch MOUNT_OPTIONS -- -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.scratch
generic/604 2s ... [failed, exit status 1]- output mismatch (see /root/xfstests-dev/results//generic/604.out.bad)
--- tests/generic/604.out 2024-09-12 12:36:14.187441830 +0100
+++ /root/xfstests-dev/results//generic/604.out.bad 2024-12-13 13:18:51.910900871 +0000
@@ -1,2 +1,4 @@
QA output created by 604
-Silence is golden
+mount error 16 = Device or resource busy
+mount -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.scratch failed +(see /root/xfstests-dev/results//generic/604.full for details)
...
(Run 'diff -u /root/xfstests-dev/tests/generic/604.out /root/xfstests-dev/results//generic/604.out.bad' to see the entire diff)
generic/633 [failed, exit status 1]- output mismatch (see /root/xfstests-dev/results//generic/633.out.bad)
--- tests/generic/633.out 2024-09-12 12:36:14.187441830 +0100
+++ /root/xfstests-dev/results//generic/633.out.bad 2024-12-13 13:18:55.958979531 +0000
@@ -1,2 +1,4 @@
QA output created by 633
Silence is golden
+idmapped-mounts.c: 307: tcore_create_in_userns - Input/output error - failure: open file
+vfstest.c: 2418: run_test - Success - failure: create operations in user namespace
...
(Run 'diff -u /root/xfstests-dev/tests/generic/633.out /root/xfstests-dev/results//generic/633.out.bad' to see the entire diff)
generic/645 [failed, exit status 1]- output mismatch (see /root/xfstests-dev/results//generic/645.out.bad)
--- tests/generic/645.out 2024-09-12 12:36:14.191441810 +0100
+++ /root/xfstests-dev/results//generic/645.out.bad 2024-12-13 13:19:25.526908024 +0000
@@ -1,2 +1,4 @@
QA output created by 645
Silence is golden
+idmapped-mounts.c: 6671: nested_userns - Invalid argument - failure: sys_mount_setattr
+vfstest.c: 2418: run_test - Invalid argument - failure: test that nested user namespaces behave correctly when attached to idmapped mounts
...
(Run 'diff -u /root/xfstests-dev/tests/generic/645.out /root/xfstests-dev/results//generic/645.out.bad' to see the entire diff)
generic/696 - output mismatch (see /root/xfstests-dev/results//generic/696.out.bad)
--- tests/generic/696.out 2024-09-12 12:36:14.195441791 +0100
+++ /root/xfstests-dev/results//generic/696.out.bad 2024-12-13 13:19:30.254804087 +0000
@@ -1,2 +1,6 @@
QA output created by 696
+idmapped-mounts.c: 7763: setgid_create_umask_idmapped - Input/output error - failure: create
+vfstest.c: 2418: run_test - Success - failure: create operations by using umask in directories with setgid bit set on idmapped mount +idmapped-mounts.c: 7763: setgid_create_umask_idmapped - Input/output error - failure: create
+vfstest.c: 2418: run_test - Success - failure: create operations by using umask in directories with setgid bit set on idmapped mount
Silence is golden
...
(Run 'diff -u /root/xfstests-dev/tests/generic/696.out /root/xfstests-dev/results//generic/696.out.bad' to see the entire diff)
HINT: You _MAY_ be missing kernel fix:
ac6800e279a2 fs: Add missing umask strip in vfs_tmpfile 1639a49ccdce fs: move S_ISGID stripping into the vfs_*() helpers
generic/697 - output mismatch (see /root/xfstests-dev/results//generic/697.out.bad)
--- tests/generic/697.out 2024-09-12 12:36:14.195441791 +0100
+++ /root/xfstests-dev/results//generic/697.out.bad 2024-12-13 13:19:31.749225548 +0000
@@ -1,2 +1,4 @@
QA output created by 697
+idmapped-mounts.c: 8218: setgid_create_acl_idmapped - Input/output error - failure: create
+vfstest.c: 2418: run_test - Success - failure: create operations by using acl in directories with setgid bit set on idmapped mount
Silence is golden
...
(Run 'diff -u /root/xfstests-dev/tests/generic/697.out /root/xfstests-dev/results//generic/697.out.bad' to see the entire diff)
HINT: You _MAY_ be missing kernel fix:
1639a49ccdce fs: move S_ISGID stripping into the vfs_*() helpers
generic/732 1s ... [failed, exit status 1]- output mismatch (see /root/xfstests-dev/results//generic/732.out.bad)
--- tests/generic/732.out 2024-09-12 12:36:14.195441791 +0100
+++ /root/xfstests-dev/results//generic/732.out.bad 2024-12-13 13:19:34.482858235 +0000
@@ -1,2 +1,5 @@
QA output created by 732
Silence is golden
+mount error 16 = Device or resource busy
+mount -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.test/mountpoint2-732 failed +(see /root/xfstests-dev/results//generic/732.full for details)
...
(Run 'diff -u /root/xfstests-dev/tests/generic/732.out /root/xfstests-dev/results//generic/732.out.bad' to see the entire diff)
Ran: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732
Failures: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732
Failed 6 of 6 tests
Re: ceph xfstests failures [was Re: [PATCH 00/10] netfs, ceph, nfs, cachefiles: Miscellaneous fixes/changes]
Hey David.
Thanks, for the find. I've seen your mail, but it was a busy week. If you can, please open a https://tracker.ceph.com/ bug and assign it to me.
Quoted 110 lines · David Howells
Quoted 2 lines · David Howells
With these patches, I can run xfstest -g quick to completion on ceph with a local cache.
I should qualify that. The thing completes and doesn't hang, but I get 6 failures:
Failures: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732Though these don't appear to be anything to do with netfslib (see attached). There are two cases where the mount is busy and the rest seems to be due to id-mapped mounts and/or user namespaces.
The xfstest local.config file looks something like:
export FSTYP=ceph
export TEST_DEV=<ipaddr>:/test
export TEST_DIR=/xfstest.test
TEST_FS_MOUNT_OPTS='-o name=admin,mds_namespace=test,fs=test,fsc' export SCRATCH_DEV=<ipaddr>:/scratch
export SCRATCH_MNT=/xfstest.scratch
export MOUNT_OPTIONS='-o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch'David
---# ./check -E .exclude generic/604 generic/633 generic/645 generic/696 generic/697 generic/732 FSTYP -- ceph PLATFORM -- Linux/x86_64 andromeda 6.13.0-rc2-build3+ #5311 SMP Fri Dec 13 09:03:34 GMT 2024 MKFS_OPTIONS -- <ipaddr>:/scratch MOUNT_OPTIONS -- -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.scratchgeneric/604 2s ... [failed, exit status 1]- output mismatch (see
/root/xfstests-dev/results//generic/604.out.bad)
--- tests/generic/604.out 2024-09-12 12:36:14.187441830 +0100
+++/root/xfstests-dev/results//generic/604.out.bad2024-12-13 13:18:51.910900871 +0000
@@ -1,2 +1,4 @@
QA output created by 604
-Silence is golden
+mount error 16 = Device or resource busy
+mount -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.scratch failed +(see/root/xfstests-dev/results//generic/604.fullfor details)
...
(Run 'diff -u/root/xfstests-dev/tests/generic/604.out/root/xfstests-dev/results//generic/604.out.bad' to see the entire diff)
generic/633 [failed, exit status 1]- output mismatch (see/root/xfstests-dev/results//generic/633.out.bad)
--- tests/generic/633.out 2024-09-12 12:36:14.187441830 +0100
+++/root/xfstests-dev/results//generic/633.out.bad2024-12-13 13:18:55.958979531 +0000
@@ -1,2 +1,4 @@
QA output created by 633
Silence is golden
+idmapped-mounts.c: 307:tcore_create_in_userns- Input/output error - failure: open file
+vfstest.c: 2418:run_test- Success - failure: create operations in user namespace
...
(Run 'diff -u/root/xfstests-dev/tests/generic/633.out/root/xfstests-dev/results//generic/633.out.bad' to see the entire diff)
generic/645 [failed, exit status 1]- output mismatch (see/root/xfstests-dev/results//generic/645.out.bad)
--- tests/generic/645.out 2024-09-12 12:36:14.191441810 +0100
+++/root/xfstests-dev/results//generic/645.out.bad2024-12-13 13:19:25.526908024 +0000
@@ -1,2 +1,4 @@
QA output created by 645
Silence is golden
+idmapped-mounts.c: 6671:nested_userns- Invalid argument - failure:sys_mount_setattr
+vfstest.c: 2418:run_test- Invalid argument - failure: test that nested user namespaces behave correctly when attached to idmapped mounts
...
(Run 'diff -u/root/xfstests-dev/tests/generic/645.out/root/xfstests-dev/results//generic/645.out.bad' to see the entire diff)
generic/696 - output mismatch (see/root/xfstests-dev/results//generic/696.out.bad)
--- tests/generic/696.out 2024-09-12 12:36:14.195441791 +0100
+++/root/xfstests-dev/results//generic/696.out.bad2024-12-13 13:19:30.254804087 +0000
@@ -1,2 +1,6 @@
QA output created by 696
+idmapped-mounts.c: 7763:setgid_create_umask_idmapped- Input/output error - failure: create
+vfstest.c: 2418:run_test- Success - failure: create operations by using umask in directories with setgid bit set on idmapped mount+idmapped-mounts.c: 7763:setgid_create_umask_idmapped- Input/output error - failure: create
+vfstest.c: 2418:run_test- Success - failure: create operations by using umask in directories with setgid bit set on idmapped mount
Silence is golden
...
(Run 'diff -u/root/xfstests-dev/tests/generic/696.out/root/xfstests-dev/results//generic/696.out.bad' to see the entire diff)HINT: You _MAY_ be missing kernel fix:
ac6800e279a2fs: Add missing umask strip invfs_tmpfile1639a49ccdcefs: moveS_ISGIDstripping into the vfs_*() helpersgeneric/697 - output mismatch (see
/root/xfstests-dev/results//generic/697.out.bad)
--- tests/generic/697.out 2024-09-12 12:36:14.195441791 +0100
+++/root/xfstests-dev/results//generic/697.out.bad2024-12-13 13:19:31.749225548 +0000
@@ -1,2 +1,4 @@
QA output created by 697
+idmapped-mounts.c: 8218:setgid_create_acl_idmapped- Input/output error - failure: create
+vfstest.c: 2418:run_test- Success - failure: create operations by using acl in directories with setgid bit set on idmapped mount
Silence is golden
...
(Run 'diff -u/root/xfstests-dev/tests/generic/697.out/root/xfstests-dev/results//generic/697.out.bad' to see the entire diff)HINT: You _MAY_ be missing kernel fix:
1639a49ccdcefs: moveS_ISGIDstripping into the vfs_*() helpersgeneric/732 1s ... [failed, exit status 1]- output mismatch (see
/root/xfstests-dev/results//generic/732.out.bad)
--- tests/generic/732.out 2024-09-12 12:36:14.195441791 +0100
+++/root/xfstests-dev/results//generic/732.out.bad2024-12-13 13:19:34.482858235 +0000
@@ -1,2 +1,5 @@
QA output created by 732
Silence is golden
+mount error 16 = Device or resource busy
+mount -o name=admin,mds_namespace=scratch,fs=scratch,fsc=scratch -o context=system_u:object_r:root_t:s0 <ipaddr>:/scratch /xfstest.test/mountpoint2-732 failed +(see/root/xfstests-dev/results//generic/732.fullfor details)
...
(Run 'diff -u/root/xfstests-dev/tests/generic/732.out/root/xfstests-dev/results//generic/732.out.bad' to see the entire diff)
Ran: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732
Failures: generic/604 generic/633 generic/645 generic/696 generic/697 generic/732
Failed 6 of 6 tests
David Howells
maintainer[PATCH 11/10] netfs: Fix is-caching check in read-retry
In reply to David Howellsnetfs: Fix is-caching check in read-retry
The read-retry code checks the NETFS_RREQ_COPY_TO_CACHE flag to determine if there might be failed reads from the cache that need turning into reads from the server, with the intention of skipping the complicated part if it can. The code that set the flag, however, got lost during the read-side rewrite.
Fix the check to see if the cache_resources are valid instead. The flag can then be removed.
--- fs/netfs/read_retry.c | 2 +- include/linux/netfs.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
fs/netfs/read_retry.c +1 -1
@@ -49,7 +49,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)* up to the first permanently failed one. */ if (!rreq->netfs_ops->prepare_read && - !test_bit (NETFS_RREQ_COPY_TO_CACHE , &rreq->flags)) { + !rreq->cache_resources.ops) {struct netfs_io_subrequest *subreq;list_for_each_entry (subreq, &rreq->subrequests, rreq_link) {
include/linux/netfs.h +0 -1
@@ -269,7 +269,6 @@ struct netfs_io_request {size_t prev_donated;/* Fallback for subreq->prev_donated */ refcount_t ref;unsigned long flags; -#define NETFS_RREQ_COPY_TO_CACHE 1/* Need to write to the cache */ #define NETFS_RREQ_NO_UNLOCK_FOLIO 2/* Don't unlock no_unlock_folio on completion */ #define NETFS_RREQ_DONT_UNLOCK_FOLIOS 3/* Don't unlock the folios on completion */ #define NETFS_RREQ_FAILED 4/* The request failed */