netfs: Read performance improvements and "single-blob" support
Subsystems (15)
David Howells
maintainerv3 cover netfs: Read performance improvements and "single-blob" support
Hi Christian, Steve, Willy,
This set of patches is primarily about two things: improving read performance and supporting monolithic single-blob objects that have to be read/written as such (e.g. AFS directory contents). The implementation of the two parts is interwoven as each makes the other possible.
READ PERFORMANCE
================
The read performance improvements are intended to speed up some loss of performance detected in cifs and to a lesser extend in afs. The problem is that we queue too many work items during the collection of read results:
each individual subrequest is collected by its own work item, and then they have to interact with each other when a series of subrequests don't exactly align with the pattern of folios that are being read by the overall request.
Whilst the processing of the pages covered by individual subrequests as they complete potentially allows folios to be woken in parallel and with minimum delay, it can shuffle wakeups for sequential reads out of order - and that is the most common I/O pattern.
The final assessment and cleanup of an operation is then held up until the last I/O completes - and for a synchronous sequential operation, this means the bouncing around of work items just adds latency.
Two changes have been made to make this work:
(1) All collection is now done in a single "work item" that works progressively through the subrequests as they complete (and also dispatches retries as necessary).
(2) For readahead and AIO, this work item be done on a workqueue and can run in parallel with the ultimate consumer of the data; for synchronous direct or unbuffered reads, the collection is run in the application thread and not offloaded.
Functions such as smb2_readv_callback() then just tell netfslib that the subrequest has terminated; netfslib does a minimal bit of processing on the spot - stat counting and tracing mostly - and then queues/wakes up the worker. This simplifies the logic as the collector just walks sequentially through the subrequests as they complete and walks through the folios, if buffered, unlocking them as it goes. It also keeps to a minimum the amount of latency injected into the filesystem's low-level I/O handling
SINGLE-BLOB OBJECT SUPPORT
==========================
Single-blob objects are files for which the content of the file must be read from or written to the server in a single operation because reading them in parts may yield inconsistent results. AFS directories are an example of this as there exists the possibility that the contents are generated on the fly and would differ between reads or might change due to third party interference.
Such objects will be written to and retrieved from the cache if one is present, though we allow/may need to propose multiple subrequests to do so. The important part is that read from/write to the *server* is monolithic.
Single blob reading is, for the moment, fully synchronous and does result collection in the application thread and, also for the moment, the API is supplied the buffer in the form of a folio_queue chain rather than using the pagecache.
AFS CHANGES
===========
This series makes a number of changes to the kafs filesystem, primarily in the area of directory handling:
(1) AFS's FetchData RPC reply processing is made partially asynchronous which allows the netfs_io_request's outstanding operation counter to be removed as part of reducing the collection to a single work item.
(2) Directory and symlink reading are plumbed through netfslib using the single-blob object API and are now cacheable with fscache. This also allows the afs_read struct to be eliminated and netfs_io_subrequest to be used directly instead.
(3) Directory and symlink content are now stored in a folio_queue buffer rather than in the pagecache. This means we don't require the RCU read lock and xarray iteration to access it, and folios won't randomly disappear under us because the VM wants them back.
There are some downsides to this, though: the storage folios are no longer known to the VM, drop_caches can't flush them, the folios are not migrateable. The inode must also be marked dirty manually to get the data written to the cache in the background.
(4) The vnode operation lock is changed from a mutex struct to a private lock implementation. The problem is that the lock now needs to be dropped in a separate thread and mutexes don't permit that.
(5) When a new directory or symlink is created, we now initialise it locally and mark it valid rather than downloading it (we know what it's likely to look like).
(6) We now use the in-directory hashtable to reduce the number of entries we need to scan when doing a lookup. The edit routines have to maintain the hash chains.
(7) Cancellation (e.g. by signal) of an async call after the rxrpc_call has been set up is now offloaded to the worker thread as there will be a notification from rxrpc upon completion. This avoids a double cleanup.
SUPPORTING CHANGES
==================
To support the above some other changes are also made:
(1) A "rolling buffer" implementation is created to abstract out the two separate folio_queue chaining implementations I had (one for read and one for write).
(2) Functions are provided to create/extend a buffer in a folio_queue chain and tear it down again. This is used to handle AFS directories, but could also be used to create bounce buffers for content crypto and transport crypto.
(3) The was_async argument is dropped from netfs_read_subreq_terminated(). Instead we wake the read collection work item by either queuing it or waking up the app thread.
(4) We don't need to use BH-excluding locks when communicating between the issuing thread and the collection thread as neither of them now run in BH context.
MISCELLANY
==========
Also included are some fixes from Matthew Wilcox that need to be applied first; a number of new tracepoints; a split of the netfslib write collection code to put retrying into its own file (it gets more complicated with content encryption).
There are also some minor fixes AFS included, including fixing the AFS directory format struct layout, reducing some directory over-invalidation and making afs_mkdir() translate EEXIST to ENOTEMPY (which is not available on all systems the servers support).
Finally, there's a patch to try and detect entry into the folio unlock function with no folio_queue structs in the buffer (which isn't allowed in the cases that can get there). This is a debugging patch, but should be minimal overhead.
The patches can also be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-writeback
CHANGES
=======
ver #3)
- In afs, fix a number of issues in asynchronous FetchData handling.
- In afs, fix a signedness issue in an error check dir editing.
- To afs, add a patch to fix handling of interruption by a signal whilst an async call is being set up.
ver #2)
- Handle that we might be in RCU pathwalk in afs_get_link().
- Fix double-call of afs_put_operation() in async FetchData.
- Don't set ->mapping on directory and symlink folios as they're not in the pagecache.
- Add an afs patch to search the directory's hash table on lookup.
- Add an afs patch to preset the contents of a new symlink on creation.
- Add an afs patch to add a tracepoint in the async FetchData response processing.
- Add a patch to report if a NULL folio_queue pointer is seen in netfs_writeback_unlock_folios().
Thanks,
David
David Howells (30):
kheaders: Ignore silly-rename files
netfs: Use a folio_queue allocation and free functions
netfs: Add a tracepoint to log the lifespan of folio_queue structs
netfs: Abstract out a rolling folio buffer implementation
netfs: Make netfs_advance_write() return size_t
netfs: Split retry code out of fs/netfs/write_collect.c
netfs: Drop the error arg from netfs_read_subreq_terminated()
netfs: Drop the was_async arg from netfs_read_subreq_terminated()
netfs: Don't use bh spinlock
afs: Don't use mutex for I/O operation lock
afs: Fix EEXIST error returned from afs_rmdir() to be ENOTEMPTY
afs: Fix directory format encoding struct
netfs: Remove some extraneous directory invalidations
cachefiles: Add some subrequest tracepoints
cachefiles: Add auxiliary data trace
afs: Add more tracepoints to do with tracking validity
netfs: Add functions to build/clean a buffer in a folio_queue
netfs: Add support for caching single monolithic objects such as AFS
dirs
afs: Make afs_init_request() get a key if not given a file
afs: Use netfslib for directories
afs: Use netfslib for symlinks, allowing them to be cached
afs: Eliminate afs_read
afs: Fix cleanup of immediately failed async calls
afs: Make {Y,}FS.FetchData an asynchronous operation
netfs: Change the read result collector to only use one work item
afs: Make afs_mkdir() locally initialise a new directory's content
afs: Use the contained hashtable to search a directory
afs: Locally initialise the contents of a new symlink on creation
afs: Add a tracepoint for afs_read_receive()
netfs: Report on NULL folioq in netfs_writeback_unlock_folios()
Matthew Wilcox (Oracle) (3):
netfs: Remove call to folio_index()
netfs: Fix a few minor bugs in netfs_page_mkwrite()
netfs: Remove unnecessary references to pages
fs/9p/vfs_addr.c | 8 +- fs/afs/Makefile | 1 + fs/afs/callback.c | 4 +- fs/afs/dir.c | 803 +++++++++++++++--------------- fs/afs/dir_edit.c | 382 ++++++++------ fs/afs/dir_search.c | 227 +++++++++ fs/afs/file.c | 260 +++++----- fs/afs/fs_operation.c | 113 ++++- fs/afs/fsclient.c | 62 +-- fs/afs/inode.c | 138 ++++- fs/afs/internal.h | 142 ++++-- fs/afs/main.c | 2 +- fs/afs/mntpt.c | 22 +- fs/afs/rotate.c | 4 +- fs/afs/rxrpc.c | 37 +- fs/afs/super.c | 4 +- fs/afs/validation.c | 31 +- fs/afs/vlclient.c | 1 + fs/afs/write.c | 16 +- fs/afs/xdr_fs.h | 2 +- fs/afs/yfsclient.c | 49 +- fs/cachefiles/io.c | 4 + fs/cachefiles/xattr.c | 9 +- fs/ceph/addr.c | 13 +- fs/netfs/Makefile | 5 +- fs/netfs/buffered_read.c | 276 ++++------ fs/netfs/buffered_write.c | 41 +- fs/netfs/direct_read.c | 80 +-- fs/netfs/direct_write.c | 10 +- fs/netfs/internal.h | 36 +- fs/netfs/main.c | 6 +- fs/netfs/misc.c | 163 +++--- fs/netfs/objects.c | 21 +- fs/netfs/read_collect.c | 703 +++++++++++++++----------- fs/netfs/read_pgpriv2.c | 35 +- fs/netfs/read_retry.c | 209 ++++---- fs/netfs/read_single.c | 195 ++++++++ fs/netfs/rolling_buffer.c | 225 +++++++++ fs/netfs/stats.c | 4 +- fs/netfs/write_collect.c | 278 ++--------- fs/netfs/write_issue.c | 239 ++++++++- fs/netfs/write_retry.c | 233 +++++++++ fs/nfs/fscache.c | 6 +- fs/nfs/fscache.h | 3 +- fs/smb/client/cifssmb.c | 12 +- fs/smb/client/file.c | 3 +- fs/smb/client/smb2ops.c | 2 +- fs/smb/client/smb2pdu.c | 14 +- include/linux/folio_queue.h | 12 +- include/linux/netfs.h | 55 +- include/linux/rolling_buffer.h | 61 +++ include/trace/events/afs.h | 210 +++++++- include/trace/events/cachefiles.h | 13 +- include/trace/events/netfs.h | 97 ++-- kernel/gen_kheaders.sh | 1 + lib/kunit_iov_iter.c | 4 +- 56 files changed, 3635 insertions(+), 1951 deletions(-) create mode 100644 fs/afs/dir_search.c create mode 100644 fs/netfs/read_single.c create mode 100644 fs/netfs/rolling_buffer.c create mode 100644 fs/netfs/write_retry.c create mode 100644 include/linux/rolling_buffer.h
David Howells
maintainerPATCH v3 1/33 kheaders: Ignore silly-rename files
Tell tar to ignore silly-rename files (".__afs*" and ".nfs*") when building the header archive. These occur when a file that is open is unlinked locally, but hasn't yet been closed. Such files are visible to the user via the getdents() syscall and so programs may want to do things with them.
During the kernel build, such files may be made during the processing of header files and the cleanup may get deferred by fput() which may result in tar seeing these files when it reads the directory, but they may have disappeared by the time it tries to open them, causing tar to fail with an error. Further, we don't want to include them in the tarball if they still exist.
With CONFIG_HEADERS_INSTALL=y, something like the following may be seen:
find: './kernel/.tmp_cpio_dir/include/dt-bindings/reset/.__afs2080': No such file or directory
tar: ./include/linux/greybus/.__afs3C95: File removed before we read it
The find warning doesn't seem to cause a problem.
Fix this by telling tar when called from in gen_kheaders.sh to exclude such files. This only affects afs and nfs; cifs uses the Windows Hidden attribute to prevent the file from being seen.
--- kernel/gen_kheaders.sh | 1 + 1 file changed, 1 insertion(+)
kernel/gen_kheaders.sh +1 -0
@@ -89,6 +89,7 @@ find $cpio_dir -type f -print0 | # Create archive and try to normalize metadata for reproducibility. tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \ + --exclude=".__afs*" --exclude=".nfs*" \ --owner=0 --group=0 --sort=name --numeric-owner --mode=u=rw,go=r,a+X \ -I $XZ -cf $tarfile -C $cpio_dir/ . > /dev/null
David Howells
maintainerPATCH v3 2/33 netfs: Remove call to folio_index()
In reply to David HowellsFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Calling folio_index() is pointless overhead; directly dereferencing folio->index is fine.
--- include/trace/events/netfs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/trace/events/netfs.h +1 -1
@@ -450,7 +450,7 @@ TRACE_EVENT(netfs_folio,struct address_space *__m =READ_ONCE (folio->mapping); __entry->ino = __m ? __m->host->i_ino :0 ; __entry->why = why; - __entry->index =folio_index (folio); + __entry->index = folio->index; __entry->nr =folio_nr_pages (folio); ),
David Howells
maintainerPATCH v3 3/33 netfs: Fix a few minor bugs in netfs_page_mkwrite()
In reply to David HowellsFrom: "Matthew Wilcox (Oracle)" <[email protected]>
We can't return with VM_FAULT_SIGBUS | VM_FAULT_LOCKED; the core code will not unlock the folio in this instance. Introduce a new "unlock" error exit to handle this case. Use it to handle the "folio is truncated" check, and change the "writeback interrupted by a fatal signal" to do a NOPAGE exit instead of letting the core code install the folio currently under writeback before killing the process.
--- fs/netfs/buffered_write.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
fs/netfs/buffered_write.c +13 -14
@@ -491,7 +491,9 @@ EXPORT_SYMBOL(netfs_file_write_iter);/* * Notification that a previously read-only page is about to become writable. -* Note that the caller indicates a single page of a multipage folio. +* The caller indicates the precise page that needs to be written to, but +* we only track group on a per-folio basis, so we block more often than +* we might otherwise. */ vm_fault_t netfs_page_mkwrite (struct vm_fault *vmf,struct netfs_group *netfs_group) {
@@ -501,7 +503,7 @@ vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_grstruct address_space *mapping = file->f_mapping;struct inode *inode =file_inode (file);struct netfs_inode *ictx =netfs_inode (inode); -vm_fault_t ret =VM_FAULT_RETRY ; +vm_fault_t ret =VM_FAULT_NOPAGE ;int err;_enter ("%lx" , folio->index);
@@ -510,21 +512,15 @@ vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_grif (folio_lock_killable (folio) <0 ) goto out; -if (folio->mapping != mapping) { -folio_unlock (folio); - ret =VM_FAULT_NOPAGE ; - goto out; - } - -if (folio_wait_writeback_killable (folio)) { - ret =VM_FAULT_LOCKED ; - goto out; - } +if (folio->mapping != mapping) + goto unlock; +if (folio_wait_writeback_killable (folio) <0 ) + goto unlock;/* Can we see a streaming write here? */ if (WARN_ON (!folio_test_uptodate (folio))) { - ret =VM_FAULT_SIGBUS |VM_FAULT_LOCKED ; - goto out; + ret =VM_FAULT_SIGBUS ; + goto unlock; } group =netfs_folio_group (folio);
@@ -559,5 +555,8 @@ vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_gr out:sb_end_pagefault (inode->i_sb);return ret; +unlock: +folio_unlock (folio); + goto out; }EXPORT_SYMBOL (netfs_page_mkwrite);
David Howells
maintainerPATCH v3 4/33 netfs: Remove unnecessary references to pages
In reply to David HowellsFrom: "Matthew Wilcox (Oracle)" <[email protected]>
These places should all use folios instead of pages.
--- fs/netfs/buffered_read.c | 8 ++++---- fs/netfs/buffered_write.c | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-)
fs/netfs/buffered_read.c +4 -4
@@ -627,7 +627,7 @@ static bool netfs_skip_folio_read(struct folio *folio, loff_t pos, size_t len,if (unlikely (always_fill)) {if (pos - offset + len <= i_size)return false;/* Page entirely before EOF */ -zero_user_segment (&folio->page,0 , plen); +folio_zero_segment (folio,0 , plen);folio_mark_uptodate (folio);return true; }
@@ -646,7 +646,7 @@ static bool netfs_skip_folio_read(struct folio *folio, loff_t pos, size_t len,return false; zero_out: -zero_user_segments (&folio->page,0 , offset, offset + len, plen); +folio_zero_segments (folio,0 , offset, offset + len, plen);return true; }
@@ -713,7 +713,7 @@ int netfs_write_begin(struct netfs_inode *ctx,if (folio_test_uptodate (folio)) goto have_folio; -/* If the page is beyond the EOF, we want to clear it - unless it's +/* If the folio is beyond the EOF, we want to clear it - unless it's * within the cache granule containing the EOF, in which case we need * to preload the granule. */
@@ -773,7 +773,7 @@ int netfs_write_begin(struct netfs_inode *ctx,EXPORT_SYMBOL (netfs_write_begin);/* -* Preload the data into a page we're proposing to write into. +* Preload the data into a folio we're proposing to write into. */ int netfs_prefetch_for_write (struct file *file,struct folio *folio,size_t offset,size_t len)
fs/netfs/buffered_write.c +7 -7
@@ -83,13 +83,13 @@ static void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,* netfs_perform_write - Copy data into the pagecache. * @iocb: The operation parameters * @iter: The source buffer -* @netfs_group: Grouping for dirty pages (eg. ceph snaps). +* @netfs_group: Grouping for dirty folios (eg. ceph snaps). * -* Copy data into pagecache pages attached to the inode specified by @iocb. +* Copy data into pagecache folios attached to the inode specified by @iocb. * The caller must hold appropriate inode locks. * -* Dirty pages are tagged with a netfs_folio struct if they're not up to date -* to indicate the range modified. Dirty pages may also be tagged with a +* Dirty folios are tagged with a netfs_folio struct if they're not up to date +* to indicate the range modified. Dirty folios may also be tagged with a * netfs-specific grouping such that data from an old group gets flushed before * a new one is started. */
@@ -223,11 +223,11 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,* we try to read it. */ if (fpos >= ctx->zero_point) { -zero_user_segment (&folio->page,0 , offset); +folio_zero_segment (folio,0 , offset); copied =copy_folio_from_iter_atomic (folio, offset, part, iter);if (unlikely (copied ==0 )) goto copy_failed; -zero_user_segment (&folio->page, offset + copied, flen); +folio_zero_segment (folio, offset + copied, flen);__netfs_set_group (folio, netfs_group);folio_mark_uptodate (folio);trace_netfs_folio (folio, netfs_modify_and_clear);
@@ -407,7 +407,7 @@ EXPORT_SYMBOL(netfs_perform_write);* netfs_buffered_write_iter_locked - write data to a file * @iocb: IO state structure (file, offset, etc.) * @from: iov_iter with data to write -* @netfs_group: Grouping for dirty pages (eg. ceph snaps). +* @netfs_group: Grouping for dirty folios (eg. ceph snaps). * * This function does all the work needed for actually writing data to a * file. It does all basic checks, removes SUID from the file, updates
David Howells
maintainerPATCH v3 5/33 netfs: Use a folio_queue allocation and free functions
In reply to David HowellsProvide and use folio_queue allocation and free functions to combine the allocation, initialisation and stat (un)accounting steps that are repeated in several places.
--- fs/netfs/buffered_read.c | 12 +++--------- fs/netfs/misc.c | 38 ++++++++++++++++++++++++++++++++++---- include/linux/netfs.h | 5 +++++ 3 files changed, 42 insertions(+), 13 deletions(-)
fs/netfs/buffered_read.c +3 -9
@@ -131,11 +131,9 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)struct folio_queue *tail = rreq->buffer_tail, *new;size_t added; - new =kmalloc (sizeof (*new),GFP_NOFS ); + new =netfs_folioq_alloc (GFP_NOFS );if (!new)return -ENOMEM ; -netfs_stat (&netfs_n_folioq); -folioq_init (new); new->prev = tail; tail->next = new; rreq->buffer_tail = new;
@@ -359,11 +357,9 @@ static int netfs_prime_buffer(struct netfs_io_request *rreq)struct folio_batch put_batch;size_t added; - folioq =kmalloc (sizeof (*folioq),GFP_KERNEL ); + folioq =netfs_folioq_alloc (GFP_KERNEL );if (!folioq)return -ENOMEM ; -netfs_stat (&netfs_n_folioq); -folioq_init (folioq); rreq->buffer = folioq; rreq->buffer_tail = folioq; rreq->submitted = rreq->start;
@@ -436,12 +432,10 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo {struct folio_queue *folioq; - folioq =kmalloc (sizeof (*folioq),GFP_KERNEL ); + folioq =netfs_folioq_alloc (GFP_KERNEL );if (!folioq)return -ENOMEM ; -netfs_stat (&netfs_n_folioq); -folioq_init (folioq);folioq_append (folioq, folio);BUG_ON (folioq_folio (folioq,0 ) != folio);BUG_ON (folioq_folio_order (folioq,0 ) !=folio_order (folio));
fs/netfs/misc.c +34 -4
@@ -8,6 +8,38 @@#include <linux/swap.h> #include "internal.h" +/** +* netfs_folioq_alloc - Allocate a folio_queue struct +* @gfp: Allocation constraints +* +* Allocate, initialise and account the folio_queue struct. +*/ +struct folio_queue *netfs_folioq_alloc (gfp_t gfp) +{ +struct folio_queue *fq; + + fq =kmalloc (sizeof (*fq), gfp); +if (fq) { +netfs_stat (&netfs_n_folioq); +folioq_init (fq); + } +return fq; +} +EXPORT_SYMBOL (netfs_folioq_alloc); + +/** +* netfs_folioq_free - Free a folio_queue struct +* @folioq: The object to free +* +* Free and unaccount the folio_queue struct. +*/ +void netfs_folioq_free (struct folio_queue *folioq) +{ +netfs_stat_d (&netfs_n_folioq); +kfree (folioq); +} +EXPORT_SYMBOL (netfs_folioq_free); +/* * Make sure there's space in the rolling queue. */
@@ -87,8 +119,7 @@ struct folio_queue *netfs_delete_buffer_head(struct netfs_io_request *wreq)if (next) next->prev =NULL ; -netfs_stat_d (&netfs_n_folioq); -kfree (head); +netfs_folioq_free (head); wreq->buffer = next;return next; }
@@ -111,8 +142,7 @@ void netfs_clear_buffer(struct netfs_io_request *rreq)folio_put (folio); } } -netfs_stat_d (&netfs_n_folioq); -kfree (p); +netfs_folioq_free (p); } }
include/linux/netfs.h +5 -0
@@ -21,6 +21,7 @@enum netfs_sreq_ref_trace ;typedef struct mempool_s mempool_t ; +struct folio_queue ;/** * folio_start_private_2 - Start an fscache write on a folio. [DEPRECATED]
@@ -454,6 +455,10 @@ void netfs_end_io_write(struct inode *inode);int netfs_start_io_direct (struct inode *inode);void netfs_end_io_direct (struct inode *inode); +/* Miscellaneous APIs. */ +struct folio_queue *netfs_folioq_alloc (gfp_t gfp); +void netfs_folioq_free (struct folio_queue *folioq); +/** * netfs_inode - Get the netfs inode context from the inode * @inode: The inode to query
David Howells
maintainerPATCH v3 6/33 netfs: Add a tracepoint to log the lifespan of folio_queue structs
In reply to David HowellsAdd a tracepoint to log the lifespan of folio_queue structs. For tracing illustrative purposes, folio_queues are tagged with the debug ID of whatever they're related to (typically a netfs_io_request) and a debug ID of their own.
--- fs/netfs/buffered_read.c | 10 ++++++--- fs/netfs/internal.h | 3 ++- fs/netfs/misc.c | 31 +++++++++++++++++---------- fs/netfs/read_collect.c | 8 +++++-- fs/netfs/write_issue.c | 2 +- fs/smb/client/smb2ops.c | 2 +- include/linux/folio_queue.h | 12 ++++++++--- include/linux/netfs.h | 6 ++++-- include/trace/events/netfs.h | 41 ++++++++++++++++++++++++++++++++++-- lib/kunit_iov_iter.c | 4 ++-- 10 files changed, 91 insertions(+), 28 deletions(-)
fs/netfs/buffered_read.c +7 -3
@@ -131,7 +131,8 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)struct folio_queue *tail = rreq->buffer_tail, *new;size_t added; - new =netfs_folioq_alloc (GFP_NOFS ); + new =netfs_folioq_alloc (rreq->debug_id,GFP_NOFS , + netfs_trace_folioq_alloc_read_prep);if (!new)return -ENOMEM ; new->prev = tail;
@@ -357,9 +358,11 @@ static int netfs_prime_buffer(struct netfs_io_request *rreq)struct folio_batch put_batch;size_t added; - folioq =netfs_folioq_alloc (GFP_KERNEL ); + folioq =netfs_folioq_alloc (rreq->debug_id,GFP_KERNEL , + netfs_trace_folioq_alloc_read_prime);if (!folioq)return -ENOMEM ; + rreq->buffer = folioq; rreq->buffer_tail = folioq; rreq->submitted = rreq->start;
@@ -432,7 +435,8 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo {struct folio_queue *folioq; - folioq =netfs_folioq_alloc (GFP_KERNEL ); + folioq =netfs_folioq_alloc (rreq->debug_id,GFP_KERNEL , + netfs_trace_folioq_alloc_read_sing);if (!folioq)return -ENOMEM ;
fs/netfs/internal.h +2 -1
@@ -58,7 +58,8 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}/* * misc.c */ -struct folio_queue *netfs_buffer_make_space (struct netfs_io_request *rreq); +struct folio_queue *netfs_buffer_make_space (struct netfs_io_request *rreq, +enum netfs_folioq_trace trace);int netfs_buffer_append_folio (struct netfs_io_request *rreq,struct folio *folio,bool needs_put);struct folio_queue *netfs_delete_buffer_head (struct netfs_io_request *wreq);
fs/netfs/misc.c +20 -11
@@ -10,18 +10,25 @@/** * netfs_folioq_alloc - Allocate a folio_queue struct +* @rreq_id: Associated debugging ID for tracing purposes * @gfp: Allocation constraints +* @trace: Trace tag to indicate the purpose of the allocation * -* Allocate, initialise and account the folio_queue struct. +* Allocate, initialise and account the folio_queue struct and log a trace line +* to mark the allocation. */ -struct folio_queue *netfs_folioq_alloc (gfp_t gfp) +struct folio_queue *netfs_folioq_alloc (unsigned int rreq_id,gfp_t gfp, +unsigned int /*enum netfs_folioq_trace*/ trace) { +static atomic_t debug_ids;struct folio_queue *fq; fq =kmalloc (sizeof (*fq), gfp);if (fq) {netfs_stat (&netfs_n_folioq); -folioq_init (fq); +folioq_init (fq, rreq_id); + fq->debug_id =atomic_inc_return (&debug_ids); +trace_netfs_folioq (fq, trace); }return fq; }
@@ -30,11 +37,14 @@ EXPORT_SYMBOL(netfs_folioq_alloc);/** * netfs_folioq_free - Free a folio_queue struct * @folioq: The object to free +* @trace: Trace tag to indicate which free * * Free and unaccount the folio_queue struct. */ -void netfs_folioq_free (struct folio_queue *folioq) +void netfs_folioq_free (struct folio_queue *folioq, +unsigned int /*enum netfs_trace_folioq*/ trace) { +trace_netfs_folioq (folioq, trace);netfs_stat_d (&netfs_n_folioq);kfree (folioq); }
@@ -43,7 +53,8 @@ EXPORT_SYMBOL(netfs_folioq_free);/* * Make sure there's space in the rolling queue. */ -struct folio_queue *netfs_buffer_make_space (struct netfs_io_request *rreq) +struct folio_queue *netfs_buffer_make_space (struct netfs_io_request *rreq, +enum netfs_folioq_trace trace) {struct folio_queue *tail = rreq->buffer_tail, *prev;unsigned int prev_nr_slots =0 ;
@@ -59,11 +70,9 @@ struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq) prev_nr_slots =folioq_nr_slots (tail); } - tail =kmalloc (sizeof (*tail),GFP_NOFS ); + tail =netfs_folioq_alloc (rreq->debug_id,GFP_NOFS , trace);if (!tail)return ERR_PTR (-ENOMEM ); -netfs_stat (&netfs_n_folioq); -folioq_init (tail); tail->prev = prev;if (prev)/* [!] NOTE: After we set prev->next, the consumer is entirely
@@ -98,7 +107,7 @@ int netfs_buffer_append_folio(struct netfs_io_request *rreq, struct folio *foliostruct folio_queue *tail;unsigned int slot, order =folio_order (folio); - tail =netfs_buffer_make_space (rreq); + tail =netfs_buffer_make_space (rreq, netfs_trace_folioq_alloc_append_folio);if (IS_ERR (tail))return PTR_ERR (tail);
@@ -119,7 +128,7 @@ struct folio_queue *netfs_delete_buffer_head(struct netfs_io_request *wreq)if (next) next->prev =NULL ; -netfs_folioq_free (head); +netfs_folioq_free (head, netfs_trace_folioq_delete); wreq->buffer = next;return next; }
@@ -142,7 +151,7 @@ void netfs_clear_buffer(struct netfs_io_request *rreq)folio_put (folio); } } -netfs_folioq_free (p); +netfs_folioq_free (p, netfs_trace_folioq_clear); } }
fs/netfs/read_collect.c +6 -2
@@ -101,6 +101,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was subreq->transferred, subreq->len)) subreq->transferred = subreq->len; +trace_netfs_folioq (folioq, netfs_trace_folioq_read_progress); next_folio: fsize =PAGE_SIZE << subreq->curr_folio_order; fpos =round_down (subreq->start + subreq->consumed, fsize);
@@ -117,9 +118,11 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool wasif (folioq) {struct folio *folio =folioq_folio (folioq, slot); -pr_err ("folioq: orders=%02x%02x%02x%02x\n" , +pr_err ("folioq: fq=%x orders=%02x%02x%02x%02x %px\n" , + folioq->debug_id, folioq->orders[0 ], folioq->orders[1 ], - folioq->orders[2 ], folioq->orders[3 ]); + folioq->orders[2 ], folioq->orders[3 ], + folioq);if (folio) pr_err("folio: %llx-%llx ix=%llx o=%u qo=%u\n", fpos, fend - 1, folio_pos(folio), folio_order(folio),
@@ -220,6 +223,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was slot =0 ; folioq = folioq->next; subreq->curr_folioq = folioq; +trace_netfs_folioq (folioq, netfs_trace_folioq_read_progress); } subreq->curr_folioq_slot = slot;if (folioq &&folioq_folio (folioq, slot))
fs/netfs/write_issue.c +1 -1
@@ -161,7 +161,7 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,*/ if (iov_iter_is_folioq (wreq_iter) && wreq_iter->folioq_slot >=folioq_nr_slots (wreq_iter->folioq)) { -netfs_buffer_make_space (wreq); +netfs_buffer_make_space (wreq, netfs_trace_folioq_prep_write); } subreq =netfs_alloc_subrequest (wreq);
fs/smb/client/smb2ops.c +1 -1
@@ -4415,7 +4415,7 @@ static struct folio_queue *cifs_alloc_folioq_buffer(ssize_t size) p =kmalloc (sizeof (*p),GFP_NOFS );if (!p) goto nomem; -folioq_init (p); +folioq_init (p,0 );if (tail) { tail->next = p; p->prev = tail;
include/linux/folio_queue.h +9 -3
@@ -37,16 +37,20 @@ struct folio_queue {#if PAGEVEC_SIZE >BITS_PER_LONG #error marks is not big enough#endif +unsigned int rreq_id; +unsigned int debug_id; };/** * folioq_init - Initialise a folio queue segment * @folioq: The segment to initialise +* @rreq_id: The request identifier to use in tracelines. * -* Initialise a folio queue segment. Note that the folio pointers are -* left uninitialised. +* Initialise a folio queue segment and set an identifier to be used in traces. +* +* Note that the folio pointers are left uninitialised. */ -static inline void folioq_init (struct folio_queue *folioq) +static inline void folioq_init (struct folio_queue *folioq,unsigned int rreq_id) {folio_batch_init (&folioq->vec); folioq->next =NULL ;
@@ -54,6 +58,8 @@ static inline void folioq_init(struct folio_queue *folioq) folioq->marks =0 ; folioq->marks2 =0 ; folioq->marks3 =0 ; + folioq->rreq_id = rreq_id; + folioq->debug_id =0 ; }/**
include/linux/netfs.h +4 -2
@@ -456,8 +456,10 @@ int netfs_start_io_direct(struct inode *inode);void netfs_end_io_direct (struct inode *inode);/* Miscellaneous APIs. */ -struct folio_queue *netfs_folioq_alloc (gfp_t gfp); -void netfs_folioq_free (struct folio_queue *folioq); +struct folio_queue *netfs_folioq_alloc (unsigned int rreq_id,gfp_t gfp, +unsigned int /*enum netfs_folioq_trace*/ trace); +void netfs_folioq_free (struct folio_queue *folioq, +unsigned int /*enum netfs_trace_folioq*/ trace);/** * netfs_inode - Get the netfs inode context from the inode
include/trace/events/netfs.h +39 -2
84 lines not on this page · raw message
lib/kunit_iov_iter.c +2 -2
@@ -392,7 +392,7 @@ static void __init iov_kunit_load_folioq(struct kunit *test,if (folioq_full (p)) { p->next =kzalloc (sizeof (struct folio_queue ),GFP_KERNEL );KUNIT_ASSERT_NOT_ERR_OR_NULL (test, p->next); -folioq_init (p->next); +folioq_init (p->next,0 ); p->next->prev = p; p = p->next; }
@@ -409,7 +409,7 @@ static struct folio_queue *iov_kunit_create_folioq(struct kunit *test) folioq =kzalloc (sizeof (struct folio_queue ),GFP_KERNEL );KUNIT_ASSERT_NOT_ERR_OR_NULL (test, folioq);kunit_add_action_or_reset (test, iov_kunit_destroy_folioq, folioq); -folioq_init (folioq); +folioq_init (folioq,0 );return folioq; }
David Howells
maintainerPATCH v3 7/33 netfs: Abstract out a rolling folio buffer implementation
In reply to David HowellsA rolling buffer is a series of folios held in a list of folio_queues. New folios and folio_queue structs may be inserted at the head simultaneously with spent ones being removed from the tail without the need for locking.
The rolling buffer includes an iov_iter and it has to be careful managing this as the list of folio_queues is extended such that an oops doesn't incurred because the iterator was pointing to the end of a folio_queue segment that got appended to and then removed.
We need to use the mechanism twice, once for read and once for write, and, in future patches, we will use a second rolling buffer to handle bounce buffering for content encryption.
--- fs/netfs/Makefile | 1 + fs/netfs/buffered_read.c | 119 ++++------------- fs/netfs/direct_read.c | 14 +- fs/netfs/direct_write.c | 10 +- fs/netfs/internal.h | 4 - fs/netfs/misc.c | 147 --------------------- fs/netfs/objects.c | 2 +- fs/netfs/read_pgpriv2.c | 32 ++--- fs/netfs/read_retry.c | 2 +- fs/netfs/rolling_buffer.c | 225 +++++++++++++++++++++++++++++++++ fs/netfs/write_collect.c | 19 +-- fs/netfs/write_issue.c | 26 ++-- include/linux/netfs.h | 10 +- include/linux/rolling_buffer.h | 61 +++++++++ include/trace/events/netfs.h | 2 + 15 files changed, 375 insertions(+), 299 deletions(-) create mode 100644 fs/netfs/rolling_buffer.c create mode 100644 include/linux/rolling_buffer.h
fs/netfs/Makefile +1 -0
@@ -13,6 +13,7 @@ netfs-y := \ read_collect.o \ read_pgpriv2.o \ read_retry.o \ + rolling_buffer.o \ write_collect.o \ write_issue.o
fs/netfs/buffered_read.c +23 -96
200 lines not on this page · raw message
fs/netfs/direct_read.c +7 -7
@@ -25,7 +25,7 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq) subreq->len = rsize;if (unlikely (rreq->io_streams[0 ].sreq_max_segs)) { -size_t limit =netfs_limit_iter (&rreq->iter,0 , rsize, +size_t limit =netfs_limit_iter (&rreq->buffer.iter,0 , rsize, rreq->io_streams[0 ].sreq_max_segs);if (limit < rsize) {
@@ -36,9 +36,9 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)trace_netfs_sreq (subreq, netfs_sreq_trace_prepare); - subreq->io_iter = rreq->iter; + subreq->io_iter = rreq->buffer.iter;iov_iter_truncate (&subreq->io_iter, subreq->len); -iov_iter_advance (&rreq->iter, subreq->len); +iov_iter_advance (&rreq->buffer.iter, subreq->len); }/*
@@ -199,15 +199,15 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i* the request. */ if (user_backed_iter (iter)) { - ret =netfs_extract_user_iter (iter, rreq->len, &rreq->iter,0 ); + ret =netfs_extract_user_iter (iter, rreq->len, &rreq->buffer.iter,0 );if (ret <0 ) goto out; - rreq->direct_bv = (struct bio_vec *)rreq->iter.bvec; + rreq->direct_bv = (struct bio_vec *)rreq->buffer.iter.bvec; rreq->direct_bv_count = ret; rreq->direct_bv_unpin =iov_iter_extract_will_pin (iter); - rreq->len =iov_iter_count (&rreq->iter); + rreq->len =iov_iter_count (&rreq->buffer.iter); }else { - rreq->iter = *iter; + rreq->buffer.iter = *iter; rreq->len = orig_count; rreq->direct_bv_unpin = false;iov_iter_advance (iter, orig_count);
fs/netfs/direct_write.c +5 -5
@@ -68,19 +68,19 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter ** request. */ if (async ||user_backed_iter (iter)) { - n =netfs_extract_user_iter (iter, len, &wreq->iter,0 ); + n =netfs_extract_user_iter (iter, len, &wreq->buffer.iter,0 );if (n <0 ) { ret = n; goto out; } - wreq->direct_bv = (struct bio_vec *)wreq->iter.bvec; + wreq->direct_bv = (struct bio_vec *)wreq->buffer.iter.bvec; wreq->direct_bv_count = n; wreq->direct_bv_unpin =iov_iter_extract_will_pin (iter); }else { - wreq->iter = *iter; + wreq->buffer.iter = *iter; } - wreq->io_iter = wreq->iter; + wreq->buffer.iter = wreq->buffer.iter; }__set_bit (NETFS_RREQ_USE_IO_ITER , &wreq->flags);
@@ -92,7 +92,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *__set_bit (NETFS_RREQ_UPLOAD_TO_SERVER , &wreq->flags);if (async) wreq->iocb = iocb; - wreq->len =iov_iter_count (&wreq->io_iter); + wreq->len =iov_iter_count (&wreq->buffer.iter); wreq->cleanup = netfs_cleanup_dio_write; ret =netfs_unbuffered_write (wreq,is_sync_kiocb (iocb), wreq->len);if (ret <0 ) {
fs/netfs/internal.h +0 -4
@@ -60,10 +60,6 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}*/ struct folio_queue *netfs_buffer_make_space (struct netfs_io_request *rreq,enum netfs_folioq_trace trace); -int netfs_buffer_append_folio (struct netfs_io_request *rreq,struct folio *folio, -bool needs_put); -struct folio_queue *netfs_delete_buffer_head (struct netfs_io_request *wreq); -void netfs_clear_buffer (struct netfs_io_request *rreq);void netfs_reset_iter (struct netfs_io_subrequest *subreq);/*
fs/netfs/misc.c +0 -147
153 lines not on this page · raw message
fs/netfs/objects.c +1 -1
@@ -143,7 +143,7 @@ static void netfs_free_request(struct work_struct *work) }kvfree (rreq->direct_bv); } -netfs_clear_buffer (rreq); +rolling_buffer_clear (&rreq->buffer);if (atomic_dec_and_test (&ictx->io_count))wake_up_var (&ictx->io_count);
fs/netfs/read_pgpriv2.c +17 -15
@@ -34,8 +34,9 @@ void netfs_pgpriv2_mark_copy_to_cache(struct netfs_io_subrequest *subreq,* [DEPRECATED] Cancel PG_private_2 on all marked folios in the event of an * unrecoverable error. */ -static void netfs_pgpriv2_cancel (struct folio_queue *folioq) +static void netfs_pgpriv2_cancel (struct rolling_buffer *buffer) { +struct folio_queue *folioq = buffer->tail;struct folio *folio;int slot;
@@ -94,7 +95,7 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct foliotrace_netfs_folio (folio, netfs_folio_trace_store_copy);/* Attach the folio to the rolling buffer. */ -if (netfs_buffer_append_folio (wreq, folio, false) <0 ) +if (rolling_buffer_append (&wreq->buffer, folio,0 ) <0 )return -ENOMEM ; cache->submit_extendable_to = fsize;
@@ -109,7 +110,7 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct folio do { ssize_t part; - wreq->io_iter.iov_offset = cache->submit_off; + wreq->buffer.iter.iov_offset = cache->submit_off; atomic64_set(&wreq->issued_to, fpos + cache->submit_off); cache->submit_extendable_to = fsize - cache->submit_off;
@@ -122,8 +123,8 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct folio cache->submit_len -= part; }while (cache->submit_len >0 ); - wreq->io_iter.iov_offset =0 ; -iov_iter_advance (&wreq->io_iter, fsize); + wreq->buffer.iter.iov_offset =0 ; +rolling_buffer_advance (&wreq->buffer, fsize);atomic64_set (&wreq->issued_to, fpos + fsize);if (flen < fsize)
@@ -151,7 +152,7 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq) goto couldnt_start;/* Need the first folio to be able to set up the op. */ -for (folioq = rreq->buffer; folioq; folioq = folioq->next) { +for (folioq = rreq->buffer.tail; folioq; folioq = folioq->next) {if (folioq->marks3) { slot =__ffs (folioq->marks3);break ;
@@ -194,7 +195,7 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)netfs_put_request (wreq, false, netfs_rreq_trace_put_return);_leave (" = %d" , error); couldnt_start: -netfs_pgpriv2_cancel (rreq->buffer); +netfs_pgpriv2_cancel (&rreq->buffer); }/*
@@ -203,13 +204,13 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)*/ bool netfs_pgpriv2_unlock_copied_folios (struct netfs_io_request *wreq) { -struct folio_queue *folioq = wreq->buffer; +struct folio_queue *folioq = wreq->buffer.tail;unsigned long long collected_to = wreq->collected_to; -unsigned int slot = wreq->buffer_head_slot; +unsigned int slot = wreq->buffer.first_tail_slot;bool made_progress = false;if (slot >=folioq_nr_slots (folioq)) { - folioq =netfs_delete_buffer_head (wreq); + folioq =rolling_buffer_delete_spent (&wreq->buffer); slot =0 ; }
@@ -248,9 +249,9 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)folioq_clear (folioq, slot); slot++;if (slot >=folioq_nr_slots (folioq)) { -if (READ_ONCE (wreq->buffer_tail) == folioq) -break ; - folioq =netfs_delete_buffer_head (wreq); + folioq =rolling_buffer_delete_spent (&wreq->buffer); +if (!folioq) + goto done; slot =0 ; }
@@ -258,7 +259,8 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)break ; } - wreq->buffer = folioq; - wreq->buffer_head_slot = slot; + wreq->buffer.tail = folioq; +done: + wreq->buffer.first_tail_slot = slot;return made_progress; }
fs/netfs/read_retry.c +1 -1
@@ -243,7 +243,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq) {struct folio_queue *p; -for (p = rreq->buffer; p; p = p->next) { +for (p = rreq->buffer.tail; p; p = p->next) {for (int slot =0 ; slot <folioq_count (p); slot++) {struct folio *folio =folioq_folio (p, slot);
fs/netfs/rolling_buffer.c new file +225 -0
225 lines not on this page · raw message
fs/netfs/write_collect.c +11 -8
@@ -83,9 +83,9 @@ int netfs_folio_written_back(struct folio *folio)static void netfs_writeback_unlock_folios (struct netfs_io_request *wreq,unsigned int *notes) { -struct folio_queue *folioq = wreq->buffer; +struct folio_queue *folioq = wreq->buffer.tail;unsigned long long collected_to = wreq->collected_to; -unsigned int slot = wreq->buffer_head_slot; +unsigned int slot = wreq->buffer.first_tail_slot;if (wreq->origin ==NETFS_PGPRIV2_COPY_TO_CACHE ) {if (netfs_pgpriv2_unlock_copied_folios (wreq))
@@ -94,7 +94,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq, }if (slot >=folioq_nr_slots (folioq)) { - folioq =netfs_delete_buffer_head (wreq); + folioq =rolling_buffer_delete_spent (&wreq->buffer); +if (!folioq) +return ; slot =0 ; }
@@ -134,9 +136,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,folioq_clear (folioq, slot); slot++;if (slot >=folioq_nr_slots (folioq)) { -if (READ_ONCE (wreq->buffer_tail) == folioq) -break ; - folioq =netfs_delete_buffer_head (wreq); + folioq =rolling_buffer_delete_spent (&wreq->buffer); +if (!folioq) + goto done; slot =0 ; }
@@ -144,8 +146,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,break ; } - wreq->buffer = folioq; - wreq->buffer_head_slot = slot; + wreq->buffer.tail = folioq; +done: + wreq->buffer.first_tail_slot = slot; }/*
fs/netfs/write_issue.c +17 -9
@@ -107,6 +107,8 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, ictx =netfs_inode (wreq->inode);if (is_buffered &&netfs_is_cache_enabled (ictx))fscache_begin_write_operation (&wreq->cache_resources,netfs_i_cookie (ictx)); +if (rolling_buffer_init (&wreq->buffer, wreq->debug_id,ITER_SOURCE ) <0 ) + goto nomem; wreq->cleaned_to = wreq->start;
@@ -129,6 +131,10 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, }return wreq; +nomem: + wreq->error = -ENOMEM ; +netfs_put_request (wreq, false, netfs_rreq_trace_put_failed); +return ERR_PTR (-ENOMEM ); }/**
@@ -153,16 +159,15 @@ static void netfs_prepare_write(struct netfs_io_request *wreq, loff_t start) {struct netfs_io_subrequest *subreq; -struct iov_iter *wreq_iter = &wreq->io_iter; +struct iov_iter *wreq_iter = &wreq->buffer.iter;/* Make sure we don't point the iterator at a used-up folio_queue * struct being used as a placeholder to prevent the queue from * collapsing. In such a case, extend the queue. */ if (iov_iter_is_folioq (wreq_iter) && - wreq_iter->folioq_slot >=folioq_nr_slots (wreq_iter->folioq)) { -netfs_buffer_make_space (wreq, netfs_trace_folioq_prep_write); - } + wreq_iter->folioq_slot >=folioq_nr_slots (wreq_iter->folioq)) +rolling_buffer_make_space (&wreq->buffer); subreq =netfs_alloc_subrequest (wreq); subreq->source = stream->source;
@@ -325,6 +330,9 @@ static int netfs_write_folio(struct netfs_io_request *wreq,_enter ("" ); +if (rolling_buffer_make_space (&wreq->buffer) <0 ) +return -ENOMEM ; +/* netfs_perform_write() may shift i_size around the page or from out * of the page to beyond it, but cannot move i_size into or through the * page since we have it locked.
@@ -429,7 +437,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq, }/* Attach the folio to the rolling buffer. */ -netfs_buffer_append_folio (wreq, folio, false); +rolling_buffer_append (&wreq->buffer, folio,0 );/* Move the submission point forward to allow for write-streaming data * not starting at the front of the page. We don't do write-streaming
@@ -476,7 +484,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,/* Advance the iterator(s). */ if (stream->submit_off > iter_off) { -iov_iter_advance (&wreq->io_iter, stream->submit_off - iter_off); +rolling_buffer_advance (&wreq->buffer, stream->submit_off - iter_off); iter_off = stream->submit_off; }
@@ -494,7 +502,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq, }if (fsize > iter_off) -iov_iter_advance (&wreq->io_iter, fsize - iter_off); +rolling_buffer_advance (&wreq->buffer, fsize - iter_off);atomic64_set (&wreq->issued_to, fpos + fsize);if (!debug)
@@ -633,7 +641,7 @@ int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_c struct folio **writethrough_cache) {_enter ("R=%x ic=%zu ws=%u cp=%zu tp=%u" , - wreq->debug_id, wreq->iter.count, wreq->wsize, copied, to_page_end); + wreq->debug_id, wreq->buffer.iter.count, wreq->wsize, copied, to_page_end);if (!*writethrough_cache) {if (folio_test_dirty (folio))
@@ -708,7 +716,7 @@ int netfs_unbuffered_write(struct netfs_io_request *wreq, bool may_wait, size_t part =netfs_advance_write (wreq, upload, start, len, false); start += part; len -= part; -iov_iter_advance (&wreq->io_iter, part); +rolling_buffer_advance (&wreq->buffer, part);if (test_bit (NETFS_RREQ_PAUSE , &wreq->flags)) {trace_netfs_rreq (wreq, netfs_rreq_trace_wait_pause);wait_on_bit (&wreq->flags,NETFS_RREQ_PAUSE ,TASK_UNINTERRUPTIBLE );
include/linux/netfs.h +4 -6
@@ -18,6 +18,7 @@#include <linux/fs.h> #include <linux/pagemap.h> #include <linux/uio.h> +#include <linux/rolling_buffer.h> enum netfs_sreq_ref_trace ;typedef struct mempool_s mempool_t ;
@@ -238,10 +239,9 @@ struct netfs_io_request {struct netfs_io_stream io_streams[2 ];/* Streams of parallel I/O operations */ #define NR_IO_STREAMS 2 //wreq->nr_io_streamsstruct netfs_group *group;/* Writeback group being written back */ -struct folio_queue *buffer;/* Head of I/O buffer */ -struct folio_queue *buffer_tail;/* Tail of I/O buffer */ -struct iov_iter iter;/* Unencrypted-side iterator */ -struct iov_iter io_iter;/* I/O (Encrypted-side) iterator */ +struct rolling_buffer buffer;/* Unencrypted buffer */ +#define NETFS_ROLLBUF_PUT_MARK ROLLBUF_MARK_1 +#define NETFS_ROLLBUF_PAGECACHE_MARK ROLLBUF_MARK_2void *netfs_priv;/* Private data for the netfs */ void *netfs_priv2;/* Private data for the netfs */ struct bio_vec *direct_bv;/* DIO buffer list (when handling iovec-iter) */
@@ -259,8 +259,6 @@ struct netfs_io_request {long error;/* 0 or error that occurred */ enum netfs_io_origin origin;/* Origin of the request */ bool direct_bv_unpin;/* T if direct_bv[] must be unpinned */ -u8 buffer_head_slot;/* First slot in ->buffer */ -u8 buffer_tail_slot;/* Next slot in ->buffer_tail */ unsigned long long i_size;/* Size of the file */ unsigned long long start;/* Start position */ atomic64_t issued_to;/* Write issuer folio cursor */
include/linux/rolling_buffer.h new file +61 -0
61 lines not on this page · raw message
include/trace/events/netfs.h +2 -0
@@ -198,7 +198,9 @@ EM(netfs_trace_folioq_alloc_read_sing, "alloc-r-sing") \ EM(netfs_trace_folioq_clear, "clear") \ EM(netfs_trace_folioq_delete, "delete") \ + EM(netfs_trace_folioq_make_space, "make-space") \ EM(netfs_trace_folioq_prep_write, "prep-wr") \ + EM(netfs_trace_folioq_rollbuf_init, "roll-init") \ E_(netfs_trace_folioq_read_progress, "r-progress")#ifndef __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
David Howells
maintainerPATCH v3 8/33 netfs: Make netfs_advance_write() return size_t
In reply to David Howellsnetfs_advance_write() calculates the amount of data it's attaching to a stream with size_t, but then returns this as an int. Switch the return value to size_t for consistency.
--- fs/netfs/internal.h | 6 +++--- fs/netfs/write_issue.c | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-)
fs/netfs/internal.h +3 -3
@@ -178,9 +178,9 @@ void netfs_reissue_write(struct netfs_io_stream *stream, struct iov_iter *source);void netfs_issue_write (struct netfs_io_request *wreq,struct netfs_io_stream *stream); -int netfs_advance_write (struct netfs_io_request *wreq, -struct netfs_io_stream *stream, -loff_t start,size_t len,bool to_eof); +size_t netfs_advance_write (struct netfs_io_request *wreq, +struct netfs_io_stream *stream, +loff_t start,size_t len,bool to_eof);struct netfs_io_request *netfs_begin_writethrough (struct kiocb *iocb,size_t len); int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc, struct folio *folio, size_t copied, bool to_page_end,
fs/netfs/write_issue.c +3 -3
@@ -271,9 +271,9 @@ void netfs_issue_write(struct netfs_io_request *wreq,* we can avoid overrunning the credits obtained (cifs) and try to parallelise * content-crypto preparation with network writes. */ -int netfs_advance_write (struct netfs_io_request *wreq, -struct netfs_io_stream *stream, -loff_t start,size_t len,bool to_eof) +size_t netfs_advance_write (struct netfs_io_request *wreq, +struct netfs_io_stream *stream, +loff_t start,size_t len,bool to_eof) {struct netfs_io_subrequest *subreq = stream->construct;size_t part;
David Howells
maintainerPATCH v3 9/33 netfs: Split retry code out of fs/netfs/write_collect.c
In reply to David HowellsSplit write-retry code out of fs/netfs/write_collect.c as it will become more elaborate when content crypto is introduced.
--- fs/netfs/Makefile | 3 +- fs/netfs/internal.h | 5 + fs/netfs/write_collect.c | 215 ------------------------------------ fs/netfs/write_retry.c | 227 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 234 insertions(+), 216 deletions(-) create mode 100644 fs/netfs/write_retry.c
fs/netfs/Makefile +2 -1
@@ -15,7 +15,8 @@ netfs-y := \ read_retry.o \ rolling_buffer.o \ write_collect.o \ - write_issue.o + write_issue.o \ + write_retry.o netfs-$(CONFIG_NETFS_STATS) += stats.o
fs/netfs/internal.h +5 -0
@@ -189,6 +189,11 @@ int netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_contr struct folio *writethrough_cache);int netfs_unbuffered_write (struct netfs_io_request *wreq,bool may_wait,size_t len); +/* +* write_retry.c +*/ +void netfs_retry_writes (struct netfs_io_request *wreq); +/* * Miscellaneous functions. */
fs/netfs/write_collect.c +0 -215
221 lines not on this page · raw message
fs/netfs/write_retry.c new file +227 -0
227 lines not on this page · raw message
David Howells
maintainerPATCH v3 10/33 netfs: Drop the error arg from netfs_read_subreq_terminated()
In reply to David HowellsDrop the error argument from netfs_read_subreq_terminated() in favour of passing the value in subreq->error.
--- fs/9p/vfs_addr.c | 3 ++- fs/afs/file.c | 15 ++++++++----- fs/ceph/addr.c | 13 +++++++---- fs/netfs/buffered_read.c | 16 +++++++------- fs/netfs/objects.c | 15 ++++++++++++- fs/netfs/read_collect.c | 47 +++++++++++++++++++++++++--------------- fs/nfs/fscache.c | 6 +++-- fs/nfs/fscache.h | 3 ++- fs/smb/client/cifssmb.c | 10 +-------- fs/smb/client/file.c | 3 ++- fs/smb/client/smb2pdu.c | 10 +-------- include/linux/netfs.h | 7 +++--- 12 files changed, 86 insertions(+), 62 deletions(-)
fs/9p/vfs_addr.c +2 -1
@@ -83,7 +83,8 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)if (!err) subreq->transferred += total; -netfs_read_subreq_terminated (subreq, err, false); + subreq->error = err; +netfs_read_subreq_terminated (subreq, false); }/**
fs/afs/file.c +10 -5
@@ -246,7 +246,8 @@ static void afs_fetch_data_notify(struct afs_operation *op) subreq->rreq->i_size = req->file_size;if (req->pos + req->actual_len >= req->file_size)__set_bit (NETFS_SREQ_HIT_EOF , &subreq->flags); -netfs_read_subreq_terminated (subreq, error, false); + subreq->error = error; +netfs_read_subreq_terminated (subreq, false); req->subreq =NULL ; }else if (req->done) { req->done (req);
@@ -301,8 +302,10 @@ int afs_fetch_data(struct afs_vnode *vnode, struct afs_read *req) op =afs_alloc_operation (req->key, vnode->volume);if (IS_ERR (op)) { -if (req->subreq) -netfs_read_subreq_terminated (req->subreq,PTR_ERR (op), false); +if (req->subreq) { + req->subreq->error =PTR_ERR (op); +netfs_read_subreq_terminated (req->subreq, false); + }return PTR_ERR (op); }
@@ -320,8 +323,10 @@ static void afs_read_worker(struct work_struct *work)struct afs_read *fsreq; fsreq =afs_alloc_read (GFP_NOFS ); -if (!fsreq) -return netfs_read_subreq_terminated (subreq, -ENOMEM , false); +if (!fsreq) { + subreq->error = -ENOMEM ; +return netfs_read_subreq_terminated (subreq, false); + } fsreq->subreq = subreq; fsreq->pos = subreq->start + subreq->transferred;
fs/ceph/addr.c +9 -4
@@ -253,8 +253,9 @@ static void finish_netfs_read(struct ceph_osd_request *req) subreq->transferred = err; err =0 ; } + subreq->error = err;trace_netfs_sreq (subreq, netfs_sreq_trace_io_progress); -netfs_read_subreq_terminated (subreq, err, false); +netfs_read_subreq_terminated (subreq, false);iput (req->r_inode);ceph_dec_osd_stopping_blocker (fsc->mdsc); }
@@ -314,7 +315,9 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)ceph_mdsc_put_request (req); out: -netfs_read_subreq_terminated (subreq, err, false); + subreq->error = err; +trace_netfs_sreq (subreq, netfs_sreq_trace_io_progress); +netfs_read_subreq_terminated (subreq, false);return true; }
@@ -426,8 +429,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)ceph_osdc_start_request (req->r_osdc, req); out:ceph_osdc_put_request (req); -if (err) -netfs_read_subreq_terminated (subreq, err, false); +if (err) { + subreq->error = err; +netfs_read_subreq_terminated (subreq, false); + }doutc (cl,"%llx.%llx result %d\n" ,ceph_vinop (inode), err); }
fs/netfs/buffered_read.c +8 -8
@@ -148,14 +148,13 @@ static void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error {struct netfs_io_subrequest *subreq = priv; -if (transferred_or_error <0 ) { -netfs_read_subreq_terminated (subreq, transferred_or_error, was_async); -return ; - } - -if (transferred_or_error >0 ) +if (transferred_or_error >0 ) { subreq->transferred += transferred_or_error; -netfs_read_subreq_terminated (subreq,0 , was_async); + subreq->error =0 ; + }else { + subreq->error = transferred_or_error; + } +netfs_read_subreq_terminated (subreq, was_async); }/*
@@ -261,7 +260,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)netfs_stat (&netfs_n_rh_zero); slice =netfs_prepare_read_iterator (subreq);__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); -netfs_read_subreq_terminated (subreq,0 , false); + subreq->error =0 ; +netfs_read_subreq_terminated (subreq, false); goto done; }
fs/netfs/objects.c +14 -1
@@ -191,7 +191,20 @@ struct netfs_io_subrequest *netfs_alloc_subrequest(struct netfs_io_request *rreq }memset (subreq,0 ,kmem_cache_size (cache)); -INIT_WORK (&subreq->work,NULL ); + +switch (rreq->origin) { +case NETFS_READAHEAD : +case NETFS_READPAGE : +case NETFS_READ_GAPS : +case NETFS_READ_FOR_WRITE : +case NETFS_DIO_READ : +INIT_WORK (&subreq->work, netfs_read_subreq_termination_worker); +break ; +default : +INIT_WORK (&subreq->work,NULL ); +break ; + } +INIT_LIST_HEAD (&subreq->rreq_link);refcount_set (&subreq->ref,2 ); subreq->rreq = rreq;
fs/netfs/read_collect.c +30 -17
91 lines not on this page · raw message
fs/nfs/fscache.c +4 -2
@@ -307,8 +307,10 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) &nfs_async_read_completion_ops); netfs =nfs_netfs_alloc (sreq); -if (!netfs) -return netfs_read_subreq_terminated (sreq, -ENOMEM , false); +if (!netfs) { + sreq->error = -ENOMEM ; +return netfs_read_subreq_terminated (sreq, false); + } pgio.pg_netfs = netfs;/* used in completion */
fs/nfs/fscache.h +2 -1
@@ -74,7 +74,8 @@ static inline void nfs_netfs_put(struct nfs_netfs_io_data *netfs)*/ netfs->sreq->transferred =min_t (s64, netfs->sreq->len,atomic64_read (&netfs->transferred)); -netfs_read_subreq_terminated (netfs->sreq, netfs->error, false); + netfs->sreq->error = netfs->error; +netfs_read_subreq_terminated (netfs->sreq, false);kfree (netfs); }static inline void nfs_netfs_inode_init (struct nfs_inode *nfsi)
fs/smb/client/cifssmb.c +1 -9
@@ -1261,14 +1261,6 @@ CIFS_open(const unsigned int xid, struct cifs_open_parms *oparms, int *oplock,return rc; } -static void cifs_readv_worker (struct work_struct *work) -{ -struct cifs_io_subrequest *rdata = - container_of(work, struct cifs_io_subrequest, subreq.work); - -netfs_read_subreq_terminated (&rdata->subreq, rdata->result, false); -} -static void cifs_readv_callback (struct mid_q_entry *mid) {
@@ -1334,8 +1326,8 @@ cifs_readv_callback(struct mid_q_entry *mid) } rdata->credits.value =0 ; + rdata->subreq.error = rdata->result; rdata->subreq.transferred += rdata->got_bytes; -INIT_WORK (&rdata->subreq.work, cifs_readv_worker);queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid);add_credits (server, &credits,0 );
fs/smb/client/file.c +2 -1
@@ -227,7 +227,8 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)return ; failed: -netfs_read_subreq_terminated (subreq, rc, false); + subreq->error = rc; +netfs_read_subreq_terminated (subreq, false); }/*
fs/smb/client/smb2pdu.c +1 -9
@@ -4513,14 +4513,6 @@ smb2_new_read_req(void **buf, unsigned int *total_len,return rc; } -static void smb2_readv_worker (struct work_struct *work) -{ -struct cifs_io_subrequest *rdata = - container_of(work, struct cifs_io_subrequest, subreq.work); - -netfs_read_subreq_terminated (&rdata->subreq, rdata->result, false); -} -static void smb2_readv_callback (struct mid_q_entry *mid) {
@@ -4633,9 +4625,9 @@ smb2_readv_callback(struct mid_q_entry *mid) server->credits, server->in_flight, 0, cifs_trace_rw_credits_read_response_clear); rdata->credits.value =0 ; + rdata->subreq.error = rdata->result; rdata->subreq.transferred += rdata->got_bytes;trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_progress); -INIT_WORK (&rdata->subreq.work, smb2_readv_worker);queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid); trace_smb3_rw_credits(rreq_debug_id, subreq_debug_index, 0,
include/linux/netfs.h +3 -4
@@ -428,10 +428,9 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp);vm_fault_t netfs_page_mkwrite (struct vm_fault *vmf,struct netfs_group *netfs_group);/* (Sub)request management API. */ -void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq, -bool was_async); -void netfs_read_subreq_terminated (struct netfs_io_subrequest *subreq, -int error,bool was_async); +void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq,bool was_async); +void netfs_read_subreq_terminated (struct netfs_io_subrequest *subreq,bool was_async); +void netfs_read_subreq_termination_worker (struct work_struct *work);void netfs_get_subrequest (struct netfs_io_subrequest *subreq,enum netfs_sreq_ref_trace what); void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
David Howells
maintainerPATCH v3 11/33 netfs: Drop the was_async arg from netfs_read_subreq_terminated()
In reply to David HowellsDrop the was_async argument from netfs_read_subreq_terminated(). Almost every caller is either in process context and passes false. Some filesystems delegate the call to a workqueue to avoid doing the work in their network message queue parsing thread.
The only exception is netfs_cache_read_terminated() which handles completion in the cache - which is usually a callback from the backing filesystem in softirq context, though it can be from process context if an error occurred. In this case, delegate to a workqueue.
--- fs/9p/vfs_addr.c | 2 +- fs/afs/file.c | 6 ++--- fs/afs/fsclient.c | 2 +- fs/afs/yfsclient.c | 2 +- fs/ceph/addr.c | 6 ++--- fs/netfs/buffered_read.c | 6 ++--- fs/netfs/direct_read.c | 2 +- fs/netfs/internal.h | 2 +- fs/netfs/objects.c | 2 +- fs/netfs/read_collect.c | 53 +++++++++------------------------------- fs/netfs/read_retry.c | 2 +- fs/nfs/fscache.c | 2 +- fs/nfs/fscache.h | 2 +- fs/smb/client/file.c | 2 +- include/linux/netfs.h | 4 +-- 15 files changed, 33 insertions(+), 62 deletions(-)
fs/9p/vfs_addr.c +1 -1
@@ -84,7 +84,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq) subreq->transferred += total; subreq->error = err; -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq); }/**
fs/afs/file.c +3 -3
@@ -247,7 +247,7 @@ static void afs_fetch_data_notify(struct afs_operation *op)if (req->pos + req->actual_len >= req->file_size)__set_bit (NETFS_SREQ_HIT_EOF , &subreq->flags); subreq->error = error; -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq); req->subreq =NULL ; }else if (req->done) { req->done (req);
@@ -304,7 +304,7 @@ int afs_fetch_data(struct afs_vnode *vnode, struct afs_read *req)if (IS_ERR (op)) {if (req->subreq) { req->subreq->error =PTR_ERR (op); -netfs_read_subreq_terminated (req->subreq, false); +netfs_read_subreq_terminated (req->subreq); }return PTR_ERR (op); }
@@ -325,7 +325,7 @@ static void afs_read_worker(struct work_struct *work) fsreq =afs_alloc_read (GFP_NOFS );if (!fsreq) { subreq->error = -ENOMEM ; -return netfs_read_subreq_terminated (subreq, false); +return netfs_read_subreq_terminated (subreq); } fsreq->subreq = subreq;
fs/afs/fsclient.c +1 -1
@@ -352,7 +352,7 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call) ret =afs_extract_data (call, true);if (req->subreq) { req->subreq->transferred += count_before - call->iov_len; -netfs_read_subreq_progress (req->subreq, false); +netfs_read_subreq_progress (req->subreq); }if (ret <0 )return ret;
fs/afs/yfsclient.c +1 -1
@@ -398,7 +398,7 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call) ret =afs_extract_data (call, true);if (req->subreq) { req->subreq->transferred += count_before - call->iov_len; -netfs_read_subreq_progress (req->subreq, false); +netfs_read_subreq_progress (req->subreq); }if (ret <0 )return ret;
fs/ceph/addr.c +3 -3
@@ -255,7 +255,7 @@ static void finish_netfs_read(struct ceph_osd_request *req) } subreq->error = err;trace_netfs_sreq (subreq, netfs_sreq_trace_io_progress); -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq);iput (req->r_inode);ceph_dec_osd_stopping_blocker (fsc->mdsc); }
@@ -317,7 +317,7 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) out: subreq->error = err;trace_netfs_sreq (subreq, netfs_sreq_trace_io_progress); -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq);return true; }
@@ -431,7 +431,7 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)ceph_osdc_put_request (req);if (err) { subreq->error = err; -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq); }doutc (cl,"%llx.%llx result %d\n" ,ceph_vinop (inode), err); }
fs/netfs/buffered_read.c +3 -3
@@ -154,7 +154,7 @@ static void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error }else { subreq->error = transferred_or_error; } -netfs_read_subreq_terminated (subreq, was_async); +schedule_work (&subreq->work); }/*
@@ -261,7 +261,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) slice =netfs_prepare_read_iterator (subreq);__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); subreq->error =0 ; -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq); goto done; }
@@ -283,7 +283,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) }while (size >0 );if (atomic_dec_and_test (&rreq->nr_outstanding)) -netfs_rreq_terminated (rreq, false); +netfs_rreq_terminated (rreq);/* Defer error return as we may need to wait for outstanding I/O. */ cmpxchg (&rreq->error,0 , ret);
fs/netfs/direct_read.c +1 -1
@@ -100,7 +100,7 @@ static int netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq) }while (size >0 );if (atomic_dec_and_test (&rreq->nr_outstanding)) -netfs_rreq_terminated (rreq, false); +netfs_rreq_terminated (rreq);return ret; }
fs/netfs/internal.h +1 -1
@@ -85,7 +85,7 @@ static inline void netfs_see_request(struct netfs_io_request *rreq,* read_collect.c */ void netfs_read_termination_worker (struct work_struct *work); -void netfs_rreq_terminated (struct netfs_io_request *rreq,bool was_async); +void netfs_rreq_terminated (struct netfs_io_request *rreq);/* * read_pgpriv2.c
fs/netfs/objects.c +1 -1
@@ -56,7 +56,7 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping, origin == NETFS_READ_GAPS || origin == NETFS_READ_FOR_WRITE || origin == NETFS_DIO_READ) -INIT_WORK (&rreq->work, netfs_read_termination_worker); +INIT_WORK (&rreq->work,NULL ); else INIT_WORK(&rreq->work, netfs_write_collection_worker);
fs/netfs/read_collect.c +12 -41
126 lines not on this page · raw message
fs/netfs/read_retry.c +1 -1
@@ -232,7 +232,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)netfs_retry_read_subrequests (rreq);if (atomic_dec_and_test (&rreq->nr_outstanding)) -netfs_rreq_terminated (rreq, false); +netfs_rreq_terminated (rreq); }/*
fs/nfs/fscache.c +1 -1
@@ -309,7 +309,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) netfs =nfs_netfs_alloc (sreq);if (!netfs) { sreq->error = -ENOMEM ; -return netfs_read_subreq_terminated (sreq, false); +return netfs_read_subreq_terminated (sreq); } pgio.pg_netfs = netfs;/* used in completion */
fs/nfs/fscache.h +1 -1
@@ -75,7 +75,7 @@ static inline void nfs_netfs_put(struct nfs_netfs_io_data *netfs) netfs->sreq->transferred =min_t (s64, netfs->sreq->len,atomic64_read (&netfs->transferred)); netfs->sreq->error = netfs->error; -netfs_read_subreq_terminated (netfs->sreq, false); +netfs_read_subreq_terminated (netfs->sreq);kfree (netfs); }static inline void nfs_netfs_inode_init (struct nfs_inode *nfsi)
fs/smb/client/file.c +1 -1
@@ -228,7 +228,7 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq) failed: subreq->error = rc; -netfs_read_subreq_terminated (subreq, false); +netfs_read_subreq_terminated (subreq); }/*
include/linux/netfs.h +2 -2
@@ -428,8 +428,8 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp);vm_fault_t netfs_page_mkwrite (struct vm_fault *vmf,struct netfs_group *netfs_group);/* (Sub)request management API. */ -void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq,bool was_async); -void netfs_read_subreq_terminated (struct netfs_io_subrequest *subreq,bool was_async); +void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq); +void netfs_read_subreq_terminated (struct netfs_io_subrequest *subreq);void netfs_read_subreq_termination_worker (struct work_struct *work);void netfs_get_subrequest (struct netfs_io_subrequest *subreq,enum netfs_sreq_ref_trace what);
David Howells
maintainerPATCH v3 12/33 netfs: Don't use bh spinlock
In reply to David HowellsAll the accessing of the subrequest lists is now done in process context, possibly in a workqueue, but not now in a BH context, so we don't need the lock against BH interference when taking the netfs_io_request::lock spinlock.
--- fs/netfs/buffered_read.c | 4 ++-- fs/netfs/direct_read.c | 4 ++-- fs/netfs/read_collect.c | 20 ++++++++++---------- fs/netfs/read_retry.c | 8 ++++---- fs/netfs/write_collect.c | 4 ++-- fs/netfs/write_issue.c | 4 ++-- 6 files changed, 22 insertions(+), 22 deletions(-)
fs/netfs/buffered_read.c +2 -2
@@ -200,12 +200,12 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) subreq->len = size;atomic_inc (&rreq->nr_outstanding); -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);list_add_tail (&subreq->rreq_link, &rreq->subrequests); subreq->prev_donated = rreq->prev_donated; rreq->prev_donated =0 ;trace_netfs_sreq (subreq, netfs_sreq_trace_added); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock); source =netfs_cache_prepare_read (rreq, subreq, rreq->i_size); subreq->source = source;
fs/netfs/direct_read.c +2 -2
@@ -68,12 +68,12 @@ static int netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq) subreq->len = size;atomic_inc (&rreq->nr_outstanding); -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);list_add_tail (&subreq->rreq_link, &rreq->subrequests); subreq->prev_donated = rreq->prev_donated; rreq->prev_donated =0 ;trace_netfs_sreq (subreq, netfs_sreq_trace_added); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock);netfs_stat (&netfs_n_rh_download);if (rreq->netfs_ops->prepare_read) {
fs/netfs/read_collect.c +10 -10
@@ -142,7 +142,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq) prev_donated =READ_ONCE (subreq->prev_donated); next_donated =READ_ONCE (subreq->next_donated);if (prev_donated || next_donated) { -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock); prev_donated = subreq->prev_donated; next_donated = subreq->next_donated; subreq->start -= prev_donated;
@@ -155,7 +155,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq) next_donated = subreq->next_donated =0 ; }trace_netfs_sreq (subreq, netfs_sreq_trace_add_donations); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock); } avail = subreq->transferred;
@@ -184,18 +184,18 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq) }else if (fpos < start) { excess = fend - subreq->start; -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);/* If we complete first on a folio split with the * preceding subreq, donate to that subreq - otherwise * we get the responsibility. */ if (subreq->prev_donated != prev_donated) { -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock); goto donation_changed; }if (list_is_first (&subreq->rreq_link, &rreq->subrequests)) { -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock);pr_err ("Can't donate prior to front\n" ); goto bad; }
@@ -211,7 +211,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq)if (subreq->consumed >= subreq->len) goto remove_subreq_locked; -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock); }else {pr_err ("fpos > start\n" ); goto bad;
@@ -239,11 +239,11 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq)/* Donate the remaining downloaded data to one of the neighbouring * subrequests. Note that we may race with them doing the same thing. */ -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);if (subreq->prev_donated != prev_donated || subreq->next_donated != next_donated) { -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock);cond_resched (); goto donation_changed; }
@@ -293,11 +293,11 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq) goto remove_subreq_locked; remove_subreq: -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock); remove_subreq_locked: subreq->consumed = subreq->len;list_del (&subreq->rreq_link); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock);netfs_put_subrequest (subreq, false, netfs_sreq_trace_put_consumed);return true;
fs/netfs/read_retry.c +4 -4
@@ -139,12 +139,12 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags);__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);list_add_tail (&subreq->rreq_link, &rreq->subrequests); subreq->prev_donated += rreq->prev_donated; rreq->prev_donated =0 ;trace_netfs_sreq (subreq, netfs_sreq_trace_retry); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock);BUG_ON (!len);
@@ -215,9 +215,9 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags);__clear_bit (NETFS_SREQ_RETRYING , &subreq->flags); } -spin_lock_bh (&rreq->lock); +spin_lock (&rreq->lock);list_splice_tail_init (&queue, &rreq->subrequests); -spin_unlock_bh (&rreq->lock); +spin_unlock (&rreq->lock); }/*
fs/netfs/write_collect.c +2 -2
@@ -238,14 +238,14 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) cancel:/* Remove if completely consumed. */ -spin_lock_bh (&wreq->lock); +spin_lock (&wreq->lock); remove = front;list_del_init (&front->rreq_link); front =list_first_entry_or_null (&stream->subrequests, struct netfs_io_subrequest, rreq_link); stream->front = front; -spin_unlock_bh (&wreq->lock); +spin_unlock (&wreq->lock); netfs_put_subrequest(remove, false, notes & SAW_FAILURE ? netfs_sreq_trace_put_cancel :
fs/netfs/write_issue.c +2 -2
@@ -203,7 +203,7 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,* the list. The collector only goes nextwards and uses the lock to * remove entries off of the front. */ -spin_lock_bh (&wreq->lock); +spin_lock (&wreq->lock);list_add_tail (&subreq->rreq_link, &stream->subrequests);if (list_is_first (&subreq->rreq_link, &stream->subrequests)) { stream->front = subreq;
@@ -214,7 +214,7 @@ static void netfs_prepare_write(struct netfs_io_request *wreq, } } -spin_unlock_bh (&wreq->lock); +spin_unlock (&wreq->lock); stream->construct = subreq; }
David Howells
maintainerPATCH v3 13/33 afs: Don't use mutex for I/O operation lock
In reply to David HowellsDon't use the standard mutex for the I/O operation lock, but rather implement our own as the standard mutex must be released in the same thread as locked it. This is a problem when it comes to doing async FetchData where the lock will be dropped from the workqueue that processed the incoming data and not from the issuing thread.
--- fs/afs/fs_operation.c | 111 +++++++++++++++++++++++++++++++++++++++--- fs/afs/internal.h | 3 +- fs/afs/super.c | 2 +- 3 files changed, 108 insertions(+), 8 deletions(-)
fs/afs/fs_operation.c +105 -6
@@ -49,6 +49,105 @@ struct afs_operation *afs_alloc_operation(struct key *key, struct afs_volume *voreturn op; } +struct afs_io_locker { +struct list_head link; +struct task_struct *task; +unsigned long have_lock; +}; + +/* +* Unlock the I/O lock on a vnode. +*/ +static void afs_unlock_for_io (struct afs_vnode *vnode) +{ +struct afs_io_locker *locker; + +spin_lock (&vnode->lock); + locker =list_first_entry_or_null (&vnode->io_lock_waiters, + struct afs_io_locker, link); +if (locker) { +list_del (&locker->link); +smp_store_release (&locker->have_lock,1 ); +smp_mb__after_atomic ();/* Store have_lock before task state */ +wake_up_process (locker->task); + }else { +clear_bit (AFS_VNODE_IO_LOCK , &vnode->flags); + } +spin_unlock (&vnode->lock); +} + +/* +* Lock the I/O lock on a vnode uninterruptibly. We can't use an ordinary +* mutex as lockdep will complain if we unlock it in the wrong thread. +*/ +static void afs_lock_for_io (struct afs_vnode *vnode) +{ +struct afs_io_locker myself = { .task = current, }; + +spin_lock (&vnode->lock); + +if (!test_and_set_bit (AFS_VNODE_IO_LOCK , &vnode->flags)) { +spin_unlock (&vnode->lock); +return ; + } + +list_add_tail (&myself.link, &vnode->io_lock_waiters); +spin_unlock (&vnode->lock); + +for (;;) { +set_current_state (TASK_UNINTERRUPTIBLE ); +if (smp_load_acquire (&myself.have_lock)) +break ; +schedule (); + } +__set_current_state (TASK_RUNNING ); +} + +/* +* Lock the I/O lock on a vnode interruptibly. We can't use an ordinary mutex +* as lockdep will complain if we unlock it in the wrong thread. +*/ +static int afs_lock_for_io_interruptible (struct afs_vnode *vnode) +{ +struct afs_io_locker myself = { .task = current, }; +int ret =0 ; + +spin_lock (&vnode->lock); + +if (!test_and_set_bit (AFS_VNODE_IO_LOCK , &vnode->flags)) { +spin_unlock (&vnode->lock); +return 0 ; + } + +list_add_tail (&myself.link, &vnode->io_lock_waiters); +spin_unlock (&vnode->lock); + +for (;;) { +set_current_state (TASK_INTERRUPTIBLE ); +if (smp_load_acquire (&myself.have_lock) || +signal_pending (current)) +break ; +schedule (); + } +__set_current_state (TASK_RUNNING ); + +/* If we got a signal, try to transfer the lock onto the next +* waiter. +*/ +if (unlikely (signal_pending (current))) { +spin_lock (&vnode->lock); +if (myself.have_lock) { +spin_unlock (&vnode->lock); +afs_unlock_for_io (vnode); + }else { +list_del (&myself.link); +spin_unlock (&vnode->lock); + } + ret = -ERESTARTSYS ; + } +return ret; +} +/* * Lock the vnode(s) being operated upon. */
@@ -60,7 +159,7 @@ static bool afs_get_io_locks(struct afs_operation *op)_enter ("" );if (op->flags &AFS_OPERATION_UNINTR ) { -mutex_lock (&vnode->io_lock); +afs_lock_for_io (vnode); op->flags |=AFS_OPERATION_LOCK_0 ;_leave (" = t [1]" );return true;
@@ -72,7 +171,7 @@ static bool afs_get_io_locks(struct afs_operation *op)if (vnode2 > vnode)swap (vnode, vnode2); -if (mutex_lock_interruptible (&vnode->io_lock) <0 ) { +if (afs_lock_for_io_interruptible (vnode) <0 ) {afs_op_set_error (op, -ERESTARTSYS ); op->flags |=AFS_OPERATION_STOP ;_leave (" = f [I 0]" );
@@ -81,10 +180,10 @@ static bool afs_get_io_locks(struct afs_operation *op) op->flags |=AFS_OPERATION_LOCK_0 ;if (vnode2) { -if (mutex_lock_interruptible_nested (&vnode2->io_lock,1 ) <0 ) { +if (afs_lock_for_io_interruptible (vnode2) <0 ) {afs_op_set_error (op, -ERESTARTSYS ); op->flags |=AFS_OPERATION_STOP ; -mutex_unlock (&vnode->io_lock); +afs_unlock_for_io (vnode); op->flags &= ~AFS_OPERATION_LOCK_0 ;_leave (" = f [I 1]" );return false;
@@ -104,9 +203,9 @@ static void afs_drop_io_locks(struct afs_operation *op)_enter ("" );if (op->flags &AFS_OPERATION_LOCK_1 ) -mutex_unlock (&vnode2->io_lock); +afs_unlock_for_io (vnode2);if (op->flags &AFS_OPERATION_LOCK_0 ) -mutex_unlock (&vnode->io_lock); +afs_unlock_for_io (vnode); } static void afs_prepare_vnode(struct afs_operation *op, struct afs_vnode_param *vp,
fs/afs/internal.h +2 -1
@@ -702,13 +702,14 @@ struct afs_vnode {struct afs_file_status status;/* AFS status info for this file */ afs_dataversion_t invalid_before;/* Child dentries are invalid before this */ struct afs_permits __rcu *permit_cache;/* cache of permits so far obtained */ -struct mutex io_lock;/* Lock for serialising I/O on this mutex */ +struct list_head io_lock_waiters;/* Threads waiting for the I/O lock */ struct rw_semaphore validate_lock;/* lock for validating this vnode */ struct rw_semaphore rmdir_lock;/* Lock for rmdir vs sillyrename */ struct key *silly_key;/* Silly rename key */ spinlock_t wb_lock;/* lock for wb_keys */ spinlock_t lock;/* waitqueue/flags lock */ unsigned long flags; +#define AFS_VNODE_IO_LOCK 0/* Set if the I/O serialisation lock is held */ #define AFS_VNODE_UNSET 1/* set if vnode attributes not yet set */ #define AFS_VNODE_DIR_VALID 2/* Set if dir contents are valid */ #define AFS_VNODE_ZAP_DATA 3/* set if vnode's data should be invalidated */
fs/afs/super.c +1 -1
@@ -663,7 +663,7 @@ static void afs_i_init_once(void *_vnode)memset (vnode,0 ,sizeof (*vnode));inode_init_once (&vnode->netfs.inode); -mutex_init (&vnode->io_lock); +INIT_LIST_HEAD (&vnode->io_lock_waiters);init_rwsem (&vnode->validate_lock);spin_lock_init (&vnode->wb_lock);spin_lock_init (&vnode->lock);
David Howells
maintainerPATCH v3 14/33 afs: Fix EEXIST error returned from afs_rmdir() to be ENOTEMPTY
In reply to David HowellsAFS servers pass back a code indicating EEXIST when they're asked to remove a directory that is not empty rather than ENOTEMPTY because not all the systems that an AFS server can run on have the latter error available and AFS preexisted the addition of that error in general.
Fix afs_rmdir() to translate EEXIST to ENOTEMPTY.
--- fs/afs/dir.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
fs/afs/dir.c +6 -1
@@ -1472,7 +1472,12 @@ static int afs_rmdir(struct inode *dir, struct dentry *dentry) op->file[1 ].vnode = vnode; } -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); + +/* Not all systems that can host afs servers have ENOTEMPTY. */ +if (ret == -EEXIST ) + ret = -ENOTEMPTY ; +return ret; error:return afs_put_operation (op);
David Howells
maintainerPATCH v3 15/33 afs: Fix directory format encoding struct
In reply to David HowellsThe AFS directory format structure, union afs_xdr_dir_block::meta, has too many alloc counter slots declared and so pushes the hash table along and over the data. This doesn't cause a problem at the moment because I'm currently ignoring the hash table and only using the correct number of alloc_ctrs in the code anyway. In future, however, I should start using the hash table to try and speed up afs_lookup().
Fix this by using the correct constant to declare the counter array.
--- fs/afs/xdr_fs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/afs/xdr_fs.h +1 -1
@@ -88,7 +88,7 @@ union afs_xdr_dir_block {struct {struct afs_xdr_dir_hdr hdr; -u8 alloc_ctrs[AFS_DIR_MAX_BLOCKS ]; +u8 alloc_ctrs[AFS_DIR_BLOCKS_WITH_CTR ];__be16 hashtable[AFS_DIR_HASHTBL_SIZE ]; } meta;
David Howells
maintainerPATCH v3 16/33 netfs: Remove some extraneous directory invalidations
In reply to David HowellsIn the directory editing code, we shouldn't re-invalidate the directory if it is already invalidated.
--- fs/afs/dir_edit.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)
fs/afs/dir_edit.c +9 -13
@@ -247,7 +247,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,*/ index = b /AFS_DIR_BLOCKS_PER_PAGE ;if (nr_blocks >=AFS_DIR_MAX_BLOCKS ) - goto error; + goto error_too_many_blocks;if (index >=folio_nr_pages (folio0)) { folio =afs_dir_get_folio (vnode, index);if (!folio)
@@ -260,7 +260,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,/* Abandon the edit if we got a callback break. */ if (!test_bit (AFS_VNODE_DIR_VALID , &vnode->flags)) - goto invalidated; + goto already_invalidated; _debug("block %u: %2u %3u %u", b,
@@ -348,9 +348,8 @@ void afs_edit_dir_add(struct afs_vnode *vnode,_leave ("" );return ; -invalidated: +already_invalidated:trace_afs_edit_dir (vnode, why, afs_edit_dir_create_inval,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags);kunmap_local (block);if (folio != folio0) {folio_unlock (folio);
@@ -358,9 +357,10 @@ void afs_edit_dir_add(struct afs_vnode *vnode, } goto out_unmap; +error_too_many_blocks: +clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); error:trace_afs_edit_dir (vnode, why, afs_edit_dir_create_error,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); goto out_unmap; }
@@ -421,7 +421,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,/* Abandon the edit if we got a callback break. */ if (!test_bit (AFS_VNODE_DIR_VALID , &vnode->flags)) - goto invalidated; + goto already_invalidated;if (b >AFS_DIR_BLOCKS_WITH_CTR || meta->meta.alloc_ctrs[b] <=AFS_DIR_SLOTS_PER_BLOCK -1 - need_slots) {
@@ -475,10 +475,9 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,_leave ("" );return ; -invalidated: +already_invalidated:trace_afs_edit_dir (vnode, why, afs_edit_dir_delete_inval,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags);kunmap_local (block);if (folio != folio0) {folio_unlock (folio);
@@ -489,7 +488,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode, error:trace_afs_edit_dir (vnode, why, afs_edit_dir_delete_error,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); goto out_unmap; }
@@ -530,7 +528,7 @@ void afs_edit_dir_update_dotdot(struct afs_vnode *vnode, struct afs_vnode *new_d/* Abandon the edit if we got a callback break. */ if (!test_bit (AFS_VNODE_DIR_VALID , &vnode->flags)) - goto invalidated; + goto already_invalidated; slot =afs_dir_scan_block (block, &dotdot_name, b);if (slot >=0 )
@@ -564,18 +562,16 @@ void afs_edit_dir_update_dotdot(struct afs_vnode *vnode, struct afs_vnode *new_d_leave ("" );return ; -invalidated: +already_invalidated:kunmap_local (block);folio_unlock (folio);folio_put (folio);trace_afs_edit_dir (vnode, why, afs_edit_dir_update_inval,0 ,0 ,0 ,0 ,".." ); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); goto out; error:trace_afs_edit_dir (vnode, why, afs_edit_dir_update_error,0 ,0 ,0 ,0 ,".." ); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); goto out; }
David Howells
maintainerPATCH v3 17/33 cachefiles: Add some subrequest tracepoints
In reply to David HowellsAdd some tracepoints into the cachefiles write paths.
--- fs/cachefiles/io.c | 4 ++++ include/trace/events/netfs.h | 3 +++ 2 files changed, 7 insertions(+)
fs/cachefiles/io.c +4 -0
@@ -13,6 +13,7 @@#include <linux/falloc.h> #include <linux/sched/mm.h> #include <trace/events/fscache.h> +#include <trace/events/netfs.h> #include "internal.h" struct cachefiles_kiocb {
@@ -366,6 +367,7 @@ static int cachefiles_write(struct netfs_cache_resources *cres,if (!fscache_wait_for_operation (cres,FSCACHE_WANT_WRITE )) {if (term_func)term_func (term_func_priv, -ENOBUFS , false); +trace_netfs_sreq (term_func_priv, netfs_sreq_trace_cache_nowrite);return -ENOBUFS ; }
@@ -695,6 +697,7 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)iov_iter_truncate (&subreq->io_iter, len); } +trace_netfs_sreq (subreq, netfs_sreq_trace_cache_prepare);cachefiles_begin_secure (cache, &saved_cred); ret =__cachefiles_prepare_write (object,cachefiles_cres_file (cres), &start, &len, len, true);
@@ -704,6 +707,7 @@ static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)return ; } +trace_netfs_sreq (subreq, netfs_sreq_trace_cache_write);cachefiles_write (&subreq->rreq->cache_resources, subreq->start, &subreq->io_iter, netfs_write_subrequest_terminated, subreq);
include/trace/events/netfs.h +3 -0
@@ -74,6 +74,9 @@#define netfs_sreq_traces \ EM(netfs_sreq_trace_add_donations, "+DON ") \ EM(netfs_sreq_trace_added, "ADD ") \ + EM(netfs_sreq_trace_cache_nowrite, "CA-NW") \ + EM(netfs_sreq_trace_cache_prepare, "CA-PR") \ + EM(netfs_sreq_trace_cache_write, "CA-WR") \ EM(netfs_sreq_trace_clear, "CLEAR") \ EM(netfs_sreq_trace_discard, "DSCRD") \ EM(netfs_sreq_trace_donate_to_prev, "DON-P") \
David Howells
maintainerPATCH v3 18/33 cachefiles: Add auxiliary data trace
In reply to David HowellsAdd a display of the first 8 bytes of the downloaded auxiliary data and of the on-disk stored auxiliary data as these are used in coherency management. In the case of afs, this holds the data version number.
--- fs/cachefiles/xattr.c | 9 ++++++++- include/trace/events/cachefiles.h | 13 ++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-)
fs/cachefiles/xattr.c +8 -1
@@ -77,6 +77,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)trace_cachefiles_vfs_error (object,file_inode (file), ret, cachefiles_trace_setxattr_error);trace_cachefiles_coherency (object,file_inode (file)->i_ino, +be64_to_cpup ((__be64 *)buf->data), buf->content, cachefiles_coherency_set_fail);if (ret != -ENOMEM )
@@ -85,6 +86,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object) "Failed to set xattr with error %d", ret); }else {trace_cachefiles_coherency (object,file_inode (file)->i_ino, +be64_to_cpup ((__be64 *)buf->data), buf->content, cachefiles_coherency_set_ok); }
@@ -126,7 +128,10 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file object, "Failed to read aux with error %zd", xlen); why = cachefiles_coherency_check_xattr; - }else if (buf->type !=CACHEFILES_COOKIE_TYPE_DATA ) { + goto out; + } + +if (buf->type !=CACHEFILES_COOKIE_TYPE_DATA ) { why = cachefiles_coherency_check_type; }else if (memcmp (buf->data, p, len) !=0 ) { why = cachefiles_coherency_check_aux;
@@ -141,7 +146,9 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file ret =0 ; } +out:trace_cachefiles_coherency (object,file_inode (file)->i_ino, +be64_to_cpup ((__be64 *)buf->data), buf->content, why);kfree (buf);return ret;
include/trace/events/cachefiles.h +10 -3
@@ -380,10 +380,11 @@ TRACE_EVENT(cachefiles_rename, TRACE_EVENT(cachefiles_coherency, TP_PROTO(struct cachefiles_object *obj, ino_t ino, + u64 disk_aux, enum cachefiles_content content, enum cachefiles_coherency_trace why), - TP_ARGS(obj, ino, content, why), + TP_ARGS(obj, ino, disk_aux, content, why), /* Note that obj may be NULL */ TP_STRUCT__entry(
@@ -391,6 +392,8 @@ TRACE_EVENT(cachefiles_coherency, __field(enum cachefiles_coherency_trace, why ) __field(enum cachefiles_content, content ) __field(u64, ino ) + __field(u64, aux ) + __field(u64, disk_aux) ), TP_fast_assign(
@@ -398,13 +401,17 @@ TRACE_EVENT(cachefiles_coherency, __entry->why = why; __entry->content = content; __entry->ino = ino; + __entry->aux =be64_to_cpup ((__be64 *)obj->cookie->inline_aux); + __entry->disk_aux = disk_aux; ), -TP_printk ("o=%08x %s B=%llx c=%u" , +TP_printk ("o=%08x %s B=%llx c=%u aux=%llx dsk=%llx" , __entry->obj,__print_symbolic (__entry->why, cachefiles_coherency_traces), __entry->ino, - __entry->content) + __entry->content, + __entry->aux, + __entry->disk_aux) ); TRACE_EVENT(cachefiles_vol_coherency,
David Howells
maintainerPATCH v3 19/33 afs: Add more tracepoints to do with tracking validity
In reply to David HowellsAdd wrappers to set and clear the callback promise and to mark a directory as invalidated, and add tracepoints to track these events:
(1) afs_cb_promise: Log when a callback promise is set on a vnode.
(2) afs_vnode_invalid: Log when the server's callback promise for a vnode is no longer valid and we need to refetch the vnode metadata.
(3) afs_dir_invalid: Log when the contents of a directory are marked invalid and requiring refetching from the server and the cache invalidating.
and two tracepoints to record data version number management:
(4) afs_set_dv: Log when the DV is recorded on a vnode.
(5) afs_dv_mismatch: Log when the DV recorded on a vnode plus the expected delta for the operation does not match the DV we got back from the server.
--- fs/afs/callback.c | 4 +- fs/afs/dir.c | 14 ++- fs/afs/dir_edit.c | 16 ++-- fs/afs/inode.c | 23 +++-- fs/afs/internal.h | 32 +++++++ fs/afs/rotate.c | 4 +- fs/afs/validation.c | 31 ++++--- include/trace/events/afs.h | 169 +++++++++++++++++++++++++++++++++++-- 8 files changed, 248 insertions(+), 45 deletions(-)
fs/afs/callback.c +2 -2
@@ -41,7 +41,7 @@ static void afs_volume_init_callback(struct afs_volume *volume)list_for_each_entry (vnode, &volume->open_mmaps, cb_mmap_link) {if (vnode->cb_v_check !=atomic_read (&volume->cb_v_break)) { -atomic64_set (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ); +afs_clear_cb_promise (vnode, afs_cb_promise_clear_vol_init_cb);queue_work (system_unbound_wq, &vnode->cb_work); } }
@@ -79,7 +79,7 @@ void __afs_break_callback(struct afs_vnode *vnode, enum afs_cb_break_reason reas_enter ("" );clear_bit (AFS_VNODE_NEW_CONTENT , &vnode->flags); -if (atomic64_xchg (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ) !=AFS_NO_CB_PROMISE ) { +if (afs_clear_cb_promise (vnode, afs_cb_promise_clear_cb_break)) { vnode->cb_break++; vnode->cb_v_check =atomic_read (&vnode->volume->cb_v_break);afs_clear_permits (vnode);
fs/afs/dir.c +6 -8
@@ -324,8 +324,7 @@ static struct afs_read *afs_read_dir(struct afs_vnode *dvnode, struct key *key) folio =filemap_get_folio (mapping, i);if (IS_ERR (folio)) { -if (test_and_clear_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) -afs_stat_v (dvnode, n_inval); +afs_invalidate_dir (dvnode, afs_dir_invalid_reclaimed_folio); folio =__filemap_get_folio (mapping, i,FGP_LOCK |FGP_CREAT , mapping->gfp_mask);
@@ -1388,8 +1387,8 @@ static void afs_dir_remove_subdir(struct dentry *dentry)clear_nlink (&vnode->netfs.inode);set_bit (AFS_VNODE_DELETED , &vnode->flags); -atomic64_set (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_clear_cb_promise (vnode, afs_cb_promise_clear_rmdir); +afs_invalidate_dir (vnode, afs_dir_invalid_subdir_removed); } }
@@ -1851,6 +1850,7 @@ static void afs_rename_success(struct afs_operation *op)write_seqlock (&vnode->cb_lock); new_dv = vnode->status.data_version +1 ; +trace_afs_set_dv (vnode, new_dv); vnode->status.data_version = new_dv;inode_set_iversion_raw (&vnode->netfs.inode, new_dv);
@@ -2063,8 +2063,7 @@ static bool afs_dir_release_folio(struct folio *folio, gfp_t gfp_flags)folio_detach_private (folio);/* The directory will need reloading. */ -if (test_and_clear_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) -afs_stat_v (dvnode, n_relpg); +afs_invalidate_dir (dvnode, afs_dir_invalid_release_folio);return true; }
@@ -2081,8 +2080,7 @@ static void afs_dir_invalidate_folio(struct folio *folio, size_t offset,BUG_ON (!folio_test_locked (folio));/* The directory will need reloading. */ -if (test_and_clear_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) -afs_stat_v (dvnode, n_inval); +afs_invalidate_dir (dvnode, afs_dir_invalid_inval_folio);/* we clean up only if the entire folio is being invalidated */ if (offset ==0 && length ==folio_size (folio))
fs/afs/dir_edit.c +8 -8
@@ -116,7 +116,7 @@ static struct folio *afs_dir_get_folio(struct afs_vnode *vnode, pgoff_t index)FGP_LOCK |FGP_ACCESSED |FGP_CREAT , mapping->gfp_mask);if (IS_ERR (folio)) { -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_get_block);return NULL ; }if (!folio_test_private (folio))
@@ -220,7 +220,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode, i_size =i_size_read (&vnode->netfs.inode);if (i_size >AFS_DIR_BLOCK_SIZE *AFS_DIR_MAX_BLOCKS || (i_size & (AFS_DIR_BLOCK_SIZE -1 ))) { -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_add_bad_size);return ; }
@@ -299,7 +299,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,* succeeded. Download the directory again. */ trace_afs_edit_dir (vnode, why, afs_edit_dir_create_nospc,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_add_no_slots); goto out_unmap; new_directory:
@@ -358,7 +358,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode, goto out_unmap; error_too_many_blocks: -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_add_too_many_blocks); error:trace_afs_edit_dir (vnode, why, afs_edit_dir_create_error,0 ,0 ,0 ,0 , name->name); goto out_unmap;
@@ -388,7 +388,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,if (i_size <AFS_DIR_BLOCK_SIZE || i_size >AFS_DIR_BLOCK_SIZE *AFS_DIR_MAX_BLOCKS || (i_size & (AFS_DIR_BLOCK_SIZE -1 ))) { -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_rem_bad_size);return ; } nr_blocks = i_size /AFS_DIR_BLOCK_SIZE ;
@@ -440,7 +440,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,/* Didn't find the dirent to clobber. Download the directory again. */ trace_afs_edit_dir (vnode, why, afs_edit_dir_delete_noent,0 ,0 ,0 ,0 , name->name); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_rem_wrong_name); goto out_unmap; found_dirent:
@@ -510,7 +510,7 @@ void afs_edit_dir_update_dotdot(struct afs_vnode *vnode, struct afs_vnode *new_d i_size =i_size_read (&vnode->netfs.inode);if (i_size <AFS_DIR_BLOCK_SIZE ) { -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_upd_bad_size);return ; } nr_blocks = i_size /AFS_DIR_BLOCK_SIZE ;
@@ -542,7 +542,7 @@ void afs_edit_dir_update_dotdot(struct afs_vnode *vnode, struct afs_vnode *new_d/* Didn't find the dirent to clobber. Download the directory again. */ trace_afs_edit_dir (vnode, why, afs_edit_dir_update_nodd,0 ,0 ,0 ,0 ,".." ); -clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags); +afs_invalidate_dir (vnode, afs_dir_invalid_edit_upd_no_dd); goto out; found_dirent:
fs/afs/inode.c +14 -9
@@ -140,15 +140,17 @@ static int afs_inode_init_from_status(struct afs_operation *op,afs_set_netfs_context (vnode); vnode->invalid_before = status->data_version; +trace_afs_set_dv (vnode, status->data_version);inode_set_iversion_raw (&vnode->netfs.inode, status->data_version);if (!vp->scb.have_cb) {/* it's a symlink we just created (the fileserver * didn't give us a callback) */ -atomic64_set (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ); +afs_clear_cb_promise (vnode, afs_cb_promise_set_new_symlink); }else { vnode->cb_server = op->server; -atomic64_set (&vnode->cb_expires_at, vp->scb.callback.expires_at); +afs_set_cb_promise (vnode, vp->scb.callback.expires_at, + afs_cb_promise_set_new_inode); }write_sequnlock (&vnode->cb_lock);
@@ -207,12 +209,17 @@ static void afs_apply_status(struct afs_operation *op,if (vp->update_ctime)inode_set_ctime_to_ts (inode, op->ctime); -if (vnode->status.data_version != status->data_version) +if (vnode->status.data_version != status->data_version) { +trace_afs_set_dv (vnode, status->data_version); data_changed = true; + } vnode->status = *status;if (vp->dv_before + vp->dv_delta != status->data_version) { +trace_afs_dv_mismatch (vnode, vp->dv_before, vp->dv_delta, + status->data_version); +if (vnode->cb_ro_snapshot ==atomic_read (&vnode->volume->cb_ro_snapshot) &&atomic64_read (&vnode->cb_expires_at) !=AFS_NO_CB_PROMISE ) pr_warn("kAFS: vnode modified {%llx:%llu} %llx->%llx %s (op=%x)\n",
@@ -223,12 +230,10 @@ static void afs_apply_status(struct afs_operation *op, op->debug_id); vnode->invalid_before = status->data_version; -if (vnode->status.type ==AFS_FTYPE_DIR ) { -if (test_and_clear_bit (AFS_VNODE_DIR_VALID , &vnode->flags)) -afs_stat_v (vnode, n_inval); - }else { +if (vnode->status.type ==AFS_FTYPE_DIR ) +afs_invalidate_dir (vnode, afs_dir_invalid_dv_mismatch); +else set_bit (AFS_VNODE_ZAP_DATA , &vnode->flags); - } change_size = true; data_changed = true; unexpected_jump = true;
@@ -273,7 +278,7 @@ static void afs_apply_callback(struct afs_operation *op,if (!afs_cb_is_broken (vp->cb_break_before, vnode)) {if (op->volume->type ==AFSVL_RWVOL ) vnode->cb_server = op->server; -atomic64_set (&vnode->cb_expires_at, cb->expires_at); +afs_set_cb_promise (vnode, cb->expires_at, afs_cb_promise_set_apply_cb); } }
fs/afs/internal.h +32 -0
@@ -1713,6 +1713,38 @@ static inline int afs_bad(struct afs_vnode *vnode, enum afs_file_error where)return -EIO ; } +/* +* Set the callback promise on a vnode. +*/ +static inline void afs_set_cb_promise (struct afs_vnode *vnode,time64_t expires_at, +enum afs_cb_promise_trace trace) +{ +atomic64_set (&vnode->cb_expires_at, expires_at); +trace_afs_cb_promise (vnode, trace); +} + +/* +* Clear the callback promise on a vnode, returning true if it was promised. +*/ +static inline bool afs_clear_cb_promise (struct afs_vnode *vnode, +enum afs_cb_promise_trace trace) +{ +trace_afs_cb_promise (vnode, trace); +return atomic64_xchg (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ) !=AFS_NO_CB_PROMISE ; +} + +/* +* Mark a directory as being invalid. +*/ +static inline void afs_invalidate_dir (struct afs_vnode *dvnode, +enum afs_dir_invalid_trace trace) +{ +if (test_and_clear_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) { +trace_afs_dir_invalid (dvnode, trace); +afs_stat_v (dvnode, n_inval); + } +} +/*****************************************************************************/ /* * debug tracing
fs/afs/rotate.c +2 -2
@@ -99,7 +99,7 @@ static bool afs_start_fs_iteration(struct afs_operation *op,write_seqlock (&vnode->cb_lock); ASSERTCMP(cb_server, ==, vnode->cb_server); vnode->cb_server =NULL ; -if (atomic64_xchg (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ) !=AFS_NO_CB_PROMISE ) +if (afs_clear_cb_promise (vnode, afs_cb_promise_clear_rotate_server)) vnode->cb_break++;write_sequnlock (&vnode->cb_lock); }
@@ -583,7 +583,7 @@ bool afs_select_fileserver(struct afs_operation *op)if (vnode->cb_server != server) { vnode->cb_server = server; vnode->cb_v_check =atomic_read (&vnode->volume->cb_v_break); -atomic64_set (&vnode->cb_expires_at,AFS_NO_CB_PROMISE ); +afs_clear_cb_promise (vnode, afs_cb_promise_clear_server_change); } retry_server:
fs/afs/validation.c +20 -11
@@ -120,22 +120,31 @@bool afs_check_validity (const struct afs_vnode *vnode) {const struct afs_volume *volume = vnode->volume; +enum afs_vnode_invalid_trace trace = afs_vnode_valid_trace; +time64_t cb_expires_at =atomic64_read (&vnode->cb_expires_at);time64_t deadline =ktime_get_real_seconds () +10 ;if (test_bit (AFS_VNODE_DELETED , &vnode->flags))return true; -if (atomic_read (&volume->cb_v_check) !=atomic_read (&volume->cb_v_break) || -atomic64_read (&vnode->cb_expires_at) <= deadline || - volume->cb_expires_at <= deadline || - vnode->cb_ro_snapshot !=atomic_read (&volume->cb_ro_snapshot) || - vnode->cb_scrub !=atomic_read (&volume->cb_scrub) || -test_bit (AFS_VNODE_ZAP_DATA , &vnode->flags)) { -_debug ("inval" ); -return false; - } - -return true; +if (atomic_read (&volume->cb_v_check) !=atomic_read (&volume->cb_v_break)) + trace = afs_vnode_invalid_trace_cb_v_break; +else if (cb_expires_at ==AFS_NO_CB_PROMISE ) + trace = afs_vnode_invalid_trace_no_cb_promise; +else if (cb_expires_at <= deadline) + trace = afs_vnode_invalid_trace_expired; +else if (volume->cb_expires_at <= deadline) + trace = afs_vnode_invalid_trace_vol_expired; +else if (vnode->cb_ro_snapshot !=atomic_read (&volume->cb_ro_snapshot)) + trace = afs_vnode_invalid_trace_cb_ro_snapshot; +else if (vnode->cb_scrub !=atomic_read (&volume->cb_scrub)) + trace = afs_vnode_invalid_trace_cb_scrub; +else if (test_bit (AFS_VNODE_ZAP_DATA , &vnode->flags)) + trace = afs_vnode_invalid_trace_zap_data; +else +return true; +trace_afs_vnode_invalid (vnode, trace); +return false; }/*
include/trace/events/afs.h +164 -5
238 lines not on this page · raw message
David Howells
maintainerPATCH v3 20/33 netfs: Add functions to build/clean a buffer in a folio_queue
In reply to David HowellsAdd two netfslib functions to build up or clean up a buffer in a folio_queue. The first, netfs_alloc_folioq_buffer() will add folios to a buffer, extending up at least to the given size. If it can, it will add multipage folios. The folios are optionally have the mapping set and will have the index set according to the distance from the front of the folio queue.
The second function will free up a folio queue and put any folios in the queue that have the first mark set.
The netfs_folio tracepoint is also altered to cope with folios that have a NULL mapping, and the folios being added/put will have trace lines emitted and will be accounted in the stats.
--- fs/netfs/misc.c | 95 ++++++++++++++++++++++++++++++++++++ include/linux/netfs.h | 6 +++ include/trace/events/netfs.h | 6 +-- 3 files changed, 103 insertions(+), 4 deletions(-)
fs/netfs/misc.c +95 -0
@@ -8,6 +8,101 @@#include <linux/swap.h> #include "internal.h" +/** +* netfs_alloc_folioq_buffer - Allocate buffer space into a folio queue +* @mapping: Address space to set on the folio (or NULL). +* @_buffer: Pointer to the folio queue to add to (may point to a NULL; updated). +* @_cur_size: Current size of the buffer (updated). +* @size: Target size of the buffer. +* @gfp: The allocation constraints. +*/ +int netfs_alloc_folioq_buffer (struct address_space *mapping, +struct folio_queue **_buffer, +size_t *_cur_size,ssize_t size,gfp_t gfp) +{ +struct folio_queue *tail = *_buffer, *p; + + size =round_up (size,PAGE_SIZE ); +if (*_cur_size >= size) +return 0 ; + +if (tail) +while (tail->next) + tail = tail->next; + +do { +struct folio *folio; +int order =0 , slot; + +if (!tail ||folioq_full (tail)) { + p =netfs_folioq_alloc (0 ,GFP_NOFS , netfs_trace_folioq_alloc_buffer); +if (!p) +return -ENOMEM ; +if (tail) { + tail->next = p; + p->prev = tail; + }else { + *_buffer = p; + } + tail = p; + } + +if (size - *_cur_size >PAGE_SIZE ) + order =umin (ilog2 (size - *_cur_size) -PAGE_SHIFT , +MAX_PAGECACHE_ORDER ); + + folio =folio_alloc (gfp, order); +if (!folio && order >0 ) + folio =folio_alloc (gfp,0 ); +if (!folio) +return -ENOMEM ; + + folio->mapping = mapping; + folio->index = *_cur_size /PAGE_SIZE ; +trace_netfs_folio (folio, netfs_folio_trace_alloc_buffer); + slot =folioq_append_mark (tail, folio); + *_cur_size +=folioq_folio_size (tail, slot); + }while (*_cur_size < size); + +return 0 ; +} +EXPORT_SYMBOL (netfs_alloc_folioq_buffer); + +/** +* netfs_free_folioq_buffer - Free a folio queue. +* @fq: The start of the folio queue to free +* +* Free up a chain of folio_queues and, if marked, the marked folios they point +* to. +*/ +void netfs_free_folioq_buffer (struct folio_queue *fq) +{ +struct folio_queue *next; +struct folio_batch fbatch; + +folio_batch_init (&fbatch); + +for (; fq; fq = next) { +for (int slot =0 ; slot <folioq_count (fq); slot++) { +struct folio *folio =folioq_folio (fq, slot); +if (!folio || + !folioq_is_marked (fq, slot)) +continue ; + +trace_netfs_folio (folio, netfs_folio_trace_put); +if (folio_batch_add (&fbatch, folio)) +folio_batch_release (&fbatch); + } + +netfs_stat_d (&netfs_n_folioq); + next = fq->next; +kfree (fq); + } + +folio_batch_release (&fbatch); +} +EXPORT_SYMBOL (netfs_free_folioq_buffer); +/* * Reset the subrequest iterator to refer just to the region remaining to be * read. The iterator may or may not have been advanced by socket ops or
include/linux/netfs.h +6 -0
@@ -458,6 +458,12 @@ struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,void netfs_folioq_free (struct folio_queue *folioq,unsigned int /*enum netfs_trace_folioq*/ trace); +/* Buffer wrangling helpers API. */ +int netfs_alloc_folioq_buffer (struct address_space *mapping, +struct folio_queue **_buffer, +size_t *_cur_size,ssize_t size,gfp_t gfp); +void netfs_free_folioq_buffer (struct folio_queue *fq); +/** * netfs_inode - Get the netfs inode context from the inode * @inode: The inode to query
include/trace/events/netfs.h +2 -4
@@ -155,6 +155,7 @@ EM(netfs_streaming_filled_page, "mod-streamw-f") \ EM(netfs_streaming_cont_filled_page, "mod-streamw-f+") \ EM(netfs_folio_trace_abandon, "abandon") \ + EM(netfs_folio_trace_alloc_buffer, "alloc-buf") \ EM(netfs_folio_trace_cancel_copy, "cancel-copy") \ EM(netfs_folio_trace_cancel_store, "cancel-store") \EM (netfs_folio_trace_clear,"clear" ) \
@@ -195,10 +196,7 @@ E_(netfs_trace_donate_to_deferred_next, "defer-next")#define netfs_folioq_traces \ - EM(netfs_trace_folioq_alloc_append_folio, "alloc-apf") \ - EM(netfs_trace_folioq_alloc_read_prep, "alloc-r-prep") \ - EM(netfs_trace_folioq_alloc_read_prime, "alloc-r-prime") \ - EM(netfs_trace_folioq_alloc_read_sing, "alloc-r-sing") \ + EM(netfs_trace_folioq_alloc_buffer, "alloc-buf") \ EM(netfs_trace_folioq_clear, "clear") \ EM(netfs_trace_folioq_delete, "delete") \ EM(netfs_trace_folioq_make_space, "make-space") \
David Howells
maintainerPATCH v3 21/33 netfs: Add support for caching single monolithic objects such as AFS dirs
In reply to David HowellsAdd support for caching the content of a file that contains a single monolithic object that must be read/written with a single I/O operation, such as an AFS directory.
--- fs/netfs/Makefile | 1 + fs/netfs/buffered_read.c | 11 +- fs/netfs/internal.h | 2 + fs/netfs/main.c | 2 + fs/netfs/objects.c | 2 + fs/netfs/read_collect.c | 45 +++++++- fs/netfs/read_single.c | 202 ++++++++++++++++++++++++++++++++++ fs/netfs/stats.c | 4 +- fs/netfs/write_collect.c | 6 +- fs/netfs/write_issue.c | 203 ++++++++++++++++++++++++++++++++++- include/linux/netfs.h | 10 ++ include/trace/events/netfs.h | 4 + 12 files changed, 478 insertions(+), 14 deletions(-) create mode 100644 fs/netfs/read_single.c
fs/netfs/Makefile +1 -0
@@ -13,6 +13,7 @@ netfs-y := \ read_collect.o \ read_pgpriv2.o \ read_retry.o \ + read_single.o \ rolling_buffer.o \ write_collect.o \ write_issue.o \
fs/netfs/buffered_read.c +8 -3
@@ -137,14 +137,17 @@ static enum netfs_io_source netfs_cache_prepare_read(struct netfs_io_request *rr loff_t i_size) {struct netfs_cache_resources *cres = &rreq->cache_resources; +enum netfs_io_source source;if (!cres->ops)return NETFS_DOWNLOAD_FROM_SERVER ; -return cres->ops->prepare_read (subreq, i_size); + source = cres->ops->prepare_read (subreq, i_size); +trace_netfs_sreq (subreq, netfs_sreq_trace_prepare); +return source; + } -static void netfs_cache_read_terminated (void *priv,ssize_t transferred_or_error, -bool was_async) +void netfs_cache_read_terminated (void *priv,ssize_t transferred_or_error,bool was_async) {struct netfs_io_subrequest *subreq = priv;
@@ -213,6 +216,8 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)unsigned long long zp =umin (ictx->zero_point, rreq->i_size);size_t len = subreq->len; +if (unlikely (rreq->origin ==NETFS_READ_SINGLE )) + zp = rreq->i_size;if (subreq->start >= zp) { subreq->source = source =NETFS_FILL_WITH_ZEROES ; goto fill_with_zeroes;
fs/netfs/internal.h +2 -0
@@ -23,6 +23,7 @@/* * buffered_read.c */ +void netfs_cache_read_terminated (void *priv,ssize_t transferred_or_error,bool was_async);int netfs_prefetch_for_write (struct file *file,struct folio *folio,size_t offset,size_t len);
@@ -110,6 +111,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq);extern atomic_t netfs_n_rh_dio_read;extern atomic_t netfs_n_rh_readahead;extern atomic_t netfs_n_rh_read_folio; +extern atomic_t netfs_n_rh_read_single;extern atomic_t netfs_n_rh_rreq;extern atomic_t netfs_n_rh_sreq;extern atomic_t netfs_n_rh_download;
fs/netfs/main.c +2 -0
@@ -37,9 +37,11 @@ static const char *netfs_origins[nr__netfs_io_origin] = { [NETFS_READAHEAD] = "RA", [NETFS_READPAGE] = "RP", [NETFS_READ_GAPS] = "RG", + [NETFS_READ_SINGLE] = "R1", [NETFS_READ_FOR_WRITE] = "RW", [NETFS_DIO_READ] = "DR", [NETFS_WRITEBACK] = "WB", + [NETFS_WRITEBACK_SINGLE] = "W1", [NETFS_WRITETHROUGH] = "WT", [NETFS_UNBUFFERED_WRITE] = "UW", [NETFS_DIO_WRITE] = "DW",
fs/netfs/objects.c +2 -0
@@ -54,6 +54,7 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,if (origin ==NETFS_READAHEAD || origin ==NETFS_READPAGE || origin ==NETFS_READ_GAPS || + origin ==NETFS_READ_SINGLE || origin ==NETFS_READ_FOR_WRITE || origin ==NETFS_DIO_READ )INIT_WORK (&rreq->work,NULL );
@@ -196,6 +197,7 @@ struct netfs_io_subrequest *netfs_alloc_subrequest(struct netfs_io_request *rreqcase NETFS_READAHEAD :case NETFS_READPAGE :case NETFS_READ_GAPS : +case NETFS_READ_SINGLE :case NETFS_READ_FOR_WRITE :case NETFS_DIO_READ :INIT_WORK (&subreq->work, netfs_read_subreq_termination_worker);
fs/netfs/read_collect.c +43 -2
58 lines not on this page · raw message
fs/netfs/read_single.c new file +202 -0
202 lines not on this page · raw message
fs/netfs/stats.c +3 -1
@@ -12,6 +12,7 @@atomic_t netfs_n_rh_dio_read;atomic_t netfs_n_rh_readahead;atomic_t netfs_n_rh_read_folio; +atomic_t netfs_n_rh_read_single;atomic_t netfs_n_rh_rreq;atomic_t netfs_n_rh_sreq;atomic_t netfs_n_rh_download;
@@ -46,10 +47,11 @@ atomic_t netfs_n_folioq;int netfs_stats_show (struct seq_file *m,void *v) { -seq_printf (m,"Reads : DR=%u RA=%u RF=%u WB=%u WBZ=%u\n" , +seq_printf (m,"Reads : DR=%u RA=%u RF=%u RS=%u WB=%u WBZ=%u\n" ,atomic_read (&netfs_n_rh_dio_read),atomic_read (&netfs_n_rh_readahead),atomic_read (&netfs_n_rh_read_folio), +atomic_read (&netfs_n_rh_read_single),atomic_read (&netfs_n_rh_write_begin),atomic_read (&netfs_n_rh_write_zskip)); seq_printf(m, "Writes : BW=%u WT=%u DW=%u WP=%u 2C=%u\n",
fs/netfs/write_collect.c +3 -3
@@ -17,7 +17,7 @@#define HIT_PENDING 0x01/* A front op was still pending */ #define NEED_REASSESS 0x02/* Need to loop round and reassess */ #define MADE_PROGRESS 0x04/* Made progress cleaning up a stream or the folio set */ -#define BUFFERED 0x08/* The pagecache needs cleaning up */ +#define NEED_UNLOCK 0x08/* The pagecache needs unlocking */ #define NEED_RETRY 0x10/* A front op requests retrying */ #define SAW_FAILURE 0x20/* One stream or hit a permanent failure */
@@ -179,7 +179,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)if (wreq->origin ==NETFS_WRITEBACK || wreq->origin ==NETFS_WRITETHROUGH || wreq->origin ==NETFS_PGPRIV2_COPY_TO_CACHE ) - notes =BUFFERED ; + notes =NEED_UNLOCK ;else notes =0 ;
@@ -276,7 +276,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)trace_netfs_collect_state (wreq, wreq->collected_to, notes);/* Unlock any folios that we have now finished with. */ -if (notes &BUFFERED ) { +if (notes &NEED_UNLOCK ) {if (wreq->cleaned_to < wreq->collected_to)netfs_writeback_unlock_folios (wreq, ¬es); }else {
fs/netfs/write_issue.c +198 -5
224 lines not on this page · raw message
include/linux/netfs.h +10 -0
@@ -73,6 +73,7 @@ struct netfs_inode {#define NETFS_ICTX_UNBUFFERED 1/* I/O should not use the pagecache */ #define NETFS_ICTX_WRITETHROUGH 2/* Write-through caching */ #define NETFS_ICTX_MODIFIED_ATTR 3/* Indicate change in mtime/ctime */ +#define NETFS_ICTX_SINGLE_NO_UPLOAD 4/* Monolithic payload, cache but no upload */ };/*
@@ -210,9 +211,11 @@ enum netfs_io_origin {NETFS_READAHEAD ,/* This read was triggered by readahead */ NETFS_READPAGE ,/* This read is a synchronous read */ NETFS_READ_GAPS ,/* This read is a synchronous read to fill gaps */ +NETFS_READ_SINGLE ,/* This read should be treated as a single object */ NETFS_READ_FOR_WRITE ,/* This read is to prepare a write */ NETFS_DIO_READ ,/* This is a direct I/O read */ NETFS_WRITEBACK ,/* This write was triggered by writepages */ +NETFS_WRITEBACK_SINGLE ,/* This monolithic write was triggered by writepages */ NETFS_WRITETHROUGH ,/* This write was made by netfs_perform_write() */ NETFS_UNBUFFERED_WRITE ,/* This is an unbuffered write */ NETFS_DIO_WRITE, /* This is a direct I/O write */
@@ -409,6 +412,13 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter * struct netfs_group *netfs_group);ssize_t netfs_file_write_iter (struct kiocb *iocb,struct iov_iter *from); +/* Single, monolithic object read/write API. */ +void netfs_single_mark_inode_dirty (struct inode *inode); +ssize_t netfs_read_single (struct inode *inode,struct file *file,struct iov_iter *iter); +int netfs_writeback_single (struct address_space *mapping, +struct writeback_control *wbc, +struct iov_iter *iter); +/* Address operations API */ struct readahead_control ;void netfs_readahead (struct readahead_control *);
include/trace/events/netfs.h +4 -0
@@ -21,6 +21,7 @@ EM(netfs_read_trace_readahead, "READAHEAD") \ EM(netfs_read_trace_readpage, "READPAGE ") \ EM(netfs_read_trace_read_gaps, "READ-GAPS") \ + EM(netfs_read_trace_read_single, "READ-SNGL") \ EM(netfs_read_trace_prefetch_for_write, "PREFETCHW") \E_ (netfs_read_trace_write_begin,"WRITEBEGN" )
@@ -35,9 +36,11 @@ EM(NETFS_READAHEAD, "RA") \ EM(NETFS_READPAGE, "RP") \ EM(NETFS_READ_GAPS, "RG") \ + EM(NETFS_READ_SINGLE, "R1") \ EM(NETFS_READ_FOR_WRITE, "RW") \ EM(NETFS_DIO_READ, "DR") \ EM(NETFS_WRITEBACK, "WB") \ + EM(NETFS_WRITEBACK_SINGLE, "W1") \ EM(NETFS_WRITETHROUGH, "WT") \ EM(NETFS_UNBUFFERED_WRITE, "UW") \EM (NETFS_DIO_WRITE ,"DW" ) \
@@ -47,6 +50,7 @@ EM(netfs_rreq_trace_assess, "ASSESS ") \ EM(netfs_rreq_trace_copy, "COPY ") \ EM(netfs_rreq_trace_collect, "COLLECT") \ + EM(netfs_rreq_trace_dirty, "DIRTY ") \ EM(netfs_rreq_trace_done, "DONE ") \ EM(netfs_rreq_trace_free, "FREE ") \EM (netfs_rreq_trace_redirty,"REDIRTY" ) \
David Howells
maintainerPATCH v3 22/33 afs: Make afs_init_request() get a key if not given a file
In reply to David HowellsIn a future patch, AFS directory caching will go through netfslib and this will involve, at times, running on behalf of ->lookup(), which doesn't provide us with a file from which we can get an authentication key.
If a file isn't provided, make afs_init_request() get a key from the process's keyrings instead when setting up a read.
--- fs/afs/file.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
fs/afs/file.c +16 -0
@@ -372,10 +372,26 @@ static int afs_symlink_read_folio(struct file *file, struct folio *folio)static int afs_init_request (struct netfs_io_request *rreq,struct file *file) { +struct afs_vnode *vnode =AFS_FS_I (rreq->inode); +if (file) rreq->netfs_priv =key_get (afs_file_key (file)); rreq->rsize =256 *1024 ; rreq->wsize =256 *1024 *1024 ; + +switch (rreq->origin) { +case NETFS_READ_SINGLE : +if (!file) { +struct key *key =afs_request_key (vnode->volume->cell); + +if (IS_ERR (key)) +return PTR_ERR (key); + rreq->netfs_priv = key; + } +break ; +default : +break ; + }return 0 ; }
David Howells
maintainerPATCH v3 23/33 afs: Use netfslib for directories
In reply to David HowellsIn the AFS ecosystem, directories are just a special type of file that is downloaded and parsed locally. Download is done by the same mechanism as ordinary files and the data can be cached. There is one important semantic restriction on directories over files: the client must download the entire directory in one go because, for example, the server could fabricate the contents of the blob on the fly with each download and give a different image each time.
So that we can cache the directory download, switch AFS directory support over to using the netfslib single-object API, thereby allowing directory content to be stored in the local cache.
To make this work, the following changes are made:
(1) A directory's contents are now stored in a folio_queue chain attached to the afs_vnode (inode) struct rather than its associated pagecache, though multipage folios are still used to hold the data. The folio queue is discarded when the directory inode is evicted.
This also helps with the phasing out of ITER_XARRAY.
(2) Various directory operations are made to use and unuse the cache cookie.
(3) The content checking, content dumping and content iteration are now performed with a standard iov_iter iterator over the contents of the folio queue.
(4) Iteration and modification must be done with the vnode's validate_lock held. In conjunction with (1), this means that the iteration can be done without the need to lock pages or take extra refs on them, unlike when accessing ->i_pages.
(5) Convert to using netfs_read_single() to read data.
(6) Provide a ->writepages() to call netfs_writeback_single() to save the data to the cache according to the VM's scheduling whilst holding the validate_lock read-locked as (4).
(7) Change local directory image editing functions:
(a) Provide a function to get a specific block by number from the folio_queue as we can no longer use the i_pages xarray to locate folios by index. This uses a cursor to remember the current position as we need to iterate through the directory contents. The block is kmapped before being returned.
(b) Make the function in (a) extend the directory by an extra folio if we run out of space.
(c) Raise the check of the block free space counter, for those blocks that have one, higher in the function to eliminate a call to get a block.
(d) Remove the page unlocking and putting done during the editing loops. This is no longer necessary as the folio_queue holds the references and the pages are no longer in the pagecache.
(e) Mark the inode dirty and pin the cache usage till writeback at the end of a successful edit.
(8) Don't set the large_folios flag on the inode as we do the allocation ourselves rather than the VM doing it automatically.
(9) Mark the inode as being a single object that isn't uploaded to the server.
(10) Enable caching on directories.
(11) Only set the upload key for writeback for regular files.
Notes:
(*) We keep the ->release_folio(), ->invalidate_folio() and
->migrate_folio() ops as we set the mapping pointer on the folio.
--- fs/afs/dir.c | 741 +++++++++++++++++++------------------ fs/afs/dir_edit.c | 183 ++++----- fs/afs/file.c | 8 + fs/afs/inode.c | 21 +- fs/afs/internal.h | 15 + fs/afs/super.c | 2 + fs/afs/write.c | 4 +- include/trace/events/afs.h | 6 +- 8 files changed, 510 insertions(+), 470 deletions(-)
fs/afs/dir.c +379 -362
@@ -13,6 +13,7 @@#include <linux/ctype.h> #include <linux/sched.h> #include <linux/iversion.h> +#include <linux/iov_iter.h> #include <linux/task_io_accounting_ops.h> #include "internal.h" #include "afs_fs.h"
@@ -42,15 +43,6 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir,static int afs_rename (struct mnt_idmap *idmap,struct inode *old_dir,struct dentry *old_dentry,struct inode *new_dir,struct dentry *new_dentry,unsigned int flags); -static bool afs_dir_release_folio (struct folio *folio,gfp_t gfp_flags); -static void afs_dir_invalidate_folio (struct folio *folio,size_t offset, -size_t length); - -static bool afs_dir_dirty_folio (struct address_space *mapping, -struct folio *folio) -{ -BUG ();/* This should never happen. */ -}const struct file_operations afs_dir_file_operations = { .open = afs_dir_open,
@@ -75,10 +67,7 @@ const struct inode_operations afs_dir_inode_operations = { };const struct address_space_operations afs_dir_aops = { - .dirty_folio = afs_dir_dirty_folio, - .release_folio = afs_dir_release_folio, - .invalidate_folio = afs_dir_invalidate_folio, - .migrate_folio = filemap_migrate_folio, + .writepages = afs_single_writepages, };const struct dentry_operations afs_fs_dentry_operations = {
@@ -105,146 +94,120 @@ struct afs_lookup_cookie {struct afs_fid fids[50 ]; }; +static void afs_dir_unuse_cookie (struct afs_vnode *dvnode,int ret) +{ +if (ret ==0 ) { +struct afs_vnode_cache_aux aux; +loff_t i_size =i_size_read (&dvnode->netfs.inode); + +afs_set_cache_aux (dvnode, &aux); +fscache_unuse_cookie (afs_vnode_cache (dvnode), &aux, &i_size); + }else { +fscache_unuse_cookie (afs_vnode_cache (dvnode),NULL ,NULL ); + } +} +/* -* Drop the refs that we're holding on the folios we were reading into. We've -* got refs on the first nr_pages pages. +* Iterate through a kmapped directory segment, dumping a summary of +* the contents. */ -static void afs_dir_read_cleanup (struct afs_read *req) +static size_t afs_dir_dump_step (void *iter_base,size_t progress,size_t len, +void *priv,void *priv2) { -struct address_space *mapping = req->vnode->netfs.inode.i_mapping; -struct folio *folio; -pgoff_t last = req->nr_pages -1 ; +do { +union afs_xdr_dir_block *block = iter_base; -XA_STATE (xas, &mapping->i_pages,0 ); +pr_warn ("[%05zx] %32phN\n" , progress, block); + iter_base +=AFS_DIR_BLOCK_SIZE ; + progress +=AFS_DIR_BLOCK_SIZE ; + len -=AFS_DIR_BLOCK_SIZE ; + }while (len >0 ); -if (unlikely (!req->nr_pages)) -return ; +return len; +} -rcu_read_lock (); -xas_for_each (&xas, folio, last) { -if (xas_retry (&xas, folio)) -continue ; -BUG_ON (xa_is_value (folio)); - ASSERTCMP(folio->mapping, ==, mapping); +/* +* Dump the contents of a directory. +*/ +static void afs_dir_dump (struct afs_vnode *dvnode) +{ +struct iov_iter iter; +unsigned long long i_size =i_size_read (&dvnode->netfs.inode); -folio_put (folio); - } +pr_warn ("DIR %llx:%llx is=%llx\n" , + dvnode->fid.vid, dvnode->fid.vnode, i_size); -rcu_read_unlock (); +iov_iter_folio_queue (&iter,ITER_SOURCE , dvnode->directory,0 ,0 , i_size); +iterate_folioq (&iter,iov_iter_count (&iter),NULL ,NULL , + afs_dir_dump_step); }/* * check that a directory folio is valid */ -static bool afs_dir_check_folio (struct afs_vnode *dvnode,struct folio *folio, -loff_t i_size) +static bool afs_dir_check_block (struct afs_vnode *dvnode,size_t progress, +union afs_xdr_dir_block *block) { -union afs_xdr_dir_block *block; -size_t offset, size; -loff_t pos; +if (block->hdr.magic !=AFS_DIR_MAGIC ) { +pr_warn ("%s(%lx): [%zx] bad magic %04x\n" , + __func__, dvnode->netfs.inode.i_ino, + progress,ntohs (block->hdr.magic)); +trace_afs_dir_check_failed (dvnode, progress); +trace_afs_file_error (dvnode, -EIO , afs_file_error_dir_bad_magic); +return false; + } -/* Determine how many magic numbers there should be in this folio, but -* we must take care because the directory may change size under us. +/* Make sure each block is NUL terminated so we can reasonably +* use string functions on it. The filenames in the folio +* *should* be NUL-terminated anyway. */ - pos =folio_pos (folio); -if (i_size <= pos) - goto checked; - - size =min_t (loff_t,folio_size (folio), i_size - pos); -for (offset =0 ; offset < size; offset +=sizeof (*block)) { - block =kmap_local_folio (folio, offset); -if (block->hdr.magic !=AFS_DIR_MAGIC ) { -printk ("kAFS: %s(%lx): [%llx] bad magic %zx/%zx is %04hx\n" , - __func__, dvnode->netfs.inode.i_ino, - pos, offset, size,ntohs (block->hdr.magic)); -trace_afs_dir_check_failed (dvnode, pos + offset, i_size); -kunmap_local (block); -trace_afs_file_error (dvnode, -EIO , afs_file_error_dir_bad_magic); - goto error; - } - -/* Make sure each block is NUL terminated so we can reasonably -* use string functions on it. The filenames in the folio -* *should* be NUL-terminated anyway. -*/ - ((u8 *)block)[AFS_DIR_BLOCK_SIZE -1 ] =0 ; - -kunmap_local (block); - } -checked: + ((u8 *)block)[AFS_DIR_BLOCK_SIZE -1 ] =0 ;afs_stat_v (dvnode, n_read_dir);return true; - -error: -return false; }/* -* Dump the contents of a directory. +* Iterate through a kmapped directory segment, checking the content. */ -static void afs_dir_dump (struct afs_vnode *dvnode,struct afs_read *req) +static size_t afs_dir_check_step (void *iter_base,size_t progress,size_t len, +void *priv,void *priv2) { -union afs_xdr_dir_block *block; -struct address_space *mapping = dvnode->netfs.inode.i_mapping; -struct folio *folio; -pgoff_t last = req->nr_pages -1 ; -size_t offset, size; - -XA_STATE (xas, &mapping->i_pages,0 ); - -pr_warn ("DIR %llx:%llx f=%llx l=%llx al=%llx\n" , - dvnode->fid.vid, dvnode->fid.vnode, - req->file_size, req->len, req->actual_len); -pr_warn ("DIR %llx %x %zx %zx\n" , - req->pos, req->nr_pages, - req->iter->iov_offset,iov_iter_count (req->iter)); - -xas_for_each (&xas, folio, last) { -if (xas_retry (&xas, folio)) -continue ; +struct afs_vnode *dvnode = priv; -BUG_ON (folio->mapping != mapping); +if (WARN_ON_ONCE (progress %AFS_DIR_BLOCK_SIZE || + len %AFS_DIR_BLOCK_SIZE )) +return len; - size =min_t (loff_t,folio_size (folio), req->actual_len -folio_pos (folio)); -for (offset =0 ; offset < size; offset +=sizeof (*block)) { - block =kmap_local_folio (folio, offset); -pr_warn ("[%02lx] %32phN\n" , folio->index + offset, block); -kunmap_local (block); - } - } +do { +if (!afs_dir_check_block (dvnode, progress, iter_base)) +break ; + iter_base +=AFS_DIR_BLOCK_SIZE ; + len -=AFS_DIR_BLOCK_SIZE ; + }while (len >0 ); + +return len; }/* -* Check all the blocks in a directory. All the folios are held pinned. +* Check all the blocks in a directory. */ -static int afs_dir_check (struct afs_vnode *dvnode,struct afs_read *req) +static int afs_dir_check (struct afs_vnode *dvnode) { -struct address_space *mapping = dvnode->netfs.inode.i_mapping; -struct folio *folio; -pgoff_t last = req->nr_pages -1 ; -int ret =0 ; +struct iov_iter iter; +unsigned long long i_size =i_size_read (&dvnode->netfs.inode); +size_t checked =0 ; -XA_STATE (xas, &mapping->i_pages,0 ); - -if (unlikely (!req->nr_pages)) +if (unlikely (!i_size))return 0 ; -rcu_read_lock (); -xas_for_each (&xas, folio, last) { -if (xas_retry (&xas, folio)) -continue ; - -BUG_ON (folio->mapping != mapping); - -if (!afs_dir_check_folio (dvnode, folio, req->actual_len)) { -afs_dir_dump (dvnode, req); - ret = -EIO ; -break ; - } +iov_iter_folio_queue (&iter,ITER_SOURCE , dvnode->directory,0 ,0 , i_size); + checked =iterate_folioq (&iter,iov_iter_count (&iter), dvnode,NULL , + afs_dir_check_step); +if (checked != i_size) { +afs_dir_dump (dvnode); +return -EIO ; } - -rcu_read_unlock (); -return ret; +return 0 ; }/*
@@ -264,115 +227,118 @@ static int afs_dir_open(struct inode *inode, struct file *file) }/* -* Read the directory into the pagecache in one go, scrubbing the previous -* contents. The list of folios is returned, pinning them so that they don't -* get reclaimed during the iteration. +* Read a file in a single download. */ -static struct afs_read *afs_read_dir (struct afs_vnode *dvnode,struct key *key) - __acquires(&dvnode->validate_lock) +static ssize_t afs_do_read_single (struct afs_vnode *dvnode,struct file *file) { -struct address_space *mapping = dvnode->netfs.inode.i_mapping; -struct afs_read *req; +struct iov_iter iter; +ssize_t ret;loff_t i_size; -int nr_pages, i; -int ret; -loff_t remote_size =0 ; - -_enter ("" ); - - req =kzalloc (sizeof (*req),GFP_KERNEL ); -if (!req) -return ERR_PTR (-ENOMEM ); +bool is_dir = (S_ISDIR (dvnode->netfs.inode.i_mode) && + !test_bit (AFS_VNODE_MOUNTPOINT , &dvnode->flags)); -refcount_set (&req->usage,1 ); - req->vnode = dvnode; - req->key =key_get (key); - req->cleanup = afs_dir_read_cleanup; - -expand: i_size =i_size_read (&dvnode->netfs.inode); -if (i_size < remote_size) - i_size = remote_size; -if (i_size <2048 ) { - ret =afs_bad (dvnode, afs_file_error_dir_small); - goto error; - } -if (i_size >2048 *1024 ) { -trace_afs_file_error (dvnode, -EFBIG , afs_file_error_dir_big); - ret = -EFBIG ; - goto error; +if (is_dir) { +if (i_size <AFS_DIR_BLOCK_SIZE ) +return afs_bad (dvnode, afs_file_error_dir_small); +if (i_size >AFS_DIR_BLOCK_SIZE *1024 ) { +trace_afs_file_error (dvnode, -EFBIG , afs_file_error_dir_big); +return -EFBIG ; + } + }else { +if (i_size >AFSPATHMAX ) { +trace_afs_file_error (dvnode, -EFBIG , afs_file_error_dir_big); +return -EFBIG ; + } } -_enter ("%llu" , i_size); +/* Expand the storage. TODO: Shrink the storage too. */ +if (dvnode->directory_size < i_size) { +size_t cur_size = dvnode->directory_size; - nr_pages = (i_size +PAGE_SIZE -1 ) /PAGE_SIZE ; + ret =netfs_alloc_folioq_buffer (NULL , + &dvnode->directory, &cur_size, i_size, +mapping_gfp_mask (dvnode->netfs.inode.i_mapping)); + dvnode->directory_size = cur_size; +if (ret <0 ) +return ret; + } - req->actual_len = i_size;/* May change */ - req->len = nr_pages *PAGE_SIZE ;/* We can ask for more than there is */ - req->data_version = dvnode->status.data_version;/* May change */ -iov_iter_xarray (&req->def_iter,ITER_DEST , &dvnode->netfs.inode.i_mapping->i_pages, -0 , i_size); - req->iter = &req->def_iter; +iov_iter_folio_queue (&iter,ITER_DEST , dvnode->directory,0 ,0 , dvnode->directory_size); -/* Fill in any gaps that we might find where the memory reclaimer has -* been at work and pin all the folios. If there are any gaps, we will -* need to reread the entire directory contents. +/* AFS requires us to perform the read of a directory synchronously as +* a single unit to avoid issues with the directory contents being +* changed between reads. */ - i = req->nr_pages; -while (i < nr_pages) { -struct folio *folio; - - folio =filemap_get_folio (mapping, i); -if (IS_ERR (folio)) { -afs_invalidate_dir (dvnode, afs_dir_invalid_reclaimed_folio); - folio =__filemap_get_folio (mapping, - i,FGP_LOCK |FGP_CREAT , - mapping->gfp_mask); -if (IS_ERR (folio)) { - ret =PTR_ERR (folio); - goto error; - } -folio_attach_private (folio, (void *)1 ); -folio_unlock (folio); + ret =netfs_read_single (&dvnode->netfs.inode, file, &iter); +if (ret >=0 ) { + i_size =i_size_read (&dvnode->netfs.inode); +if (i_size > ret) { +/* The content has grown, so we need to expand the +* buffer. +*/ + ret = -ESTALE ; + }else if (is_dir) { +int ret2 =afs_dir_check (dvnode); + +if (ret2 <0 ) + ret = ret2; + }else if (i_size <folioq_folio_size (dvnode->directory,0 )) { +/* NUL-terminate a symlink. */ +char *symlink =kmap_local_folio (folioq_folio (dvnode->directory,0 ),0 ); + + symlink[i_size] =0 ; +kunmap_local (symlink); } - - req->nr_pages +=folio_nr_pages (folio); - i +=folio_nr_pages (folio); } -/* If we're going to reload, we need to lock all the pages to prevent -* races. -*/ +return ret; +} + +ssize_t afs_read_single (struct afs_vnode *dvnode,struct file *file) +{ +ssize_t ret; + +fscache_use_cookie (afs_vnode_cache (dvnode), false); + ret =afs_do_read_single (dvnode, file); +fscache_unuse_cookie (afs_vnode_cache (dvnode),NULL ,NULL ); +return ret; +} + +/* +* Read the directory into a folio_queue buffer in one go, scrubbing the +* previous contents. We return -ESTALE if the caller needs to call us again. +*/ +static ssize_t afs_read_dir (struct afs_vnode *dvnode,struct file *file) + __acquires(&dvnode->validate_lock) +{ +ssize_t ret; +loff_t i_size; + + i_size =i_size_read (&dvnode->netfs.inode); + ret = -ERESTARTSYS ;if (down_read_killable (&dvnode->validate_lock) <0 ) goto error; -if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) - goto success; +/* We only need to reread the data if it became invalid - or if we +* haven't read it yet. +*/ +if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && + dvnode->directory) + goto valid;up_read (&dvnode->validate_lock);if (down_write_killable (&dvnode->validate_lock) <0 ) goto error; -if (!test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) { -trace_afs_reload_dir (dvnode); - ret =afs_fetch_data (dvnode, req); -if (ret <0 ) - goto error_unlock; - -task_io_account_read (PAGE_SIZE * req->nr_pages); +if (!test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags)) +afs_invalidate_cache (dvnode,0 ); -if (req->len < req->file_size) { -/* The content has grown, so we need to expand the -* buffer. -*/ -up_write (&dvnode->validate_lock); - remote_size = req->file_size; - goto expand; - } - -/* Validate the data we just read. */ - ret =afs_dir_check (dvnode, req); +if (!test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) || + !dvnode->directory) { +trace_afs_reload_dir (dvnode); + ret =afs_read_single (dvnode, file);if (ret <0 ) goto error_unlock;
@@ -382,15 +348,14 @@ static struct afs_read *afs_read_dir(struct afs_vnode *dvnode, struct key *key) }downgrade_write (&dvnode->validate_lock); -success: -return req; +valid: +return i_size; error_unlock:up_write (&dvnode->validate_lock); error: -afs_put_read (req); -_leave (" = %d" , ret); -return ERR_PTR (ret); +_leave (" = %zd" , ret); +return ret; }/*
@@ -398,79 +363,69 @@ static struct afs_read *afs_read_dir(struct afs_vnode *dvnode, struct key *key)*/ static int afs_dir_iterate_block (struct afs_vnode *dvnode,struct dir_context *ctx, -union afs_xdr_dir_block *block, -unsigned blkoff) +union afs_xdr_dir_block *block) {union afs_xdr_dirent *dire; -unsigned offset, next, curr, nr_slots; +unsigned int blknum, base, hdr, pos, next, nr_slots;size_t nlen;int tmp; -_enter ("%llx,%x" , ctx->pos, blkoff); + blknum = ctx->pos /AFS_DIR_BLOCK_SIZE ; + base = blknum *AFS_DIR_SLOTS_PER_BLOCK ; + hdr = (blknum ==0 ?AFS_DIR_RESV_BLOCKS0 :AFS_DIR_RESV_BLOCKS ); + pos =DIV_ROUND_UP (ctx->pos,AFS_DIR_DIRENT_SIZE ) - base; - curr = (ctx->pos - blkoff) /sizeof (union afs_xdr_dirent ); +_enter ("%llx,%x" , ctx->pos, blknum);/* walk through the block, an entry at a time */ -for (offset = (blkoff ==0 ?AFS_DIR_RESV_BLOCKS0 :AFS_DIR_RESV_BLOCKS ); - offset <AFS_DIR_SLOTS_PER_BLOCK ; - offset = next - ) { +for (unsigned int slot = hdr; slot <AFS_DIR_SLOTS_PER_BLOCK ; slot = next) {/* skip entries marked unused in the bitmap */ -if (!(block->hdr.bitmap[offset /8 ] & - (1 << (offset %8 )))) { -_debug ("ENT[%zu.%u]: unused" , - blkoff /sizeof (union afs_xdr_dir_block ), offset); - next = offset +1 ; -if (offset >= curr) - ctx->pos = blkoff + - next *sizeof (union afs_xdr_dirent ); +if (!(block->hdr.bitmap[slot /8 ] & + (1 << (slot %8 )))) { +_debug ("ENT[%x]: Unused" , base + slot); + next = slot +1 ; +if (next >= pos) + ctx->pos = (base + next) *sizeof (union afs_xdr_dirent );continue ; }/* got a valid entry */ - dire = &block->dirents[offset]; + dire = &block->dirents[slot]; nlen =strnlen (dire->u.name, -sizeof (*block) - - offset *sizeof (union afs_xdr_dirent )); + (unsigned long )(block +1 ) - (unsigned long )dire->u.name -1 );if (nlen >AFSNAMEMAX -1 ) { -_debug ("ENT[%zu]: name too long (len %u/%zu)" , - blkoff /sizeof (union afs_xdr_dir_block ), - offset, nlen); +_debug ("ENT[%x]: Name too long (len %zx)" , + base + slot, nlen);return afs_bad (dvnode, afs_file_error_dir_name_too_long); } -_debug ("ENT[%zu.%u]: %s %zu \"%s\"" , - blkoff /sizeof (union afs_xdr_dir_block ), offset, - (offset < curr ?"skip" :"fill" ), +_debug ("ENT[%x]: %s %zx \"%s\"" , + base + slot, (slot < pos ?"skip" :"fill" ), nlen, dire->u.name); nr_slots =afs_dir_calc_slots (nlen); - next = offset + nr_slots; + next = slot + nr_slots;if (next >AFS_DIR_SLOTS_PER_BLOCK ) { -_debug ("ENT[%zu.%u]:" -" %u extends beyond end dir block" -" (len %zu)" , - blkoff /sizeof (union afs_xdr_dir_block ), - offset, next, nlen); +_debug ("ENT[%x]: extends beyond end dir block (len %zx)" , + base + slot, nlen);return afs_bad (dvnode, afs_file_error_dir_over_end); }/* Check that the name-extension dirents are all allocated */ for (tmp =1 ; tmp < nr_slots; tmp++) { -unsigned int ix = offset + tmp; -if (!(block->hdr.bitmap[ix /8 ] & (1 << (ix %8 )))) { -_debug ("ENT[%zu.u]:" -" %u unmarked extension (%u/%u)" , - blkoff /sizeof (union afs_xdr_dir_block ), - offset, tmp, nr_slots); +unsigned int xslot = slot + tmp; + +if (!(block->hdr.bitmap[xslot /8 ] & (1 << (xslot %8 )))) { +_debug ("ENT[%x]: Unmarked extension (%x/%x)" , + base + slot, tmp, nr_slots);return afs_bad (dvnode, afs_file_error_dir_unmarked_ext); } }/* skip if starts before the current position */ -if (offset < curr) { -if (next > curr) - ctx->pos = blkoff + next *sizeof (union afs_xdr_dirent ); +if (slot < pos) { +if (next > pos) + ctx->pos = (base + next) *sizeof (union afs_xdr_dirent );continue ; }
@@ -484,7 +439,7 @@ static int afs_dir_iterate_block(struct afs_vnode *dvnode,return 0 ; } - ctx->pos = blkoff + next *sizeof (union afs_xdr_dirent ); + ctx->pos = (base + next) *sizeof (union afs_xdr_dirent ); }_leave (" = 1 [more]" );
@@ -492,67 +447,97 @@ static int afs_dir_iterate_block(struct afs_vnode *dvnode, }/* -* iterate through the data blob that lists the contents of an AFS directory +* Iterate through a kmapped directory segment. */ -static int afs_dir_iterate (struct inode *dir,struct dir_context *ctx, -struct key *key,afs_dataversion_t *_dir_version) +static size_t afs_dir_iterate_step (void *iter_base,size_t progress,size_t len, +void *priv,void *priv2) { -struct afs_vnode *dvnode =AFS_FS_I (dir); -union afs_xdr_dir_block *dblock; -struct afs_read *req; -struct folio *folio; -unsigned offset, size; +struct dir_context *ctx = priv2; +struct afs_vnode *dvnode = priv;int ret; -_enter ("{%lu},%u,," , dir->i_ino, (unsigned )ctx->pos); - -if (test_bit (AFS_VNODE_DELETED , &AFS_FS_I (dir)->flags)) { -_leave (" = -ESTALE" ); -return -ESTALE ; +if (WARN_ON_ONCE (progress %AFS_DIR_BLOCK_SIZE || + len %AFS_DIR_BLOCK_SIZE )) { +pr_err ("Mis-iteration prog=%zx len=%zx\n" , + progress %AFS_DIR_BLOCK_SIZE , + len %AFS_DIR_BLOCK_SIZE ); +return len; } - req =afs_read_dir (dvnode, key); -if (IS_ERR (req)) -return PTR_ERR (req); - *_dir_version = req->data_version; +do { + ret =afs_dir_iterate_block (dvnode, ctx, iter_base); +if (ret !=1 ) +break ; -/* round the file position up to the next entry boundary */ - ctx->pos +=sizeof (union afs_xdr_dirent ) -1 ; - ctx->pos &= ~(sizeof (union afs_xdr_dirent ) -1 ); + ctx->pos =round_up (ctx->pos,AFS_DIR_BLOCK_SIZE ); + iter_base +=AFS_DIR_BLOCK_SIZE ; + len -=AFS_DIR_BLOCK_SIZE ; + }while (len >0 ); -/* walk through the blocks in sequence */ - ret =0 ; -while (ctx->pos < req->actual_len) { -/* Fetch the appropriate folio from the directory and re-add it -* to the LRU. We have all the pages pinned with an extra ref. -*/ - folio =__filemap_get_folio (dir->i_mapping, ctx->pos /PAGE_SIZE , -FGP_ACCESSED ,0 ); -if (IS_ERR (folio)) { - ret =afs_bad (dvnode, afs_file_error_dir_missing_page); -break ; - } +return len; +} - offset =round_down (ctx->pos,sizeof (*dblock)) -folio_pos (folio); - size =min_t (loff_t,folio_size (folio), - req->actual_len -folio_pos (folio)); +/* +* Iterate through the directory folios under RCU conditions. +*/ +static int afs_dir_iterate_contents (struct inode *dir,struct dir_context *ctx) +{ +struct afs_vnode *dvnode =AFS_FS_I (dir); +struct iov_iter iter; +unsigned long long i_size =i_size_read (dir); +int ret =0 ; -do { - dblock =kmap_local_folio (folio, offset); - ret =afs_dir_iterate_block (dvnode, ctx, dblock, -folio_pos (folio) + offset); -kunmap_local (dblock); -if (ret !=1 ) - goto out; +/* Round the file position up to the next entry boundary */ + ctx->pos =round_up (ctx->pos,sizeof (union afs_xdr_dirent )); - }while (offset +=sizeof (*dblock), offset < size); +if (i_size <=0 || ctx->pos >= i_size) +return 0 ; - ret =0 ; - } +iov_iter_folio_queue (&iter,ITER_SOURCE , dvnode->directory,0 ,0 , i_size); +iov_iter_advance (&iter,round_down (ctx->pos,AFS_DIR_BLOCK_SIZE )); + +iterate_folioq (&iter,iov_iter_count (&iter), dvnode, ctx, + afs_dir_iterate_step); + +if (ret == -ESTALE ) +afs_invalidate_dir (dvnode, afs_dir_invalid_iter_stale); +return ret; +} + +/* +* iterate through the data blob that lists the contents of an AFS directory +*/ +static int afs_dir_iterate (struct inode *dir,struct dir_context *ctx, +struct file *file,afs_dataversion_t *_dir_version) +{ +struct afs_vnode *dvnode =AFS_FS_I (dir); +int retry_limit =100 ; +int ret; + +_enter ("{%lu},%llx,," , dir->i_ino, ctx->pos); + +do { +if (--retry_limit <0 ) { +pr_warn ("afs_read_dir(): Too many retries\n" ); + ret = -ESTALE ; +break ; + } + ret =afs_read_dir (dvnode, file); +if (ret <0 ) { +if (ret != -ESTALE ) +break ; +if (test_bit (AFS_VNODE_DELETED , &AFS_FS_I (dir)->flags)) { + ret = -ESTALE ; +break ; + } +continue ; + } + *_dir_version =inode_peek_iversion_raw (dir); + + ret =afs_dir_iterate_contents (dir, ctx); +up_read (&dvnode->validate_lock); + }while (ret == -ESTALE ); -out: -up_read (&dvnode->validate_lock); -afs_put_read (req);_leave (" = %d" , ret);return ret; }
@@ -564,8 +549,7 @@ static int afs_readdir(struct file *file, struct dir_context *ctx) {afs_dataversion_t dir_version; -return afs_dir_iterate (file_inode (file), ctx,afs_file_key (file), - &dir_version); +return afs_dir_iterate (file_inode (file), ctx, file, &dir_version); }/*
@@ -606,7 +590,7 @@ static bool afs_lookup_one_filldir(struct dir_context *ctx, const char *name,* - just returns the FID the dentry name maps to if found */ static int afs_do_lookup_one (struct inode *dir,struct dentry *dentry, -struct afs_fid *fid,struct key *key, +struct afs_fid *fid,afs_dataversion_t *_dir_version) {struct afs_super_info *as = dir->i_sb->s_fs_info;
@@ -620,7 +604,7 @@ static int afs_do_lookup_one(struct inode *dir, struct dentry *dentry,_enter ("{%lu},%p{%pd}," , dir->i_ino, dentry, dentry);/* search the directory */ - ret =afs_dir_iterate (dir, &cookie.ctx, key, _dir_version); + ret =afs_dir_iterate (dir, &cookie.ctx,NULL , _dir_version);if (ret <0 ) {_leave (" = %d [iter]" , ret);return ret;
@@ -787,8 +771,7 @@ static bool afs_server_supports_ibulk(struct afs_vnode *dvnode)* files in one go and create inodes for them. The inode of the file we were * asked for is returned. */ -static struct inode *afs_do_lookup (struct inode *dir,struct dentry *dentry, -struct key *key) +static struct inode *afs_do_lookup (struct inode *dir,struct dentry *dentry) {struct afs_lookup_cookie *cookie;struct afs_vnode_param *vp;
@@ -816,7 +799,7 @@ static struct inode *afs_do_lookup(struct inode *dir, struct dentry *dentry, cookie->one_only = true;/* search the directory */ - ret =afs_dir_iterate (dir, &cookie->ctx, key, &data_version); + ret =afs_dir_iterate (dir, &cookie->ctx,NULL , &data_version);if (ret <0 ) goto out;
@@ -925,8 +908,7 @@ static struct inode *afs_do_lookup(struct inode *dir, struct dentry *dentry,/* * Look up an entry in a directory with @sys substitution. */ -static struct dentry *afs_lookup_atsys (struct inode *dir,struct dentry *dentry, -struct key *key) +static struct dentry *afs_lookup_atsys (struct inode *dir,struct dentry *dentry) {struct afs_sysnames *subs;struct afs_net *net =afs_i2net (dir);
@@ -974,7 +956,6 @@ static struct dentry *afs_lookup_atsys(struct inode *dir, struct dentry *dentry,afs_put_sysnames (subs);kfree (buf); out_p: -key_put (key);return ret; }
@@ -988,7 +969,6 @@ static struct dentry *afs_lookup(struct inode *dir, struct dentry *dentry,struct afs_fid fid = {};struct inode *inode;struct dentry *d; -struct key *key;int ret; _enter("{%llx:%llu},%p{%pd},",
@@ -1006,15 +986,9 @@ static struct dentry *afs_lookup(struct inode *dir, struct dentry *dentry,return ERR_PTR (-ESTALE ); } - key =afs_request_key (dvnode->volume->cell); -if (IS_ERR (key)) { -_leave (" = %ld [key]" ,PTR_ERR (key)); -return ERR_CAST (key); - } - - ret =afs_validate (dvnode, key); + ret =afs_validate (dvnode,NULL );if (ret <0 ) { -key_put (key); +afs_dir_unuse_cookie (dvnode, ret);_leave (" = %d [val]" , ret);return ERR_PTR (ret); }
@@ -1024,11 +998,10 @@ static struct dentry *afs_lookup(struct inode *dir, struct dentry *dentry, dentry->d_name.name[dentry->d_name.len - 3] == 's' && dentry->d_name.name[dentry->d_name.len - 2] == 'y' && dentry->d_name.name[dentry->d_name.len - 1] == 's') -return afs_lookup_atsys (dir, dentry, key); +return afs_lookup_atsys (dir, dentry);afs_stat_v (dvnode, n_lookup); - inode =afs_do_lookup (dir, dentry, key); -key_put (key); + inode =afs_do_lookup (dir, dentry);if (inode ==ERR_PTR (-ENOENT )) inode =afs_try_auto_mntpt (dentry, dir);
@@ -1154,7 +1127,7 @@ static int afs_d_revalidate(struct dentry *dentry, unsigned int flags)afs_stat_v (dir, n_reval);/* search the directory for this vnode */ - ret =afs_do_lookup_one (&dir->netfs.inode, dentry, &fid, key, &dir_version); + ret =afs_do_lookup_one (&dir->netfs.inode, dentry, &fid, &dir_version);switch (ret) {case 0 :/* the filename maps to something */
@@ -1316,18 +1289,21 @@ static void afs_create_success(struct afs_operation *op)static void afs_create_edit_dir (struct afs_operation *op) { +struct netfs_cache_resources cres = {};struct afs_vnode_param *dvp = &op->file[0 ];struct afs_vnode_param *vp = &op->file[1 ];struct afs_vnode *dvnode = dvp->vnode;_enter ("op=%08x" , op->debug_id); +fscache_begin_write_operation (&cres,afs_vnode_cache (dvnode));down_write (&dvnode->validate_lock);if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && dvnode->status.data_version == dvp->dv_before + dvp->dv_delta)afs_edit_dir_add (dvnode, &op->dentry->d_name, &vp->fid, op->create.reason);up_write (&dvnode->validate_lock); +fscache_end_operation (&cres); }static void afs_create_put (struct afs_operation *op)
@@ -1355,6 +1331,7 @@ static int afs_mkdir(struct mnt_idmap *idmap, struct inode *dir, {struct afs_operation *op;struct afs_vnode *dvnode =AFS_FS_I (dir); +int ret;_enter ("{%llx:%llu},{%pd},%ho" , dvnode->fid.vid, dvnode->fid.vnode, dentry, mode);
@@ -1365,6 +1342,8 @@ static int afs_mkdir(struct mnt_idmap *idmap, struct inode *dir,return PTR_ERR (op); } +fscache_use_cookie (afs_vnode_cache (dvnode), true); +afs_op_set_vnode (op,0 , dvnode); op->file[0 ].dv_delta =1 ; op->file[0 ].modification = true;
@@ -1374,7 +1353,9 @@ static int afs_mkdir(struct mnt_idmap *idmap, struct inode *dir, op->create.reason = afs_edit_dir_for_mkdir; op->mtime =current_time (dir); op->ops = &afs_mkdir_operation; -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); +afs_dir_unuse_cookie (dvnode, ret); +return ret; }/*
@@ -1402,18 +1383,21 @@ static void afs_rmdir_success(struct afs_operation *op)static void afs_rmdir_edit_dir (struct afs_operation *op) { +struct netfs_cache_resources cres = {};struct afs_vnode_param *dvp = &op->file[0 ];struct afs_vnode *dvnode = dvp->vnode;_enter ("op=%08x" , op->debug_id);afs_dir_remove_subdir (op->dentry); +fscache_begin_write_operation (&cres,afs_vnode_cache (dvnode));down_write (&dvnode->validate_lock);if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && dvnode->status.data_version == dvp->dv_before + dvp->dv_delta)afs_edit_dir_remove (dvnode, &op->dentry->d_name, afs_edit_dir_for_rmdir);up_write (&dvnode->validate_lock); +fscache_end_operation (&cres); }static void afs_rmdir_put (struct afs_operation *op)
@@ -1448,6 +1432,8 @@ static int afs_rmdir(struct inode *dir, struct dentry *dentry)if (IS_ERR (op))return PTR_ERR (op); +fscache_use_cookie (afs_vnode_cache (dvnode), true); +afs_op_set_vnode (op,0 , dvnode); op->file[0 ].dv_delta =1 ; op->file[0 ].modification = true;
@@ -1476,10 +1462,13 @@ static int afs_rmdir(struct inode *dir, struct dentry *dentry)/* Not all systems that can host afs servers have ENOTEMPTY. */ if (ret == -EEXIST ) ret = -ENOTEMPTY ; +out: +afs_dir_unuse_cookie (dvnode, ret);return ret; error: -return afs_put_operation (op); + ret =afs_put_operation (op); + goto out; }/*
@@ -1542,16 +1531,19 @@ static void afs_unlink_success(struct afs_operation *op)static void afs_unlink_edit_dir (struct afs_operation *op) { +struct netfs_cache_resources cres = {};struct afs_vnode_param *dvp = &op->file[0 ];struct afs_vnode *dvnode = dvp->vnode;_enter ("op=%08x" , op->debug_id); +fscache_begin_write_operation (&cres,afs_vnode_cache (dvnode));down_write (&dvnode->validate_lock);if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && dvnode->status.data_version == dvp->dv_before + dvp->dv_delta)afs_edit_dir_remove (dvnode, &op->dentry->d_name, afs_edit_dir_for_unlink);up_write (&dvnode->validate_lock); +fscache_end_operation (&cres); }static void afs_unlink_put (struct afs_operation *op)
@@ -1590,6 +1582,8 @@ static int afs_unlink(struct inode *dir, struct dentry *dentry)if (IS_ERR (op))return PTR_ERR (op); +fscache_use_cookie (afs_vnode_cache (dvnode), true); +afs_op_set_vnode (op,0 , dvnode); op->file[0 ].dv_delta =1 ; op->file[0 ].modification = true;
@@ -1636,10 +1630,10 @@ static int afs_unlink(struct inode *dir, struct dentry *dentry)afs_wait_for_operation (op); } -return afs_put_operation (op); - error: -return afs_put_operation (op); + ret =afs_put_operation (op); +afs_dir_unuse_cookie (dvnode, ret); +return ret; }static const struct afs_operation_ops afs_create_operation = {
@@ -1673,6 +1667,8 @@ static int afs_create(struct mnt_idmap *idmap, struct inode *dir, goto error; } +fscache_use_cookie (afs_vnode_cache (dvnode), true); +afs_op_set_vnode (op,0 , dvnode); op->file[0 ].dv_delta =1 ; op->file[0 ].modification = true;
@@ -1683,7 +1679,9 @@ static int afs_create(struct mnt_idmap *idmap, struct inode *dir, op->create.reason = afs_edit_dir_for_create; op->mtime =current_time (dir); op->ops = &afs_create_operation; -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); +afs_dir_unuse_cookie (dvnode, ret); +return ret; error:d_drop (dentry);
@@ -1748,6 +1746,8 @@ static int afs_link(struct dentry *from, struct inode *dir, goto error; } +fscache_use_cookie (afs_vnode_cache (dvnode), true); + ret =afs_validate (vnode, op->key);if (ret <0 ) goto error_op;
@@ -1763,10 +1763,13 @@ static int afs_link(struct dentry *from, struct inode *dir, op->dentry_2 = from; op->ops = &afs_link_operation; op->create.reason = afs_edit_dir_for_link; -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); +afs_dir_unuse_cookie (dvnode, ret); +return ret; error_op:afs_put_operation (op); +afs_dir_unuse_cookie (dvnode, ret); error:d_drop (dentry);_leave (" = %d" , ret);
@@ -1810,6 +1813,8 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir, goto error; } +fscache_use_cookie (afs_vnode_cache (dvnode), true); +afs_op_set_vnode (op,0 , dvnode); op->file[0 ].dv_delta =1 ;
@@ -1818,7 +1823,9 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir, op->create.reason = afs_edit_dir_for_symlink; op->create.symlink = content; op->mtime =current_time (dir); -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); +afs_dir_unuse_cookie (dvnode, ret); +return ret; error:d_drop (dentry);
@@ -1860,6 +1867,7 @@ static void afs_rename_success(struct afs_operation *op)static void afs_rename_edit_dir (struct afs_operation *op) { +struct netfs_cache_resources orig_cres = {}, new_cres = {};struct afs_vnode_param *orig_dvp = &op->file[0 ];struct afs_vnode_param *new_dvp = &op->file[1 ];struct afs_vnode *orig_dvnode = orig_dvp->vnode;
@@ -1876,6 +1884,10 @@ static void afs_rename_edit_dir(struct afs_operation *op) op->rename.rehash =NULL ; } +fscache_begin_write_operation (&orig_cres,afs_vnode_cache (orig_dvnode)); +if (new_dvnode != orig_dvnode) +fscache_begin_write_operation (&new_cres,afs_vnode_cache (new_dvnode)); +down_write (&orig_dvnode->validate_lock);if (test_bit (AFS_VNODE_DIR_VALID , &orig_dvnode->flags) && orig_dvnode->status.data_version == orig_dvp->dv_before + orig_dvp->dv_delta)
@@ -1925,6 +1937,9 @@ static void afs_rename_edit_dir(struct afs_operation *op)d_move (old_dentry, new_dentry);up_write (&new_dvnode->validate_lock); +fscache_end_operation (&orig_cres); +if (new_dvnode != orig_dvnode) +fscache_end_operation (&new_cres); }static void afs_rename_put (struct afs_operation *op)
@@ -1977,6 +1992,10 @@ static int afs_rename(struct mnt_idmap *idmap, struct inode *old_dir,if (IS_ERR (op))return PTR_ERR (op); +fscache_use_cookie (afs_vnode_cache (orig_dvnode), true); +if (new_dvnode != orig_dvnode) +fscache_use_cookie (afs_vnode_cache (new_dvnode), true); + ret =afs_validate (vnode, op->key);afs_op_set_error (op, ret);if (ret <0 )
@@ -2044,45 +2063,43 @@ static int afs_rename(struct mnt_idmap *idmap, struct inode *old_dir,*/ d_drop (old_dentry); -return afs_do_sync_operation (op); + ret =afs_do_sync_operation (op); +out: +afs_dir_unuse_cookie (orig_dvnode, ret); +if (new_dvnode != orig_dvnode) +afs_dir_unuse_cookie (new_dvnode, ret); +return ret; error: -return afs_put_operation (op); -} - -/* -* Release a directory folio and clean up its private state if it's not busy -* - return true if the folio can now be released, false if not -*/ -static bool afs_dir_release_folio (struct folio *folio,gfp_t gfp_flags) -{ -struct afs_vnode *dvnode =AFS_FS_I (folio_inode (folio)); - -_enter ("{{%llx:%llu}[%lu]}" , dvnode->fid.vid, dvnode->fid.vnode, folio->index); - -folio_detach_private (folio); - -/* The directory will need reloading. */ -afs_invalidate_dir (dvnode, afs_dir_invalid_release_folio); -return true; + ret =afs_put_operation (op); + goto out; }/* -* Invalidate part or all of a folio. +* Write the file contents to the cache as a single blob. */ -static void afs_dir_invalidate_folio (struct folio *folio,size_t offset, -size_t length) +int afs_single_writepages (struct address_space *mapping, +struct writeback_control *wbc) { -struct afs_vnode *dvnode =AFS_FS_I (folio_inode (folio)); - -_enter ("{%lu},%zu,%zu" , folio->index, offset, length); - -BUG_ON (!folio_test_locked (folio)); +struct afs_vnode *dvnode =AFS_FS_I (mapping->host); +struct iov_iter iter; +bool is_dir = (S_ISDIR (dvnode->netfs.inode.i_mode) && + !test_bit (AFS_VNODE_MOUNTPOINT , &dvnode->flags)); +int ret =0 ; -/* The directory will need reloading. */ -afs_invalidate_dir (dvnode, afs_dir_invalid_inval_folio); +/* Need to lock to prevent the folio queue and folios from being thrown +* away. +*/ +down_read (&dvnode->validate_lock); + +if (is_dir ? +test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) : +atomic64_read (&dvnode->cb_expires_at) !=AFS_NO_CB_PROMISE ) { +iov_iter_folio_queue (&iter,ITER_SOURCE , dvnode->directory,0 ,0 , +i_size_read (&dvnode->netfs.inode)); + ret =netfs_writeback_single (mapping, wbc, &iter); + } -/* we clean up only if the entire folio is being invalidated */ -if (offset ==0 && length ==folio_size (folio)) -folio_detach_private (folio); +up_read (&dvnode->validate_lock); +return ret; }
fs/afs/dir_edit.c +82 -101
352 lines not on this page · raw message
fs/afs/file.c +8 -0
@@ -389,6 +389,14 @@ static int afs_init_request(struct netfs_io_request *rreq, struct file *file) rreq->netfs_priv = key; }break ; +case NETFS_WRITEBACK : +case NETFS_WRITETHROUGH : +case NETFS_UNBUFFERED_WRITE : +case NETFS_DIO_WRITE : +if (S_ISREG (rreq->inode->i_mode)) + rreq->io_streams[0 ].avail = true; +break ; +case NETFS_WRITEBACK_SINGLE :default :break ; }
fs/afs/inode.c +19 -2
@@ -110,7 +110,9 @@ static int afs_inode_init_from_status(struct afs_operation *op, inode->i_op = &afs_dir_inode_operations; inode->i_fop = &afs_dir_file_operations; inode->i_mapping->a_ops = &afs_dir_aops; -mapping_set_large_folios (inode->i_mapping); +__set_bit (NETFS_ICTX_SINGLE_NO_UPLOAD , &vnode->netfs.flags); +/* Assume locally cached directory data will be valid. */ +__set_bit (AFS_VNODE_DIR_VALID , &vnode->flags);break ;case AFS_FTYPE_SYMLINK :/* Symlinks with a mode of 0644 are actually mountpoints. */
@@ -440,7 +442,8 @@ static void afs_get_inode_cache(struct afs_vnode *vnode) }__packed key;struct afs_vnode_cache_aux aux; -if (vnode->status.type !=AFS_FTYPE_FILE ) { +if (vnode->status.type !=AFS_FTYPE_FILE && + vnode->status.type !=AFS_FTYPE_DIR ) { vnode->netfs.cache =NULL ;return ; }
@@ -642,6 +645,7 @@ int afs_drop_inode(struct inode *inode)void afs_evict_inode (struct inode *inode) {struct afs_vnode_cache_aux aux; +struct afs_super_info *sbi =AFS_FS_S (inode->i_sb);struct afs_vnode *vnode =AFS_FS_I (inode); _enter("{%llx:%llu.%d}",
@@ -653,8 +657,21 @@ void afs_evict_inode(struct inode *inode) ASSERTCMP(inode->i_ino, ==, vnode->fid.vnode); +if ((S_ISDIR (inode->i_mode)) && + (inode->i_state &I_DIRTY ) && + !sbi->dyn_root) { +struct writeback_control wbc = { + .sync_mode =WB_SYNC_ALL , + .for_sync = true, + .range_end =LLONG_MAX , + }; + +afs_single_writepages (inode->i_mapping, &wbc); + } +netfs_wait_for_outstanding_io (inode);truncate_inode_pages_final (&inode->i_data); +netfs_free_folioq_buffer (vnode->directory);afs_set_cache_aux (vnode, &aux);netfs_clear_inode_writeback (inode, &aux);
fs/afs/internal.h +15 -0
@@ -721,6 +721,7 @@ struct afs_vnode {#define AFS_VNODE_SILLY_DELETED 9/* Set if file has been silly-deleted */ #define AFS_VNODE_MODIFYING 10/* Set if we're performing a modification op */ +struct folio_queue *directory;/* Directory contents */ struct list_head wb_keys;/* List of keys available for writeback */ struct list_head pending_locks;/* locks waiting to be granted */ struct list_head granted_locks;/* locks granted on this file */
@@ -729,6 +730,7 @@ struct afs_vnode {ktime_t locked_at;/* Time at which lock obtained */ enum afs_lock_state lock_state : 8; afs_lock_type_t lock_type : 8; +unsigned int directory_size;/* Amount of space in ->directory */ /* outstanding callback notification on this file */ struct work_struct cb_work;/* Work for mmap'd files */
@@ -984,6 +986,16 @@ static inline void afs_invalidate_cache(struct afs_vnode *vnode, unsigned int fl i_size_read(&vnode->netfs.inode), flags); } +/* +* Directory iteration management. +*/ +struct afs_dir_iter { +struct afs_vnode *dvnode; +struct folio_queue *fq; +unsigned int fpos; +int fq_slot; +}; +#include <trace/events/afs.h> /*****************************************************************************/
@@ -1065,8 +1077,11 @@ extern const struct inode_operations afs_dir_inode_operations;extern const struct address_space_operations afs_dir_aops;extern const struct dentry_operations afs_fs_dentry_operations; +ssize_t afs_read_single (struct afs_vnode *dvnode,struct file *file);extern void afs_d_release (struct dentry *);extern void afs_check_for_remote_deletion (struct afs_operation *); +int afs_single_writepages (struct address_space *mapping, +struct writeback_control *wbc);/* * dir_edit.c
fs/afs/super.c +2 -0
@@ -696,6 +696,8 @@ static struct inode *afs_alloc_inode(struct super_block *sb) vnode->volume =NULL ; vnode->lock_key =NULL ; vnode->permit_cache =NULL ; + vnode->directory =NULL ; + vnode->directory_size =0 ; vnode->flags =1 <<AFS_VNODE_UNSET ; vnode->lock_state =AFS_VNODE_LOCK_NONE ;
fs/afs/write.c +2 -2
@@ -179,8 +179,8 @@ void afs_issue_write(struct netfs_io_subrequest *subreq)*/ void afs_begin_writeback (struct netfs_io_request *wreq) { -afs_get_writeback_key (wreq); - wreq->io_streams[0 ].avail = true; +if (S_ISREG (wreq->inode->i_mode)) +afs_get_writeback_key (wreq); }/*
include/trace/events/afs.h +3 -3
@@ -930,9 +930,9 @@ TRACE_EVENT(afs_sent_data, ); TRACE_EVENT(afs_dir_check_failed, - TP_PROTO(struct afs_vnode *vnode, loff_t off, loff_t i_size), + TP_PROTO(struct afs_vnode *vnode, loff_t off), - TP_ARGS(vnode, off, i_size), + TP_ARGS(vnode, off), TP_STRUCT__entry( __field(struct afs_vnode *, vnode)
@@ -943,7 +943,7 @@ TRACE_EVENT(afs_dir_check_failed,TP_fast_assign ( __entry->vnode = vnode; __entry->off = off; - __entry->i_size = i_size; + __entry->i_size =i_size_read (&vnode->netfs.inode); ), TP_printk("vn=%p %llx/%llx",
David Howells
maintainerRe: [PATCH v3 23/33] afs: Use netfslib for directories
The attached fix needs folding in across this patch (23), patch 24, patch 29 and patch 31.
David
---
commit 9d4429bc7bb3f2b518d6decd1ca0e99e4d80d58e
Author: David Howells <[email protected]> Date: Thu Nov 7 23:46:48 2024 +0000
afs: Fix handling of signals during readdir
When a directory is being read, whether or not the dvnode->directory buffer pointer is NULL is used to track whether we've checked fscache yet. However, if a signal occurs after the buffer being allocated but whilst we're doing the read, we may end up in an invalid state with ->directory set but no data in the buffer.
In this state, afs_readdir(), afs_lookup() and afs_d_revalidate() see corrupt directory contents leading to a variety of malfunctions.
Fix this by providing a specific flag to record whether or not we've performed a read yet - and, incidentally, sampled fscache - rather than using the value in ->directory instead.
fs/afs/dir.c +3 -2
@@ -323,7 +323,7 @@ ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)* haven't read it yet. */ if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && - dvnode->directory) { +test_bit (AFS_VNODE_DIR_READ , &dvnode->flags)) { ret = i_size; goto valid; }
@@ -336,7 +336,7 @@ ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)afs_invalidate_cache (dvnode,0 );if (!test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) || - !dvnode->directory) { + !test_bit (AFS_VNODE_DIR_READ , &dvnode->flags)) {trace_afs_reload_dir (dvnode); ret =afs_read_single (dvnode, file);if (ret <0 )
@@ -345,6 +345,7 @@ ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)// TODO: Trim excess pages set_bit (AFS_VNODE_DIR_VALID , &dvnode->flags); +set_bit (AFS_VNODE_DIR_READ , &dvnode->flags); }else { ret = i_size; }
fs/afs/dir_edit.c +1 -0
@@ -644,4 +644,5 @@ void afs_mkdir_init_dir(struct afs_vnode *dvnode, struct afs_vnode *parent_dvnodnetfs_single_mark_inode_dirty (&dvnode->netfs.inode);set_bit (AFS_VNODE_DIR_VALID , &dvnode->flags); +set_bit (AFS_VNODE_DIR_READ , &dvnode->flags); }
fs/afs/inode.c +5 -3
@@ -39,6 +39,7 @@ void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op) p =kmap_local_folio (folioq_folio (vnode->directory,0 ),0 );memcpy (p, op->create.symlink, size);kunmap_local (p); +set_bit (AFS_VNODE_DIR_READ , &vnode->flags);netfs_single_mark_inode_dirty (&vnode->netfs.inode); }
@@ -60,12 +61,12 @@ const char *afs_get_link(struct dentry *dentry, struct inode *inode,if (!dentry) {/* RCU pathwalk. */ -if (!vnode->directory || !afs_check_validity (vnode)) +if (!test_bit (AFS_VNODE_DIR_READ , &vnode->flags) || !afs_check_validity (vnode))return ERR_PTR (-ECHILD ); goto good; } -if (!vnode->directory) +if (test_bit (AFS_VNODE_DIR_READ , &vnode->flags)) goto fetch; ret =afs_validate (vnode,NULL );
@@ -73,13 +74,14 @@ const char *afs_get_link(struct dentry *dentry, struct inode *inode,return ERR_PTR (ret);if (!test_and_clear_bit (AFS_VNODE_ZAP_DATA , &vnode->flags) && - vnode->directory) +test_bit (AFS_VNODE_DIR_READ , &vnode->flags)) goto good; fetch: ret =afs_read_single (vnode,NULL );if (ret <0 )return ERR_PTR (ret); +set_bit (AFS_VNODE_DIR_READ , &vnode->flags); good: folio =folioq_folio (vnode->directory,0 );
fs/afs/internal.h +1 -0
@@ -705,6 +705,7 @@ struct afs_vnode {#define AFS_VNODE_NEW_CONTENT 8/* Set if file has new content (create/trunc-0) */ #define AFS_VNODE_SILLY_DELETED 9/* Set if file has been silly-deleted */ #define AFS_VNODE_MODIFYING 10/* Set if we're performing a modification op */ +#define AFS_VNODE_DIR_READ 11/* Set if we've read a dir's contents */ struct folio_queue *directory;/* Directory contents */ struct list_head wb_keys;/* List of keys available for writeback */
David Howells
maintainerPATCH v3 24/33 afs: Use netfslib for symlinks, allowing them to be cached
In reply to David HowellsUse netfslib to read symlinks, thereby allowing them to be cached by fscache and cachefiles.
--- fs/afs/file.c | 32 ------------------- fs/afs/inode.c | 64 +++++++++++++++++++++++++++++++++++--- fs/afs/internal.h | 4 ++- fs/afs/mntpt.c | 22 ++++++------- include/trace/events/afs.h | 1 + 5 files changed, 74 insertions(+), 49 deletions(-)
fs/afs/file.c +0 -32
@@ -20,7 +20,6 @@#include "internal.h" static int afs_file_mmap (struct file *file,struct vm_area_struct *vma); -static int afs_symlink_read_folio (struct file *file,struct folio *folio);static ssize_t afs_file_read_iter (struct kiocb *iocb,struct iov_iter *iter); static ssize_t afs_file_splice_read(struct file *in, loff_t *ppos,
@@ -61,13 +60,6 @@ const struct address_space_operations afs_file_aops = { .writepages = afs_writepages, }; -const struct address_space_operations afs_symlink_aops = { - .read_folio = afs_symlink_read_folio, - .release_folio = netfs_release_folio, - .invalidate_folio = netfs_invalidate_folio, - .migrate_folio = filemap_migrate_folio, -}; -static const struct vm_operations_struct afs_vm_ops = { .open = afs_vm_open, .close = afs_vm_close,
@@ -346,30 +338,6 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)queue_work (system_long_wq, &subreq->work); } -static int afs_symlink_read_folio (struct file *file,struct folio *folio) -{ -struct afs_vnode *vnode =AFS_FS_I (folio->mapping->host); -struct afs_read *fsreq; -int ret; - - fsreq =afs_alloc_read (GFP_NOFS ); -if (!fsreq) -return -ENOMEM ; - - fsreq->pos =folio_pos (folio); - fsreq->len =folio_size (folio); - fsreq->vnode = vnode; - fsreq->iter = &fsreq->def_iter; -iov_iter_xarray (&fsreq->def_iter,ITER_DEST , &folio->mapping->i_pages, - fsreq->pos, fsreq->len); - - ret =afs_fetch_data (fsreq->vnode, fsreq); -if (ret ==0 ) -folio_mark_uptodate (folio); -folio_unlock (folio); -return ret; -} -static int afs_init_request (struct netfs_io_request *rreq,struct file *file) {struct afs_vnode *vnode =AFS_FS_I (rreq->inode);
fs/afs/inode.c +59 -5
94 lines not on this page · raw message
fs/afs/internal.h +3 -1
@@ -1115,7 +1115,6 @@ extern void afs_dynroot_depopulate(struct super_block *);* file.c */ extern const struct address_space_operations afs_file_aops; -extern const struct address_space_operations afs_symlink_aops;extern const struct inode_operations afs_file_inode_operations;extern const struct file_operations afs_file_operations;extern const struct netfs_request_ops afs_req_ops;
@@ -1221,6 +1220,9 @@ extern void afs_fs_probe_cleanup(struct afs_net *);*/ extern const struct afs_operation_ops afs_fetch_status_operation; +const char *afs_get_link (struct dentry *dentry,struct inode *inode, +struct delayed_call *callback); +int afs_readlink (struct dentry *dentry,char __user *buffer,int buflen);extern void afs_vnode_commit_status (struct afs_operation *,struct afs_vnode_param *);extern int afs_fetch_status (struct afs_vnode *,struct key *,bool ,afs_access_t *);extern int afs_ilookup5_test_by_fid (struct inode *,void *);
fs/afs/mntpt.c +11 -11
@@ -30,7 +30,7 @@ const struct file_operations afs_mntpt_file_operations = {const struct inode_operations afs_mntpt_inode_operations = { .lookup = afs_mntpt_lookup, - .readlink = page_readlink, + .readlink = afs_readlink, .getattr = afs_getattr, };
@@ -118,9 +118,9 @@ static int afs_mntpt_set_params(struct fs_context *fc, struct dentry *mntpt) ctx->volnamesz =sizeof (afs_root_volume) -1 ; }else {/* read the contents of the AFS special symlink */ -struct page *page; +DEFINE_DELAYED_CALL (cleanup); +const char *content;loff_t size =i_size_read (d_inode (mntpt)); -char *buf;if (src_as->cell) ctx->cell =afs_use_cell (src_as->cell, afs_cell_trace_use_mntpt);
@@ -128,16 +128,16 @@ static int afs_mntpt_set_params(struct fs_context *fc, struct dentry *mntpt)if (size <2 || size >PAGE_SIZE -1 )return -EINVAL ; - page =read_mapping_page (d_inode (mntpt)->i_mapping,0 ,NULL ); -if (IS_ERR (page)) -return PTR_ERR (page); + content =afs_get_link (mntpt,d_inode (mntpt), &cleanup); +if (IS_ERR (content)) { +do_delayed_call (&cleanup); +return PTR_ERR (content); + } - buf =kmap (page); ret = -EINVAL ; -if (buf[size -1 ] =='.' ) - ret =vfs_parse_fs_string (fc,"source" , buf, size -1 ); -kunmap (page); -put_page (page); +if (content[size -1 ] =='.' ) + ret =vfs_parse_fs_string (fc,"source" , content, size -1 ); +do_delayed_call (&cleanup);if (ret <0 )return ret;
include/trace/events/afs.h +1 -0
@@ -422,6 +422,7 @@ enum yfs_cm_operation { EM(afs_file_error_dir_over_end, "DIR_ENT_OVER_END") \ EM(afs_file_error_dir_small, "DIR_SMALL") \ EM(afs_file_error_dir_unmarked_ext, "DIR_UNMARKED_EXT") \ + EM(afs_file_error_symlink_big, "SYM_BIG") \ EM(afs_file_error_mntpt, "MNTPT_READ_FAILED") \E_ (afs_file_error_writeback_fail,"WRITEBACK_FAILED" )
David Howells
maintainerPATCH v3 25/33 afs: Eliminate afs_read
In reply to David HowellsNow that directory and symlink reads go through netfslib, the afs_read struct is mostly redundant with almost all data duplicated in the netfs_io_request and netfs_io_subrequest structs that are also available any time we're doing a fetch.
Eliminate afs_read by moving the one field we still need there to the afs_call struct (we may be given a different amount of data than what we asked for and have to track what remains of that) and using the netfs_io_subrequest directly instead.
--- fs/afs/file.c | 96 +++++++++------------------------------------- fs/afs/fsclient.c | 55 +++++++++++++------------- fs/afs/inode.c | 2 + fs/afs/internal.h | 35 ++--------------- fs/afs/yfsclient.c | 47 +++++++++++------------ fs/netfs/main.c | 2 +- 6 files changed, 72 insertions(+), 165 deletions(-)
fs/afs/file.c +18 -78
@@ -200,50 +200,12 @@ int afs_release(struct inode *inode, struct file *file)return ret; } -/* -* Allocate a new read record. -*/ -struct afs_read *afs_alloc_read (gfp_t gfp) -{ -struct afs_read *req; - - req =kzalloc (sizeof (struct afs_read ), gfp); -if (req) -refcount_set (&req->usage,1 ); - -return req; -} - -/* -* Dispose of a ref to a read record. -*/ -void afs_put_read (struct afs_read *req) -{ -if (refcount_dec_and_test (&req->usage)) { -if (req->cleanup) - req->cleanup (req); -key_put (req->key); -kfree (req); - } -} -static void afs_fetch_data_notify (struct afs_operation *op) { -struct afs_read *req = op->fetch.req; -struct netfs_io_subrequest *subreq = req->subreq; -int error =afs_op_error (op); - - req->error = error; -if (subreq) { - subreq->rreq->i_size = req->file_size; -if (req->pos + req->actual_len >= req->file_size) -__set_bit (NETFS_SREQ_HIT_EOF , &subreq->flags); - subreq->error = error; -netfs_read_subreq_terminated (subreq); - req->subreq =NULL ; - }else if (req->done) { - req->done (req); - } +struct netfs_io_subrequest *subreq = op->fetch.subreq; + + subreq->error =afs_op_error (op); +netfs_read_subreq_terminated (subreq); }static void afs_fetch_data_success (struct afs_operation *op)
@@ -253,7 +215,7 @@ static void afs_fetch_data_success(struct afs_operation *op)_enter ("op=%08x" , op->debug_id);afs_vnode_commit_status (op, &op->file[0 ]);afs_stat_v (vnode, n_fetches); -atomic_long_add (op->fetch.req->actual_len, &op->net->n_fetch_bytes); +atomic_long_add (op->fetch.subreq->transferred, &op->net->n_fetch_bytes);afs_fetch_data_notify (op); }
@@ -265,11 +227,10 @@ static void afs_fetch_data_aborted(struct afs_operation *op)static void afs_fetch_data_put (struct afs_operation *op) { - op->fetch.req->error =afs_op_error (op); -afs_put_read (op->fetch.req); + op->fetch.subreq->error =afs_op_error (op); } -static const struct afs_operation_ops afs_fetch_data_operation = { +const struct afs_operation_ops afs_fetch_data_operation = { .issue_afs_rpc = afs_fs_fetch_data, .issue_yfs_rpc = yfs_fs_fetch_data, .success = afs_fetch_data_success,
@@ -281,55 +242,34 @@ static const struct afs_operation_ops afs_fetch_data_operation = {/* * Fetch file data from the volume. */ -int afs_fetch_data (struct afs_vnode *vnode,struct afs_read *req) +static void afs_read_worker (struct work_struct *work) { + struct netfs_io_subrequest *subreq = container_of(work, struct netfs_io_subrequest, work);struct afs_operation *op; +struct afs_vnode *vnode =AFS_FS_I (subreq->rreq->inode); +struct key *key = subreq->rreq->netfs_priv;_enter ("%s{%llx:%llu.%u},%x,,," , vnode->volume->name, vnode->fid.vid, vnode->fid.vnode, vnode->fid.unique, -key_serial (req->key)); +key_serial (key)); - op =afs_alloc_operation (req->key, vnode->volume); + op =afs_alloc_operation (key, vnode->volume);if (IS_ERR (op)) { -if (req->subreq) { - req->subreq->error =PTR_ERR (op); -netfs_read_subreq_terminated (req->subreq); - } -return PTR_ERR (op); + subreq->error =PTR_ERR (op); +netfs_read_subreq_terminated (subreq); +return ; }afs_op_set_vnode (op,0 , vnode); - op->fetch.req =afs_get_read (req); + op->fetch.subreq = subreq; op->ops = &afs_fetch_data_operation; -return afs_do_sync_operation (op); -} - -static void afs_read_worker (struct work_struct *work) -{ - struct netfs_io_subrequest *subreq = container_of(work, struct netfs_io_subrequest, work); -struct afs_vnode *vnode =AFS_FS_I (subreq->rreq->inode); -struct afs_read *fsreq; - - fsreq =afs_alloc_read (GFP_NOFS ); -if (!fsreq) { - subreq->error = -ENOMEM ; -return netfs_read_subreq_terminated (subreq); - } - - fsreq->subreq = subreq; - fsreq->pos = subreq->start + subreq->transferred; - fsreq->len = subreq->len - subreq->transferred; - fsreq->key =key_get (subreq->rreq->netfs_priv); - fsreq->vnode = vnode; - fsreq->iter = &subreq->io_iter;trace_netfs_sreq (subreq, netfs_sreq_trace_submit); -afs_fetch_data (fsreq->vnode, fsreq); -afs_put_read (fsreq); +afs_do_sync_operation (op); }static void afs_issue_read (struct netfs_io_subrequest *subreq)
fs/afs/fsclient.c +26 -29
146 lines not on this page · raw message
fs/afs/inode.c +2 -0
@@ -317,6 +317,8 @@ static void afs_apply_status(struct afs_operation *op,inode_set_ctime_to_ts (inode, t);inode_set_atime_to_ts (inode, t); } +if (op->ops == &afs_fetch_data_operation) + op->fetch.subreq->rreq->i_size = status->size; } }
fs/afs/internal.h +3 -32
@@ -163,6 +163,7 @@ struct afs_call {spinlock_t state_lock;int error;/* error code */ u32 abort_code;/* Remote abort ID or 0 */ +unsigned long long remaining;/* How much is left to receive */ unsigned int max_lifespan;/* Maximum lifespan in secs to set if not 0 */ unsigned request_size;/* size of request data */ unsigned reply_max;/* maximum size of reply */
@@ -232,28 +233,6 @@ static inline struct key *afs_file_key(struct file *file)return af->key; } -/* -* Record of an outstanding read operation on a vnode. -*/ -struct afs_read { -loff_t pos;/* Where to start reading */ -loff_t len;/* How much we're asking for */ -loff_t actual_len;/* How much we're actually getting */ -loff_t file_size;/* File size returned by server */ -struct key *key;/* The key to use to reissue the read */ -struct afs_vnode *vnode;/* The file being read into. */ -struct netfs_io_subrequest *subreq;/* Fscache helper read request this belongs to */ -afs_dataversion_t data_version;/* Version number returned by server */ -refcount_t usage; -unsigned int call_debug_id; -unsigned int nr_pages; -int error; -void (*done)(struct afs_read *); -void (*cleanup)(struct afs_read *); -struct iov_iter *iter;/* Iterator representing the buffer */ -struct iov_iter def_iter;/* Default iterator */ -}; -/* * AFS superblock private data * - there's one superblock per volume
@@ -910,7 +889,7 @@ struct afs_operation {bool new_negative; } rename;struct { -struct afs_read *req; +struct netfs_io_subrequest *subreq; } fetch;struct {afs_lock_type_t type;
@@ -1117,21 +1096,13 @@ extern void afs_dynroot_depopulate(struct super_block *);extern const struct address_space_operations afs_file_aops;extern const struct inode_operations afs_file_inode_operations;extern const struct file_operations afs_file_operations; +extern const struct afs_operation_ops afs_fetch_data_operation;extern const struct netfs_request_ops afs_req_ops;extern int afs_cache_wb_key (struct afs_vnode *,struct afs_file *);extern void afs_put_wb_key (struct afs_wb_key *);extern int afs_open (struct inode *,struct file *);extern int afs_release (struct inode *,struct file *); -extern int afs_fetch_data (struct afs_vnode *,struct afs_read *); -extern struct afs_read *afs_alloc_read (gfp_t ); -extern void afs_put_read (struct afs_read *); - -static inline struct afs_read *afs_get_read (struct afs_read *req) -{ -refcount_inc (&req->usage); -return req; -}/* * flock.c
fs/afs/yfsclient.c +22 -25
120 lines not on this page · raw message
fs/netfs/main.c +1 -1
@@ -118,7 +118,7 @@ static int __init netfs_init(void) goto error_reqpool; netfs_subrequest_slab =kmem_cache_create ("netfs_subrequest" , -sizeof (struct netfs_io_subrequest ),0 , +sizeof (struct netfs_io_subrequest ) +16 ,0 ,SLAB_HWCACHE_ALIGN |SLAB_ACCOUNT ,NULL );if (!netfs_subrequest_slab)
David Howells
maintainerPATCH v3 26/33 afs: Fix cleanup of immediately failed async calls
In reply to David HowellsIf we manage to begin an async call, but fail to transmit any data on it due to a signal, we then abort it which causes a race between the notification of call completion from rxrpc and our attempt to cancel the notification. The notification will be necessary, however, for async FetchData to terminate the netfs subrequest.
However, since we get a notification from rxrpc upon completion of a call (aborted or otherwise), we can just leave it to that.
This leads to calls not getting cleaned up, but appearing in /proc/net/rxrpc/calls as being aborted with code 6.
Fix this by making the "error_do_abort:" case of afs_make_call() abort the call and then abandon it to the notification handler.
--- fs/afs/internal.h | 9 +++++++++ fs/afs/rxrpc.c | 12 +++++++++--- include/trace/events/afs.h | 2 ++ 3 files changed, 20 insertions(+), 3 deletions(-)
fs/afs/internal.h +9 -0
@@ -1335,6 +1335,15 @@ extern void afs_send_simple_reply(struct afs_call *, const void *, size_t);extern int afs_extract_data (struct afs_call *,bool );extern int afs_protocol_error (struct afs_call *,enum afs_eproto_cause ); +static inline void afs_see_call (struct afs_call *call,enum afs_call_trace why) +{ +int r =refcount_read (&call->ref); + +trace_afs_call (call->debug_id, why, r, +atomic_read (&call->net->nr_outstanding_calls), +__builtin_return_address (0 )); +} +static inline void afs_make_op_call (struct afs_operation *op,struct afs_call *call,gfp_t gfp) {
fs/afs/rxrpc.c +9 -3
@@ -430,11 +430,16 @@ void afs_make_call(struct afs_call *call, gfp_t gfp)return ; error_do_abort: -if (ret != -ECONNABORTED ) { +if (ret != -ECONNABORTED )rxrpc_kernel_abort_call (call->net->socket, rxcall,RX_USER_ABORT , ret, afs_abort_send_data_error); - }else { +if (call->async) { +afs_see_call (call, afs_call_trace_async_abort); +return ; + } + +if (ret == -ECONNABORTED ) { len =0 ;iov_iter_kvec (&msg.msg_iter,ITER_DEST ,NULL ,0 ,0 ); rxrpc_kernel_recv_data(call->net->socket, rxcall,
@@ -445,6 +450,8 @@ void afs_make_call(struct afs_call *call, gfp_t gfp) call->error = ret;trace_afs_call_done (call); error_kill_call: +if (call->async) +afs_see_call (call, afs_call_trace_async_kill);if (call->type->done) call->type->done (call);
@@ -602,7 +609,6 @@ static void afs_deliver_to_call(struct afs_call *call) abort_code =0 ; call_complete:afs_set_call_complete (call, ret, remote_abort); - state =AFS_CALL_COMPLETE ; goto done; }
include/trace/events/afs.h +2 -0
@@ -118,6 +118,8 @@ enum yfs_cm_operation {*/ #define afs_call_traces \ EM(afs_call_trace_alloc, "ALLOC") \ + EM(afs_call_trace_async_abort, "ASYAB") \ + EM(afs_call_trace_async_kill, "ASYKL") \ EM(afs_call_trace_free, "FREE ") \ EM(afs_call_trace_get, "GET ") \ EM(afs_call_trace_put, "PUT ") \
David Howells
maintainerPATCH v3 27/33 afs: Make {Y,}FS.FetchData an asynchronous operation
In reply to David HowellsMake FS.FetchData and YFS.FetchData an asynchronous operation in that the request is queued in AF_RXRPC and then we return to the caller rather than waiting. Processing of the returning packets is then done inline if it's a synchronous VFS/VM call (readdir, read_folio, sync DIO, prep for write) or offloaded to a workqueue if asynchronous VM calls (eg. readahead, async DIO).
This reduces the chain of workqueues invoking workqueues and cuts out some of the overhead, driving rxrpc data extraction and netfslib read collection from a thread that's going to block to completion anyway if possible.
The ->done() call op is also split with ->immediate_cancel() handling the cancellation on failure to begin the call and ->done() handling the rest. This means that the AFS async FetchData code doesn't try to terminate the netfs subrequest twice.
--- fs/afs/file.c | 126 +++++++++++++++++++++++++++++++++++++----- fs/afs/fs_operation.c | 2 +- fs/afs/fsclient.c | 9 ++- fs/afs/internal.h | 24 ++++++++ fs/afs/main.c | 2 +- fs/afs/rxrpc.c | 25 ++------- fs/afs/vlclient.c | 1 + fs/afs/write.c | 12 ++++ fs/afs/yfsclient.c | 6 +- 9 files changed, 170 insertions(+), 37 deletions(-)
fs/afs/file.c +112 -14
@@ -225,26 +225,111 @@ static void afs_fetch_data_aborted(struct afs_operation *op)afs_fetch_data_notify (op); } -static void afs_fetch_data_put (struct afs_operation *op) -{ - op->fetch.subreq->error =afs_op_error (op); -} -const struct afs_operation_ops afs_fetch_data_operation = { .issue_afs_rpc = afs_fs_fetch_data, .issue_yfs_rpc = yfs_fs_fetch_data, .success = afs_fetch_data_success, .aborted = afs_fetch_data_aborted, .failed = afs_fetch_data_notify, - .put = afs_fetch_data_put, }; +static void afs_issue_read_call (struct afs_operation *op) +{ + op->call_responded = false; + op->call_error =0 ; + op->call_abort_code =0 ; +if (test_bit (AFS_SERVER_FL_IS_YFS , &op->server->flags)) +yfs_fs_fetch_data (op); +else +afs_fs_fetch_data (op); +} + +static void afs_end_read (struct afs_operation *op) +{ +if (op->call_responded && op->server) +set_bit (AFS_SERVER_FL_RESPONDING , &op->server->flags); + +if (!afs_op_error (op)) +afs_fetch_data_success (op); +else if (op->cumul_error.aborted) +afs_fetch_data_aborted (op); +else +afs_fetch_data_notify (op); + +afs_end_vnode_operation (op); +afs_put_operation (op); +} + +/* +* Perform I/O processing on an asynchronous call. The work item carries a ref +* to the call struct that we either need to release or to pass on. +*/ +static void afs_read_receive (struct afs_call *call) +{ +struct afs_operation *op = call->op; +enum afs_call_state state; + +_enter ("" ); + + state =READ_ONCE (call->state); +if (state ==AFS_CALL_COMPLETE ) +return ; + +while (state <AFS_CALL_COMPLETE &&READ_ONCE (call->need_attention)) { +WRITE_ONCE (call->need_attention, false); +afs_deliver_to_call (call); + state =READ_ONCE (call->state); + } + +if (state <AFS_CALL_COMPLETE ) { +netfs_read_subreq_progress (op->fetch.subreq); +if (rxrpc_kernel_check_life (call->net->socket, call->rxcall)) +return ; +/* rxrpc terminated the call. */ +afs_set_call_complete (call, call->error, call->abort_code); + } + + op->call_abort_code = call->abort_code; + op->call_error = call->error; + op->call_responded = call->responded; + op->call =NULL ; + call->op =NULL ; +afs_put_call (call); + +/* If the call failed, then we need to crank the server rotation +* handle and try the next. +*/ +if (afs_select_fileserver (op)) { +afs_issue_read_call (op); +return ; + } + +afs_end_read (op); +} + +void afs_fetch_data_async_rx (struct work_struct *work) +{ + struct afs_call *call = container_of(work, struct afs_call, async_work); + +afs_read_receive (call); +afs_put_call (call); +} + +void afs_fetch_data_immediate_cancel (struct afs_call *call) +{ +if (call->async) { +afs_get_call (call, afs_call_trace_wake); +if (!queue_work (afs_async_calls, &call->async_work)) +afs_deferred_put_call (call); +flush_work (&call->async_work); + } +} +/* * Fetch file data from the volume. */ -static void afs_read_worker (struct work_struct *work) +static void afs_issue_read (struct netfs_io_subrequest *subreq) { - struct netfs_io_subrequest *subreq = container_of(work, struct netfs_io_subrequest, work);struct afs_operation *op;struct afs_vnode *vnode =AFS_FS_I (subreq->rreq->inode);struct key *key = subreq->rreq->netfs_priv;
@@ -269,13 +354,26 @@ static void afs_read_worker(struct work_struct *work) op->ops = &afs_fetch_data_operation;trace_netfs_sreq (subreq, netfs_sreq_trace_submit); -afs_do_sync_operation (op); -} -static void afs_issue_read (struct netfs_io_subrequest *subreq) -{ -INIT_WORK (&subreq->work, afs_read_worker); -queue_work (system_long_wq, &subreq->work); +if (subreq->rreq->origin ==NETFS_READAHEAD || + subreq->rreq->iocb) { + op->flags |=AFS_OPERATION_ASYNC ; + +if (!afs_begin_vnode_operation (op)) { + subreq->error =afs_put_operation (op); +netfs_read_subreq_terminated (subreq); +return ; + } + +if (!afs_select_fileserver (op)) { +afs_end_read (op); +return ; + } + +afs_issue_read_call (op); + }else { +afs_do_sync_operation (op); + } }static int afs_init_request (struct netfs_io_request *rreq,struct file *file)
fs/afs/fs_operation.c +1 -1
@@ -256,7 +256,7 @@ bool afs_begin_vnode_operation(struct afs_operation *op)/* * Tidy up a filesystem cursor and unlock the vnode. */ -static void afs_end_vnode_operation (struct afs_operation *op) +void afs_end_vnode_operation (struct afs_operation *op) {_enter ("" );
fs/afs/fsclient.c +8 -1
@@ -352,7 +352,6 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call) ret =afs_extract_data (call, true); subreq->transferred += count_before - call->iov_len; call->remaining -= count_before - call->iov_len; -netfs_read_subreq_progress (subreq);if (ret <0 )return ret;
@@ -409,14 +408,18 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)static const struct afs_call_type afs_RXFSFetchData = { .name ="FS.FetchData" , .op = afs_FS_FetchData, + .async_rx = afs_fetch_data_async_rx, .deliver = afs_deliver_fs_fetch_data, + .immediate_cancel = afs_fetch_data_immediate_cancel, .destructor = afs_flat_call_destructor, };static const struct afs_call_type afs_RXFSFetchData64 = { .name ="FS.FetchData64" , .op = afs_FS_FetchData64, + .async_rx = afs_fetch_data_async_rx, .deliver = afs_deliver_fs_fetch_data, + .immediate_cancel = afs_fetch_data_immediate_cancel, .destructor = afs_flat_call_destructor, };
@@ -436,6 +439,9 @@ static void afs_fs_fetch_data64(struct afs_operation *op)if (!call)return afs_op_nomem (op); +if (op->flags &AFS_OPERATION_ASYNC ) + call->async = true; +/* marshall the parameters */ bp = call->request; bp[0 ] =htonl (FSFETCHDATA64 );
@@ -1730,6 +1736,7 @@ static const struct afs_call_type afs_RXFSGetCapabilities = { .op = afs_FS_GetCapabilities, .deliver = afs_deliver_fs_get_capabilities, .done = afs_fileserver_probe_result, + .immediate_cancel = afs_fileserver_probe_result, .destructor = afs_fs_get_capabilities_destructor, };
fs/afs/internal.h +24 -0
@@ -202,11 +202,17 @@ struct afs_call_type {/* clean up a call */ void (*destructor)(struct afs_call *call); +/* Async receive processing function */ +void (*async_rx)(struct work_struct *work); +/* Work function */ void (*work)(struct work_struct *work);/* Call done function (gets called immediately on success or failure) */ void (*done)(struct afs_call *call); + +/* Handle a call being immediately cancelled. */ +void (*immediate_cancel)(struct afs_call *call); };/*
@@ -941,6 +947,7 @@ struct afs_operation {#define AFS_OPERATION_TRIED_ALL 0x0400/* Set if we've tried all the fileservers */ #define AFS_OPERATION_RETRY_SERVER 0x0800/* Set if we should retry the current server */ #define AFS_OPERATION_DIR_CONFLICT 0x1000/* Set if we detected a 3rd-party dir change */ +#define AFS_OPERATION_ASYNC 0x2000/* Set if should run asynchronously */ };/*
@@ -1103,6 +1110,8 @@ extern int afs_cache_wb_key(struct afs_vnode *, struct afs_file *);extern void afs_put_wb_key (struct afs_wb_key *);extern int afs_open (struct inode *,struct file *);extern int afs_release (struct inode *,struct file *); +void afs_fetch_data_async_rx (struct work_struct *work); +void afs_fetch_data_immediate_cancel (struct afs_call *call);/* * flock.c
@@ -1154,6 +1163,7 @@ extern void afs_fs_store_acl(struct afs_operation *);extern struct afs_operation *afs_alloc_operation (struct key *,struct afs_volume *);extern int afs_put_operation (struct afs_operation *);extern bool afs_begin_vnode_operation (struct afs_operation *); +extern void afs_end_vnode_operation (struct afs_operation *op);extern void afs_wait_for_operation (struct afs_operation *);extern int afs_do_sync_operation (struct afs_operation *);
@@ -1325,6 +1335,7 @@ extern void afs_charge_preallocation(struct work_struct *);extern void afs_put_call (struct afs_call *);void afs_deferred_put_call (struct afs_call *call);void afs_make_call (struct afs_call *call,gfp_t gfp); +void afs_deliver_to_call (struct afs_call *call);void afs_wait_for_call_to_complete (struct afs_call *call); extern struct afs_call *afs_alloc_flat_call(struct afs_net *, const struct afs_call_type *,
@@ -1335,6 +1346,19 @@ extern void afs_send_simple_reply(struct afs_call *, const void *, size_t);extern int afs_extract_data (struct afs_call *,bool );extern int afs_protocol_error (struct afs_call *,enum afs_eproto_cause ); +static inline struct afs_call *afs_get_call (struct afs_call *call, +enum afs_call_trace why) +{ +int r; + +__refcount_inc (&call->ref, &r); + +trace_afs_call (call->debug_id, why, r +1 , +atomic_read (&call->net->nr_outstanding_calls), +__builtin_return_address (0 )); +return call; +} +static inline void afs_see_call (struct afs_call *call,enum afs_call_trace why) {int r =refcount_read (&call->ref);
fs/afs/main.c +1 -1
@@ -177,7 +177,7 @@ static int __init afs_init(void) afs_wq =alloc_workqueue ("afs" ,0 ,0 );if (!afs_wq) goto error_afs_wq; - afs_async_calls =alloc_workqueue ("kafsd" ,WQ_MEM_RECLAIM ,0 ); + afs_async_calls =alloc_workqueue ("kafsd" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!afs_async_calls) goto error_async; afs_lock_manager =alloc_workqueue ("kafs_lockd" ,WQ_MEM_RECLAIM ,0 );
fs/afs/rxrpc.c +6 -19
@@ -149,7 +149,8 @@ static struct afs_call *afs_alloc_call(struct afs_net *net, call->net = net; call->debug_id =atomic_inc_return (&rxrpc_debug_id);refcount_set (&call->ref,1 ); -INIT_WORK (&call->async_work, afs_process_async_call); +INIT_WORK (&call->async_work, type->async_rx ?: afs_process_async_call); +INIT_WORK (&call->work, call->type->work);INIT_WORK (&call->free_work, afs_deferred_free_worker);init_waitqueue_head (&call->waitq);spin_lock_init (&call->state_lock);
@@ -235,27 +236,12 @@ void afs_deferred_put_call(struct afs_call *call)schedule_work (&call->free_work); } -static struct afs_call *afs_get_call (struct afs_call *call, -enum afs_call_trace why) -{ -int r; - -__refcount_inc (&call->ref, &r); - -trace_afs_call (call->debug_id, why, r +1 , -atomic_read (&call->net->nr_outstanding_calls), -__builtin_return_address (0 )); -return call; -} -/* * Queue the call for actual work. */ static void afs_queue_call_work (struct afs_call *call) {if (call->type->work) { -INIT_WORK (&call->work, call->type->work); -afs_get_call (call, afs_call_trace_work);if (!queue_work (afs_wq, &call->work))afs_put_call (call);
@@ -452,8 +438,8 @@ void afs_make_call(struct afs_call *call, gfp_t gfp) error_kill_call:if (call->async)afs_see_call (call, afs_call_trace_async_kill); -if (call->type->done) - call->type->done (call); +if (call->type->immediate_cancel) + call->type->immediate_cancel (call);/* We need to dispose of the extra ref we grabbed for an async call. * The call, however, might be queued on afs_async_calls and we need to
@@ -508,7 +494,7 @@ static void afs_log_error(struct afs_call *call, s32 remote_abort)/* * deliver messages to a call */ -static void afs_deliver_to_call (struct afs_call *call) +void afs_deliver_to_call (struct afs_call *call) {enum afs_call_state state;size_t len;
@@ -809,6 +795,7 @@ static int afs_deliver_cm_op_id(struct afs_call *call)return -ENOTSUPP ;trace_afs_cb_call (call); + call->work.func = call->type->work;/* pass responsibility for the remainer of this message off to the * cache manager op */
fs/afs/vlclient.c +1 -0
@@ -370,6 +370,7 @@ static const struct afs_call_type afs_RXVLGetCapabilities = { .name = "VL.GetCapabilities", .op = afs_VL_GetCapabilities, .deliver = afs_deliver_vl_get_capabilities, + .immediate_cancel = afs_vlserver_probe_result, .done = afs_vlserver_probe_result, .destructor = afs_destroy_vl_get_capabilities, };
fs/afs/write.c +12 -0
@@ -193,6 +193,18 @@ void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stlist_first_entry (&stream->subrequests, struct netfs_io_subrequest, rreq_link); +switch (wreq->origin) { +case NETFS_READAHEAD : +case NETFS_READPAGE : +case NETFS_READ_GAPS : +case NETFS_READ_SINGLE : +case NETFS_READ_FOR_WRITE : +case NETFS_DIO_READ : +return ; +default : +break ; + } +switch (subreq->error) {case -EACCES :case -EPERM :
fs/afs/yfsclient.c +5 -1
@@ -397,7 +397,6 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call) ret =afs_extract_data (call, true); subreq->transferred += count_before - call->iov_len; -netfs_read_subreq_progress (subreq);if (ret <0 )return ret;
@@ -457,7 +456,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)static const struct afs_call_type yfs_RXYFSFetchData64 = { .name ="YFS.FetchData64" , .op = yfs_FS_FetchData64, + .async_rx = afs_fetch_data_async_rx, .deliver = yfs_deliver_fs_fetch_data64, + .immediate_cancel = afs_fetch_data_immediate_cancel, .destructor = afs_flat_call_destructor, };
@@ -486,6 +487,9 @@ void yfs_fs_fetch_data(struct afs_operation *op)if (!call)return afs_op_nomem (op); +if (op->flags &AFS_OPERATION_ASYNC ) + call->async = true; +/* marshall the parameters */ bp = call->request; bp =xdr_encode_u32 (bp,YFSFETCHDATA64 );
David Howells
maintainerPATCH v3 28/33 netfs: Change the read result collector to only use one work item
In reply to David HowellsChange the way netfslib collects read results to do all the collection for a particular read request using a single work item that walks along the subrequest queue as subrequests make progress or complete, unlocking folios progressively rather than doing the unlock in parallel as parallel requests come in.
The code is remodelled to be more like the write-side code, though only using a single stream. This makes it more directly comparable and thus easier to duplicate fixes between the two sides.
This has a number of advantages:
(1) It's simpler. There doesn't need to be a complex donation mechanism to handle mismatches between the size and alignment of subrequests and folios. The collector unlocks folios as the subrequests covering each complete.
(2) It should cause less scheduler overhead as there's a single work item in play unlocking pages in parallel when a read gets split up into a lot of subrequests instead of one per subrequest.
Whilst the parallellism is nice in theory, in practice, the vast majority of loads are sequential reads of the whole file, so committing a bunch of threads to unlocking folios out of order doesn't help in those cases.
(3) It should make it easier to implement content decryption. A folio cannot be decrypted until all the requests that contribute to it have completed - and, again, most loads are sequential and so, most of the time, we want to begin decryption sequentially (though it's great if the decryption can happen in parallel).
There is a disadvantage in that we're losing the ability to decrypt and unlock things on an as-things-arrive basis which may affect some applications.
--- fs/9p/vfs_addr.c | 5 +- fs/afs/dir.c | 8 +- fs/netfs/buffered_read.c | 138 ++++---- fs/netfs/direct_read.c | 62 ++-- fs/netfs/internal.h | 16 +- fs/netfs/main.c | 2 +- fs/netfs/objects.c | 34 +- fs/netfs/read_collect.c | 658 ++++++++++++++++++++--------------- fs/netfs/read_pgpriv2.c | 3 +- fs/netfs/read_retry.c | 207 ++++++----- fs/netfs/read_single.c | 37 +- fs/netfs/write_retry.c | 14 +- fs/smb/client/cifssmb.c | 2 + fs/smb/client/smb2pdu.c | 4 +- include/linux/netfs.h | 17 +- include/trace/events/netfs.h | 75 +--- 16 files changed, 664 insertions(+), 618 deletions(-)
fs/9p/vfs_addr.c +3 -2
@@ -79,9 +79,10 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags);if (pos + total >=i_size_read (rreq->inode))__set_bit (NETFS_SREQ_HIT_EOF , &subreq->flags); - -if (!err) +if (!err && total) { +__set_bit (NETFS_SREQ_MADE_PROGRESS , &subreq->flags); subreq->transferred += total; + } subreq->error = err;netfs_read_subreq_terminated (subreq);
fs/afs/dir.c +6 -2
@@ -325,8 +325,10 @@ static ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)* haven't read it yet. */ if (test_bit (AFS_VNODE_DIR_VALID , &dvnode->flags) && - dvnode->directory) + dvnode->directory) { + ret = i_size; goto valid; + }up_read (&dvnode->validate_lock);if (down_write_killable (&dvnode->validate_lock) <0 )
@@ -345,11 +347,13 @@ static ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)// TODO: Trim excess pages set_bit (AFS_VNODE_DIR_VALID , &dvnode->flags); + }else { + ret = i_size; }downgrade_write (&dvnode->validate_lock); valid: -return i_size; +return ret; error_unlock:up_write (&dvnode->validate_lock);
fs/netfs/buffered_read.c +66 -72
224 lines not on this page · raw message
fs/netfs/direct_read.c +36 -26
115 lines not on this page · raw message
fs/netfs/internal.h +12 -4
@@ -82,17 +82,25 @@ static inline void netfs_see_request(struct netfs_io_request *rreq,trace_netfs_rreq_ref (rreq->debug_id,refcount_read (&rreq->ref), what); } +static inline void netfs_see_subrequest (struct netfs_io_subrequest *subreq, +enum netfs_sreq_ref_trace what) +{ +trace_netfs_sreq_ref (subreq->rreq->debug_id, subreq->debug_index, +refcount_read (&subreq->ref), what); +} +/* * read_collect.c */ -void netfs_read_termination_worker (struct work_struct *work); -void netfs_rreq_terminated (struct netfs_io_request *rreq); +void netfs_read_collection_worker (struct work_struct *work); +void netfs_wake_read_collector (struct netfs_io_request *rreq); +void netfs_cache_read_terminated (void *priv,ssize_t transferred_or_error,bool was_async); +ssize_t netfs_wait_for_read (struct netfs_io_request *rreq);/* * read_pgpriv2.c */ -void netfs_pgpriv2_mark_copy_to_cache (struct netfs_io_subrequest *subreq, -struct netfs_io_request *rreq, +void netfs_pgpriv2_mark_copy_to_cache (struct netfs_io_request *rreq,struct folio_queue *folioq,int slot);void netfs_pgpriv2_write_to_the_cache (struct netfs_io_request *rreq);
fs/netfs/main.c +1 -1
@@ -71,7 +71,7 @@ static int netfs_requests_seq_show(struct seq_file *m, void *v)refcount_read (&rreq->ref), rreq->flags, rreq->error, -atomic_read (&rreq->nr_outstanding), +0 , rreq->start, rreq->submitted, rreq->len);seq_putc (m,'\n' );return 0 ;
fs/netfs/objects.c +7 -27
@@ -48,7 +48,7 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,spin_lock_init (&rreq->lock);INIT_LIST_HEAD (&rreq->io_streams[0 ].subrequests);INIT_LIST_HEAD (&rreq->io_streams[1 ].subrequests); -INIT_LIST_HEAD (&rreq->subrequests); +init_waitqueue_head (&rreq->waitq);refcount_set (&rreq->ref,1 ); if (origin == NETFS_READAHEAD ||
@@ -56,10 +56,12 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping, origin == NETFS_READ_GAPS || origin == NETFS_READ_SINGLE || origin == NETFS_READ_FOR_WRITE || - origin == NETFS_DIO_READ) -INIT_WORK (&rreq->work,NULL ); - else + origin == NETFS_DIO_READ) { + INIT_WORK(&rreq->work, netfs_read_collection_worker); + rreq->io_streams[0].avail = true; + } else {INIT_WORK (&rreq->work, netfs_write_collection_worker); + }__set_bit (NETFS_RREQ_IN_PROGRESS , &rreq->flags);if (file && file->f_flags &O_NONBLOCK )
@@ -93,14 +95,6 @@ void netfs_clear_subrequests(struct netfs_io_request *rreq, bool was_async)struct netfs_io_stream *stream;int s; -while (!list_empty (&rreq->subrequests)) { - subreq =list_first_entry (&rreq->subrequests, - struct netfs_io_subrequest, rreq_link); -list_del (&subreq->rreq_link); -netfs_put_subrequest (subreq, was_async, - netfs_sreq_trace_put_clear); - } -for (s =0 ; s <ARRAY_SIZE (rreq->io_streams); s++) { stream = &rreq->io_streams[s];while (!list_empty (&stream->subrequests)) {
@@ -192,21 +186,7 @@ struct netfs_io_subrequest *netfs_alloc_subrequest(struct netfs_io_request *rreq }memset (subreq,0 ,kmem_cache_size (cache)); - -switch (rreq->origin) { -case NETFS_READAHEAD : -case NETFS_READPAGE : -case NETFS_READ_GAPS : -case NETFS_READ_SINGLE : -case NETFS_READ_FOR_WRITE : -case NETFS_DIO_READ : -INIT_WORK (&subreq->work, netfs_read_subreq_termination_worker); -break ; -default : -INIT_WORK (&subreq->work,NULL ); -break ; - } - +INIT_WORK (&subreq->work,NULL );INIT_LIST_HEAD (&subreq->rreq_link);refcount_set (&subreq->ref,2 ); subreq->rreq = rreq;
fs/netfs/read_collect.c +368 -290
811 lines not on this page · raw message
fs/netfs/read_pgpriv2.c +1 -2
@@ -18,8 +18,7 @@* third mark in the folio queue is used to indicate that this folio needs * writing. */ -void netfs_pgpriv2_mark_copy_to_cache (struct netfs_io_subrequest *subreq, -struct netfs_io_request *rreq, +void netfs_pgpriv2_mark_copy_to_cache (struct netfs_io_request *rreq,struct folio_queue *folioq,int slot) {
fs/netfs/read_retry.c +120 -87
311 lines not on this page · raw message
fs/netfs/read_single.c +15 -22
@@ -77,6 +77,7 @@ static void netfs_single_read_cache(struct netfs_io_request *rreq, {struct netfs_cache_resources *cres = &rreq->cache_resources; +_enter ("R=%08x[%x]" , rreq->debug_id, subreq->debug_index);netfs_stat (&netfs_n_rh_read); cres->ops->read (cres, subreq->start, &subreq->io_iter,NETFS_READ_HOLE_FAIL , netfs_cache_read_terminated, subreq);
@@ -88,28 +89,28 @@ static void netfs_single_read_cache(struct netfs_io_request *rreq,*/ static int netfs_single_dispatch_read (struct netfs_io_request *rreq) { +struct netfs_io_stream *stream = &rreq->io_streams[0 ];struct netfs_io_subrequest *subreq;int ret =0 ; -atomic_set (&rreq->nr_outstanding,1 ); - subreq =netfs_alloc_subrequest (rreq); -if (!subreq) { - ret = -ENOMEM ; - goto out; - } +if (!subreq) +return -ENOMEM ; subreq->source =NETFS_DOWNLOAD_FROM_SERVER ; subreq->start =0 ; subreq->len = rreq->len; subreq->io_iter = rreq->buffer.iter; -atomic_inc (&rreq->nr_outstanding); +__set_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags); -spin_lock_bh (&rreq->lock); -list_add_tail (&subreq->rreq_link, &rreq->subrequests); +spin_lock (&rreq->lock); +list_add_tail (&subreq->rreq_link, &stream->subrequests);trace_netfs_sreq (subreq, netfs_sreq_trace_added); -spin_unlock_bh (&rreq->lock); + stream->front = subreq; +/* Store list pointers before active flag */ +smp_store_release (&stream->active, true); +spin_unlock (&rreq->lock);netfs_single_cache_prepare_read (rreq, subreq);switch (subreq->source) {
@@ -137,14 +138,12 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)break ; } -out: -if (atomic_dec_and_test (&rreq->nr_outstanding)) -netfs_rreq_terminated (rreq); +smp_wmb ();/* Write lists before ALL_QUEUED. */ +set_bit (NETFS_RREQ_ALL_QUEUED , &rreq->flags);return ret; cancel: -atomic_dec (&rreq->nr_outstanding);netfs_put_subrequest (subreq, false, netfs_sreq_trace_put_cancel); - goto out; +return ret; }/**
@@ -185,13 +184,7 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite rreq->buffer.iter = *iter;netfs_single_dispatch_read (rreq); -trace_netfs_rreq (rreq, netfs_rreq_trace_wait_ip); -wait_on_bit (&rreq->flags,NETFS_RREQ_IN_PROGRESS , -TASK_UNINTERRUPTIBLE ); - - ret = rreq->error; -if (ret ==0 ) - ret = rreq->transferred; + ret =netfs_wait_for_read (rreq);netfs_put_request (rreq, true, netfs_rreq_trace_put_return);return ret;
fs/netfs/write_retry.c +10 -4
@@ -94,15 +94,21 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,list_for_each_entry_from (subreq, &stream->subrequests, rreq_link) {if (!len)break ; -/* Renegotiate max_len (wsize) */ -trace_netfs_sreq (subreq, netfs_sreq_trace_retry); + + subreq->start = start; + subreq->len = len;__clear_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags);__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); +trace_netfs_sreq (subreq, netfs_sreq_trace_retry); + +/* Renegotiate max_len (wsize) */ + stream->sreq_max_len = len; stream->prepare_write (subreq); - part =min (len, stream->sreq_max_len); + part =umin (len, stream->sreq_max_len); +if (unlikely (stream->sreq_max_segs)) + part =netfs_limit_iter (&source,0 , part, stream->sreq_max_segs); subreq->len = part; - subreq->start = start; subreq->transferred =0 ; len -= part; start += part;
fs/smb/client/cifssmb.c +2 -0
@@ -1323,6 +1323,8 @@ cifs_readv_callback(struct mid_q_entry *mid)__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); rdata->result =0 ; } +if (rdata->got_bytes) +__set_bit (NETFS_SREQ_MADE_PROGRESS , &rdata->subreq.flags); } rdata->credits.value =0 ;
fs/smb/client/smb2pdu.c +3 -1
@@ -4620,6 +4620,8 @@ smb2_readv_callback(struct mid_q_entry *mid)__set_bit (NETFS_SREQ_HIT_EOF , &rdata->subreq.flags); rdata->result =0 ; } +if (rdata->got_bytes) +__set_bit (NETFS_SREQ_MADE_PROGRESS , &rdata->subreq.flags); } trace_smb3_rw_credits(rreq_debug_id, subreq_debug_index, rdata->credits.value, server->credits, server->in_flight,
@@ -4628,7 +4630,7 @@ smb2_readv_callback(struct mid_q_entry *mid) rdata->subreq.error = rdata->result; rdata->subreq.transferred += rdata->got_bytes;trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_progress); -queue_work (cifsiod_wq, &rdata->subreq.work); +netfs_read_subreq_terminated (&rdata->subreq);release_mid (mid); trace_smb3_rw_credits(rreq_debug_id, subreq_debug_index, 0, server->credits, server->in_flight,
include/linux/netfs.h +6 -11
@@ -181,23 +181,17 @@ struct netfs_io_subrequest {unsigned long long start;/* Where to start the I/O */ size_t len;/* Size of the I/O */ size_t transferred;/* Amount of data transferred */ -size_t consumed;/* Amount of read data consumed */ -size_t prev_donated;/* Amount of data donated from previous subreq */ -size_t next_donated;/* Amount of data donated from next subreq */ refcount_t ref;short error;/* 0 or error that occurred */ unsigned short debug_index;/* Index in list (for debugging output) */ unsigned int nr_segs;/* Number of segs in io_iter */ enum netfs_io_source source;/* Where to read from/write to */ unsigned char stream_nr;/* I/O stream this belongs to */ -unsigned char curr_folioq_slot;/* Folio currently being read */ -unsigned char curr_folio_order;/* Order of folio */ -struct folio_queue *curr_folioq;/* Queue segment in which current folio resides */ unsigned long flags;#define NETFS_SREQ_COPY_TO_CACHE 0/* Set if should copy the data to the cache */ #define NETFS_SREQ_CLEAR_TAIL 1/* Set if the rest of the read should be cleared */ #define NETFS_SREQ_SEEK_DATA_READ 3/* Set if ->read() should SEEK_DATA first */ -#define NETFS_SREQ_NO_PROGRESS 4/* Set if we didn't manage to read any data */ +#define NETFS_SREQ_MADE_PROGRESS 4/* Set if we managed to read more data */ #define NETFS_SREQ_ONDEMAND 5/* Set if it's from on-demand read mode */ #define NETFS_SREQ_BOUNDARY 6/* Set if ends on hard boundary (eg. ceph object) */ #define NETFS_SREQ_HIT_EOF 7/* Set if short due to EOF */
@@ -238,13 +232,13 @@ struct netfs_io_request {struct netfs_cache_resources cache_resources;struct readahead_control *ractl;/* Readahead descriptor */ struct list_head proc_link;/* Link in netfs_iorequests */ -struct list_head subrequests;/* Contributory I/O operations */ struct netfs_io_stream io_streams[2 ];/* Streams of parallel I/O operations */ #define NR_IO_STREAMS 2 //wreq->nr_io_streamsstruct netfs_group *group;/* Writeback group being written back */ struct rolling_buffer buffer;/* Unencrypted buffer */ #define NETFS_ROLLBUF_PUT_MARK ROLLBUF_MARK_1#define NETFS_ROLLBUF_PAGECACHE_MARK ROLLBUF_MARK_2 +wait_queue_head_t waitq;/* Processor waiter */ void *netfs_priv;/* Private data for the netfs */ void *netfs_priv2;/* Private data for the netfs */ struct bio_vec *direct_bv;/* DIO buffer list (when handling iovec-iter) */
@@ -255,7 +249,6 @@ struct netfs_io_request {atomic_t subreq_counter;/* Next subreq->debug_index */ unsigned int nr_group_rel;/* Number of refs to release on ->group */ spinlock_t lock;/* Lock for queuing subreqs */ -atomic_t nr_outstanding;/* Number of ops in progress */ unsigned long long submitted;/* Amount submitted for I/O so far */ unsigned long long len;/* Length of the request */ size_t transferred;/* Amount to be indicated as transferred */
@@ -268,14 +261,17 @@ struct netfs_io_request {unsigned long long collected_to;/* Point we've collected to */ unsigned long long cleaned_to;/* Position we've cleaned folios to */ pgoff_t no_unlock_folio;/* Don't unlock this folio after read */ -size_t prev_donated;/* Fallback for subreq->prev_donated */ +unsigned char front_folio_order;/* Order (size) of front folio */ refcount_t ref;unsigned long flags; +#define NETFS_RREQ_OFFLOAD_COLLECTION 0/* Offload collection to workqueue */ #define NETFS_RREQ_COPY_TO_CACHE 1/* Need to write to the cache */ #define NETFS_RREQ_NO_UNLOCK_FOLIO 2/* Don't unlock no_unlock_folio on completion */ #define NETFS_RREQ_DONT_UNLOCK_FOLIOS 3/* Don't unlock the folios on completion */ #define NETFS_RREQ_FAILED 4/* The request failed */ #define NETFS_RREQ_IN_PROGRESS 5/* Unlocked when the request completes */ +#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 6/* Copy current folio to cache from read */ +#define NETFS_RREQ_FOLIO_ABANDON 7/* Abandon failed folio from read */ #define NETFS_RREQ_UPLOAD_TO_SERVER 8/* Need to write to the server */ #define NETFS_RREQ_NONBLOCK 9/* Don't block if possible (O_NONBLOCK) */ #define NETFS_RREQ_BLOCKED 10/* We blocked */
@@ -440,7 +436,6 @@ vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_gr/* (Sub)request management API. */ void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq);void netfs_read_subreq_terminated (struct netfs_io_subrequest *subreq); -void netfs_read_subreq_termination_worker (struct work_struct *work);void netfs_get_subrequest (struct netfs_io_subrequest *subreq,enum netfs_sreq_ref_trace what); void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
include/trace/events/netfs.h +8 -67
118 lines not on this page · raw message
David Howells
maintainerRe: [PATCH v3 28/33] netfs: Change the read result collector to only use one work item
This patch needs the attached adjustment folding in.
David
---
commit 2c0cccc7b29a051fadb6816d31f526e4dd45ddf5
Author: David Howells <[email protected]> Date: Thu Nov 7 22:22:49 2024 +0000
netfs: Fix folio abandonment
The mechanism to handle the abandonment of a folio being read due to an error occurring in a subrequest (such as if a signal occurs) will correctly abandon folios if they're entirely processed in one go; but if the constituent subrequests aren't processed in the same scheduled work item, the note that the first one failed will get lost if no folios are processed (the ABANDON_SREQ note isn't transferred to the NETFS_RREQ_FOLIO_ABANDON flag unless we process a folio).
Fix this by simplifying the way the mechanism works. Replace the flag with a variable that records the file position to which we should abandon folios. Any folio that overlaps this region at the time of unlocking must be abandoned (and reread).
This works because subrequests are processed in order of file position and each folio is processed as soon as enough subrequest transference is available to cover it - so when the abandonment point is moved, it covers only folios that draw data from the dodgy region.
Also make sure that NETFS_SREQ_FAILED is set on failure and that stream->failed is set when we cut the stream short.
Signed-off: David Howells <[email protected]>
cc: Jeff Layton <[email protected]>
cc: [email protected]
cc: [email protected]
fs/netfs/read_collect.c +9 -4
@@ -46,7 +46,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,struct netfs_folio *finfo;struct folio *folio =folioq_folio (folioq, slot); -if (unlikely (test_bit (NETFS_RREQ_FOLIO_ABANDON , &rreq->flags))) { +if (unlikely (folio_pos (folio) < rreq->abandon_to)) {trace_netfs_folio (folio, netfs_folio_trace_abandon); goto just_unlock; }
@@ -126,8 +126,6 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,if (*notes ©_TO_CACHE )set_bit (NETFS_RREQ_FOLIO_COPY_TO_CACHE , &rreq->flags); -if (*notes &ABANDON_SREQ ) -set_bit (NETFS_RREQ_FOLIO_ABANDON , &rreq->flags); folio =folioq_folio (folioq, slot); if (WARN_ONCE(!folio_test_locked(folio),
@@ -152,7 +150,6 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq, *notes |=MADE_PROGRESS ;clear_bit (NETFS_RREQ_FOLIO_COPY_TO_CACHE , &rreq->flags); -clear_bit (NETFS_RREQ_FOLIO_ABANDON , &rreq->flags);/* Clean up the head folioq. If we clear an entire folioq, then * we can get rid of it provided it's not also the tail folioq
@@ -251,6 +248,12 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)if (test_bit (NETFS_SREQ_COPY_TO_CACHE , &front->flags)) notes |=COPY_TO_CACHE ; +if (test_bit (NETFS_SREQ_FAILED , &front->flags)) { + rreq->abandon_to = front->start + front->len; + front->transferred = front->len; + transferred = front->len; +trace_netfs_rreq (rreq, netfs_rreq_trace_set_abandon); + }if (front->start + transferred >= rreq->cleaned_to + fsize ||test_bit (NETFS_SREQ_HIT_EOF , &front->flags))netfs_read_unlock_folios (rreq, ¬es);
@@ -268,6 +271,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq) stream->error = front->error; rreq->error = front->error;set_bit (NETFS_RREQ_FAILED , &rreq->flags); + stream->failed = true; } notes |=MADE_PROGRESS |ABANDON_SREQ ; }else if (test_bit (NETFS_SREQ_NEED_RETRY , &front->flags)) {
@@ -566,6 +570,7 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)__set_bit (NETFS_SREQ_NEED_RETRY , &subreq->flags); }else {netfs_stat (&netfs_n_rh_download_failed); +__set_bit (NETFS_SREQ_FAILED , &subreq->flags); }trace_netfs_rreq (rreq, netfs_rreq_trace_set_pause);set_bit (NETFS_RREQ_PAUSE , &rreq->flags);
include/linux/netfs.h +1 -1
@@ -260,6 +260,7 @@ struct netfs_io_request {atomic64_t issued_to;/* Write issuer folio cursor */ unsigned long long collected_to;/* Point we've collected to */ unsigned long long cleaned_to;/* Position we've cleaned folios to */ +unsigned long long abandon_to;/* Position to abandon folios to */ pgoff_t no_unlock_folio;/* Don't unlock this folio after read */ unsigned char front_folio_order;/* Order (size) of front folio */ refcount_t ref;
@@ -271,7 +272,6 @@ struct netfs_io_request {#define NETFS_RREQ_FAILED 4/* The request failed */ #define NETFS_RREQ_IN_PROGRESS 5/* Unlocked when the request completes */ #define NETFS_RREQ_FOLIO_COPY_TO_CACHE 6/* Copy current folio to cache from read */ -#define NETFS_RREQ_FOLIO_ABANDON 7/* Abandon failed folio from read */ #define NETFS_RREQ_UPLOAD_TO_SERVER 8/* Need to write to the server */ #define NETFS_RREQ_NONBLOCK 9/* Don't block if possible (O_NONBLOCK) */ #define NETFS_RREQ_BLOCKED 10/* We blocked */
include/trace/events/netfs.h +1 -0
@@ -56,6 +56,7 @@ EM(netfs_rreq_trace_free, "FREE ") \ EM(netfs_rreq_trace_redirty, "REDIRTY") \ EM(netfs_rreq_trace_resubmit, "RESUBMT") \ + EM(netfs_rreq_trace_set_abandon, "S-ABNDN") \ EM(netfs_rreq_trace_set_pause, "PAUSE ") \ EM(netfs_rreq_trace_unlock, "UNLOCK ") \EM (netfs_rreq_trace_unlock_pgpriv2,"UNLCK-2" ) \
David Howells
maintainerPATCH v3 29/33 afs: Make afs_mkdir() locally initialise a new directory's content
In reply to David HowellsInitialise a new directory's content when it is created by mkdir locally rather than downloading the content from the server as we can predict what it's going to look like.
--- fs/afs/dir.c | 3 +++ fs/afs/dir_edit.c | 48 ++++++++++++++++++++++++++++++++++++++ fs/afs/internal.h | 1 + include/trace/events/afs.h | 2 ++ 4 files changed, 54 insertions(+)
fs/afs/dir.c +3 -0
@@ -1258,6 +1258,7 @@ void afs_check_for_remote_deletion(struct afs_operation *op)*/ static void afs_vnode_new_inode (struct afs_operation *op) { +struct afs_vnode_param *dvp = &op->file[0 ];struct afs_vnode_param *vp = &op->file[1 ];struct afs_vnode *vnode;struct inode *inode;
@@ -1277,6 +1278,8 @@ static void afs_vnode_new_inode(struct afs_operation *op) vnode =AFS_FS_I (inode);set_bit (AFS_VNODE_NEW_CONTENT , &vnode->flags); +if (S_ISDIR (inode->i_mode)) +afs_mkdir_init_dir (vnode, dvp->vnode);if (!afs_op_error (op))afs_cache_permit (vnode, op->key, vnode->cb_break, &vp->scb);d_instantiate (op->dentry, inode);
fs/afs/dir_edit.c +48 -0
51 lines not on this page · raw message
fs/afs/internal.h +1 -0
@@ -1077,6 +1077,7 @@ extern void afs_edit_dir_add(struct afs_vnode *, struct qstr *, struct afs_fid *extern void afs_edit_dir_remove (struct afs_vnode *,struct qstr *,enum afs_edit_dir_reason );void afs_edit_dir_update_dotdot (struct afs_vnode *vnode,struct afs_vnode *new_dvnode,enum afs_edit_dir_reason why); +void afs_mkdir_init_dir (struct afs_vnode *dvnode,struct afs_vnode *parent_vnode);/* * dir_silly.c
include/trace/events/afs.h +2 -0
@@ -350,6 +350,7 @@ enum yfs_cm_operation { EM(afs_dir_invalid_edit_add_no_slots, "edit-add-no-slots") \ EM(afs_dir_invalid_edit_add_too_many_blocks, "edit-add-too-many-blocks") \ EM(afs_dir_invalid_edit_get_block, "edit-get-block") \ + EM(afs_dir_invalid_edit_mkdir, "edit-mkdir") \ EM(afs_dir_invalid_edit_rem_bad_size, "edit-rem-bad-size") \ EM(afs_dir_invalid_edit_rem_wrong_name, "edit-rem-wrong_name") \EM (afs_dir_invalid_edit_upd_bad_size,"edit-upd-bad-size" ) \
@@ -371,6 +372,7 @@ enum yfs_cm_operation { EM(afs_edit_dir_delete_error, "d_err ") \ EM(afs_edit_dir_delete_inval, "d_invl") \ EM(afs_edit_dir_delete_noent, "d_nent") \ + EM(afs_edit_dir_mkdir, "mk_ent") \ EM(afs_edit_dir_update_dd, "u_ddot") \ EM(afs_edit_dir_update_error, "u_fail") \EM (afs_edit_dir_update_inval,"u_invl" ) \
David Howells
maintainerPATCH v3 30/33 afs: Use the contained hashtable to search a directory
In reply to David HowellsEach directory image contains a hashtable with 128 buckets to speed up searching. Currently, kafs does not use this, but rather iterates over all the occupied slots in the image as it can share this with readdir.
Switch kafs to use the hashtable for lookups to reduce the latency. Care must be taken that the hash chains are acyclic.
--- fs/afs/Makefile | 1 + fs/afs/dir.c | 42 ++++---- fs/afs/dir_edit.c | 135 +++++++++++++++++--------- fs/afs/dir_search.c | 227 ++++++++++++++++++++++++++++++++++++++++++++ fs/afs/internal.h | 18 ++++ 5 files changed, 350 insertions(+), 73 deletions(-) create mode 100644 fs/afs/dir_search.c
fs/afs/Makefile +1 -0
@@ -11,6 +11,7 @@ kafs-y := \ cmservice.o \ dir.o \ dir_edit.o \ + dir_search.o \ dir_silly.o \ dynroot.o \ file.o \
fs/afs/dir.c +16 -26
82 lines not on this page · raw message
fs/afs/dir_edit.c +88 -47
205 lines not on this page · raw message
fs/afs/dir_search.c new file +227 -0
227 lines not on this page · raw message
fs/afs/internal.h +18 -0
@@ -977,9 +977,14 @@ static inline void afs_invalidate_cache(struct afs_vnode *vnode, unsigned int fl*/ struct afs_dir_iter {struct afs_vnode *dvnode; +union afs_xdr_dir_block *block;struct folio_queue *fq;unsigned int fpos;int fq_slot; +unsigned int loop_check; +u8 nr_slots; +u8 bucket; +unsigned int prev_entry; };#include <trace/events/afs.h>
@@ -1064,6 +1069,8 @@ extern const struct address_space_operations afs_dir_aops;extern const struct dentry_operations afs_fs_dentry_operations;ssize_t afs_read_single (struct afs_vnode *dvnode,struct file *file); +ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file) + __acquires(&dvnode->validate_lock);extern void afs_d_release (struct dentry *);extern void afs_check_for_remote_deletion (struct afs_operation *); int afs_single_writepages(struct address_space *mapping,
@@ -1079,6 +1086,17 @@ void afs_edit_dir_update_dotdot(struct afs_vnode *vnode, struct afs_vnode *new_d enum afs_edit_dir_reason why);void afs_mkdir_init_dir (struct afs_vnode *dvnode,struct afs_vnode *parent_vnode); +/* +* dir_search.c +*/ +unsigned int afs_dir_hash_name (const struct qstr *name); +bool afs_dir_init_iter (struct afs_dir_iter *iter,const struct qstr *name); +union afs_xdr_dir_block *afs_dir_find_block (struct afs_dir_iter *iter,size_t block); +int afs_dir_search_bucket (struct afs_dir_iter *iter,const struct qstr *name, +struct afs_fid *_fid); +int afs_dir_search (struct afs_vnode *dvnode,struct qstr *name, +struct afs_fid *_fid,afs_dataversion_t *_dir_version); +/* * dir_silly.c */
David Howells
maintainerPATCH v3 31/33 afs: Locally initialise the contents of a new symlink on creation
In reply to David HowellsSince we know what the contents of a symlink will be when we create it on the server, initialise its contents locally too to avoid the need to download it.
--- fs/afs/dir.c | 2 ++ fs/afs/inode.c | 44 ++++++++++++++++++++++++++++++++++------ fs/afs/internal.h | 1 + fs/netfs/buffered_read.c | 2 +- fs/netfs/read_single.c | 2 +- 5 files changed, 43 insertions(+), 8 deletions(-)
fs/afs/dir.c +2 -0
@@ -1270,6 +1270,8 @@ static void afs_vnode_new_inode(struct afs_operation *op)set_bit (AFS_VNODE_NEW_CONTENT , &vnode->flags);if (S_ISDIR (inode->i_mode))afs_mkdir_init_dir (vnode, dvp->vnode); +else if (S_ISLNK (inode->i_mode)) +afs_init_new_symlink (vnode, op);if (!afs_op_error (op))afs_cache_permit (vnode, op->key, vnode->cb_break, &vp->scb);d_instantiate (op->dentry, inode);
fs/afs/inode.c +38 -6
59 lines not on this page · raw message
fs/afs/internal.h +1 -0
@@ -1220,6 +1220,7 @@ extern void afs_fs_probe_cleanup(struct afs_net *);*/ extern const struct afs_operation_ops afs_fetch_status_operation; +void afs_init_new_symlink (struct afs_vnode *vnode,struct afs_operation *op);const char *afs_get_link (struct dentry *dentry,struct inode *inode,struct delayed_call *callback);int afs_readlink (struct dentry *dentry,char __user *buffer,int buflen);
fs/netfs/buffered_read.c +1 -1
@@ -210,7 +210,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) do { struct netfs_io_subrequest *subreq; - enum netfs_io_source source = NETFS_DOWNLOAD_FROM_SERVER; + enum netfs_io_source source = NETFS_SOURCE_UNKNOWN; ssize_t slice; subreq = netfs_alloc_subrequest(rreq);
fs/netfs/read_single.c +1 -1
@@ -97,7 +97,7 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq)if (!subreq)return -ENOMEM ; - subreq->source =NETFS_DOWNLOAD_FROM_SERVER ; + subreq->source =NETFS_SOURCE_UNKNOWN ; subreq->start =0 ; subreq->len = rreq->len; subreq->io_iter = rreq->buffer.iter;
David Howells
maintainerPATCH v3 32/33 afs: Add a tracepoint for afs_read_receive()
In reply to David HowellsAdd a tracepoint for afs_read_receive() to allow potential missed wakeups to be debugged.
--- fs/afs/file.c | 1 + include/trace/events/afs.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
fs/afs/file.c +1 -0
@@ -274,6 +274,7 @@ static void afs_read_receive(struct afs_call *call) state =READ_ONCE (call->state);if (state ==AFS_CALL_COMPLETE )return ; +trace_afs_read_recv (op, call);while (state <AFS_CALL_COMPLETE &&READ_ONCE (call->need_attention)) {WRITE_ONCE (call->need_attention, false);
include/trace/events/afs.h +30 -0
@@ -1775,6 +1775,36 @@ TRACE_EVENT(afs_make_call, __entry->fid.unique) ); +TRACE_EVENT (afs_read_recv, + TP_PROTO(const struct afs_operation *op, const struct afs_call *call), + +TP_ARGS (op, call), + +TP_STRUCT__entry ( + __field(unsigned int, rreq) + __field(unsigned int, sreq) + __field(unsigned int, op) + __field(unsigned int, op_flags) + __field(unsigned int, call) + __field(enum afs_call_state, call_state) + ), + +TP_fast_assign ( + __entry->op = op->debug_id; + __entry->sreq = op->fetch.subreq->debug_index; + __entry->rreq = op->fetch.subreq->rreq->debug_id; + __entry->op_flags = op->flags; + __entry->call = call->debug_id; + __entry->call_state = call->state; + ), + +TP_printk ("R=%08x[%x] OP=%08x c=%08x cs=%x of=%x" , + __entry->rreq, __entry->sreq, + __entry->op, + __entry->call, __entry->call_state, + __entry->op_flags) + ); +#endif /* _TRACE_AFS_H */ /* This part must be outside protection */
David Howells
maintainerPATCH v3 33/33 netfs: Report on NULL folioq in netfs_writeback_unlock_folios()
In reply to David HowellsIt seems that it's possible to get to netfs_writeback_unlock_folios() with an empty rolling buffer during buffered writes. This should not be possible as the rolling buffer is initialised as the write request is set up and thereafter maintains at least one folio_queue struct therein until it gets destroyed. This allows lockless addition and removal of folio_queue structs in the buffer because, unlike with a ring buffer, the producer and consumer each only need to look at and alter one pointer into the buffer.
Now, the rolling buffer is only used for buffered I/O operations as netfs_collect_write_results() should only call netfs_writeback_unlock_folios() if the request is of origin type NETFS_WRITEBACK, NETFS_WRITETHROUGH or NETFS_PGPRIV2_COPY_TO_CACHE.
So it would seem that one of the following occurred: (1) I/O started before the request was fully initialised, (2) the origin got switched mid-flow or
(3) the request has already been freed and this is a UAF error. I think the last is the most likely.
Make netfs_writeback_unlock_folios() report information about the request and subrequests if folioq is seen to be NULL to try and help debug this, throw a warning and return.
Note that this does not try to fix the problem.
--- fs/netfs/write_collect.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
fs/netfs/write_collect.c +34 -0
@@ -21,6 +21,34 @@#define NEED_RETRY 0x10/* A front op requests retrying */ #define SAW_FAILURE 0x20/* One stream or hit a permanent failure */ +static void netfs_dump_request (const struct netfs_io_request *rreq) +{ +pr_err ("Request R=%08x r=%d fl=%lx or=%x e=%ld\n" , + rreq->debug_id,refcount_read (&rreq->ref), rreq->flags, + rreq->origin, rreq->error); +pr_err (" st=%llx tsl=%zx/%llx/%llx\n" , + rreq->start, rreq->transferred, rreq->submitted, rreq->len); +pr_err (" cci=%llx/%llx/%llx\n" , + rreq->cleaned_to, rreq->collected_to,atomic64_read (&rreq->issued_to)); +pr_err (" iw=%pSR\n" , rreq->netfs_ops->issue_write); +for (int i =0 ; i <NR_IO_STREAMS ; i++) { +const struct netfs_io_subrequest *sreq; +const struct netfs_io_stream *s = &rreq->io_streams[i]; + +pr_err (" str[%x] s=%x e=%d acnf=%u,%u,%u,%u\n" , + s->stream_nr, s->source, s->error, + s->avail, s->active, s->need_retry, s->failed); +pr_err (" str[%x] ct=%llx t=%zx\n" , + s->stream_nr, s->collected_to, s->transferred); +list_for_each_entry (sreq, &s->subrequests, rreq_link) { +pr_err (" sreq[%x:%x] sc=%u s=%llx t=%zx/%zx r=%d f=%lx\n" , + sreq->stream_nr, sreq->debug_index, sreq->source, + sreq->start, sreq->transferred, sreq->len, +refcount_read (&sreq->ref), sreq->flags); + } + } +} +/* * Successful completion of write of a folio to the server and/or cache. Note * that we are not allowed to lock the folio here on pain of deadlocking with
@@ -87,6 +115,12 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,unsigned long long collected_to = wreq->collected_to;unsigned int slot = wreq->buffer.first_tail_slot; +if (WARN_ON_ONCE (!folioq)) { +pr_err ("[!] Writeback unlock found empty rolling buffer!\n" ); +netfs_dump_request (wreq); +return ; + } +if (wreq->origin ==NETFS_PGPRIV2_COPY_TO_CACHE ) {if (netfs_pgpriv2_unlock_copied_folios (wreq)) *notes |=MADE_PROGRESS ;