netfs, cifs: Miscellaneous fixes and read/write improvements
Subsystems (16)
David Howells
maintainercover netfs, cifs: Miscellaneous fixes and read/write improvements
Hi Christian, Steve, Willy,
[!] NOTE: THE LAST PATCH IS INCOMPLETE AND ONLY PARTIALLY WORKS.
[!] Willy: Could you look at the sheaf-adding patch?
This set of patches includes some netfs and cifs fixes:
(1) Make sure we set the request length when doing an unbuffered/direct write. This was in v2 of the io_uring fix writethrough patch, but v1 got merged instead.
(2) Move CIFS_INO_MODIFIED_ATTR to netfs_inode::flags to avoid jumping through a function pointer just for that.
(3) Fix a premature issuing of write ops on a partially dirty page where the dirty part is at the end of the page and thus may be contiguous with the next page.
A couple of adjustments to the /proc/fs/netfs/stats file:
(4) All the netfs stats lines begin 'Netfs:'. Change this to something a bit more useful.
(5) Add a couple of stats counters to track the numbers of skips and waits on the per-inode writeback serialisation lock to make it easier to check for this as a source of performance loss.
Some miscellaneous bits:
(6) Enable multipage folios in 9P.
(7) Reduce the number of conditional branches in netfs_perform_write().
(8) Delete some unused netfslib functions for dealing with xarrays.
(9) In cifs, defer read completion.
(10) In cifs, only pick a channel once per read request (as per v6.9) rather than per-subrequest.
(11) In cifs, move the 'pid' from the subrequest to the request as it's the same for all.
(12) Move the max_len/max_nr_segs members from netfs_io_subrequest to netfs_io_request as they're only needed for one subreq at a time.
Then there's the main performance enhancing changes:
(13) Define a structure, struct sheaf, and a new iterator type, ITER_SHEAF, to hold a buffer as a replacement for ITER_XARRAY. See that patch for questions about naming and form.
(14) Use sheaves in the write-side helpers instead of xarrays.
(15) Simplify the write-side helpers to use sheaves to skip gaps rather than trying to work out where gaps are.
(16) In afs, make the read subrequests asynchronous, putting them into work items to allow the next patch to do progressive unlocking/reading.
(17) Overhaul the read-side helpers to improve performance. [!] NOTE: THIS IS INCOMPLETE! It works for buffered read to a large extent, but not DIO and may not cache correctly.
David
David Howells (17):
netfs: Fix io_uring based write-through
netfs, cifs: Move CIFS_INO_MODIFIED_ATTR to netfs_inode
netfs: Fix early issue of write op on partial write to folio tail
netfs: Adjust labels in /proc/fs/netfs/stats
netfs: Record contention stats for writeback lock
9p: Enable multipage folios
netfs: Reduce number of conditional branches in netfs_perform_write()
netfs: Delete some xarray-wangling functions that aren't used
cifs: Defer read completion
cifs: Only pick a channel once per read request
cifs: Move the 'pid' from the subreq to the req
netfs: Move max_len/max_nr_segs from netfs_io_subrequest to
netfs_io_stream
mm: Define struct sheaf and ITER_SHEAF to handle a sequence of folios
netfs: Use new sheaf data type and iterator instead of xarray iter
netfs: Simplify the writeback code
afs: Make read subreqs async
netfs: Speed up buffered reading
fs/9p/vfs_addr.c | 13 +- fs/9p/vfs_inode.c | 1 + fs/afs/file.c | 21 +- fs/afs/fsclient.c | 9 +- fs/afs/write.c | 4 +- fs/afs/yfsclient.c | 9 +- fs/cachefiles/io.c | 5 +- fs/ceph/addr.c | 72 ++--- fs/netfs/Makefile | 2 +- fs/netfs/buffered_read.c | 510 +++++++++++++++++++++------------ fs/netfs/buffered_write.c | 308 ++++++++++---------- fs/netfs/direct_read.c | 99 ++++++- fs/netfs/direct_write.c | 3 +- fs/netfs/internal.h | 26 +- fs/netfs/io.c | 528 +---------------------------------- fs/netfs/iterator.c | 50 ++++ fs/netfs/main.c | 7 +- fs/netfs/misc.c | 115 ++++---- fs/netfs/objects.c | 2 +- fs/netfs/read_collect.c | 450 +++++++++++++++++++++++++++++ fs/netfs/stats.c | 23 +- fs/netfs/write_collect.c | 242 +++++----------- fs/netfs/write_issue.c | 68 ++--- fs/nfs/fscache.c | 31 +- fs/nfs/fscache.h | 7 +- fs/smb/client/cifsglob.h | 4 +- fs/smb/client/cifssmb.c | 14 +- fs/smb/client/file.c | 109 +++----- fs/smb/client/smb2pdu.c | 21 +- include/linux/iov_iter.h | 57 ++++ include/linux/netfs.h | 29 +- include/linux/sheaf.h | 83 ++++++ include/linux/uio.h | 11 + include/trace/events/netfs.h | 124 +++++--- lib/iov_iter.c | 222 ++++++++++++++- lib/kunit_iov_iter.c | 259 +++++++++++++++++ lib/scatterlist.c | 69 ++++- 37 files changed, 2251 insertions(+), 1356 deletions(-) create mode 100644 fs/netfs/read_collect.c create mode 100644 include/linux/sheaf.h
David Howells
maintainerPATCH 1/17 netfs: Fix io_uring based write-through
[This was included in v2 of 9b038d004ce95551cb35381c49fe896c5bc11ffe, but v1 got pushed instead]
Fix netfs_unbuffered_write_iter_locked() to set the total request length in the netfs_io_request struct rather than leaving it as zero.
--- fs/netfs/direct_write.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/netfs/direct_write.c +2 -1
@@ -92,8 +92,9 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *__set_bit (NETFS_RREQ_UPLOAD_TO_SERVER , &wreq->flags);if (async) wreq->iocb = iocb; + wreq->len =iov_iter_count (&wreq->io_iter); wreq->cleanup = netfs_cleanup_dio_write; - ret =netfs_unbuffered_write (wreq,is_sync_kiocb (iocb),iov_iter_count (&wreq->io_iter)); + ret =netfs_unbuffered_write (wreq,is_sync_kiocb (iocb), wreq->len);if (ret <0 ) {_debug ("begin = %zd" , ret); goto out;
David Howells
maintainerPATCH 2/17 netfs, cifs: Move CIFS_INO_MODIFIED_ATTR to netfs_inode
In reply to David HowellsMove CIFS_INO_MODIFIED_ATTR to netfs_inode as NETFS_ICTX_MODIFIED_ATTR and then make netfs_perform_write() set it. This means that cifs doesn't need to implement the ->post_modify() hook.
--- fs/netfs/buffered_write.c | 10 ++++++++-- fs/smb/client/cifsglob.h | 1 - fs/smb/client/file.c | 9 +-------- include/linux/netfs.h | 1 + 4 files changed, 10 insertions(+), 11 deletions(-)
fs/netfs/buffered_write.c +8 -2
@@ -405,8 +405,14 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, }while (iov_iter_count (iter)); out: -if (likely (written) && ctx->ops->post_modify) - ctx->ops->post_modify (inode); +if (likely (written)) { +/* Set indication that ctime and mtime got updated in case +* close is deferred. +*/ +set_bit (NETFS_ICTX_MODIFIED_ATTR , &ctx->flags); +if (unlikely (ctx->ops->post_modify)) + ctx->ops->post_modify (inode); + }if (unlikely (wreq)) { ret2 =netfs_end_writethrough (wreq, &wbc, writethrough);
fs/smb/client/cifsglob.h +0 -1
@@ -1569,7 +1569,6 @@ struct cifsInodeInfo {#define CIFS_INO_DELETE_PENDING (3)/* delete pending on server */ #define CIFS_INO_INVALID_MAPPING (4)/* pagecache is invalid */ #define CIFS_INO_LOCK (5)/* lock bit for synchronization */ -#define CIFS_INO_MODIFIED_ATTR (6)/* Indicate change in mtime/ctime */ #define CIFS_INO_CLOSE_ON_LOCK (7)/* Not to defer the close when lock is set */ unsigned long flags;spinlock_t writers_lock;
fs/smb/client/file.c +1 -8
@@ -302,12 +302,6 @@ static void cifs_rreq_done(struct netfs_io_request *rreq)inode_set_atime_to_ts (inode,inode_get_mtime (inode)); } -static void cifs_post_modify (struct inode *inode) -{ -/* Indication to update ctime and mtime as close is deferred */ -set_bit (CIFS_INO_MODIFIED_ATTR , &CIFS_I (inode)->flags); -} -static void cifs_free_request (struct netfs_io_request *rreq) { struct cifs_io_request *req = container_of(rreq, struct cifs_io_request, rreq);
@@ -346,7 +340,6 @@ const struct netfs_request_ops cifs_req_ops = { .clamp_length = cifs_clamp_length, .issue_read = cifs_req_issue_read, .done = cifs_rreq_done, - .post_modify = cifs_post_modify, .begin_writeback = cifs_begin_writeback, .prepare_write = cifs_prepare_write, .issue_write = cifs_issue_write,
@@ -1369,7 +1362,7 @@ int cifs_close(struct inode *inode, struct file *file) dclose =kmalloc (sizeof (struct cifs_deferred_close ),GFP_KERNEL );if ((cfile->status_file_deleted == false) && (smb2_can_defer_close (inode, dclose))) { -if (test_and_clear_bit (CIFS_INO_MODIFIED_ATTR , &cinode->flags)) { +if (test_and_clear_bit (NETFS_ICTX_MODIFIED_ATTR , &cinode->netfs.flags)) {inode_set_mtime_to_ts (inode,inode_set_ctime_current (inode)); }
include/linux/netfs.h +1 -0
@@ -73,6 +73,7 @@ struct netfs_inode {#define NETFS_ICTX_ODIRECT 0/* The file has DIO in progress */ #define NETFS_ICTX_UNBUFFERED 1/* I/O should not use the pagecache */ #define NETFS_ICTX_WRITETHROUGH 2/* Write-through caching */ +#define NETFS_ICTX_MODIFIED_ATTR 3/* Indicate change in mtime/ctime */ #define NETFS_ICTX_USE_PGPRIV2 31/* [DEPRECATED] Use PG_private_2 to mark * write to cache on read */ };
David Howells
maintainerPATCH 3/17 netfs: Fix early issue of write op on partial write to folio tail
In reply to David HowellsDuring the writeback procedure, at the end of netfs_write_folio(), pending write operations are flushed if the amount of write-streaming data stored in a page is less than the size of the folio because if we haven't modified a folio to the end, it cannot be contiguous with the following folio... except if the dirty region of the folio is right at the end of the folio space.
Fix the test to take the offset into the folio into account as well, such that if the dirty region runs right up to the end of the folio, we leave the flushing for later.
--- fs/netfs/write_issue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/netfs/write_issue.c +1 -1
@@ -483,7 +483,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,if (!debug)kdebug ("R=%x: No submit" , wreq->debug_id); -if (flen < fsize) +if (foff + flen < fsize)for (int s =0 ; s <NR_IO_STREAMS ; s++)netfs_issue_write (wreq, &wreq->io_streams[s]);
David Howells
maintainerPATCH 4/17 netfs: Adjust labels in /proc/fs/netfs/stats
In reply to David HowellsAdjust the labels in /proc/fs/netfs/stats that refer to netfs-specific counters. These currently all begin with "Netfs", but change them to begin with more specific labels.
--- fs/netfs/stats.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
fs/netfs/stats.c +8 -8
@@ -42,39 +42,39 @@ atomic_t netfs_n_wh_write_failed;int netfs_stats_show (struct seq_file *m,void *v) { -seq_printf (m,"Netfs : DR=%u RA=%u RF=%u WB=%u WBZ=%u\n" , +seq_printf (m,"Reads : DR=%u RA=%u RF=%u WB=%u WBZ=%u\n" ,atomic_read (&netfs_n_rh_dio_read),atomic_read (&netfs_n_rh_readahead),atomic_read (&netfs_n_rh_read_folio),atomic_read (&netfs_n_rh_write_begin),atomic_read (&netfs_n_rh_write_zskip)); -seq_printf (m,"Netfs : BW=%u WT=%u DW=%u WP=%u\n" , +seq_printf (m,"Writes : BW=%u WT=%u DW=%u WP=%u\n" ,atomic_read (&netfs_n_wh_buffered_write),atomic_read (&netfs_n_wh_writethrough),atomic_read (&netfs_n_wh_dio_write),atomic_read (&netfs_n_wh_writepages)); -seq_printf (m,"Netfs : ZR=%u sh=%u sk=%u\n" , +seq_printf (m,"ZeroOps: ZR=%u sh=%u sk=%u\n" ,atomic_read (&netfs_n_rh_zero),atomic_read (&netfs_n_rh_short_read),atomic_read (&netfs_n_rh_write_zskip)); -seq_printf (m,"Netfs : DL=%u ds=%u df=%u di=%u\n" , +seq_printf (m,"DownOps: DL=%u ds=%u df=%u di=%u\n" ,atomic_read (&netfs_n_rh_download),atomic_read (&netfs_n_rh_download_done),atomic_read (&netfs_n_rh_download_failed),atomic_read (&netfs_n_rh_download_instead)); -seq_printf (m,"Netfs : RD=%u rs=%u rf=%u\n" , +seq_printf (m,"CaRdOps: RD=%u rs=%u rf=%u\n" ,atomic_read (&netfs_n_rh_read),atomic_read (&netfs_n_rh_read_done),atomic_read (&netfs_n_rh_read_failed)); -seq_printf (m,"Netfs : UL=%u us=%u uf=%u\n" , +seq_printf (m,"UpldOps: UL=%u us=%u uf=%u\n" ,atomic_read (&netfs_n_wh_upload),atomic_read (&netfs_n_wh_upload_done),atomic_read (&netfs_n_wh_upload_failed)); -seq_printf (m,"Netfs : WR=%u ws=%u wf=%u\n" , +seq_printf (m,"CaWrOps: WR=%u ws=%u wf=%u\n" ,atomic_read (&netfs_n_wh_write),atomic_read (&netfs_n_wh_write_done),atomic_read (&netfs_n_wh_write_failed)); -seq_printf (m,"Netfs : rr=%u sr=%u wsc=%u\n" , +seq_printf (m,"Objs : rr=%u sr=%u wsc=%u\n" ,atomic_read (&netfs_n_rh_rreq),atomic_read (&netfs_n_rh_sreq),atomic_read (&netfs_n_wh_wstream_conflict));
David Howells
maintainerPATCH 5/17 netfs: Record contention stats for writeback lock
In reply to David HowellsRecord statistics for contention upon the writeback serialisation lock that prevents racing writeback calls from causing each other to interleave their writebacks. These can be viewed in /proc/fs/netfs/stats on the WbLock line, with skip=N indicating the number of non-SYNC writebacks skipped and wait=N indicating the number of SYNC writebacks that waited.
--- fs/netfs/internal.h | 2 ++ fs/netfs/stats.c | 5 +++++ fs/netfs/write_issue.c | 10 +++++++--- 3 files changed, 14 insertions(+), 3 deletions(-)
fs/netfs/internal.h +2 -0
@@ -126,6 +126,8 @@ extern atomic_t netfs_n_wh_upload_failed;extern atomic_t netfs_n_wh_write;extern atomic_t netfs_n_wh_write_done;extern atomic_t netfs_n_wh_write_failed; +extern atomic_t netfs_n_wb_lock_skip; +extern atomic_t netfs_n_wb_lock_wait;int netfs_stats_show (struct seq_file *m,void *v);
fs/netfs/stats.c +5 -0
@@ -39,6 +39,8 @@ atomic_t netfs_n_wh_upload_failed;atomic_t netfs_n_wh_write;atomic_t netfs_n_wh_write_done;atomic_t netfs_n_wh_write_failed; +atomic_t netfs_n_wb_lock_skip; +atomic_t netfs_n_wb_lock_wait;int netfs_stats_show (struct seq_file *m,void *v) {
@@ -78,6 +80,9 @@ int netfs_stats_show(struct seq_file *m, void *v)atomic_read (&netfs_n_rh_rreq),atomic_read (&netfs_n_rh_sreq), atomic_read(&netfs_n_wh_wstream_conflict)); +seq_printf (m,"WbLock : skip=%u wait=%u\n" , +atomic_read (&netfs_n_wb_lock_skip), +atomic_read (&netfs_n_wb_lock_wait));return fscache_stats_show (m); }EXPORT_SYMBOL (netfs_stats_show);
fs/netfs/write_issue.c +7 -3
@@ -502,10 +502,14 @@ int netfs_writepages(struct address_space *mapping,struct folio *folio;int error =0 ; -if (wbc->sync_mode ==WB_SYNC_ALL ) +if (!mutex_trylock (&ictx->wb_lock)) { +if (wbc->sync_mode ==WB_SYNC_NONE ) { +netfs_stat (&netfs_n_wb_lock_skip); +return 0 ; + } +netfs_stat (&netfs_n_wb_lock_wait);mutex_lock (&ictx->wb_lock); -else if (!mutex_trylock (&ictx->wb_lock)) -return 0 ; + }/* Need the first folio to be able to set up the op. */ folio =writeback_iter (mapping, wbc,NULL , &error);
David Howells
maintainerPATCH 6/17 9p: Enable multipage folios
In reply to David HowellsEnable support for multipage folios on the 9P filesystem. This is all handled through netfslib and is already enabled on AFS and CIFS also.
--- fs/9p/vfs_inode.c | 1 + 1 file changed, 1 insertion(+)
fs/9p/vfs_inode.c +1 -0
@@ -295,6 +295,7 @@ int v9fs_init_inode(struct v9fs_session_info *v9ses, inode->i_op = &v9fs_file_inode_operations; inode->i_fop = &v9fs_file_operations; } +mapping_set_large_folios (inode->i_mapping);break ;case S_IFLNK :
Dominique Martinet
maintainerRe: [PATCH 06/17] 9p: Enable multipage folios
David Howells wrote on Thu, Jun 20, 2024 at 06:31:24PM +0100:
Quoted 2 lines
Enable support for multipage folios on the 9P filesystem. This is all handled through netfslib and is already enabled on AFS and CIFS also.
Since this is fairly unrelated to the other patches let's take this through the 9p tree as well - I'll run some quick tests to verify writes go from 4k to something larger and it doesn't blow up immediately and push it out for 6.11
Signature · 1 line
Dominique Martinet | Asmadeus
Re: [PATCH 06/17] 9p: Enable multipage folios
Dominique Martinet wrote on Fri, Jun 21, 2024 at 05:34:49AM +0900:
Quoted 7 lines
David Howells wrote on Thu, Jun 20, 2024 at 06:31:24PM +0100:
Quoted 2 lines
Enable support for multipage folios on the 9P filesystem. This is all handled through netfslib and is already enabled on AFS and CIFS also.
Since this is fairly unrelated to the other patches let's take this through the 9p tree as well - I'll run some quick tests to verify writes go from 4k to something larger
(huh, my memory is rotten, we were already aggregating writes at some point without this. Oh, well, at least it doesn't seem to blow up)
Quoted 1 line
and it doesn't blow up immediately and push it out for 6.11
Queued for -next.
Signature · 1 line
Dominique Martinet | Asmadeus
David Howells
maintainerPATCH 7/17 netfs: Reduce number of conditional branches in netfs_perform_write()
In reply to David HowellsReduce the number of conditional branches in netfs_perform_write() by merging in netfs_how_to_modify() and then creating a separate if-statement for each way we might modify a folio. Note that this means replicating the data copy in each path.
--- fs/netfs/buffered_write.c | 298 ++++++++++++++++------------------- include/trace/events/netfs.h | 2 - 2 files changed, 133 insertions(+), 167 deletions(-)
fs/netfs/buffered_write.c +133 -165
@@ -13,91 +13,22 @@#include <linux/pagevec.h> #include "internal.h" -/* -* Determined write method. Adjust netfs_folio_traces if this is changed. -*/ -enum netfs_how_to_modify { -NETFS_FOLIO_IS_UPTODATE ,/* Folio is uptodate already */ -NETFS_JUST_PREFETCH ,/* We have to read the folio anyway */ -NETFS_WHOLE_FOLIO_MODIFY ,/* We're going to overwrite the whole folio */ -NETFS_MODIFY_AND_CLEAR ,/* We can assume there is no data to be downloaded. */ -NETFS_STREAMING_WRITE ,/* Store incomplete data in non-uptodate page. */ -NETFS_STREAMING_WRITE_CONT ,/* Continue streaming write. */ -NETFS_FLUSH_CONTENT ,/* Flush incompatible content. */ -}; - -static void netfs_set_group (struct folio *folio,struct netfs_group *netfs_group) +static void __netfs_set_group (struct folio *folio,struct netfs_group *netfs_group) { -void *priv =folio_get_private (folio); - -if (netfs_group && (!priv || priv ==NETFS_FOLIO_COPY_TO_CACHE )) +if (netfs_group)folio_attach_private (folio,netfs_get_group (netfs_group)); -else if (!netfs_group && priv ==NETFS_FOLIO_COPY_TO_CACHE ) -folio_detach_private (folio); } -/* -* Decide how we should modify a folio. We might be attempting to do -* write-streaming, in which case we don't want to a local RMW cycle if we can -* avoid it. If we're doing local caching or content crypto, we award that -* priority over avoiding RMW. If the file is open readably, then we also -* assume that we may want to read what we wrote. -*/ -static enum netfs_how_to_modify netfs_how_to_modify (struct netfs_inode *ctx, -struct file *file, -struct folio *folio, -void *netfs_group, -size_t flen, -size_t offset, -size_t len, -bool maybe_trouble) +static void netfs_set_group (struct folio *folio,struct netfs_group *netfs_group) { -struct netfs_folio *finfo =netfs_folio_info (folio); -struct netfs_group *group =netfs_folio_group (folio); -loff_t pos =folio_file_pos (folio); - -_enter ("" ); - -if (group != netfs_group && group !=NETFS_FOLIO_COPY_TO_CACHE ) -return NETFS_FLUSH_CONTENT ; - -if (folio_test_uptodate (folio)) -return NETFS_FOLIO_IS_UPTODATE ; - -if (pos >= ctx->zero_point) -return NETFS_MODIFY_AND_CLEAR ; - -if (!maybe_trouble && offset ==0 && len >= flen) -return NETFS_WHOLE_FOLIO_MODIFY ; - -if (file->f_mode &FMODE_READ ) - goto no_write_streaming; - -if (netfs_is_cache_enabled (ctx)) { -/* We don't want to get a streaming write on a file that loses -* caching service temporarily because the backing store got -* culled. -*/ - goto no_write_streaming; - } +void *priv =folio_get_private (folio); -if (!finfo) -return NETFS_STREAMING_WRITE ; - -/* We can continue a streaming write only if it continues on from the -* previous. If it overlaps, we must flush lest we suffer a partial -* copy and disjoint dirty regions. -*/ -if (offset == finfo->dirty_offset + finfo->dirty_len) -return NETFS_STREAMING_WRITE_CONT ; -return NETFS_FLUSH_CONTENT ; - -no_write_streaming: -if (finfo) { -netfs_stat (&netfs_n_wh_wstream_conflict); -return NETFS_FLUSH_CONTENT ; +if (unlikely (priv != netfs_group)) { +if (netfs_group && (!priv || priv ==NETFS_FOLIO_COPY_TO_CACHE )) +folio_attach_private (folio,netfs_get_group (netfs_group)); +else if (!netfs_group && priv ==NETFS_FOLIO_COPY_TO_CACHE ) +folio_detach_private (folio); } -return NETFS_JUST_PREFETCH ; }/*
@@ -177,13 +108,10 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, .range_end = iocb->ki_pos + iter->count, };struct netfs_io_request *wreq =NULL ; -struct netfs_folio *finfo; -struct folio *folio, *writethrough =NULL ; -enum netfs_how_to_modify howto; -enum netfs_folio_trace trace; +struct folio *folio =NULL , *writethrough =NULL ;unsigned int bdp_flags = (iocb->ki_flags &IOCB_NOWAIT ) ?BDP_ASYNC :0 ;ssize_t written =0 , ret, ret2; -loff_t i_size, pos = iocb->ki_pos, from, to; +loff_t i_size, pos = iocb->ki_pos;size_t max_chunk =PAGE_SIZE <<MAX_PAGECACHE_ORDER ;bool maybe_trouble = false;
@@ -213,15 +141,14 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, } do { + struct netfs_folio *finfo; + struct netfs_group *group; + unsigned long long fpos; size_t flen; size_t offset; /* Offset into pagecache folio */ size_t part; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ - ret = balance_dirty_pages_ratelimited_flags(mapping, bdp_flags); - if (unlikely(ret < 0)) - break; - offset = pos & (max_chunk - 1); part = min(max_chunk - offset, iov_iter_count(iter));
@@ -247,7 +174,8 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, } flen =folio_size (folio); - offset = pos & (flen -1 ); + fpos =folio_pos (folio); + offset = pos - fpos; part =min_t (size_t, flen - offset, part);/* Wait for writeback to complete. The writeback engine owns
@@ -265,71 +193,52 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, goto error_folio_unlock; } -/* See if we need to prefetch the area we're going to modify. -* We need to do this before we get a lock on the folio in case -* there's more than one writer competing for the same cache -* block. +/* Decide how we should modify a folio. We might be attempting +* to do write-streaming, in which case we don't want to a +* local RMW cycle if we can avoid it. If we're doing local +* caching or content crypto, we award that priority over +* avoiding RMW. If the file is open readably, then we also +* assume that we may want to read what we wrote. */ - howto =netfs_how_to_modify (ctx, file, folio, netfs_group, - flen, offset, part, maybe_trouble); -_debug ("howto %u" , howto); -switch (howto) { -case NETFS_JUST_PREFETCH : - ret =netfs_prefetch_for_write (file, folio, offset, part); -if (ret <0 ) { -_debug ("prefetch = %zd" , ret); - goto error_folio_unlock; - } -break ; -case NETFS_FOLIO_IS_UPTODATE : -case NETFS_WHOLE_FOLIO_MODIFY : -case NETFS_STREAMING_WRITE_CONT : -break ; -case NETFS_MODIFY_AND_CLEAR : -zero_user_segment (&folio->page,0 , offset); -break ; -case NETFS_STREAMING_WRITE : - ret = -EIO ; -if (WARN_ON (folio_get_private (folio))) - goto error_folio_unlock; -break ; -case NETFS_FLUSH_CONTENT : -trace_netfs_folio (folio, netfs_flush_content); - from =folio_pos (folio); - to = from +folio_size (folio) -1 ; -folio_unlock (folio); -folio_put (folio); - ret =filemap_write_and_wait_range (mapping, from, to); -if (ret <0 ) - goto error_folio_unlock; -continue ; - } - -if (mapping_writably_mapped (mapping)) -flush_dcache_folio (folio); - - copied =copy_folio_from_iter_atomic (folio, offset, part, iter); - -flush_dcache_folio (folio); - -/* Deal with a (partially) failed copy */ -if (copied ==0 ) { - ret = -EFAULT ; - goto error_folio_unlock; + finfo =netfs_folio_info (folio); + group =netfs_folio_group (folio); + +if (unlikely (group != netfs_group) && + group !=NETFS_FOLIO_COPY_TO_CACHE ) + goto flush_content; + +if (folio_test_uptodate (folio)) { +if (mapping_writably_mapped (mapping)) +flush_dcache_folio (folio); + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed; +netfs_set_group (folio, netfs_group); +trace_netfs_folio (folio, netfs_folio_is_uptodate); + goto copied; } - trace = (enum netfs_folio_trace )howto; -switch (howto) { -case NETFS_FOLIO_IS_UPTODATE : -case NETFS_JUST_PREFETCH : -netfs_set_group (folio, netfs_group); -break ; -case NETFS_MODIFY_AND_CLEAR : +/* If the page is above the zero-point then we assume that the +* server would just return a block of zeros or a short read if +* we try to read it. +*/ +if (fpos >= ctx->zero_point) { +zero_user_segment (&folio->page,0 , offset); + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed;zero_user_segment (&folio->page, offset + copied, flen); -netfs_set_group (folio, netfs_group); +__netfs_set_group (folio, netfs_group);folio_mark_uptodate (folio); -break ; -case NETFS_WHOLE_FOLIO_MODIFY : +trace_netfs_folio (folio, netfs_modify_and_clear); + goto copied; + } + +/* See if we can write a whole folio in one go. */ +if (!maybe_trouble && offset ==0 && part >= flen) { + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed;if (unlikely (copied < part)) { maybe_trouble = true;iov_iter_revert (iter, copied);
@@ -337,16 +246,52 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,folio_unlock (folio); goto retry; } -netfs_set_group (folio, netfs_group); +__netfs_set_group (folio, netfs_group);folio_mark_uptodate (folio); -break ; -case NETFS_STREAMING_WRITE : +trace_netfs_folio (folio, netfs_whole_folio_modify); + goto copied; + } + +/* We don't want to do a streaming write on a file that loses +* caching service temporarily because the backing store got +* culled and we don't really want to get a streaming write on +* a file that's open for reading as ->read_folio() then has to +* be able to flush it. +*/ +if ((file->f_mode &FMODE_READ ) || +netfs_is_cache_enabled (ctx)) { +if (finfo) { +netfs_stat (&netfs_n_wh_wstream_conflict); + goto flush_content; + } + ret =netfs_prefetch_for_write (file, folio, offset, part); +if (ret <0 ) { +_debug ("prefetch = %zd" , ret); + goto error_folio_unlock; + } + + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed; +__netfs_set_group (folio, netfs_group); +trace_netfs_folio (folio, netfs_just_prefetch); + goto copied; + } + +if (!finfo) { + ret = -EIO ; +if (WARN_ON (folio_get_private (folio))) + goto error_folio_unlock; + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed;if (offset ==0 && copied == flen) { -netfs_set_group (folio, netfs_group); +__netfs_set_group (folio, netfs_group);folio_mark_uptodate (folio); - trace = netfs_streaming_filled_page; -break ; +trace_netfs_folio (folio, netfs_streaming_filled_page); + goto copied; } + finfo =kzalloc (sizeof (*finfo),GFP_KERNEL );if (!finfo) {iov_iter_revert (iter, copied);
@@ -358,9 +303,18 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter, finfo->dirty_len = copied;folio_attach_private (folio, (void *)((unsigned long )finfo |NETFS_FOLIO_INFO )); -break ; -case NETFS_STREAMING_WRITE_CONT : - finfo =netfs_folio_info (folio); +trace_netfs_folio (folio, netfs_streaming_write); + goto copied; + } + +/* We can continue a streaming write only if it continues on +* from the previous. If it overlaps, we must flush lest we +* suffer a partial copy and disjoint dirty regions. +*/ +if (offset == finfo->dirty_offset + finfo->dirty_len) { + copied =copy_folio_from_iter_atomic (folio, offset, part, iter); +if (unlikely (copied ==0 )) + goto copy_failed; finfo->dirty_len += copied;if (finfo->dirty_offset ==0 && finfo->dirty_len == flen) {if (finfo->netfs_group)
@@ -369,17 +323,25 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,folio_detach_private (folio);folio_mark_uptodate (folio);kfree (finfo); - trace = netfs_streaming_cont_filled_page; +trace_netfs_folio (folio, netfs_streaming_cont_filled_page); + }else { +trace_netfs_folio (folio, netfs_streaming_write_cont); } -break ; -default : -WARN (true,"Unexpected modify type %u ix=%lx\n" , - howto, folio->index); - ret = -EIO ; - goto error_folio_unlock; + goto copied; } -trace_netfs_folio (folio, trace); +/* Incompatible write; flush the folio and try again. */ + flush_content: +trace_netfs_folio (folio, netfs_flush_content); +folio_unlock (folio); +folio_put (folio); + ret =filemap_write_and_wait_range (mapping, fpos, fpos + flen -1 ); +if (ret <0 ) + goto error_folio_unlock; +continue ; + + copied: +flush_dcache_folio (folio);/* Update the inode size if we moved the EOF marker */ pos += copied;
@@ -401,6 +363,10 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,folio_put (folio); folio =NULL ; + ret =balance_dirty_pages_ratelimited_flags (mapping, bdp_flags); +if (unlikely (ret <0 )) +break ; +cond_resched (); }while (iov_iter_count (iter));
@@ -427,6 +393,8 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,_leave (" = %zd [%zd]" , written, ret);return written ? written : ret; +copy_failed: + ret = -EFAULT ; error_folio_unlock:folio_unlock (folio);folio_put (folio);
include/trace/events/netfs.h +0 -2
@@ -128,7 +128,6 @@ E_(netfs_sreq_trace_put_terminated, "PUT TERM ")#define netfs_folio_traces \ -/* The first few correspond to enum netfs_how_to_modify */ \ EM(netfs_folio_is_uptodate, "mod-uptodate") \ EM(netfs_just_prefetch, "mod-prefetch") \ EM(netfs_whole_folio_modify, "mod-whole-f") \
@@ -138,7 +137,6 @@ EM(netfs_flush_content, "flush") \ EM(netfs_streaming_filled_page, "mod-streamw-f") \ EM(netfs_streaming_cont_filled_page, "mod-streamw-f+") \ - /* The rest are for writeback */ \ EM(netfs_folio_trace_cancel_copy, "cancel-copy") \ EM(netfs_folio_trace_clear, "clear") \EM (netfs_folio_trace_clear_cc,"clear-cc" ) \
David Howells
maintainerPATCH 8/17 netfs: Delete some xarray-wangling functions that aren't used
In reply to David HowellsDelete some xarray-based buffer wangling functions that are intended for use with bounce buffering, but aren't used because bounce-buffering got deferred to a later patch series. Now, however, the intention is to use something other than an xarray to do this.
--- fs/netfs/internal.h | 9 ----- fs/netfs/misc.c | 81 --------------------------------------------- 2 files changed, 90 deletions(-)
fs/netfs/internal.h +0 -9
@@ -63,15 +63,6 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}/* * misc.c */ -#define NETFS_FLAG_PUT_MARK BIT(0) -#define NETFS_FLAG_PAGECACHE_MARK BIT(1) -int netfs_xa_store_and_mark (struct xarray *xa,unsigned long index, -struct folio *folio,unsigned int flags, -gfp_t gfp_mask); -int netfs_add_folios_to_buffer (struct xarray *buffer, -struct address_space *mapping, -pgoff_t index,pgoff_t to,gfp_t gfp_mask); -void netfs_clear_buffer (struct xarray *buffer);/* * objects.c
fs/netfs/misc.c +0 -81
87 lines not on this page · raw message
David Howells
maintainerPATCH 9/17 cifs: Defer read completion
In reply to David Howells--- fs/smb/client/smb2pdu.c | 15 ++++++++++++--- include/trace/events/netfs.h | 7 +++++-- 2 files changed, 17 insertions(+), 5 deletions(-)
fs/smb/client/smb2pdu.c +12 -3
@@ -4484,6 +4484,16 @@ smb2_new_read_req(void **buf, unsigned int *total_len,return rc; } +static void smb2_readv_worker (struct work_struct *work) +{ +struct cifs_io_subrequest *rdata = + container_of(work, struct cifs_io_subrequest, subreq.work); + +netfs_subreq_terminated (&rdata->subreq, + (rdata->result ==0 || rdata->result == -EAGAIN ) ? + rdata->got_bytes : rdata->result, true); +} +static void smb2_readv_callback (struct mid_q_entry *mid) {
@@ -4578,9 +4588,8 @@ smb2_readv_callback(struct mid_q_entry *mid) rdata->result =0 ; } rdata->credits.value =0 ; -netfs_subreq_terminated (&rdata->subreq, - (rdata->result ==0 || rdata->result == -EAGAIN ) ? - rdata->got_bytes : rdata->result, true); +INIT_WORK (&rdata->subreq.work, smb2_readv_worker); +queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid);add_credits (server, &credits,0 ); }
include/trace/events/netfs.h +5 -2
@@ -412,6 +412,7 @@ TRACE_EVENT(netfs_folio, __field(ino_t, ino) __field(pgoff_t, index) __field(unsigned int, nr) + __field(bool, ra_trigger) __field(enum netfs_folio_trace, why) ),
@@ -420,11 +421,13 @@ TRACE_EVENT(netfs_folio, __entry->why = why; __entry->index =folio_index (folio); __entry->nr =folio_nr_pages (folio); + __entry->ra_trigger =folio_test_readahead (folio); ), -TP_printk ("i=%05lx ix=%05lx-%05lx %s" , +TP_printk ("i=%05lx ix=%05lx-%05lx %s%s" , __entry->ino, __entry->index, __entry->index + __entry->nr -1 , -__print_symbolic (__entry->why, netfs_folio_traces)) +__print_symbolic (__entry->why, netfs_folio_traces), + __entry->ra_trigger ?" *RA*" :"" ) ); TRACE_EVENT(netfs_write_iter,
David Howells
maintainerPATCH v2 9/17 cifs: Defer read completion
Oops - a bit of debugging code had got left in this patch and I forgot to fill out the description. Updated patch attached.
David
---
cifs: Defer read completion
Defer read completion from the I/O thread to the cifsiod thread so as not to slow down the I/O thread. This restores the behaviour of v6.9.
--- fs/smb/client/smb2pdu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
fs/smb/client/smb2pdu.c +12 -3
@@ -4484,6 +4484,16 @@ smb2_new_read_req(void **buf, unsigned int *total_len,return rc; } +static void smb2_readv_worker (struct work_struct *work) +{ +struct cifs_io_subrequest *rdata = + container_of(work, struct cifs_io_subrequest, subreq.work); + +netfs_subreq_terminated (&rdata->subreq, + (rdata->result ==0 || rdata->result == -EAGAIN ) ? + rdata->got_bytes : rdata->result, true); +} +static void smb2_readv_callback (struct mid_q_entry *mid) {
@@ -4578,9 +4588,8 @@ smb2_readv_callback(struct mid_q_entry *mid) rdata->result =0 ; } rdata->credits.value =0 ; -netfs_subreq_terminated (&rdata->subreq, - (rdata->result ==0 || rdata->result == -EAGAIN ) ? - rdata->got_bytes : rdata->result, true); +INIT_WORK (&rdata->subreq.work, smb2_readv_worker); +queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid);add_credits (server, &credits,0 ); }
David Howells
maintainerPATCH 10/17 cifs: Only pick a channel once per read request
In reply to David HowellsIn cifs, only pick a channel when setting up a read request rather than doing so individually for every subrequest and instead use that channel for all. This mirrors what the code in v6.9 does.
--- fs/smb/client/cifsglob.h | 1 + fs/smb/client/file.c | 14 +++----------- 2 files changed, 4 insertions(+), 11 deletions(-)
fs/smb/client/cifsglob.h +1 -0
@@ -1494,6 +1494,7 @@ struct cifs_aio_ctx {struct cifs_io_request {struct netfs_io_request rreq;struct cifsFileInfo *cfile; +struct TCP_Server_Info *server; };/* asynchronous read support */
fs/smb/client/file.c +3 -11
@@ -134,17 +134,15 @@ static void cifs_issue_write(struct netfs_io_subrequest *subreq)static bool cifs_clamp_length (struct netfs_io_subrequest *subreq) {struct netfs_io_request *rreq = subreq->rreq; -struct TCP_Server_Info *server; struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq); +struct TCP_Server_Info *server = req->server;struct cifs_sb_info *cifs_sb =CIFS_SB (rreq->inode->i_sb);size_t rsize =0 ;int rc; rdata->xid =get_xid (); rdata->have_xid = true; - - server =cifs_pick_channel (tlink_tcon (req->cfile->tlink)->ses); rdata->server = server;if (cifs_sb->ctx->rsize ==0 )
@@ -203,14 +201,7 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq)__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); rdata->pid = pid; - rc =adjust_credits (rdata->server, &rdata->credits, rdata->subreq.len); -if (!rc) { -if (rdata->req->cfile->invalidHandle) - rc = -EAGAIN ; -else - rc = rdata->server->ops->async_readv (rdata); - } - + rc = rdata->server->ops->async_readv (rdata); out:if (rc)netfs_subreq_terminated (subreq, rc, false);
@@ -250,6 +241,7 @@ static int cifs_init_request(struct netfs_io_request *rreq, struct file *file) open_file = file->private_data; rreq->netfs_priv = file->private_data; req->cfile =cifsFileInfo_get (open_file); + req->server =cifs_pick_channel (tlink_tcon (req->cfile->tlink)->ses); }else if (rreq->origin !=NETFS_WRITEBACK ) {WARN_ON_ONCE (1 );return -EIO ;
David Howells
maintainerPATCH 11/17 cifs: Move the 'pid' from the subreq to the req
In reply to David HowellsMove the reference pid from the cifs_io_subrequest struct to the cifs_io_request struct as it's the same for all subreqs of a particular request.
--- fs/smb/client/cifsglob.h | 2 +- fs/smb/client/cifssmb.c | 8 ++++---- fs/smb/client/file.c | 10 +++------- fs/smb/client/smb2pdu.c | 4 ++-- 4 files changed, 10 insertions(+), 14 deletions(-)
fs/smb/client/cifsglob.h +1 -1
@@ -1495,6 +1495,7 @@ struct cifs_io_request {struct netfs_io_request rreq;struct cifsFileInfo *cfile;struct TCP_Server_Info *server; +pid_t pid; };/* asynchronous read support */
@@ -1505,7 +1506,6 @@ struct cifs_io_subrequest {struct cifs_io_request *req; };ssize_t got_bytes; -pid_t pid;unsigned int xid;int result;bool have_xid;
fs/smb/client/cifssmb.c +4 -4
@@ -1345,8 +1345,8 @@ cifs_async_readv(struct cifs_io_subrequest *rdata)if (rc)return rc; - smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = rdata->req->cfile->fid.netfid;
@@ -1689,8 +1689,8 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)if (rc) goto async_writev_out; - smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = wdata->req->cfile->fid.netfid;
fs/smb/client/file.c +3 -7
@@ -178,14 +178,8 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq) struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);struct cifs_sb_info *cifs_sb =CIFS_SB (rreq->inode->i_sb); -pid_t pid;int rc =0 ; -if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) - pid = req->cfile->pid; -else - pid = current->tgid;// Ummm... This may be a workqueue -cifs_dbg (FYI ,"%s: op=%08x[%x] mapping=%p len=%zu/%zu\n" , __func__, rreq->debug_id, subreq->debug_index, rreq->mapping, subreq->transferred, subreq->len);
@@ -199,7 +193,6 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq) }__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); - rdata->pid = pid; rc = rdata->server->ops->async_readv (rdata); out:
@@ -236,12 +229,15 @@ static int cifs_init_request(struct netfs_io_request *rreq, struct file *file) rreq->rsize = cifs_sb->ctx->rsize; rreq->wsize = cifs_sb->ctx->wsize; + req->pid = current->tgid;// Ummm... This may be a workqueue if (file) { open_file = file->private_data; rreq->netfs_priv = file->private_data; req->cfile =cifsFileInfo_get (open_file); req->server =cifs_pick_channel (tlink_tcon (req->cfile->tlink)->ses); +if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) + req->pid = req->cfile->pid; }else if (rreq->origin !=NETFS_WRITEBACK ) {WARN_ON_ONCE (1 );return -EIO ;
fs/smb/client/smb2pdu.c +2 -2
@@ -4621,7 +4621,7 @@ smb2_async_readv(struct cifs_io_subrequest *rdata) io_parms.length = rdata->subreq.len; io_parms.persistent_fid = rdata->req->cfile->fid.persistent_fid; io_parms.volatile_fid = rdata->req->cfile->fid.volatile_fid; - io_parms.pid = rdata->pid; + io_parms.pid = rdata->req->pid; rc =smb2_new_read_req ( (void **) &buf, &total_len, &io_parms, rdata,0 ,0 );
@@ -4873,7 +4873,7 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) .length = wdata->subreq.len, .persistent_fid = wdata->req->cfile->fid.persistent_fid, .volatile_fid = wdata->req->cfile->fid.volatile_fid, - .pid = wdata->pid, + .pid = wdata->req->pid, }; io_parms = &_io_parms;
Re: [PATCH 11/17] cifs: Move the 'pid' from the subreq to the req
minor update to the patch. Fix function ‘cifs_req_issue_read’ to remove the warning:/home/smfrench/cifs-2.6/fs/smb/client/file.c:180:30: warning: unused variable ‘cifs_sb
See attached.
Merged this and two more from the series tentatively into cifs-2.6.git for-next pending more testing/review
969b3010cbfc cifs: Only pick a channel once per read request ce5291e56081 cifs: Defer read completion
Quoted 131 lines · David Howells
Move the reference pid from the
cifs_io_subrequeststruct to thecifs_io_requeststruct as it's the same for all subreqs of a particular request.Signed-off-by David Howells <[email protected]>cc Steve French <[email protected]>cc Paulo Alcantara <[email protected]>cc Jeff Layton <[email protected]>--- fs/smb/client/cifsglob.h | 2 +- fs/smb/client/cifssmb.c | 8 ++++---- fs/smb/client/file.c | 10 +++------- fs/smb/client/smb2pdu.c | 4 ++-- 4 files changed, 10 insertions(+), 14 deletions(-)fs/smb/client/cifsglob.h +1
-1@@ -1495,6 +1495,7 @@ struct cifs_io_request {struct netfs_io_request rreq;struct cifsFileInfo *cfile;struct TCP_Server_Info *server; +pid_t pid; };/* asynchronous read support */ @@ -1505,7 +1506,6 @@ struct cifs_io_subrequest {struct cifs_io_request *req; };ssize_t got_bytes; -pid_t pid;unsigned int xid;int result;bool have_xid;fs/smb/client/cifssmb.c +4
-4@@ -1345,8 +1345,8 @@ cifs_async_readv(struct cifs_io_subrequest *rdata)if (rc)return rc; - smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = rdata->req->cfile->fid.netfid;@@ -1689,8 +1689,8 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)if (rc) goto async_writev_out; - smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = wdata->req->cfile->fid.netfid;fs/smb/client/file.c +3
-7@@ -178,14 +178,8 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq) struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);struct cifs_sb_info *cifs_sb =CIFS_SB (rreq->inode->i_sb); -pid_t pid;int rc =0 ; -if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) - pid = req->cfile->pid; -else - pid = current->tgid;// Ummm... This may be a workqueue -cifs_dbg (FYI ,"%s: op=%08x[%x] mapping=%p len=%zu/%zu\n" , __func__, rreq->debug_id, subreq->debug_index, rreq->mapping, subreq->transferred, subreq->len);@@ -199,7 +193,6 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq) }__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); - rdata->pid = pid; rc = rdata->server->ops->async_readv (rdata); out:@@ -236,12 +229,15 @@ static int cifs_init_request(struct netfs_io_request *rreq, struct file *file) rreq->rsize = cifs_sb->ctx->rsize; rreq->wsize = cifs_sb->ctx->wsize; + req->pid = current->tgid;// Ummm... This may be a workqueue if (file) { open_file = file->private_data; rreq->netfs_priv = file->private_data; req->cfile =cifsFileInfo_get (open_file); req->server =cifs_pick_channel (tlink_tcon (req->cfile->tlink)->ses); +if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) + req->pid = req->cfile->pid; }else if (rreq->origin !=NETFS_WRITEBACK ) {WARN_ON_ONCE (1 );return -EIO ;fs/smb/client/smb2pdu.c +2
-2@@ -4621,7 +4621,7 @@ smb2_async_readv(struct cifs_io_subrequest *rdata) io_parms.length = rdata->subreq.len; io_parms.persistent_fid = rdata->req->cfile->fid.persistent_fid; io_parms.volatile_fid = rdata->req->cfile->fid.volatile_fid; - io_parms.pid = rdata->pid; + io_parms.pid = rdata->req->pid; rc =smb2_new_read_req ( (void **) &buf, &total_len, &io_parms, rdata,0 ,0 );@@ -4873,7 +4873,7 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) .length = wdata->subreq.len, .persistent_fid = wdata->req->cfile->fid.persistent_fid, .volatile_fid = wdata->req->cfile->fid.volatile_fid, - .pid = wdata->pid, + .pid = wdata->req->pid, }; io_parms = &_io_parms;
Signature · 3 lines
Thanks, Steve
From 3f59138580bf8006fa99641b5803d0f683709f10 Mon Sep 17 00:00:00 2001
From: David Howells <[email protected]>
Date: Thu, 20 Jun 2024 18:31:29 +0100
Subject: [PATCH] cifs: Move the 'pid' from the subreq to the req
Move the reference pid from the cifs_io_subrequest struct to the cifs_io_request struct as it's the same for all subreqs of a particular request.
--- fs/smb/client/cifsglob.h | 2 +- fs/smb/client/cifssmb.c | 8 ++++---- fs/smb/client/file.c | 11 +++-------- fs/smb/client/smb2pdu.c | 4 ++-- 4 files changed, 10 insertions(+), 15 deletions(-)
fs/smb/client/cifsglob.h +1 -1
@@ -1495,6 +1495,7 @@ struct cifs_io_request {struct netfs_io_request rreq;struct cifsFileInfo *cfile;struct TCP_Server_Info *server; +pid_t pid; };/* asynchronous read support */
@@ -1505,7 +1506,6 @@ struct cifs_io_subrequest {struct cifs_io_request *req; };ssize_t got_bytes; -pid_t pid;unsigned int xid;int result;bool have_xid;
fs/smb/client/cifssmb.c +4 -4
@@ -1345,8 +1345,8 @@ cifs_async_readv(struct cifs_io_subrequest *rdata)if (rc)return rc; - smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )rdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(rdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = rdata->req->cfile->fid.netfid;
@@ -1689,8 +1689,8 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)if (rc) goto async_writev_out; - smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->pid); - smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->pid >>16 )); + smb->hdr.Pid =cpu_to_le16 ((__u16 )wdata->req->pid); + smb->hdr.PidHigh =cpu_to_le16 ((__u16)(wdata->req->pid >>16 )); smb->AndXCommand =0xFF ;/* none */ smb->Fid = wdata->req->cfile->fid.netfid;
fs/smb/client/file.c +3 -8
@@ -177,15 +177,8 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq)struct netfs_io_request *rreq = subreq->rreq; struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq); -struct cifs_sb_info *cifs_sb =CIFS_SB (rreq->inode->i_sb); -pid_t pid;int rc =0 ; -if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) - pid = req->cfile->pid; -else - pid = current->tgid;// Ummm... This may be a workqueue -cifs_dbg (FYI ,"%s: op=%08x[%x] mapping=%p len=%zu/%zu\n" , __func__, rreq->debug_id, subreq->debug_index, rreq->mapping, subreq->transferred, subreq->len);
@@ -199,7 +192,6 @@ static void cifs_req_issue_read(struct netfs_io_subrequest *subreq) }__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); - rdata->pid = pid; rc = rdata->server->ops->async_readv (rdata); out:
@@ -236,12 +228,15 @@ static int cifs_init_request(struct netfs_io_request *rreq, struct file *file) rreq->rsize = cifs_sb->ctx->rsize; rreq->wsize = cifs_sb->ctx->wsize; + req->pid = current->tgid;// Ummm... This may be a workqueue if (file) { open_file = file->private_data; rreq->netfs_priv = file->private_data; req->cfile =cifsFileInfo_get (open_file); req->server =cifs_pick_channel (tlink_tcon (req->cfile->tlink)->ses); +if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_RWPIDFORWARD ) + req->pid = req->cfile->pid; }else if (rreq->origin !=NETFS_WRITEBACK ) {WARN_ON_ONCE (1 );return -EIO ;
fs/smb/client/smb2pdu.c +2 -2
@@ -4621,7 +4621,7 @@ smb2_async_readv(struct cifs_io_subrequest *rdata) io_parms.length = rdata->subreq.len; io_parms.persistent_fid = rdata->req->cfile->fid.persistent_fid; io_parms.volatile_fid = rdata->req->cfile->fid.volatile_fid; - io_parms.pid = rdata->pid; + io_parms.pid = rdata->req->pid; rc =smb2_new_read_req ( (void **) &buf, &total_len, &io_parms, rdata,0 ,0 );
@@ -4873,7 +4873,7 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) .length = wdata->subreq.len, .persistent_fid = wdata->req->cfile->fid.persistent_fid, .volatile_fid = wdata->req->cfile->fid.volatile_fid, - .pid = wdata->pid, + .pid = wdata->req->pid, }; io_parms = &_io_parms;
Signature · 1 line
2.43.0
David Howells
maintainerPATCH 12/17 netfs: Move max_len/max_nr_segs from netfs_io_subrequest to netfs_io_stream
In reply to David HowellsMove max_len/max_nr_segs from struct netfs_io_subrequest to struct netfs_io_stream as we only issue one subreq at a time and then don't need these values again for that subreq unless and until we have to retry it - in which case we want to renegotiate them.
--- fs/afs/write.c | 4 +++- fs/cachefiles/io.c | 5 +++-- fs/netfs/io.c | 4 ++-- fs/netfs/write_collect.c | 10 +++++----- fs/netfs/write_issue.c | 14 +++++++------- fs/smb/client/file.c | 15 ++++++++------- include/linux/netfs.h | 4 ++-- 7 files changed, 30 insertions(+), 26 deletions(-)
fs/afs/write.c +3 -1
@@ -89,10 +89,12 @@ static const struct afs_operation_ops afs_store_data_operation = {*/ void afs_prepare_write (struct netfs_io_subrequest *subreq) { +struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr]; +//if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags)) // subreq->max_len = 512 * 1024; //else - subreq->max_len =256 *1024 *1024 ; + stream->sreq_max_len =256 *1024 *1024 ; }/*
fs/cachefiles/io.c +3 -2
@@ -627,11 +627,12 @@ static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq) {struct netfs_io_request *wreq = subreq->rreq;struct netfs_cache_resources *cres = &wreq->cache_resources; +struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];_enter ("W=%x[%x] %llx" , wreq->debug_id, subreq->debug_index, subreq->start); - subreq->max_len =ULONG_MAX ; - subreq->max_nr_segs =BIO_MAX_VECS ; + stream->sreq_max_len =UINT_MAX ; + stream->sreq_max_segs =BIO_MAX_VECS ;if (!cachefiles_cres_file (cres)) {if (!fscache_wait_for_operation (cres,FSCACHE_WANT_WRITE ))
fs/netfs/io.c +2 -2
@@ -469,9 +469,9 @@ netfs_rreq_prepare_read(struct netfs_io_request *rreq, goto out; } -if (subreq->max_nr_segs) { +if (rreq->io_streams[0 ].sreq_max_segs) { lsize =netfs_limit_iter (io_iter,0 , subreq->len, - subreq->max_nr_segs); + rreq->io_streams[0 ].sreq_max_segs);if (subreq->len > lsize) { subreq->len = lsize;trace_netfs_sreq (subreq, netfs_sreq_trace_limited);
fs/netfs/write_collect.c +5 -5
@@ -231,7 +231,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,__set_bit (NETFS_SREQ_RETRYING , &subreq->flags); stream->prepare_write (subreq); - part =min (len, subreq->max_len); + part =min (len, stream->sreq_max_len); subreq->len = part; subreq->start = start; subreq->transferred =0 ;
@@ -271,8 +271,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, subreq =netfs_alloc_subrequest (wreq); subreq->source = to->source; subreq->start = start; - subreq->max_len = len; - subreq->max_nr_segs =INT_MAX ; subreq->debug_index =atomic_inc_return (&wreq->subreq_counter); subreq->stream_nr = to->stream_nr;__set_bit (NETFS_SREQ_RETRYING , &subreq->flags);
@@ -286,10 +284,12 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, to =list_next_entry (to, rreq_link);trace_netfs_sreq (subreq, netfs_sreq_trace_retry); + stream->sreq_max_len = len; + stream->sreq_max_segs =INT_MAX ;switch (stream->source) {case NETFS_UPLOAD_TO_SERVER :netfs_stat (&netfs_n_wh_upload); - subreq->max_len =min (len, wreq->wsize); + stream->sreq_max_len =umin (len, wreq->wsize);break ;case NETFS_WRITE_TO_CACHE :netfs_stat (&netfs_n_wh_write);
@@ -300,7 +300,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, stream->prepare_write (subreq); - part =min (len, subreq->max_len); + part =umin (len, stream->sreq_max_len); subreq->len = subreq->transferred + part; len -= part; start += part;
fs/netfs/write_issue.c +7 -7
@@ -155,8 +155,6 @@ static void netfs_prepare_write(struct netfs_io_request *wreq, subreq =netfs_alloc_subrequest (wreq); subreq->source = stream->source; subreq->start = start; - subreq->max_len =ULONG_MAX ; - subreq->max_nr_segs =INT_MAX ; subreq->stream_nr = stream->stream_nr;_enter ("R=%x[%x]" , wreq->debug_id, subreq->debug_index);
@@ -167,10 +165,12 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,trace_netfs_sreq (subreq, netfs_sreq_trace_prepare); + stream->sreq_max_len =UINT_MAX ; + stream->sreq_max_segs =INT_MAX ;switch (stream->source) {case NETFS_UPLOAD_TO_SERVER :netfs_stat (&netfs_n_wh_upload); - subreq->max_len = wreq->wsize; + stream->sreq_max_len = wreq->wsize;break ;case NETFS_WRITE_TO_CACHE :netfs_stat (&netfs_n_wh_write);
@@ -287,13 +287,13 @@ int netfs_advance_write(struct netfs_io_request *wreq,netfs_prepare_write (wreq, stream, start); subreq = stream->construct; - part =min (subreq->max_len - subreq->len, len); -_debug ("part %zx/%zx %zx/%zx" , subreq->len, subreq->max_len, part, len); + part =umin (stream->sreq_max_len - subreq->len, len); +_debug ("part %zx/%zx %zx/%zx" , subreq->len, stream->sreq_max_len, part, len); subreq->len += part; subreq->nr_segs++; -if (subreq->len >= subreq->max_len || - subreq->nr_segs >= subreq->max_nr_segs || +if (subreq->len >= stream->sreq_max_len || + subreq->nr_segs >= stream->sreq_max_segs || to_eof) {netfs_issue_write (wreq, stream); subreq =NULL ;
fs/smb/client/file.c +8 -7
@@ -49,6 +49,7 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)struct cifs_io_subrequest *wdata = container_of(subreq, struct cifs_io_subrequest, subreq);struct cifs_io_request *req = wdata->req; +struct netfs_io_stream *stream = &req->rreq.io_streams[subreq->stream_nr];struct TCP_Server_Info *server;struct cifsFileInfo *open_file = req->cfile;size_t wsize = req->rreq.wsize;
@@ -73,7 +74,7 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq) } } - rc = server->ops->wait_mtu_credits (server, wsize, &wdata->subreq.max_len, + rc = server->ops->wait_mtu_credits (server, wsize, &stream->sreq_max_len, &wdata->credits);if (rc <0 ) { subreq->error = rc;
@@ -82,7 +83,7 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)#ifdef CONFIG_CIFS_SMB_DIRECT if (server->smbd_conn) - subreq->max_nr_segs = server->smbd_conn->max_frmr_depth; + stream->sreq_max_segs = server->smbd_conn->max_frmr_depth;#endif }
@@ -134,11 +135,11 @@ static void cifs_issue_write(struct netfs_io_subrequest *subreq)static bool cifs_clamp_length (struct netfs_io_subrequest *subreq) {struct netfs_io_request *rreq = subreq->rreq; +struct netfs_io_stream *stream = &rreq->io_streams[subreq->stream_nr]; struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);struct TCP_Server_Info *server = req->server;struct cifs_sb_info *cifs_sb =CIFS_SB (rreq->inode->i_sb); -size_t rsize =0 ;int rc; rdata->xid =get_xid ();
@@ -151,17 +152,17 @@ static bool cifs_clamp_length(struct netfs_io_subrequest *subreq) cifs_sb->ctx); - rc = server->ops->wait_mtu_credits (server, cifs_sb->ctx->rsize, &rsize, - &rdata->credits); + rc = server->ops->wait_mtu_credits (server, cifs_sb->ctx->rsize, + &stream->sreq_max_len, &rdata->credits);if (rc) { subreq->error = rc;return false; } - subreq->len =min_t (size_t, subreq->len, rsize); + subreq->len =umin (subreq->len, stream->sreq_max_len);#ifdef CONFIG_CIFS_SMB_DIRECT if (server->smbd_conn) - subreq->max_nr_segs = server->smbd_conn->max_frmr_depth; + stream->sreq_max_segs = server->smbd_conn->max_frmr_depth;#endif return true; }
include/linux/netfs.h +2 -2
@@ -136,6 +136,8 @@ static inline struct netfs_group *netfs_folio_group(struct folio *folio)struct netfs_io_stream {/* Submission tracking */ struct netfs_io_subrequest *construct;/* Op being constructed */ +size_t sreq_max_len;/* Maximum size of a subrequest */ +unsigned int sreq_max_segs;/* 0 or max number of segments in an iterator */ unsigned int submit_off;/* Folio offset we're submitting from */ unsigned int submit_len;/* Amount of data left to submit */ unsigned int submit_max_len;/* Amount I/O can be rounded up to */
@@ -179,14 +181,12 @@ struct netfs_io_subrequest {struct list_head rreq_link;/* Link in rreq->subrequests */ struct iov_iter io_iter;/* Iterator for this subrequest */ unsigned long long start;/* Where to start the I/O */ -size_t max_len;/* Maximum size of the I/O */ size_t len;/* Size of the I/O */ size_t transferred;/* Amount of data transferred */ refcount_t ref;short error;/* 0 or error that occurred */ unsigned short debug_index;/* Index in list (for debugging output) */ unsigned int nr_segs;/* Number of segs in io_iter */ -unsigned int max_nr_segs;/* 0 or max number of segments in an iterator */ enum netfs_io_source source;/* Where to read from/write to */ unsigned char stream_nr;/* I/O stream this belongs to */ unsigned long flags;
David Howells
maintainerPATCH 13/17 mm: Define struct sheaf and ITER_SHEAF to handle a sequence of folios
In reply to David HowellsDefine a data structure, struct sheaf, to represent a sequence of folios and a kernel-internal I/O iterator type, ITER_SHEAF, to allow a list of sheaf structures to be used to provide a buffer to iov_iter-taking functions, such as sendmsg and recvmsg.
The sheaf structure looks like:
struct sheaf {
struct sheaf *next;
struct sheaf *prev;
sheaf_slot_t slots[29];
u8 orders[29];
};It does not use a list_head so that next and/or prev can be set to NULL at the ends of the list, allowing iov_iter-handling routines to determine that they *are* the ends without needing to store the head pointer in the iov_iter struct.
The slots are a folio pointer with the bottom two bits usable for storing marks. The intention is to use at least one of them to mark folios that need putting, but that might not be ultimately necessary. Accessor functions are used to access the slots to do the masking and an additional accessor function is used to indicate the size of the array.
The order of each folio is also stored in the structure to avoid the need for iov_iter_advance() and iov_iter_revert() to have to query each folio to find its size.
With careful barriering, this can be used as an extending buffer with new folios inserted and new sheaf structs added without the need for a lock. Further, provided we always keep at least one struct in the buffer, we can also remove consumed folios and consumed structs from the head end as we without the need for locks.
[Questions/thoughts]
(1) I'm told that 'sheaf' is now being used for something else mm-related out-of-tree, so I should probably change the name to something else - but what? Continuing on the papery theme, "scroll" and "fanfold" have occurred to me, but they're probably not the best. Maybe "folio_queue" or "folio_seq"?
(2) To manage this, I need a head pointer, a tail pointer, a tail slot number (assuming insertion happens at the tail end and the next pointers point from head to tail). Should I put these into a struct of their own, say "sheaf_list" or "rolling_buffer" or "folio_queue"?
I will end up with two of these in netfs_io_request eventually, one keeping track of the pagecache I'm dealing with for buffered I/O and the other to hold a bounce buffer when we need one.
(3) I have set the number of elements to 29 so that it fits into a 32-word allocation (either 256 bytes or 512 bytes). However, should I replace slots[] with a folio_batch struct?
The disadvantage of that is that adds a bunch of fields I don't need for the purpose of the buffer and pushes the size of the struct to a non-pow-2 size. The advantage would be that I can pass the batch directly to a variety of mm routines.
(4) Should I make the slots {folio,off,len} or bio_vec?
(5) This is intended to replace ITER_XARRAY eventually. Using an xarray in I/O iteration requires the taking of the RCU read lock, doing copying under the RCU read lock, walking the xarray (which may change under us), handling retries and dealing with special values.
The advantage of ITER_XARRAY is that when we're dealing with the pagecache directly, we don't need any allocation - but if we're doing encrypted comms, there's a good chance we'd be using a bounce buffer anyway.
This will require afs, erofs, cifs and fscache to be converted to not use this. afs still uses it for dirs and symlinks; some of erofs usages should be easy to change, but there's one which won't be so easy; ceph's use via fscache can be fixed by porting ceph to netfslib; cifs is using xarray as a bounce buffer - that can be moved to use sheaves instead; and orangefs has a similar problem to erofs - maybe orangefs could use netfslib?
--- include/linux/iov_iter.h | 57 +++++++++ include/linux/sheaf.h | 83 +++++++++++++ include/linux/uio.h | 11 ++ lib/iov_iter.c | 222 ++++++++++++++++++++++++++++++++- lib/kunit_iov_iter.c | 259 +++++++++++++++++++++++++++++++++++++++ lib/scatterlist.c | 69 ++++++++++- 6 files changed, 697 insertions(+), 4 deletions(-) create mode 100644 include/linux/sheaf.h
include/linux/iov_iter.h +57 -0
@@ -10,6 +10,7 @@#include <linux/uio.h> #include <linux/bvec.h> +#include <linux/sheaf.h> typedef size_t (*iov_step_f )(void *iter_base,size_t progress,size_t len,void *priv,void *priv2);
@@ -140,6 +141,60 @@ size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2,return progress; } +/* +* Handle ITER_SHEAF. +*/ +static __always_inline +size_t iterate_sheaf(struct iov_iter *iter, size_t len, void *priv, void *priv2, +iov_step_f step) +{ +const struct sheaf *sheaf = iter->sheaf; +unsigned int slot = iter->sheaf_slot; +size_t progress =0 , skip = iter->iov_offset; + +if (slot ==sheaf_nr_slots (sheaf)) { +/* The iterator may have been extended. */ + sheaf = sheaf->next; + slot =0 ; + } + +do { +struct folio *folio =sheaf_slot_folio (sheaf, slot); +size_t part, remain, consumed; +size_t fsize; +void *base; + +if (!folio) +break ; + + fsize =sheaf_folio_size (sheaf, slot); + base =kmap_local_folio (folio, skip); + part =umin (len,PAGE_SIZE - skip %PAGE_SIZE ); + remain =step (base, progress, part, priv, priv2); +kunmap_local (base); + consumed = part - remain; + len -= consumed; + progress += consumed; + skip += consumed; +if (skip >= fsize) { + skip =0 ; + slot++; +if (slot ==sheaf_nr_slots (sheaf) && sheaf->next) { + sheaf = sheaf->next; + slot =0 ; + } + } +if (remain) +break ; + }while (len); + + iter->sheaf_slot = slot; + iter->sheaf = sheaf; + iter->iov_offset = skip; + iter->count -= progress; +return progress; +} +/* * Handle ITER_XARRAY. */
@@ -249,6 +304,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,return iterate_bvec (iter, len, priv, priv2, step);if (iov_iter_is_kvec (iter))return iterate_kvec (iter, len, priv, priv2, step); +if (iov_iter_is_sheaf (iter)) +return iterate_sheaf (iter, len, priv, priv2, step);if (iov_iter_is_xarray (iter))return iterate_xarray (iter, len, priv, priv2, step);return iterate_discard (iter, len, priv, priv2, step);
include/linux/sheaf.h new file +83 -0
83 lines not on this page · raw message
include/linux/uio.h +11 -0
@@ -11,6 +11,7 @@#include <uapi/linux/uio.h> struct page ; +struct sheaf ;typedef unsigned int __bitwise iov_iter_extraction_t ;
@@ -25,6 +26,7 @@ enum iter_type {ITER_IOVEC ,ITER_BVEC ,ITER_KVEC , +ITER_SHEAF ,ITER_XARRAY , ITER_DISCARD, };
@@ -66,6 +68,7 @@ struct iov_iter {const struct iovec *__iov;const struct kvec *kvec;const struct bio_vec *bvec; +const struct sheaf *sheaf;struct xarray *xarray;void __user *ubuf; };
@@ -74,6 +77,7 @@ struct iov_iter { };union {unsigned long nr_segs; +u8 sheaf_slot;loff_t xarray_start; }; };
@@ -126,6 +130,11 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i)return iov_iter_type (i) ==ITER_DISCARD ; } +static inline bool iov_iter_is_sheaf (const struct iov_iter *i) +{ +return iov_iter_type (i) ==ITER_SHEAF ; +} +static inline bool iov_iter_is_xarray (const struct iov_iter *i) {return iov_iter_type (i) ==ITER_XARRAY ;
@@ -273,6 +282,8 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvecvoid iov_iter_bvec (struct iov_iter *i,unsigned int direction,const struct bio_vec *bvec,unsigned long nr_segs,size_t count);void iov_iter_discard (struct iov_iter *i,unsigned int direction,size_t count); +void iov_iter_sheaf (struct iov_iter *i,unsigned int direction,const struct sheaf *sheaf, +unsigned int first_slot,unsigned int offset,size_t count);void iov_iter_xarray (struct iov_iter *i,unsigned int direction,struct xarray *xarray,loff_t start,size_t count); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
lib/iov_iter.c +220 -2
312 lines not on this page · raw message
lib/kunit_iov_iter.c +259 -0
293 lines not on this page · raw message
lib/scatterlist.c +67 -2
93 lines not on this page · raw message
David Howells
maintainerPATCH 14/17 netfs: Use new sheaf data type and iterator instead of xarray iter
In reply to David HowellsMake the netfs write-side routines use the new sheaf struct to hold a rolling buffer of folios, with the issuer adding folios at the tail and the collector removing them from the head as they're processed instead of using an xarray.
This will allow a subsequent patch to simplify the write collector.
--- fs/netfs/internal.h | 9 ++++- fs/netfs/misc.c | 74 +++++++++++++++++++++++++++++++++++ fs/netfs/objects.c | 1 + fs/netfs/stats.c | 4 +- fs/netfs/write_collect.c | 84 +++++++++++++++++++++------------------- fs/netfs/write_issue.c | 28 +++++++------- include/linux/netfs.h | 4 ++ 7 files changed, 147 insertions(+), 57 deletions(-)
fs/netfs/internal.h +8 -1
@@ -7,6 +7,7 @@#include <linux/slab.h> #include <linux/seq_file.h> +#include <linux/sheaf.h> #include <linux/netfs.h> #include <linux/fscache.h> #include <linux/fscache-cache.h>
@@ -63,6 +64,10 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}/* * misc.c */ +int netfs_buffer_append_folio (struct netfs_io_request *rreq,struct folio *folio, +bool needs_put); +struct sheaf *netfs_delete_buffer_head (struct netfs_io_request *wreq); +void netfs_clear_buffer (struct netfs_io_request *rreq);/* * objects.c
@@ -119,6 +124,7 @@ extern atomic_t netfs_n_wh_write_done;extern atomic_t netfs_n_wh_write_failed;extern atomic_t netfs_n_wb_lock_skip;extern atomic_t netfs_n_wb_lock_wait; +extern atomic_t netfs_n_sheaf;int netfs_stats_show (struct seq_file *m,void *v);
@@ -152,7 +158,8 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,loff_t start, enum netfs_io_origin origin);void netfs_reissue_write (struct netfs_io_stream *stream, -struct netfs_io_subrequest *subreq); +struct netfs_io_subrequest *subreq, +struct iov_iter *source);int netfs_advance_write (struct netfs_io_request *wreq,struct netfs_io_stream *stream,loff_t start,size_t len,bool to_eof);
fs/netfs/misc.c +74 -0
80 lines not on this page · raw message
fs/netfs/objects.c +1 -0
@@ -144,6 +144,7 @@ static void netfs_free_request(struct work_struct *work) }kvfree (rreq->direct_bv); } +netfs_clear_buffer (rreq);if (atomic_dec_and_test (&ictx->io_count))wake_up_var (&ictx->io_count);
fs/netfs/stats.c +3 -1
@@ -41,6 +41,7 @@ atomic_t netfs_n_wh_write_done;atomic_t netfs_n_wh_write_failed;atomic_t netfs_n_wb_lock_skip;atomic_t netfs_n_wb_lock_wait; +atomic_t netfs_n_sheaf;int netfs_stats_show (struct seq_file *m,void *v) {
@@ -76,9 +77,10 @@ int netfs_stats_show(struct seq_file *m, void *v)atomic_read (&netfs_n_wh_write),atomic_read (&netfs_n_wh_write_done), atomic_read(&netfs_n_wh_write_failed)); -seq_printf (m,"Objs : rr=%u sr=%u wsc=%u\n" , +seq_printf (m,"Objs : rr=%u sr=%u shf=%u wsc=%u\n" ,atomic_read (&netfs_n_rh_rreq),atomic_read (&netfs_n_rh_sreq), +atomic_read (&netfs_n_sheaf),atomic_read (&netfs_n_wh_wstream_conflict)); seq_printf(m, "WbLock : skip=%u wait=%u\n", atomic_read(&netfs_n_wb_lock_skip),
fs/netfs/write_collect.c +44 -40
143 lines not on this page · raw message
fs/netfs/write_issue.c +13 -15
@@ -211,9 +211,11 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,* netfs_write_subrequest_terminated() when complete. */ static void netfs_do_issue_write (struct netfs_io_stream *stream, -struct netfs_io_subrequest *subreq) +struct netfs_io_subrequest *subreq, +struct iov_iter *source) {struct netfs_io_request *wreq = subreq->rreq; +size_t size = subreq->len - subreq->transferred;_enter ("R=%x[%x],%zx" , wreq->debug_id, subreq->debug_index, subreq->len);
@@ -221,27 +223,20 @@ static void netfs_do_issue_write(struct netfs_io_stream *stream,return netfs_write_subrequest_terminated (subreq, subreq->error, false);// TODO: Use encrypted buffer -if (test_bit (NETFS_RREQ_USE_IO_ITER , &wreq->flags)) { - subreq->io_iter = wreq->io_iter; -iov_iter_advance (&subreq->io_iter, - subreq->start + subreq->transferred - wreq->start); -iov_iter_truncate (&subreq->io_iter, - subreq->len - subreq->transferred); - }else { -iov_iter_xarray (&subreq->io_iter,ITER_SOURCE , &wreq->mapping->i_pages, - subreq->start + subreq->transferred, - subreq->len - subreq->transferred); - } + subreq->io_iter = *source; +iov_iter_advance (source, size); +iov_iter_truncate (&subreq->io_iter, size);trace_netfs_sreq (subreq, netfs_sreq_trace_submit); stream->issue_write (subreq); }void netfs_reissue_write (struct netfs_io_stream *stream, -struct netfs_io_subrequest *subreq) +struct netfs_io_subrequest *subreq, +struct iov_iter *source) {__set_bit (NETFS_SREQ_IN_PROGRESS , &subreq->flags); -netfs_do_issue_write (stream, subreq); +netfs_do_issue_write (stream, subreq, source); } static void netfs_issue_write(struct netfs_io_request *wreq,
@@ -255,7 +250,7 @@ static void netfs_issue_write(struct netfs_io_request *wreq,if (subreq->start + subreq->len > wreq->start + wreq->submitted)WRITE_ONCE (wreq->submitted, subreq->start + subreq->len - wreq->start); -netfs_do_issue_write (stream, subreq); +netfs_do_issue_write (stream, subreq, &wreq->io_iter); }/*
@@ -420,6 +415,9 @@ static int netfs_write_folio(struct netfs_io_request *wreq,trace_netfs_folio (folio, netfs_folio_trace_store_plus); } +/* Attach the folio to the rolling buffer. */ +netfs_buffer_append_folio (wreq, folio, false); +/* Move the submission point forward to allow for write-streaming data * not starting at the front of the page. We don't do write-streaming * with the cache as the cache requires DIO alignment.
include/linux/netfs.h +4 -0
@@ -234,6 +234,8 @@ struct netfs_io_request {struct netfs_io_stream io_streams[2 ];/* Streams of parallel I/O operations */ #define NR_IO_STREAMS 2 //wreq->nr_io_streamsstruct netfs_group *group;/* Writeback group being written back */ +struct sheaf *buffer;/* Head of I/O buffer */ +struct sheaf *buffer_tail;/* Tail of I/O buffer */ struct iov_iter iter;/* Unencrypted-side iterator */ struct iov_iter io_iter;/* I/O (Encrypted-side) iterator */ void *netfs_priv;/* Private data for the netfs */
@@ -255,6 +257,8 @@ struct netfs_io_request {short error;/* 0 or error that occurred */ enum netfs_io_origin origin;/* Origin of the request */ bool direct_bv_unpin;/* T if direct_bv[] must be unpinned */ +u8 buffer_head_slot;/* First slot in ->buffer */ +u8 buffer_tail_slot;/* Next slot in ->buffer_tail */ unsigned long long i_size;/* Size of the file */ unsigned long long start;/* Start position */ atomic64_t issued_to;/* Write issuer folio cursor */
David Howells
maintainerPATCH 15/17 netfs: Simplify the writeback code
In reply to David HowellsUse the new sheaf structures to simplify the writeback code. The problem with referring to the i_pages xarray directly is that we may have gaps in the sequence of folios we're writing from that we need to skip when we're removing the writeback mark from the folios we're writing back from.
At the moment the code tries to deal with this by carefully tracking the gaps in each writeback stream (eg. write to server and write to cache) and divining when there's a gap that spans folios (something that's not helped by folios not being a consistent size).
Instead, the sheaf buffer contains pointers only the folios we're dealing with, has them in ascending order and indicates a gap by placing non-consequitive folios next to each other. This makes it possible to track where we need to clean up to by just keeping track of where we've processed to on each stream and taking the minimum.
--- fs/netfs/write_collect.c | 148 ++++++----------------------------- fs/netfs/write_issue.c | 10 +-- include/linux/netfs.h | 1 - include/trace/events/netfs.h | 33 +------- 4 files changed, 31 insertions(+), 161 deletions(-)
fs/netfs/write_collect.c +26 -122
@@ -15,15 +15,11 @@/* Notes made in the collector */ #define HIT_PENDING 0x01/* A front op was still pending */ -#define SOME_EMPTY 0x02/* One of more streams are empty */ -#define ALL_EMPTY 0x04/* All streams are empty */ -#define MAYBE_DISCONTIG 0x08/* A front op may be discontiguous (rounded to PAGE_SIZE) */ -#define NEED_REASSESS 0x10/* Need to loop round and reassess */ -#define REASSESS_DISCONTIG 0x20/* Reassess discontiguity if contiguity advances */ -#define MADE_PROGRESS 0x40/* Made progress cleaning up a stream or the folio set */ -#define BUFFERED 0x80/* The pagecache needs cleaning up */ -#define NEED_RETRY 0x100/* A front op requests retrying */ -#define SAW_FAILURE 0x200/* One stream or hit a permanent failure */ +#define NEED_REASSESS 0x02/* Need to loop round and reassess */ +#define MADE_PROGRESS 0x04/* Made progress cleaning up a stream or the folio set */ +#define BUFFERED 0x08/* The pagecache needs cleaning up */ +#define NEED_RETRY 0x10/* A front op requests retrying */ +#define SAW_FAILURE 0x20/* One stream or hit a permanent failure */ /* * Successful completion of write of a folio to the server and/or cache. Note
@@ -78,10 +74,10 @@ int netfs_folio_written_back(struct folio *folio)* Unlock any folios we've finished with. */ static void netfs_writeback_unlock_folios (struct netfs_io_request *wreq, -unsigned long long collected_to,unsigned int *notes) {struct sheaf *sheaf = wreq->buffer; +unsigned long long collected_to = wreq->collected_to;unsigned int slot = wreq->buffer_head_slot;if (slot >=sheaf_nr_slots (sheaf)) {
@@ -110,12 +106,6 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,trace_netfs_collect_folio (wreq, folio, fend, collected_to); -if (fpos + fsize > wreq->contiguity) { -trace_netfs_collect_contig (wreq, fpos + fsize, - netfs_contig_trace_unlock); - wreq->contiguity = fpos + fsize; - } -/* Unlock any folio we've transferred all of. */ if (collected_to < fend)break ;
@@ -374,7 +364,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) {struct netfs_io_subrequest *front, *remove;struct netfs_io_stream *stream; -unsigned long long collected_to; +unsigned long long collected_to, issued_to;unsigned int notes;int s;
@@ -383,28 +373,21 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)trace_netfs_rreq (wreq, netfs_rreq_trace_collect); reassess_streams: + issued_to =atomic64_read (&wreq->issued_to);smp_rmb (); collected_to =ULLONG_MAX ; -if (wreq->origin ==NETFS_WRITEBACK ) - notes =ALL_EMPTY |BUFFERED |MAYBE_DISCONTIG ; -else if (wreq->origin ==NETFS_WRITETHROUGH ) - notes =ALL_EMPTY |BUFFERED ; +if (wreq->origin ==NETFS_WRITEBACK || + wreq->origin ==NETFS_WRITETHROUGH ) + notes =BUFFERED ;else - notes =ALL_EMPTY ; + notes =0 ;/* Remove completed subrequests from the front of the streams and * advance the completion point on each stream. We stop when we hit * something that's in progress. The issuer thread may be adding stuff * to the tail whilst we're doing this. -* -* We must not, however, merge in discontiguities that span whole -* folios that aren't under writeback. This is made more complicated -* by the folios in the gap being of unpredictable sizes - if they even -* exist - but we don't want to look them up. */ for (s =0 ; s <NR_IO_STREAMS ; s++) { -loff_t rstart, rend; - stream = &wreq->io_streams[s];/* Read active flag before list pointers */ if (!smp_load_acquire (&stream->active))
@@ -416,26 +399,10 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)//_debug("sreq [%x] %llx %zx/%zx", // front->debug_index, front->start, front->transferred, front->len); -/* Stall if there may be a discontinuity. */ - rstart =round_down (front->start,PAGE_SIZE ); -if (rstart > wreq->contiguity) { -if (wreq->contiguity > stream->collected_to) { -trace_netfs_collect_gap (wreq, stream, - wreq->contiguity,'D' ); - stream->collected_to = wreq->contiguity; - } - notes |=REASSESS_DISCONTIG ; -break ; +if (stream->collected_to < front->start) { +trace_netfs_collect_gap (wreq, stream, issued_to,'F' ); + stream->collected_to = front->start; } - rend =round_up (front->start + front->len,PAGE_SIZE ); -if (rend > wreq->contiguity) { -trace_netfs_collect_contig (wreq, rend, - netfs_contig_trace_collect); - wreq->contiguity = rend; -if (notes &REASSESS_DISCONTIG ) - notes |=NEED_REASSESS ; - } - notes &= ~MAYBE_DISCONTIG ;/* Stall if the front is still undergoing I/O. */ if (test_bit (NETFS_SREQ_IN_PROGRESS , &front->flags)) {
@@ -477,15 +444,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) front =list_first_entry_or_null (&stream->subrequests, struct netfs_io_subrequest, rreq_link); stream->front = front; -if (!front) { -unsigned long long jump_to =atomic64_read (&wreq->issued_to); - -if (stream->collected_to < jump_to) { -trace_netfs_collect_gap (wreq, stream, jump_to,'A' ); - stream->collected_to = jump_to; - } - } -spin_unlock (&wreq->lock); netfs_put_subrequest(remove, false, notes & SAW_FAILURE ?
@@ -493,11 +451,16 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq) netfs_sreq_trace_put_done); } -if (front) - notes &= ~ALL_EMPTY ; -else - notes |=SOME_EMPTY ; +/* If we have an empty stream, we need to jump it forward +* otherwise the collection point will never advance. +*/ +if (!front && issued_to > stream->collected_to) { +trace_netfs_collect_gap (wreq, stream, issued_to,'E' ); + stream->collected_to = issued_to; + } +if (list_empty (&stream->subrequests)) + stream->collected_to = issued_to;if (stream->collected_to < collected_to) collected_to = stream->collected_to; }
@@ -505,36 +468,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)if (collected_to !=ULLONG_MAX && collected_to > wreq->collected_to) wreq->collected_to = collected_to; -/* If we have an empty stream, we need to jump it forward over any gap -* otherwise the collection point will never advance. -* -* Note that the issuer always adds to the stream with the lowest -* so-far submitted start, so if we see two consecutive subreqs in one -* stream with nothing between then in another stream, then the second -* stream has a gap that can be jumped. -*/ -if (notes &SOME_EMPTY ) { -unsigned long long jump_to = wreq->start +READ_ONCE (wreq->submitted); - -for (s =0 ; s <NR_IO_STREAMS ; s++) { - stream = &wreq->io_streams[s]; -if (stream->active && - stream->front && - stream->front->start < jump_to) - jump_to = stream->front->start; - } - -for (s =0 ; s <NR_IO_STREAMS ; s++) { - stream = &wreq->io_streams[s]; -if (stream->active && - !stream->front && - stream->collected_to < jump_to) { -trace_netfs_collect_gap (wreq, stream, jump_to,'B' ); - stream->collected_to = jump_to; - } - } - } -for (s =0 ; s <NR_IO_STREAMS ; s++) { stream = &wreq->io_streams[s];if (stream->active)
@@ -545,43 +478,14 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)/* Unlock any folios that we have now finished with. */ if (notes &BUFFERED ) { -unsigned long long clean_to =min (wreq->collected_to, wreq->contiguity); - -if (wreq->cleaned_to < clean_to) -netfs_writeback_unlock_folios (wreq, clean_to, ¬es); +if (wreq->cleaned_to < wreq->collected_to) +netfs_writeback_unlock_folios (wreq, ¬es); }else { wreq->cleaned_to = wreq->collected_to; }// TODO: Discard encryption buffers -/* If all streams are discontiguous with the last folio we cleared, we -* may need to skip a set of folios. -*/ -if ((notes & (MAYBE_DISCONTIG |ALL_EMPTY )) ==MAYBE_DISCONTIG ) { -unsigned long long jump_to =ULLONG_MAX ; - -for (s =0 ; s <NR_IO_STREAMS ; s++) { - stream = &wreq->io_streams[s]; -if (stream->active && stream->front && - stream->front->start < jump_to) - jump_to = stream->front->start; - } - -trace_netfs_collect_contig (wreq, jump_to, netfs_contig_trace_jump); - wreq->contiguity = jump_to; - wreq->cleaned_to = jump_to; - wreq->collected_to = jump_to; -for (s =0 ; s <NR_IO_STREAMS ; s++) { - stream = &wreq->io_streams[s]; -if (stream->collected_to < jump_to) - stream->collected_to = jump_to; - } -//cond_resched(); - notes |=MADE_PROGRESS ; - goto reassess_streams; - } -if (notes &NEED_RETRY ) goto need_retry;if ((notes &MADE_PROGRESS ) &&test_bit (NETFS_RREQ_PAUSE , &wreq->flags)) {
fs/netfs/write_issue.c +3 -7
@@ -105,7 +105,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,if (test_bit (NETFS_RREQ_WRITE_TO_CACHE , &wreq->flags))fscache_begin_write_operation (&wreq->cache_resources,netfs_i_cookie (ictx)); - wreq->contiguity = wreq->start; wreq->cleaned_to = wreq->start;INIT_WORK (&wreq->work, netfs_write_collection_worker);
@@ -247,9 +246,6 @@ static void netfs_issue_write(struct netfs_io_request *wreq,if (!subreq)return ; stream->construct =NULL ; - -if (subreq->start + subreq->len > wreq->start + wreq->submitted) -WRITE_ONCE (wreq->submitted, subreq->start + subreq->len - wreq->start);netfs_do_issue_write (stream, subreq, &wreq->io_iter); }
@@ -463,9 +459,9 @@ static int netfs_write_folio(struct netfs_io_request *wreq,break ; stream = &wreq->io_streams[choose_s]; +atomic64_set (&wreq->issued_to, fpos + stream->submit_off); part =netfs_advance_write (wreq, stream, fpos + stream->submit_off, stream->submit_len, to_eof); -atomic64_set (&wreq->issued_to, fpos + stream->submit_off); stream->submit_off += part; stream->submit_max_len -= part;if (part > stream->submit_len)
@@ -524,10 +520,10 @@ int netfs_writepages(struct address_space *mapping,netfs_stat (&netfs_n_wh_writepages); do { - _debug("wbiter %lx %llx", folio->index, wreq->start + wreq->submitted); + _debug("wbiter %lx %llx", folio->index, atomic64_read(&wreq->issued_to)); /* It appears we don't have to handle cyclic writeback wrapping. */ - WARN_ON_ONCE(wreq && folio_pos(folio) < wreq->start + wreq->submitted); + WARN_ON_ONCE(wreq && folio_pos(folio) < atomic64_read(&wreq->issued_to)); if (netfs_folio_group(folio) != NETFS_FOLIO_COPY_TO_CACHE && unlikely(!test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))) {
include/linux/netfs.h +0 -1
@@ -262,7 +262,6 @@ struct netfs_io_request {unsigned long long i_size;/* Size of the file */ unsigned long long start;/* Start position */ atomic64_t issued_to;/* Write issuer folio cursor */ -unsigned long long contiguity;/* Tracking for gaps in the writeback sequence */ unsigned long long collected_to;/* Point we've collected to */ unsigned long long cleaned_to;/* Position we've cleaned folios to */ pgoff_t no_unlock_folio;/* Don't unlock this folio after read */
include/trace/events/netfs.h +2 -31
@@ -512,33 +512,6 @@ TRACE_EVENT(netfs_collect, __entry->start + __entry->len) ); -TRACE_EVENT (netfs_collect_contig, - TP_PROTO(const struct netfs_io_request *wreq, unsigned long long to, - enum netfs_collect_contig_trace type), - -TP_ARGS (wreq, to, type), - -TP_STRUCT__entry ( - __field(unsigned int, wreq) - __field(enum netfs_collect_contig_trace, type) - __field(unsigned long long, contiguity) - __field(unsigned long long, to) - ), - -TP_fast_assign ( - __entry->wreq = wreq->debug_id; - __entry->type = type; - __entry->contiguity = wreq->contiguity; - __entry->to = to; - ), - -TP_printk ("R=%08x %llx -> %llx %s" , - __entry->wreq, - __entry->contiguity, - __entry->to, -__print_symbolic (__entry->type, netfs_collect_contig_traces)) - ); - TRACE_EVENT(netfs_collect_sreq, TP_PROTO(const struct netfs_io_request *wreq, const struct netfs_io_subrequest *subreq),
@@ -610,7 +583,6 @@ TRACE_EVENT(netfs_collect_state, __field(unsigned int, notes ) __field(unsigned long long, collected_to ) __field(unsigned long long, cleaned_to ) -__field (unsigned long long , contiguity ) ), TP_fast_assign(
@@ -618,12 +590,11 @@ TRACE_EVENT(netfs_collect_state, __entry->notes = notes; __entry->collected_to = collected_to; __entry->cleaned_to = wreq->cleaned_to; - __entry->contiguity = wreq->contiguity; ), -TP_printk ("R=%08x cto=%llx fto=%llx ctg=%llx n=%x" , +TP_printk ("R=%08x col=%llx cln=%llx n=%x" , __entry->wreq, __entry->collected_to, - __entry->cleaned_to, __entry->contiguity, + __entry->cleaned_to, __entry->notes) );
David Howells
maintainerPATCH 16/17 afs: Make read subreqs async
In reply to David HowellsPerform AFS read subrequests in a work item rather than in the calling thread. For normal buffered reads, this will allow the calling thread to copy data from the pagecache to the application at the same time as the demarshalling thread is shovelling data from skbuffs into the pagecache.
This will also allow the RA mark to trigger a new read before we've finished shovelling the data from the current one.
Note: This would be a bit safer if the FS.FetchData RPC ops returned the metadata (including the data version number) before returning the data. This would allow me to flush the pagecache before installing the new data.
In future, it may be possible to asynchronously flush the pagecache either side of the region being read.
--- fs/afs/file.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/afs/file.c +8 -1
@@ -304,8 +304,9 @@ int afs_fetch_data(struct afs_vnode *vnode, struct afs_read *req)return afs_do_sync_operation (op); } -static void afs_issue_read (struct netfs_io_subrequest *subreq) +static void afs_read_worker (struct work_struct *work) { + struct netfs_io_subrequest *subreq = container_of(work, struct netfs_io_subrequest, work);struct afs_vnode *vnode =AFS_FS_I (subreq->rreq->inode);struct afs_read *fsreq;
@@ -324,6 +325,12 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)afs_put_read (fsreq); } +static void afs_issue_read (struct netfs_io_subrequest *subreq) +{ +INIT_WORK (&subreq->work, afs_read_worker); +queue_work (system_long_wq, &subreq->work); +} +static int afs_symlink_read_folio (struct file *file,struct folio *folio) {struct afs_vnode *vnode =AFS_FS_I (folio->mapping->host);
David Howells
maintainerPATCH 17/17 netfs: Speed up buffered reading
In reply to David Howells[!!!] NOTE: THIS PATCH IS INCOMPLETE! Buffered reading mostly works, but
unbuffered/direct reads don't. Further, I have not yet re-implemented read-retry or tested local caching.
Improve the efficiency of buffered reads in a number of ways:
(1) Overhaul the algorithm in general so that it's a lot more compact and split the read submission code between buffered and unbuffered versions. The unbuffered version can be vastly simplified.
(2) Get rid of ->clamp_length(), instead calling into ->issue_read() and having it return the size of the slice issued. This gets rid of a function pointer.
(3) After determining the size of the slice it wants, ->issue_read() must call netfs_prepare_read_iterator() to load more folios into the buffer (if necessary) and to set the iterators. This allows some of the work to be done whilst I/O is in progress.
(4) netfs_subreq_terminated(), which was used to report termination, is replaced with a function, netfs_read_subreq_progress(), that can be used to report incomplete progress as well as termination.
afs can then use this to start unlocking pages as it fills them in as its transport has a packetised approach to RPCs rather than complete smaller messages favoured by, say, cifs.
(5) Read-result collection is handed off to a work queue rather than being done in the I/O thread. Multiple subrequests can be processes simultaneously.
(6) When a subrequest is collected, any folios it fully spans are collected and "spare" data on either side is donated to either the previous or the next subrequest in the sequence.
Notes:
(*) Readahead expansion is currently not working and needs investigation.
(*) Unbuffered/direct-I/O reads don't work and need debugging.
(*) Caching is untested and may not work.
(*) Failed or partial reads need retrying, but aren't yet.
(*) RDMA with cifs does appear to work, both with SIW and RXE.
--- fs/9p/vfs_addr.c | 13 +- fs/afs/file.c | 18 +- fs/afs/fsclient.c | 9 +- fs/afs/yfsclient.c | 9 +- fs/ceph/addr.c | 72 ++--- fs/netfs/Makefile | 2 +- fs/netfs/buffered_read.c | 510 +++++++++++++++++++++------------ fs/netfs/direct_read.c | 99 ++++++- fs/netfs/internal.h | 6 - fs/netfs/io.c | 528 +---------------------------------- fs/netfs/iterator.c | 50 ++++ fs/netfs/main.c | 7 +- fs/netfs/objects.c | 1 - fs/netfs/read_collect.c | 450 +++++++++++++++++++++++++++++ fs/netfs/write_issue.c | 4 - fs/nfs/fscache.c | 31 +- fs/nfs/fscache.h | 7 +- fs/smb/client/cifssmb.c | 6 +- fs/smb/client/file.c | 69 +++-- fs/smb/client/smb2pdu.c | 8 +- include/linux/netfs.h | 19 +- include/trace/events/netfs.h | 82 +++++- 22 files changed, 1171 insertions(+), 829 deletions(-) create mode 100644 fs/netfs/read_collect.c
fs/9p/vfs_addr.c +11 -2
@@ -64,12 +64,17 @@ static void v9fs_issue_write(struct netfs_io_subrequest *subreq)* v9fs_issue_read - Issue a read from 9P * @subreq: The read to make */ -static void v9fs_issue_read (struct netfs_io_subrequest *subreq) +static ssize_t v9fs_issue_read (struct netfs_io_subrequest *subreq) {struct netfs_io_request *rreq = subreq->rreq;struct p9_fid *fid = rreq->netfs_priv; +ssize_t len;int total, err; + len =netfs_prepare_read_iterator (subreq,ULONG_MAX ,0 ); +if (len <0 ) +return len; + total =p9_client_read (fid, subreq->start + subreq->transferred, &subreq->io_iter, &err);
@@ -77,7 +82,11 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)* cache won't be on server and is zeroes */ __set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); -netfs_subreq_terminated (subreq, err ?: total, false); +if (!err) + subreq->transferred += total; + +netfs_read_subreq_progress (subreq, err, false); +return len; }/**
fs/afs/file.c +12 -6
@@ -243,7 +243,7 @@ static void afs_fetch_data_notify(struct afs_operation *op) req->error = error;if (subreq) {__set_bit (NETFS_SREQ_CLEAR_TAIL , &subreq->flags); -netfs_subreq_terminated (subreq, error ?: req->actual_len, false); +netfs_read_subreq_progress (subreq, error, false); req->subreq =NULL ; }else if (req->done) { req->done (req);
@@ -293,7 +293,7 @@ int afs_fetch_data(struct afs_vnode *vnode, struct afs_read *req) op =afs_alloc_operation (req->key, vnode->volume);if (IS_ERR (op)) {if (req->subreq) -netfs_subreq_terminated (req->subreq,PTR_ERR (op), false); +netfs_read_subreq_progress (req->subreq,PTR_ERR (op), false);return PTR_ERR (op); }
@@ -312,7 +312,7 @@ static void afs_read_worker(struct work_struct *work) fsreq =afs_alloc_read (GFP_NOFS );if (!fsreq) -return netfs_subreq_terminated (subreq, -ENOMEM , false); +return netfs_read_subreq_progress (subreq, -ENOMEM , false); fsreq->subreq = subreq; fsreq->pos = subreq->start + subreq->transferred;
@@ -325,10 +325,16 @@ static void afs_read_worker(struct work_struct *work)afs_put_read (fsreq); } -static void afs_issue_read (struct netfs_io_subrequest *subreq) +static ssize_t afs_issue_read (struct netfs_io_subrequest *subreq) { -INIT_WORK (&subreq->work, afs_read_worker); -queue_work (system_long_wq, &subreq->work); +ssize_t len; + + len =netfs_prepare_read_iterator (subreq,ULONG_MAX ,0 ); +if (len >0 ) { +INIT_WORK (&subreq->work, afs_read_worker); +queue_work (system_long_wq, &subreq->work); + } +return len; }static int afs_symlink_read_folio (struct file *file,struct folio *folio)
fs/afs/fsclient.c +7 -2
@@ -304,6 +304,7 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)struct afs_vnode_param *vp = &op->file[0 ];struct afs_read *req = op->fetch.req;const __be32 *bp; +size_t count_before;int ret; _enter("{%u,%zu,%zu/%llu}",
@@ -345,10 +346,14 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)/* extract the returned data */ case 2 : -_debug ("extract data %zu/%llu" , -iov_iter_count (call->iter), req->actual_len); + count_before = call->iov_len; +_debug ("extract data %zu/%llu" , count_before, req->actual_len); ret =afs_extract_data (call, true); +if (req->subreq) { + req->subreq->transferred += count_before - call->iov_len; +netfs_read_subreq_progress (req->subreq, -EINPROGRESS , false); + }if (ret <0 )return ret;
fs/afs/yfsclient.c +7 -2
@@ -355,6 +355,7 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)struct afs_vnode_param *vp = &op->file[0 ];struct afs_read *req = op->fetch.req;const __be32 *bp; +size_t count_before;int ret; _enter("{%u,%zu, %zu/%llu}",
@@ -391,10 +392,14 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)/* extract the returned data */ case 2 : -_debug ("extract data %zu/%llu" , -iov_iter_count (call->iter), req->actual_len); + count_before = call->iov_len; +_debug ("extract data %zu/%llu" , count_before, req->actual_len); ret =afs_extract_data (call, true); +if (req->subreq) { + req->subreq->transferred += count_before - call->iov_len; +netfs_read_subreq_progress (req->subreq, -EINPROGRESS , false); + }if (ret <0 )return ret;
fs/ceph/addr.c +39 -33
147 lines not on this page · raw message
fs/netfs/Makefile +1 -1
@@ -5,12 +5,12 @@ netfs-y := \ buffered_write.o \ direct_read.o \ direct_write.o \ - io.o \ iterator.o \ locking.o \ main.o \ misc.o \ objects.o \ + read_collect.o \ write_collect.o \ write_issue.o
fs/netfs/buffered_read.c +339 -171
593 lines not on this page · raw message
fs/netfs/direct_read.c +98 -1
111 lines not on this page · raw message
fs/netfs/internal.h +0 -6
@@ -23,15 +23,9 @@/* * buffered_read.c */ -void netfs_rreq_unlock_folios (struct netfs_io_request *rreq);int netfs_prefetch_for_write (struct file *file,struct folio *folio,size_t offset,size_t len); -/* -* io.c -*/ -int netfs_begin_read (struct netfs_io_request *rreq,bool sync); -/* * main.c */
fs/netfs/io.c +2 -526
543 lines not on this page · raw message
fs/netfs/iterator.c +50 -0
59 lines not on this page · raw message
fs/netfs/main.c +3 -4
@@ -54,21 +54,20 @@ static int netfs_requests_seq_show(struct seq_file *m, void *v)if (v == &netfs_io_requests) {seq_puts (m, -"REQUEST OR REF FL ERR OPS COVERAGE\n" -"======== == === == ==== === =========\n" +"REQUEST OR REF FL ERR COVERAGE\n" +"======== == === == ==== =========\n" );return 0 ; } rreq = list_entry(v, struct netfs_io_request, proc_link);seq_printf (m, -"%08x %s %3d %2lx %4d %3d @%04llx %llx/%llx" , +"%08x %s %3d %2lx %4d @%04llx %llx/%llx" , rreq->debug_id, netfs_origins[rreq->origin],refcount_read (&rreq->ref), rreq->flags, rreq->error, -atomic_read (&rreq->nr_outstanding), rreq->start, rreq->submitted, rreq->len);seq_putc (m,'\n' );return 0 ;
fs/netfs/objects.c +0 -1
@@ -40,7 +40,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,memset (rreq,0 ,kmem_cache_size (cache)); rreq->start = start; rreq->len = len; - rreq->upper_len = len; rreq->origin = origin; rreq->netfs_ops = ctx->ops; rreq->mapping = mapping;
fs/netfs/read_collect.c new file +450 -0
450 lines not on this page · raw message
fs/netfs/write_issue.c +0 -4
@@ -158,10 +158,6 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,_enter ("R=%x[%x]" , wreq->debug_id, subreq->debug_index); -trace_netfs_sreq_ref (wreq->debug_id, subreq->debug_index, -refcount_read (&subreq->ref), - netfs_sreq_trace_new); -trace_netfs_sreq (subreq, netfs_sreq_trace_prepare); stream->sreq_max_len =UINT_MAX ;
fs/nfs/fscache.c +16 -15
@@ -286,15 +286,7 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *srereturn netfs; } -static bool nfs_netfs_clamp_length (struct netfs_io_subrequest *sreq) -{ -size_t rsize =NFS_SB (sreq->rreq->inode->i_sb)->rsize; - - sreq->len =min (sreq->len, rsize); -return true; -} - -static void nfs_netfs_issue_read (struct netfs_io_subrequest *sreq) +static ssize_t nfs_netfs_issue_read (struct netfs_io_subrequest *sreq) {struct nfs_netfs_io_data *netfs;struct nfs_pageio_descriptor pgio;
@@ -302,17 +294,26 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)struct nfs_open_context *ctx = sreq->rreq->netfs_priv;struct page *page;unsigned long idx; +pgoff_t start, last; +ssize_t len;int err; -pgoff_t start = (sreq->start + sreq->transferred) >>PAGE_SHIFT ; -pgoff_t last = ((sreq->start + sreq->len - - sreq->transferred -1 ) >>PAGE_SHIFT ); + + err =netfs_prepare_read_iterator (sreq,NFS_SB (sreq->rreq->inode->i_sb)->rsize,0 ); +if (err <0 ) +return err; + + len = sreq->len; + start = (sreq->start + sreq->transferred) >>PAGE_SHIFT ; + last = ((sreq->start + len - sreq->transferred -1 ) >>PAGE_SHIFT );nfs_pageio_init_read (&pgio, inode, false, &nfs_async_read_completion_ops); netfs =nfs_netfs_alloc (sreq); -if (!netfs) -return netfs_subreq_terminated (sreq, -ENOMEM , false); +if (!netfs) { +netfs_read_subreq_progress (sreq, -ENOMEM , false); +return -ENOMEM ; + } pgio.pg_netfs = netfs;/* used in completion */
@@ -327,6 +328,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) out:nfs_pageio_complete_read (&pgio);nfs_netfs_put (netfs); +return len; }void nfs_netfs_initiate_read (struct nfs_pgio_header *hdr)
@@ -377,5 +379,4 @@ const struct netfs_request_ops nfs_netfs_ops = { .init_request = nfs_netfs_init_request, .free_request = nfs_netfs_free_request, .issue_read = nfs_netfs_issue_read, - .clamp_length = nfs_netfs_clamp_length };
fs/nfs/fscache.h +3 -4
@@ -60,8 +60,6 @@ static inline void nfs_netfs_get(struct nfs_netfs_io_data *netfs)static inline void nfs_netfs_put (struct nfs_netfs_io_data *netfs) { -ssize_t final_len; -/* Only the last RPC completion should call netfs_subreq_terminated() */ if (!refcount_dec_and_test (&netfs->refcount))return ;
@@ -74,8 +72,9 @@ static inline void nfs_netfs_put(struct nfs_netfs_io_data *netfs)* Correct the final length here to be no larger than the netfs subrequest * length, and thus avoid netfs's "Subreq overread" warning message. */ - final_len =min_t (s64, netfs->sreq->len,atomic64_read (&netfs->transferred)); -netfs_subreq_terminated (netfs->sreq, netfs->error ?: final_len, false); + netfs->sreq->transferred =min_t (s64, netfs->sreq->len, +atomic64_read (&netfs->transferred)); +netfs_read_subreq_progress (netfs->sreq, netfs->error, false);kfree (netfs); }static inline void nfs_netfs_inode_init (struct nfs_inode *nfsi)
fs/smb/client/cifssmb.c +2 -4
@@ -1309,10 +1309,8 @@ cifs_readv_callback(struct mid_q_entry *mid)if (rdata->result ==0 || rdata->result == -EAGAIN )iov_iter_advance (&rdata->subreq.io_iter, rdata->got_bytes); rdata->credits.value =0 ; -netfs_subreq_terminated (&rdata->subreq, - (rdata->result ==0 || rdata->result == -EAGAIN ) ? - rdata->got_bytes : rdata->result, - false); + rdata->subreq.transferred += rdata->got_bytes; +netfs_read_subreq_progress (&rdata->subreq, rdata->result, false);release_mid (mid);add_credits (server, &credits,0 ); }
fs/smb/client/file.c +32 -37
120 lines not on this page · raw message
fs/smb/client/smb2pdu.c +5 -3
@@ -4489,9 +4489,7 @@ static void smb2_readv_worker(struct work_struct *work)struct cifs_io_subrequest *rdata = container_of(work, struct cifs_io_subrequest, subreq.work); -netfs_subreq_terminated (&rdata->subreq, - (rdata->result ==0 || rdata->result == -EAGAIN ) ? - rdata->got_bytes : rdata->result, true); +netfs_read_subreq_progress (&rdata->subreq, rdata->result, false); } static void
@@ -4538,6 +4536,7 @@ smb2_readv_callback(struct mid_q_entry *mid)break ;case MID_REQUEST_SUBMITTED :case MID_RETRY_NEEDED : +__set_bit (NETFS_SREQ_NEED_RETRY , &rdata->subreq.flags); rdata->result = -EAGAIN ;if (server->sign && rdata->got_bytes)/* reset bytes number since we can not check a sign */
@@ -4588,6 +4587,8 @@ smb2_readv_callback(struct mid_q_entry *mid) rdata->result =0 ; } rdata->credits.value =0 ; + rdata->subreq.transferred += rdata->got_bytes; +trace_netfs_sreq (&rdata->subreq, netfs_sreq_trace_io_progress);INIT_WORK (&rdata->subreq.work, smb2_readv_worker);queue_work (cifsiod_wq, &rdata->subreq.work);release_mid (mid);
@@ -4838,6 +4839,7 @@ smb2_writev_callback(struct mid_q_entry *mid) wdata->subreq.start, wdata->subreq.len); wdata->credits.value =0 ; +trace_netfs_sreq (&wdata->subreq, netfs_sreq_trace_io_progress);cifs_write_subrequest_terminated (wdata, result ?: written, true);release_mid (mid);add_credits (server, &credits,0 );
include/linux/netfs.h +13 -6
@@ -183,12 +183,18 @@ struct netfs_io_subrequest {unsigned long long start;/* Where to start the I/O */ size_t len;/* Size of the I/O */ size_t transferred;/* Amount of data transferred */ +size_t consumed;/* Amount of read data consumed */ +unsigned int prev_donated;/* Amount of data donated from previous subreq */ +unsigned int next_donated;/* Amount of data donated from next subreq */ refcount_t ref;short error;/* 0 or error that occurred */ unsigned short debug_index;/* Index in list (for debugging output) */ unsigned int nr_segs;/* Number of segs in io_iter */ enum netfs_io_source source;/* Where to read from/write to */ unsigned char stream_nr;/* I/O stream this belongs to */ +unsigned char curr_sheaf_slot;/* Folio currently being read */ +unsigned char curr_folio_order;/* Order of folio */ +struct sheaf *curr_sheaf;/* Sheaf in which current folio resides */ unsigned long flags;#define NETFS_SREQ_COPY_TO_CACHE 0/* Set if should copy the data to the cache */ #define NETFS_SREQ_CLEAR_TAIL 1/* Set if the rest of the read should be cleared */
@@ -229,6 +235,7 @@ struct netfs_io_request {struct address_space *mapping;/* The mapping being accessed */ struct kiocb *iocb;/* AIO completion vector */ struct netfs_cache_resources cache_resources; +struct readahead_control *ractl;/* Readahead descriptor */ struct list_head proc_link;/* Link in netfs_iorequests */ struct list_head subrequests;/* Contributory I/O operations */ struct netfs_io_stream io_streams[2 ];/* Streams of parallel I/O operations */
@@ -248,9 +255,6 @@ struct netfs_io_request {atomic_t subreq_counter;/* Next subreq->debug_index */ unsigned int nr_group_rel;/* Number of refs to release on ->group */ spinlock_t lock;/* Lock for queuing subreqs */ -atomic_t nr_outstanding;/* Number of ops in progress */ -atomic_t nr_copy_ops;/* Number of copy-to-cache ops in progress */ -size_t upper_len;/* Length can be extended to here */ unsigned long long submitted;/* Amount submitted for I/O so far */ unsigned long long len;/* Length of the request */ size_t transferred;/* Amount to be indicated as transferred */
@@ -265,6 +269,7 @@ struct netfs_io_request {unsigned long long collected_to;/* Point we've collected to */ unsigned long long cleaned_to;/* Position we've cleaned folios to */ pgoff_t no_unlock_folio;/* Don't unlock this folio after read */ +unsigned int prev_donated;/* Fallback for subreq->prev_donated */ refcount_t ref;unsigned long flags;#define NETFS_RREQ_INCOMPLETE_IO 0/* Some ioreqs terminated short or with error */
@@ -298,8 +303,7 @@ struct netfs_request_ops {/* Read request handling */ void (*expand_readahead)(struct netfs_io_request *rreq); -bool (*clamp_length)(struct netfs_io_subrequest *subreq); -void (*issue_read)(struct netfs_io_subrequest *subreq); +ssize_t (*issue_read)(struct netfs_io_subrequest *subreq);bool (*is_still_valid)(struct netfs_io_request *rreq);int (*check_write_begin)(struct file *file,loff_t pos,unsigned len,struct folio **foliop,void **_fsdata);
@@ -428,7 +432,10 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp);vm_fault_t netfs_page_mkwrite (struct vm_fault *vmf,struct netfs_group *netfs_group);/* (Sub)request management API. */ -void netfs_subreq_terminated (struct netfs_io_subrequest *,ssize_t ,bool ); +ssize_t netfs_prepare_read_iterator (struct netfs_io_subrequest *subreq,size_t rsize, +unsigned int max_segs); +void netfs_read_subreq_progress (struct netfs_io_subrequest *subreq, +int error,bool was_async);void netfs_get_subrequest (struct netfs_io_subrequest *subreq,enum netfs_sreq_ref_trace what); void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
include/trace/events/netfs.h +81 -1
129 lines not on this page · raw message