mm, iomap, NFSD: stop DONTCACHE writes reading back shared pages
Subsystems (8)
v12 cover mm, iomap, NFSD: stop DONTCACHE writes reading back shared pages
Hi,
An RWF_DONTCACHE write is buffered IO: its data goes through the page cache, and writeback drops each folio once the folio is written back. A write that starts or ends inside a folio covers that folio only partially. Usually another write covers the rest of the folio. If writeback drops the folio before the other write writes its part, the other write must first read the folio back from disk. 32 threads writing interleaved records to one XFS file, 4 KiB pages, before and after 5/7:
record pwritev2() flags 4 KiB device MiB/s
bytes reads per write
before after before after 47008 RWF_DONTCACHE 0.97 0.0005 831 2281 47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0005 601 1227 6000 RWF_DONTCACHE 0.97 0.0001 121 488 6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0002 90 206
47008 bytes is the transfer size of IO500's ior-hard, in which each rank writes interleaved records to one shared file. Any record size that is not a multiple of the page size has the same problem. 6000 bytes is a second example.
The test program and script: https://lore.kernel.org/linux-nfs/[email protected]/
- 1/7 moves code into a helper, with no functional change.
- 2/7 fixes 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback"), in v7.3: WB_DONTCACHE_DIRTY could drift.
- 3/7 reports WB_DONTCACHE_DIRTY in the bdi debugfs stats.
- 4/7 adds folio_mark_dropbehind() for a folio that is already dirty.
- 5/7 keeps a partially covered folio until the write that completes it.
- 6/7 lets a writer that syncs its own write complete those folios itself, after the sync.
- 7/7 makes NFSD do that for the misaligned ends of a direct-mode WRITE.
Merging: Chuck suggested taking the series through the mm tree. 1/7 to 6/7 apply cleanly to each of these mm branches:
mm-unstable 4318887a3efa ("fs/proc/task_mmu: fix lost maps entries above 2^63")
mm-new daed51c91051 ("riscv: mm: fix concurrency in mark_new_valid_map()")
mm-stable fa4ad798f6f6 ("Merge tag 'for-7.4-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux")7/7 needs the NFSD direct-mode changes that are in cel/nfsd-testing. The whole series applies to cel/nfsd-testing at ab036e3a0dd4 ("NFSD:
Refuse a direct READ whose aligned range exceeds the reply pages"), where each patch builds with W=1 with no warnings. So 7/7 could follow in the nfsd tree once the mm part is merged.
Changes since v11:
- 4/7: folio_mark_dropbehind() leaves a mapped folio unmarked. Through its page table, a mapped folio can be dirtied without the folio lock, and PG_dirty set before the folio is counted. With a stale dirty tag, folio_mark_dropbehind() then counted the folio in WB_DONTCACHE_DIRTY, and the dirtier counted it again. sashiko found this in its review of v11 4/7 (https://lore.kernel.org/r/[email protected]).
The folio lock keeps an unmapped folio unmapped, so the window is closed for every dirtier that does not use buffer heads. filemap_end_dropbehind() does not drop a mapped folio either.
- 6/7: filemap_dontcache_edge_complete() leaves a mapped folio alone, dirty or clean, to match.
- Jeff's Reviewed-by is dropped from 4/7, which now has a code change.
- Testing: a KUnit test (not in this series) runs the interleaving from sashiko's review step by step. On an unmapped folio, the folio is counted twice. On a mapped folio, v12 leaves it unmarked and the count is correct. The v11 tests through NFSD and with RWF_DONTCACHE give the same results with v12.
- Rebased on the current cel/nfsd-testing.
v11: https://lore.kernel.org/linux-nfs/[email protected]/
v10: https://lore.kernel.org/linux-nfs/[email protected]/
v9: https://lore.kernel.org/linux-nfs/[email protected]/
v8: https://lore.kernel.org/linux-nfs/[email protected]/
v7: https://lore.kernel.org/linux-nfs/[email protected]/
v6: https://lore.kernel.org/linux-nfs/[email protected]/
v5: https://lore.kernel.org/linux-nfs/[email protected]/
v4: https://lore.kernel.org/linux-nfs/[email protected]/
v3: https://lore.kernel.org/linux-nfs/[email protected]/
v2: https://lore.kernel.org/linux-nfs/[email protected]/
v1: https://lore.kernel.org/linux-nfs/[email protected]/
Thanks,
Mike
Mike Snitzer (7):
mm: move clearing the dropbehind flag of a found folio into a helper
mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there
mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it mm, iomap: keep a partially covered page until a DONTCACHE write completes it mm, iomap: let a DONTCACHE writer complete its shared pages itself
NFSD: drop a shared boundary page only once both WRITEs have written it
.../filesystems/nfs/nfsd-io-modes.rst | 18 ++ fs/iomap/buffered-io.c | 30 ++- fs/iomap/trace.h | 3 +- fs/nfsd/vfs.c | 107 ++++++++- include/linux/fs.h | 5 +- include/linux/iomap.h | 1 + include/linux/pagemap.h | 12 + mm/backing-dev.c | 6 + mm/filemap.c | 219 ++++++++++++++++-- 9 files changed, 370 insertions(+), 31 deletions(-)
base-commit: ab036e3a0dd410f762192f33d6244e7993b36f8d
Signature · 1 line
2.52.0
PATCH v12 1/7 mm: move clearing the dropbehind flag of a found folio into a helper
Move the code in __filemap_get_folio_mpol() that clears PG_dropbehind on a folio found by a cached lookup, and takes the folio off WB_DONTCACHE_DIRTY, into filemap_clear_dropbehind(). No functional change. The next patch changes the helper.
--- mm/filemap.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
mm/filemap.c +20 -13
@@ -1956,6 +1956,24 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)return folio; } +/* Clear the dropbehind flag of a folio that a cached lookup found. */ +static void filemap_clear_dropbehind (struct address_space *mapping, +struct folio *folio) +{ +if (!folio_test_clear_dropbehind (folio)) +return ; +if (folio_test_dirty (folio) &&mapping_can_writeback (mapping)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; +long nr =folio_nr_pages (folio); + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); +unlocked_inode_to_wb_end (inode, &cookie); + } +} +/** * __filemap_get_folio_mpol - Find and get a reference to a folio. * @mapping: The address_space to search.
@@ -2087,19 +2105,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_clear_dropbehind (folio)) { -if (folio_test_dirty (folio) && -mapping_can_writeback (mapping)) { -struct inode *inode = mapping->host; -struct bdi_writeback *wb; -struct wb_lock_cookie cookie = {}; -long nr =folio_nr_pages (folio); - - wb =unlocked_inode_to_wb_begin (inode, &cookie); -wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); -unlocked_inode_to_wb_end (inode, &cookie); - } - } +if (!(fgp_flags &FGP_DONTCACHE )) +filemap_clear_dropbehind (mapping, folio);return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);
Signature · 1 line
2.52.0
Re: [PATCH v12 1/7] mm: move clearing the dropbehind flag of a found folio into a helper
Quoted 7 lines · Mike Snitzer
Move the code in
__filemap_get_folio_mpol()that clearsPG_dropbehindon a folio found by a cached lookup, and takes the folio offWB_DONTCACHE_DIRTY, intofilemap_clear_dropbehind(). No functional change. The next patch changes the helper.Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Mike Snitzer <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 58 lines
--- mm/filemap.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)mm/filemap.c +20
-13@@ -1956,6 +1956,24 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)return folio; } +/* Clear the dropbehind flag of a folio that a cached lookup found. */ +static void filemap_clear_dropbehind (struct address_space *mapping, +struct folio *folio) +{ +if (!folio_test_clear_dropbehind (folio)) +return ; +if (folio_test_dirty (folio) &&mapping_can_writeback (mapping)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; +long nr =folio_nr_pages (folio); + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); +unlocked_inode_to_wb_end (inode, &cookie); + } +} +/** * __filemap_get_folio_mpol - Find and get a reference to a folio. * @mapping: The address_space to search. @@ -2087,19 +2105,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_clear_dropbehind (folio)) { -if (folio_test_dirty (folio) && -mapping_can_writeback (mapping)) { -struct inode *inode = mapping->host; -struct bdi_writeback *wb; -struct wb_lock_cookie cookie = {}; -long nr =folio_nr_pages (folio); - - wb =unlocked_inode_to_wb_begin (inode, &cookie); -wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); -unlocked_inode_to_wb_end (inode, &cookie); - } - } +if (!(fgp_flags &FGP_DONTCACHE )) +filemap_clear_dropbehind (mapping, folio);return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);Signature · 1 line
2.52.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v12 2/7 mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there
In reply to Mike SnitzerA cached lookup that finds a dropbehind folio clears the flag and subtracts the folio from WB_DONTCACHE_DIRTY if PG_dirty is set. That test races with the dirtying of the folio. filemap_dirty_folio() sets PG_dirty before folio_account_dirtied() counts the folio, so a lookup in between subtracts a folio that was never added. The lookup also does not hold the folio lock. Every other path that subtracts a dirty folio from the counter holds it, so the lookup can subtract the folio a second time. Either way the counter drifts, and once it is negative the DONTCACHE writeback kick writes nothing.
Fix this by having filemap_clear_dropbehind() clear the dropbehind flag of a folio that a cached lookup found, and take the folio off WB_DONTCACHE_DIRTY if it is counted there. With the folio locked and the i_pages lock held, the folio is counted if both PG_dirty and the dirty tag are set. A folio that another task has locked keeps the flag.
--- mm/filemap.c | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-)
mm/filemap.c +28 -14
@@ -1956,22 +1956,36 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)return folio; } -/* Clear the dropbehind flag of a folio that a cached lookup found. */ +/* +* Clear the dropbehind flag of a folio that a cached lookup found, and take +* the folio off WB_DONTCACHE_DIRTY if it is counted there. With the folio +* locked and the i_pages lock held, the folio is counted if both PG_dirty and +* the dirty tag are set. A folio that another task has locked keeps the flag. +*/ static void filemap_clear_dropbehind (struct address_space *mapping, -struct folio *folio) +struct folio *folio,bool locked) { -if (!folio_test_clear_dropbehind (folio)) -return ; -if (folio_test_dirty (folio) &&mapping_can_writeback (mapping)) { -struct inode *inode = mapping->host; -struct bdi_writeback *wb; -struct wb_lock_cookie cookie = {}; -long nr =folio_nr_pages (folio); +unsigned long flags; - wb =unlocked_inode_to_wb_begin (inode, &cookie); -wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); -unlocked_inode_to_wb_end (inode, &cookie); +if (!locked) { +if (!folio_trylock (folio)) +return ; +if (unlikely (folio->mapping != mapping)) + goto out; } +if (!mapping_can_writeback (mapping)) { +folio_clear_dropbehind (folio); + goto out; + } +xa_lock_irqsave (&mapping->i_pages, flags); +if (folio_test_clear_dropbehind (folio) &&folio_test_dirty (folio) && +xa_get_mark (&mapping->i_pages, folio->index,PAGECACHE_TAG_DIRTY )) +wb_stat_mod (inode_to_wb (mapping->host),WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +xa_unlock_irqrestore (&mapping->i_pages, flags); +out: +if (!locked) +folio_unlock (folio); }/**
@@ -2105,8 +2119,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (!(fgp_flags &FGP_DONTCACHE )) -filemap_clear_dropbehind (mapping, folio); +if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_dropbehind (folio)) +filemap_clear_dropbehind (mapping, folio, fgp_flags &FGP_LOCK );return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v12 2/7] mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm: unsigned negation in filemap_clear_dropbehind() corrupts counter on 32-bit --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm: unsigned negation in filemap_clear_dropbehind() corrupts counter on 32-bit
--
commit 08b68ff1ed201d0ae5bce5df36b625f50a1bfa1f
Author: Mike Snitzer <[email protected]>
mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there
This commit updates filemap_clear_dropbehind() to hold the folio lock and i_pages lock before clearing the dropbehind flag and decrementing WB_DONTCACHE_DIRTY when both PG_dirty and PAGECACHE_TAG_DIRTY are set.
Quoted 36 lines
mm/filemap.c +13
-12@@ -1956,22 +1956,36 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)return folio; } -/* Clear the dropbehind flag of a folio that a cached lookup found. */ +/* +* Clear the dropbehind flag of a folio that a cached lookup found, and take +* the folio off WB_DONTCACHE_DIRTY if it is counted there. With the folio +* locked and the i_pages lock held, the folio is counted if both PG_dirty and +* the dirty tag are set. A folio that another task has locked keeps the flag. +*/ static void filemap_clear_dropbehind (struct address_space *mapping, -struct folio *folio) +struct folio *folio,bool locked) { -if (!folio_test_clear_dropbehind (folio)) -return ; -if (folio_test_dirty (folio) &&mapping_can_writeback (mapping)) { -struct inode *inode = mapping->host; -struct bdi_writeback *wb; -struct wb_lock_cookie cookie = {}; -long nr =folio_nr_pages (folio); +unsigned long flags; - wb =unlocked_inode_to_wb_begin (inode, &cookie); -wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); -unlocked_inode_to_wb_end (inode, &cookie); +if (!locked) { +if (!folio_trylock (folio)) +return ; +if (unlikely (folio->mapping != mapping)) + goto out;
mm/filemap.c void *filemap_get_entry(struct address_space *mapping, pgoff_t index) · line 1984
Quoted 10 lines
mm/filemap.c +9
-0+ } +if (!mapping_can_writeback (mapping)) { +folio_clear_dropbehind (folio); + goto out; } +xa_lock_irqsave (&mapping->i_pages, flags); +if (folio_test_clear_dropbehind (folio) &&folio_test_dirty (folio) && +xa_get_mark (&mapping->i_pages, folio->index,PAGECACHE_TAG_DIRTY )) +wb_stat_mod (inode_to_wb (mapping->host),WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio));
[Severity: Medium]
Does passing -folio_nr_pages(folio) to wb_stat_mod() corrupt the WB_DONTCACHE_DIRTY counter on 32-bit architectures?
folio_nr_pages() returns unsigned long, whereas wb_stat_mod() takes s64. On 32-bit platforms, negating a 32-bit unsigned long produces 2^32 - nr, which is then zero-extended to s64, adding a large positive value instead of subtracting nr pages.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v12 2/7] mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there
In reply to Mike SnitzerQuoted 17 lines · Mike Snitzer
A cached lookup that finds a dropbehind folio clears the flag and subtracts the folio from
WB_DONTCACHE_DIRTYifPG_dirtyis set. That test races with the dirtying of the folio.filemap_dirty_folio()setsPG_dirtybeforefolio_account_dirtied()counts the folio, so a lookup in between subtracts a folio that was never added. The lookup also does not hold the folio lock. Every other path that subtracts a dirty folio from the counter holds it, so the lookup can subtract the folio a second time. Either way the counter drifts, and once it is negative the DONTCACHE writeback kick writes nothing.Fix this by having
filemap_clear_dropbehind()clear the dropbehind flag of a folio that a cached lookup found, and take the folio offWB_DONTCACHE_DIRTYif it is counted there. With the folio locked and thei_pageslock held, the folio is counted if bothPG_dirtyand the dirty tag are set. A folio that another task has locked keeps the flag.
Quoted 3 lines
Fixes 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")Reviewed-by Jeff Layton <[email protected]>Signed-off-by Mike Snitzer <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 71 lines
--- mm/filemap.c | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-)mm/filemap.c +28
-14@@ -1956,22 +1956,36 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)return folio; } -/* Clear the dropbehind flag of a folio that a cached lookup found. */ +/* +* Clear the dropbehind flag of a folio that a cached lookup found, and take +* the folio off WB_DONTCACHE_DIRTY if it is counted there. With the folio +* locked and the i_pages lock held, the folio is counted if both PG_dirty and +* the dirty tag are set. A folio that another task has locked keeps the flag. +*/ static void filemap_clear_dropbehind (struct address_space *mapping, -struct folio *folio) +struct folio *folio,bool locked) { -if (!folio_test_clear_dropbehind (folio)) -return ; -if (folio_test_dirty (folio) &&mapping_can_writeback (mapping)) { -struct inode *inode = mapping->host; -struct bdi_writeback *wb; -struct wb_lock_cookie cookie = {}; -long nr =folio_nr_pages (folio); +unsigned long flags; - wb =unlocked_inode_to_wb_begin (inode, &cookie); -wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr); -unlocked_inode_to_wb_end (inode, &cookie); +if (!locked) { +if (!folio_trylock (folio)) +return ; +if (unlikely (folio->mapping != mapping)) + goto out; } +if (!mapping_can_writeback (mapping)) { +folio_clear_dropbehind (folio); + goto out; + } +xa_lock_irqsave (&mapping->i_pages, flags); +if (folio_test_clear_dropbehind (folio) &&folio_test_dirty (folio) && +xa_get_mark (&mapping->i_pages, folio->index,PAGECACHE_TAG_DIRTY )) +wb_stat_mod (inode_to_wb (mapping->host),WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +xa_unlock_irqrestore (&mapping->i_pages, flags); +out: +if (!locked) +folio_unlock (folio); }/** @@ -2105,8 +2119,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (!(fgp_flags &FGP_DONTCACHE )) -filemap_clear_dropbehind (mapping, folio); +if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_dropbehind (folio)) +filemap_clear_dropbehind (mapping, folio, fgp_flags &FGP_LOCK );return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);Signature · 1 line
2.52.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v12 3/7 mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
In reply to Mike SnitzerThe per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the flusher writes back in response to an IOCB_DONTCACHE kick, but it is not reported anywhere, so the DONTCACHE dirty backlog can only be inferred from MemFree/Cached movement.
Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats and as WbDontcacheDirty in the per-wb stats, next to the Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in the per-cpu batches: the flusher targets the summed value and small backlogs would otherwise read as zero.
--- mm/backing-dev.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/backing-dev.c +6 -0
@@ -46,6 +46,7 @@ struct wb_stats {unsigned long nr_dirty_time;unsigned long nr_writeback;unsigned long nr_reclaimable; +unsigned long nr_dontcache_dirty;unsigned long nr_dirtied;unsigned long nr_written;unsigned long dirty_thresh;
@@ -78,6 +79,7 @@ static void collect_wb_stats(struct wb_stats *stats, stats->nr_writeback +=wb_stat (wb,WB_WRITEBACK ); stats->nr_reclaimable +=wb_stat (wb,WB_RECLAIMABLE ); + stats->nr_dontcache_dirty +=wb_stat_sum (wb,WB_DONTCACHE_DIRTY ); stats->nr_dirtied +=wb_stat (wb,WB_DIRTIED ); stats->nr_written +=wb_stat (wb,WB_WRITTEN ); stats->wb_thresh +=wb_calc_thresh (wb, stats->dirty_thresh);
@@ -125,6 +127,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)seq_printf (m,"BdiWriteback: %10lu kB\n" "BdiReclaimable: %10lu kB\n" +"BdiDontcacheDirty: %10lu kB\n" "BdiDirtyThresh: %10lu kB\n" "DirtyThresh: %10lu kB\n" "BackgroundThresh: %10lu kB\n"
@@ -139,6 +142,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)"state: %10lx\n" ,K (stats.nr_writeback),K (stats.nr_reclaimable), +K (stats.nr_dontcache_dirty),K (stats.wb_thresh),K (dirty_thresh), K(background_thresh),
@@ -163,6 +167,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,"WbCgIno: %10lu\n" "WbWriteback: %10lu kB\n" "WbReclaimable: %10lu kB\n" +"WbDontcacheDirty: %10lu kB\n" "WbDirtyThresh: %10lu kB\n" "WbDirtied: %10lu kB\n" "WbWritten: %10lu kB\n"
@@ -179,6 +184,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,#endif K (stats->nr_writeback),K (stats->nr_reclaimable), +K (stats->nr_dontcache_dirty),K (stats->wb_thresh),K (stats->nr_dirtied), K(stats->nr_written),
Signature · 1 line
2.52.0
Re: [PATCH v12 3/7] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
Quoted 11 lines · Mike Snitzer
The per-bdi_writeback
WB_DONTCACHE_DIRTYcounter drives how much the flusher writes back in response to anIOCB_DONTCACHEkick, but it is not reported anywhere, so the DONTCACHE dirty backlog can only be inferred from MemFree/Cached movement.Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats and as WbDontcacheDirty in the per-wb stats, next to the Reclaimable line. Use
wb_stat_sum()rather thanwb_stat()to fold in the per-cpu batches: the flusher targets the summed value and small backlogs would otherwise read as zero.
Quoted 3 lines
Fixes 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")Reviewed-by Jeff Layton <[email protected]>Signed-off-by Mike Snitzer <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 59 lines
--- mm/backing-dev.c | 6 ++++++ 1 file changed, 6 insertions(+)mm/backing-dev.c +6
-0@@ -46,6 +46,7 @@ struct wb_stats {unsigned long nr_dirty_time;unsigned long nr_writeback;unsigned long nr_reclaimable; +unsigned long nr_dontcache_dirty;unsigned long nr_dirtied;unsigned long nr_written;unsigned long dirty_thresh;@@ -78,6 +79,7 @@ static void collect_wb_stats(struct wb_stats *stats, stats->nr_writeback +=wb_stat (wb,WB_WRITEBACK ); stats->nr_reclaimable +=wb_stat (wb,WB_RECLAIMABLE ); + stats->nr_dontcache_dirty +=wb_stat_sum (wb,WB_DONTCACHE_DIRTY ); stats->nr_dirtied +=wb_stat (wb,WB_DIRTIED ); stats->nr_written +=wb_stat (wb,WB_WRITTEN ); stats->wb_thresh +=wb_calc_thresh (wb, stats->dirty_thresh);@@ -125,6 +127,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)seq_printf (m,"BdiWriteback: %10lu kB\n" "BdiReclaimable: %10lu kB\n" +"BdiDontcacheDirty: %10lu kB\n" "BdiDirtyThresh: %10lu kB\n" "DirtyThresh: %10lu kB\n" "BackgroundThresh: %10lu kB\n"@@ -139,6 +142,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)"state: %10lx\n" ,K (stats.nr_writeback),K (stats.nr_reclaimable), +K (stats.nr_dontcache_dirty),K (stats.wb_thresh),K (dirty_thresh), K(background_thresh),@@ -163,6 +167,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,"WbCgIno: %10lu\n" "WbWriteback: %10lu kB\n" "WbReclaimable: %10lu kB\n" +"WbDontcacheDirty: %10lu kB\n" "WbDirtyThresh: %10lu kB\n" "WbDirtied: %10lu kB\n" "WbWritten: %10lu kB\n" @@ -179,6 +184,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,#endif K (stats->nr_writeback),K (stats->nr_reclaimable), +K (stats->nr_dontcache_dirty),K (stats->wb_thresh),K (stats->nr_dirtied), K(stats->nr_written),Signature · 1 line
2.52.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v12 4/7 mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it
In reply to Mike SnitzerWB_DONTCACHE_DIRTY counts the dirty folios that are marked dropbehind. When a dropbehind folio is dirtied, folio_account_dirtied() adds it to the counter. When the folio is cleaned, folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it again. This balances only if the folio is marked before it is dirtied, as it is for the folios that an RWF_DONTCACHE write creates.
To prepare for NFSD to mark a folio dropbehind once both writes that share it have written their parts, handle a folio that is already dirty when it is marked. folio_set_dropbehind() only sets the flag. With it, the clean would subtract the folio from WB_DONTCACHE_DIRTY although it was never added, and the counter would go negative.
Add folio_mark_dropbehind(), the inverse of filemap_clear_dropbehind(). With the folio locked, it sets PG_dropbehind. Under the i_pages lock, it adds the folio to WB_DONTCACHE_DIRTY if both PG_dirty and the dirty tag are set. A dirtier that has set PG_dirty but not yet counted the folio finds the flag set and counts the folio itself.
Leave a mapped folio unmarked. Through its page table, a mapped folio can be dirtied without the folio lock: zap_present_folio_ptes() sets PG_dirty before the folio is counted. __folio_cancel_dirty() can leave a stale dirty tag on a folio. Then PG_dirty and the tag are both set in that window, and the folio would be counted twice. The folio lock keeps an unmapped folio unmapped. The dirtiers of an unmapped folio hold the folio lock, except mark_buffer_dirty() on a file system with buffer heads. filemap_end_dropbehind() does not drop a mapped folio either, so nothing is lost.
The name follows folio_set_dirty() and folio_mark_dirty(): the set variant only sets the flag, the mark variant also accounts for it.
--- include/linux/pagemap.h | 1 + mm/filemap.c | 42 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+)
include/linux/pagemap.h +1 -0
@@ -1306,6 +1306,7 @@ void end_page_writeback(struct page *page);void folio_end_writeback (struct folio *folio);void folio_end_writeback_no_dropbehind (struct folio *folio);void folio_end_dropbehind (struct folio *folio); +void folio_mark_dropbehind (struct folio *folio);void folio_wait_stable (struct folio *folio);void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);
mm/filemap.c +42 -0
48 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v12 5/7 mm, iomap: keep a partially covered page until a DONTCACHE write completes it
In reply to Mike SnitzerAn RWF_DONTCACHE buffered write goes through the page cache like any buffered write. It marks every folio it creates dropbehind, so writeback drops the folio as soon as the data of the write is on disk. A write that starts or ends inside a folio covers that folio only partially. Usually another write covers the rest of the folio. An example is the next record of a file written in records that are not a multiple of the page size. Writeback can drop the folio before the other write writes its part. Then the other write must first read the folio back from disk: one read per misaligned write.
Add FGP_DONTCACHE_EDGE for the lookup of such a folio. iomap sets it when a write starts or ends inside the smallest folio the mapping allows. A folio created for an edge is not marked dropbehind. A folio found uptodate already holds the data of the other write, so this write completes it. Mark it with folio_mark_dropbehind() under the folio lock, and the writeback that cleans the data of both writes drops it. With blocks smaller than the folio, the edge is not uptodate yet, so folio_dontcache_edge_written() also accepts a folio whose blocks outside this write are uptodate. A larger folio, such as one from readahead, is left alone.
The kernel cannot know if another write will come, and this commit does not assume it. It reads only the uptodate state of the folio. An edge that no other write completes, such as the last page of a file, stays cached until reclaim. Writeback drops all other folios of the write, as before.
With this commit, 32 threads write interleaved 47008-byte records to one XFS file with RWF_DONTCACHE. Device reads per write fell from 0.97 to 0.0005, and throughput rose from 831 to 2281 MiB/s. 47008 bytes is the transfer size of IO500's ior-hard, in which each rank writes interleaved records to one shared file. Any record size that is not a multiple of the page size has the same problem. With 6000-byte records, device reads per write fell from 0.97 to 0.0001, and throughput rose from 121 to 488 MiB/s.
--- fs/iomap/buffered-io.c | 26 +++++++++++++++--- include/linux/pagemap.h | 5 ++++ mm/filemap.c | 59 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 86 insertions(+), 4 deletions(-)
fs/iomap/buffered-io.c +23 -3
@@ -798,16 +798,36 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);*/ struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) { +struct address_space *mapping = iter->inode->i_mapping;fgf_t fgp =FGP_WRITEBEGIN ; +struct folio *folio;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; -if (iter->flags &IOMAP_DONTCACHE ) +if (iter->flags &IOMAP_DONTCACHE ) { +size_t unit =mapping_min_folio_nrbytes (mapping); +size_t off = pos & (unit -1 ); + fgp |=FGP_DONTCACHE ; +/* Another write may cover the rest of this unit. */ +if (off || iter->len < unit) { + fgp |=FGP_DONTCACHE_EDGE ; + len =min_t (size_t, len, unit - off); + } + } fgp |=fgf_set_order (len); -return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , - fgp,mapping_gfp_mask (iter->inode->i_mapping)); + folio =__filemap_get_folio (mapping, pos >>PAGE_SHIFT , fgp, +mapping_gfp_mask (mapping)); +/* +* With blocks smaller than the folio, an edge is not uptodate +* until this write completes it. +*/ +if ((fgp &FGP_DONTCACHE_EDGE ) && + !IS_ERR (folio) && !folio_test_uptodate (folio) && +folio_dontcache_edge_written (folio, pos, len)) +folio_mark_dropbehind (folio); +return folio; }EXPORT_SYMBOL_GPL (iomap_get_folio);
include/linux/pagemap.h +5 -0
@@ -698,6 +698,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,* * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) * * %FGP_DONTCACHE - Uncached buffered IO +* * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers the folio +* only partially. A new folio is not marked dropbehind, one found +* uptodate is. * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */
@@ -712,6 +715,7 @@ typedef unsigned int __bitwise fgf_t;#define FGP_FOR_MMAP ((__force fgf_t)0x00000040)#define FGP_STABLE ((__force fgf_t)0x00000080)#define FGP_DONTCACHE ((__force fgf_t)0x00000100) +#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)#define FGF_GET_ORDER (fgf) (((__force unsigned)fgf) >> 26)/* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
@@ -1307,6 +1311,7 @@ void folio_end_writeback(struct folio *folio);void folio_end_writeback_no_dropbehind (struct folio *folio);void folio_end_dropbehind (struct folio *folio);void folio_mark_dropbehind (struct folio *folio); +bool folio_dontcache_edge_written (struct folio *folio,loff_t pos,size_t len);void folio_wait_stable (struct folio *folio);void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);
mm/filemap.c +58 -1
77 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v12 6/7 mm, iomap: let a DONTCACHE writer complete its shared pages itself
In reply to Mike SnitzerWith the previous commit, the write that completes a shared folio marks it while it is still dirty. The folio then counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick. Some callers sync once, after writing several segments. NFSD does this for a WRITE that the client sends as FILE_SYNC or DATA_SYNC. Such a caller wakes the kick once per segment, for pages that its own sync is about to write.
Add IOCB_DONTCACHE_DEFER (with IOMAP_DONTCACHE_DEFER and FGP_DONTCACHE_DEFER). The write leaves the folios it covers only partially unmarked, and the caller completes them itself. Called right before the write, filemap_dontcache_edge_completes() tells the caller whether the other write that shares the folio already wrote its part. Called once the write and its sync are done, filemap_dontcache_edge_complete() waits for writeback in flight. Then it marks a dirty folio and kicks writeback, or drops a clean folio at once. A mapped folio stays cached, as folio_mark_dropbehind() leaves it.
Two writes that arrive together may both find the folio absent. The folio then stays cached until reclaim.
IOCB_DONTCACHE_DEFER enables NFSD, in a later commit, to avoid waking the DONTCACHE writeback kick.
--- fs/iomap/buffered-io.c | 6 +++- fs/iomap/trace.h | 3 +- include/linux/fs.h | 5 ++- include/linux/iomap.h | 1 + include/linux/pagemap.h | 6 ++++ mm/filemap.c | 75 +++++++++++++++++++++++++++++++++++++++-- 6 files changed, 91 insertions(+), 5 deletions(-)
fs/iomap/buffered-io.c +5 -1
@@ -812,6 +812,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)/* Another write may cover the rest of this unit. */ if (off || iter->len < unit) { fgp |=FGP_DONTCACHE_EDGE ; +if (iter->flags &IOMAP_DONTCACHE_DEFER ) + fgp |=FGP_DONTCACHE_DEFER ; len =min_t (size_t, len, unit - off); } }
@@ -823,7 +825,7 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)* With blocks smaller than the folio, an edge is not uptodate * until this write completes it. */ -if ((fgp &FGP_DONTCACHE_EDGE ) && +if ((fgp &FGP_DONTCACHE_EDGE ) && !(fgp &FGP_DONTCACHE_DEFER ) && !IS_ERR (folio) && !folio_test_uptodate (folio) &&folio_dontcache_edge_written (folio, pos, len))folio_mark_dropbehind (folio);
@@ -1333,6 +1335,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, iter.flags |=IOMAP_NOWAIT ;if (iocb->ki_flags &IOCB_DONTCACHE ) iter.flags |=IOMAP_DONTCACHE ; +if (iocb->ki_flags &IOCB_DONTCACHE_DEFER ) + iter.flags |=IOMAP_DONTCACHE_DEFER ;while ((ret =iomap_iter (&iter, ops)) >0 ) iter.status =iomap_write_iter (&iter, i, write_ops);
fs/iomap/trace.h +2 -1
@@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter); { IOMAP_UNSHARE, "UNSHARE" }, \ { IOMAP_DAX, "DAX" }, \ { IOMAP_ATOMIC, "ATOMIC" }, \ - { IOMAP_DONTCACHE, "DONTCACHE" } + { IOMAP_DONTCACHE, "DONTCACHE" }, \ + { IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" } #define IOMAP_F_FLAGS_STRINGS \ { IOMAP_F_NEW, "NEW" }, \
include/linux/fs.h +4 -1
@@ -357,6 +357,8 @@ struct readahead_control;/* kiocb is a read or write operation submitted by fs/aio.c. */ #define IOCB_AIO_RW (1 << 22)#define IOCB_HAS_METADATA (1 << 23) +/* With IOCB_DONTCACHE: see filemap_dontcache_edge_complete() */ +#define IOCB_DONTCACHE_DEFER (1 << 24)/* for use in trace events */ #define TRACE_IOCB_STRINGS \
@@ -374,7 +376,8 @@ struct readahead_control; { IOCB_NOIO, "NOIO" }, \ { IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \ { IOCB_AIO_RW, "AIO_RW" }, \ - { IOCB_HAS_METADATA, "AIO_HAS_METADATA" } + { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }, \ + { IOCB_DONTCACHE_DEFER, "DONTCACHE_DEFER" } struct kiocb {struct file *ki_filp;
include/linux/iomap.h +1 -0
@@ -212,6 +212,7 @@ struct iomap_write_ops {#endif /* CONFIG_FS_DAX */ #define IOMAP_ATOMIC (1 << 9)/* torn-write protection */ #define IOMAP_DONTCACHE (1 << 10) +#define IOMAP_DONTCACHE_DEFER (1 << 11)/* see IOCB_DONTCACHE_DEFER */ /* * Return the existing mapping at pos, or reserve space starting at pos for up
include/linux/pagemap.h +6 -0
@@ -701,6 +701,8 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,* * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers the folio * only partially. A new folio is not marked dropbehind, one found * uptodate is. +* * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: a folio found uptodate is +* not marked either, the caller completes it. * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */
@@ -716,6 +718,7 @@ typedef unsigned int __bitwise fgf_t;#define FGP_STABLE ((__force fgf_t)0x00000080)#define FGP_DONTCACHE ((__force fgf_t)0x00000100)#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200) +#define FGP_DONTCACHE_DEFER ((__force fgf_t)0x00000400)#define FGF_GET_ORDER (fgf) (((__force unsigned)fgf) >> 26)/* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
@@ -1312,6 +1315,9 @@ void folio_end_writeback_no_dropbehind(struct folio *folio);void folio_end_dropbehind (struct folio *folio);void folio_mark_dropbehind (struct folio *folio);bool folio_dontcache_edge_written (struct folio *folio,loff_t pos,size_t len); +bool filemap_dontcache_edge_completes (struct address_space *mapping, +loff_t pos,size_t len); +void filemap_dontcache_edge_complete (struct address_space *mapping,loff_t pos);void folio_wait_stable (struct folio *folio);void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);
mm/filemap.c +73 -2
87 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v12 7/7 NFSD: drop a shared boundary page only once both WRITEs have written it
In reply to Mike SnitzerNFSD splits a misaligned direct-mode WRITE into a direct middle and buffered DONTCACHE ends. Each end covers its first or last unit, the smallest folio the mapping allows, only partially. The other WRITE that covers the rest of that unit shares it.
Issue the buffered ends with IOCB_DONTCACHE_DEFER. Before writing an end, ask filemap_dontcache_edge_completes() whether the other WRITE has written its part of each shared unit. Once the WRITE is done, complete those units with filemap_dontcache_edge_complete(). If the client sent the WRITE as FILE_SYNC or DATA_SYNC, that is after the one vfs_fsync_range() that NFSD runs over the whole WRITE. An UNSTABLE WRITE has no fsync, so its units are completed while still dirty. A short write completes a unit only if it wrote its own part of it.
With this commit, 32 interleaved writers of 47008-byte records through NFSD DIRECT read 0.0001 to 0.0002 blocks per WRITE. Before, they read 0.99 blocks per WRITE. FILE_SYNC WRITEs no longer wake the DONTCACHE writeback kick, where they woke it 23,000 to 65,000 times a run without IOCB_DONTCACHE_DEFER. On a four-server pNFS flexfiles rig, the page cache grew by 2.9 GiB during the write phase instead of about 11 GiB.
47008 bytes is the transfer size of IO500's ior-hard, in which each rank writes interleaved records to one shared file with O_DIRECT. The Linux client sends an O_DIRECT write that fits in one WRITE as FILE_SYNC.
--- .../filesystems/nfs/nfsd-io-modes.rst | 18 +++ fs/nfsd/vfs.c | 107 ++++++++++++++++-- 2 files changed, 113 insertions(+), 12 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +18 -0
@@ -197,6 +197,24 @@ Misaligned WRITE: above. Set it to N for a workload that reads back or rewrites what it just wrote. + The page holding a start or end segment is shared with the other + WRITE that covers the rest of the page, which may arrive before, + after or at the same time, from any client. So that the second WRITE + to such a page does not have to read it back from disk, the page + cache keeps a page a DONTCACHE write covers only partially, + unmarked, for the write that completes it. NFSD asks, immediately + before writing a start or end segment, whether the other WRITE has + already written its part; if it has, NFSD marks the page DONTCACHE + once its own WRITE, including its fsync, is complete, and the page + is dropped. A page that no other WRITE completes, or that the other + WRITE writes at the same time, stays cached until it is reclaimed. + The first and last pages of a WRITE that is not split are handled + the same way. + + None of this applies when the start and end segments are normal + buffered IO, or for a filesystem that does not use iomap for + buffered writes. + Tracing: The nfsd_read_direct trace event marks a READ that NFSD issues as O_DIRECT. It records the offset and length the client requested.
fs/nfsd/vfs.c +95 -12
169 lines not on this page · raw message
Signature · 1 line
2.52.0
Jan Kara
maintainerHi!
Quoted 17 lines · Mike Snitzer
An
RWF_DONTCACHEwrite is buffered IO: its data goes through the page cache, and writeback drops each folio once the folio is written back. A write that starts or ends inside a folio covers that folio only partially. Usually another write covers the rest of the folio. If writeback drops the folio before the other write writes its part, the other write must first read the folio back from disk. 32 threads writing interleaved records to one XFS file, 4 KiB pages, before and after 5/7:record
pwritev2()flags 4 KiB device MiB/s
bytes reads per writebefore after before after 47008 RWF_DONTCACHE 0.97 0.0005 831 2281 47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0005 601 1227 6000 RWF_DONTCACHE 0.97 0.0001 121 488 6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0002 90 206
Quoted 4 lines
47008 bytes is the transfer size of IO500's ior-hard, in which each rank writes interleaved records to one shared file. Any record size that is not a multiple of the page size has the same problem. 6000 bytes is a second example.
OK, I understand the problem but frankly I hate the complexity this introduces. Patches 1-3 are good improvements, patch 4 is OK but 5 & 6 look too complex for too niche usecase to me. I didn't really pay attention to previous revisions of this patch set so I'm sorry if this was discussed but can't we just make iomap for unaligned RWF_DONTCACHE write ask for minimum-sized folios for the partial edges without FGP_DONTCACHE and use FGP_DONTCACHE only for folios fully within write? That will also solve the problem with the following write having to read again from the disk and it will be much simpler. Sure the edge folios will not be dropped after IO completion which may have smaller or larger effect (depending on the overall write size) but I'd say that's the cost of userspace making life hard for the kernel :-p
Honza
Quoted 86 lines
The test program and script: https://lore.kernel.org/linux-nfs/[email protected]/
- 1/7 moves code into a helper, with no functional change.
- 2/7 fixes88d6f128d06d("mm: track DONTCACHE dirty pages perbdi_writeback"), in v7.3:WB_DONTCACHE_DIRTYcould drift.
- 3/7 reportsWB_DONTCACHE_DIRTYin the bdi debugfs stats.
- 4/7 addsfolio_mark_dropbehind()for a folio that is already dirty.
- 5/7 keeps a partially covered folio until the write that completes it.
- 6/7 lets a writer that syncs its own write complete those folios itself, after the sync.
- 7/7 makes NFSD do that for the misaligned ends of a direct-mode WRITE.Merging: Chuck suggested taking the series through the mm tree. 1/7 to 6/7 apply cleanly to each of these mm branches:
mm-unstable 4318887a3efa ("fs/proc/task_mmu: fix lost maps entries above 2^63") mm-new daed51c91051 ("riscv: mm: fix concurrency in mark_new_valid_map()") mm-stable fa4ad798f6f6 ("Merge tag 'for-7.4-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux")7/7 needs the NFSD direct-mode changes that are in cel/nfsd-testing. The whole series applies to cel/nfsd-testing at
ab036e3a0dd4("NFSD:
Refuse a direct READ whose aligned range exceeds the reply pages"), where each patch builds with W=1 with no warnings. So 7/7 could follow in the nfsd tree once the mm part is merged.Changes since v11:
- 4/7:folio_mark_dropbehind()leaves a mapped folio unmarked. Through its page table, a mapped folio can be dirtied without the folio lock, andPG_dirtyset before the folio is counted. With a stale dirty tag,folio_mark_dropbehind()then counted the folio inWB_DONTCACHE_DIRTY, and the dirtier counted it again. sashiko found this in its review of v11 4/7 (https://lore.kernel.org/r/[email protected]).
The folio lock keeps an unmapped folio unmapped, so the window is closed for every dirtier that does not use buffer heads.filemap_end_dropbehind()does not drop a mapped folio either.
- 6/7:filemap_dontcache_edge_complete()leaves a mapped folio alone, dirty or clean, to match.
- Jeff's Reviewed-by is dropped from 4/7, which now has a code change.
- Testing: a KUnit test (not in this series) runs the interleaving from sashiko's review step by step. On an unmapped folio, the folio is counted twice. On a mapped folio, v12 leaves it unmarked and the count is correct. The v11 tests through NFSD and withRWF_DONTCACHEgive the same results with v12.
- Rebased on the current cel/nfsd-testing.v11: https://lore.kernel.org/linux-nfs/[email protected]/
v10: https://lore.kernel.org/linux-nfs/[email protected]/
v9: https://lore.kernel.org/linux-nfs/[email protected]/
v8: https://lore.kernel.org/linux-nfs/[email protected]/
v7: https://lore.kernel.org/linux-nfs/[email protected]/
v6: https://lore.kernel.org/linux-nfs/[email protected]/
v5: https://lore.kernel.org/linux-nfs/[email protected]/
v4: https://lore.kernel.org/linux-nfs/[email protected]/
v3: https://lore.kernel.org/linux-nfs/[email protected]/
v2: https://lore.kernel.org/linux-nfs/[email protected]/
v1: https://lore.kernel.org/linux-nfs/[email protected]/Thanks,
MikeMike Snitzer (7):
mm: move clearing the dropbehind flag of a found folio into a helper
mm: take a folio offWB_DONTCACHE_DIRTYonly if it was counted there
mm: exposeWB_DONTCACHE_DIRTYin bdi debugfs stats
mm: addfolio_mark_dropbehind()to mark a folio dropbehind and account it mm, iomap: keep a partially covered page until a DONTCACHE write completes it mm, iomap: let a DONTCACHE writer complete its shared pages itself
NFSD: drop a shared boundary page only once both WRITEs have written it.../filesystems/nfs/nfsd-io-modes.rst | 18 ++ fs/iomap/buffered-io.c | 30 ++- fs/iomap/trace.h | 3 +- fs/nfsd/vfs.c | 107 ++++++++- include/linux/fs.h | 5 +- include/linux/iomap.h | 1 + include/linux/pagemap.h | 12 + mm/backing-dev.c | 6 + mm/filemap.c | 219 ++++++++++++++++-- 9 files changed, 370 insertions(+), 31 deletions(-)base-commit:
ab036e3a0dd410f762192f33d6244e7993b36f8dSignature · 1 line
2.52.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 25 lines · Jan Kara
Hi!
Quoted 17 lines · Mike Snitzer
An
RWF_DONTCACHEwrite is buffered IO: its data goes through the page cache, and writeback drops each folio once the folio is written back. A write that starts or ends inside a folio covers that folio only partially. Usually another write covers the rest of the folio. If writeback drops the folio before the other write writes its part, the other write must first read the folio back from disk. 32 threads writing interleaved records to one XFS file, 4 KiB pages, before and after 5/7:record
pwritev2()flags 4 KiB device MiB/s
bytes reads per writebefore after before after 47008 RWF_DONTCACHE 0.97 0.0005 831 2281 47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0005 601 1227 6000 RWF_DONTCACHE 0.97 0.0001 121 488 6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.0002 90 206Quoted 4 lines
47008 bytes is the transfer size of IO500's ior-hard, in which each rank writes interleaved records to one shared file. Any record size that is not a multiple of the page size has the same problem. 6000 bytes is a second example.
Quoted 12 lines
OK, I understand the problem but frankly I hate the complexity this introduces. Patches 1-3 are good improvements, patch 4 is OK but 5 & 6 look too complex for too niche usecase to me. I didn't really pay attention to previous revisions of this patch set so I'm sorry if this was discussed but can't we just make iomap for unaligned
RWF_DONTCACHEwrite ask for minimum-sized folios for the partial edges withoutFGP_DONTCACHEand useFGP_DONTCACHEonly for folios fully within write? That will also solve the problem with the following write having to read again from the disk and it will be much simpler. Sure the edge folios will not be dropped after IO completion which may have smaller or larger effect (depending on the overall write size) but I'd say that's the cost of userspace making life hard for the kernel :-p
Thanks for looking, Jan.
On complexity: 5/7 and 6/7 add almost nothing to the fast path. The new code runs only at a partial edge. Aligned IO, and IO that is not RWF_DONTCACHE, pays for it with a few flag tests:
- IO that is not RWF_DONTCACHE (buffered, direct, reads): every new branch is behind IOMAP_DONTCACHE or FGP_DONTCACHE. The cost is one flag test in the found-folio path of __filemap_get_folio(), one when a folio is created, and one per write call in iomap_file_buffered_write().
- An aligned RWF_DONTCACHE write: iomap_get_folio() tests whether the folio is partially covered, which is a mask and two compares per folio. The test is false, so FGP_DONTCACHE_EDGE is not set, and the folio is created and marked dropbehind as before.
- No new locks, atomics, allocations, page flags or per-file state. The state of an edge is the uptodate state that the folio already has. Writeback and reads do not change.
- folio_dontcache_edge_written() and the two 6/7 helpers run only at a partial edge, or for a caller that asks for IOCB_DONTCACHE_DEFER (NFSD).
So the complexity is all in the misaligned case, which needs it. I'd argue that if the kernel (MM) can handle userspace making life hard it should, within reason. I think my patchset nicely isolates the complexity to userspace that makes life difficult.
The first part of what you describe is what 5/7 already does. iomap asks for a minimum-sized folio at each partial edge, and a new edge folio is not marked dropbehind. 5/7 adds one more step: the write that completes an edge marks it, so the writeback that cleans it drops it.
Without that step, every edge stays cached until reclaim. For this workload, that is a large part of what the writes put in the page cache:
- With 4 KiB pages, each record leaves about one page cached. Through LOCALIO with cached edges, 8.6% of the data stayed resident with 47008-byte records, and 66% with 6000-byte records. When the write that completes an edge drops it, 0% stayed resident.
- With 64 KiB pages, a 47008-byte record never covers a whole folio. So no folio is ever marked, and RWF_DONTCACHE caches as much as a normal buffered write.
- Jon Flynn measured cached edges through NFSD on four pNFS data servers with ior-hard. The page cache grew by 622 GiB in a 318 s write phase. With this series it grew by 2.9 GiB.
So for this misaligned workload, cached edges keep from 9% to all of the written data in the page cache, depending on the record size and the page size. The reads stop, but the page cache fills until reclaim runs.
6/7 does not change what is cached. A caller that syncs its own write (NFSD, for a FILE_SYNC WRITE) uses it to mark the shared folios after its sync. Without it, each segment wakes the DONTCACHE writeback kick for pages that the sync then writes.
Mike