mm: improve write performance with RWF_DONTCACHE
Subsystems (6)
Jeff Layton
maintainerv4 cover mm: improve write performance with RWF_DONTCACHE
Here's a new version for everyone to contemplate while traveling!
This patch series attempts to improve write performance with RWF_DONTCACHE. The main justification and benchmarks for the series are in patch #2.
This version implements a scheme that Jan Kara and Christoph Hellwig suggested during review of the earlier series: after a DONTCACHE write, kick the flusher thread to do an amount of writeback proportional to the amount written, but don't target any particular inode or pages when doing writeback.
The second patch in the series has a summary of the benchmark results. This seems to markedly improve RWF_DONTCACHE write performance in all the benchmarks I tested.
The benchmarks I used are in the last two patches. I'm not sure if we want to merge those into the tree as they are (mostly) AI slop. There is probably a better tool for this out there.
---
Changes in v4:
- Track DONTCACHE dirty pages per bdi_writeback
- New benchmark for competing buffered and dontcache writers
- New benchmark replicating Jens' original 32 concurrent writer test
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Track dirty DONTCACHE pages in the VM
- Have flusher write back a proportional number of pages after DONTCACHE write
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- kick flusher thread instead of initiating writeback inline
- add mechanism to run 'perf lock' around the testcases
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Jeff Layton (4):
mm: track DONTCACHE dirty pages per bdi_writeback
mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
testing: add nfsd-io-bench NFS server benchmark suite
testing: add dontcache-bench local filesystem benchmark suite
fs/fs-writeback.c | 60 ++ include/linux/backing-dev-defs.h | 3 + include/linux/fs.h | 6 +- include/trace/events/writeback.h | 3 +- mm/filemap.c | 13 +- mm/page-writeback.c | 6 + .../dontcache-bench/fio-jobs/axboe-write.fio | 14 + .../dontcache-bench/fio-jobs/lat-reader.fio | 12 + .../dontcache-bench/fio-jobs/multi-write.fio | 11 + .../dontcache-bench/fio-jobs/noisy-writer.fio | 12 + .../testing/dontcache-bench/fio-jobs/rand-read.fio | 13 + .../dontcache-bench/fio-jobs/rand-write.fio | 13 + .../testing/dontcache-bench/fio-jobs/seq-read.fio | 13 + .../testing/dontcache-bench/fio-jobs/seq-write.fio | 13 + .../dontcache-bench/scripts/parse-results.sh | 346 +++++++++++ .../dontcache-bench/scripts/run-benchmarks.sh | 643 +++++++++++++++++++++ .../testing/nfsd-io-bench/fio-jobs/lat-reader.fio | 15 + .../testing/nfsd-io-bench/fio-jobs/multi-write.fio | 14 + .../nfsd-io-bench/fio-jobs/noisy-writer.fio | 14 + tools/testing/nfsd-io-bench/fio-jobs/rand-read.fio | 15 + .../testing/nfsd-io-bench/fio-jobs/rand-write.fio | 15 + tools/testing/nfsd-io-bench/fio-jobs/seq-read.fio | 14 + tools/testing/nfsd-io-bench/fio-jobs/seq-write.fio | 14 + .../testing/nfsd-io-bench/scripts/parse-results.sh | 238 ++++++++ .../nfsd-io-bench/scripts/run-benchmarks.sh | 591 +++++++++++++++++++ .../testing/nfsd-io-bench/scripts/setup-server.sh | 94 +++ 26 files changed, 2199 insertions(+), 6 deletions(-)
---
base-commit: 26fd6bff2c050196005312d1d306889220952a99
change-id: 20260401-dontcache-5811efd7eaf3
Best regards,
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerPATCH v4 1/4 mm: track DONTCACHE dirty pages per bdi_writeback
Add a per-wb WB_DONTCACHE_DIRTY counter that tracks the number of dirty pages with the dropbehind flag set (i.e., pages dirtied via RWF_DONTCACHE writes).
Increment the counter alongside WB_RECLAIMABLE in folio_account_dirtied() when the folio has the dropbehind flag set, and decrement it in folio_clear_dirty_for_io() and folio_account_cleaned(). Also decrement it when a non-DONTCACHE lookup clears the dropbehind flag on a dirty folio in __filemap_get_folio_mpol(), using proper writeback domain locking.
The counter will be used by the writeback flusher to determine how many pages to write back when expediting writeback for IOCB_DONTCACHE writes, without flushing the entire BDI's dirty pages.
--- include/linux/backing-dev-defs.h | 1 + mm/filemap.c | 13 ++++++++++++- mm/page-writeback.c | 6 ++++++ 3 files changed, 19 insertions(+), 1 deletion(-)
include/linux/backing-dev-defs.h +1 -0
@@ -33,6 +33,7 @@ enum wb_stat_item {WB_WRITEBACK ,WB_DIRTIED ,WB_WRITTEN , +WB_DONTCACHE_DIRTY , NR_WB_STAT_ITEMS };
mm/filemap.c +12 -1
@@ -2052,8 +2052,19 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) +if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) { +if (folio_test_dirty (folio)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +unlocked_inode_to_wb_end (inode, &cookie); + }folio_clear_dropbehind (folio); + }return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);
mm/page-writeback.c +6 -0
@@ -2630,6 +2630,8 @@ static void folio_account_dirtied(struct folio *folio, wb =inode_to_wb (inode);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , nr);__zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr);__node_stat_mod_folio (folio,NR_DIRTIED , nr);wb_stat_mod (wb,WB_RECLAIMABLE , nr);
@@ -2651,6 +2653,8 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr);task_io_account_cancelled_write (nr *PAGE_SIZE );
@@ -2920,6 +2924,8 @@ bool folio_clear_dirty_for_io(struct folio *folio)if (folio_test_clear_dirty (folio)) {long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr); ret = true;
Signature · 1 line
2.54.0
Re: [PATCH v4 1/4] mm: track DONTCACHE dirty pages per bdi_writeback
Quoted 10 lines · Jeff Layton
Add a per-wb
WB_DONTCACHE_DIRTYcounter that tracks the number of dirty pages with the dropbehind flag set (i.e., pages dirtied viaRWF_DONTCACHEwrites).Increment the counter alongside
WB_RECLAIMABLEinfolio_account_dirtied()when the folio has the dropbehind flag set, and decrement it infolio_clear_dirty_for_io()andfolio_account_cleaned(). Also decrement it when a non-DONTCACHE lookup clears the dropbehind flag on a dirty folio in__filemap_get_folio_mpol(), using proper writeback domain locking.
Quoted 7 lines
The counter will be used by the writeback flusher to determine how many pages to write back when expediting writeback for
IOCB_DONTCACHEwrites, without flushing the entire BDI's dirty pages.Suggested-by Jan Kara <[email protected]>Assisted-by Claude:claude-opus-4-6Signed-off-by Jeff Layton <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 78 lines
--- include/linux/backing-dev-defs.h | 1 + mm/filemap.c | 13 ++++++++++++- mm/page-writeback.c | 6 ++++++ 3 files changed, 19 insertions(+), 1 deletion(-)include/linux/backing-dev-defs.h +1
-0@@ -33,6 +33,7 @@ enum wb_stat_item {WB_WRITEBACK ,WB_DIRTIED ,WB_WRITTEN , +WB_DONTCACHE_DIRTY , NR_WB_STAT_ITEMS };mm/filemap.c +12
-1@@ -2052,8 +2052,19 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) +if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) { +if (folio_test_dirty (folio)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +unlocked_inode_to_wb_end (inode, &cookie); + }folio_clear_dropbehind (folio); + }return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);mm/page-writeback.c +6
-0@@ -2630,6 +2630,8 @@ static void folio_account_dirtied(struct folio *folio, wb =inode_to_wb (inode);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , nr);__zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr);__node_stat_mod_folio (folio,NR_DIRTIED , nr);wb_stat_mod (wb,WB_RECLAIMABLE , nr);@@ -2651,6 +2653,8 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr);task_io_account_cancelled_write (nr *PAGE_SIZE );@@ -2920,6 +2924,8 @@ bool folio_clear_dirty_for_io(struct folio *folio)if (folio_test_clear_dirty (folio)) {long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr); ret = true;Signature · 2 lines
-- 2.54.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jeff Layton
maintainerPATCH v4 2/4 mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
In reply to Jeff LaytonThe IOCB_DONTCACHE writeback path in generic_write_sync() calls filemap_flush_range() on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).
Replace the inline filemap_flush_range() call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.
To avoid flushing unrelated buffered dirty data, add a dedicated WB_start_dontcache bit and wb_check_start_dontcache() handler that uses the per-wb WB_DONTCACHE_DIRTY counter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.
Like WB_start_all, the WB_start_dontcache bit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations.
Also add WB_REASON_DONTCACHE as a new writeback reason for tracing visibility, and target the correct cgroup writeback domain via unlocked_inode_to_wb_begin().
dontcache-bench results (same host, T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fio io_uring):
Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Change
seq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%
Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)
Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%
Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%
Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before After
buffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/s
Previously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
--- fs/fs-writeback.c | 60 ++++++++++++++++++++++++++++++++++++++++ include/linux/backing-dev-defs.h | 2 ++ include/linux/fs.h | 6 ++-- include/trace/events/writeback.h | 3 +- 4 files changed, 66 insertions(+), 5 deletions(-)
fs/fs-writeback.c +60 -0
@@ -1334,6 +1334,18 @@ static void wb_start_writeback(struct bdi_writeback *wb, enum wb_reason reason)wb_wakeup (wb); } +static void wb_start_dontcache_writeback (struct bdi_writeback *wb) +{ +if (!wb_has_dirty_io (wb)) +return ; + +if (test_bit (WB_start_dontcache, &wb->state) || +test_and_set_bit (WB_start_dontcache, &wb->state)) +return ; + +wb_wakeup (wb); +} +/** * wb_start_background_writeback - start background writeback * @wb: bdi_writback to write from
@@ -2373,6 +2385,28 @@ static long wb_check_start_all(struct bdi_writeback *wb)return nr_pages; } +static long wb_check_start_dontcache (struct bdi_writeback *wb) +{ +long nr_pages; + +if (!test_bit (WB_start_dontcache, &wb->state)) +return 0 ; + + nr_pages =wb_stat (wb,WB_DONTCACHE_DIRTY ); +if (nr_pages) { +struct wb_writeback_work work = { + .nr_pages = nr_pages, + .sync_mode =WB_SYNC_NONE , + .range_cyclic =1 , + .reason =WB_REASON_DONTCACHE , + }; + + nr_pages =wb_writeback (wb, &work); + } + +clear_bit (WB_start_dontcache, &wb->state); +return nr_pages; +}/* * Retrieve work items and do the writeback they describe
@@ -2394,6 +2428,11 @@ static long wb_do_writeback(struct bdi_writeback *wb)*/ wrote +=wb_check_start_all (wb); +/* +* Check for dontcache writeback request +*/ + wrote +=wb_check_start_dontcache (wb); +/* * Check for periodic writeback, kupdated() style */
@@ -2468,6 +2507,27 @@ void wakeup_flusher_threads_bdi(struct backing_dev_info *bdi,rcu_read_unlock (); } +/** +* filemap_dontcache_kick_writeback - kick flusher for IOCB_DONTCACHE writes +* @mapping: address_space that was just written to +* +* Kick the writeback flusher thread to expedite writeback of dontcache +* dirty pages. Uses a dedicated WB_start_dontcache bit so that only +* pages tracked by WB_DONTCACHE_DIRTY are written back, rather than +* flushing the entire BDI's dirty pages. +*/ +void filemap_dontcache_kick_writeback (struct address_space *mapping) +{ +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_start_dontcache_writeback (wb); +unlocked_inode_to_wb_end (inode, &cookie); +} +EXPORT_SYMBOL_GPL (filemap_dontcache_kick_writeback); +/* * Wakeup the flusher threads to start writeback of all currently dirty pages */
include/linux/backing-dev-defs.h +2 -0
@@ -26,6 +26,7 @@ enum wb_state { WB_writeback_running,/* Writeback is in progress */ WB_has_dirty_io,/* Dirty inodes on ->b_{dirty|io|more_io} */ WB_start_all,/* nr_pages == 0 (all) work pending */ + WB_start_dontcache, /* dontcache writeback pending */ }; enum wb_stat_item {
@@ -56,6 +57,7 @@ enum wb_reason {*/ WB_REASON_FORKER_THREAD ,WB_REASON_FOREIGN_FLUSH , +WB_REASON_DONTCACHE , WB_REASON_MAX, };
include/linux/fs.h +2 -4
@@ -2624,6 +2624,7 @@ extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end);int filemap_flush_range (struct address_space *mapping,loff_t start,loff_t end); +void filemap_dontcache_kick_writeback (struct address_space *mapping);static inline int file_write_and_wait (struct file *file) {
@@ -2657,10 +2658,7 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)if (ret)return ret; }else if (iocb->ki_flags &IOCB_DONTCACHE ) { -struct address_space *mapping = iocb->ki_filp->f_mapping; - -filemap_flush_range (mapping, iocb->ki_pos - count, - iocb->ki_pos -1 ); +filemap_dontcache_kick_writeback (iocb->ki_filp->f_mapping); }return count;
include/trace/events/writeback.h +2 -1
@@ -44,7 +44,8 @@ EM( WB_REASON_PERIODIC, "periodic") \ EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ EM( WB_REASON_FORKER_THREAD, "forker_thread") \ - EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush") + EM( WB_REASON_FOREIGN_FLUSH, "foreign_flush") \ + EMe(WB_REASON_DONTCACHE, "dontcache")WB_WORK_REASON
Signature · 1 line
2.54.0
Re: [PATCH v4 2/4] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
Quoted 75 lines · Jeff Layton
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.To avoid flushing unrelated buffered dirty data, add a dedicated
WB_start_dontcachebit andwb_check_start_dontcache()handler that uses the per-wbWB_DONTCACHE_DIRTYcounter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.Like
WB_start_all, theWB_start_dontcachebit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations.Also add
WB_REASON_DONTCACHEas a new writeback reason for tracing visibility, and target the correct cgroup writeback domain viaunlocked_inode_to_wb_begin().dontcache-bench results (same host,
T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fioio_uring):Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Changeseq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/sPreviously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 5 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
I like this, this is the better way to kick off the writeback.
Signature · 1 line
Jens Axboe
Re: [PATCH v4 2/4] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
In reply to Jeff LaytonQuoted 75 lines · Jeff Layton
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.To avoid flushing unrelated buffered dirty data, add a dedicated
WB_start_dontcachebit andwb_check_start_dontcache()handler that uses the per-wbWB_DONTCACHE_DIRTYcounter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.Like
WB_start_all, theWB_start_dontcachebit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations.Also add
WB_REASON_DONTCACHEas a new writeback reason for tracing visibility, and target the correct cgroup writeback domain viaunlocked_inode_to_wb_begin().dontcache-bench results (same host,
T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fioio_uring):Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Changeseq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/sPreviously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 8 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
Assisted-by Claude:claude-opus-4-6Signed-off-by Jeff Layton <[email protected]>
Nice and looks good to me now. Feel free to add:
One nit below:
Quoted 8 lines
+/**
+ *filemap_dontcache_kick_writeback- kick flusher forIOCB_DONTCACHEwrites
+ * @mapping:address_spacethat was just written to
+ *
+ * Kick the writeback flusher thread to expedite writeback of dontcache
+ * dirty pages. Uses a dedicatedWB_start_dontcachebit so that only
+ * pages tracked byWB_DONTCACHE_DIRTYare written back, rather than
+ * flushing the entire BDI's dirty pages.
This comment is a bit confusing as in fact we write arbitrary dirty pages. It is only the amount of pages that is influenced by WB_DONTCACHE_DIRTY. So I'd rephrase the last sentence like: We queue writeback for the inode's wb for as many pages as there are dontcache pages but we don't restrict writeback to dontcache pages only. This significantly improves performance over either writing all wb's pages or writing only dontcache pages. Although it doesn't guarantee quick writeback and reclaim of dontcache pages it keeps the amount of dirty pages in check and over longer term dontcache pages get written and reclaimed by background writeback even with this rough heuristic.
Honza
Quoted 78 lines
+ */
fs/fs-writeback.c +12
-0+void filemap_dontcache_kick_writeback (struct address_space *mapping) +{ +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_start_dontcache_writeback (wb); +unlocked_inode_to_wb_end (inode, &cookie); +} +EXPORT_SYMBOL_GPL (filemap_dontcache_kick_writeback); +/* * Wakeup the flusher threads to start writeback of all currently dirty pages */ include/linux/backing-dev-defs.h +2
-0@@ -26,6 +26,7 @@ enum wb_state { WB_writeback_running,/* Writeback is in progress */ WB_has_dirty_io,/* Dirty inodes on ->b_{dirty|io|more_io} */ WB_start_all,/* nr_pages == 0 (all) work pending */ + WB_start_dontcache, /* dontcache writeback pending */ }; enum wb_stat_item {@@ -56,6 +57,7 @@ enum wb_reason {*/ WB_REASON_FORKER_THREAD ,WB_REASON_FOREIGN_FLUSH , +WB_REASON_DONTCACHE , WB_REASON_MAX, };include/linux/fs.h +2
-4@@ -2624,6 +2624,7 @@ extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end);int filemap_flush_range (struct address_space *mapping,loff_t start,loff_t end); +void filemap_dontcache_kick_writeback (struct address_space *mapping);static inline int file_write_and_wait (struct file *file) {@@ -2657,10 +2658,7 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)if (ret)return ret; }else if (iocb->ki_flags &IOCB_DONTCACHE ) { -struct address_space *mapping = iocb->ki_filp->f_mapping; - -filemap_flush_range (mapping, iocb->ki_pos - count, - iocb->ki_pos -1 ); +filemap_dontcache_kick_writeback (iocb->ki_filp->f_mapping); }return count;include/trace/events/writeback.h +2
-1@@ -44,7 +44,8 @@ EM( WB_REASON_PERIODIC, "periodic") \ EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ EM( WB_REASON_FORKER_THREAD, "forker_thread") \ - EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush") + EM( WB_REASON_FOREIGN_FLUSH, "foreign_flush") \ + EMe(WB_REASON_DONTCACHE, "dontcache")WB_WORK_REASON Signature · 2 lines
-- 2.54.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jeff Layton
maintainerRe: [PATCH v4 2/4] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
Quoted 111 lines · Jan Kara
Quoted 75 lines · Jeff Layton
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.To avoid flushing unrelated buffered dirty data, add a dedicated
WB_start_dontcachebit andwb_check_start_dontcache()handler that uses the per-wbWB_DONTCACHE_DIRTYcounter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.Like
WB_start_all, theWB_start_dontcachebit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations.Also add
WB_REASON_DONTCACHEas a new writeback reason for tracing visibility, and target the correct cgroup writeback domain viaunlocked_inode_to_wb_begin().dontcache-bench results (same host,
T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fioio_uring):Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Changeseq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/sPreviously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 8 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
Assisted-by Claude:claude-opus-4-6Signed-off-by Jeff Layton <[email protected]>Nice and looks good to me now. Feel free to add:
Reviewed-by Jan Kara <[email protected]>One nit below:
Quoted 8 lines
+/**
+ *filemap_dontcache_kick_writeback- kick flusher forIOCB_DONTCACHEwrites
+ * @mapping:address_spacethat was just written to
+ *
+ * Kick the writeback flusher thread to expedite writeback of dontcache
+ * dirty pages. Uses a dedicatedWB_start_dontcachebit so that only
+ * pages tracked byWB_DONTCACHE_DIRTYare written back, rather than
+ * flushing the entire BDI's dirty pages.This comment is a bit confusing as in fact we write arbitrary dirty pages. It is only the amount of pages that is influenced by
WB_DONTCACHE_DIRTY. So I'd rephrase the last sentence like: We queue writeback for the inode's wb for as many pages as there are dontcache pages but we don't restrict writeback to dontcache pages only. This significantly improves performance over either writing all wb's pages or writing only dontcache pages. Although it doesn't guarantee quick writeback and reclaim of dontcache pages it keeps the amount of dirty pages in check and over longer term dontcache pages get written and reclaimed by background writeback even with this rough heuristic.
Quoted 2 lines
Honza
I'll add that. Thanks for the suggestion and review!
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerPATCH v4 3/4 testing: add nfsd-io-bench NFS server benchmark suite
In reply to Jeff LaytonAdd a benchmark suite for testing NFSD I/O mode performance using fio with the libnfs backend against an NFS server on localhost. Tests buffered, dontcache, and direct I/O modes via NFSD debugfs controls.
Includes:
- fio job files for sequential/random read/write, multi-writer, noisy-neighbor, and latency-sensitive reader workloads
- run-benchmarks.sh: orchestrates test matrix with mode switching
- parse-results.sh: extracts metrics from fio JSON output
- setup-server.sh: configures NFS export for testing
--- .../testing/nfsd-io-bench/fio-jobs/lat-reader.fio | 15 + .../testing/nfsd-io-bench/fio-jobs/multi-write.fio | 14 + .../nfsd-io-bench/fio-jobs/noisy-writer.fio | 14 + tools/testing/nfsd-io-bench/fio-jobs/rand-read.fio | 15 + .../testing/nfsd-io-bench/fio-jobs/rand-write.fio | 15 + tools/testing/nfsd-io-bench/fio-jobs/seq-read.fio | 14 + tools/testing/nfsd-io-bench/fio-jobs/seq-write.fio | 14 + .../testing/nfsd-io-bench/scripts/parse-results.sh | 238 +++++++++ .../nfsd-io-bench/scripts/run-benchmarks.sh | 591 +++++++++++++++++++++ .../testing/nfsd-io-bench/scripts/setup-server.sh | 94 ++++ 10 files changed, 1024 insertions(+)
tools/testing/nfsd-io-bench/fio-jobs/lat-reader.fio new file +15 -0
@@ -0,0 +1,15 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=4k +numjobs=16 +runtime=300 +time_based=1 +group_reporting=1 +rw=randread +log_avg_msec=1000 +write_bw_log=latreader +write_lat_log=latreader + +[lat_reader]
tools/testing/nfsd-io-bench/fio-jobs/multi-write.fio new file +14 -0
@@ -0,0 +1,14 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=1M +numjobs=16 +time_based=0 +group_reporting=1 +rw=write +log_avg_msec=1000 +write_bw_log=multiwrite +write_lat_log=multiwrite + +[writer]
tools/testing/nfsd-io-bench/fio-jobs/noisy-writer.fio new file +14 -0
@@ -0,0 +1,14 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=1M +numjobs=16 +time_based=0 +group_reporting=1 +rw=write +log_avg_msec=1000 +write_bw_log=noisywriter +write_lat_log=noisywriter + +[bulk_writer]
tools/testing/nfsd-io-bench/fio-jobs/rand-read.fio new file +15 -0
@@ -0,0 +1,15 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=4k +numjobs=16 +runtime=300 +time_based=1 +group_reporting=1 +rw=randread +log_avg_msec=1000 +write_bw_log=randread +write_lat_log=randread + +[randread]
tools/testing/nfsd-io-bench/fio-jobs/rand-write.fio new file +15 -0
@@ -0,0 +1,15 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=64k +numjobs=16 +runtime=300 +time_based=1 +group_reporting=1 +rw=randwrite +log_avg_msec=1000 +write_bw_log=randwrite +write_lat_log=randwrite + +[randwrite]
tools/testing/nfsd-io-bench/fio-jobs/seq-read.fio new file +14 -0
@@ -0,0 +1,14 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=1M +numjobs=16 +time_based=0 +group_reporting=1 +rw=read +log_avg_msec=1000 +write_bw_log=seqread +write_lat_log=seqread + +[seqread]
tools/testing/nfsd-io-bench/fio-jobs/seq-write.fio new file +14 -0
@@ -0,0 +1,14 @@ +[global] +ioengine=nfs +nfs_url=nfs://localhost/export +direct=0 +bs=1M +numjobs=16 +time_based=0 +group_reporting=1 +rw=write +log_avg_msec=1000 +write_bw_log=seqwrite +write_lat_log=seqwrite + +[seqwrite]
tools/testing/nfsd-io-bench/scripts/parse-results.sh new file +238 -0
238 lines not on this page · raw message
tools/testing/nfsd-io-bench/scripts/run-benchmarks.sh new file +591 -0
591 lines not on this page · raw message
tools/testing/nfsd-io-bench/scripts/setup-server.sh new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.54.0
Jeff Layton
maintainerPATCH v4 4/4 testing: add dontcache-bench local filesystem benchmark suite
In reply to Jeff LaytonAdd a benchmark suite for testing IOCB_DONTCACHE on local filesystems via fio's io_uring engine with the RWF_DONTCACHE flag.
The suite mirrors the nfsd-io-bench test matrix but uses io_uring with the "uncached" fio option instead of NFSD debugfs mode switching:
- uncached=0: standard buffered I/O
- uncached=1: RWF_DONTCACHE
- Mode 2 uses O_DIRECT via fio's --direct=1
Additionally, this benchmark includes:
- a benchmark for competing buffered vs. dontcache writers to different files on the same backing device.
- a benchmark mirroring Jens Axboe's original RWF_UNCACHED write test:
32 concurrent writers with 64K block size, time-based (300s), with per-second bandwidth logging
Includes fio job files, run-benchmarks.sh, and parse-results.sh.
--- .../dontcache-bench/fio-jobs/axboe-write.fio | 14 + .../dontcache-bench/fio-jobs/lat-reader.fio | 12 + .../dontcache-bench/fio-jobs/multi-write.fio | 11 + .../dontcache-bench/fio-jobs/noisy-writer.fio | 12 + .../testing/dontcache-bench/fio-jobs/rand-read.fio | 13 + .../dontcache-bench/fio-jobs/rand-write.fio | 13 + .../testing/dontcache-bench/fio-jobs/seq-read.fio | 13 + .../testing/dontcache-bench/fio-jobs/seq-write.fio | 13 + .../dontcache-bench/scripts/parse-results.sh | 346 +++++++++++ .../dontcache-bench/scripts/run-benchmarks.sh | 643 +++++++++++++++++++++ 10 files changed, 1090 insertions(+)
tools/testing/dontcache-bench/fio-jobs/axboe-write.fio new file +14 -0
@@ -0,0 +1,14 @@ +[global] +ioengine=io_uring +direct=0 +bs=64k +numjobs=32 +time_based +runtime=300 +rw=write +group_reporting=0 +filename_format=$jobname.$jobnum +log_avg_msec=1000 +write_bw_log=axboe-write + +[axboe-write]
tools/testing/dontcache-bench/fio-jobs/lat-reader.fio new file +12 -0
@@ -0,0 +1,12 @@ +[global] +ioengine=io_uring +direct=0 +bs=4k +numjobs=1 +time_based=0 +rw=read +log_avg_msec=1000 +write_bw_log=latreader +write_lat_log=latreader + +[latreader]
tools/testing/dontcache-bench/fio-jobs/multi-write.fio new file +11 -0
@@ -0,0 +1,11 @@ +[global] +ioengine=io_uring +direct=0 +bs=1M +numjobs=4 +time_based=0 +rw=write +group_reporting=0 +filename_format=$jobname.$jobnum + +[multiwrite]
tools/testing/dontcache-bench/fio-jobs/noisy-writer.fio new file +12 -0
@@ -0,0 +1,12 @@ +[global] +ioengine=io_uring +direct=0 +bs=1M +numjobs=1 +time_based=0 +rw=write +log_avg_msec=1000 +write_bw_log=noisywriter +write_lat_log=noisywriter + +[noisywriter]
tools/testing/dontcache-bench/fio-jobs/rand-read.fio new file +13 -0
@@ -0,0 +1,13 @@ +[global] +ioengine=io_uring +direct=0 +bs=4k +numjobs=1 +iodepth=16 +time_based=0 +rw=randread +log_avg_msec=1000 +write_bw_log=randread +write_lat_log=randread + +[randread]
tools/testing/dontcache-bench/fio-jobs/rand-write.fio new file +13 -0
@@ -0,0 +1,13 @@ +[global] +ioengine=io_uring +direct=0 +bs=4k +numjobs=1 +iodepth=16 +time_based=0 +rw=randwrite +log_avg_msec=1000 +write_bw_log=randwrite +write_lat_log=randwrite + +[randwrite]
tools/testing/dontcache-bench/fio-jobs/seq-read.fio new file +13 -0
@@ -0,0 +1,13 @@ +[global] +ioengine=io_uring +direct=0 +bs=1M +numjobs=1 +iodepth=16 +time_based=0 +rw=read +log_avg_msec=1000 +write_bw_log=seqread +write_lat_log=seqread + +[seqread]
tools/testing/dontcache-bench/fio-jobs/seq-write.fio new file +13 -0
@@ -0,0 +1,13 @@ +[global] +ioengine=io_uring +direct=0 +bs=1M +numjobs=1 +iodepth=16 +time_based=0 +rw=write +log_avg_msec=1000 +write_bw_log=seqwrite +write_lat_log=seqwrite + +[seqwrite]
tools/testing/dontcache-bench/scripts/parse-results.sh new file +346 -0
346 lines not on this page · raw message
tools/testing/dontcache-bench/scripts/run-benchmarks.sh new file +643 -0
643 lines not on this page · raw message
Signature · 1 line
2.54.0