mm: improve write performance with RWF_DONTCACHE
Subsystems (6)
Jeff Layton
maintainerv6 cover mm: improve write performance with RWF_DONTCACHE
This patch series is intended to improve write performance with RWF_DONTCACHE. After sending v5 yesterday, I decided to check what Sasiko thought about this series, and lo and behold it had found a number of bugs that I had missed before. This set should fix them. Performance is identical to the v5 set.
Because there are some substantive changes in this set, I've dropped the R-b's. Please resend them if you're OK with this version.
Christian, please consider these for v7.2.
Thanks,
---
Changes in v6:
- Use atomic folio_test_clear_dropbehind() in __filemap_get_folio_mpol() to prevent double-decrement of WB_DONTCACHE_DIRTY by concurrent readers
- Add mapping_can_writeback() guard before decrementing WB_DONTCACHE_DIRTY in __filemap_get_folio_mpol() to match the increment path
- Use wb_stat_sum() instead of wb_stat() in wb_check_start_dontcache() so small writes below the percpu batch threshold are visible to the flusher
- Use test_and_clear_bit for WB_start_dontcache before starting writeback to prevent lost wakeups from concurrent DONTCACHE writers
- Move wb_wakeup() outside the unlocked_inode_to_wb_begin/end section in filemap_dontcache_kick_writeback() to avoid spin_unlock_irq() re-enabling interrupts while the i_pages xa_lock is held during cgroup writeback switch
- Drop Reviewed-by tags due to substantive changes
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Flesh out comment over filemap_dontcache_kick_writeback()
- Drop testcases from posting
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Track DONTCACHE dirty pages per bdi_writeback
- New benchmark for competing buffered and dontcache writers
- New benchmark replicating Jens' original 32 concurrent writer test
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Track dirty DONTCACHE pages in the VM
- Have flusher write back a proportional number of pages after DONTCACHE write
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- kick flusher thread instead of initiating writeback inline
- add mechanism to run 'perf lock' around the testcases
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Jeff Layton (2):
mm: track DONTCACHE dirty pages per bdi_writeback
mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
fs/fs-writeback.c | 59 ++++++++++++++++++++++++++++++++++++++++ include/linux/backing-dev-defs.h | 3 ++ include/linux/fs.h | 6 ++-- include/trace/events/writeback.h | 3 +- mm/filemap.c | 15 ++++++++-- mm/page-writeback.c | 6 ++++ 6 files changed, 85 insertions(+), 7 deletions(-)
---
base-commit: 7e2326f4275c11652e1fdaae11de06159fef1d90
change-id: 20260401-dontcache-5811efd7eaf3
Best regards,
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerPATCH v6 1/2 mm: track DONTCACHE dirty pages per bdi_writeback
Add a per-wb WB_DONTCACHE_DIRTY counter that tracks the number of dirty pages with the dropbehind flag set (i.e., pages dirtied via RWF_DONTCACHE writes).
Increment the counter alongside WB_RECLAIMABLE in folio_account_dirtied() when the folio has the dropbehind flag set, and decrement it in folio_clear_dirty_for_io() and folio_account_cleaned(). Also decrement it when a non-DONTCACHE lookup atomically clears the dropbehind flag on a dirty folio in __filemap_get_folio_mpol(), using folio_test_clear_dropbehind() to prevent concurrent lookups from double-decrementing the counter, and guarding the decrement with mapping_can_writeback() to match the increment path.
The counter will be used by the writeback flusher to determine how many pages to write back when expediting writeback for IOCB_DONTCACHE writes, without flushing the entire BDI's dirty pages.
--- include/linux/backing-dev-defs.h | 1 + mm/filemap.c | 15 +++++++++++++-- mm/page-writeback.c | 6 ++++++ 3 files changed, 20 insertions(+), 2 deletions(-)
include/linux/backing-dev-defs.h +1 -0
@@ -33,6 +33,7 @@ enum wb_stat_item {WB_WRITEBACK ,WB_DIRTIED ,WB_WRITTEN , +WB_DONTCACHE_DIRTY , NR_WB_STAT_ITEMS };
mm/filemap.c +13 -2
@@ -2052,8 +2052,19 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) -folio_clear_dropbehind (folio); +if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_clear_dropbehind (folio)) { +if (folio_test_dirty (folio) && +mapping_can_writeback (mapping)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +unlocked_inode_to_wb_end (inode, &cookie); + } + }return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);
mm/page-writeback.c +6 -0
@@ -2630,6 +2630,8 @@ static void folio_account_dirtied(struct folio *folio, wb =inode_to_wb (inode);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , nr);__zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr);__node_stat_mod_folio (folio,NR_DIRTIED , nr);wb_stat_mod (wb,WB_RECLAIMABLE , nr);
@@ -2651,6 +2653,8 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr);task_io_account_cancelled_write (nr *PAGE_SIZE );
@@ -2920,6 +2924,8 @@ bool folio_clear_dirty_for_io(struct folio *folio)if (folio_test_clear_dirty (folio)) {long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr); ret = true;
Signature · 1 line
2.54.0
Re: [PATCH v6 1/2] mm: track DONTCACHE dirty pages per bdi_writeback
Quoted 13 lines · Jeff Layton
Add a per-wb
WB_DONTCACHE_DIRTYcounter that tracks the number of dirty pages with the dropbehind flag set (i.e., pages dirtied viaRWF_DONTCACHEwrites).Increment the counter alongside
WB_RECLAIMABLEinfolio_account_dirtied()when the folio has the dropbehind flag set, and decrement it infolio_clear_dirty_for_io()andfolio_account_cleaned(). Also decrement it when a non-DONTCACHE lookup atomically clears the dropbehind flag on a dirty folio in__filemap_get_folio_mpol(), usingfolio_test_clear_dropbehind()to prevent concurrent lookups from double-decrementing the counter, and guarding the decrement withmapping_can_writeback()to match the increment path.
Quoted 7 lines
The counter will be used by the writeback flusher to determine how many pages to write back when expediting writeback for
IOCB_DONTCACHEwrites, without flushing the entire BDI's dirty pages.Suggested-by Jan Kara <[email protected]>Assisted-by Claude:claude-opus-4-6Signed-off-by Jeff Layton <[email protected]>
Yeah, good catch with the folio_test_clear_dropbehind(). Feel free to add:
Honza
Quoted 79 lines
--- include/linux/backing-dev-defs.h | 1 + mm/filemap.c | 15 +++++++++++++-- mm/page-writeback.c | 6 ++++++ 3 files changed, 20 insertions(+), 2 deletions(-)include/linux/backing-dev-defs.h +1
-0@@ -33,6 +33,7 @@ enum wb_stat_item {WB_WRITEBACK ,WB_DIRTIED ,WB_WRITTEN , +WB_DONTCACHE_DIRTY , NR_WB_STAT_ITEMS };mm/filemap.c +13
-2@@ -2052,8 +2052,19 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,if (!folio)return ERR_PTR (-ENOENT );/* not an uncached lookup, clear uncached if set */ -if (folio_test_dropbehind (folio) && !(fgp_flags &FGP_DONTCACHE )) -folio_clear_dropbehind (folio); +if (!(fgp_flags &FGP_DONTCACHE ) &&folio_test_clear_dropbehind (folio)) { +if (folio_test_dirty (folio) && +mapping_can_writeback (mapping)) { +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , + -folio_nr_pages (folio)); +unlocked_inode_to_wb_end (inode, &cookie); + } + }return folio; }EXPORT_SYMBOL (__filemap_get_folio_mpol);mm/page-writeback.c +6
-0@@ -2630,6 +2630,8 @@ static void folio_account_dirtied(struct folio *folio, wb =inode_to_wb (inode);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , nr);__zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr);__node_stat_mod_folio (folio,NR_DIRTIED , nr);wb_stat_mod (wb,WB_RECLAIMABLE , nr);@@ -2651,6 +2653,8 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr);task_io_account_cancelled_write (nr *PAGE_SIZE );@@ -2920,6 +2924,8 @@ bool folio_clear_dirty_for_io(struct folio *folio)if (folio_test_clear_dirty (folio)) {long nr =folio_nr_pages (folio);lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); +if (folio_test_dropbehind (folio)) +wb_stat_mod (wb,WB_DONTCACHE_DIRTY , -nr);zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr);wb_stat_mod (wb,WB_RECLAIMABLE , -nr); ret = true;Signature · 2 lines
-- 2.54.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jeff Layton
maintainerPATCH v6 2/2 mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
In reply to Jeff LaytonThe IOCB_DONTCACHE writeback path in generic_write_sync() calls filemap_flush_range() on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).
Replace the inline filemap_flush_range() call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.
To avoid flushing unrelated buffered dirty data, add a dedicated WB_start_dontcache bit and wb_check_start_dontcache() handler that uses the per-wb WB_DONTCACHE_DIRTY counter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.
Like WB_start_all, the WB_start_dontcache bit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations. Use test_and_clear_bit to atomically consume the kick request before reading the dirty counter and starting writeback, so that concurrent DONTCACHE writes during writeback can re-set the bit and schedule a follow-up flusher run.
Read the dirty counter with wb_stat_sum() (aggregating per-CPU batches) rather than wb_stat() (which reads only the global counter) to ensure small writes below the percpu batch threshold are visible to the flusher.
In filemap_dontcache_kick_writeback(), set the WB_start_dontcache bit inside the unlocked_inode_to_wb_begin/end section for correct cgroup writeback domain targeting, but defer the wb_wakeup() call until after the section ends, since wb_wakeup() uses spin_unlock_irq() which would unconditionally re-enable interrupts while the i_pages xa_lock may still be held under irqsave during a cgroup writeback switch.
Also add WB_REASON_DONTCACHE as a new writeback reason for tracing visibility.
dontcache-bench results (same host, T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fio io_uring):
Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Change
seq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%
Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)
Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%
Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%
Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before After
buffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/s
Previously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
--- fs/fs-writeback.c | 59 ++++++++++++++++++++++++++++++++++++++++ include/linux/backing-dev-defs.h | 2 ++ include/linux/fs.h | 6 ++-- include/trace/events/writeback.h | 3 +- 4 files changed, 65 insertions(+), 5 deletions(-)
fs/fs-writeback.c +59 -0
@@ -2373,6 +2373,27 @@ static long wb_check_start_all(struct bdi_writeback *wb)return nr_pages; } +static long wb_check_start_dontcache (struct bdi_writeback *wb) +{ +long nr_pages; + +if (!test_and_clear_bit (WB_start_dontcache, &wb->state)) +return 0 ; + + nr_pages =wb_stat_sum (wb,WB_DONTCACHE_DIRTY ); +if (nr_pages) { +struct wb_writeback_work work = { + .nr_pages = nr_pages, + .sync_mode =WB_SYNC_NONE , + .range_cyclic =1 , + .reason =WB_REASON_DONTCACHE , + }; + + nr_pages =wb_writeback (wb, &work); + } + +return nr_pages; +}/* * Retrieve work items and do the writeback they describe
@@ -2394,6 +2415,11 @@ static long wb_do_writeback(struct bdi_writeback *wb)*/ wrote +=wb_check_start_all (wb); +/* +* Check for dontcache writeback request +*/ + wrote +=wb_check_start_dontcache (wb); +/* * Check for periodic writeback, kupdated() style */
@@ -2468,6 +2494,39 @@ void wakeup_flusher_threads_bdi(struct backing_dev_info *bdi,rcu_read_unlock (); } +/** +* filemap_dontcache_kick_writeback - kick flusher for IOCB_DONTCACHE writes +* @mapping: address_space that was just written to +* +* Kick the writeback flusher thread to expedite writeback of dontcache dirty +* pages. Queue writeback for the inode's wb for as many pages as there are +* dontcache pages, but don't restrict writeback to dontcache pages only. +* +* This significantly improves performance over either writing all wb's pages +* or writing only dontcache pages. Although it doesn't guarantee quick +* writeback and reclaim of dontcache pages, it keeps the amount of dirty pages +* in check. Over longer term dontcache pages get written and reclaimed by +* background writeback even with this rough heuristic. +*/ +void filemap_dontcache_kick_writeback (struct address_space *mapping) +{ +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; +bool need_wakeup = false; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +if (wb_has_dirty_io (wb) && + !test_bit (WB_start_dontcache, &wb->state) && + !test_and_set_bit (WB_start_dontcache, &wb->state)) + need_wakeup = true; +unlocked_inode_to_wb_end (inode, &cookie); + +if (need_wakeup) +wb_wakeup (wb); +} +EXPORT_SYMBOL_GPL (filemap_dontcache_kick_writeback); +/* * Wakeup the flusher threads to start writeback of all currently dirty pages */
include/linux/backing-dev-defs.h +2 -0
@@ -26,6 +26,7 @@ enum wb_state { WB_writeback_running,/* Writeback is in progress */ WB_has_dirty_io,/* Dirty inodes on ->b_{dirty|io|more_io} */ WB_start_all,/* nr_pages == 0 (all) work pending */ + WB_start_dontcache, /* dontcache writeback pending */ }; enum wb_stat_item {
@@ -56,6 +57,7 @@ enum wb_reason {*/ WB_REASON_FORKER_THREAD ,WB_REASON_FOREIGN_FLUSH , +WB_REASON_DONTCACHE , WB_REASON_MAX, };
include/linux/fs.h +2 -4
@@ -2624,6 +2624,7 @@ extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end);int filemap_flush_range (struct address_space *mapping,loff_t start,loff_t end); +void filemap_dontcache_kick_writeback (struct address_space *mapping);static inline int file_write_and_wait (struct file *file) {
@@ -2657,10 +2658,7 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)if (ret)return ret; }else if (iocb->ki_flags &IOCB_DONTCACHE ) { -struct address_space *mapping = iocb->ki_filp->f_mapping; - -filemap_flush_range (mapping, iocb->ki_pos - count, - iocb->ki_pos -1 ); +filemap_dontcache_kick_writeback (iocb->ki_filp->f_mapping); }return count;
include/trace/events/writeback.h +2 -1
@@ -44,7 +44,8 @@ EM( WB_REASON_PERIODIC, "periodic") \ EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ EM( WB_REASON_FORKER_THREAD, "forker_thread") \ - EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush") + EM( WB_REASON_FOREIGN_FLUSH, "foreign_flush") \ + EMe(WB_REASON_DONTCACHE, "dontcache")WB_WORK_REASON
Signature · 1 line
2.54.0
Re: [PATCH v6 2/2] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
Quoted 88 lines · Jeff Layton
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path.To avoid flushing unrelated buffered dirty data, add a dedicated
WB_start_dontcachebit andwb_check_start_dontcache()handler that uses the per-wbWB_DONTCACHE_DIRTYcounter to determine how many pages to write back. The flusher writes back that many pages from the oldest dirty inodes (not restricted to dontcache-specific inodes). This helps preserve I/O batching while limiting the scope of expedited writeback.Like
WB_start_all, theWB_start_dontcachebit coalesces multiple DONTCACHE writes into a single flusher wakeup without per-write allocations. Usetest_and_clear_bitto atomically consume the kick request before reading the dirty counter and starting writeback, so that concurrent DONTCACHE writes during writeback can re-set the bit and schedule a follow-up flusher run.Read the dirty counter with
wb_stat_sum()(aggregating per-CPU batches) rather thanwb_stat()(which reads only the global counter) to ensure small writes below the percpu batch threshold are visible to the flusher.In
filemap_dontcache_kick_writeback(), set theWB_start_dontcachebit inside the unlocked_inode_to_wb_begin/end section for correct cgroup writeback domain targeting, but defer thewb_wakeup()call until after the section ends, sincewb_wakeup()usesspin_unlock_irq()which would unconditionally re-enable interrupts while thei_pagesxa_lockmay still be held under irqsave during a cgroup writeback switch.Also add
WB_REASON_DONTCACHEas a new writeback reason for tracing visibility.dontcache-bench results (same host,
T6F_SKL_1920GBF, 251 GiB RAM, xfs on NVMe, fioio_uring):Buffered and direct I/O paths are unaffected by this patchset. All improvements are confined to the dontcache path:
Single-stream throughput (MB/s):
Before After Changeseq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%Dontcache multi-writer throughput now matches buffered (4,532 vs 4,616 MB/s).
32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/sPreviously the buffered writer starved the dontcache writer 2:1. With per-bdi_writeback tracking, both writers now receive equal bandwidth. The aggregate matches the buffered-vs-buffered baseline (863 MB/s), indicating fair sharing regardless of I/O mode.
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 8 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
Assisted-by Claude:claude-opus-4-6Signed-off-by Jeff Layton <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 153 lines
--- fs/fs-writeback.c | 59 ++++++++++++++++++++++++++++++++++++++++ include/linux/backing-dev-defs.h | 2 ++ include/linux/fs.h | 6 ++-- include/trace/events/writeback.h | 3 +- 4 files changed, 65 insertions(+), 5 deletions(-)fs/fs-writeback.c +59
-0@@ -2373,6 +2373,27 @@ static long wb_check_start_all(struct bdi_writeback *wb)return nr_pages; } +static long wb_check_start_dontcache (struct bdi_writeback *wb) +{ +long nr_pages; + +if (!test_and_clear_bit (WB_start_dontcache, &wb->state)) +return 0 ; + + nr_pages =wb_stat_sum (wb,WB_DONTCACHE_DIRTY ); +if (nr_pages) { +struct wb_writeback_work work = { + .nr_pages = nr_pages, + .sync_mode =WB_SYNC_NONE , + .range_cyclic =1 , + .reason =WB_REASON_DONTCACHE , + }; + + nr_pages =wb_writeback (wb, &work); + } + +return nr_pages; +}/* * Retrieve work items and do the writeback they describe @@ -2394,6 +2415,11 @@ static long wb_do_writeback(struct bdi_writeback *wb)*/ wrote +=wb_check_start_all (wb); +/* +* Check for dontcache writeback request +*/ + wrote +=wb_check_start_dontcache (wb); +/* * Check for periodic writeback, kupdated() style */ @@ -2468,6 +2494,39 @@ void wakeup_flusher_threads_bdi(struct backing_dev_info *bdi,rcu_read_unlock (); } +/** +* filemap_dontcache_kick_writeback - kick flusher for IOCB_DONTCACHE writes +* @mapping: address_space that was just written to +* +* Kick the writeback flusher thread to expedite writeback of dontcache dirty +* pages. Queue writeback for the inode's wb for as many pages as there are +* dontcache pages, but don't restrict writeback to dontcache pages only. +* +* This significantly improves performance over either writing all wb's pages +* or writing only dontcache pages. Although it doesn't guarantee quick +* writeback and reclaim of dontcache pages, it keeps the amount of dirty pages +* in check. Over longer term dontcache pages get written and reclaimed by +* background writeback even with this rough heuristic. +*/ +void filemap_dontcache_kick_writeback (struct address_space *mapping) +{ +struct inode *inode = mapping->host; +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; +bool need_wakeup = false; + + wb =unlocked_inode_to_wb_begin (inode, &cookie); +if (wb_has_dirty_io (wb) && + !test_bit (WB_start_dontcache, &wb->state) && + !test_and_set_bit (WB_start_dontcache, &wb->state)) + need_wakeup = true; +unlocked_inode_to_wb_end (inode, &cookie); + +if (need_wakeup) +wb_wakeup (wb); +} +EXPORT_SYMBOL_GPL (filemap_dontcache_kick_writeback); +/* * Wakeup the flusher threads to start writeback of all currently dirty pages */ include/linux/backing-dev-defs.h +2
-0@@ -26,6 +26,7 @@ enum wb_state { WB_writeback_running,/* Writeback is in progress */ WB_has_dirty_io,/* Dirty inodes on ->b_{dirty|io|more_io} */ WB_start_all,/* nr_pages == 0 (all) work pending */ + WB_start_dontcache, /* dontcache writeback pending */ }; enum wb_stat_item {@@ -56,6 +57,7 @@ enum wb_reason {*/ WB_REASON_FORKER_THREAD ,WB_REASON_FOREIGN_FLUSH , +WB_REASON_DONTCACHE , WB_REASON_MAX, };include/linux/fs.h +2
-4@@ -2624,6 +2624,7 @@ extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end);int filemap_flush_range (struct address_space *mapping,loff_t start,loff_t end); +void filemap_dontcache_kick_writeback (struct address_space *mapping);static inline int file_write_and_wait (struct file *file) {@@ -2657,10 +2658,7 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)if (ret)return ret; }else if (iocb->ki_flags &IOCB_DONTCACHE ) { -struct address_space *mapping = iocb->ki_filp->f_mapping; - -filemap_flush_range (mapping, iocb->ki_pos - count, - iocb->ki_pos -1 ); +filemap_dontcache_kick_writeback (iocb->ki_filp->f_mapping); }return count;include/trace/events/writeback.h +2
-1@@ -44,7 +44,8 @@ EM( WB_REASON_PERIODIC, "periodic") \ EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ EM( WB_REASON_FORKER_THREAD, "forker_thread") \ - EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush") + EM( WB_REASON_FOREIGN_FLUSH, "foreign_flush") \ + EMe(WB_REASON_DONTCACHE, "dontcache")WB_WORK_REASON Signature · 2 lines
-- 2.54.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Andrew Morton
maintainerRe: [PATCH v6 2/2] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
In reply to Jeff LaytonQuoted 49 lines · On Tue, 05 May 2026 20:59:49 +0200 Jeff Layton <[email protected]> wrote:
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path....
Before After Change
seq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/s...
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 5 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
Geeze, is that the best you can do ;)
Sashiko seems to have found more stuff:
https://sashiko.dev/#/patchset/[email protected]
Jeff Layton
maintainerRe: [PATCH v6 2/2] mm: kick writeback flusher for IOCB_DONTCACHE with targeted dirty tracking
Quoted 57 lines · Andrew Morton
Quoted 49 lines · On Tue, 05 May 2026 20:59:49 +0200 Jeff Layton <[email protected]> wrote:
The
IOCB_DONTCACHEwriteback path ingeneric_write_sync()callsfilemap_flush_range()on every write, submitting writeback inline in the writer's context. Perf lock contention profiling shows the performance problem is not lock contention but the writeback submission work itself — walking the page tree and submitting I/O blocks the writer for milliseconds, inflating p99.9 latency from 23ms (buffered) to 93ms (dontcache).Replace the inline
filemap_flush_range()call with a flusher kick that drains dirty pages in the background. This moves writeback submission completely off the writer's hot path....
Before After Change
seq-write/dontcache 298 897 +201% rand-write/dontcache 131 236 +80%Tail latency improvements (seq-write/dontcache):
p99: 135,266 us -> 23,986 us (-82%) p99.9: 8,925,479 us -> 28,443 us (-99.7%)Multi-writer (4 jobs, sequential write):
Before After Change dontcache aggregate (MB/s) 2,529 4,532 +79% dontcache p99 (us) 8,553 1,002 -88% dontcache p99.9 (us) 109,314 1,057 -99%32-file write (Axboe test):
Before After Change dontcache aggregate (MB/s) 1,548 3,499 +126% dontcache p99 (us) 10,170 602 -94% Peak dirty pages (MB) 1,837 213 -88%Dontcache now reaches 81% of buffered throughput (was 35%).
Competing writers (dontcache vs buffered, separate files):
Before Afterbuffered writer 868 433 MB/s dontcache writer 415 433 MB/s Aggregate 1,284 866 MB/s...
The dontcache writer's p99.9 latency collapsed from 119 ms to 33 ms (-73%), eliminating the severe periodic stalls seen in the baseline. Both writers now share identical latency profiles, matching the buffered-vs-buffered pattern.
Quoted 5 lines
The per-bdi_writeback dirty tracking dramatically reduces peak dirty pages in dontcache workloads, with the 32-file test dropping from 1.8 GB to 213 MB. Dontcache sequential write throughput triples and multi-writer throughput reaches parity with buffered I/O, with tail latencies collapsing by 1-2 orders of magnitude.
Quoted 4 lines
Geeze, is that the best you can do ;)
Sashiko seems to have found more stuff:
https://sashiko.dev/#/patchset/[email protected]
I saw those after I sent the last set and have been working on addressing them. I've also found a couple more via dueling Gemini and Claude reviews.
I'll have a v7 posting coming early next week.
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>