mm/iomap: add granular dirty and writeback accounting
Subsystems (6)
cover mm/iomap: add granular dirty and writeback accounting
This patchset is a stab at adding granular dirty and writeback stats accounting for large folios.
The dirty page balancing logic uses these stats to determine things like whether the ratelimit has been exceeded, the frequency with which pages need to be written back, if dirtying should be throttled, etc. Currently for large folios, if any byte in the folio is dirtied or written back, all the bytes in the folio are accounted as such.
In particular, there are four places where dirty and writeback stats get incremented and decremented as pages get dirtied and written back:
a) folio dirtying (filemap_dirty_folio() -> ... -> folio_account_dirtied())
- increments NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING, WB_RECLAIMABLE,
current->nr_dirtied
b) writing back a mapping (writeback_iter() -> ... -> folio_clear_dirty_for_io())
- decrements NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING, WB_RECLAIMABLE
c) starting writeback on a folio (folio_start_writeback())
- increments WB_WRITEBACK, NR_WRITEBACK, NR_ZONE_WRITE_PENDING
d) ending writeback on a folio (folio_end_writeback())
- decrements WB_WRITEBACK, NR_WRITEBACK, NR_ZONE_WRITE_PENDING
Patches 1 to 9 adds support for the 4 cases above to take in the number of pages to be accounted, instead of accounting for the entire folio.
Patch 10 adds the iomap changes that uses these new APIs. This relies on the iomap folio state bitmap to track which pages are dirty (so that we avoid any double-counting). As such we can only do granular accounting if the block size >= PAGE_SIZE.
This patchset was run through xfstests using fuse passthrough hp (with an out-of-tree kernel patch enabling fuse large folios).
This is on top of commit d5212d81 ("Merge patch series "fuse: use iomap..."") in Christian's vfs iomap tree, and on top of the patchset that removes BDI_CAP_WRITEBACK_ACCT [1].
Benchmarks using a contrived test program that writes 2 GB in 128 MB chunks to a fuse mount (with out-of-tree kernel patch that enables fuse large folios) and then does 50k 50-byte random writes showed roughly a 10% performance improvement (0.625 seconds -> 0.547 seconds for the random writes).
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Joanne Koong (10):
mm: pass number of pages to __folio_start_writeback()
mm: pass number of pages to __folio_end_writeback()
mm: add folio_end_writeback_pages() helper
mm: pass number of pages dirtied to __folio_mark_dirty()
mm: add filemap_dirty_folio_pages() helper
mm: add __folio_clear_dirty_for_io() helper
mm: add no_stats_accounting bitfield to wbc
mm: refactor clearing dirty stats into helper function
mm: add clear_dirty_for_io_stats() helper
iomap: add granular dirty and writeback accounting
fs/buffer.c | 6 +- fs/ext4/page-io.c | 2 +- fs/iomap/buffered-io.c | 136 ++++++++++++++++++++++++++++++++++--- include/linux/page-flags.h | 6 +- include/linux/pagemap.h | 4 +- include/linux/writeback.h | 6 ++ mm/filemap.c | 25 ++++--- mm/internal.h | 2 +- mm/page-writeback.c | 127 ++++++++++++++++++++++------------ 9 files changed, 246 insertions(+), 68 deletions(-)
Signature · 1 line
2.47.3
RFC PATCH 1/10 mm: pass number of pages to __folio_start_writeback()
Add an additional arg to __folio_start_writeback() that takes in the number of pages to write back.
--- fs/ext4/page-io.c | 2 +- include/linux/page-flags.h | 6 +++--- mm/page-writeback.c | 10 +++++----- 3 files changed, 9 insertions(+), 9 deletions(-)
fs/ext4/page-io.c +1 -1
@@ -580,7 +580,7 @@ int ext4_bio_write_folio(struct ext4_io_submit *io, struct folio *folio, io_folio =page_folio (bounce_page); } -__folio_start_writeback (folio, keep_towrite); +__folio_start_writeback (folio, keep_towrite,folio_nr_pages (folio));/* Now submit buffers to write */ do {
include/linux/page-flags.h +3 -3
@@ -854,13 +854,13 @@ static __always_inline void SetPageUptodate(struct page *page) CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL) -void __folio_start_writeback (struct folio *folio,bool keep_write); +void __folio_start_writeback (struct folio *folio,bool keep_write,long nr_pages);void set_page_writeback (struct page *page);#define folio_start_writeback (folio) \ - __folio_start_writeback(folio, false) + __folio_start_writeback(folio, false, folio_nr_pages(folio))#define folio_start_writeback_keepwrite (folio) \ - __folio_start_writeback(folio, true) + __folio_start_writeback(folio, true, folio_nr_pages(folio)) static __always_inline bool folio_test_head(const struct folio *folio) {
mm/page-writeback.c +5 -5
@@ -3044,9 +3044,9 @@ bool __folio_end_writeback(struct folio *folio)return ret; } -void __folio_start_writeback (struct folio *folio,bool keep_write) +void __folio_start_writeback (struct folio *folio,bool keep_write, +long nr_pages) { -long nr =folio_nr_pages (folio);struct address_space *mapping =folio_mapping (folio);int access_ret;
@@ -3067,7 +3067,7 @@ void __folio_start_writeback(struct folio *folio, bool keep_write) on_wblist =mapping_tagged (mapping,PAGECACHE_TAG_WRITEBACK );xas_set_mark (&xas,PAGECACHE_TAG_WRITEBACK ); -wb_stat_mod (wb,WB_WRITEBACK , nr); +wb_stat_mod (wb,WB_WRITEBACK , nr_pages);if (!on_wblist) {wb_inode_writeback_start (wb);/*
@@ -3088,8 +3088,8 @@ void __folio_start_writeback(struct folio *folio, bool keep_write)folio_test_set_writeback (folio); } -lruvec_stat_mod_folio (folio,NR_WRITEBACK , nr); -zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr); +lruvec_stat_mod_folio (folio,NR_WRITEBACK , nr_pages); +zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , nr_pages); access_ret =arch_make_folio_accessible (folio);/*
Signature · 1 line
2.47.3
RFC PATCH 2/10 mm: pass number of pages to __folio_end_writeback()
In reply to Joanne KoongAdd an additional arg to __folio_end_writeback() that takes in the number of pages that were written back.
--- mm/filemap.c | 2 +- mm/internal.h | 2 +- mm/page-writeback.c | 13 ++++++------- 3 files changed, 8 insertions(+), 9 deletions(-)
mm/filemap.c +1 -1
@@ -1657,7 +1657,7 @@ void folio_end_writeback(struct folio *folio)* reused before the folio_wake_bit(). */ folio_get (folio); -if (__folio_end_writeback (folio)) +if (__folio_end_writeback (folio,folio_nr_pages (folio)))folio_wake_bit (folio, PG_writeback);filemap_end_dropbehind_write (folio);
mm/internal.h +1 -1
@@ -416,7 +416,7 @@ static inline vm_fault_t vmf_anon_prepare(struct vm_fault *vmf)vm_fault_t do_swap_page (struct vm_fault *vmf);void folio_rotate_reclaimable (struct folio *folio); -bool __folio_end_writeback (struct folio *folio); +bool __folio_end_writeback (struct folio *folio,long nr_pages);void deactivate_file_folio (struct folio *folio);void folio_activate (struct folio *folio);
mm/page-writeback.c +6 -7
@@ -3008,9 +3008,8 @@ static void wb_inode_writeback_end(struct bdi_writeback *wb)spin_unlock_irqrestore (&wb->work_lock, flags); } -bool __folio_end_writeback (struct folio *folio) +bool __folio_end_writeback (struct folio *folio,long nr_pages) { -long nr =folio_nr_pages (folio);struct address_space *mapping =folio_mapping (folio);bool ret;
@@ -3024,8 +3023,8 @@ bool __folio_end_writeback(struct folio *folio)__xa_clear_mark (&mapping->i_pages,folio_index (folio),PAGECACHE_TAG_WRITEBACK ); -wb_stat_mod (wb,WB_WRITEBACK , -nr); -__wb_writeout_add (wb, nr); +wb_stat_mod (wb,WB_WRITEBACK , -nr_pages); +__wb_writeout_add (wb, nr_pages);if (!mapping_tagged (mapping,PAGECACHE_TAG_WRITEBACK )) {wb_inode_writeback_end (wb);if (mapping->host)
@@ -3037,9 +3036,9 @@ bool __folio_end_writeback(struct folio *folio) ret =folio_xor_flags_has_waiters (folio,1 << PG_writeback); } -lruvec_stat_mod_folio (folio,NR_WRITEBACK , -nr); -zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr); -node_stat_mod_folio (folio,NR_WRITTEN , nr); +lruvec_stat_mod_folio (folio,NR_WRITEBACK , -nr_pages); +zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr_pages); +node_stat_mod_folio (folio,NR_WRITTEN , nr_pages);return ret; }
Signature · 1 line
2.47.3
RFC PATCH 3/10 mm: add folio_end_writeback_pages() helper
In reply to Joanne KoongAdd folio_end_writeback_pages() which takes in the number of pages written back.
--- include/linux/pagemap.h | 1 + mm/filemap.c | 25 +++++++++++++++---------- 2 files changed, 16 insertions(+), 10 deletions(-)
include/linux/pagemap.h +1 -0
@@ -1202,6 +1202,7 @@ void folio_wait_writeback(struct folio *folio);int folio_wait_writeback_killable (struct folio *folio);void end_page_writeback (struct page *page);void folio_end_writeback (struct folio *folio); +void folio_end_writeback_pages (struct folio *folio,long nr_pages);void folio_wait_stable (struct folio *folio);void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);
mm/filemap.c +15 -10
@@ -1626,15 +1626,7 @@ static void filemap_end_dropbehind_write(struct folio *folio) } } -/** -* folio_end_writeback - End writeback against a folio. -* @folio: The folio. -* -* The folio must actually be under writeback. -* -* Context: May be called from process or interrupt context. -*/ -void folio_end_writeback (struct folio *folio) +void folio_end_writeback_pages (struct folio *folio,long nr_pages) {VM_BUG_ON_FOLIO (!folio_test_writeback (folio), folio);
@@ -1657,13 +1649,26 @@ void folio_end_writeback(struct folio *folio)* reused before the folio_wake_bit(). */ folio_get (folio); -if (__folio_end_writeback (folio,folio_nr_pages (folio))) +if (__folio_end_writeback (folio, nr_pages))folio_wake_bit (folio, PG_writeback);filemap_end_dropbehind_write (folio);acct_reclaim_writeback (folio);folio_put (folio); } + +/** +* folio_end_writeback - End writeback against a folio. +* @folio: The folio. +* +* The folio must actually be under writeback. +* +* Context: May be called from process or interrupt context. +*/ +void folio_end_writeback (struct folio *folio) +{ +folio_end_writeback_pages (folio,folio_nr_pages (folio)); +}EXPORT_SYMBOL (folio_end_writeback);/**
Signature · 1 line
2.47.3
Re: [RFC PATCH v1 03/10] mm: add folio_end_writeback_pages() helper
mm/filemap.c static void filemap_end_dropbehind_write(struct folio *folio) · line 1629
Quoted 10 lines · Joanne Koong
mm/filemap.c +1
-9-/** -* folio_end_writeback - End writeback against a folio. -* @folio: The folio. -* -* The folio must actually be under writeback. -* -* Context: May be called from process or interrupt context. -*/ -void folio_end_writeback (struct folio *folio) +void folio_end_writeback_pages (struct folio *folio,long nr_pages)
Please keep the kerneldoc comment for the now more complicated function.
RFC PATCH 4/10 mm: pass number of pages dirtied to __folio_mark_dirty()
In reply to Joanne KoongAdd an additional arg to __folio_mark_dirty() that takes in the number of pages dirtied, so that this can be passed to folio_account_dirtied() when it updates the stats.
--- fs/buffer.c | 6 ++++-- include/linux/pagemap.h | 3 ++- mm/page-writeback.c | 10 +++++----- 3 files changed, 11 insertions(+), 8 deletions(-)
fs/buffer.c +4 -2
@@ -751,7 +751,8 @@ bool block_dirty_folio(struct address_space *mapping, struct folio *folio)spin_unlock (&mapping->i_private_lock);if (newly_dirty) -__folio_mark_dirty (folio, mapping,1 ); +__folio_mark_dirty (folio, mapping,1 , +folio_nr_pages (folio));if (newly_dirty)__mark_inode_dirty (mapping->host,I_DIRTY_PAGES );
@@ -1209,7 +1210,8 @@ void mark_buffer_dirty(struct buffer_head *bh)if (!folio_test_set_dirty (folio)) { mapping = folio->mapping;if (mapping) -__folio_mark_dirty (folio, mapping,0 ); +__folio_mark_dirty (folio, mapping,0 , +folio_nr_pages (folio)); }if (mapping)__mark_inode_dirty (mapping->host,I_DIRTY_PAGES );
include/linux/pagemap.h +2 -1
@@ -1204,7 +1204,8 @@ void end_page_writeback(struct page *page);void folio_end_writeback (struct folio *folio);void folio_end_writeback_pages (struct folio *folio,long nr_pages);void folio_wait_stable (struct folio *folio); -void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn); +void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn, +long nr_pages);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);void __folio_cancel_dirty (struct folio *folio);static inline void folio_cancel_dirty (struct folio *folio)
mm/page-writeback.c +5 -5
@@ -2677,7 +2677,7 @@ EXPORT_SYMBOL(noop_dirty_folio);* NOTE: This relies on being atomic wrt interrupts. */ static void folio_account_dirtied (struct folio *folio, -struct address_space *mapping) +struct address_space *mapping,long nr) {struct inode *inode = mapping->host;
@@ -2685,7 +2685,6 @@ static void folio_account_dirtied(struct folio *folio,if (mapping_can_writeback (mapping)) {struct bdi_writeback *wb; -long nr =folio_nr_pages (folio);inode_attach_wb (inode, folio); wb =inode_to_wb (inode);
@@ -2733,14 +2732,14 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)* try_to_free_buffers() to fail. */ void __folio_mark_dirty (struct folio *folio,struct address_space *mapping, -int warn) +int warn,long nr_pages) {unsigned long flags;xa_lock_irqsave (&mapping->i_pages, flags);if (folio->mapping) {/* Race with truncate? */ WARN_ON_ONCE (warn && !folio_test_uptodate (folio)); -folio_account_dirtied (folio, mapping); +folio_account_dirtied (folio, mapping, nr_pages);__xa_set_mark (&mapping->i_pages,folio_index (folio),PAGECACHE_TAG_DIRTY ); }
@@ -2771,7 +2770,8 @@ bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)if (folio_test_set_dirty (folio))return false; -__folio_mark_dirty (folio, mapping, !folio_test_private (folio)); +__folio_mark_dirty (folio, mapping, !folio_test_private (folio), +folio_nr_pages (folio));if (mapping->host) {/* !PageAnon && !swapper_space */
Signature · 1 line
2.47.3
RFC PATCH 5/10 mm: add filemap_dirty_folio_pages() helper
In reply to Joanne KoongAdd filemap_dirty_folio_pages() which takes in the number of pages to dirty.
--- fs/buffer.c | 4 ++-- include/linux/pagemap.h | 2 +- include/linux/writeback.h | 2 ++ mm/page-writeback.c | 25 +++++++++++++++++++++---- 4 files changed, 26 insertions(+), 7 deletions(-)
fs/buffer.c +2 -2
@@ -752,7 +752,7 @@ bool block_dirty_folio(struct address_space *mapping, struct folio *folio)if (newly_dirty)__folio_mark_dirty (folio, mapping,1 , -folio_nr_pages (folio)); +folio_nr_pages (folio), true);if (newly_dirty)__mark_inode_dirty (mapping->host,I_DIRTY_PAGES );
@@ -1211,7 +1211,7 @@ void mark_buffer_dirty(struct buffer_head *bh) mapping = folio->mapping;if (mapping)__folio_mark_dirty (folio, mapping,0 , -folio_nr_pages (folio)); +folio_nr_pages (folio), true); }if (mapping)__mark_inode_dirty (mapping->host,I_DIRTY_PAGES );
include/linux/pagemap.h +1 -1
@@ -1205,7 +1205,7 @@ void folio_end_writeback(struct folio *folio);void folio_end_writeback_pages (struct folio *folio,long nr_pages);void folio_wait_stable (struct folio *folio);void __folio_mark_dirty (struct folio *folio,struct address_space *,int warn, -long nr_pages); +long nr_pages,bool newly_dirty);void folio_account_cleaned (struct folio *folio,struct bdi_writeback *wb);void __folio_cancel_dirty (struct folio *folio);static inline void folio_cancel_dirty (struct folio *folio)
include/linux/writeback.h +2 -0
@@ -383,6 +383,8 @@ void tag_pages_for_writeback(struct address_space *mapping, pgoff_t start, pgoff_t end);bool filemap_dirty_folio (struct address_space *mapping,struct folio *folio); +bool filemap_dirty_folio_pages (struct address_space *mapping, +struct folio *folio,long nr_pages);bool folio_redirty_for_writepage (struct writeback_control *,struct folio *);bool redirty_page_for_writepage (struct writeback_control *,struct page *);
mm/page-writeback.c +21 -4
@@ -2732,7 +2732,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)* try_to_free_buffers() to fail. */ void __folio_mark_dirty (struct folio *folio,struct address_space *mapping, -int warn,long nr_pages) +int warn,long nr_pages,bool newly_dirty) {unsigned long flags;
@@ -2740,12 +2740,29 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,if (folio->mapping) {/* Race with truncate? */ WARN_ON_ONCE (warn && !folio_test_uptodate (folio));folio_account_dirtied (folio, mapping, nr_pages); -__xa_set_mark (&mapping->i_pages,folio_index (folio), -PAGECACHE_TAG_DIRTY ); +if (newly_dirty) +__xa_set_mark (&mapping->i_pages,folio_index (folio), +PAGECACHE_TAG_DIRTY ); }xa_unlock_irqrestore (&mapping->i_pages, flags); } +bool filemap_dirty_folio_pages (struct address_space *mapping,struct folio *folio, +long nr_pages) +{ +bool newly_dirty = !folio_test_set_dirty (folio); + +__folio_mark_dirty (folio, mapping, !folio_test_private (folio), + nr_pages, newly_dirty); + +if (newly_dirty && mapping->host) { +/* !PageAnon && !swapper_space */ +__mark_inode_dirty (mapping->host,I_DIRTY_PAGES ); + } + +return newly_dirty; +} +/** * filemap_dirty_folio - Mark a folio dirty for filesystems which do not use buffer_heads. * @mapping: Address space this folio belongs to.
@@ -2771,7 +2788,7 @@ bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)return false;__folio_mark_dirty (folio, mapping, !folio_test_private (folio), -folio_nr_pages (folio)); +folio_nr_pages (folio), true);if (mapping->host) {/* !PageAnon && !swapper_space */
Signature · 1 line
2.47.3
Jan Kara
maintainerRe: [RFC PATCH v1 05/10] mm: add filemap_dirty_folio_pages() helper
Quoted 3 lines · Joanne Koong
Add
filemap_dirty_folio_pages()which takes in the number of pages to dirty.Signed-off-by Joanne Koong <[email protected]>
...
Quoted 14 lines
mm/page-writeback.c +1
-1@@ -2732,7 +2732,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)* try_to_free_buffers() to fail. */ void __folio_mark_dirty (struct folio *folio,struct address_space *mapping, -int warn,long nr_pages) +int warn,long nr_pages,bool newly_dirty) {unsigned long flags;
mm/page-writeback.c void __folio_mark_dirty(struct folio *folio, struct address_space *mapping, · line 2747
Quoted 10 lines
mm/page-writeback.c +3
-2@@ -2740,12 +2740,29 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,if (folio->mapping) {/* Race with truncate? */ WARN_ON_ONCE (warn && !folio_test_uptodate (folio));folio_account_dirtied (folio, mapping, nr_pages); -__xa_set_mark (&mapping->i_pages,folio_index (folio), -PAGECACHE_TAG_DIRTY ); +if (newly_dirty) +__xa_set_mark (&mapping->i_pages,folio_index (folio), +PAGECACHE_TAG_DIRTY ); }xa_unlock_irqrestore (&mapping->i_pages, flags);
I think this is a dangerous coding pattern. What is making sure that by the time you get here newly_dirty is still valid? I mean the dirtying can race e.g. with writeback and so it can happen that the page is clean by the time we get here but newly_dirty is false. We are often protected by page lock when dirtying a folio but not always... So if nothing else this requires a careful documentation about correct use.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v1 05/10] mm: add filemap_dirty_folio_pages() helper
Quoted 38 lines · Jan Kara
Quoted 3 lines · Joanne Koong
Add
filemap_dirty_folio_pages()which takes in the number of pages to dirty.Signed-off-by Joanne Koong <[email protected]>...
Quoted 14 lines
mm/page-writeback.c +1
-1@@ -2732,7 +2732,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)* try_to_free_buffers() to fail. */ void __folio_mark_dirty (struct folio *folio,struct address_space *mapping, -int warn,long nr_pages) +int warn,long nr_pages,bool newly_dirty) {unsigned long flags;mm/page-writeback.c void __folio_mark_dirty(struct folio *folio, struct address_space *mapping, · line 2747
Quoted 10 lines
mm/page-writeback.c +3
-2@@ -2740,12 +2740,29 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,if (folio->mapping) {/* Race with truncate? */ WARN_ON_ONCE (warn && !folio_test_uptodate (folio));folio_account_dirtied (folio, mapping, nr_pages); -__xa_set_mark (&mapping->i_pages,folio_index (folio), -PAGECACHE_TAG_DIRTY ); +if (newly_dirty) +__xa_set_mark (&mapping->i_pages,folio_index (folio), +PAGECACHE_TAG_DIRTY ); }xa_unlock_irqrestore (&mapping->i_pages, flags);I think this is a dangerous coding pattern. What is making sure that by the time you get here
newly_dirtyis still valid? I mean the dirtying can race e.g. with writeback and so it can happen that the page is clean by the time we get here butnewly_dirtyis false. We are often protected by page lock when dirtying a folio but not always... So if nothing else this requires a careful documentation about correct use.
Quoted 1 line
Honza
I think races against writeback and truncation could already exist here prior to this patch. afaict from the function documentation for __folio_mark_dirty(), it's up to the caller to prevent this:
* It is the caller's responsibility to prevent the folio from being truncated
* while this function is in progress, although it may have been truncated
* before this function is called. Most callers have the folio locked.
* A few have the folio blocked from truncation through other means (e.g.
* zap_vma_pages() has it mapped and is holding the page table lock).
The documentation doesn't mention anything about writeback but I think it applies here similarly.
I'm happy to do this another way though if there's a better approach here.
Thanks,
Joanne
Quoted 3 lines
--
Jan Kara <[email protected]>
SUSE Labs, CR
Re: [RFC PATCH v1 05/10] mm: add filemap_dirty_folio_pages() helper
In reply to Joanne KoongQuoted 1 line
+bool filemap_dirty_folio_pages(struct
address_space*mapping, struct folio *folio,
Overly long line here. Also this function would benefit from a kerneldoc comment.
RFC PATCH 6/10 mm: add __folio_clear_dirty_for_io() helper
In reply to Joanne KoongAdd __folio_clear_dirty_for_io() which takes in an arg for whether the folio and wb stats should be updated as part of the call or not.
--- mm/page-writeback.c | 47 +++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 19 deletions(-)
mm/page-writeback.c +28 -19
@@ -2927,21 +2927,7 @@ void __folio_cancel_dirty(struct folio *folio) }EXPORT_SYMBOL (__folio_cancel_dirty); -/* -* Clear a folio's dirty flag, while caring for dirty memory accounting. -* Returns true if the folio was previously dirty. -* -* This is for preparing to put the folio under writeout. We leave -* the folio tagged as dirty in the xarray so that a concurrent -* write-for-sync can discover it via a PAGECACHE_TAG_DIRTY walk. -* The ->writepage implementation will run either folio_start_writeback() -* or folio_mark_dirty(), at which stage we bring the folio's dirty flag -* and xarray dirty tag back into sync. -* -* This incoherency between the folio's dirty flag and xarray tag is -* unfortunate, but it only exists while the folio is locked. -*/ -bool folio_clear_dirty_for_io (struct folio *folio) +static bool __folio_clear_dirty_for_io (struct folio *folio,bool update_stats) {struct address_space *mapping =folio_mapping (folio);bool ret = false;
@@ -2990,10 +2976,14 @@ bool folio_clear_dirty_for_io(struct folio *folio)*/ wb =unlocked_inode_to_wb_begin (inode, &cookie);if (folio_test_clear_dirty (folio)) { -long nr =folio_nr_pages (folio); -lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); -zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr); -wb_stat_mod (wb,WB_RECLAIMABLE , -nr); +if (update_stats) { +long nr =folio_nr_pages (folio); +lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , + -nr); +zone_stat_mod_folio (folio, +NR_ZONE_WRITE_PENDING , -nr); +wb_stat_mod (wb,WB_RECLAIMABLE , -nr); + } ret = true; }unlocked_inode_to_wb_end (inode, &cookie);
@@ -3001,6 +2991,25 @@ bool folio_clear_dirty_for_io(struct folio *folio) }return folio_test_clear_dirty (folio); } + +/* +* Clear a folio's dirty flag, while caring for dirty memory accounting. +* Returns true if the folio was previously dirty. +* +* This is for preparing to put the folio under writeout. We leave +* the folio tagged as dirty in the xarray so that a concurrent +* write-for-sync can discover it via a PAGECACHE_TAG_DIRTY walk. +* The ->writepage implementation will run either folio_start_writeback() +* or folio_mark_dirty(), at which stage we bring the folio's dirty flag +* and xarray dirty tag back into sync. +* +* This incoherency between the folio's dirty flag and xarray tag is +* unfortunate, but it only exists while the folio is locked. +*/ +bool folio_clear_dirty_for_io (struct folio *folio) +{ +return __folio_clear_dirty_for_io (folio, true); +}EXPORT_SYMBOL (folio_clear_dirty_for_io);static void wb_inode_writeback_start (struct bdi_writeback *wb)
Signature · 1 line
2.47.3
RFC PATCH 7/10 mm: add no_stats_accounting bitfield to wbc
In reply to Joanne KoongAdd a no_stats_accounting bitfield to wbc that callers can set. Hook this up to __folio_clear_dirty_for_io() when preparing writeback.
--- include/linux/writeback.h | 3 +++ mm/page-writeback.c | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
include/linux/writeback.h +3 -0
@@ -72,6 +72,9 @@ struct writeback_control {*/ unsigned no_cgroup_owner:1; +/* Do not do any stats accounting. The caller will do this themselves */ + unsigned no_stats_accounting:1; +/* To enable batching of swap writes to non-block-device backends, * "plug" can be set point to a 'struct swap_iocb *'. When all swap * writes have been submitted, if with swap_iocb is not NULL,
mm/page-writeback.c +2 -1
@@ -2404,6 +2404,7 @@ void tag_pages_for_writeback(struct address_space *mapping, }EXPORT_SYMBOL (tag_pages_for_writeback); +static bool __folio_clear_dirty_for_io (struct folio *folio,bool update_stats);static bool folio_prepare_writeback (struct address_space *mapping,struct writeback_control *wbc,struct folio *folio) {
@@ -2430,7 +2431,7 @@ static bool folio_prepare_writeback(struct address_space *mapping, }BUG_ON (folio_test_writeback (folio)); -if (!folio_clear_dirty_for_io (folio)) +if (!__folio_clear_dirty_for_io (folio, !wbc->no_stats_accounting))return false;return true;
Signature · 1 line
2.47.3
Re: [RFC PATCH v1 07/10] mm: add no_stats_accounting bitfield to wbc
Quoted 2 lines · Joanne Koong
Add a
no_stats_accountingbitfield to wbc that callers can set. Hook this up to__folio_clear_dirty_for_io()when preparing writeback.
Please explain the use case for this a bit more in the commit log, and maybe also the comments.
RFC PATCH 8/10 mm: refactor clearing dirty stats into helper function
In reply to Joanne KoongMove logic for clearing dirty stats into a helper function both folio_account_cleaned() and __folio_clear_dirty_for_io() invoke.
--- mm/page-writeback.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
mm/page-writeback.c +12 -11
@@ -2703,6 +2703,14 @@ static void folio_account_dirtied(struct folio *folio, } } +static void __clear_dirty_for_io_stats (struct folio *folio, +struct bdi_writeback *wb,long nr_pages) +{ +lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr_pages); +zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr_pages); +wb_stat_mod (wb,WB_RECLAIMABLE , -nr_pages); +} +/* * Helper function for deaccounting dirty page without writeback. *
@@ -2711,9 +2719,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb) {long nr =folio_nr_pages (folio); -lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); -zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr); -wb_stat_mod (wb,WB_RECLAIMABLE , -nr); +__clear_dirty_for_io_stats (folio, wb, nr);task_io_account_cancelled_write (nr *PAGE_SIZE ); }
@@ -2977,14 +2983,9 @@ static bool __folio_clear_dirty_for_io(struct folio *folio, bool update_stats)*/ wb =unlocked_inode_to_wb_begin (inode, &cookie);if (folio_test_clear_dirty (folio)) { -if (update_stats) { -long nr =folio_nr_pages (folio); -lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , - -nr); -zone_stat_mod_folio (folio, -NR_ZONE_WRITE_PENDING , -nr); -wb_stat_mod (wb,WB_RECLAIMABLE , -nr); - } +if (update_stats) +__clear_dirty_for_io_stats (folio, wb, +folio_nr_pages (folio)); ret = true; }unlocked_inode_to_wb_end (inode, &cookie);
Signature · 1 line
2.47.3
Re: [RFC PATCH v1 08/10] mm: refactor clearing dirty stats into helper function
Quoted 46 lines · Joanne Koong
Move logic for clearing dirty stats into a helper function both
folio_account_cleaned()and__folio_clear_dirty_for_io()invoke.Signed-off-by Joanne Koong <[email protected]>--- mm/page-writeback.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)mm/page-writeback.c +9
-7@@ -2703,6 +2703,14 @@ static void folio_account_dirtied(struct folio *folio, } } +static void __clear_dirty_for_io_stats (struct folio *folio, +struct bdi_writeback *wb,long nr_pages) +{ +lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr_pages); +zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr_pages); +wb_stat_mod (wb,WB_RECLAIMABLE , -nr_pages); +} +/* * Helper function for deaccounting dirty page without writeback. * @@ -2711,9 +2719,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb) {long nr =folio_nr_pages (folio); -lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , -nr); -zone_stat_mod_folio (folio,NR_ZONE_WRITE_PENDING , -nr); -wb_stat_mod (wb,WB_RECLAIMABLE , -nr); +__clear_dirty_for_io_stats (folio, wb, nr);task_io_account_cancelled_write (nr *PAGE_SIZE ); }@@ -2977,14 +2983,9 @@ static bool __folio_clear_dirty_for_io(struct folio *folio, bool update_stats)*/ wb =unlocked_inode_to_wb_begin (inode, &cookie);if (folio_test_clear_dirty (folio)) { -if (update_stats) { -long nr =folio_nr_pages (folio); -lruvec_stat_mod_folio (folio,NR_FILE_DIRTY , - -nr);
mm/page-writeback.c static bool __folio_clear_dirty_for_io(struct folio *folio, bool update_stats) · line 2991
Quoted 10 lines
mm/page-writeback.c +3
-4-zone_stat_mod_folio (folio, -NR_ZONE_WRITE_PENDING , -nr); -wb_stat_mod (wb,WB_RECLAIMABLE , -nr); - } +if (update_stats) +__clear_dirty_for_io_stats (folio, wb, +folio_nr_pages (folio)); ret = true; }unlocked_inode_to_wb_end (inode, &cookie);
This seems like a nice cleanup that isn't dependent on the rest of the series.
RFC PATCH 9/10 mm: add clear_dirty_for_io_stats() helper
In reply to Joanne KoongAdd clear_dirty_for_io_stats() which clears dirty stats corresponding to a folio.
--- include/linux/writeback.h | 1 + mm/page-writeback.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
include/linux/writeback.h +1 -0
@@ -382,6 +382,7 @@ int write_cache_pages(struct address_space *mapping, void *data);int do_writepages (struct address_space *mapping,struct writeback_control *wbc);void writeback_set_ratelimit (void ); +void clear_dirty_for_io_stats (struct folio *folio,long nr_pages);void tag_pages_for_writeback (struct address_space *mapping,pgoff_t start,pgoff_t end);
mm/page-writeback.c +16 -0
@@ -2711,6 +2711,22 @@ static void __clear_dirty_for_io_stats(struct folio *folio,wb_stat_mod (wb,WB_RECLAIMABLE , -nr_pages); } +void clear_dirty_for_io_stats (struct folio *folio,long nr_pages) +{ +struct address_space *mapping =folio_mapping (folio); +struct bdi_writeback *wb; +struct wb_lock_cookie cookie = {}; +struct inode *inode; + +if (!mapping || !mapping_can_writeback (mapping)) +return ; + + inode = mapping->host; + wb =unlocked_inode_to_wb_begin (inode, &cookie); +__clear_dirty_for_io_stats (folio, wb, nr_pages); +unlocked_inode_to_wb_end (inode, &cookie); +} +/* * Helper function for deaccounting dirty page without writeback. *
Signature · 1 line
2.47.3
RFC PATCH 10/10 iomap: add granular dirty and writeback accounting
In reply to Joanne KoongAdd granular dirty and writeback accounting for large folios. These stats are used by the mm layer for dirty balancing and throttling. Having granular dirty and writeback accounting helps prevent over-aggressive balancing and throttling.
There are 4 places in iomap this commit affects:
a) filemap dirtying, which now calls filemap_dirty_folio_pages()
b) writeback_iter with setting the wbc->no_stats_accounting bit and calling clear_dirty_for_io_stats()
c) starting writeback, which now calls __folio_start_writeback()
d) ending writeback, which now calls folio_end_writeback_pages()
This relies on using the ifs->state dirty bitmap to track dirty pages in the folio. As such, this can only be utilized on filesystems where the block size >= PAGE_SIZE.
--- fs/iomap/buffered-io.c | 136 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 128 insertions(+), 8 deletions(-)
fs/iomap/buffered-io.c +128 -8
@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;/* * Each block has two bits in this bitmap:
@@ -81,6 +83,25 @@ static inline bool ifs_block_is_dirty(struct folio *folio,return test_bit (block + blks_per_folio, ifs->state); } +static unsigned ifs_count_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned block_size =1 << inode->i_blkbits; +unsigned start_blk =0 ; +unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio)); +unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + } + +return nblks * (block_size >>PAGE_SHIFT ); +} +static unsigned ifs_find_dirty_range (struct folio *folio,struct iomap_folio_state *ifs,u64 *range_start,u64 range_end) {
@@ -165,6 +186,63 @@ static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)ifs_set_range_dirty (folio, ifs, off, len); } +static long iomap_get_range_newly_dirtied (struct folio *folio,loff_t pos, +unsigned len) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned start_blk = pos >> inode->i_blkbits; +unsigned end_blk =min ((unsigned )((pos + len -1 ) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio) -1 ); +unsigned nblks =0 ; +unsigned block_size =1 << inode->i_blkbits; + +while (start_blk <= end_blk) { +if (!ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + } + +return nblks * (block_size >>PAGE_SHIFT ); +} + +static bool iomap_granular_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode; +unsigned block_size; + +if (!ifs) +return false; + + inode = folio->mapping->host; + block_size =1 << inode->i_blkbits; + +if (block_size >=PAGE_SIZE ) { +WARN_ON (block_size & (PAGE_SIZE -1 )); +return true; + } +return false; +} + +static bool iomap_dirty_folio_range (struct address_space *mapping,struct folio *folio, +loff_t pos,unsigned len) +{ +long nr_new_dirty_pages; + +if (!iomap_granular_dirty_pages (folio)) { +iomap_set_range_dirty (folio, pos, len); +return filemap_dirty_folio (mapping, folio); + } + + nr_new_dirty_pages =iomap_get_range_newly_dirtied (folio, pos, len); +if (!nr_new_dirty_pages) +return false; + +iomap_set_range_dirty (folio, pos, len); +return filemap_dirty_folio_pages (mapping, folio, nr_new_dirty_pages); +} +static struct iomap_folio_state *ifs_alloc (struct inode *inode,struct folio *folio,unsigned int flags) {
@@ -661,8 +739,7 @@ bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio)size_t len =folio_size (folio);ifs_alloc (inode, folio,0 ); -iomap_set_range_dirty (folio,0 , len); -return filemap_dirty_folio (mapping, folio); +return iomap_dirty_folio_range (mapping, folio,0 , len); }EXPORT_SYMBOL_GPL (iomap_dirty_folio);
@@ -886,8 +963,8 @@ static bool __iomap_write_end(struct inode *inode, loff_t pos, size_t len,if (unlikely (copied < len && !folio_test_uptodate (folio)))return false;iomap_set_range_uptodate (folio,offset_in_folio (folio, pos), len); -iomap_set_range_dirty (folio,offset_in_folio (folio, pos), copied); -filemap_dirty_folio (inode->i_mapping, folio); +iomap_dirty_folio_range (inode->i_mapping, folio, +offset_in_folio (folio, pos), copied);return true; }
@@ -1560,6 +1637,29 @@ void iomap_start_folio_write(struct inode *inode, struct folio *folio, }EXPORT_SYMBOL_GPL (iomap_start_folio_write); +static void iomap_folio_start_writeback (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; + +if (!iomap_granular_dirty_pages (folio)) +return folio_start_writeback (folio); + +__folio_start_writeback (folio, false, ifs->nr_pages_writeback); +} + +static void iomap_folio_end_writeback (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +long nr_pages_writeback; + +if (!iomap_granular_dirty_pages (folio)) +return folio_end_writeback (folio); + + nr_pages_writeback = ifs->nr_pages_writeback; + ifs->nr_pages_writeback =0 ; +folio_end_writeback_pages (folio, nr_pages_writeback); +} +void iomap_finish_folio_write (struct inode *inode,struct folio *folio,size_t len) {
@@ -1569,7 +1669,7 @@ void iomap_finish_folio_write(struct inode *inode, struct folio *folio,WARN_ON_ONCE (ifs &&atomic_read (&ifs->write_bytes_pending) <=0 );if (!ifs ||atomic_sub_and_test (len, &ifs->write_bytes_pending)) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }EXPORT_SYMBOL_GPL (iomap_finish_folio_write);
@@ -1657,6 +1757,21 @@ static bool iomap_writeback_handle_eof(struct folio *folio, struct inode *inode,return true; } +static void iomap_update_dirty_stats (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +long nr_dirty_pages; + +if (iomap_granular_dirty_pages (folio)) { + nr_dirty_pages =ifs_count_dirty_pages (folio); + ifs->nr_pages_writeback = nr_dirty_pages; + }else { + nr_dirty_pages =folio_nr_pages (folio); + } + +clear_dirty_for_io_stats (folio, nr_dirty_pages); +} +int iomap_writeback_folio (struct iomap_writepage_ctx *wpc,struct folio *folio) {struct iomap_folio_state *ifs = folio->private;
@@ -1674,6 +1789,8 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)trace_iomap_writeback_folio (inode, pos,folio_size (folio)); +iomap_update_dirty_stats (folio); +if (!iomap_writeback_handle_eof (folio, inode, &end_pos))return 0 ;WARN_ON_ONCE (end_pos <= pos);
@@ -1681,6 +1798,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)if (i_blocks_per_folio (inode, folio) >1 ) {if (!ifs) { ifs =ifs_alloc (inode, folio,0 ); + ifs->nr_pages_writeback =folio_nr_pages (folio);iomap_set_range_dirty (folio,0 , end_pos - pos); }
@@ -1698,7 +1816,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)* Set the writeback bit ASAP, as the I/O completion for the single * block per folio case happen hit as soon as we're submitting the bio. */ -folio_start_writeback (folio); +iomap_folio_start_writeback (folio);/* * Walk through the folio to find dirty areas to write back.
@@ -1731,10 +1849,10 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)*/ if (ifs) {if (atomic_dec_and_test (&ifs->write_bytes_pending)) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }else {if (!wb_pending) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }mapping_set_error (inode->i_mapping, error);return error;
@@ -1756,6 +1874,8 @@ iomap_writepages(struct iomap_writepage_ctx *wpc) PF_MEMALLOC))return -EIO ; + wpc->wbc->no_stats_accounting = true; +while ((folio =writeback_iter (mapping, wpc->wbc, folio, &error))) { error =iomap_writeback_folio (wpc, folio);folio_unlock (folio);
Signature · 1 line
2.47.3
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
fs/iomap/buffered-io.c struct iomap_folio_state { · line 24
Quoted 10 lines
fs/iomap/buffered-io.c +2
-0@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;
This adds more sizse to the folio state. Shouldn't this be the same as
DIV_ROUND_UP(write_bytes_pending, PAGE_SIZE)
anyway?
fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 93
Quoted 2 lines
fs/iomap/buffered-io.c +2
-0+unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio));
Overly long line. Also not sure why the cast is needed to start with?
fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 100
Quoted 7 lines
fs/iomap/buffered-io.c +7
-0+unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }
We have this pattern open coded in a few places. Maybe factor it into a helper first? And then maybe someone smart can actually make it use find_first_bit/find_next_bit.
Quoted 7 lines
fs/iomap/buffered-io.c +7
-0+static bool iomap_granular_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode; +unsigned block_size; + +if (!ifs)
fs/iomap/buffered-io.c static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len) · line 225
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+return false; + + inode = folio->mapping->host; + block_size =1 << inode->i_blkbits; + +if (block_size >=PAGE_SIZE ) { +WARN_ON (block_size & (PAGE_SIZE -1 )); +return true; + } +return false;
Do we need the WARN_ON? Both the block and page size must be powers of two, so I can't see how it would trigger. Also this can use the i_blocksize helper.
I.e. just turn this into:
return i_blocksize(folio->mapping->host) >= PAGE_SIZE;
Quoted 1 line
+static bool iomap_dirty_folio_range(struct
address_space*mapping, struct folio *folio,
Overly long line.
fs/iomap/buffered-io.c iomap_writepages(struct iomap_writepage_ctx *wpc) · line 1877
Quoted 1 line
fs/iomap/buffered-io.c +1
-0+ wpc->wbc->no_stats_accounting = true;
Who does the writeback accounting now? Maybe throw in a comment if iomap is now doing something different than all the other writeback code.
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
Quoted 15 lines · Christoph Hellwig
fs/iomap/buffered-io.c struct iomap_folio_state { · line 24
Quoted 11 lines
fs/iomap/buffered-io.c +2
-0@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;This adds more sizse to the folio state. Shouldn't this be the same as
Quoted 3 lines
DIV_ROUND_UP(write_bytes_pending, PAGE_SIZE)anyway?
I don't think we can use write_bytes_pending because writeback for a folio may be split into multiple requests (eg for fuse, if the ranges are not contiguous) and each request when it finishes will call iomap_finish_folio_write() which will decrement write_bytes_pending, but when the last folio writeback request has finished and we call folio_end_writeback_pages(), we would need the original value of write_bytes_pending before any of the decrements. write_bytes_pending gets decremented since it gets used as a refcount.
I need to look more into whether readahead/read_folio and writeback run concurrently or not but if not, maybe read_bytes_pending and write_bytes_pending could be consolidated together.
Quoted 5 lines
fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 93
Quoted 3 lines
fs/iomap/buffered-io.c +2
-0+unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio));Overly long line. Also not sure why the cast is needed to start with?
The cast is needed to avoid the compiler error of comparing a loff_t with an unsigned int. I see there's a min_t helper, I'll use that instead then.
Quoted 26 lines
fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 100
Quoted 8 lines
fs/iomap/buffered-io.c +7
-0+unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }We have this pattern open coded in a few places. Maybe factor it into a helper first? And then maybe someone smart can actually make it use find_first_bit/find_next_bit.
Quoted 13 lines
fs/iomap/buffered-io.c +13
-0+static bool iomap_granular_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode; +unsigned block_size; + +if (!ifs) +return false; + + inode = folio->mapping->host; + block_size =1 << inode->i_blkbits; + +if (block_size >=PAGE_SIZE ) {
Quoted 8 lines
fs/iomap/buffered-io.c static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len) · line 225
Quoted 4 lines
fs/iomap/buffered-io.c +4
-0+WARN_ON (block_size & (PAGE_SIZE -1 )); +return true; + } +return false;Do we need the
WARN_ON? Both the block and page size must be powers of two, so I can't see how it would trigger. Also this can use thei_blocksizehelper.
I'll get rid of the WARN_ON and will incorporate your i_blocksize helper suggestion.
Quoted 9 lines
I.e. just turn this into:
return i_blocksize(folio->mapping->host) >= PAGE_SIZE;Quoted 1 line
+static bool iomap_dirty_folio_range(struct
address_space*mapping, struct folio *folio,Overly long line.
I'll fix up the long lines in the patchset, sorry.
Quoted 6 lines
Quoted 2 lines
+
wpc->wbc->no_stats_accounting= true;Who does the writeback accounting now? Maybe throw in a comment if iomap is now doing something different than all the other writeback code.
iomap does the writeback accounting now, which happens in iomap_update_dirty_stats(). I'll add a comment about that.
Thanks,
Joanne
Quoted 1 line
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
Quoted 4 lines · Joanne Koong
I need to look more into whether readahead/read_folio and writeback run concurrently or not but if not, maybe
read_bytes_pendingandwrite_bytes_pendingcould be consolidated together.
Nvm, that doesn't work. read_folio() can still get called for a folio that's under writeback if it's not fully up to date.
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
In reply to Joanne KoongQuoted 12 lines · Joanne Koong
Add granular dirty and writeback accounting for large folios. These stats are used by the mm layer for dirty balancing and throttling. Having granular dirty and writeback accounting helps prevent over-aggressive balancing and throttling.
There are 4 places in iomap this commit affects:
a) filemap dirtying, which now callsfilemap_dirty_folio_pages()
b)writeback_iterwith setting thewbc->no_stats_accountingbit and callingclear_dirty_for_io_stats()
c) starting writeback, which now calls__folio_start_writeback()
d) ending writeback, which now callsfolio_end_writeback_pages()
Quoted 3 lines
This relies on using the
ifs->statedirty bitmap to track dirty pages in the folio. As such, this can only be utilized on filesystems where the block size >=PAGE_SIZE.
Apologies for my slow responses this month. :)
I wonder, does this cause an observable change in the writeback accounting and throttling behavior for non-fuse filesystems like XFS that use large folios? I *think* this does actually reduce throttling for XFS, but it might not be so noticeable because the limits are much more generous outside of fuse?
Quoted 27 lines
Signed-off-by Joanne Koong <[email protected]>--- fs/iomap/buffered-io.c | 136 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 128 insertions(+), 8 deletions(-)fs/iomap/buffered-io.c +7
-0@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;/* * Each block has two bits in this bitmap: @@ -81,6 +83,25 @@ static inline bool ifs_block_is_dirty(struct folio *folio,return test_bit (block + blks_per_folio, ifs->state); } +static unsigned ifs_count_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned block_size =1 << inode->i_blkbits;
fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 100
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+unsigned start_blk =0 ; +unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio)); +unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }
Hmm, isn't this bitmap_weight(ifs->state, blks_per_folio) ?
Ohh wait no, the dirty bitmap doesn't start on a byte boundary because the format of the bitmap is [uptodate bits][dirty bits].
Maybe those two should be reversed, because I bet the dirty state gets changed a lot more over the lifetime of a folio than the uptodate bits.
Quoted 17 lines
fs/iomap/buffered-io.c +10
-0+ +return nblks * (block_size >>PAGE_SHIFT ); +} +static unsigned ifs_find_dirty_range (struct folio *folio,struct iomap_folio_state *ifs,u64 *range_start,u64 range_end) {@@ -165,6 +186,63 @@ static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)ifs_set_range_dirty (folio, ifs, off, len); } +static long iomap_get_range_newly_dirtied (struct folio *folio,loff_t pos, +unsigned len) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned start_blk = pos >> inode->i_blkbits;
fs/iomap/buffered-io.c static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len) · line 204
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+unsigned end_blk =min ((unsigned )((pos + len -1 ) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio) -1 ); +unsigned nblks =0 ; +unsigned block_size =1 << inode->i_blkbits; + +while (start_blk <= end_blk) { +if (!ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }
...then this becomes (endblk - startblk) - bitmap_weight().
Quoted 178 lines
fs/iomap/buffered-io.c +91
-8+ +return nblks * (block_size >>PAGE_SHIFT ); +} + +static bool iomap_granular_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode; +unsigned block_size; + +if (!ifs) +return false; + + inode = folio->mapping->host; + block_size =1 << inode->i_blkbits; + +if (block_size >=PAGE_SIZE ) { +WARN_ON (block_size & (PAGE_SIZE -1 )); +return true; + } +return false; +} + +static bool iomap_dirty_folio_range (struct address_space *mapping,struct folio *folio, +loff_t pos,unsigned len) +{ +long nr_new_dirty_pages; + +if (!iomap_granular_dirty_pages (folio)) { +iomap_set_range_dirty (folio, pos, len); +return filemap_dirty_folio (mapping, folio); + } + + nr_new_dirty_pages =iomap_get_range_newly_dirtied (folio, pos, len); +if (!nr_new_dirty_pages) +return false; + +iomap_set_range_dirty (folio, pos, len); +return filemap_dirty_folio_pages (mapping, folio, nr_new_dirty_pages); +} +static struct iomap_folio_state *ifs_alloc (struct inode *inode,struct folio *folio,unsigned int flags) {@@ -661,8 +739,7 @@ bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio)size_t len =folio_size (folio);ifs_alloc (inode, folio,0 ); -iomap_set_range_dirty (folio,0 , len); -return filemap_dirty_folio (mapping, folio); +return iomap_dirty_folio_range (mapping, folio,0 , len); }EXPORT_SYMBOL_GPL (iomap_dirty_folio);@@ -886,8 +963,8 @@ static bool __iomap_write_end(struct inode *inode, loff_t pos, size_t len,if (unlikely (copied < len && !folio_test_uptodate (folio)))return false;iomap_set_range_uptodate (folio,offset_in_folio (folio, pos), len); -iomap_set_range_dirty (folio,offset_in_folio (folio, pos), copied); -filemap_dirty_folio (inode->i_mapping, folio); +iomap_dirty_folio_range (inode->i_mapping, folio, +offset_in_folio (folio, pos), copied);return true; }@@ -1560,6 +1637,29 @@ void iomap_start_folio_write(struct inode *inode, struct folio *folio, }EXPORT_SYMBOL_GPL (iomap_start_folio_write); +static void iomap_folio_start_writeback (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; + +if (!iomap_granular_dirty_pages (folio)) +return folio_start_writeback (folio); + +__folio_start_writeback (folio, false, ifs->nr_pages_writeback); +} + +static void iomap_folio_end_writeback (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +long nr_pages_writeback; + +if (!iomap_granular_dirty_pages (folio)) +return folio_end_writeback (folio); + + nr_pages_writeback = ifs->nr_pages_writeback; + ifs->nr_pages_writeback =0 ; +folio_end_writeback_pages (folio, nr_pages_writeback); +} +void iomap_finish_folio_write (struct inode *inode,struct folio *folio,size_t len) {@@ -1569,7 +1669,7 @@ void iomap_finish_folio_write(struct inode *inode, struct folio *folio,WARN_ON_ONCE (ifs &&atomic_read (&ifs->write_bytes_pending) <=0 );if (!ifs ||atomic_sub_and_test (len, &ifs->write_bytes_pending)) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }EXPORT_SYMBOL_GPL (iomap_finish_folio_write);@@ -1657,6 +1757,21 @@ static bool iomap_writeback_handle_eof(struct folio *folio, struct inode *inode,return true; } +static void iomap_update_dirty_stats (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +long nr_dirty_pages; + +if (iomap_granular_dirty_pages (folio)) { + nr_dirty_pages =ifs_count_dirty_pages (folio); + ifs->nr_pages_writeback = nr_dirty_pages; + }else { + nr_dirty_pages =folio_nr_pages (folio); + } + +clear_dirty_for_io_stats (folio, nr_dirty_pages); +} +int iomap_writeback_folio (struct iomap_writepage_ctx *wpc,struct folio *folio) {struct iomap_folio_state *ifs = folio->private;@@ -1674,6 +1789,8 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)trace_iomap_writeback_folio (inode, pos,folio_size (folio)); +iomap_update_dirty_stats (folio); +if (!iomap_writeback_handle_eof (folio, inode, &end_pos))return 0 ;WARN_ON_ONCE (end_pos <= pos);@@ -1681,6 +1798,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)if (i_blocks_per_folio (inode, folio) >1 ) {if (!ifs) { ifs =ifs_alloc (inode, folio,0 ); + ifs->nr_pages_writeback =folio_nr_pages (folio);iomap_set_range_dirty (folio,0 , end_pos - pos); }@@ -1698,7 +1816,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)* Set the writeback bit ASAP, as the I/O completion for the single * block per folio case happen hit as soon as we're submitting the bio. */ -folio_start_writeback (folio); +iomap_folio_start_writeback (folio);/* * Walk through the folio to find dirty areas to write back. @@ -1731,10 +1849,10 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)*/ if (ifs) {if (atomic_dec_and_test (&ifs->write_bytes_pending)) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }else {if (!wb_pending) -folio_end_writeback (folio); +iomap_folio_end_writeback (folio); }mapping_set_error (inode->i_mapping, error);return error;@@ -1756,6 +1874,8 @@ iomap_writepages(struct iomap_writepage_ctx *wpc) PF_MEMALLOC))return -EIO ; + wpc->wbc->no_stats_accounting = true; +while ((folio =writeback_iter (mapping, wpc->wbc, folio, &error))) { error =iomap_writeback_folio (wpc, folio);folio_unlock (folio);Signature · 1 line
2.47.3
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
Quoted 14 lines · Darrick J. Wong
Quoted 12 lines · Joanne Koong
Add granular dirty and writeback accounting for large folios. These stats are used by the mm layer for dirty balancing and throttling. Having granular dirty and writeback accounting helps prevent over-aggressive balancing and throttling.
There are 4 places in iomap this commit affects:
a) filemap dirtying, which now callsfilemap_dirty_folio_pages()
b)writeback_iterwith setting thewbc->no_stats_accountingbit and callingclear_dirty_for_io_stats()
c) starting writeback, which now calls__folio_start_writeback()
d) ending writeback, which now callsfolio_end_writeback_pages()
Quoted 5 lines
Quoted 3 lines
This relies on using the
ifs->statedirty bitmap to track dirty pages in the folio. As such, this can only be utilized on filesystems where the block size >=PAGE_SIZE.Apologies for my slow responses this month. :)
No worries at all, thanks for looking at this.
Quoted 6 lines
I wonder, does this cause an observable change in the writeback accounting and throttling behavior for non-fuse filesystems like XFS that use large folios? I *think* this does actually reduce throttling for XFS, but it might not be so noticeable because the limits are much more generous outside of fuse?
I haven't run any benchmarks on non-fuse filesystems yet but that's what I would expect too. Will run some benchmarks to see!
Quoted 41 lines
Quoted 28 lines
Signed-off-by Joanne Koong <[email protected]>--- fs/iomap/buffered-io.c | 136 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 128 insertions(+), 8 deletions(-)fs/iomap/buffered-io.c +7
-0@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;/* * Each block has two bits in this bitmap: @@ -81,6 +83,25 @@ static inline bool ifs_block_is_dirty(struct folio *folio,return test_bit (block + blks_per_folio, ifs->state); } +static unsigned ifs_count_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned block_size =1 << inode->i_blkbits;fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 100
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+unsigned start_blk =0 ; +unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio)); +unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }Hmm, isn't this
bitmap_weight(ifs->state, blks_per_folio)?
Quoted 5 lines
Ohh wait no, the dirty bitmap doesn't start on a byte boundary because the format of the bitmap is [uptodate bits][dirty bits].
Maybe those two should be reversed, because I bet the dirty state gets changed a lot more over the lifetime of a folio than the uptodate bits.
I think there's the find_next_bit() helper (which Christoph also pointed out) that could probably be used here instead. Or at least that's how I see a lot of the driver code doing it for their bitmaps.
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
Quoted 24 lines · Joanne Koong
Quoted 14 lines · Darrick J. Wong
Quoted 12 lines · Joanne Koong
Add granular dirty and writeback accounting for large folios. These stats are used by the mm layer for dirty balancing and throttling. Having granular dirty and writeback accounting helps prevent over-aggressive balancing and throttling.
There are 4 places in iomap this commit affects:
a) filemap dirtying, which now callsfilemap_dirty_folio_pages()
b)writeback_iterwith setting thewbc->no_stats_accountingbit and callingclear_dirty_for_io_stats()
c) starting writeback, which now calls__folio_start_writeback()
d) ending writeback, which now callsfolio_end_writeback_pages()Quoted 5 lines
Quoted 3 lines
This relies on using the
ifs->statedirty bitmap to track dirty pages in the folio. As such, this can only be utilized on filesystems where the block size >=PAGE_SIZE.Apologies for my slow responses this month. :)
No worries at all, thanks for looking at this.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
I wonder, does this cause an observable change in the writeback accounting and throttling behavior for non-fuse filesystems like XFS that use large folios? I *think* this does actually reduce throttling for XFS, but it might not be so noticeable because the limits are much more generous outside of fuse?
I haven't run any benchmarks on non-fuse filesystems yet but that's what I would expect too. Will run some benchmarks to see!
I ran some benchmarks on xfs for the contrived test case I used for fuse (eg writing 2 GB in 128 MB chunks and then doing 50k 50-byte random writes) and I don't see any noticeable performance difference.
I re-tested it on fuse but this time with strictlimiting disabled and didn't notice any difference on that either, probably because with strictlimiting off we don't run into the upper limit in that test so there's no extra throttling that needs to be mitigated.
It's unclear to me how often (if at all?) real workloads run up against their dirty/writeback limits.
Quoted 51 lines
Quoted 42 lines
Quoted 29 lines
Signed-off-by Joanne Koong <[email protected]>--- fs/iomap/buffered-io.c | 136 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 128 insertions(+), 8 deletions(-)fs/iomap/buffered-io.c +7
-0@@ -20,6 +20,8 @@ struct iomap_folio_state {spinlock_t state_lock;unsigned int read_bytes_pending;atomic_t write_bytes_pending; +/* number of pages being currently written back */ +unsigned nr_pages_writeback;/* * Each block has two bits in this bitmap: @@ -81,6 +83,25 @@ static inline bool ifs_block_is_dirty(struct folio *folio,return test_bit (block + blks_per_folio, ifs->state); } +static unsigned ifs_count_dirty_pages (struct folio *folio) +{ +struct iomap_folio_state *ifs = folio->private; +struct inode *inode = folio->mapping->host; +unsigned block_size =1 << inode->i_blkbits;fs/iomap/buffered-io.c static inline bool ifs_block_is_dirty(struct folio *folio, · line 100
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+unsigned start_blk =0 ; +unsigned end_blk =min ((unsigned )(i_size_read (inode) >> inode->i_blkbits), +i_blocks_per_folio (inode, folio)); +unsigned nblks =0 ; + +while (start_blk < end_blk) { +if (ifs_block_is_dirty (folio, ifs, start_blk)) + nblks++; + start_blk++; + }Hmm, isn't this
bitmap_weight(ifs->state, blks_per_folio)?Quoted 5 lines
Ohh wait no, the dirty bitmap doesn't start on a byte boundary because the format of the bitmap is [uptodate bits][dirty bits].
Maybe those two should be reversed, because I bet the dirty state gets changed a lot more over the lifetime of a folio than the uptodate bits.
I think there's the
find_next_bit()helper (which Christoph also pointed out) that could probably be used here instead. Or at least that's how I see a lot of the driver code doing it for their bitmaps.
Re: [RFC PATCH v1 10/10] iomap: add granular dirty and writeback accounting
Quoted 39 lines · Joanne Koong
Quoted 24 lines · Joanne Koong
Quoted 14 lines · Darrick J. Wong
Quoted 12 lines · Joanne Koong
Add granular dirty and writeback accounting for large folios. These stats are used by the mm layer for dirty balancing and throttling. Having granular dirty and writeback accounting helps prevent over-aggressive balancing and throttling.
There are 4 places in iomap this commit affects:
a) filemap dirtying, which now callsfilemap_dirty_folio_pages()
b)writeback_iterwith setting thewbc->no_stats_accountingbit and callingclear_dirty_for_io_stats()
c) starting writeback, which now calls__folio_start_writeback()
d) ending writeback, which now callsfolio_end_writeback_pages()Quoted 5 lines
Quoted 3 lines
This relies on using the
ifs->statedirty bitmap to track dirty pages in the folio. As such, this can only be utilized on filesystems where the block size >=PAGE_SIZE.Apologies for my slow responses this month. :)
No worries at all, thanks for looking at this.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
I wonder, does this cause an observable change in the writeback accounting and throttling behavior for non-fuse filesystems like XFS that use large folios? I *think* this does actually reduce throttling for XFS, but it might not be so noticeable because the limits are much more generous outside of fuse?
I haven't run any benchmarks on non-fuse filesystems yet but that's what I would expect too. Will run some benchmarks to see!
I ran some benchmarks on xfs for the contrived test case I used for fuse (eg writing 2 GB in 128 MB chunks and then doing 50k 50-byte random writes) and I don't see any noticeable performance difference.
Quoted 8 lines
I re-tested it on fuse but this time with strictlimiting disabled and didn't notice any difference on that either, probably because with strictlimiting off we don't run into the upper limit in that test so there's no extra throttling that needs to be mitigated.
It's unclear to me how often (if at all?) real workloads run up against their dirty/writeback limits.
I benchmarked it again today but this time with manually setting /proc/sys/vm/dirty_bytes to 20% of 16 GiB and /proc/sys/vm/dirty_background_bytes to 10% of 16 GB and testing it on a more intense workload (the original test scenario but on 10+ threads) and and I see results now on xfs, around 3 seconds (with some variability of taking 0.3 seconds to 5 seconds sometimes) for writes prior to this patchset vs. a pretty consistent 0.14 seconds with this patchset. I ran the test scenario setup a few times but it'd be great if someone else could also run it to verify it shows up on their system too.
I set up xfs by following the instructions in the xfstests readme:
# xfs_io -f -c "falloc 0 10g" test.img
# xfs_io -f -c "falloc 0 10g" scratch.img
# mkfs.xfs test.img
# losetup /dev/loop0 ./test.img
# losetup /dev/loop1 ./scratch.img
# mkdir -p /mnt/test && mount /dev/loop0 /mnt/testand then ran:
sudo sysctl -w vm.dirty_bytes=$((3276 * 1024 * 1024)) # roughly 20% of 16GB sudo sysctl -w vm.dirty_background_bytes=$((1638*1024*1024)) # roughly 10% of 16 GB
and then ran this test program (ai-generated) https://pastebin.com/CbcwTXjq
I'll send out an updated v2 of this series.
Thanks,
Joanne