btrfs: simplify extent buffer writeback
cover btrfs: simplify extent buffer writeback
Hello,
We currently have two different paths for writing out extent buffers, a subpage path and a normal path. This has resulted in subtle bugs with subpage code that took us a while to figure out. Additionally we have this complex interaction of get folio, find eb, see if we already started writing that eb out, write out the eb.
We already have a radix tree for our extent buffers, so we can use that similarly to how pagecache uses the radix tree. Tag the buffers with DIRTY when they're dirty, and WRITEBACK when we start writing them out.
The unfortunate part is we have to re-implement folio_batch for extent buffers, so that's where most of the new code comes from. The good part is we are now down to a single path for writing out extent buffers, it's way simpler, and in fact quite a bit faster now that we don't have all of these folio->eb transitions to deal with.
I ran this through fsperf on a VM with 8 CPUs and 16gib of ram. I used smallfiles100k, but reduced the files to 1k to make it run faster, the results are as follows, with the statistically significant improvements marked with *, there were no regressions. fsperf was run with -n 10 for both runs, so the baseline is the average 10 runs and the test is the average of 10 runs.
smallfiles100k results
metric baseline current stdev diff
================================================================================avg_commit_ms 68.58 58.44 3.35 -14.79% *
commits 270.60 254.70 16.24 -5.88%
dev_read_iops 48 48 0 0.00% dev_read_kbytes 1044 1044 0 0.00% dev_write_iops 866117.90 850028.10 14292.20 -1.86% dev_write_kbytes 10939976.40 10605701.20 351330.32 -3.06% elapsed 49.30 33 1.64 -33.06% * end_state_mount_ns 41251498.80 35773220.70 2531205.32 -13.28% * end_state_umount_ns 1.90e+09 1.50e+09 14186226.85 -21.38% * max_commit_ms 139 111.60 9.72 -19.71% * sys_cpu 4.90 3.86 0.88 -21.29% write_bw_bytes 42935768.20 64318451.10 1609415.05 49.80% * write_clat_ns_mean 366431.69 243202.60 14161.98 -33.63% * write_clat_ns_p50 49203.20 20992 264.40 -57.34% * write_clat_ns_p99 827392 653721.60 65904.74 -20.99% * write_io_kbytes 2035940 2035940 0 0.00% write_iops 10482.37 15702.75 392.92 49.80% * write_lat_ns_max 1.01e+08 90516129 3910102.06 -10.29% *
write_lat_ns_mean 366556.19 243308.48 14154.51 -33.62% *
As you can see we get about a 33% decrease runtime, with a 50% throughput increase, which is pretty significant. Thanks,
Josef
Josef Bacik (2):
btrfs: set DIRTY and WRITEBACK tags on the buffer_radix
btrfs: use buffer radix for extent buffer writeback operations
fs/btrfs/disk-io.c | 10 ++ fs/btrfs/extent_io.c | 385 ++++++++++++++++++++++------------------- fs/btrfs/extent_io.h | 1 + fs/btrfs/transaction.c | 5 +- 4 files changed, 224 insertions(+), 177 deletions(-)
Signature · 1 line
2.48.1
PATCH 1/2 btrfs: set DIRTY and WRITEBACK tags on the buffer_radix
In preparation for changing how we do writeout of extent buffers, start tagging the extent buffer radix with DIRTY and WRITEBACK to make it easier to find extent buffers that are in either state.
--- fs/btrfs/extent_io.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
fs/btrfs/extent_io.c +41 -0
@@ -1801,8 +1801,19 @@ static noinline_for_stack bool lock_extent_buffer_for_io(struct extent_buffer *e*/ spin_lock (&eb->refs_lock);if (test_and_clear_bit (EXTENT_BUFFER_DIRTY , &eb->bflags)) { +XA_STATE (xas, &fs_info->buffer_radix, + eb->start >> fs_info->sectorsize_bits); +unsigned long flags; +set_bit (EXTENT_BUFFER_WRITEBACK , &eb->bflags);spin_unlock (&eb->refs_lock); + +xas_lock_irqsave (&xas, flags); +xas_load (&xas); +xas_set_mark (&xas,PAGECACHE_TAG_WRITEBACK ); +xas_clear_mark (&xas,PAGECACHE_TAG_DIRTY ); +xas_unlock_irqrestore (&xas, flags); +btrfs_set_header_flag (eb,BTRFS_HEADER_FLAG_WRITTEN ); percpu_counter_add_batch(&fs_info->dirty_metadata_bytes, -eb->len,
@@ -1888,6 +1899,33 @@ static void set_btree_ioerr(struct extent_buffer *eb) } } +static void buffer_radix_set_mark (const struct extent_buffer *eb,xa_mark_t mark) +{ +struct btrfs_fs_info *fs_info = eb->fs_info; +XA_STATE (xas, &fs_info->buffer_radix, + eb->start >> fs_info->sectorsize_bits); +unsigned long flags; + +xas_lock_irqsave (&xas, flags); +xas_load (&xas); +xas_set_mark (&xas, mark); +xas_unlock_irqrestore (&xas, flags); +} + +static void buffer_radix_clear_mark (const struct extent_buffer *eb, +xa_mark_t mark) +{ +struct btrfs_fs_info *fs_info = eb->fs_info; +XA_STATE (xas, &fs_info->buffer_radix, + eb->start >> fs_info->sectorsize_bits); +unsigned long flags; + +xas_lock_irqsave (&xas, flags); +xas_load (&xas); +xas_clear_mark (&xas, mark); +xas_unlock_irqrestore (&xas, flags); +} +/* * The endio specific version which won't touch any unsafe spinlock in endio * context.
@@ -1920,6 +1958,7 @@ static void end_bbio_meta_write(struct btrfs_bio *bbio)btrfs_meta_folio_clear_writeback (fi.folio, eb); } +buffer_radix_clear_mark (eb,PAGECACHE_TAG_WRITEBACK );clear_bit (EXTENT_BUFFER_WRITEBACK , &eb->bflags);smp_mb__after_atomic ();wake_up_bit (&eb->bflags,EXTENT_BUFFER_WRITEBACK );
@@ -3530,6 +3569,7 @@ void btrfs_clear_buffer_dirty(struct btrfs_trans_handle *trans,if (!test_and_clear_bit (EXTENT_BUFFER_DIRTY , &eb->bflags))return ; +buffer_radix_clear_mark (eb,PAGECACHE_TAG_DIRTY );percpu_counter_add_batch (&fs_info->dirty_metadata_bytes, -eb->len, fs_info->dirty_metadata_batch);
@@ -3578,6 +3618,7 @@ void set_extent_buffer_dirty(struct extent_buffer *eb)folio_lock (eb->folios[0 ]);for (int i =0 ; i <num_extent_folios (eb); i++)btrfs_meta_folio_set_dirty (eb->folios[i], eb); +buffer_radix_set_mark (eb,PAGECACHE_TAG_DIRTY );if (subpage)folio_unlock (eb->folios[0 ]); percpu_counter_add_batch(&eb->fs_info->dirty_metadata_bytes,
Signature · 1 line
2.48.1
PATCH 2/2 btrfs: use buffer radix for extent buffer writeback operations
In reply to Josef BacikCurrently we have this ugly back and forth with the btree writeback where we find the folio, find the eb associated with that folio, and then attempt to writeback. This results in two different paths for subpage eb's and >= pagesize eb's.
Clean this up by adding our own infrastructure around looking up tag'ed eb's and writing the eb's out directly. This allows us to unify the subpage and >= pagesize IO paths, resulting in a much cleaner writeback path for extent buffers.
I ran this through fsperf on a VM with 8 CPUs and 16gib of ram. I used smallfiles100k, but reduced the files to 1k to make it run faster, the results are as follows, with the statistically significant improvements marked with *, there were no regressions. fsperf was run with -n 10 for both runs, so the baseline is the average 10 runs and the test is the average of 10 runs.
smallfiles100k results
metric baseline current stdev diff
================================================================================avg_commit_ms 68.58 58.44 3.35 -14.79% *
commits 270.60 254.70 16.24 -5.88%
dev_read_iops 48 48 0 0.00% dev_read_kbytes 1044 1044 0 0.00% dev_write_iops 866117.90 850028.10 14292.20 -1.86% dev_write_kbytes 10939976.40 10605701.20 351330.32 -3.06% elapsed 49.30 33 1.64 -33.06% * end_state_mount_ns 41251498.80 35773220.70 2531205.32 -13.28% * end_state_umount_ns 1.90e+09 1.50e+09 14186226.85 -21.38% * max_commit_ms 139 111.60 9.72 -19.71% * sys_cpu 4.90 3.86 0.88 -21.29% write_bw_bytes 42935768.20 64318451.10 1609415.05 49.80% * write_clat_ns_mean 366431.69 243202.60 14161.98 -33.63% * write_clat_ns_p50 49203.20 20992 264.40 -57.34% * write_clat_ns_p99 827392 653721.60 65904.74 -20.99% * write_io_kbytes 2035940 2035940 0 0.00% write_iops 10482.37 15702.75 392.92 49.80% * write_lat_ns_max 1.01e+08 90516129 3910102.06 -10.29% *
write_lat_ns_mean 366556.19 243308.48 14154.51 -33.62% *
As you can see we get about a 33% decrease runtime, with a 50% throughput increase, which is pretty significant.
--- fs/btrfs/disk-io.c | 10 ++ fs/btrfs/extent_io.c | 344 ++++++++++++++++++++--------------------- fs/btrfs/extent_io.h | 1 + fs/btrfs/transaction.c | 5 +- 4 files changed, 183 insertions(+), 177 deletions(-)
fs/btrfs/disk-io.c +10 -0
@@ -2762,10 +2762,20 @@ static int __cold init_tree_roots(struct btrfs_fs_info *fs_info)return ret; } +static struct lock_class_key buffer_radix_lock_key; +void btrfs_init_fs_info (struct btrfs_fs_info *fs_info) {INIT_RADIX_TREE (&fs_info->fs_roots_radix,GFP_ATOMIC );INIT_RADIX_TREE (&fs_info->buffer_radix,GFP_ATOMIC ); + +/* +* We have to do this because we also use a radix tree for the delayed +* inodes, and lockdep will sometimes confuse that radix tree for this +* one and then complain about IRQ locking differences. +*/ +lockdep_set_class (&fs_info->buffer_radix.xa_lock, + &buffer_radix_lock_key);INIT_LIST_HEAD (&fs_info->trans_list);INIT_LIST_HEAD (&fs_info->dead_roots);INIT_LIST_HEAD (&fs_info->delayed_iputs);
fs/btrfs/extent_io.c +170 -174
395 lines not on this page · raw message
fs/btrfs/extent_io.h +1 -0
@@ -240,6 +240,7 @@ void extent_write_locked_range(struct inode *inode, const struct folio *locked_fint btrfs_writepages (struct address_space *mapping,struct writeback_control *wbc);int btree_write_cache_pages (struct address_space *mapping,struct writeback_control *wbc); +void btree_wait_writeback_range (struct btrfs_fs_info *fs_info,u64 start,u64 end);void btrfs_readahead (struct readahead_control *rac);int set_folio_extent_mapped (struct folio *folio);void clear_folio_extent_mapped (struct folio *folio);
fs/btrfs/transaction.c +2 -3
@@ -1155,7 +1155,7 @@ int btrfs_write_marked_extents(struct btrfs_fs_info *fs_info,if (!ret) ret =filemap_fdatawrite_range (mapping, start, end);if (!ret && wait_writeback) - ret =filemap_fdatawait_range (mapping, start, end); +btree_wait_writeback_range (fs_info, start, end);btrfs_free_extent_state (cached_state);if (ret)break ;
@@ -1175,7 +1175,6 @@ int btrfs_write_marked_extents(struct btrfs_fs_info *fs_info,static int __btrfs_wait_marked_extents (struct btrfs_fs_info *fs_info,struct extent_io_tree *dirty_pages) { -struct address_space *mapping = fs_info->btree_inode->i_mapping;struct extent_state *cached_state =NULL ;u64 start =0 ;u64 end;
@@ -1196,7 +1195,7 @@ static int __btrfs_wait_marked_extents(struct btrfs_fs_info *fs_info,if (ret == -ENOMEM ) ret =0 ;if (!ret) - ret =filemap_fdatawait_range (mapping, start, end); +btree_wait_writeback_range (fs_info, start, end);btrfs_free_extent_state (cached_state);if (ret)break ;
Signature · 1 line
2.48.1
Quoted 46 lines · Josef Bacik
Hello,
We currently have two different paths for writing out extent buffers, a subpage path and a normal path. This has resulted in subtle bugs with subpage code that took us a while to figure out. Additionally we have this complex interaction of get folio, find eb, see if we already started writing that eb out, write out the eb.
We already have a radix tree for our extent buffers, so we can use that similarly to how pagecache uses the radix tree. Tag the buffers with DIRTY when they're dirty, and WRITEBACK when we start writing them out.
The unfortunate part is we have to re-implement
folio_batchfor extent buffers, so that's where most of the new code comes from. The good part is we are now down to a single path for writing out extent buffers, it's way simpler, and in fact quite a bit faster now that we don't have all of thesefolio->ebtransitions to deal with.I ran this through fsperf on a VM with 8 CPUs and 16gib of ram. I used smallfiles100k, but reduced the files to 1k to make it run faster, the results are as follows, with the statistically significant improvements marked with *, there were no regressions. fsperf was run with -n 10 for both runs, so the baseline is the average 10 runs and the test is the average of 10 runs.
smallfiles100k results
metric baseline current stdev diff
================================================================================avg_commit_ms68.58 58.44 3.35 -14.79% *
commits 270.60 254.70 16.24 -5.88%dev_read_iops 48 48 0 0.00% dev_read_kbytes 1044 1044 0 0.00% dev_write_iops 866117.90 850028.10 14292.20 -1.86% dev_write_kbytes 10939976.40 10605701.20 351330.32 -3.06% elapsed 49.30 33 1.64 -33.06% * end_state_mount_ns 41251498.80 35773220.70 2531205.32 -13.28% * end_state_umount_ns 1.90e+09 1.50e+09 14186226.85 -21.38% * max_commit_ms 139 111.60 9.72 -19.71% * sys_cpu 4.90 3.86 0.88 -21.29% write_bw_bytes 42935768.20 64318451.10 1609415.05 49.80% * write_clat_ns_mean 366431.69 243202.60 14161.98 -33.63% * write_clat_ns_p50 49203.20 20992 264.40 -57.34% * write_clat_ns_p99 827392 653721.60 65904.74 -20.99% * write_io_kbytes 2035940 2035940 0 0.00% write_iops 10482.37 15702.75 392.92 49.80% * write_lat_ns_max 1.01e+08 90516129 3910102.06 -10.29% *
Quoted 4 lines
write_lat_ns_mean366556.19 243308.48 14154.51 -33.62% *As you can see we get about a 33% decrease runtime, with a 50% throughput increase, which is pretty significant. Thanks,
Ignore this for now, the xarray<->radix thing isn't quite one to one, so I've got to convert the buffer radix to a proper xarray first. Thanks,
Josef