btrfs: introduce btrfs_bio::async_csum
v2 cover btrfs: introduce btrfs_bio::async_csum
[CHANGELOG]
v2:
- Fix a race where csum generation can be completely skipped If the lower storage layer advanced the bi_iter before we finished calculating the checksum, bio_one_bio() may skip generating csum completely.
Unfortunately to solve this, a new bvec_iter must be introduced and cause the size of btrfs_bio to increase.
- Remove btrfs_bio::fs_info by always requireing btrfs_bio::inode For scrub, use btree inode instead, and for the only function that requires to be called by scrub, btrfs_submit_repair_write(), use a new member, btrfs_bio::is_scrub, to check the callers.
- Add a new patch to cleanup the recursive including problems The idea is to keep the btrfs headers inside .h files to minimal, thus lower risk of recursive including.
- Remove BTRFS_MAX_BIO_SECTORS macro
It's the same as BIO_MAX_VECS, and only utilized once.
The new async_csum feature will allow btrfs to calculate checksum for data write bios and submit them in parallel.
This will reduce latency and improve write throughput when data checksum is utilized.
This will slightly reclaim the performance drop after commit 968f19c5b1b7 ("btrfs: always fallback to buffered write if the inode requires checksum").
Patch 1 is a minor cleanup to remove a single-used macro. Patch 2 is a preparation to remove btrfs_bio::fs_info, the core is to avoid recursive including.
Patch 3 is to remove btrfs_bio::fs_info, as soon we will increase the size of btrfs_bio by 16 bytes.
Patch 4~6 are preparations to make sure btrfs_bio::end_io() is called in task context.
Patch 7 enables the async checksum generation for data writes, under experimental flags.
The series will increase btrfs_bio size by 8 bytes (+16 for the new structurs, -8 for the removal of btrfs_bio::fs_info).
Since the new async_csum should be better than the csum offload anyway, we may want to deprecate the csum offload feature completely in the future.
Thankfully csum offload feature is still behind the experimental flag, thus it should not affect end users.
Qu Wenruo (7):
btrfs: replace BTRFS_MAX_BIO_SECTORS with BIO_MAX_VECS
btrfs: headers cleanup to remove unnecessary local includes
btrfs: remove btrfs_bio::fs_info by extracting it from
btrfs_bio::inode
btrfs: make sure all btrfs_bio::end_io is called in task context
btrfs: remove btrfs_fs_info::compressed_write_workers
btrfs: relax btrfs_inode::ordered_tree_lock
btrfs: introduce btrfs_bio::async_csum
fs/btrfs/accessors.h | 1 + fs/btrfs/bio.c | 136 ++++++++++++++++++++++++++-------------- fs/btrfs/bio.h | 32 ++++++---- fs/btrfs/btrfs_inode.h | 8 +-- fs/btrfs/compression.c | 33 +++------- fs/btrfs/compression.h | 13 ++-- fs/btrfs/ctree.h | 2 - fs/btrfs/defrag.c | 1 + fs/btrfs/dir-item.c | 1 + fs/btrfs/direct-io.c | 8 +-- fs/btrfs/disk-io.c | 10 +-- fs/btrfs/disk-io.h | 3 +- fs/btrfs/extent-tree.c | 1 + fs/btrfs/extent_io.c | 27 ++++---- fs/btrfs/extent_io.h | 1 - fs/btrfs/extent_map.h | 3 +- fs/btrfs/file-item.c | 64 +++++++++++++------ fs/btrfs/file-item.h | 4 +- fs/btrfs/fs.h | 1 - fs/btrfs/inode.c | 12 ++-- fs/btrfs/ordered-data.c | 57 ++++++++--------- fs/btrfs/scrub.c | 49 ++++++++------- fs/btrfs/space-info.c | 1 + fs/btrfs/subpage.h | 1 - fs/btrfs/transaction.c | 2 + fs/btrfs/transaction.h | 4 -- fs/btrfs/tree-log.c | 5 +- fs/btrfs/tree-log.h | 3 +- fs/btrfs/zoned.c | 4 +- fs/btrfs/zoned.h | 1 - 30 files changed, 264 insertions(+), 224 deletions(-)
Signature · 1 line
2.51.0
PATCH v2 1/7 btrfs: replace BTRFS_MAX_BIO_SECTORS with BIO_MAX_VECS
It's impossible to have a btrfs bio with more than BIO_MAX_VECS vectors anyway.
And there is only one location utilizing that macro, just replace it with BIO_MAX_VECS.
--- fs/btrfs/bio.h | 7 ------- fs/btrfs/direct-io.c | 2 +- 2 files changed, 1 insertion(+), 8 deletions(-)
fs/btrfs/bio.h +0 -7
@@ -18,13 +18,6 @@ struct btrfs_inode;#define BTRFS_BIO_INLINE_CSUM_SIZE 64 -/* -* Maximum number of sectors for a single bio to limit the size of the -* checksum array. This matches the number of bio_vecs per bio and thus the -* I/O size for buffered I/O. -*/ -#define BTRFS_MAX_BIO_SECTORS (256) -typedef void (*btrfs_bio_end_io_t )(struct btrfs_bio *bbio);/*
fs/btrfs/direct-io.c +1 -1
@@ -385,7 +385,7 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start,* to allocate a contiguous array for the checksums. */ if (!write) - len =min_t (u64, len, fs_info->sectorsize *BTRFS_MAX_BIO_SECTORS ); + len =min_t (u64, len, fs_info->sectorsize *BIO_MAX_VECS ); lockstart = start; lockend = start + len -1 ;
Signature · 1 line
2.51.0
PATCH v2 2/7 btrfs: headers cleanup to remove unnecessary local includes
In reply to Qu Wenruo[BUG]
When I tried to remove btrfs_bio::fs_info and use btrfs_bio::inode to grab the fs_info, the header "btrfs_inode.h" is needed to access the full btrfs_inode structure.
Then btrfs will fail to compile.
[CAUSE]
There is a recursive including chain:
"bio.h" -> "btrfs_inode.h" -> "extent_map.h" -> "compression.h" -> "bio.h"
That recursive including is causing problems for btrfs.
[ENHANCEMENT]
To reduce the risk of recursive including:
- Remove unnecessary local includes from btrfs headers Either the included header is pulled in by other headers, or is completely unnecessary.
- Remove btrfs local includes if the header only requires a pointer In that case let the implementing C file to pull the required header.
This is especially important for headers like "btrfs_inode.h" which pulls in a lot of other btrfs headers, thus it's a mine field of recursive including.
- Remove unnecessary temporary structure definition Either if we have included the header defining the structure, or completely unused.
Now including "btrfs_inode.h" inside "bio.h" is completely fine, although "btrfs_inode.h" still includes "extent_map.h", but that header only includes "fs.h", no more chain back to "bio.h".
--- fs/btrfs/accessors.h | 1 + fs/btrfs/btrfs_inode.h | 8 ++++---- fs/btrfs/compression.h | 3 --- fs/btrfs/ctree.h | 2 -- fs/btrfs/defrag.c | 1 + fs/btrfs/dir-item.c | 1 + fs/btrfs/direct-io.c | 2 ++ fs/btrfs/disk-io.c | 1 + fs/btrfs/disk-io.h | 3 ++- fs/btrfs/extent-tree.c | 1 + fs/btrfs/extent_io.h | 1 - fs/btrfs/extent_map.h | 3 +-- fs/btrfs/file-item.h | 2 +- fs/btrfs/inode.c | 1 + fs/btrfs/space-info.c | 1 + fs/btrfs/subpage.h | 1 - fs/btrfs/transaction.c | 2 ++ fs/btrfs/transaction.h | 4 ---- fs/btrfs/tree-log.c | 1 + fs/btrfs/tree-log.h | 3 +-- fs/btrfs/zoned.h | 1 - 21 files changed, 21 insertions(+), 22 deletions(-)
fs/btrfs/accessors.h +1 -0
@@ -12,6 +12,7 @@#include <linux/string.h> #include <linux/mm.h> #include <uapi/linux/btrfs_tree.h> +#include "fs.h" #include "extent_io.h" struct extent_buffer ;
fs/btrfs/btrfs_inode.h +4 -4
@@ -18,20 +18,20 @@#include <linux/lockdep.h> #include <uapi/linux/btrfs_tree.h> #include <trace/events/btrfs.h> +#include "ctree.h" #include "block-rsv.h" #include "extent_map.h" -#include "extent_io.h" #include "extent-io-tree.h" -#include "ordered-data.h" -#include "delayed-inode.h" -struct extent_state ;struct posix_acl ;struct iov_iter ;struct writeback_control ;struct btrfs_root ;struct btrfs_fs_info ;struct btrfs_trans_handle ; +struct btrfs_bio ; +struct btrfs_file_extent ; +struct btrfs_delayed_node ;/* * Since we search a directory based on f_pos (struct dir_context::pos) we have
fs/btrfs/compression.h +0 -3
@@ -14,14 +14,11 @@#include <linux/pagemap.h> #include "bio.h" #include "fs.h" -#include "messages.h" struct address_space ; -struct page ;struct inode ;struct btrfs_inode ;struct btrfs_ordered_extent ; -struct btrfs_bio ;/* * We want to make sure that amount of RAM required to uncompress an extent is
fs/btrfs/ctree.h +0 -2
@@ -17,9 +17,7 @@#include <linux/refcount.h> #include <uapi/linux/btrfs_tree.h> #include "locking.h" -#include "fs.h" #include "accessors.h" -#include "extent-io-tree.h" struct extent_buffer ;struct btrfs_block_rsv ;
fs/btrfs/defrag.c +1 -0
@@ -15,6 +15,7 @@#include "defrag.h" #include "file-item.h" #include "super.h" +#include "compression.h" static struct kmem_cache *btrfs_inode_defrag_cachep;
fs/btrfs/dir-item.c +1 -0
@@ -9,6 +9,7 @@#include "transaction.h" #include "accessors.h" #include "dir-item.h" +#include "delayed-inode.h" /* * insert a name into a directory, doing overflow properly if there is a hash
fs/btrfs/direct-io.c +2 -0
@@ -10,6 +10,8 @@#include "fs.h" #include "transaction.h" #include "volumes.h" +#include "bio.h" +#include "ordered-data.h" struct btrfs_dio_data {ssize_t submitted;
fs/btrfs/disk-io.c +1 -0
@@ -50,6 +50,7 @@#include "relocation.h" #include "scrub.h" #include "super.h" +#include "delayed-inode.h" #define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\ BTRFS_HEADER_FLAG_RELOC |\
fs/btrfs/disk-io.h +2 -1
@@ -9,7 +9,8 @@#include <linux/sizes.h> #include <linux/compiler_types.h> #include "ctree.h" -#include "fs.h" +#include "bio.h" +#include "ordered-data.h" struct block_device ;struct super_block ;
fs/btrfs/extent-tree.c +1 -0
@@ -40,6 +40,7 @@#include "orphan.h" #include "tree-checker.h" #include "raid-stripe-tree.h" +#include "delayed-inode.h" #undef SCRAMBLE_DELAYED_REFS
fs/btrfs/extent_io.h +0 -1
@@ -12,7 +12,6 @@#include <linux/rwsem.h> #include <linux/list.h> #include <linux/slab.h> -#include "compression.h" #include "messages.h" #include "ulist.h" #include "misc.h"
fs/btrfs/extent_map.h +1 -2
@@ -8,8 +8,7 @@#include <linux/rbtree.h> #include <linux/list.h> #include <linux/refcount.h> -#include "misc.h" -#include "compression.h" +#include "fs.h" struct btrfs_inode ;struct btrfs_fs_info ;
fs/btrfs/file-item.h +1 -1
@@ -7,7 +7,7 @@#include <linux/list.h> #include <uapi/linux/btrfs_tree.h> #include "ctree.h" -#include "accessors.h" +#include "ordered-data.h" struct extent_map ;struct btrfs_file_extent_item ;
fs/btrfs/inode.c +1 -0
@@ -71,6 +71,7 @@#include "backref.h" #include "raid-stripe-tree.h" #include "fiemap.h" +#include "delayed-inode.h" #define COW_FILE_RANGE_KEEP_LOCKED (1UL << 0)#define COW_FILE_RANGE_NO_INLINE (1UL << 1)
fs/btrfs/space-info.c +1 -0
@@ -15,6 +15,7 @@#include "accessors.h" #include "extent-tree.h" #include "zoned.h" +#include "delayed-inode.h" /* * HOW DOES SPACE RESERVATION WORK
fs/btrfs/subpage.h +0 -1
@@ -7,7 +7,6 @@#include <linux/atomic.h> #include <linux/sizes.h> #include "btrfs_inode.h" -#include "fs.h" struct address_space ;struct folio ;
fs/btrfs/transaction.c +2 -0
@@ -32,6 +32,8 @@#include "ioctl.h" #include "relocation.h" #include "scrub.h" +#include "ordered-data.h" +#include "delayed-inode.h" static struct kmem_cache *btrfs_trans_handle_cachep;
fs/btrfs/transaction.h +0 -4
@@ -14,10 +14,6 @@#include <linux/wait.h> #include "btrfs_inode.h" #include "delayed-ref.h" -#include "extent-io-tree.h" -#include "block-rsv.h" -#include "messages.h" -#include "misc.h" struct dentry ;struct inode ;
fs/btrfs/tree-log.c +1 -0
@@ -29,6 +29,7 @@#include "orphan.h" #include "print-tree.h" #include "tree-checker.h" +#include "delayed-inode.h" #define MAX_CONFLICT_INODES 10
fs/btrfs/tree-log.h +1 -2
@@ -8,8 +8,7 @@#include <linux/list.h> #include <linux/fs.h> -#include "messages.h" -#include "ctree.h" +#include <linux/fscrypt.h> #include "transaction.h" struct inode ;
fs/btrfs/zoned.h +0 -1
@@ -15,7 +15,6 @@#include "disk-io.h" #include "block-group.h" #include "btrfs_inode.h" -#include "fs.h" struct block_device ;struct extent_buffer ;
Signature · 1 line
2.51.0
PATCH v2 3/7 btrfs: remove btrfs_bio::fs_info by extracting it from btrfs_bio::inode
In reply to Qu WenruoCurrently there is only one caller which doesn't populate btrfs_bio::inode, and that's scrub.
The idea is scrub doesn't want any automatic csum verification nor read-repair, as everything will be handled by scrub itself.
However that behavior is really no different than metadata inode, thus we can reuse btree_inode as btrfs_bio::inode for scrub.
The only exception is in btrfs_submit_chunk() where if a bbio is from scrub or data reloc inode, we set rst_search_commit_root to true. This means we still need a way to distinguish scrub from metadata, but that can be done by a new flag inside btrfs_bio.
Now btrfs_bio::inode is a mandatory parameter, we can extract fs_info from that inode thus can remove btrfs_bio::fs_info to save 8 bytes from btrfs_bio structure.
--- fs/btrfs/bio.c | 51 ++++++++++++++++++++++-------------------- fs/btrfs/bio.h | 18 ++++++++++----- fs/btrfs/compression.c | 6 ++--- fs/btrfs/compression.h | 3 ++- fs/btrfs/direct-io.c | 4 +--- fs/btrfs/extent_io.c | 22 ++++++++---------- fs/btrfs/inode.c | 7 ++---- fs/btrfs/scrub.c | 49 ++++++++++++++++++++++------------------ fs/btrfs/zoned.c | 4 ++-- 9 files changed, 84 insertions(+), 80 deletions(-)
fs/btrfs/bio.c +27 -24
@@ -41,13 +41,17 @@ static bool bbio_has_ordered_extent(const struct btrfs_bio *bbio)* Initialize a btrfs_bio structure. This skips the embedded bio itself as it * is already initialized by the block layer. */ -void btrfs_bio_init (struct btrfs_bio *bbio,struct btrfs_fs_info *fs_info, +void btrfs_bio_init (struct btrfs_bio *bbio,struct btrfs_inode *inode,u64 file_offset,btrfs_bio_end_io_t end_io,void *private) { +/* @inode parameter is mandatory. */ +ASSERT (inode); +memset (bbio,0 , offsetof(struct btrfs_bio , bio)); - bbio->fs_info = fs_info; + bbio->inode = inode; bbio->end_io = end_io; bbio->private = private; + bbio->file_offset = file_offset;atomic_set (&bbio->pending_ios,1 );WRITE_ONCE (bbio->status,BLK_STS_OK ); }
@@ -60,7 +64,7 @@ void btrfs_bio_init(struct btrfs_bio *bbio, struct btrfs_fs_info *fs_info,* a mempool. */ struct btrfs_bio *btrfs_bio_alloc (unsigned int nr_vecs,blk_opf_t opf, -struct btrfs_fs_info *fs_info, +struct btrfs_inode *inode,u64 file_offset,btrfs_bio_end_io_t end_io,void *private) {struct btrfs_bio *bbio;
@@ -68,7 +72,7 @@ struct btrfs_bio *btrfs_bio_alloc(unsigned int nr_vecs, blk_opf_t opf, bio =bio_alloc_bioset (NULL , nr_vecs, opf,GFP_NOFS , &btrfs_bioset); bbio =btrfs_bio (bio); -btrfs_bio_init (bbio, fs_info, end_io, private); +btrfs_bio_init (bbio, inode, file_offset, end_io, private);return bbio; }
@@ -85,9 +89,7 @@ static struct btrfs_bio *btrfs_split_bio(struct btrfs_fs_info *fs_info,return ERR_CAST (bio); bbio =btrfs_bio (bio); -btrfs_bio_init (bbio, fs_info,NULL , orig_bbio); - bbio->inode = orig_bbio->inode; - bbio->file_offset = orig_bbio->file_offset; +btrfs_bio_init (bbio, orig_bbio->inode, orig_bbio->file_offset,NULL , orig_bbio); orig_bbio->file_offset += map_length;if (bbio_has_ordered_extent (bbio)) {refcount_inc (&orig_bbio->ordered->refs);
@@ -244,9 +246,8 @@ static struct btrfs_failed_bio *repair_one_sector(struct btrfs_bio *failed_bbio,bio_add_folio_nofail (repair_bio, folio, sectorsize, foff); repair_bbio =btrfs_bio (repair_bio); -btrfs_bio_init (repair_bbio, fs_info,NULL , fbio); - repair_bbio->inode = failed_bbio->inode; - repair_bbio->file_offset = failed_bbio->file_offset + bio_offset; +btrfs_bio_init (repair_bbio, failed_bbio->inode, failed_bbio->file_offset + bio_offset, +NULL , fbio); mirror =next_repair_mirror (fbio, failed_bbio->mirror_num);btrfs_debug (fs_info,"submitting repair read to mirror %d" , mirror);
@@ -332,7 +333,7 @@ static void btrfs_simple_end_io(struct bio *bio) {struct btrfs_bio *bbio =btrfs_bio (bio);struct btrfs_device *dev = bio->bi_private; -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;btrfs_bio_counter_dec (fs_info);
@@ -581,10 +582,11 @@ static void run_one_async_done(struct btrfs_work *work, bool do_free)static bool should_async_write (struct btrfs_bio *bbio) { +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;bool auto_csum_mode = true;#ifdef CONFIG_BTRFS_EXPERIMENTAL -struct btrfs_fs_devices *fs_devices = bbio->fs_info->fs_devices; +struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode);if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF )
@@ -594,7 +596,7 @@ static bool should_async_write(struct btrfs_bio *bbio)#endif /* Submit synchronously if the checksum implementation is fast. */ -if (auto_csum_mode &&test_bit (BTRFS_FS_CSUM_IMPL_FAST , &bbio->fs_info->flags)) +if (auto_csum_mode &&test_bit (BTRFS_FS_CSUM_IMPL_FAST , &fs_info->flags))return false;/*
@@ -605,7 +607,7 @@ static bool should_async_write(struct btrfs_bio *bbio)return false;/* Zoned devices require I/O to be submitted in order. */ -if ((bbio->bio.bi_opf &REQ_META ) &&btrfs_is_zoned (bbio->fs_info)) +if ((bbio->bio.bi_opf &REQ_META ) &&btrfs_is_zoned (fs_info))return false;return true;
@@ -620,7 +622,7 @@ static bool btrfs_wq_submit_bio(struct btrfs_bio *bbio,struct btrfs_io_context *bioc, struct btrfs_io_stripe *smap, int mirror_num) { - struct btrfs_fs_info *fs_info = bbio->fs_info; + struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;struct async_submit_bio *async; async =kmalloc (sizeof (*async),GFP_NOFS );
@@ -639,11 +641,12 @@ static bool btrfs_wq_submit_bio(struct btrfs_bio *bbio,static u64 btrfs_append_map_length (struct btrfs_bio *bbio,u64 map_length) { +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;unsigned int nr_segs;int sector_offset; - map_length =min (map_length, bbio->fs_info->max_zone_append_size); - sector_offset =bio_split_rw_at (&bbio->bio, &bbio->fs_info->limits, + map_length =min (map_length, fs_info->max_zone_append_size); + sector_offset =bio_split_rw_at (&bbio->bio, &fs_info->limits, &nr_segs, map_length);if (sector_offset) {/*
@@ -651,7 +654,7 @@ static u64 btrfs_append_map_length(struct btrfs_bio *bbio, u64 map_length)* sectorsize and thus cause unaligned I/Os. Fix that by * always rounding down to the nearest boundary. */ -return ALIGN_DOWN (sector_offset <<SECTOR_SHIFT , bbio->fs_info->sectorsize); +return ALIGN_DOWN (sector_offset <<SECTOR_SHIFT , fs_info->sectorsize); }return map_length; }
@@ -659,7 +662,7 @@ static u64 btrfs_append_map_length(struct btrfs_bio *bbio, u64 map_length)static bool btrfs_submit_chunk (struct btrfs_bio *bbio,int mirror_num) {struct btrfs_inode *inode = bbio->inode; -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = inode->root->fs_info;struct bio *bio = &bbio->bio;u64 logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ;u64 length = bio->bi_iter.bi_size;
@@ -670,7 +673,7 @@ static bool btrfs_submit_chunk(struct btrfs_bio *bbio, int mirror_num)blk_status_t status;int ret; -if (!bbio->inode ||btrfs_is_data_reloc_root (inode->root)) +if (bbio->is_scrub ||btrfs_is_data_reloc_root (inode->root)) smap.rst_search_commit_root = true;else smap.rst_search_commit_root = false;
@@ -782,7 +785,7 @@ static bool btrfs_submit_chunk(struct btrfs_bio *bbio, int mirror_num)static void assert_bbio_alignment (struct btrfs_bio *bbio) {#ifdef CONFIG_BTRFS_ASSERT -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;struct bio_vec bvec;struct bvec_iter iter;const u32 blocksize = fs_info->sectorsize;
@@ -885,16 +888,16 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start,*/ void btrfs_submit_repair_write (struct btrfs_bio *bbio,int mirror_num,bool dev_replace) { -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;u64 logical = bbio->bio.bi_iter.bi_sector <<SECTOR_SHIFT ;u64 length = bbio->bio.bi_iter.bi_size;struct btrfs_io_stripe smap = {0 };int ret; -ASSERT (fs_info);ASSERT (mirror_num >0 );ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE ); -ASSERT (!bbio->inode); +ASSERT (!is_data_inode (bbio->inode)); +ASSERT (bbio->is_scrub);btrfs_bio_counter_inc_blocked (fs_info); ret =btrfs_map_repair_block (fs_info, &smap, logical, length, mirror_num);
fs/btrfs/bio.h +12 -6
@@ -27,7 +27,10 @@ typedef void (*btrfs_bio_end_io_t)(struct btrfs_bio *bbio);struct btrfs_bio {/* * Inode and offset into it that this I/O operates on. -* Only set for data I/O. +* +* If the inode is a data one, csum verification and read-repair +* will be done automatically. +* If the inode is a metadata one, everything is handled by the caller. */ struct btrfs_inode *inode;u64 file_offset;
@@ -69,14 +72,17 @@ struct btrfs_bio {atomic_t pending_ios;struct work_struct end_io_work; -/* File system that this I/O operates on. */ -struct btrfs_fs_info *fs_info; -/* Save the first error status of split bio. */ blk_status_t status;/* Use the commit root to look up csums (data read bio only). */ bool csum_search_commit_root; + +/* +* Since scrub will reuse btree inode, we need this flag to distinguish +* scrub bios. +*/ +bool is_scrub;/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last.
@@ -92,10 +98,10 @@ static inline struct btrfs_bio *btrfs_bio(struct bio *bio)int __init btrfs_bioset_init (void );void __cold btrfs_bioset_exit (void ); -void btrfs_bio_init (struct btrfs_bio *bbio,struct btrfs_fs_info *fs_info, +void btrfs_bio_init (struct btrfs_bio *bbio,struct btrfs_inode *inode,u64 file_offset,btrfs_bio_end_io_t end_io,void *private);struct btrfs_bio *btrfs_bio_alloc (unsigned int nr_vecs,blk_opf_t opf, -struct btrfs_fs_info *fs_info, +struct btrfs_inode *inode,u64 file_offset,btrfs_bio_end_io_t end_io,void *private);void btrfs_bio_end_io (struct btrfs_bio *bbio,blk_status_t status);
fs/btrfs/compression.c +2 -4
@@ -67,9 +67,7 @@ static struct compressed_bio *alloc_compressed_bio(struct btrfs_inode *inode, bbio =btrfs_bio (bio_alloc_bioset (NULL ,BTRFS_MAX_COMPRESSED_PAGES , op,GFP_NOFS , &btrfs_compressed_bioset)); -btrfs_bio_init (bbio, inode->root->fs_info, end_io,NULL ); - bbio->inode = inode; - bbio->file_offset = start; +btrfs_bio_init (bbio, inode, start, end_io,NULL );return to_compressed_bio (bbio); }
@@ -354,7 +352,7 @@ static void end_bbio_compressed_write(struct btrfs_bio *bbio)static void btrfs_add_compressed_bio_folios (struct compressed_bio *cb) { -struct btrfs_fs_info *fs_info = cb->bbio.fs_info; +struct btrfs_fs_info *fs_info = cb->bbio.inode->root->fs_info;struct bio *bio = &cb->bbio.bio;u32 offset =0 ;
fs/btrfs/compression.h +2 -1
@@ -14,6 +14,7 @@#include <linux/pagemap.h> #include "bio.h" #include "fs.h" +#include "btrfs_inode.h" struct address_space ;struct inode ;
@@ -74,7 +75,7 @@ struct compressed_bio {static inline struct btrfs_fs_info *cb_to_fs_info (const struct compressed_bio *cb) { -return cb->bbio.fs_info; +return cb->bbio.inode->root->fs_info; }/* @range_end must be exclusive. */
fs/btrfs/direct-io.c +1 -3
@@ -715,10 +715,8 @@ static void btrfs_dio_submit_io(const struct iomap_iter *iter, struct bio *bio, container_of(bbio, struct btrfs_dio_private, bbio);struct btrfs_dio_data *dio_data = iter->private; -btrfs_bio_init (bbio,BTRFS_I (iter->inode)->root->fs_info, +btrfs_bio_init (bbio,BTRFS_I (iter->inode), file_offset, btrfs_dio_end_io, bio->bi_private); - bbio->inode =BTRFS_I (iter->inode); - bbio->file_offset = file_offset; dip->file_offset = file_offset; dip->bytes = bio->bi_iter.bi_size;
fs/btrfs/extent_io.c +9 -13
@@ -517,7 +517,7 @@ static void end_folio_read(struct folio *folio, bool uptodate, u64 start, u32 le*/ static void end_bbio_data_write (struct btrfs_bio *bbio) { -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;struct bio *bio = &bbio->bio;int error =blk_status_to_errno (bio->bi_status);struct folio_iter fi;
@@ -573,7 +573,7 @@ static void begin_folio_read(struct btrfs_fs_info *fs_info, struct folio *folio)*/ static void end_bbio_data_read (struct btrfs_bio *bbio) { -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info;struct bio *bio = &bbio->bio;struct folio_iter fi;
@@ -738,12 +738,10 @@ static void alloc_new_bio(struct btrfs_inode *inode,struct btrfs_fs_info *fs_info = inode->root->fs_info;struct btrfs_bio *bbio; - bbio =btrfs_bio_alloc (BIO_MAX_VECS , bio_ctrl->opf, fs_info, - bio_ctrl->end_io_func,NULL ); + bbio =btrfs_bio_alloc (BIO_MAX_VECS , bio_ctrl->opf, inode, + file_offset, bio_ctrl->end_io_func,NULL ); bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; bbio->bio.bi_write_hint = inode->vfs_inode.i_write_hint; - bbio->inode = inode; - bbio->file_offset = file_offset; bio_ctrl->bbio = bbio; bio_ctrl->len_to_oe_boundary =U32_MAX ; bio_ctrl->next_file_offset = file_offset;
@@ -2224,12 +2222,11 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb, bbio =btrfs_bio_alloc (INLINE_EXTENT_BUFFER_PAGES ,REQ_OP_WRITE |REQ_META |wbc_to_write_flags (wbc), - eb->fs_info, end_bbio_meta_write, eb); +BTRFS_I (fs_info->btree_inode), eb->start, + end_bbio_meta_write, eb); bbio->bio.bi_iter.bi_sector = eb->start >>SECTOR_SHIFT ;bio_set_dev (&bbio->bio, fs_info->fs_devices->latest_dev->bdev);wbc_init_bio (wbc, &bbio->bio); - bbio->inode =BTRFS_I (eb->fs_info->btree_inode); - bbio->file_offset = eb->start;for (int i =0 ; i <num_extent_folios (eb); i++) {struct folio *folio = eb->folios[i];u64 range_start =max_t (u64, eb->start,folio_pos (folio));
@@ -3844,6 +3841,7 @@ static void end_bbio_meta_read(struct btrfs_bio *bbio)int read_extent_buffer_pages_nowait (struct extent_buffer *eb,int mirror_num,const struct btrfs_tree_parent_check *check) { +struct btrfs_fs_info *fs_info = eb->fs_info;struct btrfs_bio *bbio;if (test_bit (EXTENT_BUFFER_UPTODATE , &eb->bflags))
@@ -3877,11 +3875,9 @@ int read_extent_buffer_pages_nowait(struct extent_buffer *eb, int mirror_num,refcount_inc (&eb->refs); bbio =btrfs_bio_alloc (INLINE_EXTENT_BUFFER_PAGES , -REQ_OP_READ |REQ_META , eb->fs_info, - end_bbio_meta_read, eb); +REQ_OP_READ |REQ_META ,BTRFS_I (fs_info->btree_inode), + eb->start, end_bbio_meta_read, eb); bbio->bio.bi_iter.bi_sector = eb->start >>SECTOR_SHIFT ; - bbio->inode =BTRFS_I (eb->fs_info->btree_inode); - bbio->file_offset = eb->start;memcpy (&bbio->parent_check, check,sizeof (*check));for (int i =0 ; i <num_extent_folios (eb); i++) {struct folio *folio = eb->folios[i];
fs/btrfs/inode.c +2 -5
@@ -9404,7 +9404,6 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, u64 disk_bytenr, u64 disk_io_size, struct page **pages, void *uring_ctx) { - struct btrfs_fs_info *fs_info = inode->root->fs_info; struct btrfs_encoded_read_private *priv, sync_priv;struct completion sync_reads;unsigned long i =0 ;
@@ -9429,10 +9428,9 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, priv->status =0 ; priv->uring_ctx = uring_ctx; - bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , fs_info, + bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , inode,0 , btrfs_encoded_read_endio, priv); bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; - bbio->inode = inode; do { size_t bytes = min_t(u64, disk_io_size, PAGE_SIZE);
@@ -9441,10 +9439,9 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,refcount_inc (&priv->pending_refs);btrfs_submit_bbio (bbio,0 ); - bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , fs_info, + bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , inode,0 , btrfs_encoded_read_endio, priv); bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; - bbio->inode = inode;continue ; }
fs/btrfs/scrub.c +27 -22
88 lines not on this page · raw message
fs/btrfs/zoned.c +2 -2
@@ -1813,14 +1813,14 @@ bool btrfs_use_zone_append(struct btrfs_bio *bbio) {u64 start = (bbio->bio.bi_iter.bi_sector <<SECTOR_SHIFT );struct btrfs_inode *inode = bbio->inode; -struct btrfs_fs_info *fs_info = bbio->fs_info; +struct btrfs_fs_info *fs_info = inode->root->fs_info;struct btrfs_block_group *cache;bool ret = false;if (!btrfs_is_zoned (fs_info))return false; -if (!inode || !is_data_inode (inode)) +if (!is_data_inode (inode))return false;if (btrfs_op (&bbio->bio) !=BTRFS_MAP_WRITE )
Signature · 1 line
2.51.0
PATCH v2 4/7 btrfs: make sure all btrfs_bio::end_io is called in task context
In reply to Qu Wenruo[BACKGROUND]
Btrfs has a lot of different bi_end_io functions, to handle different raid profiles. But they introduced a lot of different context for btrfs_bio::end_io() calls:
- Simple read bios
Run in task context, backed by either endio_meta_workers or endio_workers.
- Simple write bios
Run in irq context.
- RAID56 write or rebuild bios
Run in task context, backed by rmw_workers.
- Mirrored write bios
Run in irq context.
This is very inconsistent, and contributes to the huge amount of workqueues used in btrfs.
[ENHANCEMENT]
Make all above bios to call their btrfs_bio::end_io() in task context, backed by either endio_meta_workers for metadata, or endio_workers for data.
For simple write bios, merge the handling into simple_end_io_work().
For mirrored write bios, it will be a little more complex, since both the original or the cloned bios can run the final btrfs_bio::end_io().
Here we make sure the cloned bios are using btrfs_bioset, to reuse the end_io_work, and run both original and cloned work inside the workqueue.
And add extra ASSERT()s to make sure btrfs_bio_end_io() is running in task context.
This not only unifies the context for btrfs_bio::end_io() functions, but also opens a new door for further btrfs_bio::end_io() related cleanups.
--- fs/btrfs/bio.c | 64 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 46 insertions(+), 18 deletions(-)
fs/btrfs/bio.c +46 -18
@@ -102,6 +102,9 @@ static struct btrfs_bio *btrfs_split_bio(struct btrfs_fs_info *fs_info,void btrfs_bio_end_io (struct btrfs_bio *bbio,blk_status_t status) { +/* Make sure we're already in task context. */ +ASSERT (in_task ()); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;
@@ -318,15 +321,20 @@ static struct workqueue_struct *btrfs_end_io_wq(const struct btrfs_fs_info *fs_ireturn fs_info->endio_workers; } -static void btrfs_end_bio_work (struct work_struct *work) +static void simple_end_io_work (struct work_struct *work) { struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, end_io_work); +struct bio *bio = &bbio->bio; -/* Metadata reads are checked and repaired by the submitter. */ -if (is_data_bbio (bbio)) -btrfs_check_read_bio (bbio, bbio->bio.bi_private); -else -btrfs_bio_end_io (bbio, bbio->bio.bi_status); +if (bio_op (bio) ==REQ_OP_READ ) { +/* Metadata reads are checked and repaired by the submitter. */ +if (is_data_bbio (bbio)) +return btrfs_check_read_bio (bbio, bbio->bio.bi_private); +return btrfs_bio_end_io (bbio, bbio->bio.bi_status); + } +if (bio_is_zone_append (bio) && !bio->bi_status) +btrfs_record_physical_zoned (bbio); +btrfs_bio_end_io (bbio, bbio->bio.bi_status); }static void btrfs_simple_end_io (struct bio *bio)
@@ -340,14 +348,8 @@ static void btrfs_simple_end_io(struct bio *bio)if (bio->bi_status)btrfs_log_dev_io_error (bio, dev); -if (bio_op (bio) ==REQ_OP_READ ) { -INIT_WORK (&bbio->end_io_work, btrfs_end_bio_work); -queue_work (btrfs_end_io_wq (fs_info, bio), &bbio->end_io_work); - }else { -if (bio_is_zone_append (bio) && !bio->bi_status) -btrfs_record_physical_zoned (bbio); -btrfs_bio_end_io (bbio, bbio->bio.bi_status); - } +INIT_WORK (&bbio->end_io_work, simple_end_io_work); +queue_work (btrfs_end_io_wq (fs_info, bio), &bbio->end_io_work); }static void btrfs_raid56_end_io (struct bio *bio)
@@ -355,6 +357,9 @@ static void btrfs_raid56_end_io(struct bio *bio)struct btrfs_io_context *bioc = bio->bi_private;struct btrfs_bio *bbio =btrfs_bio (bio); +/* RAID56 endio is always handled in workqueue. */ +ASSERT (in_task ()); +btrfs_bio_counter_dec (bioc->fs_info); bbio->mirror_num = bioc->mirror_num;if (bio_op (bio) ==REQ_OP_READ &&is_data_bbio (bbio))
@@ -365,11 +370,12 @@ static void btrfs_raid56_end_io(struct bio *bio)btrfs_put_bioc (bioc); } -static void btrfs_orig_write_end_io (struct bio *bio) +static void orig_write_end_io_work (struct work_struct *work) { + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, end_io_work); +struct bio *bio = &bbio->bio;struct btrfs_io_stripe *stripe = bio->bi_private;struct btrfs_io_context *bioc = stripe->bioc; -struct btrfs_bio *bbio =btrfs_bio (bio);btrfs_bio_counter_dec (bioc->fs_info);
@@ -394,8 +400,18 @@ static void btrfs_orig_write_end_io(struct bio *bio)btrfs_put_bioc (bioc); } -static void btrfs_clone_write_end_io (struct bio *bio) +static void btrfs_orig_write_end_io (struct bio *bio) { +struct btrfs_bio *bbio =btrfs_bio (bio); + +INIT_WORK (&bbio->end_io_work, orig_write_end_io_work); +queue_work (btrfs_end_io_wq (bbio->inode->root->fs_info, bio), &bbio->end_io_work); +} + +static void clone_write_end_io_work (struct work_struct *work) +{ + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, end_io_work); +struct bio *bio = &bbio->bio;struct btrfs_io_stripe *stripe = bio->bi_private;if (bio->bi_status) {
@@ -410,6 +426,14 @@ static void btrfs_clone_write_end_io(struct bio *bio)bio_put (bio); } +static void btrfs_clone_write_end_io (struct bio *bio) +{ +struct btrfs_bio *bbio =btrfs_bio (bio); + +INIT_WORK (&bbio->end_io_work, clone_write_end_io_work); +queue_work (btrfs_end_io_wq (bbio->inode->root->fs_info, bio), &bbio->end_io_work); +} +static void btrfs_submit_dev_bio (struct btrfs_device *dev,struct bio *bio) { if (!dev || !dev->bdev ||
@@ -456,6 +480,7 @@ static void btrfs_submit_dev_bio(struct btrfs_device *dev, struct bio *bio)static void btrfs_submit_mirrored_bio (struct btrfs_io_context *bioc,int dev_nr) {struct bio *orig_bio = bioc->orig_bio, *bio; +struct btrfs_bio *orig_bbio =btrfs_bio (orig_bio);ASSERT (bio_op (orig_bio) !=REQ_OP_READ );
@@ -464,8 +489,11 @@ static void btrfs_submit_mirrored_bio(struct btrfs_io_context *bioc, int dev_nr) bio = orig_bio; bio->bi_end_io = btrfs_orig_write_end_io; }else { - bio =bio_alloc_clone (NULL , orig_bio,GFP_NOFS , &fs_bio_set); +/* We need to use endio_work to run end_io in task context. */ + bio =bio_alloc_clone (NULL , orig_bio,GFP_NOFS , &btrfs_bioset);bio_inc_remaining (orig_bio); +btrfs_bio_init (btrfs_bio (bio), orig_bbio->inode, + orig_bbio->file_offset,NULL ,NULL ); bio->bi_end_io = btrfs_clone_write_end_io; }
Signature · 1 line
2.51.0
PATCH v2 5/7 btrfs: remove btrfs_fs_info::compressed_write_workers
In reply to Qu WenruoThe reason why end_bbio_compressed_write() queues a work into compressed_write_workers wq is for end_compressed_writeback() call, as it will grab all the involved folioes and clear the writeback flags, which may sleep.
However now we always run btrfs_bio::end_io() in task context, there is no need to queue the work anymore.
Just remove btrfs_fs_info::compressed_write_workers and compressed_bio::write_end_work.
There is a comment about the works queued into compressed_write_workers, now change to flush endio wq instead, which is responsible to handle all data endio functions.
--- fs/btrfs/compression.c | 27 ++++++++------------------- fs/btrfs/compression.h | 7 ++----- fs/btrfs/disk-io.c | 9 ++------- fs/btrfs/fs.h | 1 - 4 files changed, 12 insertions(+), 32 deletions(-)
fs/btrfs/compression.c +8 -19
@@ -319,22 +319,6 @@ static noinline void end_compressed_writeback(const struct compressed_bio *cb)/* the inode may be gone now */ } -static void btrfs_finish_compressed_write_work (struct work_struct *work) -{ -struct compressed_bio *cb = - container_of(work, struct compressed_bio, write_end_work); - -btrfs_finish_ordered_extent (cb->bbio.ordered,NULL , cb->start, cb->len, - cb->bbio.bio.bi_status ==BLK_STS_OK ); - -if (cb->writeback) -end_compressed_writeback (cb); -/* Note, our inode could be gone now */ - -btrfs_free_compressed_folios (cb); -bio_put (&cb->bbio.bio); -} -/* * Do the cleanup once all the compressed pages hit the disk. This will clear * writeback on the file pages and free the compressed pages.
@@ -345,9 +329,15 @@ static void btrfs_finish_compressed_write_work(struct work_struct *work)static void end_bbio_compressed_write (struct btrfs_bio *bbio) {struct compressed_bio *cb =to_compressed_bio (bbio); -struct btrfs_fs_info *fs_info = bbio->inode->root->fs_info; -queue_work (fs_info->compressed_write_workers, &cb->write_end_work); +btrfs_finish_ordered_extent (cb->bbio.ordered,NULL , cb->start, cb->len, + cb->bbio.bio.bi_status ==BLK_STS_OK ); + +if (cb->writeback) +end_compressed_writeback (cb); +/* Note, our inode could be gone now */ +btrfs_free_compressed_folios (cb); +bio_put (&cb->bbio.bio); }static void btrfs_add_compressed_bio_folios (struct compressed_bio *cb)
@@ -400,7 +390,6 @@ void btrfs_submit_compressed_write(struct btrfs_ordered_extent *ordered, cb->compressed_folios = compressed_folios; cb->compressed_len = ordered->disk_num_bytes; cb->writeback = writeback; -INIT_WORK (&cb->write_end_work, btrfs_finish_compressed_write_work); cb->nr_folios = nr_folios; cb->bbio.bio.bi_iter.bi_sector = ordered->disk_bytenr >>SECTOR_SHIFT ; cb->bbio.ordered = ordered;
fs/btrfs/compression.h +2 -5
@@ -63,11 +63,8 @@ struct compressed_bio {/* Whether this is a write for writeback. */ bool writeback; -union { -/* For reads, this is the bio we are copying the data into */ -struct btrfs_bio *orig_bbio; -struct work_struct write_end_work; - }; +/* For reads, this is the bio we are copying the data into */ +struct btrfs_bio *orig_bbio;/* Must be last. */ struct btrfs_bio bbio;
fs/btrfs/disk-io.c +2 -7
@@ -1774,8 +1774,6 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)destroy_workqueue (fs_info->endio_workers);if (fs_info->rmw_workers)destroy_workqueue (fs_info->rmw_workers); -if (fs_info->compressed_write_workers) -destroy_workqueue (fs_info->compressed_write_workers);btrfs_destroy_workqueue (fs_info->endio_write_workers);btrfs_destroy_workqueue (fs_info->endio_freespace_worker);btrfs_destroy_workqueue (fs_info->delayed_workers);
@@ -1987,8 +1985,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) fs_info->endio_write_workers =btrfs_alloc_workqueue (fs_info,"endio-write" , flags, max_active,2 ); - fs_info->compressed_write_workers = -alloc_workqueue ("btrfs-compressed-write" , flags, max_active); fs_info->endio_freespace_worker =btrfs_alloc_workqueue (fs_info,"freespace-write" , flags, max_active,0 );
@@ -2004,7 +2000,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) if (!(fs_info->workers && fs_info->delalloc_workers && fs_info->flush_workers && fs_info->endio_workers && fs_info->endio_meta_workers && - fs_info->compressed_write_workers && fs_info->endio_write_workers && fs_info->endio_freespace_worker && fs_info->rmw_workers && fs_info->caching_workers && fs_info->fixup_workers &&
@@ -4291,7 +4286,7 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)/* * When finishing a compressed write bio we schedule a work queue item -* to finish an ordered extent - btrfs_finish_compressed_write_work() +* to finish an ordered extent - end_bbio_compressed_write() * calls btrfs_finish_ordered_extent() which in turns does a call to * btrfs_queue_ordered_fn(), and that queues the ordered extent * completion either in the endio_write_workers work queue or in the
@@ -4299,7 +4294,7 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)* below, so before we flush them we must flush this queue for the * workers of compressed writes. */ -flush_workqueue (fs_info->compressed_write_workers); +flush_workqueue (fs_info->endio_workers);/* * After we parked the cleaner kthread, ordered extents may have
fs/btrfs/fs.h +0 -1
@@ -654,7 +654,6 @@ struct btrfs_fs_info {struct workqueue_struct *endio_workers;struct workqueue_struct *endio_meta_workers;struct workqueue_struct *rmw_workers; -struct workqueue_struct *compressed_write_workers;struct btrfs_workqueue *endio_write_workers;struct btrfs_workqueue *endio_freespace_worker;struct btrfs_workqueue *caching_workers;
Signature · 1 line
2.51.0
PATCH v2 6/7 btrfs: relax btrfs_inode::ordered_tree_lock
In reply to Qu WenruoWe used IRQ version of spinlock for ordered_tree_lock, as btrfs_finish_ordered_extent() can be called in end_bbio_data_write() which was in IRQ context.
However since we're moving all the btrfs_bio::end_io() calls into task context, there is no more need to support IRQ context thus we can relax to regular spin_lock()/spin_unlock() for btrfs_inode::ordered_tree_lock.
--- fs/btrfs/extent_io.c | 5 ++-- fs/btrfs/inode.c | 4 +-- fs/btrfs/ordered-data.c | 57 ++++++++++++++++++----------------------- fs/btrfs/tree-log.c | 4 +-- 4 files changed, 31 insertions(+), 39 deletions(-)
fs/btrfs/extent_io.c +2 -3
@@ -1726,7 +1726,6 @@ static noinline_for_stack int extent_writepage_io(struct btrfs_inode *inode,if (cur >= i_size) {struct btrfs_ordered_extent *ordered; -unsigned long flags; ordered =btrfs_lookup_first_ordered_range (inode, cur, folio_end - cur);
@@ -1735,11 +1734,11 @@ static noinline_for_stack int extent_writepage_io(struct btrfs_inode *inode,* there must be an ordered extent. */ ASSERT (ordered !=NULL ); -spin_lock_irqsave (&inode->ordered_tree_lock, flags); +spin_lock (&inode->ordered_tree_lock);set_bit (BTRFS_ORDERED_TRUNCATED , &ordered->flags); ordered->truncated_len =min (ordered->truncated_len, cur - ordered->file_offset); -spin_unlock_irqrestore (&inode->ordered_tree_lock, flags); +spin_unlock (&inode->ordered_tree_lock);btrfs_put_ordered_extent (ordered); btrfs_mark_ordered_io_finished(inode, folio, cur,
fs/btrfs/inode.c +2 -2
@@ -7589,11 +7589,11 @@ static void btrfs_invalidate_folio(struct folio *folio, size_t offset,EXTENT_LOCKED |EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG, &cached_state); -spin_lock_irq (&inode->ordered_tree_lock); +spin_lock (&inode->ordered_tree_lock);set_bit (BTRFS_ORDERED_TRUNCATED , &ordered->flags); ordered->truncated_len =min (ordered->truncated_len, cur - ordered->file_offset); -spin_unlock_irq (&inode->ordered_tree_lock); +spin_unlock (&inode->ordered_tree_lock);/* * If the ordered extent has finished, we're safe to delete all
fs/btrfs/ordered-data.c +25 -32
203 lines not on this page · raw message
fs/btrfs/tree-log.c +2 -2
@@ -5410,12 +5410,12 @@ static int btrfs_log_changed_extents(struct btrfs_trans_handle *trans,set_bit (BTRFS_ORDERED_LOGGED , &ordered->flags);if (!test_bit (BTRFS_ORDERED_COMPLETE , &ordered->flags)) { -spin_lock_irq (&inode->ordered_tree_lock); +spin_lock (&inode->ordered_tree_lock);if (!test_bit (BTRFS_ORDERED_COMPLETE , &ordered->flags)) {set_bit (BTRFS_ORDERED_PENDING , &ordered->flags);atomic_inc (&trans->transaction->pending_ordered); } -spin_unlock_irq (&inode->ordered_tree_lock); +spin_unlock (&inode->ordered_tree_lock); }btrfs_put_ordered_extent (ordered); }
Signature · 1 line
2.51.0
PATCH v2 7/7 btrfs: introduce btrfs_bio::async_csum
In reply to Qu Wenruo[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.
But after commit 968f19c5b1b7 ("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.
This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling btrfs_bio::end_io() Or we can lead to use-after-free for the csum generation worker.
- Save the current bi_iter for csum_one_bio()
As the submission part can advance btrfs_bio::bio.bi_iter, if not saved csum_one_bio() may got an empty bi_iter and do not generate any checksum.
Unfortunately this means we have to increase the size of btrfs_bio for 16 bytes.
As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:
Memory bandwidth: 50GiB/s
CRC32C bandwidth: 45GiB/s
SSD bandwidth: 8GiB/s
Then btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/s
After the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/s
The difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.
The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s
Patched Datasum: 1792.26 GiB/s
Diff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)
fs/btrfs/bio.c +16 -5
@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;
@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif }/*
@@ -617,10 +624,14 @@ static bool should_async_write(struct btrfs_bio *bbio)struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode); -if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF ) -return false; - - auto_csum_mode = (csum_mode ==BTRFS_OFFLOAD_CSUM_AUTO ); +if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_ON ) +return true; +/* +* Write bios will calculate checksum and submit bio at the same time. +* Unless explicitly required don't offload serial csum calculate and bio +* submit into a workqueue. +*/ +return false;#endif /* Submit synchronously if the checksum implementation is fast. */
fs/btrfs/bio.h +7 -0
@@ -56,6 +56,9 @@ struct btrfs_bio {struct {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums; +struct work_struct csum_work; +struct completion csum_done; +struct bvec_iter csum_saved_iter;u64 orig_physical; };
@@ -83,6 +86,10 @@ struct btrfs_bio {* scrub bios. */ bool is_scrub; + +/* Whether the csum generation for data write is async. */ +bool async_csum; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last.
fs/btrfs/file-item.c +45 -19
93 lines not on this page · raw message
fs/btrfs/file-item.h +1 -1
@@ -64,7 +64,7 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,int btrfs_csum_file_blocks (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct btrfs_ordered_sum *sums); -int btrfs_csum_one_bio (struct btrfs_bio *bbio); +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async);int btrfs_alloc_dummy_sum (struct btrfs_bio *bbio); int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end, struct list_head *list, int search_commit,
Signature · 1 line
2.51.0
Quoted 71 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 5 lines
fs/btrfs/bio.c +2
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);
Can we do flush_work(&bbio->csum_work); instead here and get rid of the completion? I believe it is not needed at all.
--nX
Quoted 178 lines
fs/btrfs/bio.c +14
-5+ bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif }/* @@ -617,10 +624,14 @@ static bool should_async_write(struct btrfs_bio *bbio)struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode); -if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF ) -return false; - - auto_csum_mode = (csum_mode ==BTRFS_OFFLOAD_CSUM_AUTO ); +if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_ON ) +return true; +/* +* Write bios will calculate checksum and submit bio at the same time. +* Unless explicitly required don't offload serial csum calculate and bio +* submit into a workqueue. +*/ +return false;#endif /* Submit synchronously if the checksum implementation is fast. */ fs/btrfs/bio.h +7
-0@@ -56,6 +56,9 @@ struct btrfs_bio {struct {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums; +struct work_struct csum_work; +struct completion csum_done; +struct bvec_iter csum_saved_iter;u64 orig_physical; };@@ -83,6 +86,10 @@ struct btrfs_bio {* scrub bios. */ bool is_scrub; + +/* Whether the csum generation for data write is async. */ +bool async_csum; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last. fs/btrfs/file-item.c +45
-19@@ -18,6 +18,7 @@#include "fs.h" #include "accessors.h" #include "file-item.h" +#include "volumes.h" #define __MAX_CSUM_ITEMS (r, size) ((unsigned long)(((BTRFS_LEAF_DATA_SIZE(r) - \ sizeof(struct btrfs_item) * 2) / \@@ -764,21 +765,46 @@ int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path,return ret; } -/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);struct bio *bio = &bbio->bio; -struct btrfs_ordered_sum *sums; -struct bvec_iter iter = bio->bi_iter; +struct btrfs_ordered_sum *sums = bbio->sums; +struct bvec_iter iter = *src;phys_addr_t paddr;const u32 blocksize = fs_info->sectorsize; -int index; +int index =0 ; + + shash->tfm = fs_info->csum_shash; + +btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { +btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); + index += fs_info->csum_size; + } +} + +static void csum_one_bio_work (struct work_struct *work) +{ + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, csum_work); + +ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE ); +ASSERT (bbio->async_csum == true); +csum_one_bio (bbio, &bbio->csum_saved_iter); +complete (&bbio->csum_done); +} + +/* +* Calculate checksums of the data contained inside a bio. +*/ +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async) +{ +struct btrfs_ordered_extent *ordered = bbio->ordered; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct bio *bio = &bbio->bio; +struct btrfs_ordered_sum *sums;unsigned nofs_flag; nofs_flag =memalloc_nofs_save ();@@ -789,21 +815,21 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio)if (!sums)return -ENOMEM ; + sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; sums->len = bio->bi_iter.bi_size;INIT_LIST_HEAD (&sums->list); - - sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; - index =0 ; - - shash->tfm = fs_info->csum_shash; - -btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { -btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); - index += fs_info->csum_size; - } - bbio->sums = sums;btrfs_add_ordered_sum (ordered, sums); + +if (!async) { +csum_one_bio (bbio, &bbio->bio.bi_iter); +return 0 ; + } +init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);return 0 ; }fs/btrfs/file-item.h +1
-2@@ -64,7 +64,7 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,int btrfs_csum_file_blocks (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct btrfs_ordered_sum *sums); -int btrfs_csum_one_bio (struct btrfs_bio *bbio); +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async);int btrfs_alloc_dummy_sum (struct btrfs_bio *bbio); int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end, struct list_head *list, int search_commit, --2.51.0
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());
Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.
I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside __flush_work(), triggering the warning from check_flush_dependency().
It looks like the workqueue we're in (btrfs-endio) has a different WQ_MEM_RECLAIM flag for csum_one_bio_work().
I'll keep digging to try to use flush_work() to remove the csum_done, as that will keep the size of btrfs_bio unchanged.
Thanks,
Qu
Quoted 5 lines
--nX
Quoted 1 line
Quoted 3 lines · Qu Wenruo
在 2025/11/11 05:10, Daniel Vacek 写道:
[...]
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) /* Make sure we're already in task context. */ ASSERT(in_task());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+ if (bbio->async_csum) + wait_for_completion(&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.
Quoted 8 lines
I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.
Unfortunately it doesn't look like possible to use flush_work() here.
The point is that btrfs_bio_end_io() can be called inside a workqueue (endio_workers or rmw_workers).
Those involved workqueues have WQ_MEM_RECLAIM flags set, this means they can not flush works without breaking the forward-process guarantee. (check_flush_dependency()).
It looks like flush_work() has more complex mechanism than just waiting, not suitable for our pure waiting usage.
Thanks,
Qu
Quoted 11 lines
Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 82 lines · Qu Wenruo
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.
Quoted 7 lines
I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().
If I read the code correctly that would be solved using the btrfs_end_io_wq instead of system wq (ie. queue_work() instead of schedule_work()).
fs/btrfs/file-item.c int btrfs_csum_one_bio(struct btrfs_bio *bbio) · line 832
Quoted 5 lines
fs/btrfs/file-item.c +5
-0+init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);
Quoted 12 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 88 lines · Daniel Vacek
Quoted 87 lines · Qu Wenruo
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().
Quoted 6 lines
Quoted 2 lines
It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().If I read the code correctly that would be solved using the
btrfs_end_io_wqinstead of system wq (ie.queue_work()instead ofschedule_work()).
That will cause dependency problems. The endio work now depends on the csum work, which are both executed on the same workqueue. If the max_active is 1, endio work will deadlock waiting for the csum one.
Quoted 8 lines
fs/btrfs/file-item.c int btrfs_csum_one_bio(struct btrfs_bio *bbio) · line 832
Quoted 6 lines
fs/btrfs/file-item.c +5
-0+init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);
Nope, I even created a new workqueue for the csum work, and it doesn't workaround the workqueue dependency check.
The check inside check_flush_dependency() is:
WARN_ONCE(worker && ((worker->current_pwq->wq->flags &
(WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM),
It means the worker can not have WQ_MEM_RECLAIM flag at all. (unless it also has the LEGACY flag)
So nope, the flush_work() idea won't work inside any current btrfs workqueue, which all have WQ_MEM_RECLAIM flag set.
What we need is to make endio_workers and rmw_workers to get rid of WQ_MEM_RECLAIM, but that is a huge change and may not even work.
Thanks,
Qu
Quoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 92 lines · Qu Wenruo
Quoted 91 lines · Daniel Vacek
Quoted 89 lines · Qu Wenruo
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().
Quoted 7 lines
Quoted 3 lines
If I read the code correctly that would be solved using the
btrfs_end_io_wqinstead of system wq (ie.queue_work()instead ofschedule_work()).That will cause dependency problems. The endio work now depends on the csum work, which are both executed on the same workqueue.
If themax_activeis 1, endio work will deadlock waiting for the csum one.
When the csum work is being queued the bio was not even submitted yet. The chances are the csum work will be done even before the bio ends and the end io work is queued. But even if csum work is not done yet (or even started due to scheduling delays or previous (unrelated) worker still being blocked), it's always serialized before the end io work. So IIUC, there should be no deadlock possible. Unless I'm still missing something, workqueues could be tricky.
Quoted 10 lines
Quoted 7 lines
fs/btrfs/file-item.c int btrfs_csum_one_bio(struct btrfs_bio *bbio) · line 832
Quoted 5 lines
fs/btrfs/file-item.c +5
-0+init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);Nope, I even created a new workqueue for the csum work, and it doesn't workaround the workqueue dependency check.
Did you create it with the WQ_MEM_RECLAIM flag? As like:
alloc_workqueue("btrfs-async-csum", ... | WQ_MEM_RECLAIM, ...);
I don't see how that would trigger the warning. See below for a detailed explanation.
Quoted 8 lines
The check inside
check_flush_dependency()is:WARN_ONCE(worker && ((worker->current_pwq->wq->flags & (WQ_MEM_RECLAIM | __WQ_LEGACY)) ==
WQ_MEM_RECLAIM),It means the worker can not have
WQ_MEM_RECLAIMflag at all. (unless it also has the LEGACY flag)
I understand that if the work is queued on a wq with WQ_MEM_RECLAIM, the check_flush_dependency() returns early. Hence no need to worry about the warning's condition as it's no longer of any concern.
Quoted 2 lines
So nope, the
flush_work()idea won't work inside any current btrfs workqueue, which all haveWQ_MEM_RECLAIMflag set.
With the above being said, I still see two possible solutions. Either using the btrfs_end_io_wq() as suggested before. It should be safe, IMO. Or, if you're still worried about possible deadlocks, creating a new dedicated wq which also has the WQ_MEM_RECLAIM set (which is needed for compatibility with the context where we want to call the flush_work()).
Both ways could help getting rid of the completion in btrfs_bio, which is 32 bytes by itself.
What do I miss?
Out of curiosity, flush_work() internally also uses completion in pretty much exactly the same way as in this patch, but it's on the caller's stack (in this case on the stack which would call the btrfs_bio_end_io() modified with flush_work()). So in the end the effect would be like moving the completion from btrfs_bio to a stack.
Quoted 20 lines
What we need is to make
endio_workersandrmw_workersto get rid ofWQ_MEM_RECLAIM, but that is a huge change and may not even work.Thanks,
QuQuoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 115 lines · Daniel Vacek
Quoted 92 lines · Qu Wenruo
Quoted 91 lines · Daniel Vacek
Quoted 89 lines · Qu Wenruo
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().Quoted 7 lines
Quoted 3 lines
If I read the code correctly that would be solved using the
btrfs_end_io_wqinstead of system wq (ie.queue_work()instead ofschedule_work()).That will cause dependency problems. The endio work now depends on the csum work, which are both executed on the same workqueue.
If themax_activeis 1, endio work will deadlock waiting for the csum one.When the csum work is being queued the bio was not even submitted yet. The chances are the csum work will be done even before the bio ends and the end io work is queued. But even if csum work is not done yet (or even started due to scheduling delays or previous (unrelated) worker still being blocked), it's always serialized before the end io work. So IIUC, there should be no deadlock possible. Unless I'm still missing something, workqueues could be tricky.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
fs/btrfs/file-item.c +5
-0+init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);Nope, I even created a new workqueue for the csum work, and it doesn't workaround the workqueue dependency check.
Did you create it with the
WQ_MEM_RECLAIMflag? As like:alloc_workqueue("btrfs-async-csum", ... |
WQ_MEM_RECLAIM, ...);
Yes.
Quoted 3 lines
I don't see how that would trigger the warning. See below for a detailed explanation.
It's not the workqueue running the csum work, but the workqueue running the flush_work().
Quoted 7 lines
Quoted 7 lines
The check inside
check_flush_dependency()is:WARN_ONCE(worker && ((worker->current_pwq->wq->flags & (WQ_MEM_RECLAIM | __WQ_LEGACY)) ==
WQ_MEM_RECLAIM),
Quoted 6 lines
Quoted 2 lines
It means the worker can not have
WQ_MEM_RECLAIMflag at all. (unless it also has the LEGACY flag)I understand that if the work is queued on a wq with
WQ_MEM_RECLAIM, thecheck_flush_dependency()returns early. Hence no need to worry about the warning's condition as it's no longer of any concern.
You can just try to use flush_work() and test it by yourself.
I have tried my best to explain it, but it looks like it's at my limit.
Just try a patch removing the csum_wait, and use flush_work() instead of wait_for_completion().
Then you'll see the problem I'm hitting.
Thanks,
Qu
Quoted 42 lines
Quoted 3 lines
So nope, the
flush_work()idea won't work inside any current btrfs workqueue, which all haveWQ_MEM_RECLAIMflag set.With the above being said, I still see two possible solutions. Either using the
btrfs_end_io_wq()as suggested before. It should be safe, IMO. Or, if you're still worried about possible deadlocks, creating a new dedicated wq which also has theWQ_MEM_RECLAIMset (which is needed for compatibility with the context where we want to call theflush_work()).Both ways could help getting rid of the completion in
btrfs_bio, which is 32 bytes by itself.What do I miss?
Out of curiosity,
flush_work()internally also uses completion in pretty much exactly the same way as in this patch, but it's on the caller's stack (in this case on the stack which would call thebtrfs_bio_end_io()modified withflush_work()). So in the end the effect would be like moving the completion frombtrfs_bioto a stack.Quoted 20 lines
What we need is to make
endio_workersandrmw_workersto get rid ofWQ_MEM_RECLAIM, but that is a huge change and may not even work.Thanks,
QuQuoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 125 lines · Qu Wenruo
Quoted 115 lines · Daniel Vacek
Quoted 92 lines · Qu Wenruo
Quoted 91 lines · Daniel Vacek
Quoted 89 lines · Qu Wenruo
Quoted 75 lines · Daniel Vacek
Quoted 74 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +0
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ());Quoted 5 lines
fs/btrfs/bio.c void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status) · line 109
Quoted 2 lines
fs/btrfs/bio.c +2
-0+if (bbio->async_csum) +wait_for_completion (&bbio->csum_done);Can we do
flush_work(&bbio->csum_work);instead here and get rid of the completion? I believe it is not needed at all.I tried this idea, unfortunately causing kernel warnings.
It will trigger a warning inside
__flush_work(), triggering the warning fromcheck_flush_dependency().It looks like the workqueue we're in (btrfs-endio) has a different
WQ_MEM_RECLAIMflag forcsum_one_bio_work().Quoted 7 lines
Quoted 3 lines
If I read the code correctly that would be solved using the
btrfs_end_io_wqinstead of system wq (ie.queue_work()instead ofschedule_work()).That will cause dependency problems. The endio work now depends on the csum work, which are both executed on the same workqueue.
If themax_activeis 1, endio work will deadlock waiting for the csum one.When the csum work is being queued the bio was not even submitted yet. The chances are the csum work will be done even before the bio ends and the end io work is queued. But even if csum work is not done yet (or even started due to scheduling delays or previous (unrelated) worker still being blocked), it's always serialized before the end io work. So IIUC, there should be no deadlock possible. Unless I'm still missing something, workqueues could be tricky.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
(quoted diff) +5
-0+ init_completion(&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; + INIT_WORK(&bbio->csum_work, csum_one_bio_work); + schedule_work(&bbio->csum_work);Quoted 8 lines
Quoted 4 lines
Quoted 1 line
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);Nope, I even created a new workqueue for the csum work, and it doesn't workaround the workqueue dependency check.
Did you create it with the
WQ_MEM_RECLAIMflag? As like:alloc_workqueue("btrfs-async-csum", ... |
WQ_MEM_RECLAIM, ...);
Quoted 8 lines
Yes.
Quoted 3 lines
I don't see how that would trigger the warning. See below for a detailed explanation.
It's not the workqueue running the csum work, but the workqueue running the
flush_work().
The wq running the flush_work() is in the warning condition. *But* the early return checks the csum wq.
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
The check inside
check_flush_dependency()is:WARN_ONCE(worker && ((worker->current_pwq->wq->flags & (WQ_MEM_RECLAIM | __WQ_LEGACY)) ==
WQ_MEM_RECLAIM),Quoted 6 lines
Quoted 2 lines
It means the worker can not have
WQ_MEM_RECLAIMflag at all. (unless it also has the LEGACY flag)I understand that if the work is queued on a wq with
WQ_MEM_RECLAIM, thecheck_flush_dependency()returns early. Hence no need to worry about the warning's condition as it's no longer of any concern.
Quoted 8 lines
You can just try to use
flush_work()and test it by yourself.I have tried my best to explain it, but it looks like it's at my limit.
Just try a patch removing the
csum_wait, and useflush_work()instead ofwait_for_completion().Then you'll see the problem I'm hitting.
No luck. With the change below all fstests pass without any warning or deadlocks whatsoever. Just as I expected.
I don't think there would be any impact on performance, but that needs to be tested and confirmed.
fs/btrfs/bio.c +0 -0
@@ -108,7 +108,8 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio,
blk_status_t status)
ASSERT(in_task());
if (bbio->async_csum)- wait_for_completion(&bbio->csum_done);
+ flush_work(&bbio->csum_work);
bbio->bio.bi_status = status;
if (bbio->bio.bi_pool == &btrfs_clone_bioset) {fs/btrfs/bio.h +0 -1
@@ -58,7 +58,7 @@ struct btrfs_bio {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums;struct work_struct csum_work; -struct completion csum_done;struct bvec_iter csum_saved_iter;u64 orig_physical;
fs/btrfs/file-item.c +9 -3
@@ -790,7 +790,15 @@ static void csum_one_bio_work(struct work_struct *work)ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE );ASSERT (bbio->async_csum == true);csum_one_bio (bbio, &bbio->csum_saved_iter); -complete (&bbio->csum_done); +} + +static struct workqueue_struct *btrfs_end_io_wq (struct btrfs_fs_info *fs_info, +struct bio *bio) +{ +if (bio->bi_opf &REQ_META ) +return fs_info->endio_meta_workers; +return fs_info->endio_workers; }/*
@@ -823,12 +831,13 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio, bool async)csum_one_bio (bbio, &bbio->bio.bi_iter);return 0 ; } -init_completion (&bbio->csum_done); bbio->async_csum = true; bbio->csum_saved_iter = bbio->bio.bi_iter;INIT_WORK (&bbio->csum_work, csum_one_bio_work); -schedule_work (&bbio->csum_work); +queue_work (btrfs_end_io_wq (fs_info, bio), &bbio->csum_work);return 0 ; }
Quoted 46 lines
Thanks,
QuQuoted 42 lines
Quoted 3 lines
So nope, the
flush_work()idea won't work inside any current btrfs workqueue, which all haveWQ_MEM_RECLAIMflag set.With the above being said, I still see two possible solutions. Either using the
btrfs_end_io_wq()as suggested before. It should be safe, IMO. Or, if you're still worried about possible deadlocks, creating a new dedicated wq which also has theWQ_MEM_RECLAIMset (which is needed for compatibility with the context where we want to call theflush_work()).Both ways could help getting rid of the completion in
btrfs_bio, which is 32 bytes by itself.What do I miss?
Out of curiosity,
flush_work()internally also uses completion in pretty much exactly the same way as in this patch, but it's on the caller's stack (in this case on the stack which would call thebtrfs_bio_end_io()modified withflush_work()). So in the end the effect would be like moving the completion frombtrfs_bioto a stack.Quoted 20 lines
What we need is to make
endio_workersandrmw_workersto get rid ofWQ_MEM_RECLAIM, but that is a huge change and may not even work.Thanks,
QuQuoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
在 2025/11/13 08:13, Daniel Vacek 写道:
[...]
Quoted 82 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);Nope, I even created a new workqueue for the csum work, and it doesn't workaround the workqueue dependency check.
Did you create it with the
WQ_MEM_RECLAIMflag? As like:alloc_workqueue("btrfs-async-csum", ... |
WQ_MEM_RECLAIM, ...);Quoted 8 lines
Yes.
Quoted 3 lines
I don't see how that would trigger the warning. See below for a detailed explanation.
It's not the workqueue running the csum work, but the workqueue running the
flush_work().The wq running the
flush_work()is in the warning condition. *But* the early return checks the csum wq.Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
The check inside
check_flush_dependency()is:WARN_ONCE(worker && ((worker->current_pwq->wq->flags & (WQ_MEM_RECLAIM | __WQ_LEGACY)) ==
WQ_MEM_RECLAIM),Quoted 6 lines
Quoted 2 lines
It means the worker can not have
WQ_MEM_RECLAIMflag at all. (unless it also has the LEGACY flag)I understand that if the work is queued on a wq with
WQ_MEM_RECLAIM, thecheck_flush_dependency()returns early. Hence no need to worry about the warning's condition as it's no longer of any concern.Quoted 8 lines
You can just try to use
flush_work()and test it by yourself.I have tried my best to explain it, but it looks like it's at my limit.
Just try a patch removing the
csum_wait, and useflush_work()instead ofwait_for_completion().Then you'll see the problem I'm hitting.
No luck. With the change below all fstests pass without any warning or deadlocks whatsoever. Just as I expected.
I don't think there would be any impact on performance, but that needs to be tested and confirmed.fs/btrfs/bio.c +1
-1@@ -108,7 +108,8 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)ASSERT (in_task ());if (bbio->async_csum) -wait_for_completion (&bbio->csum_done); +flush_work (&bbio->csum_work); bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {fs/btrfs/bio.h +0
-1@@ -58,7 +58,7 @@ struct btrfs_bio {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums;struct work_struct csum_work; -struct completion csum_done;struct bvec_iter csum_saved_iter;u64 orig_physical;fs/btrfs/file-item.c +0
-0@@ -790,7 +790,15 @@ static void csum_one_bio_work(struct work_struct *work)ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE );ASSERT (bbio->async_csum == true);
fs/btrfs/file-item.c static void csum_one_bio_work(struct work_struct *work) · line 800
Quoted 10 lines
fs/btrfs/file-item.c +8
-1csum_one_bio (bbio, &bbio->csum_saved_iter); -complete (&bbio->csum_done); +} + +static struct workqueue_struct *btrfs_end_io_wq (struct btrfs_fs_info *fs_info, +struct bio *bio) +{ +if (bio->bi_opf &REQ_META ) +return fs_info->endio_meta_workers; +return fs_info->endio_workers;
For btrfs_csum_one_bio() it's definitely data IO, thus you can go with endio_workers directly.
fs/btrfs/file-item.c int btrfs_csum_one_bio(struct btrfs_bio *bbio, bool async) · line 837
Quoted 13 lines
fs/btrfs/file-item.c +1
-2}/* @@ -823,12 +831,13 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio, bool async)csum_one_bio (bbio, &bbio->bio.bi_iter);return 0 ; } -init_completion (&bbio->csum_done); bbio->async_csum = true; bbio->csum_saved_iter = bbio->bio.bi_iter;INIT_WORK (&bbio->csum_work, csum_one_bio_work); -schedule_work (&bbio->csum_work); +queue_work (btrfs_end_io_wq (fs_info, bio), &bbio->csum_work);
This is the what causing the difference, that you're calling flush_work() inside the same workqueue, at least for most cases.
However I'm still not 100% sure if it's fine, as for RAID56 the endio function is called inside rmw_workers, not the same as the regular endio_workers, causing the same cross-wq flush_work() call, which can still lead to the warning.
Personally speaking I'd prefer not to bother the cross-wq situations for flush_work() for now, which is a completely new rabbit hole.
Feel free to send out a dedicated patch removing btrfs_bio::csum_done, and we can continue the discussion there.
Thanks,
Qu
Quoted 50 lines
fs/btrfs/file-item.c +0
-0return 0 ; } > Thanks,Quoted 45 lines
Qu
Quoted 42 lines
Quoted 3 lines
So nope, the
flush_work()idea won't work inside any current btrfs workqueue, which all haveWQ_MEM_RECLAIMflag set.With the above being said, I still see two possible solutions. Either using the
btrfs_end_io_wq()as suggested before. It should be safe, IMO. Or, if you're still worried about possible deadlocks, creating a new dedicated wq which also has theWQ_MEM_RECLAIMset (which is needed for compatibility with the context where we want to call theflush_work()).Both ways could help getting rid of the completion in
btrfs_bio, which is 32 bytes by itself.What do I miss?
Out of curiosity,
flush_work()internally also uses completion in pretty much exactly the same way as in this patch, but it's on the caller's stack (in this case on the stack which would call thebtrfs_bio_end_io()modified withflush_work()). So in the end the effect would be like moving the completion frombtrfs_bioto a stack.Quoted 20 lines
What we need is to make
endio_workersandrmw_workersto get rid ofWQ_MEM_RECLAIM, but that is a huge change and may not even work.Thanks,
QuQuoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 11 lines · Qu Wenruo
Quoted 6 lines · Daniel Vacek
-
init_completion(&bbio->csum_done);bbio->async_csum= true;
bbio->csum_saved_iter=bbio->bio.bi_iter;INIT_WORK(&bbio->csum_work, csum_one_bio_work); -schedule_work(&bbio->csum_work);
+queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->csum_work);
Quoted 3 lines
This is the what causing the difference, that you're calling
flush_work()inside the same workqueue, at least for most cases.However I'm still not 100% sure if it's fine, as for RAID56 the endio function is called inside
rmw_workers, not the same as the regularendio_workers, causing the same cross-wqflush_work()call, which can still lead to the warning.
Well, for RAID56 the rmw_workers are still having WQ_MEM_RECLAIM flag, thus it will still be fine.
It's really the system wide workqueues not having that flags which causes the warning.
Quoted 4 lines
Personally speaking I'd prefer not to bother the cross-wq situations for
flush_work()for now, which is a completely new rabbit hole.Feel free to send out a dedicated patch removing
btrfs_bio::csum_done, and we can continue the discussion there.
Still, you find the fix and you deserve the patch.
Thanks,
Qu
Quoted 55 lines
Thanks,
QuQuoted 50 lines
return 0;
}Quoted 46 lines
Thanks,
QuQuoted 42 lines
Quoted 3 lines
So nope, the
flush_work()idea won't work inside any current btrfs workqueue, which all haveWQ_MEM_RECLAIMflag set.With the above being said, I still see two possible solutions. Either using the
btrfs_end_io_wq()as suggested before. It should be safe, IMO. Or, if you're still worried about possible deadlocks, creating a new dedicated wq which also has theWQ_MEM_RECLAIMset (which is needed for compatibility with the context where we want to call theflush_work()).Both ways could help getting rid of the completion in
btrfs_bio, which is 32 bytes by itself.What do I miss?
Out of curiosity,
flush_work()internally also uses completion in pretty much exactly the same way as in this patch, but it's on the caller's stack (in this case on the stack which would call thebtrfs_bio_end_io()modified withflush_work()). So in the end the effect would be like moving the completion frombtrfs_bioto a stack.Quoted 20 lines
What we need is to make
endio_workersandrmw_workersto get rid ofWQ_MEM_RECLAIM, but that is a huge change and may not even work.Thanks,
QuQuoted 13 lines
Quoted 13 lines
I'll keep digging to try to use
flush_work()to remove thecsum_done, as that will keep the size ofbtrfs_biounchanged.Thanks,
QuQuoted 5 lines
--nX
Quoted 1 line
Quoted 152 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +16
-5@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif }/* @@ -617,10 +624,14 @@ static bool should_async_write(struct btrfs_bio *bbio)struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode); -if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF ) -return false; - - auto_csum_mode = (csum_mode ==BTRFS_OFFLOAD_CSUM_AUTO ); +if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_ON ) +return true; +/* +* Write bios will calculate checksum and submit bio at the same time. +* Unless explicitly required don't offload serial csum calculate and bio +* submit into a workqueue. +*/ +return false;#endif /* Submit synchronously if the checksum implementation is fast. */ fs/btrfs/bio.h +7
-0@@ -56,6 +56,9 @@ struct btrfs_bio {struct {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums; +struct work_struct csum_work; +struct completion csum_done; +struct bvec_iter csum_saved_iter;u64 orig_physical; };@@ -83,6 +86,10 @@ struct btrfs_bio {* scrub bios. */ bool is_scrub; + +/* Whether the csum generation for data write is async. */ +bool async_csum; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last. fs/btrfs/file-item.c +1
-0@@ -18,6 +18,7 @@#include "fs.h" #include "accessors.h" #include "file-item.h" +#include "volumes.h" #define __MAX_CSUM_ITEMS (r, size) ((unsigned long)(((BTRFS_LEAF_DATA_SIZE(r) - \ sizeof(struct btrfs_item) * 2) / \@@ -764,21 +765,46 @@ int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path,return ret; }
fs/btrfs/file-item.c int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path, · line 771
Quoted 10 lines
fs/btrfs/file-item.c +1
-5-/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;
If you use bbio->fs_info, the inode can be also removed.
Quoted 27 lines
fs/btrfs/file-item.c +20
-3SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);struct bio *bio = &bbio->bio; -struct btrfs_ordered_sum *sums; -struct bvec_iter iter = bio->bi_iter; +struct btrfs_ordered_sum *sums = bbio->sums; +struct bvec_iter iter = *src;phys_addr_t paddr;const u32 blocksize = fs_info->sectorsize; -int index; +int index =0 ; + + shash->tfm = fs_info->csum_shash; + +btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { +btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); + index += fs_info->csum_size; + } +} + +static void csum_one_bio_work (struct work_struct *work) +{ + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, csum_work); + +ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE ); +ASSERT (bbio->async_csum == true); +csum_one_bio (bbio, &bbio->csum_saved_iter); +complete (&bbio->csum_done);
fs/btrfs/file-item.c int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path, · line 805
Quoted 10 lines
fs/btrfs/file-item.c +10
-0+} + +/* +* Calculate checksums of the data contained inside a bio. +*/ +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async) +{ +struct btrfs_ordered_extent *ordered = bbio->ordered; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info;
And the same here.
--nX
Quoted 55 lines
fs/btrfs/file-item.c +13
-11+struct bio *bio = &bbio->bio; +struct btrfs_ordered_sum *sums;unsigned nofs_flag; nofs_flag =memalloc_nofs_save ();@@ -789,21 +815,21 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio)if (!sums)return -ENOMEM ; + sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; sums->len = bio->bi_iter.bi_size;INIT_LIST_HEAD (&sums->list); - - sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; - index =0 ; - - shash->tfm = fs_info->csum_shash; - -btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { -btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); - index += fs_info->csum_size; - } - bbio->sums = sums;btrfs_add_ordered_sum (ordered, sums); + +if (!async) { +csum_one_bio (bbio, &bbio->bio.bi_iter); +return 0 ; + } +init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);return 0 ; }fs/btrfs/file-item.h +1
-2@@ -64,7 +64,7 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,int btrfs_csum_file_blocks (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct btrfs_ordered_sum *sums); -int btrfs_csum_one_bio (struct btrfs_bio *bbio); +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async);int btrfs_alloc_dummy_sum (struct btrfs_bio *bbio); int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end, struct list_head *list, int search_commit, --2.51.0
Quoted 1 line · Daniel Vacek
On Wed, 29 Oct 2025 at 06:43, Qu Wenruo <[email protected]> wrote:
[...]
Quoted 11 lines
fs/btrfs/file-item.c int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path, · line 771
Quoted 9 lines
fs/btrfs/file-item.c +1
-5-/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;If you use
bbio->fs_info, the inode can be also removed.
Please check the patch 3/7.
There is no more btrfs_bio::fs_info.
Quoted 15 lines · Qu Wenruo
Quoted 1 line · Daniel Vacek
On Wed, 29 Oct 2025 at 06:43, Qu Wenruo <[email protected]> wrote:
[...]
Quoted 11 lines
fs/btrfs/file-item.c int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path, · line 771
Quoted 9 lines
fs/btrfs/file-item.c +1
-5-/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;If you use
bbio->fs_info, the inode can be also removed.
Quoted 3 lines
Please check the patch 3/7.
There is no more
btrfs_bio::fs_info.
Right. I missed that. Not a big deal then. The inode is still not needed here, though.
Sorry about the noise.
Quoted 80 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +3
-0@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;
fs/btrfs/bio.c static int btrfs_bio_csum(struct btrfs_bio *bbio) · line 548
Quoted 9 lines
fs/btrfs/bio.c +5
-1@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif
Btw, what are your plans with async csums, Qu? You marked it experimental here, but I don't really see any reason to limit it this way. Shall we just go live?
--nX
Quoted 164 lines
fs/btrfs/bio.c +8
-4}/* @@ -617,10 +624,14 @@ static bool should_async_write(struct btrfs_bio *bbio)struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode); -if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF ) -return false; - - auto_csum_mode = (csum_mode ==BTRFS_OFFLOAD_CSUM_AUTO ); +if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_ON ) +return true; +/* +* Write bios will calculate checksum and submit bio at the same time. +* Unless explicitly required don't offload serial csum calculate and bio +* submit into a workqueue. +*/ +return false;#endif /* Submit synchronously if the checksum implementation is fast. */ fs/btrfs/bio.h +7
-0@@ -56,6 +56,9 @@ struct btrfs_bio {struct {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums; +struct work_struct csum_work; +struct completion csum_done; +struct bvec_iter csum_saved_iter;u64 orig_physical; };@@ -83,6 +86,10 @@ struct btrfs_bio {* scrub bios. */ bool is_scrub; + +/* Whether the csum generation for data write is async. */ +bool async_csum; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last. fs/btrfs/file-item.c +45
-19@@ -18,6 +18,7 @@#include "fs.h" #include "accessors.h" #include "file-item.h" +#include "volumes.h" #define __MAX_CSUM_ITEMS (r, size) ((unsigned long)(((BTRFS_LEAF_DATA_SIZE(r) - \ sizeof(struct btrfs_item) * 2) / \@@ -764,21 +765,46 @@ int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path,return ret; } -/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);struct bio *bio = &bbio->bio; -struct btrfs_ordered_sum *sums; -struct bvec_iter iter = bio->bi_iter; +struct btrfs_ordered_sum *sums = bbio->sums; +struct bvec_iter iter = *src;phys_addr_t paddr;const u32 blocksize = fs_info->sectorsize; -int index; +int index =0 ; + + shash->tfm = fs_info->csum_shash; + +btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { +btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); + index += fs_info->csum_size; + } +} + +static void csum_one_bio_work (struct work_struct *work) +{ + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, csum_work); + +ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE ); +ASSERT (bbio->async_csum == true); +csum_one_bio (bbio, &bbio->csum_saved_iter); +complete (&bbio->csum_done); +} + +/* +* Calculate checksums of the data contained inside a bio. +*/ +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async) +{ +struct btrfs_ordered_extent *ordered = bbio->ordered; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct bio *bio = &bbio->bio; +struct btrfs_ordered_sum *sums;unsigned nofs_flag; nofs_flag =memalloc_nofs_save ();@@ -789,21 +815,21 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio)if (!sums)return -ENOMEM ; + sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; sums->len = bio->bi_iter.bi_size;INIT_LIST_HEAD (&sums->list); - - sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; - index =0 ; - - shash->tfm = fs_info->csum_shash; - -btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { -btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); - index += fs_info->csum_size; - } - bbio->sums = sums;btrfs_add_ordered_sum (ordered, sums); + +if (!async) { +csum_one_bio (bbio, &bbio->bio.bi_iter); +return 0 ; + } +init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);return 0 ; }fs/btrfs/file-item.h +1
-2@@ -64,7 +64,7 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,int btrfs_csum_file_blocks (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct btrfs_ordered_sum *sums); -int btrfs_csum_one_bio (struct btrfs_bio *bbio); +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async);int btrfs_alloc_dummy_sum (struct btrfs_bio *bbio); int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end, struct list_head *list, int search_commit, --2.51.0
Quoted 86 lines · Daniel Vacek
Quoted 85 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +4
-1@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL
Quoted 8 lines
fs/btrfs/bio.c static int btrfs_bio_csum(struct btrfs_bio *bbio) · line 548
Quoted 4 lines
fs/btrfs/bio.c +4
-0+return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif Btw, what are your plans with async csums, Qu? You marked it experimental here, but I don't really see any reason to limit it this way. Shall we just go live?
Oh, thanks for reminding me about this.
Too many pending features make me forget that async csum is still experimental.
I'll send out a patch enabling it soon.
Thanks,
Qu
Quoted 167 lines
--nX
Quoted 164 lines
fs/btrfs/bio.c +8
-4}/* @@ -617,10 +624,14 @@ static bool should_async_write(struct btrfs_bio *bbio)struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;enum btrfs_offload_csum_mode csum_mode =READ_ONCE (fs_devices->offload_csum_mode); -if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_OFF ) -return false; - - auto_csum_mode = (csum_mode ==BTRFS_OFFLOAD_CSUM_AUTO ); +if (csum_mode ==BTRFS_OFFLOAD_CSUM_FORCE_ON ) +return true; +/* +* Write bios will calculate checksum and submit bio at the same time. +* Unless explicitly required don't offload serial csum calculate and bio +* submit into a workqueue. +*/ +return false;#endif /* Submit synchronously if the checksum implementation is fast. */ fs/btrfs/bio.h +7
-0@@ -56,6 +56,9 @@ struct btrfs_bio {struct {struct btrfs_ordered_extent *ordered;struct btrfs_ordered_sum *sums; +struct work_struct csum_work; +struct completion csum_done; +struct bvec_iter csum_saved_iter;u64 orig_physical; };@@ -83,6 +86,10 @@ struct btrfs_bio {* scrub bios. */ bool is_scrub; + +/* Whether the csum generation for data write is async. */ +bool async_csum; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last. fs/btrfs/file-item.c +45
-19@@ -18,6 +18,7 @@#include "fs.h" #include "accessors.h" #include "file-item.h" +#include "volumes.h" #define __MAX_CSUM_ITEMS (r, size) ((unsigned long)(((BTRFS_LEAF_DATA_SIZE(r) - \ sizeof(struct btrfs_item) * 2) / \@@ -764,21 +765,46 @@ int btrfs_lookup_csums_bitmap(struct btrfs_root *root, struct btrfs_path *path,return ret; } -/* -* Calculate checksums of the data contained inside a bio. -*/ -int btrfs_csum_one_bio (struct btrfs_bio *bbio) +static void csum_one_bio (struct btrfs_bio *bbio,struct bvec_iter *src) { -struct btrfs_ordered_extent *ordered = bbio->ordered;struct btrfs_inode *inode = bbio->inode;struct btrfs_fs_info *fs_info = inode->root->fs_info;SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);struct bio *bio = &bbio->bio; -struct btrfs_ordered_sum *sums; -struct bvec_iter iter = bio->bi_iter; +struct btrfs_ordered_sum *sums = bbio->sums; +struct bvec_iter iter = *src;phys_addr_t paddr;const u32 blocksize = fs_info->sectorsize; -int index; +int index =0 ; + + shash->tfm = fs_info->csum_shash; + +btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { +btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); + index += fs_info->csum_size; + } +} + +static void csum_one_bio_work (struct work_struct *work) +{ + struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, csum_work); + +ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE ); +ASSERT (bbio->async_csum == true); +csum_one_bio (bbio, &bbio->csum_saved_iter); +complete (&bbio->csum_done); +} + +/* +* Calculate checksums of the data contained inside a bio. +*/ +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async) +{ +struct btrfs_ordered_extent *ordered = bbio->ordered; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct bio *bio = &bbio->bio; +struct btrfs_ordered_sum *sums;unsigned nofs_flag; nofs_flag =memalloc_nofs_save ();@@ -789,21 +815,21 @@ int btrfs_csum_one_bio(struct btrfs_bio *bbio)if (!sums)return -ENOMEM ; + sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; sums->len = bio->bi_iter.bi_size;INIT_LIST_HEAD (&sums->list); - - sums->logical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; - index =0 ; - - shash->tfm = fs_info->csum_shash; - -btrfs_bio_for_each_block (paddr, bio, &iter, blocksize) { -btrfs_calculate_block_csum (fs_info, paddr, sums->sums + index); - index += fs_info->csum_size; - } - bbio->sums = sums;btrfs_add_ordered_sum (ordered, sums); + +if (!async) { +csum_one_bio (bbio, &bbio->bio.bi_iter); +return 0 ; + } +init_completion (&bbio->csum_done); + bbio->async_csum = true; + bbio->csum_saved_iter = bbio->bio.bi_iter; +INIT_WORK (&bbio->csum_work, csum_one_bio_work); +schedule_work (&bbio->csum_work);return 0 ; }fs/btrfs/file-item.h +1
-2@@ -64,7 +64,7 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,int btrfs_csum_file_blocks (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct btrfs_ordered_sum *sums); -int btrfs_csum_one_bio (struct btrfs_bio *bbio); +int btrfs_csum_one_bio (struct btrfs_bio *bbio,bool async);int btrfs_alloc_dummy_sum (struct btrfs_bio *bbio); int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end, struct list_head *list, int search_commit, --2.51.0
Quoted 96 lines · Qu Wenruo
Quoted 86 lines · Daniel Vacek
Quoted 85 lines · Qu Wenruo
[ENHANCEMENT]
Btrfs currently calculate its data checksum then submit the bio.But after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum"), any writes with data checksum will fallback to buffered IO, meaning the content will not change during writeback.This means we're safe to calculate the data checksum and submit the bio in parallel, and only need the following new behaviors:
- Wait the csum generation to finish before calling
btrfs_bio::end_io()Or we can lead to use-after-free for the csum generation worker.- Save the current
bi_iterforcsum_one_bio()
As the submission part can advancebtrfs_bio::bio.bi_iter, if not savedcsum_one_bio()may got an emptybi_iterand do not generate any checksum.Unfortunately this means we have to increase the size of
btrfs_biofor 16 bytes.As usual, such new feature is hidden behind the experimental flag.
[THEORETIC ANALYZE]
Consider the following theoretic hardware performance, which should be more or less close to modern mainstream hardware:Memory bandwidth: 50GiB/s CRC32C bandwidth: 45GiB/s SSD bandwidth: 8GiB/sThen btrfs write bandwidth with data checksum before the patch would be
1 / ( 1 / 50 + 1 / 45 + 1 / 8) = 5.98 GiB/sAfter the patch, the bandwidth would be:
1 / ( 1 / 50 + max( 1 / 45 + 1 / 8)) = 6.90 GiB/sThe difference would be 15.32 % improvement.
[REAL WORLD BENCHMARK]
I'm using a Zen5 (HX 370) as the host, the VM has 4GiB memory, 10 vCPUs, the storage is backed by a PCIE gen3 x4 NVME SSD.The test is a direct IO write, with 1MiB block size, write 7GiB data into a btrfs mount with data checksum. Thus the direct write will fallback to buffered one:
Vanilla Datasum: 1619.97 GiB/s Patched Datasum: 1792.26 GiB/sDiff +10.6 %
In my case, the bottleneck is the storage, thus the improvement is not reaching the theoretic one, but still some observable improvement.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/bio.c | 21 +++++++++++---- fs/btrfs/bio.h | 7 +++++ fs/btrfs/file-item.c | 64 +++++++++++++++++++++++++++++++------------- fs/btrfs/file-item.h | 2 +- 4 files changed, 69 insertions(+), 25 deletions(-)fs/btrfs/bio.c +4
-1@@ -105,6 +105,9 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status)/* Make sure we're already in task context. */ ASSERT (in_task ()); +if (bbio->async_csum) +wait_for_completion (&bbio->csum_done); + bbio->bio.bi_status = status;if (bbio->bio.bi_pool == &btrfs_clone_bioset) {struct btrfs_bio *orig_bbio = bbio->private;@@ -538,7 +541,11 @@ static int btrfs_bio_csum(struct btrfs_bio *bbio) {if (bbio->bio.bi_opf &REQ_META )return btree_csum_one_bio (bbio); -return btrfs_csum_one_bio (bbio); +#ifdef CONFIG_BTRFS_EXPERIMENTAL Quoted 8 lines
fs/btrfs/bio.c static int btrfs_bio_csum(struct btrfs_bio *bbio) · line 548
Quoted 4 lines
fs/btrfs/bio.c +4
-0+return btrfs_csum_one_bio (bbio, true); +#else +return btrfs_csum_one_bio (bbio, false); +#endif Btw, what are your plans with async csums, Qu? You marked it experimental here, but I don't really see any reason to limit it this way. Shall we just go live?
Quoted 6 lines
Oh, thanks for reminding me about this.
Too many pending features make me forget that async csum is still experimental.
I'll send out a patch enabling it soon.
No worries. I can add it to my series while cleaning up around...
--nX
Quoted 2 lines
Thanks,
Qu
Quoted 38 lines · Qu Wenruo
[CHANGELOG]
v2:
- Fix a race where csum generation can be completely skipped If the lower storage layer advanced thebi_iterbefore we finished calculating the checksum,bio_one_bio()may skip generating csum completely.Unfortunately to solve this, a new
bvec_itermust be introduced and cause the size ofbtrfs_bioto increase.- Remove
btrfs_bio::fs_infoby always requireingbtrfs_bio::inodeFor scrub, use btree inode instead, and for the only function that requires to be called by scrub,btrfs_submit_repair_write(), use a new member,btrfs_bio::is_scrub, to check the callers.- Add a new patch to cleanup the recursive including problems The idea is to keep the btrfs headers inside .h files to minimal, thus lower risk of recursive including.
- Remove
BTRFS_MAX_BIO_SECTORSmacro
It's the same asBIO_MAX_VECS, and only utilized once.The new
async_csumfeature will allow btrfs to calculate checksum for data write bios and submit them in parallel.This will reduce latency and improve write throughput when data checksum is utilized.
This will slightly reclaim the performance drop after commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum").Patch 1 is a minor cleanup to remove a single-used macro. Patch 2 is a preparation to remove
btrfs_bio::fs_info, the core is to avoid recursive including.
Patch 3 is to removebtrfs_bio::fs_info, as soon we will increase the size ofbtrfs_bioby 16 bytes.
Quoted 5 lines
Patch 4~6 are preparations to make sure
btrfs_bio::end_io()is called in task context.Patch 7 enables the async checksum generation for data writes, under experimental flags.
Ok, I think we can leave it under experimental for just one cycle.
Quoted 2 lines
The series will increase
btrfs_biosize by 8 bytes (+16 for the new structurs, -8 for the removal ofbtrfs_bio::fs_info).
The end result is still good, also the fs_info was really redundant.
Quoted 5 lines
Since the new
async_csumshould be better than the csum offload anyway, we may want to deprecate the csum offload feature completely in the future.
Thankfully csum offload feature is still behind the experimental flag, thus it should not affect end users.
Yeah we can remove it eventually. The idea was to provide a way to possibly test different ways how to improve the performance but it's a bandaid, there should be only one. The logic can decide based on various factors but the sysfs tunable should not be needed.
Quoted 9 lines
Qu Wenruo (7):
btrfs: replaceBTRFS_MAX_BIO_SECTORSwithBIO_MAX_VECS
btrfs: headers cleanup to remove unnecessary local includes
btrfs: removebtrfs_bio::fs_infoby extracting it from
btrfs_bio::inode
btrfs: make sure allbtrfs_bio::end_iois called in task context
btrfs: removebtrfs_fs_info::compressed_write_workers
btrfs: relaxbtrfs_inode::ordered_tree_lock
btrfs: introducebtrfs_bio::async_csum
Please add the patches to for-next as you see fit. I'll add them to linux-next for early testing too.