btrfs I/O completion cleanup and single device I/O optimizations
Hi all,
this series cleans up the btrfs_bio API, most prominently by splitting the end_io handler for the highlevel bio from the low-level bio bi_end_io, which are really confusingly coupled in the current code. Once that is done it then optimizes the bio submission to not allocate a btrfs_io_context for I/Os tht just go to a single device.
This series sits on top of the "fix read repair on compressed extents v2" series submitted. To make everyones life easier a git tree is also available:
git://git.infradead.org/users/hch/misc.git btrfs-bio-api-cleanup
Gitweb:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/btrfs-bio-api-cleanup
Diffstat:
compression.c | 50 +++----- disk-io.c | 16 +- extent-io-tree.h | 4 extent_io.c | 117 +++---------------- extent_io.h | 3 inode.c | 57 ++++----- raid56.c | 45 +------ raid56.h | 4 scrub.c | 7 - super.c | 6 - volumes.c | 330 ++++++++++++++++++++++++++++++++++++++----------------- volumes.h | 19 ++- 12 files changed, 343 insertions(+), 315 deletions(-)
PATCH 1/7 btrfs: don't call bioset_integrity_create for btrfs_bioset
btrfs never uses bio integrity data itself, so don't allocate the integrity pools for btrfs_bioset.
--- fs/btrfs/extent_io.c | 6 ------ 1 file changed, 6 deletions(-)
fs/btrfs/extent_io.c +0 -6
@@ -255,14 +255,8 @@ int __init extent_io_init(void) BIOSET_NEED_BVECS)) goto free_buffer_cache; -if (bioset_integrity_create (&btrfs_bioset,BIO_POOL_SIZE )) - goto free_bioset; -return 0 ; -free_bioset: -bioset_exit (&btrfs_bioset); - free_buffer_cache:kmem_cache_destroy (extent_buffer_cache); extent_buffer_cache =NULL ;
Signature · 1 line
2.30.2
Re: [PATCH 1/7] btrfs: don't call bioset_integrity_create for btrfs_bioset
Looks good,
PATCH 2/7 btrfs: move btrfs_bio allocation to volumes.c
In reply to Christoph Hellwigvolumes.c is the place that implements the storage layer using the btrfs_bio structure, so move the bio_set and allocation helpers there as well.
To make up for the new initialization boilerplate, merge the two init/exit helpers in extent_io.c into a single one.
--- fs/btrfs/extent-io-tree.h | 4 +-- fs/btrfs/extent_io.c | 74 ++++----------------------------------- fs/btrfs/extent_io.h | 3 -- fs/btrfs/super.c | 6 ++-- fs/btrfs/volumes.c | 57 ++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 3 ++ 6 files changed, 71 insertions(+), 76 deletions(-)
fs/btrfs/extent-io-tree.h +2 -2
@@ -96,8 +96,8 @@ struct extent_state {#endif }; -int __init extent_state_cache_init (void ); -void __cold extent_state_cache_exit (void ); +int __init volumes_init (void ); +void __cold volumes_exit (void ); void extent_io_tree_init(struct btrfs_fs_info *fs_info, struct extent_io_tree *tree, unsigned int owner,
fs/btrfs/extent_io.c +6 -68
111 lines not on this page · raw message
fs/btrfs/extent_io.h +0 -3
@@ -60,7 +60,6 @@ enum {struct btrfs_bio ;struct btrfs_root ;struct btrfs_inode ; -struct btrfs_io_bio ;struct btrfs_fs_info ;struct io_failure_record ;struct extent_io_tree ;
@@ -242,8 +241,6 @@ void extent_clear_unlock_delalloc(struct btrfs_inode *inode, u64 start, u64 end, u32 bits_to_clear, unsigned long page_ops);int btrfs_alloc_page_array (unsigned int nr_pages,struct page **page_array); -struct bio *btrfs_bio_alloc (unsigned int nr_iovecs); -struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size);void end_extent_writepage (struct page *page,int err,u64 start,u64 end);int btrfs_repair_eb_io_failure (const struct extent_buffer *eb,int mirror_num);
fs/btrfs/super.c +3 -3
@@ -2668,7 +2668,7 @@ static int __init init_btrfs_fs(void)if (err) goto free_cachep; - err =extent_state_cache_init (); + err =volumes_init ();if (err) goto free_extent_io;
@@ -2727,7 +2727,7 @@ static int __init init_btrfs_fs(void) free_extent_map:extent_map_exit (); free_extent_state_cache: -extent_state_cache_exit (); +volumes_exit (); free_extent_io:extent_io_exit (); free_cachep:
@@ -2748,7 +2748,7 @@ static void __exit exit_btrfs_fs(void)btrfs_prelim_ref_exit ();ordered_data_exit ();extent_map_exit (); -extent_state_cache_exit (); +volumes_exit ();extent_io_exit ();btrfs_interface_exit ();unregister_filesystem (&btrfs_fs_type);
fs/btrfs/volumes.c +57 -0
72 lines not on this page · raw message
fs/btrfs/volumes.h +3 -0
@@ -391,6 +391,9 @@ static inline struct btrfs_bio *btrfs_bio(struct bio *bio) return container_of(bio, struct btrfs_bio, bio); } +struct bio *btrfs_bio_alloc (unsigned int nr_vecs); +struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size); +static inline void btrfs_bio_free_csum (struct btrfs_bio *bbio) {if (bbio->csum != bbio->csum_inline) {
Signature · 1 line
2.30.2
Re: [PATCH 2/7] btrfs: move btrfs_bio allocation to volumes.c
Looks good,
PATCH 3/7 btrfs: pass the operation to btrfs_bio_alloc
In reply to Christoph HellwigPass the operation to btrfs_bio_alloc, matching what bio_alloc_bioset set does.
--- fs/btrfs/compression.c | 3 +-- fs/btrfs/extent_io.c | 6 ++---- fs/btrfs/inode.c | 3 +-- fs/btrfs/volumes.c | 4 ++-- fs/btrfs/volumes.h | 2 +- 5 files changed, 7 insertions(+), 11 deletions(-)
fs/btrfs/compression.c +1 -2
@@ -364,10 +364,9 @@ static struct bio *alloc_compressed_bio(struct compressed_bio *cb, u64 disk_bytestruct bio *bio;int ret; - bio =btrfs_bio_alloc (BIO_MAX_VECS ); + bio =btrfs_bio_alloc (BIO_MAX_VECS , opf); bio->bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; - bio->bi_opf = opf; bio->bi_private = cb; bio->bi_end_io = endio_func;
fs/btrfs/extent_io.c +2 -4
@@ -2627,10 +2627,9 @@ int btrfs_repair_one_sector(struct inode *inode, struct btrfs_bio *failed_bbio,return -EIO ; } - repair_bio =btrfs_bio_alloc (1 ); + repair_bio =btrfs_bio_alloc (1 ,REQ_OP_READ ); repair_bbio =btrfs_bio (repair_bio); repair_bbio->file_offset = start; - repair_bio->bi_opf =REQ_OP_READ ; repair_bio->bi_end_io = failed_bio->bi_end_io; repair_bio->bi_iter.bi_sector = failrec->logical >>9 ; repair_bio->bi_private = failed_bio->bi_private;
@@ -3265,7 +3264,7 @@ static int alloc_new_bio(struct btrfs_inode *inode,struct bio *bio;int ret; - bio =btrfs_bio_alloc (BIO_MAX_VECS ); + bio =btrfs_bio_alloc (BIO_MAX_VECS , opf);/* * For compressed page range, its disk_bytenr is always @disk_bytenr * passed in, no matter if we have added any range into previous bio.
@@ -3277,7 +3276,6 @@ static int alloc_new_bio(struct btrfs_inode *inode, bio_ctrl->bio = bio; bio_ctrl->compress_type = compress_type; bio->bi_end_io = end_io_func; - bio->bi_opf = opf; ret =calc_bio_boundaries (bio_ctrl, inode, file_offset);if (ret <0 ) goto error;
fs/btrfs/inode.c +1 -2
@@ -10419,12 +10419,11 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,size_t bytes =min_t (u64, remaining,PAGE_SIZE );if (!bio) { - bio =btrfs_bio_alloc (BIO_MAX_VECS ); + bio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ ); bio->bi_iter.bi_sector = (disk_bytenr + cur) >>SECTOR_SHIFT ; bio->bi_end_io = btrfs_encoded_read_endio; bio->bi_private = &priv; - bio->bi_opf =REQ_OP_READ ; } if (!bytes ||
fs/btrfs/volumes.c +2 -2
@@ -6626,11 +6626,11 @@ static inline void btrfs_bio_init(struct btrfs_bio *bbio)* Just like the underlying bio_alloc_bioset it will no fail as it is backed by * a mempool. */ -struct bio *btrfs_bio_alloc (unsigned int nr_vecs) +struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf) {struct bio *bio; - bio =bio_alloc_bioset (NULL , nr_vecs,0 ,GFP_NOFS , &btrfs_bioset); + bio =bio_alloc_bioset (NULL , nr_vecs, opf,GFP_NOFS , &btrfs_bioset);btrfs_bio_init (btrfs_bio (bio));return bio; }
fs/btrfs/volumes.h +1 -1
@@ -391,7 +391,7 @@ static inline struct btrfs_bio *btrfs_bio(struct bio *bio) return container_of(bio, struct btrfs_bio, bio); } -struct bio *btrfs_bio_alloc (unsigned int nr_vecs); +struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf);struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size);static inline void btrfs_bio_free_csum (struct btrfs_bio *bbio)
Signature · 1 line
2.30.2
Re: [PATCH 3/7] btrfs: pass the operation to btrfs_bio_alloc
Looks good,
PATCH 4/7 btrfs: properly abstract the parity raid bio handling
In reply to Christoph HellwigThe parity raid write/recover functionality is currently now very well abstracted from the bio submission and completion handling in volumes.c:
- the raid56 code directly completes the original btrfs_bio fed into btrfs_submit_bio instead of dispatching back to volumes.c
- the raid56 code consumes the bioc and bio_counter references taken by volumes.c, which also leads to ugly special casing of the calls from the scrub code into the raid56 code
To fix this up supply a bi_end_io handler that calls back into the volumes.c machinery, which then puts the bioc, decrements the bio_counter and completes the original bio, and updates the scrub code to also take ownership of the bioc and bio_counter in all cases.
--- fs/btrfs/raid56.c | 45 +++++++-------------------------------------- fs/btrfs/raid56.h | 4 +--- fs/btrfs/scrub.c | 7 +++++-- fs/btrfs/volumes.c | 18 +++++++++++++++++- 4 files changed, 30 insertions(+), 44 deletions(-)
fs/btrfs/raid56.c +7 -38
@@ -275,7 +275,6 @@ static void merge_rbio(struct btrfs_raid_bio *dest,/* Also inherit the bitmaps from @victim. */ bitmap_or (&dest->dbitmap, &victim->dbitmap, &dest->dbitmap, dest->stripe_nsectors); - dest->generic_bio_cnt += victim->generic_bio_cnt;bio_list_init (&victim->bio_list); }
@@ -814,8 +813,6 @@ static void rbio_orig_end_io(struct btrfs_raid_bio *rbio, blk_status_t err)struct bio *cur =bio_list_get (&rbio->bio_list);struct bio *extra; -if (rbio->generic_bio_cnt) -btrfs_bio_counter_sub (rbio->bioc->fs_info, rbio->generic_bio_cnt);/* * Clear the data bitmap, as the rbio may be cached for later usage. * do this before before unlock_stripe() so there will be no new bio
@@ -946,6 +943,7 @@ static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info,spin_lock_init (&rbio->bio_list_lock);INIT_LIST_HEAD (&rbio->stripe_cache);INIT_LIST_HEAD (&rbio->hash_list); +btrfs_get_bioc (bioc); rbio->bioc = bioc; rbio->nr_pages = num_pages; rbio->nr_sectors = num_sectors;
@@ -1813,15 +1811,12 @@ void raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc) rbio =alloc_rbio (fs_info, bioc);if (IS_ERR (rbio)) { -btrfs_put_bioc (bioc); ret =PTR_ERR (rbio); - goto out_dec_counter; + goto fail; } rbio->operation =BTRFS_RBIO_WRITE ;rbio_add_bio (rbio, bio); - rbio->generic_bio_cnt =1 ; -/* * don't plug on full rbios, just get them out the door * as quickly as we can
@@ -1829,7 +1824,7 @@ void raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc)if (rbio_is_full (rbio)) { ret =full_stripe_write (rbio);if (ret) - goto out_dec_counter; + goto fail;return ; }
@@ -1844,13 +1839,12 @@ void raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc) }else { ret =__raid56_parity_write (rbio);if (ret) - goto out_dec_counter; + goto fail; }return ; -out_dec_counter: -btrfs_bio_counter_dec (fs_info); +fail: bio->bi_status =errno_to_blk_status (ret);bio_endio (bio); }
@@ -2198,18 +2192,11 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio)* of the drive. */ void raid56_parity_recover (struct bio *bio,struct btrfs_io_context *bioc, -int mirror_num,bool generic_io) +int mirror_num) {struct btrfs_fs_info *fs_info = bioc->fs_info;struct btrfs_raid_bio *rbio; -if (generic_io) { -ASSERT (bioc->mirror_num == mirror_num); -btrfs_bio (bio)->mirror_num = mirror_num; - }else { -btrfs_get_bioc (bioc); - } - rbio =alloc_rbio (fs_info, bioc);if (IS_ERR (rbio)) { bio->bi_status =errno_to_blk_status (PTR_ERR (rbio));
@@ -2225,14 +2212,11 @@ void raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,"%s could not find the bad stripe in raid56 so that we cannot recover any more (bio has logical %llu len %llu, bioc has map_type %llu)" , __func__, bio->bi_iter.bi_sector <<9 , (u64)bio->bi_iter.bi_size, bioc->map_type); -kfree (rbio); +__free_raid_bio (rbio); bio->bi_status =BLK_STS_IOERR ; goto out_end_bio; } -if (generic_io) - rbio->generic_bio_cnt =1 ; -/* * Loop retry: * for 'mirror == 2', reconstruct from all other stripes.
@@ -2261,8 +2245,6 @@ void raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,return ; out_end_bio: -btrfs_bio_counter_dec (fs_info); -btrfs_put_bioc (bioc);bio_endio (bio); }
@@ -2326,13 +2308,6 @@ struct btrfs_raid_bio *raid56_parity_alloc_scrub_rbio(struct bio *bio,ASSERT (i < rbio->real_stripes);bitmap_copy (&rbio->dbitmap, dbitmap, stripe_nsectors); - -/* -* We have already increased bio_counter when getting bioc, record it -* so we can free it at rbio_orig_end_io(). -*/ - rbio->generic_bio_cnt =1 ; -return rbio; }
@@ -2772,12 +2747,6 @@ raid56_alloc_missing_rbio(struct bio *bio, struct btrfs_io_context *bioc)return NULL ; } -/* -* When we get bioc, we have already increased bio_counter, record it -* so we can free it at rbio_orig_end_io() -*/ - rbio->generic_bio_cnt =1 ; -return rbio; }
fs/btrfs/raid56.h +1 -3
@@ -89,8 +89,6 @@ struct btrfs_raid_bio {*/ int bio_list_bytes; -int generic_bio_cnt; -refcount_t refs;atomic_t stripes_pending;
@@ -166,7 +164,7 @@ static inline int nr_data_stripes(const struct map_lookup *map)struct btrfs_device ;void raid56_parity_recover (struct bio *bio,struct btrfs_io_context *bioc, -int mirror_num,bool generic_io); +int mirror_num);void raid56_parity_write (struct bio *bio,struct btrfs_io_context *bioc); void raid56_add_scrub_pages(struct btrfs_raid_bio *rbio, struct page *page,
fs/btrfs/scrub.c +5 -2
@@ -1381,7 +1381,7 @@ static int scrub_submit_raid56_bio_wait(struct btrfs_fs_info *fs_info, bio->bi_private = &done; bio->bi_end_io = scrub_bio_wait_endio;raid56_parity_recover (bio, sector->recover->bioc, - sector->sblock->sectors[0 ]->mirror_num, false); + sector->sblock->sectors[0 ]->mirror_num);wait_for_completion_io (&done);return blk_status_to_errno (bio->bi_status);
@@ -2102,6 +2102,7 @@ static void scrub_missing_raid56_end_io(struct bio *bio)struct scrub_block *sblock = bio->bi_private;struct btrfs_fs_info *fs_info = sblock->sctx->fs_info; +btrfs_bio_counter_dec (fs_info);if (bio->bi_status) sblock->no_io_error_seen =0 ;
@@ -2204,6 +2205,7 @@ static void scrub_missing_raid56_pages(struct scrub_block *sblock)scrub_block_get (sblock);scrub_pending_bio_inc (sctx);raid56_submit_missing_rbio (rbio); +btrfs_put_bioc (bioc);return ; rbio_out:
@@ -2774,6 +2776,7 @@ static void scrub_parity_bio_endio_worker(struct work_struct *work) work);struct scrub_ctx *sctx = sparity->sctx; +btrfs_bio_counter_dec (sctx->fs_info);scrub_free_parity (sparity);scrub_pending_bio_dec (sctx); }
@@ -2824,6 +2827,7 @@ static void scrub_parity_check_and_repair(struct scrub_parity *sparity) sparity->scrub_dev, &sparity->dbitmap, sparity->nsectors); +btrfs_put_bioc (bioc);if (!rbio) goto rbio_out;
@@ -2835,7 +2839,6 @@ static void scrub_parity_check_and_repair(struct scrub_parity *sparity)bio_put (bio); bioc_out:btrfs_bio_counter_dec (fs_info); -btrfs_put_bioc (bioc);bitmap_or (&sparity->ebitmap, &sparity->ebitmap, &sparity->dbitmap, sparity->nsectors);spin_lock (&sctx->stat_lock);
fs/btrfs/volumes.c +17 -1
@@ -6666,6 +6666,20 @@ static void btrfs_end_bio_work(struct work_struct *work) bbio->bio.bi_end_io (&bbio->bio); } +static void btrfs_raid56_end_io (struct bio *bio) +{ +struct btrfs_io_context *bioc = bio->bi_private; +struct btrfs_bio *bbio =btrfs_bio (bio); + +btrfs_bio_counter_dec (bioc->fs_info); + bbio->mirror_num = bioc->mirror_num; + bio->bi_end_io = bioc->end_io; + bio->bi_private = bioc->private; + bio->bi_end_io (bio); + +btrfs_put_bioc (bioc); +} +static void btrfs_end_bio (struct bio *bio) {struct btrfs_io_stripe *stripe = bio->bi_private;
@@ -6801,10 +6815,12 @@ void btrfs_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio, int mirrorif ((bioc->map_type &BTRFS_BLOCK_GROUP_RAID56_MASK ) && ((btrfs_op (bio) ==BTRFS_MAP_WRITE ) || (mirror_num >1 ))) { + bio->bi_private = bioc; + bio->bi_end_io = btrfs_raid56_end_io;if (btrfs_op (bio) ==BTRFS_MAP_WRITE )raid56_parity_write (bio, bioc);else -raid56_parity_recover (bio, bioc, mirror_num, true); +raid56_parity_recover (bio, bioc, mirror_num);return ; }
Signature · 1 line
2.30.2
Re: [PATCH 4/7] btrfs: properly abstract the parity raid bio handling
Quoted 1 line · Christoph Hellwig
The parity raid write/recover functionality is currently now very well not? ~^
For the rest of the patch, I don't know the raid56 code enough to feel qualified for reviewing it.
PATCH 5/7 btrfs: give struct btrfs_bio a real end_io handler
In reply to Christoph HellwigCurrently btrfs_bio end I/O handling is a bit of a mess. The bi_end_io handler and bi_private pointer of the embedded struct bio are both used to handle the completion of the high-level btrfs_bio and for the I/O completion for the low-level device that the embedded bio ends up being sent to. To support this bi_end_io and bi_private are saved into the btrfs_io_context structure and then restored after the bio sent to the underlying device has completed the actual I/O.
Untangle this by adding an end I/O handler and private data to struct btrfs_bio for the highlevel btrfs_bio based completions, and leave the actual bio bi_end_io handler and bi_private pointer entirely to the low-level device I/O.
--- fs/btrfs/compression.c | 49 +++++++++++++++--------------------- fs/btrfs/disk-io.c | 16 ++++++------ fs/btrfs/extent_io.c | 35 +++++++++++++------------- fs/btrfs/inode.c | 56 +++++++++++++++++++----------------------- fs/btrfs/volumes.c | 32 +++++++++++------------- fs/btrfs/volumes.h | 20 ++++++++++++--- 6 files changed, 100 insertions(+), 108 deletions(-)
fs/btrfs/compression.c +20 -29
@@ -151,11 +151,7 @@ static void finish_compressed_bio_read(struct compressed_bio *cb)put_page (page); } -/* Do io completion on the original bio */ -if (cb->status !=BLK_STS_OK ) { - cb->orig_bio->bi_status = cb->status; -bio_endio (cb->orig_bio); - }else { +if (cb->status ==BLK_STS_OK ) {struct bio_vec *bvec;struct bvec_iter_all iter_all;
@@ -172,10 +168,11 @@ static void finish_compressed_bio_read(struct compressed_bio *cb) bvec->bv_page, bvec_start, bvec->bv_len); } - -bio_endio (cb->orig_bio); } +/* Do io completion on the original bio */ +btrfs_bio_end_io (btrfs_bio (cb->orig_bio), cb->status); +/* Finally free the cb struct */ kfree (cb->compressed_pages);kfree (cb);
@@ -186,16 +183,15 @@ static void finish_compressed_bio_read(struct compressed_bio *cb)* before decompressing it into the original bio and freeing the uncompressed * pages. */ -static void end_compressed_bio_read (struct bio *bio) +static void end_compressed_bio_read (struct btrfs_bio *bbio) { -struct compressed_bio *cb = bio->bi_private; +struct compressed_bio *cb = bbio->private;struct inode *inode = cb->inode;struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct btrfs_inode *bi =BTRFS_I (inode);bool csum = !(bi->flags &BTRFS_INODE_NODATASUM ) && !test_bit (BTRFS_FS_STATE_NO_CSUMS , &fs_info->fs_state); -blk_status_t status = bio->bi_status; -struct btrfs_bio *bbio =btrfs_bio (bio); +blk_status_t status = bbio->bio.bi_status;struct bvec_iter iter;struct bio_vec bv;u32 offset;
@@ -229,7 +225,7 @@ static void end_compressed_bio_read(struct bio *bio)if (refcount_dec_and_test (&cb->pending_ios))finish_compressed_bio_read (cb);btrfs_bio_free_csum (bbio); -bio_put (bio); +bio_put (&bbio->bio); }/*
@@ -321,20 +317,20 @@ static void btrfs_finish_compressed_write_work(struct work_struct *work)* This also calls the writeback end hooks for the file pages so that metadata * and checksums can be updated in the file. */ -static void end_compressed_bio_write (struct bio *bio) +static void end_compressed_bio_write (struct btrfs_bio *bbio) { -struct compressed_bio *cb = bio->bi_private; +struct compressed_bio *cb = bbio->private; -if (bio->bi_status) - cb->status = bio->bi_status; +if (bbio->bio.bi_status) + cb->status = bbio->bio.bi_status;if (refcount_dec_and_test (&cb->pending_ios)) {struct btrfs_fs_info *fs_info =btrfs_sb (cb->inode->i_sb); -btrfs_record_physical_zoned (cb->inode, cb->start, bio); +btrfs_record_physical_zoned (cb->inode, cb->start, &bbio->bio);queue_work (fs_info->compressed_write_workers, &cb->write_end_work); } -bio_put (bio); +bio_put (&bbio->bio); }/*
@@ -355,7 +351,8 @@ static void end_compressed_bio_write(struct bio *bio)static struct bio *alloc_compressed_bio (struct compressed_bio *cb,u64 disk_bytenr, -unsigned int opf,bio_end_io_t endio_func, +unsigned int opf, +btrfs_bio_end_io_t endio_func,u64 *next_stripe_start) {struct btrfs_fs_info *fs_info =btrfs_sb (cb->inode->i_sb);
@@ -364,11 +361,8 @@ static struct bio *alloc_compressed_bio(struct compressed_bio *cb, u64 disk_bytestruct bio *bio;int ret; - bio =btrfs_bio_alloc (BIO_MAX_VECS , opf); - + bio =btrfs_bio_alloc (BIO_MAX_VECS , opf, endio_func, cb); bio->bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; - bio->bi_private = cb; - bio->bi_end_io = endio_func; em =btrfs_get_chunk_map (fs_info, disk_bytenr, fs_info->sectorsize);if (IS_ERR (em)) {
@@ -497,8 +491,7 @@ blk_status_t btrfs_submit_compressed_write(struct btrfs_inode *inode, u64 start,if (!skip_sum) { ret =btrfs_csum_one_bio (inode, bio, start, true);if (ret) { - bio->bi_status = ret; -bio_endio (bio); +btrfs_bio_end_io (btrfs_bio (bio), ret);break ; } }
@@ -819,8 +812,7 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, ret =btrfs_lookup_bio_sums (inode, comp_bio,NULL );if (ret) { - comp_bio->bi_status = ret; -bio_endio (comp_bio); +btrfs_bio_end_io (btrfs_bio (comp_bio), ret);break ; }
@@ -846,8 +838,7 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,kfree (cb); out:free_extent_map (em); - bio->bi_status = ret; -bio_endio (bio); +btrfs_bio_end_io (btrfs_bio (bio), ret);return ; }
fs/btrfs/disk-io.c +7 -9
@@ -728,16 +728,14 @@ static void run_one_async_start(struct btrfs_work *work)*/ static void run_one_async_done (struct btrfs_work *work) { -struct async_submit_bio *async; -struct inode *inode; - - async = container_of(work, struct async_submit_bio, work); - inode = async->inode; +struct async_submit_bio *async = + container_of(work, struct async_submit_bio, work); +struct inode *inode = async->inode; +struct btrfs_bio *bbio =btrfs_bio (async->bio);/* If an error occurred we just want to clean up the bio and move on */ if (async->status) { - async->bio->bi_status = async->status; -bio_endio (async->bio); +btrfs_bio_end_io (bbio, async->status);return ; }
@@ -838,6 +836,7 @@ static bool should_async_write(struct btrfs_fs_info *fs_info,void btrfs_submit_metadata_bio (struct inode *inode,struct bio *bio,int mirror_num) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb); +struct btrfs_bio *bbio =btrfs_bio (bio);blk_status_t ret; bio->bi_opf |=REQ_META ;
@@ -857,8 +856,7 @@ void btrfs_submit_metadata_bio(struct inode *inode, struct bio *bio, int mirror_ ret =btree_csum_one_bio (bio);if (ret) { - bio->bi_status = ret; -bio_endio (bio); +btrfs_bio_end_io (bbio, ret);return ; }
fs/btrfs/extent_io.c +17 -18
@@ -206,7 +206,7 @@ static void submit_one_bio(struct btrfs_bio_ctrl *bio_ctrl)btrfs_submit_data_read_bio (inode, bio, mirror_num, bio_ctrl->compress_type); -/* The bio is owned by the bi_end_io handler now */ +/* The bio is owned by the end_io handler now */ bio_ctrl->bio =NULL ; }
@@ -222,9 +222,8 @@ static void submit_write_bio(struct extent_page_data *epd, int ret)if (ret) {ASSERT (ret <0 ); - bio->bi_status =errno_to_blk_status (ret); -bio_endio (bio); -/* The bio is owned by the bi_end_io handler now */ +btrfs_bio_end_io (btrfs_bio (bio),errno_to_blk_status (ret)); +/* The bio is owned by the end_io handler now */ epd->bio_ctrl.bio =NULL ; }else {submit_one_bio (&epd->bio_ctrl);
@@ -2627,12 +2626,11 @@ int btrfs_repair_one_sector(struct inode *inode, struct btrfs_bio *failed_bbio,return -EIO ; } - repair_bio =btrfs_bio_alloc (1 ,REQ_OP_READ ); + repair_bio =btrfs_bio_alloc (1 ,REQ_OP_READ , failed_bbio->end_io, + failed_bbio->private); repair_bbio =btrfs_bio (repair_bio); repair_bbio->file_offset = start; - repair_bio->bi_end_io = failed_bio->bi_end_io; repair_bio->bi_iter.bi_sector = failrec->logical >>9 ; - repair_bio->bi_private = failed_bio->bi_private;if (failed_bbio->csum) {const u32 csum_size = fs_info->csum_size;
@@ -2799,8 +2797,9 @@ void end_extent_writepage(struct page *page, int err, u64 start, u64 end)* Scheduling is not allowed, so the extent state tree is expected * to have one and only one object corresponding to this IO. */ -static void end_bio_extent_writepage (struct bio *bio) +static void end_bio_extent_writepage (struct btrfs_bio *bbio) { +struct bio *bio = &bbio->bio;int error =blk_status_to_errno (bio->bi_status);struct bio_vec *bvec;u64 start;
@@ -2962,10 +2961,10 @@ static struct extent_buffer *find_extent_buffer_readpage(* Scheduling is not allowed, so the extent state tree is expected * to have one and only one object corresponding to this IO. */ -static void end_bio_extent_readpage (struct bio *bio) +static void end_bio_extent_readpage (struct btrfs_bio *bbio) { +struct bio *bio = &bbio->bio;struct bio_vec *bvec; -struct btrfs_bio *bbio =btrfs_bio (bio);struct extent_io_tree *tree, *failure_tree;struct processed_extent processed = {0 };/*
@@ -3256,7 +3255,7 @@ static int alloc_new_bio(struct btrfs_inode *inode,struct btrfs_bio_ctrl *bio_ctrl, struct writeback_control *wbc, unsigned int opf, - bio_end_io_t end_io_func, + btrfs_bio_end_io_t end_io_func, u64 disk_bytenr, u32 offset, u64 file_offset, enum btrfs_compression_type compress_type) {
@@ -3264,7 +3263,7 @@ static int alloc_new_bio(struct btrfs_inode *inode,struct bio *bio;int ret; - bio =btrfs_bio_alloc (BIO_MAX_VECS , opf); + bio =btrfs_bio_alloc (BIO_MAX_VECS , opf, end_io_func,NULL );/* * For compressed page range, its disk_bytenr is always @disk_bytenr * passed in, no matter if we have added any range into previous bio.
@@ -3275,7 +3274,6 @@ static int alloc_new_bio(struct btrfs_inode *inode, bio->bi_iter.bi_sector = (disk_bytenr + offset) >>SECTOR_SHIFT ; bio_ctrl->bio = bio; bio_ctrl->compress_type = compress_type; - bio->bi_end_io = end_io_func; ret =calc_bio_boundaries (bio_ctrl, inode, file_offset);if (ret <0 ) goto error;
@@ -3314,8 +3312,7 @@ static int alloc_new_bio(struct btrfs_inode *inode,return 0 ; error: bio_ctrl->bio =NULL ; - bio->bi_status =errno_to_blk_status (ret); -bio_endio (bio); +btrfs_bio_end_io (btrfs_bio (bio),errno_to_blk_status (ret));return ret; }
@@ -3338,7 +3335,7 @@ static int submit_extent_page(unsigned int opf,struct btrfs_bio_ctrl *bio_ctrl, struct page *page, u64 disk_bytenr, size_t size, unsigned long pg_offset, - bio_end_io_t end_io_func, + btrfs_bio_end_io_t end_io_func, enum btrfs_compression_type compress_type, bool force_bio_submit) {
@@ -4335,8 +4332,9 @@ static struct extent_buffer *find_extent_buffer_nolock(* Unlike end_bio_extent_buffer_writepage(), we only call end_page_writeback() * after all extent buffers in the page has finished their writeback. */ -static void end_bio_subpage_eb_writepage (struct bio *bio) +static void end_bio_subpage_eb_writepage (struct btrfs_bio *bbio) { +struct bio *bio = &bbio->bio;struct btrfs_fs_info *fs_info;struct bio_vec *bvec;struct bvec_iter_all iter_all;
@@ -4392,8 +4390,9 @@ static void end_bio_subpage_eb_writepage(struct bio *bio)bio_put (bio); } -static void end_bio_extent_buffer_writepage (struct bio *bio) +static void end_bio_extent_buffer_writepage (struct btrfs_bio *bbio) { +struct bio *bio = &bbio->bio;struct bio_vec *bvec;struct extent_buffer *eb;int done;
fs/btrfs/inode.c +25 -31
140 lines not on this page · raw message
fs/btrfs/volumes.c +15 -17
@@ -6614,9 +6614,12 @@ int btrfs_map_sblock(struct btrfs_fs_info *fs_info, enum btrfs_map_op op,* Initialize a btrfs_bio structure. This skips the embedded bio itself as it * is already initialized by the block layer. */ -static inline void btrfs_bio_init (struct btrfs_bio *bbio) +static inline void btrfs_bio_init (struct btrfs_bio *bbio, +btrfs_bio_end_io_t end_io,void *private) {memset (bbio,0 , offsetof(struct btrfs_bio , bio)); + bbio->end_io = end_io; + bbio->private = private; }/*
@@ -6626,16 +6629,18 @@ static inline void btrfs_bio_init(struct btrfs_bio *bbio)* Just like the underlying bio_alloc_bioset it will no fail as it is backed by * a mempool. */ -struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf) +struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf, +btrfs_bio_end_io_t end_io,void *private) {struct bio *bio; bio =bio_alloc_bioset (NULL , nr_vecs, opf,GFP_NOFS , &btrfs_bioset); -btrfs_bio_init (btrfs_bio (bio)); +btrfs_bio_init (btrfs_bio (bio), end_io, private);return bio; } -struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size) +struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size, +btrfs_bio_end_io_t end_io,void *private) {struct bio *bio;struct btrfs_bio *bbio;
@@ -6644,7 +6649,7 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, u64 offset, u64 size) bio =bio_alloc_clone (orig->bi_bdev, orig,GFP_NOFS , &btrfs_bioset); bbio =btrfs_bio (bio); -btrfs_bio_init (bbio); +btrfs_bio_init (bbio, end_io, private);bio_trim (bio, offset >>9 , size >>9 ); bbio->iter = bio->bi_iter;
@@ -6663,7 +6668,7 @@ static void btrfs_end_bio_work(struct work_struct *work)struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, end_io_work); - bbio->bio.bi_end_io (&bbio->bio); + bbio->end_io (bbio); }static void btrfs_raid56_end_io (struct bio *bio)
@@ -6673,9 +6678,7 @@ static void btrfs_raid56_end_io(struct bio *bio)btrfs_bio_counter_dec (bioc->fs_info); bbio->mirror_num = bioc->mirror_num; - bio->bi_end_io = bioc->end_io; - bio->bi_private = bioc->private; - bio->bi_end_io (bio); + bbio->end_io (bbio);btrfs_put_bioc (bioc); }
@@ -6721,14 +6724,12 @@ static void btrfs_end_bio(struct bio *bio)btrfs_bio_counter_dec (bioc->fs_info); bbio->mirror_num = bioc->mirror_num; - orig_bio->bi_end_io = bioc->end_io; - orig_bio->bi_private = bioc->private; -if (btrfs_op (orig_bio) ==BTRFS_MAP_READ ) { +if (btrfs_op (bio) ==BTRFS_MAP_READ ) { bbio->device = stripe->dev;INIT_WORK (&bbio->end_io_work, btrfs_end_bio_work);queue_work (btrfs_end_io_wq (bioc), &bbio->end_io_work); }else { - orig_bio->bi_end_io (orig_bio); + bbio->end_io (bbio); }btrfs_put_bioc (bioc);
@@ -6803,15 +6804,12 @@ void btrfs_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio, int mirror &map_length, &bioc, mirror_num, 1);if (ret) {btrfs_bio_counter_dec (fs_info); - bio->bi_status =errno_to_blk_status (ret); -bio_endio (bio); +btrfs_bio_end_io (btrfs_bio (bio),errno_to_blk_status (ret));return ; } total_devs = bioc->num_stripes; bioc->orig_bio = bio; - bioc->private = bio->bi_private; - bioc->end_io = bio->bi_end_io;if ((bioc->map_type &BTRFS_BLOCK_GROUP_RAID56_MASK ) && ((btrfs_op (bio) ==BTRFS_MAP_WRITE ) || (mirror_num >1 ))) {
fs/btrfs/volumes.h +16 -4
@@ -361,6 +361,8 @@ struct btrfs_fs_devices {*/ #define BTRFS_MAX_BIO_SECTORS (256) +typedef void (*btrfs_bio_end_io_t )(struct btrfs_bio *bbio); +/* * Additional info to pass along bio. *
@@ -376,6 +378,10 @@ struct btrfs_bio {u8 csum_inline[BTRFS_BIO_INLINE_CSUM_SIZE ];struct bvec_iter iter; +/* End I/O information supplied to btrfs_bio_alloc */ +btrfs_bio_end_io_t end_io; +void *private; +/* For read end I/O handling */ struct work_struct end_io_work;
@@ -391,8 +397,16 @@ static inline struct btrfs_bio *btrfs_bio(struct bio *bio) return container_of(bio, struct btrfs_bio, bio); } -struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf); -struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size); +struct bio *btrfs_bio_alloc (unsigned int nr_vecs,unsigned int opf, +btrfs_bio_end_io_t end_io,void *private); +struct bio *btrfs_bio_clone_partial (struct bio *orig,u64 offset,u64 size, +btrfs_bio_end_io_t end_io,void *private); + +static inline void btrfs_bio_end_io (struct btrfs_bio *bbio,blk_status_t status) +{ + bbio->bio.bi_status = status; + bbio->end_io (bbio); +}static inline void btrfs_bio_free_csum (struct btrfs_bio *bbio) {
@@ -454,9 +468,7 @@ struct btrfs_io_context {refcount_t refs;struct btrfs_fs_info *fs_info;u64 map_type;/* get from map_lookup->type */ -bio_end_io_t *end_io;struct bio *orig_bio; -void *private;atomic_t error;int max_errors;int num_stripes;
Signature · 1 line
2.30.2
PATCH 6/7 btrfs: split out the end_io handler for cloned bios
In reply to Christoph HellwigSplit the handler for cloned bios from the main end I/O handler. This cleans up the logic to be easier to follow, and prepares for additional low-level bio completion changes.
--- fs/btrfs/volumes.c | 59 +++++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 24 deletions(-)
fs/btrfs/volumes.c +35 -24
@@ -6656,6 +6656,21 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, u64 offset, u64 size,return bio; } +static void btrfs_log_dev_io_error (struct bio *bio,struct btrfs_device *dev) +{ +if (!dev || !dev->bdev) +return ; +if (bio->bi_status !=BLK_STS_IOERR && bio->bi_status !=BLK_STS_TARGET ) +return ; + +if (btrfs_op (bio) ==BTRFS_MAP_WRITE ) +btrfs_dev_stat_inc_and_print (dev,BTRFS_DEV_STAT_WRITE_ERRS ); +if (!(bio->bi_opf &REQ_RAHEAD )) +btrfs_dev_stat_inc_and_print (dev,BTRFS_DEV_STAT_READ_ERRS ); +if (bio->bi_opf &REQ_PREFLUSH ) +btrfs_dev_stat_inc_and_print (dev,BTRFS_DEV_STAT_FLUSH_ERRS ); +} +static struct workqueue_struct *btrfs_end_io_wq (struct btrfs_io_context *bioc) {if (bioc->orig_bio->bi_opf &REQ_META )
@@ -6683,34 +6698,29 @@ static void btrfs_raid56_end_io(struct bio *bio)btrfs_put_bioc (bioc); } +static void btrfs_clone_write_end_io (struct bio *bio) +{ +struct btrfs_io_stripe *stripe = bio->bi_private; + +if (bio->bi_status) { +atomic_inc (&stripe->bioc->error); +btrfs_log_dev_io_error (bio, stripe->dev); + } + +/* Pass on cotrol to the original bio this was cloned from */ +bio_endio (stripe->bioc->orig_bio); +bio_put (bio); +} +static void btrfs_end_bio (struct bio *bio) {struct btrfs_io_stripe *stripe = bio->bi_private;struct btrfs_io_context *bioc = stripe->bioc; -struct bio *orig_bio = bioc->orig_bio; -struct btrfs_bio *bbio =btrfs_bio (orig_bio); +struct btrfs_bio *bbio =btrfs_bio (bio);if (bio->bi_status) {atomic_inc (&bioc->error); -if (stripe->dev && stripe->dev->bdev && - (bio->bi_status ==BLK_STS_IOERR || - bio->bi_status ==BLK_STS_TARGET )) { -if (btrfs_op (bio) ==BTRFS_MAP_WRITE ) -btrfs_dev_stat_inc_and_print (stripe->dev, -BTRFS_DEV_STAT_WRITE_ERRS ); -else if (!(bio->bi_opf &REQ_RAHEAD )) -btrfs_dev_stat_inc_and_print (stripe->dev, -BTRFS_DEV_STAT_READ_ERRS ); -if (bio->bi_opf &REQ_PREFLUSH ) -btrfs_dev_stat_inc_and_print (stripe->dev, -BTRFS_DEV_STAT_FLUSH_ERRS ); - } - } - -if (bio != orig_bio) { -bio_endio (orig_bio); -bio_put (bio); -return ; +btrfs_log_dev_io_error (bio, stripe->dev); }/*
@@ -6718,9 +6728,9 @@ static void btrfs_end_bio(struct bio *bio)* threshold. */ if (atomic_read (&bioc->error) > bioc->max_errors) - orig_bio->bi_status =BLK_STS_IOERR ; + bio->bi_status =BLK_STS_IOERR ;else - orig_bio->bi_status =BLK_STS_OK ; + bio->bi_status =BLK_STS_OK ;btrfs_bio_counter_dec (bioc->fs_info); bbio->mirror_num = bioc->mirror_num;
@@ -6746,13 +6756,14 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc,if (clone) { bio =bio_alloc_clone (NULL , orig_bio,GFP_NOFS , &fs_bio_set);bio_inc_remaining (orig_bio); + bio->bi_end_io = btrfs_clone_write_end_io; }else { bio = orig_bio; + bio->bi_end_io = btrfs_end_bio; } bioc->stripes[dev_nr].bioc = bioc; bio->bi_private = &bioc->stripes[dev_nr]; - bio->bi_end_io = btrfs_end_bio; bio->bi_iter.bi_sector = physical >>9 ; if (!dev || !dev->bdev ||
Signature · 1 line
2.30.2
Re: [PATCH 6/7] btrfs: split out the end_io handler for cloned bios
Looks good,
PATCH 7/7 btrfs: stop allocation a btrfs_io_context for simple I/O
In reply to Christoph HellwigThe I/O context structure is only really used to pass the btrfs_device to the end I/O handler for I/Os that go to a single device.
Stop allocating the I/O context for these cases and just pass an optional btrfs_io_stripe argument to __btrfs_map_block that is filled out for this fast path. Additionally the num_mirrors argument is changed to be passed by references so that the mirror_num can be returned in addition to be passed in.
--- fs/btrfs/volumes.c | 194 +++++++++++++++++++++++++++++---------------- 1 file changed, 124 insertions(+), 70 deletions(-)
fs/btrfs/volumes.c +124 -70
@@ -250,10 +250,10 @@ static int btrfs_relocate_sys_chunks(struct btrfs_fs_info *fs_info);static void btrfs_dev_stat_print_on_error (struct btrfs_device *dev);static void btrfs_dev_stat_print_on_load (struct btrfs_device *device);static int __btrfs_map_block (struct btrfs_fs_info *fs_info, -enum btrfs_map_op op, -u64 logical,u64 *length, +enum btrfs_map_op op,u64 logical,u64 *length,struct btrfs_io_context **bioc_ret, -int mirror_num,int need_raid_map); +struct btrfs_io_stripe *smap, +int *mirror_num_p,int need_raid_map);/* * Device locking
@@ -6088,7 +6088,7 @@ static int get_extra_mirror_from_replace(struct btrfs_fs_info *fs_info,int ret =0 ; ret =__btrfs_map_block (fs_info,BTRFS_MAP_GET_READ_MIRRORS , - logical, &length, &bioc,0 ,0 ); + logical, &length, &bioc,NULL ,NULL ,0 );if (ret) {ASSERT (bioc ==NULL );return ret;
@@ -6347,11 +6347,19 @@ int btrfs_get_io_geometry(struct btrfs_fs_info *fs_info, struct extent_map *em,return 0 ; } +static void set_stripe (struct btrfs_io_stripe *dst,struct map_lookup *map, +u32 stripe_index,u64 stripe_offset,u64 stripe_nr) +{ + dst->dev = map->stripes[stripe_index].dev; + dst->physical = map->stripes[stripe_index].physical + + stripe_offset + stripe_nr * map->stripe_len; +} +static int __btrfs_map_block (struct btrfs_fs_info *fs_info, -enum btrfs_map_op op, -u64 logical,u64 *length, +enum btrfs_map_op op,u64 logical,u64 *length,struct btrfs_io_context **bioc_ret, -int mirror_num,int need_raid_map) +struct btrfs_io_stripe *smap, +int *mirror_num_p,int need_raid_map) {struct extent_map *em;struct map_lookup *map;
@@ -6362,6 +6370,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info,int data_stripes;int i;int ret =0 ; +int mirror_num = mirror_num_p ? *mirror_num_p :0 ;int num_stripes;int max_errors =0 ;int tgtdev_indexes =0 ;
@@ -6522,6 +6531,29 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, tgtdev_indexes = num_stripes; } +/* +* If this I/O maps to a single device, try to return the device and +* physical block information on the stack instead of allocating an +* I/O context structure. +*/ +if (smap && num_alloc_stripes ==1 && + !((map->type &BTRFS_BLOCK_GROUP_RAID56_MASK ) && mirror_num >1 ) && + (!need_full_stripe (op) || !dev_replace_is_ongoing || + !dev_replace->tgtdev)) { +if (unlikely (patch_the_first_stripe_for_dev_replace)) { + smap->dev = dev_replace->tgtdev; + smap->physical = physical_to_patch_in_first_stripe; + *mirror_num_p = map->num_stripes +1 ; + }else { +set_stripe (smap, map, stripe_index, stripe_offset, + stripe_nr); + *mirror_num_p = mirror_num; + } + *bioc_ret =NULL ; + ret =0 ; + goto out; + } + bioc =alloc_btrfs_io_context (fs_info, num_alloc_stripes, tgtdev_indexes);if (!bioc) { ret = -ENOMEM ;
@@ -6529,9 +6561,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, }for (i =0 ; i < num_stripes; i++) { - bioc->stripes[i].physical = map->stripes[stripe_index].physical + - stripe_offset + stripe_nr * map->stripe_len; - bioc->stripes[i].dev = map->stripes[stripe_index].dev; +set_stripe (&bioc->stripes[i], map, stripe_index, stripe_offset, + stripe_nr); stripe_index++; }
@@ -6599,7 +6630,7 @@ int btrfs_map_block(struct btrfs_fs_info *fs_info, enum btrfs_map_op op, struct btrfs_io_context **bioc_ret, int mirror_num) {return __btrfs_map_block (fs_info, op, logical, length, bioc_ret, - mirror_num,0 ); +NULL , &mirror_num,0 ); }/* For Scrub/replace */
@@ -6607,7 +6638,8 @@ int btrfs_map_sblock(struct btrfs_fs_info *fs_info, enum btrfs_map_op op, u64 logical, u64 *length, struct btrfs_io_context **bioc_ret) { - return __btrfs_map_block(fs_info, op, logical, length, bioc_ret, 0, 1); + return __btrfs_map_block(fs_info, op, logical, length, bioc_ret, + NULL, NULL, 1); }/*
@@ -6671,11 +6703,12 @@ static void btrfs_log_dev_io_error(struct bio *bio, struct btrfs_device *dev)btrfs_dev_stat_inc_and_print (dev,BTRFS_DEV_STAT_FLUSH_ERRS ); } -static struct workqueue_struct *btrfs_end_io_wq (struct btrfs_io_context *bioc) +static struct workqueue_struct *btrfs_end_io_wq (struct btrfs_fs_info *fs_info, +struct bio *bio) { -if (bioc->orig_bio->bi_opf &REQ_META ) -return bioc->fs_info->endio_meta_workers; -return bioc->fs_info->endio_workers; +if (bio->bi_opf &REQ_META ) +return fs_info->endio_meta_workers; +return fs_info->endio_workers; }static void btrfs_end_bio_work (struct work_struct *work)
@@ -6686,6 +6719,23 @@ static void btrfs_end_bio_work(struct work_struct *work) bbio->end_io (bbio); } +static void btrfs_simple_end_io (struct bio *bio) +{ +struct btrfs_fs_info *fs_info = bio->bi_private; +struct btrfs_bio *bbio =btrfs_bio (bio); + +if (bio->bi_status) +btrfs_log_dev_io_error (bio, bbio->device); +btrfs_bio_counter_dec (fs_info); + +if (bio_op (bio) ==REQ_OP_READ ) { +INIT_WORK (&bbio->end_io_work, btrfs_end_bio_work); +queue_work (btrfs_end_io_wq (fs_info, bio), &bbio->end_io_work); + }else { + bbio->end_io (bbio); + } +} +static void btrfs_raid56_end_io (struct bio *bio) {struct btrfs_io_context *bioc = bio->bi_private;
@@ -6712,7 +6762,7 @@ static void btrfs_clone_write_end_io(struct bio *bio)bio_put (bio); } -static void btrfs_end_bio (struct bio *bio) +static void btrfs_orig_write_end_io (struct bio *bio) {struct btrfs_io_stripe *stripe = bio->bi_private;struct btrfs_io_context *bioc = stripe->bioc;
@@ -6733,39 +6783,12 @@ static void btrfs_end_bio(struct bio *bio) bio->bi_status =BLK_STS_OK ;btrfs_bio_counter_dec (bioc->fs_info); - bbio->mirror_num = bioc->mirror_num; -if (btrfs_op (bio) ==BTRFS_MAP_READ ) { - bbio->device = stripe->dev; -INIT_WORK (&bbio->end_io_work, btrfs_end_bio_work); -queue_work (btrfs_end_io_wq (bioc), &bbio->end_io_work); - }else { - bbio->end_io (bbio); - } - + bbio->end_io (bbio);btrfs_put_bioc (bioc); } -static void submit_stripe_bio (struct btrfs_io_context *bioc, -struct bio *orig_bio,int dev_nr,bool clone) +static void btrfs_submit_dev_bio (struct btrfs_device *dev,struct bio *bio) { -struct btrfs_fs_info *fs_info = bioc->fs_info; -struct btrfs_device *dev = bioc->stripes[dev_nr].dev; -u64 physical = bioc->stripes[dev_nr].physical; -struct bio *bio; - -if (clone) { - bio =bio_alloc_clone (NULL , orig_bio,GFP_NOFS , &fs_bio_set); -bio_inc_remaining (orig_bio); - bio->bi_end_io = btrfs_clone_write_end_io; - }else { - bio = orig_bio; - bio->bi_end_io = btrfs_end_bio; - } - - bioc->stripes[dev_nr].bioc = bioc; - bio->bi_private = &bioc->stripes[dev_nr]; - bio->bi_iter.bi_sector = physical >>9 ; - if (!dev || !dev->bdev || test_bit(BTRFS_DEV_STATE_MISSING, &dev->dev_state) || (btrfs_op(bio) == BTRFS_MAP_WRITE &&
@@ -6781,8 +6804,11 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc,* zone */ if (bio_op (bio) ==REQ_OP_ZONE_APPEND ) { +u64 physical = bio->bi_iter.bi_sector <<SECTOR_SHIFT ; +if (btrfs_dev_is_sequential (dev, physical)) { -u64 zone_start =round_down (physical, fs_info->zone_size); +u64 zone_start =round_down (physical, + dev->fs_info->zone_size); bio->bi_iter.bi_sector = zone_start >>SECTOR_SHIFT ; }else {
@@ -6790,7 +6816,8 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc, bio->bi_opf |=REQ_OP_WRITE ; } } - btrfs_debug_in_rcu(fs_info, + + btrfs_debug_in_rcu(dev->fs_info, "%s: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u", __func__, bio_op(bio), bio->bi_opf, bio->bi_iter.bi_sector, (unsigned long)dev->bdev->bd_dev, rcu_str_deref(dev->name),
@@ -6800,39 +6827,46 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc,submit_bio (bio); } +static void btrfs_submit_mirrored_bio (struct btrfs_io_context *bioc,int dev_nr) +{ +struct bio *orig_bio = bioc->orig_bio, *bio; + +ASSERT (bio_op (orig_bio) !=REQ_OP_READ ); + +/* Reuse the bio embedded into the btrfs_bio for the last mirror */ +if (dev_nr == bioc->num_stripes -1 ) { + bio = orig_bio; + bio->bi_end_io = btrfs_orig_write_end_io; + }else { + bio =bio_alloc_clone (NULL , orig_bio,GFP_NOFS , &fs_bio_set); +bio_inc_remaining (orig_bio); + bio->bi_end_io = btrfs_clone_write_end_io; + } + + bio->bi_private = &bioc->stripes[dev_nr]; + bio->bi_iter.bi_sector = bioc->stripes[dev_nr].physical >>SECTOR_SHIFT ; + bioc->stripes[dev_nr].bioc = bioc; +btrfs_submit_dev_bio (bioc->stripes[dev_nr].dev, bio); +} +void btrfs_submit_bio (struct btrfs_fs_info *fs_info,struct bio *bio,int mirror_num) {u64 logical = bio->bi_iter.bi_sector <<9 ;u64 length = bio->bi_iter.bi_size;u64 map_length = length; -int ret; -int dev_nr; -int total_devs;struct btrfs_io_context *bioc =NULL ; +struct btrfs_io_stripe smap; +int ret;btrfs_bio_counter_inc_blocked (fs_info); - ret =__btrfs_map_block (fs_info,btrfs_op (bio), logical, - &map_length, &bioc, mirror_num,1 ); + ret =__btrfs_map_block (fs_info,btrfs_op (bio), logical, &map_length, + &bioc, &smap, &mirror_num,1 );if (ret) {btrfs_bio_counter_dec (fs_info);btrfs_bio_end_io (btrfs_bio (bio),errno_to_blk_status (ret));return ; } - total_devs = bioc->num_stripes; - bioc->orig_bio = bio; - -if ((bioc->map_type &BTRFS_BLOCK_GROUP_RAID56_MASK ) && - ((btrfs_op (bio) ==BTRFS_MAP_WRITE ) || (mirror_num >1 ))) { - bio->bi_private = bioc; - bio->bi_end_io = btrfs_raid56_end_io; -if (btrfs_op (bio) ==BTRFS_MAP_WRITE ) -raid56_parity_write (bio, bioc); -else -raid56_parity_recover (bio, bioc, mirror_num); -return ; - } -if (map_length < length) { btrfs_crit(fs_info, "mapping failed logical %llu bio len %llu len %llu",
@@ -6840,10 +6874,30 @@ void btrfs_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio, int mirrorBUG (); } -for (dev_nr =0 ; dev_nr < total_devs; dev_nr++) { -const bool should_clone = (dev_nr < total_devs -1 ); +if (!bioc) { +/* Single mirror read/write fast path */ +btrfs_bio (bio)->mirror_num = mirror_num; +btrfs_bio (bio)->device = smap.dev; + bio->bi_iter.bi_sector = smap.physical >>SECTOR_SHIFT ; + bio->bi_private = fs_info; + bio->bi_end_io = btrfs_simple_end_io; +btrfs_submit_dev_bio (smap.dev, bio); + }else if (bioc->map_type &BTRFS_BLOCK_GROUP_RAID56_MASK ) { +/* Parity RAID write or read recovery */ + bio->bi_private = bioc; + bio->bi_end_io = btrfs_raid56_end_io; +if (bio_op (bio) ==REQ_OP_READ ) +raid56_parity_recover (bio, bioc, mirror_num); +else +raid56_parity_write (bio, bioc); + }else { +/* Write to multiple mirrors */ +int total_devs = bioc->num_stripes; +int dev_nr; -submit_stripe_bio (bioc, bio, dev_nr, should_clone); + bioc->orig_bio = bio; +for (dev_nr =0 ; dev_nr < total_devs; dev_nr++) +btrfs_submit_mirrored_bio (bioc, dev_nr); } }
Signature · 1 line
2.30.2
Re: [PATCH 7/7] btrfs: stop allocation a btrfs_io_context for simple I/O
Looks good,
Re: [PATCH 7/7] btrfs: stop allocation a btrfs_io_context for simple I/O
In reply to Christoph HellwigQuoted 10 lines · Christoph Hellwig
The I/O context structure is only really used to pass the
btrfs_deviceto the end I/O handler for I/Os that go to a single device.Stop allocating the I/O context for these cases and just pass an optional
btrfs_io_stripeargument to__btrfs_map_blockthat is filled out for this fast path. Additionally thenum_mirrorsargument is changed to be passed by references so that themirror_numcan be returned in addition to be passed in.Signed-off-by Christoph Hellwig <[email protected]>
This patch IMO needs to be split:
1. Make a patch which introduces the various split endio functions.
2. Then do the cleanups around __btrfs_map_block group and how submitted bios are being initialized and processed in btrfs_submit_bio et al.
Re: [PATCH 7/7] btrfs: stop allocation a btrfs_io_context for simple I/O
Quoted 3 lines · Nikolay Borisov
This patch IMO needs to be split:
1. Make a patch which introduces the various split endio functions.
Where "the various split endio functions" is btrfs_simple_end_io?
btrfs_simple_end_io relies on the fact that it doesn't need to deal with and a free a btrfs_io_context, so without:
Quoted 2 lines
2. Then do the cleanups around
__btrfs_map_blockgroup and how submitted bios are being initialized and processed inbtrfs_submit_bioet al.
the only thing I could do is to first allocate an btrfs_io_context and just free it directly after I/O submission. Which sounds doable if we really want to split it for splitting's sake, but it probably hurts understanding the concept instead of helping that.
Johannes Thumshirn
Tested-byQuoted 19 lines · Christoph Hellwig
Hi all,
this series cleans up the
btrfs_bioAPI, most prominently by splitting theend_iohandler for the highlevel bio from the low-level biobi_end_io, which are really confusingly coupled in the current code. Once that is done it then optimizes the bio submission to not allocate abtrfs_io_contextfor I/Os tht just go to a single device.This series sits on top of the "fix read repair on compressed extents v2" series submitted. To make everyones life easier a git tree is also available:
git://git.infradead.org/users/hch/misc.gitbtrfs-bio-api-cleanupGitweb:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/btrfs-bio-api-cleanup
Diffstat:
Quoted 14 lines
compression.c | 50 +++----- disk-io.c | 16 +- extent-io-tree.h | 4 extent_io.c | 117 +++---------------- extent_io.h | 3 inode.c | 57 ++++----- raid56.c | 45 +------ raid56.h | 4 scrub.c | 7 - super.c | 6 - volumes.c | 330 ++++++++++++++++++++++++++++++++++++++----------------- volumes.h | 19 ++- 12 files changed, 343 insertions(+), 315 deletions(-)
For the whole series:
(fstests on zoned as well as a base branch for my RST work)
Quoted 19 lines · Christoph Hellwig
Hi all,
this series cleans up the
btrfs_bioAPI, most prominently by splitting theend_iohandler for the highlevel bio from the low-level biobi_end_io, which are really confusingly coupled in the current code. Once that is done it then optimizes the bio submission to not allocate abtrfs_io_contextfor I/Os tht just go to a single device.This series sits on top of the "fix read repair on compressed extents v2" series submitted. To make everyones life easier a git tree is also available:
git://git.infradead.org/users/hch/misc.gitbtrfs-bio-api-cleanupGitweb:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/btrfs-bio-api-cleanup
Diffstat:
Quoted 13 lines
compression.c | 50 +++----- disk-io.c | 16 +- extent-io-tree.h | 4 extent_io.c | 117 +++---------------- extent_io.h | 3 inode.c | 57 ++++----- raid56.c | 45 +------ raid56.h | 4 scrub.c | 7 - super.c | 6 - volumes.c | 330 ++++++++++++++++++++++++++++++++++++++----------------- volumes.h | 19 ++- 12 files changed, 343 insertions(+), 315 deletions(-)