btrfs: read repair/direct I/O improvements
Subsystems (4)
v2 cover btrfs: read repair/direct I/O improvements
From: Omar Sandoval <[email protected]>
Hi,
This is version 2 of my series of fixes, cleanups, and improvements to direct I/O and read repair. It's preparation for adding read repair to my RWF_ENCODED series [1], but it can go in independently.
Patch 1 adds a new bio helper needed for patch 4. Patches 2 and 3 are direct I/O error handling fixes. Patch 4 is a buffered read repair fix. Patch 5 is a buffered read repair improvement. Patches 6-9 are trivial cleanups. Patch 10 converts direct I/O to use refcount_t, which would've helped catch the bug fixed by patch 2. Patches 11-14 drastically simplify the direct I/O code. Patch 15 unifies buffered and direct I/O read repair, which also makes direct I/O repair actually do validation for large failed reads instead of rewriting the whole thing.
Overall, this is net about -350 lines of code and actually makes direct I/O more functional.
Note that this series causes btrfs/142 to fail. This is a bug in the test, as it assumes that direct I/O doesn't do read validation. This is fixed by the fstests patch "btrfs/14{2,3}: use dm-dust instead of fail_make_request" which I sent up yesterday [2].
Jens and Christoph are cc'd for patches 1 and 4. Instead of looking at the bio internals like I did in v1, I added a new bio_for_each_bvec_all() helper.
Changes from v1 [3]:
* Added patch 1 with bio_for_each_bvec_all() helper
* Dropped patch 8 which moved struct definition
* Fixed performance regression [4] in patch 13 caused by accidentally making all direct I/O submission asynchronous
* Fixed uninitialized assert in patch 12
* Fixed misplaced assert in btrfs_check_read_dio_bio in patch 13
* Added reviewed-bys
* Refactored btrfs_submit_direct() and btrfs_submit_direct_hook() in patch 2 (I didn't add Nikolay's reviewed-by to that one because the new patch looks fairly different from patch 1 in v1)
* Rewrapped csum calculations in patch 10 for easier readability
* Clarified several commit messages and comments
This series is based on misc-next.
Thanks!
1: https://lore.kernel.org/linux-fsdevel/[email protected]/
2: https://lore.kernel.org/fstests/d992390752c612acd0893ee3db929e77affded2b.1586983958.git.osandov@fb.com/
3: https://lore.kernel.org/linux-btrfs/[email protected]/
4: https://lore.kernel.org/lkml/20200331090145.GH11705@shao2-debian/
Omar Sandoval (15):
block: add bio_for_each_bvec_all()
btrfs: fix error handling when submitting direct I/O bio
btrfs: fix double __endio_write_update_ordered in direct I/O
btrfs: look at full bi_io_vec for repair decision
btrfs: don't do repair validation for checksum errors
btrfs: clarify btrfs_lookup_bio_sums documentation
btrfs: rename __readpage_endio_check to check_data_csum
btrfs: make btrfs_check_repairable() static
btrfs: kill btrfs_dio_private->private
btrfs: convert btrfs_dio_private->pending_bios to refcount_t
btrfs: put direct I/O checksums in btrfs_dio_private instead of bio
btrfs: get rid of one layer of bios in direct I/O
btrfs: simplify direct I/O read repair
btrfs: get rid of endio_repair_workers
btrfs: unify buffered and direct I/O read repair
.clang-format | 1 + Documentation/block/biovecs.rst | 2 + fs/btrfs/btrfs_inode.h | 26 +- fs/btrfs/ctree.h | 1 - fs/btrfs/disk-io.c | 8 +- fs/btrfs/disk-io.h | 1 - fs/btrfs/extent_io.c | 152 ++++--- fs/btrfs/extent_io.h | 19 +- fs/btrfs/file-item.c | 11 +- fs/btrfs/inode.c | 745 ++++++++------------------------ include/linux/bio.h | 8 + 11 files changed, 312 insertions(+), 662 deletions(-)
Signature · 1 line
2.26.1
PATCH v2 1/15 block: add bio_for_each_bvec_all()
From: Omar Sandoval <[email protected]>
An upcoming Btrfs fix needs to know the original size of a non-cloned bios. Rather than accessing the bvec table directly, let's add a bio_for_each_bvec_all() accessor.
--- .clang-format | 1 + Documentation/block/biovecs.rst | 2 ++ include/linux/bio.h | 8 ++++++++ 3 files changed, 11 insertions(+)
.clang-format +1 -0
@@ -80,6 +80,7 @@ ForEachMacros: - 'ax25_uid_for_each' - '__bio_for_each_bvec' - 'bio_for_each_bvec' + - 'bio_for_each_bvec_all' - 'bio_for_each_integrity_vec' - '__bio_for_each_segment' - 'bio_for_each_segment'
Documentation/block/biovecs.rst +2 -0
@@ -129,6 +129,7 @@ Usage of helpers: :: bio_for_each_segment_all() + bio_for_each_bvec_all() bio_first_bvec_all() bio_first_page_all() bio_last_bvec_all()
@@ -143,4 +144,5 @@ Usage of helpers: bio_vec' will contain a multi-page IO vector during the iteration:: bio_for_each_bvec() + bio_for_each_bvec_all() rq_for_each_bvec()
include/linux/bio.h +8 -0
@@ -169,6 +169,14 @@ static inline void bio_advance_iter(struct bio *bio, struct bvec_iter *iter,#define bio_for_each_bvec (bvl, bio, iter) \ __bio_for_each_bvec(bvl, bio, iter, (bio)->bi_iter) +/* +* Iterate over all multi-page bvecs. Drivers shouldn't use this version for the +* same reasons as bio_for_each_segment_all(). +*/ +#define bio_for_each_bvec_all (bvl, bio, i) \ + for (i = 0, bvl = bio_first_bvec_all(bio); \ + i < (bio)->bi_vcnt; i++, bvl++) \ +#define bio_iter_last (bvec, iter) ((iter).bi_size == (bvec).bv_len)static inline unsigned bio_segments (struct bio *bio)
Signature · 1 line
2.26.1
PATCH v2 2/15 btrfs: fix error handling when submitting direct I/O bio
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
In btrfs_submit_direct_hook(), if a direct I/O write doesn't span a RAID stripe or chunk, we submit orig_bio without cloning it. In this case, we don't increment pending_bios. Then, if btrfs_submit_dio_bio() fails, we decrement pending_bios to -1, and we never complete orig_bio. Fix it by initializing pending_bios to 1 instead of incrementing later.
Fixing this exposes another bug: we put orig_bio prematurely and then put it again from end_io. Fix it by not putting orig_bio.
After this change, pending_bios is really more of a reference count, but I'll leave that cleanup separate to keep the fix small.
--- fs/btrfs/inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/btrfs/inode.c +3 -3
@@ -7939,7 +7939,6 @@ static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip)/* bio split */ ASSERT (geom.len <=INT_MAX ); -atomic_inc (&dip->pending_bios); do { clone_len = min_t(int, submit_len, geom.len);
@@ -7989,7 +7988,8 @@ static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip)if (!status)return 0 ; -bio_put (bio); +if (bio != orig_bio) +bio_put (bio); out_err: dip->errors =1 ;/*
@@ -8030,7 +8030,7 @@ static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode, bio->bi_private = dip; dip->orig_bio = bio; dip->dio_bio = dio_bio; -atomic_set (&dip->pending_bios,0 ); +atomic_set (&dip->pending_bios,1 ); io_bio =btrfs_io_bio (bio); io_bio->logical = file_offset;
Signature · 1 line
2.26.1
PATCH v2 3/15 btrfs: fix double __endio_write_update_ordered in direct I/O
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
In btrfs_submit_direct(), if we fail to allocate the btrfs_dio_private, we complete the ordered extent range. However, we don't mark that the range doesn't need to be cleaned up from btrfs_direct_IO() until later. Therefore, if we fail to allocate the btrfs_dio_private, we complete the ordered extent range twice. We could fix this by updating unsubmitted_oe_range earlier, but it's cleaner to reorganize the code so that creating the btrfs_dio_private and submitting the bios are separate, and once the btrfs_dio_private is created, cleanup always happens through the btrfs_dio_private.
--- fs/btrfs/inode.c | 174 ++++++++++++++++++----------------------------- 1 file changed, 66 insertions(+), 108 deletions(-)
fs/btrfs/inode.c +66 -108
@@ -7903,14 +7903,60 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,return ret; } -static int btrfs_submit_direct_hook (struct btrfs_dio_private *dip) +/* +* If this succeeds, the btrfs_dio_private is responsible for cleaning up locked +* or ordered extents whether or not we submit any bios. +*/ +static struct btrfs_dio_private *btrfs_create_dio_private (struct bio *dio_bio, +struct inode *inode, +loff_t file_offset) { -struct inode *inode = dip->inode; +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE ); +struct btrfs_dio_private *dip; +struct bio *bio; + + dip =kzalloc (sizeof (*dip),GFP_NOFS ); +if (!dip) +return NULL ; + + bio =btrfs_bio_clone (dio_bio); + bio->bi_private = dip; +btrfs_io_bio (bio)->logical = file_offset; + + dip->private = dio_bio->bi_private; + dip->inode = inode; + dip->logical_offset = file_offset; + dip->bytes = dio_bio->bi_iter.bi_size; + dip->disk_bytenr = (u64 )dio_bio->bi_iter.bi_sector <<9 ; + dip->orig_bio = bio; + dip->dio_bio = dio_bio; +atomic_set (&dip->pending_bios,1 ); + +if (write) { +struct btrfs_dio_data *dio_data = current->journal_info; + + dio_data->unsubmitted_oe_range_end = dip->logical_offset + + dip->bytes; + dio_data->unsubmitted_oe_range_start = + dio_data->unsubmitted_oe_range_end; + + bio->bi_end_io = btrfs_endio_direct_write; + }else { + bio->bi_end_io = btrfs_endio_direct_read; + dip->subio_endio = btrfs_subio_endio_read; + } +return dip; +} + +static void btrfs_submit_direct (struct bio *dio_bio,struct inode *inode, +loff_t file_offset) +{ +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE );struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb); +struct btrfs_dio_private *dip;struct bio *bio; -struct bio *orig_bio = dip->orig_bio; -u64 start_sector = orig_bio->bi_iter.bi_sector; -u64 file_offset = dip->logical_offset; +struct bio *orig_bio; +u64 start_sector;int async_submit =0 ;u64 submit_len;int clone_offset =0 ;
@@ -7919,11 +7965,24 @@ static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip)blk_status_t status;struct btrfs_io_geometry geom; + dip =btrfs_create_dio_private (dio_bio, inode, file_offset); +if (!dip) { +if (!write) { +unlock_extent (&BTRFS_I (inode)->io_tree, file_offset, + file_offset + dio_bio->bi_iter.bi_size -1 ); + } + dio_bio->bi_status =BLK_STS_RESOURCE ; +dio_end_io (dio_bio); +return ; + } + + orig_bio = dip->orig_bio; + start_sector = orig_bio->bi_iter.bi_sector; submit_len = orig_bio->bi_iter.bi_size; ret =btrfs_get_io_geometry (fs_info,btrfs_op (orig_bio), start_sector <<9 , submit_len, &geom);if (ret) -return -EIO ; + goto out_err;if (geom.len >= submit_len) { bio = orig_bio;
@@ -7986,7 +8045,7 @@ static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip) submit: status =btrfs_submit_dio_bio (bio, inode, file_offset, async_submit);if (!status) -return 0 ; +return ;if (bio != orig_bio)bio_put (bio);
@@ -8000,107 +8059,6 @@ static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip)*/ if (atomic_dec_and_test (&dip->pending_bios))bio_io_error (dip->orig_bio); - -/* bio_end_io() will handle error, so we needn't return it */ -return 0 ; -} - -static void btrfs_submit_direct (struct bio *dio_bio,struct inode *inode, -loff_t file_offset) -{ -struct btrfs_dio_private *dip =NULL ; -struct bio *bio =NULL ; -struct btrfs_io_bio *io_bio; -bool write = (bio_op (dio_bio) ==REQ_OP_WRITE ); -int ret =0 ; - - bio =btrfs_bio_clone (dio_bio); - - dip =kzalloc (sizeof (*dip),GFP_NOFS ); -if (!dip) { - ret = -ENOMEM ; - goto free_ordered; - } - - dip->private = dio_bio->bi_private; - dip->inode = inode; - dip->logical_offset = file_offset; - dip->bytes = dio_bio->bi_iter.bi_size; - dip->disk_bytenr = (u64 )dio_bio->bi_iter.bi_sector <<9 ; - bio->bi_private = dip; - dip->orig_bio = bio; - dip->dio_bio = dio_bio; -atomic_set (&dip->pending_bios,1 ); - io_bio =btrfs_io_bio (bio); - io_bio->logical = file_offset; - -if (write) { - bio->bi_end_io = btrfs_endio_direct_write; - }else { - bio->bi_end_io = btrfs_endio_direct_read; - dip->subio_endio = btrfs_subio_endio_read; - } - -/* -* Reset the range for unsubmitted ordered extents (to a 0 length range) -* even if we fail to submit a bio, because in such case we do the -* corresponding error handling below and it must not be done a second -* time by btrfs_direct_IO(). -*/ -if (write) { -struct btrfs_dio_data *dio_data = current->journal_info; - - dio_data->unsubmitted_oe_range_end = dip->logical_offset + - dip->bytes; - dio_data->unsubmitted_oe_range_start = - dio_data->unsubmitted_oe_range_end; - } - - ret =btrfs_submit_direct_hook (dip); -if (!ret) -return ; - -btrfs_io_bio_free_csum (io_bio); - -free_ordered: -/* -* If we arrived here it means either we failed to submit the dip -* or we either failed to clone the dio_bio or failed to allocate the -* dip. If we cloned the dio_bio and allocated the dip, we can just -* call bio_endio against our io_bio so that we get proper resource -* cleanup if we fail to submit the dip, otherwise, we must do the -* same as btrfs_endio_direct_[write|read] because we can't call these -* callbacks - they require an allocated dip and a clone of dio_bio. -*/ -if (bio && dip) { -bio_io_error (bio); -/* -* The end io callbacks free our dip, do the final put on bio -* and all the cleanup and final put for dio_bio (through -* dio_end_io()). -*/ - dip =NULL ; - bio =NULL ; - }else { -if (write) -__endio_write_update_ordered (inode, - file_offset, - dio_bio->bi_iter.bi_size, - false); -else -unlock_extent (&BTRFS_I (inode)->io_tree, file_offset, - file_offset + dio_bio->bi_iter.bi_size -1 ); - - dio_bio->bi_status =BLK_STS_IOERR ; -/* -* Releases and cleans up our dio_bio, no need to bio_put() -* nor bio_endio()/bio_io_error() against dio_bio. -*/ -dio_end_io (dio_bio); - } -if (bio) -bio_put (bio); -kfree (dip); } static ssize_t check_direct_IO(struct btrfs_fs_info *fs_info,
Signature · 1 line
2.26.1
PATCH v2 5/15 btrfs: don't do repair validation for checksum errors
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
The purpose of the validation step is to distinguish between good and bad sectors in a failed multi-sector read. If a multi-sector read succeeded but some of those sectors had checksum errors, we don't need to validate anything; we know the sectors with bad checksums need to be repaired.
--- fs/btrfs/extent_io.c | 8 ++++++++ 1 file changed, 8 insertions(+)
fs/btrfs/extent_io.c +8 -0
@@ -2640,6 +2640,14 @@ static bool btrfs_io_needs_validation(struct inode *inode, struct bio *bio)u64 len =0 ;int i; +/* +* If bi_status is BLK_STS_OK, then this was a checksum error, not an +* I/O error. In this case, we already know exactly which sector was +* bad, so we don't need to validate. +*/ +if (bio->bi_status ==BLK_STS_OK ) +return false; +/* * We need to validate each sector individually if the failed I/O was * for multiple sectors.
Signature · 1 line
2.26.1
PATCH v2 4/15 btrfs: look at full bi_io_vec for repair decision
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Read repair does two things: it finds a good copy of data to return to the reader, and it corrects the bad copy on disk. If a read of multiple sectors has an I/O error, repair does an extra "validation" step that issues a separate read for each sector. This allows us to find the exact failing sectors and only rewrite those.
This heuristic is implemented in
bio_readpage_error()/btrfs_check_repairable() as:
failed_bio_pages = failed_bio->bi_iter.bi_size >> PAGE_SHIFT; if (failed_bio_pages > 1) do validation
However, at this point, bi_iter may have already been advanced. This means that we'll skip the validation step and rewrite the entire failed read.
Fix it by getting the actual size from the biovec (which we can do because this is only called for non-cloned bios, although that will change in a later commit).
--- fs/btrfs/extent_io.c | 33 +++++++++++++++++++++++++++------ fs/btrfs/extent_io.h | 5 +++-- 2 files changed, 30 insertions(+), 8 deletions(-)
fs/btrfs/extent_io.c +27 -6
@@ -2537,8 +2537,9 @@ int btrfs_get_io_failure_record(struct inode *inode, u64 start, u64 end,return 0 ; } -bool btrfs_check_repairable (struct inode *inode,unsigned failed_bio_pages, -struct io_failure_record *failrec,int failed_mirror) +bool btrfs_check_repairable (struct inode *inode,bool need_validation, +struct io_failure_record *failrec, +int failed_mirror) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);int num_copies;
@@ -2561,7 +2562,7 @@ bool btrfs_check_repairable(struct inode *inode, unsigned failed_bio_pages,* a) deliver good data to the caller * b) correct the bad sectors on disk */ -if (failed_bio_pages >1 ) { +if (need_validation) {/* * to fulfill b), we need to know the exact failing sectors, as * we don't want to rewrite any more than the failed ones. thus,
@@ -2633,6 +2634,24 @@ struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio,return bio; } +static bool btrfs_io_needs_validation (struct inode *inode,struct bio *bio) +{ +struct bio_vec *bvec; +u64 len =0 ; +int i; + +/* +* We need to validate each sector individually if the failed I/O was +* for multiple sectors. +*/ +bio_for_each_bvec_all (bvec, bio, i) { + len += bvec->bv_len; +if (len > inode->i_sb->s_blocksize) +return true; + } +return false; +} +/* * This is a generic handler for readpage errors. If other copies exist, read * those and write back good data to the failed position. Does not investigate
@@ -2647,11 +2666,11 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset,struct inode *inode = page->mapping->host;struct extent_io_tree *tree = &BTRFS_I (inode)->io_tree;struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree; +bool need_validation;struct bio *bio;int read_mode =0 ;blk_status_t status;int ret; -unsigned failed_bio_pages = failed_bio->bi_iter.bi_size >>PAGE_SHIFT ;BUG_ON (bio_op (failed_bio) ==REQ_OP_WRITE );
@@ -2659,13 +2678,15 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset,if (ret)return ret; -if (!btrfs_check_repairable (inode, failed_bio_pages, failrec, + need_validation =btrfs_io_needs_validation (inode, failed_bio); + +if (!btrfs_check_repairable (inode, need_validation, failrec, failed_mirror)) {free_io_failure (failure_tree, tree, failrec);return -EIO ; } -if (failed_bio_pages >1 ) +if (need_validation) read_mode |=REQ_FAILFAST_DEV ; phy_offset >>= inode->i_sb->s_blocksize_bits;
fs/btrfs/extent_io.h +3 -2
@@ -312,8 +312,9 @@ struct io_failure_record { }; -bool btrfs_check_repairable (struct inode *inode,unsigned failed_bio_pages, -struct io_failure_record *failrec,int fail_mirror); +bool btrfs_check_repairable (struct inode *inode,bool need_validation, +struct io_failure_record *failrec, +int failed_mirror); struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio, struct io_failure_record *failrec, struct page *page, int pg_offset, int icsum,
Signature · 1 line
2.26.1
PATCH v2 6/15 btrfs: clarify btrfs_lookup_bio_sums documentation
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Fix a couple of issues in the btrfs_lookup_bio_sums documentation:
* The bio doesn't need to be a btrfs_io_bio if dst was provided. Move the declaration in the code to make that clear, too.
* dst must be large enough to hold nblocks * csum_size, not just csum_size.
--- fs/btrfs/file-item.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
fs/btrfs/file-item.c +7 -4
@@ -242,11 +242,13 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans,/** * btrfs_lookup_bio_sums - Look up checksums for a bio. * @inode: inode that the bio is for. -* @bio: bio embedded in btrfs_io_bio. +* @bio: bio to look up. * @offset: Unless (u64)-1, look up checksums for this offset in the file. * If (u64)-1, use the page offsets from the bio instead. -* @dst: Buffer of size btrfs_super_csum_size() used to return checksum. If -* NULL, the checksum is returned in btrfs_io_bio(bio)->csum instead. +* @dst: Buffer of size nblocks * btrfs_super_csum_size() used to return +* checksum (nblocks = bio->bi_iter.bi_size / fs_info->sectorsize). If +* NULL, the checksum buffer is allocated and returned in +* btrfs_io_bio(bio)->csum instead. * * Return: BLK_STS_RESOURCE if allocating memory fails, BLK_STS_OK otherwise. */
@@ -256,7 +258,6 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio,struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct bio_vec bvec;struct bvec_iter iter; -struct btrfs_io_bio *btrfs_bio =btrfs_io_bio (bio);struct btrfs_csum_item *item =NULL ;struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct btrfs_path *path;
@@ -277,6 +278,8 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, nblocks = bio->bi_iter.bi_size >> inode->i_sb->s_blocksize_bits;if (!dst) { +struct btrfs_io_bio *btrfs_bio =btrfs_io_bio (bio); +if (nblocks * csum_size >BTRFS_BIO_INLINE_CSUM_SIZE ) { btrfs_bio->csum =kmalloc_array (nblocks, csum_size,GFP_NOFS );
Signature · 1 line
2.26.1
PATCH v2 7/15 btrfs: rename __readpage_endio_check to check_data_csum
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
__readpage_endio_check() is also used from the direct I/O read code, so give it a more descriptive name.
--- fs/btrfs/inode.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
fs/btrfs/inode.c +11 -11
@@ -2726,10 +2726,9 @@ void btrfs_writepage_endio_finish_ordered(struct page *page, u64 start,btrfs_queue_work (wq, &ordered_extent->work); } -static int __readpage_endio_check (struct inode *inode, -struct btrfs_io_bio *io_bio, -int icsum,struct page *page, -int pgoff,u64 start,size_t len) +static int check_data_csum (struct inode *inode,struct btrfs_io_bio *io_bio, +int icsum,struct page *page,int pgoff,u64 start, +size_t len) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);SHASH_DESC_ON_STACK (shash, fs_info->csum_shash);
@@ -2790,8 +2789,8 @@ static int btrfs_readpage_end_io_hook(struct btrfs_io_bio *io_bio, } phy_offset >>= inode->i_sb->s_blocksize_bits; -return __readpage_endio_check (inode, io_bio, phy_offset, page, offset, - start, (size_t )(end - start +1 )); +return check_data_csum (inode, io_bio, phy_offset, page, offset, start, + (size_t )(end - start +1 )); }/*
@@ -7584,9 +7583,9 @@ static void btrfs_retry_endio(struct bio *bio)ASSERT (!bio_flagged (bio,BIO_CLONED ));bio_for_each_segment_all (bvec, bio, iter_all) { - ret =__readpage_endio_check (inode, io_bio, i, bvec->bv_page, - bvec->bv_offset, done->start, - bvec->bv_len); + ret =check_data_csum (inode, io_bio, i, bvec->bv_page, + bvec->bv_offset, done->start, + bvec->bv_len);if (!ret) clean_io_failure(BTRFS_I(inode)->root->fs_info, failure_tree, io_tree, done->start,
@@ -7636,8 +7635,9 @@ static blk_status_t __btrfs_subio_endio_read(struct inode *inode, next_block:if (uptodate) { csum_pos =BTRFS_BYTES_TO_BLKS (fs_info, offset); - ret =__readpage_endio_check (inode, io_bio, csum_pos, - bvec.bv_page, pgoff, start, sectorsize); + ret =check_data_csum (inode, io_bio, csum_pos, + bvec.bv_page, pgoff, start, + sectorsize);if (likely (!ret)) goto next; }
Signature · 1 line
2.26.1
PATCH v2 8/15 btrfs: make btrfs_check_repairable() static
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Since its introduction in commit 2fe6303e7cd0 ("Btrfs: split bio_readpage_error into several functions"), btrfs_check_repairable() has only been used from extent_io.c where it is defined.
--- fs/btrfs/extent_io.c | 7 ++++--- fs/btrfs/extent_io.h | 3 --- 2 files changed, 4 insertions(+), 6 deletions(-)
fs/btrfs/extent_io.c +4 -3
@@ -2537,9 +2537,10 @@ int btrfs_get_io_failure_record(struct inode *inode, u64 start, u64 end,return 0 ; } -bool btrfs_check_repairable (struct inode *inode,bool need_validation, -struct io_failure_record *failrec, -int failed_mirror) +static bool btrfs_check_repairable (struct inode *inode, +bool need_validation, +struct io_failure_record *failrec, +int failed_mirror) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);int num_copies;
fs/btrfs/extent_io.h +0 -3
@@ -312,9 +312,6 @@ struct io_failure_record { }; -bool btrfs_check_repairable (struct inode *inode,bool need_validation, -struct io_failure_record *failrec, -int failed_mirror); struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio, struct io_failure_record *failrec, struct page *page, int pg_offset, int icsum,
Signature · 1 line
2.26.1
PATCH v2 9/15 btrfs: kill btrfs_dio_private->private
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
We haven't used this since commit 9be3395bcd4a ("Btrfs: use a btrfs bioset instead of abusing bio internals").
--- fs/btrfs/btrfs_inode.h | 1 - fs/btrfs/inode.c | 1 - 2 files changed, 2 deletions(-)
fs/btrfs/btrfs_inode.h +0 -1
@@ -301,7 +301,6 @@ struct btrfs_dio_private {u64 logical_offset;u64 disk_bytenr;u64 bytes; -void *private;/* number of bios pending for this dio */ atomic_t pending_bios;
fs/btrfs/inode.c +0 -1
@@ -7923,7 +7923,6 @@ static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio, bio->bi_private = dip;btrfs_io_bio (bio)->logical = file_offset; - dip->private = dio_bio->bi_private; dip->inode = inode; dip->logical_offset = file_offset; dip->bytes = dio_bio->bi_iter.bi_size;
Signature · 1 line
2.26.1
PATCH v2 10/15 btrfs: convert btrfs_dio_private->pending_bios to refcount_t
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
This is really a reference count now, so convert it to refcount_t and rename it to refs.
--- fs/btrfs/btrfs_inode.h | 8 ++++++-- fs/btrfs/inode.c | 10 +++++----- 2 files changed, 11 insertions(+), 7 deletions(-)
fs/btrfs/btrfs_inode.h +6 -2
@@ -7,6 +7,7 @@#define BTRFS_INODE_H #include <linux/hash.h> +#include <linux/refcount.h> #include "extent_map.h" #include "extent_io.h" #include "ordered-data.h"
@@ -302,8 +303,11 @@ struct btrfs_dio_private {u64 disk_bytenr;u64 bytes; -/* number of bios pending for this dio */ -atomic_t pending_bios; +/* +* References to this structure. There is one reference per in-flight +* bio plus one while we're still setting up. +*/ +refcount_t refs;/* IO errors */ int errors;
fs/btrfs/inode.c +5 -5
@@ -7811,7 +7811,7 @@ static void btrfs_end_dio_bio(struct bio *bio) }/* if there are more bios still pending for this dio, just exit */ -if (!atomic_dec_and_test (&dip->pending_bios)) +if (!refcount_dec_and_test (&dip->refs)) goto out;if (dip->errors) {
@@ -7929,7 +7929,7 @@ static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio, dip->disk_bytenr = (u64 )dio_bio->bi_iter.bi_sector <<9 ; dip->orig_bio = bio; dip->dio_bio = dio_bio; -atomic_set (&dip->pending_bios,1 ); +refcount_set (&dip->refs,1 );if (write) {struct btrfs_dio_data *dio_data = current->journal_info;
@@ -8021,13 +8021,13 @@ static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode,* count. Otherwise, the dip might get freed before we're * done setting it up. */ -atomic_inc (&dip->pending_bios); +refcount_inc (&dip->refs); status =btrfs_submit_dio_bio (bio, inode, file_offset, async_submit);if (status) {bio_put (bio); -atomic_dec (&dip->pending_bios); +refcount_dec (&dip->refs); goto out_err; }
@@ -8056,7 +8056,7 @@ static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode,* atomic operations with a return value are fully ordered as per * atomic_t.txt */ -if (atomic_dec_and_test (&dip->pending_bios)) +if (refcount_dec_and_test (&dip->refs))bio_io_error (dip->orig_bio); }
Signature · 1 line
2.26.1
PATCH v2 11/15 btrfs: put direct I/O checksums in btrfs_dio_private instead of bio
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
The next commit will get rid of btrfs_dio_private->orig_bio. The only thing we really need it for is containing all of the checksums, but we can easily put the checksum array in btrfs_dio_private and have the submitted bios reference the array. We can also look the checksums up while we're setting up instead of the current awkward logic that looks them up for orig_bio when the first split bio is submitted.
(Interestingly, btrfs_dio_private did contain the checksums before commit 23ea8e5a0767 ("Btrfs: load checksum data once when submitting a direct read io"), but it didn't look them up up front.)
--- fs/btrfs/btrfs_inode.h | 3 ++ fs/btrfs/inode.c | 70 +++++++++++++++++++----------------------- 2 files changed, 34 insertions(+), 39 deletions(-)
fs/btrfs/btrfs_inode.h +3 -0
@@ -324,6 +324,9 @@ struct btrfs_dio_private {*/ blk_status_t (*subio_endio)(struct inode *,struct btrfs_io_bio *,blk_status_t ); + +/* Checksums. */ +u8 sums[]; };/*
fs/btrfs/inode.c +31 -39
109 lines not on this page · raw message
Signature · 1 line
2.26.1
PATCH v2 12/15 btrfs: get rid of one layer of bios in direct I/O
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
In the worst case, there are _4_ layers of bios in the Btrfs direct I/O path:
1. The bio created by the generic direct I/O code (dio_bio).
2. A clone of dio_bio we create in btrfs_submit_direct() to represent the entire direct I/O range (orig_bio).
3. A partial clone of orig_bio limited to the size of a RAID stripe that we create in btrfs_submit_direct_hook().
4. Clones of each of those split bios for each RAID stripe that we create in btrfs_map_bio().
As of the previous commit, the second layer (orig_bio) is no longer needed for anything: we can split dio_bio instead, and complete dio_bio directly when all of the cloned bios complete. This lets us clean up a bunch of cruft, including dip->subio_endio and dip->errors (we can use dio_bio->bi_status instead). It also enables the next big cleanup of direct I/O read repair.
--- fs/btrfs/btrfs_inode.h | 16 ---- fs/btrfs/inode.c | 185 +++++++++++++++-------------------------- 2 files changed, 65 insertions(+), 136 deletions(-)
fs/btrfs/btrfs_inode.h +0 -16
@@ -294,11 +294,8 @@ static inline int btrfs_inode_in_log(struct btrfs_inode *inode, u64 generation)return ret; } -#define BTRFS_DIO_ORIG_BIO_SUBMITTED 0x1 -struct btrfs_dio_private {struct inode *inode; -unsigned long flags;u64 logical_offset;u64 disk_bytenr;u64 bytes;
@@ -309,22 +306,9 @@ struct btrfs_dio_private {*/ refcount_t refs; -/* IO errors */ -int errors; - -/* orig_bio is our btrfs_io_bio */ -struct bio *orig_bio; -/* dio_bio came from fs/direct-io.c */ struct bio *dio_bio; -/* -* The original bio may be split to several sub-bios, this is -* done during endio of sub-bios -*/ -blk_status_t (*subio_endio)(struct inode *,struct btrfs_io_bio *, -blk_status_t ); -/* Checksums. */ u8 sums[]; };
fs/btrfs/inode.c +65 -120
290 lines not on this page · raw message
Signature · 1 line
2.26.1
PATCH v2 15/15 btrfs: unify buffered and direct I/O read repair
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Currently, direct I/O has its own versions of bio_readpage_error() and btrfs_check_repairable() (dio_read_error() and btrfs_check_dio_repairable(), respectively). The main difference is that the direct I/O version doesn't do read validation. The rework of direct I/O repair makes it possible to do validation, so we can get rid of btrfs_check_dio_repairable() and combine bio_readpage_error() and dio_read_error() into a new helper, btrfs_submit_read_repair().
--- fs/btrfs/extent_io.c | 130 ++++++++++++++++++++----------------------- fs/btrfs/extent_io.h | 17 ++++-- fs/btrfs/inode.c | 103 ++++------------------------------ 3 files changed, 82 insertions(+), 168 deletions(-)
fs/btrfs/extent_io.c +60 -70
@@ -2602,41 +2602,6 @@ static bool btrfs_check_repairable(struct inode *inode,return true; } - -struct bio *btrfs_create_repair_bio (struct inode *inode,struct bio *failed_bio, -struct io_failure_record *failrec, -struct page *page,int pg_offset,int icsum, -bio_end_io_t *endio_func,void *data) -{ -struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb); -struct bio *bio; -struct btrfs_io_bio *btrfs_failed_bio; -struct btrfs_io_bio *btrfs_bio; - - bio =btrfs_io_bio_alloc (1 ); - bio->bi_end_io = endio_func; - bio->bi_iter.bi_sector = failrec->logical >>9 ; - bio->bi_iter.bi_size =0 ; - bio->bi_private = data; - - btrfs_failed_bio =btrfs_io_bio (failed_bio); -if (btrfs_failed_bio->csum) { -u16 csum_size =btrfs_super_csum_size (fs_info->super_copy); - - btrfs_bio =btrfs_io_bio (bio); - btrfs_bio->csum = btrfs_bio->csum_inline; - icsum *= csum_size; -memcpy (btrfs_bio->csum, btrfs_failed_bio->csum + icsum, - csum_size); - } - -bio_add_page (bio, page, failrec->len, pg_offset); -btrfs_io_bio (bio)->logical = failrec->start; -btrfs_io_bio (bio)->iter = bio->bi_iter; - -return bio; -} -static bool btrfs_io_needs_validation (struct inode *inode,struct bio *bio) {struct bio_vec *bvec;
@@ -2654,72 +2619,96 @@ static bool btrfs_io_needs_validation(struct inode *inode, struct bio *bio)/* * We need to validate each sector individually if the failed I/O was * for multiple sectors. +* +* There are a few possible bios that can end up here: +* 1. A buffered read bio, which is not cloned. +* 2. A direct I/O read bio, which is cloned. +* 3. A (buffered or direct) repair bio, which is not cloned. +* +* For cloned bios (case 2), we can get the size from +* btrfs_io_bio->iter; for non-cloned bios (cases 1 and 3), we can get +* it from the bvecs. */ -bio_for_each_bvec_all (bvec, bio, i) { - len += bvec->bv_len; -if (len > inode->i_sb->s_blocksize) +if (bio_flagged (bio,BIO_CLONED )) { +if (btrfs_io_bio (bio)->iter.bi_size > inode->i_sb->s_blocksize)return true; + }else { +bio_for_each_bvec_all (bvec, bio, i) { + len += bvec->bv_len; +if (len > inode->i_sb->s_blocksize) +return true; + } }return false; } -/* -* This is a generic handler for readpage errors. If other copies exist, read -* those and write back good data to the failed position. Does not investigate -* in remapping the failed extent elsewhere, hoping the device will be smart -* enough to do this as needed -*/ -static int bio_readpage_error (struct bio *failed_bio,u64 phy_offset, -struct page *page,u64 start,u64 end, -int failed_mirror) +blk_status_t btrfs_submit_read_repair (struct inode *inode, +struct bio *failed_bio,u64 phy_offset, +struct page *page,unsigned int pgoff, +u64 start,u64 end,int failed_mirror, +submit_bio_hook_t *submit_bio_hook) {struct io_failure_record *failrec; -struct inode *inode = page->mapping->host; +struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_io_tree *tree = &BTRFS_I (inode)->io_tree;struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree; +struct btrfs_io_bio *failed_io_bio =btrfs_io_bio (failed_bio); +int icsum = phy_offset >> inode->i_sb->s_blocksize_bits;bool need_validation; -struct bio *bio; -int read_mode =0 ; +struct bio *repair_bio; +struct btrfs_io_bio *repair_io_bio;blk_status_t status;int ret; +btrfs_info (btrfs_sb (inode->i_sb), +"Repair Read Error: read error at %llu" , start); +BUG_ON (bio_op (failed_bio) ==REQ_OP_WRITE ); ret =btrfs_get_io_failure_record (inode, start, end, &failrec);if (ret) -return ret; +return errno_to_blk_status (ret); need_validation =btrfs_io_needs_validation (inode, failed_bio);if (!btrfs_check_repairable (inode, need_validation, failrec, failed_mirror)) {free_io_failure (failure_tree, tree, failrec); -return -EIO ; +return BLK_STS_IOERR ; } + repair_bio =btrfs_io_bio_alloc (1 ); + repair_io_bio =btrfs_io_bio (repair_bio); + repair_bio->bi_opf =REQ_OP_READ ;if (need_validation) - read_mode |=REQ_FAILFAST_DEV ; + repair_bio->bi_opf |=REQ_FAILFAST_DEV ; + repair_bio->bi_end_io = failed_bio->bi_end_io; + repair_bio->bi_iter.bi_sector = failrec->logical >>9 ; + repair_bio->bi_private = failed_bio->bi_private; - phy_offset >>= inode->i_sb->s_blocksize_bits; - bio =btrfs_create_repair_bio (inode, failed_bio, failrec, page, - start -page_offset (page), - (int )phy_offset, failed_bio->bi_end_io, -NULL ); - bio->bi_opf =REQ_OP_READ | read_mode; +if (failed_io_bio->csum) { +u16 csum_size =btrfs_super_csum_size (fs_info->super_copy); + + repair_io_bio->csum = repair_io_bio->csum_inline; +memcpy (repair_io_bio->csum, + failed_io_bio->csum + csum_size * icsum, csum_size); + } + +bio_add_page (repair_bio, page, failrec->len, pgoff); + repair_io_bio->logical = failrec->start; + repair_io_bio->iter = repair_bio->bi_iter;btrfs_debug (btrfs_sb (inode->i_sb), -"Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d" , - read_mode, failrec->this_mirror, failrec->in_validation); +"Repair Read Error: submitting new read to this_mirror=%d, in_validation=%d" , + failrec->this_mirror, failrec->in_validation); - status = tree->ops->submit_bio_hook (tree->private_data, bio, failrec->this_mirror, - failrec->bio_flags); + status =submit_bio_hook (inode, repair_bio, failrec->this_mirror, + failrec->bio_flags);if (status) {free_io_failure (failure_tree, tree, failrec); -bio_put (bio); - ret =blk_status_to_errno (status); +bio_put (repair_bio); } - -return ret; +return status; }/* lots and lots of room for performance fixes in the end_bio funcs */
@@ -2891,9 +2880,10 @@ static void end_bio_extent_readpage(struct bio *bio)* If it can't handle the error it will return -EIO and * we remain responsible for that page. */ - ret =bio_readpage_error (bio, offset, page, start, end, - mirror); -if (ret ==0 ) { +if (!btrfs_submit_read_repair (inode, bio, offset, page, + start -page_offset (page), + start, end, mirror, + tree->ops->submit_bio_hook)) { uptodate = !bio->bi_status; offset += len;continue ;
fs/btrfs/extent_io.h +11 -6
@@ -66,6 +66,10 @@ struct btrfs_io_bio;struct io_failure_record ;struct extent_io_tree ; +typedef blk_status_t (submit_bio_hook_t )(struct inode *inode,struct bio *bio, +int mirror_num, +unsigned long bio_flags); +typedef blk_status_t (extent_submit_bio_start_t )(void *private_data,struct bio *bio,u64 bio_offset);
@@ -74,8 +78,7 @@ struct extent_io_ops {* The following callbacks must be always defined, the function * pointer will be called unconditionally. */ -blk_status_t (*submit_bio_hook)(struct inode *inode,struct bio *bio, -int mirror_num,unsigned long bio_flags); +submit_bio_hook_t *submit_bio_hook;int (*readpage_end_io_hook)(struct btrfs_io_bio *io_bio,u64 phy_offset,struct page *page,u64 start,u64 end,int mirror);
@@ -312,10 +315,12 @@ struct io_failure_record { }; -struct bio *btrfs_create_repair_bio (struct inode *inode,struct bio *failed_bio, -struct io_failure_record *failrec, -struct page *page,int pg_offset,int icsum, -bio_end_io_t *endio_func,void *data); +blk_status_t btrfs_submit_read_repair (struct inode *inode, +struct bio *failed_bio,u64 phy_offset, +struct page *page,unsigned int pgoff, +u64 start,u64 end,int failed_mirror, +submit_bio_hook_t *submit_bio_hook); +#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS bool find_lock_delalloc_range(struct inode *inode, struct page *locked_page, u64 *start,
fs/btrfs/inode.c +11 -92
128 lines not on this page · raw message
Signature · 1 line
2.26.1
PATCH v2 13/15 btrfs: simplify direct I/O read repair
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Direct I/O read repair was originally implemented in commit 8b110e393c5a ("Btrfs: implement repair function when direct read fails"). This implementation is unnecessarily complicated. There is major code duplication between __btrfs_subio_endio_read() (checks checksums and handles I/O errors for files with checksums), __btrfs_correct_data_nocsum() (handles I/O errors for files without checksums), btrfs_retry_endio() (checks checksums and handles I/O errors for retries of files with checksums), and btrfs_retry_endio_nocsum() (handles I/O errors for retries of files without checksum). If it sounds like these should be one function, that's because they should. Additionally, these functions are very hard to follow due to their excessive use of goto.
This commit replaces the original implementation. After the previous commit getting rid of orig_bio, we can reuse the same endio callback for repair I/O and the original I/O, we just need to track the file offset and original iterator in the repair bio. We can also unify the handling of files with and without checksums and simplify the control flow. We also no longer have to wait for each repair I/O to complete one by one.
--- fs/btrfs/extent_io.c | 2 + fs/btrfs/inode.c | 268 +++++++------------------------------------ 2 files changed, 44 insertions(+), 226 deletions(-)
fs/btrfs/extent_io.c +2 -0
@@ -2631,6 +2631,8 @@ struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio, }bio_add_page (bio, page, failrec->len, pg_offset); +btrfs_io_bio (bio)->logical = failrec->start; +btrfs_io_bio (bio)->iter = bio->bi_iter;return bio; }
fs/btrfs/inode.c +42 -226
329 lines not on this page · raw message
Signature · 1 line
2.26.1
PATCH v2 14/15 btrfs: get rid of endio_repair_workers
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
This was originally added in commit 8b110e393c5a ("Btrfs: implement repair function when direct read fails") to avoid a deadlock. In that commit, the direct I/O read endio executes on the endio_workers workqueue, submits a repair bio, and waits for it to complete. The repair bio endio must execute on a different workqueue, otherwise it could block on the endio_workers workqueue becoming available, which won't happen because the original endio is blocked on the repair bio.
As of the previous commit, the original endio doesn't wait for the repair bio, so this separate workqueue is unnecessary.
--- fs/btrfs/ctree.h | 1 - fs/btrfs/disk-io.c | 8 +------- fs/btrfs/disk-io.h | 1 - fs/btrfs/inode.c | 2 +- 4 files changed, 2 insertions(+), 10 deletions(-)
fs/btrfs/ctree.h +0 -1
@@ -758,7 +758,6 @@ struct btrfs_fs_info {struct btrfs_workqueue *endio_workers;struct btrfs_workqueue *endio_meta_workers;struct btrfs_workqueue *endio_raid56_workers; -struct btrfs_workqueue *endio_repair_workers;struct btrfs_workqueue *rmw_workers;struct btrfs_workqueue *endio_meta_write_workers;struct btrfs_workqueue *endio_write_workers;
fs/btrfs/disk-io.c +1 -7
@@ -709,9 +709,7 @@ static void end_workqueue_bio(struct bio *bio)else wq = fs_info->endio_write_workers; }else { -if (unlikely (end_io_wq->metadata ==BTRFS_WQ_ENDIO_DIO_REPAIR )) - wq = fs_info->endio_repair_workers; -else if (end_io_wq->metadata ==BTRFS_WQ_ENDIO_RAID56 ) +if (end_io_wq->metadata ==BTRFS_WQ_ENDIO_RAID56 ) wq = fs_info->endio_raid56_workers;else if (end_io_wq->metadata) wq = fs_info->endio_meta_workers;
@@ -1942,7 +1940,6 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)btrfs_destroy_workqueue (fs_info->workers);btrfs_destroy_workqueue (fs_info->endio_workers);btrfs_destroy_workqueue (fs_info->endio_raid56_workers); -btrfs_destroy_workqueue (fs_info->endio_repair_workers);btrfs_destroy_workqueue (fs_info->rmw_workers);btrfs_destroy_workqueue (fs_info->endio_write_workers);btrfs_destroy_workqueue (fs_info->endio_freespace_worker);
@@ -2148,8 +2145,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info, fs_info->endio_raid56_workers =btrfs_alloc_workqueue (fs_info,"endio-raid56" , flags, max_active,4 ); - fs_info->endio_repair_workers = -btrfs_alloc_workqueue (fs_info,"endio-repair" , flags,1 ,0 ); fs_info->rmw_workers =btrfs_alloc_workqueue (fs_info,"rmw" , flags, max_active,2 ); fs_info->endio_write_workers =
@@ -2173,7 +2168,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info, fs_info->flush_workers && fs_info->endio_workers && fs_info->endio_meta_workers && fs_info->endio_meta_write_workers && - fs_info->endio_repair_workers && fs_info->endio_write_workers && fs_info->endio_raid56_workers && fs_info->endio_freespace_worker && fs_info->rmw_workers && fs_info->caching_workers && fs_info->readahead_workers &&
fs/btrfs/disk-io.h +0 -1
@@ -25,7 +25,6 @@ enum btrfs_wq_endio_type {BTRFS_WQ_ENDIO_METADATA ,BTRFS_WQ_ENDIO_FREE_SPACE , BTRFS_WQ_ENDIO_RAID56, - BTRFS_WQ_ENDIO_DIO_REPAIR, };static inline u64 btrfs_sb_offset (int mirror)
fs/btrfs/inode.c +1 -1
@@ -7388,7 +7388,7 @@ static inline blk_status_t submit_dio_repair_bio(struct inode *inode,BUG_ON (bio_op (bio) ==REQ_OP_WRITE ); - ret =btrfs_bio_wq_end_io (fs_info, bio,BTRFS_WQ_ENDIO_DIO_REPAIR ); + ret =btrfs_bio_wq_end_io (fs_info, bio,BTRFS_WQ_ENDIO_DATA );if (ret)return ret;
Signature · 1 line
2.26.1
Quoted 31 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Hi,
This is version 2 of my series of fixes, cleanups, and improvements to direct I/O and read repair. It's preparation for adding read repair to my
RWF_ENCODEDseries [1], but it can go in independently.Patch 1 adds a new bio helper needed for patch 4. Patches 2 and 3 are direct I/O error handling fixes. Patch 4 is a buffered read repair fix. Patch 5 is a buffered read repair improvement. Patches 6-9 are trivial cleanups. Patch 10 converts direct I/O to use
refcount_t, which would've helped catch the bug fixed by patch 2. Patches 11-14 drastically simplify the direct I/O code. Patch 15 unifies buffered and direct I/O read repair, which also makes direct I/O repair actually do validation for large failed reads instead of rewriting the whole thing.Overall, this is net about -350 lines of code and actually makes direct I/O more functional.
Note that this series causes btrfs/142 to fail. This is a bug in the test, as it assumes that direct I/O doesn't do read validation. This is fixed by the fstests patch "btrfs/14{2,3}: use dm-dust instead of
fail_make_request" which I sent up yesterday [2].Jens and Christoph are cc'd for patches 1 and 4. Instead of looking at the bio internals like I did in v1, I added a new
bio_for_each_bvec_all()helper.Changes from v1 [3]:
Quoted 12 lines
* Added patch 1 with
bio_for_each_bvec_all()helper
* Dropped patch 8 which moved struct definition
* Fixed performance regression [4] in patch 13 caused by accidentally making all direct I/O submission asynchronous
* Fixed uninitialized assert in patch 12
* Fixed misplaced assert inbtrfs_check_read_dio_bioin patch 13
* Added reviewed-bys
* Refactoredbtrfs_submit_direct()andbtrfs_submit_direct_hook()in patch 2 (I didn't add Nikolay's reviewed-by to that one because the new patch looks fairly different from patch 1 in v1)
* Rewrapped csum calculations in patch 10 for easier readability
* Clarified several commit messages and comments
Thanks, I haven't read the changes yet but fstests passed. There are some conflicts with the dio-iomap patches, all seem to be in the expected range and solvable.
Re: [PATCH v2 01/15] block: add bio_for_each_bvec_all()
In reply to Omar SandovalLooks good from my PoV
Re: [PATCH v2 02/15] btrfs: fix error handling when submitting direct I/O bio
In reply to Omar SandovalLooks good,
Re: [PATCH v2 03/15] btrfs: fix double __endio_write_update_ordered in direct I/O
In reply to Omar SandovalOn 16/04/2020 23:47, Omar Sandoval wrote:
[...]
fs/btrfs/inode.c static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, · line 7914
Quoted 6 lines
fs/btrfs/inode.c +4
-1+static struct btrfs_dio_private *btrfs_create_dio_private (struct bio *dio_bio, +struct inode *inode, +loff_t file_offset) { -struct inode *inode = dip->inode; +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE );
Nit: I think the braces aren't needed here.
[...]
fs/btrfs/inode.c static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, · line 7954
Quoted 4 lines
fs/btrfs/inode.c +4
-0+static void btrfs_submit_direct (struct bio *dio_bio,struct inode *inode, +loff_t file_offset) +{ +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE );
Same here
Anyways:
Re: [PATCH v2 04/15] btrfs: look at full bi_io_vec for repair decision
In reply to Omar SandovalLooks good,
Re: [PATCH v2 05/15] btrfs: don't do repair validation for checksum errors
In reply to Omar SandovalLooks good,
Re: [PATCH v2 06/15] btrfs: clarify btrfs_lookup_bio_sums documentation
In reply to Omar SandovalQuoted 6 lines · Omar Sandoval
Fix a couple of issues in the
btrfs_lookup_bio_sumsdocumentation:* The bio doesn't need to be a
btrfs_io_bioif dst was provided. Move the declaration in the code to make that clear, too.
* dst must be large enough to hold nblocks *csum_size, not justcsum_size.
And also reduce the scope of 'btrfs_bio'?
Otherwise,
Re: [PATCH v2 09/15] btrfs: kill btrfs_dio_private->private
In reply to Omar SandovalLooks good,
Re: [PATCH v2 10/15] btrfs: convert btrfs_dio_private->pending_bios to refcount_t
In reply to Omar SandovalEasy enough,
Re: [PATCH v2 11/15] btrfs: put direct I/O checksums in btrfs_dio_private instead of bio
In reply to Omar SandovalLooks good,
Re: [PATCH v2 03/15] btrfs: fix double __endio_write_update_ordered in direct I/O
In reply to Johannes ThumshirnQuoted 10 lines · Johannes Thumshirn
On 16/04/2020 23:47, Omar Sandoval wrote:
[...]fs/btrfs/inode.c static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, · line 7914
Quoted 6 lines
fs/btrfs/inode.c +4
-1+static struct btrfs_dio_private *btrfs_create_dio_private (struct bio *dio_bio, +struct inode *inode, +loff_t file_offset) { -struct inode *inode = dip->inode; +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE );Nit: I think the braces aren't needed here.
For readability I prefer to keep them, it makes the operator precedence obvious.
Re: [PATCH v2 03/15] btrfs: fix double __endio_write_update_ordered in direct I/O
In reply to Omar SandovalQuoted 12 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
In
btrfs_submit_direct(), if we fail to allocate thebtrfs_dio_private, we complete the ordered extent range. However, we don't mark that the range doesn't need to be cleaned up frombtrfs_direct_IO()until later. Therefore, if we fail to allocate thebtrfs_dio_private, we complete the ordered extent range twice. We could fix this by updatingunsubmitted_oe_rangeearlier, but it's cleaner to reorganize the code so that creating thebtrfs_dio_privateand submitting the bios are separate, and once thebtrfs_dio_privateis created, cleanup always happens through thebtrfs_dio_private.
Quoted 2 lines
Fixes f28a49287817 ("Btrfs: fix leaking of ordered extents after direct IO write error")Signed-off-by Omar Sandoval <[email protected]>
Generally looks ok, so :
below for some remarks on the logic around unsubmitted_oe_range_start/end
Quoted 40 lines
--- fs/btrfs/inode.c | 174 ++++++++++++++++++----------------------------- 1 file changed, 66 insertions(+), 108 deletions(-)fs/btrfs/inode.c +25
-2@@ -7903,14 +7903,60 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,return ret; } -static int btrfs_submit_direct_hook (struct btrfs_dio_private *dip) +/* +* If this succeeds, the btrfs_dio_private is responsible for cleaning up locked +* or ordered extents whether or not we submit any bios. +*/ +static struct btrfs_dio_private *btrfs_create_dio_private (struct bio *dio_bio, +struct inode *inode, +loff_t file_offset) { -struct inode *inode = dip->inode; +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE ); +struct btrfs_dio_private *dip; +struct bio *bio; + + dip =kzalloc (sizeof (*dip),GFP_NOFS ); +if (!dip) +return NULL ; + + bio =btrfs_bio_clone (dio_bio); + bio->bi_private = dip; +btrfs_io_bio (bio)->logical = file_offset; + + dip->private = dio_bio->bi_private; + dip->inode = inode; + dip->logical_offset = file_offset; + dip->bytes = dio_bio->bi_iter.bi_size; + dip->disk_bytenr = (u64 )dio_bio->bi_iter.bi_sector <<9 ; + dip->orig_bio = bio;
fs/btrfs/inode.c static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, · line 7941
Quoted 10 lines
fs/btrfs/inode.c +10
-0+ dip->dio_bio = dio_bio; +atomic_set (&dip->pending_bios,1 ); + +if (write) { +struct btrfs_dio_data *dio_data = current->journal_info; + + dio_data->unsubmitted_oe_range_end = dip->logical_offset + + dip->bytes; + dio_data->unsubmitted_oe_range_start = + dio_data->unsubmitted_oe_range_end;
The logic around those 2 members is really subtle. We really have the following:
1. btrfs_direct_IO sets those two to the same value.
2. When we call __blockdev_direct_IO unless btrfs_get_blocks_direct->btrfs_get_blocks_direct_write is called to modify unsubmitted_oe_range_start so that start < end. Cleanup won't happen.
3. We come into btrfs_submit_direct - if it dip allocation fails we'd return with oe_range_end now modified so cleanup will happen.
4. If we manage to allocate the dip we reset the unsubmitted range members to be equal so that cleanup happens from btrfs_endio_direct_write.
This 4-step logic is not really obvious, especially given it's scattered across 3 functions. Perhaps a comment saying that having the 2 members being equal means cleanup in btrfs_DIRECT_io is disabled.
I'd rather have it spelled out in the changelog, I guess David can also do that ?
Quoted 9 lines
fs/btrfs/inode.c +9
-0+ + bio->bi_end_io = btrfs_endio_direct_write; + }else { + bio->bi_end_io = btrfs_endio_direct_read; + dip->subio_endio = btrfs_subio_endio_read; + } +return dip; +} +
<snip>
Re: [PATCH v2 06/15] btrfs: clarify btrfs_lookup_bio_sums documentation
In reply to Omar SandovalQuoted 11 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Fix a couple of issues in the
btrfs_lookup_bio_sumsdocumentation:* The bio doesn't need to be a
btrfs_io_bioif dst was provided. Move the declaration in the code to make that clear, too.
* dst must be large enough to hold nblocks *csum_size, not justcsum_size.Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>
Re: [PATCH v2 12/15] btrfs: get rid of one layer of bios in direct I/O
In reply to Omar SandovalQuoted 13 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
In the worst case, there are _4_ layers of bios in the Btrfs direct I/O path:
1. The bio created by the generic direct I/O code (
dio_bio).
2. A clone ofdio_biowe create inbtrfs_submit_direct()to represent the entire direct I/O range (orig_bio).
3. A partial clone oforig_biolimited to the size of a RAID stripe that we create inbtrfs_submit_direct_hook().
4. Clones of each of those split bios for each RAID stripe that we create inbtrfs_map_bio().
Quoted 6 lines
As of the previous commit, the second layer (
orig_bio) is no longer needed for anything: we can splitdio_bioinstead, and completedio_biodirectly when all of the cloned bios complete. This lets us clean up a bunch of cruft, includingdip->subio_endioanddip->errors(we can usedio_bio->bi_statusinstead). It also enables the next big cleanup of direct I/O read repair.
nit: Please explicitly mention that btrfs_dio_private_put is now not only putting a structure and doing cleanups but also serves as the io completion routine for dio_bio. Given the name of the function and its purpose I find this a bit counter-intuitive. But since this is rather subjective I'd like to ask David if he also sees it as a bit surprising?
Quoted 8 lines
Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/btrfs_inode.h | 16 ---- fs/btrfs/inode.c | 185 +++++++++++++++-------------------------- 2 files changed, 65 insertions(+), 136 deletions(-)
<snip>
fs/btrfs/inode.c static int btrfs_get_blocks_direct(struct inode *inode, sector_t iblock, · line 7360
Quoted 10 lines
fs/btrfs/inode.c +2
-0@@ -7356,6 +7356,29 @@ static int btrfs_get_blocks_direct(struct inode *inode, sector_t iblock,return ret; } +static void btrfs_dio_private_put (struct btrfs_dio_private *dip) +{
The way you've structured the code is proliferating something which has been identified as bad practice, namely - asymmetry between "refcount get" and "refcount put" operations. The put is nicely encapsulated behind an aptly named function, however getting the reference is really an open-coded refcount_inc. This has lead to at least 1 bug in the past (recently fixed by Filipe) since transaction's refcount management is similar. So I'd rather have the refcount_inc(dip->refs) be encapsulated behind a simple btrfs_dio_private_get() helper.
Quoted 24 lines
fs/btrfs/inode.c +21
-0+/* +* This implies a barrier so that stores to dio_bio->bi_status before +* this and loads of dio_bio->bi_status after this are fully ordered. +*/ +if (!refcount_dec_and_test (&dip->refs)) +return ; + +if (bio_op (dip->dio_bio) ==REQ_OP_WRITE ) { +__endio_write_update_ordered (dip->inode, dip->logical_offset, + dip->bytes, + !dip->dio_bio->bi_status); + }else { +unlock_extent (&BTRFS_I (dip->inode)->io_tree, + dip->logical_offset, + dip->logical_offset + dip->bytes -1 ); + } + +dio_end_io (dip->dio_bio); +kfree (dip); +} +static inline blk_status_t submit_dio_repair_bio (struct inode *inode,struct bio *bio,int mirror_num)
<snip>
Re: [PATCH v2 13/15] btrfs: simplify direct I/O read repair
In reply to Omar SandovalQuoted 170 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Direct I/O read repair was originally implemented in commit
8b110e393c5a("Btrfs: implement repair function when direct read fails"). This implementation is unnecessarily complicated. There is major code duplication between__btrfs_subio_endio_read()(checks checksums and handles I/O errors for files with checksums),__btrfs_correct_data_nocsum()(handles I/O errors for files without checksums),btrfs_retry_endio()(checks checksums and handles I/O errors for retries of files with checksums), andbtrfs_retry_endio_nocsum()(handles I/O errors for retries of files without checksum). If it sounds like these should be one function, that's because they should. Additionally, these functions are very hard to follow due to their excessive use of goto.This commit replaces the original implementation. After the previous commit getting rid of
orig_bio, we can reuse the same endio callback for repair I/O and the original I/O, we just need to track the file offset and original iterator in the repair bio. We can also unify the handling of files with and without checksums and simplify the control flow. We also no longer have to wait for each repair I/O to complete one by one.Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/extent_io.c | 2 + fs/btrfs/inode.c | 268 +++++++------------------------------------ 2 files changed, 44 insertions(+), 226 deletions(-)fs/btrfs/extent_io.c +2
-0@@ -2631,6 +2631,8 @@ struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio, }bio_add_page (bio, page, failrec->len, pg_offset); +btrfs_io_bio (bio)->logical = failrec->start; +btrfs_io_bio (bio)->iter = bio->bi_iter;return bio; }fs/btrfs/inode.c +19
-59@@ -7435,19 +7435,17 @@ static int btrfs_check_dio_repairable(struct inode *inode,static blk_status_t dio_read_error (struct inode *inode,struct bio *failed_bio,struct page *page,unsigned int pgoff, -u64 start,u64 end,int failed_mirror, -bio_end_io_t *repair_endio,void *repair_arg) +u64 start,u64 end,int failed_mirror) { +struct btrfs_dio_private *dip = failed_bio->bi_private;struct io_failure_record *failrec;struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree;struct bio *bio;int isector;unsigned int read_mode =0 ; -int segs;int ret;blk_status_t status; -struct bio_vec bvec;BUG_ON (bio_op (failed_bio) ==REQ_OP_WRITE );@@ -7462,261 +7460,79 @@ static blk_status_t dio_read_error(struct inode *inode, struct bio *failed_bio,return BLK_STS_IOERR ; } - segs =bio_segments (failed_bio); -bio_get_first_bvec (failed_bio, &bvec); -if (segs >1 || - (bvec.bv_len >btrfs_inode_sectorsize (inode))) +if (btrfs_io_bio (failed_bio)->iter.bi_size > inode->i_sb->s_blocksize) read_mode |=REQ_FAILFAST_DEV ; isector = start -btrfs_io_bio (failed_bio)->logical; isector >>= inode->i_sb->s_blocksize_bits; - bio =btrfs_create_repair_bio (inode, failed_bio, failrec, page, - pgoff, isector, repair_endio, repair_arg); + bio =btrfs_create_repair_bio (inode, failed_bio, failrec, page, pgoff, + isector, failed_bio->bi_end_io, dip); bio->bi_opf =REQ_OP_READ | read_mode;btrfs_debug (BTRFS_I (inode)->root->fs_info,"repair DIO read error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d" , read_mode, failrec->this_mirror, failrec->in_validation); +refcount_inc (&dip->refs); status =submit_dio_repair_bio (inode, bio, failrec->this_mirror);if (status) {free_io_failure (failure_tree, io_tree, failrec);bio_put (bio); +refcount_dec (&dip->refs); }return status; } -struct btrfs_retry_complete { -struct completion done; -struct inode *inode; -u64 start; -int uptodate; -}; - -static void btrfs_retry_endio_nocsum (struct bio *bio) -{ -struct btrfs_retry_complete *done = bio->bi_private; -struct inode *inode = done->inode; -struct bio_vec *bvec; -struct extent_io_tree *io_tree, *failure_tree; -struct bvec_iter_all iter_all; - -if (bio->bi_status) - goto end; - -ASSERT (bio->bi_vcnt ==1 ); - io_tree = &BTRFS_I (inode)->io_tree; - failure_tree = &BTRFS_I (inode)->io_failure_tree; -ASSERT (bio_first_bvec_all (bio)->bv_len ==btrfs_inode_sectorsize (inode)); - - done->uptodate =1 ; -ASSERT (!bio_flagged (bio,BIO_CLONED )); -bio_for_each_segment_all (bvec, bio, iter_all) -clean_io_failure (BTRFS_I (inode)->root->fs_info, failure_tree, - io_tree, done->start, bvec->bv_page, -btrfs_ino (BTRFS_I (inode)),0 ); -end: -complete (&done->done); -bio_put (bio); -} - -static blk_status_t __btrfs_correct_data_nocsum (struct inode *inode, -struct btrfs_io_bio *io_bio) +static blk_status_t btrfs_check_read_dio_bio (struct inode *inode, +struct btrfs_io_bio *io_bio, +const bool uptodate) { -struct btrfs_fs_info *fs_info; +struct btrfs_fs_info *fs_info =BTRFS_I (inode)->root->fs_info; +u32 sectorsize = fs_info->sectorsize; +struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree; +struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree; +const bool csum = !(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM );struct bio_vec bvec;struct bvec_iter iter; -struct btrfs_retry_complete done; -u64 start; -unsigned int pgoff; -u32 sectorsize; -int nr_sectors; -blk_status_t ret; +u64 start = io_bio->logical; +int icsum =0 ;blk_status_t err =BLK_STS_OK ; - fs_info =BTRFS_I (inode)->root->fs_info; - sectorsize = fs_info->sectorsize; - - start = io_bio->logical; - done.inode = inode; - io_bio->bio.bi_iter = io_bio->iter; +__bio_for_each_segment (bvec, &io_bio->bio, iter, io_bio->iter) { +unsigned int i, nr_sectors, pgoff; nr_sectors =BTRFS_BYTES_TO_BLKS (fs_info, bvec.bv_len); pgoff = bvec.bv_offset;
<snip>
Quoted 2 lines
fs/btrfs/inode.c +1
-0+for (i =0 ; i < nr_sectors; i++) {ASSERT (pgoff <PAGE_SIZE );
<snip>
fs/btrfs/inode.c static blk_status_t dio_read_error(struct inode *inode, struct bio *failed_bio, · line 7516
Quoted 8 lines
fs/btrfs/inode.c +8
-0+if (uptodate && + (!csum || !check_data_csum (inode, io_bio, icsum, + bvec.bv_page, pgoff, + start, sectorsize))) { +clean_io_failure (fs_info, failure_tree, io_tree, + start, bvec.bv_page, +btrfs_ino (BTRFS_I (inode)), + pgoff);
Why do you have to call clean_io_failure in case there isn't a failure since that function:
a) Will return because the tree is empty altogether (count_range_bits returns 0).
b) Or if there was a corruption a different segment - get_state_failrec would return -ENOENT.
Can't this be reworked to:
if (!uptodate || (csum && check_data_csum)) {dio_read_error
}
Quoted 21 lines
fs/btrfs/inode.c +13
-3+ }else { +blk_status_t status; + + status =dio_read_error (inode, &io_bio->bio, + bvec.bv_page, pgoff, + start, + start + sectorsize -1 , + io_bio->mirror_num); +if (status) + err = status; + } + start += sectorsize; + icsum++; pgoff += sectorsize; -ASSERT (pgoff <PAGE_SIZE ); - goto next_block; } } -return err; }
<snip>
Nikolay Borisov
Reviewed-byRe: [PATCH v2 13/15] btrfs: simplify direct I/O read repair
Quoted 192 lines · Nikolay Borisov
Quoted 170 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Direct I/O read repair was originally implemented in commit
8b110e393c5a("Btrfs: implement repair function when direct read fails"). This implementation is unnecessarily complicated. There is major code duplication between__btrfs_subio_endio_read()(checks checksums and handles I/O errors for files with checksums),__btrfs_correct_data_nocsum()(handles I/O errors for files without checksums),btrfs_retry_endio()(checks checksums and handles I/O errors for retries of files with checksums), andbtrfs_retry_endio_nocsum()(handles I/O errors for retries of files without checksum). If it sounds like these should be one function, that's because they should. Additionally, these functions are very hard to follow due to their excessive use of goto.This commit replaces the original implementation. After the previous commit getting rid of
orig_bio, we can reuse the same endio callback for repair I/O and the original I/O, we just need to track the file offset and original iterator in the repair bio. We can also unify the handling of files with and without checksums and simplify the control flow. We also no longer have to wait for each repair I/O to complete one by one.Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/extent_io.c | 2 + fs/btrfs/inode.c | 268 +++++++------------------------------------ 2 files changed, 44 insertions(+), 226 deletions(-)fs/btrfs/extent_io.c +2
-0@@ -2631,6 +2631,8 @@ struct bio *btrfs_create_repair_bio(struct inode *inode, struct bio *failed_bio, }bio_add_page (bio, page, failrec->len, pg_offset); +btrfs_io_bio (bio)->logical = failrec->start; +btrfs_io_bio (bio)->iter = bio->bi_iter;return bio; }fs/btrfs/inode.c +19
-59@@ -7435,19 +7435,17 @@ static int btrfs_check_dio_repairable(struct inode *inode,static blk_status_t dio_read_error (struct inode *inode,struct bio *failed_bio,struct page *page,unsigned int pgoff, -u64 start,u64 end,int failed_mirror, -bio_end_io_t *repair_endio,void *repair_arg) +u64 start,u64 end,int failed_mirror) { +struct btrfs_dio_private *dip = failed_bio->bi_private;struct io_failure_record *failrec;struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree;struct bio *bio;int isector;unsigned int read_mode =0 ; -int segs;int ret;blk_status_t status; -struct bio_vec bvec;BUG_ON (bio_op (failed_bio) ==REQ_OP_WRITE );@@ -7462,261 +7460,79 @@ static blk_status_t dio_read_error(struct inode *inode, struct bio *failed_bio,return BLK_STS_IOERR ; } - segs =bio_segments (failed_bio); -bio_get_first_bvec (failed_bio, &bvec); -if (segs >1 || - (bvec.bv_len >btrfs_inode_sectorsize (inode))) +if (btrfs_io_bio (failed_bio)->iter.bi_size > inode->i_sb->s_blocksize) read_mode |=REQ_FAILFAST_DEV ; isector = start -btrfs_io_bio (failed_bio)->logical; isector >>= inode->i_sb->s_blocksize_bits; - bio =btrfs_create_repair_bio (inode, failed_bio, failrec, page, - pgoff, isector, repair_endio, repair_arg); + bio =btrfs_create_repair_bio (inode, failed_bio, failrec, page, pgoff, + isector, failed_bio->bi_end_io, dip); bio->bi_opf =REQ_OP_READ | read_mode;btrfs_debug (BTRFS_I (inode)->root->fs_info,"repair DIO read error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d" , read_mode, failrec->this_mirror, failrec->in_validation); +refcount_inc (&dip->refs); status =submit_dio_repair_bio (inode, bio, failrec->this_mirror);if (status) {free_io_failure (failure_tree, io_tree, failrec);bio_put (bio); +refcount_dec (&dip->refs); }return status; } -struct btrfs_retry_complete { -struct completion done; -struct inode *inode; -u64 start; -int uptodate; -}; - -static void btrfs_retry_endio_nocsum (struct bio *bio) -{ -struct btrfs_retry_complete *done = bio->bi_private; -struct inode *inode = done->inode; -struct bio_vec *bvec; -struct extent_io_tree *io_tree, *failure_tree; -struct bvec_iter_all iter_all; - -if (bio->bi_status) - goto end; - -ASSERT (bio->bi_vcnt ==1 ); - io_tree = &BTRFS_I (inode)->io_tree; - failure_tree = &BTRFS_I (inode)->io_failure_tree; -ASSERT (bio_first_bvec_all (bio)->bv_len ==btrfs_inode_sectorsize (inode)); - - done->uptodate =1 ; -ASSERT (!bio_flagged (bio,BIO_CLONED )); -bio_for_each_segment_all (bvec, bio, iter_all) -clean_io_failure (BTRFS_I (inode)->root->fs_info, failure_tree, - io_tree, done->start, bvec->bv_page, -btrfs_ino (BTRFS_I (inode)),0 ); -end: -complete (&done->done); -bio_put (bio); -} - -static blk_status_t __btrfs_correct_data_nocsum (struct inode *inode, -struct btrfs_io_bio *io_bio) +static blk_status_t btrfs_check_read_dio_bio (struct inode *inode, +struct btrfs_io_bio *io_bio, +const bool uptodate) { -struct btrfs_fs_info *fs_info; +struct btrfs_fs_info *fs_info =BTRFS_I (inode)->root->fs_info; +u32 sectorsize = fs_info->sectorsize; +struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree; +struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree; +const bool csum = !(BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM );struct bio_vec bvec;struct bvec_iter iter; -struct btrfs_retry_complete done; -u64 start; -unsigned int pgoff; -u32 sectorsize; -int nr_sectors; -blk_status_t ret; +u64 start = io_bio->logical; +int icsum =0 ;blk_status_t err =BLK_STS_OK ; - fs_info =BTRFS_I (inode)->root->fs_info; - sectorsize = fs_info->sectorsize; - - start = io_bio->logical; - done.inode = inode; - io_bio->bio.bi_iter = io_bio->iter; +__bio_for_each_segment (bvec, &io_bio->bio, iter, io_bio->iter) { +unsigned int i, nr_sectors, pgoff; nr_sectors =BTRFS_BYTES_TO_BLKS (fs_info, bvec.bv_len); pgoff = bvec.bv_offset;<snip>
Quoted 2 lines
fs/btrfs/inode.c +1
-0+for (i =0 ; i < nr_sectors; i++) {ASSERT (pgoff <PAGE_SIZE );<snip>
fs/btrfs/inode.c static blk_status_t dio_read_error(struct inode *inode, struct bio *failed_bio, · line 7516
Quoted 8 lines
fs/btrfs/inode.c +8
-0+if (uptodate && + (!csum || !check_data_csum (inode, io_bio, icsum, + bvec.bv_page, pgoff, + start, sectorsize))) { +clean_io_failure (fs_info, failure_tree, io_tree, + start, bvec.bv_page, +btrfs_ino (BTRFS_I (inode)), + pgoff);Why do you have to call
clean_io_failurein case there isn't a failure since that function:a) Will return because the tree is empty altogether (
count_range_bitsreturns 0).
Quoted 8 lines
b) Or if there was a corruption a different segment -
get_state_failrecwould return-ENOENT.Can't this be reworked to:
if (!uptodate || (csum &&
check_data_csum)) {dio_read_error
}
Thinking about it a bit more this seems to handle the case when we have failed one read and subsequently queued a repair bio. When this repair bio finishes we'll execute the clean_ioo_failure to remove the failure rec created initially. In this case this patch LGTM, so:
Quoted 25 lines
Quoted 22 lines
fs/btrfs/inode.c +13
-3+ }else { +blk_status_t status; + + status =dio_read_error (inode, &io_bio->bio, + bvec.bv_page, pgoff, + start, + start + sectorsize -1 , + io_bio->mirror_num); +if (status) + err = status; + } + start += sectorsize; + icsum++; pgoff += sectorsize; -ASSERT (pgoff <PAGE_SIZE ); - goto next_block; } } -return err; }<snip>
Re: [PATCH v2 03/15] btrfs: fix double __endio_write_update_ordered in direct I/O
In reply to Nikolay BorisovQuoted 90 lines · Nikolay Borisov
Quoted 12 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
In
btrfs_submit_direct(), if we fail to allocate thebtrfs_dio_private, we complete the ordered extent range. However, we don't mark that the range doesn't need to be cleaned up frombtrfs_direct_IO()until later. Therefore, if we fail to allocate thebtrfs_dio_private, we complete the ordered extent range twice. We could fix this by updatingunsubmitted_oe_rangeearlier, but it's cleaner to reorganize the code so that creating thebtrfs_dio_privateand submitting the bios are separate, and once thebtrfs_dio_privateis created, cleanup always happens through thebtrfs_dio_private.Quoted 2 lines
Fixes f28a49287817 ("Btrfs: fix leaking of ordered extents after direct IO write error")Signed-off-by Omar Sandoval <[email protected]>Generally looks ok, so :
below for some remarks on the logic around unsubmitted_oe_range_start/end
Quoted 40 lines
--- fs/btrfs/inode.c | 174 ++++++++++++++++++----------------------------- 1 file changed, 66 insertions(+), 108 deletions(-)fs/btrfs/inode.c +25
-2@@ -7903,14 +7903,60 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,return ret; } -static int btrfs_submit_direct_hook (struct btrfs_dio_private *dip) +/* +* If this succeeds, the btrfs_dio_private is responsible for cleaning up locked +* or ordered extents whether or not we submit any bios. +*/ +static struct btrfs_dio_private *btrfs_create_dio_private (struct bio *dio_bio, +struct inode *inode, +loff_t file_offset) { -struct inode *inode = dip->inode; +const bool write = (bio_op (dio_bio) ==REQ_OP_WRITE ); +struct btrfs_dio_private *dip; +struct bio *bio; + + dip =kzalloc (sizeof (*dip),GFP_NOFS ); +if (!dip) +return NULL ; + + bio =btrfs_bio_clone (dio_bio); + bio->bi_private = dip; +btrfs_io_bio (bio)->logical = file_offset; + + dip->private = dio_bio->bi_private; + dip->inode = inode; + dip->logical_offset = file_offset; + dip->bytes = dio_bio->bi_iter.bi_size; + dip->disk_bytenr = (u64 )dio_bio->bi_iter.bi_sector <<9 ; + dip->orig_bio = bio;fs/btrfs/inode.c static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, · line 7941
Quoted 10 lines
fs/btrfs/inode.c +10
-0+ dip->dio_bio = dio_bio; +atomic_set (&dip->pending_bios,1 ); + +if (write) { +struct btrfs_dio_data *dio_data = current->journal_info; + + dio_data->unsubmitted_oe_range_end = dip->logical_offset + + dip->bytes; + dio_data->unsubmitted_oe_range_start = + dio_data->unsubmitted_oe_range_end;The logic around those 2 members is really subtle. We really have the following:
1.
btrfs_direct_IOsets those two to the same value.2. When we call
__blockdev_direct_IOunlessbtrfs_get_blocks_direct->btrfs_get_blocks_direct_writeis called to modifyunsubmitted_oe_range_startso that start < end. Cleanup won't happen.3. We come into
btrfs_submit_direct- if it dip allocation fails we'd return withoe_range_endnow modified so cleanup will happen.4. If we manage to allocate the dip we reset the unsubmitted range members to be equal so that cleanup happens from
btrfs_endio_direct_write.
Quoted 6 lines
This 4-step logic is not really obvious, especially given it's scattered across 3 functions. Perhaps a comment saying that having the 2 members being equal means cleanup in
btrfs_DIRECT_iois disabled.I'd rather have it spelled out in the changelog, I guess David can also do that ?
The above added to changelog and a brief comment added, thanks.
Re: [PATCH v2 11/15] btrfs: put direct I/O checksums in btrfs_dio_private instead of bio
In reply to Omar SandovalQuoted 27 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
The next commit will get rid of
btrfs_dio_private->orig_bio. The only thing we really need it for is containing all of the checksums, but we can easily put the checksum array inbtrfs_dio_privateand have the submitted bios reference the array. We can also look the checksums up while we're setting up instead of the current awkward logic that looks them up fororig_biowhen the first split bio is submitted.(Interestingly,
btrfs_dio_privatedid contain the checksums before commit23ea8e5a0767("Btrfs: load checksum data once when submitting a direct read io"), but it didn't look them up up front.)Reviewed-by Nikolay Borisov <[email protected]>Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/btrfs_inode.h | 3 ++ fs/btrfs/inode.c | 70 +++++++++++++++++++----------------------- 2 files changed, 34 insertions(+), 39 deletions(-)fs/btrfs/btrfs_inode.h +0
-0
fs/btrfs/btrfs_inode.h struct btrfs_dio_private { · line 329
Quoted 6 lines
fs/btrfs/btrfs_inode.h +3
-0@@ -324,6 +324,9 @@ struct btrfs_dio_private {*/ blk_status_t (*subio_endio)(struct inode *,struct btrfs_io_bio *,blk_status_t ); + +/* Checksums. */ +u8 sums[];
I've renamed it to 'csums'
Re: [PATCH v2 12/15] btrfs: get rid of one layer of bios in direct I/O
In reply to Nikolay BorisovQuoted 21 lines · Nikolay Borisov
Quoted 13 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
In the worst case, there are _4_ layers of bios in the Btrfs direct I/O path:
1. The bio created by the generic direct I/O code (
dio_bio).
2. A clone ofdio_biowe create inbtrfs_submit_direct()to represent the entire direct I/O range (orig_bio).
3. A partial clone oforig_biolimited to the size of a RAID stripe that we create inbtrfs_submit_direct_hook().
4. Clones of each of those split bios for each RAID stripe that we create inbtrfs_map_bio().Quoted 6 lines
As of the previous commit, the second layer (
orig_bio) is no longer needed for anything: we can splitdio_bioinstead, and completedio_biodirectly when all of the cloned bios complete. This lets us clean up a bunch of cruft, includingdip->subio_endioanddip->errors(we can usedio_bio->bi_statusinstead). It also enables the next big cleanup of direct I/O read repair.
Quoted 5 lines
nit: Please explicitly mention that
btrfs_dio_private_putis now not only putting a structure and doing cleanups but also serves as the io completion routine fordio_bio. Given the name of the function and its purpose I find this a bit counter-intuitive. But since this is rather subjective I'd like to ask David if he also sees it as a bit surprising?
I tend to agree that mixing put and end_io together is counter intuitive, but I don't se a way how to separate them.
Quoted 19 lines
Quoted 8 lines
Reviewed-by Josef Bacik <[email protected]>Signed-off-by Omar Sandoval <[email protected]>--- fs/btrfs/btrfs_inode.h | 16 ---- fs/btrfs/inode.c | 185 +++++++++++++++-------------------------- 2 files changed, 65 insertions(+), 136 deletions(-)<snip>
Quoted 8 lines
fs/btrfs/inode.c +0
-0@@ -7356,6 +7356,29 @@ static int btrfs_get_blocks_direct(struct inode *inode, sector_t iblock,return ret; }
Quoted 11 lines
fs/btrfs/inode.c static int btrfs_get_blocks_direct(struct inode *inode, sector_t iblock, · line 7360
Quoted 2 lines
fs/btrfs/inode.c +2
-0+static void btrfs_dio_private_put (struct btrfs_dio_private *dip) +{The way you've structured the code is proliferating something which has been identified as bad practice, namely - asymmetry between "refcount get" and "refcount put" operations. The put is nicely encapsulated behind an aptly named function, however getting the reference is really an open-coded
refcount_inc. This has lead to at least 1 bug in the past (recently fixed by Filipe) since transaction's refcount management is similar. So I'd rather have therefcount_inc(dip->refs)be encapsulated behind a simplebtrfs_dio_private_get()helper.
btrfs_dio_private_get for refcount_inc would be ok, but I'm not sure if you also want the put+end_io for btrfs_dio_private_put.
Eg. in submit_dio_repair_bio, there's plain refcount_inc/refcount_dec, here the end_io semantics is not expected (and won't probably happen).
In btrfs_submit_direct there's the conditional inc/dec, when transfering the 1st reference from the init.
Suggestions welcome.
Re: [PATCH v2 15/15] btrfs: unify buffered and direct I/O read repair
In reply to Omar SandovalQuoted 13 lines · Omar Sandoval
fs/btrfs/extent_io.c +8
-1+blk_status_t btrfs_submit_read_repair (struct inode *inode, +struct bio *failed_bio,u64 phy_offset, +struct page *page,unsigned int pgoff, +u64 start,u64 end,int failed_mirror, +submit_bio_hook_t *submit_bio_hook) {struct io_failure_record *failrec; -struct inode *inode = page->mapping->host; +struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_io_tree *tree = &BTRFS_I (inode)->io_tree;struct extent_io_tree *failure_tree = &BTRFS_I (inode)->io_failure_tree; +struct btrfs_io_bio *failed_io_bio =btrfs_io_bio (failed_bio); +int icsum = phy_offset >> inode->i_sb->s_blocksize_bits;
fs/btrfs/extent_io.c static bool btrfs_io_needs_validation(struct inode *inode, struct bio *bio) · line 2664
Quoted 10 lines
fs/btrfs/extent_io.c +4
-2bool need_validation; -struct bio *bio; -int read_mode =0 ; +struct bio *repair_bio; +struct btrfs_io_bio *repair_io_bio;blk_status_t status;int ret; +btrfs_info (btrfs_sb (inode->i_sb), +"Repair Read Error: read error at %llu" , start);
This is a new message, there's another one with 'repair read error' but at 'debug' level, so I'd set that one to debug too. Or maybe drop it altogether, we want to use the uevents mechanism to communicate such things to user and not flood the system log.
Quoted 9 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Omar Sandoval (15):
block: addbio_for_each_bvec_all()
btrfs: fix error handling when submitting direct I/O bio
btrfs: fix double__endio_write_update_orderedin direct I/O
btrfs: look at fullbi_io_vecfor repair decision
btrfs: don't do repair validation for checksum errors
btrfs: clarifybtrfs_lookup_bio_sumsdocumentation
btrfs: rename__readpage_endio_checktocheck_data_csum
Quoted 8 lines
btrfs: make
btrfs_check_repairable()static
btrfs: killbtrfs_dio_private->private
btrfs: convertbtrfs_dio_private->pending_biostorefcount_t
btrfs: put direct I/O checksums inbtrfs_dio_privateinstead of bio
btrfs: get rid of one layer of bios in direct I/O
btrfs: simplify direct I/O read repair
btrfs: get rid ofendio_repair_workers
btrfs: unify buffered and direct I/O read repair
Thanks to all who did the reviews, I'm adding the patchset to misc-next. There are some minor updates, in changelogs or in code. There are also some comments that might lead to more fixups but I don't think it's too serious to hold the patches unmerged.
If there are futher comments or requests for clarification, changelog updates, please let me know, I'll do that directly. We need to give it a test and also to provide a base for the dio-iomap patchset.