Fix error handling on data bio submission
cover Fix error handling on data bio submission
Internally we tried to enable a remediation to automatically re-provision machines that had checksum errors. This was based on dmesg scanning, however we discovered that we were getting transienty csum error messages. This came down to getting a transient ENOMEM while trying to lookup checksums while doing a data read (this was on memory constrained containers).
What we were doing was simply acting like there wasn't a checksum there, which would print a scary message about missing checksums. And then we'd do the read, but because we didn't have a checksum we'd complain about a checksum mismatch. Neither of these things were actually what was happening, we simply got an EIO while looking up the checksums.
Fix this by properly returning an error and erroring out the BIO with the correct error. This is actually correct, it allows us to skip the IO and also not erroneously tell the user that their checksums are invalid.
While testing this fix however I uncovered a variety of problems with our error handling when we submit. So the first two patches are to fix the main problem I wanted to fix, and the next 6 are to fix problems that happen when injecting errors into the checksum lookup path.
With these patches I'm no longer getting csum mismatch errors when I fail to lookup csums, and I'm also able to survive a xfstests run while randomly injecting errors into this path. Thanks,
Josef
Josef Bacik (8):
btrfs: make search_csum_tree return 0 if we get -EFBIG
btrfs: handle csum lookup errors properly on reads
btrfs: check correct bio in finish_compressed_bio_read
btrfs: remove the bio argument from finish_compressed_bio_read
btrfs: track compressed bio errors as blk_status_t
btrfs: do not double complete bio on errors during compressed reads
btrfs: do not try to repair bio that has no mirror set
btrfs: do not clean up repair bio if submit fails
fs/btrfs/compression.c | 50 +++++++++++++++++++++++------------------- fs/btrfs/compression.h | 2 +- fs/btrfs/extent_io.c | 25 ++++++++++++++------- fs/btrfs/file-item.c | 15 ++++++++++--- fs/btrfs/inode.c | 12 ++++++---- 5 files changed, 65 insertions(+), 39 deletions(-)
Signature · 1 line
2.26.3
PATCH 1/8 btrfs: make search_csum_tree return 0 if we get -EFBIG
We can either fail to find a csum entry at all and return -ENOENT, or we can find a range that is close, but return -EFBIG. In essence these both mean the same thing when we are doing a lookup for a csum in an existing range, we didn't find a csum. We want to treat both of these errors the same way, complain loudly that there wasn't a csum. This currently happens anyway because we do
count = search_csum_tree();
if (count <= 0) {
// reloc and error handling
}however it forces us to incorrectly treat EIO or ENOMEM errors as on disk corruption. Fix this by returning 0 if we get either -ENOENT or -EFBIG from btrfs_lookup_csum() so we can do proper error handling.
--- fs/btrfs/file-item.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/file-item.c +1 -1
@@ -305,7 +305,7 @@ static int search_csum_tree(struct btrfs_fs_info *fs_info,read_extent_buffer (path->nodes[0 ], dst, (unsigned long )item, ret * csum_size); out: -if (ret == -ENOENT ) +if (ret == -ENOENT || ret == -EFBIG ) ret =0 ;return ret; }
Signature · 1 line
2.26.3
Re: [PATCH 1/8] btrfs: make search_csum_tree return 0 if we get -EFBIG
Quoted 12 lines · Josef Bacik
We can either fail to find a csum entry at all and return
-ENOENT, or we can find a range that is close, but return-EFBIG. In essence these both mean the same thing when we are doing a lookup for a csum in an existing range, we didn't find a csum. We want to treat both of these errors the same way, complain loudly that there wasn't a csum. This currently happens anyway because we docount = search_csum_tree(); if (count <= 0) { // reloc and error handling }
Quoted 5 lines
however it forces us to incorrectly treat EIO or ENOMEM errors as on disk corruption. Fix this by returning 0 if we get either
-ENOENTor-EFBIGfrombtrfs_lookup_csum()so we can do proper error handling.Signed-off-by Josef Bacik <[email protected]>
Quoted 20 lines
--- fs/btrfs/file-item.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/btrfs/file-item.c +1
-1@@ -305,7 +305,7 @@ static int search_csum_tree(struct btrfs_fs_info *fs_info,read_extent_buffer (path->nodes[0 ], dst, (unsigned long )item, ret * csum_size); out: -if (ret == -ENOENT ) +if (ret == -ENOENT || ret == -EFBIG ) ret =0 ;return ret; }Signature · 1 line
2.26.3
Re: [PATCH 1/8] btrfs: make search_csum_tree return 0 if we get -EFBIG
In reply to Josef BacikLooks good,
PATCH 2/8 btrfs: handle csum lookup errors properly on reads
In reply to Josef BacikCurrently any error we get while trying to lookup csums during reads shows up as a missing csum, and then on the read completion side we spit out an error saying there was a csum mismatch and we increase the device corruption count.
However we could have gotten an EIO from the lookup. We could also be inside of a memory constrained container and gotten a ENOMEM while trying to do the read. In either case we don't want to make this look like a file system corruption problem, we want to make it look like the actual error it is. Capture any negative value, convert it to the appropriate blk_sts_t, free the csum array if we have one and bail.
--- fs/btrfs/file-item.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
fs/btrfs/file-item.c +11 -2
@@ -368,6 +368,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree; +struct btrfs_bio *bbio =NULL ;struct btrfs_path *path;const u32 sectorsize = fs_info->sectorsize;const u32 csum_size = fs_info->csum_size;
@@ -377,6 +378,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dstu8 *csum;const unsigned int nblocks = orig_len >> fs_info->sectorsize_bits;int count =0 ; +blk_status_t ret =BLK_STS_OK ;if ((BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM ) ||test_bit (BTRFS_FS_STATE_NO_CSUMS , &fs_info->fs_state))
@@ -400,7 +402,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dstreturn BLK_STS_RESOURCE ;if (!dst) { -struct btrfs_bio *bbio =btrfs_bio (bio); + bbio =btrfs_bio (bio);if (nblocks * csum_size >BTRFS_BIO_INLINE_CSUM_SIZE ) { bbio->csum =kmalloc_array (nblocks, csum_size,GFP_NOFS );
@@ -456,6 +458,13 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst count =search_csum_tree (fs_info, path, cur_disk_bytenr, search_len, csum_dst); +if (count <0 ) { + ret =errno_to_blk_status (count); +if (bbio) +btrfs_bio_free_csum (bbio); +break ; + } +if (count <=0 ) {/* * Either we hit a critical error or we didn't find
@@ -491,7 +500,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst }btrfs_free_path (path); -return BLK_STS_OK ; +return ret; } int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,
Signature · 1 line
2.26.3
Re: [PATCH 2/8] btrfs: handle csum lookup errors properly on reads
Quoted 48 lines · Josef Bacik
Currently any error we get while trying to lookup csums during reads shows up as a missing csum, and then on the read completion side we spit out an error saying there was a csum mismatch and we increase the device corruption count.
However we could have gotten an EIO from the lookup. We could also be inside of a memory constrained container and gotten a ENOMEM while trying to do the read. In either case we don't want to make this look like a file system corruption problem, we want to make it look like the actual error it is. Capture any negative value, convert it to the appropriate
blk_sts_t, free the csum array if we have one and bail.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/file-item.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)fs/btrfs/file-item.c +3
-1@@ -368,6 +368,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree; +struct btrfs_bio *bbio =NULL ;struct btrfs_path *path;const u32 sectorsize = fs_info->sectorsize;const u32 csum_size = fs_info->csum_size;@@ -377,6 +378,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dstu8 *csum;const unsigned int nblocks = orig_len >> fs_info->sectorsize_bits;int count =0 ; +blk_status_t ret =BLK_STS_OK ;if ((BTRFS_I (inode)->flags &BTRFS_INODE_NODATASUM ) ||test_bit (BTRFS_FS_STATE_NO_CSUMS , &fs_info->fs_state))@@ -400,7 +402,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dstreturn BLK_STS_RESOURCE ;if (!dst) { -struct btrfs_bio *bbio =btrfs_bio (bio); + bbio =btrfs_bio (bio);if (nblocks * csum_size >BTRFS_BIO_INLINE_CSUM_SIZE ) { bbio->csum =kmalloc_array (nblocks, csum_size,GFP_NOFS );@@ -456,6 +458,13 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst
fs/btrfs/file-item.c blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst · line 468
Quoted 10 lines
fs/btrfs/file-item.c +7
-0count =search_csum_tree (fs_info, path, cur_disk_bytenr, search_len, csum_dst); +if (count <0 ) { + ret =errno_to_blk_status (count); +if (bbio) +btrfs_bio_free_csum (bbio); +break ; + } +if (count <=0 ) {
This new error handling doesn't quite mesh with the existing logic, IMO.
1) count <= 0 is redundant with count < 0 above and it's confusing to think about whether this code runs for count < 0.
2) it really is not running the nodatacow thing for the reloc inodes for negative return values anymore. Is that desirable? I found the original commit about it:
'Btrfs: fix nodatasum handling in balancing code'
and it looks like that one cared about the 0 case, not the negative case, but that's not what the logic or comment suggest as they are.
I think if you switch it to explicitly:
< 0 -> error; run the new bail code.
== 0 -> not found; deal with reloc, warn otherwise.
and make a single comment covering it, then it would make sense to me.
Quoted 14 lines
fs/btrfs/file-item.c +1
-1/* * Either we hit a critical error or we didn't find @@ -491,7 +500,7 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst }btrfs_free_path (path); -return BLK_STS_OK ; +return ret; } int btrfs_lookup_csums_range(struct btrfs_root *root, u64 start, u64 end,Signature · 1 line
2.26.3
PATCH 3/8 btrfs: check correct bio in finish_compressed_bio_read
In reply to Josef BacikCommit c09abff87f90 ("btrfs: cloned bios must not be iterated by bio_for_each_segment_all") added ASSERT()'s to make sure we weren't calling bio_for_each_segment_all() on a RAID5/6 bio. However it was checking the bio that the compression code passed in, not the cb->orig_bio that we actually iterate over, so adjust this ASSERT() to check the correct bio.
--- fs/btrfs/compression.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/compression.c +1 -1
@@ -259,7 +259,7 @@ static void finish_compressed_bio_read(struct compressed_bio *cb, struct bio *bi* We have verified the checksum already, set page checked so * the end_io handlers know about it */ -ASSERT (!bio_flagged (bio,BIO_CLONED )); +ASSERT (!bio_flagged (cb->orig_bio,BIO_CLONED ));bio_for_each_segment_all (bvec, cb->orig_bio, iter_all) {u64 bvec_start =page_offset (bvec->bv_page) + bvec->bv_offset;
Signature · 1 line
2.26.3
Re: [PATCH 3/8] btrfs: check correct bio in finish_compressed_bio_read
Quoted 8 lines · Josef Bacik
Commit
c09abff87f90("btrfs: cloned bios must not be iterated bybio_for_each_segment_all") added ASSERT()'s to make sure we weren't callingbio_for_each_segment_all()on a RAID5/6 bio. However it was checking the bio that the compression code passed in, not thecb->orig_biothat we actually iterate over, so adjust thisASSERT()to check the correct bio.Signed-off-by Josef Bacik <[email protected]>
Quoted 20 lines
--- fs/btrfs/compression.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/btrfs/compression.c +1
-1@@ -259,7 +259,7 @@ static void finish_compressed_bio_read(struct compressed_bio *cb, struct bio *bi* We have verified the checksum already, set page checked so * the end_io handlers know about it */ -ASSERT (!bio_flagged (bio,BIO_CLONED )); +ASSERT (!bio_flagged (cb->orig_bio,BIO_CLONED ));bio_for_each_segment_all (bvec, cb->orig_bio, iter_all) {u64 bvec_start =page_offset (bvec->bv_page) + bvec->bv_offset;Signature · 1 line
2.26.3
Re: [PATCH 3/8] btrfs: check correct bio in finish_compressed_bio_read
In reply to Josef BacikQuoted 17 lines · Josef Bacik
Commit
c09abff87f90("btrfs: cloned bios must not be iterated bybio_for_each_segment_all") added ASSERT()'s to make sure we weren't callingbio_for_each_segment_all()on a RAID5/6 bio. However it was checking the bio that the compression code passed in, not thecb->orig_biothat we actually iterate over, so adjust thisASSERT()to check the correct bio.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/compression.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/btrfs/compression.c +0
-0
fs/btrfs/compression.c static void finish_compressed_bio_read(struct compressed_bio *cb, struct bio *bi · line 265
Quoted 8 lines
fs/btrfs/compression.c +1
-1@@ -259,7 +259,7 @@ static void finish_compressed_bio_read(struct compressed_bio *cb, struct bio *bi* We have verified the checksum already, set page checked so * the end_io handlers know about it */ -ASSERT (!bio_flagged (bio,BIO_CLONED )); +ASSERT (!bio_flagged (cb->orig_bio,BIO_CLONED ));bio_for_each_segment_all (bvec, cb->orig_bio, iter_all) {u64 bvec_start =page_offset (bvec->bv_page) + bvec->bv_offset;
Looks good,
PATCH 4/8 btrfs: remove the bio argument from finish_compressed_bio_read
In reply to Josef BacikThis bio is usually one of the compressed bio's, and we don't actually need it in this function, so remove the argument and stop passing it around.
--- fs/btrfs/compression.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
fs/btrfs/compression.c +3 -5
@@ -234,7 +234,7 @@ static bool dec_and_test_compressed_bio(struct compressed_bio *cb, struct bio *breturn last_io; } -static void finish_compressed_bio_read (struct compressed_bio *cb,struct bio *bio) +static void finish_compressed_bio_read (struct compressed_bio *cb) {unsigned int index;struct page *page;
@@ -253,8 +253,6 @@ static void finish_compressed_bio_read(struct compressed_bio *cb, struct bio *bistruct bio_vec *bvec;struct bvec_iter_all iter_all; -ASSERT (bio); -ASSERT (!bio->bi_status);/* * We have verified the checksum already, set page checked so * the end_io handlers know about it
@@ -325,7 +323,7 @@ static void end_compressed_bio_read(struct bio *bio) csum_failed:if (ret) cb->errors =1 ; -finish_compressed_bio_read (cb, bio); +finish_compressed_bio_read (cb); out:bio_put (bio); }
@@ -970,7 +968,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,*/ ASSERT (refcount_read (&cb->pending_sectors));/* Now we are the only one referring @cb, can finish it safely. */ -finish_compressed_bio_read (cb,NULL ); +finish_compressed_bio_read (cb);return ret; }
Signature · 1 line
2.26.3
Re: [PATCH 4/8] btrfs: remove the bio argument from finish_compressed_bio_read
Looks good,
PATCH 5/8 btrfs: track compressed bio errors as blk_status_t
In reply to Josef BacikRight now we just have a binary "errors" flag, so any error we get on the compressed bio's gets translated to EIO. This isn't necessarily a bad thing, but if we get an ENOMEM it may be nice to know that's what happened instead of an EIO. Track our errors as a blk_status_t, and do the appropriate setting of the errors accordingly.
--- fs/btrfs/compression.c | 26 ++++++++++++++------------ fs/btrfs/compression.h | 2 +- 2 files changed, 15 insertions(+), 13 deletions(-)
fs/btrfs/compression.c +14 -12
@@ -219,7 +219,7 @@ static bool dec_and_test_compressed_bio(struct compressed_bio *cb, struct bio *b bi_size += bvec->bv_len;if (bio->bi_status) - cb->errors =1 ; + cb->status = bio->bi_status;ASSERT (bi_size && bi_size <= cb->compressed_len); last_io = refcount_sub_and_test(bi_size >> fs_info->sectorsize_bits,
@@ -247,8 +247,9 @@ static void finish_compressed_bio_read(struct compressed_bio *cb) }/* Do io completion on the original bio */ -if (cb->errors) { -bio_io_error (cb->orig_bio); +if (cb->status !=BLK_STS_OK ) { + cb->orig_bio->bi_status = cb->status; +bio_endio (cb->orig_bio); }else {struct bio_vec *bvec;struct bvec_iter_all iter_all;
@@ -306,7 +307,7 @@ static void end_compressed_bio_read(struct bio *bio)* Some IO in this cb have failed, just skip checksum as there * is no way it could be correct. */ -if (cb->errors ==1 ) +if (cb->status !=BLK_STS_OK ) goto csum_failed; inode = cb->inode;
@@ -322,7 +323,7 @@ static void end_compressed_bio_read(struct bio *bio) csum_failed:if (ret) - cb->errors =1 ; + cb->status =errno_to_blk_status (ret);finish_compressed_bio_read (cb); out:bio_put (bio);
@@ -340,11 +341,12 @@ static noinline void end_compressed_writeback(struct inode *inode,unsigned long end_index = (cb->start + cb->len -1 ) >>PAGE_SHIFT ;struct page *pages[16 ];unsigned long nr_pages = end_index - index +1 ; +int errno =blk_status_to_errno (cb->status);int i;int ret; -if (cb->errors) -mapping_set_error (inode->i_mapping, -EIO ); +if (errno) +mapping_set_error (inode->i_mapping, errno);while (nr_pages >0 ) { ret = find_get_pages_contig(inode->i_mapping, index,
@@ -356,7 +358,7 @@ static noinline void end_compressed_writeback(struct inode *inode,continue ; }for (i =0 ; i < ret; i++) { -if (cb->errors) +if (errno)SetPageError (pages[i]);btrfs_page_clamp_clear_writeback (fs_info, pages[i], cb->start, cb->len);
@@ -372,14 +374,14 @@ static void finish_compressed_bio_write(struct compressed_bio *cb) {struct inode *inode = cb->inode;unsigned int index; +int errno =blk_status_to_errno (cb->status);/* * Ok, we're the last bio for this extent, step one is to call back * into the FS and do all the end_io operations. */ btrfs_writepage_endio_finish_ordered (BTRFS_I (inode),NULL , - cb->start, cb->start + cb->len -1 , - !cb->errors); + cb->start, cb->start + cb->len -1 , errno ==0 );end_compressed_writeback (inode, cb);/* Note, our inode could be gone now */
@@ -522,7 +524,7 @@ blk_status_t btrfs_submit_compressed_write(struct btrfs_inode *inode, u64 start,if (!cb)return BLK_STS_RESOURCE ;refcount_set (&cb->pending_sectors, compressed_len >> fs_info->sectorsize_bits); - cb->errors =0 ; + cb->status =BLK_STS_OK ; cb->inode = &inode->vfs_inode; cb->start = start; cb->len = len;
@@ -829,7 +831,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, goto out;refcount_set (&cb->pending_sectors, compressed_len >> fs_info->sectorsize_bits); - cb->errors =0 ; + cb->status =BLK_STS_OK ; cb->inode = inode; cb->mirror_num = mirror_num; sums = cb->sums;
fs/btrfs/compression.h +1 -1
@@ -55,7 +55,7 @@ struct compressed_bio {u8 compress_type;/* IO errors */ -u8 errors; +blk_status_t status;int mirror_num;/* for reads, this is the bio we are copying the data into */
Signature · 1 line
2.26.3
Re: [PATCH 5/8] btrfs: track compressed bio errors as blk_status_t
Quoted 4 lines · Josef Bacik
fs/btrfs/compression.c +0
-0@@ -372,14 +374,14 @@ static void finish_compressed_bio_write(struct compressed_bio *cb) {struct inode *inode = cb->inode;unsigned int index;
fs/btrfs/compression.c static void finish_compressed_bio_write(struct compressed_bio *cb) · line 384
Quoted 10 lines
fs/btrfs/compression.c +2
-2+int errno =blk_status_to_errno (cb->status);/* * Ok, we're the last bio for this extent, step one is to call back * into the FS and do all the end_io operations. */ btrfs_writepage_endio_finish_ordered (BTRFS_I (inode),NULL , - cb->start, cb->start + cb->len -1 , - !cb->errors); + cb->start, cb->start + cb->len -1 , errno ==0 );
No need for the errno here, it could just be:
btrfs_writepage_endio_finish_ordered(BTRFS_I(inode), NULL, cb->start, cb->start + cb->len - 1, cb->status == BLK_STS_OK);
PATCH 6/8 btrfs: do not double complete bio on errors during compressed reads
In reply to Josef BacikI hit some weird panics while fixing up the error handling from btrfs_lookup_bio_sums(). Turns out the compression path will complete the bio we use if we set up any of the compression bios and then return an error, and then btrfs_submit_data_bio() will also call bio_endio() on the bio.
Fix this by making btrfs_submit_compressed_read() responsible for calling bio_endio() on the bio if there are any errors. Currently it was only doing it if we created the compression bios, otherwise it was depending on btrfs_submit_data_bio() to do the right thing. This creates the above problem, so fix up btrfs_submit_compressed_read() to always call bio_endio() in case of an error, and then simply return from btrfs_submit_data_bio() if we had to call btrfs_submit_compressed_read().
--- fs/btrfs/compression.c | 14 +++++++++----- fs/btrfs/inode.c | 12 ++++++++---- 2 files changed, 17 insertions(+), 9 deletions(-)
fs/btrfs/compression.c +9 -5
@@ -808,7 +808,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,u64 em_len;u64 em_start;struct extent_map *em; -blk_status_t ret =BLK_STS_RESOURCE ; +blk_status_t ret;int faili =0 ;u8 *sums;
@@ -821,9 +821,12 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,read_lock (&em_tree->lock); em =lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);read_unlock (&em_tree->lock); -if (!em) -return BLK_STS_IOERR ; +if (!em) { + ret =BLK_STS_IOERR ; + goto out; + } + ret =BLK_STS_RESOURCE ;ASSERT (em->compress_type !=BTRFS_COMPRESS_NONE ); compressed_len = em->block_len; cb =kmalloc (compressed_bio_size (fs_info, compressed_len),GFP_NOFS );
@@ -858,7 +861,6 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, cb->compressed_pages[pg_index] =alloc_page (GFP_NOFS );if (!cb->compressed_pages[pg_index]) { faili = pg_index -1 ; - ret =BLK_STS_RESOURCE ; goto fail2; } }
@@ -938,7 +940,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio =NULL ; } } -return 0 ; +return BLK_STS_OK ; fail2:while (faili >=0 ) {
@@ -951,6 +953,8 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,kfree (cb); out:free_extent_map (em); + bio->bi_status = ret; +bio_endio (bio);return ret; finish_cb:if (comp_bio) {
fs/btrfs/inode.c +8 -4
@@ -2542,10 +2542,14 @@ blk_status_t btrfs_submit_data_bio(struct inode *inode, struct bio *bio, goto out;if (bio_flags &EXTENT_BIO_COMPRESSED ) { - ret =btrfs_submit_compressed_read (inode, bio, - mirror_num, - bio_flags); - goto out; +/* +* btrfs_submit_compressed_read will handle completing +* the bio if there were any errors, so just return +* here. +*/ +return btrfs_submit_compressed_read (inode, bio, + mirror_num, + bio_flags); }else {/* * Lookup bio sums does extra checks around whether we
Signature · 1 line
2.26.3
Re: [PATCH 6/8] btrfs: do not double complete bio on errors during compressed reads
Quoted 15 lines · Josef Bacik
I hit some weird panics while fixing up the error handling from
btrfs_lookup_bio_sums(). Turns out the compression path will complete the bio we use if we set up any of the compression bios and then return an error, and thenbtrfs_submit_data_bio()will also callbio_endio()on the bio.Fix this by making
btrfs_submit_compressed_read()responsible for callingbio_endio()on the bio if there are any errors. Currently it was only doing it if we created the compression bios, otherwise it was depending onbtrfs_submit_data_bio()to do the right thing. This creates the above problem, so fix upbtrfs_submit_compressed_read()to always callbio_endio()in case of an error, and then simply return frombtrfs_submit_data_bio()if we had to callbtrfs_submit_compressed_read().
Quoted 1 line
Signed-off-by Josef Bacik <[email protected]>
Quoted 20 lines
--- fs/btrfs/compression.c | 14 +++++++++----- fs/btrfs/inode.c | 12 ++++++++---- 2 files changed, 17 insertions(+), 9 deletions(-)fs/btrfs/compression.c +1
-1@@ -808,7 +808,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,u64 em_len;u64 em_start;struct extent_map *em; -blk_status_t ret =BLK_STS_RESOURCE ; +blk_status_t ret;int faili =0 ;u8 *sums;@@ -821,9 +821,12 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,read_lock (&em_tree->lock);
fs/btrfs/compression.c blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · line 829
Quoted 10 lines
fs/btrfs/compression.c +5
-2em =lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);read_unlock (&em_tree->lock); -if (!em) -return BLK_STS_IOERR ; +if (!em) { + ret =BLK_STS_IOERR ; + goto out; + } + ret =BLK_STS_RESOURCE ;
I think the error handling logic with all the special exit paths makes it worthwhile to set ret at each individual 'goto failX'.
Quoted 55 lines
fs/btrfs/compression.c +3
-2ASSERT (em->compress_type !=BTRFS_COMPRESS_NONE ); compressed_len = em->block_len; cb =kmalloc (compressed_bio_size (fs_info, compressed_len),GFP_NOFS );@@ -858,7 +861,6 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, cb->compressed_pages[pg_index] =alloc_page (GFP_NOFS );if (!cb->compressed_pages[pg_index]) { faili = pg_index -1 ; - ret =BLK_STS_RESOURCE ; goto fail2; } }@@ -938,7 +940,7 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio =NULL ; } } -return 0 ; +return BLK_STS_OK ; fail2:while (faili >=0 ) {@@ -951,6 +953,8 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,kfree (cb); out:free_extent_map (em); + bio->bi_status = ret; +bio_endio (bio);return ret; finish_cb:if (comp_bio) {fs/btrfs/inode.c +8
-4@@ -2542,10 +2542,14 @@ blk_status_t btrfs_submit_data_bio(struct inode *inode, struct bio *bio, goto out;if (bio_flags &EXTENT_BIO_COMPRESSED ) { - ret =btrfs_submit_compressed_read (inode, bio, - mirror_num, - bio_flags); - goto out; +/* +* btrfs_submit_compressed_read will handle completing +* the bio if there were any errors, so just return +* here. +*/ +return btrfs_submit_compressed_read (inode, bio, + mirror_num, + bio_flags); }else {/* * Lookup bio sums does extra checks around whether we Signature · 1 line
2.26.3
Re: [PATCH 6/8] btrfs: do not double complete bio on errors during compressed reads
Quoted 11 lines · Boris Burkov
Quoted 11 lines
fs/btrfs/compression.c +4
-2@@ -821,9 +821,12 @@ blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,read_lock (&em_tree->lock); em =lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);read_unlock (&em_tree->lock); -if (!em) -return BLK_STS_IOERR ; +if (!em) { + ret =BLK_STS_IOERR ; + goto out; + }
Quoted 4 lines
fs/btrfs/compression.c blk_status_t btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · line 829
Quoted 1 line
fs/btrfs/compression.c +1
-0+ ret =BLK_STS_RESOURCE ;I think the error handling logic with all the special exit paths makes it worthwhile to set ret at each individual 'goto failX'.
Yeah, this is IMHO more clear which error is returned from the error branches than some random value set before.
PATCH 7/8 btrfs: do not try to repair bio that has no mirror set
In reply to Josef BacikIf we fail to submit a bio for whatever reason, we may not have setup a mirror_num for that bio. This means we shouldn't try to do the repair workflow, if we do we'll hit an BUG_ON(!failrec->this_mirror) in clean_io_failure. Instead simply skip the repair workflow if we have no mirror set, and add an assert to btrfs_check_repairable() to make it easier to catch what is happening in the future.
--- fs/btrfs/extent_io.c | 9 +++++++++ 1 file changed, 9 insertions(+)
fs/btrfs/extent_io.c +9 -0
@@ -2610,6 +2610,7 @@ static bool btrfs_check_repairable(struct inode *inode,* a good copy of the failed sector and if we succeed, we have setup * everything for repair_io_failure to do the rest for us. */ +ASSERT (failed_mirror); failrec->failed_mirror = failed_mirror; failrec->this_mirror++;if (failrec->this_mirror == failed_mirror)
@@ -3067,6 +3068,14 @@ static void end_bio_extent_readpage(struct bio *bio) goto readpage_ok;if (is_data_inode (inode)) { +/* +* If we failed to submit the IO at all we'll have a +* mirror_num == 0, in which case we need to just mark +* the page with an error and unlock it and carry on. +*/ +if (mirror ==0 ) + goto readpage_ok; +/* * btrfs_submit_read_repair() will handle all the good * and bad sectors, we just continue to the next bvec.
Signature · 1 line
2.26.3
Re: [PATCH 7/8] btrfs: do not try to repair bio that has no mirror set
Quoted 8 lines · Josef Bacik
If we fail to submit a bio for whatever reason, we may not have setup a
mirror_numfor that bio. This means we shouldn't try to do the repair workflow, if we do we'll hit anBUG_ON(!failrec->this_mirror)inclean_io_failure. Instead simply skip the repair workflow if we have no mirror set, and add an assert tobtrfs_check_repairable()to make it easier to catch what is happening in the future.Signed-off-by Josef Bacik <[email protected]>
Quoted 34 lines
--- fs/btrfs/extent_io.c | 9 +++++++++ 1 file changed, 9 insertions(+)fs/btrfs/extent_io.c +9
-0@@ -2610,6 +2610,7 @@ static bool btrfs_check_repairable(struct inode *inode,* a good copy of the failed sector and if we succeed, we have setup * everything for repair_io_failure to do the rest for us. */ +ASSERT (failed_mirror); failrec->failed_mirror = failed_mirror; failrec->this_mirror++;if (failrec->this_mirror == failed_mirror)@@ -3067,6 +3068,14 @@ static void end_bio_extent_readpage(struct bio *bio) goto readpage_ok;if (is_data_inode (inode)) { +/* +* If we failed to submit the IO at all we'll have a +* mirror_num == 0, in which case we need to just mark +* the page with an error and unlock it and carry on. +*/ +if (mirror ==0 ) + goto readpage_ok; +/* * btrfs_submit_read_repair() will handle all the good * and bad sectors, we just continue to the next bvec. Signature · 1 line
2.26.3
PATCH 8/8 btrfs: do not clean up repair bio if submit fails
In reply to Josef BacikThe submit helper will always run bio_endio() on the bio if it fails to submit, so cleaning up the bio just leads to a variety of UAF and NULL pointer deref bugs because we race with the endio function that is cleaning up the bio. Instead just return STS_OK as the repair function has to continue to process the rest of the pages, and the endio for the repair bio will do the appropriate cleanup for the page that it was given.
--- fs/btrfs/extent_io.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
fs/btrfs/extent_io.c +8 -8
@@ -2640,7 +2640,6 @@ int btrfs_repair_one_sector(struct inode *inode,const int icsum = bio_offset >> fs_info->sectorsize_bits;struct bio *repair_bio;struct btrfs_bio *repair_bbio; -blk_status_t status;btrfs_debug (fs_info,"repair read error: read error at %llu" , start);
@@ -2679,13 +2678,14 @@ int btrfs_repair_one_sector(struct inode *inode,"repair read error: submitting new read to mirror %d" , failrec->this_mirror); - status =submit_bio_hook (inode, repair_bio, failrec->this_mirror, - failrec->bio_flags); -if (status) { -free_io_failure (failure_tree, tree, failrec); -bio_put (repair_bio); - } -return blk_status_to_errno (status); +/* +* At this point we have a bio, so any errors from submit_bio_hook() +* will be handled by the endio on the repair_bio, so we can't return an +* error here. +*/ +submit_bio_hook (inode, repair_bio, failrec->this_mirror, + failrec->bio_flags); +return BLK_STS_OK ; }static void end_page_read (struct page *page,bool uptodate,u64 start,u32 len)
Signature · 1 line
2.26.3
Re: [PATCH 8/8] btrfs: do not clean up repair bio if submit fails
Quoted 9 lines · Josef Bacik
The submit helper will always run
bio_endio()on the bio if it fails to submit, so cleaning up the bio just leads to a variety of UAF and NULL pointer deref bugs because we race with the endio function that is cleaning up the bio. Instead just returnSTS_OKas the repair function has to continue to process the rest of the pages, and the endio for the repair bio will do the appropriate cleanup for the page that it was given.Signed-off-by Josef Bacik <[email protected]>
Quoted 41 lines
--- fs/btrfs/extent_io.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)fs/btrfs/extent_io.c +8
-8@@ -2640,7 +2640,6 @@ int btrfs_repair_one_sector(struct inode *inode,const int icsum = bio_offset >> fs_info->sectorsize_bits;struct bio *repair_bio;struct btrfs_bio *repair_bbio; -blk_status_t status;btrfs_debug (fs_info,"repair read error: read error at %llu" , start);@@ -2679,13 +2678,14 @@ int btrfs_repair_one_sector(struct inode *inode,"repair read error: submitting new read to mirror %d" , failrec->this_mirror); - status =submit_bio_hook (inode, repair_bio, failrec->this_mirror, - failrec->bio_flags); -if (status) { -free_io_failure (failure_tree, tree, failrec); -bio_put (repair_bio); - } -return blk_status_to_errno (status); +/* +* At this point we have a bio, so any errors from submit_bio_hook() +* will be handled by the endio on the repair_bio, so we can't return an +* error here. +*/ +submit_bio_hook (inode, repair_bio, failrec->this_mirror, + failrec->bio_flags); +return BLK_STS_OK ; }static void end_page_read (struct page *page,bool uptodate,u64 start,u32 len)Signature · 1 line
2.26.3