btrfs: Enhancement to tree block validation
cover btrfs: Enhancement to tree block validation
Patchset can be fetched from github:
https://github.com/adam900710/linux/tree/write_time_tree_checker
Which is based on v5.0-rc1 tag.
Also there is no conflict rebasing the patchset to misc-next.
This patchset has the following 3 features:
- Tree block validation output enhancement
* Output validation failure timing (write time or read time)
* Always output tree block level/key mismatch error message This part is already submitted and reviewed.
- Write time tree block validation check
To catch memory corruption either from hardware or kernel. Example output would be:
BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3): delayed_refs has NO entry
- Better error handling before calling flush_write_bio() One hidden reason of calling flush_write_bio() under all cases is, flush_write_bio() will trigger endio function and endio function of epd->bio will free the bio under all cases.
So we're in fact abusing flush_write_bio() as cleanup.
Since now flush_write_bio() has its own return value, we shouldn't call flush_write_bio() no-brain, here we introduce proper cleanup helper, end_write_bio(). Now we call flush_write_bio() like:
New | Old
--------------------------------------------------------------
ret = do_some_evil(&epd); | ret = do_some_evil(&epd);
if (ret < 0) { | flush_write_bio(&epd);
end_write_bio(&epd, ret); | ^^^ submitting half-backed epd->bio?
return ret; | return ret;
} | ret = flush_write_bio(&epd); | return ret; |
Above code should be more streamline for the error handling part.
Changelog:
v2:
- Unlock locked pages in lock_extent_buffer_for_io() for error handling.
- Added Reviewed-by tags.
v3:
- Remove duplicated error message.
- Use IS_ENABLED() macro to replace #ifdef.
- Added Reviewed-by tags.
v4:
- Re-organized patch split
Now each BUG_ON() cleanup has its own patch
- Dig much further into the call sites to eliminate unexpected >0 return May be a little paranoid and abuse some ASSERT(), but it should be much safer against further code change.
- Fix the false alert caused by balance and memory pressure The fix it skip owner checker for non-essential tree at write time. Since owner root can't always be reliable, either due to commit root created in current transaction or balance + memory pressure.
v5:
- Do proper error-out handling other than relying on flush_write_bio() to clean up.
This has a side effect that no Reviewed-by tags for modified patches.
- New comment for why we don't need to do anything about ebp->bio when submit_one_bio() fails.
- Add some Reviewed-by tag.
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.
Qu Wenruo (12):
btrfs: Always output error message when key/level verification fails
btrfs: extent_io: Kill the forward declaration of flush_write_bio()
btrfs: disk-io: Show the timing of corrupted tree block explicitly
btrfs: extent_io: Move the BUG_ON() in flush_write_bio() one level up
btrfs: extent_io: Handle error better in extent_write_full_page()
btrfs: extent_io: Handle error better in btree_write_cache_pages()
btrfs: extent_io: Kill the dead branch in extent_write_cache_pages()
btrfs: extent_io: Handle error better in extent_write_locked_range()
btrfs: extent_io: Kill the BUG_ON() in lock_extent_buffer_for_io()
btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
btrfs: extent_io: Handle error better in extent_writepages()
btrfs: Do mandatory tree block check before submitting bio
fs/btrfs/disk-io.c | 23 ++++-- fs/btrfs/extent_io.c | 168 ++++++++++++++++++++++++++++------------ fs/btrfs/tree-checker.c | 24 +++++- fs/btrfs/tree-checker.h | 8 ++ 4 files changed, 164 insertions(+), 59 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/12 btrfs: Always output error message when key/level verification fails
We have internal report of strange transaction abort due to EUCLEAN without any error message.
Since error message inside verify_level_key() is only enabled for CONFIG_BTRFS_DEBUG, the error message won't output for most distro.
This patch will make the error message mandatory, so when problem happens we know what's causing the problem.
--- fs/btrfs/disk-io.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
fs/btrfs/disk-io.c +4 -6
@@ -423,12 +423,11 @@ static int verify_level_key(struct btrfs_fs_info *fs_info, found_level =btrfs_header_level (eb);if (found_level != level) { -#ifdef CONFIG_BTRFS_DEBUG -WARN_ON (1 ); +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: tree level check failed\n" );btrfs_err (fs_info,"tree level mismatch detected, bytenr=%llu level expected=%u has=%u" , eb->start, level, found_level); -#endif return -EIO ; }
@@ -449,9 +448,9 @@ static int verify_level_key(struct btrfs_fs_info *fs_info,btrfs_item_key_to_cpu (eb, &found_key,0 ); ret =btrfs_comp_cpu_keys (first_key, &found_key); -#ifdef CONFIG_BTRFS_DEBUG if (ret) { -WARN_ON (1 ); +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: tree first key check failed\n" ); btrfs_err(fs_info, "tree first key mismatch detected, bytenr=%llu parent_transid=%llu key expected=(%llu,%u,%llu) has=(%llu,%u,%llu)", eb->start, parent_transid, first_key->objectid,
@@ -459,7 +458,6 @@ static int verify_level_key(struct btrfs_fs_info *fs_info, found_key.objectid, found_key.type, found_key.offset); } -#endif return ret; }
Signature · 1 line
2.20.1
PATCH 2/12 btrfs: extent_io: Kill the forward declaration of flush_write_bio()
In reply to Qu WenruoThere is no need to forward declare flush_write_bio(), as it only depends on submit_one_bio().
Both of them are pretty small, just move them to kill the forward declaration.
--- fs/btrfs/extent_io.c | 66 +++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 34 deletions(-)
fs/btrfs/extent_io.c +32 -34
@@ -147,7 +147,38 @@ static int add_extent_changeset(struct extent_state *state, unsigned bits,return ret; } -static void flush_write_bio (struct extent_page_data *epd); +static int __must_check submit_one_bio (struct bio *bio,int mirror_num, +unsigned long bio_flags) +{ +blk_status_t ret =0 ; +struct bio_vec *bvec =bio_last_bvec_all (bio); +struct page *page = bvec->bv_page; +struct extent_io_tree *tree = bio->bi_private; +u64 start; + + start =page_offset (page) + bvec->bv_offset; + + bio->bi_private =NULL ; + +if (tree->ops) + ret = tree->ops->submit_bio_hook (tree->private_data, bio, + mirror_num, bio_flags, start); +else +btrfsic_submit_bio (bio); + +return blk_status_to_errno (ret); +} + +static void flush_write_bio (struct extent_page_data *epd) +{ +if (epd->bio) { +int ret; + + ret =submit_one_bio (epd->bio,0 ,0 ); +BUG_ON (ret <0 );/* -ENOMEM */ + epd->bio =NULL ; + } +}int __init extent_io_init (void ) {
@@ -2692,28 +2723,6 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, int offset, int size)return bio; } -static int __must_check submit_one_bio (struct bio *bio,int mirror_num, -unsigned long bio_flags) -{ -blk_status_t ret =0 ; -struct bio_vec *bvec =bio_last_bvec_all (bio); -struct page *page = bvec->bv_page; -struct extent_io_tree *tree = bio->bi_private; -u64 start; - - start =page_offset (page) + bvec->bv_offset; - - bio->bi_private =NULL ; - -if (tree->ops) - ret = tree->ops->submit_bio_hook (tree->private_data, bio, - mirror_num, bio_flags, start); -else -btrfsic_submit_bio (bio); - -return blk_status_to_errno (ret); -} -/* * @opf: bio REQ_OP_* and REQ_* flags as one value * @tree: tree so we can call our merge_bio hook
@@ -4007,17 +4016,6 @@ static int extent_write_cache_pages(struct address_space *mapping,return ret; } -static void flush_write_bio (struct extent_page_data *epd) -{ -if (epd->bio) { -int ret; - - ret =submit_one_bio (epd->bio,0 ,0 ); -BUG_ON (ret <0 );/* -ENOMEM */ - epd->bio =NULL ; - } -} -int extent_write_full_page (struct page *page,struct writeback_control *wbc) {int ret;
Signature · 1 line
2.20.1
PATCH 3/12 btrfs: disk-io: Show the timing of corrupted tree block explicitly
In reply to Qu WenruoJust add one extra line to show when the corruption is detected. Currently only read time detection is possible.
The planned distinguish line would be:
read time:
<detail report>
block=XXXXX read time tree block corruption detected
write time:
<detail report>
block=XXXXX write time tree block corruption detected
--- fs/btrfs/disk-io.c | 3 +++ 1 file changed, 3 insertions(+)
fs/btrfs/disk-io.c +3 -0
@@ -671,6 +671,9 @@ static int btree_readpage_end_io_hook(struct btrfs_io_bio *io_bio,*/ atomic_inc (&eb->io_pages);clear_extent_buffer_uptodate (eb); +btrfs_err (fs_info, +"block=%llu read time tree block corruption detected" , + eb->start); }free_extent_buffer (eb); out:
Signature · 1 line
2.20.1
PATCH 3/12 btrfs: disk-io: Show the timing of corrupted tree block explicitly
Just add one extra line to show when the corruption is detected. Currently only read time detection is possible.
The planned distinguish line would be:
read time:
<detail report>
block=XXXXX read time tree block corruption detected
write time:
<detail report>
block=XXXXX write time tree block corruption detected
---
v5.2:
- Make the read time error message only for tree checker error. RAID reconstruction can cause a lot of errors which are expected. Avoid error message flood for such use case.
--- fs/btrfs/disk-io.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/btrfs/disk-io.c +4 -0
@@ -658,6 +658,10 @@ static int btree_readpage_end_io_hook(struct btrfs_io_bio *io_bio,if (!ret)set_extent_buffer_uptodate (eb); +else +btrfs_err (fs_info, +"block=%llu read time tree block corruption detected" , + eb->start); err:if (reads_done &&test_and_clear_bit (EXTENT_BUFFER_READAHEAD , &eb->bflags))
Signature · 1 line
2.20.1
PATCH 4/12 btrfs: extent_io: Move the BUG_ON() in flush_write_bio() one level up
In reply to Qu WenruoWe have a BUG_ON() in flush_write_bio() to handle the return value of submit_one_bio().
Move the BUG_ON() one level up to all its callers.
This patch will introduce temporary variable, @flush_ret to keep code change minimal in this patch. That variable will be cleaned up when enhancing the error handling later.
--- fs/btrfs/extent_io.c | 55 +++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 14 deletions(-)
fs/btrfs/extent_io.c +41 -14
@@ -169,15 +169,28 @@ static int __must_check submit_one_bio(struct bio *bio, int mirror_num,return blk_status_to_errno (ret); } -static void flush_write_bio (struct extent_page_data *epd) +/* +* A wrapper for submit_one_bio(). +* +* Return 0 if everything is OK. +* Return <0 for error. +*/ +static int __must_check flush_write_bio (struct extent_page_data *epd) { -if (epd->bio) { -int ret; +int ret =0 ; +if (epd->bio) { ret =submit_one_bio (epd->bio,0 ,0 ); -BUG_ON (ret <0 );/* -ENOMEM */ +/* +* Clean up of epd->bio is handled by its endio function. +* And endio is either triggered by successful bio execution +* or the error handler of submit bio hook. +* So at this point, no matter what happened, we don't need +* to clean up epd->bio. +*/ epd->bio =NULL ; } +return ret; }int __init extent_io_init (void )
@@ -3510,7 +3523,8 @@ lock_extent_buffer_for_io(struct extent_buffer *eb,if (!btrfs_try_tree_write_lock (eb)) { flush =1 ; -flush_write_bio (epd); + ret =flush_write_bio (epd); +BUG_ON (ret <0 );btrfs_tree_lock (eb); }
@@ -3519,7 +3533,8 @@ lock_extent_buffer_for_io(struct extent_buffer *eb,if (!epd->sync_io)return 0 ;if (!flush) { -flush_write_bio (epd); + ret =flush_write_bio (epd); +BUG_ON (ret <0 ); flush =1 ; }while (1 ) {
@@ -3560,7 +3575,8 @@ lock_extent_buffer_for_io(struct extent_buffer *eb,if (!trylock_page (p)) {if (!flush) { -flush_write_bio (epd); + ret =flush_write_bio (epd); +BUG_ON (ret <0 ); flush =1 ; }lock_page (p);
@@ -3751,6 +3767,7 @@ int btree_write_cache_pages(struct address_space *mapping, .sync_io = wbc->sync_mode == WB_SYNC_ALL, };int ret =0 ; +int flush_ret;int done =0 ;int nr_to_write_done =0 ;struct pagevec pvec;
@@ -3850,7 +3867,8 @@ int btree_write_cache_pages(struct address_space *mapping, index =0 ; goto retry; } -flush_write_bio (&epd); + flush_ret =flush_write_bio (&epd); +BUG_ON (flush_ret <0 );return ret; }
@@ -3947,7 +3965,8 @@ static int extent_write_cache_pages(struct address_space *mapping,* tmpfs file mapping */ if (!trylock_page (page)) { -flush_write_bio (epd); + ret =flush_write_bio (epd); +BUG_ON (ret <0 );lock_page (page); }
@@ -3957,8 +3976,10 @@ static int extent_write_cache_pages(struct address_space *mapping, }if (wbc->sync_mode !=WB_SYNC_NONE ) { -if (PageWriteback (page)) -flush_write_bio (epd); +if (PageWriteback (page)) { + ret =flush_write_bio (epd); +BUG_ON (ret <0 ); + }wait_on_page_writeback (page); }
@@ -4019,6 +4040,7 @@ static int extent_write_cache_pages(struct address_space *mapping,int extent_write_full_page (struct page *page,struct writeback_control *wbc) {int ret; +int flush_ret;struct extent_page_data epd = { .bio =NULL , .tree = &BTRFS_I (page->mapping->host)->io_tree,
@@ -4028,7 +4050,8 @@ int extent_write_full_page(struct page *page, struct writeback_control *wbc) ret =__extent_writepage (page, wbc, &epd); -flush_write_bio (&epd); + flush_ret =flush_write_bio (&epd); +BUG_ON (flush_ret <0 );return ret; }
@@ -4036,6 +4059,7 @@ int extent_write_locked_range(struct inode *inode, u64 start, u64 end, int mode) {int ret =0 ; +int flush_ret;struct address_space *mapping = inode->i_mapping;struct extent_io_tree *tree = &BTRFS_I (inode)->io_tree;struct page *page;
@@ -4068,7 +4092,8 @@ int extent_write_locked_range(struct inode *inode, u64 start, u64 end, start +=PAGE_SIZE ; } -flush_write_bio (&epd); + flush_ret =flush_write_bio (&epd); +BUG_ON (flush_ret <0 );return ret; }
@@ -4076,6 +4101,7 @@ int extent_writepages(struct address_space *mapping, struct writeback_control *wbc) {int ret =0 ; +int flush_ret;struct extent_page_data epd = { .bio =NULL , .tree = &BTRFS_I (mapping->host)->io_tree,
@@ -4084,7 +4110,8 @@ int extent_writepages(struct address_space *mapping, }; ret =extent_write_cache_pages (mapping, wbc, &epd); -flush_write_bio (&epd); + flush_ret =flush_write_bio (&epd); +BUG_ON (flush_ret <0 );return ret; }
Signature · 1 line
2.20.1
PATCH 5/12 btrfs: extent_io: Handle error better in extent_write_full_page()
In reply to Qu WenruoSince now flush_write_bio() could return error, kill the BUG_ON() first.
Then don't call flush_write_bio() unconditionally, instead we check the return value from __extent_writepage() first.
If __extent_writepage() fails, we do cleanup, and return error without submitting the possible corrupted or half-baked bio.
If __extent_writepage() successes, then we call flush_write_bio() and return the result.
--- fs/btrfs/extent_io.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +21 -3
@@ -169,6 +169,16 @@ static int __must_check submit_one_bio(struct bio *bio, int mirror_num,return blk_status_to_errno (ret); } +/* A wrapper for bio_endio() to cleanup unsubmitted bios */ +static void end_write_bio (struct extent_page_data *epd,int ret) +{ +if (epd->bio) { + epd->bio->bi_status =errno_to_blk_status (ret); +bio_endio (epd->bio); + epd->bio =NULL ; + } +} +/* * A wrapper for submit_one_bio(). *
@@ -3431,6 +3441,9 @@ static noinline_for_stack int __extent_writepage_io(struct inode *inode,* records are inserted to lock ranges in the tree, and as dirty areas * are found, they are marked writeback. Then the lock bits are removed * and the end_io handler clears the writeback ranges +* +* Return 0 if everything goes well. +* Return <0 for error. */ static int __extent_writepage (struct page *page,struct writeback_control *wbc,struct extent_page_data *epd)
@@ -3500,6 +3513,7 @@ static int __extent_writepage(struct page *page, struct writeback_control *wbc,end_extent_writepage (page, ret, start, page_end); }unlock_page (page); +ASSERT (ret <=0 );return ret; done_unlocked:
@@ -4040,7 +4054,6 @@ static int extent_write_cache_pages(struct address_space *mapping,int extent_write_full_page (struct page *page,struct writeback_control *wbc) {int ret; -int flush_ret;struct extent_page_data epd = { .bio =NULL , .tree = &BTRFS_I (page->mapping->host)->io_tree,
@@ -4049,9 +4062,14 @@ int extent_write_full_page(struct page *page, struct writeback_control *wbc) }; ret =__extent_writepage (page, wbc, &epd); +ASSERT (ret <=0 ); +if (ret <0 ) { +end_write_bio (&epd, ret); +return ret; + } - flush_ret =flush_write_bio (&epd); -BUG_ON (flush_ret <0 ); + ret =flush_write_bio (&epd); +ASSERT (ret <=0 );return ret; }
Signature · 1 line
2.20.1
PATCH 6/12 btrfs: extent_io: Handle error better in btree_write_cache_pages()
In reply to Qu WenruoIn btree_write_cache_pages(), we can only get @ret <= 0. Add an ASSERT() for it just in case.
Then instead of submitting the write bio even we got some error, check the return value first.
If we have already hit some error, just clean up the corrupted or half-baked bio, and return error.
If there is no error so far, then call flush_write_bio() and return the result.
--- fs/btrfs/extent_io.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +6 -3
@@ -3781,7 +3781,6 @@ int btree_write_cache_pages(struct address_space *mapping, .sync_io = wbc->sync_mode == WB_SYNC_ALL, };int ret =0 ; -int flush_ret;int done =0 ;int nr_to_write_done =0 ;struct pagevec pvec;
@@ -3881,8 +3880,12 @@ int btree_write_cache_pages(struct address_space *mapping, index =0 ; goto retry; } - flush_ret =flush_write_bio (&epd); -BUG_ON (flush_ret <0 ); +ASSERT (ret <=0 ); +if (ret <0 ) { +end_write_bio (&epd, ret); +return ret; + } + ret =flush_write_bio (&epd);return ret; }
Signature · 1 line
2.20.1
PATCH 7/12 btrfs: extent_io: Kill the dead branch in extent_write_cache_pages()
In reply to Qu WenruoSince __extent_writepage() will no longer return >0 value, (ret == AOP_WRITEPAGE_ACTIVATE) will never be true.
Kill that dead branch.
--- fs/btrfs/extent_io.c | 5 ----- 1 file changed, 5 deletions(-)
fs/btrfs/extent_io.c +0 -5
@@ -4007,11 +4007,6 @@ static int extent_write_cache_pages(struct address_space *mapping, } ret =__extent_writepage (page, wbc, epd); - -if (unlikely (ret ==AOP_WRITEPAGE_ACTIVATE )) { -unlock_page (page); - ret =0 ; - }if (ret <0 ) {/* * done_index is set past this page,
Signature · 1 line
2.20.1
PATCH 8/12 btrfs: extent_io: Handle error better in extent_write_locked_range()
In reply to Qu WenruoDo proper cleanup if we hit any error in extent_write_locked_range(), and check the return value of flush_write_bio().
--- fs/btrfs/extent_io.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +6 -3
@@ -4075,7 +4075,6 @@ int extent_write_locked_range(struct inode *inode, u64 start, u64 end, int mode) {int ret =0 ; -int flush_ret;struct address_space *mapping = inode->i_mapping;struct extent_io_tree *tree = &BTRFS_I (inode)->io_tree;struct page *page;
@@ -4108,8 +4107,12 @@ int extent_write_locked_range(struct inode *inode, u64 start, u64 end, start +=PAGE_SIZE ; } - flush_ret =flush_write_bio (&epd); -BUG_ON (flush_ret <0 ); +ASSERT (ret <=0 ); +if (ret <0 ) { +end_write_bio (&epd, ret); +return ret; + } + ret =flush_write_bio (&epd);return ret; }
Signature · 1 line
2.20.1
PATCH 9/12 btrfs: extent_io: Kill the BUG_ON() in lock_extent_buffer_for_io()
In reply to Qu WenruoThis function needs some extra check on locked pages and eb.
For error handling we need to unlock locked pages and the eb.
Also add comment for possible return values of lock_extent_buffer_for_io().
There is a rare >0 return value branch, where all pages get locked while write bio is not flushed.
Thankfully it's handled by the only caller, btree_write_cache_pages(), as later write_one_eb() call will trigger submit_one_bio(). So there shouldn't be any problem.
--- fs/btrfs/extent_io.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-)
fs/btrfs/extent_io.c +20 -5
@@ -3526,19 +3526,25 @@ void wait_on_extent_buffer_writeback(struct extent_buffer *eb) TASK_UNINTERRUPTIBLE); } +/* +* Return 0 if nothing went wrong, its pages get locked and submitted. +* Return >0 is mostly the same as 0, except bio is not submitted. +* Return <0 if something went wrong, no page get locked. +*/ static noinline_for_stack intlock_extent_buffer_for_io (struct extent_buffer *eb,struct btrfs_fs_info *fs_info,struct extent_page_data *epd) { -int i, num_pages; +int i, num_pages, failed_page_nr;int flush =0 ;int ret =0 ;if (!btrfs_try_tree_write_lock (eb)) { - flush =1 ; ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) +return ret; + flush =1 ;btrfs_tree_lock (eb); }
@@ -3548,7 +3554,8 @@ lock_extent_buffer_for_io(struct extent_buffer *eb,return 0 ;if (!flush) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) +return ret; flush =1 ; }while (1 ) {
@@ -3590,7 +3597,10 @@ lock_extent_buffer_for_io(struct extent_buffer *eb,if (!trylock_page (p)) {if (!flush) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) { + failed_page_nr = i; + goto err_unlock; + } flush =1 ; }lock_page (p);
@@ -3598,6 +3608,11 @@ lock_extent_buffer_for_io(struct extent_buffer *eb, }return ret; +err_unlock: +/* Unlock these already locked pages */ +for (i =0 ; i < failed_page_nr; i++) +unlock_page (eb->pages[i]); +return ret; }static void end_extent_buffer_writeback (struct extent_buffer *eb)
Signature · 1 line
2.20.1
PATCH 10/12 btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
In reply to Qu Wenruo--- fs/btrfs/extent_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +8 -2
@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) { + done =1 ; +break ; + }lock_page (page); }
@@ -4010,7 +4013,10 @@ static int extent_write_cache_pages(struct address_space *mapping,if (wbc->sync_mode !=WB_SYNC_NONE ) {if (PageWriteback (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) { + done =1 ; +break ; + } }wait_on_page_writeback (page); }
Signature · 1 line
2.20.1
Re: [PATCH v5.1 10/12] btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
Quoted 11 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>--- fs/btrfs/extent_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)fs/btrfs/extent_io.c +0
-0
fs/btrfs/extent_io.c static int extent_write_cache_pages(struct address_space *mapping, · line 4001
Quoted 5 lines
fs/btrfs/extent_io.c +1
-1@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) {
This needs some more explanation why it's correct, there's conditional locking and writeback status manipulation. Thanks.
Re: [PATCH v5.1 10/12] btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
Quoted 7 lines · David Sterba
Quoted 6 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>--- fs/btrfs/extent_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
Quoted 13 lines
fs/btrfs/extent_io.c static int extent_write_cache_pages(struct address_space *mapping, · line 4001
Quoted 10 lines
fs/btrfs/extent_io.c +1
-1@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) {This needs some more explanation why it's correct, there's conditional locking and writeback status manipulation. Thanks.
Because flush_write_bio() handles the cleanup in its endio function.
Thus we don't need extra cleanup in this case.
Thanks,
Qu
Re: [PATCH v5.1 10/12] btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
Quoted 24 lines · Qu Wenruo
Quoted 7 lines · David Sterba
Quoted 6 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>--- fs/btrfs/extent_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)Quoted 13 lines
fs/btrfs/extent_io.c static int extent_write_cache_pages(struct address_space *mapping, · line 4001
Quoted 10 lines
fs/btrfs/extent_io.c +1
-1@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) {This needs some more explanation why it's correct, there's conditional locking and writeback status manipulation. Thanks.
Quoted 3 lines
Because
flush_write_bio()handles the cleanup in its endio function.Thus we don't need extra cleanup in this case.
Non-trivial changes need changelog, that's nothing new. Especially for error handling you have to provide a proof that things don't get broken.
If you spend a week working on the code, everything is trivial, but only for you and for a short period of time. That's why we need good changelogs that make sense to other people and after a long period of time. Thanks.
Re: [PATCH v5.1 10/12] btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
Quoted 29 lines · David Sterba
Quoted 24 lines · Qu Wenruo
Quoted 7 lines · David Sterba
Quoted 6 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>--- fs/btrfs/extent_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)Quoted 13 lines
fs/btrfs/extent_io.c static int extent_write_cache_pages(struct address_space *mapping, · line 4001
Quoted 10 lines
fs/btrfs/extent_io.c +1
-1@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) {This needs some more explanation why it's correct, there's conditional locking and writeback status manipulation. Thanks.
Quoted 3 lines
Because
flush_write_bio()handles the cleanup in its endio function.Thus we don't need extra cleanup in this case.
Quoted 7 lines
Non-trivial changes need changelog, that's nothing new. Especially for error handling you have to provide a proof that things don't get broken.
If you spend a week working on the code, everything is trivial, but only for you and for a short period of time. That's why we need good changelogs that make sense to other people and after a long period of time. Thanks.
That completely makes sense, since even myself need some time to get to the point why the error out works.
I'll update the change log for a comprehensive call path why it works.
Thanks,
Qu
PATCH 10/12 btrfs: extent_io: Kill the BUG_ON() in extent_write_cache_pages()
In reply to Qu WenruoIn extent_write_cache_pages() since flush_write_bio() can return error, we need some extra error handling.
For the first flush_write_bio() since we haven't locked the page, we only need to exit the loop.
For the seconds flush_write_bio() call, we have the page locked, despite that there is nothing special need to handle.
---
changelog:
v5.2:
- Update commit message to explain why the error out behavior is correct.
- Add a missing unlock_page() for the 2nd error out case.
--- fs/btrfs/extent_io.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +9 -2
@@ -3998,7 +3998,10 @@ static int extent_write_cache_pages(struct address_space *mapping,*/ if (!trylock_page (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) { + done =1 ; +break ; + }lock_page (page); }
@@ -4010,7 +4013,11 @@ static int extent_write_cache_pages(struct address_space *mapping,if (wbc->sync_mode !=WB_SYNC_NONE ) {if (PageWriteback (page)) { ret =flush_write_bio (epd); -BUG_ON (ret <0 ); +if (ret <0 ) { +unlock_page (page); + done =1 ; +break ; + } }wait_on_page_writeback (page); }
Signature · 1 line
2.21.0
PATCH 11/12 btrfs: extent_io: Handle error better in extent_writepages()
In reply to Qu WenruoDo proper cleanup if we hit any error in extent_writepages(), and check the return value of flush_write_bio().
--- fs/btrfs/extent_io.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +6 -3
@@ -4141,7 +4141,6 @@ int extent_writepages(struct address_space *mapping, struct writeback_control *wbc) {int ret =0 ; -int flush_ret;struct extent_page_data epd = { .bio =NULL , .tree = &BTRFS_I (mapping->host)->io_tree,
@@ -4150,8 +4149,12 @@ int extent_writepages(struct address_space *mapping, }; ret =extent_write_cache_pages (mapping, wbc, &epd); - flush_ret =flush_write_bio (&epd); -BUG_ON (flush_ret <0 ); +ASSERT (ret <=0 ); +if (ret <0 ) { +end_write_bio (&epd, ret); +return ret; + } + ret =flush_write_bio (&epd);return ret; }
Signature · 1 line
2.20.1
PATCH 12/12 btrfs: Do mandatory tree block check before submitting bio
In reply to Qu WenruoThere are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at btrfs_mark_buffer_dirty() time, as it's not mandatory and only for CONFIG_BTRFS_FS_CHECK_INTEGRITY enabled build, it doesn't help user to detect such problem.
This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The design points are:
- Timing of the check: Tree block write hook
This timing is chosen to reduce the overhead.
The comprehensive check should be as expensive as csum. Doing full check at btrfs_mark_buffer_dirty() is too expensive for end user.
- Loose empty leaf check
Originally for empty leaf, tree-checker will report error if it's not a tree root.
The problem for such check at write time is:
* False alert for tree root created in current transaction In that case, the commit root still needs to be written to disk. And since current root can differ from commit root, then it will cause false alert.
This happens for log tree.
* False alert for relocated tree block
Relocated tree block can be written to disk due to memory pressure, in that case an empty csum tree root can be written to disk and cause false alert, since csum root node hasn't been updated.
Although some more reliable empty leaf check is still kept as is. Namely essential trees (e.g. extent, chunk) should never be empty.
The example error output will be something like:
BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3): delayed_refs has NO entry
--- fs/btrfs/disk-io.c | 10 ++++++++++ fs/btrfs/tree-checker.c | 24 +++++++++++++++++++++--- fs/btrfs/tree-checker.h | 8 ++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +10 -0
@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_write (fs_info, buf); +if (err <0 ) { +btrfs_err (fs_info, +"block=%llu write time tree block corruption detected" , + buf->start); +return err; + }write_extent_buffer (buf, result,0 , csum_size); }
fs/btrfs/tree-checker.c +21 -3
@@ -477,7 +477,7 @@ static int check_leaf_item(struct btrfs_fs_info *fs_info, }static int check_leaf (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf, -bool check_item_data) +bool check_item_data,bool check_empty_leaf) {/* No valid key type is 0, so all key should be larger than this key */ struct btrfs_key prev_key = {0 ,0 ,0 };
@@ -516,6 +516,18 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf, owner);return -EUCLEAN ; } + +/* +* Skip empty leaf check, mostly for write time tree block +* +* Such skip mostly happens for tree block write time, as +* we can't use @owner as accurate owner indicator. +* Case like balance and new tree block created for commit root +* can break owner check easily. +*/ +if (!check_empty_leaf) +return 0 ; + key.objectid = owner; key.type =BTRFS_ROOT_ITEM_KEY ; key.offset = (u64)-1 ;
@@ -636,13 +648,19 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf,int btrfs_check_leaf_full (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, true); +return check_leaf (fs_info, leaf, true, true); }int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, false); +return check_leaf (fs_info, leaf, false, true); +} + +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf) +{ +return check_leaf (fs_info, leaf, false, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)
fs/btrfs/tree-checker.h +8 -0
@@ -23,6 +23,14 @@ int btrfs_check_leaf_full(struct btrfs_fs_info *fs_info,*/ int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf); + +/* +* Write time specific leaf checker. +* Don't check if the empty leaf belongs to a tree root. Mostly for balance +* and new tree created in current transaction. +*/ +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf);int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node);#endif
Signature · 1 line
2.20.1
Re: [PATCH v5.1 12/12] btrfs: Do mandatory tree block check before submitting bio
Quoted 61 lines · Qu Wenruo
There are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at
btrfs_mark_buffer_dirty()time, as it's not mandatory and only forCONFIG_BTRFS_FS_CHECK_INTEGRITYenabled build, it doesn't help user to detect such problem.This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The design points are:
- Timing of the check: Tree block write hook
This timing is chosen to reduce the overhead.
The comprehensive check should be as expensive as csum. Doing full check atbtrfs_mark_buffer_dirty()is too expensive for end user.- Loose empty leaf check
Originally for empty leaf, tree-checker will report error if it's not a tree root.
The problem for such check at write time is:
* False alert for tree root created in current transaction In that case, the commit root still needs to be written to disk. And since current root can differ from commit root, then it will cause false alert.
This happens for log tree.* False alert for relocated tree block
Relocated tree block can be written to disk due to memory pressure, in that case an empty csum tree root can be written to disk and cause false alert, since csum root node hasn't been updated.Although some more reliable empty leaf check is still kept as is. Namely essential trees (e.g. extent, chunk) should never be empty.
The example error output will be something like:
BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3):delayed_refshas NO entryReported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ fs/btrfs/tree-checker.c | 24 +++++++++++++++++++++--- fs/btrfs/tree-checker.h | 8 ++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)fs/btrfs/disk-io.c +0
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else {
fs/btrfs/disk-io.c static int csum_tree_block(struct btrfs_fs_info *fs_info, · line 325
Quoted 10 lines
fs/btrfs/disk-io.c +10
-0+if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_write (fs_info, buf); +if (err <0 ) { +btrfs_err (fs_info, +"block=%llu write time tree block corruption detected" , + buf->start); +return err; + }
This code should be moved in csum_dirty_buffer. Currently there is pending cleanups in csum_tree_block and the final if there will be removed and respective read/write code factored out in csum_dirty_buffer/btree_readpage_end_io_hook.
Eventually csum_tree_block's sole purpose should be to calculate the checksum and nothing more.
Quoted 77 lines
fs/btrfs/disk-io.c +0
-0write_extent_buffer (buf, result,0 , csum_size); }fs/btrfs/tree-checker.c +21
-3@@ -477,7 +477,7 @@ static int check_leaf_item(struct btrfs_fs_info *fs_info, }static int check_leaf (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf, -bool check_item_data) +bool check_item_data,bool check_empty_leaf) {/* No valid key type is 0, so all key should be larger than this key */ struct btrfs_key prev_key = {0 ,0 ,0 };@@ -516,6 +516,18 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf, owner);return -EUCLEAN ; } + +/* +* Skip empty leaf check, mostly for write time tree block +* +* Such skip mostly happens for tree block write time, as +* we can't use @owner as accurate owner indicator. +* Case like balance and new tree block created for commit root +* can break owner check easily. +*/ +if (!check_empty_leaf) +return 0 ; + key.objectid = owner; key.type =BTRFS_ROOT_ITEM_KEY ; key.offset = (u64)-1 ;@@ -636,13 +648,19 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf,int btrfs_check_leaf_full (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, true); +return check_leaf (fs_info, leaf, true, true); }int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, false); +return check_leaf (fs_info, leaf, false, true); +} + +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf) +{ +return check_leaf (fs_info, leaf, false, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)fs/btrfs/tree-checker.h +8
-0@@ -23,6 +23,14 @@ int btrfs_check_leaf_full(struct btrfs_fs_info *fs_info,*/ int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf); + +/* +* Write time specific leaf checker. +* Don't check if the empty leaf belongs to a tree root. Mostly for balance +* and new tree created in current transaction. +*/ +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf);int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node);#endif
Re: [PATCH v5.1 12/12] btrfs: Do mandatory tree block check before submitting bio
[snip]
Quoted 24 lines
Quoted 24 lines
Reported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ fs/btrfs/tree-checker.c | 24 +++++++++++++++++++++--- fs/btrfs/tree-checker.h | 8 ++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)fs/btrfs/disk-io.c +7
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_write (fs_info, buf); +if (err <0 ) { + btrfs_err(fs_info, + "block=%llu write time tree block corruption detected",
Quoted 8 lines
fs/btrfs/disk-io.c static int csum_tree_block(struct btrfs_fs_info *fs_info, · line 325
Quoted 3 lines
fs/btrfs/disk-io.c +3
-0+ buf->start); +return err; + }This code should be moved in
csum_dirty_buffer. Currently there is pending cleanups incsum_tree_blockand the final if there will be removed and respective read/write code factored out in csum_dirty_buffer/btree_readpage_end_io_hook.
I have no preference here.
As long as the timing isn't changed, I'm fine either way.
But at least, from my last check on misc-next, there is no conflict at all. So the pending cleanup isn't so pending right now?
Thanks,
Qu
Quoted 81 lines
Eventually csum_tree_block's sole purpose should be to calculate the checksum and nothing more.
Quoted 77 lines
fs/btrfs/disk-io.c +0
-0write_extent_buffer (buf, result,0 , csum_size); }fs/btrfs/tree-checker.c +21
-3@@ -477,7 +477,7 @@ static int check_leaf_item(struct btrfs_fs_info *fs_info, }static int check_leaf (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf, -bool check_item_data) +bool check_item_data,bool check_empty_leaf) {/* No valid key type is 0, so all key should be larger than this key */ struct btrfs_key prev_key = {0 ,0 ,0 };@@ -516,6 +516,18 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf, owner);return -EUCLEAN ; } + +/* +* Skip empty leaf check, mostly for write time tree block +* +* Such skip mostly happens for tree block write time, as +* we can't use @owner as accurate owner indicator. +* Case like balance and new tree block created for commit root +* can break owner check easily. +*/ +if (!check_empty_leaf) +return 0 ; + key.objectid = owner; key.type =BTRFS_ROOT_ITEM_KEY ; key.offset = (u64)-1 ;@@ -636,13 +648,19 @@ static int check_leaf(struct btrfs_fs_info *fs_info, struct extent_buffer *leaf,int btrfs_check_leaf_full (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, true); +return check_leaf (fs_info, leaf, true, true); }int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (fs_info, leaf, false); +return check_leaf (fs_info, leaf, false, true); +} + +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf) +{ +return check_leaf (fs_info, leaf, false, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)fs/btrfs/tree-checker.h +8
-0@@ -23,6 +23,14 @@ int btrfs_check_leaf_full(struct btrfs_fs_info *fs_info,*/ int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf); + +/* +* Write time specific leaf checker. +* Don't check if the empty leaf belongs to a tree root. Mostly for balance +* and new tree created in current transaction. +*/ +int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, +struct extent_buffer *leaf);int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node);#endif
Re: [PATCH v5.1 12/12] btrfs: Do mandatory tree block check before submitting bio
Quoted 34 lines · Qu Wenruo
[snip]
Quoted 24 lines
Quoted 24 lines
Reported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ fs/btrfs/tree-checker.c | 24 +++++++++++++++++++++--- fs/btrfs/tree-checker.h | 8 ++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)fs/btrfs/disk-io.c +7
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_write (fs_info, buf); +if (err <0 ) { + btrfs_err(fs_info, + "block=%llu write time tree block corruption detected",Quoted 8 lines
fs/btrfs/disk-io.c static int csum_tree_block(struct btrfs_fs_info *fs_info, · line 325
Quoted 3 lines
fs/btrfs/disk-io.c +3
-0+ buf->start); +return err; + }This code should be moved in
csum_dirty_buffer. Currently there is pending cleanups incsum_tree_blockand the final if there will be removed and respective read/write code factored out in csum_dirty_buffer/btree_readpage_end_io_hook.
Quoted 5 lines
I have no preference here.
As long as the timing isn't changed, I'm fine either way.But at least, from my last check on misc-next, there is no conflict at all. So the pending cleanup isn't so pending right now?
The pending changes are probably from Johannes, that are still in the mailinglist and I haven't processed them yet (ie. no misc-next or for-next). My plan is to merge your patchset first, the cleanups can be adapted more easily. As the merge window is approaching, all of that will happen in a week or two.
Quoted 3 lines · Qu Wenruo
Patchset can be fetched from github:
https://github.com/adam900710/linux/tree/write_time_tree_checker
Which is based on v5.0-rc1 tag.
Please base the patches on something new, a recent rc is fine (rc6 or rc7 depends what you've tested).
Quoted 1 line
Also there is no conflict rebasing the patchset to misc-next.
The patches can apply on top of misc-next, but I've seen some hunks applied with offset. While git is good, mismerges can still happen, so if you do the rebase on your side it lowers the risk.
Quoted 68 lines
This patchset has the following 3 features:
- Tree block validation output enhancement
* Output validation failure timing (write time or read time)
* Always output tree block level/key mismatch error message This part is already submitted and reviewed.- Write time tree block validation check
To catch memory corruption either from hardware or kernel. Example output would be:BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3):delayed_refshas NO entry- Better error handling before calling
flush_write_bio()One hidden reason of callingflush_write_bio()under all cases is,flush_write_bio()will trigger endio function and endio function ofepd->biowill free the bio under all cases.
So we're in fact abusingflush_write_bio()as cleanup.Since now
flush_write_bio()has its own return value, we shouldn't callflush_write_bio()no-brain, here we introduce proper cleanup helper,end_write_bio(). Now we callflush_write_bio()like:
New | Old
--------------------------------------------------------------
ret =do_some_evil(&epd); | ret =do_some_evil(&epd);
if (ret < 0) { |flush_write_bio(&epd);
end_write_bio(&epd, ret); | ^^^ submitting half-backedepd->bio?
return ret; | return ret;} | ret = flush_write_bio(&epd); | return ret; |Above code should be more streamline for the error handling part.
Changelog:
v2:
- Unlock locked pages inlock_extent_buffer_for_io()for error handling.
- Added Reviewed-by tags.v3:
- Remove duplicated error message.
- UseIS_ENABLED()macro to replace #ifdef.
- Added Reviewed-by tags.v4:
- Re-organized patch split
Now eachBUG_ON()cleanup has its own patch
- Dig much further into the call sites to eliminate unexpected >0 return May be a little paranoid and abuse someASSERT(), but it should be much safer against further code change.
- Fix the false alert caused by balance and memory pressure The fix it skip owner checker for non-essential tree at write time. Since owner root can't always be reliable, either due to commit root created in current transaction or balance + memory pressure.v5:
- Do proper error-out handling other than relying onflush_write_bio()to clean up.
This has a side effect that no Reviewed-by tags for modified patches.
- New comment for why we don't need to do anything aboutebp->biowhensubmit_one_bio()fails.
- Add some Reviewed-by tag.
Quoted 3 lines
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.
The patchset is in a good shape, I like the detailed changelogs and example messages. I'll give it some more testing though merging to 5.1 does not seem to be feasible now but we'll see. The improved error handling is not really a feature so a late pull request could be ok for that.
Thanks.
Quoted 7 lines · David Sterba
Quoted 3 lines · Qu Wenruo
Patchset can be fetched from github:
https://github.com/adam900710/linux/tree/write_time_tree_checker
Which is based on v5.0-rc1 tag.Please base the patches on something new, a recent rc is fine (rc6 or rc7 depends what you've tested).
Any recommended workflow for different timing point? E.g. always rebase to latest rc?
My current (and now is proven not proper) workflow is to stick with -rc1, and cherry pick known fixes related to test.
This provides a stable base run to spot regression caused by my patches.
Quoted 6 lines
Quoted 2 lines
Also there is no conflict rebasing the patchset to misc-next.
The patches can apply on top of misc-next, but I've seen some hunks applied with offset. While git is good, mismerges can still happen, so if you do the rebase on your side it lowers the risk.
No problem, I'll change the my workflow to more proper one.
Quoted 69 lines
Quoted 69 lines
This patchset has the following 3 features:
- Tree block validation output enhancement
* Output validation failure timing (write time or read time)
* Always output tree block level/key mismatch error message This part is already submitted and reviewed.- Write time tree block validation check
To catch memory corruption either from hardware or kernel. Example output would be:BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3):delayed_refshas NO entry- Better error handling before calling
flush_write_bio()One hidden reason of callingflush_write_bio()under all cases is,flush_write_bio()will trigger endio function and endio function ofepd->biowill free the bio under all cases.
So we're in fact abusingflush_write_bio()as cleanup.Since now
flush_write_bio()has its own return value, we shouldn't callflush_write_bio()no-brain, here we introduce proper cleanup helper,end_write_bio(). Now we callflush_write_bio()like:
New | Old
--------------------------------------------------------------
ret =do_some_evil(&epd); | ret =do_some_evil(&epd);
if (ret < 0) { |flush_write_bio(&epd);
end_write_bio(&epd, ret); | ^^^ submitting half-backedepd->bio?
return ret; | return ret;} | ret = flush_write_bio(&epd); | return ret; |Above code should be more streamline for the error handling part.
Changelog:
v2:
- Unlock locked pages inlock_extent_buffer_for_io()for error handling.
- Added Reviewed-by tags.v3:
- Remove duplicated error message.
- UseIS_ENABLED()macro to replace #ifdef.
- Added Reviewed-by tags.v4:
- Re-organized patch split
Now eachBUG_ON()cleanup has its own patch
- Dig much further into the call sites to eliminate unexpected >0 return May be a little paranoid and abuse someASSERT(), but it should be much safer against further code change.
- Fix the false alert caused by balance and memory pressure The fix it skip owner checker for non-essential tree at write time. Since owner root can't always be reliable, either due to commit root created in current transaction or balance + memory pressure.v5:
- Do proper error-out handling other than relying onflush_write_bio()to clean up.
This has a side effect that no Reviewed-by tags for modified patches.
- New comment for why we don't need to do anything aboutebp->biowhensubmit_one_bio()fails.
- Add some Reviewed-by tag.
Quoted 7 lines
Quoted 3 lines
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.The patchset is in a good shape, I like the detailed changelogs and example messages. I'll give it some more testing though merging to 5.1 does not seem to be feasible now but we'll see.
I'm fine with v5.2.
Just wonder if I need to rebase to v5.1-rc1 later on?
Thanks,
Qu
Quoted 6 lines
The improved error
handling is not really a feature so a late pull request could be ok for that.Thanks.
Quoted 8 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Quoted 3 lines · Qu Wenruo
Patchset can be fetched from github:
https://github.com/adam900710/linux/tree/write_time_tree_checker
Which is based on v5.0-rc1 tag.
Quoted 5 lines
Quoted 2 lines
Please base the patches on something new, a recent rc is fine (rc6 or rc7 depends what you've tested).
Any recommended workflow for different timing point? E.g. always rebase to latest rc?
It depends on the phase of the development cycle, last released rc is a good default choice unless you have reasons not to use it. Think about it as "what does happen when this gets merged to the expected target branch?". If you send a fix for current master, then it's perfect to use the last rc in master, or base the patch on the branch that gets pulled there (eg. misc-4.20).
Quoted 3 lines
My current (and now is proven not proper) workflow is to stick with -rc1, and cherry pick known fixes related to test.
This provides a stable base run to spot regression caused by my patches.
You could use that as a reference point, but when the patches get merged, it's on a more recent codebase so it's better that you verify that it works on your side. The rc1 is not a very good base in general as there are so many changes in the whole kernel that need to stabilize.
Quoted 4 lines · Qu Wenruo
Patchset can be fetched from github:
https://github.com/adam900710/linux/tree/write_time_tree_checker
Which is based on v5.0-rc1 tag.
Also there is no conflict rebasing the patchset to misc-next.
Now the github branch rebased to v5.0-rc7 tag.
And ran tests of btrfs auto group, no new regressions found.
Git is clever enough in this rebase, so I bother mail bombing the mail list for another minor update.
Thanks,
Qu
Quoted 93 lines
This patchset has the following 3 features:
- Tree block validation output enhancement
* Output validation failure timing (write time or read time)
* Always output tree block level/key mismatch error message This part is already submitted and reviewed.- Write time tree block validation check
To catch memory corruption either from hardware or kernel. Example output would be:BTRFS critical (device dm-3): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0) BTRFS error (device dm-3): block=1350630375424 write time tree block corruption detected
BTRFS: error (device dm-3) in btrfs_commit_transaction:2220: errno=-5 IO failure (Error while writing out transaction)
BTRFS info (device dm-3): forced readonly
BTRFS warning (device dm-3): Skipping commit of aborted transaction.
BTRFS: error (device dm-3) in cleanup_transaction:1839: errno=-5 IO failure
BTRFS info (device dm-3):delayed_refshas NO entry- Better error handling before calling
flush_write_bio()One hidden reason of callingflush_write_bio()under all cases is,flush_write_bio()will trigger endio function and endio function ofepd->biowill free the bio under all cases.
So we're in fact abusingflush_write_bio()as cleanup.Since now
flush_write_bio()has its own return value, we shouldn't callflush_write_bio()no-brain, here we introduce proper cleanup helper,end_write_bio(). Now we callflush_write_bio()like:
New | Old
--------------------------------------------------------------
ret =do_some_evil(&epd); | ret =do_some_evil(&epd);
if (ret < 0) { |flush_write_bio(&epd);
end_write_bio(&epd, ret); | ^^^ submitting half-backedepd->bio?
return ret; | return ret;} | ret = flush_write_bio(&epd); | return ret; |Above code should be more streamline for the error handling part.
Changelog:
v2:
- Unlock locked pages inlock_extent_buffer_for_io()for error handling.
- Added Reviewed-by tags.v3:
- Remove duplicated error message.
- UseIS_ENABLED()macro to replace #ifdef.
- Added Reviewed-by tags.v4:
- Re-organized patch split
Now eachBUG_ON()cleanup has its own patch
- Dig much further into the call sites to eliminate unexpected >0 return May be a little paranoid and abuse someASSERT(), but it should be much safer against further code change.
- Fix the false alert caused by balance and memory pressure The fix it skip owner checker for non-essential tree at write time. Since owner root can't always be reliable, either due to commit root created in current transaction or balance + memory pressure.v5:
- Do proper error-out handling other than relying onflush_write_bio()to clean up.
This has a side effect that no Reviewed-by tags for modified patches.
- New comment for why we don't need to do anything aboutebp->biowhensubmit_one_bio()fails.
- Add some Reviewed-by tag.v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.Qu Wenruo (12):
btrfs: Always output error message when key/level verification fails
btrfs:extent_io: Kill the forward declaration offlush_write_bio()
btrfs: disk-io: Show the timing of corrupted tree block explicitly
btrfs:extent_io: Move theBUG_ON()inflush_write_bio()one level up
btrfs:extent_io: Handle error better inextent_write_full_page()
btrfs:extent_io: Handle error better inbtree_write_cache_pages()
btrfs:extent_io: Kill the dead branch inextent_write_cache_pages()
btrfs:extent_io: Handle error better inextent_write_locked_range()
btrfs:extent_io: Kill theBUG_ON()inlock_extent_buffer_for_io()
btrfs:extent_io: Kill theBUG_ON()inextent_write_cache_pages()
btrfs:extent_io: Handle error better inextent_writepages()
btrfs: Do mandatory tree block check before submitting biofs/btrfs/disk-io.c | 23 ++++-- fs/btrfs/extent_io.c | 168 ++++++++++++++++++++++++++++------------ fs/btrfs/tree-checker.c | 24 +++++- fs/btrfs/tree-checker.h | 8 ++ 4 files changed, 164 insertions(+), 59 deletions(-)
Quoted 3 lines · Qu Wenruo
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.
I see reports from tests btrfs/124 and btrfs/125, the tested branch was for-next-20190220. Something's wrong, check the large numbers of checksums from the messages, eg.
hex(506381209866536711) = 0x707070707070707
hex(217020518514230019) = 0x303030303030303
hex(1085102592571150095) = 0xf0f0f0f0f0f0f0f
hex(72340172838076673) = 0x101010101010101
there's a pattern. And the number of the 'read time' messages is quite high to be some random fluke.
[ 2847.704946] BTRFS: device fsid 8b8c51ce-59de-439f-847a-bcfce3e10a5d devid 1 transid 7 /dev/vdb [ 2847.710386] BTRFS info (device vdb): allowing degraded mounts [ 2847.713405] BTRFS info (device vdb): disk space caching is enabled [ 2847.716586] BTRFS info (device vdb): has skinny extents [ 2847.753599] BTRFS warning (device vdb): devid 2 uuid 307c73fe-349e-4b34-8233-4e35431aae31 is missing [ 2847.758479] BTRFS warning (device vdb): devid 2 uuid 307c73fe-349e-4b34-8233-4e35431aae31 is missing [ 2859.881984] BTRFS: device fsid e4579131-1621-417b-8464-cfaf6cd15204 devid 1 transid 241 /dev/vda [ 2859.896416] BTRFS info (device vdb): disk space caching is enabled [ 2859.899049] BTRFS info (device vdb): has skinny extents [ 2859.992305] BTRFS error (device vdb): bad tree block start, want 32161792 have 0 [ 2859.996611] BTRFS error (device vdb): block=32161792 read time tree block corruption detected [ 2860.001694] BTRFS info (device vdb): read error corrected: ino 0 off 32161792 (dev /dev/vdc sector 21856) [ 2860.007079] BTRFS info (device vdb): read error corrected: ino 0 off 32165888 (dev /dev/vdc sector 21864) [ 2860.012447] BTRFS info (device vdb): read error corrected: ino 0 off 32169984 (dev /dev/vdc sector 21872) [ 2860.017817] BTRFS info (device vdb): read error corrected: ino 0 off 32174080 (dev /dev/vdc sector 21880) [ 2860.023669] BTRFS error (device vdb): bad tree block start, want 32243712 have 0 [ 2860.027865] BTRFS error (device vdb): block=32243712 read time tree block corruption detected [ 2860.032961] BTRFS info (device vdb): read error corrected: ino 0 off 32243712 (dev /dev/vdc sector 22016) [ 2860.037602] BTRFS info (device vdb): read error corrected: ino 0 off 32247808 (dev /dev/vdc sector 22024) [ 2860.042185] BTRFS info (device vdb): read error corrected: ino 0 off 32251904 (dev /dev/vdc sector 22032) [ 2860.046738] BTRFS info (device vdb): read error corrected: ino 0 off 32256000 (dev /dev/vdc sector 22040) [ 2860.089197] BTRFS info (device vdb): balance: start -d -m -s [ 2860.092227] BTRFS info (device vdb): relocating block group 2479882240 flags data [ 2860.098064] BTRFS error (device vdb): bad tree block start, want 31653888 have 0 [ 2860.102289] BTRFS error (device vdb): block=31653888 read time tree block corruption detected [ 2860.107409] BTRFS info (device vdb): read error corrected: ino 0 off 31653888 (dev /dev/vdc sector 20864) [ 2860.111997] BTRFS info (device vdb): read error corrected: ino 0 off 31657984 (dev /dev/vdc sector 20872) [ 2860.166451] BTRFS error (device vdb): bad tree block start, want 31670272 have 0 [ 2860.167172] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.167375] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.167377] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.167384] BTRFS error (device vdb): bad tree block start, want 31719424 have 0 [ 2860.167386] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.167394] BTRFS error (device vdb): bad tree block start, want 31735808 have 0 [ 2860.167396] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.170120] BTRFS error (device vdb): block=31670272 read time tree block corruption detected [ 2860.174308] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.205288] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.208875] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.244462] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.248013] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.284017] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.315705] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.336001] BTRFS error (device vdb): block=31752192 read time tree block corruption detected [ 2860.336054] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.339786] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.366693] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.389495] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.418551] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.418713] BTRFS error (device vdb): block=31883264 read time tree block corruption detected [ 2860.418721] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.418736] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.418744] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.419721] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.462401] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.490821] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.506718] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.535013] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.568602] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.590434] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.590532] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.590677] BTRFS error (device vdb): block=31997952 read time tree block corruption detected [ 2860.590688] BTRFS error (device vdb): block=32014336 read time tree block corruption detected
[ 2860.590697] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.590705] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.625834] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.658723] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.696644] BTRFS error (device vdb): block=32014336 read time tree block corruption detected [ 2860.735014] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.743240] BTRFS error (device vdb): block=32096256 read time tree block corruption detected [ 2860.780296] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.807073] BTRFS error (device vdb): block=32063488 read time tree block corruption detected [ 2860.807173] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2860.841582] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2862.476657] BTRFS info (device vdb): found 3 extents [ 2864.186610] BTRFS info (device vdb): found 3 extents [ 2864.190233] BTRFS error (device vdb): block=31637504 read time tree block corruption detected [ 2864.306413] BTRFS info (device vdb): relocating block group 2446327808 flags system [ 2864.418449] BTRFS info (device vdb): relocating block group 2177892352 flags metadata [ 2864.530002] BTRFS info (device vdb): relocating block group 1104150528 flags data|raid1 [ 2864.582024] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2097152 csum 0x6e65c49f expected csum 0x98f94189 mirror 1 [ 2864.582156] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2162688 csum 0x5ce6193f expected csum 0x98f94189 mirror 1 [ 2864.587210] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2101248 csum 0x6e65c49f expected csum 0x98f94189 mirror 1 [ 2864.594057] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2228224 csum 0x89780ae9 expected csum 0x98f94189 mirror 1 [ 2864.594156] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2359296 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594200] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2363392 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594228] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2367488 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594267] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2371584 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594291] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2490368 csum 0xddddd09d expected csum 0x98f94189 mirror 1 [ 2864.594294] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2375680 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.686125] BTRFS error (device vdb): parent transid verify failed on 30539776 wanted 8 found 5 [ 2864.859090] BTRFS error (device vdb): parent transid verify failed on 30556160 wanted 8 found 5 [ 2864.936897] BTRFS error (device vdb): parent transid verify failed on 30572544 wanted 8 found 6 [ 2865.003123] repair_io_failure: 5828 callbacks suppressed [ 2865.003127] BTRFS info (device vdb): read error corrected: ino 260 off 43462656 (dev /dev/vdc sector 2200472) [ 2865.003141] BTRFS info (device vdb): read error corrected: ino 260 off 43683840 (dev /dev/vdc sector 2200904) [ 2865.003210] BTRFS info (device vdb): read error corrected: ino 260 off 43659264 (dev /dev/vdc sector 2200856) [ 2865.003223] BTRFS info (device vdb): read error corrected: ino 260 off 43651072 (dev /dev/vdc sector 2200840) [ 2865.005980] BTRFS info (device vdb): read error corrected: ino 260 off 43663360 (dev /dev/vdc sector 2200864) [ 2865.006063] BTRFS info (device vdb): read error corrected: ino 260 off 43446272 (dev /dev/vdc sector 2200440) [ 2865.006098] BTRFS info (device vdb): read error corrected: ino 260 off 43638784 (dev /dev/vdc sector 2200816) [ 2865.010374] BTRFS info (device vdb): read error corrected: ino 260 off 43655168 (dev /dev/vdc sector 2200848) [ 2865.010425] BTRFS info (device vdb): read error corrected: ino 260 off 43700224 (dev /dev/vdc sector 2200936) [ 2865.010664] BTRFS info (device vdb): read error corrected: ino 260 off 43843584 (dev /dev/vdc sector 2201216) [ 2865.112654] BTRFS error (device vdb): parent transid verify failed on 30588928 wanted 8 found 6 [ 2865.267513] btree_readpage_end_io_hook: 34 callbacks suppressed [ 2865.267518] BTRFS error (device vdb): bad tree block start, want 30670848 have 0 [ 2865.267544] BTRFS error (device vdb): bad tree block start, want 30687232 have 0 [ 2865.267549] BTRFS error (device vdb): bad tree block start, want 30703616 have 0 [ 2865.267551] BTRFS error (device vdb): block=30703616 read time tree block corruption detected [ 2865.267578] BTRFS error (device vdb): bad tree block start, want 30736384 have 0 [ 2865.267580] BTRFS error (device vdb): block=30736384 read time tree block corruption detected [ 2865.267823] BTRFS error (device vdb): bad tree block start, want 30720000 have 0 [ 2865.267825] BTRFS error (device vdb): block=30720000 read time tree block corruption detected [ 2865.276321] BTRFS error (device vdb): block=30670848 read time tree block corruption detected [ 2865.380720] BTRFS error (device vdb): block=30687232 read time tree block corruption detected [ 2865.477717] BTRFS error (device vdb): bad tree block start, want 30687232 have 0 [ 2865.487564] BTRFS error (device vdb): block=30687232 read time tree block corruption detected [ 2865.628551] BTRFS error (device vdb): bad tree block start, want 30703616 have 0 [ 2865.636510] BTRFS error (device vdb): block=30703616 read time tree block corruption detected [ 2865.842136] BTRFS error (device vdb): bad tree block start, want 30720000 have 0 [ 2865.845901] BTRFS error (device vdb): block=30720000 read time tree block corruption detected [ 2866.072213] BTRFS error (device vdb): bad tree block start, want 30736384 have 0 [ 2866.081387] BTRFS error (device vdb): block=30736384 read time tree block corruption detected [ 2866.253507] BTRFS error (device vdb): bad tree block start, want 30769152 have 0 [ 2866.253527] BTRFS error (device vdb): block=30801920 read time tree block corruption detected
[ 2866.253538] BTRFS error (device vdb): block=30752768 read time tree block corruption detected [ 2866.253550] BTRFS error (device vdb): block=30818304 read time tree block corruption detected [ 2866.263700] BTRFS error (device vdb): block=30769152 read time tree block corruption detected [ 2866.273628] BTRFS error (device vdb): block=30785536 read time tree block corruption detected [ 2866.474143] BTRFS error (device vdb): block=30769152 read time tree block corruption detected [ 2866.576807] BTRFS error (device vdb): block=30785536 read time tree block corruption detected [ 2866.793977] BTRFS error (device vdb): block=30801920 read time tree block corruption detected [ 2866.997914] BTRFS error (device vdb): block=30818304 read time tree block corruption detected [ 2867.157625] BTRFS error (device vdb): block=30834688 read time tree block corruption detected [ 2867.157684] BTRFS error (device vdb): block=30851072 read time tree block corruption detected [ 2867.162566] BTRFS error (device vdb): block=30867456 read time tree block corruption detected [ 2867.162581] BTRFS error (device vdb): block=30883840 read time tree block corruption detected [ 2867.162591] BTRFS error (device vdb): block=30900224 read time tree block corruption detected [ 2867.393541] BTRFS error (device vdb): block=30851072 read time tree block corruption detected [ 2867.556641] BTRFS error (device vdb): block=30867456 read time tree block corruption detected [ 2867.787270] BTRFS error (device vdb): block=30883840 read time tree block corruption detected [ 2867.997127] BTRFS error (device vdb): block=30900224 read time tree block corruption detected [ 2868.205510] BTRFS error (device vdb): block=30916608 read time tree block corruption detected [ 2868.205532] BTRFS error (device vdb): block=30949376 read time tree block corruption detected [ 2868.205536] BTRFS error (device vdb): block=30932992 read time tree block corruption detected [ 2868.205562] BTRFS error (device vdb): block=30965760 read time tree block corruption detected [ 2868.205569] BTRFS error (device vdb): block=30982144 read time tree block corruption detected [ 2868.388272] BTRFS error (device vdb): block=30932992 read time tree block corruption detected [ 2868.592079] BTRFS error (device vdb): block=30949376 read time tree block corruption detected [ 2868.794379] BTRFS error (device vdb): block=30965760 read time tree block corruption detected [ 2868.999973] BTRFS error (device vdb): block=30982144 read time tree block corruption detected [ 2869.207738] BTRFS error (device vdb): block=31014912 read time tree block corruption detected [ 2869.207772] BTRFS error (device vdb): block=30998528 read time tree block corruption detected [ 2869.207849] BTRFS error (device vdb): block=31031296 read time tree block corruption detected [ 2869.207859] BTRFS error (device vdb): block=31047680 read time tree block corruption detected [ 2869.207865] BTRFS error (device vdb): block=31064064 read time tree block corruption detected [ 2869.430131] BTRFS error (device vdb): block=31014912 read time tree block corruption detected [ 2869.583069] btrfs_print_data_csum_error: 178079 callbacks suppressed [ 2869.583073] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400269312 csum 0x5ce6193f expected csum 0x98f94189 mirror 1 [ 2869.583088] BTRFS warning (device vdb): csum failed root -9 ino 260 off 399634432 csum 0x362910d4 expected csum 0x98f94189 mirror 1 [ 2869.584130] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400273408 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.586502] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400277504 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.586609] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400130048 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.601709] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400281600 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.601732] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400134144 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.606867] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400138240 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.606878] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400285696 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.606912] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400289792 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.678550] BTRFS error (device vdb): block=31031296 read time tree block corruption detected [ 2869.782906] BTRFS error (device vdb): block=31047680 read time tree block corruption detected [ 2870.010786] repair_io_failure: 89687 callbacks suppressed [ 2870.010790] BTRFS info (device vdb): read error corrected: ino 260 off 430133248 (dev /dev/vdc sector 2955688) [ 2870.010830] BTRFS info (device vdb): read error corrected: ino 260 off 430178304 (dev /dev/vdc sector 2955776) [ 2870.010844] BTRFS info (device vdb): read error corrected: ino 260 off 430116864 (dev /dev/vdc sector 2955656) [ 2870.011103] BTRFS info (device vdb): read error corrected: ino 260 off 430182400 (dev /dev/vdc sector 2955784) [ 2870.011133] BTRFS info (device vdb): read error corrected: ino 260 off 430194688 (dev /dev/vdc sector 2955808) [ 2870.011168] BTRFS info (device vdb): read error corrected: ino 260 off 430112768 (dev /dev/vdc sector 2955648) [ 2870.011313] BTRFS info (device vdb): read error corrected: ino 260 off 430071808 (dev /dev/vdc sector 2955568) [ 2870.011328] BTRFS info (device vdb): read error corrected: ino 260 off 430047232 (dev /dev/vdc sector 2955520) [ 2870.011396] BTRFS info (device vdb): read error corrected: ino 260 off 430051328 (dev /dev/vdc sector 2955528) [ 2870.011456] BTRFS info (device vdb): read error corrected: ino 260 off 429916160 (dev /dev/vdc sector 2955264) [ 2870.017915] BTRFS error (device vdb): block=31064064 read time tree block corruption detected [ 2870.254803] BTRFS error (device vdb): block=31096832 read time tree block corruption detected [ 2870.254814] BTRFS error (device vdb): block=31080448 read time tree block corruption detected [ 2870.254816] BTRFS error (device vdb): block=31113216 read time tree block corruption detected [ 2870.254851] BTRFS error (device vdb): block=31129600 read time tree block corruption detected [ 2870.254868] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2870.468150] btree_readpage_end_io_hook: 40 callbacks suppressed
[ 2870.468154] BTRFS error (device vdb): bad tree block start, want 31096832 have 0 [ 2870.481275] BTRFS error (device vdb): block=31096832 read time tree block corruption detected [ 2870.659389] BTRFS error (device vdb): bad tree block start, want 31113216 have 0 [ 2870.663667] BTRFS error (device vdb): block=31113216 read time tree block corruption detected [ 2870.919169] BTRFS error (device vdb): bad tree block start, want 31129600 have 0 [ 2870.927705] BTRFS error (device vdb): block=31129600 read time tree block corruption detected [ 2871.058301] BTRFS error (device vdb): bad tree block start, want 31145984 have 0 [ 2871.058336] BTRFS error (device vdb): bad tree block start, want 31162368 have 0 [ 2871.072181] BTRFS error (device vdb): bad tree block start, want 31178752 have 0 [ 2871.072185] BTRFS error (device vdb): bad tree block start, want 31211520 have 72340172838076673 [ 2871.072187] BTRFS error (device vdb): block=31211520 read time tree block corruption detected [ 2871.072197] BTRFS error (device vdb): bad tree block start, want 31227904 have 72340172838076673 [ 2871.072198] BTRFS error (device vdb): block=31227904 read time tree block corruption detected [ 2871.072264] BTRFS error (device vdb): bad tree block start, want 31195136 have 188978565376 [ 2871.072266] BTRFS error (device vdb): block=31195136 read time tree block corruption detected [ 2871.076187] BTRFS error (device vdb): block=31162368 read time tree block corruption detected [ 2871.081901] BTRFS error (device vdb): block=31178752 read time tree block corruption detected [ 2871.091522] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2871.686871] BTRFS error (device vdb): bad tree block start, want 31145984 have 0 [ 2871.701355] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2871.883082] BTRFS error (device vdb): block=31162368 read time tree block corruption detected [ 2872.070858] BTRFS error (device vdb): block=31227904 read time tree block corruption detected [ 2872.209604] BTRFS error (device vdb): block=31195136 read time tree block corruption detected [ 2872.524825] BTRFS error (device vdb): block=31211520 read time tree block corruption detected [ 2872.773657] BTRFS error (device vdb): block=31260672 read time tree block corruption detected [ 2872.773661] BTRFS error (device vdb): block=31277056 read time tree block corruption detected [ 2872.773668] BTRFS error (device vdb): block=31244288 read time tree block corruption detected [ 2872.773674] BTRFS error (device vdb): block=31293440 read time tree block corruption detected [ 2872.773681] BTRFS error (device vdb): block=31309824 read time tree block corruption detected [ 2872.930060] BTRFS error (device vdb): block=31260672 read time tree block corruption detected [ 2873.179606] BTRFS error (device vdb): block=31277056 read time tree block corruption detected [ 2873.346103] BTRFS error (device vdb): block=31293440 read time tree block corruption detected [ 2873.517863] BTRFS error (device vdb): block=31309824 read time tree block corruption detected [ 2873.705657] BTRFS error (device vdb): block=31326208 read time tree block corruption detected [ 2873.705665] BTRFS error (device vdb): block=31358976 read time tree block corruption detected [ 2873.705735] BTRFS error (device vdb): block=31375360 read time tree block corruption detected [ 2873.705743] BTRFS error (device vdb): block=31391744 read time tree block corruption detected [ 2873.706274] BTRFS error (device vdb): block=31342592 read time tree block corruption detected [ 2874.081625] BTRFS error (device vdb): block=31342592 read time tree block corruption detected [ 2874.408558] BTRFS error (device vdb): block=31358976 read time tree block corruption detected [ 2874.587058] btrfs_print_data_csum_error: 132544 callbacks suppressed [ 2874.587063] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688840704 csum 0x434957d2 expected csum 0x98f94189 mirror 1 [ 2874.587069] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688975872 csum 0x0107a017 expected csum 0x98f94189 mirror 1 [ 2874.587074] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688984064 csum 0x7ef574ea expected csum 0x98f94189 mirror 1 [ 2874.587090] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688988160 csum 0xf077c653 expected csum 0x98f94189 mirror 1 [ 2874.587096] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688992256 csum 0xcd9f8290 expected csum 0x98f94189 mirror 1 [ 2874.587109] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689090560 csum 0x9d2a64a2 expected csum 0x98f94189 mirror 1 [ 2874.587111] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688996352 csum 0x60f06185 expected csum 0x98f94189 mirror 1 [ 2874.587124] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689094656 csum 0x6ad9235d expected csum 0x98f94189 mirror 1 [ 2874.587129] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689098752 csum 0xe2cb848d expected csum 0x98f94189 mirror 1 [ 2874.597159] BTRFS error (device vdb): block=31375360 read time tree block corruption detected [ 2874.881178] BTRFS error (device vdb): block=31391744 read time tree block corruption detected [ 2874.892860] BTRFS error (device vdb): block=31408128 read time tree block corruption detected [ 2874.892885] BTRFS error (device vdb): block=31457280 read time tree block corruption detected [ 2874.892932] BTRFS error (device vdb): block=31440896 read time tree block corruption detected [ 2874.893058] BTRFS error (device vdb): block=31473664 read time tree block corruption detected [ 2874.893063] BTRFS error (device vdb): block=31490048 read time tree block corruption detected [ 2874.952101] BTRFS error (device vdb): block=31424512 read time tree block corruption detected [ 2875.011131] repair_io_failure: 65834 callbacks suppressed [ 2875.011136] BTRFS info (device vdb): read error corrected: ino 260 off 711217152 (dev /dev/vdc sector 3504680) [ 2875.011145] BTRFS info (device vdb): read error corrected: ino 260 off 710975488 (dev /dev/vdc sector 3504208) [ 2875.011167] BTRFS info (device vdb): read error corrected: ino 260 off 711225344 (dev /dev/vdc sector 3504696) [ 2875.011266] BTRFS info (device vdb): read error corrected: ino 260 off 711041024 (dev /dev/vdc sector 3504336)
[ 2875.011310] BTRFS info (device vdb): read error corrected: ino 260 off 711348224 (dev /dev/vdc sector 3504936) [ 2875.014273] BTRFS info (device vdb): read error corrected: ino 260 off 710987776 (dev /dev/vdc sector 3504232) [ 2875.014288] BTRFS info (device vdb): read error corrected: ino 260 off 710979584 (dev /dev/vdc sector 3504216) [ 2875.014330] BTRFS info (device vdb): read error corrected: ino 260 off 710983680 (dev /dev/vdc sector 3504224) [ 2875.014400] BTRFS info (device vdb): read error corrected: ino 260 off 710995968 (dev /dev/vdc sector 3504248) [ 2875.014595] BTRFS info (device vdb): read error corrected: ino 260 off 711266304 (dev /dev/vdc sector 3504776) [ 2875.147908] BTRFS error (device vdb): block=31408128 read time tree block corruption detected [ 2875.380354] BTRFS error (device vdb): block=31424512 read time tree block corruption detected [ 2875.565748] btree_readpage_end_io_hook: 30 callbacks suppressed [ 2875.565753] BTRFS error (device vdb): bad tree block start, want 31457280 have 0 [ 2875.581223] BTRFS error (device vdb): block=31457280 read time tree block corruption detected [ 2875.788290] BTRFS error (device vdb): bad tree block start, want 31473664 have 1085102592571150095 [ 2875.798989] BTRFS error (device vdb): block=31473664 read time tree block corruption detected [ 2877.767277] BTRFS error (device vdb): bad tree block start, want 31490048 have 1085102592571150095 [ 2877.772654] BTRFS error (device vdb): block=31490048 read time tree block corruption detected [ 2878.524728] BTRFS error (device vdb): bad tree block start, want 31506432 have 1085102592571150095 [ 2878.524766] BTRFS error (device vdb): bad tree block start, want 31522816 have 217020518514230019 [ 2878.524798] BTRFS error (device vdb): bad tree block start, want 31539200 have 217020518514230019 [ 2878.524800] BTRFS error (device vdb): block=31539200 read time tree block corruption detected [ 2878.524835] BTRFS error (device vdb): bad tree block start, want 31571968 have 217020518514230019 [ 2878.524837] BTRFS error (device vdb): block=31571968 read time tree block corruption detected [ 2878.524848] BTRFS error (device vdb): bad tree block start, want 31555584 have 217020518514230019 [ 2878.524849] BTRFS error (device vdb): block=31555584 read time tree block corruption detected [ 2878.529852] BTRFS error (device vdb): block=31506432 read time tree block corruption detected [ 2878.534202] BTRFS error (device vdb): block=31522816 read time tree block corruption detected [ 2879.386342] BTRFS error (device vdb): bad tree block start, want 31522816 have 217020518514230019 [ 2879.400043] BTRFS error (device vdb): block=31522816 read time tree block corruption detected [ 2879.638124] btrfs_print_data_csum_error: 59720 callbacks suppressed [ 2879.638128] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817057792 csum 0x576bf9a1 expected csum 0x98f94189 mirror 1 [ 2879.638139] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816893952 csum 0x7d29d791 expected csum 0x98f94189 mirror 1 [ 2879.638760] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816898048 csum 0x4c64848f expected csum 0x98f94189 mirror 1 [ 2879.638814] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816902144 csum 0x0107a017 expected csum 0x98f94189 mirror 1 [ 2879.638856] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817061888 csum 0x3dc46475 expected csum 0x98f94189 mirror 1 [ 2879.638890] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817065984 csum 0x63f98f58 expected csum 0x98f94189 mirror 1 [ 2879.638915] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817070080 csum 0x2a2250eb expected csum 0x98f94189 mirror 1 [ 2879.638940] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817074176 csum 0x90ece4b7 expected csum 0x98f94189 mirror 1 [ 2879.640899] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817000448 csum 0xe61df820 expected csum 0x98f94189 mirror 1 [ 2879.640961] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817078272 csum 0x222426ee expected csum 0x98f94189 mirror 1 [ 2880.015085] repair_io_failure: 25064 callbacks suppressed [ 2880.015089] BTRFS info (device vdb): read error corrected: ino 260 off 819994624 (dev /dev/vdc sector 3717136) [ 2880.015186] BTRFS info (device vdb): read error corrected: ino 260 off 819200000 (dev /dev/vdc sector 3715584) [ 2880.017126] BTRFS info (device vdb): read error corrected: ino 260 off 819195904 (dev /dev/vdc sector 3715576) [ 2880.017197] BTRFS info (device vdb): read error corrected: ino 260 off 819204096 (dev /dev/vdc sector 3715592) [ 2880.017226] BTRFS info (device vdb): read error corrected: ino 260 off 819212288 (dev /dev/vdc sector 3715608) [ 2880.017359] BTRFS info (device vdb): read error corrected: ino 260 off 819216384 (dev /dev/vdc sector 3715616) [ 2880.017392] BTRFS info (device vdb): read error corrected: ino 260 off 819220480 (dev /dev/vdc sector 3715624) [ 2880.017430] BTRFS info (device vdb): read error corrected: ino 260 off 819224576 (dev /dev/vdc sector 3715632) [ 2880.017598] BTRFS info (device vdb): read error corrected: ino 260 off 819228672 (dev /dev/vdc sector 3715640) [ 2880.017706] BTRFS info (device vdb): read error corrected: ino 260 off 819232768 (dev /dev/vdc sector 3715648) [ 2880.948226] BTRFS error (device vdb): bad tree block start, want 31539200 have 217020518514230019 [ 2880.952594] BTRFS error (device vdb): block=31539200 read time tree block corruption detected [ 2881.118537] BTRFS error (device vdb): bad tree block start, want 31555584 have 217020518514230019 [ 2881.126228] BTRFS error (device vdb): block=31555584 read time tree block corruption detected [ 2881.253287] BTRFS error (device vdb): bad tree block start, want 31571968 have 217020518514230019 [ 2881.259086] BTRFS error (device vdb): block=31571968 read time tree block corruption detected [ 2881.474356] BTRFS error (device vdb): bad tree block start, want 31588352 have 0 [ 2881.474364] BTRFS error (device vdb): bad tree block start, want 31604736 have 0 [ 2881.474368] BTRFS error (device vdb): bad tree block start, want 31621120 have 0 [ 2881.474371] BTRFS error (device vdb): block=31621120 read time tree block corruption detected [ 2881.474373] BTRFS error (device vdb): block=31588352 read time tree block corruption detected [ 2881.478266] BTRFS error (device vdb): block=31604736 read time tree block corruption detected [ 2881.742489] BTRFS error (device vdb): bad tree block start, want 31604736 have 0 [ 2881.746510] BTRFS error (device vdb): block=31604736 read time tree block corruption detected
[ 2882.057412] BTRFS error (device vdb): bad tree block start, want 31621120 have 0 [ 2882.073918] BTRFS error (device vdb): block=31621120 read time tree block corruption detected [ 2888.507510] BTRFS info (device vdb): found 8 extents [ 2890.081934] BTRFS info (device vdb): found 8 extents [ 2890.201963] BTRFS info (device vdb): relocating block group 30408704 flags metadata|raid1 [ 2890.348951] BTRFS info (device vdb): found 82 extents [ 2890.452777] BTRFS info (device vdb): relocating block group 22020096 flags system|raid1 [ 2890.564087] BTRFS info (device vdb): found 1 extents [ 2890.648211] BTRFS info (device vdb): balance: ended with status: 0
Quoted 6 lines · David Sterba
Quoted 3 lines · Qu Wenruo
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.I see reports from tests btrfs/124 and btrfs/125,
That two tests are RAID1 reconstruction. It's known to trigger read error on missing devices.
Especially for bytenr mismatch, as all read from missing device will result 0 filled page.
I could change the read time error message to skip bytenr/csum mismatch to avoid such report.
Quoted 8 lines
the tested branch was
for-next-20190220. Something's wrong, check the large numbers of checksums from the messages, eg.hex(506381209866536711) =
0x707070707070707
hex(217020518514230019) =0x303030303030303
hex(1085102592571150095) =0xf0f0f0f0f0f0f0f
hex(72340172838076673) =0x101010101010101
The pattern is strange. Looks like some poisoned value. Maybe we missed to clean allocated pages?
But considering the context, it's from the missing device, it shouldn't cause much problem.
Thanks,
Qu
Quoted 334 lines
there's a pattern. And the number of the 'read time' messages is quite high to be some random fluke.
[ 2847.704946] BTRFS: device fsid 8b8c51ce-59de-439f-847a-bcfce3e10a5d devid 1 transid 7 /dev/vdb [ 2847.710386] BTRFS info (device vdb): allowing degraded mounts [ 2847.713405] BTRFS info (device vdb): disk space caching is enabled [ 2847.716586] BTRFS info (device vdb): has skinny extents [ 2847.753599] BTRFS warning (device vdb): devid 2 uuid 307c73fe-349e-4b34-8233-4e35431aae31 is missing [ 2847.758479] BTRFS warning (device vdb): devid 2 uuid 307c73fe-349e-4b34-8233-4e35431aae31 is missing [ 2859.881984] BTRFS: device fsid e4579131-1621-417b-8464-cfaf6cd15204 devid 1 transid 241 /dev/vda [ 2859.896416] BTRFS info (device vdb): disk space caching is enabled [ 2859.899049] BTRFS info (device vdb): has skinny extents [ 2859.992305] BTRFS error (device vdb): bad tree block start, want 32161792 have 0 [ 2859.996611] BTRFS error (device vdb): block=32161792 read time tree block corruption detected [ 2860.001694] BTRFS info (device vdb): read error corrected: ino 0 off 32161792 (dev /dev/vdc sector 21856) [ 2860.007079] BTRFS info (device vdb): read error corrected: ino 0 off 32165888 (dev /dev/vdc sector 21864) [ 2860.012447] BTRFS info (device vdb): read error corrected: ino 0 off 32169984 (dev /dev/vdc sector 21872) [ 2860.017817] BTRFS info (device vdb): read error corrected: ino 0 off 32174080 (dev /dev/vdc sector 21880) [ 2860.023669] BTRFS error (device vdb): bad tree block start, want 32243712 have 0 [ 2860.027865] BTRFS error (device vdb): block=32243712 read time tree block corruption detected [ 2860.032961] BTRFS info (device vdb): read error corrected: ino 0 off 32243712 (dev /dev/vdc sector 22016) [ 2860.037602] BTRFS info (device vdb): read error corrected: ino 0 off 32247808 (dev /dev/vdc sector 22024) [ 2860.042185] BTRFS info (device vdb): read error corrected: ino 0 off 32251904 (dev /dev/vdc sector 22032) [ 2860.046738] BTRFS info (device vdb): read error corrected: ino 0 off 32256000 (dev /dev/vdc sector 22040) [ 2860.089197] BTRFS info (device vdb): balance: start -d -m -s [ 2860.092227] BTRFS info (device vdb): relocating block group 2479882240 flags data [ 2860.098064] BTRFS error (device vdb): bad tree block start, want 31653888 have 0 [ 2860.102289] BTRFS error (device vdb): block=31653888 read time tree block corruption detected [ 2860.107409] BTRFS info (device vdb): read error corrected: ino 0 off 31653888 (dev /dev/vdc sector 20864) [ 2860.111997] BTRFS info (device vdb): read error corrected: ino 0 off 31657984 (dev /dev/vdc sector 20872) [ 2860.166451] BTRFS error (device vdb): bad tree block start, want 31670272 have 0 [ 2860.167172] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.167375] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.167377] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.167384] BTRFS error (device vdb): bad tree block start, want 31719424 have 0 [ 2860.167386] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.167394] BTRFS error (device vdb): bad tree block start, want 31735808 have 0 [ 2860.167396] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.170120] BTRFS error (device vdb): block=31670272 read time tree block corruption detected [ 2860.174308] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.205288] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.208875] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.244462] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.248013] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.284017] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.315705] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.336001] BTRFS error (device vdb): block=31752192 read time tree block corruption detected [ 2860.336054] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.339786] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.366693] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.389495] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.418551] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.418713] BTRFS error (device vdb): block=31883264 read time tree block corruption detected [ 2860.418721] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.418736] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.418744] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.419721] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.462401] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.490821] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.506718] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.535013] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.568602] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.590434] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.590532] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.590677] BTRFS error (device vdb): block=31997952 read time tree block corruption detected [ 2860.590688] BTRFS error (device vdb): block=32014336 read time tree block corruption detected[ 2860.590697] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.590705] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.625834] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.658723] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.696644] BTRFS error (device vdb): block=32014336 read time tree block corruption detected [ 2860.735014] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.743240] BTRFS error (device vdb): block=32096256 read time tree block corruption detected [ 2860.780296] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.807073] BTRFS error (device vdb): block=32063488 read time tree block corruption detected [ 2860.807173] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2860.841582] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2862.476657] BTRFS info (device vdb): found 3 extents [ 2864.186610] BTRFS info (device vdb): found 3 extents [ 2864.190233] BTRFS error (device vdb): block=31637504 read time tree block corruption detected [ 2864.306413] BTRFS info (device vdb): relocating block group 2446327808 flags system [ 2864.418449] BTRFS info (device vdb): relocating block group 2177892352 flags metadata [ 2864.530002] BTRFS info (device vdb): relocating block group 1104150528 flags data|raid1 [ 2864.582024] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2097152 csum 0x6e65c49f expected csum 0x98f94189 mirror 1 [ 2864.582156] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2162688 csum 0x5ce6193f expected csum 0x98f94189 mirror 1 [ 2864.587210] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2101248 csum 0x6e65c49f expected csum 0x98f94189 mirror 1 [ 2864.594057] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2228224 csum 0x89780ae9 expected csum 0x98f94189 mirror 1 [ 2864.594156] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2359296 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594200] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2363392 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594228] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2367488 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594267] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2371584 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.594291] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2490368 csum 0xddddd09d expected csum 0x98f94189 mirror 1 [ 2864.594294] BTRFS warning (device vdb): csum failed root -9 ino 260 off 2375680 csum 0xfd497142 expected csum 0x98f94189 mirror 1 [ 2864.686125] BTRFS error (device vdb): parent transid verify failed on 30539776 wanted 8 found 5 [ 2864.859090] BTRFS error (device vdb): parent transid verify failed on 30556160 wanted 8 found 5 [ 2864.936897] BTRFS error (device vdb): parent transid verify failed on 30572544 wanted 8 found 6 [ 2865.003123] repair_io_failure: 5828 callbacks suppressed [ 2865.003127] BTRFS info (device vdb): read error corrected: ino 260 off 43462656 (dev /dev/vdc sector 2200472) [ 2865.003141] BTRFS info (device vdb): read error corrected: ino 260 off 43683840 (dev /dev/vdc sector 2200904) [ 2865.003210] BTRFS info (device vdb): read error corrected: ino 260 off 43659264 (dev /dev/vdc sector 2200856) [ 2865.003223] BTRFS info (device vdb): read error corrected: ino 260 off 43651072 (dev /dev/vdc sector 2200840) [ 2865.005980] BTRFS info (device vdb): read error corrected: ino 260 off 43663360 (dev /dev/vdc sector 2200864) [ 2865.006063] BTRFS info (device vdb): read error corrected: ino 260 off 43446272 (dev /dev/vdc sector 2200440) [ 2865.006098] BTRFS info (device vdb): read error corrected: ino 260 off 43638784 (dev /dev/vdc sector 2200816) [ 2865.010374] BTRFS info (device vdb): read error corrected: ino 260 off 43655168 (dev /dev/vdc sector 2200848) [ 2865.010425] BTRFS info (device vdb): read error corrected: ino 260 off 43700224 (dev /dev/vdc sector 2200936) [ 2865.010664] BTRFS info (device vdb): read error corrected: ino 260 off 43843584 (dev /dev/vdc sector 2201216) [ 2865.112654] BTRFS error (device vdb): parent transid verify failed on 30588928 wanted 8 found 6 [ 2865.267513] btree_readpage_end_io_hook: 34 callbacks suppressed [ 2865.267518] BTRFS error (device vdb): bad tree block start, want 30670848 have 0 [ 2865.267544] BTRFS error (device vdb): bad tree block start, want 30687232 have 0 [ 2865.267549] BTRFS error (device vdb): bad tree block start, want 30703616 have 0 [ 2865.267551] BTRFS error (device vdb): block=30703616 read time tree block corruption detected [ 2865.267578] BTRFS error (device vdb): bad tree block start, want 30736384 have 0 [ 2865.267580] BTRFS error (device vdb): block=30736384 read time tree block corruption detected [ 2865.267823] BTRFS error (device vdb): bad tree block start, want 30720000 have 0 [ 2865.267825] BTRFS error (device vdb): block=30720000 read time tree block corruption detected [ 2865.276321] BTRFS error (device vdb): block=30670848 read time tree block corruption detected [ 2865.380720] BTRFS error (device vdb): block=30687232 read time tree block corruption detected [ 2865.477717] BTRFS error (device vdb): bad tree block start, want 30687232 have 0 [ 2865.487564] BTRFS error (device vdb): block=30687232 read time tree block corruption detected [ 2865.628551] BTRFS error (device vdb): bad tree block start, want 30703616 have 0 [ 2865.636510] BTRFS error (device vdb): block=30703616 read time tree block corruption detected [ 2865.842136] BTRFS error (device vdb): bad tree block start, want 30720000 have 0 [ 2865.845901] BTRFS error (device vdb): block=30720000 read time tree block corruption detected [ 2866.072213] BTRFS error (device vdb): bad tree block start, want 30736384 have 0 [ 2866.081387] BTRFS error (device vdb): block=30736384 read time tree block corruption detected [ 2866.253507] BTRFS error (device vdb): bad tree block start, want 30769152 have 0 [ 2866.253527] BTRFS error (device vdb): block=30801920 read time tree block corruption detected[ 2866.253538] BTRFS error (device vdb): block=30752768 read time tree block corruption detected [ 2866.253550] BTRFS error (device vdb): block=30818304 read time tree block corruption detected [ 2866.263700] BTRFS error (device vdb): block=30769152 read time tree block corruption detected [ 2866.273628] BTRFS error (device vdb): block=30785536 read time tree block corruption detected [ 2866.474143] BTRFS error (device vdb): block=30769152 read time tree block corruption detected [ 2866.576807] BTRFS error (device vdb): block=30785536 read time tree block corruption detected [ 2866.793977] BTRFS error (device vdb): block=30801920 read time tree block corruption detected [ 2866.997914] BTRFS error (device vdb): block=30818304 read time tree block corruption detected [ 2867.157625] BTRFS error (device vdb): block=30834688 read time tree block corruption detected [ 2867.157684] BTRFS error (device vdb): block=30851072 read time tree block corruption detected [ 2867.162566] BTRFS error (device vdb): block=30867456 read time tree block corruption detected [ 2867.162581] BTRFS error (device vdb): block=30883840 read time tree block corruption detected [ 2867.162591] BTRFS error (device vdb): block=30900224 read time tree block corruption detected [ 2867.393541] BTRFS error (device vdb): block=30851072 read time tree block corruption detected [ 2867.556641] BTRFS error (device vdb): block=30867456 read time tree block corruption detected [ 2867.787270] BTRFS error (device vdb): block=30883840 read time tree block corruption detected [ 2867.997127] BTRFS error (device vdb): block=30900224 read time tree block corruption detected [ 2868.205510] BTRFS error (device vdb): block=30916608 read time tree block corruption detected [ 2868.205532] BTRFS error (device vdb): block=30949376 read time tree block corruption detected [ 2868.205536] BTRFS error (device vdb): block=30932992 read time tree block corruption detected [ 2868.205562] BTRFS error (device vdb): block=30965760 read time tree block corruption detected [ 2868.205569] BTRFS error (device vdb): block=30982144 read time tree block corruption detected [ 2868.388272] BTRFS error (device vdb): block=30932992 read time tree block corruption detected [ 2868.592079] BTRFS error (device vdb): block=30949376 read time tree block corruption detected [ 2868.794379] BTRFS error (device vdb): block=30965760 read time tree block corruption detected [ 2868.999973] BTRFS error (device vdb): block=30982144 read time tree block corruption detected [ 2869.207738] BTRFS error (device vdb): block=31014912 read time tree block corruption detected [ 2869.207772] BTRFS error (device vdb): block=30998528 read time tree block corruption detected [ 2869.207849] BTRFS error (device vdb): block=31031296 read time tree block corruption detected [ 2869.207859] BTRFS error (device vdb): block=31047680 read time tree block corruption detected [ 2869.207865] BTRFS error (device vdb): block=31064064 read time tree block corruption detected [ 2869.430131] BTRFS error (device vdb): block=31014912 read time tree block corruption detected [ 2869.583069] btrfs_print_data_csum_error: 178079 callbacks suppressed [ 2869.583073] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400269312 csum 0x5ce6193f expected csum 0x98f94189 mirror 1 [ 2869.583088] BTRFS warning (device vdb): csum failed root -9 ino 260 off 399634432 csum 0x362910d4 expected csum 0x98f94189 mirror 1 [ 2869.584130] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400273408 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.586502] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400277504 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.586609] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400130048 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.601709] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400281600 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.601732] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400134144 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.606867] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400138240 csum 0x0bd5f454 expected csum 0x98f94189 mirror 1 [ 2869.606878] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400285696 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.606912] BTRFS warning (device vdb): csum failed root -9 ino 260 off 400289792 csum 0xe3b70302 expected csum 0x98f94189 mirror 1 [ 2869.678550] BTRFS error (device vdb): block=31031296 read time tree block corruption detected [ 2869.782906] BTRFS error (device vdb): block=31047680 read time tree block corruption detected [ 2870.010786] repair_io_failure: 89687 callbacks suppressed [ 2870.010790] BTRFS info (device vdb): read error corrected: ino 260 off 430133248 (dev /dev/vdc sector 2955688) [ 2870.010830] BTRFS info (device vdb): read error corrected: ino 260 off 430178304 (dev /dev/vdc sector 2955776) [ 2870.010844] BTRFS info (device vdb): read error corrected: ino 260 off 430116864 (dev /dev/vdc sector 2955656) [ 2870.011103] BTRFS info (device vdb): read error corrected: ino 260 off 430182400 (dev /dev/vdc sector 2955784) [ 2870.011133] BTRFS info (device vdb): read error corrected: ino 260 off 430194688 (dev /dev/vdc sector 2955808) [ 2870.011168] BTRFS info (device vdb): read error corrected: ino 260 off 430112768 (dev /dev/vdc sector 2955648) [ 2870.011313] BTRFS info (device vdb): read error corrected: ino 260 off 430071808 (dev /dev/vdc sector 2955568) [ 2870.011328] BTRFS info (device vdb): read error corrected: ino 260 off 430047232 (dev /dev/vdc sector 2955520) [ 2870.011396] BTRFS info (device vdb): read error corrected: ino 260 off 430051328 (dev /dev/vdc sector 2955528) [ 2870.011456] BTRFS info (device vdb): read error corrected: ino 260 off 429916160 (dev /dev/vdc sector 2955264) [ 2870.017915] BTRFS error (device vdb): block=31064064 read time tree block corruption detected [ 2870.254803] BTRFS error (device vdb): block=31096832 read time tree block corruption detected [ 2870.254814] BTRFS error (device vdb): block=31080448 read time tree block corruption detected [ 2870.254816] BTRFS error (device vdb): block=31113216 read time tree block corruption detected [ 2870.254851] BTRFS error (device vdb): block=31129600 read time tree block corruption detected [ 2870.254868] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2870.468150] btree_readpage_end_io_hook: 40 callbacks suppressed[ 2870.468154] BTRFS error (device vdb): bad tree block start, want 31096832 have 0 [ 2870.481275] BTRFS error (device vdb): block=31096832 read time tree block corruption detected [ 2870.659389] BTRFS error (device vdb): bad tree block start, want 31113216 have 0 [ 2870.663667] BTRFS error (device vdb): block=31113216 read time tree block corruption detected [ 2870.919169] BTRFS error (device vdb): bad tree block start, want 31129600 have 0 [ 2870.927705] BTRFS error (device vdb): block=31129600 read time tree block corruption detected [ 2871.058301] BTRFS error (device vdb): bad tree block start, want 31145984 have 0 [ 2871.058336] BTRFS error (device vdb): bad tree block start, want 31162368 have 0 [ 2871.072181] BTRFS error (device vdb): bad tree block start, want 31178752 have 0 [ 2871.072185] BTRFS error (device vdb): bad tree block start, want 31211520 have 72340172838076673 [ 2871.072187] BTRFS error (device vdb): block=31211520 read time tree block corruption detected [ 2871.072197] BTRFS error (device vdb): bad tree block start, want 31227904 have 72340172838076673 [ 2871.072198] BTRFS error (device vdb): block=31227904 read time tree block corruption detected [ 2871.072264] BTRFS error (device vdb): bad tree block start, want 31195136 have 188978565376 [ 2871.072266] BTRFS error (device vdb): block=31195136 read time tree block corruption detected [ 2871.076187] BTRFS error (device vdb): block=31162368 read time tree block corruption detected [ 2871.081901] BTRFS error (device vdb): block=31178752 read time tree block corruption detected [ 2871.091522] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2871.686871] BTRFS error (device vdb): bad tree block start, want 31145984 have 0 [ 2871.701355] BTRFS error (device vdb): block=31145984 read time tree block corruption detected [ 2871.883082] BTRFS error (device vdb): block=31162368 read time tree block corruption detected [ 2872.070858] BTRFS error (device vdb): block=31227904 read time tree block corruption detected [ 2872.209604] BTRFS error (device vdb): block=31195136 read time tree block corruption detected [ 2872.524825] BTRFS error (device vdb): block=31211520 read time tree block corruption detected [ 2872.773657] BTRFS error (device vdb): block=31260672 read time tree block corruption detected [ 2872.773661] BTRFS error (device vdb): block=31277056 read time tree block corruption detected [ 2872.773668] BTRFS error (device vdb): block=31244288 read time tree block corruption detected [ 2872.773674] BTRFS error (device vdb): block=31293440 read time tree block corruption detected [ 2872.773681] BTRFS error (device vdb): block=31309824 read time tree block corruption detected [ 2872.930060] BTRFS error (device vdb): block=31260672 read time tree block corruption detected [ 2873.179606] BTRFS error (device vdb): block=31277056 read time tree block corruption detected [ 2873.346103] BTRFS error (device vdb): block=31293440 read time tree block corruption detected [ 2873.517863] BTRFS error (device vdb): block=31309824 read time tree block corruption detected [ 2873.705657] BTRFS error (device vdb): block=31326208 read time tree block corruption detected [ 2873.705665] BTRFS error (device vdb): block=31358976 read time tree block corruption detected [ 2873.705735] BTRFS error (device vdb): block=31375360 read time tree block corruption detected [ 2873.705743] BTRFS error (device vdb): block=31391744 read time tree block corruption detected [ 2873.706274] BTRFS error (device vdb): block=31342592 read time tree block corruption detected [ 2874.081625] BTRFS error (device vdb): block=31342592 read time tree block corruption detected [ 2874.408558] BTRFS error (device vdb): block=31358976 read time tree block corruption detected [ 2874.587058] btrfs_print_data_csum_error: 132544 callbacks suppressed [ 2874.587063] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688840704 csum 0x434957d2 expected csum 0x98f94189 mirror 1 [ 2874.587069] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688975872 csum 0x0107a017 expected csum 0x98f94189 mirror 1 [ 2874.587074] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688984064 csum 0x7ef574ea expected csum 0x98f94189 mirror 1 [ 2874.587090] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688988160 csum 0xf077c653 expected csum 0x98f94189 mirror 1 [ 2874.587096] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688992256 csum 0xcd9f8290 expected csum 0x98f94189 mirror 1 [ 2874.587109] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689090560 csum 0x9d2a64a2 expected csum 0x98f94189 mirror 1 [ 2874.587111] BTRFS warning (device vdb): csum failed root -9 ino 260 off 688996352 csum 0x60f06185 expected csum 0x98f94189 mirror 1 [ 2874.587124] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689094656 csum 0x6ad9235d expected csum 0x98f94189 mirror 1 [ 2874.587129] BTRFS warning (device vdb): csum failed root -9 ino 260 off 689098752 csum 0xe2cb848d expected csum 0x98f94189 mirror 1 [ 2874.597159] BTRFS error (device vdb): block=31375360 read time tree block corruption detected [ 2874.881178] BTRFS error (device vdb): block=31391744 read time tree block corruption detected [ 2874.892860] BTRFS error (device vdb): block=31408128 read time tree block corruption detected [ 2874.892885] BTRFS error (device vdb): block=31457280 read time tree block corruption detected [ 2874.892932] BTRFS error (device vdb): block=31440896 read time tree block corruption detected [ 2874.893058] BTRFS error (device vdb): block=31473664 read time tree block corruption detected [ 2874.893063] BTRFS error (device vdb): block=31490048 read time tree block corruption detected [ 2874.952101] BTRFS error (device vdb): block=31424512 read time tree block corruption detected [ 2875.011131] repair_io_failure: 65834 callbacks suppressed [ 2875.011136] BTRFS info (device vdb): read error corrected: ino 260 off 711217152 (dev /dev/vdc sector 3504680) [ 2875.011145] BTRFS info (device vdb): read error corrected: ino 260 off 710975488 (dev /dev/vdc sector 3504208) [ 2875.011167] BTRFS info (device vdb): read error corrected: ino 260 off 711225344 (dev /dev/vdc sector 3504696) [ 2875.011266] BTRFS info (device vdb): read error corrected: ino 260 off 711041024 (dev /dev/vdc sector 3504336)[ 2875.011310] BTRFS info (device vdb): read error corrected: ino 260 off 711348224 (dev /dev/vdc sector 3504936) [ 2875.014273] BTRFS info (device vdb): read error corrected: ino 260 off 710987776 (dev /dev/vdc sector 3504232) [ 2875.014288] BTRFS info (device vdb): read error corrected: ino 260 off 710979584 (dev /dev/vdc sector 3504216) [ 2875.014330] BTRFS info (device vdb): read error corrected: ino 260 off 710983680 (dev /dev/vdc sector 3504224) [ 2875.014400] BTRFS info (device vdb): read error corrected: ino 260 off 710995968 (dev /dev/vdc sector 3504248) [ 2875.014595] BTRFS info (device vdb): read error corrected: ino 260 off 711266304 (dev /dev/vdc sector 3504776) [ 2875.147908] BTRFS error (device vdb): block=31408128 read time tree block corruption detected [ 2875.380354] BTRFS error (device vdb): block=31424512 read time tree block corruption detected [ 2875.565748] btree_readpage_end_io_hook: 30 callbacks suppressed [ 2875.565753] BTRFS error (device vdb): bad tree block start, want 31457280 have 0 [ 2875.581223] BTRFS error (device vdb): block=31457280 read time tree block corruption detected [ 2875.788290] BTRFS error (device vdb): bad tree block start, want 31473664 have 1085102592571150095 [ 2875.798989] BTRFS error (device vdb): block=31473664 read time tree block corruption detected [ 2877.767277] BTRFS error (device vdb): bad tree block start, want 31490048 have 1085102592571150095 [ 2877.772654] BTRFS error (device vdb): block=31490048 read time tree block corruption detected [ 2878.524728] BTRFS error (device vdb): bad tree block start, want 31506432 have 1085102592571150095 [ 2878.524766] BTRFS error (device vdb): bad tree block start, want 31522816 have 217020518514230019 [ 2878.524798] BTRFS error (device vdb): bad tree block start, want 31539200 have 217020518514230019 [ 2878.524800] BTRFS error (device vdb): block=31539200 read time tree block corruption detected [ 2878.524835] BTRFS error (device vdb): bad tree block start, want 31571968 have 217020518514230019 [ 2878.524837] BTRFS error (device vdb): block=31571968 read time tree block corruption detected [ 2878.524848] BTRFS error (device vdb): bad tree block start, want 31555584 have 217020518514230019 [ 2878.524849] BTRFS error (device vdb): block=31555584 read time tree block corruption detected [ 2878.529852] BTRFS error (device vdb): block=31506432 read time tree block corruption detected [ 2878.534202] BTRFS error (device vdb): block=31522816 read time tree block corruption detected [ 2879.386342] BTRFS error (device vdb): bad tree block start, want 31522816 have 217020518514230019 [ 2879.400043] BTRFS error (device vdb): block=31522816 read time tree block corruption detected [ 2879.638124] btrfs_print_data_csum_error: 59720 callbacks suppressed [ 2879.638128] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817057792 csum 0x576bf9a1 expected csum 0x98f94189 mirror 1 [ 2879.638139] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816893952 csum 0x7d29d791 expected csum 0x98f94189 mirror 1 [ 2879.638760] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816898048 csum 0x4c64848f expected csum 0x98f94189 mirror 1 [ 2879.638814] BTRFS warning (device vdb): csum failed root -9 ino 260 off 816902144 csum 0x0107a017 expected csum 0x98f94189 mirror 1 [ 2879.638856] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817061888 csum 0x3dc46475 expected csum 0x98f94189 mirror 1 [ 2879.638890] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817065984 csum 0x63f98f58 expected csum 0x98f94189 mirror 1 [ 2879.638915] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817070080 csum 0x2a2250eb expected csum 0x98f94189 mirror 1 [ 2879.638940] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817074176 csum 0x90ece4b7 expected csum 0x98f94189 mirror 1 [ 2879.640899] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817000448 csum 0xe61df820 expected csum 0x98f94189 mirror 1 [ 2879.640961] BTRFS warning (device vdb): csum failed root -9 ino 260 off 817078272 csum 0x222426ee expected csum 0x98f94189 mirror 1 [ 2880.015085] repair_io_failure: 25064 callbacks suppressed [ 2880.015089] BTRFS info (device vdb): read error corrected: ino 260 off 819994624 (dev /dev/vdc sector 3717136) [ 2880.015186] BTRFS info (device vdb): read error corrected: ino 260 off 819200000 (dev /dev/vdc sector 3715584) [ 2880.017126] BTRFS info (device vdb): read error corrected: ino 260 off 819195904 (dev /dev/vdc sector 3715576) [ 2880.017197] BTRFS info (device vdb): read error corrected: ino 260 off 819204096 (dev /dev/vdc sector 3715592) [ 2880.017226] BTRFS info (device vdb): read error corrected: ino 260 off 819212288 (dev /dev/vdc sector 3715608) [ 2880.017359] BTRFS info (device vdb): read error corrected: ino 260 off 819216384 (dev /dev/vdc sector 3715616) [ 2880.017392] BTRFS info (device vdb): read error corrected: ino 260 off 819220480 (dev /dev/vdc sector 3715624) [ 2880.017430] BTRFS info (device vdb): read error corrected: ino 260 off 819224576 (dev /dev/vdc sector 3715632) [ 2880.017598] BTRFS info (device vdb): read error corrected: ino 260 off 819228672 (dev /dev/vdc sector 3715640) [ 2880.017706] BTRFS info (device vdb): read error corrected: ino 260 off 819232768 (dev /dev/vdc sector 3715648) [ 2880.948226] BTRFS error (device vdb): bad tree block start, want 31539200 have 217020518514230019 [ 2880.952594] BTRFS error (device vdb): block=31539200 read time tree block corruption detected [ 2881.118537] BTRFS error (device vdb): bad tree block start, want 31555584 have 217020518514230019 [ 2881.126228] BTRFS error (device vdb): block=31555584 read time tree block corruption detected [ 2881.253287] BTRFS error (device vdb): bad tree block start, want 31571968 have 217020518514230019 [ 2881.259086] BTRFS error (device vdb): block=31571968 read time tree block corruption detected [ 2881.474356] BTRFS error (device vdb): bad tree block start, want 31588352 have 0 [ 2881.474364] BTRFS error (device vdb): bad tree block start, want 31604736 have 0 [ 2881.474368] BTRFS error (device vdb): bad tree block start, want 31621120 have 0 [ 2881.474371] BTRFS error (device vdb): block=31621120 read time tree block corruption detected [ 2881.474373] BTRFS error (device vdb): block=31588352 read time tree block corruption detected [ 2881.478266] BTRFS error (device vdb): block=31604736 read time tree block corruption detected [ 2881.742489] BTRFS error (device vdb): bad tree block start, want 31604736 have 0 [ 2881.746510] BTRFS error (device vdb): block=31604736 read time tree block corruption detected[ 2882.057412] BTRFS error (device vdb): bad tree block start, want 31621120 have 0 [ 2882.073918] BTRFS error (device vdb): block=31621120 read time tree block corruption detected [ 2888.507510] BTRFS info (device vdb): found 8 extents [ 2890.081934] BTRFS info (device vdb): found 8 extents [ 2890.201963] BTRFS info (device vdb): relocating block group 30408704 flags metadata|raid1 [ 2890.348951] BTRFS info (device vdb): found 82 extents [ 2890.452777] BTRFS info (device vdb): relocating block group 22020096 flags system|raid1 [ 2890.564087] BTRFS info (device vdb): found 1 extents [ 2890.648211] BTRFS info (device vdb): balance: ended with status: 0
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · David Sterba
Quoted 3 lines · Qu Wenruo
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.I see reports from tests btrfs/124 and btrfs/125,
Quoted 8 lines
That two tests are RAID1 reconstruction. It's known to trigger read error on missing devices.
Especially for bytenr mismatch, as all read from missing device will result 0 filled page.
I could change the read time error message to skip bytenr/csum mismatch to avoid such report.
Yeah the output needs some tuning. See the example below.
Quoted 26 lines
Quoted 26 lines
[ 2860.001694] BTRFS info (device vdb): read error corrected: ino 0 off 32161792 (dev /dev/vdc sector 21856) [ 2860.007079] BTRFS info (device vdb): read error corrected: ino 0 off 32165888 (dev /dev/vdc sector 21864) [ 2860.012447] BTRFS info (device vdb): read error corrected: ino 0 off 32169984 (dev /dev/vdc sector 21872) [ 2860.017817] BTRFS info (device vdb): read error corrected: ino 0 off 32174080 (dev /dev/vdc sector 21880) [ 2860.023669] BTRFS error (device vdb): bad tree block start, want 32243712 have 0 [ 2860.027865] BTRFS error (device vdb): block=32243712 read time tree block corruption detected [ 2860.032961] BTRFS info (device vdb): read error corrected: ino 0 off 32243712 (dev /dev/vdc sector 22016) [ 2860.037602] BTRFS info (device vdb): read error corrected: ino 0 off 32247808 (dev /dev/vdc sector 22024) [ 2860.042185] BTRFS info (device vdb): read error corrected: ino 0 off 32251904 (dev /dev/vdc sector 22032) [ 2860.046738] BTRFS info (device vdb): read error corrected: ino 0 off 32256000 (dev /dev/vdc sector 22040) [ 2860.089197] BTRFS info (device vdb): balance: start -d -m -s [ 2860.092227] BTRFS info (device vdb): relocating block group 2479882240 flags data [ 2860.098064] BTRFS error (device vdb): bad tree block start, want 31653888 have 0 [ 2860.102289] BTRFS error (device vdb): block=31653888 read time tree block corruption detected [ 2860.107409] BTRFS info (device vdb): read error corrected: ino 0 off 31653888 (dev /dev/vdc sector 20864) [ 2860.111997] BTRFS info (device vdb): read error corrected: ino 0 off 31657984 (dev /dev/vdc sector 20872) [ 2860.166451] BTRFS error (device vdb): bad tree block start, want 31670272 have 0 [ 2860.167172] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.167375] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.167377] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.167384] BTRFS error (device vdb): bad tree block start, want 31719424 have 0 [ 2860.167386] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.167394] BTRFS error (device vdb): bad tree block start, want 31735808 have 0 [ 2860.167396] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.170120] BTRFS error (device vdb): block=31670272 read time tree block corruption detected
Quoted 8 lines
Quoted 8 lines
[ 2860.174308] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.205288] BTRFS error (device vdb): bad tree block start, want 31686656 have 0 [ 2860.208875] BTRFS error (device vdb): block=31686656 read time tree block corruption detected [ 2860.244462] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.248013] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.284017] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.315705] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.336001] BTRFS error (device vdb): block=31752192 read time tree block corruption detected
Block 31752192 does not have any previous message about a specific error (unlike the 'bad tree block' messages of other blocks). I don't think it's good to print just the 'read time' message.
Quoted 20 lines
Quoted 20 lines
[ 2860.336054] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.339786] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.366693] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.389495] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.418551] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.418713] BTRFS error (device vdb): block=31883264 read time tree block corruption detected [ 2860.418721] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.418736] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.418744] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.419721] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.462401] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.490821] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.506718] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.535013] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.568602] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.590434] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.590532] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.590677] BTRFS error (device vdb): block=31997952 read time tree block corruption detected [ 2860.590688] BTRFS error (device vdb): block=32014336 read time tree block corruption detected
Quoted 11 lines
Quoted 11 lines
[ 2860.590697] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.590705] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.625834] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.658723] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.696644] BTRFS error (device vdb): block=32014336 read time tree block corruption detected [ 2860.735014] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.743240] BTRFS error (device vdb): block=32096256 read time tree block corruption detected [ 2860.780296] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.807073] BTRFS error (device vdb): block=32063488 read time tree block corruption detected [ 2860.807173] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2860.841582] BTRFS error (device vdb): block=32079872 read time tree block corruption detected
This long sequence of messages lacks details about what happen, and there are duplicates (block 32079872). If this is caused by reads from missing device, this can potentially flood the logs.
Quoted 22 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · David Sterba
Quoted 3 lines · Qu Wenruo
v5.1:
- Add "block=%llu " output for write/read time error line.
- Also output read time error message for fsid/start/level check.I see reports from tests btrfs/124 and btrfs/125,
Quoted 8 lines
That two tests are RAID1 reconstruction. It's known to trigger read error on missing devices.
Especially for bytenr mismatch, as all read from missing device will result 0 filled page.
I could change the read time error message to skip bytenr/csum mismatch to avoid such report.
Yeah the output needs some tuning. See the example below.
Quoted 1 line
[snip]
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
[ 2860.244462] BTRFS error (device vdb): bad tree block start, want 31703040 have 0 [ 2860.248013] BTRFS error (device vdb): block=31703040 read time tree block corruption detected [ 2860.284017] BTRFS error (device vdb): block=31719424 read time tree block corruption detected [ 2860.315705] BTRFS error (device vdb): block=31735808 read time tree block corruption detected [ 2860.336001] BTRFS error (device vdb): block=31752192 read time tree block corruption detectedBlock 31752192 does not have any previous message about a specific error (unlike the 'bad tree block' messages of other blocks). I don't think it's good to print just the 'read time' message.
The first is caused by the bytenr mismatch.
But for the remaining 3, they can be caused by read error from missing device. EXTENT_BUFFER_READ_ERR would cause no detailed error report here.
Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
[ 2860.336054] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.339786] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.366693] BTRFS error (device vdb): block=31768576 read time tree block corruption detected [ 2860.389495] BTRFS error (device vdb): block=31784960 read time tree block corruption detected [ 2860.418551] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.418713] BTRFS error (device vdb): block=31883264 read time tree block corruption detected [ 2860.418721] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.418736] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.418744] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.419721] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.462401] BTRFS error (device vdb): block=31866880 read time tree block corruption detected [ 2860.490821] BTRFS error (device vdb): block=31932416 read time tree block corruption detected [ 2860.506718] BTRFS error (device vdb): block=31899648 read time tree block corruption detected [ 2860.535013] BTRFS error (device vdb): block=31916032 read time tree block corruption detected [ 2860.568602] BTRFS error (device vdb): block=31948800 read time tree block corruption detected [ 2860.590434] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.590532] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.590677] BTRFS error (device vdb): block=31997952 read time tree block corruption detected [ 2860.590688] BTRFS error (device vdb): block=32014336 read time tree block corruption detectedQuoted 11 lines
Quoted 11 lines
[ 2860.590697] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.590705] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.625834] BTRFS error (device vdb): block=31965184 read time tree block corruption detected [ 2860.658723] BTRFS error (device vdb): block=31981568 read time tree block corruption detected [ 2860.696644] BTRFS error (device vdb): block=32014336 read time tree block corruption detected [ 2860.735014] BTRFS error (device vdb): block=32030720 read time tree block corruption detected [ 2860.743240] BTRFS error (device vdb): block=32096256 read time tree block corruption detected [ 2860.780296] BTRFS error (device vdb): block=32047104 read time tree block corruption detected [ 2860.807073] BTRFS error (device vdb): block=32063488 read time tree block corruption detected [ 2860.807173] BTRFS error (device vdb): block=32079872 read time tree block corruption detected [ 2860.841582] BTRFS error (device vdb): block=32079872 read time tree block corruption detected
Quoted 3 lines
This long sequence of messages lacks details about what happen, and there are duplicates (block 32079872). If this is caused by reads from missing device, this can potentially flood the logs.
Not only from missing device, but also from stale old device.
Anyway, since I have reverted the behavior to the original one, which only output the "read time" error message for tree checker error, it should no longer flood the kernel message.
Thanks,
Qu