btrfs: raid56: fix inverted bio-list check in scrub read assembly
Subsystems (2)
PATCH 1/1 btrfs: raid56: fix inverted bio-list check in scrub read assembly
Commit 5387bd958180 ("btrfs: raid56: remove sector_ptr structure") converted the bio-list membership checks from sector pointers to physical addresses. The two conversions in rmw_assemble_write_bios() kept their polarity (skip the sector when it is NOT in the bio list, i.e. when there is nothing to write), but scrub_assemble_read_bios() has the opposite polarity -- skip the sector when it IS in the bio list, because then there is nothing to read -- and the conversion flipped it:
- sector = sector_in_rbio(rbio, stripe, sectornr, 1); - if (sector) + paddr = sector_paddr_in_rbio(rbio, stripe, sectornr, 1); + if (paddr == INVALID_PADDR) continue;
Since a parity-scrub rbio's bio list only holds the empty completion bio, the result is that scrub_assemble_read_bios() submits no reads at all. finish_parity_scrub() then compares the parity it computes from the (cached, correct) data stripes against whatever happens to be in the freshly allocated, uninitialized stripe pages:
- if the garbage differs from the computed parity, the sector is "repaired" and written back -- accidentally producing the correct on-disk result;
- if a recycled page happens to still hold the old (correct) parity content, the sector is deemed clean, dropped from dbitmap, and the actually-corrupt on-disk parity is left in place -- silently, with every scrub error counter reading zero.
The second case is intermittent because it depends on page-allocator recycling. Observed with fstests btrfs/297 (raid5, 2 devices): the corrupted P stripe intermittently stays corrupt after a scrub that reports no errors -- roughly 1/10 runs on x86-64 KVM and up to 7/8 on a UML build whose timing favors page reuse. Instrumentation of verify_one_parity_step() showed the "on disk" bytes never matching the device content (stale 0xaa / zeroed pages instead of the injected 0xff), and after this fix the injected corruption is read, detected and repaired in every run (8/8 UML, 10/10 KVM).
---
Note 1: I found, reproduced and created a fix for this problem using AI tools
Note 2: I am referencing UML (User-Mode Linux) above which is coming from the project https://github.com/mykola-lysenko/btrfs-uml-fstests/ to run xfstests in the UML. For reference only.
fs/btrfs/raid56.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/btrfs/raid56.c +4 -4
@@ -2910,11 +2910,11 @@ static int scrub_assemble_read_bios(struct btrfs_raid_bio *rbio)/* * We want to find all the sectors missing from the rbio and -* read them from the disk. If sector_paddr_in_rbio() finds a sector -* in the bio list we don't need to read it off the stripe. +* read them from the disk. If sector_paddrs_in_rbio() finds a +* sector in the bio list we don't need to read it off the +* stripe. */ - paddrs =sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ); -if (paddrs ==NULL ) +if (sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ))continue ; paddrs =rbio_stripe_paddrs (rbio, stripe, sectornr);
Signature · 1 line
2.43.0
Quoted 15 lines · Mykola Lysenko
Commit
5387bd958180("btrfs: raid56: removesector_ptrstructure") converted the bio-list membership checks from sector pointers to physical addresses. The two conversions inrmw_assemble_write_bios()kept their polarity (skip the sector when it is NOT in the bio list, i.e. when there is nothing to write), butscrub_assemble_read_bios()has the opposite polarity -- skip the sector when it IS in the bio list, because then there is nothing to read -- and the conversion flipped it:- sector = sector_in_rbio(rbio, stripe, sectornr, 1); - if (sector) + paddr = sector_paddr_in_rbio(rbio, stripe, sectornr, 1); + if (paddr == INVALID_PADDR) continue;
Quoted 3 lines
Since a parity-scrub rbio's bio list only holds the empty completion bio, the result is that
scrub_assemble_read_bios()submits no reads at all.
Then you should remove the check completely, and replace it with an ASSERT() to make sure scrub should not have any bio sectors, aka, rbio->bio_paddrs[] are all INVALID_PADDR, or all bios (should be one) in the bio_list are empty.
Otherwise the analyze looks good to me.
Quoted 11 lines
finish_parity_scrub()then compares the parity it computes from the (cached, correct) data stripes against whatever happens to be in the freshly allocated, uninitialized stripe pages:- if the garbage differs from the computed parity, the sector is "repaired" and written back -- accidentally producing the correct on-disk result;
- if a recycled page happens to still hold the old (correct) parity content, the sector is deemed clean, dropped from dbitmap, and the actually-corrupt on-disk parity is left in place -- silently, with every scrub error counter reading zero.
Unfortunately that's by design, as scrub never reports P/Q corruptions since there is no counter for them, and there is no space left for expansion either.
Quoted 11 lines
The second case is intermittent because it depends on page-allocator recycling. Observed with fstests btrfs/297 (raid5, 2 devices): the corrupted P stripe intermittently stays corrupt after a scrub that reports no errors -- roughly 1/10 runs on x86-64 KVM and up to 7/8 on a UML build whose timing favors page reuse. Instrumentation of
verify_one_parity_step()showed the "on disk" bytes never matching the device content (stale0xaa/ zeroed pages instead of the injected0xff), and after this fix the injected corruption is read, detected and repaired in every run (8/8 UML, 10/10 KVM).
Quoted 6 lines
Fixes 5387bd958180 ("btrfs: raid56: remove sector_ptr structure")CC [email protected] # 7.1+Signed-off-by Mykola Lysenko <[email protected]>---
Note 1: I found, reproduced and created a fix for this problem using AI tools
Not sure if we still require the disclosure of AI usage using Assisted-by: tag, but I'd prefer that to be extra clear.
Thanks,
Qu
Quoted 28 lines
Note 2: I am referencing UML (User-Mode Linux) above which is coming from the project https://github.com/mykola-lysenko/btrfs-uml-fstests/ to run xfstests in the UML. For reference only.
fs/btrfs/raid56.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)fs/btrfs/raid56.c +4
-4@@ -2910,11 +2910,11 @@ static int scrub_assemble_read_bios(struct btrfs_raid_bio *rbio)/* * We want to find all the sectors missing from the rbio and -* read them from the disk. If sector_paddr_in_rbio() finds a sector -* in the bio list we don't need to read it off the stripe. +* read them from the disk. If sector_paddrs_in_rbio() finds a +* sector in the bio list we don't need to read it off the +* stripe. */ - paddrs =sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ); -if (paddrs ==NULL ) +if (sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ))continue ; paddrs =rbio_stripe_paddrs (rbio, stripe, sectornr);
Hi Qu! Thanks a lot for your review!
Quoted 23 lines · Qu Wenruo
Quoted 15 lines · Mykola Lysenko
Commit
5387bd958180("btrfs: raid56: removesector_ptrstructure") converted the bio-list membership checks from sector pointers to physical addresses. The two conversions inrmw_assemble_write_bios()kept their polarity (skip the sector when it is NOT in the bio list, i.e. when there is nothing to write), butscrub_assemble_read_bios()has the opposite polarity -- skip the sector when it IS in the bio list, because then there is nothing to read -- and the conversion flipped it:- sector = sector_in_rbio(rbio, stripe, sectornr, 1); - if (sector) + paddr = sector_paddr_in_rbio(rbio, stripe, sectornr, 1); + if (paddr == INVALID_PADDR) continue;Quoted 3 lines
Since a parity-scrub rbio's bio list only holds the empty completion bio, the result is that
scrub_assemble_read_bios()submits no reads at all.
Quoted 6 lines
Then you should remove the check completely, and replace it with an
ASSERT()to make sure scrub should not have any bio sectors, aka, rbio->bio_paddrs[] are allINVALID_PADDR, or all bios (should be one) in thebio_listare empty.Otherwise the analyze looks good to me.
Sounds good, will be addressed in v2.
Quoted 13 lines
Quoted 5 lines
finish_parity_scrub()then compares the parity it computes from the (cached, correct) data stripes against whatever happens to be in the freshly allocated, uninitialized stripe pages:Quoted 7 lines
- if the garbage differs from the computed parity, the sector is "repaired" and written back -- accidentally producing the correct on-disk result;
- if a recycled page happens to still hold the old (correct) parity content, the sector is deemed clean, dropped from dbitmap, and the actually-corrupt on-disk parity is left in place -- silently, with every scrub error counter reading zero.
Quoted 3 lines
Unfortunately that's by design, as scrub never reports P/Q corruptions since there is no counter for them, and there is no space left for expansion either.
Got it, will reframe.
Quoted 19 lines
Quoted 12 lines
The second case is intermittent because it depends on page-allocator recycling. Observed with fstests btrfs/297 (raid5, 2 devices): the corrupted P stripe intermittently stays corrupt after a scrub that reports no errors -- roughly 1/10 runs on x86-64 KVM and up to 7/8 on a UML build whose timing favors page reuse. Instrumentation of
verify_one_parity_step()showed the "on disk" bytes never matching the device content (stale0xaa/ zeroed pages instead of the injected0xff), and after this fix the injected corruption is read, detected and repaired in every run (8/8 UML, 10/10 KVM).Quoted 6 lines
Fixes 5387bd958180 ("btrfs: raid56: remove sector_ptr structure")CC [email protected] # 7.1+Signed-off-by Mykola Lysenko <[email protected]>---
Note 1: I found, reproduced and created a fix for this problem using AI tools
Quoted 2 lines
Not sure if we still require the disclosure of AI usage using
Assisted-by tag, but I'd prefer that to be extra clear.
Sounds good.
Quoted 3 lines
Thanks,
Qu
Thank you!
Quoted 30 lines
Quoted 29 lines
Note 2: I am referencing UML (User-Mode Linux) above which is coming from the project https://github.com/mykola-lysenko/btrfs-uml-fstests/ to run xfstests in the UML. For reference only.
fs/btrfs/raid56.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)fs/btrfs/raid56.c +4
-4@@ -2910,11 +2910,11 @@ static int scrub_assemble_read_bios(struct btrfs_raid_bio *rbio)/* * We want to find all the sectors missing from the rbio and -* read them from the disk. If sector_paddr_in_rbio() finds a sector -* in the bio list we don't need to read it off the stripe. +* read them from the disk. If sector_paddrs_in_rbio() finds a +* sector in the bio list we don't need to read it off the +* stripe. */ - paddrs =sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ); -if (paddrs ==NULL ) +if (sector_paddrs_in_rbio (rbio, stripe, sectornr,1 ))continue ; paddrs =rbio_stripe_paddrs (rbio, stripe, sectornr);
Quoted 12 lines · Qu Wenruo
Quoted 11 lines
finish_parity_scrub()then compares the parity it computes from the (cached, correct) data stripes against whatever happens to be in the freshly allocated, uninitialized stripe pages:- if the garbage differs from the computed parity, the sector is "repaired" and written back -- accidentally producing the correct on-disk result;
- if a recycled page happens to still hold the old (correct) parity content, the sector is deemed clean, dropped from dbitmap, and the actually-corrupt on-disk parity is left in place -- silently, with every scrub error counter reading zero.
Quoted 3 lines
Unfortunately that's by design, as scrub never reports P/Q corruptions since there is no counter for them, and there is no space left for expansion either.
Right, unfortunately scrub was not designed with future extensions of the stats. We can add v2 of the scrub progress ioctl as this does the reporting part, and not the main scrub ioctl.