btrfs: scrub: don't count zoned repairs as corrected on a degraded mount
PATCH 1/1 btrfs: scrub: don't count zoned repairs as corrected on a degraded mount
On zoned devices scrub can't rewrite a bad copy in place. It calls btrfs_repair_one_zone() to relocate the block group instead, and counts the repaired blocks as corrected.
On a degraded mount btrfs_repair_one_zone() relocates nothing ("Do not attempt to repair in degraded state"), but scrub still counts the blocks as corrected and logs "fixed up error" for each. The bad copies stay, and the next scrub finds and "fixes" the same errors again. With zoned RAID1 mounted -o degraded with both devices present and read errors in one device's metadata zone, two scrubs in a row both report:
Error summary: read=96
Corrected: 36
Uncorrectable: 60Count such errors as uncorrectable, which the uapi header defines as "no intact copy was found or the writeback failed". Dev-replace, which writes the repaired blocks to the target device, and read-only scrub are not changed.
---
Keeping these errors as corrected and only logging that the repair was skipped would leave a false positive: the counters would still report the bad copies as fixed, and the next scrub finds them again. A new counter for errors with an intact copy that were not written back would be another option, but it changes the uapi. A read-only scrub, which also counts errors with an intact copy as corrected without writing anything, is not changed here.
Found with FuzzyRAID, a fault-injection tester for RAID that is not public at the moment. The analysis, the fix and the commit message were prepared with the help of an LLM, which also tested it. I reviewed everything.
It also applies on top of "btrfs: scrub: fix several races which can lead to UAF" (v2), with offsets.
fs/btrfs/scrub.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
fs/btrfs/scrub.c +21 -1
@@ -323,6 +323,12 @@ struct scrub_error_records {*/ unsigned long init_error_bitmap; +/* +* Set if the repaired blocks could not be written back, thus all the +* initial errors are still on disk. +*/ +bool repair_skipped; +unsigned int nr_io_errors;unsigned int nr_csum_errors;unsigned int nr_meta_errors;
@@ -992,7 +998,7 @@ static void scrub_stripe_report_errors(struct scrub_ctx *sctx,struct btrfs_fs_info *fs_info = sctx->fs_info;struct btrfs_device *dev =NULL ;const unsigned long extent_bitmap =scrub_bitmap_read_has_extent (stripe); -const unsigned long error_bitmap =scrub_bitmap_read_error (stripe); +unsigned long error_bitmap =scrub_bitmap_read_error (stripe);u64 physical =0 ;int nr_data_sectors =0 ;int nr_meta_sectors =0 ;
@@ -1003,6 +1009,11 @@ static void scrub_stripe_report_errors(struct scrub_ctx *sctx,if (test_bit (SCRUB_STRIPE_FLAG_NO_REPORT , &stripe->state))return ; +/* Not written back, the repaired blocks are still bad on disk. */ +if (errors->repair_skipped) +bitmap_or (&error_bitmap, &error_bitmap, + &errors->init_error_bitmap, stripe->nr_sectors); +/* * Init needed infos for error reporting. *
@@ -1227,6 +1238,15 @@ static void scrub_stripe_read_repair_worker(struct work_struct *work)if (!sctx->readonly && !bitmap_empty (&repaired, stripe->nr_sectors)) {if (btrfs_is_zoned (fs_info)) {btrfs_repair_one_zone (fs_info, sctx->stripes[0 ].bg->start); +/* +* On a degraded mount btrfs_repair_one_zone() does not +* relocate the block group and the bad copies stay, +* unless dev-replace writes the repaired blocks to the +* target device. +*/ +if (btrfs_test_opt (fs_info,DEGRADED ) && + !sctx->is_dev_replace) + errors.repair_skipped = true; }else {scrub_write_sectors (sctx, stripe, repaired, false);wait_scrub_stripe_io (stripe);
Signature · 1 line
2.53.0
Quoted 6 lines · Miroslav Cimerman
On zoned devices scrub can't rewrite a bad copy in place. It calls
btrfs_repair_one_zone()to relocate the block group instead, and counts the repaired blocks as corrected.On a degraded mount
btrfs_repair_one_zone()relocates nothing ("Do not attempt to repair in degraded state"),
I think this is a little weird, any special reason for this in the first place?
Quoted 10 lines
but scrub still counts the
blocks as corrected and logs "fixed up error" for each. The bad copies stay, and the next scrub finds and "fixes" the same errors again. With zoned RAID1 mounted -o degraded with both devices present and read errors in one device's metadata zone, two scrubs in a row both report:Error summary: read=96 Corrected: 36 Uncorrectable: 60
Quoted 6 lines
Count such errors as uncorrectable, which the uapi header defines as "no intact copy was found or the writeback failed". Dev-replace, which writes the repaired blocks to the target device, and read-only scrub are not changed.
Fixes 54765392a1b9 ("btrfs: scrub: introduce helper to queue a stripe for scrub")
I do not think this is the correct fixes tag, that commit is just calling btrfs_repair_one_zone(), and btrfs is already calling that before.
Quoted 75 lines
Signed-off-by Miroslav Cimerman <[email protected]>---
Keeping these errors as corrected and only logging that the repair was skipped would leave a false positive: the counters would still report the bad copies as fixed, and the next scrub finds them again. A new counter for errors with an intact copy that were not written back would be another option, but it changes the uapi. A read-only scrub, which also counts errors with an intact copy as corrected without writing anything, is not changed here.Found with FuzzyRAID, a fault-injection tester for RAID that is not public at the moment. The analysis, the fix and the commit message were prepared with the help of an LLM, which also tested it. I reviewed everything.
It also applies on top of "btrfs: scrub: fix several races which can lead to UAF" (v2), with offsets.
fs/btrfs/scrub.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)fs/btrfs/scrub.c +21
-1@@ -323,6 +323,12 @@ struct scrub_error_records {*/ unsigned long init_error_bitmap; +/* +* Set if the repaired blocks could not be written back, thus all the +* initial errors are still on disk. +*/ +bool repair_skipped; +unsigned int nr_io_errors;unsigned int nr_csum_errors;unsigned int nr_meta_errors;@@ -992,7 +998,7 @@ static void scrub_stripe_report_errors(struct scrub_ctx *sctx,struct btrfs_fs_info *fs_info = sctx->fs_info;struct btrfs_device *dev =NULL ;const unsigned long extent_bitmap =scrub_bitmap_read_has_extent (stripe); -const unsigned long error_bitmap =scrub_bitmap_read_error (stripe); +unsigned long error_bitmap =scrub_bitmap_read_error (stripe);u64 physical =0 ;int nr_data_sectors =0 ;int nr_meta_sectors =0 ;@@ -1003,6 +1009,11 @@ static void scrub_stripe_report_errors(struct scrub_ctx *sctx,if (test_bit (SCRUB_STRIPE_FLAG_NO_REPORT , &stripe->state))return ; +/* Not written back, the repaired blocks are still bad on disk. */ +if (errors->repair_skipped) +bitmap_or (&error_bitmap, &error_bitmap, + &errors->init_error_bitmap, stripe->nr_sectors); +/* * Init needed infos for error reporting. * @@ -1227,6 +1238,15 @@ static void scrub_stripe_read_repair_worker(struct work_struct *work)if (!sctx->readonly && !bitmap_empty (&repaired, stripe->nr_sectors)) {if (btrfs_is_zoned (fs_info)) {btrfs_repair_one_zone (fs_info, sctx->stripes[0 ].bg->start); +/* +* On a degraded mount btrfs_repair_one_zone() does not +* relocate the block group and the bad copies stay, +* unless dev-replace writes the repaired blocks to the +* target device. +*/ +if (btrfs_test_opt (fs_info,DEGRADED ) && + !sctx->is_dev_replace) + errors.repair_skipped = true; }else {scrub_write_sectors (sctx, stripe, repaired, false);wait_scrub_stripe_io (stripe);
Quoted 7 lines · Qu Wenruo
Quoted 2 lines · Miroslav Cimerman
On a degraded mount
btrfs_repair_one_zone()relocates nothing ("Do not attempt to repair in degraded state"),I think this is a little weird, any special reason for this in the first place?
It was probably meant to check for a missing device (so that a mirror's redundancy won't be downgraded by the relocation). But because it checks with the mount flag, a -o degraded mount with no missing devices does not get bad copies repaired by scrub.
I propose a v2 series, where 1/2 replaces the current mount flag check with a missing devices check, and 2/2 is this patch modified to use the same missing devices check as 1/2, so they are in sync. 2/2 is still needed because for example RAID1C3 can have all: a good copy, bad copy, and a missing device.
Are you okay with the proposal, Qu?
Quoted 4 lines
Quoted 1 line
Fixes 54765392a1b9 ("btrfs: scrub: introduce helper to queue a stripe for scrub")I do not think this is the correct fixes tag, that commit is just calling
btrfs_repair_one_zone(), and btrfs is already calling that before.
You are right, the Fixes tag should be:
Thanks,
Miroslav
Quoted 4 lines · Miroslav Cimerman
Quoted 1 line · Qu Wenruo
在 2026/11/19 04:26, Miroslav Cimerman 写道:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
On a degraded mount
btrfs_repair_one_zone()relocates nothing ("Do not attempt to repair in degraded state"),I think this is a little weird, any special reason for this in the first place?
It was probably meant to check for a missing device (so that a mirror's redundancy won't be downgraded by the relocation).
If it's a workaround for the well known new-SINGLE chunks during degraded writes, it may be a little over-killed.
A lot of btrfs-progs commands will do give extra warning when such unexpected profiles are detected, so end users can later balance them out after recovering. (It's not the best solution, but it is not zoned specific either)
I'm more concerned if it's some extra requirement for zoned RAID profiles, e.g. to keep the writer pointers in sync etc.
Quoted 9 lines
But because it checks
with the mount flag, a -o degraded mount with no missing devices does not get bad copies repaired by scrub.I propose a v2 series, where 1/2 replaces the current mount flag check with a missing devices check, and 2/2 is this patch modified to use the same missing devices check as 1/2, so they are in sync. 2/2 is still needed because for example RAID1C3 can have all: a good copy, bad copy, and a missing device.
And I'd prefer if you can get some helper to know if this block group can be repaired, so that no duplicated checks between repair and scrub.
Thanks,
Qu
Quoted 14 lines
Are you okay with the proposal, Qu?
Quoted 4 lines
Quoted 1 line
Fixes 54765392a1b9 ("btrfs: scrub: introduce helper to queue a stripe for scrub")I do not think this is the correct fixes tag, that commit is just calling
btrfs_repair_one_zone(), and btrfs is already calling that before.You are right, the Fixes tag should be:
Fixes f7ef5287a63d ("btrfs: zoned: relocate block group to repair IO failure in zoned filesystems")Thanks,
Miroslav
Quoted 2 lines · Qu Wenruo
If it's a workaround for the well known new-SINGLE chunks during degraded writes, it may be a little over-killed.
True, balance and block group reclaim already relocate block groups on degraded mounts with no such check.
Quoted 2 lines
I'm more concerned if it's some extra requirement for zoned RAID profiles, e.g. to keep the writer pointers in sync etc.
None that the repair would break: a regular mount needs equal write pointers, but a missing device falls behind on every degraded write anyway, not only on a repair. The problem is when that device comes back stale: two days ago my tester found that a degraded mount could then allocate over committed extents, which aborted the transaction or crashed. The series "btrfs: zoned: RAID1 block groups with missing or stale stripes" fixes that:
https://lore.kernel.org/linux-btrfs/[email protected]/
In testing with the skip removed, the relocation with a device really missing works: RAID1 on three devices stays RAID1, RAID1C3 on three devices gets single block groups, a second scrub is clean, and the missing device mounts normally when it comes back.
One thing it needs: without the skip, a degraded mount that reads a bad copy starts the relocation during mount. In testing, it raced with the mount, which failed and hit the uncommitted transaction ASSERT on its error path. "btrfs: zoned: wait for the relocating repair kthread during unmount" prevents that: with it, no repair starts before BTRFS_FS_OPEN is set:
https://lore.kernel.org/linux-btrfs/[email protected]/
Quoted 2 lines
And I'd prefer if you can get some helper to know if this block group can be repaired, so that no duplicated checks between repair and scrub.
With that repair kthread patch, the skip can be dropped completely, so no check is left to share, and scrub needs no change for degraded mounts. It still counts errors as corrected when a repair is not started, e.g. while another one runs, just as on a regular mount.
In testing, with the series and the repair kthread patch applied and the skip removed, a stale device came back. A few scrubs on the degraded mount relocated the block groups it was behind on, and then a regular mount worked again, without a balance. Only a block group where scrub finds no error still needs a balance, e.g. when the missed range holds only nodatasum data.
One downside: when fewer devices are left than the profile needs, the repair relocates the block group as single, without the missing device's copy. A RAID1C3 or RAID1C4 block group then keeps one copy until a balance. Without the repair, it would get its copies back when the device returns.
Are you OK with a v2 that drops the skip completely, on top of that repair kthread patch?
Thanks,
Miroslav
Quoted 47 lines · Miroslav Cimerman
Quoted 2 lines · Qu Wenruo
If it's a workaround for the well known new-SINGLE chunks during degraded writes, it may be a little over-killed.
True, balance and block group reclaim already relocate block groups on degraded mounts with no such check.
Quoted 2 lines
I'm more concerned if it's some extra requirement for zoned RAID profiles, e.g. to keep the writer pointers in sync etc.
None that the repair would break: a regular mount needs equal write pointers, but a missing device falls behind on every degraded write anyway, not only on a repair. The problem is when that device comes back stale: two days ago my tester found that a degraded mount could then allocate over committed extents, which aborted the transaction or crashed. The series "btrfs: zoned: RAID1 block groups with missing or stale stripes" fixes that:
https://lore.kernel.org/linux-btrfs/[email protected]/In testing with the skip removed, the relocation with a device really missing works: RAID1 on three devices stays RAID1, RAID1C3 on three devices gets single block groups, a second scrub is clean, and the missing device mounts normally when it comes back.
One thing it needs: without the skip, a degraded mount that reads a bad copy starts the relocation during mount. In testing, it raced with the mount, which failed and hit the uncommitted transaction ASSERT on its error path. "btrfs: zoned: wait for the relocating repair kthread during unmount" prevents that: with it, no repair starts before
BTRFS_FS_OPENis set:
https://lore.kernel.org/linux-btrfs/[email protected]/Quoted 2 lines
And I'd prefer if you can get some helper to know if this block group can be repaired, so that no duplicated checks between repair and scrub.
With that repair kthread patch, the skip can be dropped completely, so no check is left to share, and scrub needs no change for degraded mounts. It still counts errors as corrected when a repair is not started, e.g. while another one runs, just as on a regular mount.
In testing, with the series and the repair kthread patch applied and the skip removed, a stale device came back. A few scrubs on the degraded mount relocated the block groups it was behind on, and then a regular mount worked again, without a balance. Only a block group where scrub finds no error still needs a balance, e.g. when the missed range holds only nodatasum data.
Quoted 8 lines
One downside: when fewer devices are left than the profile needs, the repair relocates the block group as single, without the missing device's copy. A RAID1C3 or RAID1C4 block group then keeps one copy until a balance. Without the repair, it would get its copies back when the device returns.
Are you OK with a v2 that drops the skip completely, on top of that repair kthread patch?
I'm fine dropping that check, but it would be better to be a dedicated patch for extra review from zoned experts.
Thanks,
Qu
Quoted 3 lines
Thanks,
Miroslav