Uncorrectable error during multiple scrub (raid5 recovery).
Hi,
Uncorrectable error during multiple scrub (raid5 recovery).
This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.
reproducer:
mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT
for((i=1;i<=15;++i)); do
#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
sync; sleep 5; sync; sleep 5; sync; sleep 25;
# change the device to discard in every loop
j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M
btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'done
This problem will not happen if we change the test data to simpler one.
# about 220M data of '/usr/bin' to single 1G file
This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'
so it seems that some info in memory is wrong after RAID5 recovery?
[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
Best Regards
Wang Yugui ([email protected])
2022/08/14
Quoted 4 lines · Wang Yugui
Hi,
Uncorrectable error during multiple scrub (raid5 recovery).
Quoted 8 lines
This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.
reproducer:
mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
Please remove unnecessary comments if it's not explaining anything.
It just makes the test case much harder to read.
Quoted 9 lines
/bin/cp-a/usr/bin$SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNTfor((i=1;i<=15;++i)); do
#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime sync; sleep 5; sync; sleep 5; sync; sleep 25;
If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.
Quoted 16 lines
# change the device to discard in every loop j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} )) /usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2Mbtrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
done
This problem will not happen if we change the test data to simpler one.
# about 220M data of '/usr/bin' to single 1G fileThis problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'
Quoted 6 lines
so it seems that some info in memory is wrong after RAID5 recovery?
[1]
Subject: misc-next and for-next: kernel BUG atfs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
That case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/
Thanks,
Qu
Quoted 6 lines
Best Regards
Wang Yugui ([email protected])
2022/08/14
Hi,
Quoted 48 lines · Wang Yugui
Quoted 4 lines
Hi,
Uncorrectable error during multiple scrub (raid5 recovery).
Quoted 8 lines
This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.
reproducer:
mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatimePlease remove unnecessary comments if it's not explaining anything.
It just makes the test case much harder to read.
Quoted 9 lines
/bin/cp-a/usr/bin$SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNTfor((i=1;i<=15;++i)); do
#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime sync; sleep 5; sync; sleep 5; sync; sleep 25;If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.
Quoted 18 lines
# change the device to discard in every loop j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} )) /usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2Mbtrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
done
This problem will not happen if we change the test data to simpler one.
# about 220M data of '/usr/bin' to single 1G fileThis problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'so it seems that some info in memory is wrong after RAID5 recovery?
Quoted 8 lines
Quoted 4 lines
[1]
Subject: misc-next and for-next: kernel BUG atfs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#tThat case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/
This problem still happen on linux 5.20(20220812) with the flowing patches.
Subject: btrfs: scrub: properly report super block errors in system log
Subject: btrfs: scrub: try to fix super block errors (v2)Subject: btrfs: don't merge pages into bio if their page offset is not
Best Regards
Wang Yugui ([email protected])
2022/08/14
Quoted 55 lines · Wang Yugui
Hi,
Quoted 53 lines · Wang Yugui
Quoted 4 lines
Hi,
Uncorrectable error during multiple scrub (raid5 recovery).
Quoted 8 lines
This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.
reproducer:
mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatimePlease remove unnecessary comments if it's not explaining anything.
It just makes the test case much harder to read.
Quoted 9 lines
/bin/cp-a/usr/bin$SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNTfor((i=1;i<=15;++i)); do
#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime sync; sleep 5; sync; sleep 5; sync; sleep 25;If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.
Quoted 22 lines
# change the device to discard in every loop j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} )) /usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2Mbtrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
done
This problem will not happen if we change the test data to simpler one.
# about 220M data of '/usr/bin' to single 1G fileThis problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'so it seems that some info in memory is wrong after RAID5 recovery?
[1]
Subject: misc-next and for-next: kernel BUG atfs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
Quoted 5 lines
Quoted 3 lines
That case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/This problem still happen on linux 5.20(20220812) with the flowing patches.
Then please provide a human readable reproducer.
The current one is not really clear which part is commented out.
Quoted 10 lines
Subject: btrfs: scrub: properly report super block errors in system log
Subject: btrfs: scrub: try to fix super block errors (v2)Subject: btrfs: don't merge pages into bio if their page offset is notBest Regards
Wang Yugui ([email protected])
2022/08/14
Hi,
Quoted 26 lines
Uncorrectable error during multiple scrub (raid5 recovery).
This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.
reproducer:
mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
/bin/cp-a/usr/bin$SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNTfor((i=1;i<=15;++i)); do
#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime sync; sleep 5; sync; sleep 5; sync; sleep 25; # change the device to discard in every loop j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} )) /usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
Quoted 1 line
done
Add some info about the logical of this reproducer.
For btrfs RAID5, if only one disk is corrupted (blkdiscard in this reproducer) at the same time, we can recovery it by 'btrfs scrub'.
Once this recovery is finished, and then if another disk is corrupted (blkdiscard in this reproducer) too, we can recovery it by 'btrfs scrub' too.
We do multiple RAID5 recovery(btrfs scrub) with different disk, as a way to try to detect some mismatched info inside.
Best Regards
Wang Yugui ([email protected])
2022/08/14
Quoted 19 lines
This problem will not happen if we change the test data to simpler one.
# about 220M data of '/usr/bin' to single 1G fileThis problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'so it seems that some info in memory is wrong after RAID5 recovery?
[1]
Subject: misc-next and for-next: kernel BUG atfs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#tBest Regards
Wang Yugui ([email protected])
2022/08/14