Skip to contentSkip to conversation

Uncorrectable error during multiple scrub (raid5 recovery).

DISCUSSION Wang Yugui 2 participants 5 messages Aug 14, 2022 Btrfs file system

lore
Share & tools
lore

Wang Yugui

author
#0 lore raw

Hi,

Uncorrectable error during multiple scrub (raid5 recovery).

This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.

reproducer:

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime

/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT

for((i=1;i<=15;++i)); do

	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
	sync; sleep 5; sync; sleep 5; sync; sleep 25;

	# change the device to discard in every loop
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M

	btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'

done

This problem will not happen if we change the test data to simpler one.

# about 220M data of '/usr/bin' to single 1G file

This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'

so it seems that some info in memory is wrong after RAID5 recovery?

[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t

Best Regards
Wang Yugui ([email protected])
2022/08/14

#1 lore raw
Quoted 4 lines · Wang Yugui

Hi,

Uncorrectable error during multiple scrub (raid5 recovery).

Quoted 8 lines

This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.

reproducer:

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime

Please remove unnecessary comments if it's not explaining anything.

It just makes the test case much harder to read.

Quoted 9 lines

/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT

for((i=1;i<=15;++i)); do

	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
	sync; sleep 5; sync; sleep 5; sync; sleep 25;

If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.

Quoted 16 lines
	# change the device to discard in every loop
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M

btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'

done

This problem will not happen if we change the test data to simpler one.

# about 220M data of '/usr/bin' to single 1G file

This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'

Quoted 6 lines

so it seems that some info in memory is wrong after RAID5 recovery?

[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t

That case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/

Thanks,
Qu

Quoted 6 lines

Best Regards
Wang Yugui ([email protected])
2022/08/14

author
#2 lore raw

Hi,

Quoted 48 lines · Wang Yugui
Quoted 4 lines

Hi,

Uncorrectable error during multiple scrub (raid5 recovery).

Quoted 8 lines

This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.

reproducer:

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime

Please remove unnecessary comments if it's not explaining anything.

It just makes the test case much harder to read.

Quoted 9 lines

/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT

for((i=1;i<=15;++i)); do

	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
	sync; sleep 5; sync; sleep 5; sync; sleep 25;

If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.

Quoted 18 lines
	# change the device to discard in every loop
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M

btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'

done

This problem will not happen if we change the test data to simpler one.

# about 220M data of '/usr/bin' to single 1G file

This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'

so it seems that some info in memory is wrong after RAID5 recovery?

Quoted 8 lines
Quoted 4 lines

[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t

That case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/

This problem still happen on linux 5.20(20220812) with the flowing patches.

Subject: btrfs: scrub: properly report super block errors in system log
Subject: btrfs: scrub: try to fix super block errors (v2)Subject: btrfs: don't merge pages into bio if their page offset is not

Best Regards
Wang Yugui ([email protected])
2022/08/14

#3 lore raw
Quoted 55 lines · Wang Yugui

Hi,

Quoted 53 lines · Wang Yugui
Quoted 4 lines

Hi,

Uncorrectable error during multiple scrub (raid5 recovery).

Quoted 8 lines

This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.

reproducer:

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime

Please remove unnecessary comments if it's not explaining anything.

It just makes the test case much harder to read.

Quoted 9 lines

/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT

for((i=1;i<=15;++i)); do

	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
	sync; sleep 5; sync; sleep 5; sync; sleep 25;

If you really want to provide a reproducer, either explain why this is needed or it will just waste time of everyone who is trying this test.

Quoted 22 lines
	# change the device to discard in every loop
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M

btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'

done

This problem will not happen if we change the test data to simpler one.

# about 220M data of '/usr/bin' to single 1G file

This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'

so it seems that some info in memory is wrong after RAID5 recovery?

[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t

Quoted 5 lines
Quoted 3 lines

That case is in fact not related to RAID56, and we already have the fix for it:
https://lore.kernel.org/linux-btrfs/1d9b69af6ce0a79e54fbaafcc65ead8f71b54b60.1660377678.git.wqu@suse.com/

This problem still happen on linux 5.20(20220812) with the flowing patches.

Then please provide a human readable reproducer.

The current one is not really clear which part is commented out.

Quoted 10 lines

Subject: btrfs: scrub: properly report super block errors in system log
Subject: btrfs: scrub: try to fix super block errors (v2)Subject: btrfs: don't merge pages into bio if their page offset is not

Best Regards
Wang Yugui ([email protected])
2022/08/14

author
#4 lore raw
In reply to Wang Yugui

Hi,

Quoted 26 lines

Uncorrectable error during multiple scrub (raid5 recovery).

This reproducer is based on some reproducer [1], but it seems a new problem, so I open a new thread.

reproducer:

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL} SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime

/bin/cp -a /usr/bin $SCRATCH_MNT/
#(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img du -sh $SCRATCH_MNT

for((i=1;i<=15;++i)); do

	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
	sync; sleep 5; sync; sleep 5; sync; sleep 25;

	# change the device to discard in every loop
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M

	btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
Quoted 1 line

done

Add some info about the logical of this reproducer.

For btrfs RAID5, if only one disk is corrupted (blkdiscard in this reproducer) at the same time, we can recovery it by 'btrfs scrub'.

Once this recovery is finished, and then if another disk is corrupted (blkdiscard in this reproducer) too, we can recovery it by 'btrfs scrub' too.

We do multiple RAID5 recovery(btrfs scrub) with different disk, as a way to try to detect some mismatched info inside.

Best Regards
Wang Yugui ([email protected])
2022/08/14

Quoted 19 lines

This problem will not happen if we change the test data to simpler one.

# about 220M data of '/usr/bin' to single 1G file

This problem will not happen if we clear cache with 'umount; mount' between multiple loop.
# 'sync; sleep 5; ...' to 'umount; mount'

so it seems that some info in memory is wrong after RAID5 recovery?

[1]
Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t

Best Regards
Wang Yugui ([email protected])
2022/08/14