generic: check if one fs can detect damage at/after fs thaw
PATCH 1/1 generic: check if one fs can detect damage at/after fs thaw
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)
Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.
Then corrupt the whole fs by overwriting the block device with 0xcd (default seed from xfs_io pwrite command).
Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing -EUCLEAN.
For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return -EROFS.
For XFS, it will detect the corruption at touch time, return -EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)
--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+) create mode 100755 tests/generic/702 create mode 100644 tests/generic/702.out
tests/generic/702 new file +61 -0
@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time. +# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time" + +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journal +sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.full + +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNT + +# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 use +# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fi + +# success, all done +status=0 +exit
tests/generic/702.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected)
Signature · 1 line
2.38.0
Quoted 2 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow
"hibernation".
Quoted 10 lines
one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.
Quoted 6 lines
Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing
-EUCLEAN.
Heh, yikes. That's pretty scary for ext4 since it still uses buffer heads from the block device to read/store metadata and older kernels are known to have crashing problems if (say) the feature bits in the primary superblock get changed.
I wonder if this should force errors=remount-ro for ext4 since errors=continue is dangerous and erorrs=panic will crash the test machine.
Quoted 2 lines
For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return
-EROFS.
What /does/ btrfs check, specifically? Reading this makes me wonder if xfs shouldn't re-read its primary super on thaw to check that nobody ran us over with a backhoe, though that wouldn't help us in the hibernation case. (Or does it? Is userspace/systemd finally smart enough to freeze filesystems?)
Quoted 3 lines
For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)
Yep.
Quoted 22 lines
Signed-off-by Qu Wenruo <[email protected]>--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+) create mode 100755 tests/generic/702 create mode 100644 tests/generic/702.outtests/generic/702 new file +8
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time.
tests/generic/702 line 18
Quoted 10 lines
tests/generic/702 new file +10
-0+# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time"
Hmmm, it's not very useful for a test failure on (say) xfs spitting out a message about how this "may" get fixed with a btrfs patch. How about:
$FSTYP = btrfs && _fixed_by_kernel_commit a05d3c915314 \
"btrfs: check superbloc..."
Quoted 13 lines
tests/generic/702 new file +13
-0+ +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journal
tests/generic/702 line 41
Quoted 10 lines
tests/generic/702 new file +10
-0+sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.full
directio and a larger buffer size to speed this up? e.g.
$XFS_IO_PROG -d -c 'pwrite -b 1m 0 512M' -c sync $SCRATCH_DEV
tests/generic/702 line 44
Quoted 3 lines
tests/generic/702 new file +3
-0+ +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNT
I'm a little surprised you don't check for btrfs returning an error here...?
tests/generic/702 line 57
Quoted 12 lines
tests/generic/702 new file +12
-0+# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 use +# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fi
But otherwise this test looks reasonable so far.
--D
Quoted 15 lines
tests/generic/702 new file +4
-0+ +# success, all done +status=0 +exittests/generic/702.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected)Signature · 1 line
2.38.0
Quoted 28 lines · Darrick J. Wong
Quoted 2 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow"hibernation".
Quoted 10 lines
one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Quoted 6 lines
Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing
-EUCLEAN.Heh, yikes. That's pretty scary for ext4 since it still uses buffer heads from the block device to read/store metadata and older kernels are known to have crashing problems if (say) the feature bits in the primary superblock get changed.
Quoted 8 lines
I wonder if this should force errors=remount-ro for ext4 since errors=continue is dangerous and erorrs=panic will crash the test machine.
Quoted 2 lines
For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return
-EROFS.What /does/ btrfs check, specifically?
- Read sb without using cache
- The same mount time sanity checks on the superblock Which already implies an fsid check.
- Extra generation check
To make sure no one has touched out cake.
Quoted 5 lines
Reading this makes me wonder if
xfs shouldn't re-read its primary super on thaw to check that nobody ran us over with a backhoe, though that wouldn't help us in the hibernation case. (Or does it? Is userspace/systemd finally smart enough to freeze filesystems?)
I doubt if userspace/systemd is that smart, because the error report is running not-that-old distro.
Especially for hibernation there is really no way for anyone to know if our cakes are touched.
Quoted 44 lines
Quoted 4 lines
For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Yep.
Quoted 22 lines
Signed-off-by Qu Wenruo <[email protected]>--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+)create mode 100755 tests/generic/702
create mode 100644 tests/generic/702.outtests/generic/702 new file +8
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time.tests/generic/702 line 18
Quoted 10 lines
tests/generic/702 new file +10
-0+# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time"Hmmm, it's not very useful for a test failure on (say) xfs spitting out a message about how this "may" get fixed with a btrfs patch. How about:
Quoted 2 lines
$FSTYP = btrfs &&
_fixed_by_kernel_commita05d3c915314\
"btrfs: check superbloc..."
That sounds pretty good.
Quoted 27 lines
Quoted 14 lines
tests/generic/702 new file +13
-0+ +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journaltests/generic/702 line 41
Quoted 10 lines
tests/generic/702 new file +10
-0+sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.fulldirectio and a larger buffer size to speed this up? e.g.
Quoted 1 line
$XFS_IO_PROG -d -c 'pwrite -b 1m 0 512M' -c sync $SCRATCH_DEV
I guess no need for directio especially we're doing a sync after the write. Although larger blocksize may only help a little considering by default it's already buffered write.
Quoted 7 lines
tests/generic/702 line 44
Quoted 4 lines
tests/generic/702 new file +3
-0+ +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNTI'm a little surprised you don't check for btrfs returning an error here...?
Great you have asked!
This is the special pitfall related to thaw error handling.
If we return an error for .unfreeze_fs hook, the VFS treats it as we failed to thaw the fs, and will still consider the fs frozen.
Thus for now, btrfs only output error message into dmesg during thaw, but always return 0 to workaround it.
We may want a better way for .unfreeze_fs hook to distinguish between "something really went wrong, but please consider it unfreezed" and "nope, please keep it frozen".
Thanks,
Qu
Quoted 33 lines
tests/generic/702 line 57
Quoted 13 lines
tests/generic/702 new file +12
-0+# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 use +# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fiBut otherwise this test looks reasonable so far.
--D
Quoted 15 lines
tests/generic/702 new file +4
-0+ +# success, all done +status=0 +exittests/generic/702.out new file +2
-1@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected) --2.38.0
Quoted 40 lines · Qu Wenruo
Quoted 28 lines · Darrick J. Wong
Quoted 2 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow"hibernation".
Quoted 10 lines
one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Quoted 6 lines
Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing
-EUCLEAN.Heh, yikes. That's pretty scary for ext4 since it still uses buffer heads from the block device to read/store metadata and older kernels are known to have crashing problems if (say) the feature bits in the primary superblock get changed.
Quoted 8 lines
I wonder if this should force errors=remount-ro for ext4 since errors=continue is dangerous and erorrs=panic will crash the test machine.
Quoted 2 lines
For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return
-EROFS.What /does/ btrfs check, specifically?
Quoted 7 lines
- Read sb without using cache
- The same mount time sanity checks on the superblock Which already implies an fsid check.
- Extra generation check
To make sure no one has touched out cake.
Ah, ok, so you compare the ondisk super with the incore version and complain if they don't match. Makes sense.
Quoted 11 lines
Quoted 5 lines
Reading this makes me wonder if
xfs shouldn't re-read its primary super on thaw to check that nobody ran us over with a backhoe, though that wouldn't help us in the hibernation case. (Or does it? Is userspace/systemd finally smart enough to freeze filesystems?)I doubt if userspace/systemd is that smart, because the error report is running not-that-old distro.
Especially for hibernation there is really no way for anyone to know if our cakes are touched.
Yeah, short of encrypting the primary super. :)
Quoted 76 lines
Quoted 44 lines
Quoted 4 lines
For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Yep.
Quoted 22 lines
Signed-off-by Qu Wenruo <[email protected]>--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+)create mode 100755 tests/generic/702
create mode 100644 tests/generic/702.outtests/generic/702 new file +8
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time.tests/generic/702 line 18
Quoted 10 lines
tests/generic/702 new file +10
-0+# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time"Hmmm, it's not very useful for a test failure on (say) xfs spitting out a message about how this "may" get fixed with a btrfs patch. How about:
Quoted 2 lines
$FSTYP = btrfs &&
_fixed_by_kernel_commita05d3c915314\
"btrfs: check superbloc..."That sounds pretty good.
Quoted 27 lines
Quoted 14 lines
tests/generic/702 new file +13
-0+ +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journaltests/generic/702 line 41
Quoted 10 lines
tests/generic/702 new file +10
-0+sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.fulldirectio and a larger buffer size to speed this up? e.g.
Quoted 5 lines
Quoted 1 line
$XFS_IO_PROG -d -c 'pwrite -b 1m 0 512M' -c sync $SCRATCH_DEV
I guess no need for directio especially we're doing a sync after the write. Although larger blocksize may only help a little considering by default it's already buffered write.
<nod>
Quoted 15 lines
Quoted 7 lines
tests/generic/702 line 44
Quoted 4 lines
tests/generic/702 new file +3
-0+ +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNTI'm a little surprised you don't check for btrfs returning an error here...?
Great you have asked!
This is the special pitfall related to thaw error handling.
If we return an error for .unfreeze_fs hook, the VFS treats it as we failed to thaw the fs, and will still consider the fs frozen.
Quoted 6 lines
Thus for now, btrfs only output error message into dmesg during thaw, but always return 0 to workaround it.
We may want a better way for .unfreeze_fs hook to distinguish between "something really went wrong, but please consider it unfreezed" and "nope, please keep it frozen".
Ah, I guess it makes sense that you have to access the fs post-thaw to find out if it's still alive.
--D
Quoted 36 lines
Thanks,
QuQuoted 33 lines
tests/generic/702 line 57
Quoted 13 lines
tests/generic/702 new file +12
-0+# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 use +# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fiBut otherwise this test looks reasonable so far.
--D
Quoted 15 lines
tests/generic/702 new file +4
-0+ +# success, all done +status=0 +exittests/generic/702.out new file +2
-1@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected) --2.38.0
Quoted 9 lines · Darrick J. Wong
Quoted 8 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
Quoted 6 lines
Quoted 6 lines
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).
It seems to me that the test case is testing something very different from the originally stated concern, and what btrfs is testing.
The original concern is "something else modified the file system", which btrfs is testing by checking whether the file system generation number is different from the last recorded transaction id.
The test is "something has trashed the file system by filling the block device by 0xcd; let's see how quickly the file system notices" which is quite different from the scenario described in the link and the commit description a05d3c915314 ("btrfs: check superblock to ensure the fs was not modified at thaw time").
Quoted 5 lines
What /does/ btrfs check, specifically? Reading this makes me wonder if xfs shouldn't re-read its primary super on thaw to check that nobody ran us over with a backhoe, though that wouldn't help us in the hibernation case. (Or does it? Is userspace/systemd finally smart enough to freeze filesystems?)
From looking at the commit described below, it appears to do some basic superblock sanity checks, and then it checks to see if the last commited transaction has changed from what has been recorded in the superblock.
The simple stupid thing I could add in ext4 is to simply make a full copy of the ext4 superblock, and if *anything* in that 1k set of bytes has changed between the freeze and the thaw, call ext4_error(), and mark the file system corrupted.
We've been talking about changing the default for ext4 to remount the file system read-only, and if we did this then the behavior would be the same as btrfs. Or maybe in the specific case of the superblock has changed between freeze and thaw, we will always remount the file system read-only.
Quoted 5 lines
Quoted 3 lines
For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Yep.
I would suggest not putting this test in generic/NNN, but put it in shared, and to let each file system opt-in to this test. There are a whole bunch of file systems such such as jfs, reiserfs, vfat, exfat, etc., which could run this test, and depending on the specifics of how a file system might behave to determine whether the test "passes" or "fails" seems wrong.
After all, what you're really doing is protecting against a specific form of "stupid user trick", and other Linux file systems happen to do something different when you completely trash the file system by overwriting the block device with 0xcd, callign what some other file system, whether it be f2fs, exfat, overlayfs, nfs, as a "failure" doesn't seem right.
Moving it into shared also means you don't have to add extra checks to make sure the test gets skipped if there is no block device to trash (for example, if you are testing overlayfs, nfs, tmpfs, etc.)
Cheers,
- Ted
Quoted 24 lines · Theodore Ts'o
Quoted 9 lines · Darrick J. Wong
Quoted 8 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
Quoted 6 lines
Quoted 6 lines
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).It seems to me that the test case is testing something very different from the originally stated concern, and what btrfs is testing.
The original concern is "something else modified the file system", which btrfs is testing by checking whether the file system generation number is different from the last recorded transaction id.
Quoted 5 lines
The test is "something has trashed the file system by filling the block device by
0xcd; let's see how quickly the file system notices" which is quite different from the scenario described in the link and the commit descriptiona05d3c915314("btrfs: check superblock to ensure the fs was not modified at thaw time").
Yes, you're right.
The problem here is I have no good way to just mount the frozen fs and update it.
One of the btrfs specific problem is, btrfs will reject fs with the same fsid.
Thus even if we do a binary copy of the original fs, we can not mount it to modify, at least not for btrfs.
Any good ideas for this? Maybe relying on btrfs-progs to do the write?
Quoted 12 lines
Quoted 6 lines
What /does/ btrfs check, specifically? Reading this makes me wonder if xfs shouldn't re-read its primary super on thaw to check that nobody ran us over with a backhoe, though that wouldn't help us in the hibernation case. (Or does it? Is userspace/systemd finally smart enough to freeze filesystems?)
From looking at the commit described below, it appears to do some
basic superblock sanity checks, and then it checks to see if the last commited transaction has changed from what has been recorded in the superblock.
Quoted 4 lines
The simple stupid thing I could add in ext4 is to simply make a full copy of the ext4 superblock, and if *anything* in that 1k set of bytes has changed between the freeze and the thaw, call
ext4_error(), and mark the file system corrupted.
Yes, for EXT4/XFS it can be a simple memcmp(), but for btrfs we have multiple devices, and super block for each device has something different (related to that device).
Thus at least for btrfs, we can not do a full memcmp() level check, but you get the idea correctly.
Quoted 27 lines
We've been talking about changing the default for ext4 to remount the file system read-only, and if we did this then the behavior would be the same as btrfs. Or maybe in the specific case of the superblock has changed between freeze and thaw, we will always remount the file system read-only.
Quoted 5 lines
Quoted 3 lines
For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Yep.
I would suggest not putting this test in generic/NNN, but put it in shared, and to let each file system opt-in to this test. There are a whole bunch of file systems such such as jfs, reiserfs, vfat, exfat, etc., which could run this test, and depending on the specifics of how a file system might behave to determine whether the test "passes" or "fails" seems wrong.
After all, what you're really doing is protecting against a specific form of "stupid user trick", and other Linux file systems happen to do something different when you completely trash the file system by overwriting the block device with
0xcd, callign what some other file system, whether it be f2fs, exfat, overlayfs, nfs, as a "failure" doesn't seem right.
Quoted 3 lines
Moving it into shared also means you don't have to add extra checks to make sure the test gets skipped if there is no block device to trash (for example, if you are testing overlayfs, nfs, tmpfs, etc.)
Thanks for the advice, would move it to shared in next update.
Thanks,
Qu
Quoted 4 lines
Cheers,
- Ted
Quoted 87 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing
-EUCLEAN.For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return
-EROFS.For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Signed-off-by Qu Wenruo <[email protected]>--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+) create mode 100755 tests/generic/702 create mode 100644 tests/generic/702.outtests/generic/702 new file +47
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time. +# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time" + +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journal +sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.full + +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNT + +# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 use
tests/generic/702 line 57
Quoted 10 lines
tests/generic/702 new file +10
-0+# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fi
Thanks for all help to review!
That _require_freeze will skip exfat and others which not support freeze.
And _require_block_device $SCRATCH_DEV helps you to avoid this test run/fail on overlayfs, nfs, tmpfs, etc. Due to you try to write the $SCRATCH_DEV directly.
And you can use _supported_fs ^f2fs to skip this test from some specified fs if they're not suit for this test.
But I'm wondering if the last test step will fail on every fs soon? Except you're trying to test how fast a fs can find itself is corrupted. Or how about give some fs more chance/time to detect errors? Likes do more operations which enough to trigger errors on most fs?
Thanks,
Zorro
Quoted 15 lines
tests/generic/702 new file +4
-0+ +# success, all done +status=0 +exittests/generic/702.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected)Signature · 1 line
2.38.0
Quoted 106 lines · Zorro Lang
Quoted 87 lines · Qu Wenruo
[BACKGROUND]
There is bug report from btrfs mailing list that, hiberation can allow one to modify the frozen filesystem unexpectedly (using another OS).
(https://lore.kernel.org/linux-btrfs/[email protected]/)Later btrfs adds the check to make sure the fs is not changed unexpectedly, to prevent corruption from happening.
[TESTCASE]
Here the new test case will create a basic filesystem, fill it with something by using fsstress, then sync the fs, and finally freeze the fs.Then corrupt the whole fs by overwriting the block device with
0xcd(default seed fromxfs_iopwrite command).Finally we thaw the fs, and try if we can create a new file.
for EXT4, it will detect the corruption at touch time, causing
-EUCLEAN.For Btrfs, it will detect the corruption at thaw time, marking the fs RO immediately, and later touch will return
-EROFS.For XFS, it will detect the corruption at touch time, return
-EUCLEAN. (Without the cache drop, XFS seems to be very happy using the cache info to do the work without any error though.)Signed-off-by Qu Wenruo <[email protected]>--- tests/generic/702 | 61 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/702.out | 2 ++ 2 files changed, 63 insertions(+)create mode 100755 tests/generic/702
create mode 100644 tests/generic/702.outtests/generic/702 new file +47
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 702 +# +# Test if the filesystem can detect the underlying disk has changed at +# thaw time. +# +. ./common/preamble +. ./common/filter +_begin_fstest freeze quick + +# real QA test starts here + +_supported_fs generic +_fixed_by_kernel_commit a05d3c915314 \ + "btrfs: check superblock to ensure the fs was not modified at thaw time" + +# We will corrupt the device completely, thus should not check it after the test. +_require_scratch_nocheck +_require_freeze + +# Limit the fs to 512M so we won't waste too much time screwing it up later. +_scratch_mkfs_sized $((512 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount + +# Populate the fs with something. +$FSSTRESS_PROG -n 500 -d $SCRATCH_MNT >> $seqres.full + +# Sync to make sure no dirty journal +sync + +# Drop all cache, so later write will need to read from disk, increasing +# the chance of detecting the corruption. +echo 3 > /proc/sys/vm/drop_caches + +$XFS_IO_PROG -x -c "freeze" $SCRATCH_MNT + +# Now screw up the block device +$XFS_IO_PROG -f -c "pwrite 0 512M" -c sync $SCRATCH_DEV >> $seqres.full + +# Thaw the fs, it may or may not report error, we will check it manually later. +$XFS_IO_PROG -x -c "thaw" $SCRATCH_MNT + +# If the fs detects something wrong, it should trigger error now. +# We don't use the error message as golden output, as btrfs and ext4 usetests/generic/702 line 57
Quoted 10 lines
tests/generic/702 new file +10
-0+# different error number for different reasons. +# (btrfs detects the change immediately at thaw time and mark the fs RO, thus +# touch returns -EROFS, while ext4 detects the change at journal write time, +# returning -EUCLEAN). +touch $SCRATCH_MNT/foobar >>$seqres.full 2>&1 +if [ $? -eq 0 ]; then + echo "Failed to detect corrupted fs" +else + echo "Detected corrupted fs (expected)" +fiThanks for all help to review!
That
_require_freezewill skip exfat and others which not support freeze.And
_require_block_device $SCRATCH_DEVhelps you to avoid this test run/fail on overlayfs, nfs, tmpfs, etc. Due to you try to write the $SCRATCH_DEV directly.
Quoted 7 lines
And you can use
_supported_fs ^f2fsto skip this test from some specified fs if they're not suit for this test.But I'm wondering if the last test step will fail on every fs soon? Except you're trying to test how fast a fs can find itself is corrupted. Or how about give some fs more chance/time to detect errors? Likes do more operations which enough to trigger errors on most fs?
I don't think any timeout/extra work would help, there is already a drop cache to make sure the fs won't have any cached info.
Or fs like old btrfs or XFS can easily continue without any problem by using all the cached info.
And without cached info, any write should cause the fs to read its metadata and detect the problem.
Thanks,
Qu
Quoted 19 lines
Thanks,
ZorroQuoted 15 lines
tests/generic/702 new file +4
-0+ +# success, all done +status=0 +exittests/generic/702.out new file +2
-1@@ -0,0 +1,2 @@ +QA output created by 702 +Detected corrupted fs (expected) --2.38.0