Skip to contentSkip to conversation

fstests: generic: add a test case to check fsverity reads

PATCH Superseded Qu Wenruo 5 participants 9 messages Oct 11 – Oct 17 Btrfs file system

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH v3 1/1 fstests: generic: add a test case to check fsverity reads

There is a bug report that with the recently enabled large folios support, fsverity enabled files on btrfs can trigger reproducible warnings about file corruption, but in the end no error is returned to user space.

It turns out to be a bug in the btrfs fsverity integration, where blocks not included in the read range are passed to be verified.

This is a long-standing bug where btrfs fsverity always assumed that one folio contains exactly one fs block, ignoring the fact that btrfs had block size < page size support before fsverity support was added.

Add a new regression test case for it. The workload is pretty simple:

- Create a 32M file

- Punch a 64K hole for every 128K
This will limit the read bio size to 64K, which is smaller than a lot of large folios, and thus can trigger the btrfs bug.

- Enable fsverity digest and drop page cache

- Read that file

- Verify there is no "FILE CORRUPTED!" warning

---
Changelog
v3:
- Disable signature verification temporarily for the test Pointed out by Dhruva, as it's a user configurable behavior that if signature is required, reading the file without a signature will lead to an EPERM error.

v2:
- Do not use btrfs specific mount options to limit read bio size Use holes to limit the size of read bios instead, now the test case can be generic other than btrfs specific.

- Add the missing _require_check_dmesg call

---
 tests/generic/807     | 65 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/807.out |  2 ++
 2 files changed, 67 insertions(+)
 create mode 100755 tests/generic/807
 create mode 100644 tests/generic/807.out
tests/generic/807 new file +65 -0
new file mode 100755
index 00000000..bdd89a08
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 SUSE S.A.  All Rights Reserved.
+#
+# FS QA Test 807
+#
+# Verify that reading a large file with holes won't trigger fsverity warnings
+#
+. ./common/preamble
+_begin_fstest auto quick verity
+
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"
+
+. ./common/verity
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	_restore_fsverity_signatures
+	rm -f $tmp.*
+}
+
+_require_xfs_io_command "fpunch"
+_require_check_dmesg
+_require_scratch_verity
+_disable_fsverity_signatures
+
+filesize=$(( 32 * 1024 * 1024 ))
+interval=$(( 128 * 1024 ))
+holesize=$(( 64 * 1024 ))
+
+_scratch_mkfs_verity >> $seqres.full
+_scratch_mount
+
+# Create a large enough file so that readahead is triggered, and
+# large folios are allocated during readahead.
+_pwrite_byte "0xcd" 0 "$filesize" "$SCRATCH_MNT/foobar" > /dev/null
+
+# Create a hole for every $interval bytes. This should limit the read to
+# $(( $interval - $holesize)), which should be much smaller than the maximum
+# folio size (2M for x86_64).
+for (( i = 0; i < $(( $filesize / $interval )); i++ )); do
+	$XFS_IO_PROG -c "fpunch $(( $i * $interval )) $holesize" \
+		"$SCRATCH_MNT/foobar" > /dev/null
+done
+
+_fsv_enable "$SCRATCH_MNT/foobar"
+
+# Cycle mount to drop page cache
+_scratch_cycle_mount
+
+# Read the file, unpatched btrfs will verify blocks outside the read range and
+# trigger fs-verity warnings.
+# But the read itself should still be fine, as the failed part will be retried.
+cat "$SCRATCH_MNT/foobar" > /dev/null
+
+# Check for any kernel warnings from fs-verity on file corruption.
+if _check_dmesg_for "FILE CORRUPTED!"; then
+	echo "fs-verity detected corruption"
+fi
+echo "Silence is golden"
+
+_exit 0
tests/generic/807.out new file +2 -0
new file mode 100644
index 00000000..3752a5f7
@@ -0,0 +1,2 @@
+QA output created by 807
+Silence is golden
Signature · 1 line
2.56.0
Tested-by Reviewed-by on v3 1/1
#2 lore raw
In reply to Qu Wenruo
Quoted 26 lines · Qu Wenruo

There is a bug report that with the recently enabled large folios support, fsverity enabled files on btrfs can trigger reproducible warnings about file corruption, but in the end no error is returned to user space.

It turns out to be a bug in the btrfs fsverity integration, where blocks not included in the read range are passed to be verified.

This is a long-standing bug where btrfs fsverity always assumed that one folio contains exactly one fs block, ignoring the fact that btrfs had block size < page size support before fsverity support was added.

Add a new regression test case for it. The workload is pretty simple:

- Create a 32M file

- Punch a 64K hole for every 128K
This will limit the read bio size to 64K, which is smaller than a lot of large folios, and thus can trigger the btrfs bug.

- Enable fsverity digest and drop page cache

- Read that file

- Verify there is no "FILE CORRUPTED!" warning

Quoted 8 lines

---
Changelog
v3:
- Disable signature verification temporarily for the test Pointed out by Dhruva, as it's a user configurable behavior that if signature is required, reading the file without a signature will lead to an EPERM error.

Thanks, looks good.

Regards,
Dhruva

#3 lore raw
In reply to Qu Wenruo

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

tests/generic/807 line 57

Quoted 4 lines
tests/generic/807 new file +4 -0
+# Read the file, unpatched btrfs will verify blocks outside the read range and
+# trigger fs-verity warnings.
+# But the read itself should still be fine, as the failed part will be retried.
+cat "$SCRATCH_MNT/foobar" > /dev/null

How about using md5sum and putting the expected value in the .out file, so the contents are validated too?

tests/generic/807 line 62

Quoted 4 lines
tests/generic/807 new file +4 -0
+# Check for any kernel warnings from fs-verity on file corruption.
+if _check_dmesg_for "FILE CORRUPTED!"; then
+	echo "fs-verity detected corruption"
+fi

Is there a way to use the fserror support being added by
https://lore.kernel.org/fsverity/[email protected]/
instead of relying on the unstable text printed to the kernel log? If the string is changed at all, this test will become ineffective.

- Eric

author
#4 lore raw
Quoted 6 lines · Eric Biggers

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

It need as least 7.2 upstream, which enables large data folios for btrfs by default.

And I can always reproduce it here, mind to provide more info about the passed run?

Quoted 8 lines

tests/generic/807 line 57

Quoted 5 lines
tests/generic/807 new file +4 -0
+# Read the file, unpatched btrfs will verify blocks outside the read range and
+# trigger fs-verity warnings.
+# But the read itself should still be fine, as the failed part will be retried.
+cat "$SCRATCH_MNT/foobar" > /dev/null

How about using md5sum and putting the expected value in the .out file, so the contents are validated too?

That won't catch the problem.

The root problem is that, for a large folio, a read bio can only cover part of the folio.
And at end io time, unpatched btrfs call fsverity_verify_folio() which can have data that is still under read, thus fsverity will report errors. However after each read, the correct data are already copied into the page cache.

So after a fsverity verification failure, MM/VFS will retry the read, and since other parts may have already copied the correct data into page cache, eventually the incorrect fsverity_verify_folio() will succeed, thus no read error is returned to the user space.

So md5sum will always be correct.

Quoted 10 lines

tests/generic/807 line 62

Quoted 5 lines
tests/generic/807 new file +4 -0
+# Check for any kernel warnings from fs-verity on file corruption.
+if _check_dmesg_for "FILE CORRUPTED!"; then
+	echo "fs-verity detected corruption"
+fi

Is there a way to use the fserror support being added by
https://lore.kernel.org/fsverity/[email protected]/
instead of relying on the unstable text printed to the kernel log? If the string is changed at all, this test will become ineffective.

That's a better idea, I'll need to check if it's possible grab fserror info from btrfs.

Thanks,
Qu

Quoted 2 lines

- Eric

#5 lore raw
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · Eric Biggers

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

Quoted 5 lines

It need as least 7.2 upstream, which enables large data folios for btrfs by default.

And I can always reproduce it here, mind to provide more info about the passed run?

This is on the latest mainline (2f2f7eeef8b0). Not sure what else you're looking for.

Quoted 9 lines
Quoted 7 lines

tests/generic/807 line 57

Quoted 4 lines
tests/generic/807 new file +4 -0
+# Read the file, unpatched btrfs will verify blocks outside the read range and
+# trigger fs-verity warnings.
+# But the read itself should still be fine, as the failed part will be retried.
+cat "$SCRATCH_MNT/foobar" > /dev/null

How about using md5sum and putting the expected value in the .out file, so the contents are validated too?

That won't catch the problem.

I know, but it might as well be checked too.

- Eric

#6 lore raw
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · Eric Biggers

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

Quoted 8 lines
Quoted 5 lines

It need as least 7.2 upstream, which enables large data folios for btrfs by default.

And I can always reproduce it here, mind to provide more info about the passed run?

This is on the latest mainline (2f2f7eeef8b0). Not sure what else you're looking for.

It can be racy, since the fsverity call is at endio time, if all the endio of the same large folio is called after all the reads of the folio finished, it can also pass the verification.

But I don't think it's that possible, unless you have blazing fast storage.
As the first hole read should trigger the verification for the whole folio and trigger an fsverity error immediately.

I can enlarge the file size, or even git it a loop to run multiple times, but I just can't figure out why...

Or is it possible that the kernel config doesn't have CONFIG_TRANSPARENT_HUGEPAGE?

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

tests/generic/807 line 57

Quoted 5 lines
tests/generic/807 new file +4 -0
+# Read the file, unpatched btrfs will verify blocks outside the read range and
+# trigger fs-verity warnings.
+# But the read itself should still be fine, as the failed part will be retried.
+cat "$SCRATCH_MNT/foobar" > /dev/null

How about using md5sum and putting the expected value in the .out file, so the contents are validated too?

Quoted 3 lines
Quoted 1 line

That won't catch the problem.

I know, but it might as well be checked too.

I'm fine adding the md5sum output.

Thanks,
Qu

Quoted 2 lines

- Eric

#7 lore raw
Quoted 31 lines · Qu Wenruo
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · Eric Biggers

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

Quoted 8 lines
Quoted 5 lines

It need as least 7.2 upstream, which enables large data folios for btrfs by default.

And I can always reproduce it here, mind to provide more info about the passed run?

This is on the latest mainline (2f2f7eeef8b0). Not sure what else you're looking for.

It can be racy, since the fsverity call is at endio time, if all the endio of the same large folio is called after all the reads of the folio finished, it can also pass the verification.

But I don't think it's that possible, unless you have blazing fast storage. As the first hole read should trigger the verification for the whole folio and trigger an fsverity error immediately.

Quoted 5 lines

I can enlarge the file size, or even git it a loop to run multiple times, but I just can't figure out why...

Or is it possible that the kernel config doesn't have CONFIG_TRANSPARENT_HUGEPAGE?

Indeed, I didn't have CONFIG_TRANSPARENT_HUGEPAGE enabled. Enabling it makes the test fail as expected. Maybe would be helpful to mention that prerequisite in a comment in the test.

- Eric

#8 lore raw
Quoted 35 lines · Eric Biggers
Quoted 34 lines · Qu Wenruo
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Qu Wenruo
Quoted 6 lines · Eric Biggers

tests/generic/807 line 13

Quoted 2 lines · Qu Wenruo
tests/generic/807 new file +2 -0
+_fixed_by_fs_commit btrfs xxxxxxxxxxxx \
+	"btrfs: fix the fsverity callback where unexpected range is verified"

Even without that commit, 'kvm-xfstests -c btrfs generic/807' still passes. Under what circumstances will this test fail?

Quoted 8 lines
Quoted 5 lines

It need as least 7.2 upstream, which enables large data folios for btrfs by default.

And I can always reproduce it here, mind to provide more info about the passed run?

This is on the latest mainline (2f2f7eeef8b0). Not sure what else you're looking for.

It can be racy, since the fsverity call is at endio time, if all the endio of the same large folio is called after all the reads of the folio finished, it can also pass the verification.

But I don't think it's that possible, unless you have blazing fast storage. As the first hole read should trigger the verification for the whole folio and trigger an fsverity error immediately.

I can enlarge the file size, or even git it a loop to run multiple times, but I just can't figure out why...

Quoted 6 lines
Quoted 2 lines

Or is it possible that the kernel config doesn't have CONFIG_TRANSPARENT_HUGEPAGE?

Indeed, I didn't have CONFIG_TRANSPARENT_HUGEPAGE enabled. Enabling it makes the test fail as expected. Maybe would be helpful to mention that prerequisite in a comment in the test.

Sure, I'll address all the comments in the next update.

Although for the fserror reporting part, btrfs doesn't support the full fserror yet. So I'm afraid the test will rely on the dmesg part for error detection for now.

Thanks,
Qu

Quoted 2 lines

- Eric