Skip to contentSkip to conversation

btrfs: exercise NOCOMPRESS range and EOF handling

PATCH Superseded Dhruva G 3 participants 7 messages Oct 11 – Oct 13 1 message not in the archive Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Dhruva G

author
#1 lore raw

PATCH 1/3 btrfs: exercise NOCOMPRESS range and EOF handling

Check that a bounded --nocomp request rewrites compressed extents in the selected range while distant untouched regions still point to their original compressed storage.
Cover a partial final sector, zero length, a length extending past EOF, start at EOF, and the empty-file no-op. Preserve contents and sizes across the conversion and a remount.

Add shared FIEMAP and integrity helpers for the NOCOMPRESS tests. Require full mapping coverage and inspect every overlapping extent so holes and partly converted ranges cannot pass. Include a small ioctl helper to probe kernel support and check errno without btrfs-progs translation.

---
 .gitignore                |   1 +
 common/btrfs_nocomp       | 134 ++++++++++++++++++++++++++++++++++++++
 src/Makefile              |   2 +-
 src/btrfs_defrag_nocomp.c |  63 ++++++++++++++++++
 tests/btrfs/355           |  80 +++++++++++++++++++++++
 tests/btrfs/355.out       |   2 +
 6 files changed, 281 insertions(+), 1 deletion(-)
 create mode 100644 common/btrfs_nocomp
 create mode 100644 src/btrfs_defrag_nocomp.c
 create mode 100755 tests/btrfs/355
 create mode 100644 tests/btrfs/355.out
.gitignore +1 -0
index 0b6b9452..1bfa12ef 100644
@@ -216,6 +216,7 @@ tags
 /src/truncate
 /src/t_btrfs_received_uuid_ioctl
 /src/btrfs_ioctl
+/src/btrfs_defrag_nocomp
 
 # Symlinked files
 /tests/generic/035.out
common/btrfs_nocomp new file +134 -0

134 lines not on this page · raw message

src/Makefile +1 -1
index 76cf50c3..07bf681d 100644
@@ -36,7 +36,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \
 	fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \
 	detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \
 	uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \
-	rw_hint fs-monitor btrfs_ioctl
+	rw_hint fs-monitor btrfs_ioctl btrfs_defrag_nocomp
 
 EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \
 	      btrfs_crc32c_forged_name.py popdir.pl popattr.py \
src/btrfs_defrag_nocomp.c new file +63 -0

63 lines not on this page · raw message

tests/btrfs/355 new file +80 -0

80 lines not on this page · raw message

tests/btrfs/355.out new file +2 -0
new file mode 100644
index 00000000..4e4c5e98
@@ -0,0 +1,2 @@
+QA output created by 355
+Silence is golden
Signature · 1 line
2.43.0
#2 lore raw
Quoted 44 lines · Dhruva G

Check that a bounded --nocomp request rewrites compressed extents in the selected range while distant untouched regions still point to their original compressed storage.
Cover a partial final sector, zero length, a length extending past EOF, start at EOF, and the empty-file no-op. Preserve contents and sizes across the conversion and a remount.

Add shared FIEMAP and integrity helpers for the NOCOMPRESS tests. Require full mapping coverage and inspect every overlapping extent so holes and partly converted ranges cannot pass. Include a small ioctl helper to probe kernel support and check errno without btrfs-progs translation.

---
  .gitignore                |   1 +
  common/btrfs_nocomp       | 134 ++++++++++++++++++++++++++++++++++++++
  src/Makefile              |   2 +-
  src/btrfs_defrag_nocomp.c |  63 ++++++++++++++++++
  tests/btrfs/355           |  80 +++++++++++++++++++++++
  tests/btrfs/355.out       |   2 +
  6 files changed, 281 insertions(+), 1 deletion(-)

create mode 100644 common/btrfs_nocomp
create mode 100644 src/btrfs_defrag_nocomp.c
create mode 100755 tests/btrfs/355
create mode 100644 tests/btrfs/355.out

.gitignore +1 -0
index 0b6b9452..1bfa12ef 100644
@@ -216,6 +216,7 @@ tags
  /src/truncate
  /src/t_btrfs_received_uuid_ioctl
  /src/btrfs_ioctl
+/src/btrfs_defrag_nocomp
 
  # Symlinked files
  /tests/generic/035.out
common/btrfs_nocomp new file +0 -0
new file mode 100644
index 00000000..7e2111e1

common/btrfs_nocomp line 3

Quoted 3 lines
common/btrfs_nocomp new file +3 -0
@@ -0,0 +1,134 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# Helpers for BTRFS_DEFRAG_RANGE_NOCOMPRESS tests.

I don't like a dedicated file just do all those.
Why not put them into common/btrfs?

common/btrfs_nocomp line 5

Quoted 2 lines
common/btrfs_nocomp new file +2 -0
+
+_nc_require()

And "nc" is too generic, _require_btrfs_defrag_nocompress() would make more sense.

All the namings here look just some temporary scaffolding, not for generic purposes.

Quoted 16 lines
common/btrfs_nocomp new file +16 -0
+{
+	_require_scratch
+	_require_test_program btrfs_defrag_nocomp
+	_require_xfs_io_command fiemap ranged
+	_require_btrfs_command filesystem defragment --nocomp
+	_require_btrfs_no_nodatacow
+	_require_btrfs_no_nodatasum
+	_require_btrfs_no_autodefrag
+	_require_btrfs_support_sectorsize 4096
+	_wants_kernel_commit 009b2056cb25 \
+		"btrfs: defrag: add flag to force no-compression"
+}
+
+_nc_write()
+{
+	$XFS_IO_PROG -f -c "pwrite -S $4 -b 128K $2 $3" -c fsync "$1" \

common/btrfs_nocomp line 31

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+		>> "$seqres.full" 2>&1 || _fail "write failed: $1"
+}
+
+_nc_probe()
+{
+	local result
+
+	_nc_write "$SCRATCH_MNT/probe" 0 131072 0x5a
+	result=$($here/src/btrfs_defrag_nocomp "$SCRATCH_MNT/probe" 0 131072) ||
+		_fail "NOCOMPRESS probe failed"

Why we need such dedicated probe program while the kernel is already rejecting unknown flags with ENOTTY?

And I think this probing should also be done inside the _require_btrfs_defrag_nocompress().

Quoted 19 lines
common/btrfs_nocomp new file +19 -0
+	case "$result" in
+	OK) ;;
+	EOPNOTSUPP|ENOTTY) _notrun "kernel does not support NOCOMPRESS" ;;
+	*) _fail "unexpected NOCOMPRESS probe result: $result" ;;
+	esac
+	rm -f "$SCRATCH_MNT/probe"
+}
+
+_nc_defrag()
+{
+	local file="${@: -1}"
+
+	$BTRFS_UTIL_PROG filesystem defragment --nocomp "$@" \
+		>> "$seqres.full" 2>&1 || _fail "NOCOMPRESS defrag failed"
+	$XFS_IO_PROG -c fsync "$file" >> "$seqres.full" 2>&1 ||
+		_fail "fsync after defrag failed"
+}
+
+_nc_map()

common/btrfs_nocomp line 60

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+{
+	$XFS_IO_PROG -c "fiemap -v $2 $3" "$1" > "$tmp.nc-map" \
+		2>> "$seqres.full" || _fail "fiemap failed: $1"
+	cat "$tmp.nc-map" >> "$seqres.full"
+}
+
+# FIEMAP prints logical offsets in 512-byte units. Check every overlapping
+# extent and complete coverage, so a hole or a partly converted range fails.
+# The caller supplies byte offsets and lengths, including a partial EOF sector.
+_nc_check_range()

This looks a little overkilled to me.

For regular hole/compression flags, what about filtering out the non-hole logical bytenr (e.g. replace them with 'XXX' or 'regular'), then output to the stdout, and use golden output to verify?

author
#3 lore raw
Quoted 54 lines · Qu Wenruo
Quoted 44 lines · Dhruva G

Check that a bounded --nocomp request rewrites compressed extents in the selected range while distant untouched regions still point to their original compressed storage.
Cover a partial final sector, zero length, a length extending past EOF, start at EOF, and the empty-file no-op. Preserve contents and sizes across the conversion and a remount.

Add shared FIEMAP and integrity helpers for the NOCOMPRESS tests. Require full mapping coverage and inspect every overlapping extent so holes and partly converted ranges cannot pass. Include a small ioctl helper to probe kernel support and check errno without btrfs-progs translation.

---
  .gitignore                |   1 +
  common/btrfs_nocomp       | 134 ++++++++++++++++++++++++++++++++++++++
  src/Makefile              |   2 +-
  src/btrfs_defrag_nocomp.c |  63 ++++++++++++++++++
  tests/btrfs/355           |  80 +++++++++++++++++++++++
  tests/btrfs/355.out       |   2 +
  6 files changed, 281 insertions(+), 1 deletion(-)

create mode 100644 common/btrfs_nocomp
create mode 100644 src/btrfs_defrag_nocomp.c
create mode 100755 tests/btrfs/355
create mode 100644 tests/btrfs/355.out

.gitignore +1 -0
index 0b6b9452..1bfa12ef 100644
@@ -216,6 +216,7 @@ tags
  /src/truncate
  /src/t_btrfs_received_uuid_ioctl
  /src/btrfs_ioctl
+/src/btrfs_defrag_nocomp
 
  # Symlinked files
  /tests/generic/035.out
common/btrfs_nocomp new file +0 -0
new file mode 100644
index 00000000..7e2111e1

common/btrfs_nocomp line 3

Quoted 3 lines
common/btrfs_nocomp new file +3 -0
@@ -0,0 +1,134 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# Helpers for BTRFS_DEFRAG_RANGE_NOCOMPRESS tests.

I don't like a dedicated file just do all those.
Why not put them into common/btrfs?

Quoted 8 lines

common/btrfs_nocomp line 5

Quoted 2 lines
common/btrfs_nocomp new file +2 -0
+
+_nc_require()

And "nc" is too generic, _require_btrfs_defrag_nocompress() would make more sense.

All the namings here look just some temporary scaffolding, not for generic purposes.

Agreed. I'll move the shared helpers into common/btrfs and use more descriptive names in v2.

Quoted 31 lines
Quoted 17 lines
common/btrfs_nocomp new file +16 -0
+{
+	_require_scratch
+	_require_test_program btrfs_defrag_nocomp
+	_require_xfs_io_command fiemap ranged
+	_require_btrfs_command filesystem defragment --nocomp
+	_require_btrfs_no_nodatacow
+	_require_btrfs_no_nodatasum
+	_require_btrfs_no_autodefrag
+	_require_btrfs_support_sectorsize 4096
+	_wants_kernel_commit 009b2056cb25 \
+		"btrfs: defrag: add flag to force no-compression"
+}
+
+_nc_write()
+{
+	$XFS_IO_PROG -f -c "pwrite -S $4 -b 128K $2 $3" -c fsync "$1" \

common/btrfs_nocomp line 31

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+		>> "$seqres.full" 2>&1 || _fail "write failed: $1"
+}
+
+_nc_probe()
+{
+	local result
+
+	_nc_write "$SCRATCH_MNT/probe" 0 131072 0x5a
+	result=$($here/src/btrfs_defrag_nocomp "$SCRATCH_MNT/probe" 0 131072) ||
+		_fail "NOCOMPRESS probe failed"

Why we need such dedicated probe program while the kernel is already rejecting unknown flags with ENOTTY?

Quoted 2 lines

And I think this probing should also be done inside the _require_btrfs_defrag_nocompress().

Agreed. I'll do this in v2.

Quoted 32 lines
Quoted 20 lines
common/btrfs_nocomp new file +19 -0
+	case "$result" in
+	OK) ;;
+	EOPNOTSUPP|ENOTTY) _notrun "kernel does not support NOCOMPRESS" ;;
+	*) _fail "unexpected NOCOMPRESS probe result: $result" ;;
+	esac
+	rm -f "$SCRATCH_MNT/probe"
+}
+
+_nc_defrag()
+{
+	local file="${@: -1}"
+
+	$BTRFS_UTIL_PROG filesystem defragment --nocomp "$@" \
+		>> "$seqres.full" 2>&1 || _fail "NOCOMPRESS defrag failed"
+	$XFS_IO_PROG -c fsync "$file" >> "$seqres.full" 2>&1 ||
+		_fail "fsync after defrag failed"
+}
+
+_nc_map()

common/btrfs_nocomp line 60

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+{
+	$XFS_IO_PROG -c "fiemap -v $2 $3" "$1" > "$tmp.nc-map" \
+		2>> "$seqres.full" || _fail "fiemap failed: $1"
+	cat "$tmp.nc-map" >> "$seqres.full"
+}
+
+# FIEMAP prints logical offsets in 512-byte units. Check every overlapping
+# extent and complete coverage, so a hole or a partly converted range fails.
+# The caller supplies byte offsets and lengths, including a partial EOF sector.
+_nc_check_range()

This looks a little overkilled to me.

Quoted 3 lines
Quoted 1 line

For regular hole/compression flags, what about filtering out the

non-hole logical bytenr (e.g. replace them with 'XXX' or 'regular'), then output to the stdout, and use golden output to verify?

For patch 3, the test also needs to distinguish compressed inline extents from compressed non-inline extents. _filter_fiemap_flags 1 retains the encoded flag, but currently filters out the inline flag.

Would you prefer a small opt-in extension to retain the inline flag, or a test-local filter? With the extension, existing callers would retain their current output, and the inline test could use:
_filter_fiemap_flags 1 1.

Thanks for reviewing and suggestions!

Regards,
Dhruva Gole

#4 lore raw
Quoted 1 line · Dhruva G

On 12-10-2026 03:30, Qu Wenruo wrote:

[...]

Quoted 10 lines
Quoted 4 lines

This looks a little overkilled to me.

Quoted 1 line

For regular hole/compression flags, what about filtering out the

non-hole logical bytenr (e.g. replace them with 'XXX' or 'regular'), then output to the stdout, and use golden output to verify?

For patch 3, the test also needs to distinguish compressed inline extents from compressed non-inline extents. _filter_fiemap_flags 1 retains the encoded flag, but currently filters out the inline flag.

Would you prefer a small opt-in extension to retain the inline flag, or a test-local filter?

I think for now a local filter would be good enough if there is only one test case requiring this.

But in the long run, we want a filter that only filters out the bytenr. Maybe something like _filter_fiemap_keep_flags().

Quoted 2 lines

With the extension, existing callers would retain their current output, and the inline test could use:
_filter_fiemap_flags 1 1.

More than one boolean parameters in a row is never a good idea to me.

Thus I'd prefer a dedicated filter.

Thanks,
Qu

Quoted 5 lines

Thanks for reviewing and suggestions!

Regards,
Dhruva Gole

#5 lore raw
In reply to Dhruva G
Quoted 45 lines · Dhruva G

Check that a bounded --nocomp request rewrites compressed extents in the selected range while distant untouched regions still point to their original compressed storage.
Cover a partial final sector, zero length, a length extending past EOF, start at EOF, and the empty-file no-op. Preserve contents and sizes across the conversion and a remount.

Add shared FIEMAP and integrity helpers for the NOCOMPRESS tests. Require full mapping coverage and inspect every overlapping extent so holes and partly converted ranges cannot pass. Include a small ioctl helper to probe kernel support and check errno without btrfs-progs translation.

---
 .gitignore                |   1 +
 common/btrfs_nocomp       | 134 ++++++++++++++++++++++++++++++++++++++
 src/Makefile              |   2 +-
 src/btrfs_defrag_nocomp.c |  63 ++++++++++++++++++
 tests/btrfs/355           |  80 +++++++++++++++++++++++
 tests/btrfs/355.out       |   2 +
 6 files changed, 281 insertions(+), 1 deletion(-)
 create mode 100644 common/btrfs_nocomp
 create mode 100644 src/btrfs_defrag_nocomp.c
 create mode 100755 tests/btrfs/355
 create mode 100644 tests/btrfs/355.out
.gitignore +1 -0
index 0b6b9452..1bfa12ef 100644
@@ -216,6 +216,7 @@ tags
 /src/truncate
 /src/t_btrfs_received_uuid_ioctl
 /src/btrfs_ioctl
+/src/btrfs_defrag_nocomp
 
 # Symlinked files
 /tests/generic/035.out
common/btrfs_nocomp new file +0 -0
new file mode 100644
index 00000000..7e2111e1

common/btrfs_nocomp line 6

Quoted 6 lines
common/btrfs_nocomp new file +6 -0
@@ -0,0 +1,134 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# Helpers for BTRFS_DEFRAG_RANGE_NOCOMPRESS tests.
+
+_nc_require()
+{

Such a odd name because:

1. All our require functions are named as _require_something, but here the logic is inverted.

2. "nc" is not really intuitive, is rather cryptic, specially taking into account this is specific for the no compress option for defrag - there's no indication in the name this is defrag related.

Quoted 7 lines
common/btrfs_nocomp new file +7 -0
+       _require_scratch
+       _require_test_program btrfs_defrag_nocomp
+       _require_xfs_io_command fiemap ranged
+       _require_btrfs_command filesystem defragment --nocomp
+       _require_btrfs_no_nodatacow
+       _require_btrfs_no_nodatasum
+       _require_btrfs_no_autodefrag

common/btrfs_nocomp line 23

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+       _require_btrfs_support_sectorsize 4096
+       _wants_kernel_commit 009b2056cb25 \
+               "btrfs: defrag: add flag to force no-compression"
+}
+
+_nc_write()
+{
+       $XFS_IO_PROG -f -c "pwrite -S $4 -b 128K $2 $3" -c fsync "$1" \
+               >> "$seqres.full" 2>&1 || _fail "write failed: $1"
+}

A useless wrapper. Just make the tests call XFS_IO_PROG, this does not add any value at all.

Quoted 5 lines
common/btrfs_nocomp new file +5 -0
+
+_nc_probe()
+{
+       local result
+

common/btrfs_nocomp line 38

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+       _nc_write "$SCRATCH_MNT/probe" 0 131072 0x5a
+       result=$($here/src/btrfs_defrag_nocomp "$SCRATCH_MNT/probe" 0 131072) ||
+               _fail "NOCOMPRESS probe failed"
+       case "$result" in
+       OK) ;;
+       EOPNOTSUPP|ENOTTY) _notrun "kernel does not support NOCOMPRESS" ;;
+       *) _fail "unexpected NOCOMPRESS probe result: $result" ;;
+       esac
+       rm -f "$SCRATCH_MNT/probe"
+}

Too complex, we usually just call the user space utility on a simple scenario that should never fail and if it fails we skip the test saying the feature is not supported by the kernel.
And we do it in a _require_something function as well.

common/btrfs_nocomp line 48

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+
+_nc_defrag()
+{
+       local file="${@: -1}"
+
+       $BTRFS_UTIL_PROG filesystem defragment --nocomp "$@" \
+               >> "$seqres.full" 2>&1 || _fail "NOCOMPRESS defrag failed"
+       $XFS_IO_PROG -c fsync "$file" >> "$seqres.full" 2>&1 ||
+               _fail "fsync after defrag failed"
+}

Another pointless wrapper.
I also dislike this practice of adding a _fail for everything. Just let the tests fail with a golden output mismatch.

common/btrfs_nocomp line 55

Quoted 7 lines
common/btrfs_nocomp new file +7 -0
+
+_nc_map()
+{
+       $XFS_IO_PROG -c "fiemap -v $2 $3" "$1" > "$tmp.nc-map" \
+               2>> "$seqres.full" || _fail "fiemap failed: $1"
+       cat "$tmp.nc-map" >> "$seqres.full"
+}

Yet another wrapper.

Quoted 34 lines
common/btrfs_nocomp new file +34 -0
+
+# FIEMAP prints logical offsets in 512-byte units. Check every overlapping
+# extent and complete coverage, so a hole or a partly converted range fails.
+# The caller supplies byte offsets and lengths, including a partial EOF sector.
+_nc_check_range()
+{
+       _nc_map "$1" "$2" "$3"
+       $AWK_PROG -v start="$2" -v len="$3" -v mode="$4" '
+       BEGIN {
+               first = int(start / 512)
+               end = int((start + len + 511) / 512)
+               next_block = first
+       }
+       $1 ~ /^[0-9]+:$/ && $5 ~ /^0x/ {
+               gsub(/[^0-9]/, " ", $2)
+               split($2, logical)
+               lo = logical[1] + 0
+               hi = logical[2] + 1
+               if (hi <= first || lo >= end)
+                       next
+               if (mode == "hole") {
+                       bad = 1
+                       next
+               }
+               flags = strtonum($5)
+               encoded = and(flags, 0x8) != 0
+               unwritten = and(flags, 0x800) != 0
+               inline = and(flags, 0x200) != 0
+               if (and(flags, 0x6) || lo > next_block)
+                       bad = 1
+               if (mode == "compressed" && (!encoded || unwritten || inline))
+                       bad = 1
+               if (mode == "regular" && (encoded || unwritten || inline))
+                       bad = 1

common/btrfs_nocomp line 99

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+               if (mode == "inline" && (!encoded || unwritten || !inline))
+                       bad = 1
+               if (mode == "prealloc" && (encoded || !unwritten || inline))
+                       bad = 1
+               if (hi > next_block)
+                       next_block = hi
+       }
+       END { exit bad || (mode != "hole" && next_block < end) }
+       ' "$tmp.nc-map" || _fail "unexpected $4 mapping: $1 ($2, $3)"
+}

Why do we need this level of complexity?
Adding an awk script to process fiemap output and detect unexpected results is overly complex.

Why not do what we do in every other test that checks the extent layout of a file?
Simply call fiemap in the test and add the expected output to the golden output file.
That's simple, does the work and it's the preferred way to do things in fstests.

Quoted 4 lines
common/btrfs_nocomp new file +4 -0
+
+# Ignore flags such as LAST, which may change when another range is rewritten.
+_nc_mapping()
+{

common/btrfs_nocomp line 113

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+       _nc_map "$1" "$2" "$3"
+       _filter_xfs_io_fiemap < "$tmp.nc-map" | \
+               $AWK_PROG '{ print $1, $2, $3, $4 }'
+}
+
+_nc_save_file()
+{
+       md5sum "$1" >> "$tmp.nc-checksums" || _fail "checksum failed: $1"
+       stat -c '%s %n' "$1" >> "$tmp.nc-sizes" || _fail "stat failed: $1"
+}

This name "save_file" is totally confusing.

Quoted 11 lines
common/btrfs_nocomp new file +11 -0
+
+_nc_verify_files()
+{
+       local size file actual
+
+       md5sum -c "$tmp.nc-checksums" >> "$seqres.full" 2>&1 ||
+               _fail "file contents changed"
+       while read -r size file; do
+               actual=$(stat -c %s "$file") || _fail "stat failed: $file"
+               [ "$actual" = "$size" ] || _fail "file size changed: $file"
+       done < "$tmp.nc-sizes"

common/btrfs_nocomp line 134

Quoted 10 lines
common/btrfs_nocomp new file +10 -0
+}
+
+_nc_raw_expect()
+{
+       local expected="$1" result
+       shift
+       result=$($here/src/btrfs_defrag_nocomp "$@") || _fail "raw ioctl failed"
+       [ "$result" = "$expected" ] ||
+               _fail "expected $expected, got $result"
+}

This is all stuff we can do in the tests by calling md5sum and placing the expected digest in the golden output file, like we do in many other tests.

Quoted 19 lines
src/Makefile +1 -1
index 76cf50c3..07bf681d 100644
@@ -36,7 +36,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \
        fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \
        detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \
        uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \
-       rw_hint fs-monitor btrfs_ioctl
+       rw_hint fs-monitor btrfs_ioctl btrfs_defrag_nocomp
 
 EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \
              btrfs_crc32c_forged_name.py popdir.pl popattr.py \
src/btrfs_defrag_nocomp.c new file +0 -0
new file mode 100644
index 00000000..4b066418

src/btrfs_defrag_nocomp.c line 3

Quoted 3 lines
src/btrfs_defrag_nocomp.c new file +3 -0
@@ -0,0 +1,63 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+/* Issue NOCOMPRESS requests without userspace range/error translation. */

Why do we need a C program?
Why not just have the tests call btrfs-progs: btrfs filesystem defragment --nocomp ....

This patchset adds an insane level of complexity for no good reason, making the tests hard to read and maintain.

I can't see why just calling btrfs-progs, fiemap and md5sum directly in the tests and then placing the expected output in the golden file is not enough...

Thanks.

Quoted 157 lines
src/btrfs_defrag_nocomp.c new file +60 -0
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/btrfs.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#ifndef BTRFS_DEFRAG_RANGE_NOCOMPRESS
+#define BTRFS_DEFRAG_RANGE_NOCOMPRESS (1ULL << 3)
+#endif
+
+static unsigned long long number(const char *s)
+{
+       char *end;
+       unsigned long long value;
+
+       errno = 0;
+       value = strtoull(s, &end, 0);
+       if (errno || *end || end == s || *s == '-') {
+               fprintf(stderr, "invalid number: %s\n", s);
+               exit(2);
+       }
+       return value;
+}
+
+int main(int argc, char **argv)
+{
+       struct btrfs_ioctl_defrag_range_args range = { 0 };
+       int fd, err = 0;
+       const char *result;
+
+       if (argc != 4) {
+               fprintf(stderr, "usage: %s file start len\n", argv[0]);
+               return 2;
+       }
+       range.flags = BTRFS_DEFRAG_RANGE_NOCOMPRESS;
+       range.start = number(argv[2]);
+       range.len = number(argv[3]);
+       fd = open(argv[1], O_RDONLY | O_CLOEXEC);
+       if (fd < 0) {
+               perror("open");
+               return 2;
+       }
+       if (ioctl(fd, BTRFS_IOC_DEFRAG_RANGE, &range) < 0)
+               err = errno;
+       close(fd);
+       switch (err) {
+       case 0: result = "OK"; break;
+       case EINVAL: result = "EINVAL"; break;
+       case EOPNOTSUPP: result = "EOPNOTSUPP"; break;
+       case ENOTTY: result = "ENOTTY"; break;
+       default:
+               fprintf(stderr, "unexpected errno %d (%s)\n", err, strerror(err));
+               return 1;
+       }
+       puts(result);
+       return 0;
+}
tests/btrfs/355 new file +80 -0
new file mode 100755
index 00000000..12befb91
@@ -0,0 +1,80 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# FS QA Test 355
+#
+# Verify bounded NOCOMPRESS conversion and EOF range handling.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress fiemap
+
+. ./common/filter
+. ./common/btrfs_nocomp
+
+_nc_require
+_scratch_mkfs -s 4k >> "$seqres.full" 2>&1 || _fail "mkfs failed"
+_scratch_mount -o compress=zlib
+_nc_probe
+
+file="$SCRATCH_MNT/range"
+_nc_write "$file" 0 4194304 0x5a
+_nc_save_file "$file"
+_nc_check_range "$file" 0 4194304 compressed
+_nc_mapping "$file" 0 131072 > "$tmp.prefix"
+_nc_mapping "$file" 4063232 131072 > "$tmp.suffix"
+
+_nc_defrag -s 1M -l 512K "$file"
+_nc_check_range "$file" 1048576 524288 regular
+_nc_check_range "$file" 0 1048576 compressed
+_nc_check_range "$file" 1572864 2621440 compressed
+_nc_mapping "$file" 0 131072 > "$tmp.prefix-after"
+_nc_mapping "$file" 4063232 131072 > "$tmp.suffix-after"
+cmp -s "$tmp.prefix" "$tmp.prefix-after" || _fail "prefix mapping changed"
+cmp -s "$tmp.suffix" "$tmp.suffix-after" || _fail "suffix mapping changed"
+
+# The two-sector tail can be compressed, unlike a single-sector tail on a
+# 4K filesystem. First convert a one-byte range, then use an unaligned start
+# and end the request at EOF, one byte into the final 4K sector.
+tail="$SCRATCH_MNT/partial"
+_nc_write "$tail" 0 135169 0x39
+_nc_save_file "$tail"
+_nc_check_range "$tail" 131072 4097 compressed
+_nc_defrag -s 128K -l 1 "$tail"
+_nc_check_range "$tail" 131072 4096 regular
+_nc_defrag -s 131073 -l 4096 "$tail"
+_nc_check_range "$tail" 131072 4097 regular
+
+# Zero length means to EOF; a length extending past EOF must be clamped.
+for mode in zero huge; do
+       f="$SCRATCH_MNT/$mode"
+       _nc_write "$f" 0 262144 0x38
+       _nc_save_file "$f"
+       _nc_check_range "$f" 0 262144 compressed
+       case "$mode" in
+       zero) _nc_defrag -s 128K -l 0 "$f" ;;
+       huge) _nc_defrag -s 128K -l 16M "$f" ;;
+       esac
+       _nc_check_range "$f" 0 131072 compressed
+       _nc_check_range "$f" 131072 131072 regular
+done
+
+# A nonempty file rejects start == EOF, but an empty file is a no-op.
+_nc_raw_expect EINVAL "$file" 4194304 4096
+touch "$SCRATCH_MNT/empty" || _fail "create empty file failed"
+_nc_save_file "$SCRATCH_MNT/empty"
+_nc_raw_expect OK "$SCRATCH_MNT/empty" 0 0
+_nc_verify_files
+_scratch_cycle_mount compress=zlib
+_nc_verify_files
+_nc_check_range "$file" 1048576 524288 regular
+_nc_check_range "$file" 0 1048576 compressed
+_nc_check_range "$file" 1572864 2621440 compressed
+_nc_check_range "$tail" 131072 4097 regular
+for mode in zero huge; do
+       _nc_check_range "$SCRATCH_MNT/$mode" 0 131072 compressed
+       _nc_check_range "$SCRATCH_MNT/$mode" 131072 131072 regular
+done
+
+echo "Silence is golden"
+_exit 0
tests/btrfs/355.out new file +2 -1
new file mode 100644
index 00000000..4e4c5e98
@@ -0,0 +1,2 @@
+QA output created by 355
+Silence is golden
--

2.43.0

author
#6 lore raw

PATCH 2/3 btrfs: check NOCOMPRESS preserves holes and preallocation

In reply to a message not in the archive

Place compressed data islands around holes and preallocated extents, including a KEEP_SIZE reservation beyond EOF. Defragment the file with --nocomp and require every data island to become uncompressed.

Check that each hole stays unmapped and both reservations retain their complete physical mappings and UNWRITTEN state. Verify the layout, contents, and file size again after remounting with compression enabled. This extends btrfs/257 to NOCOMPRESS and compressed sparse files.

---
 tests/btrfs/356     | 71 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/356.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/btrfs/356
 create mode 100644 tests/btrfs/356.out
tests/btrfs/356 new file +71 -0
new file mode 100755
index 00000000..829c6e95
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# FS QA Test 356
+#
+# Verify NOCOMPRESS skips holes and preallocated extents.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress prealloc fiemap
+
+. ./common/filter
+. ./common/btrfs_nocomp
+
+_nc_require
+_require_xfs_io_command falloc -k
+_scratch_mkfs -s 4k >> "$seqres.full" 2>&1 || _fail "mkfs failed"
+_scratch_mount -o compress=zlib
+_nc_probe
+
+file="$SCRATCH_MNT/islands"
+_nc_write "$file" 0 131072 0x31
+_nc_write "$file" 524288 131072 0x32
+_nc_write "$file" 1048576 131072 0x33
+for offset in 0 524288 1048576; do
+	_nc_check_range "$file" "$offset" 131072 compressed
+done
+$XFS_IO_PROG -c 'falloc 256K 128K' -c 'falloc -k 1280K 128K' -c fsync \
+	"$file" >> "$seqres.full" 2>&1 || _fail "fallocate failed"
+[ "$(stat -c %s "$file")" = 1179648 ] || _fail "fallocate extended EOF"
+_nc_save_file "$file"
+
+check_layout()
+{
+	local offset
+
+	for offset in 262144 1310720; do
+		_nc_check_range "$file" "$offset" 131072 prealloc
+		_nc_mapping "$file" "$offset" 131072 > "$tmp.prealloc-$offset-now"
+		cmp -s "$tmp.prealloc-$offset" "$tmp.prealloc-$offset-now" ||
+			_fail "prealloc mapping changed at $offset"
+	done
+	_nc_check_range "$file" 131072 131072 hole
+	_nc_check_range "$file" 393216 131072 hole
+	_nc_check_range "$file" 655360 393216 hole
+	_nc_check_range "$file" 1179648 131072 hole
+}
+
+# Include the KEEP_SIZE reservation beyond EOF in the baseline. FIEMAP
+# reports preallocation there even though there are no bytes to read.
+for offset in 262144 1310720; do
+	_nc_check_range "$file" "$offset" 131072 prealloc
+	_nc_mapping "$file" "$offset" 131072 > "$tmp.prealloc-$offset"
+done
+check_layout
+_nc_defrag "$file"
+for offset in 0 524288 1048576; do
+	_nc_check_range "$file" "$offset" 131072 regular
+done
+check_layout
+_nc_verify_files
+
+_scratch_cycle_mount compress=zlib
+_nc_verify_files
+check_layout
+for offset in 0 524288 1048576; do
+	_nc_check_range "$file" "$offset" 131072 regular
+done
+
+echo "Silence is golden"
+_exit 0
tests/btrfs/356.out new file +2 -0
new file mode 100644
index 00000000..78eedd26
@@ -0,0 +1,2 @@
+QA output created by 356
+Silence is golden
Signature · 1 line
2.43.0
author
#7 lore raw

PATCH 3/3 btrfs: cover NOCOMPRESS inline eligibility and short tails

In reply to a message not in the archive

Check that --nocomp preserves a compressed inline extent while it remains eligible under max_inline, including after a remount. Lower max_inline to zero and require the same extent to become regular, uncompressed data.

Exercise one-byte, sector-minus-one, sector-plus-one, and maximum-compressed-extent-plus-one file sizes. Check every mapped byte range for regular uncompressed data after defrag, and verify all file contents, sizes, and converted mappings again after a remount.

Use FIEMAP INLINE and ENCODED flags instead of parsing dump-tree output, and skip configurations without inline extent creation support.

---
 tests/btrfs/357     | 59 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/357.out |  2 ++
 2 files changed, 61 insertions(+)
 create mode 100755 tests/btrfs/357
 create mode 100644 tests/btrfs/357.out
tests/btrfs/357 new file +59 -0
new file mode 100755
index 00000000..0d7b1e33
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# FS QA Test 357
+#
+# Verify NOCOMPRESS inline eligibility and short EOF tails.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress fiemap
+
+. ./common/filter
+. ./common/btrfs_nocomp
+
+_nc_require
+_require_btrfs_inline_extents_creation
+_scratch_mkfs -s 4k >> "$seqres.full" 2>&1 || _fail "mkfs failed"
+_scratch_mount -o compress=zlib,max_inline=4096
+_nc_probe
+
+file="$SCRATCH_MNT/inline"
+_nc_write "$file" 0 2048 0x34
+_nc_save_file "$file"
+_nc_check_range "$file" 0 2048 inline
+_nc_defrag "$file"
+_nc_check_range "$file" 0 2048 inline
+_nc_verify_files
+_scratch_cycle_mount compress=zlib,max_inline=4096
+_nc_check_range "$file" 0 2048 inline
+_nc_verify_files
+
+# Lowering max_inline makes the same extent eligible for conversion.
+_scratch_cycle_mount compress=zlib,max_inline=0
+_nc_defrag "$file"
+_nc_check_range "$file" 0 2048 regular
+_nc_verify_files
+
+for length in 1 4095 4097 131073; do
+	f="$SCRATCH_MNT/tail-$length"
+	_nc_write "$f" 0 "$length" 0x36
+	_nc_save_file "$f"
+	# One-sector files cannot save a sector through compression. The larger
+	# fixtures must contain compressed data before testing conversion.
+	if [ "$length" -gt 4096 ]; then
+		_nc_check_range "$f" 0 4096 compressed
+	fi
+	_nc_defrag "$f"
+	_nc_check_range "$f" 0 "$length" regular
+done
+_nc_verify_files
+_scratch_cycle_mount compress=zlib,max_inline=0
+_nc_verify_files
+_nc_check_range "$file" 0 2048 regular
+for length in 1 4095 4097 131073; do
+	_nc_check_range "$SCRATCH_MNT/tail-$length" 0 "$length" regular
+done
+
+echo "Silence is golden"
+_exit 0
tests/btrfs/357.out new file +2 -0
new file mode 100644
index 00000000..25875c6d
@@ -0,0 +1,2 @@
+QA output created by 357
+Silence is golden
Signature · 1 line
2.43.0