Skip to contentSkip to conversation

btrfs: add test case to verify "btrfs filesystem defrag -c" behavior

PATCH Superseded Qu Wenruo 3 participants 3 messages Feb 2, 2022 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 btrfs: add test case to verify "btrfs filesystem defrag -c" behavior

Despite the regular file defragging, "btrfs filesystem defrag" provides an option, -c, to convert all data extents (except holes and preallocated ranges) to a new compression algorithm.

The special behavior here is, unlike defrag which is not going to touch extents which are adajacent to preallocated/hole ranges, with -c, all non-hole/non-preallocated extents should be defragged and converted to the new compression algorithm.

This test case will ensure the old behavior is properly kept.

Currently both old kernels (v5.15 and older) and newer kernel with refactored defrag (v5.16 and newer) can pass the tests.

---
 tests/btrfs/258     | 153 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/258.out |   2 +
 2 files changed, 155 insertions(+)
 create mode 100755 tests/btrfs/258
 create mode 100644 tests/btrfs/258.out
tests/btrfs/258 new file +153 -0
new file mode 100755
index 00000000..a82e5af9
@@ -0,0 +1,153 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 258
+#
+# Make sure "btrfs filesystem defrag" can still convert the compression
+# algorithm of all regular extents.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress
+
+# Override the default cleanup function.
+# _cleanup()
+# {
+# 	cd /
+# 	rm -r -f $tmp.*
+# }
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch
+
+get_inode_number()
+{
+	local file="$1"
+
+	stat -c "%i" "$file"
+}
+
+get_file_extent()
+{
+	local file="$1"
+	local offset="$2"
+	local ino=$(get_inode_number "$file")
+	local file_extent_key="($ino EXTENT_DATA $offset)"
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 5 $SCRATCH_DEV |\
+		grep -A4 "$file_extent_key"
+}
+
+check_file_extent()
+{
+	local file="$1"
+	local offset="$2"
+	local expected="$3"
+
+	echo "=== file extent at file '$file' offset $offset ===" >> $seqres.full
+	get_file_extent "$file" "$offset" > $tmp.output
+	cat $tmp.output >> $seqres.full
+	grep -q "$expected" $tmp.output ||\
+		echo "file \"$file\" offset $offset doesn't have expected string \"$expected\""
+}
+
+# Unlike file extents whose btrfs specific attributes need to be grabbed from
+# dump-tree, we can check holes by fiemap. In fact recent no-holes feature
+# even makes it unable to grab holes from dump-tree.
+check_hole()
+{
+	local file="$1"
+	local offset="$2"
+	local len="$3"
+
+	output=$($XFS_IO_PROG -c "fiemap $offset $len" "$file" |\
+		 _filter_xfs_io_fiemap | head -n1)
+	if [ -z $output ]; then
+		echo "=== file extent at file '$file' offset $offset is a hole ===" \
+			>> $seqres.full
+	else
+		echo "=== file extent at file '$file' offset $offset is not a hole ==="
+	fi
+}
+
+# Needs 4K sectorsize as the test is crafted using that sectorsize
+_require_btrfs_support_sectorsize 4096
+
+_scratch_mkfs -s 4k >> $seqres.full 2>&1
+
+# Initial data is compressed using lzo
+_scratch_mount -o compress=lzo,compress-force=lzo
+
+# file 'large' has all of its compressed extents at their maximum size
+$XFS_IO_PROG -f -c "pwrite 0 1m" "$SCRATCH_MNT/large" >> $seqres.full
+
+# file 'fragment' has all of its compressed extents adjacent to
+# preallocated/hole ranges, which should not be defragged with regular
+# defrag ioctl, but should still be defragged by "btrfs fi defrag -c"
+$XFS_IO_PROG -f -c "pwrite 0 16k" \
+		-c "pwrite 32k 16k" -c "pwrite 64k 16k" \
+		"$SCRATCH_MNT/fragment" >> $seqres.full
+sync
+# We only do the falloc after the compressed data reached disk.
+# Or the inode could have PREALLOC flag, and prevent the
+# data from being compressed.
+$XFS_IO_PROG -f -c "falloc 16k 16k" "$SCRATCH_MNT/fragment"
+sync
+
+echo "====== Before the defrag ======" >> $seqres.full
+
+# Should be lzo compressed 
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 2"
+
+# Should be lzo compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 2"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+
+# Should be lzo compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 2"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be lzo compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 2"
+
+$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/large" -czstd \
+	"$SCRATCH_MNT/fragment" >> $seqres.full
+# Need to commit the transaction or dump-tree won't grab the new
+# metadata on-disk.
+sync
+
+echo "====== After the defrag ======" >> $seqres.full
+
+# Should be zstd compressed 
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 3"
+
+# Should be zstd compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 3"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+
+# Should be zstd compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 3"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be zstd compressed 
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 3"
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/258.out new file +2 -0
new file mode 100644
index 00000000..9d47016c
@@ -0,0 +1,2 @@
+QA output created by 258
+Silence is golden
Signature · 1 line
2.34.1
#1 lore raw
Quoted 7 lines · Qu Wenruo

Despite the regular file defragging, "btrfs filesystem defrag" provides an option, -c, to convert all data extents (except holes and preallocated ranges) to a new compression algorithm.

The special behavior here is, unlike defrag which is not going to touch extents which are adajacent to preallocated/hole ranges, with -c, all

adajacent -> adjacent

Quoted 23 lines

non-hole/non-preallocated extents should be defragged and converted to the new compression algorithm.

This test case will ensure the old behavior is properly kept.

Currently both old kernels (v5.15 and older) and newer kernel with refactored defrag (v5.16 and newer) can pass the tests.

---
 tests/btrfs/258     | 153 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/258.out |   2 +
 2 files changed, 155 insertions(+)
 create mode 100755 tests/btrfs/258
 create mode 100644 tests/btrfs/258.out
tests/btrfs/258 new file +1 -0
new file mode 100755
index 00000000..a82e5af9
@@ -0,0 +1,153 @@
+#! /bin/bash

tests/btrfs/258 line 11

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 258
+#
+# Make sure "btrfs filesystem defrag" can still convert the compression
+# algorithm of all regular extents.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress

Missing 'prealloc' group, as the test uses fallocate.

Quoted 23 lines
tests/btrfs/258 new file +23 -0
+
+# Override the default cleanup function.
+# _cleanup()
+# {
+#      cd /
+#      rm -r -f $tmp.*
+# }
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch
+
+get_inode_number()
+{
+       local file="$1"
+
+       stat -c "%i" "$file"
+}

tests/btrfs/258 line 44

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+
+get_file_extent()
+{
+       local file="$1"
+       local offset="$2"
+       local ino=$(get_inode_number "$file")
+       local file_extent_key="($ino EXTENT_DATA $offset)"
+
+       $BTRFS_UTIL_PROG inspect-internal dump-tree -t 5 $SCRATCH_DEV |\
+               grep -A4 "$file_extent_key"

Misses a "_require_btrfs_command inspect-internal dump-tree" at the top.

Quoted 7 lines
tests/btrfs/258 new file +7 -0
+}
+
+check_file_extent()
+{
+       local file="$1"
+       local offset="$2"
+       local expected="$3"

tests/btrfs/258 line 61

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+
+       echo "=== file extent at file '$file' offset $offset ===" >> $seqres.full
+       get_file_extent "$file" "$offset" > $tmp.output
+       cat $tmp.output >> $seqres.full
+       grep -q "$expected" $tmp.output ||\
+               echo "file \"$file\" offset $offset doesn't have expected string \"$expected\""
+}
+
+# Unlike file extents whose btrfs specific attributes need to be grabbed from
+# dump-tree, we can check holes by fiemap. In fact recent no-holes feature

no-holes was added in 2013, so I wouldn't call it recent. What you can say is that mkfs enables it by default in new versions of btrfs-progs.

tests/btrfs/258 line 70

Quoted 9 lines
tests/btrfs/258 new file +9 -0
+# even makes it unable to grab holes from dump-tree.
+check_hole()
+{
+       local file="$1"
+       local offset="$2"
+       local len="$3"
+
+       output=$($XFS_IO_PROG -c "fiemap $offset $len" "$file" |\
+                _filter_xfs_io_fiemap | head -n1)

As the test is using ranged fiemap, it should add:

_require_xfs_io_command "fiemap" "ranged"

At the top.

Quoted 5 lines
tests/btrfs/258 new file +5 -0
+       if [ -z $output ]; then
+               echo "=== file extent at file '$file' offset $offset is a hole ===" \
+                       >> $seqres.full
+       else
+               echo "=== file extent at file '$file' offset $offset is not a hole ==="

tests/btrfs/258 line 85

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+       fi
+}
+
+# Needs 4K sectorsize as the test is crafted using that sectorsize
+_require_btrfs_support_sectorsize 4096
+
+_scratch_mkfs -s 4k >> $seqres.full 2>&1
+
+# Initial data is compressed using lzo
+_scratch_mount -o compress=lzo,compress-force=lzo

Why both compress options? Why isn't only compress-force enough? It's redundant to use compress when using compress-force, or did I miss something?

Quoted 5 lines
tests/btrfs/258 new file +5 -0
+
+# file 'large' has all of its compressed extents at their maximum size
+$XFS_IO_PROG -f -c "pwrite 0 1m" "$SCRATCH_MNT/large" >> $seqres.full
+
+# file 'fragment' has all of its compressed extents adjacent to

tests/btrfs/258 line 100

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# preallocated/hole ranges, which should not be defragged with regular
+# defrag ioctl, but should still be defragged by "btrfs fi defrag -c"
+$XFS_IO_PROG -f -c "pwrite 0 16k" \
+               -c "pwrite 32k 16k" -c "pwrite 64k 16k" \
+               "$SCRATCH_MNT/fragment" >> $seqres.full
+sync
+# We only do the falloc after the compressed data reached disk.
+# Or the inode could have PREALLOC flag, and prevent the
+# data from being compressed.
+$XFS_IO_PROG -f -c "falloc 16k 16k" "$SCRATCH_MNT/fragment"

-f should go away, the file already exists.

Also, missing a:

_require_xfs_io_command "falloc"

at the top.

Quoted 13 lines
tests/btrfs/258 new file +13 -0
+sync
+
+echo "====== Before the defrag ======" >> $seqres.full
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 2"
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 2"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+

tests/btrfs/258 line 123

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 2"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 2"
+
+$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/large" -czstd \

Tests should use the full name of btrfs commands and no abbreviations. So it should use "defragment" and not "defrag", even if it seems very unlikely the shorter name
will ever be unsupported.

Thanks.

Quoted 41 lines
tests/btrfs/258 new file +30 -0
+       "$SCRATCH_MNT/fragment" >> $seqres.full
+# Need to commit the transaction or dump-tree won't grab the new
+# metadata on-disk.
+sync
+
+echo "====== After the defrag ======" >> $seqres.full
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 3"
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 3"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 3"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 3"
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/258.out new file +2 -1
new file mode 100644
index 00000000..9d47016c
@@ -0,0 +1,2 @@
+QA output created by 258
+Silence is golden
--

2.34.1

#2 lore raw
Quoted 132 lines · Filipe Manana
Quoted 7 lines · Qu Wenruo

Despite the regular file defragging, "btrfs filesystem defrag" provides an option, -c, to convert all data extents (except holes and preallocated ranges) to a new compression algorithm.

The special behavior here is, unlike defrag which is not going to touch extents which are adajacent to preallocated/hole ranges, with -c, all

adajacent -> adjacent

Quoted 23 lines

non-hole/non-preallocated extents should be defragged and converted to the new compression algorithm.

This test case will ensure the old behavior is properly kept.

Currently both old kernels (v5.15 and older) and newer kernel with refactored defrag (v5.16 and newer) can pass the tests.

---
  tests/btrfs/258     | 153 ++++++++++++++++++++++++++++++++++++++++++++
  tests/btrfs/258.out |   2 +
  2 files changed, 155 insertions(+)

create mode 100755 tests/btrfs/258
create mode 100644 tests/btrfs/258.out

tests/btrfs/258 new file +1 -0
new file mode 100755
index 00000000..a82e5af9
@@ -0,0 +1,153 @@
+#! /bin/bash

tests/btrfs/258 line 11

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 258
+#
+# Make sure "btrfs filesystem defrag" can still convert the compression
+# algorithm of all regular extents.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag compress

Missing 'prealloc' group, as the test uses fallocate.

Quoted 23 lines
tests/btrfs/258 new file +23 -0
+
+# Override the default cleanup function.
+# _cleanup()
+# {
+#      cd /
+#      rm -r -f $tmp.*
+# }
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch
+
+get_inode_number()
+{
+       local file="$1"
+
+       stat -c "%i" "$file"
+}

tests/btrfs/258 line 44

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+
+get_file_extent()
+{
+       local file="$1"
+       local offset="$2"
+       local ino=$(get_inode_number "$file")
+       local file_extent_key="($ino EXTENT_DATA $offset)"
+
+       $BTRFS_UTIL_PROG inspect-internal dump-tree -t 5 $SCRATCH_DEV |\
+               grep -A4 "$file_extent_key"

Misses a "_require_btrfs_command inspect-internal dump-tree" at the top.

Quoted 7 lines
tests/btrfs/258 new file +7 -0
+}
+
+check_file_extent()
+{
+       local file="$1"
+       local offset="$2"
+       local expected="$3"

tests/btrfs/258 line 61

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+
+       echo "=== file extent at file '$file' offset $offset ===" >> $seqres.full
+       get_file_extent "$file" "$offset" > $tmp.output
+       cat $tmp.output >> $seqres.full
+       grep -q "$expected" $tmp.output ||\
+               echo "file \"$file\" offset $offset doesn't have expected string \"$expected\""
+}
+
+# Unlike file extents whose btrfs specific attributes need to be grabbed from
+# dump-tree, we can check holes by fiemap. In fact recent no-holes feature

no-holes was added in 2013, so I wouldn't call it recent. What you can say is that mkfs enables it by default in new versions of btrfs-progs.

tests/btrfs/258 line 70

Quoted 9 lines
tests/btrfs/258 new file +9 -0
+# even makes it unable to grab holes from dump-tree.
+check_hole()
+{
+       local file="$1"
+       local offset="$2"
+       local len="$3"
+
+       output=$($XFS_IO_PROG -c "fiemap $offset $len" "$file" |\
+                _filter_xfs_io_fiemap | head -n1)

As the test is using ranged fiemap, it should add:

_require_xfs_io_command "fiemap" "ranged"

At the top.

Quoted 11 lines
tests/btrfs/258 new file +11 -0
+       if [ -z $output ]; then
+               echo "=== file extent at file '$file' offset $offset is a hole ===" \
+                       >> $seqres.full
+       else
+               echo "=== file extent at file '$file' offset $offset is not a hole ==="
+       fi
+}
+
+# Needs 4K sectorsize as the test is crafted using that sectorsize
+_require_btrfs_support_sectorsize 4096
+
Quoted 8 lines

tests/btrfs/258 line 85

Quoted 4 lines
tests/btrfs/258 new file +4 -0
+_scratch_mkfs -s 4k >> $seqres.full 2>&1
+
+# Initial data is compressed using lzo
+_scratch_mount -o compress=lzo,compress-force=lzo

Why both compress options? Why isn't only compress-force enough? It's redundant to use compress when using compress-force, or did I miss something?

My bad, the thing here is from my initial runs, the fragment file never go its extents written as compressed.

It turns out that my initial runs are always doing pwrite->falloc->pwrite->pwrite, causing the PREALLOC inode flag and tries go NOCOW path first, then fallback to COW without compression.

The root reason is spotted later, but forget to change the mount option which is tried to solve the problem but won't really work.

All the comments will be addressed in the next update.

Thanks for the review,
Qu

Quoted 97 lines
Quoted 6 lines
tests/btrfs/258 new file +5 -0
+
+# file 'large' has all of its compressed extents at their maximum size
+$XFS_IO_PROG -f -c "pwrite 0 1m" "$SCRATCH_MNT/large" >> $seqres.full
+
+# file 'fragment' has all of its compressed extents adjacent to

tests/btrfs/258 line 100

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# preallocated/hole ranges, which should not be defragged with regular
+# defrag ioctl, but should still be defragged by "btrfs fi defrag -c"
+$XFS_IO_PROG -f -c "pwrite 0 16k" \
+               -c "pwrite 32k 16k" -c "pwrite 64k 16k" \
+               "$SCRATCH_MNT/fragment" >> $seqres.full
+sync
+# We only do the falloc after the compressed data reached disk.
+# Or the inode could have PREALLOC flag, and prevent the
+# data from being compressed.
+$XFS_IO_PROG -f -c "falloc 16k 16k" "$SCRATCH_MNT/fragment"

-f should go away, the file already exists.

Also, missing a:

_require_xfs_io_command "falloc"

at the top.

Quoted 13 lines
tests/btrfs/258 new file +13 -0
+sync
+
+echo "====== Before the defrag ======" >> $seqres.full
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 2"
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 2"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+

tests/btrfs/258 line 123

Quoted 10 lines
tests/btrfs/258 new file +10 -0
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 2"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be lzo compressed
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 2"
+
+$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/large" -czstd \

Tests should use the full name of btrfs commands and no abbreviations. So it should use "defragment" and not "defrag", even if it seems very unlikely the shorter name
will ever be unsupported.

Thanks.

Quoted 41 lines
tests/btrfs/258 new file +30 -0
+       "$SCRATCH_MNT/fragment" >> $seqres.full
+# Need to commit the transaction or dump-tree won't grab the new
+# metadata on-disk.
+sync
+
+echo "====== After the defrag ======" >> $seqres.full
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/large" 0 "compression 3"
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 0 "compression 3"
+
+# Should be preallocated
+check_file_extent "$SCRATCH_MNT/fragment" 16384 "type 2"
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 32768 "compression 3"
+
+# Should be hole
+check_hole "$SCRATCH_MNT/fragment" 49152 16384
+
+# Should be zstd compressed
+check_file_extent "$SCRATCH_MNT/fragment" 65536 "compression 3"
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/258.out new file +2 -1
new file mode 100644
index 00000000..9d47016c
@@ -0,0 +1,2 @@
+QA output created by 258
+Silence is golden
--

2.34.1