Skip to contentSkip to conversation

f2fs/006: add testcase to check out-of-space case

PATCH Superseded Chao Yu 2 participants 8 messages Oct 15, 2024 – Oct 23, 2024

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Chao Yu

author
#0 lore raw

PATCH 1/2 f2fs/006: add testcase to check out-of-space case

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function, resulting in panic.

Cc Jaegeuk Kim <[email protected]>
---
 tests/f2fs/006     | 52 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/006.out |  6 ++++++
 2 files changed, 58 insertions(+)
 create mode 100755 tests/f2fs/006
 create mode 100644 tests/f2fs/006.out
tests/f2fs/006 new file +52 -0
new file mode 100755
index 00000000..b359ef8f
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function, resulting in
+# panic.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	rm -f $img
+	_scratch_unmount >> $seqres.full
+	cd /
+	rm -r -f $tmp.*
+}
+
+_require_scratch
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+img=$SCRATCH_MNT/f2fs.img
+mnt=$SCRATCH_MNT/f2fs.mnt
+testfile=$mnt/testfile
+
+mkdir $mnt
+dd if=/dev/zero of=$img bs=1M count=100 2>/dev/null
+$MKFS_F2FS_PROG -f $img >/dev/null 2>&1
+sync
+
+# use mode=lfs to let f2fs always triggers OPU
+mount -t $FSTYP -o loop,mode=lfs,checkpoint=disable:10%,noinline_dentry $img $mnt
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+
+mount -o remount,checkpoint=enable $mnt
+umount $mnt
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..a2c7ba48
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing '/mnt/scratch_f2fs/f2fs.mnt/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
author
#1 lore raw

PATCH 2/2 f2fs/007: add testcase to check consistency of compressed inode metadata

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>
---
 tests/f2fs/007     | 116 +++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/007.out |   4 ++
 2 files changed, 120 insertions(+)
 create mode 100755 tests/f2fs/007
 create mode 100644 tests/f2fs/007.out
tests/f2fs/007 new file +116 -0
new file mode 100755
index 00000000..274be806
@@ -0,0 +1,116 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	cd /
+	rm -r -f tmp.*
+}
+
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile30
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile60
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile90
+buffer_compress_percentage=90
+EOF
+
+_require_fio $fio_config
+_require_scratch
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io compress $testfile >> $seqres.full
+	cblocks=`f2fs_io get_cblocks $testfile`
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after compress"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io release_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 2. check after releasement
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after release_cblocks"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io reserve_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 3. check after rservation
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after reserve_cblocks"
+	fi
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io decompress $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 4. check after decompression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after decompress"
+	fi
+done
+
+status=0
+exit
tests/f2fs/007.out new file +4 -0
new file mode 100644
index 00000000..2ea71c18
@@ -0,0 +1,4 @@
+QA output created by 007
+compression ratio is: 64 / 256
+compression ratio is: 128 / 256
+compression ratio is: 192 / 256
Signature · 1 line
2.40.1
#4 lore raw
In reply to Chao Yu
Quoted 53 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function, resulting in panic.

Cc Jaegeuk Kim <[email protected]>
---
 tests/f2fs/006     | 52 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/006.out |  6 ++++++
 2 files changed, 58 insertions(+)
 create mode 100755 tests/f2fs/006
 create mode 100644 tests/f2fs/006.out
tests/f2fs/006 new file +30 -0
new file mode 100755
index 00000000..b359ef8f
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function, resulting in
+# panic.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	rm -f $img
+	_scratch_unmount >> $seqres.full
+	cd /
+	rm -r -f $tmp.*
+}
+
+_require_scratch
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+img=$SCRATCH_MNT/f2fs.img

tests/f2fs/006 line 40

Quoted 10 lines
tests/f2fs/006 new file +10 -0
+mnt=$SCRATCH_MNT/f2fs.mnt
+testfile=$mnt/testfile
+
+mkdir $mnt
+dd if=/dev/zero of=$img bs=1M count=100 2>/dev/null
+$MKFS_F2FS_PROG -f $img >/dev/null 2>&1
+sync
+
+# use mode=lfs to let f2fs always triggers OPU
+mount -t $FSTYP -o loop,mode=lfs,checkpoint=disable:10%,noinline_dentry $img $mnt

Hi Chao,

Is the loop device necessary? What if use SCRATCH_DEV and SCRATCH_MNT directly?

Thanks,
Zorro

Quoted 27 lines
tests/f2fs/006 new file +12 -0
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+
+mount -o remount,checkpoint=enable $mnt
+umount $mnt
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..a2c7ba48
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing '/mnt/scratch_f2fs/f2fs.mnt/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
author
#5 lore raw
Quoted 67 lines · Zorro Lang
Quoted 53 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function, resulting in panic.

Cc Jaegeuk Kim <[email protected]>
---
  tests/f2fs/006     | 52 ++++++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/006.out |  6 ++++++
  2 files changed, 58 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +30 -0
new file mode 100755
index 00000000..b359ef8f
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function, resulting in
+# panic.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	rm -f $img
+	_scratch_unmount >> $seqres.full
+	cd /
+	rm -r -f $tmp.*
+}
+
+_require_scratch
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+img=$SCRATCH_MNT/f2fs.img

tests/f2fs/006 line 40

Quoted 10 lines
tests/f2fs/006 new file +10 -0
+mnt=$SCRATCH_MNT/f2fs.mnt
+testfile=$mnt/testfile
+
+mkdir $mnt
+dd if=/dev/zero of=$img bs=1M count=100 2>/dev/null
+$MKFS_F2FS_PROG -f $img >/dev/null 2>&1
+sync
+
+# use mode=lfs to let f2fs always triggers OPU
+mount -t $FSTYP -o loop,mode=lfs,checkpoint=disable:10%,noinline_dentry $img $mnt

Hi Chao,

Quoted 1 line

Is the loop device necessary? What if use SCRATCH_DEV and SCRATCH_MNT directly?

Hi Zorro,

It uses loop device to limit image size, so that we can speed up padding steps of the test since it depends on ENOSPC state.

Or maybe we can mkfs.f2fs $SCRATCH_DEV w/ specified sector size? Any suggestion?

Thanks,

Quoted 32 lines

Thanks,
Zorro

Quoted 27 lines
tests/f2fs/006 new file +12 -0
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+
+mount -o remount,checkpoint=enable $mnt
+umount $mnt
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..a2c7ba48
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing '/mnt/scratch_f2fs/f2fs.mnt/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
#2 lore raw

Re: [PATCH 2/2] f2fs/007: add testcase to check consistency of compressed inode metadata

In reply to Chao Yu
Quoted 20 lines · Chao Yu

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>
---
 tests/f2fs/007     | 116 +++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/007.out |   4 ++
 2 files changed, 120 insertions(+)
 create mode 100755 tests/f2fs/007
 create mode 100644 tests/f2fs/007.out
tests/f2fs/007 new file +0 -0
new file mode 100755
index 00000000..274be806

tests/f2fs/007 line 7

Quoted 7 lines
tests/f2fs/007 new file +7 -0
@@ -0,0 +1,116 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata

Hi Chao,

Thanks for the new f2fs test cases.

If there's a known fix for this regression, please mark it by:

_fixed_by_kernel_commit .....

tests/f2fs/007 line 12

Quoted 5 lines
tests/f2fs/007 new file +5 -0
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick

^^^^
rw compress ?

tests/f2fs/007 line 18

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+
+_cleanup()
+{
+	cd /
+	rm -r -f tmp.*
+}

This's same with default _cleanup, so it can be removed.

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+

tests/f2fs/007 line 32

Quoted 10 lines
tests/f2fs/007 new file +10 -0
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile30
            ^^^^^^^^^^

$SCRATCH_MNT or ${testfile_prefix}30 ?

tests/f2fs/007 line 43

Quoted 11 lines
tests/f2fs/007 new file +11 -0
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile60

Same as above.

tests/f2fs/007 line 54

Quoted 11 lines
tests/f2fs/007 new file +11 -0
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile90

Same as above

tests/f2fs/007 line 59

Quoted 5 lines
tests/f2fs/007 new file +5 -0
+buffer_compress_percentage=90
+EOF
+
+_require_fio $fio_config
+_require_scratch

I'd like to call _require_scratch at the beginning, especially you use SCRATCH_* things in $fio_config. You can refer to generic/095.

tests/f2fs/007 line 60

Quoted 1 line
tests/f2fs/007 new file +1 -0
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full

Can you make sure this mkfs never fail? Maybe:

    || _fail "...."

tests/f2fs/007 line 72

Quoted 12 lines
tests/f2fs/007 new file +12 -0
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io compress $testfile >> $seqres.full
        ^^^^^^^

$F2FS_IO_PROG

tests/f2fs/007 line 73

Quoted 1 line
tests/f2fs/007 new file +1 -0
+	cblocks=`f2fs_io get_cblocks $testfile`
                 ^^^^^^^

$F2FS_IO_PROG

tests/f2fs/007 line 79

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs ?

If f2fs needs some custom operations, please change the _check_scratch_fs or _check_generic_filesystem.

tests/f2fs/007 line 85

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after compress"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io release_cblocks $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 89

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 2. check after releasement
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

tests/f2fs/007 line 95

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after release_cblocks"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io reserve_cblocks $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 99

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 3. check after rservation
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

tests/f2fs/007 line 105

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after reserve_cblocks"
+	fi
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io decompress $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 109

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 4. check after decompression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

Thanks,
Zorro

Quoted 20 lines
tests/f2fs/007 new file +7 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after decompress"
+	fi
+done
+
+status=0
+exit
tests/f2fs/007.out new file +4 -0
new file mode 100644
index 00000000..2ea71c18
@@ -0,0 +1,4 @@
+QA output created by 007
+compression ratio is: 64 / 256
+compression ratio is: 128 / 256
+compression ratio is: 192 / 256
Signature · 1 line
2.40.1
#6 lore raw
In reply to Chao Yu
Quoted 70 lines · Chao Yu
Quoted 67 lines · Zorro Lang
Quoted 53 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function, resulting in panic.

Cc Jaegeuk Kim <[email protected]>
---
  tests/f2fs/006     | 52 ++++++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/006.out |  6 ++++++
  2 files changed, 58 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +30 -0
new file mode 100755
index 00000000..b359ef8f
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function, resulting in
+# panic.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	rm -f $img
+	_scratch_unmount >> $seqres.full
+	cd /
+	rm -r -f $tmp.*
+}
+
+_require_scratch
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+img=$SCRATCH_MNT/f2fs.img

tests/f2fs/006 line 40

Quoted 10 lines
tests/f2fs/006 new file +10 -0
+mnt=$SCRATCH_MNT/f2fs.mnt
+testfile=$mnt/testfile
+
+mkdir $mnt
+dd if=/dev/zero of=$img bs=1M count=100 2>/dev/null
+$MKFS_F2FS_PROG -f $img >/dev/null 2>&1
+sync
+
+# use mode=lfs to let f2fs always triggers OPU
+mount -t $FSTYP -o loop,mode=lfs,checkpoint=disable:10%,noinline_dentry $img $mnt

Hi Chao,

Quoted 1 line

Is the loop device necessary? What if use SCRATCH_DEV and SCRATCH_MNT directly?

Quoted 7 lines

Hi Zorro,

It uses loop device to limit image size, so that we can speed up padding steps of the test since it depends on ENOSPC state.

Or maybe we can mkfs.f2fs $SCRATCH_DEV w/ specified sector size? Any suggestion?

Can scratch_mkfs_sized() help? I saw it supports f2fs.

Thanks,
Zorro

Quoted 36 lines

Thanks,

Quoted 32 lines

Thanks,
Zorro

Quoted 27 lines
tests/f2fs/006 new file +12 -0
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+
+mount -o remount,checkpoint=enable $mnt
+umount $mnt
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..a2c7ba48
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing '/mnt/scratch_f2fs/f2fs.mnt/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
author
#7 lore raw
Quoted 73 lines · Zorro Lang
Quoted 72 lines · Chao Yu
Quoted 67 lines · Zorro Lang
Quoted 53 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function, resulting in panic.

Cc Jaegeuk Kim <[email protected]>
---
   tests/f2fs/006     | 52 ++++++++++++++++++++++++++++++++++++++++++++++
   tests/f2fs/006.out |  6 ++++++
   2 files changed, 58 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +30 -0
new file mode 100755
index 00000000..b359ef8f
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function, resulting in
+# panic.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_cleanup()
+{
+	rm -f $img
+	_scratch_unmount >> $seqres.full
+	cd /
+	rm -r -f $tmp.*
+}
+
+_require_scratch
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+img=$SCRATCH_MNT/f2fs.img

tests/f2fs/006 line 40

Quoted 10 lines
tests/f2fs/006 new file +10 -0
+mnt=$SCRATCH_MNT/f2fs.mnt
+testfile=$mnt/testfile
+
+mkdir $mnt
+dd if=/dev/zero of=$img bs=1M count=100 2>/dev/null
+$MKFS_F2FS_PROG -f $img >/dev/null 2>&1
+sync
+
+# use mode=lfs to let f2fs always triggers OPU
+mount -t $FSTYP -o loop,mode=lfs,checkpoint=disable:10%,noinline_dentry $img $mnt

Hi Chao,

Quoted 1 line

Is the loop device necessary? What if use SCRATCH_DEV and SCRATCH_MNT directly?

Hi Zorro,

Quoted 7 lines
Quoted 5 lines

It uses loop device to limit image size, so that we can speed up padding steps of the test since it depends on ENOSPC state.

Or maybe we can mkfs.f2fs $SCRATCH_DEV w/ specified sector size? Any suggestion?

Can scratch_mkfs_sized() help? I saw it supports f2fs.

Yes, let me update this patch, thanks.

Thanks,

Quoted 41 lines

Thanks,
Zorro

Quoted 36 lines

Thanks,

Quoted 32 lines

Thanks,
Zorro

Quoted 27 lines
tests/f2fs/006 new file +12 -0
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+
+mount -o remount,checkpoint=enable $mnt
+umount $mnt
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..a2c7ba48
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing '/mnt/scratch_f2fs/f2fs.mnt/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
author
#3 lore raw

Re: [PATCH 2/2] f2fs/007: add testcase to check consistency of compressed inode metadata

In reply to Zorro Lang
Quoted 31 lines · Zorro Lang
Quoted 20 lines · Chao Yu

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>
---
  tests/f2fs/007     | 116 +++++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/007.out |   4 ++
  2 files changed, 120 insertions(+)

create mode 100755 tests/f2fs/007
create mode 100644 tests/f2fs/007.out

tests/f2fs/007 new file +0 -0
new file mode 100755
index 00000000..274be806

tests/f2fs/007 line 7

Quoted 7 lines
tests/f2fs/007 new file +7 -0
@@ -0,0 +1,116 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata

Hi Chao,

Quoted 5 lines

Thanks for the new f2fs test cases.

If there's a known fix for this regression, please mark it by:

_fixed_by_kernel_commit .....

Zorro,

Yes, but not been merged yet, so let me add as below and replace commit_id after upstreaming the fix.

_fixed_by_kernel_commit xxxxxxxxxxxx \
"f2fs: compress: fix inconsistent update of i_blocks in release_compress_blocks and reserve_compress_blocks"

Thanks for all comments below, will fix all.

Thanks,

Quoted 181 lines

tests/f2fs/007 line 12

Quoted 6 lines
tests/f2fs/007 new file +5 -0
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick

^^^^
rw compress ?

tests/f2fs/007 line 18

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+
+_cleanup()
+{
+	cd /
+	rm -r -f tmp.*
+}

This's same with default _cleanup, so it can be removed.

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+

tests/f2fs/007 line 32

Quoted 10 lines
tests/f2fs/007 new file +10 -0
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile30
             ^^^^^^^^^^

$SCRATCH_MNT or ${testfile_prefix}30 ?

tests/f2fs/007 line 43

Quoted 11 lines
tests/f2fs/007 new file +11 -0
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile60

Same as above.

tests/f2fs/007 line 54

Quoted 11 lines
tests/f2fs/007 new file +11 -0
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=/mnt/scratch_f2fs/testfile90

Same as above

tests/f2fs/007 line 59

Quoted 5 lines
tests/f2fs/007 new file +5 -0
+buffer_compress_percentage=90
+EOF
+
+_require_fio $fio_config
+_require_scratch

I'd like to call _require_scratch at the beginning, especially you use SCRATCH_* things in $fio_config. You can refer to generic/095.

tests/f2fs/007 line 60

Quoted 1 line
tests/f2fs/007 new file +1 -0
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full

Can you make sure this mkfs never fail? Maybe:

     || _fail "...."

tests/f2fs/007 line 72

Quoted 12 lines
tests/f2fs/007 new file +12 -0
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io compress $testfile >> $seqres.full
         ^^^^^^^

$F2FS_IO_PROG

tests/f2fs/007 line 73

Quoted 1 line
tests/f2fs/007 new file +1 -0
+	cblocks=`f2fs_io get_cblocks $testfile`
                  ^^^^^^^

$F2FS_IO_PROG

tests/f2fs/007 line 79

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs ?

If f2fs needs some custom operations, please change the _check_scratch_fs or _check_generic_filesystem.

tests/f2fs/007 line 85

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after compress"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io release_cblocks $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 89

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 2. check after releasement
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

tests/f2fs/007 line 95

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after release_cblocks"
+	fi
+
+	_scratch_mount >> $seqres.full
+	f2fs_io reserve_cblocks $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 99

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 3. check after rservation
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

tests/f2fs/007 line 105

Quoted 6 lines
tests/f2fs/007 new file +6 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after reserve_cblocks"
+	fi
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	f2fs_io decompress $testfile >> $seqres.full

$F2FS_IO_PROG

tests/f2fs/007 line 109

Quoted 4 lines
tests/f2fs/007 new file +4 -0
+	_scratch_unmount
+
+	# 4. check after decompression
+	fsck -t $FSTYP -f $SCRATCH_DEV >> $seqres.full

_check_scratch_fs

Thanks,
Zorro

Quoted 20 lines
tests/f2fs/007 new file +7 -0
+	if [ $? -ne 0 ]; then
+		_fail "filesystem becomes corrupted after decompress"
+	fi
+done
+
+status=0
+exit
tests/f2fs/007.out new file +4 -0
new file mode 100644
index 00000000..2ea71c18
@@ -0,0 +1,4 @@
+QA output created by 007
+compression ratio is: 64 / 256
+compression ratio is: 128 / 256
+compression ratio is: 192 / 256
Signature · 1 line
2.40.1