Skip to contentSkip to conversation

f2fs/006: add testcase to check out-of-space case

PATCH Superseded Chao Yu 2 participants 8 messages Oct 28, 2024 – Oct 29, 2024

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Chao Yu

author
#0 lore raw

PATCH v3 1/2 f2fs/006: add testcase to check out-of-space case

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function. If kernel config CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise, dd may encounter I/O error.

Cc Jaegeuk Kim <[email protected]>

---
v3:
- explain more about behavior on kernel w/ different config
- use _filter_scratch to filter $SCRATCH_MNT

 tests/f2fs/006     | 42 ++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/006.out |  6 ++++++
 2 files changed, 48 insertions(+)
 create mode 100755 tests/f2fs/006
 create mode 100644 tests/f2fs/006.out
tests/f2fs/006 new file +42 -0
new file mode 100755
index 00000000..f9560b2b
@@ -0,0 +1,42 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function. If kernel config
+# CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise,
+# dd may encounter I/O error.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"f2fs: fix to account dirty data in __get_secs_required()"
+
+# Import common functions.
+. ./common/filter
+
+_require_scratch
+_scratch_mkfs_sized $((1024*1024*100)) >> $seqres.full
+
+# use mode=lfs to let f2fs always triggers OPU
+_scratch_mount -o mode=lfs,checkpoint=disable:10%,noinline_dentry >> $seqres.full
+
+testfile=$SCRATCH_MNT/testfile
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 | _filter_scratch
+
+_scratch_remount checkpoint=enable
+
+status=0
+exit
tests/f2fs/006.out new file +6 -0
new file mode 100644
index 00000000..2dc9efda
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing 'SCRATCH_MNT/testfile': No space left on device
+3+0 records in
+2+0 records out
Signature · 1 line
2.40.1
author
#1 lore raw

PATCH v3 2/2 f2fs/007: add testcase to check consistency of compressed inode metadata

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>

---
v3:
- move _require_scratch before use of $SCRATCH_MNT
- output message before _check_scratch_fs

 tests/f2fs/007     | 108 +++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/007.out |  16 +++++++
 2 files changed, 124 insertions(+)
 create mode 100755 tests/f2fs/007
 create mode 100644 tests/f2fs/007.out
tests/f2fs/007 new file +108 -0
new file mode 100755
index 00000000..805d88fa
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick rw compress
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+        "f2fs: compress: fix inconsistent update of i_blocks in release_compress_blocks and reserve_compress_blocks"
+
+_require_scratch
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}30
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}60
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}90
+buffer_compress_percentage=90
+EOF
+
+_require_fio $fio_config
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full || _fail "mkfs failed"
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+# force to repaire if filesystem is corrupted
+export FSCK_OPTIONS="-f $FSCK_OPTIONS"
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG compress $testfile >> $seqres.full
+	cblocks=`$F2FS_IO_PROG get_cblocks $testfile`
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	echo "check fs after compress"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG release_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 2. check after releasement
+	echo "check fs after release_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG reserve_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 3. check after rservation
+	echo "check fs after reserve_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG decompress $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 4. check after decompression
+	echo "check fs after decompress"
+	_check_scratch_fs
+done
+
+status=0
+exit
tests/f2fs/007.out new file +16 -0
new file mode 100644
index 00000000..a4b76300
@@ -0,0 +1,16 @@
+QA output created by 007
+compression ratio is: 64 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 128 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 192 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
Signature · 1 line
2.40.1
Reviewed-by on v3 2/2
#2 lore raw

Re: [PATCH v3 2/2] f2fs/007: add testcase to check consistency of compressed inode metadata

Quoted 78 lines · Chao Yu

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>

---
v3:
- move _require_scratch before use of $SCRATCH_MNT
- output message before _check_scratch_fs

 tests/f2fs/007     | 108 +++++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/007.out |  16 +++++++
 2 files changed, 124 insertions(+)
 create mode 100755 tests/f2fs/007
 create mode 100644 tests/f2fs/007.out
tests/f2fs/007 new file +55 -0
new file mode 100755
index 00000000..805d88fa
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick rw compress
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+        "f2fs: compress: fix inconsistent update of i_blocks in release_compress_blocks and reserve_compress_blocks"
+
+_require_scratch
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}30
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}60
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}90
+buffer_compress_percentage=90
+EOF
+

tests/f2fs/007 line 65

Quoted 10 lines
tests/f2fs/007 new file +10 -0
+_require_fio $fio_config
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full || _fail "mkfs failed"
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+# force to repaire if filesystem is corrupted

repair

tests/f2fs/007 line 66

Quoted 1 line
tests/f2fs/007 new file +1 -0
+export FSCK_OPTIONS="-f $FSCK_OPTIONS"

JFYI, if f2f2 always hope to force to repair, you can think about changing the _fsck_opts() in common/config.

Others look good to me, I'll help to fix above typo when I merge it. Thanks for you patch.

Quoted 67 lines
tests/f2fs/007 new file +42 -0
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG compress $testfile >> $seqres.full
+	cblocks=`$F2FS_IO_PROG get_cblocks $testfile`
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	echo "check fs after compress"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG release_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 2. check after releasement
+	echo "check fs after release_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG reserve_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 3. check after rservation
+	echo "check fs after reserve_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG decompress $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 4. check after decompression
+	echo "check fs after decompress"
+	_check_scratch_fs
+done
+
+status=0
+exit
tests/f2fs/007.out new file +16 -0
new file mode 100644
index 00000000..a4b76300
@@ -0,0 +1,16 @@
+QA output created by 007
+compression ratio is: 64 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 128 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 192 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
Signature · 1 line
2.40.1
author
#3 lore raw

Re: [PATCH v3 2/2] f2fs/007: add testcase to check consistency of compressed inode metadata

Quoted 92 lines · Zorro Lang
Quoted 78 lines · Chao Yu

metadata of compressed inode should always be consistent after file compression, reservation, releasement and decompression, let's add a testcase to check it.

Cc Jaegeuk Kim <[email protected]>
Cc Qi Han <[email protected]>

---
v3:
- move _require_scratch before use of $SCRATCH_MNT
- output message before _check_scratch_fs

  tests/f2fs/007     | 108 +++++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/007.out |  16 +++++++
  2 files changed, 124 insertions(+)

create mode 100755 tests/f2fs/007
create mode 100644 tests/f2fs/007.out

tests/f2fs/007 new file +55 -0
new file mode 100755
index 00000000..805d88fa
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/007
+#
+# This is a regression test to check whether compressed metadata
+# can become inconsistent after file compression, reservation
+# releasement, and decompression.
+#
+. ./common/preamble
+_begin_fstest auto quick rw compress
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+        "f2fs: compress: fix inconsistent update of i_blocks in release_compress_blocks and reserve_compress_blocks"
+
+_require_scratch
+testfile_prefix=$SCRATCH_MNT/testfile
+fio_config=$tmp.fio
+
+cat >$fio_config <<EOF
+[write_compressed_data_30]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}30
+buffer_compress_percentage=30
+
+[write_compressed_data_60]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}60
+buffer_compress_percentage=60
+
+[write_compressed_data_90]
+name=mytest
+ioengine=psync
+rw=write
+direct=0
+bs=1M
+filesize=1M
+numjobs=1
+filename=${testfile_prefix}90
+buffer_compress_percentage=90
+EOF
+

tests/f2fs/007 line 65

Quoted 10 lines
tests/f2fs/007 new file +10 -0
+_require_fio $fio_config
+_scratch_mkfs "-f -O extra_attr,compression" >> $seqres.full || _fail "mkfs failed"
+_scratch_mount "-o compress_mode=user,compress_extension=*" >> $seqres.full
+
+echo -e "Run fio to initialize file w/ specified compress ratio" >> $seqres.full
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config >> $seqres.full
+_scratch_unmount
+
+# force to repaire if filesystem is corrupted

repair

Quoted 4 lines

tests/f2fs/007 line 66

Quoted 1 line
tests/f2fs/007 new file +1 -0
+export FSCK_OPTIONS="-f $FSCK_OPTIONS"

JFYI, if f2f2 always hope to force to repair, you can think about changing the _fsck_opts() in common/config.

Got it, thanks for your reminder.

Quoted 2 lines

Others look good to me, I'll help to fix above typo when I merge it. Thanks for you patch.

Thank you!

Thanks,

Quoted 71 lines
Quoted 67 lines
tests/f2fs/007 new file +42 -0
+
+for i in 30 60 90; do
+	testfile=$testfile_prefix$i
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG compress $testfile >> $seqres.full
+	cblocks=`$F2FS_IO_PROG get_cblocks $testfile`
+	echo "compression ratio is: "$cblocks" / 256"
+
+	_scratch_unmount
+
+	# 1. check after compression
+	echo "check fs after compress"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG release_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 2. check after releasement
+	echo "check fs after release_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount >> $seqres.full
+	$F2FS_IO_PROG reserve_cblocks $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 3. check after rservation
+	echo "check fs after reserve_cblocks"
+	_check_scratch_fs
+
+	_scratch_mount "-o compress_mode=user" >> $seqres.full
+	$F2FS_IO_PROG decompress $testfile >> $seqres.full
+	_scratch_unmount
+
+	# 4. check after decompression
+	echo "check fs after decompress"
+	_check_scratch_fs
+done
+
+status=0
+exit
tests/f2fs/007.out new file +16 -0
new file mode 100644
index 00000000..a4b76300
@@ -0,0 +1,16 @@
+QA output created by 007
+compression ratio is: 64 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 128 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
+compression ratio is: 192 / 256
+check fs after compress
+check fs after release_cblocks
+check fs after reserve_cblocks
+check fs after decompress
Signature · 1 line
2.40.1
#4 lore raw
In reply to Chao Yu
Quoted 75 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function. If kernel config CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise, dd may encounter I/O error.

Cc Jaegeuk Kim <[email protected]>

---
v3:
- explain more about behavior on kernel w/ different config
- use _filter_scratch to filter $SCRATCH_MNT

 tests/f2fs/006     | 42 ++++++++++++++++++++++++++++++++++++++++++
 tests/f2fs/006.out |  6 ++++++
 2 files changed, 48 insertions(+)
 create mode 100755 tests/f2fs/006
 create mode 100644 tests/f2fs/006.out
tests/f2fs/006 new file +42 -0
new file mode 100755
index 00000000..f9560b2b
@@ -0,0 +1,42 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function. If kernel config
+# CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise,
+# dd may encounter I/O error.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"f2fs: fix to account dirty data in __get_secs_required()"
+
+# Import common functions.
+. ./common/filter
+
+_require_scratch
+_scratch_mkfs_sized $((1024*1024*100)) >> $seqres.full
+
+# use mode=lfs to let f2fs always triggers OPU
+_scratch_mount -o mode=lfs,checkpoint=disable:10%,noinline_dentry >> $seqres.full
+
+testfile=$SCRATCH_MNT/testfile
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 | _filter_scratch
+
+_scratch_remount checkpoint=enable
+
+status=0
+exit
tests/f2fs/006.out new file +0 -0
new file mode 100644
index 00000000..2dc9efda

tests/f2fs/006.out line 6

Quoted 6 lines
tests/f2fs/006.out new file +6 -0
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing 'SCRATCH_MNT/testfile': No space left on device
+3+0 records in
+2+0 records out

Hi Chao,

One more question about this patch.

I'm wondering can this output always be matched? If the test doesn't care the dd output, you can filter out them, to avoid it break golden image. Especially the second one, can you be sure it always "3 in and 2 out" before returning ENOSPC?

Thanks,
Zorro

Quoted 3 lines
Signature · 1 line
2.40.1
author
#5 lore raw
Quoted 85 lines · Zorro Lang
Quoted 75 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function. If kernel config CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise, dd may encounter I/O error.

Cc Jaegeuk Kim <[email protected]>

---
v3:
- explain more about behavior on kernel w/ different config
- use _filter_scratch to filter $SCRATCH_MNT

  tests/f2fs/006     | 42 ++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/006.out |  6 ++++++
  2 files changed, 48 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +42 -0
new file mode 100755
index 00000000..f9560b2b
@@ -0,0 +1,42 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function. If kernel config
+# CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise,
+# dd may encounter I/O error.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"f2fs: fix to account dirty data in __get_secs_required()"
+
+# Import common functions.
+. ./common/filter
+
+_require_scratch
+_scratch_mkfs_sized $((1024*1024*100)) >> $seqres.full
+
+# use mode=lfs to let f2fs always triggers OPU
+_scratch_mount -o mode=lfs,checkpoint=disable:10%,noinline_dentry >> $seqres.full
+
+testfile=$SCRATCH_MNT/testfile
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 | _filter_scratch
+
+_scratch_remount checkpoint=enable
+
+status=0
+exit
tests/f2fs/006.out new file +0 -0
new file mode 100644
index 00000000..2dc9efda

tests/f2fs/006.out line 6

Quoted 6 lines
tests/f2fs/006.out new file +6 -0
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing 'SCRATCH_MNT/testfile': No space left on device
+3+0 records in
+2+0 records out

Hi Chao,

Quoted 5 lines

One more question about this patch.

I'm wondering can this output always be matched? If the test doesn't care the dd output, you can filter out them, to avoid it break golden image. Especially the second one, can you be sure it always "3 in and 2 out" before returning ENOSPC?

Zorro,

I got your concern, how about using _check_dmesg to catch the kernel bug instead? since once the bug was triggered, f2fs will call BUG_ON or WARN_ON optionally anyway.

Thanks,

Quoted 8 lines

Thanks,
Zorro

Quoted 3 lines
Signature · 1 line
2.40.1
#6 lore raw
Quoted 92 lines · Chao Yu
Quoted 85 lines · Zorro Lang
Quoted 75 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function. If kernel config CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise, dd may encounter I/O error.

Cc Jaegeuk Kim <[email protected]>

---
v3:
- explain more about behavior on kernel w/ different config
- use _filter_scratch to filter $SCRATCH_MNT

  tests/f2fs/006     | 42 ++++++++++++++++++++++++++++++++++++++++++
  tests/f2fs/006.out |  6 ++++++
  2 files changed, 48 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +42 -0
new file mode 100755
index 00000000..f9560b2b
@@ -0,0 +1,42 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function. If kernel config
+# CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise,
+# dd may encounter I/O error.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"f2fs: fix to account dirty data in __get_secs_required()"
+
+# Import common functions.
+. ./common/filter
+
+_require_scratch
+_scratch_mkfs_sized $((1024*1024*100)) >> $seqres.full
+
+# use mode=lfs to let f2fs always triggers OPU
+_scratch_mount -o mode=lfs,checkpoint=disable:10%,noinline_dentry >> $seqres.full
+
+testfile=$SCRATCH_MNT/testfile
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 | _filter_scratch
+
+_scratch_remount checkpoint=enable
+
+status=0
+exit
tests/f2fs/006.out new file +0 -0
new file mode 100644
index 00000000..2dc9efda

tests/f2fs/006.out line 6

Quoted 6 lines
tests/f2fs/006.out new file +6 -0
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing 'SCRATCH_MNT/testfile': No space left on device
+3+0 records in
+2+0 records out

Hi Chao,

Quoted 5 lines

One more question about this patch.

I'm wondering can this output always be matched? If the test doesn't care the dd output, you can filter out them, to avoid it break golden image. Especially the second one, can you be sure it always "3 in and 2 out" before returning ENOSPC?

Quoted 5 lines

Zorro,

I got your concern, how about using _check_dmesg to catch the kernel bug instead? since once the bug was triggered, f2fs will call BUG_ON or WARN_ON optionally anyway.

Hi Chao,

The _check_dmesg is always called at the end of each test. Except you need a dmesg filter, or you don't need to call it manually.

My concern is "unstable output cause unexpected failure on fixed kernel". If you think the last "dd" output is unstable and useless on fixed kernel, you can filter it out, just let the dmesg/hang break the test.

e.g.

# The 2nd dd might run out of space, and trigger a kernel warning or hang on

  # unfixed kernel
  dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >/dev/null
  dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 >/dev/null | _filter_scratch

or if you feel the error output isn't stable either, you can:

# The 2nd dd might run out of space, and trigger a kernel warning or hang on

  # unfixed kernel
  dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >>$seqres.full
  dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >>$seqres.full 2>&1
  ...
  echo "Silence is golden"

Or if you need those output, please make sure it's a stable output on any fixed system :)

Thanks,
Zorro

Quoted 12 lines

Thanks,

Quoted 8 lines

Thanks,
Zorro

Quoted 3 lines
Signature · 1 line
2.40.1
author
#7 lore raw
Quoted 95 lines · Zorro Lang
Quoted 94 lines · Chao Yu
Quoted 85 lines · Zorro Lang
Quoted 75 lines · Chao Yu

This is a regression test to check whether f2fs handles dirty data correctly when checkpoint is disabled, if lfs mode is on, it will trigger OPU for all overwritten data, this will cost free segments, so f2fs must account overwritten data as OPU data when calculating free space, otherwise, it may run out of free segments in f2fs' allocation function. If kernel config CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise, dd may encounter I/O error.

Cc Jaegeuk Kim <[email protected]>

---
v3:
- explain more about behavior on kernel w/ different config
- use _filter_scratch to filter $SCRATCH_MNT

   tests/f2fs/006     | 42 ++++++++++++++++++++++++++++++++++++++++++
   tests/f2fs/006.out |  6 ++++++
   2 files changed, 48 insertions(+)

create mode 100755 tests/f2fs/006
create mode 100644 tests/f2fs/006.out

tests/f2fs/006 new file +42 -0
new file mode 100755
index 00000000..f9560b2b
@@ -0,0 +1,42 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oppo.  All Rights Reserved.
+#
+# FS QA Test No. f2fs/006
+#
+# This is a regression test to check whether f2fs handles dirty
+# data correctly when checkpoint is disabled, if lfs mode is on,
+# it will trigger OPU for all overwritten data, this will cost
+# free segments, so f2fs must account overwritten data as OPU
+# data when calculating free space, otherwise, it may run out
+# of free segments in f2fs' allocation function. If kernel config
+# CONFIG_F2FS_CHECK_FS is on, it will cause system panic, otherwise,
+# dd may encounter I/O error.
+#
+. ./common/preamble
+_begin_fstest auto quick
+
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"f2fs: fix to account dirty data in __get_secs_required()"
+
+# Import common functions.
+. ./common/filter
+
+_require_scratch
+_scratch_mkfs_sized $((1024*1024*100)) >> $seqres.full
+
+# use mode=lfs to let f2fs always triggers OPU
+_scratch_mount -o mode=lfs,checkpoint=disable:10%,noinline_dentry >> $seqres.full
+
+testfile=$SCRATCH_MNT/testfile
+
+dd if=/dev/zero of=$testfile bs=1M count=50 2>/dev/null
+
+# it may run out of free space of f2fs and hang kernel
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync
+dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 | _filter_scratch
+
+_scratch_remount checkpoint=enable
+
+status=0
+exit
tests/f2fs/006.out new file +0 -0
new file mode 100644
index 00000000..2dc9efda

tests/f2fs/006.out line 6

Quoted 6 lines
tests/f2fs/006.out new file +6 -0
@@ -0,0 +1,6 @@
+QA output created by 006
+50+0 records in
+50+0 records out
+dd: error writing 'SCRATCH_MNT/testfile': No space left on device
+3+0 records in
+2+0 records out

Hi Chao,

Quoted 5 lines

One more question about this patch.

I'm wondering can this output always be matched? If the test doesn't care the dd output, you can filter out them, to avoid it break golden image. Especially the second one, can you be sure it always "3 in and 2 out" before returning ENOSPC?

Zorro,

Quoted 8 lines
Quoted 3 lines

I got your concern, how about using _check_dmesg to catch the kernel bug instead? since once the bug was triggered, f2fs will call BUG_ON or WARN_ON optionally anyway.

Hi Chao,

The _check_dmesg is always called at the end of each test. Except you need a dmesg filter, or you don't need to call it manually.

Zorro,

Oh, alright.

Quoted 4 lines

My concern is "unstable output cause unexpected failure on fixed kernel". If you think the last "dd" output is unstable and useless on fixed kernel, you can filter it out, just let the dmesg/hang break the test.

I just confirmed that the output may change according to mkfs.f2fs option, so I guess we'd better ignore output of dd, and check dmesg instead.

Quoted 8 lines

e.g.

# The 2nd dd might run out of space, and trigger a kernel warning or hang on

   # unfixed kernel
   dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >/dev/null
   dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync 2>&1 >/dev/null | _filter_scratch
Quoted 8 lines

or if you feel the error output isn't stable either, you can:

# The 2nd dd might run out of space, and trigger a kernel warning or hang on

   # unfixed kernel
   dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >>$seqres.full
   dd if=/dev/zero of=$testfile bs=1M count=50 conv=notrunc conv=fsync >>$seqres.full 2>&1
   ...
   echo "Silence is golden"

I think we can use above code to make output stable, thanks for your suggestion. :)

Thanks,

Quoted 20 lines

Or if you need those output, please make sure it's a stable output on any fixed system :)

Thanks,
Zorro

Quoted 12 lines

Thanks,

Quoted 8 lines

Thanks,
Zorro

Quoted 3 lines
Signature · 1 line
2.40.1