f2fs: test for race condition in between atomic_write and gc
PATCH 1/1 f2fs: test for race condition in between atomic_write and gc
Test that we will simulate sqlite atomic write logic w/ below steps:
1. create a regular file, and initialize it w/ 0xff data
2. start transaction (via F2FS_IOC_START_ATOMIC_WRITE) on it
3. write transaction data
4. trigger foreground GC to migrate data block of the file
5. commit and end the transaction (via F2FS_IOC_COMMIT_ATOMIC_WRITE)
6. check consistency of transaction w/ in-memory and on-disk data This is a regression test to check handling of race condition in between atomic_write and GC.
--- tests/f2fs/003 | 61 ++++++++++++++++++++++++++++++++++++++++++++++ tests/f2fs/003.out | 11 +++++++++ 2 files changed, 72 insertions(+) create mode 100755 tests/f2fs/003 create mode 100644 tests/f2fs/003.out
tests/f2fs/003 new file +61 -0
@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oppo. All Rights Reserved. +# +# FS QA Test No. f2fs/003 +# +# Test that we will simulate sqlite atomic write logic w/ below steps: +# 1. create a regular file, and initialize it w/ 0xff data +# 2. start transaction (via F2FS_IOC_START_ATOMIC_WRITE) on it +# 3. write transaction data +# 4. trigger foreground GC to migrate data block of the file +# 5. commit and end the transaction (via F2FS_IOC_COMMIT_ATOMIC_WRITE) +# 6. check consistency of transaction w/ in-memory and on-disk data +# This is a regression test to check handling of race condition in +# between atomic_write and GC. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_supported_fs f2fs +_require_scratch +_require_xfs_io_command "pwrite" +_require_xfs_io_command "fpunch" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full + +dbfile=$SCRATCH_MNT/dbfile +foo=$SCRATCH_MNT/foo +bar=$SCRATCH_MNT/bar + +$XFS_IO_PROG -c "pwrite 0 512k -S 0xff" -c "fsync" -f $dbfile >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $foo >> $seqres.full +sync + +# start atomic_write on dbfile & write data to dbfile +$F2FS_IO_PROG write 1 0 32 zero atomic_commit $dbfile 3000 >> $seqres.full & +$XFS_IO_PROG -c "fpunch 0 2m" $foo >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $bar >> $seqres.full + +# persist all dirty data +sync +echo 3 > /proc/sys/vm/drop_caches + +# trigger foreground GC to migrate data block of atomic_file +$F2FS_IO_PROG gc 1 $SCRATCH_MNT > /dev/null 2>&1 + +# wait for atomic_write commit completion +sleep 5 +# print in-memory data +od -x $dbfile +echo 3 > /proc/sys/vm/drop_caches +# print on-disk data +od -x $dbfile + +_scratch_unmount + +status=0 +exit
tests/f2fs/003.out new file +11 -0
@@ -0,0 +1,11 @@ +QA output created by 003 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000
Signature · 1 line
2.40.1
Quoted 9 lines · Chao Yu
Test that we will simulate sqlite atomic write logic w/ below steps:
1. create a regular file, and initialize it w/0xffdata
2. start transaction (viaF2FS_IOC_START_ATOMIC_WRITE) on it
3. write transaction data
4. trigger foreground GC to migrate data block of the file
5. commit and end the transaction (viaF2FS_IOC_COMMIT_ATOMIC_WRITE)
6. check consistency of transaction w/ in-memory and on-disk data This is a regression test to check handling of race condition in betweenatomic_writeand GC.
Hi Chao,
Sorry for the late reviewing. As this's a regression test, so which onne kernel commit fix this known issue, please specify it by:
_fixed_by_kernel_commit $commit_id $commit_subject
Quoted 29 lines
Cc Jaegeuk Kim <[email protected]>Cc Daeho Jeong <[email protected]>Signed-off-by Chao Yu <[email protected]>--- tests/f2fs/003 | 61 ++++++++++++++++++++++++++++++++++++++++++++++ tests/f2fs/003.out | 11 +++++++++ 2 files changed, 72 insertions(+) create mode 100755 tests/f2fs/003 create mode 100644 tests/f2fs/003.outtests/f2fs/003 new file +12
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oppo. All Rights Reserved. +# +# FS QA Test No. f2fs/003 +# +# Test that we will simulate sqlite atomic write logic w/ below steps: +# 1. create a regular file, and initialize it w/ 0xff data +# 2. start transaction (via F2FS_IOC_START_ATOMIC_WRITE) on it +# 3. write transaction data +# 4. trigger foreground GC to migrate data block of the file +# 5. commit and end the transaction (via F2FS_IOC_COMMIT_ATOMIC_WRITE)
tests/f2fs/003 line 22
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# 6. check consistency of transaction w/ in-memory and on-disk data +# This is a regression test to check handling of race condition in +# between atomic_write and GC. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_supported_fs f2fs
^^^^^
This line is not necessary now.
tests/f2fs/003 line 24
Quoted 2 lines
tests/f2fs/003 new file +2
-0+_require_scratch +_require_xfs_io_command "pwrite"
pwrite is a basic command of xfs_io, so I think this line is not necessary.
Quoted 5 lines
tests/f2fs/003 new file +5
-0+_require_xfs_io_command "fpunch" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full +
tests/f2fs/003 line 39
Quoted 10 lines
tests/f2fs/003 new file +10
-0+dbfile=$SCRATCH_MNT/dbfile +foo=$SCRATCH_MNT/foo +bar=$SCRATCH_MNT/bar + +$XFS_IO_PROG -c "pwrite 0 512k -S 0xff" -c "fsync" -f $dbfile >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $foo >> $seqres.full +sync + +# start atomic_write on dbfile & write data to dbfile +$F2FS_IO_PROG write 1 0 32 zero atomic_commit $dbfile 3000 >> $seqres.full &
As there's a background process, shouldn't we do kill and wait in _cleanup to make sure unmount won't hit EBUSY?
Quoted 7 lines
tests/f2fs/003 new file +7
-0+$XFS_IO_PROG -c "fpunch 0 2m" $foo >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $bar >> $seqres.full + +# persist all dirty data +sync +echo 3 > /proc/sys/vm/drop_caches +
tests/f2fs/003 line 56
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# trigger foreground GC to migrate data block of atomic_file +$F2FS_IO_PROG gc 1 $SCRATCH_MNT > /dev/null 2>&1 + +# wait for atomic_write commit completion +sleep 5 +# print in-memory data +od -x $dbfile +echo 3 > /proc/sys/vm/drop_caches +# print on-disk data +od -x $dbfile
There's a common helper named "_hexdump" in common/rc, can that be used?
tests/f2fs/003 line 58
Quoted 2 lines
tests/f2fs/003 new file +2
-0+ +_scratch_unmount
The SCRATCH_DEV will be unmounted at the end of the test, don't need to do this manually if it's not a necessary test step.
Thanks,
Zorro
Quoted 23 lines
tests/f2fs/003 new file +3
-0+ +status=0 +exittests/f2fs/003.out new file +11
-0@@ -0,0 +1,11 @@ +QA output created by 003 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000Signature · 1 line
2.40.1
Quoted 11 lines · Zorro Lang
Quoted 9 lines · Chao Yu
Test that we will simulate sqlite atomic write logic w/ below steps:
1. create a regular file, and initialize it w/0xffdata
2. start transaction (viaF2FS_IOC_START_ATOMIC_WRITE) on it
3. write transaction data
4. trigger foreground GC to migrate data block of the file
5. commit and end the transaction (viaF2FS_IOC_COMMIT_ATOMIC_WRITE)
6. check consistency of transaction w/ in-memory and on-disk data This is a regression test to check handling of race condition in betweenatomic_writeand GC.
Quoted 6 lines
Hi Chao,
Sorry for the late reviewing. As this's a regression test, so which onne kernel commit fix this known issue, please specify it by:
_fixed_by_kernel_commit$commit_id $commit_subject
Hi Zorro,
Since the fixed patch has not been merged yet, so what about adding this line after merging the fix?
Quoted 87 lines
Quoted 30 lines
Cc Jaegeuk Kim <[email protected]>Cc Daeho Jeong <[email protected]>Signed-off-by Chao Yu <[email protected]>--- tests/f2fs/003 | 61 ++++++++++++++++++++++++++++++++++++++++++++++ tests/f2fs/003.out | 11 +++++++++ 2 files changed, 72 insertions(+)create mode 100755 tests/f2fs/003
create mode 100644 tests/f2fs/003.outtests/f2fs/003 new file +12
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oppo. All Rights Reserved. +# +# FS QA Test No. f2fs/003 +# +# Test that we will simulate sqlite atomic write logic w/ below steps: +# 1. create a regular file, and initialize it w/ 0xff data +# 2. start transaction (via F2FS_IOC_START_ATOMIC_WRITE) on it +# 3. write transaction data +# 4. trigger foreground GC to migrate data block of the file +# 5. commit and end the transaction (via F2FS_IOC_COMMIT_ATOMIC_WRITE)tests/f2fs/003 line 22
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# 6. check consistency of transaction w/ in-memory and on-disk data +# This is a regression test to check handling of race condition in +# between atomic_write and GC. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_supported_fs f2fs^^^^^This line is not necessary now.
tests/f2fs/003 line 24
Quoted 2 lines
tests/f2fs/003 new file +2
-0+_require_scratch +_require_xfs_io_command "pwrite"pwrite is a basic command of
xfs_io, so I think this line is not necessary.Quoted 5 lines
tests/f2fs/003 new file +5
-0+_require_xfs_io_command "fpunch" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full +tests/f2fs/003 line 39
Quoted 10 lines
tests/f2fs/003 new file +10
-0+dbfile=$SCRATCH_MNT/dbfile +foo=$SCRATCH_MNT/foo +bar=$SCRATCH_MNT/bar + +$XFS_IO_PROG -c "pwrite 0 512k -S 0xff" -c "fsync" -f $dbfile >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $foo >> $seqres.full +sync + +# start atomic_write on dbfile & write data to dbfile +$F2FS_IO_PROG write 1 0 32 zero atomic_commit $dbfile 3000 >> $seqres.full &As there's a background process, shouldn't we do kill and wait in
_cleanupto make sure unmount won't hit EBUSY?Quoted 7 lines
tests/f2fs/003 new file +7
-0+$XFS_IO_PROG -c "fpunch 0 2m" $foo >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $bar >> $seqres.full + +# persist all dirty data +sync +echo 3 > /proc/sys/vm/drop_caches +tests/f2fs/003 line 56
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# trigger foreground GC to migrate data block of atomic_file +$F2FS_IO_PROG gc 1 $SCRATCH_MNT > /dev/null 2>&1 + +# wait for atomic_write commit completion +sleep 5 +# print in-memory data +od -x $dbfile +echo 3 > /proc/sys/vm/drop_caches +# print on-disk data +od -x $dbfileThere's a common helper named "
_hexdump" in common/rc, can that be used?
Quoted 5 lines
tests/f2fs/003 line 58
Quoted 2 lines
tests/f2fs/003 new file +2
-0+ +_scratch_unmountThe
SCRATCH_DEVwill be unmounted at the end of the test, don't need to do this manually if it's not a necessary test step.
Let me update this patch according to your comments, thanks.
Thanks,
Quoted 28 lines
Thanks,
ZorroQuoted 23 lines
tests/f2fs/003 new file +3
-0+ +status=0 +exittests/f2fs/003.out new file +11
-0@@ -0,0 +1,11 @@ +QA output created by 003 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000Signature · 1 line
2.40.1
Quoted 17 lines · Chao Yu
Quoted 16 lines · Zorro Lang
Quoted 9 lines · Chao Yu
Test that we will simulate sqlite atomic write logic w/ below steps:
1. create a regular file, and initialize it w/0xffdata
2. start transaction (viaF2FS_IOC_START_ATOMIC_WRITE) on it
3. write transaction data
4. trigger foreground GC to migrate data block of the file
5. commit and end the transaction (viaF2FS_IOC_COMMIT_ATOMIC_WRITE)
6. check consistency of transaction w/ in-memory and on-disk data This is a regression test to check handling of race condition in betweenatomic_writeand GC.Hi Chao,
Sorry for the late reviewing. As this's a regression test, so which onne kernel commit fix this known issue, please specify it by:
Quoted 6 lines
Quoted 1 line
_fixed_by_kernel_commit$commit_id $commit_subjectHi Zorro,
Since the fixed patch has not been merged yet, so what about adding this line after merging the fix?
Oh, sure, but generally they'd like to use "xxxxxxxxxxxx" to be the $commit_id (as a reminder) if there's not an assured id number. Then replace the "xxxxxx...." with the right id number after the patch merged. But it depends on you, just don't forget that :)
Thanks,
Zorro
Quoted 127 lines
Quoted 88 lines
Quoted 31 lines
Cc Jaegeuk Kim <[email protected]>Cc Daeho Jeong <[email protected]>Signed-off-by Chao Yu <[email protected]>--- tests/f2fs/003 | 61 ++++++++++++++++++++++++++++++++++++++++++++++ tests/f2fs/003.out | 11 +++++++++ 2 files changed, 72 insertions(+)create mode 100755 tests/f2fs/003
create mode 100644 tests/f2fs/003.outtests/f2fs/003 new file +12
-0@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oppo. All Rights Reserved. +# +# FS QA Test No. f2fs/003 +# +# Test that we will simulate sqlite atomic write logic w/ below steps: +# 1. create a regular file, and initialize it w/ 0xff data +# 2. start transaction (via F2FS_IOC_START_ATOMIC_WRITE) on it +# 3. write transaction data +# 4. trigger foreground GC to migrate data block of the file +# 5. commit and end the transaction (via F2FS_IOC_COMMIT_ATOMIC_WRITE)tests/f2fs/003 line 22
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# 6. check consistency of transaction w/ in-memory and on-disk data +# This is a regression test to check handling of race condition in +# between atomic_write and GC. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_supported_fs f2fs^^^^^This line is not necessary now.
tests/f2fs/003 line 24
Quoted 2 lines
tests/f2fs/003 new file +2
-0+_require_scratch +_require_xfs_io_command "pwrite"pwrite is a basic command of
xfs_io, so I think this line is not necessary.Quoted 5 lines
tests/f2fs/003 new file +5
-0+_require_xfs_io_command "fpunch" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full +tests/f2fs/003 line 39
Quoted 10 lines
tests/f2fs/003 new file +10
-0+dbfile=$SCRATCH_MNT/dbfile +foo=$SCRATCH_MNT/foo +bar=$SCRATCH_MNT/bar + +$XFS_IO_PROG -c "pwrite 0 512k -S 0xff" -c "fsync" -f $dbfile >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $foo >> $seqres.full +sync + +# start atomic_write on dbfile & write data to dbfile +$F2FS_IO_PROG write 1 0 32 zero atomic_commit $dbfile 3000 >> $seqres.full &As there's a background process, shouldn't we do kill and wait in
_cleanupto make sure unmount won't hit EBUSY?Quoted 7 lines
tests/f2fs/003 new file +7
-0+$XFS_IO_PROG -c "fpunch 0 2m" $foo >> $seqres.full +$XFS_IO_PROG -c "pwrite 0 2m" -c "fsync" -f $bar >> $seqres.full + +# persist all dirty data +sync +echo 3 > /proc/sys/vm/drop_caches +tests/f2fs/003 line 56
Quoted 10 lines
tests/f2fs/003 new file +10
-0+# trigger foreground GC to migrate data block of atomic_file +$F2FS_IO_PROG gc 1 $SCRATCH_MNT > /dev/null 2>&1 + +# wait for atomic_write commit completion +sleep 5 +# print in-memory data +od -x $dbfile +echo 3 > /proc/sys/vm/drop_caches +# print on-disk data +od -x $dbfileThere's a common helper named "
_hexdump" in common/rc, can that be used?Quoted 5 lines
tests/f2fs/003 line 58
Quoted 2 lines
tests/f2fs/003 new file +2
-0+ +_scratch_unmountThe
SCRATCH_DEVwill be unmounted at the end of the test, don't need to do this manually if it's not a necessary test step.Let me update this patch according to your comments, thanks.
Thanks,
Quoted 28 lines
Thanks,
ZorroQuoted 23 lines
tests/f2fs/003 new file +3
-0+ +status=0 +exittests/f2fs/003.out new file +11
-0@@ -0,0 +1,11 @@ +QA output created by 003 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000 +0000000 0000 0000 0000 0000 0000 0000 0000 0000 +* +0400000 ffff ffff ffff ffff ffff ffff ffff ffff +* +2000000Signature · 1 line
2.40.1