bcachefs support
cover bcachefs support
A small patch adding bcachefs support, and two other patches for consideration:
Kent Overstreet (3):
Initial bcachefs support
Improved .gitignore
Use --yes option to lvcreate
.gitignore | 3 +++ common/attr | 6 ++++++ common/config | 3 +++ common/dmlogwrites | 7 +++++++ common/quota | 4 ++-- common/rc | 31 +++++++++++++++++++++++++++++++ tests/generic/042 | 3 ++- tests/generic/081 | 2 +- tests/generic/108 | 2 +- tests/generic/425 | 3 +++ tests/generic/441 | 2 +- tests/generic/482 | 27 ++++++++++++++++++++------- tests/generic/558 | 2 ++ 13 files changed, 82 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/3 Initial bcachefs support
From: Kent Overstreet <[email protected]>
--- common/attr | 6 ++++++ common/config | 3 +++ common/dmlogwrites | 7 +++++++ common/quota | 4 ++-- common/rc | 31 +++++++++++++++++++++++++++++++ tests/generic/042 | 3 ++- tests/generic/425 | 3 +++ tests/generic/441 | 2 +- tests/generic/482 | 27 ++++++++++++++++++++------- tests/generic/558 | 2 ++ 10 files changed, 77 insertions(+), 11 deletions(-)
common/attr +6 -0
@@ -33,6 +33,9 @@ _acl_get_max() echo 506 fi ;; + bcachefs) + echo 251 + ;; *) echo 0 ;;
@@ -273,6 +276,9 @@ pvfs2) 9p|ceph|nfs) MAX_ATTRVAL_SIZE=65536 ;; +bcachefs) + MAX_ATTRVAL_SIZE=1024 + ;; *) # Assume max ~1 block of attrs BLOCK_SIZE=`_get_block_size $TEST_DIR`
common/config +3 -0
@@ -415,6 +415,9 @@ _mkfs_opts() btrfs) export MKFS_OPTIONS="$BTRFS_MKFS_OPTIONS" ;; + bcachefs) + export MKFS_OPTIONS="--errors=panic" + ;; *) ;; esac
common/dmlogwrites +7 -0
@@ -111,6 +111,13 @@ _log_writes_replay_log() [ -z "$_blkdev" ] && _fail \ "block dev must be specified for _log_writes_replay_log" + if [ "$FSTYP" = "bcachefs" ]; then + # bcachefs gets confused if we're replaying the history out of + # order, and we see writes on the device from a newer point in + # time than what the superblock points to: + dd if=/dev/zero of=$SCRATCH_DEV bs=1M oflag=direct >& /dev/null + fi + $here/src/log-writes/replay-log --log $LOGWRITES_DEV --find \ --end-mark $_mark >> $seqres.full 2>&1 [ $? -ne 0 ] && _fail "mark '$_mark' does not exist"
common/quota +2 -2
@@ -17,7 +17,7 @@ _require_quota() _notrun "Installed kernel does not support quotas" fi ;; - gfs2|ocfs2) + gfs2|ocfs2|bcachefs) ;; xfs) if [ ! -f /proc/fs/xfs/xqmstat ]; then
@@ -278,7 +278,7 @@ _check_quota_usage() VFS_QUOTA=0 case $FSTYP in - ext2|ext3|ext4|ext4dev|f2fs|reiserfs|gfs2) + ext2|ext3|ext4|ext4dev|f2fs|reiserfs|gfs2|bcachefs) VFS_QUOTA=1 quotaon -f -u -g $SCRATCH_MNT 2>/dev/null ;;
common/rc +31 -0
@@ -334,6 +334,7 @@ _try_scratch_mount() return $? fi _mount -t $FSTYP `_scratch_mount_options $*` + return } # mount scratch device with given options and _fail if mount fails
@@ -667,6 +668,9 @@ _test_mkfs() ext2|ext3|ext4) $MKFS_PROG -t $FSTYP -- -F $MKFS_OPTIONS $* $TEST_DEV ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $TEST_DEV + ;; *) yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $TEST_DEV ;;
@@ -706,6 +710,10 @@ _mkfs_dev() $MKFS_PROG -t $FSTYP -- -f $MKFS_OPTIONS $* \ 2>$tmp.mkfserr 1>$tmp.mkfsstd ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* \ + 2>$tmp_dir.mkfserr 1>$tmp_dir.mkfsstd + ;; *) yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* \ 2>$tmp.mkfserr 1>$tmp.mkfsstd
@@ -803,6 +811,10 @@ _scratch_mkfs() mkfs_cmd="yes | $MKFS_PROG -t $FSTYP --" mkfs_filter="grep -v -e ^mkfs\.ocfs2" ;; + bcachefs) + mkfs_cmd="$MKFS_PROG -t $FSTYP --" + mkfs_filter="cat" + ;; *) mkfs_cmd="yes | $MKFS_PROG -t $FSTYP --" mkfs_filter="cat"
@@ -1065,6 +1077,9 @@ _scratch_mkfs_sized() fi export MOUNT_OPTIONS="-o size=$fssize $TMPFS_MOUNT_OPTIONS" ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS --fs_size=$fssize --block_size=$blocksize $SCRATCH_DEV + ;; *) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_sized" ;;
@@ -1133,6 +1148,9 @@ _scratch_mkfs_blocksized() ocfs2) yes | ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize -C $blocksize $SCRATCH_DEV ;; + bcachefs) + ${MKFS_PROG}.$FSTYP $MKFS_OPTIONS --block_size=$blocksize $SCRATCH_DEV + ;; *) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_blocksized" ;;
@@ -1179,6 +1197,19 @@ _repair_scratch_fs() fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1 + res=$? + case $res in + 0) + res=0 + ;; + *) + _dump_err2 "fsck.$FSTYP failed, err=$res" + ;; + esac + return $res + ;; *) local dev=$SCRATCH_DEV local fstyp=$FSTYP
tests/generic/042 +2 -1
@@ -63,7 +63,8 @@ _crashtest() # We should /never/ see 0xCD in the file, because we wrote that pattern # to the filesystem image to expose stale data. - if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then + # The file is not required to exist since we didn't sync before going down: + if [[ -f $file ]] && hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then echo "Saw stale data!!!" hexdump $file fi
tests/generic/425 +3 -0
@@ -30,6 +30,9 @@ _cleanup() # real QA test starts here _supported_fs generic + +[ $FSTYP = bcachefs ] && _notrun "bcachefs does not store xattrs in blocks" + _require_scratch _require_attrs _require_xfs_io_command "fiemap" "-a"
tests/generic/441 +1 -1
@@ -40,7 +40,7 @@ case $FSTYP in btrfs) _notrun "btrfs has a specialized test for this" ;; - ext3|ext4|xfs) + ext3|ext4|xfs|bcachefs) # Do the more thorough test if we have a logdev _has_logdev && sflag='' ;;
tests/generic/482 +20 -7
@@ -77,16 +77,29 @@ prev=$(_log_writes_mark_to_entry_number mkfs) cur=$(_log_writes_find_next_fua $prev) [ -z "$cur" ] && _fail "failed to locate next FUA write" +if [ "$FSTYP" = "bcachefs" ]; then + _dmthin_cleanup + _dmthin_init $devsize $devsize $csize $lowspace +fi + while [ ! -z "$cur" ]; do _log_writes_replay_log_range $cur $DMTHIN_VOL_DEV >> $seqres.full - # Here we need extra mount to replay the log, mainly for journal based - # fs, as their fsck will report dirty log as error. - # We don't care to preserve any data on the replay dev, as we can replay - # back to the point we need, and in fact sometimes creating/deleting - # snapshots repeatedly can be slower than replaying the log. - _dmthin_mount - _dmthin_check_fs + if [ "$FSTYP" = "bcachefs" ]; then + # bcachefs will get confused if fsck does writes to replay the log, + # but then we replay writes from an earlier point in time on the + # same fs - but fsck in -n mode won't do any writes: + _check_scratch_fs -n $DMTHIN_VOL_DEV + else + # Here we need extra mount to replay the log, mainly for journal based + # fs, as their fsck will report dirty log as error. + # We don't care to preserve any data on the replay dev, as we can replay + # back to the point we need, and in fact sometimes creating/deleting + # snapshots repeatedly can be slower than replaying the log. + + _dmthin_mount + _dmthin_check_fs + fi prev=$cur cur=$(_log_writes_find_next_fua $(($cur + 1)))
tests/generic/558 +2 -0
@@ -55,6 +55,8 @@ _scratch_mount i=0 free_inode=`_get_free_inode $SCRATCH_MNT` file_per_dir=1000 +[ $FSTYP = bcachefs ] && file_per_dir=10000 + loop=$((free_inode / file_per_dir + 1)) mkdir -p $SCRATCH_MNT/testdir
Signature · 1 line
2.31.1
Re: [PATCH 1/3] Initial bcachefs support
Quoted 1 line · Kent Overstreet
From: Kent Overstreet <[email protected]>
Better to add commit logs at least to give an example about how to setup fstests to test bcachefs.
Quoted 44 lines
Signed-off-by Kent Overstreet <[email protected]>--- common/attr | 6 ++++++ common/config | 3 +++ common/dmlogwrites | 7 +++++++ common/quota | 4 ++-- common/rc | 31 +++++++++++++++++++++++++++++++ tests/generic/042 | 3 ++- tests/generic/425 | 3 +++ tests/generic/441 | 2 +- tests/generic/482 | 27 ++++++++++++++++++++------- tests/generic/558 | 2 ++ 10 files changed, 77 insertions(+), 11 deletions(-)common/attr +6
-0@@ -33,6 +33,9 @@ _acl_get_max() echo 506 fi ;; + bcachefs) + echo 251 + ;; *) echo 0 ;;@@ -273,6 +276,9 @@ pvfs2) 9p|ceph|nfs) MAX_ATTRVAL_SIZE=65536 ;; +bcachefs) + MAX_ATTRVAL_SIZE=1024 + ;; *) # Assume max ~1 block of attrs BLOCK_SIZE=`_get_block_size $TEST_DIR`common/config +0
-0
common/config _mkfs_opts() · line 419
Quoted 5 lines
common/config +2
-0@@ -415,6 +415,9 @@ _mkfs_opts() btrfs) export MKFS_OPTIONS="$BTRFS_MKFS_OPTIONS" ;; + bcachefs) + export MKFS_OPTIONS="--errors=panic"
I think this might be useful for bcachefs developers to find bugs earlier, but not suitable for people like QA, as this may crash their test machines and stop the whole test.
You could always set MKFS_OPTIONS to "--errors=panic" explicitly when needed.
Quoted 9 lines
common/config +1
-0+ ;; *) ;; esaccommon/dmlogwrites +0
-0
common/dmlogwrites _log_writes_replay_log() · line 118
Quoted 8 lines
common/dmlogwrites +5
-0@@ -111,6 +111,13 @@ _log_writes_replay_log() [ -z "$_blkdev" ] && _fail \ "block dev must be specified for _log_writes_replay_log" + if [ "$FSTYP" = "bcachefs" ]; then + # bcachefs gets confused if we're replaying the history out of + # order, and we see writes on the device from a newer point in + # time than what the superblock points to: + dd if=/dev/zero of=$SCRATCH_DEV bs=1M oflag=direct >& /dev/null
I don't know bcachefs internals, I'm not sure I understand this, clearing the first 1M of SCRATCH_DEV seems to clear superblock, but I'm still not sure why it's needed. Does wipefs work?
Quoted 32 lines
common/dmlogwrites +2
-0+ fi + $here/src/log-writes/replay-log --log $LOGWRITES_DEV --find \ --end-mark $_mark >> $seqres.full 2>&1 [ $? -ne 0 ] && _fail "mark '$_mark' does not exist"common/quota +2
-2@@ -17,7 +17,7 @@ _require_quota() _notrun "Installed kernel does not support quotas" fi ;; - gfs2|ocfs2) + gfs2|ocfs2|bcachefs) ;; xfs) if [ ! -f /proc/fs/xfs/xqmstat ]; then@@ -278,7 +278,7 @@ _check_quota_usage() VFS_QUOTA=0 case $FSTYP in - ext2|ext3|ext4|ext4dev|f2fs|reiserfs|gfs2) + ext2|ext3|ext4|ext4dev|f2fs|reiserfs|gfs2|bcachefs) VFS_QUOTA=1 quotaon -f -u -g $SCRATCH_MNT 2>/dev/null ;;common/rc +0
-0
common/rc _try_scratch_mount() · line 337
Quoted 4 lines
common/rc +1
-0@@ -334,6 +334,7 @@ _try_scratch_mount() return $? fi _mount -t $FSTYP `_scratch_mount_options $*` + return
Seems not necessary.
common/rc _test_mkfs() · line 673
Quoted 10 lines
common/rc +3
-0} # mount scratch device with given options and _fail if mount fails@@ -667,6 +668,9 @@ _test_mkfs() ext2|ext3|ext4) $MKFS_PROG -t $FSTYP -- -F $MKFS_OPTIONS $* $TEST_DEV ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $TEST_DEV + ;;
I think we could just use the default mkfs command below. The only difference is dropping the "yes | " part, but that does nothing if mkfs doesn't read "yes" or "no" from stdin.
common/rc _mkfs_dev() · line 716
Quoted 11 lines
common/rc +4
-0*) yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $TEST_DEV ;;@@ -706,6 +710,10 @@ _mkfs_dev() $MKFS_PROG -t $FSTYP -- -f $MKFS_OPTIONS $* \ 2>$tmp.mkfserr 1>$tmp.mkfsstd ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* \ + 2>$tmp_dir.mkfserr 1>$tmp_dir.mkfsstd + ;;
Same as above.
common/rc _scratch_mkfs() · line 817
Quoted 11 lines
common/rc +4
-0*) yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* \ 2>$tmp.mkfserr 1>$tmp.mkfsstd@@ -803,6 +811,10 @@ _scratch_mkfs() mkfs_cmd="yes | $MKFS_PROG -t $FSTYP --" mkfs_filter="grep -v -e ^mkfs\.ocfs2" ;; + bcachefs) + mkfs_cmd="$MKFS_PROG -t $FSTYP --" + mkfs_filter="cat" + ;;
Same here.
Quoted 14 lines
common/rc +3
-0*) mkfs_cmd="yes | $MKFS_PROG -t $FSTYP --" mkfs_filter="cat"@@ -1065,6 +1077,9 @@ _scratch_mkfs_sized() fi export MOUNT_OPTIONS="-o size=$fssize $TMPFS_MOUNT_OPTIONS" ;; + bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS --fs_size=$fssize --block_size=$blocksize $SCRATCH_DEV + ;; *) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_sized" ;;
common/rc _scratch_mkfs_blocksized() · line 1153
Quoted 6 lines
common/rc +3
-0@@ -1133,6 +1148,9 @@ _scratch_mkfs_blocksized() ocfs2) yes | ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize -C $blocksize $SCRATCH_DEV ;; + bcachefs) + ${MKFS_PROG}.$FSTYP $MKFS_OPTIONS --block_size=$blocksize $SCRATCH_DEV + ;;
Better to be consistent and use "${MKFS_PROG} -t $FSTYP ..." instead of ${MKFS_PROG}.$FSTYP
common/rc _repair_scratch_fs() · line 1201
Quoted 9 lines
common/rc +2
-0*) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_blocksized" ;;@@ -1179,6 +1197,19 @@ _repair_scratch_fs() fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1
_repair_scratch_fs() is supposed to actually fix the errors, does "fsck -n" fix errors for bcachefs?
Quoted 19 lines
common/rc +11
-0+ res=$? + case $res in + 0) + res=0 + ;; + *) + _dump_err2 "fsck.$FSTYP failed, err=$res" + ;; + esac + return $res + ;; *) local dev=$SCRATCH_DEV local fstyp=$FSTYPtests/generic/042 +0
-0
tests/generic/042 _crashtest() · line 70
Quoted 9 lines
tests/generic/042 +2
-1@@ -63,7 +63,8 @@ _crashtest() # We should /never/ see 0xCD in the file, because we wrote that pattern # to the filesystem image to expose stale data. - if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then + # The file is not required to exist since we didn't sync before going down: + if [[ -f $file ]] && hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then echo "Saw stale data!!!" hexdump $file fi
Updates for individual test should be in a separate patch.
Quoted 73 lines
tests/generic/425 +3
-0@@ -30,6 +30,9 @@ _cleanup() # real QA test starts here _supported_fs generic + +[ $FSTYP = bcachefs ] && _notrun "bcachefs does not store xattrs in blocks" + _require_scratch _require_attrs _require_xfs_io_command "fiemap" "-a"tests/generic/441 +1
-1@@ -40,7 +40,7 @@ case $FSTYP in btrfs) _notrun "btrfs has a specialized test for this" ;; - ext3|ext4|xfs) + ext3|ext4|xfs|bcachefs) # Do the more thorough test if we have a logdev _has_logdev && sflag='' ;;tests/generic/482 +20
-7@@ -77,16 +77,29 @@ prev=$(_log_writes_mark_to_entry_number mkfs) cur=$(_log_writes_find_next_fua $prev) [ -z "$cur" ] && _fail "failed to locate next FUA write" +if [ "$FSTYP" = "bcachefs" ]; then + _dmthin_cleanup + _dmthin_init $devsize $devsize $csize $lowspace +fi + while [ ! -z "$cur" ]; do _log_writes_replay_log_range $cur $DMTHIN_VOL_DEV >> $seqres.full - # Here we need extra mount to replay the log, mainly for journal based - # fs, as their fsck will report dirty log as error. - # We don't care to preserve any data on the replay dev, as we can replay - # back to the point we need, and in fact sometimes creating/deleting - # snapshots repeatedly can be slower than replaying the log. - _dmthin_mount - _dmthin_check_fs + if [ "$FSTYP" = "bcachefs" ]; then + # bcachefs will get confused if fsck does writes to replay the log, + # but then we replay writes from an earlier point in time on the + # same fs - but fsck in -n mode won't do any writes: + _check_scratch_fs -n $DMTHIN_VOL_DEV + else + # Here we need extra mount to replay the log, mainly for journal based + # fs, as their fsck will report dirty log as error. + # We don't care to preserve any data on the replay dev, as we can replay + # back to the point we need, and in fact sometimes creating/deleting + # snapshots repeatedly can be slower than replaying the log. + + _dmthin_mount + _dmthin_check_fs + fi prev=$cur cur=$(_log_writes_find_next_fua $(($cur + 1)))tests/generic/558 +0
-0
tests/generic/558 _scratch_mount · line 59
Quoted 5 lines
tests/generic/558 +2
-0@@ -55,6 +55,8 @@ _scratch_mount i=0 free_inode=`_get_free_inode $SCRATCH_MNT` file_per_dir=1000 +[ $FSTYP = bcachefs ] && file_per_dir=10000 +
Some comments would be good.
Thanks,
Eryu
Quoted 5 lines
tests/generic/558 +0
-0loop=$((free_inode / file_per_dir + 1)) mkdir -p $SCRATCH_MNT/testdirSignature · 1 line
2.31.1
Re: [PATCH 1/3] Initial bcachefs support
Quoted 8 lines · Eryu Guan
Quoted 1 line · Kent Overstreet
From: Kent Overstreet <[email protected]>
Better to add commit logs at least to give an example about how to setup fstests to test bcachefs.
You could always set
MKFS_OPTIONSto "--errors=panic" explicitly when needed.
Forgot that was an option - doing that now.
Quoted 13 lines
Quoted 13 lines
common/config +1
-0+ ;; *) ;; esaccommon/dmlogwrites +0
-0@@ -111,6 +111,13 @@ _log_writes_replay_log() [ -z "$_blkdev" ] && _fail \ "block dev must be specified for _log_writes_replay_log"
Quoted 9 lines
common/dmlogwrites _log_writes_replay_log() · line 118
Quoted 5 lines
common/dmlogwrites +5
-0+ if [ "$FSTYP" = "bcachefs" ]; then + # bcachefs gets confused if we're replaying the history out of + # order, and we see writes on the device from a newer point in + # time than what the superblock points to: + dd if=/dev/zero of=$SCRATCH_DEV bs=1M oflag=direct >& /dev/nullI don't know bcachefs internals, I'm not sure I understand this, clearing the first 1M of
SCRATCH_DEVseems to clear superblock, but I'm still not sure why it's needed. Does wipefs work?
It's not just the superblock we need to clear, it's really all metadata - the journal, and btree nodes are also log structured in bcachefs. So 1M actually isn't sufficient - the better solution would be to either
- change the tests to check the markers in the log in the correct order, so we never see metadata from a future point in time, also making sure we don't do any writes to the filesystem when we're checking the different markers, or
- just replay to a new dm-thin device
This is basically what I did for generic/482, the 1M zerout is really just a hack for 455 and 457 and should probably be moved there, unless you've got another suggestion.
Quoted 11 lines
common/rc _try_scratch_mount() · line 337
Quoted 9 lines
common/rc +1
-0@@ -334,6 +334,7 @@ _try_scratch_mount() return $? fi _mount -t $FSTYP `_scratch_mount_options $*` + returnSeems not necessary.
Not sure how that got in, dropped it.
Quoted 7 lines
common/rc _test_mkfs() · line 673
Quoted 3 lines
common/rc +3
-0+ bcachefs) + $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $TEST_DEV + ;;I think we could just use the default mkfs command below. The only difference is dropping the "yes | " part, but that does nothing if mkfs doesn't read "yes" or "no" from stdin.
That dates from when my test environment had SIGPIPE set up wrong (systemd!), it's fixed now so I've dropped these.
Quoted 4 lines
Quoted 4 lines
common/rc +0
-0*) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_blocksized" ;;@@ -1179,6 +1197,19 @@ _repair_scratch_fs()
Quoted 8 lines
common/rc _repair_scratch_fs() · line 1201
Quoted 5 lines
common/rc +2
-0fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1
_repair_scratch_fs()is supposed to actually fix the errors, does "fsck -n" fix errors for bcachefs?
No - but with bcachefs fsck finding errors _always_ indicates a bug, so for the purposes of these tests I think this is the right thing to do - I don't want the tests to pass if fsck is finding and fixing errors.
Quoted 6 lines
Quoted 6 lines
tests/generic/042 +0
-0@@ -63,7 +63,8 @@ _crashtest()
Quoted 10 lines
tests/generic/042 _crashtest() · line 70
Quoted 8 lines
tests/generic/042 +2
-1# We should /never/ see 0xCD in the file, because we wrote that pattern # to the filesystem image to expose stale data. - if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then + # The file is not required to exist since we didn't sync before going down: + if [[ -f $file ]] && hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then echo "Saw stale data!!!" hexdump $file fiUpdates for individual test should be in a separate patch.
Ok, I'll split those out.
Re: [PATCH 1/3] Initial bcachefs support
On Sun, May 23, 2021 at 06:51:49PM -0400, Kent Overstreet wrote:
[snip]
Quoted 11 lines
Quoted 10 lines
common/rc _repair_scratch_fs() · line 1201
Quoted 7 lines
common/rc +2
-0;;@@ -1179,6 +1197,19 @@ _repair_scratch_fs() fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1
_repair_scratch_fs()is supposed to actually fix the errors, does "fsck -n" fix errors for bcachefs?
Quoted 3 lines
No - but with bcachefs fsck finding errors _always_ indicates a bug, so for the purposes of these tests I think this is the right thing to do - I don't want the tests to pass if fsck is finding and fixing errors.
Then _check_scratch_fs() should be used instead, which will fail the test if any fsck finds any corruptions. _repair_scratch_fs() is meant to fix errors, and only report failure when there's unfixable errors.
Thanks,
Eryu
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
tests/generic/042 +0
-0@@ -63,7 +63,8 @@ _crashtest()Quoted 10 lines
tests/generic/042 _crashtest() · line 70
Quoted 8 lines
tests/generic/042 +2
-1# We should /never/ see 0xCD in the file, because we wrote that pattern # to the filesystem image to expose stale data. - if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then + # The file is not required to exist since we didn't sync before going down: + if [[ -f $file ]] && hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then echo "Saw stale data!!!" hexdump $file fiUpdates for individual test should be in a separate patch.
Ok, I'll split those out.
Re: [PATCH 1/3] Initial bcachefs support
Quoted 14 lines · Eryu Guan
On Sun, May 23, 2021 at 06:51:49PM -0400, Kent Overstreet wrote:
[snip]Quoted 11 lines
Quoted 10 lines
common/rc _repair_scratch_fs() · line 1201
Quoted 7 lines
common/rc +2
-0;;@@ -1179,6 +1197,19 @@ _repair_scratch_fs() fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1
_repair_scratch_fs()is supposed to actually fix the errors, does "fsck -n" fix errors for bcachefs?
Quoted 7 lines
Quoted 3 lines
No - but with bcachefs fsck finding errors _always_ indicates a bug, so for the purposes of these tests I think this is the right thing to do - I don't want the tests to pass if fsck is finding and fixing errors.
Then
_check_scratch_fs()should be used instead, which will fail the test if any fsck finds any corruptions._repair_scratch_fs()is meant to fix errors, and only report failure when there's unfixable errors.
I see no reason to make such a change to generic tests that were written for other filesystems, when this gets me exactly what I want.
Re: [PATCH 1/3] Initial bcachefs support
Quoted 19 lines · Kent Overstreet
Quoted 18 lines · Eryu Guan
On Sun, May 23, 2021 at 06:51:49PM -0400, Kent Overstreet wrote:
[snip]Quoted 14 lines
Quoted 10 lines
common/rc _repair_scratch_fs() · line 1201
Quoted 7 lines
common/rc +2
-0;;@@ -1179,6 +1197,19 @@ _repair_scratch_fs() fi return $res ;; + bcachefs) + fsck -t $FSTYP -n $SCRATCH_DEV 2>&1
_repair_scratch_fs()is supposed to actually fix the errors, does "fsck -n" fix errors for bcachefs?No - but with bcachefs fsck finding errors _always_ indicates a bug, so for the purposes of these tests I think this is the right thing to do - I don't want the tests to pass if fsck is finding and fixing errors.
Quoted 6 lines
Quoted 3 lines
Then
_check_scratch_fs()should be used instead, which will fail the test if any fsck finds any corruptions._repair_scratch_fs()is meant to fix errors, and only report failure when there's unfixable errors.I see no reason to make such a change to generic tests that were written for other filesystems, when this gets me exactly what I want.
Oh, I noticed that "with bcachefs fsck finding errors _always_ indicates a bug" now.. Then I think using fsck -n is fine here, but better with comments to describe why this is ok for bcachefs.
Also, we could just use _check_scratch_fs here instead of the open-coded fsck command. As _check_scratch_fs calls _check_generic_filesystem() which calls fsck -n internally, and handles mount/umount device correctly and prints pretty log if there's any errors.
Thanks,
Eryu
Re: [PATCH 1/3] Initial bcachefs support
Quoted 8 lines · Eryu Guan
Oh, I noticed that "with bcachefs fsck finding errors _always_ indicates a bug" now.. Then I think using fsck -n is fine here, but better with comments to describe why this is ok for bcachefs.
Also, we could just use
_check_scratch_fshere instead of the open-coded fsck command. As_check_scratch_fscalls_check_generic_filesystem()which calls fsck -n internally, and handles mount/umount device correctly and prints pretty log if there's any errors.
That should work.
PATCH 2/3 Improved .gitignore
In reply to Kent Overstreet--- .gitignore | 3 +++ 1 file changed, 3 insertions(+)
.gitignore +3 -0
@@ -4,6 +4,9 @@ .dep .libs .ltdep +.* +cscope.* +tags /local.config /results
Signature · 1 line
2.31.1
PATCH 3/3 Use --yes option to lvcreate
In reply to Kent OverstreetThis fixes spurious test failures caused by broken pipe messages.
--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tests/generic/081 +1 -1
@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1 # wait for lvcreation to fully complete $UDEV_SETTLE_PROG >>$seqres.full 2>&1
tests/generic/108 +1 -1
@@ -56,7 +56,7 @@ $LVM_PROG pvcreate -f $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -i 2 -I 4m -L 275m -n $lvname $vgname \ +$LVM_PROG lvcreate --yes -i 2 -I 4m -L 275m -n $lvname $vgname \ >>$seqres.full 2>&1 # wait for lv creation to fully complete $UDEV_SETTLE_PROG >>$seqres.full 2>&1
Signature · 1 line
2.31.1
Re: [PATCH 3/3] Use --yes option to lvcreate
Quoted 13 lines · Kent Overstreet
This fixes spurious test failures caused by broken pipe messages.
Signed-off-by Kent Overstreet <[email protected]>--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/generic/081 +0
-0
tests/generic/081 _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 · line 73
Quoted 5 lines
tests/generic/081 +1
-1@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1
Please see above comments, we use yes pipe intentionally. I don't see how this would result in broken pipe. Would you please provide more details? And let's see if we could fix the broken pipe issue.
Thanks,
Eryu
Quoted 18 lines
tests/generic/081 +0
-0# wait for lvcreation to fully complete $UDEV_SETTLE_PROG >>$seqres.full 2>&1tests/generic/108 +1
-1@@ -56,7 +56,7 @@ $LVM_PROG pvcreate -f $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -i 2 -I 4m -L 275m -n $lvname $vgname \ +$LVM_PROG lvcreate --yes -i 2 -I 4m -L 275m -n $lvname $vgname \ >>$seqres.full 2>&1 # wait for lv creation to fully complete $UDEV_SETTLE_PROG >>$seqres.full 2>&1Signature · 1 line
2.31.1
Re: [PATCH 3/3] Use --yes option to lvcreate
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Kent Overstreet
This fixes spurious test failures caused by broken pipe messages.
Signed-off-by Kent Overstreet <[email protected]>--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
Quoted 14 lines
tests/generic/081 _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 · line 73
Quoted 10 lines
tests/generic/081 +1
-1@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1Please see above comments, we use yes pipe intentionally. I don't see how this would result in broken pipe. Would you please provide more details? And let's see if we could fix the broken pipe issue.
If lvcreate never ask y/n - never reads from standard input, then echo sees a broken pipe when it tries to write. That's what I get without this patch.
Re: [PATCH 3/3] Use --yes option to lvcreate
Quoted 25 lines · Kent Overstreet
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Kent Overstreet
This fixes spurious test failures caused by broken pipe messages.
Signed-off-by Kent Overstreet <[email protected]>--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)Quoted 14 lines
tests/generic/081 _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 · line 73
Quoted 10 lines
tests/generic/081 +1
-1@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1Please see above comments, we use yes pipe intentionally. I don't see how this would result in broken pipe. Would you please provide more details? And let's see if we could fix the broken pipe issue.
Quoted 2 lines
If lvcreate never ask y/n - never reads from standard input, then echo sees a broken pipe when it tries to write. That's what I get without this patch.
I think it's something in how ktest sets up the environment. I also see the SIGPIPEs when using your ktest scripts, but not when ssh'ing into the guest and running the test.
What that thing is, I don't know. I'm not tall enough to understand signal handling.
Re: [PATCH 3/3] Use --yes option to lvcreate
Quoted 29 lines · Matthew Wilcox
Quoted 25 lines · Kent Overstreet
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Kent Overstreet
This fixes spurious test failures caused by broken pipe messages.
Signed-off-by Kent Overstreet <[email protected]>--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)Quoted 14 lines
tests/generic/081 _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 · line 73
Quoted 10 lines
tests/generic/081 +1
-1@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1Please see above comments, we use yes pipe intentionally. I don't see how this would result in broken pipe. Would you please provide more details? And let's see if we could fix the broken pipe issue.
Quoted 2 lines
If lvcreate never ask y/n - never reads from standard input, then echo sees a broken pipe when it tries to write. That's what I get without this patch.
Quoted 6 lines
I think it's something in how ktest sets up the environment. I also see the SIGPIPEs when using your ktest scripts, but not when ssh'ing into the guest and running the test.
What that thing is, I don't know. I'm not tall enough to understand signal handling.
See xfstests commit 9bcb266cd778 ("generic/397: be compatible with ignored SIGPIPE") for an example of the same problem being fixed in another test, with some more explanation.
But it's better to just not execute xfstests (or any shell script, for that matter) in an environment where SIGPIPE is ignored.
- Eric
Re: [PATCH 3/3] Use --yes option to lvcreate
Quoted 37 lines · Eric Biggers
Quoted 29 lines · Matthew Wilcox
Quoted 25 lines · Kent Overstreet
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Kent Overstreet
This fixes spurious test failures caused by broken pipe messages.
Signed-off-by Kent Overstreet <[email protected]>--- tests/generic/081 | 2 +- tests/generic/108 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)Quoted 14 lines
tests/generic/081 _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 · line 73
Quoted 10 lines
tests/generic/081 +1
-1@@ -70,7 +70,7 @@ _scratch_mkfs_sized $((300 * 1024 * 1024)) >>$seqres.full 2>&1 $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm # (like 2.02.95 in RHEL6) don't support --yes option -yes | $LVM_PROG lvcreate -L 256M -n $lvname $vgname >>$seqres.full 2>&1 +$LVM_PROG lvcreate --yes -L 256M -n $lvname $vgname >>$seqres.full 2>&1Please see above comments, we use yes pipe intentionally. I don't see how this would result in broken pipe. Would you please provide more details? And let's see if we could fix the broken pipe issue.
Quoted 2 lines
If lvcreate never ask y/n - never reads from standard input, then echo sees a broken pipe when it tries to write. That's what I get without this patch.
Quoted 6 lines
I think it's something in how ktest sets up the environment. I also see the SIGPIPEs when using your ktest scripts, but not when ssh'ing into the guest and running the test.
What that thing is, I don't know. I'm not tall enough to understand signal handling.
Quoted 6 lines
See xfstests commit
9bcb266cd778("generic/397: be compatible with ignored SIGPIPE") for an example of the same problem being fixed in another test, with some more explanation.But it's better to just not execute xfstests (or any shell script, for that matter) in an environment where SIGPIPE is ignored.
Ah, thanks for the info. I'll see if I can figure out what's mucking with the signal handler (probably systemd...)
Quoted 1 line · Kent Overstreet
A small patch adding bcachefs support, and two other patches for consideration:
As bcachefs is not upstream yet, I think we should re-visit bcachefs support after it's in upstream.
OTOH, I have some minor comments go to patch 1.
Thanks,
Eryu
Quoted 23 lines
Kent Overstreet (3):
Initial bcachefs support
Improved .gitignore
Use --yes option to lvcreate.gitignore | 3 +++ common/attr | 6 ++++++ common/config | 3 +++ common/dmlogwrites | 7 +++++++ common/quota | 4 ++-- common/rc | 31 +++++++++++++++++++++++++++++++ tests/generic/042 | 3 ++- tests/generic/081 | 2 +- tests/generic/108 | 2 +- tests/generic/425 | 3 +++ tests/generic/441 | 2 +- tests/generic/482 | 27 ++++++++++++++++++++------- tests/generic/558 | 2 ++ 13 files changed, 82 insertions(+), 13 deletions(-)Signature · 1 line
2.31.1
Quoted 5 lines · Eryu Guan
Quoted 1 line · Kent Overstreet
A small patch adding bcachefs support, and two other patches for consideration:
As bcachefs is not upstream yet, I think we should re-visit bcachefs support after it's in upstream.
I disagree completely. I've been waiting for this to land for some time so I can actually run fstests against bcachefs easily to evaluate it's current state of stability and support. The plans are to get bcachefs merged upstream, and so having support already in fstests makes it much easier for reviewers and developers to actually run tests and find problems prior to merging.
As an upstream developer and someone who will be reviewing bcachefs when it is next proposed for merge, I would much prefer to see extensive and long term fstests coverage *before* the code is even merged upstream. Given that filesystems take years to develop to the point where they are stable and ready for merge, saying "can't enable the test environment until it is merged upstream" is not very helpful.
As a general principle, we want developers of new filesystems to start using fstests early in the development process of their filesystem. We should be encouraging new filesystems to be added to fstests, not saying "we only support upstream filesystems". If the filesystem plans to be merged upstream, then fstests support for that filesystem should be there long before the filesytsem is even proposed for merge. We need to help people get new filesystems upstream, not place arbitrary "not upstream so not supported" catch-22s in their way...
Hence I ask that you merge bcachefs support to help the process of getting bcachefs suport upstream.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 32 lines · Dave Chinner
Quoted 5 lines · Eryu Guan
Quoted 1 line · Kent Overstreet
A small patch adding bcachefs support, and two other patches for consideration:
As bcachefs is not upstream yet, I think we should re-visit bcachefs support after it's in upstream.
I disagree completely. I've been waiting for this to land for some time so I can actually run fstests against bcachefs easily to evaluate it's current state of stability and support. The plans are to get bcachefs merged upstream, and so having support already in fstests makes it much easier for reviewers and developers to actually run tests and find problems prior to merging.
As an upstream developer and someone who will be reviewing bcachefs when it is next proposed for merge, I would much prefer to see extensive and long term fstests coverage *before* the code is even merged upstream. Given that filesystems take years to develop to the point where they are stable and ready for merge, saying "can't enable the test environment until it is merged upstream" is not very helpful.
As a general principle, we want developers of new filesystems to start using fstests early in the development process of their filesystem. We should be encouraging new filesystems to be added to fstests, not saying "we only support upstream filesystems". If the filesystem plans to be merged upstream, then fstests support for that filesystem should be there long before the filesytsem is even proposed for merge. We need to help people get new filesystems upstream, not place arbitrary "not upstream so not supported" catch-22s in their way...
Quoted 2 lines
Hence I ask that you merge bcachefs support to help the process of getting bcachefs suport upstream.
/me notes that both Eryu and Dave have been willing to merge surprisingly large quantities of code for XFS reflink and online fsck long before either of those features landed in mainline, so I think it's perfectly fine to merge Kent's relatively small changes to enable 'FSTYP=bcachefs'.
(With all of Eryu's review comments fixed, obviously...)
--D
Quoted 6 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 23 lines · Darrick J. Wong
Quoted 22 lines · Dave Chinner
Quoted 5 lines · Eryu Guan
Quoted 1 line · Kent Overstreet
A small patch adding bcachefs support, and two other patches for consideration:
As bcachefs is not upstream yet, I think we should re-visit bcachefs support after it's in upstream.
I disagree completely. I've been waiting for this to land for some time so I can actually run fstests against bcachefs easily to evaluate it's current state of stability and support. The plans are to get bcachefs merged upstream, and so having support already in fstests makes it much easier for reviewers and developers to actually run tests and find problems prior to merging.
As an upstream developer and someone who will be reviewing bcachefs when it is next proposed for merge, I would much prefer to see extensive and long term fstests coverage *before* the code is even merged upstream. Given that filesystems take years to develop to the point where they are stable and ready for merge, saying "can't enable the test environment until it is merged upstream" is not very helpful.
Quoted 9 lines
Quoted 9 lines
As a general principle, we want developers of new filesystems to start using fstests early in the development process of their filesystem. We should be encouraging new filesystems to be added to fstests, not saying "we only support upstream filesystems". If the filesystem plans to be merged upstream, then fstests support for that filesystem should be there long before the filesytsem is even proposed for merge. We need to help people get new filesystems upstream, not place arbitrary "not upstream so not supported" catch-22s in their way...
OK, that makes sense. Actually, I should have made my "upstream first" more clear. I'd love to merge non-upstream features/new fs supports if the proposed new feature/new filesystem has been developmented actively and the community has generally made the agreement that will merge the new feature/filesystem when it's in a good shape. IOW, it's not some random new features/filesystems, which are very likely to be dropped in the half way.
And providing such info in the patch is very helpful to reviewers.
Quoted 3 lines
Quoted 3 lines
Hence I ask that you merge bcachefs support to help the process of getting bcachefs suport upstream.
Sure, bcachefs looks promising to me now :)
Quoted 4 lines
/me notes that both Eryu and Dave have been willing to merge surprisingly large quantities of code for XFS reflink and online fsck long before either of those features landed in mainline, so I think it's
Because I know you and the xfs commnity will work on it continuously and very unlikely make the new tests dead code :)
Thanks,
Eryu
Quoted 13 lines
perfectly fine to merge Kent's relatively small changes to enable 'FSTYP=bcachefs'.
(With all of Eryu's review comments fixed, obviously...)
--D
Quoted 6 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]