generic: test cloning large exents to a file with many small extents
PATCH 1/1 generic: test cloning large exents to a file with many small extents
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extents
The test currently passes on xfs.
--- tests/generic/558 | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 81 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.out
tests/generic/558 new file +75 -0
@@ -0,0 +1,75 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096 +num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +done + +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (for btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done + +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar" $SCRATCH_MNT/foo >>/dev/null + +_scratch_remount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exit
tests/generic/558.out new file +5 -0
@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728
tests/generic/group +1 -0
@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone
Signature · 1 line
2.11.0
Quoted 60 lines · On Thu, Jun 27, 2019 at 06:00:30PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>--- tests/generic/558 | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 81 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +30
-0@@ -0,0 +1,75 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic
tests/generic/558 line 40
Quoted 10 lines
tests/generic/558 new file +10
-0+_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096
What if the fs block size is 64k?
tests/generic/558 line 48
Quoted 8 lines
tests/generic/558 new file +8
-0+num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +done
I wouldn't have thought that this would actually succeed on xfs because you could lay extents down one after the other, but then started seeing this in the filefrag output:
File size of /opt/foo is 528384 (129 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
3: 17.. 17: 52.. 52: 1: 37: 4: 18.. 18: 67.. 67: 1: 53: 5: 19.. 19: 81.. 81: 1: 68: 6: 20.. 20: 94.. 94: 1: 82: 7: 21.. 21: 106.. 106: 1: 95: 8: 22.. 22: 117.. 117: 1: 107: 9: 23.. 23: 127.. 127: 1: 118: 10: 24.. 24: 136.. 136: 1: 128: 11: 25.. 25: 144.. 144: 1: 137: 12: 26.. 26: 151.. 151: 1: 145: 13: 27.. 27: 157.. 157: 1: 152: 14: 28.. 28: 162.. 162: 1: 158: 15: 29.. 29: 166.. 166: 1: 163: 16: 30.. 30: 169.. 169: 1: 167: 17: 31.. 32: 171.. 172: 2: 170: 18: 33.. 33: 188.. 188: 1: 173:
52, 67, 81, 94, 106, 117, 127, 136, 44, 151, 157, 162, 166, 169, 171...
+15 +14 +13 +12 +11 +10 +9 +8 +7 +6 +5 +4 +3 +2...
Hm, I wonder what quirk of the xfs allocator this might be?
Quoted 5 lines
tests/generic/558 new file +5
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null +
tests/generic/558 line 63
Quoted 10 lines
tests/generic/558 new file +10
-0+# Fill the fs (for btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done
_fill_fs?
tests/generic/558 line 67
Quoted 4 lines
tests/generic/558 new file +4
-0+ +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar" $SCRATCH_MNT/foo >>/dev/null
_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo ?
--D
Quoted 31 lines
tests/generic/558 new file +8
-0+ +_scratch_remount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-0@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto cloneSignature · 1 line
2.11.0
Quoted 66 lines · Darrick J. Wong
Quoted 64 lines · On Thu, Jun 27, 2019 at 06:00:30PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>--- tests/generic/558 | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 81 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +34
-0@@ -0,0 +1,75 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full
Quoted 8 lines
tests/generic/558 line 40
Quoted 6 lines
tests/generic/558 new file +6
-0+ +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096What if the fs block size is 64k?
Then we get extents of 64Kb instead of 4Kb. Works on btrfs. Is it a problem for xfs (or any other fs)?
Quoted 33 lines
tests/generic/558 line 48
Quoted 9 lines
tests/generic/558 new file +8
-0+num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +doneI wouldn't have thought that this would actually succeed on xfs because you could lay extents down one after the other, but then started seeing this in the filefrag output:
File size of
/opt/foois 528384 (129 blocks of 4096 bytes)
ext:logical_offset:physical_offset: length: expected: flags:3: 17.. 17: 52.. 52: 1: 37: 4: 18.. 18: 67.. 67: 1: 53: 5: 19.. 19: 81.. 81: 1: 68: 6: 20.. 20: 94.. 94: 1: 82: 7: 21.. 21: 106.. 106: 1: 95: 8: 22.. 22: 117.. 117: 1: 107: 9: 23.. 23: 127.. 127: 1: 118: 10: 24.. 24: 136.. 136: 1: 128: 11: 25.. 25: 144.. 144: 1: 137: 12: 26.. 26: 151.. 151: 1: 145: 13: 27.. 27: 157.. 157: 1: 152: 14: 28.. 28: 162.. 162: 1: 158: 15: 29.. 29: 166.. 166: 1: 163: 16: 30.. 30: 169.. 169: 1: 167: 17: 31.. 32: 171.. 172: 2: 170: 18: 33.. 33: 188.. 188: 1: 173:
Quoted 5 lines
52, 67, 81, 94, 106, 117, 127, 136, 44, 151, 157, 162, 166, 169, 171...
+15 +14 +13 +12 +11 +10 +9 +8 +7 +6 +5 +4 +3 +2...
Hm, I wonder what quirk of the xfs allocator this might be?
Don't ask me, I have no idea how it works :)
Quoted 10 lines
Quoted 10 lines
tests/generic/558 new file +9
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (for btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed).
Quoted 8 lines
tests/generic/558 line 63
Quoted 6 lines
tests/generic/558 new file +6
-0+i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done
_fill_fs?
No, because that will do fallocate. Writing 2Kb files here is on purpose because that will fill the metadata
block group(s) (on btrfs we inline extents in the btree (metadata) as long as they are not bigger than 2Kb, by default).
Using fallocate, we end up using data block groups.
Quoted 7 lines
tests/generic/558 line 67
Quoted 5 lines
tests/generic/558 new file +4
-0+ +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar" $SCRATCH_MNT/foo >>/dev/null
_reflink$SCRATCH_MNT/bar $SCRATCH_MNT/foo ?
Does exactly the same. I don't mind changing it however.
Thanks.
Quoted 34 lines
--D
Quoted 31 lines
tests/generic/558 new file +8
-0+ +_scratch_remount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-1@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone --2.11.0
Quoted 76 lines · Filipe Manana
Quoted 66 lines · Darrick J. Wong
Quoted 64 lines · On Thu, Jun 27, 2019 at 06:00:30PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>--- tests/generic/558 | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 81 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +34
-0@@ -0,0 +1,75 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.fullQuoted 8 lines
tests/generic/558 line 40
Quoted 6 lines
tests/generic/558 new file +6
-0+ +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096What if the fs block size is 64k?
Quoted 2 lines
Then we get extents of 64Kb instead of 4Kb. Works on btrfs. Is it a problem for xfs (or any other fs)?
It shouldn't be; I was merely wondering if 2048 extents was enough to trigger the enospc on btrfs.
Quoted 33 lines
Quoted 33 lines
tests/generic/558 line 48
Quoted 9 lines
tests/generic/558 new file +8
-0+num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +doneI wouldn't have thought that this would actually succeed on xfs because you could lay extents down one after the other, but then started seeing this in the filefrag output:
File size of
/opt/foois 528384 (129 blocks of 4096 bytes)
ext:logical_offset:physical_offset: length: expected: flags:3: 17.. 17: 52.. 52: 1: 37: 4: 18.. 18: 67.. 67: 1: 53: 5: 19.. 19: 81.. 81: 1: 68: 6: 20.. 20: 94.. 94: 1: 82: 7: 21.. 21: 106.. 106: 1: 95: 8: 22.. 22: 117.. 117: 1: 107: 9: 23.. 23: 127.. 127: 1: 118: 10: 24.. 24: 136.. 136: 1: 128: 11: 25.. 25: 144.. 144: 1: 137: 12: 26.. 26: 151.. 151: 1: 145: 13: 27.. 27: 157.. 157: 1: 152: 14: 28.. 28: 162.. 162: 1: 158: 15: 29.. 29: 166.. 166: 1: 163: 16: 30.. 30: 169.. 169: 1: 167: 17: 31.. 32: 171.. 172: 2: 170: 18: 33.. 33: 188.. 188: 1: 173:
Quoted 7 lines
Quoted 5 lines
52, 67, 81, 94, 106, 117, 127, 136, 44, 151, 157, 162, 166, 169, 171...
+15 +14 +13 +12 +11 +10 +9 +8 +7 +6 +5 +4 +3 +2...
Hm, I wonder what quirk of the xfs allocator this might be?
Don't ask me, I have no idea how it works :)
Ah, it's because xfs frees post-eof allocations every time we close a file... and I never got the series that fixes this finalized and sent upstream.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
tests/generic/558 new file +9
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (for btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed).Quoted 8 lines
tests/generic/558 line 63
Quoted 6 lines
tests/generic/558 new file +6
-0+i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done
_fill_fs?
Quoted 5 lines
No, because that will do fallocate. Writing 2Kb files here is on purpose because that will fill the metadata
block group(s) (on btrfs we inline extents in the btree (metadata) as long as they are not bigger than 2Kb, by default).
Using fallocate, we end up using data block groups.
Ok.
Quoted 9 lines
Quoted 7 lines
tests/generic/558 line 67
Quoted 5 lines
tests/generic/558 new file +4
-0+ +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar" $SCRATCH_MNT/foo >>/dev/null
_reflink$SCRATCH_MNT/bar $SCRATCH_MNT/foo ?Does exactly the same. I don't mind changing it however.
<Nod>
--D
Quoted 36 lines
Thanks.
Quoted 34 lines
--D
Quoted 31 lines
tests/generic/558 new file +8
-0+ +_scratch_remount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-1@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone --2.11.0
PATCH v2 1/1 generic: test cloning large exents to a file with many small extents
In reply to [email protected]From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extents
The test currently passes on xfs.
---
V2: Use _scratch_cycle_mount instead of _scratch_remount, as we want to see
if the operation was durably persisted (otherwise we are seeing content from the page cache).
Use _reflink instead of calling xfs_io with the reflink command. Make the comment before filling the filesystem more clear about why it is done the way it is instead of using _fill_fs.
tests/generic/558 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 86 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.out
tests/generic/558 new file +80 -0
@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096 +num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +done + +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (For btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed, we +# create files with 2Kb because that results in extents inlined in the metadata +# (btree leafs) and it's the fastest way to fill metadata space on btrfs, by +# default btrfs inlines up to 2Kb of data). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done + +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full + +# Unmount and mount the filesystem again to verify the operation was durably +# persisted. +_scratch_cycle_mount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exit
tests/generic/558.out new file +5 -0
@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728
tests/generic/group +1 -0
@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone
Signature · 1 line
2.11.0
Quoted 76 lines · On Fri, Jun 28, 2019 at 11:08:36PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>---
V2: Use
_scratch_cycle_mountinstead of_scratch_remount, as we want to see
if the operation was durably persisted (otherwise we are seeing content from the page cache).
Use_reflinkinstead of callingxfs_iowith the reflink command. Make the comment before filling the filesystem more clear about why it is done the way it is instead of using_fill_fs.tests/generic/558 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 86 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +38
-0@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount +
tests/generic/558 line 48
Quoted 10 lines
tests/generic/558 new file +10
-0+file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096 +num_extents=$(( $file_size / $extent_size )) + +# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +done
This is taking too long time (1000+s) to finish when testing on XFS. I'm wondering if we could take use of src/punch-alternating to punch out every other block to create a file with many small extents.
i.e. with the following diffs, test runs & passes with XFS within 90s while still reproduces the btrfs bug (note that I have to increase the file_size to 200M to reproduce the btrfs bug, while 256M seems bring test time back to 1000+s). Would you please check if this works for you?
Thanks,
Eryu
tests/generic/558 +6 -6
@@ -30,22 +30,22 @@ _cleanup() _supported_fs generic _supported_os Linux _require_scratch_reflink +_require_test_program "punch-alternating" +_require_xfs_io_command "fpunch" rm -f $seqres.full _scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -file_size=$(( 128 * 1024 * 1024 )) # 128Mb +file_size=$(( 200 * 1024 * 1024 )) # 200MB extent_size=4096 num_extents=$(( $file_size / $extent_size )) # Create a file with many small extents. -for ((i = 0; i < $num_extents; i++)); do - offset=$(( $i * $extent_size )) - $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ - $SCRATCH_MNT/foo >>/dev/null -done +$XFS_IO_PROG -f -c "pwrite -S 0xe5 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/foo >>/dev/null +$here/src/punch-alternating $SCRATCH_MNT/foo >> $seqres.full # Create file bar with the same size that file foo has but with large extents. $XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \
tests/generic/558.out +1 -1
@@ -2,4 +2,4 @@ QA output created by 558 File foo data after cloning and remount: 0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 * -134217728 +209715200
Quoted 55 lines
tests/generic/558 new file +32
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (For btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed, we +# create files with 2Kb because that results in extents inlined in the metadata +# (btree leafs) and it's the fastest way to fill metadata space on btrfs, by +# default btrfs inlines up to 2Kb of data). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done + +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full + +# Unmount and mount the filesystem again to verify the operation was durably +# persisted. +_scratch_cycle_mount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-0@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto cloneSignature · 1 line
2.11.0
Quoted 82 lines · Eryu Guan
Quoted 80 lines · On Fri, Jun 28, 2019 at 11:08:36PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>---
V2: Use
_scratch_cycle_mountinstead of_scratch_remount, as we want to see
if the operation was durably persisted (otherwise we are seeing content from the page cache).
Use_reflinkinstead of callingxfs_iowith the reflink command. Make the comment before filling the filesystem more clear about why it is done the way it is instead of using_fill_fs.tests/generic/558 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 86 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +42
-0@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096 +num_extents=$(( $file_size / $extent_size )) +
Quoted 8 lines
tests/generic/558 line 48
Quoted 6 lines
tests/generic/558 new file +6
-0+# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +doneThis is taking too long time (1000+s) to finish when testing on XFS.
Hum, for on 5.2-rc, debug kernel (lockdep, plenty of other debug stuff enabled that slows things down), it takes about 350 seconds for me.
Quoted 3 lines
I'm
wondering if we could take use of src/punch-alternating to punch out every other block to create a file with many small extents.
That's fine, I didn't remember about that.
Quoted 5 lines
i.e. with the following diffs, test runs & passes with XFS within 90s while still reproduces the btrfs bug (note that I have to increase the
file_sizeto 200M to reproduce the btrfs bug, while 256M seems bring test time back to 1000+s). Would you please check if this works for you?
It does, thanks!
And, because of the holes, it uncovered a bug in my fix (second patch).
Now it takes ~15 seconds on btrfs instead of ~300s and ~100s on xfs for me.
Quoted 15 lines
Thanks,
Eryutests/generic/558 +2
-0@@ -30,22 +30,22 @@ _cleanup() _supported_fs generic _supported_os Linux _require_scratch_reflink +_require_test_program "punch-alternating" +_require_xfs_io_command "fpunch"
tests/generic/558 _cleanup() · line 43
Quoted 10 lines
tests/generic/558 +1
-1rm -f $seqres.full _scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -file_size=$(( 128 * 1024 * 1024 )) # 128Mb +file_size=$(( 200 * 1024 * 1024 )) # 200MB extent_size=4096 num_extents=$(( $file_size / $extent_size ))
These two variables, extent_size and num_extents, are no longer needed.
Do you want me to send another version of the patch or can you apply this patch on top of it?
Thanks.
Quoted 79 lines
tests/generic/558 +3
-5# Create a file with many small extents. -for ((i = 0; i < $num_extents; i++)); do - offset=$(( $i * $extent_size )) - $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ - $SCRATCH_MNT/foo >>/dev/null -done +$XFS_IO_PROG -f -c "pwrite -S 0xe5 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/foo >>/dev/null +$here/src/punch-alternating $SCRATCH_MNT/foo >> $seqres.full # Create file bar with the same size that file foo has but with large extents. $XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \tests/generic/558.out +1
-1@@ -2,4 +2,4 @@ QA output created by 558 File foo data after cloning and remount: 0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 * -134217728 +209715200Quoted 56 lines
tests/generic/558 new file +32
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (For btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed, we +# create files with 2Kb because that results in extents inlined in the metadata +# (btree leafs) and it's the fastest way to fill metadata space on btrfs, by +# default btrfs inlines up to 2Kb of data). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done + +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full + +# Unmount and mount the filesystem again to verify the operation was durably +# persisted. +_scratch_cycle_mount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-1@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone --2.11.0
Quoted 102 lines · Filipe Manana
Quoted 82 lines · Eryu Guan
Quoted 80 lines · On Fri, Jun 28, 2019 at 11:08:36PM +0100, [email protected] wrote:
From: Filipe Manana <[email protected]>
Test that if we clone a file with some large extents into a file that has many small extents, when the fs is nearly full, the clone operation does not fail and produces the correct result.
This is motivated by a bug found in btrfs wich is fixed by the following patches for the linux kernel:
[PATCH 1/2] Btrfs: factor out extent dropping code from hole punch handler [PATCH 2/2] Btrfs: fix ENOSPC errors, leading to transaction aborts, when
cloning extentsThe test currently passes on xfs.
Signed-off-by Filipe Manana <[email protected]>---
V2: Use
_scratch_cycle_mountinstead of_scratch_remount, as we want to see
if the operation was durably persisted (otherwise we are seeing content from the page cache).
Use_reflinkinstead of callingxfs_iowith the reflink command. Make the comment before filling the filesystem more clear about why it is done the way it is instead of using_fill_fs.tests/generic/558 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/558.out | 5 ++++ tests/generic/group | 1 + 3 files changed, 86 insertions(+) create mode 100755 tests/generic/558 create mode 100644 tests/generic/558.outtests/generic/558 new file +42
-0@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FSQA Test No. 558 +# +# Test that if we clone a file with some large extents into a file that has +# many small extents, when the fs is nearly full, the clone operation does +# not fail and produces the correct result. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_require_scratch_reflink + +rm -f $seqres.full + +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mount + +file_size=$(( 128 * 1024 * 1024 )) # 128Mb +extent_size=4096 +num_extents=$(( $file_size / $extent_size )) +Quoted 8 lines
tests/generic/558 line 48
Quoted 6 lines
tests/generic/558 new file +6
-0+# Create a file with many small extents. +for ((i = 0; i < $num_extents; i++)); do + offset=$(( $i * $extent_size )) + $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ + $SCRATCH_MNT/foo >>/dev/null +doneThis is taking too long time (1000+s) to finish when testing on XFS.
Hum, for on 5.2-rc, debug kernel (lockdep, plenty of other debug stuff enabled that slows things down), it takes about 350 seconds for me.
Quoted 3 lines
I'm
wondering if we could take use of src/punch-alternating to punch out every other block to create a file with many small extents.That's fine, I didn't remember about that.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
i.e. with the following diffs, test runs & passes with XFS within 90s while still reproduces the btrfs bug (note that I have to increase the
file_sizeto 200M to reproduce the btrfs bug, while 256M seems bring test time back to 1000+s). Would you please check if this works for you?It does, thanks!
And, because of the holes, it uncovered a bug in my fix (second patch).
That's great :)
Quoted 2 lines
Now it takes ~15 seconds on btrfs instead of ~300s and ~100s on xfs for me.
Sounds good!
Quoted 29 lines
Quoted 16 lines
Thanks,
Eryutests/generic/558 +2
-0@@ -30,22 +30,22 @@ _cleanup() _supported_fs generic _supported_os Linux _require_scratch_reflink +_require_test_program "punch-alternating" +_require_xfs_io_command "fpunch"tests/generic/558 _cleanup() · line 43
Quoted 10 lines
tests/generic/558 +1
-1rm -f $seqres.full _scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -file_size=$(( 128 * 1024 * 1024 )) # 128Mb +file_size=$(( 200 * 1024 * 1024 )) # 200MB extent_size=4096 num_extents=$(( $file_size / $extent_size ))These two variables,
extent_sizeandnum_extents, are no longer needed.
Quoted 2 lines
Do you want me to send another version of the patch or can you apply this patch on top of it?
I can fix the patch on commit. Thanks for the confirmation!
Eryu
Quoted 82 lines
Thanks.
Quoted 79 lines
tests/generic/558 +3
-5# Create a file with many small extents. -for ((i = 0; i < $num_extents; i++)); do - offset=$(( $i * $extent_size )) - $XFS_IO_PROG -f -s -c "pwrite -S 0xe5 $offset $extent_size" \ - $SCRATCH_MNT/foo >>/dev/null -done +$XFS_IO_PROG -f -c "pwrite -S 0xe5 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/foo >>/dev/null +$here/src/punch-alternating $SCRATCH_MNT/foo >> $seqres.full # Create file bar with the same size that file foo has but with large extents. $XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \tests/generic/558.out +1
-1@@ -2,4 +2,4 @@ QA output created by 558 File foo data after cloning and remount: 0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 * -134217728 +209715200Quoted 56 lines
tests/generic/558 new file +32
-0+ +# Create file bar with the same size that file foo has but with large extents. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $file_size 0 $file_size" \ + $SCRATCH_MNT/bar >>/dev/null + +# Fill the fs (For btrfs we are interested in filling all unallocated space +# and most of the existing metadata block group(s), so that after this there +# will be no unallocated space and metadata space will be mostly full but with +# more than enough free space for the clone operation below to succeed, we +# create files with 2Kb because that results in extents inlined in the metadata +# (btree leafs) and it's the fastest way to fill metadata space on btrfs, by +# default btrfs inlines up to 2Kb of data). +i=1 +while true; do + $XFS_IO_PROG -f -c "pwrite 0 2K" $SCRATCH_MNT/filler_$i &> /dev/null + [ $? -ne 0 ] && break + i=$(( i + 1 )) +done + +# Now clone file bar into file foo. This is supposed to succeed and not fail +# with ENOSPC for example. +_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full + +# Unmount and mount the filesystem again to verify the operation was durably +# persisted. +_scratch_cycle_mount + +echo "File foo data after cloning and remount:" +od -A d -t x1 $SCRATCH_MNT/foo + +status=0 +exittests/generic/558.out new file +5
-0@@ -0,0 +1,5 @@ +QA output created by 558 +File foo data after cloning and remount: +0000000 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 c7 +* +134217728tests/generic/group +1
-1@@ -560,3 +560,4 @@ 555 auto quick cap 556 auto quick casefold 557 auto quick log +558 auto clone --2.11.0