generic/224: Increase filesystem instance size to 1.5 GiB
Subsystems (2)
PATCH 1/1 generic/224: Increase filesystem instance size to 1.5 GiB
For small filesystem instances (i.e. size <= 1 GiB), mkfs.btrfs fails when "data block size" does not match with the "metadata block size" specified on the mkfs.btrfs command line. This commit increases the size of filesystem instance created so that the test can be executed on subpagesize-blocksize Btrfs instances which have different values for data and metadata blocksizes.
Increasing the filesystem size had a negligible impact on the runtime of the test as indicated by the following table:
|------------+--------------+----------------| | Filesystem | 1GiB FS size | 1.5GiB FS size | |------------+--------------+----------------| | Btrfs | 21secs | 24secs | | Ext4 | 12secs | 15secs | | Xfs | 10secs | 11secs | |------------+--------------+----------------|
Timing information reported in the above table were the worst-case times observed when running the test on a spinning disk.
--- tests/generic/224 | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tests/generic/224 +3 -2
@@ -53,8 +53,9 @@ _supported_os Linux _require_scratch -# make a 1GB filesystem -_scratch_mkfs_sized `expr 1024 \* 1024 \* 1024` > $seqres.full 2>&1 +# make a 1.5GB filesystem +FS_SIZE=$(echo "1.5 $((1024 * 1024 * 1024))" | awk '{printf "%d", $1 * $2}') +_scratch_mkfs_sized $FS_SIZE > $seqres.full 2>&1 _scratch_mount >> $seqres.full 2>&1 # set the reserved block pool to almost empty for XFS
Signature · 1 line
2.1.0
Quoted 5 lines · Chandan Rajendra
For small filesystem instances (i.e. size <= 1 GiB), mkfs.btrfs fails when "data block size" does not match with the "metadata block size" specified on the mkfs.btrfs command line. This commit increases the size of filesystem instance created so that the test can be executed on subpagesize-blocksize Btrfs instances which have different values for data and metadata blocksizes.
Stupid question --- why isn't this considered a bug in mkfs.btrfs? Does btrfs simply not support file systems <= 1 GB? So if someone has a 1GB USB disk or SD card, what's the official advice from the btrfs developers? Use xfs or ext4?
- Ted
Quoted 11 lines · Theodore Ts'o
Quoted 5 lines · Chandan Rajendra
For small filesystem instances (i.e. size <= 1 GiB), mkfs.btrfs fails when "data block size" does not match with the "metadata block size" specified on the mkfs.btrfs command line. This commit increases the size of filesystem instance created so that the test can be executed on subpagesize-blocksize Btrfs instances which have different values for data and metadata blocksizes.
Stupid question --- why isn't this considered a bug in mkfs.btrfs? Does btrfs simply not support file systems <= 1 GB? So if someone has a 1GB USB disk or SD card, what's the official advice from the btrfs developers? Use xfs or ext4?
AFAIK, it shouldn't be failing that way, and should automatically switch to mixed mode allocation. A 1G filesystem should work fine for BTRFS, but smaller ones will have higher chances of ENOSPC issues (inversely proportional to the size of the FS). I would advise against using BTRFS on such a small disk (I avoid using it on anything smaller than 4G personally), but I'm not one of the developers, and the fact that I feel it isn't a good idea doesn't mean it shouldn't work.
Quoted 7 lines · Austin S Hemmelgarn
AFAIK, it shouldn't be failing that way, and should automatically switch to mixed mode allocation. A 1G filesystem should work fine for BTRFS, but smaller ones will have higher chances of ENOSPC issues (inversely proportional to the size of the FS). I would advise against using BTRFS on such a small disk (I avoid using it on anything smaller than 4G personally), but I'm not one of the developers, and the fact that I feel it isn't a good idea doesn't mean it shouldn't work.
Instead of modifying generic/224, maybe it would be better to have a way to specify a minimum file system size on a per-file system basis. That way, if some file system does have a minimum size of say, 1G or 4G, it can be configured in one place, instead of needing to modify every test that uses a small file system size, or forcing all file systems to use a larger file systems just for the benefit of a single file system?
Or maybe just fix mkfs.btrfs?
- Ted
Quoted 7 lines · On Monday 31 Aug 2015 14:11:27 Theodore Ts'o wrote:
Quoted 6 lines · Chandan Rajendra
For small filesystem instances (i.e. size <= 1 GiB), mkfs.btrfs fails when "data block size" does not match with the "metadata block size" specified on the mkfs.btrfs command line. This commit increases the size of filesystem instance created so that the test can be executed on subpagesize-blocksize Btrfs instances which have different values for data and metadata blocksizes.
Quoted 5 lines
Stupid question --- why isn't this considered a bug in mkfs.btrfs? Does btrfs simply not support file systems <= 1 GB? So if someone has a 1GB USB disk or SD card, what's the official advice from the btrfs developers? Use xfs or ext4?
Ted, Btrfs does indeed support filesystem instances <= 1GiB. When creating such instances, mixed block groups are created i.e. These block groups hold both data and metadata. Hence the requirement of having matching "data block size" and "metadata block size" for filesystems <= 1 GiB.
mkfs.btrfs when invoked on small filesystems by "not" specifying any block sizes (i.e. mkfs.btrfs -f /dev/sda1) will automatically create filesystem instance with "data block size" == "metadata block size". However in the subpagesize-blocksize scenario, we need to specify both data and metadata block size on the command line (For e.g. mkfs.btrfs -f -s 4096 -n 16384 /dev/sda1). In this case, Since the user is forcing the block sizes and it is impossible to have mixed block groups with differing data and metadata block sizes, mkfs.btrfs will fail.
Signature · 1 line
chandan
Quoted 21 lines · On Tuesday 01 Sep 2015 05:49:14 Chandan Rajendra wrote:
Quoted 10 lines · On Monday 31 Aug 2015 14:11:27 Theodore Ts'o wrote:
Quoted 8 lines · Chandan Rajendra
For small filesystem instances (i.e. size <= 1 GiB), mkfs.btrfs fails when
"data block size" does not match with the "metadata block size" specified
on the mkfs.btrfs command line. This commit increases the size of filesystem instance created so that the test can be executed on subpagesize-blocksize Btrfs instances which have different values for data and metadata blocksizes.Quoted 4 lines
Stupid question --- why isn't this considered a bug in mkfs.btrfs? Does btrfs simply not support file systems <= 1 GB? So if someone has a 1GB USB disk or SD card, what's the official advice from the btrfs developers? Use xfs or ext4?
Ted, Btrfs does indeed support filesystem instances <= 1GiB. When creating such instances, mixed block groups are created i.e. These block groups hold both data and metadata. Hence the requirement of having matching "data block size" and "metadata block size" for filesystems <= 1 GiB.
Quoted 8 lines
mkfs.btrfs when invoked on small filesystems by "not" specifying any block sizes (i.e. mkfs.btrfs -f
/dev/sda1) will automatically create filesystem instance with "data block size" == "metadata block size". However in the subpagesize-blocksize scenario, we need to specify both data and metadata block size on the command line (For e.g. mkfs.btrfs -f -s 4096 -n 16384/dev/sda1). In this case, Since the user is forcing the block sizes and it is impossible to have mixed block groups with differing data and metadata block sizes, mkfs.btrfs will fail.
Also, For small filesystems, "mkfs.btrfs -f /dev/sda1 -s 4096 -n 4096" works fine since the command is invoked with the same size for both data and metadata blocks.
Signature · 1 line
chandan
Quoted 8 lines · Chandan Rajendra
mkfs.btrfs when invoked on small filesystems by "not" specifying any block sizes (i.e. mkfs.btrfs -f
/dev/sda1) will automatically create filesystem instance with "data block size" == "metadata block size". However in the subpagesize-blocksize scenario, we need to specify both data and metadata block size on the command line (For e.g. mkfs.btrfs -f -s 4096 -n 16384/dev/sda1). In this case, Since the user is forcing the block sizes and it is impossible to have mixed block groups with differing data and metadata block sizes, mkfs.btrfs will fail.
Ok, so the issue is that for this particular test configuration, btrfs has a minimum file system size. What about changing _scratch_mkfs_sized so that if MIN_FS_SIZE is set, the file system created will be at least MIN_FS_SIZE in size.
This way it sets the minimum file system size for all tests, not just generic/224, and any test configuration, whether it be ext4, xfs, or btrfs where the data and metadata block size are the same, don't have to take extra time -- only the test configuration of btrfs with data_block_size != metadata_block_size.
Cheers,
- Ted
Quoted 15 lines · On Monday 31 Aug 2015 22:15:10 Theodore Ts'o wrote:
Quoted 8 lines · Chandan Rajendra
mkfs.btrfs when invoked on small filesystems by "not" specifying any block sizes (i.e. mkfs.btrfs -f
/dev/sda1) will automatically create filesystem instance with "data block size" == "metadata block size". However in the subpagesize-blocksize scenario, we need to specify both data and metadata block size on the command line (For e.g. mkfs.btrfs -f -s 4096 -n 16384/dev/sda1). In this case, Since the user is forcing the block sizes and it is impossible to have mixed block groups with differing data and metadata block sizes, mkfs.btrfs will fail.Ok, so the issue is that for this particular test configuration, btrfs has a minimum file system size. What about changing
_scratch_mkfs_sizedso that ifMIN_FS_SIZEis set, the file system created will be at leastMIN_FS_SIZEin size.
Quoted 6 lines
This way it sets the minimum file system size for all tests, not just generic/224, and any test configuration, whether it be ext4, xfs, or btrfs where the data and metadata block size are the same, don't have to take extra time -- only the test configuration of btrfs with
data_block_size!=metadata_block_size.
I agree with the approach you have suggested. I will write up a patch and send it across the mailing list. Thanks Ted.
Signature · 1 line
chandan
Chandan Rajendra wrote on 2015/09/01 08:03 +0530:
Quoted 16 lines · On Monday 31 Aug 2015 22:15:10 Theodore Ts'o wrote:
Quoted 15 lines · Chandan Rajendra
Quoted 8 lines
mkfs.btrfs when invoked on small filesystems by "not" specifying any block sizes (i.e. mkfs.btrfs -f
/dev/sda1) will automatically create filesystem instance with "data block size" == "metadata block size". However in the subpagesize-blocksize scenario, we need to specify both data and metadata block size on the command line (For e.g. mkfs.btrfs -f -s 4096 -n 16384/dev/sda1). In this case, Since the user is forcing the block sizes and it is impossible to have mixed block groups with differing data and metadata block sizes, mkfs.btrfs will fail.Ok, so the issue is that for this particular test configuration, btrfs has a minimum file system size. What about changing
_scratch_mkfs_sizedso that ifMIN_FS_SIZEis set, the file system created will be at leastMIN_FS_SIZEin size.
Quoted 9 lines
Quoted 6 lines
This way it sets the minimum file system size for all tests, not just generic/224, and any test configuration, whether it be ext4, xfs, or btrfs where the data and metadata block size are the same, don't have to take extra time -- only the test configuration of btrfs with
data_block_size!=metadata_block_size.I agree with the approach you have suggested. I will write up a patch and send it across the mailing list. Thanks Ted.
Hi Chandan,
Although the reply may be a little late, but I think it's still better to fix the bug in mkfs.btrfs.
The core problem is, why mkfs.btrfs is insisting on make the fs into mixed-bg.
Auto detection for small device and make it into mixed-bg should be a "OPTIONAL" feature, not a "mandatory" one.
So IMHO, the best method is to disable mixed-bg when nodesize/sectorsize is given and differs and user doesn't force mkfs to create mixed-bg.
Thanks,
Qu
On Friday 11 Sep 2015 09:38:43 Qu Wenruo wrote:
Qu, Sorry for the late response. My email client's filtering had broken and hence I did not see your response until now.
Quoted 5 lines
Hi Chandan,
Although the reply may be a little late, but I think it's still better to fix the bug in mkfs.btrfs.
Quoted 9 lines
The core problem is, why mkfs.btrfs is insisting on make the fs into mixed-bg.
Auto detection for small device and make it into mixed-bg should be a "OPTIONAL" feature, not a "mandatory" one.
So IMHO, the best method is to disable mixed-bg when nodesize/sectorsize is given and differs and user doesn't force mkfs to create mixed-bg.
My understanding of the chunk allocator (i.e. __btrfs_alloc_chunk()) was incomplete. Looking into the code more thoroughly I now realize that 'small filesystem detection and mixed-bg creation' can be made optional. Thanks for the suggestion.
Signature · 1 line
chandan