btrfs: Add new test for setting the chunk size.
PATCH v4 1/1 btrfs: Add new test for setting the chunk size.
Summary:
Add new testcase for testing the new btrfs sysfs knob to change the chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.
Note: this test needs to force the allocation of space. It uses the /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.
---
V4: - fixed indentation in common/btrfs
- Removed UUID code, which is no longer necessary (provided by helper function)
- Used new helper _get_btrfs_sysfs_attr
- Direct output to /dev/null
V3: - removed tests for system block type.
- added failure case for system block type.
- renamed stripe_size to chunk_size
V2: - added new functions to common/btrfs and use them in the new test
- _require_btrfs_sysfs_attr - Make sure btrfs supports a sysfs setting
- _get_btrfs_sysfs_attr - Read sysfs value
- _set_btrfs_sysfs_attr - Write sysfs value
- create file system of required size with _scratch_mkfs_sized
- use shortened error message
- Remove last logging message
--- common/btrfs | 73 ++++++++++++ tests/btrfs/248 | 280 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/248.out | 11 ++ 3 files changed, 364 insertions(+) create mode 100755 tests/btrfs/248 create mode 100644 tests/btrfs/248.out
common/btrfs +73 -0
@@ -445,3 +445,76 @@ _scratch_btrfs_is_zoned() [ `_zone_type ${SCRATCH_DEV}` != "none" ] && return 0 return 1 } + +# Print the content of /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _get_btrfs_sysfs_attr /mnt/scratch allocation/data/stripe_size +_get_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2 + + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + cat /sys/fs/btrfs/${uuid}/${attr} +} + +# Write "content" into /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# - content: the content of $attr +# +# Usage example: +# _set_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_set_btrfs_sysfs_attr() +{ + local mnt=$1 + shift + local attr=$1 + shift + local content="$*" + + if [ ! -e "$mnt" -o -z "$attr" -o -z "$content" ];then + _fail "Usage: _set_btrfs_sysfs_attr <mounted_directory> <attr> <content>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + echo "$content" > /sys/fs/btrfs/${uuid}/${attr} 2>/dev/null + if [ $? -ne 0 ]; then + _dump_err "_set_btrfs_sysfs_attr cannot write ${attr}" + fi +} + +# Verify if the sysfs entry in /sys/fs/btrfs/$UUID/$ATTR exists +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _require_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_require_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2 + + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + if [[ ! -e /sys/fs/btrfs/${uuid}/${attr} ]]; then + _notrun "Btrfs does not support sysfs $attr" + fi +} +
tests/btrfs/248 new file +280 -0
280 lines not on this page · raw message
tests/btrfs/248.out new file +11 -0
@@ -0,0 +1,11 @@ +QA output created by 248 +Capture default chunk sizes. +First allocation. +Second allocation. +Calculate request size so last memory allocation cannot be completely fullfilled. +Third allocation. +Force allocation of system block type must fail. +_set_btrfs_sysfs_attr cannot write allocation/system/force_chunk_alloc +Verify first allocation. +Verify second allocation. +Verify third allocation.
Signature · 1 line
2.30.2
Quoted 11 lines · Stefan Roesch
Summary:
Add new testcase for testing the new btrfs sysfs knob to change the chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.
Quoted 7 lines
Note: this test needs to force the allocation of space. It uses the /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.Signed-off-by Stefan Roesch <[email protected]>
Thanks,
Josef
Quoted 11 lines · Stefan Roesch
Summary:
Add new testcase for testing the new btrfs sysfs knob to change the chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.
Quoted 7 lines
Note: this test needs to force the allocation of space. It uses the /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.Signed-off-by Stefan Roesch <[email protected]>
Sorry for the late review.. I noticed some other minor issues below.
Quoted 72 lines
---
V4: - fixed indentation in common/btrfs
- Removed UUID code, which is no longer necessary (provided by helper function)
- Used new helper_get_btrfs_sysfs_attr
- Direct output to/dev/null
V3: - removed tests for system block type.
- added failure case for system block type.
- renamedstripe_sizetochunk_size
V2: - added new functions to common/btrfs and use them in the new test
-_require_btrfs_sysfs_attr- Make sure btrfs supports a sysfs setting
-_get_btrfs_sysfs_attr- Read sysfs value
-_set_btrfs_sysfs_attr- Write sysfs value
- create file system of required size with_scratch_mkfs_sized
- use shortened error message
- Remove last logging message--- common/btrfs | 73 ++++++++++++ tests/btrfs/049 | 280 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/049.out | 11 ++ 3 files changed, 364 insertions(+) create mode 100755 tests/btrfs/049 create mode 100644 tests/btrfs/049.outcommon/btrfs +39
-0@@ -445,3 +445,76 @@ _scratch_btrfs_is_zoned() [ `_zone_type ${SCRATCH_DEV}` != "none" ] && return 0 return 1 } + +# Print the content of /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _get_btrfs_sysfs_attr /mnt/scratch allocation/data/stripe_size +_get_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2 + + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + cat /sys/fs/btrfs/${uuid}/${attr} +} + +# Write "content" into /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# - content: the content of $attr +# +# Usage example: +# _set_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_set_btrfs_sysfs_attr() +{ + local mnt=$1 + shift + local attr=$1 + shift + local content="$*" +
common/btrfs _scratch_btrfs_is_zoned() · line 496
Quoted 10 lines
common/btrfs +10
-0+ if [ ! -e "$mnt" -o -z "$attr" -o -z "$content" ];then + _fail "Usage: _set_btrfs_sysfs_attr <mounted_directory> <attr> <content>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + echo "$content" > /sys/fs/btrfs/${uuid}/${attr} 2>/dev/null + if [ $? -ne 0 ]; then + _dump_err "_set_btrfs_sysfs_attr cannot write ${attr}" + fi +}
It seems that above two functions are almost identical to the generic counter parts in common/rc, the only difference is _[gs]et_sysfs_attr() takes device as first arg and uses the short dev name not filesystem UUID.
So I'm wondering if it's possible to extend _[gs]et_sysfs_attr() to support btrfs as well, i.e. we sill pass device as the first arg, and find uuid by dev if $FSTYP is btrfs, e.g.
local dname
case "$FSTYP" in
btrfs)
dname=$(findmnt -n -o UUID ${dev}) ;;
*)
dname=$(_short_dev $dev) ;;
esacQuoted 13 lines
common/btrfs +13
-0+ +# Verify if the sysfs entry in /sys/fs/btrfs/$UUID/$ATTR exists +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _require_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_require_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2
common/btrfs _scratch_btrfs_is_zoned() · line 519
Quoted 10 lines
common/btrfs +10
-0+ + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + if [[ ! -e /sys/fs/btrfs/${uuid}/${attr} ]]; then + _notrun "Btrfs does not support sysfs $attr" + fi +}
I think this could be done by extending _require_fs_sysfs() in the same way.
Quoted 44 lines
common/btrfs +1
-0+tests/btrfs/049 new file +37
-0@@ -0,0 +1,280 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2021 Facebook. All Rights Reserved. +# +# FS QA Test 250 +# +# Test the new /sys/fs/btrfs/<uuid>/allocation/<block-type>/chunk_size +# setting. This setting allows the admin to change the chunk size +# setting for the next allocation. +# +# Test 1: +# Allocate storage for all three block types (data, metadata and system) +# with the default chunk size. +# +# Test 2: +# Set a new chunk size to double the default size and allocate space +# for all new block types with the new chunk size. +# +# Test 3: +# Pick an allocation size that is used in a loop and make sure the last +# allocation cannot be partially fullfilled. +# +# Note: Variable naming uses the following convention: if a variable name +# ends in "_B" then its a byte value, if it ends in "_MB" then the +# value is in megabytes. +# +. ./common/preamble +_begin_fstest auto + +test_file="${TEST_DIR}/${seq}" +seq=`basename $0` +seqres="${RESULT_DIR}/${seq}" + +# Override the default cleanup function. +_cleanup() +{ + cd /
tests/btrfs/049 line 47
Quoted 10 lines
tests/btrfs/049 new file +10
-0+ rm -f "$test_file" +} + +# Parse a size string which is in the format "XX.XXMib". +# +# Parameters: +# - (IN) Block group type (Data, Metadata, System) +# - (INOUT) Variable to store block group size in MB +# +_parse_size_string() {
Local functions don't need the leading underscore, the same applies to other local functions in this test (except _cleanup).
tests/btrfs/049 line 48
Quoted 1 line
tests/btrfs/049 new file +1
-0+ local SIZE=$(echo "$1" | awk 'match($0, /([0-9.]+)/) { print substr($0, RSTART, RLENGTH) }')
Use $AWK_PROG instead of bare awk.
tests/btrfs/049 line 58
Quoted 10 lines
tests/btrfs/049 new file +10
-0+ eval $2="${SIZE%.*}" +} + +# Determine the size of the device in MB. +# +# Parameters: +# - (INOUT) Variable to store device size in MB +# +_device_size() { + btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid >> $seqres.full
Use $BTRFS_UTIL_PROG instead of bare 'btrfs', also please use full subcommand instead of abbr. like 'fi'.
Quoted 30 lines
tests/btrfs/049 new file +30
-0+ local SIZE=$(btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid) + _parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC + eval $1=${SIZE_ALLOC%.*} +} + +# Determine the free space of a block group in MB. +# +# Parameters: +# - (INOUT) Variable to store free space in MB +# +_free_space() { + local SIZE=$(btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid) + _parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC + _parse_size_string $(echo "${SIZE}" | awk '{print $6}') SIZE_USED + eval $1=$(expr ${SIZE_ALLOC} - ${SIZE_USED}) +} + +# Determine how much space in MB has been allocated to a block group. +# +# Parameters: +# - (IN) Block group type (Data, Metadata, System) +# - (INOUT) Variable to store block group size in MB +# +_alloc_size() { + local SIZE_STRING=$(btrfs filesystem df ${SCRATCH_MNT} -m | grep "$1" | awk '{print $3}') + _parse_size_string ${SIZE_STRING} BLOCK_GROUP_SIZE + eval $2="${BLOCK_GROUP_SIZE}" +} + +. ./common/filter
tests/btrfs/049 line 98
Quoted 10 lines
tests/btrfs/049 new file +10
-0+_supported_fs btrfs +_require_test +_require_scratch +_require_btrfs_fs_sysfs + +# Delete log file if it exists. +rm -f "${seqres}.full" + +# Make filesystem. +_require_scratch_size $((10 * 1024 * 1024)) #kB
_scratch_mkfs_sized() will check for the required size and _notrun if dev is too small, so there's no need to call above _require rule.
Quoted 80 lines
tests/btrfs/049 new file +80
-0+_scratch_mkfs_sized $((10 * 1024 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +# Check if there is sufficient sysfs support. +_require_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size +_require_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc + +# Get free space. +_free_space FREE_SPACE_MB +_device_size DEVICE_SIZE_MB + +echo "free space = ${FREE_SPACE_MB}MB" >> ${seqres}.full + +# Get chunk sizes. +echo "Capture default chunk sizes." +FIRST_DATA_CHUNK_SIZE_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size) +FIRST_METADATA_CHUNK_SIZE_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size) + +echo "Orig Data chunk size = ${FIRST_DATA_CHUNK_SIZE_B}" >> ${seqres}.full +echo "Orig Metaata chunk size = ${FIRST_METADATA_CHUNK_SIZE_B}" >> ${seqres}.full + +INIT_ALLOC_SIZE_MB=$(expr \( ${FIRST_DATA_CHUNK_SIZE_B} + ${FIRST_METADATA_CHUNK_SIZE_B} \) / 1024 / 1024) +echo "Allocation size for initial allocation = ${INIT_ALLOC_SIZE_MB}MB" >> $seqres.full + +# +# Do first allocation with the default chunk sizes for the different block +# types. +# +echo "First allocation." +_alloc_size "Data" DATA_SIZE_START_MB +_alloc_size "Metadata" METADATA_SIZE_START_MB + +echo "Block group Data alloc size = ${DATA_SIZE_START_MB}MB" >> $seqres.full +echo "Block group Metadata alloc size = ${METADATA_SIZE_START_MB}MB" >> $seqres.full + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc 1 + +_alloc_size "Data" FIRST_DATA_SIZE_MB +_alloc_size "Metadata" FIRST_METADATA_SIZE_MB + +echo "First block group Data alloc size = ${FIRST_DATA_SIZE_MB}MB" >> ${seqres}.full +echo "First block group Metadata alloc size = ${FIRST_METADATA_SIZE_MB}MB" >> ${seqres}.full + +_free_space FREE_SPACE_AFTER +echo "Free space after first allocation = ${FREE_SPACE_AFTER}MB" >> ${seqres}.full + +# +# Do allocation with the doubled chunk sizes for the different block types. +# +echo "Second allocation." +SECOND_DATA_CHUNK_SIZE_B=$(expr ${FIRST_DATA_CHUNK_SIZE_B} \* 2) +SECOND_METADATA_CHUNK_SIZE_B=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} \* 2) + +echo "Second block group Data calc alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full +echo "Second block group Metadata calc alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full + +# Stripe size is limited to 10% of device size. +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then + SECOND_DATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024) +fi +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then + SECOND_METADATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024) +fi + +echo "Second block group Data alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full +echo "Second block group Metadata alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size ${SECOND_DATA_CHUNK_SIZE_B} +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size ${SECOND_METADATA_CHUNK_SIZE_B} + +SECOND_DATA_CHUNK_SIZE_READ_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size) +SECOND_METADATA_CHUNK_SIZE_READ_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size) + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 +echo "Allocated data chunk" >> $seqres.full +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc 1 +echo "Allocated metadata chunk" >> $seqres.full + +_alloc_size "Data" SECOND_DATA_SIZE_MB
tests/btrfs/049 line 188
Quoted 10 lines
tests/btrfs/049 new file +10
-0+_alloc_size "Metadata" SECOND_METADATA_SIZE_MB +_alloc_size "System" SECOND_SYSTEM_SIZE_MB + +echo "Calculate request size so last memory allocation cannot be completely fullfilled." +_free_space FREE_SPACE_MB + +# Find request size whose space allocation cannot be completely fullfilled. +THIRD_DATA_CHUNK_SIZE_MB=$(expr 256) +until [ ${THIRD_DATA_CHUNK_SIZE_MB} -gt $(expr 7 \* 1024) ] +do
Please unify the code style as
until [ ... ]; do
<do something>
done
tests/btrfs/049 line 191
Quoted 3 lines
tests/btrfs/049 new file +3
-0+ if [ $((FREE_SPACE_MB%THIRD_DATA_CHUNK_SIZE_MB)) -ne 0 ]; then + break + fi
Mixed space and tab as indention above?
Quoted 34 lines
tests/btrfs/049 new file +34
-0+ THIRD_DATA_CHUNK_SIZE_MB=$((THIRD_DATA_CHUNK_SIZE_MB+256)) +done + +if [[ ${THIRD_DATA_CHUNK_SIZE_MB} -eq $(expr 7 \* 1024) ]]; then + _fail "Cannot find allocation size for partial block allocation." +fi + +THIRD_DATA_CHUNK_SIZE_B=$(expr ${THIRD_DATA_CHUNK_SIZE_MB} \* 1024 \* 1024) +echo "Allocation size in mb = ${THIRD_DATA_CHUNK_SIZE_MB}" >> ${seqres}.full +echo "Allocation size in bytes = ${THIRD_DATA_CHUNK_SIZE_B}" >> ${seqres}.full + +# +# Do allocation until free space is exhausted. +# +echo "Third allocation." +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size ${THIRD_DATA_CHUNK_SIZE_B} + +_free_space FREE_SPACE_MB +while [ $FREE_SPACE_MB -gt $THIRD_DATA_CHUNK_SIZE_MB ] +do + _alloc_size "Data" THIRD_DATA_SIZE_MB + _set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 + + _free_space FREE_SPACE_MB +done + +_alloc_size "Data" FOURTH_DATA_SIZE_MB + +# +# Force chunk allocation of system block tyep must fail. +# +echo "Force allocation of system block type must fail." +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/system/force_chunk_alloc 1 2>/dev/null +
tests/btrfs/049 line 235
Quoted 10 lines
tests/btrfs/049 new file +10
-0+# +# Verification of initial allocation. +# +echo "Verify first allocation." +FIRST_DATA_CHUNK_SIZE_MB=$(expr ${FIRST_DATA_CHUNK_SIZE_B} / 1024 / 1024) +FIRST_METADATA_CHUNK_SIZE_MB=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} / 1024 / 1024) + +# if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne $(expr ${FIRST_DATA_SIZE_MB}) ]]; then +if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne ${FIRST_DATA_SIZE_MB} ]]; then + _fail "tInitial data allocation not correct."
No need to call _fail, we could just echo this message and it will break the golden image and fail the test. Also, dumping the expected and actual value to stdout may help debug failure.
The same applies to other _fail calls below as well.
Thanks,
Eryu
Quoted 64 lines
tests/btrfs/049 new file +45
-0+fi + +if [[ $(expr ${FIRST_METADATA_CHUNK_SIZE_MB} + ${METADATA_SIZE_START_MB}) -ne ${FIRST_METADATA_SIZE_MB} ]]; then + _fail "Initial metadata allocation not correct." +fi + +# +# Verification of second allocation. +# +echo "Verify second allocation." +SECOND_DATA_CHUNK_SIZE_MB=$(expr ${SECOND_DATA_CHUNK_SIZE_B} / 1024 / 1024) +SECOND_METADATA_CHUNK_SIZE_MB=$(expr ${SECOND_METADATA_CHUNK_SIZE_B} / 1024 / 1024) + +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -ne ${SECOND_DATA_CHUNK_SIZE_READ_B} ]]; then + _fail "Value written to allocation/data/chunk_size and read value are different." +fi + +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -ne ${SECOND_METADATA_CHUNK_SIZE_READ_B} ]]; then + _fail "Value written to allocation/metadata/chunk_size and read value are different." +fi + +if [[ $(expr ${SECOND_DATA_CHUNK_SIZE_MB} + ${FIRST_DATA_SIZE_MB}) -ne ${SECOND_DATA_SIZE_MB} ]]; then + _fail "Data allocation after chunk size change not correct." +fi + +if [[ $(expr ${SECOND_METADATA_CHUNK_SIZE_MB} + ${FIRST_METADATA_SIZE_MB}) -ne ${SECOND_METADATA_SIZE_MB} ]]; then + _fail "Metadata allocation after chunk size change not correct." +fi + +# +# Verification of third allocation. +# +echo "Verify third allocation." +if [[ ${FREE_SPACE_MB} -gt ${THIRD_DATA_CHUNK_SIZE_MB} ]]; then + _fail "Free space after allocating over memlimit is too high." +fi + +# The + 1 is required as 1MB is always kept as free space. +if [[ $(expr ${THIRD_DATA_CHUNK_SIZE_MB} + ${THIRD_DATA_SIZE_MB} + 1) -le $(expr ${FOURTH_DATA_SIZE_MB}) ]]; then + _fail "Allocation until out of memory: last memory allocation size is not correct." +fi + +status=0 +exit +tests/btrfs/049.out new file +11
-0@@ -0,0 +1,11 @@ +QA output created by 049 +Capture default chunk sizes. +First allocation. +Second allocation. +Calculate request size so last memory allocation cannot be completely fullfilled. +Third allocation. +Force allocation of system block type must fail. +_set_btrfs_sysfs_attr cannot write allocation/system/force_chunk_alloc +Verify first allocation. +Verify second allocation. +Verify third allocation.Signature · 1 line
2.30.2
Quoted 110 lines · Eryu Guan
Quoted 11 lines · Stefan Roesch
Summary:
Add new testcase for testing the new btrfs sysfs knob to change the chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.Quoted 7 lines
Note: this test needs to force the allocation of space. It uses the /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.Signed-off-by Stefan Roesch <[email protected]>Sorry for the late review.. I noticed some other minor issues below.
Quoted 72 lines
---
V4: - fixed indentation in common/btrfs
- Removed UUID code, which is no longer necessary (provided by helper function)
- Used new helper_get_btrfs_sysfs_attr
- Direct output to/dev/null
V3: - removed tests for system block type.
- added failure case for system block type.
- renamedstripe_sizetochunk_size
V2: - added new functions to common/btrfs and use them in the new test
-_require_btrfs_sysfs_attr- Make sure btrfs supports a sysfs setting
-_get_btrfs_sysfs_attr- Read sysfs value
-_set_btrfs_sysfs_attr- Write sysfs value
- create file system of required size with_scratch_mkfs_sized
- use shortened error message
- Remove last logging message--- common/btrfs | 73 ++++++++++++ tests/btrfs/049 | 280 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/049.out | 11 ++ 3 files changed, 364 insertions(+) create mode 100755 tests/btrfs/049 create mode 100644 tests/btrfs/049.outcommon/btrfs +39
-0@@ -445,3 +445,76 @@ _scratch_btrfs_is_zoned() [ `_zone_type ${SCRATCH_DEV}` != "none" ] && return 0 return 1 } + +# Print the content of /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _get_btrfs_sysfs_attr /mnt/scratch allocation/data/stripe_size +_get_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2 + + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + cat /sys/fs/btrfs/${uuid}/${attr} +} + +# Write "content" into /sys/fs/btrfs/$UUID/$ATTR +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# - content: the content of $attr +# +# Usage example: +# _set_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_set_btrfs_sysfs_attr() +{ + local mnt=$1 + shift + local attr=$1 + shift + local content="$*" +common/btrfs _scratch_btrfs_is_zoned() · line 496
Quoted 10 lines
common/btrfs +10
-0+ if [ ! -e "$mnt" -o -z "$attr" -o -z "$content" ];then + _fail "Usage: _set_btrfs_sysfs_attr <mounted_directory> <attr> <content>" + fi + + local uuid=$(findmnt -n -o UUID ${mnt}) + echo "$content" > /sys/fs/btrfs/${uuid}/${attr} 2>/dev/null + if [ $? -ne 0 ]; then + _dump_err "_set_btrfs_sysfs_attr cannot write ${attr}" + fi +}It seems that above two functions are almost identical to the generic counter parts in common/rc, the only difference is _[gs]et_sysfs_attr() takes device as first arg and uses the short dev name not filesystem UUID.
Quoted 3 lines
So I'm wondering if it's possible to extend _[gs]et_sysfs_attr() to support btrfs as well, i.e. we sill pass device as the first arg, and find uuid by dev if $FSTYP is btrfs, e.g.
Done.
Quoted 27 lines
local dname case "$FSTYP" in btrfs) dname=$(findmnt -n -o UUID ${dev}) ;; *) dname=$(_short_dev $dev) ;; esacQuoted 18 lines
common/btrfs +18
-0+ +# Verify if the sysfs entry in /sys/fs/btrfs/$UUID/$ATTR exists +# +# All arguments are necessary, and in this order: +# - mnt: mount point name, e.g. $SCRATCH_MNT +# - attr: path name under /sys/fs/btrfs/$uuid/$attr +# +# Usage example: +# _require_btrfs_sysfs_attr /mnt/scratch allocation/data/chunk_size +_require_btrfs_sysfs_attr() +{ + local mnt=$1 + local attr=$2 + + if [ ! -e "$mnt" -o -z "$attr" ];then + _fail "Usage: _get_btrfs_sysfs_attr <mounted_directory> <attr>" + fi +
Quoted 9 lines
common/btrfs _scratch_btrfs_is_zoned() · line 519
Quoted 5 lines
common/btrfs +5
-0+ local uuid=$(findmnt -n -o UUID ${mnt}) + if [[ ! -e /sys/fs/btrfs/${uuid}/${attr} ]]; then + _notrun "Btrfs does not support sysfs $attr" + fi +}I think this could be done by extending
_require_fs_sysfs()in the same way.
Unfortunately the function require_fs_sysfs_attr does not pass in the device, it uses the environment variable $TEST_DEV directly.
Quoted 49 lines
Quoted 49 lines
common/btrfs +1
-0+tests/btrfs/049 new file +42
-0@@ -0,0 +1,280 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2021 Facebook. All Rights Reserved. +# +# FS QA Test 250 +# +# Test the new /sys/fs/btrfs/<uuid>/allocation/<block-type>/chunk_size +# setting. This setting allows the admin to change the chunk size +# setting for the next allocation. +# +# Test 1: +# Allocate storage for all three block types (data, metadata and system) +# with the default chunk size. +# +# Test 2: +# Set a new chunk size to double the default size and allocate space +# for all new block types with the new chunk size. +# +# Test 3: +# Pick an allocation size that is used in a loop and make sure the last +# allocation cannot be partially fullfilled. +# +# Note: Variable naming uses the following convention: if a variable name +# ends in "_B" then its a byte value, if it ends in "_MB" then the +# value is in megabytes. +# +. ./common/preamble +_begin_fstest auto + +test_file="${TEST_DIR}/${seq}" +seq=`basename $0` +seqres="${RESULT_DIR}/${seq}" + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -f "$test_file" +} + +# Parse a size string which is in the format "XX.XXMib". +#
Quoted 8 lines
tests/btrfs/049 line 47
Quoted 5 lines
tests/btrfs/049 new file +5
-0+# Parameters: +# - (IN) Block group type (Data, Metadata, System) +# - (INOUT) Variable to store block group size in MB +# +_parse_size_string() {Local functions don't need the leading underscore, the same applies to other local functions in this test (except
_cleanup).
Done.
Quoted 4 lines
tests/btrfs/049 line 48
Quoted 2 lines
tests/btrfs/049 new file +1
-0+ local SIZE=$(echo "$1" | awk 'match($0, /([0-9.]+)/) { print substr($0, RSTART, RLENGTH) }')Use $AWK_PROG instead of bare awk.
Done.
Quoted 15 lines
tests/btrfs/049 line 58
Quoted 11 lines
tests/btrfs/049 new file +10
-0+ eval $2="${SIZE%.*}" +} + +# Determine the size of the device in MB. +# +# Parameters: +# - (INOUT) Variable to store device size in MB +# +_device_size() { + btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid >> $seqres.fullUse $BTRFS_UTIL_PROG instead of bare 'btrfs', also please use full subcommand instead of abbr. like 'fi'.
Done.
Quoted 35 lines
Quoted 35 lines
tests/btrfs/049 new file +35
-0+ local SIZE=$(btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid) + _parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC + eval $1=${SIZE_ALLOC%.*} +} + +# Determine the free space of a block group in MB. +# +# Parameters: +# - (INOUT) Variable to store free space in MB +# +_free_space() { + local SIZE=$(btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid) + _parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC + _parse_size_string $(echo "${SIZE}" | awk '{print $6}') SIZE_USED + eval $1=$(expr ${SIZE_ALLOC} - ${SIZE_USED}) +} + +# Determine how much space in MB has been allocated to a block group. +# +# Parameters: +# - (IN) Block group type (Data, Metadata, System) +# - (INOUT) Variable to store block group size in MB +# +_alloc_size() { + local SIZE_STRING=$(btrfs filesystem df ${SCRATCH_MNT} -m | grep "$1" | awk '{print $3}') + _parse_size_string ${SIZE_STRING} BLOCK_GROUP_SIZE + eval $2="${BLOCK_GROUP_SIZE}" +} + +. ./common/filter +_supported_fs btrfs +_require_test +_require_scratch +_require_btrfs_fs_sysfs +
Quoted 9 lines
tests/btrfs/049 line 98
Quoted 5 lines
tests/btrfs/049 new file +5
-0+# Delete log file if it exists. +rm -f "${seqres}.full" + +# Make filesystem. +_require_scratch_size $((10 * 1024 * 1024)) #kB
_scratch_mkfs_sized()will check for the required size and_notrunif dev is too small, so there's no need to call above_requirerule.
Done.
Quoted 93 lines
Quoted 80 lines
tests/btrfs/049 new file +80
-0+_scratch_mkfs_sized $((10 * 1024 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +# Check if there is sufficient sysfs support. +_require_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size +_require_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc + +# Get free space. +_free_space FREE_SPACE_MB +_device_size DEVICE_SIZE_MB + +echo "free space = ${FREE_SPACE_MB}MB" >> ${seqres}.full + +# Get chunk sizes. +echo "Capture default chunk sizes." +FIRST_DATA_CHUNK_SIZE_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size) +FIRST_METADATA_CHUNK_SIZE_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size) + +echo "Orig Data chunk size = ${FIRST_DATA_CHUNK_SIZE_B}" >> ${seqres}.full +echo "Orig Metaata chunk size = ${FIRST_METADATA_CHUNK_SIZE_B}" >> ${seqres}.full + +INIT_ALLOC_SIZE_MB=$(expr \( ${FIRST_DATA_CHUNK_SIZE_B} + ${FIRST_METADATA_CHUNK_SIZE_B} \) / 1024 / 1024) +echo "Allocation size for initial allocation = ${INIT_ALLOC_SIZE_MB}MB" >> $seqres.full + +# +# Do first allocation with the default chunk sizes for the different block +# types. +# +echo "First allocation." +_alloc_size "Data" DATA_SIZE_START_MB +_alloc_size "Metadata" METADATA_SIZE_START_MB + +echo "Block group Data alloc size = ${DATA_SIZE_START_MB}MB" >> $seqres.full +echo "Block group Metadata alloc size = ${METADATA_SIZE_START_MB}MB" >> $seqres.full + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc 1 + +_alloc_size "Data" FIRST_DATA_SIZE_MB +_alloc_size "Metadata" FIRST_METADATA_SIZE_MB + +echo "First block group Data alloc size = ${FIRST_DATA_SIZE_MB}MB" >> ${seqres}.full +echo "First block group Metadata alloc size = ${FIRST_METADATA_SIZE_MB}MB" >> ${seqres}.full + +_free_space FREE_SPACE_AFTER +echo "Free space after first allocation = ${FREE_SPACE_AFTER}MB" >> ${seqres}.full + +# +# Do allocation with the doubled chunk sizes for the different block types. +# +echo "Second allocation." +SECOND_DATA_CHUNK_SIZE_B=$(expr ${FIRST_DATA_CHUNK_SIZE_B} \* 2) +SECOND_METADATA_CHUNK_SIZE_B=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} \* 2) + +echo "Second block group Data calc alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full +echo "Second block group Metadata calc alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full + +# Stripe size is limited to 10% of device size. +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then + SECOND_DATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024) +fi +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then + SECOND_METADATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024) +fi + +echo "Second block group Data alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full +echo "Second block group Metadata alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size ${SECOND_DATA_CHUNK_SIZE_B} +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size ${SECOND_METADATA_CHUNK_SIZE_B} + +SECOND_DATA_CHUNK_SIZE_READ_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size) +SECOND_METADATA_CHUNK_SIZE_READ_B=$(_get_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/chunk_size) + +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 +echo "Allocated data chunk" >> $seqres.full +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/metadata/force_chunk_alloc 1 +echo "Allocated metadata chunk" >> $seqres.full + +_alloc_size "Data" SECOND_DATA_SIZE_MBtests/btrfs/049 line 188
Quoted 10 lines
tests/btrfs/049 new file +10
-0+_alloc_size "Metadata" SECOND_METADATA_SIZE_MB +_alloc_size "System" SECOND_SYSTEM_SIZE_MB + +echo "Calculate request size so last memory allocation cannot be completely fullfilled." +_free_space FREE_SPACE_MB + +# Find request size whose space allocation cannot be completely fullfilled. +THIRD_DATA_CHUNK_SIZE_MB=$(expr 256) +until [ ${THIRD_DATA_CHUNK_SIZE_MB} -gt $(expr 7 \* 1024) ] +doPlease unify the code style as
Quoted 4 lines
until [ ... ]; do
<do something>
done
Done.
Quoted 55 lines
tests/btrfs/049 line 191
Quoted 3 lines
tests/btrfs/049 new file +3
-0+ if [ $((FREE_SPACE_MB%THIRD_DATA_CHUNK_SIZE_MB)) -ne 0 ]; then + break + fiMixed space and tab as indention above?
Quoted 34 lines
tests/btrfs/049 new file +34
-0+ THIRD_DATA_CHUNK_SIZE_MB=$((THIRD_DATA_CHUNK_SIZE_MB+256)) +done + +if [[ ${THIRD_DATA_CHUNK_SIZE_MB} -eq $(expr 7 \* 1024) ]]; then + _fail "Cannot find allocation size for partial block allocation." +fi + +THIRD_DATA_CHUNK_SIZE_B=$(expr ${THIRD_DATA_CHUNK_SIZE_MB} \* 1024 \* 1024) +echo "Allocation size in mb = ${THIRD_DATA_CHUNK_SIZE_MB}" >> ${seqres}.full +echo "Allocation size in bytes = ${THIRD_DATA_CHUNK_SIZE_B}" >> ${seqres}.full + +# +# Do allocation until free space is exhausted. +# +echo "Third allocation." +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/chunk_size ${THIRD_DATA_CHUNK_SIZE_B} + +_free_space FREE_SPACE_MB +while [ $FREE_SPACE_MB -gt $THIRD_DATA_CHUNK_SIZE_MB ] +do + _alloc_size "Data" THIRD_DATA_SIZE_MB + _set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/data/force_chunk_alloc 1 + + _free_space FREE_SPACE_MB +done + +_alloc_size "Data" FOURTH_DATA_SIZE_MB + +# +# Force chunk allocation of system block tyep must fail. +# +echo "Force allocation of system block type must fail." +_set_btrfs_sysfs_attr ${SCRATCH_MNT} allocation/system/force_chunk_alloc 1 2>/dev/null +tests/btrfs/049 line 235
Quoted 10 lines
tests/btrfs/049 new file +10
-0+# +# Verification of initial allocation. +# +echo "Verify first allocation." +FIRST_DATA_CHUNK_SIZE_MB=$(expr ${FIRST_DATA_CHUNK_SIZE_B} / 1024 / 1024) +FIRST_METADATA_CHUNK_SIZE_MB=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} / 1024 / 1024) + +# if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne $(expr ${FIRST_DATA_SIZE_MB}) ]]; then +if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne ${FIRST_DATA_SIZE_MB} ]]; then + _fail "tInitial data allocation not correct."No need to call
_fail, we could just echo this message and it will break the golden image and fail the test. Also, dumping the expected and actual value to stdout may help debug failure.
Quoted 2 lines
The same applies to other
_failcalls below as well.
Done.
Quoted 67 lines
Thanks,
EryuQuoted 64 lines
tests/btrfs/049 new file +45
-0+fi + +if [[ $(expr ${FIRST_METADATA_CHUNK_SIZE_MB} + ${METADATA_SIZE_START_MB}) -ne ${FIRST_METADATA_SIZE_MB} ]]; then + _fail "Initial metadata allocation not correct." +fi + +# +# Verification of second allocation. +# +echo "Verify second allocation." +SECOND_DATA_CHUNK_SIZE_MB=$(expr ${SECOND_DATA_CHUNK_SIZE_B} / 1024 / 1024) +SECOND_METADATA_CHUNK_SIZE_MB=$(expr ${SECOND_METADATA_CHUNK_SIZE_B} / 1024 / 1024) + +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -ne ${SECOND_DATA_CHUNK_SIZE_READ_B} ]]; then + _fail "Value written to allocation/data/chunk_size and read value are different." +fi + +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -ne ${SECOND_METADATA_CHUNK_SIZE_READ_B} ]]; then + _fail "Value written to allocation/metadata/chunk_size and read value are different." +fi + +if [[ $(expr ${SECOND_DATA_CHUNK_SIZE_MB} + ${FIRST_DATA_SIZE_MB}) -ne ${SECOND_DATA_SIZE_MB} ]]; then + _fail "Data allocation after chunk size change not correct." +fi + +if [[ $(expr ${SECOND_METADATA_CHUNK_SIZE_MB} + ${FIRST_METADATA_SIZE_MB}) -ne ${SECOND_METADATA_SIZE_MB} ]]; then + _fail "Metadata allocation after chunk size change not correct." +fi + +# +# Verification of third allocation. +# +echo "Verify third allocation." +if [[ ${FREE_SPACE_MB} -gt ${THIRD_DATA_CHUNK_SIZE_MB} ]]; then + _fail "Free space after allocating over memlimit is too high." +fi + +# The + 1 is required as 1MB is always kept as free space. +if [[ $(expr ${THIRD_DATA_CHUNK_SIZE_MB} + ${THIRD_DATA_SIZE_MB} + 1) -le $(expr ${FOURTH_DATA_SIZE_MB}) ]]; then + _fail "Allocation until out of memory: last memory allocation size is not correct." +fi + +status=0 +exit +tests/btrfs/049.out new file +11
-0@@ -0,0 +1,11 @@ +QA output created by 049 +Capture default chunk sizes. +First allocation. +Second allocation. +Calculate request size so last memory allocation cannot be completely fullfilled. +Third allocation. +Force allocation of system block type must fail. +_set_btrfs_sysfs_attr cannot write allocation/system/force_chunk_alloc +Verify first allocation. +Verify second allocation. +Verify third allocation.Signature · 1 line
2.30.2