shared/298: Wire btrfs support in get_free_sectors
PATCH v2 1/1 shared/298: Wire btrfs support in get_free_sectors
Add support for btrfs in shared/298. Achieve this by introducing 2 new awk scripts that parse relevant btrfs structures and print holes. Additionally modify the test to create larger - 3gb filesystem in the case of btrfs. This is needed so that distinct block groups are used for data and metadata.
---
V2:
* Changed the way args are passed to mkfs.btrfs to preserve existing options, yet override data/metadata profile settings
parse-dev-tree.awk | 47 +++++++++++++++++++ parse-extent-tree.awk | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/shared/298 | 36 +++++++++++++-- 3 files changed, 205 insertions(+), 3 deletions(-) create mode 100755 parse-dev-tree.awk create mode 100755 parse-extent-tree.awk
parse-dev-tree.awk new file +47 -0
@@ -0,0 +1,47 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' device tree for holes, required parameters passed on command +# line: +# * spb - how many bytes per sector, this is used so that the numbers +# returned by the script are in sectors. +# * devsize - size of the device in bytes, used to output the final +# hole (if any) + +function get_extent_size(line, tmp) { + split(line, tmp) + return tmp[6] +} + +function get_extent_offset(line, tmp) { + split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] +} + +BEGIN { + dev_extent_match="^.item [0-9]* key \\([0-9]* DEV_EXTENT [0-9]*\\).*" + dev_extent_len_match="^\\s*chunk_objectid [0-9]* chunk_offset [0-9]* length [0-9]*$" +} + +{ + if (match($0,dev_extent_match)) { + extent_offset = get_extent_offset($0) + if (prev_extent_end) { + hole_size = extent_offset - prev_extent_end + if (hole_size > 0) { + print prev_extent_end / spb, int((extent_offset - 1) / spb) + } + } + } else if (match($0, dev_extent_len_match)) { + extent_size = get_extent_size($0) + prev_extent_end = extent_offset + extent_size + } +} + +END { + if (prev_extent_end) { + print prev_extent_end / spb, int((devsize - 1) / spb) + } +} +
parse-extent-tree.awk new file +125 -0
125 lines not on this page · raw message
tests/shared/298 +33 -3
@@ -15,14 +15,24 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common/rc -_supported_fs ext4 xfs +_supported_fs ext4 xfs btrfs _supported_os Linux _require_test _require_loop _require_fstrim _require_xfs_io_command "fiemap" -_require_fs_space $TEST_DIR 307200 +if [ "$FSTYP" = "btrfs" ]; then + # 3g for btrfs to have distinct bgs + _require_fs_space $TEST_DIR 3145728 + fssize=3000 +else + _require_fs_space $TEST_DIR 307200 + fssize=300 +fi + [ "$FSTYP" = "ext4" ] && _require_dumpe2fs +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-super +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-tree _cleanup() {
@@ -61,6 +71,25 @@ get_free_sectors() $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \ '{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }' ;; + btrfs) + local device_size=$($BTRFS_UTIL_PROG filesystem show --raw $loop_mnt 2>&1 \ + | sed -n "s/^.*size \([0-9]*\).*$/\1/p") + + local nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $img_file \ + | sed -n 's/nodesize\s*\(.*\)/\1/p') + + # Get holes within block groups + local btrfs_free=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t extent $img_file \ + | $AWK_PROG -v spb=512 -v nodesize=$nodesize -f parse-extent-tree.awk) + + btrfs_free+="\n" + + # Get holes within unallocated space on disk + btrfs_free+=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t dev $img_file \ + | $AWK_PROG -v spb=512 -v devsize=$device_size -f parse-dev-tree.awk) + + echo -e "$btrfs_free" + ;; esac }
@@ -105,7 +134,7 @@ here=`pwd` tmp=`mktemp -d` img_file=$TEST_DIR/$$.fs -dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null +dd if=/dev/zero of=$img_file bs=1M count=$fssize &> /dev/null loop_dev=$(_create_loop_device $img_file) loop_mnt=$tmp/loop_mnt
@@ -118,6 +147,7 @@ merged_sectors="$tmp/merged_free_sectors" mkdir $loop_mnt [ "$FSTYP" = "xfs" ] && MKFS_OPTIONS="-f $MKFS_OPTIONS" +[ "$FSTYP" = "btrfs" ] && MKFS_OPTIONS="$MKFS_OPTIONS -f -dsingle -msingle" _mkfs_dev $loop_dev _mount $loop_dev $loop_mnt
Signature · 1 line
2.7.4
Quoted 7 lines · Nikolay Borisov
Add support for btrfs in shared/298. Achieve this by introducing 2 new awk scripts that parse relevant btrfs structures and print holes. Additionally modify the test to create larger - 3gb filesystem in the case of btrfs. This is needed so that distinct block groups are used for data and metadata.
Signed-off-by Nikolay Borisov <[email protected]>
Ping, I would really like to get this ASAP to get coverage for btrfs.
Quoted 272 lines
---
V2:
* Changed the way args are passed to mkfs.btrfs to preserve existing options, yet override data/metadata profile settingsparse-dev-tree.awk | 47 +++++++++++++++++++ parse-extent-tree.awk | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/shared/298 | 36 +++++++++++++-- 3 files changed, 205 insertions(+), 3 deletions(-) create mode 100755 parse-dev-tree.awk create mode 100755 parse-extent-tree.awkparse-dev-tree.awk new file +47
-0@@ -0,0 +1,47 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' device tree for holes, required parameters passed on command +# line: +# * spb - how many bytes per sector, this is used so that the numbers +# returned by the script are in sectors. +# * devsize - size of the device in bytes, used to output the final +# hole (if any) + +function get_extent_size(line, tmp) { + split(line, tmp) + return tmp[6] +} + +function get_extent_offset(line, tmp) { + split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] +} + +BEGIN { + dev_extent_match="^.item [0-9]* key \\([0-9]* DEV_EXTENT [0-9]*\\).*" + dev_extent_len_match="^\\s*chunk_objectid [0-9]* chunk_offset [0-9]* length [0-9]*$" +} + +{ + if (match($0,dev_extent_match)) { + extent_offset = get_extent_offset($0) + if (prev_extent_end) { + hole_size = extent_offset - prev_extent_end + if (hole_size > 0) { + print prev_extent_end / spb, int((extent_offset - 1) / spb) + } + } + } else if (match($0, dev_extent_len_match)) { + extent_size = get_extent_size($0) + prev_extent_end = extent_offset + extent_size + } +} + +END { + if (prev_extent_end) { + print prev_extent_end / spb, int((devsize - 1) / spb) + } +} +parse-extent-tree.awk new file +125
-0@@ -0,0 +1,125 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' extent tree for holes, required parameters passed on command +# line: +# * spb - how many bytes per sector, this is used so that the numbers +# returned by the script are in sectors. +# * nodesize - size of metadata extents, used for internal calculations + +function get_extent_size(line, tmp) { + if (line ~ data_match || line ~ bg_match) { + split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] + } else if (line ~ metadata_match) { + return nodesize + } +} + +function get_extent_offset(line, tmp) { + split(line, tmp) + gsub(/\(/,"",tmp[4]) + return tmp[4] +} + +function print_array( base_offset, bg_line) +{ + if (match(lines[0], bg_match)) { + #we don't have an extent at the beginning of of blockgroup, so we + #have a hole between blockgroup offset and first extent offset + bg_line = lines[0] + prev_size=0 + prev_offset=get_extent_offset(bg_line) + delete lines[0] + } else { + #we have an extent at the beginning of block group, so initialize + #the prev_* vars correctly + bg_line = lines[1] + prev_size = get_extent_size(lines[0]) + prev_offset = get_extent_offset(lines[0]) + delete lines[1] + delete lines[0] + } + + bg_offset=get_extent_offset(bg_line) + bgend=bg_offset + get_extent_size(bg_line) + + for (i in lines) { + cur_size = get_extent_size(lines[i]) + cur_offset = get_extent_offset(lines[i]) + if (cur_offset != prev_offset + prev_size) + print int((prev_size + prev_offset) / spb), int((cur_offset-1) / spb) + prev_size = cur_size + prev_offset = cur_offset + } + + print int((prev_size + prev_offset) / spb), int((bgend-1) / spb) + total_printed++ + delete lines +} + +BEGIN { + loi_match="^.item [0-9]* key \\([0-9]* (BLOCK_GROUP_ITEM|METADATA_ITEM|EXTENT_ITEM) [0-9]*\\).*" + metadata_match="^.item [0-9]* key \\([0-9]* METADATA_ITEM [0-9]*\\).*" + data_match="^.item [0-9]* key \\([0-9]* EXTENT_ITEM [0-9]*\\).*" + bg_match="^.item [0-9]* key \\([0-9]* BLOCK_GROUP_ITEM [0-9]*\\).*" + node_match="^node.*$" + leaf_match="^leaf [0-9]* flags" + line_count=0 + total_printed=0 + skip_lines=0 +} + +{ + #skip lines not belonging to a leaf + if (match($0,node_match)) { + skip_lines=1 + } else if (match($0, leaf_match)) { + skip_lines=0 + } + + if (!match($0,loi_match) || skip_lines == 1) next; + + #we have a line of interest, we need to parse it. First check if there is + #anything in the array + if (line_count==0) { + lines[line_count++]=$0; + } else { + prev_line=lines[line_count-1] + split(prev_line, prev_line_fields) + prev_objectid=prev_line_fields[4] + objectid=$4 + + if (objectid == prev_objectid && match($0, bg_match)) { + if (total_printed>0) { + #We are adding a BG after we have added its first extent + #previously, consider this a record ending event and just print + #the array + + delete lines[line_count-1] + print_array() + #we now start a new array with current and previous lines + line_count=0 + lines[line_count++]=prev_line + lines[line_count++]=$0 + } else { + #first 2 added lines are EXTENT and BG that match, in this case + #just add them + lines[line_count++]=$0 + + } + } else if (match($0, bg_match)) { + #ordinary end of record + print_array() + line_count=0 + lines[line_count++]=$0 + } else { + lines[line_count++]=$0 + } + } +} + +END { + print_array() +}tests/shared/298 +33
-3@@ -15,14 +15,24 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common/rc -_supported_fs ext4 xfs +_supported_fs ext4 xfs btrfs _supported_os Linux _require_test _require_loop _require_fstrim _require_xfs_io_command "fiemap" -_require_fs_space $TEST_DIR 307200 +if [ "$FSTYP" = "btrfs" ]; then + # 3g for btrfs to have distinct bgs + _require_fs_space $TEST_DIR 3145728 + fssize=3000 +else + _require_fs_space $TEST_DIR 307200 + fssize=300 +fi + [ "$FSTYP" = "ext4" ] && _require_dumpe2fs +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-super +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-tree _cleanup() {@@ -61,6 +71,25 @@ get_free_sectors() $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \ '{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }' ;; + btrfs) + local device_size=$($BTRFS_UTIL_PROG filesystem show --raw $loop_mnt 2>&1 \ + | sed -n "s/^.*size \([0-9]*\).*$/\1/p") + + local nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $img_file \ + | sed -n 's/nodesize\s*\(.*\)/\1/p') + + # Get holes within block groups + local btrfs_free=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t extent $img_file \ + | $AWK_PROG -v spb=512 -v nodesize=$nodesize -f parse-extent-tree.awk) + + btrfs_free+="\n" + + # Get holes within unallocated space on disk + btrfs_free+=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t dev $img_file \ + | $AWK_PROG -v spb=512 -v devsize=$device_size -f parse-dev-tree.awk) + + echo -e "$btrfs_free" + ;; esac }@@ -105,7 +134,7 @@ here=`pwd` tmp=`mktemp -d` img_file=$TEST_DIR/$$.fs -dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null +dd if=/dev/zero of=$img_file bs=1M count=$fssize &> /dev/null loop_dev=$(_create_loop_device $img_file) loop_mnt=$tmp/loop_mnt@@ -118,6 +147,7 @@ merged_sectors="$tmp/merged_free_sectors" mkdir $loop_mnt [ "$FSTYP" = "xfs" ] && MKFS_OPTIONS="-f $MKFS_OPTIONS" +[ "$FSTYP" = "btrfs" ] && MKFS_OPTIONS="$MKFS_OPTIONS -f -dsingle -msingle" _mkfs_dev $loop_dev _mount $loop_dev $loop_mnt
Quoted 7 lines · Nikolay Borisov
Add support for btrfs in shared/298. Achieve this by introducing 2 new awk scripts that parse relevant btrfs structures and print holes. Additionally modify the test to create larger - 3gb filesystem in the case of btrfs. This is needed so that distinct block groups are used for data and metadata.
Signed-off-by Nikolay Borisov <[email protected]>
Sorry for the late review.. I find that parsing btrfs extent and dev tree is very btrfs-specific, it'd be great if btrfs folks could help review the two awk scripts as well!
Quoted 8 lines
---
V2:
* Changed the way args are passed to mkfs.btrfs to preserve existing options, yet override data/metadata profile settingsparse-dev-tree.awk | 47 +++++++++++++++++++ parse-extent-tree.awk | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++
I'd prefer placing these files in src dir, instead of dumping them in top dir directly.
Quoted 11 lines
tests/shared/298 | 36 +++++++++++++-- 3 files changed, 205 insertions(+), 3 deletions(-) create mode 100755 parse-dev-tree.awk create mode 100755 parse-extent-tree.awkparse-dev-tree.awk new file +0
-0
parse-dev-tree.awk line 4
Quoted 4 lines
parse-dev-tree.awk new file +4
-0@@ -0,0 +1,47 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' device tree for holes, required parameters passed on command
I find this description not very useful, would you please describe the expected output and format as well?
parse-dev-tree.awk line 6
Quoted 2 lines
parse-dev-tree.awk new file +2
-0+# line: +# * spb - how many bytes per sector, this is used so that the numbers
^^^ This is misleading, in shared/298 spb represents "sector per block", but here it's really sector size.
parse-dev-tree.awk line 8
Quoted 2 lines
parse-dev-tree.awk new file +2
-0+# returned by the script are in sectors. +# * devsize - size of the device in bytes, used to output the final
This line is not aligned with above line, it contains leading tab.
parse-dev-tree.awk line 11
Quoted 3 lines
parse-dev-tree.awk new file +3
-0+# hole (if any) + +function get_extent_size(line, tmp) {
Would you please document the expected intput and output in comment as well? So it's easier to review.
Also, is the 'tmp' argument really needed?
parse-dev-tree.awk line 16
Quoted 5 lines
parse-dev-tree.awk new file +5
-0+ split(line, tmp) + return tmp[6] +} + +function get_extent_offset(line, tmp) {
Same here.
Quoted 37 lines
parse-dev-tree.awk new file +31
-0+ split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] +} + +BEGIN { + dev_extent_match="^.item [0-9]* key \\([0-9]* DEV_EXTENT [0-9]*\\).*" + dev_extent_len_match="^\\s*chunk_objectid [0-9]* chunk_offset [0-9]* length [0-9]*$" +} + +{ + if (match($0,dev_extent_match)) { + extent_offset = get_extent_offset($0) + if (prev_extent_end) { + hole_size = extent_offset - prev_extent_end + if (hole_size > 0) { + print prev_extent_end / spb, int((extent_offset - 1) / spb) + } + } + } else if (match($0, dev_extent_len_match)) { + extent_size = get_extent_size($0) + prev_extent_end = extent_offset + extent_size + } +} + +END { + if (prev_extent_end) { + print prev_extent_end / spb, int((devsize - 1) / spb) + } +} +parse-extent-tree.awk new file +0
-0
parse-extent-tree.awk line 4
Quoted 4 lines
parse-extent-tree.awk new file +4
-0@@ -0,0 +1,125 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' extent tree for holes, required parameters passed on command
Same here, please provide more details.
parse-extent-tree.awk line 6
Quoted 2 lines
parse-extent-tree.awk new file +2
-0+# line: +# * spb - how many bytes per sector, this is used so that the numbers
And replace 'spb' with a more proper variable name.
parse-extent-tree.awk line 8
Quoted 2 lines
parse-extent-tree.awk new file +2
-0+# returned by the script are in sectors. +# * nodesize - size of metadata extents, used for internal calculations
Indention issue too.
Quoted 8 lines
parse-extent-tree.awk new file +8
-0+ +function get_extent_size(line, tmp) { + if (line ~ data_match || line ~ bg_match) { + split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] + } else if (line ~ metadata_match) { + return nodesize
parse-extent-tree.awk line 26
Quoted 10 lines
parse-extent-tree.awk new file +10
-0+ } +} + +function get_extent_offset(line, tmp) { + split(line, tmp) + gsub(/\(/,"",tmp[4]) + return tmp[4] +} + +function print_array( base_offset, bg_line)
Document the expected input and output of these functions too.
And why there're so many spaces before 'base_offset' argument?
parse-extent-tree.awk line 29
Quoted 3 lines
parse-extent-tree.awk new file +3
-0+{ + if (match(lines[0], bg_match)) { + #we don't have an extent at the beginning of of blockgroup, so we
Add a space after '#' for comments.
Quoted 139 lines
parse-extent-tree.awk new file +96
-0+ #have a hole between blockgroup offset and first extent offset + bg_line = lines[0] + prev_size=0 + prev_offset=get_extent_offset(bg_line) + delete lines[0] + } else { + #we have an extent at the beginning of block group, so initialize + #the prev_* vars correctly + bg_line = lines[1] + prev_size = get_extent_size(lines[0]) + prev_offset = get_extent_offset(lines[0]) + delete lines[1] + delete lines[0] + } + + bg_offset=get_extent_offset(bg_line) + bgend=bg_offset + get_extent_size(bg_line) + + for (i in lines) { + cur_size = get_extent_size(lines[i]) + cur_offset = get_extent_offset(lines[i]) + if (cur_offset != prev_offset + prev_size) + print int((prev_size + prev_offset) / spb), int((cur_offset-1) / spb) + prev_size = cur_size + prev_offset = cur_offset + } + + print int((prev_size + prev_offset) / spb), int((bgend-1) / spb) + total_printed++ + delete lines +} + +BEGIN { + loi_match="^.item [0-9]* key \\([0-9]* (BLOCK_GROUP_ITEM|METADATA_ITEM|EXTENT_ITEM) [0-9]*\\).*" + metadata_match="^.item [0-9]* key \\([0-9]* METADATA_ITEM [0-9]*\\).*" + data_match="^.item [0-9]* key \\([0-9]* EXTENT_ITEM [0-9]*\\).*" + bg_match="^.item [0-9]* key \\([0-9]* BLOCK_GROUP_ITEM [0-9]*\\).*" + node_match="^node.*$" + leaf_match="^leaf [0-9]* flags" + line_count=0 + total_printed=0 + skip_lines=0 +} + +{ + #skip lines not belonging to a leaf + if (match($0,node_match)) { + skip_lines=1 + } else if (match($0, leaf_match)) { + skip_lines=0 + } + + if (!match($0,loi_match) || skip_lines == 1) next; + + #we have a line of interest, we need to parse it. First check if there is + #anything in the array + if (line_count==0) { + lines[line_count++]=$0; + } else { + prev_line=lines[line_count-1] + split(prev_line, prev_line_fields) + prev_objectid=prev_line_fields[4] + objectid=$4 + + if (objectid == prev_objectid && match($0, bg_match)) { + if (total_printed>0) { + #We are adding a BG after we have added its first extent + #previously, consider this a record ending event and just print + #the array + + delete lines[line_count-1] + print_array() + #we now start a new array with current and previous lines + line_count=0 + lines[line_count++]=prev_line + lines[line_count++]=$0 + } else { + #first 2 added lines are EXTENT and BG that match, in this case + #just add them + lines[line_count++]=$0 + + } + } else if (match($0, bg_match)) { + #ordinary end of record + print_array() + line_count=0 + lines[line_count++]=$0 + } else { + lines[line_count++]=$0 + } + } +} + +END { + print_array() +}tests/shared/298 +20
-2@@ -15,14 +15,24 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common/rc -_supported_fs ext4 xfs +_supported_fs ext4 xfs btrfs _supported_os Linux _require_test _require_loop _require_fstrim _require_xfs_io_command "fiemap" -_require_fs_space $TEST_DIR 307200 +if [ "$FSTYP" = "btrfs" ]; then + # 3g for btrfs to have distinct bgs + _require_fs_space $TEST_DIR 3145728 + fssize=3000 +else + _require_fs_space $TEST_DIR 307200 + fssize=300 +fi + [ "$FSTYP" = "ext4" ] && _require_dumpe2fs +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-super +[ "$FSTYP" = "btrfs" ] && _require_btrfs_command inspect-internal dump-tree _cleanup() {@@ -61,6 +71,25 @@ get_free_sectors() $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \ '{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }' ;; + btrfs) + local device_size=$($BTRFS_UTIL_PROG filesystem show --raw $loop_mnt 2>&1 \ + | sed -n "s/^.*size \([0-9]*\).*$/\1/p") + + local nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $img_file \ + | sed -n 's/nodesize\s*\(.*\)/\1/p') + + # Get holes within block groups
tests/shared/298 get_free_sectors() · line 91
Quoted 10 lines
tests/shared/298 +10
-0+ local btrfs_free=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t extent $img_file \ + | $AWK_PROG -v spb=512 -v nodesize=$nodesize -f parse-extent-tree.awk) + + btrfs_free+="\n" + + # Get holes within unallocated space on disk + btrfs_free+=$($BTRFS_UTIL_PROG inspect-internal dump-tree -t dev $img_file \ + | $AWK_PROG -v spb=512 -v devsize=$device_size -f parse-dev-tree.awk) + + echo -e "$btrfs_free"
I don't think the "$btrfs_free" variable is needed, we could just let awk print the result out.
Thanks,
Eryu
Quoted 24 lines
tests/shared/298 +3
-1+ ;; esac }@@ -105,7 +134,7 @@ here=`pwd` tmp=`mktemp -d` img_file=$TEST_DIR/$$.fs -dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null +dd if=/dev/zero of=$img_file bs=1M count=$fssize &> /dev/null loop_dev=$(_create_loop_device $img_file) loop_mnt=$tmp/loop_mnt@@ -118,6 +147,7 @@ merged_sectors="$tmp/merged_free_sectors" mkdir $loop_mnt [ "$FSTYP" = "xfs" ] && MKFS_OPTIONS="-f $MKFS_OPTIONS" +[ "$FSTYP" = "btrfs" ] && MKFS_OPTIONS="$MKFS_OPTIONS -f -dsingle -msingle" _mkfs_dev $loop_dev _mount $loop_dev $loop_mntSignature · 1 line
2.7.4
Quoted 19 lines · Eryu Guan
Quoted 7 lines · Nikolay Borisov
Add support for btrfs in shared/298. Achieve this by introducing 2 new awk scripts that parse relevant btrfs structures and print holes. Additionally modify the test to create larger - 3gb filesystem in the case of btrfs. This is needed so that distinct block groups are used for data and metadata.
Signed-off-by Nikolay Borisov <[email protected]>Sorry for the late review.. I find that parsing btrfs extent and dev tree is very btrfs-specific, it'd be great if btrfs folks could help review the two awk scripts as well!
Quoted 6 lines
---
V2:
* Changed the way args are passed to mkfs.btrfs to preserve existing options, yet override data/metadata profile settings
Quoted 5 lines
Quoted 2 lines
parse-dev-tree.awk | 47 +++++++++++++++++++ parse-extent-tree.awk | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++I'd prefer placing these files in src dir, instead of dumping them in top dir directly.
Ok
Quoted 6 lines
Quoted 6 lines
tests/shared/298 | 36 +++++++++++++-- 3 files changed, 205 insertions(+), 3 deletions(-) create mode 100755 parse-dev-tree.awk create mode 100755 parse-extent-tree.awk
Quoted 13 lines
parse-dev-tree.awk line 4
Quoted 10 lines
parse-dev-tree.awk new file +4
-0@@ -0,0 +1,47 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' device tree for holes, required parameters passed on commandI find this description not very useful, would you please describe the expected output and format as well?
Ok will make it a bit more verbose
Quoted 5 lines
parse-dev-tree.awk line 6
Quoted 3 lines
parse-dev-tree.awk new file +2
-0+# line: +# * spb - how many bytes per sector, this is used so that the numbers^^^ This is misleading, in shared/298 spb represents "sector per block", but here it's really sector size.
Yeah probably should rename it.
Quoted 6 lines
parse-dev-tree.awk line 8
Quoted 3 lines
parse-dev-tree.awk new file +2
-0+# returned by the script are in sectors. +# * devsize - size of the device in bytes, used to output the finalThis line is not aligned with above line, it contains leading tab.
Quoted 8 lines
parse-dev-tree.awk line 11
Quoted 3 lines
parse-dev-tree.awk new file +3
-0+# hole (if any) + +function get_extent_size(line, tmp) {Would you please document the expected intput and output in comment as well? So it's easier to review.
Also, is the 'tmp' argument really needed?
tmp in this case is really a function-local variable as per:
https://www.gnu.org/software/gawk/manual/html_node/Variable-Scope.html :
"Unlike in many languages, there is no way to make a variable local to a { … } block in awk, but you can make a variable local to a function. It is good practice to do so whenever a variable is needed only in that function."
Quoted 84 lines
parse-dev-tree.awk line 16
Quoted 6 lines
parse-dev-tree.awk new file +5
-0+ split(line, tmp) + return tmp[6] +} + +function get_extent_offset(line, tmp) {Same here.
Quoted 37 lines
parse-dev-tree.awk new file +31
-0+ split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] +} + +BEGIN { + dev_extent_match="^.item [0-9]* key \\([0-9]* DEV_EXTENT [0-9]*\\).*" + dev_extent_len_match="^\\s*chunk_objectid [0-9]* chunk_offset [0-9]* length [0-9]*$" +} + +{ + if (match($0,dev_extent_match)) { + extent_offset = get_extent_offset($0) + if (prev_extent_end) { + hole_size = extent_offset - prev_extent_end + if (hole_size > 0) { + print prev_extent_end / spb, int((extent_offset - 1) / spb) + } + } + } else if (match($0, dev_extent_len_match)) { + extent_size = get_extent_size($0) + prev_extent_end = extent_offset + extent_size + } +} + +END { + if (prev_extent_end) { + print prev_extent_end / spb, int((devsize - 1) / spb) + } +} +parse-extent-tree.awk new file +0
-0parse-extent-tree.awk line 4
Quoted 4 lines
parse-extent-tree.awk new file +4
-0@@ -0,0 +1,125 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2019 Nikolay Borisov, SUSE LLC. All Rights Reserved. +# +# Parses btrfs' extent tree for holes, required parameters passed on commandSame here, please provide more details.
parse-extent-tree.awk line 6
Quoted 2 lines
parse-extent-tree.awk new file +2
-0+# line: +# * spb - how many bytes per sector, this is used so that the numbersAnd replace 'spb' with a more proper variable name.
parse-extent-tree.awk line 8
Quoted 2 lines
parse-extent-tree.awk new file +2
-0+# returned by the script are in sectors. +# * nodesize - size of metadata extents, used for internal calculationsIndention issue too.
Quoted 8 lines
parse-extent-tree.awk new file +8
-0+ +function get_extent_size(line, tmp) { + if (line ~ data_match || line ~ bg_match) { + split(line, tmp) + gsub(/\)/,"", tmp[6]) + return tmp[6] + } else if (line ~ metadata_match) { + return nodesizeparse-extent-tree.awk line 26
Quoted 10 lines
parse-extent-tree.awk new file +10
-0+ } +} + +function get_extent_offset(line, tmp) { + split(line, tmp) + gsub(/\(/,"",tmp[4]) + return tmp[4] +} + +function print_array( base_offset, bg_line)Document the expected input and output of these functions too.
Quoted 1 line
And why there're so many spaces before '
base_offset' argument?
Again, those 2 are used as local variables as per early cited documentation.
Quoted 6 lines
parse-extent-tree.awk line 29
Quoted 4 lines
parse-extent-tree.awk new file +3
-0+{ + if (match(lines[0], bg_match)) { + #we don't have an extent at the beginning of of blockgroup, so weAdd a space after '#' for comments.
Ok
<snip>
Quoted 2 lines
Quoted 1 line