Skip to contentSkip to conversation

: generic/020: fix MAX_ATTRVAL_SIZE values

PATCH New David Disseldorp 1 participant 5 messages Apr 13, 2022

lore
Share & tools
lore
#0 lore raw

v3 cover : generic/020: fix MAX_ATTRVAL_SIZE values

MAX_ATTRVAL_SIZE provides a per-filesystem maximum xattr value length limit. There are a few problems with the current values:
- XFS, UDF and Btrfs are incorrectly hardcoded to use 64 bytes
+ This is a regression from the larger value used prior to fff4359d
+ Btrfs's should be calculated using nodesize and xattr name length
- NFS currently uses a 64K limit
+ This may be above the server's underlying failsystem limit, so use a conservative estimate instead

The changes are split into four patches to ease review, but feel free to squash as deemed necessary.

Changes since v2:
- move _attr_get_max() helper into generic/020 and use test-local $max_attrs / $max_attrval_size variables
- minor commit message changes

 common/attr           |  75 ---------------------------
 tests/generic/020     | 116 +++++++++++++++++++++++++++++++++++++++---
 tests/generic/020.out |   2 +-
 3 files changed, 109 insertions(+), 84 deletions(-)
#1 lore raw

PATCH v3 1/4 generic/020: move MAX_ATTRS and MAX_ATTRVAL_SIZE logic

No functional change. MAX_ATTRS and MAX_ATTRVAL_SIZE are only used within generic/020, so move the logic for determining these values over there.

---
 common/attr       | 75 ----------------------------------------------
 tests/generic/020 | 76 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 75 deletions(-)
common/attr +0 -75
index dae8a1bb..cce4d1b2 100644
@@ -264,80 +264,5 @@ _getfattr()
 	return ${PIPESTATUS[0]}
 }
 
-# set maximum total attr space based on fs type
-case "$FSTYP" in
-xfs|udf|pvfs2|9p|ceph|nfs)
-	MAX_ATTRS=1000
-	;;
-ext2|ext3|ext4)
-	# For 4k blocksizes, most of the attributes have an attr_name of
-	# "attribute_NN" which is 12, and "value_NN" which is 8.
-	# But for larger block sizes, we start having extended attributes of the
-	# form "attribute_NNN" or "attribute_NNNN", and "value_NNN" and
-	# "value_NNNN", which causes the round(len(..), 4) to jump up by 4
-	# bytes.  So round_up(len(attr_name, 4)) becomes 16 instead of 12, and
-	# round_up(len(value, 4)) becomes 12 instead of 8.
-	#
-	# For 64K blocksize the calculation becomes
-	# 	max_attrs = (block_size - 32) / (16 + 12 + 16)
-	# or
-	# 	max_attrs = (block_size - 32) / 44
-	#
-	# For 4K blocksize:-
-	# 	max_attrs = (block_size - 32) / (16 + 8 + 12)
-	# or
-	# 	max_attrs = (block_size - 32) / 36
-	#
-	# Note (for 4K bs) above are exact calculations for attrs of type
-	# attribute_NN with values of type value_NN.
-	# With above calculations, for 4k blocksize max_attrs becomes 112.
-	# This means we can have few attrs of type attribute_NNN with values of
-	# type value_NNN. To avoid/handle this we need to add extra 4 bytes of
-	# headroom.
-	#
-	# So for 4K, the calculations becomes:-
-	# 	max_attrs = (block_size - 32) / (16 + 8 + 12 + 4)
-	# or
-	# 	max_attrs = (block_size - 32) / 40
-	#
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	if [ $BLOCK_SIZE -le 4096 ]; then
-		let MAX_ATTRS=$((($BLOCK_SIZE - 32) / (16 + 8 + 12 + 4)))
-	else
-		let MAX_ATTRS=$((($BLOCK_SIZE - 32) / (16 + 12 + 16 )))
-	fi
-	;;
-*)
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	# user.attribute_XXX="value.XXX" is about 32 bytes; leave some overhead
-	let MAX_ATTRS=$BLOCK_SIZE/40
-esac
-
-export MAX_ATTRS
-
-# Set max attr value size based on fs type
-case "$FSTYP" in
-xfs|udf|btrfs)
-	MAX_ATTRVAL_SIZE=64
-	;;
-pvfs2)
-	MAX_ATTRVAL_SIZE=8192
-	;;
-9p|ceph|nfs)
-	MAX_ATTRVAL_SIZE=65536
-	;;
-bcachefs)
-	MAX_ATTRVAL_SIZE=1024
-	;;
-*)
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	# leave a little overhead
-	let MAX_ATTRVAL_SIZE=$BLOCK_SIZE-256
-esac
-
-export MAX_ATTRVAL_SIZE
 # make sure this script returns success
 /bin/true
tests/generic/020 +76 -0

82 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/4 generic/020: add and use _attr_get_max()

In reply to David Disseldorp

In preparation for taking into account the attr name length when calculating $MAX_ATTRVAL_SIZE, put the current logic in a _attr_get_max() helper function and set local $max_attrval_size / $max_attrs variables instead of using export.

---
 tests/generic/020 | 178 +++++++++++++++++++++++++---------------------
 1 file changed, 95 insertions(+), 83 deletions(-)
tests/generic/020 +95 -83
index c2c285f6..cbd3f227 100755
@@ -51,81 +51,89 @@ _attr_list()
     fi
 }
 
-# set maximum total attr space based on fs type
-case "$FSTYP" in
-xfs|udf|pvfs2|9p|ceph|nfs)
-	MAX_ATTRS=1000
-	;;
-ext2|ext3|ext4)
-	# For 4k blocksizes, most of the attributes have an attr_name of
-	# "attribute_NN" which is 12, and "value_NN" which is 8.
-	# But for larger block sizes, we start having extended attributes of the
-	# form "attribute_NNN" or "attribute_NNNN", and "value_NNN" and
-	# "value_NNNN", which causes the round(len(..), 4) to jump up by 4
-	# bytes.  So round_up(len(attr_name, 4)) becomes 16 instead of 12, and
-	# round_up(len(value, 4)) becomes 12 instead of 8.
-	#
-	# For 64K blocksize the calculation becomes
-	# 	max_attrs = (block_size - 32) / (16 + 12 + 16)
-	# or
-	# 	max_attrs = (block_size - 32) / 44
-	#
-	# For 4K blocksize:-
-	# 	max_attrs = (block_size - 32) / (16 + 8 + 12)
-	# or
-	# 	max_attrs = (block_size - 32) / 36
-	#
-	# Note (for 4K bs) above are exact calculations for attrs of type
-	# attribute_NN with values of type value_NN.
-	# With above calculations, for 4k blocksize max_attrs becomes 112.
-	# This means we can have few attrs of type attribute_NNN with values of
-	# type value_NNN. To avoid/handle this we need to add extra 4 bytes of
-	# headroom.
-	#
-	# So for 4K, the calculations becomes:-
-	# 	max_attrs = (block_size - 32) / (16 + 8 + 12 + 4)
-	# or
-	# 	max_attrs = (block_size - 32) / 40
-	#
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	if [ $BLOCK_SIZE -le 4096 ]; then
-		let MAX_ATTRS=$((($BLOCK_SIZE - 32) / (16 + 8 + 12 + 4)))
-	else
-		let MAX_ATTRS=$((($BLOCK_SIZE - 32) / (16 + 12 + 16 )))
-	fi
-	;;
-*)
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	# user.attribute_XXX="value.XXX" is about 32 bytes; leave some overhead
-	let MAX_ATTRS=$BLOCK_SIZE/40
-esac
-
-export MAX_ATTRS
-
-# Set max attr value size based on fs type
-case "$FSTYP" in
-xfs|udf|btrfs)
-	MAX_ATTRVAL_SIZE=64
-	;;
-pvfs2)
-	MAX_ATTRVAL_SIZE=8192
-	;;
-9p|ceph|nfs)
-	MAX_ATTRVAL_SIZE=65536
-	;;
-bcachefs)
-	MAX_ATTRVAL_SIZE=1024
-	;;
-*)
-	# Assume max ~1 block of attrs
-	BLOCK_SIZE=`_get_block_size $TEST_DIR`
-	# leave a little overhead
-	let MAX_ATTRVAL_SIZE=$BLOCK_SIZE-256
-esac
-
-export MAX_ATTRVAL_SIZE
+# set fs-specific max_attrs and max_attrval_size values. The parameter
+# @max_attrval_namelen is required for filesystems which take into account attr
+# name lengths (including namespace prefix) when determining limits.
+_attr_get_max()
+{
+	local max_attrval_namelen="$1"
+
+	# set maximum total attr space based on fs type
+	case "$FSTYP" in
+	xfs|udf|pvfs2|9p|ceph|nfs)
+		max_attrs=1000
+		;;
+	ext2|ext3|ext4)
+		# For 4k blocksizes, most of the attributes have an attr_name of
+		# "attribute_NN" which is 12, and "value_NN" which is 8.
+		# But for larger block sizes, we start having extended
+		# attributes of the
+		# form "attribute_NNN" or "attribute_NNNN", and "value_NNN" and
+		# "value_NNNN", which causes the round(len(..), 4) to jump up by
+		# 4 bytes.  So round_up(len(attr_name, 4)) becomes 16 instead of
+		# 12, and round_up(len(value, 4)) becomes 12 instead of 8.
+		#
+		# For 64K blocksize the calculation becomes
+		# 	max_attrs = (block_size - 32) / (16 + 12 + 16)
+		# or
+		# 	max_attrs = (block_size - 32) / 44
+		#
+		# For 4K blocksize:-
+		# 	max_attrs = (block_size - 32) / (16 + 8 + 12)
+		# or
+		# 	max_attrs = (block_size - 32) / 36
+		#
+		# Note (for 4K bs) above are exact calculations for attrs of
+		# type attribute_NN with values of type value_NN.
+		# With above calculations, for 4k blocksize max_attrs becomes
+		# 112.
+		# This means we can have few attrs of type attribute_NNN with
+		# values of
+		# type value_NNN. To avoid/handle this we need to add extra 4
+		# bytes of headroom.
+		#
+		# So for 4K, the calculations becomes:-
+		# 	max_attrs = (block_size - 32) / (16 + 8 + 12 + 4)
+		# or
+		# 	max_attrs = (block_size - 32) / 40
+		#
+		# Assume max ~1 block of attrs
+		BLOCK_SIZE=`_get_block_size $TEST_DIR`
+		if [ $BLOCK_SIZE -le 4096 ]; then
+			let max_attrs=$((($BLOCK_SIZE - 32) / (16 + 8 + 12 + 4)))
+		else
+			let max_attrs=$((($BLOCK_SIZE - 32) / (16 + 12 + 16 )))
+		fi
+		;;
+	*)
+		# Assume max ~1 block of attrs
+		BLOCK_SIZE=`_get_block_size $TEST_DIR`
+		# user.attribute_XXX="value.XXX" is about 32 bytes; leave some
+		# overhead
+		let max_attrs=$BLOCK_SIZE/40
+	esac
+
+	# Set max attr value size based on fs type
+	case "$FSTYP" in
+	xfs|udf|btrfs)
+		max_attrval_size=64
+		;;
+	pvfs2)
+		max_attrval_size=8192
+		;;
+	9p|ceph|nfs)
+		max_attrval_size=65536
+		;;
+	bcachefs)
+		max_attrval_size=1024
+		;;
+	*)
+		# Assume max ~1 block of attrs
+		BLOCK_SIZE=`_get_block_size $TEST_DIR`
+		# leave a little overhead
+		let max_attrval_size=$BLOCK_SIZE-256
+	esac
+}
 
 # real QA test starts here
 _supported_fs generic
@@ -161,10 +169,13 @@ echo "*** remove attribute"
 _attr -r fish $testfile
 _attr_list $testfile
 
+max_attrval_name="long_attr"	# add 5 for "user." prefix
+_attr_get_max "$(( 5 + ${#max_attrval_name} ))"
+
 echo "*** add lots of attributes"
 v=0
 
-while [ $v -lt $MAX_ATTRS ]
+while [ $v -lt $max_attrs ]
 do
     echo -n "value_$v" | attr -s "attribute_$v" $testfile >>$seqres.full
     if [ $? -ne 0 ]
@@ -185,11 +196,11 @@ _getfattr --absolute-names $testfile \
 	/^[ 	]*$/ { next }
         { l++ } 
 	END {print "   *** " (l - 1) " attribute(s)" }' \
-    | sed s/$MAX_ATTRS/MAX_ATTRS/
+    | sed s/$max_attrs/MAX_ATTRS/
 
 echo "*** remove lots of attributes"
 v=0
-while [ $v -lt $MAX_ATTRS ]
+while [ $v -lt $max_attrs ]
 do
     if ! $ATTR_PROG -r "attribute_$v" $testfile >>$seqres.full
     then
@@ -203,12 +214,13 @@ done
 _attr_list $testfile
 
 echo "*** really long value"
-dd if=/dev/zero bs=1 count=$MAX_ATTRVAL_SIZE 2>/dev/null \
-    | _attr -s "long_attr" $testfile >/dev/null
+dd if=/dev/zero bs=1 count=$max_attrval_size 2>/dev/null \
+    | _attr -s "$max_attrval_name" $testfile >/dev/null
 
-OCTAL_SIZE=`echo "obase=8; $MAX_ATTRVAL_SIZE" | bc`
-_attr -q -g "long_attr" $testfile | od -t x1 | sed -e "s/^0*$OCTAL_SIZE$/ATTRSIZE/"    
-_attr -r "long_attr" $testfile >/dev/null
+OCTAL_SIZE=`echo "obase=8; $max_attrval_size" | bc`
+_attr -q -g "$max_attrval_name" $testfile | od -t x1 \
+    | sed -e "s/^0*$OCTAL_SIZE$/ATTRSIZE/"
+_attr -r "$max_attrval_name" $testfile >/dev/null
 
 echo "*** set/get/remove really long names (expect failure)"
 short="XXXXXXXXXX"
Signature · 1 line
2.34.1
#3 lore raw

PATCH v3 3/4 generic/020: fix max_attrval_size for XFS, UDF, Btrfs and NFS

In reply to David Disseldorp

As found by Dave Chinner, fff4359d ("020: make this xattr test generic") unintentionally changed the long attribute value length from 100K to 64 *bytes* for XFS, UDF and Btrfs.
Update XFS and UDF to use a 64K $max_attrval_size value. For Btrfs, use the nodesize, xattr length and tree entry overhead sizes to calculate the maximum.
NFS doesn't provide a way to find out the $max_attrval_size for the underlying filesystem on the server, so just use a rough 1K limit.

---
 tests/generic/020 | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
tests/generic/020 +16 -4
index cbd3f227..f50a4f87 100755
@@ -113,20 +113,32 @@ _attr_get_max()
 		let max_attrs=$BLOCK_SIZE/40
 	esac
 
-	# Set max attr value size based on fs type
+	# Set max attr value size in bytes based on fs type
 	case "$FSTYP" in
-	xfs|udf|btrfs)
-		max_attrval_size=64
+	btrfs)
+		_require_btrfs_command inspect-internal dump-super
+		local ns=$($BTRFS_UTIL_PROG inspect-internal dump-super \
+				$TEST_DEV | sed -n 's/nodesize\s*\(.*\)/\1/p')
+		[ -n "$ns" ] || _fail "failed to obtain nodesize"
+		# max == nodesize - sizeof(struct btrfs_header)
+		#		- sizeof(struct btrfs_item)
+		#		- sizeof(struct btrfs_dir_item) - name_len
+		max_attrval_size=$(( $ns - 156 - $max_attrval_namelen ))
 		;;
 	pvfs2)
 		max_attrval_size=8192
 		;;
-	9p|ceph|nfs)
+	xfs|udf|9p|ceph)
 		max_attrval_size=65536
 		;;
 	bcachefs)
 		max_attrval_size=1024
 		;;
+	nfs)
+		# NFS doesn't provide a way to find out the max_attrval_size for
+		# the underlying filesystem, so just use the lowest value above.
+		max_attrval_size=1024
+		;;
 	*)
 		# Assume max ~1 block of attrs
 		BLOCK_SIZE=`_get_block_size $TEST_DIR`
Signature · 1 line
2.34.1
#4 lore raw

PATCH v3 4/4 generic/020: fix max_attrval_size output filter

In reply to David Disseldorp

The current attr -g "$max_attrval_name" output filter is broken if $max_attrval_size isn't 16-byte aligned, due to od's duplicate suppression behaviour.
Fix it by having od only dump one byte per line.

---
 tests/generic/020     | 2 +-
 tests/generic/020.out | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tests/generic/020 +1 -1
index f50a4f87..d8648e96 100755
@@ -230,7 +230,7 @@ dd if=/dev/zero bs=1 count=$max_attrval_size 2>/dev/null \
     | _attr -s "$max_attrval_name" $testfile >/dev/null
 
 OCTAL_SIZE=`echo "obase=8; $max_attrval_size" | bc`
-_attr -q -g "$max_attrval_name" $testfile | od -t x1 \
+_attr -q -g "$max_attrval_name" $testfile | od -w1 -t x1 \
     | sed -e "s/^0*$OCTAL_SIZE$/ATTRSIZE/"
 _attr -r "$max_attrval_name" $testfile >/dev/null
 
tests/generic/020.out +1 -1
index 7e3e65bd..0dc5e09f 100644
@@ -47,7 +47,7 @@ user.snrub="fish2\012"
 user.snrub="fish2\012"
 
 *** really long value
-0000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+0000000 00
 *
 ATTRSIZE
 *** set/get/remove really long names (expect failure)
Signature · 1 line
2.34.1