Skip to contentSkip to conversation

generic/233,270: unlimit the max locked memory size for io_uring

PATCH Superseded Zorro Lang 3 participants 6 messages Jan 27, 2021 – Feb 7, 2021 1 message not in the archive

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Zorro Lang

author
#0 lore raw

PATCH 1/1 generic/233,270: unlimit the max locked memory size for io_uring

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

---
 tests/generic/233 | 8 ++++++++
 tests/generic/270 | 8 ++++++++
 2 files changed, 16 insertions(+)
tests/generic/233 +8 -0
index 7eda5774..342690c2 100755
@@ -43,6 +43,13 @@ _fsstress()
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
 -n $count -d $out -p 7`
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited
 	echo "fsstress $args" >> $seqres.full
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $seqres.full | _filter_num
 	then
@@ -50,6 +57,7 @@ _fsstress()
 		cat $tmp.out | tee -a $seqres.full
 		status=1
 	fi
+	ulimit -l $lmem
 }
 
 # real QA test starts here
tests/generic/270 +8 -0
index 3d8656d4..bd52d56e 100755
@@ -37,6 +37,13 @@ _workout()
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
 	$SETCAP_PROG cap_chown=epi  $tmp.fsstress.bin
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited
 	(su $qa_user -c "$tmp.fsstress.bin $args" &) > /dev/null 2>&1
 
 	echo "Run dd writers in parallel"
@@ -50,6 +57,7 @@ _workout()
 	done
 
 	$KILLALL_PROG -w $tmp.fsstress.bin
+	ulimit -l $lmem
 }
 
 # real QA test starts here
Signature · 1 line
2.29.2
#1 lore raw
Quoted 21 lines · Zorro Lang

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

---
 tests/generic/233 | 8 ++++++++
 tests/generic/270 | 8 ++++++++
 2 files changed, 16 insertions(+)
tests/generic/233 +0 -0
index 7eda5774..342690c2 100755

tests/generic/233 _fsstress() · line 52

Quoted 10 lines
tests/generic/233 +7 -0
@@ -43,6 +43,13 @@ _fsstress()
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
 -n $count -d $out -p 7`
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited

Should this apply to all the other tests that run fsstress?

$ grep -l FSSTRESS_PROG tests/ | wc -l
94

--D

Quoted 40 lines
tests/generic/233 +1 -0
 	echo "fsstress $args" >> $seqres.full
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $seqres.full | _filter_num
 	then
@@ -50,6 +57,7 @@ _fsstress()
 		cat $tmp.out | tee -a $seqres.full
 		status=1
 	fi
+	ulimit -l $lmem
 }
 
 # real QA test starts here
tests/generic/270 +8 -0
index 3d8656d4..bd52d56e 100755
@@ -37,6 +37,13 @@ _workout()
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
 	$SETCAP_PROG cap_chown=epi  $tmp.fsstress.bin
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited
 	(su $qa_user -c "$tmp.fsstress.bin $args" &) > /dev/null 2>&1
 
 	echo "Run dd writers in parallel"
@@ -50,6 +57,7 @@ _workout()
 	done
 
 	$KILLALL_PROG -w $tmp.fsstress.bin
+	ulimit -l $lmem
 }
 
 # real QA test starts here
Signature · 1 line
2.29.2
author
#2 lore raw
Quoted 35 lines · Darrick J. Wong
Quoted 21 lines · Zorro Lang

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

---
 tests/generic/233 | 8 ++++++++
 tests/generic/270 | 8 ++++++++
 2 files changed, 16 insertions(+)
tests/generic/233 +0 -0
index 7eda5774..342690c2 100755

tests/generic/233 _fsstress() · line 52

Quoted 10 lines
tests/generic/233 +7 -0
@@ -43,6 +43,13 @@ _fsstress()
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
 -n $count -d $out -p 7`
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited

Should this apply to all the other tests that run fsstress?

Quoted 2 lines
$ grep -l FSSTRESS_PROG tests/ | wc -l
94

The root user isn't under this restriction, only g/233 and g/270 run fsstress with non-root user ($qa_user).
Hmm... is that possible to run xfstests with a non-root user? Or you'd like to write a common helper _fsstress() (if run fsx with non-root user, I doubt it's needed too)...

Thanks,
Zorro

Quoted 44 lines

--D

Quoted 40 lines
tests/generic/233 +1 -0
 	echo "fsstress $args" >> $seqres.full
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $seqres.full | _filter_num
 	then
@@ -50,6 +57,7 @@ _fsstress()
 		cat $tmp.out | tee -a $seqres.full
 		status=1
 	fi
+	ulimit -l $lmem
 }
 
 # real QA test starts here
tests/generic/270 +8 -0
index 3d8656d4..bd52d56e 100755
@@ -37,6 +37,13 @@ _workout()
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
 	$SETCAP_PROG cap_chown=epi  $tmp.fsstress.bin
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited
 	(su $qa_user -c "$tmp.fsstress.bin $args" &) > /dev/null 2>&1
 
 	echo "Run dd writers in parallel"
@@ -50,6 +57,7 @@ _workout()
 	done
 
 	$KILLALL_PROG -w $tmp.fsstress.bin
+	ulimit -l $lmem
 }
 
 # real QA test starts here
Signature · 1 line
2.29.2
#3 lore raw
Quoted 36 lines · Zorro Lang
Quoted 35 lines · Darrick J. Wong
Quoted 21 lines · Zorro Lang

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

---
 tests/generic/233 | 8 ++++++++
 tests/generic/270 | 8 ++++++++
 2 files changed, 16 insertions(+)
tests/generic/233 +0 -0
index 7eda5774..342690c2 100755

tests/generic/233 _fsstress() · line 52

Quoted 10 lines
tests/generic/233 +7 -0
@@ -43,6 +43,13 @@ _fsstress()
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
 -n $count -d $out -p 7`
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited

Should this apply to all the other tests that run fsstress?

Quoted 6 lines
Quoted 2 lines
$ grep -l FSSTRESS_PROG tests/ | wc -l
94

The root user isn't under this restriction, only g/233 and g/270 run fsstress with non-root user ($qa_user).
Hmm... is that possible to run xfstests with a non-root user? Or you'd like to

No, fstests is required to be run by root user, there's a check in 'check' script :)

Quoted 2 lines

write a common helper _fsstress() (if run fsx with non-root user, I doubt it's needed too)...

Perhaps we should raise the limit in fsstress and fsx while setting up io_uring_queue_init()? So we don't have to add this ulimit settings to every test that runs fsstress/fsx with regular user.

Thanks,
Eryu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 51 lines · Zorro Lang
Quoted 50 lines · Eryu Guan
Quoted 36 lines · Zorro Lang
Quoted 35 lines · Darrick J. Wong
Quoted 21 lines · Zorro Lang

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

---
 tests/generic/233 | 8 ++++++++
 tests/generic/270 | 8 ++++++++
 2 files changed, 16 insertions(+)
tests/generic/233 +0 -0
index 7eda5774..342690c2 100755

tests/generic/233 _fsstress() · line 52

Quoted 10 lines
tests/generic/233 +7 -0
@@ -43,6 +43,13 @@ _fsstress()
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
 -n $count -d $out -p 7`
 
+	# io_uring accounts memory it needs under the rlimit memlocked option,
+	# which can be quite low on some setups (especially 64K pagesize). root
+	# isn't under this restriction, but regular users are. To avoid the
+	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
+	# temporarily.
+	lmem=`ulimit -l`
+	ulimit -l unlimited

Should this apply to all the other tests that run fsstress?

Quoted 6 lines
Quoted 2 lines
$ grep -l FSSTRESS_PROG tests/ | wc -l
94

The root user isn't under this restriction, only g/233 and g/270 run fsstress with non-root user ($qa_user).
Hmm... is that possible to run xfstests with a non-root user? Or you'd like to

No, fstests is required to be run by root user, there's a check in 'check' script :)

Quoted 2 lines

write a common helper _fsstress() (if run fsx with non-root user, I doubt it's needed too)...

Quoted 8 lines
Quoted 3 lines

Perhaps we should raise the limit in fsstress and fsx while setting up io_uring_queue_init()? So we don't have to add this ulimit settings to every test that runs fsstress/fsx with regular user.

Hmm... I don't know if it's the best idea, to change a sysconfig in fsstress.

If by any chance, in the future, someone case uses fsstress to test a sysconfig which is changed by fsstress :)

And fsstress is invoked by a regular user, and doesn't have the permission to change rlimit..

Thanks,
Eryu

#6 lore raw
In reply to Zorro Lang
Quoted 9 lines · Zorro Lang

The ltp/fsstress always fails on io_uring_queue_init() by returnning ENOMEM. Due to io_uring accounts memory it needs under the rlimit memlocked option, which can be quite low on some setups, especially on 64K pagesize machine. root isn't under this restriction, but regular users are. So only g/233 and g/270 which use $qa_user to run fsstress are failed.

To avoid this failure, set max locked memory to unlimited before doing fsstress, then restore it after test done.

I'm wondering if we could just set 'ulimit -l unlimited' in check? so all tests inherits the unlimited mlock setting. For root, this doesn't change anything, but command runs by regular user could get unlimited lockable memory. And new tests won't have to worry about running fsstress/fsx as $qa_user.

Thanks,
Eryu