Skip to contentSkip to conversation

NFS test fixes

PATCH In review Martin Doucha 4 participants 21 messages Sep 19, 2023 – Nov 13, 2023

lore
Share & tools
lore
#0 lore raw

cover NFS test fixes

The nfs03.sh test has a race condition which may randomly cause NFS unmount to fail and trigger strange behavior. Add NFS mount/unmount logging and unmount success check for easier debugging. Also fix the race condition in nfs03.sh

Patch 3 should be merged after new LTP release.

Martin Doucha (3):
nfs_lib.sh: Log NFS mount and unmount
nfs03.sh: Wait for all files to be removed
nfs_lib.sh: Fail the test if NFS unmount fails

 testcases/network/nfs/nfs_stress/nfs03.sh   | 2 ++
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 9 +++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)
Signature · 5 lines
2.42.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

PATCH 1/3 nfs_lib.sh: Log NFS mount and unmount

---
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
testcases/network/nfs/nfs_stress/nfs_lib.sh +7 -2
index abf7ba5a2..a996f7cc8 100644
@@ -191,7 +191,9 @@ nfs_setup()
 
 		remote_dir="$(get_remote_dir $i $type)"
 		nfs_setup_server "$remote_dir" "$(($$ + n))"
-		nfs_mount "$(get_local_dir $i $n)" "$remote_dir" "-o proto=$type,vers=$i"
+		local_dir="$(get_local_dir $i $n)"
+		tst_res TINFO "Mounting $local_dir"
+		nfs_mount "$local_dir" "$remote_dir" "-o proto=$type,vers=$i"
 
 		n=$(( n + 1 ))
 	done
@@ -210,7 +212,10 @@ nfs_cleanup()
 	local n=0
 	for i in $VERSION; do
 		local_dir="$(get_local_dir $i $n)"
-		grep -q "$local_dir" /proc/mounts && umount $local_dir
+		if grep -q "$local_dir" /proc/mounts; then
+			tst_res TINFO "Unmounting $local_dir"
+			umount $local_dir
+		fi
 		n=$(( n + 1 ))
 	done
 	sleep 2
Signature · 5 lines
2.42.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/3
#2 lore raw

Re: [LTP] [PATCH 1/3] nfs_lib.sh: Log NFS mount and unmount

Hi Martin,

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/3
#3 lore raw

Re: [LTP] [PATCH 1/3] nfs_lib.sh: Log NFS mount and unmount

In reply to Martin Doucha
Quoted 34 lines · Martin Doucha
---
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
testcases/network/nfs/nfs_stress/nfs_lib.sh +7 -2
index abf7ba5a2..a996f7cc8 100644
@@ -191,7 +191,9 @@ nfs_setup()
 
 		remote_dir="$(get_remote_dir $i $type)"
 		nfs_setup_server "$remote_dir" "$(($$ + n))"
-		nfs_mount "$(get_local_dir $i $n)" "$remote_dir" "-o proto=$type,vers=$i"
+		local_dir="$(get_local_dir $i $n)"
+		tst_res TINFO "Mounting $local_dir"
+		nfs_mount "$local_dir" "$remote_dir" "-o proto=$type,vers=$i"
 
 		n=$(( n + 1 ))
 	done
@@ -210,7 +212,10 @@ nfs_cleanup()
 	local n=0
 	for i in $VERSION; do
 		local_dir="$(get_local_dir $i $n)"
-		grep -q "$local_dir" /proc/mounts && umount $local_dir
+		if grep -q "$local_dir" /proc/mounts; then
+			tst_res TINFO "Unmounting $local_dir"
+			umount $local_dir
+		fi
 		n=$(( n + 1 ))
 	done
 	sleep 2
Signature · 1 line
2.42.0
Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

PATCH 2/3 nfs03.sh: Wait for all files to be removed

In reply to Martin Doucha

A race condition between the two parallel calls of rm_files may cause NFS unmount to fail and then the NFS client will get confused by remote filesystem change which leads to strange failures. Make sure both rm_files calls finish before returning from the main test function.

---
 testcases/network/nfs/nfs_stress/nfs03.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/network/nfs/nfs_stress/nfs03.sh +2 -0
index e5f4de67c..e9ef5fb78 100755
@@ -66,11 +66,13 @@ do_test()
 	cd ../dir1
 	wait $pid1
 	rm_files &
+	pid3=$!
 
 	tst_res TINFO "cd dir2 & removing files"
 	cd ../dir2
 	wait $pid2
 	rm_files
+	wait $pid3
 
 	tst_res TPASS "test done"
 }
Signature · 5 lines
2.42.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 2/3
#6 lore raw

Re: [LTP] [PATCH 2/3] nfs03.sh: Wait for all files to be removed

Hi Martin,

Good idea, if the test timeouts we know it's due removing files.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw

Re: [LTP] [PATCH 2/3] nfs03.sh: Wait for all files to be removed

In reply to Martin Doucha

Hello,

Quoted 4 lines · Martin Doucha

A race condition between the two parallel calls of rm_files may cause NFS unmount to fail and then the NFS client will get confused by remote filesystem change which leads to strange failures. Make sure both rm_files calls finish before returning from the main test function.

You mean unmounting fails if there are still operations in progress?

Like does umount fail with EBUSY?

Quoted 26 lines
---
 testcases/network/nfs/nfs_stress/nfs03.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/network/nfs/nfs_stress/nfs03.sh +2 -0
index e5f4de67c..e9ef5fb78 100755
@@ -66,11 +66,13 @@ do_test()
 	cd ../dir1
 	wait $pid1
 	rm_files &
+	pid3=$!
 
 	tst_res TINFO "cd dir2 & removing files"
 	cd ../dir2
 	wait $pid2
 	rm_files
+	wait $pid3
 
 	tst_res TPASS "test done"
 }
Signature · 1 line
2.42.0
Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw

Re: [LTP] [PATCH 2/3] nfs03.sh: Wait for all files to be removed

Quoted 4 lines · Richard Palethorpe

Hello,

Martin Doucha <[email protected]> writes:

Quoted 8 lines
Quoted 4 lines

A race condition between the two parallel calls of rm_files may cause NFS unmount to fail and then the NFS client will get confused by remote filesystem change which leads to strange failures. Make sure both rm_files calls finish before returning from the main test function.

You mean unmounting fails if there are still operations in progress?

Like does umount fail with EBUSY?

Yes, the unmount will fail with EBUSY. Then the test filesystem gets deleted while still mounted over NFS and a new NFS mount gets added on top. Then the NFS client randomly gets confused and complains about fsid change.

Quoted 29 lines
Quoted 27 lines
---
  testcases/network/nfs/nfs_stress/nfs03.sh | 2 ++
  1 file changed, 2 insertions(+)
testcases/network/nfs/nfs_stress/nfs03.sh +2 -0
index e5f4de67c..e9ef5fb78 100755
@@ -66,11 +66,13 @@ do_test()
  	cd ../dir1
  	wait $pid1
  	rm_files &
+	pid3=$!
 
  	tst_res TINFO "cd dir2 & removing files"
  	cd ../dir2
  	wait $pid2
  	rm_files
+	wait $pid3
 
  	tst_res TPASS "test done"
  }
Signature · 1 line
2.42.0
Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 2/3
#9 lore raw

Re: [LTP] [PATCH 2/3] nfs03.sh: Wait for all files to be removed

In reply to Martin Doucha
Quoted 30 lines · Martin Doucha

A race condition between the two parallel calls of rm_files may cause NFS unmount to fail and then the NFS client will get confused by remote filesystem change which leads to strange failures. Make sure both rm_files calls finish before returning from the main test function.

---
 testcases/network/nfs/nfs_stress/nfs03.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/network/nfs/nfs_stress/nfs03.sh +2 -0
index e5f4de67c..e9ef5fb78 100755
@@ -66,11 +66,13 @@ do_test()
 	cd ../dir1
 	wait $pid1
 	rm_files &
+	pid3=$!
 
 	tst_res TINFO "cd dir2 & removing files"
 	cd ../dir2
 	wait $pid2
 	rm_files
+	wait $pid3
 
 	tst_res TPASS "test done"
 }
Signature · 1 line
2.42.0
Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#11 lore raw

Re: [LTP] [PATCH 2/3] nfs03.sh: Wait for all files to be removed

Hi all,

Quoted 2 lines

Hi!

I merged these two patches.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#12 lore raw

PATCH 3/3 nfs_lib.sh: Fail the test if NFS unmount fails

In reply to Martin Doucha
---
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
index a996f7cc8..099c78759 100644
@@ -214,7 +214,7 @@ nfs_cleanup()
 		local_dir="$(get_local_dir $i $n)"
 		if grep -q "$local_dir" /proc/mounts; then
 			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"
 		fi
 		n=$(( n + 1 ))
 	done
Signature · 5 lines
2.42.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 3/3
#13 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Hi Martin,

Quoted 10 lines
---
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/network/nfs/nfs_stress/nfs_lib.sh +0 -0
index a996f7cc8..099c78759 100644

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 5 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
@@ -214,7 +214,7 @@ nfs_cleanup()
 		local_dir="$(get_local_dir $i $n)"
 		if grep -q "$local_dir" /proc/mounts; then
 			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I'd personally add TWARN (i.e. less critical error). @Cyril WDYT?

Kind regards,
Petr

Quoted 3 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +0 -0
 		fi
 		n=$(( n + 1 ))
 	done
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 3/3
#14 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

In reply to Martin Doucha
Quoted 20 lines · Martin Doucha
---
 testcases/network/nfs/nfs_stress/nfs_lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
index a996f7cc8..099c78759 100644
@@ -214,7 +214,7 @@ nfs_cleanup()
 		local_dir="$(get_local_dir $i $n)"
 		if grep -q "$local_dir" /proc/mounts; then
 			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"
 		fi
 		n=$(( n + 1 ))
 	done
Signature · 1 line
2.42.0
Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 3/3
#15 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

In reply to Martin Doucha

Hi!

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 10 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
index a996f7cc8..099c78759 100644
@@ -214,7 +214,7 @@ nfs_cleanup()
 		local_dir="$(get_local_dir $i $n)"
 		if grep -q "$local_dir" /proc/mounts; then
 			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from the previous patches, should be applied after the release.

Otherwise:

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#16 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Quoted 13 lines

Hi!

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 10 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
index a996f7cc8..099c78759 100644
@@ -214,7 +214,7 @@ nfs_cleanup()
 		local_dir="$(get_local_dir $i $n)"
 		if grep -q "$local_dir" /proc/mounts; then
 			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from

Right, TBROK looks to be the best.
Martin, if you're ok with the change, I'll update it before merge (after the release).

Kind regards,
Petr

Quoted 5 lines

the previous patches, should be applied after the release.

Otherwise:

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#17 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Quoted 14 lines · Petr Vorel
Quoted 13 lines

Hi!

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 10 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
index a996f7cc8..099c78759 100644
@@ -214,7 +214,7 @@ nfs_cleanup()
  		local_dir="$(get_local_dir $i $n)"
  		if grep -q "$local_dir" /proc/mounts; then
  			tst_res TINFO "Unmounting $local_dir"
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from

Quoted 3 lines

Right, TBROK looks to be the best.
Martin, if you're ok with the change, I'll update it before merge (after the release).

I don't see the point. This is a cleanup function. The TBROK will be changed to TWARN and the test will continue anyway.

Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#18 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Hi,

...

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 2 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from

Right, TBROK looks to be the best.
Martin, if you're ok with the change, I'll update it before merge (after the release).

I don't see the point. This is a cleanup function. The TBROK will be changed to TWARN and the test will continue anyway.

But TFAIL in cleanup function looks to me strange as well. Should we endup with TWARN then?

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#19 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Quoted 8 lines · Petr Vorel

Hi,

...

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 2 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Right, TBROK looks to be the best.
Martin, if you're ok with the change, I'll update it before merge (after the release).

I don't see the point. This is a cleanup function. The TBROK will be changed to TWARN and the test will continue anyway.

But TFAIL in cleanup function looks to me strange as well. Should we endup with TWARN then?

TWARN makes sense. Feel free to change it during merge.

Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#20 lore raw

Re: [LTP] [PATCH 3/3] nfs_lib.sh: Fail the test if NFS unmount fails

Quoted 12 lines · Petr Vorel
Quoted 11 lines

Hi,

...

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

testcases/network/nfs/nfs_stress/nfs_lib.sh nfs_cleanup() · line 217

Quoted 2 lines
testcases/network/nfs/nfs_stress/nfs_lib.sh +1 -1
-			umount $local_dir
+			umount $local_dir || tst_res TFAIL "Unmount failed"

I suppose that this should be TBROK instead. And that this, apart from

Right, TBROK looks to be the best.
Martin, if you're ok with the change, I'll update it before merge (after the release).

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I don't see the point. This is a cleanup function. The TBROK will be changed to TWARN and the test will continue anyway.

But TFAIL in cleanup function looks to me strange as well. Should we endup with TWARN then?

TWARN makes sense. Feel free to change it during merge.

Good, changed and merged. Thanks!

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp