Skip to contentSkip to conversation

common: Fix file leak in _get_max_file_size

PATCH In review Pavel Reichl 3 participants 14 messages Sep 22, 2022 – Sep 23, 2022 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 common: Fix file leak in _get_max_file_size

This is obviously mostly problematic for FS lacking support for sparse files.

---
 common/rc         | 1 +
 tests/generic/692 | 0
 2 files changed, 1 insertion(+)
 mode change 100644 => 100755 tests/generic/692
common/rc +1 -0
index 228fcb37..c9078649 100644
@@ -4637,6 +4637,7 @@ _get_max_file_size()
 			l=$m
 		fi
 	done
+	rm -f $testfile
 	echo $l
 }
 
tests/generic/692 +0 -0
old mode 100644
new mode 100755
Signature · 1 line
2.37.3
#1 lore raw

PATCH 2/4 g/299: speed up the test

Move the _require_xfs_io_command "falloc" above the potencially 'expensive' function _get_max_file_size

---
 tests/generic/299 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/299 +1 -1
index 8e1a112c..1bb371a3 100755
@@ -24,6 +24,7 @@ _require_scratch
 _require_odirect
 _require_aio
 _require_block_device $SCRATCH_DEV
+_require_xfs_io_command "falloc"
 
 NUM_JOBS=$((4*LOAD_FACTOR))
 BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
@@ -96,7 +97,6 @@ filename=buffered-aio-verifier
 EOF
 
 _require_fio $fio_config
-_require_xfs_io_command "falloc"
 
 _scratch_mkfs >> $seqres.full 2>&1
 _scratch_mount
Signature · 1 line
2.37.3
#2 lore raw

Re: [PATCH 2/4] g/299: speed up the test

Quoted 5 lines · Pavel Reichl

Move the _require_xfs_io_command "falloc" above the potencially 'expensive' function _get_max_file_size

---

It's not for speeding up this test, I think you're trying to notrun this case on a fs which doesn't support fallocate and sparse file but support truncate syscall. Anyway I don't have objection on this patch.

Quoted 26 lines
 tests/generic/299 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/299 +1 -1
index 8e1a112c..1bb371a3 100755
@@ -24,6 +24,7 @@ _require_scratch
 _require_odirect
 _require_aio
 _require_block_device $SCRATCH_DEV
+_require_xfs_io_command "falloc"
 
 NUM_JOBS=$((4*LOAD_FACTOR))
 BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
@@ -96,7 +97,6 @@ filename=buffered-aio-verifier
 EOF
 
 _require_fio $fio_config
-_require_xfs_io_command "falloc"
 
 _scratch_mkfs >> $seqres.full 2>&1
 _scratch_mount
Signature · 1 line
2.37.3
#3 lore raw

PATCH 3/4 generic: cleanup test files

In reply to Pavel Reichl

There seems to be some seek_sanity_testfile files that are not cleaned up and take up space.

-rwxr-xr-x. 1 root root 8.0G Sep 22 13:39 seek_sanity_testfile10

---
 tests/generic/285 | 2 +-
 tests/generic/436 | 2 +-
 tests/generic/445 | 2 +-
 tests/generic/448 | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
tests/generic/285 +1 -1
index 6c755b5f..d1850021 100755
@@ -28,7 +28,7 @@ _require_test_program "seek_sanity_test"
 # Override the default cleanup function.
 _cleanup()
 {
-	eval "rm -f $BASE_TEST_FILE.*"
+	rm -f $BASE_TEST_FILE*
 }
 
 _run_seek_sanity_test $BASE_TEST_FILE > $seqres.full 2>&1 ||
tests/generic/436 +1 -1
index 7fb24b41..d54af436 100755
@@ -16,7 +16,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile
 # Override the default cleanup function.
 _cleanup()
 {
-	rm -f $tmp.* $BASE_TEST_FILE.*
+	rm -f $tmp.* $BASE_TEST_FILE*
 }
 
 # Import common functions.
tests/generic/445 +1 -1
index 05d1c5b3..d9e9f84b 100755
@@ -16,7 +16,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile
 # Override the default cleanup function.
 _cleanup()
 {
-	rm -f $tmp.* $BASE_TEST_FILE.*
+	rm -f $tmp.* $BASE_TEST_FILE*
 }
 
 # Import common functions.
tests/generic/448 +1 -1
index d66967bc..11945549 100755
@@ -15,7 +15,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile_$seq
 _cleanup()
 {
 	cd /
-	rm -f $tmp.* $BASE_TEST_FILE
+	rm -f $tmp.* $BASE_TEST_FILE*
 }
 
 # Import common functions.
Signature · 1 line
2.37.3
#4 lore raw

Re: [PATCH 3/4] generic: cleanup test files

Quoted 18 lines · Pavel Reichl

There seems to be some seek_sanity_testfile files that are not cleaned up and take up space.

-rwxr-xr-x. 1 root root 8.0G Sep 22 13:39 seek_sanity_testfile10

---
 tests/generic/285 | 2 +-
 tests/generic/436 | 2 +-
 tests/generic/445 | 2 +-
 tests/generic/448 | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
tests/generic/285 +0 -0
index 6c755b5f..d1850021 100755

tests/generic/285 _require_test_program "seek_sanity_test" · line 31

Quoted 5 lines
tests/generic/285 +1 -1
@@ -28,7 +28,7 @@ _require_test_program "seek_sanity_test"
 # Override the default cleanup function.
 _cleanup()
 {
-	eval "rm -f $BASE_TEST_FILE.*"
+	rm -f $BASE_TEST_FILE*

OK, this eval looks not reasonable.

tests/generic/436 BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile

Quoted 8 lines
tests/generic/285 +0 -0
 }
 
 _run_seek_sanity_test $BASE_TEST_FILE > $seqres.full 2>&1 ||
tests/generic/436 +0 -0
index 7fb24b41..d54af436 100755
@@ -16,7 +16,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile

I'm wondering two questions:
1) If these cases all use $TEST_DIR/seek_sanity_testfile, and don't really remove these seek_sanity_testfile* files, will them affect each testing result? Can we set BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile_$seq at least, to avoid they use same files?

2) If these $BASE_TEST_FILE* files won't take too much space, and won't affect later testing, how about keeping them in $TEST_DIR, then remove these specific _cleanup(). From my testing, after running g/285, g/436, g/445 and g/448, I got [1]. Looks like they only takes ~25M space.

Thanks,
Zorro

[1]

# du -h /mnt/test/
25M     /mnt/test/
# du -h /mnt/test/seek_sanity_testfile*
4.0K    /mnt/test/seek_sanity_testfile
0       /mnt/test/seek_sanity_testfile01
4.0K    /mnt/test/seek_sanity_testfile02
12K     /mnt/test/seek_sanity_testfile03
4.0K    /mnt/test/seek_sanity_testfile04
4.0K    /mnt/test/seek_sanity_testfile05
8.0K    /mnt/test/seek_sanity_testfile06
44K     /mnt/test/seek_sanity_testfile07
44K     /mnt/test/seek_sanity_testfile08
404K    /mnt/test/seek_sanity_testfile09
128K    /mnt/test/seek_sanity_testfile10
128K    /mnt/test/seek_sanity_testfile11
128K    /mnt/test/seek_sanity_testfile12
4.0M    /mnt/test/seek_sanity_testfile13
4.0M    /mnt/test/seek_sanity_testfile14
12M     /mnt/test/seek_sanity_testfile15
4.0M    /mnt/test/seek_sanity_testfile16
0       /mnt/test/seek_sanity_testfile17
0       /mnt/test/seek_sanity_testfile_44818
Quoted 40 lines
tests/generic/436 +1 -1
 # Override the default cleanup function.
 _cleanup()
 {
-	rm -f $tmp.* $BASE_TEST_FILE.*
+	rm -f $tmp.* $BASE_TEST_FILE*
 }
 
 # Import common functions.
tests/generic/445 +1 -1
index 05d1c5b3..d9e9f84b 100755
@@ -16,7 +16,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile
 # Override the default cleanup function.
 _cleanup()
 {
-	rm -f $tmp.* $BASE_TEST_FILE.*
+	rm -f $tmp.* $BASE_TEST_FILE*
 }
 
 # Import common functions.
tests/generic/448 +1 -1
index d66967bc..11945549 100755
@@ -15,7 +15,7 @@ BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile_$seq
 _cleanup()
 {
 	cd /
-	rm -f $tmp.* $BASE_TEST_FILE
+	rm -f $tmp.* $BASE_TEST_FILE*
 
 
 

}

# Import common functions.

Signature · 1 line
2.37.3

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Re: [PATCH 3/4] generic: cleanup test files

Quoted 10 lines · Pavel Reichl

2) If these $BASE_TEST_FILE* files won't take too much space, and won't affect later testing, how about keeping them in $TEST_DIR, then remove these specific _cleanup(). From my testing, after running g/285, g/436, g/445 and g/448, I got [1]. Looks like they only takes ~25M space.

Hi,

as I mentioned in the commit message - in my case I'm getting:
8.0G seek_sanity_testfile10
So I think it's good idea to solve it somehow.

OK, you can combine patch 1,3,4 into one patch, due to they all take too much space on filesystem which doesn't support sparse file, so we make sure these files be cleanup.

#7 lore raw

Re: [PATCH 3/4] generic: cleanup test files

In reply to Zorro Lang
Quoted 6 lines

I'm wondering two questions:
1) If these cases all use $TEST_DIR/seek_sanity_testfile, and don't really remove these seek_sanity_testfile* files, will them affect each testing result? Can we set BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile_$seq at least, to avoid they use same files?

I'm happy to try to implement this idea.

#8 lore raw

Re: [PATCH 3/4] generic: cleanup test files

Quoted 7 lines · Pavel Reichl
Quoted 6 lines

I'm wondering two questions:
1) If these cases all use $TEST_DIR/seek_sanity_testfile, and don't really remove these seek_sanity_testfile* files, will them affect each testing result? Can we set BASE_TEST_FILE=$TEST_DIR/seek_sanity_testfile_$seq at least, to avoid they use same files?

I'm happy to try to implement this idea.

If you need to remove them in cleanup, then the name won't conflict :)

Quoted 1 line
#9 lore raw

PATCH 4/4 g/394: clean up test files

In reply to Pavel Reichl

Clean up test files taking space.

-rwxr-xr-x. 1 root root 1.0G Sep 22 14:58 394.1183899 -rwxr-xr-x. 1 root root 1.0G Sep 22 14:58 394.1183899-1
-rwxr-xr-x. 1 root root 0 sep 22 14:58 394.1183899+1

---
 tests/generic/394 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/394 +1 -1
index fc03e2f3..cbc2ce13 100755
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	ulimit -f unlimited
-	rm -f $tmp.*
+	rm -f $tmp.* $TEST_DIR/$seq.*
 }
 
 # Import common functions.
Signature · 1 line
2.37.3
#10 lore raw

Re: [PATCH 4/4] g/394: clean up test files

Quoted 16 lines · Pavel Reichl

Clean up test files taking space.

(quoted diff) +0 -3
-rwxr-xr-x. 1 root root 1.0G Sep 22 14:58 394.1183899
-rwxr-xr-x. 1 root root 1.0G Sep 22 14:58 394.1183899-1
-rwxr-xr-x. 1 root root    0 sep 22 14:58 394.1183899+1
---
 tests/generic/394 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/394 +0 -0
index fc03e2f3..cbc2ce13 100755

tests/generic/394 _cleanup() · line 17

Quoted 5 lines
tests/generic/394 +1 -1
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	ulimit -f unlimited
-	rm -f $tmp.*
+	rm -f $tmp.* $TEST_DIR/$seq.*

I think you can combine this patch with patch 1/4, due to they're for same purpose, all due to some fs (e.g. exfat) doesn't support sparse file, so we need to remove those large files created by truncate. If you find same issue on other places, better to fix them in same patch.

Quoted 6 lines
tests/generic/394 +0 -0
 }
 
 # Import common functions.
Signature · 1 line
2.37.3
#11 lore raw
In reply to Pavel Reichl
Quoted 15 lines · Pavel Reichl

This is obviously mostly problematic for FS lacking support for sparse files.

---
 common/rc         | 1 +
 tests/generic/692 | 0
 2 files changed, 1 insertion(+)
 mode change 100644 => 100755 tests/generic/692
common/rc +0 -0
index 228fcb37..c9078649 100644

common/rc _get_max_file_size() · line 4640

Quoted 4 lines
common/rc +1 -0
@@ -4637,6 +4637,7 @@ _get_max_file_size()
 			l=$m
 		fi
 	done
+	rm -f $testfile

OK, as "truncate" will allocate real space in exfat and other fs not support sparse file, I think we'd better to remove it. If you find more similar situation, please remove them in same one patch :)

tests/generic/692

Quoted 6 lines
common/rc +0 -0
 	echo $l
 }
 
tests/generic/692 +0 -0
old mode 100644
new mode 100755

Hmm... is this another fix? Although I don't know if it's worth to be a single patch, if it doesn't cause anything wrong?

Thanks,
Zorro

Quoted 3 lines
Signature · 1 line
2.37.3
#12 lore raw
Quoted 7 lines

tests/generic/692

Quoted 5 lines
tests/generic/692 +0 -0
old mode 100644
new mode 100755

Hmm... is this another fix? Although I don't know if it's worth to be a single patch, if it doesn't cause anything wrong?

Hmm, sorry about this change, it's not on purpose, I'm not sure what happened here. I'll remove it in the next iteration

#13 lore raw
In reply to Pavel Reichl
Quoted 15 lines · Pavel Reichl

This is obviously mostly problematic for FS lacking support for sparse files.

---
 common/rc         | 1 +
 tests/generic/692 | 0
 2 files changed, 1 insertion(+)
 mode change 100644 => 100755 tests/generic/692
common/rc +0 -0
index 228fcb37..c9078649 100644

common/rc _get_max_file_size() · line 4642

Quoted 6 lines
common/rc +1 -0
@@ -4637,6 +4637,7 @@ _get_max_file_size()
 			l=$m
 		fi
 	done
+	rm -f $testfile
 	echo $l
 }

Removing the file is fine, but I didn't have filesystems that lack support for sparse files in mind when I wrote this function. Maybe it should look at $FSTYP first, and only use the generic algorithm as a fallback for unhandled types?

- Eric

#14 lore raw
Quoted 11 lines · Eric Biggers
Quoted 10 lines · Pavel Reichl

This is obviously mostly problematic for FS lacking support for sparse files.

---
 common/rc         | 1 +
 tests/generic/692 | 0
 2 files changed, 1 insertion(+)
 mode change 100644 => 100755 tests/generic/692
Quoted 15 lines

common/rc _get_max_file_size() · line 4642

Quoted 11 lines
common/rc +1 -0
index 228fcb37..c9078649 100644
@@ -4637,6 +4637,7 @@ _get_max_file_size()
 			l=$m
 		fi
 	done
+	rm -f $testfile
 	echo $l
 }

Removing the file is fine, but I didn't have filesystems that lack support for sparse files in mind when I wrote this function. Maybe it should look at $FSTYP first, and only use the generic algorithm as a fallback for unhandled types?

That might be good idea. The current algorithm takes too long time for fs which doesn't support sparse file.

Anyway, this $testfile should be removed. If Pavel would like to change this function likes:

_get_max_file_size()

{
	case $FSTYP:
	exfat)
		# calculate the max file size of exfat according to blocksize, then
		echo $max_file_size_of_exfat
		;;
	# Add ext* if extN forks think it's better
	*)
		# By default, do the algorithm this function does originally
		;;
	esac
}

he can split this part as a separated patch, then let's review it. Or we can have this patch at first, then let Eric change it when he'd like to do that.

Thanks,
Zorro

Quoted 3 lines

- Eric