Skip to contentSkip to conversation

generic/175, generic/176: cleanup testdir before exit

PATCH In review Murphy Zhou 2 participants 7 messages Jan 13, 2020 – Jan 17, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic/175, generic/176: cleanup testdir before exit

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +1 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
+rm -rf $testdir
 # success, all done
 status=0
 exit
tests/generic/176 +1 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
+rm -rf $testdir
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#1 lore raw
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Quoted 13 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 4 lines
tests/generic/175 +1 -0
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

--D

Quoted 18 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +1 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
+rm -rf $testdir
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#2 lore raw
Quoted 11 lines · Darrick J. Wong
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear.

Quoted 17 lines
Quoted 17 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 3 lines

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 1 line
tests/generic/175 +1 -0
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

That's for devices I believe..

Thanks,
Murphy

Quoted 21 lines

--D

Quoted 18 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +1 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
+rm -rf $testdir
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#3 lore raw
Quoted 33 lines · Murphy Zhou
Quoted 11 lines · Darrick J. Wong
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear.

Quoted 17 lines
Quoted 17 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 3 lines

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 1 line
tests/generic/175 +1 -0
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

That's for devices I believe..

It exists to zap the scratch devices into enough of a clean state that whatever contents a test wrote to the scratch devices cannot easily (e.g. mount) leak into the next test. IOWs, exactly where you'd want FSTYP=NFS to delete whatever junk is in the scratch mount.

Plus then you get a clean nfs scratch mount at the start of each test without needing to whackamole each test.

Quoted 17 lines

Thanks,
Murphy

Quoted 14 lines

--D

Quoted 11 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +0 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 1 line
Quoted 1 line

tests/generic/176 bytes=$((blocks_needed * blksz)) · line 76

Quoted 1 line
tests/generic/176 +1 -0
+rm -rf $testdir

Also this will massively slow down generic/17[56] on btrfs/XFS considering that we don't care about deleting the massively reflinked file we created.

--D

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
tests/generic/176 +0 -0
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#4 lore raw
Quoted 65 lines · Darrick J. Wong
Quoted 33 lines · Murphy Zhou
Quoted 11 lines · Darrick J. Wong
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear.

Quoted 17 lines
Quoted 17 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 3 lines

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 1 line
tests/generic/175 +1 -0
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

That's for devices I believe..

It exists to zap the scratch devices into enough of a clean state that whatever contents a test wrote to the scratch devices cannot easily (e.g. mount) leak into the next test. IOWs, exactly where you'd want FSTYP=NFS to delete whatever junk is in the scratch mount.

Plus then you get a clean nfs scratch mount at the start of each test without needing to whackamole each test.

Quoted 17 lines

Thanks,
Murphy

Quoted 14 lines

--D

Quoted 11 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +0 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 1 line
Quoted 1 line

tests/generic/176 bytes=$((blocks_needed * blksz)) · line 76

Quoted 1 line
tests/generic/176 +1 -0
+rm -rf $testdir

Also this will massively slow down generic/17[56] on btrfs/XFS considering that we don't care about deleting the massively reflinked file we created.

Deleting them would be a good testcase to me. After all the tests, deleting them should not cause any problem, or spending to much time. Leaving them behind and letting them get formatted by mkfs, definitely will be faster, but may make potential problems not discovered.

If you think the deleting will be extremely slow, maybe it's time to optimize how xfs deleting reflinked files.

Thanks,
Murphy

Quoted 9 lines

--D

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
tests/generic/176 +0 -0
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#5 lore raw
Quoted 72 lines · Murphy Zhou
Quoted 65 lines · Darrick J. Wong
Quoted 33 lines · Murphy Zhou
Quoted 11 lines · Darrick J. Wong
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear.

Quoted 17 lines
Quoted 17 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 3 lines

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 1 line
tests/generic/175 +1 -0
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

That's for devices I believe..

It exists to zap the scratch devices into enough of a clean state that whatever contents a test wrote to the scratch devices cannot easily (e.g. mount) leak into the next test. IOWs, exactly where you'd want FSTYP=NFS to delete whatever junk is in the scratch mount.

Plus then you get a clean nfs scratch mount at the start of each test without needing to whackamole each test.

Quoted 17 lines

Thanks,
Murphy

Quoted 14 lines

--D

Quoted 11 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +0 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 1 line
Quoted 1 line

tests/generic/176 bytes=$((blocks_needed * blksz)) · line 76

Quoted 1 line
tests/generic/176 +1 -0
+rm -rf $testdir

Also this will massively slow down generic/17[56] on btrfs/XFS considering that we don't care about deleting the massively reflinked file we created.

Quoted 5 lines

Deleting them would be a good testcase to me. After all the tests, deleting them should not cause any problem, or spending to much time. Leaving them behind and letting them get formatted by mkfs, definitely will be faster, but may make potential problems not discovered.

In that case, please rewrite the commit message to say that we're enhancing this stress test to include inactivation of the files it creates.

Quoted 2 lines

If you think the deleting will be extremely slow, maybe it's time to optimize how xfs deleting reflinked files.

They already exist & have been out for review for years:

https://lore.kernel.org/linux-xfs/157784092020.1362752.15046503361741521784.stgit@magnolia/

--D

Quoted 13 lines

Thanks,
Murphy

Quoted 9 lines

--D

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
tests/generic/176 +0 -0
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1
#6 lore raw
Quoted 83 lines · Darrick J. Wong
Quoted 72 lines · Murphy Zhou
Quoted 65 lines · Darrick J. Wong
Quoted 33 lines · Murphy Zhou
Quoted 11 lines · Darrick J. Wong
Quoted 5 lines · Murphy Zhou

Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175.

Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next?

Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear.

Quoted 17 lines
Quoted 17 lines

To be clear, cleanup testdir before exit.

---
 tests/generic/175 | 1 +
 tests/generic/176 | 1 +
 2 files changed, 2 insertions(+)
tests/generic/175 +0 -0
index 79e5b3d6..bd966a28 100755
@@ -61,6 +61,7 @@ bytes=$((blks * blksz))
 echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 3 lines

tests/generic/175 bytes=$((blks * blksz)) · line 64

Quoted 1 line
tests/generic/175 +1 -0
+rm -rf $testdir

Or put another way, this probably ought to be in _try_wipe_scratch_devs()

That's for devices I believe..

It exists to zap the scratch devices into enough of a clean state that whatever contents a test wrote to the scratch devices cannot easily (e.g. mount) leak into the next test. IOWs, exactly where you'd want FSTYP=NFS to delete whatever junk is in the scratch mount.

Plus then you get a clean nfs scratch mount at the start of each test without needing to whackamole each test.

Quoted 17 lines

Thanks,
Murphy

Quoted 14 lines

--D

Quoted 11 lines
tests/generic/175 +0 -0
 # success, all done
 status=0
 exit
tests/generic/176 +0 -0
index a084578a..bc83762e 100755
@@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz))
 echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full"
 _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full"
 
Quoted 5 lines
Quoted 1 line
Quoted 1 line

tests/generic/176 bytes=$((blocks_needed * blksz)) · line 76

Quoted 1 line
tests/generic/176 +1 -0
+rm -rf $testdir

Also this will massively slow down generic/17[56] on btrfs/XFS considering that we don't care about deleting the massively reflinked file we created.

Quoted 5 lines

Deleting them would be a good testcase to me. After all the tests, deleting them should not cause any problem, or spending to much time. Leaving them behind and letting them get formatted by mkfs, definitely will be faster, but may make potential problems not discovered.

In that case, please rewrite the commit message to say that we're enhancing this stress test to include inactivation of the files it creates.

Quoted 6 lines
Quoted 2 lines

If you think the deleting will be extremely slow, maybe it's time to optimize how xfs deleting reflinked files.

They already exist & have been out for review for years:

https://lore.kernel.org/linux-xfs/157784092020.1362752.15046503361741521784.stgit@magnolia/

Great! Going to run some tests.

Thanks very much for reviewing!

Murphy

Quoted 16 lines

--D

Quoted 13 lines

Thanks,
Murphy

Quoted 9 lines

--D

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
tests/generic/176 +0 -0
 # success, all done
 status=0
 exit
Signature · 1 line
2.20.1