shared/010: avoid dedupe testing blocked on large fs

PATCH In review Zorro Lang 3 participants 6 messages Sep 12, 2018 – Sep 15, 2018

Share & tools

Zorro Lang

author

2018-09-12 15:19 UTC #0 lore raw

PATCH 1/1 shared/010: avoid dedupe testing blocked on large fs

When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.

Signed-off-by Zorro Lang <[email protected]>

---

Hi,

Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.

If maintainer feels I should fix it in another patch, please tell me:-P

Thanks,
Zorro

 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

tests/shared/010 +4 -2

index 1817081b..04f55890 100755
@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"
+mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $SCRATCH_MNT -l 0 >> $seqres.full 2>&1 &
+$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
 touch $dupe_run
 for ((i = 0; i < $((2 * LOAD_FACTOR)); i++)); do
 	while [ -e $dupe_run ]; do
-		$DUPEREMOVE_PROG -dr --dedupe-options=same $SCRATCH_MNT/ \
+		$DUPEREMOVE_PROG -dr --dedupe-options=same $testdir \
 			>>$seqres.full 2>&1
 	done &
 	dedup_pids="$! $dedup_pids"

Signature · 1 line

2.14.4

Eric Sandeen

Reviewed-by on v1 1/1

2018-09-13 21:38 UTC #1 lore raw

Quoted 6 lines · Zorro Lang

When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.
Signed-off-by Zorro Lang <[email protected]>

Looks fine to me -

Reviewed-by Eric Sandeen <[email protected]>

But - I'm not sure what you mean about "testdir?"

Quoted 40 lines

---

Hi,

Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.

If maintainer feels I should fix it in another patch, please tell me:-P

Thanks,
Zorro

 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

tests/shared/010 +4 -2

index 1817081b..04f55890 100755
@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"
+mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $SCRATCH_MNT -l 0 >> $seqres.full 2>&1 &
+$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
 touch $dupe_run
 for ((i = 0; i < $((2 * LOAD_FACTOR)); i++)); do
 	while [ -e $dupe_run ]; do
-		$DUPEREMOVE_PROG -dr --dedupe-options=same $SCRATCH_MNT/ \
+		$DUPEREMOVE_PROG -dr --dedupe-options=same $testdir \
 			>>$seqres.full 2>&1
 	done &
 	dedup_pids="$! $dedup_pids"

Zorro Lang

author

2018-09-14 06:21 UTC #2 lore raw

Quoted 6 lines · Eric Sandeen

Quoted 5 lines · Zorro Lang
When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.

Quoted 7 lines

Quoted 1 line
Signed-off-by Zorro Lang <[email protected]>
Looks fine to me -
Reviewed-by Eric Sandeen <[email protected]>
But - I'm not sure what you mean about "testdir?"

Read shared/010 before merge this patch, you'll find a "$testdir" on this line:
find $testdir -type f -exec md5sum {} \; > ${tmp}.md5sum

But that "$testdir" never be assigned, it's a mistake I left :)

Thanks,
Zorro

Quoted 41 lines

---

Hi,

Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.

If maintainer feels I should fix it in another patch, please tell me:-P

Thanks,
Zorro

 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

tests/shared/010 +4 -2

index 1817081b..04f55890 100755
@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"
+mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $SCRATCH_MNT -l 0 >> $seqres.full 2>&1 &
+$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
 touch $dupe_run
 for ((i = 0; i < $((2 * LOAD_FACTOR)); i++)); do
 	while [ -e $dupe_run ]; do
-		$DUPEREMOVE_PROG -dr --dedupe-options=same $SCRATCH_MNT/ \
+		$DUPEREMOVE_PROG -dr --dedupe-options=same $testdir \
 			>>$seqres.full 2>&1
 	done &
 	dedup_pids="$! $dedup_pids"

Dave Chinner

2018-09-14 04:13 UTC #3 lore raw

In reply to Zorro Lang

Quoted 26 lines · Zorro Lang

When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.
Signed-off-by Zorro Lang <[email protected]>
---
Hi,
Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.
If maintainer feels I should fix it in another patch, please tell me:-P
Thanks,
Zorro
 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/shared/010 +0 -0
index 1817081b..04f55890 100755

tests/shared/010 function end_test() · line 68

Quoted 4 lines

tests/shared/010 +1 -0

@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"

Can we not call a temporary on the scratch device "testdir"?

Because the "test dir" name specifically refers to the test device mount point, and using it for anything else (even though it's a different variable) is just confusing.

Call it something like "local_dir" or "work_dir" or "stress_dir". But not "testdir".

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]

Zorro Lang

author

2018-09-14 06:24 UTC #4 lore raw

Quoted 34 lines · Dave Chinner

Quoted 26 lines · Zorro Lang
When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.
Signed-off-by Zorro Lang <[email protected]>
---
Hi,
Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.
If maintainer feels I should fix it in another patch, please tell me:-P
Thanks,
Zorro
 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/shared/010 +0 -0
index 1817081b..04f55890 100755
tests/shared/010 function end_test() · line 68
Quoted 4 lines
tests/shared/010 +1 -0
@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"
Can we not call a temporary on the scratch device "testdir"?

Quoted 6 lines

Because the "test dir" name specifically refers to the test device mount point, and using it for anything else (even though it's a different variable) is just confusing.
Call it something like "local_dir" or "work_dir" or "stress_dir". But not "testdir".

Hmm... by running:

# grep -rsn testdir xfstests/tests

I found there're lots of "testdir"... do you need I change them all, or just begin to notice this issue from this patch?

Thanks,
Zorro

Quoted 7 lines

Cheers,
Dave.
Signature · 2 lines
Dave Chinner
[email protected]

Dave Chinner

2018-09-15 03:18 UTC #5 lore raw

Quoted 39 lines · Zorro Lang

Quoted 38 lines · Dave Chinner
Quoted 26 lines · Zorro Lang
When test on large fs (--large-fs), xfstests preallocates a large file in SCRATCH_MNT/ at first. Duperemove will take too long time to deal with that large file (many days on 500T XFS). So move working directory to a sub-dir underlying $SCRATCH_MNT/.
Signed-off-by Zorro Lang <[email protected]>
---
Hi,
Besides fix this issue, this patch fix another issue passingly. I left a bad variable named "testdir" in this case. This patch can fix it.
If maintainer feels I should fix it in another patch, please tell me:-P
Thanks,
Zorro
 tests/shared/010 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/shared/010 +0 -0
index 1817081b..04f55890 100755
tests/shared/010 function end_test() · line 68
Quoted 4 lines
tests/shared/010 +1 -0
@@ -65,15 +65,17 @@ function end_test()
 sleep_time=$((50 * TIME_FACTOR))
 
 # Start fsstress
+testdir="$SCRATCH_MNT/dir"
Can we not call a temporary on the scratch device "testdir"?
Because the "test dir" name specifically refers to the test device mount point, and using it for anything else (even though it's a different variable) is just confusing.

Quoted 8 lines

Quoted 2 lines
Call it something like "local_dir" or "work_dir" or "stress_dir". But not "testdir".
Hmm... by running:
# grep -rsn testdir xfstests/tests
I found there're lots of "testdir"... do you need I change them all, or just begin to notice this issue from this patch?

Just this test for now, but it would be good to clean up all the other tests at some point.

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]