Skip to contentSkip to conversation

generic/707: Test moving directory while being grown

PATCH Superseded Jan Kara 3 participants 4 messages Jan 26, 2023 – Jan 30, 2023

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jan Kara

author
#0 lore raw

PATCH 1/1 generic/707: Test moving directory while being grown

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

---
 tests/generic/707     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out
tests/generic/707 new file +54 -0
new file mode 100755
index 000000000000..25cbb7e98a23
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+cd $SCRATCH_MNT
+
+# Loop multiple times trying to hit the race
+loops=100
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir dir0/dir
+	pushd dir0/dir &>/dev/null
+	create_files &
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv dir$i/dir dir$((i+1))/dir
+	done
+	wait
+	rm -r dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3
Reviewed-by on v1 1/1
#1 lore raw
Quoted 7 lines · Jan Kara

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

Looks fine to me.

Quoted 78 lines
---
 tests/generic/707     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out
tests/generic/707 new file +54 -0
new file mode 100755
index 000000000000..25cbb7e98a23
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+cd $SCRATCH_MNT
+
+# Loop multiple times trying to hit the race
+loops=100
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir dir0/dir
+	pushd dir0/dir &>/dev/null
+	create_files &
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv dir$i/dir dir$((i+1))/dir
+	done
+	wait
+	rm -r dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3
#2 lore raw
In reply to Jan Kara
Quoted 30 lines · Jan Kara

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

---
 tests/generic/707     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out
tests/generic/707 new file +10 -0
new file mode 100755
index 000000000000..25cbb7e98a23
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 

tests/generic/707 line 20

Quoted 10 lines
tests/generic/707 new file +10 -0
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+cd $SCRATCH_MNT

Is this necessary? Generally we use absolute path, e.g.

xxxfile=$SCRATCH_MNT/xxxxxxx
touch $xxxfile

tests/generic/707 line 25

Quoted 5 lines
tests/generic/707 new file +5 -0
+
+# Loop multiple times trying to hit the race
+loops=100
+files=500
+moves=500

You might would like to try LOAD_FACTOR or TIME_FACTOR parameters?

Quoted 7 lines
tests/generic/707 new file +7 -0
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i

tests/generic/707 line 42

Quoted 10 lines
tests/generic/707 new file +10 -0
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir dir0/dir
+	pushd dir0/dir &>/dev/null

Use absolute path can help save the pushd&popd operations.

tests/generic/707 line 43

Quoted 1 line
tests/generic/707 new file +1 -0
+	create_files &

If there're background processes, we should deal with it in _cleanup() clearly.

Thanks,
Zorro

Quoted 22 lines
tests/generic/707 new file +11 -0
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv dir$i/dir dir$((i+1))/dir
+	done
+	wait
+	rm -r dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3
author
#3 lore raw
Quoted 44 lines · Zorro Lang
Quoted 30 lines · Jan Kara

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

---
 tests/generic/707     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out
tests/generic/707 new file +10 -0
new file mode 100755
index 000000000000..25cbb7e98a23
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 

tests/generic/707 line 20

Quoted 10 lines
tests/generic/707 new file +10 -0
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+cd $SCRATCH_MNT

Is this necessary? Generally we use absolute path, e.g.

Quoted 2 lines

xxxfile=$SCRATCH_MNT/xxxxxxx
touch $xxxfile

So here I can use absolute path without a problem.

Quoted 7 lines

tests/generic/707 line 25

Quoted 5 lines
tests/generic/707 new file +5 -0
+
+# Loop multiple times trying to hit the race
+loops=100
+files=500
+moves=500

You might would like to try LOAD_FACTOR or TIME_FACTOR parameters?

Will do.

Quoted 11 lines
Quoted 11 lines
tests/generic/707 new file +11 -0
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
Quoted 8 lines

tests/generic/707 line 42

Quoted 6 lines
tests/generic/707 new file +6 -0
+	mkdir dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir dir0/dir
+	pushd dir0/dir &>/dev/null

Use absolute path can help save the pushd&popd operations.

Not really. The thing is the background process needs to start inside the directory that is being moved by the foreground process. Because the absolute path constantly changes due to movement it is not really possible to implement this without changing cwd.

Quoted 3 lines

tests/generic/707 line 43

Quoted 1 line
tests/generic/707 new file +1 -0
+	create_files &

If there're background processes, we should deal with it in _cleanup() clearly.

Right, I'll add that. Thanks for review!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR