Skip to contentSkip to conversation

generic/707: Test moving directory while being grown

PATCH Superseded Jan Kara 3 participants 4 messages Jan 30, 2023 – Jan 31, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jan Kara

author
#0 lore raw

PATCH v2 1/1 generic/707: Test moving directory while being grown

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

---
 tests/generic/707     | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 65 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out

Changes since v1:
* Added TIME_FACTOR multiplier
* Handle background process in _cleanup
* Use absolute paths where possible

tests/generic/707 new file +63 -0
new file mode 100755
index 000000000000..d1d299563813
@@ -0,0 +1,63 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+_cleanup()
+{
+	if [ -n "$BGPID" ]; then
+		# Stop background process
+		kill -9 $BGPID &>/dev/null
+		wait
+	fi
+}
+
+# Loop multiple times trying to hit the race
+loops=$((100*TIME_FACTOR))
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir $SCRATCH_MNT/dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir $SCRATCH_MNT/dir0/dir
+	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	create_files &
+	BGPID=$!
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
+	done
+	wait
+	BGPID=""
+	rm -r $SCRATCH_MNT/dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3
#1 lore raw
Quoted 26 lines · Jan Kara

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

---
 tests/generic/707     | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 65 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out

Changes since v1:
* Added TIME_FACTOR multiplier
* Handle background process in _cleanup
* Use absolute paths where possible

tests/generic/707 new file +0 -0
new file mode 100755
index 000000000000..d1d299563813

tests/generic/707 line 10

Quoted 10 lines
tests/generic/707 new file +10 -0
@@ -0,0 +1,63 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 
    ^

A trailing whitespace at here :)

Quoted 6 lines
tests/generic/707 new file +6 -0
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+

tests/generic/707 line 26

Quoted 10 lines
tests/generic/707 new file +10 -0
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+_cleanup()
+{
+	if [ -n "$BGPID" ]; then
+		# Stop background process
+		kill -9 $BGPID &>/dev/null
+		wait
+	fi

Neet to keep the code in common/preamble:_cleanup(), if you cover it:

        cd /
        rm -f $tmp.*

especially you use pushd and popd.

I forgot to ask, is this case written for someone known fix or change? If so, better to use _fixed_by_kernel_commit or _wants_kernel_commit.

Quoted 21 lines
tests/generic/707 new file +21 -0
+}
+
+# Loop multiple times trying to hit the race
+loops=$((100*TIME_FACTOR))
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir $SCRATCH_MNT/dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do

tests/generic/707 line 57

Quoted 10 lines
tests/generic/707 new file +10 -0
+	mkdir $SCRATCH_MNT/dir0/dir
+	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	create_files &
+	BGPID=$!
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
+	done
+	wait
+	BGPID=""

unset BGPID might be better, but anyway :)

Others looks good to me.

Thanks,
Zorro

Quoted 17 lines
tests/generic/707 new file +6 -0
+	rm -r $SCRATCH_MNT/dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3
author
#2 lore raw
Quoted 20 lines · Zorro Lang
Quoted 6 lines · Jan Kara

+. ./common/preamble

tests/generic/707 new file +5 -0
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+

tests/generic/707 line 26

Quoted 10 lines
tests/generic/707 new file +10 -0
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+_cleanup()
+{
+	if [ -n "$BGPID" ]; then
+		# Stop background process
+		kill -9 $BGPID &>/dev/null
+		wait
+	fi

Neet to keep the code in common/preamble:_cleanup(), if you cover it:

Quoted 4 lines
        cd /
        rm -f $tmp.*

especially you use pushd and popd.

OK, done.

Quoted 2 lines

I forgot to ask, is this case written for someone known fix or change? If so, better to use _fixed_by_kernel_commit or _wants_kernel_commit.

Yes, good point. It tests for UDF & ext4 corruption issue. Ext4 commit is not yet merged but I'll add there something.

Quoted 25 lines
Quoted 25 lines
tests/generic/707 new file +25 -0
+}
+
+# Loop multiple times trying to hit the race
+loops=$((100*TIME_FACTOR))
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir $SCRATCH_MNT/dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir $SCRATCH_MNT/dir0/dir
+	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	create_files &
+	BGPID=$!
Quoted 8 lines

tests/generic/707 line 57

Quoted 6 lines
tests/generic/707 new file +6 -0
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
+	done
+	wait
+	BGPID=""

unset BGPID might be better, but anyway :)

Yeah, nicer. Done.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Reviewed-by on v2 1/1
#3 lore raw
In reply to Jan Kara
Quoted 8 lines · Jan Kara

Test how the filesystem can handle moving a directory to a different directory (so that parent pointer gets updated) while it is grown. Ext4 and UDF had a bug where if the directory got converted to a different type due to growth while rename is running, the filesystem got corrupted.

With Zoro's suggested changes, this looks fine.

Quoted 92 lines
---
 tests/generic/707     | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/707.out |  2 ++
 2 files changed, 65 insertions(+)
 create mode 100755 tests/generic/707
 create mode 100644 tests/generic/707.out

Changes since v1:
* Added TIME_FACTOR multiplier
* Handle background process in _cleanup
* Use absolute paths where possible

tests/generic/707 new file +63 -0
new file mode 100755
index 000000000000..d1d299563813
@@ -0,0 +1,63 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Jan Kara, SUSE Linux.  All Rights Reserved.
+#
+# FS QA Test 707
+#
+# This is a test verifying whether the filesystem can gracefully handle
+# modifying of a directory while it is being moved, in particular the cases
+# where directory format changes
+# 
+. ./common/preamble
+_begin_fstest auto
+
+_supported_fs generic
+_require_scratch
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+_cleanup()
+{
+	if [ -n "$BGPID" ]; then
+		# Stop background process
+		kill -9 $BGPID &>/dev/null
+		wait
+	fi
+}
+
+# Loop multiple times trying to hit the race
+loops=$((100*TIME_FACTOR))
+files=500
+moves=500
+
+create_files()
+{
+	# We use slightly longer file name to make directory grow faster and
+	# hopefully convert between various types
+	for (( i = 0; i < $files; i++ )); do
+		touch somewhatlongerfilename$i
+	done
+}
+
+for (( i = 0; i <= $moves; i++ )); do
+	mkdir $SCRATCH_MNT/dir$i
+done
+
+for (( l = 0; l < $loops; l++ )); do
+	mkdir $SCRATCH_MNT/dir0/dir
+	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	create_files &
+	BGPID=$!
+	popd &>/dev/null
+	for (( i = 0; i < $moves; i++ )); do
+		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
+	done
+	wait
+	BGPID=""
+	rm -r $SCRATCH_MNT/dir$moves/dir
+done
+
+echo "Silence is golden"
+status=0
+exit
tests/generic/707.out new file +2 -0
new file mode 100644
index 000000000000..8e57a1d8c971
@@ -0,0 +1,2 @@
+QA output created by 707
+Silence is golden
Signature · 1 line
2.35.3