Skip to contentSkip to conversation

xfs: test the deadlock between the AGI and AGF with RENAME_WHITEOUT

PATCH Superseded kaixuxia 4 participants 7 messages Sep 11, 2019 – Sep 16, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

kaixuxia

author
#0 lore raw

PATCH 2/2 xfs: test the deadlock between the AGI and AGF with RENAME_WHITEOUT

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +99 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"
+_scratch_mount
+
+files=250000
+
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
#1 lore raw
Quoted 44 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +25 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks

tests/xfs/512 line 35

Quoted 10 lines
tests/xfs/512 new file +10 -0
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

tests/xfs/512 line 46

Quoted 11 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done

The same goes for the other while loops below that increment up to $files.

tests/xfs/512 line 58

Quoted 12 lines
tests/xfs/512 new file +12 -0
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

	prefix=`for i in $(seq 0 245); do echo -n a; done`
Quoted 12 lines
tests/xfs/512 new file +12 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir

tests/xfs/512 line 80

Quoted 10 lines
tests/xfs/512 new file +10 -0
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

tests/xfs/512 line 96

Quoted 9 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

Brian

Quoted 25 lines
tests/xfs/512 new file +3 -0
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
#2 lore raw
Quoted 50 lines · Brian Foster
Quoted 49 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +30 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
Quoted 8 lines

tests/xfs/512 line 35

Quoted 5 lines
tests/xfs/512 new file +5 -0
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

There was a comment in v1, but not in this v2, we should keep that comment.

Quoted 15 lines

tests/xfs/512 line 46

Quoted 12 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

Quoted 6 lines
	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done

The same goes for the other while loops below that increment up to $files.

Agreed, but looks like echo (which is a bash builtin) is faster than touch (which requires forking new process every loop).

Quoted 18 lines

tests/xfs/512 line 58

Quoted 13 lines
tests/xfs/512 new file +12 -0
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

Quoted 1 line
	prefix=`for i in $(seq 0 245); do echo -n a; done`

Or

	prefix=`$PERL_PROG -e 'print "a"x256;'`

? Which seems a bit simpler to me.

Quoted 44 lines
Quoted 13 lines
tests/xfs/512 new file +12 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir

tests/xfs/512 line 80

Quoted 10 lines
tests/xfs/512 new file +10 -0
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

Quoted 15 lines

tests/xfs/512 line 96

Quoted 9 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

The deadlock has been fixed, so I think it's fine to leave dangerous group.

Quoted 2 lines

Brian

Thanks a lot for the review!

Eryu

Quoted 26 lines
Quoted 26 lines
tests/xfs/512 new file +3 -0
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
#3 lore raw
Quoted 82 lines · Eryu Guan
Quoted 50 lines · Brian Foster
Quoted 49 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +30 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
Quoted 8 lines

tests/xfs/512 line 35

Quoted 5 lines
tests/xfs/512 new file +5 -0
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

There was a comment in v1, but not in this v2, we should keep that comment.

Quoted 19 lines

tests/xfs/512 line 46

Quoted 12 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done
Quoted 6 lines
Quoted 2 lines

The same goes for the other while loops below that increment up to $files.

Agreed, but looks like echo (which is a bash builtin) is faster than touch (which requires forking new process every loop).

Ah, interesting. I suppose that makes sense if there's tangible benefit. Would that benefit stand if we created an internal _touch helper or some such instead of open-coding it everywhere?

Quoted 86 lines
Quoted 18 lines

tests/xfs/512 line 58

Quoted 13 lines
tests/xfs/512 new file +12 -0
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

Quoted 1 line
	prefix=`for i in $(seq 0 245); do echo -n a; done`

Or

	prefix=`$PERL_PROG -e 'print "a"x256;'`

? Which seems a bit simpler to me.

Quoted 44 lines
Quoted 13 lines
tests/xfs/512 new file +12 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir

tests/xfs/512 line 80

Quoted 10 lines
tests/xfs/512 new file +10 -0
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

Quoted 15 lines

tests/xfs/512 line 96

Quoted 9 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

Quoted 3 lines

The deadlock has been fixed, so I think it's fine to leave dangerous group.

Do you mean to leave it in or out?

In general, what's the approach for dealing with dangerous tests that are no longer dangerous? Leave them indefinitely or remove them after a period of time? I tend to skip dangerous tests on regression runs just because I'm not looking for a deadlock or crash to disrupt a long running test.

Brian

Quoted 33 lines
Quoted 2 lines

Brian

Thanks a lot for the review!

Eryu

Quoted 26 lines
Quoted 26 lines
tests/xfs/512 new file +3 -0
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
#4 lore raw
Quoted 86 lines · Brian Foster
Quoted 85 lines · Eryu Guan
Quoted 50 lines · Brian Foster
Quoted 49 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +30 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
Quoted 8 lines

tests/xfs/512 line 35

Quoted 5 lines
tests/xfs/512 new file +5 -0
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

There was a comment in v1, but not in this v2, we should keep that comment.

Quoted 21 lines

tests/xfs/512 line 46

Quoted 12 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done

The same goes for the other while loops below that increment up to $files.

Quoted 7 lines
Quoted 3 lines

Agreed, but looks like echo (which is a bash builtin) is faster than touch (which requires forking new process every loop).

Ah, interesting. I suppose that makes sense if there's tangible benefit. Would that benefit stand if we created an internal _touch helper or some such instead of open-coding it everywhere?

IMHO, creating a _touch helper doesn't gain much than open-coding it, it's probably just a one-liner helper anyway.

Quoted 87 lines
Quoted 87 lines
Quoted 19 lines
Quoted 4 lines
tests/xfs/512 new file +2 -0
+}
+

tests/xfs/512 line 58

Quoted 10 lines
tests/xfs/512 new file +10 -0
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

Quoted 1 line
	prefix=`for i in $(seq 0 245); do echo -n a; done`

Or

	prefix=`$PERL_PROG -e 'print "a"x256;'`

? Which seems a bit simpler to me.

Quoted 59 lines
Quoted 13 lines
tests/xfs/512 new file +12 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir

tests/xfs/512 line 80

Quoted 10 lines
tests/xfs/512 new file +10 -0
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

tests/xfs/512 line 96

Quoted 9 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

Quoted 5 lines
Quoted 3 lines

The deadlock has been fixed, so I think it's fine to leave dangerous group.

Do you mean to leave it in or out?

Don't add 'dangerous' group.

Quoted 6 lines

In general, what's the approach for dealing with dangerous tests that are no longer dangerous? Leave them indefinitely or remove them after a period of time? I tend to skip dangerous tests on regression runs just because I'm not looking for a deadlock or crash to disrupt a long running test.

I think the definition of 'dangerous' group is a bit vague over time (at least to me), I'd define it as 'would crash or hang the test machine and cause tests to be interrupted & re-run'. So I'd remove them after the problem is fixed in latest upstream kernel/userspace tools.

And I tend to not add new 'dangerous' tests, as we usually push such tests to fstests upstream only after the fixes are upstreamed, and only add new 'dangerous' tests when we know the fix is not going to be there for the near future.

Thanks,
Eryu

author
#5 lore raw
In reply to Brian Foster
Quoted 50 lines · Brian Foster
Quoted 49 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +30 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
Quoted 8 lines

tests/xfs/512 line 35

Quoted 5 lines
tests/xfs/512 new file +5 -0
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

Will add the comment in the next version.

Quoted 59 lines

tests/xfs/512 line 46

Quoted 12 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done

The same goes for the other while loops below that increment up to $files.

tests/xfs/512 line 58

Quoted 12 lines
tests/xfs/512 new file +12 -0
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

	prefix=`for i in $(seq 0 245); do echo -n a; done`
Quoted 18 lines
tests/xfs/512 new file +18 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
Quoted 8 lines

tests/xfs/512 line 47

Quoted 4 lines
tests/xfs/512 new file +4 -0
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

Right.

Quoted 7 lines

tests/xfs/512 line 83

Quoted 4 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

The smaller block size, the more frequently that the blocks are allocated when creating files...

Quoted 9 lines

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

Hmm, 250000 is just a random number big enough for this test. The smaller file counts maybe is enough. Actually, the create_file() call run fast than the rename_whiteout() call, so we can choose two different file counts for them. Anyway, I would address the file counts problem in V3...

Quoted 44 lines

tests/xfs/512 line 96

Quoted 10 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

Brian

Quoted 25 lines
tests/xfs/512 new file +3 -0
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
Signature · 1 line
kaixuxia
#6 lore raw
Quoted 132 lines · kaixuxia
Quoted 50 lines · Brian Foster
Quoted 49 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +30 -0
new file mode 100755
index 0000000..754f102
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/renameat2
+
+rm -f $seqres.full
Quoted 8 lines

tests/xfs/512 line 35

Quoted 5 lines
tests/xfs/512 new file +5 -0
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch_nocheck

Why _nocheck? AFAICT the filesystem shouldn't end up intentionally corrupted.

Will add the comment in the next version.

Quoted 59 lines

tests/xfs/512 line 46

Quoted 12 lines
tests/xfs/512 new file +11 -0
+_requires_renameat2 whiteout
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done

Something like the following is a bit more simple, IMO:

	for i in $(seq 1 $files); do
		touch $SCRATCH_MNT/f.$i
	done

The same goes for the other while loops below that increment up to $files.

tests/xfs/512 line 58

Quoted 12 lines
tests/xfs/512 new file +12 -0
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# a long filename could increase the possibility that target_dp
+	# allocate new blocks(acquire the AGF lock) to store the filename
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+

The max filename length is 256 bytes. You could do something like the following to increase name length (leaving room for the file index and terminating NULL) if it helps the test:

	prefix=`for i in $(seq 0 245); do echo -n a; done`
Quoted 18 lines
tests/xfs/512 new file +18 -0
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
Quoted 8 lines

tests/xfs/512 line 47

Quoted 4 lines
tests/xfs/512 new file +4 -0
+		echo > $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}

You could generalize this function to take a target directory parameter and just call it twice (once to prepare and again for the create workload).

Right.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+
+_scratch_mkfs_xfs -bsize=1024 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"

Why -bsize=1k? Does that make the reproducer more effective?

The smaller block size, the more frequently that the blocks are allocated when creating files...

If 1k block size is not a "must have", we could leave that part to MKFS_OPTIONS and don't specify block size in the test, i.e. testing with different MKFS_OPTIONS could cover various block sizes, e.g.

MKFS_OPTIONS="-bsize=1k,-m crc=1,reflink=1" ./check ...

Quoted 10 lines
Quoted 9 lines

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+_scratch_mount
+
+files=250000
+

Have you tested effectiveness of reproducing the issue with smaller file counts? A brief comment here to document where the value comes from might be useful. Somewhat related, how long does this test take on fixed kernels?

Quoted 4 lines

Hmm, 250000 is just a random number big enough for this test. The smaller file counts maybe is enough. Actually, the create_file() call run fast than the rename_whiteout() call, so we can choose two different file counts for them. Anyway, I would address the file counts problem in V3...

Yeah, it's good to figure out a small file count (no need to be smallest) that could reproduce the deadlock reliably.

Thanks,
Eryu

Quoted 47 lines
Quoted 44 lines

tests/xfs/512 line 96

Quoted 10 lines
tests/xfs/512 new file +9 -0
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+

I wonder if this should be in the dangerous group as well. I go back and forth on that though because I tend to filter out dangerous tests and the test won't be so risky once the fix proliferates. Perhaps that's just a matter of removing it from the dangerous group after a long enough period of time.

Brian

Quoted 25 lines
tests/xfs/512 new file +3 -0
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
Signature · 1 line
kaixuxia