Skip to contentSkip to conversation

xfs: test the deadlock between the AGI and AGF with RENAME_WHITEOUT

PATCH In review kaixuxia 2 participants 3 messages Sep 4, 2019 – Sep 9, 2019

lore
Share & tools
lore

kaixuxia

author
#0 lore raw

PATCH 1/1 xfs: test the deadlock between the AGI and AGF with RENAME_WHITEOUT

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |   2 ++
 tests/xfs/group   |   1 +
 3 files changed, 103 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +100 -0
new file mode 100755
index 0000000..0e95fb7
@@ -0,0 +1,100 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch
+
+# Single AG will cause default xfs_repair to fail. This test need a
+# single AG fs, so ignore the check.
+_require_scratch_nocheck
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# just get a random long name...
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg
+
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+_scratch_mkfs_xfs -bsize=512 -dagcount=1 >> $seqres.full 2>&1 ||
+	_fail "mkfs failed"
+_scratch_mount
+
+files=250000
+
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
#1 lore raw
Quoted 43 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |   2 ++
 tests/xfs/group   |   1 +
 3 files changed, 103 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +24 -0
new file mode 100755
index 0000000..0e95fb7
@@ -0,0 +1,100 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+

tests/xfs/512 line 34

Quoted 10 lines
tests/xfs/512 new file +10 -0
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch

Only _require_scratch_nocheck is suffiecient.

tests/xfs/512 line 38

Quoted 4 lines
tests/xfs/512 new file +4 -0
+
+# Single AG will cause default xfs_repair to fail. This test need a
+# single AG fs, so ignore the check.
+_require_scratch_nocheck

Also need to

. ./common/rename
...

_requires_renameat2

Also, this test requires RENAME_WHITEOUT, I'd suggest enhance src/renameat2.c to support check for if a given rename flag is supported by kernel, and refactor the checks in generic/02[45] and generic/078 into _requires_renameat2 to use the new functionality. e.g.

# without option, behavior stays unchanged, check for renameat2 syscall
# support
_requires_renameat2
# check if renameat2 upports RENAME_WHITEOUT flag
_requires_renameat2 whiteout
# check if renameat2 upports RENAME_EXCHANGE flag
_requires_renameat2 exchange

tests/xfs/512 line 47

Quoted 9 lines
tests/xfs/512 new file +9 -0
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1

Creating 250000 4k files take a long time. Does file content really matters? I guess racing RENAME_WHITEOUT with file creation is enough, is it possible to just create many empty files? e.g.

echo > $file

this saves a lot time.

tests/xfs/512 line 58

Quoted 11 lines
tests/xfs/512 new file +11 -0
+	done
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# just get a random long name...
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg

Better to explain why long file name is required.

Quoted 12 lines
tests/xfs/512 new file +12 -0
+
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir

tests/xfs/512 line 80

Quoted 10 lines
tests/xfs/512 new file +10 -0
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1
+	done

Same here, does creating empty files work?

tests/xfs/512 line 83

Quoted 3 lines
tests/xfs/512 new file +3 -0
+}
+
+_scratch_mkfs_xfs -bsize=512 -dagcount=1 >> $seqres.full 2>&1 ||

This doesn't work because crc is on by default, as crc requires minimum 1k block size. Is 512 block size really needed?

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+	_fail "mkfs failed"
+_scratch_mount
+
+files=250000

If we could reduce file number to create, test could run faster as well. Running test with less than 250000 files couldn't reproduce the deadlock?

Thanks,
Eryu

Quoted 35 lines
tests/xfs/512 new file +13 -0
+
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
author
#2 lore raw
Quoted 48 lines · Eryu Guan
Quoted 47 lines · kaixuxia

There is ABBA deadlock bug between the AGI and AGF when performing rename() with RENAME_WHITEOUT flag, and add this testcase to make sure the rename() call works well.

---
 tests/xfs/512     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/512.out |   2 ++
 tests/xfs/group   |   1 +
 3 files changed, 103 insertions(+)
 create mode 100755 tests/xfs/512
 create mode 100644 tests/xfs/512.out
tests/xfs/512 new file +28 -0
new file mode 100755
index 0000000..0e95fb7
@@ -0,0 +1,100 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Tencent.  All Rights Reserved.
+#
+# FS QA Test 512
+#
+# Test the ABBA deadlock case between the AGI and AGF When performing
+# rename operation with RENAME_WHITEOUT flag.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
Quoted 8 lines

tests/xfs/512 line 34

Quoted 6 lines
tests/xfs/512 new file +6 -0
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_scratch

Only _require_scratch_nocheck is suffiecient.

Yeah, will fix it.

Quoted 22 lines

tests/xfs/512 line 38

Quoted 5 lines
tests/xfs/512 new file +4 -0
+
+# Single AG will cause default xfs_repair to fail. This test need a
+# single AG fs, so ignore the check.
+_require_scratch_nocheck

Also need to

. ./common/rename
...

_requires_renameat2

Also, this test requires RENAME_WHITEOUT, I'd suggest enhance src/renameat2.c to support check for if a given rename flag is supported by kernel, and refactor the checks in generic/02[45] and generic/078 into _requires_renameat2 to use the new functionality. e.g.

# without option, behavior stays unchanged, check for renameat2 syscall
# support
_requires_renameat2
Quoted 6 lines
# check if renameat2 upports RENAME_WHITEOUT flag
_requires_renameat2 whiteout
# check if renameat2 upports RENAME_EXCHANGE flag
_requires_renameat2 exchange

Right, this testcase requires RENAME_WHITEOUT, it is necessary that check whether the given rename flag is supported by kernel. I will try to refactor the corresponding tests and functions.

Quoted 14 lines

tests/xfs/512 line 47

Quoted 9 lines
tests/xfs/512 new file +9 -0
+
+prepare_file()
+{
+	# create many small files for the rename with RENAME_WHITEOUT
+	i=0
+	while [ $i -le $files ]; do
+		file=$SCRATCH_MNT/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1

Creating 250000 4k files take a long time. Does file content really matters? I guess racing RENAME_WHITEOUT with file creation is enough, is it possible to just create many empty files? e.g.

Quoted 3 lines

echo > $file

this saves a lot time.

Yeah, create empty files is better.

Quoted 15 lines

tests/xfs/512 line 58

Quoted 12 lines
tests/xfs/512 new file +11 -0
+	done
+}
+
+rename_whiteout()
+{
+	# create the rename targetdir
+	renamedir=$SCRATCH_MNT/renamedir
+	mkdir $renamedir
+
+	# just get a random long name...
+	longnamepre=FFFsafdsagafsadfagasdjfalskdgakdlsglkasdg

Better to explain why long file name is required.

Will add the explanation. The long file name will increase the possibility that target_dp allocate new blocks(acquire the AGF lock) to store the filename.

Quoted 16 lines
Quoted 16 lines
tests/xfs/512 new file +16 -0
+
+	# now try to do rename with RENAME_WHITEOUT flag
+	i=0
+	while [ $i -le $files ]; do
+		src/renameat2 -w $SCRATCH_MNT/f$i $renamedir/$longnamepre$i >/dev/null 2>&1
+		let i=$i+1
+	done
+}
+
+create_file()
+{
+	# create the targetdir
+	createdir=$SCRATCH_MNT/createdir
+	mkdir $createdir
+
+	# try to create file at the same time to hit the deadlock
Quoted 8 lines

tests/xfs/512 line 80

Quoted 6 lines
tests/xfs/512 new file +6 -0
+	i=0
+	while [ $i -le $files ]; do
+		file=$createdir/f$i
+		$XFS_IO_PROG -f -d -c 'pwrite -b 4k 0 4k' $file >/dev/null 2>&1
+		let i=$i+1
+	done

Same here, does creating empty files work?

Yeah.

Quoted 8 lines

tests/xfs/512 line 83

Quoted 4 lines
tests/xfs/512 new file +3 -0
+}
+
+_scratch_mkfs_xfs -bsize=512 -dagcount=1 >> $seqres.full 2>&1 ||

This doesn't work because crc is on by default, as crc requires minimum 1k block size. Is 512 block size really needed?

Quoted 8 lines

tests/xfs/512 line 87

Quoted 4 lines
tests/xfs/512 new file +4 -0
+	_fail "mkfs failed"
+_scratch_mount
+
+files=250000

If we could reduce file number to create, test could run faster as well. Running test with less than 250000 files couldn't reproduce the deadlock?

I used this magic number in my local test environment and reproduced the deadlock every time. Creating empty files would reduce the run time...

Thanks.

Quoted 39 lines

Thanks,
Eryu

Quoted 35 lines
tests/xfs/512 new file +13 -0
+
+prepare_file
+rename_whiteout &
+create_file &
+
+wait
+echo Silence is golden
+
+# Failure comes in the form of a deadlock.
+
+# success, all done
+status=0
+exit
tests/xfs/512.out new file +2 -0
new file mode 100644
index 0000000..0aabdef
@@ -0,0 +1,2 @@
+QA output created by 512
+Silence is golden
tests/xfs/group +1 -0
index a7ad300..ed250d6 100644
@@ -509,3 +509,4 @@
 509 auto ioctl
 510 auto ioctl quick
 511 auto quick quota
+512 auto rename
Signature · 4 lines
1.8.3.1

-- 
kaixuxia
Signature · 1 line
kaixuxia