Skip to contentSkip to conversation

generic/new: drop caches while freeze

PATCH Superseded Murphy Zhou 2 participants 5 messages Dec 13, 2023 – Dec 15, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic/new: drop caches while freeze

---
 tests/generic/734     | 34 ++++++++++++++++++++++++++++++++++
 tests/generic/734.out |  2 ++
 2 files changed, 36 insertions(+)
 create mode 100755 tests/generic/734
 create mode 100644 tests/generic/734.out
tests/generic/734 new file +34 -0
new file mode 100755
index 00000000..8ca91930
@@ -0,0 +1,34 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 734
+#
+# Test possible deadlock of umount and reclaim memory
+# when there are EOF blocks in files.
+#
+. ./common/preamble
+_begin_fstest freeze auto quick
+
+_supported_fs generic
+_require_scratch
+
+$XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+
+sync
+
+fsfreeze -f $SCRATCH_MNT
+
+# This will hang if bug reproduces
+echo 3 > /proc/sys/vm/drop_caches &
+
+# Manually unfreeze after a while
+sleep 5
+fsfreeze -u $SCRATCH_MNT
+
+wait
+# success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/734.out new file +2 -0
new file mode 100644
index 00000000..4299839b
@@ -0,0 +1,2 @@
+QA output created by 734
+Silence is golden
Signature · 1 line
2.31.1
#1 lore raw
Quoted 3 lines · Murphy Zhou
---
 tests/generic/734     | 34 ++++++++++++++++++++++++++++++++++

Hi Murphy,

Thanks for this new test. Sorry that g/734 has been taken last weekend, please rebase to the latest for-next branch.

Quoted 11 lines
 tests/generic/734.out |  2 ++
 2 files changed, 36 insertions(+)
 create mode 100755 tests/generic/734
 create mode 100644 tests/generic/734.out
tests/generic/734 new file +0 -0
new file mode 100755
index 00000000..8ca91930

tests/generic/734 line 7

Quoted 7 lines
tests/generic/734 new file +7 -0
@@ -0,0 +1,34 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 734
+#
+# Test possible deadlock of umount and reclaim memory
+# when there are EOF blocks in files.

Is there a known bug fix for this case? If yes, please specify that by:
"_fixed_by_kernel_commit xxxxxxxxxx xxxxxxxxxxxxxxxx".

tests/generic/734 line 13

Quoted 6 lines
tests/generic/734 new file +6 -0
+#
+. ./common/preamble
+_begin_fstest freeze auto quick
+
+_supported_fs generic
+_require_scratch

This's a geneirc case, not all fs support fsfreeze, so ... _require_freeze

And ...

_scratch_mkfs >> $seqres.full ??
_scratch_mount ??

tests/generic/734 line 15

Quoted 2 lines
tests/generic/734 new file +2 -0
+
+$XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full

Could you explain why there're two duplicated lines below? If it's intended, better to have a comment to explain why do it twice (not once or more times).

tests/generic/734 line 21

Quoted 6 lines
tests/generic/734 new file +6 -0
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+
+sync
+
+fsfreeze -f $SCRATCH_MNT

All cases in fstests do fs freeze by xfs_freeze. If there's not special reason, better to keep consistency. Due to the _require_freeze checks that too.

tests/generic/734 line 24

Quoted 3 lines
tests/generic/734 new file +3 -0
+
+# This will hang if bug reproduces
+echo 3 > /proc/sys/vm/drop_caches &

I'm wondering, if the data in cache has been written back and been cleared before fsfreeze or this step, what will happen? If the cache data is necessary, should we load/read more data/files to provide that?

tests/generic/734 line 30

Quoted 6 lines
tests/generic/734 new file +6 -0
+
+# Manually unfreeze after a while
+sleep 5
+fsfreeze -u $SCRATCH_MNT
+
+wait

As there's a fs freeze in this case, so we'd better to do a "forced" unfreeze in _cleanup, to make sure the fs will be unfreezed 100%, no matter what. So you can move these two lines to _cleanup.

Quoted 16 lines
tests/generic/734 new file +4 -0
+# success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/734.out new file +2 -0
new file mode 100644
index 00000000..4299839b
@@ -0,0 +1,2 @@
+QA output created by 734
+Silence is golden
Signature · 1 line
2.31.1
#2 lore raw

Hi Zorro,

Quoted 11 lines · Zorro Lang
Quoted 3 lines · Murphy Zhou
---
 tests/generic/734     | 34 ++++++++++++++++++++++++++++++++++

Hi Murphy,

Thanks for this new test. Sorry that g/734 has been taken last weekend, please rebase to the latest for-next branch.

Got it.

Quoted 6 lines
Quoted 6 lines
 tests/generic/734.out |  2 ++
 2 files changed, 36 insertions(+)
 create mode 100755 tests/generic/734
 create mode 100644 tests/generic/734.out
Quoted 16 lines

tests/generic/734 line 7

Quoted 13 lines
tests/generic/734 new file +7 -0
new file mode 100755
index 00000000..8ca91930
@@ -0,0 +1,34 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 734
+#
+# Test possible deadlock of umount and reclaim memory
+# when there are EOF blocks in files.

Is there a known bug fix for this case? If yes, please specify that by:
"_fixed_by_kernel_commit xxxxxxxxxx xxxxxxxxxxxxxxxx".

Maybe a patchset but I'm not sure of it. In this situation I think we can skip this part?

Quoted 10 lines

tests/generic/734 line 13

Quoted 7 lines
tests/generic/734 new file +6 -0
+#
+. ./common/preamble
+_begin_fstest freeze auto quick
+
+_supported_fs generic
+_require_scratch

This's a geneirc case, not all fs support fsfreeze, so ... _require_freeze

Got it.

Quoted 5 lines

And ...

_scratch_mkfs >> $seqres.full ??
_scratch_mount ??

OK.

Quoted 6 lines

tests/generic/734 line 15

Quoted 3 lines
tests/generic/734 new file +2 -0
+
+$XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full

Could you explain why there're two duplicated lines below? If it's intended, better to have a comment to explain why do it twice (not once or more times).

OK.

Quoted 11 lines

tests/generic/734 line 21

Quoted 7 lines
tests/generic/734 new file +6 -0
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+cat $SCRATCH_MNT/testfile >> $SCRATCH_MNT/testfile1
+
+sync
+
+fsfreeze -f $SCRATCH_MNT

All cases in fstests do fs freeze by xfs_freeze. If there's not special reason, better to keep consistency. Due to the _require_freeze checks that too.

OK.

Quoted 8 lines

tests/generic/734 line 24

Quoted 4 lines
tests/generic/734 new file +3 -0
+
+# This will hang if bug reproduces
+echo 3 > /proc/sys/vm/drop_caches &

I'm wondering, if the data in cache has been written back and been cleared before fsfreeze or this step, what will happen? If the cache data is necessary, should we load/read more data/files to provide that?

Agree. I'll add more files.

Quoted 11 lines

tests/generic/734 line 30

Quoted 7 lines
tests/generic/734 new file +6 -0
+
+# Manually unfreeze after a while
+sleep 5
+fsfreeze -u $SCRATCH_MNT
+
+wait

As there's a fs freeze in this case, so we'd better to do a "forced" unfreeze in _cleanup, to make sure the fs will be unfreezed 100%, no matter what. So you can move these two lines to _cleanup.

Good idea!

Thanks for reviewing!

Murphy

Quoted 18 lines
Quoted 17 lines
tests/generic/734 new file +4 -0
+# success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/734.out new file +2 -1
new file mode 100644
index 00000000..4299839b
@@ -0,0 +1,2 @@
+QA output created by 734
+Silence is golden
--

2.31.1

#3 lore raw

PATCH v2 1/1 generic/new: drop caches while freeze

---
v2:
re-number to 738
mkfs and mount scratch in the begining
create more files and comment why
unfreeze in cleanup

 tests/generic/738     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/738.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/738
 create mode 100644 tests/generic/738.out
tests/generic/738 new file +54 -0
new file mode 100755
index 00000000..9f1f9d6b
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 738
+#
+# Test possible deadlock of umount and reclaim memory
+# when there are EOF blocks in files.
+#
+. ./common/preamble
+_begin_fstest freeze auto quick
+
+_cleanup()
+{
+	xfs_freeze -u $SCRATCH_MNT
+	wait
+	cd /
+	rm -r -f $tmp.*
+}
+
+_supported_fs generic
+_require_scratch
+_require_freeze
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+
+create_eof_block_file()
+{
+	# Create several EOF blocks in the new file
+	for j in $(seq 1 5); do
+		cat $SCRATCH_MNT/testfile >> $1
+	done
+}
+
+$XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full
+
+# Create enough files to make sure there is enough cache
+for i in $(seq 0 1024); do
+	create_eof_block_file $SCRATCH_MNT/$i
+done
+
+sync
+
+xfs_freeze -f $SCRATCH_MNT
+
+# This will hang if bug reproduces
+echo 3 > /proc/sys/vm/drop_caches &
+
+# Wait a while before exiting and unfreezing.
+sleep 3
+# success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/738.out new file +2 -0
new file mode 100644
index 00000000..5719515a
@@ -0,0 +1,2 @@
+QA output created by 738
+Silence is golden
Signature · 1 line
2.31.1
Reviewed-by on v2 1/1
#4 lore raw
Quoted 7 lines · Murphy Zhou

---
v2:
re-number to 738
mkfs and mount scratch in the begining
create more files and comment why
unfreeze in cleanup

This version looks better to me. I'll merge this patch with a few of small changes. Thanks for this new coverage!

Quoted 78 lines
 tests/generic/738     | 54 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/738.out |  2 ++
 2 files changed, 56 insertions(+)
 create mode 100755 tests/generic/738
 create mode 100644 tests/generic/738.out
tests/generic/738 new file +54 -0
new file mode 100755
index 00000000..9f1f9d6b
@@ -0,0 +1,54 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 738
+#
+# Test possible deadlock of umount and reclaim memory
+# when there are EOF blocks in files.
+#
+. ./common/preamble
+_begin_fstest freeze auto quick
+
+_cleanup()
+{
+	xfs_freeze -u $SCRATCH_MNT
+	wait
+	cd /
+	rm -r -f $tmp.*
+}
+
+_supported_fs generic
+_require_scratch
+_require_freeze
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+
+create_eof_block_file()
+{
+	# Create several EOF blocks in the new file
+	for j in $(seq 1 5); do
+		cat $SCRATCH_MNT/testfile >> $1
+	done
+}
+
+$XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full
+
+# Create enough files to make sure there is enough cache
+for i in $(seq 0 1024); do
+	create_eof_block_file $SCRATCH_MNT/$i
+done
+
+sync
+
+xfs_freeze -f $SCRATCH_MNT
+
+# This will hang if bug reproduces
+echo 3 > /proc/sys/vm/drop_caches &
+
+# Wait a while before exiting and unfreezing.
+sleep 3
+# success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/738.out new file +2 -0
new file mode 100644
index 00000000..5719515a
@@ -0,0 +1,2 @@
+QA output created by 738
+Silence is golden
Signature · 1 line
2.31.1