fstests: generic/366: add a new test case to verify if certain fio load will hang the filesystem
PATCH v2 1/1 fstests: generic/366: add a new test case to verify if certain fio load will hang the filesystem
[BUG]
During the development to make btrfs pass generic/563 (which needs to make btrfs to support partial folios), generic/095 causes hangs during tests.
The call trace for the hanging process looks like this:
__switch_to+0xf8/0x168 __schedule+0x328/0x8a8 schedule+0x54/0x140 io_schedule+0x44/0x68 folio_wait_bit_common+0x198/0x3f8 __folio_lock+0x24/0x40 extent_write_cache_pages+0x2e0/0x4c0 [btrfs] btrfs_writepages+0x94/0x158 [btrfs] do_writepages+0x74/0x190 filemap_fdatawrite_wbc+0x88/0xc8 __filemap_fdatawrite_range+0x6c/0xa8 filemap_fdatawrite_range+0x1c/0x30 btrfs_start_ordered_extent+0x264/0x2e0 [btrfs] btrfs_lock_and_flush_ordered_range+0x8c/0x160 [btrfs] __get_extent_map+0xa0/0x220 [btrfs] btrfs_do_readpage+0x1bc/0x5d8 [btrfs] btrfs_read_folio+0x50/0xa0 [btrfs] filemap_read_folio+0x54/0x110 filemap_update_page+0x2e0/0x3b8 filemap_get_pages+0x228/0x4d8 filemap_read+0x11c/0x3b8 btrfs_file_read_iter+0x74/0x90 [btrfs] new_sync_read+0xd0/0x1d0 vfs_read+0x1a0/0x1f0
[CAUSE]
The root cause is a btrfs specific behavior that during a folio read, we can trigger writeback of the same folio, which will try to lock the same folio already locked by the read process.
The fix is already sent to the mailing list:
https://lore.kernel.org/linux-btrfs/62bf73ada7be2888d45a787c2b6fd252103a5d25.1729725088.git.wqu@suse.com/
This problem can only happen if all the following conditions are met:
- The sector size of btrfs is smaller than page size To have partial uptodate folios.
- Btrfs won't read the full folio if buffered write is block aligned This is done by the not yet merged patch:
https://lore.kernel.org/linux-btrfs/ac2639ec4e9ac176d33e95ef7ecf008fa6be5461.1727833878.git.wqu@suse.com/
[TEST CASE]
During the debugging of that generic/095 hang, I extracted a minimal reproducer which is much smaller and faster, although it still requires several runs to trigger a hang.
The test case will run the fio workload 32 times by default, which is more than enough to trigger the hang.
---
Changelog:
v2:
- Fix the duplicated _fixed_by_kernel_commit line
- Fix a typo in the commit message
- Update the comments to show the workload and how it hangs btrfs
--- tests/generic/366 | 106 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/366.out | 2 + 2 files changed, 108 insertions(+) create mode 100755 tests/generic/366 create mode 100644 tests/generic/366.out
tests/generic/366 new file +106 -0
@@ -0,0 +1,106 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2024 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 366 +# +# Test if mixed direct read, direct write and buffered write on the same file will +# hang the filesystem. +# +# This is exposed by an incoming btrfs feature, which allows a folio to be +# partial uptodate if the buffered write range is block aligned but not yet +# full folio aligned. +# +# Such behavior makes btrfs to hang reliably under generic/095. +# This is the extracted minimal reproducer for 4k block size and 64K page size. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_require_scratch +_require_odirect +_require_aio + +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: avoid deadlock when reading a partial uptodate folio" + +iterations=$((32 * LOAD_FACTOR)) + +fio_config=$tmp.fio +fio_out=$tmp.fio.out +blksz=`$here/src/min_dio_alignment $SCRATCH_MNT $SCRATCH_DEV` +cat >$fio_config <<EOF +[global] +bs=8k +iodepth=1 +randrepeat=1 +size=256k +directory=$SCRATCH_MNT +numjobs=1 +[job1] +ioengine=sync +bs=512 +direct=1 +rw=randread +filename=file1 +[job2] +ioengine=libaio +rw=randwrite +direct=1 +filename=file1 +[job3] +ioengine=posixaio +rw=randwrite +filename=file1 +EOF +_require_fio $fio_config + +for (( i = 0; i < $iterations; i++)); do + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + # There's a known EIO failure to report collisions between directio and buffered + # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error + # at here. + # + # And for btrfs if sector size < page size, if we have a partial + # uptodate folio caused by a buffered write, e.g: + # + # 0 16K 32K 48K 64K + # | |///////////| + # \- sector Uptodate|Dirty + # + # Then writeback happens and finished, but btrfs' ordered extent not + # yet finished. + # In that case, the folio can be released from the page cache (since + # the folio is not under IO/lock). + # + # Then new buffered writes into the folio happened, re-dirty the folio: + # 0 16K 32K 48K 64K + # |//////////| |///////////| + # \- sector Uptodate|Dirty \- No sector flags + # extent map PINNED + # OE still here + # + # Now read is triggered on that folio. + # Btrfs will need to wait for any existing ordered extents in the folio range, + # that wait will also trigger writeback if the folio is dirty. + # That writeback will happen for range [48K, 64K), but since the whole folio + # is locked for read, writeback will also try to lock the same folio, causing + # a deadlock. + $FIO_PROG $fio_config --ignore_error=,EIO --output=$fio_out + # umount before checking dmesg in case umount triggers any WARNING or Oops + _scratch_unmount + + _check_dmesg _filter_aiodio_dmesg + + echo "=== fio $i/$iterations ===" >> $seqres.full + cat $fio_out >> $seqres.full +done + +echo "Silence is golden" + +# success, all done +status=0 +exit
tests/generic/366.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 366 +Silence is golden
Signature · 1 line
2.46.0
Quoted 85 lines · Qu Wenruo
[BUG]
During the development to make btrfs pass generic/563 (which needs to make btrfs to support partial folios), generic/095 causes hangs during tests.The call trace for the hanging process looks like this:
__switch_to+0xf8/0x168 __schedule+0x328/0x8a8 schedule+0x54/0x140 io_schedule+0x44/0x68 folio_wait_bit_common+0x198/0x3f8 __folio_lock+0x24/0x40 extent_write_cache_pages+0x2e0/0x4c0 [btrfs] btrfs_writepages+0x94/0x158 [btrfs] do_writepages+0x74/0x190 filemap_fdatawrite_wbc+0x88/0xc8 __filemap_fdatawrite_range+0x6c/0xa8 filemap_fdatawrite_range+0x1c/0x30 btrfs_start_ordered_extent+0x264/0x2e0 [btrfs] btrfs_lock_and_flush_ordered_range+0x8c/0x160 [btrfs] __get_extent_map+0xa0/0x220 [btrfs] btrfs_do_readpage+0x1bc/0x5d8 [btrfs] btrfs_read_folio+0x50/0xa0 [btrfs] filemap_read_folio+0x54/0x110 filemap_update_page+0x2e0/0x3b8 filemap_get_pages+0x228/0x4d8 filemap_read+0x11c/0x3b8 btrfs_file_read_iter+0x74/0x90 [btrfs] new_sync_read+0xd0/0x1d0 vfs_read+0x1a0/0x1f0[CAUSE]
The root cause is a btrfs specific behavior that during a folio read, we can trigger writeback of the same folio, which will try to lock the same folio already locked by the read process.The fix is already sent to the mailing list:
https://lore.kernel.org/linux-btrfs/62bf73ada7be2888d45a787c2b6fd252103a5d25.1729725088.git.wqu@suse.com/This problem can only happen if all the following conditions are met:
- The sector size of btrfs is smaller than page size To have partial uptodate folios.
- Btrfs won't read the full folio if buffered write is block aligned This is done by the not yet merged patch:
https://lore.kernel.org/linux-btrfs/ac2639ec4e9ac176d33e95ef7ecf008fa6be5461.1727833878.git.wqu@suse.com/[TEST CASE]
During the debugging of that generic/095 hang, I extracted a minimal reproducer which is much smaller and faster, although it still requires several runs to trigger a hang.The test case will run the fio workload 32 times by default, which is more than enough to trigger the hang.
Signed-off-by Qu Wenruo <[email protected]>---
Changelog:
v2:
- Fix the duplicated_fixed_by_kernel_commitline
- Fix a typo in the commit message
- Update the comments to show the workload and how it hangs btrfs--- tests/generic/366 | 106 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/366.out | 2 + 2 files changed, 108 insertions(+) create mode 100755 tests/generic/366 create mode 100644 tests/generic/366.outtests/generic/366 new file +8
-0@@ -0,0 +1,106 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2024 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 366 +# +# Test if mixed direct read, direct write and buffered write on the same file will +# hang the filesystem.
tests/generic/366 line 18
Quoted 10 lines
tests/generic/366 new file +10
-0+# +# This is exposed by an incoming btrfs feature, which allows a folio to be +# partial uptodate if the buffered write range is block aligned but not yet +# full folio aligned. +# +# Such behavior makes btrfs to hang reliably under generic/095. +# This is the extracted minimal reproducer for 4k block size and 64K page size. +# +. ./common/preamble +_begin_fstest auto quick
^^
rwI think we can keep the "rw" group as g/095 does.
Quoted 64 lines
tests/generic/366 new file +64
-0+ +. ./common/filter + +_require_scratch +_require_odirect +_require_aio + +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: avoid deadlock when reading a partial uptodate folio" + +iterations=$((32 * LOAD_FACTOR)) + +fio_config=$tmp.fio +fio_out=$tmp.fio.out +blksz=`$here/src/min_dio_alignment $SCRATCH_MNT $SCRATCH_DEV` +cat >$fio_config <<EOF +[global] +bs=8k +iodepth=1 +randrepeat=1 +size=256k +directory=$SCRATCH_MNT +numjobs=1 +[job1] +ioengine=sync +bs=512 +direct=1 +rw=randread +filename=file1 +[job2] +ioengine=libaio +rw=randwrite +direct=1 +filename=file1 +[job3] +ioengine=posixaio +rw=randwrite +filename=file1 +EOF +_require_fio $fio_config + +for (( i = 0; i < $iterations; i++)); do + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + # There's a known EIO failure to report collisions between directio and buffered + # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error + # at here. + # + # And for btrfs if sector size < page size, if we have a partial + # uptodate folio caused by a buffered write, e.g: + # + # 0 16K 32K 48K 64K + # | |///////////| + # \- sector Uptodate|Dirty + # + # Then writeback happens and finished, but btrfs' ordered extent not + # yet finished. + # In that case, the folio can be released from the page cache (since + # the folio is not under IO/lock). + # + # Then new buffered writes into the folio happened, re-dirty the folio: + # 0 16K 32K 48K 64K + # |//////////| |///////////| + # \- sector Uptodate|Dirty \- No sector flags
tests/generic/366 line 92
Quoted 10 lines
tests/generic/366 new file +10
-0+ # extent map PINNED + # OE still here + # + # Now read is triggered on that folio. + # Btrfs will need to wait for any existing ordered extents in the folio range, + # that wait will also trigger writeback if the folio is dirty. + # That writeback will happen for range [48K, 64K), but since the whole folio + # is locked for read, writeback will also try to lock the same folio, causing + # a deadlock. + $FIO_PROG $fio_config --ignore_error=,EIO --output=$fio_out
Looks like this test doesn't mix DIO and buffered IO, so this EIO ignoring might not be needed.
tests/generic/366 line 96
Quoted 4 lines
tests/generic/366 new file +4
-0+ # umount before checking dmesg in case umount triggers any WARNING or Oops + _scratch_unmount + + _check_dmesg _filter_aiodio_dmesg
This test removed mmap test part, so this dmesg filter might not be needed either ? If so, don't need to import above "./common/filter" either.
Others looks good to me.
Thanks,
Zorro
Quoted 22 lines
tests/generic/366 new file +10
-0+ + echo "=== fio $i/$iterations ===" >> $seqres.full + cat $fio_out >> $seqres.full +done + +echo "Silence is golden" + +# success, all done +status=0 +exittests/generic/366.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 366 +Silence is goldenSignature · 1 line
2.46.0
Quoted 1 line · Zorro Lang
On Tue, Oct 29, 2024 at 10:05:25AM +1030, Qu Wenruo wrote:
[...]
Quoted 7 lines
tests/generic/366 line 18
Quoted 3 lines
tests/generic/366 new file +3
-0+# +. ./common/preamble +_begin_fstest auto quick^^ rwI think we can keep the "rw" group as g/095 does.
If that's the only missing part, mind to add that rw group when merging?
The remaining two points may not need modification, explained inlined.
Quoted 10 lines
tests/generic/366 line 92
Quoted 7 lines
tests/generic/366 new file +7
-0+ # Now read is triggered on that folio. + # Btrfs will need to wait for any existing ordered extents in the folio range, + # that wait will also trigger writeback if the folio is dirty. + # That writeback will happen for range [48K, 64K), but since the whole folio + # is locked for read, writeback will also try to lock the same folio, causing + # a deadlock. + $FIO_PROG $fio_config --ignore_error=,EIO --output=$fio_outLooks like this test doesn't mix DIO and buffered IO, so this EIO ignoring might not be needed.
I'm not sure about the EIO part, but at least job2 and job3 are all doing writes, one is buffered and the other is direct. So it's still mixing both.
Quoted 8 lines
tests/generic/366 line 96
Quoted 5 lines
tests/generic/366 new file +4
-0+ # umount before checking dmesg in case umount triggers any WARNING or Oops + _scratch_unmount + + _check_dmesg _filter_aiodio_dmesgThis test removed mmap test part, so this dmesg filter might not be needed either ? If so, don't need to import above "./common/filter" either.
Since it's still mixing buffered and direct IO, I can hit the aiodio error message just like this:
[91619.077752] BTRFS warning (device dm-2): read-write for sector size 4096 with page size 65536 is experimental [91619.086204] BTRFS info (device dm-2): checking UUID tree [91619.473510] Page cache invalidation failure on direct I/O. Possible
data corruption due to collision with buffered I/O!
So I'm afraid we need the filter too.
Thanks,
Qu
Quoted 30 lines
Others looks good to me.
Thanks,
ZorroQuoted 22 lines
tests/generic/366 new file +10
-0+ + echo "=== fio $i/$iterations ===" >> $seqres.full + cat $fio_out >> $seqres.full +done + +echo "Silence is golden" + +# success, all done +status=0 +exittests/generic/366.out new file +2
-1@@ -0,0 +1,2 @@ +QA output created by 366 +Silence is golden --2.46.0
Zorro Lang
Reviewed-byQuoted 44 lines · Qu Wenruo
Quoted 1 line · Zorro Lang
On Tue, Oct 29, 2024 at 10:05:25AM +1030, Qu Wenruo wrote:
[...]
Quoted 7 lines
tests/generic/366 line 18
Quoted 3 lines
tests/generic/366 new file +3
-0+# +. ./common/preamble +_begin_fstest auto quick^^ rwI think we can keep the "rw" group as g/095 does.
If that's the only missing part, mind to add that rw group when merging?
The remaining two points may not need modification, explained inlined.
Quoted 10 lines
tests/generic/366 line 92
Quoted 7 lines
tests/generic/366 new file +7
-0+ # Now read is triggered on that folio. + # Btrfs will need to wait for any existing ordered extents in the folio range, + # that wait will also trigger writeback if the folio is dirty. + # That writeback will happen for range [48K, 64K), but since the whole folio + # is locked for read, writeback will also try to lock the same folio, causing + # a deadlock. + $FIO_PROG $fio_config --ignore_error=,EIO --output=$fio_outLooks like this test doesn't mix DIO and buffered IO, so this EIO ignoring might not be needed.
I'm not sure about the EIO part, but at least job2 and job3 are all doing writes, one is buffered and the other is direct. So it's still mixing both.
Quoted 8 lines
tests/generic/366 line 96
Quoted 5 lines
tests/generic/366 new file +4
-0+ # umount before checking dmesg in case umount triggers any WARNING or Oops + _scratch_unmount + + _check_dmesg _filter_aiodio_dmesgThis test removed mmap test part, so this dmesg filter might not be needed either ? If so, don't need to import above "./common/filter" either.
Since it's still mixing buffered and direct IO, I can hit the aiodio error message just like this:
Quoted 7 lines
[91619.077752] BTRFS warning (device dm-2): read-write for sector size 4096 with page size 65536 is experimental [91619.086204] BTRFS info (device dm-2): checking UUID tree [91619.473510] Page cache invalidation failure on direct I/O. Possibledata corruption due to collision with buffered I/O!
So I'm afraid we need the filter too.
Oh, thanks for your confirming. I can merge this patch with that tiny change. If there's not more review points from btrfs list, I'll merge it. It looks good to me now.
Quoted 35 lines
Thanks,
QuQuoted 30 lines
Others looks good to me.
Thanks,
ZorroQuoted 22 lines
tests/generic/366 new file +10
-0+ + echo "=== fio $i/$iterations ===" >> $seqres.full + cat $fio_out >> $seqres.full +done + +echo "Silence is golden" + +# success, all done +status=0 +exittests/generic/366.out new file +2
-1@@ -0,0 +1,2 @@ +QA output created by 366 +Silence is golden --2.46.0
Quoted 170 lines · Qu Wenruo
[BUG]
During the development to make btrfs pass generic/563 (which needs to make btrfs to support partial folios), generic/095 causes hangs during tests.The call trace for the hanging process looks like this:
__switch_to+0xf8/0x168 __schedule+0x328/0x8a8 schedule+0x54/0x140 io_schedule+0x44/0x68 folio_wait_bit_common+0x198/0x3f8 __folio_lock+0x24/0x40 extent_write_cache_pages+0x2e0/0x4c0 [btrfs] btrfs_writepages+0x94/0x158 [btrfs] do_writepages+0x74/0x190 filemap_fdatawrite_wbc+0x88/0xc8 __filemap_fdatawrite_range+0x6c/0xa8 filemap_fdatawrite_range+0x1c/0x30 btrfs_start_ordered_extent+0x264/0x2e0 [btrfs] btrfs_lock_and_flush_ordered_range+0x8c/0x160 [btrfs] __get_extent_map+0xa0/0x220 [btrfs] btrfs_do_readpage+0x1bc/0x5d8 [btrfs] btrfs_read_folio+0x50/0xa0 [btrfs] filemap_read_folio+0x54/0x110 filemap_update_page+0x2e0/0x3b8 filemap_get_pages+0x228/0x4d8 filemap_read+0x11c/0x3b8 btrfs_file_read_iter+0x74/0x90 [btrfs] new_sync_read+0xd0/0x1d0 vfs_read+0x1a0/0x1f0[CAUSE]
The root cause is a btrfs specific behavior that during a folio read, we can trigger writeback of the same folio, which will try to lock the same folio already locked by the read process.The fix is already sent to the mailing list:
https://lore.kernel.org/linux-btrfs/62bf73ada7be2888d45a787c2b6fd252103a5d25.1729725088.git.wqu@suse.com/This problem can only happen if all the following conditions are met:
- The sector size of btrfs is smaller than page size To have partial uptodate folios.
- Btrfs won't read the full folio if buffered write is block aligned This is done by the not yet merged patch:
https://lore.kernel.org/linux-btrfs/ac2639ec4e9ac176d33e95ef7ecf008fa6be5461.1727833878.git.wqu@suse.com/[TEST CASE]
During the debugging of that generic/095 hang, I extracted a minimal reproducer which is much smaller and faster, although it still requires several runs to trigger a hang.The test case will run the fio workload 32 times by default, which is more than enough to trigger the hang.
Signed-off-by Qu Wenruo <[email protected]>---
Changelog:
v2:
- Fix the duplicated_fixed_by_kernel_commitline
- Fix a typo in the commit message
- Update the comments to show the workload and how it hangs btrfs--- tests/generic/366 | 106 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/366.out | 2 + 2 files changed, 108 insertions(+)create mode 100755 tests/generic/366
create mode 100644 tests/generic/366.outtests/generic/366 new file +93
-0@@ -0,0 +1,106 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2024 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 366 +# +# Test if mixed direct read, direct write and buffered write on the same file will +# hang the filesystem. +# +# This is exposed by an incoming btrfs feature, which allows a folio to be +# partial uptodate if the buffered write range is block aligned but not yet +# full folio aligned. +# +# Such behavior makes btrfs to hang reliably under generic/095. +# This is the extracted minimal reproducer for 4k block size and 64K page size. +# +. ./common/preamble +_begin_fstest auto quick + +. ./common/filter + +_require_scratch +_require_odirect +_require_aio + +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: avoid deadlock when reading a partial uptodate folio" + +iterations=$((32 * LOAD_FACTOR)) + +fio_config=$tmp.fio +fio_out=$tmp.fio.out +blksz=`$here/src/min_dio_alignment $SCRATCH_MNT $SCRATCH_DEV` +cat >$fio_config <<EOF +[global] +bs=8k +iodepth=1 +randrepeat=1 +size=256k +directory=$SCRATCH_MNT +numjobs=1 +[job1] +ioengine=sync +bs=512 +direct=1 +rw=randread +filename=file1 +[job2] +ioengine=libaio +rw=randwrite +direct=1 +filename=file1 +[job3] +ioengine=posixaio +rw=randwrite +filename=file1 +EOF +_require_fio $fio_config + +for (( i = 0; i < $iterations; i++)); do + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + # There's a known EIO failure to report collisions between directio and buffered + # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error + # at here. + # + # And for btrfs if sector size < page size, if we have a partial + # uptodate folio caused by a buffered write, e.g: + # + # 0 16K 32K 48K 64K + # | |///////////| + # \- sector Uptodate|Dirty + # + # Then writeback happens and finished, but btrfs' ordered extent not + # yet finished. + # In that case, the folio can be released from the page cache (since + # the folio is not under IO/lock). + # + # Then new buffered writes into the folio happened, re-dirty the folio: + # 0 16K 32K 48K 64K + # |//////////| |///////////| + # \- sector Uptodate|Dirty \- No sector flags + # extent map PINNED + # OE still here + # + # Now read is triggered on that folio. + # Btrfs will need to wait for any existing ordered extents in the folio range, + # that wait will also trigger writeback if the folio is dirty. + # That writeback will happen for range [48K, 64K), but since the whole folio + # is locked for read, writeback will also try to lock the same folio, causing + # a deadlock. + $FIO_PROG $fio_config --ignore_error=,EIO --output=$fio_out + # umount before checking dmesg in case umount triggers any WARNING or Oops
tests/generic/366 line 103
Quoted 10 lines
tests/generic/366 new file +10
-0+ _scratch_unmount + + _check_dmesg _filter_aiodio_dmesg + + echo "=== fio $i/$iterations ===" >> $seqres.full + cat $fio_out >> $seqres.full +done + +echo "Silence is golden" +
Thanks!
Quoted 11 lines
tests/generic/366 new file +3
-0+# success, all done +status=0 +exittests/generic/366.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 366 +Silence is golden