error writing primary super block on zoned btrfs
Hi Naohiro, (and willy for insights on the pagecache, see below),
when running plain fsx on zoned btrfs on a null_blk devices as below:
dev="/sys/kernel/config/nullb/nullb1"
size=12800 # MB
mkdir ${dev}
echo 2 > "${dev}"/submit_queues
echo 2 > "${dev}"/queue_mode
echo 2 > "${dev}"/irqmode
echo "${size}" > "${dev}"/size
echo 1 > "${dev}"/zoned
echo 0 > "${dev}"/zone_nr_conv
echo 128 > "${dev}"/zone_size
echo 96 > "${dev}"/zone_capacity
echo 14 > "${dev}"/zone_max_active
echo 1 > "${dev}"/memory_backed
echo 1000000 > "${dev}"/completion_nsec
echo 1 > "${dev}"/power
mkfs.btrfs -m single /dev/nullb1
mount /dev/nullb1 /mnt/test/
~/xfstests-dev/ltp/fsx /mnt/test/foobar
fsx will eventually after ~10 minutes fail with the following left in dmesg:
[ 1185.480200] BTRFS error (device nullb1): error writing primary super block to device 1 [ 1185.480988] BTRFS: error (device nullb1) in write_all_supers:4488: errno=-5 IO failure (1 errors while writing supers) [ 1185.481971] BTRFS info (device nullb1: state E): forced readonly [ 1185.482521] BTRFS: error (device nullb1: state EA) in btrfs_sync_log:3341: errno=-5 IO failure
I tracked this down to the find_get_page call in wait_dev_supers returning NULL, and digging furter it seems to come from xa_is_value() in __filemap_get_folio returnin true. I'm not sure why we'd see a value here in the block device mapping and why that only happens in zoned mode (the same config on regular device ran for 10 hours last night).
Quoted 32 lines · Christoph Hellwig
Hi Naohiro, (and willy for insights on the pagecache, see below),
when running plain fsx on zoned btrfs on a
null_blkdevices as below:dev="/sys/kernel/config/nullb/nullb1"
size=12800 # MBmkdir ${dev}
echo 2 > "${dev}"/submit_queues
echo 2 > "${dev}"/queue_mode
echo 2 > "${dev}"/irqmode
echo "${size}" > "${dev}"/size
echo 1 > "${dev}"/zoned
echo 0 > "${dev}"/zone_nr_conv
echo 128 > "${dev}"/zone_size
echo 96 > "${dev}"/zone_capacity
echo 14 > "${dev}"/zone_max_active
echo 1 > "${dev}"/memory_backed
echo 1000000 > "${dev}"/completion_nsec
echo 1 > "${dev}"/power
mkfs.btrfs -m single/dev/nullb1
mount/dev/nullb1/mnt/test/
~/xfstests-dev/ltp/fsx/mnt/test/foobarfsx will eventually after ~10 minutes fail with the following left in dmesg:
[ 1185.480200] BTRFS error (device nullb1): error writing primary super block to device 1 [ 1185.480988] BTRFS: error (device nullb1) in write_all_supers:4488: errno=-5 IO failure (1 errors while writing supers) [ 1185.481971] BTRFS info (device nullb1: state E): forced readonly [ 1185.482521] BTRFS: error (device nullb1: state EA) in btrfs_sync_log:3341: errno=-5 IO failure
Quoted 6 lines
I tracked this down to the
find_get_pagecall inwait_dev_supersreturning NULL, and digging furter it seems to come fromxa_is_value()in__filemap_get_folioreturnin true. I'm not sure why we'd see a value here in the block device mapping and why that only happens in zoned mode (the same config on regular device ran for 10 hours last night).
A "value" entry in the block device's i_pages will be a shadow entry -- that is, the page has reached the end of the LRU list and been discarded, so we made a note that we would have liked to keep it in the LRU list, but we didn't have enough memory in the system to do so. That helps us put it back in the right position in the LRU list when it gets brought back in from disc.
I'd sugegst something else has gone wrong, maybe the refcount should have been kept elevated to prevent the superblock from being paged out. I find it hard to believe that we can be so low on memory that we need to page out a superblock to make room for some other memory allocation.
(Although maybe if you have millions of unusued filesystems mounted ...?)
Quoted 7 lines · Matthew Wilcox
A "value" entry in the block device's
i_pageswill be a shadow entry -- that is, the page has reached the end of the LRU list and been discarded, so we made a note that we would have liked to keep it in the LRU list, but we didn't have enough memory in the system to do so. That helps us put it back in the right position in the LRU list when it gets brought back in from disc.
Quoted 6 lines
I'd sugegst something else has gone wrong, maybe the refcount should have been kept elevated to prevent the superblock from being paged out. I find it hard to believe that we can be so low on memory that we need to page out a superblock to make room for some other memory allocation.
(Although maybe if you have millions of unusued filesystems mounted ...?)
This is a freshly booted VM running the test on the only non-root disk file system. So yeah, there must be a logic error somewhere in the use of the page cache.
Quoted 47 lines · Matthew Wilcox
Quoted 32 lines · Christoph Hellwig
Hi Naohiro, (and willy for insights on the pagecache, see below),
when running plain fsx on zoned btrfs on a
null_blkdevices as below:dev="/sys/kernel/config/nullb/nullb1"
size=12800 # MBmkdir ${dev}
echo 2 > "${dev}"/submit_queues
echo 2 > "${dev}"/queue_mode
echo 2 > "${dev}"/irqmode
echo "${size}" > "${dev}"/size
echo 1 > "${dev}"/zoned
echo 0 > "${dev}"/zone_nr_conv
echo 128 > "${dev}"/zone_size
echo 96 > "${dev}"/zone_capacity
echo 14 > "${dev}"/zone_max_active
echo 1 > "${dev}"/memory_backed
echo 1000000 > "${dev}"/completion_nsec
echo 1 > "${dev}"/power
mkfs.btrfs -m single/dev/nullb1
mount/dev/nullb1/mnt/test/
~/xfstests-dev/ltp/fsx/mnt/test/foobarfsx will eventually after ~10 minutes fail with the following left in dmesg:
[ 1185.480200] BTRFS error (device nullb1): error writing primary super block to device 1 [ 1185.480988] BTRFS: error (device nullb1) in write_all_supers:4488: errno=-5 IO failure (1 errors while writing supers) [ 1185.481971] BTRFS info (device nullb1: state E): forced readonly [ 1185.482521] BTRFS: error (device nullb1: state EA) in btrfs_sync_log:3341: errno=-5 IO failureQuoted 6 lines
I tracked this down to the
find_get_pagecall inwait_dev_supersreturning NULL, and digging furter it seems to come fromxa_is_value()in__filemap_get_folioreturnin true. I'm not sure why we'd see a value here in the block device mapping and why that only happens in zoned mode (the same config on regular device ran for 10 hours last night).A "value" entry in the block device's
i_pageswill be a shadow entry -- that is, the page has reached the end of the LRU list and been discarded, so we made a note that we would have liked to keep it in the LRU list, but we didn't have enough memory in the system to do so. That helps us put it back in the right position in the LRU list when it gets brought back in from disc.
Quoted 7 lines
I'd sugegst something else has gone wrong, maybe the refcount should have been kept elevated to prevent the superblock from being paged out. I find it hard to believe that we can be so low on memory that we need to page out a superblock to make room for some other memory allocation.
(Although maybe if you have millions of unusued filesystems mounted ...?)
Ha but zoned btrfs uses two zones as a ringbuffer for its super-block, could it be, that we're accumulating too many page references somewhere? And then it behaves like having millions of filesystems mounted?
Quoted 3 lines · Johannes Thumshirn
Ha but zoned btrfs uses two zones as a ringbuffer for its super-block, could it be, that we're accumulating too many page references somewhere? And then it behaves like having millions of filesystems mounted?
That fact the superblock moves for zoned devices probably has something to do with it. But the whole code leaves me really puzzling.
Why does wait_dev_supers even do a find_get_page vs just stashing three page pointers away in the btrfs_device structure?
Why does this abuse wait_on_page_locked vs using a completion?
Why does the code count errors while only an error on the primary superblock has any consequences?
What is the point of the secodary superblocks if they aren't written on fsync?
How does just setting the whole page uptodat work on file systems with a block size smaller than the page size where we don't know what is in the rest of the page?
Quoted 10 lines · Christoph Hellwig
Quoted 3 lines · Johannes Thumshirn
Ha but zoned btrfs uses two zones as a ringbuffer for its super-block, could it be, that we're accumulating too many page references somewhere? And then it behaves like having millions of filesystems mounted?
That fact the superblock moves for zoned devices probably has something to do with it. But the whole code leaves me really puzzling.
Why does
wait_dev_superseven do afind_get_pagevs just stashing three page pointers away in thebtrfs_devicestructure?
The superblock used to be written written using buffer heads, the current code is direct transformation of the buffer head API to bios, so it's still using page cache.
I've sent a patchset to write it with separate pages, but this breaks userspace as the reads are from page cache. This should be done by direct io, also I'll need more time to test it properly, the kernel/userspace interactions were missed initially.
Quoted 1 line
Why does this abuse
wait_on_page_lockedvs using a completion?
This is I think still from the buffer head times, the page lock waiting was available and hasn't been converted to completion.
Quoted 2 lines
Why does the code count errors while only an error on the primary superblock has any consequences?
Because the primary superblock is the important one, error on the secondary should not bring down the filesystem if the primary can be written.
Quoted 2 lines
What is the point of the secodary superblocks if they aren't written on fsync?
Writing superblock is an IO hit, used to be noticeable on rotational devices, and fsync is meant to be fast. It's I believe a performance optimization. The secondary superblocks are rarely used so they don't get the same treatment as primary.
Quoted 3 lines
How does just setting the whole page uptodat work on file systems with a block size smaller than the page size where we don't know what is in the rest of the page?
This was pointed out by Matthew some time ago, the part of the page after superblock will be uninitialized.
Quoted 12 lines · David Sterba
Quoted 10 lines · Christoph Hellwig
Quoted 3 lines · Johannes Thumshirn
Ha but zoned btrfs uses two zones as a ringbuffer for its super-block, could it be, that we're accumulating too many page references somewhere? And then it behaves like having millions of filesystems mounted?
That fact the superblock moves for zoned devices probably has something to do with it. But the whole code leaves me really puzzling.
Why does
wait_dev_superseven do afind_get_pagevs just stashing three page pointers away in thebtrfs_devicestructure?
Quoted 8 lines
The superblock used to be written written using buffer heads, the current code is direct transformation of the buffer head API to bios, so it's still using page cache.
I've sent a patchset to write it with separate pages, but this breaks userspace as the reads are from page cache. This should be done by direct io, also I'll need more time to test it properly, the kernel/userspace interactions were missed initially.
I have wip (tests pass) that does the own page write, waiting using a completion and own bio, ie. avoiding page cache. The super block read side however uses page cache in several places. When a device is not part of a mounted filesystem thre's no difference, no concurrent writers and readers, but for zoned mode it is a problem in one cases, when both zones are full and the older needs to be determined by reading the superblock.
This happens on a mounted filesystem and it's read and write so it needs to be converted to own page read too so I can't merge the write part as-is. Maybe there's a middle ground, but otherwise the page cache based read requires restructuring as it's done accross several functions.