Skip to contentSkip to conversation

__buffer_migrate_page() vs ll_rw_block()

DISCUSSION Vincent Whitchurch 2 participants 2 messages Jun 1, 2021 – Jun 2, 2021 Memory management

lore
Share & tools
lore
#0 lore raw

I'm seeing occasional squashfs read failures ("squashfs_read_data failed to read block") when compaction is run at the same time as reading from squashfs, with something like the commands below. The kernel version is the latest stable/v5.4 kernel, v5.4.123.

while :; do echo 1 > /proc/sys/vm/compact_memory; done &
while :; do echo 3 > /proc/sys/vm/drop_caches; find fs/ > /dev/null; done &

On this kernel, squashfs uses ll_rw_block(). The problem is that ll_rw_block() ignores BHs which it can't get a lock on, but __buffer_migrate_page() can take the lock on the BHs in order to check if they can be migrated. If __buffer_migrate_page() holds the lock at the same time that ll_rw_block() wants it, the BH is skipped and I/O is not issued for these blocks, and squashfs ends up seeing !buffer_uptodate() and erroring out.

On newer kernels, squashfs doesn't use ll_rw_block() anymore, but I still see other users of that function in other filesystems, and AFAICS the underlying problem of the race with __buffer_migrate_page() has not yet been fixed.

I'd be happy to receive any suggestions about the right way to fix this.

Thank you.

#1 lore raw
Quoted 16 lines · Vincent Whitchurch

I'm seeing occasional squashfs read failures ("squashfs_read_data failed to read block") when compaction is run at the same time as reading from squashfs, with something like the commands below. The kernel version is the latest stable/v5.4 kernel, v5.4.123.

while :; do echo 1 > /proc/sys/vm/compact_memory; done &
while :; do echo 3 > /proc/sys/vm/drop_caches; find fs/ > /dev/null; done &

On this kernel, squashfs uses ll_rw_block(). The problem is that ll_rw_block() ignores BHs which it can't get a lock on, but __buffer_migrate_page() can take the lock on the BHs in order to check if they can be migrated. If __buffer_migrate_page() holds the lock at the same time that ll_rw_block() wants it, the BH is skipped and I/O is not issued for these blocks, and squashfs ends up seeing !buffer_uptodate() and erroring out.

Quoted 6 lines

On newer kernels, squashfs doesn't use ll_rw_block() anymore, but I still see other users of that function in other filesystems, and AFAICS the underlying problem of the race with __buffer_migrate_page() has not yet been fixed.

I'd be happy to receive any suggestions about the right way to fix this.

Well, if you care about squashfs, the best solution would IMO be to backport 93e72b3c612adc ("squashfs: migrate from ll_rw_block usage to BIO") and the followup fixes (there are several of them as far as I can see by Fixes tags) to your kernel - stable tree would be IMHO willing to take the series if you add your justification that it fixes read failures for you.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR