the kernel crashes when storing an EXT4 file system in a ZRAM device
Subsystems (2)
Hi,
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.
Can you take a look please?
[1] https://bugzilla.kernel.org/show_bug.cgi?id=219548
[2] https://lore.kernel.org/linux-kernel/[email protected]
Quoted 8 lines · Sergey Senozhatsky
Hi,
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under
write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.
Do you set BLK_FEAT_STABLE_WRITES on zram?
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under
write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.Do you set
BLK_FEAT_STABLE_WRITESon zram?
Yes, zram sets BLK_FEAT_STABLE_WRITES and BLK_FEAT_SYNCHRONOUS.
Quoted 8 lines · Sergey Senozhatsky
Hi,
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under
write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.
This smells like ext4 doesn't respect BDI_CAP_STABLE_WRITES somewhere.
Quoted 9 lines · Sergey Senozhatsky
Hi,
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under
write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.
Quoted 4 lines
Can you take a look please?
[1] https://bugzilla.kernel.org/show_bug.cgi?id=219548
[2] https://lore.kernel.org/linux-kernel/[email protected]
The link in [2] is a bit busted, since the message in question wasn't cc'ed to LKML, but rather to mm-commits. But dropping "/linux-kernel" allows the link to work, and what's interesting is this message from that thread:
https://lore.kernel.org/all/[email protected]/
The blocks which are getting modified while a write is in flight are ext4 metadata blocks, which are in the buffer cache. Ext4 is modifying those blocks via bh->b_data, and ext4 isn't issuing the write; those are happenig via the buffer cache's writeback functions.
Hmmm.... was the user using an ext4 file system with the journal disabled, by any chance? If ext4 is using the journal (which is the common case), metadata blocks only get modified via jbd2 journal functions, and a blocks only get modified when they are part of a jbd2 transaction --- and while the transaction is active, the buffer cache writeback is disabled. It's only after the transaction is committed that are dirty blocks associated with that transaction are allowed to be written back. So I *think* the only way we could run into problems is ext4's jbd2 journalling is disabled.
More generally, any file system which uses the buffer cache, and doesn't use jbd2 to control when writeback happens, I think is going to be at risk with a block device which requires stable writes. The only way to fix this, really, is to have the buffer cache code copy the data to a bounce buffer, and then issue the write from the bounce buffer.
- Ted
Quoted 12 lines · Theodore Ts'o
Quoted 11 lines · Sergey Senozhatsky
Hi,
We've got two reports [1] [2] (could be the same person) which suggest that ext4 may change page content while the page is under
write(). The particular problem here the case when ext4 is on the zram device. zram compresses every page written to it, so if the page content can be modified concurrently with zram's compression then we can't really use zram with ext4.Can you take a look please?
Quoted 7 lines
Quoted 2 lines
[1] https://bugzilla.kernel.org/show_bug.cgi?id=219548
[2] https://lore.kernel.org/linux-kernel/[email protected]The link in [2] is a bit busted, since the message in question wasn't cc'ed to LKML, but rather to mm-commits. But dropping "/linux-kernel" allows the link to work, and what's interesting is this message from that thread:
My bad.
Quoted 1 line
Let me Cc Yu Huabing on this:
Quoted 21 lines
The blocks which are gtting modified while a write is in flight are ext4 metadata blocks, which are in the buffer cache. Ext4 is modifying those blocks via
bh->b_data, and ext4 isn't issuing the write; those are happenig via the buffer cache's writeback functions.Hmmm.... was the user using an ext4 file system with the journal disabled, by any chance? If ext4 is using the journal (which is the common case), metadata blocks only get modified via jbd2 journal functions, and a blocks only get modified when they are part of a jbd2 transaction --- and while the transaction is active, the buffer cache writeback is disabled. It's only after the transaction is committed that are dirty blocks associated with that transaction are allowed to be written back. So I *think* the only way we could run into problems is ext4's jbd2 journalling is disabled.
More generally, any file system which uses the buffer cache, and doesn't use jbd2 to control when writeback happens, I think is going to be at risk with a block device which requires stable writes. The only way to fix this, really, is to have the buffer cache code copy the data to a bounce buffer, and then issue the write from the bounce buffer.
Quoted 6 lines · Theodore Ts'o
More generally, any file system which uses the buffer cache, and doesn't use jbd2 to control when writeback happens, I think is going to be at risk with a block device which requires stable writes. The only way to fix this, really, is to have the buffer cache code copy the data to a bounce buffer, and then issue the write from the bounce buffer.
Should there be a pr_warn_once when using a file systems using the legacy buffer cache interfaces on a device that requires stable pages?
Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Theodore Ts'o
More generally, any file system which uses the buffer cache, and doesn't use jbd2 to control when writeback happens, I think is going to be at risk with a block device which requires stable writes. The only way to fix this, really, is to have the buffer cache code copy the data to a bounce buffer, and then issue the write from the bounce buffer.
Should there be a
pr_warn_oncewhen using a file systems using the legacy buffer cache interfaces on a device that requires stable pages?
Well, either that, or we need to teach the buffer cache writeback code to issue writes through a bounce buffer if the device requires stable writes.
I'll note that this could also manifest if some program was writing to a device that requires stable writes using buffered I/O. For example, if they are using postgres, which won't be switching to direct I/O for another 2-5 years (depending on how optimistic you are and how willing enterprise customers will be to move to the latest version of Postgres; some are stillu using very ancient Postgres for the same reason that RHEL 7 systems based on the 3.10 kernel are still in production use even today.)
For this particular use case, which is running VM's on Chromium/ChromeOS, I suspect we do need to have some kind of solution other than triggering a WARN_ON. Besides, I'd really rather not get the kind of syzbot noise we would have by having some scheme that would be trivially easy for syzbot to trigger. (We're not should use WARN_ON for things that can be triggered by Stupid User Tricks, because syzbot fuzzers can be so ingenious. :-)
- Ted
Quoted 3 lines · Theodore Ts'o
For this particular use case, which is running VM's on Chromium/ChromeOS, I suspect we do need to have some kind of solution other than triggering a
WARN_ON.
Sorry, I didn't complete my thought here. We could just say, "don't use ext4 without a journal in a Chrome VM." But if the are going to allow the VM's access to external USB storage, then ext4 in no-journal mode will be the least of their problems. People trying to access USB thumbdrives or sdcards from their digital cameras using FAT file sytems will be trigerring ZBLK buffer overflow kernel crashes left, right, and center. Especially if they are on a low-cost ChromeOS device with a tiny amount of meory, such that memory pressure should be considered a foregone conclusion. :-)
- Ted
Quoted 7 lines · Theodore Ts'o
The blocks which are getting modified while a write is in flight are ext4 metadata blocks, which are in the buffer cache. Ext4 is modifying those blocks via
bh->b_data, and ext4 isn't issuing the write; those are happenig via the buffer cache's writeback functions.Hmmm.... was the user using an ext4 file system with the journal disabled, by any chance?
I believe you are right, at least that's what caiqingfu said [1]:
echo 524288000 > /sys/devices/virtual/block/zram0/disksize mkfs.ext4 -O ^has_journal -b 4096 -F -L TEMP -m 0 /dev/zram0 mkdir /tmp/zram
mount -t ext4 -o errors=continue,nosuid,nodev,noatime /dev/zram0 /tmp/zram