Skip to contentSkip to conversation

mm: move migration work around to buffer-heads

PATCH In review Luis Chamberlain 5 participants 17 messages Mar 30, 2025 – Apr 4, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover mm: move migration work around to buffer-heads

We have an eye-sore of a spin lock held during page migration which was added for a ext4 jbd corruption fix for which we have no clear public corruption data. We want to remove the spin lock on mm/migrate so to help buffer-head filesystems embrace large folios, since we can cond_resched() on large folios on folio_mc_copy(). I've managed to reproduce a corruption by just removing the spinlock and stressing ext4 with generic/750, a corruption happens after 3 hours.

The spin lock was added to help ext4 jbd and other users of buffer_migrate_folio_norefs(), so the block device cache and nilfs2. This does the work to move the heuristic needed to avoid page migration to back to the buffere-head code on __find_get_block_slow() and only to users of buffer_migrate_folio_norefs(). I have ran generic/750 over 20 hours and don't see the corruption issue.

I've also ran this patchset against all the following ext4 profiles on all fstests tests and have found no regression, I've published the baseline based on linux-next tag next-20250328 onto kdevops [0]. For further sanity I've also tested this patchset against blktests as well and found no regressions.

ext4-defaults
ext4-1k
ext4-2k
ext4-4k
ext4-bigalloc16k-4k
ext4-bigalloc32k-4k
ext4-bigalloc64k-4k
ext4-bigalloc1024k-4k
ext4-bigalloc2048k-4k
ext4-advanced-features

[0] https://github.com/linux-kdevops/kdevops/commit/3ecd638e67b14162b76b733a120e6e1b55698cc9

Luis Chamberlain (3):
mm/migrate: add might_sleep() on __migrate_folio() fs/buffer: avoid races with folio migrations on

    __find_get_block_slow()

mm/migrate: avoid atomic context on buffer_migrate_folio_norefs()
migration

 fs/buffer.c  | 9 +++++++++
 mm/migrate.c | 6 +++---
 2 files changed, 12 insertions(+), 3 deletions(-)
Signature · 1 line
2.47.2
#1 lore raw

PATCH 1/3 mm/migrate: add might_sleep() on __migrate_folio()

When we do page migration of large folios folio_mc_copy() can cond_resched() *iff* we are on a large folio. There's a hairy bug reported by both 0-day [0] and syzbot [1] where it has been detected we can call folio_mc_copy() in atomic context. While, technically speaking that should in theory be only possible today from buffer-head filesystems using buffer_migrate_folio_norefs() on page migration the only buffer-head large folio filesystem -- the block device cache, and so with block devices with large block sizes. However tracing shows that folio_mc_copy() *isn't* being called as often as we'd expect from buffer_migrate_folio_norefs() path as we're likely bailing early now thanks to the check added by commit 060913999d7a ("mm: migrate: support poisoned recover from migrate folio").

*Most* folio_mc_copy() calls in turn end up *not* being in atomic context, and so we won't hit a splat when using:

CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_ATOMIC_SLEEP=y

But we *want* to help proactively find callers of __migrate_folio() in atomic context, so make might_sleep() explicit to help us root out large folio atomic callers of migrate_folio().

---
 mm/migrate.c | 2 ++
 1 file changed, 2 insertions(+)
mm/migrate.c +2 -0
index f3ee6d8d5e2e..712ddd11f3f0 100644
@@ -751,6 +751,8 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 {
 	int rc, expected_count = folio_expected_refs(mapping, src);
 
+	might_sleep();
+
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
Signature · 1 line
2.47.2
#2 lore raw

Re: [PATCH 1/3] mm/migrate: add might_sleep() on __migrate_folio()

Quoted 5 lines · Luis Chamberlain

However tracing shows that folio_mc_copy() *isn't* being called as often as we'd expect from buffer_migrate_folio_norefs() path as we're likely bailing early now thanks to the check added by commit 060913999d7a ("mm: migrate: support poisoned recover from migrate folio").

Umm. You're saying that most folios we try to migrate have extra refs? That seems unexpected; does it indicate a bug in 060913999d7a?

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 754

Quoted 6 lines

+++ b/mm/migrate.c

mm/migrate.c +1 -0
@@ -751,6 +751,8 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 {
 	int rc, expected_count = folio_expected_refs(mapping, src);
 
+	might_sleep();

We deliberately don't sleep when the folio is only a single page. So this needs to be:

	might_sleep_if(folio_test_large(folio));
Quoted 6 lines
mm/migrate.c +0 -0
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
Signature · 1 line
2.47.2
#3 lore raw

Re: [PATCH 1/3] mm/migrate: add might_sleep() on __migrate_folio()

Quoted 9 lines · Matthew Wilcox
Quoted 5 lines · Luis Chamberlain

However tracing shows that folio_mc_copy() *isn't* being called as often as we'd expect from buffer_migrate_folio_norefs() path as we're likely bailing early now thanks to the check added by commit 060913999d7a ("mm: migrate: support poisoned recover from migrate folio").

Umm. You're saying that most folios we try to migrate have extra refs? That seems unexpected; does it indicate a bug in 060913999d7a?

I've debugged this further, the migration does succeed and I don't see any failures due to the new refcheck added by 060913999d7a. I've added stats in a out of tree patch [0] in case folks find this useful, I could submit this. But the point is that even if you use dd against a large block device you won't always end up trying to migrate large folios *right away* even if you trigger folio migration through compaction, specially if you use a large bs on dd like bs=1M. Using a size matching more close to the logical block size will trigger large folio migration much faster.

Example of the stats:

# cat /sys/kernel/debug/mm/migrate/bh/stats

[buffer_migrate_folio]

                    calls       9874
                  success       9854
                    fails       20

[buffer_migrate_folio_norefs]

                    calls       3694
                  success       1651
                    fails       2043
          no-head-success       532
            no-head-fails       0
                  invalid       2040
                    valid       1119
            valid-success       1119
              valid-fails       0

Success ratios:
buffer_migrate_folio: 99% success (9854/9874) buffer_migrate_folio_norefs: 44% success (1651/3694)

Quoted 11 lines

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 754

Quoted 6 lines

+++ b/mm/migrate.c

mm/migrate.c +1 -0
@@ -751,6 +751,8 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 {
 	int rc, expected_count = folio_expected_refs(mapping, src);
 
+	might_sleep();

We deliberately don't sleep when the folio is only a single page. So this needs to be:

	might_sleep_if(folio_test_large(folio));

That does reduce the scope of our test coverage but, sure.

[0] https://lore.kernel.org/all/[email protected]/

Luis

#4 lore raw

Re: [PATCH 1/3] mm/migrate: add might_sleep() on __migrate_folio()

In reply to Matthew Wilcox
Quoted 4 lines · Matthew Wilcox

We deliberately don't sleep when the folio is only a single page. So this needs to be:

	might_sleep_if(folio_test_large(folio));

Along with willy's suggestion:

... and the same rules apply for copy_folio():

fs/aio.c: atomic context but no large folio: safe

fs/fuse/file.c: no large folio: safe

fs/nilfs2/page.c: blocking safe, has large folio: safe

mm/migrate_device.c: no large folio: safe

mm/shmem.c: blocking safe, has large folio: safe shmem_replace_folio() could use a might_sleep() I guess.

Thanks,
Davidlohr

#5 lore raw

PATCH 2/3 fs/buffer: avoid races with folio migrations on __find_get_block_slow()

In reply to Luis Chamberlain

Filesystems which use buffer-heads where it cannot guarantees that the there are no other references to the folio, for example with a folio lock, must use buffer_migrate_folio_norefs() for the address space mapping migrate_folio() callback. There are only 3 filesystems which use this callback:

1) the block device cache
2) ext4 for its ext4_journalled_aops
3) nilfs2

The commit ebdf4de5642fb6 ("mm: migrate: fix reference check race between __find_get_block() and migration") added a spin lock to prevent races with page migration which ext4 users were reporting through the SUSE bugzilla (bnc#1137609 [0]). Although implicit, the spinlock is only held for users of buffer_migrate_folio_norefs() which was added by commit 89cb0888ca148 ("mm: migrate: provide buffer_migrate_page_norefs()") to support page migration on block device folios. Later commit dae999602eeb ("ext4: stop providing .writepage hook") made ext4_journalled_aops use the same callback. It is worth elaborating on why ext4 journalled aops uses this:

so that buffers cannot be modified under jdb2's hands as that can cause data corruption. For example when commit code does writeout of transaction buffers in jbd2_journal_write_metadata_buffer(), we don't hold page lock or have page writeback bit set or have the buffer locked. So page migration code would go and happily migrate the page elsewhere while the copy is running thus corrupting data.

Although we don't have exact traces of the filesystem corruption we can can reproduce fs corruption one ext4 by just removing the spinlock and stress testing the filesystem with generic/750, we eventually end up after 3 hours of testing with kdevops using libvirt on the ext4 profile ext4-4k. Things like the below as reported recently [1]:

Mar 28 03:36:37 extra-ext4-4k unknown: run fstests generic/750 at 2025-03-28 03:36:37
<-- etc -->
Mar 28 05:57:09 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_get_first_dir_block:3538: inode #5174: comm fsstress: directory missing '.'
Mar 28 06:04:43 extra-ext4-4k kernel: EXT4-fs warning (device loop5): ext4_empty_dir:3088: inode #5176: comm fsstress: directory missing '.'
Mar 28 06:42:05 extra-ext4-4k kernel: EXT4-fs error (device loop5): __ext4_find_entry:1626: inode #5173: comm fsstress: checksumming directory block 0
Mar 28 08:16:43 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_find_extent:938: inode #1104560: comm fsstress: pblk 4932229 bad header/extent: invalid magic - magic 8383, entries 33667, max 33667(0), depth 33667(0)

The block device cache is a user of buffer_migrate_folio_norefs() and it supports large folios, in that case we can sleep on folio_mc_copy() on page migration on a cond_resched(). So we want to avoid requiring a spin lock even on the buffer_migrate_folio_norefs() case so to enable large folios on buffer-head folio migration. To address this we must avoid races with folio migration in a different way.

This provides an alternative by avoiding giving away a folio in __find_get_block_slow() on folio migration candidates so to enable us to let us later rip out the spin_lock() held on the folio migration buffer_migrate_folio_norefs() path. We limit the scope of this sanity check only for filesystems which cannot provide any guarantees that there are no references to the folio, so only users of the folio migration callback buffer_migrate_folio_norefs().

Although we have no direct clear semantics to check if a folio is being evaluated for folio migration we know that folio migration happens LRU folios [2]. Since folio migration must not be called with folio_test_writeback() folios we can skip these folios as well. The other corner case we can be concerned is for a drive implement mops, but the docs indicate VM seems to use lru for that too. A real concern to have here is if the check is starving readers or writers who want to read a block into the page cache and it is part of the LRU. The path __getblk_slow() will first try __find_get_block() which uses __filemap_get_folio() without FGP_CREAT, and if it fails it will call grow_buffers() which calls again __filemap_get_folio() but with with FGP_CREAT now, but __filemap_get_folio() won't create a folio for us if it already exists. So if the folio was in LRU __getblk_slow() will essentially end up checking again for the folio until its gone from the page cache or migration ended, effectively preventing a race with folio migration which is what we want.

This commit and the subsequent one prove to be an alternative to fix the filesystem corruption noted above.

---
 fs/buffer.c | 9 +++++++++
 1 file changed, 9 insertions(+)
fs/buffer.c +9 -0
index c7abb4a029dc..a4e4455a6ce2 100644
@@ -208,6 +208,15 @@ __find_get_block_slow(struct block_device *bdev, sector_t block)
 	head = folio_buffers(folio);
 	if (!head)
 		goto out_unlock;
+
+	if (folio->mapping->a_ops->migrate_folio &&
+	    folio->mapping->a_ops->migrate_folio == buffer_migrate_folio_norefs) {
+		if (folio_test_lru(folio) &&
+		    folio_test_locked(folio) &&
+		    !folio_test_writeback(folio))
+			goto out_unlock;
+	}
+
 	bh = head;
 	do {
 		if (!buffer_mapped(bh))
Signature · 1 line
2.47.2
#6 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 6 lines · Luis Chamberlain

Although we don't have exact traces of the filesystem corruption we can can reproduce fs corruption one ext4 by just removing the spinlock and stress testing the filesystem with generic/750, we eventually end up after 3 hours of testing with kdevops using libvirt on the ext4 profile ext4-4k. Things like the below as reported recently [1]:

Quoted 6 lines
Mar 28 03:36:37 extra-ext4-4k unknown: run fstests generic/750 at 2025-03-28 03:36:37
<-- etc -->
Mar 28 05:57:09 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_get_first_dir_block:3538: inode #5174: comm fsstress: directory missing '.'
Mar 28 06:04:43 extra-ext4-4k kernel: EXT4-fs warning (device loop5): ext4_empty_dir:3088: inode #5176: comm fsstress: directory missing '.'
Mar 28 06:42:05 extra-ext4-4k kernel: EXT4-fs error (device loop5): __ext4_find_entry:1626: inode #5173: comm fsstress: checksumming directory block 0
Mar 28 08:16:43 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_find_extent:938: inode #1104560: comm fsstress: pblk 4932229 bad header/extent: invalid magic - magic 8383, entries 33667, max 33667(0), depth 33667(0)

I reproduced again a corruption with ext4 when we remove the spin lock alone with generic/750, the trace looks slightly different, and this time I ran the test with ext4 with 2k block size filesystem. I also reverted both "mm: migrate: remove folio_migrate_copy()" and "mm: migrate: support poisoned recover from migrate folio" just in case the extra check added by the later was helping avoid the corruption. I also added some debugfs stats one can use to verify exact values of how buffer_migrate_folio_norefs() can fail. I put this tree on linux-kdevops 20250321-accel-ext4-bug branch [0] for other folks' convenience. This time it took ~ 5 hours to reproduce and we start off with a jbd jbd2_journal_dirty_metadata() kernel warning followed by putting the filesystem in read-only mode:

Mar 31 06:51:30 extra-ext4-2k kernel: Linux version 6.14.0-rc7-next-20250321-00004-ge4b961f1dadb (mcgrof@beef) (gcc (Debian 14.2.0-16) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #92 SMP PREEMPT_DYNAMIC Mon Mar 31 06:43:08 UTC 2025
Mar 31 06:51:30 extra-ext4-2k kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.14.0-rc7-next-20250321-00004-ge4b961f1dadb root=PARTUUID=503fa6f2-2d5b-4d7e-8cf8-3a811de326ce ro console=tty0 console=tty1 console=ttyS0,115200n8 console=ttyS0

<-- snip -->

Mar 31 06:56:05 extra-ext4-2k unknown: run fstests generic/750 at 2025-03-31 06:56:05
Mar 31 06:56:06 extra-ext4-2k kernel: EXT4-fs (loop5): mounted filesystem 858bf9bf-6582-42d3-b2f5-41c7061033df r/w with ordered data mode. Quota mode: none.
Mar 31 07:02:04 extra-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Mar 31 07:35:22 extra-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Mar 31 07:53:05 extra-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Mar 31 11:47:01 extra-ext4-2k kernel: ------------[ cut here ]------------
Mar 31 11:47:01 extra-ext4-2k kernel: WARNING: CPU: 5 PID: 3092 at fs/jbd2/transaction.c:1552 jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel: Modules linked in: loop sunrpc 9p nls_iso8859_1 nls_cp437 vfat crc32c_generic fat kvm_intel kvm ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio virtio_balloon virtio_console button evdev joydev serio_raw nvme_fabrics drm dm_mod nvme_core nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover failover virtio_blk psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Mar 31 11:47:01 extra-ext4-2k kernel: CPU: 5 UID: 0 PID: 3092 Comm: fsstress Not tainted 6.14.0-rc7-next-20250321-00004-ge4b961f1dadb #92 PREEMPT(full) 
Mar 31 11:47:01 extra-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.11-5 01/28/2025
Mar 31 11:47:01 extra-ext4-2k kernel: RIP: 0010:jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel: Code: 30 0f 84 5b fe ff ff 0f 0b 41 bc 8b ff ff ff e9 69 fe ff ff 48 8b 04 24 4c 8b 48 70 4d 39 cf 0f 84 53 ff ff ff e9 22 c5 00 00 <0f> 0b 41 bc e4 ff ff ff e9 41 ff ff ff 0f 0b 90 0f 1f 40 00 90 90
Mar 31 11:47:01 extra-ext4-2k kernel: RSP: 0018:ffffb8bd83d2f7f8 EFLAGS: 00010246
Mar 31 11:47:01 extra-ext4-2k kernel: RAX: 0000000000000001 RBX: ffff9d72cb6d7528 RCX: 00000000000000fd
Mar 31 11:47:01 extra-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: RBP: ffff9d72cf5127b8 R08: ffff9d72cf5127b8 R09: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R10: ffff9d733c8d983c R11: 0000000000006496 R12: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R13: ffff9d72c258cce8 R14: ffff9d72cb6d7530 R15: ffff9d72c25ff000
Mar 31 11:47:01 extra-ext4-2k kernel: FS:  00007f1f5ae7f740(0000) GS:ffff9d7386349000(0000) knlGS:0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 31 11:47:01 extra-ext4-2k kernel: CR2: 000055a76a03bfa8 CR3: 0000000108f76001 CR4: 0000000000772ef0
Mar 31 11:47:01 extra-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 31 11:47:01 extra-ext4-2k kernel: PKRU: 55555554
Mar 31 11:47:01 extra-ext4-2k kernel: Call Trace:
Mar 31 11:47:01 extra-ext4-2k kernel:  <TASK>
Mar 31 11:47:01 extra-ext4-2k kernel:  ? jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __warn.cold+0x93/0xf8
Mar 31 11:47:01 extra-ext4-2k kernel:  ? jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? report_bug+0xe6/0x170
Mar 31 11:47:01 extra-ext4-2k kernel:  ? jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? handle_bug+0x199/0x260
Mar 31 11:47:01 extra-ext4-2k kernel:  ? exc_invalid_op+0x13/0x60
Mar 31 11:47:01 extra-ext4-2k kernel:  ? asm_exc_invalid_op+0x16/0x20
Mar 31 11:47:01 extra-ext4-2k kernel:  ? jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? jbd2_journal_dirty_metadata+0x9e/0x230 [jbd2]
Mar 31 11:47:01 extra-ext4-2k kernel:  __ext4_handle_dirty_metadata+0x6d/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_da_write_end+0xa3/0x3c0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  do_writepages+0xdd/0x250
Mar 31 11:47:01 extra-ext4-2k kernel:  filemap_fdatawrite_wbc+0x48/0x60
Mar 31 11:47:01 extra-ext4-2k kernel:  __filemap_fdatawrite_range+0x5b/0x80
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_release_file+0x6d/0xa0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  __fput+0xf4/0x2b0
Mar 31 11:47:01 extra-ext4-2k kernel:  __x64_sys_close+0x39/0x80
Mar 31 11:47:01 extra-ext4-2k kernel:  do_syscall_64+0x4b/0x120
Mar 31 11:47:01 extra-ext4-2k kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Mar 31 11:47:01 extra-ext4-2k kernel: RIP: 0033:0x7f1f5af11687
Mar 31 11:47:01 extra-ext4-2k kernel: Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff
Mar 31 11:47:01 extra-ext4-2k kernel: RSP: 002b:00007ffc2509e1d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Mar 31 11:47:01 extra-ext4-2k kernel: RAX: ffffffffffffffda RBX: 00007f1f5ae7f740 RCX: 00007f1f5af11687
Mar 31 11:47:01 extra-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
Mar 31 11:47:01 extra-ext4-2k kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R13: 00007ffc2509e7c0 R14: 0000000000000003 R15: 0000000000000004
Mar 31 11:47:01 extra-ext4-2k kernel:  </TASK>
Mar 31 11:47:01 extra-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Mar 31 11:47:01 extra-ext4-2k kernel: ------------[ cut here ]------------
Mar 31 11:47:01 extra-ext4-2k kernel: WARNING: CPU: 5 PID: 3092 at fs/ext4/ext4_jbd2.c:360 __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel: Modules linked in: loop sunrpc 9p nls_iso8859_1 nls_cp437 vfat crc32c_generic fat kvm_intel kvm ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio virtio_balloon virtio_console button evdev joydev serio_raw nvme_fabrics drm dm_mod nvme_core nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover failover virtio_blk psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Mar 31 11:47:01 extra-ext4-2k kernel: CPU: 5 UID: 0 PID: 3092 Comm: fsstress Tainted: G        W           6.14.0-rc7-next-20250321-00004-ge4b961f1dadb #92 PREEMPT(full) 
Mar 31 11:47:01 extra-ext4-2k kernel: Tainted: [W]=WARN
Mar 31 11:47:01 extra-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.11-5 01/28/2025
Mar 31 11:47:01 extra-ext4-2k kernel: RIP: 0010:__ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel: Code: 00 00 00 44 89 fa 4c 89 f6 49 c7 c1 df 2b 97 c0 4c 89 e7 41 bd fb ff ff ff e8 6a ce 05 00 eb 97 48 89 df e8 60 99 8a f5 eb 8a <0f> 0b 48 c7 c2 60 42 96 c0 45 89 e8 48 89 e9 44 89 fe 4c 89 f7 e8
Mar 31 11:47:01 extra-ext4-2k kernel: RSP: 0018:ffffb8bd83d2f838 EFLAGS: 00010286
Mar 31 11:47:01 extra-ext4-2k kernel: RAX: ffff9d72c04ef800 RBX: ffff9d72cf5127b8 RCX: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00000000ffffffff
Mar 31 11:47:01 extra-ext4-2k kernel: RBP: ffff9d72c258cce8 R08: ffff9d72cf5127b8 R09: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R10: ffff9d733c8d983c R11: 0000000000006496 R12: ffff9d71d05fe378
Mar 31 11:47:01 extra-ext4-2k kernel: R13: 00000000ffffffe4 R14: ffffffffc0964720 R15: 0000000000000556
Mar 31 11:47:01 extra-ext4-2k kernel: FS:  00007f1f5ae7f740(0000) GS:ffff9d7386349000(0000) knlGS:0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 31 11:47:01 extra-ext4-2k kernel: CR2: 000055a76a03bfa8 CR3: 0000000108f76001 CR4: 0000000000772ef0
Mar 31 11:47:01 extra-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 31 11:47:01 extra-ext4-2k kernel: PKRU: 55555554
Mar 31 11:47:01 extra-ext4-2k kernel: Call Trace:
Mar 31 11:47:01 extra-ext4-2k kernel:  <TASK>
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __warn.cold+0x93/0xf8
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? report_bug+0xe6/0x170
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? handle_bug+0x199/0x260
Mar 31 11:47:01 extra-ext4-2k kernel:  ? exc_invalid_op+0x13/0x60
Mar 31 11:47:01 extra-ext4-2k kernel:  ? asm_exc_invalid_op+0x16/0x20
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? __ext4_handle_dirty_metadata+0x6d/0x190 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_da_write_end+0xa3/0x3c0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  do_writepages+0xdd/0x250
Mar 31 11:47:01 extra-ext4-2k kernel:  filemap_fdatawrite_wbc+0x48/0x60
Mar 31 11:47:01 extra-ext4-2k kernel:  __filemap_fdatawrite_range+0x5b/0x80
Mar 31 11:47:01 extra-ext4-2k kernel:  ext4_release_file+0x6d/0xa0 [ext4]
Mar 31 11:47:01 extra-ext4-2k kernel:  __fput+0xf4/0x2b0
Mar 31 11:47:01 extra-ext4-2k kernel:  __x64_sys_close+0x39/0x80
Mar 31 11:47:01 extra-ext4-2k kernel:  do_syscall_64+0x4b/0x120
Mar 31 11:47:01 extra-ext4-2k kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Mar 31 11:47:01 extra-ext4-2k kernel: RIP: 0033:0x7f1f5af11687
Mar 31 11:47:01 extra-ext4-2k kernel: Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff
Mar 31 11:47:01 extra-ext4-2k kernel: RSP: 002b:00007ffc2509e1d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Mar 31 11:47:01 extra-ext4-2k kernel: RAX: ffffffffffffffda RBX: 00007f1f5ae7f740 RCX: 00007f1f5af11687
Mar 31 11:47:01 extra-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
Mar 31 11:47:01 extra-ext4-2k kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
Mar 31 11:47:01 extra-ext4-2k kernel: R13: 00007ffc2509e7c0 R14: 0000000000000003 R15: 0000000000000004
Mar 31 11:47:01 extra-ext4-2k kernel:  </TASK>
Mar 31 11:47:01 extra-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs: ext4_ext_grow_indepth:1366: aborting transaction: error 28 in __ext4_handle_dirty_metadata
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5): ext4_ext_grow_indepth:1366: inode #213689: block 9062541: comm fsstress: journal_dirty_metadata failed: handle type 2 started at line 2721, credits 11/0, errcode -28
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_mb_clear_bb:6550: Readonly filesystem
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_reserve_inode_write:5870: Readonly filesystem
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2336: inode #213689: comm fsstress: mark_inode_dirty error
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2338: comm fsstress: Failed to mark inode 213689 dirty
Mar 31 11:47:01 extra-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_do_writepages:2751: error 28

So at least we now have 2 different public filesystem corruptions traces with ext4 without the spin lock.

[0] https://github.com/linux-kdevops/linux/tree/20250321-accel-ext4-bug

Luis

#7 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 14 lines · Luis Chamberlain
Quoted 6 lines · Luis Chamberlain

Although we don't have exact traces of the filesystem corruption we can can reproduce fs corruption one ext4 by just removing the spinlock and stress testing the filesystem with generic/750, we eventually end up after 3 hours of testing with kdevops using libvirt on the ext4 profile ext4-4k. Things like the below as reported recently [1]:

Quoted 6 lines
Mar 28 03:36:37 extra-ext4-4k unknown: run fstests generic/750 at 2025-03-28 03:36:37
<-- etc -->
Mar 28 05:57:09 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_get_first_dir_block:3538: inode #5174: comm fsstress: directory missing '.'
Mar 28 06:04:43 extra-ext4-4k kernel: EXT4-fs warning (device loop5): ext4_empty_dir:3088: inode #5176: comm fsstress: directory missing '.'
Mar 28 06:42:05 extra-ext4-4k kernel: EXT4-fs error (device loop5): __ext4_find_entry:1626: inode #5173: comm fsstress: checksumming directory block 0
Mar 28 08:16:43 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_find_extent:938: inode #1104560: comm fsstress: pblk 4932229 bad header/extent: invalid magic - magic 8383, entries 33667, max 33667(0), depth 33667(0)
Quoted 6 lines

I reproduced again a corruption with ext4 when we remove the spin lock alone with generic/750, the trace looks slightly different, and this time I ran the test with ext4 with 2k block size filesystem. I also reverted both "mm: migrate: remove folio_migrate_copy()" and "mm: migrate: support poisoned recover from migrate folio" just in case the extra check added by the later was helping avoid the corruption.

Since we didn't have much data over this filesystem corruption I figured it would be good to expand on that with more information. I reproduced yet again with just generic/750 and the spinlock removed in about ~6 hours:

Apr 01 18:17:16 fix-ext4-2k kernel: Linux version 6.14.0-12246-g6ee4cc7e5950-dirty (mcgrof@beef) (gcc (Debian 14.2.0-6) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #9 SMP PREEMPT_DYNAMIC Tue Apr  1 18:12:51 PDT 2025
<-- snip -->
Apr 01 18:17:43 fix-ext4-2k unknown: run fstests generic/750 at 2025-04-01 18:17:43
Apr 01 18:17:44 fix-ext4-2k kernel: EXT4-fs (loop5): mounted filesystem 218f1368-a3fd-42d5-9869-82b1539c5e74 r/w with ordered data mode. Quota mode: none.
Apr 02 00:18:24 fix-ext4-2k kernel: ------------[ cut here ]------------
Apr 02 00:18:24 fix-ext4-2k kernel: WARNING: CPU: 5 PID: 5588 at fs/jbd2/transaction.c:1552 jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Apr 02 00:18:24 fix-ext4-2k kernel: Modules linked in: loop sunrpc nls_iso8859_1 kvm_intel nls_cp437 9p vfat fat kvm crc32c_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio virtio_console virtio_balloon button evdev joydev serio_raw nvme_fabrics dm_mod nvme_core drm nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover failover virtio_blk psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Apr 02 00:18:24 fix-ext4-2k kernel: CPU: 5 UID: 0 PID: 5588 Comm: kworker/u38:0 Not tainted 6.14.0-12246-g6ee4cc7e5950-dirty #9 PREEMPT(full) 
Apr 02 00:18:24 fix-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.08-2 10/04/2024
Apr 02 00:18:24 fix-ext4-2k kernel: Workqueue: writeback wb_workfn (flush-7:5)
Apr 02 00:18:24 fix-ext4-2k kernel: RIP: 0010:jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Apr 02 00:18:24 fix-ext4-2k kernel: Code: 30 0f 84 5b fe ff ff 0f 0b 41 bc 8b ff ff ff e9 69 fe ff ff 48 8b 04 24 4c 8b 48 70 4d 39 cf 0f 84 53 ff ff ff e9 42 c5 00 00 <0f> 0b 41 bc e4 ff ff ff e9 41 ff ff ff 0f 0b 90 0f 1f 40 00 90 90
Apr 02 00:18:24 fix-ext4-2k kernel: RSP: 0018:ffffb44046157588 EFLAGS: 00010246
Apr 02 00:18:24 fix-ext4-2k kernel: RAX: 0000000000000001 RBX: ffff9a73d04a58e8 RCX: 00000000000000fd
Apr 02 00:18:24 fix-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: RBP: ffff9a74c2c4e478 R08: ffff9a74c2c4e478 R09: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: R10: 000000007b4f6d91 R11: 0000000000003151 R12: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: R13: ffff9a74c8d44460 R14: ffff9a73d04a58f0 R15: ffff9a74e92be300
Apr 02 00:18:24 fix-ext4-2k kernel: FS:  0000000000000000(0000) GS:ffff9a7597a0c000(0000) knlGS:0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 02 00:18:24 fix-ext4-2k kernel: CR2: 00007ff766871000 CR3: 0000000116e6a002 CR4: 0000000000772ef0
Apr 02 00:18:24 fix-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Apr 02 00:18:24 fix-ext4-2k kernel: PKRU: 55555554
Apr 02 00:18:24 fix-ext4-2k kernel: Call Trace:
Apr 02 00:18:24 fix-ext4-2k kernel:  <TASK>
Apr 02 00:18:24 fix-ext4-2k kernel:  __ext4_handle_dirty_metadata+0x6d/0x190 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_stack_trace_consume_entry+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ? arch_stack_walk+0x88/0xf0
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  do_writepages+0xdd/0x250
Apr 02 00:18:24 fix-ext4-2k kernel:  __writeback_single_inode+0x41/0x330
Apr 02 00:18:24 fix-ext4-2k kernel:  writeback_sb_inodes+0x21b/0x4e0
Apr 02 00:18:24 fix-ext4-2k kernel:  wb_writeback+0x89/0x320
Apr 02 00:18:24 fix-ext4-2k kernel:  wb_workfn+0xbe/0x440
Apr 02 00:18:24 fix-ext4-2k kernel:  ? queue_delayed_work_on+0x6b/0x80
Apr 02 00:18:24 fix-ext4-2k kernel:  process_one_work+0x188/0x350
Apr 02 00:18:24 fix-ext4-2k kernel:  worker_thread+0x255/0x3a0
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  kthread+0x112/0x250
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ? _raw_spin_unlock+0x15/0x30
Apr 02 00:18:24 fix-ext4-2k kernel:  ? finish_task_switch.isra.0+0x94/0x290
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ret_from_fork+0x2d/0x50
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 00:18:24 fix-ext4-2k kernel:  </TASK>
Apr 02 00:18:24 fix-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Apr 02 00:18:24 fix-ext4-2k kernel: ------------[ cut here ]------------
Apr 02 00:18:24 fix-ext4-2k kernel: WARNING: CPU: 5 PID: 5588 at fs/ext4/ext4_jbd2.c:360 __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel: Modules linked in: loop sunrpc nls_iso8859_1 kvm_intel nls_cp437 9p vfat fat kvm crc32c_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio virtio_console virtio_balloon button evdev joydev serio_raw nvme_fabrics dm_mod nvme_core drm nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover failover virtio_blk psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Apr 02 00:18:24 fix-ext4-2k kernel: CPU: 5 UID: 0 PID: 5588 Comm: kworker/u38:0 Tainted: G        W           6.14.0-12246-g6ee4cc7e5950-dirty #9 PREEMPT(full) 
Apr 02 00:18:24 fix-ext4-2k kernel: Tainted: [W]=WARN
Apr 02 00:18:24 fix-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.08-2 10/04/2024
Apr 02 00:18:24 fix-ext4-2k kernel: Workqueue: writeback wb_workfn (flush-7:5)
Apr 02 00:18:24 fix-ext4-2k kernel: RIP: 0010:__ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel: Code: 00 00 00 44 89 fa 4c 89 f6 49 c7 c1 3b 1c 77 c0 4c 89 e7 41 bd fb ff ff ff e8 5a ce 05 00 eb 97 48 89 df e8 c0 a4 4a e2 eb 8a <0f> 0b 48 c7 c2 a0 32 76 c0 45 89 e8 48 89 e9 44 89 fe 4c 89 f7 e8
Apr 02 00:18:24 fix-ext4-2k kernel: RSP: 0018:ffffb440461575c8 EFLAGS: 00010286
Apr 02 00:18:24 fix-ext4-2k kernel: RAX: ffff9a74e1d5d800 RBX: ffff9a74c2c4e478 RCX: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00000000ffffffff
Apr 02 00:18:24 fix-ext4-2k kernel: RBP: ffff9a74c8d44460 R08: ffff9a74c2c4e478 R09: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: R10: 000000007b4f6d91 R11: 0000000000003151 R12: ffff9a73c12236c8
Apr 02 00:18:24 fix-ext4-2k kernel: R13: 00000000ffffffe4 R14: ffffffffc0763760 R15: 0000000000000556
Apr 02 00:18:24 fix-ext4-2k kernel: FS:  0000000000000000(0000) GS:ffff9a7597a0c000(0000) knlGS:0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 02 00:18:24 fix-ext4-2k kernel: CR2: 00007ff766871000 CR3: 0000000116e6a002 CR4: 0000000000772ef0
Apr 02 00:18:24 fix-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 02 00:18:24 fix-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Apr 02 00:18:24 fix-ext4-2k kernel: PKRU: 55555554
Apr 02 00:18:24 fix-ext4-2k kernel: Call Trace:
Apr 02 00:18:24 fix-ext4-2k kernel:  <TASK>
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_stack_trace_consume_entry+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ? arch_stack_walk+0x88/0xf0
Apr 02 00:18:24 fix-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 00:18:24 fix-ext4-2k kernel:  do_writepages+0xdd/0x250
Apr 02 00:18:24 fix-ext4-2k kernel:  __writeback_single_inode+0x41/0x330
Apr 02 00:18:24 fix-ext4-2k kernel:  writeback_sb_inodes+0x21b/0x4e0
Apr 02 00:18:24 fix-ext4-2k kernel:  wb_writeback+0x89/0x320
Apr 02 00:18:24 fix-ext4-2k kernel:  wb_workfn+0xbe/0x440
Apr 02 00:18:24 fix-ext4-2k kernel:  ? queue_delayed_work_on+0x6b/0x80
Apr 02 00:18:24 fix-ext4-2k kernel:  process_one_work+0x188/0x350
Apr 02 00:18:24 fix-ext4-2k kernel:  worker_thread+0x255/0x3a0
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  kthread+0x112/0x250
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ? _raw_spin_unlock+0x15/0x30
Apr 02 00:18:24 fix-ext4-2k kernel:  ? finish_task_switch.isra.0+0x94/0x290
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ret_from_fork+0x2d/0x50
Apr 02 00:18:24 fix-ext4-2k kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 00:18:24 fix-ext4-2k kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 00:18:24 fix-ext4-2k kernel:  </TASK>
Apr 02 00:18:24 fix-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs: ext4_ext_grow_indepth:1366: aborting transaction: error 28 in __ext4_handle_dirty_metadata
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5): ext4_ext_grow_indepth:1366: inode #1010806: block 3112384: comm kworker/u38:0: journal_dirty_metadata failed: handle type 2 started at line 2721, credits 11/0, errcode -28
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_mb_clear_bb:6550: Readonly filesystem
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_reserve_inode_write:5870: Readonly filesystem
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2336: inode #1010806: comm kworker/u38:0: mark_inode_dirty error
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2338: comm kworker/u38:0: Failed to mark inode 1010806 dirty
Apr 02 00:18:24 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_do_writepages:2751: error 28
#8 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 22 lines · Luis Chamberlain
Quoted 14 lines · Luis Chamberlain
Quoted 6 lines · Luis Chamberlain

Although we don't have exact traces of the filesystem corruption we can can reproduce fs corruption one ext4 by just removing the spinlock and stress testing the filesystem with generic/750, we eventually end up after 3 hours of testing with kdevops using libvirt on the ext4 profile ext4-4k. Things like the below as reported recently [1]:

Quoted 6 lines
Mar 28 03:36:37 extra-ext4-4k unknown: run fstests generic/750 at 2025-03-28 03:36:37
<-- etc -->
Mar 28 05:57:09 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_get_first_dir_block:3538: inode #5174: comm fsstress: directory missing '.'
Mar 28 06:04:43 extra-ext4-4k kernel: EXT4-fs warning (device loop5): ext4_empty_dir:3088: inode #5176: comm fsstress: directory missing '.'
Mar 28 06:42:05 extra-ext4-4k kernel: EXT4-fs error (device loop5): __ext4_find_entry:1626: inode #5173: comm fsstress: checksumming directory block 0
Mar 28 08:16:43 extra-ext4-4k kernel: EXT4-fs error (device loop5): ext4_find_extent:938: inode #1104560: comm fsstress: pblk 4932229 bad header/extent: invalid magic - magic 8383, entries 33667, max 33667(0), depth 33667(0)
Quoted 6 lines

I reproduced again a corruption with ext4 when we remove the spin lock alone with generic/750, the trace looks slightly different, and this time I ran the test with ext4 with 2k block size filesystem. I also reverted both "mm: migrate: remove folio_migrate_copy()" and "mm: migrate: support poisoned recover from migrate folio" just in case the extra check added by the later was helping avoid the corruption.

Quoted 3 lines

Since we didn't have much data over this filesystem corruption I figured it would be good to expand on that with more information. I reproduced yet again with just generic/750 and the spinlock removed in about ~6 hours:

Here's another for the collection, this one reproduced also on a 2k ext4 filesytem within just an hour, and also runs into "error count since last fsck":

Apr 02 17:49:49 fix-ext4-2k kernel: Linux version 6.14.0-12248-g87aeaf206469 (mcgrof@beef) (gcc (Debian 14.2.0-6) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #16 SMP PREEMPT_DYNAMIC Wed Apr  2 17:43:14 PDT 2025

<-- snip -->

Apr 02 17:50:50 fix-ext4-2k unknown: run fstests generic/750 at 2025-04-02 17:50:50
Apr 02 17:50:51 fix-ext4-2k kernel: EXT4-fs (loop5): mounted filesystem 2f6642d1-eab9-481a-86b4-dac9cc760f63 r/w with ordered data mode. Quota mode: none.
Apr 02 18:00:45 fix-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Apr 02 18:13:27 fix-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Apr 02 18:47:36 fix-ext4-2k kernel: ------------[ cut here ]------------
Apr 02 18:47:36 fix-ext4-2k kernel: WARNING: CPU: 4 PID: 2515 at fs/jbd2/transaction.c:1552 jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Apr 02 18:47:36 fix-ext4-2k kernel: Modules linked in: loop sunrpc 9p nls_iso8859_1 nls_cp437 vfat crc32c_generic fat kvm_intel kvm ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio joydev virtio_console virtio_balloon button evdev serio_raw nvme_fabrics nvme_core dm_mod drm nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover virtio_blk failover psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Apr 02 18:47:36 fix-ext4-2k kernel: CPU: 4 UID: 0 PID: 2515 Comm: fsstress Not tainted 6.14.0-12248-g87aeaf206469 #16 PREEMPT(full) 
Apr 02 18:47:36 fix-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.08-2 10/04/2024
Apr 02 18:47:36 fix-ext4-2k kernel: RIP: 0010:jbd2_journal_dirty_metadata+0x21c/0x230 [jbd2]
Apr 02 18:47:36 fix-ext4-2k kernel: Code: 30 0f 84 5b fe ff ff 0f 0b 41 bc 8b ff ff ff e9 69 fe ff ff 48 8b 04 24 4c 8b 48 70 4d 39 cf 0f 84 53 ff ff ff e9 42 c5 00 00 <0f> 0b 41 bc e4 ff ff ff e9 41 ff ff ff 0f 0b 90 0f 1f 40 00 90 90
Apr 02 18:47:36 fix-ext4-2k kernel: RSP: 0018:ffffa464824d77f8 EFLAGS: 00010246
Apr 02 18:47:36 fix-ext4-2k kernel: RAX: 0000000000000001 RBX: ffff95c8866129d8 RCX: 00000000000000fd
Apr 02 18:47:36 fix-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: RBP: ffff95c88726ac98 R08: ffff95c88726ac98 R09: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R10: 00000000b2ae40cd R11: 00000000000064a2 R12: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R13: ffff95c9849e7f88 R14: ffff95c8866129e0 R15: ffff95c986708180
Apr 02 18:47:36 fix-ext4-2k kernel: FS:  00007f230b292740(0000) GS:ffff95ca439cc000(0000) knlGS:0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 02 18:47:36 fix-ext4-2k kernel: CR2: 00007f230b28a000 CR3: 000000012517c003 CR4: 0000000000772ef0
Apr 02 18:47:36 fix-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Apr 02 18:47:36 fix-ext4-2k kernel: PKRU: 55555554
Apr 02 18:47:36 fix-ext4-2k kernel: Call Trace:
Apr 02 18:47:36 fix-ext4-2k kernel:  <TASK>
Apr 02 18:47:36 fix-ext4-2k kernel:  __ext4_handle_dirty_metadata+0x6d/0x190 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  do_writepages+0xdd/0x250
Apr 02 18:47:36 fix-ext4-2k kernel:  ? __call_rcu_common.constprop.0+0x130/0x290
Apr 02 18:47:36 fix-ext4-2k kernel:  ? iter_file_splice_write+0x1d6/0x4f0
Apr 02 18:47:36 fix-ext4-2k kernel:  ? kfree+0x1cc/0x370
Apr 02 18:47:36 fix-ext4-2k kernel:  filemap_fdatawrite_wbc+0x48/0x60
Apr 02 18:47:36 fix-ext4-2k kernel:  __filemap_fdatawrite_range+0x5b/0x80
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_release_file+0x6d/0xa0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  __fput+0xf4/0x2b0
Apr 02 18:47:36 fix-ext4-2k kernel:  __x64_sys_close+0x39/0x80
Apr 02 18:47:36 fix-ext4-2k kernel:  do_syscall_64+0x4b/0x120
Apr 02 18:47:36 fix-ext4-2k kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 02 18:47:36 fix-ext4-2k kernel: RIP: 0033:0x7f230b324687
Apr 02 18:47:36 fix-ext4-2k kernel: Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff
Apr 02 18:47:36 fix-ext4-2k kernel: RSP: 002b:00007ffc1efe2bb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Apr 02 18:47:36 fix-ext4-2k kernel: RAX: ffffffffffffffda RBX: 00007f230b292740 RCX: 00007f230b324687
Apr 02 18:47:36 fix-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
Apr 02 18:47:36 fix-ext4-2k kernel: RBP: 00007ffc1efe2c60 R08: 0000000000000000 R09: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R13: 00007ffc1efe2db0 R14: 00007ffc1efe31b0 R15: 000000000000622d
Apr 02 18:47:36 fix-ext4-2k kernel:  </TASK>
Apr 02 18:47:36 fix-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Apr 02 18:47:36 fix-ext4-2k kernel: ------------[ cut here ]------------
Apr 02 18:47:36 fix-ext4-2k kernel: WARNING: CPU: 2 PID: 2515 at fs/ext4/ext4_jbd2.c:360 __ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel: Modules linked in: loop sunrpc 9p nls_iso8859_1 nls_cp437 vfat crc32c_generic fat kvm_intel kvm ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd 9pnet_virtio joydev virtio_console virtio_balloon button evdev serio_raw nvme_fabrics nvme_core dm_mod drm nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod virtio_net net_failover virtio_blk failover psmouse virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
Apr 02 18:47:36 fix-ext4-2k kernel: CPU: 2 UID: 0 PID: 2515 Comm: fsstress Tainted: G        W           6.14.0-12248-g87aeaf206469 #16 PREEMPT(full) 
Apr 02 18:47:36 fix-ext4-2k kernel: Tainted: [W]=WARN
Apr 02 18:47:36 fix-ext4-2k kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.08-2 10/04/2024
Apr 02 18:47:36 fix-ext4-2k kernel: RIP: 0010:__ext4_handle_dirty_metadata+0x102/0x190 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel: Code: 00 00 00 44 89 fa 4c 89 f6 49 c7 c1 3b 2c aa c0 4c 89 e7 41 bd fb ff ff ff e8 5a ce 05 00 eb 97 48 89 df e8 c0 a4 0a f6 eb 8a <0f> 0b 48 c7 c2 a0 42 a9 c0 45 89 e8 48 89 e9 44 89 fe 4c 89 f7 e8
Apr 02 18:47:36 fix-ext4-2k kernel: RSP: 0018:ffffa464824d7838 EFLAGS: 00010286
Apr 02 18:47:36 fix-ext4-2k kernel: RAX: ffff95c998971800 RBX: ffff95c88726ac98 RCX: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00000000ffffffff
Apr 02 18:47:36 fix-ext4-2k kernel: RBP: ffff95c9849e7f88 R08: ffff95c88726ac98 R09: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R10: 00000000b2ae40cd R11: 00000000000064a2 R12: ffff95c999e4c430
Apr 02 18:47:36 fix-ext4-2k kernel: R13: 00000000ffffffe4 R14: ffffffffc0a94760 R15: 0000000000000556
Apr 02 18:47:36 fix-ext4-2k kernel: FS:  00007f230b292740(0000) GS:ffff95ca4394c000(0000) knlGS:0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 02 18:47:36 fix-ext4-2k kernel: CR2: 00007f230b291000 CR3: 000000012517c006 CR4: 0000000000772ef0
Apr 02 18:47:36 fix-ext4-2k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Apr 02 18:47:36 fix-ext4-2k kernel: PKRU: 55555554
Apr 02 18:47:36 fix-ext4-2k kernel: Call Trace:
Apr 02 18:47:36 fix-ext4-2k kernel:  <TASK>
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_ext_insert_extent+0x5c1/0x1510 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_cache_extents+0x5a/0xd0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_find_extent+0x37c/0x3a0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_ext_map_blocks+0x51e/0x1900 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? mpage_map_and_submit_buffers+0x23f/0x270 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_map_blocks+0x11a/0x4d0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? kmem_cache_alloc_noprof+0x321/0x3e0
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_do_writepages+0x762/0xd40 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ? ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_writepages+0xd7/0x1b0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  do_writepages+0xdd/0x250
Apr 02 18:47:36 fix-ext4-2k kernel:  ? __call_rcu_common.constprop.0+0x130/0x290
Apr 02 18:47:36 fix-ext4-2k kernel:  ? iter_file_splice_write+0x1d6/0x4f0
Apr 02 18:47:36 fix-ext4-2k kernel:  ? kfree+0x1cc/0x370
Apr 02 18:47:36 fix-ext4-2k kernel:  filemap_fdatawrite_wbc+0x48/0x60
Apr 02 18:47:36 fix-ext4-2k kernel:  __filemap_fdatawrite_range+0x5b/0x80
Apr 02 18:47:36 fix-ext4-2k kernel:  ext4_release_file+0x6d/0xa0 [ext4]
Apr 02 18:47:36 fix-ext4-2k kernel:  __fput+0xf4/0x2b0
Apr 02 18:47:36 fix-ext4-2k kernel:  __x64_sys_close+0x39/0x80
Apr 02 18:47:36 fix-ext4-2k kernel:  do_syscall_64+0x4b/0x120
Apr 02 18:47:36 fix-ext4-2k kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 02 18:47:36 fix-ext4-2k kernel: RIP: 0033:0x7f230b324687
Apr 02 18:47:36 fix-ext4-2k kernel: Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff
Apr 02 18:47:36 fix-ext4-2k kernel: RSP: 002b:00007ffc1efe2bb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Apr 02 18:47:36 fix-ext4-2k kernel: RAX: ffffffffffffffda RBX: 00007f230b292740 RCX: 00007f230b324687
Apr 02 18:47:36 fix-ext4-2k kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
Apr 02 18:47:36 fix-ext4-2k kernel: RBP: 00007ffc1efe2c60 R08: 0000000000000000 R09: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
Apr 02 18:47:36 fix-ext4-2k kernel: R13: 00007ffc1efe2db0 R14: 00007ffc1efe31b0 R15: 000000000000622d
Apr 02 18:47:36 fix-ext4-2k kernel:  </TASK>
Apr 02 18:47:36 fix-ext4-2k kernel: ---[ end trace 0000000000000000 ]---
Apr 02 18:47:36 fix-ext4-2k kernel: EXT4-fs: ext4_ext_grow_indepth:1366: aborting transaction: error 28 in __ext4_handle_dirty_metadata
Apr 02 18:47:36 fix-ext4-2k kernel: EXT4-fs error (device loop5): ext4_ext_grow_indepth:1366: inode #755485: block 3924606: comm fsstress: journal_dirty_metadata failed: handle type 2 started at line 2721, credits 11/0, errcode -28
Apr 02 18:47:36 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_mb_clear_bb:6550: Readonly filesystem
Apr 02 18:47:36 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_reserve_inode_write:5870: Readonly filesystem
Apr 02 18:47:37 fix-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2336: inode #755485: comm fsstress: mark_inode_dirty error
Apr 02 18:47:37 fix-ext4-2k kernel: EXT4-fs error (device loop5): mpage_map_and_submit_extent:2338: comm fsstress: Failed to mark inode 755485 dirty
Apr 02 18:47:37 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_do_writepages:2751: error 28
Apr 03 05:43:31 fix-ext4-2k kernel: NOHZ tick-stop error: local softirq work is pending, handler #10!!!
Apr 03 19:32:37 fix-ext4-2k kernel: EXT4-fs (loop5): error count since last fsck: 6
Apr 03 19:32:37 fix-ext4-2k kernel: EXT4-fs (loop5): initial error at time 1743644856: ext4_ext_grow_indepth:1366: inode 755485: block 3924606
Apr 03 19:32:37 fix-ext4-2k kernel: EXT4-fs (loop5): last error at time 1743644857: ext4_do_writepages:2751
Apr 04 08:12:34 fix-ext4-2k kernel: EXT4-fs: ext4_ext_grow_indepth:1366: aborting transaction: error 28 in __ext4_handle_dirty_metadata
Apr 04 08:12:35 fix-ext4-2k kernel: EXT4-fs error (device loop5): ext4_ext_grow_indepth:1366: inode #330657: block 855807: comm kworker/u35:9: journal_dirty_metadata failed: handle type 2 started at line 2721, credits 11/0, errcode -28
Apr 04 08:12:35 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_mb_clear_bb:6550: Readonly filesystem
Apr 04 08:12:35 fix-ext4-2k kernel: EXT4-fs error (device loop5) in ext4_do_writepages:2751: error 28
reviewer
#9 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

In reply to Luis Chamberlain

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 213

Quoted 11 lines · Luis Chamberlain
fs/buffer.c +3 -0
index c7abb4a029dc..a4e4455a6ce2 100644
@@ -208,6 +208,15 @@ __find_get_block_slow(struct block_device *bdev, sector_t block)
 	head = folio_buffers(folio);
 	if (!head)
 		goto out_unlock;
+
+	if (folio->mapping->a_ops->migrate_folio &&
+	    folio->mapping->a_ops->migrate_folio == buffer_migrate_folio_norefs) {

This is always true for bdev mapping we have here, isn't it?

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 214

Quoted 1 line
fs/buffer.c +1 -0
+		if (folio_test_lru(folio) &&

Do you expect bdev page cache to contain non-LRU folios? I thought every pagecache folio is on LRU so this seems pointless as well? But I may be missing something here.

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 217

Quoted 3 lines
fs/buffer.c +3 -0
+		    folio_test_locked(folio) &&
+		    !folio_test_writeback(folio))
+			goto out_unlock;

I find this problematic. It fixes the race with migration, alright (although IMO we should have a comment very well explaining the interplay of folio lock and mapping->private_lock to make this work - probably in buffer_migrate_folio_norefs() - and reference it from here), but there are places which expect that if __find_get_block() doesn't return anything, this block is not cached in the buffer cache. And your change breaks this assumption. Look for example at write_boundary_block(), that will fail to write the block it should write if it races with someone locking the folio after your changes. Similarly the code tracking state of deleted metadata blocks in fs/jbd2/revoke.c will fail to properly update buffer's state if __find_get_block() suddently starts returning NULL although the buffer is present in cache.

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 222

Quoted 5 lines
fs/buffer.c +2 -0
+	}
+
 	bh = head;
 	do {
 		if (!buffer_mapped(bh))

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#10 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 4 lines · Jan Kara

I find this problematic. It fixes the race with migration, alright (although IMO we should have a comment very well explaining the interplay of folio lock and mapping->private_lock to make this work - probably in buffer_migrate_folio_norefs() - and reference it from here), but there are

Quoted 8 lines

places which expect that if __find_get_block() doesn't return anything, this block is not cached in the buffer cache. And your change breaks this assumption. Look for example at write_boundary_block(), that will fail to write the block it should write if it races with someone locking the folio after your changes. Similarly the code tracking state of deleted metadata blocks in fs/jbd2/revoke.c will fail to properly update buffer's state if __find_get_block() suddently starts returning NULL although the buffer is present in cache.

Yeah - one thing I was thinking about, _iff_ failing lookups (__find_get_block() returning nil) during migration is in fact permitted, was adding a BH_migrate flag and serialize vs __buffer_migrate_folio() entirely. Semantically there are no users, and none are added during this window, but as a consequence I suppose one thread could see the page not cached, act upon that, then see it cached once the migration is done and get confused(?). So I don't see a problem here for write_boundary_block() specifically, but I'm probably overlooking others.

Now, if bailing on the lookup is not an option, meaning it must wait for the migration to complete, I'm not sure large folios will ever be compatible with the "Various filesystems appear to want __find_get_block to be non-blocking." comment.

So the below could be tucked in for norefs only (because this is about the addr space i_private_lock), but this also shortens the hold time; if that matters at all, of course, vs changing the migration semantics.

Thanks,
Davidlohr

---8<----------------------------

fs/buffer.c +0 -0
index cc8452f60251..f585339ae2e4 100644
@@ -208,6 +208,14 @@ __find_get_block_slow(struct block_device *bdev, sector_t block)
	head = folio_buffers(folio);
	if (!head)
		goto out_unlock;

+
+ bh = head;
+ do {
+ if (test_bit(BH_migrate, &bh->b_state))
+ goto out_unlock;
+ bh = bh->b_this_page;
+ } while (bh != head);
+

	bh = head;
	do {
		if (!buffer_mapped(bh))
include/linux/buffer_head.h +0 -0
index 932139c5d46f..e956a1509a05 100644
@@ -34,6 +34,7 @@ enum bh_state_bits {

BH_Meta, /* Buffer contains metadata */
BH_Prio, /* Buffer should be submitted with REQ_PRIO */
BH_Defer_Completion, /* Defer AIO completion to workqueue */
+ BH_migrate, /* Buffer is being migrated */

BH_PrivateStart,/* not a state bit, but the first bit available
* for private allocation by other entities

mm/migrate.c +0 -0
index a073eb6c5009..0ffa8b478fd3 100644
@@ -846,6 +846,12 @@ static int __buffer_migrate_folio(struct address_space *mapping,
	if (!buffer_migrate_lock_buffers(head, mode))
		return -EAGAIN;

+ bh = head;
+ do {
+ set_bit(BH_migrate, &bh->b_state);
+ bh = bh->b_this_page;
+ } while (bh != head);
+

	if (check_refs) {
		bool busy;
		bool invalidated = false;
@@ -861,12 +867,12 @@ static int __buffer_migrate_folio(struct address_space *mapping,
			}
			bh = bh->b_this_page;
		} while (bh != head);
+		spin_unlock(&mapping->i_private_lock);
		if (busy) {
			if (invalidated) {
				rc = -EAGAIN;
				goto unlock_buffers;
			}
-			spin_unlock(&mapping->i_private_lock);
			invalidate_bh_lrus();
			invalidated = true;
			goto recheck_buffers;
@@ -884,10 +890,9 @@ static int __buffer_migrate_folio(struct address_space *mapping,
	} while (bh != head);

unlock_buffers:
- if (check_refs)

-		spin_unlock(&mapping->i_private_lock);
	bh = head;
	do {
+		clear_bit(BH_migrate, &bh->b_state)
		unlock_buffer(bh);
		bh = bh->b_this_page;
	} while (bh != head);
#11 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 3 lines · Davidlohr Bueso

So the below could be tucked in for norefs only (because this is about the addr space i_private_lock), but this also shortens the hold time; if that matters at all, of course, vs changing the migration semantics.

I like this approach a lot better. One wrinkle is that it doesn't seem that we need to set the BH_Migrate bit on every buffer; we could define that it's only set on the head BH, right?

#12 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Davidlohr Bueso

So the below could be tucked in for norefs only (because this is about the addr space i_private_lock), but this also shortens the hold time; if that matters at all, of course, vs changing the migration semantics.

I like this approach a lot better. One wrinkle is that it doesn't seem that we need to set the BH_Migrate bit on every buffer; we could define that it's only set on the head BH, right?

Yes, we are also only doing this for block devices, and for migration purposes. Even though a bit from one buffer may be desirable it makes no sense to allow for that in case migration is taking place. So indeed we have no need to add the flag for all buffers.

I think the remaining question is what users of __find_get_block_slow() can really block, and well I've started trying to determine that with coccinelle [0], its gonna take some more time.

Perhaps its easier to ask, why would a block device mapping want to allow __find_get_block_slow() to not block?

[0] https://lkml.kernel.org/r/[email protected]

Luis

reviewer
#13 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 15 lines · Luis Chamberlain
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Davidlohr Bueso

So the below could be tucked in for norefs only (because this is about the addr space i_private_lock), but this also shortens the hold time; if that matters at all, of course, vs changing the migration semantics.

I like this approach a lot better. One wrinkle is that it doesn't seem that we need to set the BH_Migrate bit on every buffer; we could define that it's only set on the head BH, right?

Yes, we are also only doing this for block devices, and for migration purposes. Even though a bit from one buffer may be desirable it makes no sense to allow for that in case migration is taking place. So indeed we have no need to add the flag for all buffers.

Quoted 6 lines

I think the remaining question is what users of __find_get_block_slow() can really block, and well I've started trying to determine that with coccinelle [0], its gonna take some more time.

Perhaps its easier to ask, why would a block device mapping want to allow __find_get_block_slow() to not block?

So I've audited all callers of __find_get_block_slow() (there aren't that many) and most of them are actually fine with sleeping these days.

Analysis:
__find_get_block_slow() is only used from __find_get_block().

__find_get_block() is used from:
write_boundary_block() - locks the buffer so can sleep bdev_getblk() - allocates buffers with 'gfp' mask. We use GFP_NOWAIT mask
from some places (generally doing readahead). For callers where !gfpflags_allow_blocking() we should bail rather than block on migration. Callers are currently fine with this, we should probably document that bdev_getblk() with restrictive gfp mask may fail even if bh is present - or perhaps make this even more explicit in the API by providing bdev_try_getblk() and make bdev_getblk() assert gfp mask allows sleeping.
__getblk_slow() - only called from bdev_getblk(). Probably should fold
there.
ocfs2_force_read_journal() - allows sleeping as it does IO jbd2_journal_revoke() - can sleep (has might_sleep() in the beginning) jbd2_journal_cancel_revoke() - only used from do_get_write_access() and
do_get_create_access() which do sleep. So can sleep.
jbd2_clear_buffer_revoked_flags() - only called from journal commit code
which sleeps. So can sleep.

The last user is sb_find_get_block() which is used from:
hpfs_prefetch_sectors() - prefers bail rather than blocking fat_dir_readahead() - prefers bail rather than blocking exfat_dir_readahead() - prefers bail rather than blocking ext4_free_blocks() - can sleep
ext4_getblk() - depending on EXT4_GET_BLOCKS_CACHED_NOWAIT flag either can
sleep or must bail (and is fine with it) rather than sleeping
fs/ext4/ialloc.c:recently_deleted() - this one is the most problematic
place. It must bail rather than sleeping (called under a spinlock) but it depends on the fact that if bh is not returned, then the data has been written out and evicted from memory. Luckily, the usage of recently_deleted() is mostly an optimization to reduce damage in case of crash so rare false failure should be OK. Ted, what is your opinion?

And this is actually all. So it seems that if we give possibility to callers to tell whether they want to bail or wait for migration, things should work out fine.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#14 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

Quoted 6 lines · Jan Kara

fs/ext4/ialloc.c:recently_deleted() - this one is the most problematic
place. It must bail rather than sleeping (called under a spinlock) but it depends on the fact that if bh is not returned, then the data has been written out and evicted from memory. Luckily, the usage of recently_deleted() is mostly an optimization to reduce damage in case of crash so rare false failure should be OK. Ted, what is your opinion?

Yes, if we can just assume that inode has not been recently deleted in the rare case where a miogration is taking place, that should be fine. So in practice, recently_deleted() could just call some variant of find_get_block() (with some flag ) which returns NULL if we need to sleep (e.g., if it is not in the buffer cache so a read would need to take place, or we need to wait for the page migration to complete), that should work fine.

Thanks,

- Ted

#15 lore raw

Re: [PATCH 2/3] fs/buffer: avoid races with folio migrations on __find_get_block_slow()

In reply to Jan Kara
Quoted 14 lines · Jan Kara

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 213

Quoted 11 lines · Luis Chamberlain
fs/buffer.c +3 -0
index c7abb4a029dc..a4e4455a6ce2 100644
@@ -208,6 +208,15 @@ __find_get_block_slow(struct block_device *bdev, sector_t block)
 	head = folio_buffers(folio);
 	if (!head)
 		goto out_unlock;
+
+	if (folio->mapping->a_ops->migrate_folio &&
+	    folio->mapping->a_ops->migrate_folio == buffer_migrate_folio_norefs) {

This is always true for bdev mapping we have here, isn't it?

Yes, thanks!

Quoted 6 lines

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 214

Quoted 1 line
fs/buffer.c +1 -0
+		if (folio_test_lru(folio) &&

Do you expect bdev page cache to contain non-LRU folios? I thought every pagecache folio is on LRU so this seems pointless as well? But I may be missing something here.

Quoted 11 lines

fs/buffer.c __find_get_block_slow(struct block_device *bdev, sector_t block) · line 217

Quoted 3 lines
fs/buffer.c +3 -0
+		    folio_test_locked(folio) &&
+		    !folio_test_writeback(folio))
+			goto out_unlock;

I find this problematic. It fixes the race with migration, alright (although IMO we should have a comment very well explaining the interplay of folio lock and mapping->private_lock to make this work - probably in buffer_migrate_folio_norefs() - and reference it from here), but there are places which expect that if __find_get_block() doesn't return anything, this block is not cached in the buffer cache. And your change breaks this assumption.

Yup agreed!

Luis

#16 lore raw

PATCH 3/3 mm/migrate: avoid atomic context on buffer_migrate_folio_norefs() migration

In reply to Luis Chamberlain

The buffer_migrate_folio_norefs() should avoid holding the spin lock held in order to ensure we can support large folios. The prior commit "fs/buffer: avoid races with folio migrations on __find_get_block_slow()" ripped out the only rationale for having the atomic context, so we can remove the spin lock call now.

Reported-by [email protected]
Fixes 3c20917120ce ("block/bdev: enable large folio support for large logical block sizes")
---
 mm/migrate.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
mm/migrate.c +1 -3
index 712ddd11f3f0..f3047c685706 100644
@@ -861,12 +861,12 @@ static int __buffer_migrate_folio(struct address_space *mapping,
 			}
 			bh = bh->b_this_page;
 		} while (bh != head);
+		spin_unlock(&mapping->i_private_lock);
 		if (busy) {
 			if (invalidated) {
 				rc = -EAGAIN;
 				goto unlock_buffers;
 			}
-			spin_unlock(&mapping->i_private_lock);
 			invalidate_bh_lrus();
 			invalidated = true;
 			goto recheck_buffers;
@@ -884,8 +884,6 @@ static int __buffer_migrate_folio(struct address_space *mapping,
 	} while (bh != head);
 
 unlock_buffers:
-	if (check_refs)
-		spin_unlock(&mapping->i_private_lock);
 	bh = head;
 	do {
 		unlock_buffer(bh);
Signature · 1 line
2.47.2