New: general protection fault in ext4 during simultaneous online resize and write operations

https://bugzilla.kernel.org/show_bug.cgi?id=206443

Bug ID: 206443
Summary: general protection fault in ext4 during simultaneous
online resize and write operations
Product: File System
Version: 2.5
Kernel Version: 5.5
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: ext4
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Created attachment 287189
--> https://bugzilla.kernel.org/attachment.cgi?id=287189&action=edit
proposed_fix.patch

While writing to an ext4 file system partition during simultaneous online resize a general protection fault was encountered.

Reproducer:

truncate -s 100G /tmp/foo
sudo bash -c 'while true; do dd if=/dev/zero of=/mnt/xxx bs=1M count=1; sync; rm /mnt/xxx; done' &
while true; do mkfs.ext4 -b 1024 -E resize=26213883 /tmp/foo 2096635 -F; sudo mount -o loop /tmp/foo /mnt; sudo resize2fs /dev/loop0 26213883; sudo umount /mnt; done

The following call trace was observed:

[  886.837106] RIP: 0010:ext4_get_group_desc+0x46/0xa0 [ext4]
[  886.844922] Code: 41 8b 8a a8 00 00 00 41 89 f1 41 8b 42 38 41 d3 e9 49 8b
4a 70 83 e8 01 45 89 c8 21 f0 4a 8b 0c c1 48 85 c9 74 30 49 0f af 02 <48> 03 41
28 48 85 d2 74 03 48 89 0a f3 c3 41 89 f0 48 c7 c1 b8 47
[  886.857215] RSP: 0018:ffffc9000018f7d0 EFLAGS: 00010202
[  886.860998] RAX: 0000000000000040 RBX: ffff8887d634a000 RCX:
6f26075a7d3c6d9e
[  886.865578] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff8887d634a000
[  886.870148] RBP: ffff8887d634c000 R08: 0000000000000000 R09:
0000000000000000
[  886.874731] R10: ffff8887d634c000 R11: 0000000000000000 R12:
0000000000000001
[  886.879306] R13: 0000000000000000 R14: ffff8887d634c000 R15:
ffff8887d0b32000
[  886.883881] FS:  0000000000000000(0000) GS:ffff8887dfa00000(0000)
knlGS:0000000000000000
[  886.890293] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  886.894293] CR2: 00007fc03aa34f30 CR3: 000000000200a006 CR4:
00000000007606e0
[  886.898875] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  886.903522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  886.908267] PKRU: 55555554
[  886.911046] Call Trace:
[  886.913708]  ext4_read_block_bitmap_nowait+0x2a/0x600 [ext4]
[  886.917718]  ext4_read_block_bitmap+0x14/0x50 [ext4]
[  886.921408]  ext4_mb_mark_diskspace_used+0x58/0x380 [ext4]
[  886.925317]  ext4_mb_new_blocks+0x2a4/0x720 [ext4]
[  886.928928]  ? ext4_find_extent+0x295/0x2e0 [ext4]
[  886.932543]  ext4_ext_map_blocks+0xa60/0xd70 [ext4]
[  886.936191]  ? __lock_page_killable+0x240/0x260
[  886.939698]  ext4_map_blocks+0x3ae/0x5d0 [ext4]
[  886.943205]  ext4_writepages+0x7bc/0xe70 [ext4]
[  886.946712]  ? nvme_queue_rq+0x4d8/0xa90 [nvme]
[  886.950207]  ? __update_load_avg_cfs_rq+0x12b/0x2b0
[  886.953846]  ? __update_load_avg_cfs_rq+0x12b/0x2b0
[  886.957491]  ? do_writepages+0x4b/0xe0
[  886.960673]  ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
[  886.964458]  do_writepages+0x4b/0xe0
[  886.967566]  ? enqueue_task_fair+0xa8/0xa40
[  886.970914]  ? __writeback_single_inode+0x3d/0x320
[  886.974521]  __writeback_single_inode+0x3d/0x320
[  886.978052]  ? ttwu_do_wakeup+0x19/0x140
[  886.981288]  writeback_sb_inodes+0x1b5/0x490
[  886.984675]  __writeback_inodes_wb+0x5d/0xb0
[  886.988069]  wb_writeback+0x265/0x2f0
[  886.991208]  ? wb_workfn+0x33f/0x400
[  886.994315]  wb_workfn+0x33f/0x400
[  886.997341]  process_one_work+0x195/0x380
[  887.000623]  worker_thread+0x30/0x390
[  887.003759]  ? process_one_work+0x380/0x380
[  887.007112]  kthread+0x113/0x130
[  887.010065]  ? kthread_park+0x90/0x90
[  887.013199]  ret_from_fork+0x35/0x40
[  887.016305] Modules linked in: loop ext4 crc16 mbcache jbd2 sunrpc mousedev

evdev psmouse button ena ip_tables x_tables xfs libcrc32c crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd nvme cryptd glue_helper nvme_core dm_mirror dm_region_hash dm_log dm_mod dax ipv6 crc_ccitt nf_defrag_ipv6 autofs4

[  887.035407] ---[ end trace dc25e57808972176 ]---
[  887.038960] RIP: 0010:ext4_get_group_desc+0x46/0xa0 [ext4]
[  887.042867] Code: 41 8b 8a a8 00 00 00 41 89 f1 41 8b 42 38 41 d3 e9 49 8b
4a 70 83 e8 01 45 89 c8 21 f0 4a 8b 0c c1 48 85 c9 74 30 49 0f af 02 <48> 03 41
28 48 85 d2 74 03 48 89 0a f3 c3 41 89 f0 48 c7 c1 b8 47
[  887.055189] RSP: 0018:ffffc9000018f7d0 EFLAGS: 00010202
[  887.058992] RAX: 0000000000000040 RBX: ffff8887d634a000 RCX:
6f26075a7d3c6d9e
[  887.063592] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff8887d634a000
[  887.068197] RBP: ffff8887d634c000 R08: 0000000000000000 R09:
0000000000000000
[  887.072795] R10: ffff8887d634c000 R11: 0000000000000000 R12:
0000000000000001
[  887.077399] R13: 0000000000000000 R14: ffff8887d634c000 R15:
ffff8887d0b32000
[  887.081996] FS:  0000000000000000(0000) GS:ffff8887dfa00000(0000)
knlGS:0000000000000000
[  887.088436] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  887.092447] CR2: 00007fc03aa34f30 CR3: 000000000200a006 CR4:
00000000007606e0
[  887.097060] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  887.101651] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  887.106252] PKRU: 55555554

Progress:

This looked likely to be a use after free in ext4_get_group_desc() of EXT4_SB(sb)->s_group_desc while it was being resized in add_new_gdb():
o_group_desc = EXT4_SB(sb)->s_group_desc;
{snip}
EXT4_SB(sb)->s_group_desc = n_group_desc; EXT4_SB(sb)->s_gdb_count++;
kvfree(o_group_desc);

Proposed fix:

The attached patch was proposed as a fix to use rcu locking around the access to s_group_desc.
A test run with this patch was done and while the initial problem was no longer encountered. New call traces (attached) were encountered this time.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #1 from Suraj ([email protected]) ---
Created attachment 287191
--> https://bugzilla.kernel.org/attachment.cgi?id=287191&action=edit
testcase

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #2 from Suraj ([email protected]) ---
Created attachment 287193
--> https://bugzilla.kernel.org/attachment.cgi?id=287193&action=edit
ext4_mb_load_buddy_gfp.trace

Call trace in ext4_mb_load_buddy_gfp

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #3 from Suraj ([email protected]) ---
Created attachment 287195
--> https://bugzilla.kernel.org/attachment.cgi?id=287195&action=edit
ext4_free_blocks.trace

Call trace in ext4_free_blocks

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #4 from Suraj ([email protected]) ---
Created attachment 287197
--> https://bugzilla.kernel.org/attachment.cgi?id=287197&action=edit
ext4_free_inode.trace

Call trace in ext4_free_inode

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #5 from Suraj ([email protected]) ---
Created attachment 287199
--> https://bugzilla.kernel.org/attachment.cgi?id=287199&action=edit
__ext4_new_inode.trace

Call trace in __ext4_new_inode

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #6 from Suraj ([email protected]) ---
Initial bug was hit reliably (~95% of the time) within 30 minutes. The following traces only occurred on ~50% of runs and some times taking up to 5 hours to hit.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #7 from Suraj ([email protected]) ---
The other crashes look to be related to access sbi->s_flex_groups and sbi->s_group_info which are reallocated in the same way that s_group_desc is on resize.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #8 from Suraj ([email protected]) ---
I've attached 2 additional patches which seem to resolve the s_group_info and s_flex_group issues in the same way as the s_group_desc reallocation was resolved. That is to used rcu to protect access to the pointer.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #9 from Suraj ([email protected]) ---
Created attachment 287249
--> https://bugzilla.kernel.org/attachment.cgi?id=287249&action=edit
s_group_info.patch

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #10 from Suraj ([email protected]) ---
Created attachment 287251
--> https://bugzilla.kernel.org/attachment.cgi?id=287251&action=edit
s_flex_group.patch

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #11 from Theodore Tso ([email protected]) --- Hi Suraj,

The two patches s_group_info.patch and s_flex_group.patch appear to use the helper function rcu_sbi_array_dereference() without defining it?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #12 from Theodore Tso ([email protected]) ---
I've posted a proposed improvement[1] to the first proposed patch[2] on LKML.

[1] https://lore.kernel.org/r/[email protected]
[2] https://bugzilla.kernel.org/attachment.cgi?id=287189

Suraj, please note that your patches are whitespace damaged, and are lacking
the Developer's Certification of Origin. In the future, it would save me a
lot of time you take a look at the Submitting Patches[3] instructions from the kernel documentation.

[3] https://www.kernel.org/doc/html/latest/process/submitting-patches.html

You can either use e-mail to [email protected] or attach patches to a Bugzilla entry. although the former is certainly preferred. It's better to send a proposal to the [email protected], since that way the patch can also get tracked via patchwork[4], and on lore.kernel.org, as in [1] above.

[4] http://patchwork.ozlabs.org/project/linux-ext4/list/

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #13 from Suraj ([email protected]) ---
Your approach using call_rcu is very similar to something I was also exploring so that looks good to me.

Apologies, I didn't realise that patches attached to the BZ had to be mailing list ready.

There is also the same issue present with the resizing of the s_group_info and s_flex_groups arrays which are addressed in the following patches which I have posted to the EXT4 mailing list:
[1/3] ext4: introduce macro sbi_array_rcu_deref() to access rcu protected fields [1]
[2/3] ext4: fix potential race between s_group_info online resizing and access
[2]
[3/3] ext4: fix potential race between s_flex_groups online resizing and access
[3]

[1] http://patchwork.ozlabs.org/patch/1240507/
[2] http://patchwork.ozlabs.org/patch/1240506/
[3] http://patchwork.ozlabs.org/patch/1240509/

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=206443

--- Comment #14 from Theodore Tso ([email protected]) ---
Patches to BZ don't have to be perfect, or mailing list ready. But it would be nice if they actually applied (e.g., not be white-space damaged) and if they actually compiled (not be missing macro definitions). :-)

In my experience, bugzilla is good for collecting data when we are trying to
root-cause a problem. But it's a lot more work to look at a bug in BZ, since
we have to download it first. Where as if it is sent to the mailing list,
it's a lot easier to review it and to send back comments.

For that matter, it's fine to send patches to the mailing list that aren't
ready to be applied. Using a [PATCH RFC] subject prefix is a good way to make
that clear; Linus Torvalds has been known to post patches with "Warning! I haven't even tried to compile it yet"; this is just to show the approach I'm
thinking of. What's important is to make sure expectations are set for why
the patch is being sent to the list or being uploaded to BZ.

Thanks for your work on this bug!

You are receiving this mail because:
You are watching the assignee of the bug.