Skip to contentSkip to conversation

linux-next/fs-next released on 6th September fails to boot

BUG REPORT Chandan Babu R 3 participants 17 messages Sep 9, 2024 – Sep 10, 2024

Subsystems (2)
lore
Share & tools
lore

Chandan Babu R

author Reviewed-by Christian Brauner
#0 lore raw

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
[   42.672406]  cuse_channel_open+0x540/0x710 [cuse]
[   42.672415]  misc_open+0x2a7/0x3a0
[   42.672424]  chrdev_open+0x1ef/0x5f0
[   42.672432]  ? __pfx_chrdev_open+0x10/0x10
[   42.672439]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672443]  ? security_file_open+0x3bb/0x720
[   42.672451]  do_dentry_open+0x43d/0x1200
[   42.672459]  ? __pfx_chrdev_open+0x10/0x10
[   42.672468]  vfs_open+0x79/0x340
[   42.672475]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672482]  do_open+0x68c/0x11e0
[   42.672489]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672495]  ? __pfx_do_open+0x10/0x10
[   42.672501]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672506]  ? open_last_lookups+0x2a2/0x1370
[   42.672515]  path_openat+0x24f/0x640
[   42.672522]  ? __pfx_path_openat+0x10/0x10
[   42.723972]  ? stack_depot_save_flags+0x45/0x4b0
[   42.724787]  ? __fput+0x43c/0xa70
[   42.725100]  do_filp_open+0x1b3/0x3e0
[   42.725710]  ? poison_slab_object+0x10d/0x190
[   42.726145]  ? __kasan_slab_free+0x33/0x50
[   42.726570]  ? __pfx_do_filp_open+0x10/0x10
[   42.726981]  ? do_syscall_64+0x64/0x170
[   42.727418]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   42.728018]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.728505]  ? do_raw_spin_lock+0x131/0x270
[   42.728922]  ? __pfx_do_raw_spin_lock+0x10/0x10
[   42.729494]  ? do_raw_spin_unlock+0x14c/0x1f0
[   42.729992]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.730889]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.732178]  ? alloc_fd+0x176/0x5e0
[   42.732585]  do_sys_openat2+0x122/0x160
[   42.732929]  ? __pfx_do_sys_openat2+0x10/0x10
[   42.733448]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.734013]  ? __pfx_map_id_up+0x10/0x10
[   42.734482]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.735529]  ? __memcg_slab_free_hook+0x292/0x500
[   42.736131]  __x64_sys_openat+0x123/0x1e0
[   42.736526]  ? __pfx___x64_sys_openat+0x10/0x10
[   42.737369]  ? __x64_sys_close+0x7c/0xd0
[   42.737717]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.738192]  ? syscall_trace_enter+0x11e/0x1b0
[   42.738739]  do_syscall_64+0x64/0x170
[   42.739113]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   42.739638] RIP: 0033:0x7f28cd13e87b
[   42.740038] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[   42.741943] RSP: 002b:00007ffc992546c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[   42.742951] RAX: ffffffffffffffda RBX: 00007f28cd44f1ee RCX: 00007f28cd13e87b
[   42.743660] RDX: 0000000000000002 RSI: 00007f28cd44f2fa RDI: 00000000ffffff9c
[   42.744518] RBP: 00007f28cd44f2fa R08: 0000000000000000 R09: 0000000000000001
[   42.745211] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[   42.745920] R13: 00007f28cd44f2fa R14: 0000000000000000 R15: 0000000000000003
[   42.746708]  </TASK>
[   42.746937] Modules linked in: cuse vfat fat ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common kvm_amd ccp bochs drm_vram_helper kvm drm_ttm_helper ttm pcspkr i2c_piix4 drm_kms_helper i2c_smbus pvpanic_mmio pvpanic joydev sch_fq_codel drm fuse xfs nvme_tcp nvme_fabrics nvme_core sd_mod sg virtio_net net_failover virtio_scsi failover crct10dif_pclmul crc32_pclmul ata_generic pata_acpi ata_piix ghash_clmulni_intel virtio_pci sha512_ssse3 virtio_pci_legacy_dev sha256_ssse3 virtio_pci_modern_dev sha1_ssse3 libata serio_raw dm_multipath btrfs blake2b_generic xor zstd_compress raid6_pq sunrpc dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi qemu_fw_cfg aesni_intel crypto_simd cryptd
[   42.754333] ---[ end trace 0000000000000000 ]---
[   42.756899] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.757851] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.760334] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.760940] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.761697] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.763009] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.763920] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.764839] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.765716] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.766890] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.767828] CR2: 00007f3237366208 CR3: 000000012c79e001 CR4: 0000000000770ef0
[   42.768730] PKRU: 55555554
[   42.769022] Kernel panic - not syncing: Fatal exception
[   42.770758] Kernel Offset: 0x7200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   42.771947] ---[ end Kernel panic - not syncing: Fatal exception ]---

However, the machine boots successfully when the following commit is reverted,

commit 9dc504f244895def513a0f2982c909103d4ab345

Author: Alexander Mikhalitsyn <[email protected]>
Date:   Tue Sep 3 17:16:26 2024 +0200

virtio_fs: allow idmapped mounts

Allow idmapped mounts for virtiofs.
It's absolutely safe as for virtiofs we have the same feature negotiation mechanism as for classical fuse filesystems. This does not affect any existing setups anyhow.

virtiofsd support:
https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/245

Signature · 1 line
Chandan
#1 lore raw

Am Mo., 9. Sept. 2024 um 12:25 Uhr schrieb Chandan Babu R <[email protected]>:

Quoted 2 lines

Hi,

Hi Chandan,

Quoted 111 lines

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
[   42.672406]  cuse_channel_open+0x540/0x710 [cuse]
[   42.672415]  misc_open+0x2a7/0x3a0
[   42.672424]  chrdev_open+0x1ef/0x5f0
[   42.672432]  ? __pfx_chrdev_open+0x10/0x10
[   42.672439]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672443]  ? security_file_open+0x3bb/0x720
[   42.672451]  do_dentry_open+0x43d/0x1200
[   42.672459]  ? __pfx_chrdev_open+0x10/0x10
[   42.672468]  vfs_open+0x79/0x340
[   42.672475]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672482]  do_open+0x68c/0x11e0
[   42.672489]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672495]  ? __pfx_do_open+0x10/0x10
[   42.672501]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672506]  ? open_last_lookups+0x2a2/0x1370
[   42.672515]  path_openat+0x24f/0x640
[   42.672522]  ? __pfx_path_openat+0x10/0x10
[   42.723972]  ? stack_depot_save_flags+0x45/0x4b0
[   42.724787]  ? __fput+0x43c/0xa70
[   42.725100]  do_filp_open+0x1b3/0x3e0
[   42.725710]  ? poison_slab_object+0x10d/0x190
[   42.726145]  ? __kasan_slab_free+0x33/0x50
[   42.726570]  ? __pfx_do_filp_open+0x10/0x10
[   42.726981]  ? do_syscall_64+0x64/0x170
[   42.727418]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   42.728018]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.728505]  ? do_raw_spin_lock+0x131/0x270
[   42.728922]  ? __pfx_do_raw_spin_lock+0x10/0x10
[   42.729494]  ? do_raw_spin_unlock+0x14c/0x1f0
[   42.729992]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.730889]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.732178]  ? alloc_fd+0x176/0x5e0
[   42.732585]  do_sys_openat2+0x122/0x160
[   42.732929]  ? __pfx_do_sys_openat2+0x10/0x10
[   42.733448]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.734013]  ? __pfx_map_id_up+0x10/0x10
[   42.734482]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.735529]  ? __memcg_slab_free_hook+0x292/0x500
[   42.736131]  __x64_sys_openat+0x123/0x1e0
[   42.736526]  ? __pfx___x64_sys_openat+0x10/0x10
[   42.737369]  ? __x64_sys_close+0x7c/0xd0
[   42.737717]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.738192]  ? syscall_trace_enter+0x11e/0x1b0
[   42.738739]  do_syscall_64+0x64/0x170
[   42.739113]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   42.739638] RIP: 0033:0x7f28cd13e87b
[   42.740038] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[   42.741943] RSP: 002b:00007ffc992546c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[   42.742951] RAX: ffffffffffffffda RBX: 00007f28cd44f1ee RCX: 00007f28cd13e87b
[   42.743660] RDX: 0000000000000002 RSI: 00007f28cd44f2fa RDI: 00000000ffffff9c
[   42.744518] RBP: 00007f28cd44f2fa R08: 0000000000000000 R09: 0000000000000001
[   42.745211] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[   42.745920] R13: 00007f28cd44f2fa R14: 0000000000000000 R15: 0000000000000003
[   42.746708]  </TASK>
[   42.746937] Modules linked in: cuse vfat fat ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common kvm_amd ccp bochs drm_vram_helper kvm drm_ttm_helper ttm pcspkr i2c_piix4 drm_kms_helper i2c_smbus pvpanic_mmio pvpanic joydev sch_fq_codel drm fuse xfs nvme_tcp nvme_fabrics nvme_core sd_mod sg virtio_net net_failover virtio_scsi failover crct10dif_pclmul crc32_pclmul ata_generic pata_acpi ata_piix ghash_clmulni_intel virtio_pci sha512_ssse3 virtio_pci_legacy_dev sha256_ssse3 virtio_pci_modern_dev sha1_ssse3 libata serio_raw dm_multipath btrfs blake2b_generic xor zstd_compress raid6_pq sunrpc dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi qemu_fw_cfg aesni_intel crypto_simd cryptd
[   42.754333] ---[ end trace 0000000000000000 ]---
[   42.756899] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.757851] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.760334] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.760940] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.761697] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.763009] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.763920] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.764839] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.765716] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.766890] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.767828] CR2: 00007f3237366208 CR3: 000000012c79e001 CR4: 0000000000770ef0
[   42.768730] PKRU: 55555554
[   42.769022] Kernel panic - not syncing: Fatal exception
[   42.770758] Kernel Offset: 0x7200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   42.771947] ---[ end Kernel panic - not syncing: Fatal exception ]---
Quoted 1 line

However, the machine boots successfully when the following commit is reverted,

Thanks for your report. I'll check what's going on.

Can you provide a bit more info about your setup? Which distro you use inside a VM, etc.

And it is weird that reverting ("virtio_fs: allow idmapped mounts") helps you as this commit only enables idmapped
mounts support for virtiofs if *daemon* supports it. But I'm pretty sure that you daemon does not support it
as this change was not yet merged to upstream virtiofsd
(https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/245).

Kind regards,
Alex

Quoted 22 lines

commit 9dc504f244895def513a0f2982c909103d4ab345

Author: Alexander Mikhalitsyn <[email protected]>
Date:   Tue Sep 3 17:16:26 2024 +0200

virtio_fs: allow idmapped mounts

Allow idmapped mounts for virtiofs.
It's absolutely safe as for virtiofs we have the same feature negotiation mechanism as for classical fuse filesystems. This does not affect any existing setups anyhow.

virtiofsd support:
https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/245

--
Chandan

#2 lore raw
In reply to Chandan Babu R
Quoted 32 lines · Chandan Babu R

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

That's why I'm insistent passing NULL is a problem. If I'm not mistaken this should be fixed by Alex's patchset to not pass NULL. I'll go review that now.

#3 lore raw

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

It must be something else. Maybe there is a mistake during merge? I'll check.

Quoted 4 lines

That's why I'm insistent passing NULL is a problem. If I'm not mistaken this should be fixed by Alex's patchset to not pass NULL. I'll go review that now.

Cool! Thanks, Christian!

#4 lore raw
Quoted 51 lines · Alexander Mikhalitsyn

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 49 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

Quoted 7 lines
Quoted 2 lines

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

#5 lore raw

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

#6 lore raw
Quoted 69 lines · Alexander Mikhalitsyn

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

#7 lore raw

Am Mo., 9. Sept. 2024 um 13:07 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 79 lines · Alexander Mikhalitsyn
Quoted 69 lines

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

Quoted 8 lines
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

Yes, we can sleep on this thing (and do a context switch), but won't leave the fuse_get_req()
function and nobody can free idmap before we exit from fuse_get_req() and all the functions upper the stack.

So, my point is that if we use an idmap pointer from a VFS callback argument and never preserve this pointer anywhere in
other data structures (but just pass it down the stack to helper functions) then we don't need to care about the idmap lifetime because it's controlled automatically. But if we start to deal with idmap in rcu callbacks, works, kthreads (like it was in cephfs) then yes. We do care.

Please, correct me if I'm saying something stupid :)

#8 lore raw
Quoted 103 lines · Alexander Mikhalitsyn

Am Mo., 9. Sept. 2024 um 13:07 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 79 lines · Alexander Mikhalitsyn
Quoted 69 lines

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

Quoted 8 lines
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

Yes, we can sleep on this thing (and do a context switch), but won't leave the fuse_get_req()
function and nobody can free idmap before we exit from fuse_get_req() and all the functions upper the stack.

So, my point is that if we use an idmap pointer from a VFS callback argument and never preserve this pointer anywhere in
other data structures (but just pass it down the stack to helper functions) then we don't need to care about the idmap lifetime because it's controlled automatically. But if we start to deal with idmap in rcu callbacks, works, kthreads (like it was in cephfs) then yes. We do care.

Quoted 1 line

Please, correct me if I'm saying something stupid :)

No, you're right. I'm task switching too much. It should also be irrelevant here since @idmap is NULL anyway in your current patch.

#9 lore raw

Am Mo., 9. Sept. 2024 um 14:32 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 105 lines · Alexander Mikhalitsyn
Quoted 103 lines

Am Mo., 9. Sept. 2024 um 13:07 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 79 lines · Alexander Mikhalitsyn
Quoted 69 lines

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

Quoted 8 lines
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

Yes, we can sleep on this thing (and do a context switch), but won't leave the fuse_get_req()
function and nobody can free idmap before we exit from fuse_get_req() and all the functions upper the stack.

So, my point is that if we use an idmap pointer from a VFS callback argument and never preserve this pointer anywhere in
other data structures (but just pass it down the stack to helper functions) then we don't need to care about the idmap lifetime because it's controlled automatically. But if we start to deal with idmap in rcu callbacks, works, kthreads (like it was in cephfs) then yes. We do care.

Quoted 3 lines
Quoted 1 line

Please, correct me if I'm saying something stupid :)

No, you're right. I'm task switching too much. It should also be

I can imagine ;-)

No worries about this one, I'll debug it and get back with results.

Quoted 1 line

irrelevant here since @idmap is NULL anyway in your current patch.

#10 lore raw

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

fs/fuse/dev.c +0 -0
index 7885f071fa1e..f130b23d6850 100644
@@ -148,7 +148,7 @@ static struct fuse_req *fuse_get_req(struct

mnt_idmap *idmap,

        if (for_background)
                __set_bit(FR_BACKGROUND, &req->flags);

- if ((fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {

+       if (!fm->sb || (fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
                kuid_t idmapped_fsuid;
                kgid_t idmapped_fsgid;

From your call stack if looks caused by CUSE case. It's my bad that I have not considered this use case for fuse_get_req() and there is, obviously,
no such thing as fm->sb for CUSE scenario.

I'm really sorry about that.

Kind regards,
Alex

Am Mo., 9. Sept. 2024 um 14:43 Uhr schrieb Alexander Mikhalitsyn <[email protected]>:

Quoted 117 lines

Am Mo., 9. Sept. 2024 um 14:32 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 105 lines · Alexander Mikhalitsyn
Quoted 103 lines

Am Mo., 9. Sept. 2024 um 13:07 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 79 lines · Alexander Mikhalitsyn
Quoted 69 lines

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

Quoted 8 lines
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

Yes, we can sleep on this thing (and do a context switch), but won't leave the fuse_get_req()
function and nobody can free idmap before we exit from fuse_get_req() and all the functions upper the stack.

So, my point is that if we use an idmap pointer from a VFS callback argument and never preserve this pointer anywhere in
other data structures (but just pass it down the stack to helper functions) then we don't need to care about the idmap lifetime because it's controlled automatically. But if we start to deal with idmap in rcu callbacks, works, kthreads (like it was in cephfs) then yes. We do care.

Quoted 3 lines
Quoted 1 line

Please, correct me if I'm saying something stupid :)

No, you're right. I'm task switching too much. It should also be

I can imagine ;-)

No worries about this one, I'll debug it and get back with results.

Quoted 1 line

irrelevant here since @idmap is NULL anyway in your current patch.

#11 lore raw

I'll send a patch with this fix a bit later, once
https://lore.kernel.org/linux-fsdevel/[email protected]/
is merged to prevent merge conflicts later on.

Am Mo., 9. Sept. 2024 um 17:56 Uhr schrieb Alexander Mikhalitsyn <[email protected]>:

Quoted 148 lines

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

fs/fuse/dev.c +1 -1
index 7885f071fa1e..f130b23d6850 100644
@@ -148,7 +148,7 @@ static struct fuse_req *fuse_get_req(struct
 mnt_idmap *idmap,
        if (for_background)
                __set_bit(FR_BACKGROUND, &req->flags);
 
-       if ((fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
+       if (!fm->sb || (fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
                kuid_t idmapped_fsuid;
                kgid_t idmapped_fsgid;

From your call stack if looks caused by CUSE case. It's my bad that I have not considered this use case for fuse_get_req() and there is, obviously,
no such thing as fm->sb for CUSE scenario.

I'm really sorry about that.

Kind regards,
Alex

Am Mo., 9. Sept. 2024 um 14:43 Uhr schrieb Alexander Mikhalitsyn <[email protected]>:

Quoted 117 lines

Am Mo., 9. Sept. 2024 um 14:32 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 105 lines · Alexander Mikhalitsyn
Quoted 103 lines

Am Mo., 9. Sept. 2024 um 13:07 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 79 lines · Alexander Mikhalitsyn
Quoted 69 lines

Am Mo., 9. Sept. 2024 um 13:00 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 59 lines · Alexander Mikhalitsyn
Quoted 57 lines

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

Quoted 5 lines

It is a background request, but we handle all idmappings stuff when we form fuse_header structure for the userspace.
So it is *before* all background stuff. Also, we never keep struct mnt_idmap pointers anywhere in fuse filesystem data structures. => no need to take references

Hm, ok but what about

Quoted 8 lines
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
                                !fuse_block_alloc(fc, for_background)))
                        goto out;
        }

?

Yes, we can sleep on this thing (and do a context switch), but won't leave the fuse_get_req()
function and nobody can free idmap before we exit from fuse_get_req() and all the functions upper the stack.

So, my point is that if we use an idmap pointer from a VFS callback argument and never preserve this pointer anywhere in
other data structures (but just pass it down the stack to helper functions) then we don't need to care about the idmap lifetime because it's controlled automatically. But if we start to deal with idmap in rcu callbacks, works, kthreads (like it was in cephfs) then yes. We do care.

Quoted 3 lines
Quoted 1 line

Please, correct me if I'm saying something stupid :)

No, you're right. I'm task switching too much. It should also be

I can imagine ;-)

No worries about this one, I'll debug it and get back with results.

Quoted 1 line

irrelevant here since @idmap is NULL anyway in your current patch.

#12 lore raw
Quoted 11 lines · Alexander Mikhalitsyn

I'll send a patch with this fix a bit later, once
https://lore.kernel.org/linux-fsdevel/[email protected]/
is merged to prevent merge conflicts later on.

Am Mo., 9. Sept. 2024 um 17:56 Uhr schrieb Alexander Mikhalitsyn <[email protected]>:

Quoted 5 lines

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

Hi Alexander,

I am sorry. I now see that the bug is present even after commit 9dc504f244895def513a0f2982c909103d4ab345 (virtio_fs: allow idmapped mounts) is reverted. I was using kexec to boot into new kernels during the bisect operation.

However, now when I do a normal kernel installation (i.e. make modules_install && make install) and reboot into the new kernel, I am noticing that the kernel fails to boot even with alleged bad commit reverted. I will write back with more details.

Apologies, for the inadvertent mistake.

Signature · 1 line
Chandan
#13 lore raw

Am Di., 10. Sept. 2024 um 08:45 Uhr schrieb Chandan Babu R <[email protected]>:

Quoted 9 lines · Alexander Mikhalitsyn
Quoted 7 lines

I'll send a patch with this fix a bit later, once
https://lore.kernel.org/linux-fsdevel/[email protected]/
is merged to prevent merge conflicts later on.

Am Mo., 9. Sept. 2024 um 17:56 Uhr schrieb Alexander Mikhalitsyn <[email protected]>:

Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

Hi Alexander,

Hi Chandan,

Quoted 5 lines

I am sorry. I now see that the bug is present even after commit 9dc504f244895def513a0f2982c909103d4ab345 (virtio_fs: allow idmapped mounts) is reverted. I was using kexec to boot into new kernels during the bisect operation.

It's not a surprise, that revert of 9dc504f244895def513a0f2982c909103d4ab345 (virtio_fs: allow idmapped mounts)
itself does not help, because really this commit does nothing and can take effect only if the userspace side
activates idmapping support.

At the same time, from the call stack that you provided I can see that crash happens on:

42.672223] <TASK>

[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0
x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
<CUT>
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
[   42.672406]  cuse_channel_open+0x540/0x710 [cuse]
[   42.672415]  misc_open+0x2a7/0x3a0
[   42.672424]  chrdev_open+0x1ef/0x5f0
<CUT>
[   42.726570]  ? __pfx_do_filp_open+0x10/0x10
[   42.726981]  ? do_syscall_64+0x64/0x170

which corresponds to CUSE.

I would suggest applying that patch I sent yesterday and try it out.

Quoted 7 lines

However, now when I do a normal kernel installation (i.e. make modules_install && make install) and reboot into the new kernel, I am noticing that the kernel fails to boot even with alleged bad commit reverted. I will write back with more details.

Apologies, for the inadvertent mistake.

Kind regards,
Alex

Quoted 3 lines

--
Chandan

#14 lore raw
In reply to Alexander Mikhalitsyn
Quoted 18 lines · Alexander Mikhalitsyn

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

fs/fuse/dev.c +1 -1
index 7885f071fa1e..f130b23d6850 100644
@@ -148,7 +148,7 @@ static struct fuse_req *fuse_get_req(struct
 mnt_idmap *idmap,
        if (for_background)
                __set_bit(FR_BACKGROUND, &req->flags);
 
-       if ((fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
+       if (!fm->sb || (fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
                kuid_t idmapped_fsuid;
                kgid_t idmapped_fsgid;
Quoted 5 lines

From your call stack if looks caused by CUSE case. It's my bad that I have not considered this use case for fuse_get_req() and there is, obviously,
no such thing as fm->sb for CUSE scenario.

The above patch solves the problem. Thank you!

Signature · 1 line
Chandan
#15 lore raw

Am Di., 10. Sept. 2024 um 09:51 Uhr schrieb Chandan Babu R <[email protected]>:

Quoted 20 lines · Alexander Mikhalitsyn
Quoted 18 lines

Dear Chandan,

Please can you check if the following patch resolves issue for your workload:

fs/fuse/dev.c +1 -1
index 7885f071fa1e..f130b23d6850 100644
@@ -148,7 +148,7 @@ static struct fuse_req *fuse_get_req(struct
 mnt_idmap *idmap,
        if (for_background)
                __set_bit(FR_BACKGROUND, &req->flags);
 
-       if ((fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
+       if (!fm->sb || (fm->sb->s_iflags & SB_I_NOIDMAP) || idmap) {
                kuid_t idmapped_fsuid;
                kgid_t idmapped_fsgid;
Quoted 7 lines
Quoted 5 lines

From your call stack if looks caused by CUSE case. It's my bad that I have not considered this use case for fuse_get_req() and there is, obviously,
no such thing as fm->sb for CUSE scenario.

The above patch solves the problem. Thank you!

Awesome! Thanks, Chandan!

I'll send a patch to LKML a bit later once Miklos take
https://lore.kernel.org/linux-fsdevel/[email protected]
series.

Kind regards,
Alex

Quoted 3 lines

--
Chandan

#16 lore raw
In reply to Christian Brauner
Quoted 58 lines · Christian Brauner
Quoted 57 lines · Alexander Mikhalitsyn

Am Mo., 9. Sept. 2024 um 12:40 Uhr schrieb Christian Brauner <[email protected]>:

Quoted 43 lines · Chandan Babu R
Quoted 32 lines

Hi,

linux-next/fs-next released on 6th September is failing to boot on a x86 guest,

[   42.659136] Oops: general protection fault, probably for non-canonical address 0xdffffc000000000b: 0000 [#1] PREEMPT SMP KASAN NOPTI
[   42.660501] fbcon: Taking over console
[   42.660930] KASAN: null-ptr-deref in range [0x0000000000000058-0x000000000000005f]
[   42.661752] CPU: 1 UID: 0 PID: 1589 Comm: dtprobed Not tainted 6.11.0-rc6+ #1
[   42.662565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023
[   42.663472] RIP: 0010:fuse_get_req+0x36b/0x990 [fuse]
[   42.664046] Code: 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 8c 05 00 00 48 b8 00 00 00 00 00 fc ff df 48 8b 6d 08 48 8d 7d 58 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 4d 05 00 00 f6 45 59 20 0f 85 06 03 00 00 48 83
[   42.666945] RSP: 0018:ffffc900009a7730 EFLAGS: 00010212
[   42.668837] RAX: dffffc0000000000 RBX: 1ffff92000134eed RCX: ffffffffc20dec9a
[   42.670122] RDX: 000000000000000b RSI: 0000000000000008 RDI: 0000000000000058
[   42.672154] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1022110172
[   42.672160] R10: ffff888110880b97 R11: ffffc900009a737a R12: 0000000000000001
[   42.672179] R13: ffff888110880b60 R14: ffff888110880b90 R15: ffff888169973840
[   42.672186] FS:  00007f28cd21d7c0(0000) GS:ffff8883ef280000(0000) knlGS:0000000000000000
[   42.672191] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   42.[ CR02: ;32m00007f3237366208 CR3: 0  OK  79e001 CR4: 0000000000770ef0
[   42.672214] PKRU: 55555554
[   42.672218] Call Trace:
[   42.672223]  <TASK>
[   42.672226]  ? die_addr+0x41/0xa0
[   42.672238]  ? exc_general_protection+0x14c/0x230
[   42.672250]  ? asm_exc_general_protection+0x26/0x30
[   42.672260]  ? fuse_get_req+0x77a/0x990 [fuse]
[   42.672281]  ? fuse_get_req+0x36b/0x990 [fuse]
[   42.672300]  ? kasan_unpoison+0x27/0x60
[   42.672310]  ? __pfx_fuse_get_req+0x10/0x10 [fuse]
[   42.672327]  ? srso_alias_return_thunk+0x5/0xfbef5
Quoted 8 lines
[   42.672333]  ? alloc_pages_mpol_noprof+0x195/0x440
[   42.672340]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672345]  ? kasan_unpoison+0x27/0x60
[   42.672350]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672355]  ? __kasan_slab_alloc+0x4d/0x90
[   42.672362]  ? srso_alias_return_thunk+0x5/0xfbef5
[   42.672367]  ? __kmalloc_cache_noprof+0x134/0x350
[   42.672376]  fuse_simple_background+0xe7/0x180 [fuse]
Quoted 8 lines

I think this is basically:

fuse_simple_background()
-> !args->force
-> fuse_get_req(NULL, fm, true);

and there you have fm->sb->s_iflags & SB_I_NOIDMAP with idmap == NULL afaict.

Yeah, but fuse_get_req() is ready for idmap == NULL case:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/fs/fuse/dev.c?h=fs-next#n111

Quoted 7 lines
Quoted 1 line

It must be something else. Maybe there is a mistake during merge? I'll check.

Oh yeah, I'm blind.

Well, this is a background request? In that case can't the idmap go away in the meantime and become freed? If so, then you need mnt_idmap_get() and mnt_idmap_put().

So sm like:

fs/fuse/dev.c +6 -0
index 7885f071fa1e..a7bf7f6d17e6 100644
@@ -117,6 +117,10 @@ static struct fuse_req *fuse_get_req(struct mnt_idmap *idmap,
        int err;
        atomic_inc(&fc->num_waiting);
 
+       /* Ensure that @idmap can't just go away. */
+       if (for_background)
+               mnt_idmap_get(idmap);
+
        if (fuse_block_alloc(fc, for_background)) {
                err = -EINTR;
                if (wait_event_killable_exclusive(fc->blocked_waitq,
@@ -166,6 +170,8 @@ static struct fuse_req *fuse_get_req(struct mnt_idmap *idmap,
                if (unlikely(req->in.h.uid == ((uid_t)-1) ||
                             req->in.h.gid == ((gid_t)-1))) {
                        fuse_put_request(req);
+                       if (for_background)
+                               mnt_idmap_put(idmap);
                        return ERR_PTR(-EOVERFLOW);
                }
        } else {