locking/urgent for v6.17-rc1
Subsystems (8)
Hi Linus,
please pull a locking/urgent fix for v6.17-rc1.
Thx.
---
The following changes since commit 98e8f2c0e0930feee6a2538450c74d9d7de0a9cc:
Merge tag 'x86-platform-2025-07-29' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip (2025-07-29 20:05:06 -0700)
are available in the Git repository at:
ssh://[email protected]/pub/scm/linux/kernel/git/tip/tip tags/locking_urgent_for_v6.17_rc1
for you to fetch changes up to e703b7e247503b8bf87b62c02a4392749b09eca8:
futex: Move futex cleanup to __mmdrop() (2025-08-02 15:11:52 +0200)
----------------------------------------------------------------
- Prevent a futex hash leak due to different mm lifetimes
----------------------------------------------------------------
Thomas Gleixner (1):
futex: Move futex cleanup to __mmdrop()
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteThe pull request you sent on Sat, 9 Aug 2025 20:02:07 +0200:
The pull request you sent on Sat, 9 Aug 2025 20:02:07 +0200:
Quoted 1 line
ssh://[email protected]/pub/scm/linux/kernel/git/tip/tip tags/locking_urgent_for_v6.17_rc1
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/8e8f6b635fae254252f7f52dd3e79fb68d06c332
Thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html
Quoted 2 lines · Borislav Petkov
please pull a locking/urgent fix for v6.17-rc1.
Ok, so this clearly wasn't a fix.
Quoted 2 lines
Thomas Gleixner (1):
futex: Move futex cleanup to__mmdrop()
So this causes problems, because __mmdrop is not done in thread context, and the kvfree() calls then cause issues:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Hilf Danton sent out a patch, but honestly, that patch looks like pure bandaid, and will make the exit path horribly much slower by moving things into workqueues. It might not be visible in profiles exactly *because* it's then hidden in workqueues, but it's not great.
I think it's a mistake to allow vmalloc'ing those hashes in the first place, and I suggest the local hash be size-limited to the point where it's just a kmalloc() and thus works in all contexts.
Or maybe the mistake was the mm-private hashing in the first place. Maybe that hash shouldn't be allocated at mm_alloc() -> futex_mm_init() at all. Only initialized by the futex code when needed, and then dropped in exit_mmap().
So the problems seem deeper than just "free'd in the wrong context".
Linus
Quoted 24 lines · Linus Torvalds
Quoted 2 lines · Borislav Petkov
please pull a locking/urgent fix for v6.17-rc1.
Ok, so this clearly wasn't a fix.
Quoted 2 lines
Thomas Gleixner (1):
futex: Move futex cleanup to__mmdrop()So this causes problems, because
__mmdropis not done in thread context, and thekvfree()calls then cause issues:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/Hilf Danton sent out a patch, but honestly, that patch looks like pure bandaid, and will make the exit path horribly much slower by moving things into workqueues. It might not be visible in profiles exactly *because* it's then hidden in workqueues, but it's not great.
I think it's a mistake to allow vmalloc'ing those hashes in the first place, and I suggest the local hash be size-limited to the point where it's just a
kmalloc()and thus works in all contexts.
Quoted 6 lines
Or maybe the mistake was the mm-private hashing in the first place. Maybe that hash shouldn't be allocated at
mm_alloc()->futex_mm_init()at all. Only initialized by the futex code when needed, and then dropped inexit_mmap().So the problems seem deeper than just "free'd in the wrong context".
Piggybacking the futex private hashing attention, the new fanciness is causing crashes in my testing. The crashes are 100% reproducible, but my reproducer is simply running a variety of tests in parallel, i.e. isn't very debug-friendly, and the code itself is black magic to me, so all I've done is bisect.
I reported the issue on the original thread, but haven't seen any follow-up.
Sebastian Andrzej Siewior
maintainerQuoted 8 lines · Sean Christopherson
Piggybacking the futex private hashing attention, the new fanciness is causing crashes in my testing. The crashes are 100% reproducible, but my reproducer is simply running a variety of tests in parallel, i.e. isn't very debug-friendly, and the code itself is black magic to me, so all I've done is bisect.
I reported the issue on the original thread, but haven't seen any follow-up.
I somehow missed it. Can you try rc2 with the patch I just sent?
Sebastian
Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 8 lines · Sean Christopherson
Piggybacking the futex private hashing attention, the new fanciness is causing crashes in my testing. The crashes are 100% reproducible, but my reproducer is simply running a variety of tests in parallel, i.e. isn't very debug-friendly, and the code itself is black magic to me, so all I've done is bisect.
I reported the issue on the original thread, but haven't seen any follow-up.
I somehow missed it. Can you try rc2 with the patch I just sent?
No dice, fails with the same signature.
I got a trimmed down reproduer. Load KVM, run this in the background (in a loop) to constantly trigger try_to_wake_up() on relevant tasks (needs to be run as root):
echo Y > /sys/module/kvm/parameters/nx_huge_pages echo N > /sys/module/kvm/parameters/nx_huge_pages sleep .2
and then run the hardware_disable_test KVM selftest (from tools/testing/selftests/kvm/hardware_disable_test.c).
Strace on hardware_disable_test spewed a whole pile of these
wait4(32861, 0x7ffc66475dec, WNOHANG, NULL) = 0
futex(0x7fb735c43000, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
immediately before the crash. I assume it corresponds to this:
/* Child is still running, keep waiting. */ if (pid != waitpid(pid, &status, WNOHANG)) continue;
I also got a new splat on the "WARN_ON_ONCE(ret < 0);" at the end of __futex_ref_atomic_end(). This happened during boot; AFAICT our userspace was setting up cgroups. In this case, the system hung and I had to reboot.
------------[ cut here ]------------ WARNING: CPU: 45 PID: 0 at kernel/futex/core.c:1604 futex_ref_rcu+0xbf/0xf0 Modules linked in: vfat fat i2c_mux_pca954x i2c_mux spidev cdc_acm xhci_pci xhci_hcd gq(O) sha3_generic CPU: 45 UID: 0 PID: 0 Comm: swapper/45 Tainted: G S O 6.17.0-smp--1278d576b27d-futex #886 NONE Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE Hardware name: Google LLC Indus/Indus_QC_03, BIOS 30.110.0 09/13/2024 RIP: 0010:futex_ref_rcu+0xbf/0xf0 Code: c7 04 0a 00 00 00 00 48 ff c0 eb c2 65 ff 01 89 e8 4c 01 f0 48 ff c0 48 89 c1 f0 48 0f c1 8b 48 01 00 00 48 01 c1 74 06 79 0c <0f> 0b eb 08 48 89 df e8 55 0a f9 ff 48 89 df 5b 41 5e 5d e9 f9 5c RSP: 0018:ffffa43c8d440ec8 EFLAGS: 00010286 RAX: 8000000000000000 RBX: ffff933782245080 RCX: ffffffffffffffff RDX: 0000000000000060 RSI: 0000000000000060 RDI: ffffffffac840520 RBP: 0000000000000000 R08: ffff933680044d00 R09: ffffffff00000000 R10: ffff9365c9b59e00 R11: ffff9365c9b59e00 R12: ffffffffab77ac10 R13: ffff9337822451b8 R14: 7fffffffffffffff R15: ffff9365c749de00 FS: 0000000000000000(0000) GS:ffff9395514f2000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fad8a21cf38 CR3: 00000055c062b002 CR4: 00000000007706f0 PKRU: 55555554 Call Trace: <IRQ> rcu_do_batch+0x250/0x7e0 rcu_core+0x12f/0x230 handle_softirqs+0xc8/0x280 __irq_exit_rcu+0x48/0x100 sysvec_apic_timer_interrupt+0x74/0x80 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:cpuidle_enter_state+0xfb/0x290 Code: bb f6 ff ff 49 89 c4 8b 73 04 bf ff ff ff ff e8 9b 68 d8 ff 31 ff e8 f4 32 48 ff 80 7c 24 04 00 74 05 e8 c8 68 d8 ff fb 85 ed <0f> 88 ba 00 00 00 89 e9 48 6b f9 68 4c 8b 44 24 08 49 8b 54 38 30 RSP: 0018:ffffa43c803d3e80 EFLAGS: 00000206 RAX: ffff9395514f2000 RBX: ffff9394ff776548 RCX: 000000000000001f RDX: 000000000018ec50 RSI: 000000000000002d RDI: 0000000000000000 RBP: 0000000000000003 R08: 0000000000000002 R09: 0000000000000002 R10: 00000000000003dc R11: 0000000000000389 R12: 00000010fb32644d R13: 00000010fb2333f7 R14: ffffffffad276f68 R15: 0000000000000003 cpuidle_enter+0x2c/0x40 do_idle+0x1ac/0x250 cpu_startup_entry+0x2a/0x30 start_secondary+0x80/0x80 common_startup_64+0x13e/0x140 </TASK> ---[ end trace 0000000000000000 ]---
Heh, and two more when booting a different system. Guess it's my lucky day. This time whatever went sideways didn't appear to be fatal as the system booted and I could ssh in. One is the same WARN as above, and the second WARN on the system hit the
WARN_ON_ONCE(atomic_long_read(&mm->futex_atomic) != 0);
in futex_hash_allocate().
------------[ cut here ]------------ WARNING: CPU: 120 PID: 11779 at kernel/futex/core.c:1553 futex_hash_allocate+0x436/0x450 Modules linked in: vfat fat ccp k10temp i2c_piix4 cdc_acm xhci_pci xhci_hcd gq(O) sha3_generic CPU: 120 UID: 0 PID: 11779 Comm: borglet Tainted: G U W O 6.17.0-smp--1278d576b27d-futex #886 NONE Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.64.2-0 12/26/2024 RIP: 0010:futex_hash_allocate+0x436/0x450 Code: 31 ff 65 48 8b 05 ba bc ae 02 48 3b 44 24 48 75 20 44 89 f8 48 83 c4 50 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b e9 9d fe ff ff <0f> 0b e9 c0 fe ff ff e8 ce 99 af 00 66 66 66 66 66 2e 0f 1f 84 00 RSP: 0018:ffffbbc0f1237d10 EFLAGS: 00010286 RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffa25747532180 RDX: 0000000000000400 RSI: 000000000000ffc0 RDI: 00000000000039b8 RBP: ffffa296a2620000 R08: 00000000004029c0 R09: 00000000ffffffff R10: 00000000ffffffff R11: 0000000000010040 R12: ffffa2571336b700 R13: ffffa2571336b600 R14: ffffa2571336b600 R15: ffffa296b9270000 FS: 00007f6bbd3297c0(0000) GS:ffffa2d5a31b2000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f6bae810f38 CR3: 00000001330a4001 CR4: 0000000000770ef0 PKRU: 55555554 Call Trace: <TASK> ? cgroup_can_fork+0x258/0x420 copy_process+0xae3/0xff0 kernel_clone+0x99/0x320 __x64_sys_clone+0xc8/0xf0 do_syscall_64+0x6f/0x1f0 ? arch_exit_to_user_mode_prepare+0x9/0x50 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7f6bbd466051 Code: 48 85 ff 74 3d 48 85 f6 74 38 48 83 ee 10 48 89 4e 08 48 89 3e 48 89 d7 4c 89 c2 4d 89 c8 4c 8b 54 24 08 b8 38 00 00 00 0f 05 <48> 85 c0 7c 13 74 01 c3 31 ed 58 5f ff d0 48 89 c7 b8 3c 00 00 00 RSP: 002b:00007fffff2eda98 EFLAGS: 00000206 ORIG_RAX: 0000000000000038 RAX: ffffffffffffffda RBX: 00007f6bae812700 RCX: 00007f6bbd466051 RDX: 00007f6bae8129d0 RSI: 00007f6bae810f30 RDI: 00000000003d0f00 RBP: 00007fffff2edad0 R08: 00007f6bae812700 R09: 00007f6bae812700 R10: 00007f6bae8129d0 R11: 0000000000000206 R12: 00007f6bae8129d0 R13: 00007fffff2edb66 R14: 00007fffff2edbd0 R15: 00007f6bae810f40 </TASK> ---[ end trace 0000000000000000 ]---
Sebastian Andrzej Siewior
maintainerQuoted 9 lines · Sean Christopherson
Quoted 3 lines
Quoted 1 line
I somehow missed it. Can you try rc2 with the patch I just sent?
No dice, fails with the same signature.
I got a trimmed down reproduer. Load KVM, run this in the background (in a loop) to constantly trigger
try_to_wake_up()on relevant tasks (needs to be run as root):
Quoted 6 lines
echo Y >
/sys/module/kvm/parameters/nx_huge_pagesecho N >/sys/module/kvm/parameters/nx_huge_pagessleep .2and then run the
hardware_disable_testKVM selftest (fromtools/testing/selftests/kvm/hardware_disable_test.c).
With this information I was able to reproduce what you had in the link a the top. I don't know why it happens. It hangs and lockdep isn't happy with the lock - it seems to be a valid task_struct::pi_lock lock for one of the kvm-nx-lpage-recovery threads.
I got rid of all free_percpu() and kvfree() in futex/core.c (and leak memory, yes) and this still happens.
I was able to avoid the crash if I skip the assignment of the second private hash but it turned out that I was not patient enough.
The strange part here is that the private hash is not used. The private hash gets allocated and resized because hardware_disable_test creates a lot of threads. But it is not used, it just sits around and waits to be cleared.
And it also seems to happen if I tell futex_hash_allocate_default() not to do anything at all.
kvm-nx-lpage-recovery shares the mm but it grabs a reference. It might be a coincidence but the task, on which the wakeup chokes, seems to be gone according to my traces. And with
kernel/vhost_task.c +4 -1
@@ -75,7 +84,10 @@ static int vhost_task_fn(void *data)*/ void vhost_task_wake (struct vhost_task *vtsk) { -wake_up_process (vtsk->task); +mutex_lock (&vtsk->exit_mutex); +if (!test_bit (VHOST_TASK_FLAGS_KILLED , &vtsk->flags)) +wake_up_process (vtsk->task); +mutex_unlock (&vtsk->exit_mutex); }EXPORT_SYMBOL_GPL (vhost_task_wake);
it doesn't crash anymore. Could it attempts to wake a task that is gone?
Quoted 4 lines
Strace on
hardware_disable_testspewed a whole pile of these
wait4(32861, 0x7ffc66475dec, WNOHANG, NULL)= 0
futex(0x7fb735c43000, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
That is a shared FUTEX and is probably part pthread_join().
Quoted 9 lines
immediately before the crash. I assume it corresponds to this:
/* Child is still running, keep waiting. */ if (pid != waitpid(pid, &status, WNOHANG)) continue;I also got a new splat on the "
WARN_ON_ONCE(ret < 0);" at the end of__futex_ref_atomic_end(). This happened during boot; AFAICT our userspace was setting up cgroups. In this case, the system hung and I had to reboot.
This is odd
Quoted 2 lines
------------[ cut here ]------------ WARNING: CPU: 45 PID: 0 at kernel/futex/core.c:1604 futex_ref_rcu+0xbf/0xf0
…
Quoted 8 lines
Heh, and two more when booting a different system. Guess it's my lucky day. This time whatever went sideways didn't appear to be fatal as the system booted and I could ssh in. One is the same WARN as above, and the second WARN on the system hit the
WARN_ON_ONCE(atomic_long_read(&mm->futex_atomic) != 0);in
futex_hash_allocate().
This means the counter don't add up after the switch. Not sure how. This seems to be a random task but it might be part of the previous splat.
Sebastian
Quoted 20 lines · Sebastian Andrzej Siewior
On 2025-08-22 17:28:02 [-0700], Sean Christopherson wrote:
kvm-nx-lpage-recovery shares the mm but it grabs a reference. It might be a coincidence but the task, on which the wakeup chokes, seems to be gone according to my traces. And withkernel/vhost_task.c +4
-1@@ -75,7 +84,10 @@ static int vhost_task_fn(void *data)*/ void vhost_task_wake (struct vhost_task *vtsk) { -wake_up_process (vtsk->task); +mutex_lock (&vtsk->exit_mutex); +if (!test_bit (VHOST_TASK_FLAGS_KILLED , &vtsk->flags)) +wake_up_process (vtsk->task); +mutex_unlock (&vtsk->exit_mutex); }EXPORT_SYMBOL_GPL (vhost_task_wake);
Quoted 1 line
it doesn't crash anymore. Could it attempts to wake a task that is gone?
Oh fudge, that indeed is what's happening.
Each VM that KVM creates has a kvm-nx-lpage-recovery task, and KVM wakes all such tasks across all VMs in response to any change to the hugepage recovery settings, i.e. when privileged userspace changes any of the associate module params.
KVM holds a global lock when walking the list of VMs and so guarantees the VM hasn't fully exited, but nothing prevents the recovery task from getting a signal and exiting long before the VM is destroyed. hardware_disable_test is (deliberately?) not very tidy, and exits without explicitly closing the VM and vCPU fds, and so its recovery task gets terminated via signal instead of by KVM explicitly calling vhost_task_stop() when the VM is being destroyed.
The basic gist of the above diff works, but unfortunately simply taking vtsk->exit_mutex in vhost_task_wake() doesn't appear to be an option because the vhost code appears to have gone through a lot of effort to avoid waking an exited task.
I think we can also add some sanity checks and hints to help future users of the vhost task code from running into the same problem.
I'll post a proper series.
Thanks a ton, I owe you a drink of your choice :-)
Quoted 27 lines
Quoted 4 lines
Strace on
hardware_disable_testspewed a whole pile of these
wait4(32861, 0x7ffc66475dec, WNOHANG, NULL)= 0
futex(0x7fb735c43000, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)That is a shared FUTEX and is probably part
pthread_join().Quoted 9 lines
immediately before the crash. I assume it corresponds to this:
/* Child is still running, keep waiting. */ if (pid != waitpid(pid, &status, WNOHANG)) continue;I also got a new splat on the "
WARN_ON_ONCE(ret < 0);" at the end of__futex_ref_atomic_end(). This happened during boot; AFAICT our userspace was setting up cgroups. In this case, the system hung and I had to reboot.This is odd
Quoted 2 lines
------------[ cut here ]------------ WARNING: CPU: 45 PID: 0 at kernel/futex/core.c:1604 futex_ref_rcu+0xbf/0xf0…
Quoted 5 lines
Heh, and two more when booting a different system. Guess it's my lucky day. This time whatever went sideways didn't appear to be fatal as the system booted and I could ssh in. One is the same WARN as above, and the second WARN on the system hit the
Quoted 6 lines
Quoted 3 lines
WARN_ON_ONCE(atomic_long_read(&mm->futex_atomic) != 0);in
futex_hash_allocate().This means the counter don't add up after the switch. Not sure how. This seems to be a random task but it might be part of the previous splat.
Yeah, IIRC, those only showed up when I kexec'd into a new kernel instead of doing a normal reboot, so it may have been some weird leftovers and/or PEBKAC? I'll file a new bug report if I see either of those warnings again.
(Re)boot hangs and refcount errors on next-20250829
I think I hit the the same error (and some more ...) when booting next-20250829 on my amd64 laptop (no VMs and no kexec were used here):
When I try booting next-20250829 on my debian stable(trixie) amd64 system the boot process hangs every few attempts (without any log messages recorded). When the boot process succeeds the following error message appears in dmesg:
[ 8.337248] [ T44] ------------[ cut here ]------------
[ 8.337250] [ T44] WARNING: kernel/futex/core.c:1604 at futex_ref_rcu+0xe8/0x100, CPU#6: rcuc/6/44
[ 8.337257] [ T44] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel btusb snd_intel_dspcfg snd_soc_dmic snd_acp3x_pdm_dma snd_acp3x_rn snd_hda_codec btrtl uvcvideo snd_soc_core btintel snd_hda_core btbcm videobuf2_vmalloc btmtk videobuf2_memops snd_hwdep uvc videobuf2_v4l2 snd_pcm_oss bluetooth videodev snd_mixer_oss snd_pcm snd_rn_pci_acp3x snd_acp_config videobuf2_common snd_soc_acpi msi_wmi ecdh_generic ecc mc sparse_keymap snd_timer edac_mce_amd wmi_bmof snd snd_pci_acp3x k10temp ccp soundcore battery ac button joydev hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_prox hid_sensor_als hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf industrialio amd_pmc hid_sensor_iio_common evdev mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 mac80211 libarc4 cfg80211 rfkill msr fuse nvme_fabrics efi_pstore configfs
[ 8.337310] [ T44] efivarfs autofs4 ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm drm_panel_backlight_quirks drm_exec drm_suballoc_helper drm_buddy gpu_sched xhci_pci amdxcp xhci_hcd hid_sensor_hub psmouse drm_display_helper mfd_core hid_multitouch hid_generic drm_kms_helper i2c_hid_acpi serio_raw usbcore amd_sfh i2c_hid r8169 cec nvme hid crc16 nvme_core i2c_piix4 usb_common i2c_smbus i2c_designware_platform i2c_designware_core
[ 8.337337] [ T44] CPU: 6 UID: 0 PID: 44 Comm: rcuc/6 Not tainted 6.17.0-rc3-next-20250829-master #215 PREEMPT_{RT,(full)}
[ 8.337340] [ T44] Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
[ 8.337342] [ T44] RIP: 0010:futex_ref_rcu+0xe8/0x100
[ 8.337344] [ T44] Code: be ff ff ff ff ff ff ff 7f 48 89 b3 88 01 00 00 c7 00 01 00 00 00 48 8d bb 78 01 00 00 5b 48 c7 c6 70 d7 b4 a5 e9 78 7a fd ff <0f> 0b eb ab 0f 0b e9 31 ff ff ff 0f 0b eb c6 66 0f 1f 84 00 00 00
[ 8.337346] [ T44] RSP: 0018:ffffb438402dfdb0 EFLAGS: 00010286
[ 8.337348] [ T44] RAX: 7fffffffffffffff RBX: ffff8aab87fa1900 RCX: 0000000000000010
[ 8.337350] [ T44] RDX: ffffffffffffffff RSI: ffffffffffffffff RDI: 000000000000000f
[ 8.337351] [ T44] RBP: ffff8aab80958000 R08: 0000000000000000 R09: 0000000000000006
[ 8.337352] [ T44] R10: 0000000000220021 R11: 0000000000000000 R12: ffff8aae5e7a2300
[ 8.337353] [ T44] R13: ffffb438402dfdf0 R14: 0000000000000016 R15: 0000000000000000
[ 8.337354] [ T44] FS: 0000000000000000(0000) GS:ffff8aaeb79bc000(0000) knlGS:0000000000000000
[ 8.337356] [ T44] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8.337357] [ T44] CR2: 00007fbce40135f8 CR3: 0000000108b48000 CR4: 0000000000750ef0
[ 8.337358] [ T44] PKRU: 55555554
[ 8.337359] [ T44] Call Trace:
[ 8.337361] [ T44] <TASK>
[ 8.337363] [ T44] rcu_do_batch+0x1b6/0x540
[ 8.337366] [ T44] ? rcu_do_batch+0x156/0x540
[ 8.337369] [ T44] rcu_core+0x141/0x250
[ 8.337371] [ T44] ? rcu_cpu_kthread+0x22/0xe0
[ 8.337373] [ T44] rcu_cpu_kthread+0x85/0xe0
[ 8.337375] [ T44] ? sort_range+0x20/0x20
[ 8.337378] [ T44] smpboot_thread_fn+0xd8/0x1d0
[ 8.337381] [ T44] kthread+0xe9/0x1e0
[ 8.337383] [ T44] ? kthreads_online_cpu+0x100/0x100
[ 8.337385] [ T44] ? kthreads_online_cpu+0x100/0x100
[ 8.337387] [ T44] ret_from_fork+0x18e/0x1c0
[ 8.337391] [ T44] ? kthreads_online_cpu+0x100/0x100
[ 8.337392] [ T44] ret_from_fork_asm+0x11/0x20
[ 8.337398] [ T44] </TASK>
[ 8.337398] [ T44] ---[ end trace 0000000000000000 ]---Rebooting (normal reboot no kexec) also hangs sometimes and using netconsole I could capture the following error message:
T124;------------[ cut here ]------------
T124;percpu ref (css_release) <= 0 (-1) after switching to atomic
T124;WARNING: lib/percpu-refcount.c:197 at percpu_ref_switch_to_atomic_rcu+0x194/0x1a0, CPU#11: rcuc/11/124
T124,ncfrag=0/1022;Modules linked in: netconsole ccm snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_acp3x_pdm_dma snd_soc_dmic snd_acp3x_rn btusb snd_hda_codec btrtl uvcvideo snd_soc_core btintel snd_hda_core btbcm videobuf2_vmalloc btmtk snd_hwdep videobuf2_memops snd_pcm_oss uvc videobuf2_v4l2 bluetooth snd_mixer_oss videodev snd_pcm snd_rn_pci_acp3x snd_acp_config videobuf2_common snd_timer msi_wmi snd_soc_acpi ecdh_generic ecc sparse_keymap mc snd wmi_bmof edac_mce_amd soundcore k10temp snd_pci_acp3x ccp ac battery button joydev hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_magn_3d hid_sensor_als hid_sensor_prox hid_sensor_trigger industrialio_triggered_buffer kfifo_buf industrialio amd_pmc hid_sensor_iio_common evdev mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 T124,ncfrag=956/1022;mac80211 libarc4 cfg80211 rfkill msr fuse nvme_fabrics efi_pstore
T124; configfs efivarfs autofs4 ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm drm_panel_backlight_quirks drm_exec drm_suballoc_helper drm_buddy gpu_sched xhci_pci amdxcp drm_display_helper xhci_hcd hid_sensor_hub hid_multitouch mfd_core hid_generic usbcore i2c_hid_acpi psmouse amd_sfh drm_kms_helper nvme i2c_hid hid serio_raw nvme_core cec r8169 i2c_piix4 usb_common crc16 i2c_smbus i2c_designware_platform i2c_designware_core
T124;CPU: 11 UID: 0 PID: 124 Comm: rcuc/11 Tainted: G W 6.17.0-rc3-next-20250829-master #215 PREEMPT_{RT,(full)}
T124;Tainted: [W]=WARN
T124;Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
T124;RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x194/0x1a0
T124;Code: c0 83 f8 03 0f 8f 12 ff ff ff e9 a8 25 ba ff 48 8b 53 e0 48 8b 73 e8 48 c7 c7 f0 1c 8c 8e c6 05 ec 7c a7 00 01 e8 3c a9 c1 ff <0f> 0b eb c4 0f 1f 84 00 00 00 00 00 41 55 41 54 55 48 89 fd 53 48
T124;RSP: 0018:ffffb86540587da0 EFLAGS: 00010282
Aug 31 12:01:07 localhost 4,1297,74466412,-,caller=T124;RAX: 0000000000000000 RBX: ffff893620a67c60 RCX: 0000000000000027
T124;RDX: ffff8938de8d6d48 RSI: 0000000000000001 RDI: ffff8938de8d6d40
T124;RBP: 8000000000000001 R08: 0000000000000f15 R09: 00000000ffffffff
T124;R10: 0000000000000001 R11: 0000000000000008 R12: ffff893620a67c40
T124;R13: ffffb86540587df0 R14: 000000000000001c R15: 0000000000000000
T124;FS: 0000000000000000(0000) GS:ffff89394f8fc000(0000) knlGS:0000000000000000
T124;CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
T124;CR2: 00007fa2fc9b8b60 CR3: 0000000107e89000 CR4: 0000000000750ef0
T124;PKRU: 55555554
T124;Call Trace:
T124; <TASK>
T124; rcu_do_batch+0x1b6/0x540
T124; ? rcu_do_batch+0x156/0x540
T124; rcu_core+0x141/0x250
T124; ? rcu_cpu_kthread+0x22/0xe0
T124; rcu_cpu_kthread+0x85/0xe0
T124; ? sort_range+0x20/0x20
T124; smpboot_thread_fn+0xd8/0x1d0
T124; kthread+0xe9/0x1e0
T124; ? kthreads_online_cpu+0x100/0x100
T124; ? kthreads_online_cpu+0x100/0x100
T124; ret_from_fork+0x18e/0x1c0
T124; ? kthreads_online_cpu+0x100/0x100
T124; ret_from_fork_asm+0x11/0x20
T124; </TASK>
T124;---[ end trace 0000000000000000 ]---
T124;percpu_ref_switch_to_atomic_rcu: percpu_ref_switch_to_atomic_rcu(): percpu_ref underflow slab kmalloc-64 start ffff893620a67c40 pointer offset 0 size 64 T153;unregister_netdevice: waiting for lo to become free. Usage count = -2
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
T153;unregister_netdevice: waiting for lo to become free. Usage count = -1
Both errors messages occur with and without PREEMPT_RT.
I bisected this (between v6.17-rc3 and next-20250829) using the first error (WARNING: kernel/futex/core.c:1604) as signal for a bad commit. During the bisection some additional and more severe errors occured, I do not know yet if or how all these errors are linked:
At commit e255111fdcae a NULL pointer dereference happened:
[ 8.612505] [ T159] BUG: kernel NULL pointer dereference, address: 0000000000000020
[ 8.612508] [ T159] #PF: supervisor read access in kernel mode
[ 8.612509] [ T159] #PF: error_code(0x0000) - not-present page
[ 8.612511] [ T159] PGD 0 P4D 0
[ 8.612513] [ T159] Oops: Oops: 0000 [#1] SMP NOPTI
[ 8.612517] [ T159] CPU: 8 UID: 0 PID: 159 Comm: kworker/u64:14 Tainted: G W 6.17.0-rc3-bisect-00614-ge255111fdcae #219 PREEMPT_{RT,(full)}
[ 8.612521] [ T159] Tainted: [W]=WARN
[ 8.612522] [ T159] Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
[ 8.612523] [ T159] Workqueue: events_unbound flush_memcg_stats_dwork
[ 8.612528] [ T159] RIP: 0010:css_rstat_flush+0xe7/0x510
[ 8.612532] [ T159] Code: 48 01 d0 48 89 04 24 4d 63 fc 48 8b 3c 24 e8 00 02 31 00 48 89 c6 48 85 c0 74 69 48 89 00 48 8b 70 08 4a 8b 0c fd 80 2c 74 82 <48> 8b 46 20 48 8b 96 b8 00 00 00 48 83 7c 01 08 00 48 8d 3c 08 74
[ 8.612534] [ T159] RSP: 0018:ffffa8ef806abdf8 EFLAGS: 00010282
[ 8.612536] [ T159] RAX: ffffc8ef7fbde7b0 RBX: ffff9ce2c0261380 RCX: ffff9ce61ba00000
[ 8.612537] [ T159] RDX: ffffc8ef7fbde7b0 RSI: 0000000000000000 RDI: ffff9ce59e7e7488
[ 8.612538] [ T159] RBP: 0000000000000007 R08: ffff9ce59e7a7478 R09: ffff9ce2dbd30d00
[ 8.612539] [ T159] R10: 0000000000000028 R11: ffff9ce59e7a6e80 R12: 0000000000000007
[ 8.612540] [ T159] R13: ffff9ce59e7e7490 R14: ffffffffffffffff R15: 0000000000000007
[ 8.612541] [ T159] FS: 0000000000000000(0000) GS:ffff9ce61ba40000(0000) knlGS:0000000000000000
[ 8.612542] [ T159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8.612544] [ T159] CR2: 0000000000000020 CR3: 00000001d5e22000 CR4: 0000000000750ef0
[ 8.612545] [ T159] PKRU: 55555554
[ 8.612545] [ T159] Call Trace:
[ 8.612547] [ T159] <TASK>
[ 8.612550] [ T159] flush_memcg_stats_dwork+0x37/0x60
[ 8.612552] [ T159] process_one_work+0x17b/0x290
[ 8.612557] [ T159] worker_thread+0x2ca/0x400
[ 8.612560] [ T159] ? rescuer_thread+0x4f0/0x4f0
[ 8.612562] [ T159] kthread+0xe9/0x1e0
[ 8.612564] [ T159] ? kthreads_online_cpu+0x100/0x100
[ 8.612566] [ T159] ? kthreads_online_cpu+0x100/0x100
[ 8.612568] [ T159] ret_from_fork+0x18e/0x1c0
[ 8.612571] [ T159] ? kthreads_online_cpu+0x100/0x100
[ 8.612573] [ T159] ret_from_fork_asm+0x11/0x20
[ 8.612578] [ T159] </TASK>
[ 8.612579] [ T159] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi btusb snd_hda_intel btrtl snd_intel_dspcfg btintel uvcvideo snd_hda_codec btbcm btmtk snd_soc_dmic snd_acp3x_pdm_dma snd_acp3x_rn videobuf2_vmalloc snd_soc_core snd_hda_core videobuf2_memops uvc snd_hwdep videobuf2_v4l2 bluetooth snd_pcm_oss videodev snd_mixer_oss snd_pcm snd_rn_pci_acp3x snd_acp_config videobuf2_common snd_timer snd_soc_acpi msi_wmi ecdh_generic ecc sparse_keymap mc snd wmi_bmof edac_mce_amd k10temp soundcore ccp snd_pci_acp3x battery ac button hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_prox hid_sensor_magn_3d hid_sensor_als joydev hid_sensor_trigger industrialio_triggered_buffer kfifo_buf industrialio amd_pmc evdev hid_sensor_iio_common mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 mac80211 libarc4 cfg80211 rfkill msr nvme_fabrics fuse configfs efi_pstore
[ 8.612632] [ T159] efivarfs autofs4 ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm drm_panel_backlight_quirks drm_exec drm_suballoc_helper amdxcp drm_buddy xhci_pci gpu_sched xhci_hcd drm_display_helper hid_sensor_hub hid_multitouch mfd_core hid_generic i2c_hid_acpi psmouse usbcore drm_kms_helper nvme amd_sfh i2c_hid hid serio_raw cec i2c_piix4 nvme_core r8169 i2c_smbus usb_common crc16 i2c_designware_platform i2c_designware_core
[ 8.612658] [ T159] CR2: 0000000000000020
[ 8.612660] [ T159] ---[ end trace 0000000000000000 ]---
[ 8.733174] [ T159] RIP: 0010:css_rstat_flush+0xe7/0x510
[ 8.733174] [ T159] Code: 48 01 d0 48 89 04 24 4d 63 fc 48 8b 3c 24 e8 00 02 31 00 48 89 c6 48 85 c0 74 69 48 89 00 48 8b 70 08 4a 8b 0c fd 80 2c 74 82 <48> 8b 46 20 48 8b 96 b8 00 00 00 48 83 7c 01 08 00 48 8d 3c 08 74
[ 8.733174] [ T159] RSP: 0018:ffffa8ef806abdf8 EFLAGS: 00010282
[ 8.733174] [ T159] RAX: ffffc8ef7fbde7b0 RBX: ffff9ce2c0261380 RCX: ffff9ce61ba00000
[ 8.733174] [ T159] RDX: ffffc8ef7fbde7b0 RSI: 0000000000000000 RDI: ffff9ce59e7e7488
[ 8.733174] [ T159] RBP: 0000000000000007 R08: ffff9ce59e7a7478 R09: ffff9ce2dbd30d00
[ 8.733174] [ T159] R10: 0000000000000028 R11: ffff9ce59e7a6e80 R12: 0000000000000007
[ 8.733174] [ T159] R13: ffff9ce59e7e7490 R14: ffffffffffffffff R15: 0000000000000007
[ 8.733174] [ T159] FS: 0000000000000000(0000) GS:ffff9ce61ba40000(0000) knlGS:0000000000000000
[ 8.739103] [ T159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8.739103] [ T159] CR2: 0000000000000020 CR3: 00000001d5e22000 CR4: 0000000000750ef0
[ 8.739103] [ T159] PKRU: 55555554At commit 4d8bb6e8f4a2 a bad rss-counter state appeared (CCing akpm due to the rss-counter state):
[ 7.370238] [ T84] BUG: Bad rss-counter state mm:000000007094a157 type:MM_FILEPAGES val:-14 Comm:rcuc/1 Pid:84
[ 7.370244] [ T84] BUG: Bad rss-counter state mm:000000007094a157 type:MM_ANONPAGES val:1 Comm:rcuc/1 Pid:84
[ 7.384238] [ T100] BUG: Bad rss-counter state mm:000000007369ddda type:MM_FILEPAGES val:-11 Comm:rcuc/5 Pid:100
[ 7.384241] [ T100] BUG: Bad rss-counter state mm:000000007369ddda type:MM_ANONPAGES val:-28 Comm:rcuc/5 Pid:100
[ 7.396237] [ T76] BUG: Bad rss-counter state mm:0000000088a6974c type:MM_FILEPAGES val:-3 Comm:rcuc/14 Pid:76
[ 7.396241] [ T76] BUG: Bad rss-counter state mm:0000000088a6974c type:MM_SWAPENTS val:48 Comm:rcuc/14 Pid:76
[ 7.396243] [ T76] BUG: Bad rss-counter state mm:0000000088a6974c type:MM_SHMEMPAGES val:-1 Comm:rcuc/14 Pid:76
[ 7.414298] [ T108] ------------[ cut here ]------------
[ 7.414300] [ T108] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after switching to atomic
[ 7.414309] [ T108] WARNING: CPU: 7 PID: 108 at lib/percpu-refcount.c:197 percpu_ref_switch_to_atomic_rcu+0x194/0x1a0
[ 7.414315] [ T108] Modules linked in: rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg btusb snd_hda_codec btrtl btintel uvcvideo btbcm snd_acp3x_pdm_dma snd_soc_dmic snd_acp3x_rn snd_hda_core btmtk snd_soc_core snd_hwdep videobuf2_vmalloc videobuf2_memops snd_pcm_oss uvc videobuf2_v4l2 bluetooth snd_mixer_oss videodev snd_pcm snd_rn_pci_acp3x snd_acp_config snd_timer videobuf2_common snd_soc_acpi msi_wmi ecdh_generic ecc sparse_keymap mc wmi_bmof edac_mce_amd snd snd_pci_acp3x k10temp soundcore ccp battery ac button joydev hid_sensor_accel_3d hid_sensor_gyro_3d hid_sensor_als hid_sensor_prox hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf amd_pmc industrialio hid_sensor_iio_common evdev mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 mac80211 libarc4 cfg80211 rfkill msr fuse nvme_fabrics efi_pstore configfs efivarfs autofs4 ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm
[ 7.414370] [ T108] drm_panel_backlight_quirks drm_exec xhci_pci drm_suballoc_helper amdxcp xhci_hcd drm_buddy hid_multitouch gpu_sched hid_sensor_hub mfd_core hid_generic drm_display_helper i2c_hid_acpi nvme amd_sfh i2c_hid drm_kms_helper usbcore psmouse hid nvme_core r8169 cec serio_raw i2c_piix4 usb_common i2c_smbus crc16 i2c_designware_platform i2c_designware_core
[ 7.414390] [ T108] CPU: 7 UID: 0 PID: 108 Comm: rcuc/7 Not tainted 6.17.0-rc3-bisect-00368-g4d8bb6e8f4a2 #222 PREEMPT_{RT,(full)}
[ 7.414393] [ T108] Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
[ 7.414394] [ T108] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x194/0x1a0
[ 7.414397] [ T108] Code: c0 83 f8 03 0f 8f 12 ff ff ff e9 08 77 ba ff 48 8b 53 e0 48 8b 73 e8 48 c7 c7 a8 ef 2b b8 c6 05 0c cd a7 00 01 e8 fc eb c1 ff <0f> 0b eb c4 0f 1f 84 00 00 00 00 00 41 55 41 54 55 48 89 fd 53 48
[ 7.414399] [ T108] RSP: 0018:ffff9fff804ffda0 EFLAGS: 00010282
[ 7.414401] [ T108] RAX: 0000000000000000 RBX: ffff8df6a5583660 RCX: 0000000000000027
[ 7.414402] [ T108] RDX: ffff8df95e7d6d48 RSI: 0000000000000001 RDI: ffff8df95e7d6d40
[ 7.414403] [ T108] RBP: 7ffffffffffffffe R08: 0000000000000dcb R09: 00000000ffffffff
[ 7.414405] [ T108] R10: 0000000000000001 R11: 0000000000000008 R12: ffff8df6a5583640
[ 7.414406] [ T108] R13: ffff9fff804ffdf0 R14: 00000000000000b4 R15: 0000000000000000
[ 7.414407] [ T108] FS: 0000000000000000(0000) GS:ffff8df9a5e00000(0000) knlGS:0000000000000000
[ 7.414408] [ T108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7.414409] [ T108] CR2: 000055a8070e5908 CR3: 0000000101a48000 CR4: 0000000000750ef0
[ 7.414411] [ T108] PKRU: 55555554
[ 7.414411] [ T108] Call Trace:
[ 7.414413] [ T108] <TASK>
[ 7.414415] [ T108] rcu_do_batch+0x1b6/0x540
[ 7.414420] [ T108] ? rcu_do_batch+0x156/0x540
[ 7.414424] [ T108] rcu_core+0x141/0x250
[ 7.414426] [ T108] ? rcu_cpu_kthread+0x22/0xe0
[ 7.414428] [ T108] rcu_cpu_kthread+0x85/0xe0
[ 7.414431] [ T108] ? sort_range+0x20/0x20
[ 7.414432] [ T108] smpboot_thread_fn+0xd8/0x1d0
[ 7.414434] [ T108] kthread+0xe9/0x1e0
[ 7.414437] [ T108] ? kthreads_online_cpu+0x100/0x100
[ 7.414438] [ T108] ? kthreads_online_cpu+0x100/0x100
[ 7.414441] [ T108] ret_from_fork+0x18e/0x1c0
[ 7.414444] [ T108] ? kthreads_online_cpu+0x100/0x100
[ 7.414446] [ T108] ret_from_fork_asm+0x11/0x20
[ 7.414451] [ T108] </TASK>
[ 7.414452] [ T108] ---[ end trace 0000000000000000 ]---
[ 7.414453] [ T108] percpu_ref_switch_to_atomic_rcu: percpu_ref_switch_to_atomic_rcu(): percpu_ref underflow slab kmalloc-64 start ffff8df6a5583640 pointer offset 0 size 64
[ 8.255241] [ T124] ------------[ cut here ]------------
[ 8.255243] [ T124] WARNING: CPU: 11 PID: 124 at kernel/futex/core.c:1604 futex_ref_rcu+0xe8/0x100
[ 8.255249] [ T124] Modules linked in: snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg btusb snd_hda_codec btrtl btintel uvcvideo btbcm snd_acp3x_pdm_dma snd_soc_dmic snd_acp3x_rn snd_hda_core btmtk snd_soc_core snd_hwdep videobuf2_vmalloc videobuf2_memops snd_pcm_oss uvc videobuf2_v4l2 bluetooth snd_mixer_oss videodev snd_pcm snd_rn_pci_acp3x snd_acp_config snd_timer videobuf2_common snd_soc_acpi msi_wmi ecdh_generic ecc sparse_keymap mc wmi_bmof edac_mce_amd snd snd_pci_acp3x k10temp soundcore ccp battery ac button joydev hid_sensor_accel_3d hid_sensor_gyro_3d hid_sensor_als hid_sensor_prox hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf amd_pmc industrialio hid_sensor_iio_common evdev mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 mac80211 libarc4 cfg80211 rfkill msr fuse nvme_fabrics efi_pstore configfs efivarfs autofs4
[ 8.255293] [ T124] ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm drm_panel_backlight_quirks drm_exec xhci_pci drm_suballoc_helper amdxcp xhci_hcd drm_buddy hid_multitouch gpu_sched hid_sensor_hub mfd_core hid_generic drm_display_helper i2c_hid_acpi nvme amd_sfh i2c_hid drm_kms_helper usbcore psmouse hid nvme_core r8169 cec serio_raw i2c_piix4 usb_common i2c_smbus crc16 i2c_designware_platform i2c_designware_core
[ 8.255314] [ T124] CPU: 11 UID: 0 PID: 124 Comm: rcuc/11 Tainted: G W 6.17.0-rc3-bisect-00368-g4d8bb6e8f4a2 #222 PREEMPT_{RT,(full)}
[ 8.255317] [ T124] Tainted: [W]=WARN
[ 8.255317] [ T124] Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
[ 8.255318] [ T124] RIP: 0010:futex_ref_rcu+0xe8/0x100
[ 8.255321] [ T124] Code: be ff ff ff ff ff ff ff 7f 48 89 b3 88 01 00 00 c7 00 01 00 00 00 48 8d bb 78 01 00 00 5b 48 c7 c6 f0 c5 74 b7 e9 f8 7a fd ff <0f> 0b eb ab 0f 0b e9 31 ff ff ff 0f 0b eb c6 66 0f 1f 84 00 00 00
[ 8.255322] [ T124] RSP: 0018:ffff9fff80587db0 EFLAGS: 00010286
[ 8.255324] [ T124] RAX: 7fffffffffffffff RBX: ffff8df6a02bbe80 RCX: 0000000000000010
[ 8.255325] [ T124] RDX: ffffffffffffffff RSI: ffffffffffffffff RDI: 000000000000000f
[ 8.255326] [ T124] RBP: ffff8df680b4a000 R08: 0000000000000000 R09: 0000000000000005
[ 8.255327] [ T124] R10: 0000000000200017 R11: 0000000000000000 R12: ffff8df95e8e2300
[ 8.255327] [ T124] R13: ffff9fff80587df0 R14: 0000000000000007 R15: 0000000000000000
[ 8.255328] [ T124] FS: 0000000000000000(0000) GS:ffff8df9a5f00000(0000) knlGS:0000000000000000
[ 8.255330] [ T124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8.255331] [ T124] CR2: 0000561aa64a1020 CR3: 00000001209dd000 CR4: 0000000000750ef0
[ 8.255331] [ T124] PKRU: 55555554
[ 8.255332] [ T124] Call Trace:
[ 8.255334] [ T124] <TASK>
[ 8.255335] [ T124] rcu_do_batch+0x1b6/0x540
[ 8.255339] [ T124] ? rcu_do_batch+0x156/0x540
[ 8.255342] [ T124] rcu_core+0x141/0x250
[ 8.255344] [ T124] ? rcu_cpu_kthread+0x22/0xe0
[ 8.255345] [ T124] rcu_cpu_kthread+0x85/0xe0
[ 8.255347] [ T124] ? sort_range+0x20/0x20
[ 8.255349] [ T124] smpboot_thread_fn+0xd8/0x1d0
[ 8.255351] [ T124] kthread+0xe9/0x1e0
[ 8.255354] [ T124] ? kthreads_online_cpu+0x100/0x100
[ 8.255355] [ T124] ret_from_fork+0x18e/0x1c0
[ 8.255358] [ T124] ? kthreads_online_cpu+0x100/0x100
[ 8.255360] [ T124] ret_from_fork_asm+0x11/0x20
[ 8.255364] [ T124] </TASK>
[ 8.255365] [ T124] ---[ end trace 0000000000000000 ]---The result of the bisection was this as the first bad commit:1b708b38414d ("futex: Move futex_hash_free() back to __mmput()")
I reverted commit 1b708b38414d in next-20250829 and all the errors disappeared, but do not know yet how all these errors are linked.
Bert Karwatzki
Re: (Re)boot hangs and refcount errors on next-20250829
I've found the missing piece of the patch here:
https://lore.kernel.org/all/175664303766.1920.13212322629796216488.tip-bot2@tip-bot2/#t
kernel/futex/core.c +1 -0
@@ -1722,6 +1722,7 @@ int futex_mm_init(struct mm_struct *mm)RCU_INIT_POINTER (mm->futex_phash,NULL ); mm->futex_phash_new =NULL ;/* futex-ref */ + mm->futex_ref =NULL ;atomic_long_set (&mm->futex_atomic,0 ); mm->futex_batches =get_state_synchronize_rcu ();return 0 ;
This solves all the mentioned Problems!
Bert Karwatzki
Re: (Re)boot hangs and refcount errors on next-20250829
In reply to Bert KarwatzkiQuoted 9 lines · Bert Karwatzki
I think I hit the the same error (and some more ...) when booting next-20250829 on my amd64 laptop (no VMs and no kexec were used here):
When I try booting next-20250829 on my debian stable(trixie) amd64 system the boot process hangs every few attempts (without any log messages recorded). When the boot process succeeds the following error message appears in dmesg:
[ 8.337248] [ T44] ------------[ cut here ]------------ [ 8.337250] [ T44] WARNING: kernel/futex/core.c:1604 at futex_ref_rcu+0xe8/0x100, CPU#6: rcuc/6/44
I believe that got fixed, pls try:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=locking/urgent
which got updated to contain the fix for the warning above.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSebastian Andrzej Siewior
maintainerQuoted 12 lines · Linus Torvalds
Quoted 2 lines · Borislav Petkov
please pull a locking/urgent fix for v6.17-rc1.
Ok, so this clearly wasn't a fix.
Quoted 2 lines
Thomas Gleixner (1):
futex: Move futex cleanup to__mmdrop()So this causes problems, because
__mmdropis not done in thread context, and thekvfree()calls then cause issues:
Quoted 7 lines
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/Hilf Danton sent out a patch, but honestly, that patch looks like pure bandaid, and will make the exit path horribly much slower by moving things into workqueues. It might not be visible in profiles exactly *because* it's then hidden in workqueues, but it's not great.
vfree() has an in_interrupt() check. Extending it to an irq-check would check this but not a section where a spin_lock_t is held since we can't check for disabled preemption. And therefore another band aid.
Quoted 3 lines
I think it's a mistake to allow vmalloc'ing those hashes in the first place, and I suggest the local hash be size-limited to the point where it's just a
kmalloc()and thus works in all contexts.
default auto scaling with 512 CPUs without lockdep maxes out
64 + (64 * 512 * 4) = 128KiB + 64.
For kmalloc() the slab cache is used up to 8KiB which crosses the limit with 32 CPUs. Then we have kmalloc()'s max allocation limit which is at 4MiB.
Given that Jakub hits the warning after sometime might indicate that it was possible to fulfill the requirement initially but over time the memory became too fragmented. So it allocated virtual memory.
Quoted 4 lines
Or maybe the mistake was the mm-private hashing in the first place. Maybe that hash shouldn't be allocated at
mm_alloc()->futex_mm_init()at all. Only initialized by the futex code when needed, and then dropped inexit_mmap().
mm_alloc() is fine. This does only an alloc_percpu() and its counter part can be used in atomic context. So the hash pointer should end up in kfree(NULL).
Let me stare at the initial report leading to the fix. Maybe we can avoid the leak and the atomic context altogether.
Quoted 3 lines
So the problems seem deeper than just "free'd in the wrong context".
Linus
Sebastian
Sebastian Andrzej Siewior
maintainerPATCH 1/1 futex: Move futex_hash_free() back to __mmput()
To avoid a memory leak via mm_alloc() + mmdrop() the futex cleanup code has been moved to __mmdrop(). This resulted in a warnings if the futex hash table has been allocated via vmalloc() the mmdrop() was invoked from atomic context.
The free path must stay in __mmput() to ensure it is invoked from preemptible context.
In order to avoid the memory leak, delay the allocation of mm_struct::mm->futex_ref to futex_hash_allocate(). This works because neither the per-CPU counter nor the private hash has been allocated and therefore
- futex_private_hash() callers (such as exit_pi_state_list()) don't acquire reference if there is no private hash yet. There is also no reference put.
- Regular callers (futex_hash()) fallback to global hash. No reference counting here.
The futex_ref member can be allocated in futex_hash_allocate() before the private hash itself is allocated. This happens either while the first thread is created or on request. In both cases the process has just a single thread so there can be either futex operation in progress or the request to create a private hash.
Move futex_hash_free() back to __mmput();
Move the allocation of mm_struct::futex_ref to futex_hash_allocate().
--- kernel/fork.c | 2 +- kernel/futex/core.c | 15 +++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
kernel/fork.c +1 -1
@@ -689,7 +689,6 @@ void __mmdrop(struct mm_struct *mm)mm_pasid_drop (mm);mm_destroy_cid (mm);percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); -futex_hash_free (mm);free_mm (mm); }
@@ -1138,6 +1137,7 @@ static inline void __mmput(struct mm_struct *mm)if (mm->binfmt)module_put (mm->binfmt->module);lru_gen_del_mm (mm); +futex_hash_free (mm);mmdrop (mm); }
kernel/futex/core.c +11 -4
@@ -1724,10 +1724,6 @@ int futex_mm_init(struct mm_struct *mm)/* futex-ref */ atomic_long_set (&mm->futex_atomic,0 ); mm->futex_batches =get_state_synchronize_rcu (); - mm->futex_ref = alloc_percpu(unsigned int); -if (!mm->futex_ref) -return -ENOMEM ; -this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ return 0 ; }
@@ -1801,6 +1797,17 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) } } +if (!mm->futex_ref) { +/* +* This will always be allocated by the first thread and +* therefore requires no locking. +*/ + mm->futex_ref = alloc_percpu(unsigned int); +if (!mm->futex_ref) +return -ENOMEM ; +this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ + } + fph =kvzalloc (struct_size (fph, queues, hash_slots),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);if (!fph)
Signature · 1 line
2.50.1
[tip: locking/urgent] futex: Move futex_hash_free() back to __mmput()
The following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 1b708b38414d32838baa39c9dee59d40731ed202 Gitweb: https://git.kernel.org/tip/1b708b38414d32838baa39c9dee59d40731ed202 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Fri, 22 Aug 2025 16:12:38 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 27 Aug 2025 13:31:07 +02:00
futex: Move futex_hash_free() back to __mmput()
To avoid a memory leak via mm_alloc() + mmdrop() the futex cleanup code has been moved to __mmdrop(). This resulted in a warnings if the futex hash table has been allocated via vmalloc() the mmdrop() was invoked from atomic context.
The free path must stay in __mmput() to ensure it is invoked from preemptible context.
In order to avoid the memory leak, delay the allocation of mm_struct::mm->futex_ref to futex_hash_allocate(). This works because neither the per-CPU counter nor the private hash has been allocated and therefore
- futex_private_hash() callers (such as exit_pi_state_list()) don't acquire reference if there is no private hash yet. There is also no reference put.
- Regular callers (futex_hash()) fallback to global hash. No reference counting here.
The futex_ref member can be allocated in futex_hash_allocate() before the private hash itself is allocated. This happens either while the first thread is created or on request. In both cases the process has just a single thread so there can be either futex operation in progress or the request to create a private hash.
Move futex_hash_free() back to __mmput();
Move the allocation of mm_struct::futex_ref to futex_hash_allocate().
--- kernel/fork.c | 2 +- kernel/futex/core.c | 15 +++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
kernel/fork.c +1 -1
@@ -689,7 +689,6 @@ void __mmdrop(struct mm_struct *mm)mm_pasid_drop (mm);mm_destroy_cid (mm);percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); -futex_hash_free (mm);free_mm (mm); }
@@ -1138,6 +1137,7 @@ static inline void __mmput(struct mm_struct *mm)if (mm->binfmt)module_put (mm->binfmt->module);lru_gen_del_mm (mm); +futex_hash_free (mm);mmdrop (mm); }
kernel/futex/core.c +11 -4
@@ -1724,10 +1724,6 @@ int futex_mm_init(struct mm_struct *mm)/* futex-ref */ atomic_long_set (&mm->futex_atomic,0 ); mm->futex_batches =get_state_synchronize_rcu (); - mm->futex_ref = alloc_percpu(unsigned int); -if (!mm->futex_ref) -return -ENOMEM ; -this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ return 0 ; }
@@ -1801,6 +1797,17 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) } } +if (!mm->futex_ref) { +/* +* This will always be allocated by the first thread and +* therefore requires no locking. +*/ + mm->futex_ref = alloc_percpu(unsigned int); +if (!mm->futex_ref) +return -ENOMEM ; +this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ + } + fph =kvzalloc (struct_size (fph, queues, hash_slots),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);if (!fph)
[tip: locking/urgent] futex: Move futex_hash_free() back to __mmput()
In reply to Sebastian Andrzej SiewiorThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: d9b05321e21e4b218de4ce8a590bf375f58b6346 Gitweb: https://git.kernel.org/tip/d9b05321e21e4b218de4ce8a590bf375f58b6346 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Fri, 22 Aug 2025 16:12:38 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Sun, 31 Aug 2025 11:48:19 +02:00
futex: Move futex_hash_free() back to __mmput()
To avoid a memory leak via mm_alloc() + mmdrop() the futex cleanup code has been moved to __mmdrop(). This resulted in a warnings if the futex hash table has been allocated via vmalloc() the mmdrop() was invoked from atomic context.
The free path must stay in __mmput() to ensure it is invoked from preemptible context.
In order to avoid the memory leak, delay the allocation of mm_struct::mm->futex_ref to futex_hash_allocate(). This works because neither the per-CPU counter nor the private hash has been allocated and therefore
- futex_private_hash() callers (such as exit_pi_state_list()) don't acquire reference if there is no private hash yet. There is also no reference put.
- Regular callers (futex_hash()) fallback to global hash. No reference counting here.
The futex_ref member can be allocated in futex_hash_allocate() before the private hash itself is allocated. This happens either while the first thread is created or on request. In both cases the process has just a single thread so there can be either futex operation in progress or the request to create a private hash.
Move futex_hash_free() back to __mmput();
Move the allocation of mm_struct::futex_ref to futex_hash_allocate().
[ bp: Fold a follow-up fix to prevent a use-after-free:
https://lore.kernel.org/r/[email protected] ]
--- kernel/fork.c | 2 +- kernel/futex/core.c | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-)
kernel/fork.c +1 -1
@@ -689,7 +689,6 @@ void __mmdrop(struct mm_struct *mm)mm_pasid_drop (mm);mm_destroy_cid (mm);percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); -futex_hash_free (mm);free_mm (mm); }
@@ -1138,6 +1137,7 @@ static inline void __mmput(struct mm_struct *mm)if (mm->binfmt)module_put (mm->binfmt->module);lru_gen_del_mm (mm); +futex_hash_free (mm);mmdrop (mm); }
kernel/futex/core.c +12 -4
@@ -1722,12 +1722,9 @@ int futex_mm_init(struct mm_struct *mm)RCU_INIT_POINTER (mm->futex_phash,NULL ); mm->futex_phash_new =NULL ;/* futex-ref */ + mm->futex_ref =NULL ;atomic_long_set (&mm->futex_atomic,0 ); mm->futex_batches =get_state_synchronize_rcu (); - mm->futex_ref = alloc_percpu(unsigned int); -if (!mm->futex_ref) -return -ENOMEM ; -this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ return 0 ; }
@@ -1801,6 +1798,17 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) } } +if (!mm->futex_ref) { +/* +* This will always be allocated by the first thread and +* therefore requires no locking. +*/ + mm->futex_ref = alloc_percpu(unsigned int); +if (!mm->futex_ref) +return -ENOMEM ; +this_cpu_inc (*mm->futex_ref);/* 0 -> 1 */ + } + fph =kvzalloc (struct_size (fph, queues, hash_slots),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);if (!fph)