[6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...
====================================================
Hi David,
Quoted 71 lines
Hi,
Caught a kernel panic when rebooting, system stuck until pressing power button. I have only a screenshot when it happens, following logs were extracted from a captured picture.
863.881960]
sysved_call_function_single+0x4c/0xc0
863.881301]asm_sysvec_call_function_single+0x16/0x20
869.881344] RIP: 0633:0x7f9alcea3367
663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
869.881567] RAX:0000562060c962d0RBX: 0000000000000002 RCX:00007f9a1cff1c60863.881625] RDX:00007f9a0c000030RSI: 00007f9alcff1c60 RDI:00007f9a1ca91c20863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09:00007f9a1d6217a0869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12:00007f9a1d70c020869.881798] R13:00007fffa5270030R14:00007fffa526fd00R15: 0000000000000000 863.881860] </TASK>
863.881876) Modules linked in:snd_seq_dummy(E)snd_hrtimer(E)...
...
863.887142] button (E)
863.912127] CR2:ffffe4afcc079650
863.914593] --- [ end trace 0000000000000000 1--
864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
864.052929] RAX: 0000000000000002 RBX:ffff8e8eeed2a580RCX:ffff8e8bded9bf00864.055518] RDX:000000c92340b051RSI:000000c92340b051RDI: ffff 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000 864.063183] R13:ffff8e8eeed2a580R14: 0000000000000007 R15:ffffe4afcc079650864.065729] FS:00007f9a1ca91940(0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
864.070898] CR2:ffffe4afcc079650CR3:00000001136d8000CR4:0000000000350ef0864.673523] Kernel panic - not syncing: Fatal exception in interrupt 864.076410] Kernel Offset:0xc00000from0xffffffff81000000(relocation range:0xff864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---This happens ever since 6.15-rc1, from time to time, I would get kernel panic when reboot; it is only recently that I figured out a precedure reproducing this with *high* probability:
1. create a cgroup.
2.perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP)for each cpu with following attrs:attr.type = PERF_TYPE_SOFTWARE; attr.size = sizeof(attr); attr.config = PERF_COUNT_SW_CPU_CLOCK; attr.sample_freq = 9999; attr.freq = 1; attr.wakeup_events = 16; attr.sample_type = PERF_SAMPLE_CALLCHAIN; attr.sample_max_stack = 32; attr.exclude_callchain_user = 1;3. close all
perf_event_openafter several minutes
4. rebootAnd after an exhausting bisect on
events/core.c, (I need 5 rounds to conclude a good bisect) I think I reach the conclusion, with very high probability, that this is caused bycommit
a3c3c66670cee11eb13aa43905904bf29cb92d32Author: Yeoreum Yun <[email protected]> Date: Wed Mar 26 08:20:03 2025 +0000perf/core: Fix
child_total_time_enabledaccounting bug at task exitReverting this can fix it: I run the test 10 rounds, no kernel panic observed.
The changes made to
__perf_remove_from_contextby commit a3c3c6667("perf/core:
Fixchild_total_time_enabledaccounting bug at task exit") has wider effect than the callchain mentioned in commit message, and I think an esay fix would be just restricting the effect to that callchain only, and restore other changes back.
Quoted 4 lines
I have test the patch below several rounds, and so far so good, and I will have more tests on it.
Signed-off-by David Wang <[email protected]>
Thanks for your reporting and Sorry for my bad.
By my change, the tracking nr_cgorups is broken which could make a dangling pointer for cpuctx->cgrp.
Could you test with below change please?
kernel/events/core.c +10 -12
@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)if (event->group_leader == event)del_event_from_groups (event, ctx); -/* -* If event was in error state, then keep it -* that way, otherwise bogus counts will be -* returned on read(). The only way to get out -* of error state is by explicit re-enabling -* of the event -*/ -if (event->state >PERF_EVENT_STATE_OFF ) { -perf_cgroup_event_disable (event, ctx); -perf_event_set_state (event,PERF_EVENT_STATE_OFF ); - } - ctx->generation++; event->pmu_ctx->nr_events--; }
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,ctx_time_update (cpuctx, ctx); +/* +* If event was in error state, then keep it +* that way, otherwise bogus counts will be +* returned on read(). The only way to get out +* of error state is by explicit re-enabling +* of the event +*/ +if (event->state >PERF_EVENT_STATE_OFF ) +perf_cgroup_event_disable (event, ctx); +/* * Ensure event_sched_out() switches to OFF, at the very least * this avoids raising perf_pending_task() at this time.
Thanks
--
Sincerely,
Yeoreum Yun
Quoted 130 lines · Yeoreum Yun
Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...====================================================
Hi David,Quoted 71 lines
Hi,
Caught a kernel panic when rebooting, system stuck until pressing power button. I have only a screenshot when it happens, following logs were extracted from a captured picture.
863.881960]
sysved_call_function_single+0x4c/0xc0
863.881301]asm_sysvec_call_function_single+0x16/0x20
869.881344] RIP: 0633:0x7f9alcea3367
663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
869.881567] RAX:0000562060c962d0RBX: 0000000000000002 RCX:00007f9a1cff1c60863.881625] RDX:00007f9a0c000030RSI: 00007f9alcff1c60 RDI:00007f9a1ca91c20863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09:00007f9a1d6217a0869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12:00007f9a1d70c020869.881798] R13:00007fffa5270030R14:00007fffa526fd00R15: 0000000000000000 863.881860] </TASK>
863.881876) Modules linked in:snd_seq_dummy(E)snd_hrtimer(E)...
...
863.887142] button (E)
863.912127] CR2:ffffe4afcc079650
863.914593] --- [ end trace 0000000000000000 1--
864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
864.052929] RAX: 0000000000000002 RBX:ffff8e8eeed2a580RCX:ffff8e8bded9bf00864.055518] RDX:000000c92340b051RSI:000000c92340b051RDI: ffff 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000 864.063183] R13:ffff8e8eeed2a580R14: 0000000000000007 R15:ffffe4afcc079650864.065729] FS:00007f9a1ca91940(0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
864.070898] CR2:ffffe4afcc079650CR3:00000001136d8000CR4:0000000000350ef0864.673523] Kernel panic - not syncing: Fatal exception in interrupt 864.076410] Kernel Offset:0xc00000from0xffffffff81000000(relocation range:0xff864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---This happens ever since 6.15-rc1, from time to time, I would get kernel panic when reboot; it is only recently that I figured out a precedure reproducing this with *high* probability:
1. create a cgroup.
2.perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP)for each cpu with following attrs:attr.type = PERF_TYPE_SOFTWARE; attr.size = sizeof(attr); attr.config = PERF_COUNT_SW_CPU_CLOCK; attr.sample_freq = 9999; attr.freq = 1; attr.wakeup_events = 16; attr.sample_type = PERF_SAMPLE_CALLCHAIN; attr.sample_max_stack = 32; attr.exclude_callchain_user = 1;3. close all
perf_event_openafter several minutes
4. rebootAnd after an exhausting bisect on
events/core.c, (I need 5 rounds to conclude a good bisect) I think I reach the conclusion, with very high probability, that this is caused bycommit
a3c3c66670cee11eb13aa43905904bf29cb92d32Author: Yeoreum Yun <[email protected]> Date: Wed Mar 26 08:20:03 2025 +0000perf/core: Fix
child_total_time_enabledaccounting bug at task exitReverting this can fix it: I run the test 10 rounds, no kernel panic observed.
The changes made to
__perf_remove_from_contextby commit a3c3c6667("perf/core:
Fixchild_total_time_enabledaccounting bug at task exit") has wider effect than the callchain mentioned in commit message, and I think an esay fix would be just restricting the effect to that callchain only, and restore other changes back.Quoted 4 lines
I have test the patch below several rounds, and so far so good, and I will have more tests on it.
Signed-off-by David Wang <[email protected]>Thanks for your reporting and Sorry for my bad.
By my change, the trackingnr_cgorupsis broken which could make a dangling pointer forcpuctx->cgrp.Could you test with below change please?
kernel/events/core.c +10
-12@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)if (event->group_leader == event)del_event_from_groups (event, ctx); -/* -* If event was in error state, then keep it -* that way, otherwise bogus counts will be -* returned on read(). The only way to get out -* of error state is by explicit re-enabling -* of the event -*/ -if (event->state >PERF_EVENT_STATE_OFF ) { -perf_cgroup_event_disable (event, ctx); -perf_event_set_state (event,PERF_EVENT_STATE_OFF ); - } - ctx->generation++; event->pmu_ctx->nr_events--; }@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,ctx_time_update (cpuctx, ctx); +/* +* If event was in error state, then keep it +* that way, otherwise bogus counts will be +* returned on read(). The only way to get out +* of error state is by explicit re-enabling +* of the event +*/ +if (event->state >PERF_EVENT_STATE_OFF ) +perf_cgroup_event_disable (event, ctx); +/* * Ensure event_sched_out() switches to OFF, at the very least * this avoids raising perf_pending_task() at this time.
Quoted 6 lines
Thanks
--
Sincerely,
Yeoreum Yun
Before I start testing, I feel concerned about following chain:
./kernel/fork.c:bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()
This patch seems changes the behavior in this callchain. Would this have other side-effect?
David
Hi David,
Quoted 10 lines
Before I start testing, I feel concerned about following chain:
./kernel/fork.c:bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()This patch seems changes the behavior in this callchain. Would this have other side-effect?
What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.
also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.
Thanks.
--
Sincerely,
Yeoreum Yun
Quoted 18 lines · Yeoreum Yun
Hi David,
Quoted 10 lines
Before I start testing, I feel concerned about following chain:
./kernel/fork.c:bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()This patch seems changes the behavior in this callchain. Would this have other side-effect?
What behavior is changed you're worry about?
both error patch is handled by__perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.
Quoted 8 lines
also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.
Thanks.
--
Sincerely,
Yeoreum Yun
Am I reading it wrong?
The call chain I mentioned above dose not walk through __perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...
David
Hi David,
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Before I start testing, I feel concerned about following chain:
./kernel/fork.c:bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()This patch seems changes the behavior in this callchain. Would this have other side-effect?
What behavior is changed you're worry about?
both error patch is handled by__perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.
Quoted 8 lines
Quoted 4 lines
also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.
Thanks.
Am I reading it wrong?
The call chain I mentioned above dose not walk through__perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...
Since commit 90661365021a
("perf Unify perf_event_free_task() / perf_evenet_exit_task_context()")
perf_event_free_task() is integrated with perf_event_exit_task_context() So, it calls __perf_remove_from_context().
In v6.15, I think you can test with below change only:
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
ctx_time_update(cpuctx, ctx);
+ /*
+ * If event was in error state, then keep it + * that way, otherwise bogus counts will be + * returned on read(). The only way to get out + * of error state is by explicit re-enabling + * of the event + */
+ if (event->state > PERF_EVENT_STATE_OFF)
+ perf_cgroup_event_disable(event, ctx);
+
/*
* Ensure event_sched_out() switches to OFF, at the very least
* this avoids raising perf_pending_task() at this time.
not with modification with "list_del_event()".
Thanks
--
Sincerely,
Yeoreum Yun
David Wang
Tested-byQuoted 27 lines · Yeoreum Yun
Hi David,
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Before I start testing, I feel concerned about following chain:
./kernel/fork.c:bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()This patch seems changes the behavior in this callchain. Would this have other side-effect?
What behavior is changed you're worry about?
both error patch is handled by__perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.Quoted 8 lines
Quoted 4 lines
also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.
Thanks.
Am I reading it wrong?
The call chain I mentioned above dose not walk through__perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...
Quoted 5 lines
Since commit
90661365021a
("perf Unifyperf_event_free_task()/perf_evenet_exit_task_context()")
perf_event_free_task()is integrated withperf_event_exit_task_context()So, it calls__perf_remove_from_context().
Good to know~
Quoted 20 lines
In v6.15, I think you can test with below change only:
kernel/events/core.c +10
-0@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,ctx_time_update (cpuctx, ctx); +/* +* If event was in error state, then keep it +* that way, otherwise bogus counts will be +* returned on read(). The only way to get out +* of error state is by explicit re-enabling +* of the event +*/ +if (event->state >PERF_EVENT_STATE_OFF ) +perf_cgroup_event_disable (event, ctx); +/* * Ensure event_sched_out() switches to OFF, at the very least * this avoids raising perf_pending_task() at this time.
Quoted 1 line
not with modification with "
list_del_event()".
... I apply your original patch on 6.15, up to now, 7 rounds of test show no sign of kernel panic. I think, the patch does fix it.
Thanks
David