Skip to contentSkip to conversation

[6.15][perf] Kernel panic not syncing: Fatal exception in interrupt

BUG Yeoreum Yun 2 participants 6 messages Jun 2, 2025 perf events

lore
Share & tools
lore
#0 lore raw

Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...

====================================================
Hi David,

Quoted 71 lines

Hi,

Caught a kernel panic when rebooting, system stuck until pressing power button. I have only a screenshot when it happens, following logs were extracted from a captured picture.

863.881960] sysved_call_function_sing le+0x4c/0xc0
863.881301] asm_sysvec_call_function_single+0x16/0x20
869.881344] RIP: 0633:0x7f9alcea3367
663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60 863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20 863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0 869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020 869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000 863.881860] </TASK>
863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
...
863.887142] button (E)
863.912127] CR2: ffffe4afcc079650
863.914593] --- [ end trace 0000000000000000 1--
864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00 864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000 864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650 864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0 864.673523] Kernel panic - not syncing: Fatal exception in interrupt 864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff 864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---

This happens ever since 6.15-rc1, from time to time, I would get kernel panic when reboot; it is only recently that I figured out a precedure reproducing this with *high* probability:

1. create a cgroup.
2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:

	attr.type = PERF_TYPE_SOFTWARE;
	attr.size = sizeof(attr);
	attr.config = PERF_COUNT_SW_CPU_CLOCK;
	attr.sample_freq = 9999;
	attr.freq = 1;
	attr.wakeup_events = 16;
	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
	attr.sample_max_stack = 32;
	attr.exclude_callchain_user = 1;

3. close all perf_event_open after several minutes
4. reboot

And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect) I think I reach the conclusion, with very high probability, that this is caused by

commit a3c3c66670cee11eb13aa43905904bf29cb92d32

Author: Yeoreum Yun <[email protected]>
Date:   Wed Mar 26 08:20:03 2025 +0000

perf/core: Fix child_total_time_enabled accounting bug at task exit

Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.

The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
Fix child_total_time_enabled accounting bug at task exit") has wider effect than the callchain mentioned in commit message, and I think an esay fix would be just restricting the effect to that callchain only, and restore other changes back.

Quoted 4 lines

I have test the patch below several rounds, and so far so good, and I will have more tests on it.

Thanks for your reporting and Sorry for my bad.
By my change, the tracking nr_cgorups is broken which could make a dangling pointer for cpuctx->cgrp.

Could you test with below change please?

kernel/events/core.c +10 -12
index 95e703891b24..d0a9096735b9 100644
@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
        if (event->group_leader == event)
                del_event_from_groups(event, ctx);
 
-       /*
-        * If event was in error state, then keep it
-        * that way, otherwise bogus counts will be
-        * returned on read(). The only way to get out
-        * of error state is by explicit re-enabling
-        * of the event
-        */
-       if (event->state > PERF_EVENT_STATE_OFF) {
-               perf_cgroup_event_disable(event, ctx);
-               perf_event_set_state(event, PERF_EVENT_STATE_OFF);
-       }
-
        ctx->generation++;
        event->pmu_ctx->nr_events--;
 }
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
 
        ctx_time_update(cpuctx, ctx);
 
+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
        /*
         * Ensure event_sched_out() switches to OFF, at the very least
         * this avoids raising perf_pending_task() at this time.

Thanks

--
Sincerely,
Yeoreum Yun

#1 lore raw
Quoted 130 lines · Yeoreum Yun

Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...

====================================================
Hi David,

Quoted 71 lines

Hi,

Caught a kernel panic when rebooting, system stuck until pressing power button. I have only a screenshot when it happens, following logs were extracted from a captured picture.

863.881960] sysved_call_function_sing le+0x4c/0xc0
863.881301] asm_sysvec_call_function_single+0x16/0x20
869.881344] RIP: 0633:0x7f9alcea3367
663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60 863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20 863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0 869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020 869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000 863.881860] </TASK>
863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
...
863.887142] button (E)
863.912127] CR2: ffffe4afcc079650
863.914593] --- [ end trace 0000000000000000 1--
864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00 864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000 864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650 864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0 864.673523] Kernel panic - not syncing: Fatal exception in interrupt 864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff 864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---

This happens ever since 6.15-rc1, from time to time, I would get kernel panic when reboot; it is only recently that I figured out a precedure reproducing this with *high* probability:

1. create a cgroup.
2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:

	attr.type = PERF_TYPE_SOFTWARE;
	attr.size = sizeof(attr);
	attr.config = PERF_COUNT_SW_CPU_CLOCK;
	attr.sample_freq = 9999;
	attr.freq = 1;
	attr.wakeup_events = 16;
	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
	attr.sample_max_stack = 32;
	attr.exclude_callchain_user = 1;

3. close all perf_event_open after several minutes
4. reboot

And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect) I think I reach the conclusion, with very high probability, that this is caused by

commit a3c3c66670cee11eb13aa43905904bf29cb92d32

Author: Yeoreum Yun <[email protected]>
Date:   Wed Mar 26 08:20:03 2025 +0000

perf/core: Fix child_total_time_enabled accounting bug at task exit

Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.

The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
Fix child_total_time_enabled accounting bug at task exit") has wider effect than the callchain mentioned in commit message, and I think an esay fix would be just restricting the effect to that callchain only, and restore other changes back.

Quoted 4 lines

I have test the patch below several rounds, and so far so good, and I will have more tests on it.

Thanks for your reporting and Sorry for my bad.
By my change, the tracking nr_cgorups is broken which could make a dangling pointer for cpuctx->cgrp.

Could you test with below change please?

kernel/events/core.c +10 -12
index 95e703891b24..d0a9096735b9 100644
@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
       if (event->group_leader == event)
               del_event_from_groups(event, ctx);
 
-       /*
-        * If event was in error state, then keep it
-        * that way, otherwise bogus counts will be
-        * returned on read(). The only way to get out
-        * of error state is by explicit re-enabling
-        * of the event
-        */
-       if (event->state > PERF_EVENT_STATE_OFF) {
-               perf_cgroup_event_disable(event, ctx);
-               perf_event_set_state(event, PERF_EVENT_STATE_OFF);
-       }
-
       ctx->generation++;
       event->pmu_ctx->nr_events--;
 }
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
 
       ctx_time_update(cpuctx, ctx);
 
+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
       /*
        * Ensure event_sched_out() switches to OFF, at the very least
        * this avoids raising perf_pending_task() at this time.
Quoted 6 lines

Thanks

--
Sincerely,
Yeoreum Yun

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()

This patch seems changes the behavior in this callchain. Would this have other side-effect?

David

#2 lore raw
In reply to Yeoreum Yun

Hi David,

Quoted 10 lines

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()

This patch seems changes the behavior in this callchain. Would this have other side-effect?

What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.

also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.

Thanks.

--
Sincerely,
Yeoreum Yun

#3 lore raw
Quoted 18 lines · Yeoreum Yun

Hi David,

Quoted 10 lines

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()

This patch seems changes the behavior in this callchain. Would this have other side-effect?

What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.

Quoted 8 lines

also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.

Thanks.

--
Sincerely,
Yeoreum Yun

Am I reading it wrong?
The call chain I mentioned above dose not walk through __perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...

David

#4 lore raw
In reply to Yeoreum Yun

Hi David,

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()

This patch seems changes the behavior in this callchain. Would this have other side-effect?

What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.

Quoted 8 lines
Quoted 4 lines

also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.

Thanks.

Am I reading it wrong?
The call chain I mentioned above dose not walk through __perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...

Since commit 90661365021a
("perf Unify perf_event_free_task() / perf_evenet_exit_task_context()")

perf_event_free_task() is integrated with perf_event_exit_task_context() So, it calls __perf_remove_from_context().

In v6.15, I think you can test with below change only:
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,

        ctx_time_update(cpuctx, ctx);

+ /*

+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */

+ if (event->state > PERF_EVENT_STATE_OFF)
+ perf_cgroup_event_disable(event, ctx);
+
/*
* Ensure event_sched_out() switches to OFF, at the very least
* this avoids raising perf_pending_task() at this time.

not with modification with "list_del_event()".

Thanks
--
Sincerely,
Yeoreum Yun

Tested-by
#5 lore raw
Quoted 27 lines · Yeoreum Yun

Hi David,

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
perf_event_free_task()
perf_free_event()
list_del_event()

This patch seems changes the behavior in this callchain. Would this have other side-effect?

What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(), There wouldn't be no problem since this patch just move the time of disabling cgroup before changing event state.

Quoted 8 lines
Quoted 4 lines

also, the cgroup event is for only cpuctx not added in taskctx. So, there's no effect for event attached in taskctx.

Thanks.

Am I reading it wrong?
The call chain I mentioned above dose not walk through __perf_remove_from_context, It is a fail path in fork, which happens rarely, but still possible. I guess...

Quoted 5 lines

Since commit 90661365021a
("perf Unify perf_event_free_task() / perf_evenet_exit_task_context()")

perf_event_free_task() is integrated with perf_event_exit_task_context() So, it calls __perf_remove_from_context().

Good to know~

Quoted 20 lines

In v6.15, I think you can test with below change only:

kernel/events/core.c +10 -0
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
 
       ctx_time_update(cpuctx, ctx);
 
+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
       /*
        * Ensure event_sched_out() switches to OFF, at the very least
        * this avoids raising perf_pending_task() at this time.
Quoted 1 line

not with modification with "list_del_event()".

... I apply your original patch on 6.15, up to now, 7 rounds of test show no sign of kernel panic. I think, the patch does fix it.

Thanks
David