WARN_ON_ONCE() hit at kernel/events/core.c:330
I use linux 5.1.0-rc3 on s390 and got this WARN_ON_ONCE message:
WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330
event_function_local.constprop.79+0xe2/0xe8which was introduced with
commit cca2094605ef ("perf/core: Fix event_function_local()").
This is the WARN_ON_ONCE message, which sometimes shows up in the kernel log:
[ 4598.316519] WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330
event_function_local.constprop.79+0xe2/0xe8
[ 4598.316524] Kernel panic - not syncing: panic_on_warn set ...
[ 4598.316527] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G OE
5.1.0-20190402.rc3.git0.5e7a8ca31926.300.fc29.s390x+git #1
[ 4598.316529] Hardware name: IBM 2964 NC9 712 (LPAR)
[ 4598.316531] Call Trace:
[ 4598.316534] ([<0000000000112eb8>] show_stack+0x58/0x70)
[ 4598.316538] [<0000000000a820aa>] dump_stack+0x7a/0xa8
[ 4598.316541] [<0000000000143b52>] panic+0x11a/0x2d0
[ 4598.316543] [<00000000001439b0>] __warn+0xf8/0x118
[ 4598.316545] [<0000000000a811a8>] report_bug+0xd8/0x150
[ 4598.316547] [<00000000001014ac>] do_report_trap+0xc4/0xe0
[ 4598.316549] [<0000000000101680>] illegal_op+0x138/0x150
[ 4598.316552] [<0000000000aa233c>] pgm_check_handler+0x1cc/0x220
[ 4598.316554] [<00000000002a6d02>] event_function_local.constprop.79+
0xe2/0xe8
[ 4598.316556] ([<00000000002a6c7a>] event_function_local.constprop.79+
0x5a/0xe8)
[ 4598.316559] [<00000000002aa400>] perf_pending_event+0x88/0xb0
[ 4598.316561] [<0000000000271a8c>] irq_work_run_list+0x8c/0xb8
[ 4598.316563] [<0000000000271d78>] irq_work_tick+0x48/0x68
[ 4598.316566] [<00000000001d6058>] update_process_times+0x68/0x80
[ 4598.316568] [<00000000001e6c10>] tick_sched_handle.isra.6+0x50/0x60
[ 4598.316570] [<00000000001e6c7e>] tick_sched_timer+0x5e/0xb8
[ 4598.316573] [<00000000001d6b6a>] __hrtimer_run_queues+0x10a/0x2c0
[ 4598.316575] [<00000000001d7a88>] hrtimer_interrupt+0x138/0x2a8
[ 4598.316577] [<000000000010c3e4>] do_IRQ+0xac/0xb0
[ 4598.316597] [<0000000000aa2744>] ext_int_handler+0x128/0x12c
[ 4598.316600] [<00000000001034f6>] enabled_wait+0x46/0xd0
[ 4598.316602] ([<000003e000d2fe10>] 0x3e000d2fe10)
[ 4598.316604] [<0000000000103842>] arch_cpu_idle+0x3a/0x50
[ 4598.316606] [<000000000017b808>] do_idle+0x180/0x1b0
[ 4598.316608] [<000000000017ba06>] cpu_startup_entry+0x36/0x40
[ 4598.316611] [<0000000000115d16>] smp_init_secondary+0xd6/0xf0
[ 4598.316613] [<000000000011521e>] smp_start_secondary+0x86/0x98Any ideas or hints who to avoid/fix this warning?
I ran a user space program which uses
1. perf_event_open() system call with type HARDWARE and PERF_COUNT_HW_CPU_CYCLES in sampling mode.
2. Assigns signal SIGIO to the file descriptor returned from
perf_event_open() using:
rc = fcntl(fd, F_SETFL, O_RDWR | O_NONBLOCK | O_ASYNC);
rc |= fcntl(fd, F_SETSIG, signo);
rc |= fcntl(fd, F_SETOWN, getpid()); 3. The signal handler increments some variables and issues ioctl(fd, PERF_EVENT_IOC_REFRESH, 1);
to restart signal delivery.
I can send you the test program if needed.
Thanks a lot for your help.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 47 lines · Thomas-Mich Richter
I use linux 5.1.0-rc3 on s390 and got this
WARN_ON_ONCEmessage:WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8which was introduced with
commitcca2094605ef("perf/core: Fixevent_function_local()").This is the
WARN_ON_ONCEmessage, which sometimes shows up in the kernel log:[ 4598.316519] WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8 [ 4598.316524] Kernel panic - not syncing: panic_on_warn set ... [ 4598.316527] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G OE 5.1.0-20190402.rc3.git0.5e7a8ca31926.300.fc29.s390x+git #1 [ 4598.316529] Hardware name: IBM 2964 NC9 712 (LPAR) [ 4598.316531] Call Trace: [ 4598.316534] ([<0000000000112eb8>] show_stack+0x58/0x70) [ 4598.316538] [<0000000000a820aa>] dump_stack+0x7a/0xa8 [ 4598.316541] [<0000000000143b52>] panic+0x11a/0x2d0 [ 4598.316543] [<00000000001439b0>] __warn+0xf8/0x118 [ 4598.316545] [<0000000000a811a8>] report_bug+0xd8/0x150 [ 4598.316547] [<00000000001014ac>] do_report_trap+0xc4/0xe0 [ 4598.316549] [<0000000000101680>] illegal_op+0x138/0x150 [ 4598.316552] [<0000000000aa233c>] pgm_check_handler+0x1cc/0x220 [ 4598.316554] [<00000000002a6d02>] event_function_local.constprop.79+ 0xe2/0xe8 [ 4598.316556] ([<00000000002a6c7a>] event_function_local.constprop.79+ 0x5a/0xe8) [ 4598.316559] [<00000000002aa400>] perf_pending_event+0x88/0xb0 [ 4598.316561] [<0000000000271a8c>] irq_work_run_list+0x8c/0xb8 [ 4598.316563] [<0000000000271d78>] irq_work_tick+0x48/0x68 [ 4598.316566] [<00000000001d6058>] update_process_times+0x68/0x80 [ 4598.316568] [<00000000001e6c10>] tick_sched_handle.isra.6+0x50/0x60 [ 4598.316570] [<00000000001e6c7e>] tick_sched_timer+0x5e/0xb8 [ 4598.316573] [<00000000001d6b6a>] __hrtimer_run_queues+0x10a/0x2c0 [ 4598.316575] [<00000000001d7a88>] hrtimer_interrupt+0x138/0x2a8 [ 4598.316577] [<000000000010c3e4>] do_IRQ+0xac/0xb0 [ 4598.316597] [<0000000000aa2744>] ext_int_handler+0x128/0x12c [ 4598.316600] [<00000000001034f6>] enabled_wait+0x46/0xd0 [ 4598.316602] ([<000003e000d2fe10>] 0x3e000d2fe10) [ 4598.316604] [<0000000000103842>] arch_cpu_idle+0x3a/0x50 [ 4598.316606] [<000000000017b808>] do_idle+0x180/0x1b0 [ 4598.316608] [<000000000017ba06>] cpu_startup_entry+0x36/0x40 [ 4598.316611] [<0000000000115d16>] smp_init_secondary+0xd6/0xf0 [ 4598.316613] [<000000000011521e>] smp_start_secondary+0x86/0x98
Quoted 1 line
Any ideas or hints who to avoid/fix this warning?
Some thoughts here:
https://lkml.kernel.org/r/[email protected]
tl;dr, I've no frigging clue.
Quoted 10 lines
I ran a user space program which uses
1.
perf_event_open()system call with type HARDWARE andPERF_COUNT_HW_CPU_CYCLESin sampling mode.
2. Assigns signal SIGIO to the file descriptor returned fromperf_event_open() using: rc = fcntl(fd, F_SETFL, O_RDWR | O_NONBLOCK | O_ASYNC); rc |= fcntl(fd, F_SETSIG, signo); rc |= fcntl(fd, F_SETOWN, getpid());
Quoted 5 lines
3. The signal handler increments some variables and issues
ioctl(fd, PERF_EVENT_IOC_REFRESH, 1);
to restart signal delivery.I can send you the test program if needed.
Does it reproduce on x86 without virt on? I don't have a PPC LPAR to test things on.
Quoted 66 lines · Peter Zijlstra
Quoted 47 lines · Thomas-Mich Richter
I use linux 5.1.0-rc3 on s390 and got this
WARN_ON_ONCEmessage:WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8which was introduced with
commitcca2094605ef("perf/core: Fixevent_function_local()").This is the
WARN_ON_ONCEmessage, which sometimes shows up in the kernel log:[ 4598.316519] WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8 [ 4598.316524] Kernel panic - not syncing: panic_on_warn set ... [ 4598.316527] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G OE 5.1.0-20190402.rc3.git0.5e7a8ca31926.300.fc29.s390x+git #1 [ 4598.316529] Hardware name: IBM 2964 NC9 712 (LPAR) [ 4598.316531] Call Trace: [ 4598.316534] ([<0000000000112eb8>] show_stack+0x58/0x70) [ 4598.316538] [<0000000000a820aa>] dump_stack+0x7a/0xa8 [ 4598.316541] [<0000000000143b52>] panic+0x11a/0x2d0 [ 4598.316543] [<00000000001439b0>] __warn+0xf8/0x118 [ 4598.316545] [<0000000000a811a8>] report_bug+0xd8/0x150 [ 4598.316547] [<00000000001014ac>] do_report_trap+0xc4/0xe0 [ 4598.316549] [<0000000000101680>] illegal_op+0x138/0x150 [ 4598.316552] [<0000000000aa233c>] pgm_check_handler+0x1cc/0x220 [ 4598.316554] [<00000000002a6d02>] event_function_local.constprop.79+ 0xe2/0xe8 [ 4598.316556] ([<00000000002a6c7a>] event_function_local.constprop.79+ 0x5a/0xe8) [ 4598.316559] [<00000000002aa400>] perf_pending_event+0x88/0xb0 [ 4598.316561] [<0000000000271a8c>] irq_work_run_list+0x8c/0xb8 [ 4598.316563] [<0000000000271d78>] irq_work_tick+0x48/0x68 [ 4598.316566] [<00000000001d6058>] update_process_times+0x68/0x80 [ 4598.316568] [<00000000001e6c10>] tick_sched_handle.isra.6+0x50/0x60 [ 4598.316570] [<00000000001e6c7e>] tick_sched_timer+0x5e/0xb8 [ 4598.316573] [<00000000001d6b6a>] __hrtimer_run_queues+0x10a/0x2c0 [ 4598.316575] [<00000000001d7a88>] hrtimer_interrupt+0x138/0x2a8 [ 4598.316577] [<000000000010c3e4>] do_IRQ+0xac/0xb0 [ 4598.316597] [<0000000000aa2744>] ext_int_handler+0x128/0x12c [ 4598.316600] [<00000000001034f6>] enabled_wait+0x46/0xd0 [ 4598.316602] ([<000003e000d2fe10>] 0x3e000d2fe10) [ 4598.316604] [<0000000000103842>] arch_cpu_idle+0x3a/0x50 [ 4598.316606] [<000000000017b808>] do_idle+0x180/0x1b0 [ 4598.316608] [<000000000017ba06>] cpu_startup_entry+0x36/0x40 [ 4598.316611] [<0000000000115d16>] smp_init_secondary+0xd6/0xf0 [ 4598.316613] [<000000000011521e>] smp_start_secondary+0x86/0x98Quoted 1 line
Any ideas or hints who to avoid/fix this warning?
Some thoughts here:
https://lkml.kernel.org/r/[email protected]
tl;dr, I've no frigging clue.
Quoted 10 lines
I ran a user space program which uses
1.
perf_event_open()system call with type HARDWARE andPERF_COUNT_HW_CPU_CYCLESin sampling mode.
2. Assigns signal SIGIO to the file descriptor returned fromperf_event_open() using: rc = fcntl(fd, F_SETFL, O_RDWR | O_NONBLOCK | O_ASYNC); rc |= fcntl(fd, F_SETSIG, signo); rc |= fcntl(fd, F_SETOWN, getpid());
Quoted 9 lines
Quoted 5 lines
3. The signal handler increments some variables and issues
ioctl(fd, PERF_EVENT_IOC_REFRESH, 1);
to restart signal delivery.I can send you the test program if needed.
Does it reproduce on x86 without virt on? I don't have a PPC LPAR to test things on.
It happens on an s390 mainframe (not PowerPC) without KVM or z/VM virtualisation. I'll try to repro it on x86.
PS: Put my team members Heiko and Hendrik on cc.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Thomas-Mich Richter
I use linux 5.1.0-rc3 on s390 and got this
WARN_ON_ONCEmessage:WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8which was introduced with
commitcca2094605ef("perf/core: Fixevent_function_local()").
..snip....
Quoted 8 lines
Quoted 1 line
Any ideas or hints who to avoid/fix this warning?
Some thoughts here:
https://lkml.kernel.org/r/[email protected]
tl;dr, I've no frigging clue.
I have read this thread and at the end you mentioned:
Humm, but in that case:
context_switch()
prepare_task_switch()
perf_event_task_sched_out()
__perf_event_task_sched_out()
perf_event_context_sched_out()
task_ctx_sched_out()
ctx_sched_out()
group_sched_out()
event_sched_out()
if (event->pending_disable) Would have already cleared the pending_disable state, so the IPI would not have ran perf_event_disable_local() in the first place.
Our test system is configured to panic in WARN_ON_ONCE(). I looked at the dump. The event triggering WARN_ON_ONCE:
crash> struct perf_event.oncpu 0x1f9b24800
oncpu = 0
crash> struct perf_event.state 0x1f9b24800
state = PERF_EVENT_STATE_ACTIVE
crash>
This means the code in
static void event_sched_out(....)
{
....
event->pmu->del(event, 0);
event->oncpu = -1;
if (event->pending_disable) {
event->pending_disable = 0;
state = PERF_EVENT_STATE_OFF;
}
perf_event_set_state(event, state);
...
}has not finished and returned from this function. So the task was not completely context-switched out from CPU 0 while the interrupt handler was executing on CPU 15:
static void perf_pending_event(...)
{
....
if (event->pending_disable) {
event->pending_disable = 0;
perf_event_disable_local(event); <--- Causes the WARN_ON_ONCE()
}
.....
}I think there is a race, especially when the interrupt handler on CPU 15 was invoked via timer interrupt an runs on a different CPU.
Quoted 4 lines
Does it reproduce on x86 without virt on? I don't have a PPC LPAR to test things on.
s390 LPARs run under hipervisor control, no chance to run any OS without it.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 75 lines · Thomas-Mich Richter
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Thomas-Mich Richter
I use linux 5.1.0-rc3 on s390 and got this
WARN_ON_ONCEmessage:WARNING: CPU: 15 PID: 0 at kernel/events/core.c:330 event_function_local.constprop.79+0xe2/0xe8which was introduced with
commitcca2094605ef("perf/core: Fixevent_function_local()").
..snip....Quoted 8 lines
Quoted 1 line
Any ideas or hints who to avoid/fix this warning?
Some thoughts here:
https://lkml.kernel.org/r/[email protected]
tl;dr, I've no frigging clue.
I have read this thread and at the end you mentioned:
Humm, but in that case:
context_switch() prepare_task_switch() perf_event_task_sched_out() __perf_event_task_sched_out() perf_event_context_sched_out() task_ctx_sched_out() ctx_sched_out() group_sched_out() event_sched_out() if (event->pending_disable) Would have already cleared the pending_disable state, so the IPI would not have ran perf_event_disable_local() in the first place.Our test system is configured to panic in
WARN_ON_ONCE(). I looked at the dump. The event triggeringWARN_ON_ONCE:crash> struct perf_event.oncpu
0x1f9b24800
oncpu = 0
crash> struct perf_event.state0x1f9b24800
state =PERF_EVENT_STATE_ACTIVE
crash>This means the code in
static voidevent_sched_out(....){ .... event->pmu->del(event, 0); event->oncpu = -1; if (event->pending_disable) { event->pending_disable = 0; state = PERF_EVENT_STATE_OFF; } perf_event_set_state(event, state); ... }has not finished and returned from this function. So the task was not completely context-switched out from CPU 0 while the interrupt handler was executing on CPU 15:
static void
perf_pending_event(...){ .... if (event->pending_disable) { event->pending_disable = 0; perf_event_disable_local(event); <--- Causes the WARN_ON_ONCE() } ..... }
Quoted 2 lines
I think there is a race, especially when the interrupt handler on CPU 15 was invoked via timer interrupt an runs on a different CPU.
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>
What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);
and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls perf_event_disable_inatomic() _again_ (this would mean irq_work_queue() failing, which we don't check) (and schedule out again, although that's not required).
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Quoted 5 lines
Quoted 3 lines
Does it reproduce on x86 without virt on? I don't have a PPC LPAR to test things on.
s390 LPARs run under hipervisor control, no chance to run any OS without it.
Yah, I know.. Same difference though; I also don't have an s390.
Peter Zijlstra
maintainerQuoted 56 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).
Quoted 4 lines
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Arghh... s390 doesn't implement arch_irq_work_raise(), which makes it far far worse.
I have a hack that might've cured it, were it not for that. Let me think more still..
Peter Zijlstra
maintainerQuoted 60 lines · Peter Zijlstra
Quoted 59 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Quoted 7 lines
Quoted 1 line
Let me think about this a little more...
Arghh... s390 doesn't implement
arch_irq_work_raise(), which makes it far far worse.I have a hack that might've cured it, were it not for that. Let me think more still..
Could you educate me on the s390 PMU, afaict only the SF one has a sampling interrupt (cpumf_measurement_alert), is that NMI-like or a regular IRQ ?
Peter Zijlstra
maintainerQuoted 56 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).
Quoted 4 lines
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Does the below cure things? It's not exactly pretty, but it could just do the trick.
---
kernel/events/core.c +6 -5
@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event, event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ; state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);
@@ -2198,7 +2198,7 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();irq_work_queue (&event->pending); }
@@ -5822,8 +5822,8 @@ static void perf_pending_event(struct irq_work *entry)* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ;perf_event_disable_local (event); }
@@ -10236,6 +10236,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,init_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);
Quoted 104 lines · Peter Zijlstra
Quoted 56 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).Quoted 4 lines
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Does the below cure things? It's not exactly pretty, but it could just do the trick.
---
kernel/events/core.c +5
-5@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event, event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ; state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);@@ -2198,7 +2198,7 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();irq_work_queue (&event->pending); }@@ -5822,8 +5822,8 @@ static void perf_pending_event(struct irq_work *entry)* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ;perf_event_disable_local (event); }
kernel/events/core.c perf_event_alloc(struct perf_event_attr *attr, int cpu, · line 10242
Quoted 7 lines
kernel/events/core.c +1
-0@@ -10236,6 +10236,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,init_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);
Thanks for the quick reply, will give it a try...
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Quoted 58 lines · Peter Zijlstra
Quoted 56 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).
Quoted 8 lines
Quoted 4 lines
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Does the below cure things? It's not exactly pretty, but it could just do the trick.
Thanks a lot for the patch, I have built a new kernel and let it run over the week end.
s390 does not have a PMI, all interrupts (including the measurement interrupts from the PMU) are normal, maskable interrupts.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 64 lines · Thomas-Mich Richter
Quoted 63 lines · Peter Zijlstra
Quoted 60 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Quoted 8 lines
Quoted 3 lines
Does the below cure things? It's not exactly pretty, but it could just do the trick.
Thanks a lot for the patch, I have built a new kernel and let it run over the week end.
s390 does not have a PMI, all interrupts (including the measurement interrupts from the PMU) are normal, maskable interrupts.
Please implement arch_irq_work_raise() for s390 though, traditionally that was only required if you had NMI-like PMis, and IRQ based PMIs would have to run irq_work_run() at the end of their handler.
The s390-sf handler does not do this, but even it if were to do that, it wouldn't be sufficient, since you also drain the buffer from pmu::stop().
Not doing this will get you some 'weird' behavoiur. As in, your signals could be delivered very late etc.
Quoted 104 lines · Peter Zijlstra
Quoted 56 lines · Peter Zijlstra
That is not entirely the scenario I talked about, but *groan*.
So what I meant was:
CPU-0 CPU-n __schedule() local_irq_disable() ... deactivate_task(prev); try_to_wake_up(@p) ... smp_cond_load_acquire(&p->on_cpu, !VAL); <PMI> .. perf_event_disable_inatomic() event->pending_disable = 1; irq_work_queue() /* self-IPI */ </PMI> context_switch() prepare_task_switch() perf_event_task_sched_out() // the above chain that clears pending_disable finish_task_switch() finish_task() smp_store_release(prev->on_cpu, 0); /* finally.... */ // take woken // context_switch to @p finish_lock_switch() raw_spin_unlock_irq() /* w00t, IRQs enabled, self-IPI time */ <self-IPI> perf_pending_event() // event->pending_disable == 0 </self-IPI>What you're suggesting, is that the time between:
smp_store_release(prev->on_cpu, 0);and
<self-IPI>
on CPU-0 is sufficient for CPU-n to context switch to the task, enable the event there, trigger a PMI that calls
perf_event_disable_inatomic()_again_ (this would meanirq_work_queue()failing, which we don't check) (and schedule out again, although that's not required).Quoted 4 lines
This being virt that might actually be possible if (v)CPU-0 takes a nap I suppose.
Let me think about this a little more...
Does the below cure things? It's not exactly pretty, but it could just do the trick.
---
kernel/events/core.c +5
-5@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event, event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ; state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);@@ -2198,7 +2198,7 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();irq_work_queue (&event->pending); }@@ -5822,8 +5822,8 @@ static void perf_pending_event(struct irq_work *entry)* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ;perf_event_disable_local (event); }
kernel/events/core.c perf_event_alloc(struct perf_event_attr *attr, int cpu, · line 10242
Quoted 7 lines
kernel/events/core.c +1
-0@@ -10236,6 +10236,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,init_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);
Peter,
very good news, your fix ran over the weekend without any hit!!!
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 46 lines · Thomas-Mich Richter
Quoted 46 lines
Does the below cure things? It's not exactly pretty, but it could just do the trick.
---
kernel/events/core.c +6
-5@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event, event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ; state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);@@ -2198,7 +2198,7 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();irq_work_queue (&event->pending); }@@ -5822,8 +5822,8 @@ static void perf_pending_event(struct irq_work *entry)* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ;perf_event_disable_local (event); }@@ -10236,6 +10236,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,init_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);
Quoted 8 lines
kernel/events/core.c perf_event_alloc(struct perf_event_attr *attr, int cpu, · line 10242
Quoted 2 lines
kernel/events/core.c +0
-0mutex_init (&event->mmap_mutex);Peter,
very good news, your fix ran over the weekend without any hit!!!
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Could I convince you to implement arch_irq_work_raise() for s390?
Quoted 52 lines · Peter Zijlstra
Quoted 51 lines · Thomas-Mich Richter
Quoted 41 lines
Does the below cure things? It's not exactly pretty, but it could just do the trick.
---
kernel/events/core.c +5
-5@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event, event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ; state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);@@ -2198,7 +2198,7 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();irq_work_queue (&event->pending); }@@ -5822,8 +5822,8 @@ static void perf_pending_event(struct irq_work *entry)* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; +if (event->pending_disable ==smp_processor_id ()) { + event->pending_disable =-1 ;perf_event_disable_local (event); }kernel/events/core.c perf_event_alloc(struct perf_event_attr *attr, int cpu, · line 10242
Quoted 7 lines
kernel/events/core.c +1
-0@@ -10236,6 +10236,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,init_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);Peter,
Quoted 8 lines
Quoted 3 lines
very good news, your fix ran over the weekend without any hit!!!
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Could I convince you to implement
arch_irq_work_raise()for s390?
Yes, I am convinced, however I need to discuss this with the s390 maintainers Martin Schwidesfky and Heiko Carstens first.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 7 lines · Peter Zijlstra
Quoted 3 lines · Thomas-Mich Richter
very good news, your fix ran over the weekend without any hit!!!
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Hi Thomas, find below.
Sadly, while writing the Changelog I ended up with a 'completely' differet patch again, could I bother you to test this one too?
---
Subject: perf: Fix perf_event_disable_inatomic()
From: Peter Zijlstra <[email protected]>
Date: Thu, 4 Apr 2019 15:03:00 +0200
Thomas-Mich Richter reported he triggered a WARN from event_function_local() on his s390. The problem boils down to:
CPU-A CPU-B
perf_event_overflow()
perf_event_disable_inatomic()
@pending_disable = 1
irq_work_queue();sched-out
event_sched_out()
@pending_disable = 0
sched-in
perf_event_overflow()
perf_event_disable_inatomic()
@pending_disable = 1;
irq_work_queue(); // FAILS irq_work_run()
perf_pending_event()
if (@pending_disable)
perf_event_disable_local(); // WHOOPSThe problem exists in generic, but s390 is particularly sensitive because it doesn't implement arch_irq_work_raise(), nor does it call irq_work_run() from it's PMU interrupt handler (nor would that be sufficient in this case, because s390 also generates perf_event_overflow() from pmu::stop). Add to that the fact that s390 is a virtual architecture and (virtual) CPU-A can stall long enough for the above race to happen, even if it would self-IPI.
Adding an irq_work_syn() to event_sched_in() would work for all hardare PMUs that properly use irq_work_run() but fails for software PMUs.
Instead encode the CPU number in @pending_disable, such that we can tell which CPU requested the disable. This then allows us to detect the above scenario and even redirect the IPI to make up for the failed queue.
--- kernel/events/core.c | 52 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-)
kernel/events/core.c +43 -9
@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (READ_ONCE (event->pending_disable) >=0 ) { +WRITE_ONCE (event->pending_disable,-1 ); state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);
@@ -2198,7 +2198,8 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; +WRITE_ONCE (event->pending_disable,smp_processor_id ()); +/* can fail, see perf_pending_event_disable() */ irq_work_queue (&event->pending); }
@@ -5810,10 +5811,45 @@ void perf_event_wakeup(struct perf_event } } +static void perf_pending_event_disable (struct perf_event *event) +{ +int cpu =READ_ONCE (event->pending_disable); + +if (cpu <0 ) +return ; + +if (cpu ==smp_processor_id ()) { +WRITE_ONCE (event->pending_disable,-1 ); +perf_event_disable_local (event); +return ; + } + +/* +* CPU-A CPU-B +* +* perf_event_disable_inatomic() +* @pending_disable = CPU-A; +* irq_work_queue(); +* +* sched-out +* @pending_disable = -1; +* +* sched-in +* perf_event_disable_inatomic() +* @pending_disable = CPU-B; +* irq_work_queue(); // FAILS +* +* irq_work_run() +* perf_pending_event() +* +* But the event runs on CPU-B and wants disabling there. +*/ +irq_work_queue_on (&event->pending, cpu); +} +static void perf_pending_event (struct irq_work *entry) { -struct perf_event *event =container_of (entry, - struct perf_event, pending); + struct perf_event *event = container_of(entry, struct perf_event, pending);int rctx; rctx =perf_swevent_get_recursion_context ();
@@ -5822,10 +5858,7 @@ static void perf_pending_event(struct ir* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; -perf_event_disable_local (event); - } +perf_pending_event_disable (event);if (event->pending_wakeup) { event->pending_wakeup =0 ;
@@ -10236,6 +10269,7 @@ perf_event_alloc(struct perf_event_attrinit_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Thomas-Mich Richter
very good news, your fix ran over the weekend without any hit!!!
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Hi Thomas, find below.
Sadly, while writing the Changelog I ended up with a 'completely' differet patch again, could I bother you to test this one too?
Hi Peter, no problem, test is now running overnight. Let you know the outcome tomorrow...
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Thomas-Mich Richter
Tested-byQuoted 5 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Thomas-Mich Richter
very good news, your fix ran over the weekend without any hit!!!
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Hi Thomas, find below.
Sadly, while writing the Changelog I ended up with a 'completely' differet patch again, could I bother you to test this one too?
Hi Peter,
I verified your second patch, it ran overnight without any message in the kernel log. Thanks again.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 16 lines · Thomas-Mich Richter
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Thomas-Mich Richter
very good news, your fix ran over the weekend without any hit!!!
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Hi Thomas, find below.
Sadly, while writing the Changelog I ended up with a 'completely' differet patch again, could I bother you to test this one too?
Quoted 6 lines
Hi Peter,
I verified your second patch, it ran overnight without any message in the kernel log. Thanks again.
Tested-by Thomas Richter <[email protected]>
Most excellent, thanks for that!
Quoted 52 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Thomas-Mich Richter
very good news, your fix ran over the weekend without any hit!!!
Thanks very much for your help. Do you submit this patch to the kernel mailing list?
Most excellent, let me go write a Changelog.
Hi Thomas, find below.
Sadly, while writing the Changelog I ended up with a 'completely' differet patch again, could I bother you to test this one too?
---
Subject: perf: Fixperf_event_disable_inatomic()
From: Peter Zijlstra <[email protected]>
Date: Thu, 4 Apr 2019 15:03:00 +0200Thomas-Mich Richter reported he triggered a WARN from
event_function_local()on his s390. The problem boils down to:CPU-A CPU-B
perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1 irq_work_queue();sched-out
event_sched_out() @pending_disable = 0 sched-in perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1; irq_work_queue(); // FAILS
irq_work_run()perf_pending_event() if (@pending_disable) perf_event_disable_local(); // WHOOPSThe problem exists in generic, but s390 is particularly sensitive because it doesn't implement
arch_irq_work_raise(), nor does it callirq_work_run()from it's PMU interrupt handler (nor would that be sufficient in this case, because s390 also generatesperf_event_overflow()frompmu::stop). Add to that the fact that s390 is a virtual architecture and (virtual) CPU-A can stall long enough for the above race to happen, even if it would self-IPI.
Quoted 2 lines
Adding an
irq_work_syn()toevent_sched_in()would work for all hardare PMUs that properly useirq_work_run()but fails for software PMUs.
Typo: s/syn/sync/
Quoted 6 lines
Instead encode the CPU number in @pending_disable, such that we can tell which CPU requested the disable. This then allows us to detect the above scenario and even redirect the IPI to make up for the failed queue.
Quoted 9 lines
Cc Heiko Carstens <[email protected]>Cc Kees Cook <[email protected]>Cc Hendrik Brueckner <[email protected]>Cc Martin Schwidefsky <[email protected]>Cc Mark Rutland <[email protected]>Cc Jiri Olsa <[email protected]>Reported-by Thomas-Mich Richter <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
I can't think of a nicer way of handling this, so FWIW:
Mark.
Quoted 95 lines
--- kernel/events/core.c | 52 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-)kernel/events/core.c +43
-9@@ -2009,8 +2009,8 @@ event_sched_out(struct perf_event *event event->pmu->del (event,0 ); event->oncpu =-1 ; -if (event->pending_disable) { - event->pending_disable =0 ; +if (READ_ONCE (event->pending_disable) >=0 ) { +WRITE_ONCE (event->pending_disable,-1 ); state =PERF_EVENT_STATE_OFF ; }perf_event_set_state (event, state);@@ -2198,7 +2198,8 @@ EXPORT_SYMBOL_GPL(perf_event_disable);void perf_event_disable_inatomic (struct perf_event *event) { - event->pending_disable =1 ; +WRITE_ONCE (event->pending_disable,smp_processor_id ()); +/* can fail, see perf_pending_event_disable() */ irq_work_queue (&event->pending); }@@ -5810,10 +5811,45 @@ void perf_event_wakeup(struct perf_event } } +static void perf_pending_event_disable (struct perf_event *event) +{ +int cpu =READ_ONCE (event->pending_disable); + +if (cpu <0 ) +return ; + +if (cpu ==smp_processor_id ()) { +WRITE_ONCE (event->pending_disable,-1 ); +perf_event_disable_local (event); +return ; + } + +/* +* CPU-A CPU-B +* +* perf_event_disable_inatomic() +* @pending_disable = CPU-A; +* irq_work_queue(); +* +* sched-out +* @pending_disable = -1; +* +* sched-in +* perf_event_disable_inatomic() +* @pending_disable = CPU-B; +* irq_work_queue(); // FAILS +* +* irq_work_run() +* perf_pending_event() +* +* But the event runs on CPU-B and wants disabling there. +*/ +irq_work_queue_on (&event->pending, cpu); +} +static void perf_pending_event (struct irq_work *entry) { -struct perf_event *event =container_of (entry, - struct perf_event, pending); + struct perf_event *event = container_of(entry, struct perf_event, pending);int rctx; rctx =perf_swevent_get_recursion_context ();@@ -5822,10 +5858,7 @@ static void perf_pending_event(struct ir* and we won't recurse 'further'. */ -if (event->pending_disable) { - event->pending_disable =0 ; -perf_event_disable_local (event); - } +perf_pending_event_disable (event);if (event->pending_wakeup) { event->pending_wakeup =0 ;@@ -10236,6 +10269,7 @@ perf_event_alloc(struct perf_event_attrinit_waitqueue_head (&event->waitq); + event->pending_disable =-1 ;init_irq_work (&event->pending, perf_pending_event);mutex_init (&event->mmap_mutex);
Quoted 4 lines · Mark Rutland
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
On Mon, Apr 08, 2019 at 09:12:28AM +0200, Thomas-Mich Richter wrote:
.....
Quoted 16 lines
Quoted 6 lines
Instead encode the CPU number in @pending_disable, such that we can tell which CPU requested the disable. This then allows us to detect the above scenario and even redirect the IPI to make up for the failed queue.
Quoted 9 lines
Cc Heiko Carstens <[email protected]>Cc Kees Cook <[email protected]>Cc Hendrik Brueckner <[email protected]>Cc Martin Schwidefsky <[email protected]>Cc Mark Rutland <[email protected]>Cc Jiri Olsa <[email protected]>Reported-by Thomas-Mich Richter <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 7 lines
I can't think of a nicer way of handling this, so FWIW:
Acked-by Mark Rutland <[email protected]>Mark.
Quoted 1 line
---
Thanks for the fix with commit id 86071b11317550d994b55ce5e31aa06bcad783b5.
However doing an fgrep on the pending_disable member of struct perf_event reveals two more hits in file kernel/events/ringbuffer.c when events have auxiliary data.
Not sure if this needs to be addresses too, just wanted to let you know.
Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Peter Zijlstra
maintainerQuoted 7 lines · Thomas-Mich Richter
Thanks for the fix with commit id
86071b11317550d994b55ce5e31aa06bcad783b5.However doing an fgrep on the
pending_disablemember of structperf_eventreveals two more hits in filekernel/events/ringbuffer.cwhen events have auxiliary data.Not sure if this needs to be addresses too, just wanted to let you know.
*groan* indeed, and yes that would need fixing too. I completely missed the AUX stuff using that too.
I think we can simply do the below on top, Alexander?
--- kernel/events/ring_buffer.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/events/ring_buffer.c +2 -2
@@ -392,7 +392,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,* store that will be enabled on successful return */ if (!handle->size) {/* A, matches D */ - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();perf_output_wakeup (handle);local_set (&rb->aux_nest,0 ); goto err_put;
@@ -480,7 +480,7 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size)if (wakeup) {if (handle->aux_flags &PERF_AUX_FLAG_TRUNCATED ) - handle->event->pending_disable =1 ; + handle->event->pending_disable =smp_processor_id ();perf_output_wakeup (handle); }
Alexander Shishkin
reviewerQuoted 7 lines · Peter Zijlstra
Quoted 6 lines · Thomas-Mich Richter
Thanks for the fix with commit id
86071b11317550d994b55ce5e31aa06bcad783b5.However doing an fgrep on the
pending_disablemember of structperf_eventreveals two more hits in filekernel/events/ringbuffer.cwhen events have auxiliary data.
Quoted 6 lines
Quoted 1 line
Not sure if this needs to be addresses too, just wanted to let you know.
*groan* indeed, and yes that would need fixing too. I completely missed the AUX stuff using that too.
I think we can simply do the below on top, Alexander?
Looks good. FWIW, renaming pending_disable in the first place to match the fact that it's a cpu# now, would have probably caught all these.
Quoted 18 lines
--- kernel/events/ring_buffer.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/events/ring_buffer.c +1
-1@@ -392,7 +392,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,* store that will be enabled on successful return */ if (!handle->size) {/* A, matches D */ - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();perf_output_wakeup (handle);local_set (&rb->aux_nest,0 ); goto err_put;
kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size) · line 485
Quoted 7 lines
kernel/events/ring_buffer.c +1
-1@@ -480,7 +480,7 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size)if (wakeup) {if (handle->aux_flags &PERF_AUX_FLAG_TRUNCATED ) - handle->event->pending_disable =1 ; + handle->event->pending_disable =smp_processor_id ();perf_output_wakeup (handle); }
Thanks,
--
Alex