Core2 cpu triggers hard lockup with perf test
Subsystems (3)
hi,
we are getting hard lockups on Core2 cpus (model 23) just by running 'perf test'
PID: 10425 TASK: ffff880068562e00 CPU: 3 COMMAND: "perf"
#0 [ffff88007d985a08] machine_kexec at ffffffff8105521b #1 [ffff88007d985a68] crash_kexec at ffffffff810f7412
#2 [ffff88007d985b38] panic at ffffffff8163c031
#3 [ffff88007d985bb8] watchdog_overflow_callback at ffffffff81120472 #4 [ffff88007d985bc8] __perf_event_overflow at ffffffff81164e0e #5 [ffff88007d985c00] perf_event_overflow at ffffffff81165a44 #6 [ffff88007d985c10] intel_pmu_handle_irq at ffffffff81033198 #7 [ffff88007d985e60] perf_event_nmi_handler at ffffffff8164be8b #8 [ffff88007d985e80] nmi_handle at ffffffff8164b5d9
#9 [ffff88007d985ec8] do_nmi at ffffffff8164b789
#10 [ffff88007d985ef0] end_repeat_nmi at ffffffff8164aa13
[exception RIP: intel_pmu_enable_all+17]
RIP: ffffffff81032301 RSP: ffff88005e917c98 RFLAGS: 00000046
RAX: ffff88007d98cd20 RBX: ffff88005e991000 RCX: 000000000000038f
RDX: 0000000000000007 RSI: 0000000000000003 RDI: 0000000000000000
RBP: ffff88005e917cd8 R8: ffffffffffffff85 R9: 000000ffffffffff
R10: ffff88007d98c100 R11: ffff88005e9179e0 R12: ffff88007d98bd10
R13: ffff88007d98b9e0 R14: ffff88007d98bc08 R15: 0000000000000002
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018--- <NMI exception stack> ---
#11 [ffff88005e917c98] intel_pmu_enable_all at ffffffff81032301 #12 [ffff88005e917c98] x86_pmu_enable at ffffffff8102ba24 #13 [ffff88005e917ce0] perf_pmu_enable at ffffffff81160457 #14 [ffff88005e917cf0] perf_event_context_sched_in at ffffffff81161930 #15 [ffff88005e917d20] perf_event_exec at ffffffff811621db #16 [ffff88005e917d68] setup_new_exec at ffffffff811edffd #17 [ffff88005e917d88] load_elf_binary at ffffffff81240ed9 #18 [ffff88005e917e58] search_binary_handler at ffffffff811ec89d #19 [ffff88005e917ea0] do_execve_common at ffffffff811ede04 #20 [ffff88005e917f30] sys_execve at ffffffff811ee199
#21 [ffff88005e917f50] stub_execve at ffffffff816531a9
the reproducer seems to be hw event with very small period like (thanks Arnaldo ;-):
perf record -e cycles -c 123 kill
I bisected it down to the:
156174999dd1 perf/intel/x86: Enlarge the PEBS buffer
Looks like the bigger PEBS buffer together with event being marked as PERF_X86_EVENT_FREERUNNING will block the CPU right after the event is enabled before it could reach local_irq_enable and trigger the NMI watchdog.
I can't find what's special about Core2 CPU PEBS setup, it seems that oher CPUs are ok (tried on ivb/snb/hsw).
reverting the 156174999dd1 fixed the issue for me
ideas? thanks,
jirka
Peter Zijlstra
maintainerQuoted 8 lines · Jiri Olsa
we are getting hard lockups on Core2 cpus (model 23)
I can't find what's special about Core2 CPU PEBS setup, it seems that oher CPUs are ok (tried on ivb/snb/hsw).
reverting the
156174999dd1fixed the issue for meideas? thanks,
The obvious difference between Core2 and later chips is that Core2 only has PEBS on a single counter (cnt0).
I'll try and have a closer look on Monday, I should still have a Core2 class machine around the house somewhere.
Quoted 4 lines
I can't find what's special about Core2 CPU PEBS setup, it seems that oher CPUs are ok (tried on ivb/snb/hsw).
reverting the
156174999dd1fixed the issue for me
Ok multi-record PEbS was never tested on Core 2. I suppose we can enable it only on Nehalem+
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines
I can't find what's special about Core2 CPU PEBS setup, it seems that oher CPUs are ok (tried on ivb/snb/hsw).
reverting the
156174999dd1fixed the issue for meideas? thanks,
I think we may just disable the multiple pebs support for core2 as the patch below.
In SDM "18.4.4.4 Re-configuring PEBS Facilities" it mentioned that a quiescent period is needed between stopping the prior event counting and setting up a new PEBS event when software needs to reconfigure PEBS facilities. The quiescent period is to allow any latent residual PEBS records to complete its capture at their previously specified buffer address
That requirement only can be found in Core Microarchitecture.
I think it may implies that there is some observed delay in writing PEBS buffer. So if perf record precise hw event with very small period, the slow PEBS writing may lockup the CPU. If so, I think disabling the multiple pebs should be a good way.
--- arch/x86/events/intel/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +2 -2
@@ -2577,8 +2577,8 @@ static int intel_pmu_hw_config(struct perf_event *event)if (event->attr.precise_ip) {if (!event->attr.freq) { event->hw.flags |=PERF_X86_EVENT_AUTO_RELOAD ; -if (!(event->attr.sample_type & - ~intel_pmu_free_running_flags (event))) +if ((x86_pmu.intel_cap.pebs_format >0 ) && + !(event->attr.sample_type & ~intel_pmu_free_running_flags (event))) event->hw.flags |=PERF_X86_EVENT_FREERUNNING ; }if (x86_pmu.pebs_aliases)
Signature · 4 lines
2.5.0 Thanks, Kan
Quoted 20 lines · Kan
Quoted 9 lines
I can't find what's special about Core2 CPU PEBS setup, it seems that oher CPUs are ok (tried on ivb/snb/hsw).
reverting the
156174999dd1fixed the issue for meideas? thanks,
I think we may just disable the multiple pebs support for core2 as the patch below.
In SDM "18.4.4.4 Re-configuring PEBS Facilities" it mentioned that a quiescent period is needed between stopping the prior event counting and setting up a new PEBS event when software needs to reconfigure PEBS facilities. The quiescent period is to allow any latent residual PEBS records to complete its capture at their previously specified buffer address
That requirement only can be found in Core Microarchitecture.
Quoted 6 lines
I think it may implies that there is some observed delay in writing PEBS buffer. So if perf record precise hw event with very small period, the slow PEBS writing may lockup the CPU. If so, I think disabling the multiple pebs should be a good way.
hi,
got same lockup with the patch:
[ 167.486514] Kernel panic - not syncing: Hard LOCKUP [ 167.486514] CPU: 3 PID: 10656 Comm: perf Not tainted 4.5.0-rc4+ #7 [ 167.486514] Hardware name: System Manufacturer To Be Filled By O.E.M. Product Name To Be Filled By O.E.M./BB Name To be filled by O.E.M., BIOS CGELIA55.86 [ 167.486514] 0000000000000086 0000000084986595 ffff88007d985b28 ffffffff8133983f [ 167.486514] ffffffff8191b723 0000000000000000 ffff88007d985ba8 ffffffff811872d1 [ 167.486514] ffff880000000008 ffff88007d985bb8 ffff88007d985b58 0000000084986595 [ 167.486514] Call Trace: [ 167.486514] <NMI> [<ffffffff8133983f>] dump_stack+0x63/0x84 [ 167.486514] [<ffffffff811872d1>] panic+0xe2/0x229 [ 167.486514] [<ffffffff8113dc30>] watchdog_overflow_callback+0x100/0x100 [ 167.486514] [<ffffffff8117ee18>] __perf_event_overflow+0x88/0x1c0 [ 167.486514] [<ffffffff8117f994>] perf_event_overflow+0x14/0x20 [ 167.486514] [<ffffffff8100c42f>] intel_pmu_handle_irq+0x1df/0x460 [ 167.486514] [<ffffffff81052e3f>] ? native_apic_wait_icr_idle+0x1f/0x30 [ 167.486514] [<ffffffff81032cc5>] ? arch_irq_work_raise+0x35/0x40 [ 167.486514] [<ffffffff8100563d>] perf_event_nmi_handler+0x2d/0x50 [ 167.486514] [<ffffffff810313a2>] nmi_handle+0x62/0xf0 [ 167.486514] [<ffffffff81031a06>] default_do_nmi+0xf6/0x120 [ 167.486514] [<ffffffff81031b11>] do_nmi+0xe1/0x150 [ 167.486514] [<ffffffff816ad5f1>] end_repeat_nmi+0x1a/0x1e [ 167.486514] [<ffffffff81063a16>] ? native_write_msr_safe+0x6/0x30 [ 167.486514] [<ffffffff81063a16>] ? native_write_msr_safe+0x6/0x30 [ 167.486514] [<ffffffff81063a16>] ? native_write_msr_safe+0x6/0x30 [ 167.486514] <<EOE>> [<ffffffff8100b5cd>] ? __intel_pmu_enable_all.isra.12+0x4d/0xb0 [ 167.486514] [<ffffffff8100b640>] intel_pmu_enable_all+0x10/0x20 [ 167.486514] [<ffffffff810072c3>] x86_pmu_enable+0x263/0x2f0 [ 167.486514] [<ffffffff81179a72>] perf_pmu_enable+0x22/0x30 [ 167.486514] [<ffffffff8117a721>] ctx_resched+0x51/0x60 [ 167.486514] [<ffffffff8117b2ff>] perf_event_exec+0x10f/0x140 [ 167.486514] [<ffffffff8121949d>] setup_new_exec+0x6d/0x1a0 [ 167.486514] [<ffffffff8126b58a>] load_elf_binary+0x37a/0x10e0 [ 167.486514] [<ffffffff811b77f2>] ? get_user_pages+0x52/0x60 [ 167.486514] [<ffffffff8121779e>] search_binary_handler+0x9e/0x1e0 [ 167.486514] [<ffffffff812191f4>] do_execveat_common.isra.34+0x554/0x6e0 [ 167.486514] [<ffffffff8121960a>] SyS_execve+0x3a/0x50 [ 167.486514] [<ffffffff816ab195>] stub_execve+0x5/0x5 [ 167.486514] [<ffffffff816aaeee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
jirka
Peter Zijlstra
maintainerQuoted 7 lines · Kan
In SDM "18.4.4.4 Re-configuring PEBS Facilities" it mentioned that a quiescent period is needed between stopping the prior event counting and setting up a new PEBS event when software needs to reconfigure PEBS facilities. The quiescent period is to allow any latent residual PEBS records to complete its capture at their previously specified buffer address
That requirement only can be found in Core Microarchitecture.
But that should apply to all (PEBS) event scheduling, not just the multi thing.
Also very convenient that quiescent period is so well defined. How long should we wait, a day?
Quoted 1 line
I think it may implies that there is some observed delay in writing PEBS buffer.
Doesn't it explicitly state just that?
Quoted 2 lines
So if perf record precise hw event with very small period, the slow PEBS writing may lockup the CPU.
And I still don't see how this would explain a lockup in the MSR writes.
[ Jiri, can you disable that stupid panic on hard lockup and let it run for a while, see if all the lockup msgs hit the same IP? Also, can you look where exactly that IP lives in the code? ]
So I suspect it actually just did the PERF_GLOBAL_CTRL write, how else would the hardware watchdog trigger on that same CPU.
After that, there's only BTS muck, which you're not using, so WTH is it actually stuck on?
Quoted 1 line
If so, I think disabling the multiple pebs should be a good way.
As said, this should affect any and all PEBS event scheduling, not just the multi stuff.
Quoted 20 lines · Peter Zijlstra
Quoted 7 lines · Kan
In SDM "18.4.4.4 Re-configuring PEBS Facilities" it mentioned that a quiescent period is needed between stopping the prior event counting and setting up a new PEBS event when software needs to reconfigure PEBS facilities. The quiescent period is to allow any latent residual PEBS records to complete its capture at their previously specified buffer address
That requirement only can be found in Core Microarchitecture.
But that should apply to all (PEBS) event scheduling, not just the multi thing.
Also very convenient that quiescent period is so well defined. How long should we wait, a day?
Quoted 1 line
I think it may implies that there is some observed delay in writing PEBS buffer.
Doesn't it explicitly state just that?
Quoted 8 lines
Quoted 2 lines
So if perf record precise hw event with very small period, the slow PEBS writing may lockup the CPU.
And I still don't see how this would explain a lockup in the MSR writes.
[ Jiri, can you disable that stupid panic on hard lockup and let it run for a while, see if all the lockup msgs hit the same IP? Also, can you look where exactly that IP lives in the code? ]
im on it.. also the patch that makes this happen just enlarge the buffer for PEBS:
156174999dd1 perf/intel/x86: Enlarge the PEBS buffer
but I did not find anyaPEBS buffer lenght limitations in SDM
jirka
Quoted 11 lines
So I suspect it actually just did the
PERF_GLOBAL_CTRLwrite, how else would the hardware watchdog trigger on that same CPU.After that, there's only BTS muck, which you're not using, so WTH is it actually stuck on?
Quoted 1 line
If so, I think disabling the multiple pebs should be a good way.
As said, this should affect any and all PEBS event scheduling, not just the multi stuff.
Peter Zijlstra
maintainerQuoted 6 lines · Jiri Olsa
Quoted 4 lines
[ Jiri, can you disable that stupid panic on hard lockup and let it run for a while, see if all the lockup msgs hit the same IP? Also, can you look where exactly that IP lives in the code? ]
im on it..
Thanks!
Quoted 6 lines
also the patch that makes this happen just
enlarge the buffer for PEBS:
156174999dd1perf/intel/x86: Enlarge the PEBS bufferbut I did not find anyaPEBS buffer lenght limitations in SDM
Probably just makes it easier to tickle.
Quoted 6 lines
im on it.. also the patch that makes this happen just enlarge the buffer for PEBS:
156174999dd1perf/intel/x86: Enlarge the PEBS bufferbut I did not find anyaPEBS buffer lenght limitations in SDM
May be the easiest would be to just keep the old buffer size on Merom.
-Andi
Peter Zijlstra
maintainerQuoted 9 lines · Andi Kleen
Quoted 6 lines
im on it.. also the patch that makes this happen just enlarge the buffer for PEBS:
156174999dd1perf/intel/x86: Enlarge the PEBS bufferbut I did not find anyaPEBS buffer lenght limitations in SDM
May be the easiest would be to just keep the old buffer size on Merom.
I would still like to know why it breaks though. Just changing code about because it makes fail go away is never the right answer.
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Andi Kleen
Quoted 5 lines
im on it.. also the patch that makes this happen just enlarge the buffer for PEBS:
156174999dd1perf/intel/x86: Enlarge the PEBS buffer
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
but I did not find anyaPEBS buffer lenght limitations in SDM
May be the easiest would be to just keep the old buffer size on Merom.
I would still like to know why it breaks though. Just changing code about because it makes fail go away is never the right answer.
I had to go throught several config switch offs,
however now it's pretty clear where it hangs
I got one (just one occurance):
[ 125.982977] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1^M [ 125.982977] Modules linked in: rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ocrdma ib_core snd_hda_codec_idt snd_hda_codec_generic ib_addr snd_hda_intel snd_hda_codec snd_hda_core coretemp kvm_intel snd_hwdep snd_seq kvm snd_seq_device snd_pcm snd_timer snd nfsd soundcore shpchp pcspkr auth_rpcgss ppdev nfs_acl sg irqbypass parport_serial parport_pc lockd i2c_i801 parport acpi_cpufreq grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom i915 video i2c_algo_bit ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e drm ptp pps_core i2c_core lpfc libata scsi_transport_fc be2net vxlan ip6_udp_tunnel udp_tunnel fjes dm_mirror dm_region_hash dm_log dm_mod^M [ 125.982977] CPU: 1 PID: 5586 Comm: perf Not tainted 4.5.0-rc4fix+ #10^M [ 125.982977] Hardware name: System Manufacturer To Be Filled By O.E.M. Product Name To Be Filled By O.E.M./BB Name To be filled by O.E.M., BIOS CGELIA55.86^M [ 125.982977] 0000000000000086 00000000522d7515 ffff88007d885bb0 ffffffff8131dd11^M [ 125.982977] 0000000000000000 0000000000000000 ffff88007d885bc8 ffffffff811279cb^M [ 125.982977] ffff88007a548000 ffff88007d885c00 ffffffff81168618 0000000000000002^M [ 125.982977] Call Trace:^M [ 125.982977] <NMI> [<ffffffff8131dd11>] dump_stack+0x4d/0x6c^M [ 125.982977] [<ffffffff811279cb>] watchdog_overflow_callback+0xeb/0x100^M [ 125.982977] [<ffffffff81168618>] __perf_event_overflow+0x88/0x1c0^M [ 125.982977] [<ffffffff81169194>] perf_event_overflow+0x14/0x20^M [ 125.982977] [<ffffffff8100b656>] intel_pmu_handle_irq+0x1e6/0x4a0^M [ 125.982977] [<ffffffff810413bf>] ? native_apic_wait_icr_idle+0x1f/0x30^M [ 125.982977] [<ffffffff81021d15>] ? arch_irq_work_raise+0x35/0x40^M [ 125.982977] [<ffffffff8115c54e>] ? irq_work_queue+0x6e/0x80^M [ 125.982977] [<ffffffff8100466d>] perf_event_nmi_handler+0x2d/0x50^M [ 125.982977] [<ffffffff81020382>] nmi_handle+0x62/0xf0^M [ 125.982977] [<ffffffff810209d6>] default_do_nmi+0xe6/0x110^M [ 125.982977] [<ffffffff81020ad7>] do_nmi+0xd7/0x140^M [ 125.982977] [<ffffffff8167af37>] end_repeat_nmi+0x1a/0x1e^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] <<EOE>> [<ffffffff8100af10>] intel_pmu_enable_all+0x10/0x20^M [ 125.982977] [<ffffffff81006283>] x86_pmu_enable+0x263/0x2f0^M [ 125.982977] [<ffffffff811632d2>] perf_pmu_enable+0x22/0x30^M [ 125.982977] [<ffffffff81163f51>] ctx_resched+0x51/0x60^M [ 125.982977] [<ffffffff81164b09>] perf_event_exec+0x109/0x150^M [ 125.982977] [<ffffffff811fff7d>] setup_new_exec+0x6d/0x1a0^M [ 125.982977] [<ffffffff8125104a>] load_elf_binary+0x37a/0x10e0^M [ 125.982977] [<ffffffff811a06c2>] ? get_user_pages+0x52/0x60^M [ 125.982977] [<ffffffff811fe32e>] search_binary_handler+0x9e/0x1e0^M [ 125.982977] [<ffffffff811ffccd>] do_execveat_common.isra.37+0x54d/0x6e0^M [ 125.982977] [<ffffffff812000ea>] SyS_execve+0x3a/0x50^M [ 125.982977] [<ffffffff81679065>] stub_execve+0x5/0x5^M [ 125.982977] [<ffffffff81678dd7>] ? entry_SYSCALL_64_fastpath+0x12/0x6a^M
and several rcu stalls.. I guess that's because above path did rcu_read_lock:
[ 166.667008] INFO: rcu_sched detected stalls on CPUs/tasks:^M [ 166.668003] 1-...: (16 GPs behind) idle=b13/140000000000000/0 softirq=6921/6923 fqs=19790 ^M [ 166.774020] (detected by 2, t=60168 jiffies, g=6522, c=6521, q=0)^M [ 166.774020] Task dump for CPU 1:^M [ 166.774020] perf R running task 0 5586 5585 0x00000088^M [ 166.774020] 0000000000000001 ffff88007144fd70 ffffffff81164b09 ffff88007d899808^M [ 166.774020] 0000000000000286 ffff88007d899808 ffff880076f8e7c0 ffff880079734200^M [ 166.774020] ffff880071450000 0000000000000000 ffff880078dfc500 ffff88007144fd90^M [ 166.774020] Call Trace:^M [ 166.774020] [<ffffffff81164b09>] ? perf_event_exec+0x109/0x150^M [ 166.774020] [<ffffffff811fff7d>] ? setup_new_exec+0x6d/0x1a0^M [ 166.774020] [<ffffffff8125104a>] ? load_elf_binary+0x37a/0x10e0^M [ 166.774020] [<ffffffff811a06c2>] ? get_user_pages+0x52/0x60^M [ 166.774020] [<ffffffff811fe32e>] ? search_binary_handler+0x9e/0x1e0^M [ 166.774020] [<ffffffff811ffccd>] ? do_execveat_common.isra.37+0x54d/0x6e0^M [ 166.774020] [<ffffffff812000ea>] ? SyS_execve+0x3a/0x50^M [ 166.774020] [<ffffffff81679065>] ? stub_execve+0x5/0x5^M [ 166.774020] [<ffffffff81678dd7>] ? entry_SYSCALL_64_fastpath+0x12/0x6a^M [ 346.672008] INFO: rcu_sched detected stalls on CPUs/tasks:^M
the exception addr is on wrmsr:
ffffffff8100ae30 <__intel_pmu_enable_all.isra.11>:
ffffffff8100ae30: e8 bb 02 67 00 callq ffffffff8167b0f0 <__fentry__> ffffffff8100ae35: 55 push %rbp ffffffff8100ae36: 48 89 e5 mov %rsp,%rbp ffffffff8100ae39: 41 54 push %r12 ffffffff8100ae3b: 41 89 fc mov %edi,%r12d ffffffff8100ae3e: 53 push %rbx ffffffff8100ae3f: 48 c7 c3 80 a3 00 00 mov $0xa380,%rbx ffffffff8100ae46: 65 48 03 1d d2 f2 ff add %gs:0x7efff2d2(%rip),%rbx # a120 <this_cpu_off> ffffffff8100ae4d: 7e ffffffff8100ae4e: e8 6d 49 00 00 callq ffffffff8100f7c0 <intel_pmu_pebs_enable_all> ffffffff8100ae53: 41 0f b6 fc movzbl %r12b,%edi ffffffff8100ae57: e8 94 58 00 00 callq ffffffff810106f0 <intel_pmu_lbr_enable_all> ffffffff8100ae5c: 48 8b 83 68 0c 00 00 mov 0xc68(%rbx),%rax ffffffff8100ae63: b9 8f 03 00 00 mov $0x38f,%ecx ffffffff8100ae68: 48 f7 d0 not %rax ffffffff8100ae6b: 48 23 05 26 80 ad 00 and 0xad8026(%rip),%rax # ffffffff81ae2e98 <x86_pmu+0x138> ffffffff8100ae72: 48 89 c2 mov %rax,%rdx ffffffff8100ae75: 48 c1 ea 20 shr $0x20,%rdx ffffffff8100ae79: 0f 30 wrmsr
I tried what Andy suggested below (not sure what he meant by Merom, I took PEBS format0 instead), works for me
jirka
---
arch/x86/events/intel/ds.c +4 -2
@@ -269,7 +269,7 @@ static int alloc_pebs_buffer(int cpu)if (!x86_pmu.pebs)return 0 ; - buffer =kzalloc_node (PEBS_BUFFER_SIZE ,GFP_KERNEL , node); + buffer =kzalloc_node (x86_pmu.pebs_buffer_size,GFP_KERNEL , node);if (unlikely (!buffer))return -ENOMEM ;
@@ -286,7 +286,7 @@ static int alloc_pebs_buffer(int cpu)per_cpu (insn_buffer, cpu) = ibuffer; } - max =PEBS_BUFFER_SIZE / x86_pmu.pebs_record_size; + max = x86_pmu.pebs_buffer_size / x86_pmu.pebs_record_size; ds->pebs_buffer_base = (u64 )(unsigned long )buffer; ds->pebs_index = ds->pebs_buffer_base;
@@ -1319,6 +1319,7 @@ void __init intel_ds_init(void) x86_pmu.bts =boot_cpu_has (X86_FEATURE_BTS ); x86_pmu.pebs =boot_cpu_has (X86_FEATURE_PEBS ); + x86_pmu.pebs_buffer_size =PEBS_BUFFER_SIZE ;if (x86_pmu.pebs) {char pebs_type = x86_pmu.intel_cap.pebs_trap ?'+' :'-' ;int format = x86_pmu.intel_cap.pebs_format;
@@ -1327,6 +1328,7 @@ void __init intel_ds_init(void)case 0 :pr_cont ("PEBS fmt0%c, " , pebs_type); x86_pmu.pebs_record_size =sizeof (struct pebs_record_core ); + x86_pmu.pebs_buffer_size =PAGE_SIZE ; x86_pmu.drain_pebs = intel_pmu_drain_pebs_core;break ;
arch/x86/events/perf_event.h +1 -0
@@ -586,6 +586,7 @@ struct x86_pmu { pebs_broken :1 , pebs_prec_dist :1;int pebs_record_size; +int pebs_buffer_size;void (*drain_pebs)(struct pt_regs *regs);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);
Andi Kleen
Reviewed-byQuoted 2 lines
I tried what Andy suggested below (not sure what he meant by Merom, I took PEBS format0 instead), works for me
Thanks Jiri. Patch looks good to me.
We may want to make this buffer size a configurable anyways because with multi-record PEBS it can make sense to use much larger buffers to reduce overhead. But that could be done separately.
-Andi
Peter Zijlstra
maintainerQuoted 31 lines · Jiri Olsa
[ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] <<EOE>> [<ffffffff8100af10>] intel_pmu_enable_all+0x10/0x20^M [ 125.982977] [<ffffffff81006283>] x86_pmu_enable+0x263/0x2f0^M [ 125.982977] [<ffffffff811632d2>] perf_pmu_enable+0x22/0x30^M [ 125.982977] [<ffffffff81163f51>] ctx_resched+0x51/0x60^M [ 125.982977] [<ffffffff81164b09>] perf_event_exec+0x109/0x150^M [ 125.982977] [<ffffffff811fff7d>] setup_new_exec+0x6d/0x1a0^M [ 125.982977] [<ffffffff8125104a>] load_elf_binary+0x37a/0x10e0^M [ 125.982977] [<ffffffff811a06c2>] ? get_user_pages+0x52/0x60^M [ 125.982977] [<ffffffff811fe32e>] search_binary_handler+0x9e/0x1e0^M [ 125.982977] [<ffffffff811ffccd>] do_execveat_common.isra.37+0x54d/0x6e0^M [ 125.982977] [<ffffffff812000ea>] SyS_execve+0x3a/0x50^M [ 125.982977] [<ffffffff81679065>] stub_execve+0x5/0x5^M [ 125.982977] [<ffffffff81678dd7>] ? entry_SYSCALL_64_fastpath+0x12/0x6a^Mthe exception addr is on wrmsr:
ffffffff8100ae30<__intel_pmu_enable_all.isra.11>:ffffffff8100ae30: e8 bb 02 67 00 callq ffffffff8167b0f0 <__fentry__> ffffffff8100ae35: 55 push %rbp ffffffff8100ae36: 48 89 e5 mov %rsp,%rbp ffffffff8100ae39: 41 54 push %r12 ffffffff8100ae3b: 41 89 fc mov %edi,%r12d ffffffff8100ae3e: 53 push %rbx ffffffff8100ae3f: 48 c7 c3 80 a3 00 00 mov $0xa380,%rbx ffffffff8100ae46: 65 48 03 1d d2 f2 ff add %gs:0x7efff2d2(%rip),%rbx # a120 <this_cpu_off> ffffffff8100ae4d: 7e ffffffff8100ae4e: e8 6d 49 00 00 callq ffffffff8100f7c0 <intel_pmu_pebs_enable_all> ffffffff8100ae53: 41 0f b6 fc movzbl %r12b,%edi
Quoted 9 lines
ffffffff8100ae57: e8 94 58 00 00 callq ffffffff810106f0 <intel_pmu_lbr_enable_all> ffffffff8100ae5c: 48 8b 83 68 0c 00 00 mov 0xc68(%rbx),%rax ffffffff8100ae63: b9 8f 03 00 00 mov $0x38f,%ecx ffffffff8100ae68: 48 f7 d0 not %rax ffffffff8100ae6b: 48 23 05 26 80 ad 00 and 0xad8026(%rip),%rax # ffffffff81ae2e98 <x86_pmu+0x138> ffffffff8100ae72: 48 89 c2 mov %rax,%rdx ffffffff8100ae75: 48 c1 ea 20 shr $0x20,%rdx ffffffff8100ae79: 0f 30 wrmsr
That's the PERF_GLOBAL_CTRL, right? But it must have succeeded, otherwise the NMI watchdog would never have fired.
Something is hosed alright.
I think I've seen my IVB-EP do something similar. But mostly that machine gets stuck in intel_bts_enable_local().
Quoted 43 lines · Peter Zijlstra
Quoted 31 lines · Jiri Olsa
[ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M [ 125.982977] <<EOE>> [<ffffffff8100af10>] intel_pmu_enable_all+0x10/0x20^M [ 125.982977] [<ffffffff81006283>] x86_pmu_enable+0x263/0x2f0^M [ 125.982977] [<ffffffff811632d2>] perf_pmu_enable+0x22/0x30^M [ 125.982977] [<ffffffff81163f51>] ctx_resched+0x51/0x60^M [ 125.982977] [<ffffffff81164b09>] perf_event_exec+0x109/0x150^M [ 125.982977] [<ffffffff811fff7d>] setup_new_exec+0x6d/0x1a0^M [ 125.982977] [<ffffffff8125104a>] load_elf_binary+0x37a/0x10e0^M [ 125.982977] [<ffffffff811a06c2>] ? get_user_pages+0x52/0x60^M [ 125.982977] [<ffffffff811fe32e>] search_binary_handler+0x9e/0x1e0^M [ 125.982977] [<ffffffff811ffccd>] do_execveat_common.isra.37+0x54d/0x6e0^M [ 125.982977] [<ffffffff812000ea>] SyS_execve+0x3a/0x50^M [ 125.982977] [<ffffffff81679065>] stub_execve+0x5/0x5^M [ 125.982977] [<ffffffff81678dd7>] ? entry_SYSCALL_64_fastpath+0x12/0x6a^Mthe exception addr is on wrmsr:
ffffffff8100ae30<__intel_pmu_enable_all.isra.11>:ffffffff8100ae30: e8 bb 02 67 00 callq ffffffff8167b0f0 <__fentry__> ffffffff8100ae35: 55 push %rbp ffffffff8100ae36: 48 89 e5 mov %rsp,%rbp ffffffff8100ae39: 41 54 push %r12 ffffffff8100ae3b: 41 89 fc mov %edi,%r12d ffffffff8100ae3e: 53 push %rbx ffffffff8100ae3f: 48 c7 c3 80 a3 00 00 mov $0xa380,%rbx ffffffff8100ae46: 65 48 03 1d d2 f2 ff add %gs:0x7efff2d2(%rip),%rbx # a120 <this_cpu_off> ffffffff8100ae4d: 7e ffffffff8100ae4e: e8 6d 49 00 00 callq ffffffff8100f7c0 <intel_pmu_pebs_enable_all> ffffffff8100ae53: 41 0f b6 fc movzbl %r12b,%ediQuoted 9 lines
ffffffff8100ae57: e8 94 58 00 00 callq ffffffff810106f0 <intel_pmu_lbr_enable_all> ffffffff8100ae5c: 48 8b 83 68 0c 00 00 mov 0xc68(%rbx),%rax ffffffff8100ae63: b9 8f 03 00 00 mov $0x38f,%ecx ffffffff8100ae68: 48 f7 d0 not %rax ffffffff8100ae6b: 48 23 05 26 80 ad 00 and 0xad8026(%rip),%rax # ffffffff81ae2e98 <x86_pmu+0x138> ffffffff8100ae72: 48 89 c2 mov %rax,%rdx ffffffff8100ae75: 48 c1 ea 20 shr $0x20,%rdx ffffffff8100ae79: 0f 30 wrmsr
Quoted 1 line
That's the
PERF_GLOBAL_CTRL, right? But it must have succeeded,
yep, should be this one:
static void __intel_pmu_enable_all(int added, bool pmi)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_pebs_enable_all();
intel_pmu_lbr_enable_all(pmi);Quoted 1 line
Quoted 1 line
Quoted 1 line
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
Quoted 1 line
otherwise the NMI watchdog would never have fired.
so NMI wouldn't trigger if CPU is inside wrmsr?
jirka
Quoted 7 lines
Something is hosed alright.
I think I've seen my IVB-EP do something similar. But mostly that machine gets stuck in
intel_bts_enable_local().
Peter Zijlstra
maintainerQuoted 10 lines · Jiri Olsa
Quoted 1 line
That's the
PERF_GLOBAL_CTRL, right? But it must have succeeded,yep, should be this one:
static void
__intel_pmu_enable_all(int added, bool pmi){ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); intel_pmu_pebs_enable_all(); intel_pmu_lbr_enable_all(pmi);
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);Quoted 1 line
otherwise the NMI watchdog would never have fired.
so NMI wouldn't trigger if CPU is inside wrmsr?
Well, anything goes with MSR writes, that's all a magic heap of micro-code.
But at the very least it did actually enable the counters, otherwise the counter used for the NMI watchdog could not fire, it too would still be disabled.
Peter Zijlstra
maintainerQuoted 2 lines · Jiri Olsa
I tried what Andy suggested below (not sure what he meant by Merom, I took PEBS format0 instead), works for me
Model 15, see intel_pmu_init(). But you're actually running on a Penryn I suspect, since we disabled PEBS for Merom.
There's also a bunch of Atoms that uses PEBS format 0, no idea if they're affected too.
Quoted 32 lines
---
arch/x86/events/intel/ds.c +3
-2@@ -269,7 +269,7 @@ static int alloc_pebs_buffer(int cpu)if (!x86_pmu.pebs)return 0 ; - buffer =kzalloc_node (PEBS_BUFFER_SIZE ,GFP_KERNEL , node); + buffer =kzalloc_node (x86_pmu.pebs_buffer_size,GFP_KERNEL , node);if (unlikely (!buffer))return -ENOMEM ;@@ -286,7 +286,7 @@ static int alloc_pebs_buffer(int cpu)per_cpu (insn_buffer, cpu) = ibuffer; } - max =PEBS_BUFFER_SIZE / x86_pmu.pebs_record_size; + max = x86_pmu.pebs_buffer_size / x86_pmu.pebs_record_size; ds->pebs_buffer_base = (u64 )(unsigned long )buffer; ds->pebs_index = ds->pebs_buffer_base;@@ -1319,6 +1319,7 @@ void __init intel_ds_init(void) x86_pmu.bts =boot_cpu_has (X86_FEATURE_BTS ); x86_pmu.pebs =boot_cpu_has (X86_FEATURE_PEBS ); + x86_pmu.pebs_buffer_size =PEBS_BUFFER_SIZE ;if (x86_pmu.pebs) {char pebs_type = x86_pmu.intel_cap.pebs_trap ?'+' :'-' ;int format = x86_pmu.intel_cap.pebs_format;
arch/x86/events/intel/ds.c void __init intel_ds_init(void) · line 1330
Quoted 3 lines
arch/x86/events/intel/ds.c +0
-0@@ -1327,6 +1328,7 @@ void __init intel_ds_init(void)case 0 :pr_cont ("PEBS fmt0%c, " , pebs_type); x86_pmu.pebs_record_size =sizeof (struct pebs_record_core );
At the very least this wants a comment along the lines of:
/*
* Using >PAGE_SIZE buffers makes the WRMSR to
* PERF_GLOBAL_CTRL in intel_pmu_enable_all()
* mysteriously hang on Core2.
*
* As a workaround, we don't do this. */
Quoted 16 lines
arch/x86/events/intel/ds.c +1
-0+ x86_pmu.pebs_buffer_size =PAGE_SIZE ; x86_pmu.drain_pebs = intel_pmu_drain_pebs_core;break ;arch/x86/events/perf_event.h +1
-0@@ -586,6 +586,7 @@ struct x86_pmu { pebs_broken :1 , pebs_prec_dist :1;int pebs_record_size; +int pebs_buffer_size;void (*drain_pebs)(struct pt_regs *regs);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);
PATCH 1/1 perf x86: Use PAGE_SIZE for PEBS buffer size on Core2
On Tue, Mar 01, 2016 at 07:12:07PM +0100, Peter Zijlstra wrote:
SNIP
Quoted 19 lines
Review void __init intel_ds_init(void) · line 1330
Quoted 4 lines
(quoted diff) +0
-0@@ -1327,6 +1328,7 @@ void __init intel_ds_init(void) case 0: pr_cont("PEBS fmt0%c, ", pebs_type); x86_pmu.pebs_record_size = sizeof(struct pebs_record_core);At the very least this wants a comment along the lines of:
/*
* Using >PAGE_SIZE buffers makes the WRMSR to
*PERF_GLOBAL_CTRLinintel_pmu_enable_all()
* mysteriously hang on Core2.
*
* As a workaround, we don't do this. */Quoted 4 lines
(quoted diff) +1
-0+ x86_pmu.pebs_buffer_size = PAGE_SIZE; x86_pmu.drain_pebs = intel_pmu_drain_pebs_core; break;
Quoted 12 lines
arch/x86/events/perf_event.h struct x86_pmu { · line 592
Quoted 12 lines
arch/x86/events/perf_event.h +1
-0@@ -586,6 +586,7 @@ struct x86_pmu { pebs_broken :1 , pebs_prec_dist :1;int pebs_record_size; +int pebs_buffer_size;void (*drain_pebs)(struct pt_regs *regs);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);
sending updated patch
jirka
---
Using PAGE_SIZE buffers makes the WRMSR toPERF_GLOBAL_CTRL in intel_pmu_enable_all()
mysteriously hang on Core2. As a workaround,
we don't do this.
The hard lockup is easily triggered by running 'perf test attr' repeatedly. Most of the time it gets stuck on sample session with small periods.
# perf test attr -vv
14: struct perf_event_attr setup :
--- start ---
...
'PERF_TEST_ATTR=/tmp/tmpuEKz3B /usr/bin/perf record -o /tmp/tmpuEKz3B/perf.data -c 123 kill >/dev/null 2>&1' ret 1--- arch/x86/events/intel/ds.c | 13 +++++++++++-- arch/x86/events/perf_event.h | 1 + 2 files changed, 12 insertions(+), 2 deletions(-)
arch/x86/events/intel/ds.c +11 -2
@@ -269,7 +269,7 @@ static int alloc_pebs_buffer(int cpu)if (!x86_pmu.pebs)return 0 ; - buffer =kzalloc_node (PEBS_BUFFER_SIZE ,GFP_KERNEL , node); + buffer =kzalloc_node (x86_pmu.pebs_buffer_size,GFP_KERNEL , node);if (unlikely (!buffer))return -ENOMEM ;
@@ -286,7 +286,7 @@ static int alloc_pebs_buffer(int cpu)per_cpu (insn_buffer, cpu) = ibuffer; } - max =PEBS_BUFFER_SIZE / x86_pmu.pebs_record_size; + max = x86_pmu.pebs_buffer_size / x86_pmu.pebs_record_size; ds->pebs_buffer_base = (u64 )(unsigned long )buffer; ds->pebs_index = ds->pebs_buffer_base;
@@ -1319,6 +1319,7 @@ void __init intel_ds_init(void) x86_pmu.bts =boot_cpu_has (X86_FEATURE_BTS ); x86_pmu.pebs =boot_cpu_has (X86_FEATURE_PEBS ); + x86_pmu.pebs_buffer_size =PEBS_BUFFER_SIZE ;if (x86_pmu.pebs) {char pebs_type = x86_pmu.intel_cap.pebs_trap ?'+' :'-' ;int format = x86_pmu.intel_cap.pebs_format;
@@ -1327,6 +1328,14 @@ void __init intel_ds_init(void)case 0 :pr_cont ("PEBS fmt0%c, " , pebs_type); x86_pmu.pebs_record_size =sizeof (struct pebs_record_core ); +/* +* Using >PAGE_SIZE buffers makes the WRMSR to +* PERF_GLOBAL_CTRL in intel_pmu_enable_all() +* mysteriously hang on Core2. +* +* As a workaround, we don't do this. +*/ + x86_pmu.pebs_buffer_size =PAGE_SIZE ; x86_pmu.drain_pebs = intel_pmu_drain_pebs_core;break ;
arch/x86/events/perf_event.h +1 -0
@@ -586,6 +586,7 @@ struct x86_pmu { pebs_broken :1 , pebs_prec_dist :1;int pebs_record_size; +int pebs_buffer_size;void (*drain_pebs)(struct pt_regs *regs);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);
Signature · 1 line
2.4.3
tip-bot for Jiri Olsa
merge bot[tip:perf/core] perf/x86/intel: Use PAGE_SIZE for PEBS buffer size on Core2
perf/x86/intel: Use PAGE_SIZE for PEBS buffer size on Core2
Commit-ID: e72daf3f4d764c47fb71c9bdc7f9c54a503825b1 Gitweb: http://git.kernel.org/tip/e72daf3f4d764c47fb71c9bdc7f9c54a503825b1 Author: Jiri Olsa <[email protected]> AuthorDate: Tue, 1 Mar 2016 20:03:52 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 8 Mar 2016 12:18:32 +0100
perf/x86/intel: Use PAGE_SIZE for PEBS buffer size on Core2
Using PAGE_SIZE buffers makes the WRMSR to PERF_GLOBAL_CTRL in intel_pmu_enable_all() mysteriously hang on Core2. As a workaround, we don't do this.
The hard lockup is easily triggered by running 'perf test attr' repeatedly. Most of the time it gets stuck on sample session with small periods.
# perf test attr -vv
14: struct perf_event_attr setup :
--- start ---
...
'PERF_TEST_ATTR=/tmp/tmpuEKz3B /usr/bin/perf record -o /tmp/tmpuEKz3B/perf.data -c 123 kill >/dev/null 2>&1' ret 1--- arch/x86/events/intel/ds.c | 13 +++++++++++-- arch/x86/events/perf_event.h | 1 + 2 files changed, 12 insertions(+), 2 deletions(-)
arch/x86/events/intel/ds.c +11 -2
@@ -269,7 +269,7 @@ static int alloc_pebs_buffer(int cpu)if (!x86_pmu.pebs)return 0 ; - buffer =kzalloc_node (PEBS_BUFFER_SIZE ,GFP_KERNEL , node); + buffer =kzalloc_node (x86_pmu.pebs_buffer_size,GFP_KERNEL , node);if (unlikely (!buffer))return -ENOMEM ;
@@ -286,7 +286,7 @@ static int alloc_pebs_buffer(int cpu)per_cpu (insn_buffer, cpu) = ibuffer; } - max =PEBS_BUFFER_SIZE / x86_pmu.pebs_record_size; + max = x86_pmu.pebs_buffer_size / x86_pmu.pebs_record_size; ds->pebs_buffer_base = (u64 )(unsigned long )buffer; ds->pebs_index = ds->pebs_buffer_base;
@@ -1319,6 +1319,7 @@ void __init intel_ds_init(void) x86_pmu.bts =boot_cpu_has (X86_FEATURE_BTS ); x86_pmu.pebs =boot_cpu_has (X86_FEATURE_PEBS ); + x86_pmu.pebs_buffer_size =PEBS_BUFFER_SIZE ;if (x86_pmu.pebs) {char pebs_type = x86_pmu.intel_cap.pebs_trap ?'+' :'-' ;int format = x86_pmu.intel_cap.pebs_format;
@@ -1327,6 +1328,14 @@ void __init intel_ds_init(void)case 0 :pr_cont ("PEBS fmt0%c, " , pebs_type); x86_pmu.pebs_record_size =sizeof (struct pebs_record_core ); +/* +* Using >PAGE_SIZE buffers makes the WRMSR to +* PERF_GLOBAL_CTRL in intel_pmu_enable_all() +* mysteriously hang on Core2. +* +* As a workaround, we don't do this. +*/ + x86_pmu.pebs_buffer_size =PAGE_SIZE ; x86_pmu.drain_pebs = intel_pmu_drain_pebs_core;break ;
arch/x86/events/perf_event.h +1 -0
@@ -586,6 +586,7 @@ struct x86_pmu { pebs_broken :1 , pebs_prec_dist :1;int pebs_record_size; +int pebs_buffer_size;void (*drain_pebs)(struct pt_regs *regs);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);