Skip to contentSkip to conversation

perf fixes

GIT PULL Ingo Molnar 3 participants 13 messages Feb 26, 2013 – Mar 15, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest perf-urgent-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf-urgent-for-linus

HEAD: ff1fb5f6b4925a536ffb8171e5f2dbd01ccfeb97 Merge branch 'tip/perf/core' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace into perf/urgent

Thanks,

Ingo

------------------>
Stephane Eranian (1):
perf/x86: Add Intel IvyBridge event scheduling constraints

Steven Rostedt (1):
tracing/syscalls: Allow archs to ignore tracing compat syscalls

Steven Rostedt (Red Hat) (1):
ftrace: Call ftrace cleanup module notifier after all other notifiers

 arch/x86/include/asm/ftrace.h          | 24 ++++++++++++++++++
 arch/x86/include/asm/thread_info.h     |  1 -
 arch/x86/kernel/cpu/perf_event_intel.c | 23 ++++++++++++++++-
 kernel/trace/ftrace.c                  | 46 +++++++++++++++++++++++-----------
 kernel/trace/trace_syscalls.c          | 43 +++++++++++++++++++++++++++----
 5 files changed, 116 insertions(+), 21 deletions(-)
arch/x86/include/asm/ftrace.h +24 -0
index 86cb51e..0525a8b 100644
@@ -72,4 +72,28 @@ int ftrace_int3_handler(struct pt_regs *regs);
 #endif /* __ASSEMBLY__ */
 #endif /* CONFIG_FUNCTION_TRACER */
 
+
+#if !defined(__ASSEMBLY__) && !defined(COMPILE_OFFSETS)
+
+#if defined(CONFIG_FTRACE_SYSCALLS) && defined(CONFIG_IA32_EMULATION)
+#include <asm/compat.h>
+
+/*
+ * Because ia32 syscalls do not map to x86_64 syscall numbers
+ * this screws up the trace output when tracing a ia32 task.
+ * Instead of reporting bogus syscalls, just do not trace them.
+ *
+ * If the user realy wants these, then they should use the
+ * raw syscall tracepoints with filtering.
+ */
+#define ARCH_TRACE_IGNORE_COMPAT_SYSCALLS 1
+static inline bool arch_trace_is_compat_syscall(struct pt_regs *regs)
+{
+	if (is_compat_task())
+		return true;
+	return false;
+}
+#endif /* CONFIG_FTRACE_SYSCALLS && CONFIG_IA32_EMULATION */
+#endif /* !__ASSEMBLY__  && !COMPILE_OFFSETS */
+
 #endif /* _ASM_X86_FTRACE_H */
arch/x86/include/asm/thread_info.h +0 -1
index 2d946e6..2cd056e 100644
@@ -20,7 +20,6 @@
 struct task_struct;
 struct exec_domain;
 #include <asm/processor.h>
-#include <asm/ftrace.h>
 #include <linux/atomic.h>
 
 struct thread_info {
arch/x86/kernel/cpu/perf_event_intel.c +22 -1
index 4914e94..529c893 100644
@@ -107,6 +107,27 @@ static struct event_constraint intel_snb_event_constraints[] __read_mostly =
 	EVENT_CONSTRAINT_END
 };
 
+static struct event_constraint intel_ivb_event_constraints[] __read_mostly =
+{
+	FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */
+	FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */
+	FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */
+	INTEL_UEVENT_CONSTRAINT(0x0148, 0x4), /* L1D_PEND_MISS.PENDING */
+	INTEL_UEVENT_CONSTRAINT(0x0279, 0xf), /* IDQ.EMTPY */
+	INTEL_UEVENT_CONSTRAINT(0x019c, 0xf), /* IDQ_UOPS_NOT_DELIVERED.CORE */
+	INTEL_UEVENT_CONSTRAINT(0x04a3, 0xf), /* CYCLE_ACTIVITY.CYCLES_NO_EXECUTE */
+	INTEL_UEVENT_CONSTRAINT(0x05a3, 0xf), /* CYCLE_ACTIVITY.STALLS_L2_PENDING */
+	INTEL_UEVENT_CONSTRAINT(0x06a3, 0xf), /* CYCLE_ACTIVITY.STALLS_LDM_PENDING */
+	INTEL_UEVENT_CONSTRAINT(0x08a3, 0x4), /* CYCLE_ACTIVITY.CYCLES_L1D_PENDING */
+	INTEL_UEVENT_CONSTRAINT(0x0ca3, 0x4), /* CYCLE_ACTIVITY.STALLS_L1D_PENDING */
+	INTEL_UEVENT_CONSTRAINT(0x01c0, 0x2), /* INST_RETIRED.PREC_DIST */
+	INTEL_EVENT_CONSTRAINT(0xd0, 0xf), /* MEM_UOPS_RETIRED.* */
+	INTEL_EVENT_CONSTRAINT(0xd1, 0xf), /* MEM_LOAD_UOPS_RETIRED.* */
+	INTEL_EVENT_CONSTRAINT(0xd2, 0xf), /* MEM_LOAD_UOPS_LLC_HIT_RETIRED.* */
+	INTEL_EVENT_CONSTRAINT(0xd3, 0xf), /*  MEM_LOAD_UOPS_LLC_MISS_RETIRED.* */
+	EVENT_CONSTRAINT_END
+};
+
 static struct extra_reg intel_westmere_extra_regs[] __read_mostly =
 {
 	INTEL_EVENT_EXTRA_REG(0xb7, MSR_OFFCORE_RSP_0, 0xffff, RSP_0),
@@ -2095,7 +2116,7 @@ __init int intel_pmu_init(void)
 
 		intel_pmu_lbr_init_snb();
 
-		x86_pmu.event_constraints = intel_snb_event_constraints;
+		x86_pmu.event_constraints = intel_ivb_event_constraints;
 		x86_pmu.pebs_constraints = intel_ivb_pebs_event_constraints;
 		x86_pmu.pebs_aliases = intel_pebs_aliases_snb;
 		x86_pmu.extra_regs = intel_snb_extra_regs;
kernel/trace/ftrace.c +32 -14

78 lines not on this page · raw message

kernel/trace/trace_syscalls.c +38 -5

80 lines not on this page · raw message

#1 lore raw
Quoted 3 lines · Ingo Molnar

Stephane Eranian (1):
perf/x86: Add Intel IvyBridge event scheduling constraints

So thanks to my new Chromebook, I'm trying to do perf stuff on IvyBridge. I was hoping the whole exact cycle counting would work better, since the last machine it was reliable for me was my old Westmere desktop, and my previous laptop (Sandybridge MacBook Air) didn't work very well due to CPU errata.

Sadly, my new Ivybridge laptop doesn't seem to do very well either, but I suspect it's the kernel, not the hardware this time. A simple

perf record -f -e cycles:pp make -j

in the kernel directory (fully built kernel, so the load doesn't actually go crazy despite the "-j") will *sometimes* totally hang the machine with current git.

I get a NULL pointer dereference in intel_pmu_enable_all, so I don't think it's a hardware bug.

And to make things interesting, I seem to be able to only reproduce this *after* a suspend cycle. That may be just happenstance, since it seemed to be hard to replicate and most of the time it has happened under X with no messages visible at all, but that *seems* to be the pattern.

And the one time I got it to happen on the text console, things scrolled off (watchdog warnings due to lockups), but I did get a NULL pointer dereference in intel_pmu_enable_all().

I'll try to reproduce it and get a picture, but thought I'd send out the email with just this, since maybe the "suspend cycle" plus "NULL pointer in intel_pmu_enable_all()" makes somebody go "Duh!", or at least replicate it...

Linus

#2 lore raw
Quoted 7 lines · Linus Torvalds

And to make things interesting, I seem to be able to only reproduce this *after* a suspend cycle. That may be just happenstance, since it seemed to be hard to replicate and most of the time it has happened under X with no messages visible at all, but that *seems* to be the pattern.

Quoted 5 lines

And the one time I got it to happen on the text console, things scrolled off (watchdog warnings due to lockups), but I did get a NULL pointer dereference in intel_pmu_enable_all().

I'll try to reproduce it and get a picture,

Theory more or less confirmed.

It does need a suspend/resume cycle, and I have a picture. The oops happens immediately when trying to do any perf work after the first suspend, before suspending I seem to be able to reliably use perf. It could still be just random flakiness, but I don't think so.

The NULL pointer dereference is at intel_pmu_enable_all+0x4d/0xa0 for me, which seems to be the load of the

    if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))

thing. It says

   BUG: unable to handle NULL pointer dereference at 0000000000000028

But that error makes no sense. The code at that EIP is

48 8b 83 00 02 00 00 mov 0x200(%rbx),%rax <-- trapping instruction

and the value printed out for %rbx is 0xffff80014f20b8e0, so it should *not* be a NULL pointer dereference (and "cpuc" was also used just before the wrmsrl).

So I suspect that the "wrmsrl" that was just before that instruction does something odd, and the PMU is in some odd state, so that the NULL pointer dereference actually has something to do with *that*, rather than the instruction itself.

The callchain looks normal. It's

finish_task_switch ->

    __perf_event_task_sched_in ->
      perf_event_context_sched_in ->
        perf_pmu_enable ->
          x86_pmu_enable ->
            intel_pmu_enable_all()

The immediately preceding wrmsrl was done with rax=0xf, rdx=0x7, rcx=0x38f according to the register dump (but the picture isn't great, so the numbers aren't 100% reliable).

Does this give any clues?

Linus

#3 lore raw

Hi,

Quoted 14 lines · Linus Torvalds
Quoted 11 lines · Linus Torvalds

And to make things interesting, I seem to be able to only reproduce this *after* a suspend cycle. That may be just happenstance, since it seemed to be hard to replicate and most of the time it has happened under X with no messages visible at all, but that *seems* to be the pattern.

And the one time I got it to happen on the text console, things scrolled off (watchdog warnings due to lockups), but I did get a NULL pointer dereference in intel_pmu_enable_all().

Quoted 9 lines
Quoted 1 line

I'll try to reproduce it and get a picture,

Theory more or less confirmed.

It does need a suspend/resume cycle, and I have a picture. The oops happens immediately when trying to do any perf work after the first suspend, before suspending I seem to be able to reliably use perf. It could still be just random flakiness, but I don't think so.

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

But still don't see how a wrmsrl could corrupt a cpuc.

Quoted 34 lines

The NULL pointer dereference is at intel_pmu_enable_all+0x4d/0xa0 for me, which seems to be the load of the

    if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))

thing. It says

   BUG: unable to handle NULL pointer dereference at 0000000000000028

But that error makes no sense. The code at that EIP is

48 8b 83 00 02 00 00 mov 0x200(%rbx),%rax <-- trapping instruction

and the value printed out for %rbx is 0xffff80014f20b8e0, so it should *not* be a NULL pointer dereference (and "cpuc" was also used just before the wrmsrl).

So I suspect that the "wrmsrl" that was just before that instruction does something odd, and the PMU is in some odd state, so that the NULL pointer dereference actually has something to do with *that*, rather than the instruction itself.

The callchain looks normal. It's

finish_task_switch ->

    __perf_event_task_sched_in ->
      perf_event_context_sched_in ->
        perf_pmu_enable ->
          x86_pmu_enable ->
            intel_pmu_enable_all()
Quoted 4 lines

The immediately preceding wrmsrl was done with rax=0xf, rdx=0x7, rcx=0x38f according to the register dump (but the picture isn't great, so the numbers aren't 100% reliable).

Value 0x38f for GLOBAL_CTRL is valid. And 0x70000000f is valid too for the counter bitmask (4 generic counters + 3 fixed counters).

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

Quoted 3 lines

Does this give any clues?

Linus

#4 lore raw
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

So I'd assume it should happen with any IvyBridge.

Linus

#5 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

Quoted 9 lines

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Ok, simpler test case then.

Quoted 8 lines
Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

So I'd assume it should happen with any IvyBridge.

Will try on a desktop IvyBridge too.

#6 lore raw
Quoted 23 lines · Stephane Eranian
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

Quoted 9 lines

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Ok, simpler test case then.

Quoted 3 lines
Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

Quoted 6 lines
Quoted 5 lines

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

So I'd assume it should happen with any IvyBridge.

Will try on a desktop IvyBridge too.

Ok, it happens on my IVB desktop too, so I can investigate...

#7 lore raw
Quoted 24 lines · Stephane Eranian
Quoted 23 lines · Stephane Eranian
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

Quoted 9 lines

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Ok, simpler test case then.

Quoted 3 lines
Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

So I'd assume it should happen with any IvyBridge.

Will try on a desktop IvyBridge too.

Ok, it happens on my IVB desktop too, so I can investigate...

It's not specific to IVB either, it hangs on my Nehalem desktop as well.

#8 lore raw
Quoted 28 lines · Stephane Eranian
Quoted 27 lines · Stephane Eranian
Quoted 26 lines · Stephane Eranian
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

Quoted 9 lines

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Ok, simpler test case then.

Quoted 6 lines
Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So I'd assume it should happen with any IvyBridge.

Will try on a desktop IvyBridge too.

Ok, it happens on my IVB desktop too, so I can investigate...

It's not specific to IVB either, it hangs on my Nehalem desktop as well.

Looks related to PEBS. If I drop the :pp the machine does not hang. Even a single :p hangs it. So it is possible something is not properly restored in the
DS state after a resume or is corrupted by the suspend.

#9 lore raw

Linus,

I bet if you force the affinity of your perf record to be on a CPU other than CPU0, you will not get the crash.

This is what I am seeing now. I appears on resume, CPU0 hotplug callbacks for perf_events are not invoked leaving DS_AREA MSR to 0.

Can you confirm on your machine?

Quoted 41 lines · Stephane Eranian
Quoted 28 lines · Stephane Eranian
Quoted 27 lines · Stephane Eranian
Quoted 26 lines · Stephane Eranian
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Stephane Eranian

Could be related to suspend/resume. But were you running perf across that resume/suspend cycle?

No.

Quoted 9 lines

In most cases I was running a perf record before and after (but not *while* suspending)

In at least one other crash, I didn't run perf before at all, so the first time I used perf was after the resume.

So in no cases did I actually have any perf stuff active over the suspend itself.

Ok, simpler test case then.

Quoted 6 lines
Quoted 2 lines

Let's see if we can reproduce the problem on the same ChromeBook you have. Don't have one myself.

I don't imagine it should be about chromebook per se, because afaik all of pmu suspend/resume is done by the kernel, no firmware involved.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So I'd assume it should happen with any IvyBridge.

Will try on a desktop IvyBridge too.

Ok, it happens on my IVB desktop too, so I can investigate...

It's not specific to IVB either, it hangs on my Nehalem desktop as well.

Looks related to PEBS. If I drop the :pp the machine does not hang. Even a single :p hangs it. So it is possible something is not properly restored in the
DS state after a resume or is corrupted by the suspend.

#10 lore raw
Quoted 9 lines · Stephane Eranian

I bet if you force the affinity of your perf record to be on a CPU other than CPU0, you will not get the crash.

This is what I am seeing now. I appears on resume, CPU0 hotplug callbacks for perf_events are not invoked leaving DS_AREA MSR to 0.

Can you confirm on your machine?

I'm not even going to bother confirming it, because I think you're right, and I think the reason is clear: the DS initialization code uses the CPU_UP notifiers.

And that's sufficient for CPU hotplug, which is what suspend/resume ends up doing for all but the boot CPU. But the boot CPU is not hotplugged.

Using CPU_UP notifiers is wrong, and they get called too late anyway.

The code should use a real resume method. Or, better yet, just do it right, and do it from __restore_processor_state().

Those f*cking CPU notifiers are a pain in the ass, and the tend to be invariably broken, and they have their own idiotic hacks that are equally broken (ie that x86_pmu_notifier() thing seems to make up its own suspend/resume with "x86_pmu.cpu_prepare/cpu_starting/cpu_dying/cpu_dead" things.

I guess we could make the BP do a fake cpu notifier thing around the suspend of the boot processor as well, but most of the per-CPU stuff seems to be perfectly fine without it (ie mtrr, apic, etc etc all use the suspend/resume infrastructure) and doesn't need that kind of stuff.

Linus

#11 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 8 lines · Stephane Eranian

I bet if you force the affinity of your perf record to be on a CPU other than CPU0, you will not get the crash.

This is what I am seeing now. I appears on resume, CPU0 hotplug callbacks for perf_events are not invoked leaving DS_AREA MSR to 0.

Quoted 6 lines
Quoted 1 line

Can you confirm on your machine?

I'm not even going to bother confirming it, because I think you're right, and I think the reason is clear: the DS initialization code uses the CPU_UP notifiers.

Ok, I instrumented the pebs_enable() function and I confirm that DS_AREA=0 on resume.

So what seems broken here for me is that on suspend, the cpu notifier ends up calling fini_debug_store() to clear DS_AREA for CPU0. But on resume, the same notifier does NOT call the init_debug_store(). I don't understand this asymmetry. You either do neither or you do both.

Quoted 22 lines

And that's sufficient for CPU hotplug, which is what suspend/resume ends up doing for all but the boot CPU. But the boot CPU is not hotplugged.

Using CPU_UP notifiers is wrong, and they get called too late anyway.

The code should use a real resume method. Or, better yet, just do it right, and do it from __restore_processor_state().

Those f*cking CPU notifiers are a pain in the ass, and the tend to be invariably broken, and they have their own idiotic hacks that are equally broken (ie that x86_pmu_notifier() thing seems to make up its own suspend/resume with "x86_pmu.cpu_prepare/cpu_starting/cpu_dying/cpu_dead" things.

I guess we could make the BP do a fake cpu notifier thing around the suspend of the boot processor as well, but most of the per-CPU stuff seems to be perfectly fine without it (ie mtrr, apic, etc etc all use the suspend/resume infrastructure) and doesn't need that kind of stuff.

Linus

#12 lore raw
Quoted 17 lines · Stephane Eranian
Quoted 9 lines · Linus Torvalds
Quoted 8 lines · Stephane Eranian

I bet if you force the affinity of your perf record to be on a CPU other than CPU0, you will not get the crash.

This is what I am seeing now. I appears on resume, CPU0 hotplug callbacks for perf_events are not invoked leaving DS_AREA MSR to 0.

Quoted 6 lines
Quoted 1 line

Can you confirm on your machine?

I'm not even going to bother confirming it, because I think you're right, and I think the reason is clear: the DS initialization code uses the CPU_UP notifiers.

Quoted 9 lines

Ok, I instrumented the pebs_enable() function and I confirm that DS_AREA=0 on resume.

So what seems broken here for me is that on suspend, the cpu notifier ends up calling fini_debug_store() to clear DS_AREA for CPU0. But on resume, the same notifier does NOT call the init_debug_store(). I don't understand this asymmetry. You either do neither or you do both.

Ok, corrections. I ran some more tests. On the suspend path, the cpu notifier is not called for CPU0. However when the machine comes back up, DS_AREA is 0 which is the power-up default value. And given that the notifier is not called for CPU0, that is the value we inherit later on and which causes the crash.

Quoted 10 lines
Quoted 10 lines

And that's sufficient for CPU hotplug, which is what suspend/resume ends up doing for all but the boot CPU. But the boot CPU is not hotplugged.

Using CPU_UP notifiers is wrong, and they get called too late anyway.

The code should use a real resume method. Or, better yet, just do it right, and do it from __restore_processor_state().

I will produce a patch to use this function, it's simple enough.

Quoted 13 lines
Quoted 13 lines

Those f*cking CPU notifiers are a pain in the ass, and the tend to be invariably broken, and they have their own idiotic hacks that are equally broken (ie that x86_pmu_notifier() thing seems to make up its own suspend/resume with "x86_pmu.cpu_prepare/cpu_starting/cpu_dying/cpu_dead" things.

I guess we could make the BP do a fake cpu notifier thing around the suspend of the boot processor as well, but most of the per-CPU stuff seems to be perfectly fine without it (ie mtrr, apic, etc etc all use the suspend/resume infrastructure) and doesn't need that kind of stuff.

Linus