perf, ftrace and MCEs
Subsystems (4)
Borislav Petkov
maintainerHi,
so I finally had some spare time to stare at perf/ftrace code and ponder on how to use those facilities for MCE collecting and reporting. Btw, I have to say, it took me quite a while to understand what goes where - my suggestion to anyone who tries to understand how perf/ftrace works is to do make <file.i> where there is at least one trace_XXX emit record function call and start untangling code paths from there.
So anyway, here are some questions I had, I just as well may've missed something so please correct me if I'm wrong:
1. Since machine checks can happen at any time, we need to have the MCE tracepoint (trace_mce_record() in <include/trace/events/mce.h>) always enabled. This, in turn, means that we need the ftrace/perf infrastructure always compiled in (lockless ring buffer, perf_event.c stuff) on any x86 system so that MCEs can be handled at anytime. Is this going to be ok to be enabled on _all_ machines, hmmm... I dunno, maybe only a subset of those facilites at least.
2. Tangential to 1., we need that "thin" software layer prepared for decoding and reporting them as early as possible. event_trace_init() is an fs_initcall and executed too late, IMHO. The ->perf_event_enable in the ftrace_event_call is enabled even later on the perf init path over the sys_perf_even_open which is at userspace time. In our case, this is going be executed by the error logging and decoding daemon I guess.
3. Since we want to listen for MCEs all the time, the concept of enabling and disabling those events does not apply in the sense of performance profiling. IOW, MCEs need to be able to be logged to the ring buffer at any time. I guess this is easily done - we simply enable MCE events at the earliest moment possible and disable them on shutdown; done.
So yeah, some food for thought but what do you guys think?
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Steven Rostedt
maintainerQuoted 12 lines · Borislav Petkov
Hi,
so I finally had some spare time to stare at perf/ftrace code and ponder on how to use those facilities for MCE collecting and reporting. Btw, I have to say, it took me quite a while to understand what goes where - my suggestion to anyone who tries to understand how perf/ftrace works is to do make <file.i> where there is at least one
trace_XXXemit record function call and start untangling code paths from there.So anyway, here are some questions I had, I just as well may've missed something so please correct me if I'm wrong:
Quoted 7 lines
1. Since machine checks can happen at any time, we need to have the MCE tracepoint (
trace_mce_record()in <include/trace/events/mce.h>) always enabled. This, in turn, means that we need the ftrace/perf infrastructure always compiled in (lockless ring buffer,perf_event.cstuff) on any x86 system so that MCEs can be handled at anytime. Is this going to be ok to be enabled on _all_ machines, hmmm... I dunno, maybe only a subset of those facilites at least.
I'm not exactly sure what you goal is, but if you need to do something directly, you can bypass ftrace and perf. All trace events can be connected by anything even when ftrace and perf are not enabled.
That is, you need to connect to the tracepoint and write you own callback. This can be done pretty much at anytime during boot up. To see how to connect to a trace point, you can look at register_trace_sched_switch() in kernel/trace/ftrace.c. This registers a callback to the trace_sched_switch() trace point in sched.c.
Quoted 8 lines
2. Tangential to 1., we need that "thin" software layer prepared for decoding and reporting them as early as possible.
event_trace_init()is anfs_initcalland executed too late, IMHO. The ->perf_event_enable in theftrace_event_callis enabled even later on the perf init path over thesys_perf_even_openwhich is at userspace time. In our case, this is going be executed by the error logging and decoding daemon I guess.
Quoted 6 lines
3. Since we want to listen for MCEs all the time, the concept of enabling and disabling those events does not apply in the sense of performance profiling. IOW, MCEs need to be able to be logged to the ring buffer at any time. I guess this is easily done - we simply enable MCE events at the earliest moment possible and disable them on shutdown; done.
This looks like a good reason to have your own handler. More than one callback may be registered to a tracepoint, so you do not need to worry about having other handlers affect your code.
-- Steve
Quoted 5 lines
So yeah, some food for thought but what do you guys think?
Thanks.
Borislav Petkov
maintainerFrom: Steven Rostedt <[email protected]>
Date: Mon, May 03, 2010 at 10:41:12AM -0400
Hi Steven,
Quoted 21 lines · Borislav Petkov
Quoted 12 lines
Hi,
so I finally had some spare time to stare at perf/ftrace code and ponder on how to use those facilities for MCE collecting and reporting. Btw, I have to say, it took me quite a while to understand what goes where - my suggestion to anyone who tries to understand how perf/ftrace works is to do make <file.i> where there is at least one
trace_XXXemit record function call and start untangling code paths from there.So anyway, here are some questions I had, I just as well may've missed something so please correct me if I'm wrong:
Quoted 7 lines
1. Since machine checks can happen at any time, we need to have the MCE tracepoint (
trace_mce_record()in <include/trace/events/mce.h>) always enabled. This, in turn, means that we need the ftrace/perf infrastructure always compiled in (lockless ring buffer,perf_event.cstuff) on any x86 system so that MCEs can be handled at anytime. Is this going to be ok to be enabled on _all_ machines, hmmm... I dunno, maybe only a subset of those facilites at least.
Quoted 3 lines
I'm not exactly sure what you goal is, but if you need to do something directly, you can bypass ftrace and perf. All trace events can be connected by anything even when ftrace and perf are not enabled.
Right, so the idea is to use the perf/ftrace infrastructure to detect failing hardware which is signalled through machine checks, among others. I'm thinking a lockless ring buffer would be cool so we can execute in any context... wait a minute, right, we have that already.
However, if I use the perf/ftrace facilities, I have to have them enabled on every system since machine checks are core processor functionality and the software support for those has to be always enabled. And the code has to be small and execute fast since after a critical mcheck happens all bets are off.
So I'm thinking maybe a core perf/ftrace stuff which is thin and is always enabled... but I'm not sure for I haven't stared at the code enough yet.
Quoted 5 lines
That is, you need to connect to the tracepoint and write you own callback. This can be done pretty much at anytime during boot up. To see how to connect to a trace point, you can look at
register_trace_sched_switch()inkernel/trace/ftrace.c. This registers a callback to thetrace_sched_switch()trace point insched.c.
The base tracepoint functionality should suffice for now but this is definitely a cool point and good to know, thanks.
Quoted 15 lines
Quoted 8 lines
2. Tangential to 1., we need that "thin" software layer prepared for decoding and reporting them as early as possible.
event_trace_init()is anfs_initcalland executed too late, IMHO. The ->perf_event_enable in theftrace_event_callis enabled even later on the perf init path over thesys_perf_even_openwhich is at userspace time. In our case, this is going be executed by the error logging and decoding daemon I guess.Quoted 6 lines
3. Since we want to listen for MCEs all the time, the concept of enabling and disabling those events does not apply in the sense of performance profiling. IOW, MCEs need to be able to be logged to the ring buffer at any time. I guess this is easily done - we simply enable MCE events at the earliest moment possible and disable them on shutdown; done.
Quoted 3 lines
This looks like a good reason to have your own handler. More than one callback may be registered to a tracepoint, so you do not need to worry about having other handlers affect your code.
Yep, good ideas, thanks. /me goes back to the drawing board.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 2 lines · Borislav Petkov
so I finally had some spare time to stare at perf/ftrace code and ponder on how to use those facilities for MCE collecting and reporting. Btw, I
A good beginning of any such investigations would be to describe what exact problems you're trying to solve here.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 12 lines · Borislav Petkov
Hi,
so I finally had some spare time to stare at perf/ftrace code and ponder on how to use those facilities for MCE collecting and reporting. Btw, I have to say, it took me quite a while to understand what goes where - my suggestion to anyone who tries to understand how perf/ftrace works is to do make <file.i> where there is at least one
trace_XXXemit record function call and start untangling code paths from there.So anyway, here are some questions I had, I just as well may've missed something so please correct me if I'm wrong:
Quoted 7 lines
1. Since machine checks can happen at any time, we need to have the MCE tracepoint (
trace_mce_record()in <include/trace/events/mce.h>) always enabled. This, in turn, means that we need the ftrace/perf infrastructure always compiled in (lockless ring buffer,perf_event.cstuff) on any x86 system so that MCEs can be handled at anytime. Is this going to be ok to be enabled on _all_ machines, hmmm... I dunno, maybe only a subset of those facilites at least.
Yeah - and this happens on x86 anyway so you can rely on it.
Quoted 6 lines
2. Tangential to 1., we need that "thin" software layer prepared for decoding and reporting them as early as possible.
event_trace_init()is anfs_initcalland executed too late, IMHO. The ->perf_event_enable in theftrace_event_callis enabled even later on the perf init path over thesys_perf_even_openwhich is at userspace time. In our case, this is going be executed by the error logging and decoding daemon I guess.
We could certainly move bits of this initialization earlier.
Also we could add the notion of 'persistent' events that dont have a user-space process attached to them - and which would buffer to a certain degree. Such persistent events could be initialized during bootup and the daemon could pick up the events later on.
In-kernel actions/policy could work off the callback mechanism. (See for example how the new NMI watchdog code in tip:perf/nmi makes use of it - or how the hw-breakpoints code utilizes it.) These would work even if there's no user-space daemon attached (or if the daemon has been stopped). So i dont see a significant design problem here - it's all natural extensions of existing perf facilities and could be used for other purposes as well.
Quoted 3 lines
3. Since we want to listen for MCEs all the time, the concept of enabling and disabling those events does not apply in the sense of performance profiling. [...]
Correct.
Quoted 5 lines
[...] IOW, MCEs need to be able to be logged to the ring buffer at any time. I guess this is easily done - we simply enable MCE events at the earliest moment possible and disable them on shutdown; done.
So yeah, some food for thought but what do you guys think?
Note that it doesnt _have to_ go to the ftrace ring-buffer. If the daemon (or whatever facility picking up the events) keeps a global (per cpu) MCE perf event enabled all the time then it might be doing that regardless of ftrace.
Some decoupling from ftrace could be done here easily. I'd suggest to not worry about it - once we have the MCE event code we can certainly reshape the underlying support code to be more readily available/configurable. (or even built-in) This is not really a significant design issue.
To start with this, a quick initial prototype could use the 'perf trace' live mode tracing script. (See latest -tip, 'perf trace --script <script-name>' and 'perf record -o -' to activate live mode.)
Note that there's also 'perf inject' now, which can be used to simulate rare events and test the daemon's reactions to it. (Right now perf-inject is only used to inject certain special build-id events, but it can be used for the injection of MCE events as well.)
Thanks,
Ingo
Borislav Petkov
maintainerFrom: Ingo Molnar <[email protected]>
Date: Tue, May 04, 2010 at 01:32:27PM +0200
Hi,
Quoted 3 lines
To start with this, a quick initial prototype could use the 'perf trace' live mode tracing script. (See latest -tip, 'perf trace --script <script-name>' and 'perf record -o -' to activate live mode.)
so I did some experimenting with this and have a pretty rough prototype which conveys decoded MCEs to userspace where they're read with perf. More specifically, I did
perf record -e mce:mce_record -a
after tweaking the mce_record tracepoint to include the decoded error string.
And then doing
perf trace -g python
perf trace -s perf-trace.py
got me:
in trace_beginmce__mce_record 6 00600.700632283 0 init mcgcap=262, mcgstatus=0, bank=4, status=15888347641659525651, addr=26682366720, misc=13837309867997528064, ip=0, cs=0, tsc=0, walltime=1273928155, cpu=6, cpuid=1052561, apicid=6, socketid=0, cpuvendor=2, decoded_err= Northbridge Error, node 1ECC/ChipKill ECC error.
CE err addr: 0x636649b00
CE page 0x636649, offset 0xb00, grain 0, syndrome 0x1fd, row 3, channel 0
Transaction type: generic read(mem access), no t
in trace_end
which shows the signature of an ECC which I injected earlier over the EDAC sysfs interface. And yes, the decoded_err appears truncated so I'll have to think of a slicker way to collect that info.
Although they're pretty rough yet, I've attached the relevant patches so that one could get an impression of where we're moving here.
0001-amd64_edac-Remove-polling-mechanism.patch removes the EDAC polling mechanism in favor of hooking into the machine_check_poll polling function using the atomic notifier which we already use for uncorrectable errors.
The other two
0002-mce-trace-Add-decoded-string-to-mce_record-s-format.patch 0003-edac-mce-Prepare-error-decoded-info.patch
add that decoded_err string. I'm open for better ideas here though.
Concerning the early MCE logging and reporting, I'm thinking of using the mce.c ring buffer temporarily until the ftrace buffer has been initialized and then copying all records into the last. We might do a more elegant solution in the future after all that bootmem churn has quieted down and allocate memory early for a dedicated MCE ring buffer or whatever.
Wrt critical MCEs, I'm leaning towards bypassing perf/ftrace subsystem altogether in favor of executing the smallest amount of code possible like, for example, switching to a tty, dumping the decoded error and in certain cases not panicking but shutting down gracefully after a timeout. Of course, graceful shutdown is completely dependent on the type of hw failure and in some cases we can't do anything else but freeze in order to prevent faulty data propagation.
I'm sure there's more...
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 1 line
From
47927877707e57f7cda3093d426160bb70654291Mon Sep 17 00:00:00 2001
From: Borislav Petkov <[email protected]>
Date: Sat, 15 May 2010 13:51:57 +0200
Subject: [PATCH 1/4] amd64_edac: Remove polling mechanism
Switch to using the machine check polling mechanism of the mcheck core instead of duplicating functionality.
--- arch/x86/kernel/cpu/mcheck/mce.c | 1 + drivers/edac/amd64_edac.c | 118 -------------------------------------- 2 files changed, 1 insertions(+), 118 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +1 -0
@@ -581,6 +581,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)*/ if (!(flags &MCP_DONTLOG ) && !mce_dont_log_ce) {mce_log (&m); +atomic_notifier_call_chain (&x86_mce_decoder_chain,0 , &m);add_taint (TAINT_MACHINE_CHECK ); }
drivers/edac/amd64_edac.c +0 -118
136 lines not on this page · raw message
Signature · 1 line
1.6.4.4
Quoted 1 line
From
e5e0209868763073b4a82c6874a7e9d21fd7c8e5Mon Sep 17 00:00:00 2001
From: Borislav Petkov <[email protected]>
Date: Sat, 27 Mar 2010 17:25:02 +0100
Subject: [PATCH 2/4] mce, trace: Add decoded string to mce_record's format
Put the decoded error string into the trace record.
--- arch/x86/kernel/cpu/mcheck/mce.c | 2 +- include/trace/events/mce.h | 43 +++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 20 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +1 -1
@@ -161,7 +161,7 @@ void mce_log(struct mce *mce)unsigned next, entry;/* Emit the trace record: */ -trace_mce_record (mce); +trace_mce_record (mce,"" ); mce->finished =0 ;wmb ();
include/trace/events/mce.h +24 -19
69 lines not on this page · raw message
Signature · 1 line
1.6.4.4
Quoted 1 line
From
980c6fdf00cf23ef76481a4c94ba682c0ff80d61Mon Sep 17 00:00:00 2001
From: Borislav Petkov <[email protected]>
Date: Sat, 27 Mar 2010 18:42:08 +0100
Subject: [PATCH 3/4] edac, mce: Prepare error decoded info
Add a buffer where CECC error info is stored and dump it later into the trace record.
--- arch/x86/kernel/cpu/mcheck/mce.c | 2 + drivers/edac/amd64_edac.c | 4 ++- drivers/edac/edac_mc.c | 7 ++++ drivers/edac/edac_mce_amd.c | 60 +++++++++++++++++++++++++++++++------ drivers/edac/edac_mce_amd.h | 1 + 5 files changed, 63 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +2 -0
@@ -160,8 +160,10 @@ void mce_log(struct mce *mce) {unsigned next, entry; +#ifndef CONFIG_EDAC_DECODE_MCE /* Emit the trace record: */ trace_mce_record (mce,"" ); +#endif mce->finished =0 ;wmb ();
drivers/edac/amd64_edac.c +3 -1
@@ -1993,7 +1993,9 @@ static void amd64_handle_ce(struct mem_ctl_info *mci, sys_addr = pvt->ops->get_error_address (mci, info);amd64_mc_printk (mci,KERN_ERR , -"CE ERROR_ADDRESS= 0x%llx\n" , sys_addr); +"CE err addr: 0x%llx\n" , sys_addr); + +edac_snprintf ("CE err addr: 0x%llx\n" , sys_addr); pvt->ops->map_sysaddr_to_csrow (mci, info, sys_addr); }
drivers/edac/edac_mc.c +7 -0
@@ -33,6 +33,7 @@#include <asm/edac.h> #include "edac_core.h" #include "edac_module.h" +#include "edac_mce_amd.h" /* lock to memory controller's control array */ static DEFINE_MUTEX (mem_ctls_mutex);
@@ -702,6 +703,12 @@ void edac_mc_handle_ce(struct mem_ctl_info *mci, mci->csrows[row].grain, syndrome, row, channel, mci->csrows[row].channels[channel].label, msg); +edac_snprintf ("CE page 0x%lx, offset 0x%lx, grain %d, syndrome " +"0x%lx, row %d, channel %d\n" , + page_frame_number, offset_in_page, + mci->csrows[row].grain, syndrome, row, channel); + + mci->ce_count++; mci->csrows[row].ce_count++; mci->csrows[row].channels[channel].ce_count++;
drivers/edac/edac_mce_amd.c +50 -10
124 lines not on this page · raw message
drivers/edac/edac_mce_amd.h +1 -0
@@ -66,4 +66,5 @@ void amd_register_ecc_decoder(void (*f)(int, struct err_regs *));void amd_unregister_ecc_decoder (void (*f)(int ,struct err_regs *));void amd_decode_nb_mce (int ,struct err_regs *,int ); +void edac_snprintf (const char *fmt, ...);#endif /* _EDAC_MCE_AMD_H */
Signature · 1 line
1.6.4.4
Quoted 5 lines · Borislav Petkov
From: Ingo Molnar <[email protected]>
Date: Tue, May 04, 2010 at 01:32:27PM +0200Hi,
Quoted 7 lines
Quoted 3 lines
To start with this, a quick initial prototype could use the 'perf trace' live mode tracing script. (See latest -tip, 'perf trace --script <script-name>' and 'perf record -o -' to activate live mode.)
so I did some experimenting with this and have a pretty rough prototype which conveys decoded MCEs to userspace where they're read with perf.
nice :-)
Quoted 36 lines
More specifically, I did
perf record -e mce:mce_record -a
after tweaking the
mce_recordtracepoint to include the decoded error string.And then doing
perf trace -g python
perf trace -sperf-trace.pygot me:
in
trace_beginmce__mce_record6 00600.700632283 0 init mcgcap=262, mcgstatus=0, bank=4, status=15888347641659525651, addr=26682366720, misc=13837309867997528064, ip=0, cs=0, tsc=0, walltime=1273928155, cpu=6, cpuid=1052561, apicid=6, socketid=0, cpuvendor=2, decoded_err= Northbridge Error, node 1ECC/ChipKill ECC error.
CE err addr:0x636649b00
CE page0x636649, offset0xb00, grain 0, syndrome0x1fd, row 3, channel 0
Transaction type: generic read(mem access), no t
intrace_endwhich shows the signature of an ECC which I injected earlier over the EDAC sysfs interface. And yes, the
decoded_errappears truncated so I'll have to think of a slicker way to collect that info.Although they're pretty rough yet, I've attached the relevant patches so that one could get an impression of where we're moving here.
0001-amd64_edac-Remove-polling-mechanism.patch removes the EDAC polling mechanism in favor of hooking into the
machine_check_pollpolling function using the atomic notifier which we already use for uncorrectable errors.
Quoted 7 lines
The other two
0002-mce-trace-Add-decoded-string-to-mce_record-s-format.patch 0003-edac-mce-Prepare-error-decoded-info.patch
add that
decoded_errstring. I'm open for better ideas here though.
Yes, this is exactly what i was thinking about.
Quoted 8 lines
Concerning the early MCE logging and reporting, I'm thinking of using the
mce.cring buffer temporarily until the ftrace buffer has been initialized and then copying all records into the last. We might do a more elegant solution in the future after all that bootmem churn has quieted down and allocate memory early for a dedicated MCE ring buffer or whatever.
Agreed.
There's overlap here with the boot tracer as well, which we want to convert over to perf events as well.
That could be achieved via the concept of 'persistent events', which are basically task-less events brought active and attached to a buffer space to dump the events to.
That buffer space could be initialized very early on.
Quoted 10 lines
Wrt critical MCEs, I'm leaning towards bypassing perf/ftrace subsystem altogether in favor of executing the smallest amount of code possible like, for example, switching to a tty, dumping the decoded error and in certain cases not panicking but shutting down gracefully after a timeout. Of course, graceful shutdown is completely dependent on the type of hw failure and in some cases we can't do anything else but freeze in order to prevent faulty data propagation.
I agree that the critical functionality itself should be implemented in the kernel - and not all routed through a user-space component.
But please generate the callbacks via perf events, like the new watchdog code does in -tip (tip:perf/nmi):
cafcd80: lockup_detector: Cross arch compile fixes
23637d4: lockup_detector: Introduce CONFIG_HARDLOCKUP_DETECTOR
c01d432: lockup_detector: Adapt CONFIG_PERF_EVENT_NMI to other archs
e16bb1d: lockup_detector: Update some config
5e85391: x86, watchdog: Fix build error in hw_nmi.c
0167c78: watchdog: Export touch_softlockup_watchdog
19cc36c: lockup_detector: Fix forgotten config conversion
89d7ce2: lockup_detector: Make BOOTPARAM_SOFTLOCKUP_PANIC depend on LOCKUP_DETECTOR
d7c5473: lockup_detector: Separate touch_nmi_watchdog code path from touch_watchdog 10f9014: x86: Cleanup hw_nmi.c cruft
7cbb7e7: x86: Move trigger_all_cpu_backtrace to its own die_notifier
f69bcf6: lockup_detector: Remove nmi_watchdog.c file
2508ce1: lockup_detector: Remove old softlockup code
332fbdb: lockup_detector: Touch_softlockup cleanups and softlockup_tick removal 58687ac: lockup_detector: Combine nmi_watchdog and softlockup detector 5671a10: nmi_watchdog: Tell the world we're active
c99c30f: nmi_watchdog: Turn it off by default
47195d5: nmi_watchdog: Clean up various small details
2cc4452: nmi_watchdog: Fix undefined 'apic' build bug
96ca402: nmi_watchdog: Properly configure for software events
6081b6c: nmi_watchdog: support for oprofile
cf454ae: nmi_watchdog: Fallback to software events when no hardware pmu detected 504d7cf: nmi_watchdog: Compile and portability fixes
c3128fb: nmi_watchdog: Use a boolean config flag for compiling 8e7672c: nmi_watchdog: Only enable on x86 for now
84e478c: nmi_watchdog: Config option to enable new nmi_watchdog 1fb9d6a: nmi_watchdog: Add new, generic implementation, using perf events
e40b172: x86: Move notify_die from nmi.c to traps.c
Then those callbacks can be used to implement some minimal critical functionality: panic/print decisions, tty switching, etc.
The essence of the method is (see kernel/watchdog.c):
event = perf_event_create_kernel_counter(
&wd_attr, hotcpu, -1, wd_overflow);
the wd_overflow() is the callback you get. It's called when the event triggers. You can put arbitrary functionality into that callback.
Doing it this way is basically a constant, gradual process of unifying our currently woefully inconsistent callbacks within the x86 platform and the core kernel, and collecting these events into the TRACE_EVENT and perf umbrella - without any loss in functionality.
You can still call back on those events and add arbitrary EDAC and RAS features on top of that. (if there's something you cannot do via that method, or if it feels clumsy please let us know and we'll fix/enhance it.)
The advantage of that approach is that almost as a side-effect we also gain an event described in a structured way - which other tools can (and do) make use of.
Another related development is the ongoing work to describe events in sysfs, not in debugfs. The current plans put them under special files:
/sys/kernel/sched/events/wakeup/id
/sys/kernel/sched/events/wakeup/format
/sys/devices/system/cpu/events/cycles/id /sys/devices/system/cpu/events/instructions/id
/sys/kernel/events/mce/mce_record/id
/sys/kernel/events/mce/mce_record/format
The exact placement is not yet final - we could
reasonably put the MCE events under/sys/devices/system/cpu/events/mce/ as well - and
alias them to multiple places.
Likewise, memory controller events could show up
under /sys/devices/system/node/events/ ?
All events could be found via /sys/class/events/ and could be enumerated from there by EDAC/RAS tools.
See the discussion and patch from Lin Ming that
started this on lkml:
[RFC][PATCH 3/9] perf: export registerred pmus via sysfs
Plus, mid-term/long-term we could also split EDAC/RAS functionality from tools/perf/ and put it into tools/ras/, tools/mce/ or tools/edac/, and librarize common functionality to share as much code as possible.
We dont 'have to' have a 'perf mce' sub-tool - that's just a convenient starting place for you i suspect.
Thanks,
Ingo
Borislav Petkov
maintainerFrom: Ingo Molnar <[email protected]>
Date: Sun, May 16, 2010 at 01:26:41PM +0200
Hi,
Quoted 24 lines
Quoted 24 lines
in
trace_beginmce__mce_record6 00600.700632283 0 init mcgcap=262, mcgstatus=0, bank=4, status=15888347641659525651, addr=26682366720, misc=13837309867997528064, ip=0, cs=0, tsc=0, walltime=1273928155, cpu=6, cpuid=1052561, apicid=6, socketid=0, cpuvendor=2, decoded_err= Northbridge Error, node 1ECC/ChipKill ECC error.
CE err addr:0x636649b00
CE page0x636649, offset0xb00, grain 0, syndrome0x1fd, row 3, channel 0
Transaction type: generic read(mem access), no t
intrace_endwhich shows the signature of an ECC which I injected earlier over the EDAC sysfs interface. And yes, the
decoded_errappears truncated so I'll have to think of a slicker way to collect that info.Although they're pretty rough yet, I've attached the relevant patches so that one could get an impression of where we're moving here.
0001-amd64_edac-Remove-polling-mechanism.patch removes the EDAC polling mechanism in favor of hooking into the
machine_check_pollpolling function using the atomic notifier which we already use for uncorrectable errors.The other two
Quoted 7 lines
Quoted 5 lines
0002-mce-trace-Add-decoded-string-to-mce_record-s-format.patch 0003-edac-mce-Prepare-error-decoded-info.patch
add that
decoded_errstring. I'm open for better ideas here though.Yes, this is exactly what i was thinking about.
The decoded string handling is still clumsy since it is of variable length and I have to allocate a string of the maximal possible length of any error for it to not get truncated.
We could avoid this by spitting error codes of fixed length, instead, which the RAS daemon would map to strings in userspace but the coding scheme would need some thinking so that it works adequately for all possible error types.
Anyway, we can always improve this if needed and as we go...
Quoted 11 lines
Quoted 8 lines
Concerning the early MCE logging and reporting, I'm thinking of using the
mce.cring buffer temporarily until the ftrace buffer has been initialized and then copying all records into the last. We might do a more elegant solution in the future after all that bootmem churn has quieted down and allocate memory early for a dedicated MCE ring buffer or whatever.Agreed.
Quoted 7 lines
There's overlap here with the boot tracer as well, which we want to convert over to perf events as well.
That could be achieved via the concept of 'persistent events', which are basically task-less events brought active and attached to a buffer space to dump the events to.
Yep, sounds neat.
Quoted 13 lines
That buffer space could be initialized very early on.
Quoted 10 lines
Wrt critical MCEs, I'm leaning towards bypassing perf/ftrace subsystem altogether in favor of executing the smallest amount of code possible like, for example, switching to a tty, dumping the decoded error and in certain cases not panicking but shutting down gracefully after a timeout. Of course, graceful shutdown is completely dependent on the type of hw failure and in some cases we can't do anything else but freeze in order to prevent faulty data propagation.
Quoted 7 lines
I agree that the critical functionality itself should be implemented in the kernel - and not all routed through a user-space component.
But please generate the callbacks via perf events, like the new watchdog code does in -tip (tip:perf/nmi):
[snip]
My only concern here is that going over the perf events and callbacks adds unnecessary additional code when we're in emergency mode handling an uncorrectable error. However, I don't know how much that code would be and whether its overhead would be relevant or not...
Quoted 4 lines
Then those callbacks can be used to implement some minimal critical functionality: panic/print decisions, tty switching, etc.
Quoted 8 lines
The essence of the method is (see
kernel/watchdog.c):event = perf_event_create_kernel_counter(
&wd_attr, hotcpu, -1, wd_overflow);the
wd_overflow()is the callback you get. It's called when the event triggers. You can put arbitrary functionality into that callback.
Correct me if I'm wrong but since the trace_mce_record() is a tracepoint, we don't want to register callbacks over the perf_event* interface but rather use the ftrace path, like Steven's example with register_trace_sched_switch().
If we do it this way, there's no overhead at all and we add all the needed functionality like tty switching and emergency shutdown preparations to the proper path in the proper order - right after having decoded the MCE and still in the NMI context of do_machine_check, i.e. at the earliest possible moment and without wasting time.
And with tracepoints we still have the event unification/enumeration you mention below so I think it can't get any better than that :).
Quoted 13 lines
Doing it this way is basically a constant, gradual process of unifying our currently woefully inconsistent callbacks within the x86 platform and the core kernel, and collecting these events into the
TRACE_EVENTand perf umbrella - without any loss in functionality.You can still call back on those events and add arbitrary EDAC and RAS features on top of that. (if there's something you cannot do via that method, or if it feels clumsy please let us know and we'll fix/enhance it.)
Quoted 4 lines
The advantage of that approach is that almost as a side-effect we also gain an event described in a structured way - which other tools can (and do) make use of.
see above.
Quoted 10 lines
Another related development is the ongoing work to describe events in sysfs, not in debugfs. The current plans put them under special files:
/sys/kernel/sched/events/wakeup/id/sys/kernel/sched/events/wakeup/format
/sys/devices/system/cpu/events/cycles/id/sys/devices/system/cpu/events/instructions/id
Quoted 7 lines
/sys/kernel/events/mce/mce_record/id/sys/kernel/events/mce/mce_record/formatThe exact placement is not yet final - we could reasonably put the MCE events under
/sys/devices/system/cpu/events/mce/as well - and alias them to multiple places.
Well, since MCEs are per CPU, the proper way to map those would be
/sys/devices/system/cpu/cpuX/events/mce/
With this, we could have the additional functionality to disable some MCEs per CPU if it makes sense for certain cases...
Hmm?
Quoted 9 lines
Likewise, memory controller events could show up under
/sys/devices/system/node/events/?All events could be found via
/sys/class/events/and could be enumerated from there by EDAC/RAS tools.See the discussion and patch from Lin Ming that started this on lkml:
Quoted 7 lines
[RFC][PATCH 3/9] perf: export registerred pmus via sysfs
Plus, mid-term/long-term we could also split EDAC/RAS functionality from
tools/perf/and put it intotools/ras/,tools/mce/ortools/edac/, and librarize common functionality to share as much code as possible.
Well, both mce and edac are a subset of RAS so calling it perf/ras/ is the most sensible way to go IMHO. Also, this is where the perf inject can be reused since we can inject true hardware errors and not only simulate them in software.
Quoted 2 lines
We dont 'have to' have a 'perf mce' sub-tool - that's just a convenient starting place for you i suspect.
ditto, 'perf ras' is what I'm thinking.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.