Skip to contentSkip to conversation

RAS trace event proto

PATCH Borislav Petkov 4 participants 19 messages Feb 20, 2012 – Feb 27, 2012

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi all,

here's a dirty patch which should hopefully show what I have in mind wrt using tracepoints for RAS events. The code compiles and should only give an initial idea, it is subject to significant changes until it reaches its final form thus it is only exemplary and not final in any case.

Notes:

* So there are two RAS tracepoints: trace_mce_record which dumps the MCE-related errors + their decoded info and trace_hw_error which simply carries a string to userspace. The second one can be used for non-MCA errors.

* When prepping the string for the tracepoint, we cache the string by calling ras_printk which buffers the so-far done string internally, so everything that wants to dump into it needs to be converted to use ras_printk.

* Which brings me to it: ras_printk() is x86-only and it could probably be moved to an arch-agnostic place for the other arches. I'd leave it x86-only for now, for testing purposes, and then later the other arches could consider using it (this is wrt non-x86 EDAC drivers).

* When writing a 1 into /sys/devices/system/ras/agent, we enable the string buffering functionality - this could be done by the RAS daemon or whatever agent is requesting putting hw errors info into tracing.

* I'd like to have conditional printk-ing in trace_mce_record depending on the TP args, Steve probably knows what can be done:

@Steven:

I'd like to do the following:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", str1, arg1, arg2)

and have it print only the first arg, i.e. the string and drop the rest of the args while still doing the TP_fast_assign into the ring buffer and carrying the stuff to its consumers. Background is that I want to dump the decoded string of a hardware error, if it is decoded, but carry the MCE info to userspace and only dump the fields of the MCE if I haven't managed to decode it, i.e. str1 == "".

So, my question is, can I do something like:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", __print_conditional(str1, arg1, arg2))

where __print_conditional is a vararg macro which calls a ftrace_print_cond() which prints only str1 if strlen(str1) > 0 and otherwise calls a vsnprintf() variant to deal with the va_args?

As always, all comments are welcome.

--

Quoted 1 line

From e06143929d7d6cbed7bec1a7f4976f595a2537da Mon Sep 17 00:00:00 2001

From: Borislav Petkov <[email protected]>
Date: Mon, 20 Feb 2012 14:52:19 +0100
Subject: [PATCH] RAS trace event proto

---
 arch/x86/Kconfig                 |    9 ++
 arch/x86/Makefile                |    3 +
 arch/x86/include/asm/ras.h       |   17 +++
 arch/x86/kernel/cpu/mcheck/mce.c |    2 +-
 arch/x86/ras/Makefile            |    1 +
 arch/x86/ras/ras.c               |  121 +++++++++++++++++++++
 drivers/edac/amd64_edac.c        |    8 +-
 drivers/edac/edac_mc.c           |    9 ++-
 drivers/edac/mce_amd.c           |  218 ++++++++++++++++++++------------------
 include/trace/events/mce.h       |   26 ++++-
 10 files changed, 306 insertions(+), 108 deletions(-)
 create mode 100644 arch/x86/include/asm/ras.h
 create mode 100644 arch/x86/ras/Makefile
 create mode 100644 arch/x86/ras/ras.c
arch/x86/Kconfig +9 -0
index 5bed94e189fa..bda1480241b2 100644
@@ -657,6 +657,15 @@ config X86_CYCLONE_TIMER
 	def_bool y
 	depends on X86_SUMMIT
 
+config X86_RAS
+	def_bool y
+	prompt "X86 RAS features"
+	---help---
+	A collection of Reliability, Availability and Serviceability
+	software features which aim to enable hardware error logging
+	and reporting. Leave it at 'y' unless you really know what
+	you're doing
+
 source "arch/x86/Kconfig.cpu"
 
 config HPET_TIMER
arch/x86/Makefile +3 -0
index 209ba1294592..a6b6bb1f308b 100644
@@ -146,6 +146,9 @@ drivers-$(CONFIG_OPROFILE) += arch/x86/oprofile/
 # suspend and hibernation support
 drivers-$(CONFIG_PM) += arch/x86/power/
 
+# RAS support
+core-y += arch/x86/ras/
+
 drivers-$(CONFIG_FB) += arch/x86/video/
 
 ####
arch/x86/include/asm/ras.h new file +17 -0
new file mode 100644
index 000000000000..27333cfd7534
@@ -0,0 +1,17 @@
+#ifndef _ASM_X86_RAS_H
+#define _ASM_X86_RAS_H
+
+#define ERR_STRING_SZ 200
+
+extern bool ras_agent;
+extern char *decoded_err_str;
+
+enum ras_printk_flags {
+	PR_EMERG	= 0,
+	PR_WARNING	= 1,
+	PR_CONT		= 2,
+	RAS_EOFLAGS,
+};
+extern void ras_printk(enum ras_printk_flags flags, const char *fmt, ...);
+
+#endif /* _ASM_X86_RAS_H */
arch/x86/kernel/cpu/mcheck/mce.c +1 -1
index 5a11ae2e9e91..072e020ecaf3 100644
@@ -144,7 +144,7 @@ void mce_log(struct mce *mce)
 	int ret = 0;
 
 	/* Emit the trace record: */
-	trace_mce_record(mce);
+	trace_mce_record("", mce);
 
 	ret = atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, mce);
 	if (ret == NOTIFY_STOP)
arch/x86/ras/Makefile new file +1 -0
new file mode 100644
index 000000000000..7a70bb5cd057
@@ -0,0 +1 @@
+obj-y		:= ras.o
arch/x86/ras/ras.c new file +121 -0

121 lines not on this page · raw message

drivers/edac/amd64_edac.c +6 -2
index c9eee6d33e9a..8a42e591508d 100644
@@ -1,6 +1,7 @@
-#include "amd64_edac.h"
 #include <asm/amd_nb.h>
+#include <asm/ras.h>
 
+#include "amd64_edac.h"
 static struct edac_pci_ctl_info *amd64_ctl_pci;
 
 static int report_gart_errors;
@@ -1901,7 +1902,10 @@ static void amd64_handle_ce(struct mem_ctl_info *mci, struct mce *m)
 	sys_addr = get_error_address(m);
 	syndrome = extract_syndrome(m->status);
 
-	amd64_mc_err(mci, "CE ERROR_ADDRESS= 0x%llx\n", sys_addr);
+	if (ras_agent)
+		ras_printk(PR_EMERG, "err addr: 0x%llx", sys_addr);
+	else
+		amd64_mc_err(mci, "CE ERROR_ADDRESS= 0x%llx\n", sys_addr);
 
 	pvt->ops->map_sysaddr_to_csrow(mci, sys_addr, syndrome);
 }
drivers/edac/edac_mc.c +8 -1
index ca6c04d350ee..772d712a6f74 100644
@@ -30,8 +30,10 @@
 #include <asm/uaccess.h>
 #include <asm/page.h>
 #include <asm/edac.h>
+#include <asm/ras.h>
 #include "edac_core.h"
 #include "edac_module.h"
+#include "mce_amd.h"
 
 /* lock to memory controller's control array */
 static DEFINE_MUTEX(mem_ctls_mutex);
@@ -701,7 +703,11 @@ void edac_mc_handle_ce(struct mem_ctl_info *mci,
 		return;
 	}
 
-	if (edac_mc_get_log_ce())
+	if (edac_mc_get_log_ce()) {
+		if (ras_agent)
+			ras_printk(PR_CONT, "row: %d, channel: %d\n",
+				   row, channel);
+
 		/* FIXME - put in DIMM location */
 		edac_mc_printk(mci, KERN_WARNING,
 			"CE page 0x%lx, offset 0x%lx, grain %d, syndrome "
@@ -709,6 +715,7 @@ void edac_mc_handle_ce(struct mem_ctl_info *mci,
 			page_frame_number, offset_in_page,
 			mci->csrows[row].grain, syndrome, row, channel,
 			mci->csrows[row].channels[channel].label, msg);
+	}
 
 	mci->ce_count++;
 	mci->csrows[row].ce_count++;
drivers/edac/mce_amd.c +117 -101

603 lines not on this page · raw message

include/trace/events/mce.h +23 -3
index 4cbbcef6baa8..1a7cd471a771 100644
@@ -10,11 +10,12 @@
 
 TRACE_EVENT(mce_record,
 
-	TP_PROTO(struct mce *m),
+	TP_PROTO(const char *msg, struct mce *m),
 
-	TP_ARGS(m),
+	TP_ARGS(msg, m),
 
 	TP_STRUCT__entry(
+		__string(	msg,		msg		)
 		__field(	u64,		mcgcap		)
 		__field(	u64,		mcgstatus	)
 		__field(	u64,		status		)
@@ -33,6 +34,7 @@ TRACE_EVENT(mce_record,
 	),
 
 	TP_fast_assign(
+		__assign_str(msg,	msg);
 		__entry->mcgcap		= m->mcgcap;
 		__entry->mcgstatus	= m->mcgstatus;
 		__entry->status		= m->status;
@@ -50,7 +52,8 @@ TRACE_EVENT(mce_record,
 		__entry->cpuvendor	= m->cpuvendor;
 	),
 
-	TP_printk("CPU: %d, MCGc/s: %llx/%llx, MC%d: %016Lx, ADDR/MISC: %016Lx/%016Lx, RIP: %02x:<%016Lx>, TSC: %llx, PROCESSOR: %u:%x, TIME: %llu, SOCKET: %u, APIC: %x",
+	TP_printk("%s\n(CPU: %d, MCGc/s: %llx/%llx, MC%d: %016Lx, ADDR/MISC: %016Lx/%016Lx, RIP: %02x:<%016Lx>, TSC: %llx, PROCESSOR: %u:%x, TIME: %llu, SOCKET: %u, APIC: %x)",
+		__get_str(msg),
 		__entry->cpu,
 		__entry->mcgcap, __entry->mcgstatus,
 		__entry->bank, __entry->status,
@@ -63,6 +66,23 @@ TRACE_EVENT(mce_record,
 		__entry->apicid)
 );
 
+TRACE_EVENT(hw_error,
+
+	TP_PROTO(const char *msg),
+
+	TP_ARGS(msg),
+
+	TP_STRUCT__entry(
+		__string(msg, msg)
+	),
+
+	TP_fast_assign(
+		__assign_str(msg, msg);
+	),
+
+	TP_printk(HW_ERR "%s\n", __get_str(msg))
+);
+
 #endif /* _TRACE_MCE_H */
 
 /* This part must be outside protection */
Signature · 12 lines
1.7.8.rc0


-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
maintainer
#1 lore raw
Quoted 2 lines · Borislav Petkov

* I'd like to have conditional printk-ing in trace_mce_record depending on the TP args, Steve probably knows what can be done:

/me is confused :-)

Quoted 14 lines

@Steven:

I'd like to do the following:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", str1, arg1, arg2)

and have it print only the first arg, i.e. the string and drop the rest of the args while still doing the TP_fast_assign into the ring buffer and carrying the stuff to its consumers. Background is that I want to dump the decoded string of a hardware error, if it is decoded, but carry the MCE info to userspace and only dump the fields of the MCE if I haven't managed to decode it, i.e. str1 == "".

Quoted 3 lines

So, my question is, can I do something like:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", __print_conditional(str1, arg1, arg2))

You want to affect the output of ftrace?

perf and even trace-cmd do the parsing later and can be overridden. Well, perf can be when we finally get it to use the updated trace-cmd parser.

-- Steve

Quoted 5 lines

where __print_conditional is a vararg macro which calls a ftrace_print_cond() which prints only str1 if strlen(str1) > 0 and otherwise calls a vsnprintf() variant to deal with the va_args?

#2 lore raw
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Borislav Petkov

* I'd like to have conditional printk-ing in trace_mce_record depending on the TP args, Steve probably knows what can be done:

/me is confused :-)

Yeah, sorry. I'll explain to you in more detail when you get over here next month :-).

Quoted 16 lines
Quoted 16 lines

@Steven:

I'd like to do the following:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", str1, arg1, arg2)

and have it print only the first arg, i.e. the string and drop the rest of the args while still doing the TP_fast_assign into the ring buffer and carrying the stuff to its consumers. Background is that I want to dump the decoded string of a hardware error, if it is decoded, but carry the MCE info to userspace and only dump the fields of the MCE if I haven't managed to decode it, i.e. str1 == "".

So, my question is, can I do something like:

Quoted 7 lines
Quoted 1 line
	TP_printk("%s, ARG1: %d, ARG2: %c ...", __print_conditional(str1, arg1, arg2))

You want to affect the output of ftrace?

perf and even trace-cmd do the parsing later and can be overridden. Well, perf can be when we finally get it to use the updated trace-cmd parser.

Right, that's another possibility. I simply didn't want to burden the ring buffer with useless stuff we aren't going to use but I guess a couple of tens of bytes aren't the world :)

Thanks.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#3 lore raw
In reply to Borislav Petkov

Em 20-02-2012 12:59, Borislav Petkov escreveu:

Quoted 46 lines

Hi all,

here's a dirty patch which should hopefully show what I have in mind wrt using tracepoints for RAS events. The code compiles and should only give an initial idea, it is subject to significant changes until it reaches its final form thus it is only exemplary and not final in any case.

Notes:

* So there are two RAS tracepoints: trace_mce_record which dumps the MCE-related errors + their decoded info and trace_hw_error which simply carries a string to userspace. The second one can be used for non-MCA errors.

* When prepping the string for the tracepoint, we cache the string by calling ras_printk which buffers the so-far done string internally, so everything that wants to dump into it needs to be converted to use ras_printk.

* Which brings me to it: ras_printk() is x86-only and it could probably be moved to an arch-agnostic place for the other arches. I'd leave it x86-only for now, for testing purposes, and then later the other arches could consider using it (this is wrt non-x86 EDAC drivers).

* When writing a 1 into /sys/devices/system/ras/agent, we enable the string buffering functionality - this could be done by the RAS daemon or whatever agent is requesting putting hw errors info into tracing.

* I'd like to have conditional printk-ing in trace_mce_record depending on the TP args, Steve probably knows what can be done:

@Steven:

I'd like to do the following:

	TP_printk("%s, ARG1: %d, ARG2: %c ...", str1, arg1, arg2)

and have it print only the first arg, i.e. the string and drop the rest of the args while still doing the TP_fast_assign into the ring buffer and carrying the stuff to its consumers. Background is that I want to dump the decoded string of a hardware error, if it is decoded, but carry the MCE info to userspace and only dump the fields of the MCE if I haven't managed to decode it, i.e. str1 == "".

So, my question is, can I do something like:

Quoted 7 lines
	TP_printk("%s, ARG1: %d, ARG2: %c ...", __print_conditional(str1, arg1, arg2))

where __print_conditional is a vararg macro which calls a ftrace_print_cond() which prints only str1 if strlen(str1) > 0 and otherwise calls a vsnprintf() variant to deal with the va_args?

As always, all comments are welcome.

Boris,

The idea of having a separate ring buffer for the hardware errors seem to be interesting (the code there at ras.c seems incomplete on this dirty patch - at least I couldn't see how userspace would retrieve the messages, if the has agent is enabled).

I prefer to code it arch-agnostic since the beginning, as there's nothing there that it is arch-specific.

It is not clear from this patch how errors are supposed to be reported, on your proposal, as, currently, you're using trace_mce_record() to report just the MCE error, without any parsing, using an empty string for the error description.

I _think_ you're planning to move trace_mce_record() to happen inside the driver (or at the edac core), for the cases where the error is parsed inside amd64_edac.

You're using ras_printk() to report not only RAS errors, but also errors that happened inside the amd64_edac driver, like:

+ ras_printk(PR_EMERG, "You shouldn't be seeing an LS MCE on this"
+ " cpu family, please report on LKML.\n");

That sounds confusing.

It also should be noticed that the amd64_edac driver reports error on a different way than other drivers. Except for amd64_edac, the other drivers prepare an error string on some internal buffer, and then they call the RAS core (edac) to present that error to userspace. For example:

static void i5000_process_fatal_error_info(struct mem_ctl_info *mci,

					struct i5000_error_info *info,
					int handle_errors)
{
...
/* Only 1 bit will be on */
	switch (allErrors) {
	case FERR_FAT_M1ERR:
		specific = "Alert on non-redundant retry or fast "
				"reset timeout";
		break;
...
		/* Form out message */
		snprintf(msg, sizeof(msg),
			 "(Branch=%d DRAM-Bank=%d RDWR=%s RAS=%d "
			 "CAS=%d, UE Err=0x%x (%s))",
			 branch >> 1, bank, rdwr ? "Write" : "Read", ras, cas,
			 ue_errors, specific);

		/* Call the helper to output message */
		edac_mc_handle_fbd_ue(mci, rank, channel, channel + 1, msg);

So, the "msg" buffer will contain the entire error message, and the core call will increment the error counters, will identify the silkscreen label associated with the memory location and output the error.

In the case of amd64_edac, the error message is generated using KERN_COUNT, e. g. instead of storing it into some buffer and then doing just one call to output it, it calls pr_count() several times (or, after your patch, ras_printk).

As the idea is to generate just one event for one hardware error, the better is to change the logic inside amd64_edac to use driver-specific buffer for the error message, and then call the RAS core error handler to output it, and let the core to generate the trace event, as this simplifies the logic on all drivers, as it removes redundancy, as reporting an error requires several operations, like:
- incrementing error counts;
- retrieving the motherboard silkscreen labels;
- printing the error at dmesg and/or generating a trace event.

<snip/>

include/trace/events/mce.h line 13

Quoted 4 lines

TRACE_EVENT(mce_record,

include/trace/events/mce.h +1 -1
-	TP_PROTO(struct mce *m),
+	TP_PROTO(const char *msg, struct mce *m),

That doesn't sound good on my eyes. It is basically preserving all MCE register values as-is (u32/u64 values), and folding all parsed information into just one string field. I fail to understand why it is more important to store in binary format the content of MCE registers than the other datat here.

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

It also requires that a RAS software to parse the messages in order to detect, for example, where the memory silkscreen label is inside it. This can be very tricky, as each driver could fill it with a different way, making very hard for RAS userspace developers to write a code that would be generic.

My proposal is to have more fields there than just one message. On my patches, the proposal is to use, instead:

TRACE_EVENT(mc_error_mce,

TP_PROTO(const unsigned int err_type,
const unsigned int mc_index,
const char *msg,
const char *label,
const char *location,
const char *detail,
const char *driver_detail,
const struct mce *m),

Where:

	err_type - identifies if the error is corrected, uncorrected, fatal;
	mc_index - identifies the memory controller instance;
	msg - the error cause, like "read error" , "write error", etc;
	label - The silkscreen label of the affected component(s);
	location - the memory hierarchy location of the error (csrow/channel or
		   branch/channel/dimm, depending on the memory hierarchy);
	detail - contains the error page, offset, grain and syndrome.
	driver_detail - contains additional driver-specific details about
		 the error (other data that the driver error parser may get).

With the above, all possible usecases are covered. The normal usecase where the user just needs to know if there were memory corruption, and what memory needs to be replaced is covered by "err_type", "msg" and "label".

If more information is required, the other fields will provide him enough detail to either replace the bad DRAM chip or to provide enough data for the user to discuss the issues with the hardware vendor.

Regards,
Mauro

#4 lore raw
Quoted 1 line · Mauro Carvalho Chehab

The idea of having a separate ring buffer for the hardware errors seem

Not a separate ring buffer - ras_printk buffers the string before its being sent through the tracepoint, i.e. once the string is done, we do:

	trace_mce_record(decoded_err_str, m);

in amd_decode_mce(). The tracepoint can be invoked in the same manner from other places.

Quoted 3 lines

to be interesting (the code there at ras.c seems incomplete on this dirty patch - at least I couldn't see how userspace would retrieve the messages, if the has agent is enabled).

The normal way you retrieve an ftrace trace.

Quoted 2 lines

I prefer to code it arch-agnostic since the beginning, as there's nothing there that it is arch-specific.

Let me quote my last email:

"* Which brings me to it: ras_printk() is x86-only and it could probably be moved to an arch-agnostic place for the other arches. I'd leave it x86-only for now, for testing purposes, and then later the other arches could consider using it (this is wrt non-x86 EDAC drivers)."

Quoted 4 lines

It is not clear from this patch how errors are supposed to be reported, on your proposal, as, currently, you're using trace_mce_record() to report just the MCE error, without any parsing, using an empty string for the error description.

Let me quote my last email:

"* So there are two RAS tracepoints: trace_mce_record which dumps the MCE-related errors + their decoded info and trace_hw_error which simply carries a string to userspace. The second one can be used for non-MCA errors."

Quoted 3 lines

I _think_ you're planning to move trace_mce_record() to happen inside the driver (or at the edac core), for the cases where the error is parsed inside amd64_edac.

No.

Quoted 2 lines

You're using ras_printk() to report not only RAS errors, but also errors that happened inside the amd64_edac driver, like:

Not the amd64_edac driver - this is the routine that decodes LS MCEs.

Quoted 5 lines
drivers/edac/mce_amd.c +2 -0
+		ras_printk(PR_EMERG, "You shouldn't be seeing an LS MCE on this"
+				     " cpu family, please report on LKML.\n");

That sounds confusing.

No, that's ok since we're carrying a string of a decoded MCE to userspace - if, instead, we have _another_ string saying that we don't decode LS MCEs for a CPU family, it's still a string so we're fine. We could do additional mappings like "DECODE_ERROR" in front of it so that the userspace tools don't choke on it.

Quoted 4 lines

It also should be noticed that the amd64_edac driver reports error on a different way than other drivers. Except for amd64_edac, the other drivers prepare an error string on some internal buffer, and then they call the RAS core (edac) to present that error to userspace. For example:

This is what I do too - ras_printk() buffers the string internally before sendint it out through the tracepoint.

Quoted 24 lines

static void i5000_process_fatal_error_info(struct mem_ctl_info *mci,

					struct i5000_error_info *info,
					int handle_errors)
{
...
/* Only 1 bit will be on */
	switch (allErrors) {
	case FERR_FAT_M1ERR:
		specific = "Alert on non-redundant retry or fast "
				"reset timeout";
		break;
...
		/* Form out message */
		snprintf(msg, sizeof(msg),
			 "(Branch=%d DRAM-Bank=%d RDWR=%s RAS=%d "
			 "CAS=%d, UE Err=0x%x (%s))",
			 branch >> 1, bank, rdwr ? "Write" : "Read", ras, cas,
			 ue_errors, specific);

		/* Call the helper to output message */
		edac_mc_handle_fbd_ue(mci, rank, channel, channel + 1, msg);
Quoted 7 lines

So, the "msg" buffer will contain the entire error message, and the core call will increment the error counters, will identify the silkscreen label associated with the memory location and output the error.

In the case of amd64_edac, the error message is generated using KERN_COUNT, e. g. instead of storing it into some buffer and then doing just one call to output it, it calls pr_count() several times (or, after your patch, ras_printk).

I think you're confusing amd64_edac with <drivers/edac/mce_amd.c> which contains the code that decodes _ALL_ MCEs and not only DRAM ECC errors which amd64_edac decodes.

Quoted 10 lines

As the idea is to generate just one event for one hardware error, the better is to change the logic inside amd64_edac to use driver-specific buffer for the error message, and then call the RAS core error handler to output it, and let the core to generate the trace event, as this simplifies the logic on all drivers, as it removes redundancy, as reporting an error requires several operations, like:
- incrementing error counts;
- retrieving the motherboard silkscreen labels;
- printing the error at dmesg and/or generating a trace event.

This is what the code does already.

Quoted 5 lines

<snip/>

Quoted 2 lines

TRACE_EVENT(mce_record,

Quoted 7 lines

include/trace/events/mce.h line 13

Quoted 2 lines
include/trace/events/mce.h +1 -1
-	TP_PROTO(struct mce *m),
+	TP_PROTO(const char *msg, struct mce *m),

That doesn't sound good on my eyes. It is basically preserving all MCE register values as-is (u32/u64 values), and folding all parsed information into just one string field. I fail to understand why it is more important to store in binary format the content of MCE registers than the other datat here.

The msg arg is the decoded string and not the MCE register values. We carry the register values to userspace just in case, say, a RAS daemon would like to look at them too. This makes sense in cases where a RAS daemon, for example, wants to count how many errors have happened per rank and at which address, etc...

Quoted 2 lines

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

Huh, because you can't grep through the trace anymore...?

Quoted 5 lines

It also requires that a RAS software to parse the messages in order to detect, for example, where the memory silkscreen label is inside it. This can be very tricky, as each driver could fill it with a different way, making very hard for RAS userspace developers to write a code that would be generic.

We can add an additional arg called char *silkscreen in case it is needed.

Quoted 5 lines

My proposal is to have more fields there than just one message. On my patches, the proposal is to use, instead:

TRACE_EVENT(mc_error_mce,

Quoted 8 lines

TP_PROTO(const unsigned int err_type,
const unsigned int mc_index,
const char *msg,
const char *label,
const char *location,
const char *detail,
const char *driver_detail,
const struct mce *m),

Hehe, so basically instead of one string you have a bunch of strings in addition to some unneeded fields. I fail to see the difference.

The greatest common denominator between hw errors is a single string containing the whole error message, that's it! The moment you start implementing an arch-agnostic solution, a single string is all you can use. And adding a tracepoint for each error type is a no-go too, as I've explained it to you already.

Quoted 12 lines

Where:

	err_type - identifies if the error is corrected, uncorrected, fatal;
	mc_index - identifies the memory controller instance;
	msg - the error cause, like "read error" , "write error", etc;
	label - The silkscreen label of the affected component(s);
	location - the memory hierarchy location of the error (csrow/channel or
		   branch/channel/dimm, depending on the memory hierarchy);
	detail - contains the error page, offset, grain and syndrome.
	driver_detail - contains additional driver-specific details about
		 the error (other data that the driver error parser may get).
Quoted 1 line

With the above, all possible usecases are covered. The normal usecase

As I've already explained it to you multiple times, this does NOT cover all possible cases - this covers only DRAM errors and adds unnecessarily too much info. If you start adding tracepoints for all the other error types hw can report, you're going to enter Bloatland very quickly.

Quoted 2 lines

where the user just needs to know if there were memory corruption, and what memory needs to be replaced is covered by "err_type", "msg" and "label".

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
maintainer
#5 lore raw
Quoted 5 lines · Borislav Petkov
Quoted 2 lines

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

Huh, because you can't grep through the trace anymore...?

I believe Mauro is talking about the tracing filters used by both perf and ftrace that lets you ignore trace events when the contents of the event does not match the filter. This is filtering out events before they go to the buffer.

-- Steve

#6 lore raw
Quoted 5 lines · Steven Rostedt
Quoted 3 lines · Borislav Petkov
Quoted 2 lines

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

Quoted 7 lines
Quoted 2 lines

Huh, because you can't grep through the trace anymore...?

I believe Mauro is talking about the tracing filters used by both perf and ftrace that lets you ignore trace events when the contents of the event does not match the filter. This is filtering out events before they go to the buffer.

Oh ok, in that case we could filter the errors - if needed - before they get even reported. I say "if needed" because normally we want to collect all hw errors in the trace, IMHO.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#7 lore raw

Em 21-02-2012 12:59, Borislav Petkov escreveu:

Quoted 6 lines · Steven Rostedt
Quoted 5 lines · Borislav Petkov
Quoted 4 lines
Quoted 3 lines

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Huh, because you can't grep through the trace anymore...?

I believe Mauro is talking about the tracing filters used by both perf and ftrace that lets you ignore trace events when the contents of the event does not match the filter. This is filtering out events before they go to the buffer.

Yes, that's what I'm talking about.

Another advantage of using separate fields for the silkscreen label and for the additional details is that the userspace program that it is displaying it (trace-cmd or whatever) could have different profiles to either display just the error message and the silkscreen label or to print the entire error message.

Quoted 3 lines

Oh ok, in that case we could filter the errors - if needed - before they get even reported. I say "if needed" because normally we want to collect all hw errors in the trace, IMHO.

#18 lore raw
In reply to Borislav Petkov

Em 21-02-2012 12:12, Borislav Petkov escreveu:

Quoted 9 lines · Mauro Carvalho Chehab
Quoted 1 line

The idea of having a separate ring buffer for the hardware errors seem

Not a separate ring buffer - ras_printk buffers the string before its being sent through the tracepoint, i.e. once the string is done, we do:

	trace_mce_record(decoded_err_str, m);

in amd_decode_mce(). The tracepoint can be invoked in the same manner from other places.

Ah, ok. I missed this call for trace_mce_record() on my first review.

The ras.c seems to be a way to help your amd64 mce parser. After reviewing how errors are output on the other drivers, I don't think that these routines would help the other drivers, as, on all other places, there is a driver buffer that does the same.

Also, it doesn't sound a good idea to have this hunk:

drivers/edac/edac_mc.c +5 -1
@@ -701,7 +703,11 @@ void edac_mc_handle_ce(struct mem_ctl_info *mci,
 		return;
 	}
 
-	if (edac_mc_get_log_ce())
+	if (edac_mc_get_log_ce()) {
+		if (ras_agent)
+			ras_printk(PR_CONT, "row: %d, channel: %d\n",
+				   row, channel);
+
 		/* FIXME - put in DIMM location */
 		edac_mc_printk(mci, KERN_WARNING,
 			"CE page 0x%lx, offset 0x%lx, grain %d, syndrome "

The routine that handles the error should be generating the trace, and not filling a buffer for the trace to happen on some other place.

Quoted 9 lines
Quoted 3 lines

to be interesting (the code there at ras.c seems incomplete on this dirty patch - at least I couldn't see how userspace would retrieve the messages, if the has agent is enabled).

The normal way you retrieve an ftrace trace.

Quoted 2 lines

I prefer to code it arch-agnostic since the beginning, as there's nothing there that it is arch-specific.

Quoted 6 lines

Let me quote my last email:

"* Which brings me to it: ras_printk() is x86-only and it could probably be moved to an arch-agnostic place for the other arches. I'd leave it x86-only for now, for testing purposes, and then later the other arches could consider using it (this is wrt non-x86 EDAC drivers)."

The ras_printk() seems to be mce_amd only.

Quoted 12 lines
Quoted 4 lines

It is not clear from this patch how errors are supposed to be reported, on your proposal, as, currently, you're using trace_mce_record() to report just the MCE error, without any parsing, using an empty string for the error description.

Let me quote my last email:

"* So there are two RAS tracepoints: trace_mce_record which dumps the MCE-related errors + their decoded info and trace_hw_error which simply carries a string to userspace. The second one can be used for non-MCA errors."

Quoted 5 lines
Quoted 3 lines

I _think_ you're planning to move trace_mce_record() to happen inside the driver (or at the edac core), for the cases where the error is parsed inside amd64_edac.

No.

The main purpose of edac_mc_handle_ce() & friends is to report the error to userspace and increment the error counters. Not having the trace call there is a failure on doing that.

Quoted 5 lines
Quoted 3 lines

You're using ras_printk() to report not only RAS errors, but also errors that happened inside the amd64_edac driver, like:

Not the amd64_edac driver - this is the routine that decodes LS MCEs.

Ok.

Quoted 13 lines
Quoted 6 lines
(quoted diff) +2 -0
+		ras_printk(PR_EMERG, "You shouldn't be seeing an LS MCE on this"
+				     " cpu family, please report on LKML.\n");

That sounds confusing.

No, that's ok since we're carrying a string of a decoded MCE to userspace - if, instead, we have _another_ string saying that we don't decode LS MCEs for a CPU family, it's still a string so we're fine. We could do additional mappings like "DECODE_ERROR" in front of it so that the userspace tools don't choke on it.

Quoted 7 lines
Quoted 4 lines

It also should be noticed that the amd64_edac driver reports error on a different way than other drivers. Except for amd64_edac, the other drivers prepare an error string on some internal buffer, and then they call the RAS core (edac) to present that error to userspace. For example:

This is what I do too - ras_printk() buffers the string internally before sendint it out through the tracepoint.

So, it is your way to implement an amd64_edac/mce_amd string buffer. I don't see any value of changing the other drivers to do it, as, on all other drivers, a simple char * buffer would produce the same result.

<snip/>

Quoted 10 lines
Quoted 9 lines

Review

Quoted 4 lines

TRACE_EVENT(mce_record,

(quoted diff) +1 -1
-	TP_PROTO(struct mce *m),
+	TP_PROTO(const char *msg, struct mce *m),

That doesn't sound good on my eyes. It is basically preserving all MCE register values as-is (u32/u64 values), and folding all parsed information into just one string field. I fail to understand why it is more important to store in binary format the content of MCE registers than the other datat here.

Quoted 5 lines

The msg arg is the decoded string and not the MCE register values. We carry the register values to userspace just in case, say, a RAS daemon would like to look at them too. This makes sense in cases where a RAS daemon, for example, wants to count how many errors have happened per rank and at which address, etc...

The rank information may not be there! On Sandy Bridge, the rank is the result of a complex decoding logic that _doesn't_ use the MCE registers. In a matter of fact, the only thing that it is provided at the MCE registers on Sandy Bridge is the address and the error type. All the other information is parsed by using non-MCE registers: memory controller, channel, rank, dimm. So, exporting struct mce to userspace doesn't help a RAS daemon for Sandy Bridge.

On the other hand, the "location" field on my proposal will always have the error location (csrow/dimm or branch/channel/slot, depending on the memory architecture). This makes much more sense to have there than the raw MCE register values.

On the other hand, on Nehalem, the MCE registers contains the memory location. So, a RAS daemon could use them, or could equally use the location field.

<snip/>

Quoted 9 lines
Quoted 6 lines

It also requires that a RAS software to parse the messages in order to detect, for example, where the memory silkscreen label is inside it. This can be very tricky, as each driver could fill it with a different way, making very hard for RAS userspace developers to write a code that would be generic.

We can add an additional arg called char *silkscreen in case it is needed.

It is not just the silkscreen label. See my patch series, if you want to see the dirty details on all drivers, or try to convert the other drivers to use your proposal, and you'll see what I'm talking about.

Quoted 14 lines
Quoted 5 lines

My proposal is to have more fields there than just one message. On my patches, the proposal is to use, instead:

TRACE_EVENT(mc_error_mce,

Quoted 8 lines

TP_PROTO(const unsigned int err_type,
const unsigned int mc_index,
const char *msg,
const char *label,
const char *location,
const char *detail,
const char *driver_detail,
const struct mce *m),

Quoted 2 lines

Hehe, so basically instead of one string you have a bunch of strings in addition to some unneeded fields. I fail to see the difference.

Some of them could be, instead, integers or some other way to represent the error details. For example, the location could be, instead:

	u32 layer0_pos, u32 layer1_pos, u32 layer2_pos

to represent the memory location (csrow/channel, branch/channel/slot or channel/slot, depending on how the location is defined on the memory hierarchy.

The bad thing of using this way is that, when just 2 layers are filled, layer2_pos will be meaningless.

The detail could be, instead:
u32 page, u32 offset, u32 grain, u32 syndrome
or even:
u64 address, u32 grain, u32 syndrome

The other fields are, per their nature, strings (error message, DIMM label, extra driver-specific details).

I'm not entirely convinced to use strings there. Comments are welcome.

Quoted 2 lines

The greatest common denominator between hw errors is a single string containing the whole error message, that's it!

If you use this argument, the MCE fields should also be converted to string, as less than 10% of the existing drivers have it.

Quoted 4 lines

The moment you start
implementing an arch-agnostic solution, a single string is all you can use. And adding a tracepoint for each error type is a no-go too, as I've explained it to you already.

This trace can easily be generalized to cover any type of error, like:

TP_PROTO(const unsigned int error_severity,
const unsigned int error_type, const unsigned int instance, const char *error_msg, const char *silkscreen_label, const char *location, const char *detail,
const char *driver_detail, const struct mce *m),

as there's nothing there that it is really memory specific:

- error_severity: corrected, uncorrected, fatal, ...;
- error_type: memory error, PCI error, bus error, ...;
- instance: an instance number to uniquelly identify the affected resource;
- silkscreen_label: can be a memory slot, a bus slot, a cpu slot, ..., depending on the error type;
- location: "csrow 1 channel 0" for MC; "07:05.0" for PCI,
"Bus 003 Device 001" for USB, ...
- detail: some additional error_type-specific detail handled by the core;
- driver_detail: some additional error_type-specific detail handled by the driver.

By having several fields, RAS software can group errors by location, by label, etc.

Quoted 13 lines
Quoted 13 lines

Where:

	err_type - identifies if the error is corrected, uncorrected, fatal;
	mc_index - identifies the memory controller instance;
	msg - the error cause, like "read error" , "write error", etc;
	label - The silkscreen label of the affected component(s);
	location - the memory hierarchy location of the error (csrow/channel or
		   branch/channel/dimm, depending on the memory hierarchy);
	detail - contains the error page, offset, grain and syndrome.
	driver_detail - contains additional driver-specific details about
		 the error (other data that the driver error parser may get).
Quoted 6 lines
Quoted 1 line

With the above, all possible usecases are covered. The normal usecase

As I've already explained it to you multiple times, this does NOT cover all possible cases - this covers only DRAM errors and adds unnecessarily too much info. If you start adding tracepoints for all the other error types hw can report, you're going to enter Bloatland very quickly.

I fail to see it. Most cases will fail on the above trace.

Quoted 3 lines
Quoted 2 lines

where the user just needs to know if there were memory corruption, and what memory needs to be replaced is covered by "err_type", "msg" and "label".

Regards,
Mauro

maintainer
#8 lore raw
In reply to Borislav Petkov
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Also, folding everything into just one string prevents (or make hard) the usage of the perf filters, for example, to filter only the errors at the memory controller 1.

Huh, because you can't grep through the trace anymore...?

Quoted 8 lines
Quoted 4 lines

I believe Mauro is talking about the tracing filters used by both perf and ftrace that lets you ignore trace events when the contents of the event does not match the filter. This is filtering out events before they go to the buffer.

Oh ok, in that case we could filter the errors - if needed - before they get even reported. I say "if needed" because normally we want to collect all hw errors in the trace, IMHO.

I'm also struggling to understand an end-user use case where you would want filtering. Mauro - can you expand a bit on why someone would just want to see the errors from memory controller 1?

My mental model of the world is that large systems have some background noise - a trickle of corrected errors that happen in normal operation. User shouldn't care about these errors unless they breach some threshold.

When something goes wrong, you may see a storm of corrected errors, or some uncorrected errors. In either case you'd like to get as much information as possible to identify the component that is at fault. I'd definitely like to see some structure to the error reporting, so that mining for data patterns in a storm isn't hideously platform dependent.

It might be easier to evaluate the competing ideas here with some sample output in addition to the code.

-Tony

#9 lore raw
Quoted 8 lines · Tony

I'm also struggling to understand an end-user use case where you would want filtering. Mauro - can you expand a bit on why someone would just want to see the errors from memory controller 1?

My mental model of the world is that large systems have some background noise - a trickle of corrected errors that happen in normal operation. User shouldn't care about these errors unless they breach some threshold.

Quoted 5 lines

When something goes wrong, you may see a storm of corrected errors, or some uncorrected errors. In either case you'd like to get as much information as possible to identify the component that is at fault. I'd definitely like to see some structure to the error reporting, so that mining for data patterns in a storm isn't hideously platform dependent.

Yep, I'm on the same page here.

Quoted 2 lines

It might be easier to evaluate the competing ideas here with some sample output in addition to the code.

Well, to clarify:

When you get a decoded error, you get the same format as what you get in dmesg, for example:

[ 2666.646070] [Hardware Error]: CPU:64   MC1_STATUS[-|CE|MiscV|PCC|-|CECC]: 0x9a05c00007010011
[ 2666.655003] [Hardware Error]: Instruction Cache Error: L1 TLB multimatch.
[ 2666.655008] [Hardware Error]: cache level: L1, tx: INSN

And with the decoded string tracepoint, that thing above is a single string. If you use trace_mce_record(), you still can get the single MCE fields which we carry to userspace from struct mce, _in addition_. The hypothetical problem is for userspace not being able to use the tracepoint format to parse reported fields easily and in an unambiguous manner. Instead, it gets a single string which, I admit, is not that pretty.

Now, the problem is if we want to use a single tracepoint for all errors
- it is unfeasible that any fields sharing can be done there except maybe the TSC stamp when it happened, the CPU that caught it and etc. not so important details.

IOW, the error format is different for each error type, almost, and there's no marrying between them. OTOH, ff we start adding tracepoints for each error type, we'll hit the other end - bloat. So also a no-no.

Maybe the compromise would be to define a single tracepoint per _hardware_ error reporting scheme. That is, MCA has an own tracepoint, PCIE AER has its own error reporting tracepoint, then there's an EDAC !x86 one which doesn't use MCA for reporting and also any other scheme a hw vendor would come up with...

This will keep the bloat level to a minimum, keep the TPs apart and hopefully make all of us happy :).

Opinions?

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#10 lore raw

Em 22-02-2012 08:43, Borislav Petkov escreveu:

Quoted 9 lines · Tony
Quoted 8 lines

I'm also struggling to understand an end-user use case where you would want filtering. Mauro - can you expand a bit on why someone would just want to see the errors from memory controller 1?

My mental model of the world is that large systems have some background noise - a trickle of corrected errors that happen in normal operation. User shouldn't care about these errors unless they breach some threshold.

Quoted 7 lines
Quoted 5 lines

When something goes wrong, you may see a storm of corrected errors, or some uncorrected errors. In either case you'd like to get as much information as possible to identify the component that is at fault. I'd definitely like to see some structure to the error reporting, so that mining for data patterns in a storm isn't hideously platform dependent.

Yep, I'm on the same page here.

The error counters will be incremented at the sysfs even if the events are filtered. A RAS software could start with the tracing disabled, and only enable the events as a hole or partially if the errors are above a certain limit (or rate limit).

Quoted 8 lines
Quoted 2 lines

It might be easier to evaluate the competing ideas here with some sample output in addition to the code.

Well, to clarify:

When you get a decoded error, you get the same format as what you get in dmesg, for example:

Quoted 7 lines
[ 2666.646070] [Hardware Error]: CPU:64   MC1_STATUS[-|CE|MiscV|PCC|-|CECC]: 0x9a05c00007010011
[ 2666.655003] [Hardware Error]: Instruction Cache Error: L1 TLB multimatch.
[ 2666.655008] [Hardware Error]: cache level: L1, tx: INSN

And with the decoded string tracepoint, that thing above is a single string. If you use trace_mce_record(), you still can get the single MCE fields which we carry to userspace from struct mce, _in addition_.

Using the same concept I've adopted for my EDAC patches, I would map the above into 3 fields:

CPU instance = 64
error message = Instruction Cache Error: L1 TLB multimatch. detail = cache level: L1, tx: INSN
(or, maybe, detail = [-|CE|MiscV|PCC|-|CECC] cache level: L1, tx: INSN)

Those fields contain what userspace needs, and it is easy for it to parse, as different things are on different places.

Quoted 4 lines

The hypothetical problem is for userspace not being able to use the tracepoint format to parse reported fields easily and in an unambiguous manner. Instead, it gets a single string which, I admit, is not that pretty.

Yes.

Quoted 4 lines

Now, the problem is if we want to use a single tracepoint for all errors
- it is unfeasible that any fields sharing can be done there except maybe the TSC stamp when it happened, the CPU that caught it and etc. not so important details.

Well, in this example, 3 fields could be very similar to the ones I used for the memory errors.

"silkscreen label" may make some sense, in order to convert from a CPU core and from some logical CPU number into the right CPU socket.

Other fields, like "location" wouldn't make much sense (as location in this case matches the CPU number), but those could simply be filled with a blank string.

I don't see any problem on having some fields that would be filled with a blank string (or that it would contain a value like -1 - in the case of integers - to mean that such value is not relevant for that error type).

Quoted 3 lines

IOW, the error format is different for each error type, almost, and there's no marrying between them. OTOH, ff we start adding tracepoints for each error type, we'll hit the other end - bloat. So also a no-no.

Indeed, one tracepoint per error type is a bad idea.

Quoted 8 lines

Maybe the compromise would be to define a single tracepoint per _hardware_ error reporting scheme. That is, MCA has an own tracepoint, PCIE AER has its own error reporting tracepoint, then there's an EDAC !x86 one which doesn't use MCA for reporting and also any other scheme a hw vendor would come up with...

This will keep the bloat level to a minimum, keep the TPs apart and hopefully make all of us happy :).

That sounds interesting.

There is also another alternative: one tracepoint per HW block (where HW block in this context is: memory, cpu, PCI, ...).

I think we should start with one tracepoint per hw reporting scheme, and see how it fits, using the parameters I found it is common denominator for the memory errors.

Regards,
Mauro

#11 lore raw
Quoted 7 lines · Mauro Carvalho Chehab

Using the same concept I've adopted for my EDAC patches, I would map the above into 3 fields:

CPU instance = 64
error message = Instruction Cache Error: L1 TLB multimatch. detail = cache level: L1, tx: INSN
(or, maybe, detail = [-|CE|MiscV|PCC|-|CECC] cache level: L1, tx: INSN)

No, this is not going to fly the moment you decide to dump MCi_ADDR because it is relevant for a certain types of errors:

[ 1121.970020] [Hardware Error]: CPU:64   MC2_STATUS[Over|CE|-|-|AddrV|CECC]: 0xd400400000000813
[ 1121.979039] [Hardware Error]:  MC2_ADDR: 0xbabedeaddeadbeef
[ 1121.979042] [Hardware Error]: Bus Unit Error: RD/ECC error in data read from NB.
[ 1121.979047] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: SRC (no timeout)

The whole decoded thing above is a string and the MCA registers are passed on into the trace, _in_ _addition_.

IOW, for each tracepoint, the format should be a string which is possibly empty and describes the error message additionally, and the remaining register attributes, one per field. Mapping the hw error scheme to some memory error format you've come up with is a bad idea and a no-no.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#12 lore raw

Em 22-02-2012 10:25, Borislav Petkov escreveu:

Quoted 9 lines · Mauro Carvalho Chehab
Quoted 7 lines

Using the same concept I've adopted for my EDAC patches, I would map the above into 3 fields:

CPU instance = 64
error message = Instruction Cache Error: L1 TLB multimatch. detail = cache level: L1, tx: INSN
(or, maybe, detail = [-|CE|MiscV|PCC|-|CECC] cache level: L1, tx: INSN)

Quoted 7 lines

No, this is not going to fly the moment you decide to dump MCi_ADDR because it is relevant for a certain types of errors:

[ 1121.970020] [Hardware Error]: CPU:64   MC2_STATUS[Over|CE|-|-|AddrV|CECC]: 0xd400400000000813
[ 1121.979039] [Hardware Error]:  MC2_ADDR: 0xbabedeaddeadbeef
[ 1121.979042] [Hardware Error]: Bus Unit Error: RD/ECC error in data read from NB.
[ 1121.979047] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: SRC (no timeout)

This could be mapped as:

CPU instance = 64
error message = Bus Unit Error: RD/ECC error in data read from NB
detail = [Over|CE|-|-|AddrV|CECC] cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: SRC (no timeout) address: 0xbabedeaddeadbeef

Quoted 2 lines

The whole decoded thing above is a string and the MCA registers are passed on into the trace, _in_ _addition_.

The MCA registers are also there on their own fields, so there's no need to also store them at the decoded string. Yet, they could be there at the detail.

What I'm proposing is that the "error message" should contain the most relevant information for the end user, while additional technical details would be at detail.

That makes easy for a RAS daemon to store and group similar errors. Messing all information inside one string would be very hard for it to parse.

Quoted 5 lines

IOW, for each tracepoint, the format should be a string which is possibly empty and describes the error message additionally, and the remaining register attributes, one per field. Mapping the hw error scheme to some memory error format you've come up with is a bad idea and a no-no.

The "remaining register attributes" is a no-no: such concept would assume that the remaining information is inside a MCA register (or inside a well-known group of registers). This is not always true.

As I said several times, the memory error location, on Sandy Bridge, is not stored on any register. The driver has to use the MCA error address as a starting point, and navigate into a tree of about 200+ registers, in order to get the DIMM channel, rank and slot.

Passing the values of all those 200+ registers to userspace wouldn't make any sense, as only a few is used to parse the error. What it makes sense is to pass the memory location to userspace.

To make things worse, when the memory controller is at mirror mode, or at lockstep mode, Sandy Bridge won't provide a single location. Instead, it will provide a channel mask, meaning that one of the two DIMMs (or four DIMMs, if both modes are enabled) is the root cause of the error.

The same also applies to FB-DIMM drivers.

So, the error location has to be either a string or up to 3 integers to represent the memory location, plus a mask, in order to represent lockstep and mirror modes.

Using a string makes it easier.

So, in other words, I think that the _best_ way to represent an MCA type of error is to use an event with the following prototype:

TP_PROTO(const unsigned int error_severity,
const unsigned int error_type, const unsigned int instance, const char *error_msg, const char *silkscreen_label, const char *location, const char *detail,
const char *driver_detail, const struct mce *m),

Where the above fields are:

- error_severity: corrected, uncorrected, fatal, ...;
- error_type: memory error, PCI error, bus error, ...;
- instance: an instance number to uniquelly identify the affected resource;
- silkscreen_label: can be a memory slot, a bus slot, a cpu slot, ..., depending on the error type;
- location: "csrow 1 channel 0" for MC; "07:05.0" for PCI,
"Bus 003 Device 001" for USB, ...
- detail: some additional error_type-specific detail handled by the core;
- driver_detail: some additional error_type-specific detail handled by the driver.

@Steven:

In a matter of fact, the error_severity and error_type should be enum's. We should not use enum's on ioctl's, as enum size is not portable. Can it be used inside perf, or should them be exported, on perf, via integer (like the above proposal)?

Regards,
Mauro.

#13 lore raw
Quoted 2 lines · Mauro Carvalho Chehab

So, in other words, I think that the _best_ way to represent an MCA type of error is to use an event with the following prototype:

No, it's not.

And if you're going to repeat yourself for the 100th time as to why, don't even bother replying.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#14 lore raw

Em 22-02-2012 12:05, Borislav Petkov escreveu:

Quoted 6 lines · Mauro Carvalho Chehab
Quoted 3 lines

So, in other words, I think that the _best_ way to represent an MCA type of error is to use an event with the following prototype:

No, it's not.

Why? It fits with all existing edac drivers, and it seems to fit with mce_amd, based on the examples you gave.

Quoted 3 lines

And if you're going to repeat yourself for the 100th time as to why, don't even bother replying.

Well, a single string is not ok, as it is not easily parseable and it doesn't allow to use the perf filters.

We need to agree on some trace format that fits, and that covers the needs by the existing drivers.

I'm ok, if you didn't like the format I've proposed, if you have a better proposal.

Regards,
Mauro.

maintainer
#15 lore raw
In reply to Mauro Carvalho Chehab
Quoted 5 lines · Mauro Carvalho Chehab

@Steven:

In a matter of fact, the error_severity and error_type should be enum's. We should not use enum's on ioctl's, as enum size is not portable. Can it be used inside perf, or should them be exported, on perf, via integer (like the above proposal)?

I don't know exactly how perf does it today, but I'm sure it's similar to trace-cmd.

You can use an enum, as it just comes through as a number. Each field is given the size of that field. If it's an int it stays an int.

Note though, perf and trace-cmd do not know how to convert the number back to a enum name. You can use the __print_symbols() feature, but that's about it.

-- Steve

#16 lore raw
In reply to Borislav Petkov
Quoted 2 lines · Borislav Petkov

This will keep the bloat level to a minimum, keep the TPs apart and hopefully make all of us happy :).

Btw, here's how the rough MCE TP trace_mce_record() looks like:

       mcegen.py-2715  [001] .N..  1049.818840: mce_record: [Hardware Error]: CPU:0     MC4_STATUS[Over|UE|-|PCC|AddrV|UECC]: 0xf604a00006080a41

[Hardware Error]: MC4_ADDR: 0xbabedeaddeadbeef
[Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected
(CPU: 0, MCGc/s: 0/0, MC4: f604a00006080a41, ADDR/MISC: babedeaddeadbeef/dead57ac1ba0babe, RIP: 00:<0000000000000000>, TSC: 0, PROCESSOR: 0:0, TIME: 0, SOCKET: 0, APIC: 0)

Basically, the userspace daemon will consume the error string (after it's been massaged into looking prettier and smaller :-)) (1st arg) and dump it to some logs, and use some of the MCE fields to do error collection and thresholding/ratelimiting/whatever.

While at it, I'm also looking very critically at the fields SOCKET, APIC, TSC (we have walltime) for I'd like to drop them. Also, MC4 should be MC4_STATUS btw.

To be continued...

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#17 lore raw

Hi Tony,

Quoted 16 lines · Borislav Petkov
Quoted 2 lines · Borislav Petkov

This will keep the bloat level to a minimum, keep the TPs apart and hopefully make all of us happy :).

Btw, here's how the rough MCE TP trace_mce_record() looks like:

       mcegen.py-2715  [001] .N..  1049.818840: mce_record: [Hardware Error]: CPU:0     MC4_STATUS[Over|UE|-|PCC|AddrV|UECC]: 0xf604a00006080a41

[Hardware Error]: MC4_ADDR: 0xbabedeaddeadbeef
[Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected
(CPU: 0, MCGc/s: 0/0, MC4: f604a00006080a41, ADDR/MISC: babedeaddeadbeef/dead57ac1ba0babe, RIP: 00:<0000000000000000>, TSC: 0, PROCESSOR: 0:0, TIME: 0, SOCKET: 0, APIC: 0)

Basically, the userspace daemon will consume the error string (after it's been massaged into looking prettier and smaller :-)) (1st arg) and dump it to some logs, and use some of the MCE fields to do error collection and thresholding/ratelimiting/whatever.

Quoted 5 lines

While at it, I'm also looking very critically at the fields SOCKET, APIC, TSC (we have walltime) for I'd like to drop them. Also, MC4 should be MC4_STATUS btw.

To be continued...

new week, new stuff:

Here's how the MCE TP looks like with a couple of MCEs injected:

       mcegen.py-2318  [001] .N..   580.902409: mce_record: [Hardware Error]: CPU:0 MC4_STATUS[Over|CE|-|PCC|AddrV|CECC]: 0xd604c00006080a41 MC4_ADDR: 0x0000000000000016

[Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected on the NB.
[Hardware Error]: ERR_ADDR: 0x16 row: 0, channel: 0
[Hardware Error]: cache level: L1, mem/io: MEM, mem-tx: DWR, part-proc: RES (no timeout)
[Hardware Error]: CPU: 0, MCGc/s: 0/0, MC4: d604c00006080a41, ADDR/MISC: 0000000000000016/dead57ac1ba0babe, RIP: 00:<0000000000000000>, TSC: 0, TIME: 0)

       mcegen.py-2326  [001] .N..   598.795494: mce_record: [Hardware Error]: CPU:0 MC4_STATUS[Over|UE|MiscV|PCC|-|UECC]: 0xfa002000001c011b[Hardware Error]: Northbridge Error (node 0): L3 ECC data cache error.

[Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
[Hardware Error]: CPU: 0, MCGc/s: 0/0, MC4: fa002000001c011b, ADDR/MISC: 0000000000000016/dead57ac1ba0babe, RIP: 00:<0000000000000000>, TSC: 0, TIME: 0)

       mcegen.py-2343  [013] .N..   619.620698: mce_record: [Hardware Error]: CPU:0 MC4_STATUS[-|UE|MiscV|PCC|-|UECC]: 0xba002100000f001b[Hardware Error]: Northbridge Error (node 0): GART Table Walk data error.

[Hardware Error]: cache level: L3/GEN, tx: GEN
[Hardware Error]: CPU: 0, MCGc/s: 0/0, MC4: ba002100000f001b, ADDR/MISC: 0000000000000016/dead57ac1ba0babe, RIP: 00:<0000000000000000>, TSC: 0, TIME: 0)

Basically the lines excluding the last one are the string message generated by the decoding code and collected into the ras decode buffer using ras_printk. Btw, the buffer enlarges itself on demand when we're close to filling it up with the decoding info.

The last line is the MCE TP with useless IMO fields removed which will be used by the RAS daemon in userspace.

I'll be splitting the single patch into multiple, more digestible chunks for review now.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551