Skip to contentSkip to conversation

x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer

PATCH In review Huang Ying 3 participants 9 messages Jun 4, 2009 – Jun 8, 2009 x86

lore
Share & tools
lore

Huang Ying

author
#0 lore raw

PATCH 1/1 x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer

Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:

- One ring buffer for each CPU

+ MCEs are added to corresponding local per-CPU buffer, instead of one big global buffer. Contention/unfairness between CPUs is eleminated.

+ MCE records are read out and removed from per-CPU buffers by mutex protected global reader function. Because there are no many readers in system to contend in most cases.

- Per-CPU ring buffer data structure

+ An array is used to hold MCE records. integer "head" indicates next writing position and integer "tail" indicates next reading position.

+ To distinguish buffer empty and full, head and tail wrap to 0 at MCE_LOG_LIMIT instead of MCE_LOG_LEN. Then the real next writing position is head % MCE_LOG_LEN, and real next reading position is tail % MCE_LOG_LEN. If buffer is empty, head == tail, if buffer is full, head % MCE_LOG_LEN == tail % MCE_LOG_LEN and head != tail.

- Lock-less for writer side

+ MCE log writer may come from NMI, so the writer side must be lock-less. For per-CPU buffer of one CPU, writers may come from process, IRQ or NMI context, so "head" is increased with cmpxchg_local() to allocate buffer space.

+ Reader side is protected with a mutex to guarantee only one reader is active in the whole system.

Performance test show that the throughput of per-CPU mcelog buffer can reach 430k records/s compared with 5.3k records/s for original implementation on a 2-core 2.1GHz Core2 machine.

ChangeLog:

v4:

- Rebased on x86-tip.git x86/mce3 branch

- Fix a synchronization issue about mce.finished

- Fix comment style

v3:

- Use DEFINE_PER_CPU to allocate per_cpu mcelog buffer.

- Use cond_resched() to prevent possible system not reponsing issue for large user buffer.

v2:

- Use alloc_percpu() to allocate per_cpu mcelog buffer.

- Use ndelay to implement witer timeout.

---
 arch/x86/include/asm/mce.h       |   17 +-
 arch/x86/kernel/cpu/mcheck/mce.c |  277 ++++++++++++++++++++++++---------------
 2 files changed, 183 insertions(+), 111 deletions(-)
arch/x86/include/asm/mce.h +12 -5
@@ -52,20 +52,27 @@ struct mce {
  * is set.
  */
 
-#define MCE_LOG_LEN 32
+#define MCE_LOG_LEN		32
+#define MCE_LOG_LIMIT		(MCE_LOG_LEN * 2 - 1)
+
+static inline int mce_log_index(int n)
+{
+	return n >= MCE_LOG_LEN ? n - MCE_LOG_LEN : n;
+}
+
+struct mce_log_cpu;
 
 struct mce_log {
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };
 
 #define MCE_OVERFLOW 0		/* bit 0 in flags means overflow */
 
-#define MCE_LOG_SIGNATURE	"MACHINECHECK"
+#define MCE_LOG_SIGNATURE	"MACHINECHEC2"
 
 #define MCE_GET_RECORD_LEN   _IOR('M', 1, int)
 #define MCE_GET_LOG_LEN      _IOR('M', 2, int)
arch/x86/kernel/cpu/mcheck/mce.c +171 -106

347 lines not on this page · raw message

Acked-by on v1 1/1
#1 lore raw
Quoted 2 lines · Huang Ying

Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:

Looks good. Thanks.

Quoted 4 lines

Performance test show that the throughput of per-CPU mcelog buffer can reach 430k records/s compared with 5.3k records/s for original implementation on a 2-core 2.1GHz Core2 machine.

I should add the main reason to add the patch is less that performance (normally there shouldn't be that many errors, so it doesn't matter too much), but that it scales to a larger number of CPUs. The old mce log ring had a fixed sized buffer that was hard to extend and didn't scale on larger systems, which could cause overflows and other issues.
Also this fixes a long standing known livelock in the old code where a very high continuous error rate could livelock the reader.

-Andi

#2 lore raw
In reply to Huang Ying
Quoted 2 lines · Huang Ying

Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:

That would be great. This should be in .31, I think.

Some minor points:

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

Quoted 7 lines
arch/x86/kernel/cpu/mcheck/mce.c +3 -0
@@ -642,6 +669,16 @@ static int mce_cap_init(void)
 	return 0;
 }
 
+/*
+ * Initialize MCE per-CPU log buffer
+ */

arch/x86/kernel/cpu/mcheck/mce.c static int mce_cap_init(void) · line 684

Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce.c +7 -0
+static __cpuinit void mce_log_init(void)
+{
+	if (mcelog.mcelog_cpus)
+		return;
+	mcelog.mcelog_cpus = &per_cpu_var(mce_log_cpus);
+}
+
 static void mce_init(void)
 {
 	mce_banks_t all_banks;

Next,

arch/x86/kernel/cpu/mcheck/mce.c static int mce_release(struct inode *ino · line 920

Quoted 7 lines
arch/x86/kernel/cpu/mcheck/mce.c +7 -0
+static int mce_empty_cpu(struct mce_log_cpu *mcelog_cpu)
+{
+	int head, tail;
+	head = mcelog_cpu->head;
+	tail = mcelog_cpu->tail;
+	return head == tail;
+}

Are there any race condition?
Why we cannot have "return (mcelog_cpu->head == mcelog_cpu->tail);"?

Last,

Quoted 8 lines
arch/x86/kernel/cpu/mcheck/mce.c +8 -0
+static ssize_t mce_read(struct file *filp, char __user *inubuf, size_t usize,
+			loff_t *off)
+{
+	char __user *ubuf = inubuf;
+	struct mce_log_cpu *mcelog_cpu;
+	int cpu, new_mce, err = 0;
+	static DEFINE_MUTEX(mce_read_mutex);
+

(snip)

arch/x86/kernel/cpu/mcheck/mce.c static int mce_release(struct inode *ino · line 969

Quoted 2 lines
arch/x86/kernel/cpu/mcheck/mce.c +1 -0
+	return err ? : ubuf - inubuf;
 }

It would be better to put "static DEFINE_MUTEX(mce_read_mutex)" to outside of the function.

And it looks work, but I'd prefer:

	return err ? err : ubuf - inubuf;

Overall it looks good.
Unfortunately there were updates on tip/mce3 so I could not apply this patch on the top of the branch. I'd appreciate it if you could rebase this patch again.

Thanks,
H.Seto

author
#3 lore raw
Quoted 8 lines · Hidetoshi Seto
Quoted 2 lines · Huang Ying

Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:

That would be great. This should be in .31, I think.

Some minor points:

Quoted 14 lines

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

It is mainly used by something like kdump, which can search "MACHINECHEC2", and analyze mce_log. mcelog_cpus can help kdump find the real mcelog storage.

Quoted 21 lines
Quoted 7 lines
arch/x86/kernel/cpu/mcheck/mce.c +3 -0
@@ -642,6 +669,16 @@ static int mce_cap_init(void)
 	return 0;
 }
 
+/*
+ * Initialize MCE per-CPU log buffer
+ */

arch/x86/kernel/cpu/mcheck/mce.c static int mce_cap_init(void) · line 684

Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce.c +7 -0
+static __cpuinit void mce_log_init(void)
+{
+	if (mcelog.mcelog_cpus)
+		return;
+	mcelog.mcelog_cpus = &per_cpu_var(mce_log_cpus);
+}
+
 static void mce_init(void)
 {
 	mce_banks_t all_banks;

Next,

Quoted 10 lines

arch/x86/kernel/cpu/mcheck/mce.c static int mce_release(struct inode *ino · line 920

Quoted 7 lines
arch/x86/kernel/cpu/mcheck/mce.c +7 -0
+static int mce_empty_cpu(struct mce_log_cpu *mcelog_cpu)
+{
+	int head, tail;
+	head = mcelog_cpu->head;
+	tail = mcelog_cpu->tail;
+	return head == tail;
+}

Are there any race condition?
Why we cannot have "return (mcelog_cpu->head == mcelog_cpu->tail);"?

Yes, it is better, I will change this.

Quoted 11 lines

Last,

Quoted 8 lines
arch/x86/kernel/cpu/mcheck/mce.c +8 -0
+static ssize_t mce_read(struct file *filp, char __user *inubuf, size_t usize,
+			loff_t *off)
+{
+	char __user *ubuf = inubuf;
+	struct mce_log_cpu *mcelog_cpu;
+	int cpu, new_mce, err = 0;
+	static DEFINE_MUTEX(mce_read_mutex);
+

(snip)

Quoted 8 lines

arch/x86/kernel/cpu/mcheck/mce.c static int mce_release(struct inode *ino · line 969

Quoted 2 lines
arch/x86/kernel/cpu/mcheck/mce.c +1 -0
+	return err ? : ubuf - inubuf;
 }

It would be better to put "static DEFINE_MUTEX(mce_read_mutex)" to outside of the function.

And it looks work, but I'd prefer:

	return err ? err : ubuf - inubuf;

OK, I will change this.

Quoted 4 lines

Overall it looks good.
Unfortunately there were updates on tip/mce3 so I could not apply this patch on the top of the branch. I'd appreciate it if you could rebase this patch again.

Yes. I will re-base it.

Best Regards,
Huang Ying

#4 lore raw
Quoted 16 lines · Huang Ying
Quoted 14 lines · Hidetoshi Seto

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines · Huang Ying

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

Quoted 3 lines

It is mainly used by something like kdump, which can search "MACHINECHEC2", and analyze mce_log. mcelog_cpus can help kdump find the real mcelog storage.

Hum, but it help tools only to find one of buffers, not to find all.

I think it would be better help for tools if we have another signature on struct mce_log_cpu, e.g.:

+struct mce_log_cpu {
+ char signature[**]; /* "MCE_LOG_CPU_VER_1" or so */
+ __u32 cpuid; /* cpuid or extcpu, same as struct mce */
+ int head;
+ int tail;
+ unsigned long flags;
+ struct mce entry[MCE_LOG_LEN];
+};

How about this?

Thanks,
H.Seto

author
#5 lore raw
Quoted 35 lines · Hidetoshi Seto
Quoted 16 lines · Huang Ying
Quoted 14 lines · Hidetoshi Seto

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines · Huang Ying

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

Quoted 3 lines

It is mainly used by something like kdump, which can search "MACHINECHEC2", and analyze mce_log. mcelog_cpus can help kdump find the real mcelog storage.

Hum, but it help tools only to find one of buffers, not to find all.

I think it would be better help for tools if we have another signature on struct mce_log_cpu, e.g.:

+struct mce_log_cpu {

(quoted diff) +2 -0
+ 	char signature[**];	/* "MCE_LOG_CPU_VER_1" or so */
+	__u32 cpuid;		/* cpuid or extcpu, same as struct mce */
 +	int head;
 +	int tail;
 +	unsigned long flags;
 +	struct mce entry[MCE_LOG_LEN];
 +};
Quoted 1 line

How about this?

+ mcelog.mcelog_cpus = &per_cpu_var(mce_log_cpus);

So mcelog.mcelog_cpus are pointed to the buffers of all CPUs, not just that of one CPU. You can find them by analyzing PER cpu data structure.

Best Regards,
Huang Ying

#6 lore raw
Quoted 38 lines · Huang Ying
Quoted 34 lines · Hidetoshi Seto
Quoted 15 lines · Huang Ying
Quoted 14 lines · Hidetoshi Seto

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines · Huang Ying

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

Quoted 3 lines

It is mainly used by something like kdump, which can search "MACHINECHEC2", and analyze mce_log. mcelog_cpus can help kdump find the real mcelog storage.

Hum, but it help tools only to find one of buffers, not to find all.

I think it would be better help for tools if we have another signature on struct mce_log_cpu, e.g.:

+struct mce_log_cpu {

(quoted diff) +2 -0
+ 	char signature[**];	/* "MCE_LOG_CPU_VER_1" or so */
+	__u32 cpuid;		/* cpuid or extcpu, same as struct mce */
 +	int head;
 +	int tail;
 +	unsigned long flags;
 +	struct mce entry[MCE_LOG_LEN];
 +};

How about this?

arch/x86/kernel/cpu/mcheck/mce.c +1 -0
+	mcelog.mcelog_cpus = &per_cpu_var(mce_log_cpus);
Quoted 2 lines

So mcelog.mcelog_cpus are pointed to the buffers of all CPUs, not just that of one CPU. You can find them by analyzing PER cpu data structure.

I thought that the signature is used to find the structure without such analyzing. If a tool can analyze PER cpu data, then it likely know where the mce_log_cpu is, so mcelog.mcelog_cpus will not be required.

#define per_cpu_var(var) per_cpu__##var

#define per_cpu(var, cpu) \
        (*SHIFT_PERCPU_PTR(&per_cpu_var(var), per_cpu_offset(cpu)))

OK, I have mistook per_cpu_var() for per_cpu().
Then mcelog.mcelog_cpus does not point any of the buffers, but tells where the buffer is locating in each of PER cpu data.
Tools have to get per_cpu_offset(cpu) to know where the PER cpu data allocated. According to pcpu_alloc_bootmem(), it might consider NUMA. Anyway It will take a bit more effort.

So still I think the mcelog.mcelog_cpus is mostly pointless.

Thanks,
H.Seto

author
#7 lore raw
Quoted 51 lines · Hidetoshi Seto
Quoted 38 lines · Huang Ying
Quoted 34 lines · Hidetoshi Seto
Quoted 15 lines · Huang Ying
Quoted 14 lines · Hidetoshi Seto

arch/x86/include/asm/mce.h struct mce { · line 71

Quoted 10 lines · Huang Ying

struct mce_log {

arch/x86/include/asm/mce.h +2 -3
-	char signature[12]; /* "MACHINECHECK" */
+	char signature[12]; /* "MACHINECHEC2" */
 	unsigned len;	    /* = MCE_LOG_LEN */
-	unsigned next;
 	unsigned flags;
 	unsigned pad0;
-	struct mce entry[MCE_LOG_LEN];
+	struct mce_log_cpu *mcelog_cpus;
 };

What is this *mcelog_cpus to be used for?
It seems it will point one of per-CPU buffers (maybe cpu#0's buffer) if I have read the following mce_log_init() correctly.

Quoted 3 lines

It is mainly used by something like kdump, which can search "MACHINECHEC2", and analyze mce_log. mcelog_cpus can help kdump find the real mcelog storage.

Hum, but it help tools only to find one of buffers, not to find all.

I think it would be better help for tools if we have another signature on struct mce_log_cpu, e.g.:

+struct mce_log_cpu {

(quoted diff) +2 -0
+ 	char signature[**];	/* "MCE_LOG_CPU_VER_1" or so */
+	__u32 cpuid;		/* cpuid or extcpu, same as struct mce */
 +	int head;
 +	int tail;
 +	unsigned long flags;
 +	struct mce entry[MCE_LOG_LEN];
 +};

How about this?

arch/x86/kernel/cpu/mcheck/mce.c +1 -0
+	mcelog.mcelog_cpus = &per_cpu_var(mce_log_cpus);
Quoted 2 lines

So mcelog.mcelog_cpus are pointed to the buffers of all CPUs, not just that of one CPU. You can find them by analyzing PER cpu data structure.

I thought that the signature is used to find the structure without such analyzing. If a tool can analyze PER cpu data, then it likely know where the mce_log_cpu is, so mcelog.mcelog_cpus will not be required.

#define per_cpu_var(var) per_cpu__##var

#define per_cpu(var, cpu) \
        (*SHIFT_PERCPU_PTR(&per_cpu_var(var), per_cpu_offset(cpu)))
Quoted 8 lines

OK, I have mistook per_cpu_var() for per_cpu().
Then mcelog.mcelog_cpus does not point any of the buffers, but tells where the buffer is locating in each of PER cpu data.
Tools have to get per_cpu_offset(cpu) to know where the PER cpu data allocated. According to pcpu_alloc_bootmem(), it might consider NUMA. Anyway It will take a bit more effort.

So still I think the mcelog.mcelog_cpus is mostly pointless.

Emm, it seem hard to analyze mce_log_cpu from per_cpu_var(mce_log_cpus), how about following method:

struct mce_log {
    char signature[12];
    ...
    struct mce_log_cpu *mcelog_cpus[];
};

void mcelog_init(void)

{
    int cpu;
    mcelog.mcelog_cpus = kmalloc(sizeof(void *) * num_possible_cpus();
    for_each_possible_cpu(cpu)
	mcelog.mcelog_cpus[cpu] = &per_cpu(mce_log_cpus, cpu);
}

Best Regards,
Huang Ying

#8 lore raw
Quoted 9 lines · Huang Ying

Emm, it seem hard to analyze mce_log_cpu from per_cpu_var(mce_log_cpus), how about following method:

struct mce_log {
    char signature[12];
    ...
    struct mce_log_cpu *mcelog_cpus[];
};
Quoted 7 lines

void mcelog_init(void)

{
    int cpu;
    mcelog.mcelog_cpus = kmalloc(sizeof(void *) * num_possible_cpus();
    for_each_possible_cpu(cpu)
	mcelog.mcelog_cpus[cpu] = &per_cpu(mce_log_cpus, cpu);
}

It would be good. Using kzalloc() will be better. It might be a bit much, but how about adding:

+ mcelog.nr_mcelog_cpus = num_possible_cpus();

Be careful that mcheck_init(), caller of mcelog_init(), will be called for each booted CPU. You will need:

+ if (mcelog.mcelog_cpus)
+ return;

Thanks,
H.Seto