x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer
PATCH 1/1 x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer
Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:
- One ring buffer for each CPU
+ MCEs are added to corresponding local per-CPU buffer, instead of one big global buffer. Contention/unfairness between CPUs is eleminated.
+ MCE records are read out and removed from per-CPU buffers by mutex protected global reader function. Because there are no many readers in system to contend in most cases.
- Per-CPU ring buffer data structure
+ An array is used to hold MCE records. integer "head" indicates next writing position and integer "tail" indicates next reading position.
+ To distinguish buffer empty and full, head and tail wrap to 0 at MCE_LOG_LIMIT instead of MCE_LOG_LEN. Then the real next writing position is head % MCE_LOG_LEN, and real next reading position is tail % MCE_LOG_LEN. If buffer is empty, head == tail, if buffer is full, head % MCE_LOG_LEN == tail % MCE_LOG_LEN and head != tail.
- Lock-less for writer side
+ MCE log writer may come from NMI, so the writer side must be lock-less. For per-CPU buffer of one CPU, writers may come from process, IRQ or NMI context, so "head" is increased with cmpxchg_local() to allocate buffer space.
+ Reader side is protected with a mutex to guarantee only one reader is active in the whole system.
Performance test show that the throughput of per-CPU mcelog buffer can reach 430k records/s compared with 5.3k records/s for original implementation on a 2-core 2.1GHz Core2 machine.
ChangeLog:
v3:
- Use DEFINE_PER_CPU to allocate per_cpu mcelog buffer.
- Use cond_resched() to prevent possible system not reponsing issue for large user buffer.
v2:
- Use alloc_percpu() to allocate per_cpu mcelog buffer.
- Use ndelay to implement witer timeout.
--- arch/x86/include/asm/mce.h | 17 +- arch/x86/kernel/cpu/mcheck/mce_64.c | 258 ++++++++++++++++++++++-------------- 2 files changed, 172 insertions(+), 103 deletions(-)
arch/x86/include/asm/mce.h +12 -5
@@ -50,20 +50,27 @@ struct mce {* is set. */ -#define MCE_LOG_LEN 32 +#define MCE_LOG_LEN 32 +#define MCE_LOG_LIMIT (MCE_LOG_LEN * 2 - 1) + +static inline int mce_log_index (int n) +{ +return n >=MCE_LOG_LEN ? n -MCE_LOG_LEN : n; +} + +struct mce_log_cpu ;struct mce_log { -char signature[12 ];/* "MACHINECHECK" */ +char signature[12 ];/* "MACHINECHEC2" */ unsigned len;/* = MCE_LOG_LEN */ -unsigned next;unsigned flags;unsigned pad0; -struct mce entry[MCE_LOG_LEN ]; +struct mce_log_cpu *mcelog_cpus; };#define MCE_OVERFLOW 0/* bit 0 in flags means overflow */ -#define MCE_LOG_SIGNATURE "MACHINECHECK" +#define MCE_LOG_SIGNATURE "MACHINECHEC2"#define MCE_GET_RECORD_LEN _IOR('M', 1, int)#define MCE_GET_LOG_LEN _IOR('M', 2, int)
arch/x86/kernel/cpu/mcheck/mce_64.c +160 -98
327 lines not on this page · raw message
Quoted 3 lines · Huang Ying
Signed-off-by Huang Ying <[email protected]>CC Andi Kleen <[email protected]>
Looks good. I think this should go into 2.6.31
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.