x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer
PATCH 1/1 x86: MCE: Re-implement MCE log ring buffer as per-CPU ring buffer
Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:
- One ring buffer for each CPU
+ MCEs are added to corresponding local per-CPU buffer, instead of one big global buffer. Contention/unfairness between CPUs is eleminated.
+ MCE records are read out and removed from per-CPU buffers by mutex protected global reader function. Because there are no many readers in system to contend in most cases.
- Per-CPU ring buffer data structure
+ An array is used to hold MCE records. integer "head" indicates next writing position and integer "tail" indicates next reading position.
+ To distinguish buffer empty and full, head and tail wrap to 0 at MCE_LOG_LIMIT instead of MCE_LOG_LEN. Then the real next writing position is head % MCE_LOG_LEN, and real next reading position is tail % MCE_LOG_LEN. If buffer is empty, head == tail, if buffer is full, head % MCE_LOG_LEN == tail % MCE_LOG_LEN and head != tail.
- Lock-less for writer side
+ MCE log writer may come from NMI, so the writer side must be lock-less. For per-CPU buffer of one CPU, writers may come from process, IRQ or NMI context, so "head" is increased with cmpxchg_local() to allocate buffer space.
+ Reader side is protected with a mutex to guarantee only one reader is active in the whole system.
Performance test show that the throughput of per-CPU mcelog buffer can reach 430k records/s compared with 5.3k records/s for original implementation on a 2-core 2.1GHz Core2 machine.
ChangeLog:
v2:
- Use alloc_percpu() to allocate per_cpu mcelog buffer
- Use ndelay to implement witer timeout
--- arch/x86/include/asm/mce.h | 13 + arch/x86/kernel/cpu/mcheck/mce_64.c | 257 ++++++++++++++++++++++-------------- 2 files changed, 169 insertions(+), 101 deletions(-)
arch/x86/include/asm/mce.h +10 -3
@@ -50,15 +50,22 @@ struct mce {* is set. */ -#define MCE_LOG_LEN 32 +#define MCE_LOG_LEN 32 +#define MCE_LOG_LIMIT (MCE_LOG_LEN * 2 - 1) + +static inline int mce_log_index (int n) +{ +return n >=MCE_LOG_LEN ? n -MCE_LOG_LEN : n; +} + +struct mce_log_cpu ;struct mce_log {char signature[12 ];/* "MACHINECHECK" */ unsigned len;/* = MCE_LOG_LEN */ -unsigned next;unsigned flags;unsigned pad0; -struct mce entry[MCE_LOG_LEN ]; +struct mce_log_cpu *mcelog_cpus; };#define MCE_OVERFLOW 0/* bit 0 in flags means overflow */
arch/x86/kernel/cpu/mcheck/mce_64.c +159 -98
326 lines not on this page · raw message
Quoted 6 lines · Huang Ying
ChangeLog:
v2:
- Use
alloc_percpu()to allocateper_cpumcelog buffer
Sorry, why didn't you just use DEFINE_PER_CPU ? That should work as well and will be shorter.
Another thing I noticed. the "MACHINECHECK" signature was originally for crash dump tools to find the log. If you change the format you should change it to MACHINECHEC2 or so.
arch/x86/kernel/cpu/mcheck/mce_64.c static int mce_release(struct inode *ino · line 824
Quoted 6 lines
arch/x86/kernel/cpu/mcheck/mce_64.c +6
-0+size_t usize_limit; + +/* Too large user buffer size may cause system not response */ + usize_limit =num_possible_cpus () *MCE_LOG_LEN *sizeof (struct mce ); +if (usize > usize_limit) + usize = usize_limit;
Did you ever track down what happens here? I still find it worrying
Otherwise looks good.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Huang Ying
ChangeLog:
v2:
- Use
alloc_percpu()to allocateper_cpumcelog bufferSorry, why didn't you just use
DEFINE_PER_CPU? That should work as well and will be shorter.
OK. I can do that.
Quoted 3 lines
Another thing I noticed. the "MACHINECHECK" signature was originally for crash dump tools to find the log. If you change the format you should change it to MACHINECHEC2 or so.
Oh, thanks, I will change this.
Quoted 8 lines
arch/x86/kernel/cpu/mcheck/mce_64.c static int mce_release(struct inode *ino · line 824
Quoted 6 lines
arch/x86/kernel/cpu/mcheck/mce_64.c +6
-0+size_t usize_limit; + +/* Too large user buffer size may cause system not response */ + usize_limit =num_possible_cpus () *MCE_LOG_LEN *sizeof (struct mce ); +if (usize > usize_limit) + usize = usize_limit;Did you ever track down what happens here? I still find it worrying
Maybe the comment is a little confusing. What we do is limit the size of user buffer size to prevent system from not response. If the user buffer size is limited, the time taken by mce_read() can be reasonable, and will not cause system not to response.
But maybe num_possible_cpus() * MCE_LOG_LEN is not a good upper limit, because possible cpus can be fairly large. Maybe we can add another limit, something as follow:
#define MCE_READ_RECORD_LIMIT 512
/*
* Too large user buffer size may cause system not response, so constrain
* the effective size of user buffer
*/usize_limit = num_possible_cpus() * MCE_LOG_LEN * sizeof(struct mce); if (usize_limit > MCE_RECOARD_LIMIT * sizeof(struct mce))
usize_limit = MCE_RECOARD_LIMIT * sizeof(struct mce); if (usize > usize_limit) usize = usize_limit;
Best Regards,
Huang Ying
Quoted 2 lines
Maybe the comment is a little confusing. What we do is limit the size of user buffer size to prevent system from not response. If the user buffer
Ah. The standard way to handle this is to add cond_resched()s where appropiate. Maybe that would be an alternative or is that not possible? You have to drop locks of course.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines · Andi Kleen
Quoted 2 lines
Maybe the comment is a little confusing. What we do is limit the size of user buffer size to prevent system from not response. If the user buffer
Ah. The standard way to handle this is to add cond_resched()s where appropiate. Maybe that would be an alternative or is that not possible? You have to drop locks of course.
Yes, I will use cond_resched() instead.
Best Regards,
Huang Ying
Quoted 32 lines · Huang Ying
Re-implement MCE log ring buffer as per-CPU ring buffer for better scalability. Basic design is as follow:
- One ring buffer for each CPU
+ MCEs are added to corresponding local per-CPU buffer, instead of one big global buffer. Contention/unfairness between CPUs is eleminated.
+ MCE records are read out and removed from per-CPU buffers by mutex protected global reader function. Because there are no many readers in system to contend in most cases.
- Per-CPU ring buffer data structure
+ An array is used to hold MCE records. integer "head" indicates next writing position and integer "tail" indicates next reading position.
+ To distinguish buffer empty and full, head and tail wrap to 0 at
MCE_LOG_LIMITinstead ofMCE_LOG_LEN. Then the real next writing position is head %MCE_LOG_LEN, and real next reading position is tail %MCE_LOG_LEN. If buffer is empty, head == tail, if buffer is full, head %MCE_LOG_LEN== tail %MCE_LOG_LENand head != tail.- Lock-less for writer side
+ MCE log writer may come from NMI, so the writer side must be lock-less. For per-CPU buffer of one CPU, writers may come from process, IRQ or NMI context, so "head" is increased with
cmpxchg_local()to allocate buffer space.
Quoted 7 lines
+ Reader side is protected with a mutex to guarantee only one reader is active in the whole system.
Performance test show that the throughput of per-CPU mcelog buffer can reach 430k records/s compared with 5.3k records/s for original implementation on a 2-core 2.1GHz Core2 machine.
We're talking about Machine Check Exceptions here, right? Is there a valid scenario where you care about performance? I always thought that an MCE meant something seriously went wrong, log the event and reboot the machine -- possibly start ordering replacement parts.
But now you're saying we want to be able to record more than 5.3k events a second on this? Sounds daft to me.
Also, it sounds like something that might fit the ftrace ringbuffer thingy.
Quoted 2 lines
We're talking about Machine Check Exceptions here, right?
No, this is actually more about machine check events (CMCI or poll), not int #18.
Quoted 2 lines
Is there a
valid scenario where you care about performance?
Yes corrected errors. They just get reported while the system keep running.
In practice they can happen in bursts for a relatively short time. Another case was dumb QA setups where people using high frequency injection of corrected errors (that was one case the old code didn't handle too well). Yes I would agree with you that long term high frequency is not an interesting case still.
However there's another reason why this patchkit is very useful. The old code didn't scale to a large number of CPUs, that is why the per CPU buffers are needed.
Quoted 2 lines
Also, it sounds like something that might fit the ftrace ringbuffer thingy.
We went over this before. Please read this relevant threads.
-Andi