x86/RAS: Correctable Errors Collector
Subsystems (3)
Borislav Petkov
maintainercover x86/RAS: Correctable Errors Collector
From: Borislav Petkov <[email protected]>
Hi guys,
here's v1, all feedback I know of has been addressed. So I guess it is time. :)
We don't have it default y yet but will make it so after it has seen wider testing. The end goal is to have it running by default so that transient correctable ECC errors don't generate error logs and upset people unnecessarily.
The other good thing resulting from this patchset is that we have *all* MCE consumers lined up in a notifier with priorities. This way we have a single chain which gets to see error records and not some wild variety of hooks here and there.
Last but not least, /dev/mcelog has been deprecated and all the code is behind a CONFIG_X86_MCELOG config option.
Btw, patch 1 is for urgent.
Please apply,
thanks.
Changelog:
=========
v0:
here's the latest incarnation of the CEC collector. I think I've taken care of all review comments but feel free to correct me here. The introductory comment in cec.c should explain the whole deal - I'm referring to there so that we have that text in the actual source and not spread it around commit messages. So pls have a look there for more info.
The thing has knobs in debugfs now which can control its operation, I hope I've chosen sane default values.
Andi Kleen (1):
x86/mce: Don't print MCEs when mcelog is active
Borislav Petkov (4):
x86/MCE: Rename mce_log()'s argument x86/MCE: Rename mce_log to mce_log_buffer
RAS: Add a Corrected Errors Collector x86/mce: Do not register notifiers with invalid prio
Tony Luck (1):
x86/mce: Deprecate /dev/mcelog
Documentation/admin-guide/kernel-parameters.txt | 6 + arch/x86/Kconfig | 10 +- arch/x86/include/asm/mce.h | 12 +- arch/x86/kernel/cpu/mcheck/Makefile | 2 + arch/x86/kernel/cpu/mcheck/dev-mcelog.c | 397 ++++++++++++++++++ arch/x86/kernel/cpu/mcheck/mce-internal.h | 8 + arch/x86/kernel/cpu/mcheck/mce.c | 501 +++++----------------- arch/x86/ras/Kconfig | 14 + drivers/ras/Makefile | 3 +- drivers/ras/cec.c | 532 ++++++++++++++++++++++++ drivers/ras/debugfs.c | 2 +- drivers/ras/debugfs.h | 8 + drivers/ras/ras.c | 11 + include/linux/ras.h | 13 +- 14 files changed, 1105 insertions(+), 414 deletions(-) create mode 100644 arch/x86/kernel/cpu/mcheck/dev-mcelog.c create mode 100644 drivers/ras/cec.c create mode 100644 drivers/ras/debugfs.h
Signature · 1 line
2.11.0
Borislav Petkov
maintainerPATCH 1/6 x86/mce: Don't print MCEs when mcelog is active
From: Andi Kleen <[email protected]>
Since
cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")
all MCEs are printed even when mcelog is running. Fix the regression to not print to dmesg when mcelog is running as it is a consumer too.
--- arch/x86/kernel/cpu/mcheck/mce.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/mcheck/mce.c +6 -1
@@ -55,6 +55,8 @@static DEFINE_MUTEX (mce_chrdev_read_mutex); +static int mce_chrdev_open_count;/* #times opened */ +#define mce_log_get_idx_check (p) \ ({ \ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
@@ -599,6 +601,10 @@ static int mce_default_notifier(struct notifier_block *nb, unsigned long val,if (atomic_read (&num_notifiers) >2 )return NOTIFY_DONE ; +/* Don't print when mcelog is running */ +if (mce_chrdev_open_count >0 ) +return NOTIFY_DONE ; +__print_mce (m);return NOTIFY_DONE ;
@@ -1848,7 +1854,6 @@ void mcheck_cpu_clear(struct cpuinfo_x86 *c)*/ static DEFINE_SPINLOCK (mce_chrdev_state_lock); -static int mce_chrdev_open_count;/* #times opened */ static int mce_chrdev_open_exclu;/* already open exclusive? */ static int mce_chrdev_open (struct inode *inode,struct file *file)
Signature · 1 line
2.11.0
tip-bot for Andi Kleen
merge bot[tip:ras/core] x86/mce: Don't print MCEs when mcelog is active
x86/mce: Don't print MCEs when mcelog is active
Commit-ID: cc66afea58f858ff6da7f79b8a595a67bbb4f9a9 Gitweb: http://git.kernel.org/tip/cc66afea58f858ff6da7f79b8a595a67bbb4f9a9 Author: Andi Kleen <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:32:59 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:53:52 +0200
x86/mce: Don't print MCEs when mcelog is active
Since:
cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")
all MCEs are printed even when mcelog is running. Fix the regression to not print to dmesg when mcelog is running as it is a consumer too.
--- arch/x86/kernel/cpu/mcheck/mce.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/mcheck/mce.c +6 -1
@@ -54,6 +54,8 @@static DEFINE_MUTEX (mce_chrdev_read_mutex); +static int mce_chrdev_open_count;/* #times opened */ +#define mce_log_get_idx_check (p) \ ({ \ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
@@ -598,6 +600,10 @@ static int mce_default_notifier(struct notifier_block *nb, unsigned long val,if (atomic_read (&num_notifiers) >2 )return NOTIFY_DONE ; +/* Don't print when mcelog is running */ +if (mce_chrdev_open_count >0 ) +return NOTIFY_DONE ; +__print_mce (m);return NOTIFY_DONE ;
@@ -1828,7 +1834,6 @@ void mcheck_cpu_clear(struct cpuinfo_x86 *c)*/ static DEFINE_SPINLOCK (mce_chrdev_state_lock); -static int mce_chrdev_open_count;/* #times opened */ static int mce_chrdev_open_exclu;/* already open exclusive? */ static int mce_chrdev_open (struct inode *inode,struct file *file)
Borislav Petkov
maintainerPATCH 2/6 x86/MCE: Rename mce_log()'s argument
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
We call it everywhere "struct mce *m". Adjust that here too to avoid confusion.
No functionality change.
--- arch/x86/kernel/cpu/mcheck/mce.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +4 -4
@@ -158,14 +158,14 @@ static struct mce_log mcelog = { .recordlen = sizeof(struct mce), }; -void mce_log (struct mce *mce) +void mce_log (struct mce *m) {unsigned next, entry;/* Emit the trace record: */ -trace_mce_record (mce); +trace_mce_record (m); -if (!mce_gen_pool_add (mce)) +if (!mce_gen_pool_add (m))irq_work_queue (&mce_irq_work);wmb ();
@@ -195,7 +195,7 @@ void mce_log(struct mce *mce)if (cmpxchg (&mcelog.next, entry, next) == entry)break ; } -memcpy (mcelog.entry + entry, mce,sizeof (struct mce )); +memcpy (mcelog.entry + entry, m,sizeof (struct mce ));wmb (); mcelog.entry[entry].finished =1 ;wmb ();
Signature · 1 line
2.11.0
tip-bot for Borislav Petkov
merge bot[tip:ras/core] x86/mce: Rename mce_log()'s argument
x86/mce: Rename mce_log()'s argument
Commit-ID: fe3ed20fddfd8c6a4dcfb8e43de80f269d1f3f2a Gitweb: http://git.kernel.org/tip/fe3ed20fddfd8c6a4dcfb8e43de80f269d1f3f2a Author: Borislav Petkov <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:33:00 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:54:38 +0200
x86/mce: Rename mce_log()'s argument
We call it everywhere "struct mce *m". Adjust that here too to avoid confusion.
No functionality change.
--- arch/x86/kernel/cpu/mcheck/mce.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +4 -4
@@ -158,14 +158,14 @@ static struct mce_log mcelog = { .recordlen = sizeof(struct mce), }; -void mce_log (struct mce *mce) +void mce_log (struct mce *m) {unsigned next, entry;/* Emit the trace record: */ -trace_mce_record (mce); +trace_mce_record (m); -if (!mce_gen_pool_add (mce)) +if (!mce_gen_pool_add (m))irq_work_queue (&mce_irq_work);wmb ();
@@ -195,7 +195,7 @@ void mce_log(struct mce *mce)if (cmpxchg (&mcelog.next, entry, next) == entry)break ; } -memcpy (mcelog.entry + entry, mce,sizeof (struct mce )); +memcpy (mcelog.entry + entry, m,sizeof (struct mce ));wmb (); mcelog.entry[entry].finished =1 ;wmb ();
Borislav Petkov
maintainerPATCH 3/6 x86/MCE: Rename mce_log to mce_log_buffer
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
It is confusing when staring at "struct mce_log mcelog" and then there's also a function called mce_log(). So call the buffer what it is.
No functionality change.
--- arch/x86/include/asm/mce.h | 2 +- arch/x86/kernel/cpu/mcheck/mce.c | 30 +++++++++++++++--------------- 2 files changed, 16 insertions(+), 16 deletions(-)
arch/x86/include/asm/mce.h +1 -1
@@ -128,7 +128,7 @@* debugging tools. Each entry is only valid when its finished flag * is set. */ -struct mce_log { +struct mce_log_buffer {char signature[12 ];/* "MACHINECHECK" */ unsigned len;/* = MCE_LOG_LEN */ unsigned next;
arch/x86/kernel/cpu/mcheck/mce.c +15 -15
@@ -152,7 +152,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);* separate MCEs from kernel messages to avoid bogus bug reports. */ -static struct mce_log mcelog = { +static struct mce_log_buffer mcelog_buf = { .signature =MCE_LOG_SIGNATURE , .len =MCE_LOG_LEN , .recordlen =sizeof (struct mce ),
@@ -170,7 +170,7 @@ void mce_log(struct mce *m)wmb ();for (;;) { - entry =mce_log_get_idx_check (mcelog.next); + entry =mce_log_get_idx_check (mcelog_buf.next);for (;;) {/*
@@ -180,11 +180,11 @@ void mce_log(struct mce *m)*/ if (entry >=MCE_LOG_LEN ) {set_bit (MCE_OVERFLOW , - (unsigned long *)&mcelog.flags); + (unsigned long *)&mcelog_buf.flags);return ; }/* Old left over entry. Skip: */ -if (mcelog.entry[entry].finished) { +if (mcelog_buf.entry[entry].finished) { entry++;continue ; }
@@ -192,12 +192,12 @@ void mce_log(struct mce *m) }smp_rmb (); next = entry +1 ; -if (cmpxchg (&mcelog.next, entry, next) == entry) +if (cmpxchg (&mcelog_buf.next, entry, next) == entry)break ; } -memcpy (mcelog.entry + entry, m,sizeof (struct mce )); +memcpy (mcelog_buf.entry + entry, m,sizeof (struct mce ));wmb (); - mcelog.entry[entry].finished =1 ; + mcelog_buf.entry[entry].finished =1 ;wmb ();set_bit (0 , &mce_need_notify);
@@ -1958,7 +1958,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf, goto out; } - next =mce_log_get_idx_check (mcelog.next); + next =mce_log_get_idx_check (mcelog_buf.next);/* Only supports full reads right now */ err = -EINVAL ;
@@ -1970,7 +1970,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf, do { for (i = prev; i < next; i++) { unsigned long start = jiffies; - struct mce *m = &mcelog.entry[i]; + struct mce *m = &mcelog_buf.entry[i]; while (!m->finished) { if (time_after_eq(jiffies, start + 2)) {
@@ -1986,10 +1986,10 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf, ; } -memset (mcelog.entry + prev,0 , +memset (mcelog_buf.entry + prev,0 , (next - prev) *sizeof (struct mce )); prev = next; - next =cmpxchg (&mcelog.next, prev,0 ); + next =cmpxchg (&mcelog_buf.next, prev,0 ); }while (next != prev);synchronize_sched ();
@@ -2001,7 +2001,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,on_each_cpu (collect_tscs, cpu_tsc,1 );for (i = next; i <MCE_LOG_LEN ; i++) { -struct mce *m = &mcelog.entry[i]; +struct mce *m = &mcelog_buf.entry[i];if (m->finished && m->tsc < cpu_tsc[m->cpu]) { err |=copy_to_user (buf, m,sizeof (*m));
@@ -2024,7 +2024,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,static unsigned int mce_chrdev_poll (struct file *file,poll_table *wait) {poll_wait (file, &mce_chrdev_wait, wait); -if (READ_ONCE (mcelog.next)) +if (READ_ONCE (mcelog_buf.next))return POLLIN |POLLRDNORM ;if (!mce_apei_read_done &&apei_check_mce ())return POLLIN |POLLRDNORM ;
@@ -2048,8 +2048,8 @@ static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,unsigned flags;do { - flags = mcelog.flags; - }while (cmpxchg (&mcelog.flags, flags,0 ) != flags); + flags = mcelog_buf.flags; + }while (cmpxchg (&mcelog_buf.flags, flags,0 ) != flags);return put_user (flags, p); }
Signature · 1 line
2.11.0
tip-bot for Borislav Petkov
merge bot[tip:ras/core] x86/mce: Rename mce_log to mce_log_buffer
x86/mce: Rename mce_log to mce_log_buffer
Commit-ID: e64edfcce9c738300b4102d0739577d6ecc96d4a Gitweb: http://git.kernel.org/tip/e64edfcce9c738300b4102d0739577d6ecc96d4a Author: Borislav Petkov <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:33:01 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:54:42 +0200
x86/mce: Rename mce_log to mce_log_buffer
It is confusing when staring at "struct mce_log mcelog" and then there's also a function called mce_log(). So call the buffer what it is.
No functionality change.
--- arch/x86/include/asm/mce.h | 2 +- arch/x86/kernel/cpu/mcheck/mce.c | 30 +++++++++++++++--------------- 2 files changed, 16 insertions(+), 16 deletions(-)
arch/x86/include/asm/mce.h +1 -1
@@ -128,7 +128,7 @@* debugging tools. Each entry is only valid when its finished flag * is set. */ -struct mce_log { +struct mce_log_buffer {char signature[12 ];/* "MACHINECHECK" */ unsigned len;/* = MCE_LOG_LEN */ unsigned next;
arch/x86/kernel/cpu/mcheck/mce.c +15 -15
@@ -152,7 +152,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);* separate MCEs from kernel messages to avoid bogus bug reports. */ -static struct mce_log mcelog = { +static struct mce_log_buffer mcelog_buf = { .signature =MCE_LOG_SIGNATURE , .len =MCE_LOG_LEN , .recordlen =sizeof (struct mce ),
@@ -170,7 +170,7 @@ void mce_log(struct mce *m)wmb ();for (;;) { - entry =mce_log_get_idx_check (mcelog.next); + entry =mce_log_get_idx_check (mcelog_buf.next);for (;;) {/*
@@ -180,11 +180,11 @@ void mce_log(struct mce *m)*/ if (entry >=MCE_LOG_LEN ) {set_bit (MCE_OVERFLOW , - (unsigned long *)&mcelog.flags); + (unsigned long *)&mcelog_buf.flags);return ; }/* Old left over entry. Skip: */ -if (mcelog.entry[entry].finished) { +if (mcelog_buf.entry[entry].finished) { entry++;continue ; }
@@ -192,12 +192,12 @@ void mce_log(struct mce *m) }smp_rmb (); next = entry +1 ; -if (cmpxchg (&mcelog.next, entry, next) == entry) +if (cmpxchg (&mcelog_buf.next, entry, next) == entry)break ; } -memcpy (mcelog.entry + entry, m,sizeof (struct mce )); +memcpy (mcelog_buf.entry + entry, m,sizeof (struct mce ));wmb (); - mcelog.entry[entry].finished =1 ; + mcelog_buf.entry[entry].finished =1 ;wmb ();set_bit (0 , &mce_need_notify);
@@ -1958,7 +1958,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf, goto out; } - next =mce_log_get_idx_check (mcelog.next); + next =mce_log_get_idx_check (mcelog_buf.next);/* Only supports full reads right now */ err = -EINVAL ;
@@ -1970,7 +1970,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf, do { for (i = prev; i < next; i++) { unsigned long start = jiffies; - struct mce *m = &mcelog.entry[i]; + struct mce *m = &mcelog_buf.entry[i]; while (!m->finished) { if (time_after_eq(jiffies, start + 2)) {
@@ -1986,10 +1986,10 @@ timeout: ; } -memset (mcelog.entry + prev,0 , +memset (mcelog_buf.entry + prev,0 , (next - prev) *sizeof (struct mce )); prev = next; - next =cmpxchg (&mcelog.next, prev,0 ); + next =cmpxchg (&mcelog_buf.next, prev,0 ); }while (next != prev);synchronize_sched ();
@@ -2001,7 +2001,7 @@ timeout:on_each_cpu (collect_tscs, cpu_tsc,1 );for (i = next; i <MCE_LOG_LEN ; i++) { -struct mce *m = &mcelog.entry[i]; +struct mce *m = &mcelog_buf.entry[i];if (m->finished && m->tsc < cpu_tsc[m->cpu]) { err |=copy_to_user (buf, m,sizeof (*m));
@@ -2024,7 +2024,7 @@ out:static unsigned int mce_chrdev_poll (struct file *file,poll_table *wait) {poll_wait (file, &mce_chrdev_wait, wait); -if (READ_ONCE (mcelog.next)) +if (READ_ONCE (mcelog_buf.next))return POLLIN |POLLRDNORM ;if (!mce_apei_read_done &&apei_check_mce ())return POLLIN |POLLRDNORM ;
@@ -2048,8 +2048,8 @@ static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,unsigned flags;do { - flags = mcelog.flags; - }while (cmpxchg (&mcelog.flags, flags,0 ) != flags); + flags = mcelog_buf.flags; + }while (cmpxchg (&mcelog_buf.flags, flags,0 ) != flags);return put_user (flags, p); }
Borislav Petkov
maintainerPATCH 4/6 RAS: Add a Corrected Errors Collector
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
A simple data structure for collecting correctable errors along with accessors. More detailed description in the code itself.
The error decoding is done with the decoding chain now and mce_first_notifier() gets to see the error first and the CEC decides whether to log it and then the rest of the chain doesn't hear about it - basically the main reason for the CE collector - or to continue running the notifiers.
When the CEC hits the action threshold, it will try to soft-offine the page containing the ECC and then the whole decoding chain gets to see the error.
--- Documentation/admin-guide/kernel-parameters.txt | 6 + arch/x86/include/asm/mce.h | 9 +- arch/x86/kernel/cpu/mcheck/mce.c | 191 +++++---- arch/x86/ras/Kconfig | 14 + drivers/ras/Makefile | 3 +- drivers/ras/cec.c | 532 ++++++++++++++++++++++++ drivers/ras/debugfs.c | 2 +- drivers/ras/debugfs.h | 8 + drivers/ras/ras.c | 11 + include/linux/ras.h | 13 +- 10 files changed, 706 insertions(+), 83 deletions(-) create mode 100644 drivers/ras/cec.c create mode 100644 drivers/ras/debugfs.h
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -3173,6 +3173,12 @@ ramdisk_size= [RAM] Sizes of RAM disks in kilobytes See Documentation/blockdev/ramdisk.txt. + ras=option[,option,...] [KNL] RAS-specific options + + cec_disable [X86] + Disable the Correctable Errors Collector, + see CONFIG_RAS_CEC help text. + rcu_nocbs= [KNL] The argument is a cpu list, as described above.
arch/x86/include/asm/mce.h +5 -4
@@ -191,10 +191,11 @@ extern struct mca_config mca_cfg;extern struct mca_msr_regs msr_ops;enum mce_notifier_prios { -MCE_PRIO_SRAO =INT_MAX , -MCE_PRIO_EXTLOG =INT_MAX -1 , -MCE_PRIO_NFIT =INT_MAX -2 , -MCE_PRIO_EDAC =INT_MAX -3 , +MCE_PRIO_FIRST =INT_MAX , +MCE_PRIO_SRAO =INT_MAX -1 , +MCE_PRIO_EXTLOG =INT_MAX -2 , +MCE_PRIO_NFIT =INT_MAX -3 , +MCE_PRIO_EDAC =INT_MAX -4 ,MCE_PRIO_LOWEST =0 , };
arch/x86/kernel/cpu/mcheck/mce.c +115 -76
247 lines not on this page · raw message
arch/x86/ras/Kconfig +14 -0
@@ -7,3 +7,17 @@ config MCE_AMD_INJ aspects of the MCE handling code. WARNING: Do not even assume this interface is staying stable! + +config RAS_CEC + bool "Correctable Errors Collector" + depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS + ---help--- + This is a small cache which collects correctable memory errors per 4K + page PFN and counts their repeated occurrence. Once the counter for a + PFN overflows, we try to soft-offline that page as we take it to mean + that it has reached a relatively high error count and would probably + be best if we don't use it anymore. + + Bear in mind that this is absolutely useless if your platform doesn't + have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS. +
drivers/ras/Makefile +2 -1
@@ -1 +1,2 @@ -obj-$(CONFIG_RAS) += ras.o debugfs.o +obj-$(CONFIG_RAS) += ras.o debugfs.o +obj-$(CONFIG_RAS_CEC) += cec.o
drivers/ras/cec.c new file +532 -0
532 lines not on this page · raw message
drivers/ras/debugfs.c +1 -1
@@ -1,6 +1,6 @@#include <linux/debugfs.h> -static struct dentry *ras_debugfs_dir; +struct dentry *ras_debugfs_dir;static atomic_t trace_count =ATOMIC_INIT (0 );
drivers/ras/debugfs.h new file +8 -0
@@ -0,0 +1,8 @@ +#ifndef __RAS_DEBUGFS_H__ +#define __RAS_DEBUGFS_H__ + +#include <linux/debugfs.h> + +extern struct dentry *ras_debugfs_dir; + +#endif /* __RAS_DEBUGFS_H__ */
drivers/ras/ras.c +11 -0
@@ -27,3 +27,14 @@ subsys_initcall(ras_init);EXPORT_TRACEPOINT_SYMBOL_GPL (extlog_mem_event);#endif EXPORT_TRACEPOINT_SYMBOL_GPL (mc_event); + + +int __init parse_ras_param (char *str) +{ +#ifdef CONFIG_RAS_CEC +parse_cec_param (str); +#endif + +return 1 ; +} +__setup ("ras" , parse_ras_param);
include/linux/ras.h +12 -1
@@ -1,14 +1,25 @@#ifndef __RAS_H__#define __RAS_H__ +#include <asm/errno.h> +#ifdef CONFIG_DEBUG_FS int ras_userspace_consumers (void );void ras_debugfs_init (void );int ras_add_daemon_trace (void );#else static inline int ras_userspace_consumers (void ) {return 0 ; } -static inline void ras_debugfs_init (void ) {return ; } +static inline void ras_debugfs_init (void ) { }static inline int ras_add_daemon_trace (void ) {return 0 ; }#endif +#ifdef CONFIG_RAS_CEC +void __init cec_init (void ); +int __init parse_cec_param (char *str); +int cec_add_elem (u64 pfn); +#else +static inline void __init cec_init (void ) { } +static inline int cec_add_elem (u64 pfn) {return -ENODEV ; }#endif + +#endif /* __RAS_H__ */
Signature · 1 line
2.11.0
tip-bot for Borislav Petkov
merge bot[tip:ras/core] RAS: Add a Corrected Errors Collector
RAS: Add a Corrected Errors Collector
Commit-ID: 011d8261117249eab97bc86a8e1ac7731e03e319 Gitweb: http://git.kernel.org/tip/011d8261117249eab97bc86a8e1ac7731e03e319 Author: Borislav Petkov <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:33:02 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:54:48 +0200
RAS: Add a Corrected Errors Collector
Introduce a simple data structure for collecting correctable errors along with accessors. More detailed description in the code itself.
The error decoding is done with the decoding chain now and mce_first_notifier() gets to see the error first and the CEC decides whether to log it and then the rest of the chain doesn't hear about it - basically the main reason for the CE collector - or to continue running the notifiers.
When the CEC hits the action threshold, it will try to soft-offine the page containing the ECC and then the whole decoding chain gets to see the error.
--- Documentation/admin-guide/kernel-parameters.txt | 6 + arch/x86/include/asm/mce.h | 9 +- arch/x86/kernel/cpu/mcheck/mce.c | 191 +++++---- arch/x86/ras/Kconfig | 14 + drivers/ras/Makefile | 3 +- drivers/ras/cec.c | 532 ++++++++++++++++++++++++ drivers/ras/debugfs.c | 2 +- drivers/ras/debugfs.h | 8 + drivers/ras/ras.c | 11 + include/linux/ras.h | 13 +- 10 files changed, 706 insertions(+), 83 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -3172,6 +3172,12 @@ ramdisk_size= [RAM] Sizes of RAM disks in kilobytes See Documentation/blockdev/ramdisk.txt. + ras=option[,option,...] [KNL] RAS-specific options + + cec_disable [X86] + Disable the Correctable Errors Collector, + see CONFIG_RAS_CEC help text. + rcu_nocbs= [KNL] The argument is a cpu list, as described above.
arch/x86/include/asm/mce.h +5 -4
@@ -191,10 +191,11 @@ extern struct mca_config mca_cfg;extern struct mca_msr_regs msr_ops;enum mce_notifier_prios { -MCE_PRIO_SRAO =INT_MAX , -MCE_PRIO_EXTLOG =INT_MAX -1 , -MCE_PRIO_NFIT =INT_MAX -2 , -MCE_PRIO_EDAC =INT_MAX -3 , +MCE_PRIO_FIRST =INT_MAX , +MCE_PRIO_SRAO =INT_MAX -1 , +MCE_PRIO_EXTLOG =INT_MAX -2 , +MCE_PRIO_NFIT =INT_MAX -3 , +MCE_PRIO_EDAC =INT_MAX -4 ,MCE_PRIO_LOWEST =0 , };
arch/x86/kernel/cpu/mcheck/mce.c +115 -76
247 lines not on this page · raw message
arch/x86/ras/Kconfig +14 -0
@@ -7,3 +7,17 @@ config MCE_AMD_INJ aspects of the MCE handling code. WARNING: Do not even assume this interface is staying stable! + +config RAS_CEC + bool "Correctable Errors Collector" + depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS + ---help--- + This is a small cache which collects correctable memory errors per 4K + page PFN and counts their repeated occurrence. Once the counter for a + PFN overflows, we try to soft-offline that page as we take it to mean + that it has reached a relatively high error count and would probably + be best if we don't use it anymore. + + Bear in mind that this is absolutely useless if your platform doesn't + have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS. +
drivers/ras/Makefile +2 -1
@@ -1 +1,2 @@ -obj-$(CONFIG_RAS) += ras.o debugfs.o +obj-$(CONFIG_RAS) += ras.o debugfs.o +obj-$(CONFIG_RAS_CEC) += cec.o
drivers/ras/cec.c new file +532 -0
532 lines not on this page · raw message
drivers/ras/debugfs.c +1 -1
@@ -1,6 +1,6 @@#include <linux/debugfs.h> -static struct dentry *ras_debugfs_dir; +struct dentry *ras_debugfs_dir;static atomic_t trace_count =ATOMIC_INIT (0 );
drivers/ras/debugfs.h new file +8 -0
@@ -0,0 +1,8 @@ +#ifndef __RAS_DEBUGFS_H__ +#define __RAS_DEBUGFS_H__ + +#include <linux/debugfs.h> + +extern struct dentry *ras_debugfs_dir; + +#endif /* __RAS_DEBUGFS_H__ */
drivers/ras/ras.c +11 -0
@@ -27,3 +27,14 @@ subsys_initcall(ras_init);EXPORT_TRACEPOINT_SYMBOL_GPL (extlog_mem_event);#endif EXPORT_TRACEPOINT_SYMBOL_GPL (mc_event); + + +int __init parse_ras_param (char *str) +{ +#ifdef CONFIG_RAS_CEC +parse_cec_param (str); +#endif + +return 1 ; +} +__setup ("ras" , parse_ras_param);
include/linux/ras.h +12 -1
@@ -1,14 +1,25 @@#ifndef __RAS_H__#define __RAS_H__ +#include <asm/errno.h> +#ifdef CONFIG_DEBUG_FS int ras_userspace_consumers (void );void ras_debugfs_init (void );int ras_add_daemon_trace (void );#else static inline int ras_userspace_consumers (void ) {return 0 ; } -static inline void ras_debugfs_init (void ) {return ; } +static inline void ras_debugfs_init (void ) { }static inline int ras_add_daemon_trace (void ) {return 0 ; }#endif +#ifdef CONFIG_RAS_CEC +void __init cec_init (void ); +int __init parse_cec_param (char *str); +int cec_add_elem (u64 pfn); +#else +static inline void __init cec_init (void ) { } +static inline int cec_add_elem (u64 pfn) {return -ENODEV ; }#endif + +#endif /* __RAS_H__ */
Borislav Petkov
maintainerPATCH 5/6 x86/mce: Deprecate /dev/mcelog
In reply to Borislav PetkovFrom: Tony Luck <[email protected]>
Move all code relating to /dev/mcelog to a separate source file. /dev/mcelog driver can now operate from the machine check notifier with lowest prio.
Boris:
* Move the mce_helper and trigger functionality behind CONFIG_X86_MCELOG.
--- arch/x86/Kconfig | 10 +- arch/x86/include/asm/mce.h | 1 + arch/x86/kernel/cpu/mcheck/Makefile | 2 + arch/x86/kernel/cpu/mcheck/dev-mcelog.c | 397 ++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/mcheck/mce-internal.h | 8 + arch/x86/kernel/cpu/mcheck/mce.c | 372 +--------------------------- 6 files changed, 426 insertions(+), 364 deletions(-) create mode 100644 arch/x86/kernel/cpu/mcheck/dev-mcelog.c
arch/x86/Kconfig +9 -1
@@ -1044,6 +1044,14 @@ config X86_MCE The action the kernel takes depends on the severity of the problem, ranging from warning messages to halting the machine. +config X86_MCELOG + bool "Support for deprecated /dev/mcelog character device" + depends on X86_MCE + ---help--- + Enable support for /dev/mcelog which is needed by the old mcelog + userspace logging daemon. Consider switching to the new generation + rasdaemon solution. + config X86_MCE_INTEL def_bool y prompt "Intel MCE features"
@@ -1073,7 +1081,7 @@ config X86_MCE_THRESHOLD def_bool y config X86_MCE_INJECT - depends on X86_MCE && X86_LOCAL_APIC + depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG tristate "Machine check injector support" ---help--- Provide support for injecting machine checks for testing purposes.
arch/x86/include/asm/mce.h +1 -0
@@ -196,6 +196,7 @@ enum mce_notifier_prios {MCE_PRIO_EXTLOG =INT_MAX -2 ,MCE_PRIO_NFIT =INT_MAX -3 ,MCE_PRIO_EDAC =INT_MAX -4 , +MCE_PRIO_MCELOG =1 , MCE_PRIO_LOWEST = 0, };
arch/x86/kernel/cpu/mcheck/Makefile +2 -0
@@ -9,3 +9,5 @@ obj-$(CONFIG_X86_MCE_INJECT) += mce-inject.o obj-$(CONFIG_X86_THERMAL_VECTOR) += therm_throt.o obj-$(CONFIG_ACPI_APEI) += mce-apei.o + +obj-$(CONFIG_X86_MCELOG) += dev-mcelog.o
arch/x86/kernel/cpu/mcheck/dev-mcelog.c new file +397 -0
397 lines not on this page · raw message
arch/x86/kernel/cpu/mcheck/mce-internal.h +8 -0
@@ -96,3 +96,11 @@ static inline bool mce_cmp(struct mce *m1, struct mce *m2) m1->addr != m2->addr || m1->misc != m2->misc; } + +extern struct device_attribute dev_attr_trigger; + +#ifdef CONFIG_X86_MCELOG +extern void mce_work_trigger (void ); +#else +static inline void mce_work_trigger (void ) { } +#endif
arch/x86/kernel/cpu/mcheck/mce.c +9 -363
476 lines not on this page · raw message
Signature · 1 line
2.11.0
tip-bot for Tony Luck
merge bot[tip:ras/core] x86/mce: Factor out and deprecate the /dev/mcelog driver
x86/mce: Factor out and deprecate the /dev/mcelog driver
Commit-ID: 5de97c9f6d85fd83af76e09e338b18e7adb1ae60 Gitweb: http://git.kernel.org/tip/5de97c9f6d85fd83af76e09e338b18e7adb1ae60 Author: Tony Luck <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:33:03 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:55:01 +0200
x86/mce: Factor out and deprecate the /dev/mcelog driver
Move all code relating to /dev/mcelog to a separate source file. /dev/mcelog driver can now operate from the machine check notifier with lowest prio.
--- arch/x86/Kconfig | 10 +- arch/x86/include/asm/mce.h | 1 + arch/x86/kernel/cpu/mcheck/Makefile | 2 + arch/x86/kernel/cpu/mcheck/dev-mcelog.c | 397 ++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/mcheck/mce-internal.h | 8 + arch/x86/kernel/cpu/mcheck/mce.c | 372 +--------------------------- 6 files changed, 426 insertions(+), 364 deletions(-)
arch/x86/Kconfig +9 -1
@@ -1043,6 +1043,14 @@ config X86_MCE The action the kernel takes depends on the severity of the problem, ranging from warning messages to halting the machine. +config X86_MCELOG_LEGACY + bool "Support for deprecated /dev/mcelog character device" + depends on X86_MCE + ---help--- + Enable support for /dev/mcelog which is needed by the old mcelog + userspace logging daemon. Consider switching to the new generation + rasdaemon solution. + config X86_MCE_INTEL def_bool y prompt "Intel MCE features"
@@ -1072,7 +1080,7 @@ config X86_MCE_THRESHOLD def_bool y config X86_MCE_INJECT - depends on X86_MCE && X86_LOCAL_APIC + depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG_LEGACY tristate "Machine check injector support" ---help--- Provide support for injecting machine checks for testing purposes.
arch/x86/include/asm/mce.h +1 -0
@@ -196,6 +196,7 @@ enum mce_notifier_prios {MCE_PRIO_EXTLOG =INT_MAX -2 ,MCE_PRIO_NFIT =INT_MAX -3 ,MCE_PRIO_EDAC =INT_MAX -4 , +MCE_PRIO_MCELOG =1 , MCE_PRIO_LOWEST = 0, };
arch/x86/kernel/cpu/mcheck/Makefile +2 -0
@@ -9,3 +9,5 @@ obj-$(CONFIG_X86_MCE_INJECT) += mce-inject.o obj-$(CONFIG_X86_THERMAL_VECTOR) += therm_throt.o obj-$(CONFIG_ACPI_APEI) += mce-apei.o + +obj-$(CONFIG_X86_MCELOG_LEGACY) += dev-mcelog.o
arch/x86/kernel/cpu/mcheck/dev-mcelog.c new file +397 -0
397 lines not on this page · raw message
arch/x86/kernel/cpu/mcheck/mce-internal.h +8 -0
@@ -96,3 +96,11 @@ static inline bool mce_cmp(struct mce *m1, struct mce *m2) m1->addr != m2->addr || m1->misc != m2->misc; } + +extern struct device_attribute dev_attr_trigger; + +#ifdef CONFIG_X86_MCELOG_LEGACY +extern void mce_work_trigger (void ); +#else +static inline void mce_work_trigger (void ) { } +#endif
arch/x86/kernel/cpu/mcheck/mce.c +9 -363
476 lines not on this page · raw message
Borislav Petkov
maintainerPATCH 6/6 x86/mce: Do not register notifiers with invalid prio
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
This is just a defensive precaution: do not register notifiers with a priority which would disrupt the error handling in the notifiers with prio higher than MCE_PRIO_EDAC.
--- arch/x86/kernel/cpu/mcheck/mce.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +3 -2
@@ -157,9 +157,10 @@ static atomic_t num_notifiers;void mce_register_decode_chain (struct notifier_block *nb) { -atomic_inc (&num_notifiers); +if (WARN_ON (nb->priority >MCE_PRIO_LOWEST && nb->priority <MCE_PRIO_EDAC )) +return ; -WARN_ON (nb->priority >MCE_PRIO_LOWEST && nb->priority <MCE_PRIO_EDAC ); +atomic_inc (&num_notifiers);atomic_notifier_chain_register (&x86_mce_decoder_chain, nb); }
Signature · 1 line
2.11.0
tip-bot for Borislav Petkov
merge bot[tip:ras/core] x86/mce: Do not register notifiers with invalid prio
x86/mce: Do not register notifiers with invalid prio
Commit-ID: 32b40a82e84414499bb4d635cafb9de9e45e6c38 Gitweb: http://git.kernel.org/tip/32b40a82e84414499bb4d635cafb9de9e45e6c38 Author: Borislav Petkov <[email protected]> AuthorDate: Mon, 27 Mar 2017 11:33:04 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 Mar 2017 08:55:15 +0200
x86/mce: Do not register notifiers with invalid prio
This is just a defensive precaution: do not register notifiers with a priority which would disrupt the error handling in the notifiers with prio higher than MCE_PRIO_EDAC.
--- arch/x86/kernel/cpu/mcheck/mce.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +3 -2
@@ -157,9 +157,10 @@ static atomic_t num_notifiers;void mce_register_decode_chain (struct notifier_block *nb) { -atomic_inc (&num_notifiers); +if (WARN_ON (nb->priority >MCE_PRIO_LOWEST && nb->priority <MCE_PRIO_EDAC )) +return ; -WARN_ON (nb->priority >MCE_PRIO_LOWEST && nb->priority <MCE_PRIO_EDAC ); +atomic_inc (&num_notifiers);atomic_notifier_chain_register (&x86_mce_decoder_chain, nb); }