Skip to contentSkip to conversation

x86/RAS: Correctable Errors Collector

PATCH New Borislav Petkov 2 participants 13 messages Mar 27, 2017 – Mar 28, 2017

Subsystems (3)
lore
Share & tools
lore

Borislav Petkov

author maintainer
#0 lore raw

cover x86/RAS: Correctable Errors Collector

From: Borislav Petkov <[email protected]>

Hi guys,

here's v1, all feedback I know of has been addressed. So I guess it is time. :)

We don't have it default y yet but will make it so after it has seen wider testing. The end goal is to have it running by default so that transient correctable ECC errors don't generate error logs and upset people unnecessarily.

The other good thing resulting from this patchset is that we have *all* MCE consumers lined up in a notifier with priorities. This way we have a single chain which gets to see error records and not some wild variety of hooks here and there.

Last but not least, /dev/mcelog has been deprecated and all the code is behind a CONFIG_X86_MCELOG config option.

Btw, patch 1 is for urgent.

Please apply,
thanks.

Changelog:
=========

v0:

here's the latest incarnation of the CEC collector. I think I've taken care of all review comments but feel free to correct me here. The introductory comment in cec.c should explain the whole deal - I'm referring to there so that we have that text in the actual source and not spread it around commit messages. So pls have a look there for more info.

The thing has knobs in debugfs now which can control its operation, I hope I've chosen sane default values.

Andi Kleen (1):
x86/mce: Don't print MCEs when mcelog is active

Borislav Petkov (4):
x86/MCE: Rename mce_log()'s argument x86/MCE: Rename mce_log to mce_log_buffer
RAS: Add a Corrected Errors Collector x86/mce: Do not register notifiers with invalid prio

Tony Luck (1):
x86/mce: Deprecate /dev/mcelog

 Documentation/admin-guide/kernel-parameters.txt |   6 +
 arch/x86/Kconfig                                |  10 +-
 arch/x86/include/asm/mce.h                      |  12 +-
 arch/x86/kernel/cpu/mcheck/Makefile             |   2 +
 arch/x86/kernel/cpu/mcheck/dev-mcelog.c         | 397 ++++++++++++++++++
 arch/x86/kernel/cpu/mcheck/mce-internal.h       |   8 +
 arch/x86/kernel/cpu/mcheck/mce.c                | 501 +++++-----------------
 arch/x86/ras/Kconfig                            |  14 +
 drivers/ras/Makefile                            |   3 +-
 drivers/ras/cec.c                               | 532 ++++++++++++++++++++++++
 drivers/ras/debugfs.c                           |   2 +-
 drivers/ras/debugfs.h                           |   8 +
 drivers/ras/ras.c                               |  11 +
 include/linux/ras.h                             |  13 +-
 14 files changed, 1105 insertions(+), 414 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/mcheck/dev-mcelog.c
 create mode 100644 drivers/ras/cec.c
 create mode 100644 drivers/ras/debugfs.h
Signature · 1 line
2.11.0
author maintainer
#1 lore raw

PATCH 1/6 x86/mce: Don't print MCEs when mcelog is active

From: Andi Kleen <[email protected]>

Since

cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")

all MCEs are printed even when mcelog is running. Fix the regression to not print to dmesg when mcelog is running as it is a consumer too.

Cc [email protected] # 4.10..
Cc Tony Luck <[email protected]>
Cc linux-edac <[email protected]>
Cc x86-ml <[email protected]>
Fixes cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")
[ Massage commit message. ]
---
 arch/x86/kernel/cpu/mcheck/mce.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/mcheck/mce.c +6 -1
index 5e365a2fabe5..e95e02734f25 100644
@@ -55,6 +55,8 @@
 
 static DEFINE_MUTEX(mce_chrdev_read_mutex);
 
+static int mce_chrdev_open_count;	/* #times opened */
+
 #define mce_log_get_idx_check(p) \
 ({ \
 	RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
@@ -599,6 +601,10 @@ static int mce_default_notifier(struct notifier_block *nb, unsigned long val,
 	if (atomic_read(&num_notifiers) > 2)
 		return NOTIFY_DONE;
 
+	/* Don't print when mcelog is running */
+	if (mce_chrdev_open_count > 0)
+		return NOTIFY_DONE;
+
 	__print_mce(m);
 
 	return NOTIFY_DONE;
@@ -1848,7 +1854,6 @@ void mcheck_cpu_clear(struct cpuinfo_x86 *c)
  */
 
 static DEFINE_SPINLOCK(mce_chrdev_state_lock);
-static int mce_chrdev_open_count;	/* #times opened */
 static int mce_chrdev_open_exclu;	/* already open exclusive? */
 
 static int mce_chrdev_open(struct inode *inode, struct file *file)
Signature · 1 line
2.11.0
#2 lore raw

[tip:ras/core] x86/mce: Don't print MCEs when mcelog is active

x86/mce: Don't print MCEs when mcelog is active
Commit-ID:  cc66afea58f858ff6da7f79b8a595a67bbb4f9a9
Gitweb:     http://git.kernel.org/tip/cc66afea58f858ff6da7f79b8a595a67bbb4f9a9
Author:     Andi Kleen <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:32:59 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:53:52 +0200

x86/mce: Don't print MCEs when mcelog is active

Since:

cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")

all MCEs are printed even when mcelog is running. Fix the regression to not print to dmesg when mcelog is running as it is a consumer too.

[ Massage commit message. ]
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tony Luck <[email protected]>
Cc linux-edac <[email protected]>
Cc [email protected] # 4.10..
Fixes cd9c57cad3fe ("x86/MCE: Dump MCE to dmesg if no consumers")
---
 arch/x86/kernel/cpu/mcheck/mce.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/mcheck/mce.c +6 -1
index 8e9725c..5accfbd 100644
@@ -54,6 +54,8 @@
 
 static DEFINE_MUTEX(mce_chrdev_read_mutex);
 
+static int mce_chrdev_open_count;	/* #times opened */
+
 #define mce_log_get_idx_check(p) \
 ({ \
 	RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
@@ -598,6 +600,10 @@ static int mce_default_notifier(struct notifier_block *nb, unsigned long val,
 	if (atomic_read(&num_notifiers) > 2)
 		return NOTIFY_DONE;
 
+	/* Don't print when mcelog is running */
+	if (mce_chrdev_open_count > 0)
+		return NOTIFY_DONE;
+
 	__print_mce(m);
 
 	return NOTIFY_DONE;
@@ -1828,7 +1834,6 @@ void mcheck_cpu_clear(struct cpuinfo_x86 *c)
  */
 
 static DEFINE_SPINLOCK(mce_chrdev_state_lock);
-static int mce_chrdev_open_count;	/* #times opened */
 static int mce_chrdev_open_exclu;	/* already open exclusive? */
 
 static int mce_chrdev_open(struct inode *inode, struct file *file)
author maintainer
#3 lore raw

PATCH 2/6 x86/MCE: Rename mce_log()'s argument

In reply to Borislav Petkov

From: Borislav Petkov <[email protected]>

We call it everywhere "struct mce *m". Adjust that here too to avoid confusion.

No functionality change.

---
 arch/x86/kernel/cpu/mcheck/mce.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +4 -4
index e95e02734f25..49673b20f17f 100644
@@ -158,14 +158,14 @@ static struct mce_log mcelog = {
 	.recordlen	= sizeof(struct mce),
 };
 
-void mce_log(struct mce *mce)
+void mce_log(struct mce *m)
 {
 	unsigned next, entry;
 
 	/* Emit the trace record: */
-	trace_mce_record(mce);
+	trace_mce_record(m);
 
-	if (!mce_gen_pool_add(mce))
+	if (!mce_gen_pool_add(m))
 		irq_work_queue(&mce_irq_work);
 
 	wmb();
@@ -195,7 +195,7 @@ void mce_log(struct mce *mce)
 		if (cmpxchg(&mcelog.next, entry, next) == entry)
 			break;
 	}
-	memcpy(mcelog.entry + entry, mce, sizeof(struct mce));
+	memcpy(mcelog.entry + entry, m, sizeof(struct mce));
 	wmb();
 	mcelog.entry[entry].finished = 1;
 	wmb();
Signature · 1 line
2.11.0
#4 lore raw

[tip:ras/core] x86/mce: Rename mce_log()'s argument

x86/mce: Rename mce_log()'s argument
Commit-ID:  fe3ed20fddfd8c6a4dcfb8e43de80f269d1f3f2a
Gitweb:     http://git.kernel.org/tip/fe3ed20fddfd8c6a4dcfb8e43de80f269d1f3f2a
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:33:00 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:54:38 +0200

x86/mce: Rename mce_log()'s argument

We call it everywhere "struct mce *m". Adjust that here too to avoid confusion.

No functionality change.

---
 arch/x86/kernel/cpu/mcheck/mce.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +4 -4
index e95e027..49673b2 100644
@@ -158,14 +158,14 @@ static struct mce_log mcelog = {
 	.recordlen	= sizeof(struct mce),
 };
 
-void mce_log(struct mce *mce)
+void mce_log(struct mce *m)
 {
 	unsigned next, entry;
 
 	/* Emit the trace record: */
-	trace_mce_record(mce);
+	trace_mce_record(m);
 
-	if (!mce_gen_pool_add(mce))
+	if (!mce_gen_pool_add(m))
 		irq_work_queue(&mce_irq_work);
 
 	wmb();
@@ -195,7 +195,7 @@ void mce_log(struct mce *mce)
 		if (cmpxchg(&mcelog.next, entry, next) == entry)
 			break;
 	}
-	memcpy(mcelog.entry + entry, mce, sizeof(struct mce));
+	memcpy(mcelog.entry + entry, m, sizeof(struct mce));
 	wmb();
 	mcelog.entry[entry].finished = 1;
 	wmb();
author maintainer
#5 lore raw

PATCH 3/6 x86/MCE: Rename mce_log to mce_log_buffer

In reply to Borislav Petkov

From: Borislav Petkov <[email protected]>

It is confusing when staring at "struct mce_log mcelog" and then there's also a function called mce_log(). So call the buffer what it is.

No functionality change.

---
 arch/x86/include/asm/mce.h       |  2 +-
 arch/x86/kernel/cpu/mcheck/mce.c | 30 +++++++++++++++---------------
 2 files changed, 16 insertions(+), 16 deletions(-)
arch/x86/include/asm/mce.h +1 -1
index e63873683d4a..0512dcc11750 100644
@@ -128,7 +128,7 @@
  * debugging tools.  Each entry is only valid when its finished flag
  * is set.
  */
-struct mce_log {
+struct mce_log_buffer {
 	char signature[12]; /* "MACHINECHECK" */
 	unsigned len;	    /* = MCE_LOG_LEN */
 	unsigned next;
arch/x86/kernel/cpu/mcheck/mce.c +15 -15
index 49673b20f17f..8ada093d4e40 100644
@@ -152,7 +152,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);
  * separate MCEs from kernel messages to avoid bogus bug reports.
  */
 
-static struct mce_log mcelog = {
+static struct mce_log_buffer mcelog_buf = {
 	.signature	= MCE_LOG_SIGNATURE,
 	.len		= MCE_LOG_LEN,
 	.recordlen	= sizeof(struct mce),
@@ -170,7 +170,7 @@ void mce_log(struct mce *m)
 
 	wmb();
 	for (;;) {
-		entry = mce_log_get_idx_check(mcelog.next);
+		entry = mce_log_get_idx_check(mcelog_buf.next);
 		for (;;) {
 
 			/*
@@ -180,11 +180,11 @@ void mce_log(struct mce *m)
 			 */
 			if (entry >= MCE_LOG_LEN) {
 				set_bit(MCE_OVERFLOW,
-					(unsigned long *)&mcelog.flags);
+					(unsigned long *)&mcelog_buf.flags);
 				return;
 			}
 			/* Old left over entry. Skip: */
-			if (mcelog.entry[entry].finished) {
+			if (mcelog_buf.entry[entry].finished) {
 				entry++;
 				continue;
 			}
@@ -192,12 +192,12 @@ void mce_log(struct mce *m)
 		}
 		smp_rmb();
 		next = entry + 1;
-		if (cmpxchg(&mcelog.next, entry, next) == entry)
+		if (cmpxchg(&mcelog_buf.next, entry, next) == entry)
 			break;
 	}
-	memcpy(mcelog.entry + entry, m, sizeof(struct mce));
+	memcpy(mcelog_buf.entry + entry, m, sizeof(struct mce));
 	wmb();
-	mcelog.entry[entry].finished = 1;
+	mcelog_buf.entry[entry].finished = 1;
 	wmb();
 
 	set_bit(0, &mce_need_notify);
@@ -1958,7 +1958,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 			goto out;
 	}
 
-	next = mce_log_get_idx_check(mcelog.next);
+	next = mce_log_get_idx_check(mcelog_buf.next);
 
 	/* Only supports full reads right now */
 	err = -EINVAL;
@@ -1970,7 +1970,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 	do {
 		for (i = prev; i < next; i++) {
 			unsigned long start = jiffies;
-			struct mce *m = &mcelog.entry[i];
+			struct mce *m = &mcelog_buf.entry[i];
 
 			while (!m->finished) {
 				if (time_after_eq(jiffies, start + 2)) {
@@ -1986,10 +1986,10 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 			;
 		}
 
-		memset(mcelog.entry + prev, 0,
+		memset(mcelog_buf.entry + prev, 0,
 		       (next - prev) * sizeof(struct mce));
 		prev = next;
-		next = cmpxchg(&mcelog.next, prev, 0);
+		next = cmpxchg(&mcelog_buf.next, prev, 0);
 	} while (next != prev);
 
 	synchronize_sched();
@@ -2001,7 +2001,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 	on_each_cpu(collect_tscs, cpu_tsc, 1);
 
 	for (i = next; i < MCE_LOG_LEN; i++) {
-		struct mce *m = &mcelog.entry[i];
+		struct mce *m = &mcelog_buf.entry[i];
 
 		if (m->finished && m->tsc < cpu_tsc[m->cpu]) {
 			err |= copy_to_user(buf, m, sizeof(*m));
@@ -2024,7 +2024,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 static unsigned int mce_chrdev_poll(struct file *file, poll_table *wait)
 {
 	poll_wait(file, &mce_chrdev_wait, wait);
-	if (READ_ONCE(mcelog.next))
+	if (READ_ONCE(mcelog_buf.next))
 		return POLLIN | POLLRDNORM;
 	if (!mce_apei_read_done && apei_check_mce())
 		return POLLIN | POLLRDNORM;
@@ -2048,8 +2048,8 @@ static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,
 		unsigned flags;
 
 		do {
-			flags = mcelog.flags;
-		} while (cmpxchg(&mcelog.flags, flags, 0) != flags);
+			flags = mcelog_buf.flags;
+		} while (cmpxchg(&mcelog_buf.flags, flags, 0) != flags);
 
 		return put_user(flags, p);
 	}
Signature · 1 line
2.11.0
#6 lore raw

[tip:ras/core] x86/mce: Rename mce_log to mce_log_buffer

x86/mce: Rename mce_log to mce_log_buffer
Commit-ID:  e64edfcce9c738300b4102d0739577d6ecc96d4a
Gitweb:     http://git.kernel.org/tip/e64edfcce9c738300b4102d0739577d6ecc96d4a
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:33:01 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:54:42 +0200

x86/mce: Rename mce_log to mce_log_buffer

It is confusing when staring at "struct mce_log mcelog" and then there's also a function called mce_log(). So call the buffer what it is.

No functionality change.

---
 arch/x86/include/asm/mce.h       |  2 +-
 arch/x86/kernel/cpu/mcheck/mce.c | 30 +++++++++++++++---------------
 2 files changed, 16 insertions(+), 16 deletions(-)
arch/x86/include/asm/mce.h +1 -1
index e638736..0512dcc 100644
@@ -128,7 +128,7 @@
  * debugging tools.  Each entry is only valid when its finished flag
  * is set.
  */
-struct mce_log {
+struct mce_log_buffer {
 	char signature[12]; /* "MACHINECHECK" */
 	unsigned len;	    /* = MCE_LOG_LEN */
 	unsigned next;
arch/x86/kernel/cpu/mcheck/mce.c +15 -15
index 49673b2..8ada093 100644
@@ -152,7 +152,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);
  * separate MCEs from kernel messages to avoid bogus bug reports.
  */
 
-static struct mce_log mcelog = {
+static struct mce_log_buffer mcelog_buf = {
 	.signature	= MCE_LOG_SIGNATURE,
 	.len		= MCE_LOG_LEN,
 	.recordlen	= sizeof(struct mce),
@@ -170,7 +170,7 @@ void mce_log(struct mce *m)
 
 	wmb();
 	for (;;) {
-		entry = mce_log_get_idx_check(mcelog.next);
+		entry = mce_log_get_idx_check(mcelog_buf.next);
 		for (;;) {
 
 			/*
@@ -180,11 +180,11 @@ void mce_log(struct mce *m)
 			 */
 			if (entry >= MCE_LOG_LEN) {
 				set_bit(MCE_OVERFLOW,
-					(unsigned long *)&mcelog.flags);
+					(unsigned long *)&mcelog_buf.flags);
 				return;
 			}
 			/* Old left over entry. Skip: */
-			if (mcelog.entry[entry].finished) {
+			if (mcelog_buf.entry[entry].finished) {
 				entry++;
 				continue;
 			}
@@ -192,12 +192,12 @@ void mce_log(struct mce *m)
 		}
 		smp_rmb();
 		next = entry + 1;
-		if (cmpxchg(&mcelog.next, entry, next) == entry)
+		if (cmpxchg(&mcelog_buf.next, entry, next) == entry)
 			break;
 	}
-	memcpy(mcelog.entry + entry, m, sizeof(struct mce));
+	memcpy(mcelog_buf.entry + entry, m, sizeof(struct mce));
 	wmb();
-	mcelog.entry[entry].finished = 1;
+	mcelog_buf.entry[entry].finished = 1;
 	wmb();
 
 	set_bit(0, &mce_need_notify);
@@ -1958,7 +1958,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 			goto out;
 	}
 
-	next = mce_log_get_idx_check(mcelog.next);
+	next = mce_log_get_idx_check(mcelog_buf.next);
 
 	/* Only supports full reads right now */
 	err = -EINVAL;
@@ -1970,7 +1970,7 @@ static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
 	do {
 		for (i = prev; i < next; i++) {
 			unsigned long start = jiffies;
-			struct mce *m = &mcelog.entry[i];
+			struct mce *m = &mcelog_buf.entry[i];
 
 			while (!m->finished) {
 				if (time_after_eq(jiffies, start + 2)) {
@@ -1986,10 +1986,10 @@ timeout:
 			;
 		}
 
-		memset(mcelog.entry + prev, 0,
+		memset(mcelog_buf.entry + prev, 0,
 		       (next - prev) * sizeof(struct mce));
 		prev = next;
-		next = cmpxchg(&mcelog.next, prev, 0);
+		next = cmpxchg(&mcelog_buf.next, prev, 0);
 	} while (next != prev);
 
 	synchronize_sched();
@@ -2001,7 +2001,7 @@ timeout:
 	on_each_cpu(collect_tscs, cpu_tsc, 1);
 
 	for (i = next; i < MCE_LOG_LEN; i++) {
-		struct mce *m = &mcelog.entry[i];
+		struct mce *m = &mcelog_buf.entry[i];
 
 		if (m->finished && m->tsc < cpu_tsc[m->cpu]) {
 			err |= copy_to_user(buf, m, sizeof(*m));
@@ -2024,7 +2024,7 @@ out:
 static unsigned int mce_chrdev_poll(struct file *file, poll_table *wait)
 {
 	poll_wait(file, &mce_chrdev_wait, wait);
-	if (READ_ONCE(mcelog.next))
+	if (READ_ONCE(mcelog_buf.next))
 		return POLLIN | POLLRDNORM;
 	if (!mce_apei_read_done && apei_check_mce())
 		return POLLIN | POLLRDNORM;
@@ -2048,8 +2048,8 @@ static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,
 		unsigned flags;
 
 		do {
-			flags = mcelog.flags;
-		} while (cmpxchg(&mcelog.flags, flags, 0) != flags);
+			flags = mcelog_buf.flags;
+		} while (cmpxchg(&mcelog_buf.flags, flags, 0) != flags);
 
 		return put_user(flags, p);
 	}
author maintainer
#7 lore raw

PATCH 4/6 RAS: Add a Corrected Errors Collector

In reply to Borislav Petkov

From: Borislav Petkov <[email protected]>

A simple data structure for collecting correctable errors along with accessors. More detailed description in the code itself.

The error decoding is done with the decoding chain now and mce_first_notifier() gets to see the error first and the CEC decides whether to log it and then the rest of the chain doesn't hear about it - basically the main reason for the CE collector - or to continue running the notifiers.

When the CEC hits the action threshold, it will try to soft-offine the page containing the ECC and then the whole decoding chain gets to see the error.

---
 Documentation/admin-guide/kernel-parameters.txt |   6 +
 arch/x86/include/asm/mce.h                      |   9 +-
 arch/x86/kernel/cpu/mcheck/mce.c                | 191 +++++----
 arch/x86/ras/Kconfig                            |  14 +
 drivers/ras/Makefile                            |   3 +-
 drivers/ras/cec.c                               | 532 ++++++++++++++++++++++++
 drivers/ras/debugfs.c                           |   2 +-
 drivers/ras/debugfs.h                           |   8 +
 drivers/ras/ras.c                               |  11 +
 include/linux/ras.h                             |  13 +-
 10 files changed, 706 insertions(+), 83 deletions(-)
 create mode 100644 drivers/ras/cec.c
 create mode 100644 drivers/ras/debugfs.h
Documentation/admin-guide/kernel-parameters.txt +6 -0
index ea1293d60f17..77f285c802e4 100644
@@ -3173,6 +3173,12 @@
 	ramdisk_size=	[RAM] Sizes of RAM disks in kilobytes
 			See Documentation/blockdev/ramdisk.txt.
 
+	ras=option[,option,...]	[KNL] RAS-specific options
+
+		cec_disable	[X86]
+				Disable the Correctable Errors Collector,
+				see CONFIG_RAS_CEC help text.
+
 	rcu_nocbs=	[KNL]
 			The argument is a cpu list, as described above.
 
arch/x86/include/asm/mce.h +5 -4
index 0512dcc11750..c5ae545d27d8 100644
@@ -191,10 +191,11 @@ extern struct mca_config mca_cfg;
 extern struct mca_msr_regs msr_ops;
 
 enum mce_notifier_prios {
-	MCE_PRIO_SRAO		= INT_MAX,
-	MCE_PRIO_EXTLOG		= INT_MAX - 1,
-	MCE_PRIO_NFIT		= INT_MAX - 2,
-	MCE_PRIO_EDAC		= INT_MAX - 3,
+	MCE_PRIO_FIRST		= INT_MAX,
+	MCE_PRIO_SRAO		= INT_MAX - 1,
+	MCE_PRIO_EXTLOG		= INT_MAX - 2,
+	MCE_PRIO_NFIT		= INT_MAX - 3,
+	MCE_PRIO_EDAC		= INT_MAX - 4,
 	MCE_PRIO_LOWEST		= 0,
 };
 
arch/x86/kernel/cpu/mcheck/mce.c +115 -76

247 lines not on this page · raw message

arch/x86/ras/Kconfig +14 -0
index 0bc60a308730..2a2d89d39af6 100644
@@ -7,3 +7,17 @@ config MCE_AMD_INJ
 	  aspects of the MCE handling code.
 
 	  WARNING: Do not even assume this interface is staying stable!
+
+config RAS_CEC
+	bool "Correctable Errors Collector"
+	depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS
+	---help---
+	  This is a small cache which collects correctable memory errors per 4K
+	  page PFN and counts their repeated occurrence. Once the counter for a
+	  PFN overflows, we try to soft-offline that page as we take it to mean
+	  that it has reached a relatively high error count and would probably
+	  be best if we don't use it anymore.
+
+	  Bear in mind that this is absolutely useless if your platform doesn't
+	  have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS.
+
drivers/ras/Makefile +2 -1
index d7f73341ced3..7b26dd3aa5d0 100644
@@ -1 +1,2 @@
-obj-$(CONFIG_RAS) += ras.o debugfs.o
+obj-$(CONFIG_RAS)	+= ras.o debugfs.o
+obj-$(CONFIG_RAS_CEC)	+= cec.o
drivers/ras/cec.c new file +532 -0

532 lines not on this page · raw message

drivers/ras/debugfs.c +1 -1
index 0322acf67ea5..501603057dff 100644
@@ -1,6 +1,6 @@
 #include <linux/debugfs.h>
 
-static struct dentry *ras_debugfs_dir;
+struct dentry *ras_debugfs_dir;
 
 static atomic_t trace_count = ATOMIC_INIT(0);
 
drivers/ras/debugfs.h new file +8 -0
new file mode 100644
index 000000000000..db72e4513191
@@ -0,0 +1,8 @@
+#ifndef __RAS_DEBUGFS_H__
+#define __RAS_DEBUGFS_H__
+
+#include <linux/debugfs.h>
+
+extern struct dentry *ras_debugfs_dir;
+
+#endif /* __RAS_DEBUGFS_H__ */
drivers/ras/ras.c +11 -0
index b67dd362b7b6..94f8038864b4 100644
@@ -27,3 +27,14 @@ subsys_initcall(ras_init);
 EXPORT_TRACEPOINT_SYMBOL_GPL(extlog_mem_event);
 #endif
 EXPORT_TRACEPOINT_SYMBOL_GPL(mc_event);
+
+
+int __init parse_ras_param(char *str)
+{
+#ifdef CONFIG_RAS_CEC
+	parse_cec_param(str);
+#endif
+
+	return 1;
+}
+__setup("ras", parse_ras_param);
include/linux/ras.h +12 -1
index 2aceeafd6fe5..ffb147185e8d 100644
@@ -1,14 +1,25 @@
 #ifndef __RAS_H__
 #define __RAS_H__
 
+#include <asm/errno.h>
+
 #ifdef CONFIG_DEBUG_FS
 int ras_userspace_consumers(void);
 void ras_debugfs_init(void);
 int ras_add_daemon_trace(void);
 #else
 static inline int ras_userspace_consumers(void) { return 0; }
-static inline void ras_debugfs_init(void) { return; }
+static inline void ras_debugfs_init(void) { }
 static inline int ras_add_daemon_trace(void) { return 0; }
 #endif
 
+#ifdef CONFIG_RAS_CEC
+void __init cec_init(void);
+int __init parse_cec_param(char *str);
+int cec_add_elem(u64 pfn);
+#else
+static inline void __init cec_init(void)	{ }
+static inline int cec_add_elem(u64 pfn)		{ return -ENODEV; }
 #endif
+
+#endif /* __RAS_H__ */
Signature · 1 line
2.11.0
#8 lore raw

[tip:ras/core] RAS: Add a Corrected Errors Collector

RAS: Add a Corrected Errors Collector
Commit-ID:  011d8261117249eab97bc86a8e1ac7731e03e319
Gitweb:     http://git.kernel.org/tip/011d8261117249eab97bc86a8e1ac7731e03e319
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:33:02 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:54:48 +0200

RAS: Add a Corrected Errors Collector

Introduce a simple data structure for collecting correctable errors along with accessors. More detailed description in the code itself.

The error decoding is done with the decoding chain now and mce_first_notifier() gets to see the error first and the CEC decides whether to log it and then the rest of the chain doesn't hear about it - basically the main reason for the CE collector - or to continue running the notifiers.

When the CEC hits the action threshold, it will try to soft-offine the page containing the ECC and then the whole decoding chain gets to see the error.

---
 Documentation/admin-guide/kernel-parameters.txt |   6 +
 arch/x86/include/asm/mce.h                      |   9 +-
 arch/x86/kernel/cpu/mcheck/mce.c                | 191 +++++----
 arch/x86/ras/Kconfig                            |  14 +
 drivers/ras/Makefile                            |   3 +-
 drivers/ras/cec.c                               | 532 ++++++++++++++++++++++++
 drivers/ras/debugfs.c                           |   2 +-
 drivers/ras/debugfs.h                           |   8 +
 drivers/ras/ras.c                               |  11 +
 include/linux/ras.h                             |  13 +-
 10 files changed, 706 insertions(+), 83 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +6 -0
index 2ba45ca..927b8b8 100644
@@ -3172,6 +3172,12 @@
 	ramdisk_size=	[RAM] Sizes of RAM disks in kilobytes
 			See Documentation/blockdev/ramdisk.txt.
 
+	ras=option[,option,...]	[KNL] RAS-specific options
+
+		cec_disable	[X86]
+				Disable the Correctable Errors Collector,
+				see CONFIG_RAS_CEC help text.
+
 	rcu_nocbs=	[KNL]
 			The argument is a cpu list, as described above.
 
arch/x86/include/asm/mce.h +5 -4
index 0512dcc..c5ae545 100644
@@ -191,10 +191,11 @@ extern struct mca_config mca_cfg;
 extern struct mca_msr_regs msr_ops;
 
 enum mce_notifier_prios {
-	MCE_PRIO_SRAO		= INT_MAX,
-	MCE_PRIO_EXTLOG		= INT_MAX - 1,
-	MCE_PRIO_NFIT		= INT_MAX - 2,
-	MCE_PRIO_EDAC		= INT_MAX - 3,
+	MCE_PRIO_FIRST		= INT_MAX,
+	MCE_PRIO_SRAO		= INT_MAX - 1,
+	MCE_PRIO_EXTLOG		= INT_MAX - 2,
+	MCE_PRIO_NFIT		= INT_MAX - 3,
+	MCE_PRIO_EDAC		= INT_MAX - 4,
 	MCE_PRIO_LOWEST		= 0,
 };
 
arch/x86/kernel/cpu/mcheck/mce.c +115 -76

247 lines not on this page · raw message

arch/x86/ras/Kconfig +14 -0
index 0bc60a3..2a2d89d 100644
@@ -7,3 +7,17 @@ config MCE_AMD_INJ
 	  aspects of the MCE handling code.
 
 	  WARNING: Do not even assume this interface is staying stable!
+
+config RAS_CEC
+	bool "Correctable Errors Collector"
+	depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS
+	---help---
+	  This is a small cache which collects correctable memory errors per 4K
+	  page PFN and counts their repeated occurrence. Once the counter for a
+	  PFN overflows, we try to soft-offline that page as we take it to mean
+	  that it has reached a relatively high error count and would probably
+	  be best if we don't use it anymore.
+
+	  Bear in mind that this is absolutely useless if your platform doesn't
+	  have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS.
+
drivers/ras/Makefile +2 -1
index d7f7334..7b26dd3 100644
@@ -1 +1,2 @@
-obj-$(CONFIG_RAS) += ras.o debugfs.o
+obj-$(CONFIG_RAS)	+= ras.o debugfs.o
+obj-$(CONFIG_RAS_CEC)	+= cec.o
drivers/ras/cec.c new file +532 -0

532 lines not on this page · raw message

drivers/ras/debugfs.c +1 -1
index 0322acf..5016030 100644
@@ -1,6 +1,6 @@
 #include <linux/debugfs.h>
 
-static struct dentry *ras_debugfs_dir;
+struct dentry *ras_debugfs_dir;
 
 static atomic_t trace_count = ATOMIC_INIT(0);
 
drivers/ras/debugfs.h new file +8 -0
new file mode 100644
index 0000000..db72e45
@@ -0,0 +1,8 @@
+#ifndef __RAS_DEBUGFS_H__
+#define __RAS_DEBUGFS_H__
+
+#include <linux/debugfs.h>
+
+extern struct dentry *ras_debugfs_dir;
+
+#endif /* __RAS_DEBUGFS_H__ */
drivers/ras/ras.c +11 -0
index b67dd36..94f8038 100644
@@ -27,3 +27,14 @@ subsys_initcall(ras_init);
 EXPORT_TRACEPOINT_SYMBOL_GPL(extlog_mem_event);
 #endif
 EXPORT_TRACEPOINT_SYMBOL_GPL(mc_event);
+
+
+int __init parse_ras_param(char *str)
+{
+#ifdef CONFIG_RAS_CEC
+	parse_cec_param(str);
+#endif
+
+	return 1;
+}
+__setup("ras", parse_ras_param);
include/linux/ras.h +12 -1
index 2aceeaf..ffb1471 100644
@@ -1,14 +1,25 @@
 #ifndef __RAS_H__
 #define __RAS_H__
 
+#include <asm/errno.h>
+
 #ifdef CONFIG_DEBUG_FS
 int ras_userspace_consumers(void);
 void ras_debugfs_init(void);
 int ras_add_daemon_trace(void);
 #else
 static inline int ras_userspace_consumers(void) { return 0; }
-static inline void ras_debugfs_init(void) { return; }
+static inline void ras_debugfs_init(void) { }
 static inline int ras_add_daemon_trace(void) { return 0; }
 #endif
 
+#ifdef CONFIG_RAS_CEC
+void __init cec_init(void);
+int __init parse_cec_param(char *str);
+int cec_add_elem(u64 pfn);
+#else
+static inline void __init cec_init(void)	{ }
+static inline int cec_add_elem(u64 pfn)		{ return -ENODEV; }
 #endif
+
+#endif /* __RAS_H__ */
author maintainer
#9 lore raw

PATCH 5/6 x86/mce: Deprecate /dev/mcelog

In reply to Borislav Petkov

From: Tony Luck <[email protected]>

Move all code relating to /dev/mcelog to a separate source file. /dev/mcelog driver can now operate from the machine check notifier with lowest prio.

Boris:
* Move the mce_helper and trigger functionality behind CONFIG_X86_MCELOG.

---
 arch/x86/Kconfig                          |  10 +-
 arch/x86/include/asm/mce.h                |   1 +
 arch/x86/kernel/cpu/mcheck/Makefile       |   2 +
 arch/x86/kernel/cpu/mcheck/dev-mcelog.c   | 397 ++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/mcheck/mce-internal.h |   8 +
 arch/x86/kernel/cpu/mcheck/mce.c          | 372 +---------------------------
 6 files changed, 426 insertions(+), 364 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/mcheck/dev-mcelog.c
arch/x86/Kconfig +9 -1
index 8977d9c77373..b3de6fab7503 100644
@@ -1044,6 +1044,14 @@ config X86_MCE
 	  The action the kernel takes depends on the severity of the problem,
 	  ranging from warning messages to halting the machine.
 
+config X86_MCELOG
+	bool "Support for deprecated /dev/mcelog character device"
+	depends on X86_MCE
+	---help---
+	  Enable support for /dev/mcelog which is needed by the old mcelog
+	  userspace logging daemon. Consider switching to the new generation
+	  rasdaemon solution.
+
 config X86_MCE_INTEL
 	def_bool y
 	prompt "Intel MCE features"
@@ -1073,7 +1081,7 @@ config X86_MCE_THRESHOLD
 	def_bool y
 
 config X86_MCE_INJECT
-	depends on X86_MCE && X86_LOCAL_APIC
+	depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG
 	tristate "Machine check injector support"
 	---help---
 	  Provide support for injecting machine checks for testing purposes.
arch/x86/include/asm/mce.h +1 -0
index c5ae545d27d8..4fd5195deed0 100644
@@ -196,6 +196,7 @@ enum mce_notifier_prios {
 	MCE_PRIO_EXTLOG		= INT_MAX - 2,
 	MCE_PRIO_NFIT		= INT_MAX - 3,
 	MCE_PRIO_EDAC		= INT_MAX - 4,
+	MCE_PRIO_MCELOG		= 1,
 	MCE_PRIO_LOWEST		= 0,
 };
 
arch/x86/kernel/cpu/mcheck/Makefile +2 -0
index a3311c886194..950e9ff602ea 100644
@@ -9,3 +9,5 @@ obj-$(CONFIG_X86_MCE_INJECT)	+= mce-inject.o
 obj-$(CONFIG_X86_THERMAL_VECTOR) += therm_throt.o
 
 obj-$(CONFIG_ACPI_APEI)		+= mce-apei.o
+
+obj-$(CONFIG_X86_MCELOG)	+= dev-mcelog.o
arch/x86/kernel/cpu/mcheck/dev-mcelog.c new file +397 -0

397 lines not on this page · raw message

arch/x86/kernel/cpu/mcheck/mce-internal.h +8 -0
index 903043e6a62b..f096d64485d0 100644
@@ -96,3 +96,11 @@ static inline bool mce_cmp(struct mce *m1, struct mce *m2)
 		m1->addr != m2->addr ||
 		m1->misc != m2->misc;
 }
+
+extern struct device_attribute dev_attr_trigger;
+
+#ifdef CONFIG_X86_MCELOG
+extern void mce_work_trigger(void);
+#else
+static inline void mce_work_trigger(void)	{ }
+#endif
arch/x86/kernel/cpu/mcheck/mce.c +9 -363

476 lines not on this page · raw message

Signature · 1 line
2.11.0
#10 lore raw

[tip:ras/core] x86/mce: Factor out and deprecate the /dev/mcelog driver

x86/mce: Factor out and deprecate the /dev/mcelog driver
Commit-ID:  5de97c9f6d85fd83af76e09e338b18e7adb1ae60
Gitweb:     http://git.kernel.org/tip/5de97c9f6d85fd83af76e09e338b18e7adb1ae60
Author:     Tony Luck <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:33:03 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:55:01 +0200

x86/mce: Factor out and deprecate the /dev/mcelog driver

Move all code relating to /dev/mcelog to a separate source file. /dev/mcelog driver can now operate from the machine check notifier with lowest prio.

[ Move the mce_helper and trigger functionality behind CONFIG_X86_MCELOG_LEGACY. ]
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc linux-edac <[email protected]>
[ Renamed CONFIG_X86_MCELOG to CONFIG_X86_MCELOG_LEGACY. ]
---
 arch/x86/Kconfig                          |  10 +-
 arch/x86/include/asm/mce.h                |   1 +
 arch/x86/kernel/cpu/mcheck/Makefile       |   2 +
 arch/x86/kernel/cpu/mcheck/dev-mcelog.c   | 397 ++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/mcheck/mce-internal.h |   8 +
 arch/x86/kernel/cpu/mcheck/mce.c          | 372 +---------------------------
 6 files changed, 426 insertions(+), 364 deletions(-)
arch/x86/Kconfig +9 -1
index cc98d5a..43e6bac 100644
@@ -1043,6 +1043,14 @@ config X86_MCE
 	  The action the kernel takes depends on the severity of the problem,
 	  ranging from warning messages to halting the machine.
 
+config X86_MCELOG_LEGACY
+	bool "Support for deprecated /dev/mcelog character device"
+	depends on X86_MCE
+	---help---
+	  Enable support for /dev/mcelog which is needed by the old mcelog
+	  userspace logging daemon. Consider switching to the new generation
+	  rasdaemon solution.
+
 config X86_MCE_INTEL
 	def_bool y
 	prompt "Intel MCE features"
@@ -1072,7 +1080,7 @@ config X86_MCE_THRESHOLD
 	def_bool y
 
 config X86_MCE_INJECT
-	depends on X86_MCE && X86_LOCAL_APIC
+	depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG_LEGACY
 	tristate "Machine check injector support"
 	---help---
 	  Provide support for injecting machine checks for testing purposes.
arch/x86/include/asm/mce.h +1 -0
index c5ae545..4fd5195 100644
@@ -196,6 +196,7 @@ enum mce_notifier_prios {
 	MCE_PRIO_EXTLOG		= INT_MAX - 2,
 	MCE_PRIO_NFIT		= INT_MAX - 3,
 	MCE_PRIO_EDAC		= INT_MAX - 4,
+	MCE_PRIO_MCELOG		= 1,
 	MCE_PRIO_LOWEST		= 0,
 };
 
arch/x86/kernel/cpu/mcheck/Makefile +2 -0
index a3311c8..43051f0 100644
@@ -9,3 +9,5 @@ obj-$(CONFIG_X86_MCE_INJECT)	+= mce-inject.o
 obj-$(CONFIG_X86_THERMAL_VECTOR) += therm_throt.o
 
 obj-$(CONFIG_ACPI_APEI)		+= mce-apei.o
+
+obj-$(CONFIG_X86_MCELOG_LEGACY)	+= dev-mcelog.o
arch/x86/kernel/cpu/mcheck/dev-mcelog.c new file +397 -0

397 lines not on this page · raw message

arch/x86/kernel/cpu/mcheck/mce-internal.h +8 -0
index 903043e..7f2a7c5 100644
@@ -96,3 +96,11 @@ static inline bool mce_cmp(struct mce *m1, struct mce *m2)
 		m1->addr != m2->addr ||
 		m1->misc != m2->misc;
 }
+
+extern struct device_attribute dev_attr_trigger;
+
+#ifdef CONFIG_X86_MCELOG_LEGACY
+extern void mce_work_trigger(void);
+#else
+static inline void mce_work_trigger(void)	{ }
+#endif
arch/x86/kernel/cpu/mcheck/mce.c +9 -363

476 lines not on this page · raw message

author maintainer
#11 lore raw

PATCH 6/6 x86/mce: Do not register notifiers with invalid prio

In reply to Borislav Petkov

From: Borislav Petkov <[email protected]>

This is just a defensive precaution: do not register notifiers with a priority which would disrupt the error handling in the notifiers with prio higher than MCE_PRIO_EDAC.

---
 arch/x86/kernel/cpu/mcheck/mce.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +3 -2
index 8e119156429b..4b57b8336ecd 100644
@@ -157,9 +157,10 @@ static atomic_t num_notifiers;
 
 void mce_register_decode_chain(struct notifier_block *nb)
 {
-	atomic_inc(&num_notifiers);
+	if (WARN_ON(nb->priority > MCE_PRIO_LOWEST && nb->priority < MCE_PRIO_EDAC))
+		return;
 
-	WARN_ON(nb->priority > MCE_PRIO_LOWEST && nb->priority < MCE_PRIO_EDAC);
+	atomic_inc(&num_notifiers);
 
 	atomic_notifier_chain_register(&x86_mce_decoder_chain, nb);
 }
Signature · 1 line
2.11.0
#12 lore raw

[tip:ras/core] x86/mce: Do not register notifiers with invalid prio

x86/mce: Do not register notifiers with invalid prio
Commit-ID:  32b40a82e84414499bb4d635cafb9de9e45e6c38
Gitweb:     http://git.kernel.org/tip/32b40a82e84414499bb4d635cafb9de9e45e6c38
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 27 Mar 2017 11:33:04 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 28 Mar 2017 08:55:15 +0200

x86/mce: Do not register notifiers with invalid prio

This is just a defensive precaution: do not register notifiers with a priority which would disrupt the error handling in the notifiers with prio higher than MCE_PRIO_EDAC.

---
 arch/x86/kernel/cpu/mcheck/mce.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +3 -2
index 36082c7..a09bb67 100644
@@ -157,9 +157,10 @@ static atomic_t num_notifiers;
 
 void mce_register_decode_chain(struct notifier_block *nb)
 {
-	atomic_inc(&num_notifiers);
+	if (WARN_ON(nb->priority > MCE_PRIO_LOWEST && nb->priority < MCE_PRIO_EDAC))
+		return;
 
-	WARN_ON(nb->priority > MCE_PRIO_LOWEST && nb->priority < MCE_PRIO_EDAC);
+	atomic_inc(&num_notifiers);
 
 	atomic_notifier_chain_register(&x86_mce_decoder_chain, nb);
 }