Skip to contentSkip to conversation

mce: prevent concurrent polling of MCE events

PATCH Applied Aristeu Rozanski 5 participants 38 messages May 15, 2023 – Jul 21, 2023

Subsystems (3)

v1 of 4 Applied Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mce: prevent concurrent polling of MCE events

Error injection in modern HP machines with CMCI disabled will cause the injected MCE to be found only by polling. Because these newer machines have a big number of CPUs per package, it makes a lot more likely for multiple CPUs polling IMC registers (that are shared in the same package) at same time, causing multiple reports of the same MCE.

arch/x86/kernel/cpu/mce/core.c +3 -0
@@ -1597,6 +1597,7 @@ static unsigned long check_interval = INITIAL_CHECK_INTERVAL;
 
 static DEFINE_PER_CPU(unsigned long, mce_next_interval); /* in jiffies */
 static DEFINE_PER_CPU(struct timer_list, mce_timer);
+static DEFINE_RAW_SPINLOCK(timer_fn_lock);
 
 static unsigned long mce_adjust_timer_default(unsigned long interval)
 {
@@ -1628,7 +1629,9 @@ static void mce_timer_fn(struct timer_list *t)
 	iv = __this_cpu_read(mce_next_interval);
 
 	if (mce_available(this_cpu_ptr(&cpu_info))) {
+		raw_spin_lock(&timer_fn_lock);
 		machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+		raw_spin_unlock(&timer_fn_lock);
 
 		if (mce_intel_cmci_poll()) {
 			iv = mce_adjust_timer(iv);
maintainer
#1 lore raw
Quoted 4 lines · Aristeu Rozanski

Error injection in modern HP machines with CMCI disabled will cause the injected MCE to be found only by polling. Because these newer machines have a big number of CPUs per package, it makes a lot more likely for multiple CPUs polling IMC registers (that are shared in the same package) at same time,

So the MCE code already has some notion of shared banks and tries to pay attention to it.

I think it'll be cleaner if machine_check_poll() paid attention to that and polled only on the smallest CPU number which shares the bank instead of doing an unconditional wholesale spinlock grabbing which is not really needed...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#2 lore raw
Quoted 2 lines

So the MCE code already has some notion of shared banks and tries to pay attention to it.

The MCE code only knows sharing scope if CMCI is enabled. The algorithm to determine ownership of a shared bank is "first to find the bank is the owner" as determined by successful setting of bit 30 in IA32_MCi_CTL2. There isn't any other practical way to determine scope of bank sharing.

In Aristeu's case with the HP systems CMCI is disabled. I saw the same on a system here when booted with "mce=no_cmci". Twelve CPUs (out of 144) were polling on the same jiffy and reported the same error.

-Tony

maintainer
#3 lore raw
Quoted 4 lines · Tony

The MCE code only knows sharing scope if CMCI is enabled. The algorithm to determine ownership of a shared bank is "first to find the bank is the owner" as determined by successful setting of bit 30 in IA32_MCi_CTL2. There isn't any other practical way to determine scope of bank sharing.

I'm not talking about this ownership - I'm talking about how many logical cores get to see the same error info because the bank is replicated across all of them in *hardware*.

Quoted 3 lines

In Aristeu's case with the HP systems CMCI is disabled. I saw the same on a system here when booted with "mce=no_cmci". Twelve CPUs (out of 144) were polling on the same jiffy and reported the same error.

Is CMCI disabled a valid use case or is this someone doing testing?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#4 lore raw
Quoted 3 lines

I'm not talking about this ownership - I'm talking about how many logical cores get to see the same error info because the bank is replicated across all of them in *hardware*.

The hardware doesn't enumerate that banks are shared, or which logical CPUs share each bank. The sharing scope can, and has, changed from one generation to another. E.g. Banks 0 and 1 were core scoped from Nehalem until Cascade Lake. But Icelake and subsequent (so far) CPUs made them thread scoped.

-Tony

maintainer
#5 lore raw
Quoted 9 lines · Tony
Quoted 3 lines

I'm not talking about this ownership - I'm talking about how many logical cores get to see the same error info because the bank is replicated across all of them in *hardware*.

The hardware doesn't enumerate that banks are shared, or which logical CPUs share each bank. The sharing scope can, and has, changed from one generation to another. E.g. Banks 0 and 1 were core scoped from Nehalem until Cascade Lake. But Icelake and subsequent (so far) CPUs made them thread scoped.

So this unconditional serialization is just gross. This is a per-cpu timer grabbing a global lock. It is going to make unrelated threads all wait on a single lock while polling, for nothing.

On a huge box with gazillion cores and when the timers fire just at the right time, they're all going to hit that lock real hard.

There has to be a better way...

Also, from my previous mail: Is CMCI disabled a valid use case or is this someone doing testing?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#6 lore raw
Quoted 6 lines

So this unconditional serialization is just gross. This is a per-cpu timer grabbing a global lock. It is going to make unrelated threads all wait on a single lock while polling, for nothing.

On a huge box with gazillion cores and when the timers fire just at the right time, they're all going to hit that lock real hard.

Linux does try to avoid all the CPUs taking the polling interrupt at the same time. See my earlier e-mail:

https://lore.kernel.org/all/[email protected]/

but there are often more CPUs than HZ ticks to spread things out. So still get a few collisions.

Quoted 1 line

There has to be a better way...

Options:
1) Divide up the lock.
1a) One lock per machine check bank
1b) One lock per socket
1c) Combo of 1a & 1b
2) Enumerate the bank sharing with a big table per CPU model number
[No, we really don't want to do that!]
3) Allow multiple CPUs to read the same error. Add a filter to suppress duplicates in the logging path.

Quoted 2 lines

Also, from my previous mail: Is CMCI disabled a valid use case or is this someone doing testing?

I disabled CMCI for debugging reasons. Aristeu needs to comment on his use case.

-Tony

#7 lore raw
Quoted 2 lines · Tony

I disabled CMCI for debugging reasons. Aristeu needs to comment on his use case.

Partner has a toggle in BIOS and reported the issue. I suspect it has to do with a big mutual customer that wants to reduce CE storms caused CPU spikes to a minimal, but I'll get more details.

Signature · 1 line
Aristeu
maintainer
#8 lore raw
Quoted 3 lines

Partner has a toggle in BIOS and reported the issue. I suspect it has to do with a big mutual customer that wants to reduce CE storms caused CPU spikes to a minimal, but I'll get more details.

If CMCI storms are their problem ... perhaps this is an answer:

https://lore.kernel.org/all/[email protected]/

-Tony

#9 lore raw
Quoted 3 lines · Tony

If CMCI storms are their problem ... perhaps this is an answer:

https://lore.kernel.org/all/[email protected]/

Apparently they're disabling CMCI for CE and UCNA events when OS-first is enabled for two entire lines of products. So there might be more behind this than I suspected. I just sent a message to their engineers to find out what's going on.

Now on the CMCI storm mitigation, wouldn't a system that got a storm be subject to this same issue of duplicated events since it switches to polling?

Signature · 1 line
Aristeu
maintainer
#10 lore raw
Quoted 2 lines

Now on the CMCI storm mitigation, wouldn't a system that got a storm be subject to this same issue of duplicated events since it switches to polling?

Current code switches to polling. But I think that it makes use of the bank ownership information that it derived when CMCI was originally enabled. It wouldn't surprise me if things go wrong if you mix in CPU hotplug with a storm and offline the CPU that "owns" a bank. Perhaps the ownership will get passed to another CPU in the same scope. But I'm not sure.

Those new patches don't switch to polling. They just reset the threshold in MCi_CTL2 to a very big value to throttle the number of interrupts for corrected errors, while still processing the UCNA errors that are also signaled using CMCI right away.

-Tony

maintainer
#11 lore raw
In reply to Aristeu Rozanski
Quoted 9 lines · Aristeu Rozanski
Quoted 3 lines · Tony

If CMCI storms are their problem ... perhaps this is an answer:

https://lore.kernel.org/all/[email protected]/

Apparently they're disabling CMCI for CE and UCNA events when OS-first is enabled for two entire lines of products. So there might be more behind this than I suspected. I just sent a message to their engineers to find out what's going on.

Might also ask them to test Tony's patches, see whether they help.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#12 lore raw
In reply to Luck, Tony
Quoted 2 lines · Tony

I disabled CMCI for debugging reasons. Aristeu needs to comment on his use case.

Got an answer from them. They disable it in two lines of products (one uses IceLake, the other uses Sapphire Rapids) and in these lines they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled. They did make Intel aware of it (maybe you heard details about it? If not I can get them in contact with you) but it's not certain if this can be fixed or not.

Signature · 1 line
Aristeu
#13 lore raw
Quoted 10 lines · Aristeu Rozanski
Quoted 2 lines · Tony

I disabled CMCI for debugging reasons. Aristeu needs to comment on his use case.

Got an answer from them. They disable it in two lines of products (one uses IceLake, the other uses Sapphire Rapids) and in these lines they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled. They did make Intel aware of it (maybe you heard details about it? If not I can get them in contact with you) but it's not certain if this can be fixed or not.

Tony?

Signature · 1 line
Aristeu
maintainer
#14 lore raw
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

I disabled CMCI for debugging reasons. Aristeu needs to comment on his use case.

Got an answer from them. They disable it in two lines of products (one uses IceLake, the other uses Sapphire Rapids) and in these lines they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled. They did make Intel aware of it (maybe you heard details about it? If not I can get them in contact with you) but it's not certain if this can be fixed or not.

Since mce=no_cmci is a supported mode, and end-users are making use of it on production systems ... I think Linux needs a patch to prevent multiple CPUs logging the same error.

I don't have any other ideas on how to do that besides Aristeu's patch to add a spinlock to prevent simultaneous access. This isn't a hot path, and in most cases not contended, so I don't see much downside.

-Tony

maintainer
#15 lore raw
Quoted 5 lines · Tony

Since mce=no_cmci is a supported mode, and end-users are making use of it on production systems ... I think Linux needs a patch to prevent multiple CPUs logging the same error.

I don't have any other ideas

I have some:

1. One can teach mce_gen_pool_add() to scan what's already on the list and drop duplicate errors. Not sure if it is worth it.

or

2. Carve out that abysmal locking into a function called poll_cmci_disabled() or so, run this function only on CMCI machines and make sure the use case is properly explained in the commit message.

I have no clue what "they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled."

I thought HP use firmware first and all that magic EFI gunk to do RAS and don't rely on OS facilities. Oh well...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#16 lore raw
Quoted 6 lines
Quoted 1 line

I don't have any other ideas

I have some:

1. One can teach mce_gen_pool_add() to scan what's already on the list and drop duplicate errors. Not sure if it is worth it.

That would need to take care to only strip out truly duplicated reports, but leave in genuine repeats of errors from the same location. I agree this path isn't worth taking.

Quoted 6 lines

or

2. Carve out that abysmal locking into a function called poll_cmci_disabled() or so, run this function only on CMCI machines and make sure the use case is properly explained in the commit message.

How much code do you want to duplicate? machine_check_poll() is ~100 lines, most of in the racy section from:

                m.status = mce_rdmsrl(mca_msr_reg(i, MCA_STATUS));
to
                mce_wrmsrl(mca_msr_reg(i, MCA_STATUS), 0);

That looks like long term maintenance pain to keep the CMCI on/off versions in step.

Quoted 2 lines

I have no clue what "they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled."

I can't parse that either. Maybe rooted in the fact that UCNA ignores the threshold in MCi_CTL2 ... but maybe there's some missing description of what the SMI handler is doing.

Quoted 2 lines

I thought HP use firmware first and all that magic EFI gunk to do RAS and don't rely on OS facilities. Oh well...

HPE are big fans of firmware first for corrected errors. It's more complicated for uncorrected ones as the OS really needs to know about those.

-Tony

maintainer
#17 lore raw
Quoted 2 lines · Tony

That would need to take care to only strip out truly duplicated reports, but leave in genuine repeats of errors from the same location.

Yeah, timestamps will make them unique.

Quoted 1 line

I agree this path isn't worth taking.

I have a feeling we might need to dedup at some point but we can do that in luserspace.

Quoted 1 line

How much code do you want to duplicate?

The usual. See untested diff below.

Quoted 3 lines

I can't parse that either. Maybe rooted in the fact that UCNA ignores the threshold in MCi_CTL2 ... but maybe there's some missing description of what the SMI handler is doing.

If you don't know what this does either, then this is going nowhere before it is properly explained.

Quoted 2 lines

HPE are big fans of firmware first for corrected errors. It's more complicated for uncorrected ones as the OS really needs to know about those.

Bah, the firmware is so powerful and perfect. What does it need the OS for? </sarcasm>

---

Btw, why was that spinlock raw?

arch/x86/kernel/cpu/mce/core.c +8 -1
index 22dfcb2adcd7..bcdf5b52003c 100644
@@ -1595,6 +1595,13 @@ static unsigned long mce_adjust_timer_default(unsigned long interval)
 
 static unsigned long (*mce_adjust_timer)(unsigned long interval) = mce_adjust_timer_default;
 
+static void poll_cmci_disabled(void)
+{
+	machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+}
+
+static void (*poll_cmci_disabled)(void) = poll_cmci_disabled_default;
+
 static void __start_timer(struct timer_list *t, unsigned long interval)
 {
 	unsigned long when = jiffies + interval;
@@ -1618,7 +1625,7 @@ static void mce_timer_fn(struct timer_list *t)
 	iv = __this_cpu_read(mce_next_interval);
 
 	if (mce_available(this_cpu_ptr(&cpu_info))) {
-		machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+		poll_cmci_disabled();
 
 		if (mce_intel_cmci_poll()) {
 			iv = mce_adjust_timer(iv);
arch/x86/kernel/cpu/mce/intel.c +13 -1
index 95275a5e57e0..4722a57baf1e 100644
@@ -71,6 +71,8 @@ enum {
 	CMCI_STORM_SUBSIDED,
 };
 
+static DEFINE_SPINLOCK(timer_fn_lock);
+
 static atomic_t cmci_storm_on_cpus;
 
 static int cmci_supported(int *banks)
@@ -426,12 +428,22 @@ void cmci_disable_bank(int bank)
 	raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
 }
 
+static void poll_cmci_disabled_intel(void)
+{
+	raw_spin_lock(&timer_fn_lock);
+	machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+	raw_spin_unlock(&timer_fn_lock);
+}
+
 void intel_init_cmci(void)
 {
 	int banks;
 
-	if (!cmci_supported(&banks))
+	if (!cmci_supported(&banks)) {
+		if (mca_cfg.cmci_disabled)
+			poll_cmci_disabled = poll_cmci_disabled_intel;
 		return;
+	}
 
 	mce_threshold_vector = intel_threshold_interrupt;
 	cmci_discover(banks);
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#18 lore raw
Quoted 3 lines
Quoted 1 line

How much code do you want to duplicate?

The usual. See untested diff below.

That looks like it covers the "mce=cmci_off" case. What about the case where CMCI is disabled temporarily because of a storm?

-Tony

maintainer
#19 lore raw
Quoted 2 lines · Tony

That looks like it covers the "mce=cmci_off" case. What about the case where CMCI is disabled temporarily because of a storm?

Then the decision which function to call will have to be dynamic and not with function pointers assigned once, statically.

This was just an example about how it could be done.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#21 lore raw
In reply to Borislav Petkov
Quoted 2 lines · Borislav Petkov

I have a feeling we might need to dedup at some point but we can do that in luserspace.

Problem with throwing this userspace is that it won't get rid of the duplication on the kernel log.

Signature · 1 line
Aristeu
maintainer
#22 lore raw
Quoted 2 lines · Aristeu Rozanski

Problem with throwing this userspace is that it won't get rid of the duplication on the kernel log.

The kernel log never was and is a source to be scraped for hw errors. The right thing is called tracepoints and rasdaemon uses them already.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#23 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Aristeu Rozanski

Problem with throwing this userspace is that it won't get rid of the duplication on the kernel log.

The kernel log never was and is a source to be scraped for hw errors. The right thing is called tracepoints and rasdaemon uses them already.

Yes, but I'm talking about flooding the log with duplicates.

Signature · 1 line
Aristeu
maintainer
#24 lore raw
Quoted 1 line · Aristeu Rozanski

Yes, but I'm talking about flooding the log with duplicates.

And?

We don't *have* to log them if someone's too confused by staring at something someone should not be staring at in the first place.

But let's see how valid that "use case" actually is first...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#25 lore raw
Quoted 1 line

But let's see how valid that "use case" actually is first...

I'm going to meet with the users later this week to find out what they are doing.

-Tony

#26 lore raw
In reply to Borislav Petkov
Quoted 7 lines · Borislav Petkov
Quoted 1 line · Aristeu Rozanski

Yes, but I'm talking about flooding the log with duplicates.

And?

We don't *have* to log them if someone's too confused by staring at something someone should not be staring at in the first place.

It makes support people's lives worse by reducing the amount of useful information in dmesg, when it can be avoided, in the case of duplicates. Getting rid of the messages completely would be harmful as well, not always customers run rasdaemon or provide the database with events, which sometimes can be useful as hints.

Quoted 1 line

But let's see how valid that "use case" actually is first...

Of course. Tony is already aware who inside Intel talked to them and they'll talk.

--
Aristeu

maintainer
#27 lore raw
Quoted 5 lines · Aristeu Rozanski

It makes support people's lives worse by reducing the amount of useful information in dmesg, when it can be avoided, in the case of duplicates. Getting rid of the messages completely would be harmful as well, not always customers run rasdaemon or provide the database with events, which sometimes can be useful as hints.

Lemme make sure you realize what you're arguing for: you want to not log MCE duplicates in a ring buffer which can get overwritten due to a bunch of reasons and as such is best effort only anyway.

Hell, it can get overwritten even by non-duplicate MCEs if they're just enough many to overflow the buffer's size.

So relying on dmesg for any serious error counting or whatnot is a exercise in futility.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#28 lore raw
Quoted 9 lines · Borislav Petkov

Lemme make sure you realize what you're arguing for: you want to not log MCE duplicates in a ring buffer which can get overwritten due to a bunch of reasons and as such is best effort only anyway.

Hell, it can get overwritten even by non-duplicate MCEs if they're just enough many to overflow the buffer's size.

So relying on dmesg for any serious error counting or whatnot is a exercise in futility.

I'm not sure when I made you believe I want to do error counting or error parsing from the kernel log.

I started saying that we need to avoid, when possible, to waste kernel log. Duplicates are a waste. You mentioned not even logging MCEs at all and I said that I believe it's not right, as it might be useful to be aware of MCEs being delivered before a crash.

--
Aristeu

maintainer
#29 lore raw
Quoted 4 lines · Aristeu Rozanski

I started saying that we need to avoid, when possible, to waste kernel log. Duplicates are a waste. You mentioned not even logging MCEs at all and I said that I believe it's not right, as it might be useful to be aware of MCEs being delivered before a crash.

You don't have to summarize the whole thread - I know what we're talking about.

And just to give you another example: if the MCE which caused that crash gets overwritten in dmesg, your saved kernel log is useless as you want the *first* one logged.

That's all I'm saying - the kernel log is best effort - nothing more.

Independent of that, yes, we will try not to pollute it with duplicates once we know what the issue exactly is which makes people disable CMCI.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#30 lore raw
Quoted 2 lines

Independent of that, yes, we will try not to pollute it with duplicates once we know what the issue exactly is which makes people disable CMCI.

I talked to the vendor today. They have customers who want the OS to see all reported corrected errors, but they also need to have their firmware keep logs of all uncorrected errors (so their field engineers can replace the DIMMs that are throwing the UC errors).

Intel machine check architecture has conflated processing of corrected errors and "UCNA" (the log that comes from the memory controller when ECC says the error is uncorrected ... this is distinct from the log that comes from the core when the poison data is consumed and signals a machine check).

So the closest they can get to meeting both the customer requirement and their service requirement is to have their BIOS configure to get an SMI for the UCNA *and* tell the users to run in mce=no_cmci mode. If Linux enables CMCI, then the h/w will also generate an SMI for every corrected error. Neither the customer, nor the vendor wants that overhead.

Bottom line: This is a legitimate, production, use case.

-Tony

maintainer
#31 lore raw
Quoted 8 lines · Tony
Quoted 2 lines

Independent of that, yes, we will try not to pollute it with duplicates once we know what the issue exactly is which makes people disable CMCI.

I talked to the vendor today. They have customers who want the OS to see all reported corrected errors, but they also need to have their firmware keep logs of all uncorrected errors (so their field engineers can replace the DIMMs that are throwing the UC errors).

Quoted 4 lines

Intel machine check architecture has conflated processing of corrected errors and "UCNA" (the log that comes from the memory controller when ECC says the error is uncorrected ... this is distinct from the log that comes from the core when the poison data is consumed and signals a machine check).

I'd expect poison data consumption not to be a "UCNA" - i.e., "No Action" required :)

Quoted 7 lines

So the closest they can get to meeting both the customer requirement and their service requirement is to have their BIOS configure to get an SMI for the UCNA *and* tell the users to run in mce=no_cmci mode. If Linux enables CMCI, then the h/w will also generate an SMI for every corrected error. Neither the customer, nor the vendor wants that overhead.

Bottom line: This is a legitimate, production, use case.

Ok, that makes more sense.

Thanks.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#32 lore raw
Quoted 9 lines · Borislav Petkov
Quoted 8 lines · Tony
Quoted 2 lines

Independent of that, yes, we will try not to pollute it with duplicates once we know what the issue exactly is which makes people disable CMCI.

I talked to the vendor today. They have customers who want the OS to see all reported corrected errors, but they also need to have their firmware keep logs of all uncorrected errors (so their field engineers can replace the DIMMs that are throwing the UC errors).

Quoted 8 lines
Quoted 4 lines

Intel machine check architecture has conflated processing of corrected errors and "UCNA" (the log that comes from the memory controller when ECC says the error is uncorrected ... this is distinct from the log that comes from the core when the poison data is consumed and signals a machine check).

I'd expect poison data consumption not to be a "UCNA" - i.e., "No Action" required :)

So "UCNA" is like the AMD "Deferred" severity it seems. How is this different from "Action Optional"? I've been equating DFR and AO.

Quoted 10 lines
Quoted 7 lines

So the closest they can get to meeting both the customer requirement and their service requirement is to have their BIOS configure to get an SMI for the UCNA *and* tell the users to run in mce=no_cmci mode. If Linux enables CMCI, then the h/w will also generate an SMI for every corrected error. Neither the customer, nor the vendor wants that overhead.

Bottom line: This is a legitimate, production, use case.

Ok, that makes more sense.

Please do restrict this change to Intel, or at least !AMD, systems. AMD MCA implemenation guarantees each MCA bank can only be accessed by a single logical CPU, so there is no contention.

Thanks,
Yazen

maintainer
#33 lore raw
Quoted 2 lines

So "UCNA" is like the AMD "Deferred" severity it seems. How is this different from "Action Optional"? I've been equating DFR and AO.

Categories of uncorrected errors memory errors on Intel

1) "UCNA" ... these are logged by memory controllers when ECC says that a memory read cannot supply correct data. If CMCI is enabled, signaled with CMCI. Note that these will occur on prefetch or speculative reads as well as "regular" reads. The data might never be consumed.

2) "SRAO". This is now legacy. Pre-Icelake systems log these for uncorrected errors found by the patrol scrubber, and for evictions of poison from L3 cache (if that poison was due to an ECC failure in the cache itself, not for poison created elsewhere and currently resident in the cache). Signaled with a broadcast machine check. Icelake and newer systems use UCNA for these.

3) "SRAR". Attempt to consume poison (either data read or instruction fetch). Signaled with machine check. Pre-Skylake this was broadcast. Skylake and newer have an opt-in mechanism to request #MC delivery to just the logical CPU trying to consume (Linux always opts-in).

UCNA = Uncorrected No Action required. But Linux does take action to try to offline the page.

SRAO = Software Recoverable Action Optional. As with UCNA Linux tries to offline the page.

SRAR = Software Recoverable Action Required. Linux will replace a clean page with a new copy if it can (think read-only text pages mapped from ELF executable). If not it sends SIGBUS to the application. Some SRAR in the kernel are recoverable ... see the copy_mc*() functions.

-Tony

#34 lore raw
Quoted 3 lines · Tony
Quoted 2 lines

So "UCNA" is like the AMD "Deferred" severity it seems. How is this different from "Action Optional"? I've been equating DFR and AO.

Thanks Tony.

Quoted 6 lines

Categories of uncorrected errors memory errors on Intel

1) "UCNA" ... these are logged by memory controllers when ECC says that a memory read cannot supply correct data. If CMCI is enabled, signaled with CMCI. Note that these will occur on prefetch or speculative reads as well as "regular" reads. The data might never be consumed.

Yes, this is like AMD.

Key differences:
* Logged using "Deferred" severity. However, deferred errors aren't always from the memory controller. So there still needs to be an error code check in addition to severity.
* Signalled with a Deferred error APIC interrupt. This way UC errors can be signalled independently of CEs.

Quoted 5 lines

2) "SRAO". This is now legacy. Pre-Icelake systems log these for uncorrected errors found by the patrol scrubber, and for evictions of poison from L3 cache (if that poison was due to an ECC failure in the cache itself, not for poison created elsewhere and currently resident in the cache). Signaled with a broadcast machine check. Icelake and newer systems use UCNA for these.

Yes, this mostly fails within the Deferred/UCNA case for AMD also.

Quoted 7 lines

3) "SRAR". Attempt to consume poison (either data read or instruction fetch). Signaled with machine check. Pre-Skylake this was broadcast. Skylake and newer have an opt-in mechanism to request #MC delivery to just the logical CPU trying to consume (Linux always opts-in).

UCNA = Uncorrected No Action required. But Linux does take action to try to offline the page.

That's right. So we'll use this for the Deferred memory error case. But there will need to be updates for these to be actionable on AMD systems.

Quoted 2 lines

SRAO = Software Recoverable Action Optional. As with UCNA Linux tries to offline the page.

Okay, so ignore these going forward.

Quoted 4 lines

SRAR = Software Recoverable Action Required. Linux will replace a clean page with a new copy if it can (think read-only text pages mapped from ELF executable). If not it sends SIGBUS to the application. Some SRAR in the kernel are recoverable ... see the copy_mc*() functions.

Yep, this mostly works. There's still an AMD IF Unit quirk that needs to be handled. And the kernel recovery cases needs to be tested.

Thanks again!

-Yazen

#35 lore raw
In reply to Borislav Petkov

Hi Borislav,

Quoted 2 lines · Borislav Petkov

Independent of that, yes, we will try not to pollute it with duplicates once we know what the issue exactly is which makes people disable CMCI.

how about this (untested, with possibly a short comment on intel_cmci_poll_lock()):

arch/x86/kernel/cpu/mce/core.c +2 -0
@@ -1618,7 +1618,9 @@ static void mce_timer_fn(struct timer_list *t)
 	iv = __this_cpu_read(mce_next_interval);
 
 	if (mce_available(this_cpu_ptr(&cpu_info))) {
+		bool locked = intel_cmci_poll_lock();
 		machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+		intel_cmci_poll_unlock(locked);
 
 		if (mce_intel_cmci_poll()) {
 			iv = mce_adjust_timer(iv);
arch/x86/kernel/cpu/mce/intel.c +31 -4
index 95275a5e57e0..c688c088f5cf 100644
@@ -73,13 +73,10 @@ enum {
 
 static atomic_t cmci_storm_on_cpus;
 
-static int cmci_supported(int *banks)
+static int cmci_supported_hw(int *banks)
 {
 	u64 cap;
 
-	if (mca_cfg.cmci_disabled || mca_cfg.ignore_ce)
-		return 0;
-
 	/*
 	 * Vendor check is not strictly needed, but the initial
 	 * initialization is vendor keyed and this
@@ -96,6 +93,14 @@ static int cmci_supported(int *banks)
 	return !!(cap & MCG_CMCI_P);
 }
 
+static int cmci_supported(int *banks)
+{
+	if (mca_cfg.cmci_disabled || mca_cfg.ignore_ce)
+		return 0;
+
+	return cmci_supported_hw(banks);
+}
+
 static bool lmce_supported(void)
 {
 	u64 tmp;
@@ -519,3 +524,25 @@ bool intel_filter_mce(struct mce *m)
 
 	return false;
 }
+
+static DEFINE_SPINLOCK(cmci_poll_lock);
+bool intel_cmci_poll_lock(void)
+{
+	int banks;
+
+	if (!cmci_supported_hw(&banks))
+		return false;
+
+	spin_lock(&cmci_poll_lock);
+
+	return true;
+}
+
+void intel_cmci_poll_unlock(bool locked)
+{
+	if (!locked)
+		return;
+
+	spin_unlock(&cmci_poll_lock);
+}
+
arch/x86/kernel/cpu/mce/internal.h +4 -0
index d2412ce2d312..25bcc4a13e8c 100644
@@ -49,6 +49,8 @@ void intel_init_cmci(void);
 void intel_init_lmce(void);
 void intel_clear_lmce(void);
 bool intel_filter_mce(struct mce *m);
+bool intel_cmci_poll_lock(void);
+void intel_cmci_poll_unlock(bool locked);
 #else
 # define cmci_intel_adjust_timer mce_adjust_timer_default
 static inline bool mce_intel_cmci_poll(void) { return false; }
@@ -58,6 +60,8 @@ static inline void intel_init_cmci(void) { }
 static inline void intel_init_lmce(void) { }
 static inline void intel_clear_lmce(void) { }
 static inline bool intel_filter_mce(struct mce *m) { return false; }
+static inline bool intel_cmci_poll_lock(void) { return false; }
+static inline void intel_cmci_poll_unlock(bool locked) { }
 #endif
 
 void mce_timer_kick(unsigned long interval);
Signature · 1 line
Aristeu
#36 lore raw
In reply to Borislav Petkov
Quoted 2 lines · Borislav Petkov

I have no clue what "they can use thresholding to signal UCNA without signaling any corrected events and it won't work with CMCI enabled."

That's pretty much exactly what was passed to me. I'll ask who the Intel contact was and inform Tony so they can talk internally.

--
Aristeu

#37 lore raw

[tip: ras/core] x86/mce: Prevent duplicate error records

In reply to Aristeu Rozanski
The following commit has been merged into the ras/core branch of tip:

The following commit has been merged into the ras/core branch of tip:

Commit-ID:     c3629dd7e67d6ec5705d33b0de0d142c972fe573
Gitweb:        https://git.kernel.org/tip/c3629dd7e67d6ec5705d33b0de0d142c972fe573
Author:        Borislav Petkov (AMD) <[email protected]>
AuthorDate:    Wed, 19 Jul 2023 14:19:50 +02:00
Committer:     Borislav Petkov (AMD) <[email protected]>
CommitterDate: Fri, 21 Jul 2023 18:55:46 +02:00

x86/mce: Prevent duplicate error records

A legitimate use case of the MCA infrastructure is to have the firmware log all uncorrectable errors and also, have the OS see all correctable errors.

The uncorrectable, UCNA errors are usually configured to be reported through an SMI. CMCI, which is the correctable error reporting interrupt, uses SMI too and having both enabled, leads to unnecessary overhead.

So what ends up happening is, people disable CMCI in the wild and leave on only the UCNA SMI.

When CMCI is disabled, the MCA infrastructure resorts to polling the MCA banks. If a MCA MSR is shared between the logical threads, one error ends up getting logged multiple times as the polling runs on every logical thread.

Therefore, introduce locking on the Intel side of the polling routine to prevent such duplicate error records from appearing.

Based on a patch by Aristeu Rozanski <[email protected]>.

---
 arch/x86/kernel/cpu/mce/core.c     |  9 ++++++++-
 arch/x86/kernel/cpu/mce/intel.c    | 19 ++++++++++++++++++-
 arch/x86/kernel/cpu/mce/internal.h |  1 +
 3 files changed, 27 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mce/core.c +8 -1
index 89e2aab..b8ad5a5 100644
@@ -1608,6 +1608,13 @@ static void __start_timer(struct timer_list *t, unsigned long interval)
 	local_irq_restore(flags);
 }
 
+static void mc_poll_banks_default(void)
+{
+	machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+}
+
+void (*mc_poll_banks)(void) = mc_poll_banks_default;
+
 static void mce_timer_fn(struct timer_list *t)
 {
 	struct timer_list *cpu_t = this_cpu_ptr(&mce_timer);
@@ -1618,7 +1625,7 @@ static void mce_timer_fn(struct timer_list *t)
 	iv = __this_cpu_read(mce_next_interval);
 
 	if (mce_available(this_cpu_ptr(&cpu_info))) {
-		machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+		mc_poll_banks();
 
 		if (mce_intel_cmci_poll()) {
 			iv = mce_adjust_timer(iv);
arch/x86/kernel/cpu/mce/intel.c +18 -1
index 95275a5..f532355 100644
@@ -56,6 +56,13 @@ static DEFINE_PER_CPU(int, cmci_backoff_cnt);
  */
 static DEFINE_RAW_SPINLOCK(cmci_discover_lock);
 
+/*
+ * On systems that do support CMCI but it's disabled, polling for MCEs can
+ * cause the same event to be reported multiple times because IA32_MCi_STATUS
+ * is shared by the same package.
+ */
+static DEFINE_SPINLOCK(cmci_poll_lock);
+
 #define CMCI_THRESHOLD		1
 #define CMCI_POLL_INTERVAL	(30 * HZ)
 #define CMCI_STORM_INTERVAL	(HZ)
@@ -426,12 +433,22 @@ void cmci_disable_bank(int bank)
 	raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
 }
 
+/* Bank polling function when CMCI is disabled. */
+static void cmci_mc_poll_banks(void)
+{
+	spin_lock(&cmci_poll_lock);
+	machine_check_poll(0, this_cpu_ptr(&mce_poll_banks));
+	spin_unlock(&cmci_poll_lock);
+}
+
 void intel_init_cmci(void)
 {
 	int banks;
 
-	if (!cmci_supported(&banks))
+	if (!cmci_supported(&banks)) {
+		mc_poll_banks = cmci_mc_poll_banks;
 		return;
+	}
 
 	mce_threshold_vector = intel_threshold_interrupt;
 	cmci_discover(banks);
arch/x86/kernel/cpu/mce/internal.h +1 -0
index d2412ce..ed4a71c 100644
@@ -274,4 +274,5 @@ static __always_inline u32 mca_msr_reg(int bank, enum mca_msr reg)
 	return 0;
 }
 
+extern void (*mc_poll_banks)(void);
 #endif /* __X86_MCE_INTERNAL_H__ */