Skip to contentSkip to conversation

x86, amd, mce: Prevent potential cpu-online oops

PATCH In review Daniel J Blueman 4 participants 11 messages Apr 4, 2013 – Apr 9, 2013 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86, amd, mce: Prevent potential cpu-online oops

On platforms where all Northbridges may not be visible (due to routing, eg on NumaConnect systems), prevent oopsing due to stale pointer access when offlining cores.

---
 arch/x86/kernel/cpu/mcheck/mce_amd.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/mcheck/mce_amd.c +6 -5
index 1ac581f..53a58c2 100644
@@ -578,8 +578,11 @@ static __cpuinit int threshold_create_bank(unsigned int cpu, unsigned int bank)
 	if (shared_bank[bank]) {
 		nb = node_to_amd_nb(amd_get_nb_id(cpu));
 
+		if (WARN_ON_ONCE(!nb))
+			goto out;
+
 		/* threshold descriptor already initialized on this node? */
-		if (nb && nb->bank4) {
+		if (nb->bank4) {
 			/* yes, use it */
 			b = nb->bank4;
 			err = kobject_add(b->kobj, &dev->kobj, name);
@@ -613,10 +616,8 @@ static __cpuinit int threshold_create_bank(unsigned int cpu, unsigned int bank)
 		atomic_set(&b->cpus, 1);
 
 		/* nb is already initialized, see above */
-		if (nb) {
-			WARN_ON(nb->bank4);
-			nb->bank4 = b;
-		}
+		WARN_ON(nb->bank4);
+		nb->bank4 = b;
 	}
 
 	err = allocate_threshold_blocks(cpu, bank, 0,
Signature · 1 line
1.7.4.1
#1 lore raw

+ if (WARN_ON_ONCE(!nb))
+ goto out;
+

WARN_ON_ONCE() will drop a stack trace to the console - is that going to be useful?

If you want a message perhaps:

		if (!nb) {
			printk_once("something interesting about not having access to north bridge\n")
			goto out;
		}

-Tony

maintainer
#2 lore raw
In reply to Daniel J Blueman
Quoted 6 lines · Daniel J Blueman

On platforms where all Northbridges may not be visible (due to routing, eg on NumaConnect systems), prevent oopsing due to stale pointer access when offlining cores.

Huh, what's up?

This one is almost reverting 21c5e50e15b1a which you wrote in the first place. What's happening? What stale pointer access, where? We have the if (nb ..) guards there.

This commit message needs a *lot* more explanation about what's going on and why we're reverting 21c5e50e15b1a. And why the special handling for shared banks? I presume you offline some of the cores and there's a dangling pointer but again, there are the nb validity guards...

/me is genuinely confused.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#3 lore raw
Quoted 14 lines · Borislav Petkov
Quoted 6 lines · Daniel J Blueman

On platforms where all Northbridges may not be visible (due to routing, eg on NumaConnect systems), prevent oopsing due to stale pointer access when offlining cores.

Huh, what's up?

This one is almost reverting 21c5e50e15b1a which you wrote in the first place. What's happening? What stale pointer access, where? We have the if (nb ..) guards there.

Quoted 7 lines

This commit message needs a *lot* more explanation about what's going on and why we're reverting 21c5e50e15b1a. And why the special handling for shared banks? I presume you offline some of the cores and there's a dangling pointer but again, there are the nb validity guards...

/me is genuinely confused.

You get oopses when offlining cores when there's no NB struct for the shared MC4 bank. In threshold_remove_bank(), there's no "if (!nb)" guard :

	if (shared_bank[bank]) {
		if (!atomic_dec_and_test(&b->cpus)) {
			__threshold_remove_blocks(b);
			per_cpu(threshold_banks, cpu)[bank] = NULL;
			return;
		} else {
			/*
			 * the last CPU on this node using the shared bank is
			 * going away, remove that bank now.
			 */
			nb = node_to_amd_nb(amd_get_nb_id(cpu));
			nb->bank4 = NULL;
		}
	}

nb->bank4 = NULL will oops, since nb is NULL.

It made more sense (to me) to skip the creation of MC4 all together if you can't find the matching northbridge since you can't reliably do the dec_and_test() reference counting on the shared bank when you don't have the common NB struct for all the shared cores.

Or am I just smoking the wrong stuff ?

Cheers,
Steffen

maintainer
#4 lore raw
Quoted 6 lines · Steffen Persvold

It made more sense (to me) to skip the creation of MC4 all together if you can't find the matching northbridge since you can't reliably do the dec_and_test() reference counting on the shared bank when you don't have the common NB struct for all the shared cores.

Or am I just smoking the wrong stuff ?

No, actually *this* explanation should've been in the commit message. You numascale people do crazy things with the hardware :) so explaining yourself more verbosely is an absolute must if anyone is to understand why you're changing the code.

So please write a detailed commit message why you need this change, don't be afraid to talk about the big picture.

Also, I'm guessing this is urgent stuff and it needs to go into 3.9? Yes, no? If yes, this patch should probably be tagged for stable.

Also, please redo this patch against tip:x86/ras which already has patches touching mce_amd.c.

Oh, and lastly, needless to say, it needs to be tested on a "normal", i.e. !numascale AMD multinode box, in case you haven't done so yet. :-)

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
-- 
#5 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Steffen Persvold

It made more sense (to me) to skip the creation of MC4 all together if you can't find the matching northbridge since you can't reliably do the dec_and_test() reference counting on the shared bank when you don't have the common NB struct for all the shared cores.

Quoted 6 lines
Quoted 1 line

Or am I just smoking the wrong stuff ?

No, actually *this* explanation should've been in the commit message. You numascale people do crazy things with the hardware :) so explaining yourself more verbosely is an absolute must if anyone is to understand why you're changing the code.

Ok :)

Quoted 3 lines

So please write a detailed commit message why you need this change, don't be afraid to talk about the big picture.

Will do.

Quoted 3 lines

Also, I'm guessing this is urgent stuff and it needs to go into 3.9? Yes, no? If yes, this patch should probably be tagged for stable.

Yes. We found the issue on -stable at first (3.8.2 iirc) because it doesn't have the multi-domain support we needed (which is added in 3.9).

Quoted 3 lines

Also, please redo this patch against tip:x86/ras which already has patches touching mce_amd.c.

Ok.

Quoted 4 lines

Oh, and lastly, needless to say, it needs to be tested on a "normal", i.e. !numascale AMD multinode box, in case you haven't done so yet. :-)

It has been tested on "normal" platforms and NumaConnect platforms (Fam10h and Fam15h AMD processors, SCM and MCM versions).

Cheers,
Steffen

#6 lore raw
In reply to Borislav Petkov
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Steffen Persvold

It made more sense (to me) to skip the creation of MC4 all together if you can't find the matching northbridge since you can't reliably do the dec_and_test() reference counting on the shared bank when you don't have the common NB struct for all the shared cores.

Quoted 7 lines
Quoted 1 line

Or am I just smoking the wrong stuff ?

No, actually *this* explanation should've been in the commit message. You numascale people do crazy things with the hardware :) so explaining yourself more verbosely is an absolute must if anyone is to understand why you're changing the code.

Boris,

A question came up. Why have this "shared" bank concept for the kobjects at all ? What's the advantage ? Before our patch, when running on our architecture but without pci domains for "slave" servers, everything was working fine except the de-allocation oops due to the NULL pointer when offlining cores.

Why not let all cores just create their individual kobject and skip this "shared" nb->bank4 concept ? Any disadvantage to that (apart from the obvious storage bloat?).

Cheers,
Steffen

maintainer
#7 lore raw
Quoted 3 lines · Steffen Persvold

Why not let all cores just create their individual kobject and skip this "shared" nb->bank4 concept ? Any disadvantage to that (apart from the obvious storage bloat?).

Well, bank4 is shared across cores on the northbridge in *hardware*. So it is only logical to represent the hardware layout correctly in software.

Also, if you want to configure any settings over one core's sysfs nodes, you want those to be visible across all cores automagically:

# echo 12 > machinecheck2/northbridge/dram/threshold_limit
# grep . -Er /sys/devices/system/machinecheck | grep -E "dram.*threshold_limit"
/sys/devices/system/machinecheck/machinecheck0/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck1/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck2/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck3/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck4/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck5/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck6/northbridge/dram/threshold_limit:12
/sys/devices/system/machinecheck/machinecheck7/northbridge/dram/threshold_limit:12

HTH.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#8 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Steffen Persvold

Why not let all cores just create their individual kobject and skip this "shared" nb->bank4 concept ? Any disadvantage to that (apart from the obvious storage bloat?).

Well, bank4 is shared across cores on the northbridge in *hardware*.

Well, yes I was aware of that :)

Quoted 5 lines

So it is only logical to represent the hardware layout correctly in software.

Also, if you want to configure any settings over one core's sysfs nodes, you want those to be visible across all cores automagically:

Hmm, yes of course. This of course breaks on our slave servers when the shared mechanism doesn't work properly (i.e NB not visible). Then all cores gets individual kobjects and there can be discrepancies between what the hardware is programmed to and what is reflected in /sys on some cores..

Ok, we go with our first approach to not create MC4 at all if NB isn't visible.

We'll redo the patch against the tip:x86/ras branch.

Cheers,
Steffen

maintainer
#9 lore raw
Quoted 5 lines · Steffen Persvold

Hmm, yes of course. This of course breaks on our slave servers when the shared mechanism doesn't work properly (i.e NB not visible). Then all cores gets individual kobjects and there can be discrepancies between what the hardware is programmed to and what is reflected in /sys on some cores..

Hold on, are you saying you have cores with an invisible NB? How does that even work? Or is it only invisible to sw?

Which would explain why you want the bank4 thing to be per-core. However, the ugliness with this approach is that you need to note which cores share the bank and do the proper updates on all sharing cores upon modifications from sysfs.

Oh well, if you can come up with a clean patchset doing that and without introducing too much bloat, we can talk about it... :-)

Quoted 2 lines

Ok, we go with our first approach to not create MC4 at all if NB isn't visible.

... but this would be the simpler fix. Unless it is still not fixing everything for you guys.

Quoted 1 line

We'll redo the patch against the tip:x86/ras branch.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
-- 
#10 lore raw
Quoted 9 lines · Borislav Petkov
Quoted 5 lines · Steffen Persvold

Hmm, yes of course. This of course breaks on our slave servers when the shared mechanism doesn't work properly (i.e NB not visible). Then all cores gets individual kobjects and there can be discrepancies between what the hardware is programmed to and what is reflected in /sys on some cores..

Hold on, are you saying you have cores with an invisible NB? How does that even work? Or is it only invisible to sw?

only invisible to the kernel because the multi-pci-domains isn't working pre 3.9 on our architecture.

cheers,
Steffen