Skip to contentSkip to conversation

NOPL on 32-bit

PATCH In review Borislav Petkov 5 participants 21 messages Oct 3, 2010 – Oct 5, 2010 x86

lore
Share & tools
lore

Borislav Petkov

author maintainer
#0 lore raw

Hi,

it looks like we currently clear X86_FEATURE_NOPL in detect_nopl() unconditionally and not only on 32-bit, as it should be according to ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8.

I'm thinking maybe something like the following might be in order

--

arch/x86/kernel/cpu/common.c +2 -0
index caa967d..9a2cfeb 100644
@@ -713,7 +713,9 @@ void __init early_cpu_init(void)
  */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)

--

or maybe get even more radical and rip out the whole X86_FEATURE_NOPL thing?

Signature · 2 lines
Regards/Gruss,
    Boris.
#1 lore raw
Quoted 15 lines · Borislav Petkov

Hi,

it looks like we currently clear X86_FEATURE_NOPL in detect_nopl() unconditionally and not only on 32-bit, as it should be according to ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8.

I'm thinking maybe something like the following might be in order

--

arch/x86/kernel/cpu/common.c +0 -0
index caa967d..9a2cfeb 100644

arch/x86/kernel/cpu/common.c void __init early_cpu_init(void)

Quoted 10 lines
arch/x86/kernel/cpu/common.c +2 -1
@@ -713,7 +713,9 @@ void __init early_cpu_init(void)
  */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)
--

Yes, that's a bug. It should be disabled only on 32 bits.

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
author maintainer
#2 lore raw

PATCH 1/1 x86, cpu: X86_FEATURE_NOPL should be disabled on 32-bit only

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit.

---
 arch/x86/kernel/cpu/common.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/common.c +2 -0
index caa967d..9a2cfeb 100644
@@ -713,7 +713,9 @@ void __init early_cpu_init(void)
  */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)
Signature · 5 lines
1.7.2.3

-- 
Regards/Gruss,
    Boris.
#3 lore raw

Re: [PATCH] x86, cpu: X86_FEATURE_NOPL should be disabled on 32-bit only

I think we actually need to set it on 64 bits.

Quoted 30 lines · Borislav Petkov

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit.

---
arch/x86/kernel/cpu/common.c | 2 ++

1 files changed, 2 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/common.c +2 -0
index caa967d..9a2cfeb 100644
@@ -713,7 +713,9 @@ void __init early_cpu_init(void)
 */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)
Signature · 5 lines
1.7.2.3

-- 
Regards/Gruss,
   Boris.
Signature · 1 line
Sent from my mobile phone.  Please pardon any lack of formatting.
author maintainer
#4 lore raw

Re: [PATCH] x86, cpu: X86_FEATURE_NOPL should be disabled on 32-bit only

From: "H. Peter Anvin" <[email protected]>
Date: Sun, Oct 03, 2010 at 11:19:18AM -0700

Quoted 1 line

I think we actually need to set it on 64 bits.

Ok, correct me if I'm wrong but I think it is already "indirectly" set on 64-bits through REQUIRED_MASK3:

#define REQUIRED_MASK3 (NEED_NOPL)

which is defined as

#define NEED_NOPL (1<<(X86_FEATURE_NOPL & 31))

when either CONFIG_X86_P6_NOP or CONFIG_X86_64 are selected.

and cpu_has() does

#define cpu_has(c, bit)                                                 \
        (__builtin_constant_p(bit) &&                                   \
	...
           (((bit)>>5)==3 && (1UL<<((bit)&31) & REQUIRED_MASK3)) ||     \
	...
	? 1 : test_cpu_cap(c, bit))

which returns 1 when testing for X86_FEATURE_NOPL, no?

Signature · 2 lines
Regards/Gruss,
    Boris.
#5 lore raw

Re: [PATCH] x86, cpu: X86_FEATURE_NOPL should be disabled on 32-bit only

Sort of... only if the input is a constant.

Quoted 30 lines · Borislav Petkov

From: "H. Peter Anvin" <[email protected]>
Date: Sun, Oct 03, 2010 at 11:19:18AM -0700

Quoted 1 line

I think we actually need to set it on 64 bits.

Ok, correct me if I'm wrong but I think it is already "indirectly" set on 64-bits through REQUIRED_MASK3:

#define REQUIRED_MASK3 (NEED_NOPL)

which is defined as

#define NEED_NOPL (1<<(X86_FEATURE_NOPL & 31))

when either CONFIG_X86_P6_NOP or CONFIG_X86_64 are selected.

and cpu_has() does

#define cpu_has(c, bit)                                                 \
       (__builtin_constant_p(bit) &&                                   \
	...
          (((bit)>>5)==3 && (1UL<<((bit)&31) & REQUIRED_MASK3)) ||     \
	...
	? 1 : test_cpu_cap(c, bit))

which returns 1 when testing for X86_FEATURE_NOPL, no?

Signature · 2 lines
Regards/Gruss,
   Boris.
Signature · 1 line
Sent from my mobile phone.  Please pardon any lack of formatting.
author maintainer
#6 lore raw

PATCH 1/1 x86, cpu: Fix X86_FEATURE_NOPL

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit. For that, set it explicitly for non-constant arguments of cpu_has(). Update comment for future reference.

---
 arch/x86/kernel/cpu/common.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/common.c +10 -5
index caa967d..4b68bda 100644
@@ -704,16 +704,21 @@ void __init early_cpu_init(void)
 }
 
 /*
- * The NOPL instruction is supposed to exist on all CPUs with
- * family >= 6; unfortunately, that's not true in practice because
- * of early VIA chips and (more importantly) broken virtualizers that
- * are not easy to detect.  In the latter case it doesn't even *fail*
- * reliably, so probing for it doesn't even work.  Disable it completely
+ * The NOPL instruction is supposed to exist on all CPUs of family >= 6;
+ * unfortunately, that's not true in practice because of early VIA
+ * chips and (more importantly) broken virtualizers that are not easy
+ * to detect. In the latter case it doesn't even *fail* reliably, so
+ * probing for it doesn't even work. Disable it completely on 32-bit
  * unless we can find a reliable way to detect all the broken cases.
+ * Enable it explicitly on 64-bit for non-constant inputs of cpu_has().
  */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#else
+	set_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)
Signature · 5 lines
1.7.2.3

-- 
Regards/Gruss,
    Boris.
author maintainer
#7 lore raw

[tip:x86/cpu] x86, cpu: Fix X86_FEATURE_NOPL

Commit-ID:  366d4a43b1a7a861c356a0e407c4f03f454d42ea
Gitweb:     http://git.kernel.org/tip/366d4a43b1a7a861c356a0e407c4f03f454d42ea
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 4 Oct 2010 09:31:27 +0200
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Mon, 4 Oct 2010 11:22:24 -0700

x86, cpu: Fix X86_FEATURE_NOPL

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit. For that, set it explicitly for non-constant arguments of cpu_has(). Update comment for future reference.

LKML-Reference: <[email protected]>

Cc Ryan O'Neill <[email protected]>
Cc Linus Torvalds <[email protected]>
---
 arch/x86/kernel/cpu/common.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/common.c +10 -5
index f2f9ac7..927e630 100644
@@ -704,16 +704,21 @@ void __init early_cpu_init(void)
 }
 
 /*
- * The NOPL instruction is supposed to exist on all CPUs with
- * family >= 6; unfortunately, that's not true in practice because
- * of early VIA chips and (more importantly) broken virtualizers that
- * are not easy to detect.  In the latter case it doesn't even *fail*
- * reliably, so probing for it doesn't even work.  Disable it completely
+ * The NOPL instruction is supposed to exist on all CPUs of family >= 6;
+ * unfortunately, that's not true in practice because of early VIA
+ * chips and (more importantly) broken virtualizers that are not easy
+ * to detect. In the latter case it doesn't even *fail* reliably, so
+ * probing for it doesn't even work. Disable it completely on 32-bit
  * unless we can find a reliable way to detect all the broken cases.
+ * Enable it explicitly on 64-bit for non-constant inputs of cpu_has().
  */
 static void __cpuinit detect_nopl(struct cpuinfo_x86 *c)
 {
+#ifdef CONFIG_X86_32
 	clear_cpu_cap(c, X86_FEATURE_NOPL);
+#else
+	set_cpu_cap(c, X86_FEATURE_NOPL);
+#endif
 }
 
 static void __cpuinit generic_identify(struct cpuinfo_x86 *c)
author maintainer
#8 lore raw

Re: [tip:x86/cpu] x86, cpu: Fix X86_FEATURE_NOPL

From: tip-bot for Borislav Petkov <[email protected]>
Date: Mon, Oct 04, 2010 at 08:36:15PM +0000

Quoted 15 lines
Commit-ID:  366d4a43b1a7a861c356a0e407c4f03f454d42ea
Gitweb:     http://git.kernel.org/tip/366d4a43b1a7a861c356a0e407c4f03f454d42ea
Author:     Borislav Petkov <[email protected]>
AuthorDate: Mon, 4 Oct 2010 09:31:27 +0200
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Mon, 4 Oct 2010 11:22:24 -0700

x86, cpu: Fix X86_FEATURE_NOPL

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit. For that, set it explicitly for non-constant arguments of cpu_has(). Update comment for future reference.

Quoted 5 lines

LKML-Reference: <[email protected]>

Cc Ryan O'Neill <[email protected]>
Cc Linus Torvalds <[email protected]>

tag it for -stable too?

Signature · 2 lines
Regards/Gruss,
Boris.
#9 lore raw

Re: [tip:x86/cpu] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 3 lines · Borislav Petkov

tag it for -stable too?

What is the flaw that justifies it for stable, or even .36? It seems relatively harmless, with at most a very minor performance issue, unless I'm missing something?

-hpa

author maintainer
#10 lore raw

Re: [tip:x86/cpu] x86, cpu: Fix X86_FEATURE_NOPL

From: "H. Peter Anvin" <[email protected]>
Date: Tue, Oct 05, 2010 at 09:30:37AM -0700

Quoted 8 lines · Borislav Petkov
Quoted 3 lines

tag it for -stable too?

What is the flaw that justifies it for stable, or even .36? It seems relatively harmless, with at most a very minor performance issue, unless I'm missing something?

I was thinking more along the lines of this being partially broken when supplying non-constant arguments to cpu_has(), something like <arch/x86/kernel/cpu/proc.c:show_cpuinfo()>, for example. But all this could cause is /proc/cpuinfo not to report the "nopl" feature bit. I guess this is too minor an issue to even to be considered for stable.

Signature · 2 lines
Regards/Gruss,
Boris.
#11 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

In reply to Borislav Petkov
Quoted 6 lines · Borislav Petkov

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit. For that, set it explicitly for non-constant arguments of cpu_has(). Update comment for future reference.

I don't think this is right.

The cpu_has() logic depends not on x86-64, but on X86_P6_NOP.

Which has

        depends on X86_64
        depends on (MCORE2 || MPENTIUM4 || MPSC)

as its config rules, not just X86_64.

So I think your patch is bogus. It makes the current situation even _more_ confusing than it is.

So what is it? Should we get rid of that odd X86_P6_NOP thing entirely? Or should we use it consistently for the "this machine has NOPL"? Should we always use P6_NOP for x86-64 and just remove the "MCORE2 || MPENTIUM4 || MPSC" thing?

Whatever we do, I don't think this patch is the right one.

Linus

#12 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Borislav Petkov

ba0593bf553c450a03dbc5f8c1f0ff58b778a0c8 cleared the aforementioned cpuid bit only on 32-bit due to various problems with Virtual PC. This somehow got lost during the 32- + 64-bit merge so restore the feature bit on 64-bit. For that, set it explicitly for non-constant arguments of cpu_has(). Update comment for future reference.

Quoted 4 lines

I don't think this is right.

The cpu_has() logic depends not on x86-64, but on X86_P6_NOP.

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Obviously, if we *use* P6 NOPs they better be available on the processor, but we also are pretty sure that every 64-bit processor supports then

Quoted 6 lines

Which has

        depends on X86_64
        depends on (MCORE2 || MPENTIUM4 || MPSC)

as its config rules, not just X86_64.

Right; the top clause, of course, was added later, as we found out that it was unsafe to ever use NOPL on 32 bits, because of Microsoft f*ckups.

CONFIG_X86_P6_NOP was intended to indicate that using NOPL is *preferred*, whereas the CPUID bit -- cpu_has() -- was (and is) intended to indicate that NOPL is *supported*, not necessarily preferred.

As such, the code I believe is technically correct for the current situation (NOPL is always supported on 64 bits, never on 32 bits), but as you quite correctly point out it is definitely more confusing than is desirable; this is probably also reflected by the following code in alternative.c:

static const unsigned char *const *__init_or_module find_nop_table(void)

{
        if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL &&
            boot_cpu_has(X86_FEATURE_NOPL))
                return p6_nops;
        else
                return k8_nops;
}

The vendor check here is really ugly.

The only case where we need CONFIG_X86_P6_NOP as a compile-time check is for the ASM_NOP* macros; for dynamic code we're of course better off with a runtime check, but it would be better if that was part of the CPU routines. Perhaps X86_FEATURE_FAST_NOPL or something like that.

-hpa

#13 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 3 lines · H. Peter Anvin

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Ahh. Right you are. The place that depends on just P6_NOP is the default NOP choice logic in <asm/nops.h>

But the end result ends up being the same: can we please clean this all up so that it isn't so confusing? Rather than add to the confusion?

Linus

#14 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 5 lines · Linus Torvalds
Quoted 3 lines · H. Peter Anvin

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Quoted 7 lines

Ahh. Right you are. The place that depends on just P6_NOP is the default NOP choice logic in <asm/nops.h>

But the end result ends up being the same: can we please clean this all up so that it isn't so confusing? Rather than add to the confusion?

Agreed that this should be cleaned up. However, in the meantime I'd like to keep Borislav's patch in the tree since it makes the code technically correct at least.

OK?

-hpa

author maintainer
#15 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

From: "H. Peter Anvin" <[email protected]>
Date: Mon, Oct 04, 2010 at 02:21:33PM -0700

Quoted 9 lines · Linus Torvalds
Quoted 8 lines · H. Peter Anvin
Quoted 3 lines

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Ahh. Right you are. The place that depends on just P6_NOP is the default NOP choice logic in <asm/nops.h>

Quoted 8 lines
Quoted 4 lines

But the end result ends up being the same: can we please clean this all up so that it isn't so confusing? Rather than add to the confusion?

Agreed that this should be cleaned up. However, in the meantime I'd like to keep Borislav's patch in the tree since it makes the code technically correct at least.

I think the cleanup should be easy: on 64-bit unconditionally return p6_nops in find_nop_table() since every 64-bit processor should support them and on 32-bit never return p6_nops since X86_FEATURE_NOPL is not set there and fall back to intel_nops on non-AMD. Which means we can get rid of X86_FEATURE_NOPL altogether. Too radical?

Hmmm...

Signature · 2 lines
Regards/Gruss,
    Boris.
#16 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 11 lines
Quoted 4 lines

Agreed that this should be cleaned up. However, in the meantime I'd like to keep Borislav's patch in the tree since it makes the code technically correct at least.

I think the cleanup should be easy: on 64-bit unconditionally return p6_nops in find_nop_table() since every 64-bit processor should support them and on 32-bit never return p6_nops since X86_FEATURE_NOPL is not set there and fall back to intel_nops on non-AMD. Which means we can get rid of X86_FEATURE_NOPL altogether. Too radical?

This is fine with me, but you may want to check with your employer if the Intel NOPs will perform slower than the 66 ... 90 NOPs on AMD processors.

-hpa

#17 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

In reply to Borislav Petkov
Quoted 7 lines · Borislav Petkov

I think the cleanup should be easy: on 64-bit unconditionally return p6_nops in find_nop_table() since every 64-bit processor should support them and on 32-bit never return p6_nops since X86_FEATURE_NOPL is not set there and fall back to intel_nops on non-AMD. Which means we can get rid of X86_FEATURE_NOPL altogether. Too radical?

FWIW, if P6_NOPs work as well as 66...90 on AMD, I'd much rather switch to the fixed sequence for all 64-bit processors.

-hpa

author maintainer
#18 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

From: "H. Peter Anvin" <[email protected]>
Date: Mon, Oct 04, 2010 at 02:53:15PM -0700

Quoted 10 lines · Borislav Petkov
Quoted 7 lines

I think the cleanup should be easy: on 64-bit unconditionally return p6_nops in find_nop_table() since every 64-bit processor should support them and on 32-bit never return p6_nops since X86_FEATURE_NOPL is not set there and fall back to intel_nops on non-AMD. Which means we can get rid of X86_FEATURE_NOPL altogether. Too radical?

FWIW, if P6_NOPs work as well as 66...90 on AMD,

Yeah, let me get back to you on that :).

Quoted 2 lines

I'd much rather switch to the fixed sequence for all 64-bit
processors.

Signature · 2 lines
Regards/Gruss,
    Boris.
#19 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

In reply to H. Peter Anvin
Quoted 9 lines · H. Peter Anvin
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · H. Peter Anvin

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Ahh. Right you are. The place that depends on just P6_NOP is the default NOP choice logic in <asm/nops.h>

Quoted 8 lines
Quoted 4 lines

But the end result ends up being the same: can we please clean this all up so that it isn't so confusing? Rather than add to the confusion?

Agreed that this should be cleaned up.  However, in the meantime I'd like to keep Borislav's patch in the tree since it makes the code technically correct at least.

Another piece of the confusion I noticed a couple of days ago:
X86_MINIMUM_CPU_FAMILY defaults to "6" if X86_32 && X86_P6_NOP; whereas X86_P6_NOP depends on X86_64.

Hugh

#20 lore raw

Re: [PATCH] x86, cpu: Fix X86_FEATURE_NOPL

Quoted 15 lines · Hugh Dickins
Quoted 14 lines · H. Peter Anvin
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · H. Peter Anvin

Actually, cpu_has() depends on:
#if defined(CONFIG_X86_P6_NOP) || defined(CONFIG_X86_64)

Ahh. Right you are. The place that depends on just P6_NOP is the default NOP choice logic in <asm/nops.h>

But the end result ends up being the same: can we please clean this all up so that it isn't so confusing? Rather than add to the confusion?

Quoted 8 lines
Quoted 3 lines

Agreed that this should be cleaned up. However, in the meantime I'd like to keep Borislav's patch in the tree since it makes the code technically correct at least.

Another piece of the confusion I noticed a couple of days ago:
X86_MINIMUM_CPU_FAMILY defaults to "6" if X86_32 && X86_P6_NOP; whereas X86_P6_NOP depends on X86_64.

Again, it's completely consistent -- if you keep in mind that CONFIG_X86_P6_NOP depending on X86_64 is a policy decision; that policy can theoretically be changed. However, it really doesn't seem worth it to ever contemplate at this point.

-hpa