Skip to contentSkip to conversation

Pentium Pro - sysenter - doublefault

PATCH In review Jim Houston 9 participants 19 messages Aug 21, 2003 – Sep 8, 2003

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Pentium Pro - sysenter - doublefault

Hi Everyone,

I upgraded my Pentium Pro system to Redhat 9, installed a linux-2.6.0-test3 kernel, and it fails with a double-fault when init starts.

The code which decides if it is o.k. to use sysenter is broken for some Pentium Pro cpus ,in particular, this bit of code from arch/i386/kernel/cpu/intel.c:

	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
		clear_bit(X86_FEATURE_SEP, c->x86_capability);

On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP. This results in a double-fault when init starts. The double-fault happens on the sysexit. The new double-fault handler caught this nicely, and I was able to debug this with kgdb.

The logic above is exactly what Intel says to do in "IA-32 Intel® Architecture Software Developer's Manual, Volume 2: Instruction Set Reference" on page 3-767. It also says that sysenter was added to the Pentium II.

I checked the Pentium Pro and Pentium II Specifications Update manuals hoping to find the details to justify the "mask < 3" portion of the test above. They both describe sysenter related errata but none which was fixed in mask 3.

The attached patch avoids using sysenter on all Pentium Pro systems.

Jim Houston - Concurrent Computer Corp.

diff -urN linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c
--- linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c 2003-08-20
10:30:14.000000000 -0400
+++ linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c 2003-08-21
14:39:35.000000000 -0400
@@ -246,7 +246,7 @@
}

/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
- if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
+ if ( c->x86 == 6 && c->x86_model < 3)
clear_bit(X86_FEATURE_SEP, c->x86_capability);

/* Names for the Pentium II/Celeron processors

#1 lore raw
Quoted 19 lines · Jim Houston

Hi Everyone,

I upgraded my Pentium Pro system to Redhat 9, installed a linux-2.6.0-test3 kernel, and it fails with a double-fault when init starts.

The code which decides if it is o.k. to use sysenter is broken for some Pentium Pro cpus ,in particular, this bit of code from arch/i386/kernel/cpu/intel.c:

	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
		clear_bit(X86_FEATURE_SEP, c->x86_capability);

On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP. This results in a double-fault when init starts. The double-fault happens on the sysexit. The new double-fault handler caught this nicely, and I was able to debug this with kgdb.

Quoted 4 lines

The logic above is exactly what Intel says to do in "IA-32 Intel® Architecture Software Developer's Manual, Volume 2: Instruction Set Reference" on page 3-767. It also says that sysenter was added to the Pentium II.

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID), and it says (section 3.4) that SEP is unsupported when the signature as a whole is less that 0x633. This means all PPros, and PII Model 3s with steppings less than 3.

The problem is that the kernel check you quoted above is buggy: the c->x86_model < 3 && c->x86_mask < 3 part fails for late-stepping PPros since c->x86_mask >= 3 for them. The test should be rewritten as:

        if (c->x86 == 6 && (c->x86_model < 3 ||
                            (c->x86_model == 3 && c->x86_mask < 3)))
                clear_bit(X86_FEATURE_SEP, c->x86_capability);

/Mikael

#2 lore raw
Quoted 10 lines · Mikael Pettersson
Quoted 4 lines

The logic above is exactly what Intel says to do in "IA-32 Intel® Architecture Software Developer's Manual, Volume 2: Instruction Set Reference" on page 3-767. It also says that sysenter was added to the Pentium II.

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID), and it says (section 3.4) that SEP is unsupported when the signature as a whole is less that 0x633. This means all PPros, and PII Model 3s with steppings less than 3.

Hi Dave, Everyone,

This make sense. Here is Mikael's suggested code as a patch.

Dave, I picked your name from the maintainers list. Please feed this patch up the chain.

Jim Houston - Concurrent Computer Corp.

diff -urN linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c
--- linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c 2003-08-20
10:30:14.000000000 -0400
+++ linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c 2003-08-21
21:34:40.000000000 -0400
@@ -246,7 +246,8 @@
}

/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
- if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
+ if ( c->x86 == 6 && ((c->x86_model < 3) ||
+ (c->x86_model == 3 && c->x86_mask < 3)))
clear_bit(X86_FEATURE_SEP, c->x86_capability);

/* Names for the Pentium II/Celeron processors

#3 lore raw
In reply to Mikael Pettersson
Quoted 9 lines · Mikael Pettersson
Quoted 4 lines

The logic above is exactly what Intel says to do in "IA-32 Intel® Architecture Software Developer's Manual, Volume 2: Instruction Set Reference" on page 3-767. It also says that sysenter was added to the Pentium II.

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID), and it says (section 3.4) that SEP is unsupported when the signature as a whole is less that 0x633. This means all PPros, and PII Model 3s with steppings less than 3.

So (double-checking) the pseudo-code in "IA-32: Intel Architecture Software Development Manual, Volume 2: Instruction Set Reference" is buggy?

Oh my! Perhaps there are other bugs in that behemoth of a manual, too :/

-- Jamie

#4 lore raw
In reply to Mikael Pettersson
Quoted 4 lines · Mikael Pettersson

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID), and it says (section 3.4) that SEP is unsupported when the signature as a whole is less that 0x633. This means all PPros, and PII Model 3s with steppings less than 3.

"SEP is unsupported". It's interesting that Pentium Pro erratum #82 is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment selector' to SS and CS registers", implying that SYSENTER does _something_ useful on PPros.

-- Jamie

#5 lore raw

Hi!

Quoted 9 lines
Quoted 4 lines

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID), and it says (section 3.4) that SEP is unsupported when the signature as a whole is less that 0x633. This means all PPros, and PII Model 3s with steppings less than 3.

"SEP is unsupported". It's interesting that Pentium Pro erratum #82 is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment selector' to SS and CS registers", implying that SYSENTER does _something_ useful on PPros.

Well, with CS==0 machine is not going to survive too long. If it only happens sometimes you might catch the double fault and fixup, but....
Pavel

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#6 lore raw
Quoted 8 lines · Pavel Machek
Quoted 4 lines

"SEP is unsupported". It's interesting that Pentium Pro erratum #82 is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment selector' to SS and CS registers", implying that SYSENTER does _something_ useful on PPros.

Well, with CS==0 machine is not going to survive too long. If it only happens sometimes you might catch the double fault and fixup, but....

The erratum only applies when you load CS==0 _deliberately_, by setting the MSR to that.

I'm wondering what happens when you don't do silly things - what is the undocumented behaviour of SYSENTER/SYSEXIT on those chips?

I vaguely recall reading details about the behaviour change made by Intel, around the time it was done, but I can't see to find it anywhere.

-- Jamie

#7 lore raw
In reply to Jim Houston
Quoted 4 lines · Jim Houston

On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP. This results in a double-fault when init starts. The double-fault happens on the sysexit. The new double-fault handler caught this nicely, and I was able to debug this with kgdb.

Does anyone know what the syenter & sysexit instructions do on these early PPro CPUs?

The Intel documentation is vague, saying only to avoid using them. I'd like to know what happens if userspace does "sysenter" on one of these systems. Does it issue Invalid Opcode, General Protection fault, or something else?

Jim you can answer this as you have such a Ppro. Could you please run this very simple userspace program for me, and report the result?

	int main() { __asm__ ("sysenter"); return 0; }

I expect it to die with SIGILL on Pentium and earlier chips, and SIGSEGV on "good" PPro and later chips running kernels which don't enable the sysenter instruction.

But what does it do on your early Intel PPro, the one which is the subject of this thread?

Thanks,
-- Jamie

#8 lore raw

On Mon, Aug 25, 2003 at 05:05:14AM +0100, Jamie Lokier wrote:
...

Quoted 4 lines

Jim you can answer this as you have such a Ppro. Could you please run this very simple userspace program for me, and report the result?

	int main() { __asm__ ("sysenter"); return 0; }

I tested on two boxes:

Stepping 1 ppro:  SIGSEGV
Stepping 7 ppro:  SIGSEGV

If you need additional info, please just ask.

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
#9 lore raw
Quoted 9 lines · Jakob Oestergaard
Quoted 4 lines

Jim you can answer this as you have such a Ppro. Could you please run this very simple userspace program for me, and report the result?

	int main() { __asm__ ("sysenter"); return 0; }

I tested on two boxes:

Stepping 1 ppro:  SIGSEGV
Stepping 7 ppro:  SIGSEGV

Thank you!

So that means the sysenter instruction _does_ exist on the PPro and early Pentium II, but it isn't usable.

It's safe so long as it is disabled, but as you found, when you enable and use it, it doesn't work as expected. (I wonder what the actual behaviour of sysenter/sysexit are on these CPUs.)

-- Jamie

#10 lore raw
Quoted 2 lines · Jamie Lokier

So that means the sysenter instruction _does_ exist on the PPro and early Pentium II, but it isn't usable.

If anyone has information on what the SYSENTER and SYSEXIT instructions actually do on Intel Pentium Pro or stepping<3 Pentium II processors, I am very interested.

I'm intrigued to know if the buggy behaviour of these instructions is really unsafe, or simply hard to use so Intel changed the behaviour. (An example of hard to use would be SYSENTER not disabling interrupts). If they are safe but hard to use, perhaps the ingenuity of kernel hackers can work around the hardness >:)

Does anyone have a contact at Intel for this question?

Thanks,
-- Jamie

#11 lore raw
Quoted 7 lines · Jamie Lokier
Quoted 2 lines · Jamie Lokier

So that means the sysenter instruction _does_ exist on the PPro and early Pentium II, but it isn't usable.

If anyone has information on what the SYSENTER and SYSEXIT instructions actually do on Intel Pentium Pro or stepping<3 Pentium II processors, I am very interested.

I dug up a little more from the archeological site (machine room - sigh)...

-------------------------------------------
model name : Pentium II (Deschutes)
stepping : 2

$ ./syse 
Segmentation fault
-------------------------------------------
model name      : Celeron (Mendocino)
stepping        : 0
cpu MHz         : 334.097
$ ./syse
Segmentation fault
-------------------------------------------
CPU: Pentium II/Pentium II Xeon/Celeron (299.94-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x651  Stepping = 1
$ ./syse
Bus error (core dumped)
-------------------------------------------

The last one is a FreeBSD box.

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
#12 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier
Quoted 2 lines · Jamie Lokier

So that means the sysenter instruction _does_ exist on the PPro and early Pentium II, but it isn't usable.

If anyone has information on what the SYSENTER and SYSEXIT instructions actually do on Intel Pentium Pro or stepping<3 Pentium II processors, I am very interested.

Quoted 5 lines

I'm intrigued to know if the buggy behaviour of these instructions is really unsafe, or simply hard to use so Intel changed the behaviour. (An example of hard to use would be SYSENTER not disabling interrupts). If they are safe but hard to use, perhaps the ingenuity of kernel hackers can work around the hardness >:)

Hi Jamie,

I tried your test on my machine. It fails with a segmentation fault. I noticed that the Pentium II specifications update manual starts with rev C0 stepping (ignoring mask rev < 3).
I'm inclined to forgive Intel for not publishing the scary errata that goes with the first few mask revs, particularly for an old product.

When I was chasing the original problem, I added tracing code (compiling the kernel with finstrument-functions) so that when I got into kgdb after the double-fault I could see that it had just completed a umask system call. I'm assuming that it failed on the sysexit.

I keep the old Pentium Pro around because it has an NMI interrupt button.

I'm happy that Linus has merged the fix to disable correctly disable sysenter for these machines.

Jim Houston - Concurrent Computer Corp

#13 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier

Could you please run this very simple userspace program for me, and report the result?

	int main() { __asm__ ("sysenter"); return 0; }

I expect it to die with SIGILL on Pentium and earlier chips, and SIGSEGV on "good" PPro and later chips running kernels which don't enable the sysenter instruction.

OK, since I get something different to the other reports I saw:

1:20PM-malvern-0-534-% ./sysenter
1:20PM-malvern-STKFLT-535-% echo $?
144

/proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 5
model name : Pentium II (Deschutes)
stepping : 0
cpu MHz : 333.495
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr
bogomips : 657.40

uname -a:
Linux malvern 2.6.0-test2-mm5 #2 Fri Aug 8 12:06:50 BST 2003 i686 unknown

HTH

Signature · 3 lines
Richard \\\ SuperH Core+Debug Architect /// .. At home ..
  P.    /// [email protected]  ///  [email protected]
Curnow  \\\ http://www.superh.com/    ///  www.rc0.org.uk
#14 lore raw
Quoted 5 lines · Richard Curnow

OK, since I get something different to the other reports I saw:

1:20PM-malvern-0-534-% ./sysenter
1:20PM-malvern-STKFLT-535-% echo $?
144

Hi Richard,

That's because you ran it on a 2.5/2.6 kernel, right? The test code is meant for 2.4 kernels and earlier :)

Here is a more universal test:

	int main () {
		asm ("movl %%esp,%%ebp;sysenter" : : "a" (1), "b" (0));
		return 0;
	}

I expect it to do the first of these which is applicable:

- raise SIGILL on Pentium and earlier Intel CPUs
- raise SIGILL on non-Intel CPUs which don't have the SEP capability
- raise SIGSEGV on Pentium Pro CPUs
- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping < 3
- raise SIGSEGV on 2.4 kernels
- exit with status 0 on 2.6 kernels

Enjoy,
-- Jamie

#15 lore raw
Quoted 2 lines · Jamie Lokier

Here is a more universal test:

[snip]

Quoted 1 line

- exit with status 0 on 2.6 kernels

Yes, confirmed, that's what it did on 2.6.0-test2-mm5 here. Sorry, I can't try 2.4 right now.

Signature · 3 lines
Richard \\\ SuperH Core+Debug Architect /// .. At home ..
  P.    /// [email protected]  ///  [email protected]
Curnow  \\\ http://www.superh.com/    ///  www.rc0.org.uk
#16 lore raw
In reply to Jamie Lokier

did you notice the announcement from www.apache.org and www.debian.org?

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jamie Lokier
Sent: Wednesday, August 27, 2003 10:01 PM
To: Jim Houston; [email protected]; [email protected]
Subject: Re: [PATCH] Pentium Pro - sysenter - doublefault

Quoted 4 lines · Richard Curnow

OK, since I get something different to the other reports I saw:

1:20PM-malvern-0-534-% ./sysenter 1:20PM-malvern-STKFLT-535-% echo
$? 144

Hi Richard,

That's because you ran it on a 2.5/2.6 kernel, right? The test code is meant for 2.4 kernels and earlier :)

Here is a more universal test:

	int main () {
		asm ("movl %%esp,%%ebp;sysenter" : : "a" (1), "b" (0));
		return 0;
	}

I expect it to do the first of these which is applicable:

- raise SIGILL on Pentium and earlier Intel CPUs
- raise SIGILL on non-Intel CPUs which don't have the SEP
capability
- raise SIGSEGV on Pentium Pro CPUs
- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping
< 3
- raise SIGSEGV on 2.4 kernels
- exit with status 0 on 2.6 kernels

Enjoy,
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#17 lore raw
In reply to Jamie Lokier

Jamie Lokier wrote:

<SNIP>

Quoted 11 lines

I expect it to do the first of these which is applicable:

- raise SIGILL on Pentium and earlier Intel CPUs
- raise SIGILL on non-Intel CPUs which don't have the SEP capability
- raise SIGSEGV on Pentium Pro CPUs
- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping < 3
- raise SIGSEGV on 2.4 kernels
- exit with status 0 on 2.6 kernels

Enjoy,
-- Jamie

As expected I get a SIGILL on P166 with 2.4.22

-sb
-----------------------------------------------
The price of freedom? Ask your Senator how much
the RIAA gave him for his Lexus.

#18 lore raw
In reply to Jamie Lokier

* Jamie Lokier <[email protected]>:

Quoted 11 lines
Quoted 5 lines · Richard Curnow

OK, since I get something different to the other reports I saw:

1:20PM-malvern-0-534-% ./sysenter
1:20PM-malvern-STKFLT-535-% echo $?
144

Hi Richard,

That's because you ran it on a 2.5/2.6 kernel, right? The test code is meant for 2.4 kernels and earlier :)

If this is of any help..

- -

pvsavola@a11a:~/code$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 1
model name      : Pentium Pro
stepping        : 6
cpu MHz         : 199.312
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov
bogomips        : 397.31
pvsavola@a11a:~/code$ ./sysent ; echo $?
Segmentation fault
139
pvsavola@a11a:~/code$ uname -a
Linux a11a 2.4.19-ck3-rmap #1 Mon Aug 26 21:38:49 EEST 2002 i686 GNU/Linux
- -
Signature · 1 line
   Psi -- <http://www.iki.fi/pasi.savolainen>