Skip to contentSkip to conversation

x86-64: Disable local APIC timer use on AMD systems with C1E

PATCH In review Thomas Gleixner 3 participants 6 messages Sep 25, 2007 – Sep 26, 2007

lore
Share & tools
lore

Thomas Gleixner

author
#0 lore raw

PATCH 1/1 x86-64: Disable local APIC timer use on AMD systems with C1E

commit 3556ddfa9284a86a59a9b78fe5894430f6ab4eef titled

[PATCH] x86-64: Disable local APIC timer use on AMD systems with C1E

solves a problem with AMD dual core laptops e.g. HP nx6325 (Turion 64 X2) with C1E enabled:

When both cores go into idle at the same time, then the system switches into C1E state, which is basically the same as C3. This stops the local apic timer.

This was debugged right after the dyntick merge on i386 and despite the patch title it fixes only the 32 bit path.

x86_64 is still missing this fix. It seems that mainline is not really affected by this issue, as the PIT is running and keeps jiffies incrementing, but that's just waiting for trouble.

-mm suffers from this problem due to the x86_64 high resolution timer patches.

This is a quick and dirty port of the i386 code to x86_64.

I spent quite a time with Rafael to debug the -mm / hrt wreckage until someone pointed us to this. I really had forgotten that we debugged this half a year ago already.

Sigh, is it just me or is there something yelling arch/x86 into my ear?

Signed-off-by Thomas Gleixner <[email protected]>
arch/x86_64/kernel/setup.c +34 -0
index af838f6..32054bf 100644
@@ -546,6 +546,37 @@ static void __init amd_detect_cmp(struct cpuinfo_x86 *c)
 #endif
 }
 
+#define ENABLE_C1E_MASK		0x18000000
+#define CPUID_PROCESSOR_SIGNATURE	1
+#define CPUID_XFAM		0x0ff00000
+#define CPUID_XFAM_K8		0x00000000
+#define CPUID_XFAM_10H		0x00100000
+#define CPUID_XFAM_11H		0x00200000
+#define CPUID_XMOD		0x000f0000
+#define CPUID_XMOD_REV_F	0x00040000
+
+/* AMD systems with C1E don't have a working lAPIC timer. Check for that. */
+static __cpuinit int amd_apic_timer_broken(void)
+{
+	u32 lo, hi;
+	u32 eax = cpuid_eax(CPUID_PROCESSOR_SIGNATURE);
+	switch (eax & CPUID_XFAM) {
+	case CPUID_XFAM_K8:
+		if ((eax & CPUID_XMOD) < CPUID_XMOD_REV_F)
+			break;
+	case CPUID_XFAM_10H:
+	case CPUID_XFAM_11H:
+		rdmsr(MSR_K8_ENABLE_C1E, lo, hi);
+		if (lo & ENABLE_C1E_MASK)
+			return 1;
+		break;
+	default:
+		/* err on the side of caution */
+		return 1;
+	}
+	return 0;
+}
+
 static void __cpuinit init_amd(struct cpuinfo_x86 *c)
 {
 	unsigned level;

@@ -617,6 +648,9 @@ static void __cpuinit init_amd(struct cpuinfo_x86 *c)
 	/* Family 10 doesn't support C states in MWAIT so don't use it */
 	if (c->x86 == 0x10 && !force_mwait)
 		clear_bit(X86_FEATURE_MWAIT, &c->x86_capability);
+
+	if (amd_apic_timer_broken())
+		disable_apic_timer = 1;
 }
 
 static void __cpuinit detect_ht(struct cpuinfo_x86 *c)
include/asm-x86_64/apic.h +1 -0
index 85125ef..e458020 100644
@@ -20,6 +20,7 @@ extern int apic_verbosity;
 extern int apic_runs_main_timer;
 extern int ioapic_force;
 extern int apic_mapped;
+extern int disable_apic_timer;
 
 /*
  * Define the default level of output to be very little
#1 lore raw
Quoted 23 lines · Thomas Gleixner

commit 3556ddfa9284a86a59a9b78fe5894430f6ab4eef titled

[PATCH] x86-64: Disable local APIC timer use on AMD systems with C1E

solves a problem with AMD dual core laptops e.g. HP nx6325 (Turion 64 X2) with C1E enabled:

When both cores go into idle at the same time, then the system switches into C1E state, which is basically the same as C3. This stops the local apic timer.

This was debugged right after the dyntick merge on i386 and despite the patch title it fixes only the 32 bit path.

x86_64 is still missing this fix. It seems that mainline is not really affected by this issue, as the PIT is running and keeps jiffies incrementing, but that's just waiting for trouble.

-mm suffers from this problem due to the x86_64 high resolution timer patches.

This is a quick and dirty port of the i386 code to x86_64.

Quoted 7 lines

I spent quite a time with Rafael to debug the -mm / hrt wreckage until someone pointed us to this. I really had forgotten that we debugged this half a year ago already.

Sigh, is it just me or is there something yelling arch/x86 into my ear?

Signed-off-by Thomas Gleixner <[email protected]>

I have reworked the patch a bit so that it applies on top of 2.6.23-rc8-mm1 and compiles (my version is attached).

With this patch applied, the kernel boots correctly on the nx6325.

Greetings,
Rafael

---
 arch/x86_64/kernel/apic.c  |    2 +-
 arch/x86_64/kernel/setup.c |   34 ++++++++++++++++++++++++++++++++++
 include/asm-x86_64/apic.h  |    1 +
 3 files changed, 36 insertions(+), 1 deletion(-)
arch/x86_64/kernel/setup.c +34 -0
Index: linux-2.6.23-rc8-mm1/arch/x86_64/kernel/setup.c
===================================================================
@@ -620,6 +620,37 @@ static void __cpuinit early_init_amd(str
 #endif
 }
 
+#define ENABLE_C1E_MASK		0x18000000
+#define CPUID_PROCESSOR_SIGNATURE	1
+#define CPUID_XFAM		0x0ff00000
+#define CPUID_XFAM_K8		0x00000000
+#define CPUID_XFAM_10H		0x00100000
+#define CPUID_XFAM_11H		0x00200000
+#define CPUID_XMOD		0x000f0000
+#define CPUID_XMOD_REV_F	0x00040000
+
+/* AMD systems with C1E don't have a working lAPIC timer. Check for that. */
+static __cpuinit int amd_apic_timer_broken(void)
+{
+	u32 lo, hi;
+	u32 eax = cpuid_eax(CPUID_PROCESSOR_SIGNATURE);
+	switch (eax & CPUID_XFAM) {
+	case CPUID_XFAM_K8:
+		if ((eax & CPUID_XMOD) < CPUID_XMOD_REV_F)
+			break;
+	case CPUID_XFAM_10H:
+	case CPUID_XFAM_11H:
+		rdmsr(MSR_K8_ENABLE_C1E, lo, hi);
+		if (lo & ENABLE_C1E_MASK)
+			return 1;
+		break;
+	default:
+		/* err on the side of caution */
+		return 1;
+	}
+	return 0;
+}
+
 static void __cpuinit init_amd(struct cpuinfo_x86 *c)
 {
 	unsigned level, flags, dummy;

@@ -692,6 +723,9 @@ static void __cpuinit init_amd(struct cp
 	if (c->x86 == 0x10 && !force_mwait)
 		clear_bit(X86_FEATURE_MWAIT, &c->x86_capability);
 
+	if (amd_apic_timer_broken())
+		disable_apic_timer = 1;
+
 	if (c->x86 >= 0xf && c->x86 <= 0x11 &&
 	    !rdmsr_safe(MSR_VM_CR, &flags, &dummy) &&
 	    (flags & 0x18))
include/asm-x86_64/apic.h +1 -0
Index: linux-2.6.23-rc8-mm1/include/asm-x86_64/apic.h
===================================================================
@@ -19,6 +19,7 @@
 extern int apic_verbosity;
 extern int apic_runs_main_timer;
 extern int ioapic_force;
+extern int disable_apic_timer;
 
 /*
  * Define the default level of output to be very little
arch/x86_64/kernel/apic.c +1 -1
Index: linux-2.6.23-rc8-mm1/arch/x86_64/kernel/apic.c
===================================================================
@@ -41,8 +41,8 @@
 #include <asm/apic.h>
 
 int apic_verbosity;
+int disable_apic_timer __initdata;
 static int apic_calibrate_pmtmr __initdata;
-static int disable_apic_timer __initdata;
 
 /* Local APIC timer works in C2? */
 int local_apic_timer_c2_ok;
author
#2 lore raw
Quoted 4 lines · Rafael J. Wysocki

I have reworked the patch a bit so that it applies on top of 2.6.23-rc8-mm1 and compiles (my version is attached).

With this patch applied, the kernel boots correctly on the nx6325.

I know. It's basically enforced "noapictimer".

But this still does not explain why your nasty box booted current mainline with "apicmaintimer" on the kernel command line.

tglx

#3 lore raw
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines · Rafael J. Wysocki

I have reworked the patch a bit so that it applies on top of 2.6.23-rc8-mm1 and compiles (my version is attached).

With this patch applied, the kernel boots correctly on the nx6325.

I know. It's basically enforced "noapictimer".

But this still does not explain why your nasty box booted current mainline with "apicmaintimer" on the kernel command line.

Now it doesn't do that any more.

As I wrote in the other thread, I think that _sometimes_ the CPU doesn't enter the C1E state (or how it's called) and then the box boots and works.

Greetings,
Rafael

#4 lore raw
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Thomas Gleixner
Quoted 4 lines · Rafael J. Wysocki

I have reworked the patch a bit so that it applies on top of 2.6.23-rc8-mm1 and compiles (my version is attached).

With this patch applied, the kernel boots correctly on the nx6325.

I know. It's basically enforced "noapictimer".

Quoted 8 lines
Quoted 2 lines

But this still does not explain why your nasty box booted current mainline with "apicmaintimer" on the kernel command line.

Now it doesn't do that any more.

As I wrote in the other thread, I think that _sometimes_ the CPU doesn't enter the C1E state (or how it's called) and then the box boots and works.

Can you see any correlation between being on AC or on battery and this bug occuring? Sometimes the BIOS might even do C states depending on whether the box was booted up on AC or on battery. (and sometimes the kernel calibrates things differently when booting up on battery [lower speed] - so this can have an effect too.)

Ingo

#5 lore raw
Quoted 15 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines · Rafael J. Wysocki

I have reworked the patch a bit so that it applies on top of 2.6.23-rc8-mm1 and compiles (my version is attached).

With this patch applied, the kernel boots correctly on the nx6325.

I know. It's basically enforced "noapictimer".

But this still does not explain why your nasty box booted current mainline with "apicmaintimer" on the kernel command line.

Quoted 8 lines
Quoted 5 lines

Now it doesn't do that any more.

As I wrote in the other thread, I think that _sometimes_ the CPU doesn't enter the C1E state (or how it's called) and then the box boots and works.

Can you see any correlation between being on AC or on battery and this bug occuring?

No. All tests were done with the AC power supply plugged in.

Quoted 4 lines

Sometimes the BIOS might even do C states depending on whether the box was booted up on AC or on battery. (and sometimes the kernel calibrates things differently when booting up on battery [lower speed] - so this can have an effect too.)

On my box there are hidden history effects related to other things too (eg. sometimes after a fresh boot the processor generates lots of ACPI events such that it takes 25% of CPU power to handle them alone).

Greetings,
Rafael