Performance/resume issues on Toshiba NB305

I've been looking into a couple of problems with this machine that have me a bit stumped at the moment. The two problems may or may not be related, so I'm including details about both issues below. If anyone has any ideas, I'd love to hear them. Note that these are not recent regressions; they've been around since at least 2.6.32.

The CPU in this machine is an Atom N450.

When booted normally the performance on this machine is very poor, but when booted with any of nohz=off, nolapic, or nohpet it improves significantly. The performance also improves if I use the patch below to force the hpet to remain in periodic mode (with hpet=periodic on the command line).

One other thing I noticed when I had added some logging related to hpet rearming is 3-5 second periods of no log activity occurring fairly frequently, whereas such inactive periods are infrequent when performance is good and are also infrequent on another machine with very similar hardware and no performance issues.

The machine also hangs for 5 minutes during resume, unless booted with both nohz=off and highres=off, or with hpet=periodic using the patch below. I've traced this down to hanging in an SMI handler during the ACPI _WAK method execution. The 5 minutes corresponds to how long it takes for the low 32 bits of the hpet to wrap in this machine, and since the options that eliminate the hang result in the hpet being in periodic mode during _WAK method execution I suspect that the SMI handler is hanging until a timer interrupt happens.

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

Note also that I've already looked into the triggering for IRQ0, and it seems to be correct.

Thanks,
Seth

arch/x86/kernel/hpet.c +7 -0

index 4ff5968..ab10012 100644
@@ -87,6 +87,7 @@ static inline void hpet_clear_mapping(void)
 static int boot_hpet_disable;
 int hpet_force_user;
 static int hpet_verbose;
+static int hpet_oneshot_disable;
 
 static int __init hpet_setup(char *str)
 {

@@ -97,6 +98,8 @@ static int __init hpet_setup(char *str)
 			hpet_force_user = 1;
 		if (!strncmp("verbose", str, 7))
 			hpet_verbose = 1;
+		if (!strncmp("periodic", str, 8))
+			hpet_oneshot_disable = 1;
 	}
 	return 1;
 }

@@ -306,6 +309,10 @@ static void hpet_legacy_clockevent_register(void)
 	hpet_clockevent.min_delta_ns = clockevent_delta2ns(HPET_MIN_PROG_DELTA,
 							   &hpet_clockevent);
 
+	/* Disable one-shot mode if requested */
+	if (hpet_oneshot_disable)
+		hpet_clockevent.features &= ~CLOCK_EVT_FEAT_ONESHOT;
+
 	/*
 	 * Start hpet with the boot cpu mask and make it
 	 * global after the IO_APIC has been initialized.

kernel/time/tick-broadcast.c +6 -1

index 48b2761..8e84c11 100644
@@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
 
 	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 
-	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	bc = tick_broadcast_device.evtdev;
+	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
+		goto unlock;
+
+	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	if (bc)
 		tick_broadcast_setup_oneshot(bc);
+
+unlock:
 	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
 }
 

Would be helpful to Cc the people that may help you.

Quoted 29 lines · Seth Forshee

I've been looking into a couple of problems with this machine that have me a bit stumped at the moment. The two problems may or may not be related, so I'm including details about both issues below. If anyone has any ideas, I'd love to hear them. Note that these are not recent regressions; they've been around since at least 2.6.32.

The CPU in this machine is an Atom N450.

When booted normally the performance on this machine is very poor, but when booted with any of nohz=off, nolapic, or nohpet it improves significantly. The performance also improves if I use the patch below to force the hpet to remain in periodic mode (with hpet=periodic on the command line).

One other thing I noticed when I had added some logging related to hpet rearming is 3-5 second periods of no log activity occurring fairly frequently, whereas such inactive periods are infrequent when performance is good and are also infrequent on another machine with very similar hardware and no performance issues.

The machine also hangs for 5 minutes during resume, unless booted with both nohz=off and highres=off, or with hpet=periodic using the patch below. I've traced this down to hanging in an SMI handler during the ACPI _WAK method execution. The 5 minutes corresponds to how long it takes for the low 32 bits of the hpet to wrap in this machine, and since the options that eliminate the hang result in the hpet being in periodic mode during _WAK method execution I suspect that the SMI handler is hanging until a timer interrupt happens.

Quoted 4 lines

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

Sounds like a buggy BIOS to me. Is there upgrades available for this box?

Quoted 7 lines · Steven Rostedt

Quoted 4 lines

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

Sounds like a buggy BIOS to me. Is there upgrades available for this box?

It does sound like a buggy BIOS. I've already flashed the latest BIOS though, so no luck there.

Quoted 6 lines · Seth Forshee

Quoted 5 lines · Steven Rostedt

Quoted 4 lines

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

Quoted 5 lines

Quoted 2 lines

Sounds like a buggy BIOS to me. Is there upgrades available for this box?

It does sound like a buggy BIOS. I've already flashed the latest BIOS though, so no luck there.

Has the BIOS some knob to disable that feature^Wcrap? It's often hidden behind some "$manufacturer super extra mode".

I've been looking into a couple of problems with this machine that have me a bit stumped at the moment. The two problems may or may not be related, so I'm including details about both issues below. If anyone has any ideas, I'd love to hear them. Note that these are not recent regressions; they've been around since at least 2.6.32.

The CPU in this machine is an Atom N450.

When booted normally the performance on this machine is very poor, but when booted with any of nohz=off, nolapic, or nohpet it improves significantly. The performance also improves if I use the patch below to force the hpet to remain in periodic mode (with hpet=periodic on the command line).

One other thing I noticed when I had added some logging related to hpet rearming is 3-5 second periods of no log activity occurring fairly frequently, whereas such inactive periods are infrequent when performance is good and are also infrequent on another machine with very similar hardware and no performance issues.

The machine also hangs for 5 minutes during resume, unless booted with both nohz=off and highres=off, or with hpet=periodic using the patch below. I've traced this down to hanging in an SMI handler during the ACPI _WAK method execution. The 5 minutes corresponds to how long it takes for the low 32 bits of the hpet to wrap in this machine, and since the options that eliminate the hang result in the hpet being in periodic mode during _WAK method execution I suspect that the SMI handler is hanging until a timer interrupt happens.

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

kernel/time/tick-broadcast.c +0 -1

@@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
 
 	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 
-	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	bc = tick_broadcast_device.evtdev;

On Fri, 25 Feb 2011, Seth Forshee wrote:

Please always Cc the relevant maintainers. I just got notified by someone who accidentaly stumbled over that.

Quoted 14 lines

I've been looking into a couple of problems with this machine that have me a bit stumped at the moment. The two problems may or may not be related, so I'm including details about both issues below. If anyone has any ideas, I'd love to hear them. Note that these are not recent regressions; they've been around since at least 2.6.32.

The CPU in this machine is an Atom N450.

When booted normally the performance on this machine is very poor, but when booted with any of nohz=off, nolapic, or nohpet it improves significantly. The performance also improves if I use the patch below to force the hpet to remain in periodic mode (with hpet=periodic on the command line).

Quoted 5 lines

One other thing I noticed when I had added some logging related to hpet rearming is 3-5 second periods of no log activity occurring fairly frequently, whereas such inactive periods are infrequent when performance is good and are also infrequent on another machine with very similar hardware and no performance issues.

That seems to be related to low power states. When the machine goes idle we switch into lower power states and that requires to use the hpet instead of the local apic timer as that one stops.

You could verify that theory by booting with processor.max_cstate=1

Quoted 9 lines

The machine also hangs for 5 minutes during resume, unless booted with both nohz=off and highres=off, or with hpet=periodic using the patch below. I've traced this down to hanging in an SMI handler during the ACPI _WAK method execution. The 5 minutes corresponds to how long it takes for the low 32 bits of the hpet to wrap in this machine, and since the options that eliminate the hang result in the hpet being in periodic mode during _WAK method execution I suspect that the SMI handler is hanging until a timer interrupt happens.

Quoted 4 lines

One possible explanation here is that the performance problems are also related to hangs in SMI handlers until there's a timer interrupt, although I don't know how that explains why some of the command line options eliminate the performance issues.

Well, that's simple. If you disable NOHZ then we never go into deep idle because the next timer interrupt will arrive in a very short time span. nohpet makes it use the PIT which has a very short (25ms) maximum oneshot time. nolapic disables a lot of the functionality as well.

Quoted 6 lines

kernel/time/tick-broadcast.c +0 -0

@@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
 
 	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 

kernel/time/tick-broadcast.c void tick_broadcast_switch_to_oneshot(void) · line 578

Quoted 12 lines

kernel/time/tick-broadcast.c +6 -1

-	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	bc = tick_broadcast_device.evtdev;
+	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
+		goto unlock;
+
+	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	if (bc)
 		tick_broadcast_setup_oneshot(bc);
+
+unlock:
 	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
 }

Why would you need that? We should not call that when the broadcast device does not have TICKDEV_MODE_ONESHOT. If we do the bug is somewhere else.

Quoted 6 lines · Thomas Gleixner

On Fri, 25 Feb 2011, Seth Forshee wrote:
That seems to be related to low power states. When the machine goes idle we switch into lower power states and that requires to use the hpet instead of the local apic timer as that one stops.

You could verify that theory by booting with processor.max_cstate=1

This fixes the performance in combination with intel_idle.max_cstate=0. Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.

Is the answer to quirk the machine to avoid deep C-states, or is there some better way I can fix this?

Quoted 7 lines

Review

Quoted 3 lines

(quoted diff) +1 -0

+unlock:
 	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
 }

Why would you need that? We should not call that when the broadcast device does not have TICKDEV_MODE_ONESHOT. If we do the bug is somewhere else.

Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the HPET without this change the HPET got put into oneshot mode. The local tick device is checked before switching to nohz, but not the broadcast device. This change was just a quick hack to get around that and test my theory.

Let's wait for the intel and acpi folks. It would be interesting what the new intel toy says to your BIOS.

http://biosbits.org/

Quoted 5 lines

Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the HPET without this change the HPET got put into oneshot mode. The local tick device is checked before switching to nohz, but not the broadcast device. This change was just a quick hack to get around that and test my theory.

Indeed. The patch below should cure that.

Thanks,

tglx

----------->
Subject: clockevents-fix-broadcast.patch
From: Thomas Gleixner <[email protected]>
Date: Fri, 25 Feb 2011 22:34:23 +0100

Signed-off-by Thomas Gleixner <[email protected]>

Quoted 4 lines · Thomas Gleixner

Let's wait for the intel and acpi folks. It would be interesting what the new intel toy says to your BIOS.

http://biosbits.org/

Not much because it doesn't know about my processor. About all I could get out of it is that my MSRs are inconsistent, SMI latency is bad, and the round-trip latency via MWAIT test gives elapsed time = 285ms with 229 iterations/ms.

Quoted 7 lines

Quoted 5 lines

Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the HPET without this change the HPET got put into oneshot mode. The local tick device is checked before switching to nohz, but not the broadcast device. This change was just a quick hack to get around that and test my theory.

Indeed. The patch below should cure that.

It works. I was wondering whether or not I should put the broadcast device in periodic mode with the local ones in nohz; I guess your patch answers my question.

Quoted 4 lines · Seth Forshee

Quoted 3 lines · Thomas Gleixner

Let's wait for the intel and acpi folks. It would be interesting what the new intel toy says to your BIOS.

Quoted 6 lines

Quoted 1 line

http://biosbits.org/

Not much because it doesn't know about my processor. About all I could get out of it is that my MSRs are inconsistent, SMI latency is bad, and the round-trip latency via MWAIT test gives elapsed time = 285ms with 229 iterations/ms.

Ouch.

Looks like the Toshiba NB305 has an Intel Atom processor in it. bits-327, released today, now detects Intel Atom processors, and no longer shows them as unknown. We don't have any processor-specific tests for Atom yet, though we may add some in the future. All of the general tests should work, however.

Can you provide more details on which MSRs show up as inconsistent on your system? They might represent actual bugs in your BIOS, but we might just need to add a few more entries to the Atom CPU-specific MSR blacklist as expected inconsistencies.

Quoted 5 lines · Burt Triplett

Looks like the Toshiba NB305 has an Intel Atom processor in it. bits-327, released today, now detects Intel Atom processors, and no longer shows them as unknown. We don't have any processor-specific tests for Atom yet, though we may add some in the future. All of the general tests should work, however.

The new release detects the Atom processor okay, but in bits-327 the system resets during the SMI frequency/latency test, whereas it didn't with bits-316.

Quoted 4 lines

Can you provide more details on which MSRs show up as inconsistent on your system? They might represent actual bugs in your BIOS, but we might just need to add a few more entries to the Atom CPU-specific MSR blacklist as expected inconsistencies.

Here's what it says:

   (MSR 0x39 consistent): FAIL
   (MSR 0x199 consistent): FAIL
   (MSR 0x1a0 consistent): FAIL
   Summary: 7917 passed, 3 failed

I have 3 systems with this same processor, and they all report the same thing here.

Quoted 3 lines

The new release detects the Atom processor okay, but in bits-327 the system resets during the SMI frequency/latency test, whereas it didn't with bits-316.

The SMI Frequency and Latency test should be fixed in bits-329.

To get more information on how the MSRs differ between CPUs, you can turn on verbose mode. Go to the GRUB command line by hitting 'c', and enter "test_options -v 2". Then hit Esc to go back to the menu, and re-run the MSR consistency check. It will now show you the value of the MSR on each CPU.

MSR 0x1a0 is IA32_MISC_ENABLE, and generally any inconsistency in that MSR represents a BIOS bug: it didn't enable features identically on all CPUs. Once you see which bits differ, you can look at the Intel Software Developer's Manual (specifically volume 3B) to find out what those bits mean and which specific features the BIOS inconsistently enabled.

MSR 0x199 is IA32_PERF_CTL. A difference in that MSR usually means the BIOS left CPUs in different P-states when booting, which it really shouldn't do. Again, you can decode the result with the Intel SDM.

Looking into MSR 0x39.

Quoted 5 lines · Burt Triplett

Quoted 3 lines

The new release detects the Atom processor okay, but in bits-327 the system resets during the SMI frequency/latency test, whereas it didn't with bits-316.

The SMI Frequency and Latency test should be fixed in bits-329.

Yes, it's fixed now, thanks.

Quoted 6 lines

To get more information on how the MSRs differ between CPUs, you can turn on verbose mode. Go to the GRUB command line by hitting 'c', and enter "test_options -v 2". Then hit Esc to go back to the menu, and re-run the MSR consistency check. It will now show you the value of the MSR on each CPU.

Quoted 6 lines

MSR 0x1a0 is IA32_MISC_ENABLE, and generally any inconsistency in that MSR represents a BIOS bug: it didn't enable features identically on all CPUs. Once you see which bits differ, you can look at the Intel Software Developer's Manual (specifically volume 3B) to find out what those bits mean and which specific features the BIOS inconsistently enabled.

(MSR 0x1a0 consistent) FAIL (2 different values) 1 CPUs 0x0000000364950089: 0x0
1 CPUs 0x0000000364950081: 0x1

Bit 3: Automatic Thermal Control Circuit Enable

I'm not sure of the impact of this, since there's really only one core in the CPU. One of the others differs has bit 3 inconsistent as well. On the third bit 3 is the same but bit 0 (fast-strings enable) is inconsistent.

Quoted 4 lines

MSR 0x199 is IA32_PERF_CTL. A difference in that MSR usually means the BIOS left CPUs in different P-states when booting, which it really shouldn't do. Again, you can decode the result with the Intel SDM.

(MSR 0x199 consistent): FAIL (2 different values) 1 CPUs 0x0000000000000a1f: 0x0
1 CPUs 0x0000000000000613: 0x1

So yes, the cores are in different P-states. The results from the other machines are similar, but the values for CPU1 are slightly different for each machine. CPU1 has the same value on all three machines.

Quoted 1 line

Looking into MSR 0x39.

Yeah, this one isn't in the SDM at all. Here are the values I got:

(MSR 0x39 consistent): FAIL (2 different values) 1 CPUs 0x0000000000000000: 0x0
1 CPUs 0x0000000000000001: 0x1

Values are identical for all three machines.

I'm not really getting the idea that any of these is contributing to the problems I'm seeing with this machine though.

Quoted 10 lines · Seth Forshee

Quoted 6 lines · Thomas Gleixner

On Fri, 25 Feb 2011, Seth Forshee wrote:
That seems to be related to low power states. When the machine goes idle we switch into lower power states and that requires to use the hpet instead of the local apic timer as that one stops.

You could verify that theory by booting with processor.max_cstate=1

This fixes the performance in combination with intel_idle.max_cstate=0. Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.

That was expected :)

Quoted 2 lines

Is the answer to quirk the machine to avoid deep C-states, or is there some better way I can fix this?

Let's wait for the intel and acpi folks. It would be interesting what the new intel toy says to your BIOS.

Quoted 12 lines · Thomas Gleixner

Quoted 10 lines · Seth Forshee

Quoted 6 lines · Thomas Gleixner

On Fri, 25 Feb 2011, Seth Forshee wrote:
That seems to be related to low power states. When the machine goes idle we switch into lower power states and that requires to use the hpet instead of the local apic timer as that one stops.

You could verify that theory by booting with processor.max_cstate=1

This fixes the performance in combination with intel_idle.max_cstate=0. Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.

Quoted 7 lines

That was expected :)

Quoted 2 lines

Is the answer to quirk the machine to avoid deep C-states, or is there some better way I can fix this?

Let's wait for the intel and acpi folks. It would be interesting what the new intel toy says to your BIOS.

Since the discussion on this issue died without really getting anywhere, I went ahead and threw together the patch below to disable anything deeper than C1 for this machine. I hope a better solution can be found, but if not would something like this be an acceptable workaround?

As for the hangs during resume, unless someone has a better suggestion I guess I'll start looking into forcing the HPET to remain in periodic mode throughout suspend.