Skip to contentSkip to conversation

2.6.23-mm1 breaks C-state support on Intel T7200 x86_64

DISCUSSION [email protected] 3 participants 8 messages Nov 8, 2007 – Nov 12, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

(Sorry for not reporting this sooner - I haven't been running off battery much in the last 3 weeks, so I didn't notice it till now...)

Dell Latitude D820 laptop, T7200 Core2 Duo CPU, x86_64 kernel.

As reported by 'powertop' on a basically idle machine:

2.6.23-mm1:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        (100.0%)        2.00 Ghz     0.8%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3                0.0ms ( 0.0%)         1000 Mhz    99.2%

2.6.23-rc8-mm2:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 0.3%)         2.00 Ghz     0.0%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3               31.5ms (99.7%)         1000 Mhz   100.0%

In addition, the ACPI power estimate reported about 25 watts for 23-mm1, but only 21 watts for -rc8-mm2, a significant regression.

I bisected this down to this set of patches:

pm-qos-infrastructure-and-interface.patch pm-qos-infrastructure-and-interface-fix.patch pm-qos-infrastructure-and-interface-vs-git-acpi.patch pm-qos-infrastructure-and-interface-vs-git-acpi-2.patch latencyc-use-qos-infrastructure.patch

The patch says:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

#1 lore raw
Quoted 46 lines · On Thu, 08 Nov 2007 12:19:44 -0500 [email protected] wrote:

(Sorry for not reporting this sooner - I haven't been running off battery much in the last 3 weeks, so I didn't notice it till now...)

Dell Latitude D820 laptop, T7200 Core2 Duo CPU, x86_64 kernel.

As reported by 'powertop' on a basically idle machine:

2.6.23-mm1:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        (100.0%)        2.00 Ghz     0.8%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3                0.0ms ( 0.0%)         1000 Mhz    99.2%

2.6.23-rc8-mm2:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 0.3%)         2.00 Ghz     0.0%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3               31.5ms (99.7%)         1000 Mhz   100.0%

In addition, the ACPI power estimate reported about 25 watts for 23-mm1, but only 21 watts for -rc8-mm2, a significant regression.

I bisected this down to this set of patches:

pm-qos-infrastructure-and-interface.patch pm-qos-infrastructure-and-interface-fix.patch pm-qos-infrastructure-and-interface-vs-git-acpi.patch pm-qos-infrastructure-and-interface-vs-git-acpi-2.patch latencyc-use-qos-infrastructure.patch

The patch says:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

Quoted 5 lines

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

That's a great report, thanks. Over to you, Mark ;)

btw, I also have a note here that these patches caused Rafael to see an smp_call_function() inside local_irq_save(). Did that get fixed?

#2 lore raw
Quoted 52 lines · Andrew Morton
Quoted 46 lines · On Thu, 08 Nov 2007 12:19:44 -0500 [email protected] wrote:

(Sorry for not reporting this sooner - I haven't been running off battery much in the last 3 weeks, so I didn't notice it till now...)

Dell Latitude D820 laptop, T7200 Core2 Duo CPU, x86_64 kernel.

As reported by 'powertop' on a basically idle machine:

2.6.23-mm1:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        (100.0%)        2.00 Ghz     0.8%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3                0.0ms ( 0.0%)         1000 Mhz    99.2%

2.6.23-rc8-mm2:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 0.3%)         2.00 Ghz     0.0%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3               31.5ms (99.7%)         1000 Mhz   100.0%

In addition, the ACPI power estimate reported about 25 watts for 23-mm1, but only 21 watts for -rc8-mm2, a significant regression.

I bisected this down to this set of patches:

pm-qos-infrastructure-and-interface.patch pm-qos-infrastructure-and-interface-fix.patch pm-qos-infrastructure-and-interface-vs-git-acpi.patch pm-qos-infrastructure-and-interface-vs-git-acpi-2.patch latencyc-use-qos-infrastructure.patch

The patch says:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

Quoted 5 lines

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

Quoted 4 lines

That's a great report, thanks. Over to you, Mark ;)

btw, I also have a note here that these patches caused Rafael to see an smp_call_function() inside local_irq_save(). Did that get fixed?

Ah, I see the problem. I think I posted a fix to this. The problem is that what's in the mm1 tree has a parameter PM_QOS_IDLE that needed to be PM_QOS_CPU_DMA_LATENCY.

I'm not sure what's in the current MM tree at this point so I can't say its been fixed. Is there an easy way from me to see what's currently in MM?

FWIW I think I fixed this when I fixed up Rafael's issue. Would you like me to send out a re-fresh patch against 2.6.23-mm1?

--mgross

#3 lore raw
Quoted 10 lines · On Thu, 8 Nov 2007 12:03:52 -0800 Mark Gross <[email protected]> wrote:

...

Quoted 8 lines
Quoted 7 lines

call.

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

Quoted 8 lines
Quoted 4 lines

That's a great report, thanks. Over to you, Mark ;)

btw, I also have a note here that these patches caused Rafael to see an smp_call_function() inside local_irq_save(). Did that get fixed?

Ah, I see the problem. I think I posted a fix to this. The problem is that what's in the mm1 tree has a parameter PM_QOS_IDLE that needed to be PM_QOS_CPU_DMA_LATENCY.

That doesn't ring a bell.

Quoted 3 lines

I'm not sure what's in the current MM tree at this point so I can't say its been fixed. Is there an easy way from me to see what's currently in MM?

Not terribly. ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/mm/broken-out-2007-11-06-02-32.tar.gz is from two days ago.

Quoted 2 lines

FWIW I think I fixed this when I fixed up Rafael's issue. Would you like me to send out a re-fresh patch against 2.6.23-mm1?

sure.

#4 lore raw
In reply to [email protected]
Quoted 17 lines · On Thu, Nov 08, 2007 at 12:19:44PM -0500, [email protected] wrote:

(Sorry for not reporting this sooner - I haven't been running off battery much in the last 3 weeks, so I didn't notice it till now...)

Dell Latitude D820 laptop, T7200 Core2 Duo CPU, x86_64 kernel.

As reported by 'powertop' on a basically idle machine:

2.6.23-mm1:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        (100.0%)        2.00 Ghz     0.8%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3                0.0ms ( 0.0%)         1000 Mhz    99.2%

2.6.23-rc8-mm2:

Quoted 8 lines

Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 0.3%)         2.00 Ghz     0.0%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3               31.5ms (99.7%)         1000 Mhz   100.0%

In addition, the ACPI power estimate reported about 25 watts for 23-mm1, but only 21 watts for -rc8-mm2, a significant regression.

well, thats because you burn less watts if you get into C3.

Quoted 8 lines

I bisected this down to this set of patches:

pm-qos-infrastructure-and-interface.patch pm-qos-infrastructure-and-interface-fix.patch pm-qos-infrastructure-and-interface-vs-git-acpi.patch pm-qos-infrastructure-and-interface-vs-git-acpi-2.patch latencyc-use-qos-infrastructure.patch

yipes! I'll look at it right away. It looks like an integration issue with CPU-IDLE patches (those control the C-state entry). I'll get it fixed up.

Quoted 12 lines

The patch says:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

Quoted 5 lines

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

you won't have such a process (at least I highly doubt you do) I need to fix this. Thanks for taking the time to bisect it and reporting it to me!

--mgross

#5 lore raw
In reply to [email protected]
Quoted 45 lines · On Thu, Nov 08, 2007 at 12:19:44PM -0500, [email protected] wrote:

(Sorry for not reporting this sooner - I haven't been running off battery much in the last 3 weeks, so I didn't notice it till now...)

Dell Latitude D820 laptop, T7200 Core2 Duo CPU, x86_64 kernel.

As reported by 'powertop' on a basically idle machine:

2.6.23-mm1:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        (100.0%)        2.00 Ghz     0.8%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3                0.0ms ( 0.0%)         1000 Mhz    99.2%

2.6.23-rc8-mm2:

Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 0.3%)         2.00 Ghz     0.0%
C1                0.0ms ( 0.0%)         1.67 Ghz     0.0%
C2                0.0ms ( 0.0%)         1333 Mhz     0.0%
C3               31.5ms (99.7%)         1000 Mhz   100.0%

In addition, the ACPI power estimate reported about 25 watts for 23-mm1, but only 21 watts for -rc8-mm2, a significant regression.

I bisected this down to this set of patches:

pm-qos-infrastructure-and-interface.patch pm-qos-infrastructure-and-interface-fix.patch pm-qos-infrastructure-and-interface-vs-git-acpi.patch pm-qos-infrastructure-and-interface-vs-git-acpi-2.patch latencyc-use-qos-infrastructure.patch

The patch says:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

Quoted 8 lines

I shouldn't have to have a process open a /dev/file, write a number, and then stay around forever so the file doesn't close in order to get the same behavior I was getting by default before. What needs to happen to get this to not be a behavior regression/change?

wing patch fixes up the cpuidle / pm-qos integration.

I suspect that this is folded into another mm patch but it should fix C-state issue identified.

--mgross

-------------

drivers/cpuidle/cpuidle.c +1 -1
Index: linux-2.6.23-mm1/drivers/cpuidle/cpuidle.c
===================================================================
@@ -268,7 +268,7 @@
 
 static inline void latency_notifier_init(struct notifier_block *n)
 {
-        pm_qos_add_notifier(PM_QOS_CPUIDLE, n);
+	pm_qos_add_notifier(PM_QOS_CPU_DMA_LATENCY, n);
 }
 
 #else /* CONFIG_SMP */
drivers/cpuidle/governors/ladder.c +1 -1
Index: linux-2.6.23-mm1/drivers/cpuidle/governors/ladder.c
===================================================================
@@ -82,7 +82,7 @@
 	if (last_idx < dev->state_count - 1 &&
 	    last_residency > last_state->threshold.promotion_time &&
 	    dev->states[last_idx + 1].exit_latency <=
-			pm_qos_requirement(PM_QOS_CPUIDLE)) {
+			pm_qos_requirement(PM_QOS_CPU_DMA_LATENCY)) {
 		last_state->stats.promotion_count++;
 		last_state->stats.demotion_count = 0;
 		if (last_state->stats.promotion_count >= last_state->threshold.promotion_count) {
drivers/cpuidle/governors/menu.c +2 -1
Index: linux-2.6.23-mm1/drivers/cpuidle/governors/menu.c
===================================================================
@@ -48,7 +48,8 @@
 			break;
 		if (s->target_residency > data->predicted_us)
 			break;
-		if (s->exit_latency > pm_qos_requirement(PM_QOS_CPUIDLE))
+		if (s->exit_latency >
+			pm_qos_requirement(PM_QOS_CPU_DMA_LATENCY))
 			break;
 	}
 
include/linux/pm_qos_params.h +1 -12
Index: linux-2.6.23-mm1/include/linux/pm_qos_params.h
===================================================================
@@ -6,23 +6,12 @@
 #include <linux/notifier.h>
 #include <linux/miscdevice.h>
 
-struct requirement_list {
-	struct list_head list;
-	union {
-		s32 value;
-		s32 usec;
-		s32 kbps;
-	};
-	char *name;
-};
-
 #define PM_QOS_RESERVED 0
 #define PM_QOS_CPU_DMA_LATENCY 1
 #define PM_QOS_NETWORK_LATENCY 2
 #define PM_QOS_NETWORK_THROUGHPUT 3
-#define PM_QOS_CPUIDLE 4
 
-#define PM_QOS_NUM_CLASSES 5
+#define PM_QOS_NUM_CLASSES 4
 #define PM_QOS_DEFAULT_VALUE -1
 
 int pm_qos_add_requirement(int qos, char *name, s32 value);
kernel/pm_qos_params.c +10 -0
Index: linux-2.6.23-mm1/kernel/pm_qos_params.c
===================================================================
@@ -47,6 +47,16 @@
  * held, taken with _irqsave.  One lock to rule them all
  */
 
+struct requirement_list {
+	struct list_head list;
+	union {
+		s32 value;
+		s32 usec;
+		s32 kbps;
+	};
+	char *name;
+};
+
 struct pm_qos_object {
 	struct requirement_list requirements;
 	struct srcu_notifier_head notifiers;
#6 lore raw
Quoted 4 lines · Mark Gross

wing patch fixes up the cpuidle / pm-qos integration.

I suspect that this is folded into another mm patch but it should fix C-state issue identified.

Confirming that patch left my CPUs mostly in C3 again. Thanks.

I'll have to let Mark and Andrew figure out this code's status in the -mm queue.

#7 lore raw
Quoted 10 lines · On Fri, Nov 09, 2007 at 03:24:44PM -0500, [email protected] wrote:
Quoted 4 lines · Mark Gross

wing patch fixes up the cpuidle / pm-qos integration.

I suspect that this is folded into another mm patch but it should fix C-state issue identified.

Confirming that patch left my CPUs mostly in C3 again. Thanks.

I'll have to let Mark and Andrew figure out this code's status in the -mm queue.

I checked on this last week, I'm pretty sure its in Andrew's current tree.

Thanks again for looking at this.

--mgross