Skip to contentSkip to conversation

Tracing of power:power_start events doesn't work

DISCUSSION Ronny Tschüter 6 participants 12 messages May 3, 2010 – Jun 29, 2010 ACPI

lore
Share & tools
lore
#0 lore raw

Hello,

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing

echo "power:power_start" | sudo tee set_event

on the console, the trace doesn't contain such events. power:power_frequency and power:power_end events are reported correctly by the system. I read in the mailing list that Robert Schoene had a similiar problem, but with power:power_end events. Is there a known bug regarding to trace_power_start methods in process.c?

Bye Ronny

Signature · 12 lines
Ronny Tschueter

Technische Universität Dresden
Center for Information Services
and High Performance Computing (ZIH)
D-01062 Dresden
Germany

Office: Willersbau, room A31
Phone:	(+49) 351/463-35448
E-Mail: [email protected]
WWW:    http://www.tu-dresden.de/zih
#1 lore raw
Quoted 6 lines · =?ISO-8859-15?Q?Ronny_Tsch=FCter?= <[email protected]> writes:

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing echo "power:power_start" | sudo tee set_event
on the console, the trace doesn't contain such
events [...]

FWIW, on that kernel version (Fedora 13 on Athlon X2), systemtap hooks into all those tracepoints fine, and

# stap -e 'probe kernel.trace("power_start") { log($$parms) }'

produces plenty of traffic.

- FChE

#2 lore raw
In reply to Ronny Tschüter
Quoted 6 lines · Ronny Tschüter

Hello,

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing

Quoted 7 lines

echo "power:power_start" | sudo tee set_event

on the console, the trace doesn't contain such events. power:power_frequency and power:power_end events are reported correctly by the system. I read in the mailing list that Robert Schoene had a similiar problem, but with power:power_end events. Is there a known bug regarding to trace_power_start methods in process.c?

Is this for perf or ftrace? I'm assuming you did this from the debugfs tracing directory. The set_event file is for ftrace not perf.

-- Steve

#3 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Ronny Tschüter

Hello,

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing

echo "power:power_start" | sudo tee set_event

Quoted 9 lines
Quoted 6 lines

on the console, the trace doesn't contain such events. power:power_frequency and power:power_end events are reported correctly by the system. I read in the mailing list that Robert Schoene had a similiar problem, but with power:power_end events. Is there a known bug regarding to trace_power_start methods in process.c?

Is this for perf or ftrace? I'm assuming you did this from the debugfs tracing directory. The set_event file is for ftrace not perf.

It is for perf - i'm interested in event tracing. I used the event tracing system like it is described in Documentation/trace/events.txt. In addition i wrote a program which uses syscall(__NR_perf_counter_open, ... ) to trace power events. But the result is the same - power:power_end events are reported, power:power_start events not.

Bye Ronny

#4 lore raw
Quoted 20 lines · Ronny Tschüter
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Ronny Tschüter

Hello,

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing

echo "power:power_start" | sudo tee set_event

Quoted 9 lines
Quoted 6 lines

on the console, the trace doesn't contain such events. power:power_frequency and power:power_end events are reported correctly by the system. I read in the mailing list that Robert Schoene had a similiar problem, but with power:power_end events. Is there a known bug regarding to trace_power_start methods in process.c?

Is this for perf or ftrace? I'm assuming you did this from the debugfs tracing directory. The set_event file is for ftrace not perf.

Quoted 7 lines

It is for perf - i'm interested in event tracing. I used the event tracing system like it is described in Documentation/trace/events.txt. In addition i wrote a program which uses syscall(__NR_perf_counter_open, ... ) to trace power events. But the result is the same - power:power_end events are reported, power:power_start events not.

Bye Ronny

Hi, can you please compare the perf and ftrace output to see if ftrace works while perf doesn't for example?

In perf:

	perf record -f -c 1 -a -e power:power_start
	^C
	perf report

In ftrace:
echo 1 > $DEBUGFS/tracing/events/power/power_start/enable cat $DEBUGFS/trace_pipe

Thanks.

#5 lore raw
Quoted 43 lines · Frederic Weisbecker
Quoted 24 lines · Ronny Tschüter
Quoted 18 lines · Steven Rostedt
Quoted 6 lines · Ronny Tschüter

Hello,

i use kernel version 2.6.33.2 (x86-64, Core 2 Duo) and the perf events to track frequency and c states of my cpu. But if i want to trace power:power_start events, it does not work. After typing

Quoted 9 lines

echo "power:power_start" | sudo tee set_event

on the console, the trace doesn't contain such events. power:power_frequency and power:power_end events are reported correctly by the system. I read in the mailing list that Robert Schoene had a similiar problem, but with power:power_end events. Is there a known bug regarding to trace_power_start methods in process.c?

Quoted 4 lines

Is this for perf or ftrace? I'm assuming you did this from the debugfs tracing directory. The set_event file is for ftrace not perf.

Quoted 8 lines

It is for perf - i'm interested in event tracing. I used the event tracing system like it is described in Documentation/trace/events.txt. In addition i wrote a program which uses syscall(__NR_perf_counter_open, ... ) to trace power events. But the result is the same - power:power_end events are reported, power:power_start events not.

Bye Ronny

Hi, can you please compare the perf and ftrace output to see if ftrace works while perf doesn't for example?

In perf:

	perf record -f -c 1 -a -e power:power_start
	^C
	perf report
Quoted 5 lines

In ftrace:
echo 1> $DEBUGFS/tracing/events/power/power_start/enable cat $DEBUGFS/trace_pipe

Thanks

Hi

IN PERF:

-> patched kernel (see my mail from yesterday)
perf record -f -c 1 -a -e power:power_start
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.500 MB perf.data (~21826 samples) ] perf report
# Samples: 7880
#
# Overhead Command Shared Object Symbol

# ........  ...............  .................  ......
#
    57.13%          swapper                  0  [k] 0000000000000000
    42.86%             init                  0  [k] 0000000000000000
     0.01%          swapper  [kernel]           [k] 0000000000000000
#
# (For a higher level overview, try: perf report --sort comm,dso)
#

-> unpatched kernel
perf record -f -c 1 -a -e power:power_start
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.263 MB perf.data (~11484 samples) ] perf report
# Samples: 0
#

# Overhead          Command  Shared Object  Symbol
# ........  ...............  .............  ......

#
#
# (For a higher level overview, try: perf report --sort comm,dso) #

IN FTRACE:
-> patched kernel
cat trace_pipe shows events like

<idle>-0     [001] 59067.030622: power_start: type=1 state=2
<idle>-0     [001] 59067.032514: power_start: type=1 state=2
<idle>-0     [000] 59067.032522: power_start: type=1 state=2
<idle>-0     [001] 59067.032561: power_start: type=1 state=2
<idle>-0     [000] 59067.037527: power_start: type=1 state=2
<idle>-0     [001] 59067.037528: power_start: type=1 state=2

-> unpatched kernel
cat trace returns no output

Bye

#6 lore raw
In reply to Steven Rostedt

Hello,

I took a look at ftrace and tried to narrow my problem down.

 1) <idle>-0    |                     |  cpuidle_idle_call() {
 1) <idle>-0    |                     |    menu_select() {
 1) <idle>-0    |   0.219 us    |      pm_qos_requirement();
 1) <idle>-0    |   0.219 us    |      tick_nohz_get_sleep_length();
 1) <idle>-0    |   0.210 us    |      nr_iowait_cpu();
 1) <idle>-0    |   0.213 us    |      this_cpu_load();
 1) <idle>-0    |   0.213 us    |      nr_iowait_cpu();
 1) <idle>-0    |   2.472 us    |    }
 1) <idle>-0    |                     |    acpi_idle_enter_simple() {
 1) <idle>-0    |                     |      
lapic_timer_state_broadcast() {
 1) <idle>-0    |                     |        clockevents_notify() {
 1) <idle>-0    |   0.234 us    |          _raw_spin_lock_irqsave();
 1) <idle>-0    |                     |          
raw_notifier_call_chain() {
 1) <idle>-0    |                     |            tick_notify() {
 1) <idle>-0    |                     |              

tick_broadcast_oneshot_control() {
1) <idle>-0 | 0.237 us | _raw_spin_lock_irqsave();
1) <idle>-0 | |
clockevents_set_mode() {
1) <idle>-0 | |
lapic_timer_setup() {
1) <idle>-0 | 0.222 us |
native_apic_mem_read();
1) <idle>-0 | 0.213 us |
native_apic_mem_write();
1) <idle>-0 | 0.210 us |
native_apic_mem_write();

 1) <idle>-0    |   1.554 us    |                  }
 1) <idle>-0    |   2.007 us    |                }
 1) <idle>-0    |                     |                
tick_dev_program_event() {
 1) <idle>-0    |                     |                  ktime_get() {
 1) <idle>-0    |   0.606 us    |                    read_hpet();
 1) <idle>-0    |   1.053 us    |                  }
 1) <idle>-0    |                     |                  
clockevents_program_event() {
 1) <idle>-0    |                     |                    
hpet_legacy_next_event() {
 1) <idle>-0    |   1.692 us    |                      hpet_next_event();
 1) <idle>-0    |   2.190 us    |                    }
 1) <idle>-0    |   2.664 us    |                  }
 1) <idle>-0    |   4.410 us    |                }
 1) <idle>-0    |   0.237 us    |                
_raw_spin_unlock_irqrestore();
 1) <idle>-0    |   8.022 us    |              }
 1) <idle>-0    |   8.460 us    |            }
 1) <idle>-0    |   8.943 us    |          }
 1) <idle>-0    |   0.231 us    |          _raw_spin_unlock_irqrestore();
 1) <idle>-0    | + 10.296 us   |        }
 1) <idle>-0    | + 10.734 us   |      }
 1) <idle>-0    |                     |      ktime_get_real() {
 1) <idle>-0    |                     |        getnstimeofday() {
 1) <idle>-0    |   0.630 us    |          read_hpet();
 1) <idle>-0    |   1.389 us    |        }
 1) <idle>-0    |   1.830 us    |      }
 1) <idle>-0    | ! 3833.301 us |      acpi_idle_do_entry()

Like the callgraph shows, my machine uses the cpuidle framework with menu governor and an acpi-based driver to handle idle states. Unfortunately I can't find any instrumentation for tracing power events in the cpuidle.c and processor_idle.c (acpi idle management). The following patch achieved success.

new/processor_idle.c renamed from old/processor_idle.c +1 -0
index cc978a8..1668d34 100644
@@ -42,6 +42,7 @@
 #include <linux/clockchips.h>
 #include <linux/cpuidle.h>
 #include <linux/irqflags.h>
+#include <trace/events/power.h>
 
 /*
  * Include the apic definitions for x86 to have the APIC timer related 

defines
@@ -796,6 +797,18 @@ static int acpi_idle_bm_check(void)
*/
static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx) {
+ switch (cx->type) {
+ case ACPI_STATE_C1:
+ trace_power_start(POWER_CSTATE, 1);
+ break;
+ case ACPI_STATE_C2:
+ trace_power_start(POWER_CSTATE, 2);
+ break;
+ case ACPI_STATE_C3:
+ trace_power_start(POWER_CSTATE, 3);
+ break;
+ }
+

     /* Don't trace irqs off for idle */
     stop_critical_timings();
     if (cx->entry_method == ACPI_CSTATE_FFH) {
#7 lore raw
Quoted 9 lines · Ronny Tschüter

/*
* Include the apic definitions for x86 to have the APIC timer related
defines

(quoted diff) +2 -0
@@ -796,6 +797,18 @@ static int acpi_idle_bm_check(void)
   */
  static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
  {
+    switch (cx->type) {
+    case ACPI_STATE_C1:

Review static int acpi_idle_bm_check(void) · line 811

Quoted 10 lines
(quoted diff) +10 -0
+        trace_power_start(POWER_CSTATE, 1);
+        break;
+    case ACPI_STATE_C2:
+        trace_power_start(POWER_CSTATE, 2);
+        break;
+    case ACPI_STATE_C3:
+        trace_power_start(POWER_CSTATE, 3);
+        break;
+    }
+

Depending on gcc, the above can bloat the code since each call to trace_power_start() is a macro expanded. Try to call it just once. Perhaps one of the following:

	trace_power_start(POWER_CSTATE,
		cx->type == ACPI_STATE_C1 ? 1 :
		cx->type == ACPI_STATE_C2 ? 2 :
		3);

Or make a local variable instead, although the above would compile out completely if tracing is not enabled.

	int type:

	switch (cx->type) {
	case ACPI_STATE_C1:
		type = 1;
		break;
	case ACPI_STATE_C2:
		type = 2;
		break;
	case ACPI_STATE_C3:
		type = 3;
		break;
	}
	trace_power_start(POWER_CSTATE, type);

-- Steve

Quoted 4 lines
(quoted diff) +0 -0
      /* Don't trace irqs off for idle */
      stop_critical_timings();
      if (cx->entry_method == ACPI_CSTATE_FFH) {
#8 lore raw
Quoted 17 lines · Steven Rostedt
Quoted 15 lines · Ronny Tsch�ter

/*
* Include the apic definitions for x86 to have the APIC timer related
defines

(quoted diff) +8 -0
@@ -796,6 +797,18 @@ static int acpi_idle_bm_check(void)
    */
   static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
   {
+    switch (cx->type) {
+    case ACPI_STATE_C1:
+        trace_power_start(POWER_CSTATE, 1);
+        break;
+    case ACPI_STATE_C2:
+        trace_power_start(POWER_CSTATE, 2);
+        break;
+    case ACPI_STATE_C3:
Quoted 8 lines

Review static int acpi_idle_bm_check(void) · line 811

Quoted 4 lines
(quoted diff) +4 -0
+        trace_power_start(POWER_CSTATE, 3);
+        break;
+    }
+

Depending on gcc, the above can bloat the code since each call to trace_power_start() is a macro expanded. Try to call it just once. Perhaps one of the following:

the code is also incorrect fundamentally.
You need to pass in the mwait value or equivalent; the ACPI STATE type is pretty much useless random garbage and should completely be ignored.

#9 lore raw

Am Mittwoch, den 05.05.2010, 07:33 -0700 schrieb Arjan van de Ven:

Quoted 27 lines · Steven Rostedt
Quoted 17 lines · Ronny Tsch�ter
Quoted 16 lines

/*
* Include the apic definitions for x86 to have the APIC timer related
defines

(quoted diff) +8 -0
@@ -796,6 +797,18 @@ static int acpi_idle_bm_check(void)
    */
   static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
   {
+    switch (cx->type) {
+    case ACPI_STATE_C1:
+        trace_power_start(POWER_CSTATE, 1);
+        break;
+    case ACPI_STATE_C2:
+        trace_power_start(POWER_CSTATE, 2);
+        break;
+    case ACPI_STATE_C3:
Quoted 8 lines

Review static int acpi_idle_bm_check(void) · line 811

Quoted 4 lines
(quoted diff) +4 -0
+        trace_power_start(POWER_CSTATE, 3);
+        break;
+    }
+

Depending on gcc, the above can bloat the code since each call to trace_power_start() is a macro expanded. Try to call it just once. Perhaps one of the following:

Quoted 3 lines

the code is also incorrect fundamentally.
You need to pass in the mwait value or equivalent; the ACPI STATE type is pretty much useless random garbage and should completely be ignored.

You're correct for your case (switching to c-state via monitor/mwait). But the current implementation is broken too. It works only if your kernel uses processor_idle AND monitor/mwait. For all other cases it fails - may it be on Ronnys system, that uses IO port based C-states or my system, that uses cpu_idle (not processor_idle).

I'd suggest, that it should work for IO port based switches and could be included in the syntactic block of that case.
Afaik, halt means that the processor goes to C1. So in this block, there could be a power_start event too with 1 as new state. (Maybe there are problems with AMDs C1E which would be reported as C1.)

Robert

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 7 lines
Robert Schoene
Technische Universitaet Dresden
Zentrum fuer Informationsdienste und Hochleistungsrechnen
01062 Dresden

Tel.: (0351) 463-42483, Fax: (0351) 463-37773
E-Mail: [email protected]
#10 lore raw
Quoted 41 lines · Robert Schöne

On Wed, den 05.05.2010-05-05, 07:33 -0700 schrieb Arjan van de Ven:

Quoted 29 lines · Steven Rostedt
Quoted 19 lines · Ronny Tsch�ter
Quoted 17 lines

/*
* Include the apic definitions for x86 to have the APIC timer related
defines

(quoted diff) +8 -0
@@ -796,6 +797,18 @@ static int acpi_idle_bm_check(void)
     */
    static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
    {
+    switch (cx->type) {
+    case ACPI_STATE_C1:
+        trace_power_start(POWER_CSTATE, 1);
+        break;
+    case ACPI_STATE_C2:
+        trace_power_start(POWER_CSTATE, 2);
+        break;
+    case ACPI_STATE_C3:
Quoted 9 lines

Review static int acpi_idle_bm_check(void) · line 812

Quoted 5 lines
(quoted diff) +4 -0
+        trace_power_start(POWER_CSTATE, 3);
+        break;
+    }
+
 

Depending on gcc, the above can bloat the code since each call to trace_power_start() is a macro expanded. Try to call it just once. Perhaps one of the following:

Quoted 4 lines

the code is also incorrect fundamentally.
You need to pass in the mwait value or equivalent; the ACPI STATE type is pretty much useless random garbage and should completely be ignored.

You're correct for your case (switching to c-state via monitor/mwait). But the current implementation is broken too. It works only if your kernel uses processor_idle AND monitor/mwait. For all other cases it fails - may it be on Ronnys system, that uses IO port based C-states or my system, that uses cpu_idle (not processor_idle).

Quoted 8 lines

I'd suggest, that it should work for IO port based switches and could be included in the syntactic block of that case.
Afaik, halt means that the processor goes to C1. So in this block, there could be a power_start event too with 1 as new state. (Maybe there are problems with AMDs C1E which would be reported as C1.)

Robert

Unfortunately the bug still exists in kernel 2.6.34. On my system I'm not able to trace power:power_start events, because they are not reported. Regarding to the preceding comments I moved my code snippet into the IO port based C-state part of acpi_idle_do_entry(). Therefore it should not interfere with the other two C-state calls in this function. Furthermore a lot of other code sequences in processor_idle.c use the ACPI STATE type in if- or switch-statements. So I don't think that ACPI STATE type is useless garbage.

Finally here is my new patch:

new/processor_idle.c renamed from old/processor_idle.c +1 -0
index 5939e7f..5818522 100644
@@ -43,6 +43,7 @@
 #include <linux/clockchips.h>
 #include <linux/cpuidle.h>
 #include <linux/irqflags.h>
+#include <trace/events/power.h>
 
 /*
  * Include the apic definitions for x86 to have the APIC timer related 

defines
@@ -807,6 +808,10 @@ static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)

     } else {
         int unused;
         /* IO port based C-state */
+        trace_power_start(POWER_CSTATE,
+        cx->type == ACPI_STATE_C1 ? 1 :
+        cx->type == ACPI_STATE_C2 ? 2 :
+        3);
         inb(cx->address);
         /* Dummy wait op - must do something useless after P_LVL2 read
            because chipsets cannot guarantee that STPCLK# signal
#11 lore raw

I use Linux performance counters to trace some performance related events. Unfortunately power:power_start events are not reported on my machine. In /drivers/acpi/processor_idle.c I found three different ways to call a C-state, but only one way (via architectural FFH based C-state and using MONITOR/MWAIT) is currently instrumented. Here is my patch to instrument the remaining two possibilities:

drivers/acpi/processor_idle.c +1 -0
index b1b3856..08e7c42 100644
@@ -43,6 +43,7 @@
 #include <linux/clockchips.h>
 #include <linux/cpuidle.h>
 #include <linux/irqflags.h>
+#include <trace/events/power.h>
 
 /*
  * Include the apic definitions for x86 to have the APIC timer related 

defines
@@ -136,6 +137,7 @@ static void acpi_safe_halt(void)

      */
     smp_mb();
     if (!need_resched()) {
+        trace_power_start(POWER_CSTATE,1);
         safe_halt();
         local_irq_disable();
     }
@@ -797,6 +799,10 @@ static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
     } else {
         int unused;
         /* IO port based C-state */
+        trace_power_start(POWER_CSTATE,
+        cx->type == ACPI_STATE_C1 ? 1 :
+        cx->type == ACPI_STATE_C2 ? 2 :
+        3);
         inb(cx->address);
         /* Dummy wait op - must do something useless after P_LVL2 read
            because chipsets cannot guarantee that STPCLK# signal