Skip to contentSkip to conversation

tracing/function-graph-tracer: use the more lightweight local clock

PATCH In review Frederic Weisbecker 3 participants 14 messages Mar 5, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 2/2 tracing/function-graph-tracer: use the more lightweight local clock

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

---
 arch/x86/kernel/ftrace.c             |    2 +-
 kernel/trace/trace_functions_graph.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/ftrace.c +1 -1
index 3925ec0..40960c2 100644
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = sched_clock();
 
 	if (ftrace_push_return_trace(old, calltime,
 				self_addr, &trace.depth) == -EBUSY) {
kernel/trace/trace_functions_graph.c +1 -1
index 2461732..c5038f4 100644
@@ -112,7 +112,7 @@ unsigned long ftrace_return_to_handler(void)
 	unsigned long ret;
 
 	ftrace_pop_return_trace(&trace, &ret);
-	trace.rettime = cpu_clock(raw_smp_processor_id());
+	trace.rettime = sched_clock();
 	ftrace_graph_return(&trace);
 
 	if (unlikely(!ret)) {
Signature · 1 line
1.6.1
#1 lore raw
Quoted 42 lines · Frederic Weisbecker

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

---
 arch/x86/kernel/ftrace.c             |    2 +-
 kernel/trace/trace_functions_graph.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/ftrace.c +1 -1
index 3925ec0..40960c2 100644
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = sched_clock();
 
 	if (ftrace_push_return_trace(old, calltime,
 				self_addr, &trace.depth) == -EBUSY) {
kernel/trace/trace_functions_graph.c +1 -1
index 2461732..c5038f4 100644
@@ -112,7 +112,7 @@ unsigned long ftrace_return_to_handler(void)
 	unsigned long ret;
 
 	ftrace_pop_return_trace(&trace, &ret);
-	trace.rettime = cpu_clock(raw_smp_processor_id());
+	trace.rettime = sched_clock();
 	ftrace_graph_return(&trace);
 
 	if (unlikely(!ret)) {
Signature · 1 line
1.6.1

Note that these two patches solve the function graph tracer hangs you reported. But there are still sometimes large latencies (which can reach 4 seconds) and the reason seems to me obviously come from the timer interrupt tracing. If you look at the following timer interrupt trace:

 0)   ==========> |
 0)               |          smp_apic_timer_interrupt() {
 0)               |            ack_APIC_irq() {
 0)               |              apic_write() {
 0)   3.597 us    |                native_apic_mem_write();
 0) + 13.203 us   |              }
 0) + 22.573 us   |            }
 0)               |            irq_enter() {
 0)   3.749 us    |              idle_cpu();
 0) + 14.016 us   |            }
 0)               |            tick_handle_periodic() {
 0)               |              tick_periodic() {
 0)   8.882 us    |                _spin_lock();
 0)               |                do_timer() {
 0)               |                  update_wall_time() {
 0)   3.720 us    |                    clocksource_read();
 0)   3.621 us    |                    update_xtime_cache();
 0)               |                    clocksource_get_next() {
 0)   6.355 us    |                      _spin_lock_irqsave();
 0)   5.862 us    |                      _spin_unlock_irqrestore();
 0) + 28.665 us   |                    }
 0) + 60.193 us   |                  }
 0) + 70.076 us   |                }
 0)   4.299 us    |                _spin_unlock();
 0)               |                update_process_times() {
 0)               |                  account_process_tick() {
 0)   4.603 us    |                    account_system_time();
 0) + 14.154 us   |                  }
 0)               |                  run_local_timers() {
 0)               |                    hrtimer_run_queues() {
 0)   3.784 us    |                      current_kernel_time();
 0)               |                      timespec_to_ktime() {
 0)   3.360 us    |                        ktime_set();
 0) + 12.532 us   |                      }
 0)               |                      timespec_to_ktime() {
 0)   3.216 us    |                        ktime_set();
 0) + 12.342 us   |                      }
 0)   6.177 us    |                      _spin_lock();
 0)   4.258 us    |                      _spin_unlock();
 0) + 74.455 us   |                    }
 0)               |                    raise_softirq() {
 0)               |                      raise_softirq_irqoff() {
 0) + 13.328 us   |                    }
 0)               |                    softlockup_tick() {
 0)   6.376 us    |                      get_timestamp();
 0) + 16.665 us   |                    }
 0) ! 126.019 us  |                  }
 0)               |                  rcu_pending() {
 0)               |                    __rcu_pending() {
 0)   3.753 us    |                      cpumask_check();
 0) + 14.203 us   |                    }
 0) + 23.865 us   |                  }
 0)               |                  rcu_check_callbacks() {
 0)   3.552 us    |                    idle_cpu();
 0)               |                    raise_rcu_softirq() {
 0)               |                      raise_softirq() {
 0)   3.340 us    |                        raise_softirq_irqoff();
 0) + 13.029 us   |                      }
 0) + 22.580 us   |                    }
 0) + 41.989 us   |                  }
 0)   3.597 us    |                  printk_tick();
 0)               |                  scheduler_tick() {
 0)               |                    ktime_get() {
 0)               |                      ktime_get_ts() {
 0)               |                        getnstimeofday() {
 0)   3.742 us    |                          clocksource_read();
 0) + 14.141 us   |                        }
 0)   3.291 us    |                        set_normalized_timespec();
 0) + 75.737 us   |                      }
 0)               |                      timespec_to_ktime() {
 0)   3.122 us    |                        ktime_set();
 0) + 12.275 us   |                      }
 0) ! 103.588 us  |                    }
 0)   6.814 us    |                    _spin_lock();
 0)   4.347 us    |                    update_rq_clock();
 0)               |                    task_tick_fair() {
 0)               |                      update_curr() {
 0)   3.280 us    |                        calc_delta_fair();
 0)               |                        update_min_vruntime() {
 0)   3.164 us    |                          max_vruntime();
 0) + 12.409 us   |                        }
 0)   3.415 us    |                        account_group_exec_runtime();
 0) + 40.947 us   |                      }
 0) + 50.545 us   |                    }
 0)               |                    perf_counter_task_tick() {
 0)               |                      perf_counter_task_sched_out() {
 0)               |                        __perf_counter_sched_out() {
 0)   6.940 us    |                          _spin_lock();
 0)   4.804 us    |                          _spin_unlock();
 0) + 28.069 us   |                        }
 0) + 37.414 us   |                      }
 0)               |                      perf_counter_task_sched_in() {
 0)   5.857 us    |                        _spin_lock();
 0)   4.405 us    |                        _spin_unlock();
 0) + 26.177 us   |                      }
 0) + 78.616 us   |                    }
 0)   4.018 us    |                    _spin_unlock();
 0)   3.187 us    |                    idle_cpu();
 0) ! 299.382 us  |                  }
 0)               |                  run_posix_cpu_timers() {
 0)   3.300 us    |                    task_cputime_zero();
 0)   3.256 us    |                    task_cputime_zero();
 0) + 21.977 us   |                  }
 0) ! 577.188 us  |                }
 0)   3.425 us    |                profile_tick();
 0) ! 698.681 us  |              }
 0) ! 707.982 us  |            }
 0)   3.275 us    |            perf_counter_unthrottle();
 0)               |            irq_exit() {
 0)               |              do_softirq() {
 0)               |                __do_softirq() {
 0)   4.372 us    |                  __local_bh_disable();
 0)               |                  run_timer_softirq() {
 0)   3.677 us    |                    hrtimer_run_pending();
 0)   7.721 us    |                    _spin_lock_irq();
 0)   3.441 us    |                    list_replace_init();
 0)   5.117 us    |                    _spin_unlock_irq();
 0) + 49.610 us   |                  }
 0)               |                  rcu_process_callbacks() {
 0)   5.340 us    |                    __rcu_process_callbacks();
 0)   3.695 us    |                    __rcu_process_callbacks();
 0) + 25.576 us   |                  }
 0)   4.666 us    |                  _local_bh_enable();
 0) ! 115.634 us  |                }
 0) ! 126.637 us  |              }
 0) ! 136.176 us  |            }
 0) ! 918.236 us  |          }
 0)   <========== |
 0) ! 1392.029 us |        }

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

#2 lore raw
Quoted 11 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

#3 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging... :-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

#4 lore raw
Quoted 40 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

Quoted 8 lines

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging...
:-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

yeah. perfcounters and KernelTop might not work on a PII CPU out of box though.

But hacking perfcounters and looking at perfstat/kerneltop output is serious amount of fun so if you are interested you could try to implement support for it. Do you have any box where perfcounters work? (that would be Core2 Intel boxes or pretty much any AMD box)

You could have a look at how oprofile works on your box - the code for PII CPUs should be in arch/x86/oprofile/op_model_ppro.c.

There's also hardcoded support for a single perfcounter in the nmi_watchdog=2 code, in arch/x86/kernel/cpu/perfctr-watchdog.c, for pretty much any x86 CPU that has a PMU.

Plus there's also the CPU documentation on Intel's site. It's quite well written and pretty well structured. The URL for the CPU's PMU ("Performance Monitoring") should be:

http://download.intel.com/design/processor/manuals/253669.pdf

As a last resort ;-)

Ingo

#9 lore raw
In reply to Frederic Weisbecker
Commit-ID:  bf467399b5f6301e7a6dc21460e66b1f40e16c7e
Gitweb:     http://git.kernel.org/tip/bf467399b5f6301e7a6dc21460e66b1f40e16c7e
Author:     "Frederic Weisbecker" <[email protected]>
AuthorDate: Thu, 5 Mar 2009 01:49:22 +0100
Commit:     Ingo Molnar <[email protected]>
CommitDate: Thu, 5 Mar 2009 12:01:52 +0100

tracing/function-graph-tracer: use the more lightweight local clock

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

Cc Steven Rostedt <[email protected]>
Cc Peter Zijlstra <[email protected]>

LKML-Reference: <[email protected]>

---
 arch/x86/kernel/ftrace.c             |    2 +-
 kernel/trace/trace_functions_graph.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/ftrace.c +1 -1
index 3925ec0..a85da17 100644
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = trace_clock_local();
 
 	if (ftrace_push_return_trace(old, calltime,
 				self_addr, &trace.depth) == -EBUSY) {
kernel/trace/trace_functions_graph.c +1 -1
index c009553..e527f2f 100644
@@ -112,7 +112,7 @@ unsigned long ftrace_return_to_handler(void)
 	unsigned long ret;
 
 	ftrace_pop_return_trace(&trace, &ret);
-	trace.rettime = cpu_clock(raw_smp_processor_id());
+	trace.rettime = trace_clock_local();
 	ftrace_graph_return(&trace);
 
 	if (unlikely(!ret)) {
#10 lore raw
In reply to Frederic Weisbecker
Quoted 18 lines · Frederic Weisbecker

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

---
 arch/x86/kernel/ftrace.c             |    2 +-
 kernel/trace/trace_functions_graph.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/ftrace.c +0 -0
index 3925ec0..40960c2 100644

arch/x86/kernel/ftrace.c void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr) · line 439

Quoted 5 lines
arch/x86/kernel/ftrace.c +1 -1
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = sched_clock();

I changed this to trace_clock_local() :-)

(btw, we should turn trace_clock_local() into an inline function)

There will be one problem though: function trace elapsed time measurements across idle. Those are not correctly measured by the local clock.

Ingo

#5 lore raw
In reply to Ingo Molnar
Quoted 73 lines · Ingo Molnar
Quoted 40 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

Quoted 8 lines

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging...
:-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

yeah. perfcounters and KernelTop might not work on a PII CPU out of box though.

But hacking perfcounters and looking at perfstat/kerneltop output is serious amount of fun so if you are interested you could try to implement support for it. Do you have any box where perfcounters work? (that would be Core2 Intel boxes or pretty much any AMD box)

You could have a look at how oprofile works on your box - the code for PII CPUs should be in arch/x86/oprofile/op_model_ppro.c.

There's also hardcoded support for a single perfcounter in the nmi_watchdog=2 code, in arch/x86/kernel/cpu/perfctr-watchdog.c, for pretty much any x86 CPU that has a PMU.

Plus there's also the CPU documentation on Intel's site. It's quite well written and pretty well structured. The URL for the CPU's PMU ("Performance Monitoring") should be:

Quoted 5 lines

http://download.intel.com/design/processor/manuals/253669.pdf

As a last resort ;-)

Ingo

Ah yes, That could be fun!
So, by reading your description, it should work on my labtop I guess?

-> Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz

Anyway, I will give it a try and see what I can do. Thanks for the pointers.

#13 lore raw
In reply to Frederic Weisbecker
Commit-ID:  0012693ad4f636c720fed3802027f9427962f540
Gitweb:     http://git.kernel.org/tip/0012693ad4f636c720fed3802027f9427962f540
Author:     "Frederic Weisbecker" <[email protected]>
AuthorDate: Thu, 5 Mar 2009 01:49:22 +0100
Commit:     Ingo Molnar <[email protected]>
CommitDate: Thu, 5 Mar 2009 12:14:41 +0100

tracing/function-graph-tracer: use the more lightweight local clock

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

Cc Steven Rostedt <[email protected]>
Cc Peter Zijlstra <[email protected]>

LKML-Reference: <[email protected]>

---
 arch/x86/kernel/ftrace.c             |    2 +-
 include/linux/ftrace.h               |   13 +++++++------
 kernel/trace/trace_functions_graph.c |    2 +-
 3 files changed, 9 insertions(+), 8 deletions(-)
arch/x86/kernel/ftrace.c +1 -1
index 3925ec0..a85da17 100644
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = trace_clock_local();
 
 	if (ftrace_push_return_trace(old, calltime,
 				self_addr, &trace.depth) == -EBUSY) {
include/linux/ftrace.h +7 -6
index 1f69ac7..6ea62ac 100644
@@ -1,15 +1,16 @@
 #ifndef _LINUX_FTRACE_H
 #define _LINUX_FTRACE_H
 
-#include <linux/linkage.h>
-#include <linux/fs.h>
-#include <linux/ktime.h>
-#include <linux/init.h>
-#include <linux/types.h>
-#include <linux/module.h>
+#include <linux/trace_clock.h>
 #include <linux/kallsyms.h>
+#include <linux/linkage.h>
 #include <linux/bitops.h>
+#include <linux/module.h>
+#include <linux/ktime.h>
 #include <linux/sched.h>
+#include <linux/types.h>
+#include <linux/init.h>
+#include <linux/fs.h>
 
 #include <asm/ftrace.h>
 
kernel/trace/trace_functions_graph.c +1 -1
index c009553..e527f2f 100644
@@ -112,7 +112,7 @@ unsigned long ftrace_return_to_handler(void)
 	unsigned long ret;
 
 	ftrace_pop_return_trace(&trace, &ret);
-	trace.rettime = cpu_clock(raw_smp_processor_id());
+	trace.rettime = trace_clock_local();
 	ftrace_graph_return(&trace);
 
 	if (unlikely(!ret)) {
#11 lore raw
In reply to Ingo Molnar
Quoted 24 lines · Ingo Molnar
Quoted 21 lines · Frederic Weisbecker

Impact: decrease hangs risks with the graph tracer on slow systems

Since the function graph tracer can spend too much time on timer interrupts, it's better now to use the more lightweight local clock. Anyway, the function graph traces are more reliable on a per cpu trace.

---
 arch/x86/kernel/ftrace.c             |    2 +-
 kernel/trace/trace_functions_graph.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/ftrace.c +0 -0
index 3925ec0..40960c2 100644
@@ -436,7 +436,7 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr)
 		return;
 	}
 
Quoted 7 lines

arch/x86/kernel/ftrace.c void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr) · line 439

Quoted 2 lines
arch/x86/kernel/ftrace.c +1 -1
-	calltime = cpu_clock(raw_smp_processor_id());
+	calltime = sched_clock();

I changed this to trace_clock_local() :-)

(btw, we should turn trace_clock_local() into an inline function)

Ok.

Quoted 3 lines

There will be one problem though: function trace elapsed time measurements across idle. Those are not correctly measured by the local clock.

Oh, why? If I'm not wrong it uses sched_clock() which uses tsc on x86. And this register always goes forward at the same rate. Unless some Cpu decrease their frequency while beeing idle for some time and then become unreliable?

Quoted 1 line

Ingo

#12 lore raw
Quoted 8 lines · Frederic Weisbecker
Quoted 3 lines

There will be one problem though: function trace elapsed time measurements across idle. Those are not correctly measured by the local clock.

Oh, why? If I'm not wrong it uses sched_clock() which uses tsc on x86. And this register always goes forward at the same rate. Unless some Cpu decrease their frequency while beeing idle for some time and then become unreliable?

TSC stops in idle on many CPUs.

Ingo

#6 lore raw
In reply to Frederic Weisbecker
Quoted 76 lines · Frederic Weisbecker
Quoted 75 lines · Ingo Molnar
Quoted 40 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

Quoted 8 lines

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging...
:-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

yeah. perfcounters and KernelTop might not work on a PII CPU out of box though.

But hacking perfcounters and looking at perfstat/kerneltop output is serious amount of fun so if you are interested you could try to implement support for it. Do you have any box where perfcounters work? (that would be Core2 Intel boxes or pretty much any AMD box)

You could have a look at how oprofile works on your box - the code for PII CPUs should be in arch/x86/oprofile/op_model_ppro.c.

There's also hardcoded support for a single perfcounter in the nmi_watchdog=2 code, in arch/x86/kernel/cpu/perfctr-watchdog.c, for pretty much any x86 CPU that has a PMU.

Plus there's also the CPU documentation on Intel's site. It's quite well written and pretty well structured. The URL for the CPU's PMU ("Performance Monitoring") should be:

http://download.intel.com/design/processor/manuals/253669.pdf

Quoted 8 lines
Quoted 3 lines

As a last resort ;-)

Ingo

Ah yes, That could be fun!
So, by reading your description, it should work on my labtop I guess?

-> Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz

Yeah, should work fine there - so that should be a good reference point to start off. Let me know if you see any bugs/problems.

Quoted 2 lines

Anyway, I will give it a try and see what I can do. Thanks for the pointers.

You are welcome.

Ingo

#7 lore raw
Quoted 92 lines · Ingo Molnar
Quoted 76 lines · Frederic Weisbecker
Quoted 75 lines · Ingo Molnar
Quoted 40 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

Quoted 8 lines

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging...
:-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

yeah. perfcounters and KernelTop might not work on a PII CPU out of box though.

But hacking perfcounters and looking at perfstat/kerneltop output is serious amount of fun so if you are interested you could try to implement support for it. Do you have any box where perfcounters work? (that would be Core2 Intel boxes or pretty much any AMD box)

You could have a look at how oprofile works on your box - the code for PII CPUs should be in arch/x86/oprofile/op_model_ppro.c.

There's also hardcoded support for a single perfcounter in the nmi_watchdog=2 code, in arch/x86/kernel/cpu/perfctr-watchdog.c, for pretty much any x86 CPU that has a PMU.

Plus there's also the CPU documentation on Intel's site. It's quite well written and pretty well structured. The URL for the CPU's PMU ("Performance Monitoring") should be:

http://download.intel.com/design/processor/manuals/253669.pdf

Quoted 8 lines
Quoted 3 lines

As a last resort ;-)

Ingo

Ah yes, That could be fun!
So, by reading your description, it should work on my labtop I guess?

-> Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz

Yeah, should work fine there - so that should be a good reference point to start off. Let me know if you see any bugs/problems.

Quoted 6 lines
Quoted 2 lines

Anyway, I will give it a try and see what I can do. Thanks for the pointers.

You are welcome.

Ingo

Ho, that's impressive:

$ ./perfstat /bin/echo 1
1

Performance counter stats for '/bin/echo':

5.681909 task clock ticks (msecs)

2 CPU migrations (events)
9 context switches (events)
422 pagefaults (events)
4986950 CPU cycles (events)
4870587 instructions (events)
62881 cache references (events)
4882 cache misses (events)

Wall-clock time elapsed: 9.046821 msecs

So I guess that for it to be useful on fine grained profiling, it's better to include the percounters syscalls inside the application to profile a single function for example?

I've parsed a bit the intel documentation and perfcounters source code. It looks like the current implementation for intel and amd share enough similar properties so that they have been factorized in a single file.

This similar property is the use of an msr, while P6 family use the pmc.

Perhaps I can expand the struct pmc_x86_ops to guess whether we want to use rdmsr or rdpmc.

Hmm it seems the implementation would be enough different to deserve a new file for P6.
Will see...

#8 lore raw
Quoted 117 lines · Frederic Weisbecker
Quoted 92 lines · Ingo Molnar
Quoted 76 lines · Frederic Weisbecker
Quoted 75 lines · Ingo Molnar
Quoted 40 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker

It takes 1 ms to execute while tracing.
Considering my frequency is 250 Hz, it means 1/4 of the system is used on timer interrupt while tracing.

For now the hang is fixed, but not the awful latency. And I'm just too frightened to test it on 1000 Hz.

Quoted 6 lines
Quoted 4 lines

But I plan to add a kind of watchdog to check how many time we spent inside an interrupt while graph tracing.
By checking this time against the current Hz value, I could decide to abort the tracing for all irq.

That would basically render the thing useless :-(

It would be only for slow machines :-)
I'm talking about something that happened on a Pentium II.

Quoted 2 lines

Is it specifically function_graph that is so expensive? If so, is that because of the function exit hook?

Yes, specifically the function_graph, the function tracer is not concerned. The function graph tracer takes more than double overhead compared to the function tracer.

Usually the function tracer hooks directly the the function that insert the event, it's pretty straightforward.

The function graph does much more work:

entry: basic checks, take the time, push the infos on the stack, insert an event
on the ring-buffer, hook the return value.
return: pop the infos from stack, insert an event on the ring-buffer, jump
to the original caller.

Quoted 8 lines

It has a high cost... which makes me sad because I plan to port it in on Arm and I fear the little Arm boad I recently purshased will not let me trace the interrupts without hanging...
:-(

I guess I should start thinking on some optimizations, perhaps using perfcounter?

yeah. perfcounters and KernelTop might not work on a PII CPU out of box though.

But hacking perfcounters and looking at perfstat/kerneltop output is serious amount of fun so if you are interested you could try to implement support for it. Do you have any box where perfcounters work? (that would be Core2 Intel boxes or pretty much any AMD box)

You could have a look at how oprofile works on your box - the code for PII CPUs should be in arch/x86/oprofile/op_model_ppro.c.

There's also hardcoded support for a single perfcounter in the nmi_watchdog=2 code, in arch/x86/kernel/cpu/perfctr-watchdog.c, for pretty much any x86 CPU that has a PMU.

Plus there's also the CPU documentation on Intel's site. It's quite well written and pretty well structured. The URL for the CPU's PMU ("Performance Monitoring") should be:

http://download.intel.com/design/processor/manuals/253669.pdf

Quoted 8 lines
Quoted 3 lines

As a last resort ;-)

Ingo

Ah yes, That could be fun!
So, by reading your description, it should work on my labtop I guess?

-> Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz

Yeah, should work fine there - so that should be a good reference point to start off. Let me know if you see any bugs/problems.

Quoted 6 lines
Quoted 2 lines

Anyway, I will give it a try and see what I can do. Thanks for the pointers.

You are welcome.

Ingo

Ho, that's impressive:

$ ./perfstat /bin/echo 1
1

Performance counter stats for '/bin/echo':

5.681909 task clock ticks (msecs)

2 CPU migrations (events)
9 context switches (events)
422 pagefaults (events)
4986950 CPU cycles (events)
4870587 instructions (events)
62881 cache references (events)
4882 cache misses (events)

Quoted 5 lines

Wall-clock time elapsed: 9.046821 msecs

So I guess that for it to be useful on fine grained profiling, it's better to include the percounters syscalls inside the application to profile a single function for example?

Correct, it takes 4.8 million instructions to execute a simple shell script on Linux ...

For your purposes to profile the function tracer, kerneltop is a much more natural choice.

perfstat is best when used for performance/behavioral comparisons of a given workload. It gives a flat number for the metrics it follows, that does not give you much of an idea about exactly where those extra instructions and the extra cost comes from.

Quoted 5 lines

I've parsed a bit the intel documentation and perfcounters source code. It looks like the current implementation for intel and amd share enough similar properties so that they have been factorized in a single file.

Quoted 8 lines

This similar property is the use of an msr, while P6 family use the pmc.

Perhaps I can expand the struct pmc_x86_ops to guess whether we want to use rdmsr or rdpmc.

Hmm it seems the implementation would be enough different to deserve a new file for P6. Will see...

Yeah, if there's not enough similarities then you can go for a separate perf_counters_p6.c file.

Ingo