Updated rtc-debug patch
Andrew, this is an updated version of your rtc-debug patch with some minor changes:
- Histogram bins are 1 usec instead of 10
- Histogram displays one bin per line rather than 5. You may prefer the other way, but I think this makes it easier to gnuplot the results.
- Line numbers tweaked to apply cleanly
This should make it easy to measure the much improved latencies we are getting now. I have never posted a kernel patch before, apologies if I got the 'Signed-Off-By' wrong.
For anyone unfamiliar this patch is designed to be used with amlat:
http://www.zip.com.au/~akpm/linux/schedlat.html#amlat
amlat itself does not need to be modified to use with recent kernels.
Example output with the voluntary-preempt-S0:
rtc histogram:
31 3152
32 2654
33 2895
34 1673934
35 631584
36 59189
37 62070
38 57622
39 40790
40 28464
41 17503
42 8406
43 3457
44 1586
45 1194
46 1175
47 520
48 439
49 457
50 493
51 413
52 329
53 258
54 4041
55 1638
56 659
57 23485
58 13043
59 41537
60 31171
61 1475
62 1208
63 1318
64 1303
65 1424
66 1515
67 1451
68 1208
69 1053
70 919
71 696
72 440
73 311
74 192
75 114
76 83
77 89
78 59
79 58
80 46
81 60
82 32
83 19
84 27
85 16
86 8
87 6
88 10
89 7
90 8
91 5
92 6
93 6
94 2
95 2
96 1
97 3
98 1
99 1
100 1
101 2
105 3
106 1
107 1
108 3
109 2
110 1
116 1
117 1
132 1
135 1
136 1
Total samples: 2729359
arch/i386/kernel/traps.c renamed from arch/i386/kernel/traps.c~rtc-debug +39 -2
@@ -197,14 +197,51 @@ void show_stack(struct task_struct *taskfor (i =0 ; i < kstack_depth_to_print; i++) {if (kstack_end (stack))break ; -if (i && ((i %8 ) ==0 )) -printk ("\n " ); +if (i && ((i %8 ) ==0 )) { +printk ("\n" ); +printk (" " ); + }printk ("%08lx " , *stack++); }printk ("\nCall Trace:\n" );show_trace (task, esp); } +static void show_trace_local (void ) +{ +printk ("CPU %d:\n" ,smp_processor_id ()); +show_trace (0 ,0 ); +} + +#ifdef CONFIG_SMP +static atomic_t trace_cpu; + +static void show_trace_one (void *dummy) +{ +while (atomic_read (&trace_cpu) !=smp_processor_id ()) + ; +show_trace_local (); +atomic_inc (&trace_cpu); +while (atomic_read (&trace_cpu) !=num_online_cpus ()) + ; +} + +void show_trace_smp (void ) +{ +atomic_set (&trace_cpu,0 ); +smp_call_function (show_trace_one,0 ,1 ,0 ); +show_trace_one (0 ); +} + +#else + +void show_trace_smp (void ) +{ +show_trace_local (); +} + +#endif +/* * The architecture-independent dump_stack generator */
drivers/char/rtc.c renamed from drivers/char/rtc.c~rtc-debug +162 -5
227 lines not on this page · raw message
_
Quoted 3 lines · Lee Revell
Andrew, this is an updated version of your rtc-debug patch with some minor changes:
Thanks Lee. I was just talking about doing this today because now that the latencies are getting better I wanted to be able to generate some nice graphs to show some interested folks.
kr
Quoted 9 lines · K.R. Foley
Quoted 3 lines · Lee Revell
Andrew, this is an updated version of your rtc-debug patch with some minor changes:
Thanks Lee. I was just talking about doing this today because now that the latencies are getting better I wanted to be able to generate some nice graphs to show some interested folks.
I think it is a pretty good sign that we are having to "zoom in" these tools by 10-1000x to analyze the remaining data.
Here are some of my results so far:
http://krustophenia.net/testresults.php?dataset=2.6.9-rc1-bk12-S0
We are getting pretty close to the point where we can analyze those graphs and be able to say the spike at X is caused by Y code path. For example the spikes above 100 usecs are certainly caused by the netif_receive_skb path.
It might be useful is for the latency tracer to have a preempt_min_latency as well as a preempt_max_latency setting. Then, if you were to hypothesize that the spike between 55 and 60 usecs is caused by a certain code path, you could set the window and verify this.
Here's a modified version the hist-to-jpg from Andrew's amlat package. It to produce graphs in the same format as the right hand graph from my testresults.php script (PNG format, data style boxes, and logarithmic scaling on the Y axis).
http://krustophenia.net/hist-to-jpg.sh.txt
Here's the PHP script:
http://krustophenia.net/testresults.phps
I would like to put together a collection of latency profiling tools for the 2.6 kernel. Mostly because I am afraid someone will post a link to my graphs on slashdot and knock me off the internet for a day.
If you have anything you find useful, send it to me and I will put a page together. It would be nice to have some standard tools we can use to compare systems.
Lee