documentation for basic guide to profiling
PATCH 1/1 documentation for basic guide to profiling
I was trying to write some simple docs on how to do profiling for people to use for really basic stuff. I got it all wrong, but John's kindly corrected it ;-) Andrew asked me to do this as a patch for the documentation directory ... feedback would be much appreciated (yes, it's oversimplified - it's meant to be).
Documentation/basic_profiling.txt +44 -0
@@ -0,0 +1,44 @@ +These instructions are deliberately very basic. If you want something clever, +go read the real docs ;-) Please don't add more stuff, but feel free to +correct my mistakes ;-) ([email protected]) +Thanks to John Levon and Dave Hansen for help writing this. + +<test> is the thing you're trying to measure. +Make sure you have the correct System.map / vmlinux referenced! +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel +to copy config files, system.map, vmlinux to /boot. + +Readprofile +----------- +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/ +add "profile=2" to the kernel command line. + +clear echo 2 > /proc/profile + <test> +dump output readprofile -m /boot/System.map > catured_profile + +Oprofile +-------- +get source (I use 0.5) from http://oprofile.sourceforge.net/ +add "poll=idle" to the kernel command line +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel +./configure --with-kernel-support +make install + +One time setup (pick appropriate one for your CPU): +P3 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000 +Athalon opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=RETIRED_INSNS --ctr0-count=100000 +P4 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=GLOBAL_POWER_EVENTS \ + --ctr0-unit-mask=1 --ctr0-count=100000 + +start daemon opcontrol --start-daemon +clear opcontrol --reset +start opcontrol --start + <test> +stop opcontrol --stop +dump output oprofpp -dl -i /boot/vmlinux > output_file +
Review
Quoted 4 lines · Martin J. Bligh
...
(quoted diff) +2
-0+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from +ftp://ftp.kernel.org/pub/linux/ people/mbligh/tools/readprofile/
kernel/
Documentation/basic_profiling.txt line 19
Quoted 5 lines
Documentation/basic_profiling.txt +5
-0+add "profile=2" to the kernel command line. + +clear echo 2 > /proc/profile + <test> +dump output readprofile -m /boot/System.map > catured_profile
util-linux-2.11z (at least) has the fixed readprofile.
Of course, installing standard util-linux turns your boot process into a complete mess because vendors have added incompatible features to their versions. But it seems to struggle through.
Documentation/basic_profiling.txt line 32
Quoted 12 lines
Documentation/basic_profiling.txt +12
-0+Oprofile +-------- +get source (I use 0.5) from http://oprofile.sourceforge.net/ +add "poll=idle" to the kernel command line +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel +./configure --with-kernel-support +make install + +One time setup (pick appropriate one for your CPU): +P3 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000 +Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
Athlon
Documentation/basic_profiling.txt line 44
Quoted 12 lines
Documentation/basic_profiling.txt +12
-0+ --ctr0-event=RETIRED_INSNS --ctr0-count=100000 +P4 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=GLOBAL_POWER_EVENTS \ + --ctr0-unit-mask=1 --ctr0-count=100000 + +start daemon opcontrol --start-daemon +clear opcontrol --reset +start opcontrol --start + <test> +stop opcontrol --stop +dump output oprofpp -dl -i /boot/vmlinux > output_file +
OK. Might be worth adding a pointer to this in REPORTING-BUGS, but nobody reads that anyway.
Quoted 2 lines · Andrew Morton
Quoted 1 line
+Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
Athlon
Also x86-64 as of 0.5.1
Dave
Quoted 6 lines · Dave Jones
Quoted 2 lines · Andrew Morton
Quoted 1 line
+Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
Athlon
Also x86-64 as of 0.5.1
Alpha is supported too... (as least I saw the kernel part go in)
Quoted 2 lines
Alpha is supported too... (as least I saw the kernel part go in)
sparc64 and ppc64 too, but only timer tick profiling so far.
Anton
Quoted 3 lines
Quoted 1 line
Alpha is supported too... (as least I saw the kernel part go in)
sparc64 and ppc64 too, but only timer tick profiling so far.
I need the magic incantation for oprofile for these then ...
Actually, better still, we need a wrapper script that works out this from /proc/cpuinfo and auto-sets it up for you, if someone who knows enough about different Pentium types knows how ... I'm happy to go write it if it's easy to detect ...
My laptop says:
model name : Pentium III (Coppermine)
but a list of pattern matches from someone (or list of people) who know how this works would be very helpful (ie matching up cpuinfo to magic incantation to oprofile).
M.
Quoted 11 lines · Martin J. Bligh
Quoted 3 lines
Quoted 1 line
Alpha is supported too... (as least I saw the kernel part go in)
sparc64 and ppc64 too, but only timer tick profiling so far.
I need the magic incantation for oprofile for these then ...
Actually, better still, we need a wrapper script that works out this from
/proc/cpuinfoand auto-sets it up for you, if someone who knows enough about different Pentium types knows how ... I'm happy to go write it if it's easy to detect ...
Quoted 7 lines
My laptop says:
model name : Pentium III (Coppermine)
but a list of pattern matches from someone (or list of people) who know how this works would be very helpful (ie matching up cpuinfo to magic incantation to oprofile).
Actually, oprofile already does this somehow. opcontrol --list-events will tell you the perf counters for your particular CPU. When my scripts are deciding whether the machine is P3/4, they grep through this list.
I think the proper way to do this is in oprofile itself. John mentioned earlier in this thread that they have considered doing default events, but there may be a better way to do it.
Signature · 2 lines
Dave Hansen [email protected]
OK, fixed a couple more things (spelling, and pointer to new tools). I don't really want to add a list of 200 arches, but I'll do a script with any info people email me the cpuinfo output and correct oprofile setting for ... I'll do that for the existing ones instead of what's there now as well.
Documentation/basic_profiling.txt +48 -0
@@ -0,0 +1,48 @@ +These instructions are deliberately very basic. If you want something clever, +go read the real docs ;-) Please don't add more stuff, but feel free to +correct my mistakes ;-) ([email protected]) +Thanks to John Levon and Dave Hansen for help writing this. + +<test> is the thing you're trying to measure. +Make sure you have the correct System.map / vmlinux referenced! +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel +to copy config files, system.map, vmlinux to /boot. + +Readprofile +----------- +You need a fixed readprofile command for 2.5 ... either get hold of +a current version from: +http://www.kernel.org/pub/linux/utils/util-linux/ +or get readprofile binary fixed for 2.5 / akpm's 2.5 patch from +ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/readprofile/ + +Add "profile=2" to the kernel command line. + +clear readprofile -r + <test> +dump output readprofile -m /boot/System.map > captured_profile + +Oprofile +-------- +get source (I use 0.5) from http://oprofile.sourceforge.net/ +add "poll=idle" to the kernel command line +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel +./configure --with-kernel-support +make install + +One time setup (pick appropriate one for your CPU): +P3 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000 +Athlon/x86-64 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=RETIRED_INSNS --ctr0-count=100000 +P4 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=GLOBAL_POWER_EVENTS \ + --ctr0-unit-mask=1 --ctr0-count=100000 + +start daemon opcontrol --start-daemon +clear opcontrol --reset +start opcontrol --start + <test> +stop opcontrol --stop +dump output oprofpp -dl -i /boot/vmlinux > output_file +
Quoted 1 line · Martin J. Bligh
+start daemon opcontrol --start-daemon
--start implies starting the daemon if it isn't started already.
Quoted 1 line
+stop opcontrol --stop
--stop unsupported. use "--shutdown"
Quoted 1 line
+dump output oprofpp -dl -i
/boot/vmlinux>output_file
opcontrol --dump
op_time -l is also worth adding to this doc imo.
Dave
Quoted 8 lines · Dave Jones
Quoted 1 line
+start daemon opcontrol --start-daemon
--start implies starting the daemon if it isn't started already.
Quoted 1 line
+stop opcontrol --stop
--stop unsupported. use "--shutdown"
Stop running 2.4 ! :)
--stop works on 2.5 only ...
regards
john
Quoted 6 lines · John Levon
Quoted 2 lines
Quoted 1 line
+stop opcontrol --stop
--stop unsupported. use "--shutdown"
Stop running 2.4 ! :)
--stop works on 2.5 only ...
That seems quite confusing. Why not make --stop an alias for --shutdown if running on 2.4 ?
Dave
Quoted 2 lines
--start implies starting the daemon if it isn't started already.
I think John suggested to do those seperately, to minimise overhead or something.
Quoted 5 lines
Quoted 1 line
+dump output oprofpp -dl -i
/boot/vmlinux>output_fileopcontrol --dump
op_time-l is also worth adding to this doc imo.
OK, will add.
M.
Quoted 5 lines · Martin J. Bligh
Quoted 2 lines
--start implies starting the daemon if it isn't started already.
I think John suggested to do those seperately, to minimise overhead or something.
On a heavily loaded system, --start can take quite a while if the daemon isn't already running. Before starting my machine-killing benchmark (which I want to profile), I use --start-daemon which gets most of the work out of the way. The subsequent --start, which occurs while the benchmark is already steaming along, will happen quickly and with much less impact to the benchmark results.
Signature · 2 lines
Dave Hansen [email protected]
Hi!
Documentation/basic_profiling.txt line 28
Quoted 4 lines
Documentation/basic_profiling.txt +4
-0+Oprofile +-------- +get source (I use 0.5) from http://oprofile.sourceforge.net/ +add "poll=idle" to the kernel command line
This should be idle=poll, AFAICS.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
On Fri, 28 Feb 2003 00:12:42 -0800
"Martin J. Bligh" <[email protected]> wrote:
| I was trying to write some simple docs on how to do profiling for people | to use for really basic stuff. I got it all wrong, but John's kindly | corrected it ;-) Andrew asked me to do this as a patch for the | documentation directory ... feedback would be much appreciated | (yes, it's oversimplified - it's meant to be). | | diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt | --- virgin/Documentation/basic_profiling.txt Wed Dec 31 16:00:00 1969 | +++ oprofile_doc/Documentation/basic_profiling.txt Fri Feb 28 00:05:59 2003 | @@ -0,0 +1,44 @@ | +These instructions are deliberately very basic. If you want something clever, | +go read the real docs ;-) Please don't add more stuff, but feel free to | +correct my mistakes ;-) ([email protected]) | +Thanks to John Levon and Dave Hansen for help writing this. | + | +<test> is the thing you're trying to measure. | +Make sure you have the correct System.map / vmlinux referenced! | +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel | +to copy config files, system.map, vmlinux to /boot. | + | +Readprofile | +----------- | +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from | +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/ | +add "profile=2" to the kernel command line.
These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 > /proc/profile
man page says to use "readprofile -r". Doesn't that still work?
| + <test>
| +dump output readprofile -m /boot/System.map > catured_profile
> captured_profile
--
~Randy
Quoted 4 lines
These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 >/proc/profile
man page says to use "readprofile -r". Doesn't that still work?
Dunno. I always have done the above ... have you been using -r with success? If so, I'll change it.
M.
Quoted 7 lines · Martin J. Bligh
Quoted 4 lines
These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 >/proc/profile
man page says to use "readprofile -r". Doesn't that still work?Dunno. I always have done the above ... have you been using -r with success? If so, I'll change it.
It's what I've always used.
Signature · 2 lines
Dave Hansen [email protected]
On Fri, 28 Feb 2003 10:13:51 -0800
Dave Hansen <[email protected]> wrote:
| Martin J. Bligh wrote: | >>These: ^------------v should be the same value (as you have it). | >> v | >>| +clear echo 2 > /proc/profile | >>man page says to use "readprofile -r". Doesn't that still work? | > | > Dunno. I always have done the above ... have you been using -r with | > success? If so, I'll change it. | | It's what I've always used.
Same here (-r).
--
~Randy
OK, fixed a couple of things ... thanks for the feedback everyone.
Documentation/basic_profiling.txt +44 -0
@@ -0,0 +1,44 @@ +These instructions are deliberately very basic. If you want something clever, +go read the real docs ;-) Please don't add more stuff, but feel free to +correct my mistakes ;-) ([email protected]) +Thanks to John Levon and Dave Hansen for help writing this. + +<test> is the thing you're trying to measure. +Make sure you have the correct System.map / vmlinux referenced! +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel +to copy config files, system.map, vmlinux to /boot. + +Readprofile +----------- +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/ +add "profile=2" to the kernel command line. + +clear readprofile -r + <test> +dump output readprofile -m /boot/System.map > catured_profile + +Oprofile +-------- +get source (I use 0.5) from http://oprofile.sourceforge.net/ +add "poll=idle" to the kernel command line +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel +./configure --with-kernel-support +make install + +One time setup (pick appropriate one for your CPU): +P3 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000 +Athlon/x86-64 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=RETIRED_INSNS --ctr0-count=100000 +P4 opcontrol --setup --vmlinux=/boot/vmlinux \ + --ctr0-event=GLOBAL_POWER_EVENTS \ + --ctr0-unit-mask=1 --ctr0-count=100000 + +start daemon opcontrol --start-daemon +clear opcontrol --reset +start opcontrol --start + <test> +stop opcontrol --stop +dump output oprofpp -dl -i /boot/vmlinux > output_file +
Documentation/basic_profiling.txt line 17
Quoted 7 lines · Martin J. Bligh
Documentation/basic_profiling.txt +7
-0+Readprofile +----------- +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/ +add "profile=2" to the kernel command line. + +clear echo 2 > /proc/profile
As far as I can see, the 2 is meaningless here. This should just be
echo > /proc/profile
(On SMP when writing sizeof(int) bytes, the value written is significant. But 1 or 2 is not sizeof(int).)
Andries
-----
Fragment of some notes:
<sect1>Profiling the kernel<p>
There are several facilities to see where the kernel spends its resources. A simple one is the profiling function, that stores the current EIP (instruction pointer) at each clock tick. <p>
Boot the kernel with command line option <tt>profile=2</tt> (or some other number instead of 2). This will cause
a file <tt>/proc/profile</tt> to be created.
The number given after <tt>profile=</tt> is the number of positions EIP is shifted right when profiling. So a large number gives a coarse profile.
The counters are reset by writing to <tt>/proc/profile</tt>. The utility <tt>readprofile</tt> will output statistics for you. It does not sort - you have to invoke <tt>sort</tt> explicitly. But given a memory map it will translate addresses to kernel symbols. <p>
See <tt>kernel/profile.c</tt> and <tt>fs/proc/proc_misc.c</tt> and <tt>readprofile(1)</tt>.
<p>
For example:
<verb>
# echo > /proc/profile ... # readprofile -m System.map-2.5.59 | sort -nr | head -2 510502 total 0.1534 508548 default_idle 10594.7500 </verb>