Skip to contentSkip to conversation

documentation for basic guide to profiling

PATCH In review Martin J. Bligh 10 participants 20 messages Feb 28, 2003 – Mar 3, 2003 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 documentation for basic guide to profiling

I was trying to write some simple docs on how to do profiling for people to use for really basic stuff. I got it all wrong, but John's kindly corrected it ;-) Andrew asked me to do this as a patch for the documentation directory ... feedback would be much appreciated (yes, it's oversimplified - it's meant to be).

Documentation/basic_profiling.txt +44 -0
diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear		echo 2 > /proc/profile
+		<test>
+dump output	readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+
#1 lore raw

Review

Quoted 4 lines · Martin J. Bligh

...

(quoted diff) +2 -0
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/       people/mbligh/tools/readprofile/

kernel/

Documentation/basic_profiling.txt line 19

Quoted 5 lines
Documentation/basic_profiling.txt +5 -0
+add "profile=2" to the kernel command line.
+
+clear		echo 2 > /proc/profile
+		<test>
+dump output	readprofile -m /boot/System.map > catured_profile

util-linux-2.11z (at least) has the fixed readprofile.

Of course, installing standard util-linux turns your boot process into a complete mess because vendors have added incompatible features to their versions. But it seems to struggle through.

Documentation/basic_profiling.txt line 32

Quoted 12 lines
Documentation/basic_profiling.txt +12 -0
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \

Athlon

Documentation/basic_profiling.txt line 44

Quoted 12 lines
Documentation/basic_profiling.txt +12 -0
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+

OK. Might be worth adding a pointer to this in REPORTING-BUGS, but nobody reads that anyway.

#2 lore raw
Quoted 2 lines · Andrew Morton
Quoted 1 line

+Athalon opcontrol --setup --vmlinux=/boot/vmlinux \

Athlon

Also x86-64 as of 0.5.1

Dave

#3 lore raw
Quoted 6 lines · Dave Jones
Quoted 2 lines · Andrew Morton
Quoted 1 line

+Athalon opcontrol --setup --vmlinux=/boot/vmlinux \

Athlon

Also x86-64 as of 0.5.1

Alpha is supported too... (as least I saw the kernel part go in)

#4 lore raw
Quoted 2 lines

Alpha is supported too... (as least I saw the kernel part go in)

sparc64 and ppc64 too, but only timer tick profiling so far.

Anton

#5 lore raw
Quoted 3 lines
Quoted 1 line

Alpha is supported too... (as least I saw the kernel part go in)

sparc64 and ppc64 too, but only timer tick profiling so far.

I need the magic incantation for oprofile for these then ...

Actually, better still, we need a wrapper script that works out this from /proc/cpuinfo and auto-sets it up for you, if someone who knows enough about different Pentium types knows how ... I'm happy to go write it if it's easy to detect ...

My laptop says:

model name : Pentium III (Coppermine)

but a list of pattern matches from someone (or list of people) who know how this works would be very helpful (ie matching up cpuinfo to magic incantation to oprofile).

M.

#6 lore raw
Quoted 11 lines · Martin J. Bligh
Quoted 3 lines
Quoted 1 line

Alpha is supported too... (as least I saw the kernel part go in)

sparc64 and ppc64 too, but only timer tick profiling so far.

I need the magic incantation for oprofile for these then ...

Actually, better still, we need a wrapper script that works out this from /proc/cpuinfo and auto-sets it up for you, if someone who knows enough about different Pentium types knows how ... I'm happy to go write it if it's easy to detect ...

Quoted 7 lines

My laptop says:

model name : Pentium III (Coppermine)

but a list of pattern matches from someone (or list of people) who know how this works would be very helpful (ie matching up cpuinfo to magic incantation to oprofile).

Actually, oprofile already does this somehow. opcontrol --list-events will tell you the perf counters for your particular CPU. When my scripts are deciding whether the machine is P3/4, they grep through this list.

I think the proper way to do this is in oprofile itself. John mentioned earlier in this thread that they have considered doing default events, but there may be a better way to do it.

Signature · 2 lines
Dave Hansen
[email protected]
#7 lore raw
In reply to Anton Blanchard

OK, fixed a couple more things (spelling, and pointer to new tools). I don't really want to add a list of 200 arches, but I'll do a script with any info people email me the cpuinfo output and correct oprofile setting for ... I'll do that for the existing ones instead of what's there now as well.

Documentation/basic_profiling.txt +48 -0
diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
@@ -0,0 +1,48 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+You need a fixed readprofile command for 2.5 ... either get hold of
+a current version from:
+http://www.kernel.org/pub/linux/utils/util-linux/
+or get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/readprofile/
+
+Add "profile=2" to the kernel command line.
+
+clear		readprofile -r
+		<test>
+dump output	readprofile -m /boot/System.map > captured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64	opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+
#8 lore raw
Quoted 1 line · Martin J. Bligh

+start daemon opcontrol --start-daemon

--start implies starting the daemon if it isn't started already.

Quoted 1 line

+stop opcontrol --stop

--stop unsupported. use "--shutdown"

Quoted 1 line

+dump output oprofpp -dl -i /boot/vmlinux > output_file

opcontrol --dump

op_time -l is also worth adding to this doc imo.

Dave

#9 lore raw
Quoted 8 lines · Dave Jones
Quoted 1 line

+start daemon opcontrol --start-daemon

--start implies starting the daemon if it isn't started already.

Quoted 1 line

+stop opcontrol --stop

--stop unsupported. use "--shutdown"

Stop running 2.4 ! :)

--stop works on 2.5 only ...

regards
john

#10 lore raw
Quoted 6 lines · John Levon
Quoted 2 lines
Quoted 1 line

+stop opcontrol --stop

--stop unsupported. use "--shutdown"

Stop running 2.4 ! :)

--stop works on 2.5 only ...

That seems quite confusing. Why not make --stop an alias for --shutdown if running on 2.4 ?

Dave

#11 lore raw
In reply to Dave Jones
Quoted 2 lines

--start implies starting the daemon if it isn't started already.

I think John suggested to do those seperately, to minimise overhead or something.

Quoted 5 lines
Quoted 1 line

+dump output oprofpp -dl -i /boot/vmlinux > output_file

opcontrol --dump

op_time -l is also worth adding to this doc imo.

OK, will add.

M.

#12 lore raw
Quoted 5 lines · Martin J. Bligh
Quoted 2 lines

--start implies starting the daemon if it isn't started already.

I think John suggested to do those seperately, to minimise overhead or something.

On a heavily loaded system, --start can take quite a while if the daemon isn't already running. Before starting my machine-killing benchmark (which I want to profile), I use --start-daemon which gets most of the work out of the way. The subsequent --start, which occurs while the benchmark is already steaming along, will happen quickly and with much less impact to the benchmark results.

Signature · 2 lines
Dave Hansen
[email protected]
#13 lore raw
In reply to Martin J. Bligh

Hi!

Documentation/basic_profiling.txt line 28

Quoted 4 lines
Documentation/basic_profiling.txt +4 -0
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 

This should be idle=poll, AFAICS.
Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#14 lore raw
In reply to Martin J. Bligh

On Fri, 28 Feb 2003 00:12:42 -0800
"Martin J. Bligh" <[email protected]> wrote:

| I was trying to write some simple docs on how to do profiling for people 
| to use for really basic stuff. I got it all wrong, but John's kindly 
| corrected  it ;-) Andrew asked me to do this as a patch for the 
| documentation directory ... feedback would be much appreciated 
| (yes, it's oversimplified - it's meant to be).
| 
| diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
| --- virgin/Documentation/basic_profiling.txt	Wed Dec 31 16:00:00 1969
| +++ oprofile_doc/Documentation/basic_profiling.txt	Fri Feb 28 00:05:59 2003
| @@ -0,0 +1,44 @@
| +These instructions are deliberately very basic. If you want something clever,
| +go read the real docs ;-) Please don't add more stuff, but feel free to 
| +correct my mistakes ;-)    ([email protected])
| +Thanks to John Levon and Dave Hansen for help writing this.
| +
| +<test> is the thing you're trying to measure.
| +Make sure you have the correct System.map / vmlinux referenced!
| +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
| +to copy config files, system.map, vmlinux to /boot.
| +
| +Readprofile
| +-----------
| +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
| +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
| +add "profile=2" to the kernel command line.

These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 > /proc/profile
man page says to use "readprofile -r". Doesn't that still work?

| + <test>
| +dump output readprofile -m /boot/System.map > catured_profile
> captured_profile

--
~Randy

#15 lore raw
Quoted 4 lines

These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 > /proc/profile
man page says to use "readprofile -r". Doesn't that still work?

Dunno. I always have done the above ... have you been using -r with success? If so, I'll change it.

M.

#16 lore raw
Quoted 7 lines · Martin J. Bligh
Quoted 4 lines

These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 > /proc/profile
man page says to use "readprofile -r". Doesn't that still work?

Dunno. I always have done the above ... have you been using -r with success? If so, I'll change it.

It's what I've always used.

Signature · 2 lines
Dave Hansen
[email protected]
#17 lore raw

On Fri, 28 Feb 2003 10:13:51 -0800
Dave Hansen <[email protected]> wrote:

| Martin J. Bligh wrote:
| >>These:          ^------------v  should be the same value (as you have it).
| >>                             v
| >>| +clear		echo 2 > /proc/profile
| >>man page says to use "readprofile -r".  Doesn't that still work?
| > 
| > Dunno. I always have done the above ... have you been using -r with
| > success? If so, I'll change it.
| 
| It's what I've always used.

Same here (-r).

--
~Randy

#18 lore raw

OK, fixed a couple of things ... thanks for the feedback everyone.

Documentation/basic_profiling.txt +44 -0
diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear		readprofile -r
+		<test>
+dump output	readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64	opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+
#19 lore raw
In reply to Martin J. Bligh

Documentation/basic_profiling.txt line 17

Quoted 7 lines · Martin J. Bligh
Documentation/basic_profiling.txt +7 -0
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear		echo 2 > /proc/profile

As far as I can see, the 2 is meaningless here. This should just be

echo > /proc/profile

(On SMP when writing sizeof(int) bytes, the value written is significant. But 1 or 2 is not sizeof(int).)

Andries

-----
Fragment of some notes:

<sect1>Profiling the kernel<p>
There are several facilities to see where the kernel spends its resources. A simple one is the profiling function, that stores the current EIP (instruction pointer) at each clock tick. <p>
Boot the kernel with command line option <tt>profile=2</tt> (or some other number instead of 2). This will cause
a file <tt>/proc/profile</tt> to be created.
The number given after <tt>profile=</tt> is the number of positions EIP is shifted right when profiling. So a large number gives a coarse profile.
The counters are reset by writing to <tt>/proc/profile</tt>. The utility <tt>readprofile</tt> will output statistics for you. It does not sort - you have to invoke <tt>sort</tt> explicitly. But given a memory map it will translate addresses to kernel symbols. <p>
See <tt>kernel/profile.c</tt> and <tt>fs/proc/proc_misc.c</tt> and <tt>readprofile(1)</tt>.
<p>
For example:
<verb>

# echo > /proc/profile
...
# readprofile -m System.map-2.5.59 | sort -nr | head -2
510502 total                                      0.1534
508548 default_idle                           10594.7500
</verb>