Skip to contentSkip to conversation

perf record/report: Add call graph / call chain profiling

tip:perfcounters/core tip-bot for Ingo Molnar 4 participants 8 messages Jun 14, 2009 – Jun 17, 2009 perf events

lore
Share & tools
lore
#0 lore raw
Commit-ID:  3efa1cc99ec51bc7a7ae0011a16619fd20dbe6ea
Gitweb:     http://git.kernel.org/tip/3efa1cc99ec51bc7a7ae0011a16619fd20dbe6ea
Author:     Ingo Molnar <[email protected]>
AuthorDate: Sun, 14 Jun 2009 15:04:15 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 14 Jun 2009 20:34:06 +0200

perf record/report: Add call graph / call chain profiling

Add the first steps of call-graph profiling:

- add the -c (--call-graph) option to perf record
- parse the call-graph record and printout out under -D (--dump-trace)

The call-graph data is not put into the histogram yet, but it can be seen that it's being processed correctly:

0x3ce0 [0x38]: event: 35
.
. ... raw event: size 56 bytes

.  0000:  23 00 00 00 05 00 38 00 d4 df 0e 81 ff ff ff ff  #.....8........
.  0010:  60 0b 00 00 60 0b 00 00 03 00 00 00 01 00 02 00  `...`..........
.  0020:  d4 df 0e 81 ff ff ff ff a0 61 ed 41 36 00 00 00  .........a.A6..
.  0030:  04 92 e6 41 36 00 00 00                          .a.A6..
.
0x3ce0 [0x38]: PERF_EVENT (IP, 5): 2912: 0xffffffff810edfd4 period: 1
... chain: u:2, k:1, nr:3
.....  0: 0xffffffff810edfd4
.....  1: 0x3641ed61a0
.....  2: 0x3641e69204

... thread: perf:2912
...... dso: [kernel]

This shows a 3-entry call-graph: with 1 kernel-space and two user-space entries

Cc: Frederic Weisbecker <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Arjan van de Ven <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
LKML-Reference: <new-submission>

---
 tools/perf/builtin-record.c |    8 ++++++
 tools/perf/builtin-report.c |   57 ++++++++++++++++++++++++++++++++++---------
 2 files changed, 53 insertions(+), 12 deletions(-)
tools/perf/builtin-record.c +8 -0
index 0f5771f..a177a59 100644
@@ -37,6 +37,7 @@ static pid_t			target_pid			= -1;
 static int			inherit				= 1;
 static int			force				= 0;
 static int			append_file			= 0;
+static int			call_graph			= 0;
 static int			verbose				= 0;
 
 static long			samples;
@@ -351,11 +352,16 @@ static void create_counter(int counter, int cpu, pid_t pid)
 	int track = 1;
 
 	attr->sample_type	= PERF_SAMPLE_IP | PERF_SAMPLE_TID;
+
 	if (freq) {
 		attr->sample_type	|= PERF_SAMPLE_PERIOD;
 		attr->freq		= 1;
 		attr->sample_freq	= freq;
 	}
+
+	if (call_graph)
+		attr->sample_type	|= PERF_SAMPLE_CALLCHAIN;
+
 	attr->mmap		= track;
 	attr->comm		= track;
 	attr->inherit		= (cpu < 0) && inherit;
@@ -555,6 +561,8 @@ static const struct option options[] = {
 		    "profile at this frequency"),
 	OPT_INTEGER('m', "mmap-pages", &mmap_pages,
 		    "number of mmap data pages"),
+	OPT_BOOLEAN('g', "call-graph", &call_graph,
+		    "do call-graph (stack chain/backtrace) recording"),
 	OPT_BOOLEAN('v', "verbose", &verbose,
 		    "be more verbose (show counter open errors, etc)"),
 	OPT_END()
tools/perf/builtin-report.c +45 -12

107 lines not on this page · raw message

#1 lore raw

tools/perf/builtin-report.c static int full_paths; · line 54

Quoted 8 lines · tip-bot for Ingo Molnar
tools/perf/builtin-report.c +8 -0
+struct ip_chain_event {
+	__u16 nr;
+	__u16 hv;
+	__u16 kernel;
+	__u16 user;
+	__u64 ips[];
+};
+

The following error on my 32bit box:

cc1: warnings being treated as errors
builtin-report.c: In function 'process_overflow_event':
builtin-report.c:984: error: cast to pointer from integer of different size
make: *** [builtin-report.o] Error 1

unsigned long ips[] perhaps?

#2 lore raw
In reply to tip-bot for Ingo Molnar
Quoted 103 lines · tip-bot for Ingo Molnar
Commit-ID:  3efa1cc99ec51bc7a7ae0011a16619fd20dbe6ea
Gitweb:     http://git.kernel.org/tip/3efa1cc99ec51bc7a7ae0011a16619fd20dbe6ea
Author:     Ingo Molnar <[email protected]>
AuthorDate: Sun, 14 Jun 2009 15:04:15 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 14 Jun 2009 20:34:06 +0200

perf record/report: Add call graph / call chain profiling

Add the first steps of call-graph profiling:

- add the -c (--call-graph) option to perf record
- parse the call-graph record and printout out under -D (--dump-trace)

The call-graph data is not put into the histogram yet, but it can be seen that it's being processed correctly:

0x3ce0 [0x38]: event: 35
.
. ... raw event: size 56 bytes

.  0000:  23 00 00 00 05 00 38 00 d4 df 0e 81 ff ff ff ff  #.....8........
.  0010:  60 0b 00 00 60 0b 00 00 03 00 00 00 01 00 02 00  `...`..........
.  0020:  d4 df 0e 81 ff ff ff ff a0 61 ed 41 36 00 00 00  .........a.A6..
.  0030:  04 92 e6 41 36 00 00 00                          .a.A6..
.
0x3ce0 [0x38]: PERF_EVENT (IP, 5): 2912: 0xffffffff810edfd4 period: 1
... chain: u:2, k:1, nr:3
.....  0: 0xffffffff810edfd4
.....  1: 0x3641ed61a0
.....  2: 0x3641e69204

... thread: perf:2912
...... dso: [kernel]

This shows a 3-entry call-graph: with 1 kernel-space and two user-space entries

Cc: Frederic Weisbecker <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Arjan van de Ven <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
LKML-Reference: <new-submission>

---
 tools/perf/builtin-record.c |    8 ++++++
 tools/perf/builtin-report.c |   57 ++++++++++++++++++++++++++++++++++---------
 2 files changed, 53 insertions(+), 12 deletions(-)
tools/perf/builtin-record.c +8 -0
index 0f5771f..a177a59 100644
@@ -37,6 +37,7 @@ static pid_t			target_pid			= -1;
 static int			inherit				= 1;
 static int			force				= 0;
 static int			append_file			= 0;
+static int			call_graph			= 0;
 static int			verbose				= 0;
 
 static long			samples;
@@ -351,11 +352,16 @@ static void create_counter(int counter, int cpu, pid_t pid)
 	int track = 1;
 
 	attr->sample_type	= PERF_SAMPLE_IP | PERF_SAMPLE_TID;
+
 	if (freq) {
 		attr->sample_type	|= PERF_SAMPLE_PERIOD;
 		attr->freq		= 1;
 		attr->sample_freq	= freq;
 	}
+
+	if (call_graph)
+		attr->sample_type	|= PERF_SAMPLE_CALLCHAIN;
+
 	attr->mmap		= track;
 	attr->comm		= track;
 	attr->inherit		= (cpu < 0) && inherit;
@@ -555,6 +561,8 @@ static const struct option options[] = {
 		    "profile at this frequency"),
 	OPT_INTEGER('m', "mmap-pages", &mmap_pages,
 		    "number of mmap data pages"),
+	OPT_BOOLEAN('g', "call-graph", &call_graph,
+		    "do call-graph (stack chain/backtrace) recording"),
 	OPT_BOOLEAN('v', "verbose", &verbose,
 		    "be more verbose (show counter open errors, etc)"),
 	OPT_END()
tools/perf/builtin-report.c +1 -0
index 37515da..aebba56 100644
@@ -36,6 +36,7 @@ static int		show_mask = SHOW_KERNEL | SHOW_USER | SHOW_HV;
 
 static int		dump_trace = 0;
 #define dprintf(x...)	do { if (dump_trace) printf(x); } while (0)
+#define cdprintf(x...)	do { if (dump_trace) color_fprintf(stdout, color, x); } while (0)
 
 static int		verbose;
 static int		full_paths;

tools/perf/builtin-report.c static int full_paths; · line 48

Quoted 5 lines
tools/perf/builtin-report.c +2 -0
@@ -43,11 +44,19 @@ static int		full_paths;
 static unsigned long	page_size;
 static unsigned long	mmap_window = 32;
 
+struct ip_chain_event {
+	__u16 nr;

Is it needed to have the nr encoded in the ip_chain? We can already find it by doing kernel + user.

Thanks,
Frederic.

Quoted 98 lines
tools/perf/builtin-report.c +42 -12
+	__u16 hv;
+	__u16 kernel;
+	__u16 user;
+	__u64 ips[];
+};
+
 struct ip_event {
 	struct perf_event_header header;
 	__u64 ip;
 	__u32 pid, tid;
-	__u64 period;
+	unsigned char __more_data[];
 };
 
 struct mmap_event {
@@ -944,9 +953,13 @@ process_overflow_event(event_t *event, unsigned long offset, unsigned long head)
 	__u64 ip = event->ip.ip;
 	__u64 period = 1;
 	struct map *map = NULL;
+	void *more_data = event->ip.__more_data;
+	struct ip_chain_event *chain;
 
-	if (event->header.type & PERF_SAMPLE_PERIOD)
-		period = event->ip.period;
+	if (event->header.type & PERF_SAMPLE_PERIOD) {
+		period = *(__u64 *)more_data;
+		more_data += sizeof(__u64);
+	}
 
 	dprintf("%p [%p]: PERF_EVENT (IP, %d): %d: %p period: %Ld\n",
 		(void *)(offset + head),
@@ -956,6 +969,22 @@ process_overflow_event(event_t *event, unsigned long offset, unsigned long head)
 		(void *)(long)ip,
 		(long long)period);
 
+	if (event->header.type & PERF_SAMPLE_CALLCHAIN) {
+		int i;
+
+		chain = (void *)more_data;
+
+		if (dump_trace) {
+			dprintf("... chain: u:%d, k:%d, nr:%d\n",
+				chain->user,
+				chain->kernel,
+				chain->nr);
+
+			for (i = 0; i < chain->nr; i++)
+				dprintf("..... %2d: %p\n", i, (void *)chain->ips[i]);
+		}
+	}
+
 	dprintf(" ... thread: %s:%d\n", thread->comm, thread->pid);
 
 	if (thread == NULL) {
@@ -1098,30 +1127,34 @@ process_period_event(event_t *event, unsigned long offset, unsigned long head)
 static void trace_event(event_t *event)
 {
 	unsigned char *raw_event = (void *)event;
+	char *color = PERF_COLOR_BLUE;
 	int i, j;
 
 	if (!dump_trace)
 		return;
 
-	dprintf(".\n. ... raw event: size %d bytes\n", event->header.size);
+	dprintf(".");
+	cdprintf("\n. ... raw event: size %d bytes\n", event->header.size);
 
 	for (i = 0; i < event->header.size; i++) {
-		if ((i & 15) == 0)
-			dprintf(".  %04x: ", i);
+		if ((i & 15) == 0) {
+			dprintf(".");
+			cdprintf("  %04x: ", i);
+		}
 
-		dprintf(" %02x", raw_event[i]);
+		cdprintf(" %02x", raw_event[i]);
 
 		if (((i & 15) == 15) || i == event->header.size-1) {
-			dprintf("  ");
+			cdprintf("  ");
 			for (j = 0; j < 15-(i & 15); j++)
-				dprintf("   ");
+				cdprintf("   ");
 			for (j = 0; j < (i & 15); j++) {
 				if (isprint(raw_event[i-15+j]))
-					dprintf("%c", raw_event[i-15+j]);
+					cdprintf("%c", raw_event[i-15+j]);
 				else
-					dprintf(".");
+					cdprintf(".");
 			}
-			dprintf("\n");
+			cdprintf("\n");
 		}
 	}
 	dprintf(".\n");
#3 lore raw
Quoted 9 lines · Frederic Weisbecker

tools/perf/builtin-report.c static int full_paths; · line 48

Quoted 6 lines
tools/perf/builtin-report.c +2 -0
@@ -43,11 +44,19 @@ static int		full_paths;
 static unsigned long	page_size;
 static unsigned long	mmap_window = 32;
 
+struct ip_chain_event {
+	__u16 nr;

Is it needed to have the nr encoded in the ip_chain? We can already find it by doing kernel + user.

That's a good observation. Since we havent exposed the call-chain bits in upstream version of the tools, we could still improve on this a little bit.

I think the best would be context separators which occupy a special address in some quiet corner of the 64-bit address space.

That way we'd have streams of u64 entries:

ip-1
ip-2
CONTEXT_IRQ
ip-3
ip-4
CONTEXT_SYSCALL
ip-5
ip-6

The following contexts IDs would be useful:

CONTEXT_NMI
CONTEXT_HARDIRQ
CONTEXT_SOFTIRQ
CONTEXT_KERNEL
CONTEXT_USER
CONTEXT_GUEST_NMI
CONTEXT_GUEST_HARDIRQ
CONTEXT_GUEST_SOFTIRQ
CONTEXT_GUEST_KERNEL
CONTEXT_GUEST_USER

The context IDs would occupy some rare and unlikely-to-be-allocated-soon corner of the address space - say startig at 0x8765432112345000. (and real RIPs would be filtered and nudged just outside that space of a handful IDs.)

The advantage would be that this is infinitely flexible and extensible - any level of nesting can be expressed without having separate fields for nr_hv_guest_irq, etc. It's also pretty fast to parse.

Hm?

Ingo

#7 lore raw
In reply to Frederic Weisbecker
Quoted 7 lines · Frederic Weisbecker

tools/perf/builtin-report.c static int full_paths; · line 48

Quoted 2 lines
tools/perf/builtin-report.c +2 -0
+struct ip_chain_event {
+     __u16 nr;

Is it needed to have the nr encoded in the ip_chain? We can already find it by doing kernel + user.

strictly speaking: hv+kernel+user
but yeah, we could possibly replace it with another context, guest perhaps.

suggestions anyone?

Quoted 6 lines
Quoted 6 lines
tools/perf/builtin-report.c +5 -0
+     __u16 hv;
+     __u16 kernel;
+     __u16 user;
+     __u64 ips[];
+};
#4 lore raw
In reply to Ingo Molnar
Quoted 43 lines · Ingo Molnar
Quoted 9 lines · Frederic Weisbecker

tools/perf/builtin-report.c static int full_paths; · line 48

Quoted 6 lines
tools/perf/builtin-report.c +2 -0
@@ -43,11 +44,19 @@ static int		full_paths;
 static unsigned long	page_size;
 static unsigned long	mmap_window = 32;
 
+struct ip_chain_event {
+	__u16 nr;

Is it needed to have the nr encoded in the ip_chain? We can already find it by doing kernel + user.

That's a good observation. Since we havent exposed the call-chain bits in upstream version of the tools, we could still improve on this a little bit.

I think the best would be context separators which occupy a special address in some quiet corner of the 64-bit address space.

That way we'd have streams of u64 entries:

   ip-1
   ip-2
   CONTEXT_IRQ
   ip-3
   ip-4
   CONTEXT_SYSCALL
   ip-5
   ip-6

The following contexts IDs would be useful:

CONTEXT_NMI
CONTEXT_HARDIRQ
CONTEXT_SOFTIRQ
CONTEXT_KERNEL
CONTEXT_USER
CONTEXT_GUEST_NMI
CONTEXT_GUEST_HARDIRQ
CONTEXT_GUEST_SOFTIRQ
CONTEXT_GUEST_KERNEL
CONTEXT_GUEST_USER

Quoted 4 lines

The context IDs would occupy some rare and unlikely-to-be-allocated-soon corner of the address space - say startig at 0x8765432112345000. (and real RIPs would be filtered and nudged just outside that space of a handful IDs.)

Right, that works too, but should we use (u64)-1..-4095 for that? We already use that range for things like ERR_PTR() so its very unlikely we have something sensible mapped there.

#6 lore raw
In reply to Ingo Molnar
Quoted 49 lines · Ingo Molnar
Quoted 9 lines · Frederic Weisbecker

tools/perf/builtin-report.c static int full_paths; · line 48

Quoted 6 lines
tools/perf/builtin-report.c +2 -0
@@ -43,11 +44,19 @@ static int		full_paths;
 static unsigned long	page_size;
 static unsigned long	mmap_window = 32;
 
+struct ip_chain_event {
+	__u16 nr;

Is it needed to have the nr encoded in the ip_chain? We can already find it by doing kernel + user.

That's a good observation. Since we havent exposed the call-chain bits in upstream version of the tools, we could still improve on this a little bit.

I think the best would be context separators which occupy a special address in some quiet corner of the 64-bit address space.

That way we'd have streams of u64 entries:

   ip-1
   ip-2
   CONTEXT_IRQ
   ip-3
   ip-4
   CONTEXT_SYSCALL
   ip-5
   ip-6

The following contexts IDs would be useful:

CONTEXT_NMI
CONTEXT_HARDIRQ
CONTEXT_SOFTIRQ
CONTEXT_KERNEL
CONTEXT_USER
CONTEXT_GUEST_NMI
CONTEXT_GUEST_HARDIRQ
CONTEXT_GUEST_SOFTIRQ
CONTEXT_GUEST_KERNEL
CONTEXT_GUEST_USER

The context IDs would occupy some rare and unlikely-to-be-allocated-soon corner of the address space - say startig at 0x8765432112345000. (and real RIPs would be filtered and nudged just outside that space of a handful IDs.)

Quoted 4 lines

The advantage would be that this is infinitely flexible and extensible - any level of nesting can be expressed without having separate fields for nr_hv_guest_irq, etc. It's also pretty fast to parse.

Indeed, nice idea!

Quoted 3 lines

Hm?

Ingo

#5 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines

The context IDs would occupy some rare and unlikely-to-be-allocated-soon corner of the address space - say startig at 0x8765432112345000. (and real RIPs would be filtered and nudged just outside that space of a handful IDs.)

Right, that works too, but should we use (u64)-1..-4095 for that? We already use that range for things like ERR_PTR() so its very unlikely we have something sensible mapped there.

Makes sense. It's also an easier (and more natural) enumeration method.

Ingo