perf-probe: issue with latest fedora kernel

PATCH New Francis Moreau 4 participants 30 messages Dec 9, 2010 – Dec 27, 2010 perf events

lore

Share & tools

lore

Francis Moreau

author

2010-12-09 15:00 UTC #0 lore raw

Hello,

I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).

I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.

I added it to the buildid cache:

$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK

But when running perf-probe(1) like the following:

  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

it fails.

One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.

Could anybody tell me what I'm doing wrong ?

Thanks

Signature · 1 line

Francis

Arnaldo Carvalho de Melo

2010-12-09 15:35 UTC #1 lore raw

Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:

Quoted 17 lines

I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
I added it to the buildid cache:
$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
But when running perf-probe(1) like the following:
  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

Quoted 6 lines

it fails.
One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.
Could anybody tell me what I'm doing wrong ?

Can you try using:

$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>

So that it tries it directly instead of looking into the buildid cache?

I'll check if perf probe actually looks in the buildid cache, till then, please try specifying it directly,

Regards,

- Arnaldo

Francis Moreau

author

2010-12-09 16:34 UTC #2 lore raw

Quoted 24 lines · Arnaldo Carvalho de Melo

Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 23 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
I added it to the buildid cache:
$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
But when running perf-probe(1) like the following:
  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)
it fails.
One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.

Quoted 8 lines

Quoted 1 line
Could anybody tell me what I'm doing wrong ?
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?

Well that still fail:

$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

but that's not really suprising if perf checks that the running kernel build-id is the same as the one in:
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

It really looks like something is broken in the F14 kernel-debuginfo package.

thanks

Signature · 1 line

Francis

Masami Hiramatsu

2010-12-10 03:30 UTC #3 lore raw

Quoted 41 lines · Francis Moreau

Quoted 24 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 23 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
I added it to the buildid cache:
$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
But when running perf-probe(1) like the following:
  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)
it fails.
One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.
Quoted 8 lines
Quoted 1 line
Could anybody tell me what I'm doing wrong ?
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Well that still fail:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

Quoted 6 lines

but that's not really suprising if perf checks that the running kernel build-id is the same as the one in:
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
It really looks like something is broken in the F14 kernel-debuginfo package.

Hmm, curious. I'll check it after setting up Fedora 14. Thank you for reporting :-)

Signature · 4 lines

Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Franck Bui-Huu

2010-12-10 07:30 UTC #9 lore raw

In reply to Arnaldo Carvalho de Melo

Quoted 26 lines · Arnaldo Carvalho de Melo

Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 18 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
I added it to the buildid cache:
$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
But when running perf-probe(1) like the following:
  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)
Quoted 6 lines
it fails.
One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.
Could anybody tell me what I'm doing wrong ?

Quoted 8 lines

Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
I'll check if perf probe actually looks in the buildid cache, till then, please try specifying it directly,

Looks like a standard path is missing...

What about adding the one above since it looks like Fedora is installing their debug kernel images at this place ?

tools/perf/util/symbol.c +7 -1

index d628c8d..0f914c7 100644
@@ -2260,7 +2262,7 @@ static int vmlinux_path__init(void)
 	if (uname(&uts) < 0)
 		return -1;
 
-	vmlinux_path = malloc(sizeof(char *) * 5);
+	vmlinux_path = malloc(sizeof(char *) * 6);
 	if (vmlinux_path == NULL)
 		return -1;

@@ -2288,6 +2290,12 @@ static int vmlinux_path__init(void)
 	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
 		goto out_fail;
 	++vmlinux_path__nr_entries;
+	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
+		 uts.release);
+	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
+	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
+		goto out_fail;
+	++vmlinux_path__nr_entries;
 
 	return 0;

Signature · 1 line

		Franck

Franck Bui-Huu

2010-12-10 07:53 UTC #4 lore raw

In reply to Masami Hiramatsu

Hi,

Quoted 1 line · Masami Hiramatsu

(2010/12/10 1:34), Francis Moreau wrote:

[...]

Quoted 11 lines

Quoted 11 lines
Well that still fail:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)
but that's not really suprising if perf checks that the running kernel build-id is the same as the one in:
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

Quoted 5 lines

Quoted 2 lines
It really looks like something is broken in the F14 kernel-debuginfo package.
Hmm, curious. I'll check it after setting up Fedora 14. Thank you for reporting :-)

BTW, it seems to me that perf-probe(1) is ignoring the kernel path given by '-k' switch.

Looking at kernel_get_module_path(), it only considers the standard paths given by vmlinux_path[].

I would have seen this, no ?

tools/perf/builtin-probe.c +5 -0

index 2e000c0..add163c 100644
@@ -249,6 +249,11 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	     !params.show_lines))
 		usage_with_options(probe_usage, options);
 
+	/*
+	 * Only consider the user's kernel image path if given.
+	 */
+	symbol_conf.try_vmlinux_path = (symbol_conf.vmlinux_name == NULL);
+
 	if (params.list_events) {
 		if (params.mod_events) {
 			pr_err("  Error: Don't use --list with --add/--del.\n");

tools/perf/util/probe-event.c +18 -7

index 694f04b..558545e 100644
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {

@@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
-	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
-			pr_debug("Failed to load kernel map.\n");
-			return NULL;
-		}
 	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {
+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)
+			goto found;
+		return NULL;
+	}
+
+	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
+		pr_debug("Failed to load kernel map.\n");
+		return NULL;
+	}
+
 found:
 	return dso->long_name;
 }

tools/perf/util/symbol.c +2 -2

index a3a0c43..4e42e0d 100644
@@ -1780,8 +1780,8 @@ out_failure:
 	return -1;
 }
 
-static int dso__load_vmlinux(struct dso *self, struct map *map,
-			     const char *vmlinux, symbol_filter_t filter)
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter)
 {
 	int err = -1, fd;

tools/perf/util/symbol.h +2 -0

index 038f220..6c6eafd 100644
@@ -166,6 +166,8 @@ void dso__sort_by_name(struct dso *self, enum map_type type);
 struct dso *__dsos__findnew(struct list_head *head, const char *name);
 
 int dso__load(struct dso *self, struct map *map, symbol_filter_t filter);
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter);
 int dso__load_vmlinux_path(struct dso *self, struct map *map,
 			   symbol_filter_t filter);
 int dso__load_kallsyms(struct dso *self, const char *filename, struct map *map,

Signature · 1 line

		Franck

Masami Hiramatsu

2010-12-10 08:27 UTC #5 lore raw

Quoted 8 lines · Franck Bui-Huu

BTW, it seems to me that perf-probe(1) is ignoring the kernel path given by '-k' switch.
Looking at kernel_get_module_path(), it only considers the standard paths given by vmlinux_path[].
I would have seen this, no ?

Right! Thank you for finding the bug!

Quoted 31 lines

tools/perf/util/probe-event.c +8 -7

index 694f04b..558545e 100644
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {

@@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
-	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
-			pr_debug("Failed to load kernel map.\n");
-			return NULL;
-		}
 	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {

tools/perf/util/probe-event.c const char *kernel_get_module_path(const char *module) · line 144

Quoted 10 lines

tools/perf/util/probe-event.c +10 -0

+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)
+			goto found;
+		return NULL;
+	}
+
+	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
+		pr_debug("Failed to load kernel map.\n");
+		return NULL;
+	}
+

Hmm, could you put these changes into an else block? Because this part is only for the kernel vmlinux, and not for modules.

Thanks,

Signature · 4 lines

Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Franck Bui-Huu

2010-12-10 13:00 UTC #6 lore raw

Quoted 1 line · Masami Hiramatsu

(2010/12/10 16:53), Franck Bui-Huu wrote:

[...]

Quoted 24 lines

tools/perf/util/probe-event.c +2 -7

index 694f04b..558545e 100644
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {

@@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
-	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
-			pr_debug("Failed to load kernel map.\n");
-			return NULL;
-		}

Quoted 8 lines

tools/perf/util/probe-event.c const char *kernel_get_module_path(const char *module) · line 135
Quoted 8 lines
tools/perf/util/probe-event.c +7 -0
 	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {
+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)

BTW, the condition seems wrong, it should have been '<= 0', that is if an error occured or no symbol was loaded.

Quoted 5 lines

tools/perf/util/probe-event.c const char *kernel_get_module_path(const char *module) · line 140
Quoted 5 lines
tools/perf/util/probe-event.c +5 -0
+			goto found;
+		return NULL;
+	}
+
+	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {

This one should be '<= 0' too, since if no symbol is loaded, the image is probably wrong.

Quoted 8 lines

tools/perf/util/probe-event.c const char *kernel_get_module_path(const char *module) · line 144
Quoted 4 lines
tools/perf/util/probe-event.c +4 -0
+		pr_debug("Failed to load kernel map.\n");
+		return NULL;
+	}
+
Hmm, could you put these changes into an else block? Because this part is only for the kernel vmlinux, and not for modules.

Ok, I'll do that.

Thanks.

Signature · 1 line

		Franck

Franck Bui-Huu

2010-12-10 13:06 UTC #7 lore raw

PATCH 1/2 perf-probe: do use the kernel image path given by 'k' option

From: Franck Bui-Huu <[email protected]>

Signed-off-by Franck Bui-Huu <[email protected]>

---
 tools/perf/builtin-probe.c    |    5 +++++
 tools/perf/util/probe-event.c |   15 ++++++++++++---
 tools/perf/util/symbol.c      |    4 ++--
 tools/perf/util/symbol.h      |    2 ++
 4 files changed, 21 insertions(+), 5 deletions(-)

tools/perf/builtin-probe.c +5 -0

index 2e000c0..add163c 100644
@@ -249,6 +249,11 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	     !params.show_lines))
 		usage_with_options(probe_usage, options);
 
+	/*
+	 * Only consider the user's kernel image path if given.
+	 */
+	symbol_conf.try_vmlinux_path = (symbol_conf.vmlinux_name == NULL);
+
 	if (params.list_events) {
 		if (params.mod_events) {
 			pr_err("  Error: Don't use --list with --add/--del.\n");

tools/perf/util/probe-event.c +12 -3

index 694f04b..99f9933 100644
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {

@@ -123,10 +125,17 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
+	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {
+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) <= 0)
+			return NULL;
 	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
+		if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
 			pr_debug("Failed to load kernel map.\n");
 			return NULL;
 		}

tools/perf/util/symbol.c +2 -2

index a3a0c43..4e42e0d 100644
@@ -1780,8 +1780,8 @@ out_failure:
 	return -1;
 }
 
-static int dso__load_vmlinux(struct dso *self, struct map *map,
-			     const char *vmlinux, symbol_filter_t filter)
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter)
 {
 	int err = -1, fd;

tools/perf/util/symbol.h +2 -0

index 038f220..6c6eafd 100644
@@ -166,6 +166,8 @@ void dso__sort_by_name(struct dso *self, enum map_type type);
 struct dso *__dsos__findnew(struct list_head *head, const char *name);
 
 int dso__load(struct dso *self, struct map *map, symbol_filter_t filter);
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter);
 int dso__load_vmlinux_path(struct dso *self, struct map *map,
 			   symbol_filter_t filter);
 int dso__load_kallsyms(struct dso *self, const char *filename, struct map *map,

Signature · 1 line

1.7.3.2

Franck Bui-Huu

2010-12-10 13:07 UTC #8 lore raw

PATCH 2/2 perf-probe: fail if the kernel image contains no symbol

In reply to Franck Bui-Huu

From: Franck Bui-Huu <[email protected]>

This fix is for kernel images found in the standard paths.

Signed-off-by Franck Bui-Huu <[email protected]>

---
 tools/perf/util/probe-event.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

tools/perf/util/probe-event.c +1 -1

index 99f9933..fc2b0b4 100644
@@ -135,7 +135,7 @@ const char *kernel_get_module_path(const char *module)
 		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) <= 0)
 			return NULL;
 	} else {
-		if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
+		if (dso__load_vmlinux_path(dso, map, NULL) <= 0) {
 			pr_debug("Failed to load kernel map.\n");
 			return NULL;
 		}

Signature · 5 lines

1.7.3.2


-- 
		Franck

Arnaldo Carvalho de Melo

2010-12-12 14:14 UTC #10 lore raw

In reply to Franck Bui-Huu

Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:

Quoted 4 lines

Looks like a standard path is missing...
What about adding the one above since it looks like Fedora is installing their debug kernel images at this place ?

Can I get your signed-off-by on this one?

- Arnaldo

Quoted 28 lines

tools/perf/util/symbol.c +7 -1

index d628c8d..0f914c7 100644
@@ -2260,7 +2262,7 @@ static int vmlinux_path__init(void)
 	if (uname(&uts) < 0)
 		return -1;
 
-	vmlinux_path = malloc(sizeof(char *) * 5);
+	vmlinux_path = malloc(sizeof(char *) * 6);
 	if (vmlinux_path == NULL)
 		return -1;

@@ -2288,6 +2290,12 @@ static int vmlinux_path__init(void)
 	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
 		goto out_fail;
 	++vmlinux_path__nr_entries;
+	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
+		 uts.release);
+	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
+	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
+		goto out_fail;
+	++vmlinux_path__nr_entries;
 
 	return 0;

Signature · 1 line

		Franck

Franck Bui-Huu

2010-12-12 22:15 UTC #11 lore raw

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:
Quoted 4 lines
Looks like a standard path is missing...
What about adding the one above since it looks like Fedora is installing their debug kernel images at this place ?
Can I get your signed-off-by on this one?

Of course.

[PATCH] perf-tools: add one more default search path for kernel image

From: Franck Bui-Huu <[email protected]>

It looks like Fedora is using this one.

Reported-by Francis Moreau <[email protected]>

Signed-off-by Franck Bui-Huu <[email protected]>

---
 tools/perf/util/symbol.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

tools/perf/util/symbol.c +7 -1

index d628c8d..a3a0c43 100644
@@ -2260,7 +2260,7 @@ static int vmlinux_path__init(void)
 	if (uname(&uts) < 0)
 		return -1;
 
-	vmlinux_path = malloc(sizeof(char *) * 5);
+	vmlinux_path = malloc(sizeof(char *) * 6);
 	if (vmlinux_path == NULL)
 		return -1;

@@ -2288,6 +2288,12 @@ static int vmlinux_path__init(void)
 	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
 		goto out_fail;
 	++vmlinux_path__nr_entries;
+	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
+		 uts.release);
+	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
+	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
+		goto out_fail;
+	++vmlinux_path__nr_entries;
 
 	return 0;

Signature · 1 line

1.7.3.2

Francis Moreau

author

2010-12-13 10:02 UTC #13 lore raw

In reply to Arnaldo Carvalho de Melo

Hi,

Quoted 26 lines · Arnaldo Carvalho de Melo

Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 18 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
I added it to the buildid cache:
$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
But when running perf-probe(1) like the following:
  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)
Quoted 6 lines
it fails.
One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.
Could anybody tell me what I'm doing wrong ?

Quoted 8 lines

Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
I'll check if perf probe actually looks in the buildid cache, till then, please try specifying it directly,

Does anybody know where I can report this issue to Fedora community ?

Signature · 1 line

Francis

Arnaldo Carvalho de Melo

2010-12-13 12:58 UTC #14 lore raw

Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:

Quoted 8 lines

Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.

Quoted 7 lines

Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?

Please create a ticket at:

http://bugzilla.redhat.com

More specifically:

https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora

Mention this changeset:

http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c

It will make the -k argument work.

There is another that will look automatically at this location, but I haven't merged yet, will post here when done.

- Arnaldo

Francis Moreau

author

2010-12-13 13:09 UTC #15 lore raw

Quoted 25 lines · Arnaldo Carvalho de Melo

Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 8 lines
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:
http://bugzilla.redhat.com
More specifically:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora

Quoted 8 lines

Mention this changeset:
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
It will make the -k argument work.
There is another that will look automatically at this location, but I haven't merged yet, will post here when done.

Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.

The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.

I think it's a distrib issue, hence my question about Fedora report.

Thanks

Signature · 1 line

Francis

Masami Hiramatsu

2010-12-13 14:04 UTC #16 lore raw

Quoted 31 lines · Francis Moreau

Quoted 29 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 8 lines
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:
http://bugzilla.redhat.com
More specifically:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
Mention this changeset:
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c

Quoted 7 lines

Quoted 4 lines
It will make the -k argument work.
There is another that will look automatically at this location, but I haven't merged yet, will post here when done.
Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.

I agree with you.

Quoted 5 lines

The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.
I think it's a distrib issue, hence my question about Fedora report.

Yes, that is just for fedora people who can easily check how to reproduce and make sure the problem is here. Then they can try to solve the problem until the issue is resolved.

Thank you,

Signature · 4 lines

Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Francis Moreau

author

2010-12-13 14:16 UTC #17 lore raw

Quoted 47 lines · Masami Hiramatsu

Quoted 32 lines · Francis Moreau
Quoted 30 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 8 lines
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:
http://bugzilla.redhat.com
More specifically:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
Mention this changeset:
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
Quoted 7 lines
Quoted 4 lines
It will make the -k argument work.
There is another that will look automatically at this location, but I haven't merged yet, will post here when done.
Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.
I agree with you.
Quoted 4 lines
The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.

Quoted 6 lines

Quoted 1 line
I think it's a distrib issue, hence my question about Fedora report.
Yes, that is just for fedora people who can easily check how to reproduce and make sure the problem is here. Then they can try to solve the problem until the issue is resolved.

I created this report:

https://bugzilla.redhat.com/show_bug.cgi?id=662653

Thanks

Signature · 1 line

Francis

Arnaldo Carvalho de Melo

2010-12-13 14:22 UTC #20 lore raw

In reply to Francis Moreau

Em Mon, Dec 13, 2010 at 02:08:56PM +0100, Francis Moreau escreveu:

Quoted 18 lines

Quoted 16 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 6 lines
Quoted 5 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 4 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:

Quoted 8 lines

Quoted 1 line
http://bugzilla.redhat.com
Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.
The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.

So? The problem for me is that:

1. looks at /sys/kernel/notes and gets the running kernel build-id
2. fails to locate a vmlinux with a matching build-id
3. ignores the -k specified path
4. The mentioned patch makes it consider the -k specified path

So it should look at the -k specified path and make sure that it has a matching build-id, if it has, problem solved, no?

Quoted 1 line

I think it's a distrib issue, hence my question about Fedora report.

Well, here I have fedora 14 and:

[root@felicio /]# uname -r
2.6.35.9-64.fc14.x86_64
[root@felicio /]# rpm -qa | grep kernel-debuginfo kernel-debuginfo-2.6.35.9-64.fc14.x86_64 kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64 [root@felicio /]# perf probe -L icmp_rcv
Failed to open net/ipv4/icmp.c: No such file or directory
Error: Failed to show lines. (-2)
<icmp_rcv:0>
[root@felicio /]# rpm -ql kernel-debuginfo-common-x86_64 | grep net/ipv4/icmp.c /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/net/ipv4/icmp.c [root@felicio /]# cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/ [root@felicio linux-2.6.35.x86_64]# perf probe -L icmp_rcv | head -30 Source file is shorter than expected.
<icmp_rcv:0>

      0  int icmp_rcv(struct sk_buff *skb)
      1  {
         	struct icmphdr *icmph;
         	struct rtable *rt = skb_rtable(skb);
         	struct net *net = dev_net(rt->u.dst.dev);
         
      6  	if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
         		struct sec_path *sp = skb_sec_path(skb);
         		int nh;
         
     10  		if (!(sp && sp->xvec[sp->len - 1]->props.flags &
         				 XFRM_STATE_ICMP))
         			goto drop;
         
     14  		if (!pskb_may_pull(skb, sizeof(*icmph) + sizeof(struct iphdr)))
         			goto drop;
         
         		nh = skb_network_offset(skb);
         		skb_set_network_header(skb, sizeof(*icmph));
         
     20  		if (!xfrm4_policy_check_reverse(NULL, XFRM_POLICY_IN, skb))
         			goto drop;
         
         		skb_set_network_header(skb, nh);
         	}
         
     26  	ICMP_INC_STATS_BH(net, ICMP_MIB_INMSGS);

[root@felicio linux-2.6.35.x86_64]# cat /proc/net/snmp | grep Icmp:
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 4 1 2 0 0 0 0 1 1 0 0 0 0 119 0 116 0 0 0 0 3 0 0 0 0 0
[root@felicio linux-2.6.35.x86_64]# perf probe icmp_rcv:26
Add new event:
probe:icmp_rcv (on icmp_rcv:26)

You can now use it on all perf tools, such as:

perf record -e probe:icmp_rcv -aR sleep 1

[root@felicio linux-2.6.35.x86_64]# perf record -e probe:icmp_rcv -ag ^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.535 MB perf.data (~23357 samples) ]

[root@felicio linux-2.6.35.x86_64]# perf report
# Events: 3 cycles
#
# Overhead Command Shared Object Symbol

# ........  ...............  .................  ......
#
    66.67%          swapper  [kernel.kallsyms]  [k] icmp_rcv
                    |
                    --- icmp_rcv
                        ip_local_deliver_finish
                        NF_HOOK.clone.8
                        ip_local_deliver
                        ip_rcv_finish
                        NF_HOOK.clone.8
                        ip_rcv
                        __netif_receive_skb
                        netif_receive_skb
                        napi_skb_finish
                        napi_gro_receive
                        igb_poll
                        net_rx_action
                        __do_softirq
                        call_softirq
                        do_softirq
                        irq_exit
                        do_IRQ
                        common_interrupt
                        default_idle
                        cpu_idle
                        start_secondary

    33.33%              ssh  [kernel.kallsyms]  [k] icmp_rcv
                        |
                        --- icmp_rcv
                            ip_local_deliver_finish
                            NF_HOOK.clone.8
                            ip_local_deliver
                            ip_rcv_finish
                            NF_HOOK.clone.8
                            ip_rcv
                            __netif_receive_skb
                            netif_receive_skb
                            napi_skb_finish
                            napi_gro_receive
                            igb_poll
                            net_rx_action
                            __do_softirq
                            call_softirq

[root@felicio linux-2.6.35.x86_64]#
[root@felicio linux-2.6.35.x86_64]# cat /proc/net/snmp | grep Icmp:
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 7 1 2 0 0 0 0 4 1 0 0 0 0 123 0 117 0 0 0 0 3 3 0 0 0 0
[root@felicio linux-2.6.35.x86_64]#

All seems to work, right?

There are two problems here:

on the 'report' header, as we didn't collect "cycles", but "probe:icmp_rcv" events, that is another fix that needs to be backported to fedora, but a minor one.

Also 'perf probe -L' should look use /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for sources.

That can be found using:

[root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5

    <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64

[root@felicio linux-2.6.35.x86_64]#

<0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)

    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
    <99>   DW_AT_low_pc      : 0x0      
    <a1>   DW_AT_entry_pc    : 0x0      
    <a9>   DW_AT_ranges      : 0x340    
    <ad>   DW_AT_stmt_list   : 0x114

I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags, Masami?

All this with:

[root@felicio linux-2.6.35.x86_64]# type perf perf is hashed (/usr/bin/perf)
[root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/bin/perf perf-2.6.35.9-64.fc14.x86_64
[root@felicio linux-2.6.35.x86_64]#

I.e. the perf binary shipping in fedora, without the recent patches.

To finish the session:

[root@felicio linux-2.6.35.x86_64]# perf probe --list
probe:icmp_rcv (on icmp_rcv:26@ipv4/icmp.c)
[root@felicio linux-2.6.35.x86_64]# perf probe --del icmp_rcv Remove event: probe:icmp_rcv
[root@felicio linux-2.6.35.x86_64]# perf probe --list [root@felicio linux-2.6.35.x86_64]#

And if you run it in verbose mode we can see what was the vmlinux file used:

[root@felicio linux-2.6.35.x86_64]# perf probe -v icmp_rcv:26 probe-definition(0): icmp_rcv:26
symbol:icmp_rcv file:(null) line:26 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (6 entries long)
Using /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux for symbols Try to open /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux Probe line found: line[571]:999 addr:0xffffffff81416655
Probe point found: icmp_rcv+208
find 1 kprobe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Add new event:
Writing event: p:probe/icmp_rcv icmp_rcv+208
probe:icmp_rcv (on icmp_rcv:26)

You can now use it on all perf tools, such as:

perf record -e probe:icmp_rcv -aR sleep 1

[root@felicio linux-2.6.35.x86_64]#

It was /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux, that is present on:

[root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux kernel-debuginfo-2.6.35.9-64.fc14.x86_64

And...

[root@felicio linux-2.6.35.x86_64]# perf buildid-list 882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]

This is the build-id of the running kernel, and:

[root@felicio linux-2.6.35.x86_64]# perf buildid-cache -v --add /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux Adding 882b1b53eb1d653200e8fa7100273aa8493896c4 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux: FAIL /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux already in the cache
[root@felicio linux-2.6.35.x86_64]#

The running kernel build-id matches the build-id in the vmlinux file found in the kernel-debuginfo package.

So can you please give us the versions for the kernel, kernel-debuginfo package? Perhaps you have a new debuginfo package and is running an older kernel?

Use the perf buildid-list trick to get the build-id in readable format.

Thanks,

- Arnaldo

Arnaldo Carvalho de Melo

2010-12-13 14:25 UTC #19 lore raw

In reply to Masami Hiramatsu

Em Mon, Dec 13, 2010 at 11:04:36PM +0900, Masami Hiramatsu escreveu:

Quoted 16 lines

Quoted 15 lines · Francis Moreau
Quoted 8 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 7 lines
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.

Quoted 6 lines

Quoted 1 line
I think it's a distrib issue, hence my question about Fedora report.
Yes, that is just for fedora people who can easily check how to reproduce and make sure the problem is here. Then they can try to solve the problem until the issue is resolved.

I just tried it, works for me.

I added the sequence of commands, perhaps he can find where the problem lies following them.

- Arnaldo

Francis Moreau

author

2010-12-13 15:11 UTC #21 lore raw

In reply to Arnaldo Carvalho de Melo

Quoted 31 lines · Arnaldo Carvalho de Melo

Em Mon, Dec 13, 2010 at 02:08:56PM +0100, Francis Moreau escreveu:
Quoted 19 lines
Quoted 17 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 7 lines
Quoted 5 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 4 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:
Quoted 8 lines
Quoted 1 line
http://bugzilla.redhat.com
Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.
The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.
So? The problem for me is that:

Quoted 7 lines

1. looks at /sys/kernel/notes and gets the running kernel build-id
2. fails to locate a vmlinux with a matching build-id
3. ignores the -k specified path
4. The mentioned patch makes it consider the -k specified path
So it should look at the -k specified path and make sure that it has a matching build-id, if it has, problem solved, no?

But as I said, there's no matching build-id. I understand that -k is currently broken but suppose that it works, which vmlinux should I use then ?

$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64

$ rpm -ql kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 | grep vmlinux
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

So the debug package installed the kernel image with debug info into

/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug

directory. Please note that this path can't be found by perf without the recent patches.

To check the build id of this image, I do:

$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 ...

This is the only way I know to dump the debug id of an object.

The buildid of the running kernel:

$ perf buildid-list
882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]

So they both don't match.

Quoted 5 lines

Quoted 3 lines
I think it's a distrib issue, hence my question about Fedora report.
Well, here I have fedora 14 and:

[...]

Quoted 10 lines

All this with:
[root@felicio linux-2.6.35.x86_64]# type perf perf is hashed (/usr/bin/perf)
[root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/bin/perf perf-2.6.35.9-64.fc14.x86_64
[root@felicio linux-2.6.35.x86_64]#
I.e. the perf binary shipping in fedora, without the recent patches.

In that case how perf can find the right vmlinux in your system since it needs to be patched to find the vmlinux image in the /usr/lib/debug/lib/modules/%s.debug directory ?

Quoted 4 lines

To finish the session:

[...]

Quoted 11 lines

And...
[root@felicio linux-2.6.35.x86_64]# perf buildid-list 882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]
This is the build-id of the running kernel, and:
[root@felicio linux-2.6.35.x86_64]# perf buildid-cache -v --add /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux Adding 882b1b53eb1d653200e8fa7100273aa8493896c4 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux: FAIL /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux already in the cache

Ah, that's interesting...

In the path you're using, you loose the '.debug', therefore this image doesn't seem to belong to kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64.

And that explains how perf can find the debug kernel image.

Signature · 1 line

Francis

Arnaldo Carvalho de Melo

2010-12-13 16:40 UTC #22 lore raw

Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:

Quoted 10 lines

But as I said, there's no matching build-id. I understand that -k is currently broken but suppose that it works, which vmlinux should I use then ?
$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64

Bzzt, here is the problem: you're not using the right package, kernel-debug-debuginfo is for the kernel-debug package, a kernel built with lots of CONFIG_DEBUG_ options turned on, please install the right package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64

Quoted 6 lines

$ rpm -ql kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 | grep vmlinux
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

So the debug package installed the kernel image with debug info into

   /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug

Right, this matches what is in the kernel in the kernel-debug package.

Please install the right debuginfo package and try again

- Arnaldo

Arnaldo Carvalho de Melo

2010-12-13 16:40 UTC #12 lore raw

In reply to Franck Bui-Huu

Em Sun, Dec 12, 2010 at 11:15:26PM +0100, Franck Bui-Huu escreveu:

Quoted 11 lines

Quoted 8 lines · Arnaldo Carvalho de Melo
Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:
Quoted 4 lines
Looks like a standard path is missing...
What about adding the one above since it looks like Fedora is installing their debug kernel images at this place ?
Can I get your signed-off-by on this one?

Quoted 7 lines

Of course.
[PATCH] perf-tools: add one more default search path for kernel image
From: Franck Bui-Huu <[email protected]>
It looks like Fedora is using this one.

Nope, this is not needed, see the rest of the discussion please.

- Arnaldo

Francis Moreau

author

2010-12-13 19:08 UTC #23 lore raw

In reply to Arnaldo Carvalho de Melo

Quoted 13 lines · Arnaldo Carvalho de Melo

Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
Quoted 11 lines
But as I said, there's no matching build-id. I understand that -k is currently broken but suppose that it works, which vmlinux should I use then ?
$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64

Quoted 4 lines

Bzzt, here is the problem: you're not using the right package, kernel-debug-debuginfo is for the kernel-debug package, a kernel built with lots of CONFIG_DEBUG_ options turned on, please install the right package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64

Ah, that's it.

Thanks !

Signature · 1 line

Francis

Francis Moreau

author

2010-12-13 19:08 UTC #18 lore raw

In reply to Francis Moreau

Quoted 51 lines · Francis Moreau

Quoted 49 lines · Masami Hiramatsu
Quoted 32 lines · Francis Moreau
Quoted 30 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
Quoted 8 lines
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
Quoted 5 lines
I'm trying to use perf-probe(1) with the latest F14 kernel (2.6.35.9-64.fc14.x86_64).
I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which contains the running vmlinux image AFAICT.
Quoted 7 lines
Quoted 5 lines
Can you try using:
$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
So that it tries it directly instead of looking into the buildid cache?
Does anybody know where I can report this issue to Fedora community ?
Please create a ticket at:
http://bugzilla.redhat.com
More specifically:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
Mention this changeset:
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
Quoted 7 lines
Quoted 4 lines
It will make the -k argument work.
There is another that will look automatically at this location, but I haven't merged yet, will post here when done.
Hmm, I don't think that any patches recently posted in response of this thread will solve the problem.
I agree with you.
Quoted 5 lines
The primarily concern here, is that the running kernel has a different build-id than the debug image installed by the corresponding debug package.
I think it's a distrib issue, hence my question about Fedora report.

Quoted 9 lines

Quoted 4 lines
Yes, that is just for fedora people who can easily check how to reproduce and make sure the problem is here. Then they can try to solve the problem until the issue is resolved.
I created this report:
https://bugzilla.redhat.com/show_bug.cgi?id=662653

Now closed.

Signature · 1 line

Francis

Arnaldo Carvalho de Melo

2010-12-13 19:17 UTC #24 lore raw

In reply to Francis Moreau

Em Mon, Dec 13, 2010 at 08:08:08PM +0100, Francis Moreau escreveu:

Quoted 14 lines

Quoted 11 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
Quoted 10 lines
But as I said, there's no matching build-id. I understand that -k is currently broken but suppose that it works, which vmlinux should I use then ?
$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64

Quoted 6 lines

Quoted 4 lines
Bzzt, here is the problem: you're not using the right package, kernel-debug-debuginfo is for the kernel-debug package, a kernel built with lots of CONFIG_DEBUG_ options turned on, please install the right package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64
Ah, that's it.

Yeah, those package names are a bit confusing :-)

Please let us know about any other problem you have using these tools, any feedback is _greatly_ appreciated.

- Arnaldo

Francis Moreau

author

2010-12-13 19:32 UTC #25 lore raw

Quoted 21 lines · Arnaldo Carvalho de Melo

Em Mon, Dec 13, 2010 at 08:08:08PM +0100, Francis Moreau escreveu:
Quoted 20 lines
Quoted 17 lines · Arnaldo Carvalho de Melo
Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
Quoted 10 lines
But as I said, there's no matching build-id. I understand that -k is currently broken but suppose that it works, which vmlinux should I use then ?
$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64
Bzzt, here is the problem: you're not using the right package, kernel-debug-debuginfo is for the kernel-debug package, a kernel built with lots of CONFIG_DEBUG_ options turned on, please install the right package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64

Quoted 6 lines

Quoted 1 line
Ah, that's it.
Yeah, those package names are a bit confusing :-)
Please let us know about any other problem you have using these tools, any feedback is _greatly_ appreciated.

Don't worry, I won't miss an opportunity.

Thanks

Signature · 1 line

Francis

Masami Hiramatsu

2010-12-15 08:57 UTC #26 lore raw

In reply to Arnaldo Carvalho de Melo

Quoted 30 lines · Arnaldo Carvalho de Melo

There are two problems here:
on the 'report' header, as we didn't collect "cycles", but "probe:icmp_rcv" events, that is another fix that needs to be backported to fedora, but a minor one.
Also 'perf probe -L' should look use /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for sources.
That can be found using:
[root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
    <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
[root@felicio linux-2.6.35.x86_64]#
<0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
    <99>   DW_AT_low_pc      : 0x0      
    <a1>   DW_AT_entry_pc    : 0x0      
    <a9>   DW_AT_ranges      : 0x340    
    <ad>   DW_AT_stmt_list   : 0x114

Quoted 2 lines

I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags, Masami?

It already supports DW_AT_comp_dir on upstream kernel, see find_line_range() in utils/probe-finder.c ;-)

Thank you,

Signature · 4 lines

Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Arnaldo Carvalho de Melo

2010-12-15 18:54 UTC #27 lore raw

Em Wed, Dec 15, 2010 at 05:57:11PM +0900, Masami Hiramatsu escreveu:

Quoted 13 lines

Quoted 11 lines · Arnaldo Carvalho de Melo
Also 'perf probe -L' should look use /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for sources.
That can be found using:
<0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 

Quoted 5 lines

Quoted 2 lines
I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags, Masami?
It already supports DW_AT_comp_dir on upstream kernel, see find_line_range() in utils/probe-finder.c ;-)

I happily stand corrected, was using the f14 one :-)

Great work!

- Arnaldo

Franck Bui-Huu

2010-12-23 13:59 UTC #28 lore raw

In reply to Masami Hiramatsu

Hello Masami,

Quoted 32 lines · Masami Hiramatsu

Quoted 30 lines · Arnaldo Carvalho de Melo
There are two problems here:
on the 'report' header, as we didn't collect "cycles", but "probe:icmp_rcv" events, that is another fix that needs to be backported to fedora, but a minor one.
Also 'perf probe -L' should look use /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for sources.
That can be found using:
[root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
    <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
[root@felicio linux-2.6.35.x86_64]#
<0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
    <99>   DW_AT_low_pc      : 0x0      
    <a1>   DW_AT_entry_pc    : 0x0      
    <a9>   DW_AT_ranges      : 0x340    
    <ad>   DW_AT_stmt_list   : 0x114

Quoted 6 lines

Quoted 2 lines
I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags, Masami?
It already supports DW_AT_comp_dir on upstream kernel, see find_line_range() in utils/probe-finder.c ;-)

It looks like adding probe with lazy matching pattern does not:

   $ cd /dev
   $ perf probe -n -a "icmp.c;sk=*"
   Failed to open net/ipv4/icmp.c: Operation not permitted
     Error: Failed to add events. (-2)
   $ cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/
   $ perf probe -n -a "icmp.c;sk=*"
   Add new events:
     probe:icmp_push_reply (on @cmp.c)
     probe:icmp_reply     (on @cmp.c)
     probe:icmp_reply_1   (on @cmp.c)
     probe:icmp_send      (on @cmp.c)
     probe:icmp_send_1    (on @cmp.c)
     probe:icmp_error     (on @cmp.c)
     probe:icmp_error_1   (on @cmp.c)
     probe:icmp_error_2   (on @cmp.c)
     probe:icmp_error_3   (on @cmp.c)

You can now use it on all perf tools, such as:

perf record -e probe:icmp_error_3 -aR sleep 1

Below is a quick and dirty fix (sorry, I don't have any time to make it better for now), also note that the note of the file (icmp.c) reported is screwed...

tools/perf/util/probe-event.c +1 -1

index 10ad1ad..53c2f97 100644
@@ -241,7 +241,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
  * a newly allocated path on success.
  * Return 0 if file was found and readable, -errno otherwise.
  */
-static int get_real_path(const char *raw_path, const char *comp_dir,
+int get_real_path(const char *raw_path, const char *comp_dir,
 			 char **new_path)
 {
 	const char *prefix = symbol_conf.source_prefix;

tools/perf/util/probe-finder.c +9 -1

index 3991d73..88829ef 100644
@@ -1126,9 +1126,17 @@ static int find_probe_point_lazy(Dwarf_Die *sp_die, struct probe_finder *pf)
 	int ret = 0;
 
 	if (list_empty(&pf->lcache)) {
+		const char *comp_dir = cu_get_comp_dir(&pf->cu_die);
+		char *fullpath;
+
+		ret = get_real_path(pf->fname, comp_dir, &fullpath);
+		if (ret < 0)
+			return ret;
+
 		/* Matching lazy line pattern */
-		ret = find_lazy_match_lines(&pf->lcache, pf->fname,
+		ret = find_lazy_match_lines(&pf->lcache, fullpath,
 					    pf->pev->point.lazy_line);
+		free(fullpath);
 		if (ret == 0) {
 			pr_debug("No matched lines found in %s.\n", pf->fname);
 			return 0;

tools/perf/util/probe-finder.h +3 -0

index bba69d4..665938a 100644
@@ -15,6 +15,9 @@ static inline int is_c_varname(const char *name)
 	return isalpha(name[0]) || name[0] == '_';
 }
 
+extern int get_real_path(const char *raw_path, const char *comp_dir,
+		  char **new_path);
+
 #ifdef DWARF_SUPPORT
 /* Find probe_trace_events specified by perf_probe_event from debuginfo */
 extern int find_probe_trace_events(int fd, struct perf_probe_event *pev,

Signature · 1 line

		Franck

Franck Bui-Huu

2010-12-27 20:50 UTC #29 lore raw

Masami,

It seems you forget this one ....

Thanks

Quoted 120 lines · Franck Bui-Huu

Hello Masami,

Quoted 32 lines · Masami Hiramatsu

Quoted 30 lines · Arnaldo Carvalho de Melo
There are two problems here:
on the 'report' header, as we didn't collect "cycles", but "probe:icmp_rcv" events, that is another fix that needs to be backported to fedora, but a minor one.
Also 'perf probe -L' should look use /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for sources.
That can be found using:
[root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
    <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
[root@felicio linux-2.6.35.x86_64]#
<0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
    <99>   DW_AT_low_pc      : 0x0      
    <a1>   DW_AT_entry_pc    : 0x0      
    <a9>   DW_AT_ranges      : 0x340    
    <ad>   DW_AT_stmt_list   : 0x114

Quoted 6 lines

Quoted 2 lines
I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags, Masami?
It already supports DW_AT_comp_dir on upstream kernel, see find_line_range() in utils/probe-finder.c ;-)

It looks like adding probe with lazy matching pattern does not:

   $ cd /dev
   $ perf probe -n -a "icmp.c;sk=*"
   Failed to open net/ipv4/icmp.c: Operation not permitted
     Error: Failed to add events. (-2)
   $ cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/
   $ perf probe -n -a "icmp.c;sk=*"
   Add new events:
     probe:icmp_push_reply (on @cmp.c)
     probe:icmp_reply     (on @cmp.c)
     probe:icmp_reply_1   (on @cmp.c)
     probe:icmp_send      (on @cmp.c)
     probe:icmp_send_1    (on @cmp.c)
     probe:icmp_error     (on @cmp.c)
     probe:icmp_error_1   (on @cmp.c)
     probe:icmp_error_2   (on @cmp.c)
     probe:icmp_error_3   (on @cmp.c)

You can now use it on all perf tools, such as:

perf record -e probe:icmp_error_3 -aR sleep 1

Below is a quick and dirty fix (sorry, I don't have any time to make it better for now), also note that the note of the file (icmp.c) reported is screwed...

tools/perf/util/probe-event.c +1 -1

index 10ad1ad..53c2f97 100644
@@ -241,7 +241,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
  * a newly allocated path on success.
  * Return 0 if file was found and readable, -errno otherwise.
  */
-static int get_real_path(const char *raw_path, const char *comp_dir,
+int get_real_path(const char *raw_path, const char *comp_dir,
 			 char **new_path)
 {
 	const char *prefix = symbol_conf.source_prefix;

tools/perf/util/probe-finder.c +9 -1

index 3991d73..88829ef 100644
@@ -1126,9 +1126,17 @@ static int find_probe_point_lazy(Dwarf_Die *sp_die, struct probe_finder *pf)
 	int ret = 0;
 
 	if (list_empty(&pf->lcache)) {
+		const char *comp_dir = cu_get_comp_dir(&pf->cu_die);
+		char *fullpath;
+
+		ret = get_real_path(pf->fname, comp_dir, &fullpath);
+		if (ret < 0)
+			return ret;
+
 		/* Matching lazy line pattern */
-		ret = find_lazy_match_lines(&pf->lcache, pf->fname,
+		ret = find_lazy_match_lines(&pf->lcache, fullpath,
 					    pf->pev->point.lazy_line);
+		free(fullpath);
 		if (ret == 0) {
 			pr_debug("No matched lines found in %s.\n", pf->fname);
 			return 0;

tools/perf/util/probe-finder.h +3 -0

index bba69d4..665938a 100644
@@ -15,6 +15,9 @@ static inline int is_c_varname(const char *name)
 	return isalpha(name[0]) || name[0] == '_';
 }
 
+extern int get_real_path(const char *raw_path, const char *comp_dir,
+		  char **new_path);
+
 #ifdef DWARF_SUPPORT
 /* Find probe_trace_events specified by perf_probe_event from debuginfo */
 extern int find_probe_trace_events(int fd, struct perf_probe_event *pev,

Signature · 1 line

		Franck