Skip to contentSkip to conversation

perf probe: failed to add events

bug report In review Joseph Qi 6 participants 16 messages Feb 26, 2019 – Mar 25, 2019 perf events

lore
Share & tools
lore

Joseph Qi

author
#0 lore raw

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.

I've also tried kernel v4.20, and it can pass.

So I've bisected and finally found the first good commit is:
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline which is based on another commit:
98f05b5138f0 Use the TSS sp2 slot for SYSCALL/SYSRET scratch space

Once I've backpoted these two commits into 4.19.24, the above case can pass, though I'm not sure how it is fixed.
So is there any plan to let them go into stable as well?

Thanks,
Joseph

#1 lore raw
Quoted 10 lines · Joseph Qi

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

Quoted 8 lines

So I've bisected and finally found the first good commit is:
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline which is based on another commit:
98f05b5138f0 Use the TSS sp2 slot for SYSCALL/SYSRET scratch space

Once I've backpoted these two commits into 4.19.24, the above case can pass, though I'm not sure how it is fixed.
So is there any plan to let them go into stable as well?

If they are needed, I'll gladly queue them up, but this feels like something might have broken, so it should be easier to just revert the offending commit instead.

Andy, any ideas?

thanks,

greg k-h

author
#2 lore raw
Quoted 6 lines · Greg KH
Quoted 5 lines · Joseph Qi

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

Quoted 9 lines
Quoted 5 lines
$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

From my test, v4.19.0 also has this issue.
Bisect locates that it is introduced by commit bf904d2762ee "x86/pti/64: Remove the SYSCALL64 entry trampoline".

Thanks,
Joseph

Quoted 19 lines
Quoted 8 lines

So I've bisected and finally found the first good commit is:
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline which is based on another commit:
98f05b5138f0 Use the TSS sp2 slot for SYSCALL/SYSRET scratch space

Once I've backpoted these two commits into 4.19.24, the above case can pass, though I'm not sure how it is fixed.
So is there any plan to let them go into stable as well?

If they are needed, I'll gladly queue them up, but this feels like something might have broken, so it should be easier to just revert the offending commit instead.

Andy, any ideas?

thanks,

greg k-h

#3 lore raw
Quoted 13 lines · Joseph Qi
Quoted 10 lines · Greg KH
Quoted 9 lines · Joseph Qi

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

From my test, v4.19.0 also has this issue.

Bisect locates that it is introduced by commit bf904d2762ee "x86/pti/64: Remove the SYSCALL64 entry trampoline".

But that commit was in 4.20, not 4.19. So if this never worked, it's not a regression?

confused,

greg k-h

#4 lore raw

Em Tue, Feb 26, 2019 at 02:08:02PM +0100, Greg KH escreveu:

Quoted 19 lines · Joseph Qi
Quoted 18 lines · Greg KH
Quoted 15 lines · Joseph Qi
Quoted 10 lines

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

From my test, v4.19.0 also has this issue.

Bisect locates that it is introduced by commit bf904d2762ee "x86/pti/64: Remove the SYSCALL64 entry trampoline".

But that commit was in 4.20, not 4.19. So if this never worked, it's not a regression?

confused,

Adrian, Ideas?

- Arnaldo

reviewer
#5 lore raw
Quoted 24 lines · Arnaldo Carvalho de Melo

Em Tue, Feb 26, 2019 at 02:08:02PM +0100, Greg KH escreveu:

Quoted 23 lines · Joseph Qi
Quoted 21 lines · Greg KH
Quoted 10 lines · Joseph Qi
Quoted 9 lines

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.
Quoted 6 lines
Quoted 1 line

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

From my test, v4.19.0 also has this issue.

Bisect locates that it is introduced by commit bf904d2762ee "x86/pti/64: Remove the SYSCALL64 entry trampoline".

Quoted 7 lines
Quoted 4 lines

But that commit was in 4.20, not 4.19. So if this never worked, it's not a regression?

confused,

Adrian, Ideas?

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index e86f8be89157..6cd96f9b346d 100644
@@ -157,8 +157,10 @@ static struct map *kernel_get_module_map(const char *module)
 	if (module && strchr(module, '/'))
 		return dso__new_map(module);
 
-	if (!module)
-		module = "kernel";
+	if (!module) {
+		pos = machine__kernel_map(host_machine);
+		return map__get(pos);
+	}
 
 	for (pos = maps__first(maps); pos; pos = map__next(pos)) {
 		/* short_name is "[module]" */
Signature · 1 line
2.19.1
author Tested-by
#6 lore raw

Hi Adrian,

Quoted 6 lines · Adrian Hunter

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Quoted 8 lines

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")

Below is my investigation result before, FYI.
the first bad commit (v4.18 ~ v4.19):
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines revert this commit on 4.19.0, it works.
the first good commit again (v4.19 ~ v4.20):
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline backported this commit as well as the related commit 98f05b5138f0 on v4.19.24, it works.

And I've tested your fix on v4.19.24, it also works.

reviewer
#7 lore raw
Quoted 13 lines · Joseph Qi

Hi Adrian,

Quoted 10 lines · Adrian Hunter

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Quoted 8 lines
Quoted 4 lines

Fix by using machine__kernel_map() to get the main kernel map.

Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")

Below is my investigation result before, FYI.
the first bad commit (v4.18 ~ v4.19):
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines

Yes we should add a fixes tag for that also.

Quoted 9 lines

revert this commit on 4.19.0, it works.
the first good commit again (v4.19 ~ v4.20):
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline backported this commit as well as the related commit 98f05b5138f0 on v4.19.24, it works.

And I've tested your fix on v4.19.24, it also works.

author
#8 lore raw

Hi Adrian,

Quoted 16 lines · Adrian Hunter
Quoted 15 lines · Joseph Qi

Hi Adrian,

Quoted 12 lines · Adrian Hunter

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")

Below is my investigation result before, FYI.
the first bad commit (v4.18 ~ v4.19):
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines

Yes we should add a fixes tag for that also.

Have you sent out the patch officially?
I can't find it in the mail list.

Thanks,
Joseph

Quoted 9 lines
Quoted 9 lines

revert this commit on 4.19.0, it works.
the first good commit again (v4.19 ~ v4.20):
bf904d2762ee x86/pti/64: Remove the SYSCALL64 entry trampoline backported this commit as well as the related commit 98f05b5138f0 on v4.19.24, it works.

And I've tested your fix on v4.19.24, it also works.

reviewer
#9 lore raw

PATCH 1/1 perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
Fixes d83212d5dd67 ("kallsyms, x86: Export addresses of PTI entry trampolines")
---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index e86f8be89157..6cd96f9b346d 100644
@@ -157,8 +157,10 @@ static struct map *kernel_get_module_map(const char *module)
 	if (module && strchr(module, '/'))
 		return dso__new_map(module);
 
-	if (!module)
-		module = "kernel";
+	if (!module) {
+		pos = machine__kernel_map(host_machine);
+		return map__get(pos);
+	}
 
 	for (pos = maps__first(maps); pos; pos = map__next(pos)) {
 		/* short_name is "[module]" */
Signature · 1 line
2.19.1
#10 lore raw

Re: [PATCH] perf probe: Fix getting the kernel map

Em Mon, Mar 04, 2019 at 03:13:21PM +0200, Adrian Hunter escreveu:

Quoted 5 lines

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Masami, can I get your Acked-by?

- Arnaldo

Quoted 28 lines
Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
Fixes d83212d5dd67 ("kallsyms, x86: Export addresses of PTI entry trampolines")
---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index e86f8be89157..6cd96f9b346d 100644
@@ -157,8 +157,10 @@ static struct map *kernel_get_module_map(const char *module)
 	if (module && strchr(module, '/'))
 		return dso__new_map(module);
 
-	if (!module)
-		module = "kernel";
+	if (!module) {
+		pos = machine__kernel_map(host_machine);
+		return map__get(pos);
+	}
 
 	for (pos = maps__first(maps); pos; pos = map__next(pos)) {
 		/* short_name is "[module]" */
Signature · 1 line
2.19.1
Signature · 1 line
- Arnaldo
#11 lore raw

Re: [PATCH] perf probe: Fix getting the kernel map

Quoted 8 lines · On Fri, 8 Mar 2019 11:39:26 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Mon, Mar 04, 2019 at 03:13:21PM +0200, Adrian Hunter escreveu:

Quoted 5 lines

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Masami, can I get your Acked-by?

Yes, this looks good to me :)

Thank you Adrian!

Quoted 35 lines

- Arnaldo

Quoted 28 lines
Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
Fixes d83212d5dd67 ("kallsyms, x86: Export addresses of PTI entry trampolines")
---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index e86f8be89157..6cd96f9b346d 100644
@@ -157,8 +157,10 @@ static struct map *kernel_get_module_map(const char *module)
 	if (module && strchr(module, '/'))
 		return dso__new_map(module);
 
-	if (!module)
-		module = "kernel";
+	if (!module) {
+		pos = machine__kernel_map(host_machine);
+		return map__get(pos);
+	}
 
 	for (pos = maps__first(maps); pos; pos = map__next(pos)) {
 		/* short_name is "[module]" */
Signature · 1 line
2.19.1
Signature · 1 line
- Arnaldo
Signature · 1 line
Masami Hiramatsu <[email protected]>
#12 lore raw

[tip:perf/urgent] perf probe: Fix getting the kernel map

In reply to Adrian Hunter
perf probe: Fix getting the kernel map
Commit-ID:  eaeffeb9838a7c0dec981d258666bfcc0fa6a947
Gitweb:     https://git.kernel.org/tip/eaeffeb9838a7c0dec981d258666bfcc0fa6a947
Author:     Adrian Hunter <[email protected]>
AuthorDate: Mon, 4 Mar 2019 15:13:21 +0200
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Cc Alexander Shishkin <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Jiufei Xue <[email protected]>
Cc Peter Zijlstra <[email protected]>
Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
Fixes d83212d5dd67 ("kallsyms, x86: Export addresses of PTI entry trampolines")
---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index a1b8d9649ca7..198e09ff611e 100644
@@ -160,8 +160,10 @@ static struct map *kernel_get_module_map(const char *module)
 	if (module && strchr(module, '/'))
 		return dso__new_map(module);
 
-	if (!module)
-		module = "kernel";
+	if (!module) {
+		pos = machine__kernel_map(host_machine);
+		return map__get(pos);
+	}
 
 	for (pos = maps__first(maps); pos; pos = map__next(pos)) {
 		/* short_name is "[module]" */
#13 lore raw
In reply to Adrian Hunter
Quoted 16 lines · Adrian Hunter
Quoted 15 lines · Joseph Qi

Hi Adrian,

Quoted 12 lines · Adrian Hunter

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")

Below is my investigation result before, FYI.
the first bad commit (v4.18 ~ v4.19):
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines

Yes we should add a fixes tag for that also.

So, what do I need to do here for the stable tree(s) to resolve this issue?

thanks,

greg k-h

reviewer
#14 lore raw
Quoted 20 lines · Greg KH
Quoted 19 lines · Adrian Hunter
Quoted 18 lines · Joseph Qi

Hi Adrian,

Quoted 14 lines · Adrian Hunter

Seems to be fixed by this:

From: Adrian Hunter <[email protected]>
Date: Wed, 27 Feb 2019 05:35:25 +0200
Subject: [PATCH] perf probe: Fix getting the kernel map

Since commit 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines"), perf tools has been creating more than one kernel map, however 'perf probe' assumed there could be only one.

Fix by using machine__kernel_map() to get the main kernel map.

Fixes 4d99e4136580 ("perf machine: Workaround missing maps for x86 PTI entry trampolines")
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Below is my investigation result before, FYI.
the first bad commit (v4.18 ~ v4.19):
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines

Yes we should add a fixes tag for that also.

So, what do I need to do here for the stable tree(s) to resolve this issue?

Nothing.

Patch was re-sent with a stable tag so will go through like normal:

	https://lore.kernel.org/lkml/[email protected]/
author
#15 lore raw
In reply to Greg KH
Quoted 19 lines · Greg KH
Quoted 18 lines · Joseph Qi
Quoted 15 lines · Greg KH
Quoted 10 lines · Joseph Qi

Hi,

I'm using kernel v4.19.24 and have found that there is an issue when using perf probe to define a new dynamic tracepoint.

$ perf probe -a handle_mm_fault
Failed to write event: Numerical result out of range
  Error: Failed to add events.

I've also tried kernel v4.20, and it can pass.

Ick, has this ever worked on the 4.19 stable tree? If so, any chance you can run 'git bisect' to find the offending commit?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

From my test, v4.19.0 also has this issue.

Bisect locates that it is introduced by commit bf904d2762ee "x86/pti/64: Remove the SYSCALL64 entry trampoline".

But that commit was in 4.20, not 4.19. So if this never worked, it's not a regression?

confused,

Oops, my fault.
The first bad commit is:
d83212d5dd67 kallsyms, x86: Export addresses of PTI entry trampolines

It is between v4.18 and v4.19.

Quoted 3 lines

greg k-h