Skip to contentSkip to conversation

perf test: Fix probe testsuite with a new error message

PATCH In review Namhyung Kim 3 participants 7 messages Oct 10, 2024 – Oct 16, 2024 1 message not in the archive perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH 1/1 perf test: Fix probe testsuite with a new error message

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.
Cc Masami Hiramatsu <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Veronika Molnarova <[email protected]>
---
 tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh +2 -1
index b5dc10b2a73810b3..01e5e09053c37e46 100755
@@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
 REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
 REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
 REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
-../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
+REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
+../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
 CHECK_EXIT_CODE=$?
 
 print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
Signature · 1 line
2.47.0.rc0.187.ge670bccf7e-goog
#1 lore raw
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

I'll try to find out the threads and see what happened.

- Arnaldo

Quoted 24 lines
Cc Masami Hiramatsu <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Veronika Molnarova <[email protected]>
---
 tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh +2 -1
index b5dc10b2a73810b3..01e5e09053c37e46 100755
@@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
 REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
 REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
 REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
-../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
+REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
+../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
 CHECK_EXIT_CODE=$?
 
 print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
Signature · 1 line
2.47.0.rc0.187.ge670bccf7e-goog
#2 lore raw
Quoted 19 lines · Arnaldo Carvalho de Melo
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.
Quoted 8 lines

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

I'll try to find out the threads and see what happened.

https://lore.kernel.org/all/[email protected]/T/#u

Veronika, was there a v3?

Thanks,

- Arnaldo

#3 lore raw
Quoted 27 lines · Arnaldo Carvalho de Melo
Quoted 26 lines · Arnaldo Carvalho de Melo
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

Quoted 7 lines
Quoted 1 line

I'll try to find out the threads and see what happened.

https://lore.kernel.org/all/[email protected]/T/#u

Veronika, was there a v3?

Thanks,

Well the patch was already applied upstream so we are talking about a fix not v3, right? As Arnaldo mentioned, I got together a fix patch that checks the CU for the Assembly for blacklisted probe. Michael also suggested the possibility to check multiple blacklisted probes if we are not able to check for the Assembly, so I also added that.

I will send the fix patch soon as part of the perftool upstreaming patchset to prevent conflicts happening. Hopefully, this can be at the start of the next week if all of the checks pass.

Thanks,
Veronika

Quoted 3 lines

- Arnaldo

A message that is not in the archive (sent off-list or not yet received): CA+JHD91skZHrEanhANQzWaq=4wpZomTn8Vab4j+wT2kHdibTkA@mail.gmail.com

#5 lore raw
Quoted 46 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · Veronika Molnarova
Quoted 29 lines · Arnaldo Carvalho de Melo
Quoted 27 lines · Arnaldo Carvalho de Melo
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

I'll try to find out the threads and see what happened.

Quoted 7 lines
Quoted 5 lines

https://lore.kernel.org/all/[email protected]/T/#u

Veronika, was there a v3?

Thanks,

Well the patch was already applied upstream so we are talking about a fix not v3, right?

Quoted 7 lines

Can you provide a url for the patch that was applied upstream?

I just checked at

https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next

And I'm not seeing it.

Well the patch adding the failing test case that you can see at

https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next

There was the whole patchset that contained the failing probe test case. The patchset was applied and merged upstream so there won't be a v3 for the failing test case, instead we will send a fix patch resolving the mentioned issue. Hopefully this explains the confusion that I think happened.

We will send the next patchset soon containing also the patch fixing the probe issue with bad dwarf so if there will still be some confusion that should resolve that.

- Veronika

Quoted 22 lines

- Arnaldo

As Arnaldo mentioned, I got together a fix patch that checks

Quoted 16 lines

the CU for the Assembly for blacklisted probe. Michael also suggested the possibility to check multiple blacklisted probes if we are not able to check
for the Assembly, so I also added that.

I will send the fix patch soon as part of the perftool upstreaming patchset to prevent conflicts happening. Hopefully, this can be at the start of the next week if all of the checks pass.

Thanks,
Veronika

Quoted 3 lines

- Arnaldo

author maintainer
#6 lore raw

Hello,

Quoted 59 lines · Veronika Molnarova
Quoted 46 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · Veronika Molnarova
Quoted 29 lines · Arnaldo Carvalho de Melo
Quoted 27 lines · Arnaldo Carvalho de Melo
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

I'll try to find out the threads and see what happened.

Quoted 7 lines
Quoted 5 lines

https://lore.kernel.org/all/[email protected]/T/#u

Veronika, was there a v3?

Thanks,

Well the patch was already applied upstream so we are talking about a fix not v3, right?

Quoted 7 lines

Can you provide a url for the patch that was applied upstream?

I just checked at

https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next

And I'm not seeing it.

Well the patch adding the failing test case that you can see at

Quoted 7 lines

https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next

There was the whole patchset that contained the failing probe test case. The patchset was applied and merged upstream so there won't be a v3 for the failing test case, instead we will send a fix patch resolving the mentioned issue. Hopefully this explains the confusion that I think happened.

Ok, thanks for the explanation. I see it's merged already and I'm waiting for the fix only.

Quoted 4 lines

We will send the next patchset soon containing also the patch fixing the probe issue with bad dwarf so if there will still be some confusion that should resolve that.

It'd be nice if you can send the fix first separately. As the whole patchset might take some time for review, I want to fix the existing problem ASAP.

Thanks,
Namhyung

author maintainer
#7 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 19 lines · Arnaldo Carvalho de Melo
Quoted 14 lines · Namhyung Kim

On my system, it's constantly failing because of new error message from perf probe. It should update the regex pattern to match the message - "A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error
  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.
Quoted 3 lines
  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.
Quoted 6 lines

We discussed this in the past, I came up with a similar patch, Veronika rightly pointed out that this may point to a real problem, Masami said that since these are for DWARF from assembly those are known issues, I suggested Veronika checked if the CU where the function came from was generated from Assembly (there are DWARF tags that have that info), IIRC she said she would try to do it.

Oh ok. I was lost in the thread. :)

Quoted 2 lines

I'll try to find out the threads and see what happened.

Thanks, I will wait for the next step.
Namhyung

Quoted 25 lines
Quoted 25 lines
Cc Masami Hiramatsu <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Veronika Molnarova <[email protected]>
---
 tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh +2 -1
index b5dc10b2a73810b3..01e5e09053c37e46 100755
@@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
 REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
 REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
 REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
-../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
+REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
+../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
 CHECK_EXIT_CODE=$?
 
 print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
Signature · 1 line
2.47.0.rc0.187.ge670bccf7e-goog