Skip to contentSkip to conversation

perf probe can't remove probes

BUG Arnaldo Carvalho de Melo 2 participants 7 messages Nov 25, 2020 – Dec 16, 2020

lore
Share & tools
lore
#0 lore raw

Masami, have you stumbled on this already?

[root@seventh ~]# perf probe security_locked_down%return 'ret=$retval' Added new event:
probe:security_locked_down__return (on security_locked_down%return with ret=$retval)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down__return -aR sleep 1

[root@seventh ~]# perf probe security_locked_down what Added new event:
probe:security_locked_down (on security_locked_down with what)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# uname -r
5.10.0-rc3.bpfsign+
[root@seventh ~]# perf probe -l
probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what) probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
[root@seventh ~]# perf probe -D '*:*'
Semantic error :There is non-digit char in line number.

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
                          Show trace event definition of given traceevent for k/uprobe_events.

[root@seventh ~]# perf probe probe:security_locked_down Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@seventh ~]# perf probe probe:security_locked_down__return Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@seventh ~]# cat /sys/kernel/debug/kprobes/
blacklist enabled list
[root@seventh ~]# cat /sys/kernel/debug/kprobes/list

ffffffff8248b350  k  security_locked_down+0x0    [FTRACE]
ffffffff8248b350  r  security_locked_down+0x0    [FTRACE]

[root@seventh ~]#

[root@seventh ~]# cat /etc/fedora-release
Fedora release 33 (Thirty Three)
[root@seventh ~]# gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.1 20201016 (Red Hat 10.2.1-6) (GCC)
[root@seventh ~]# rpm -q elfutils
elfutils-0.182-1.fc33.x86_64
[root@seventh ~]#

- Arnaldo

#1 lore raw

Hi Arnaldo,

Quoted 31 lines · On Wed, 25 Nov 2020 14:27:55 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Masami, have you stumbled on this already?

[root@seventh ~]# perf probe security_locked_down%return 'ret=$retval' Added new event:
probe:security_locked_down__return (on security_locked_down%return with ret=$retval)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down__return -aR sleep 1

[root@seventh ~]# perf probe security_locked_down what Added new event:
probe:security_locked_down (on security_locked_down with what)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# uname -r
5.10.0-rc3.bpfsign+
[root@seventh ~]# perf probe -l
probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what) probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
[root@seventh ~]# perf probe -D '*:*'
Semantic error :There is non-digit char in line number.

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

Quoted 9 lines
    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
                          Show trace event definition of given traceevent for k/uprobe_events.

As you can see, "-D" is showing definition. Not delete. (*) Delete is "-d" or "--del".

(*) this option is for different version of kernel, remote-machine and boot-time tracing.

Quoted 6 lines

[root@seventh ~]# perf probe probe:security_locked_down Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@seventh ~]# perf probe probe:security_locked_down__return Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Since you don't pass any option, both are for adding new probe event.

What happen if you run

$ perf probe -d "*:*"

?

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#2 lore raw

Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:

Quoted 43 lines

Hi Arnaldo,

Quoted 38 lines · On Wed, 25 Nov 2020 14:27:55 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Masami, have you stumbled on this already?

[root@seventh ~]# perf probe security_locked_down%return 'ret=$retval' Added new event:
probe:security_locked_down__return (on security_locked_down%return with ret=$retval)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down__return -aR sleep 1

[root@seventh ~]# perf probe security_locked_down what Added new event:
probe:security_locked_down (on security_locked_down with what)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# uname -r
5.10.0-rc3.bpfsign+
[root@seventh ~]# perf probe -l
probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what) probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
[root@seventh ~]# perf probe -D '*:*'
Semantic error :There is non-digit char in line number.

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs
Quoted 5 lines
Quoted 2 lines

-D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
Show trace event definition of given traceevent for k/uprobe_events.

As you can see, "-D" is showing definition. Not delete. (*) Delete is "-d" or "--del".

Yeah, I was in a hurry and looked at just the first line right after the command, didn't want to forget reporting it so sent the "bug" report, d0h, sorry about the noise, using -d or --del works.

But having both -d and -D, in retrospect, wasn't such a good idea :-\

- Arnaldo

Quoted 22 lines

(*) this option is for different version of kernel, remote-machine and boot-time tracing.

Quoted 6 lines

[root@seventh ~]# perf probe probe:security_locked_down Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@seventh ~]# perf probe probe:security_locked_down__return Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Since you don't pass any option, both are for adding new probe event.

What happen if you run

$ perf probe -d "*:*"

?

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
Signature · 1 line
- Arnaldo
#3 lore raw
Quoted 47 lines · On Thu, 26 Nov 2020 14:26:03 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:

Quoted 46 lines

Hi Arnaldo,

Quoted 40 lines · On Wed, 25 Nov 2020 14:27:55 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Masami, have you stumbled on this already?

[root@seventh ~]# perf probe security_locked_down%return 'ret=$retval' Added new event:
probe:security_locked_down__return (on security_locked_down%return with ret=$retval)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down__return -aR sleep 1

[root@seventh ~]# perf probe security_locked_down what Added new event:
probe:security_locked_down (on security_locked_down with what)

You can now use it in all perf tools, such as:

perf record -e probe:security_locked_down -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# uname -r
5.10.0-rc3.bpfsign+
[root@seventh ~]# perf probe -l
probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what) probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
[root@seventh ~]# perf probe -D '*:*'
Semantic error :There is non-digit char in line number.

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
                          Show trace event definition of given traceevent for k/uprobe_events.
Quoted 8 lines
Quoted 2 lines

As you can see, "-D" is showing definition. Not delete. (*) Delete is "-d" or "--del".

Yeah, I was in a hurry and looked at just the first line right after the command, didn't want to forget reporting it so sent the "bug" report, d0h, sorry about the noise, using -d or --del works.

But having both -d and -D, in retrospect, wasn't such a good idea :-\

Sorry for confusing :(

Hmm, would we better to remove -D and keep only --definition? But it is already there, I think we should keep this option for backward compatibility.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#4 lore raw

Em Wed, Dec 16, 2020 at 09:16:40AM +0900, Masami Hiramatsu escreveu:

Quoted 1 line

On Thu, 26 Nov 2020 14:26:03 -0300

Quoted 9 lines · Arnaldo Carvalho de Melo
Quoted 8 lines

Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:

Quoted 3 lines · On Wed, 25 Nov 2020 14:27:55 -0300 Arnaldo Carvalho de Melo <[email protected]>…

As you can see, "-D" is showing definition. Not delete. (*) Delete is "-d" or "--del".

Yeah, I was in a hurry and looked at just the first line right after the command, didn't want to forget reporting it so sent the "bug" report, d0h, sorry about the noise, using -d or --del works.

Quoted 7 lines
Quoted 1 line

But having both -d and -D, in retrospect, wasn't such a good idea :-\

Sorry for confusing :(

Hmm, would we better to remove -D and keep only --definition? But it is already there, I think we should keep this option for backward compatibility.

Nah, since we have it there, lets take this just as a lesson and in the future use single letter options just for core functionality, leaving things like --definition to be just a long option.

We can't change it now :-)

- Arnaldo

#5 lore raw
In reply to Masami Hiramatsu

Hi Masami,

Any idea why listing the source code doesn't work while simply
adding the probe works?

[root@seventh ~]# perf probe verify_pkcs7_signature Added new event:
probe:verify_pkcs7_signature (on verify_pkcs7_signature)

You can now use it in all perf tools, such as:

perf record -e probe:verify_pkcs7_signature -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# readelf -wi /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux | grep DW_AT_producer -m2

    <1c>   DW_AT_producer    : (indirect string, offset: 0x50): GNU AS 2.35
    <2f>   DW_AT_producer    : (indirect string, offset: 0x2daf): GNU C89 10.2.1 20201016 (Red Hat 10.2.1-6) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu90 -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -fno-strict-overflow -fstack-check=no -fconserve-stack -fcf-protection=none -fno-stack-protector

[root@seventh ~]#

[root@seventh ~]# perf probe -L verify_pkcs7_signature
Specified source line is not found.
Error: Failed to show lines.
[root@seventh ~]# grep verify_pkcs7_signature /proc/kallsyms
ffffffff9f2617b0 T verify_pkcs7_signature
ffffffffa047ac30 r __ksymtab_verify_pkcs7_signature
ffffffffa0482e38 r __kstrtabns_verify_pkcs7_signature
ffffffffa0483b3b r __kstrtab_verify_pkcs7_signature
[root@seventh ~]# perf probe -L verify_pkcs7_signature
Specified source line is not found.
Error: Failed to show lines.
[root@seventh ~]# perf probe -v -L verify_pkcs7_signature
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux
fname: /home/acme/git/bpf/include/linux/verification.h, lineno:38
New line range: 38 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)
[root@seventh ~]# readelf -wi /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux | grep -B2 -A8 verify_pkcs7_signature
<1><1a362a2>: Abbrev Number: 64 (DW_TAG_subprogram)

    <1a362a3>   DW_AT_external    : 1
    <1a362a3>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <1a362a7>   DW_AT_decl_file   : 52
    <1a362a8>   DW_AT_decl_line   : 38
    <1a362a9>   DW_AT_decl_column : 12
    <1a362aa>   DW_AT_prototyped  : 1
    <1a362aa>   DW_AT_type        : <0x1a3333f>
    <1a362ae>   DW_AT_sibling     : <0x1a36313>
 <2><1a362b2>: Abbrev Number: 65 (DW_TAG_formal_parameter)
    <1a362b3>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data
--
 <1><24fa79d>: Abbrev Number: 83 (DW_TAG_subprogram)
    <24fa79e>   DW_AT_external    : 1
    <24fa79e>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <24fa7a2>   DW_AT_decl_file   : 218
    <24fa7a3>   DW_AT_decl_line   : 38
    <24fa7a4>   DW_AT_decl_column : 12
    <24fa7a5>   DW_AT_prototyped  : 1
    <24fa7a5>   DW_AT_type        : <0x24e1e05>
    <24fa7a9>   DW_AT_sibling     : <0x24fa80e>
 <2><24fa7ad>: Abbrev Number: 41 (DW_TAG_formal_parameter)
    <24fa7ae>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data
--
    <26258b8>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
 <1><26258c2>: Abbrev Number: 116 (DW_TAG_variable)
    <26258c3>   DW_AT_name        : (indirect string, offset: 0x10523f): __UNIQUE_ID___addressable_verify_pkcs7_signature164
    <26258c7>   DW_AT_decl_file   : 1
    <26258c8>   DW_AT_decl_line   : 305
    <26258ca>   DW_AT_decl_column : 1
    <26258cb>   DW_AT_type        : <0x2620ca6>
    <26258cf>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
 <1><26258d9>: Abbrev Number: 117 (DW_TAG_subprogram)
    <26258da>   DW_AT_external    : 1
    <26258da>   DW_AT_name        : (indirect string, offset: 0x1050d9): pkcs7_free_message
--
 <1><2625c2e>: Abbrev Number: 120 (DW_TAG_subprogram)
    <2625c2f>   DW_AT_external    : 1
    <2625c2f>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <2625c33>   DW_AT_decl_file   : 1
    <2625c34>   DW_AT_decl_line   : 282
    <2625c36>   DW_AT_decl_column : 5
    <2625c37>   DW_AT_prototyped  : 1
    <2625c37>   DW_AT_type        : <0x26209e9>
    <2625c3b>   DW_AT_low_pc      : 0xffffffff812617b0
    <2625c43>   DW_AT_high_pc     : 0x73
    <2625c4b>   DW_AT_frame_base  : 1 byte block: 9c 	(DW_OP_call_frame_cfa)
--
 <1><495560e>: Abbrev Number: 68 (DW_TAG_subprogram)
    <495560f>   DW_AT_external    : 1
    <495560f>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <4955613>   DW_AT_decl_file   : 61
    <4955614>   DW_AT_decl_line   : 38
    <4955615>   DW_AT_decl_column : 12
    <4955616>   DW_AT_prototyped  : 1
    <4955616>   DW_AT_type        : <0x49519fd>
    <495561a>   DW_AT_sibling     : <0x495567f>
 <2><495561e>: Abbrev Number: 74 (DW_TAG_formal_parameter)
    <495561f>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data
#6 lore raw

Em Tue, Dec 01, 2020 at 10:25:17AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 4 lines

Hi Masami,

Any idea why listing the source code doesn't work while simply
adding the probe works?

<SNIP>

Quoted 3 lines

[root@seventh ~]# perf probe -L verify_pkcs7_signature Specified source line is not found.
Error: Failed to show lines.

Yeah, those fixes you sent, when cherry-picked into this bpf-next/master based branch, do the trick, so nevermind, you fixed this already, these fixes were already merged by Linus, bpf-next will rebase at some point, etc. :-)

- Arnaldo

[acme@five bpf]$ git log --oneline -4
0f3147557ea8143d (HEAD -> bpfsign) perf probe: Change function definition check due to broken DWARF 0e104f490d8f86b4 perf probe: Fix to die_entrypc() returns error correctly 9423e1f089643738 libbpf: Check if the kernel supports signatures before associating them 580616996c498b70 libbpf: Attach signature ELF sections to signed ELF program sections [acme@five bpf]$

[root@seventh ~]# perf probe -L verify_pkcs7_signature <verify_pkcs7_signature@/home/acme/git/bpf/certs/system_keyring.c:0>

      0  int verify_pkcs7_signature(const void *data, size_t len,
                                   const void *raw_pkcs7, size_t pkcs7_len,
                                   struct key *trusted_keys,
                                   enum key_being_used_for usage,
                                   int (*view_content)(void *ctx,
                                                       const void *data, size_t len,
                                                       size_t asn1hdrlen),
                                   void *ctx)
         {
      9         struct pkcs7_message *pkcs7;
                int ret;

                pkcs7 = pkcs7_parse_message(raw_pkcs7, pkcs7_len);
     13         if (IS_ERR(pkcs7))
     14                 return PTR_ERR(pkcs7);

     16         ret = verify_pkcs7_message_sig(data, len, pkcs7, trusted_keys, usage,
                                               view_content, ctx);

     19         pkcs7_free_message(pkcs7);
     20         pr_devel("<==%s() = %d\n", __func__, ret);
                return ret;
         }
         EXPORT_SYMBOL_GPL(verify_pkcs7_signature);

[root@seventh ~]#