Skip to contentSkip to conversation

tracing/x86: Update syscall trace events to handle new x86 syscall func names

PATCH Superseded Steven Rostedt 3 participants 16 messages Apr 17, 2018 – Apr 18, 2018

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

PATCH v2 1/1 tracing/x86: Update syscall trace events to handle new x86 syscall func names

From: "Steven Rostedt (VMware)" <[email protected]>

Arnaldo noticed that the latest kernel is missing the syscall event system directory in x86. I bisected it down to d5a00528b58c ("syscalls/core, syscalls/x86: Rename struct pt_regs-based sys_*() to __x64_sys_*()").

The system call trace events are special, as there is only one trace event for all system calls (the raw_syscalls). But a macro that wraps the system calls creates meta data for them that copies the name to find the system call that maps to the system call table (the number). At boot up, it does a kallsyms lookup for this mapping. If it does not find a function, then that system call is ignored.

Because the x86 system calls had "__x64_" appended to the "sys" for the names, they do not match the default compare algorithm. As this was a problem for power pc, the algorithm can be overwritten by the architecture. The solution is to have x86 have its own algorithm to do the compare and this brings back the system call trace events.

Fixes d5a00528b58c ("syscalls/core, syscalls/x86: Rename struct pt_regs-based sys_*() to __x64_sys_*()")

---
Changes since v1:

Added back original compare to not miss 32bit kernel syscalls

 arch/x86/include/asm/ftrace.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/ftrace.h +10 -0
index 09ad88572746..975b8c73bd50 100644
@@ -31,6 +31,16 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
 	return addr;
 }
 
+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
+{
+	/*
+	 * Compare the symbol name with the system call name. Skip the
+	 * "__x64_sys" or simple "sys" prefix.
+	 */
+	return !strcmp(sym + 9, name + 3) || !strcmp(sym + 3, name + 3);
+}
+
 #ifdef CONFIG_DYNAMIC_FTRACE
 
 struct dyn_arch_ftrace {
Signature · 1 line
2.13.6
#1 lore raw
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

Looks good otherwise.

Thanks,
Dominik

#2 lore raw

Em Tue, Apr 17, 2018 at 07:22:36PM +0200, Dominik Brodowski escreveu:

Quoted 3 lines
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

So this should have covered 0-parameter syscalls, ok, I'm double checking that the last patch is the one I have running... Because 0-parameter syscalls are not working for me, i.e. no syscalls:sys_enter_getppid, for instance.

- Arnaldo

Quoted 4 lines

Looks good otherwise.

Thanks,
Dominik

#3 lore raw

Em Tue, Apr 17, 2018 at 03:04:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

Em Tue, Apr 17, 2018 at 07:22:36PM +0200, Dominik Brodowski escreveu:

Quoted 3 lines
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

So this should have covered 0-parameter syscalls, ok, I'm double checking that the last patch is the one I have running... Because 0-parameter syscalls are not working for me, i.e. no syscalls:sys_enter_getppid, for instance.

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory) [root@jouet ~]#

[root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format
ls: cannot access '/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format': No such file or directory [root@jouet ~]#

With:

+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name) +{
+ /*

+        * Compare the symbol name with the system call name. Skip the
+        * "__x64_sys" prefix.
+        */

+ return !strcmp(sym + 9, name + 3) || !strcmp(sym + 3, name + 3);
+}

in place.

- Arnaldo

author maintainer
#4 lore raw
Quoted 34 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Apr 17, 2018 at 03:04:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

Em Tue, Apr 17, 2018 at 07:22:36PM +0200, Dominik Brodowski escreveu:

Quoted 3 lines
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

So this should have covered 0-parameter syscalls, ok, I'm double checking that the last patch is the one I have running... Because 0-parameter syscalls are not working for me, i.e. no syscalls:sys_enter_getppid, for instance.

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory) [root@jouet ~]#

[root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format
ls: cannot access '/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format': No such file or directory [root@jouet ~]#

With:

arch/x86/include/asm/ftrace.h +9 -0
+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
+{
+       /*
+        * Compare the symbol name with the system call name. Skip the
+        * "__x64_sys" prefix.
+        */
+       return !strcmp(sym + 9, name + 3) || !strcmp(sym + 3, name + 3);
+}
Quoted 2 lines

in place.

Yeah, I did a test too, and found that I'm missing sys_clock_nanosleep. I'm looking into it.

-- Steve

author maintainer
#5 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 34 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Apr 17, 2018 at 03:04:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

Em Tue, Apr 17, 2018 at 07:22:36PM +0200, Dominik Brodowski escreveu:

Quoted 3 lines
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

So this should have covered 0-parameter syscalls, ok, I'm double checking that the last patch is the one I have running... Because 0-parameter syscalls are not working for me, i.e. no syscalls:sys_enter_getppid, for instance.

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory) [root@jouet ~]#

[root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format
ls: cannot access '/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format': No such file or directory [root@jouet ~]#

With:

(quoted diff) +9 -0
+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
+{
+       /*
+        * Compare the symbol name with the system call name. Skip the
+        * "__x64_sys" prefix.
+        */
+       return !strcmp(sym + 9, name + 3) || !strcmp(sym + 3, name + 3);
+}
Quoted 1 line

in place.

It doesn't have to do with the number of parameters, not everything has "__x64" on it.

Try this patch:

-- Steve

arch/x86/include/asm/ftrace.h +13 -0
index 09ad88572746..15582648887f 100644
@@ -31,6 +31,19 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
 	return addr;
 }
 
+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
+{
+	/*
+	 * Compare the symbol name with the system call name. Skip the
+	 * "__x64_sys", "__ia32_sys" or simple "sys" prefix.
+	 */
+	return !strcmp(sym + 3, name + 3) ||
+		(!strncmp(sym, "__x64_", 6) && !strcmp(sym + 9, name + 3)) ||
+		(!strncmp(sym, "__ia32_", 7) && !strcmp(sym + 10, name + 3));
+
+}
+
 #ifdef CONFIG_DYNAMIC_FTRACE
 
 struct dyn_arch_ftrace {
#6 lore raw

Em Tue, Apr 17, 2018 at 05:41:28PM -0400, Steven Rostedt escreveu:

Quoted 11 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Quoted 5 lines

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory)

It doesn't have to do with the number of parameters, not everything has "__x64" on it.

Try this patch:

Trying...

Quoted 24 lines
arch/x86/include/asm/ftrace.h +13 -0
index 09ad88572746..15582648887f 100644
@@ -31,6 +31,19 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
 	return addr;
 }
 
+#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
+static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
+{
+	/*
+	 * Compare the symbol name with the system call name. Skip the
+	 * "__x64_sys", "__ia32_sys" or simple "sys" prefix.
+	 */
+	return !strcmp(sym + 3, name + 3) ||
+		(!strncmp(sym, "__x64_", 6) && !strcmp(sym + 9, name + 3)) ||
+		(!strncmp(sym, "__ia32_", 7) && !strcmp(sym + 10, name + 3));
+
+}
+
 #ifdef CONFIG_DYNAMIC_FTRACE
 
 struct dyn_arch_ftrace {
author maintainer
#7 lore raw
Quoted 8 lines · On Wed, 18 Apr 2018 09:53:22 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Apr 17, 2018 at 05:41:28PM -0400, Steven Rostedt escreveu:

Quoted 6 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Quoted 5 lines

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory)

Quoted 6 lines
Quoted 4 lines

It doesn't have to do with the number of parameters, not everything has "__x64" on it.

Try this patch:

Trying...

You're keeping me in suspense!

-- Steve

#8 lore raw

Em Wed, Apr 18, 2018 at 10:36:06AM -0400, Steven Rostedt escreveu:

Quoted 10 lines
Quoted 8 lines · On Wed, 18 Apr 2018 09:53:22 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Apr 17, 2018 at 05:41:28PM -0400, Steven Rostedt escreveu:

Quoted 6 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Quoted 5 lines

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory)

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

It doesn't have to do with the number of parameters, not everything has "__x64" on it.

Try this patch:

Trying...

You're keeping me in suspense!

I switched locations, had trouble reconnecting, those tests are ok now, there is just one case left, related to the syscall routines renames, but not related to the syscalls:sys_{enter,exit}_NAME tracepoints:

[root@jouet ~]# perf test

 1: vmlinux symtab matches kallsyms                       : Ok
 2: Detect openat syscall event                           : Ok
 3: Detect openat syscall event on all cpus               : Ok
 4: Read samples using the mmap interface                 : Ok
 5: Test data source output                               : Ok
 6: Parse event definition strings                        : Ok
 7: Simple expression parser                              : Ok
 8: PERF_RECORD_* events & perf_sample fields             : Ok
 9: Parse perf pmu format                                 : Ok
10: DSO data read                                         : Ok
11: DSO data cache                                        : Ok
12: DSO data reopen                                       : Ok
13: Roundtrip evsel->name                                 : Ok
14: Parse sched tracepoints fields                        : Ok
15: syscalls:sys_enter_openat event fields                : Ok
16: Setup struct perf_event_attr                          : Ok
17: Match and link multiple hists                         : Ok
18: 'import perf' in python                               : Ok
19: Breakpoint overflow signal handler                    : Ok
20: Breakpoint overflow sampling                          : Ok
21: Breakpoint accounting                                 : Ok
22: Number of exit events of a simple workload            : Ok
23: Software clock events period values                   : Ok
24: Object code reading                                   : Ok
25: Sample parsing                                        : Ok
26: Use a dummy software event to keep tracking           : Ok
27: Parse with no sample_id_all bit set                   : Ok
28: Filter hist entries                                   : Ok
29: Lookup mmap thread                                    : Ok
30: Share thread mg                                       : Ok
31: Sort output of hist entries                           : Ok
32: Cumulate child hist entries                           : Ok
33: Track with sched_switch                               : Ok
34: Filter fds with revents mask in a fdarray             : Ok
35: Add fd to a fdarray, making it autogrow               : Ok
36: kmod_path__parse                                      : Ok
37: Thread map                                            : Ok
38: LLVM search and compile                               :
38.1: Basic BPF llvm compile                              : Ok
38.2: kbuild searching                                    : Ok
38.3: Compile source for BPF prologue generation          : Ok
38.4: Compile source for BPF relocation                   : Ok
39: Session topology                                      : Ok
40: BPF filter                                            :
40.1: Basic BPF filtering                                 : FAILED!
40.2: BPF pinning                                         : Skip
40.3: BPF prologue generation                             : Skip
40.4: BPF relocation checker                              : Skip
41: Synthesize thread map                                 : Ok
42: Remove thread map                                     : Ok
43: Synthesize cpu map                                    : Ok
44: Synthesize stat config                                : Ok
45: Synthesize stat                                       : Ok
46: Synthesize stat round                                 : Ok
47: Synthesize attr update                                : Ok
48: Event times                                           : Ok
49: Read backward ring buffer                             : Ok
50: Print cpu map                                         : Ok
51: Probe SDT events                                      : Ok
52: is_printable_array                                    : Ok
53: Print bitmap                                          : Ok
54: perf hooks                                            : Ok
55: builtin clang support                                 : Skip (not compiled in)
56: unit_number__scnprintf                                : Ok
57: mem2node                                              : Ok
58: x86 rdpmc                                             : Ok
59: Convert perf time to TSC                              : Ok
60: DWARF unwind                                          : Ok
61: x86 instruction decoder - new instructions            : Ok
62: Use vfs_getname probe to get syscall args filenames   : Ok
63: probe libc's inet_pton & backtrace it with ping       : Ok

64: Check open filename arg using perf trace + vfs_getname: Ok
65: probe libc's inet_pton & backtrace it with ping : Ok
66: Add vfs_getname probe to get syscall args filenames : Ok
[root@jouet ~]#

If we use -v for that test we see the problem:

To the point:

Probe point 'SyS_epoll_pwait' not found.

This is not there anymore, I'll change this test to first figure out what is the syscall routine for the epoll_pwait syscall so that it works with pre-syscall-routines-rename and after that.

- Arnaldo

all details:

[root@jouet ~]# perf test -v BPF
40: BPF filter :
40.1: Basic BPF filtering :
--- start ---
test child forked, pid 13225
Kernel build dir is set to /lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build
set env: KBUILD_DIR=/lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build
unset env: KBUILD_OPTS
include option is set to -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h set env: NR_CPUS=4
set env: LINUX_VERSION_CODE=0x41100
set env: CLANG_EXEC=/usr/lib64/ccache/clang
set env: CLANG_OPTIONS=-xc
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h set env: WORKING_DIR=/lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build
set env: CLANG_SOURCE=-
llvm compiling command template: echo '/*
* bpf-script-example.c
* Test basic LLVM building
*/
#ifndef LINUX_VERSION_CODE

# error Need LINUX_VERSION_CODE
# error Example: for 4.2 kernel, put 'clang-opt="-DLINUX_VERSION_CODE=0x40200" into llvm section of ~/.perfconfig'
#endif
#define BPF_ANY 0
#define BPF_MAP_TYPE_ARRAY 2
#define BPF_FUNC_map_lookup_elem 1
#define BPF_FUNC_map_update_elem 2

static void *(*bpf_map_lookup_elem)(void *map, void *key) =

	(void *) BPF_FUNC_map_lookup_elem;
static void *(*bpf_map_update_elem)(void *map, void *key, void *value, int flags) =
	(void *) BPF_FUNC_map_update_elem;
struct bpf_map_def {
	unsigned int type;
	unsigned int key_size;
	unsigned int value_size;
	unsigned int max_entries;
};

#define SEC(NAME) __attribute__((section(NAME), used)) struct bpf_map_def SEC("maps") flip_table = {
.type = BPF_MAP_TYPE_ARRAY,
.key_size = sizeof(int),
.value_size = sizeof(int), .max_entries = 1,
};

SEC("func=SyS_epoll_pwait")
int bpf_func__SyS_epoll_pwait(void *ctx)

{
	int ind =0;
	int *flag = bpf_map_lookup_elem(&flip_table, &ind);
	int new_flag;
	if (!flag)
		return 0;
	/* flip flag and store back */
	new_flag = !*flag;
	bpf_map_update_elem(&flip_table, &ind, &new_flag, BPF_ANY);
	return new_flag;
}

char _license[] SEC("license") = "GPL";
int _version SEC("version") = LINUX_VERSION_CODE;
' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o - llvm compiling command : echo '/*
* bpf-script-example.c
* Test basic LLVM building
*/
#ifndef LINUX_VERSION_CODE

# error Need LINUX_VERSION_CODE
# error Example: for 4.2 kernel, put 'clang-opt=-DLINUX_VERSION_CODE=0x40200 into llvm section of ~/.perfconfig'
#endif
#define BPF_ANY 0
#define BPF_MAP_TYPE_ARRAY 2
#define BPF_FUNC_map_lookup_elem 1
#define BPF_FUNC_map_update_elem 2

static void *(*bpf_map_lookup_elem)(void *map, void *key) =

	(void *) BPF_FUNC_map_lookup_elem;
static void *(*bpf_map_update_elem)(void *map, void *key, void *value, int flags) =
	(void *) BPF_FUNC_map_update_elem;
struct bpf_map_def {
	unsigned int type;
	unsigned int key_size;
	unsigned int value_size;
	unsigned int max_entries;
};

#define SEC(NAME) __attribute__((section(NAME), used)) struct bpf_map_def SEC(maps) flip_table = {
.type = BPF_MAP_TYPE_ARRAY,
.key_size = sizeof(int),
.value_size = sizeof(int), .max_entries = 1,
};

SEC(func=SyS_epoll_pwait)
int bpf_func__SyS_epoll_pwait(void *ctx)

{
	int ind =0;
	int *flag = bpf_map_lookup_elem(&flip_table, &ind);
	int new_flag;
	if (!flag)
		return 0;
	/* flip flag and store back */
	new_flag = !*flag;
	bpf_map_update_elem(&flip_table, &ind, &new_flag, BPF_ANY);
	return new_flag;
}

char _license[] SEC(license) = GPL;
int _version SEC(version) = LINUX_VERSION_CODE;
' | /usr/lib64/ccache/clang -D__KERNEL__ -D__NR_CPUS__=4 -DLINUX_VERSION_CODE=0x41100 -xc -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h -Wno-unused-value -Wno-pointer-sign -working-directory /lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build -c - -target bpf -O2 -o -
libbpf: loading object '[basic_bpf_test]' from buffer
libbpf: section(1) .strtab, size 122, link 0, flags 0, type=3
libbpf: skip section(1) .strtab
libbpf: section(2) .text, size 0, link 0, flags 6, type=1
libbpf: skip section(2) .text
libbpf: section(3) func=SyS_epoll_pwait, size 192, link 0, flags 6, type=1
libbpf: found program func=SyS_epoll_pwait
libbpf: section(4) .relfunc=SyS_epoll_pwait, size 32, link 8, flags 0, type=9
libbpf: section(5) maps, size 16, link 0, flags 3, type=1
libbpf: section(6) license, size 4, link 0, flags 3, type=1
libbpf: license of [basic_bpf_test] is GPL
libbpf: section(7) version, size 4, link 0, flags 3, type=1
libbpf: kernel version of [basic_bpf_test] is 41100
libbpf: section(8) .symtab, size 168, link 1, flags 0, type=2
libbpf: maps in [basic_bpf_test]: 1 maps in 16 bytes
libbpf: map 0 is "flip_table"
libbpf: collecting relocating info for: 'func=SyS_epoll_pwait'
libbpf: relo for 6 value 0 name 81
libbpf: relocation: insn_idx=4
libbpf: relocation: find map 0 (flip_table) for insn 4
libbpf: relo for 6 value 0 name 81
libbpf: relocation: insn_idx=17
libbpf: relocation: find map 0 (flip_table) for insn 17
bpf: config program 'func=SyS_epoll_pwait'
symbol:SyS_epoll_pwait file:(null) line:0 offset:0 return:0 lazy:(null)
bpf: config 'func=SyS_epoll_pwait' is ok
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.17.0-rc1-00024-g7e4e440a91c8-dirty/build/vmlinux Try to find probe point from debuginfo.
Probe point 'SyS_epoll_pwait' not found.
bpf_probe: failed to convert perf probe events
Failed to add events selected by BPF
test child finished with -1
---- end ----
BPF filter subtest 0: FAILED!
40.2: BPF pinning :
--- force skipped ---
BPF filter subtest 1: Skip
40.3: BPF prologue generation :
--- force skipped ---
BPF filter subtest 2: Skip
40.4: BPF relocation checker :
--- force skipped ---
BPF filter subtest 3: Skip
[root@jouet ~]#

#9 lore raw

Em Wed, Apr 18, 2018 at 12:02:12PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 31 lines

Em Wed, Apr 18, 2018 at 10:36:06AM -0400, Steven Rostedt escreveu:

Quoted 10 lines
Quoted 8 lines · On Wed, 18 Apr 2018 09:53:22 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Apr 17, 2018 at 05:41:28PM -0400, Steven Rostedt escreveu:

Quoted 6 lines · On Tue, 17 Apr 2018 15:13:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Quoted 5 lines

Yeah, failing:

[root@jouet ~]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory)

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

It doesn't have to do with the number of parameters, not everything has "__x64" on it.

Try this patch:

Trying...

You're keeping me in suspense!

I switched locations, had trouble reconnecting, those tests are ok now, there is just one case left, related to the syscall routines renames, but not related to the syscalls:sys_{enter,exit}_NAME tracepoints:

40: BPF filter                                            :
40.1: Basic BPF filtering                                 : FAILED!
40.2: BPF pinning                                         : Skip
40.3: BPF prologue generation                             : Skip
40.4: BPF relocation checker                              : Skip

If we use -v for that test we see the problem:

Quoted 7 lines

To the point:

Probe point 'SyS_epoll_pwait' not found.

This is not there anymore, I'll change this test to first figure out what is the syscall routine for the epoll_pwait syscall so that it works with pre-syscall-routines-rename and after that.

This does the trick, by not using the main syscall routine, but one called from it and not renamed, should work with older kernels.

This test should be improved to look if the desired routine is in place, if not just skip the test and tell about the unavailability of the wanted function, but that is for later.

- Arnaldo

tools/perf/tests/bpf-script-example.c +1 -1
index e4123c1b0e88..1ca5106df5f1 100644
@@ -31,7 +31,7 @@ struct bpf_map_def SEC("maps") flip_table = {
 	.max_entries = 1,
 };
 
-SEC("func=SyS_epoll_pwait")
+SEC("func=do_epoll_wait")
 int bpf_func__SyS_epoll_pwait(void *ctx)
 {
 	int ind =0;
author maintainer
#10 lore raw
Quoted 6 lines · On Wed, 18 Apr 2018 12:17:16 -0300 Arnaldo Carvalho de Melo <[email protected]>…

This does the trick, by not using the main syscall routine, but one called from it and not renamed, should work with older kernels.

This test should be improved to look if the desired routine is in place, if not just skip the test and tell about the unavailability of the wanted function, but that is for later.

Does this mean you can give me a "Tested-by" for that last patch?

-- Steve

maintainer Tested-by
#11 lore raw

Em Wed, Apr 18, 2018 at 11:20:33AM -0400, Steven Rostedt escreveu:

Quoted 11 lines
Quoted 6 lines · On Wed, 18 Apr 2018 12:17:16 -0300 Arnaldo Carvalho de Melo <[email protected]>…

This does the trick, by not using the main syscall routine, but one called from it and not renamed, should work with older kernels.

This test should be improved to look if the desired routine is in place, if not just skip the test and tell about the unavailability of the wanted function, but that is for later.

Does this mean you can give me a "Tested-by" for that last patch?

Here it is, just written down:

- Arnaldo

author maintainer
#12 lore raw
Quoted 11 lines · On Wed, 18 Apr 2018 12:25:36 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Wed, Apr 18, 2018 at 11:20:33AM -0400, Steven Rostedt escreveu:

Quoted 10 lines
Quoted 6 lines · On Wed, 18 Apr 2018 12:17:16 -0300 Arnaldo Carvalho de Melo <[email protected]>…

This does the trick, by not using the main syscall routine, but one called from it and not renamed, should work with older kernels.

This test should be improved to look if the desired routine is in place, if not just skip the test and tell about the unavailability of the wanted function, but that is for later.

Quoted 6 lines
Quoted 1 line

Does this mean you can give me a "Tested-by" for that last patch?

Here it is, just written down:

Thanks!

-- Steve

author maintainer
#13 lore raw
In reply to Dominik Brodowski
Quoted 6 lines · On Tue, 17 Apr 2018 19:22:36 +0200 Dominik Brodowski…
Quoted 1 line

Added back original compare to not miss 32bit kernel syscalls

s/32bit/32bit and 0-parameter syscalls.

Looks good otherwise.

You want to send an Acked or Reviewed-by to my final patch (v3).

-- Steve

maintainer Tested-by on v2 1/1
#14 lore raw
In reply to Steven Rostedt

Em Tue, Apr 17, 2018 at 01:07:02PM -0400, Steven Rostedt escreveu:

Quoted 19 lines

From: "Steven Rostedt (VMware)" <[email protected]>

Arnaldo noticed that the latest kernel is missing the syscall event system directory in x86. I bisected it down to d5a00528b58c ("syscalls/core, syscalls/x86: Rename struct pt_regs-based sys_*() to __x64_sys_*()").

The system call trace events are special, as there is only one trace event for all system calls (the raw_syscalls). But a macro that wraps the system calls creates meta data for them that copies the name to find the system call that maps to the system call table (the number). At boot up, it does a kallsyms lookup for this mapping. If it does not find a function, then that system call is ignored.

Because the x86 system calls had "__x64_" appended to the "sys" for the names, they do not match the default compare algorithm. As this was a problem for power pc, the algorithm can be overwritten by the architecture. The solution is to have x86 have its own algorithm to do the compare and this brings back the system call trace events.

Quoted 3 lines
Fixes d5a00528b58c ("syscalls/core, syscalls/x86: Rename struct pt_regs-based sys_*() to __x64_sys_*()")

Again on x86_64, the number of files in /sys/kernel/debug/tracing/events/syscalls/ is the same, all my perf related tests pass.

Now to the next seemingly kernel related test failure, unrelated to this:

4: Read samples using the mmap interface : FAILED!

- Arnaldo

#15 lore raw

Em Tue, Apr 17, 2018 at 02:29:40PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 13 lines

Em Tue, Apr 17, 2018 at 01:07:02PM -0400, Steven Rostedt escreveu:

Quoted 9 lines

Because the x86 system calls had "__x64_" appended to the "sys" for the names, they do not match the default compare algorithm. As this was a problem for power pc, the algorithm can be overwritten by the architecture. The solution is to have x86 have its own algorithm to do the compare and this brings back the system call trace events.

Fixes d5a00528b58c ("syscalls/core, syscalls/x86: Rename struct pt_regs-based sys_*() to __x64_sys_*()")
Quoted 8 lines

Again on x86_64, the number of files in /sys/kernel/debug/tracing/events/syscalls/ is the same, all my perf related tests pass.

Now to the next seemingly kernel related test failure, unrelated to this:

4: Read samples using the mmap interface : FAILED!

Ok, I take that back, it is related:

[root@jouet perf]# perf test -F -v "mmap interface" |& grep -v plugins
4: Read samples using the mmap interface :
--- start ---
perf_evsel__new(sys_enter_getppid)
---- end ----
Read samples using the mmap interface: FAILED!
[root@jouet perf]#

[root@jouet perf]# strace -e openat -e file perf test -F -v "mmap interface" |& grep syscalls
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getsid/format", O_RDONLY) = 3 openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/syscalls/sys_enter_getppid/format", O_RDONLY) = -1 ENOENT (No such file or directory) [root@jouet perf]#

Where did syscalls:sys_enter_getppid go? Zero argument syscall, investigating...

- Arnaldo