Skip to contentSkip to conversation

perf test: Fix test case perf trace BTF general tests

PATCH In review Thomas Richter 5 participants 17 messages Nov 17, 2025 – Nov 27, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf test: Fix test case perf trace BTF general tests

The following test case fails on linux-next repo:

❯ uname -a

 Linux s83lp47.lnxne.boe 6.18.0-20251116.rc5.git0.0f2995693867.63.\
      fc42.s390x+next #1 SMP Sun Nov 16 20:05:28 CET 2025 s390x GNU/Linux
 # perf test -Fv 109
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Buffer augmentation test failed, output:
 buffer content
 echo/23281 write(1, buffer conten, 15, "") = 15
 ---- end ----
 109: perf trace BTF general tests            : FAILED!
 #

The root case is a changed output format on linux-next. There is an addional "" string as forth parameter in the write() line. Here is the detailed output on linux-repo.
Please note that this depends on the kernel and not on the perf tool.

Output on linux next kernel:

 # uname -a
 Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC ...
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/7676 write(1, buffer content\10, 15, "") = 15
 #

Output on linux kernel:

 # uname -a
 Linux b3560002.lnxne.boe 6.18.0-rc5m-perf #6 ....
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/36932 write(1, buffer content\10, 15) = 15
 #

Add the optional forth parameter in the extented regular expression to accept both output formats.

Output after:

 # ./perf test -Fv 'perf trace BTF general tests'
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Testing perf trace's struct augmentation
 ---- end ----
 115: perf trace BTF general tests            : Ok
 #
---
 tools/perf/tests/shell/trace_btf_general.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/trace_btf_general.sh +1 -1
index ef2da806be6b..9cd6180062d8 100755
@@ -39,7 +39,7 @@ trace_test_buffer() {
   echo "Testing perf trace's buffer augmentation"
   # echo will insert a newline (\10) at the end of the buffer
   output="$(perf trace --sort-events -e write --max-events=1 -- echo "${buffer}" 2>&1)"
-  if ! echo "$output" | grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  if ! echo "$output" | grep -qE "^echo/[0-9]+ write\([0-9]+, ${buffer}\\\\10, [0-9]+(, ..)?\) += +[0-9]+$"
   then
     printf "Buffer augmentation test failed, output:\n$output\n"
     err=1
Signature · 1 line
2.51.1
maintainer
#1 lore raw

Hello,

Quoted 18 lines · Thomas Richter

The following test case fails on linux-next repo:

❯ uname -a

 Linux s83lp47.lnxne.boe 6.18.0-20251116.rc5.git0.0f2995693867.63.\
      fc42.s390x+next #1 SMP Sun Nov 16 20:05:28 CET 2025 s390x GNU/Linux
 # perf test -Fv 109
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Buffer augmentation test failed, output:
 buffer content
 echo/23281 write(1, buffer conten, 15, "") = 15
 ---- end ----
 109: perf trace BTF general tests            : FAILED!
 #
Quoted 4 lines

The root case is a changed output format on linux-next. There is an addional "" string as forth parameter in the write() line. Here is the detailed output on linux-repo.
Please note that this depends on the kernel and not on the perf tool.

Thanks for the report. Do you know what the 4th arg is? It'd be nice if you can dump the contents of the event format which is /sys/kernel/tracing/events/syscalls/sys_enter_write/format.

Thanks,
Namhyung

Quoted 56 lines

Output on linux next kernel:

 # uname -a
 Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC ...
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/7676 write(1, buffer content\10, 15, "") = 15
 #

Output on linux kernel:

 # uname -a
 Linux b3560002.lnxne.boe 6.18.0-rc5m-perf #6 ....
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/36932 write(1, buffer content\10, 15) = 15
 #

Add the optional forth parameter in the extented regular expression to accept both output formats.

Output after:

 # ./perf test -Fv 'perf trace BTF general tests'
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Testing perf trace's struct augmentation
 ---- end ----
 115: perf trace BTF general tests            : Ok
 #
---
 tools/perf/tests/shell/trace_btf_general.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/trace_btf_general.sh +1 -1
index ef2da806be6b..9cd6180062d8 100755
@@ -39,7 +39,7 @@ trace_test_buffer() {
   echo "Testing perf trace's buffer augmentation"
   # echo will insert a newline (\10) at the end of the buffer
   output="$(perf trace --sort-events -e write --max-events=1 -- echo "${buffer}" 2>&1)"
-  if ! echo "$output" | grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  if ! echo "$output" | grep -qE "^echo/[0-9]+ write\([0-9]+, ${buffer}\\\\10, [0-9]+(, ..)?\) += +[0-9]+$"
   then
     printf "Buffer augmentation test failed, output:\n$output\n"
     err=1
Signature · 1 line
2.51.1
#2 lore raw
Quoted 26 lines · Namhyung Kim

Hello,

Quoted 18 lines · Thomas Richter

The following test case fails on linux-next repo:

❯ uname -a

 Linux s83lp47.lnxne.boe 6.18.0-20251116.rc5.git0.0f2995693867.63.\
      fc42.s390x+next #1 SMP Sun Nov 16 20:05:28 CET 2025 s390x GNU/Linux
 # perf test -Fv 109
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Buffer augmentation test failed, output:
 buffer content
 echo/23281 write(1, buffer conten, 15, "") = 15
 ---- end ----
 109: perf trace BTF general tests            : FAILED!
 #
Quoted 4 lines

The root case is a changed output format on linux-next. There is an addional "" string as forth parameter in the write() line. Here is the detailed output on linux-repo.
Please note that this depends on the kernel and not on the perf tool.

Quoted 7 lines

Thanks for the report. Do you know what the 4th arg is? It'd be nice if you can dump the contents of the event format which is /sys/kernel/tracing/events/syscalls/sys_enter_write/format.

Thanks,
Namhyung

Here is the output from my x86 virtual machine with linux-next 20251114 tag.

bash-5.3# uname -a
Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC Mon Nov 17 11:24:02 CET 2025 x86_64 GNU/Linux bash-5.3# cat /sys/kernel/tracing/events/syscalls/sys_enter_write/format
name: sys_enter_write
ID: 758
format:

	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;

print fmt: "fd: 0x%08lx, buf: 0x%08lx (%s), count: 0x%08lx", ((unsigned long)(REC->fd)), ((unsigned long)(REC->buf)), __print_dynamic_array(__buf_val, 1), ((unsigned long)(REC->count)) bash-5.3#

Hope this helps.

Quoted 56 lines
Quoted 56 lines

Output on linux next kernel:

 # uname -a
 Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC ...
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/7676 write(1, buffer content\10, 15, "") = 15
 #

Output on linux kernel:

 # uname -a
 Linux b3560002.lnxne.boe 6.18.0-rc5m-perf #6 ....
 # perf config trace.show_arg_names=false trace.show_duration=false \
	trace.show_timestamp=false trace.args_alignment=0
 # ./perf trace --sort-events -e write --max-events=1 \
	-- echo 'buffer content' 1>/dev/null
 echo/36932 write(1, buffer content\10, 15) = 15
 #

Add the optional forth parameter in the extented regular expression to accept both output formats.

Output after:

 # ./perf test -Fv 'perf trace BTF general tests'
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Testing perf trace's struct augmentation
 ---- end ----
 115: perf trace BTF general tests            : Ok
 #
---
 tools/perf/tests/shell/trace_btf_general.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/trace_btf_general.sh +1 -1
index ef2da806be6b..9cd6180062d8 100755
@@ -39,7 +39,7 @@ trace_test_buffer() {
   echo "Testing perf trace's buffer augmentation"
   # echo will insert a newline (\10) at the end of the buffer
   output="$(perf trace --sort-events -e write --max-events=1 -- echo "${buffer}" 2>&1)"
-  if ! echo "$output" | grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  if ! echo "$output" | grep -qE "^echo/[0-9]+ write\([0-9]+, ${buffer}\\\\10, [0-9]+(, ..)?\) += +[0-9]+$"
   then
     printf "Buffer augmentation test failed, output:\n$output\n"
     err=1
Signature · 1 line
2.51.1
Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
maintainer
#3 lore raw
Quoted 31 lines · Thomas Richter
Quoted 30 lines · Namhyung Kim

Hello,

Quoted 18 lines · Thomas Richter

The following test case fails on linux-next repo:

❯ uname -a

 Linux s83lp47.lnxne.boe 6.18.0-20251116.rc5.git0.0f2995693867.63.\
      fc42.s390x+next #1 SMP Sun Nov 16 20:05:28 CET 2025 s390x GNU/Linux
 # perf test -Fv 109
 --- start ---
 Checking if vmlinux BTF exists
 Testing perf trace's string augmentation
 Testing perf trace's buffer augmentation
 Buffer augmentation test failed, output:
 buffer content
 echo/23281 write(1, buffer conten, 15, "") = 15
 ---- end ----
 109: perf trace BTF general tests            : FAILED!
 #
Quoted 4 lines

The root case is a changed output format on linux-next. There is an addional "" string as forth parameter in the write() line. Here is the detailed output on linux-repo.
Please note that this depends on the kernel and not on the perf tool.

Thanks for the report. Do you know what the 4th arg is? It'd be nice if you can dump the contents of the event format which is /sys/kernel/tracing/events/syscalls/sys_enter_write/format.

Quoted 5 lines
Quoted 3 lines

Thanks,
Namhyung

Here is the output from my x86 virtual machine with linux-next 20251114 tag.

Thanks for sharing this!

Quoted 7 lines

bash-5.3# uname -a
Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC Mon Nov 17 11:24:02 CET 2025 x86_64 GNU/Linux bash-5.3# cat /sys/kernel/tracing/events/syscalls/sys_enter_write/format
name: sys_enter_write
ID: 758
format:

Quoted 10 lines
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;

Indeed, I see this new field __buf_val.

Steve, is this what you added recently for taking user contents? Hmm.. this makes perf trace confused wrt the syscall parameters. Is it always __buf_val or has any patterns?

Quoted 5 lines

print fmt: "fd: 0x%08lx, buf: 0x%08lx (%s), count: 0x%08lx", ((unsigned long)(REC->fd)), ((unsigned long)(REC->buf)), __print_dynamic_array(__buf_val, 1), ((unsigned long)(REC->count)) bash-5.3#

Hope this helps.

Yes it did, thanks!
Namhyung

maintainer
#4 lore raw
Quoted 17 lines · On Mon, 17 Nov 2025 22:43:21 -0800 Namhyung Kim <[email protected]> wrote:
Quoted 6 lines

bash-5.3# uname -a
Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC Mon Nov 17 11:24:02 CET 2025 x86_64 GNU/Linux bash-5.3# cat /sys/kernel/tracing/events/syscalls/sys_enter_write/format
name: sys_enter_write
ID: 758
format:

Quoted 10 lines
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;  
Quoted 3 lines

Indeed, I see this new field __buf_val.

Steve, is this what you added recently for taking user contents?

Yes.

Quoted 2 lines

Hmm.. this makes perf trace confused wrt the syscall parameters. Is it always __buf_val or has any patterns?

Really? It still uses libtraceevent right? I made sure that this didn't break trace-cmd and thought that perf would work too.

-- Steve

maintainer
#5 lore raw
Quoted 24 lines · Steven Rostedt
Quoted 17 lines · On Mon, 17 Nov 2025 22:43:21 -0800 Namhyung Kim <[email protected]> wrote:
Quoted 6 lines

bash-5.3# uname -a
Linux f43 6.18.0-rc5-next-20251114tmr-n #1 SMP PREEMPT_DYNAMIC Mon Nov 17 11:24:02 CET 2025 x86_64 GNU/Linux bash-5.3# cat /sys/kernel/tracing/events/syscalls/sys_enter_write/format
name: sys_enter_write
ID: 758
format:

Quoted 10 lines
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;  
Quoted 3 lines

Indeed, I see this new field __buf_val.

Steve, is this what you added recently for taking user contents?

Quoted 7 lines

Yes.

Quoted 2 lines

Hmm.. this makes perf trace confused wrt the syscall parameters. Is it always __buf_val or has any patterns?

Really? It still uses libtraceevent right? I made sure that this didn't break trace-cmd and thought that perf would work too.

It doesn't completely break perf trace but added new parameter for the write syscall at the end. IIUC perf trace iterates the format fields after __syscall_nr and take them all as syscall parameters.

Thanks,
Namhyung

maintainer
#6 lore raw
Quoted 6 lines · On Tue, 18 Nov 2025 20:36:46 -0800 Namhyung Kim <[email protected]> wrote:
Quoted 2 lines

Really? It still uses libtraceevent right? I made sure that this didn't break trace-cmd and thought that perf would work too.

It doesn't completely break perf trace but added new parameter for the write syscall at the end. IIUC perf trace iterates the format fields after __syscall_nr and take them all as syscall parameters.

Is this a regression? Or can perf be fixed?

I just ran it and I have this:

542.337 ( 0.131 ms): sshd-session/1189 write(fd: 7<socket:[9749]>, buf: , count: 268) = 268

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

-- Steve

maintainer
#7 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Tue, 18 Nov 2025 20:36:46 -0800 Namhyung Kim <[email protected]> wrote:
Quoted 2 lines

Really? It still uses libtraceevent right? I made sure that this didn't break trace-cmd and thought that perf would work too.

It doesn't completely break perf trace but added new parameter for the write syscall at the end. IIUC perf trace iterates the format fields after __syscall_nr and take them all as syscall parameters.

Quoted 8 lines

Is this a regression? Or can perf be fixed?

I just ran it and I have this:

   542.337 ( 0.131 ms): sshd-session/1189 write(fd: 7<socket:[9749]>, buf: , count: 268)                        = 268

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

Yep, it reads the content using BPF. This is on my 6.16 kernel.

  $ sudo perf trace -e write -- /bin/echo hello
  hello
       0.000 ( 0.014 ms): echo/61922 write(fd: 1, buf: hello\10, count: 6)                                 = 6

Thanks,
Namhyung

#8 lore raw
Quoted 20 lines · Namhyung Kim
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Tue, 18 Nov 2025 20:36:46 -0800 Namhyung Kim <[email protected]> wrote:
Quoted 2 lines

Really? It still uses libtraceevent right? I made sure that this didn't break trace-cmd and thought that perf would work too.

It doesn't completely break perf trace but added new parameter for the write syscall at the end. IIUC perf trace iterates the format fields after __syscall_nr and take them all as syscall parameters.

Quoted 8 lines

Is this a regression? Or can perf be fixed?

I just ran it and I have this:

   542.337 ( 0.131 ms): sshd-session/1189 write(fd: 7<socket:[9749]>, buf: , count: 268)                        = 268

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

Quoted 10 lines

Yep, it reads the content using BPF. This is on my 6.16 kernel.

  $ sudo perf trace -e write -- /bin/echo hello
  hello
       0.000 ( 0.014 ms): echo/61922 write(fd: 1, buf: hello\10, count: 6)                                 = 6

Thanks,
Namhyung

Hello Namhyung, Steven,

friendly ping... any progress here?

Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
maintainer
#9 lore raw
Quoted 9 lines · On Wed, 26 Nov 2025 08:13:00 +0100 Thomas Richter <[email protected]> wrote:
Quoted 9 lines
Quoted 2 lines

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

Yep, it reads the content using BPF. This is on my 6.16 kernel.

  $ sudo perf trace -e write -- /bin/echo hello
  hello
       0.000 ( 0.014 ms): echo/61922 write(fd: 1, buf: hello\10, count: 6)                                 = 6
Quoted 9 lines
Quoted 4 lines

Thanks,
Namhyung

Hello Namhyung, Steven,

friendly ping... any progress here?

I honestly have no clue how to fix this, as I don't even know where to look. Is it BPF that is messing up? If so, where's the BPF program that is doing this.

I thought BPF is supposed to handle updates and should never cause API breakage?

I'll continue to look at the builtin-trace.c, but it seems that the BPF program it's attached to is handing it garbage with:

perf trace -e syscalls:sys_enter_write

The new fields are at the end. The BPF program should simply ignore those values. But again, I don't know where this BPF program lives.

-- Steve

maintainer
#10 lore raw

Arnaldo,

How can I make perf trace not confused by the extra fields in the system call trace events?

Ftrace can now show the contents of the system call user space buffers, but it appears that this breaks perf!!!

system: syscalls
name: sys_enter_write
ID: 791
format:

	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;

That new __buf_val appears to confuse perf, but I'm having a hell of a time trying to figure out where it reads it!

-- Steve

Quoted 38 lines · On Wed, 26 Nov 2025 10:24:01 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 9 lines · On Wed, 26 Nov 2025 08:13:00 +0100 Thomas Richter <[email protected]> wrote:
Quoted 9 lines
Quoted 2 lines

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

Yep, it reads the content using BPF. This is on my 6.16 kernel.

  $ sudo perf trace -e write -- /bin/echo hello
  hello
       0.000 ( 0.014 ms): echo/61922 write(fd: 1, buf: hello\10, count: 6)                                 = 6
Quoted 9 lines
Quoted 4 lines

Thanks,
Namhyung

Hello Namhyung, Steven,

friendly ping... any progress here?

I honestly have no clue how to fix this, as I don't even know where to look. Is it BPF that is messing up? If so, where's the BPF program that is doing this.

I thought BPF is supposed to handle updates and should never cause API breakage?

I'll continue to look at the builtin-trace.c, but it seems that the BPF program it's attached to is handing it garbage with:

perf trace -e syscalls:sys_enter_write

The new fields are at the end. The BPF program should simply ignore those values. But again, I don't know where this BPF program lives.

-- Steve

maintainer
#11 lore raw
Quoted 23 lines · On Wed, 26 Nov 2025 12:12:29 -0500 Steven Rostedt <[email protected]> wrote:

Arnaldo,

How can I make perf trace not confused by the extra fields in the system call trace events?

Ftrace can now show the contents of the system call user space buffers, but it appears that this breaks perf!!!

system: syscalls
name: sys_enter_write
ID: 791
format:

	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;
Quoted 3 lines

That new __buf_val appears to confuse perf, but I'm having a hell of a time trying to figure out where it reads it!

Hmm, it gets less confused (at least it doesn't crash), when I don't have perf read the extra values.

Thomas, if you add the below patch, does it fix things for you?

-- Steve

kernel/trace/trace_syscalls.c +0 -13
index e96d0063cbcf..add809d226dc 100644
@@ -1403,7 +1403,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
 	struct hlist_head *head;
 	unsigned long args[6];
 	bool valid_prog_array;
-	bool mayfault;
 	char *user_ptr;
 	int user_sizes[SYSCALL_FAULT_MAX_CNT] = {};
 	int buf_size = CONFIG_TRACE_SYSCALL_BUF_SIZE_DEFAULT;
@@ -1431,15 +1430,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
 
 	syscall_get_arguments(current, regs, args);
 
-	/* Check if this syscall event faults in user space memory */
-	mayfault = sys_data->user_mask != 0;
-
-	if (mayfault) {
-		if (syscall_get_data(sys_data, args, &user_ptr,
-				     &size, user_sizes, &uargs, buf_size) < 0)
-			return;
-	}
-
 	head = this_cpu_ptr(sys_data->enter_event->perf_events);
 	valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
 	if (!valid_prog_array && hlist_empty(head))
@@ -1457,9 +1447,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
 	rec->nr = syscall_nr;
 	memcpy(&rec->args, args, sizeof(unsigned long) * sys_data->nb_args);
 
-	if (mayfault)
-		syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs);
-
 	if ((valid_prog_array &&
 	     !perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) ||
 	    hlist_empty(head)) {
maintainer
#12 lore raw
In reply to Steven Rostedt
Quoted 24 lines · Steven Rostedt

Arnaldo,

How can I make perf trace not confused by the extra fields in the system call trace events?

Ftrace can now show the contents of the system call user space buffers, but it appears that this breaks perf!!!

system: syscalls
name: sys_enter_write
ID: 791
format:

	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:unsigned int fd;	offset:16;	size:8;	signed:0;
	field:const char * buf;	offset:24;	size:8;	signed:0;
	field:size_t count;	offset:32;	size:8;	signed:0;
	field:__data_loc char[] __buf_val;	offset:40;	size:4;	signed:0;
Quoted 2 lines

That new __buf_val appears to confuse perf, but I'm having a hell of a time trying to figure out where it reads it!

I've discussed with Steven and concluded that we should change perf to ignore fields with "__data_loc char[]" type in syscalls. Let me take a look.

Thanks,
Namhyung

#13 lore raw

Hi guys,

Quoted 26 lines · Namhyung Kim
Quoted 24 lines · Steven Rostedt

Arnaldo,

How can I make perf trace not confused by the extra fields in the system call trace events?

Ftrace can now show the contents of the system call user space buffers, but it appears that this breaks perf!!!

system: syscalls
name: sys_enter_write
ID: 791
format:

      field:unsigned short common_type;       offset:0;       size:2; signed:0;
      field:unsigned char common_flags;       offset:2;       size:1; signed:0;
      field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
      field:int common_pid;   offset:4;       size:4; signed:1;

      field:int __syscall_nr; offset:8;       size:4; signed:1;
      field:unsigned int fd;  offset:16;      size:8; signed:0;
      field:const char * buf; offset:24;      size:8; signed:0;
      field:size_t count;     offset:32;      size:8; signed:0;
      field:__data_loc char[] __buf_val;      offset:40;      size:4; signed:0;
Quoted 6 lines
Quoted 2 lines

That new __buf_val appears to confuse perf, but I'm having a hell of a time trying to figure out where it reads it!

I've discussed with Steven and concluded that we should change perf to ignore fields with "__data_loc char[]" type in syscalls. Let me take a look.

Thanks, I'll also give it a look.

Thanks,
Howard

Quoted 5 lines

Thanks,
Namhyung

#14 lore raw
In reply to Steven Rostedt
Quoted 17 lines · Steven Rostedt
Quoted 14 lines · On Wed, 26 Nov 2025 08:13:00 +0100 Thomas Richter <[email protected]> wrote:
Quoted 13 lines
Quoted 2 lines

I haven't tried it without the patches. Does it usually show what "buf" is? Now with the reading of user space, it can show the content too!

Yep, it reads the content using BPF. This is on my 6.16 kernel.

  $ sudo perf trace -e write -- /bin/echo hello
  hello
       0.000 ( 0.014 ms): echo/61922 write(fd: 1, buf: hello\10, count: 6)                                 = 6

Thanks,
Namhyung

Quoted 8 lines
Quoted 4 lines

Hello Namhyung, Steven,

friendly ping... any progress here?

I honestly have no clue how to fix this, as I don't even know where to look. Is it BPF that is messing up? If so, where's the BPF program that is doing this.

Yeah, sounds very familiar... happens to me all the time :-)>

Quoted 6 lines

I thought BPF is supposed to handle updates and should never cause API breakage?

I'll continue to look at the builtin-trace.c, but it seems that the BPF program it's attached to is handing it garbage with:

Quoted 6 lines

perf trace -e syscalls:sys_enter_write

The new fields are at the end. The BPF program should simply ignore those values. But again, I don't know where this BPF program lives.

-- Steve

Ok, this sounds that the issue will linger around for some time.

Can we then adopt the test case check for perf test case 'perf trace BTF general tests' to accept that strange 4th parameter?
Right now this test fails on a daily basis in linux-next and will most likely be in linux repo soon, where it will also fail.

This is sort of 'unfortunate' that our daily continuous integration testing reports several errors per day...
If you guys object to that fix I sent out a few days ago (which is absolutely ok), I will disable the test in our CI test suite.

Thanks a lot.

Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Tested-by Reviewed-by on v1 1/1
#15 lore raw
In reply to Thomas Richter
Quoted 2 lines · Thomas Richter

The following test case fails on linux-next repo:

---8<--- snip ---8<---

Quoted 12 lines

#

---
 tools/perf/tests/shell/trace_btf_general.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/trace_btf_general.sh +0 -0
index ef2da806be6b..9cd6180062d8 100755

tools/perf/tests/shell/trace_btf_general.sh trace_test_buffer() { · line 45

Quoted 8 lines
tools/perf/tests/shell/trace_btf_general.sh +1 -1
@@ -39,7 +39,7 @@ trace_test_buffer() {
   echo "Testing perf trace's buffer augmentation"
   # echo will insert a newline (\10) at the end of the buffer
   output="$(perf trace --sort-events -e write --max-events=1 -- echo "${buffer}" 2>&1)"
-  if ! echo "$output" | grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  if ! echo "$output" | grep -qE "^echo/[0-9]+ write\([0-9]+, ${buffer}\\\\10, [0-9]+(, ..)?\) += +[0-9]+$"
   then
     printf "Buffer augmentation test failed, output:\n$output\n"
     err=1

---------------8<---------------

Thank you Thomas

Nit: maybe (, \"\")? instead of (, ..)? for clarity?

#16 lore raw
Quoted 25 lines · Jan Polensky
Quoted 2 lines · Thomas Richter

The following test case fails on linux-next repo:

---8<--- snip ---8<---

Quoted 12 lines

#

---
 tools/perf/tests/shell/trace_btf_general.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/trace_btf_general.sh +0 -0
index ef2da806be6b..9cd6180062d8 100755

tools/perf/tests/shell/trace_btf_general.sh trace_test_buffer() { · line 45

Quoted 8 lines
tools/perf/tests/shell/trace_btf_general.sh +1 -1
@@ -39,7 +39,7 @@ trace_test_buffer() {
   echo "Testing perf trace's buffer augmentation"
   # echo will insert a newline (\10) at the end of the buffer
   output="$(perf trace --sort-events -e write --max-events=1 -- echo "${buffer}" 2>&1)"
-  if ! echo "$output" | grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  if ! echo "$output" | grep -qE "^echo/[0-9]+ write\([0-9]+, ${buffer}\\\\10, [0-9]+(, ..)?\) += +[0-9]+$"
   then
     printf "Buffer augmentation test failed, output:\n$output\n"
     err=1

---------------8<---------------

Quoted 7 lines

Thank you Thomas

Nit: maybe (, \"\")? instead of (, ..)? for clarity?

Thanks will do it.

Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294