TP_printk() bug with %c, and more?
Hello Linux tracing maintainers,
I've come across an unexpected behaviour in the kernel tracing infrastructure that looks like a bug, or maybe two.
Cc-ing ASoC maintainers for as it appeared using ASoC traces, but it does not look ASoC-specific.
It all started when using this trace-cmd sequence on an ARM64 board running a mainline 6.8.0-rc7 kernel:
trace-cmd record -e snd_soc_dapm_path ./my-play
trace-cmd report
While this produces perfectly valid traces for other asoc events, the snd_soc_dapm_path produces:
snd_soc_dapm_path: >c<* MIC1_EN <- (direct) <-
instead of the expected:
snd_soc_dapm_path: *MIC1 <- (direct) <- MIC1_EN
The originating macro is:
TP_printk("%c%s %s %s %s %s",
(int) __entry->path_node &&
(int) __entry->path_connect ? '*' : ' ',
__get_str(wname), DAPM_ARROW(__entry->path_dir),
__get_str(pname), DAPM_ARROW(__entry->path_dir),
__get_str(pnname))It appears as if the %c placeholder always produces the three ">c<" characters, the '*' or ' ' char is printed as the first %s, all the other strings are shifted right by one position and the last string is never printed.
On my x86_64 laptop running the default Ubuntu kernel (6.5) I'm able to trace a few events having a '%c' in their TP_printk() macros and the result is:
intel_pipe_update_start: dev 0000:00:02.0, pipe >c<, frame=1, scanline=107856, min=2208, max=2154
originating from:
TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u",
Here it looks like the %c produced ">c<" again, but apparently without any shifting.
Back on the ARM64 board I found a couple interesting clues.
First, using the <debugfs>/tracing/ interface instead of trace-cmd, I'm getting correctly formatted strings:
trace-cmd: snd_soc_dapm_path: >c<* HPOUT_L -> (direct) -> debugfs: snd_soc_dapm_path: *HPOUT_L <- (direct) <- HPOUT_POP_SOUND_L
Notice the arrows pointing to the opposite direction though. The correct arrow is the one in the debugfs run.
Second, I tried a simple test:
TP_printk("(%c,%c,%c,%c) [%s,%s,%s,%s]",
'A',
'B',
'C',
'D',
"Just",
"a",
"stupid",
"test") and this logs:
snd_soc_dapm_path: (>c<,>c<,>c<,>c<) [A,B,C,D]
so it looks like there really is something wrong with %c in TP_printk(), and the %c in the format string do not consume any parameters, de facto shifting them to the right.
As one may expect, avoiding the %c fixes formatting:
- TP_printk("%c%s %s %s %s %s",
+ TP_printk("%s%s %s %s %s %s",
(int) __entry->path_node &&
- (int) __entry->path_connect ? '*' : ' ',
+ (int) __entry->path_connect ? "*" : " ",
__get_str(wname), DAPM_ARROW(__entry->path_dir),
__get_str(pname), DAPM_ARROW(__entry->path_dir),
__get_str(pnname))With this change, the string formatting is correct both with debugfs and trace-cmd, but the arrows are still wrong with trace-cmd.
I have no idea how to further debug this and after a quick look at the macros I can honestly say I'm not feeling brave enough to dig into them in a late Friday afternoon.
Any hints?
Am I doing anything wrong?
Is %c supposed to work in tracing macros?
Best regards,
Luca
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Steven Rostedt
maintainerQuoted 1 line · On Fri, 15 Mar 2024 17:49:00 +0100 Luca Ceresoli <[email protected]>…
Hello Linux tracing maintainers,
Hi Luca!
Quoted 36 lines
I've come across an unexpected behaviour in the kernel tracing infrastructure that looks like a bug, or maybe two.
Cc-ing ASoC maintainers for as it appeared using ASoC traces, but it does not look ASoC-specific.
It all started when using this trace-cmd sequence on an ARM64 board running a mainline 6.8.0-rc7 kernel:
trace-cmd record -e
snd_soc_dapm_path./my-play
trace-cmd reportWhile this produces perfectly valid traces for other asoc events, the
snd_soc_dapm_pathproduces:
snd_soc_dapm_path: >c<*MIC1_EN<- (direct) <-instead of the expected:
snd_soc_dapm_path: *MIC1 <- (direct) <-MIC1_ENThe originating macro is:
TP_printk("%c%s %s %s %s %s", (int) __entry->path_node && (int) __entry->path_connect ? '*' : ' ', __get_str(wname), DAPM_ARROW(__entry->path_dir), __get_str(pname), DAPM_ARROW(__entry->path_dir), __get_str(pnname))It appears as if the %c placeholder always produces the three ">c<" characters, the '*' or ' ' char is printed as the first %s, all the other strings are shifted right by one position and the last string is never printed.
Quoted 7 lines
On my
x86_64laptop running the default Ubuntu kernel (6.5) I'm able to trace a few events having a '%c' in theirTP_printk()macros and the result is:
intel_pipe_update_start: dev 0000:00:02.0, pipe >c<, frame=1, scanline=107856, min=2208, max=2154
What does /sys/kernel/tracing/trace show?
If that's fine, then the bug is in libtraceevent and not the kernel.
I'm testing it out now, and I see %c not being processed properly by libtraceevent. I'll take a deeper look.
Thanks for the report.
-- Steve
Quoted 21 lines
originating from:
TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u",
Here it looks like the %c produced ">c<" again, but apparently without any shifting.
Back on the ARM64 board I found a couple interesting clues.
First, using the <debugfs>/tracing/ interface instead of trace-cmd, I'm getting correctly formatted strings:
trace-cmd: snd_soc_dapm_path: >c<* HPOUT_L -> (direct) -> debugfs: snd_soc_dapm_path: *HPOUT_L <- (direct) <- HPOUT_POP_SOUND_LNotice the arrows pointing to the opposite direction though. The correct arrow is the one in the debugfs run.
Second, I tried a simple test:
TP_printk("(%c,%c,%c,%c) [%s,%s,%s,%s]",
Hello Steven,
thanks for the quick feedback!
Quoted 47 lines · On Fri, 15 Mar 2024 13:21:46 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 15 Mar 2024 17:49:00 +0100 Luca Ceresoli <[email protected]>…
Hello Linux tracing maintainers,
Hi Luca!
Quoted 40 lines
I've come across an unexpected behaviour in the kernel tracing infrastructure that looks like a bug, or maybe two.
Cc-ing ASoC maintainers for as it appeared using ASoC traces, but it does not look ASoC-specific.
It all started when using this trace-cmd sequence on an ARM64 board running a mainline 6.8.0-rc7 kernel:
trace-cmd record -e
snd_soc_dapm_path./my-play
trace-cmd reportWhile this produces perfectly valid traces for other asoc events, the
snd_soc_dapm_pathproduces:
snd_soc_dapm_path: >c<*MIC1_EN<- (direct) <-instead of the expected:
snd_soc_dapm_path: *MIC1 <- (direct) <-MIC1_ENThe originating macro is:
TP_printk("%c%s %s %s %s %s", (int) __entry->path_node && (int) __entry->path_connect ? '*' : ' ', __get_str(wname), DAPM_ARROW(__entry->path_dir), __get_str(pname), DAPM_ARROW(__entry->path_dir), __get_str(pnname))It appears as if the %c placeholder always produces the three ">c<" characters, the '*' or ' ' char is printed as the first %s, all the other strings are shifted right by one position and the last string is never printed.
On my
x86_64laptop running the default Ubuntu kernel (6.5) I'm able to trace a few events having a '%c' in theirTP_printk()macros and the result is:
Quoted 5 lines
Quoted 3 lines
intel_pipe_update_start: dev 0000:00:02.0, pipe >c<, frame=1, scanline=107856, min=2208, max=2154What does
/sys/kernel/tracing/traceshow?
It is correct:
intel_pipe_update_start: dev 0000:00:02.0, pipe B, frame=377644, scanline=1466, min=2154, max=2159
Quoted 4 lines
If that's fine, then the bug is in libtraceevent and not the kernel.
I'm testing it out now, and I see %c not being processed properly by libtraceevent. I'll take a deeper look.
Thanks.
Quoted 9 lines
Quoted 9 lines
originating from:
TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u",
Here it looks like the %c produced ">c<" again, but apparently without any shifting.
Back on the ARM64 board I found a couple interesting clues.
Quoted 8 lines
Quoted 8 lines
First, using the <debugfs>/tracing/ interface instead of trace-cmd, I'm getting correctly formatted strings:
trace-cmd: snd_soc_dapm_path: >c<* HPOUT_L -> (direct) -> debugfs: snd_soc_dapm_path: *HPOUT_L <- (direct) <- HPOUT_POP_SOUND_LNotice the arrows pointing to the opposite direction though. The correct arrow is the one in the debugfs run.
This other issue appears a separate bug however.
Luca
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Steven Rostedt
maintainerQuoted 69 lines · On Fri, 15 Mar 2024 19:03:12 +0100 Luca Ceresoli <[email protected]>…
Quoted 40 lines
Quoted 40 lines
I've come across an unexpected behaviour in the kernel tracing infrastructure that looks like a bug, or maybe two.
Cc-ing ASoC maintainers for as it appeared using ASoC traces, but it does not look ASoC-specific.
It all started when using this trace-cmd sequence on an ARM64 board running a mainline 6.8.0-rc7 kernel:
trace-cmd record -e
snd_soc_dapm_path./my-play
trace-cmd reportWhile this produces perfectly valid traces for other asoc events, the
snd_soc_dapm_pathproduces:
snd_soc_dapm_path: >c<*MIC1_EN<- (direct) <-instead of the expected:
snd_soc_dapm_path: *MIC1 <- (direct) <-MIC1_ENThe originating macro is:
TP_printk("%c%s %s %s %s %s", (int) __entry->path_node && (int) __entry->path_connect ? '*' : ' ', __get_str(wname), DAPM_ARROW(__entry->path_dir), __get_str(pname), DAPM_ARROW(__entry->path_dir), __get_str(pnname))It appears as if the %c placeholder always produces the three ">c<" characters, the '*' or ' ' char is printed as the first %s, all the other strings are shifted right by one position and the last string is never printed.
On my
x86_64laptop running the default Ubuntu kernel (6.5) I'm able to trace a few events having a '%c' in theirTP_printk()macros and the result is:Quoted 5 lines
Quoted 3 lines
intel_pipe_update_start: dev 0000:00:02.0, pipe >c<, frame=1, scanline=107856, min=2208, max=2154What does
/sys/kernel/tracing/traceshow?It is correct:
intel_pipe_update_start: dev 0000:00:02.0, pipe B, frame=377644, scanline=1466, min=2154, max=2159Quoted 4 lines
If that's fine, then the bug is in libtraceevent and not the kernel.
I'm testing it out now, and I see %c not being processed properly by libtraceevent. I'll take a deeper look.
Thanks.
Quoted 12 lines
Quoted 12 lines
originating from:
TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u",
Here it looks like the %c produced ">c<" again, but apparently without any shifting.
Back on the ARM64 board I found a couple interesting clues.
First, using the <debugfs>/tracing/ interface instead of trace-cmd, I'm getting correctly formatted strings:
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
trace-cmd: snd_soc_dapm_path: >c<* HPOUT_L -> (direct) -> debugfs: snd_soc_dapm_path: *HPOUT_L <- (direct) <- HPOUT_POP_SOUND_LNotice the arrows pointing to the opposite direction though. The correct arrow is the one in the debugfs run.
This other issue appears a separate bug however.
Can you make user you have the latest libtraceevent from:
git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
And apply this patch.
Thanks,
-- Steve
src/event-parse.c +12 -1
@@ -3732,8 +3732,19 @@ process_arg_token(struct tep_event *event, struct tep_print_arg *arg, arg->atom.atom = atom;break ; -case TEP_EVENT_DQUOTE :case TEP_EVENT_SQUOTE : + arg->type =TEP_PRINT_ATOM ; +/* Make characters into numbers */ +if (asprintf (&arg->atom.atom,"%d" , token[0 ]) <0 ) { +free_token (token); + *tok =NULL ; + arg->atom.atom =NULL ; +return TEP_EVENT_ERROR ; + } +free_token (token); + type =read_token_item (event->tep, &token); +break ; +case TEP_EVENT_DQUOTE : arg->type =TEP_PRINT_ATOM ; arg->atom.atom = token; type =read_token_item (event->tep, &token);
Hello Steven,
Quoted 105 lines · On Fri, 15 Mar 2024 14:58:52 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 69 lines · On Fri, 15 Mar 2024 19:03:12 +0100 Luca Ceresoli <[email protected]>…
Quoted 40 lines
Quoted 40 lines
I've come across an unexpected behaviour in the kernel tracing infrastructure that looks like a bug, or maybe two.
Cc-ing ASoC maintainers for as it appeared using ASoC traces, but it does not look ASoC-specific.
It all started when using this trace-cmd sequence on an ARM64 board running a mainline 6.8.0-rc7 kernel:
trace-cmd record -e
snd_soc_dapm_path./my-play
trace-cmd reportWhile this produces perfectly valid traces for other asoc events, the
snd_soc_dapm_pathproduces:
snd_soc_dapm_path: >c<*MIC1_EN<- (direct) <-instead of the expected:
snd_soc_dapm_path: *MIC1 <- (direct) <-MIC1_ENThe originating macro is:
TP_printk("%c%s %s %s %s %s", (int) __entry->path_node && (int) __entry->path_connect ? '*' : ' ', __get_str(wname), DAPM_ARROW(__entry->path_dir), __get_str(pname), DAPM_ARROW(__entry->path_dir), __get_str(pnname))It appears as if the %c placeholder always produces the three ">c<" characters, the '*' or ' ' char is printed as the first %s, all the other strings are shifted right by one position and the last string is never printed.
On my
x86_64laptop running the default Ubuntu kernel (6.5) I'm able to trace a few events having a '%c' in theirTP_printk()macros and the result is:Quoted 5 lines
Quoted 3 lines
intel_pipe_update_start: dev 0000:00:02.0, pipe >c<, frame=1, scanline=107856, min=2208, max=2154What does
/sys/kernel/tracing/traceshow?It is correct:
intel_pipe_update_start: dev 0000:00:02.0, pipe B, frame=377644, scanline=1466, min=2154, max=2159Quoted 4 lines
If that's fine, then the bug is in libtraceevent and not the kernel.
I'm testing it out now, and I see %c not being processed properly by libtraceevent. I'll take a deeper look.
Thanks.
Quoted 12 lines
Quoted 12 lines
originating from:
TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u",
Here it looks like the %c produced ">c<" again, but apparently without any shifting.
Back on the ARM64 board I found a couple interesting clues.
First, using the <debugfs>/tracing/ interface instead of trace-cmd, I'm getting correctly formatted strings:
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
trace-cmd: snd_soc_dapm_path: >c<* HPOUT_L -> (direct) -> debugfs: snd_soc_dapm_path: *HPOUT_L <- (direct) <- HPOUT_POP_SOUND_LNotice the arrows pointing to the opposite direction though. The correct arrow is the one in the debugfs run.
This other issue appears a separate bug however.
Can you make user you have the latest libtraceevent from:
git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.gitAnd apply this patch.
Thanks,
-- Steve
src/event-parse.c +5
-1@@ -3732,8 +3732,19 @@ process_arg_token(struct tep_event *event, struct tep_print_arg *arg, arg->atom.atom = atom;break ; -case TEP_EVENT_DQUOTE :case TEP_EVENT_SQUOTE : + arg->type =TEP_PRINT_ATOM ; +/* Make characters into numbers */ +if (asprintf (&arg->atom.atom,"%d" , token[0 ]) <0 ) { +free_token (token); + *tok =NULL ;
src/event-parse.c process_arg_token(struct tep_event *event, struct tep_print_arg *arg, · line 3750
Quoted 10 lines
src/event-parse.c +7
-0+ arg->atom.atom =NULL ; +return TEP_EVENT_ERROR ; + } +free_token (token); + type =read_token_item (event->tep, &token); +break ; +case TEP_EVENT_DQUOTE : arg->type =TEP_PRINT_ATOM ; arg->atom.atom = token; type =read_token_item (event->tep, &token);
Indeed I was on an older version, apologies.
I upgraded both libtraceevent and trace-cmd to master and applied your patch, now the %c is formatted correctly.
However the arrows are still reversed.
Is this what you were expecting?
Luca
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Steven Rostedt
maintainerQuoted 8 lines · On Mon, 18 Mar 2024 16:43:07 +0100 Luca Ceresoli <[email protected]>…
Indeed I was on an older version, apologies.
I upgraded both libtraceevent and trace-cmd to master and applied your patch, now the %c is formatted correctly.
However the arrows are still reversed.
Is this what you were expecting?
No, I didn't look at the arrows, just the %c issue. I'll try to get some time to do that.
-- Steve
Steven Rostedt
maintainerQuoted 1 line · On Mon, 18 Mar 2024 16:43:07 +0100 Luca Ceresoli <[email protected]>…
However the arrows are still reversed.
This requires a kernel change. The problem is that the print fmt has:
print fmt: "%c%s %s %s %s %s", (int) REC->path_node && (int) REC->path_connect ? '*' : ' ', __get_str(wname), (((REC->path_dir) == SND_SOC_DAPM_DIR_OUT) ? "->" : "<-"), __get_str(pname), (((REC->path_dir) == SND_SOC_DAPM_DIR_OUT) ? "->" : "<-"), __get_str(pnname)
User space (trace-cmd and perf) have no idea what SND_SOC_DAPM_DIR_OUT is. The kernel needs to convert that, otherwise the parsing will fail, or it will default it to zero.
-- Steve
Hello Steven,
Quoted 5 lines · On Mon, 15 Apr 2024 04:44:30 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Mon, 18 Mar 2024 16:43:07 +0100 Luca Ceresoli <[email protected]>…
However the arrows are still reversed.
Quoted 7 lines
This requires a kernel change. The problem is that the print fmt has:
print fmt: "%c%s %s %s %s %s", (int)
REC->path_node&& (int)REC->path_connect? '*' : ' ',__get_str(wname), (((REC->path_dir) ==SND_SOC_DAPM_DIR_OUT) ? "->" : "<-"),__get_str(pname), (((REC->path_dir) ==SND_SOC_DAPM_DIR_OUT) ? "->" : "<-"),__get_str(pnname)User space (trace-cmd and perf) have no idea what
SND_SOC_DAPM_DIR_OUTis. The kernel needs to convert that, otherwise the parsing will fail, or it will default it to zero.
Thanks for the insight. I'm definitely trying to fix this based on your hint as soon as I get my hand on a board.
Luca
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Steven Rostedt
maintainerQuoted 2 lines · On Tue, 16 Apr 2024 04:08:46 +0200 Luca Ceresoli <[email protected]>…
Thanks for the insight. I'm definitely trying to fix this based on your hint as soon as I get my hand on a board.
I have a patch I forgot to send out. Let me do that now.
-- Steve