5.19 printk breaks message ordering
Subsystems (4)
Hi John & folks,
In 5.19, I'm seeing some changes in message ordering / interleaving which lead to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually writes() to stdout. For example, here's a snippet from build.wireguard.com:
[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K
[ 0.469738] Write protecting the kernel read-only data: 10240k
[ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K
[ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K
[ 0.475236] Run /init as init process WireGuard Test Suite on Linux 5.19.0-rc2+ x86_64
[+] Mounting filesystems...
[+] Module self-tests:
* allowedips self-tests: pass
* nonce counter self-tests: pass
* ratelimiter self-tests: pass
[+] Enabling logging...
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log output
Now, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
I assume this is mostly caused by your threaded printk patchset, which moves some of that printing into a worker, which means it's more dependent on the scheduler than before. This probably has important benefits. But it certainly makes CI and related debugging a bit trickier as a result.
So I was wondering if there was some way to boot the kernel with a command line option or compile-time flag that always flushes printk messages when they're made, or does something to make the ordering a bit more faithful.
Thanks,
Jason
Quoted 4 lines · Jason A. Donenfeld
So I was wondering if there was some way to boot the kernel with a command line option or compile-time flag that always flushes printk messages when they're made, or does something to make the ordering a bit more faithful.
I'll scratch my own itch. Patch incoming.
Jason
PATCH 1/1 printk: allow direct console printing to be enabled always
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually writes() to stdout. For example, here's a snippet from build.wireguard.com:
[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K
[ 0.469738] Write protecting the kernel read-only data: 10240k
[ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K
[ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K
[ 0.475236] Run /init as init process WireGuard Test Suite on Linux 5.19.0-rc2+ x86_64
[+] Mounting filesystems...
[+] Module self-tests:
* allowedips self-tests: pass
* nonce counter self-tests: pass
* ratelimiter self-tests: pass
[+] Enabling logging...
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log output
Now, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
This is caused by threaded printk. In order to restore this in debugging sessions and in CI, this commit adds the ability to always use direct printk, either set by default at compile time, or overridden with a runtime command line switch.
--- Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++ init/Kconfig | 12 ++++++++++++ kernel/printk/printk.c | 12 ++++++++++++ 3 files changed, 32 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -4389,6 +4389,14 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.always_direct= + Rather than using kthreads for printk output, always + write to the console immediately. This has performance + implications, but will result in a more faithful + ordering and interleaving with other processes writing + to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable) + processor.max_cstate= [HW,ACPI] Limit processor to maximum C-state max_cstate=9 overrides any DMI blacklist limit.
init/Kconfig +12 -0
@@ -798,6 +798,18 @@ config PRINTK_INDEX There is no additional runtime cost to printk with this enabled. +config PRINTK_ALWAYS_DIRECT + bool "Flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always write to the + console immediately. This has performance implications, but will + result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.always_direct=0/1. + # # Architectures with an unreliable sched_clock() should select this: #
kernel/printk/printk.c +12 -0
@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool always_direct_printk =IS_ENABLED (CONFIG_PRINTK_ALWAYS_DIRECT ); + +static int __init control_always_direct_printk (char *str) +{ +return kstrtobool (str, &always_direct_printk); +} +__setup ("printk.always_direct=" , control_always_direct_printk); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
@@ -471,6 +479,10 @@ void printk_prefer_direct_exit(void)*/ static inline bool allow_direct_printing (void ) { +/* If the user has explicitly enabled this to be on always. */ +if (always_direct_printk) +return true; +/* * Checking kthread availability is a possible race because the * kthread printers can become permanently disabled during runtime.
Signature · 1 line
2.35.1
Randy Dunlap
reviewerRe: [PATCH] printk: allow direct console printing to be enabled always
Quoted 5 lines · Jason A. Donenfeld
Documentation/admin-guide/kernel-parameters.txt +0
-0
Documentation/admin-guide/kernel-parameters.txt line 4398
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -4389,6 +4389,14 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.always_direct= + Rather than using kthreads for printk output, always + write to the console immediately. This has performance + implications, but will result in a more faithful + ordering and interleaving with other processes writing + to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable)
or:
+ Format: <bool> (1/Y/y/T/t/on=enable, 0/N/n/F/f/off=disable)
Signature · 1 line
~Randy
Re: [PATCH] printk: allow direct console printing to be enabled always
Quoted 23 lines · Randy Dunlap
Quoted 6 lines · Jason A. Donenfeld
Documentation/admin-guide/kernel-parameters.txt +0
-0Documentation/admin-guide/kernel-parameters.txt line 4398
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -4389,6 +4389,14 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.always_direct= + Rather than using kthreads for printk output, always + write to the console immediately. This has performance + implications, but will result in a more faithful + ordering and interleaving with other processes writing + to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable)or:
Review
Quoted 1 line
(quoted diff) +1
-0+ Format: <bool> (1/Y/y/T/t/on=enable, 0/N/n/F/f/off=disable)
This is the same as the lines above and below. So I think you'd be better off sending an independent patch that fixes all of these in one fell swoop, rather than introducing an inconsistency in this one.
Jason
John Ogness
maintainerRe: [PATCH] printk: allow direct console printing to be enabled always
In reply to Jason A. DonenfeldQuoted 12 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K [ 0.469738] Write protecting the kernel read-only data: 10240k [ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K [ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K [ 0.475236] Run /init as init process
Quoted 16 lines
WireGuard Test Suite on Linux 5.19.0-rc2+ x86_64
[+] Mounting filesystems...
[+] Module self-tests:
* allowedips self-tests: pass
* nonce counter self-tests: pass
* ratelimiter self-tests: pass
[+] Enabling logging...
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1As Petr mentioned, this issue is not new. But with kthreads it has become the common case.
Documentation/admin-guide/kernel-parameters.txt line 4392
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +1
-0+ printk.always_direct=
Do we need the word "always" in there? I would prefer the simplied:
printk.direct=
Documentation/admin-guide/kernel-parameters.txt line 4394
Quoted 2 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ Rather than using kthreads for printk output, always + write to the console immediately. This has performance
The "best effort" part needs to be in there. Perhaps:
Rather than using kthreads for printk output, always attempt to write to the console immediately.
Documentation/admin-guide/kernel-parameters.txt line 4397
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ implications, but will result in a more faithful + ordering and interleaving with other processes writing + to the console.
My main concern is that "direct printing" is not synchronous. Since 2.4.10 it has not been printk's goal to be synchronous. Your patch is making the issue less likely again, but not solving it. And since this is the first time we are documenting the printk printing behavior, we should be careful to not mislead users to think it is truly direct.
The planned solution is atomic consoles, which are able to provide a truly synchronous printk. But they are not yet available mainline.
init/Kconfig config PRINTK_INDEX · line 801
Quoted 9 lines
init/Kconfig +1
-0@@ -798,6 +798,18 @@ config PRINTK_INDEX There is no additional runtime cost to printk with this enabled. +config PRINTK_ALWAYS_DIRECT
(perhaps)
config PRINTK_DIRECT
init/Kconfig config PRINTK_INDEX · line 802
Quoted 1 line
init/Kconfig +1
-0+ bool "Flush printk output immediately"
Attempt to flush printk output immediately
init/Kconfig config PRINTK_INDEX · line 805
Quoted 3 lines
init/Kconfig +3
-0+ depends on PRINTK + help + Rather than using kthreads for printk output, always write to the
always attempt to write
init/Kconfig config PRINTK_INDEX · line 811
Quoted 6 lines
init/Kconfig +6
-0+ console immediately. This has performance implications, but will + result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.always_direct=0/1.
(perhaps)
printk.direct=0/1
kernel/printk/printk.c void printk_prefer_direct_exit(void) · line 484
Quoted 11 lines
kernel/printk/printk.c +3
-0@@ -471,6 +479,10 @@ void printk_prefer_direct_exit(void)*/ static inline bool allow_direct_printing (void ) { +/* If the user has explicitly enabled this to be on always. */ +if (always_direct_printk) +return true;
There is no reason to start the threads if they won't be used. Perhaps something like this instead:
-------- BEGIN HUNK -------
@@ -3602,11 +3610,13 @@ static int __init printk_activate_kthreads(void)
{
struct console *con;
- console_lock();
- printk_kthreads_available = true;
- for_each_console(con)
- printk_start_kthread(con);
- console_unlock();
+ if (!always_direct_printk) {
+ console_lock();
+ printk_kthreads_available = true;
+ for_each_console(con)
+ printk_start_kthread(con);
+ console_unlock();
+ }
return 0;
}
-------- END HUNK -------
Some general comments from me...
Direct printing has a lot of technical issues. It is sometimes directly responsible for kernels dying. It is sometimes directly responsible for preventing important information from being made available at crash time. For the fbcon, direct printing is a ticking timebomb. And in many cases it isn't even truly doing direct printing anyway.
Direct printing (in its current form) must be phased out at some point. We are working to bring true synchronous console printing mainline.
Aside from my above comments, I have no problems with this patch.
John Ogness
Re: [PATCH] printk: allow direct console printing to be enabled always
Hi John,
Thanks for your review. I'll have a v2 for you shortly.
Quoted 4 lines · John Ogness
Documentation/admin-guide/kernel-parameters.txt line 4392
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +1
-0+ printk.always_direct=Do we need the word "always" in there? I would prefer the simplied:
printk.direct=
Sure, no problem. I'll do that for v2.
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 4394
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ Rather than using kthreads for printk output, always + write to the console immediately. This has performanceThe "best effort" part needs to be in there. Perhaps:
Rather than using kthreads for printk output, always attempt to write to the console immediately.
Ack.
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt line 4397
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ implications, but will result in a more faithful + ordering and interleaving with other processes writing + to the console.My main concern is that "direct printing" is not synchronous. Since 2.4.10 it has not been printk's goal to be synchronous. Your patch is making the issue less likely again, but not solving it. And since this is the first time we are documenting the printk printing behavior, we should be careful to not mislead users to think it is truly direct.
Sure, that makes sense. The "attempt to" language clarifies that well.
Quoted 4 lines
Quoted 1 line
+config
PRINTK_ALWAYS_DIRECT(perhaps)
configPRINTK_DIRECT
Ack.
Quoted 4 lines
init/Kconfig config PRINTK_INDEX · line 802
Quoted 2 lines
init/Kconfig +1
-0+ bool "Flush printk output immediately"Attempt to flush printk output immediately
Ack.
Quoted 6 lines
init/Kconfig config PRINTK_INDEX · line 805
Quoted 4 lines
init/Kconfig +3
-0+ depends on PRINTK + help + Rather than using kthreads for printk output, always write to thealways attempt to write
Ack.
Quoted 10 lines
init/Kconfig config PRINTK_INDEX · line 811
Quoted 7 lines
init/Kconfig +6
-0+ console immediately. This has performance implications, but will + result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.always_direct=0/1.(perhaps)
printk.direct=0/1
Ack.
Quoted 27 lines
kernel/printk/printk.c void printk_prefer_direct_exit(void) · line 484
Quoted 12 lines
kernel/printk/printk.c +3
-0@@ -471,6 +479,10 @@ void printk_prefer_direct_exit(void)*/ static inline bool allow_direct_printing (void ) { +/* If the user has explicitly enabled this to be on always. */ +if (always_direct_printk) +return true;There is no reason to start the threads if they won't be used. Perhaps something like this instead:
-------- BEGIN HUNK -------
(quoted diff) +1
-5@@ -3602,11 +3610,13 @@ static int __init printk_activate_kthreads(void) { struct console *con; - console_lock(); - printk_kthreads_available = true; - for_each_console(con) - printk_start_kthread(con); - console_unlock(); + if (!always_direct_printk) {
Review static int __init printk_activate_kthreads(void)
Quoted 10 lines
(quoted diff) +6
-1+ console_lock(); + printk_kthreads_available = true; + for_each_console(con) + printk_start_kthread(con); + console_unlock(); + } return 0; } -------- END HUNK -------
Good thinking. I'll do it like this.
Quoted 11 lines
Direct printing has a lot of technical issues. It is sometimes directly responsible for kernels dying. It is sometimes directly responsible for preventing important information from being made available at crash time. For the fbcon, direct printing is a ticking timebomb. And in many cases it isn't even truly doing direct printing anyway.
Direct printing (in its current form) must be phased out at some point. We are working to bring true synchronous console printing mainline.
Aside from my above comments, I have no problems with this patch.
That's a fair point, though it remains a valuable resource for debugging and CI. The way this patch is written, it defaults to off, and the help text is kind of discouraging of its use, which is what we want I think.
v2 on its way.
Jason
PATCH v2 1/1 printk: allow direct console printing to be enabled always
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually writes() to stdout. For example, here's a snippet from build.wireguard.com:
[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K
[ 0.469738] Write protecting the kernel read-only data: 10240k
[ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K
[ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K
[ 0.475236] Run /init as init process WireGuard Test Suite on Linux 5.19.0-rc2+ x86_64
[+] Mounting filesystems...
[+] Module self-tests:
* allowedips self-tests: pass
* nonce counter self-tests: pass
* ratelimiter self-tests: pass
[+] Enabling logging...
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log output
Now, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
This is caused by threaded printk. In order to restore this in debugging sessions and in CI, this commit adds the ability to always use direct printk, either set by default at compile time, or overridden with a runtime command line switch.
--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ init/Kconfig | 12 ++++++++++++ kernel/printk/printk.c | 11 +++++++++++ 3 files changed, 30 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -4389,6 +4389,13 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.direct= Rather than using kthreads for printk output, always + attempt to write to the console immediately. This has + performance implications, but will result in a more + faithful ordering and interleaving with other + processes writing to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable) + processor.max_cstate= [HW,ACPI] Limit processor to maximum C-state max_cstate=9 overrides any DMI blacklist limit.
init/Kconfig +12 -0
@@ -798,6 +798,18 @@ config PRINTK_INDEX There is no additional runtime cost to printk with this enabled. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + # # Architectures with an unreliable sched_clock() should select this: #
kernel/printk/printk.c +11 -0
@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct =IS_ENABLED (CONFIG_PRINTK_DIRECT ); + +static int __init control_printk_direct (char *str) +{ +return kstrtobool (str, &printk_direct); +} +__setup ("printk.direct=" , control_printk_direct); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
@@ -3602,6 +3610,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct) +return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)
Signature · 1 line
2.35.1
John Ogness
maintainerRe: [PATCH v2] printk: allow direct console printing to be enabled always
init/Kconfig
Quoted 4 lines · Jason A. Donenfeld
init/Kconfig +0
-0
Sorry, I missed this in your v1. But I think this config belongs in lib/Kconfig.debug under the "printk and dmesg options" menu.
Quoted 24 lines
init/Kconfig +12
-0@@ -798,6 +798,18 @@ config PRINTK_INDEX There is no additional runtime cost to printk with this enabled. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + # # Architectures with an unreliable sched_clock() should select this: #kernel/printk/printk.c +0
-0
kernel/printk/printk.c static int __init control_devkmsg(char *str) · line 181
Quoted 4 lines
kernel/printk/printk.c +1
-0@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct =IS_ENABLED (CONFIG_PRINTK_DIRECT );
I understand why you would name the variable to match the boot arg. But I would prefer the _internal_ variable had a name that makes it clear (to us developers) that it is a permanent setting. Perhaps printk_direct_only or printk_direct_always?
The reason is because when kthreads are active, direct printing is turned on and off dynamically (using @printk_prefer_direct). And if this new variable is named @printk_direct, one could easily mistake its meaning to be related to the dynamic turning on and off.
Quoted 11 lines
kernel/printk/printk.c +7
-0+ +static int __init control_printk_direct (char *str) +{ +return kstrtobool (str, &printk_direct); +} +__setup ("printk.direct=" , control_printk_direct); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3613
Quoted 4 lines
kernel/printk/printk.c +1
-0@@ -3602,6 +3610,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct)
I'm wondering if we should output a message here. My suggestion is:
pr_info("printing threads disabled, using direct printing\n");
kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3618
Quoted 5 lines
kernel/printk/printk.c +2
-0+return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)
Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
John Ogness
Re: [PATCH v2] printk: allow direct console printing to be enabled always
Hi John,
Quoted 8 lines · John Ogness
init/Kconfig
Quoted 4 lines · Jason A. Donenfeld
init/Kconfig +0
-0Sorry, I missed this in your v1. But I think this config belongs in
lib/Kconfig.debugunder the "printk and dmesg options" menu.
That's better. It helps drive home that it's a debug thing.
Quoted 6 lines
Quoted 1 line
+static bool
printk_direct=IS_ENABLED(CONFIG_PRINTK_DIRECT);I understand why you would name the variable to match the boot arg. But I would prefer the _internal_ variable had a name that makes it clear (to us developers) that it is a permanent setting. Perhaps
printk_direct_onlyorprintk_direct_always?
Sure, I'll do that. The variable can also be __initdata, since it's only used inside of an __init function.
Quoted 5 lines
init/Kconfig
Quoted 1 line
init/Kconfig +1
-0+ if (printk_direct)I'm wondering if we should output a message here. My suggestion is:
pr_info("printing threads disabled, using direct printing\n");
That seems a bit heavy to me, and just adds more log spam. Moving it into the debug kconfig zone seems like the right way to handle this instead.
Quoted 10 lines
init/Kconfig
Quoted 5 lines
init/Kconfig +2
-0+ return 0; + console_lock(); printk_kthreads_available = true; for_each_console(con)Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
Fair enough. It's easy enough to send a v3, and I agree with most of your suggestions, so I'll send that now, and we'll wait to hear if the others think it's fine too.
Jason
PATCH v3 1/1 printk: allow direct console printing to be enabled always
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually writes() to stdout. For example, here's a snippet from build.wireguard.com:
[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K
[ 0.469738] Write protecting the kernel read-only data: 10240k
[ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K
[ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K
[ 0.475236] Run /init as init process WireGuard Test Suite on Linux 5.19.0-rc2+ x86_64
[+] Mounting filesystems...
[+] Module self-tests:
* allowedips self-tests: pass
* nonce counter self-tests: pass
* ratelimiter self-tests: pass
[+] Enabling logging...
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log output
Now, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
This is caused by threaded printk. In order to restore this in debugging sessions and in CI, this commit adds the ability to always use direct printk, either set by default at compile time, or overridden with a runtime command line switch.
--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ kernel/printk/printk.c | 11 +++++++++++ lib/Kconfig.debug | 12 ++++++++++++ 3 files changed, 30 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -4424,6 +4424,13 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.direct= Rather than using kthreads for printk output, always + attempt to write to the console immediately. This has + performance implications, but will result in a more + faithful ordering and interleaving with other + processes writing to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable) + processor.max_cstate= [HW,ACPI] Limit processor to maximum C-state max_cstate=9 overrides any DMI blacklist limit.
kernel/printk/printk.c +11 -0
@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct_only__initdata =IS_ENABLED (CONFIG_PRINTK_DIRECT ); + +static int __init control_printk_direct_only (char *str) +{ +return kstrtobool (str, &printk_direct_only); +} +__setup ("printk.direct=" , control_printk_direct_only); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
@@ -3605,6 +3613,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct_only) +return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)
lib/Kconfig.debug +12 -0
@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + config STACKTRACE_BUILD_ID bool "Show build ID information in stacktraces" depends on PRINTK
Signature · 1 line
2.35.1
Petr Mladek
maintainerRe: [PATCH v3] printk: allow direct console printing to be enabled always
Quoted 14 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:lib/Kconfig.debug +2
-0@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately"
lib/Kconfig.debug config PRINTK_CALLER · line 49
Quoted 10 lines
lib/Kconfig.debug +10
-0+ depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. +
I am fine with the patch except for the config option. Is there any particular reason why we need it, please?
We should be careful to add new build options in general because they make building the kernel more complicated. People need to understand what each option is about and what they want.
Kernel parameters are less intrusive. People need to care about it only when they need some special behavior. And everyone has its own default command line anyway. For example, I always use ignore_loglevel.
Best Regards,
Petr
Re: [PATCH v3] printk: allow direct console printing to be enabled always
Hi Petr,
Quoted 13 lines · Petr Mladek
Quoted 12 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:lib/Kconfig.debug +0
-0@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface.
Quoted 15 lines
lib/Kconfig.debug config PRINTK_CALLER · line 49
Quoted 12 lines
lib/Kconfig.debug +12
-0+config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. +I am fine with the patch except for the config option. Is there any particular reason why we need it, please?
It's a debug option that I'd like to be able to easily build my CI kernels with, just like various other debug options.
Jason
Quoted 3 lines
We should be careful to add new build options in general because they make building the kernel more complicated. People need to understand what each option is about and what they want.
This one doesn't, really. It'd hidden away in the debug menu.
Jason
RE: [PATCH v3] printk: allow direct console printing to be enabled always
In reply to Petr MladekFrom: Petr Mladek
Quoted 31 lines
Sent: 20 June 2022 17:58
Quoted 14 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:lib/Kconfig.debug +2
-0@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately"lib/Kconfig.debug config PRINTK_CALLER · line 49
Quoted 10 lines
lib/Kconfig.debug +10
-0+ depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. +I am fine with the patch except for the config option. Is there any particular reason why we need it, please?
Quoted 3 lines
We should be careful to add new build options in general because they make building the kernel more complicated. People need to understand what each option is about and what they want.
Especially since most people use pre-built kernels.
I suspect that kernels only get build for very big and very small/embedded systems.
For the latter you may want to completely remove using kthreads for printk in order to shrink the kernel size.
But that doesn't look like what this option is for.
David
Quoted 7 lines
Kernel parameters are less intrusive. People need to care about it only when they need some special behavior. And everyone has its own default command line anyway. For example, I always use
ignore_loglevel.Best Regards,
Petr
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v3] printk: allow direct console printing to be enabled always
In reply to Jason A. DonenfeldHey John, Petr, Sergey,
Quoted 141 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K [ 0.469738] Write protecting the kernel read-only data: 10240k [ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K [ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K [ 0.475236] Run /init as init processWireGuard Test Suite on Linux 5.19.0-rc2+ x86_64 [+] Mounting filesystems... [+] Module self-tests: * allowedips self-tests: pass * nonce counter self-tests: pass * ratelimiter self-tests: pass [+] Enabling logging... [+] Launching tests... [ 0.475237] with arguments: [ 0.475238] /init [ 0.475238] with environment: [ 0.475239] HOME=/ [ 0.475240] TERM=linux [+] ip netns add wg-test-46-0 [+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log outputNow, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
This is caused by threaded printk. In order to restore this in debugging sessions and in CI, this commit adds the ability to always use direct printk, either set by default at compile time, or overridden with a runtime command line switch.
Cc John Ogness <[email protected]>Cc Petr Mladek <[email protected]>Cc Marco Elver <[email protected]>Fixes 09c5ba0aa2fc ("printk: add kthread console printers")Signed-off-by Jason A. Donenfeld <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ kernel/printk/printk.c | 11 +++++++++++ lib/Kconfig.debug | 12 ++++++++++++ 3 files changed, 30 insertions(+)Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -4424,6 +4424,13 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.direct= Rather than using kthreads for printk output, always + attempt to write to the console immediately. This has + performance implications, but will result in a more + faithful ordering and interleaving with other + processes writing to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable) + processor.max_cstate= [HW,ACPI] Limit processor to maximum C-state max_cstate=9 overrides any DMI blacklist limit.kernel/printk/printk.c +11
-0@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct_only__initdata =IS_ENABLED (CONFIG_PRINTK_DIRECT ); + +static int __init control_printk_direct_only (char *str) +{ +return kstrtobool (str, &printk_direct_only); +} +__setup ("printk.direct=" , control_printk_direct_only); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,@@ -3605,6 +3613,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct_only) +return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)lib/Kconfig.debug +12
-0@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + config STACKTRACE_BUILD_ID bool "Show build ID information in stacktraces" depends on PRINTKSignature · 1 line
2.35.1
From reading the other thread with Linus about console driver locks, it sounds like there's a good chance threaded printk will be reverted wholesale, which would of course fix this regression here. But before that happens, could you make sure this v3 gets applied, so that whenever threaded printk is reintroduced, I don't run into the same problem and repeat all the same motions to draw this up again?
Thanks,
Jason
Re: [PATCH v3] printk: allow direct console printing to be enabled always
Quoted 152 lines · Jason A. Donenfeld
Hey John, Petr, Sergey,
Quoted 141 lines · Jason A. Donenfeld
In 5.19, there are some changes in printk message ordering / interleaving which leads to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K [ 0.469738] Write protecting the kernel read-only data: 10240k [ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K [ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K [ 0.475236] Run /init as init processWireGuard Test Suite on Linux 5.19.0-rc2+ x86_64 [+] Mounting filesystems... [+] Module self-tests: * allowedips self-tests: pass * nonce counter self-tests: pass * ratelimiter self-tests: pass [+] Enabling logging... [+] Launching tests... [ 0.475237] with arguments: [ 0.475238] /init [ 0.475238] with environment: [ 0.475239] HOME=/ [ 0.475240] TERM=linux [+] ip netns add wg-test-46-0 [+] ip netns add wg-test-46-1Before the "with arguments:" and such would print prior to the "wireguard test suite on linux 5.19" banner. Now it shows after.
I see the same thing with "Freeing unused kernel image (text/rodata gap) memory" printing interwoven into the console of my initramfs on my laptop. And so forth.
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log outputNow, the kernel log outputs are all over the place and out of order with the sequence of commands. This makes debugging issues somewhat tricky, because post hoc ergo propter hoc winds up being a good intuition to follow when tracking down bugs, and now the post hoc part is muddled.
This is caused by threaded printk. In order to restore this in debugging sessions and in CI, this commit adds the ability to always use direct printk, either set by default at compile time, or overridden with a runtime command line switch.
Cc John Ogness <[email protected]>Cc Petr Mladek <[email protected]>Cc Marco Elver <[email protected]>Fixes 09c5ba0aa2fc ("printk: add kthread console printers")Signed-off-by Jason A. Donenfeld <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ kernel/printk/printk.c | 11 +++++++++++ lib/Kconfig.debug | 12 ++++++++++++ 3 files changed, 30 insertions(+)Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -4424,6 +4424,13 @@ printk.time= Show timing data prefixed to each printk message line Format: <bool> (1/Y/y=enable, 0/N/n=disable) + printk.direct= Rather than using kthreads for printk output, always + attempt to write to the console immediately. This has + performance implications, but will result in a more + faithful ordering and interleaving with other + processes writing to the console. + Format: <bool> (1/Y/y=enable, 0/N/n=disable) + processor.max_cstate= [HW,ACPI] Limit processor to maximum C-state max_cstate=9 overrides any DMI blacklist limit.kernel/printk/printk.c +11
-0@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct_only__initdata =IS_ENABLED (CONFIG_PRINTK_DIRECT ); + +static int __init control_printk_direct_only (char *str) +{ +return kstrtobool (str, &printk_direct_only); +} +__setup ("printk.direct=" , control_printk_direct_only); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,@@ -3605,6 +3613,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct_only) +return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)lib/Kconfig.debug +12
-0@@ -35,6 +35,18 @@ config PRINTK_CALLER no option to enable/disable at the kernel command line parameter or sysfs interface. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + config STACKTRACE_BUILD_ID bool "Show build ID information in stacktraces" depends on PRINTKSignature · 1 line
2.35.1From reading the other thread with Linus about console driver locks, it sounds like there's a good chance threaded printk will be reverted wholesale, which would of course fix this regression here. But before that happens, could you make sure this v3 gets applied, so that whenever threaded printk is reintroduced, I don't run into the same problem and repeat all the same motions to draw this up again?
Quoted 2 lines
Thanks,
Jason
Another thing I'm finding is that without this patch, printing various messages from userspace, and then calling reboot() results in those messages never actually making it to the console, making my test suite panic() handler kind of useless.
This patch presumably fixes it, since it becomes direct. But maybe if people don't want direct mode, but still don't want messages to get lost on reboot, you might consider some kind of flushing?
Jason
RE: [PATCH v2] printk: allow direct console printing to be enabled always
In reply to John OgnessFrom: John Ogness
Quoted 45 lines
Sent: 20 June 2022 00:17
init/Kconfig
Quoted 4 lines · Jason A. Donenfeld
init/Kconfig +0
-0Sorry, I missed this in your v1. But I think this config belongs in
lib/Kconfig.debugunder the "printk and dmesg options" menu.Quoted 24 lines
init/Kconfig +12
-0@@ -798,6 +798,18 @@ config PRINTK_INDEX There is no additional runtime cost to printk with this enabled. +config PRINTK_DIRECT + bool "Attempt to flush printk output immediately" + depends on PRINTK + help + Rather than using kthreads for printk output, always attempt to write + to the console immediately. This has performance implications, but + will result in a more faithful ordering and interleaving with other + processes writing to the console. + + Say N here unless you really need this. This may also be controlled + at boot time with printk.direct=0/1. + # # Architectures with an unreliable sched_clock() should select this: #kernel/printk/printk.c +0
-0kernel/printk/printk.c static int __init control_devkmsg(char *str) · line 181
Quoted 4 lines
kernel/printk/printk.c +1
-0@@ -178,6 +178,14 @@ static int __init control_devkmsg(char *str) }__setup ("printk.devkmsg=" , control_devkmsg); +static bool printk_direct =IS_ENABLED (CONFIG_PRINTK_DIRECT );I understand why you would name the variable to match the boot arg. But I would prefer the _internal_ variable had a name that makes it clear (to us developers) that it is a permanent setting. Perhaps
printk_direct_onlyorprintk_direct_always?
Quoted 4 lines
The reason is because when kthreads are active, direct printing is turned on and off dynamically (using @printk_prefer_direct). And if this new variable is named @printk_direct, one could easily mistake its meaning to be related to the dynamic turning on and off.
Do you need both variables?
I presume the whole lot can be compiled out (for small kernels). Apart from that having a sysctl to control which message levels get 'direct printing' (much like the one that controls whether they get printed at all) would surely be enough.
That just leaves the question of the initial level at boot.
David
Quoted 32 lines
Quoted 12 lines
kernel/printk/printk.c +7
-0+ +static int __init control_printk_direct (char *str) +{ +return kstrtobool (str, &printk_direct); +} +__setup ("printk.direct=" , control_printk_direct); +char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE ] ="ratelimit" ;#if defined(CONFIG_PRINTK ) && defined(CONFIG_SYSCTL ) int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3613
Quoted 4 lines
kernel/printk/printk.c +1
-0@@ -3602,6 +3610,9 @@ static int __init printk_activate_kthreads(void) {struct console *con; +if (printk_direct)I'm wondering if we should output a message here. My suggestion is:
pr_info("printing threads disabled, using direct printing\n");kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3618
Quoted 5 lines
kernel/printk/printk.c +2
-0+return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
Quoted 1 line
John Ogness
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Sergey Senozhatsky
reviewerRe: [PATCH v2] printk: allow direct console printing to be enabled always
In reply to John OgnessQuoted 4 lines · John Ogness
I'm wondering if we should output a message here. My suggestion is:
pr_info("printing threads disabled, using direct printing\n");
Quoted 10 lines
kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3618
Quoted 5 lines
kernel/printk/printk.c +2
-0+return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
A tricky situation. I think John already mentioned in another thread that any such config/boot param potentially can become a default value for some setups; at the same time, well, yeah, I see what commit message talks about.
May I just dump some thoughts?
I guess user-space "[+] some userspace command" are write()-s to /dev/ttyX which end up in uart circular buffer and which are printed from console IRQ handler (where it handles both TX and RX)
console_irq()
{
int count = XXX;
spin lock irqsafe port->lock
RX
do {
TX char from xmit->buf
} while (--count);
spin unlock irqrestore port->lock
}The "[ 1.2345 ] some kernel log output" (which I assume are printed
by user-space from some syscall?) now take a much more lengthy path to console and are always async/deferred.
I know that John and Petr (rightfully so) will hate what I'm about to say, but would it make sense/be possible/etc. to look into possibility to address that "deferred kthread vs IRQ" race by moving prb entries emit to the same IRQ handler that emits chars from uart xmit buffer? In other words:
console_irq()
{
int count = XXX;
spin lock irqsafe port->lock
RX
do {
TX char from xmit->buf
} while (--count);+ count = XXX;
+ do {
+ console_emit_next_record();
+ } while (--count);
spin unlock irqrestore port->lock }
Re: [PATCH v2] printk: allow direct console printing to be enabled always
Hi Sergey,
Quoted 16 lines · Sergey Senozhatsky
Quoted 4 lines · John Ogness
I'm wondering if we should output a message here. My suggestion is:
pr_info("printing threads disabled, using direct printing\n");Quoted 10 lines
kernel/printk/printk.c static int __init printk_activate_kthreads(void) · line 3618
Quoted 5 lines
kernel/printk/printk.c +2
-0+return 0 ; +console_lock (); printk_kthreads_available = true;for_each_console (con)Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
Quoted 4 lines
A tricky situation. I think John already mentioned in another thread that any such config/boot param potentially can become a default value for some setups; at the same time, well, yeah, I see what commit message talks about.
Moving it into the debug section of kconfig hopefully sends the right message there. Plus the help text makes it clear there are performance implications and such.
Jason
Sergey Senozhatsky
reviewerRe: [PATCH v2] printk: allow direct console printing to be enabled always
Quoted 5 lines · Jason A. Donenfeld
Quoted 5 lines
Quoted 4 lines
Otherwise it looks OK to me. But you may want to wait on a response from Petr, Sergey, or Steven before sending a v3. You are adding a kernel config and a boot argument. Both of these are sensitive topics that require more feedback from others.
Quoted 8 lines
Quoted 4 lines
A tricky situation. I think John already mentioned in another thread that any such config/boot param potentially can become a default value for some setups; at the same time, well, yeah, I see what commit message talks about.
Moving it into the debug section of kconfig hopefully sends the right message there. Plus the help text makes it clear there are performance implications and such.
No objections from me.
John Ogness
maintainerRe: [PATCH v2] printk: allow direct console printing to be enabled always
In reply to Sergey SenozhatskyQuoted 3 lines · Sergey Senozhatsky
would it make sense/be possible/etc. to look into possibility to address that "deferred kthread vs IRQ" race by moving prb entries emit to the same IRQ handler that emits chars from uart xmit buffer?
I think this is an interesting idea. We could have a function that would allow serial drivers (if they had a console on them) to flush any pending printk messages for their console.
Each driver would need to call this function for themselves. And also the driver would need to ensure that it is not holding any locks that the console-part of the driver will need. And it would need to be best effort, because the related kthread may already be in the process of printing. But in general I think it could be a nice feature to improve the sort-of-synchronous behavior of printk for drivers that use it.
John Ogness
Petr Mladek
maintainerQuoted 24 lines · Jason A. Donenfeld
Hi John & folks,
In 5.19, I'm seeing some changes in message ordering / interleaving which lead to confusion. The most obvious (and benign) example appears on system boot, in which the "Run /init as init process" message gets intermixed with the messages that init actually
writes()to stdout. For example, here's a snippet from build.wireguard.com:[ 0.469732] Freeing unused kernel image (initmem) memory: 4576K [ 0.469738] Write protecting the kernel read-only data: 10240k [ 0.473823] Freeing unused kernel image (text/rodata gap) memory: 2044K [ 0.475228] Freeing unused kernel image (rodata/data gap) memory: 1136K [ 0.475236] Run /init as init processWireGuard Test Suite on Linux 5.19.0-rc2+ x86_64 [+] Mounting filesystems... [+] Module self-tests: * allowedips self-tests: pass * nonce counter self-tests: pass * ratelimiter self-tests: pass [+] Enabling logging...
Quoted 8 lines
[+] Launching tests...
[ 0.475237] with arguments:
[ 0.475238] /init
[ 0.475238] with environment:
[ 0.475239] HOME=/
[ 0.475240] TERM=linux
[+] ip netns add wg-test-46-0
[+] ip netns add wg-test-46-1I see.
Quoted 4 lines
But the bigger issue for me is that it makes it very confusing to interpret CI results later on. Prior, I would nice a nice correlation of:
Quoted 8 lines
[+] some userspace command
[ 1.2345 ] some kernel log output
[+] some userspace command
[ 1.2346 ] some kernel log output
[+] some userspace command
[ 1.2347 ] some kernel log outputI assume this is mostly caused by your threaded printk patchset
Console has never been fully synchronous. printk() did console_trylock() and flushed the message to the console only the lock was available. The console kthreads made it asynchronous always when the kthreads are available and system is in normal state.
Quoted 2 lines
This probably has important benefits. But it certainly makes CI and related debugging a bit trickier as a result.
I could imagine.
Quoted 4 lines
So I was wondering if there was some way to boot the kernel with a command line option or compile-time flag that always flushes printk messages when they're made, or does something to make the ordering a bit more faithful.
I am pretty sure that we will have to add such an option sooner or later. We did not want to do it from the beginning because otherwise people would use it and would not tell use about their problematic use-cases ;-)
In fact, in your case you might get even better synchronization if you do it the other way and write userspace messages into the kernel log via /dev/kmsg:
echo "Hello world" > /dev/kmsg
That said. I am going to look at your patch the following week. I also want to wait for an opinion from John.
Best Regards,
Petr
Quoted 5 lines · Petr Mladek
In fact, in your case you might get even better synchronization if you do it the other way and write userspace messages into the kernel log via
/dev/kmsg:echo "Hello world" >
/dev/kmsg
The idea is to interleave stderr/stdout properly, as CI has done for ages, which threaded printk breaks, and which my patch (now posted) fixes behind a default-off option. You can't dup2 /dev/kmsg to 1 and 2, as the write() semantics are different, with respect to buffering and with respect to new lines. The interfaces aren't comparable, so this isn't quite a solution. (Besides, not breaking things as they currently exist has a certain benefit.)
Jason
From: Petr Mladek
Quoted 1 line
Sent: 17 June 2022 15:21
...
Quoted 6 lines
Quoted 1 line
I assume this is mostly caused by your threaded printk patchset
Console has never been fully synchronous.
printk()didconsole_trylock()and flushed the message to the console only the lock was available. The console kthreads made it asynchronous always when the kthreads are available and system is in normal state.
What priority do these kthreads run at?
I'd have thought they ought to run at a high priority? That should tend to give kernel messages priority over user ones.
Quite how high is another matter.
Probably a bit below the RT/FIFO:50 of threaded ISR.
Although if an x86 one ends up doing software scrolling of the vga text buffer using the ISA speed accesses that usually requires maybe not!
Maybe that is faster on modern systems - but I doubt it.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
John Ogness
maintainerQuoted 1 line · David Laight
What priority do these kthreads run at?
120 (SCHED_OTHER, nice=0)
Quoted 5 lines
I'd have thought they ought to run at a high priority? That should tend to give kernel messages priority over user ones.
Quite how high is another matter.
Probably a bit below the RT/FIFO:50 of threaded ISR.
As a default value, I recommend keeping to the SCHED_OTHER policy as a default. Perhaps a nice value of -20? There are quite a few kernel threads using that as their default:
# ps -Leo ni,command | grep ^-20 | sort -20 [acpi_thermal_pm] -20 [ata_sff] -20 [blkcg_punt_bio] -20 [cfg80211] -20 [inet_frag_wq] -20 [ipv6_addrconf] -20 [kblockd] -20 [kworker/0:0H-events_highpri] -20 [kworker/0:1H-events_highpri] -20 [md] -20 [mld] -20 [mm_percpu_wq] -20 [netns] -20 [nfsiod] -20 [rcu_gp] -20 [rcu_par_gp] -20 [rpciod] -20 [scsi_tmf_0] -20 [scsi_tmf_1] -20 [writeback] -20 [xprtiod]
John Ogness
From: John Ogness
Quoted 10 lines
Sent: 19 June 2022 09:16
Quoted 1 line · David Laight
What priority do these kthreads run at?
120 (
SCHED_OTHER, nice=0)Quoted 3 lines
I'd have thought they ought to run at a high priority? That should tend to give kernel messages priority over user ones.
Quoted 6 lines
Quoted 2 lines
Quite how high is another matter.
Probably a bit below the RT/FIFO:50 of threaded ISR.As a default value, I recommend keeping to the
SCHED_OTHERpolicy as a default. Perhaps a nice value of -20? There are quite a few kernel threads using that as their default:
That doesn't mean it is a sensible priority :-)
Running at (SCHED_OTHER, nice=0) is almost certainly worse. There is little guarantee they'll run if the system is busy and has non-default priority user threads.
I know there is the NIBY style 'my process is more important than yours' But processes that don't run for very long, or have to run in order to keep the system working properly, almost certainly need to be higher priority than the lowest RT one.
I've been fighting system thread priorities on a system that is doing a lot of real time audio over UDP (ie RTP).
The application threads processing the RTP need to run in preference to all other application threads (nothing else is that time critical). Processor affinities don't help - they can only really be used to move things away from some cpu, and I need to use the idle time. Running the RTP threads at a RT priority works reasonably well except that some system threads (like the softint ones napi uses) get blocked - causing lost packets.
Threaded napi helps - but only if I run the threads under the RT scheduler.
David
Quoted 24 lines
# ps -Leo ni,command | grep ^-20 | sort -20 [acpi_thermal_pm] -20 [ata_sff] -20 [blkcg_punt_bio] -20 [cfg80211] -20 [inet_frag_wq] -20 [ipv6_addrconf] -20 [kblockd] -20 [kworker/0:0H-events_highpri] -20 [kworker/0:1H-events_highpri] -20 [md] -20 [mld] -20 [mm_percpu_wq] -20 [netns] -20 [nfsiod] -20 [rcu_gp] -20 [rcu_par_gp] -20 [rpciod] -20 [scsi_tmf_0] -20 [scsi_tmf_1] -20 [writeback] -20 [xprtiod]
Quoted 1 line
John Ogness
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)