printk: Introduce loglevel bypass for pstore console
Subsystems (3)
cover printk: Introduce loglevel bypass for pstore console
Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.
In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.
This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.
Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.
Tzung-Bi Shih (2):
printk: Introduce CON_BYPASS_LOGLEVEL flag
pstore: Bypass loglevel suppression for pstore console
fs/pstore/platform.c | 4 ++++ fs/pstore/ram.c | 3 ++- include/linux/console.h | 4 ++++ include/linux/pstore.h | 2 ++ kernel/printk/nbcon.c | 7 +++++-- kernel/printk/printk.c | 6 ++++-- 6 files changed, 21 insertions(+), 5 deletions(-)
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH 1/2 printk: Introduce CON_BYPASS_LOGLEVEL flag
Currently, offline consoles (e.g., pstore_console backed by ramoops) inherently rely on the global console_loglevel setting. If a system is configured with a quiet serial console, the pstore console is silenced as well. Consequently, during a panic or lockup, relevant verbose logs leading up to the crash are lost.
Introduce the CON_BYPASS_LOGLEVEL flag to allow consoles to bypass log level suppression.
--- include/linux/console.h | 4 ++++ kernel/printk/nbcon.c | 7 +++++-- kernel/printk/printk.c | 6 ++++-- 3 files changed, 13 insertions(+), 4 deletions(-)
include/linux/console.h +4 -0
@@ -180,6 +180,9 @@ static inline void con_debug_leave(void) { }* constraints. * @CON_NBCON_ATOMIC_UNSAFE: The write_atomic() callback is not safe and is * therefore only used by nbcon_atomic_flush_unsafe(). +* @CON_BYPASS_LOGLEVEL The console bypasses loglevel filtering. Every message +* in the ringbuffer is emitted, regardless of the global +* console_loglevel setting. */ enum cons_flags {CON_PRINTBUFFER =BIT (0 ),
@@ -192,6 +195,7 @@ enum cons_flags {CON_SUSPENDED =BIT (7 ),CON_NBCON =BIT (8 ),CON_NBCON_ATOMIC_UNSAFE =BIT (9 ), + CON_BYPASS_LOGLEVEL = BIT(10), };/**
kernel/printk/nbcon.c +5 -2
@@ -981,7 +981,9 @@ static bool nbcon_emit_next_record(struct nbcon_write_context *wctxt, bool use_a {struct nbcon_context *ctxt = &ACCESS_PRIVATE (wctxt, ctxt);struct console *con = ctxt->console; -bool is_extended =console_srcu_read_flags (con) &CON_EXTENDED ; +short console_flags =console_srcu_read_flags (con); +bool is_extended = console_flags &CON_EXTENDED ; +bool may_suppress = !(console_flags &CON_BYPASS_LOGLEVEL );struct printk_message pmsg = { .pbufs = ctxt->pbufs, };
@@ -1014,7 +1016,8 @@ static bool nbcon_emit_next_record(struct nbcon_write_context *wctxt, bool use_aif (!nbcon_context_enter_unsafe (ctxt))return false; - ctxt->backlog =printk_get_next_message (&pmsg, ctxt->seq, is_extended, true); + ctxt->backlog =printk_get_next_message (&pmsg, ctxt->seq, is_extended, + may_suppress);if (!ctxt->backlog)return nbcon_context_exit_unsafe (ctxt);
kernel/printk/printk.c +4 -2
@@ -3125,7 +3125,9 @@ struct printk_buffers printk_shared_pbufs;*/ static bool console_emit_next_record (struct console *con,bool *handover,int cookie) { -bool is_extended =console_srcu_read_flags (con) &CON_EXTENDED ; +short console_flags =console_srcu_read_flags (con); +bool is_extended = console_flags &CON_EXTENDED ; +bool may_suppress = !(console_flags &CON_BYPASS_LOGLEVEL );char *outbuf = &printk_shared_pbufs.outbuf[0 ];struct printk_message pmsg = { .pbufs = &printk_shared_pbufs,
@@ -3134,7 +3136,7 @@ static bool console_emit_next_record(struct console *con, bool *handover, int co *handover = false; -if (!printk_get_next_message (&pmsg, con->seq, is_extended, true)) +if (!printk_get_next_message (&pmsg, con->seq, is_extended, may_suppress))return false; con->dropped += pmsg.dropped;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
PATCH 2/2 pstore: Bypass loglevel suppression for pstore console
In reply to Tzung-Bi ShihThe pstore console largely serves to continuously dump kernel messages to persistent backends (e.g., ramoops). Currently, it inherits the global console_loglevel constraints. Relevant verbose logs could be omitted from a crash dump if the system uses a strict log level to prevent spam and bottlenecks.
Unlike physical consoles, writing records to RAM (ramoops) is extremely fast. Applying loglevel limits to pstore provides no system performance benefit, but sometimes costs engineers the crucial verbose debugging context leading up to a kernel panic or lockup.
Introduce PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL to allow fast pstore backends to capture full and unsuppressed logs directly regardless of the system's global loglevel setting. Opt-in for ramoops.
--- fs/pstore/platform.c | 4 ++++ fs/pstore/ram.c | 3 ++- include/linux/pstore.h | 2 ++ 3 files changed, 8 insertions(+), 1 deletion(-)
fs/pstore/platform.c +4 -0
@@ -421,6 +421,10 @@ static void pstore_register_console(void)* calls may have changed settings (specifically CON_ENABLED). */ pstore_console.flags =CON_PRINTBUFFER |CON_ENABLED |CON_ANYTIME ; + +if (psinfo->flags &PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL ) + pstore_console.flags |=CON_BYPASS_LOGLEVEL ; +register_console (&pstore_console); }
fs/pstore/ram.c +2 -1
@@ -825,7 +825,8 @@ static int ramoops_probe(struct platform_device *pdev) cxt->pstore.max_reason = pdata->max_reason; }if (cxt->console_size) - cxt->pstore.flags |=PSTORE_FLAGS_CONSOLE ; + cxt->pstore.flags |=PSTORE_FLAGS_CONSOLE | +PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL ;if (cxt->max_ftrace_cnt) cxt->pstore.flags |=PSTORE_FLAGS_FTRACE ;if (cxt->pmsg_size)
include/linux/pstore.h +2 -0
@@ -206,6 +206,8 @@ struct pstore_info {#define PSTORE_FLAGS_CONSOLE BIT(1)#define PSTORE_FLAGS_FTRACE BIT(2)#define PSTORE_FLAGS_PMSG BIT(3) +/* Bypass loglevel for fast consoles */ +#define PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL BIT(4)extern int pstore_register (struct pstore_info *);extern void pstore_unregister (struct pstore_info *);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
Quoted 13 lines · Tzung-Bi Shih
Currently, offline consoles (e.g.,
pstore_consolebacked by ramoops) inherit the system's globalconsole_loglevel.In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.
This series introduces
CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.
Quoted 6 lines
Patch 1 introduces the
CON_BYPASS_LOGLEVELflag.
Patch 2 adopts the flag inpstore_console.Tzung-Bi Shih (2):
printk: IntroduceCON_BYPASS_LOGLEVELflag
pstore: Bypass loglevel suppression for pstore console
Hi everyone,
Just a gentle ping on this series. Does it make sense to you?
Petr Mladek
maintainerQuoted 10 lines · Tzung-Bi Shih
Currently, offline consoles (e.g.,
pstore_consolebacked by ramoops) inherit the system's globalconsole_loglevel.In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.
Quoted 5 lines
This series introduces
CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.Patch 1 introduces the
CON_BYPASS_LOGLEVELflag.
Patch 2 adopts the flag inpstore_console.
I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/
It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/
I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.
Best Regards,
Petr
PS: Thanks for reminder. I wanted to answer this mail the same day
when I got it. But I have to interrupted and it somehow fallen though cracks.
Quoted 21 lines · Petr Mladek
Quoted 10 lines · Tzung-Bi Shih
Currently, offline consoles (e.g.,
pstore_consolebacked by ramoops) inherit the system's globalconsole_loglevel.In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.
Quoted 5 lines
This series introduces
CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.Patch 1 introduces the
CON_BYPASS_LOGLEVELflag.
Patch 2 adopts the flag inpstore_console.I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/
Quoted 6 lines
It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/
I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.
Thanks for the context. I completely agree that the per-console loglevel is the proper long-term solution, and I understand that console registration cleanup and the larger printk reworks are massive and complex efforts.
Because those foundational changes naturally take time, I was wondering if we could consider my approach as a narrow and localized exception just for offline consoles like pstore.
Given that we already use LOG_FORCE_CON to bypass loglevels on a per-message basis, introducing a symmetric per-console flag like CON_BYPASS_LOGLEVEL might be an acceptable short-term solution.
Would you be open to a localized flag like this in the interim, or do you strongly prefer to hold off until the full per-console loglevel architecture lands?
John Ogness
maintainerQuoted 36 lines · Tzung-Bi Shih
Quoted 20 lines
Quoted 10 lines
Currently, offline consoles (e.g.,
pstore_consolebacked by ramoops) inherit the system's globalconsole_loglevel.In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.
Quoted 5 lines
This series introduces
CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.Patch 1 introduces the
CON_BYPASS_LOGLEVELflag.
Patch 2 adopts the flag inpstore_console.I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/Quoted 6 lines
It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/
I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.
Thanks for the context. I completely agree that the per-console loglevel is the proper long-term solution, and I understand that console registration cleanup and the larger printk reworks are massive and complex efforts.
Because those foundational changes naturally take time, I was wondering if we could consider my approach as a narrow and localized exception just for offline consoles like pstore.
Quoted 7 lines
Given that we already use
LOG_FORCE_CONto bypass loglevels on a per-message basis, introducing a symmetric per-console flag likeCON_BYPASS_LOGLEVELmight be an acceptable short-term solution.Would you be open to a localized flag like this in the interim, or do you strongly prefer to hold off until the full per-console loglevel architecture lands?
I agree with Petr that we need to go with per-console loglevels here. However, we really need to prioritize our efforts to get that per-console loglevel series mainline (which means getting the cleanup series mainline first).
@Petr: Maybe we can find some time in Prag to sit down together and go through the rather large queue of various printk series that are waiting to be reviewed.
John