Skip to contentSkip to conversation

printk: Introduce loglevel bypass for pstore console

PATCH In review Tzung-Bi Shih 3 participants 7 messages Aug 31 – Sep 24

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover printk: Introduce loglevel bypass for pstore console

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.

In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.

This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.

Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.

Tzung-Bi Shih (2):
printk: Introduce CON_BYPASS_LOGLEVEL flag
pstore: Bypass loglevel suppression for pstore console

 fs/pstore/platform.c    | 4 ++++
 fs/pstore/ram.c         | 3 ++-
 include/linux/console.h | 4 ++++
 include/linux/pstore.h  | 2 ++
 kernel/printk/nbcon.c   | 7 +++++--
 kernel/printk/printk.c  | 6 ++++--
 6 files changed, 21 insertions(+), 5 deletions(-)
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#1 lore raw

PATCH 1/2 printk: Introduce CON_BYPASS_LOGLEVEL flag

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherently rely on the global console_loglevel setting. If a system is configured with a quiet serial console, the pstore console is silenced as well. Consequently, during a panic or lockup, relevant verbose logs leading up to the crash are lost.

Introduce the CON_BYPASS_LOGLEVEL flag to allow consoles to bypass log level suppression.

---
 include/linux/console.h | 4 ++++
 kernel/printk/nbcon.c   | 7 +++++--
 kernel/printk/printk.c  | 6 ++++--
 3 files changed, 13 insertions(+), 4 deletions(-)
include/linux/console.h +4 -0
index d624200cfc17..00ce21d73256 100644
@@ -180,6 +180,9 @@ static inline void con_debug_leave(void) { }
  *			constraints.
  * @CON_NBCON_ATOMIC_UNSAFE: The write_atomic() callback is not safe and is
  *			therefore only used by nbcon_atomic_flush_unsafe().
+ * @CON_BYPASS_LOGLEVEL	The console bypasses loglevel filtering. Every message
+ *			in the ringbuffer is emitted, regardless of the global
+ *			console_loglevel setting.
  */
 enum cons_flags {
 	CON_PRINTBUFFER		= BIT(0),
@@ -192,6 +195,7 @@ enum cons_flags {
 	CON_SUSPENDED		= BIT(7),
 	CON_NBCON		= BIT(8),
 	CON_NBCON_ATOMIC_UNSAFE	= BIT(9),
+	CON_BYPASS_LOGLEVEL	= BIT(10),
 };
 
 /**
kernel/printk/nbcon.c +5 -2
index 4b03b019cd5e..bb5669cbf51b 100644
@@ -981,7 +981,9 @@ static bool nbcon_emit_next_record(struct nbcon_write_context *wctxt, bool use_a
 {
 	struct nbcon_context *ctxt = &ACCESS_PRIVATE(wctxt, ctxt);
 	struct console *con = ctxt->console;
-	bool is_extended = console_srcu_read_flags(con) & CON_EXTENDED;
+	short console_flags = console_srcu_read_flags(con);
+	bool is_extended = console_flags & CON_EXTENDED;
+	bool may_suppress = !(console_flags & CON_BYPASS_LOGLEVEL);
 	struct printk_message pmsg = {
 		.pbufs = ctxt->pbufs,
 	};
@@ -1014,7 +1016,8 @@ static bool nbcon_emit_next_record(struct nbcon_write_context *wctxt, bool use_a
 	if (!nbcon_context_enter_unsafe(ctxt))
 		return false;
 
-	ctxt->backlog = printk_get_next_message(&pmsg, ctxt->seq, is_extended, true);
+	ctxt->backlog = printk_get_next_message(&pmsg, ctxt->seq, is_extended,
+						may_suppress);
 	if (!ctxt->backlog)
 		return nbcon_context_exit_unsafe(ctxt);
 
kernel/printk/printk.c +4 -2
index 3fcdf4b4e2e5..b8a7a8b69af7 100644
@@ -3125,7 +3125,9 @@ struct printk_buffers printk_shared_pbufs;
  */
 static bool console_emit_next_record(struct console *con, bool *handover, int cookie)
 {
-	bool is_extended = console_srcu_read_flags(con) & CON_EXTENDED;
+	short console_flags = console_srcu_read_flags(con);
+	bool is_extended = console_flags & CON_EXTENDED;
+	bool may_suppress = !(console_flags & CON_BYPASS_LOGLEVEL);
 	char *outbuf = &printk_shared_pbufs.outbuf[0];
 	struct printk_message pmsg = {
 		.pbufs = &printk_shared_pbufs,
@@ -3134,7 +3136,7 @@ static bool console_emit_next_record(struct console *con, bool *handover, int co
 
 	*handover = false;
 
-	if (!printk_get_next_message(&pmsg, con->seq, is_extended, true))
+	if (!printk_get_next_message(&pmsg, con->seq, is_extended, may_suppress))
 		return false;
 
 	con->dropped += pmsg.dropped;
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#2 lore raw

PATCH 2/2 pstore: Bypass loglevel suppression for pstore console

In reply to Tzung-Bi Shih

The pstore console largely serves to continuously dump kernel messages to persistent backends (e.g., ramoops). Currently, it inherits the global console_loglevel constraints. Relevant verbose logs could be omitted from a crash dump if the system uses a strict log level to prevent spam and bottlenecks.

Unlike physical consoles, writing records to RAM (ramoops) is extremely fast. Applying loglevel limits to pstore provides no system performance benefit, but sometimes costs engineers the crucial verbose debugging context leading up to a kernel panic or lockup.

Introduce PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL to allow fast pstore backends to capture full and unsuppressed logs directly regardless of the system's global loglevel setting. Opt-in for ramoops.

---
 fs/pstore/platform.c   | 4 ++++
 fs/pstore/ram.c        | 3 ++-
 include/linux/pstore.h | 2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)
fs/pstore/platform.c +4 -0
index 1d76c9d92056..88b894163b25 100644
@@ -421,6 +421,10 @@ static void pstore_register_console(void)
 	 * calls may have changed settings (specifically CON_ENABLED).
 	 */
 	pstore_console.flags = CON_PRINTBUFFER | CON_ENABLED | CON_ANYTIME;
+
+	if (psinfo->flags & PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL)
+		pstore_console.flags |= CON_BYPASS_LOGLEVEL;
+
 	register_console(&pstore_console);
 }
 
fs/pstore/ram.c +2 -1
index 2eb0d4fa2186..945dd55be97a 100644
@@ -825,7 +825,8 @@ static int ramoops_probe(struct platform_device *pdev)
 		cxt->pstore.max_reason = pdata->max_reason;
 	}
 	if (cxt->console_size)
-		cxt->pstore.flags |= PSTORE_FLAGS_CONSOLE;
+		cxt->pstore.flags |= PSTORE_FLAGS_CONSOLE |
+				     PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL;
 	if (cxt->max_ftrace_cnt)
 		cxt->pstore.flags |= PSTORE_FLAGS_FTRACE;
 	if (cxt->pmsg_size)
include/linux/pstore.h +2 -0
index fed601053c51..34b4acfa21ce 100644
@@ -206,6 +206,8 @@ struct pstore_info {
 #define PSTORE_FLAGS_CONSOLE	BIT(1)
 #define PSTORE_FLAGS_FTRACE	BIT(2)
 #define PSTORE_FLAGS_PMSG	BIT(3)
+/* Bypass loglevel for fast consoles */
+#define PSTORE_FLAGS_CONSOLE_BYPASS_LOGLEVEL BIT(4)
 
 extern int pstore_register(struct pstore_info *);
 extern void pstore_unregister(struct pstore_info *);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#3 lore raw
In reply to Tzung-Bi Shih
Quoted 13 lines · Tzung-Bi Shih

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.

In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.

This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.

Quoted 6 lines

Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.

Tzung-Bi Shih (2):
printk: Introduce CON_BYPASS_LOGLEVEL flag
pstore: Bypass loglevel suppression for pstore console

Hi everyone,

Just a gentle ping on this series. Does it make sense to you?

maintainer
#4 lore raw
In reply to Tzung-Bi Shih
Quoted 10 lines · Tzung-Bi Shih

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.

In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.

Quoted 5 lines

This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.

Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.

I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/

It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/

I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.

Best Regards,
Petr

PS: Thanks for reminder. I wanted to answer this mail the same day
when I got it. But I have to interrupted and it somehow fallen though cracks.

#5 lore raw
Quoted 21 lines · Petr Mladek
Quoted 10 lines · Tzung-Bi Shih

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.

In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.

Quoted 5 lines

This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.

Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.

I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/

I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.

Thanks for the context. I completely agree that the per-console loglevel is the proper long-term solution, and I understand that console registration cleanup and the larger printk reworks are massive and complex efforts.

Because those foundational changes naturally take time, I was wondering if we could consider my approach as a narrow and localized exception just for offline consoles like pstore.

Given that we already use LOG_FORCE_CON to bypass loglevels on a per-message basis, introducing a symmetric per-console flag like CON_BYPASS_LOGLEVEL might be an acceptable short-term solution.

Would you be open to a localized flag like this in the interim, or do you strongly prefer to hold off until the full per-console loglevel architecture lands?

maintainer
#6 lore raw
Quoted 36 lines · Tzung-Bi Shih
Quoted 20 lines
Quoted 10 lines

Currently, offline consoles (e.g., pstore_console backed by ramoops) inherit the system's global console_loglevel.

In production environments, physical serial consoles (UARTs) are often heavily rate-limited to prevent verbose messages from causing severe execution bottlenecks. Unfortunately, this identically silences the pstore console. When a kernel panic or lockup occurs, engineers are left inspecting a crash dump that has been stripped of the verbose logs that could have identified the root cause.

Quoted 5 lines

This series introduces CON_BYPASS_LOGLEVEL, a simple flag that allows specialized offline consoles to request unsuppressed logs directly.

Patch 1 introduces the CON_BYPASS_LOGLEVEL flag.
Patch 2 adopts the flag in pstore_console.

I believe that a better solution for this problem would be adding the per-console loglevel feature, see
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

It is currently blocked by a cleanup of the console registration code, see https://lore.kernel.org/all/[email protected]/

I am going to send v5 of the console registration code cleanup soon, hopefully later this week or the following week. It would be nice when it gets some more review.

Thanks for the context. I completely agree that the per-console loglevel is the proper long-term solution, and I understand that console registration cleanup and the larger printk reworks are massive and complex efforts.

Because those foundational changes naturally take time, I was wondering if we could consider my approach as a narrow and localized exception just for offline consoles like pstore.

Quoted 7 lines

Given that we already use LOG_FORCE_CON to bypass loglevels on a per-message basis, introducing a symmetric per-console flag like CON_BYPASS_LOGLEVEL might be an acceptable short-term solution.

Would you be open to a localized flag like this in the interim, or do you strongly prefer to hold off until the full per-console loglevel architecture lands?

I agree with Petr that we need to go with per-console loglevels here. However, we really need to prioritize our efforts to get that per-console loglevel series mainline (which means getting the cleanup series mainline first).

@Petr: Maybe we can find some time in Prag to sit down together and go through the rather large queue of various printk series that are waiting to be reviewed.

John