Add support to replay kernel log on console via sysrq

Hi,

This patch series enables one to replay the kernel log messages in printk ring buffer on consoles via sysrq. This is useful to view the kernel messages when terminal is unresponsive to enter commands like dmesg and syslog services are also disabled, especially on embedded targets. Although debug features like kdb/kgdb already allow this, these debug configs should be enabled which is often not the case.

In the first commit, a generic function console_replay_all() is added that replays the kernel log messages on consoles. To do this, code under CONSOLE_REPLAY_ALL mode in console_flush_on_panic() is taken out to a helper function __console_rewind_all() to set the console sequence number to oldest record in the printk buffer. This function can be called safely under console_lock(). console_replay_all() tries to get console subsystem lock using console_trylock() and if successful, calls __console_rewind_all() to reset the sequence number. Finally it calls console_unlock() which flushes out the contents of printk buffer to consoles. The console loglevel will determine which all kernel log messages are displayed.

In the second commit, code is added to call console_replay_all() from the sysrq key handler when sysrq+R is pressed. Document is also updated describing the use case and limitations.

Limitations:
- User may have to press the key combination multiple times if console lock is not obtained at the time of key press.
- If console lock owner is stuck, console_trylock() will fail continuously and messages won't be displayed.

These limitations can be overcome once atomic consoles are available. Currently, it is best effort.

Links to previous discussion:
- https://lore.kernel.org/all/[email protected]/T/#t
- https://lore.kernel.org/all/[email protected]/T/#t
- https://lore.kernel.org/linux-serial/[email protected]/

Changelog:
V4 -> V5:
- renamed console_rewind_all() to __console_rewind_all()
- modified comments for __console_rewind_all()
- renamed dump_printk_buffer() to console_replay_all()
- used sysrq+R instead of sysrq+D
- removed workqueue based implementation and used console_trylock()

V3 -> V4:
- refactor code in console_flush_on_panic() under CONSOLE_REPLAY_ALL mode
- add new function console_rewind_all()
- use console_rewind_all() instead of ksmg_dump*() in dump_printk_buffer()

V2 -> V3:
- split the implementation into two commits
- added function in printk.c to dump printk buffer to consoles
- added Suggested-by tag
- removed code to dump printk buffer from sysrq.c and called new function

V1 -> V2:
- modified kernel ring buffer to printk ring buffer
- allocated buf dynamically to prevent stack frame size warnings
- used buf of size 2048 to match PRINTK_MESSAGE_MAX and added comment

-- Sreenath

Sreenath Vijayan (2):
printk: Add function to replay kernel log on consoles tty/sysrq: Replay kernel log messages on consoles via sysrq

 Documentation/admin-guide/sysrq.rst |  8 +++
 drivers/tty/sysrq.c                 | 13 ++++-
 include/linux/printk.h              |  4 ++
 kernel/printk/printk.c              | 77 ++++++++++++++++++++---------
 4 files changed, 77 insertions(+), 25 deletions(-)

2.25.1

Add a generic function console_replay_all() for replaying the kernel log on consoles, in any context. It would allow viewing the logs on an unresponsive terminal via sysrq.

Reuse the existing code from console_flush_on_panic() for resetting the sequence numbers, by introducing a new helper function __console_rewind_all(). It is safe to be called under console_lock().

Try to acquire lock on the console subsystem without waiting. If successful, reset the sequence number to oldest available record on all consoles and call console_unlock() which will automatically flush the messages to the consoles.

---
 include/linux/printk.h |  4 +++
 kernel/printk/printk.c | 77 +++++++++++++++++++++++++++++-------------
 2 files changed, 57 insertions(+), 24 deletions(-)

index 8ef499ab3c1e..8df2ac6c0e1f 100644
@@ -192,6 +192,7 @@ void show_regs_print_info(const char *log_lvl);
 extern asmlinkage void dump_stack_lvl(const char *log_lvl) __cold;
 extern asmlinkage void dump_stack(void) __cold;
 void printk_trigger_flush(void);
+void console_replay_all(void);
 #else
 static inline __printf(1, 0)
 int vprintk(const char *s, va_list args)

@@ -271,6 +272,9 @@ static inline void dump_stack(void)
 static inline void printk_trigger_flush(void)
 {
 }
+static inline void console_replay_all(void)
+{
+}
 #endif
 
 #ifdef CONFIG_SMP

2.25.1

When terminal is unresponsive, one cannot use dmesg to view the printk ring buffer messages. Also, syslog services may be disabled, especially on embedded systems, to check the messages after a reboot. In this scenario, replay the messages in printk ring buffer on consoles via sysrq by pressing sysrq+R.

The console loglevel will determine which all kernel log messages are displayed. The messages will be displayed only when console_trylock() succeeds. Users could repeat the sysrq key when it fails. If the owner of console subsystem lock is stuck, repeating the key won't work.

---
 Documentation/admin-guide/sysrq.rst |  8 ++++++++
 drivers/tty/sysrq.c                 | 13 ++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

index 2f2e5bd440f9..8098a1b4908d 100644
@@ -161,6 +161,8 @@ Command	    Function
             will be printed to your console. (``0``, for example would make
             it so that only emergency messages like PANICs or OOPSes would
             make it to your console.)
+
+``R``	    Replay the kernel log messages on consoles.
 =========== ===================================================================
 
 Okay, so what can I use them for?

@@ -211,6 +213,12 @@ processes.
 "just thaw ``it(j)``" is useful if your system becomes unresponsive due to a
 frozen (probably root) filesystem via the FIFREEZE ioctl.
 
+``Replay logs(R)`` is useful to view the kernel log messages when system is hung
+or you are not able to use dmesg command to view the messages in printk buffer.
+User may have to press the key combination multiple times if console system is
+busy. If it is completely locked up, then messages won't be printed. Output
+messages depend on current console loglevel.
+
 Sometimes SysRq seems to get 'stuck' after using it, what can I do?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

index 02217e3c916b..c4deea41a8b5 100644
@@ -450,6 +450,17 @@ static const struct sysrq_key_op sysrq_unrt_op = {
 	.enable_mask	= SYSRQ_ENABLE_RTNICE,
 };
 
+static void sysrq_handle_replay_logs(u8 key)
+{
+	console_replay_all();
+}
+static struct sysrq_key_op sysrq_replay_logs_op = {
+	.handler        = sysrq_handle_replay_logs,
+	.help_msg       = "replay-kernel-logs(R)",
+	.action_msg     = "replay kernel logs on consoles",
+	.enable_mask    = SYSRQ_ENABLE_DUMP,
+};
+
 /* Key Operations table and lock */
 static DEFINE_SPINLOCK(sysrq_key_table_lock);
 

@@ -519,7 +530,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {
 	NULL,				/* O */
 	NULL,				/* P */
 	NULL,				/* Q */
-	NULL,				/* R */
+	&sysrq_replay_logs_op,		/* R */
 	NULL,				/* S */
 	NULL,				/* T */
 	NULL,				/* U */

2.25.1