Add support to replay kernel log on consoles via sysrq
Subsystems (3)
v6 cover Add support to replay kernel log on consoles via sysrq
Hi,
This patch series enables one to replay the kernel log messages in printk ring buffer on consoles via sysrq. This is useful to view the kernel messages when terminal is unresponsive to enter commands like dmesg and syslog services are also disabled, especially on embedded targets. Although debug features like kdb/kgdb already allow this, these debug configs should be enabled which is often not the case.
In the first commit, a generic function console_replay_all() is added that replays the kernel log messages on consoles. To do this, code under CONSOLE_REPLAY_ALL mode in console_flush_on_panic() is taken out to a helper function __console_rewind_all() to set the console sequence number to oldest record in the printk buffer. This function can be called safely under console_lock(). console_replay_all() tries to get console subsystem lock using console_trylock() and if successful, calls __console_rewind_all() to reset the sequence number. Finally it calls console_unlock() which flushes out the contents of printk buffer to consoles. The console loglevel will determine which all kernel log messages are displayed.
In the second commit, code is added to call console_replay_all() from the sysrq key handler when sysrq+R is pressed. Document is also updated describing the use case and limitations.
Limitations:
- User may have to press the key combination multiple times if console lock is not obtained at the time of key press.
- If console lock owner is stuck, console_trylock() will fail continuously and messages won't be displayed.
These limitations can be overcome once atomic consoles are available. Currently, it is best effort.
Links to previous discussion:
- https://lore.kernel.org/all/[email protected]/T/#t
- https://lore.kernel.org/all/[email protected]/T/#t
- https://lore.kernel.org/all/[email protected]/T/#t
- https://lore.kernel.org/linux-serial/[email protected]/
Changelog:
V5 -> V6:
- added reference to sysrq[0-9] to change loglevel
- fixed action_msg by using Replay instead of replay
V4 -> V5:
- renamed console_rewind_all() to __console_rewind_all()
- modified comments for __console_rewind_all()
- renamed dump_printk_buffer() to console_replay_all()
- used sysrq+R instead of sysrq+D
- removed workqueue based implementation and used console_trylock()
V3 -> V4:
- refactor code in console_flush_on_panic() under CONSOLE_REPLAY_ALL mode
- add new function console_rewind_all()
- use console_rewind_all() instead of ksmg_dump*() in dump_printk_buffer()
V2 -> V3:
- split the implementation into two commits
- added function in printk.c to dump printk buffer to consoles
- added Suggested-by tag
- removed code to dump printk buffer from sysrq.c and called new function
V1 -> V2:
- modified kernel ring buffer to printk ring buffer
- allocated buf dynamically to prevent stack frame size warnings
- used buf of size 2048 to match PRINTK_MESSAGE_MAX and added comment
-- Sreenath
Sreenath Vijayan (2):
printk: Add function to replay kernel log on consoles tty/sysrq: Replay kernel log messages on consoles via sysrq
Documentation/admin-guide/sysrq.rst | 9 ++++ drivers/tty/sysrq.c | 13 ++++- include/linux/printk.h | 4 ++ kernel/printk/printk.c | 77 ++++++++++++++++++++--------- 4 files changed, 78 insertions(+), 25 deletions(-)
Signature · 1 line
2.25.1
PATCH v6 1/2 printk: Add function to replay kernel log on consoles
Add a generic function console_replay_all() for replaying the kernel log on consoles, in any context. It would allow viewing the logs on an unresponsive terminal via sysrq.
Reuse the existing code from console_flush_on_panic() for resetting the sequence numbers, by introducing a new helper function __console_rewind_all(). It is safe to be called under console_lock().
Try to acquire lock on the console subsystem without waiting. If successful, reset the sequence number to oldest available record on all consoles and call console_unlock() which will automatically flush the messages to the consoles.
--- include/linux/printk.h | 4 +++ kernel/printk/printk.c | 77 +++++++++++++++++++++++++++++------------- 2 files changed, 57 insertions(+), 24 deletions(-)
include/linux/printk.h +4 -0
@@ -192,6 +192,7 @@ void show_regs_print_info(const char *log_lvl); extern asmlinkage void dump_stack_lvl(const char *log_lvl) __cold; extern asmlinkage void dump_stack(void) __cold;void printk_trigger_flush (void ); +void console_replay_all (void );#else static inline __printf(1, 0) int vprintk(const char *s, va_list args)
@@ -271,6 +272,9 @@ static inline void dump_stack(void)static inline void printk_trigger_flush (void ) { } +static inline void console_replay_all (void ) +{ +}#endif #ifdef CONFIG_SMP
kernel/printk/printk.c +53 -24
95 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v6 1/2] printk: Add function to replay kernel log on consoles
Quoted 17 lines · Sreenath Vijayan
Add a generic function
console_replay_all()for replaying the kernel log on consoles, in any context. It would allow viewing the logs on an unresponsive terminal via sysrq.Reuse the existing code from
console_flush_on_panic()for resetting the sequence numbers, by introducing a new helper function__console_rewind_all(). It is safe to be called underconsole_lock().Try to acquire lock on the console subsystem without waiting. If successful, reset the sequence number to oldest available record on all consoles and call
console_unlock()which will automatically flush the messages to the consoles.
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4275
Quoted 5 lines
kernel/printk/printk.c +2
-0@@ -4259,6 +4271,23 @@ void kmsg_dump_rewind(struct kmsg_dump_iter *iter) }EXPORT_SYMBOL_GPL (kmsg_dump_rewind); +/** +* console_replay_all - replay kernel log on consoles
It should rather be called "console_try_replay_all()" to make it clear that it is just the best effort.
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4281
Quoted 6 lines
kernel/printk/printk.c +6
-0+* +* Try to obtain lock on console subsystem and replay all +* available records in printk buffer on the consoles. +* Does nothing if lock is not obtained. +* +* Context: Any context.
This should be:
* Context: Any, except for NMI
Basically only lockless code is safe in NMI which is not the case here.
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4291
Quoted 10 lines
kernel/printk/printk.c +9
-0+*/ +void console_replay_all (void ) +{ +if (console_trylock ()) { +__console_rewind_all (); +/* Consoles are flushed as part of console_unlock(). */ +console_unlock (); + } +}#endif
Otherwise, it looks good. With the two changes:
I am sorry for the late review. I have been snowed under tasks. Also I had healthy problems.
I have seen a mail that Greg has queued the patch in tty-next. I am not sure if it still can be fixed. It will be perfectly fine to change this by a followup patch.
Best Regards,
Petr
Re: [PATCH v6 1/2] printk: Add function to replay kernel log on consoles
Quoted 22 lines · Petr Mladek
Quoted 20 lines · Sreenath Vijayan
Add a generic function
console_replay_all()for replaying the kernel log on consoles, in any context. It would allow viewing the logs on an unresponsive terminal via sysrq.Reuse the existing code from
console_flush_on_panic()for resetting the sequence numbers, by introducing a new helper function__console_rewind_all(). It is safe to be called underconsole_lock().Try to acquire lock on the console subsystem without waiting. If successful, reset the sequence number to oldest available record on all consoles and call
console_unlock()which will automatically flush the messages to the consoles.kernel/printk/printk.c +0
-0@@ -4259,6 +4271,23 @@ void kmsg_dump_rewind(struct kmsg_dump_iter *iter) }EXPORT_SYMBOL_GPL (kmsg_dump_rewind);
Quoted 6 lines
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4275
Quoted 2 lines
kernel/printk/printk.c +2
-0+/** +* console_replay_all - replay kernel log on consoles It should rather be called "
console_try_replay_all()" to make it clear that it is just the best effort.
Ok, noted.
Quoted 7 lines
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4281
Quoted 6 lines
kernel/printk/printk.c +6
-0+* +* Try to obtain lock on console subsystem and replay all +* available records in printk buffer on the consoles. +* Does nothing if lock is not obtained. +* +* Context: Any context.
Quoted 6 lines
This should be:
* Context: Any, except for NMI
Basically only lockless code is safe in NMI which is not the case here.
Understood.
Quoted 18 lines
kernel/printk/printk.c void kmsg_dump_rewind(struct kmsg_dump_iter *iter) · line 4291
Quoted 10 lines
kernel/printk/printk.c +9
-0+*/ +void console_replay_all (void ) +{ +if (console_trylock ()) { +__console_rewind_all (); +/* Consoles are flushed as part of console_unlock(). */ +console_unlock (); + } +}#endif Otherwise, it looks good. With the two changes:
Reviewed-by Petr Mladek <[email protected]>I am sorry for the late review. I have been snowed under tasks. Also I had healthy problems.
Quoted 6 lines
I have seen a mail that Greg has queued the patch in tty-next. I am not sure if it still can be fixed. It will be perfectly fine to change this by a followup patch.
Best Regards,
Petr
Thank you for the review comments. As the patch has been merged to mainline, I will soon send a followup patch with "Fixes:" tag.
Regards,
Sreenath
PATCH v6 2/2 tty/sysrq: Replay kernel log messages on consoles via sysrq
In reply to Sreenath VijayanWhen terminal is unresponsive, one cannot use dmesg to view the printk ring buffer messages. Also, syslog services may be disabled, especially on embedded systems, to check the messages after a reboot. In this scenario, replay the messages in printk ring buffer on consoles via sysrq by pressing sysrq+R.
The console loglevel will determine which all kernel log messages are displayed. The messages will be displayed only when console_trylock() succeeds. Users could repeat the sysrq key when it fails. If the owner of console subsystem lock is stuck, repeating the key won't work.
--- Documentation/admin-guide/sysrq.rst | 9 +++++++++ drivers/tty/sysrq.c | 13 ++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysrq.rst +9 -0
@@ -161,6 +161,8 @@ Command Function will be printed to your console. (``0``, for example would make it so that only emergency messages like PANICs or OOPSes would make it to your console.) + +``R`` Replay the kernel log messages on consoles. =========== =================================================================== Okay, so what can I use them for?
@@ -211,6 +213,13 @@ processes. "just thaw ``it(j)``" is useful if your system becomes unresponsive due to a frozen (probably root) filesystem via the FIFREEZE ioctl. +``Replay logs(R)`` is useful to view the kernel log messages when system is hung +or you are not able to use dmesg command to view the messages in printk buffer. +User may have to press the key combination multiple times if console system is +busy. If it is completely locked up, then messages won't be printed. Output +messages depend on current console loglevel, which can be modified using +sysrq[0-9] (see above). + Sometimes SysRq seems to get 'stuck' after using it, what can I do? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/tty/sysrq.c +12 -1
@@ -450,6 +450,17 @@ static const struct sysrq_key_op sysrq_unrt_op = { .enable_mask = SYSRQ_ENABLE_RTNICE, }; +static void sysrq_handle_replay_logs (u8 key) +{ +console_replay_all (); +} +static struct sysrq_key_op sysrq_replay_logs_op = { + .handler = sysrq_handle_replay_logs, + .help_msg ="replay-kernel-logs(R)" , + .action_msg ="Replay kernel logs on consoles" , + .enable_mask =SYSRQ_ENABLE_DUMP , +}; +/* Key Operations table and lock */ static DEFINE_SPINLOCK (sysrq_key_table_lock);
@@ -519,7 +530,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {NULL ,/* O */ NULL ,/* P */ NULL ,/* Q */ -NULL ,/* R */ + &sysrq_replay_logs_op,/* R */ NULL ,/* S */ NULL ,/* T */ NULL, /* U */
Signature · 1 line
2.25.1
Re: [PATCH v6 2/2] tty/sysrq: Replay kernel log messages on consoles via sysrq
Quoted 82 lines · Sreenath Vijayan
When terminal is unresponsive, one cannot use dmesg to view the printk ring buffer messages. Also, syslog services may be disabled, especially on embedded systems, to check the messages after a reboot. In this scenario, replay the messages in printk ring buffer on consoles via sysrq by pressing sysrq+R.
The console loglevel will determine which all kernel log messages are displayed. The messages will be displayed only when
console_trylock()succeeds. Users could repeat the sysrq key when it fails. If the owner of console subsystem lock is stuck, repeating the key won't work.Suggested-by Petr Mladek <[email protected]>Signed-off-by Shimoyashiki Taichi <[email protected]>Reviewed-by John Ogness <[email protected]>Signed-off-by Sreenath Vijayan <[email protected]>--- Documentation/admin-guide/sysrq.rst | 9 +++++++++ drivers/tty/sysrq.c | 13 ++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-)Documentation/admin-guide/sysrq.rst +9
-0@@ -161,6 +161,8 @@ Command Function will be printed to your console. (``0``, for example would make it so that only emergency messages like PANICs or OOPSes would make it to your console.) + +``R`` Replay the kernel log messages on consoles. =========== =================================================================== Okay, so what can I use them for?@@ -211,6 +213,13 @@ processes. "just thaw ``it(j)``" is useful if your system becomes unresponsive due to a frozen (probably root) filesystem via the FIFREEZE ioctl. +``Replay logs(R)`` is useful to view the kernel log messages when system is hung +or you are not able to use dmesg command to view the messages in printk buffer. +User may have to press the key combination multiple times if console system is +busy. If it is completely locked up, then messages won't be printed. Output +messages depend on current console loglevel, which can be modified using +sysrq[0-9] (see above). + Sometimes SysRq seems to get 'stuck' after using it, what can I do? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~drivers/tty/sysrq.c +12
-1@@ -450,6 +450,17 @@ static const struct sysrq_key_op sysrq_unrt_op = { .enable_mask = SYSRQ_ENABLE_RTNICE, }; +static void sysrq_handle_replay_logs (u8 key) +{ +console_replay_all (); +} +static struct sysrq_key_op sysrq_replay_logs_op = { + .handler = sysrq_handle_replay_logs, + .help_msg ="replay-kernel-logs(R)" , + .action_msg ="Replay kernel logs on consoles" , + .enable_mask =SYSRQ_ENABLE_DUMP , +}; +/* Key Operations table and lock */ static DEFINE_SPINLOCK (sysrq_key_table_lock);@@ -519,7 +530,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {NULL ,/* O */ NULL ,/* P */ NULL ,/* Q */ -NULL ,/* R */ + &sysrq_replay_logs_op,/* R */ NULL ,/* S */ NULL ,/* T */ NULL, /* U */Signature · 1 line
2.25.1
Hi,
Kindly let me know if there are any review comments for the above changes.
Regards,
Sreenath
Re: [PATCH v6 2/2] tty/sysrq: Replay kernel log messages on consoles via sysrq
In reply to Sreenath VijayanQuoted 12 lines · Sreenath Vijayan
When terminal is unresponsive, one cannot use dmesg to view the printk ring buffer messages. Also, syslog services may be disabled, especially on embedded systems, to check the messages after a reboot. In this scenario, replay the messages in printk ring buffer on consoles via sysrq by pressing sysrq+R.
The console loglevel will determine which all kernel log messages are displayed. The messages will be displayed only when
console_trylock()succeeds. Users could repeat the sysrq key when it fails. If the owner of console subsystem lock is stuck, repeating the key won't work.
Quoted 4 lines
Suggested-by Petr Mladek <[email protected]>Signed-off-by Shimoyashiki Taichi <[email protected]>Reviewed-by John Ogness <[email protected]>Signed-off-by Sreenath Vijayan <[email protected]>
Best Regards,
Petr