implement threaded console printing
Subsystems (9)
John Ogness
maintainerv4 cover implement threaded console printing
This is v4 of a series to implement a kthread for each registered console. v3 is here [0]. The kthreads locklessly retrieve the records from the printk ringbuffer and also do not cause any lock contention between each other. This allows consoles to run at full speed. For example, a netconsole is able to dump records much faster than a serial or vt console. Also, during normal operation, printk() callers are completely decoupled from console printing.
There are situations where kthread printing is not sufficient. For example, during panic situations, where the kthreads may not get a chance to schedule. In such cases, the current method of attempting to print directly within the printk() caller context is used. New functions printk_prefer_direct_enter() and printk_prefer_direct_exit() are made available to mark areas of the kernel where direct printing is preferred. (These should only be areas that do not occur during normal operation.)
This series also introduces pr_flush(): a might_sleep() function that will block until all active printing threads have caught up to the latest record at the time of the pr_flush() call. This function is useful, for example, to wait until pending records are flushed to consoles before suspending.
Note that this series does *not* increase the reliability of console printing. Rather it focuses on the non-interference aspect of printk() by decoupling printk() callers from printing (during normal operation). Nonetheless, the reliability aspect should not worsen due to this series.
John Ogness
[0] https://lore.kernel.org/lkml/[email protected]
Changes since v3:
- For defer_console_output(), call allow_direct_printing() instead of only checking @printk_prefer_direct.
- Remove console_lock_single_hold() and console_unlock_single_release() functions. Use the console_lock for console_stop(), console_start(), unregister_console(), printk_kthread_func().
- Introduce macros console_flags_set() and console_flags_clear() to adjust con->flags using READ_ONCE()/WRITE_ONCE() in order to guarantee consistent values for the variable. (This does not make the RMW operations atomic, but the console_lock and con->lock are still used to synchronize between tasks that modify con->flags.)
- Add and/or expand comments for allow_direct_printing(), console_cpu_notify(), __console_unlock(), console_stop(), console_start(), unregister_console(), printk_kthread_func(), defer_console_output().
John Ogness (15):
printk: rename cpulock functions
printk: cpu sync always disable interrupts
printk: add missing memory barrier to wake_up_klogd()
printk: wake up all waiters
printk: wake waiters for safe and NMI contexts
printk: get caller_id/timestamp after migration disable
printk: call boot_delay_msec() in printk_delay()
printk: add con_printk() macro for console details
printk: refactor and rework printing logic
printk: move buffer definitions into console_emit_next_record() caller
printk: add pr_flush()
printk: add functions to prefer direct printing
printk: add kthread console printers
printk: extend console_lock for proper kthread support
printk: remove @console_locked
drivers/tty/sysrq.c | 2 + include/linux/console.h | 19 + include/linux/printk.h | 82 ++- kernel/hung_task.c | 11 +- kernel/panic.c | 4 + kernel/printk/printk.c | 1234 +++++++++++++++++++++++++++++---------- kernel/rcu/tree_stall.h | 2 + kernel/reboot.c | 14 +- kernel/watchdog.c | 4 + kernel/watchdog_hld.c | 4 + lib/dump_stack.c | 4 +- lib/nmi_backtrace.c | 4 +- 12 files changed, 1059 insertions(+), 325 deletions(-)
base-commit: 84d7df104dbab9c3dda8f2c5b46f9a6fc256fe02
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 1/15 printk: rename cpulock functions
Since the printk cpulock is CPU-reentrant and since it is used in all contexts, its usage must be carefully considered and most likely will require programming locklessly. To avoid mistaking the printk cpulock as a typical lock, rename it to cpu_sync. The main functions then become:
printk_cpu_sync_get_irqsave(flags);
printk_cpu_sync_put_irqrestore(flags);Add extra notes of caution in the function description to help developers understand the requirements for correct usage.
--- include/linux/printk.h | 54 +++++++++++++++++++------------- kernel/printk/printk.c | 71 +++++++++++++++++++++--------------------- lib/dump_stack.c | 4 +-- lib/nmi_backtrace.c | 4 +-- 4 files changed, 73 insertions(+), 60 deletions(-)
include/linux/printk.h +33 -21
@@ -277,43 +277,55 @@ static inline void printk_trigger_flush(void)#endif #ifdef CONFIG_SMP -extern int __printk_cpu_trylock (void ); -extern void __printk_wait_on_cpu_lock (void ); -extern void __printk_cpu_unlock (void ); +extern int __printk_cpu_sync_try_get (void ); +extern void __printk_cpu_sync_wait (void ); +extern void __printk_cpu_sync_put (void );/** -* printk_cpu_lock_irqsave() - Acquire the printk cpu-reentrant spinning -* lock and disable interrupts. +* printk_cpu_sync_get_irqsave() - Acquire the printk cpu-reentrant spinning +* lock and disable interrupts. * @flags: Stack-allocated storage for saving local interrupt state, -* to be passed to printk_cpu_unlock_irqrestore(). +* to be passed to printk_cpu_sync_put_irqrestore(). * * If the lock is owned by another CPU, spin until it becomes available. * Interrupts are restored while spinning. +* +* CAUTION: This function must be used carefully. It does not behave like a +* typical lock. Here are important things to watch out for... +* +* * This function is reentrant on the same CPU. Therefore the calling +* code must not assume exclusive access to data if code accessing the +* data can run reentrant or within NMI context on the same CPU. +* +* * If there exists usage of this function from NMI context, it becomes +* unsafe to perform any type of locking or spinning to wait for other +* CPUs after calling this function from any context. This includes +* using spinlocks or any other busy-waiting synchronization methods. */ -#define printk_cpu_lock_irqsave (flags) \ - for (;;) { \ - local_irq_save(flags); \ - if (__printk_cpu_trylock()) \ - break; \ - local_irq_restore(flags); \ - __printk_wait_on_cpu_lock(); \ +#define printk_cpu_sync_get_irqsave (flags) \ + for (;;) { \ + local_irq_save(flags); \ + if (__printk_cpu_sync_try_get()) \ + break; \ + local_irq_restore(flags); \ + __printk_cpu_sync_wait(); \ }/** -* printk_cpu_unlock_irqrestore() - Release the printk cpu-reentrant spinning -* lock and restore interrupts. -* @flags: Caller's saved interrupt state, from printk_cpu_lock_irqsave(). +* printk_cpu_sync_put_irqrestore() - Release the printk cpu-reentrant spinning +* lock and restore interrupts. +* @flags: Caller's saved interrupt state, from printk_cpu_sync_get_irqsave(). */ -#define printk_cpu_unlock_irqrestore (flags) \ +#define printk_cpu_sync_put_irqrestore (flags) \ do { \ - __printk_cpu_unlock(); \ + __printk_cpu_sync_put(); \ local_irq_restore(flags); \ - } while (0) \ + } while (0)#else -#define printk_cpu_lock_irqsave (flags) ((void)flags) -#define printk_cpu_unlock_irqrestore (flags) ((void)flags) +#define printk_cpu_sync_get_irqsave (flags) ((void)flags) +#define printk_cpu_sync_put_irqrestore (flags) ((void)flags)#endif /* CONFIG_SMP */
kernel/printk/printk.c +36 -35
148 lines not on this page · raw message
lib/dump_stack.c +2 -2
@@ -102,9 +102,9 @@ asmlinkage __visible void dump_stack_lvl(const char *log_lvl)* Permit this cpu to perform nested stack dumps while serialising * against other CPUs */ -printk_cpu_lock_irqsave (flags); +printk_cpu_sync_get_irqsave (flags);__dump_stack (log_lvl); -printk_cpu_unlock_irqrestore (flags); +printk_cpu_sync_put_irqrestore (flags); }EXPORT_SYMBOL (dump_stack_lvl);
lib/nmi_backtrace.c +2 -2
@@ -99,7 +99,7 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)* Allow nested NMI backtraces while serializing * against other CPUs. */ -printk_cpu_lock_irqsave (flags); +printk_cpu_sync_get_irqsave (flags);if (!READ_ONCE (backtrace_idle) && regs &&cpu_in_idle (instruction_pointer (regs))) {pr_warn ("NMI backtrace for cpu %d skipped: idling at %pS\n" , cpu, (void *)instruction_pointer (regs));
@@ -110,7 +110,7 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)else dump_stack (); } -printk_cpu_unlock_irqrestore (flags); +printk_cpu_sync_put_irqrestore (flags);cpumask_clear_cpu (cpu,to_cpumask (backtrace_mask));return true; }
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 2/15 printk: cpu sync always disable interrupts
In reply to John OgnessThe CPU sync functions are a NOP for !CONFIG_SMP. But for !CONFIG_SMP they still need to disable interrupts in order to preserve context within the CPU sync sections.
--- include/linux/printk.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
include/linux/printk.h +9 -9
@@ -281,9 +281,16 @@ extern int __printk_cpu_sync_try_get(void);extern void __printk_cpu_sync_wait (void );extern void __printk_cpu_sync_put (void ); +#else + +#define __printk_cpu_sync_try_get () true +#define __printk_cpu_sync_wait () +#define __printk_cpu_sync_put () +#endif /* CONFIG_SMP */ +/** -* printk_cpu_sync_get_irqsave() - Acquire the printk cpu-reentrant spinning -* lock and disable interrupts. +* printk_cpu_sync_get_irqsave() - Disable interrupts and acquire the printk +* cpu-reentrant spinning lock. * @flags: Stack-allocated storage for saving local interrupt state, * to be passed to printk_cpu_sync_put_irqrestore(). *
@@ -322,13 +329,6 @@ extern void __printk_cpu_sync_put(void);local_irq_restore (flags); \ }while (0 ) -#else - -#define printk_cpu_sync_get_irqsave (flags) ((void)flags) -#define printk_cpu_sync_put_irqrestore (flags) ((void)flags) - -#endif /* CONFIG_SMP */ - extern int kptr_restrict;/**
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 3/15 printk: add missing memory barrier to wake_up_klogd()
In reply to John OgnessIt is important that any new records are visible to preparing waiters before the waker checks if the wait queue is empty. Otherwise it is possible that:
- there are new records available
- the waker sees an empty wait queue and does not wake
- the preparing waiter sees no new records and begins to wait
This is exactly the problem that the function description of waitqueue_active() warns about.
Use wq_has_sleeper() instead of waitqueue_active() because it includes the necessary full memory barrier.
--- kernel/printk/printk.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-)
kernel/printk/printk.c +36 -3
@@ -746,8 +746,19 @@ static ssize_t devkmsg_read(struct file *file, char __user *buf, goto out; } +/* +* Guarantee this task is visible on the waitqueue before +* checking the wake condition. +* +* The full memory barrier within set_current_state() of +* prepare_to_wait_event() pairs with the full memory barrier +* within wq_has_sleeper(). +* +* This pairs with wake_up_klogd:A. +*/ ret =wait_event_interruptible (log_wait, -prb_read_valid (prb,atomic64_read (&user->seq), r)); +prb_read_valid (prb, +atomic64_read (&user->seq), r));/* LMM(devkmsg_read:A) */ if (ret) goto out; }
@@ -1513,7 +1524,18 @@ static int syslog_print(char __user *buf, int size) seq = syslog_seq;mutex_unlock (&syslog_lock); - len =wait_event_interruptible (log_wait,prb_read_valid (prb, seq,NULL )); +/* +* Guarantee this task is visible on the waitqueue before +* checking the wake condition. +* +* The full memory barrier within set_current_state() of +* prepare_to_wait_event() pairs with the full memory barrier +* within wq_has_sleeper(). +* +* This pairs with wake_up_klogd:A. +*/ + len =wait_event_interruptible (log_wait, +prb_read_valid (prb, seq,NULL ));/* LMM(syslog_print:A) */ mutex_lock (&syslog_lock);if (len)
@@ -3316,7 +3338,18 @@ void wake_up_klogd(void)return ;preempt_disable (); -if (waitqueue_active (&log_wait)) { +/* +* Guarantee any new records can be seen by tasks preparing to wait +* before this context checks if the wait queue is empty. +* +* The full memory barrier within wq_has_sleeper() pairs with the full +* memory barrier within set_current_state() of +* prepare_to_wait_event(), which is called after ___wait_event() adds +* the waiter but before it has checked the wait condition. +* +* This pairs with devkmsg_read:A and syslog_print:A. +*/ +if (wq_has_sleeper (&log_wait)) {/* LMM(wake_up_klogd:A) */ this_cpu_or (printk_pending,PRINTK_PENDING_WAKEUP );irq_work_queue (this_cpu_ptr (&wake_up_klogd_work)); }
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 4/15 printk: wake up all waiters
In reply to John OgnessThere can be multiple tasks waiting for new records. They should all be woken. Use wake_up_interruptible_all() instead of wake_up_interruptible().
--- kernel/printk/printk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk/printk.c +1 -1
@@ -3326,7 +3326,7 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work) }if (pending &PRINTK_PENDING_WAKEUP ) -wake_up_interruptible (&log_wait); +wake_up_interruptible_all (&log_wait); } static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) =
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 5/15 printk: wake waiters for safe and NMI contexts
In reply to John OgnessWhen printk() is called from safe or NMI contexts, it will directly store the record (vprintk_store()) and then defer the console output. However, defer_console_output() only causes console printing and does not wake any waiters of new records.
Wake waiters from defer_console_output() so that they also are aware of the new records from safe and NMI contexts.
--- kernel/printk/printk.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/printk/printk.c +16 -12
@@ -754,7 +754,7 @@ static ssize_t devkmsg_read(struct file *file, char __user *buf,* prepare_to_wait_event() pairs with the full memory barrier * within wq_has_sleeper(). * -* This pairs with wake_up_klogd:A. +* This pairs with __wake_up_klogd:A. */ ret = wait_event_interruptible(log_wait, prb_read_valid(prb,
@@ -1532,7 +1532,7 @@ static int syslog_print(char __user *buf, int size)* prepare_to_wait_event() pairs with the full memory barrier * within wq_has_sleeper(). * -* This pairs with wake_up_klogd:A. +* This pairs with __wake_up_klogd:A. */ len =wait_event_interruptible (log_wait,prb_read_valid (prb, seq,NULL ));/* LMM(syslog_print:A) */
@@ -3332,7 +3332,7 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work) static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) =IRQ_WORK_INIT_LAZY (wake_up_klogd_work_func); -void wake_up_klogd (void ) +static void __wake_up_klogd (int val) {if (!printk_percpu_data_ready ())return ;
@@ -3349,22 +3349,26 @@ void wake_up_klogd(void)* * This pairs with devkmsg_read:A and syslog_print:A. */ -if (wq_has_sleeper (&log_wait)) {/* LMM(wake_up_klogd:A) */ -this_cpu_or (printk_pending,PRINTK_PENDING_WAKEUP ); +if (wq_has_sleeper (&log_wait) ||/* LMM(__wake_up_klogd:A) */ + (val &PRINTK_PENDING_OUTPUT )) { +this_cpu_or (printk_pending, val);irq_work_queue (this_cpu_ptr (&wake_up_klogd_work)); }preempt_enable (); } -void defer_console_output (void ) +void wake_up_klogd (void ) { -if (!printk_percpu_data_ready ()) -return ; +__wake_up_klogd (PRINTK_PENDING_WAKEUP ); +} -preempt_disable (); -this_cpu_or (printk_pending,PRINTK_PENDING_OUTPUT ); -irq_work_queue (this_cpu_ptr (&wake_up_klogd_work)); -preempt_enable (); +void defer_console_output (void ) +{ +/* +* New messages may have been added directly to the ringbuffer +* using vprintk_store(), so wake any waiters as well. +*/ +__wake_up_klogd (PRINTK_PENDING_WAKEUP |PRINTK_PENDING_OUTPUT ); }void printk_trigger_flush (void )
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 6/15 printk: get caller_id/timestamp after migration disable
In reply to John OgnessCurrently the local CPU timestamp and caller_id for the record are collected while migration is enabled. Since this information is CPU-specific, it should be collected with migration disabled.
Migration is disabled immediately after collecting this information anyway, so just move the information collection to after the migration disabling.
--- kernel/printk/printk.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/printk/printk.c +6 -4
@@ -2063,7 +2063,7 @@ static inline void printk_delay(void)static inline u32 printk_caller_id (void ) {return in_task () ?task_pid_nr (current) : -0x80000000 +raw_smp_processor_id (); +0x80000000 +smp_processor_id (); }/**
@@ -2145,7 +2145,6 @@ int vprintk_store(int facility, int level,const struct dev_printk_info *dev_info, const char *fmt, va_list args) { - const u32 caller_id = printk_caller_id(); struct prb_reserved_entry e;enum printk_info_flags flags =0 ;struct printk_record r;
@@ -2155,10 +2154,14 @@ int vprintk_store(int facility, int level,u8 *recursion_ptr;u16 reserve_size;va_list args2; +u32 caller_id;u16 text_len;int ret =0 ;u64 ts_nsec; +if (!printk_enter_irqsave (recursion_ptr, irqflags)) +return 0 ; +/* * Since the duration of printk() can vary depending on the message * and state of the ringbuffer, grab the timestamp now so that it is
@@ -2167,8 +2170,7 @@ int vprintk_store(int facility, int level,*/ ts_nsec =local_clock (); -if (!printk_enter_irqsave (recursion_ptr, irqflags)) -return 0 ; + caller_id =printk_caller_id ();/* * The sprintf needs to come first since the syslog prefix might be
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 7/15 printk: call boot_delay_msec() in printk_delay()
In reply to John Ognessboot_delay_msec() is always called immediately before printk_delay() so just call it from within printk_delay().
--- kernel/printk/printk.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/printk/printk.c +4 -3
@@ -2048,8 +2048,10 @@ static u8 *__printk_recursion_counter(void)int printk_delay_msec__read_mostly ; -static inline void printk_delay (void ) +static inline void printk_delay (int level) { +boot_delay_msec (level); +if (unlikely (printk_delay_msec)) {int m = printk_delay_msec;
@@ -2274,8 +2276,7 @@ asmlinkage int vprintk_emit(int facility, int level, in_sched = true; } -boot_delay_msec (level); -printk_delay (); +printk_delay (level); printed_len =vprintk_store (facility, level, dev_info, fmt, args);
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 8/15 printk: add con_printk() macro for console details
In reply to John OgnessIt is useful to generate log messages that include details about the related console. Rather than duplicate the code to assemble the details, put that code into a macro con_printk().
Once console printers become threaded, this macro will find more users.
--- kernel/printk/printk.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
kernel/printk/printk.c +7 -6
@@ -3015,6 +3015,11 @@ static void try_enable_default_console(struct console *newcon) newcon->flags |=CON_CONSDEV ; } +#define con_printk (lvl, con, fmt, ...) \ + printk(lvl pr_fmt("%sconsole [%s%d] " fmt), \ + (con->flags & CON_BOOT) ? "boot" : "", \ + con->name, con->index, ##__VA_ARGS__) +/* * The console driver calls this routine during kernel initialization * to register the console printing procedure with printk() and to
@@ -3153,9 +3158,7 @@ void register_console(struct console *newcon)* users know there might be something in the kernel's log buffer that * went to the bootconsole (that they do not see on the real console) */ -pr_info ("%sconsole [%s%d] enabled\n" , - (newcon->flags &CON_BOOT ) ?"boot" :"" , - newcon->name, newcon->index); +con_printk (KERN_INFO , newcon,"enabled\n" );if (bootcon_enabled && ((newcon->flags & (CON_CONSDEV |CON_BOOT )) ==CON_CONSDEV ) && !keep_bootcon) {
@@ -3174,9 +3177,7 @@ int unregister_console(struct console *console)struct console *con;int res; -pr_info ("%sconsole [%s%d] disabled\n" , - (console->flags &CON_BOOT ) ?"boot" :"" , - console->name, console->index); +con_printk (KERN_INFO , console,"disabled\n" ); res =_braille_unregister_console (console);if (res <0 )
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 9/15 printk: refactor and rework printing logic
In reply to John OgnessRefactor/rework printing logic in order to prepare for moving to threaded console printing.
- Move @console_seq into struct console so that the current "position" of each console can be tracked individually.
- Move @console_dropped into struct console so that the current drop count of each console can be tracked individually.
- Modify printing logic so that each console independently loads, prepares, and prints its next record.
- Remove exclusive_console logic. Since console positions are handled independently, replaying past records occurs naturally.
- Update the comments explaining why preemption is disabled while printing from printk() context.
With these changes, there is a change in behavior: the console replaying the log (formerly exclusive console) will no longer block other consoles. New messages appear on the other consoles while the newly added console is still replaying.
--- include/linux/console.h | 2 + kernel/printk/printk.c | 441 +++++++++++++++++++++------------------- 2 files changed, 230 insertions(+), 213 deletions(-)
include/linux/console.h +2 -0
@@ -151,6 +151,8 @@ struct console {int cflag;uint ispeed;uint ospeed; +u64 seq; +unsigned long dropped;void *data;struct console *next; };
kernel/printk/printk.c +228 -213
556 lines not on this page · raw message
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 10/15 printk: move buffer definitions into console_emit_next_record() caller
In reply to John OgnessExtended consoles print extended messages and do not print messages about dropped records.
Non-extended consoles print "normal" messages as well as extra messages about dropped records.
Currently the buffers for these various message types are defined within the functions that might use them and their usage is based upon the CON_EXTENDED flag. This will be a problem when moving to kthread printers because each printer must be able to provide its own buffers.
Move all the message buffer definitions outside of console_emit_next_record(). The caller knows if extended or dropped messages should be printed and can specify the appropriate buffers to use. The console_emit_next_record() and call_console_driver() functions can know what to print based on whether specified buffers are non-NULL.
With this change, buffer definition/allocation/specification is separated from the code that does the various types of string printing.
--- kernel/printk/printk.c | 60 ++++++++++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 17 deletions(-)
kernel/printk/printk.c +43 -17
@@ -394,6 +394,9 @@ static struct latched_seq clear_seq = {/* the maximum size of a formatted record (i.e. with prefix added per line) */ #define CONSOLE_LOG_MAX 1024 +/* the maximum size for a dropped text message */ +#define DROPPED_TEXT_MAX 64 +/* the maximum size allowed to be reserved for a record */ #define LOG_LINE_MAX (CONSOLE_LOG_MAX - PREFIX_MAX)
@@ -1923,18 +1926,18 @@ static int console_trylock_spinning(void)/* * Call the specified console driver, asking it to write out the specified -* text and length. For non-extended consoles, if any records have been +* text and length. If @dropped_text is non-NULL and any records have been * dropped, a dropped message will be written out first. */ -static void call_console_driver (struct console *con,const char *text,size_t len) +static void call_console_driver (struct console *con,const char *text,size_t len, +char *dropped_text) { -static char dropped_text[64 ];size_t dropped_len;trace_console_rcuidle (text, len); -if (con->dropped && !(con->flags &CON_EXTENDED )) { - dropped_len =snprintf (dropped_text,sizeof (dropped_text), +if (con->dropped && dropped_text) { + dropped_len =snprintf (dropped_text,DROPPED_TEXT_MAX ,"** %lu printk messages dropped **\n" , con->dropped); con->dropped =0 ;
@@ -2296,6 +2299,7 @@ EXPORT_SYMBOL(_printk);#else /* CONFIG_PRINTK */ #define CONSOLE_LOG_MAX 0 +#define DROPPED_TEXT_MAX 0#define printk_time false#define prb_read_valid (rb, seq, r) false
@@ -2319,7 +2323,10 @@ static ssize_t msg_print_ext_body(char *buf, size_t size, struct dev_printk_info *dev_info) { return 0; }static void console_lock_spinning_enable (void ) { }static int console_lock_spinning_disable_and_check (void ) {return 0 ; } -static void call_console_driver (struct console *con,const char *text,size_t len) { } +static void call_console_driver (struct console *con,const char *text,size_t len, +char *dropped_text) +{ +}static bool suppress_message_printing (int level) {return false; }#endif /* CONFIG_PRINTK */
@@ -2644,6 +2651,14 @@ static void __console_unlock(void)* Print one record for the given console. The record printed is whatever * record is the next available record for the given console. * +* @text is a buffer of size CONSOLE_LOG_MAX. +* +* If extended messages should be printed, @ext_text is a buffer of size +* CONSOLE_EXT_LOG_MAX. Otherwise @ext_text must be NULL. +* +* If dropped messages should be printed, @dropped_text is a buffer of size +* DROPPED_TEXT_MAX. Otherwise @dropped_text must be NULL. +* * @handover will be set to true if a printk waiter has taken over the * console_lock, in which case the caller is no longer holding the * console_lock. Otherwise it is set to false.
@@ -2653,10 +2668,9 @@ static void __console_unlock(void)* * Requires the console_lock. */ -static bool console_emit_next_record (struct console *con,bool *handover) +static bool console_emit_next_record (struct console *con,char *text,char *ext_text, +char *dropped_text,bool *handover) { -static char ext_text[CONSOLE_EXT_LOG_MAX ]; -static char text[CONSOLE_LOG_MAX ];static int panic_console_dropped;struct printk_info info;struct printk_record r;
@@ -2664,7 +2678,7 @@ static bool console_emit_next_record(struct console *con, bool *handover)char *write_text;size_t len; -prb_rec_init_rd (&r, &info, text,sizeof (text)); +prb_rec_init_rd (&r, &info, text,CONSOLE_LOG_MAX ); *handover = false;
@@ -2686,13 +2700,13 @@ static bool console_emit_next_record(struct console *con, bool *handover) goto skip; } -if (con->flags &CON_EXTENDED ) { - write_text = &ext_text[0 ]; - len =info_print_ext_header (ext_text,sizeof (ext_text), r.info); - len +=msg_print_ext_body (ext_text + len,sizeof (ext_text) - len, +if (ext_text) { + write_text = ext_text; + len =info_print_ext_header (ext_text,CONSOLE_EXT_LOG_MAX , r.info); + len +=msg_print_ext_body (ext_text + len,CONSOLE_EXT_LOG_MAX - len, &r.text_buf[0 ], r.info->text_len, &r.info->dev_info); }else { - write_text = &text[0 ]; + write_text = text; len =record_print_text (&r, console_msg_format &MSG_FORMAT_SYSLOG , printk_time); }
@@ -2710,7 +2724,7 @@ static bool console_emit_next_record(struct console *con, bool *handover)console_lock_spinning_enable ();stop_critical_timings ();/* don't trace print latency */ -call_console_driver (con, write_text, len); +call_console_driver (con, write_text, len, dropped_text);start_critical_timings (); con->seq++;
@@ -2746,6 +2760,9 @@ static bool console_emit_next_record(struct console *con, bool *handover)*/ static bool console_flush_all (bool do_cond_resched,u64 *next_seq,bool *handover) { +static char dropped_text[DROPPED_TEXT_MAX ]; +static char ext_text[CONSOLE_EXT_LOG_MAX ]; +static char text[CONSOLE_LOG_MAX ];bool any_usable = false;struct console *con;bool any_progress;
@@ -2763,7 +2780,16 @@ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handovecontinue ; any_usable = true; - progress =console_emit_next_record (con, handover); +if (con->flags &CON_EXTENDED ) { +/* Extended consoles do not print "dropped messages". */ + progress =console_emit_next_record (con, &text[0 ], + &ext_text[0 ],NULL , + handover); + }else { + progress =console_emit_next_record (con, &text[0 ], +NULL , &dropped_text[0 ], + handover); + }if (*handover)return false;
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 11/15 printk: add pr_flush()
In reply to John OgnessProvide a might-sleep function to allow waiting for console printers to catch up to the latest logged message.
Use pr_flush() whenever it is desirable to get buffered messages printed before continuing: suspend_console(), resume_console(), console_stop(), console_start(), console_unblank().
--- include/linux/printk.h | 7 ++++ kernel/printk/printk.c | 83 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+)
include/linux/printk.h +7 -0
@@ -170,6 +170,8 @@ extern void __printk_safe_exit(void);#define printk_deferred_enter __printk_safe_enter#define printk_deferred_exit __printk_safe_exit +extern bool pr_flush (int timeout_ms,bool reset_on_progress); +/* * Please don't use printk_ratelimit(), because it shares ratelimiting state * with all other unrelated printk_ratelimit() callsites. Instead use
@@ -220,6 +222,11 @@ static inline void printk_deferred_exit(void) { } +static inline bool pr_flush (int timeout_ms,bool reset_on_progress) +{ +return true; +} +static inline int printk_ratelimit (void ) {return 0 ;
kernel/printk/printk.c +83 -0
131 lines not on this page · raw message
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 12/15 printk: add functions to prefer direct printing
In reply to John OgnessOnce kthread printing is available, console printing will no longer occur in the context of the printk caller. However, there are some special contexts where it is desirable for the printk caller to directly print out kernel messages. Using pr_flush() to wait for threaded printers is only possible if the caller is in a sleepable context and the kthreads are active. That is not always the case.
Introduce printk_prefer_direct_enter() and printk_prefer_direct_exit() functions to explicitly (and globally) activate/deactivate preferred direct console printing. The term "direct console printing" refers to printing to all enabled consoles from the context of the printk caller. The term "prefer" is used because this type of printing is only best effort. If the console is currently locked or other printers are already actively printing, the printk caller will need to rely on the other contexts to handle the printing.
This preferred direct printing is how all printing has been handled until now (unless it was explicitly deferred).
When kthread printing is introduced, there may be some unanticipated problems due to kthreads being unable to flush important messages. In order to minimize such risks, preferred direct printing is activated for the primary important messages when the system experiences general types of major errors. These are:
- emergency reboot/shutdown
- cpu and rcu stalls
- hard and soft lockups
- hung tasks
- warn
- sysrq
Note that since kthread printing does not yet exist, no behavior changes result from this commit. This is only implementing the counter and marking the various places where preferred direct printing is active.
--- drivers/tty/sysrq.c | 2 ++ include/linux/printk.h | 11 +++++++++++ kernel/hung_task.c | 11 ++++++++++- kernel/panic.c | 4 ++++ kernel/printk/printk.c | 28 ++++++++++++++++++++++++++++ kernel/rcu/tree_stall.h | 2 ++ kernel/reboot.c | 14 +++++++++++++- kernel/watchdog.c | 4 ++++ kernel/watchdog_hld.c | 4 ++++ 9 files changed, 78 insertions(+), 2 deletions(-)
drivers/tty/sysrq.c +2 -0
@@ -578,6 +578,7 @@ void __handle_sysrq(int key, bool check_mask)rcu_sysrq_start ();rcu_read_lock (); +printk_prefer_direct_enter ();/* * Raise the apparent loglevel to maximum so that the sysrq header * is shown to provide the user with positive feedback. We do not
@@ -619,6 +620,7 @@ void __handle_sysrq(int key, bool check_mask)pr_cont ("\n" ); console_loglevel = orig_log_level; } +printk_prefer_direct_exit ();rcu_read_unlock ();rcu_sysrq_end ();
include/linux/printk.h +11 -0
@@ -170,6 +170,9 @@ extern void __printk_safe_exit(void);#define printk_deferred_enter __printk_safe_enter#define printk_deferred_exit __printk_safe_exit +extern void printk_prefer_direct_enter (void ); +extern void printk_prefer_direct_exit (void ); +extern bool pr_flush (int timeout_ms,bool reset_on_progress);/*
@@ -222,6 +225,14 @@ static inline void printk_deferred_exit(void) { } +static inline void printk_prefer_direct_enter (void ) +{ +} + +static inline void printk_prefer_direct_exit (void ) +{ +} +static inline bool pr_flush (int timeout_ms,bool reset_on_progress) {return true;
kernel/hung_task.c +10 -1
@@ -127,6 +127,8 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)* complain: */ if (sysctl_hung_task_warnings) { +printk_prefer_direct_enter (); +if (sysctl_hung_task_warnings >0 ) sysctl_hung_task_warnings--; pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
@@ -142,6 +144,8 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)if (sysctl_hung_task_all_cpu_backtrace) hung_task_show_all_bt = true; + +printk_prefer_direct_exit (); }touch_nmi_watchdog ();
@@ -204,12 +208,17 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) } unlock:rcu_read_unlock (); -if (hung_task_show_lock) +if (hung_task_show_lock) { +printk_prefer_direct_enter ();debug_show_all_locks (); +printk_prefer_direct_exit (); + }if (hung_task_show_all_bt) { hung_task_show_all_bt = false; +printk_prefer_direct_enter ();trigger_all_cpu_backtrace (); +printk_prefer_direct_exit (); }if (hung_task_call_panic)
kernel/panic.c +4 -0
@@ -560,6 +560,8 @@ void __warn(const char *file, int line, void *caller, unsigned taint, {disable_trace_on_warning (); +printk_prefer_direct_enter (); +if (file) pr_warn("WARNING: CPU: %d PID: %d at %s:%d %pS\n", raw_smp_processor_id(), current->pid, file, line,
@@ -597,6 +599,8 @@ void __warn(const char *file, int line, void *caller, unsigned taint,/* Just a warning, don't kill lockdep. */ add_taint (taint,LOCKDEP_STILL_OK ); + +printk_prefer_direct_exit (); }#ifndef __WARN_FLAGS
kernel/printk/printk.c +28 -0
@@ -362,6 +362,34 @@ static int console_msg_format = MSG_FORMAT_DEFAULT;static DEFINE_MUTEX (syslog_lock);#ifdef CONFIG_PRINTK +static atomic_t printk_prefer_direct =ATOMIC_INIT (0 ); + +/** +* printk_prefer_direct_enter - cause printk() calls to attempt direct +* printing to all enabled consoles +* +* Since it is not possible to call into the console printing code from any +* context, there is no guarantee that direct printing will occur. +* +* This globally effects all printk() callers. +* +* Context: Any context. +*/ +void printk_prefer_direct_enter (void ) +{ +atomic_inc (&printk_prefer_direct); +} + +/** +* printk_prefer_direct_exit - restore printk() behavior +* +* Context: Any context. +*/ +void printk_prefer_direct_exit (void ) +{ +WARN_ON (atomic_dec_if_positive (&printk_prefer_direct) <0 ); +} +DECLARE_WAIT_QUEUE_HEAD (log_wait);/* All 3 protected by @syslog_lock. */ /* the next printk record to read by syslog(READ) or /proc/kmsg */
kernel/rcu/tree_stall.h +2 -0
@@ -619,6 +619,7 @@ static void print_cpu_stall(unsigned long gps)* See Documentation/RCU/stallwarn.rst for info on how to debug * RCU CPU stall warnings. */ +printk_prefer_direct_enter ();trace_rcu_stall_warning (rcu_state.name,TPS ("SelfDetected" ));pr_err ("INFO: %s self-detected stall on CPU\n" , rcu_state.name);raw_spin_lock_irqsave_rcu_node (rdp->mynode, flags);
@@ -656,6 +657,7 @@ static void print_cpu_stall(unsigned long gps)*/ set_tsk_need_resched (current);set_preempt_need_resched (); +printk_prefer_direct_exit (); }static void check_cpu_stall (struct rcu_data *rdp)
kernel/reboot.c +13 -1
@@ -447,9 +447,11 @@ static int __orderly_reboot(void) ret =run_cmd (reboot_cmd);if (ret) { +printk_prefer_direct_enter ();pr_warn ("Failed to start orderly reboot: forcing the issue\n" );emergency_sync ();kernel_restart (NULL ); +printk_prefer_direct_exit (); }return ret;
@@ -462,6 +464,7 @@ static int __orderly_poweroff(bool force) ret =run_cmd (poweroff_cmd);if (ret && force) { +printk_prefer_direct_enter ();pr_warn ("Failed to start orderly shutdown: forcing the issue\n" );/*
@@ -471,6 +474,7 @@ static int __orderly_poweroff(bool force)*/ emergency_sync ();kernel_power_off (); +printk_prefer_direct_exit (); }return ret;
@@ -528,6 +532,8 @@ EXPORT_SYMBOL_GPL(orderly_reboot);*/ static void hw_failure_emergency_poweroff_func (struct work_struct *work) { +printk_prefer_direct_enter (); +/* * We have reached here after the emergency shutdown waiting period has * expired. This means orderly_poweroff has not been able to shut off
@@ -544,6 +550,8 @@ static void hw_failure_emergency_poweroff_func(struct work_struct *work)*/ pr_emerg ("Hardware protection shutdown failed. Trying emergency restart\n" );emergency_restart (); + +printk_prefer_direct_exit (); } static DECLARE_DELAYED_WORK(hw_failure_emergency_poweroff_work,
@@ -582,11 +590,13 @@ void hw_protection_shutdown(const char *reason, int ms_until_forced) {static atomic_t allow_proceed =ATOMIC_INIT (1 ); +printk_prefer_direct_enter (); +pr_emerg ("HARDWARE PROTECTION shutdown (%s)\n" , reason);/* Shutdown should be initiated only once. */ if (!atomic_dec_and_test (&allow_proceed)) -return ; + goto out;/* * Queue a backup emergency shutdown in the event of
@@ -594,6 +604,8 @@ void hw_protection_shutdown(const char *reason, int ms_until_forced)*/ hw_failure_emergency_poweroff (ms_until_forced);orderly_poweroff (true); +out: +printk_prefer_direct_exit (); }EXPORT_SYMBOL_GPL (hw_protection_shutdown);
kernel/watchdog.c +4 -0
@@ -424,6 +424,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)/* Start period for the next softlockup warning. */ update_report_ts (); +printk_prefer_direct_enter (); +pr_emerg ("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n" ,smp_processor_id (), duration, current->comm,task_pid_nr (current));
@@ -442,6 +444,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK );if (softlockup_panic)panic ("softlockup: hung tasks" ); + +printk_prefer_direct_exit (); }return HRTIMER_RESTART ;
kernel/watchdog_hld.c +4 -0
@@ -135,6 +135,8 @@ static void watchdog_overflow_callback(struct perf_event *event,if (__this_cpu_read (hard_watchdog_warn) == true)return ; +printk_prefer_direct_enter (); +pr_emerg ("Watchdog detected hard LOCKUP on cpu %d\n" , this_cpu);print_modules ();
@@ -155,6 +157,8 @@ static void watchdog_overflow_callback(struct perf_event *event,if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +printk_prefer_direct_exit (); +__this_cpu_write (hard_watchdog_warn, true);return ; }
Signature · 1 line
2.30.2
John Ogness
maintainerPATCH v4 13/15 printk: add kthread console printers
In reply to John OgnessCreate a kthread for each console to perform console printing. During normal operation (@system_state == SYSTEM_RUNNING), the kthread printers are responsible for all printing on their respective consoles.
During non-normal operation, console printing is done as it has been:
within the context of the printk caller or within irqwork triggered by the printk caller, referred to as direct printing.
Since threaded console printers are responsible for all printing during normal operation, this also includes messages generated via deferred printk calls. If direct printing is in effect during a deferred printk call, the queued irqwork will perform the direct printing. To make it clear that this is the only time that the irqwork will perform direct printing, rename the flag PRINTK_PENDING_OUTPUT to PRINTK_PENDING_DIRECT_OUTPUT.
Threaded console printers synchronize against each other and against console lockers by taking the console lock for each message that is printed.
Note that the kthread printers do not care about direct printing. They will always try to print if new records are available. They can be blocked by direct printing, but will be woken again once direct printing is finished.
Console unregistration is a bit tricky because the associated kthread printer cannot be stopped while the console lock is held. A policy is implemented that states: whichever task clears con->thread (under the console lock) is responsible for stopping the kthread. unregister_console() will clear con->thread while the console lock is held and then stop the kthread after releasing the console lock.
For consoles that have implemented the exit() callback, the kthread is stopped before exit() is called.
--- include/linux/console.h | 2 + kernel/printk/printk.c | 329 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 309 insertions(+), 22 deletions(-)
include/linux/console.h +2 -0
@@ -153,6 +153,8 @@ struct console {uint ospeed;u64 seq;unsigned long dropped; +struct task_struct *thread; +void *data;struct console *next; };
kernel/printk/printk.c +307 -22
472 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH printk v4 13/15] printk: add kthread console printers
Quoted 34 lines · John Ogness
Create a kthread for each console to perform console printing. During normal operation (@system_state ==
SYSTEM_RUNNING), the kthread printers are responsible for all printing on their respective consoles.During non-normal operation, console printing is done as it has been:
within the context of the printk caller or within irqwork triggered by the printk caller, referred to as direct printing.Since threaded console printers are responsible for all printing during normal operation, this also includes messages generated via deferred printk calls. If direct printing is in effect during a deferred printk call, the queued irqwork will perform the direct printing. To make it clear that this is the only time that the irqwork will perform direct printing, rename the flag
PRINTK_PENDING_OUTPUTtoPRINTK_PENDING_DIRECT_OUTPUT.Threaded console printers synchronize against each other and against console lockers by taking the console lock for each message that is printed.
Note that the kthread printers do not care about direct printing. They will always try to print if new records are available. They can be blocked by direct printing, but will be woken again once direct printing is finished.
Console unregistration is a bit tricky because the associated kthread printer cannot be stopped while the console lock is held. A policy is implemented that states: whichever task clears
con->thread(under the console lock) is responsible for stopping the kthread.unregister_console()will clearcon->threadwhile the console lock is held and then stop the kthread after releasing the console lock.
Quoted 4 lines
For consoles that have implemented the
exit()callback, the kthread is stopped beforeexit()is called.Signed-off-by John Ogness <[email protected]>
Looks good to me.
Best Regards,
Petr
John Ogness
maintainerPATCH v4 14/15 printk: extend console_lock for proper kthread support
In reply to John OgnessCurrently threaded console printers synchronize against each other using console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.
But the threaded consoles printers do still need to synchronize against console_lock() callers. Introduce a per-console mutex and a new console flag CON_THD_BLOCKED to provide this synchronization.
console_lock() is modified so that it must acquire the mutex of each console in order to set the CON_THD_BLOCKED flag. Console printing threads will acquire their mutex while printing a record. If CON_THD_BLOCKED was set, the thread will go back to sleep instead of printing.
The reason for the CON_THD_BLOCKED flag is so that console_lock() callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking.
Threaded console printers also need to synchronize against console_trylock() callers. Since console_trylock() may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock() cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task via console_trylock().
Note that @console_sem is still used to provide synchronization between console_lock() and console_trylock() callers.
A locking overview for console_lock(), console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock()
{
down(&console_sem);
for_each_console(con) {
mutex_lock(&con->lock);
con->flags |= CON_THD_BLOCKED;
mutex_unlock(&con->lock);
}
/* console_lock acquired */
}console_trylock()
{
if (down_trylock(&console_sem) == 0) {
if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) {
/* console_lock acquired */
}
}
}threaded_printer()
{
mutex_lock(&con->lock);
if (!(con->flags & CON_THD_BLOCKED)) {
/* console_lock() callers blocked */
if (atomic_inc_unless_negative(&console_kthreads_active)) {
/* console_trylock() callers blocked */
con->write();
atomic_dec(&console_lock_count);
}
}
mutex_unlock(&con->lock);
}The console owner and waiter logic now only applies between contexts that have taken the console_lock via console_trylock(). Threaded printers never take the console_lock, so they do not have a console_lock to handover. Tasks that have used console_lock() will block the threaded printers using a mutex and if the console_lock is handed over to an atomic context, it would be unable to unblock the threaded printers. However, the console_trylock() case is really the only scenario that is interesting for handovers anyway.
@panic_console_dropped must change to atomic_t since it is no longer protected exclusively by the console_lock.
Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in __console_unlock(). This means wake_up_klogd() calls following a console_unlock() are no longer necessary and are removed.
Also note that threaded printers no longer need to check @console_suspended. The check for the CON_THD_BLOCKED flag implicitly covers the suspended console case.
--- include/linux/console.h | 15 ++ kernel/printk/printk.c | 296 +++++++++++++++++++++++++++++++--------- 2 files changed, 248 insertions(+), 63 deletions(-)
include/linux/console.h +15 -0
@@ -16,6 +16,7 @@#include <linux/atomic.h> #include <linux/types.h> +#include <linux/mutex.h> struct vc_data ;struct console_font_op ;
@@ -136,6 +137,7 @@ static inline int con_debug_leave(void)#define CON_ANYTIME (16)/* Safe to call when cpu is offline */ #define CON_BRL (32)/* Used for a braille device */ #define CON_EXTENDED (64)/* Use the extended output format a la /dev/kmsg */ +#define CON_THD_BLOCKED (128)/* Thread blocked because console is locked */ struct console {char name[16 ];
@@ -155,6 +157,19 @@ struct console {unsigned long dropped;struct task_struct *thread; +/* +* The per-console lock is used by printing kthreads to synchronize +* this console with callers of console_lock(). This is necessary in +* order to allow printing kthreads to run in parallel to each other, +* while each safely accessing their own @flags and synchronizing +* against direct printing via console_lock/console_unlock. +* +* Note: For synchronizing against direct printing via +* console_trylock/console_unlock, see the static global +* variable @console_kthreads_active. +*/ +struct mutex lock; +void *data;struct console *next; };
kernel/printk/printk.c +233 -63
485 lines not on this page · raw message
Signature · 1 line
2.30.2
John Ogness
maintainerRe: [PATCH printk v4 14/15] printk: extend console_lock for proper kthread support
Hi Petr,
If v4 ends up being acceptable for linux-next, I would request you fold a couple cosmetic changes into this patch.
Quoted 6 lines · John Ogness
+/*
+ * Since the kthread printers do not acquire theconsole_lockbut do need to
+ * access @flags, they could experience races because other tasks
+ * (synchronizing using theconsole_lock) can modify @flags. These macros are
+ * available to at least provide atomic variable updates so that the kthread
+ * printers can see consistent values.
This last sentence is bad. It should not use the words "atomic" and "updates". Please change it to:
These macros are available to store the new value in a way that will provide consistent load values for kthread printers. Tasks using these macros must still do so under the console_lock.
[...]
Quoted 3 lines
EXPORT_SYMBOL(console_stop);+
Please remove this accidental blank line.
Quoted 5 lines
void
console_start(struct console *console)
{kernel/printk/printk.c +0
-2console_lock (); - console->flags |=CON_ENABLED ; -console_unlock ();
kernel/printk/printk.c void console_stop(struct console *console) · line 3228
Quoted 10 lines
kernel/printk/printk.c +3
-2-/* Wake the newly enabled kthread printer. */ -wake_up_klogd (); +/* Can cause races for printk_kthread_func(). */ +console_flags_set (console->flags,CON_ENABLED ); +console_unlock ();__pr_flush (console,1000 , true); }EXPORT_SYMBOL (console_start);
John
Re: [PATCH printk v4 14/15] printk: extend console_lock for proper kthread support
In reply to John OgnessQuoted 84 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console flagCON_THD_BLOCKEDto provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set theCON_THD_BLOCKEDflag. Console printing threads will acquire their mutex while printing a record. IfCON_THD_BLOCKEDwas set, the thread will go back to sleep instead of printing.The reason for the
CON_THD_BLOCKEDflag is so thatconsole_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->flags |= CON_THD_BLOCKED; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!(con->flags & CON_THD_BLOCKED)) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.
Quoted 2 lines
@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.
I have finally understood why console_lock_single_hold() solved the problem with con->flags. We should describe it in the commit message as well. Something like:
@con->flags must be updated WRITE_ONCE() under console_lock when the related kthread is running. The kthread printers read the flags only under con->mutex. They have to see the CON_THD_BLOCKED flag when the value might not be consistent.
Sigh, I agree that this approach is error prone and kind of ugly. The approach with console_lock_single_hold() was not ideal either. I still have to think about it.
Anyway, the approach with READ_ONCE()/WRITE_ONCE() looks good enough for now.
Quoted 10 lines
Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the
CON_THD_BLOCKEDflag implicitly covers the suspended console case.Signed-off-by John Ogness <[email protected]>
I do not see any further problem with this patch. So, with the updated commit message and comment above the macros:
Best Regards,
Petr
John Ogness
maintainerPATCH v5 1/1 printk: extend console_lock for per-console locking
In reply to John OgnessCurrently threaded console printers synchronize against each other using console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.
But the threaded consoles printers do still need to synchronize against console_lock() callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock() is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.
The reason for the @blocked boolean field is so that console_lock() callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.
Threaded console printers also need to synchronize against console_trylock() callers. Since console_trylock() may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock() cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task via console_trylock().
Note that @console_sem is still used to provide synchronization between console_lock() and console_trylock() callers.
A locking overview for console_lock(), console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock()
{
down(&console_sem);
for_each_console(con) {
mutex_lock(&con->lock);
con->blocked = true;
mutex_unlock(&con->lock);
}
/* console_lock acquired */
}console_trylock()
{
if (down_trylock(&console_sem) == 0) {
if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) {
/* console_lock acquired */
}
}
}threaded_printer()
{
mutex_lock(&con->lock);
if (!con->blocked) {
/* console_lock() callers blocked */
if (atomic_inc_unless_negative(&console_kthreads_active)) {
/* console_trylock() callers blocked */
con->write();
atomic_dec(&console_lock_count);
}
}
mutex_unlock(&con->lock);
}The console owner and waiter logic now only applies between contexts that have taken the console_lock via console_trylock(). Threaded printers never take the console_lock, so they do not have a console_lock to handover. Tasks that have used console_lock() will block the threaded printers using a mutex and if the console_lock is handed over to an atomic context, it would be unable to unblock the threaded printers. However, the console_trylock() case is really the only scenario that is interesting for handovers anyway.
@panic_console_dropped must change to atomic_t since it is no longer protected exclusively by the console_lock.
Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in __console_unlock(). This means wake_up_klogd() calls following a console_unlock() are no longer necessary and are removed.
Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
---
Changes since v4 of this patch:
- Use new @blocked field instead of CON_THD_BLOCKED flag.
- Remove console_flags_set()/console_flags_clear() macros for updating @flags (and remove their race comments).
- For printer_should_wake() and printk_kthread_func(), check @blocked before checking @flags.
- Update commit message and comments appropriately.
include/linux/console.h | 15 +++ kernel/printk/printk.c | 261 +++++++++++++++++++++++++++++++--------- 2 files changed, 220 insertions(+), 56 deletions(-)
include/linux/console.h +15 -0
@@ -16,6 +16,7 @@#include <linux/atomic.h> #include <linux/types.h> +#include <linux/mutex.h> struct vc_data ;struct console_font_op ;
@@ -154,6 +155,20 @@ struct console {u64 seq;unsigned long dropped;struct task_struct *thread; +bool blocked; + +/* +* The per-console lock is used by printing kthreads to synchronize +* this console with callers of console_lock(). This is necessary in +* order to allow printing kthreads to run in parallel to each other, +* while each safely accessing the @blocked field and synchronizing +* against direct printing via console_lock/console_unlock. +* +* Note: For synchronizing against direct printing via +* console_trylock/console_unlock, see the static global +* variable @console_kthreads_active. +*/ +struct mutex lock;void *data;struct console *next;
kernel/printk/printk.c +205 -56
417 lines not on this page · raw message
base-commit: 09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7
Signature · 1 line
2.30.2
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 95 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->blocked = true; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!con->blocked) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.
Quoted 5 lines
Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Signed-off-by John Ogness <[email protected]>
Nice, it it better than v4. I am going to push this for linux-next.
See below a comment about the possible future direction.
Quoted 6 lines
---
Changes since v4 of this patch:
- Use new @blocked field instead of
CON_THD_BLOCKEDflag.
Quoted 7 lines
- Remove console_flags_set()/console_flags_clear() macros for updating @flags (and remove their race comments).
- For
printer_should_wake()andprintk_kthread_func(), check @blocked before checking @flags.- Update commit message and comments appropriately.
Excellent work!
Quoted 5 lines
kernel/printk/printk.c +0
-0@@ -3665,15 +3802,27 @@ static int printk_kthread_func(void *data)if (error)continue ;
kernel/printk/printk.c static int printk_kthread_func(void *data) · line 3810
Quoted 10 lines
kernel/printk/printk.c +5
-3-console_lock (); + error =mutex_lock_interruptible (&con->lock); +if (error) +continue ; -if (console_suspended) { -up_console_sem (); +if (con->blocked || + !console_kthread_printing_tryenter ()) {
It is great that you moved both conditions. I have just realized how much information and functionality is accumulated here:
+ "con->blocked" is set when anyone else took @console_sem via console_lock() or when the console is suspended.
+ console_kthread_printing_tryenter() has two functions. It fails when anyone else took @console_sem via console_trylock(). Also it blocks console_trylock(). Note that console_lock() is blocked because it has to wait for con->lock.
I missed the trylock part when proposed the more safe API in the other thread, see https://lore.kernel.org/r/YmKnp3Ccu7laW3E4@alley
The safe single console lock would need to do something like:
/*
* Safe way to take con->lock. It makes sure that @console_sem is
* not taken and blocks anyone from taking @console_sem. */
void single_console_lock(struct console *con)
{try_again:
error = wait_event_interruptible(con->lock_wait,
(!con->blocked &&
!console_kthreads_atomically_blocked()));
/* Spurious wakeup */
if (error)
goto try_again;
mutex_lock(&con->lock);
/*
* Check is the console is blocked by @console_sem taken via
* console_lock() or if it is suspended.
*/
if (con->blocked) {
mutex_unlock(@con->lock);
goto try_again;
}
/*
* Try to block console_trylock(). Otherwise, we are blocked by
* @console_set taken via console_trylock().
*/
if (!console_kthread_printing_tryenter()) {
mutex_unlock(@con->lock);
goto try_again;
}
/*
* Eureka! We own @con->lock and both console_lock() and
* console_trylock() are blocked.
*/
}Best Regards,
Petr
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 100 lines · Petr Mladek
Quoted 99 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->blocked = true; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!con->blocked) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Quoted 5 lines
Quoted 1 line
Signed-off-by John Ogness <[email protected]>Nice, it it better than v4. I am going to push this for linux-next.
Reviewed-by Petr Mladek <[email protected]>
JFYI, I have just pushed this patch instead of the one from v4 into printk/linux.git, branch rework/kthreads.
It means that this branch has been rebased. It will be used in the next refresh of linux-next.
Best Regards,
Petr
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi,
Quoted 102 lines · Petr Mladek
Quoted 101 lines · Petr Mladek
Quoted 100 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->blocked = true; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!con->blocked) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Signed-off-by John Ogness <[email protected]>
Quoted 8 lines
Quoted 3 lines
Nice, it it better than v4. I am going to push this for linux-next.
Reviewed-by Petr Mladek <[email protected]>JFYI, I have just pushed this patch instead of the one from v4 into printk/linux.git, branch rework/kthreads.
It means that this branch has been rebased. It will be used in the next refresh of linux-next.
This patchset landed in linux next-20220426. In my tests I've found that it causes deadlock on all my Amlogic Meson G12B/SM1 based boards: Odroid C4/N2 and Khadas VIM3/VIM3l. The deadlock happens when system boots to userspace and getty (with automated login) is executed. I even see the bash prompt, but then the console is freezed. Reverting this patch (e00cc0e1cbf4) on top of linux-next (together with 6b3d71e87892 to make revert clean) fixes the issue.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-amlogic mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-amlogic
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 114 lines · Marek Szyprowski
Hi,
Quoted 102 lines · Petr Mladek
Quoted 101 lines · Petr Mladek
Quoted 100 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->blocked = true; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!con->blocked) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Signed-off-by John Ogness <[email protected]>Quoted 8 lines
Quoted 3 lines
Nice, it it better than v4. I am going to push this for linux-next.
Reviewed-by Petr Mladek <[email protected]>JFYI, I have just pushed this patch instead of the one from v4 into printk/linux.git, branch rework/kthreads.
It means that this branch has been rebased. It will be used in the next refresh of linux-next.
Quoted 7 lines
This patchset landed in linux next-20220426. In my tests I've found that it causes deadlock on all my Amlogic Meson G12B/SM1 based boards: Odroid C4/N2 and Khadas VIM3/VIM3l. The deadlock happens when system boots to userspace and getty (with automated login) is executed. I even see the bash prompt, but then the console is freezed. Reverting this patch (
e00cc0e1cbf4) on top of linux-next (together with6b3d71e87892to make revert clean) fixes the issue.
Thanks a lot for the report!
Just by chance, do you have the log from the dead-locked boot stored in userspace and can you share it? I mean the log stored in /var/log/dmesg or journaltctl.
In the worst case, it might help to see log from the boot with the reverted patch. I would help us to see the ordering of various console-related operations on your system.
And regarding the console. Is it the graphics console (ttyX) or a serial one (ttyS) or yet another one?
Best Regards,
Petr
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi,
Quoted 119 lines · Petr Mladek
Quoted 111 lines · Marek Szyprowski
Quoted 102 lines · Petr Mladek
Quoted 101 lines · Petr Mladek
Quoted 100 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock(){ down(&console_sem); for_each_console(con) { mutex_lock(&con->lock); con->blocked = true; mutex_unlock(&con->lock); } /* console_lock acquired */ }
console_trylock(){ if (down_trylock(&console_sem) == 0) { if (atomic_cmpxchg(&console_kthreads_active, 0, -1) == 0) { /* console_lock acquired */ } } }
threaded_printer(){ mutex_lock(&con->lock); if (!con->blocked) { /* console_lock() callers blocked */ if (atomic_inc_unless_negative(&console_kthreads_active)) { /* console_trylock() callers blocked */ con->write(); atomic_dec(&console_lock_count); } } mutex_unlock(&con->lock); }The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Signed-off-by John Ogness <[email protected]>Quoted 8 lines
Quoted 3 lines
Nice, it it better than v4. I am going to push this for linux-next.
Reviewed-by Petr Mladek <[email protected]>JFYI, I have just pushed this patch instead of the one from v4 into printk/linux.git, branch rework/kthreads.
It means that this branch has been rebased. It will be used in the next refresh of linux-next.
Quoted 7 lines
This patchset landed in linux next-20220426. In my tests I've found that it causes deadlock on all my Amlogic Meson G12B/SM1 based boards: Odroid C4/N2 and Khadas VIM3/VIM3l. The deadlock happens when system boots to userspace and getty (with automated login) is executed. I even see the bash prompt, but then the console is freezed. Reverting this patch (
e00cc0e1cbf4) on top of linux-next (together with6b3d71e87892to make revert clean) fixes the issue.
Quoted 5 lines
Thanks a lot for the report!
Just by chance, do you have the log from the dead-locked boot stored in userspace and can you share it? I mean the log stored in
/var/log/dmesgor journaltctl.
If there would be any messages, I expect them to be visible on the serial kernel console.
Quoted 6 lines
In the worst case, it might help to see log from the boot with the reverted patch. I would help us to see the ordering of various console-related operations on your system.
And regarding the console. Is it the graphics console (ttyX) or a serial one (ttyS) or yet another one?
Serial console, /dev/ttyAML0 with kernel console enabled. Later a DRM driver is loaded (meson_drm), which initializes its fbdev emulation with its console.
However it looks that I've trusted automatic bisect a bit too much and had a bit of luck while checking the reverts on top of linux-next. The issue is not 100% reproducible, so I've did this bisection again manually with more tries. The real commit causing the issue is 09c5ba0aa2fc ("printk: add kthread console printers"). Reverting the following 3 commits 6b3d71e878920b085dd823bc422951bb6f143505, e00cc0e1cbf4ea5a63d66c8de8d79519855fb231 and 09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7 on top of linux-next makes the system fully operational again.
I've also tried to disable the DRM driver and its fbdev and console (by adding modprobe.blacklist=meson_drm to kernel cmdline), but this didn't help. Here is the full serial console log:
If there is anything you would like me to try, let me know.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-amlogic mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-amlogic
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi Marek,
Quoted 3 lines · Marek Szyprowski
Here is the full serial console log:
Here are a few ideas from me:
1. For next-20220427 the printk-threaded series was slightly changed. I do not expect it to work any different, but I would prefer we are debugging the current version. If possible, could you move to next-20220427?
2. I noticed you boot with the kernel boot arguments "earlycon" and "no_console_suspend". Could you try booting without this? I expect this will make no difference.
3. It looks like the problem happens quite late in the boot process. I expect it is due to some userspace process that is running that is interacting with printk (either /dev/kmsg or /proc/kmsg) and is causing problems. If you boot with init=/bin/sh then I expect the system is running fine. (You don't have much of a system running, but it should not hang.) We need to isolate which userspace process is triggering the issue.
4. Have you tried issuing magic sysrq commands on the serial line? (For example, sending a break signal and then the letter 't' or sending a break signal and then the letter 'c'?) That might trigger various dumps so that we can see the system state.
5. You are not running a VT console, so the graphics driver should not be affecting the printk subsystem at all. I expect your autologin is also starting various services and programs. If you disable the automatic login and instead manually login (perhaps as another user) can you manually start those services one at a time to see at what point the system hangs?
Thanks for you help with this!
John Ogness
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 18 lines · John Ogness
Hi Marek,
Quoted 3 lines · Marek Szyprowski
Here is the full serial console log:
Here are a few ideas from me:
1. For next-20220427 the printk-threaded series was slightly changed. I do not expect it to work any different, but I would prefer we are debugging the current version. If possible, could you move to next-20220427?
2. I noticed you boot with the kernel boot arguments "earlycon" and "
no_console_suspend". Could you try booting without this? I expect this will make no difference.
Quoted 7 lines
3. It looks like the problem happens quite late in the boot process. I expect it is due to some userspace process that is running that is interacting with printk (either
/dev/kmsgor/proc/kmsg) and is causing problems. If you boot with init=/bin/sh then I expect the system is running fine. (You don't have much of a system running, but it should not hang.) We need to isolate which userspace process is triggering the issue.
Interesting idea.
Quoted 4 lines
4. Have you tried issuing magic sysrq commands on the serial line? (For example, sending a break signal and then the letter 't' or sending a break signal and then the letter 'c'?) That might trigger various dumps so that we can see the system state.
I see that sshd is started. If you are able to connect the system with the frozen login via ssh then it might be easier to trigger sysrq via procfs, for example:
#> echo t >/proc/sysrq-trigger
"sysrq t" should print state of all processes. It might show what process is hanging and where.
Quoted 6 lines
5. You are not running a VT console, so the graphics driver should not be affecting the printk subsystem at all. I expect your autologin is also starting various services and programs. If you disable the automatic login and instead manually login (perhaps as another user) can you manually start those services one at a time to see at what point the system hangs?
Yeah, I am not able to reproduce it and some more clues would help.
Best Regards,
Petr
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to John OgnessQuoted 5 lines · John Ogness
Hi Marek,
Quoted 2 lines · Marek Szyprowski
Here is the full serial console log:
Quoted 8 lines
Quoted 1 line
Here are a few ideas from me:
3. It looks like the problem happens quite late in the boot process. I expect it is due to some userspace process that is running that is interacting with printk (either
/dev/kmsgor/proc/kmsg) and is causing problems.
I did not find an real issue in the code handling /dev/kmsg, /proc/kmsg, or syslog sycall API. There might be just few small changes:
1. There is an increased number of spurious wakeups because log_wait is shared between upstream readers and printk kthreads. And we newly wake up waiters from both vprintk_emit() and __console_unlock() code paths.
It might affect especially the pooling APIs: kmsg_pool(), devkmsg_pool()). They might return 0 more often than before.
2. 4th patch replaced wake_up_interruptible(&log_wait) with wake_up_interruptible_all(&log_wait). As a result, all readers are woken at the same time.
It is perfectly fine because the log buffer is lockless. And all readers should be either independent or synchronized against each other.
Any of the above changes should not introduce new problems. But they might make some old problem (race) more visible.
I spent quite some time reviewing the code and testing. But I neither see any problem nor I am able to reproduce it. Some more clues from Marek would be needed.
Best Regards,
Petr
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to John OgnessHi John,
Quoted 10 lines · John Ogness
Quoted 3 lines · Marek Szyprowski
Here is the full serial console log:
Here are a few ideas from me:
1. For next-20220427 the printk-threaded series was slightly changed. I do not expect it to work any different, but I would prefer we are debugging the current version. If possible, could you move to next-20220427?
I've moved to next-20220429. Nothing changed compared to next-20220427.
Quoted 3 lines
2. I noticed you boot with the kernel boot arguments "earlycon" and "
no_console_suspend". Could you try booting without this? I expect this will make no difference.
Well, nothing changed.
Quoted 7 lines
3. It looks like the problem happens quite late in the boot process. I expect it is due to some userspace process that is running that is interacting with printk (either
/dev/kmsgor/proc/kmsg) and is causing problems. If you boot with init=/bin/sh then I expect the system is running fine. (You don't have much of a system running, but it should not hang.) We need to isolate which userspace process is triggering the issue.
The same issue happens if I boot with init=/bin/bash
Quoted 5 lines
4. Have you tried issuing magic sysrq commands on the serial line? (For example, sending a break signal and then the letter 't' or sending a break signal and then the letter 'c'?) That might trigger various dumps so that we can see the system state.
Quoted 8 lines
5. You are not running a VT console, so the graphics driver should not be affecting the printk subsystem at all. I expect your autologin is also starting various services and programs. If you disable the automatic login and instead manually login (perhaps as another user) can you manually start those services one at a time to see at what point the system hangs?
Thanks for you help with this!
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via /proc/sysrq-trigger (triggering it from UART console via break doesn't work).
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has been
updated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
Here is the output of 't' magic sys request:
If you have any more suggestion what to check let me know.
This issue must be somehow related to the way the UART driver works on the Amlogic Meson boards. The other boards based on different SoCs (Exynos, QCOM, BCM) I have in my test farm (with the same userspace and configuration) work fine with those patches.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-amlogic mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-amlogic
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.
If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello > /dev/kmsg
(You might need to increase your loglevel to see it.)
Quoted 6 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 3 lines
Here is the output of 't' magic sys request:
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Could you post your kernel config?
John
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 4 lines · John Ogness
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Right, this is how it looks like.
Quoted 10 lines
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello >
/dev/kmsg(You might need to increase your loglevel to see it.)
Data written to /dev/kmsg and all kernel logs were always displayed correctly. Also data written directly to /dev/ttyAML0 is displayed properly on the console. The latter doesn't however trigger the input related activity.
It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
Quoted 6 lines
Quoted 6 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
Quoted 8 lines
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 3 lines
Here is the output of 't' magic sys request:
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Right, good catch. For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Quoted 1 line
Could you post your kernel config?
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 10 lines · Marek Szyprowski
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
I suppose if you login via ssh and check /proc/interrupts, then type some things over serial, then check /proc/interrupts again, you will see there have been no interrupts for the uart. But interrupts for other devices are happening. Is this correct?
Quoted 4 lines
For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
This still looks odd to me. We should be seeing a trace originating from ret_from_fork+0x10/0x20 and kthread+0x118/0x11c.
I wonder if the early creation of the thread is somehow causing problems. Could you try the following patch to see if it makes a difference? I would also like to see the sysrq-t output with this patch applied:
---------------- BEGIN PATCH ---------------
kernel/printk/printk.c +1 -1
@@ -3837,7 +3837,7 @@ static int __init printk_activate_kthreads(void)return 0 ; } -early_initcall (printk_activate_kthreads); +late_initcall (printk_activate_kthreads);#if definedCONFIG_PRINTK /* If @con is specified, only wait for that console. Otherwise wait for all. */
---------------- END PATCH ---------------
Thanks for your help with this!
John
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 11 lines · John Ogness
Quoted 10 lines · Marek Szyprowski
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
Quoted 4 lines
I suppose if you login via ssh and check
/proc/interrupts, then type some things over serial, then check/proc/interruptsagain, you will see there have been no interrupts for the uart. But interrupts for other devices are happening. Is this correct?
Right. The counter for ttyAML0 is not increased when lockup happens and I type something to the uart console.
Quoted 28 lines
Quoted 4 lines
For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
This still looks odd to me. We should be seeing a trace originating from
ret_from_fork+0x10/0x20andkthread+0x118/0x11c.I wonder if the early creation of the thread is somehow causing problems. Could you try the following patch to see if it makes a difference? I would also like to see the sysrq-t output with this patch applied:
---------------- BEGIN PATCH ---------------
kernel/printk/printk.c +1
-2@@ -3837,7 +3837,7 @@ static int __init printk_activate_kthreads(void)return 0 ; } -early_initcall (printk_activate_kthreads); +late_initcall (printk_activate_kthreads);#if definedCONFIG_PRINTK /* If @con is specified, only wait for that console. Otherwise wait for all. */ ---------------- END PATCH ---------------
Quoted 1 line
Thanks for your help with this!
Well, nothing has changed. The lockup still happens.
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 7 lines · Marek Szyprowski
Quoted 4 lines
I suppose if you login via ssh and check
/proc/interrupts, then type some things over serial, then check/proc/interruptsagain, you will see there have been no interrupts for the uart. But interrupts for other devices are happening. Is this correct?Right. The counter for ttyAML0 is not increased when lockup happens and I type something to the uart console.
Hmmm. This would imply that the interrupts are disabled fo the UART.
Just to be sure that we haven't corrupted something in the driver, if you make the following change, everything works, right?
--------- BEGIN PATCH ------
kernel/printk/printk.c +1 -1
@@ -3578,7 +3578,7 @@ static int __init printk_activate_kthreads(void)struct console *con;console_lock (); - printk_kthreads_available = true; +//printk_kthreads_available = true; for_each_console (con)printk_start_kthread (con);console_unlock ();
--------- END PATCH ------
The above change will cause the kthreads to not print and instead always fallback to the direct method.
John
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 28 lines · John Ogness
Quoted 6 lines · Marek Szyprowski
Quoted 4 lines
I suppose if you login via ssh and check
/proc/interrupts, then type some things over serial, then check/proc/interruptsagain, you will see there have been no interrupts for the uart. But interrupts for other devices are happening. Is this correct?Right. The counter for ttyAML0 is not increased when lockup happens and I type something to the uart console.
Hmmm. This would imply that the interrupts are disabled fo the UART.
Just to be sure that we haven't corrupted something in the driver, if you make the following change, everything works, right?
--------- BEGIN PATCH ------
kernel/printk/printk.c +1
-2@@ -3578,7 +3578,7 @@ static int __init printk_activate_kthreads(void)struct console *con;console_lock (); - printk_kthreads_available = true; +//printk_kthreads_available = true; for_each_console (con)printk_start_kthread (con);console_unlock (); --------- END PATCH ------
Quoted 2 lines
The above change will cause the kthreads to not print and instead always fallback to the direct method.
With the above change console always works.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-amlogic mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-amlogic
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to John OgnessHi John,
Quoted 22 lines · John Ogness
Quoted 10 lines · Marek Szyprowski
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
I suppose if you login via ssh and check
/proc/interrupts, then type some things over serial, then check/proc/interruptsagain, you will see there have been no interrupts for the uart. But interrupts for other devices are happening. Is this correct?Quoted 4 lines
For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Quoted 7 lines
This still looks odd to me. We should be seeing a trace originating from
ret_from_fork+0x10/0x20andkthread+0x118/0x11c.I wonder if the early creation of the thread is somehow causing problems. Could you try the following patch to see if it makes a difference? I would also like to see the sysrq-t output with this patch applied:
On one board, I'm seeing a new splat during early boot, pointing to printk_activate_kthreads:
Calibrating delay loop (skipped), value calculated using timer
frequency.. 48.00 BogoMIPS (lpj=96000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
=============================
[ BUG: Invalid wait context ]
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431 Not tainted
-----------------------------
swapper/0/1 is trying to lock:
ffffffc00910bac8 (base_crng.lock){....}-{3:3}, at:
crng_make_state+0x148/0x1e4
other info that might help us debug this:
context-{5:5}
2 locks held by swapper/0/1:
#0: ffffffc008f8ae00 (console_lock){+.+.}-{0:0}, at:
printk_activate_kthreads+0x10/0x54
#1: ffffffc009da4a28 (&meta->lock){....}-{2:2}, at:
__kfence_alloc+0x378/0x5c4
stack backtrace:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431
Hardware name: Renesas Ebisu-4D board based on r8a77990 (DT)
Call trace:
dump_backtrace.part.0+0x98/0xc0
show_stack+0x14/0x28
dump_stack_lvl+0xac/0xec
dump_stack+0x14/0x2c
__lock_acquire+0x388/0x10a0
lock_acquire+0x190/0x2c0
_raw_spin_lock_irqsave+0x6c/0x94
crng_make_state+0x148/0x1e4
_get_random_bytes.part.0+0x4c/0xe8
get_random_u32+0x4c/0x140
__kfence_alloc+0x460/0x5c4
kmem_cache_alloc_trace+0x194/0x1dc
__kthread_create_on_node+0x5c/0x1a8
kthread_create_on_node+0x58/0x7c
printk_start_kthread.part.0+0x34/0xa8
printk_activate_kthreads+0x4c/0x54
do_one_initcall+0xec/0x278
kernel_init_freeable+0x11c/0x214
kernel_init+0x24/0x124
ret_from_fork+0x10/0x20
rcu: Hierarchical SRCU implementation.
printk: console [tty0] printing thread started
EFI services will not be available.
smp: Bringing up secondary CPUs ...
Detected VIPT I-cache on CPU1
CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
smp: Brought up 1 node, 2 CPUs
SMP: Total of 2 processors activated.Quoted 6 lines
---------------- BEGIN PATCH ---------------
kernel/printk/printk.c +0
-0
kernel/printk/printk.c static int __init printk_activate_kthreads(void)
Quoted 9 lines
kernel/printk/printk.c +1
-2@@ -3837,7 +3837,7 @@ static int __init printk_activate_kthreads(void)return 0 ; } -early_initcall (printk_activate_kthreads); +late_initcall (printk_activate_kthreads);#if definedCONFIG_PRINTK /* If @con is specified, only wait for that console. Otherwise wait for all. */ ---------------- END PATCH ---------------
Doesn't seem to make much of a difference, only a slightly different backtrace, compared to the above:
Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
+rcu: Hierarchical SRCU implementation.
=============================
[ BUG: Invalid wait context ]
-5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431 Not tainted +5.19.0-rc1-ebisu-00802-g06a0dd60d6e4-dirty #433 Not tainted
-----------------------------
swapper/0/1 is trying to lock:
ffffffc00910bac8 (base_crng.lock){....}-{3:3}, at:
crng_make_state+0x148/0x1e4
other info that might help us debug this:
context-{5:5}
-2 locks held by swapper/0/1:
- #0: ffffffc008f8ae00 (console_lock){+.+.}-{0:0}, at:
printk_activate_kthreads+0x10/0x54
- #1: ffffffc009da4a28 (&meta->lock){....}-{2:2}, at:
__kfence_alloc+0x378/0x5c4
+1 lock held by swapper/0/1:
+ #0: ffffffc009da4a28 (&meta->lock){....}-{2:2}, at:
__kfence_alloc+0x378/0x5c4
stack backtrace:
-CPU: 0 PID: 1 Comm: swapper/0 Not tainted
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431
+CPU: 0 PID: 1 Comm: swapper/0 Not tainted
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4-dirty #433
Hardware name: Renesas Ebisu-4D board based on r8a77990 (DT)
Call trace:
dump_backtrace.part.0+0x98/0xc0
@@ -33,20 +32,14 @@ Call trace:
kmem_cache_alloc_trace+0x194/0x1dc
__kthread_create_on_node+0x5c/0x1a8
kthread_create_on_node+0x58/0x7c
- printk_start_kthread.part.0+0x34/0xa8
- printk_activate_kthreads+0x4c/0x54
+ rcu_spawn_gp_kthread+0x54/0x208
do_one_initcall+0xec/0x278
kernel_init_freeable+0x11c/0x214
kernel_init+0x24/0x124
ret_from_fork+0x10/0x20
-rcu: Hierarchical SRCU implementation.
-printk: console [tty0] printing thread started
EFI services will not be available.
smp: Bringing up secondary CPUs ...
Detected VIPT I-cache on CPU1
...
+printk: console [tty0] printing thread startedGr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
(Added RANDOM NUMBER DRIVER and KFENCE people.)
Hi Geert,
Quoted 31 lines · Geert Uytterhoeven
=============================
[ BUG: Invalid wait context ]
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431 Not tainted
-----------------------------
swapper/0/1 is trying to lock:
ffffffc00910bac8 (base_crng.lock){....}-{3:3}, at:
crng_make_state+0x148/0x1e4
other info that might help us debug this:
context-{5:5}
2 locks held by swapper/0/1:
#0: ffffffc008f8ae00 (console_lock){+.+.}-{0:0}, at:
printk_activate_kthreads+0x10/0x54
#1: ffffffc009da4a28 (&meta->lock){....}-{2:2}, at:
__kfence_alloc+0x378/0x5c4
stack backtrace:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted
5.19.0-rc1-ebisu-00802-g06a0dd60d6e4 #431
Hardware name: Renesas Ebisu-4D board based on r8a77990 (DT)
Call trace:
dump_backtrace.part.0+0x98/0xc0
show_stack+0x14/0x28
dump_stack_lvl+0xac/0xec
dump_stack+0x14/0x2c
__lock_acquire+0x388/0x10a0
lock_acquire+0x190/0x2c0
_raw_spin_lock_irqsave+0x6c/0x94
crng_make_state+0x148/0x1e4
_get_random_bytes.part.0+0x4c/0xe8
get_random_u32+0x4c/0x140
__kfence_alloc+0x460/0x5c4
kmem_cache_alloc_trace+0x194/0x1dcQuoted 8 lines
__kthread_create_on_node+0x5c/0x1a8
kthread_create_on_node+0x58/0x7c
printk_start_kthread.part.0+0x34/0xa8
printk_activate_kthreads+0x4c/0x54
do_one_initcall+0xec/0x278
kernel_init_freeable+0x11c/0x214
kernel_init+0x24/0x124
ret_from_fork+0x10/0x20I am guessing you have CONFIG_PROVE_RAW_LOCK_NESTING enabled?
We are seeing a spinlock (base_crng.lock) taken while holding a raw_spinlock (meta->lock).
kfence_guarded_alloc()
raw_spin_trylock_irqsave(&meta->lock, flags)
prandom_u32_max()
prandom_u32()
get_random_u32()
get_random_bytes()
_get_random_bytes()
crng_make_state()
spin_lock_irqsave(&base_crng.lock, flags);I expect it is allowed to create kthreads via kthread_run() in early_initcalls.
John Ogness
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 1 line · John Ogness
(Added RANDOM NUMBER DRIVER and KFENCE people.)
Thanks.
Quoted 15 lines
I am guessing you have
CONFIG_PROVE_RAW_LOCK_NESTINGenabled?We are seeing a spinlock (base_crng.lock) taken while holding a
raw_spinlock(meta->lock).
kfence_guarded_alloc()
raw_spin_trylock_irqsave(&meta->lock, flags)prandom_u32_max() prandom_u32() get_random_u32() get_random_bytes() _get_random_bytes() crng_make_state() spin_lock_irqsave(&base_crng.lock, flags);
Quoted 2 lines
I expect it is allowed to create kthreads via
kthread_run()inearly_initcalls.
AFAIK, CONFIG_PROVE_RAW_LOCK_NESTING is useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add a depends on !KFENCE to PROVE_RAW_LOCK_NESTING?
Jason
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 23 lines · Jason A. Donenfeld
Hi John,
Quoted 1 line · John Ogness
(Added RANDOM NUMBER DRIVER and KFENCE people.)
Thanks.
Quoted 15 lines
I am guessing you have
CONFIG_PROVE_RAW_LOCK_NESTINGenabled?We are seeing a spinlock (base_crng.lock) taken while holding a
raw_spinlock(meta->lock).
kfence_guarded_alloc()
raw_spin_trylock_irqsave(&meta->lock, flags)prandom_u32_max() prandom_u32() get_random_u32() get_random_bytes() _get_random_bytes() crng_make_state() spin_lock_irqsave(&base_crng.lock, flags);
Quoted 8 lines
Quoted 2 lines
I expect it is allowed to create kthreads via
kthread_run()inearly_initcalls.AFAIK,
CONFIG_PROVE_RAW_LOCK_NESTINGis useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add adepends on !KFENCEtoPROVE_RAW_LOCK_NESTING?
Don't know if there are other good solutions (of similar simplicity). But fwiw this is not about the target production environment. Real production uses of RT kernels will probably not enable LOCKDEP, PROVE_RAW_LOCK_NESTING and other debugging configs.
This is about detecting as many bugs as possible in testing environments. And testing environments can well have both LOCKDEP and KFENCE enabled. Any such limitation will require doubling the number of tested configurations.
Btw, should this new CONFIG_PROVE_RAW_LOCK_NESTING be generally enabled on testing systems? We don't have it enabled on syzbot.
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi Dmitry,
Quoted 7 lines · Dmitry Vyukov
Quoted 5 lines
AFAIK,
CONFIG_PROVE_RAW_LOCK_NESTINGis useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add adepends on !KFENCEtoPROVE_RAW_LOCK_NESTING?Don't know if there are other good solutions (of similar simplicity).
Fortunately, I found one that solves things without needing to compromise on anything:
https://lore.kernel.org/lkml/[email protected]/
Quoted 2 lines
Btw, should this new
CONFIG_PROVE_RAW_LOCK_NESTINGbe generally enabled on testing systems? We don't have it enabled on syzbot.
Last time I spoke with RT people about this, the goal was eventually to *always* enable it when lock proving is enabled, but there are too many bugs and cases now to do that, so it's an opt-in. I might be misremembering, though, so CC'ing Sebastian in case he wants to chime in.
Jason
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 10 lines · Jason A. Donenfeld
Hi Dmitry,
Quoted 6 lines · Dmitry Vyukov
Quoted 5 lines
AFAIK,
CONFIG_PROVE_RAW_LOCK_NESTINGis useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add adepends on !KFENCEtoPROVE_RAW_LOCK_NESTING?
Quoted 5 lines
Quoted 1 line
Don't know if there are other good solutions (of similar simplicity).
Fortunately, I found one that solves things without needing to compromise on anything:
https://lore.kernel.org/lkml/[email protected]/
Cool! Thanks!
Quoted 8 lines
Quoted 2 lines
Btw, should this new
CONFIG_PROVE_RAW_LOCK_NESTINGbe generally enabled on testing systems? We don't have it enabled on syzbot.Last time I spoke with RT people about this, the goal was eventually to *always* enable it when lock proving is enabled, but there are too many bugs and cases now to do that, so it's an opt-in. I might be misremembering, though, so CC'ing Sebastian in case he wants to chime in.
OK, we will wait then.
Little point in doubling the number of reports for known issues.
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Jason A. DonenfeldQuoted 15 lines · Jason A. Donenfeld
Hi Dmitry,
Quoted 7 lines · Dmitry Vyukov
Quoted 5 lines
AFAIK,
CONFIG_PROVE_RAW_LOCK_NESTINGis useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add adepends on !KFENCEtoPROVE_RAW_LOCK_NESTING?Don't know if there are other good solutions (of similar simplicity).
Fortunately, I found one that solves things without needing to compromise on anything:
https://lore.kernel.org/lkml/[email protected]/
Quoted 8 lines
Quoted 2 lines
Btw, should this new
CONFIG_PROVE_RAW_LOCK_NESTINGbe generally enabled on testing systems? We don't have it enabled on syzbot.Last time I spoke with RT people about this, the goal was eventually to *always* enable it when lock proving is enabled, but there are too many bugs and cases now to do that, so it's an opt-in. I might be misremembering, though, so CC'ing Sebastian in case he wants to chime in.
That is basically still the case. If CONFIG_PROVE_RAW_LOCK_NESTING yells then there will be yelling on PREEMPT_RT, too. We would like to get things fixed ;)
Without going through this thread, John is looking at printk and printk triggers a few of those. That is one of reasons why this is not enabled by default.
Quoted 1 line
Jason
Sebastian
_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Jason A. DonenfeldHey again,
Quoted 22 lines · Jason A. Donenfeld
Hi John,
Quoted 1 line · John Ogness
(Added RANDOM NUMBER DRIVER and KFENCE people.)
Thanks.
Quoted 15 lines
I am guessing you have
CONFIG_PROVE_RAW_LOCK_NESTINGenabled?We are seeing a spinlock (base_crng.lock) taken while holding a
raw_spinlock(meta->lock).
kfence_guarded_alloc()
raw_spin_trylock_irqsave(&meta->lock, flags)prandom_u32_max() prandom_u32() get_random_u32() get_random_bytes() _get_random_bytes() crng_make_state() spin_lock_irqsave(&base_crng.lock, flags);
Quoted 8 lines
Quoted 2 lines
I expect it is allowed to create kthreads via
kthread_run()inearly_initcalls.AFAIK,
CONFIG_PROVE_RAW_LOCK_NESTINGis useful for teasing out cases where RT's raw spinlocks will nest wrong with RT's sleeping spinlocks. But nobody who wants an RT kernel will be using KFENCE. So this seems like a non-issue? Maybe just add adepends on !KFENCEtoPROVE_RAW_LOCK_NESTING?
On second thought, the fix is trivial:
https://lore.kernel.org/lkml/[email protected]/
Jason
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Marek SzyprowskiQuoted 26 lines · Marek Szyprowski
Hi John,
Quoted 4 lines · John Ogness
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Right, this is how it looks like.
Quoted 10 lines
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello >
/dev/kmsg(You might need to increase your loglevel to see it.)
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.
Quoted 5 lines
It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
This is really interesting.
'date >/dev/kmsg' should be handled like a normal printk(). It should get pushed to the console using printk kthread, that calls call_console_driver() that calls con->write() callback. In our case, it should be meson_serial_console_write().
I am not sure if meson_serial_console_write() is used also when writing via /dev/ttyAML0.
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 3 lines
Here is the output of 't' magic sys request:
Quoted 8 lines
Quoted 2 lines
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Right, good catch. For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Strange. The backtrace is weird here too:
[ 50.514509] task:pr/ttyAML0 state:R running task stack: 0 pid: 65 ppid: 2 flags:0x00000008 [ 50.514540] Call trace: [ 50.514548] __switch_to+0xe8/0x160 [ 50.514561] meson_serial_console+0x78/0x118
There should be kthread() and printk_kthread_func() on the stack.
Hmm, meson_serial_console+0x78/0x118 is weird on its own. meson_serial_console is the name of the structure. I would expect a name of the .write callback, like meson_serial_console_write()
Best Regards,
Petr
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi Petr,
Quoted 64 lines · Petr Mladek
Quoted 22 lines · Marek Szyprowski
Quoted 4 lines · John Ogness
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Right, this is how it looks like.
Quoted 10 lines
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello >
/dev/kmsg(You might need to increase your loglevel to see it.)
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.Quoted 5 lines
It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
This is really interesting.
'date >/dev/kmsg' should be handled like a normal
printk(). It should get pushed to the console using printk kthread, that callscall_console_driver()that callscon->write()callback. In our case, it should bemeson_serial_console_write().I am not sure if
meson_serial_console_write()is used also when writing via/dev/ttyAML0.Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 2 lines
Here is the output of 't' magic sys request:
Quoted 8 lines
Quoted 3 lines
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Right, good catch. For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Strange. The backtrace is weird here too:
[ 50.514509] task:pr/ttyAML0 state:R running task stack: 0 pid: 65 ppid: 2 flags:0x00000008 [ 50.514540] Call trace: [ 50.514548] __switch_to+0xe8/0x160 [ 50.514561] meson_serial_console+0x78/0x118
Quoted 6 lines
There should be
kthread()andprintk_kthread_func()on the stack.Hmm,
meson_serial_console+0x78/0x118is weird on its own.meson_serial_consoleis the name of the structure. I would expect a name of the .write callback, likemeson_serial_console_write()
Well, this sounds a bit like a stack corruption. For the reference, I've checked what magic sysrq 't' reports for my other test boards:
RaspberryPi4:
[ 166.702431] task:pr/ttyS0 state:R stack: 0 pid: 64 ppid: 2 flags:0x00000008
[ 166.711069] Call trace:
[ 166.713647] __switch_to+0xe8/0x160
[ 166.717216] __schedule+0x2f4/0x9f0
[ 166.720862] log_wait+0x0/0x50
[ 166.724081] task:vfio-irqfd-clea state:I stack: 0 pid: 65 ppid: 2 flags:0x00000008
[ 166.732698] Call trace:
ARM Juno R1:
[ 74.356562] task:pr/ttyAMA0 state:R running task stack: 0 pid: 47 ppid: 2 flags:0x00000008
[ 74.356605] Call trace:
[ 74.356617] __switch_to+0xe8/0x160
[ 74.356637] amba_console+0x78/0x118
[ 74.356657] task:kworker/2:1 state:I stack: 0 pid: 48 ppid: 2 flags:0x00000008
[ 74.356695] Workqueue: 0x0 (mm_percpu_wq)
[ 74.356738] Call trace:
QEMU virt/arm64:
[ 174.155760] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 174.156305] Call trace:
[ 174.156529] __switch_to+0xe8/0x160
[ 174.157131] 0xffff5ebbbfdd62d8
In the last case it doesn't happen always. In the other runs I got the following log from QEMU virt/arm64:
[ 200.537579] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 200.538121] Call trace:
[ 200.538341] __switch_to+0xe8/0x160
[ 200.538583] __schedule+0x2f4/0x9f0
[ 200.538822] schedule+0x54/0xd0
[ 200.539047] printk_kthread_func+0x2d8/0x3bc
[ 200.539301] kthread+0x118/0x11c
[ 200.539523] ret_from_fork+0x10/0x20
I hope that at least the qemu case will let you to analyze it by yourself. I run my test system with the following command:
qemu-system-aarch64 -kernel virt/Image -append "console=ttyAMA0 root=/dev/vda rootwait" -M virt -cpu cortex-a57 -smp 2 -m 1024 -device virtio-blk-device,drive=virtio-blk0 -drive file=qemu-virt-rootfs.raw,id=virtio-blk0,if=none,format=raw -netdev user,id=user -device virtio-net-device,netdev=user -display none
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 103 lines · Marek Szyprowski
Hi Petr,
Quoted 64 lines · Petr Mladek
Quoted 22 lines · Marek Szyprowski
Quoted 4 lines · John Ogness
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Right, this is how it looks like.
Quoted 10 lines
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello >
/dev/kmsg(You might need to increase your loglevel to see it.)
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.Quoted 5 lines
It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
This is really interesting.
'date >/dev/kmsg' should be handled like a normal
printk(). It should get pushed to the console using printk kthread, that callscall_console_driver()that callscon->write()callback. In our case, it should bemeson_serial_console_write().I am not sure if
meson_serial_console_write()is used also when writing via/dev/ttyAML0.Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 2 lines
Here is the output of 't' magic sys request:
Quoted 8 lines
Quoted 3 lines
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Right, good catch. For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Strange. The backtrace is weird here too:
[ 50.514509] task:pr/ttyAML0 state:R running task stack: 0 pid: 65 ppid: 2 flags:0x00000008 [ 50.514540] Call trace: [ 50.514548] __switch_to+0xe8/0x160 [ 50.514561] meson_serial_console+0x78/0x118Quoted 6 lines
There should be
kthread()andprintk_kthread_func()on the stack.Hmm,
meson_serial_console+0x78/0x118is weird on its own.meson_serial_consoleis the name of the structure. I would expect a name of the .write callback, likemeson_serial_console_write()Well, this sounds a bit like a stack corruption. For the reference, I've checked what magic sysrq 't' reports for my other test boards:
RaspberryPi4:
[ 166.702431] task:pr/ttyS0 state:R stack: 0 pid: 64 ppid: 2 flags:0x00000008
[ 166.711069] Call trace:
[ 166.713647]__switch_to+0xe8/0x160
[ 166.717216]__schedule+0x2f4/0x9f0
[ 166.720862]log_wait+0x0/0x50
[ 166.724081] task:vfio-irqfd-clea state:I stack: 0 pid: 65 ppid: 2 flags:0x00000008
[ 166.732698] Call trace:ARM Juno R1:
[ 74.356562] task:pr/ttyAMA0 state:R running task stack: 0 pid: 47 ppid: 2 flags:0x00000008
[ 74.356605] Call trace:
[ 74.356617]__switch_to+0xe8/0x160
[ 74.356637]amba_console+0x78/0x118
[ 74.356657] task:kworker/2:1 state:I stack: 0 pid: 48 ppid: 2 flags:0x00000008
[ 74.356695] Workqueue:0x0(mm_percpu_wq)
[ 74.356738] Call trace:
Quoted 7 lines
QEMU
virt/arm64:[ 174.155760] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 174.156305] Call trace:
[ 174.156529]__switch_to+0xe8/0x160
[ 174.157131]0xffff5ebbbfdd62d8
You mentioned in the other mail that the other boards work as expected. I mean that console gets stuck only on the meson board. Is it true, please?
The stack looks really weird. But another weird thing is that even the meson board is able to show the messages, for example, using echo hello >/dev/kmsg. It suggests that the kthreads somehow work.
There is also a possibility that this code path is optimized some special way and the unwinder has troubles to show the stack correctly.
Quoted 12 lines
In the last case it doesn't happen always. In the other runs I got the following log from QEMU
virt/arm64:[ 200.537579] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 200.538121] Call trace:
[ 200.538341]__switch_to+0xe8/0x160
[ 200.538583]__schedule+0x2f4/0x9f0
[ 200.538822]schedule+0x54/0xd0
[ 200.539047]printk_kthread_func+0x2d8/0x3bc
[ 200.539301]kthread+0x118/0x11c
[ 200.539523]ret_from_fork+0x10/0x20
This is what I would expect when the kthread is in an interruptible sleep waiting for new strings to handle.
BTW: This is what I see on my x86_64 test system:
[61892.932242] task:pr/tty0 state:S stack: 0 pid: 14 ppid: 2 flags:0x00004000 [61892.932250] Call Trace: [61892.932253] <TASK> [61892.932263] __schedule+0x376/0xbb0 [61892.932284] schedule+0x44/0xb0 [61892.932290] printk_kthread_func+0x18f/0x370 [61892.932303] ? schedstat_stop+0x10/0x10 [61892.932316] ? console_start+0x30/0x30 [61892.932322] kthread+0xf2/0x120 [61892.932327] ? kthread_complete_and_exit+0x20/0x20 [61892.932338] ret_from_fork+0x1f/0x30 [61892.932370] </TASK> [61892.932373] task:pr/ttyS0 state:R running task stack: 0 pid: 15 ppid: 2 flags:0x00004000 [61892.932391] Call Trace: [61892.932398] <TASK> [61892.932427] ? printk_kthread_func+0x15b/0x370 [61892.932436] ? schedstat_stop+0x10/0x10 [61892.932449] ? console_start+0x30/0x30 [61892.932455] ? kthread+0xf2/0x120 [61892.932460] ? kthread_complete_and_exit+0x20/0x20 [61892.932471] ? ret_from_fork+0x1f/0x30 [61892.932502] </TASK>
pr/tty0 is in the interruptible sleep and the stack looks reasonable.
pr/ttyS0 is in runnable state and the stack is weird. The '?' means that this address was found on the stack, it belongs to some function but the unwinder is not able to assign it to the current call path by going back via the return addresses stored on the stack.
Quoted 8 lines
I hope that at least the qemu case will let you to analyze it by yourself. I run my test system with the following command:
qemu-system-aarch64 -kernel
virt/Image-append "console=ttyAMA0 root=/dev/vda rootwait" -M virt -cpu cortex-a57 -smp 2 -m 1024 -device virtio-blk-device,drive=virtio-blk0 -drive file=qemu-virt-rootfs.raw,id=virtio-blk0,if=none,format=raw -netdev user,id=user -device virtio-net-device,netdev=user -display none
Thanks a lot for all the information. It is really helpful.
Best Regards,
Petr
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi,
Quoted 103 lines · Petr Mladek
Quoted 102 lines · Marek Szyprowski
Quoted 64 lines · Petr Mladek
Quoted 22 lines · Marek Szyprowski
Quoted 4 lines · John Ogness
Quoted 1 line · Marek Szyprowski
The same issue happens if I boot with init=/bin/bash
Very interesting. Since you are seeing all the output up until you try doing something, I guess receiving UART data is triggering the issue.
Right, this is how it looks like.
Quoted 10 lines
Quoted 3 lines
I found something really interesting. When lockup happens, I'm still able to log via ssh and trigger any magic sysrq action via
/proc/sysrq-trigger.If you boot the system and directly login via ssh (without sending any data via serial), can you trigger printk output on the UART? For example, with:
echo hello >
/dev/kmsg(You might need to increase your loglevel to see it.)
Data written to
/dev/kmsgand all kernel logs were always displayed correctly. Also data written directly to/dev/ttyAML0is displayed properly on the console. The latter doesn't however trigger the input related activity.Quoted 5 lines
It looks that the data read from the uart is delivered only if other activity happens on the kernel console. If I type 'reboot' and press enter, nothing happens immediately. If I type 'date >/dev/ttyAML0' via ssh then, I only see the date printed on the console. However if I type 'date >/dev/kmsg', the the date is printed and reboot happens.
This is really interesting.
'date >/dev/kmsg' should be handled like a normal
printk(). It should get pushed to the console using printk kthread, that callscall_console_driver()that callscon->write()callback. In our case, it should bemeson_serial_console_write().I am not sure if
meson_serial_console_write()is used also when writing via/dev/ttyAML0.Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
It turned out that the UART console is somehow blocked, but it receives and buffers all the input. For example after issuing "echo
Quoted 1 line
/proc/sysrq-trigger" from the ssh console, the UART console has beenupdated and I see the magic sysrq banner and then all the commands I blindly typed in the UART console! However this doesn't unblock the console.
sysrq falls back to direct printing. This would imply that the kthread printer is somehow unable to print.
Quoted 2 lines
Here is the output of 't' magic sys request:
Quoted 8 lines
Quoted 3 lines
It looks like the call trace for the printing kthread (pr/ttyAML0) is corrupt.
Right, good catch. For comparison, here is a 't' sysrq result from the 'working' serial console (next-20220429), which happens usually 1 of 4 boots:
Strange. The backtrace is weird here too:
[ 50.514509] task:pr/ttyAML0 state:R running task stack: 0 pid: 65 ppid: 2 flags:0x00000008 [ 50.514540] Call trace: [ 50.514548] __switch_to+0xe8/0x160 [ 50.514561] meson_serial_console+0x78/0x118Quoted 6 lines
There should be
kthread()andprintk_kthread_func()on the stack.Hmm,
meson_serial_console+0x78/0x118is weird on its own.meson_serial_consoleis the name of the structure. I would expect a name of the .write callback, likemeson_serial_console_write()Well, this sounds a bit like a stack corruption. For the reference, I've checked what magic sysrq 't' reports for my other test boards:
RaspberryPi4:
[ 166.702431] task:pr/ttyS0 state:R stack: 0 pid: 64 ppid: 2 flags:0x00000008
[ 166.711069] Call trace:
[ 166.713647]__switch_to+0xe8/0x160
[ 166.717216]__schedule+0x2f4/0x9f0
[ 166.720862]log_wait+0x0/0x50
[ 166.724081] task:vfio-irqfd-clea state:I stack: 0 pid: 65 ppid: 2 flags:0x00000008
[ 166.732698] Call trace:ARM Juno R1:
[ 74.356562] task:pr/ttyAMA0 state:R running task stack: 0 pid: 47 ppid: 2 flags:0x00000008
[ 74.356605] Call trace:
[ 74.356617]__switch_to+0xe8/0x160
[ 74.356637]amba_console+0x78/0x118
[ 74.356657] task:kworker/2:1 state:I stack: 0 pid: 48 ppid: 2 flags:0x00000008
[ 74.356695] Workqueue:0x0(mm_percpu_wq)
[ 74.356738] Call trace:QEMU
virt/arm64:
Quoted 8 lines
Quoted 5 lines
[ 174.155760] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 174.156305] Call trace:
[ 174.156529]__switch_to+0xe8/0x160
[ 174.157131]0xffff5ebbbfdd62d8You mentioned in the other mail that the other boards work as expected. I mean that console gets stuck only on the meson board. Is it true, please?
Right. Even on Meson based boards the console is operational about 1 of 4 boots.
Quoted 8 lines
The stack looks really weird. But another weird thing is that even the meson board is able to show the messages, for example, using echo hello >/dev/kmsg. It suggests that the kthreads somehow work.
There is also a possibility that this code path is optimized some special way and the unwinder has troubles to show the stack correctly.
I doubt that this is a result of the compiler's optimization. See my logs from QEMU's virt machine. I've managed to capture 2 states of ttyAMA0 task. One shows some kind of stack corruption imho. It doesn't happen always though.
Quoted 1 line
...
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Marek SzyprowskiQuoted 7 lines · Marek Szyprowski
QEMU
virt/arm64:[ 174.155760] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008
[ 174.156305] Call trace:
[ 174.156529]__switch_to+0xe8/0x160
[ 174.157131]0xffff5ebbbfdd62d8
I can reproduce the apparent stack corruption with qemu:
[ 5.545268] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008 [ 5.545520] Call trace: [ 5.545620] __switch_to+0x104/0x160 [ 5.545796] __schedule+0x2f4/0x9f0 [ 5.546122] schedule+0x54/0xd0 [ 5.546206] 0x0
When it happens, the printk-kthread is the only one with the corrupted stack. It seems I am doing something wrong when creating the kthread? I will investigate this.
Thanks Marek for helping us to narrow this down.
John
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 8 lines · John Ogness
I can reproduce the apparent stack corruption with qemu:
[ 5.545268] task:pr/ttyAMA0 state:S stack: 0 pid: 26 ppid: 2 flags:0x00000008 [ 5.545520] Call trace: [ 5.545620] __switch_to+0x104/0x160 [ 5.545796] __schedule+0x2f4/0x9f0 [ 5.546122] schedule+0x54/0xd0 [ 5.546206] 0x0
I believe I am chasing a ghost. I can rather easily reproduce these strange call traces, but if another sysrq-t is sent afterwards, the call trace is OK. Also, I added trace_dump_stack() into the printk-kthread main loop to dump the stack on every iteration. There I never see any corruption, even though the timestamps are near the sysrq-t dump showing corruption. Moving trace_dump_stack() into amba-pl011:pl011_console_write() also showed no stack corruption at very near times when sysrq-t did.
And it should be noted that the console-hanging issues reported in this thread _cannot_ be reproduced with qemu.
So I will stop focussing on this "corrupt stack" thing and instead investigate what the meson driver is doing that causes it to get stuck. Since interrupts do not even fire, I'm guessing that the RX interrupts are not being re-enabled (AML_UART_RX_INT_EN) for some code path. This bit is only explicitly set once, in meson_uart_startup(). Whenever the bit is cleared, later the previous value is restored. This is assumed to mean the interrupt gets re-enabled. But if there is some code path where multiple CPUs can modify the register, then the interrupt could end up permanently disabled.
I will go through and check if all access to AML_UART_CONTROL is protected by port->lock.
John
_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi Marek,
Quoted 2 lines · John Ogness
I will go through and check if all access to
AML_UART_CONTROLis protected byport->lock.
The startup() callback of the uart_ops is not called with the port locked. I'm having difficulties identifying if the startup() callback can occur after the console was already registered via meson_uart_init() and could be actively printing, but I see other serial drivers are protecting their registers in the startup() callback with the port->lock.
Could you try booting the meson hardware with the following change? (And removing any previous debug changes I posted?)
John
drivers/tty/serial/meson_uart.c +5 -0
@@ -267,9 +267,12 @@ static void meson_uart_reset(struct uart_port *port)static int meson_uart_startup (struct uart_port *port) { +unsigned long flags;u32 val;int ret =0 ; +spin_lock_irqsave (&port->lock, flags); + val =readl (port->membase +AML_UART_CONTROL ); val |=AML_UART_CLEAR_ERR ;writel (val, port->membase +AML_UART_CONTROL );
@@ -285,6 +288,8 @@ static int meson_uart_startup(struct uart_port *port) val = (AML_UART_RECV_IRQ (1 ) |AML_UART_XMIT_IRQ (port->fifosize /2 ));writel (val, port->membase +AML_UART_MISC ); +spin_unlock_irqrestore (&port->lock, flags); + ret =request_irq (port->irq, meson_uart_interrupt,0 , port->name, port);
_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
Marek Szyprowski
Tested-byRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 10 lines · John Ogness
Quoted 2 lines · John Ogness
I will go through and check if all access to
AML_UART_CONTROLis protected byport->lock.The
startup()callback of theuart_opsis not called with the port locked. I'm having difficulties identifying if thestartup()callback can occur after the console was already registered viameson_uart_init()and could be actively printing, but I see other serial drivers are protecting their registers in thestartup()callback with theport->lock.
Quoted 2 lines
Could you try booting the meson hardware with the following change? (And removing any previous debug changes I posted?)
Bingo! It looks that the startup() is called when getty initializes console. This fixed the issues observed on the Amlogic Meson based boards.
Feel free to add:
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi,
Quoted 16 lines · Marek Szyprowski
Hi John,
Quoted 10 lines · John Ogness
Quoted 2 lines · John Ogness
I will go through and check if all access to
AML_UART_CONTROLis protected byport->lock.The
startup()callback of theuart_opsis not called with the port locked. I'm having difficulties identifying if thestartup()callback can occur after the console was already registered viameson_uart_init()and could be actively printing, but I see other serial drivers are protecting their registers in thestartup()callback with theport->lock.Quoted 2 lines
Could you try booting the meson hardware with the following change? (And removing any previous debug changes I posted?)
Quoted 8 lines
Bingo! It looks that the
startup()is called when getty initializes console. This fixed the issues observed on the Amlogic Meson based boards.Feel free to add:
Tested-by Marek Szyprowski <[email protected]>Best regards
Thanks all for figuring out the issue, perhaps other uart drivers could fall in the same issue if startup code isn't protected with lock ?
Neil
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi Neil,
Quoted 3 lines · Neil Armstrong
Thanks all for figuring out the issue, perhaps other uart drivers could fall in the same issue if startup code isn't protected with lock?
When preparing for the official patch submission [0], I needed quite a bit of time to understand why another function (meson_uart_reset) should not and cannot acquire the port->lock.
I then started investigating some other drivers and indeed I see lots of potential problems. Any console initializing port->lock from the driver's probe() is probably wrong (and there are lots of them). But as I've learned with the meson driver, the details are subtle. Each driver will need to be carefully evaluated to see if it is actually safe.
uart_ops->startup() is called without holding port->lock. If the device is a console, it is already registered and printing.
driver->probe() is called without holding port->lock. If the device is a console, it is already registered and printing.
For both functions, port->lock might not be initialized yet, so blindly acquiring it is wrong.
Note that this is not related to the introduction of kthread printing.
I've put it on my TODO list to go through the ~76 console drivers to investigate their startup() and probe() implementations. But I will not be able to do this quickly. My time might be better spent writing to all the maintainers asking them to please verify the usage.
John Ogness
Petr Mladek
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Marek SzyprowskiQuoted 12 lines · Marek Szyprowski
Hi John,
Quoted 9 lines · John Ogness
Quoted 2 lines · John Ogness
I will go through and check if all access to
AML_UART_CONTROLis protected byport->lock.The
startup()callback of theuart_opsis not called with the port locked. I'm having difficulties identifying if thestartup()callback can occur after the console was already registered viameson_uart_init()and could be actively printing, but I see other serial drivers are protecting their registers in thestartup()callback with theport->lock.
I guess that it is used by the early console before the racy code is called.
Quoted 9 lines
Quoted 2 lines
Could you try booting the meson hardware with the following change? (And removing any previous debug changes I posted?)
Bingo! It looks that the
startup()is called when getty initializes console. This fixed the issues observed on the Amlogic Meson based boards.Feel free to add:
Tested-by Marek Szyprowski <[email protected]>
Uff, it is a huge relief that it has got fixed.
Best Regards,
Petr
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Marek SzyprowskiQuoted 9 lines · Marek Szyprowski
Quoted 2 lines
Could you try booting the meson hardware with the following change? (And removing any previous debug changes I posted?)
Bingo! It looks that the
startup()is called when getty initializes console. This fixed the issues observed on the Amlogic Meson based boards.Feel free to add:
Tested-by Marek Szyprowski <[email protected]>
Thanks Marek. I will post an official patch on the correct people/lists.
John
_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to Marek SzyprowskiHi All,
Quoted 114 lines · Marek Szyprowski
Quoted 104 lines · Petr Mladek
Quoted 103 lines · Petr Mladek
Quoted 102 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.But the threaded consoles printers do still need to synchronize against
console_lock()callers. Introduce a per-console mutex and a new console boolean field @blocked to provide this synchronization.
console_lock()is modified so that it must acquire the mutex of each console in order to set the @blocked field. Console printing threads will acquire their mutex while printing a record. If @blocked was set, the thread will go back to sleep instead of printing.The reason for the @blocked boolean field is so that
console_lock()callers do not need to acquire multiple console mutexes simultaneously, which would introduce unnecessary complexity due to nested mutex locking. Also, a new field was chosen instead of adding a new @flags value so that the blocked status could be checked without concern of reading inconsistent values due to @flags updates from other contexts.Threaded console printers also need to synchronize against
console_trylock()callers. Sinceconsole_trylock()may be called from any context, the per-console mutex cannot be used for this synchronization. (mutex_trylock()cannot be called from atomic contexts.) Introduce a global atomic counter to identify if any threaded printers are active. The threaded printers will also check the atomic counter to identify if the console has been locked by another task viaconsole_trylock().Note that @console_sem is still used to provide synchronization between
console_lock()andconsole_trylock()callers.A locking overview for
console_lock(),console_trylock(), and the threaded printers is as follows (pseudo code):
console_lock()
{
down(&console_sem);
for_each_console(con){
mutex_lock(&con->lock);
con->blocked= true;
mutex_unlock(&con->lock);
}
/*console_lockacquired */
}
console_trylock()
{
if (down_trylock(&console_sem)== 0) {
if (atomic_cmpxchg(&console_kthreads_active, 0, -1)== 0) {
/*console_lockacquired */
}
}
}
threaded_printer()
{
mutex_lock(&con->lock);
if (!con->blocked) {
/*console_lock()callers blocked */if (
atomic_inc_unless_negative(&console_kthreads_active)) { /*console_trylock()callers blocked */
con->write();
atomic_dec(&console_lock_count);
}
}
mutex_unlock(&con->lock);
}The console owner and waiter logic now only applies between contexts that have taken the
console_lockviaconsole_trylock(). Threaded printers never take theconsole_lock, so they do not have aconsole_lockto handover. Tasks that have usedconsole_lock()will block the threaded printers using a mutex and if theconsole_lockis handed over to an atomic context, it would be unable to unblock the threaded printers. However, theconsole_trylock()case is really the only scenario that is interesting for handovers anyway.@panic_console_dropped must change to
atomic_tsince it is no longer protected exclusively by theconsole_lock.Since threaded printers remain asleep if they see that the console is locked, they now must be explicitly woken in
__console_unlock(). This meanswake_up_klogd()calls following aconsole_unlock()are no longer necessary and are removed.Also note that threaded printers no longer need to check @console_suspended. The check for the @blocked field implicitly covers the suspended console case.
Signed-off-by John Ogness <[email protected]>Quoted 8 lines
Quoted 3 lines
Nice, it it better than v4. I am going to push this for linux-next.
Reviewed-by Petr Mladek <[email protected]>JFYI, I have just pushed this patch instead of the one from v4 into printk/linux.git, branch rework/kthreads.
It means that this branch has been rebased. It will be used in the next refresh of linux-next.
Quoted 8 lines
This patchset landed in linux next-20220426. In my tests I've found that it causes deadlock on all my Amlogic Meson G12B/SM1 based boards:
Odroid C4/N2 and Khadas VIM3/VIM3l. The deadlock happens when system boots to userspace and getty (with automated login) is executed. I even see the bash prompt, but then the console is freezed. Reverting this patch (e00cc0e1cbf4) on top of linux-next (together with6b3d71e87892to make revert clean) fixes the issue.
The Amlogic Meson related issue has been investigated and fixed:
https://lore.kernel.org/all/[email protected]/
but I just found that there is one more issue.
It appears on QCom-based DragonBoard 410c SBC (arch/arm64/boot/dts/qcom/apq8016-sbc.dts). To see it on today's linux next-20220506, one has to revert 42cd402b8fd4672b692400fe5f9eecd55d2794ac, otherwise lockdep triggers other warning and it is turned off too early:
================================
WARNING: inconsistent lock state
5.18.0-rc5-next-20220506+ #11869 Not tainted
--------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:ffff80000aaa8478 (&port_lock_key){?.+.}-{2:2}, at: msm_uart_irq+0x38/0x750 {HARDIRQ-ON-W} state was registered at:
lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock+0x5c/0x80 __msm_console_write+0x1ac/0x220 msm_console_write+0x48/0x60 __console_emit_next_record+0x188/0x420 printk_kthread_func+0x3a0/0x3bc kthread+0x118/0x11c ret_from_fork+0x10/0x20 irq event stamp: 12182 hardirqs last enabled at (12181): [<ffff800008e3d2a8>] cpuidle_enter_state+0xc4/0x30c stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.0-rc5-next-20220506+ #11869 Hardware name: Qualcomm Technologies, Inc. APQ 8016 SBC (DT) Call trace: dump_backtrace.part.0+0xd0/0xe0 show_stack+0x18/0x6c dump_stack_lvl+0x8c/0xb8 dump_stack+0x18/0x34 print_usage_bug.part.0+0x208/0x22c mark_lock+0x710/0x954 __lock_acquire+0x9fc/0x20cc lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock_irqsave+0x80/0xcc msm_uart_irq+0x38/0x750 __handle_irq_event_percpu+0xac/0x3d0 handle_irq_event+0x4c/0x120 handle_fasteoi_irq+0xa4/0x1a0 generic_handle_domain_irq+0x3c/0x60 gic_handle_irq+0x44/0xc4 call_on_irq_stack+0x2c/0x54 do_interrupt_handler+0x80/0x84 el1_interrupt+0x34/0x64 el1h_64_irq_handler+0x18/0x24 el1h_64_irq+0x64/0x68 cpuidle_enter_state+0xcc/0x30c cpuidle_enter+0x38/0x50 do_idle+0x22c/0x2bc cpu_startup_entry+0x28/0x30 rest_init+0x110/0x190 arch_post_acpi_subsys_init+0x0/0x18 start_kernel+0x6c4/0x704 __primary_switched+0xc0/0xc8 INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
Reverting the following patches on top of linux next-20220506 fixes/hides this lockdep warning:
6b3d71e87892 ("printk: remove @console_locked") 8e274732115f ("printk: extend console_lock for per-console locking") 09c5ba0aa2fc ("printk: add kthread console printers")
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 67 lines · Marek Szyprowski
The Amlogic Meson related issue has been investigated and fixed:
https://lore.kernel.org/all/[email protected]/
but I just found that there is one more issue.
It appears on QCom-based DragonBoard 410c SBC (
arch/arm64/boot/dts/qcom/apq8016-sbc.dts). To see it on today's linux next-20220506, one has to revert42cd402b8fd4672b692400fe5f9eecd55d2794ac, otherwise lockdep triggers other warning and it is turned off too early:================================
WARNING: inconsistent lock state
5.18.0-rc5-next-20220506+ #11869 Not tainted
--------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:ffff80000aaa8478(&port_lock_key){?.+.}-{2:2}, at:msm_uart_irq+0x38/0x750{HARDIRQ-ON-W} state was registered at:lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock+0x5c/0x80 __msm_console_write+0x1ac/0x220 msm_console_write+0x48/0x60 __console_emit_next_record+0x188/0x420 printk_kthread_func+0x3a0/0x3bc kthread+0x118/0x11c ret_from_fork+0x10/0x20 irq event stamp: 12182 hardirqs last enabled at (12181): [<ffff800008e3d2a8>] cpuidle_enter_state+0xc4/0x30c stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.0-rc5-next-20220506+ #11869 Hardware name: Qualcomm Technologies, Inc. APQ 8016 SBC (DT) Call trace: dump_backtrace.part.0+0xd0/0xe0 show_stack+0x18/0x6c dump_stack_lvl+0x8c/0xb8 dump_stack+0x18/0x34 print_usage_bug.part.0+0x208/0x22c mark_lock+0x710/0x954 __lock_acquire+0x9fc/0x20cc lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock_irqsave+0x80/0xcc msm_uart_irq+0x38/0x750 __handle_irq_event_percpu+0xac/0x3d0 handle_irq_event+0x4c/0x120 handle_fasteoi_irq+0xa4/0x1a0 generic_handle_domain_irq+0x3c/0x60 gic_handle_irq+0x44/0xc4 call_on_irq_stack+0x2c/0x54 do_interrupt_handler+0x80/0x84 el1_interrupt+0x34/0x64 el1h_64_irq_handler+0x18/0x24 el1h_64_irq+0x64/0x68 cpuidle_enter_state+0xcc/0x30c cpuidle_enter+0x38/0x50 do_idle+0x22c/0x2bc cpu_startup_entry+0x28/0x30 rest_init+0x110/0x190 arch_post_acpi_subsys_init+0x0/0x18 start_kernel+0x6c4/0x704 __primary_switched+0xc0/0xc8 INIT: version 2.88 booting
Quoted 1 line
[info] Using makefile-style concurrent boot in runlevel S.
The console write() callback for the msm driver (__msm_console_write) assumes interrupts are off and is doing a spin_lock(&port->lock) rather than spin_lock_irqsave(&port->lock, flags).
The following change should address the issue:
John
drivers/tty/serial/msm_serial.c +3 -2
@@ -1599,6 +1599,7 @@ static inline struct uart_port *msm_get_port_from_line(unsigned int line)static void __msm_console_write (struct uart_port *port,const char *s,unsigned int count,bool is_uartdm) { +unsigned long flags;int i;int num_newlines =0 ;bool replaced = false;
@@ -1621,7 +1622,7 @@ static void __msm_console_write(struct uart_port *port, const char *s,else if (oops_in_progress) locked =spin_trylock (&port->lock);else -spin_lock (&port->lock); +spin_lock_irqsave (&port->lock, flags);if (is_uartdm)msm_reset_dm_count (port, count);
@@ -1660,7 +1661,7 @@ static void __msm_console_write(struct uart_port *port, const char *s, }if (locked) -spin_unlock (&port->lock); +spin_unlock_irqrestore (&port->lock, flags); } static void msm_console_write(struct console *co, const char *s,
Marek Szyprowski
Tested-byRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 69 lines · John Ogness
Quoted 67 lines · Marek Szyprowski
The Amlogic Meson related issue has been investigated and fixed:
https://lore.kernel.org/all/[email protected]/
but I just found that there is one more issue.
It appears on QCom-based DragonBoard 410c SBC (
arch/arm64/boot/dts/qcom/apq8016-sbc.dts). To see it on today's linux next-20220506, one has to revert42cd402b8fd4672b692400fe5f9eecd55d2794ac, otherwise lockdep triggers other warning and it is turned off too early:================================
WARNING: inconsistent lock state
5.18.0-rc5-next-20220506+ #11869 Not tainted
--------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:ffff80000aaa8478(&port_lock_key){?.+.}-{2:2}, at:msm_uart_irq+0x38/0x750{HARDIRQ-ON-W} state was registered at:lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock+0x5c/0x80 __msm_console_write+0x1ac/0x220 msm_console_write+0x48/0x60 __console_emit_next_record+0x188/0x420 printk_kthread_func+0x3a0/0x3bc kthread+0x118/0x11c ret_from_fork+0x10/0x20 irq event stamp: 12182 hardirqs last enabled at (12181): [<ffff800008e3d2a8>] cpuidle_enter_state+0xc4/0x30c stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.0-rc5-next-20220506+ #11869 Hardware name: Qualcomm Technologies, Inc. APQ 8016 SBC (DT) Call trace: dump_backtrace.part.0+0xd0/0xe0 show_stack+0x18/0x6c dump_stack_lvl+0x8c/0xb8 dump_stack+0x18/0x34 print_usage_bug.part.0+0x208/0x22c mark_lock+0x710/0x954 __lock_acquire+0x9fc/0x20cc lock_acquire.part.0+0xe0/0x230 lock_acquire+0x68/0x84 _raw_spin_lock_irqsave+0x80/0xcc msm_uart_irq+0x38/0x750 __handle_irq_event_percpu+0xac/0x3d0 handle_irq_event+0x4c/0x120 handle_fasteoi_irq+0xa4/0x1a0 generic_handle_domain_irq+0x3c/0x60 gic_handle_irq+0x44/0xc4 call_on_irq_stack+0x2c/0x54 do_interrupt_handler+0x80/0x84 el1_interrupt+0x34/0x64 el1h_64_irq_handler+0x18/0x24 el1h_64_irq+0x64/0x68 cpuidle_enter_state+0xcc/0x30c cpuidle_enter+0x38/0x50 do_idle+0x22c/0x2bc cpu_startup_entry+0x28/0x30 rest_init+0x110/0x190 arch_post_acpi_subsys_init+0x0/0x18 start_kernel+0x6c4/0x704 __primary_switched+0xc0/0xc8 INIT: version 2.88 bootingQuoted 1 line
[info] Using makefile-style concurrent boot in runlevel S.
Quoted 5 lines
The console
write()callback for the msm driver (__msm_console_write) assumes interrupts are off and is doing aspin_lock(&port->lock)rather thanspin_lock_irqsave(&port->lock, flags).The following change should address the issue:
Right, this help. spin_trylock() should be also converted imho, see below.
Quoted 15 lines
John
drivers/tty/serial/msm_serial.c +1
-0@@ -1599,6 +1599,7 @@ static inline struct uart_port *msm_get_port_from_line(unsigned int line)static void __msm_console_write (struct uart_port *port,const char *s,unsigned int count,bool is_uartdm) { +unsigned long flags;int i;int num_newlines =0 ;bool replaced = false;
drivers/tty/serial/msm_serial.c static void __msm_console_write(struct uart_port *port, const char *s, · line 1623
Quoted 2 lines
drivers/tty/serial/msm_serial.c +0
-0@@ -1621,7 +1622,7 @@ static void __msm_console_write(struct uart_port *port, const char *s,else if (oops_in_progress) locked =spin_trylock (&port->lock);
locked = spin_trylock_irqsave(&port->lock, flags);
Quoted 7 lines
drivers/tty/serial/msm_serial.c +1
-1else -spin_lock (&port->lock); +spin_lock_irqsave (&port->lock, flags);if (is_uartdm)msm_reset_dm_count (port, count);
drivers/tty/serial/msm_serial.c static void __msm_console_write(struct uart_port *port, const char *s, · line 1667
Quoted 8 lines
drivers/tty/serial/msm_serial.c +1
-1@@ -1660,7 +1661,7 @@ static void __msm_console_write(struct uart_port *port, const char *s, }if (locked) -spin_unlock (&port->lock); +spin_unlock_irqrestore (&port->lock, flags); } static void msm_console_write(struct console *co, const char *s,
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
In reply to John OgnessHi John,
Quoted 6 lines · John Ogness
Currently threaded console printers synchronize against each other using
console_lock(). However, different console drivers are unrelated and do not require any synchronization between each other. Removing the synchronization between the threaded console printers will allow each console to print at its own speed.
[...]
Quoted 1 line
Signed-off-by John Ogness <[email protected]>
Thanks for your patch, which is now commit 8e274732115f63c1 ("printk: extend console_lock for per-console locking") in v5.19-rc1.
I have bisected another intriguing issue to this commit: on SiPEED MAiX BiT (Canaan K210 riscv), it no longer prints the line detecting ttySIF0, i.e. the console output changes like:
spi-nor spi1.0: gd25lq128d (16384 Kbytes)
i2c_dev: i2c /dev entries driver
k210-fpioa 502b0000.pinmux: K210 FPIOA pin controller
-38000000.serial: ttySIF0 at MMIO 0x38000000 (irq = 1, base_baud =115200) is a SiFive UART v0
printk: console [ttySIF0] enabled
printk: bootconsole [sifive0] disabled
printk: console [ttySIF0] printing thread started
As this patch does not make any changes to drivers/tty/, and ttySIF0 does work (it's the console), I looked in /proc/kmsg, and bingo, the missing line is there, so it is generated, but never printed.
I tried taking the port spinlock in sifive_serial_startup(), as suggested for the meson driver, but that doesn't make a difference.
Do you have a clue?
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
John Ogness
maintainerRe: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Quoted 8 lines · Geert Uytterhoeven
I have bisected another intriguing issue to this commit: on SiPEED MAiX BiT (Canaan K210 riscv), it no longer prints the line detecting ttySIF0, i.e. the console output changes like:
spi-nor spi1.0: gd25lq128d (16384 Kbytes) i2c_dev: i2c /dev entries driver k210-fpioa 502b0000.pinmux: K210 FPIOA pin controller -38000000.serial: ttySIF0 at MMIO 0x38000000 (irq = 1, base_baud =
Quoted 8 lines
115200) is a SiFive UART v0
printk: console [ttySIF0] enabled
printk: bootconsole [sifive0] disabled
printk: console [ttySIF0] printing thread startedAs this patch does not make any changes to
drivers/tty/, and ttySIF0 does work (it's the console), I looked in/proc/kmsg, and bingo, the missing line is there, so it is generated, but never printed.
What is sifive0? Are you using the earlycon driver to create an early boot console? Can I see your boot args?
There is a known issue the that earlycon does not synchronize with normal consoles. A patch was recently posted [0] on LKML.
Quoted 2 lines
I tried taking the port spinlock in
sifive_serial_startup(), as suggested for the meson driver, but that doesn't make a difference.
It may not have made a difference for you, but it should be there. sifive_serial_startup() is writing to SIFIVE_SERIAL_IE_OFFS without taking port->lock. sifive_serial_console_write() also writes to this register (under port->lock). This could lead to RX watermark interrupts being disabled for some time. The problem is not as bad as it was with the meson driver because __ssp_enable_rxwm() is updating the shadow copy. But still, it is a bug. And anyway we shouldn't have 2 CPUs writing to a register simultaneously.
John Ogness
Re: [PATCH printk v5 1/1] printk: extend console_lock for per-console locking
Hi John,
Quoted 14 lines · John Ogness
Quoted 13 lines · Geert Uytterhoeven
I have bisected another intriguing issue to this commit: on SiPEED MAiX BiT (Canaan K210 riscv), it no longer prints the line detecting ttySIF0, i.e. the console output changes like:
spi-nor spi1.0: gd25lq128d (16384 Kbytes) i2c_dev: i2c /dev entries driver k210-fpioa 502b0000.pinmux: K210 FPIOA pin controller -38000000.serial: ttySIF0 at MMIO 0x38000000 (irq = 1, base_baud =115200) is a SiFive UART v0
printk: console [ttySIF0] enabled
printk: bootconsole [sifive0] disabled
printk: console [ttySIF0] printing thread started
Quoted 6 lines
Quoted 3 lines
As this patch does not make any changes to
drivers/tty/, and ttySIF0 does work (it's the console), I looked in/proc/kmsg, and bingo, the missing line is there, so it is generated, but never printed.What is sifive0? Are you using the earlycon driver to create an early boot console? Can I see your boot args?
earlycon: sifive0 at MMIO 0x0000000038000000 (options '115200n8')
printk: bootconsole [sifive0] enabled
Kernel command line: earlycon console=ttySIF0 rootdelay=2 root=/dev/mmcblk0p1 ro
Quoted 4 lines
There is a known issue the that earlycon does not synchronize with normal consoles. A patch was recently posted [0] on LKML.
Thank you, that did the trick!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
John Ogness
maintainerPATCH v4 15/15 printk: remove @console_locked
In reply to John OgnessThe static global variable @console_locked is used to help debug VT code to make sure that certain code paths are running with the console_lock held. However, this information is also available with the static global variable @console_kthreads_blocked (for locking via console_lock()), and the static global variable @console_kthreads_active (for locking via console_trylock()).
Remove @console_locked and update is_console_locked() to use the alternative variables.
--- kernel/printk/printk.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)
kernel/printk/printk.c +14 -15
@@ -340,15 +340,7 @@ static void console_kthreads_unblock(void) console_kthreads_blocked = false; } -/* -* This is used for debugging the mess that is the VT code by -* keeping track if we have the console semaphore held. It's -* definitely not the perfect debug tool (we don't know if _WE_ -* hold it and are racing, but it helps tracking those weird code -* paths in the console code where we end up in places I want -* locked without the console semaphore held). -*/ -static int console_locked, console_suspended; +static int console_suspended;/* * Array of consoles built from command line options (console=)
@@ -2719,7 +2711,6 @@ void console_lock(void)if (console_suspended)return ;console_kthreads_block (); - console_locked =1 ; console_may_schedule =1 ; }EXPORT_SYMBOL (console_lock);
@@ -2744,15 +2735,26 @@ int console_trylock(void)up_console_sem ();return 0 ; } - console_locked =1 ; console_may_schedule =0 ;return 1 ; }EXPORT_SYMBOL (console_trylock); +/* +* This is used to help to make sure that certain paths within the VT code are +* running with the console lock held. It is definitely not the perfect debug +* tool (it is not known if the VT code is the task holding the console lock), +* but it helps tracking those weird code paths in the console code such as +* when the console is suspended: where the console is not locked but no +* console printing may occur. +* +* Note: This returns true when the console is suspended but is not locked. +* This is intentional because the VT code must consider that situation +* the same as if the console was locked. +*/ int is_console_locked (void ) { -return (console_locked ||atomic_read (&console_kthreads_active)); +return (console_kthreads_blocked ||atomic_read (&console_kthreads_active)); }EXPORT_SYMBOL (is_console_locked);
@@ -2808,8 +2810,6 @@ static inline bool console_is_usable(struct console *con)static void __console_unlock (void ) { - console_locked =0 ; -/* * Depending on whether console_lock() or console_trylock() was used, * appropriately allow the kthread printers to continue.
@@ -3127,7 +3127,6 @@ void console_unblank(void) }else console_lock (); - console_locked =1 ; console_may_schedule =0 ;for_each_console (c)if ((c->flags &CON_ENABLED ) && c->unblank)
Signature · 1 line
2.30.2
Petr Mladek
maintainerQuoted 23 lines · John Ogness
This is v4 of a series to implement a kthread for each registered console. v3 is here [0]. The kthreads locklessly retrieve the records from the printk ringbuffer and also do not cause any lock contention between each other. This allows consoles to run at full speed. For example, a netconsole is able to dump records much faster than a serial or vt console. Also, during normal operation,
printk()callers are completely decoupled from console printing.There are situations where kthread printing is not sufficient. For example, during panic situations, where the kthreads may not get a chance to schedule. In such cases, the current method of attempting to print directly within the
printk()caller context is used. New functionsprintk_prefer_direct_enter()andprintk_prefer_direct_exit()are made available to mark areas of the kernel where direct printing is preferred. (These should only be areas that do not occur during normal operation.)This series also introduces
pr_flush(): amight_sleep()function that will block until all active printing threads have caught up to the latest record at the time of thepr_flush()call. This function is useful, for example, to wait until pending records are flushed to consoles before suspending.
Quoted 5 lines
Note that this series does *not* increase the reliability of console printing. Rather it focuses on the non-interference aspect of
printk()by decouplingprintk()callers from printing (during normal operation). Nonetheless, the reliability aspect should not worsen due to this series.
This version looks good enough for linux-next. I do not see any functional problem and it should work as designed. It is time to see how it works in various "real life" work loads.
I am going to push it later today unless anyone (John) complains ;-)
Best Regards,
Petr
Petr Mladek
maintainerQuoted 30 lines · Petr Mladek
Quoted 23 lines · John Ogness
This is v4 of a series to implement a kthread for each registered console. v3 is here [0]. The kthreads locklessly retrieve the records from the printk ringbuffer and also do not cause any lock contention between each other. This allows consoles to run at full speed. For example, a netconsole is able to dump records much faster than a serial or vt console. Also, during normal operation,
printk()callers are completely decoupled from console printing.There are situations where kthread printing is not sufficient. For example, during panic situations, where the kthreads may not get a chance to schedule. In such cases, the current method of attempting to print directly within the
printk()caller context is used. New functionsprintk_prefer_direct_enter()andprintk_prefer_direct_exit()are made available to mark areas of the kernel where direct printing is preferred. (These should only be areas that do not occur during normal operation.)This series also introduces
pr_flush(): amight_sleep()function that will block until all active printing threads have caught up to the latest record at the time of thepr_flush()call. This function is useful, for example, to wait until pending records are flushed to consoles before suspending.Quoted 5 lines
Note that this series does *not* increase the reliability of console printing. Rather it focuses on the non-interference aspect of
printk()by decouplingprintk()callers from printing (during normal operation). Nonetheless, the reliability aspect should not worsen due to this series.
Quoted 5 lines
This version looks good enough for linux-next. I do not see any functional problem and it should work as designed. It is time to see how it works in various "real life" work loads.
I am going to push it later today unless anyone (John) complains ;-)
I have pushed the patchset into printk/linux.git, branch rework/kthreads. Also I merged it into for-next branch.
We are still discussing better solution of the complicated locking scheme[0]. The main purpose is to make it easier and more safe to use. Anyway, the current code looks safe. Any potential improvements should not affect the behavior.
So, it is time to test it in linux-next. Let's see how survives hammering of various robots and people testing on linux-next. I keep my fingers crossed.
Best Regards,
Petr