Bcc:
Subject: Re: [PATCH] printk: avoid livelock if another CPU printks
continuously
Reply-To:
In-Reply-To: <[email protected]>
Quoted 15 lines · Denys Vlasenko
At the end of each
printk(), kernel attempts to takeconsole_sem. If this succeeds, it feeds buffered message data to console devices until there is nothing left, and releasesconsole_sem:if (console_trylock_for_printk(this_cpu)) console_unlock();The livelock exists because code in
console_unlock()has no limit on the amount of buffered data it would process underconsole_sem. This is bad ifprintk()was called with IRQs disabled.This patch makes
console_unlock()releaseconsole_semafter 5 iterations, which usually amounts to 5 lines of printk messages, and give other printk'ing CPUs a chance to acquireconsole_sem.
Quoted 5 lines
If some CPU grabs it,
console_unlock()finishes.
If no one takes the semaphore,console_unlock()re-acquires it and loops back for another cycle of console output.This seems to be a hard-to-trigger, but long-existing problem:
Yup, and there are more people trying to handle this. I add some of them into CC.
Sadly, the problem is much more complicated that it looks. Jan Kara (jack) has already provided many possible solutions that were not accepted. The last one can be seen at
http://thread.gmane.org/gmane.linux.kernel/2105183/focus=2113787
See below some comments to your approach.
Quoted 44 lines
Signed-off-by Denys Vlasenko <[email protected]>CC Steven Rostedt <[email protected]>CC Tejun Heo <[email protected]>CC Peter Hurley <[email protected]>--- kernel/printk/printk.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)kernel/printk/printk.c +5
-0@@ -2235,6 +2235,7 @@ void console_unlock(void)unsigned long flags;bool wake_klogd = false;bool do_cond_resched, retry; +unsigned cnt;if (console_suspended) {up_console_sem ();@@ -2257,6 +2258,7 @@ void console_unlock(void)/* flush buffered message fragment immediately to console */ console_cont_flush (text,sizeof (text)); again: + cnt =5 ;for (;;) {struct printk_log *msg;size_t ext_len =0 ;@@ -2284,6 +2286,9 @@ skip:if (console_seq == log_next_seq)break ; +if (--cnt ==0 ) +break ;/* Someone else printk's like crazy */ + msg =log_from_idx (console_idx);if (msg->flags &LOG_NOCONS ) {/* @@ -2350,6 +2355,26 @@ skip:if (retry &&console_trylock ()) goto again;
kernel/printk/printk.c skip: · line 2367
Quoted 10 lines
kernel/printk/printk.c +10
-0+if (cnt ==0 ) { +/* +* Other CPU(s) printk like crazy, filling log_buf[]. +* Try to get rid of the "honor" of servicing their data: +* give _them_ time to grab console_sem and start working. +*/ + cnt =9999 ; +while (--cnt !=0 ) { +cpu_relax (); +if (console_seq == log_next_seq) {
This condition is true when all available messages are printed to the console. It means that there is nothing to do at all. It is quite late. A much better solution would be to store console_seq to a local variable and check it is being modified by an other CPU.
kernel/printk/printk.c skip: · line 2375
Quoted 8 lines
kernel/printk/printk.c +8
-0+/* Good, other CPU entered "for(;;)" loop */ + goto out; + } + } +/* No one seems to be willing to take it... */ +if (console_trylock ()) + goto again;/* we took it */ +/* Nope, someone else holds console_sem! Good */
The cycle gives a big chance other CPUs to enter console_unlock(). It means that more CPUs might end up in the above busy cycle.
It gives a chance to move the printing to another CPU. It likely slows down the flood of messages because the producer end up here as well.
So, it probably works but the performance is far from optimal. Many CPUs might end up doing nothing. I am afraid that this is not the right way to go.
Best Regards,
Petr
Quoted 6 lines · On Wed, 10 Feb 2016 15:36:49 +0100 Petr Mladek <[email protected]> wrote:
Bcc:
Subject: Re: [PATCH] printk: avoid livelock if another CPU printks
continuously
Reply-To:
In-Reply-To: <[email protected]>
Hmm, playing with mail headers?
Quoted 7 lines
Quoted 7 lines
(quoted diff) +7
-0+ if (cnt == 0) { + /* + * Other CPU(s) printk like crazy, filling log_buf[]. + * Try to get rid of the "honor" of servicing their data: + * give _them_ time to grab console_sem and start working. + */ + cnt = 9999;
Quoted 9 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ while (--cnt != 0) { + cpu_relax(); + if (console_seq == log_next_seq) {This condition is true when all available messages are printed to the console. It means that there is nothing to do at all. It is quite late. A much better solution would be to store
console_seqto a local variable and check it is being modified by an other CPU.
Yep, I recommended the same thing.
Quoted 13 lines
Review
Quoted 9 lines
(quoted diff) +8
-0+ /* Good, other CPU entered "for(;;)" loop */ + goto out; + } + } + /* No one seems to be willing to take it... */ + if (console_trylock()) + goto again; /* we took it */ + /* Nope, someone else holds console_sem! Good */The cycle gives a big chance other CPUs to enter
console_unlock(). It means that more CPUs might end up in the above busy cycle.
Quoted 7 lines
It gives a chance to move the printing to another CPU. It likely slows down the flood of messages because the producer end up here as well.
So, it probably works but the performance is far from optimal. Many CPUs might end up doing nothing. I am afraid that this is not the right way to go.
Note, it's not that performance critical, and the loop only happens if someone else is adding to the console, which hopefully, should be rare.
-- Steve