Skip to contentSkip to conversation

DISCUSSION Petr Mladek 2 participants 2 messages Feb 10, 2016

lore
Share & tools
lore
#0 lore raw

Bcc:
Subject: Re: [PATCH] printk: avoid livelock if another CPU printks
continuously
Reply-To:
In-Reply-To: <[email protected]>

Quoted 15 lines · Denys Vlasenko

At the end of each printk(), kernel attempts to take console_sem. If this succeeds, it feeds buffered message data to console devices until there is nothing left, and releases console_sem:

        if (console_trylock_for_printk(this_cpu))
                console_unlock();

The livelock exists because code in console_unlock() has no limit on the amount of buffered data it would process under console_sem. This is bad if printk() was called with IRQs disabled.

This patch makes console_unlock() release console_sem after 5 iterations, which usually amounts to 5 lines of printk messages, and give other printk'ing CPUs a chance to acquire console_sem.

Quoted 5 lines

If some CPU grabs it, console_unlock() finishes.
If no one takes the semaphore, console_unlock() re-acquires it and loops back for another cycle of console output.

This seems to be a hard-to-trigger, but long-existing problem:

Yup, and there are more people trying to handle this. I add some of them into CC.

Sadly, the problem is much more complicated that it looks. Jan Kara (jack) has already provided many possible solutions that were not accepted. The last one can be seen at
http://thread.gmane.org/gmane.linux.kernel/2105183/focus=2113787

See below some comments to your approach.

Quoted 44 lines
---
 kernel/printk/printk.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
kernel/printk/printk.c +5 -0
index c963ba5..ca4f9d55 100644
@@ -2235,6 +2235,7 @@ void console_unlock(void)
 	unsigned long flags;
 	bool wake_klogd = false;
 	bool do_cond_resched, retry;
+	unsigned cnt;
 
 	if (console_suspended) {
 		up_console_sem();
@@ -2257,6 +2258,7 @@ void console_unlock(void)
 	/* flush buffered message fragment immediately to console */
 	console_cont_flush(text, sizeof(text));
 again:
+	cnt = 5;
 	for (;;) {
 		struct printk_log *msg;
 		size_t ext_len = 0;
@@ -2284,6 +2286,9 @@ skip:
 		if (console_seq == log_next_seq)
 			break;
 
+		if (--cnt == 0)
+			break;	/* Someone else printk's like crazy */
+
 		msg = log_from_idx(console_idx);
 		if (msg->flags & LOG_NOCONS) {
 			/*
@@ -2350,6 +2355,26 @@ skip:
 	if (retry && console_trylock())
 		goto again;
 

kernel/printk/printk.c skip: · line 2367

Quoted 10 lines
kernel/printk/printk.c +10 -0
+	if (cnt == 0) {
+		/*
+		 * Other CPU(s) printk like crazy, filling log_buf[].
+		 * Try to get rid of the "honor" of servicing their data:
+		 * give _them_ time to grab console_sem and start working.
+		 */
+		cnt = 9999;
+		while (--cnt != 0) {
+			cpu_relax();
+			if (console_seq == log_next_seq) {

This condition is true when all available messages are printed to the console. It means that there is nothing to do at all. It is quite late. A much better solution would be to store console_seq to a local variable and check it is being modified by an other CPU.

kernel/printk/printk.c skip: · line 2375

Quoted 8 lines
kernel/printk/printk.c +8 -0
+				/* Good, other CPU entered "for(;;)" loop */
+				goto out;
+			}
+		}
+		/* No one seems to be willing to take it... */
+		if (console_trylock())
+			goto again; /* we took it */
+		/* Nope, someone else holds console_sem! Good */

The cycle gives a big chance other CPUs to enter console_unlock(). It means that more CPUs might end up in the above busy cycle.

It gives a chance to move the printing to another CPU. It likely slows down the flood of messages because the producer end up here as well.

So, it probably works but the performance is far from optimal. Many CPUs might end up doing nothing. I am afraid that this is not the right way to go.

Best Regards,
Petr

#1 lore raw
Quoted 6 lines · On Wed, 10 Feb 2016 15:36:49 +0100 Petr Mladek <[email protected]> wrote:

Bcc:
Subject: Re: [PATCH] printk: avoid livelock if another CPU printks
continuously
Reply-To:
In-Reply-To: <[email protected]>

Hmm, playing with mail headers?

Quoted 7 lines
Quoted 7 lines
(quoted diff) +7 -0
+	if (cnt == 0) {
+		/*
+		 * Other CPU(s) printk like crazy, filling log_buf[].
+		 * Try to get rid of the "honor" of servicing their data:
+		 * give _them_ time to grab console_sem and start working.
+		 */
+		cnt = 9999;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		while (--cnt != 0) {
+			cpu_relax();
+			if (console_seq == log_next_seq) {  

This condition is true when all available messages are printed to the console. It means that there is nothing to do at all. It is quite late. A much better solution would be to store console_seq to a local variable and check it is being modified by an other CPU.

Yep, I recommended the same thing.

Quoted 13 lines

Review

Quoted 9 lines
(quoted diff) +8 -0
+				/* Good, other CPU entered "for(;;)" loop */
+				goto out;
+			}
+		}
+		/* No one seems to be willing to take it... */
+		if (console_trylock())
+			goto again; /* we took it */
+		/* Nope, someone else holds console_sem! Good */  

The cycle gives a big chance other CPUs to enter console_unlock(). It means that more CPUs might end up in the above busy cycle.

Quoted 7 lines

It gives a chance to move the printing to another CPU. It likely slows down the flood of messages because the producer end up here as well.

So, it probably works but the performance is far from optimal. Many CPUs might end up doing nothing. I am afraid that this is not the right way to go.

Note, it's not that performance critical, and the loop only happens if someone else is adding to the console, which hopefully, should be rare.

-- Steve