Skip to contentSkip to conversation

printk: Start printing handover kthreads on demand

PATCH Sergey Senozhatsky 1 participant 2 messages Dec 10, 2015 Printk

lore
Share & tools
lore
#0 lore raw

Review

Quoted 10 lines
(quoted diff) +10 -0
+static void printk_offload_init(void)
+{
+	mutex_lock(&printk_kthread_mutex);
+	if (num_possible_cpus() <= 1) {
+		/* Offloading doesn't make sense. Disable print offloading. */
+		printk_offload_chars = 0;
+	} else
+		printk_start_offload_kthreads();
+	mutex_unlock(&printk_kthread_mutex);
+}

A silly minor nitpick

+	} else {
		printk_start_offload_kthreads();

+ }

	-ss
#1 lore raw
Quoted 3 lines · Sergey Senozhatsky

A silly minor nitpick

ah.. nope, I lied.

this part

Quoted 14 lines

+static int printk_start_offload_kthreads(void)
{

(quoted diff) +0 -1
-	struct console *con;
	int i;
	struct task_struct *task;
(quoted diff) +8 -0
+	/* Does handover of printing make any sense? */
+	if (printk_offload_chars == 0 || num_possible_cpus() <= 1)
+		return 0;
+	for (i = 0; i < PRINTING_TASKS; i++) {
+		if (printing_kthread[i])
+			continue;
+		task = kthread_run(printing_task, NULL, "print/%d", i);
+		if (IS_ERR(task))

Review

Quoted 10 lines
(quoted diff) +10 -0
+			goto out_err;
+		printing_kthread[i] = task;
+	}
+	return 0;
+out_err:
+	pr_err("printk: Cannot create printing thread: %ld\n", PTR_ERR(task));
+	/* Disable offloading if creating kthreads failed */
+	printk_offload_chars = 0;
+	return PTR_ERR(task);
+}

we can keep printk_offload_chars if we have created at least one thread, in theory. if we are in heavy or nearly hevy (which is 'a process will spin in unlock_console() with preemption_disabled for far too long') printk traffic case (and that's what I want to fix from my side) then we have at least 1 process doing printk->console_unlock() very often, so breaking that 'while (1)' console_unlock() loop is a good thing here. which may be a corner case.

if we want to zero `printk_offload_chars' then how about bringing back Tejun Heo's "printk: do cond_resched() between lines while outputting to consoles"?

Besides, we need this for !CONFIG_PRINTK_OFFLOAD kernel.

===8<====

if printk_start_offload_kthreads() has failed or if for some other reason `printk_offload_chars' ended up to be 0, we still need to try to break up long console_unlock() loops and try to reschedule.

---
 kernel/printk/printk.c | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
kernel/printk/printk.c +26 -1
index 21b0fb9..fc8c493 100644
@@ -2285,6 +2285,16 @@ static bool cpu_stop_printing(int printed_chars)
 	return false;
 }
 
+static bool cpu_should_cond_resched(bool do_cond_resched)
+{
+	/* Oops? Print everything now to maximize chances user will see it */
+	if (oops_in_progress)
+		return false;
+	if (!printk_offload_chars && do_cond_resched)
+		return true;
+	return false;
+}
+
 /**
  * console_unlock - unlock the console system
  *
@@ -2309,7 +2319,7 @@ void console_unlock(void)
 	unsigned long flags;
 	bool wake_klogd = false;
 	bool retry;
-	bool hand_over = false;
+	bool hand_over = false, do_cond_resched;
 	int printed_chars = 0;
 
 	if (console_suspended) {
@@ -2317,6 +2327,15 @@ void console_unlock(void)
 		return;
 	}
 
+	/*
+	 * We may end up dumping a lot of lines, for example, if called
+	 * from console registration path, and should invoke cond_resched()
+	 * between lines if allowable.  Not doing so can cause a very long
+	 * scheduling stall on a slow console leading to RCU stall and
+	 * softlockup warnings which exacerbate the issue with more
+	 * messages practically incapacitating the system.
+	 */
+	do_cond_resched = console_may_schedule;
 	console_may_schedule = 0;
 
 	/* flush buffered message fragment immediately to console */
@@ -2397,6 +2416,12 @@ skip:
 		call_console_drivers(level, ext_text, ext_len, text, len);
 		start_critical_timings();
 		printed_chars += len;
+
+		if (unlikely(cpu_should_cond_resched(do_cond_resched))) {
+			raw_spin_unlock_irqrestore(&print_lock, flags);
+			cond_resched();
+			raw_spin_lock_irqsave(&print_lock, flags);
+		}
 	}
 
 	/* Release the exclusive_console once it is used */
Signature · 1 line
2.6.3