Skip to contentSkip to conversation

make printk work again

PATCH In review Peter Zijlstra 5 participants 17 messages Oct 18, 2016 – Nov 29, 2016 Printk

lore
Share & tools
lore
#0 lore raw

cover make printk work again

This basically fixes printk by evading everything it does.

There's too many problems with printk, from sleeping locks to broken console drivers. Stop using it.

The early_console drivers are the most robust and simple, use those.

#1 lore raw

PATCH 1/3 printk: Fix kdb_trap_printk placement

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

---
 kernel/printk/printk.c |   19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
kernel/printk/printk.c +8 -11
@@ -1750,6 +1750,13 @@ asmlinkage int vprintk_emit(int facility
 	/* cpu currently holding logbuf_lock in this function */
 	static unsigned int logbuf_cpu = UINT_MAX;
 
+#ifdef CONFIG_KGDB_KDB
+	if (unlikely(kdb_trap_printk)) {
+		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
+		return r;
+	}
+#endif
+
 	if (level == LOGLEVEL_SCHED) {
 		level = LOGLEVEL_DEFAULT;
 		in_sched = true;
@@ -1932,17 +1939,7 @@ EXPORT_SYMBOL(printk_emit);
 
 int vprintk_default(const char *fmt, va_list args)
 {
-	int r;
-
-#ifdef CONFIG_KGDB_KDB
-	if (unlikely(kdb_trap_printk)) {
-		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
-		return r;
-	}
-#endif
-	r = vprintk_emit(0, LOGLEVEL_DEFAULT, NULL, 0, fmt, args);
-
-	return r;
+	return vprintk_emit(0, LOGLEVEL_DEFAULT, NULL, 0, fmt, args);
 }
 EXPORT_SYMBOL_GPL(vprintk_default);
 
maintainer Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] printk: Fix kdb_trap_printk placement

Quoted 5 lines · Peter Zijlstra

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

Good question! vkdb_printf() tries to avoid a deadlock but the code is racy:

int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap) {
[...]

	/* Serialize kdb_printf if multiple cpus try to write at once.
	 * But if any cpu goes recursive in kdb, just print the output,
	 * even if it is interleaved with any other text.
	 */
	if (!KDB_STATE(PRINTF_LOCK)) {
		KDB_STATE_SET(PRINTF_LOCK);
		spin_lock_irqsave(&kdb_printf_lock, flags);
		got_printf_lock = 1;
		atomic_inc(&kdb_event);
	} else {
		__acquire(kdb_printf_lock);
	}

Let's have the following situation:

CPU1 CPU2

if (!KDB_STATE(PRINTF_LOCK)) {
	KDB_STATE_SET(PRINTF_LOCK);

					if (!KDB_STATE(PRINTF_LOCK)) {
					} else {
						__acquire(kdb_printf_lock);
					}

Now, both CPUs are in the critical section and happily writing over each other, e.g. in

	vsnprintf(next_avail, size_avail, fmt, ap);

I quess that we want to fix this race. But I am not sure if it will be done an NMI-safe way. I am going to send a patch for this.

Well, vkdb_printf() is called later when the messages are pushed to the main logbuffer by printk_nmi_flush_line(). It is not perfect but...

Quoted 2 lines

Otherwise, your patch makes sense:

Best Regards,
Petr

#3 lore raw

Re: [PATCH 1/3] printk: Fix kdb_trap_printk placement

Quoted 38 lines · Petr Mladek
Quoted 5 lines · Peter Zijlstra

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

Good question! vkdb_printf() tries to avoid a deadlock but the code is racy:

int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap) {
[...]

	/* Serialize kdb_printf if multiple cpus try to write at once.
	 * But if any cpu goes recursive in kdb, just print the output,
	 * even if it is interleaved with any other text.
	 */
	if (!KDB_STATE(PRINTF_LOCK)) {
		KDB_STATE_SET(PRINTF_LOCK);
		spin_lock_irqsave(&kdb_printf_lock, flags);
		got_printf_lock = 1;
		atomic_inc(&kdb_event);
	} else {
		__acquire(kdb_printf_lock);
	}

Let's have the following situation:

CPU1 CPU2

if (!KDB_STATE(PRINTF_LOCK)) {
	KDB_STATE_SET(PRINTF_LOCK);

					if (!KDB_STATE(PRINTF_LOCK)) {
					} else {
						__acquire(kdb_printf_lock);
					}
Quoted 7 lines

Now, both CPUs are in the critical section and happily writing over each other, e.g. in

	vsnprintf(next_avail, size_avail, fmt, ap);

I quess that we want to fix this race. But I am not sure if it will be done an NMI-safe way. I am going to send a patch for this.

Something like patch 3 in this series should do I suppose. But the vkdb_printf() thing using spin_lock_irqsave() seems to suggest it was never meant to be used from NMI context.

Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] printk: Fix kdb_trap_printk placement

In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

looks good to me.

	-ss
maintainer Reviewed-by Sergey Senozhatsky Reviewed-by on v1 1/3
#5 lore raw

Re: [PATCH 1/3] printk: Fix kdb_trap_printk placement

In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

I think that nobody saw this problem because kdb_trap_printk was used only for a limited number of printk's. It is just a trick how to use generic functions to print messages in the kdb context, e.g. for getting backtraces.

But the patch makes sense.

Quoted 2 lines

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

It is perfectly fine. Messages from NMI context are not meant for kdb anyway.

Quoted 9 lines
---
 kernel/printk/printk.c |   19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)

kernel/printk/printk.c asmlinkage int vprintk_emit(int facility · line 1756

Quoted 7 lines
kernel/printk/printk.c +4 -0
@@ -1750,6 +1750,13 @@ asmlinkage int vprintk_emit(int facility
 	/* cpu currently holding logbuf_lock in this function */
 	static unsigned int logbuf_cpu = UINT_MAX;
 
+#ifdef CONFIG_KGDB_KDB
+	if (unlikely(kdb_trap_printk)) {
+		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
+		return r;

r is not defined. It is fixed in the next patch but it breaks bisectability.

Please, find below an updated patch that also includes my Reviewed-by.

Quoted 1 line

From 5adf18de45ba986ea3ae611446828238f4d65fe0 Mon Sep 17 00:00:00 2001

From: Peter Zijlstra <[email protected]>
Date: Tue, 18 Oct 2016 19:08:31 +0200
Subject: [PATCH 1/3] printk: Fix kdb_trap_printk placement

Some people figured vprintk_emit() makes for a nice API and exported it, bypassing the kdb trap.

This still leaves vprintk_nmi() outside of the kbd reach, should that be fixed too?

---
 kernel/printk/printk.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
kernel/printk/printk.c +6 -11
index f7a55e9ff2f7..541ce7705353 100644
@@ -1781,6 +1781,11 @@ asmlinkage int vprintk_emit(int facility, int level,
 	/* cpu currently holding logbuf_lock in this function */
 	static unsigned int logbuf_cpu = UINT_MAX;
 
+#ifdef CONFIG_KGDB_KDB
+	if (unlikely(kdb_trap_printk))
+		return vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
+#endif
+
 	if (level == LOGLEVEL_SCHED) {
 		level = LOGLEVEL_DEFAULT;
 		in_sched = true;
@@ -1923,17 +1928,7 @@ asmlinkage int printk_emit(int facility, int level,
 
 int vprintk_default(const char *fmt, va_list args)
 {
-	int r;
-
-#ifdef CONFIG_KGDB_KDB
-	if (unlikely(kdb_trap_printk)) {
-		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
-		return r;
-	}
-#endif
-	r = vprintk_emit(0, LOGLEVEL_DEFAULT, NULL, 0, fmt, args);
-
-	return r;
+	return vprintk_emit(0, LOGLEVEL_DEFAULT, NULL, 0, fmt, args);
 }
 EXPORT_SYMBOL_GPL(vprintk_default);
 
Signature · 1 line
1.8.5.6
#6 lore raw

PATCH 2/3 early_printk: Add force_early_printk kernel parameter

In reply to Peter Zijlstra

Add add the 'force_early_printk' kernel parameter to override printk() and force it into early_printk(). This bypasses all the cruft and fail from printk() and makes things work again.

---
 kernel/printk/printk.c |   74 ++++++++++++++++++++++++++++++++-----------------
 1 file changed, 49 insertions(+), 25 deletions(-)
kernel/printk/printk.c +49 -25
@@ -344,6 +344,42 @@ __packed __aligned(4)
 #endif
 ;
 
+#ifdef CONFIG_EARLY_PRINTK
+struct console *early_console;
+
+static bool __read_mostly force_early_printk;
+
+static int __init force_early_printk_setup(char *str)
+{
+	force_early_printk = true;
+	return 0;
+}
+early_param("force_early_printk", force_early_printk_setup);
+
+static int early_vprintk(const char *fmt, va_list args)
+{
+	char buf[512];
+	int n;
+
+	n = vscnprintf(buf, sizeof(buf), fmt, args);
+	early_console->write(early_console, buf, n);
+
+	return n;
+}
+
+asmlinkage __visible void early_printk(const char *fmt, ...)
+{
+	va_list ap;
+
+	if (!early_console)
+		return;
+
+	va_start(ap, fmt);
+	early_vprintk(fmt, ap);
+	va_end(ap);
+}
+#endif
+
 /*
  * The logbuf_lock protects kmsg buffer, indices, counters.  This can be taken
  * within the scheduler's rq lock. It must be released before calling
@@ -1751,10 +1787,13 @@ asmlinkage int vprintk_emit(int facility
 	static unsigned int logbuf_cpu = UINT_MAX;
 
 #ifdef CONFIG_KGDB_KDB
-	if (unlikely(kdb_trap_printk)) {
-		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
-		return r;
-	}
+	if (unlikely(kdb_trap_printk))
+		return vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
+#endif
+
+#ifdef CONFIG_EARLY_PRINTK
+	if (force_early_printk && early_console)
+		return early_vprintk(fmt, args);
 #endif
 
 	if (level == LOGLEVEL_SCHED) {
@@ -1970,7 +2009,12 @@ asmlinkage __visible int printk(const ch
 	int r;
 
 	va_start(args, fmt);
-	r = vprintk_func(fmt, args);
+#ifdef CONFIG_EARLY_PRINTK
+	if (force_early_printk && early_console)
+		r = vprintk_default(fmt, args);
+	else
+#endif
+		r = vprintk_func(fmt, args);
 	va_end(args);
 
 	return r;
@@ -2020,26 +2064,6 @@ DEFINE_PER_CPU(printk_func_t, printk_fun
 
 #endif /* CONFIG_PRINTK */
 
-#ifdef CONFIG_EARLY_PRINTK
-struct console *early_console;
-
-asmlinkage __visible void early_printk(const char *fmt, ...)
-{
-	va_list ap;
-	char buf[512];
-	int n;
-
-	if (!early_console)
-		return;
-
-	va_start(ap, fmt);
-	n = vscnprintf(buf, sizeof(buf), fmt, ap);
-	va_end(ap);
-
-	early_console->write(early_console, buf, n);
-}
-#endif
-
 static int __add_preferred_console(char *name, int idx, char *options,
 				   char *brl_options)
 {
maintainer Reviewed-by on v1 2/3
#7 lore raw

Re: [PATCH 2/3] early_printk: Add force_early_printk kernel parameter

Quoted 3 lines · Peter Zijlstra

Add add the 'force_early_printk' kernel parameter to override printk() and force it into early_printk(). This bypasses all the cruft and fail from printk() and makes things work again.

IMHO, the patch makes perfect sense and helps with debugging hard problems.

kernel/printk/printk.c asmlinkage int vprintk_emit(int facility · line 1792

Quoted 13 lines
kernel/printk/printk.c +3 -4
@@ -1751,10 +1787,13 @@ asmlinkage int vprintk_emit(int facility
 	static unsigned int logbuf_cpu = UINT_MAX;
 
 #ifdef CONFIG_KGDB_KDB
-	if (unlikely(kdb_trap_printk)) {
-		r = vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
-		return r;
-	}
+	if (unlikely(kdb_trap_printk))
+		return vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
+#endif

Please, find below an updated patch that has also my Reviewed-by.

In particular, the above change was moved to the first patch to fix bisectability. The result after applying both patches is still exactly the same.

Quoted 1 line

From e1c00ae67d07767ec8e5bddb1113c2badf31f4bd Mon Sep 17 00:00:00 2001

From: Petr Mladek <[email protected]>
Date: Tue, 29 Nov 2016 13:32:56 +0100
Subject: [PATCH 2/3] early_printk: Add force_early_printk kernel parameter

Add add the 'force_early_printk' kernel parameter to override printk() and force it into early_printk(). This bypasses all the cruft and fail from printk() and makes things work again.

---
 kernel/printk/printk.c | 68 ++++++++++++++++++++++++++++++++++----------------
 1 file changed, 47 insertions(+), 21 deletions(-)
kernel/printk/printk.c +47 -21
index 541ce7705353..bb612e5c2e00 100644
@@ -344,6 +344,42 @@ __packed __aligned(4)
 #endif
 ;
 
+#ifdef CONFIG_EARLY_PRINTK
+struct console *early_console;
+
+static bool __read_mostly force_early_printk;
+
+static int __init force_early_printk_setup(char *str)
+{
+	force_early_printk = true;
+	return 0;
+}
+early_param("force_early_printk", force_early_printk_setup);
+
+static int early_vprintk(const char *fmt, va_list args)
+{
+	char buf[512];
+	int n;
+
+	n = vscnprintf(buf, sizeof(buf), fmt, args);
+	early_console->write(early_console, buf, n);
+
+	return n;
+}
+
+asmlinkage __visible void early_printk(const char *fmt, ...)
+{
+	va_list ap;
+
+	if (!early_console)
+		return;
+
+	va_start(ap, fmt);
+	early_vprintk(fmt, ap);
+	va_end(ap);
+}
+#endif
+
 /*
  * The logbuf_lock protects kmsg buffer, indices, counters.  This can be taken
  * within the scheduler's rq lock. It must be released before calling
@@ -1786,6 +1822,11 @@ asmlinkage int vprintk_emit(int facility, int level,
 		return vkdb_printf(KDB_MSGSRC_PRINTK, fmt, args);
 #endif
 
+#ifdef CONFIG_EARLY_PRINTK
+	if (force_early_printk && early_console)
+		return early_vprintk(fmt, args);
+#endif
+
 	if (level == LOGLEVEL_SCHED) {
 		level = LOGLEVEL_DEFAULT;
 		in_sched = true;
@@ -1959,7 +2000,12 @@ asmlinkage __visible int printk(const char *fmt, ...)
 	int r;
 
 	va_start(args, fmt);
-	r = vprintk_func(fmt, args);
+#ifdef CONFIG_EARLY_PRINTK
+	if (force_early_printk && early_console)
+		r = vprintk_default(fmt, args);
+	else
+#endif
+		r = vprintk_func(fmt, args);
 	va_end(args);
 
 	return r;
@@ -2009,26 +2055,6 @@ static size_t msg_print_text(const struct printk_log *msg, enum log_flags prev,
 
 #endif /* CONFIG_PRINTK */
 
-#ifdef CONFIG_EARLY_PRINTK
-struct console *early_console;
-
-asmlinkage __visible void early_printk(const char *fmt, ...)
-{
-	va_list ap;
-	char buf[512];
-	int n;
-
-	if (!early_console)
-		return;
-
-	va_start(ap, fmt);
-	n = vscnprintf(buf, sizeof(buf), fmt, ap);
-	va_end(ap);
-
-	early_console->write(early_console, buf, n);
-}
-#endif
-
 static int __add_preferred_console(char *name, int idx, char *options,
 				   char *brl_options)
 {
Signature · 1 line
1.8.5.6
#8 lore raw

PATCH 3/3 early_printk: Add simple serialization to early_vprintk()

In reply to Peter Zijlstra

In order to avoid multiple CPUs banging on the serial port at the same time, add simple serialization. This explicitly deals with nested contexts (like IRQs etc.).

---
 kernel/printk/printk.c |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
kernel/printk/printk.c +15 -1
@@ -356,14 +356,28 @@ static int __init force_early_printk_set
 }
 early_param("force_early_printk", force_early_printk_setup);
 
+static int early_printk_cpu = -1;
+
 static int early_vprintk(const char *fmt, va_list args)
 {
+	int n, cpu, old;
 	char buf[512];
-	int n;
+
+	cpu = get_cpu();
+	for (;;) {
+		old = cmpxchg(&early_printk_cpu, -1, cpu);
+		if (old == -1 || old == cpu)
+			break;
+
+		cpu_relax();
+	}
 
 	n = vscnprintf(buf, sizeof(buf), fmt, args);
 	early_console->write(early_console, buf, n);
 
+	smp_store_release(&early_printk_cpu, old);
+	put_cpu();
+
 	return n;
 }
 
#9 lore raw

Re: [PATCH 3/3] early_printk: Add simple serialization to early_vprintk()

Quoted 6 lines · On Tue, 18 Oct 2016 19:08:33 +0200 Peter Zijlstra <[email protected]> wrote:

In order to avoid multiple CPUs banging on the serial port at the same time, add simple serialization. This explicitly deals with nested contexts (like IRQs etc.).

---

hehe, I have the exact same patch for my -rt work.

-- Steve

Quoted 36 lines
 kernel/printk/printk.c |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
kernel/printk/printk.c +15 -1
@@ -356,14 +356,28 @@ static int __init force_early_printk_set
 }
 early_param("force_early_printk", force_early_printk_setup);
 
+static int early_printk_cpu = -1;
+
 static int early_vprintk(const char *fmt, va_list args)
 {
+	int n, cpu, old;
 	char buf[512];
-	int n;
+
+	cpu = get_cpu();
+	for (;;) {
+		old = cmpxchg(&early_printk_cpu, -1, cpu);
+		if (old == -1 || old == cpu)
+			break;
+
+		cpu_relax();
+	}
 
 	n = vscnprintf(buf, sizeof(buf), fmt, args);
 	early_console->write(early_console, buf, n);
 
+	smp_store_release(&early_printk_cpu, old);
+	put_cpu();
+
 	return n;
 }
 
#10 lore raw

Re: [PATCH 3/3] early_printk: Add simple serialization to early_vprintk()

Quoted 15 lines · Steven Rostedt

hehe, I have the exact same patch for my -rt work.

Review

Quoted 12 lines
(quoted diff) +9 -0
+static int early_printk_cpu = -1;
+
 static int early_vprintk(const char *fmt, va_list args)
 {
+	int n, cpu, old;
 	char buf[512];
+
+	cpu = get_cpu();
+	for (;;) {
+		old = cmpxchg(&early_printk_cpu, -1, cpu);
+		if (old == -1 || old == cpu)
+			break;

Looking at this again, we really should not spin using cmpxchg(), that thrashes the cacheline.

The below is slightly better spinning... then again, this isn't performance code.

kernel/printk/printk.c +8 -2
@@ -376,10 +376,16 @@ static int early_vprintk(const char *fmt
 
 	cpu = get_cpu();
 	for (;;) {
-		old = cmpxchg(&early_printk_cpu, -1, cpu);
-		if (old == -1 || old == cpu)
+		old = READ_ONCE(early_printk_cpu);
+		if (old == cpu)
 			break;
 
+		if (old == -1) {
+			old = cmpxchg(&early_printk_cpu, -1, cpu);
+			if (old == -1 || old == cpu)
+				break;
+		}
+
 		cpu_relax();
 	}
 
#11 lore raw

Re: [PATCH 3/3] early_printk: Add simple serialization to early_vprintk()

Quoted 7 lines · On Tue, 18 Oct 2016 19:30:46 +0200 Peter Zijlstra <[email protected]> wrote:

On Tue, Oct 18, 2016 at 01:19:51PM -0400, Steven Rostedt wrote:

Looking at this again, we really should not spin using cmpxchg(), that thrashes the cacheline.

The below is slightly better spinning... then again, this isn't performance code.

Yeah, this is similar to the hack I had.

May want to fold this in your previous version.

-- Steve

Quoted 22 lines
kernel/printk/printk.c +8 -2
@@ -376,10 +376,16 @@ static int early_vprintk(const char *fmt
 
 	cpu = get_cpu();
 	for (;;) {
-		old = cmpxchg(&early_printk_cpu, -1, cpu);
-		if (old == -1 || old == cpu)
+		old = READ_ONCE(early_printk_cpu);
+		if (old == cpu)
 			break;
 
+		if (old == -1) {
+			old = cmpxchg(&early_printk_cpu, -1, cpu);
+			if (old == -1 || old == cpu)
+				break;
+		}
+
 		cpu_relax();
 	}
 
maintainer
#12 lore raw

Re: [PATCH 3/3] early_printk: Add simple serialization to early_vprintk()

In reply to Peter Zijlstra
Quoted 5 lines · Peter Zijlstra

In order to avoid multiple CPUs banging on the serial port at the same time, add simple serialization. This explicitly deals with nested contexts (like IRQs etc.).

Makes sense. Just a small comment below.

Quoted 7 lines
kernel/printk/printk.c +1 -0
@@ -356,14 +356,28 @@ static int __init force_early_printk_set
 }
 early_param("force_early_printk", force_early_printk_setup);
 
+static int early_printk_cpu = -1;

[...]

kernel/printk/printk.c static int __init force_early_printk_set · line 378

Quoted 12 lines
kernel/printk/printk.c +8 -0
+	for (;;) {
+		old = cmpxchg(&early_printk_cpu, -1, cpu);
+		if (old == -1 || old == cpu)
+			break;
+
+		cpu_relax();
+	}
 
 	n = vscnprintf(buf, sizeof(buf), fmt, args);
 	early_console->write(early_console, buf, n);
 
+	smp_store_release(&early_printk_cpu, old);

checkpatch.pl complains about using a barrier without a comment. The code is simple but it still might help to add something like:

	/* Releasing early_printk_cpu custom lock. */

Best Regards,
Petr

#13 lore raw
In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

This basically fixes printk by evading everything it does.

There's too many problems with printk, from sleeping locks to broken console drivers. Stop using it.

The early_console drivers are the most robust and simple, use those.

ACK

Can we also please ban that pre Because it's a bit tricky.emption thing?

	-ss
#14 lore raw
Quoted 11 lines · Sergey Senozhatsky
Quoted 6 lines · Peter Zijlstra

This basically fixes printk by evading everything it does.

There's too many problems with printk, from sleeping locks to broken console drivers. Stop using it.

The early_console drivers are the most robust and simple, use those.

ACK

Can we also please ban that pre Because it's a bit tricky.emption thing?

Ha, so that's actually quite complicated. The 3rd patch does what is simple and robust. Doing more gets very tricky real quick.

So actual preemption is disabled per get_cpu()/put_cpu(), and cross-cpu interleaving is disabled by that 'lock'. But interrupt (and NMI) nesting are still possible.

It does sporadically happen, but I've not really found it to be a problem when reading the output.

#15 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra

This basically fixes printk by evading everything it does.

There's too many problems with printk, from sleeping locks to broken console drivers. Stop using it.

I agree that printk is fragile and your patches are likely fine for machine where you do kernel development. However for production servers with hundreds of SCSI LUNs assigned I don't think it is a viable solution - I'm pretty sure those machines would take ages to boot (if they ever boot) with early_printk implementation. So do you intend this as "the ultimate printk solution" or just a "kernel developers debugging aid"? :)

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#16 lore raw
Quoted 6 lines · Jan Kara
Quoted 4 lines · Peter Zijlstra

This basically fixes printk by evading everything it does.

There's too many problems with printk, from sleeping locks to broken console drivers. Stop using it.

Quoted 6 lines

I agree that printk is fragile and your patches are likely fine for machine where you do kernel development. However for production servers with hundreds of SCSI LUNs assigned I don't think it is a viable solution - I'm pretty sure those machines would take ages to boot (if they ever boot) with early_printk implementation. So do you intend this as "the ultimate printk solution" or just a "kernel developers debugging aid"? :)

Mostly just to scratch my itch. It also completely kills dmesg, which I can't see being popular :-) On the plus side, these 3 little patches seems like something mergable (as opposed to my previous approach which basically just deleted everything printk). The first also fixes a real, and long standing, issue with kdb_printk.

But I really do not see how printk and the console drivers as exist today can ever yield something reliable. We've been stacking band-aids on it for a while now, and I think we're at the point where we should say stop this madness.

Either do a complete overhaul and redesign of the entire stack, or just give up and admit its broken crap and leave it rot.