Skip to contentSkip to conversation

panic/printk/x86: Prevent some more printk-related deadlocks in panic()

PATCH In review Petr Mladek 3 participants 17 messages Jul 16, 2019 – Jul 31, 2019

Subsystems (2)
lore
Share & tools
lore

Petr Mladek

author maintainer
#0 lore raw

cover panic/printk/x86: Prevent some more printk-related deadlocks in panic()

Hi,

I have found some spare duct tape and wrapped some more printk-related deadlocks in panic().

More seriously, someone reported a deadlock in panic(). Some non-trivial debugging pointed out a problem with the following combination:

+ x86_64 architecture
+ panic()
+ pstore configured as message dumper (kmsg_dump())
+ crash kernel configured
+ crash_kexec_post_notifiers

In this case, CPUs are stopped by crash_smp_send_stop(). It uses NMI but it does not modify cpu_online_mask. Therefore logbuf_lock might stay locked, see 2nd patch for more details.

The above is a real corner case. But similar problem seems to be even in the common situations on architectures that do not use NMI in smp_send_stop() as a fallback, see 1st patch.

Back to the duct tape. I hope that we will get rid of these problems with the lockless printk ringbuffer rather sooner than later. But it still might take some time. And the two fixes might be useful also for stable kernels.

Petr Mladek (2):
printk/panic: Access the main printk log in panic() only when safe printk/panic/x86: Allow to access printk log buffer after
crash_smp_send_stop()

 arch/x86/kernel/crash.c     |  6 +++++-
 include/linux/printk.h      |  6 ++++++
 kernel/panic.c              | 49 +++++++++++++++++++++++++++------------------
 kernel/printk/printk_safe.c | 37 ++++++++++++++++++++++------------
 4 files changed, 65 insertions(+), 33 deletions(-)
Signature · 1 line
2.16.4
author maintainer
#1 lore raw

PATCH 1/2 printk/panic: Access the main printk log in panic() only when safe

Kernel tries hard to store and show printk messages when panicking. Even logbuf_lock gets re-initialized when only one CPU is running after smp_send_stop().

Unfortunately, smp_send_stop() might fail on architectures that do not use NMI as a fallback. Then printk log buffer might stay locked and a deadlock is almost inevitable.

Now, printk_safe_flush_on_panic() is safe because it accesses the main log buffer only when the lock is available. But kmgs_dump() and console_unlock() callers might still cause a deadlock.

A better approach is to move printk lock busting into a separate function. And call all dependent operations only when the lock is not blocked.

---
 include/linux/printk.h      |  6 ++++++
 kernel/panic.c              | 49 +++++++++++++++++++++++++++------------------
 kernel/printk/printk_safe.c | 35 ++++++++++++++++++++------------
 3 files changed, 58 insertions(+), 32 deletions(-)
include/linux/printk.h +6 -0
index cefd374c47b1..4d15a0eda9c6 100644
@@ -202,6 +202,7 @@ __printf(1, 2) void dump_stack_set_arch_desc(const char *fmt, ...);
 void dump_stack_print_info(const char *log_lvl);
 void show_regs_print_info(const char *log_lvl);
 extern asmlinkage void dump_stack(void) __cold;
+extern int printk_bust_lock_safe(void);
 extern void printk_safe_init(void);
 extern void printk_safe_flush(void);
 extern void printk_safe_flush_on_panic(void);
@@ -269,6 +270,11 @@ static inline void dump_stack(void)
 {
 }
 
+static inline int printk_bust_lock_safe(void)
+{
+	return 0;
+}
+
 static inline void printk_safe_init(void)
 {
 }
kernel/panic.c +30 -19

73 lines not on this page · raw message

kernel/printk/printk_safe.c +22 -13
index b4045e782743..71d4b763f811 100644
@@ -253,6 +253,26 @@ void printk_safe_flush(void)
 	}
 }
 
+/**
+ * printk_try_bust_lock - make printk log accessible when safe
+ *
+ * Return 0 when the log is accessible. Return -EWOULDBLOCK when
+ * it is not safe and likely to cause a deadlock.
+ */
+int printk_bust_lock_safe(void)
+{
+	if (!raw_spin_is_locked(&logbuf_lock))
+		return 0;
+
+	if (num_online_cpus() == 1) {
+		debug_locks_off();
+		raw_spin_lock_init(&logbuf_lock);
+		return 0;
+	}
+
+	return -EWOULDBLOCK;
+}
+
 /**
  * printk_safe_flush_on_panic - flush all per-cpu nmi buffers when the system
  *	goes down.
@@ -265,19 +285,8 @@ void printk_safe_flush(void)
  */
 void printk_safe_flush_on_panic(void)
 {
-	/*
-	 * Make sure that we could access the main ring buffer.
-	 * Do not risk a double release when more CPUs are up.
-	 */
-	if (raw_spin_is_locked(&logbuf_lock)) {
-		if (num_online_cpus() > 1)
-			return;
-
-		debug_locks_off();
-		raw_spin_lock_init(&logbuf_lock);
-	}
-
-	printk_safe_flush();
+	if (printk_bust_lock_safe() == 0)
+		printk_safe_flush();
 }
 
 #ifdef CONFIG_PRINTK_NMI
Signature · 1 line
2.16.4
#2 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 7 lines · Petr Mladek

Kernel tries hard to store and show printk messages when panicking. Even logbuf_lock gets re-initialized when only one CPU is running after smp_send_stop().

Unfortunately, smp_send_stop() might fail on architectures that do not use NMI as a fallback. Then printk log buffer might stay locked and a deadlock is almost inevitable.

I'd say that deadlock is still almost inevitable.

panic-CPU syncs with the printing-CPU before it attempts to SMP_STOP. If there is an active printing-CPU, which is looping in console_unlock(), taking logbuf_lock in order to msg_print_text() and stuff, then panic-CPU will spin on console_owner waiting for that printing-CPU to handover printing duties.

	pr_emerg("Kernel panic - not syncing");
	smp_send_stop();

If printing-CPU goes nuts under logbuf_lock, has corrupted IDT or anything else, then we will not progress with panic(). panic-CPU will deadlock. If not on
pr_emerg("Kernel panic - not syncing")

then on another pr_emerg(), right before the NMI-fallback.

	static void native_stop_other_cpus()
	{
	...
		pr_emerg("Shutting down cpus with NMI\n");
		           ^^ deadlock here
		apic->send_IPI_allbutself(NMI_VECTOR);
		           ^^ not going to happen
	...
	}

And it's not only x86. In many cases if we fail to SMP_STOP other CPUs, and one of hem is holding logbuf_lock then we are done with panic(). We will not return from smp_send_stop().

arm/kernel/smp.c

void smp_send_stop(void)

{
	...
	if (num_online_cpus() > 1)
		pr_warn("SMP: failed to stop secondary CPUs\n");
}

arm64/kernel/smp.c

void crash_smp_send_stop(void)

{
	...
	pr_crit("SMP: stopping secondary CPUs\n");
	smp_cross_call(&mask, IPI_CPU_CRASH_STOP);

	...
	if (atomic_read(&waiting_for_crash_ipi) > 0)
		pr_warning("SMP: failed to stop secondary CPUs %*pbl\n",
				cpumask_pr_args(&mask));
	...
}

arm64/kernel/smp.c

void smp_send_stop(void)

{
	...
	if (num_online_cpus() > 1)
		pr_warning("SMP: failed to stop secondary CPUs %*pbl\n",
				cpumask_pr_args(cpu_online_mask));
	...
}

riscv/kernel/smp.c

void smp_send_stop(void)

{
	...
	if (num_online_cpus() > 1)
		pr_warn("SMP: failed to stop secondary CPUs %*pbl\n",
			cpumask_pr_args(cpu_online_mask));
	...
}

And so on.

	-ss
author maintainer
#3 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 11 lines · Sergey Senozhatsky
Quoted 7 lines · Petr Mladek

Kernel tries hard to store and show printk messages when panicking. Even logbuf_lock gets re-initialized when only one CPU is running after smp_send_stop().

Unfortunately, smp_send_stop() might fail on architectures that do not use NMI as a fallback. Then printk log buffer might stay locked and a deadlock is almost inevitable.

I'd say that deadlock is still almost inevitable.

Quoted 8 lines

panic-CPU syncs with the printing-CPU before it attempts to SMP_STOP. If there is an active printing-CPU, which is looping in console_unlock(), taking logbuf_lock in order to msg_print_text() and stuff, then panic-CPU will spin on console_owner waiting for that printing-CPU to handover printing duties.

	pr_emerg("Kernel panic - not syncing");
	smp_send_stop();

Good point. I forgot the handover logic. Well, it is enabled only around call_console_drivers(). Therefore it is not under lockbuf_lock.

I had in mind some infinite loop or deadlock in vprintk_store(). There was at least one long time ago (warning triggered by leap second).

Quoted 6 lines

If printing-CPU goes nuts under logbuf_lock, has corrupted IDT or anything else, then we will not progress with panic(). panic-CPU will deadlock. If not on
pr_emerg("Kernel panic - not syncing")

then on another pr_emerg(), right before the NMI-fallback.

Nested printk() should not be problem thanks to printk_safe.

Also printk_safe_flush_on_panic() is safe because it checks whether the lock is available.

The problem is kmsg_dump() and console_unlock() called from console_unblank() and console_flush_on_panic(). They do not check whether the lock is available.

This patch does not help in all possible scenarios. But I still believe that it will help in some.

Well, I am primary interested into the 2nd patch. It fixes a real life bug report.

Best Regards,
Petr

#4 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 30 lines · Petr Mladek
Quoted 11 lines · Sergey Senozhatsky
Quoted 7 lines · Petr Mladek

Kernel tries hard to store and show printk messages when panicking. Even logbuf_lock gets re-initialized when only one CPU is running after smp_send_stop().

Unfortunately, smp_send_stop() might fail on architectures that do not use NMI as a fallback. Then printk log buffer might stay locked and a deadlock is almost inevitable.

I'd say that deadlock is still almost inevitable.

Quoted 8 lines

panic-CPU syncs with the printing-CPU before it attempts to SMP_STOP. If there is an active printing-CPU, which is looping in console_unlock(), taking logbuf_lock in order to msg_print_text() and stuff, then panic-CPU will spin on console_owner waiting for that printing-CPU to handover printing duties.

	pr_emerg("Kernel panic - not syncing");
	smp_send_stop();

Good point. I forgot the handover logic. Well, it is enabled only around call_console_drivers(). Therefore it is not under lockbuf_lock.

I had in mind some infinite loop or deadlock in vprintk_store(). There was at least one long time ago (warning triggered by leap second).

Quoted 8 lines
Quoted 6 lines

If printing-CPU goes nuts under logbuf_lock, has corrupted IDT or anything else, then we will not progress with panic(). panic-CPU will deadlock. If not on
pr_emerg("Kernel panic - not syncing")

then on another pr_emerg(), right before the NMI-fallback.

Nested printk() should not be problem thanks to printk_safe.

Where do nested printk()-s come from? Which one of the following scenarios you cover in commit message:

scenario 1

- we have CPUB which holds logbuf_lock
- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock stays locked on remote CPU

scenario 2

- we have CPUA which holds logbuf_lock
- we have panic() on CPUA, but it cannot bring down some other CPUB so logbuf_lock stays locked on local CPU, and it cannot re-init logbuf.

	-ss
author maintainer
#5 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 40 lines · Sergey Senozhatsky
Quoted 30 lines · Petr Mladek
Quoted 11 lines · Sergey Senozhatsky
Quoted 7 lines · Petr Mladek

Kernel tries hard to store and show printk messages when panicking. Even logbuf_lock gets re-initialized when only one CPU is running after smp_send_stop().

Unfortunately, smp_send_stop() might fail on architectures that do not use NMI as a fallback. Then printk log buffer might stay locked and a deadlock is almost inevitable.

I'd say that deadlock is still almost inevitable.

Quoted 8 lines

panic-CPU syncs with the printing-CPU before it attempts to SMP_STOP. If there is an active printing-CPU, which is looping in console_unlock(), taking logbuf_lock in order to msg_print_text() and stuff, then panic-CPU will spin on console_owner waiting for that printing-CPU to handover printing duties.

	pr_emerg("Kernel panic - not syncing");
	smp_send_stop();

Good point. I forgot the handover logic. Well, it is enabled only around call_console_drivers(). Therefore it is not under lockbuf_lock.

I had in mind some infinite loop or deadlock in vprintk_store(). There was at least one long time ago (warning triggered by leap second).

Quoted 8 lines
Quoted 6 lines

If printing-CPU goes nuts under logbuf_lock, has corrupted IDT or anything else, then we will not progress with panic(). panic-CPU will deadlock. If not on
pr_emerg("Kernel panic - not syncing")

then on another pr_emerg(), right before the NMI-fallback.

Nested printk() should not be problem thanks to printk_safe.

Quoted 8 lines

Where do nested printk()-s come from? Which one of the following scenarios you cover in commit message:

scenario 1

- we have CPUB which holds logbuf_lock
- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock stays locked on remote CPU

No, this scenario is not affected by this patch. It would always lead to a deadlock.

Quoted 6 lines

scenario 2

- we have CPUA which holds logbuf_lock
- we have panic() on CPUA, but it cannot bring down some other CPUB so logbuf_lock stays locked on local CPU, and it cannot re-init logbuf.

This scenario should get better handled by this patch. The difference will be when smp_send_stop() is not able to stop all CPUs:

+ Before:
+ printk_safe_flush_on_panic() will keep logbuf_lock locked and do nothing.

+ kmsg_dump(), console_unblank(), or console_flush_on_panic() will deadlock when they try to get logbuf_lock(). They will not be able to process any single line.

+ After:
+ printk_bust_lock_safe() will keep logbuf_lock locked

+ All functions using logbuf_lock will not get called. We will not see the messages (as previously) but the system will not deadlock.

But there is one more scenario 3:

- we have CPUB which loops or is deadlocked in IRQ context

- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock might be takes and release from time to time by CPUB

Hmm, this scenario might be handled a bit _worse_ by this patch:

+ Before:
+ printk_safe_flush_on_panic() will not touch logbuf_lock The messages will get flushed according to the state of logbuf_lock at the moment when it is being checked.

+ kmsg_dump(), console_unblank(), or console_flush_on_panic() will be able to do their job.

+ After:
+ printk_safe_flush_on_panic(), kmsg_dump(), console_unblank(), and console_flush_on_panic() could finish the job. But they will get called _only_ when logbuf_lock is released at the moment when it is being checked by printk_bust_lock_safe().

Resume:

From my POV, the 3rd scenario is the most likely one. Therefore this patch would make more bad than good.

It might be possible to somehow detect if lockbuf_lock is released from time the time on the non-stopped CPU. But it would be hairy. IMHO, it is not worth it.

Thanks a lot for helping me to sort the ideas. I suggest to forget this patch and work on lockless ringbuffer.

Best Regards,
Petr

#6 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

On (07/19/19 14:57), Petr Mladek wrote:
[..]

Quoted 11 lines
Quoted 8 lines

Where do nested printk()-s come from? Which one of the following scenarios you cover in commit message:

scenario 1

- we have CPUB which holds logbuf_lock
- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock stays locked on remote CPU

No, this scenario is not affected by this patch. It would always lead to a deadlock.

Agreed, in many cases we won't be able to panic() the system properly, deadlocking somewhere in smp_send_stop().

Quoted 6 lines
Quoted 6 lines

scenario 2

- we have CPUA which holds logbuf_lock
- we have panic() on CPUA, but it cannot bring down some other CPUB so logbuf_lock stays locked on local CPU, and it cannot re-init logbuf.

[..]

Quoted 11 lines

+ Before:
+ printk_safe_flush_on_panic() will keep logbuf_lock locked and do nothing.

+ kmsg_dump(), console_unblank(), or console_flush_on_panic() will deadlock when they try to get logbuf_lock(). They will not be able to process any single line.

+ After:
+ printk_bust_lock_safe() will keep logbuf_lock locked

Quoted 6 lines

+ All functions using logbuf_lock will not get called. We will not see the messages (as previously) but the system will not deadlock.

But there is one more scenario 3:

Yes!

Quoted 5 lines

- we have CPUB which loops or is deadlocked in IRQ context

- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock might be takes and release from time to time by CPUB

Great!

This is the only case when we actually need to pay attention to num_online_cpus(), because there is an active logbuf_lock owner; in any other case we can unconditionally re-init printk() locks.

But there is more to it.

Note, that the problem in scenario 3 is bigger than just logbuf_lock. Regardless of logbuf implementation we will not be able to panic() the system.

If we have a never ending source of printk() messages, coming from misbehaving CPU which stuck in printing loop in IRQ context, then flush_on_panic() will never end or kmsg dump will never stop, etc. We need to cut off misbehaving CPUs. Panic CPU waits (for up to 1 second?) in smp_send_stop() for secondary CPUs to die, if some secondary CPUs are still chatty after that then most likely those CPUs don't have anything good to say, just a pointless flood of same messages over and over again; which, however, will not let panic CPU to proceed.

And this is where the idea of "disconnecting" those CPUs from main logbuf come from.

So what we can do:
- smp_send_stop()
- disconnect all-but-self from logbuf (via printk-safe)
- wait for 1 or 2 more extra seconds for secondary CPUs to leave console_unlock() and to redirect printks to per-CPU buffers
- after that we are sort of good-to-go: re-init printk locks and do kmsg_dump, flush_on_panic().

Note, misbehaving CPUs will write to per-CPU buffers, they are not expected to be able to flush per-CPU buffers to the main logbuf. That will require enabled IRQs, which should deliver stop IPI. But we can do even more - just disable print_safe irq work on disconnect CPUs.

So, shall we try one more time with the "disconnect" misbehaving CPUs approach? I can send an RFC patch.

	-ss
author maintainer
#7 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 19 lines · Sergey Senozhatsky
Quoted 1 line · Petr Mladek

But there is one more scenario 3:

Yes!

Quoted 5 lines

- we have CPUB which loops or is deadlocked in IRQ context

- we have CPUA which panic()-s the system, but can't bring CPUB down, so logbuf_lock might be takes and release from time to time by CPUB

Great!

This is the only case when we actually need to pay attention to num_online_cpus(), because there is an active logbuf_lock owner; in any other case we can unconditionally re-init printk() locks.

But there is more to it.

Quoted 7 lines

Note, that the problem in scenario 3 is bigger than just logbuf_lock. Regardless of logbuf implementation we will not be able to panic() the system.

If we have a never ending source of printk() messages, coming from misbehaving CPU which stuck in printing loop in IRQ context, then flush_on_panic() will never end or kmsg dump will never stop, etc.

Yes.

Quoted 6 lines

We need to cut off misbehaving CPUs. Panic CPU waits (for up to 1 second?) in smp_send_stop() for secondary CPUs to die, if some secondary CPUs are still chatty after that then most likely those CPUs don't have anything good to say, just a pointless flood of same messages over and over again; which, however, will not let panic CPU to proceed.

Makes sense.

Quoted 6 lines

And this is where the idea of "disconnecting" those CPUs from main logbuf come from.

So what we can do:
- smp_send_stop()
- disconnect all-but-self from logbuf (via printk-safe)

printk_safe is not really necessary. As you wrote, nobody is interested into messages from CPUs that are supposed to be stopped.

It might be enough to set some global variable, for example, with the CPU number that is calling panic() and is the only one allowed to print messages from this point on.

It might even be used to force console lock owner to leave the cycle immediately.

Quoted 4 lines

- wait for 1 or 2 more extra seconds for secondary CPUs to leave console_unlock() and to redirect printks to per-CPU buffers
- after that we are sort of good-to-go: re-init printk locks and do kmsg_dump, flush_on_panic().

Sounds good.

Quoted 7 lines

Note, misbehaving CPUs will write to per-CPU buffers, they are not expected to be able to flush per-CPU buffers to the main logbuf. That will require enabled IRQs, which should deliver stop IPI. But we can do even more - just disable print_safe irq work on disconnect CPUs.

So, shall we try one more time with the "disconnect" misbehaving CPUs approach? I can send an RFC patch.

IMHO, it will be acceptable only when it is reasonably simple and straightforward. The panic() code is full of special hacks and it is already complicated to follow all the twists.

Especially because this scenario came up from a theoretical discussion. Note that my original, real bug report, was with logbuf_lock NMI, enabled kdump notifiers, ...

Best Regards,
Petr

#8 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Sorry for the delayed response.

On (07/24/19 14:27), Petr Mladek wrote:
[..]

Quoted 10 lines
Quoted 6 lines

And this is where the idea of "disconnecting" those CPUs from main logbuf come from.

So what we can do:
- smp_send_stop()
- disconnect all-but-self from logbuf (via printk-safe)

printk_safe is not really necessary. As you wrote, nobody is interested into messages from CPUs that are supposed to be stopped.

OK.
Doing it in printk.c makes it easier to handle console_owner/waiter, which are not exported.

Quoted 3 lines

It might be enough to set some global variable, for example, with the CPU number that is calling panic() and is the only one allowed to print messages from this point on.

Sounds good.

Quoted 2 lines

It might even be used to force console lock owner to leave the cycle immediately.

Yes, makes sense.

[..]

Quoted 6 lines
Quoted 2 lines

So, shall we try one more time with the "disconnect" misbehaving CPUs approach? I can send an RFC patch.

IMHO, it will be acceptable only when it is reasonably simple and straightforward. The panic() code is full of special hacks and it is already complicated to follow all the twists.

When you have a chance, mind to take a look at the patch below? Doesn't look very difficult (half of it are white-spaces and comments, I believe).

Quoted 3 lines

Especially because this scenario came up from a theoretical discussion. Note that my original, real bug report, was with logbuf_lock NMI, enabled kdump notifiers, ...

I understand. The idea is that this patch should handle your real scenario + theoretical scenario.

---
 include/linux/printk.h |  2 ++
 kernel/panic.c         |  9 ++++++-
 kernel/printk/printk.c | 53 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+), 1 deletion(-)
include/linux/printk.h +2 -0
index 57c9473f4a81..016f5ba06e94 100644
@@ -322,6 +322,8 @@ static inline void printk_safe_flush_on_panic(void)
 }
 #endif
 
+void printk_enter_panic_mode(int cpu);
+
 extern int kptr_restrict;
 
 #ifndef pr_fmt
kernel/panic.c +8 -1
index d1ece4c363b9..85fac975a90f 100644
@@ -254,13 +254,20 @@ void panic(const char *fmt, ...)
 		crash_smp_send_stop();
 	}
 
+	/* Misbehaving secondary CPUs cannot printk() to the main logbuf now */
+	printk_enter_panic_mode(this_cpu);
+
 	/*
 	 * Run any panic handlers, including those that might need to
 	 * add information to the kmsg dump output.
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	/* Call flush even twice. It tries harder with a single online CPU */
+	/*
+	 * Call flush even twice. It tries harder with a single online CPU.
+	 * Even if we failed to stop some of secondary CPUs we have printk
+	 * locks re-initialized and keep secondary CPUs off printk().
+	 */
 	printk_safe_flush_on_panic();
 	kmsg_dump(KMSG_DUMP_PANIC);
 
kernel/printk/printk.c +53 -0

77 lines not on this page · raw message

Signature · 1 line
2.22.0
#9 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

Quoted 3 lines · Sergey Senozhatsky

When you have a chance, mind to take a look at the patch below? Doesn't look very difficult (half of it are white-spaces and comments, I believe).

I'm very sorry for annoyance.

Updated version:
-- passes !PRINTK build
-- moved WRITE_ONCE(console_waiter, false) in printk_enter_panic_mode() -- added panic_in_progress_on_other_cpu() to console_trylock_spinning()

No more updates this week. Will wait for feedback.

Once again, sorry!

---
 include/linux/printk.h |  5 ++++
 kernel/panic.c         |  9 +++++-
 kernel/printk/printk.c | 64 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 1 deletion(-)
include/linux/printk.h +5 -0
index 57c9473f4a81..8293156d8243 100644
@@ -245,6 +245,7 @@ extern asmlinkage void dump_stack(void) __cold;
 extern void printk_safe_init(void);
 extern void printk_safe_flush(void);
 extern void printk_safe_flush_on_panic(void);
+extern void printk_enter_panic_mode(int cpu);
 #else
 static inline __printf(1, 0)
 int vprintk(const char *s, va_list args)
@@ -320,6 +321,10 @@ static inline void printk_safe_flush(void)
 static inline void printk_safe_flush_on_panic(void)
 {
 }
+
+static inline void printk_enter_panic_mode(int cpu)
+{
+}
 #endif
 
 extern int kptr_restrict;
kernel/panic.c +8 -1
index d1ece4c363b9..85fac975a90f 100644
@@ -254,13 +254,20 @@ void panic(const char *fmt, ...)
 		crash_smp_send_stop();
 	}
 
+	/* Misbehaving secondary CPUs cannot printk() to the main logbuf now */
+	printk_enter_panic_mode(this_cpu);
+
 	/*
 	 * Run any panic handlers, including those that might need to
 	 * add information to the kmsg dump output.
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	/* Call flush even twice. It tries harder with a single online CPU */
+	/*
+	 * Call flush even twice. It tries harder with a single online CPU.
+	 * Even if we failed to stop some of secondary CPUs we have printk
+	 * locks re-initialized and keep secondary CPUs off printk().
+	 */
 	printk_safe_flush_on_panic();
 	kmsg_dump(KMSG_DUMP_PANIC);
 
kernel/printk/printk.c +64 -0

94 lines not on this page · raw message

Signature · 1 line
2.22.0
#10 lore raw

Re: [PATCH 1/2] printk/panic: Access the main printk log in panic() only when safe

In reply to Petr Mladek

On (07/16/19 09:28), Petr Mladek wrote:
[..]

kernel/printk/printk_safe.c void printk_safe_flush(void) · line 274

Quoted 13 lines
kernel/printk/printk_safe.c +13 -0
+int printk_bust_lock_safe(void)
+{
+	if (!raw_spin_is_locked(&logbuf_lock))
+		return 0;
+
+	if (num_online_cpus() == 1) {
+		debug_locks_off();
+		raw_spin_lock_init(&logbuf_lock);
+		return 0;
+	}
+
+	return -EWOULDBLOCK;
+}

A side note:

I'd say that we also need to clear console_owner and re-init console_owner_lock spin_lock, panic CPU can spin forever otherwise; and I think it would also be reasonable to re-init console_sem's spin_lock, yet another lock on which panic CPU can spin forever. (Assuming that one of remote CPUs that we have NMI killed had owned any of those locks).

Console drivers' locks are mostly OK, as long as drivers check oops_in_progress in ->write() path and act accordingly, but printk() has to deal with more than one lock (logbuf_lock) before it invokes console drivers.

	-ss
author maintainer
#11 lore raw

PATCH 2/2 printk/panic/x86: Allow to access printk log buffer after crash_smp_send_stop()

In reply to Petr Mladek

crash_smp_send_stop() is a special variant of smp_send_stop(). It is used when crash_kexec_post_notifiers are enabled. CPUs are stopped but cpu_online_mask is not updated so that the original information is visible in crashdump. See the commit 0ee59413c967c35a6dd ("x86/panic:
replace smp_send_stop() with kdump friendly version in panic path") for more details.

crash_smp_send_stop() uses NMI to stop the CPUs. Then logbuf_lock might stay locked but printk_bust_lock_safe() does not know that CPUs are stopped.

Solution is to force logbuf_lock re-initialization from crash_smp_send_stop().

Note that x86 seems to be the only architecture that implements crash_smp_send_stop() and uses NMI at the same time. Other architectures could not guarantee that the CPUs were really stopped.

---
 arch/x86/kernel/crash.c     | 6 +++++-
 include/linux/printk.h      | 4 ++--
 kernel/panic.c              | 2 +-
 kernel/printk/printk_safe.c | 8 +++++---
 4 files changed, 13 insertions(+), 7 deletions(-)
arch/x86/kernel/crash.c +5 -1
index 2bf70a2fed90..9a497eb37bf7 100644
@@ -99,7 +99,11 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
 void kdump_nmi_shootdown_cpus(void)
 {
 	nmi_shootdown_cpus(kdump_nmi_callback);
-
+	/*
+	 * CPUs are stopped but it is not visible via cpus_online
+	 * bitmap. Bust logbuf_lock to make kmsg_dump() working.
+	 */
+	printk_bust_lock_safe(true);
 	disable_local_APIC();
 }
 
include/linux/printk.h +2 -2
index 4d15a0eda9c6..c050f1dafc32 100644
@@ -202,7 +202,7 @@ __printf(1, 2) void dump_stack_set_arch_desc(const char *fmt, ...);
 void dump_stack_print_info(const char *log_lvl);
 void show_regs_print_info(const char *log_lvl);
 extern asmlinkage void dump_stack(void) __cold;
-extern int printk_bust_lock_safe(void);
+extern int printk_bust_lock_safe(bool kdump_smp_stop);
 extern void printk_safe_init(void);
 extern void printk_safe_flush(void);
 extern void printk_safe_flush_on_panic(void);
@@ -270,7 +270,7 @@ static inline void dump_stack(void)
 {
 }
 
-static inline int printk_bust_lock_safe(void)
+static inline int printk_bust_lock_safe(bool kdump_smp_stop)
 {
 	return 0;
 }
kernel/panic.c +1 -1
index aa50cdb75022..54fae99e7a7e 100644
@@ -274,7 +274,7 @@ void panic(const char *fmt, ...)
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
 	/* Call flush even twice. It tries harder with a single online CPU */
-	printk_blocked = printk_bust_lock_safe();
+	printk_blocked = printk_bust_lock_safe(false);
 	if (!printk_blocked) {
 		printk_safe_flush_on_panic();
 		kmsg_dump(KMSG_DUMP_PANIC);
kernel/printk/printk_safe.c +5 -3
index 71d4b763f811..e26304277886 100644
@@ -255,16 +255,18 @@ void printk_safe_flush(void)
 
 /**
  * printk_try_bust_lock - make printk log accessible when safe
+ * @kdump_smp_stop: true when called after kdump stopped CPUs via NMI
+ *	but did not update the number of online CPUs.
  *
  * Return 0 when the log is accessible. Return -EWOULDBLOCK when
  * it is not safe and likely to cause a deadlock.
  */
-int printk_bust_lock_safe(void)
+int printk_bust_lock_safe(bool kdump_smp_stop)
 {
 	if (!raw_spin_is_locked(&logbuf_lock))
 		return 0;
 
-	if (num_online_cpus() == 1) {
+	if (num_online_cpus() == 1 || kdump_smp_stop)  {
 		debug_locks_off();
 		raw_spin_lock_init(&logbuf_lock);
 		return 0;
@@ -285,7 +287,7 @@ int printk_bust_lock_safe(void)
  */
 void printk_safe_flush_on_panic(void)
 {
-	if (printk_bust_lock_safe() == 0)
+	if (printk_bust_lock_safe(false) == 0)
 		printk_safe_flush();
 }
 
Signature · 1 line
2.16.4
#12 lore raw

Re: [PATCH 2/2] printk/panic/x86: Allow to access printk log buffer after crash_smp_send_stop()

On (07/16/19 09:28), Petr Mladek wrote:
[..]

kernel/printk/printk_safe.c void printk_safe_flush(void) · line 272

Quoted 10 lines
kernel/printk/printk_safe.c +2 -1
+int printk_bust_lock_safe(bool kdump_smp_stop)
 {
 	if (!raw_spin_is_locked(&logbuf_lock))
 		return 0;
 
-	if (num_online_cpus() == 1) {
+	if (num_online_cpus() == 1 || kdump_smp_stop)  {
 		debug_locks_off();
 		raw_spin_lock_init(&logbuf_lock);
 		return 0;

Let me test the waters. Criticize the following idea:

Can we, sort of, disconnect "supposed to be dead" CPUs from printk() so then we can unconditionally re-init printk() from panic-CPU?

We have per-CPU printk_state; so panic-CPU can set, let's say, DEAD_CPUS_TELL_NO_TALES bit on all CPUs but self, and vprintk_func() will do nothing if DEAD_CPUS_TELL_NO_TALES bit set on particular CPU. Foreign CPUs are not even supposed to be alive, and smp_send_stop() waits for IPI acks from secondary CPUs long enough on average (need to check that) so if one of the CPUs is misbehaving and doesn't want to die (geez...) we will just "disconnect" it from printk() to minimize possible logbuf/console drivers interventions and then proceed with panic; assuming that misbehaving CPUs are actually up to something sane. Sometimes, you know, in some cases, those CPUs are already dead:
either accidentally powered off, or went completely nuts and do nothing, etc. etc. but we still can kdump() and console_flush_on_panic().

	-ss
#13 lore raw

Re: [PATCH 2/2] printk/panic/x86: Allow to access printk log buffer after crash_smp_send_stop()

Quoted 19 lines · Sergey Senozhatsky

On (07/16/19 09:28), Petr Mladek wrote:
[..]

kernel/printk/printk_safe.c void printk_safe_flush(void) · line 272

Quoted 10 lines
kernel/printk/printk_safe.c +2 -1
+int printk_bust_lock_safe(bool kdump_smp_stop)
 {
      if (!raw_spin_is_locked(&logbuf_lock))
              return 0;
 
-     if (num_online_cpus() == 1) {
+     if (num_online_cpus() == 1 || kdump_smp_stop)  {
              debug_locks_off();
              raw_spin_lock_init(&logbuf_lock);
              return 0;

Let me test the waters. Criticize the following idea:

Can we, sort of, disconnect "supposed to be dead" CPUs from printk() so then we can unconditionally re-init printk() from panic-CPU?

Quoted 12 lines

We have per-CPU printk_state; so panic-CPU can set, let's say, DEAD_CPUS_TELL_NO_TALES bit on all CPUs but self, and vprintk_func() will do nothing if DEAD_CPUS_TELL_NO_TALES bit set on particular CPU. Foreign CPUs are not even supposed to be alive, and smp_send_stop() waits for IPI acks from secondary CPUs long enough on average (need to check that) so if one of the CPUs is misbehaving and doesn't want to die (geez...) we will just "disconnect" it from printk() to minimize possible logbuf/console drivers interventions and then proceed with panic; assuming that misbehaving CPUs are actually up to something sane. Sometimes, you know, in some cases, those CPUs are already dead:
either accidentally powered off, or went completely nuts and do nothing, etc. etc. but we still can kdump() and console_flush_on_panic().

Good idea.
Panic-CPU could just increment state to reroute printk into 'safe' per-cpu buffer.

Quoted 2 lines
        -ss
#14 lore raw

Re: [PATCH 2/2] printk/panic/x86: Allow to access printk log buffer after crash_smp_send_stop()

Quoted 18 lines · Konstantin Khlebnikov
Quoted 5 lines

Let me test the waters. Criticize the following idea:

Can we, sort of, disconnect "supposed to be dead" CPUs from printk() so then we can unconditionally re-init printk() from panic-CPU?

Quoted 12 lines

We have per-CPU printk_state; so panic-CPU can set, let's say, DEAD_CPUS_TELL_NO_TALES bit on all CPUs but self, and vprintk_func() will do nothing if DEAD_CPUS_TELL_NO_TALES bit set on particular CPU. Foreign CPUs are not even supposed to be alive, and smp_send_stop() waits for IPI acks from secondary CPUs long enough on average (need to check that) so if one of the CPUs is misbehaving and doesn't want to die (geez...) we will just "disconnect" it from printk() to minimize possible logbuf/console drivers interventions and then proceed with panic; assuming that misbehaving CPUs are actually up to something sane. Sometimes, you know, in some cases, those CPUs are already dead:
either accidentally powered off, or went completely nuts and do nothing, etc. etc. but we still can kdump() and console_flush_on_panic().

Quoted 3 lines

Good idea.
Panic-CPU could just increment state to reroute printk into 'safe' per-cpu buffer.

Yeah, that's better.

So we can do something like this

@@ -269,15 +269,21 @@ void printk_safe_flush_on_panic(void)
* Make sure that we could access the main ring buffer.
* Do not risk a double release when more CPUs are up. */
- if (raw_spin_is_locked(&logbuf_lock)) {
- if (num_online_cpus() > 1)
- return;
+ debug_locks_off();
+ raw_spin_lock_init(&logbuf_lock);
+ /* + re-init the rest of printk() locks */
+ printk_safe_flush();
+}

[..]

+void printk_switch_to_panic_mode(int panic_cpu)
+{
+ int cpu;

+ for_each_possible_cpu(cpu) {
+ if (cpu == panic_cpu)
+ continue;
+ per_cpu(printk_context, cpu) = 42;
+ }
}

And call printk_switch_to_panic_mode() from panic(). And we don't need to touch arch code (it also covers the case when some new ARCH will gain NMI support).

	-ss
author maintainer
#15 lore raw

Re: [PATCH 2/2] printk/panic/x86: Allow to access printk log buffer after crash_smp_send_stop()

Quoted 28 lines · Sergey Senozhatsky
Quoted 18 lines · Konstantin Khlebnikov
Quoted 5 lines

Let me test the waters. Criticize the following idea:

Can we, sort of, disconnect "supposed to be dead" CPUs from printk() so then we can unconditionally re-init printk() from panic-CPU?

Quoted 12 lines

We have per-CPU printk_state; so panic-CPU can set, let's say, DEAD_CPUS_TELL_NO_TALES bit on all CPUs but self, and vprintk_func() will do nothing if DEAD_CPUS_TELL_NO_TALES bit set on particular CPU. Foreign CPUs are not even supposed to be alive, and smp_send_stop() waits for IPI acks from secondary CPUs long enough on average (need to check that) so if one of the CPUs is misbehaving and doesn't want to die (geez...) we will just "disconnect" it from printk() to minimize possible logbuf/console drivers interventions and then proceed with panic; assuming that misbehaving CPUs are actually up to something sane. Sometimes, you know, in some cases, those CPUs are already dead:
either accidentally powered off, or went completely nuts and do nothing, etc. etc. but we still can kdump() and console_flush_on_panic().

Quoted 3 lines

Good idea.
Panic-CPU could just increment state to reroute printk into 'safe' per-cpu buffer.

Yeah, that's better.

So we can do something like this

Review void printk_safe_flush_on_panic(void) · line 273

Quoted 8 lines
(quoted diff) +2 -3
@@ -269,15 +269,21 @@ void printk_safe_flush_on_panic(void)
 	 * Make sure that we could access the main ring buffer.
 	 * Do not risk a double release when more CPUs are up.
 	 */
-	if (raw_spin_is_locked(&logbuf_lock)) {
-		if (num_online_cpus() > 1)
-			return;
+	debug_locks_off();
+	raw_spin_lock_init(&logbuf_lock);

Hmm, the check for num_online_cpus() is there to avoid deadlock caused be double unlock. The unconditional init would bring it back.

It is about what compromise we use. We either try to get the messages out, init the lock and risk a deadlock. Or we do not risk the deadlock.

The current code is not consistent. printk_safe_flush_on_panic() is conservative and does not risk the deadlock. While kmsg_dump(), console_unblank() and console_flush_on_panic() risk the deadlock.

The 1st patch tries to make it more consistent. It makes all the above functions as conservative as printk_safe_flush_on_panic() regarding logbuf_lock. While they still risk a deadlock with console-specific locks.

The reasoning might be:

+ The code under logbuf_lock mostly just manipulates strings. There are no nested locks. Infinite loops would most likely happen also during normal operation. By other words, it is not easy to stay locked in logbuf_lock (he, he, the last famous words).

+ On the other hand, a lot of code is done with disabled interrupts. It is easier to imagine that a CPU could not get stopped by normal interrupt because some deadlock or infinite loop in interrupt context. It might even be a code calling console_unlock(). Re-initializing logbuf_lock might be unnecessary risk.

+ We are probably more relaxed with console specific locks because most of them are ignored in panic after bust_spinlocks().

Anyway, this patchset is primary about logbuf_lock related deadlocks. I needed a fix for our customers. I do not want to spend too much time on upstreaming. IMHO, it is better to invest into the lockless ring buffer.

If we could agree that the patches makes things better (more consistent, more safe in kdump with enabled notifiers) and the complexity is acceptable. Then let's accept them (with some trivial improvements).

If they are too controversial, risky, or complex then let's abandon them and concentrate on lockless ring buffer.

Best Regards,
Petr

Reviewed-by Konstantin Khebnikov on the v1 series
#16 lore raw
In reply to Petr Mladek
Quoted 19 lines · Petr Mladek

Hi,

I have found some spare duct tape and wrapped some more printk-related deadlocks in panic().

More seriously, someone reported a deadlock in panic(). Some non-trivial debugging pointed out a problem with the following combination:

+ x86_64 architecture
+ panic()
+ pstore configured as message dumper (kmsg_dump())
+ crash kernel configured
+ crash_kexec_post_notifiers

In this case, CPUs are stopped by crash_smp_send_stop(). It uses NMI but it does not modify cpu_online_mask. Therefore logbuf_lock might stay locked, see 2nd patch for more details.

Quoted 8 lines

The above is a real corner case. But similar problem seems to be even in the common situations on architectures that do not use NMI in smp_send_stop() as a fallback, see 1st patch.

Back to the duct tape. I hope that we will get rid of these problems with the lockless printk ringbuffer rather sooner than later. But it still might take some time. And the two fixes might be useful also for stable kernels.

Looks good.

Quoted 16 lines

Petr Mladek (2):
printk/panic: Access the main printk log in panic() only when safe printk/panic/x86: Allow to access printk log buffer after
crash_smp_send_stop()

 arch/x86/kernel/crash.c     |  6 +++++-
 include/linux/printk.h      |  6 ++++++
 kernel/panic.c              | 49 +++++++++++++++++++++++++++------------------
 kernel/printk/printk_safe.c | 37 ++++++++++++++++++++++------------
 4 files changed, 65 insertions(+), 33 deletions(-)

--
2.16.4