panic: fix panic_force_cpu= redirect races and NMI bypass

The panic_force_cpu= parameter redirects a panic to a specific CPU so the crash kernel runs there. The redirect code in panic_try_force_cpu() had two races and an NMI bypass, all found by Sashiko. This series closes them and kills one more hack that the rework turned up.

Changes since v6, all from Petr's review:

Patch 4 now fixes the -Wformat-security warning that the __printf annotation tripped in hpwdt, the closest caller: nmi_panic() there passes a stack buffer, so it calls nmi_panic(regs, "%s", panic_msg) now. The "every caller passes a plain string literal" claim is gone from the changelog, it was wrong anyway, hpwdt never did. The duplicate __printf annotation was dropped from the .c definition, the header carries it.

Patch 6 dropped the "return value is not even checked" sentence from the changelog. The kmalloc is gone entirely in this version, and the self report on failure was deliberate anyway. The now constant msg variable is gone too, panic_force_buf is passed directly.

Everything else is unchanged from v6. Patches 1, 2, 3 and 5 carry Petr's Reviewed-by from the v6 thread. The double warning when a CPU goes offline while marked in the panic mask and the set_cpu_online() asymmetry in nmi_panic stay as they are, Petr called both not worth touching. Rebased onto current master, the arch_do_panic rework from Aug 31 merged cleanly on top.

Panic from NMI still goes through the redirect path first, so the crash kernel ends up on the CPU the admin asked for. The redirect buffer is a static 1KB now, no initcall, no fallback message.

Bradley Morgan (6):
panic: fix redirect CPU race in panic_try_force_cpu()
panic: flatten nmi_panic control flow
panic: fix va_list reuse in panic_try_force_cpu()
panic: restore variable arguments to nmi_panic()
panic: allow force_cpu redirect from an NMI
panic: kill the "buffer unavailable" redirect fallback

 drivers/watchdog/hpwdt.c |  2 +-
 include/linux/panic.h    |  3 +-
 kernel/panic.c           | 80 ++++++++++++++++++++++------------------
 3 files changed, 47 insertions(+), 38 deletions(-)

2.47.3

The cmpxchg() in panic_try_force_cpu() makes sure that only one CPU tries to redirect panic() to the requested CPU. It is similar to the cmpxchg() in panic_try_start() which makes sure that only one CPU does the panic(). In both situations, only the winner of cmpxchg() should proceed further. Other CPUs should go offline.

There is a bug because the cmpxchg loser returns false and falls through into vpanic(). Two non-target CPUs A and B panic, the requested CPU is C:

cpu A cpu B
---------- ----------
panic() panic()
vpanic() vpanic()
panic_try_force_cpu() panic_try_force_cpu()
cmpxchg wins cmpxchg fails
redirect = A old_cpu = A
IPI -> C return false <- BUG
return true panic_try_start() wins
panic_smp_self_stop() __crash_kexec() on B
(A stops) (target C bypassed)

The loser must stop, not fall through. It cannot just return true, though. A CPU that already won the redirect cmpxchg can reenter panic_try_force_cpu() on the same CPU, for example a nested NMI during the message formatting, before the IPI is sent:

             cpu A (1st)                  cpu A (nested)
          ----------                     ----------
    panic()
    vpanic()
    panic_try_force_cpu()
        cmpxchg wins (redirect = A)
        vsnprintf(msg) ...
            <-- NMI, nested panic -->
                                     panic()
                                     vpanic()
                                     panic_try_force_cpu()
                                         cmpxchg fails
                                         old_cpu == A (this CPU)
                                         return true   <- would halt
                                     panic_smp_self_stop()
                                     (IPI never sent, panic abandoned)

Check old_cpu against this_cpu so a second call from the same CPU returns false and falls through to panic_try_start() instead.

Also fix the panic_in_progress() check. We must not redirect when panic_cpu is already assigned. Return true to stop when the panic is on another CPU, false to proceed when it is this one.

Update the panic_try_force_cpu() doc comment for the new return value semantics.

---
 kernel/panic.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

index 50715f14cf04..08072bfae422 100644
@@ -371,8 +371,9 @@ int __weak panic_smp_redirect_cpu(int target_cpu, void *msg)
  * for the crash kernel to function correctly. This function redirects
  * panic handling to the CPU specified via the panic_force_cpu= boot parameter.
  *
- * Returns false if panic should proceed on current CPU.
- * Returns true if panic was redirected.
+ * Returns true when this CPU must stop: the panic was redirected or is
+ * already running on another CPU.
+ * Returns false when panic() should proceed on this CPU.
  */
 __printf(1, 0)
 static bool panic_try_force_cpu(const char *fmt, va_list args)

@@ -396,16 +397,20 @@ static bool panic_try_force_cpu(const char *fmt, va_list args)
 		return false;
 	}
 
-	/* Another panic already in progress */
+	/*
+	 * Don't redirect when a panic is already in progress. Stop this
+	 * CPU when it's another one, proceed when it's this one.
+	 */
 	if (panic_in_progress())
-		return false;
+		return panic_on_other_cpu();
 
 	/*
-	 * Only one CPU can do the redirect. Use atomic cmpxchg to ensure
-	 * we don't race with another CPU also trying to redirect.
+	 * Only one CPU can do the redirection. Others should go offline.
+	 * Continue with panic() when we already tried the redirection
+	 * from this CPU before, for example via nmi_panic().
 	 */
 	if (!atomic_try_cmpxchg(&panic_redirect_cpu, &old_cpu, this_cpu))
-		return false;
+		return old_cpu != this_cpu;
 
 	/*
 	 * Use dynamically allocated buffer if available, otherwise

2.47.3

Oh god sake there wasn't a issue in like the last 5 versions --- Thanks!
https://lore.kernel.org/all/[email protected]/

panic() is __noreturn, so the else after panic_try_start() is dead. Drop it so the force_cpu path can be added cleanly on top.

---
 kernel/panic.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 08072bfae422..5646d4fb82b7 100644
@@ -517,7 +517,8 @@ void nmi_panic(struct pt_regs *regs, const char *msg)
 {
 	if (panic_try_start())
 		panic("%s", msg);
-	else if (panic_on_other_cpu())
+
+	if (panic_on_other_cpu())
 		nmi_panic_self_stop(regs);
 }
 EXPORT_SYMBOL(nmi_panic);

2.47.3

vsnprintf() consumes the caller's va_list. When the redirect fails, vpanic() reuses it for the panic message, which is undefined behavior. Use va_copy().

---
 kernel/panic.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

index 5646d4fb82b7..7388eb81a1c4 100644
@@ -417,7 +417,12 @@ static bool panic_try_force_cpu(const char *fmt, va_list args)
 	 * fall back to static message for early boot panics or allocation failure.
 	 */
 	if (panic_force_buf) {
-		vsnprintf(panic_force_buf, PANIC_MSG_BUFSZ, fmt, args);
+		va_list ap;
+
+		/* Do not consume args, the caller reuses it if we fail */
+		va_copy(ap, args);
+		vsnprintf(panic_force_buf, PANIC_MSG_BUFSZ, fmt, ap);
+		va_end(ap);
 		msg = panic_force_buf;
 	} else {
 		msg = "Redirected panic (buffer unavailable)";

2.47.3

nmi_panic() used to accept variable arguments until commit ebc41f20d77f ("panic: change nmi_panic from macro to function") flattened it to a final message string. vpanic() did not exist back then, so the function had to format through panic("%s", msg).

Bring the variable arguments back and format with vpanic() directly. The next patch makes nmi_panic() try the panic_force_cpu= redirect before claiming panic_cpu, which needs the arguments twice: once to format the message for the redirected CPU and once for vpanic() when no redirect happens. Passing a final string would lose that.

No current caller passes a string with format specifiers. The closest one is hpwdt_pretimeout(), which builds panic_msg with hex_byte_pack() and has only two variants, both plain strings. But the new __printf() annotation on nmi_panic() would warn with -Wformat-security there because the buffer is passed directly as the format argument, so switch it to nmi_panic(regs, "%s", panic_msg).

---
 drivers/watchdog/hpwdt.c |  2 +-
 include/linux/panic.h    |  3 ++-
 kernel/panic.c           | 10 ++++++++--
 3 files changed, 11 insertions(+), 4 deletions(-)

index 8af1fad2de0b..78227d200afe 100644
@@ -199,7 +199,7 @@ static int hpwdt_pretimeout(unsigned int ulReason, struct pt_regs *regs)
 	}
 
 	hex_byte_pack(panic_msg, nmistat);
-	nmi_panic(regs, panic_msg);
+	nmi_panic(regs, "%s", panic_msg);
 
 	return NMI_HANDLED;
 }

index 98dd7dfd27de..17e61b61c45f 100644
@@ -13,7 +13,8 @@ __printf(1, 2)
 void panic(const char *fmt, ...) __noreturn __cold;
 __printf(1, 0)
 void vpanic(const char *fmt, va_list args) __noreturn __cold;
-void nmi_panic(struct pt_regs *regs, const char *msg);
+__printf(2, 3)
+void nmi_panic(struct pt_regs *regs, const char *fmt, ...);
 void check_panic_on_warn(const char *origin);
 extern void oops_enter(void);
 extern void oops_exit(void);

index 7388eb81a1c4..e240ca06faab 100644
@@ -518,13 +518,19 @@ EXPORT_SYMBOL(panic_on_other_cpu);
  * nmi_panic_self_stop() which can provide architecture dependent code such
  * as saving register state for crash dump.
  */
-void nmi_panic(struct pt_regs *regs, const char *msg)
+void nmi_panic(struct pt_regs *regs, const char *fmt, ...)
 {
+	va_list args;
+
+	va_start(args, fmt);
+
 	if (panic_try_start())
-		panic("%s", msg);
+		vpanic(fmt, args);
 
 	if (panic_on_other_cpu())
 		nmi_panic_self_stop(regs);
+
+	va_end(args);
 }
 EXPORT_SYMBOL(nmi_panic);
 

2.47.3

The reasoning is not correct. It would be perfectly fine to pass the final string.

I would write something like:

<paste>
Bring the variable arguments back and format with vpanic() directly. The next patch makes nmi_panic() try the panic_force_cpu= redirect before claiming panic_cpu. panic_try_force_cpu() needs to print the message but it is used also by panic() which accepts variable argument list. The string could be formatted only when a CPU gets assigned to process the redirection. Otherwise, there might be a race when writing to the panic_force_buf.
</paste>

With the updated commit message:

Best Regards,
Petr

nmi_panic() claims panic_cpu via panic_try_start() before calling panic(). When the panic later reaches panic_try_force_cpu(), the panic_in_progress() check sees panic_cpu set and refuses to redirect. The crash kernel runs on the CPU that took the NMI instead of the CPU requested with panic_force_cpu=:

nmi_panic()
panic_try_start() wins, panic_cpu = X
panic("%s", msg)
vpanic()
panic_try_force_cpu()
panic_in_progress() true, panic_cpu is X
return false redirect bypassed
panic_try_start() already won
__crash_kexec() on X, not the requested CPU

Try the redirect before claiming panic_cpu instead, as suggested by Petr Mladek. nmi_panic() now calls panic_try_force_cpu() first and claims panic_cpu only when no redirect happened. The requested CPU claims panic_cpu itself when it runs panic(), so panic_cpu does not need to be handed off. panic_try_force_cpu() copies the arguments before formatting (patch 3), so nmi_panic() can pass them to vpanic() again when no redirect happens.

The redirect IPI is sent with smp_call_function_single_async(), which is not guaranteed to work from NMI context. Treat it as best effort. It is worth the risk because the redirection is only used when the crash kernel would not work on the panicking CPU anyway.

Keep returning when the panic is already running on this CPU. A nested NMI, for example with unknown_nmi_panic while this CPU is inside panic(), must return and let the interrupted panic() continue instead of parking the CPU in nmi_panic_self_stop().

Mark the redirecting CPU offline before stopping it, like vpanic() does, so that panic_other_cpus_shutdown() on the target CPU does not wait for it.

---
 kernel/panic.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

index e240ca06faab..29c981926f5f 100644
@@ -513,10 +513,11 @@ bool panic_on_other_cpu(void)
 EXPORT_SYMBOL(panic_on_other_cpu);
 
 /*
- * A variant of panic() called from NMI context. We return if we've already
- * panicked on this CPU. If another CPU already panicked, loop in
- * nmi_panic_self_stop() which can provide architecture dependent code such
- * as saving register state for crash dump.
+ * A variant of panic() called from NMI context. The panic is first
+ * redirected to the CPU requested via panic_force_cpu=, when configured.
+ * We return if we've already panicked on this CPU. If another CPU already
+ * panicked, loop in nmi_panic_self_stop() which can provide architecture
+ * dependent code for saving register state for crash dump.
  */
 void nmi_panic(struct pt_regs *regs, const char *fmt, ...)
 {

@@ -524,6 +525,16 @@ void nmi_panic(struct pt_regs *regs, const char *fmt, ...)
 
 	va_start(args, fmt);
 
+	/* Try to redirect to the requested CPU before claiming panic_cpu. */
+	if (panic_try_force_cpu(fmt, args)) {
+		/*
+		 * Mark ourselves offline so panic_other_cpus_shutdown() won't
+		 * wait for us on architectures that check num_online_cpus().
+		 */
+		set_cpu_online(raw_smp_processor_id(), false);
+		nmi_panic_self_stop(regs);
+	}
+
 	if (panic_try_start())
 		vpanic(fmt, args);
 

2.47.3

The redirect buffer is a disgusting terrible hack. panic_force_buf is kmalloc'ed in a late_initcall, and until then the redirect delivers this as the panic message:

Redirected panic (buffer unavailable)

The whole point of the redirect is to hand the panic message to the target CPU, so the crash kernel boots knowing it panicked and not why. And that window is the entire boot, from the early_param to the late_initcall, which is exactly when you most want the message.

Make it a static 1KB buffer and kill the initcall. The cost is 1KB of .bss in SMP crash dump builds, and it is only ever touched when panic_force_cpu= is set anyway. The local msg variable is gone too, the buffer goes directly to panic_smp_redirect_cpu().

---
 kernel/panic.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)

index 29c981926f5f..170744163fc2 100644
@@ -307,7 +307,7 @@ atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
 atomic_t panic_redirect_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
 
 #if defined(CONFIG_SMP) && defined(CONFIG_CRASH_DUMP)
-static char *panic_force_buf;
+static char panic_force_buf[PANIC_MSG_BUFSZ];
 
 static int __init panic_force_cpu_setup(char *str)
 {

@@ -326,17 +326,6 @@ static int __init panic_force_cpu_setup(char *str)
 }
 early_param("panic_force_cpu", panic_force_cpu_setup);
 
-static int __init panic_force_cpu_late_init(void)
-{
-	if (panic_force_cpu < 0)
-		return 0;
-
-	panic_force_buf = kmalloc(PANIC_MSG_BUFSZ, GFP_KERNEL);
-
-	return 0;
-}
-late_initcall(panic_force_cpu_late_init);
-
 static void do_panic_on_target_cpu(void *info)
 {
 	panic("%s", (char *)info);

@@ -380,7 +369,7 @@ static bool panic_try_force_cpu(const char *fmt, va_list args)
 {
 	int this_cpu = raw_smp_processor_id();
 	int old_cpu = PANIC_CPU_INVALID;
-	const char *msg;
+	va_list ap;
 
 	/* Feature not enabled via boot parameter */
 	if (panic_force_cpu < 0)

@@ -413,20 +402,11 @@ static bool panic_try_force_cpu(const char *fmt, va_list args)
 		return old_cpu != this_cpu;
 
 	/*
-	 * Use dynamically allocated buffer if available, otherwise
-	 * fall back to static message for early boot panics or allocation failure.
+	 * Do not consume args, the caller reuses them if we fail.
 	 */
-	if (panic_force_buf) {
-		va_list ap;
-
-		/* Do not consume args, the caller reuses it if we fail */
-		va_copy(ap, args);
-		vsnprintf(panic_force_buf, PANIC_MSG_BUFSZ, fmt, ap);
-		va_end(ap);
-		msg = panic_force_buf;
-	} else {
-		msg = "Redirected panic (buffer unavailable)";
-	}
+	va_copy(ap, args);
+	vsnprintf(panic_force_buf, PANIC_MSG_BUFSZ, fmt, ap);
+	va_end(ap);
 
 	console_verbose();
 	bust_spinlocks(1);

@@ -441,7 +421,7 @@ static bool panic_try_force_cpu(const char *fmt, va_list args)
 		dump_stack();
 	}
 
-	if (panic_smp_redirect_cpu(panic_force_cpu, (void *)msg) != 0) {
+	if (panic_smp_redirect_cpu(panic_force_cpu, panic_force_buf) != 0) {
 		atomic_set(&panic_redirect_cpu, PANIC_CPU_INVALID);
 		pr_warn("panic: failed to redirect to CPU %d, continuing on CPU %d\n",
 			panic_force_cpu, this_cpu);

2.47.3

Look good to me:

Best Regards,
Petr

IMHO, this version is ready to go. It would be great to fix the commit message in the 4th patch but I think that Andrew could do the fixup.

Best Regards,
Petr

Akpm? Up for it?

--- Thanks!
"I'm not a very positive person" - Linus torvalds

Done, thanks.