Skip to contentSkip to conversation

dump_stack: Allow runtime updates of the hardware description

RFC PATCH New Nathan Lynch 1 participant 6 messages Jan 18, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover dump_stack: Allow runtime updates of the hardware description

When the kernel emits a stack trace, typically it includes a hardware description string, e.g.

  Kernel panic - not syncing: sysrq triggered crash
  CPU: 6 PID: 46433 Comm: bash Tainted: G        W          6.7.0-rc2+ #83
Quoted 1 line
Hardware name: IBM,9040-MR9 POWER9 (architected) 0x4e2102 0xf000005 of:IBM,FW950.01 (VM950_047) hv:phyp pSeries
  Call Trace:
   dump_stack_lvl+0xc4/0x170 (unreliable)
   panic+0x39c/0x584
   sysrq_handle_crash+0x80/0xe0
   __handle_sysrq+0x208/0x4bc
   [...]

This string is a statically allocated buffer populated during boot by arch code calling dump_stack_set_arch_desc(). For most platforms this is sufficient.

But the string may become inaccurate on the IBM PowerVM platform due to live migration between machine models and firmware versions. Stack dumps emitted after a migration reflect the machine on which the kernel booted, not necessarily the machine on which it is currently running. This is potentially confusing for anyone investigating kernel issues on the platform.

To address this, this series introduces a new function that safely updates the hardware description string and updates the powerpc pseries platform code to call it after a migration. The series also includes changes addressing minor latent issues identified during the implementation.

Platforms which do not need the new functionality remain unchanged.

For this initial version at least, the powerpc/pseries part includes some "self-test" code that 1. verifies that reconstructing the hardware description string late in boot matches the one that was built earlier, and 2. fully exercises the update path before any migrations occur. This could be dropped or made configurable in the future.

---
Nathan Lynch (5):
dump_stack: Make arch description buffer __ro_after_init
dump_stack: Allow update of arch description string at runtime powerpc/prom: Add CPU info to hardware description string later powerpc/pseries: Prepare pseries_add_hw_description() for runtime use powerpc/pseries: Update hardware description string after migration

 arch/powerpc/kernel/prom.c                | 12 +++--
 arch/powerpc/platforms/pseries/mobility.c |  5 ++
 arch/powerpc/platforms/pseries/pseries.h  |  1 +
 arch/powerpc/platforms/pseries/setup.c    | 80 +++++++++++++++++++++++++++++--
 include/linux/printk.h                    |  5 ++
 lib/dump_stack.c                          | 57 ++++++++++++++++++++--
 6 files changed, 146 insertions(+), 14 deletions(-)

---
base-commit: 44a1aad2fe6c10bfe0589d8047057b10a4c18a19
change-id: 20240111-update-dump-stack-arch-str-7f0880d23f30

Best regards,

Signature · 1 line
Nathan Lynch <[email protected]>
#1 lore raw

RFC PATCH 1/5 dump_stack: Make arch description buffer __ro_after_init

The static hardware description buffer is populated by arch code during boot and should not change afterwards, so mark it __ro_after_init.

---
 lib/dump_stack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/dump_stack.c +2 -1
index 83471e81501a..1057f102f6f2 100644
@@ -6,6 +6,7 @@
 
 #include <linux/kernel.h>
 #include <linux/buildid.h>
+#include <linux/cache.h>
 #include <linux/export.h>
 #include <linux/sched.h>
 #include <linux/sched/debug.h>
@@ -15,7 +16,7 @@
 #include <linux/utsname.h>
 #include <linux/stop_machine.h>
 
-static char dump_stack_arch_desc_str[128];
+static char dump_stack_arch_desc_str[128] __ro_after_init;
 
 /**
  * dump_stack_set_arch_desc - set arch-specific str to show with task dumps
Signature · 1 line
2.43.0
#2 lore raw

RFC PATCH 2/5 dump_stack: Allow update of arch description string at runtime

In reply to Nathan Lynch

The IBM PowerVM platform (targeted by powerpc/pseries) exposes the physical machine model and firmware version to partitions (guests), and this information is used to populate the arch description string, e.g.

  IBM,8408-E8E POWER8E (raw) 0x4b0201 0xf000004 \
    of:IBM,FW860.50 (SV860_146) hv:phyp pSeries

The platform supports live migration of partitions between different machine models and firmware versions, so the arch description string set at boot can become inaccurate, potentially misleading anyone who's analyzing stack traces produced after a migration.

Introduce a RCU-guarded pointer to the current arch description string, initializing it to the static buffer populated at boot. Add to dump_stack_print_info() a RCU read-side critical section that accesses the buffer through this pointer. The majority of architectures which don't need to update the string after boot incur only an additional indirection.

As for platforms which do need that ability, they can use dump_stack_update_arch_desc(), which allocates and formats a new buffer, updates the pointer, and if appropriate frees the previous buffer.

---
 include/linux/printk.h |  5 +++++
 lib/dump_stack.c       | 54 ++++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 55 insertions(+), 4 deletions(-)
include/linux/printk.h +5 -0
index 8ef499ab3c1e..6138ae019d2a 100644
@@ -187,6 +187,7 @@ u32 log_buf_len_get(void);
 void log_buf_vmcoreinfo_setup(void);
 void __init setup_log_buf(int early);
 __printf(1, 2) void dump_stack_set_arch_desc(const char *fmt, ...);
+__printf(1, 2) void dump_stack_update_arch_desc(const char *fmt, ...);
 void dump_stack_print_info(const char *log_lvl);
 void show_regs_print_info(const char *log_lvl);
 extern asmlinkage void dump_stack_lvl(const char *log_lvl) __cold;
@@ -253,6 +254,10 @@ static inline __printf(1, 2) void dump_stack_set_arch_desc(const char *fmt, ...)
 {
 }
 
+static inline __printf(1, 2) void dump_stack_update_arch_desc(const char *fmt, ...)
+{
+}
+
 static inline void dump_stack_print_info(const char *log_lvl)
 {
 }
lib/dump_stack.c +50 -4

93 lines not on this page · raw message

Signature · 1 line
2.43.0
#3 lore raw

RFC PATCH 3/5 powerpc/prom: Add CPU info to hardware description string later

In reply to Nathan Lynch

cur_cpu_spec->cpu_name is appended to ppc_hw_desc before cur_cpu_spec has taken on its final value. This is illustrated on pseries by comparing the CPU name as reported at boot ("POWER8E (raw)") to the contents of /proc/cpuinfo ("POWER8 (architected)"):

  $ dmesg | grep Hardware
  Hardware name: IBM,8408-E8E POWER8E (raw) 0x4b0201 0xf000004 \
    of:IBM,FW860.50 (SV860_146) hv:phyp pSeries
  $ grep -m 1 ^cpu /proc/cpuinfo
  cpu             : POWER8 (architected), altivec supported

Some 44x models would appear to be affected as well; see identical_pvr_fixup().

This results in incorrect CPU information in stack dumps -- ppc_hw_desc is an input to dump_stack_set_arch_desc().

Delay gathering the CPU name until after all potential calls to identify_cpu().

Fixes bd649d40e0f2 ("powerpc: Add PVR & CPU name to hardware description")
---
 arch/powerpc/kernel/prom.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/powerpc/kernel/prom.c +8 -4
index 0b5878c3125b..c12b4434336f 100644
@@ -327,6 +327,7 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
 					  void *data)
 {
 	const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+	const __be32 *cpu_version = NULL;
 	const __be32 *prop;
 	const __be32 *intserv;
 	int i, nthreads;
@@ -398,7 +399,7 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
 		prop = of_get_flat_dt_prop(node, "cpu-version", NULL);
 		if (prop && (be32_to_cpup(prop) & 0xff000000) == 0x0f000000) {
 			identify_cpu(0, be32_to_cpup(prop));
-			seq_buf_printf(&ppc_hw_desc, "0x%04x ", be32_to_cpup(prop));
+			cpu_version = prop;
 		}
 
 		check_cpu_feature_properties(node);
@@ -409,6 +410,12 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
 	}
 
 	identical_pvr_fixup(node);
+
+	// We can now add the CPU name & PVR to the hardware description
+	seq_buf_printf(&ppc_hw_desc, "%s 0x%04lx ", cur_cpu_spec->cpu_name, mfspr(SPRN_PVR));
+	if (cpu_version)
+		seq_buf_printf(&ppc_hw_desc, "0x%04x ", be32_to_cpup(cpu_version));
+
 	init_mmu_slb_size(node);
 
 #ifdef CONFIG_PPC64
@@ -846,9 +853,6 @@ void __init early_init_devtree(void *params)
 
 	dt_cpu_ftrs_scan();
 
-	// We can now add the CPU name & PVR to the hardware description
-	seq_buf_printf(&ppc_hw_desc, "%s 0x%04lx ", cur_cpu_spec->cpu_name, mfspr(SPRN_PVR));
-
 	/* Retrieve CPU related informations from the flat tree
 	 * (altivec support, boot CPU ID, ...)
 	 */
Signature · 1 line
2.43.0
#4 lore raw

RFC PATCH 4/5 powerpc/pseries: Prepare pseries_add_hw_description() for runtime use

In reply to Nathan Lynch

pseries_add_hw_description() will be used after boot to update the hardware description string emitted in stack dumps. Remove the __init and make it take a seq_buf * parameter instead of referencing ppc_hw_desc directly.

---
 arch/powerpc/platforms/pseries/setup.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
arch/powerpc/platforms/pseries/setup.c +5 -5
index ecea85c74c43..9ae1951f8312 100644
@@ -1007,7 +1007,7 @@ static void __init pSeries_cmo_feature_init(void)
 	pr_debug(" <- fw_cmo_feature_init()\n");
 }
 
-static void __init pseries_add_hw_description(void)
+static void pseries_add_hw_description(struct seq_buf *sb)
 {
 	struct device_node *dn;
 	const char *s;
@@ -1015,7 +1015,7 @@ static void __init pseries_add_hw_description(void)
 	dn = of_find_node_by_path("/openprom");
 	if (dn) {
 		if (of_property_read_string(dn, "model", &s) == 0)
-			seq_buf_printf(&ppc_hw_desc, "of:%s ", s);
+			seq_buf_printf(sb, "of:%s ", s);
 
 		of_node_put(dn);
 	}
@@ -1023,7 +1023,7 @@ static void __init pseries_add_hw_description(void)
 	dn = of_find_node_by_path("/hypervisor");
 	if (dn) {
 		if (of_property_read_string(dn, "compatible", &s) == 0)
-			seq_buf_printf(&ppc_hw_desc, "hv:%s ", s);
+			seq_buf_printf(sb, "hv:%s ", s);
 
 		of_node_put(dn);
 		return;
@@ -1031,7 +1031,7 @@ static void __init pseries_add_hw_description(void)
 
 	if (of_property_read_bool(of_root, "ibm,powervm-partition") ||
 	    of_property_read_bool(of_root, "ibm,fw-net-version"))
-		seq_buf_printf(&ppc_hw_desc, "hv:phyp ");
+		seq_buf_printf(sb, "hv:phyp ");
 }
 
 /*
@@ -1041,7 +1041,7 @@ static void __init pseries_init(void)
 {
 	pr_debug(" -> pseries_init()\n");
 
-	pseries_add_hw_description();
+	pseries_add_hw_description(&ppc_hw_desc);
 
 #ifdef CONFIG_HVC_CONSOLE
 	if (firmware_has_feature(FW_FEATURE_LPAR))
Signature · 1 line
2.43.0
#5 lore raw

RFC PATCH 5/5 powerpc/pseries: Update hardware description string after migration

In reply to Nathan Lynch

Introduce code that rebuilds the short hardware description printed by stack traces. This sort of duplicates some code from boot (prom.c mainly), but that code populates the string as early as possible using APIs that aren't available later. So sharing all the code between the boot and runtime versions isn't feasible.

To prevent "drift" between the boot and runtime versions, rebuild the description using the new runtime APIs in a late initcall and warn if it doesn't match the one built earlier. The initcall also invokes dump_stack_update_arch_desc() twice to fully exercise it before any partition migration occurs. These checks could be dropped or made configurable later.

Call pseries_update_hw_description() immediately after updating the device tree when resuming from a partition migration.

---
 arch/powerpc/platforms/pseries/mobility.c |  5 +++
 arch/powerpc/platforms/pseries/pseries.h  |  1 +
 arch/powerpc/platforms/pseries/setup.c    | 70 +++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
arch/powerpc/platforms/pseries/mobility.c +5 -0
index 1798f0f14d58..ff573cb5aee5 100644
@@ -378,6 +378,11 @@ void post_mobility_fixup(void)
 	rc = pseries_devicetree_update(MIGRATION_SCOPE);
 	if (rc)
 		pr_err("device tree update failed: %d\n", rc);
+	/*
+	 * Rebuild the hardware description printed in stack traces
+	 * using the updated device tree.
+	 */
+	pseries_update_hw_description();
 
 	cacheinfo_rebuild();
 
arch/powerpc/platforms/pseries/pseries.h +1 -0
index bba4ad192b0f..810a64fccc7e 100644
@@ -56,6 +56,7 @@ extern int dlpar_acquire_drc(u32 drc_index);
 extern int dlpar_release_drc(u32 drc_index);
 extern int dlpar_unisolate_drc(u32 drc_index);
 extern void post_mobility_fixup(void);
+void pseries_update_hw_description(void);
 
 void queue_hotplug_event(struct pseries_hp_errorlog *hp_errlog);
 int handle_dlpar_errorlog(struct pseries_hp_errorlog *hp_errlog);
arch/powerpc/platforms/pseries/setup.c +70 -0

76 lines not on this page · raw message

Signature · 1 line
2.43.0