Consider the size of the added CPU nodes in the kexec FDT
v2 cover Consider the size of the added CPU nodes in the kexec FDT
When adding CPUs to an already big system (test show it seems to start with more than 256 CPUs), the kernel is showing error messages when building the FDT for the kexec kernel (kdump or kexec).
It's worth to mention that the kdump kernel is reloaded after a CPU add operation.
The messages look like (property's name may vary):
10175.025675] Unable to add 32-64-bridge property: FDT_ERR_NOSPACE
This happens because the size of the FDT is computed based on the size of the FDT the kernel received at boot time. There is additional space added in kexec_extra_fdt_size_ppc64() for the added memory but nothing is done for the added CPUs.
This patch adds this feature so adding new CPUs will increase the size of the FDT for the kexec kernel.
To compute the additional size required, the number of CPU nodes of the initial FDT (the one the kernel receive at boot time) are recorded. When a kexec FDT is created, the number of CPU nodes in the current FDT is used to compute the additional size.
The first patch of this series is creating a variable provided by the boot code when parsing the initial FDT at boot time.
The second patch is computing the required additional space.
This has been tested on a PowerVM LPAR running with than 256 CPUs in shared mode, adding 320 CPUs to this LPAR.
Changes in v2:
- Fix build issue, moving definition in prom.h
Laurent Dufour (2):
powerpc: export the CPU node count
powerpc: Take in account addition CPU node when building kexec FDT
arch/powerpc/include/asm/prom.h | 1 + arch/powerpc/kernel/prom.c | 3 ++ arch/powerpc/kexec/file_load_64.c | 60 ++++++++++++++++++++++++++++++- 3 files changed, 63 insertions(+), 1 deletion(-)
Signature · 1 line
2.38.1
PATCH v2 1/2 powerpc: export the CPU node count
At boot time, the FDT is parsed to compute the number of CPUs. In addition count the number of CPU nodes and export it.
This is useful when building the FDT for a kexeced kernel since we need to take in account the CPU node added since the boot time during CPU hotplug operations.
--- arch/powerpc/include/asm/prom.h | 1 + arch/powerpc/kernel/prom.c | 3 +++ 2 files changed, 4 insertions(+)
arch/powerpc/include/asm/prom.h +1 -0
@@ -85,6 +85,7 @@ struct of_drc_info {extern int of_read_drc_info_cell (struct property **prop,const __be32 **curval,struct of_drc_info *data); +extern unsigned int boot_cpu_node_count;/* * There are two methods for telling firmware what our capabilities are.
arch/powerpc/kernel/prom.c +3 -0
@@ -72,6 +72,7 @@ int __initdata iommu_is_off;int __initdata iommu_force_on;unsigned long tce_alloc_start, tce_alloc_end;u64 ppc64_rma_size; +unsigned int boot_cpu_node_count__ro_after_init ;#endif static phys_addr_t first_memblock_size;static int __initdata boot_cpu_count;
@@ -335,6 +336,8 @@ static int __init early_init_dt_scan_cpus(unsigned long node,if (type ==NULL ||strcmp (type,"cpu" ) !=0 )return 0 ; + boot_cpu_node_count++; +/* Get physical cpuid */ intserv =of_get_flat_dt_prop (node,"ibm,ppc-interrupt-server#s" , &len);if (!intserv)
Signature · 1 line
2.38.1
PATCH v2 2/2 powerpc: Take in account addition CPU node when building kexec FDT
In reply to Laurent DufourOn a system with a large number of CPUs, the creation of the FDT for a kexec kernel may fail because the allocated FDT is not large enough.
When this happens, such a message is displayed on the console:
Unable to add ibm,processor-vadd-size property: FDT_ERR_NOSPACE
The property's name may change depending when the buffer overwrite is detected.
Obviously the created FDT is missing information, and it is expected that system dump or kexec kernel failed to run properly.
When the FDT is allocated, the size of the FDT the kernel received at boot time is used and an extra size can be applied. Currently, only memory added after boot time is taken in account, not the CPU nodes.
The extra size should take in account these additional CPU nodes and compute the required extra space. To achieve that, the size of a CPU node, including its subnode is computed once and multiplied by the number of additional CPU nodes.
The assumption is that the size of the CPU node is _same_ for all the node, the only variable part should be the name "PowerPC,POWERxx@##" where "##" may vary a bit.
--- arch/powerpc/kexec/file_load_64.c | 60 ++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)
arch/powerpc/kexec/file_load_64.c +59 -1
@@ -26,6 +26,7 @@#include <asm/firmware.h> #include <asm/kexec_ranges.h> #include <asm/crashdump-ppc64.h> +#include <asm/prom.h> struct umem_info {u64 *buf;/* data buffer for usable-memory property */
@@ -928,6 +929,46 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,return ret; } +/** +* get_cpu_node_size - Compute the size of a CPU node in the FDT. +* This should be done only once and the value is stored in +* a static variable. +* Returns the max size of a CPU node in the FDT. +*/ +static unsigned int cpu_node_size (void ) +{ +static unsigned int cpu_node_size; +struct device_node *dn; +struct property *pp; + +/* +* Don't compute it twice, we are assuming that the per CPU node size +* doesn't change during the system's life. +*/ +if (cpu_node_size) +return cpu_node_size; + + dn =of_find_node_by_type (NULL ,"cpu" ); +if (!dn) { +/* Unlikely to happen */ +WARN_ON_ONCE (1 ); +return 0 ; + } + +/* +* We compute the sub node size for a CPU node, assuming it +* will be the same for all. +*/ + cpu_node_size +=strlen (dn->name) +5 ; +for_each_property_of_node (dn, pp) { + cpu_node_size +=strlen (pp->name); + cpu_node_size += pp->length; + } + +of_node_put (dn); +return cpu_node_size; +} +/** * kexec_extra_fdt_size_ppc64 - Return the estimated additional size needed to * setup FDT for kexec/kdump kernel.
@@ -937,7 +978,10 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,*/ unsigned int kexec_extra_fdt_size_ppc64 (struct kimage *image) { +struct device_node *dn;u64 usm_entries; +unsigned int cpu_nodes =0 ; +unsigned int extra_size;if (image->type !=KEXEC_TYPE_CRASH )return 0 ;
@@ -949,7 +993,21 @@ unsigned int kexec_extra_fdt_size_ppc64(struct kimage *image)*/ usm_entries = ((memblock_end_of_DRAM () /drmem_lmb_size ()) + (2 * (resource_size (&crashk_res) /drmem_lmb_size ()))); -return (unsigned int )(usm_entries *sizeof (u64)); + + extra_size = (unsigned int )(usm_entries *sizeof (u64)); + +/* +* Get the number of CPU nodes in the current DT. This allows to +* reserve places for CPU nodes added since the boot time. +*/ +for_each_node_by_type (dn,"cpu" ) { + cpu_nodes++; + } + +if (cpu_nodes > boot_cpu_node_count) + extra_size += (cpu_nodes - boot_cpu_node_count) *cpu_node_size (); + +return extra_size; }/**
Signature · 1 line
2.38.1
Michael Ellerman
AppliedQuoted 8 lines · Laurent Dufour
When adding CPUs to an already big system (test show it seems to start with more than 256 CPUs), the kernel is showing error messages when building the FDT for the kexec kernel (kdump or kexec).
It's worth to mention that the kdump kernel is reloaded after a CPU add operation.
[...]
Applied to powerpc/next.
[1/2] powerpc: export the CPU node count
https://git.kernel.org/powerpc/c/e13d23a404f2e6dfaf8b1ef7d161a0836fce4fa5 [2/2] powerpc: Take in account addition CPU node when building kexec FDT https://git.kernel.org/powerpc/c/340a4a9f8773e102cc5ef531665970a686dfa245
cheers