In kernel handling of CPU hotplug events for crash kernel
Subsystems (6)
v8 cover In kernel handling of CPU hotplug events for crash kernel
The Problem:
============
Post hotplug/DLPAR events the capture kernel holds stale information about the system. Dump collection with stale capture kernel might end up in dump capture failure or an inaccurate dump collection.
Existing solution:
==================
The existing solution to keep the capture kernel up-to-date by monitoring hotplug event via udev rule and trigger a full capture kernel reload for every hotplug event.
Shortcomings:
------------------------------------------------
- Leaves a window where kernel crash might not lead to a successful dump collection.
- Reloading all kexec components for each hotplug is inefficient.
- udev rules are prone to races if hotplug events are frequent.
More about issues with an existing solution is posted here:
- https://lkml.org/lkml/2020/12/14/532
- https://lists.ozlabs.org/pipermail/linuxppc-dev/2022-February/240254.html
Proposed Solution:
==================
Instead of reloading all kexec segments on hotplug event, this patch series focuses on updating only the relevant kexec segment. Once the kexec segments are loaded in the kernel reserved area then an arch-specific hotplug handler will update the relevant kexec segment based on hotplug event type.
Series Dependecies
==================
This patch series implements the crash hotplug handler on PowerPC. The generic for crash hotplug update is introduced by https://lkml.org/lkml/2023/1/18/1420 patch series.
Git tree for testing:
=====================
The below git tree has this patch series applied on top of dependent patch series.
https://github.com/sourabhjains/linux/commits/in-kernel-crash-update
Note: only kexec_file_load syscall will work. For kexec_load mirnor changes are required in kexec tool.
To realise the feature the kdump udev rules must be disabled for CPU/Memory hotplug events. Comment out the below line in kdump udev rule file:
RHEL: /usr/lib/udev/rules.d/98-kexec.rules
#SUBSYSTEM=="cpu", ACTION=="online", GOTO="kdump_reload_cpu"
#SUBSYSTEM=="memory", ACTION=="online", GOTO="kdump_reload_mem" #SUBSYSTEM=="memory", ACTION=="offline", GOTO="kdump_reload_mem"
SLES: /usr/lib/kdump/70-kdump.rules
#SUBSYSTEM=="memory", ACTION=="add|remove", GOTO="kdump_try_restart" #SUBSYSTEM=="cpu", ACTION=="online", GOTO="kdump_try_restart"
---
Changelog:
v7 -> v8:
- Restrict fdt_index initialization to machine_kexec_post_load it work for both kexec_load and kexec_file_load.[3/8] Laurent Dufour
- Updated the logic to find the number of offline core. [6/8]
- Changed the logic to find the elfcore program header to accommodate future memory ranges due memory hotplug events. [8/8]
v6 -> v7
- added a new config to configure this feature
- pass hotplug action type to arch specific handler
v5 -> v6
- Added crash memory hotplug support
v4 -> v5:
- Replace COFNIG_CRASH_HOTPLUG with CONFIG_HOTPLUG_CPU.
- Move fdt segment identification for kexec_load case to load path instead of crash hotplug handler
- Keep new attribute defined under kimage_arch to track FDT segment under CONFIG_HOTPLUG_CPU config.
v3 -> v4:
- Update the logic to find the additional space needed for hotadd CPUs post kexec load. Refer "[RFC v4 PATCH 4/5] powerpc/crash hp: add crash hotplug support for kexec_file_load" patch to know more about the change.
- Fix a couple of typo.
- Replace pr_err to pr_info_once to warn user about memory hotplug support.
- In crash hotplug handle exit the for loop if FDT segment is found.
v2 -> v3
- Move fdt_index and fdt_index_vaild variables to kimage_arch struct.
- Rebase patche on top of https://lkml.org/lkml/2022/3/3/674 [v5]
- Fixed warning reported by checpatch script
v1 -> v2:
- Use generic hotplug handler introduced by https://lkml.org/lkml/2022/2/9/1406, a significant change from v1.
Sourabh Jain (8):
powerpc/kexec: turn some static helper functions public powerpc/crash hp: introduce a new config option CRASH_HOTPLUG powerpc/crash: update kimage_arch struct
crash: add phdr for possible CPUs in elfcorehdr
crash: pass hotplug action type to arch crash hotplug handler powerpc/crash: add crash CPU hotplug support
crash: forward memory_notify args to arch crash hotplug handler powerpc/kexec: add crash memory hotplug support
arch/powerpc/Kconfig | 12 + arch/powerpc/include/asm/kexec.h | 18 ++ arch/powerpc/include/asm/kexec_ranges.h | 1 + arch/powerpc/kexec/core_64.c | 335 ++++++++++++++++++++++++ arch/powerpc/kexec/elf_64.c | 19 +- arch/powerpc/kexec/file_load_64.c | 237 +++-------------- arch/powerpc/kexec/ranges.c | 60 +++++ arch/x86/include/asm/kexec.h | 3 +- arch/x86/kernel/crash.c | 5 +- include/linux/kexec.h | 6 +- kernel/crash_core.c | 23 +- 11 files changed, 502 insertions(+), 217 deletions(-)
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 1/8 powerpc/kexec: turn some static helper functions public
Move update_cpus_node and get_crash_memory_ranges functions from kexec/file_load.c to kexec/core_64.c to make these functions usable by other kexec compoenets.
Later in the series, both functions are used for in-kernel updates to kexec segments in the event of CPU/Memory hotplug for both kexec_load and kexec_file_load system call.
No functional change intended.
--- arch/powerpc/include/asm/kexec.h | 6 ++ arch/powerpc/kexec/core_64.c | 166 ++++++++++++++++++++++++++++++ arch/powerpc/kexec/file_load_64.c | 162 ----------------------------- 3 files changed, 172 insertions(+), 162 deletions(-)
arch/powerpc/include/asm/kexec.h +6 -0
@@ -99,6 +99,12 @@ void relocate_new_kernel(unsigned long indirection_page, unsigned long reboot_covoid kexec_copy_flush (struct kimage *image); +#ifdef CONFIG_PPC64 +struct crash_mem ; +int update_cpus_node (void *fdt); +int get_crash_memory_ranges (struct crash_mem **mem_ranges); +#endif +#if defined(CONFIG_CRASH_DUMP ) && defined(CONFIG_PPC_RTAS )void crash_free_reserved_phys_range (unsigned long begin,unsigned long end);#define crash_free_reserved_phys_range crash_free_reserved_phys_range
arch/powerpc/kexec/core_64.c +166 -0
184 lines not on this page · raw message
arch/powerpc/kexec/file_load_64.c +0 -162
174 lines not on this page · raw message
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 2/8 powerpc/crash hp: introduce a new config option CRASH_HOTPLUG
In reply to Sourabh JainThe config option CRASH_HOTPLUG enables in kernel update to kexec segments due to CPU/Memory hotplug events. By default it is disabled.
--- arch/powerpc/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/powerpc/Kconfig +12 -0
@@ -668,6 +668,18 @@ config CRASH_DUMP The same kernel binary can be used as production kernel and dump capture kernel. +config CRASH_HOTPLUG + bool "Update crash capture system on CPU/Memory hotplug event" + default n + depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) + help + In kernel update to relevant kexec segments due to change + in system configuration, rather reloading all the kexec + segments again from userspace by monitoring CPU/Memory + hotplug event in userspace using udev. + + If unsure, say Y. + config FA_DUMP bool "Firmware-assisted dump" depends on PPC64 && (PPC_RTAS || PPC_POWERNV)
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 3/8 powerpc/crash: update kimage_arch struct
In reply to Sourabh JainAdd a new member "fdt_index" to kimage_arch struct to hold the index of the FDT (Flattened Device Tree) segment in the kexec segment array.
Having direct access to FDT segment will help arch crash hotplug handler to avoid looping kexec segment array to identify the FDT segment index for every FDT update on hotplug events.
--- arch/powerpc/include/asm/kexec.h | 7 +++++++ arch/powerpc/kexec/core_64.c | 33 ++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
arch/powerpc/include/asm/kexec.h +7 -0
@@ -103,6 +103,10 @@ void kexec_copy_flush(struct kimage *image);struct crash_mem ;int update_cpus_node (void *fdt);int get_crash_memory_ranges (struct crash_mem **mem_ranges); +#if defined(CONFIG_CRASH_HOTPLUG ) +int machine_kexec_post_load (struct kimage *image); +#define machine_kexec_post_load machine_kexec_post_load +#endif #endif #if defined(CONFIG_CRASH_DUMP ) && defined(CONFIG_PPC_RTAS )
@@ -118,6 +122,9 @@ extern const struct kexec_file_ops kexec_elf64_ops;struct kimage_arch {struct crash_mem *exclude_ranges; +#if defined(CONFIG_CRASH_HOTPLUG ) +int fdt_index; +#endif unsigned long backup_start;void *backup_buf;void *fdt;
arch/powerpc/kexec/core_64.c +33 -0
@@ -77,6 +77,39 @@ int machine_kexec_prepare(struct kimage *image)return 0 ; } +#if defined(CONFIG_CRASH_HOTPLUG ) +/* +* Find the index of the FDT segment in kexec segment array and assign +* it to kimage's member fdt_index to have direct access to FDT +* segment later on. +*/ +int machine_kexec_post_load (struct kimage *kimage) +{ +int i; +void *ptr; +unsigned long mem; + +/* Mark fdt_index invalid */ + kimage->arch.fdt_index =-1 ; + +/* fdt_index remains invalid for if it not crash kernel load */ +if (kimage->type !=KEXEC_TYPE_CRASH ) +return 0 ; + +for (i =0 ; i < kimage->nr_segments; i++) { + mem = kimage->segment[i].mem; + ptr =__va (mem); + +if (ptr &&fdt_magic (ptr) ==FDT_MAGIC ) { + kimage->arch.fdt_index = i; +break ; + } + } + +return 0 ; +} +#endif +/* Called during kexec sequence with MMU off */ static notrace void copy_segments(unsigned long ind) {
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 4/8 crash: add phdr for possible CPUs in elfcorehdr
In reply to Sourabh JainOn architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for kexec_load case.
--- kernel/crash_core.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
kernel/crash_core.c +6 -3
@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize =sizeof (Elf64_Ehdr); ehdr->e_phentsize =sizeof (Elf64_Phdr); -/* Prepare one phdr of type PT_NOTE for each present CPU */ -for_each_present_cpu (cpu) { +/* Prepare one phdr of type PT_NOTE for possible CPU with crash note. */ +for_each_possible_cpu (cpu) {#ifdef CONFIG_CRASH_HOTPLUG if (IS_ENABLED (CONFIG_HOTPLUG_CPU )) {/* Skip the soon-to-be offlined cpu */
@@ -373,8 +373,11 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, continue; }#endif - phdr->p_type =PT_NOTE ; notes_addr =per_cpu_ptr_to_phys (per_cpu_ptr (crash_notes, cpu)); +if (!notes_addr) +continue ; + + phdr->p_type =PT_NOTE ; phdr->p_offset = phdr->p_paddr = notes_addr; phdr->p_filesz = phdr->p_memsz =sizeof (note_buf_t); (ehdr->e_phnum)++;
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 4/8] crash: add phdr for possible CPUs in elfcorehdr
Quoted 17 lines · Sourabh Jain
On architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for
kexec_loadcase.Signed-off-by Sourabh Jain <[email protected]>--- kernel/crash_core.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)kernel/crash_core.c +0
-0
kernel/crash_core.c int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, · line 368
Quoted 7 lines
kernel/crash_core.c +2
-2@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize =sizeof (Elf64_Ehdr); ehdr->e_phentsize =sizeof (Elf64_Phdr); -/* Prepare one phdr of type PT_NOTE for each present CPU */ -for_each_present_cpu (cpu) { +/* Prepare one phdr of type PT_NOTE for possible CPU with crash note. */ +for_each_possible_cpu (cpu) {
Sourabh,
Thomas Gleixner is suggesting moving away from for_each_present_cpu() to for_each_online_cpu(). Using for_each_online_cpu() is going to the minimum number of needed, whereas your approach of for_each_possible_cpu() would be to the maximum number needed.
What would be the ramifications to ppc for moving towards for_each_online_cpu()?
In my next patch series, I have finally figured out how to use cpuhp framework to where it is possible to use for_each_online_cpu() here, but that is at odds with your changes here.
Thanks,
eric
Quoted 16 lines
kernel/crash_core.c +4
-1#ifdef CONFIG_CRASH_HOTPLUG if (IS_ENABLED (CONFIG_HOTPLUG_CPU )) {/* Skip the soon-to-be offlined cpu */ @@ -373,8 +373,11 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, continue; }#endif - phdr->p_type =PT_NOTE ; notes_addr =per_cpu_ptr_to_phys (per_cpu_ptr (crash_notes, cpu)); +if (!notes_addr) +continue ; + + phdr->p_type =PT_NOTE ; phdr->p_offset = phdr->p_paddr = notes_addr; phdr->p_filesz = phdr->p_memsz =sizeof (note_buf_t); (ehdr->e_phnum)++;
Re: [PATCH v8 4/8] crash: add phdr for possible CPUs in elfcorehdr
Quoted 30 lines · Eric DeVolder
Quoted 17 lines · Sourabh Jain
On architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for
kexec_loadcase.Signed-off-by Sourabh Jain <[email protected]>---
kernel/crash_core.c| 9 ++++++---1 file changed, 6 insertions(+), 3 deletions(-)kernel/crash_core.c +0
-0kernel/crash_core.c int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, · line 367
Quoted 6 lines
kernel/crash_core.c +2
-2@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize = sizeof(Elf64_Ehdr); ehdr->e_phentsize = sizeof(Elf64_Phdr); - /* Prepare one phdr of type PT_NOTE for each present CPU */ - for_each_present_cpu(cpu) { + /* Prepare one phdr of type PT_NOTE for possible CPU with crash note. */ + for_each_possible_cpu(cpu) {Sourabh,
Thomas Gleixner is suggesting moving away fromfor_each_present_cpu()tofor_each_online_cpu(). Usingfor_each_online_cpu()is going to the minimum number of needed, whereas your approach offor_each_possible_cpu()would be to the maximum number needed.
Quoted 8 lines
What would be the ramifications to ppc for moving towards
for_each_online_cpu()?In my next patch series, I have finally figured out how to use cpuhp framework to where it is possible to use
for_each_online_cpu()here, but that is at odds with your changes here.Thanks,
eric
Without knowing the ramifications of changing to for_each_online_cpu(), I currently am using the following:
kernel/crash_core.c +3 -0
@@ -366,6 +366,9 @@ int crash_prepare_elf64_headers(struct crash_mem *mem, int n/* Prepare one phdr of type PT_NOTE for each present CPU */ for_each_present_cpu (cpu) { +if (IS_ENABLED (CONFIG_CRASH_HOTPLUG )) { +if (!cpu_online (cpu))continue ; + } phdr->p_type =PT_NOTE ; notes_addr =per_cpu_ptr_to_phys (per_cpu_ptr (crash_notes, cpu)); phdr->p_offset = phdr->p_paddr = notes_addr;
Thomas points out that the above can be simply the for_each_online_cpu(), but again I'm not sure how that impacts ppc, which appears to layout all possible cpus rather than just online ones. How are present but not online cpus handled by crash analysis utility?
eric
Quoted 17 lines
Quoted 7 lines
kernel/crash_core.c +0
-0#ifdef CONFIG_CRASH_HOTPLUG if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) { /* Skip the soon-to-be offlined cpu */@@ -373,8 +373,11 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, continue; }kernel/crash_core.c int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, · line 383
Quoted 10 lines
kernel/crash_core.c +4
-1#endif - phdr->p_type = PT_NOTE; notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu)); + if (!notes_addr) + continue; + + phdr->p_type = PT_NOTE; phdr->p_offset = phdr->p_paddr = notes_addr; phdr->p_filesz = phdr->p_memsz = sizeof(note_buf_t); (ehdr->e_phnum)++;
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 4/8] crash: add phdr for possible CPUs in elfcorehdr
Quoted 51 lines · Eric DeVolder
Quoted 32 lines · Eric DeVolder
Quoted 23 lines · Sourabh Jain
On architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for
kexec_loadcase.Signed-off-by Sourabh Jain <[email protected]>---
kernel/crash_core.c| 9 ++++++---1 file changed, 6 insertions(+), 3 deletions(-)kernel/crash_core.c +1
-2@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize = sizeof(Elf64_Ehdr); ehdr->e_phentsize = sizeof(Elf64_Phdr); - /* Prepare one phdr of type PT_NOTE for each present CPU */ - for_each_present_cpu(cpu) { + /* Prepare one phdr of type PT_NOTE for possible CPU with crashQuoted 2 lines
note. */
+for_each_possible_cpu(cpu){Sourabh,
Thomas Gleixner is suggesting moving away fromfor_each_present_cpu()tofor_each_online_cpu(). Usingfor_each_online_cpu()is going to the minimum number of needed, whereas your approach offor_each_possible_cpu()would be to the maximum number needed.Quoted 8 lines
What would be the ramifications to ppc for moving towards
for_each_online_cpu()?In my next patch series, I have finally figured out how to use cpuhp framework to where it is possible to use
for_each_online_cpu()here, but that is at odds with your changes here.Thanks,
ericWithout knowing the ramifications of changing to
for_each_online_cpu(), I currently am using the following:kernel/crash_core.c +0
-0
kernel/crash_core.c int crash_prepare_elf64_headers(struct crash_mem · line 372
Quoted 7 lines
kernel/crash_core.c +3
-0@@ -366,6 +366,9 @@ int crash_prepare_elf64_headers(struct crash_mem *mem, int n /* Prepare one phdr of type PT_NOTE for each present CPU */ for_each_present_cpu(cpu) { + if (IS_ENABLED(CONFIG_CRASH_HOTPLUG)) { + if (!cpu_online(cpu)) continue; + }
How about let the arch decide the list of CPUs they want to pack in? Because on PowerPC the crash notes are created for possible CPUs and we can utilize this to avoid re-generating elfcorehdr for every hotplug operation.
Quoted 10 lines
kernel/crash_core.c +0
-0phdr->p_type =PT_NOTE ; notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
phdr->p_offset=phdr->p_paddr=notes_addr;Thomas points out that the above can be simply the
for_each_online_cpu(), but again I'm not sure how that impacts ppc,which appears to layout all possible cpus rather
than just online ones. How are present but not online cpus handled by crash analysis utility?
As per my testing all worked fine if we replace for_each_present_cpu with for_each_online_cpu but again I don't know the reason why it worked. I will investigate it and let you know.
How packing PT_LOAD for present CPU is impacting x86? Because on PowerPC when system is on crash path it only populates the crash notes for online CPUs, and skip all other CPUs.
- Sourabh Jain
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 4/8] crash: add phdr for possible CPUs in elfcorehdr
In reply to Eric DeVolderQuoted 32 lines · Eric DeVolder
Quoted 20 lines · Sourabh Jain
On architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for
kexec_loadcase.Signed-off-by Sourabh Jain <[email protected]>---
kernel/crash_core.c| 9 ++++++---1 file changed, 6 insertions(+), 3 deletions(-)kernel/crash_core.c +0
-0@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize = sizeof(Elf64_Ehdr); ehdr->e_phentsize = sizeof(Elf64_Phdr);Quoted 5 lines
- /* Prepare one phdr of type
PT_NOTEfor each present CPU */ -for_each_present_cpu(cpu){
+ /* Prepare one phdr of typePT_NOTEfor possible CPU with crash note. */
+for_each_possible_cpu(cpu){Sourabh,
Thomas Gleixner is suggesting moving away fromfor_each_present_cpu()tofor_each_online_cpu(). Usingfor_each_online_cpu()is going to the minimum number of needed, whereas your approach offor_each_possible_cpu()would be to the maximum number needed.
Quoted 1 line
What would be the ramifications to ppc for moving towards
for_each_online_cpu()?
I was in the impression that if CPU notes are missing for offline CPUs in the elfcorehdr then makedumpfile will mess up the CPU IDs.
But somehow replacing for_each_present_cpu with for_each_online_cpu worked on PowerPC, even after disabling a couple of CPUs.
So things are fine if we pack PT_LOAD for online CPUs instead of present CPUs but, I need to investigate how makedumpfile is able to map PT_LOAD to online CPUs.
- Sourabh Jain
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 4/8] crash: add phdr for possible CPUs in elfcorehdr
In reply to Eric DeVolderQuoted 32 lines · Eric DeVolder
Quoted 20 lines · Sourabh Jain
On architectures like PowerPC the crash notes are available for all possible CPUs. So let's populate the elfcorehdr for all possible CPUs having crash notes to avoid updating elfcorehdr during in-kernel crash update on CPU hotplug events.
The similar technique is used in kexec-tool for
kexec_loadcase.Signed-off-by Sourabh Jain <[email protected]>---
kernel/crash_core.c| 9 ++++++---1 file changed, 6 insertions(+), 3 deletions(-)kernel/crash_core.c +0
-0@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem, ehdr->e_ehsize = sizeof(Elf64_Ehdr); ehdr->e_phentsize = sizeof(Elf64_Phdr);Quoted 5 lines
- /* Prepare one phdr of type
PT_NOTEfor each present CPU */ -for_each_present_cpu(cpu){
+ /* Prepare one phdr of typePT_NOTEfor possible CPU with crash note. */
+for_each_possible_cpu(cpu){Sourabh,
Thomas Gleixner is suggesting moving away fromfor_each_present_cpu()tofor_each_online_cpu(). Usingfor_each_online_cpu()is going to the minimum number of needed, whereas your approach offor_each_possible_cpu()would be to the maximum number needed.
Quoted 3 lines
What would be the ramifications to ppc for moving towards
for_each_online_cpu()?In my next patch series, I have finally figured out how to use cpuhp framework to where it is possible to use
for_each_online_cpu()here, but that is at odds with your changes here.
I was in the impression that if CPU notes are missing for offline CPUs in the elfcorehdr then makedumpfile will mess up the CPU IDs.
But somehow replacing for_each_present_cpu with for_each_online_cpu worked on PowerPC, even after disabling a couple of CPUs.
So things are fine if we pack PT_LOAD for online CPUs instead of present CPUs but, I need to investigate how makedumpfile is able to map PT_LOAD to online CPUs.
- Sourabh Jain
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 5/8 crash: pass hotplug action type to arch crash hotplug handler
In reply to Sourabh JainOn PowerPC, the crash update action for CPU and Memory hotplug events is not the same. Since arch_crash_handle_hotplug_event is a common crash hotplug handler for both CPU and Memory hot un/plug events, it is hard to differentiate which hotplug event triggered the arch crash hotplug handler and decide the crash update action accordingly.
Pass the hotplug action type to arch specific crash hotplug handler so that corresponding crash update action can be taken based on the hotplug action type.
--- arch/x86/include/asm/kexec.h | 2 +- arch/x86/kernel/crash.c | 3 ++- include/linux/kexec.h | 2 +- kernel/crash_core.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/include/asm/kexec.h +1 -1
@@ -213,7 +213,7 @@ extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;extern void kdump_nmi_shootdown_cpus (void );#ifdef CONFIG_CRASH_HOTPLUG -void arch_crash_handle_hotplug_event (struct kimage *image); +void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action);#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event#ifdef CONFIG_HOTPLUG_CPU
arch/x86/kernel/crash.c +2 -1
@@ -463,12 +463,13 @@ int crash_load_segments(struct kimage *image)/** * arch_crash_handle_hotplug_event() - Handle hotplug elfcorehdr changes * @image: the active struct kimage +* @hp_action: the hot un/plug action being handled * * To accurately reflect hot un/plug changes, the new elfcorehdr * is prepared in a kernel buffer, and then it is written on top * of the existing/old elfcorehdr. */ -void arch_crash_handle_hotplug_event (struct kimage *image) +void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action) {void *elfbuf =NULL , *old_elfcorehdr;unsigned long nr_mem_ranges;
include/linux/kexec.h +1 -1
@@ -509,7 +509,7 @@ static inline void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages) {#endif #ifndef arch_crash_handle_hotplug_event -static inline void arch_crash_handle_hotplug_event (struct kimage *image) { } +static inline void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action) { }#endif #else /* !CONFIG_KEXEC_CORE */
kernel/crash_core.c +1 -1
@@ -789,7 +789,7 @@ static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu) cpu :KEXEC_CRASH_HP_INVALID_CPU ;/* Now invoke arch-specific update handler */ -arch_crash_handle_hotplug_event (image); +arch_crash_handle_hotplug_event (image, hp_action);/* No longer handling a hotplug event */ image->hotplug_event = false;
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 6/8 powerpc/crash: add crash CPU hotplug support
In reply to Sourabh JainIntroduce powerpc crash hotplug handler to update the necessary kexec segments on CPU/Memory hotplug events. A common crash hotplug handler is triggered from generic infrastructure for both CPU/Memory hot un/plugged events but in this patch, only CPU hot un/plugged events are introduced. The memory hotplug support is added in upcoming patches.
The elfcorehdr segment is used to exchange the CPU and other dump related information between the kernels. Ideally, the elfcorehdr segment needs to be recreated on CPU hotplug events to reflect the changes. But on PowerPC elfcorehdr is built with all possible CPUs instead of just online CPUs hence no elfcorehdr segment update/recreation is needed.
In addition to elfcorehdr there is one more kexec segment that holds CPU data FDT (Flattened Device Tree). To boot the PowerPC kernel the crashing CPU has to be part of the FDT segment. If the kdump kernel doesn't find the crashing CPU in the FDT segment, it fails to boot.
The only action needed on PowerPC to handle the crash CPU hotplug event is to add hot added CPUs in the FDT segment to avoid kdump kernel boot failure in case the system crashes on hot added CPU.
So for the CPU hot add events, the FDT segment is updated with hot added CPU and Since there is no need to remove the hot unplugged CPUs from the FDT segment hence no action taken on CPU hot remove event in PowerPC arch crash hotplug handler.
To accommodate a growing number of CPUs, FDT is built with additional buffer space to ensure that it can hold all possible CPUs.
The changes done here will also work for kexec_load system call given that the kexec tool builds the FDT segment with additional space to accommodate all possible CPUs as it is done for kexec_file_load system call in the kernel.
Since memory crash hotplug support is not there yet the crash hotplug handler simply warns the user and returns.
--- arch/powerpc/include/asm/kexec.h | 5 ++ arch/powerpc/kexec/core_64.c | 82 +++++++++++++++++++++++++++++++ arch/powerpc/kexec/elf_64.c | 19 ++++++- arch/powerpc/kexec/file_load_64.c | 39 --------------- 4 files changed, 105 insertions(+), 40 deletions(-)
arch/powerpc/include/asm/kexec.h +5 -0
@@ -101,11 +101,16 @@ void kexec_copy_flush(struct kimage *image);#ifdef CONFIG_PPC64 struct crash_mem ; +unsigned int cpu_node_size (void );int update_cpus_node (void *fdt);int get_crash_memory_ranges (struct crash_mem **mem_ranges);#if defined(CONFIG_CRASH_HOTPLUG )int machine_kexec_post_load (struct kimage *image);#define machine_kexec_post_load machine_kexec_post_load + +void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action); +#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event +#endif #endif
arch/powerpc/kexec/core_64.c +82 -0
94 lines not on this page · raw message
arch/powerpc/kexec/elf_64.c +18 -1
@@ -42,6 +42,9 @@ static void *elf64_load(struct kimage *image, char *kernel_buf,struct kexec_buf pbuf = { .image = image, .buf_min =0 , .buf_max = ppc64_rma_size, .top_down = true, .mem =KEXEC_BUF_MEM_UNKNOWN }; +#if defined(CONFIG_CRASH_HOTPLUG ) +unsigned int offline_core_count; +#endif ret =kexec_build_elf_info (kernel_buf, kernel_len, &ehdr, &elf_info);if (ret)
@@ -119,10 +122,24 @@ static void *elf64_load(struct kimage *image, char *kernel_buf,fdt_pack (fdt); kbuf.buffer = fdt; - kbuf.bufsz = kbuf.memsz =fdt_totalsize (fdt); + kbuf.bufsz =fdt_totalsize (fdt); kbuf.buf_align =PAGE_SIZE ; kbuf.top_down = true; kbuf.mem =KEXEC_BUF_MEM_UNKNOWN ; +#if defined(CONFIG_CRASH_HOTPLUG ) +if (image->type ==KEXEC_TYPE_CRASH ) { +/* Calculate the buffer space needed to accommodate more CPU nodes in +* crash FDT post capture kernel load due to CPU hotplug events. +*/ + offline_core_count = (num_possible_cpus () -num_present_cpus ()) / threads_per_core; + kbuf.memsz =fdt_totalsize (fdt) + offline_core_count *cpu_node_size (); +fdt_set_totalsize (fdt, kbuf.memsz); + } else +#endif + { + kbuf.memsz =fdt_totalsize (fdt); + } + ret =kexec_add_buffer (&kbuf);if (ret) goto out_free_fdt;
arch/powerpc/kexec/file_load_64.c +0 -39
@@ -854,45 +854,6 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,return ret; } -/** -* get_cpu_node_size - Compute the size of a CPU node in the FDT. -* This should be done only once and the value is stored in -* a static variable. -* Returns the max size of a CPU node in the FDT. -*/ -static unsigned int cpu_node_size (void ) -{ -static unsigned int size; -struct device_node *dn; -struct property *pp; - -/* -* Don't compute it twice, we are assuming that the per CPU node size -* doesn't change during the system's life. -*/ -if (size) -return size; - - dn =of_find_node_by_type (NULL ,"cpu" ); -if (WARN_ON_ONCE (!dn)) { -// Unlikely to happen -return 0 ; - } - -/* -* We compute the sub node size for a CPU node, assuming it -* will be the same for all. -*/ - size +=strlen (dn->name) +5 ; -for_each_property_of_node (dn, pp) { - size +=strlen (pp->name); - size += pp->length; - } - -of_node_put (dn); -return size; -} -/** * kexec_extra_fdt_size_ppc64 - Return the estimated additional size needed to * setup FDT for kexec/kdump kernel.
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 7/8 crash: forward memory_notify args to arch crash hotplug handler
In reply to Sourabh JainThe way memory hot remove is handled on PowerPC, it is hard to update the elfcorehdr without memory_notify args.
On PowePC memblock data structure is used to prepare elfcorehdr for kdump. Since the notifier used for memory hotplug crash handler get initiated before the memblock data structure update happens (as depicted below), the newly prepared elfcorehdr still holds the old memory regions. So if the system crash with obsolete elfcorehdr, makedumpfile failed to collect vmcore.
Sequence of actions done on PowerPC to serve the memory hot remove:
Initiate memory hot remove
|
v
offline pages
|
v
initiate memory notify call chain
for MEM_OFFLINE event.
(same is used for crash update)
|
v
prepare new elfcorehdr for kdump using
memblock data structure
|
vupdate memblock data structure
How passing memory_notify to arch crash hotplug handler will help?
memory_notify holds the start PFN and page count, with that base address and size of hot unplugged memory can calculated and same can be use to avoid hot unplugged memory region to get added in the elfcorehdr.
--- arch/powerpc/include/asm/kexec.h | 2 +- arch/powerpc/kexec/core_64.c | 3 ++- arch/x86/include/asm/kexec.h | 3 ++- arch/x86/kernel/crash.c | 4 +++- include/linux/kexec.h | 6 +++++- kernel/crash_core.c | 14 +++++++------- 6 files changed, 20 insertions(+), 12 deletions(-)
arch/powerpc/include/asm/kexec.h +1 -1
@@ -108,7 +108,7 @@ int get_crash_memory_ranges(struct crash_mem **mem_ranges);int machine_kexec_post_load (struct kimage *image);#define machine_kexec_post_load machine_kexec_post_load -void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action); +void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action,void *arg);#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event#endif
arch/powerpc/kexec/core_64.c +2 -1
@@ -620,11 +620,12 @@ int update_cpus_node(void *fdt)* arch_crash_hotplug_handler() - Handle hotplug kexec segements changes FDT, elfcorehdr * @image: the active struct kimage * @hp_action: the hot un/plug action being handled +* @arg: struct memory_notify data handler * * To accurately reflect CPU hot un/plug changes, the FDT must be updated with the * new list of CPUs. */ -void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action) +void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action,void *arg) {void *fdt;
arch/x86/include/asm/kexec.h +2 -1
@@ -213,7 +213,8 @@ extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;extern void kdump_nmi_shootdown_cpus (void );#ifdef CONFIG_CRASH_HOTPLUG -void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action); +void arch_crash_handle_hotplug_event (struct kimage *image, +unsigned int hp_action,void *arg);#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event#ifdef CONFIG_HOTPLUG_CPU
arch/x86/kernel/crash.c +3 -1
@@ -464,12 +464,14 @@ int crash_load_segments(struct kimage *image)* arch_crash_handle_hotplug_event() - Handle hotplug elfcorehdr changes * @image: the active struct kimage * @hp_action: the hot un/plug action being handled +* @arg: struct memory_notify data handler * * To accurately reflect hot un/plug changes, the new elfcorehdr * is prepared in a kernel buffer, and then it is written on top * of the existing/old elfcorehdr. */ -void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action) +void arch_crash_handle_hotplug_event (struct kimage *image, +unsigned int hp_action,void *arg) {void *elfbuf =NULL , *old_elfcorehdr;unsigned long nr_mem_ranges;
include/linux/kexec.h +5 -1
@@ -509,7 +509,11 @@ static inline void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages) {#endif #ifndef arch_crash_handle_hotplug_event -static inline void arch_crash_handle_hotplug_event (struct kimage *image,unsigned int hp_action) { } +static inline void arch_crash_handle_hotplug_event (struct kimage *image, +unsigned int hp_action, +void *arg) +{ +}#endif #else /* !CONFIG_KEXEC_CORE */
kernel/crash_core.c +7 -7
@@ -727,7 +727,7 @@ subsys_initcall(crash_save_vmcoreinfo_init);* list of segments it checks (since the elfcorehdr changes and thus * would require an update to purgatory itself to update the digest). */ -static void handle_hotplug_event (unsigned int hp_action,unsigned int cpu) +static void handle_hotplug_event (unsigned int hp_action,unsigned int cpu,void *arg) {/* Obtain lock while changing crash information */ if (kexec_trylock ()) {
@@ -789,7 +789,7 @@ static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu) cpu :KEXEC_CRASH_HP_INVALID_CPU ;/* Now invoke arch-specific update handler */ -arch_crash_handle_hotplug_event (image, hp_action); +arch_crash_handle_hotplug_event (image, hp_action, arg);/* No longer handling a hotplug event */ image->hotplug_event = false;
@@ -804,17 +804,17 @@ static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu) } } -static int crash_memhp_notifier (struct notifier_block *nb,unsigned long val,void *v) +static int crash_memhp_notifier (struct notifier_block *nb,unsigned long val,void *arg) {switch (val) {case MEM_ONLINE :handle_hotplug_event (KEXEC_CRASH_HP_ADD_MEMORY , -KEXEC_CRASH_HP_INVALID_CPU ); +KEXEC_CRASH_HP_INVALID_CPU , arg);break ;case MEM_OFFLINE :handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_MEMORY , -KEXEC_CRASH_HP_INVALID_CPU ); +KEXEC_CRASH_HP_INVALID_CPU , arg);break ; }return NOTIFY_OK ;
@@ -827,13 +827,13 @@ static struct notifier_block crash_memhp_nb = {static int crash_cpuhp_online (unsigned int cpu) { -handle_hotplug_event (KEXEC_CRASH_HP_ADD_CPU , cpu); +handle_hotplug_event (KEXEC_CRASH_HP_ADD_CPU , cpu,NULL );return 0 ; }static int crash_cpuhp_offline (unsigned int cpu) { -handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_CPU , cpu); +handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_CPU , cpu,NULL );return 0 ; }
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v8 8/8 powerpc/kexec: add crash memory hotplug support
In reply to Sourabh JainExtend PowerPC arch crash hotplug handler to support memory hotplug events. Since elfcorehdr is used to exchange the memory info between the kernels hence it needs to be recreated to reflect the changes due to memory hotplug events.
The way memory hotplug events are handled on PowerPC and the notifier call chain used in generic code to trigger the arch crash handler, the process to recreate the elfcorehdr is different for memory add and remove events.
In the hot remove case, the memory region is first marked offline then the notifier call chain is triggered (same is used to initiate arch crash hotplug handler) and at last the memblock structure is updated. Whereas in the hot add case, memblock structure is updated before the notifier call chain is triggered.
On PowerPC, memblock structure is used to identify the memory ranges for elfcorehdr. In case of memory hot remove the memblock structure is updated after the arch crash hotplug handler is triggered, hence an additional step is taken to ensure that memory ranges used to build elfcorehdr do not include hot removed memory. Whereas no such extra steps are needed for the hot add case because memblock structure is updated before the arch crash hotplug handler is triggered.
The hot removed memory is explicitly removed from the memory ranges list before creating elfcorehdr for the hot remove case.
To accommodate a growing number of memory regions, elfcorehdr is built with additional buffer space to hold max memory regions.
The changes done here will also work for the kexec_load system call given that the kexec tool builds the elfcoredhr with additional space to accommodate future memory regions as it is done for kexec_file_load system call.
--- arch/powerpc/include/asm/kexec_ranges.h | 1 + arch/powerpc/kexec/core_64.c | 59 ++++++++++++++++++++++-- arch/powerpc/kexec/file_load_64.c | 36 ++++++++++++++- arch/powerpc/kexec/ranges.c | 60 +++++++++++++++++++++++++ 4 files changed, 151 insertions(+), 5 deletions(-)
arch/powerpc/include/asm/kexec_ranges.h +1 -0
@@ -7,6 +7,7 @@void sort_memory_ranges (struct crash_mem *mrngs,bool merge);struct crash_mem *realloc_mem_ranges (struct crash_mem **mem_ranges);int add_mem_range (struct crash_mem **mem_ranges,u64 base,u64 size); +int remove_mem_range (struct crash_mem **mem_ranges,u64 base,u64 size);int add_tce_mem_ranges (struct crash_mem **mem_ranges);int add_initrd_mem_range (struct crash_mem **mem_ranges);#ifdef CONFIG_PPC_64S_HASH_MMU
arch/powerpc/kexec/core_64.c +56 -3
84 lines not on this page · raw message
arch/powerpc/kexec/file_load_64.c +34 -2
@@ -21,6 +21,8 @@#include <linux/memblock.h> #include <linux/slab.h> #include <linux/vmalloc.h> +#include <linux/elf.h> +#include <asm/setup.h> #include <asm/drmem.h> #include <asm/firmware.h>
@@ -704,6 +706,30 @@ static void update_backup_region_phdr(struct kimage *image, Elf64_Ehdr *ehdr) } } +/* get_max_phdr - Find the total number of Phdr needed to represent the +* max memory in the kdump elfcorehdr. +* +* @cmem: crash memory ranges in the system. +*/ +static int get_max_phdr (struct crash_mem *cmem) +{ +int max_lmb; + +/* In the worst case, a Phdr is needed for every other LMB to be represented +* as an individual crash range. +*/ + max_lmb =memory_hotplug_max () /2 *drmem_lmb_size (); + +/* Do not cross the Phdr max limit of the elf header. +* Avoid counting Phdr for crash ranges (cmem->nr_ranges) which +* are already part of elfcorehdr. +*/ +if (max_lmb >PN_XNUM ) +return PN_XNUM - cmem->nr_ranges; + +return max_lmb - cmem->nr_ranges; +} +/** * load_elfcorehdr_segment - Setup crash memory ranges and initialize elfcorehdr * segment needed to load kdump kernel.
@@ -735,7 +761,13 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) kbuf->buffer = headers; kbuf->mem =KEXEC_BUF_MEM_UNKNOWN ; - kbuf->bufsz = kbuf->memsz = headers_sz; + kbuf->bufsz = headers_sz; +/* Additional buffer space to accommodate future memory ranges */ +#if defined(CONFIG_MEMORY_HOTPLUG ) + kbuf->memsz = headers_sz +get_max_phdr (cmem) *sizeof (Elf64_Phdr); +#else + kbuf->memsz = headers_sz; +#endif kbuf->top_down = false; ret =kexec_add_buffer (kbuf);
@@ -745,7 +777,7 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) } image->elf_load_addr = kbuf->mem; - image->elf_headers_sz = headers_sz; + image->elf_headers_sz = kbuf->memsz; image->elf_headers = headers; out:kfree (cmem);
arch/powerpc/kexec/ranges.c +60 -0
66 lines not on this page · raw message
Signature · 7 lines
2.39.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec