crash: Kernel handling of CPU and memory hot un/plug
v17 cover crash: Kernel handling of CPU and memory hot un/plug
When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions).
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace.
This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates.
In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.
To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file /usr/lib/udev/rules.d/98-kexec.rules:
# The kernel handles updates to crash elfcorehdr for cpu and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
These lines will cause cpu and memory hot un/plug events to be skipped within this rule file, with this changset applied.
- Change to the kexec_file_load for loading the kdump kernel:
Eg. on RHEL: in /usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to select kexec_file_load syscall.
This kernel patchset also supports kexec_load() with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:
http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
To use the kexec-tools patch, apply, build and install kexec-tools, then change the kdumpctl's standard_kexec_args to replace the -s with --hotplug. The removal of -s reverts to the kexec_load syscall and the addition of --hotplug invokes the changes put forth in the kexec-tools patch.
Regards,
eric
---
v17: 18jan2023
- Rebased onto 6.2.0-rc4
- Moved a bit of code around so that kexec_load()-only builds work, per Sourabh.
- Corrected computation of number of memory region Phdrs needed when x86 memory hotplug is not enabled, per Baoquan.
v16: 5jan2023
https://lkml.org/lkml/2023/1/5/673
- Rebased onto 6.2.0-rc2
- Corrected error identified by Baoquan.
v15: 9dec2022
https://lkml.org/lkml/2022/12/9/520
- Rebased onto 6.1.0-rc8
- Replaced arch_un/map_crash_pages() with direct use of kun/map_local_pages(), per Boris.
- Some x86 changes, per Boris.
v14: 16nov2022
https://lkml.org/lkml/2022/11/16/1645
- Rebased onto 6.1.0-rc5
- Introduced CRASH_HOTPLUG Kconfig item to better fine tune compilation of feature components, per Boris.
- Removed hp_action parameter to arch_crash_handle_hotplug_event() as it is unused.
v13: 31oct2022
https://lkml.org/lkml/2022/10/31/854
- Rebased onto 6.1.0-rc3, which means converting to use the new kexec_trylock() away from mutex_lock(kexec_mutex).
- Moved arch_un/map_crash_pages() into kexec.h and default implementation using k/unmap_local_pages().
- Changed more #ifdef's into IS_ENABLED()
- Changed CRASH_MAX_MEMORY_RANGES to 8192 from 32768, and it moved into x86 crash.c as #define rather Kconfig item, per Boris.
- Check number of Phdrs against PN_XNUM, max possible.
v12: 9sep2022
https://lkml.org/lkml/2022/9/9/1358
- Rebased onto 6.0-rc4
- Addressed some minor formatting items, per Baoquan
v11: 26aug2022
https://lkml.org/lkml/2022/8/26/963
- Rebased onto 6.0-rc2
- Redid the rework of __weak to use asm/kexec.h, per Baoquan
- Reworked some comments and minor items, per Baoquan
v10: 21jul2022
https://lkml.org/lkml/2022/7/21/1007
- Rebased to 5.19.0-rc7
- Per Sourabh, corrected build issue with arch_un/map_crash_pages() for architectures not supporting this feature.
- Per David Hildebrand, removed the WARN_ONCE() altogether.
- Per David Hansen, converted to use of kmap_local_page().
- Per Baoquan He, replaced use of __weak with the kexec technique.
v9: 13jun2022
https://lkml.org/lkml/2022/6/13/3382
- Rebased to 5.18.0
- Per Sourabh, moved crash_prepare_elf64_headers() into common crash_core.c to avoid compile issues with kexec_load only path.
- Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
- Changed the __weak arch_crash_handle_hotplug_event() to utilize WARN_ONCE() instead of WARN(). Fix some formatting issues.
- Per Sourabh, introduced sysfs attribute crash_hotplug for memory and CPUs; for use by userspace (udev) to determine if the kernel performs crash hot un/plug support.
- Per Sourabh, moved the code detecting the elfcorehdr segment from arch/x86 into crash_core:handle_hotplug_event() so both kexec_load and kexec_file_load can benefit.
- Updated userspace kexec-tools kexec utility to reflect change to using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
- Updated the new proposed udev rules to reflect using the sysfs attributes crash_hotplug.
v8: 5may2022
https://lkml.org/lkml/2022/5/5/1133
- Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define is not needed. Also use of IS_ENABLED() rather than #ifdef's. Renamed crash_hotplug_handler() to handle_hotplug_event(). And other corrections.
- Per Baoquan, minimized the parameters to the arch_crash_ handle_hotplug_event() to hp_action and cpu.
- Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
- Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change by David Hildebrand. Folded this patch into the x86 kexec_file_load support patch.
v7: 13apr2022
https://lkml.org/lkml/2022/4/13/850
- Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
v6: 1apr2022
https://lkml.org/lkml/2022/4/1/1203
- Reword commit messages and some comment cleanup per Baoquan.
- Changed elf_index to elfcorehdr_index for clarity.
- Minor code changes per Baoquan.
v5: 3mar2022
https://lkml.org/lkml/2022/3/3/674
- Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per David Hildenbrand.
- Refactored slightly a few patches per Baoquan recommendation.
v4: 9feb2022
https://lkml.org/lkml/2022/2/9/1406
- Refactored patches per Baoquan suggestsions.
- A few corrections, per Baoquan.
v3: 10jan2022
https://lkml.org/lkml/2022/1/10/1212
- Rebasing per Baoquan He request.
- Changed memory notifier per David Hildenbrand.
- Providing example kexec userspace change in cover letter.
RFC v2: 7dec2021
https://lkml.org/lkml/2021/12/7/1088
- Acting upon Baoquan He suggestion of removing elfcorehdr from the purgatory list of segments, removed purgatory code from patchset, and it is signficiantly simpler now.
RFC v1: 18nov2021
https://lkml.org/lkml/2021/11/18/845
- working patchset demonstrating kernel handling of hotplug updates to x86 elfcorehdr for kexec_file_load
RFC: 14dec2020
https://lkml.org/lkml/2020/12/14/532
- proposed concept of allowing kernel to handle hotplug update of elfcorehdr
---
Eric DeVolder (6):
crash: move a few code bits to setup support of crash hotplug
crash: prototype change for crash_prepare_elf64_headers()
crash: add generic infrastructure for crash hotplug support
kexec: exclude elfcorehdr from the segment digest
kexec: exclude hot remove cpu from elfcorehdr notes x86/crash: add x86 crash hotplug support
arch/arm64/kernel/machine_kexec_file.c | 6 +- arch/powerpc/kexec/file_load_64.c | 2 +- arch/riscv/kernel/elf_kexec.c | 7 +- arch/x86/Kconfig | 12 + arch/x86/include/asm/kexec.h | 15 ++ arch/x86/kernel/crash.c | 125 ++++++++- include/linux/crash_core.h | 8 + include/linux/kexec.h | 45 ++-- kernel/crash_core.c | 337 +++++++++++++++++++++++++ kernel/kexec_file.c | 187 +------------- 10 files changed, 533 insertions(+), 211 deletions(-)
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v17 1/6 crash: move a few code bits to setup support of crash hotplug
The crash hotplug support leans on the work for the kexec_file_load() syscall. To support crash hotplug for the kexec_load() syscall, a few bits of code need to be move outside of CONFIG_KEXEC_FILE.
The crash_prepare_elf64_headers() and crash_exclude_mem_range() functions, as well as the struct crash_mem are needed for support of crash hotplug on the kexec_load() path, as well as kexec_file_load(). As such, need to move these bits out of kexec_file.c and into a common location crash_core.c.
No functionality change.
--- include/linux/kexec.h | 30 +++---- kernel/crash_core.c | 182 ++++++++++++++++++++++++++++++++++++++++++ kernel/kexec_file.c | 181 ----------------------------------------- 3 files changed, 197 insertions(+), 196 deletions(-)
include/linux/kexec.h +15 -15
@@ -105,6 +105,21 @@ struct compat_kexec_segment { };#endif +/* Alignment required for elf header segment */ +#define ELF_CORE_HEADER_ALIGN 4096 + +struct crash_mem { +unsigned int max_nr_ranges; +unsigned int nr_ranges; +struct range ranges[]; +}; + +extern int crash_exclude_mem_range (struct crash_mem *mem, +unsigned long long mstart, +unsigned long long mend); +extern int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, +void **addr,unsigned long *sz); +#ifdef CONFIG_KEXEC_FILE struct purgatory_info {/*
@@ -238,21 +253,6 @@ static inline int arch_kexec_locate_mem_hole(struct kexec_buf *kbuf) }#endif -/* Alignment required for elf header segment */ -#define ELF_CORE_HEADER_ALIGN 4096 - -struct crash_mem { -unsigned int max_nr_ranges; -unsigned int nr_ranges; -struct range ranges[]; -}; - -extern int crash_exclude_mem_range (struct crash_mem *mem, -unsigned long long mstart, -unsigned long long mend); -extern int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, -void **addr,unsigned long *sz); -#ifndef arch_kexec_apply_relocations_add/* * arch_kexec_apply_relocations_add - apply relocations of type RELA
kernel/crash_core.c +182 -0
194 lines not on this page · raw message
kernel/kexec_file.c +0 -181
184 lines not on this page · raw message
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v17 2/6 crash: prototype change for crash_prepare_elf64_headers()
In reply to Eric DeVolderFrom within crash_prepare_elf64_headers() there is a need to reference the struct kimage hotplug members. As such, this change passes the struct kimage as a parameter to the crash_prepare_elf64_headers(). The hotplug members are added in "crash: add generic infrastructure for crash hotplug support".
This is preparation for later patch, no functionality change.
--- arch/arm64/kernel/machine_kexec_file.c | 6 +++--- arch/powerpc/kexec/file_load_64.c | 2 +- arch/riscv/kernel/elf_kexec.c | 7 ++++--- arch/x86/kernel/crash.c | 2 +- include/linux/kexec.h | 7 +++++-- kernel/crash_core.c | 4 ++-- 6 files changed, 16 insertions(+), 12 deletions(-)
arch/arm64/kernel/machine_kexec_file.c +3 -3
@@ -39,7 +39,7 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image)return kexec_image_post_load_cleanup_default (image); } -static int prepare_elf_headers (void **addr,unsigned long *sz) +static int prepare_elf_headers (struct kimage *image,void **addr,unsigned long *sz) {struct crash_mem *cmem;unsigned int nr_ranges;
@@ -64,7 +64,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz) }/* Exclude crashkernel region */ - ret =crash_exclude_mem_range (cmem, crashk_res.start, crashk_res.end); + ret =crash_exclude_mem_range (image, cmem, crashk_res.start, crashk_res.end);if (ret) goto out;
@@ -74,7 +74,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz) goto out; } - ret =crash_prepare_elf64_headers (cmem, true, addr, sz); + ret =crash_prepare_elf64_headers (image, cmem, true, addr, sz); out:kfree (cmem);
arch/powerpc/kexec/file_load_64.c +1 -1
@@ -799,7 +799,7 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) goto out;/* Setup elfcorehdr segment */ - ret =crash_prepare_elf64_headers (cmem, false, &headers, &headers_sz); + ret =crash_prepare_elf64_headers (image, cmem, false, &headers, &headers_sz);if (ret) {pr_err ("Failed to prepare elf headers for the core\n" ); goto out;
arch/riscv/kernel/elf_kexec.c +4 -3
@@ -130,7 +130,8 @@ static int prepare_elf64_ram_headers_callback(struct resource *res, void *arg)return 0 ; } -static int prepare_elf_headers (void **addr,unsigned long *sz) +static int prepare_elf_headers (struct kimage *image, +void **addr,unsigned long *sz) {struct crash_mem *cmem;unsigned int nr_ranges;
@@ -152,7 +153,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz)/* Exclude crashkernel region */ ret =crash_exclude_mem_range (cmem, crashk_res.start, crashk_res.end);if (!ret) - ret =crash_prepare_elf64_headers (cmem, true, addr, sz); + ret =crash_prepare_elf64_headers (image, cmem, true, addr, sz); out:kfree (cmem);
@@ -224,7 +225,7 @@ static void *elf_kexec_load(struct kimage *image, char *kernel_buf,/* Add elfcorehdr */ if (image->type ==KEXEC_TYPE_CRASH ) { - ret =prepare_elf_headers (&headers, &headers_sz); + ret =prepare_elf_headers (image, &headers, &headers_sz);if (ret) {pr_err ("Preparing elf core header failed\n" ); goto out;
arch/x86/kernel/crash.c +1 -1
@@ -265,7 +265,7 @@ static int prepare_elf_headers(struct kimage *image, void **addr, goto out;/* By default prepare 64bit headers */ - ret =crash_prepare_elf64_headers (cmem,IS_ENABLED (CONFIG_X86_64 ), addr, sz); + ret =crash_prepare_elf64_headers (image, cmem,IS_ENABLED (CONFIG_X86_64 ), addr, sz); out:vfree (cmem);
include/linux/kexec.h +5 -2
@@ -117,8 +117,11 @@ struct crash_mem {extern int crash_exclude_mem_range (struct crash_mem *mem,unsigned long long mstart,unsigned long long mend); -extern int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, -void **addr,unsigned long *sz); +extern int crash_prepare_elf64_headers (struct kimage *image, +struct crash_mem *mem, +int need_kernel_map, +void **addr, +unsigned long *sz);#ifdef CONFIG_KEXEC_FILE struct purgatory_info {
kernel/crash_core.c +2 -2
@@ -315,8 +315,8 @@ static int __init parse_crashkernel_dummy(char *arg) }early_param ("crashkernel" , parse_crashkernel_dummy); -int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, -void **addr,unsigned long *sz) +int crash_prepare_elf64_headers (struct kimage *image,struct crash_mem *mem, +int need_kernel_map,void **addr,unsigned long *sz) {Elf64_Ehdr *ehdr;Elf64_Phdr *phdr;
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v17 3/6 crash: add generic infrastructure for crash hotplug support
In reply to Eric DeVolderCPU and memory change notifications are received in order to regenerate the elfcorehdr.
To support cpu hotplug, a callback is registered to capture the CPUHP_AP_ONLINE_DYN online and offline events via cpuhp_setup_state_nocalls().
To support memory hotplug, a notifier is registered to capture the MEM_ONLINE and MEM_OFFLINE events via register_memory_notifier().
The cpu callback and memory notifiers call handle_hotplug_event() which performs needed tasks and then dispatches the event to the architecture specific arch_crash_handle_hotplug_event(). During the process, the kexec_lock is held.
--- include/linux/crash_core.h | 8 +++ include/linux/kexec.h | 12 ++++ kernel/crash_core.c | 138 +++++++++++++++++++++++++++++++++++++ 3 files changed, 158 insertions(+)
include/linux/crash_core.h +8 -0
@@ -84,4 +84,12 @@ int parse_crashkernel_high(char *cmdline, unsigned long long system_ram,int parse_crashkernel_low (char *cmdline,unsigned long long system_ram,unsigned long long *crash_size,unsigned long long *crash_base); +#define KEXEC_CRASH_HP_REMOVE_CPU 0 +#define KEXEC_CRASH_HP_ADD_CPU 1 +#define KEXEC_CRASH_HP_REMOVE_MEMORY 2 +#define KEXEC_CRASH_HP_ADD_MEMORY 3 +#define KEXEC_CRASH_HP_INVALID_CPU -1U + +struct kimage ; +#endif /* LINUX_CRASH_CORE_H */
include/linux/kexec.h +12 -0
@@ -33,6 +33,7 @@ extern note_buf_t __percpu *crash_notes;#include <linux/compat.h> #include <linux/ioport.h> #include <linux/module.h> +#include <linux/highmem.h> #include <asm/kexec.h> /* Verify architecture specific macros are defined */
@@ -371,6 +372,13 @@ struct kimage {struct purgatory_info purgatory_info;#endif +#ifdef CONFIG_CRASH_HOTPLUG +bool hotplug_event; +unsigned int offlinecpu; +bool elfcorehdr_index_valid; +int elfcorehdr_index; +#endif +#ifdef CONFIG_IMA_KEXEC /* Virtual address of IMA measurement buffer for kexec syscall */ void *ima_buffer;
@@ -500,6 +508,10 @@ static inline int arch_kexec_post_alloc_pages(void *vaddr, unsigned int pages, gstatic inline void arch_kexec_pre_free_pages (void *vaddr,unsigned int pages) { }#endif +#ifndef arch_crash_handle_hotplug_event +static inline void arch_crash_handle_hotplug_event (struct kimage *image) { } +#endif +#else /* !CONFIG_KEXEC_CORE */ struct pt_regs ;struct task_struct ;
kernel/crash_core.c +138 -0
153 lines not on this page · raw message
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v17 3/6] crash: add generic infrastructure for crash hotplug support
Eric!
Quoted 6 lines · Eric DeVolder
CPU and memory change notifications are received in order to regenerate the elfcorehdr.
To support cpu hotplug, a callback is registered to capture the
CPUHP_AP_ONLINE_DYNonline and offline events viacpuhp_setup_state_nocalls().
This sentence does not make sense. The callback is not registered to capture CPUHP_AP_ONLINE_DYN events.
What this does is: It installs a dynamic CPU hotplug state with callbacks for online and offline. These callbacks store information about a CPU coming up and going down. Right?
But why are they required and what's the value?
This changelog tells WHAT it does and not WHY. I can see the WHAT from the patch itself.
Don't tell me the WHY is in the cover letter. The cover letter is not part of the commits and changelogs have to be self contained.
Now let me cite from your cover letter:
Quoted 5 lines
When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions).
The CPU hotplug state you are using for this is patently inaccurate too. With your approach the CPU is tracked as online very late in the hotplug process and tracked as offline very early on unplug.
So if the kernel crashes before/after the plug/unplug tracking event then your recorded state is bogus and given the amount of callbacks between the real online/offline and the recording point there is a pretty large window.
You can argue that this is better than the current state and considered good enough for whatever reason, but such information wants to be in the changelog, no?
Thanks,
tglx
Hint: The requirements for changelogs are well documented in Documentation/process/
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v17 3/6] crash: add generic infrastructure for crash hotplug support
Quoted 10 lines · Thomas Gleixner
Eric!
Quoted 6 lines · Eric DeVolder
CPU and memory change notifications are received in order to regenerate the elfcorehdr.
To support cpu hotplug, a callback is registered to capture the
CPUHP_AP_ONLINE_DYNonline and offline events viacpuhp_setup_state_nocalls().
Quoted 5 lines
This sentence does not make sense. The callback is not registered to capture
CPUHP_AP_ONLINE_DYNevents >
What this does is: It installs a dynamic CPU hotplug state with callbacks for online and offline. These callbacks store information about a CPU coming up and going down. Right?
I agree, the wording is wrong; this code taps into that state, as you suggest, in order to handle the online and offline events.
Quoted 16 lines
But why are they required and what's the value?
This changelog tells WHAT it does and not WHY. I can see the WHAT from the patch itself.
Don't tell me the WHY is in the cover letter. The cover letter is not part of the commits and changelogs have to be self contained.
Now let me cite from your cover letter:
Quoted 5 lines
When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions).
I'll work to improve the wording and why for the next iteration.
Quoted 5 lines
The CPU hotplug state you are using for this is patently inaccurate too. With your approach the CPU is tracked as online very late in the hotplug process and tracked as offline very early on unplug.
Quoted 8 lines
So if the kernel crashes before/after the plug/unplug tracking event then your recorded state is bogus and given the amount of callbacks between the real online/offline and the recording point there is a pretty large window.
You can argue that this is better than the current state and considered good enough for whatever reason, but such information wants to be in the changelog, no?
I agree! I admit that CPUHP_AP_ONLINE_DYN may (is) not the best choice. I did spend time looking at the cpu hotplug infrastructure, but did not learn a better/correct way. Fwiw: https://lore.kernel.org/lkml/[email protected]/:
"The second problem is the use of CPUHP_AP_ONLINE_DYN. The cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the elfcorehdr, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers(). Perhaps if value(s) new/different than CPUHP_AP_ONLINE_DYN to cpuhp_setup_state() was utilized, then the offline cpu would no longer be in foreach_present_cpu(), and this change could be eliminated. I do not understand cpuhp_setup_state() well enough to choose, or create, appropriate value(s)."
The problem described (and worked around in this patch series) is the behavior/window you point out. I'd prefer to narrow the window, if possible. The states/values I tried did not work; any suggestions for a more appropriate state/value would be most welcomed!
Quoted 8 lines
Thanks,
tglx
Hint: The requirements for changelogs are well documented in
Documentation/process/
Thomas, thank you for looking at this!
eric
PATCH v17 4/6 kexec: exclude elfcorehdr from the segment digest
In reply to Eric DeVolderWhen a crash kernel is loaded via the kexec_file_load() syscall, the kernel places the various segments (ie crash kernel, crash initrd, boot_params, elfcorehdr, purgatory, etc) in memory. For those architectures that utilize purgatory, a hash digest of the segments is calculated for integrity checking. This digest is embedded into the purgatory image prior to placing purgatory in memory.
Since hotplug events cause changes to the elfcorehdr, purgatory integrity checking fails (at crash time, and no kdump created). As a result, this change explicitly excludes the elfcorehdr segment from the list of segments used to create the digest. By doing so, this permits changes to the elfcorehdr in response to hotplug events, without having to also reload purgatory due to the change to the digest.
--- kernel/kexec_file.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/kexec_file.c +6 -0
@@ -723,6 +723,12 @@ static int kexec_calculate_store_digests(struct kimage *image)for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment; +#ifdef CONFIG_CRASH_HOTPLUG +/* Exclude elfcorehdr segment to allow future changes via hotplug */ +if (image->elfcorehdr_index_valid && (j == image->elfcorehdr_index)) +continue ; +#endif + ksegment = &image->segment[i];/* * Skip purgatory as it will be modified once we put digest
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v17 5/6 kexec: exclude hot remove cpu from elfcorehdr notes
In reply to Eric DeVolderDue to use of CPUHP_AP_ONLINE_DYN, upon CPU unplug, the CPU is still in the for_each_present_cpu() list when within the handle_hotplug_event(). Thus the CPU must be explicitly excluded when building the new list of CPUs.
This change identifies in handle_hotplug_event() the CPU to be excluded, and the check for excluding the CPU in crash_prepare_elf64_headers().
--- kernel/crash_core.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/crash_core.c +17 -0
@@ -366,6 +366,13 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem,/* Prepare one phdr of type PT_NOTE for each present CPU */ for_each_present_cpu (cpu) { +#ifdef CONFIG_CRASH_HOTPLUG +if (IS_ENABLED (CONFIG_HOTPLUG_CPU )) { +/* Skip the soon-to-be offlined cpu */ +if (image->hotplug_event && (cpu == image->offlinecpu)) +continue ; + } +#endif phdr->p_type =PT_NOTE ; notes_addr =per_cpu_ptr_to_phys (per_cpu_ptr (crash_notes, cpu)); phdr->p_offset = phdr->p_paddr = notes_addr;
@@ -768,6 +775,16 @@ static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu)/* Flag to differentiate between normal load and hotplug */ image->hotplug_event = true; +/* +* Due to use of CPUHP_AP_ONLINE_DYN, upon unplug and during +* this callback, the CPU is still in the for_each_present_cpu() +* list. Must explicitly look to exclude this CPU when building +* new list. +*/ + image->offlinecpu = + (hp_action ==KEXEC_CRASH_HP_REMOVE_CPU ) ? + cpu :KEXEC_CRASH_HP_INVALID_CPU ; +/* Now invoke arch-specific update handler */ arch_crash_handle_hotplug_event (image);
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v17 6/6 x86/crash: add x86 crash hotplug support
In reply to Eric DeVolderWhen CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr. The segment containing the elfcorehdr is identified at run time in crash_core:handle_hotplug_event(), which works for both the kexec_load() and kexec_file_load() syscalls.
In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_MAX_MEMORY_RANGES description.
--- arch/x86/Kconfig | 12 ++++ arch/x86/include/asm/kexec.h | 15 +++++ arch/x86/kernel/crash.c | 123 +++++++++++++++++++++++++++++++++-- 3 files changed, 143 insertions(+), 7 deletions(-)
arch/x86/Kconfig +12 -0
@@ -2119,6 +2119,18 @@ config CRASH_DUMP (CONFIG_RELOCATABLE=y). For more details see Documentation/admin-guide/kdump/kdump.rst +config CRASH_HOTPLUG + bool "Update the crash elfcorehdr on system configuration changes" + default n + depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) + help + Enable direct update to the crash elfcorehdr (which contains + the list of CPUs and memory regions to be dumped upon a crash) + in response to hot plug/unplug of CPUs or memory. This is a much + more advanced approach than userspace attempting that. + + If unsure, say Y. + config KEXEC_JUMP bool "kexec jump" depends on KEXEC && HIBERNATION
arch/x86/include/asm/kexec.h +15 -0
@@ -212,6 +212,21 @@ typedef void crash_vmclear_fn(void);extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;extern void kdump_nmi_shootdown_cpus (void ); +#ifdef CONFIG_CRASH_HOTPLUG +void arch_crash_handle_hotplug_event (struct kimage *image); +#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event + +#ifdef CONFIG_HOTPLUG_CPU +static inline int crash_hotplug_cpu_support (void ) {return 1 ; } +#define crash_hotplug_cpu_support crash_hotplug_cpu_support +#endif + +#ifdef CONFIG_MEMORY_HOTPLUG +static inline int crash_hotplug_memory_support (void ) {return 1 ; } +#define crash_hotplug_memory_support crash_hotplug_memory_support +#endif +#endif +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_KEXEC_H */
arch/x86/kernel/crash.c +116 -7
170 lines not on this page · raw message
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec