crash: Kernel handling of CPU and memory hot un/plug
Subsystems (9)
v21 cover crash: Kernel handling of CPU and memory hot un/plug
Once the kdump service is loaded, if changes to CPUs or memory occur, either by hot un/plug or off/onlining, the crash elfcorehdr must also be updated.
The elfcorehdr describes to kdump the CPUs and memory in the system, and any inaccuracies can result in a vmcore with missing CPU context or memory regions.
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by the userspace kexec utility. In the original post I outlined the significant performance problems related to offloading this activity to userspace.
This patchset introduces a generic crash handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, from either hot un/plug or off/onlining, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate elfcorehdr update.
Note the description in patch 'crash: change crash_prepare_elf64_headers() to for_each_possible_cpu()' and 'x86/crash: optimize CPU changes' that enables further optimizations related to CPU plug/unplug/online/offline performance of elfcorehdr updates.
In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.
To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file /usr/lib/udev/rules.d/98-kexec.rules:
# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
With this changeset applied, the two rules evaluate to false for CPU and memory change events and thus skip the userspace unload-then-reload of kdump.
- Change to the kexec_file_load for loading the kdump kernel:
Eg. on RHEL: in /usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to select kexec_file_load() syscall.
This kernel patchset also supports kexec_load() with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:
http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
To use the kexec-tools patch, apply, build and install kexec-tools, then change the kdumpctl's standard_kexec_args to replace the -s with --hotplug. The removal of -s reverts to the kexec_load syscall and the addition of --hotplug invokes the changes put forth in the kexec-tools patch.
Regards,
eric
---
v21: 4apr2023
- Rebased onto 6.3.0-rc5
- Additional simplification of indentation in crash_handle_hotplug_event(), per Baoquan.
v20: 17mar2023
https://lkml.org/lkml/2023/3/17/1169
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.3.0-rc2
- Defaulting CRASH_HOTPLUG for x86 to Y, per Sourabh.
- Explicitly initializing image->hp_action, per Baoquan.
- Simplified kexec_trylock() in crash_handle_hotplug_event(), per Baoquan.
- Applied Sourabh's Reviewed-by to the series.
v19: 6mar2023
https://lkml.org/lkml/2023/3/6/1358
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.2.0
- Did away with offlinecpu, per Thomas Gleixner.
- Changed to CPUHP_BP_PREPARE_DYN instead of CPUHP_AP_ONLINE_DYN.
- Did away with elfcorehdr_index_valid, per Sourabh.
- Convert to for_each_possible_cpu() in crash_prepare_elf64_headers() per Sourabh.
- Small optimization for x86 cpu changes.
v18: 31jan2023
https://lkml.org/lkml/2023/1/31/1356
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.2.0-rc6
- Renamed struct kimage member hotplug_event to hp_action, and re-enumerated the KEXEC_CRASH_HP_x items, adding _NONE at 0.
- Moved to cpuhp state CPUHP_BP_PREPARE_DYN instead of CPUHP_AP_ONLINE_DYN in order to minimize window of time CPU is not reflected in elfcorehdr.
- Reworked some of the comments and commit messages to offer more of the why, than what, per Thomas Gleixner.
v17: 18jan2023
https://lkml.org/lkml/2023/1/18/1420
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.2.0-rc4
- Moved a bit of code around so that kexec_load()-only builds work, per Sourabh.
- Corrected computation of number of memory region Phdrs needed when x86 memory hotplug is not enabled, per Baoquan.
v16: 5jan2023
https://lkml.org/lkml/2023/1/5/673
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.2.0-rc2
- Corrected error identified by Baoquan.
v15: 9dec2022
https://lkml.org/lkml/2022/12/9/520
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.1.0-rc8
- Replaced arch_un/map_crash_pages() with direct use of kun/map_local_pages(), per Boris.
- Some x86 changes, per Boris.
v14: 16nov2022
https://lkml.org/lkml/2022/11/16/1645
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.1.0-rc5
- Introduced CRASH_HOTPLUG Kconfig item to better fine tune compilation of feature components, per Boris.
- Removed hp_action parameter to arch_crash_handle_hotplug_event() as it is unused.
v13: 31oct2022
https://lkml.org/lkml/2022/10/31/854
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.1.0-rc3, which means converting to use the new kexec_trylock() away from mutex_lock(kexec_mutex).
- Moved arch_un/map_crash_pages() into kexec.h and default implementation using k/unmap_local_pages().
- Changed more #ifdef's into IS_ENABLED()
- Changed CRASH_MAX_MEMORY_RANGES to 8192 from 32768, and it moved into x86 crash.c as #define rather Kconfig item, per Boris.
- Check number of Phdrs against PN_XNUM, max possible.
v12: 9sep2022
https://lkml.org/lkml/2022/9/9/1358
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.0-rc4
- Addressed some minor formatting items, per Baoquan
v11: 26aug2022
https://lkml.org/lkml/2022/8/26/963
https://lore.kernel.org/lkml/[email protected]/
- Rebased onto 6.0-rc2
- Redid the rework of __weak to use asm/kexec.h, per Baoquan
- Reworked some comments and minor items, per Baoquan
v10: 21jul2022
https://lkml.org/lkml/2022/7/21/1007
https://lore.kernel.org/lkml/[email protected]/
- Rebased to 5.19.0-rc7
- Per Sourabh, corrected build issue with arch_un/map_crash_pages() for architectures not supporting this feature.
- Per David Hildebrand, removed the WARN_ONCE() altogether.
- Per David Hansen, converted to use of kmap_local_page().
- Per Baoquan He, replaced use of __weak with the kexec technique.
v9: 13jun2022
https://lkml.org/lkml/2022/6/13/3382
https://lore.kernel.org/lkml/[email protected]/
- Rebased to 5.18.0
- Per Sourabh, moved crash_prepare_elf64_headers() into common crash_core.c to avoid compile issues with kexec_load only path.
- Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
- Changed the __weak arch_crash_handle_hotplug_event() to utilize WARN_ONCE() instead of WARN(). Fix some formatting issues.
- Per Sourabh, introduced sysfs attribute crash_hotplug for memory and CPUs; for use by userspace (udev) to determine if the kernel performs crash hot un/plug support.
- Per Sourabh, moved the code detecting the elfcorehdr segment from arch/x86 into crash_core:handle_hotplug_event() so both kexec_load and kexec_file_load can benefit.
- Updated userspace kexec-tools kexec utility to reflect change to using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
- Updated the new proposed udev rules to reflect using the sysfs attributes crash_hotplug.
v8: 5may2022
https://lkml.org/lkml/2022/5/5/1133
https://lore.kernel.org/lkml/[email protected]/
- Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define is not needed. Also use of IS_ENABLED() rather than #ifdef's. Renamed crash_hotplug_handler() to handle_hotplug_event(). And other corrections.
- Per Baoquan, minimized the parameters to the arch_crash_ handle_hotplug_event() to hp_action and cpu.
- Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
- Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change by David Hildebrand. Folded this patch into the x86 kexec_file_load support patch.
v7: 13apr2022
https://lkml.org/lkml/2022/4/13/850
https://lore.kernel.org/lkml/[email protected]/
- Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
v6: 1apr2022
https://lkml.org/lkml/2022/4/1/1203
https://lore.kernel.org/lkml/[email protected]/
- Reword commit messages and some comment cleanup per Baoquan.
- Changed elf_index to elfcorehdr_index for clarity.
- Minor code changes per Baoquan.
v5: 3mar2022
https://lkml.org/lkml/2022/3/3/674
https://lore.kernel.org/lkml/[email protected]/
- Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per David Hildenbrand.
- Refactored slightly a few patches per Baoquan recommendation.
v4: 9feb2022
https://lkml.org/lkml/2022/2/9/1406
https://lore.kernel.org/lkml/[email protected]/
- Refactored patches per Baoquan suggestsions.
- A few corrections, per Baoquan.
v3: 10jan2022
https://lkml.org/lkml/2022/1/10/1212
https://lore.kernel.org/lkml/[email protected]/
- Rebasing per Baoquan He request.
- Changed memory notifier per David Hildenbrand.
- Providing example kexec userspace change in cover letter.
RFC v2: 7dec2021
https://lkml.org/lkml/2021/12/7/1088
https://lore.kernel.org/lkml/[email protected]/
- Acting upon Baoquan He suggestion of removing elfcorehdr from the purgatory list of segments, removed purgatory code from patchset, and it is signficiantly simpler now.
RFC v1: 18nov2021
https://lkml.org/lkml/2021/11/18/845
https://lore.kernel.org/lkml/[email protected]/
- working patchset demonstrating kernel handling of hotplug updates to x86 elfcorehdr for kexec_file_load
RFC: 14dec2020
https://lkml.org/lkml/2020/12/14/532
https://lore.kernel.org/lkml/[email protected]/
- proposed concept of allowing kernel to handle hotplug update of elfcorehdr
---
Eric DeVolder (7):
crash: move a few code bits to setup support of crash hotplug
crash: add generic infrastructure for crash hotplug support
kexec: exclude elfcorehdr from the segment digest
crash: memory and CPU hotplug sysfs attributes
x86/crash: add x86 crash hotplug support
crash: change crash_prepare_elf64_headers() to for_each_possible_cpu() x86/crash: optimize CPU changes
.../admin-guide/mm/memory-hotplug.rst | 8 + Documentation/core-api/cpu_hotplug.rst | 18 + arch/x86/Kconfig | 13 + arch/x86/include/asm/kexec.h | 15 + arch/x86/kernel/crash.c | 129 ++++++- drivers/base/cpu.c | 14 + drivers/base/memory.c | 13 + include/linux/crash_core.h | 9 + include/linux/kexec.h | 49 ++- kernel/crash_core.c | 324 ++++++++++++++++++ kernel/kexec_core.c | 6 + kernel/kexec_file.c | 187 +--------- 12 files changed, 582 insertions(+), 203 deletions(-)
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 1/7 crash: move a few code bits to setup support of crash hotplug
The crash hotplug support leans on the work for the kexec_file_load() syscall. To also support the kexec_load() syscall, a few bits of code need to be move outside of CONFIG_KEXEC_FILE. As such, these bits are moved out of kexec_file.c and into a common location crash_core.c.
No functionality change intended.
--- include/linux/kexec.h | 30 +++---- kernel/crash_core.c | 182 ++++++++++++++++++++++++++++++++++++++++++ kernel/kexec_file.c | 181 ----------------------------------------- 3 files changed, 197 insertions(+), 196 deletions(-)
include/linux/kexec.h +15 -15
@@ -105,6 +105,21 @@ struct compat_kexec_segment { };#endif +/* Alignment required for elf header segment */ +#define ELF_CORE_HEADER_ALIGN 4096 + +struct crash_mem { +unsigned int max_nr_ranges; +unsigned int nr_ranges; +struct range ranges[]; +}; + +extern int crash_exclude_mem_range (struct crash_mem *mem, +unsigned long long mstart, +unsigned long long mend); +extern int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, +void **addr,unsigned long *sz); +#ifdef CONFIG_KEXEC_FILE struct purgatory_info {/*
@@ -238,21 +253,6 @@ static inline int arch_kexec_locate_mem_hole(struct kexec_buf *kbuf) }#endif -/* Alignment required for elf header segment */ -#define ELF_CORE_HEADER_ALIGN 4096 - -struct crash_mem { -unsigned int max_nr_ranges; -unsigned int nr_ranges; -struct range ranges[]; -}; - -extern int crash_exclude_mem_range (struct crash_mem *mem, -unsigned long long mstart, -unsigned long long mend); -extern int crash_prepare_elf64_headers (struct crash_mem *mem,int need_kernel_map, -void **addr,unsigned long *sz); -#ifndef arch_kexec_apply_relocations_add/* * arch_kexec_apply_relocations_add - apply relocations of type RELA
kernel/crash_core.c +182 -0
194 lines not on this page · raw message
kernel/kexec_file.c +0 -181
184 lines not on this page · raw message
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 2/7 crash: add generic infrastructure for crash hotplug support
In reply to Eric DeVolderTo support crash hotplug, a mechanism is needed to update the crash elfcorehdr upon CPU or memory changes (eg. hot un/plug or off/ onlining). The crash elfcorehdr describes the CPUs and memory to be written into the vmcore.
To track CPU changes, callbacks are registered with the cpuhp mechanism via cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN). The crash hotplug elfcorehdr update has no explicit ordering requirement (relative to other cpuhp states), so meets the criteria for utilizing CPUHP_BP_PREPARE_DYN. CPUHP_BP_PREPARE_DYN is a dynamic state and avoids the need to introduce a new state for crash hotplug. Also, this is the last state in the PREPARE group, just prior to the STARTING group, which is very close to the CPU starting up in a plug/online situation, or stopping in a unplug/ offline situation. This minimizes the window of time during an actual plug/online or unplug/offline situation in which the elfcorehdr would be inaccurate. Note that for a CPU being unplugged or offlined, the CPU will still be present in the list of CPUs generated by crash_prepare_elf64_headers(). However, there is no need to explicitly omit the CPU, see the discussion in the patch 'crash: change crash_prepare_elf64_headers() to for_each_possible_cpu()'.
To track memory changes, a notifier is registered to capture the memblock MEM_ONLINE and MEM_OFFLINE events via register_memory_notifier().
The CPU callbacks and memory notifiers invoke crash_handle_hotplug_event() which performs needed tasks and then dispatches the event to the architecture specific arch_crash_handle_hotplug_event() to update the elfcorehdr with the current state of CPUs and memory. During the process, the kexec_lock is held.
--- include/linux/crash_core.h | 9 +++ include/linux/kexec.h | 11 +++ kernel/crash_core.c | 142 +++++++++++++++++++++++++++++++++++++ kernel/kexec_core.c | 6 ++ 4 files changed, 168 insertions(+)
include/linux/crash_core.h +9 -0
@@ -84,4 +84,13 @@ int parse_crashkernel_high(char *cmdline, unsigned long long system_ram,int parse_crashkernel_low (char *cmdline,unsigned long long system_ram,unsigned long long *crash_size,unsigned long long *crash_base); +#define KEXEC_CRASH_HP_NONE 0 +#define KEXEC_CRASH_HP_ADD_CPU 1 +#define KEXEC_CRASH_HP_REMOVE_CPU 2 +#define KEXEC_CRASH_HP_ADD_MEMORY 3 +#define KEXEC_CRASH_HP_REMOVE_MEMORY 4 +#define KEXEC_CRASH_HP_INVALID_CPU -1U + +struct kimage ; +#endif /* LINUX_CRASH_CORE_H */
include/linux/kexec.h +11 -0
@@ -33,6 +33,7 @@ extern note_buf_t __percpu *crash_notes;#include <linux/compat.h> #include <linux/ioport.h> #include <linux/module.h> +#include <linux/highmem.h> #include <asm/kexec.h> /* Verify architecture specific macros are defined */
@@ -368,6 +369,12 @@ struct kimage {struct purgatory_info purgatory_info;#endif +#ifdef CONFIG_CRASH_HOTPLUG +int hp_action; +int elfcorehdr_index; +bool elfcorehdr_updated; +#endif +#ifdef CONFIG_IMA_KEXEC /* Virtual address of IMA measurement buffer for kexec syscall */ void *ima_buffer;
@@ -498,6 +505,10 @@ static inline int arch_kexec_post_alloc_pages(void *vaddr, unsigned int pages, gstatic inline void arch_kexec_pre_free_pages (void *vaddr,unsigned int pages) { }#endif +#ifndef arch_crash_handle_hotplug_event +static inline void arch_crash_handle_hotplug_event (struct kimage *image) { } +#endif +#else /* !CONFIG_KEXEC_CORE */ struct pt_regs ;struct task_struct ;
kernel/crash_core.c +142 -0
157 lines not on this page · raw message
kernel/kexec_core.c +6 -0
@@ -277,6 +277,12 @@ struct kimage *do_kimage_alloc_init(void)/* Initialize the list of unusable pages */ INIT_LIST_HEAD (&image->unusable_pages); +#ifdef CONFIG_CRASH_HOTPLUG + image->hp_action =KEXEC_CRASH_HP_NONE ; + image->elfcorehdr_index =-1 ; + image->elfcorehdr_updated = false; +#endif +return image; }
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
On 04/04/23 at 02:03pm, Eric DeVolder wrote:
......
Quoted 11 lines
kernel/crash_core.c +11
-0+static void crash_handle_hotplug_event (unsigned int hp_action,unsigned int cpu) +{ +struct kimage *image; + +/* Obtain lock while changing crash information */ +if (!kexec_trylock ()) { +pr_info ("kexec_trylock() failed, elfcorehdr may be inaccurate\n" ); +return ; + } + +/* Check kdump is not loaded */
kernel/crash_core.c static int __init crash_save_vmcoreinfo_init(void) · line 741
Quoted 10 lines
kernel/crash_core.c +10
-0+if (!kexec_crash_image) + goto out; + + image = kexec_crash_image; + +if (hp_action ==KEXEC_CRASH_HP_ADD_CPU || + hp_action ==KEXEC_CRASH_HP_REMOVE_CPU ) +pr_debug ("hp_action %u, cpu %u\n" , hp_action, cpu); +else +pr_debug ("hp_action %u\n" , hp_action);
Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Quoted 120 lines
kernel/crash_core.c +100
-0+ +/* +* When the struct kimage is allocated, the elfcorehdr_index +* is set to -1. Find the segment containing the elfcorehdr, +* if not already found. This works for both the kexec_load +* and kexec_file_load paths. +*/ +if (image->elfcorehdr_index <0 ) { +unsigned long mem; +unsigned char *ptr; +unsigned int n; + +for (n =0 ; n < image->nr_segments; n++) { + mem = image->segment[n].mem; + ptr =kmap_local_page (pfn_to_page (mem >>PAGE_SHIFT )); +if (ptr) { +/* The segment containing elfcorehdr */ +if (memcmp (ptr,ELFMAG ,SELFMAG ) ==0 ) + image->elfcorehdr_index = (int )n; +kunmap_local (ptr); + } + } + } + +if (image->elfcorehdr_index <0 ) { +pr_err ("unable to locate elfcorehdr segment" ); + goto out; + } + +/* Needed in order for the segments to be updated */ +arch_kexec_unprotect_crashkres (); + +/* Differentiate between normal load and hotplug update */ + image->hp_action = hp_action; + +/* Now invoke arch-specific update handler */ +arch_crash_handle_hotplug_event (image); + +/* No longer handling a hotplug event */ + image->hp_action =KEXEC_CRASH_HP_NONE ; + image->elfcorehdr_updated = true; + +/* Change back to read-only */ +arch_kexec_protect_crashkres (); + +out: +/* Release lock now that update complete */ +kexec_unlock (); +} + +static int crash_memhp_notifier (struct notifier_block *nb,unsigned long val,void *v) +{ +switch (val) { +case MEM_ONLINE : +crash_handle_hotplug_event (KEXEC_CRASH_HP_ADD_MEMORY , +KEXEC_CRASH_HP_INVALID_CPU ); +break ; + +case MEM_OFFLINE : +crash_handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_MEMORY , +KEXEC_CRASH_HP_INVALID_CPU ); +break ; + } +return NOTIFY_OK ; +} + +static struct notifier_block crash_memhp_nb = { + .notifier_call = crash_memhp_notifier, + .priority =0 +}; + +static int crash_cpuhp_online (unsigned int cpu) +{ +crash_handle_hotplug_event (KEXEC_CRASH_HP_ADD_CPU , cpu); +return 0 ; +} + +static int crash_cpuhp_offline (unsigned int cpu) +{ +crash_handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_CPU , cpu); +return 0 ; +} + +static int __init crash_hotplug_init (void ) +{ +int result =0 ; + +if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG )) +register_memory_notifier (&crash_memhp_nb); + +if (IS_ENABLED (CONFIG_HOTPLUG_CPU )) { + result =cpuhp_setup_state_nocalls (CPUHP_BP_PREPARE_DYN , +"crash/cpuhp" , crash_cpuhp_online, crash_cpuhp_offline); + } + +return result; +} + +subsys_initcall (crash_hotplug_init); +#endif kernel/kexec_core.c +6
-0@@ -277,6 +277,12 @@ struct kimage *do_kimage_alloc_init(void)/* Initialize the list of unusable pages */ INIT_LIST_HEAD (&image->unusable_pages); +#ifdef CONFIG_CRASH_HOTPLUG + image->hp_action =KEXEC_CRASH_HP_NONE ; + image->elfcorehdr_index =-1 ; + image->elfcorehdr_updated = false; +#endif +return image; }Signature · 1 line
2.31.1
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
Quoted 19 lines · Baoquan He
On 04/04/23 at 02:03pm, Eric DeVolder wrote:
......Quoted 17 lines
kernel/crash_core.c +17
-0+static void crash_handle_hotplug_event (unsigned int hp_action,unsigned int cpu) +{ +struct kimage *image; + +/* Obtain lock while changing crash information */ +if (!kexec_trylock ()) { +pr_info ("kexec_trylock() failed, elfcorehdr may be inaccurate\n" ); +return ; + } + +/* Check kdump is not loaded */ +if (!kexec_crash_image) + goto out; + + image = kexec_crash_image; + + if (hp_action == KEXEC_CRASH_HP_ADD_CPU ||
Quoted 9 lines
kernel/crash_core.c static int __init crash_save_vmcoreinfo_init(void) · line 741
Quoted 4 lines
kernel/crash_core.c +4
-0+ hp_action == KEXEC_CRASH_HP_REMOVE_CPU) +pr_debug ("hp_action %u, cpu %u\n" , hp_action, cpu); + else + pr_debug("hp_action %u\n", hp_action);Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Baoquan,
Ah, actually until recently it was used to track the 'offlinecpu' in this function, but tglx pointed out that was un-necessary. That resulted in dropping the code in this function dealing with offlinecpu, leaving this as its only use in this function.
The printing of cpu number is not necessary, but helpful; I use it for debugging.
The printing of memory range is also not necessary, but in order to do that, should we choose to do so, requires passing in the memory range to this function. This patch series did do this early on, and by v7 I dropped it at your urging (https://lore.kernel.org/lkml/[email protected]/). At the time, I provided it since I considered this generic infrastructure, but I could not defend it since x86 didn't need it. However, PPC now needs this, and is now carrying this as part of PPC support of CRASH_HOTPLUG (https://lore.kernel.org/linuxppc-dev/[email protected]/T/#u).
If you'd rather I pickup the memory range handling again, I can do that. I think I'd likely change this function to be:
void crash_handle_hotplug_event(unsigned int hp_action, unsigned int cpu,
struct memory_notify *mhp);
where on a CPU op the 'cpu' parameter would be valid and 'mhp' NULL, and on a memory op, the 'mhp' would be valid and 'cpu' parameter invalid(0).
I'd likely then stuff these two parameters into struct kimage so that it can be utilized by arch-specific handler, if needed.
And of course, would print out the memory range for debug purposes.
Let me know what you think.
eric
Quoted 117 lines
Quoted 117 lines
kernel/crash_core.c +100
-0+ +/* +* When the struct kimage is allocated, the elfcorehdr_index +* is set to -1. Find the segment containing the elfcorehdr, +* if not already found. This works for both the kexec_load +* and kexec_file_load paths. +*/ +if (image->elfcorehdr_index <0 ) { +unsigned long mem; +unsigned char *ptr; +unsigned int n; + +for (n =0 ; n < image->nr_segments; n++) { + mem = image->segment[n].mem; + ptr =kmap_local_page (pfn_to_page (mem >>PAGE_SHIFT )); +if (ptr) { +/* The segment containing elfcorehdr */ +if (memcmp (ptr,ELFMAG ,SELFMAG ) ==0 ) + image->elfcorehdr_index = (int )n; +kunmap_local (ptr); + } + } + } + +if (image->elfcorehdr_index <0 ) { +pr_err ("unable to locate elfcorehdr segment" ); + goto out; + } + +/* Needed in order for the segments to be updated */ +arch_kexec_unprotect_crashkres (); + +/* Differentiate between normal load and hotplug update */ + image->hp_action = hp_action; + +/* Now invoke arch-specific update handler */ +arch_crash_handle_hotplug_event (image); + +/* No longer handling a hotplug event */ + image->hp_action =KEXEC_CRASH_HP_NONE ; + image->elfcorehdr_updated = true; + +/* Change back to read-only */ +arch_kexec_protect_crashkres (); + +out: +/* Release lock now that update complete */ +kexec_unlock (); +} + +static int crash_memhp_notifier (struct notifier_block *nb,unsigned long val,void *v) +{ +switch (val) { +case MEM_ONLINE : +crash_handle_hotplug_event (KEXEC_CRASH_HP_ADD_MEMORY , +KEXEC_CRASH_HP_INVALID_CPU ); +break ; + +case MEM_OFFLINE : +crash_handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_MEMORY , +KEXEC_CRASH_HP_INVALID_CPU ); +break ; + } +return NOTIFY_OK ; +} + +static struct notifier_block crash_memhp_nb = { + .notifier_call = crash_memhp_notifier, + .priority =0 +}; + +static int crash_cpuhp_online (unsigned int cpu) +{ +crash_handle_hotplug_event (KEXEC_CRASH_HP_ADD_CPU , cpu); +return 0 ; +} + +static int crash_cpuhp_offline (unsigned int cpu) +{ +crash_handle_hotplug_event (KEXEC_CRASH_HP_REMOVE_CPU , cpu); +return 0 ; +} + +static int __init crash_hotplug_init (void ) +{ +int result =0 ; + +if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG )) +register_memory_notifier (&crash_memhp_nb); + +if (IS_ENABLED (CONFIG_HOTPLUG_CPU )) { + result =cpuhp_setup_state_nocalls (CPUHP_BP_PREPARE_DYN , +"crash/cpuhp" , crash_cpuhp_online, crash_cpuhp_offline); + } + +return result; +} + +subsys_initcall (crash_hotplug_init); +#endif kernel/kexec_core.c +6
-0@@ -277,6 +277,12 @@ struct kimage *do_kimage_alloc_init(void)/* Initialize the list of unusable pages */ INIT_LIST_HEAD (&image->unusable_pages); +#ifdef CONFIG_CRASH_HOTPLUG + image->hp_action =KEXEC_CRASH_HP_NONE ; + image->elfcorehdr_index =-1 ; + image->elfcorehdr_updated = false; +#endif +return image; }
Quoted 4 lines
Quoted 3 lines
Signature · 1 line
2.31.1
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
Quoted 31 lines · Eric DeVolder
Quoted 19 lines · Baoquan He
On 04/04/23 at 02:03pm, Eric DeVolder wrote:
......Quoted 17 lines
kernel/crash_core.c +17
-0+static void crash_handle_hotplug_event (unsigned int hp_action,unsigned int cpu) +{ +struct kimage *image; + +/* Obtain lock while changing crash information */ +if (!kexec_trylock ()) { +pr_info ("kexec_trylock() failed, elfcorehdr may be inaccurate\n" ); +return ; + } + +/* Check kdump is not loaded */ +if (!kexec_crash_image) + goto out; + + image = kexec_crash_image; + + if (hp_action == KEXEC_CRASH_HP_ADD_CPU ||Quoted 9 lines
kernel/crash_core.c static int __init crash_save_vmcoreinfo_init(void) · line 741
Quoted 4 lines
kernel/crash_core.c +4
-0+ hp_action == KEXEC_CRASH_HP_REMOVE_CPU) +pr_debug ("hp_action %u, cpu %u\n" , hp_action, cpu); + else + pr_debug("hp_action %u\n", hp_action);Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Quoted 8 lines
Baoquan,
Ah, actually until recently it was used to track the 'offlinecpu' in this function, but tglx pointed out that was un-necessary. That resulted in dropping the code in this function dealing with offlinecpu, leaving this as its only use in this function.
The printing of cpu number is not necessary, but helpful; I use it for debugging.
OK, I see. I am not requesting memory range printing, just try to prove cpu number printing is not so justified. If it's helpful, I am OK with it. Let's see if other people have concern about this.
Quoted 18 lines
The printing of memory range is also not necessary, but in order to do that,
should we choose to do so, requires passing in the memory range to this
function. This patch series did do this early on, and by v7 I dropped it at
your urging (https://lore.kernel.org/lkml/[email protected]/).
At the time, I provided it since I considered this generic infrastructure,
but I could not defend it since x86 didn't need it. However, PPC now needs
this, and is now carrying this as part of PPC support ofCRASH_HOTPLUG(https://lore.kernel.org/linuxppc-dev/[email protected]/T/#u).If you'd rather I pickup the memory range handling again, I can do that. I think I'd likely change this function to be:
void crash_handle_hotplug_event(unsigned int
hp_action, unsigned int cpu,struct memory_notify *mhp);where on a CPU op the 'cpu' parameter would be valid and 'mhp' NULL, and on a memory op, the 'mhp' would be valid and 'cpu' parameter invalid(0).
Quoted 6 lines
I'd likely then stuff these two parameters into struct kimage so that it can be utilized by arch-specific handler, if needed.
And of course, would print out the memory range for debug purposes.
Let me know what you think.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
Quoted 39 lines · Baoquan He
Quoted 38 lines · Eric DeVolder
Quoted 19 lines · Baoquan He
On 04/04/23 at 02:03pm, Eric DeVolder wrote:
......Quoted 17 lines
kernel/crash_core.c +17
-0+static void crash_handle_hotplug_event (unsigned int hp_action,unsigned int cpu) +{ +struct kimage *image; + +/* Obtain lock while changing crash information */ +if (!kexec_trylock ()) { +pr_info ("kexec_trylock() failed, elfcorehdr may be inaccurate\n" ); +return ; + } + +/* Check kdump is not loaded */ +if (!kexec_crash_image) + goto out; + + image = kexec_crash_image; + + if (hp_action == KEXEC_CRASH_HP_ADD_CPU ||Quoted 9 lines
kernel/crash_core.c static int __init crash_save_vmcoreinfo_init(void) · line 741
Quoted 4 lines
kernel/crash_core.c +4
-0+ hp_action == KEXEC_CRASH_HP_REMOVE_CPU) +pr_debug ("hp_action %u, cpu %u\n" , hp_action, cpu); + else + pr_debug("hp_action %u\n", hp_action);Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Baoquan,
Ah, actually until recently it was used to track the 'offlinecpu' in this function, but tglx pointed out that was un-necessary. That resulted in dropping the code in this function dealing with offlinecpu, leaving this as its only use in this function.
Quoted 6 lines
Quoted 1 line
The printing of cpu number is not necessary, but helpful; I use it for debugging.
OK, I see. I am not requesting memory range printing, just try to prove cpu number printing is not so justified. If it's helpful, I am OK with it. Let's see if other people have concern about this.
I do not plan on adding the memory range printing.
Quoted 18 lines
Quoted 18 lines
The printing of memory range is also not necessary, but in order to do that,
should we choose to do so, requires passing in the memory range to this
function. This patch series did do this early on, and by v7 I dropped it at
your urging (https://lore.kernel.org/lkml/[email protected]/).
At the time, I provided it since I considered this generic infrastructure,
but I could not defend it since x86 didn't need it. However, PPC now needs
this, and is now carrying this as part of PPC support ofCRASH_HOTPLUG(https://lore.kernel.org/linuxppc-dev/[email protected]/T/#u).If you'd rather I pickup the memory range handling again, I can do that. I think I'd likely change this function to be:
void crash_handle_hotplug_event(unsigned int
hp_action, unsigned int cpu,struct memory_notify *mhp);where on a CPU op the 'cpu' parameter would be valid and 'mhp' NULL, and on a memory op, the 'mhp' would be valid and 'cpu' parameter invalid(0).
Quoted 7 lines
Quoted 6 lines
I'd likely then stuff these two parameters into struct kimage so that it can be utilized by arch-specific handler, if needed.
And of course, would print out the memory range for debug purposes.
Let me know what you think.
I do not plan on adding the memory range handling; I'll let Sourabh do that as he has a use case for it.
As such, I don't see any other request for changes.
Thanks!
eric
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
On 04/18/23 at 08:55am, Eric DeVolder wrote:
......
Quoted 41 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Baoquan,
Ah, actually until recently it was used to track the 'offlinecpu' in this function, but tglx pointed out that was un-necessary. That resulted in dropping the code in this function dealing with offlinecpu, leaving this as its only use in this function.
Quoted 6 lines
Quoted 1 line
The printing of cpu number is not necessary, but helpful; I use it for debugging.
OK, I see. I am not requesting memory range printing, just try to prove cpu number printing is not so justified. If it's helpful, I am OK with it. Let's see if other people have concern about this.
I do not plan on adding the memory range printing.
Quoted 21 lines
Quoted 21 lines
The printing of memory range is also not necessary, but in order to do that,
should we choose to do so, requires passing in the memory range to this
function. This patch series did do this early on, and by v7 I dropped it at
your urging (https://lore.kernel.org/lkml/[email protected]/).
At the time, I provided it since I considered this generic infrastructure,
but I could not defend it since x86 didn't need it. However, PPC now needs
this, and is now carrying this as part of PPC support ofCRASH_HOTPLUG(https://lore.kernel.org/linuxppc-dev/[email protected]/T/#u).If you'd rather I pickup the memory range handling again, I can do that. I think I'd likely change this function to be:
void crash_handle_hotplug_event(unsigned int
hp_action, unsigned int cpu,struct memory_notify *mhp);where on a CPU op the 'cpu' parameter would be valid and 'mhp' NULL, and on a memory op, the 'mhp' would be valid and 'cpu' parameter invalid(0).
I'd likely then stuff these two parameters into struct kimage so that it can be utilized by arch-specific handler, if needed.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
And of course, would print out the memory range for debug purposes.
Let me know what you think.
I do not plan on adding the memory range handling; I'll let Sourabh do that as he has a use case for it.
As such, I don't see any other request for changes.
OK, then I have no concern about this patchset. Thanks a lot for all these effort, Eric.
Hi x86 maintainers,
Could you help check if there's anything we need improve, or consider taking this patchset?
Thanks
Baoquan
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 2/7] crash: add generic infrastructure for crash hotplug support
In reply to Eric DeVolderQuoted 46 lines · Eric DeVolder
Quoted 21 lines · Baoquan He
On 04/04/23 at 02:03pm, Eric DeVolder wrote:
......Quoted 19 lines
+static void
crash_handle_hotplug_event(unsigned int hp_action, unsigned int cpu)kernel/crash_core.c +6
-0+{ + struct kimage *image; + + /* Obtain lock while changing crash information */ + if (!kexec_trylock()) { + pr_info("kexec_trylock() failed, elfcorehdr may beinaccurate\n");
+ return;
+ }kernel/crash_core.c +8
-0+ + /* Check kdump is not loaded */ + if (!kexec_crash_image) + goto out; + + image = kexec_crash_image; + + if (hp_action == KEXEC_CRASH_HP_ADD_CPU ||Quoted 9 lines
Quoted 4 lines
+
hp_action==KEXEC_CRASH_HP_REMOVE_CPU)
+pr_debug("hp_action %u, cpu %u\n", hp_action, cpu); + else
+pr_debug("hp_action %u\n", hp_action);Seems we passed in the cpu number just for printing here. Wondering why we don't print out hot added/removed memory ranges. Is the cpu number printing necessary?
Baoquan,
Ah, actually until recently it was used to track the 'offlinecpu' in this function, but tglx pointed out that was un-necessary. That resulted in dropping the code in this function dealing with offlinecpu, leaving this as its only use in this function.
The printing of cpu number is not necessary, but helpful; I use it for debugging.
The printing of memory range is also not necessary, but in order to do that, should we choose to do so, requires passing in the memory range to this function. This patch series did do this early on, and by v7 I dropped it at your urging (https://lore.kernel.org/lkml/[email protected]/). At the time, I provided it since I considered this generic infrastructure, but I could not defend it since x86 didn't need it. However, PPC now needs this, and is now carrying this as part of PPC support of
CRASH_HOTPLUG(https://lore.kernel.org/linuxppc-dev/[email protected]/T/#u).If you'd rather I pickup the memory range handling again, I can do that. I think I'd likely change this function to be:
void
crash_handle_hotplug_event(unsigned int hp_action, unsigned int cpu, struct memory_notify *mhp);
Quoted 6 lines
where on a CPU op the 'cpu' parameter would be valid and 'mhp' NULL, and on a memory op, the 'mhp' would be valid and 'cpu' parameter invalid(0).
I'd likely then stuff these two parameters into struct kimage so that it can be utilized by arch-specific handler, if needed.
And of course, would print out the memory range for debug purposes.
I think passing memory_notify as parameter is a better approach compare to adding the same into struct kimage. Because once the crash hotplug event is served the memory_notify object is not useful.
- Sourabh
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 3/7 kexec: exclude elfcorehdr from the segment digest
In reply to Eric DeVolderWhen a crash kernel is loaded via the kexec_file_load() syscall, the kernel places the various segments (ie crash kernel, crash initrd, boot_params, elfcorehdr, purgatory, etc) in memory. For those architectures that utilize purgatory, a hash digest of the segments is calculated for integrity checking. This digest is embedded into the purgatory image prior to placing purgatory in memory.
This patchset updates the elfcorehdr on CPU or memory changes. However, changes to the elfcorehdr in turn cause purgatory integrity checking to fail (at crash time, and no vmcore created). Therefore, this patch explicitly excludes the elfcorehdr segment from the list of segments used to create the digest. By doing so, this permits updates to the elfcorehdr in response to CPU or memory changes, and avoids the need to also recompute the hash digest and reload purgatory.
--- kernel/kexec_file.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/kexec_file.c +6 -0
@@ -726,6 +726,12 @@ static int kexec_calculate_store_digests(struct kimage *image)for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment; +#ifdef CONFIG_CRASH_HOTPLUG +/* Exclude elfcorehdr segment to allow future changes via hotplug */ +if (j == image->elfcorehdr_index) +continue ; +#endif + ksegment = &image->segment[i];/* * Skip purgatory as it will be modified once we put digest
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 4/7 crash: memory and CPU hotplug sysfs attributes
In reply to Eric DeVolderThis introduces the crash_hotplug attribute for memory and CPUs for use by userspace. This change directly facilitates the udev rule for managing userspace re-loading of the crash kernel upon hot un/plug changes.
For memory, this changeset introduces the crash_hotplug attribute to the /sys/devices/system/memory directory. For example:
# udevadm info --attribute-walk /sys/devices/system/memory/memory81
looking at device '/devices/system/memory/memory81':
KERNEL=="memory81"
SUBSYSTEM=="memory"
DRIVER==""
ATTR{online}=="1"
ATTR{phys_device}=="0"
ATTR{phys_index}=="00000051"
ATTR{removable}=="1"
ATTR{state}=="online"
ATTR{valid_zones}=="Movable"looking at parent device '/devices/system/memory':
KERNELS=="memory"
SUBSYSTEMS==""
DRIVERS==""
ATTRS{auto_online_blocks}=="offline"
ATTRS{block_size_bytes}=="8000000"
ATTRS{crash_hotplug}=="1"For CPUs, this changeset introduces the crash_hotplug attribute to the /sys/devices/system/cpu directory. For example:
# udevadm info --attribute-walk /sys/devices/system/cpu/cpu0
looking at device '/devices/system/cpu/cpu0':
KERNEL=="cpu0"
SUBSYSTEM=="cpu"
DRIVER=="processor"
ATTR{crash_notes}=="277c38600"
ATTR{crash_notes_size}=="368"
ATTR{online}=="1"looking at parent device '/devices/system/cpu':
KERNELS=="cpu"
SUBSYSTEMS==""
DRIVERS==""
ATTRS{crash_hotplug}=="1"
ATTRS{isolated}==""
ATTRS{kernel_max}=="8191"
ATTRS{nohz_full}==" (null)"
ATTRS{offline}=="4-7"
ATTRS{online}=="0-3"
ATTRS{possible}=="0-7"
ATTRS{present}=="0-3"With these sysfs attributes in place, it is possible to efficiently instruct the udev rule to skip crash kernel reloading for kernels configured with crash hotplug support.
For example, the following is the proposed udev rule change for RHEL system 98-kexec.rules (as the first lines of the rule file):
# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
When examined in the context of 98-kexec.rules, the above change tests if crash_hotplug is set, and if so, it skips the userspace initiated unload-then-reload of the crash kernel.
CPU and memory checks are separated in accordance with CONFIG_HOTPLUG_CPU and CONFIG_MEMORY_HOTPLUG kernel config options. If an architecture supports, for example, memory hotplug but not CPU hotplug, then the /sys/devices/system/memory/crash_hotplug attribute file is present, but the /sys/devices/system/cpu/crash_hotplug attribute file will NOT be present. Thus the udev rule will skip userspace processing of memory hot un/plug events, but the udev rule will evaluate false for CPU events, thus allowing userspace to process CPU hot un/plug events (ie the unload-then-reload of the kdump capture kernel).
--- .../admin-guide/mm/memory-hotplug.rst | 8 ++++++++ Documentation/core-api/cpu_hotplug.rst | 18 ++++++++++++++++++ drivers/base/cpu.c | 14 ++++++++++++++ drivers/base/memory.c | 13 +++++++++++++ include/linux/kexec.h | 8 ++++++++ 5 files changed, 61 insertions(+)
Documentation/admin-guide/mm/memory-hotplug.rst +8 -0
@@ -291,6 +291,14 @@ The following files are currently defined: Availability depends on the CONFIG_ARCH_MEMORY_PROBE kernel configuration option. ``uevent`` read-write: generic udev file for device subsystems. +``crash_hotplug`` read-only: when changes to the system memory map + occur due to hot un/plug of memory, this file contains + '1' if the kernel updates the kdump capture kernel memory + map itself (via elfcorehdr), or '0' if userspace must update + the kdump capture kernel memory map. + + Availability depends on the CONFIG_MEMORY_HOTPLUG kernel + configuration option. ====================== ========================================================= .. note::
Documentation/core-api/cpu_hotplug.rst +18 -0
@@ -750,6 +750,24 @@ will receive all events. A script like:: can process the event further. +When changes to the CPUs in the system occur, the sysfs file +/sys/devices/system/cpu/crash_hotplug contains '1' if the kernel +updates the kdump capture kernel list of CPUs itself (via elfcorehdr), +or '0' if userspace must update the kdump capture kernel list of CPUs. + +The availability depends on the CONFIG_HOTPLUG_CPU kernel configuration +option. + +To skip userspace processing of CPU hot un/plug events for kdump +(ie the unload-then-reload to obtain a current list of CPUs), this sysfs +file can be used in a udev rule as follows: + + SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" + +For a cpu hot un/plug event, if the architecture supports kernel updates +of the elfcorehdr (which contains the list of CPUs), then the rule skips +the unload-then-reload of the kdump capture kernel. + Kernel Inline Documentations Reference ======================================
drivers/base/cpu.c +14 -0
@@ -282,6 +282,17 @@ static ssize_t print_cpus_nohz_full(struct device *dev, static DEVICE_ATTR(nohz_full, 0444, print_cpus_nohz_full, NULL);#endif +#ifdef CONFIG_HOTPLUG_CPU +#include <linux/kexec.h> +static ssize_t crash_hotplug_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +return sprintf (buf,"%d\n" ,crash_hotplug_cpu_support ()); +} +static DEVICE_ATTR_ADMIN_RO (crash_hotplug); +#endif +static void cpu_device_release (struct device *dev) {/*
@@ -469,6 +480,9 @@ static struct attribute *cpu_root_attrs[] = {#ifdef CONFIG_NO_HZ_FULL &dev_attr_nohz_full.attr,#endif +#ifdef CONFIG_HOTPLUG_CPU + &dev_attr_crash_hotplug.attr, +#endif #ifdef CONFIG_GENERIC_CPU_AUTOPROBE &dev_attr_modalias.attr,#endif
drivers/base/memory.c +13 -0
@@ -490,6 +490,16 @@ static ssize_t auto_online_blocks_store(struct device *dev,static DEVICE_ATTR_RW (auto_online_blocks); +#ifdef CONFIG_MEMORY_HOTPLUG +#include <linux/kexec.h> +static ssize_t crash_hotplug_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +return sprintf (buf,"%d\n" ,crash_hotplug_memory_support ()); +} +static DEVICE_ATTR_RO (crash_hotplug); +#endif +/* * Some architectures will have custom drivers to do this, and * will not need to do it from userspace. The fake hot-add code
@@ -889,6 +899,9 @@ static struct attribute *memory_root_attrs[] = { &dev_attr_block_size_bytes.attr, &dev_attr_auto_online_blocks.attr, +#ifdef CONFIG_MEMORY_HOTPLUG + &dev_attr_crash_hotplug.attr, +#endif NULL };
include/linux/kexec.h +8 -0
@@ -509,6 +509,14 @@ static inline void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages) {static inline void arch_crash_handle_hotplug_event (struct kimage *image) { }#endif +#ifndef crash_hotplug_cpu_support +static inline int crash_hotplug_cpu_support (void ) {return 0 ; } +#endif + +#ifndef crash_hotplug_memory_support +static inline int crash_hotplug_memory_support (void ) {return 0 ; } +#endif +#else /* !CONFIG_KEXEC_CORE */ struct pt_regs ;struct task_struct ;
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 5/7 x86/crash: add x86 crash hotplug support
In reply to Eric DeVolderWhen CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the kexec_load() and kexec_file_load() syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.
In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_MAX_MEMORY_RANGES description. This is used only on the kexec_file_load() syscall; for kexec_load() userspace will need to size the segment similarly.
To accommodate kexec_load() syscall in the absence of kexec_file_load() syscall support, and with CONFIG_CRASH_HOTPLUG enabled, it is necessary to move prepare_elf_headers() and dependents outside of CONFIG_KEXEC_FILE.
--- arch/x86/Kconfig | 13 ++++ arch/x86/include/asm/kexec.h | 15 +++++ arch/x86/kernel/crash.c | 119 ++++++++++++++++++++++++++++++++--- 3 files changed, 140 insertions(+), 7 deletions(-)
arch/x86/Kconfig +13 -0
@@ -2119,6 +2119,19 @@ config CRASH_DUMP (CONFIG_RELOCATABLE=y). For more details see Documentation/admin-guide/kdump/kdump.rst +config CRASH_HOTPLUG + bool "Update the crash elfcorehdr on system configuration changes" + default y + depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) + help + Enable direct update to the crash elfcorehdr (which contains + the list of CPUs and memory regions to be dumped upon a crash) + in response to hot plug/unplug or online/offline of CPUs or + memory. This is a much more advanced approach than userspace + attempting that. + + If unsure, say Y. + config KEXEC_JUMP bool "kexec jump" depends on KEXEC && HIBERNATION
arch/x86/include/asm/kexec.h +15 -0
@@ -212,6 +212,21 @@ typedef void crash_vmclear_fn(void);extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;extern void kdump_nmi_shootdown_cpus (void ); +#ifdef CONFIG_CRASH_HOTPLUG +void arch_crash_handle_hotplug_event (struct kimage *image); +#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event + +#ifdef CONFIG_HOTPLUG_CPU +static inline int crash_hotplug_cpu_support (void ) {return 1 ; } +#define crash_hotplug_cpu_support crash_hotplug_cpu_support +#endif + +#ifdef CONFIG_MEMORY_HOTPLUG +static inline int crash_hotplug_memory_support (void ) {return 1 ; } +#define crash_hotplug_memory_support crash_hotplug_memory_support +#endif +#endif +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_KEXEC_H */
arch/x86/kernel/crash.c +112 -7
166 lines not on this page · raw message
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Hi Eric,
Quoted 16 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.
Quoted 7 lines
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.To accommodate
kexec_load()syscall in the absence of
Firstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
One concern though is that this change assumes corresponding support in kexec-tools. Without that support kexec_load would fail to boot with digest verification failure, iiuc.
I would suggest a flag to advertise to the kernel that kexec-tools/ userspace wants in-kernel update. Something like KEXEC_IN_KERNEL_UPDATE on top of existing flags like KEXEC_ON_CRASH & KEXEC_PRESERVE_CONTEXT. This flag can be used to decide whether in-kernel update needs to be enforced or not. That should make transition to this change smoother without having to break userspace.
Thanks
Hari
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 25 lines · Hari Bathini
Hi Eric,
Quoted 22 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.
Quoted 8 lines
Quoted 1 line
To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.
Eric has posted patchset to modify kexec_tools to support that, please see the link Eric pasted in the cover letter.
http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 31 lines · Baoquan He
Quoted 30 lines · Hari Bathini
Hi Eric,
Quoted 16 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.Quoted 7 lines
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
Quoted 8 lines
Quoted 3 lines
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.Eric has posted patchset to modify
kexec_toolsto support that, please see the link Eric pasted in the cover letter.http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
Right, Baoquan.
I did see that and if I read the code correctly, without that patchset kexec_load would fail. Not with an explicit error that hotplug support is missing or such but it would simply fail to boot into capture kernel with digest verification failure.
My suggestion was to avoid that userspace tool breakage for older kexec-tools version by introducing a new kexec flag that can tell kernel that kexec-tools is ready to use this in-kernel update support. So, if kexec_load happens without the flag, avoid doing an in-kernel update on hotplug. I hope that clears the confusion.
Thanks
Hari
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 48 lines · Hari Bathini
Quoted 31 lines · Baoquan He
Quoted 30 lines · Hari Bathini
Hi Eric,
Quoted 16 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.Quoted 7 lines
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
Quoted 8 lines
Quoted 3 lines
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.Eric has posted patchset to modify
kexec_toolsto support that, please see the link Eric pasted in the cover letter.http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
Right, Baoquan.
I did see that and if I read the code correctly, without that patchset
kexec_loadwould fail. Not with an explicit error that hotplug support is missing or such but it would simply fail to boot into capture kernel with digest verification failure.
Quoted 5 lines
My suggestion was to avoid that userspace tool breakage for older kexec-tools version by introducing a new kexec flag that can tell kernel that kexec-tools is ready to use this in-kernel update support. So, if
kexec_loadhappens without the flag, avoid doing an in-kernel update on hotplug. I hope that clears the confusion.
Yeah, sounds like a good idea. It may be extended in later patch.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 49 lines · Baoquan He
Quoted 48 lines · Hari Bathini
Quoted 31 lines · Baoquan He
Quoted 30 lines · Hari Bathini
Hi Eric,
Quoted 16 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.Quoted 7 lines
To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
Quoted 8 lines
Quoted 3 lines
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.Eric has posted patchset to modify
kexec_toolsto support that, please see the link Eric pasted in the cover letter.http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
Right, Baoquan.
I did see that and if I read the code correctly, without that patchset
kexec_loadwould fail. Not with an explicit error that hotplug support is missing or such but it would simply fail to boot into capture kernel with digest verification failure.
Quoted 7 lines
Quoted 5 lines
My suggestion was to avoid that userspace tool breakage for older kexec-tools version by introducing a new kexec flag that can tell kernel that kexec-tools is ready to use this in-kernel update support. So, if
kexec_loadhappens without the flag, avoid doing an in-kernel update on hotplug. I hope that clears the confusion.Yeah, sounds like a good idea. It may be extended in later patch.
Fixing it in this series itself would be a cleaner way, I guess.
Thanks
Hari
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 30 lines · Hari Bathini
Quoted 28 lines · Baoquan He
Quoted 27 lines · Hari Bathini
Quoted 26 lines · Baoquan He
Quoted 25 lines · Hari Bathini
Hi Eric,
Quoted 22 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
Thank you!
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.
Yes, you've correctly identified that if a hotplug change occurs following kexec_load (made with kexec-tools unaltered for hotplug), then a subsequent panic would in fact fail the purgatory digest verification, and kdump would not happen.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Eric has posted patchset to modify
kexec_toolsto support that, please see the link Eric pasted in the cover letter.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Right, Baoquan.
I did see that and if I read the code correctly, without that patchset
kexec_loadwould fail. Not with an explicit error that hotplug support is missing or such but it would simply fail to boot into capture kernel with digest verification failure.
This is correct.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
My suggestion was to avoid that userspace tool breakage for older kexec-tools version by introducing a new kexec flag that can tell kernel that kexec-tools is ready to use this in-kernel update support. So, if
kexec_loadhappens without the flag, avoid doing an in-kernel update on hotplug. I hope that clears the confusion.Yeah, sounds like a good idea. It may be extended in later patch.
Fixing it in this series itself would be a cleaner way, I guess.
You're suggestion of using a flag makes alot of sense; it is an indication to the kernel that it is valid/okay to modify the kexec_load elfcorehdr. Only kexec-tools that understands this (meaning the elfcorehdr buffer is appropriately sized *and* excludes the elfcorehdr from the purgatory check) would set that flag.
The roll-out of this feature needs to be coordinated, no doubt. There are three pieces to this puzzle: this kernel series, the udev rule changes, and the changes to kexec-tools for kexec_load.
I consider the udev rule changes critical to making this feature work efficiently. I also think that deploying the udev rules immediately is doable since nothing references them, yet; they would be NOPs. And they would be in place when the kernel and/or kexec-tool changes deploy.
However, your point about supporting kexec_load with and without this new flag means the sysfs nodes upon which the udev rule change rely need to be a bit smarter now. (I'm assuming these udev rules will be generally accepted as-is, as they are simple and efficient.)
The sysfs crash_hotplug nodes need to take into account kexec_file_load vs (kexec_load && new_flag). Generally speaking these crash_hotplug sysfs nodes we want to be 1 going forward, but where kexec_load/kexec-tools is older and/or no new_flag, it needs to be 0. In this way the udev rules can remain as proposed and work properly for kexec_file_load and both flavors of kexec_load.
Good catch! I'll post v22 soon.
Eric
Quoted 3 lines
Thanks
Hari
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v21 5/7] x86/crash: add x86 crash hotplug support
Quoted 94 lines · Eric DeVolder
Quoted 30 lines · Hari Bathini
Quoted 28 lines · Baoquan He
Quoted 27 lines · Hari Bathini
Quoted 26 lines · Baoquan He
Quoted 25 lines · Hari Bathini
Hi Eric,
Quoted 22 lines · Eric DeVolder
When CPU or memory is hot un/plugged, or off/onlined, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.
The segment containing the elfcorehdr is identified at run-time in crash_core:crash_handle_hotplug_event(), which works for both the
kexec_load()andkexec_file_load()syscalls. A new elfcorehdr is generated from the available CPUs and memory into a buffer, and then installed over the top of the existing elfcorehdr.In the patch 'kexec: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or
boot_params(as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the
CRASH_MAX_MEMORY_RANGESdescription. This is used only on thekexec_file_load()syscall; forkexec_load()userspace will need to size the segment similarly.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
To accommodate
kexec_load()syscall in the absence ofFirstly, thanks! This series is a nice improvement to kdump support in hotplug environment.
Thank you!
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
One concern though is that this change assumes corresponding support in kexec-tools. Without that support
kexec_loadwould fail to boot with digest verification failure, iiuc.Yes, you've correctly identified that if a hotplug change occurs following
kexec_load(made with kexec-tools unaltered for hotplug), then a subsequent panic would in fact fail the purgatory digest verification, and kdump would not happen.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Eric has posted patchset to modify
kexec_toolsto support that, please see the link Eric pasted in the cover letter.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Right, Baoquan.
I did see that and if I read the code correctly, without that patchset
kexec_loadwould fail. Not with an explicit error that hotplug support is missing or such but it would simply fail to boot into capture kernel with digest verification failure.This is correct.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
My suggestion was to avoid that userspace tool breakage for older kexec-tools version by introducing a new kexec flag that can tell kernel that kexec-tools is ready to use this in-kernel update support. So, if
kexec_loadhappens without the flag, avoid doing an in-kernel update on hotplug. I hope that clears the confusion.Yeah, sounds like a good idea. It may be extended in later patch.
Fixing it in this series itself would be a cleaner way, I guess.
You're suggestion of using a flag makes alot of sense; it is an indication to the kernel that it is valid/okay to modify the
kexec_loadelfcorehdr. Only kexec-tools that understands this (meaning the elfcorehdr buffer is appropriately sized *and* excludes the elfcorehdr from the purgatory check) would set that flag.The roll-out of this feature needs to be coordinated, no doubt. There are three pieces to this puzzle: this kernel series, the udev rule changes, and the changes to kexec-tools for
kexec_load.I consider the udev rule changes critical to making this feature work efficiently. I also think that deploying the udev rules immediately is doable since nothing references them, yet; they would be NOPs. And they would be in place when the kernel and/or kexec-tool changes deploy.
However, your point about supporting
kexec_loadwith and without this new flag means the sysfs nodes upon which the udev rule change rely need to be a bit smarter now. (I'm assuming these udev rules will be generally accepted as-is, as they are simple and efficient.)
Quoted 5 lines
The sysfs
crash_hotplugnodes need to take into accountkexec_file_loadvs (kexec_load&&new_flag). Generally speaking thesecrash_hotplugsysfs nodes we want to be 1 going forward, but where kexec_load/kexec-tools is older and/or nonew_flag, it needs to be 0. In this way the udev rules can remain as proposed and work properly forkexec_file_loadand both flavors ofkexec_load.
Right. That is the tricky part. kdump scripts and kexec-tools have to be in sync if udev rules have to just rely on crash_hotplug.
Thanks
Hari
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 6/7 crash: change crash_prepare_elf64_headers() to for_each_possible_cpu()
In reply to Eric DeVolderThe function crash_prepare_elf64_headers() generates the elfcorehdr which describes the CPUs and memory in the system for the crash kernel. In particular, it writes out ELF PT_NOTEs for memory regions and the CPUs in the system.
With respect to the CPUs, the current implementation utilizes for_each_present_cpu() which means that as CPUs are added and removed, the elfcorehdr must again be updated to reflect the new set of CPUs.
The reasoning behind the change to use for_each_possible_cpu(), is:
- At kernel boot time, all percpu crash_notes are allocated for all possible CPUs; that is, crash_notes are not allocated dynamically when CPUs are plugged/unplugged. Thus the crash_notes for each possible CPU are always available.
- The crash_prepare_elf64_headers() creates an ELF PT_NOTE per CPU. Changing to for_each_possible_cpu() is valid as the crash_notes pointed to by each CPU PT_NOTE are present and always valid.
Furthermore, examining a common crash processing path of:
kernel panic -> crash kernel -> makedumpfile -> 'crash' analyzer
elfcorehdr /proc/vmcore vmcore
reveals how the ELF CPU PT_NOTEs are utilized:
- Upon panic, each CPU is sent an IPI and shuts itself down, recording its state in its crash_notes. When all CPUs are shutdown, the crash kernel is launched with a pointer to the elfcorehdr.
- The crash kernel via linux/fs/proc/vmcore.c does not examine or use the contents of the PT_NOTEs, it exposes them via /proc/vmcore.
- The makedumpfile utility uses /proc/vmcore and reads the CPU PT_NOTEs to craft a nr_cpus variable, which is reported in a header but otherwise generally unused. Makedumpfile creates the vmcore.
- The 'crash' dump analyzer does not appear to reference the CPU PT_NOTEs. Instead it looks-up the cpu_[possible|present|onlin]_mask symbols and directly examines those structure contents from vmcore memory. From that information it is able to determine which CPUs are present and online, and locate the corresponding crash_notes. Said differently, it appears that 'crash' analyzer does not rely on the ELF PT_NOTEs for CPUs; rather it obtains the information directly via kernel symbols and the memory within the vmcore.
(There maybe other vmcore generating and analysis tools that do use these PT_NOTEs, but 'makedumpfile' and 'crash' seems to be the most common solution.)
This change results in the benefit of having all CPUs described in the elfcorehdr, and therefore reducing the need to re-generate the elfcorehdr on CPU changes, at the small expense of an additional 56 bytes per PT_NOTE for not-present-but-possible CPUs.
On systems where kexec_file_load() syscall is utilized, all the above is valid. On systems where kexec_load() syscall is utilized, there may be the need for the elfcorehdr to be regenerated once. The reason being that some archs only populate the 'present' CPUs in the /sys/devices/system/cpus entries, which the userspace 'kexec' utility uses to generate the userspace-supplied elfcorehdr. In this situation, one memory or CPU change will rewrite the elfcorehdr via the crash_prepare_elf64_headers() function and now all possible CPUs will be described, just as with kexec_file_load() syscall.
--- kernel/crash_core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/crash_core.c +2 -2
@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map, ehdr->e_ehsize =sizeof (Elf64_Ehdr); ehdr->e_phentsize =sizeof (Elf64_Phdr); -/* Prepare one phdr of type PT_NOTE for each present CPU */ -for_each_present_cpu (cpu) { +/* Prepare one phdr of type PT_NOTE for each possible CPU */ +for_each_possible_cpu (cpu) { phdr->p_type =PT_NOTE ; notes_addr =per_cpu_ptr_to_phys (per_cpu_ptr (crash_notes, cpu)); phdr->p_offset = phdr->p_paddr = notes_addr;
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v21 7/7 x86/crash: optimize CPU changes
In reply to Eric DeVolderThis patch is dependent upon the patch 'crash: change crash_prepare_elf64_headers() to for_each_possible_cpu()'. With that patch, crash_prepare_elf64_headers() writes out an ELF CPU PT_NOTE for all possible CPUs, thus further CPU changes to the elfcorehdr are not needed.
This change works for kexec_file_load() and kexec_load() syscalls. For kexec_file_load(), crash_prepare_elf64_headers() is utilized directly and thus all ELF CPU PT_NOTEs are in the elfcorehdr already. This is the kimage->file_mode term.
For kexec_load() syscall, one CPU or memory change will cause the elfcorehdr to be updated via crash_prepare_elf64_headers() and at that point all ELF CPU PT_NOTEs are in the elfcorehdr. This is the kimage->elfcorehdr_updated term.
This code is intentionally *NOT* hoisted into crash_handle_hotplug_event() as it would prevent the arch-specific handler from running for CPU changes. This would break PPC, for example, which needs to update other information besides the elfcorehdr, on CPU changes.
--- arch/x86/kernel/crash.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kernel/crash.c +10 -0
@@ -456,6 +456,16 @@ void arch_crash_handle_hotplug_event(struct kimage *image)unsigned long mem, memsz;unsigned long elfsz =0 ; +/* As crash_prepare_elf64_headers() has already described all +* possible CPUs, there is no need to update the elfcorehdr +* for additional CPU changes. This works for both kexec_load() +* and kexec_file_load() syscalls. +*/ +if ((image->file_mode || image->elfcorehdr_updated) && + ((image->hp_action ==KEXEC_CRASH_HP_ADD_CPU ) || + (image->hp_action ==KEXEC_CRASH_HP_REMOVE_CPU ))) +return ; +/* * Create the new elfcorehdr reflecting the changes to CPU and/or * memory resources.
Signature · 7 lines
2.31.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 54 lines · Eric DeVolder
Once the kdump service is loaded, if changes to CPUs or memory occur, either by hot un/plug or off/onlining, the crash elfcorehdr must also be updated.
The elfcorehdr describes to kdump the CPUs and memory in the system, and any inaccuracies can result in a vmcore with missing CPU context or memory regions.
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd,
boot_params, purgatory and elfcorehdr) by the userspace kexec utility. In the original post I outlined the significant performance problems related to offloading this activity to userspace.This patchset introduces a generic crash handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, from either hot un/plug or off/onlining, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate elfcorehdr update.
Note the description in patch 'crash: change
crash_prepare_elf64_headers()tofor_each_possible_cpu()' and 'x86/crash: optimize CPU changes' that enables further optimizations related to CPU plug/unplug/online/offline performance of elfcorehdr updates.In the case of
x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
With this changeset applied, the two rules evaluate to false for CPU and memory change events and thus skip the userspace unload-then-reload of kdump.
- Change to the
kexec_file_loadfor loading the kdump kernel:
Eg. on RHEL: in/usr/bin/kdumpctl, change to:standard_kexec_args="-p -d -s"which adds the -s to select
kexec_file_load()syscall.This kernel patchset also supports
kexec_load()with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:
Quoted 7 lines
http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
To use the kexec-tools patch, apply, build and install kexec-tools, then change the kdumpctl's
standard_kexec_argsto replace the -s with --hotplug. The removal of -s reverts to thekexec_loadsyscall and the addition of --hotplug invokes the changes put forth in the kexec-tools patch.
Other than the nitpick in patch 2, this series looks good to me.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 63 lines · Baoquan He
Quoted 54 lines · Eric DeVolder
Once the kdump service is loaded, if changes to CPUs or memory occur, either by hot un/plug or off/onlining, the crash elfcorehdr must also be updated.
The elfcorehdr describes to kdump the CPUs and memory in the system, and any inaccuracies can result in a vmcore with missing CPU context or memory regions.
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd,
boot_params, purgatory and elfcorehdr) by the userspace kexec utility. In the original post I outlined the significant performance problems related to offloading this activity to userspace.This patchset introduces a generic crash handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, from either hot un/plug or off/onlining, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate elfcorehdr update.
Note the description in patch 'crash: change
crash_prepare_elf64_headers()tofor_each_possible_cpu()' and 'x86/crash: optimize CPU changes' that enables further optimizations related to CPU plug/unplug/online/offline performance of elfcorehdr updates.In the case of
x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
With this changeset applied, the two rules evaluate to false for CPU and memory change events and thus skip the userspace unload-then-reload of kdump.
- Change to the
kexec_file_loadfor loading the kdump kernel:
Eg. on RHEL: in/usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to selectkexec_file_load()syscall.This kernel patchset also supports
kexec_load()with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:Quoted 7 lines
http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
To use the kexec-tools patch, apply, build and install kexec-tools, then change the kdumpctl's
standard_kexec_argsto replace the -s with --hotplug. The removal of -s reverts to thekexec_loadsyscall and the addition of --hotplug invokes the changes put forth in the kexec-tools patch.
Quoted 4 lines
Other than the nitpick in patch 2, this series looks good to me.
Acked-by Baoquan He <[email protected]>
Thank you! I'll await your preference on memory range handling. eric
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi Eric,
Quoted 50 lines · Eric DeVolder
Once the kdump service is loaded, if changes to CPUs or memory occur, either by hot un/plug or off/onlining, the crash elfcorehdr must also be updated.
The elfcorehdr describes to kdump the CPUs and memory in the system, and any inaccuracies can result in a vmcore with missing CPU context or memory regions.
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd,
boot_params, purgatory and elfcorehdr) by the userspace kexec utility. In the original post I outlined the significant performance problems related to offloading this activity to userspace.This patchset introduces a generic crash handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, from either hot un/plug or off/onlining, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate elfcorehdr update.
Note the description in patch 'crash: change
crash_prepare_elf64_headers()tofor_each_possible_cpu()' and 'x86/crash: optimize CPU changes' that enables further optimizations related to CPU plug/unplug/online/offline performance of elfcorehdr updates.In the case of
x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
With this changeset applied, the two rules evaluate to false for CPU and memory change events and thus skip the userspace unload-then-reload of kdump.
- Change to the
kexec_file_loadfor loading the kdump kernel:
Eg. on RHEL: in/usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to selectkexec_file_load()syscall.
Quoted 5 lines
This kernel patchset also supports
kexec_load()with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
With the in-kernel update, the size is anyway getting calculated in
kernel for kexec_file_load case, how about passing the recommended size
for elfcorehdr segment and any other segment that needs an in-kernel update as a sysfs attribute instead of kexec-tools having to do the calculation again in the userspace (get_elfcorehdrsz()) for kexec_load
case. That makes segment size calculation less error prone.
Thanks
Hari
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 59 lines · Hari Bathini
Hi Eric,
Quoted 50 lines · Eric DeVolder
Once the kdump service is loaded, if changes to CPUs or memory occur, either by hot un/plug or off/onlining, the crash elfcorehdr must also be updated.
The elfcorehdr describes to kdump the CPUs and memory in the system, and any inaccuracies can result in a vmcore with missing CPU context or memory regions.
The current solution utilizes udev to initiate an unload-then-reload of the kdump image (eg. kernel, initrd,
boot_params, purgatory and elfcorehdr) by the userspace kexec utility. In the original post I outlined the significant performance problems related to offloading this activity to userspace.This patchset introduces a generic crash handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, from either hot un/plug or off/onlining, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate elfcorehdr update.
Note the description in patch 'crash: change
crash_prepare_elf64_headers()tofor_each_possible_cpu()' and 'x86/crash: optimize CPU changes' that enables further optimizations related to CPU plug/unplug/online/offline performance of elfcorehdr updates.In the case of
x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory; thus no involvement with userspace needed.To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:
- Prevent udev from updating kdump crash kernel on hot un/plug changes. Add the following as the first lines to the RHEL udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:# The kernel updates the crash elfcorehdr for CPU and memory changes SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end" SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
With this changeset applied, the two rules evaluate to false for CPU and memory change events and thus skip the userspace unload-then-reload of kdump.
- Change to the
kexec_file_loadfor loading the kdump kernel:
Eg. on RHEL: in/usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to selectkexec_file_load()syscall.Quoted 5 lines
This kernel patchset also supports
kexec_load()with a modified kexec userspace utility. A working changeset to the kexec userspace utility is posted to the kexec-tools mailing list here:http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
Quoted 8 lines
With the in-kernel update, the size is anyway getting calculated in
kernel forkexec_file_loadcase, how about passing the recommended size
for elfcorehdr segment and any other segment that needs an in-kernel update as a sysfs attribute instead of kexec-tools having to do the calculation again in the userspace (get_elfcorehdrsz()) forkexec_load
case. That makes segment size calculation less error prone.Thanks
Hari
Makes sense, I'll add that to v22.
Thanks!
eric
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec