Skip to contentSkip to conversation

crash: Kernel handling of CPU and memory hot un/plug

PATCH Superseded Eric DeVolder 3 participants 26 messages Mar 3, 2022 – Apr 1, 2022

Subsystems (7)
x86 PowerPC arm64 Live update +3 more subsystems

v5 of 28 Superseded Latest is v28 →

lore
Share & tools
lore
#0 lore raw

v5 cover crash: Kernel handling of CPU and memory hot un/plug

When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions).

The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace.

This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates.

In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed.

To realize the benefits/test this patchset, one must make a couple of minor changes to userspace:

- Disable the udev rule for updating kdump on hot un/plug changes Eg. on RHEL: rm -f /usr/lib/udev/rules.d/98-kexec.rules or other technique to neuter the rule.

- Change to the kexec_file_load for loading the kdump kernel:
Eg. on RHEL: in /usr/bin/kdumpctl, change to:

    standard_kexec_args="-p -d -s"

which adds the -s to select kexec_file_load syscall.

This patchset supports kexec_load with a modified kexec userspace utility, and a working changeset to the kexec userspace utility is provided here (and to use, the above change to standard_kexec_args would be, for example, to append --hotplug-size=262144 instead of -s).

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c index 9826f6d..06adb7e 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -48,6 +48,7 @@
#include <x86/x86-linux.h>

extern struct arch_options_t arch_options;
+extern unsigned long long hotplug_size;

static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
struct crash_elf_info *elf_info)
@@ -975,6 +976,13 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
} else {
memsz = bufsz;
}
+
+ /* If hotplug support enabled, use that size */
+ if (hotplug_size) {
+ memsz = hotplug_size;
+ }
+
+ info->elfcorehdr =
elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
max_addr, -1);
dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
diff --git a/kexec/kexec.c b/kexec/kexec.c
index f63b36b..9569d9a 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -58,6 +58,7 @@

unsigned long long mem_min = 0;
unsigned long long mem_max = ULONG_MAX;
+unsigned long long hotplug_size = 0;
static unsigned long kexec_flags = 0;
/* Flags for kexec file (fd) based syscall */
static unsigned long kexec_file_flags = 0;
@@ -672,6 +673,12 @@ static void update_purgatory(struct kexec_info *info)
if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
continue;
}
+ /* Don't include elfcorehdr in the checksum, if hotplug
+ * support enabled.
+ */
+ if (hotplug_size && (info->segment[i].mem == (void *)info->elfcorehdr)) {
+ continue;
+ }
sha256_update(&ctx, info->segment[i].buf,
info->segment[i].bufsz);
nullsz = info->segment[i].memsz - info->segment[i].bufsz;
@@ -1504,6 +1511,17 @@ int main(int argc, char *argv[])
case OPT_PRINT_CKR_SIZE:
print_crashkernel_region_size();
return 0;
+ case OPT_HOTPLUG_SIZE:
+ /* Reserved the specified size for hotplug growth */
+ hotplug_size = strtoul(optarg, &endptr, 0);
+ if (*endptr) {
+ fprintf(stderr,
+ "Bad option value in --hotplug-size=%s\n",
+ optarg);
+ usage();
+ return 1;
+ }
+ break;
default:
break;
}
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 595dd68..b30dda4 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -169,6 +169,7 @@ struct kexec_info {
int command_line_len;

int skip_checks;
+ unsigned long elfcorehdr;
};

struct arch_map_entry {
@@ -231,7 +232,8 @@ extern int file_types;
#define OPT_PRINT_CKR_SIZE 262
#define OPT_LOAD_LIVE_UPDATE 263
#define OPT_EXEC_LIVE_UPDATE 264
-#define OPT_MAX 265
+#define OPT_HOTPLUG_SIZE 265
+#define OPT_MAX 266
#define KEXEC_OPTIONS \
{ "help", 0, 0, OPT_HELP }, \
{ "version", 0, 0, OPT_VERSION }, \
@@ -258,6 +260,7 @@ extern int file_types;
{ "debug", 0, 0, OPT_DEBUG }, \
{ "status", 0, 0, OPT_STATUS }, \
{ "print-ckr-size", 0, 0, OPT_PRINT_CKR_SIZE }, \
+ { "hotplug-size", 2, 0, OPT_HOTPLUG_SIZE }, \

  #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"

Regards,
eric
---
v5: 3mar2022
- Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per David Hildenbrand.
- Refactored slightly a few patches per Baoquan recommendation.

v4: 9feb2022
https://lkml.org/lkml/2022/2/9/1406
- Refactored patches per Baoquan suggestsions.
- A few corrections, per Baoquan.

v3: 10jan2022
https://lkml.org/lkml/2022/1/10/1212
- Rebasing per Baoquan He request.
- Changed memory notifier per David Hildenbrand.
- Providing example kexec userspace change in cover letter.

RFC v2: 7dec2021
https://lkml.org/lkml/2021/12/7/1088
- Acting upon Baoquan He suggestion of removing elfcorehdr from the purgatory list of segments, removed purgatory code from patchset, and it is signficiantly simpler now.

RFC v1: 18nov2021
https://lkml.org/lkml/2021/11/18/845
- working patchset demonstrating kernel handling of hotplug updates to x86 elfcorehdr for kexec_file_load

RFC: 14dec2020
https://lkml.org/lkml/2020/12/14/532
- proposed concept of allowing kernel to handle hotplug update of elfcorehdr
---

Eric DeVolder (8):
x86/crash: fix minor typo/bug in debug message x86/crash hp: Introduce CRASH_HOTPLUG configuration options
crash: prototype change for crash_prepare_elf64_headers
crash: generic crash hotplug support infrastructure
kexec: exclude elfcorehdr from the segment digest
kexec: exclude hot remove cpu from elfcorehdr notes x86/crash: Add x86 crash hotplug support for kexec_file_load x86/crash: Add x86 crash hotplug support for kexec_load

 arch/arm64/kernel/machine_kexec_file.c |   6 +-
 arch/powerpc/kexec/file_load_64.c      |   2 +-
 arch/x86/Kconfig                       |  26 +++++
 arch/x86/kernel/crash.c                | 151 ++++++++++++++++++++++++-
 include/linux/kexec.h                  |  21 +++-
 kernel/crash_core.c                    | 118 +++++++++++++++++++
 kernel/kexec_file.c                    |  15 ++-
 7 files changed, 329 insertions(+), 10 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH v5 1/8 x86/crash: fix minor typo/bug in debug message

The pr_debug() intends to display the memsz member, but the parameter is actually the bufsz member (which is already displayed). Correct this to display memsz value.

---
 arch/x86/kernel/crash.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/crash.c +1 -1
index e8326a8d1c5d..9730c88530fc 100644
@@ -407,7 +407,7 @@ int crash_load_segments(struct kimage *image)
 	}
 	image->elf_load_addr = kbuf.mem;
 	pr_debug("Loaded ELF headers at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
-		 image->elf_load_addr, kbuf.bufsz, kbuf.bufsz);
+		 image->elf_load_addr, kbuf.bufsz, kbuf.memsz);
 
 	return ret;
 }
Signature · 1 line
2.27.0
#2 lore raw

PATCH v5 2/8 x86/crash hp: Introduce CRASH_HOTPLUG configuration options

In reply to Eric DeVolder

Support for CPU and memory hotplug for crash is controlled by the CRASH_HOTPLUG configuration option, introduced by this patch.

The CRASH_HOTPLUG_ELFCOREHDR_SZ related configuration option is also introduced with this patch.

---
 arch/x86/Kconfig | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
arch/x86/Kconfig +26 -0
index 9f5bd41bf660..37eb26563f07 100644
@@ -2061,6 +2061,32 @@ config CRASH_DUMP
 	  (CONFIG_RELOCATABLE=y).
 	  For more details see Documentation/admin-guide/kdump/kdump.rst
 
+config CRASH_HOTPLUG
+	bool "kernel updates of crash elfcorehdr"
+	depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) && KEXEC_FILE
+	help
+	  Enable the kernel to update the crash elfcorehdr (which contains
+	  the list of CPUs and memory regions) directly when hot plug/unplug
+	  of CPUs or memory. Otherwise userspace must monitor these hot
+	  plug/unplug change notifications via udev in order to
+	  unload-then-reload the crash kernel so that the list of CPUs and
+	  memory regions is kept up-to-date. Note that the udev CPU and
+	  memory change notifications still occur (however, userspace is not
+	  required to monitor for crash dump purposes).
+
+config CRASH_HOTPLUG_ELFCOREHDR_SZ
+	depends on CRASH_HOTPLUG
+	int
+	default 131072
+	help
+	  Specify the maximum size of the elfcorehdr buffer/segment.
+	  The 128KiB default is sized so that it can accommodate 2048
+	  Elf64_Phdr, where each Phdr represents either a CPU or a
+	  region of memory.
+	  For example, this size can accommodate a machine with up to 1024
+	  CPUs and up to 1024 memory regions, eg. as represented by the
+	  'System RAM' entries in /proc/iomem.
+
 config KEXEC_JUMP
 	bool "kexec jump"
 	depends on KEXEC && HIBERNATION
Signature · 1 line
2.27.0
#3 lore raw

Re: [PATCH v5 2/8] x86/crash hp: Introduce CRASH_HOTPLUG configuration options

Quoted 5 lines · Eric DeVolder

Support for CPU and memory hotplug for crash is controlled by the CRASH_HOTPLUG configuration option, introduced by this patch.

The CRASH_HOTPLUG_ELFCOREHDR_SZ related configuration option is also introduced with this patch.

Maybe can rephrase it as:

x86/crash: Introduce new options to support cpu and memory hotplug of crash

CRASH_HOTPLUG is to enable cpu and memory hotplug of crash.

CRASH_HOTPLUG_ELFCOREHDR_SZ is used to Specify the maximum size of the elfcorehdr buffer/segment.

This is a preparation for later usage.

Otherwise, this looks good to me.

Quoted 46 lines
---
 arch/x86/Kconfig | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
arch/x86/Kconfig +26 -0
index 9f5bd41bf660..37eb26563f07 100644
@@ -2061,6 +2061,32 @@ config CRASH_DUMP
 	  (CONFIG_RELOCATABLE=y).
 	  For more details see Documentation/admin-guide/kdump/kdump.rst
 
+config CRASH_HOTPLUG
+	bool "kernel updates of crash elfcorehdr"
+	depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) && KEXEC_FILE
+	help
+	  Enable the kernel to update the crash elfcorehdr (which contains
+	  the list of CPUs and memory regions) directly when hot plug/unplug
+	  of CPUs or memory. Otherwise userspace must monitor these hot
+	  plug/unplug change notifications via udev in order to
+	  unload-then-reload the crash kernel so that the list of CPUs and
+	  memory regions is kept up-to-date. Note that the udev CPU and
+	  memory change notifications still occur (however, userspace is not
+	  required to monitor for crash dump purposes).
+
+config CRASH_HOTPLUG_ELFCOREHDR_SZ
+	depends on CRASH_HOTPLUG
+	int
+	default 131072
+	help
+	  Specify the maximum size of the elfcorehdr buffer/segment.
+	  The 128KiB default is sized so that it can accommodate 2048
+	  Elf64_Phdr, where each Phdr represents either a CPU or a
+	  region of memory.
+	  For example, this size can accommodate a machine with up to 1024
+	  CPUs and up to 1024 memory regions, eg. as represented by the
+	  'System RAM' entries in /proc/iomem.
+
 config KEXEC_JUMP
 	bool "kexec jump"
 	depends on KEXEC && HIBERNATION
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH v5 2/8] x86/crash hp: Introduce CRASH_HOTPLUG configuration options

Quoted 11 lines · Baoquan He
Quoted 5 lines · Eric DeVolder

Support for CPU and memory hotplug for crash is controlled by the CRASH_HOTPLUG configuration option, introduced by this patch.

The CRASH_HOTPLUG_ELFCOREHDR_SZ related configuration option is also introduced with this patch.

Maybe can rephrase it as:

x86/crash: Introduce new options to support cpu and memory hotplug of crash

Quoted 8 lines

CRASH_HOTPLUG is to enable cpu and memory hotplug of crash.

CRASH_HOTPLUG_ELFCOREHDR_SZ is used to Specify the maximum size of the elfcorehdr buffer/segment.

This is a preparation for later usage.

Otherwise, this looks good to me.

Commit message updated.
eric

Quoted 49 lines
Quoted 48 lines
---
  arch/x86/Kconfig | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
arch/x86/Kconfig +26 -0
index 9f5bd41bf660..37eb26563f07 100644
@@ -2061,6 +2061,32 @@ config CRASH_DUMP
  	  (CONFIG_RELOCATABLE=y).
  	  For more details see Documentation/admin-guide/kdump/kdump.rst
 
+config CRASH_HOTPLUG
+	bool "kernel updates of crash elfcorehdr"
+	depends on CRASH_DUMP && (HOTPLUG_CPU || MEMORY_HOTPLUG) && KEXEC_FILE
+	help
+	  Enable the kernel to update the crash elfcorehdr (which contains
+	  the list of CPUs and memory regions) directly when hot plug/unplug
+	  of CPUs or memory. Otherwise userspace must monitor these hot
+	  plug/unplug change notifications via udev in order to
+	  unload-then-reload the crash kernel so that the list of CPUs and
+	  memory regions is kept up-to-date. Note that the udev CPU and
+	  memory change notifications still occur (however, userspace is not
+	  required to monitor for crash dump purposes).
+
+config CRASH_HOTPLUG_ELFCOREHDR_SZ
+	depends on CRASH_HOTPLUG
+	int
+	default 131072
+	help
+	  Specify the maximum size of the elfcorehdr buffer/segment.
+	  The 128KiB default is sized so that it can accommodate 2048
+	  Elf64_Phdr, where each Phdr represents either a CPU or a
+	  region of memory.
+	  For example, this size can accommodate a machine with up to 1024
+	  CPUs and up to 1024 memory regions, eg. as represented by the
+	  'System RAM' entries in /proc/iomem.
+
  config KEXEC_JUMP
  	bool "kexec jump"
  	depends on KEXEC && HIBERNATION
Signature · 1 line
2.27.0
#5 lore raw

PATCH v5 3/8 crash: prototype change for crash_prepare_elf64_headers

In reply to Eric DeVolder

From within crash_prepare_elf64_headers() there is a need to reference the struct kimage hotplug members. As such, this change passes the struct kimage as a parameter to the crash_prepare_elf64_headers().

This is preparation for later patch, no functionality change.

---
 arch/arm64/kernel/machine_kexec_file.c | 6 +++---
 arch/powerpc/kexec/file_load_64.c      | 2 +-
 arch/x86/kernel/crash.c                | 3 ++-
 include/linux/kexec.h                  | 5 +++--
 kernel/kexec_file.c                    | 4 ++--
 5 files changed, 11 insertions(+), 9 deletions(-)
arch/arm64/kernel/machine_kexec_file.c +3 -3
index 59c648d51848..7dbafb42ecf2 100644
@@ -39,7 +39,7 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image)
 	return kexec_image_post_load_cleanup_default(image);
 }
 
-static int prepare_elf_headers(void **addr, unsigned long *sz)
+static int prepare_elf_headers(struct kimage *image, void **addr, unsigned long *sz)
 {
 	struct crash_mem *cmem;
 	unsigned int nr_ranges;
@@ -67,7 +67,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz)
 	ret = crash_exclude_mem_range(cmem, crashk_res.start, crashk_res.end);
 
 	if (!ret)
-		ret =  crash_prepare_elf64_headers(cmem, true, addr, sz);
+		ret =  crash_prepare_elf64_headers(image, cmem, true, addr, sz);
 
 	kfree(cmem);
 	return ret;
@@ -96,7 +96,7 @@ int load_other_segments(struct kimage *image,
 
 	/* load elf core header */
 	if (image->type == KEXEC_TYPE_CRASH) {
-		ret = prepare_elf_headers(&headers, &headers_sz);
+		ret = prepare_elf_headers(image, &headers, &headers_sz);
 		if (ret) {
 			pr_err("Preparing elf core header failed\n");
 			goto out_err;
arch/powerpc/kexec/file_load_64.c +1 -1
index b4981b651d9a..07da6bf1cf24 100644
@@ -797,7 +797,7 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf)
 		goto out;
 
 	/* Setup elfcorehdr segment */
-	ret = crash_prepare_elf64_headers(cmem, false, &headers, &headers_sz);
+	ret = crash_prepare_elf64_headers(image, cmem, false, &headers, &headers_sz);
 	if (ret) {
 		pr_err("Failed to prepare elf headers for the core\n");
 		goto out;
arch/x86/kernel/crash.c +2 -1
index 9730c88530fc..9db41cce8d97 100644
@@ -265,7 +265,8 @@ static int prepare_elf_headers(struct kimage *image, void **addr,
 		goto out;
 
 	/* By default prepare 64bit headers */
-	ret =  crash_prepare_elf64_headers(cmem, IS_ENABLED(CONFIG_X86_64), addr, sz);
+	ret =  crash_prepare_elf64_headers(image, cmem,
+				IS_ENABLED(CONFIG_X86_64), addr, sz);
 
 out:
 	vfree(cmem);
include/linux/kexec.h +3 -2
index 0c994ae37729..d7b59248441b 100644
@@ -221,8 +221,9 @@ struct crash_mem {
 extern int crash_exclude_mem_range(struct crash_mem *mem,
 				   unsigned long long mstart,
 				   unsigned long long mend);
-extern int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
-				       void **addr, unsigned long *sz);
+extern int crash_prepare_elf64_headers(struct kimage *image,
+	struct crash_mem *mem, int kernel_map,
+	void **addr, unsigned long *sz);
 #endif /* CONFIG_KEXEC_FILE */
 
 #ifdef CONFIG_KEXEC_ELF
kernel/kexec_file.c +2 -2
index 8347fc158d2b..801d0d0a5012 100644
@@ -1260,8 +1260,8 @@ int crash_exclude_mem_range(struct crash_mem *mem,
 	return 0;
 }
 
-int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
-			  void **addr, unsigned long *sz)
+int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem,
+	int kernel_map, void **addr, unsigned long *sz)
 {
 	Elf64_Ehdr *ehdr;
 	Elf64_Phdr *phdr;
Signature · 1 line
2.27.0
#6 lore raw

PATCH v5 4/8 crash: generic crash hotplug support infrastructure

In reply to Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

---
 include/linux/kexec.h |  16 +++++++
 kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
 
 	/* Information for loading purgatory */
 	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
 #endif
 
 #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
 	unsigned long elf_load_addr;
 };
 
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
 /* kexec interface functions */
 extern void machine_kexec(struct kimage *image);
 extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0

123 lines not on this page · raw message

Signature · 1 line
2.27.0
#7 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Hello Eric,

Quoted 36 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

---
  include/linux/kexec.h |  16 +++++++
  kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 124 insertions(+)
include/linux/kexec.h +0 -0
index d7b59248441b..b11d75a6b2bc 100644

include/linux/kexec.h struct kimage { · line 308

Quoted 9 lines
include/linux/kexec.h +6 -0
@@ -300,6 +300,13 @@ struct kimage {
 
  	/* Information for loading purgatory */
  	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;

How about keeping an array to track all kexec segment index need to be updated in
crash hotplug handler.

struct hp_segment {
?? name;
?? index;
?? is_valid;
?}

It will be helpful if architecture need to updated multiple kexec segments? for a hotplug event.

For example, on PowerPC, we might need to update FDT and elfcorehdr on memory hot plug/unplug.

Thanks,
Sourabh Jain

Quoted 149 lines
include/linux/kexec.h +10 -0
+#endif
  #endif
 
  #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
  	unsigned long elf_load_addr;
  };
 
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
  /* kexec interface functions */
  extern void machine_kexec(struct kimage *image);
  extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
  #include <linux/init.h>
  #include <linux/utsname.h>
  #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
 
  #include <asm/page.h>
  #include <asm/sections.h>
 
  #include <crypto/sha1.h>
 
+#include "kexec_internal.h"
+
  /* vmcoreinfo stuff */
  unsigned char *vmcoreinfo_data;
  size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
  }
 
  subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
#8 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 57 lines · Sourabh Jain

Hello Eric,

Quoted 38 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

---
? include/linux/kexec.h |? 16 +++++++
? kernel/crash_core.c?? | 108 ++++++++++++++++++++++++++++++++++++++++++ ? 2 files changed, 124 insertions(+)

include/linux/kexec.h +0 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {

????? /* Information for loading purgatory */
????? struct purgatory_info purgatory_info;

include/linux/kexec.h

Quoted 6 lines
include/linux/kexec.h +6 -0
+
+#ifdef CONFIG_CRASH_HOTPLUG
+??? bool hotplug_event;
+??? int offlinecpu;
+??? bool elf_index_valid;
+??? int elf_index;

How about keeping an array to track all kexec segment index need to be updated in crash hotplug handler.

struct hp_segment {
?? name;
?? index;
?? is_valid;
?}

Quoted 6 lines

It will be helpful if architecture need to updated multiple kexec segments? for a hotplug event.

For example, on PowerPC, we might need to update FDT and elfcorehdr on memory hot plug/unplug.

Thanks,
Sourabh Jain

Sourabh,
I'm OK with that. Another idea might be if there are just two, and one of them is elfcorehdr, then perhaps in addition to elf_index and elf_index_valid, maybe we add an arch_index and arch_index_valid? In the case of PPC, the FDT would be stored in arch_index?

Either way.
Thanks!
eric

Quoted 145 lines
Quoted 145 lines

+#endif
? #endif
? #ifdef CONFIG_IMA_KEXEC

include/linux/kexec.h +9 -0
@@ -316,6 +323,15 @@ struct kimage {
 ????? unsigned long elf_load_addr;
 ? };
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+??? unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU?? 0
+#define KEXEC_CRASH_HP_ADD_CPU????? 1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY?? 3
+#endif /* CONFIG_CRASH_HOTPLUG */
+

? /* kexec interface functions */
? extern void machine_kexec(struct kimage *image); ? extern int machine_kexec_prepare(struct kimage *image);

kernel/crash_core.c +0 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@

? #include <linux/init.h>
? #include <linux/utsname.h>
? #include <linux/vmalloc.h>

kernel/crash_core.c +3 -0
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>

? #include <asm/page.h>
? #include <asm/sections.h>
? #include <crypto/sha1.h>

kernel/crash_core.c +2 -0
+#include "kexec_internal.h"
+

? /* vmcoreinfo stuff */
? unsigned char *vmcoreinfo_data;
? size_t vmcoreinfo_size;

kernel/crash_core.c +103 -0
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
 ? }
 ? subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+??? unsigned int hp_action, unsigned long a, unsigned long b)
+{
+??? pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+??? unsigned long a, unsigned long b)
+{
+??? /* Obtain lock while changing crash information */
+??? if (!mutex_trylock(&kexec_mutex))
+??????? return;
+
+??? /* Check kdump is loaded */
+??? if (kexec_crash_image) {
+??????? pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+??????????? a, b);
+
+??????? /* Needed in order for the segments to be updated */
+??????? arch_kexec_unprotect_crashkres();
+
+??????? /* Flag to differentiate between normal load and hotplug */
+??????? kexec_crash_image->hotplug_event = true;
+
+??????? /* Now invoke arch-specific update handler */
+??????? arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+??????? /* No longer handling a hotplug event */
+??????? kexec_crash_image->hotplug_event = false;
+
+??????? /* Change back to read-only */
+??????? arch_kexec_protect_crashkres();
+??? }
+
+??? /* Release lock now that update complete */
+??? mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+??? unsigned long val, void *v)
+{
+??? struct memory_notify *mhp = v;
+??? unsigned long start, end;
+
+??? start = mhp->start_pfn << PAGE_SHIFT;
+??? end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+??? switch (val) {
+??? case MEM_ONLINE:
+??????? crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+??????????? start, end-start);
+??????? break;
+
+??? case MEM_OFFLINE:
+??????? crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+??????????? start, end-start);
+??????? break;
+??? }
+??? return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+??? .notifier_call = crash_memhp_notifier,
+??? .priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+??? crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+??? return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+??? crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+??? return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+??? int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+??? register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+??? result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+??????????????? "crash/cpuhp",
+??????????????? crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+??? return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
#9 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 68 lines · Eric DeVolder
Quoted 65 lines · Sourabh Jain

Hello Eric,

Quoted 39 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

---
? include/linux/kexec.h |? 16 +++++++
? kernel/crash_core.c?? | 108
++++++++++++++++++++++++++++++++++++++++++
? 2 files changed, 124 insertions(+)

include/linux/kexec.h +0 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {

????? /* Information for loading purgatory */
????? struct purgatory_info purgatory_info;

include/linux/kexec.h

Quoted 6 lines
include/linux/kexec.h +6 -0
+
+#ifdef CONFIG_CRASH_HOTPLUG
+??? bool hotplug_event;
+??? int offlinecpu;
+??? bool elf_index_valid;
+??? int elf_index;

How about keeping an array to track all kexec segment index need to be updated in
crash hotplug handler.

struct hp_segment {
??? name;
??? index;
??? is_valid;
??}

It will be helpful if architecture need to updated multiple kexec segments? for a hotplug event.

For example, on PowerPC, we might need to update FDT and elfcorehdr on memory hot plug/unplug.

Quoted 8 lines
Quoted 2 lines

Thanks,
Sourabh Jain

Sourabh,
I'm OK with that. Another idea might be if there are just two, and one of them is elfcorehdr, then perhaps in addition to elf_index and elf_index_valid, maybe we add an arch_index and arch_index_valid? In the case of PPC, the FDT would be stored in arch_index?

Yes it seems like we might not need to keep more than two kexec indexes. Since this indexes are arch specific lets push them to struct kimage_arch (part of kimage). So for now I will push fdt_index to struct kimage_arch.

Thanks,
Sourabh Jain

#10 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

In reply to Eric DeVolder
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() will dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls(). The callbacks then call crash_hotplug_handler() to handle.

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

Quoted 173 lines
---
 include/linux/kexec.h |  16 +++++++
 kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
 
 	/* Information for loading purgatory */
 	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
 #endif
 
 #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
 	unsigned long elf_load_addr;
 };
 
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
 /* kexec interface functions */
 extern void machine_kexec(struct kimage *image);
 extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
 #include <linux/init.h>
 #include <linux/utsname.h>
 #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
 
 #include <crypto/sha1.h>
 
+#include "kexec_internal.h"
+
 /* vmcoreinfo stuff */
 unsigned char *vmcoreinfo_data;
 size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
 }
 
 subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#11 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

These callbacks and notifier will call crash_hotplug_handler() to handle captured event when invoked.

Quoted 175 lines
Quoted 174 lines
---
 include/linux/kexec.h |  16 +++++++
 kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
 
 	/* Information for loading purgatory */
 	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
 #endif
 
 #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
 	unsigned long elf_load_addr;
 };
 
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
 /* kexec interface functions */
 extern void machine_kexec(struct kimage *image);
 extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
 #include <linux/init.h>
 #include <linux/utsname.h>
 #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
 
 #include <crypto/sha1.h>
 
+#include "kexec_internal.h"
+
 /* vmcoreinfo stuff */
 unsigned char *vmcoreinfo_data;
 size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
 }
 
 subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#12 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 35 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Quoted 8 lines

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Quoted 183 lines

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

These callbacks and notifier will call crash_hotplug_handler() to handle captured event when invoked.

Quoted 175 lines
Quoted 174 lines
---
  include/linux/kexec.h |  16 +++++++
  kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
 
  	/* Information for loading purgatory */
  	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
  #endif
 
  #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
  	unsigned long elf_load_addr;
  };
 
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
  /* kexec interface functions */
  extern void machine_kexec(struct kimage *image);
  extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
  #include <linux/init.h>
  #include <linux/utsname.h>
  #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
 
  #include <asm/page.h>
  #include <asm/sections.h>
 
  #include <crypto/sha1.h>
 
+#include "kexec_internal.h"
+
  /* vmcoreinfo stuff */
  unsigned char *vmcoreinfo_data;
  size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
  }
 
  subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#13 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 45 lines · Eric DeVolder

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 40 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

Quoted 5 lines
Quoted 3 lines

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Right.

While checking it, I notice hp_action which you don't use actually. Can you reconsider that part of design, the hp_action, the a, b parameter passed to handler?

Quoted 178 lines
Quoted 177 lines

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

These callbacks and notifier will call crash_hotplug_handler() to handle captured event when invoked.

Quoted 168 lines
Quoted 167 lines
---
  include/linux/kexec.h |  16 +++++++
  kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
  	/* Information for loading purgatory */
  	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
  #endif
  #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
  	unsigned long elf_load_addr;
  };
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
  /* kexec interface functions */
  extern void machine_kexec(struct kimage *image);
  extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
  #include <linux/init.h>
  #include <linux/utsname.h>
  #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
  #include <asm/page.h>
  #include <asm/sections.h>
  #include <crypto/sha1.h>
+#include "kexec_internal.h"
+
  /* vmcoreinfo stuff */
  unsigned char *vmcoreinfo_data;
  size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
  }
  subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#14 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 49 lines · Baoquan He
Quoted 48 lines · Eric DeVolder

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 43 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

Quoted 8 lines
Quoted 2 lines

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Right.

While checking it, I notice hp_action which you don't use actually. Can you reconsider that part of design, the hp_action, the a, b parameter passed to handler?

Sure I can remove. I initially put in there as this was generic infrastructure and not sure if it would benefit others.
eric

Quoted 180 lines
Quoted 179 lines
Quoted 178 lines

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

These callbacks and notifier will call crash_hotplug_handler() to handle captured event when invoked.

Quoted 168 lines
Quoted 167 lines
---
   include/linux/kexec.h |  16 +++++++
   kernel/crash_core.c   | 108 ++++++++++++++++++++++++++++++++++++++++++
   2 files changed, 124 insertions(+)
include/linux/kexec.h +16 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {
   	/* Information for loading purgatory */
   	struct purgatory_info purgatory_info;
+
+#ifdef CONFIG_CRASH_HOTPLUG
+	bool hotplug_event;
+	int offlinecpu;
+	bool elf_index_valid;
+	int elf_index;
+#endif
   #endif
   #ifdef CONFIG_IMA_KEXEC
@@ -316,6 +323,15 @@ struct kimage {
   	unsigned long elf_load_addr;
   };
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU   0
+#define KEXEC_CRASH_HP_ADD_CPU      1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY   3
+#endif /* CONFIG_CRASH_HOTPLUG */
+
   /* kexec interface functions */
   extern void machine_kexec(struct kimage *image);
   extern int machine_kexec_prepare(struct kimage *image);
kernel/crash_core.c +108 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@
   #include <linux/init.h>
   #include <linux/utsname.h>
   #include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
   #include <asm/page.h>
   #include <asm/sections.h>
   #include <crypto/sha1.h>
+#include "kexec_internal.h"
+
   /* vmcoreinfo stuff */
   unsigned char *vmcoreinfo_data;
   size_t vmcoreinfo_size;
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
   }
   subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+	unsigned long a, unsigned long b)
+{
+	/* Obtain lock while changing crash information */
+	if (!mutex_trylock(&kexec_mutex))
+		return;
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+			a, b);
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		kexec_crash_image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+		/* No longer handling a hotplug event */
+		kexec_crash_image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+	unsigned long val, void *v)
+{
+	struct memory_notify *mhp = v;
+	unsigned long start, end;
+
+	start = mhp->start_pfn << PAGE_SHIFT;
+	end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+	switch (val) {
+	case MEM_ONLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+			start, end-start);
+		break;
+
+	case MEM_OFFLINE:
+		crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+			start, end-start);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+	return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+	register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+	result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				"crash/cpuhp",
+				crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#15 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Baoquan, a comment below.
eric

Quoted 57 lines · Eric DeVolder
Quoted 54 lines · Baoquan He
Quoted 45 lines · Eric DeVolder

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 40 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

Quoted 5 lines
Quoted 3 lines

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Right.

Quoted 8 lines
Quoted 3 lines

While checking it, I notice hp_action which you don't use actually. Can you reconsider that part of design, the hp_action, the a, b parameter passed to handler?

Sure I can remove. I initially put in there as this was generic infrastructure and not sure if it would benefit others.
eric

Actually, I will keep the hp_action as the work by Sourabh Jain for PPC uses the hp_action. I'll drop the a and b.

Also, shall I post v6, or are you still looking at patches 7 and 8?

Thanks,
eric

Quoted 180 lines
Quoted 180 lines
Quoted 179 lines
Quoted 178 lines

To support memory hotplug, a notifier crash_memhp_nb is registered to memory_chain to watch MEM_ONLINE and MEM_OFFLINE events.

These callbacks and notifier will call crash_hotplug_handler() to handle captured event when invoked.

Quoted 168 lines
Quoted 167 lines

---
?? include/linux/kexec.h |? 16 +++++++
?? kernel/crash_core.c?? | 108 ++++++++++++++++++++++++++++++++++++++++++ ?? 2 files changed, 124 insertions(+)

include/linux/kexec.h +0 -0
index d7b59248441b..b11d75a6b2bc 100644
@@ -300,6 +300,13 @@ struct kimage {

?????? /* Information for loading purgatory */
?????? struct purgatory_info purgatory_info;

include/linux/kexec.h +7 -0
+
+#ifdef CONFIG_CRASH_HOTPLUG
+??? bool hotplug_event;
+??? int offlinecpu;
+??? bool elf_index_valid;
+??? int elf_index;
+#endif

?? #endif
?? #ifdef CONFIG_IMA_KEXEC

include/linux/kexec.h +9 -0
@@ -316,6 +323,15 @@ struct kimage {
 ?????? unsigned long elf_load_addr;
 ?? };
+#ifdef CONFIG_CRASH_HOTPLUG
+void arch_crash_hotplug_handler(struct kimage *image,
+??? unsigned int hp_action, unsigned long a, unsigned long b);
+#define KEXEC_CRASH_HP_REMOVE_CPU?? 0
+#define KEXEC_CRASH_HP_ADD_CPU????? 1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY 2
+#define KEXEC_CRASH_HP_ADD_MEMORY?? 3
+#endif /* CONFIG_CRASH_HOTPLUG */
+

?? /* kexec interface functions */
?? extern void machine_kexec(struct kimage *image); ?? extern int machine_kexec_prepare(struct kimage *image);

kernel/crash_core.c +0 -0
index 256cf6db573c..76959d440f71 100644
@@ -9,12 +9,17 @@

?? #include <linux/init.h>
?? #include <linux/utsname.h>
?? #include <linux/vmalloc.h>

kernel/crash_core.c +3 -0
+#include <linux/highmem.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>

?? #include <asm/page.h>
?? #include <asm/sections.h>
?? #include <crypto/sha1.h>

kernel/crash_core.c +2 -0
+#include "kexec_internal.h"
+

?? /* vmcoreinfo stuff */
?? unsigned char *vmcoreinfo_data;
?? size_t vmcoreinfo_size;

kernel/crash_core.c +103 -0
@@ -491,3 +496,106 @@ static int __init crash_save_vmcoreinfo_init(void)
 ?? }
 ?? subsys_initcall(crash_save_vmcoreinfo_init);
+
+#ifdef CONFIG_CRASH_HOTPLUG
+void __weak arch_crash_hotplug_handler(struct kimage *image,
+??? unsigned int hp_action, unsigned long a, unsigned long b)
+{
+??? pr_warn("crash hp: %s not implemented", __func__);
+}
+
+static void crash_hotplug_handler(unsigned int hp_action,
+??? unsigned long a, unsigned long b)
+{
+??? /* Obtain lock while changing crash information */
+??? if (!mutex_trylock(&kexec_mutex))
+??????? return;
+
+??? /* Check kdump is loaded */
+??? if (kexec_crash_image) {
+??????? pr_debug("crash hp: hp_action %u, a %lu, b %lu", hp_action,
+??????????? a, b);
+
+??????? /* Needed in order for the segments to be updated */
+??????? arch_kexec_unprotect_crashkres();
+
+??????? /* Flag to differentiate between normal load and hotplug */
+??????? kexec_crash_image->hotplug_event = true;
+
+??????? /* Now invoke arch-specific update handler */
+??????? arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
+
+??????? /* No longer handling a hotplug event */
+??????? kexec_crash_image->hotplug_event = false;
+
+??????? /* Change back to read-only */
+??????? arch_kexec_protect_crashkres();
+??? }
+
+??? /* Release lock now that update complete */
+??? mutex_unlock(&kexec_mutex);
+}
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int crash_memhp_notifier(struct notifier_block *nb,
+??? unsigned long val, void *v)
+{
+??? struct memory_notify *mhp = v;
+??? unsigned long start, end;
+
+??? start = mhp->start_pfn << PAGE_SHIFT;
+??? end = ((mhp->start_pfn + mhp->nr_pages) << PAGE_SHIFT) - 1;
+
+??? switch (val) {
+??? case MEM_ONLINE:
+??????? crash_hotplug_handler(KEXEC_CRASH_HP_ADD_MEMORY,
+??????????? start, end-start);
+??????? break;
+
+??? case MEM_OFFLINE:
+??????? crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_MEMORY,
+??????????? start, end-start);
+??????? break;
+??? }
+??? return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+??? .notifier_call = crash_memhp_notifier,
+??? .priority = 0
+};
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+static int crash_cpuhp_online(unsigned int cpu)
+{
+??? crash_hotplug_handler(KEXEC_CRASH_HP_ADD_CPU, cpu, 0);
+??? return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+??? crash_hotplug_handler(KEXEC_CRASH_HP_REMOVE_CPU, cpu, 0);
+??? return 0;
+}
+#endif
+
+static int __init crash_hotplug_init(void)
+{
+??? int result = 0;
+
+#if defined(CONFIG_MEMORY_HOTPLUG)
+??? register_memory_notifier(&crash_memhp_nb);
+#endif
+
+#if defined(CONFIG_HOTPLUG_CPU)
+??? result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+??????????????? "crash/cpuhp",
+??????????????? crash_cpuhp_online, crash_cpuhp_offline);
+#endif
+
+??? return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#16 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 65 lines · Eric DeVolder

Baoquan, a comment below.
eric

Quoted 61 lines · Eric DeVolder
Quoted 57 lines · Baoquan He
Quoted 45 lines · Eric DeVolder

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 40 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

Quoted 5 lines
Quoted 3 lines

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Right.

While checking it, I notice hp_action which you don't use actually. Can you reconsider that part of design, the hp_action, the a, b parameter passed to handler?

Quoted 7 lines
Quoted 4 lines

Sure I can remove. I initially put in there as this was generic infrastructure and not sure if it would benefit others. eric

Actually, I will keep the hp_action as the work by Sourabh Jain for PPC uses the hp_action. I'll drop the a and b.

Sounds great.

Quoted 2 lines

Also, shall I post v6, or are you still looking at patches 7 and 8?

Will check today, thanks for the effort.

#17 lore raw

Re: [PATCH v5 4/8] crash: generic crash hotplug support infrastructure

Quoted 54 lines · Baoquan He
Quoted 53 lines · Eric DeVolder

Baoquan, a comment below.
eric

Quoted 49 lines · Eric DeVolder
Quoted 46 lines · Baoquan He
Quoted 45 lines · Eric DeVolder

Baoquan,
Thanks, I've offered a minor correction below.
eric

Quoted 40 lines · Baoquan He
Quoted 22 lines · Baoquan He
Quoted 21 lines · Eric DeVolder

This patch introduces a generic crash hot plug/unplug infrastructure for CPU and memory changes. Upon CPU and memory changes, a generic crash_hotplug_handler() obtains the appropriate lock, does some important house keeping and then dispatches the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(), and when that handler returns, the lock is released.

This patch modifies crash_core.c to implement a subsys_initcall() function that installs handlers for hot plug/unplug events. If CPU hotplug is enabled, then cpuhp_setup_state() is invoked to register a handler for CPU changes. Similarly, if memory hotplug is enabled, then register_memory_notifier() is invoked to install a handler for memory changes. These handlers in turn invoke the common generic handler crash_hotplug_handler().

On the CPU side, cpuhp_setup_state_nocalls() is invoked with parameter CPUHP_AP_ONLINE_DYN. While this works, when a CPU is being unplugged, the CPU still shows up in foreach_present_cpu() during the regeneration of the new CPU list, thus the need to explicitly check and exclude the soon-to-be offlined CPU in crash_prepare_elf64_headers().

Quoted 8 lines
Quoted 3 lines

On the memory side, each un/plugged memory block passes through the handler. For example, if a 1GiB DIMM is hotplugged, that generate 8 memory events, one for each 128MiB memblock.

I rewrite the log as below with my understanding. Hope it's simpler to help people get what's going on here. Please consider to take if it's OK to you or adjust based on this. The code looks good to me.

Made some tuning:

crash: add generic infrastructure for crash hotplug support

Upon CPU and memory changes, a generic crash_hotplug_handler() is added to dispatch the hot plug/unplug event to the architecture specific arch_crash_hotplug_handler(). During the process, kexec_mutex need be held.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

To support cpu hotplug, one callback pair are registered to capture KEXEC_CRASH_HP_ADD_CPU and KEXEC_CRASH_HP_REMOVE_CPU events via cpuhp_setup_state_nocalls().

s/KEXEC_CRASH_HP_ADD}REMOVE_CPU/CPUHP_AP_ONLINE_DYN/ as the KEXEC_CRASH are the names I've introduced with this patch?

Right.

Updated commit message.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

While checking it, I notice hp_action which you don't use actually. Can you reconsider that part of design, the hp_action, the a, b parameter passed to handler?

Sure I can remove. I initially put in there as this was generic infrastructure and not sure if it would benefit others. eric

Quoted 4 lines
Quoted 2 lines

Actually, I will keep the hp_action as the work by Sourabh Jain for PPC uses the hp_action. I'll drop the a and b.

Sounds great.

Turns out hp_action and a are utilized, so I just left it alone. If you'd rather I remove b, I can do so.

Quoted 6 lines
Quoted 3 lines

Also, shall I post v6, or are you still looking at patches 7 and 8?

Will check today, thanks for the effort.

#18 lore raw

PATCH v5 5/8 kexec: exclude elfcorehdr from the segment digest

In reply to Eric DeVolder

When a crash kernel is loaded via the kexec_file_load syscall, the kernel places the various segments (ie crash kernel, crash initrd, boot_params, elfcorehdr, purgatory, etc) in memory. For those architectures that utilize purgatory, a hash digest of the segments is calculated for integrity checking. This digest is embedded into the purgatory image prior to placing purgatory in memory.

Since hotplug events cause changes to the elfcorehdr, purgatory integrity checking fails (at crash time, and no kdump created). As a result, this change explicitly excludes the elfcorehdr segment from the list of segments used to create the digest. By doing so, this permits changes to the elfcorehdr in response to hotplug events, without having to also reload purgatory due to the change to the digest.

---
 kernel/kexec_file.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/kexec_file.c +6 -0
index 801d0d0a5012..93bb0c0ce66c 100644
@@ -765,6 +765,12 @@ static int kexec_calculate_store_digests(struct kimage *image)
 	for (j = i = 0; i < image->nr_segments; i++) {
 		struct kexec_segment *ksegment;
 
+#ifdef CONFIG_CRASH_HOTPLUG
+		/* This segment excluded to allow future changes via hotplug */
+		if (image->elf_index_valid && (j == image->elf_index))
+			continue;
+#endif
+
 		ksegment = &image->segment[i];
 		/*
 		 * Skip purgatory as it will be modified once we put digest
Signature · 1 line
2.27.0
#19 lore raw

PATCH v5 6/8 kexec: exclude hot remove cpu from elfcorehdr notes

In reply to Eric DeVolder

Due to use of CPUHP_AP_ONLINE_DYN, upon CPU unplug, the CPU is still in the for_each_present_cpu() list when within the crash_hotplug_handler(). Thus the CPU must be explicitly excluded when building the new list of CPUs.

This change identifies in crash_hotplug_handler() the CPU to be excluded, and the check for excluding the CPU in crash_prepare_elf64_headers().

---
 kernel/crash_core.c | 10 ++++++++++
 kernel/kexec_file.c |  5 +++++
 2 files changed, 15 insertions(+)
kernel/crash_core.c +10 -0
index 76959d440f71..d3fd51f2195c 100644
@@ -522,6 +522,16 @@ static void crash_hotplug_handler(unsigned int hp_action,
 		/* Flag to differentiate between normal load and hotplug */
 		kexec_crash_image->hotplug_event = true;
 
+		/*
+		 * Due to use of CPUHP_AP_ONLINE_DYN, upon unplug and during
+		 * this callback, the CPU is still in the for_each_present_cpu()
+		 * list. Must explicitly look to exclude this CPU when building
+		 * new list.
+		 */
+		kexec_crash_image->offlinecpu =
+			(hp_action == KEXEC_CRASH_HP_REMOVE_CPU) ?
+				(unsigned int)a : ~0U;
+
 		/* Now invoke arch-specific update handler */
 		arch_crash_hotplug_handler(kexec_crash_image, hp_action, a, b);
 
kernel/kexec_file.c +5 -0
index 93bb0c0ce66c..339995d42169 100644
@@ -1314,6 +1314,11 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem,
 
 	/* Prepare one phdr of type PT_NOTE for each present CPU */
 	for_each_present_cpu(cpu) {
+#ifdef CONFIG_CRASH_HOTPLUG
+		/* Skip the soon-to-be offlined cpu */
+		if (image->hotplug_event && (cpu == image->offlinecpu))
+			continue;
+#endif
 		phdr->p_type = PT_NOTE;
 		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
 		phdr->p_offset = phdr->p_paddr = notes_addr;
Signature · 1 line
2.27.0
#20 lore raw

PATCH v5 7/8 x86/crash: Add x86 crash hotplug support for kexec_file_load

In reply to Eric DeVolder

For x86_64, when CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.

To update the elfcorehdr for x86_64, a new elfcorehdr must be generated from the available CPUs and memory. The new elfcorehdr is prepared into a buffer, and if no errors occur, it is installed over the top of the existing elfcorehdr.

In the patch 'crash hp: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.

To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_HOTPLUG_ELFCOREHDR_SZ configure item.

With this change, the kexec_file_load syscall (not kexec_load) is supported. When loading the crash kernel via kexec_file_load, the elfcorehdr is identified at load time in crash_load_segments().

---
 arch/x86/kernel/crash.c | 120 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
arch/x86/kernel/crash.c +120 -0
index 9db41cce8d97..5da30e2bc780 100644
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 #include <linux/memblock.h>
+#include <linux/highmem.h>
 
 #include <asm/processor.h>
 #include <asm/hardirq.h>
@@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
 	image->elf_headers = kbuf.buffer;
 	image->elf_headers_sz = kbuf.bufsz;
 
+#ifdef CONFIG_CRASH_HOTPLUG
+	/* Ensure elfcorehdr segment large enough for hotplug changes */
+	kbuf.memsz = CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ;
+	/* For marking as usable to crash kernel */
+	image->elf_headers_sz = kbuf.memsz;
+	/* Record the index of the elfcorehdr segment */
+	image->elf_index = image->nr_segments;
+	image->elf_index_valid = true;
+#else
 	kbuf.memsz = kbuf.bufsz;
+#endif
 	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
 	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
 	ret = kexec_add_buffer(&kbuf);
@@ -413,3 +424,112 @@ int crash_load_segments(struct kimage *image)
 	return ret;
 }
 #endif /* CONFIG_KEXEC_FILE */
+
+#ifdef CONFIG_CRASH_HOTPLUG
+static void *map_crash_pages(unsigned long paddr, unsigned long size)
+{
+	/*
+	 * NOTE: The addresses and sizes passed to this routine have
+	 * already been fully aligned on page boundaries. There is no
+	 * need for massaging the address or size.
+	 */
+	void *ptr = NULL;
+
+	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
+	if (size > 0) {
+		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
+
+		ptr = kmap(page);
+	}
+
+	return ptr;
+}
+
+static void unmap_crash_pages(void **ptr)
+{
+	if (ptr) {
+		if (*ptr)
+			kunmap(*ptr);
+		*ptr = NULL;
+	}
+}
+
+/**
+ * arch_crash_hotplug_handler() - Handle hotplug elfcorehdr changes
+ * @image: the active struct kimage
+ * @hp_action: the hot un/plug action being handled
+ * @a: first parameter dependent upon hp_action
+ * @b: first parameter dependent upon hp_action
+ *
+ * To accurately reflect hot un/plug changes, the elfcorehdr (which
+ * is passed to the crash kernel via the elfcorehdr= parameter)
+ * must be updated with the new list of CPUs and memories. The new
+ * elfcorehdr is prepared in a kernel buffer, and if no errors,
+ * then it is written on top of the existing/old elfcorehdr.
+ *
+ * For hotplug changes to elfcorehdr to work, two conditions are
+ * needed:
+ * First, the segment containing the elfcorehdr must be large enough
+ * to permit a growing number of resources. See
+ * CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ.
+ * Second, purgatory must explicitly exclude the elfcorehdr from the
+ * list of segments it checks (since the elfcorehdr changes and thus
+ * would require an update to purgatory itself to update the digest).
+ *
+ */
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	struct kexec_segment *ksegment;
+	unsigned char *ptr = NULL;
+	unsigned long elfsz = 0;
+	void *elfbuf = NULL;
+	unsigned long mem, memsz;
+
+	/* Must have valid elfcorehdr index */
+	if (!image->elf_index_valid) {
+		pr_err("crash hp: unable to locate elfcorehdr segment");
+		goto out;
+	}
+
+	ksegment = &image->segment[image->elf_index];
+	mem = ksegment->mem;
+	memsz = ksegment->memsz;
+
+	/*
+	 * Create the new elfcorehdr reflecting the changes to CPU and/or
+	 * memory resources. The elfcorehdr segment memsz must be
+	 * sufficiently large to accommodate increases due to hotplug
+	 * activity. See CRASH_HOTPLUG_ELFCOREHDR_SZ.
+	 */
+	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
+		pr_err("crash hp: unable to prepare elfcore headers");
+		goto out;
+	}
+	if (elfsz > memsz) {
+		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz %lu",
+			elfsz, memsz);
+		goto out;
+	}
+
+	/*
+	 * At this point, we are all but assured of success.
+	 * Copy new elfcorehdr into destination.
+	 */
+	ptr = map_crash_pages(mem, memsz);
+	if (ptr) {
+		/* Temporarily invalidate the crash image while it is replaced */
+		xchg(&kexec_crash_image, NULL);
+		/* Write the new elfcorehdr into memory */
+		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
+		/* The crash image is now valid once again */
+		xchg(&kexec_crash_image, image);
+	}
+	unmap_crash_pages((void **)&ptr);
+	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
+
+out:
+	if (elfbuf)
+		vfree(elfbuf);
+}
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#21 lore raw

Re: [PATCH v5 7/8] x86/crash: Add x86 crash hotplug support for kexec_file_load

Quoted 15 lines · Eric DeVolder

For x86_64, when CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.

To update the elfcorehdr for x86_64, a new elfcorehdr must be generated from the available CPUs and memory. The new elfcorehdr is prepared into a buffer, and if no errors occur, it is installed over the top of the existing elfcorehdr.

In the patch 'crash hp: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.

Quoted 7 lines

To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_HOTPLUG_ELFCOREHDR_SZ configure item.

With this change, the kexec_file_load syscall (not kexec_load) is supported. When loading the crash kernel via kexec_file_load, the elfcorehdr is identified at load time in crash_load_segments().

I tune the log a little, please check:

[PATCH v5 7/8] x86/crash: Add x86 crash hotplug support for kexec_file_load

For x86_64, when CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.

To update the elfcorehdr for x86_64, a new elfcorehdr need be generated from the available CPUs and memory. The new elfcorehdr is prepared into a buffer, then installed over the top of the existing elfcorehdr.

In the patch 'kexec: exclude elfcorehdr from the segment digest', the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.

To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_HOTPLUG_ELFCOREHDR_SZ configure item.

With this change, crash hotplug for kexec_file_load syscall is supported. When loading kdump kernel via kexec_file_load, the elfcorehdr is identified at load time in crash_load_segments().

Quoted 73 lines
---
 arch/x86/kernel/crash.c | 120 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
arch/x86/kernel/crash.c +42 -0
index 9db41cce8d97..5da30e2bc780 100644
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 #include <linux/memblock.h>
+#include <linux/highmem.h>
 
 #include <asm/processor.h>
 #include <asm/hardirq.h>
@@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
 	image->elf_headers = kbuf.buffer;
 	image->elf_headers_sz = kbuf.bufsz;
 
+#ifdef CONFIG_CRASH_HOTPLUG
+	/* Ensure elfcorehdr segment large enough for hotplug changes */
+	kbuf.memsz = CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ;
+	/* For marking as usable to crash kernel */
+	image->elf_headers_sz = kbuf.memsz;
+	/* Record the index of the elfcorehdr segment */
+	image->elf_index = image->nr_segments;
+	image->elf_index_valid = true;
+#else
 	kbuf.memsz = kbuf.bufsz;
+#endif
 	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
 	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
 	ret = kexec_add_buffer(&kbuf);
@@ -413,3 +424,112 @@ int crash_load_segments(struct kimage *image)
 	return ret;
 }
 #endif /* CONFIG_KEXEC_FILE */
+
+#ifdef CONFIG_CRASH_HOTPLUG
+static void *map_crash_pages(unsigned long paddr, unsigned long size)
+{
+	/*
+	 * NOTE: The addresses and sizes passed to this routine have
+	 * already been fully aligned on page boundaries. There is no
+	 * need for massaging the address or size.
+	 */
+	void *ptr = NULL;
+
+	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
+	if (size > 0) {
+		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
+
+		ptr = kmap(page);
+	}
+
+	return ptr;
+}
+
+static void unmap_crash_pages(void **ptr)
+{
+	if (ptr) {
+		if (*ptr)
+			kunmap(*ptr);
+		*ptr = NULL;
+	}
+}
+
+/**

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 467

Quoted 10 lines
arch/x86/kernel/crash.c +10 -0
+ * arch_crash_hotplug_handler() - Handle hotplug elfcorehdr changes
+ * @image: the active struct kimage
+ * @hp_action: the hot un/plug action being handled
+ * @a: first parameter dependent upon hp_action
+ * @b: first parameter dependent upon hp_action
+ *
+ * To accurately reflect hot un/plug changes, the elfcorehdr (which
+ * is passed to the crash kernel via the elfcorehdr= parameter)
+ * must be updated with the new list of CPUs and memories. The new
+ * elfcorehdr is prepared in a kernel buffer, and if no errors,

~~~~~~~~~~~~~~~~~~
can be removed.

Quoted 12 lines
arch/x86/kernel/crash.c +12 -0
+ * then it is written on top of the existing/old elfcorehdr.
+ *
+ * For hotplug changes to elfcorehdr to work, two conditions are
+ * needed:
+ * First, the segment containing the elfcorehdr must be large enough
+ * to permit a growing number of resources. See
+ * CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ.
+ * Second, purgatory must explicitly exclude the elfcorehdr from the
+ * list of segments it checks (since the elfcorehdr changes and thus
+ * would require an update to purgatory itself to update the digest).
+ *
+ */

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 489

Quoted 10 lines
arch/x86/kernel/crash.c +10 -0
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
+	struct kexec_segment *ksegment;
+	unsigned char *ptr = NULL;
+	unsigned long elfsz = 0;
+	void *elfbuf = NULL;
+	unsigned long mem, memsz;
+
+	/* Must have valid elfcorehdr index */

Redundant code comment can be removed.

Quoted 5 lines
arch/x86/kernel/crash.c +5 -0
+	if (!image->elf_index_valid) {
+		pr_err("crash hp: unable to locate elfcorehdr segment");
+		goto out;
+	}
+

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 504

Quoted 10 lines
arch/x86/kernel/crash.c +10 -0
+	ksegment = &image->segment[image->elf_index];
+	mem = ksegment->mem;
+	memsz = ksegment->memsz;
+
+	/*
+	 * Create the new elfcorehdr reflecting the changes to CPU and/or
+	 * memory resources. The elfcorehdr segment memsz must be
+	 * sufficiently large to accommodate increases due to hotplug
+	 * activity. See CRASH_HOTPLUG_ELFCOREHDR_SZ.
+	 */

This paragraph could be duplicated with the part in kernel-doc. Considering to drop one?

Quoted 9 lines
arch/x86/kernel/crash.c +9 -0
+	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
+		pr_err("crash hp: unable to prepare elfcore headers");
+		goto out;
+	}
+	if (elfsz > memsz) {
+		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz %lu",
+			elfsz, memsz);
+		goto out;
+	}

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 523

Quoted 10 lines
arch/x86/kernel/crash.c +10 -0
+
+	/*
+	 * At this point, we are all but assured of success.
+	 * Copy new elfcorehdr into destination.
+	 */
+	ptr = map_crash_pages(mem, memsz);
+	if (ptr) {
+		/* Temporarily invalidate the crash image while it is replaced */
+		xchg(&kexec_crash_image, NULL);
+		/* Write the new elfcorehdr into memory */

No need, the code is self explanatory.

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 525

Quoted 2 lines
arch/x86/kernel/crash.c +2 -0
+		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
+		/* The crash image is now valid once again */

ditto. Sometime over commenting is not suggested.

Quoted 13 lines
arch/x86/kernel/crash.c +10 -0
+		xchg(&kexec_crash_image, image);
+	}
+	unmap_crash_pages((void **)&ptr);
+	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
+
+out:
+	if (elfbuf)
+		vfree(elfbuf);
+}
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#22 lore raw

Re: [PATCH v5 7/8] x86/crash: Add x86 crash hotplug support for kexec_file_load

Quoted 43 lines · Baoquan He
Quoted 15 lines · Eric DeVolder

For x86_64, when CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.

To update the elfcorehdr for x86_64, a new elfcorehdr must be generated from the available CPUs and memory. The new elfcorehdr is prepared into a buffer, and if no errors occur, it is installed over the top of the existing elfcorehdr.

In the patch 'crash hp: exclude elfcorehdr from the segment digest' the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.

Quoted 7 lines

To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_HOTPLUG_ELFCOREHDR_SZ configure item.

With this change, the kexec_file_load syscall (not kexec_load) is supported. When loading the crash kernel via kexec_file_load, the elfcorehdr is identified at load time in crash_load_segments().

I tune the log a little, please check:

[PATCH v5 7/8] x86/crash: Add x86 crash hotplug support for kexec_file_load

For x86_64, when CPU or memory is hot un/plugged, the crash elfcorehdr, which describes the CPUs and memory in the system, must also be updated.

To update the elfcorehdr for x86_64, a new elfcorehdr need be generated from the available CPUs and memory. The new elfcorehdr is prepared into a buffer, then installed over the top of the existing elfcorehdr.

In the patch 'kexec: exclude elfcorehdr from the segment digest', the need to update purgatory due to the change in elfcorehdr was eliminated. As a result, no changes to purgatory or boot_params (as the elfcorehdr= kernel command line parameter pointer remains unchanged and correct) are needed, just elfcorehdr.

Quoted 8 lines

To accommodate a growing number of resources via hotplug, the elfcorehdr segment must be sufficiently large enough to accommodate changes, see the CRASH_HOTPLUG_ELFCOREHDR_SZ configure item.

With this change, crash hotplug for kexec_file_load syscall is supported. When loading kdump kernel via kexec_file_load, the elfcorehdr is identified at load time in crash_load_segments().

Updated the commit message.

Quoted 77 lines
Quoted 77 lines
---
  arch/x86/kernel/crash.c | 120 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 120 insertions(+)
arch/x86/kernel/crash.c +46 -0
index 9db41cce8d97..5da30e2bc780 100644
@@ -25,6 +25,7 @@
  #include <linux/slab.h>
  #include <linux/vmalloc.h>
  #include <linux/memblock.h>
+#include <linux/highmem.h>
 
  #include <asm/processor.h>
  #include <asm/hardirq.h>
@@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
  	image->elf_headers = kbuf.buffer;
  	image->elf_headers_sz = kbuf.bufsz;
 
+#ifdef CONFIG_CRASH_HOTPLUG
+	/* Ensure elfcorehdr segment large enough for hotplug changes */
+	kbuf.memsz = CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ;
+	/* For marking as usable to crash kernel */
+	image->elf_headers_sz = kbuf.memsz;
+	/* Record the index of the elfcorehdr segment */
+	image->elf_index = image->nr_segments;
+	image->elf_index_valid = true;
+#else
  	kbuf.memsz = kbuf.bufsz;
+#endif
  	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
  	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
  	ret = kexec_add_buffer(&kbuf);
@@ -413,3 +424,112 @@ int crash_load_segments(struct kimage *image)
  	return ret;
  }
  #endif /* CONFIG_KEXEC_FILE */
+
+#ifdef CONFIG_CRASH_HOTPLUG
+static void *map_crash_pages(unsigned long paddr, unsigned long size)
+{
+	/*
+	 * NOTE: The addresses and sizes passed to this routine have
+	 * already been fully aligned on page boundaries. There is no
+	 * need for massaging the address or size.
+	 */
+	void *ptr = NULL;
+
+	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
+	if (size > 0) {
+		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
+
+		ptr = kmap(page);
+	}
+
+	return ptr;
+}
+
+static void unmap_crash_pages(void **ptr)
+{
+	if (ptr) {
+		if (*ptr)
+			kunmap(*ptr);
+		*ptr = NULL;
+	}
+}
+
+/**
+ * arch_crash_hotplug_handler() - Handle hotplug elfcorehdr changes
+ * @image: the active struct kimage
+ * @hp_action: the hot un/plug action being handled
+ * @a: first parameter dependent upon hp_action
Quoted 8 lines

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 467

Quoted 6 lines
arch/x86/kernel/crash.c +6 -0
+ * @b: first parameter dependent upon hp_action
+ *
+ * To accurately reflect hot un/plug changes, the elfcorehdr (which
+ * is passed to the crash kernel via the elfcorehdr= parameter)
+ * must be updated with the new list of CPUs and memories. The new
+ * elfcorehdr is prepared in a kernel buffer, and if no errors,

~~~~~~~~~~~~~~~~~~
can be removed.

done

Quoted 15 lines
Quoted 15 lines
arch/x86/kernel/crash.c +15 -0
+ * then it is written on top of the existing/old elfcorehdr.
+ *
+ * For hotplug changes to elfcorehdr to work, two conditions are
+ * needed:
+ * First, the segment containing the elfcorehdr must be large enough
+ * to permit a growing number of resources. See
+ * CONFIG_CRASH_HOTPLUG_ELFCOREHDR_SZ.
+ * Second, purgatory must explicitly exclude the elfcorehdr from the
+ * list of segments it checks (since the elfcorehdr changes and thus
+ * would require an update to purgatory itself to update the digest).
+ *
+ */
+void arch_crash_hotplug_handler(struct kimage *image,
+	unsigned int hp_action, unsigned long a, unsigned long b)
+{
Quoted 8 lines

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 489

Quoted 7 lines
arch/x86/kernel/crash.c +7 -0
+	struct kexec_segment *ksegment;
+	unsigned char *ptr = NULL;
+	unsigned long elfsz = 0;
+	void *elfbuf = NULL;
+	unsigned long mem, memsz;
+
+	/* Must have valid elfcorehdr index */

Redundant code comment can be removed.

done

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/crash.c +9 -0
+	if (!image->elf_index_valid) {
+		pr_err("crash hp: unable to locate elfcorehdr segment");
+		goto out;
+	}
+
+	ksegment = &image->segment[image->elf_index];
+	mem = ksegment->mem;
+	memsz = ksegment->memsz;
+
Quoted 8 lines

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 504

Quoted 6 lines
arch/x86/kernel/crash.c +6 -0
+	/*
+	 * Create the new elfcorehdr reflecting the changes to CPU and/or
+	 * memory resources. The elfcorehdr segment memsz must be
+	 * sufficiently large to accommodate increases due to hotplug
+	 * activity. See CRASH_HOTPLUG_ELFCOREHDR_SZ.
+	 */

This paragraph could be duplicated with the part in kernel-doc. Considering to drop one?

done

Quoted 12 lines
Quoted 12 lines
arch/x86/kernel/crash.c +12 -0
+	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
+		pr_err("crash hp: unable to prepare elfcore headers");
+		goto out;
+	}
+	if (elfsz > memsz) {
+		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz %lu",
+			elfsz, memsz);
+		goto out;
+	}
+
+	/*
+	 * At this point, we are all but assured of success.
Quoted 8 lines

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 523

Quoted 7 lines
arch/x86/kernel/crash.c +7 -0
+	 * Copy new elfcorehdr into destination.
+	 */
+	ptr = map_crash_pages(mem, memsz);
+	if (ptr) {
+		/* Temporarily invalidate the crash image while it is replaced */
+		xchg(&kexec_crash_image, NULL);
+		/* Write the new elfcorehdr into memory */

No need, the code is self explanatory.

done

Quoted 4 lines

arch/x86/kernel/crash.c int crash_load_segments(struct kimage *image) · line 525

Quoted 2 lines
arch/x86/kernel/crash.c +2 -0
+		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
+		/* The crash image is now valid once again */

ditto. Sometime over commenting is not suggested.

done

Quoted 15 lines
Quoted 14 lines
arch/x86/kernel/crash.c +10 -0
+		xchg(&kexec_crash_image, image);
+	}
+	unmap_crash_pages((void **)&ptr);
+	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
+
+out:
+	if (elfbuf)
+		vfree(elfbuf);
+}
+#endif /* CONFIG_CRASH_HOTPLUG */
Signature · 1 line
2.27.0
#23 lore raw

PATCH v5 8/8 x86/crash: Add x86 crash hotplug support for kexec_load

In reply to Eric DeVolder

For kexec_file_load support, the loading of the crash kernel occurs entirely within the kernel, and as such the elfcorehdr is readily identified (so that it can be modified upon hotplug events).

This change enables support for kexec_load by identifying the elfcorehdr segment in the arch_crash_hotplug_handler(), if it has not already been identified.

In general, support for kexec_load requires corresponding changes to the userspace kexec-tools utility. It is the responsibility of the userspace kexec utility to ensure that:
- the elfcorehdr segment is sufficiently large enough to accommodate hotplug changes, ala CRASH_HOTPLUG_ELFCOREHDR_SZ.
- provides a purgatory that excludes the elfcorehdr from its list of run-time segments to check.
These changes to the userspace kexec utility are available, but not yet accepted upstream.

---
 arch/x86/kernel/crash.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
arch/x86/kernel/crash.c +26 -0
index 5da30e2bc780..45cc6e3af63e 100644
@@ -486,6 +486,32 @@ void arch_crash_hotplug_handler(struct kimage *image,
 	void *elfbuf = NULL;
 	unsigned long mem, memsz;
 
+	/*
+	 * When the struct kimage is alloced, it is wiped to zero, so
+	 * the elf_index_valid defaults to false. It is set on the
+	 * kexec_file_load path, or here for kexec_load.
+	 */
+	if (!image->elf_index_valid) {
+		unsigned int n;
+
+		for (n = 0; n < image->nr_segments; n++) {
+			mem = image->segment[n].mem;
+			memsz = image->segment[n].memsz;
+			ptr = map_crash_pages(mem, memsz);
+			if (ptr) {
+				/* The segment containing elfcorehdr */
+				if ((ptr[0] == 0x7F) &&
+					(ptr[1] == 'E') &&
+					(ptr[2] == 'L') &&
+					(ptr[3] == 'F')) {
+					image->elf_index = (int)n;
+					image->elf_index_valid = true;
+				}
+			}
+			unmap_crash_pages((void **)&ptr);
+		}
+	}
+
 	/* Must have valid elfcorehdr index */
 	if (!image->elf_index_valid) {
 		pr_err("crash hp: unable to locate elfcorehdr segment");
Signature · 1 line
2.27.0
#24 lore raw

Re: [PATCH v5 8/8] x86/crash: Add x86 crash hotplug support for kexec_load

Quoted 8 lines · Eric DeVolder

For kexec_file_load support, the loading of the crash kernel occurs entirely within the kernel, and as such the elfcorehdr is readily identified (so that it can be modified upon hotplug events).

This change enables support for kexec_load by identifying the elfcorehdr segment in the arch_crash_hotplug_handler(), if it has not already been identified.

Quoted 9 lines

In general, support for kexec_load requires corresponding changes to the userspace kexec-tools utility. It is the responsibility of the userspace kexec utility to ensure that:
- the elfcorehdr segment is sufficiently large enough to accommodate hotplug changes, ala CRASH_HOTPLUG_ELFCOREHDR_SZ.
- provides a purgatory that excludes the elfcorehdr from its list of run-time segments to check.
These changes to the userspace kexec utility are available, but not yet accepted upstream.

These backgroud information should be put in cover letter, or in patch where they won't be grabbed into the final commit log.

Quoted 3 lines

---

Here, if you want to put some status or backgroud information and expect they are not got into log, this is a appropriate place.

Quoted 8 lines
 arch/x86/kernel/crash.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
arch/x86/kernel/crash.c +0 -0
index 5da30e2bc780..45cc6e3af63e 100644

arch/x86/kernel/crash.c void arch_crash_hotplug_handler(struct kimage *image, · line 492

Quoted 7 lines
arch/x86/kernel/crash.c +4 -0
@@ -486,6 +486,32 @@ void arch_crash_hotplug_handler(struct kimage *image,
 	void *elfbuf = NULL;
 	unsigned long mem, memsz;
 
+	/*
+	 * When the struct kimage is alloced, it is wiped to zero, so
+	 * the elf_index_valid defaults to false. It is set on the
+	 * kexec_file_load path, or here for kexec_load.

I would make the last line as:
kexec_file_load path, or here for kexec_load if not already identified..

Quoted 4 lines
arch/x86/kernel/crash.c +4 -0
+	 */
+	if (!image->elf_index_valid) {
+		unsigned int n;
+

arch/x86/kernel/crash.c void arch_crash_hotplug_handler(struct kimage *image, · line 506

Quoted 10 lines
arch/x86/kernel/crash.c +10 -0
+		for (n = 0; n < image->nr_segments; n++) {
+			mem = image->segment[n].mem;
+			memsz = image->segment[n].memsz;
+			ptr = map_crash_pages(mem, memsz);
+			if (ptr) {
+				/* The segment containing elfcorehdr */
+				if ((ptr[0] == 0x7F) &&
+					(ptr[1] == 'E') &&
+					(ptr[2] == 'L') &&
+					(ptr[3] == 'F')) {

Can it be like memcmp(ptr, ELFMAG, SELFMAG) as we have done in parse_crash_elf_headers()? With that, one line is taken.

Quoted 14 lines
arch/x86/kernel/crash.c +8 -0
+					image->elf_index = (int)n;
+					image->elf_index_valid = true;
+				}
+			}
+			unmap_crash_pages((void **)&ptr);
+		}
+	}
+
 	/* Must have valid elfcorehdr index */
 	if (!image->elf_index_valid) {
 		pr_err("crash hp: unable to locate elfcorehdr segment");
Signature · 1 line
2.27.0
#25 lore raw

Re: [PATCH v5 8/8] x86/crash: Add x86 crash hotplug support for kexec_load

Quoted 19 lines · Baoquan He
Quoted 8 lines · Eric DeVolder

For kexec_file_load support, the loading of the crash kernel occurs entirely within the kernel, and as such the elfcorehdr is readily identified (so that it can be modified upon hotplug events).

This change enables support for kexec_load by identifying the elfcorehdr segment in the arch_crash_hotplug_handler(), if it has not already been identified.

Quoted 9 lines

In general, support for kexec_load requires corresponding changes to the userspace kexec-tools utility. It is the responsibility of the userspace kexec utility to ensure that:
- the elfcorehdr segment is sufficiently large enough to accommodate hotplug changes, ala CRASH_HOTPLUG_ELFCOREHDR_SZ.
- provides a purgatory that excludes the elfcorehdr from its list of run-time segments to check.
These changes to the userspace kexec utility are available, but not yet accepted upstream.

Quoted 2 lines

These backgroud information should be put in cover letter, or in patch where they won't be grabbed into the final commit log.

Removed.

Quoted 19 lines
Quoted 4 lines

---

Here, if you want to put some status or backgroud information and expect they are not got into log, this is a appropriate place.

Quoted 11 lines
  arch/x86/kernel/crash.c | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
arch/x86/kernel/crash.c +0 -0
index 5da30e2bc780..45cc6e3af63e 100644
@@ -486,6 +486,32 @@ void arch_crash_hotplug_handler(struct kimage *image,
  	void *elfbuf = NULL;
  	unsigned long mem, memsz;
 
Quoted 8 lines

arch/x86/kernel/crash.c void arch_crash_hotplug_handler(struct kimage *image, · line 492

Quoted 4 lines
arch/x86/kernel/crash.c +4 -0
+	/*
+	 * When the struct kimage is alloced, it is wiped to zero, so
+	 * the elf_index_valid defaults to false. It is set on the
+	 * kexec_file_load path, or here for kexec_load.

I would make the last line as:
kexec_file_load path, or here for kexec_load if not already
identified..

done

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/crash.c +9 -0
+	 */
+	if (!image->elf_index_valid) {
+		unsigned int n;
+
+		for (n = 0; n < image->nr_segments; n++) {
+			mem = image->segment[n].mem;
+			memsz = image->segment[n].memsz;
+			ptr = map_crash_pages(mem, memsz);
+			if (ptr) {
Quoted 8 lines

arch/x86/kernel/crash.c void arch_crash_hotplug_handler(struct kimage *image, · line 506

Quoted 5 lines
arch/x86/kernel/crash.c +5 -0
+				/* The segment containing elfcorehdr */
+				if ((ptr[0] == 0x7F) &&
+					(ptr[1] == 'E') &&
+					(ptr[2] == 'L') &&
+					(ptr[3] == 'F')) {

Can it be like memcmp(ptr, ELFMAG, SELFMAG) as we have done in parse_crash_elf_headers()? With that, one line is taken.

done

Quoted 16 lines
Quoted 15 lines
arch/x86/kernel/crash.c +8 -0
+					image->elf_index = (int)n;
+					image->elf_index_valid = true;
+				}
+			}
+			unmap_crash_pages((void **)&ptr);
+		}
+	}
+
  	/* Must have valid elfcorehdr index */
  	if (!image->elf_index_valid) {
  		pr_err("crash hp: unable to locate elfcorehdr segment");
Signature · 1 line
2.27.0