powerpc: make fadump resilient with memory add/remove events
Subsystems (2)
v2 cover powerpc: make fadump resilient with memory add/remove events
Problem:
========
Due to changes in memory resources caused by either memory hotplug or online/offline events, the elfcorehdr, which describes the cpus and memory of the crashed kernel to the kernel that collects the dump (knows as second/fadump kernel), becomes outdated. Consequently, attempting dump collection with an outdated elfcorehdr can lead to failed or inaccurate dump collection.
Memory hotplug or online/offline events is referred as memory add/remove events in reset of the patch series.
Existing solution:
==================
Monitor memory add/remove events in userspace using udev rules, and re-register fadump whenever there are changes in memory resources. This leads to the creation of a new elfcorehdr with updated system memory information.
Challenges with existing solution:
==================================
1. Performing bulk memory add/remove with udev-based fadump re-registration can lead to race conditions and, more importantly, it creates a wide window during which fadump is inactive until all memory add/remove events are settled.
2. Re-registering fadump for every memory add/remove event is inefficient.
3. Memory for elfcorehdr is allocated based on the memblock regions available during early boot and remains fixed thereafter. However, if the elfcorehdr is later recreated with additional memblock regions, its size will increase, potentially leading to memory corruption.
Proposed solution:
==================
Address the aforementioned challenges by shifting the creation of elfcorehdr from the first kernel (also referred as the crashed kernel), where it was created and frequently recreated for every memory add/remove event, to the fadump kernel. As a result, the elfcorehdr only needs to be created once, thus eliminating the necessity to re-register fadump during memory add/remove events.
To know more about elfcorehdr creation in the fadump kernel, refer to the first patch in this series.
The second patch includes a new sysfs interface that tells userspace that fadump re-registration isn't needed for memory add/remove events. note that userspace changes do not need to be in sync with kernel changes; they can roll out independently.
Since there are significant changes in the fadump implementation, the third patch updates the fadump documentation to reflect the changes made in this patch series.
Kernel tree rebased on 6.6-rc3 with patch series applied:
=========================================================
https://github.com/sourabhjains/linux/tree/fadump-mem-hotplug
Userspace changes:
==================
To realize this feature, one must update the kdump udev rules to prevent fadump re-registration during memory add/remove events.
On rhel apply the following changes to file/usr/lib/udev/rules.d/98-kexec.rules
-run+="/bin/sh -c '/usr/bin/systemctl is-active kdump.service || exit 0; /usr/bin/systemd-run --quiet --no-block /usr/lib/udev/kdump-udev-throttler'"
+# don't re-register fadump if the value of the node
+# /sys/kernel/fadump/hotplug_ready is 1.
+
+run+="/bin/sh -c '/usr/bin/systemctl is-active kdump.service || exit 0; ! test -f /sys/kernel/fadump_enabled || cat /sys/kernel/fadump_enabled | grep 0 || ! test -f /sys/kernel/fadump/hotplug_ready || cat /sys/kernel/fadump/hotplug_ready | grep 0 || exit 0; /usr/bin/systemd-run --quiet --no-block /usr/lib/udev/kdump-udev-throttler'"
Changelog:
==========
v1 -> v2
- Fixed a few indentation issues reported by the checkpatch script.
- Rebased it to 6.6.0-rc3
Sourabh Jain (3):
powerpc: make fadump resilient with memory add/remove events powerpc/fadump: add hotplug_ready sysfs interface Documentation/powerpc: update fadump implementation details
Documentation/ABI/testing/sysfs-kernel-fadump | 12 + .../powerpc/firmware-assisted-dump.rst | 91 ++--- arch/powerpc/include/asm/fadump-internal.h | 24 +- arch/powerpc/kernel/fadump.c | 371 +++++++++++------- arch/powerpc/platforms/powernv/opal-fadump.c | 18 +- arch/powerpc/platforms/pseries/rtas-fadump.c | 23 +- 6 files changed, 307 insertions(+), 232 deletions(-)
Signature · 1 line
2.41.0
PATCH v2 1/3 powerpc: make fadump resilient with memory add/remove events
Due to changes in memory resources caused by either memory hotplug or online/offline events, the elfcorehdr, which describes the CPUs and memory of the crashed kernel to the kernel that collects the dump (knows as second/fadump kernel), becomes outdated. Consequently, attempting dump collection with an outdated elfcorehdr can lead to failed or inaccurate dump collection.
Memory hotplug or online/offline events is referred as memory add/remove events in reset of the commit message.
The current solution to address the aforementioned issue is as follows:
Monitor memory add/remove events in userspace using udev rules, and re-register fadump whenever there are changes in memory resources. This leads to the creation of a new elfcorehdr with updated system memory information.
There are several notable issues associated with re-registering fadump for every memory add/remove events.
1. Bulk memory add/remove events with udev-based fadump re-registration can lead to race conditions and, more importantly, it creates a wide window during which fadump is inactive until all memory add/remove events are settled.
2. Re-registering fadump for every memory add/remove event is inefficient.
3. The memory for elfcorehdr is allocated based on the memblock regions available during early boot and remains fixed thereafter. However, if elfcorehdr is later recreated with to additional memblock regions, its size will increase, potentially leading to memory corruption.
Address the aforementioned challenges by shifting the creation of elfcorehdr from the first kernel (also referred as the crashed kernel), where it was created and frequently recreated for every memory add/remove event, to the fadump kernel. As a result, the elfcorehdr only needs to be created once, thus eliminating the necessity to re-register fadump during memory add/remove events.
At present, the first kernel prepares the elfcorehdr and stores it in the fadump reserved area. Subsequently, it includes the start address of the elfcorehdr in the elfcorehdr_addr attribute of the fadump header. The fadump header holds information like elfcorehdr address, crashing CPU details, etc. The first kernel prepares the fadump header and stores it in the fadump reserved area. In the event of first kernel crash the second boots and access the fadump header prepared by first kernel and do the following in a platform-specific function [rtas|opal]_fadump_process:
1. Sanity check for fadump header
2. Update CPU notes in elfcorehdr
3. Set the global variable elfcorehdr_addr to the address of the fadump header's elfcorehdr. For vmcore module to process it later on.
Along with the above, update the setup_fadump()/fadump.c to create elfcorehdr in second/fadump kernel.
Section below outlines the information required to create the elfcorehdr and the changes made to make it available to the fadump kernel if it's not already.
To create elfcorehdr, the following crashed kernel information is
required: CPU notes, vmcoreinfo, and memory ranges.
At present, the CPU notes are already prepared in the fadump kernel, so no changes are needed in that regard. The fadump kernel has access to all crashed kernel memory regions, including boot memory regions that are relocated by firmware to fadump reserved areas, so no changes for that either. However, it is necessary to add new members to the fadump header, i.e., the 'fadump_crash_info_header' structure, in order to pass the crashed kernel's vmcoreinfo address and size to the fadump kernel.
Table 1 below illustrates kernel's ability to collect dump if either the first/crashed kernel or the second/fadump kernel does not have the changes introduced here. Consider the 'old kernel' as the kernel without this patch, and the 'new kernel' as the kernel with this patch included.
+----------+------------------------+------------------------+-------+ | scenario | first/crashed kernel | second/fadump kernel | Dump | +----------+------------------------+------------------------+-------+ | 1 | old kernel | new kernel | Yes | +----------+------------------------+------------------------+-------+ | 2 | new kernel | old kernel | No | +----------+------------------------+------------------------+-------+
Table 1
Scenario 1:
-----------
Since the magic number of fadump header is updated, the second kernel can differentiate the crashed kernel is of type 'new kernel' or 'old kernel' and act accordingly. In this scenario, since the crashed kernel is of type 'old kernel,' the fadump kernel skips elfcorehdr creation and uses the one prepared in the first kernel itself to collect the dump.
Scenario 2:
-----------
Since 'old kernel' as the fadump kernel is NOT capable of processing fadump header with updated magic number from 'new kernel' hence it gracefully fails with the below error and dump collection fails in this scenario.
[ 0.007365] rtas fadump: Crash info header is not valid.
Add a version field to the fadump_crash_info_header structure to avoid the need to change its magic number in the future. Adding a version field to the fadump header was one of the TODO items listed in Documentation/powerpc/firmware-assisted-dump.rst.
--- arch/powerpc/include/asm/fadump-internal.h | 24 +- arch/powerpc/kernel/fadump.c | 357 +++++++++++-------- arch/powerpc/platforms/powernv/opal-fadump.c | 18 +- arch/powerpc/platforms/pseries/rtas-fadump.c | 23 +- 4 files changed, 239 insertions(+), 183 deletions(-)
arch/powerpc/include/asm/fadump-internal.h +23 -1
@@ -42,7 +42,25 @@ static inline u64 fadump_str_to_u64(const char *str)#define FADUMP_CPU_UNKNOWN (~((u32)0)) -#define FADUMP_CRASH_INFO_MAGIC fadump_str_to_u64("FADMPINF") +/* +* The introduction of new fields in the fadump crash info header has +* led to a change in the magic key, from `FADMPINF` to `FADMPSIG`. +* This alteration ensures backward compatibility, enabling the kernel +* with the updated fadump crash info to handle kernel dumps from older +* kernels. +* +* To prevent the need for further changes to the magic number in the +* event of future modifications to the fadump header, a version field +* has been introduced to track the fadump crash info header version. +* +* Historically, there was no connection between the magic number and +* the fadump crash info header version. However, moving forward, the +* `FADMPINF` magic number in header will be treated as version 0, while +* the `FADMPSIG` magic number in header will include a version field to +* determine its version. +*/ +#define FADUMP_CRASH_INFO_MAGIC fadump_str_to_u64("FADMPSIG") +#define FADUMP_VERSION 1/* fadump crash info structure */ struct fadump_crash_info_header {
@@ -51,6 +69,10 @@ struct fadump_crash_info_header {u32 crashing_cpu;struct pt_regs regs;struct cpumask cpu_mask; +u32 version; +u64 elfcorehdr_size; +u64 vmcoreinfo_raddr; +u64 vmcoreinfo_size; };struct fadump_memory_range {
arch/powerpc/kernel/fadump.c +214 -143
461 lines not on this page · raw message
arch/powerpc/platforms/powernv/opal-fadump.c +1 -17
@@ -526,12 +526,7 @@ static int __init opal_fadump_process(struct fw_dump *fadump_conf)if (!opal_fdm_active || !fadump_conf->fadumphdr_addr)return rc; -/* Validate the fadump crash info header */ fdh =__va (fadump_conf->fadumphdr_addr); -if (fdh->magic_number !=FADUMP_CRASH_INFO_MAGIC ) { -pr_err ("Crash info header is not valid.\n" ); -return rc; - }#ifdef CONFIG_OPAL_CORE /*
@@ -545,18 +540,7 @@ static int __init opal_fadump_process(struct fw_dump *fadump_conf) kernel_initiated = true;#endif - rc =opal_fadump_build_cpu_notes (fadump_conf, fdh); -if (rc) -return rc; - -/* -* We are done validating dump info and elfcore header is now ready -* to be exported. set elfcorehdr_addr so that vmcore module will -* export the elfcore header through '/proc/vmcore'. -*/ - elfcorehdr_addr = fdh->elfcorehdr_addr; - -return rc; +return opal_fadump_build_cpu_notes (fadump_conf, fdh); } static void opal_fadump_region_show(struct fw_dump *fadump_conf,
arch/powerpc/platforms/pseries/rtas-fadump.c +1 -22
@@ -394,9 +394,6 @@ static int __init rtas_fadump_build_cpu_notes(struct fw_dump *fadump_conf)*/ static int __init rtas_fadump_process (struct fw_dump *fadump_conf) { -struct fadump_crash_info_header *fdh; -int rc =0 ; -if (!fdm_active || !fadump_conf->fadumphdr_addr)return -EINVAL ;
@@ -415,25 +412,7 @@ static int __init rtas_fadump_process(struct fw_dump *fadump_conf)return -EINVAL ; } -/* Validate the fadump crash info header */ - fdh =__va (fadump_conf->fadumphdr_addr); -if (fdh->magic_number !=FADUMP_CRASH_INFO_MAGIC ) { -pr_err ("Crash info header is not valid.\n" ); -return -EINVAL ; - } - - rc =rtas_fadump_build_cpu_notes (fadump_conf); -if (rc) -return rc; - -/* -* We are done validating dump info and elfcore header is now ready -* to be exported. set elfcorehdr_addr so that vmcore module will -* export the elfcore header through '/proc/vmcore'. -*/ - elfcorehdr_addr = fdh->elfcorehdr_addr; - -return 0 ; +return rtas_fadump_build_cpu_notes (fadump_conf); } static void rtas_fadump_region_show(struct fw_dump *fadump_conf,
Signature · 1 line
2.41.0
PATCH v2 2/3 powerpc/fadump: add hotplug_ready sysfs interface
In reply to Sourabh JainThe elfcorehdr describes the CPUs and memory of the crashed kernel to the kernel that captures the dump, known as the second or fadump kernel. The elfcorehdr needs to be updated if the system's memory changes due to memory hotplug or online/offline events.
Currently, memory hotplug events are monitored in userspace by udev rules, and fadump is re-registered, which recreates the elfcorehdr with the latest available memory in the system.
However, the previous patch ("powerpc: make fadump resilient with memory add/remove events") moved the creation of elfcorehdr to the second or fadump kernel. This eliminates the need to regenerate the elfcorehdr during memory hotplug or online/offline events.
Create a sysfs entry at /sys/kernel/fadump/hotplug_ready to let userspace know that fadump re-registration is not required for memory add/remove events.
--- Documentation/ABI/testing/sysfs-kernel-fadump | 12 ++++++++++++ arch/powerpc/kernel/fadump.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)
Documentation/ABI/testing/sysfs-kernel-fadump +12 -0
@@ -38,3 +38,15 @@ Contact: [email protected] Description: read only Provide information about the amount of memory reserved by FADump to save the crash dump in bytes. +What: /sys/kernel/fadump/hotplug_ready +Date: Sep 2023 +Contact: [email protected] +Description: read only + The Kdump scripts utilize udev rules to monitor memory add/remove + events, ensuring that FADUMP is automatically re-registered when + system memory changes occur. This re-registration was necessary + to update the elfcorehdr, which describes the system memory to the + second kernel. Now If this sysfs node holds a value of 1, it + indicates to userspace that FADUMP does not require re-registration + since the elfcorehdr is now generated in the second kernel. +User: kexec-tools
arch/powerpc/kernel/fadump.c +14 -0
@@ -1473,6 +1473,18 @@ static ssize_t enabled_show(struct kobject *kobj,return sprintf (buf,"%d\n" , fw_dump.fadump_enabled); } +/* +* /sys/kernel/fadump/hotplug_ready sysfs node only returns 1, +* which inidcates to usersapce that fadump re-registration is not +* required on memory hotplug events. +*/ +static ssize_t hotplug_ready_show (struct kobject *kobj, +struct kobj_attribute *attr, +char *buf) +{ +return sprintf (buf,"%d\n" ,1 ); +} +static ssize_t mem_reserved_show (struct kobject *kobj,struct kobj_attribute *attr,char *buf)
@@ -1545,11 +1557,13 @@ static struct kobj_attribute release_attr = __ATTR_WO(release_mem);static struct kobj_attribute enable_attr =__ATTR_RO (enabled);static struct kobj_attribute register_attr =__ATTR_RW (registered);static struct kobj_attribute mem_reserved_attr =__ATTR_RO (mem_reserved); +static struct kobj_attribute hotplug_ready_attr =__ATTR_RO (hotplug_ready);static struct attribute *fadump_attrs[] = { &enable_attr.attr, ®ister_attr.attr, &mem_reserved_attr.attr, + &hotplug_ready_attr.attr,NULL , };
Signature · 1 line
2.41.0
PATCH v2 3/3 Documentation/powerpc: update fadump implementation details
In reply to Sourabh JainThe patch titled ("powerpc: make fadump resilient with memory add/remove events") has made significant changes to the implementation of fadump, particularly on elfcorehdr creation and fadump crash info header structure. Therefore, updating the fadump implementation documentation to reflect those changes.
Following updates are done to firmware assisted dump documentation:
1. The elfcorehdr is no longer stored after fadump HDR in the reserved dump area. Instead, the second kernel dynamically allocates memory for the elfcorehdr within the address range from 0 to the boot memory size. Therefore, update figures 1 and 2 of Memory Reservation during the first and second kernels to reflect this change.
2. A version field has been added to the fadump header to maintain backward compatibility without changing the fadump header magic number in the future. Therefore, remove the corresponding TODO from the document.
--- .../powerpc/firmware-assisted-dump.rst | 91 +++++++++---------- 1 file changed, 42 insertions(+), 49 deletions(-)
Documentation/powerpc/firmware-assisted-dump.rst +42 -49
@@ -134,12 +134,12 @@ that are run. If there is dump data, then the memory is held. If there is no waiting dump data, then only the memory required to -hold CPU state, HPTE region, boot memory dump, FADump header and -elfcore header, is usually reserved at an offset greater than boot -memory size (see Fig. 1). This area is *not* released: this region -will be kept permanently reserved, so that it can act as a receptacle -for a copy of the boot memory content in addition to CPU state and -HPTE region, in the case a crash does occur. +hold CPU state, HPTE region, boot memory dump, and FADump header is +usually reserved at an offset greater than boot memory size (see Fig. 1). +This area is *not* released: this region will be kept permanently +reserved, so that it can act as a receptacle for a copy of the boot +memory content in addition to CPU state and HPTE region, in the case +a crash does occur. Since this reserved memory area is used only after the system crash, there is no point in blocking this significant chunk of memory from
@@ -153,22 +153,22 @@ that were present in CMA region:: o Memory Reservation during first kernel - Low memory Top of memory - 0 boot memory size |<--- Reserved dump area --->| | - | | | Permanent Reservation | | - V V | | V - +-----------+-----/ /---+---+----+-------+-----+-----+----+--+ - | | |///|////| DUMP | HDR | ELF |////| | - +-----------+-----/ /---+---+----+-------+-----+-----+----+--+ - | ^ ^ ^ ^ ^ - | | | | | | - \ CPU HPTE / | | - ------------------------------ | | - Boot memory content gets transferred | | - to reserved area by firmware at the | | - time of crash. | | - FADump Header | - (meta area) | + Low memory Top of memory + 0 boot memory size |<------ Reserved dump area ----->| | + | | | Permanent Reservation | | + V V | | V + +-----------+-----/ /---+---+----+-----------+-------+----+-----+ + | | |///|////| DUMP | HDR |////| | + +-----------+-----/ /---+---+----+-----------+-------+----+-----+ + | ^ ^ ^ ^ ^ + | | | | | | + \ CPU HPTE / | | + -------------------------------- | | + Boot memory content gets transferred | | + to reserved area by firmware at the | | + time of crash. | | + FADump Header | + (meta area) | | | Metadata: This area holds a metadata structure whose
@@ -186,13 +186,20 @@ that were present in CMA region:: 0 boot memory size | | |<------------ Crash preserved area ------------>| V V |<--- Reserved dump area --->| | - +-----------+-----/ /---+---+----+-------+-----+-----+----+--+ - | | |///|////| DUMP | HDR | ELF |////| | - +-----------+-----/ /---+---+----+-------+-----+-----+----+--+ - | | - V V - Used by second /proc/vmcore - kernel to boot + +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ + | |ELF| | |///|////| DUMP | HDR |/////| | + +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ + | | | | | | + ----- ------------------------------ --------------- + \ | | + \ | | + \ | | + \ | ---------------------------- + \ | / + \ | / + \ | / + /proc/vmcore + +---+ |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
@@ -200,6 +207,12 @@ that were present in CMA region:: does not have CPU & HPTE regions while Metadata region is not supported on pSeries currently. + +---+ + |ELF| -> elfcorehdr, it is created in second kernel after crash. + +---+ + + Note: Memory from 0 to the boot memory size is used by second kernel + Fig. 2
@@ -353,26 +366,6 @@ TODO: - Need to come up with the better approach to find out more accurate boot memory size that is required for a kernel to boot successfully when booted with restricted memory. - - The FADump implementation introduces a FADump crash info structure - in the scratch area before the ELF core header. The idea of introducing - this structure is to pass some important crash info data to the second - kernel which will help second kernel to populate ELF core header with - correct data before it gets exported through /proc/vmcore. The current - design implementation does not address a possibility of introducing - additional fields (in future) to this structure without affecting - compatibility. Need to come up with the better approach to address this. - - The possible approaches are: - - 1. Introduce version field for version tracking, bump up the version - whenever a new field is added to the structure in future. The version - field can be used to find out what fields are valid for the current - version of the structure. - 2. Reserve the area of predefined size (say PAGE_SIZE) for this - structure and have unused area as reserved (initialized to zero) - for future field additions. - - The advantage of approach 1 over 2 is we don't need to reserve extra space. Author: Mahesh Salgaonkar <[email protected]>
Signature · 1 line
2.41.0