Skip to content

powerpc: make fadump resilient with memory add/remove events

Sourabh Jain linuxppc-dev 10 versions – Applied

Status from this message: Applied to powerpc/next.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v10 Latest3 patches 1 replyApplied
  2. v93 patches 0 repliesSuperseded
  3. v83 patches 3 repliesSuperseded
Earlier versions · 7
VersionPostedPatchesRepliesStatus
v73 patches2 repliesSuperseded
v63 patches0 repliesSuperseded
v53 patches12 repliesSuperseded
v43 patches0 repliesSuperseded
v33 patches3 repliesSuperseded
v23 patches0 repliesSuperseded
v13 patches0 repliesSuperseded

Changelog from the cover letter

==========
v10: 23 Apr 2024
- Fix a type cast build error. 1/3
- Rebase it to 6.9-rc5.

Patches in v10 Applied

#Subject and filesSizeRepliesApplied
1/3powerpc: make fadump resilient with memory add/remove eventsarch/powerpc/include/asm/fadump-internal.h arch/powerpc/kernel/fadump.c arch/powerpc/platforms/powernv/opal-fadump.c +1 more+242 -2060 replies
2/3powerpc/fadump: add hotplug_ready sysfs interfaceDocumentation/ABI/testing/sysfs-kernel-fadump arch/powerpc/kernel/fadump.c+25 -00 replies
3/3Documentation/powerpc: update fadump implementation detailsDocumentation/arch/powerpc/firmware-assisted-dump.rst+42 -490 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v10 yet.

Read the v10 thread

Cover letter [PATCH v10 0/3] powerpc: make fadump resilient with memory add/remove events

Problem:
========
Due to changes in memory resources caused by either memory hotplug or online/offline events, the elfcorehdr, which describes the cpus and memory of the crashed kernel to the kernel that collects the dump (known as second/fadump kernel), becomes outdated. Consequently, attempting dump collection with an outdated elfcorehdr can lead to failed or inaccurate dump collection.

Memory hotplug or online/offline events is referred as memory add/remove events in reset of the patch series.

Existing solution:
==================
Monitor memory add/remove events in userspace using udev rules, and re-register fadump whenever there are changes in memory resources. This leads to the creation of a new elfcorehdr with updated system memory information.

Challenges with existing solution:
==================================
1. Performing bulk memory add/remove with udev-based fadump re-registration can lead to race conditions and, more importantly, it creates a large wide window during which fadump is inactive until all memory add/remove events are settled.
2. Re-registering fadump for every memory add/remove event is inefficient.
3. Memory for elfcorehdr is allocated based on the memblock regions available during first kernel early boot and it remains fixed thereafter. However, if the elfcorehdr is later recreated with additional memblock regions, its size will increase, potentially leading to memory corruption.

Proposed solution:
==================
Address the aforementioned challenges by shifting the creation of elfcorehdr from the first kernel (also referred as the crashed kernel), where it was created and frequently recreated for every memory add/remove event, to the fadump kernel. As a result, the elfcorehdr only needs to be created once, thus eliminating the necessity to re-register fadump during memory add/remove events.

To know more about elfcorehdr creation in the fadump kernel, refer to the first patch in this series.

The second patch includes a new sysfs interface that tells userspace that fadump re-registration isn't needed for memory add/remove events. note that userspace changes do not need to be in sync with kernel changes; they can roll out independently.

Since there are significant changes in the fadump implementation, the third patch updates the fadump documentation to reflect the changes made in this patch series.

Kernel tree rebased on 6.9.0-rc5 with patch series applied:
===========================================================
https://github.com/sourabhjains/linux/tree/fadump-mem-hotplug-v10

Userspace changes:
==================
To realize this feature, one must update the kdump udev rules to prevent fadump re-registration during memory add/remove events.

On rhel apply the following changes to file
/usr/lib/udev/rules.d/98-kexec.rules

-RUN+="/bin/sh -c '/usr/bin/systemctl is-active kdump.service || exit 0; /usr/bin/systemd-run --quiet --no-block /usr/lib/udev/kdump-udev-throttler'"
+# don't re-register fadump if the value of the node
+# /sys/kernel/fadump/hotplug_ready is 1.
+
+RUN+="/bin/sh -c '/usr/bin/systemctl is-active kdump.service || exit 0; ! test -f /sys/kernel/fadump_enabled || cat /sys/kernel/fadump_enabled | grep 0 || ! test -f /sys/kernel/fadump/hotplug_ready || cat /sys/kernel/fadump/hotplug_ready | grep 0 || exit 0; /usr/bin/systemd-run --quiet --no-block /usr/lib/udev/kdump-udev-throttler'"

Changelog:
==========
v10: 23 Apr 2024
- Fix a type cast build error. 1/3
- Rebase it to 6.9-rc5.

v9: 16 Apr 2024
https://lore.kernel.org/all/[email protected]/
- Set the physical address of elfcorehdr to elfcorehdr_addr. 1/3
- Set elfcorehdr_addr to ELFCORE_ADDR_ERR before freeing the elfcorehdr. 1/3
- Mark the populate_elf_pt_load function as __init. 1/3
- Rename a function from process_fadump to fadump_process. 1/3
- Make minor changes to the commit message and a couple of comments in 1/3.
- Update date of introduction of /sys/kernel/fadump/hotplug_ready sysfs. 2/3
- Rebase it to 6.9-rc3.

v8: 16 Feb 2024
https://lore.kernel.org/all/[email protected]/
- Move elfcorehdr_addr and elfcorehdr_size struct attributes from struct fadump_crash_info_header to struct fw_dump.
- Make minor changes in commit message 1/3.
- Rebase it to 6.8-rc4.

v7: 11 Jan 2024
https://lore.kernel.org/all/[email protected]/
- Rebase it to 6.7

v6: 8 Dec 2023
https://lore.kernel.org/all/[email protected]/
- Add size fields for pt_regs and cpumask in the fadump header structure
- Don't process the dump if the size of pt_regs and cpu_mask is not same in the crashed and fadump kernel
- Include an additional check for endianness mismatch when the magic number doesn't match, to print the relevant error message
- Don't process the dump if the fadump header contains an old magic number
- Rebased it to 6.7.0-rc4

v5: 29 Oct 2023
https://lore.kernel.org/all/[email protected]/
- Fix a comment on the first patch

v4: 21 Oct 2023
https://lore.kernel.org/all/[email protected]/
- Fix a build warning about type casting

v3: 9 Oct 2023
https://lore.kernel.org/all/[email protected]/
- Assign physical address of elfcorehdr to fdh->elfcorehdr_addr
- Rename a variable, boot_mem_dest_addr -> boot_mem_dest_offset

v2: 25 Sep 2023
https://lore.kernel.org/all/[email protected]/
- Fixed a few indentation issues reported by the checkpatch script.
- Rebased it to 6.6.0-rc3

v1: 17 Sep 2023
https://lore.kernel.org/all/[email protected]/

Cc Aditya Gupta <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Hari Bathini <[email protected]>
Cc Mahesh Salgaonkar <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Naveen N Rao <[email protected]>

Sourabh Jain (3):
powerpc: make fadump resilient with memory add/remove events powerpc/fadump: add hotplug_ready sysfs interface Documentation/powerpc: update fadump implementation details

 Documentation/ABI/testing/sysfs-kernel-fadump |  11 +
 .../arch/powerpc/firmware-assisted-dump.rst   |  91 ++---
 arch/powerpc/include/asm/fadump-internal.h    |  31 +-
 arch/powerpc/kernel/fadump.c                  | 375 ++++++++++--------
 arch/powerpc/platforms/powernv/opal-fadump.c  |  22 +-
 arch/powerpc/platforms/pseries/rtas-fadump.c  |  34 +-
 6 files changed, 309 insertions(+), 255 deletions(-)
Signature · 1 line
2.44.0