kexec: Fix CMA and elf header leaks and rework

The retry loop drops nr_segments, leaking the CMA pages of the failed segments. Release them first, and disable CMA for the retry: it ignores buf_min and would hand out the same region forever.

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Breno Leitao <[email protected]>

Cc Pratyush Yadav <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Yeoreum Yun <[email protected]>

Cc Baoquan He <[email protected]>

Cc [email protected]

Fixes 07d24902977e ("kexec: enable CMA based contiguous allocation")

Signed-off-by Jinjie Ruan <[email protected]>

---
 arch/arm64/kernel/kexec_image.c        | 8 ++++++++
 arch/arm64/kernel/machine_kexec_file.c | 5 ++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

arch/arm64/kernel/kexec_image.c +8 -0

index 74d24c8aa2fa..6ed1991e3142 100644
@@ -111,6 +111,14 @@ static void *image_load(struct kimage *image,
 		 * We couldn't find space for the other segments; erase the
 		 * kernel segment and try the next available hole.
 		 */
+		/*
+		 * kexec_alloc_contig() ignores buf_min, so disable CMA to
+		 * let the retry make progress.
+		 */
+		if (image->segment_cma[kernel_segment_number])
+			image->no_cma = true;
+
+		kexec_free_segment_cma(image, kernel_segment_number);
 		image->nr_segments -= 1;
 		kbuf.buf_min = kernel_segment->mem + kernel_segment->memsz;
 		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;

arch/arm64/kernel/machine_kexec_file.c +4 -1

index 854d872dfd0f..e48f29167b38 100644
@@ -179,7 +179,10 @@ int load_other_segments(struct kimage *image,
 	return 0;
 
 out_err:
-	image->nr_segments = orig_segments;
+	while (image->nr_segments > orig_segments) {
+		kexec_free_segment_cma(image, image->nr_segments - 1);
+		image->nr_segments--;
+	}
 	kvfree(dtb);
 	return ret;
 }

The retry loop drops nr_segments, leaking the CMA pages of the failed segments. Release them first, and disable CMA for the retry: it ignores buf_min and would hand out the same region forever.

Cc Huacai Chen <[email protected]>

Cc WANG Xuerui <[email protected]>

Cc Youling Tang <[email protected]>

Cc Mike Rapoport (Microsoft) <[email protected]>

Cc Sourabh Jain <[email protected]>

Cc Kees Cook <[email protected]>

Cc [email protected]

Link https://sashiko.dev/#/patchset/20260729031235.2840255-1-ruanjinjie%40huawei.com

Fixes 55d990f0084c ("LoongArch: Add EFI binary support for kexec_file")

Signed-off-by Jinjie Ruan <[email protected]>

---
 arch/loongarch/kernel/kexec_efi.c          | 8 ++++++++
 arch/loongarch/kernel/machine_kexec_file.c | 6 +++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

arch/loongarch/kernel/kexec_efi.c +8 -0

index 5ee78ebb1546..d42711b895a2 100644
@@ -86,6 +86,14 @@ static void *efi_kexec_load(struct kimage *image,
 		 * We couldn't find space for the other segments; erase the
 		 * kernel segment and try the next available hole.
 		 */
+		/*
+		 * kexec_alloc_contig() ignores buf_min, so disable CMA to
+		 * let the retry make progress.
+		 */
+		if (image->segment_cma[kernel_segment_number])
+			image->no_cma = true;
+
+		kexec_free_segment_cma(image, kernel_segment_number);
 		image->nr_segments -= 1;
 		kbuf.buf_min = kernel_segment->mem + kernel_segment->memsz;
 		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;

arch/loongarch/kernel/machine_kexec_file.c +5 -1

index 5412aa9f3568..62a5be102065 100644
@@ -217,7 +217,11 @@ int load_other_segments(struct kimage *image,
 	return 0;
 
 out_err:
-	image->nr_segments = orig_segments;
+	while (image->nr_segments > orig_segments) {
+		kexec_free_segment_cma(image, image->nr_segments - 1);
+		image->nr_segments--;
+	}
+
 	kfree(modified_cmdline);
 	return ret;
 }

arm64, LoongArch and RISC-V duplicate the elf core header and initrd loading. Move it into a common helper with one arch hook for the boot parameters; a failure releases everything added for the attempt.

Cc Andrew Morton <[email protected]>

Cc Baoquan He <[email protected]>

Cc Mike Rapoport <[email protected]>

Suggested-by Mike Rapoport <[email protected]>

Signed-off-by Jinjie Ruan <[email protected]>

---
 include/linux/kexec.h | 15 ++++++++
 kernel/kexec_file.c   | 90 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 105 insertions(+)

include/linux/kexec.h +0 -0

index faba26f871a4a..de1d83ddc38bd 100644
@@ -259,6 +259,21 @@ int kexec_kernel_verify_pe_sig(const char *kernel, unsigned long kernel_len);
 extern int kexec_add_buffer(struct kexec_buf *kbuf);

include/linux/kexec.h +0 -0

 #ifndef arch_kexec_locate_mem_hole
 /**
  * arch_kexec_locate_mem_hole - Find free memory to place the segments.

kernel/kexec_file.c +7 -0

index 62d4cba278a57..8b80f1186875b 100644
@@ -1295,3 +1295,93 @@ int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name,
 	return 0;
 }
 #endif /* CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY */
+
+int __weak arch_kexec_setup_other_segments(struct kimage *image,
+					   struct kexec_load_params *params,
+					   unsigned long initrd_load_addr)
+{
+	return 0;
+}

kernel/kexec_file.c +5 -0

+
+	unsigned long orig_segments = image->nr_segments;
+	unsigned long initrd_load_addr = 0;
+	int ret = 0;
+

kernel/kexec_file.c +45 -0

+
+		image->elf_headers = kbuf.buffer;
+		image->elf_headers_sz = kbuf.bufsz;
+
+		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
+		kbuf.memsz = kbuf.bufsz;
+		kbuf.buf_align = SZ_64K; /* largest supported page size */
+		kbuf.top_down = true;
+
+		ret = kexec_add_buffer(&kbuf);
+		if (ret)
+			goto out_err;
+
+		image->elf_load_addr = kbuf.mem;
+
+		kexec_dprintk("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			      image->elf_load_addr, kbuf.bufsz, kbuf.memsz);
+	}
+#endif
+
+	if (params->initrd) {
+		kbuf.buffer = params->initrd;
+		kbuf.bufsz = params->initrd_len;
+		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
+		kbuf.memsz = params->initrd_len;
+		kbuf.buf_align = 0;
+		/* within 1GB-aligned window of up to 32GB in size */
+		if (!top_down)
+			kbuf.buf_max = round_down(params->kernel_load_addr, SZ_1G) +
+					(unsigned long)SZ_1G * 32;
+		kbuf.top_down = top_down;
+
+		ret = kexec_add_buffer(&kbuf);
+		if (ret)
+			goto out_err;
+		initrd_load_addr = kbuf.mem;
+
+		kexec_dprintk("Loaded initrd at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			      initrd_load_addr, kbuf.bufsz, kbuf.memsz);
+	}
+
+	ret = arch_kexec_setup_other_segments(image, params,
+					      initrd_load_addr);
+	if (ret)
+		goto out_err;

Hi Jinjie,

Quoted 4 lines

arm64, LoongArch and RISC-V duplicate the elf core header and initrd loading. Move it into a common helper with one arch hook for the boot parameters; a failure releases everything added for the attempt.

I like the idea of pulling out the common code, but I'm not convinced

that the split you suggest is right.

Quoted 19 lines

Cc Andrew Morton <[email protected]>

Cc Baoquan He <[email protected]>

Cc Mike Rapoport <[email protected]>

Suggested-by Mike Rapoport <[email protected]>

Signed-off-by Jinjie Ruan <[email protected]>

---
 include/linux/kexec.h | 15 ++++++++
 kernel/kexec_file.c   | 90 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 105 insertions(+)

include/linux/kexec.h +0 -0

index faba26f871a4a..de1d83ddc38bd 100644
@@ -259,6 +259,21 @@ int kexec_kernel_verify_pe_sig(const char *kernel, unsigned long kernel_len);
 extern int kexec_add_buffer(struct kexec_buf *kbuf);
 int kexec_locate_mem_hole(struct kexec_buf *kbuf);
 

include/linux/kexec.h int kexec_kernel_verify_pe_sig(const char *kernel, unsigned long kernel_len); · line 269

Quoted 8 lines

include/linux/kexec.h +8 -0

+struct kexec_load_params {
+	unsigned long kernel_load_addr;
+	unsigned long kernel_size;
+	char *initrd;
+	unsigned long initrd_len;
+	char *cmdline;
+	unsigned long cmdline_len;
+};

initrd and cmdline are already in image->initrd_buf, initrd_buf_len, cmdline_buf and cmdline_buf_len, that's where fops->load() gets them from.

And the outputs are scattered: elf_load_addr goes to the image, the initrd address is passed as a separate argument to the hook, and LoongArch infers "there is an initrd" from initrd_load_addr != 0.

include/linux/kexec.h int kexec_kernel_verify_pe_sig(const char *kernel, unsigned long kernel_len); · line 276

Quoted 10 lines

include/linux/kexec.h +7 -0

+
+int arch_kexec_setup_other_segments(struct kimage *image,
+				    struct kexec_load_params *params,
+				    unsigned long initrd_load_addr);
+int kexec_load_other_segments(struct kimage *image,
+			      struct kexec_load_params *params, bool top_down);
+

The call chain is now fops->load() in arch -> kexec_load_other_segments() in core -> arch_kexec_setup_other_segments() back in arch. That's a
midlayer: the core helper owns the sequence and the arch fills a hole in the middle of it.

The weak default is never used, all three users override it, and x86, powerpc and s390 build a helper and a hook that nothing calls.

Quoted 25 lines

include/linux/kexec.h +0 -0

 #ifndef arch_kexec_locate_mem_hole
 /**
  * arch_kexec_locate_mem_hole - Find free memory to place the segments.

kernel/kexec_file.c +13 -0

index 62d4cba278a57..8b80f1186875b 100644
@@ -1295,3 +1295,93 @@ int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name,
 	return 0;
 }
 #endif /* CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY */
+
+int __weak arch_kexec_setup_other_segments(struct kimage *image,
+					   struct kexec_load_params *params,
+					   unsigned long initrd_load_addr)
+{
+	return 0;
+}
+
+int kexec_load_other_segments(struct kimage *image,
+			      struct kexec_load_params *params, bool top_down)
+{
+	struct kexec_buf kbuf = {
+		.image = image,

kernel/kexec_file.c int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name, · line 1314

Quoted 4 lines

kernel/kexec_file.c +4 -0

+		/* Do not allocate anything below the kernel */
+		.buf_min = params->kernel_load_addr + params->kernel_size,
+		.buf_max = top_down ? PFN_PHYS(max_low_pfn) : ULONG_MAX,
+	};

top_down does not describe a direction, it selects "RISC-V policy" vs "arm64 policy": the upper limit for all buffers, the initrd direction and the 32G initrd window.

These are boot protocol constraints and they belong to the architecture.

The 32G window comes from arm64 booting.rst and the SZ_64K alignment below is "largest supported page size" on arm64 and LoongArch. With this patch

they become generic and RISC-V silently switches from ELF_CORE_HEADER_ALIGN to 64K in 15/15.

Quoted 14 lines

kernel/kexec_file.c +12 -0

+
+	unsigned long orig_segments = image->nr_segments;
+	unsigned long initrd_load_addr = 0;
+	int ret = 0;
+
+#ifdef CONFIG_CRASH_DUMP
+	if (image->type == KEXEC_TYPE_CRASH) {
+		ret = crash_prepare_headers(true, &kbuf.buffer, &kbuf.bufsz, NULL);
+		if (ret) {
+			pr_err("Preparing elf core header failed\n");
+			goto out_err;
+		}

kernel/kexec_file.c int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name, · line 1329

Quoted 3 lines

kernel/kexec_file.c +3 -0

+
+		if (unlikely(image->elf_headers))
+			vfree(image->elf_headers);

The headers don't depend on where the kernel is placed, yet they are rebuilt on every iteration of the arm64 and LoongArch retry loops. That's why the vfree() above is needed, and with the out_err path freeing them it's now dead code.

Building the headers once and only placing them in the loop removes the leak instead of fixing it.

Quoted 54 lines

kernel/kexec_file.c +52 -0

+
+		image->elf_headers = kbuf.buffer;
+		image->elf_headers_sz = kbuf.bufsz;
+
+		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
+		kbuf.memsz = kbuf.bufsz;
+		kbuf.buf_align = SZ_64K; /* largest supported page size */
+		kbuf.top_down = true;
+
+		ret = kexec_add_buffer(&kbuf);
+		if (ret)
+			goto out_err;
+
+		image->elf_load_addr = kbuf.mem;
+
+		kexec_dprintk("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			      image->elf_load_addr, kbuf.bufsz, kbuf.memsz);
+	}
+#endif
+
+	if (params->initrd) {
+		kbuf.buffer = params->initrd;
+		kbuf.bufsz = params->initrd_len;
+		kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
+		kbuf.memsz = params->initrd_len;
+		kbuf.buf_align = 0;
+		/* within 1GB-aligned window of up to 32GB in size */
+		if (!top_down)
+			kbuf.buf_max = round_down(params->kernel_load_addr, SZ_1G) +
+					(unsigned long)SZ_1G * 32;
+		kbuf.top_down = top_down;
+
+		ret = kexec_add_buffer(&kbuf);
+		if (ret)
+			goto out_err;
+		initrd_load_addr = kbuf.mem;
+
+		kexec_dprintk("Loaded initrd at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			      initrd_load_addr, kbuf.bufsz, kbuf.memsz);
+	}
+
+	ret = arch_kexec_setup_other_segments(image, params,
+					      initrd_load_addr);
+	if (ret)
+		goto out_err;
+
+	return 0;
+
+out_err:
+	while (image->nr_segments > orig_segments) {
+		kexec_free_segment_cma(image, image->nr_segments - 1);
+		image->nr_segments--;

kernel/kexec_file.c int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name, · line 1384

Quoted 3 lines

kernel/kexec_file.c +3 -0

+	}
+
+	kexec_free_elf_headers(image);

This drops segments added by the arch hook (DTB, RISC-V purgatory, arm64 dm-crypt keys) without knowing anything about the state attached to them, like image->dm_crypt_keys_addr or the purgatory info. The hook, in turn, frees its buffers but not its segments.

Nobody owns the unwind, so "a failure releases everything added for the attempt" is not quite what happens.

The conversions also bury ordering changes in "simplify" patches: on arm64 dm-crypt keys move after the initrd, and on RISC-V the initrd moves before the purgatory. And dm-crypt keys loading is not an arm64 boot parameter, x86 and powerpc use it as well.

kernel/kexec_file.c int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name, · line 1387

Quoted 3 lines

kernel/kexec_file.c +3 -0

+
+	return ret;
+}

What I had in mind was to make the core provide building blocks and let the architectures compose them, e.g. a common rollback function that frees CMAs and drops segments above nr. The retry loops in arm64 and LoongArch image loaders can use it instead of open coding kexec_free_segment_cma() and nr_segments--. There also could be a helper

for adding FDT for example and maybe some other common parts, like, say, efl headers.

Arch's load_other_segments() could use these helpers and still define their own policy for segment placement.

ARM64, LoongArch and RISC-V load the elf core header, initrd, DTB and cmdline parameters with almost identical code. This series fixes the bugs in that path. And as Mike suggested, rework the common part into a helper.

Patch 1 stops using the Image text_offset for arm64, which breaks CMA segments in both the load and release paths, and rejects images with a non-zero value.

Patches 2-7 fix the CMA page and elf core header leaks in the retry loops in arm64 and LoongArch, also fix a modified_cmdline leak in LoongArch.

Patches 8-10 pass the kbuf fields to crash_prepare_headers() to simplfy code.

Patch 11 frees the elf core header from the generic post-load cleanup.

Patches 12-15 add kexec_load_other_segments() and convert the three architectures, and arch_kexec_setup_other_segments() builds the architecture specific boot parameters (a DTB or a command line).

Tested with kexec_file_load() and kdump on arm64, RISC-V and LoongArch under QEMU, including the non-zero text_offset rejection.

This series is also available at:
https://github.com/ruanjinjie-eng/linux/tree/kexec_cleanup_v7

Changes in v7:
- Rework the kexec_file load code to avoid code duplication
as Mike suggested.
- Add Acked-by.
- Drop the two non-zero text_offset fix patches
and deprecate .text_offset in arm64 as Mike suggested.
- Drop the x86-32 fix patch.
- Drop the TOCTOU race fix patches.
- Link to v6: https://lore.kernel.org/all/[email protected]/

Previous Versions are as below:
- v5: https://lore.kernel.org/all/[email protected]/
- v4: https://lore.kernel.org/all/[email protected]/
- v3: https://lore.kernel.org/all/[email protected]/
- v2: https://lore.kernel.org/all/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/#t

Jinjie Ruan (15):
arm64: kexec_file: Stop using the Image text_offset
kexec: Extract kexec_free_segment_cma() from kimage_free_cma()
arm64: kexec_file: Fix CMA page leaks in segment placement retry loops
arm64: kexec_file: Fix elf_headers memory leak in retry loop
LoongArch: kexec_file: Fix CMA page leaks in segment placement retry
loops
LoongArch: kexec_file: Fix elf_headers memory leak in retry loop
LoongArch: kexec_file: Fix a modified_cmdline leak
arm64: kexec_file: Simplify load_other_segments()
LoongArch: kexec_file: Simplify load_other_segments()
riscv: kexec_file: Simplify load_extra_segments()
kexec: Free the elf core header in kimage_file_post_load_cleanup()
kexec: Add kexec_load_other_segments() helper
arm64: kexec_file: Use kexec_load_other_segments() to simplify code
LoongArch: kexec_file: Use kexec_load_other_segments() to simplify
code
riscv: kexec_file: Use kexec_load_other_segments() to simplify code

 arch/arm64/include/asm/kexec.h             |   5 -
 arch/arm64/kernel/kexec_image.c            |  33 +++++--
 arch/arm64/kernel/machine_kexec_file.c     | 103 ++++-----------------
 arch/loongarch/include/asm/kexec.h         |   4 -
 arch/loongarch/kernel/kexec_efi.c          |  20 +++-
 arch/loongarch/kernel/kexec_elf.c          |  11 ++-
 arch/loongarch/kernel/machine_kexec.c      |   2 +
 arch/loongarch/kernel/machine_kexec_file.c |  96 +++----------------
 arch/powerpc/kexec/file_load_64.c          |   4 -
 arch/riscv/include/asm/kexec.h             |   4 -
 arch/riscv/kernel/kexec_elf.c              |  11 ++-
 arch/riscv/kernel/kexec_image.c            |  11 ++-
 arch/riscv/kernel/machine_kexec_file.c     |  79 ++++------------
 arch/x86/kernel/machine_kexec_64.c         |   4 -
 include/linux/kexec.h                      |  17 ++++
 kernel/kexec_core.c                        |  25 ++---
 kernel/kexec_file.c                        |  99 ++++++++++++++++++++
 17 files changed, 248 insertions(+), 280 deletions(-)

ARM64, LoongArch and RISC-V load the elf core header, initrd, DTB and cmdline parameters with almost identical code. This series fixes the bugs in that path. And as Mike suggested, rework the common part into a helper.

Patch 1 stops using the Image text_offset for arm64, which breaks CMA segments in both the load and release paths, and rejects images with a non-zero value.

Patches 2-7 fix the CMA page and elf core header leaks in the retry loops in arm64 and LoongArch, also fix a modified_cmdline leak in LoongArch.

Patches 8-10 pass the kbuf fields to crash_prepare_headers() to simplfy code.

Patch 11 frees the elf core header from the generic post-load cleanup.

Patches 12-15 add kexec_load_other_segments() and convert the three architectures, and arch_kexec_setup_other_segments() builds the architecture specific boot parameters (a DTB or a command line).

Hi Jinjie,

Quoted 9 lines

ARM64, LoongArch and RISC-V load the elf core header, initrd, DTB and cmdline parameters with almost identical code. This series fixes the bugs in that path. And as Mike suggested, rework the common part into a helper.

Patch 1 stops using the Image text_offset for arm64, which breaks CMA segments in both the load and release paths, and rejects images with a non-zero value.

Quoted 3 lines

Patches 2-7 fix the CMA page and elf core header leaks in the retry loops in arm64 and LoongArch, also fix a modified_cmdline leak in LoongArch.

These mix fixes that can be easily merged via arch trees and something that depends on a change to the generic kexec code.

Besides, elf_headers leak and arm64's text_offset fixes should be applied to most if not all stable kernels, while CMA fixes are only for the relatively new kernels.

I'd suggest splitting

arm64: kexec_file: Stop using the Image text_offset
arm64: kexec_file: Fix elf_headers memory leak in retry loop

and sending as pure arm64 material, and

LoongArch: kexec_file: Fix elf_headers memory leak in retry loop
LoongArch: kexec_file: Fix a modified_cmdline leak

as pure LoongArch material.

And then working on CMA fixes after these land.

If there are real dependencies or potential conflicts, b4 prep can help you with managing them.

Quoted 4 lines

Patches 8-10 pass the kbuf fields to crash_prepare_headers() to simplfy code.

Quoted 5 lines

Patch 11 frees the elf core header from the generic post-load cleanup.

Patches 12-15 add kexec_load_other_segments() and convert the three architectures, and arch_kexec_setup_other_segments() builds the architecture specific boot parameters (a DTB or a command line).

I think the refactoring should go before the identical fixes in arch/ code.

Quoted 5 lines · Mike Rapoport

Quoted 5 lines

Patches 8-10 pass the kbuf fields to crash_prepare_headers() to simplfy code.

Patch 11 frees the elf core header from the generic post-load cleanup.

Quoted 6 lines

Quoted 3 lines

Patches 12-15 add kexec_load_other_segments() and convert the three architectures, and arch_kexec_setup_other_segments() builds the architecture specific boot parameters (a DTB or a command line).

I think the refactoring should go before the identical fixes in arch/ code.

However, this will cause trouble for the fix patch in the stable round.

Signature · 2 lines

Best regards,
Jinjie

Quoted 11 lines · Jinjie Ruan

Quoted 9 lines · Mike Rapoport

Quoted 8 lines

Patches 8-10 pass the kbuf fields to crash_prepare_headers() to simplfy code.

Patch 11 frees the elf core header from the generic post-load cleanup.

Patches 12-15 add kexec_load_other_segments() and convert the three architectures, and arch_kexec_setup_other_segments() builds the architecture specific boot parameters (a DTB or a command line).

Quoted 4 lines

Quoted 2 lines

I think the refactoring should go before the identical fixes in arch/ code.

However, this will cause trouble for the fix patch in the stable round.

That's right, but since actual fixes anyway require changes to kexec core, I think it's better to refactor first. The core than will take care of proper freeing in the most cases.

And since CMA support in kexec is quite new I think the entire set can be backported to stable then.

I'm planning to have a closer look at the refactoring Really Soon :)

Quoted 5 lines

Signature · 3 lines

-- 
Best regards,
Jinjie