Skip to contentSkip to conversation

kexec: Enable CMA based contiguous allocation

PATCH Superseded Alexander Graf 3 participants 6 messages May 12, 2025 – May 24, 2025

Subsystems (3)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Alexander Graf

author reviewer
#0 lore raw

PATCH 1/1 kexec: Enable CMA based contiguous allocation

When booting a new kernel with kexec_file, the kernel picks a target location that the kernel should live at, then allocates random pages, checks whether any of those patches magically happens to coincide with a target address range and if so, uses them for that range.

For every page allocated this way, it then creates a page list that the relocation code - code that executes while all CPUs are off and we are just about to jump into the new kernel - copies to their final memory location. We can not put them there before, because chances are pretty good that at least some page in the target range is already in use by the currently running Linux environment.

All of this is inefficient.

Since kexec got introduced, Linux has gained the CMA framework which can perform physically contiguous memory mappings, while keeping that memory available for movable memory when it is not needed for contiguous allocations. The default CMA allocator is for DMA allocations.

This patch adds logic to the kexec file loader to attempt to place the target payload at a location allocated from CMA. If successful, it uses that memory range directly instead of creating copy instructions during the hot phase. To ensure that there is a safety net in case anything goes wrong with the CMA allocation, it also adds a flag for user space to force disable CMA allocations.

Using CMA allocations has two advantages:

1) Faster. There is no more need to copy in the hot phase.
2) More robust. Even if by accident some page is still in use for DMA, the new kernel image will be safe from that access because it resides in a memory region that is considered allocated in the old kernel and has a chance to reinitialize that component.

---
 include/linux/kexec.h      |  6 +++
 include/uapi/linux/kexec.h |  1 +
 kernel/kexec_core.c        | 82 ++++++++++++++++++++++++++++++++++++++
 kernel/kexec_file.c        | 55 +++++++++++++++++++++++--
 4 files changed, 140 insertions(+), 4 deletions(-)
include/linux/kexec.h +6 -0
index c8971861521a..421af2a303b6 100644
@@ -91,6 +91,9 @@ struct kexec_segment {
 	size_t bufsz;
 	unsigned long mem;
 	size_t memsz;
+
+	/* Pointer to contiguous CMA allocation or NULL */
+	struct page *cma;
 };
 
 #ifdef CONFIG_COMPAT
@@ -169,6 +172,7 @@ int kexec_image_post_load_cleanup_default(struct kimage *image);
  * @buf_min:	The buffer can't be placed below this address.
  * @buf_max:	The buffer can't be placed above this address.
  * @top_down:	Allocate from top of memory.
+ * @cma:	CMA page if the buffer is backed by CMA.
  */
 struct kexec_buf {
 	struct kimage *image;
@@ -180,6 +184,7 @@ struct kexec_buf {
 	unsigned long buf_min;
 	unsigned long buf_max;
 	bool top_down;
+	struct page *cma;
 };
 
 int kexec_load_purgatory(struct kimage *image, struct kexec_buf *kbuf);
@@ -331,6 +336,7 @@ struct kimage {
 	 */
 	unsigned int hotplug_support:1;
 #endif
+	unsigned int no_cma : 1;
 
 #ifdef ARCH_HAS_KIMAGE_ARCH
 	struct kimage_arch arch;
include/uapi/linux/kexec.h +1 -0
index 5ae1741ea8ea..8958ebfcff94 100644
@@ -27,6 +27,7 @@
 #define KEXEC_FILE_ON_CRASH	0x00000002
 #define KEXEC_FILE_NO_INITRAMFS	0x00000004
 #define KEXEC_FILE_DEBUG	0x00000008
+#define KEXEC_FILE_NO_CMA	0x00000010
 
 /* These values match the ELF architecture values.
  * Unless there is a good reason that should continue to be the case.
kernel/kexec_core.c +82 -0

112 lines not on this page · raw message

kernel/kexec_file.c +51 -4

85 lines not on this page · raw message

Signature · 12 lines
2.34.1




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
maintainer
#1 lore raw
Quoted 26 lines · On Mon, 12 May 2025 22:57:52 +0000 Alexander Graf <[email protected]> wrote:

When booting a new kernel with kexec_file, the kernel picks a target location that the kernel should live at, then allocates random pages, checks whether any of those patches magically happens to coincide with a target address range and if so, uses them for that range.

For every page allocated this way, it then creates a page list that the relocation code - code that executes while all CPUs are off and we are just about to jump into the new kernel - copies to their final memory location. We can not put them there before, because chances are pretty good that at least some page in the target range is already in use by the currently running Linux environment.

All of this is inefficient.

Since kexec got introduced, Linux has gained the CMA framework which can perform physically contiguous memory mappings, while keeping that memory available for movable memory when it is not needed for contiguous allocations. The default CMA allocator is for DMA allocations.

This patch adds logic to the kexec file loader to attempt to place the target payload at a location allocated from CMA. If successful, it uses that memory range directly instead of creating copy instructions during the hot phase. To ensure that there is a safety net in case anything goes wrong with the CMA allocation, it also adds a flag for user space to force disable CMA allocations.

Quoted 3 lines

Using CMA allocations has two advantages:

1) Faster. There is no more need to copy in the hot phase.

How much faster? Kinda matters as "fast" is the whole point of the patch!

Quoted 4 lines

2) More robust. Even if by accident some page is still in use for DMA, the new kernel image will be safe from that access because it resides in a memory region that is considered allocated in the old kernel and has a chance to reinitialize that component.

Is this known to be a problem in current code?

Some minor observations:

include/linux/kexec.h struct kimage { · line 339

Quoted 10 lines
include/linux/kexec.h +1 -0
 
...
@@ -331,6 +336,7 @@ struct kimage {
 	 */
 	unsigned int hotplug_support:1;
 #endif
+	unsigned int no_cma : 1;

"no_cma:1" is more conventional.

Quoted 18 lines
include/linux/kexec.h +0 -0
 #ifdef ARCH_HAS_KIMAGE_ARCH
 	struct kimage_arch arch;
include/uapi/linux/kexec.h +9 -0
index 5ae1741ea8ea..8958ebfcff94 100644
 
...
 
+static int kimage_load_cma_segment(struct kimage *image, struct kexec_segment *segment)
+{
+	unsigned long maddr;
+	size_t ubytes, mbytes;
+	int result = 0;
+	unsigned char __user *buf = NULL;
+	unsigned char *kbuf = NULL;
+	char *ptr = page_address(segment->cma);
+

include/uapi/linux/kexec.h

Quoted 10 lines
include/uapi/linux/kexec.h +10 -0
+	if (image->file_mode)
+		kbuf = segment->kbuf;
+	else
+		buf = segment->buf;
+	ubytes = segment->bufsz;
+	mbytes = segment->memsz;
+	maddr = segment->mem;
+
+	/* Initialize the buffer with zeros to allow for smaller input buffers */
+	memset(ptr, 0, mbytes);

Would it be more efficient to zero the remainder after performing the copy?

Quoted 46 lines
include/uapi/linux/kexec.h +37 -0
+	/* Then copy from source buffer to the CMA one */
+	while (mbytes) {
+		size_t uchunk, mchunk;
+
+		ptr += maddr & ~PAGE_MASK;
+		mchunk = min_t(size_t, mbytes,
+				PAGE_SIZE - (maddr & ~PAGE_MASK));
+		uchunk = min(ubytes, mchunk);
+
+		if (uchunk) {
+			/* For file based kexec, source pages are in kernel memory */
+			if (image->file_mode)
+				memcpy(ptr, kbuf, uchunk);
+			else
+				result = copy_from_user(ptr, buf, uchunk);
+			ubytes -= uchunk;
+			if (image->file_mode)
+				kbuf += uchunk;
+			else
+				buf += uchunk;
+		}
+
+		if (result) {
+			result = -EFAULT;
+			goto out;
+		}
+
+		ptr    += mchunk;
+		maddr  += mchunk;
+		mbytes -= mchunk;
+
+		cond_resched();
+	}
+out:
+	return result;
+}
+
 
...
kernel/kexec_file.c +0 -0
 
...

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 640

Quoted 6 lines
kernel/kexec_file.c +3 -0
@@ -632,6 +635,38 @@ static int kexec_walk_resources(struct kexec_buf *kbuf,
 		return walk_system_ram_res(0, ULONG_MAX, kbuf, func);
 }
 
+static int kexec_alloc_contig(struct kexec_buf *kbuf)
+{
+	u32 pages = (u32)(kbuf->memsz >> PAGE_SHIFT);

I don't think the cast is needed?

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 647

Quoted 7 lines
kernel/kexec_file.c +7 -0
+	unsigned long mem;
+	struct page *p;
+
+	if (kbuf->image->no_cma)
+		return -EPERM;
+
+	p = dma_alloc_from_contiguous(NULL, pages, get_order(kbuf->buf_align), true);

dma_alloc_from_contiguous()'s count' arg is size_t. Making pages' size_t seems best here. (And nr_pages would be a better identifier!)

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 649

Quoted 2 lines
kernel/kexec_file.c +2 -0
+	if (!p)
+		return -EADDRNOTAVAIL;

EADDRNOTAVAIL is a networking thing. People will be surprised to see kexec returning networking errors. Perhaps choose something more appropriate?

Quoted 22 lines
kernel/kexec_file.c +19 -0
+	pr_debug("allocated %d DMA pages at 0x%lx", pages, page_to_boot_pfn(p));
+
+	mem = page_to_boot_pfn(p) << PAGE_SHIFT;
+
+	if (kimage_is_destination_range(kbuf->image, mem, mem + kbuf->memsz)) {
+		/* Our region is already in use by a statically defined one. Bail out. */
+		pr_debug("CMA overlaps existing mem: 0x%lx+0x%lx\n", mem, kbuf->memsz);
+		dma_release_from_contiguous(NULL, p, pages);
+		return -EADDRNOTAVAIL;
+	}
+
+	kbuf->mem = page_to_boot_pfn(p) << PAGE_SHIFT;
+	kbuf->cma = p;
+
+	arch_kexec_post_alloc_pages(page_address(p), pages, 0);
+
+	return 0;
+}
+
 
...
#2 lore raw
Quoted 28 lines · Andrew Morton
Quoted 26 lines · On Mon, 12 May 2025 22:57:52 +0000 Alexander Graf <[email protected]> wrote:

When booting a new kernel with kexec_file, the kernel picks a target location that the kernel should live at, then allocates random pages, checks whether any of those patches magically happens to coincide with a target address range and if so, uses them for that range.

For every page allocated this way, it then creates a page list that the relocation code - code that executes while all CPUs are off and we are just about to jump into the new kernel - copies to their final memory location. We can not put them there before, because chances are pretty good that at least some page in the target range is already in use by the currently running Linux environment.

All of this is inefficient.

Since kexec got introduced, Linux has gained the CMA framework which can perform physically contiguous memory mappings, while keeping that memory available for movable memory when it is not needed for contiguous allocations. The default CMA allocator is for DMA allocations.

This patch adds logic to the kexec file loader to attempt to place the target payload at a location allocated from CMA. If successful, it uses that memory range directly instead of creating copy instructions during the hot phase. To ensure that there is a safety net in case anything goes wrong with the CMA allocation, it also adds a flag for user space to force disable CMA allocations.

Quoted 12 lines
Quoted 3 lines

Using CMA allocations has two advantages:

  1) Faster. There is no more need to copy in the hot phase.

How much faster?  Kinda matters as "fast" is the whole point of the patch!

Quoted 4 lines

  2) More robust. Even if by accident some page is still in use for DMA, the new kernel image will be safe from that access because it resides in a memory region that is considered allocated in the old kernel and has a chance to reinitialize that component.

https://lore.kernel.org/all/[email protected]/> Is this known to be a problem in current code?

Oh $DEITY yes. The Arm Generic Interrupt Controller is, to quote a dear friend, "a cautionary tale of how not to approach a hardware design".

It does a whole bunch of arbitrary DMA all over the place, and doesn't even live behind an IOMMU. And doesn't *stop* doing DMA unless you ask it *really* nicely; merely shutting down the offending high-level components isn't always enough, because they might still to write back some caches.

Here's one of the latest examples (not actually the one which has been breaking kexec for us, as far as we know, but an example of the genre):
https://lore.kernel.org/all/[email protected]/

So putting the new kernel into a physical memory region which was considered 'free' by the previous kernel, as Alex explains, is actually a very good defence-in-depth mechanism to protect against such issues.

maintainer
#3 lore raw
Quoted 19 lines · On Mon, 12 May 2025 19:02:20 -0700 David Woodhouse <[email protected]> wrote:
Quoted 9 lines
Quoted 4 lines

  2) More robust. Even if by accident some page is still in use for DMA, the new kernel image will be safe from that access because it resides in a memory region that is considered allocated in the old kernel and has a chance to reinitialize that component.

https://lore.kernel.org/all/[email protected]/>

Is this known to be a problem in current code?

Oh $DEITY yes. The Arm Generic Interrupt Controller is, to quote a dear friend, "a cautionary tale of how not to approach a hardware design".

It does a whole bunch of arbitrary DMA all over the place, and doesn't even live behind an IOMMU. And doesn't *stop* doing DMA unless you ask it *really* nicely; merely shutting down the offending high-level components isn't always enough, because they might still to write back some caches.

Quoted 7 lines

Here's one of the latest examples (not actually the one which has been breaking kexec for us, as far as we know, but an example of the genre):
https://lore.kernel.org/all/[email protected]/

So putting the new kernel into a physical memory region which was considered 'free' by the previous kernel, as Alex explains, is actually a very good defence-in-depth mechanism to protect against such issues.

Lol, it sounds like you're having fun over there.

Alexander, can you please repackage David's info to your taste and include it in the changelog? Escalating the value of the patch from "might speed it up, don't know how much" to "addresses grievous real-world issues" is helpful to the patch's case!

author reviewer
#4 lore raw
In reply to Andrew Morton

Hi Andrew,

Quoted 28 lines · Andrew Morton
Quoted 26 lines · On Mon, 12 May 2025 22:57:52 +0000 Alexander Graf <[email protected]> wrote:

When booting a new kernel with kexec_file, the kernel picks a target location that the kernel should live at, then allocates random pages, checks whether any of those patches magically happens to coincide with a target address range and if so, uses them for that range.

For every page allocated this way, it then creates a page list that the relocation code - code that executes while all CPUs are off and we are just about to jump into the new kernel - copies to their final memory location. We can not put them there before, because chances are pretty good that at least some page in the target range is already in use by the currently running Linux environment.

All of this is inefficient.

Since kexec got introduced, Linux has gained the CMA framework which can perform physically contiguous memory mappings, while keeping that memory available for movable memory when it is not needed for contiguous allocations. The default CMA allocator is for DMA allocations.

This patch adds logic to the kexec file loader to attempt to place the target payload at a location allocated from CMA. If successful, it uses that memory range directly instead of creating copy instructions during the hot phase. To ensure that there is a safety net in case anything goes wrong with the CMA allocation, it also adds a flag for user space to force disable CMA allocations.

Quoted 4 lines
Quoted 3 lines

Using CMA allocations has two advantages:

1) Faster. There is no more need to copy in the hot phase.

How much faster? Kinda matters as "fast" is the whole point of the patch!

It mostly depends on your memory bandwidth, cache size and the amount of payload you kexec into. Today, we copy kernel, initrd and dtb during the kexec hot phase. With this patch, we copy none.

At a hypothetical payload size of 100MiB and an AMLogic S905 we're looking at 50ms. On an Ice Lake system at about 4ms.

As for real life benchmarks, on an Apple M1 VM with 53MiB payload I measure a 2.2ms speedup.

That said, to me robustness is at least as important as speed here. The current scheme creates memory allocation conflicts almost by design, which increase the odds of something going wrong.

Quoted 5 lines
Quoted 4 lines

2) More robust. Even if by accident some page is still in use for DMA, the new kernel image will be safe from that access because it resides in a memory region that is considered allocated in the old kernel and has a chance to reinitialize that component.

Is this known to be a problem in current code?

Yes, see David's email. I'll clarify in the patch description.

Quoted 6 lines

Some minor observations:

Quoted 3 lines
include/linux/kexec.h +0 -0
Quoted 8 lines

include/linux/kexec.h struct kimage { · line 339

Quoted 7 lines

...

include/linux/kexec.h +1 -0
@@ -331,6 +336,7 @@ struct kimage {
        */
       unsigned int hotplug_support:1;
  #endif
+     unsigned int no_cma : 1;

"no_cma:1" is more conventional.

Fixed.

Quoted 22 lines
Quoted 22 lines
include/linux/kexec.h +0 -0
  #ifdef ARCH_HAS_KIMAGE_ARCH
       struct kimage_arch arch;
include/uapi/linux/kexec.h +12 -0
index 5ae1741ea8ea..8958ebfcff94 100644
 
...
 
+static int kimage_load_cma_segment(struct kimage *image, struct kexec_segment *segment)
+{
+     unsigned long maddr;
+     size_t ubytes, mbytes;
+     int result = 0;
+     unsigned char __user *buf = NULL;
+     unsigned char *kbuf = NULL;
+     char *ptr = page_address(segment->cma);
+
+     if (image->file_mode)
+             kbuf = segment->kbuf;
+     else
Quoted 8 lines

include/uapi/linux/kexec.h

Quoted 7 lines
include/uapi/linux/kexec.h +7 -0
+             buf = segment->buf;
+     ubytes = segment->bufsz;
+     mbytes = segment->memsz;
+     maddr = segment->mem;
+
+     /* Initialize the buffer with zeros to allow for smaller input buffers */
+     memset(ptr, 0, mbytes);

Would it be more efficient to zero the remainder after performing the copy?

Yup. Changed :).

Quoted 46 lines
Quoted 46 lines
include/uapi/linux/kexec.h +37 -0
+     /* Then copy from source buffer to the CMA one */
+     while (mbytes) {
+             size_t uchunk, mchunk;
+
+             ptr += maddr & ~PAGE_MASK;
+             mchunk = min_t(size_t, mbytes,
+                             PAGE_SIZE - (maddr & ~PAGE_MASK));
+             uchunk = min(ubytes, mchunk);
+
+             if (uchunk) {
+                     /* For file based kexec, source pages are in kernel memory */
+                     if (image->file_mode)
+                             memcpy(ptr, kbuf, uchunk);
+                     else
+                             result = copy_from_user(ptr, buf, uchunk);
+                     ubytes -= uchunk;
+                     if (image->file_mode)
+                             kbuf += uchunk;
+                     else
+                             buf += uchunk;
+             }
+
+             if (result) {
+                     result = -EFAULT;
+                     goto out;
+             }
+
+             ptr    += mchunk;
+             maddr  += mchunk;
+             mbytes -= mchunk;
+
+             cond_resched();
+     }
+out:
+     return result;
+}
+
 
...
kernel/kexec_file.c +0 -0
 
...
Quoted 8 lines

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 640

Quoted 7 lines
kernel/kexec_file.c +3 -0
@@ -632,6 +635,38 @@ static int kexec_walk_resources(struct kexec_buf *kbuf,
               return walk_system_ram_res(0, ULONG_MAX, kbuf, func);
  }
 
+static int kexec_alloc_contig(struct kexec_buf *kbuf)
+{
+     u32 pages = (u32)(kbuf->memsz >> PAGE_SHIFT);

I don't think the cast is needed?

I added it to make it more explicit we're casting (and hence potentially lose a few bits). At size_t, that is no longer necessary.

Quoted 12 lines

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 647

Quoted 8 lines
kernel/kexec_file.c +7 -0
+     unsigned long mem;
+     struct page *p;
+
+     if (kbuf->image->no_cma)
+             return -EPERM;
+
+     p = dma_alloc_from_contiguous(NULL, pages, get_order(kbuf->buf_align), true);

dma_alloc_from_contiguous()'s count' arg is size_t. Making pages' size_t seems best here. (And nr_pages would be a better identifier!)

Quoted 5 lines

kernel/kexec_file.c static int kexec_walk_resources(struct kexec_buf *kbuf, · line 649

Quoted 2 lines
kernel/kexec_file.c +2 -0
+     if (!p)
+             return -EADDRNOTAVAIL;

EADDRNOTAVAIL is a networking thing. People will be surprised to see kexec returning networking errors. Perhaps choose something more appropriate?

I was surprised too when I saw it in the surrounding code. It's the error code that locate_mem_hole() returns, so I copied it. Let me change it at least in this function to something sensible.

Alex

Amazon Web Services Development Center Germany GmbH Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

#5 lore raw
In reply to Alexander Graf
Quoted 9 lines · Alexander Graf
include/uapi/linux/kexec.h +1 -0
@@ -27,6 +27,7 @@
  #define KEXEC_FILE_ON_CRASH	0x00000002
  #define KEXEC_FILE_NO_INITRAMFS	0x00000004
  #define KEXEC_FILE_DEBUG	0x00000008
+#define KEXEC_FILE_NO_CMA	0x00000010

Gives me EINVAL when I try to use it. This helps (but should it be permitted only on architectures which implement it?):

include/linux/kexec.h +1 -1
@@ -385,7 +385,7 @@ extern int kexec_load_disabled;
 /* List of defined/legal kexec file flags */
 #define KEXEC_FILE_FLAGS       (KEXEC_FILE_UNLOAD | KEXEC_FILE_ON_CRASH | \
                                 KEXEC_FILE_NO_INITRAMFS | \
-                                KEXEC_FILE_COOPERATIVE)
+                                KEXEC_FILE_COOPERATIVE | KEXEC_FILE_NO_CMA)
 
 /* Location of a reserved region to hold the crash kernel.
  */