KHO: kfence + KHO memory corruption fix

It is invalid for KHO metadata or preserved memory regions to be located within the KHO scratch area, as this area is overwritten when the next kernel is loaded, and used early in boot by the next kernel. This can lead to memory corruption.

Adds checks to kho_preserve_* and KHO's internal metadata allocators (xa_load_or_alloc, new_chunk) to verify that the physical address of the memory does not overlap with any defined scratch region. If an overlap is detected, the operation will fail and a WARN_ON is triggered. To avoid performance overhead in production kernels, these checks are enabled only when CONFIG_KEXEC_HANDOVER_DEBUG is selected.

Signed-off-by Pasha Tatashin <[email protected]>

kernel/kexec_handover.c +8 -6

@@ -133,26 +135,26 @@ static struct kho_out kho_out = {
 
 static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz)
 {
-	void *elm, *res;
+	void *res = xa_load(xa, index);
 
-	elm = xa_load(xa, index);
-	if (elm)
-		return elm;
+	if (res)
+		return res;
+
+	void *elm __free(kfree) = kzalloc(sz, GFP_KERNEL);
 
-	elm = kzalloc(sz, GFP_KERNEL);
 	if (!elm)
 		return ERR_PTR(-ENOMEM);
 
+	if (WARN_ON(kho_scratch_overlap(virt_to_phys(elm), sz)))
+		return ERR_PTR(-EINVAL);
+
 	res = xa_cmpxchg(xa, index, NULL, elm, GFP_KERNEL);
 	if (xa_is_err(res))
-		res = ERR_PTR(xa_err(res));

It is invalid for KHO metadata or preserved memory regions to be located within the KHO scratch area, as this area is overwritten when the next kernel is loaded, and used early in boot by the next kernel. This can lead to memory corruption.

Adds checks to kho_preserve_* and KHO's internal metadata allocators (xa_load_or_alloc, new_chunk) to verify that the physical address of the memory does not overlap with any defined scratch region. If an overlap is detected, the operation will fail and a WARN_ON is triggered. To avoid performance overhead in production kernels, these checks are enabled only when CONFIG_KEXEC_HANDOVER_DEBUG is selected.

Quoted 6 lines · Pasha Tatashin

It is invalid for KHO metadata or preserved memory regions to be located within the KHO scratch area, as this area is overwritten when the next kernel is loaded, and used early in boot by the next kernel. This can lead to memory corruption.

Quoted 6 lines

Adds checks to kho_preserve_* and KHO's internal metadata allocators (xa_load_or_alloc, new_chunk) to verify that the physical address of the memory does not overlap with any defined scratch region. If an overlap is detected, the operation will fail and a WARN_ON is triggered. To avoid performance overhead in production kernels, these checks are enabled only when CONFIG_KEXEC_HANDOVER_DEBUG is selected.

How many scratch regions are there in practice? Checking unconditionally seems like a small price to pay to avoid possible memory corruption. Especially since most KHO preservation should happen while the VM is still running (so does not have to by hyper-optimized).

kernel/kexec_handover.c static struct kho_out kho_out = { · line 143

Quoted 13 lines

static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz) {

kernel/kexec_handover.c +5 -4

-       void *elm, *res;
+       void *res = xa_load(xa, index);
 
-       elm = xa_load(xa, index);
-       if (elm)
-               return elm;
+       if (res)
+               return res;
+
+       void *elm __free(kfree) = kzalloc(sz, GFP_KERNEL);

nit: This breaks the local style of always declaring variables at the beginning of blocks.

Quoted 13 lines

Quoted 12 lines

static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz) {

kernel/kexec_handover.c +5 -4

-       void *elm, *res;
+       void *res = xa_load(xa, index);
 
-       elm = xa_load(xa, index);
-       if (elm)
-               return elm;
+       if (res)
+               return res;
+
+       void *elm __free(kfree) = kzalloc(sz, GFP_KERNEL);

Quoted 2 lines

nit: This breaks the local style of always declaring variables at the beginning of blocks.

I think this suggestion came from Mike, to me it looks alright, as it is only part of the clean-up path.

It is invalid for KHO metadata or preserved memory regions to be located within the KHO scratch area, as this area is overwritten when the next kernel is loaded, and used early in boot by the next kernel. This can lead to memory corruption.

Adds checks to kho_preserve_* and KHO's internal metadata allocators (xa_load_or_alloc, new_chunk) to verify that the physical address of the memory does not overlap with any defined scratch region. If an overlap is detected, the operation will fail and a WARN_ON is triggered. To avoid performance overhead in production kernels, these checks are enabled only when CONFIG_KEXEC_HANDOVER_DEBUG is selected.

Signed-off-by Pasha Tatashin <[email protected]>

---
 kernel/Kconfig.kexec             |  9 ++++++
 kernel/Makefile                  |  1 +
 kernel/kexec_handover.c          | 53 ++++++++++++++++++++++----------
 kernel/kexec_handover_debug.c    | 25 +++++++++++++++
 kernel/kexec_handover_internal.h | 16 ++++++++++
 5 files changed, 87 insertions(+), 17 deletions(-)
 create mode 100644 kernel/kexec_handover_debug.c
 create mode 100644 kernel/kexec_handover_internal.h

kernel/Kconfig.kexec +0 -0

index 422270d64820..c94d36b5fcd9 100644

kernel/Kconfig.kexec +6 -0

+	help
+	  This option enables extra sanity checks for the Kexec Handover
+	  subsystem. Since, KHO performance is crucial in live update
+	  scenarios and the extra code might be adding overhead it is
+	  only optionally enabled.
+
 config CRASH_DUMP
 	bool "kernel crash dumps"
 	default ARCH_DEFAULT_CRASH_DUMP

Hi Pasha,

Quoted 28 lines · Pasha Tatashin

It is invalid for KHO metadata or preserved memory regions to be located within the KHO scratch area, as this area is overwritten when the next kernel is loaded, and used early in boot by the next kernel. This can lead to memory corruption.

Adds checks to kho_preserve_* and KHO's internal metadata allocators (xa_load_or_alloc, new_chunk) to verify that the physical address of the memory does not overlap with any defined scratch region. If an overlap is detected, the operation will fail and a WARN_ON is triggered. To avoid performance overhead in production kernels, these checks are enabled only when CONFIG_KEXEC_HANDOVER_DEBUG is selected.

Signed-off-by Pasha Tatashin <[email protected]>

---
 kernel/Kconfig.kexec             |  9 ++++++
 kernel/Makefile                  |  1 +
 kernel/kexec_handover.c          | 53 ++++++++++++++++++++++----------
 kernel/kexec_handover_debug.c    | 25 +++++++++++++++
 kernel/kexec_handover_internal.h | 16 ++++++++++
 5 files changed, 87 insertions(+), 17 deletions(-)
 create mode 100644 kernel/kexec_handover_debug.c
 create mode 100644 kernel/kexec_handover_internal.h

kernel/Kconfig.kexec +0 -0

index 422270d64820..c94d36b5fcd9 100644

kernel/Kconfig.kexec config KEXEC_HANDOVER · line 114

Quoted 6 lines

kernel/Kconfig.kexec +3 -0

@@ -109,6 +109,15 @@ config KEXEC_HANDOVER
        to keep data or state alive across the kexec. For this to work,
        both source and target kernels need to have this option enabled.
 
+config KEXEC_HANDOVER_DEBUG
+     bool "Enable Kexec Handover debug checks"
+     depends on KEXEC_HANDOVER_DEBUGFS

I missed that in the earlier review, should be "depends on KEXEC_HANDOVER"

@Andrew, can you please fold this into what's now commit 0e0faeffd144 ("kho: warn and fail on metadata or preserved memory in scratch area")

kernel/Kconfig.kexec +1 -1

index c94d36b5fcd9..54e581072617 100644
@@ -111,7 +111,7 @@ config KEXEC_HANDOVER
 
 config KEXEC_HANDOVER_DEBUG
        bool "Enable Kexec Handover debug checks"
-       depends on KEXEC_HANDOVER_DEBUGFS
+       depends on KEXEC_HANDOVER
        help
          This option enables extra sanity checks for the Kexec Handover
          subsystem. Since, KHO performance is crucial in live update

kernel/Kconfig.kexec config KEXEC_HANDOVER · line 123

Quoted 10 lines

kernel/Kconfig.kexec +6 -0

+     help
+       This option enables extra sanity checks for the Kexec Handover
+       subsystem. Since, KHO performance is crucial in live update
+       scenarios and the extra code might be adding overhead it is
+       only optionally enabled.
+
 config CRASH_DUMP
      bool "kernel crash dumps"
      default ARCH_DEFAULT_CRASH_DUMP

--
Sincerely yours,
Mike.

Would you like me to resend the series with the "+ depends on
KEXEC_HANDOVER" fix from Mike, or would you apply it into your tree directly?

KHO memory preservation metadata is preserved in 512 byte chunks which requires their allocation from slab allocator. Slabs are not safe to be used with KHO because of kfence, and because partial slabs may lead leaks to the next kernel. Change the size to be PAGE_SIZE.

The kfence specifically may cause memory corruption, where it randomly provides slab objects that can be within the scratch area. The reason for that is that kfence allocates its objects prior to KHO scratch is marked as CMA region.

While this change could potentially increase metadata overhead on systems with sparsely preserved memory, this is being mitigated by ongoing work to reduce sparseness during preservation via 1G guest pages. Furthermore, this change aligns with future work on a stateless KHO, which will also use page-sized bitmaps for its radix tree metadata.

Signed-off-by Pasha Tatashin <[email protected]>

kernel/kexec_handover.c +1 -1

-static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz)
+static void *xa_load_or_alloc(struct xarray *xa, unsigned long index)
 {
        void *res = xa_load(xa, index);
 
        if (res)
                return res;

Quoted 7 lines · Pasha Tatashin

kernel/kexec_handover.c +1 -1

-static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz)
+static void *xa_load_or_alloc(struct xarray *xa, unsigned long index)
 {
        void *res = xa_load(xa, index);
 
        if (res)
                return res;

kernel/kexec_handover.c static struct kho_out kho_out = { · line 151

Quoted 10 lines

kernel/kexec_handover.c +2 -2

 
-       void *elm __free(kfree) = kzalloc(sz, GFP_KERNEL);
+       void *elm __free(kfree) = kzalloc(PAGE_SIZE, GFP_KERNEL);
 
        if (!elm)
                return ERR_PTR(-ENOMEM);
 
-       if (WARN_ON(kho_scratch_overlap(virt_to_phys(elm), sz)))
+       if (WARN_ON(kho_scratch_overlap(virt_to_phys(elm), PAGE_SIZE)))
                return ERR_PTR(-EINVAL);

Reading xa_load_or_alloc() is a bit confusing now.

It seems very generic (returns a void *) but now hard-codes a size (PAGE_SIZE). You have to look at the caller to see it is allocating for a struct kho_mem_phys_bits, and then at the definition of struct kho_mem_phys_bits to see the static_assert() that this struct is always PAGE_SIZE.

I would either keep letting the caller passing in size (if you think this code is going to be re-used) or just commit to making xa_load_or_alloc() specific to kho_mem_phys_bits. e.g. Change the return type to struct kho_mem_phys_bits * and use sizeof() instead of PAGE_SIZE.

Quoted 30 lines · David Matlack

Quoted 7 lines · Pasha Tatashin

kernel/kexec_handover.c +1 -1

-static void *xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t sz)
+static void *xa_load_or_alloc(struct xarray *xa, unsigned long index)
 {
        void *res = xa_load(xa, index);
 
        if (res)
                return res;

kernel/kexec_handover.c static struct kho_out kho_out = { · line 151

Quoted 10 lines

kernel/kexec_handover.c +2 -2

 
-       void *elm __free(kfree) = kzalloc(sz, GFP_KERNEL);
+       void *elm __free(kfree) = kzalloc(PAGE_SIZE, GFP_KERNEL);
 
        if (!elm)
                return ERR_PTR(-ENOMEM);
 
-       if (WARN_ON(kho_scratch_overlap(virt_to_phys(elm), sz)))
+       if (WARN_ON(kho_scratch_overlap(virt_to_phys(elm), PAGE_SIZE)))
                return ERR_PTR(-EINVAL);

Reading xa_load_or_alloc() is a bit confusing now.

It seems very generic (returns a void *) but now hard-codes a size (PAGE_SIZE). You have to look at the caller to see it is allocating for a struct kho_mem_phys_bits, and then at the definition of struct kho_mem_phys_bits to see the static_assert() that this struct is always PAGE_SIZE.

Quoted 5 lines

I would either keep letting the caller passing in size (if you think this code is going to be re-used) or just commit to making xa_load_or_alloc() specific to kho_mem_phys_bits. e.g. Change the return type to struct kho_mem_phys_bits * and use sizeof() instead of PAGE_SIZE.

I see that you replace kzalloc() with get_zeroed_page() in the next patch. So the latter option is probably better, and maybe move static assert down here and use BUILD_BUG_ON()? That way readers can easily see that we are allocating for struct kho_mem_phys_bits *and* that that struct is guaranteed to be PAGE_SIZE'd.

KHO allocates metadata for its preserved memory map using the slab allocator via kzalloc(). This metadata is temporary and is used by the next kernel during early boot to find preserved memory.

A problem arises when KFENCE is enabled. kzalloc() calls can be randomly intercepted by kfence_alloc(), which services the allocation from a dedicated KFENCE memory pool. This pool is allocated early in boot via memblock.

When booting via KHO, the memblock allocator is restricted to a "scratch area", forcing the KFENCE pool to be allocated within it. This creates a conflict, as the scratch area is expected to be ephemeral and overwriteable by a subsequent kexec. If KHO metadata is placed in this KFENCE pool, it leads to memory corruption when the next kernel is loaded.

To fix this, modify KHO to allocate its metadata directly from the buddy allocator instead of slab.

Fixes fc33e4b44b27 ("kexec: enable KHO support for memory preservation")

Signed-off-by Pasha Tatashin <[email protected]>

Reviewed-by Pratyush Yadav <[email protected]>

Quoted 17 lines · Pasha Tatashin

KHO allocates metadata for its preserved memory map using the slab allocator via kzalloc(). This metadata is temporary and is used by the next kernel during early boot to find preserved memory.

A problem arises when KFENCE is enabled. kzalloc() calls can be randomly intercepted by kfence_alloc(), which services the allocation from a dedicated KFENCE memory pool. This pool is allocated early in boot via memblock.

When booting via KHO, the memblock allocator is restricted to a "scratch area", forcing the KFENCE pool to be allocated within it. This creates a conflict, as the scratch area is expected to be ephemeral and overwriteable by a subsequent kexec. If KHO metadata is placed in this KFENCE pool, it leads to memory corruption when the next kernel is loaded.

Quoted 6 lines

To fix this, modify KHO to allocate its metadata directly from the buddy allocator instead of slab.

Fixes fc33e4b44b27 ("kexec: enable KHO support for memory preservation")

Signed-off-by Pasha Tatashin <[email protected]>

Reviewed-by Pratyush Yadav <[email protected]>

Reviewed-by David Matlack <[email protected]>

This series fixes a memory corruption bug in KHO that occurs when KFENCE is enabled.

The root cause is that KHO metadata, allocated via kzalloc(), can be randomly serviced by kfence_alloc(). When a kernel boots via KHO, the early memblock allocator is restricted to a "scratch area". This forces the KFENCE pool to be allocated within this scratch area, creating a conflict. If KHO metadata is subsequently placed in this pool, it gets corrupted during the next kexec operation.

Patch 1/3 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG) that adds checks to detect and fail any operation that attempts to place KHO metadata or preserved memory within the scratch area. This serves as a validation and diagnostic tool to confirm the problem without affecting production builds.

Patch 2/3 Increases bitmap to PAGE_SIZE, so buddy allocator can be used.

Patch 3/3 Provides the fix by modifying KHO to allocate its metadata directly from the buddy allocator instead of slab. This bypasses the KFENCE interception entirely.

Pasha Tatashin (3):
liveupdate: kho: warn and fail on metadata or preserved memory in
scratch area
liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE
liveupdate: kho: allocate metadata directly from the buddy allocator

 include/linux/gfp.h              |  3 ++
 kernel/Kconfig.kexec             |  9 ++++
 kernel/Makefile                  |  1 +
 kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
 kernel/kexec_handover_debug.c    | 25 +++++++++++
 kernel/kexec_handover_internal.h | 16 +++++++
 6 files changed, 100 insertions(+), 26 deletions(-)
 create mode 100644 kernel/kexec_handover_debug.c
 create mode 100644 kernel/kexec_handover_internal.h

base-commit: 6548d364a3e850326831799d7e3ea2d7bb97ba08

Signature · 1 line

2.51.0.869.ge66316f041-goog

Quoted 22 lines · Pasha Tatashin

This series fixes a memory corruption bug in KHO that occurs when KFENCE is enabled.

The root cause is that KHO metadata, allocated via kzalloc(), can be randomly serviced by kfence_alloc(). When a kernel boots via KHO, the early memblock allocator is restricted to a "scratch area". This forces the KFENCE pool to be allocated within this scratch area, creating a conflict. If KHO metadata is subsequently placed in this pool, it gets corrupted during the next kexec operation.

Patch 1/3 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG) that adds checks to detect and fail any operation that attempts to place KHO metadata or preserved memory within the scratch area. This serves as a validation and diagnostic tool to confirm the problem without affecting production builds.

Patch 2/3 Increases bitmap to PAGE_SIZE, so buddy allocator can be used.

Patch 3/3 Provides the fix by modifying KHO to allocate its metadata directly from the buddy allocator instead of slab. This bypasses the KFENCE interception entirely.

Quoted 5 lines

Pasha Tatashin (3):
liveupdate: kho: warn and fail on metadata or preserved memory in
scratch area
liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE
liveupdate: kho: allocate metadata directly from the buddy allocator

With liveupdate: dropped from the subjects

Reviewed-by Mike Rapoport (Microsoft) <[email protected]>

Quoted 11 lines

 include/linux/gfp.h              |  3 ++
 kernel/Kconfig.kexec             |  9 ++++
 kernel/Makefile                  |  1 +
 kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
 kernel/kexec_handover_debug.c    | 25 +++++++++++
 kernel/kexec_handover_internal.h | 16 +++++++
 6 files changed, 100 insertions(+), 26 deletions(-)
 create mode 100644 kernel/kexec_handover_debug.c
 create mode 100644 kernel/kexec_handover_internal.h

Quoted 4 lines

base-commit: 6548d364a3e850326831799d7e3ea2d7bb97ba08 --
2.51.0.869.ge66316f041-goog

--
Sincerely yours,
Mike.

Quoted 1 line

With liveupdate: dropped from the subjects

I noticed "liveupdate: " subject prefix left over only after sending these patches. Andrew, would you like me to resend them, or could you remove the prefix from these patches?

Quoted 5 lines · On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <[email protected]>…

Quoted 1 line

With liveupdate: dropped from the subjects

I noticed "liveupdate: " subject prefix left over only after sending these patches. Andrew, would you like me to resend them, or could you remove the prefix from these patches?

No problem.

What should we do about -stable kernels?

It doesn't seem worthwhile to backport a 3-patch series for a pretty obscure bug. Perhaps we could merge a patch which disables this

combination in Kconfig, as a 6.18-rcX hotfix with a cc:stable.

Then for 6.19-rc1 we add this series and a fourth patch which undoes that Kconfig change?

Quoted 9 lines · Andrew Morton

Quoted 5 lines · On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <[email protected]>…

Quoted 1 line

With liveupdate: dropped from the subjects

I noticed "liveupdate: " subject prefix left over only after sending these patches. Andrew, would you like me to resend them, or could you remove the prefix from these patches?

Quoted 6 lines

No problem.

What should we do about -stable kernels?

It doesn't seem worthwhile to backport a 3-patch series for a pretty obscure bug. Perhaps we could merge a patch which disables this

We are using KHO and have had obscure crashes due to this memory corruption, with stacks all over the place. I would prefer this fix to be properly backported to stable so we can also automatically consume it once we switch to the upstream KHO. I do not think disabling kfence in the Google fleet to resolve this problem would work for us, so if it is not going to be part of stable, we would have to backport it manually anyway.

Thanks,
Pasha

Quoted 4 lines

combination in Kconfig, as a 6.18-rcX hotfix with a cc:stable.

Then for 6.19-rc1 we add this series and a fourth patch which undoes that Kconfig change?

Quoted 7 lines

We are using KHO and have had obscure crashes due to this memory corruption, with stacks all over the place. I would prefer this fix to be properly backported to stable so we can also automatically consume it once we switch to the upstream KHO. I do not think disabling kfence in the Google fleet to resolve this problem would work for us, so if it is not going to be part of stable, we would have to backport it manually anyway.

The backport to stable is only relevant to 6.17 that's going to be EOL soon anyway. Do you really think it's worth the effort?

Quoted 13 lines · Andrew Morton

Quoted 5 lines · On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <[email protected]>…

Quoted 1 line

With liveupdate: dropped from the subjects

I noticed "liveupdate: " subject prefix left over only after sending these patches. Andrew, would you like me to resend them, or could you remove the prefix from these patches?

No problem.

What should we do about -stable kernels?

Quoted 2 lines

It doesn't seem worthwhile to backport a 3-patch series for a pretty obscure bug. Perhaps we could merge a patch which disables this

We are using KHO and have had obscure crashes due to this memory corruption, with stacks all over the place. I would prefer this fix to be properly backported to stable so we can also automatically consume it once we switch to the upstream KHO.