KHO handover hangs when memmap reserves PMEM
Subsystems (4)
Hello,
I am reporting a regression in KHO handover.
In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards), the first kernel reserves RAM for the downloaded ISO with a memmap directive such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the first kernel reaches "kexec_core: Starting new kernel", but the second kernel produces no further output and QEMU spins at 100% CPU. On unaffected kernels, the second kernel starts and finds /dev/pmem0 with the expected "Persistent Memory (legacy)" range in /proc/iomem.
A source bisect identified 3f2ad90060f6 ("kho: adopt radix tree for preserved memory tracking") as the first bad commit; reverting it at the bisected revision restored the handover. The regression appears related to KHO metadata being accessed after the receiving kernel's memmap directive changes how the metadata's physical range is mapped.
#regzbot introduced: 3f2ad90060f6
I have an AI-generated candidate fix that changes 22 files. I have not attached or published it: although the AI did extensive verification on both x64 and arm64, I cannot review or explain the complete change, so I am not submitting it as a fix. If one of you thinks an unreviewed candidate would be useful for further analysis, please let me know.
Pratyush Yadav
maintainerHi,
Quoted 11 lines · Chris Bainbridge
Hello,
I am reporting a regression in KHO handover.
In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards), the first kernel reserves RAM for the downloaded ISO with a memmap directive such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the first kernel reaches "
kexec_core: Starting new kernel", but the second kernel produces no further output and QEMU spins at 100% CPU. On unaffected kernels, the second kernel starts and finds/dev/pmem0with the expected "Persistent Memory (legacy)" range in/proc/iomem.
Just to confirm, the first kernel boots without memmap= right?
If so, I have seen this before. Michal (+Cc) found this originally in our downstream test suite.
I think the main problem is that the radix tree pages can be anywhere in memory. So on the first kernel, they can land in an area that the second kernel's memmap= reserves. So kho_memory_init() might end up using memory that memmap= reserved. I'm not sure why exactly the next kernel doesn't produce any output, but IIRC memmap= memory is not mapped to the direct map, so accessing those pages likely causes a page fault.
It is not easy to fix. We would need to make memmap= aware of KHO and skip the radix tree pages, but that is pretty complicated to do for early boot code. And honestly, I don't think it is worth it either.
So I'd suggest you turn KHO off for the kexec that adds memmap= if you can.
Quoted 14 lines
A source bisect identified
3f2ad90060f6("kho: adopt radix tree for preserved memory tracking") as the first bad commit; reverting it at the bisected revision restored the handover. The regression appears related to KHO metadata being accessed after the receiving kernel's memmap directive changes how the metadata's physical range is mapped.#regzbot introduced:
3f2ad90060f6I have an AI-generated candidate fix that changes 22 files. I have not attached or published it: although the AI did extensive verification on both x64 and arm64, I cannot review or explain the complete change, so I am not submitting it as a fix. If one of you thinks an unreviewed candidate would be useful for further analysis, please let me know.
Signature · 2 lines
Regards, Pratyush Yadav
Quoted 2 lines · Pratyush Yadav
Just to confirm, the first kernel boots without memmap= right?
Yes.
Quoted 6 lines
I think the main problem is that the radix tree pages can be anywhere in memory. So on the first kernel, they can land in an area that the second kernel's memmap= reserves. So
kho_memory_init()might end up using memory that memmap= reserved. I'm not sure why exactly the next kernel doesn't produce any output, but IIRC memmap= memory is not mapped to the direct map, so accessing those pages likely causes a page fault.
That's correct, kho_memory_init() page faults. It happens before the serial console is available, so there is no output.
Pratyush Yadav
maintainerQuoted 13 lines · Chris Bainbridge
Quoted 2 lines · Pratyush Yadav
Just to confirm, the first kernel boots without memmap= right?
Yes.
Quoted 6 lines
I think the main problem is that the radix tree pages can be anywhere in memory. So on the first kernel, they can land in an area that the second kernel's memmap= reserves. So
kho_memory_init()might end up using memory that memmap= reserved. I'm not sure why exactly the next kernel doesn't produce any output, but IIRC memmap= memory is not mapped to the direct map, so accessing those pages likely causes a page fault.
Quoted 2 lines
That's correct,
kho_memory_init()page faults. It happens before the serial console is available, so there is no output.
Yeah, then I think you need an extra kexec with KHO off to enable memmap=. Or I guess hot-unplug the memory you'd like to memmap= early on first boot, which should make sure no radix tree pages land there.
Apart from that, I don't have any good ideas to properly solve this problem.
Signature · 2 lines
Regards, Pratyush Yadav
Mike Rapoport
maintainerQuoted 11 lines · Chris Bainbridge
Hello,
I am reporting a regression in KHO handover.
In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards), the first kernel reserves RAM for the downloaded ISO with a memmap directive such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the first kernel reaches "
kexec_core: Starting new kernel", but the second kernel produces no further output and QEMU spins at 100% CPU. On unaffected kernels, the second kernel starts and finds/dev/pmem0with the expected "Persistent Memory (legacy)" range in/proc/iomem.
TO make sure I understand this correctly, you enable KHO in the builds of the installer kernel and then the installer command line explicitly enables KHO.
Is there anything you actually preserve with KHO when kexec'ing the second kernel?
Quoted 14 lines
A source bisect identified
3f2ad90060f6("kho: adopt radix tree for preserved memory tracking") as the first bad commit; reverting it at the bisected revision restored the handover. The regression appears related to KHO metadata being accessed after the receiving kernel's memmap directive changes how the metadata's physical range is mapped.#regzbot introduced:
3f2ad90060f6I have an AI-generated candidate fix that changes 22 files. I have not attached or published it: although the AI did extensive verification on both x64 and arm64, I cannot review or explain the complete change, so I am not submitting it as a fix. If one of you thinks an unreviewed candidate would be useful for further analysis, please let me know.
Signature · 2 lines
Sincerely yours, Mike.
Quoted 13 lines · Mike Rapoport
Quoted 11 lines · Chris Bainbridge
Hello,
I am reporting a regression in KHO handover.
In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards), the first kernel reserves RAM for the downloaded ISO with a memmap directive such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the first kernel reaches "
kexec_core: Starting new kernel", but the second kernel produces no further output and QEMU spins at 100% CPU. On unaffected kernels, the second kernel starts and finds/dev/pmem0with the expected "Persistent Memory (legacy)" range in/proc/iomem.
Quoted 3 lines
TO make sure I understand this correctly, you enable KHO in the builds of the installer kernel and then the installer command line explicitly enables KHO.
Yes.
Quoted 2 lines
Is there anything you actually preserve with KHO when kexec'ing the second kernel?
No. kho=off is a valid workaround for the mini ISO. I only reported it as a regression here because this was working until the bisected commit was merged. If this is intended behaviour then it is not a problem.
Mike Rapoport
maintainerQuoted 18 lines · Chris Bainbridge
Quoted 13 lines · Mike Rapoport
Quoted 11 lines · Chris Bainbridge
Hello,
I am reporting a regression in KHO handover.
In the most recent Ubuntu 26.10 beta mini-ISO releases (2026-09-18 onwards), the first kernel reserves RAM for the downloaded ISO with a memmap directive such as memmap=<size>!4G, then kexecs a second kernel. On affected kernels, the first kernel reaches "
kexec_core: Starting new kernel", but the second kernel produces no further output and QEMU spins at 100% CPU. On unaffected kernels, the second kernel starts and finds/dev/pmem0with the expected "Persistent Memory (legacy)" range in/proc/iomem.Quoted 3 lines
TO make sure I understand this correctly, you enable KHO in the builds of the installer kernel and then the installer command line explicitly enables KHO.
Quoted 6 lines
Yes.
Quoted 2 lines
Is there anything you actually preserve with KHO when kexec'ing the second kernel?
No. kho=off is a valid workaround for the mini ISO.
I'd even say that setting CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=n and even CONFIG_KEXEC_HANDOVER=n is a reasonable choice for a distro kernel.
We don't have CONFIG_EXPERIMENTAL anymore, otherwise kho and luo would be gated by it.
Quoted 3 lines
I only reported it as a regression here because this was working until the bisected commit was merged. If this is intended behaviour then it is not a problem.
Thanks for the clarification.
This is a bug, but since it happens for a combination of an opt-in option (kho) and a hack (memmap=) I consider it low priority.
Signature · 2 lines
Sincerely yours, Mike.