LoongArch: kexec: avoid clobbering the QEMU FDT
Subsystems (3)
v4 cover LoongArch: kexec: avoid clobbering the QEMU FDT
This series fixes two ways the LoongArch kexec path can clobber the firmware-provided FDT that QEMU places at physical 0x100000, which made the second kernel boot silently (earlycon auto-detection reads a corrupted FDT).
Patch 1 moves the relocation trampoline off the hardcoded 0x100000 onto the kexec core's control_code_page (as arm64/riscv do), so machine_kexec_prepare() no longer overwrites the FDT.
Patch 2 removes the remaining fixed command-line address (KEXEC_CMDLINE_ADDR, 0x108000). QEMU's machine FDT declares fdt_totalsize = 1MB and thus claims the whole [0x100000, 0x200000) upper half of the reserved 2MB, so 0x108000 lives inside the FDT and only survives because the FDT's real content is small (~3.6KB) and 0x108000 falls in the trailing slack. The command line is passed to the second kernel in register a1, not at a fixed address, so it is loaded into its own kexec segment instead, removing the overlap regardless of FDT size.
This addresses the two questions raised on the v3 trampoline patch [1]:
- "how to avoid the cmdline overlapping FDT" -> patch 2.
- "how to avoid the control page being corrupted by the new kernel" -> the trampoline is executed only by the current kernel and is dead once the new kernel starts, so being overwritten afterwards is harmless; the kexec core also keeps the control page outside every relocation destination, so it cannot be corrupted during relocation. The command line is different (it is read by the new kernel), which is why only the trampoline uses the core page.
Tested with tools/testing/selftests/kho/vmtest.sh on LoongArch (QEMU, kexec_file_load): the second kernel boots with the full command line and earlycon, and KHO restore succeeds.
This series depends on [2] ("LoongArch: kexec: Fix address space mismatch in command line lookup"), whose char head[] change patch 2 builds on.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Add patch 2: load the command line in its own kexec segment, dropping the fixed KEXEC_CMDLINE_ADDR that overlaps the QEMU FDT (addresses the command line concern from the v3 review).
- Patch 1 (relocation trampoline): drop the KEXEC_CMDLINE_ADDR comment added in v3 that claimed 0x108000 does not overlap the FDT (it does; patch 2 removes the fixed address entirely). No functional change otherwise.
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
George Guo (2):
LoongArch: kexec: use core control page for relocation trampoline to
avoid QEMU FDT conflict
LoongArch: kexec: load the command line in its own segment
arch/loongarch/include/asm/kexec.h | 1 + arch/loongarch/kernel/machine_kexec.c | 38 ++++++++++------------ arch/loongarch/kernel/machine_kexec_file.c | 33 +++++++++++++++++-- 3 files changed, 49 insertions(+), 23 deletions(-)
Signature · 1 line
2.25.1
PATCH v4 1/2 LoongArch: kexec: use core control page for relocation trampoline to avoid QEMU FDT conflict
From: George Guo <[email protected]>
KEXEC_CONTROL_CODE was hardcoded to TO_CACHE(0x100000). QEMU places its machine FDT at physical 0x100000 when booting with '-kernel', so machine_kexec_prepare() overwrote the FDT with the relocation trampoline. The kexec'd kernel's fdt_setup() then read trampoline code instead of a valid FDT, earlycon auto-detection failed, and the second kernel booted silently with no console output.
The trampoline does not need a fixed address. It is executed by the current kernel to relocate and enter the new kernel, and is dead once the new kernel starts. Drop KEXEC_CONTROL_CODE and reuse the control_code_page that the kexec core already allocates (as arm64 and riscv do). That page is excluded from the relocation copy and lives nowhere near 0x100000, so the QEMU FDT is left intact.
--- arch/loongarch/kernel/machine_kexec.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
arch/loongarch/kernel/machine_kexec.c +8 -5
@@ -21,8 +21,6 @@#include <asm/cacheflush.h> #include <asm/page.h> -/* 0x100000 ~ 0x200000 is safe */ -#define KEXEC_CONTROL_CODE TO_CACHE(0x100000UL)#define KEXEC_CMDLINE_ADDR TO_CACHE(0x108000UL)static unsigned long reboot_code_buffer;
@@ -76,9 +74,14 @@ int machine_kexec_prepare(struct kimage *kimage) } } -/* kexec/kdump need a safe page to save reboot_code_buffer */ - kimage->control_code_page =virt_to_page ((void *)KEXEC_CONTROL_CODE ); - +/* +* kexec/kdump need a safe page to save reboot_code_buffer. Reuse the +* control_code_page allocated by the kexec core (as arm64 and riscv +* do) instead of a fixed address: the trampoline is only executed by +* the current kernel before entering the new kernel, so it needs no +* fixed or reserved location. This also stops machine_kexec_prepare() +* from overwriting QEMU's machine FDT at 0x100000. +*/ reboot_code_buffer = (unsigned long )page_address (kimage->control_code_page);memcpy ((void *)reboot_code_buffer, relocate_new_kernel, relocate_new_kernel_size);
Signature · 1 line
2.25.1
PATCH v4 2/2 LoongArch: kexec: load the command line in its own segment
In reply to George GuoFrom: George Guo <[email protected]>
The kexec'd kernel receives its command line through register a1, which machine_kexec_prepare() pointed at the fixed address KEXEC_CMDLINE_ADDR (0x108000) in the reserved first 2MB. That address overlaps the firmware-provided FDT. QEMU places its machine FDT at physical 0x100000; dumping the FDT it generates makes the overlap concrete:
$ qemu-system-loongarch64 -M virt -cpu la464 -m 1G -smp 2 \
-machine dumpdtb=qemu.dtb
$ fdtdump qemu.dtb
// totalsize: 0x100000 (1048576)
// off_dt_strings: 0xd04
// size_dt_strings: 0x179The FDT declares fdt_totalsize = 1MB, so it claims the entire [0x100000, 0x200000) upper half of the reserved region, even though its actual content ends at off_dt_strings + size_dt_strings = 0xe7d (~3.6KB). KEXEC_CMDLINE_ADDR (0x108000 = base + 32KB) therefore sits inside the FDT region and only survives because it lands in the trailing slack. A larger FDT (more CPU or memory nodes) whose content grows past 32KB would be clobbered when the command line is written, reproducing the silent second boot that earlier motivated moving the relocation trampoline off 0x100000.
The command line is passed by register, not at a hardcoded location, so it does not need a fixed address. Load it into its own kexec segment instead:
- In file mode, add modified_cmdline as a segment with kexec_add_buffer() and store its destination (TO_CACHE(kbuf.mem)) in arch.cmdline_ptr. The buffer is now owned by the image and freed in post_load_cleanup(), which also fixes a pre-existing leak of modified_cmdline.
- In the legacy kexec_load() path, point arch.cmdline_ptr at the destination of the user-supplied command line segment instead of copying it to the fixed address.
Drop KEXEC_CMDLINE_ADDR. The kexec core places the segment clear of the firmware FDT, so the overlap is removed regardless of FDT size.
Tested with tools/testing/selftests/kho/vmtest.sh on LoongArch (file mode, kexec_file_load): the second kernel comes up with the full command line ("kexec_file console=ttyS0 earlycon kho=on panic=-1") and earlycon, and KHO restore succeeds.
--- arch/loongarch/include/asm/kexec.h | 1 + arch/loongarch/kernel/machine_kexec.c | 25 +++++++--------- arch/loongarch/kernel/machine_kexec_file.c | 33 ++++++++++++++++++++-- 3 files changed, 41 insertions(+), 18 deletions(-)
arch/loongarch/include/asm/kexec.h +1 -0
@@ -38,6 +38,7 @@ static inline void crash_setup_regs(struct pt_regs *newregs,struct kimage_arch {unsigned long efi_boot;unsigned long cmdline_ptr; +void *cmdline_buf;/* file mode: cmdline segment buffer, freed at cleanup */ unsigned long systable_ptr; };
arch/loongarch/kernel/machine_kexec.c +10 -15
@@ -21,8 +21,6 @@#include <asm/cacheflush.h> #include <asm/page.h> -#define KEXEC_CMDLINE_ADDR TO_CACHE(0x108000UL) -static unsigned long reboot_code_buffer;static cpumask_t cpus_in_crash =CPU_MASK_NONE ;
@@ -41,29 +39,26 @@ int machine_kexec_prepare(struct kimage *kimage) {int i;char *bootloader ="kexec" ; -void *cmdline_ptr = (void *)KEXEC_CMDLINE_ADDR ; kimage->arch.efi_boot = fw_arg0; kimage->arch.systable_ptr = fw_arg2; -if (kimage->file_mode ==1 ) { -/* -* kimage->cmdline_buf will be released in kexec_file_load, so copy -* to the KEXEC_CMDLINE_ADDR safe area. -*/ -memcpy ((void *)KEXEC_CMDLINE_ADDR , (void *)kimage->arch.cmdline_ptr, -strlen ((char *)kimage->arch.cmdline_ptr) +1 ); - kimage->arch.cmdline_ptr = (unsigned long )KEXEC_CMDLINE_ADDR ; - }else { +/* +* In file mode load_other_segments() already placed the command line in +* its own segment and stored its address in arch.cmdline_ptr. For the +* legacy kexec_load() path the command line is supplied in a segment +* whose buffer starts with the bootloader name; point the second kernel +* at that segment's destination instead of copying it to a fixed +* address that can overlap the firmware FDT. +*/ +if (kimage->file_mode !=1 ) {char head[8 ]; -/* Find the command line */ for (i =0 ; i < kimage->nr_segments; i++) {if (copy_from_user (head, kimage->segment[i].buf,strlen (bootloader)))continue ;if (!strncmp (bootloader, head,strlen (bootloader))) { -if (!copy_from_user (cmdline_ptr, kimage->segment[i].buf,COMMAND_LINE_SIZE )) - kimage->arch.cmdline_ptr = (unsigned long )cmdline_ptr; + kimage->arch.cmdline_ptr =TO_CACHE (kimage->segment[i].mem);break ; } }
arch/loongarch/kernel/machine_kexec_file.c +30 -3
@@ -32,6 +32,9 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image) image->elf_headers =NULL ; image->elf_headers_sz =0 ; +kfree (image->arch.cmdline_buf); + image->arch.cmdline_buf =NULL ; +return kexec_image_post_load_cleanup_default (image); }
@@ -152,6 +155,8 @@ int load_other_segments(struct kimage *image, modified_cmdline =kzalloc (COMMAND_LINE_SIZE ,GFP_KERNEL );if (!modified_cmdline)return -EINVAL ; +/* Owned by the image from here on; freed in post_load_cleanup(). */ + image->arch.cmdline_buf = modified_cmdline;cmdline_add_loader (&cmdline_tmplen, modified_cmdline);/* Ensure it's null terminated */
@@ -227,13 +232,35 @@ int load_other_segments(struct kimage *image, }memcpy (modified_cmdline + cmdline_tmplen, cmdline, cmdline_len); - cmdline = modified_cmdline; - image->arch.cmdline_ptr = (unsigned long )cmdline; + +/* +* Load the command line into its own segment instead of copying it to +* a fixed address in the reserved low memory, which can overlap the +* firmware-provided FDT (e.g. QEMU places a 1MB machine FDT at +* 0x100000). The second kernel receives this segment's address in +* register a1; see machine_kexec(). +*/ + kbuf.buffer = modified_cmdline; + kbuf.bufsz =COMMAND_LINE_SIZE ; + kbuf.mem =KEXEC_BUF_MEM_UNKNOWN ; + kbuf.memsz =COMMAND_LINE_SIZE ; + kbuf.buf_align =PAGE_SIZE ; + kbuf.buf_max =ULONG_MAX ; + kbuf.top_down = true; + + ret =kexec_add_buffer (&kbuf); +if (ret <0 ) + goto out_err; + + image->arch.cmdline_ptr =TO_CACHE (kbuf.mem); + +kexec_dprintk ("Loaded command line at 0x%lx memsz=0x%x\n" , + kbuf.mem,COMMAND_LINE_SIZE );return 0 ; out_err: image->nr_segments = orig_segments; -kfree (modified_cmdline); +/* modified_cmdline (image->arch.cmdline_buf) is freed in cleanup. */ return ret; }
Signature · 1 line
2.25.1
Huacai Chen
maintainerHi, George,
Quoted 18 lines · George Guo
This series fixes two ways the LoongArch kexec path can clobber the firmware-provided FDT that QEMU places at physical
0x100000, which made the second kernel boot silently (earlycon auto-detection reads a corrupted FDT).Patch 1 moves the relocation trampoline off the hardcoded
0x100000onto the kexec core'scontrol_code_page(as arm64/riscv do), somachine_kexec_prepare()no longer overwrites the FDT.Patch 2 removes the remaining fixed command-line address (
KEXEC_CMDLINE_ADDR,0x108000). QEMU's machine FDT declaresfdt_totalsize= 1MB and thus claims the whole [0x100000,0x200000) upper half of the reserved 2MB, so0x108000lives inside the FDT and only survives because the FDT's real content is small (~3.6KB) and0x108000falls in the trailing slack. The command line is passed to the second kernel in register a1, not at a fixed address, so it is loaded into its own kexec segment instead, removing the overlap regardless of FDT size.
Quoted 5 lines
This addresses the two questions raised on the v3 trampoline patch [1]:
- "how to avoid the cmdline overlapping FDT" -> patch 2.
- "how to avoid the control page being corrupted by the new kernel" -> the trampoline is executed only by the current kernel and is dead once the new kernel starts, so being overwritten afterwards is harmless; the kexec core
Who told you the trampoline is dead code? All non-boot CPUs are running in the trampoline, you can use maxcpus=1 to go to the new kernel, and do something, and then hotplug the non-boot CPUs at any time.
The trampoline should live all the time for safety, ARM and RISC-V both have the trouble, especially when memory is small.
And I don't want to hear "If others can do something but LoongArch can't, then LoongArch has a drawback; if others can't do something but LoongArch can, then it is useless".
BTW, you have never replied to my comments, all you have done is submit new versions again and again. Who told you to do things like this? You want to take your ill code again and again, or your employer counts the number of your E-mails in the mailist as the amount of your work?
Huacai
Quoted 39 lines
also keeps the control page outside every relocation destination, so it cannot be corrupted during relocation. The command line is different (it is read by the new kernel), which is why only the trampoline uses the core page.
Tested with
tools/testing/selftests/kho/vmtest.shon LoongArch (QEMU,kexec_file_load): the second kernel boots with the full command line and earlycon, and KHO restore succeeds.This series depends on [2] ("LoongArch: kexec: Fix address space mismatch in command line lookup"), whose char head[] change patch 2 builds on.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/Changes in v4:
- Add patch 2: load the command line in its own kexec segment, dropping the fixedKEXEC_CMDLINE_ADDRthat overlaps the QEMU FDT (addresses the command line concern from the v3 review).
- Patch 1 (relocation trampoline): drop theKEXEC_CMDLINE_ADDRcomment added in v3 that claimed0x108000does not overlap the FDT (it does; patch 2 removes the fixed address entirely). No functional change otherwise.v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/George Guo (2):
LoongArch: kexec: use core control page for relocation trampoline to
avoid QEMU FDT conflict
LoongArch: kexec: load the command line in its own segmentarch/loongarch/include/asm/kexec.h | 1 + arch/loongarch/kernel/machine_kexec.c | 38 ++++++++++------------ arch/loongarch/kernel/machine_kexec_file.c | 33 +++++++++++++++++-- 3 files changed, 49 insertions(+), 23 deletions(-)--
2.25.1
Hi Huacai,
Thanks for the correction, and I am sorry for the way I handled this.
Quoted 4 lines
Who told you the trampoline is dead code? All non-boot CPUs are running in the trampoline, you can use maxcpus=1 to go to the new kernel, and do something, and then hotplug the non-boot CPUs at any time.
You are right, and I was wrong to call the trampoline dead once the new kernel starts. The non-boot CPUs spin in kexec_smp_wait inside the trampoline buffer and only leave it when the new kernel brings them online, so with maxcpus=1 they stay there until a later CPU hotplug. The buffer has to stay valid the whole time.
That breaks my patch 1. control_code_page is allocated by the kexec core and is not reserved in the new kernel, so the new kernel can reuse that page while the non-boot CPUs are still spinning in it, and small memory makes that more likely. The fixed KEXEC_CONTROL_CODE at 0x100000 avoided this because the first 2MB is memblock_reserve()d in both kernels; its only problem was that it overlaps QEMU's machine FDT at 0x100000. So my Q2 answer in the cover letter was also wrong, and "as arm64/riscv do" does not apply here.
So the trampoline still needs to live in the cross-kexec reserved first 2MB, just not on top of the FDT. The FDT declares fdt_totalsize 1MB and covers [0x100000, 0x200000), so the room left is below 0x100000.
Do you have a different fix in mind? I would like to agree on the direction with you before I send anything.
Quoted 2 lines
BTW, you have never replied to my comments, all you have done is submit new versions again and again.
That is fair, and it was my mistake. I should have replied to your review in the thread and discussed it before sending a new version, instead of putting my answers in a cover letter. I will reply to comments first from now on.
I will hold the series until we settle the trampoline approach.
Thanks,
George
Huacai Chen
maintainerQuoted 35 lines · George Guo
Hi Huacai,
Thanks for the correction, and I am sorry for the way I handled this.
Quoted 4 lines
Who told you the trampoline is dead code? All non-boot CPUs are running in the trampoline, you can use maxcpus=1 to go to the new kernel, and do something, and then hotplug the non-boot CPUs at any time.
You are right, and I was wrong to call the trampoline dead once the new kernel starts. The non-boot CPUs spin in
kexec_smp_waitinside the trampoline buffer and only leave it when the new kernel brings them online, so with maxcpus=1 they stay there until a later CPU hotplug. The buffer has to stay valid the whole time.That breaks my patch 1.
control_code_pageis allocated by the kexec core and is not reserved in the new kernel, so the new kernel can reuse that page while the non-boot CPUs are still spinning in it, and small memory makes that more likely. The fixedKEXEC_CONTROL_CODEat0x100000avoided this because the first 2MB is memblock_reserve()d in both kernels; its only problem was that it overlaps QEMU's machine FDT at0x100000. So my Q2 answer in the cover letter was also wrong, and "as arm64/riscv do" does not apply here.So the trampoline still needs to live in the cross-kexec reserved first 2MB, just not on top of the FDT. The FDT declares
fdt_totalsize1MB and covers [0x100000,0x200000), so the room left is below0x100000.Do you have a different fix in mind? I would like to agree on the direction with you before I send anything.
Quoted 2 lines
BTW, you have never replied to my comments, all you have done is submit new versions again and again.
Quoted 6 lines
That is fair, and it was my mistake. I should have replied to your review in the thread and discussed it before sending a new version, instead of putting my answers in a cover letter. I will reply to comments first from now on.
I will hold the series until we settle the trampoline approach.
I had proposed a reasonable solution as early as in v1.
Huacai
Quoted 3 lines
Thanks,
George