Skip to contentSkip to conversation

riscv: Enable kexec handover and live update

PATCH Superseded Chen Pei 2 participants 10 messages Oct 23 – Oct 26

Subsystems (5)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Chen Pei

author
#0 lore raw

cover riscv: Enable kexec handover and live update

Host kernel updates in the cloud should not require draining the
machine: next to live migration to another host, replacing the kernel in place while preserving selected workload state ("live update") is what KHO/LUO were built for, and LUO's documentation names hypervisor hosts as its primary use case. This series closes the gap for riscv64 and validates it with both upstream test runners.

KHO has been architecture independent since it was merged: the outgoing kernel publishes the handover FDT and the scratch descriptor array in the /chosen node of the DTB it builds for the incoming kernel, and the incoming kernel collects both while scanning that node -- which is why the arm64 integration (commit 274cdcb1c004) needed only three lines of arch code.

RISC-V already walks those same generic paths (load_extra_segments() -> of_kexec_alloc_and_setup_fdt(), parse_dtb() -> early_init_dt_scan()). The one real blocker is ordering: setup_arch() runs early_ioremap_setup() *after* parse_dtb(), so kho_populate()'s early_memremap() of the handover FDT returns "offset + 0" -- NULL for a page aligned address -- and the handover is dropped:

KHO: setup: failed to memremap FDT (0x810bb000)
KHO: disabling KHO revival

Patch 1 fixes that ordering, matching arm64/x86 and standing on its own regardless of KHO. Patch 2 enables ARCH_SUPPORTS_KEXEC_HANDOVER as "def_bool 64BIT && MMU" rather than arm64's "def_bool y":
ARCH_SUPPORTS_KEXEC_FILE is already 64BIT-only, and KEXEC_HANDOVER selects CMA, which depends on MMU. Patches 3 and 4 teach the two upstream vmtest runners about riscv64. LIVEUPDATE depends only on KEXEC_HANDOVER, so patch 2 makes it selectable on riscv as well.

No DT schema changes are needed: linux,kho-fdt / linux,kho-scratch are the properties arm64 introduced (commit 274cdcb1c004), and their binding lives in the external dt-schema repository [1] rather than in Documentation.

[1] https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/chosen.yaml

Test
----
Testing on qemu-system-riscv64 (virt, -cpu max, TCG); method and logs under the --- of patches 3 and 4:
- kho vmtest: "ok 1 KHO succeeded"
- liveupdate vmtest, all four suites PASSED (simple, multi-session,

    500 files, 2000 sessions)

- riscv defconfig with and without CONFIG_KEXEC_HANDOVER both boot to userspace unchanged; builds: riscv defconfig (+KHO), rv32_defconfig, arm64, x86_64

Additionally, a workload-continuity test was run on the same setup, outside the selftests: a counter process was snapshot with criu 4.3, the dump was carried across the kexec in an LUO-preserved memfd, and criu restored it in the new kernel. The process resumed with the same PID and open log file and kept counting from where it was dumped (500006 -> 500007, no lost or duplicated ticks).

Chen Pei (4):
riscv: Call early_ioremap_setup() before parsing the DTB
riscv: Add support for kexec handover (KHO)
selftests: kho: Add riscv support
selftests: liveupdate: Add riscv support

 arch/riscv/Kconfig                                | 3 +++
 arch/riscv/kernel/setup.c                         | 2 +-
 tools/testing/selftests/kho/riscv.conf            | 9 +++++++++
 tools/testing/selftests/kho/vmtest.sh             | 3 ++-
 tools/testing/selftests/liveupdate/config.riscv64 | 3 +++
 tools/testing/selftests/liveupdate/run-vmtests.sh | 2 +-
 tools/testing/selftests/liveupdate/vmtest.sh      | 8 +++++++-
 7 files changed, 26 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/kho/riscv.conf
 create mode 100644 tools/testing/selftests/liveupdate/config.riscv64
Signature · 1 line
2.50.1
author
#1 lore raw

PATCH 1/4 riscv: Call early_ioremap_setup() before parsing the DTB

early_ioremap_setup() fills in the slot_virt[] array whose entries the return value of early_ioremap()/early_memremap() is composed from. On RISC-V it ran after parse_dtb(), so an early_memremap() reached from the device tree scan returned "offset + 0" -- NULL for a page aligned address -- and the caller could only report a mapping failure. This stayed latent because nothing reached from parse_dtb() asked for an early mapping before; the DT based kexec handover parsing is the first such caller, as kho_populate() maps the handover FDT and the scratch descriptor array with early_memremap().

Move the call to the top of setup_arch(), which is how arm64 and x86 already order it (early_fixmap_init()/early_ioremap_init() before setup_machine_fdt()/parse_setup_data()). The BTMAP window is mapped into early_pg_dir by setup_vm(), and RISC-V neither calls early_ioremap_reset() nor defines __late_set_fixmap(), so the virtual addresses stay identical across the switch to the final page tables and nothing else in setup_arch() depends on the previous ordering.

---
 arch/riscv/kernel/setup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/kernel/setup.c +1 -1
index d79f99b86033..45e2ef9d32fa 100644
@@ -308,12 +308,12 @@ extern void __init init_rt_signal_env(void);
 
 void __init setup_arch(char **cmdline_p)
 {
+	early_ioremap_setup();
 	parse_dtb();
 	setup_initial_init_mm(_stext, _etext, _edata, _end);
 
 	*cmdline_p = boot_command_line;
 
-	early_ioremap_setup();
 	sbi_init();
 	jump_label_init();
 	parse_early_param();
Signature · 1 line
2.50.1
#2 lore raw

Re: [PATCH 1/4] riscv: Call early_ioremap_setup() before parsing the DTB

Sashiko has reviewed this patch and found no issues.
Quoted 18 lines

early_ioremap_setup() fills in the slot_virt[] array whose entries the return value of early_ioremap()/early_memremap() is composed from. On RISC-V it ran after parse_dtb(), so an early_memremap() reached from the device tree scan returned "offset + 0" -- NULL for a page aligned address -- and the caller could only report a mapping failure. This stayed latent because nothing reached from parse_dtb() asked for an early mapping before; the DT based kexec handover parsing is the first such caller, as kho_populate() maps the handover FDT and the scratch descriptor array with early_memremap().

Move the call to the top of setup_arch(), which is how arm64 and x86 already order it (early_fixmap_init()/early_ioremap_init() before setup_machine_fdt()/parse_setup_data()). The BTMAP window is mapped into early_pg_dir by setup_vm(), and RISC-V neither calls early_ioremap_reset() nor defines __late_set_fixmap(), so the virtual addresses stay identical across the switch to the final page tables and nothing else in setup_arch() depends on the previous ordering.

Quoted 1 line

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#3 lore raw

PATCH 2/4 riscv: Add support for kexec handover (KHO)

In reply to Chen Pei

KHO is architecture independent: the outgoing kernel publishes the handover FDT and the scratch descriptor array in the /chosen node of the DTB it builds for the incoming kernel (kho_add_chosen(), reached from of_kexec_alloc_and_setup_fdt()), and the incoming kernel picks both up while scanning that node (early_init_dt_check_kho() -> kho_populate()).

RISC-V already builds that DTB in load_extra_segments() and passes it to the incoming kernel in a1, so only the architecture symbol is missing. Depend on 64BIT to match ARCH_SUPPORTS_KEXEC_FILE, and on MMU because KEXEC_HANDOVER selects CMA, which depends on MMU.

---
 arch/riscv/Kconfig | 3 +++
 1 file changed, 3 insertions(+)
arch/riscv/Kconfig +3 -0
index 5874ca30f4f6..4f7d2f3f80b1 100644
@@ -1125,6 +1125,9 @@ config ARCH_SELECTS_KEXEC_FILE
 config ARCH_SUPPORTS_KEXEC_PURGATORY
 	def_bool ARCH_SUPPORTS_KEXEC_FILE
 
+config ARCH_SUPPORTS_KEXEC_HANDOVER
+	def_bool 64BIT && MMU
+
 config ARCH_SUPPORTS_CRASH_DUMP
 	def_bool y
 
Signature · 1 line
2.50.1
#4 lore raw

Re: [PATCH 2/4] riscv: Add support for kexec handover (KHO)

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] riscv: kho: BUG_ON() in linear_mapping_pa_to_va() during efi_init() - [Medium] riscv: kho: bootmem…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] riscv: kho: BUG_ON() in linear_mapping_pa_to_va() during efi_init()
- [Medium] riscv: kho: bootmem reservation fails when DRAM starts at or above 4 GiB --

commit 82f9788fea5dca057c5c8476c422bb40724521df
Author: Chen Pei <[email protected]>

riscv: Add support for kexec handover (KHO)

This commit enables ARCH_SUPPORTS_KEXEC_HANDOVER on 64-bit RISC-V systems with MMU support so that kexec handover (KHO) can be used on RISC-V.

arch/riscv/Kconfig config ARCH_SELECTS_KEXEC_FILE · line 1129

Quoted 10 lines
arch/riscv/Kconfig +2 -0
index e7891a2a89da6..a5c56252ff44a 100644
@@ -1125,6 +1125,9 @@ config ARCH_SELECTS_KEXEC_FILE
 config ARCH_SUPPORTS_KEXEC_PURGATORY
 	def_bool ARCH_SUPPORTS_KEXEC_FILE
 
+config ARCH_SUPPORTS_KEXEC_HANDOVER
+	def_bool 64BIT && MMU

[Severity: High]
Can enabling ARCH_SUPPORTS_KEXEC_HANDOVER trigger a BUG_ON() in linear_mapping_pa_to_va() during efi_init() when booting a KHO kexec kernel on a 64-bit RISC-V UEFI system?

In arch/riscv/kernel/setup.c:setup_arch(), parse_dtb() runs before efi_init(), and efi_init() runs before paging_init():

    parse_dtb();
    setup_initial_init_mm(_stext, _etext, _edata, _end);

    *cmdline_p = boot_command_line;

    early_ioremap_setup();
    sbi_init();
    jump_label_init();
     parse_early_param();

    efi_init();
    paging_init();

During parse_dtb(), early_init_dt_scan() -> early_init_dt_check_kho() -> kho_populate() sets kho_in.fdt_phys to the physical address of the handover FDT.

Next, efi_init() -> reserve_regions() in drivers/firmware/efi/efi-init.c checks is_kho_boot(), which calls kho_get_fdt() in kernel/liveupdate/kexec_handover.c:

    static const void *kho_get_fdt(void)
    {
            return kho_in.fdt_phys ? phys_to_virt(kho_in.fdt_phys) : NULL;
    }

However, on 64-bit RISC-V, kernel_map.va_pa_offset is initialized to 0UL in setup_vm() and is not populated until paging_init() -> setup_bootmem() in arch/riscv/mm/init.c:

    /*
     * In 64-bit, any use of __va/__pa before this point is wrong as we
     * did not know the start of DRAM before.
     */
    if (IS_ENABLED(CONFIG_64BIT) && IS_ENABLED(CONFIG_MMU))
            kernel_map.va_pa_offset = PAGE_OFFSET - phys_ram_base;

When CONFIG_DEBUG_VIRTUAL=y is enabled, calling phys_to_virt(kho_in.fdt_phys) before setup_bootmem() reaches linear_mapping_pa_to_va() in arch/riscv/mm/physaddr.c:

    void *linear_mapping_pa_to_va(unsigned long x)
    {
            BUG_ON(!kernel_map.va_pa_offset);

            return ((void *)((unsigned long)(x) + kernel_map.va_pa_offset));
    }

Would this panic the incoming kernel during efi_init() (or compute an untranslated address in kho_get_fdt() when CONFIG_DEBUG_VIRTUAL=n)?

[Severity: Medium]
Also, will kho_reserve_bootmem() fail on 64-bit RISC-V platforms where DRAM starts at or above 4 GiB, or record a 0-byte region when all DRAM lies below 4 GiB?

Unlike arm64 (which defines ARCH_LOW_ADDRESS_LIMIT via arm64_dma_phys_limit), RISC-V uses the default ARCH_LOW_ADDRESS_LIMIT of 0xffffffffUL from include/linux/memblock.h, while arch_mm_preinit() in arch/riscv/mm/init.c supports systems where memblock_start_of_DRAM() >= dma32_phys_limit (4 GiB) or max_pfn <= PFN_DOWN(dma32_phys_limit).

In kernel/liveupdate/kexec_handover.c:kho_reserve_bootmem(), both the lowmem and global allocations are performed unconditionally without checking whether bootmem_size_lowmem or bootmem_size_global is 0:

    size = bootmem_size_lowmem;
    addr = memblock_phys_alloc_range(size, BOOTMEM_ALIGNMENT_BYTES, 0,
                                     ARCH_LOW_ADDRESS_LIMIT);
    if (!addr) {
            pr_err("Failed to reserve lowmem bootmem\n");
            goto err_free_bootmem_desc;
    }

    kho_bootmem[i].addr = addr;
    kho_bootmem[i].size = size;
    i++;

    /* reserve large contiguous area for allocations without nid */
    size = bootmem_size_global;
    addr = memblock_phys_alloc(size, BOOTMEM_ALIGNMENT_BYTES);
    if (!addr) {
            pr_err("Failed to reserve global bootmem\n");
            goto err_free_bootmem_areas;
    }

On systems where all DRAM starts at or above 4 GiB, bootmem_size_lowmem is 0 and memblock_phys_alloc_range(0, BOOTMEM_ALIGNMENT_BYTES, 0, ARCH_LOW_ADDRESS_LIMIT) finds no memory below 0xffffffffUL, returning 0 and disabling KHO. Conversely, on systems where all DRAM is below 4 GiB, bootmem_size_global is 0 and memblock_phys_alloc(0, BOOTMEM_ALIGNMENT_BYTES) records a 0-byte entry in kho_bootmem[].

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#5 lore raw

Re: [PATCH 2/4] riscv: Add support for kexec handover (KHO)

Quoted 4 lines

[Severity: High]
Can enabling ARCH_SUPPORTS_KEXEC_HANDOVER trigger a BUG_ON() in linear_mapping_pa_to_va() during efi_init() when booting a KHO kexec kernel on a 64-bit RISC-V UEFI system?

Confirmed. Reproduced on qemu virt booting through edk2 with DEBUG_VIRTUAL enabled and "kho=on" on the outgoing kernel: the first handover dies in the incoming one during EFI setup, before the console registers, so it stays silent without an earlycon.

This will be fixed in v2.

Quoted 4 lines

[Severity: Medium]
Also, will kho_reserve_bootmem() fail on 64-bit RISC-V platforms where DRAM starts at or above 4 GiB, or record a 0-byte region when all DRAM lies below 4 GiB?

Confirmed, both halves. That is generic KHO code, not the riscv enablement, so I will send a separate patch for it.

Best regards,
Chen

author
#6 lore raw

PATCH 3/4 selftests: kho: Add riscv support

In reply to Chen Pei

Teach the KHO vmtest runner about riscv64: map the target to ARCH riscv and describe the QEMU virt machine, the kernel image and its console -- a 16550A port probed from the device tree on virt, hence the SERIAL_8250{,_CONSOLE} and SERIAL_OF_PLATFORM symbols. Also select CONFIG_SPARSEMEM_MANUAL: the runner requires CONFIG_DEFERRED_STRUCT_PAGE_INIT, which depends on SPARSEMEM, while riscv falls back to FLATMEM (no ARCH_SPARSEMEM_DEFAULT), so without it the test would just be skipped.

---
Test: qemu-system-riscv64 (virt, -cpu max, TCG), in-tree runner "tools/testing/selftests/kho/vmtest.sh -t riscv64" (boots the Image and kexec'es into the same Image).

KHO: found kexec handover data.
KHO: exec from: 7.3.0-rc7-00108-gc6e90ee4ccbb (count 1)
test_kho: KHO restore succeeded
ok 1 KHO succeeded
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

 tools/testing/selftests/kho/riscv.conf | 9 +++++++++
 tools/testing/selftests/kho/vmtest.sh  | 3 ++-
 2 files changed, 11 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/kho/riscv.conf
tools/testing/selftests/kho/riscv.conf new file +9 -0
new file mode 100644
index 000000000000..ec2a46cd45d5
@@ -0,0 +1,9 @@
+QEMU_CMD="qemu-system-riscv64 -M virt -cpu max"
+QEMU_KCONFIG="
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_OF_PLATFORM=y
+CONFIG_SPARSEMEM_MANUAL=y
+"
+KERNEL_IMAGE="Image"
+KERNEL_CMDLINE="console=ttyS0"
tools/testing/selftests/kho/vmtest.sh +2 -1
index 0014bd76e88d..325af59f653a 100755
@@ -21,7 +21,7 @@ Options:
 	-d)	path to the kernel build directory
 	-j)	number of jobs for compilation, similar to -j in make
 	-t)	run test for target_arch, requires CROSS_COMPILE set
-		supported targets: aarch64, x86_64
+		supported targets: aarch64, x86_64, riscv64
 	-h)	display this help
 EOF
 }
@@ -127,6 +127,7 @@ function target_to_arch() {
 	case $target in
 	     aarch64) echo "arm64" ;;
 	     x86_64) echo "x86" ;;
+	     riscv64) echo "riscv" ;;
 	     *) skip "architecture $target is not supported"
 	esac
 }
Signature · 1 line
2.50.1
#7 lore raw

Re: [PATCH 3/4] selftests: kho: Add riscv support

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

Teach the KHO vmtest runner about riscv64: map the target to ARCH riscv and describe the QEMU virt machine, the kernel image and its console -- a 16550A port probed from the device tree on virt, hence the SERIAL_8250{,_CONSOLE} and SERIAL_OF_PLATFORM symbols. Also select CONFIG_SPARSEMEM_MANUAL: the runner requires CONFIG_DEFERRED_STRUCT_PAGE_INIT, which depends on SPARSEMEM, while riscv falls back to FLATMEM (no ARCH_SPARSEMEM_DEFAULT), so without it the test would just be skipped.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#8 lore raw

PATCH 4/4 selftests: liveupdate: Add riscv support

In reply to Chen Pei

Teach the live update vmtest runner about riscv64: map the target to ARCH riscv, describe the QEMU virt machine, the kernel image and its console, and add the arch config fragment holding the three symbols needed to get a console on the virt machine's 16550A port.

KHO is the only dependency of LIVEUPDATE (kernel/liveupdate/Kconfig), so now that riscv has KHO the LUO tests only need the runner to stop rejecting the architecture.

Add riscv64 to the CROSS_COMPILE auto-detection list and to the target list of the batch runner as well, matching the existing aarch64/x86_64 entries.

---
Test: qemu-system-riscv64 (virt, -cpu max, -m 1G, -smp 2, TCG), in-tree runner "tools/testing/selftests/liveupdate/vmtest.sh -t riscv64 -T <suite>", all four suites:

  luo_kexec_simple:    SIMPLE KEXEC TEST PASSED
  luo_multi_session:   MULTI-SESSION KEXEC TEST PASSED
  luo_stress_files:    MANY-FILES KEXEC TEST PASSED (500 files)

luo_stress_sessions: MANY-SESSIONS KEXEC TEST PASSED (2000 sessions)

The simple suite's serial log shows the full handover round trip:

KHO: found kexec handover data.
luo: Retrieved live update data, liveupdate number: 1 # [STAGE 2] Test data verified successfully. --- SIMPLE KEXEC TEST PASSED ---

 tools/testing/selftests/liveupdate/config.riscv64 | 3 +++
 tools/testing/selftests/liveupdate/run-vmtests.sh | 2 +-
 tools/testing/selftests/liveupdate/vmtest.sh      | 8 +++++++-
 3 files changed, 11 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/liveupdate/config.riscv64
tools/testing/selftests/liveupdate/config.riscv64 new file +3 -0
new file mode 100644
index 000000000000..edbc551199b0
@@ -0,0 +1,3 @@
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_OF_PLATFORM=y
tools/testing/selftests/liveupdate/run-vmtests.sh +1 -1
index d656ce58c5a7..2e3f5f6f96b3 100755
@@ -4,7 +4,7 @@
 SCRIPT_DIR=$(dirname "$(realpath "$0")")
 TEST_RUNNER="$SCRIPT_DIR/vmtest.sh"
 
-TARGETS=("x86_64" "aarch64")
+TARGETS=("x86_64" "aarch64" "riscv64")
 
 GREEN='\033[0;32m'
 RED='\033[0;31m'
tools/testing/selftests/liveupdate/vmtest.sh +7 -1
index 64fd2ab55b76..ee554d889e1b 100755
@@ -25,6 +25,10 @@ function get_arch_conf() {
 		QEMU_CMD="qemu-system-x86_64"
 		KERNEL_IMAGE="bzImage"
 		KERNEL_CMDLINE="console=ttyS0"
+	elif [[ "$arch" == "riscv" ]]; then
+		QEMU_CMD="qemu-system-riscv64 -M virt -cpu max"
+		KERNEL_IMAGE="Image"
+		KERNEL_CMDLINE="console=ttyS0"
 	else
 		echo "Unsupported architecture: $arch"
 		exit 1
@@ -37,7 +41,7 @@ $0 [-d build_dir] [-j jobs] [-t target_arch] [-T test_name] [-w workspace_dir] [
 Options:
 	-d)	path to the kernel build directory (default: .luo_test_build.<arch>)
 	-j)	number of jobs for compilation
-	-t)	run test for target_arch (aarch64, x86_64)
+	-t)	run test for target_arch (aarch64, x86_64, riscv64)
 	-T)	test name to run (default: luo_kexec_simple)
 	-w)	custom workspace directory (default: creates temp dir)
 	-k)	keep workspace directory after successful test
@@ -87,6 +91,7 @@ function detect_cross_compile() {
 	case "$target" in
 		aarch64) candidate="aarch64-linux-gnu-" ;;
 		x86_64)  candidate="x86_64-linux-gnu-" ;;
+		riscv64) candidate="riscv64-linux-gnu-" ;;
 		*)       skip "Auto-detection for target '$target' not supported. Please set CROSS_COMPILE manually." ;;
 	esac
 
@@ -194,6 +199,7 @@ function target_to_arch() {
 	case $target in
 	     aarch64) echo "arm64" ;;
 	     x86_64) echo "x86" ;;
+	     riscv64) echo "riscv" ;;
 	     *) skip "architecture $target is not supported"
 	esac
 }
Signature · 1 line
2.50.1
#9 lore raw

Re: [PATCH 4/4] selftests: liveupdate: Add riscv support

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

Teach the live update vmtest runner about riscv64: map the target to ARCH riscv, describe the QEMU virt machine, the kernel image and its console, and add the arch config fragment holding the three symbols needed to get a console on the virt machine's 16550A port.

KHO is the only dependency of LIVEUPDATE (kernel/liveupdate/Kconfig), so now that riscv has KHO the LUO tests only need the runner to stop rejecting the architecture.

Quoted 5 lines

Add riscv64 to the CROSS_COMPILE auto-detection list and to the target list of the batch runner as well, matching the existing aarch64/x86_64 entries.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4