SMP support for RISC-V
Subsystems (2)
v3 cover SMP support for RISC-V
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration. Patch [1] has been dropped and will be replaced with a U-Boot patch.
[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/
Changes in v3:
- Print error if riscv_send_ipi() fails
- Adjust error message for failures of riscv_clear_ipi() to match error message for failures of riscv_send_ipi()
- New patch to save the hart ID in register tp instead of s0
- Adjust patch to use the new location of the hart ID (register tp)
- New patch to hang if relocation of secondary harts fails
Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available harts as indicated by the available_harts mask
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independently
Lukas Auer (11):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: save hart ID in register tp instead of s0
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: hang if relocation of secondary harts fails
riscv: fu540: enable SMP
riscv: qemu: enable SMP
arch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 167 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 + arch/riscv/include/asm/sbi.h | 94 +++++++++++++++ arch/riscv/include/asm/smp.h | 53 +++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 ++++ arch/riscv/lib/smp.c | 118 +++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 507 insertions(+), 12 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c
Signature · 1 line
2.20.1
PATCH v3 1/11 riscv: add infrastructure for calling functions on other harts
Harts on RISC-V boot independently, U-Boot is responsible for managing them. Functions are called on other harts with smp_call_function(), which sends inter-processor interrupts (IPIs) to all other available harts. Available harts are those marked as available in the device tree and present in the available_harts mask stored in global data. The available_harts mask is used to register all harts that have entered U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handler handle_ipi() must be called on software interrupts to handle the request and call the specified function.
Functions are stored in the ipi_data data structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks.
---
Changes in v3:
- Print error if riscv_send_ipi() fails
- Adjust error message for failures of riscv_clear_ipi() to match error message for failures of riscv_send_ipi()
Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available harts as indicated by the available_harts mask
arch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 118 +++++++++++++++++++++++++++ 5 files changed, 197 insertions(+) create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/smp.c
arch/riscv/Kconfig +19 -0
@@ -120,4 +120,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN default 0x1000 +config SMP + bool "Symmetric Multi-Processing" + help + This enables support for systems with more than one CPU. If + you say N here, U-Boot will run on single and multiprocessor + machines, but will use only one CPU of a multiprocessor + machine. If you say Y here, U-Boot will run on many, but not + all, single processor machines. + +config NR_CPUS + int "Maximum number of CPUs (2-32)" + range 2 32 + depends on SMP + default 8 + help + On multiprocessor machines, U-Boot sets up a stack for each CPU. + Stack memory is pre-allocated. U-Boot must therefore know the + maximum number of CPUs that may be present. + endmenu
arch/riscv/include/asm/global_data.h +6 -0
@@ -10,12 +10,18 @@#ifndef __ASM_GBL_DATA_H#define __ASM_GBL_DATA_H +#include <asm/smp.h> +/* Architecture-specific global data */ struct arch_global_data {long boot_hart;/* boot hart id */ #ifdef CONFIG_SIFIVE_CLINT void __iomem *clint;/* clint base address */ #endif +#ifdef CONFIG_SMP +struct ipi_data ipi[CONFIG_NR_CPUS ]; +#endif +ulong available_harts; };#include <asm-generic/global_data.h>
arch/riscv/include/asm/smp.h new file +53 -0
53 lines not on this page · raw message
arch/riscv/lib/Makefile +1 -0
@@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o obj-y += setjmp.o +obj-$(CONFIG_SMP) += smp.o # For building EFI apps CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)
arch/riscv/lib/smp.c new file +118 -0
118 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v3 2/11 riscv: import the supervisor binary interface header file
In reply to Lukas AuerImport the supervisor binary interface (SBI) header file from Linux (arch/riscv/include/asm/sbi.h). The last change to it was in commit 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI").
---
Changes in v3: None
Changes in v2: None
arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 arch/riscv/include/asm/sbi.h
arch/riscv/include/asm/sbi.h new file +94 -0
@@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015 Regents of the University of California +* +* Taken from Linux arch/riscv/include/asm/sbi.h +*/ + +#ifndef _ASM_RISCV_SBI_H +#define _ASM_RISCV_SBI_H + +#include <linux/types.h> + +#define SBI_SET_TIMER 0 +#define SBI_CONSOLE_PUTCHAR 1 +#define SBI_CONSOLE_GETCHAR 2 +#define SBI_CLEAR_IPI 3 +#define SBI_SEND_IPI 4 +#define SBI_REMOTE_FENCE_I 5 +#define SBI_REMOTE_SFENCE_VMA 6 +#define SBI_REMOTE_SFENCE_VMA_ASID 7 +#define SBI_SHUTDOWN 8 + +#define SBI_CALL (which, arg0, arg1, arg2) ({ \ + register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ + register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ + register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ + register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ + asm volatile ("ecall" \ + : "+r" (a0) \ + : "r" (a1), "r" (a2), "r" (a7) \ + : "memory"); \ + a0; \ +}) + +/* Lazy implementations until SBI is finalized */ +#define SBI_CALL_0 (which) SBI_CALL(which, 0, 0, 0) +#define SBI_CALL_1 (which, arg0) SBI_CALL(which, arg0, 0, 0) +#define SBI_CALL_2 (which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0) + +static inline void sbi_console_putchar (int ch) +{ +SBI_CALL_1 (SBI_CONSOLE_PUTCHAR , ch); +} + +static inline int sbi_console_getchar (void ) +{ +return SBI_CALL_0 (SBI_CONSOLE_GETCHAR ); +} + +static inline void sbi_set_timer (uint64_t stime_value) +{ +#if __riscv_xlen ==32 +SBI_CALL_2 (SBI_SET_TIMER , stime_value, stime_value >>32 ); +#else +SBI_CALL_1 (SBI_SET_TIMER , stime_value); +#endif +} + +static inline void sbi_shutdown (void ) +{ +SBI_CALL_0 (SBI_SHUTDOWN ); +} + +static inline void sbi_clear_ipi (void ) +{ +SBI_CALL_0 (SBI_CLEAR_IPI ); +} + +static inline void sbi_send_ipi (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_SEND_IPI , hart_mask); +} + +static inline void sbi_remote_fence_i (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_REMOTE_FENCE_I , hart_mask); +} + +static inline void sbi_remote_sfence_vma (const unsigned long *hart_mask, +unsigned long start, +unsigned long size) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA , hart_mask); +} + +static inline void sbi_remote_sfence_vma_asid (const unsigned long *hart_mask, +unsigned long start, +unsigned long size, +unsigned long asid) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA_ASID , hart_mask); +} + +#endif
Signature · 1 line
2.20.1
PATCH v3 3/11 riscv: implement IPI platform functions using SBI
In reply to Lukas AuerThe supervisor binary interface (SBI) provides the necessary functions to implement the platform IPI functions riscv_send_ipi() and riscv_clear_ipi(). Use it to implement them.
This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs running in supervisor mode. Support for machine mode is already available for CPUs that include the SiFive CLINT.
---
Changes in v3: None
Changes in v2: None
arch/riscv/Kconfig | 5 +++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+) create mode 100644 arch/riscv/lib/sbi_ipi.c
arch/riscv/Kconfig +5 -0
@@ -139,4 +139,9 @@ config NR_CPUS Stack memory is pre-allocated. U-Boot must therefore know the maximum number of CPUs that may be present. +config SBI_IPI + bool + default y if RISCV_SMODE + depends on SMP + endmenu
arch/riscv/lib/Makefile +1 -0
@@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o obj-y += setjmp.o obj-$(CONFIG_SMP) += smp.o
arch/riscv/lib/sbi_ipi.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#include <common.h> +#include <asm/sbi.h> + +int riscv_send_ipi (int hart) +{ +ulong mask; + + mask =1UL << hart; +sbi_send_ipi (&mask); + +return 0 ; +} + +int riscv_clear_ipi (int hart) +{ +sbi_clear_ipi (); + +return 0 ; +}
Signature · 1 line
2.20.1
PATCH v3 4/11 riscv: delay initialization of caches and debug UART
In reply to Lukas AuerMove the initialization of the caches and the debug UART until after board_init_f_init_reserve. This is in preparation for SMP support, where code prior to this point will be executed by all harts. This ensures that initialization will only be performed once on the main hart running U-Boot.
---
Changes in v3: None
Changes in v2: None
arch/riscv/cpu/start.S | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/riscv/cpu/start.S +8 -8
@@ -45,10 +45,6 @@ _start: /* mask all interrupts */ csrw MODE_PREFIX(ie), zero - /* Enable cache */ - jal icache_enable - jal dcache_enable - /* * Set stackpointer in internal/ex RAM to call board_init_f */
@@ -57,10 +53,6 @@ call_board_init_f: li t1, CONFIG_SYS_INIT_SP_ADDR and sp, t1, t0 /* force 16 byte alignment */ -#ifdef CONFIG_DEBUG_UART - jal debug_uart_init -#endif - call_board_init_f_0: mv a0, sp jal board_init_f_alloc_reserve
@@ -74,6 +66,14 @@ call_board_init_f_0: /* save the boot hart id to global_data */ SREG s0, GD_BOOT_HART(gp) + /* Enable cache */ + jal icache_enable + jal dcache_enable + +#ifdef CONFIG_DEBUG_UART + jal debug_uart_init +#endif + mv a0, zero /* a0 <-- boot_flags = 0 */ la t5, board_init_f jr t5 /* jump to board_init_f() */
Signature · 1 line
2.20.1
PATCH v3 5/11 riscv: save hart ID in register tp instead of s0
In reply to Lukas AuerThe hart ID passed by the previous boot stage is currently stored in register s0. If we divert the control flow inside a function, which is required as part of multi-hart support, the function epilog may not be called, clobbering register s0. Save the hart ID in the unallocatable register tp instead to protect the hart ID.
---
Changes in v3:
- New patch to save the hart ID in register tp instead of s0
Changes in v2: None
arch/riscv/cpu/start.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/riscv/cpu/start.S +2 -2
@@ -36,7 +36,7 @@ .globl _start _start: /* save hart id and dtb pointer */ - mv s0, a0 + mv tp, a0 mv s1, a1 la t0, trap_entry
@@ -64,7 +64,7 @@ call_board_init_f_0: jal board_init_f_init_reserve /* save the boot hart id to global_data */ - SREG s0, GD_BOOT_HART(gp) + SREG tp, GD_BOOT_HART(gp) /* Enable cache */ jal icache_enable
Signature · 1 line
2.20.1
PATCH v3 5/11 riscv: save hart ID in register tp instead of s0
Quoted 10 lines · Lukas Auer
The hart ID passed by the previous boot stage is currently stored in register s0. If we divert the control flow inside a function, which is required as part of multi-hart support, the function epilog may not be called, clobbering register s0. Save the hart ID in the unallocatable register tp instead to protect the hart ID.
Signed-off-by Lukas Auer <[email protected]>---
Quoted 8 lines
Changes in v3:
- New patch to save the hart ID in register tp instead of s0Changes in v2: None
arch/riscv/cpu/start.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v3 5/11 riscv: save hart ID in register tp instead of s0
Quoted 39 lines
Quoted 39 lines
From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de]
Sent: Monday, March 18, 2019 2:29 AM
To: u-boot at lists.denx.de
Cc: Anup Patel; Andreas Schwab; Bin Meng; Atish Patra; Palmer Dabbelt; Lukas Auer; Atish Patra; Anup Patel; Rick Jian-Zhi Chen(陳建志)
Subject: [PATCH v3 05/11] riscv: save hart ID in register tp instead of s0The hart ID passed by the previous boot stage is currently stored in register s0. If we divert the control flow inside a function, which is required as part of multi-hart support, the function epilog may not be called, clobbering register s0. Save the hart ID in the unallocatable register tp instead to protect the hart ID.
Signed-off-by Lukas Auer <[email protected]>---
Changes in v3:
- New patch to save the hart ID in register tp instead of s0Changes in v2: None
arch/riscv/cpu/start.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/riscv/cpu/start.S index renamed from arch/riscv/cpu/start.S +0
-0
a30f6f7194..bcc0ff696d100644arch/riscv/cpu/start.S +1
-1@@ -36,7 +36,7 @@ .globl _start _start: /* save hart id and dtb pointer */ - mv s0, a0 + mv tp, a0 mv s1, a1 la t0, trap_entry@@ -64,7 +64,7 @@ call_board_init_f_0: jal board_init_f_init_reserve
Quoted 6 lines
arch/riscv/cpu/start.S call_board_init_f_0: · line 70
Quoted 6 lines
arch/riscv/cpu/start.S +1
-1/* save the boot hart id to global_data */ - SREG s0, GD_BOOT_HART(gp) + SREG tp, GD_BOOT_HART(gp) /* Enable cache */ jal icache_enable
PATCH v3 5/11 riscv: save hart ID in register tp instead of s0
In reply to Lukas AuerQuoted 53 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Sunday, March 17, 2019 11:59 PM
To: u-boot at lists.denx.de
Cc: Anup Patel <[email protected]>; Andreas Schwab <[email protected]>; Bin Meng <[email protected]>; Atish Patra <[email protected]>; Palmer Dabbelt <[email protected]>; Lukas Auer <[email protected]>; Atish Patra <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>
Subject: [PATCH v3 05/11] riscv: save hart ID in register tp instead of s0The hart ID passed by the previous boot stage is currently stored in register s0. If we divert the control flow inside a function, which is required as part of multi-hart support, the function epilog may not be called, clobbering register s0. Save the hart ID in the unallocatable register tp instead to protect the hart ID.
Signed-off-by Lukas Auer <[email protected]>---
Changes in v3:
- New patch to save the hart ID in register tp instead of s0Changes in v2: None
arch/riscv/cpu/start.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/riscv/cpu/start.S index renamed from arch/riscv/cpu/start.S +0
-0
a30f6f7194..bcc0ff696d100644arch/riscv/cpu/start.S +2
-3@@ -36,7 +36,7 @@ .globl _start _start: /* save hart id and dtb pointer */ - mv s0, a0 + mv tp, a0 mv s1, a1 la t0, trap_entry@@ -64,7 +64,7 @@ call_board_init_f_0: jal board_init_f_init_reserve /* save the boot hart id to global_data */ - SREG s0, GD_BOOT_HART(gp) + SREG tp, GD_BOOT_HART(gp) /* Enable cache */ jal icache_enable --
Quoted 1 line
2.20.1
PATCH v3 6/11 riscv: add support for multi-hart systems
In reply to Lukas AuerOn RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to manage all harts in the system. All harts entering U-Boot are registered in the available_harts mask stored in global data. A hart lottery system as used in the Linux kernel selects the hart U-Boot runs on. All other harts are halted. U-Boot can delegate functions to them using smp_call_function().
Every hart has a valid pointer to the global data structure and a 8KiB stack by default. The stack size is set with CONFIG_STACK_SIZE_SHIFT.
---
Changes in v3:
- Adjust patch to use the new location of the hart ID (register tp)
Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask
arch/riscv/Kconfig | 4 ++ arch/riscv/cpu/cpu.c | 9 ++- arch/riscv/cpu/start.S | 134 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + arch/riscv/lib/asm-offsets.c | 1 + 5 files changed, 147 insertions(+), 2 deletions(-)
arch/riscv/Kconfig +4 -0
@@ -144,4 +144,8 @@ config SBI_IPI default y if RISCV_SMODE depends on SMP +config STACK_SIZE_SHIFT + int + default 13 + endmenu
arch/riscv/cpu/cpu.c +8 -1
@@ -12,10 +12,17 @@#include <dm/uclass-internal.h> /* -* prior_stage_fdt_address must be stored in the data section since it is used +* The variables here must be stored in the data section since they are used * before the bss section is available. */ phys_addr_t prior_stage_fdt_address __attribute__((section(".data"))); +u32 hart_lottery __attribute__((section(".data"))) = 0; + +/* +* The main hart running U-Boot has acquired available_harts_lock until it has +* finished initialization of global data. +*/ +u32 available_harts_lock =1 ;static inline bool supports_extension (char ext) {
arch/riscv/cpu/start.S +133 -1
181 lines not on this page · raw message
arch/riscv/include/asm/csr.h +1 -0
@@ -46,6 +46,7 @@#endif /* Interrupt Enable and Interrupt Pending flags */ +#define MIE_MSIE _AC(0x00000008, UL)/* Software Interrupt Enable */ #define SIE_SSIE _AC(0x00000002, UL)/* Software Interrupt Enable */ #define SIE_STIE _AC(0x00000020, UL)/* Timer Interrupt Enable */
arch/riscv/lib/asm-offsets.c +1 -0
@@ -14,6 +14,7 @@int main (void ) { DEFINE(GD_BOOT_HART, offsetof(gd_t, arch.boot_hart)); + DEFINE(GD_AVAILABLE_HARTS, offsetof(gd_t, arch.available_harts));return 0 ; }
Signature · 1 line
2.20.1
PATCH v3 7/11 riscv: boot images passed to bootm on all harts
In reply to Lukas Auer---
Changes in v3: None
Changes in v2: None
arch/riscv/lib/bootm.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/riscv/lib/bootm.c +12 -1
@@ -13,6 +13,7 @@#include <image.h> #include <asm/byteorder.h> #include <asm/csr.h> +#include <asm/smp.h> #include <dm/device.h> #include <dm/root.h> #include <u-boot/zlib.h>
@@ -81,6 +82,9 @@ static void boot_jump_linux(bootm_headers_t *images, int flag) {void (*kernel)(ulong hart,void *dtb);int fake = (flag &BOOTM_STATE_OS_FAKE_GO ); +#ifdef CONFIG_SMP +int ret; +#endif kernel = (void (*)(ulong ,void *))images->ep;
@@ -92,8 +96,15 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)announce_and_cleanup (fake);if (!fake) { -if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) +if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) { +#ifdef CONFIG_SMP + ret =smp_call_function (images->ep, + (ulong )images->ft_addr,0 ); +if (ret) +hang (); +#endif kernel (gd->arch.boot_hart, images->ft_addr); + } } }
Signature · 1 line
2.20.1
PATCH v3 8/11 riscv: do not rely on hart ID passed by previous boot stage
In reply to Lukas AuerRISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
---
Changes in v3: None
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
arch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/riscv/cpu/start.S +4 -0
@@ -36,6 +36,10 @@ .section .text .globl _start _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif + /* save hart id and dtb pointer */ mv tp, a0 mv s1, a1
Signature · 1 line
2.20.1
PATCH v3 9/11 riscv: hang if relocation of secondary harts fails
In reply to Lukas AuerPrint an error message and hang if smp_call_function() returns an error, indicating that relocation of the secondary harts has failed.
---
Changes in v3:
- New patch to hang if relocation of secondary harts fails
Changes in v2: None
arch/riscv/cpu/start.S | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/riscv/cpu/start.S +12 -1
@@ -33,6 +33,10 @@ #define SYM_SIZE 0x18 #endif +.section .data +secondary_harts_relocation_error: + .ascii "Relocation of secondary harts has failed, error %d\n" + .section .text .globl _start _start:
@@ -274,8 +278,15 @@ relocate_secondary_harts: mv a2, s3 jal smp_call_function + /* hang if relocation of secondary harts has failed */ + beqz a0, 1f + mv a1, a0 + la a0, secondary_harts_relocation_error + jal printf + jal hang + /* restore relocation offset */ - mv t6, s5 +1: mv t6, s5 #endif /*
Signature · 1 line
2.20.1
PATCH v3 9/11 riscv: hang if relocation of secondary harts fails
Quoted 7 lines · Lukas Auer
Print an error message and hang if
smp_call_function()returns an error, indicating that relocation of the secondary harts has failed.Signed-off-by Lukas Auer <[email protected]>---
Quoted 8 lines
Changes in v3:
- New patch to hang if relocation of secondary harts failsChanges in v2: None
arch/riscv/cpu/start.S | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
PATCH v3 9/11 riscv: hang if relocation of secondary harts fails
In reply to Lukas AuerQuoted 60 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Sunday, March 17, 2019 11:59 PM
To: u-boot at lists.denx.de
Cc: Anup Patel <[email protected]>; Andreas Schwab <[email protected]>; Bin Meng <[email protected]>; Atish Patra <[email protected]>; Palmer Dabbelt <[email protected]>; Lukas Auer <[email protected]>; Atish Patra <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>
Subject: [PATCH v3 09/11] riscv: hang if relocation of secondary harts failsPrint an error message and hang if
smp_call_function()returns an error, indicating that relocation of the secondary harts has failed.Signed-off-by Lukas Auer <[email protected]>---
Changes in v3:
- New patch to hang if relocation of secondary harts failsChanges in v2: None
arch/riscv/cpu/start.S | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/riscv/cpu/start.S index renamed from arch/riscv/cpu/start.S +0
-0
5ac899b141..a4433fbd6b100644arch/riscv/cpu/start.S +12
-2@@ -33,6 +33,10 @@ #define SYM_SIZE 0x18 #endif +.section .data +secondary_harts_relocation_error: + .ascii "Relocation of secondary harts has failed, error %d\n" + .section .text .globl _start _start:@@ -274,8 +278,15 @@ relocate_secondary_harts: mv a2, s3 jal smp_call_function + /* hang if relocation of secondary harts has failed */ + beqz a0, 1f + mv a1, a0 + la a0, secondary_harts_relocation_error + jal printf + jal hang + /* restore relocation offset */ - mv t6, s5 +1: mv t6, s5 #endif /* --
Quoted 1 line
2.20.1
PATCH v3 10/11 riscv: fu540: enable SMP
In reply to Lukas Auer---
Changes in v3: None
Changes in v2:
- New patch to enable SMP on the SiFive FU540, which was previously sent independently
board/sifive/fu540/Kconfig | 1 + 1 file changed, 1 insertion(+)
board/sifive/fu540/Kconfig +1 -0
@@ -38,5 +38,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy imply PHY_LIB imply PHY_MSCC imply SIFIVE_SERIAL + imply SMP endif
Signature · 1 line
2.20.1
PATCH v3 11/11 riscv: qemu: enable SMP
In reply to Lukas Auer---
Changes in v3: None
Changes in v2: None
board/emulation/qemu-riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)
board/emulation/qemu-riscv/Kconfig +1 -0
@@ -34,5 +34,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy imply BOARD_LATE_INIT imply OF_BOARD_SETUP imply SIFIVE_SERIAL + imply SMP endif
Signature · 1 line
2.20.1
v3 cover SMP support for RISC-V
In reply to Lukas AuerQuoted 6 lines · lukas.auer at aisec.fraunhofer.de
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
Quoted 7 lines
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration. Patch [1] has been dropped and will be replaced with a U-Boot patch.
As far as I understand it we're taking a different approach here, so this patch won't be going in.
Quoted 1 line
This should be in rc0, LMK if I screwed something up.
Thanks for the patches, and also for testing on the board :)
Quoted 53 lines
Changes in v3:
- Print error ifriscv_send_ipi()fails
- Adjust error message for failures ofriscv_clear_ipi()to match error message for failures ofriscv_send_ipi()
- New patch to save the hart ID in register tp instead of s0
- Adjust patch to use the new location of the hart ID (register tp)
- New patch to hang if relocation of secondary harts failsChanges in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independentlyLukas Auer (11):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: save hart ID in register tp instead of s0
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: hang if relocation of secondary harts fails
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 167 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 + arch/riscv/include/asm/sbi.h | 94 +++++++++++++++ arch/riscv/include/asm/smp.h | 53 +++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 ++++ arch/riscv/lib/smp.c | 118 +++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 507 insertions(+), 12 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c
Auer
v3 cover SMP support for RISC-V
Quoted 26 lines · Palmer Dabbelt
Quoted 18 lines · lukas.auer at aisec.fraunhofer.de
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for
hardware thread). Images passed to bootm will be started on all harts.
The bootm command is currently the only one that will boot images on all
harts, bootefi is not yet supported.The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using
BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.
Patch [1] has been dropped and will be replaced with a U-Boot patch.Quoted 1 line
As far as I understand it we're taking a different approach here, so this patch
won't be going in.
Quoted 6 lines
Quoted 1 line
This should be in rc0, LMK if I screwed something up.
Thanks for the patches, and also for testing on the board :)
Everything looks good, thanks for merging the patch!
U-Boot with the SMP patches now runs in supervisor mode on mainline QEMU. I will send a U-Boot patch later this week to get it running in machine mode on mainline QEMU as well.
Thanks,
Lukas
Quoted 58 lines
Quoted 58 lines
Changes in v3:
- Print error ifriscv_send_ipi()fails
- Adjust error message for failures ofriscv_clear_ipi()to match error
message for failures ofriscv_send_ipi()
- New patch to save the hart ID in register tp instead of s0
- Adjust patch to use the new location of the hart ID (register tp)
- New patch to hang if relocation of secondary harts failsChanges in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available
harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid
CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent
independentlyLukas Auer (11):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: save hart ID in register tp instead of s0
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: hang if relocation of secondary harts fails
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 167 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 + arch/riscv/include/asm/sbi.h | 94 +++++++++++++++ arch/riscv/include/asm/smp.h | 53 +++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 ++++ arch/riscv/lib/smp.c | 118 +++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 507 insertions(+), 12 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c
v3 cover SMP support for RISC-V
In reply to Lukas AuerQuoted 9 lines · Lukas Auer
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Can you describe the test configuration and boot flow a little more, or post an SDcard image that boots with the switches as shown in the readme at [1]
I don't see any board-specific memory initialization code anywhere, so I assume you are still using the original SiFive FSBL, and not the u-boot version that includes the memory init code [2]
[1] https://github.com/sifive/freedom-u-sdk
[2] https://github.com/sifive/HiFive_U-Boot
Quoted 67 lines
Mainline QEMU requires two patches [1, 2] to run in this configuration. Patch [1] has been dropped and will be replaced with a U-Boot patch.
[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/Changes in v3:
- Print error ifriscv_send_ipi()fails
- Adjust error message for failures ofriscv_clear_ipi()to match error message for failures ofriscv_send_ipi()
- New patch to save the hart ID in register tp instead of s0
- Adjust patch to use the new location of the hart ID (register tp)
- New patch to hang if relocation of secondary harts failsChanges in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independentlyLukas Auer (11):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: save hart ID in register tp instead of s0
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: hang if relocation of secondary harts fails
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 167 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 + arch/riscv/include/asm/sbi.h | 94 +++++++++++++++ arch/riscv/include/asm/smp.h | 53 +++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 ++++ arch/riscv/lib/smp.c | 118 +++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 507 insertions(+), 12 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.cSignature · 6 lines
2.20.1 _______________________________________________ U-Boot mailing list U-Boot at lists.denx.de https://lists.denx.de/listinfo/u-boot
v3 cover SMP support for RISC-V
Quoted 12 lines · Troy Benjegerdes
Quoted 9 lines · Lukas Auer
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Quoted 7 lines
Can you describe the test configuration and boot flow a little more, or post an SDcard image that boots with the switches as shown in the readme at [1]
I don't see any board-specific memory initialization code anywhere, so I assume you are still using the original SiFive FSBL, and not the u-boot version that includes the memory init code [2]
Correct. Mainline U-Boot on the FU540 board runs on S-mode currently. You can refer to documentation doc/README.sifive-fu540
Quoted 3 lines
[1] https://github.com/sifive/freedom-u-sdk
[2] https://github.com/sifive/HiFive_U-Boot
Regards,
Bin
v3 cover SMP support for RISC-V
Quoted 16 lines · Bin Meng
Quoted 15 lines · Troy Benjegerdes
Quoted 9 lines · Lukas Auer
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Can you describe the test configuration and boot flow a little more, or post an SDcard image that boots with the switches as shown in the readme at [1]
Quoted 8 lines
Quoted 4 lines
I don't see any board-specific memory initialization code anywhere, so I assume you are still using the original SiFive FSBL, and not the u-boot version that includes the memory init code [2]
Correct. Mainline U-Boot on the FU540 board runs on S-mode currently. You can refer to documentation doc/README.sifive-fu540
To add Bin's comment, the boot flow described in README.sifive-fu540 is
ZSBL->FSBL->OpenSBI/BBL->U-Boot(S Mode)->Linux
FYI: For BBL, you need to hack U-Boot to add serial console in DT.
IMHO, we should avoid forks and use upstream code as much as possible.
The memory initialization patches in [2] were never upstreamed. It would be great if that can upstream them so that we can replace FSBL as well. U-Boot SPL support for RISC-V would even be better.
Regards,
Atish
Quoted 11 lines
Quoted 3 lines
[1] https://github.com/sifive/freedom-u-sdk
[2] https://github.com/sifive/HiFive_U-BootRegards,
Bin
_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot
v3 cover SMP support for RISC-V
I think, the boot flow explained above is supported the sifive fu540 board default boot mode.
if we want to select debug mode, then, does this patches work with the u-boot to TFTP booting(dhcp, bootp) in debug mode?
Regards
Padmarao
Quoted 63 lines · Atish Patra
Quoted 1 line · Bin Meng
On Thu, Mar 21, 2019 at 11:39 PM Troy Benjegerdes <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines · Lukas Auer
Quoted 4 lines
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on
all
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode),
using
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
BBL and OpenSBI.
Can you describe the test configuration and boot flow a little more, or
post an
Quoted 3 lines
Quoted 3 lines
SDcard image that boots with the switches as shown in the readme at [1]
I don't see any board-specific memory initialization code anywhere, so
I assume
Quoted 1 line
Quoted 1 line
you are still using the original SiFive FSBL, and not the u-boot
version that
Quoted 6 lines
Quoted 2 lines
includes the memory init code [2]
Correct. Mainline U-Boot on the FU540 board runs on S-mode currently. You can refer to documentation doc/README.sifive-fu540
To add Bin's comment, the boot flow described in README.sifive-fu540 is
ZSBL->FSBL->OpenSBI/BBL->U-Boot(S Mode)->Linux
FYI: For BBL, you need to hack U-Boot to add serial console in DT.
IMHO, we should avoid forks and use upstream code as much as possible.
The memory initialization patches in [2] were never upstreamed. It would be great if that can upstream them so that we can replace FSBL as well. U-Boot SPL support for RISC-V would even be better.
Regards,
AtishQuoted 11 lines
Quoted 3 lines
[1] https://github.com/sifive/freedom-u-sdk
[2] https://github.com/sifive/HiFive_U-BootRegards,
Bin
_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot