SMP support for RISC-V
Subsystems (2)
v2 cover SMP support for RISC-V
This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.
[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/
Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available harts as indicated by the available_harts mask
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independently
Lukas Auer (9):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: fu540: enable SMP
riscv: qemu: enable SMP
arch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 152 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/sbi.h | 94 +++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 +++++ arch/riscv/lib/smp.c | 116 ++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 492 insertions(+), 10 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c
Signature · 1 line
2.20.1
PATCH v2 1/9 riscv: add infrastructure for calling functions on other harts
Harts on RISC-V boot independently, U-Boot is responsible for managing them. Functions are called on other harts with smp_call_function(), which sends inter-processor interrupts (IPIs) to all other available harts. Available harts are those marked as available in the device tree and present in the available_harts mask stored in global data. The available_harts mask is used to register all harts that have entered U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handler handle_ipi() must be called on software interrupts to handle the request and call the specified function.
Functions are stored in the ipi_data data structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks.
---
Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available harts as indicated by the available_harts mask
arch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 116 +++++++++++++++++++++++++++ 5 files changed, 195 insertions(+) create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/smp.c
arch/riscv/Kconfig +19 -0
@@ -120,4 +120,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN default 0x1000 +config SMP + bool "Symmetric Multi-Processing" + help + This enables support for systems with more than one CPU. If + you say N here, U-Boot will run on single and multiprocessor + machines, but will use only one CPU of a multiprocessor + machine. If you say Y here, U-Boot will run on many, but not + all, single processor machines. + +config NR_CPUS + int "Maximum number of CPUs (2-32)" + range 2 32 + depends on SMP + default 8 + help + On multiprocessor machines, U-Boot sets up a stack for each CPU. + Stack memory is pre-allocated. U-Boot must therefore know the + maximum number of CPUs that may be present. + endmenu
arch/riscv/include/asm/global_data.h +6 -0
@@ -10,12 +10,18 @@#ifndef __ASM_GBL_DATA_H#define __ASM_GBL_DATA_H +#include <asm/smp.h> +/* Architecture-specific global data */ struct arch_global_data {long boot_hart;/* boot hart id */ #ifdef CONFIG_SIFIVE_CLINT void __iomem *clint;/* clint base address */ #endif +#ifdef CONFIG_SMP +struct ipi_data ipi[CONFIG_NR_CPUS ]; +#endif +ulong available_harts; };#include <asm-generic/global_data.h>
arch/riscv/include/asm/smp.h new file +53 -0
53 lines not on this page · raw message
arch/riscv/lib/Makefile +1 -0
@@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o obj-y += setjmp.o +obj-$(CONFIG_SMP) += smp.o # For building EFI apps CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)
arch/riscv/lib/smp.c new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v2 1/9 riscv: add infrastructure for calling functions on other harts
Quoted 31 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; Andreas
Schwab <[email protected]>; Palmer Dabbelt <[email protected]>; Alexander Graf <[email protected]>; Lukas Auer <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>
Subject: [PATCH v2 1/9] riscv: add infrastructure for calling functions on other hartsHarts on RISC-V boot independently, U-Boot is responsible for managing them. Functions are called on other harts with
smp_call_function(), which sends inter-processor interrupts (IPIs) to all other available harts. Available harts are those marked as available in the device tree and present in theavailable_hartsmask stored in global data. Theavailable_hartsmask is used to register all harts that have entered U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handlerhandle_ipi()must be called on software interrupts to handle the request and call the specified function.Functions are stored in the
ipi_datadata structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks.
Quoted 8 lines
Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
Regards,
Anup
PATCH v2 1/9 riscv: add infrastructure for calling functions on other harts
In reply to Lukas AuerQuoted 242 lines · Lukas Auer
Harts on RISC-V boot independently, U-Boot is responsible for managing them. Functions are called on other harts with
smp_call_function(), which sends inter-processor interrupts (IPIs) to all other available harts. Available harts are those marked as available in the device tree and present in theavailable_hartsmask stored in global data. Theavailable_hartsmask is used to register all harts that have entered U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handlerhandle_ipi()must be called on software interrupts to handle the request and call the specified function.Functions are stored in the
ipi_datadata structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmaskarch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 116 +++++++++++++++++++++++++++ 5 files changed, 195 insertions(+)create mode 100644
arch/riscv/include/asm/smp.h
create mode 100644arch/riscv/lib/smp.carch/riscv/Kconfig +19
-0@@ -120,4 +120,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN default 0x1000 +config SMP + bool "Symmetric Multi-Processing" + help + This enables support for systems with more than one CPU. If + you say N here, U-Boot will run on single and multiprocessor + machines, but will use only one CPU of a multiprocessor + machine. If you say Y here, U-Boot will run on many, but not + all, single processor machines. + +config NR_CPUS + int "Maximum number of CPUs (2-32)" + range 2 32 + depends on SMP + default 8 + help + On multiprocessor machines, U-Boot sets up a stack for each CPU. + Stack memory is pre-allocated. U-Boot must therefore know the + maximum number of CPUs that may be present. + endmenuarch/riscv/include/asm/global_data.h +6
-0@@ -10,12 +10,18 @@#ifndef __ASM_GBL_DATA_H#define __ASM_GBL_DATA_H +#include <asm/smp.h> +/* Architecture-specific global data */ struct arch_global_data {long boot_hart;/* boot hart id */ #ifdef CONFIG_SIFIVE_CLINT void __iomem *clint;/* clint base address */ #endif +#ifdef CONFIG_SMP +struct ipi_data ipi[CONFIG_NR_CPUS ]; +#endif +ulong available_harts; };#include <asm-generic/global_data.h> arch/riscv/include/asm/smp.h new file +53
-0@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#ifndef _ASM_RISCV_SMP_H +#define _ASM_RISCV_SMP_H + +/** +* struct ipi_data - Inter-processor interrupt (IPI) data structure +* +* IPIs are used for SMP support to communicate to other harts what function to +* call. Functions are in the form +* void (*addr)(ulong hart, ulong arg0, ulong arg1). +* +* The function address and the two arguments, arg0 and arg1, are stored in the +* IPI data structure. The hart ID is inserted by the hart handling the IPI and +* calling the function. +* +* @addr: Address of function +* @arg0: First argument of function +* @arg1: Second argument of function +*/ +struct ipi_data { +ulong addr; +ulong arg0; +ulong arg1; +}; + +/** +* handle_ipi() - interrupt handler for software interrupts +* +* The IPI interrupt handler must be called to handle software interrupts. It +* calls the function specified in the hart's IPI data structure. +* +* @hart: Hart ID of the current hart +*/ +void handle_ipi (ulong hart); + +/** +* smp_call_function() - Call a function on all other harts +* +* Send IPIs with the specified function call to all harts. +* +* @addr: Address of function +* @arg0: First argument of function +* @arg1: Second argument of function +* @return 0 if OK, -ve on error +*/ +int smp_call_function (ulong addr,ulong arg0,ulong arg1); + +#endif arch/riscv/lib/Makefile +1
-0@@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o obj-y += setjmp.o +obj-$(CONFIG_SMP) += smp.o # For building EFI apps CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)arch/riscv/lib/smp.c new file +79
-0@@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#include <common.h> +#include <dm.h> +#include <asm/barrier.h> +#include <asm/smp.h> + +DECLARE_GLOBAL_DATA_PTR ; + +/** +* riscv_send_ipi() - Send inter-processor interrupt (IPI) +* +* Platform code must provide this function. +* +* @hart: Hart ID of receiving hart +* @return 0 if OK, -ve on error +*/ +extern int riscv_send_ipi (int hart); + +/** +* riscv_clear_ipi() - Clear inter-processor interrupt (IPI) +* +* Platform code must provide this function. +* +* @hart: Hart ID of hart to be cleared +* @return 0 if OK, -ve on error +*/ +extern int riscv_clear_ipi (int hart); + +static int send_ipi_many (struct ipi_data *ipi) +{ +ofnode node, cpus; +u32 reg; +int ret; + + cpus =ofnode_path ("/cpus" ); +if (!ofnode_valid (cpus)) { +pr_err ("Can't find cpus node!\n" ); +return -EINVAL ; + } + +ofnode_for_each_subnode (node, cpus) { +/* skip if hart is marked as not available in the device tree */ +if (!ofnode_is_available (node)) +continue ; + +/* read hart ID of CPU */ + ret =ofnode_read_u32 (node,"reg" , ®); +if (ret) +continue ; + +/* skip if it is the hart we are running on */ +if (reg == gd->arch.boot_hart) +continue ; + +if (reg >=CONFIG_NR_CPUS ) { +pr_err ("Hart ID %d is out of range, increase CONFIG_NR_CPUS\n" , + reg); + continue + } + +/* skip if hart is not available */ +if (!(gd->arch.available_harts & (1 << reg))) +continue ; + + gd->arch.ipi[reg].addr = ipi->addr; + gd->arch.ipi[reg].arg0 = ipi->arg0; + gd->arch.ipi[reg].arg1 = ipi->arg1; + + ret =riscv_send_ipi (reg); +if (ret) +return ret; + } + +return 0 ;
arch/riscv/lib/smp.c line 89
Quoted 10 lines
arch/riscv/lib/smp.c new file +10
-0+} + +void handle_ipi (ulong hart) +{ +int ret; +void (*smp_function)(ulong hart,ulong arg0,ulong arg1); + +if (hart >=CONFIG_NR_CPUS ) +return ; +
A warning here may help.
arch/riscv/lib/smp.c line 95
Quoted 6 lines
arch/riscv/lib/smp.c new file +6
-0+ ret =riscv_clear_ipi (hart); +if (ret) { +pr_err ("Cannot clear IPI\n" ); +return ; + } +
Currently, sbi_clear_ipi() doesn't return anything and riscv_clear_ipi() is always returning 0. I guess you kept the check here so that we don't have make changes in future. But IMHO, we should add it if SBI spec is changed so that sbi_clear_ipi can return errors.
Regards,
Atish
Quoted 22 lines
arch/riscv/lib/smp.c new file +21
-0+__smp_mb (); + + smp_function = (void (*)(ulong ,ulong ,ulong ))gd->arch.ipi[hart].addr; +invalidate_icache_all (); + +smp_function (hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1); +} + +int smp_call_function (ulong addr,ulong arg0,ulong arg1) +{ +int ret =0 ; +struct ipi_data ipi; + + ipi.addr = addr; + ipi.arg0 = arg0; + ipi.arg1 = arg1; + + ret =send_ipi_many (&ipi); + +return ret; +}
Auer
PATCH v2 1/9 riscv: add infrastructure for calling functions on other harts
Quoted 270 lines · Atish Patra
Quoted 269 lines · Lukas Auer
Harts on RISC-V boot independently, U-Boot is responsible for managing
them. Functions are called on other harts withsmp_call_function(), which sends inter-processor interrupts (IPIs) to all other available
harts. Available harts are those marked as available in the device tree
and present in theavailable_hartsmask stored in global data. Theavailable_hartsmask is used to register all harts that have entered
U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the
hart ID of the hart calling the function. On the other harts, the IPI
interrupt handlerhandle_ipi()must be called on software interrupts to
handle the request and call the specified function.Functions are stored in the
ipi_datadata structure. Every hart has its
own data structure in global data. While this is not required at the
moment (all harts are expected to boot Linux), this does allow future
expansion, where other harts may be used for monitoring or other tasks.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available
harts as indicated by theavailable_hartsmaskarch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 116 +++++++++++++++++++++++++++ 5 files changed, 195 insertions(+)create mode 100644
arch/riscv/include/asm/smp.h
create mode 100644arch/riscv/lib/smp.carch/riscv/Kconfig +16
-0@@ -120,4 +120,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN default 0x1000 +config SMP + bool "Symmetric Multi-Processing" + help + This enables support for systems with more than one CPU. If + you say N here, U-Boot will run on single and multiprocessor + machines, but will use only one CPU of a multiprocessor + machine. If you say Y here, U-Boot will run on many, but not + all, single processor machines. + +config NR_CPUS + int "Maximum number of CPUs (2-32)" + range 2 32 + depends on SMP + default 8 + help + On multiprocessor machines, U-Boot sets up a stack for eachCPU.
arch/riscv/Kconfig +3
-0+ Stack memory is pre-allocated. U-Boot must therefore know the + maximum number of CPUs that may be present. + endmenuarch/riscv/include/asm/global_data.h +6
-0@@ -10,12 +10,18 @@#ifndef __ASM_GBL_DATA_H#define __ASM_GBL_DATA_H +#include <asm/smp.h> +/* Architecture-specific global data */ struct arch_global_data {long boot_hart;/* boot hart id */ #ifdef CONFIG_SIFIVE_CLINT void __iomem *clint;/* clint base address */ #endif +#ifdef CONFIG_SMP +struct ipi_data ipi[CONFIG_NR_CPUS ]; +#endif +ulong available_harts; };#include <asm-generic/global_data.h> arch/riscv/include/asm/smp.h new file +11
-0@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#ifndef _ASM_RISCV_SMP_H +#define _ASM_RISCV_SMP_H + +/** +* struct ipi_data - Inter-processor interrupt (IPI) data structure
+ *
+ * IPIs are used for SMP support to communicate to other harts what function to
+ * call. Functions are in the form
+ * void (*addr)(ulong hart, ulong arg0, ulong arg1).
+ *
+ * The function address and the two arguments, arg0 and arg1, are stored in the
+ * IPI data structure. The hart ID is inserted by the hart handling the IPI and
+ * calling the function.
+ *
+ * @addr: Address of function
+ * @arg0: First argument of function
+ * @arg1: Second argument of function
+ */arch/riscv/include/asm/smp.h new file +10
-0+struct ipi_data { +ulong addr; +ulong arg0; +ulong arg1; +}; + +/** +* handle_ipi() - interrupt handler for software interrupts +* +* The IPI interrupt handler must be called to handle software interrupts. It
+ * calls the function specified in the hart's IPI data structure.
+ *
+ * @hart: Hart ID of the current hart
+ */arch/riscv/include/asm/smp.h new file +15
-0+void handle_ipi (ulong hart); + +/** +* smp_call_function() - Call a function on all other harts +* +* Send IPIs with the specified function call to all harts. +* +* @addr: Address of function +* @arg0: First argument of function +* @arg1: Second argument of function +* @return 0 if OK, -ve on error +*/ +int smp_call_function (ulong addr,ulong arg0,ulong arg1); + +#endif arch/riscv/lib/Makefile +1
-0@@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o obj-y += setjmp.o +obj-$(CONFIG_SMP) += smp.o # For building EFI apps CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)arch/riscv/lib/smp.c new file +47
-0@@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#include <common.h> +#include <dm.h> +#include <asm/barrier.h> +#include <asm/smp.h> + +DECLARE_GLOBAL_DATA_PTR ; + +/** +* riscv_send_ipi() - Send inter-processor interrupt (IPI) +* +* Platform code must provide this function. +* +* @hart: Hart ID of receiving hart +* @return 0 if OK, -ve on error +*/ +extern int riscv_send_ipi (int hart); + +/** +* riscv_clear_ipi() - Clear inter-processor interrupt (IPI) +* +* Platform code must provide this function. +* +* @hart: Hart ID of hart to be cleared +* @return 0 if OK, -ve on error +*/ +extern int riscv_clear_ipi (int hart); + +static int send_ipi_many (struct ipi_data *ipi) +{ +ofnode node, cpus; +u32 reg; +int ret; + + cpus =ofnode_path ("/cpus" ); +if (!ofnode_valid (cpus)) { +pr_err ("Can't find cpus node!\n" ); +return -EINVAL ; + } + +ofnode_for_each_subnode (node, cpus) { +/* skip if hart is marked as not available in the device tree */
arch/riscv/lib/smp.c new file +14
-0+if (!ofnode_is_available (node)) +continue ; + +/* read hart ID of CPU */ + ret =ofnode_read_u32 (node,"reg" , ®); +if (ret) +continue ; + +/* skip if it is the hart we are running on */ +if (reg == gd->arch.boot_hart) +continue ; + +if (reg >=CONFIG_NR_CPUS ) { + pr_err("Hart ID %d is out of range, increaseCONFIG_NR_CPUS\n",
arch/riscv/lib/smp.c new file +22
-0+ reg); + continue + } + +/* skip if hart is not available */ +if (!(gd->arch.available_harts & (1 << reg))) +continue ; + + gd->arch.ipi[reg].addr = ipi->addr; + gd->arch.ipi[reg].arg0 = ipi->arg0; + gd->arch.ipi[reg].arg1 = ipi->arg1; + + ret =riscv_send_ipi (reg); +if (ret) +return ret; + } + +return 0 ; +} + +void handle_ipi (ulong hart) +{
Quoted 9 lines
arch/riscv/lib/smp.c line 89
Quoted 6 lines
arch/riscv/lib/smp.c new file +6
-0+int ret; +void (*smp_function)(ulong hart,ulong arg0,ulong arg1); + +if (hart >=CONFIG_NR_CPUS ) +return ; +A warning here may help.
This is meant more as a sanity check. Harts with an ID greater than CONFIG_NR_CPUS should never run handle_ipi().
Warnings are already printed by the hart sending the IPIs, in send_ipi_many() called by smp_call_function().
Quoted 7 lines
arch/riscv/lib/smp.c line 95
Quoted 6 lines
arch/riscv/lib/smp.c new file +6
-0+ ret =riscv_clear_ipi (hart); +if (ret) { +pr_err ("Cannot clear IPI\n" ); +return ; + } +
Quoted 8 lines
Currently,
sbi_clear_ipi()doesn't return anything andriscv_clear_ipi()
is always returning 0. I guess you kept the check here so that we don't
have make changes in future. But IMHO, we should add it if SBI spec is
changed so thatsbi_clear_ipican return errors.
The check is required if U-Boot is running in machine mode. The IPI implementation using the CLINT driver returns an error if it cannot get the memory address of the CLINT device.
I just noticed that no error is printed if riscv_send_ipi() returns an error in send_ipi_many(). I will add one in the next version.
Thanks,
Lukas
Quoted 26 lines
Regards,
AtishQuoted 24 lines
arch/riscv/lib/smp.c new file +3
-0+__smp_mb (); + + smp_function = (void (*)(ulong, ulong, ulong))gd-Quoted 1 line
arch.ipi[hart].addr;
arch/riscv/lib/smp.c new file +3
-0+invalidate_icache_all (); + + smp_function(hart, gd->arch.ipi[hart].arg0, gd-Quoted 1 line
arch.ipi[hart].arg1);
arch/riscv/lib/smp.c new file +15
-0+} + +int smp_call_function (ulong addr,ulong arg0,ulong arg1) +{ +int ret =0 ; +struct ipi_data ipi; + + ipi.addr = addr; + ipi.arg0 = arg0; + ipi.arg1 = arg1; + + ret =send_ipi_many (&ipi); + +return ret; +}
PATCH v2 1/9 riscv: add infrastructure for calling functions on other harts
In reply to Lukas AuerQuoted 27 lines · Lukas Auer
Harts on RISC-V boot independently, U-Boot is responsible for managing them. Functions are called on other harts with
smp_call_function(), which sends inter-processor interrupts (IPIs) to all other available harts. Available harts are those marked as available in the device tree and present in theavailable_hartsmask stored in global data. Theavailable_hartsmask is used to register all harts that have entered U-Boot. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handlerhandle_ipi()must be called on software interrupts to handle the request and call the specified function.Functions are stored in the
ipi_datadata structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
Quoted 9 lines
arch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 116 +++++++++++++++++++++++++++ 5 files changed, 195 insertions(+) create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/smp.c
PATCH v2 2/9 riscv: import the supervisor binary interface header file
In reply to Lukas AuerImport the supervisor binary interface (SBI) header file from Linux (arch/riscv/include/asm/sbi.h). The last change to it was in commit 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI").
---
Changes in v2: None
arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 arch/riscv/include/asm/sbi.h
arch/riscv/include/asm/sbi.h new file +94 -0
@@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015 Regents of the University of California +* +* Taken from Linux arch/riscv/include/asm/sbi.h +*/ + +#ifndef _ASM_RISCV_SBI_H +#define _ASM_RISCV_SBI_H + +#include <linux/types.h> + +#define SBI_SET_TIMER 0 +#define SBI_CONSOLE_PUTCHAR 1 +#define SBI_CONSOLE_GETCHAR 2 +#define SBI_CLEAR_IPI 3 +#define SBI_SEND_IPI 4 +#define SBI_REMOTE_FENCE_I 5 +#define SBI_REMOTE_SFENCE_VMA 6 +#define SBI_REMOTE_SFENCE_VMA_ASID 7 +#define SBI_SHUTDOWN 8 + +#define SBI_CALL (which, arg0, arg1, arg2) ({ \ + register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ + register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ + register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ + register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ + asm volatile ("ecall" \ + : "+r" (a0) \ + : "r" (a1), "r" (a2), "r" (a7) \ + : "memory"); \ + a0; \ +}) + +/* Lazy implementations until SBI is finalized */ +#define SBI_CALL_0 (which) SBI_CALL(which, 0, 0, 0) +#define SBI_CALL_1 (which, arg0) SBI_CALL(which, arg0, 0, 0) +#define SBI_CALL_2 (which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0) + +static inline void sbi_console_putchar (int ch) +{ +SBI_CALL_1 (SBI_CONSOLE_PUTCHAR , ch); +} + +static inline int sbi_console_getchar (void ) +{ +return SBI_CALL_0 (SBI_CONSOLE_GETCHAR ); +} + +static inline void sbi_set_timer (uint64_t stime_value) +{ +#if __riscv_xlen ==32 +SBI_CALL_2 (SBI_SET_TIMER , stime_value, stime_value >>32 ); +#else +SBI_CALL_1 (SBI_SET_TIMER , stime_value); +#endif +} + +static inline void sbi_shutdown (void ) +{ +SBI_CALL_0 (SBI_SHUTDOWN ); +} + +static inline void sbi_clear_ipi (void ) +{ +SBI_CALL_0 (SBI_CLEAR_IPI ); +} + +static inline void sbi_send_ipi (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_SEND_IPI , hart_mask); +} + +static inline void sbi_remote_fence_i (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_REMOTE_FENCE_I , hart_mask); +} + +static inline void sbi_remote_sfence_vma (const unsigned long *hart_mask, +unsigned long start, +unsigned long size) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA , hart_mask); +} + +static inline void sbi_remote_sfence_vma_asid (const unsigned long *hart_mask, +unsigned long start, +unsigned long size, +unsigned long asid) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA_ASID , hart_mask); +} + +#endif
Signature · 1 line
2.20.1
PATCH v2 2/9 riscv: import the supervisor binary interface header file
Quoted 106 lines · Lukas Auer
Import the supervisor binary interface (SBI) header file from Linux (
arch/riscv/include/asm/sbi.h). The last change to it was in commit6d60b6ee0c97("RISC-V: Device, timer, IRQs, and the SBI").Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Bin Meng <[email protected]>---
Changes in v2: None
arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+)create mode 100644
arch/riscv/include/asm/sbi.harch/riscv/include/asm/sbi.h new file +84
-0@@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015 Regents of the University of California +* +* Taken from Linux arch/riscv/include/asm/sbi.h +*/ + +#ifndef _ASM_RISCV_SBI_H +#define _ASM_RISCV_SBI_H + +#include <linux/types.h> + +#define SBI_SET_TIMER 0 +#define SBI_CONSOLE_PUTCHAR 1 +#define SBI_CONSOLE_GETCHAR 2 +#define SBI_CLEAR_IPI 3 +#define SBI_SEND_IPI 4 +#define SBI_REMOTE_FENCE_I 5 +#define SBI_REMOTE_SFENCE_VMA 6 +#define SBI_REMOTE_SFENCE_VMA_ASID 7 +#define SBI_SHUTDOWN 8 + +#define SBI_CALL (which, arg0, arg1, arg2) ({ \ + register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ + register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ + register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ + register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ + asm volatile ("ecall" \ + : "+r" (a0) \ + : "r" (a1), "r" (a2), "r" (a7) \ + : "memory"); \ + a0; \ +}) + +/* Lazy implementations until SBI is finalized */ +#define SBI_CALL_0 (which) SBI_CALL(which, 0, 0, 0) +#define SBI_CALL_1 (which, arg0) SBI_CALL(which, arg0, 0, 0) +#define SBI_CALL_2 (which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0) + +static inline void sbi_console_putchar (int ch) +{ +SBI_CALL_1 (SBI_CONSOLE_PUTCHAR , ch); +} + +static inline int sbi_console_getchar (void ) +{ +return SBI_CALL_0 (SBI_CONSOLE_GETCHAR ); +} + +static inline void sbi_set_timer (uint64_t stime_value) +{ +#if __riscv_xlen ==32 +SBI_CALL_2 (SBI_SET_TIMER , stime_value, stime_value >>32 ); +#else +SBI_CALL_1 (SBI_SET_TIMER , stime_value); +#endif +} + +static inline void sbi_shutdown (void ) +{ +SBI_CALL_0 (SBI_SHUTDOWN ); +} + +static inline void sbi_clear_ipi (void ) +{ +SBI_CALL_0 (SBI_CLEAR_IPI ); +} + +static inline void sbi_send_ipi (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_SEND_IPI , hart_mask); +} + +static inline void sbi_remote_fence_i (const unsigned long *hart_mask) +{ +SBI_CALL_1 (SBI_REMOTE_FENCE_I , hart_mask); +} + +static inline void sbi_remote_sfence_vma (const unsigned long *hart_mask, +unsigned long start, +unsigned long size) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA , hart_mask); +}
arch/riscv/include/asm/sbi.h line 94
Quoted 10 lines
arch/riscv/include/asm/sbi.h new file +10
-0+ +static inline void sbi_remote_sfence_vma_asid (const unsigned long *hart_mask, +unsigned long start, +unsigned long size, +unsigned long asid) +{ +SBI_CALL_1 (SBI_REMOTE_SFENCE_VMA_ASID , hart_mask); +} + +#endif
Regards,
Atish
PATCH v2 3/9 riscv: implement IPI platform functions using SBI
In reply to Lukas AuerThe supervisor binary interface (SBI) provides the necessary functions to implement the platform IPI functions riscv_send_ipi() and riscv_clear_ipi(). Use it to implement them.
This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs running in supervisor mode. Support for machine mode is already available for CPUs that include the SiFive CLINT.
---
Changes in v2: None
arch/riscv/Kconfig | 5 +++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+) create mode 100644 arch/riscv/lib/sbi_ipi.c
arch/riscv/Kconfig +5 -0
@@ -139,4 +139,9 @@ config NR_CPUS Stack memory is pre-allocated. U-Boot must therefore know the maximum number of CPUs that may be present. +config SBI_IPI + bool + default y if RISCV_SMODE + depends on SMP + endmenu
arch/riscv/lib/Makefile +1 -0
@@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o obj-y += setjmp.o obj-$(CONFIG_SMP) += smp.o
arch/riscv/lib/sbi_ipi.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#include <common.h> +#include <asm/sbi.h> + +int riscv_send_ipi (int hart) +{ +ulong mask; + + mask =1UL << hart; +sbi_send_ipi (&mask); + +return 0 ; +} + +int riscv_clear_ipi (int hart) +{ +sbi_clear_ipi (); + +return 0 ; +}
Signature · 1 line
2.20.1
PATCH v2 3/9 riscv: implement IPI platform functions using SBI
Quoted 69 lines · Lukas Auer
The supervisor binary interface (SBI) provides the necessary functions to implement the platform IPI functions
riscv_send_ipi()andriscv_clear_ipi(). Use it to implement them.This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs running in supervisor mode. Support for machine mode is already available for CPUs that include the SiFive CLINT.
Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Bin Meng <[email protected]>---
Changes in v2: None
arch/riscv/Kconfig | 5 +++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+)create mode 100644
arch/riscv/lib/sbi_ipi.carch/riscv/Kconfig +5
-0@@ -139,4 +139,9 @@ config NR_CPUS Stack memory is pre-allocated. U-Boot must therefore know the maximum number of CPUs that may be present. +config SBI_IPI + bool + default y if RISCV_SMODE + depends on SMP + endmenuarch/riscv/lib/Makefile +1
-0@@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o obj-y += setjmp.o obj-$(CONFIG_SMP) += smp.oarch/riscv/lib/sbi_ipi.c new file +15
-0@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (C) 2019 Fraunhofer AISEC, +* Lukas Auer <[email protected]> +*/ + +#include <common.h> +#include <asm/sbi.h> + +int riscv_send_ipi (int hart) +{ +ulong mask; + + mask =1UL << hart; +sbi_send_ipi (&mask);
arch/riscv/lib/sbi_ipi.c line 25
Quoted 10 lines
arch/riscv/lib/sbi_ipi.c new file +10
-0+ +return 0 ; +} + +int riscv_clear_ipi (int hart) +{ +sbi_clear_ipi (); + +return 0 ; +}
LGTM.
Regards,
Atish
PATCH v2 3/9 riscv: implement IPI platform functions using SBI
In reply to Lukas AuerQuoted 14 lines · Lukas Auer
The supervisor binary interface (SBI) provides the necessary functions to implement the platform IPI functions
riscv_send_ipi()andriscv_clear_ipi(). Use it to implement them.This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs running in supervisor mode. Support for machine mode is already available for CPUs that include the SiFive CLINT.
Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Bin Meng <[email protected]>---
Quoted 8 lines
Changes in v2: None
arch/riscv/Kconfig | 5 +++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+) create mode 100644 arch/riscv/lib/sbi_ipi.c
PATCH v2 4/9 riscv: delay initialization of caches and debug UART
In reply to Lukas AuerMove the initialization of the caches and the debug UART until after board_init_f_init_reserve. This is in preparation for SMP support, where code prior to this point will be executed by all harts. This ensures that initialization will only be performed once for the main hart running U-Boot.
---
Changes in v2: None
arch/riscv/cpu/start.S | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/riscv/cpu/start.S +8 -8
@@ -45,10 +45,6 @@ _start: /* mask all interrupts */ csrw MODE_PREFIX(ie), zero - /* Enable cache */ - jal icache_enable - jal dcache_enable - /* * Set stackpointer in internal/ex RAM to call board_init_f */
@@ -57,10 +53,6 @@ call_board_init_f: li t1, CONFIG_SYS_INIT_SP_ADDR and sp, t1, t0 /* force 16 byte alignment */ -#ifdef CONFIG_DEBUG_UART - jal debug_uart_init -#endif - call_board_init_f_0: mv a0, sp jal board_init_f_alloc_reserve
@@ -74,6 +66,14 @@ call_board_init_f_0: /* save the boot hart id to global_data */ SREG s0, GD_BOOT_HART(gp) + /* Enable cache */ + jal icache_enable + jal dcache_enable + +#ifdef CONFIG_DEBUG_UART + jal debug_uart_init +#endif + mv a0, zero /* a0 <-- boot_flags = 0 */ la t5, board_init_f jr t5 /* jump to board_init_f() */
Signature · 1 line
2.20.1
PATCH v2 5/9 riscv: add support for multi-hart systems
In reply to Lukas AuerOn RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to manage all harts in the system. All harts entering U-Boot are registered in the available_harts mask stored in global data. A hart lottery system as used in the Linux kernel selects the hart U-Boot runs on. All other harts are halted. U-Boot can delegate functions to them using smp_call_function().
Every hart has a valid pointer to the global data structure and a 8KiB stack by default. The stack size is set with CONFIG_STACK_SIZE_SHIFT.
---
Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask
arch/riscv/Kconfig | 4 ++ arch/riscv/cpu/cpu.c | 9 ++- arch/riscv/cpu/start.S | 134 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + arch/riscv/lib/asm-offsets.c | 1 + 5 files changed, 147 insertions(+), 2 deletions(-)
arch/riscv/Kconfig +4 -0
@@ -144,4 +144,8 @@ config SBI_IPI default y if RISCV_SMODE depends on SMP +config STACK_SIZE_SHIFT + int + default 13 + endmenu
arch/riscv/cpu/cpu.c +8 -1
@@ -12,10 +12,17 @@#include <dm/uclass-internal.h> /* -* prior_stage_fdt_address must be stored in the data section since it is used +* The variables here must be stored in the data section since they are used * before the bss section is available. */ phys_addr_t prior_stage_fdt_address __attribute__((section(".data"))); +u32 hart_lottery __attribute__((section(".data"))) = 0; + +/* +* The main hart running U-Boot has acquired available_harts_lock until it has +* finished initialization of global data. +*/ +u32 available_harts_lock =1 ;static inline bool supports_extension (char ext) {
arch/riscv/cpu/start.S +133 -1
181 lines not on this page · raw message
arch/riscv/include/asm/csr.h +1 -0
@@ -46,6 +46,7 @@#endif /* Interrupt Enable and Interrupt Pending flags */ +#define MIE_MSIE _AC(0x00000008, UL)/* Software Interrupt Enable */ #define SIE_SSIE _AC(0x00000002, UL)/* Software Interrupt Enable */ #define SIE_STIE _AC(0x00000020, UL)/* Timer Interrupt Enable */
arch/riscv/lib/asm-offsets.c +1 -0
@@ -14,6 +14,7 @@int main (void ) { DEFINE(GD_BOOT_HART, offsetof(gd_t, arch.boot_hart)); + DEFINE(GD_AVAILABLE_HARTS, offsetof(gd_t, arch.available_harts));return 0 ; }
Signature · 1 line
2.20.1
PATCH v2 5/9 riscv: add support for multi-hart systems
Quoted 33 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; Andreas Schwab <[email protected]>; Palmer Dabbelt <[email protected]>; Alexander Graf <[email protected]>; Lukas Auer <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>; Baruch Siach <[email protected]>; Stefan Roese <[email protected]>; Daniel Schwierzeck <[email protected]>
Subject: [PATCH v2 5/9] riscv: add support for multi-hart systemsOn RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to manage all harts in the system. All harts entering U-Boot are registered in the
available_hartsmask stored in global data. A hart lottery system as used in the Linux kernel selects the hart U-Boot runs on. All other harts are halted. U-Boot can delegate functions to them usingsmp_call_function().Every hart has a valid pointer to the global data structure and a 8KiB stack by default. The stack size is set with
CONFIG_STACK_SIZE_SHIFT.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
Quoted 6 lines
arch/riscv/Kconfig | 4 ++ arch/riscv/cpu/cpu.c | 9 ++- arch/riscv/cpu/start.S | 134 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + arch/riscv/lib/asm-offsets.c | 1 + 5 files changed, 147 insertions(+), 2 deletions(-)
Regards,
Anup
PATCH v2 5/9 riscv: add support for multi-hart systems
In reply to Lukas AuerQuoted 20 lines · Lukas Auer
On RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to manage all harts in the system. All harts entering U-Boot are registered in the
available_hartsmask stored in global data. A hart lottery system as used in the Linux kernel selects the hart U-Boot runs on. All other harts are halted. U-Boot can delegate functions to them usingsmp_call_function().Every hart has a valid pointer to the global data structure and a 8KiB stack by default. The stack size is set with
CONFIG_STACK_SIZE_SHIFT.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
Quoted 7 lines
arch/riscv/Kconfig | 4 ++ arch/riscv/cpu/cpu.c | 9 ++- arch/riscv/cpu/start.S | 134 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + arch/riscv/lib/asm-offsets.c | 1 + 5 files changed, 147 insertions(+), 2 deletions(-)
Looks quite good!
Auer
PATCH v2 5/9 riscv: add support for multi-hart systems
Quoted 36 lines · Bin Meng
Quoted 25 lines · Lukas Auer
On RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to
manage all harts in the system. All harts entering U-Boot are registered
in theavailable_hartsmask stored in global data. A hart lottery system
as used in the Linux kernel selects the hart U-Boot runs on. All other
harts are halted. U-Boot can delegate functions to them usingsmp_call_function().Every hart has a valid pointer to the global data structure and a 8KiB
stack by default. The stack size is set withCONFIG_STACK_SIZE_SHIFT.Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmaskQuoted 8 lines
arch/riscv/Kconfig | 4 ++ arch/riscv/cpu/cpu.c | 9 ++- arch/riscv/cpu/start.S | 134 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + arch/riscv/lib/asm-offsets.c | 1 + 5 files changed, 147 insertions(+), 2 deletions(-)
Quoted 2 lines
Looks quite good!
Thank you and thanks for testing and reviewing the series!
Lukas
PATCH v2 6/9 riscv: boot images passed to bootm on all harts
In reply to Lukas Auer---
Changes in v2: None
arch/riscv/lib/bootm.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/riscv/lib/bootm.c +12 -1
@@ -13,6 +13,7 @@#include <image.h> #include <asm/byteorder.h> #include <asm/csr.h> +#include <asm/smp.h> #include <dm/device.h> #include <dm/root.h> #include <u-boot/zlib.h>
@@ -81,6 +82,9 @@ static void boot_jump_linux(bootm_headers_t *images, int flag) {void (*kernel)(ulong hart,void *dtb);int fake = (flag &BOOTM_STATE_OS_FAKE_GO ); +#ifdef CONFIG_SMP +int ret; +#endif kernel = (void (*)(ulong ,void *))images->ep;
@@ -92,8 +96,15 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)announce_and_cleanup (fake);if (!fake) { -if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) +if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) { +#ifdef CONFIG_SMP + ret =smp_call_function (images->ep, + (ulong )images->ft_addr,0 ); +if (ret) +hang (); +#endif kernel (gd->arch.boot_hart, images->ft_addr); + } } }
Signature · 1 line
2.20.1
PATCH v2 6/9 riscv: boot images passed to bootm on all harts
Quoted 11 lines · Lukas Auer
Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Bin Meng <[email protected]>---
Changes in v2: None
arch/riscv/lib/bootm.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
PATCH v2 7/9 riscv: do not rely on hart ID passed by previous boot stage
In reply to Lukas AuerRISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
---
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
arch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/riscv/cpu/start.S +4 -0
@@ -36,6 +36,10 @@ .section .text .globl _start _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif + /* save hart id and dtb pointer */ mv s0, a0 mv s1, a1
Signature · 1 line
2.20.1
PATCH v2 7/9 riscv: do not rely on hart ID passed by previous boot stage
Quoted 50 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; Andreas Schwab <[email protected]>; Palmer Dabbelt <[email protected]>; Alexander Graf <[email protected]>; Lukas Auer <[email protected]>; Rick Chen <[email protected]>; Anup Patel <[email protected]>
Subject: [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stageRISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
Suggested-by Rick Chen <[email protected]>Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-modearch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)arch/riscv/cpu/start.S index renamed from arch/riscv/cpu/start.S +0
-0
79b753847c..d4daa6e0bf100644arch/riscv/cpu/start.S +4
-1@@ -36,6 +36,10 @@ .section .text .globl _start _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif + /* save hart id and dtb pointer */ mv s0, a0 mv s1, a1 --
Quoted 1 line
2.20.1
Regards,
Anup
PATCH v2 7/9 riscv: do not rely on hart ID passed by previous boot stage
In reply to Lukas AuerQuoted 25 lines · Lukas Auer
RISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
Suggested-by Rick Chen <[email protected]>Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-modearch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)arch/riscv/cpu/start.S +0
-0
arch/riscv/cpu/start.S line 45
Quoted 10 lines
arch/riscv/cpu/start.S +4
-0@@ -36,6 +36,10 @@ .section .text .globl _start _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif + /* save hart id and dtb pointer */ mv s0, a0 mv s1, a1
Regards,
Atish
PATCH v2 7/9 riscv: do not rely on hart ID passed by previous boot stage
In reply to Lukas AuerQuoted 11 lines · Lukas Auer
RISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
Suggested-by Rick Chen <[email protected]>Signed-off-by Lukas Auer <[email protected]>
Quoted 9 lines
---
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-modearch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v2 7/9 riscv: do not rely on hart ID passed by previous boot stage
Hi Lukas
<[email protected]> 於 2019年3月12日 週二 下午7:04寫道:
Quoted 30 lines
Quoted 30 lines
-----Original Message-----
From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de]
Sent: Wednesday, March 06, 2019 6:53 AM
To: u-boot at lists.denx.de
Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer Dabbelt; Alexander Graf; Lukas Auer; Rick Jian-Zhi Chen(陳建志); Anup Patel
Subject: [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stageRISC-V U-Boot expects the hart ID to be passed to it via register a0 by the previous boot stage. Machine mode firmware such as BBL and OpenSBI do this when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in machine mode, this task must be handled by the boot ROM. Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid possible problems on RISC-V processors with a boot ROM that does not handle this task.
Suggested-by Rick Chen <[email protected]>Signed-off-by Lukas Auer <[email protected]>---
Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-modearch/riscv/cpu/start.S | 4 ++++ 1 file changed, 4 insertions(+)
Quoted 15 lines
Quoted 5 lines
arch/riscv/cpu/start.S index renamed from arch/riscv/cpu/start.S +0
-0
79b753847c..d4daa6e0bf100644arch/riscv/cpu/start.S line 45
Quoted 10 lines
arch/riscv/cpu/start.S +4
-0@@ -36,6 +36,10 @@ .section .text .globl _start _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif + /* save hart id and dtb pointer */ mv s0, a0 mv s1, a1
Thanks
Rick
PATCH v2 8/9 riscv: fu540: enable SMP
In reply to Lukas Auer---
Changes in v2:
- New patch to enable SMP on the SiFive FU540, which was previously sent independently
board/sifive/fu540/Kconfig | 1 + 1 file changed, 1 insertion(+)
board/sifive/fu540/Kconfig +1 -0
@@ -38,5 +38,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy imply PHY_LIB imply PHY_MSCC imply SIFIVE_SERIAL + imply SMP endif
Signature · 1 line
2.20.1
PATCH v2 8/9 riscv: fu540: enable SMP
Quoted 12 lines · Lukas Auer
Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>---
Changes in v2:
- New patch to enable SMP on the SiFive FU540, which was previously sent independentlyboard/sifive/fu540/Kconfig | 1 + 1 file changed, 1 insertion(+)
PATCH v2 9/9 riscv: qemu: enable SMP
In reply to Lukas Auer---
Changes in v2: None
board/emulation/qemu-riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)
board/emulation/qemu-riscv/Kconfig +1 -0
@@ -34,5 +34,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy imply BOARD_LATE_INIT imply OF_BOARD_SETUP imply SIFIVE_SERIAL + imply SMP endif
Signature · 1 line
2.20.1
PATCH v2 9/9 riscv: qemu: enable SMP
Quoted 11 lines · Lukas Auer
Signed-off-by Lukas Auer <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Bin Meng <[email protected]>---
Changes in v2: None
board/emulation/qemu-riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)
v2 cover SMP support for RISC-V
In reply to Lukas AuerQuoted 70 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; Andreas Schwab <[email protected]>; Palmer Dabbelt <[email protected]>; Alexander Graf <[email protected]>; Lukas Auer <[email protected]>; Anup Patel <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>; Baruch Siach <[email protected]>; Atish Patra <[email protected]>; Stefan Roese <[email protected]>; Paul Walmsley <[email protected]>; Daniel Schwierzeck <[email protected]>
Subject: [PATCH v2 0/9] SMP support for RISC-VThis patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.
The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independentlyLukas Auer (9):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 152 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/sbi.h | 94 +++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 +++++ arch/riscv/lib/smp.c | 116 ++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 492 insertions(+), 10 deletions(-) create mode 100644
Quoted 6 lines
arch/riscv/include/asm/sbi.hcreate mode 100644arch/riscv/include/asm/smp.hcreate mode 100644arch/riscv/lib/sbi_ipi.ccreate mode 100644arch/riscv/lib/smp.c--
2.20.1
I tried this series on U-Boot-2019.04-rc3 and works fine on SiFive Unleashed board.
I have also pushed these patches to riscv_sifive_fu540_smp_v4 branch of https://github.com/avpatel/u-boot.git
Regards,
Anup
Auer
v2 cover SMP support for RISC-V
Quoted 88 lines · Anup Patel
Quoted 79 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; Andreas Schwab <[email protected]>; Palmer Dabbelt <[email protected]>; Alexander Graf <[email protected]>; Lukas Auer <[email protected]>; Anup Patel <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>; Baruch Siach <[email protected]>; Atish Patra <[email protected]> ;
Stefan Roese <[email protected]>; Paul Walmsley <[email protected]> ;
Daniel Schwierzeck <[email protected]>
Subject: [PATCH v2 0/9] SMP support for RISC-VThis patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to
run on multi-hart systems (hart is the RISC-V terminology for hardware
thread). Images passed to bootm will be started on all harts. The bootm command is currently the only one that will boot images on all
harts, bootefi is not yet supported.The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using
BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as
indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in
machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent
independentlyLukas Auer (9):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 152 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/sbi.h | 94 +++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 +++++ arch/riscv/lib/smp.c | 116 ++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 492 insertions(+), 10 deletions(-) create modeQuoted 8 lines
100644
arch/riscv/include/asm/sbi.hcreate mode 100644arch/riscv/include/asm/smp.hcreate mode 100644arch/riscv/lib/sbi_ipi.c
create mode 100644arch/riscv/lib/smp.c--
2.20.1
Quoted 8 lines
I tried this series on U-Boot-2019.04-rc3 and works fine on SiFive Unleashed board.
Tested-by Anup Patel <[email protected]>I have also pushed these patches to
riscv_sifive_fu540_smp_v4branch of https://github.com/avpatel/u-boot.git
Thank you for testing the series, Anup!
Did you observe anymore issues, where not all harts enter Linux?
Thanks,
Lukas
v2 cover SMP support for RISC-V
Quoted 97 lines
-----Original Message-----
From: Auer, Lukas <[email protected]>
Sent: Wednesday, March 6, 2019 2:52 PM
To: u-boot at lists.denx.de; Anup Patel <[email protected]>
Cc: paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; schwab at suse.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 6 lines · Anup Patel
Quoted 6 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; AndreasSchwab
Quoted 1 line
Quoted 1 line
<[email protected]>; Palmer Dabbelt <[email protected]>; Alexander
Graf
Quoted 75 lines
Quoted 64 lines
<[email protected]>; Lukas Auer <[email protected]>; Anup Patel <[email protected]>; Anup Patel <[email protected]>; Rick Chen <[email protected]>; Baruch Siach <[email protected]>; Atish Patra <[email protected]> ; Stefan Roese <[email protected]>; Paul Walmsley <[email protected]> ; Daniel Schwierzeck <[email protected]>
Subject: [PATCH v2 0/9] SMP support for RISC-VThis patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems (hart is the RISC-V terminology for hardware thread). Images passed to bootm will be started on all harts.
The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported.The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board (supervisor mode), using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent independentlyLukas Auer (9):
riscv: add infrastructure for calling functions on other harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 152 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/sbi.h | 94 +++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 +++++ arch/riscv/lib/smp.c | 116 ++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 492 insertions(+), 10 deletions(-) create modeQuoted 7 lines
100644
arch/riscv/include/asm/sbi.hcreate mode 100644arch/riscv/include/asm/smp.hcreate mode 100644arch/riscv/lib/sbi_ipi.ccreate mode 100644arch/riscv/lib/smp.c--
2.20.1I tried this series on U-Boot-2019.04-rc3 and works fine on SiFive Unleashed board.
Quoted 8 lines
Quoted 5 lines
Tested-by Anup Patel <[email protected]>I have also pushed these patches to
riscv_sifive_fu540_smp_v4branch of https://github.com/avpatel/u-boot.gitThank you for testing the series, Anup!
Did you observe anymore issues, where not all harts enter Linux?
I tried using cold-boot (using power-on button) worked fine for me 10 times. I did not try more.
We are trying to make OpenSBI stable with warm-boot (using reset button) as well but I have not tried that with U-Boot using reset button.
Regards,
Anup
Quoted 3 lines
Thanks,
Lukas
v2 cover SMP support for RISC-V
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51 core), which will then fail to start userspace, since that cannot cope with the missing fpu.
Andreas.
Signature · 3 lines
Andreas Schwab, SUSE Labs, schwab at suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
v2 cover SMP support for RISC-V
Quoted 13 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>; u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
Quoted 3 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51 core), which will then fail to start userspace, since that cannot cope with the missing fpu.
That's not possible because in this series we have "available_hart_mask" to track HARTs that entered U-Boot.
Recently, Atish made some progress with OpenSBI warm-boot issues. I will let him provide details about it.
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 11 lines · Anup Patel
Quoted 11 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>; u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
Quoted 5 lines
Quoted 3 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51 core), which will then fail to start userspace, since that cannot cope with the missing fpu.
That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____
/ __ \ / ____| _ \_ _|
| | | |_ __ ___ _ __ | (___ | |_) || |
| | | | '_ \ / _ \ '_ \ \___ \| _ < | |
| |__| | |_) | __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|____/_____|
| |
|_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base : 0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1
PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiB
Andreas.
Signature · 3 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
v2 cover SMP support for RISC-V
Quoted 45 lines · Andreas Schwab
Quoted 11 lines · Anup Patel
Quoted 11 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>; u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 5 lines
Quoted 3 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51 core), which will then fail to start userspace, since that cannot cope with the missing fpu.
That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____/ __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
Quoted 5 lines
U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiB
How does this prove that U-Boot is booting on HART 0?
This seems to be warm reset issues for which fixes from Atish are not yet merged.
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 46 lines · Anup Patel
Quoted 45 lines · Andreas Schwab
Quoted 11 lines · Anup Patel
Quoted 11 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>; u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 5 lines
Quoted 3 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51 core), which will then fail to start userspace, since that cannot cope with the missing fpu.
That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____/ __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
Quoted 7 lines
Quoted 5 lines
U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiBHow does this prove that U-Boot is booting on HART 0?
See the CPU isa.
Andreas.
Signature · 3 lines
Andreas Schwab, SUSE Labs, schwab at suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Auer
v2 cover SMP support for RISC-V
Quoted 56 lines · Andreas Schwab
Quoted 54 lines · Anup Patel
Quoted 52 lines · Andreas Schwab
Quoted 14 lines · Anup Patel
Quoted 14 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>;
u-boot at lists.denx.de;
paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
<[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 7 lines
Quoted 5 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
core), which will then fail to start userspace, since that cannot cope with the
missing fpu.That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____/ __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiBHow does this prove that U-Boot is booting on HART 0?
See the CPU isa.
Interesting.. U-Boot assumes that it can run on any core it is started on. In this case, OpenSBI must have booted its payload on hart 0.
Lukas
v2 cover SMP support for RISC-V
Quoted 62 lines · Lukas
Quoted 60 lines · Andreas Schwab
Quoted 58 lines · Anup Patel
Quoted 55 lines · Andreas Schwab
Quoted 14 lines · Anup Patel
Quoted 14 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>;
u-boot at lists.denx.de;
paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
<[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 7 lines
Quoted 5 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
core), which will then fail to start userspace, since that cannot cope with the
missing fpu.That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____/ __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiB
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
How does this prove that U-Boot is booting on HART 0?
See the CPU isa.
Interesting.. U-Boot assumes that it can run on any core it is started on. In this case, OpenSBI must have booted its payload on hart 0.
This is certainly not reproducible on cold-boot at my end but this does mean OpenSBI has booted HART0 and let it jump to U-Boot.
Now OpenSBI (by default) on SiFive FU540 does not allow HART0 to go forward due to lack of S-mode. I think this is definitely the warm-boot issue where OpenSBI sees corrupted memory contents.
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 72 lines · Anup Patel
Quoted 62 lines · Lukas
Quoted 60 lines · Andreas Schwab
Quoted 58 lines · Anup Patel
Quoted 55 lines · Andreas Schwab
Quoted 14 lines · Anup Patel
Quoted 14 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>;
u-boot at lists.denx.de;
paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
<[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 7 lines
Quoted 5 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
core), which will then fail to start userspace, since that cannot cope with the
missing fpu.That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____ / __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiBQuoted 7 lines
Quoted 4 lines
Quoted 1 line
How does this prove that U-Boot is booting on HART 0?
See the CPU isa.
Interesting.. U-Boot assumes that it can run on any core it is started on. In this case, OpenSBI must have booted its payload on hart 0.
Quoted 7 lines
This is certainly not reproducible on cold-boot at my end but this does mean OpenSBI has booted HART0 and let it jump to U-Boot.
Now OpenSBI (by default) on SiFive FU540 does not allow HART0 to go forward due to lack of S-mode. I think this is definitely the warm-boot issue where OpenSBI sees corrupted memory contents.
I am able to test both warm-boot and cold-boot several times(>10) without any issue with following pending PR in openSBI.
https://github.com/riscv/opensbi/pull/84
@Andreas @Anup: Can you please apply the above PR on top of master and verify at your end as well?
All the harts booted in Linux every time as well.
Regards,
Atish
Quoted 3 lines
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 91 lines
-----Original Message-----
From: Atish Patra <[email protected]>
Sent: Thursday, March 7, 2019 5:20 AM
To: Anup Patel <[email protected]>; Auer, Lukas <[email protected]>
Cc: schwab at suse.de; paul.walmsley at sifive.com; agraf at suse.de; u- boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de; palmer at sifive.com; Anup Patel <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 69 lines · Anup Patel
Quoted 59 lines · Lukas
Quoted 57 lines · Andreas Schwab
Quoted 55 lines · Anup Patel
Quoted 52 lines · Andreas Schwab
Quoted 12 lines · Anup Patel
Quoted 12 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Wednesday, March 6, 2019 4:27 PM
To: Anup Patel <[email protected]>
Cc: Auer, Lukas <[email protected]>;
u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 6 lines
Quoted 4 lines
Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
core), which will then fail to start userspace, since that cannot cope with the missing fpu.That's not possible
Yes, it is.
OpenSBI v0.3 (Mar 6 2019 10:55:01)
____ _____ ____ _____ / __ \ / ____| _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |____) | |_) || |_ \____/| .__/ \___|_| |_|_____/|____/_____| | | |_|Platform Name : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs : 5
Current Hart : 2
Firmware Base :0x80000000
Firmware Size : 88 KB
Runtime SBI Version : 0.1PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
CPU: rv64imac Model: sifive,hifive-unleashed-a00 DRAM: 8 GiBQuoted 7 lines
Quoted 4 lines
Quoted 1 line
How does this prove that U-Boot is booting on HART 0?
See the CPU isa.
Interesting.. U-Boot assumes that it can run on any core it is started on. In this case, OpenSBI must have booted its payload on hart 0.
Quoted 7 lines
This is certainly not reproducible on cold-boot at my end but this does mean OpenSBI has booted HART0 and let it jump to U-Boot.
Now OpenSBI (by default) on SiFive FU540 does not allow HART0 to go forward due to lack of S-mode. I think this is definitely the warm-boot issue where OpenSBI sees corrupted memory contents.
Quoted 4 lines
I am able to test both warm-boot and cold-boot several times(>10) without any issue with following pending PR in openSBI.
Thanks Atish, your changes have been merged in OpenSBI.
Quoted 5 lines
@Andreas @Anup: Can you please apply the above PR on top of master and verify at your end as well?
All the harts booted in Linux every time as well.
I tried warm-boot 10 times from U-Boot prompt and I did not see any issue at all.
I also tried warm-boot 10 times from Linux prompt and I did not see any Issue at all.
In both above cases, it came back to U-Boot prompt after warm-boot.
Also, tried cold-boot couple of times. I works perfectly fine as well.
Like I mentioned, there is no functional issue with this series. The warm-boot issues were fixed in OpenSBI.
@Andreas, please try at your end.
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 4 lines · Anup Patel
Like I mentioned, there is no functional issue with this series. The warm-boot issues were fixed in OpenSBI.
@Andreas, please try at your end.
As long as issue#65 isn't fixed opensbi is mostly a no-go for me. At least it gives me more reasons to press the reset button. :-)
Andreas.
Signature · 3 lines
Andreas Schwab, SUSE Labs, schwab at suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
v2 cover SMP support for RISC-V
Quoted 15 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Thursday, March 7, 2019 2:50 PM
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Auer, Lukas <[email protected]>; paul.walmsley at sifive.com; agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VOn Mär 07 2019, Anup Patel <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
Like I mentioned, there is no functional issue with this series. The warm-boot issues were fixed in OpenSBI.
@Andreas, please try at your end.
As long as issue#65 isn't fixed opensbi is mostly a no-go for me. At least it gives me more reasons to press the reset button. :-)
The reset button works fine for me an Atish. I am sure it works fine for lot of other folks too.
BTW, as-per discussion with SiFive folks the reset button on Unleashed Board is not much tested and it can misbehave on certain boards. It is quite possible that you might have a "flaky" board.
Regards,
Anup
v2 cover SMP support for RISC-V
Quoted 2 lines · Anup Patel
The reset button works fine for me an Atish. I am sure it works fine for lot of other folks too.
There is no issue with the reset button, only with openSBI.
Andreas.
Signature · 3 lines
Andreas Schwab, SUSE Labs, schwab at suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
v2 cover SMP support for RISC-V
In reply to Anup PatelQuoted 23 lines · Anup Patel
Quoted 15 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Thursday, March 7, 2019 2:50 PM
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Auer, Lukas <[email protected]>; paul.walmsley at sifive.com; agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
rick at andestech.com; sr at denx.de; palmer at sifive.com
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VOn Mär 07 2019, Anup Patel <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
Like I mentioned, there is no functional issue with this series. The warm-boot issues were fixed in OpenSBI.
@Andreas, please try at your end.
As long as issue#65 isn't fixed opensbi is mostly a no-go for me. At least it gives me more reasons to press the reset button. :-)
Quoted 6 lines
The reset button works fine for me an Atish. I am sure it works fine for lot of other folks too.
BTW, as-per discussion with SiFive folks the reset button on Unleashed Board is not much tested and it can misbehave on certain boards. It is quite possible that you might have a "flaky" board.
I don't think the reset button differs between boards. As far as I know, the issues are really just that it doesn't reset everything -- specifically some of the IP on the chip (clock, power, JTAG) isn't reset and nothing on the board (SD, ethernet, PCIe, etc) is reset. This frequently results in flakiness when debugging drivers, but the cores and memory system should all be OK.
Is that issue 65 on github.com/opensbi? If so it clearly says this isn't a reset button issue.
v2 cover SMP support for RISC-V
Quoted 45 lines
-----Original Message-----
From: Palmer Dabbelt <[email protected]>
Sent: Monday, March 11, 2019 5:26 PM
To: Anup Patel <[email protected]>
Cc: schwab at suse.de; Atish Patra <[email protected]>;
anup at brainfault.org; lukas.auer at aisec.fraunhofer.de; Paul Walmsley <[email protected]>; agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de
Subject: RE: [PATCH v2 0/9] SMP support for RISC-VQuoted 10 lines · Anup Patel
Quoted 10 lines
-----Original Message-----
From: Andreas Schwab <[email protected]>
Sent: Thursday, March 7, 2019 2:50 PM
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Auer, Lukas <[email protected]>; paul.walmsley at sifive.com; agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com;bmeng.cn at gmail.com;
Quoted 13 lines
Quoted 5 lines
rick at andestech.com; sr at denx.de; palmer at sifive.com
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VOn Mär 07 2019, Anup Patel <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
Like I mentioned, there is no functional issue with this series. The warm-boot issues were fixed in OpenSBI.
@Andreas, please try at your end.
As long as issue#65 isn't fixed opensbi is mostly a no-go for me. At least it gives me more reasons to press the reset button. :-)
Quoted 6 lines
The reset button works fine for me an Atish. I am sure it works fine for lot of other folks too.
BTW, as-per discussion with SiFive folks the reset button on Unleashed Board is not much tested and it can misbehave on certain boards. It is quite possible that you might have a "flaky" board.
Quoted 8 lines
I don't think the reset button differs between boards. As far as I know, the issues are really just that it doesn't reset everything -- specifically some of the IP on the chip (clock, power, JTAG) isn't reset and nothing on the board (SD, ethernet, PCIe, etc) is reset. This frequently results in flakiness when debugging drivers, but the cores and memory system should all be OK.
Is that issue 65 on github.com/opensbi? If so it clearly says this isn't a reset button issue.
The issue#65 on githuh.com/opensbi is not a clearly defined and it went in various directions. We tried various things suggested by Andreas and we were only able to replicate issue with reset-button press. This fixed now and reset-button press works perfectly fine with OpenSBI.
Apart from reset-button thingy, we tried all other things reported by Andreas but we were not able to reproduce issue at our end.
Regards,
Anup
Auer
v2 cover SMP support for RISC-V
In reply to Anup PatelQuoted 122 lines · Anup Patel
Quoted 114 lines
-----Original Message-----
From: Auer, Lukas <[email protected]>
Sent: Wednesday, March 6, 2019 2:52 PM
To: u-boot at lists.denx.de; Anup Patel <[email protected]>
Cc: paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org; baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com ;
rick at andestech.com; sr at denx.de; schwab at suse.de; palmer at sifive.com; Atish Patra <[email protected]>
Subject: Re: [PATCH v2 0/9] SMP support for RISC-VQuoted 6 lines · Anup Patel
Quoted 6 lines
-----Original Message-----
From: Lukas Auer <[email protected]>
Sent: Wednesday, March 6, 2019 4:23 AM
To: u-boot at lists.denx.de
Cc: Atish Patra <[email protected]>; Anup Patel <[email protected]>; Bin Meng <[email protected]>; AndreasSchwab
Quoted 1 line
Quoted 1 line
<[email protected]>; Palmer Dabbelt <[email protected]>; Alexander
Graf
Quoted 93 lines
Quoted 84 lines
<[email protected]>; Lukas Auer <[email protected]>; Anup
Patel <[email protected]>; Anup Patel <[email protected]>; Rick
Chen <[email protected]>; Baruch Siach <[email protected]>; Atish
Patra <[email protected]> ; Stefan Roese <[email protected]>; Paul Walmsley <[email protected]> ; Daniel Schwierzeck <[email protected]>
Subject: [PATCH v2 0/9] SMP support for RISC-VThis patch series adds SMP support for RISC-V to U-Boot. It allows
U-Boot to run on multi-hart systems (hart is the RISC-V terminology
for hardware thread). Images passed to bootm will be started on all
harts.
The bootm command is currently the only one that will boot images on
all harts, bootefi is not yet supported.The patches have been successfully tested on both QEMU (machine and
supervisor mode) and the HiFive Unleashed board (supervisor mode),
using BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/Changes in v2:
- Remove unneeded quotes fromNR_CPUSKconfigentry
- Move memory barrier fromsend_ipi_many()tohandle_ipi()
- Add check insend_ipi_manyso that IPIs are only sent to available
harts as indicated by theavailable_hartsmask
- Implement hart lottery to pick main hart to run U-Boot
- RemoveCONFIG_MAIN_HARTas it is not required anymore
- Register available harts in theavailable_hartsmask
- New patch to populate register a0 with the hart ID from the mhartid CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously
sent independentlyLukas Auer (9):
riscv: add infrastructure for calling functions on other
harts
riscv: import the supervisor binary interface header file
riscv: implement IPI platform functions using SBI
riscv: delay initialization of caches and debug UART
riscv: add support for multi-hart systems
riscv: boot images passed to bootm on all harts
riscv: do not rely on hart ID passed by previous boot stage
riscv: fu540: enable SMP
riscv: qemu: enable SMParch/riscv/Kconfig | 28 +++++ arch/riscv/cpu/cpu.c | 9 +- arch/riscv/cpu/start.S | 152 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 6 ++ arch/riscv/include/asm/sbi.h | 94 +++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/asm-offsets.c | 1 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 +++++ arch/riscv/lib/smp.c | 116 ++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + board/sifive/fu540/Kconfig | 1 + 14 files changed, 492 insertions(+), 10 deletions(-) createmode
100644arch/riscv/include/asm/sbi.hcreate mode 100644arch/riscv/include/asm/smp.hcreate mode 100644arch/riscv/lib/sbi_ipi.ccreate mode 100644arch/riscv/lib/smp.cQuoted 2 lines
--
2.20.1I tried this series on U-Boot-2019.04-rc3 and works fine on SiFive
Unleashed board.Tested-by Anup Patel <[email protected]>Quoted 7 lines
Quoted 4 lines
I have also pushed these patches to
riscv_sifive_fu540_smp_v4branch
of https://github.com/avpatel/u-boot.gitThank you for testing the series, Anup!
Did you observe anymore issues, where not all harts enter Linux?
Quoted 7 lines
I tried using cold-boot (using power-on button) worked fine for me 10 times. I did not try more.
We are trying to make OpenSBI stable with warm-boot (using reset button) as well but I have not tried that with U-Boot using reset button.
Great, so it seems the cold-boot issue is fixed now. Thanks for testing this!
Lukas