RISC-V: Increase range and default value of NR_CPUS
PATCH v2 1/1 RISC-V: Increase range and default value of NR_CPUS
Currently, the range and default value of NR_CPUS is too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the current NR_CPUS is restrictive for QEMU as well. Other major architectures (such as ARM64, x86_64, MIPS, etc) have a much higher range and default value of NR_CPUS.
This patch increases NR_CPUS range to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).
---
Changes since v1:
- Updated NR_CPUS range to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.
--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/riscv/Kconfig +4 -3
@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N. config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512 depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BIT config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"
Signature · 1 line
2.25.1
Palmer Dabbelt
maintainerQuoted 28 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-1@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N. config NR_CPUS - int "Maximum number of CPUs (2-32)"
arch/riscv/Kconfig config SMP · line 284
Quoted 10 lines
arch/riscv/Kconfig +4
-2- range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512 depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BIT config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"
I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.
---
arch/riscv/Kconfig | 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMPIf you don't know what to do here, say N.
Quoted 5 lines
Quoted 5 lines
config
NR_CPUS
- int "Maximum number of CPUs (2-32)"
- range 2 32
+ int "Maximum number of CPUs (2-512)"
+ range 2 512
For SBI_V01=y there seems to be a hard constraint to XLEN bits. See __sbi_v01_cpumask_to_hartmask() in rch/riscv/kernel/sbi.c.
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32 SBI_V01 && 32BIT
range 2 64 SBI_V01 && 64BIT
Quoted 9 lines
Quoted 7 lines
depends on SMP
- default "8"
+ default "32" if 32BIT
+ default "64" if 64BITconfig
HOTPLUG_CPU
bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on x86_64 or arm64 with as many hardware threads as available (128 is not uncommon).
OpenSBI currently is limited to 128 threads:include/sbi/sbi_hartmask.h:22:
#define SBI_HARTMASK_MAX_BITS 128
This is just an arbitrary value we can be modified.
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of CONFIG_SYS_MALLOC_F_LEN that is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#t
With QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
Best regards
Heinrich
Quoted 8 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 38 lines · Heinrich Schuchardt
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N.Quoted 5 lines
arch/riscv/Kconfig config SMP · line 278
Quoted 5 lines
arch/riscv/Kconfig +2
-2config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512For SBI_V01=y there seems to be a hard constraint to XLEN bits. See
__sbi_v01_cpumask_to_hartmask()inrch/riscv/kernel/sbi.c.
Quoted 5 lines
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32SBI_V01&& 32BIT
range 2 64SBI_V01&& 64BIT
This is just making it unnecessarily complicated for supporting SBI v0.1
How about removing SBI v0.1 support and the spin-wait CPU operations from arch/riscv ?
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
arch/riscv/Kconfig +2
-1depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BITQuoted 8 lines
arch/riscv/Kconfig config SMP · line 284
Quoted 2 lines
arch/riscv/Kconfig +0
-0config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Quoted 8 lines
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on
x86_64or arm64 with as many hardware threads as available (128 is not uncommon).OpenSBI currently is limited to 128 threads:
include/sbi/sbi_hartmask.h:22:
#defineSBI_HARTMASK_MAX_BITS128
This is just an arbitrary value we can be modified.
Yes, this limit will be gradually increased with some improvements to optimize runtime memory used by OpenSBI.
Quoted 9 lines
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of
CONFIG_SYS_MALLOC_F_LENthat is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#tWith QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
Thanks for trying this patch.
Regards,
Anup
Quoted 17 lines
Best regards
Heinrich
Quoted 11 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 46 lines · Anup Patel
Quoted 38 lines · Heinrich Schuchardt
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N.Quoted 5 lines
arch/riscv/Kconfig config SMP · line 278
Quoted 5 lines
arch/riscv/Kconfig +2
-2config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512For SBI_V01=y there seems to be a hard constraint to XLEN bits. See
__sbi_v01_cpumask_to_hartmask()inrch/riscv/kernel/sbi.c.Quoted 5 lines
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32SBI_V01&& 32BIT
range 2 64SBI_V01&& 64BIT
Quoted 5 lines
This is just making it unnecessarily complicated for supporting SBI v0.1
How about removing SBI v0.1 support and the spin-wait CPU operations from
arch/riscv?
The SBI v0.1 specification was only a draft. Only the v1.0 version has ever been ratified.
It would be good to remove this legacy code from Linux and U-Boot.
By the way, why does upstream OpenSBI claim to be conformant to SBI v0.3 and not to v1.0?
include/sbi/sbi_ecall.h:16:
#define SBI_ECALL_VERSION_MAJOR 0
#define SBI_ECALL_VERSION_MINOR 3
Best regards
Heinrich
Quoted 54 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
arch/riscv/Kconfig +2
-1depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BITQuoted 8 lines
arch/riscv/Kconfig config SMP · line 284
Quoted 2 lines
arch/riscv/Kconfig +0
-0config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Quoted 8 lines
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on
x86_64or arm64 with as many hardware threads as available (128 is not uncommon).OpenSBI currently is limited to 128 threads:
include/sbi/sbi_hartmask.h:22:
#defineSBI_HARTMASK_MAX_BITS128
This is just an arbitrary value we can be modified.Yes, this limit will be gradually increased with some improvements to optimize runtime memory used by OpenSBI.
Quoted 9 lines
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of
CONFIG_SYS_MALLOC_F_LENthat is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#tWith QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
Thanks for trying this patch.
Regards,
AnupQuoted 11 lines
Best regards
Heinrich
Quoted 6 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
Quoted 54 lines · Heinrich Schuchardt
Quoted 46 lines · Anup Patel
Quoted 38 lines · Heinrich Schuchardt
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N.Quoted 5 lines
arch/riscv/Kconfig config SMP · line 278
Quoted 5 lines
arch/riscv/Kconfig +2
-2config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512For SBI_V01=y there seems to be a hard constraint to XLEN bits. See
__sbi_v01_cpumask_to_hartmask()inrch/riscv/kernel/sbi.c.Quoted 5 lines
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32SBI_V01&& 32BIT
range 2 64SBI_V01&& 64BITQuoted 5 lines
This is just making it unnecessarily complicated for supporting SBI v0.1
How about removing SBI v0.1 support and the spin-wait CPU operations from
arch/riscv?
Quoted 7 lines
The SBI v0.1 specification was only a draft. Only the v1.0 version has ever been ratified.
It would be good to remove this legacy code from Linux and U-Boot.
By the way, why does upstream OpenSBI claim to be conformant to SBI v0.3 and not to v1.0?
The ratification process for SBI v1.0 was in early stages when OpenSBI v1.0 was being released so we decided to keep the SBI v0.3 spec version. The next OpenSBI v1.1 release (due in June 2022) will change to SBI v1.0
Regards,
Anup
Quoted 64 lines
include/sbi/sbi_ecall.h:16:#define
SBI_ECALL_VERSION_MAJOR0
#defineSBI_ECALL_VERSION_MINOR3Best regards
Heinrich
Quoted 54 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
arch/riscv/Kconfig +2
-1depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BITQuoted 8 lines
arch/riscv/Kconfig config SMP · line 284
Quoted 2 lines
arch/riscv/Kconfig +0
-0config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Quoted 8 lines
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on
x86_64or arm64 with as many hardware threads as available (128 is not uncommon).OpenSBI currently is limited to 128 threads:
include/sbi/sbi_hartmask.h:22:
#defineSBI_HARTMASK_MAX_BITS128
This is just an arbitrary value we can be modified.Yes, this limit will be gradually increased with some improvements to optimize runtime memory used by OpenSBI.
Quoted 9 lines
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of
CONFIG_SYS_MALLOC_F_LENthat is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#tWith QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
Thanks for trying this patch.
Regards,
AnupQuoted 11 lines
Best regards
Heinrich
Quoted 6 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
Quoted 60 lines · Anup Patel
Quoted 57 lines · Heinrich Schuchardt
Quoted 46 lines · Anup Patel
Quoted 38 lines · Heinrich Schuchardt
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N.Quoted 5 lines
arch/riscv/Kconfig config SMP · line 278
Quoted 5 lines
arch/riscv/Kconfig +2
-2config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512For SBI_V01=y there seems to be a hard constraint to XLEN bits. See
__sbi_v01_cpumask_to_hartmask()inrch/riscv/kernel/sbi.c.Quoted 5 lines
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32SBI_V01&& 32BIT
range 2 64SBI_V01&& 64BITQuoted 5 lines
This is just making it unnecessarily complicated for supporting SBI v0.1
How about removing SBI v0.1 support and the spin-wait CPU operations from
arch/riscv?The SBI v0.1 specification was only a draft. Only the v1.0 version has ever been ratified.
Quoted 9 lines
Quoted 4 lines
It would be good to remove this legacy code from Linux and U-Boot.
By the way, why does upstream OpenSBI claim to be conformant to SBI v0.3 and not to v1.0?
The ratification process for SBI v1.0 was in early stages when OpenSBI v1.0 was being released so we decided to keep the SBI v0.3 spec version. The next OpenSBI v1.1 release (due in June 2022) will change to SBI v1.0
Yes. We are in the final stages of the official ratification of SBI v1.0. Once that ratified version is released,
OpenSBI will be upgraded to support that.
Quoted 67 lines
Regards,
AnupQuoted 64 lines
include/sbi/sbi_ecall.h:16:#define
SBI_ECALL_VERSION_MAJOR0
#defineSBI_ECALL_VERSION_MINOR3Best regards
Heinrich
Quoted 54 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
arch/riscv/Kconfig +2
-1depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BITQuoted 8 lines
arch/riscv/Kconfig config SMP · line 284
Quoted 2 lines
arch/riscv/Kconfig +0
-0config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Quoted 8 lines
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on
x86_64or arm64 with as many hardware threads as available (128 is not uncommon).OpenSBI currently is limited to 128 threads:
include/sbi/sbi_hartmask.h:22:
#defineSBI_HARTMASK_MAX_BITS128
This is just an arbitrary value we can be modified.Yes, this limit will be gradually increased with some improvements to optimize runtime memory used by OpenSBI.
Quoted 9 lines
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of
CONFIG_SYS_MALLOC_F_LENthat is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#tWith QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
Thanks for trying this patch.
Regards,
AnupQuoted 11 lines
Best regards
Heinrich
Quoted 6 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
Signature · 2 lines
Regards, Atish
Quoted 38 lines · Heinrich Schuchardt
Quoted 27 lines · Palmer Dabbelt
Quoted 26 lines · On Sat, 19 Mar 2022 05:12:06 PDT (-0700), [email protected] wrote:
Currently, the range and default value of
NR_CPUSis too restrictive for high-end RISC-V systems with large number of HARTs. The latest QEMU virt machine supports upto 512 CPUs so the currentNR_CPUSis restrictive for QEMU as well. Other major architectures (such as ARM64,x86_64, MIPS, etc) have a much higher range and default value ofNR_CPUS.This patch increases
NR_CPUSrange to 2-512 and default value to XLEN (i.e. 32 for RV32 and 64 for RV64).Signed-off-by Anup Patel <[email protected]>---
Changes since v1:
- UpdatedNR_CPUSrange to 2-512 which reflects maximum number of CPUs supported by QEMU virt machine.--- arch/riscv/Kconfig | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)arch/riscv/Kconfig +0
-0@@ -274,10 +274,11 @@ config SMP If you don't know what to do here, say N.Quoted 5 lines
arch/riscv/Kconfig config SMP · line 278
Quoted 5 lines
arch/riscv/Kconfig +2
-2config NR_CPUS - int "Maximum number of CPUs (2-32)" - range 2 32 + int "Maximum number of CPUs (2-512)" + range 2 512For SBI_V01=y there seems to be a hard constraint to XLEN bits. See
__sbi_v01_cpumask_to_hartmask()inrch/riscv/kernel/sbi.c.
Quoted 6 lines
So shouldn't this be something like:
range 2 512 !SBI_V01
range 2 32SBI_V01&& 32BIT
range 2 64SBI_V01&& 64BIT
Yes. In addition to that, we should disable RISCV_BOOT_SPINWAIT as well so that you don't end up in case where -smp argument in Qemu < NR_CPUs
Palmer already has a patch for that.
https://git.kernel.org/pub/scm/linux/kernel/git/palmer/linux.git/commit/?h=riscv-spinwait
Quoted 23 lines
Quoted 5 lines
Quoted 5 lines
arch/riscv/Kconfig +2
-1depends on SMP - default "8" + default "32" if 32BIT + default "64" if 64BITQuoted 8 lines
arch/riscv/Kconfig config SMP · line 284
Quoted 2 lines
arch/riscv/Kconfig +0
-0config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"I'm getting all sorts of boot issues with more than 32 CPUs, even on the latest QEMU master. I'm not opposed to increasing the CPU count in theory, but if we're going to have a setting that goes up to a huge number it needs to at least boot. I've got 64 host threads, so it shouldn't just be a scheduling thing.
Currently high performing hardware for RISC-V is missing. So it makes sense to build software via QEMU on
x86_64or arm64 with as many hardware threads as available (128 is not uncommon).OpenSBI currently is limited to 128 threads:
include/sbi/sbi_hartmask.h:22:
#defineSBI_HARTMASK_MAX_BITS128
This is just an arbitrary value we can be modified.
Quoted 9 lines
U-Boot v2022.04 qemu-riscv64_smode_defconfig has a value of
CONFIG_SYS_MALLOC_F_LENthat is to low. This leads to a boot failure for more than 16 harts. A patch to correct this is pending:
[PATCH v2 1/1] riscv: alloc space exhausted
https://lore.kernel.org/u-boot/CAN5B=eKt=tFLZ2z3aNHJqsnJzpdA0oikcrC2i1_=ZDD=f+M0jA@mail.gmail.com/T/#tWith QEMU 7.0 and the U-Boot fix booting into a 5.17 defconfig kernel with 64 virtual cores worked fine for me.
For me, OpenSBI -> Linux path works for 128 harts for bunch of configs from here
https://github.com/palmer-dabbelt/riscv-systems-ci/tree/master/configs/linux
with RISCV_BOOT_SPINWAIT disabled.
Quoted 16 lines
Best regards
Heinrich
Quoted 11 lines
If there was some hardware that actually boots on these I'd be happy to take it, but given that it's just QEMU I'd prefer to sort out the bugs first. It's probably just latent bugs somewhere, but allowing users to turn on configs we know don't work just seems like the wrong way to go.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Signature · 2 lines
Regards, Atish