Skip to contentSkip to conversation

Coldboot wait improvements

PATCH New Anup Patel 2 participants 4 messages Mar 19, 2024 – Apr 5, 2024 RISC-V

lore
Share & tools
lore

Anup Patel

author
#0 lore raw

cover Coldboot wait improvements

These are some of the simplifications/improvements which were found upon trying QEMU virt machine with large number of HARTs.

These patches can also be found in the coldboot_wait_simplify_v1 branch at https://github.com/avpatel/opensbi.git

Anup Patel (2):
lib: sbi: Simplify wait_for_coldboot() implementation
lib: sbi: Wakeup non-coldboot HARTs early in the coldboot path

 include/sbi/riscv_barrier.h |  6 ++-
 lib/sbi/sbi_init.c          | 81 ++++++-------------------------------
 2 files changed, 17 insertions(+), 70 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/2 lib: sbi: Simplify wait_for_coldboot() implementation

On QEMU virt machine with large number of HARTs, some of the HARTs randomly fail to come out of wait_for_coldboot() due to one of the following race-conditions:

1) Failing HARTs are not able to acquire the coldboot_lock and update the coldboot_hartmask in wait_for_coldboot() before the coldboot HART acquires the coldboot_lock and sends IPI in wake_coldboot_harts() hence the failing HARTs never receive IPI from the coldboot HART.

2) Failing HARTs acquire the coldbood_lock and update the coldboot_hartmask before coldboot HART does sbi_scratch_init() so the sbi_hartmask_set_hartid() does not update the coldboot_hartmask on the failing HARTs hence they never receive IPI from the coldboot HART.

To address this, use a simple busy-loop in wait_for_coldboot() for polling on coldboot_done flag.

---
 include/sbi/riscv_barrier.h |  6 +++-
 lib/sbi/sbi_init.c          | 70 ++-----------------------------------
 2 files changed, 8 insertions(+), 68 deletions(-)
include/sbi/riscv_barrier.h +5 -1
index 1fba8b8..3d4a038 100644
@@ -40,7 +40,11 @@
 #define smp_wmb()		RISCV_FENCE(w,w)
 
 /* CPU relax for busy loop */
-#define cpu_relax()		asm volatile ("" : : : "memory")
+#define cpu_relax()						\
+do {								\
+	unsigned long __t;					\
+	__asm__ __volatile__ ("div %0, %0, zero" : "=r" (__t));	\
+} while (0)
 
 /* clang-format on */
 
lib/sbi/sbi_init.c +3 -67

92 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/2 lib: sbi: Wakeup non-coldboot HARTs early in the coldboot path

In reply to Anup Patel

Currently, all non-coldboot HARTs busy spin in wait_for_coldboot() until the entire coldboot init sequence is completed.

This means:
1) On QEMU, all non-coldboot HARTs will eat host CPU time and also slow down the coldboot HART until the entire coldboot init sequence is completed.
2) On real HW, all non-coldboot HARTs will consume more CPU power until the entire coldboot init sequence is completed.

To address this, wake up all non-coldboot HARTs as early as possible in the coldboot init sequence.

---
 lib/sbi/sbi_init.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/sbi/sbi_init.c +9 -2
index b3f3e38..8fdc8de 100644
@@ -243,6 +243,14 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
+	/*
+	 * All non-coldboot HARTs do HSM initialization (i.e. enter HSM state
+	 * machine) at the start of the warmboot path so it is wasteful to
+	 * have these HARTs busy spin in wait_for_coldboot() until coldboot
+	 * path is completed.
+	 */
+	wake_coldboot_harts(scratch, hartid);
+
 	rc = sbi_platform_early_init(plat, true);
 	if (rc)
 		sbi_hart_hang();
@@ -348,8 +356,6 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 		sbi_hart_hang();
 	}
 
-	wake_coldboot_harts(scratch, hartid);
-
 	count = sbi_scratch_offset_ptr(scratch, init_count_offset);
 	(*count)++;
 
@@ -369,6 +375,7 @@ static void __noreturn init_warm_startup(struct sbi_scratch *scratch,
 	count = sbi_scratch_offset_ptr(scratch, entry_count_offset);
 	(*count)++;
 
+	/* Note: This has to be first thing in warmboot init sequence */
 	rc = sbi_hsm_init(scratch, hartid, false);
 	if (rc)
 		sbi_hart_hang();
Signature · 1 line
2.34.1
Applied
#3 lore raw

cover Coldboot wait improvements

In reply to Anup Patel
Quoted 10 lines · On Tue, Mar 19, 2024 at 8:29?PM Anup Patel <[email protected]> wrote:

These are some of the simplifications/improvements which were found upon trying QEMU virt machine with large number of HARTs.

These patches can also be found in the coldboot_wait_simplify_v1 branch at https://github.com/avpatel/opensbi.git

Anup Patel (2):
lib: sbi: Simplify wait_for_coldboot() implementation
lib: sbi: Wakeup non-coldboot HARTs early in the coldboot path

Applied this series to the riscv/opensbi repo.

Regards,
Anup

Quoted 7 lines
 include/sbi/riscv_barrier.h |  6 ++-
 lib/sbi/sbi_init.c          | 81 ++++++-------------------------------
 2 files changed, 17 insertions(+), 70 deletions(-)

--
2.34.1