Skip to contentSkip to conversation

OpenSBI domain support

PATCH Superseded Anup Patel 4 participants 39 messages Sep 25, 2020 – Oct 19, 2020

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Anup Patel

author
#0 lore raw

cover OpenSBI domain support

This series extends OpenSBI to provide system-level partitioning where each logical system-level partition is referred to as OpenSBI domain. The last patch of this series provides documentation for OpenSBI domain support.

It will be responsibility of OpenSBI platform support to populate OpenSBI domains based on use-case of platform vendor.

We will be having separate series to extend OpenSBI generic platform so that OpenSBI generic platform can populate OpenSBI domains from DTB passed by the previous booting stage which in-turn provides more configurability to the previous booting stage.

These patches are also available in domains_v1 branch at
https://github.com/avpatel/opensbi.git

Anup Patel (16):
include: sbi: Remove redundant includes from sbi_platform.h
include: sbi: Remove pmp_region callbacks from sbi_platform_operations
lib: sbi: Configure PMP late in coldboot and warmboot path
lib: sbi: Remove redundant hartid parameter from sbi_hart_init()
include: sbi: Make hartmask pointer as const in
sbi_hartmask_test_hart()
lib: sbi: Add initial domain support
lib: sbi: Extend sbi_hsm_hart_started_mask() for domains
lib: sbi: Extend sbi_hsm_hart_start() for domains
lib: sbi: Optimize sbi_hsm_hart_started_mask() implementation
lib: sbi: Extend sbi_system_reset() for domains
lib: utils: Update fdt_reserved_memory_fixup() to use current domain
lib: utils: Update fdt_cpu_fixup() to use current domain
lib: sbi: Remove redundant sbi_hart_pmp_xyz() functions
lib: sbi: Configure PMP based on domain memory regions
lib: sbi: Display domain details in boot prints
docs: Add initial documentation for domain support

 README.md                  |   3 +
 docs/domain_support.md     | 105 +++++++++
 docs/doxygen.cfg           |   1 +
 include/sbi/sbi_domain.h   | 144 ++++++++++++
 include/sbi/sbi_hart.h     |  13 +-
 include/sbi/sbi_hartmask.h |   2 +-
 include/sbi/sbi_hsm.h      |  12 +-
 include/sbi/sbi_platform.h |  56 ++---
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 461 +++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_ecall_hsm.c    |   8 +-
 lib/sbi/sbi_ecall_legacy.c |   4 +-
 lib/sbi/sbi_hart.c         | 152 ++++--------
 lib/sbi/sbi_hsm.c          |  59 +++--
 lib/sbi/sbi_init.c         |  59 +++--
 lib/sbi/sbi_ipi.c          |   6 +-
 lib/sbi/sbi_system.c       |  11 +-
 lib/utils/fdt/fdt_fixup.c  |  55 ++---
 18 files changed, 906 insertions(+), 246 deletions(-)
 create mode 100644 docs/domain_support.md
 create mode 100644 include/sbi/sbi_domain.h
 create mode 100644 lib/sbi/sbi_domain.c
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/16 include: sbi: Remove redundant includes from sbi_platform.h

The sbi_ecall.h and sbi_trap.h are not required to be included in sbi_platform.h hence we remove it. This way sources requiring these headers will have to explicitly include it thereby reducing implicit includes.

---
 include/sbi/sbi_platform.h | 4 ++--
 lib/sbi/sbi_hart.c         | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
include/sbi/sbi_platform.h +2 -2
index 8087148..f51e36a 100644
@@ -40,12 +40,12 @@
 
 #ifndef __ASSEMBLY__
 
-#include <sbi/sbi_ecall.h>
 #include <sbi/sbi_error.h>
 #include <sbi/sbi_scratch.h>
-#include <sbi/sbi_trap.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_trap_info;
+
 /** Possible feature flags of a platform */
 enum sbi_platform_features {
 	/** Platform has timer value */
lib/sbi/sbi_hart.c +1 -0
index 6413194..55a04da 100644
@@ -19,6 +19,7 @@
 #include <sbi/sbi_math.h>
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_string.h>
+#include <sbi/sbi_trap.h>
 
 extern void __sbi_expected_trap(void);
 extern void __sbi_expected_trap_hext(void);
Signature · 1 line
2.25.1
#2 lore raw

PATCH 1/16 include: sbi: Remove redundant includes from sbi_platform.h

Quoted 47 lines · Anup Patel

The sbi_ecall.h and sbi_trap.h are not required to be included in sbi_platform.h hence we remove it. This way sources requiring these headers will have to explicitly include it thereby reducing implicit includes.

---
 include/sbi/sbi_platform.h | 4 ++--
 lib/sbi/sbi_hart.c         | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
include/sbi/sbi_platform.h +2 -2
index 8087148..f51e36a 100644
@@ -40,12 +40,12 @@
 
 #ifndef __ASSEMBLY__
 
-#include <sbi/sbi_ecall.h>
 #include <sbi/sbi_error.h>
 #include <sbi/sbi_scratch.h>
-#include <sbi/sbi_trap.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_trap_info;
+
 /** Possible feature flags of a platform */
 enum sbi_platform_features {
        /** Platform has timer value */
lib/sbi/sbi_hart.c +1 -1
index 6413194..55a04da 100644
@@ -19,6 +19,7 @@
 #include <sbi/sbi_math.h>
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_string.h>
+#include <sbi/sbi_trap.h>
 
 extern void __sbi_expected_trap(void);
 extern void __sbi_expected_trap_hext(void);
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
author
#3 lore raw

PATCH 2/16 include: sbi: Remove pmp_region callbacks from sbi_platform_operations

In reply to Anup Patel

The pmp_region_xyz() callbacks are not required in sbi_platform_operations because we will be using OpenSBI domain memory regions to program the PMP CSRs.

---
 include/sbi/sbi_platform.h | 50 --------------------------------------
 lib/sbi/sbi_hart.c         | 14 +----------
 2 files changed, 1 insertion(+), 63 deletions(-)
include/sbi/sbi_platform.h +0 -50
index f51e36a..f2c3237 100644
@@ -89,15 +89,6 @@ struct sbi_platform_operations {
 	 */
 	int (*misa_get_xlen)(void);
 
-	/** Get number of PMP regions for given HART */
-	u32 (*pmp_region_count)(u32 hartid);
-	/**
-	 * Get PMP regions details (namely: protection, base address,
-	 * and size) for given HART
-	 */
-	int (*pmp_region_info)(u32 hartid, u32 index, ulong *prot, ulong *addr,
-			       ulong *log2size);
-
 	/** Write a character to the platform console output */
 	void (*console_putc)(char ch);
 	/** Read a character from the platform console input */
@@ -456,47 +447,6 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
 	return -1;
 }
 
-/**
- * Get the number of PMP regions of a HART
- *
- * @param plat pointer to struct sbi_platform
- * @param hartid HART ID
- *
- * @return number of PMP regions
- */
-static inline u32 sbi_platform_pmp_region_count(const struct sbi_platform *plat,
-						u32 hartid)
-{
-	if (plat && sbi_platform_ops(plat)->pmp_region_count)
-		return sbi_platform_ops(plat)->pmp_region_count(hartid);
-	return 0;
-}
-
-/**
- * Get PMP regions details (namely: protection, base address,
- * and size) of a HART
- *
- * @param plat pointer to struct sbi_platform
- * @param hartid HART ID
- * @param index index of PMP region for which we want details
- * @param prot output pointer for PMP region protection
- * @param addr output pointer for PMP region base address
- * @param log2size output pointer for log-of-2 PMP region size
- *
- * @return 0 on success and negative error code on failure
- */
-static inline int sbi_platform_pmp_region_info(const struct sbi_platform *plat,
-						u32 hartid, u32 index,
-						ulong *prot, ulong *addr,
-						ulong *log2size)
-{
-	if (plat && sbi_platform_ops(plat)->pmp_region_info)
-		return sbi_platform_ops(plat)->pmp_region_info(hartid, index,
-							       prot, addr,
-							       log2size);
-	return 0;
-}
-
 /**
  * Write a character to the platform console output
  *
lib/sbi/sbi_hart.c +1 -13
index 55a04da..edf79b7 100644
@@ -226,10 +226,8 @@ int sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
 
 static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
 {
-	u32 i, pmp_idx = 0, pmp_count, count;
+	u32 pmp_idx = 0;
 	unsigned long fw_start, fw_size_log2;
-	ulong prot, addr, log2size;
-	const struct sbi_platform *plat = sbi_platform_ptr(scratch);
 
 	if (!sbi_hart_pmp_count(scratch))
 		return 0;
@@ -239,16 +237,6 @@ static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
 	fw_start = scratch->fw_start & ~((1UL << fw_size_log2) - 1UL);
 	pmp_set(pmp_idx++, 0, fw_start, fw_size_log2);
 
-	/* Platform specific PMP regions */
-	count = sbi_platform_pmp_region_count(plat, hartid);
-	pmp_count = sbi_hart_pmp_count(scratch);
-	for (i = 0; i < count && pmp_idx < (pmp_count - 1); i++) {
-		if (sbi_platform_pmp_region_info(plat, hartid, i, &prot, &addr,
-						 &log2size))
-			continue;
-		pmp_set(pmp_idx++, prot, addr, log2size);
-	}
-
 	/*
 	 * Default PMP region for allowing S-mode and U-mode access to
 	 * memory not covered by:
Signature · 1 line
2.25.1
#4 lore raw

PATCH 2/16 include: sbi: Remove pmp_region callbacks from sbi_platform_operations

Quoted 116 lines · Anup Patel

The pmp_region_xyz() callbacks are not required in sbi_platform_operations because we will be using OpenSBI domain memory regions to program the PMP CSRs.

---
 include/sbi/sbi_platform.h | 50 --------------------------------------
 lib/sbi/sbi_hart.c         | 14 +----------
 2 files changed, 1 insertion(+), 63 deletions(-)
include/sbi/sbi_platform.h +0 -50
index f51e36a..f2c3237 100644
@@ -89,15 +89,6 @@ struct sbi_platform_operations {
         */
        int (*misa_get_xlen)(void);
 
-       /** Get number of PMP regions for given HART */
-       u32 (*pmp_region_count)(u32 hartid);
-       /**
-        * Get PMP regions details (namely: protection, base address,
-        * and size) for given HART
-        */
-       int (*pmp_region_info)(u32 hartid, u32 index, ulong *prot, ulong *addr,
-                              ulong *log2size);
-
        /** Write a character to the platform console output */
        void (*console_putc)(char ch);
        /** Read a character from the platform console input */
@@ -456,47 +447,6 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
        return -1;
 }
 
-/**
- * Get the number of PMP regions of a HART
- *
- * @param plat pointer to struct sbi_platform
- * @param hartid HART ID
- *
- * @return number of PMP regions
- */
-static inline u32 sbi_platform_pmp_region_count(const struct sbi_platform *plat,
-                                               u32 hartid)
-{
-       if (plat && sbi_platform_ops(plat)->pmp_region_count)
-               return sbi_platform_ops(plat)->pmp_region_count(hartid);
-       return 0;
-}
-
-/**
- * Get PMP regions details (namely: protection, base address,
- * and size) of a HART
- *
- * @param plat pointer to struct sbi_platform
- * @param hartid HART ID
- * @param index index of PMP region for which we want details
- * @param prot output pointer for PMP region protection
- * @param addr output pointer for PMP region base address
- * @param log2size output pointer for log-of-2 PMP region size
- *
- * @return 0 on success and negative error code on failure
- */
-static inline int sbi_platform_pmp_region_info(const struct sbi_platform *plat,
-                                               u32 hartid, u32 index,
-                                               ulong *prot, ulong *addr,
-                                               ulong *log2size)
-{
-       if (plat && sbi_platform_ops(plat)->pmp_region_info)
-               return sbi_platform_ops(plat)->pmp_region_info(hartid, index,
-                                                              prot, addr,
-                                                              log2size);
-       return 0;
-}
-
 /**
  * Write a character to the platform console output
  *
lib/sbi/sbi_hart.c +1 -14
index 55a04da..edf79b7 100644
@@ -226,10 +226,8 @@ int sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
 
 static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
 {
-       u32 i, pmp_idx = 0, pmp_count, count;
+       u32 pmp_idx = 0;
        unsigned long fw_start, fw_size_log2;
-       ulong prot, addr, log2size;
-       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
 
        if (!sbi_hart_pmp_count(scratch))
                return 0;
@@ -239,16 +237,6 @@ static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
        fw_start = scratch->fw_start & ~((1UL << fw_size_log2) - 1UL);
        pmp_set(pmp_idx++, 0, fw_start, fw_size_log2);
 
-       /* Platform specific PMP regions */
-       count = sbi_platform_pmp_region_count(plat, hartid);
-       pmp_count = sbi_hart_pmp_count(scratch);
-       for (i = 0; i < count && pmp_idx < (pmp_count - 1); i++) {
-               if (sbi_platform_pmp_region_info(plat, hartid, i, &prot, &addr,
-                                                &log2size))
-                       continue;
-               pmp_set(pmp_idx++, prot, addr, log2size);
-       }
-
        /*
         * Default PMP region for allowing S-mode and U-mode access to
         * memory not covered by:
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
author
#5 lore raw

PATCH 3/16 lib: sbi: Configure PMP late in coldboot and warmboot path

In reply to Anup Patel

We factor-out PMP configuration from sbi_hart_init() as a separate sbi_hart_pmp_configure() function and do the PMP configuration very late in coldboot and warmboot path just before platform_final_init().

Eventually, this enable us to configure PMP totally based on OpenSBI domain where OpenSBI domains can be parsed from device-tree in any of the platform operations except platform_final_init().

---
 include/sbi/sbi_hart.h | 1 +
 lib/sbi/sbi_hart.c     | 4 ++--
 lib/sbi/sbi_init.c     | 8 ++++++++
 3 files changed, 11 insertions(+), 2 deletions(-)
include/sbi/sbi_hart.h +1 -0
index 0ba68f0..9207f4c 100644
@@ -45,6 +45,7 @@ void sbi_hart_pmp_dump(struct sbi_scratch *scratch);
 int  sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
 			     unsigned long daddr, unsigned long mode,
 			     unsigned long attr);
+int sbi_hart_pmp_configure(struct sbi_scratch *scratch);
 bool sbi_hart_has_feature(struct sbi_scratch *scratch, unsigned long feature);
 void sbi_hart_get_features_str(struct sbi_scratch *scratch,
 			       char *features_str, int nfstr);
lib/sbi/sbi_hart.c +2 -2
index edf79b7..7244f1c 100644
@@ -224,7 +224,7 @@ int sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
 	return SBI_OK;
 }
 
-static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
+int sbi_hart_pmp_configure(struct sbi_scratch *scratch)
 {
 	u32 pmp_idx = 0;
 	unsigned long fw_start, fw_size_log2;
@@ -462,7 +462,7 @@ int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
 	if (rc)
 		return rc;
 
-	return pmp_init(scratch, hartid);
+	return 0;
 }
 
 void __attribute__((noreturn)) sbi_hart_hang(void)
lib/sbi/sbi_init.c +8 -0
index 65d81ba..85a5f13 100644
@@ -210,6 +210,10 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
+	rc = sbi_hart_pmp_configure(scratch);
+	if (rc)
+		sbi_hart_hang();
+
 	rc = sbi_platform_final_init(plat, TRUE);
 	if (rc)
 		sbi_hart_hang();
@@ -266,6 +270,10 @@ static void __noreturn init_warmboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
+	rc = sbi_hart_pmp_configure(scratch);
+	if (rc)
+		sbi_hart_hang();
+
 	rc = sbi_platform_final_init(plat, FALSE);
 	if (rc)
 		sbi_hart_hang();
Signature · 1 line
2.25.1
#6 lore raw

PATCH 3/16 lib: sbi: Configure PMP late in coldboot and warmboot path

Quoted 80 lines · Anup Patel

We factor-out PMP configuration from sbi_hart_init() as a separate sbi_hart_pmp_configure() function and do the PMP configuration very late in coldboot and warmboot path just before platform_final_init().

Eventually, this enable us to configure PMP totally based on OpenSBI domain where OpenSBI domains can be parsed from device-tree in any of the platform operations except platform_final_init().

---
 include/sbi/sbi_hart.h | 1 +
 lib/sbi/sbi_hart.c     | 4 ++--
 lib/sbi/sbi_init.c     | 8 ++++++++
 3 files changed, 11 insertions(+), 2 deletions(-)
include/sbi/sbi_hart.h +1 -0
index 0ba68f0..9207f4c 100644
@@ -45,6 +45,7 @@ void sbi_hart_pmp_dump(struct sbi_scratch *scratch);
 int  sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
                             unsigned long daddr, unsigned long mode,
                             unsigned long attr);
+int sbi_hart_pmp_configure(struct sbi_scratch *scratch);
 bool sbi_hart_has_feature(struct sbi_scratch *scratch, unsigned long feature);
 void sbi_hart_get_features_str(struct sbi_scratch *scratch,
                               char *features_str, int nfstr);
lib/sbi/sbi_hart.c +2 -2
index edf79b7..7244f1c 100644
@@ -224,7 +224,7 @@ int sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
        return SBI_OK;
 }
 
-static int pmp_init(struct sbi_scratch *scratch, u32 hartid)
+int sbi_hart_pmp_configure(struct sbi_scratch *scratch)
 {
        u32 pmp_idx = 0;
        unsigned long fw_start, fw_size_log2;
@@ -462,7 +462,7 @@ int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
        if (rc)
                return rc;
 
-       return pmp_init(scratch, hartid);
+       return 0;
 }
 
 void __attribute__((noreturn)) sbi_hart_hang(void)
lib/sbi/sbi_init.c +8 -1
index 65d81ba..85a5f13 100644
@@ -210,6 +210,10 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       rc = sbi_hart_pmp_configure(scratch);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_platform_final_init(plat, TRUE);
        if (rc)
                sbi_hart_hang();
@@ -266,6 +270,10 @@ static void __noreturn init_warmboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       rc = sbi_hart_pmp_configure(scratch);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_platform_final_init(plat, FALSE);
        if (rc)
                sbi_hart_hang();
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
author
#7 lore raw

PATCH 4/16 lib: sbi: Remove redundant hartid parameter from sbi_hart_init()

In reply to Anup Patel

The hartid parameter of sbi_hart_init() is not used anywhere in sbi_hart_init() implementation so let's remove it.

---
 include/sbi/sbi_hart.h |  2 +-
 lib/sbi/sbi_hart.c     | 14 +++++++-------
 lib/sbi/sbi_init.c     |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)
include/sbi/sbi_hart.h +1 -1
index 9207f4c..2efe44d 100644
@@ -27,7 +27,7 @@ enum sbi_hart_features {
 
 struct sbi_scratch;
 
-int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
+int sbi_hart_init(struct sbi_scratch *scratch, bool cold_boot);
 
 extern void (*sbi_hart_expected_trap)(void);
 static inline ulong sbi_hart_expected_trap_addr(void)
lib/sbi/sbi_hart.c +7 -7
index 7244f1c..88ffab0 100644
@@ -33,7 +33,7 @@ struct hart_features {
 };
 static unsigned long hart_features_offset;
 
-static void mstatus_init(struct sbi_scratch *scratch, u32 hartid)
+static void mstatus_init(struct sbi_scratch *scratch)
 {
 	unsigned long mstatus_val = 0;
 
@@ -62,7 +62,7 @@ static void mstatus_init(struct sbi_scratch *scratch, u32 hartid)
 		csr_write(CSR_SATP, 0);
 }
 
-static int fp_init(u32 hartid)
+static int fp_init(struct sbi_scratch *scratch)
 {
 #ifdef __riscv_flen
 	int i;
@@ -83,7 +83,7 @@ static int fp_init(u32 hartid)
 	return 0;
 }
 
-static int delegate_traps(struct sbi_scratch *scratch, u32 hartid)
+static int delegate_traps(struct sbi_scratch *scratch)
 {
 	const struct sbi_platform *plat = sbi_platform_ptr(scratch);
 	unsigned long interrupts, exceptions;
@@ -435,7 +435,7 @@ __mhpm_skip:
 		hfeatures->features |= SBI_HART_HAS_TIME;
 }
 
-int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
+int sbi_hart_init(struct sbi_scratch *scratch, bool cold_boot)
 {
 	int rc;
 
@@ -452,13 +452,13 @@ int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
 
 	hart_detect_features(scratch);
 
-	mstatus_init(scratch, hartid);
+	mstatus_init(scratch);
 
-	rc = fp_init(hartid);
+	rc = fp_init(scratch);
 	if (rc)
 		return rc;
 
-	rc = delegate_traps(scratch, hartid);
+	rc = delegate_traps(scratch);
 	if (rc)
 		return rc;
 
lib/sbi/sbi_init.c +2 -2
index 85a5f13..5cedb15 100644
@@ -182,7 +182,7 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
-	rc = sbi_hart_init(scratch, hartid, TRUE);
+	rc = sbi_hart_init(scratch, TRUE);
 	if (rc)
 		sbi_hart_hang();
 
@@ -250,7 +250,7 @@ static void __noreturn init_warmboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
-	rc = sbi_hart_init(scratch, hartid, FALSE);
+	rc = sbi_hart_init(scratch, FALSE);
 	if (rc)
 		sbi_hart_hang();
 
Signature · 1 line
2.25.1
#8 lore raw

PATCH 4/16 lib: sbi: Remove redundant hartid parameter from sbi_hart_init()

Quoted 107 lines · Anup Patel

The hartid parameter of sbi_hart_init() is not used anywhere in sbi_hart_init() implementation so let's remove it.

---
 include/sbi/sbi_hart.h |  2 +-
 lib/sbi/sbi_hart.c     | 14 +++++++-------
 lib/sbi/sbi_init.c     |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)
include/sbi/sbi_hart.h +1 -1
index 9207f4c..2efe44d 100644
@@ -27,7 +27,7 @@ enum sbi_hart_features {
 
 struct sbi_scratch;
 
-int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
+int sbi_hart_init(struct sbi_scratch *scratch, bool cold_boot);
 
 extern void (*sbi_hart_expected_trap)(void);
 static inline ulong sbi_hart_expected_trap_addr(void)
lib/sbi/sbi_hart.c +7 -7
index 7244f1c..88ffab0 100644
@@ -33,7 +33,7 @@ struct hart_features {
 };
 static unsigned long hart_features_offset;
 
-static void mstatus_init(struct sbi_scratch *scratch, u32 hartid)
+static void mstatus_init(struct sbi_scratch *scratch)
 {
        unsigned long mstatus_val = 0;
 
@@ -62,7 +62,7 @@ static void mstatus_init(struct sbi_scratch *scratch, u32 hartid)
                csr_write(CSR_SATP, 0);
 }
 
-static int fp_init(u32 hartid)
+static int fp_init(struct sbi_scratch *scratch)
 {
 #ifdef __riscv_flen
        int i;
@@ -83,7 +83,7 @@ static int fp_init(u32 hartid)
        return 0;
 }
 
-static int delegate_traps(struct sbi_scratch *scratch, u32 hartid)
+static int delegate_traps(struct sbi_scratch *scratch)
 {
        const struct sbi_platform *plat = sbi_platform_ptr(scratch);
        unsigned long interrupts, exceptions;
@@ -435,7 +435,7 @@ __mhpm_skip:
                hfeatures->features |= SBI_HART_HAS_TIME;
 }
 
-int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
+int sbi_hart_init(struct sbi_scratch *scratch, bool cold_boot)
 {
        int rc;
 
@@ -452,13 +452,13 @@ int sbi_hart_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot)
 
        hart_detect_features(scratch);
 
-       mstatus_init(scratch, hartid);
+       mstatus_init(scratch);
 
-       rc = fp_init(hartid);
+       rc = fp_init(scratch);
        if (rc)
                return rc;
 
-       rc = delegate_traps(scratch, hartid);
+       rc = delegate_traps(scratch);
        if (rc)
                return rc;
 
lib/sbi/sbi_init.c +2 -3
index 85a5f13..5cedb15 100644
@@ -182,7 +182,7 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
-       rc = sbi_hart_init(scratch, hartid, TRUE);
+       rc = sbi_hart_init(scratch, TRUE);
        if (rc)
                sbi_hart_hang();
 
@@ -250,7 +250,7 @@ static void __noreturn init_warmboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
-       rc = sbi_hart_init(scratch, hartid, FALSE);
+       rc = sbi_hart_init(scratch, FALSE);
        if (rc)
                sbi_hart_hang();
 
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
author
#9 lore raw

PATCH 5/16 include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

In reply to Anup Patel

The sbi_hartmask_test_hart() does not modify hartmask so the hartmask pointer parameter should be declared const.

---
 include/sbi/sbi_hartmask.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/sbi/sbi_hartmask.h +1 -1
index 77e42cb..f1cef0c 100644
@@ -67,7 +67,7 @@ static inline void sbi_hartmask_clear_hart(u32 h, struct sbi_hartmask *m)
  * @param h HART id to test
  * @param m the hartmask pointer
  */
-static inline int sbi_hartmask_test_hart(u32 h, struct sbi_hartmask *m)
+static inline int sbi_hartmask_test_hart(u32 h, const struct sbi_hartmask *m)
 {
 	if (h < SBI_HARTMASK_MAX_BITS)
 		return __test_bit(h, m->bits);
Signature · 1 line
2.25.1
#10 lore raw

PATCH 5/16 include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

Quoted 26 lines · Anup Patel

The sbi_hartmask_test_hart() does not modify hartmask so the hartmask pointer parameter should be declared const.

---
 include/sbi/sbi_hartmask.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/sbi/sbi_hartmask.h +1 -2
index 77e42cb..f1cef0c 100644
@@ -67,7 +67,7 @@ static inline void sbi_hartmask_clear_hart(u32 h, struct sbi_hartmask *m)
  * @param h HART id to test
  * @param m the hartmask pointer
  */
-static inline int sbi_hartmask_test_hart(u32 h, struct sbi_hartmask *m)
+static inline int sbi_hartmask_test_hart(u32 h, const struct sbi_hartmask *m)
 {
        if (h < SBI_HARTMASK_MAX_BITS)
                return __test_bit(h, m->bits);
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
#11 lore raw

PATCH 5/16 include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

In reply to Anup Patel
Quoted 1 line · Anup Patel

[PATCH 05/16] include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

Nit: "Mark ... as const"? Alternatively, "Make ... const"?

--Sean

author
#12 lore raw

PATCH 5/16 include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

Quoted 11 lines

-----Original Message-----
From: Sean Anderson <[email protected]>
Sent: 02 October 2020 21:37
To: Anup Patel <[email protected]>; Atish Patra <[email protected]>; Alistair Francis <[email protected]>
Cc: Anup Patel <[email protected]>; opensbi at lists.infradead.org
Subject: Re: [PATCH 05/16] include: sbi: Make hartmask pointer as const in sbi_hartmask_test_hart()

Quoted 5 lines
Quoted 1 line · Anup Patel

[PATCH 05/16] include: sbi: Make hartmask pointer as const in

sbi_hartmask_test_hart()

Nit: "Mark ... as const"? Alternatively, "Make ... const"?

Okay, I will update patch subject to "Mark hartmask pointer as const ..."

Regards,
Anup

author
#13 lore raw

PATCH 6/16 lib: sbi: Add initial domain support

In reply to Anup Patel

An OpenSBI domain is a logical entity representing a set of HARTs and a set of memory regions for these HARTs.

The OpenSBI domains support will allow OpenSBI platforms and previous booting stage (i.e. U-Boot SPL, Coreboot, etc) to partition a system into multiple domains where each domain will run it's own software.

For inter-domain isolation, OpenSBI will eventually use various HW features such as PMP, ePMP, IOPMP, SiFive shield, etc but initial implementation only use HW PMP support.

This patch provides initial implementation of OpenSBI domains where we have a root/default domain and OpenSBI platforms can provide non-root/custom domains using domain_get() callback.

---
 include/sbi/sbi_domain.h   | 138 ++++++++++++++
 include/sbi/sbi_platform.h |  20 ++
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 365 +++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_init.c         |  20 ++
 5 files changed, 544 insertions(+)
 create mode 100644 include/sbi/sbi_domain.h
 create mode 100644 lib/sbi/sbi_domain.c
include/sbi/sbi_domain.h new file +138 -0
new file mode 100644
index 0000000..a3f174c
@@ -0,0 +1,138 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#ifndef __SBI_DOMAIN_H__
+#define __SBI_DOMAIN_H__
+
+#include <sbi/sbi_types.h>
+#include <sbi/sbi_hartmask.h>
+
+struct sbi_scratch;
+
+/** Representation of OpenSBI domain memory region */
+struct sbi_domain_memregion {
+	/**
+	 * Size of memory region as power of 2
+	 * It has to be minimum 3 and maximum __riscv_xlen
+	 */
+	unsigned long order;
+	/**
+	 * Base address of memory region
+	 * It must be 2^order aligned address
+	 */
+	unsigned long base;
+	/** Access flags of memory region */
+#define SBI_DOMAIN_MEMREGION_READABLE		(1UL << 0)
+#define SBI_DOMAIN_MEMREGION_WRITEABLE		(1UL << 1)
+#define SBI_DOMAIN_MEMREGION_EXECUTABLE		(1UL << 2)
+#define SBI_DOMAIN_MEMREGION_MMIO		(1UL << 3)
+#define SBI_DOMAIN_MEMREGION_MMODE		(1UL << 4)
+	unsigned long flags;
+};
+
+/** Representation of OpenSBI domain */
+struct sbi_domain {
+	/**
+	 * Logical index of this domain
+	 * Note: This set by sbi_domain_finalize() in the coldboot path
+	 */
+	u32 index;
+	/**
+	 * HARTs assigned to this domain
+	 * Note: This set by sbi_domain_init() and sbi_domain_finalize()
+	 * in the coldboot path
+	 */
+	struct sbi_hartmask assigned_harts;
+	/** Name of this domain */
+	char name[64];
+	/** Possible HARTs in this domain */
+	const struct sbi_hartmask *possible_harts;
+	/** Array of memory regions terminated by a region with order zero */
+	struct sbi_domain_memregion *regions;
+	/** HART id of the HART booting this domain */
+	u32 boot_hartid;
+	/** Arg1 (or 'a1' register) of next booting stage for this domain */
+	unsigned long next_arg1;
+	/** Address of next booting stage for this domain */
+	unsigned long next_addr;
+	/** Priviledge mode of next booting stage for this domain */
+	unsigned long next_mode;
+	/** Is domain allowed to reset the system */
+	bool system_reset_allowed;
+};
+
+/** HART id to domain table */
+extern struct sbi_domain *hartid_to_domain_table[];
+
+/** Get pointer to sbi_domain from HART id */
+#define sbi_hartid_to_domain(__hartid) \
+	hartid_to_domain_table[__hartid]
+
+/** Get pointer to sbi_domain for current HART */
+#define sbi_domain_thishart_ptr() \
+	sbi_hartid_to_domain(current_hartid())
+
+/** Index to domain table */
+extern struct sbi_domain *domidx_to_domain_table[];
+
+/** Get pointer to sbi_domain from index */
+#define sbi_index_to_domain(__index) \
+	domidx_to_domain_table[__index]
+
+/** Iterate over each domain */
+#define sbi_domain_for_each(__i, __d) \
+	for ((__i) = 0; ((__d) = sbi_index_to_domain(__i)); (__i)++)
+
+/** Iterate over each memory regon of a domain */
+#define sbi_domain_for_each_memregion(__d, __r) \
+	for ((__r) = (__d)->regions; (__r)->order; (__r)++)
+
+/**
+ * Check whether given HART is assigned to specified domain
+ * @param dom pointer to domain
+ * @param hartid the HART ID
+ * @return TRUE if HART is assigned to domain otherwise FALSE
+ */
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid);
+
+/**
+ * Get ulong assigned HART mask for given domain and HART base ID
+ * @param dom pointer to domain
+ * @param hbase the HART base ID
+ * @return ulong possible HART mask
+ * Note: the return ulong mask will be set to zero on failure.
+ */
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase);
+
+/** Initialize a domain memory region as firmware region */
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg);
+
+/**
+ * Check whether we can access specified address for given mode and
+ * memory region flags under a domain
+ * @param dom pointer to domain
+ * @param addr the address to be checked
+ * @param mode the privilege mode of access
+ * @param mmio the memory type of access is MMIO
+ * @param read the access is read
+ * @param write the access is write
+ * @param execute the access is instruction execute
+ * @return TRUE if access allowed otherwise FALSE
+ */
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+			   unsigned long addr, unsigned long mode, bool mmio,
+			   bool read, bool write, bool execute);
+
+/** Finalize domain tables and startup non-root domains */
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid);
+
+/** Initialize domains */
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid);
+
+#endif
include/sbi/sbi_platform.h +20 -0
index f2c3237..e1355d8 100644
@@ -44,6 +44,7 @@
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_domain;
 struct sbi_trap_info;
 
 /** Possible feature flags of a platform */
@@ -89,6 +90,9 @@ struct sbi_platform_operations {
 	 */
 	int (*misa_get_xlen)(void);
 
+	/** Get domain pointer for given HART id */
+	struct sbi_domain *(*domain_get)(u32 hartid);
+
 	/** Write a character to the platform console output */
 	void (*console_putc)(char ch);
 	/** Read a character from the platform console input */
@@ -447,6 +451,22 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
 	return -1;
 }
 
+/**
+ * Get domain pointer for given HART
+ *
+ * @param plat pointer to struct sbi_platform
+ * @param hartid shorthand letter for CPU extensions
+ *
+ * @return non-NULL domain pointer on success and NULL on failure
+ */
+static inline struct sbi_domain *sbi_platform_domain_get(
+				const struct sbi_platform *plat, u32 hartid)
+{
+	if (plat && sbi_platform_ops(plat)->domain_get)
+		return sbi_platform_ops(plat)->domain_get(hartid);
+	return NULL;
+}
+
 /**
  * Write a character to the platform console output
  *
lib/sbi/objects.mk +1 -0
index fa808a0..6f2c06f 100644
@@ -15,6 +15,7 @@ libsbi-objs-y += riscv_locks.o
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o
+libsbi-objs-y += sbi_domain.o
 libsbi-objs-y += sbi_ecall.o
 libsbi-objs-y += sbi_ecall_base.o
 libsbi-objs-y += sbi_ecall_hsm.o
lib/sbi/sbi_domain.c new file +365 -0

365 lines not on this page · raw message

lib/sbi/sbi_init.c +20 -0
index 5cedb15..406cb3f 100644
@@ -12,6 +12,7 @@
 #include <sbi/riscv_barrier.h>
 #include <sbi/riscv_locks.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hartmask.h>
@@ -169,6 +170,11 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
+	/* Note: This has to be second thing in coldboot init sequence */
+	rc = sbi_domain_init(scratch, hartid);
+	if (rc)
+		sbi_hart_hang();
+
 	init_count_offset = sbi_scratch_alloc_offset(__SIZEOF_POINTER__,
 						     "INIT_COUNT");
 	if (!init_count_offset)
@@ -210,10 +216,24 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
 	if (rc)
 		sbi_hart_hang();
 
+	/*
+	 * Note: Finalize domains after HSM initialization so that we
+	 * can startup non-root domains.
+	 * Note: Finalize domains before HART PMP configuration so
+	 * that we use correct domain for configuring PMP.
+	 */
+	rc = sbi_domain_finalize(scratch, hartid);
+	if (rc)
+		sbi_hart_hang();
+
 	rc = sbi_hart_pmp_configure(scratch);
 	if (rc)
 		sbi_hart_hang();
 
+	/*
+	 * Note: Platform final initialization should be last so that
+	 * it sees correct domain assignment and PMP configuration.
+	 */
 	rc = sbi_platform_final_init(plat, TRUE);
 	if (rc)
 		sbi_hart_hang();
Signature · 1 line
2.25.1
#14 lore raw

PATCH 6/16 lib: sbi: Add initial domain support

Quoted 58 lines · Anup Patel

An OpenSBI domain is a logical entity representing a set of HARTs and a set of memory regions for these HARTs.

The OpenSBI domains support will allow OpenSBI platforms and previous booting stage (i.e. U-Boot SPL, Coreboot, etc) to partition a system into multiple domains where each domain will run it's own software.

For inter-domain isolation, OpenSBI will eventually use various HW features such as PMP, ePMP, IOPMP, SiFive shield, etc but initial implementation only use HW PMP support.

This patch provides initial implementation of OpenSBI domains where we have a root/default domain and OpenSBI platforms can provide non-root/custom domains using domain_get() callback.

---
 include/sbi/sbi_domain.h   | 138 ++++++++++++++
 include/sbi/sbi_platform.h |  20 ++
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 365 +++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_init.c         |  20 ++
 5 files changed, 544 insertions(+)
 create mode 100644 include/sbi/sbi_domain.h
 create mode 100644 lib/sbi/sbi_domain.c
include/sbi/sbi_domain.h new file +25 -0
new file mode 100644
index 0000000..a3f174c
@@ -0,0 +1,138 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#ifndef __SBI_DOMAIN_H__
+#define __SBI_DOMAIN_H__
+
+#include <sbi/sbi_types.h>
+#include <sbi/sbi_hartmask.h>
+
+struct sbi_scratch;
+
+/** Representation of OpenSBI domain memory region */
+struct sbi_domain_memregion {
+       /**
+        * Size of memory region as power of 2
+        * It has to be minimum 3 and maximum __riscv_xlen
+        */
+       unsigned long order;
+       /**

include/sbi/sbi_domain.h line 35

Quoted 10 lines
include/sbi/sbi_domain.h new file +10 -0
+        * Base address of memory region
+        * It must be 2^order aligned address
+        */
+       unsigned long base;
+       /** Access flags of memory region */
+#define SBI_DOMAIN_MEMREGION_READABLE          (1UL << 0)
+#define SBI_DOMAIN_MEMREGION_WRITEABLE         (1UL << 1)
+#define SBI_DOMAIN_MEMREGION_EXECUTABLE                (1UL << 2)
+#define SBI_DOMAIN_MEMREGION_MMIO              (1UL << 3)
+#define SBI_DOMAIN_MEMREGION_MMODE             (1UL << 4)

We won't need this if we don't support M-mode domain software. Correct ?

Quoted 19 lines
include/sbi/sbi_domain.h new file +19 -0
+       unsigned long flags;
+};
+
+/** Representation of OpenSBI domain */
+struct sbi_domain {
+       /**
+        * Logical index of this domain
+        * Note: This set by sbi_domain_finalize() in the coldboot path
+        */
+       u32 index;
+       /**
+        * HARTs assigned to this domain
+        * Note: This set by sbi_domain_init() and sbi_domain_finalize()
+        * in the coldboot path
+        */
+       struct sbi_hartmask assigned_harts;
+       /** Name of this domain */
+       char name[64];
+       /** Possible HARTs in this domain */

include/sbi/sbi_domain.h line 64

Quoted 10 lines
include/sbi/sbi_domain.h new file +10 -0
+       const struct sbi_hartmask *possible_harts;
+       /** Array of memory regions terminated by a region with order zero */
+       struct sbi_domain_memregion *regions;
+       /** HART id of the HART booting this domain */
+       u32 boot_hartid;
+       /** Arg1 (or 'a1' register) of next booting stage for this domain */
+       unsigned long next_arg1;
+       /** Address of next booting stage for this domain */
+       unsigned long next_addr;
+       /** Priviledge mode of next booting stage for this domain */

/Priviledge/Privilege

Quoted 18 lines
include/sbi/sbi_domain.h new file +18 -0
+       unsigned long next_mode;
+       /** Is domain allowed to reset the system */
+       bool system_reset_allowed;
+};
+
+/** HART id to domain table */
+extern struct sbi_domain *hartid_to_domain_table[];
+
+/** Get pointer to sbi_domain from HART id */
+#define sbi_hartid_to_domain(__hartid) \
+       hartid_to_domain_table[__hartid]
+
+/** Get pointer to sbi_domain for current HART */
+#define sbi_domain_thishart_ptr() \
+       sbi_hartid_to_domain(current_hartid())
+
+/** Index to domain table */
+extern struct sbi_domain *domidx_to_domain_table[];

include/sbi/sbi_domain.h line 92

Quoted 10 lines
include/sbi/sbi_domain.h new file +10 -0
+
+/** Get pointer to sbi_domain from index */
+#define sbi_index_to_domain(__index) \
+       domidx_to_domain_table[__index]
+
+/** Iterate over each domain */
+#define sbi_domain_for_each(__i, __d) \
+       for ((__i) = 0; ((__d) = sbi_index_to_domain(__i)); (__i)++)
+
+/** Iterate over each memory regon of a domain */

regon/region

include/sbi/sbi_domain.h line 103

Quoted 11 lines
include/sbi/sbi_domain.h new file +11 -0
+#define sbi_domain_for_each_memregion(__d, __r) \
+       for ((__r) = (__d)->regions; (__r)->order; (__r)++)
+
+/**
+ * Check whether given HART is assigned to specified domain
+ * @param dom pointer to domain
+ * @param hartid the HART ID
+ * @return TRUE if HART is assigned to domain otherwise FALSE
+ */
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid);
+

May be a better name is
sbi_domain_is_assigned_hart ?

include/sbi/sbi_domain.h line 105

Quoted 2 lines
include/sbi/sbi_domain.h new file +2 -0
+/**
+ * Get ulong assigned HART mask for given domain and HART base ID

Get assigned HART mask in ulong for given domain and HART base ID

include/sbi/sbi_domain.h line 112

Quoted 7 lines
include/sbi/sbi_domain.h new file +7 -0
+ * @param dom pointer to domain
+ * @param hbase the HART base ID
+ * @return ulong possible HART mask
+ * Note: the return ulong mask will be set to zero on failure.
+ */
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase);
+

Maybe a better name is
sbi_domain_get_assigned_hartmask ?

Quoted 9 lines
include/sbi/sbi_domain.h new file +9 -0
+/** Initialize a domain memory region as firmware region */
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg);
+
+/**
+ * Check whether we can access specified address for given mode and
+ * memory region flags under a domain
+ * @param dom pointer to domain
+ * @param addr the address to be checked
+ * @param mode the privilege mode of access

include/sbi/sbi_domain.h line 131

Quoted 10 lines
include/sbi/sbi_domain.h new file +10 -0
+ * @param mmio the memory type of access is MMIO
+ * @param read the access is read
+ * @param write the access is write
+ * @param execute the access is instruction execute
+ * @return TRUE if access allowed otherwise FALSE
+ */
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute);
+

We can reduce the last 3 arguments to 1 if the caller passes one value after ORing.

Quoted 80 lines
include/sbi/sbi_domain.h new file +7 -0
+/** Finalize domain tables and startup non-root domains */
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid);
+
+/** Initialize domains */
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid);
+
+#endif
include/sbi/sbi_platform.h +20 -0
index f2c3237..e1355d8 100644
@@ -44,6 +44,7 @@
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_domain;
 struct sbi_trap_info;
 
 /** Possible feature flags of a platform */
@@ -89,6 +90,9 @@ struct sbi_platform_operations {
         */
        int (*misa_get_xlen)(void);
 
+       /** Get domain pointer for given HART id */
+       struct sbi_domain *(*domain_get)(u32 hartid);
+
        /** Write a character to the platform console output */
        void (*console_putc)(char ch);
        /** Read a character from the platform console input */
@@ -447,6 +451,22 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
        return -1;
 }
 
+/**
+ * Get domain pointer for given HART
+ *
+ * @param plat pointer to struct sbi_platform
+ * @param hartid shorthand letter for CPU extensions
+ *
+ * @return non-NULL domain pointer on success and NULL on failure
+ */
+static inline struct sbi_domain *sbi_platform_domain_get(
+                               const struct sbi_platform *plat, u32 hartid)
+{
+       if (plat && sbi_platform_ops(plat)->domain_get)
+               return sbi_platform_ops(plat)->domain_get(hartid);
+       return NULL;
+}
+
 /**
  * Write a character to the platform console output
  *
lib/sbi/objects.mk +1 -0
index fa808a0..6f2c06f 100644
@@ -15,6 +15,7 @@ libsbi-objs-y += riscv_locks.o
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o
+libsbi-objs-y += sbi_domain.o
 libsbi-objs-y += sbi_ecall.o
 libsbi-objs-y += sbi_ecall_base.o
 libsbi-objs-y += sbi_ecall_hsm.o
lib/sbi/sbi_domain.c new file +10 -0
new file mode 100644
index 0000000..8d5dae1
@@ -0,0 +1,365 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#include <sbi/riscv_asm.h>

lib/sbi/sbi_domain.c line 20

Quoted 10 lines
lib/sbi/sbi_domain.c new file +10 -0
+#include <sbi/sbi_domain.h>
+#include <sbi/sbi_hartmask.h>
+#include <sbi/sbi_hsm.h>
+#include <sbi/sbi_math.h>
+#include <sbi/sbi_platform.h>
+#include <sbi/sbi_scratch.h>
+#include <sbi/sbi_string.h>
+
+struct sbi_domain *hartid_to_domain_table[SBI_HARTMASK_MAX_BITS] = { 0 };
+struct sbi_domain *domidx_to_domain_table[SBI_HARTMASK_MAX_BITS + 1] = { 0 };

Do we really need 128 as the upper range of domain index ? I think it is better to define a smaller value(8 or 16 should be reasonable) in a different macro similar to SBI_DOM_IDX_MAX_BITS.

Quoted 157 lines
lib/sbi/sbi_domain.c new file +157 -0
+
+static u32 domain_count = 0;
+
+static struct sbi_hartmask root_hmask = { 0 };
+
+#define ROOT_FW_REGION         0
+#define ROOT_ALL_REGION        1
+#define ROOT_END_REGION        2
+static struct sbi_domain_memregion root_memregs[ROOT_END_REGION + 1] = { 0 };
+
+static struct sbi_domain root = {
+       .name = "root",
+       .possible_harts = &root_hmask,
+       .regions = root_memregs,
+       .system_reset_allowed = TRUE,
+};
+
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid)
+{
+       if (dom)
+               return sbi_hartmask_test_hart(hartid, &dom->assigned_harts);
+
+       return FALSE;
+}
+
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase)
+{
+       ulong ret, bword, boff;
+
+       if (!dom)
+               return 0;
+
+       bword = BIT_WORD(hbase);
+       boff = BIT_WORD_OFFSET(hbase);
+
+       ret = sbi_hartmask_bits(&dom->assigned_harts)[bword++] >> boff;
+       if (boff && bword < BIT_WORD(SBI_HARTMASK_MAX_BITS)) {
+               ret |= (sbi_hartmask_bits(&dom->assigned_harts)[bword] &
+                       (BIT(boff) - 1UL)) << (BITS_PER_LONG - boff);
+       }
+
+       return ret;
+}
+
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg)
+{
+       if (!reg)
+               return;
+
+       sbi_memcpy(reg, &root_memregs[ROOT_FW_REGION], sizeof(*reg));
+}
+
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute)
+{
+       struct sbi_domain_memregion *reg;
+       unsigned long rstart, rend, rflags, rwx = 0;
+
+       if (!dom)
+               return FALSE;
+
+       if (read)
+               rwx |= SBI_DOMAIN_MEMREGION_READABLE;
+       if (write)
+               rwx |= SBI_DOMAIN_MEMREGION_WRITEABLE;
+       if (execute)
+               rwx |= SBI_DOMAIN_MEMREGION_EXECUTABLE;
+
+       sbi_domain_for_each_memregion(dom, reg) {
+               rflags = reg->flags;
+               if (mode == PRV_M && !(rflags & SBI_DOMAIN_MEMREGION_MMODE))
+                       continue;
+
+               rstart = reg->base;
+               rend = (reg->order < __riscv_xlen) ?
+                       rstart + ((1UL << reg->order) - 1) : -1UL;
+               if (rstart <= addr && addr <= rend) {
+                       if ((mmio && !(rflags & SBI_DOMAIN_MEMREGION_MMIO)) ||
+                           (!mmio && (rflags & SBI_DOMAIN_MEMREGION_MMIO)))
+                               return FALSE;
+                       return ((rflags & rwx) == rwx) ? TRUE : FALSE;
+               }
+       }
+
+       return (mode == PRV_M) ? TRUE : FALSE;
+}
+
+/* Check if region comply with constraints */
+static bool is_region_valid(const struct sbi_domain_memregion *reg)
+{
+       if (reg->order < 3 || __riscv_xlen < reg->order)
+               return FALSE;
+
+       if (reg->base & (BIT(reg->order) - 1))
+               return FALSE;
+
+       return TRUE;
+}
+
+/** Check if regionA is sub-region of regionB */
+static bool is_region_subset(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       ulong regA_start = regA->base;
+       ulong regA_end = regA->base + (BIT(regA->order) - 1);
+       ulong regB_start = regB->base;
+       ulong regB_end = regB->base + (BIT(regA->order) - 1);
+
+       if ((regB_start <= regA_start) &&
+           (regA_start < regB_end) &&
+           (regB_start < regA_end) &&
+           (regA_end <= regB_end))
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA conflicts regionB */
+static bool is_region_conflict(const struct sbi_domain_memregion *regA,
+                               const struct sbi_domain_memregion *regB)
+{
+       if ((is_region_subset(regA, regB) || is_region_subset(regB, regA)) &&
+           regA->flags == regB->flags)
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA should be placed before regionB */
+static bool is_region_before(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       if (regA->order < regB->order)
+               return TRUE;
+
+       if ((regA->order == regB->order) &&
+           (regA->base < regB->base))
+               return TRUE;
+
+       return FALSE;
+}
+
+static int sanitize_domain(const struct sbi_platform *plat,
+                          struct sbi_domain *dom)
+{
+       u32 i, j, count;
+       bool have_fw_reg;
+       struct sbi_domain_memregion treg, *reg, *reg1;
+
+       /* Check possible HARTs */
+       if (!dom->possible_harts)
+               return SBI_EINVAL;
+       sbi_hartmask_for_each_hart(i, dom->possible_harts) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       return SBI_EINVAL;
+       };

lib/sbi/sbi_domain.c line 187

Quoted 10 lines
lib/sbi/sbi_domain.c new file +10 -0
+
+       /* Check memory regions */
+       if (!dom->regions)
+               return SBI_EINVAL;
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (!is_region_valid(reg))
+                       return SBI_EINVAL;
+       }
+
+       /* Count memory regions and check precense of firmware region */

precense/presence

Quoted 231 lines
lib/sbi/sbi_domain.c new file +178 -0
+       count = 0;
+       have_fw_reg = FALSE;
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (reg->order == root_memregs[ROOT_FW_REGION].order &&
+                   reg->base == root_memregs[ROOT_FW_REGION].base &&
+                   reg->flags == root_memregs[ROOT_FW_REGION].flags)
+                       have_fw_reg = TRUE;
+               count++;
+       }
+       if (!have_fw_reg)
+               return SBI_EINVAL;
+
+       /* Sort the memory regions */
+       for (i = 0; i < (count - 1); i++) {
+               reg = &dom->regions[i];
+               for (j = i + 1; j < count; j++) {
+                       reg1 = &dom->regions[j];
+
+                       if (is_region_conflict(reg1, reg))
+                               return SBI_EINVAL;
+
+                       if (!is_region_before(reg1, reg))
+                               continue;
+
+                       sbi_memcpy(&treg, reg1, sizeof(treg));
+                       sbi_memcpy(reg1, reg, sizeof(treg));
+                       sbi_memcpy(reg, &treg, sizeof(treg));
+               }
+       }
+
+       /*
+        * We don't need to check boot HART id of domain because if boot
+        * HART id is not possible/assigned to this domain then it won't
+        * be started at boot-time by sbi_domain_finalize().
+        */
+
+       /* Check next mode */
+       if (dom->next_mode != PRV_M &&
+           dom->next_mode != PRV_S &&
+           dom->next_mode != PRV_U)
+               return SBI_EINVAL;
+
+       /* Check next address and next mode*/
+       if (!sbi_domain_check_addr(dom, dom->next_addr, dom->next_mode,
+                                  FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
+
+       return 0;
+}
+
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       int rc;
+       u32 i, j, dhart;
+       bool dom_exists;
+       struct sbi_domain *dom, *tdom;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Discover domains */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               /* Ignore invalid HART */
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+
+               /* Get domain assigned to HART */
+               dom = sbi_platform_domain_get(plat, i);
+               if (!dom)
+                       continue;
+
+               /* Check if domain already discovered */
+               dom_exists = FALSE;
+               sbi_domain_for_each(j, tdom) {
+                       if (tdom == dom) {
+                               dom_exists = TRUE;
+                               break;
+                       }
+               }
+
+               /* Newly discovered domain */
+               if (!dom_exists) {
+                       /* Sanitize discovered domain */
+                       rc = sanitize_domain(plat, dom);
+                       if (rc)
+                               return rc;
+
+                       /* Assign index to domain */
+                       dom->index = domain_count++;
+                       domidx_to_domain_table[dom->index] = dom;
+
+                       /* Clear assigned HARTs of domain */
+                       sbi_hartmask_clear_all(&dom->assigned_harts);
+               }
+
+               /* Assign domain to HART if HART is a possible HART */
+               if (sbi_hartmask_test_hart(i, dom->possible_harts)) {
+                       tdom = hartid_to_domain_table[i];
+                       if (tdom)
+                               sbi_hartmask_clear_hart(i,
+                                               &tdom->assigned_harts);
+                       hartid_to_domain_table[i] = dom;
+                       sbi_hartmask_set_hart(i, &dom->assigned_harts);
+               }
+       }
+
+       /* Startup boot HART of domains */
+       sbi_domain_for_each(i, dom) {
+               /* Domain boot HART */
+               dhart = dom->boot_hartid;
+
+               /* Ignore if boot HART not possible for this domain */
+               if (!sbi_hartmask_test_hart(i, dom->possible_harts))
+                       continue;
+
+               /* Ignore if boot HART assigned different domain */
+               if (sbi_hartid_to_domain(dhart) != dom ||
+                   !sbi_hartmask_test_hart(i, &dom->assigned_harts))
+                       continue;
+
+               /* Startup boot HART of domain */
+               if (dhart == cold_hartid) {
+                       scratch->next_addr = dom->next_addr;
+                       scratch->next_mode = dom->next_mode;
+                       scratch->next_arg1 = dom->next_arg1;
+               } else {
+                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
+                                       dom->next_mode, dom->next_arg1);
+                       if (rc)
+                               return rc;
+               }
+       }
+
+       return 0;
+}
+
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       u32 i;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Root domain firmware memory region */
+       root_memregs[ROOT_FW_REGION].order = log2roundup(scratch->fw_size);
+       root_memregs[ROOT_FW_REGION].base = scratch->fw_start &
+                               ~((1UL << root_memregs[0].order) - 1UL);
+       root_memregs[ROOT_FW_REGION].flags = 0;
+
+       /* Root domain allow everything memory region */
+       root_memregs[ROOT_ALL_REGION].order = __riscv_xlen;
+       root_memregs[ROOT_ALL_REGION].base = 0;
+       root_memregs[ROOT_ALL_REGION].flags = (SBI_DOMAIN_MEMREGION_READABLE |
+                                               SBI_DOMAIN_MEMREGION_WRITEABLE |
+                                               SBI_DOMAIN_MEMREGION_EXECUTABLE);
+
+       /* Root domain memory region end */
+       root_memregs[ROOT_END_REGION].order = 0;
+
+       /* Root domain boot HART id is same as coldboot HART id */
+       root.boot_hartid = cold_hartid;
+
+       /* Root domain next booting stage details */
+       root.next_arg1 = scratch->next_arg1;
+       root.next_addr = scratch->next_addr;
+       root.next_mode = scratch->next_mode;
+
+       /* Select root domain for all valid HARTs */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+               sbi_hartmask_set_hart(i, &root_hmask);
+               hartid_to_domain_table[i] = &root;
+               sbi_hartmask_set_hart(i, &root.assigned_harts);
+       }
+
+       /* Set root domain index */
+       root.index = domain_count++;
+       domidx_to_domain_table[root.index] = &root;
+
+       return 0;
+}
lib/sbi/sbi_init.c +20 -1
index 5cedb15..406cb3f 100644
@@ -12,6 +12,7 @@
 #include <sbi/riscv_barrier.h>
 #include <sbi/riscv_locks.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hartmask.h>
@@ -169,6 +170,11 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /* Note: This has to be second thing in coldboot init sequence */
+       rc = sbi_domain_init(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        init_count_offset = sbi_scratch_alloc_offset(__SIZEOF_POINTER__,
                                                     "INIT_COUNT");
        if (!init_count_offset)
@@ -210,10 +216,24 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Finalize domains after HSM initialization so that we
+        * can startup non-root domains.
+        * Note: Finalize domains before HART PMP configuration so
+        * that we use correct domain for configuring PMP.
+        */
+       rc = sbi_domain_finalize(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_hart_pmp_configure(scratch);
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Platform final initialization should be last so that
+        * it sees correct domain assignment and PMP configuration.
+        */
        rc = sbi_platform_final_init(plat, TRUE);
        if (rc)
                sbi_hart_hang();
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

#15 lore raw

PATCH 6/16 lib: sbi: Add initial domain support

Quoted 64 lines · Atish Patra
Quoted 62 lines · Anup Patel

An OpenSBI domain is a logical entity representing a set of HARTs and a set of memory regions for these HARTs.

The OpenSBI domains support will allow OpenSBI platforms and previous booting stage (i.e. U-Boot SPL, Coreboot, etc) to partition a system into multiple domains where each domain will run it's own software.

For inter-domain isolation, OpenSBI will eventually use various HW features such as PMP, ePMP, IOPMP, SiFive shield, etc but initial implementation only use HW PMP support.

This patch provides initial implementation of OpenSBI domains where we have a root/default domain and OpenSBI platforms can provide non-root/custom domains using domain_get() callback.

---
 include/sbi/sbi_domain.h   | 138 ++++++++++++++
 include/sbi/sbi_platform.h |  20 ++
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 365 +++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_init.c         |  20 ++
 5 files changed, 544 insertions(+)
 create mode 100644 include/sbi/sbi_domain.h
 create mode 100644 lib/sbi/sbi_domain.c
include/sbi/sbi_domain.h new file +29 -0
new file mode 100644
index 0000000..a3f174c
@@ -0,0 +1,138 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#ifndef __SBI_DOMAIN_H__
+#define __SBI_DOMAIN_H__
+
+#include <sbi/sbi_types.h>
+#include <sbi/sbi_hartmask.h>
+
+struct sbi_scratch;
+
+/** Representation of OpenSBI domain memory region */
+struct sbi_domain_memregion {
+       /**
+        * Size of memory region as power of 2
+        * It has to be minimum 3 and maximum __riscv_xlen
+        */
+       unsigned long order;
+       /**
+        * Base address of memory region
+        * It must be 2^order aligned address
+        */
+       unsigned long base;
Quoted 8 lines

include/sbi/sbi_domain.h line 35

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+       /** Access flags of memory region */
+#define SBI_DOMAIN_MEMREGION_READABLE          (1UL << 0)
+#define SBI_DOMAIN_MEMREGION_WRITEABLE         (1UL << 1)
+#define SBI_DOMAIN_MEMREGION_EXECUTABLE                (1UL << 2)
+#define SBI_DOMAIN_MEMREGION_MMIO              (1UL << 3)
+#define SBI_DOMAIN_MEMREGION_MMODE             (1UL << 4)

We won't need this if we don't support M-mode domain software. Correct ?

This flag is for a memregion which requires access checks for OpenSBI.

It's not related to the next stage being M-mode.

Quoted 24 lines
Quoted 24 lines
include/sbi/sbi_domain.h new file +23 -0
+       unsigned long flags;
+};
+
+/** Representation of OpenSBI domain */
+struct sbi_domain {
+       /**
+        * Logical index of this domain
+        * Note: This set by sbi_domain_finalize() in the coldboot path
+        */
+       u32 index;
+       /**
+        * HARTs assigned to this domain
+        * Note: This set by sbi_domain_init() and sbi_domain_finalize()
+        * in the coldboot path
+        */
+       struct sbi_hartmask assigned_harts;
+       /** Name of this domain */
+       char name[64];
+       /** Possible HARTs in this domain */
+       const struct sbi_hartmask *possible_harts;
+       /** Array of memory regions terminated by a region with order zero */
+       struct sbi_domain_memregion *regions;
+       /** HART id of the HART booting this domain */
Quoted 8 lines

include/sbi/sbi_domain.h line 64

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+       u32 boot_hartid;
+       /** Arg1 (or 'a1' register) of next booting stage for this domain */
+       unsigned long next_arg1;
+       /** Address of next booting stage for this domain */
+       unsigned long next_addr;
+       /** Priviledge mode of next booting stage for this domain */

/Priviledge/Privilege

Okay, will update.

Quoted 23 lines
Quoted 23 lines
include/sbi/sbi_domain.h new file +22 -0
+       unsigned long next_mode;
+       /** Is domain allowed to reset the system */
+       bool system_reset_allowed;
+};
+
+/** HART id to domain table */
+extern struct sbi_domain *hartid_to_domain_table[];
+
+/** Get pointer to sbi_domain from HART id */
+#define sbi_hartid_to_domain(__hartid) \
+       hartid_to_domain_table[__hartid]
+
+/** Get pointer to sbi_domain for current HART */
+#define sbi_domain_thishart_ptr() \
+       sbi_hartid_to_domain(current_hartid())
+
+/** Index to domain table */
+extern struct sbi_domain *domidx_to_domain_table[];
+
+/** Get pointer to sbi_domain from index */
+#define sbi_index_to_domain(__index) \
+       domidx_to_domain_table[__index]
Quoted 8 lines

include/sbi/sbi_domain.h line 92

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+
+/** Iterate over each domain */
+#define sbi_domain_for_each(__i, __d) \
+       for ((__i) = 0; ((__d) = sbi_index_to_domain(__i)); (__i)++)
+
+/** Iterate over each memory regon of a domain */

regon/region

Okay, will update.

Quoted 14 lines

include/sbi/sbi_domain.h line 103

Quoted 12 lines
include/sbi/sbi_domain.h new file +11 -0
+#define sbi_domain_for_each_memregion(__d, __r) \
+       for ((__r) = (__d)->regions; (__r)->order; (__r)++)
+
+/**
+ * Check whether given HART is assigned to specified domain
+ * @param dom pointer to domain
+ * @param hartid the HART ID
+ * @return TRUE if HART is assigned to domain otherwise FALSE
+ */
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid);
+

May be a better name is
sbi_domain_is_assigned_hart ?

Okay, will rename.

Quoted 6 lines

include/sbi/sbi_domain.h line 105

Quoted 3 lines
include/sbi/sbi_domain.h new file +2 -0
+/**
+ * Get ulong assigned HART mask for given domain and HART base ID

Get assigned HART mask in ulong for given domain and HART base ID

Quoted 10 lines

include/sbi/sbi_domain.h line 112

Quoted 7 lines
include/sbi/sbi_domain.h new file +7 -0
+ * @param dom pointer to domain
+ * @param hbase the HART base ID
+ * @return ulong possible HART mask
+ * Note: the return ulong mask will be set to zero on failure.
+ */
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase);
+

Maybe a better name is
sbi_domain_get_assigned_hartmask ?

Okay, will rename.

Quoted 16 lines
Quoted 16 lines
include/sbi/sbi_domain.h new file +14 -0
+/** Initialize a domain memory region as firmware region */
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg);
+
+/**
+ * Check whether we can access specified address for given mode and
+ * memory region flags under a domain
+ * @param dom pointer to domain
+ * @param addr the address to be checked
+ * @param mode the privilege mode of access
+ * @param mmio the memory type of access is MMIO
+ * @param read the access is read
+ * @param write the access is write
+ * @param execute the access is instruction execute
+ * @return TRUE if access allowed otherwise FALSE
Quoted 8 lines

include/sbi/sbi_domain.h line 131

Quoted 5 lines
include/sbi/sbi_domain.h new file +5 -0
+ */
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute);
+

We can reduce the last 3 arguments to 1 if the caller passes one value after ORing.

For making it one argument, we will have to define separate flags. Let's have three parameters for now.

Quoted 87 lines
Quoted 87 lines
include/sbi/sbi_domain.h new file +7 -0
+/** Finalize domain tables and startup non-root domains */
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid);
+
+/** Initialize domains */
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid);
+
+#endif
include/sbi/sbi_platform.h +20 -0
index f2c3237..e1355d8 100644
@@ -44,6 +44,7 @@
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_domain;
 struct sbi_trap_info;
 
 /** Possible feature flags of a platform */
@@ -89,6 +90,9 @@ struct sbi_platform_operations {
         */
        int (*misa_get_xlen)(void);
 
+       /** Get domain pointer for given HART id */
+       struct sbi_domain *(*domain_get)(u32 hartid);
+
        /** Write a character to the platform console output */
        void (*console_putc)(char ch);
        /** Read a character from the platform console input */
@@ -447,6 +451,22 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
        return -1;
 }
 
+/**
+ * Get domain pointer for given HART
+ *
+ * @param plat pointer to struct sbi_platform
+ * @param hartid shorthand letter for CPU extensions
+ *
+ * @return non-NULL domain pointer on success and NULL on failure
+ */
+static inline struct sbi_domain *sbi_platform_domain_get(
+                               const struct sbi_platform *plat, u32 hartid)
+{
+       if (plat && sbi_platform_ops(plat)->domain_get)
+               return sbi_platform_ops(plat)->domain_get(hartid);
+       return NULL;
+}
+
 /**
  * Write a character to the platform console output
  *
lib/sbi/objects.mk +1 -0
index fa808a0..6f2c06f 100644
@@ -15,6 +15,7 @@ libsbi-objs-y += riscv_locks.o
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o
+libsbi-objs-y += sbi_domain.o
 libsbi-objs-y += sbi_ecall.o
 libsbi-objs-y += sbi_ecall_base.o
 libsbi-objs-y += sbi_ecall_hsm.o
lib/sbi/sbi_domain.c new file +16 -0
new file mode 100644
index 0000000..8d5dae1
@@ -0,0 +1,365 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#include <sbi/riscv_asm.h>
+#include <sbi/sbi_domain.h>
+#include <sbi/sbi_hartmask.h>
+#include <sbi/sbi_hsm.h>
+#include <sbi/sbi_math.h>
+#include <sbi/sbi_platform.h>
+#include <sbi/sbi_scratch.h>
Quoted 8 lines

lib/sbi/sbi_domain.c line 20

Quoted 4 lines
lib/sbi/sbi_domain.c new file +4 -0
+#include <sbi/sbi_string.h>
+
+struct sbi_domain *hartid_to_domain_table[SBI_HARTMASK_MAX_BITS] = { 0 };
+struct sbi_domain *domidx_to_domain_table[SBI_HARTMASK_MAX_BITS + 1] = { 0 };

Do we really need 128 as the upper range of domain index ? I think it is better to define a smaller value(8 or 16 should be reasonable) in a different macro similar to SBI_DOM_IDX_MAX_BITS.

Okay, I will add SBI_DOMAIN_MAX_IDX define.

Quoted 162 lines
Quoted 162 lines
lib/sbi/sbi_domain.c new file +161 -0
+
+static u32 domain_count = 0;
+
+static struct sbi_hartmask root_hmask = { 0 };
+
+#define ROOT_FW_REGION         0
+#define ROOT_ALL_REGION        1
+#define ROOT_END_REGION        2
+static struct sbi_domain_memregion root_memregs[ROOT_END_REGION + 1] = { 0 };
+
+static struct sbi_domain root = {
+       .name = "root",
+       .possible_harts = &root_hmask,
+       .regions = root_memregs,
+       .system_reset_allowed = TRUE,
+};
+
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid)
+{
+       if (dom)
+               return sbi_hartmask_test_hart(hartid, &dom->assigned_harts);
+
+       return FALSE;
+}
+
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase)
+{
+       ulong ret, bword, boff;
+
+       if (!dom)
+               return 0;
+
+       bword = BIT_WORD(hbase);
+       boff = BIT_WORD_OFFSET(hbase);
+
+       ret = sbi_hartmask_bits(&dom->assigned_harts)[bword++] >> boff;
+       if (boff && bword < BIT_WORD(SBI_HARTMASK_MAX_BITS)) {
+               ret |= (sbi_hartmask_bits(&dom->assigned_harts)[bword] &
+                       (BIT(boff) - 1UL)) << (BITS_PER_LONG - boff);
+       }
+
+       return ret;
+}
+
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg)
+{
+       if (!reg)
+               return;
+
+       sbi_memcpy(reg, &root_memregs[ROOT_FW_REGION], sizeof(*reg));
+}
+
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute)
+{
+       struct sbi_domain_memregion *reg;
+       unsigned long rstart, rend, rflags, rwx = 0;
+
+       if (!dom)
+               return FALSE;
+
+       if (read)
+               rwx |= SBI_DOMAIN_MEMREGION_READABLE;
+       if (write)
+               rwx |= SBI_DOMAIN_MEMREGION_WRITEABLE;
+       if (execute)
+               rwx |= SBI_DOMAIN_MEMREGION_EXECUTABLE;
+
+       sbi_domain_for_each_memregion(dom, reg) {
+               rflags = reg->flags;
+               if (mode == PRV_M && !(rflags & SBI_DOMAIN_MEMREGION_MMODE))
+                       continue;
+
+               rstart = reg->base;
+               rend = (reg->order < __riscv_xlen) ?
+                       rstart + ((1UL << reg->order) - 1) : -1UL;
+               if (rstart <= addr && addr <= rend) {
+                       if ((mmio && !(rflags & SBI_DOMAIN_MEMREGION_MMIO)) ||
+                           (!mmio && (rflags & SBI_DOMAIN_MEMREGION_MMIO)))
+                               return FALSE;
+                       return ((rflags & rwx) == rwx) ? TRUE : FALSE;
+               }
+       }
+
+       return (mode == PRV_M) ? TRUE : FALSE;
+}
+
+/* Check if region comply with constraints */
+static bool is_region_valid(const struct sbi_domain_memregion *reg)
+{
+       if (reg->order < 3 || __riscv_xlen < reg->order)
+               return FALSE;
+
+       if (reg->base & (BIT(reg->order) - 1))
+               return FALSE;
+
+       return TRUE;
+}
+
+/** Check if regionA is sub-region of regionB */
+static bool is_region_subset(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       ulong regA_start = regA->base;
+       ulong regA_end = regA->base + (BIT(regA->order) - 1);
+       ulong regB_start = regB->base;
+       ulong regB_end = regB->base + (BIT(regA->order) - 1);
+
+       if ((regB_start <= regA_start) &&
+           (regA_start < regB_end) &&
+           (regB_start < regA_end) &&
+           (regA_end <= regB_end))
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA conflicts regionB */
+static bool is_region_conflict(const struct sbi_domain_memregion *regA,
+                               const struct sbi_domain_memregion *regB)
+{
+       if ((is_region_subset(regA, regB) || is_region_subset(regB, regA)) &&
+           regA->flags == regB->flags)
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA should be placed before regionB */
+static bool is_region_before(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       if (regA->order < regB->order)
+               return TRUE;
+
+       if ((regA->order == regB->order) &&
+           (regA->base < regB->base))
+               return TRUE;
+
+       return FALSE;
+}
+
+static int sanitize_domain(const struct sbi_platform *plat,
+                          struct sbi_domain *dom)
+{
+       u32 i, j, count;
+       bool have_fw_reg;
+       struct sbi_domain_memregion treg, *reg, *reg1;
+
+       /* Check possible HARTs */
+       if (!dom->possible_harts)
+               return SBI_EINVAL;
+       sbi_hartmask_for_each_hart(i, dom->possible_harts) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       return SBI_EINVAL;
+       };
+
+       /* Check memory regions */
+       if (!dom->regions)
+               return SBI_EINVAL;
Quoted 8 lines

lib/sbi/sbi_domain.c line 187

Quoted 6 lines
lib/sbi/sbi_domain.c new file +6 -0
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (!is_region_valid(reg))
+                       return SBI_EINVAL;
+       }
+
+       /* Count memory regions and check precense of firmware region */

precense/presence

Okay, will update.

Quoted 232 lines
Quoted 232 lines
lib/sbi/sbi_domain.c new file +178 -0
+       count = 0;
+       have_fw_reg = FALSE;
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (reg->order == root_memregs[ROOT_FW_REGION].order &&
+                   reg->base == root_memregs[ROOT_FW_REGION].base &&
+                   reg->flags == root_memregs[ROOT_FW_REGION].flags)
+                       have_fw_reg = TRUE;
+               count++;
+       }
+       if (!have_fw_reg)
+               return SBI_EINVAL;
+
+       /* Sort the memory regions */
+       for (i = 0; i < (count - 1); i++) {
+               reg = &dom->regions[i];
+               for (j = i + 1; j < count; j++) {
+                       reg1 = &dom->regions[j];
+
+                       if (is_region_conflict(reg1, reg))
+                               return SBI_EINVAL;
+
+                       if (!is_region_before(reg1, reg))
+                               continue;
+
+                       sbi_memcpy(&treg, reg1, sizeof(treg));
+                       sbi_memcpy(reg1, reg, sizeof(treg));
+                       sbi_memcpy(reg, &treg, sizeof(treg));
+               }
+       }
+
+       /*
+        * We don't need to check boot HART id of domain because if boot
+        * HART id is not possible/assigned to this domain then it won't
+        * be started at boot-time by sbi_domain_finalize().
+        */
+
+       /* Check next mode */
+       if (dom->next_mode != PRV_M &&
+           dom->next_mode != PRV_S &&
+           dom->next_mode != PRV_U)
+               return SBI_EINVAL;
+
+       /* Check next address and next mode*/
+       if (!sbi_domain_check_addr(dom, dom->next_addr, dom->next_mode,
+                                  FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
+
+       return 0;
+}
+
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       int rc;
+       u32 i, j, dhart;
+       bool dom_exists;
+       struct sbi_domain *dom, *tdom;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Discover domains */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               /* Ignore invalid HART */
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+
+               /* Get domain assigned to HART */
+               dom = sbi_platform_domain_get(plat, i);
+               if (!dom)
+                       continue;
+
+               /* Check if domain already discovered */
+               dom_exists = FALSE;
+               sbi_domain_for_each(j, tdom) {
+                       if (tdom == dom) {
+                               dom_exists = TRUE;
+                               break;
+                       }
+               }
+
+               /* Newly discovered domain */
+               if (!dom_exists) {
+                       /* Sanitize discovered domain */
+                       rc = sanitize_domain(plat, dom);
+                       if (rc)
+                               return rc;
+
+                       /* Assign index to domain */
+                       dom->index = domain_count++;
+                       domidx_to_domain_table[dom->index] = dom;
+
+                       /* Clear assigned HARTs of domain */
+                       sbi_hartmask_clear_all(&dom->assigned_harts);
+               }
+
+               /* Assign domain to HART if HART is a possible HART */
+               if (sbi_hartmask_test_hart(i, dom->possible_harts)) {
+                       tdom = hartid_to_domain_table[i];
+                       if (tdom)
+                               sbi_hartmask_clear_hart(i,
+                                               &tdom->assigned_harts);
+                       hartid_to_domain_table[i] = dom;
+                       sbi_hartmask_set_hart(i, &dom->assigned_harts);
+               }
+       }
+
+       /* Startup boot HART of domains */
+       sbi_domain_for_each(i, dom) {
+               /* Domain boot HART */
+               dhart = dom->boot_hartid;
+
+               /* Ignore if boot HART not possible for this domain */
+               if (!sbi_hartmask_test_hart(i, dom->possible_harts))
+                       continue;
+
+               /* Ignore if boot HART assigned different domain */
+               if (sbi_hartid_to_domain(dhart) != dom ||
+                   !sbi_hartmask_test_hart(i, &dom->assigned_harts))
+                       continue;
+
+               /* Startup boot HART of domain */
+               if (dhart == cold_hartid) {
+                       scratch->next_addr = dom->next_addr;
+                       scratch->next_mode = dom->next_mode;
+                       scratch->next_arg1 = dom->next_arg1;
+               } else {
+                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
+                                       dom->next_mode, dom->next_arg1);
+                       if (rc)
+                               return rc;
+               }
+       }
+
+       return 0;
+}
+
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       u32 i;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Root domain firmware memory region */
+       root_memregs[ROOT_FW_REGION].order = log2roundup(scratch->fw_size);
+       root_memregs[ROOT_FW_REGION].base = scratch->fw_start &
+                               ~((1UL << root_memregs[0].order) - 1UL);
+       root_memregs[ROOT_FW_REGION].flags = 0;
+
+       /* Root domain allow everything memory region */
+       root_memregs[ROOT_ALL_REGION].order = __riscv_xlen;
+       root_memregs[ROOT_ALL_REGION].base = 0;
+       root_memregs[ROOT_ALL_REGION].flags = (SBI_DOMAIN_MEMREGION_READABLE |
+                                               SBI_DOMAIN_MEMREGION_WRITEABLE |
+                                               SBI_DOMAIN_MEMREGION_EXECUTABLE);
+
+       /* Root domain memory region end */
+       root_memregs[ROOT_END_REGION].order = 0;
+
+       /* Root domain boot HART id is same as coldboot HART id */
+       root.boot_hartid = cold_hartid;
+
+       /* Root domain next booting stage details */
+       root.next_arg1 = scratch->next_arg1;
+       root.next_addr = scratch->next_addr;
+       root.next_mode = scratch->next_mode;
+
+       /* Select root domain for all valid HARTs */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+               sbi_hartmask_set_hart(i, &root_hmask);
+               hartid_to_domain_table[i] = &root;
+               sbi_hartmask_set_hart(i, &root.assigned_harts);
+       }
+
+       /* Set root domain index */
+       root.index = domain_count++;
+       domidx_to_domain_table[root.index] = &root;
+
+       return 0;
+}
lib/sbi/sbi_init.c +20 -1
index 5cedb15..406cb3f 100644
@@ -12,6 +12,7 @@
 #include <sbi/riscv_barrier.h>
 #include <sbi/riscv_locks.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hartmask.h>
@@ -169,6 +170,11 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /* Note: This has to be second thing in coldboot init sequence */
+       rc = sbi_domain_init(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        init_count_offset = sbi_scratch_alloc_offset(__SIZEOF_POINTER__,
                                                     "INIT_COUNT");
        if (!init_count_offset)
@@ -210,10 +216,24 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Finalize domains after HSM initialization so that we
+        * can startup non-root domains.
+        * Note: Finalize domains before HART PMP configuration so
+        * that we use correct domain for configuring PMP.
+        */
+       rc = sbi_domain_finalize(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_hart_pmp_configure(scratch);
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Platform final initialization should be last so that
+        * it sees correct domain assignment and PMP configuration.
+        */
        rc = sbi_platform_final_init(plat, TRUE);
        if (rc)
                sbi_hart_hang();
--

2.25.1

Quoted 10 lines
Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Regards,
Anup

#16 lore raw

PATCH 6/16 lib: sbi: Add initial domain support

Quoted 75 lines · Anup Patel
Quoted 64 lines · Atish Patra
Quoted 62 lines · Anup Patel

An OpenSBI domain is a logical entity representing a set of HARTs and a set of memory regions for these HARTs.

The OpenSBI domains support will allow OpenSBI platforms and previous booting stage (i.e. U-Boot SPL, Coreboot, etc) to partition a system into multiple domains where each domain will run it's own software.

For inter-domain isolation, OpenSBI will eventually use various HW features such as PMP, ePMP, IOPMP, SiFive shield, etc but initial implementation only use HW PMP support.

This patch provides initial implementation of OpenSBI domains where we have a root/default domain and OpenSBI platforms can provide non-root/custom domains using domain_get() callback.

---
 include/sbi/sbi_domain.h   | 138 ++++++++++++++
 include/sbi/sbi_platform.h |  20 ++
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 365 +++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_init.c         |  20 ++
 5 files changed, 544 insertions(+)
 create mode 100644 include/sbi/sbi_domain.h
 create mode 100644 lib/sbi/sbi_domain.c
include/sbi/sbi_domain.h new file +29 -0
new file mode 100644
index 0000000..a3f174c
@@ -0,0 +1,138 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#ifndef __SBI_DOMAIN_H__
+#define __SBI_DOMAIN_H__
+
+#include <sbi/sbi_types.h>
+#include <sbi/sbi_hartmask.h>
+
+struct sbi_scratch;
+
+/** Representation of OpenSBI domain memory region */
+struct sbi_domain_memregion {
+       /**
+        * Size of memory region as power of 2
+        * It has to be minimum 3 and maximum __riscv_xlen
+        */
+       unsigned long order;
+       /**
+        * Base address of memory region
+        * It must be 2^order aligned address
+        */
+       unsigned long base;
Quoted 8 lines

include/sbi/sbi_domain.h line 35

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+       /** Access flags of memory region */
+#define SBI_DOMAIN_MEMREGION_READABLE          (1UL << 0)
+#define SBI_DOMAIN_MEMREGION_WRITEABLE         (1UL << 1)
+#define SBI_DOMAIN_MEMREGION_EXECUTABLE                (1UL << 2)
+#define SBI_DOMAIN_MEMREGION_MMIO              (1UL << 3)
+#define SBI_DOMAIN_MEMREGION_MMODE             (1UL << 4)

We won't need this if we don't support M-mode domain software. Correct ?

Quoted 4 lines

This flag is for a memregion which requires access checks for OpenSBI.

It's not related to the next stage being M-mode.

Ah got it.

Quoted 130 lines
Quoted 24 lines
Quoted 24 lines
include/sbi/sbi_domain.h new file +23 -0
+       unsigned long flags;
+};
+
+/** Representation of OpenSBI domain */
+struct sbi_domain {
+       /**
+        * Logical index of this domain
+        * Note: This set by sbi_domain_finalize() in the coldboot path
+        */
+       u32 index;
+       /**
+        * HARTs assigned to this domain
+        * Note: This set by sbi_domain_init() and sbi_domain_finalize()
+        * in the coldboot path
+        */
+       struct sbi_hartmask assigned_harts;
+       /** Name of this domain */
+       char name[64];
+       /** Possible HARTs in this domain */
+       const struct sbi_hartmask *possible_harts;
+       /** Array of memory regions terminated by a region with order zero */
+       struct sbi_domain_memregion *regions;
+       /** HART id of the HART booting this domain */
Quoted 8 lines

include/sbi/sbi_domain.h line 64

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+       u32 boot_hartid;
+       /** Arg1 (or 'a1' register) of next booting stage for this domain */
+       unsigned long next_arg1;
+       /** Address of next booting stage for this domain */
+       unsigned long next_addr;
+       /** Priviledge mode of next booting stage for this domain */

/Priviledge/Privilege

Okay, will update.

Quoted 23 lines
Quoted 23 lines
include/sbi/sbi_domain.h new file +22 -0
+       unsigned long next_mode;
+       /** Is domain allowed to reset the system */
+       bool system_reset_allowed;
+};
+
+/** HART id to domain table */
+extern struct sbi_domain *hartid_to_domain_table[];
+
+/** Get pointer to sbi_domain from HART id */
+#define sbi_hartid_to_domain(__hartid) \
+       hartid_to_domain_table[__hartid]
+
+/** Get pointer to sbi_domain for current HART */
+#define sbi_domain_thishart_ptr() \
+       sbi_hartid_to_domain(current_hartid())
+
+/** Index to domain table */
+extern struct sbi_domain *domidx_to_domain_table[];
+
+/** Get pointer to sbi_domain from index */
+#define sbi_index_to_domain(__index) \
+       domidx_to_domain_table[__index]
Quoted 8 lines

include/sbi/sbi_domain.h line 92

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+
+/** Iterate over each domain */
+#define sbi_domain_for_each(__i, __d) \
+       for ((__i) = 0; ((__d) = sbi_index_to_domain(__i)); (__i)++)
+
+/** Iterate over each memory regon of a domain */

regon/region

Okay, will update.

Quoted 14 lines

include/sbi/sbi_domain.h line 103

Quoted 12 lines
include/sbi/sbi_domain.h new file +11 -0
+#define sbi_domain_for_each_memregion(__d, __r) \
+       for ((__r) = (__d)->regions; (__r)->order; (__r)++)
+
+/**
+ * Check whether given HART is assigned to specified domain
+ * @param dom pointer to domain
+ * @param hartid the HART ID
+ * @return TRUE if HART is assigned to domain otherwise FALSE
+ */
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid);
+

May be a better name is
sbi_domain_is_assigned_hart ?

Okay, will rename.

Quoted 6 lines

include/sbi/sbi_domain.h line 105

Quoted 3 lines
include/sbi/sbi_domain.h new file +2 -0
+/**
+ * Get ulong assigned HART mask for given domain and HART base ID

Get assigned HART mask in ulong for given domain and HART base ID

Quoted 10 lines

include/sbi/sbi_domain.h line 112

Quoted 7 lines
include/sbi/sbi_domain.h new file +7 -0
+ * @param dom pointer to domain
+ * @param hbase the HART base ID
+ * @return ulong possible HART mask
+ * Note: the return ulong mask will be set to zero on failure.
+ */
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase);
+

Maybe a better name is
sbi_domain_get_assigned_hartmask ?

Okay, will rename.

Quoted 16 lines
Quoted 16 lines
include/sbi/sbi_domain.h new file +14 -0
+/** Initialize a domain memory region as firmware region */
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg);
+
+/**
+ * Check whether we can access specified address for given mode and
+ * memory region flags under a domain
+ * @param dom pointer to domain
+ * @param addr the address to be checked
+ * @param mode the privilege mode of access
+ * @param mmio the memory type of access is MMIO
+ * @param read the access is read
+ * @param write the access is write
+ * @param execute the access is instruction execute
+ * @return TRUE if access allowed otherwise FALSE
Quoted 8 lines

include/sbi/sbi_domain.h line 131

Quoted 5 lines
include/sbi/sbi_domain.h new file +5 -0
+ */
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute);
+

We can reduce the last 3 arguments to 1 if the caller passes one value after ORing.

Quoted 3 lines

For making it one argument, we will have to define separate flags. Let's have three parameters for now.

We can use the flags SBI_DOMAIN_MEMREGION_XXX. no ?

Quoted 516 lines
Quoted 87 lines
Quoted 87 lines
include/sbi/sbi_domain.h new file +7 -0
+/** Finalize domain tables and startup non-root domains */
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid);
+
+/** Initialize domains */
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid);
+
+#endif
include/sbi/sbi_platform.h +20 -0
index f2c3237..e1355d8 100644
@@ -44,6 +44,7 @@
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_domain;
 struct sbi_trap_info;
 
 /** Possible feature flags of a platform */
@@ -89,6 +90,9 @@ struct sbi_platform_operations {
         */
        int (*misa_get_xlen)(void);
 
+       /** Get domain pointer for given HART id */
+       struct sbi_domain *(*domain_get)(u32 hartid);
+
        /** Write a character to the platform console output */
        void (*console_putc)(char ch);
        /** Read a character from the platform console input */
@@ -447,6 +451,22 @@ static inline int sbi_platform_misa_xlen(const struct sbi_platform *plat)
        return -1;
 }
 
+/**
+ * Get domain pointer for given HART
+ *
+ * @param plat pointer to struct sbi_platform
+ * @param hartid shorthand letter for CPU extensions
+ *
+ * @return non-NULL domain pointer on success and NULL on failure
+ */
+static inline struct sbi_domain *sbi_platform_domain_get(
+                               const struct sbi_platform *plat, u32 hartid)
+{
+       if (plat && sbi_platform_ops(plat)->domain_get)
+               return sbi_platform_ops(plat)->domain_get(hartid);
+       return NULL;
+}
+
 /**
  * Write a character to the platform console output
  *
lib/sbi/objects.mk +1 -0
index fa808a0..6f2c06f 100644
@@ -15,6 +15,7 @@ libsbi-objs-y += riscv_locks.o
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o
+libsbi-objs-y += sbi_domain.o
 libsbi-objs-y += sbi_ecall.o
 libsbi-objs-y += sbi_ecall_base.o
 libsbi-objs-y += sbi_ecall_hsm.o
lib/sbi/sbi_domain.c new file +16 -0
new file mode 100644
index 0000000..8d5dae1
@@ -0,0 +1,365 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#include <sbi/riscv_asm.h>
+#include <sbi/sbi_domain.h>
+#include <sbi/sbi_hartmask.h>
+#include <sbi/sbi_hsm.h>
+#include <sbi/sbi_math.h>
+#include <sbi/sbi_platform.h>
+#include <sbi/sbi_scratch.h>
Quoted 8 lines

lib/sbi/sbi_domain.c line 20

Quoted 4 lines
lib/sbi/sbi_domain.c new file +4 -0
+#include <sbi/sbi_string.h>
+
+struct sbi_domain *hartid_to_domain_table[SBI_HARTMASK_MAX_BITS] = { 0 };
+struct sbi_domain *domidx_to_domain_table[SBI_HARTMASK_MAX_BITS + 1] = { 0 };

Do we really need 128 as the upper range of domain index ? I think it is better to define a smaller value(8 or 16 should be reasonable) in a different macro similar to SBI_DOM_IDX_MAX_BITS.

Okay, I will add SBI_DOMAIN_MAX_IDX define.

Quoted 162 lines
Quoted 162 lines
lib/sbi/sbi_domain.c new file +161 -0
+
+static u32 domain_count = 0;
+
+static struct sbi_hartmask root_hmask = { 0 };
+
+#define ROOT_FW_REGION         0
+#define ROOT_ALL_REGION        1
+#define ROOT_END_REGION        2
+static struct sbi_domain_memregion root_memregs[ROOT_END_REGION + 1] = { 0 };
+
+static struct sbi_domain root = {
+       .name = "root",
+       .possible_harts = &root_hmask,
+       .regions = root_memregs,
+       .system_reset_allowed = TRUE,
+};
+
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32 hartid)
+{
+       if (dom)
+               return sbi_hartmask_test_hart(hartid, &dom->assigned_harts);
+
+       return FALSE;
+}
+
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom, ulong hbase)
+{
+       ulong ret, bword, boff;
+
+       if (!dom)
+               return 0;
+
+       bword = BIT_WORD(hbase);
+       boff = BIT_WORD_OFFSET(hbase);
+
+       ret = sbi_hartmask_bits(&dom->assigned_harts)[bword++] >> boff;
+       if (boff && bword < BIT_WORD(SBI_HARTMASK_MAX_BITS)) {
+               ret |= (sbi_hartmask_bits(&dom->assigned_harts)[bword] &
+                       (BIT(boff) - 1UL)) << (BITS_PER_LONG - boff);
+       }
+
+       return ret;
+}
+
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg)
+{
+       if (!reg)
+               return;
+
+       sbi_memcpy(reg, &root_memregs[ROOT_FW_REGION], sizeof(*reg));
+}
+
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute)
+{
+       struct sbi_domain_memregion *reg;
+       unsigned long rstart, rend, rflags, rwx = 0;
+
+       if (!dom)
+               return FALSE;
+
+       if (read)
+               rwx |= SBI_DOMAIN_MEMREGION_READABLE;
+       if (write)
+               rwx |= SBI_DOMAIN_MEMREGION_WRITEABLE;
+       if (execute)
+               rwx |= SBI_DOMAIN_MEMREGION_EXECUTABLE;
+
+       sbi_domain_for_each_memregion(dom, reg) {
+               rflags = reg->flags;
+               if (mode == PRV_M && !(rflags & SBI_DOMAIN_MEMREGION_MMODE))
+                       continue;
+
+               rstart = reg->base;
+               rend = (reg->order < __riscv_xlen) ?
+                       rstart + ((1UL << reg->order) - 1) : -1UL;
+               if (rstart <= addr && addr <= rend) {
+                       if ((mmio && !(rflags & SBI_DOMAIN_MEMREGION_MMIO)) ||
+                           (!mmio && (rflags & SBI_DOMAIN_MEMREGION_MMIO)))
+                               return FALSE;
+                       return ((rflags & rwx) == rwx) ? TRUE : FALSE;
+               }
+       }
+
+       return (mode == PRV_M) ? TRUE : FALSE;
+}
+
+/* Check if region comply with constraints */
+static bool is_region_valid(const struct sbi_domain_memregion *reg)
+{
+       if (reg->order < 3 || __riscv_xlen < reg->order)
+               return FALSE;
+
+       if (reg->base & (BIT(reg->order) - 1))
+               return FALSE;
+
+       return TRUE;
+}
+
+/** Check if regionA is sub-region of regionB */
+static bool is_region_subset(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       ulong regA_start = regA->base;
+       ulong regA_end = regA->base + (BIT(regA->order) - 1);
+       ulong regB_start = regB->base;
+       ulong regB_end = regB->base + (BIT(regA->order) - 1);
+
+       if ((regB_start <= regA_start) &&
+           (regA_start < regB_end) &&
+           (regB_start < regA_end) &&
+           (regA_end <= regB_end))
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA conflicts regionB */
+static bool is_region_conflict(const struct sbi_domain_memregion *regA,
+                               const struct sbi_domain_memregion *regB)
+{
+       if ((is_region_subset(regA, regB) || is_region_subset(regB, regA)) &&
+           regA->flags == regB->flags)
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA should be placed before regionB */
+static bool is_region_before(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion *regB)
+{
+       if (regA->order < regB->order)
+               return TRUE;
+
+       if ((regA->order == regB->order) &&
+           (regA->base < regB->base))
+               return TRUE;
+
+       return FALSE;
+}
+
+static int sanitize_domain(const struct sbi_platform *plat,
+                          struct sbi_domain *dom)
+{
+       u32 i, j, count;
+       bool have_fw_reg;
+       struct sbi_domain_memregion treg, *reg, *reg1;
+
+       /* Check possible HARTs */
+       if (!dom->possible_harts)
+               return SBI_EINVAL;
+       sbi_hartmask_for_each_hart(i, dom->possible_harts) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       return SBI_EINVAL;
+       };
+
+       /* Check memory regions */
+       if (!dom->regions)
+               return SBI_EINVAL;
Quoted 8 lines

lib/sbi/sbi_domain.c line 187

Quoted 6 lines
lib/sbi/sbi_domain.c new file +6 -0
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (!is_region_valid(reg))
+                       return SBI_EINVAL;
+       }
+
+       /* Count memory regions and check precense of firmware region */

precense/presence

Okay, will update.

Quoted 232 lines
Quoted 232 lines
lib/sbi/sbi_domain.c new file +178 -0
+       count = 0;
+       have_fw_reg = FALSE;
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (reg->order == root_memregs[ROOT_FW_REGION].order &&
+                   reg->base == root_memregs[ROOT_FW_REGION].base &&
+                   reg->flags == root_memregs[ROOT_FW_REGION].flags)
+                       have_fw_reg = TRUE;
+               count++;
+       }
+       if (!have_fw_reg)
+               return SBI_EINVAL;
+
+       /* Sort the memory regions */
+       for (i = 0; i < (count - 1); i++) {
+               reg = &dom->regions[i];
+               for (j = i + 1; j < count; j++) {
+                       reg1 = &dom->regions[j];
+
+                       if (is_region_conflict(reg1, reg))
+                               return SBI_EINVAL;
+
+                       if (!is_region_before(reg1, reg))
+                               continue;
+
+                       sbi_memcpy(&treg, reg1, sizeof(treg));
+                       sbi_memcpy(reg1, reg, sizeof(treg));
+                       sbi_memcpy(reg, &treg, sizeof(treg));
+               }
+       }
+
+       /*
+        * We don't need to check boot HART id of domain because if boot
+        * HART id is not possible/assigned to this domain then it won't
+        * be started at boot-time by sbi_domain_finalize().
+        */
+
+       /* Check next mode */
+       if (dom->next_mode != PRV_M &&
+           dom->next_mode != PRV_S &&
+           dom->next_mode != PRV_U)
+               return SBI_EINVAL;
+
+       /* Check next address and next mode*/
+       if (!sbi_domain_check_addr(dom, dom->next_addr, dom->next_mode,
+                                  FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
+
+       return 0;
+}
+
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       int rc;
+       u32 i, j, dhart;
+       bool dom_exists;
+       struct sbi_domain *dom, *tdom;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Discover domains */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               /* Ignore invalid HART */
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+
+               /* Get domain assigned to HART */
+               dom = sbi_platform_domain_get(plat, i);
+               if (!dom)
+                       continue;
+
+               /* Check if domain already discovered */
+               dom_exists = FALSE;
+               sbi_domain_for_each(j, tdom) {
+                       if (tdom == dom) {
+                               dom_exists = TRUE;
+                               break;
+                       }
+               }
+
+               /* Newly discovered domain */
+               if (!dom_exists) {
+                       /* Sanitize discovered domain */
+                       rc = sanitize_domain(plat, dom);
+                       if (rc)
+                               return rc;
+
+                       /* Assign index to domain */
+                       dom->index = domain_count++;
+                       domidx_to_domain_table[dom->index] = dom;
+
+                       /* Clear assigned HARTs of domain */
+                       sbi_hartmask_clear_all(&dom->assigned_harts);
+               }
+
+               /* Assign domain to HART if HART is a possible HART */
+               if (sbi_hartmask_test_hart(i, dom->possible_harts)) {
+                       tdom = hartid_to_domain_table[i];
+                       if (tdom)
+                               sbi_hartmask_clear_hart(i,
+                                               &tdom->assigned_harts);
+                       hartid_to_domain_table[i] = dom;
+                       sbi_hartmask_set_hart(i, &dom->assigned_harts);
+               }
+       }
+
+       /* Startup boot HART of domains */
+       sbi_domain_for_each(i, dom) {
+               /* Domain boot HART */
+               dhart = dom->boot_hartid;
+
+               /* Ignore if boot HART not possible for this domain */
+               if (!sbi_hartmask_test_hart(i, dom->possible_harts))
+                       continue;
+
+               /* Ignore if boot HART assigned different domain */
+               if (sbi_hartid_to_domain(dhart) != dom ||
+                   !sbi_hartmask_test_hart(i, &dom->assigned_harts))
+                       continue;
+
+               /* Startup boot HART of domain */
+               if (dhart == cold_hartid) {
+                       scratch->next_addr = dom->next_addr;
+                       scratch->next_mode = dom->next_mode;
+                       scratch->next_arg1 = dom->next_arg1;
+               } else {
+                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
+                                       dom->next_mode, dom->next_arg1);
+                       if (rc)
+                               return rc;
+               }
+       }
+
+       return 0;
+}
+
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
+{
+       u32 i;
+       const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+
+       /* Root domain firmware memory region */
+       root_memregs[ROOT_FW_REGION].order = log2roundup(scratch->fw_size);
+       root_memregs[ROOT_FW_REGION].base = scratch->fw_start &
+                               ~((1UL << root_memregs[0].order) - 1UL);
+       root_memregs[ROOT_FW_REGION].flags = 0;
+
+       /* Root domain allow everything memory region */
+       root_memregs[ROOT_ALL_REGION].order = __riscv_xlen;
+       root_memregs[ROOT_ALL_REGION].base = 0;
+       root_memregs[ROOT_ALL_REGION].flags = (SBI_DOMAIN_MEMREGION_READABLE |
+                                               SBI_DOMAIN_MEMREGION_WRITEABLE |
+                                               SBI_DOMAIN_MEMREGION_EXECUTABLE);
+
+       /* Root domain memory region end */
+       root_memregs[ROOT_END_REGION].order = 0;
+
+       /* Root domain boot HART id is same as coldboot HART id */
+       root.boot_hartid = cold_hartid;
+
+       /* Root domain next booting stage details */
+       root.next_arg1 = scratch->next_arg1;
+       root.next_addr = scratch->next_addr;
+       root.next_mode = scratch->next_mode;
+
+       /* Select root domain for all valid HARTs */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+               sbi_hartmask_set_hart(i, &root_hmask);
+               hartid_to_domain_table[i] = &root;
+               sbi_hartmask_set_hart(i, &root.assigned_harts);
+       }
+
+       /* Set root domain index */
+       root.index = domain_count++;
+       domidx_to_domain_table[root.index] = &root;
+
+       return 0;
+}
lib/sbi/sbi_init.c +20 -1
index 5cedb15..406cb3f 100644
@@ -12,6 +12,7 @@
 #include <sbi/riscv_barrier.h>
 #include <sbi/riscv_locks.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hartmask.h>
@@ -169,6 +170,11 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /* Note: This has to be second thing in coldboot init sequence */
+       rc = sbi_domain_init(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        init_count_offset = sbi_scratch_alloc_offset(__SIZEOF_POINTER__,
                                                     "INIT_COUNT");
        if (!init_count_offset)
@@ -210,10 +216,24 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Finalize domains after HSM initialization so that we
+        * can startup non-root domains.
+        * Note: Finalize domains before HART PMP configuration so
+        * that we use correct domain for configuring PMP.
+        */
+       rc = sbi_domain_finalize(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_hart_pmp_configure(scratch);
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Platform final initialization should be last so that
+        * it sees correct domain assignment and PMP configuration.
+        */
        rc = sbi_platform_final_init(plat, TRUE);
        if (rc)
                sbi_hart_hang();
--

2.25.1

Quoted 10 lines
Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Regards,
Anup

Signature · 2 lines
Regards,
Atish
author
#17 lore raw

PATCH 6/16 lib: sbi: Add initial domain support

Quoted 228 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 19 October 2020 05:28
To: Anup Patel <[email protected]>
Cc: Anup Patel <[email protected]>; Atish Patra <[email protected]>; Alistair Francis <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 06/16] lib: sbi: Add initial domain support

Quoted 4 lines · Anup Patel
Quoted 2 lines · Atish Patra

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines

An OpenSBI domain is a logical entity representing a set of HARTs and a set of memory regions for these HARTs.

The OpenSBI domains support will allow OpenSBI platforms and previous booting stage (i.e. U-Boot SPL, Coreboot, etc) to partition a system into multiple domains where each domain will run it's

own software.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

For inter-domain isolation, OpenSBI will eventually use various HW features such as PMP, ePMP, IOPMP, SiFive shield, etc but initial implementation only use HW PMP support.

This patch provides initial implementation of OpenSBI domains where we have a root/default domain and OpenSBI platforms can provide non-root/custom domains using domain_get() callback.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 include/sbi/sbi_domain.h   | 138 ++++++++++++++
 include/sbi/sbi_platform.h |  20 ++
 lib/sbi/objects.mk         |   1 +
 lib/sbi/sbi_domain.c       | 365

+++++++++++++++++++++++++++++++++++++

Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
 lib/sbi/sbi_init.c         |  20 ++
 5 files changed, 544 insertions(+)  create mode 100644

include/sbi/sbi_domain.h create mode 100644 lib/sbi/sbi_domain.c

include/sbi/sbi_domain.h new file +29 -0
new file mode 100644 index 0000000..a3f174c
@@ -0,0 +1,138 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#ifndef __SBI_DOMAIN_H__
+#define __SBI_DOMAIN_H__
+
+#include <sbi/sbi_types.h>
+#include <sbi/sbi_hartmask.h>
+
+struct sbi_scratch;
+
+/** Representation of OpenSBI domain memory region */ struct
+sbi_domain_memregion {
+       /**
+        * Size of memory region as power of 2
+        * It has to be minimum 3 and maximum __riscv_xlen
+        */
+       unsigned long order;
+       /**
+        * Base address of memory region
+        * It must be 2^order aligned address
+        */
+       unsigned long base;
Quoted 8 lines
Quoted 8 lines

include/sbi/sbi_domain.h line 35

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+       /** Access flags of memory region */
+#define SBI_DOMAIN_MEMREGION_READABLE          (1UL << 0)
+#define SBI_DOMAIN_MEMREGION_WRITEABLE         (1UL << 1)
+#define SBI_DOMAIN_MEMREGION_EXECUTABLE                (1UL << 2)
+#define SBI_DOMAIN_MEMREGION_MMIO              (1UL << 3)
+#define SBI_DOMAIN_MEMREGION_MMODE             (1UL << 4)

We won't need this if we don't support M-mode domain software.

Correct ?

Quoted 5 lines

This flag is for a memregion which requires access checks for OpenSBI.

It's not related to the next stage being M-mode.

Ah got it.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
include/sbi/sbi_domain.h new file +12 -0
+       unsigned long flags;
+};
+
+/** Representation of OpenSBI domain */ struct sbi_domain {
+       /**
+        * Logical index of this domain
+        * Note: This set by sbi_domain_finalize() in the coldboot path
+        */
+       u32 index;
+       /**
+        * HARTs assigned to this domain
+        * Note: This set by sbi_domain_init() and sbi_domain_finalize()
Quoted 8 lines
Quoted 8 lines

include/sbi/sbi_domain.h

Quoted 8 lines
include/sbi/sbi_domain.h new file +8 -0
+        * in the coldboot path
+        */
+       struct sbi_hartmask assigned_harts;
+       /** Name of this domain */
+       char name[64];
+       /** Possible HARTs in this domain */
+       const struct sbi_hartmask *possible_harts;
+       /** Array of memory regions terminated by a region with order

zero */

Quoted 110 lines
Quoted 11 lines

include/sbi/sbi_domain.h line 101

Quoted 9 lines
include/sbi/sbi_domain.h new file +9 -0
+       struct sbi_domain_memregion *regions;
+       /** HART id of the HART booting this domain */
+       u32 boot_hartid;
+       /** Arg1 (or 'a1' register) of next booting stage for this domain */
+       unsigned long next_arg1;
+       /** Address of next booting stage for this domain */
+       unsigned long next_addr;
+       /** Priviledge mode of next booting stage for this domain
+*/

/Priviledge/Privilege

Okay, will update.

Quoted 23 lines
Quoted 23 lines
include/sbi/sbi_domain.h new file +22 -0
+       unsigned long next_mode;
+       /** Is domain allowed to reset the system */
+       bool system_reset_allowed; };
+
+/** HART id to domain table */
+extern struct sbi_domain *hartid_to_domain_table[];
+
+/** Get pointer to sbi_domain from HART id */ #define
+sbi_hartid_to_domain(__hartid) \
+       hartid_to_domain_table[__hartid]
+
+/** Get pointer to sbi_domain for current HART */ #define
+sbi_domain_thishart_ptr() \
+       sbi_hartid_to_domain(current_hartid())
+
+/** Index to domain table */
+extern struct sbi_domain *domidx_to_domain_table[];
+
+/** Get pointer to sbi_domain from index */ #define
+sbi_index_to_domain(__index) \
+       domidx_to_domain_table[__index]
+
Quoted 8 lines

include/sbi/sbi_domain.h line 92

Quoted 6 lines
include/sbi/sbi_domain.h new file +6 -0
+/** Iterate over each domain */
+#define sbi_domain_for_each(__i, __d) \
+       for ((__i) = 0; ((__d) = sbi_index_to_domain(__i));
+(__i)++)
+
+/** Iterate over each memory regon of a domain */

regon/region

Okay, will update.

Quoted 14 lines

include/sbi/sbi_domain.h line 103

Quoted 12 lines
include/sbi/sbi_domain.h new file +11 -0
+#define sbi_domain_for_each_memregion(__d, __r) \
+       for ((__r) = (__d)->regions; (__r)->order; (__r)++)
+
+/**
+ * Check whether given HART is assigned to specified domain
+ * @param dom pointer to domain
+ * @param hartid the HART ID
+ * @return TRUE if HART is assigned to domain otherwise FALSE  */
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32
+hartid);
+

May be a better name is
sbi_domain_is_assigned_hart ?

Okay, will rename.

Quoted 6 lines

include/sbi/sbi_domain.h line 105

Quoted 3 lines
include/sbi/sbi_domain.h new file +2 -0
+/**
+ * Get ulong assigned HART mask for given domain and HART base ID

Get assigned HART mask in ulong for given domain and HART base ID

Quoted 11 lines

include/sbi/sbi_domain.h line 112

Quoted 8 lines
include/sbi/sbi_domain.h new file +8 -0
+ * @param dom pointer to domain
+ * @param hbase the HART base ID
+ * @return ulong possible HART mask
+ * Note: the return ulong mask will be set to zero on failure.
+ */
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom,
+ulong hbase);
+

Maybe a better name is
sbi_domain_get_assigned_hartmask ?

Okay, will rename.

Quoted 24 lines
Quoted 11 lines
include/sbi/sbi_domain.h new file +9 -0
+/** Initialize a domain memory region as firmware region */ void
+sbi_domain_memregion_initfw(struct sbi_domain_memregion *reg);
+
+/**
+ * Check whether we can access specified address for given mode
+and
+ * memory region flags under a domain
+ * @param dom pointer to domain
+ * @param addr the address to be checked

include/sbi/sbi_domain.h line 131

Quoted 10 lines
include/sbi/sbi_domain.h new file +10 -0
+ * @param mode the privilege mode of access
+ * @param mmio the memory type of access is MMIO
+ * @param read the access is read
+ * @param write the access is write
+ * @param execute the access is instruction execute
+ * @return TRUE if access allowed otherwise FALSE  */ bool
+sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute);
+

We can reduce the last 3 arguments to 1 if the caller passes one value after ORing.

Quoted 5 lines
Quoted 3 lines

For making it one argument, we will have to define separate flags. Let's have three parameters for now.

We can use the flags SBI_DOMAIN_MEMREGION_XXX. no ?

The SBI_DOMAIN_MEMREGION_XXX are domain memory region flags and it can have flags which don't describe access type.

Better to have separate SBI_DOMAIN_ACCESS_XXX flags.

Quoted 530 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
include/sbi/sbi_domain.h new file +9 -0
+/** Finalize domain tables and startup non-root domains */ int
+sbi_domain_finalize(struct sbi_scratch *scratch, u32
+cold_hartid);
+
+/** Initialize domains */
+int sbi_domain_init(struct sbi_scratch *scratch, u32
+cold_hartid);
+
+#endif
(quoted diff) +0 -0

b/include/sbi/sbi_platform.h index f2c3237..e1355d8 100644

include/sbi/sbi_platform.h +1 -0
@@ -44,6 +44,7 @@
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_version.h>
 
+struct sbi_domain;
 struct sbi_trap_info;
 
 /** Possible feature flags of a platform */ @@ -89,6 +90,9 @@
struct sbi_platform_operations {
         */
        int (*misa_get_xlen)(void);
Quoted 7 lines
Quoted 7 lines

include/sbi/sbi_platform.h struct sbi_platform_operations {

Quoted 7 lines
include/sbi/sbi_platform.h +3 -1
+       /** Get domain pointer for given HART id */
+       struct sbi_domain *(*domain_get)(u32 hartid);
+
        /** Write a character to the platform console output */
        void (*console_putc)(char ch);
        /** Read a character from the platform console input */ @@
-447,6 +451,22 @@ static inline int sbi_platform_misa_xlen(const struct

sbi_platform *plat)

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
        return -1;

}

include/sbi/sbi_platform.h +15 -0
+/**
+ * Get domain pointer for given HART
+ *
+ * @param plat pointer to struct sbi_platform
+ * @param hartid shorthand letter for CPU extensions
+ *
+ * @return non-NULL domain pointer on success and NULL on failure
+*/ static inline struct sbi_domain *sbi_platform_domain_get(
+                               const struct sbi_platform *plat,
+u32 hartid) {
+       if (plat && sbi_platform_ops(plat)->domain_get)
+               return sbi_platform_ops(plat)->domain_get(hartid);
+       return NULL;
+}
+
 /**
  * Write a character to the platform console output
  *
lib/sbi/objects.mk index renamed from lib/sbi/objects.mk +0 -0

fa808a0..6f2c06f 100644

lib/sbi/objects.mk +0 -0
@@ -15,6 +15,7 @@ libsbi-objs-y += riscv_locks.o  libsbi-objs-y +=

sbi_bitmap.o libsbi-objs-y += sbi_bitops.o libsbi-objs-y += sbi_console.o
+libsbi-objs-y += sbi_domain.o
libsbi-objs-y += sbi_ecall.o
libsbi-objs-y += sbi_ecall_base.o libsbi-objs-y +=
sbi_ecall_hsm.o diff --git a/lib/sbi/sbi_domain.c b/lib/sbi/sbi_domain.c new file mode 100644 index 0000000..8d5dae1

lib/sbi/sbi_domain.c new file +11 -0
@@ -0,0 +1,365 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2020 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <[email protected]>
+ */
+
+#include <sbi/riscv_asm.h>
+#include <sbi/sbi_domain.h>
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_domain.c

Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+#include <sbi/sbi_hartmask.h>
+#include <sbi/sbi_hsm.h>
+#include <sbi/sbi_math.h>
+#include <sbi/sbi_platform.h>
+#include <sbi/sbi_scratch.h>
+#include <sbi/sbi_string.h>
+
+struct sbi_domain

*hartid_to_domain_table[SBI_HARTMASK_MAX_BITS]

Quoted 10 lines
Quoted 6 lines

lib/sbi/sbi_domain.c line 21

Quoted 2 lines
lib/sbi/sbi_domain.c new file +2 -0
+= { 0 }; struct sbi_domain
+*domidx_to_domain_table[SBI_HARTMASK_MAX_BITS + 1] = { 0 };

Do we really need 128 as the upper range of domain index ? I think it is better to define a smaller value(8 or 16 should be reasonable) in a different macro similar to SBI_DOM_IDX_MAX_BITS.

Okay, I will add SBI_DOMAIN_MAX_IDX define.

Quoted 1 line
Quoted 9 lines
Quoted 9 lines

lib/sbi/sbi_domain.c

Quoted 9 lines
lib/sbi/sbi_domain.c new file +9 -0
+
+static u32 domain_count = 0;
+
+static struct sbi_hartmask root_hmask = { 0 };
+
+#define ROOT_FW_REGION         0
+#define ROOT_ALL_REGION        1
+#define ROOT_END_REGION        2
+static struct sbi_domain_memregion

root_memregs[ROOT_END_REGION +

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
lib/sbi/sbi_domain.c new file +20 -0
+1] = { 0 };
+
+static struct sbi_domain root = {
+       .name = "root",
+       .possible_harts = &root_hmask,
+       .regions = root_memregs,
+       .system_reset_allowed = TRUE, };
+
+bool sbi_domain_assigned_hart(const struct sbi_domain *dom, u32
+hartid) {
+       if (dom)
+               return sbi_hartmask_test_hart(hartid,
+&dom->assigned_harts);
+
+       return FALSE;
+}
+
+ulong sbi_domain_assigned_hartmask(const struct sbi_domain *dom,
+ulong hbase) {
+       ulong ret, bword, boff;
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_domain.c

Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+
+       if (!dom)
+               return 0;
+
+       bword = BIT_WORD(hbase);
+       boff = BIT_WORD_OFFSET(hbase);
+
+       ret = sbi_hartmask_bits(&dom->assigned_harts)[bword++] >>

boff;

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
lib/sbi/sbi_domain.c new file +27 -0
+       if (boff && bword < BIT_WORD(SBI_HARTMASK_MAX_BITS)) {
+               ret |= (sbi_hartmask_bits(&dom->assigned_harts)[bword] &
+                       (BIT(boff) - 1UL)) << (BITS_PER_LONG - boff);
+       }
+
+       return ret;
+}
+
+void sbi_domain_memregion_initfw(struct sbi_domain_memregion
+*reg) {
+       if (!reg)
+               return;
+
+       sbi_memcpy(reg, &root_memregs[ROOT_FW_REGION],
+sizeof(*reg)); }
+
+bool sbi_domain_check_addr(const struct sbi_domain *dom,
+                          unsigned long addr, unsigned long mode, bool mmio,
+                          bool read, bool write, bool execute) {
+       struct sbi_domain_memregion *reg;
+       unsigned long rstart, rend, rflags, rwx = 0;
+
+       if (!dom)
+               return FALSE;
+
+       if (read)
+               rwx |= SBI_DOMAIN_MEMREGION_READABLE;
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_domain.c

Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+       if (write)
+               rwx |= SBI_DOMAIN_MEMREGION_WRITEABLE;
+       if (execute)
+               rwx |= SBI_DOMAIN_MEMREGION_EXECUTABLE;
+
+       sbi_domain_for_each_memregion(dom, reg) {
+               rflags = reg->flags;
+               if (mode == PRV_M && !(rflags &

SBI_DOMAIN_MEMREGION_MMODE))

Quoted 7 lines
Quoted 7 lines

lib/sbi/sbi_domain.c

Quoted 7 lines
lib/sbi/sbi_domain.c new file +7 -0
+                       continue;
+
+               rstart = reg->base;
+               rend = (reg->order < __riscv_xlen) ?
+                       rstart + ((1UL << reg->order) - 1) : -1UL;
+               if (rstart <= addr && addr <= rend) {
+                       if ((mmio && !(rflags &

SBI_DOMAIN_MEMREGION_MMIO)) ||

Quoted 1 line
Quoted 1 line

lib/sbi/sbi_domain.c line 101

Quoted 1 line
lib/sbi/sbi_domain.c new file +1 -0
+                           (!mmio && (rflags &

SBI_DOMAIN_MEMREGION_MMIO)))

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
lib/sbi/sbi_domain.c new file +33 -0
+                               return FALSE;
+                       return ((rflags & rwx) == rwx) ? TRUE : FALSE;
+               }
+       }
+
+       return (mode == PRV_M) ? TRUE : FALSE; }
+
+/* Check if region comply with constraints */ static bool
+is_region_valid(const struct sbi_domain_memregion *reg) {
+       if (reg->order < 3 || __riscv_xlen < reg->order)
+               return FALSE;
+
+       if (reg->base & (BIT(reg->order) - 1))
+               return FALSE;
+
+       return TRUE;
+}
+
+/** Check if regionA is sub-region of regionB */ static bool
+is_region_subset(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion
+*regB) {
+       ulong regA_start = regA->base;
+       ulong regA_end = regA->base + (BIT(regA->order) - 1);
+       ulong regB_start = regB->base;
+       ulong regB_end = regB->base + (BIT(regA->order) - 1);
+
+       if ((regB_start <= regA_start) &&
+           (regA_start < regB_end) &&
+           (regB_start < regA_end) &&
+           (regA_end <= regB_end))
+               return TRUE;
+
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_domain.c line 142

Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+       return FALSE;
+}
+
+/** Check if regionA conflicts regionB */ static bool
+is_region_conflict(const struct sbi_domain_memregion *regA,
+                               const struct sbi_domain_memregion
+*regB) {
+       if ((is_region_subset(regA, regB) || is_region_subset(regB, regA))

&&

Quoted 47 lines
Quoted 38 lines
Quoted 38 lines
lib/sbi/sbi_domain.c new file +38 -0
+           regA->flags == regB->flags)
+               return TRUE;
+
+       return FALSE;
+}
+
+/** Check if regionA should be placed before regionB */ static
+bool is_region_before(const struct sbi_domain_memregion *regA,
+                            const struct sbi_domain_memregion
+*regB) {
+       if (regA->order < regB->order)
+               return TRUE;
+
+       if ((regA->order == regB->order) &&
+           (regA->base < regB->base))
+               return TRUE;
+
+       return FALSE;
+}
+
+static int sanitize_domain(const struct sbi_platform *plat,
+                          struct sbi_domain *dom) {
+       u32 i, j, count;
+       bool have_fw_reg;
+       struct sbi_domain_memregion treg, *reg, *reg1;
+
+       /* Check possible HARTs */
+       if (!dom->possible_harts)
+               return SBI_EINVAL;
+       sbi_hartmask_for_each_hart(i, dom->possible_harts) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       return SBI_EINVAL;
+       };
+
+       /* Check memory regions */
+       if (!dom->regions)
+               return SBI_EINVAL;
+       sbi_domain_for_each_memregion(dom, reg) {
Quoted 8 lines

lib/sbi/sbi_domain.c

Quoted 6 lines
lib/sbi/sbi_domain.c new file +6 -0
+               if (!is_region_valid(reg))
+                       return SBI_EINVAL;
+       }
+
+       /* Count memory regions and check precense of firmware
+ region */

precense/presence

Quoted 7 lines

Okay, will update.

Quoted 5 lines
Quoted 5 lines
lib/sbi/sbi_domain.c new file +4 -0
+       count = 0;
+       have_fw_reg = FALSE;
+       sbi_domain_for_each_memregion(dom, reg) {
+               if (reg->order == root_memregs[ROOT_FW_REGION].order

&&

Quoted 133 lines
Quoted 32 lines
Quoted 32 lines
lib/sbi/sbi_domain.c new file +32 -0
+                   reg->base == root_memregs[ROOT_FW_REGION].base &&
+                   reg->flags == root_memregs[ROOT_FW_REGION].flags)
+                       have_fw_reg = TRUE;
+               count++;
+       }
+       if (!have_fw_reg)
+               return SBI_EINVAL;
+
+       /* Sort the memory regions */
+       for (i = 0; i < (count - 1); i++) {
+               reg = &dom->regions[i];
+               for (j = i + 1; j < count; j++) {
+                       reg1 = &dom->regions[j];
+
+                       if (is_region_conflict(reg1, reg))
+                               return SBI_EINVAL;
+
+                       if (!is_region_before(reg1, reg))
+                               continue;
+
+                       sbi_memcpy(&treg, reg1, sizeof(treg));
+                       sbi_memcpy(reg1, reg, sizeof(treg));
+                       sbi_memcpy(reg, &treg, sizeof(treg));
+               }
+       }
+
+       /*
+        * We don't need to check boot HART id of domain because if boot
+        * HART id is not possible/assigned to this domain then it won't
+        * be started at boot-time by sbi_domain_finalize().
+        */
+
Quoted 8 lines

lib/sbi/sbi_domain.c

Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+       /* Check next mode */
+       if (dom->next_mode != PRV_M &&
+           dom->next_mode != PRV_S &&
+           dom->next_mode != PRV_U)
+               return SBI_EINVAL;
+
+       /* Check next address and next mode*/
+       if (!sbi_domain_check_addr(dom, dom->next_addr, dom-

next_mode,

Quoted 92 lines
Quoted 92 lines
lib/sbi/sbi_domain.c new file +92 -0
+                                  FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
+
+       return 0;
+}
+
+int sbi_domain_finalize(struct sbi_scratch *scratch, u32
+cold_hartid) {
+       int rc;
+       u32 i, j, dhart;
+       bool dom_exists;
+       struct sbi_domain *dom, *tdom;
+       const struct sbi_platform *plat =
+sbi_platform_ptr(scratch);
+
+       /* Discover domains */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               /* Ignore invalid HART */
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+
+               /* Get domain assigned to HART */
+               dom = sbi_platform_domain_get(plat, i);
+               if (!dom)
+                       continue;
+
+               /* Check if domain already discovered */
+               dom_exists = FALSE;
+               sbi_domain_for_each(j, tdom) {
+                       if (tdom == dom) {
+                               dom_exists = TRUE;
+                               break;
+                       }
+               }
+
+               /* Newly discovered domain */
+               if (!dom_exists) {
+                       /* Sanitize discovered domain */
+                       rc = sanitize_domain(plat, dom);
+                       if (rc)
+                               return rc;
+
+                       /* Assign index to domain */
+                       dom->index = domain_count++;
+                       domidx_to_domain_table[dom->index] = dom;
+
+                       /* Clear assigned HARTs of domain */
+                       sbi_hartmask_clear_all(&dom->assigned_harts);
+               }
+
+               /* Assign domain to HART if HART is a possible HART */
+               if (sbi_hartmask_test_hart(i, dom->possible_harts)) {
+                       tdom = hartid_to_domain_table[i];
+                       if (tdom)
+                               sbi_hartmask_clear_hart(i,
+                                               &tdom->assigned_harts);
+                       hartid_to_domain_table[i] = dom;
+                       sbi_hartmask_set_hart(i, &dom->assigned_harts);
+               }
+       }
+
+       /* Startup boot HART of domains */
+       sbi_domain_for_each(i, dom) {
+               /* Domain boot HART */
+               dhart = dom->boot_hartid;
+
+               /* Ignore if boot HART not possible for this domain */
+               if (!sbi_hartmask_test_hart(i, dom->possible_harts))
+                       continue;
+
+               /* Ignore if boot HART assigned different domain */
+               if (sbi_hartid_to_domain(dhart) != dom ||
+                   !sbi_hartmask_test_hart(i, &dom->assigned_harts))
+                       continue;
+
+               /* Startup boot HART of domain */
+               if (dhart == cold_hartid) {
+                       scratch->next_addr = dom->next_addr;
+                       scratch->next_mode = dom->next_mode;
+                       scratch->next_arg1 = dom->next_arg1;
+               } else {
+                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
+                                       dom->next_mode, dom->next_arg1);
+                       if (rc)
+                               return rc;
+               }
+       }
+
+       return 0;
+}
+
+int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
Quoted 16 lines
Quoted 7 lines

lib/sbi/sbi_domain.c

Quoted 7 lines
lib/sbi/sbi_domain.c new file +7 -0
+{
+       u32 i;
+       const struct sbi_platform *plat =
+sbi_platform_ptr(scratch);
+
+       /* Root domain firmware memory region */
+       root_memregs[ROOT_FW_REGION].order = log2roundup(scratch-

fw_size);

Quoted 8 lines
Quoted 8 lines
lib/sbi/sbi_domain.c new file +8 -0
+       root_memregs[ROOT_FW_REGION].base = scratch->fw_start &
+                               ~((1UL << root_memregs[0].order) - 1UL);
+       root_memregs[ROOT_FW_REGION].flags = 0;
+
+       /* Root domain allow everything memory region */
+       root_memregs[ROOT_ALL_REGION].order = __riscv_xlen;
+       root_memregs[ROOT_ALL_REGION].base = 0;
+       root_memregs[ROOT_ALL_REGION].flags =

(SBI_DOMAIN_MEMREGION_READABLE |

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/sbi/sbi_domain.c new file +30 -0
+                                               SBI_DOMAIN_MEMREGION_WRITEABLE |
+
+ SBI_DOMAIN_MEMREGION_EXECUTABLE);
+
+       /* Root domain memory region end */
+       root_memregs[ROOT_END_REGION].order = 0;
+
+       /* Root domain boot HART id is same as coldboot HART id */
+       root.boot_hartid = cold_hartid;
+
+       /* Root domain next booting stage details */
+       root.next_arg1 = scratch->next_arg1;
+       root.next_addr = scratch->next_addr;
+       root.next_mode = scratch->next_mode;
+
+       /* Select root domain for all valid HARTs */
+       for (i = 0; i < SBI_HARTMASK_MAX_BITS; i++) {
+               if (sbi_platform_hart_invalid(plat, i))
+                       continue;
+               sbi_hartmask_set_hart(i, &root_hmask);
+               hartid_to_domain_table[i] = &root;
+               sbi_hartmask_set_hart(i, &root.assigned_harts);
+       }
+
+       /* Set root domain index */
+       root.index = domain_count++;
+       domidx_to_domain_table[root.index] = &root;
+
+       return 0;
+}
lib/sbi/sbi_init.c index renamed from lib/sbi/sbi_init.c +0 -0

5cedb15..406cb3f 100644

lib/sbi/sbi_init.c +0 -0
@@ -12,6 +12,7 @@
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_init.c static void __noreturn init_coldboot(struct

Quoted 8 lines
lib/sbi/sbi_init.c +1 -0
 #include <sbi/riscv_barrier.h>
 #include <sbi/riscv_locks.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hartmask.h>
@@ -169,6 +170,11 @@ static void __noreturn init_coldboot(struct

sbi_scratch *scratch, u32 hartid)

Quoted 9 lines
Quoted 9 lines

lib/sbi/sbi_init.c static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)

Quoted 9 lines
lib/sbi/sbi_init.c +5 -0
        if (rc)
                sbi_hart_hang();
 
+       /* Note: This has to be second thing in coldboot init sequence */
+       rc = sbi_domain_init(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        init_count_offset =

sbi_scratch_alloc_offset(__SIZEOF_POINTER__,

Quoted 3 lines
Quoted 3 lines

lib/sbi/sbi_init.c static void __noreturn init_coldboot(struct

Quoted 3 lines
lib/sbi/sbi_init.c +0 -0
                                                     "INIT_COUNT");
        if (!init_count_offset)
@@ -210,10 +216,24 @@ static void __noreturn init_coldboot(struct

sbi_scratch *scratch, u32 hartid)

Quoted 39 lines
Quoted 38 lines
Quoted 28 lines
lib/sbi/sbi_init.c +14 -1
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Finalize domains after HSM initialization so that we
+        * can startup non-root domains.
+        * Note: Finalize domains before HART PMP configuration so
+        * that we use correct domain for configuring PMP.
+        */
+       rc = sbi_domain_finalize(scratch, hartid);
+       if (rc)
+               sbi_hart_hang();
+
        rc = sbi_hart_pmp_configure(scratch);
        if (rc)
                sbi_hart_hang();
 
+       /*
+        * Note: Platform final initialization should be last so that
+        * it sees correct domain assignment and PMP configuration.
+        */
        rc = sbi_platform_final_init(plat, TRUE);
        if (rc)
                sbi_hart_hang();
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Quoted 8 lines
Quoted 2 lines

Regards,
Anup

--
Regards,
Atish

Regards
Anup

author
#18 lore raw

PATCH 7/16 lib: sbi: Extend sbi_hsm_hart_started_mask() for domains

In reply to Anup Patel

The sbi_hsm_hart_started_mask() API should take one more parameter to allow caller specify domain under which started_mask is being generated. Further, the sbi_hsm_hart_started_mask() depends on sbi_hsm_hart_get_state() which also should return HART state under specified domain.

This patch updates both sbi_hsm_hart_started_mask() and sbi_hsm_hart_get_state() as-per above.

---
 include/sbi/sbi_hsm.h      |  7 ++++---
 lib/sbi/sbi_ecall_hsm.c    |  4 +++-
 lib/sbi/sbi_ecall_legacy.c |  4 +++-
 lib/sbi/sbi_hsm.c          | 17 ++++++++++-------
 lib/sbi/sbi_ipi.c          |  6 ++++--
 lib/sbi/sbi_system.c       |  4 +++-
 6 files changed, 27 insertions(+), 15 deletions(-)
include/sbi/sbi_hsm.h +4 -3
index 18d129b..296b267 100644
@@ -19,6 +19,7 @@
 #define SBI_HART_STARTED	3
 #define SBI_HART_UNKNOWN	4
 
+struct sbi_domain;
 struct sbi_scratch;
 
 int sbi_hsm_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
@@ -27,10 +28,10 @@ void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);
 int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 		       ulong saddr, ulong smode, ulong priv);
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
-int sbi_hsm_hart_get_state(u32 hartid);
+int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid);
 int sbi_hsm_hart_state_to_status(int state);
-bool sbi_hsm_hart_started(u32 hartid);
-int sbi_hsm_hart_started_mask(ulong hbase, ulong *out_hmask);
+int sbi_hsm_hart_started_mask(const struct sbi_domain *dom,
+			      ulong hbase, ulong *out_hmask);
 void sbi_hsm_prepare_next_jump(struct sbi_scratch *scratch, u32 hartid);
 
 #endif
lib/sbi/sbi_ecall_hsm.c +3 -1
index 992c93a..3698a41 100644
@@ -7,6 +7,7 @@
  *   Atish Patra <[email protected]>
  */
 
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_ecall_interface.h>
 #include <sbi/sbi_error.h>
@@ -34,7 +35,8 @@ static int sbi_ecall_hsm_handler(unsigned long extid, unsigned long funcid,
 		ret = sbi_hsm_hart_stop(scratch, TRUE);
 		break;
 	case SBI_EXT_HSM_HART_GET_STATUS:
-		hstate = sbi_hsm_hart_get_state(args[0]);
+		hstate = sbi_hsm_hart_get_state(sbi_domain_thishart_ptr(),
+						args[0]);
 		ret = sbi_hsm_hart_state_to_status(hstate);
 		break;
 	default:
lib/sbi/sbi_ecall_legacy.c +3 -1
index 3bfe749..909cbe3 100644
@@ -10,6 +10,7 @@
 
 #include <sbi/riscv_asm.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_ecall_interface.h>
 #include <sbi/sbi_error.h>
@@ -33,7 +34,8 @@ static int sbi_load_hart_mask_unpriv(ulong *pmask, ulong *hmask,
 		if (uptrap->cause)
 			return SBI_ETRAP;
 	} else {
-		sbi_hsm_hart_started_mask(0, &mask);
+		sbi_hsm_hart_started_mask(sbi_domain_thishart_ptr(),
+					  0, &mask);
 	}
 	*hmask = mask;
 
lib/sbi/sbi_hsm.c +10 -7
index 8c1b7b0..1f8dadc 100644
@@ -13,6 +13,7 @@
 #include <sbi/riscv_atomic.h>
 #include <sbi/sbi_bitops.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_error.h>
 #include <sbi/sbi_ecall_interface.h>
 #include <sbi/sbi_hart.h>
@@ -56,13 +57,13 @@ int sbi_hsm_hart_state_to_status(int state)
 	return ret;
 }
 
-int sbi_hsm_hart_get_state(u32 hartid)
+int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid)
 {
 	struct sbi_hsm_data *hdata;
 	struct sbi_scratch *scratch;
 
 	scratch = sbi_hartid_to_scratch(hartid);
-	if (!scratch)
+	if (!scratch || !sbi_domain_assigned_hart(dom, hartid))
 		return SBI_HART_UNKNOWN;
 
 	hdata = sbi_scratch_offset_ptr(scratch, hart_data_offset);
@@ -70,9 +71,9 @@ int sbi_hsm_hart_get_state(u32 hartid)
 	return atomic_read(&hdata->state);
 }
 
-bool sbi_hsm_hart_started(u32 hartid)
+static bool sbi_hsm_hart_started(const struct sbi_domain *dom, u32 hartid)
 {
-	if (sbi_hsm_hart_get_state(hartid) == SBI_HART_STARTED)
+	if (sbi_hsm_hart_get_state(dom, hartid) == SBI_HART_STARTED)
 		return TRUE;
 	else
 		return FALSE;
@@ -80,12 +81,14 @@ bool sbi_hsm_hart_started(u32 hartid)
 
 /**
  * Get ulong HART mask for given HART base ID
+ * @param dom the domain to be used for output HART mask
  * @param hbase the HART base ID
  * @param out_hmask the output ulong HART mask
  * @return 0 on success and SBI_Exxx (< 0) on failure
  * Note: the output HART mask will be set to zero on failure as well.
  */
-int sbi_hsm_hart_started_mask(ulong hbase, ulong *out_hmask)
+int sbi_hsm_hart_started_mask(const struct sbi_domain *dom,
+			      ulong hbase, ulong *out_hmask)
 {
 	ulong i;
 	ulong hcount = sbi_scratch_last_hartid() + 1;
@@ -97,7 +100,7 @@ int sbi_hsm_hart_started_mask(ulong hbase, ulong *out_hmask)
 		hcount = BITS_PER_LONG;
 
 	for (i = hbase; i < hcount; i++) {
-		if (sbi_hsm_hart_get_state(i) == SBI_HART_STARTED)
+		if (sbi_hsm_hart_get_state(dom, i) == SBI_HART_STARTED)
 			*out_hmask |= 1UL << (i - hbase);
 	}
 
@@ -259,7 +262,7 @@ int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow)
 	struct sbi_hsm_data *hdata = sbi_scratch_offset_ptr(scratch,
 							    hart_data_offset);
 
-	if (!sbi_hsm_hart_started(hartid))
+	if (!sbi_hsm_hart_started(sbi_domain_thishart_ptr(), hartid))
 		return SBI_EINVAL;
 
 	oldstate = atomic_cmpxchg(&hdata->state, SBI_HART_STARTED,
lib/sbi/sbi_ipi.c +4 -2
index a27dea0..4347832 100644
@@ -12,6 +12,7 @@
 #include <sbi/riscv_atomic.h>
 #include <sbi/riscv_barrier.h>
 #include <sbi/sbi_bitops.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_error.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hsm.h>
@@ -77,10 +78,11 @@ int sbi_ipi_send_many(ulong hmask, ulong hbase, u32 event, void *data)
 {
 	int rc;
 	ulong i, m;
+	struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
 
 	if (hbase != -1UL) {
-		rc = sbi_hsm_hart_started_mask(hbase, &m);
+		rc = sbi_hsm_hart_started_mask(dom, hbase, &m);
 		if (rc)
 			return rc;
 		m &= hmask;
@@ -92,7 +94,7 @@ int sbi_ipi_send_many(ulong hmask, ulong hbase, u32 event, void *data)
 		}
 	} else {
 		hbase = 0;
-		while (!sbi_hsm_hart_started_mask(hbase, &m)) {
+		while (!sbi_hsm_hart_started_mask(dom, hbase, &m)) {
 			/* Send IPIs */
 			for (i = hbase; m; i++, m >>= 1) {
 				if (m & 1UL)
lib/sbi/sbi_system.c +3 -1
index 6f7be14..a938003 100644
@@ -10,6 +10,7 @@
 
 #include <sbi/riscv_asm.h>
 #include <sbi/sbi_bitops.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_hsm.h>
 #include <sbi/sbi_platform.h>
@@ -21,10 +22,11 @@ void __noreturn sbi_system_reset(u32 platform_reset_type)
 {
 	ulong hbase = 0, hmask;
 	u32 cur_hartid = current_hartid();
+	struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
 
 	/* Send HALT IPI to every hart other than the current hart */
-	while (!sbi_hsm_hart_started_mask(hbase, &hmask)) {
+	while (!sbi_hsm_hart_started_mask(dom, hbase, &hmask)) {
 		if (hbase <= cur_hartid)
 			hmask &= ~(1UL << (cur_hartid - hbase));
 		if (hmask)
Signature · 1 line
2.25.1
author
#19 lore raw

PATCH 8/16 lib: sbi: Extend sbi_hsm_hart_start() for domains

In reply to Anup Patel

The sbi_hsm_hart_start() should consider the domain under which we are trying to start the HART. This will help ensure that HART A can start HART B only if both HARTs A and B belong to the same domain.

We also have a special case when we bring-up boot HART of non-root domains in sbi_domain_finalize() where we should skip domain checks in sbi_hsm_hart_start(). To achieve this, sbi_hsm_hart_start() should do domain checks only when domain parameter is non-NULL.

This patch extends sbi_hsm_hart_start() as-per above.

---
 include/sbi/sbi_hsm.h   |  5 +++--
 lib/sbi/sbi_domain.c    |  6 ++++--
 lib/sbi/sbi_ecall_hsm.c |  4 ++--
 lib/sbi/sbi_hsm.c       | 16 ++++++++--------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/sbi/sbi_hsm.h +3 -2
index 296b267..4823383 100644
@@ -25,8 +25,9 @@ struct sbi_scratch;
 int sbi_hsm_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
 void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-		       ulong saddr, ulong smode, ulong priv);
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+		       const struct sbi_domain *dom,
+		       u32 hartid, ulong saddr, ulong smode, ulong priv);
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
 int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid);
 int sbi_hsm_hart_state_to_status(int state);
lib/sbi/sbi_domain.c +4 -2
index 8d5dae1..df0f2a5 100644
@@ -309,8 +309,10 @@ int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
 			scratch->next_mode = dom->next_mode;
 			scratch->next_arg1 = dom->next_arg1;
 		} else {
-			rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
-					dom->next_mode, dom->next_arg1);
+			rc = sbi_hsm_hart_start(scratch, NULL, dhart,
+						dom->next_addr,
+						dom->next_mode,
+						dom->next_arg1);
 			if (rc)
 				return rc;
 		}
lib/sbi/sbi_ecall_hsm.c +2 -2
index 3698a41..376740c 100644
@@ -28,8 +28,8 @@ static int sbi_ecall_hsm_handler(unsigned long extid, unsigned long funcid,
 	case SBI_EXT_HSM_HART_START:
 		smode = csr_read(CSR_MSTATUS);
 		smode = (smode & MSTATUS_MPP) >> MSTATUS_MPP_SHIFT;
-		ret = sbi_hsm_hart_start(scratch, args[0], args[1],
-					 smode, args[2]);
+		ret = sbi_hsm_hart_start(scratch, sbi_domain_thishart_ptr(),
+					 args[0], args[1], smode, args[2]);
 		break;
 	case SBI_EXT_HSM_HART_STOP:
 		ret = sbi_hsm_hart_stop(scratch, TRUE);
lib/sbi/sbi_hsm.c +8 -8
index 1f8dadc..c16d542 100644
@@ -205,10 +205,10 @@ fail_exit:
 	sbi_hart_hang();
 }
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-		       ulong saddr, ulong smode, ulong priv)
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+		       const struct sbi_domain *dom,
+		       u32 hartid, ulong saddr, ulong smode, ulong priv)
 {
-	int rc;
 	unsigned long init_count;
 	unsigned int hstate;
 	struct sbi_scratch *rscratch;
@@ -217,6 +217,11 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 
 	if (smode != PRV_M && smode != PRV_S && smode != PRV_U)
 		return SBI_EINVAL;
+	if (dom && !sbi_domain_assigned_hart(dom, hartid))
+		return SBI_EINVAL;
+	if (dom && !sbi_domain_check_addr(dom, saddr, smode,
+					  FALSE, FALSE, FALSE, TRUE))
+		return SBI_EINVAL;
 
 	rscratch = sbi_hartid_to_scratch(hartid);
 	if (!rscratch)
@@ -234,11 +239,6 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 	if (hstate != SBI_HART_STOPPED)
 		return SBI_EINVAL;
 
-	rc = sbi_hart_pmp_check_addr(scratch, saddr, smode, PMP_X);
-	if (rc)
-		return rc;
-	//TODO: We also need to check saddr for valid physical address as well.
-
 	init_count = sbi_init_count(hartid);
 	rscratch->next_arg1 = priv;
 	rscratch->next_addr = saddr;
Signature · 1 line
2.25.1
#20 lore raw

PATCH 8/16 lib: sbi: Extend sbi_hsm_hart_start() for domains

Quoted 99 lines · Anup Patel

The sbi_hsm_hart_start() should consider the domain under which we are trying to start the HART. This will help ensure that HART A can start HART B only if both HARTs A and B belong to the same domain.

We also have a special case when we bring-up boot HART of non-root domains in sbi_domain_finalize() where we should skip domain checks in sbi_hsm_hart_start(). To achieve this, sbi_hsm_hart_start() should do domain checks only when domain parameter is non-NULL.

This patch extends sbi_hsm_hart_start() as-per above.

---
 include/sbi/sbi_hsm.h   |  5 +++--
 lib/sbi/sbi_domain.c    |  6 ++++--
 lib/sbi/sbi_ecall_hsm.c |  4 ++--
 lib/sbi/sbi_hsm.c       | 16 ++++++++--------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/sbi/sbi_hsm.h +3 -2
index 296b267..4823383 100644
@@ -25,8 +25,9 @@ struct sbi_scratch;
 int sbi_hsm_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
 void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv);
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv);
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
 int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid);
 int sbi_hsm_hart_state_to_status(int state);
lib/sbi/sbi_domain.c +4 -2
index 8d5dae1..df0f2a5 100644
@@ -309,8 +309,10 @@ int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
                        scratch->next_mode = dom->next_mode;
                        scratch->next_arg1 = dom->next_arg1;
                } else {
-                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
-                                       dom->next_mode, dom->next_arg1);
+                       rc = sbi_hsm_hart_start(scratch, NULL, dhart,
+                                               dom->next_addr,
+                                               dom->next_mode,
+                                               dom->next_arg1);
                        if (rc)
                                return rc;
                }
lib/sbi/sbi_ecall_hsm.c +2 -2
index 3698a41..376740c 100644
@@ -28,8 +28,8 @@ static int sbi_ecall_hsm_handler(unsigned long extid, unsigned long funcid,
        case SBI_EXT_HSM_HART_START:
                smode = csr_read(CSR_MSTATUS);
                smode = (smode & MSTATUS_MPP) >> MSTATUS_MPP_SHIFT;
-               ret = sbi_hsm_hart_start(scratch, args[0], args[1],
-                                        smode, args[2]);
+               ret = sbi_hsm_hart_start(scratch, sbi_domain_thishart_ptr(),
+                                        args[0], args[1], smode, args[2]);
                break;
        case SBI_EXT_HSM_HART_STOP:
                ret = sbi_hsm_hart_stop(scratch, TRUE);
lib/sbi/sbi_hsm.c +8 -3
index 1f8dadc..c16d542 100644
@@ -205,10 +205,10 @@ fail_exit:
        sbi_hart_hang();
 }
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv)
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv)
 {
-       int rc;
        unsigned long init_count;
        unsigned int hstate;
        struct sbi_scratch *rscratch;
@@ -217,6 +217,11 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 
        if (smode != PRV_M && smode != PRV_S && smode != PRV_U)
                return SBI_EINVAL;
+       if (dom && !sbi_domain_assigned_hart(dom, hartid))
+               return SBI_EINVAL;
+       if (dom && !sbi_domain_check_addr(dom, saddr, smode,
+                                         FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
 
        rscratch = sbi_hartid_to_scratch(hartid);
        if (!rscratch)

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid, · line 241

Quoted 8 lines
lib/sbi/sbi_hsm.c +0 -5
@@ -234,11 +239,6 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
        if (hstate != SBI_HART_STOPPED)
                return SBI_EINVAL;
 
-       rc = sbi_hart_pmp_check_addr(scratch, saddr, smode, PMP_X);
-       if (rc)
-               return rc;
-       //TODO: We also need to check saddr for valid physical address as well.
-

Why is the check being removed ? IIRC, this API was improved in your last series to include the mode but removed completely later in this series.
Maybe I missed something in the series or yet to review.

Quoted 11 lines
lib/sbi/sbi_hsm.c +0 -1
        init_count = sbi_init_count(hartid);
        rscratch->next_arg1 = priv;
        rscratch->next_addr = saddr;
--

2.25.1

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
#21 lore raw

PATCH 8/16 lib: sbi: Extend sbi_hsm_hart_start() for domains

Quoted 104 lines · Atish Patra
Quoted 102 lines · Anup Patel

The sbi_hsm_hart_start() should consider the domain under which we are trying to start the HART. This will help ensure that HART A can start HART B only if both HARTs A and B belong to the same domain.

We also have a special case when we bring-up boot HART of non-root domains in sbi_domain_finalize() where we should skip domain checks in sbi_hsm_hart_start(). To achieve this, sbi_hsm_hart_start() should do domain checks only when domain parameter is non-NULL.

This patch extends sbi_hsm_hart_start() as-per above.

---
 include/sbi/sbi_hsm.h   |  5 +++--
 lib/sbi/sbi_domain.c    |  6 ++++--
 lib/sbi/sbi_ecall_hsm.c |  4 ++--
 lib/sbi/sbi_hsm.c       | 16 ++++++++--------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/sbi/sbi_hsm.h +3 -2
index 296b267..4823383 100644
@@ -25,8 +25,9 @@ struct sbi_scratch;
 int sbi_hsm_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
 void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv);
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv);
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
 int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid);
 int sbi_hsm_hart_state_to_status(int state);
lib/sbi/sbi_domain.c +4 -2
index 8d5dae1..df0f2a5 100644
@@ -309,8 +309,10 @@ int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
                        scratch->next_mode = dom->next_mode;
                        scratch->next_arg1 = dom->next_arg1;
                } else {
-                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
-                                       dom->next_mode, dom->next_arg1);
+                       rc = sbi_hsm_hart_start(scratch, NULL, dhart,
+                                               dom->next_addr,
+                                               dom->next_mode,
+                                               dom->next_arg1);
                        if (rc)
                                return rc;
                }
lib/sbi/sbi_ecall_hsm.c +2 -2
index 3698a41..376740c 100644
@@ -28,8 +28,8 @@ static int sbi_ecall_hsm_handler(unsigned long extid, unsigned long funcid,
        case SBI_EXT_HSM_HART_START:
                smode = csr_read(CSR_MSTATUS);
                smode = (smode & MSTATUS_MPP) >> MSTATUS_MPP_SHIFT;
-               ret = sbi_hsm_hart_start(scratch, args[0], args[1],
-                                        smode, args[2]);
+               ret = sbi_hsm_hart_start(scratch, sbi_domain_thishart_ptr(),
+                                        args[0], args[1], smode, args[2]);
                break;
        case SBI_EXT_HSM_HART_STOP:
                ret = sbi_hsm_hart_stop(scratch, TRUE);
lib/sbi/sbi_hsm.c +8 -3
index 1f8dadc..c16d542 100644
@@ -205,10 +205,10 @@ fail_exit:
        sbi_hart_hang();
 }
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv)
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv)
 {
-       int rc;
        unsigned long init_count;
        unsigned int hstate;
        struct sbi_scratch *rscratch;
@@ -217,6 +217,11 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 
        if (smode != PRV_M && smode != PRV_S && smode != PRV_U)
                return SBI_EINVAL;
+       if (dom && !sbi_domain_assigned_hart(dom, hartid))
+               return SBI_EINVAL;
+       if (dom && !sbi_domain_check_addr(dom, saddr, smode,
+                                         FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
 
        rscratch = sbi_hartid_to_scratch(hartid);
        if (!rscratch)
@@ -234,11 +239,6 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
        if (hstate != SBI_HART_STOPPED)
                return SBI_EINVAL;
 
Quoted 10 lines

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid, · line 241

Quoted 5 lines
lib/sbi/sbi_hsm.c +0 -5
-       rc = sbi_hart_pmp_check_addr(scratch, saddr, smode, PMP_X);
-       if (rc)
-               return rc;
-       //TODO: We also need to check saddr for valid physical address as well.
-

Why is the check being removed ? IIRC, this API was improved in your last series to include the mode but removed completely later in this series.
Maybe I missed something in the series or yet to review.

The sbi_hart_pmp_check_addr() call is redundant here because the sbi_domain_check_addr() above does the required checks.

Quoted 15 lines
Quoted 8 lines
lib/sbi/sbi_hsm.c +0 -1
        init_count = sbi_init_count(hartid);
        rscratch->next_arg1 = priv;
        rscratch->next_addr = saddr;
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Quoted 3 lines

--
Regards,
Atish

Regards,
Anup

#22 lore raw

PATCH 8/16 lib: sbi: Extend sbi_hsm_hart_start() for domains

Quoted 117 lines · Anup Patel
Quoted 104 lines · Atish Patra
Quoted 102 lines · Anup Patel

The sbi_hsm_hart_start() should consider the domain under which we are trying to start the HART. This will help ensure that HART A can start HART B only if both HARTs A and B belong to the same domain.

We also have a special case when we bring-up boot HART of non-root domains in sbi_domain_finalize() where we should skip domain checks in sbi_hsm_hart_start(). To achieve this, sbi_hsm_hart_start() should do domain checks only when domain parameter is non-NULL.

This patch extends sbi_hsm_hart_start() as-per above.

---
 include/sbi/sbi_hsm.h   |  5 +++--
 lib/sbi/sbi_domain.c    |  6 ++++--
 lib/sbi/sbi_ecall_hsm.c |  4 ++--
 lib/sbi/sbi_hsm.c       | 16 ++++++++--------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/sbi/sbi_hsm.h +3 -2
index 296b267..4823383 100644
@@ -25,8 +25,9 @@ struct sbi_scratch;
 int sbi_hsm_init(struct sbi_scratch *scratch, u32 hartid, bool cold_boot);
 void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv);
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv);
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
 int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid);
 int sbi_hsm_hart_state_to_status(int state);
lib/sbi/sbi_domain.c +4 -2
index 8d5dae1..df0f2a5 100644
@@ -309,8 +309,10 @@ int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid)
                        scratch->next_mode = dom->next_mode;
                        scratch->next_arg1 = dom->next_arg1;
                } else {
-                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
-                                       dom->next_mode, dom->next_arg1);
+                       rc = sbi_hsm_hart_start(scratch, NULL, dhart,
+                                               dom->next_addr,
+                                               dom->next_mode,
+                                               dom->next_arg1);
                        if (rc)
                                return rc;
                }
lib/sbi/sbi_ecall_hsm.c +2 -2
index 3698a41..376740c 100644
@@ -28,8 +28,8 @@ static int sbi_ecall_hsm_handler(unsigned long extid, unsigned long funcid,
        case SBI_EXT_HSM_HART_START:
                smode = csr_read(CSR_MSTATUS);
                smode = (smode & MSTATUS_MPP) >> MSTATUS_MPP_SHIFT;
-               ret = sbi_hsm_hart_start(scratch, args[0], args[1],
-                                        smode, args[2]);
+               ret = sbi_hsm_hart_start(scratch, sbi_domain_thishart_ptr(),
+                                        args[0], args[1], smode, args[2]);
                break;
        case SBI_EXT_HSM_HART_STOP:
                ret = sbi_hsm_hart_stop(scratch, TRUE);
lib/sbi/sbi_hsm.c +8 -3
index 1f8dadc..c16d542 100644
@@ -205,10 +205,10 @@ fail_exit:
        sbi_hart_hang();
 }
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv)
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong priv)
 {
-       int rc;
        unsigned long init_count;
        unsigned int hstate;
        struct sbi_scratch *rscratch;
@@ -217,6 +217,11 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
 
        if (smode != PRV_M && smode != PRV_S && smode != PRV_U)
                return SBI_EINVAL;
+       if (dom && !sbi_domain_assigned_hart(dom, hartid))
+               return SBI_EINVAL;
+       if (dom && !sbi_domain_check_addr(dom, saddr, smode,
+                                         FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
 
        rscratch = sbi_hartid_to_scratch(hartid);
        if (!rscratch)
@@ -234,11 +239,6 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
        if (hstate != SBI_HART_STOPPED)
                return SBI_EINVAL;
 
Quoted 10 lines

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid, · line 241

Quoted 5 lines
lib/sbi/sbi_hsm.c +0 -5
-       rc = sbi_hart_pmp_check_addr(scratch, saddr, smode, PMP_X);
-       if (rc)
-               return rc;
-       //TODO: We also need to check saddr for valid physical address as well.
-

Why is the check being removed ? IIRC, this API was improved in your last series to include the mode but removed completely later in this series.
Maybe I missed something in the series or yet to review.

Quoted 3 lines

The sbi_hart_pmp_check_addr() call is redundant here because the sbi_domain_check_addr() above does the required checks.

Yes. I know. I was asking sbi_hart_pmp_check_addr improved in domain preparation series
but removed in actual domain series. That's why, I was asking why it was improved last time
if it was to be removed in the next series. I guess you changed your implementation approach in between :).

Quoted 21 lines
Quoted 15 lines
Quoted 8 lines
lib/sbi/sbi_hsm.c +0 -1
        init_count = sbi_init_count(hartid);
        rscratch->next_arg1 = priv;
        rscratch->next_addr = saddr;
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Quoted 3 lines

--
Regards,
Atish

Regards,
Anup

Signature · 2 lines
Regards,
Atish
author
#23 lore raw

PATCH 8/16 lib: sbi: Extend sbi_hsm_hart_start() for domains

Quoted 138 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 19 October 2020 05:25
To: Anup Patel <[email protected]>
Cc: Anup Patel <[email protected]>; Atish Patra <[email protected]>; Alistair Francis <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 08/16] lib: sbi: Extend sbi_hsm_hart_start() for domains

Quoted 4 lines · Anup Patel
Quoted 2 lines · Atish Patra

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 9 lines · wrote:
Quoted 9 lines
Quoted 9 lines

The sbi_hsm_hart_start() should consider the domain under which we are trying to start the HART. This will help ensure that HART A can start HART B only if both HARTs A and B belong to the same domain.

We also have a special case when we bring-up boot HART of non-root domains in sbi_domain_finalize() where we should skip domain checks in sbi_hsm_hart_start(). To achieve this, sbi_hsm_hart_start() should do domain checks only when domain

parameter is non-NULL.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

This patch extends sbi_hsm_hart_start() as-per above.

---
 include/sbi/sbi_hsm.h   |  5 +++--
 lib/sbi/sbi_domain.c    |  6 ++++--
 lib/sbi/sbi_ecall_hsm.c |  4 ++--
 lib/sbi/sbi_hsm.c       | 16 ++++++++--------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/sbi/sbi_hsm.h index renamed from include/sbi/sbi_hsm.h +0 -0

296b267..4823383 100644

include/sbi/sbi_hsm.h +0 -0
@@ -25,8 +25,9 @@ struct sbi_scratch;  int sbi_hsm_init(struct

sbi_scratch *scratch, u32 hartid, bool cold_boot); void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch);

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
include/sbi/sbi_hsm.h +4 -2
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv);
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong
+priv);
Quoted 8 lines

lib/sbi/sbi_domain.c int sbi_domain_finalize(struct sbi_scratch

Quoted 8 lines

int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow);
int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid); int sbi_hsm_hart_state_to_status(int state); diff --git a/lib/sbi/sbi_domain.c b/lib/sbi/sbi_domain.c index 8d5dae1..df0f2a5 100644

lib/sbi/sbi_domain.c +0 -0
@@ -309,8 +309,10 @@ int sbi_domain_finalize(struct sbi_scratch

*scratch, u32 cold_hartid)

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/sbi/sbi_domain.c +4 -2
                        scratch->next_mode = dom->next_mode;
                        scratch->next_arg1 = dom->next_arg1;
                } else {
-                       rc = sbi_hsm_hart_start(scratch, dhart, dom->next_addr,
-                                       dom->next_mode, dom->next_arg1);
+                       rc = sbi_hsm_hart_start(scratch, NULL, dhart,
+                                               dom->next_addr,
+                                               dom->next_mode,
+                                               dom->next_arg1);
Quoted 8 lines
Quoted 8 lines

lib/sbi/sbi_ecall_hsm.c static int sbi_ecall_hsm_handler(unsigned long

Quoted 8 lines
lib/sbi/sbi_domain.c +0 -0
                        if (rc)
                                return rc;
                }
lib/sbi/sbi_ecall_hsm.c +0 -0
index 3698a41..376740c 100644
@@ -28,8 +28,8 @@ static int sbi_ecall_hsm_handler(unsigned long

extid, unsigned long funcid,

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
lib/sbi/sbi_ecall_hsm.c +3 -2
        case SBI_EXT_HSM_HART_START:
                smode = csr_read(CSR_MSTATUS);
                smode = (smode & MSTATUS_MPP) >> MSTATUS_MPP_SHIFT;
-               ret = sbi_hsm_hart_start(scratch, args[0], args[1],
-                                        smode, args[2]);
+               ret = sbi_hsm_hart_start(scratch, sbi_domain_thishart_ptr(),
+                                        args[0], args[1], smode,
+ args[2]);
                break;
        case SBI_EXT_HSM_HART_STOP:
                ret = sbi_hsm_hart_stop(scratch, TRUE); diff --git

a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c index 1f8dadc..c16d542 100644

lib/sbi/sbi_hsm.c +4 -3
@@ -205,10 +205,10 @@ fail_exit:
        sbi_hart_hang();
 }
 
-int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,
-                      ulong saddr, ulong smode, ulong priv)
+int sbi_hsm_hart_start(struct sbi_scratch *scratch,
+                      const struct sbi_domain *dom,
+                      u32 hartid, ulong saddr, ulong smode, ulong
+priv)
 {
-       int rc;
        unsigned long init_count;
        unsigned int hstate;
        struct sbi_scratch *rscratch; @@ -217,6 +217,11 @@ int

sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,

        if (smode != PRV_M && smode != PRV_S && smode != PRV_U)
                return SBI_EINVAL;
lib/sbi/sbi_hsm.c +5 -0
+       if (dom && !sbi_domain_assigned_hart(dom, hartid))
+               return SBI_EINVAL;
+       if (dom && !sbi_domain_check_addr(dom, saddr, smode,
+                                         FALSE, FALSE, FALSE, TRUE))
+               return SBI_EINVAL;
Quoted 3 lines
Quoted 3 lines

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch

Quoted 3 lines
        rscratch = sbi_hartid_to_scratch(hartid);
        if (!rscratch)
lib/sbi/sbi_hsm.c +0 -0
@@ -234,11 +239,6 @@ int sbi_hsm_hart_start(struct sbi_scratch

*scratch, u32 hartid,

Quoted 7 lines
Quoted 7 lines

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid,

Quoted 7 lines
lib/sbi/sbi_hsm.c +0 -4
        if (hstate != SBI_HART_STOPPED)
                return SBI_EINVAL;
 
-       rc = sbi_hart_pmp_check_addr(scratch, saddr, smode, PMP_X);
-       if (rc)
-               return rc;
-       //TODO: We also need to check saddr for valid physical address as

well.

Quoted 7 lines
Quoted 6 lines

lib/sbi/sbi_hsm.c int sbi_hsm_hart_start(struct sbi_scratch *scratch, u32 hartid, · line 241

Quoted 1 line
lib/sbi/sbi_hsm.c +0 -1
-

Why is the check being removed ? IIRC, this API was improved in your last series to include the mode but removed completely later in this series.
Maybe I missed something in the series or yet to review.

Quoted 8 lines
Quoted 3 lines

The sbi_hart_pmp_check_addr() call is redundant here because the sbi_domain_check_addr() above does the required checks.

Yes. I know. I was asking sbi_hart_pmp_check_addr improved in domain preparation series but removed in actual domain series. That's why, I was asking why it was improved last time if it was to be removed in the next series. I guess you changed your implementation approach in between :).

Yes, initially I thought of reusing sbi_hart_pmp_check_addr() but later on I realized that when IOPMP comes-in we will end-up with similar checks for IOPMP. It's better to have a centralized address check function in domains which can be used from anywhere.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 9 lines
lib/sbi/sbi_hsm.c +0 -1
        init_count = sbi_init_count(hartid);
        rscratch->next_arg1 = priv;
        rscratch->next_addr = saddr;
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Quoted 8 lines
Quoted 2 lines

Regards,
Anup

--
Regards,
Atish

Regards,
Anup

author
#24 lore raw

PATCH 9/16 lib: sbi: Optimize sbi_hsm_hart_started_mask() implementation

In reply to Anup Patel

Instead of calling sbi_hsm_hart_get_state() in a loop, we can simply call a new inline __sbi_hsm_hart_get_state() which only takes "hartid" and enforce domain checks using sbi_domain_assigned_hartmask().

This patch optimizes sbi_hsm_hart_started_mask() as-per above.

---
 lib/sbi/sbi_hsm.c | 34 ++++++++++++++++++++++------------
 1 file changed, 22 insertions(+), 12 deletions(-)
lib/sbi/sbi_hsm.c +22 -12
index c16d542..d443ba8 100644
@@ -57,20 +57,27 @@ int sbi_hsm_hart_state_to_status(int state)
 	return ret;
 }
 
-int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid)
+static inline int __sbi_hsm_hart_get_state(u32 hartid)
 {
 	struct sbi_hsm_data *hdata;
 	struct sbi_scratch *scratch;
 
 	scratch = sbi_hartid_to_scratch(hartid);
-	if (!scratch || !sbi_domain_assigned_hart(dom, hartid))
+	if (!scratch)
 		return SBI_HART_UNKNOWN;
 
 	hdata = sbi_scratch_offset_ptr(scratch, hart_data_offset);
-
 	return atomic_read(&hdata->state);
 }
 
+int sbi_hsm_hart_get_state(const struct sbi_domain *dom, u32 hartid)
+{
+	if (!sbi_domain_assigned_hart(dom, hartid))
+		return SBI_HART_UNKNOWN;
+
+	return __sbi_hsm_hart_get_state(hartid);
+}
+
 static bool sbi_hsm_hart_started(const struct sbi_domain *dom, u32 hartid)
 {
 	if (sbi_hsm_hart_get_state(dom, hartid) == SBI_HART_STARTED)
@@ -90,18 +97,21 @@ static bool sbi_hsm_hart_started(const struct sbi_domain *dom, u32 hartid)
 int sbi_hsm_hart_started_mask(const struct sbi_domain *dom,
 			      ulong hbase, ulong *out_hmask)
 {
-	ulong i;
-	ulong hcount = sbi_scratch_last_hartid() + 1;
+	ulong i, hmask, dmask;
+	ulong hend = sbi_scratch_last_hartid() + 1;
 
 	*out_hmask = 0;
-	if (hcount <= hbase)
+	if (hend <= hbase)
 		return SBI_EINVAL;
-	if (BITS_PER_LONG < (hcount - hbase))
-		hcount = BITS_PER_LONG;
-
-	for (i = hbase; i < hcount; i++) {
-		if (sbi_hsm_hart_get_state(dom, i) == SBI_HART_STARTED)
-			*out_hmask |= 1UL << (i - hbase);
+	if (BITS_PER_LONG < (hend - hbase))
+		hend = hbase + BITS_PER_LONG;
+
+	dmask = sbi_domain_assigned_hartmask(dom, hbase);
+	for (i = hbase; i < hend; i++) {
+		hmask = 1UL << (i - hbase);
+		if ((dmask & hmask) &&
+		    (__sbi_hsm_hart_get_state(i) == SBI_HART_STARTED))
+			*out_hmask |= hmask;
 	}
 
 	return 0;
Signature · 1 line
2.25.1
author
#25 lore raw

PATCH 10/16 lib: sbi: Extend sbi_system_reset() for domains

In reply to Anup Patel

The sbi_system_reset() should issue platform system reset only if domain of current HART is allowed to do system reset.

This patch extends sbi_system_reset() as-per above.

---
 lib/sbi/sbi_system.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
lib/sbi/sbi_system.c +4 -3
index a938003..96290b3 100644
@@ -37,9 +37,10 @@ void __noreturn sbi_system_reset(u32 platform_reset_type)
 	/* Stop current HART */
 	sbi_hsm_hart_stop(scratch, FALSE);
 
-	/* Platform specific reset */
-	sbi_platform_system_reset(sbi_platform_ptr(scratch),
-				  platform_reset_type);
+	/* Platform specific reset if domain allowed system reset */
+	if (dom->system_reset_allowed)
+		sbi_platform_system_reset(sbi_platform_ptr(scratch),
+					  platform_reset_type);
 
 	/* If platform specific reset did not work then do sbi_exit() */
 	sbi_exit(scratch);
Signature · 1 line
2.25.1
author
#26 lore raw

PATCH 11/16 lib: utils: Update fdt_reserved_memory_fixup() to use current domain

In reply to Anup Patel

Now that each HART is mapped to a domain having a set of memory regions, we update fdt_reserved_memory_fixup() to use domain memory regions for adding reserved memory nodes in device tree.

We also change reserved memory node name prefix from "mmode_pmp" to "mmode_resv" because domain memory regions can impact other hardware configurations (such as IOPMP, etc) along with PMP.

---
 lib/utils/fdt/fdt_fixup.c | 47 ++++++++++++++++++---------------------
 1 file changed, 22 insertions(+), 25 deletions(-)
lib/utils/fdt/fdt_fixup.c +22 -25
index 8f554e1..e9e1dff 100644
@@ -9,6 +9,7 @@
 
 #include <libfdt.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_math.h>
 #include <sbi/sbi_hart.h>
 #include <sbi/sbi_platform.h>
@@ -97,11 +98,11 @@ static int fdt_resv_memory_update_node(void *fdt, unsigned long addr,
 
 	if (na > 1 && addr_high)
 		sbi_snprintf(name, sizeof(name),
-			     "mmode_pmp%d@%x,%x", index,
+			     "mmode_resv%d@%x,%x", index,
 			     addr_high, addr_low);
 	else
 		sbi_snprintf(name, sizeof(name),
-			     "mmode_pmp%d@%x", index,
+			     "mmode_resv%d@%x", index,
 			     addr_low);
 
 	subnode = fdt_add_subnode(fdt, parent, name);
@@ -153,10 +154,11 @@ static int fdt_resv_memory_update_node(void *fdt, unsigned long addr,
  */
 int fdt_reserved_memory_fixup(void *fdt)
 {
+	struct sbi_domain_memregion *reg;
+	struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
-	unsigned long prot, addr, size;
-	int parent, i, j;
-	int err;
+	unsigned long addr, size;
+	int err, parent, i;
 	int na = fdt_address_cells(fdt, 0);
 	int ns = fdt_size_cells(fdt, 0);
 
@@ -203,34 +205,29 @@ int fdt_reserved_memory_fixup(void *fdt)
 	 * We assume the given device tree does not contain any memory region
 	 * child node protected by PMP. Normally PMP programming happens at
 	 * M-mode firmware. The memory space used by OpenSBI is protected.
-	 * Some additional memory spaces may be protected by platform codes.
+	 * Some additional memory spaces may be protected by domain memory
+	 * regions.
 	 *
 	 * With above assumption, we create child nodes directly.
 	 */
 
-	if (!sbi_hart_pmp_count(scratch)) {
-		/*
-		 * Update the DT with firmware start & size even if PMP is not
-		 * supported. This makes sure that supervisor OS is always
-		 * aware of OpenSBI resident memory area.
-		 */
-		addr = scratch->fw_start & ~(scratch->fw_size - 1UL);
-		size = (1UL << log2roundup(scratch->fw_size));
-		return fdt_resv_memory_update_node(fdt, addr, size,
-						   0, parent, true);
-	}
-
-	for (i = 0, j = 0; i < sbi_hart_pmp_count(scratch); i++) {
-		err = sbi_hart_pmp_get(scratch, i, &prot, &addr, &size);
-		if (err)
+	i = 0;
+	sbi_domain_for_each_memregion(dom, reg) {
+		/* Ignore MMIO or READABLE or WRITABLE or EXECUTABLE regions */
+		if (reg->flags & SBI_DOMAIN_MEMREGION_MMIO)
+			continue;
+		if (reg->flags & SBI_DOMAIN_MEMREGION_READABLE)
 			continue;
-		if (!(prot & PMP_A))
+		if (reg->flags & SBI_DOMAIN_MEMREGION_WRITEABLE)
 			continue;
-		if (prot & (PMP_R | PMP_W | PMP_X))
+		if (reg->flags & SBI_DOMAIN_MEMREGION_EXECUTABLE)
 			continue;
 
-		fdt_resv_memory_update_node(fdt, addr, size, j, parent, false);
-		j++;
+		addr = reg->base;
+		size = 1UL << reg->order;
+		fdt_resv_memory_update_node(fdt, addr, size, i, parent,
+			(sbi_hart_pmp_count(scratch)) ? false : true);
+		i++;
 	}
 
 	return 0;
Signature · 1 line
2.25.1
author
#27 lore raw

PATCH 12/16 lib: utils: Update fdt_cpu_fixup() to use current domain

In reply to Anup Patel

The fdt_cpu_fixup() should disable a HART in DT if the HART is not assigned to the current HART domain. This patch updates fdt_cpu_fixup() accordingly.

---
 lib/utils/fdt/fdt_fixup.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
lib/utils/fdt/fdt_fixup.c +3 -5
index e9e1dff..436179a 100644
@@ -12,7 +12,6 @@
 #include <sbi/sbi_domain.h>
 #include <sbi/sbi_math.h>
 #include <sbi/sbi_hart.h>
-#include <sbi/sbi_platform.h>
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_string.h>
 #include <sbi_utils/fdt/fdt_fixup.h>
@@ -20,8 +19,7 @@
 
 void fdt_cpu_fixup(void *fdt)
 {
-	struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
-	const struct sbi_platform *plat = sbi_platform_ptr(scratch);
+	struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	int err, cpu_offset, cpus_offset, len;
 	const char *mmu_type;
 	u32 hartid;
@@ -41,12 +39,12 @@ void fdt_cpu_fixup(void *fdt)
 
 		/*
 		 * Disable a HART DT node if one of the following is true:
-		 * 1. The HART is marked invalid by platform support
+		 * 1. The HART is not assigned to the current domain
 		 * 2. MMU is not available for the HART
 		 */
 
 		mmu_type = fdt_getprop(fdt, cpu_offset, "mmu-type", &len);
-		if (sbi_platform_hart_invalid(plat, hartid) ||
+		if (!sbi_domain_assigned_hart(dom, hartid) ||
 		    !mmu_type || !len)
 			fdt_setprop_string(fdt, cpu_offset, "status",
 					   "disabled");
Signature · 1 line
2.25.1
author
#28 lore raw

PATCH 13/16 lib: sbi: Remove redundant sbi_hart_pmp_xyz() functions

In reply to Anup Patel

The sbi_hart_pmp_get() and sbi_hart_pmp_check_addr() functions are not used anymore hence this patch removes these functions.

---
 include/sbi/sbi_hart.h |  6 ------
 lib/sbi/sbi_hart.c     | 42 ------------------------------------------
 2 files changed, 48 deletions(-)
include/sbi/sbi_hart.h +0 -6
index 2efe44d..79d745a 100644
@@ -38,13 +38,7 @@ static inline ulong sbi_hart_expected_trap_addr(void)
 unsigned int sbi_hart_mhpm_count(struct sbi_scratch *scratch);
 void sbi_hart_delegation_dump(struct sbi_scratch *scratch);
 unsigned int sbi_hart_pmp_count(struct sbi_scratch *scratch);
-int sbi_hart_pmp_get(struct sbi_scratch *scratch, unsigned int n,
-		     unsigned long *prot_out, unsigned long *addr_out,
-		     unsigned long *size);
 void sbi_hart_pmp_dump(struct sbi_scratch *scratch);
-int  sbi_hart_pmp_check_addr(struct sbi_scratch *scratch,
-			     unsigned long daddr, unsigned long mode,
-			     unsigned long attr);
 int sbi_hart_pmp_configure(struct sbi_scratch *scratch);
 bool sbi_hart_has_feature(struct sbi_scratch *scratch, unsigned long feature);
 void sbi_hart_get_features_str(struct sbi_scratch *scratch,
lib/sbi/sbi_hart.c +0 -42

54 lines not on this page · raw message

Signature · 1 line
2.25.1
author
#29 lore raw

PATCH 14/16 lib: sbi: Configure PMP based on domain memory regions

In reply to Anup Patel

The PMP configuration on each HART should be only based on the memory regions of the assigned domain. This patch updates the sbi_hart_pmp_configure() function accordingly.

---
 lib/sbi/sbi_hart.c | 37 ++++++++++++++++++++++---------------
 1 file changed, 22 insertions(+), 15 deletions(-)
lib/sbi/sbi_hart.c +22 -15
index ea5d479..1871a1e 100644
@@ -13,6 +13,7 @@
 #include <sbi/riscv_fp.h>
 #include <sbi/sbi_bitops.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_domain.h>
 #include <sbi/sbi_csr_detect.h>
 #include <sbi/sbi_error.h>
 #include <sbi/sbi_hart.h>
@@ -184,24 +185,30 @@ void sbi_hart_pmp_dump(struct sbi_scratch *scratch)
 
 int sbi_hart_pmp_configure(struct sbi_scratch *scratch)
 {
-	u32 pmp_idx = 0;
-	unsigned long fw_start, fw_size_log2;
+	struct sbi_domain_memregion *reg;
+	struct sbi_domain *dom = sbi_domain_thishart_ptr();
+	unsigned int pmp_idx = 0, pmp_flags;
+	unsigned int pmp_count = sbi_hart_pmp_count(scratch);
 
-	if (!sbi_hart_pmp_count(scratch))
+	if (!pmp_count)
 		return 0;
 
-	/* Firmware PMP region to protect OpenSBI firmware */
-	fw_size_log2 = log2roundup(scratch->fw_size);
-	fw_start = scratch->fw_start & ~((1UL << fw_size_log2) - 1UL);
-	pmp_set(pmp_idx++, 0, fw_start, fw_size_log2);
-
-	/*
-	 * Default PMP region for allowing S-mode and U-mode access to
-	 * memory not covered by:
-	 * 1) Firmware PMP region
-	 * 2) Platform specific PMP regions
-	 */
-	pmp_set(pmp_idx++, PMP_R | PMP_W | PMP_X, 0, __riscv_xlen);
+	sbi_domain_for_each_memregion(dom, reg) {
+		if (pmp_count <= pmp_idx)
+			break;
+
+		pmp_flags = 0;
+		if (reg->flags & SBI_DOMAIN_MEMREGION_READABLE)
+			pmp_flags |= PMP_R;
+		if (reg->flags & SBI_DOMAIN_MEMREGION_WRITEABLE)
+			pmp_flags |= PMP_W;
+		if (reg->flags & SBI_DOMAIN_MEMREGION_EXECUTABLE)
+			pmp_flags |= PMP_X;
+		if (reg->flags & SBI_DOMAIN_MEMREGION_MMODE)
+			pmp_flags |= PMP_L;
+
+		pmp_set(pmp_idx++, pmp_flags, reg->base, reg->order);
+	}
 
 	return 0;
 }
Signature · 1 line
2.25.1
author
#30 lore raw

PATCH 15/16 lib: sbi: Display domain details in boot prints

In reply to Anup Patel

We extend boot prints to display details of each domain. In the process, we remove sbi_hart_pmp_dump() because it shows redundant information which domain details already show.

---
 include/sbi/sbi_domain.h |  6 +++
 include/sbi/sbi_hart.h   |  4 +-
 lib/sbi/sbi_domain.c     | 94 ++++++++++++++++++++++++++++++++++++++++
 lib/sbi/sbi_hart.c       | 44 +++++--------------
 lib/sbi/sbi_init.c       | 27 +++++++-----
 5 files changed, 127 insertions(+), 48 deletions(-)
include/sbi/sbi_domain.h +6 -0
index a3f174c..3b69c95 100644
@@ -129,6 +129,12 @@ bool sbi_domain_check_addr(const struct sbi_domain *dom,
 			   unsigned long addr, unsigned long mode, bool mmio,
 			   bool read, bool write, bool execute);
 
+/** Dump domain details on the console */
+void sbi_domain_dump(const struct sbi_domain *dom, const char *suffix);
+
+/** Dump all domain details on the console */
+void sbi_domain_dump_all(const char *suffix);
+
 /** Finalize domain tables and startup non-root domains */
 int sbi_domain_finalize(struct sbi_scratch *scratch, u32 cold_hartid);
 
include/sbi/sbi_hart.h +2 -2
index 79d745a..3ac1500 100644
@@ -36,9 +36,9 @@ static inline ulong sbi_hart_expected_trap_addr(void)
 }
 
 unsigned int sbi_hart_mhpm_count(struct sbi_scratch *scratch);
-void sbi_hart_delegation_dump(struct sbi_scratch *scratch);
+void sbi_hart_delegation_dump(struct sbi_scratch *scratch,
+			      const char *prefix, const char *suffix);
 unsigned int sbi_hart_pmp_count(struct sbi_scratch *scratch);
-void sbi_hart_pmp_dump(struct sbi_scratch *scratch);
 int sbi_hart_pmp_configure(struct sbi_scratch *scratch);
 bool sbi_hart_has_feature(struct sbi_scratch *scratch, unsigned long feature);
 void sbi_hart_get_features_str(struct sbi_scratch *scratch,
lib/sbi/sbi_domain.c +94 -0

106 lines not on this page · raw message

lib/sbi/sbi_hart.c +10 -34

63 lines not on this page · raw message

lib/sbi/sbi_init.c +15 -12
index 406cb3f..5151d36 100644
@@ -39,6 +39,7 @@ static void sbi_boot_prints(struct sbi_scratch *scratch, u32 hartid)
 {
 	int xlen;
 	char str[128];
+	const struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	const struct sbi_platform *plat = sbi_platform_ptr(scratch);
 
 #ifdef OPENSBI_VERSION_GIT
@@ -64,27 +65,29 @@ static void sbi_boot_prints(struct sbi_scratch *scratch, u32 hartid)
 	sbi_printf("Platform HART Count : %u\n",
 		   sbi_platform_hart_count(plat));
 
-	/* Boot HART details */
-	sbi_printf("Boot HART ID        : %u\n", hartid);
-	misa_string(xlen, str, sizeof(str));
-	sbi_printf("Boot HART ISA       : %s\n", str);
-	sbi_hart_get_features_str(scratch, str, sizeof(str));
-	sbi_printf("BOOT HART Features  : %s\n", str);
-	sbi_printf("BOOT HART PMP Count : %d\n", sbi_hart_pmp_count(scratch));
-	sbi_printf("BOOT HART MHPM Count: %d\n", sbi_hart_mhpm_count(scratch));
-
 	/* Firmware details */
 	sbi_printf("Firmware Base       : 0x%lx\n", scratch->fw_start);
 	sbi_printf("Firmware Size       : %d KB\n",
 		   (u32)(scratch->fw_size / 1024));
 
-	/* Generic details */
+	/* SBI details */
 	sbi_printf("Runtime SBI Version : %d.%d\n",
 		   sbi_ecall_version_major(), sbi_ecall_version_minor());
 	sbi_printf("\n");
 
-	sbi_hart_delegation_dump(scratch);
-	sbi_hart_pmp_dump(scratch);
+	/* Domain details */
+	sbi_domain_dump_all("");
+
+	/* Boot HART details */
+	sbi_printf("Boot HART ID        : %u\n", hartid);
+	sbi_printf("Boot HART Domain    : %s\n", dom->name);
+	misa_string(xlen, str, sizeof(str));
+	sbi_printf("Boot HART ISA       : %s\n", str);
+	sbi_hart_get_features_str(scratch, str, sizeof(str));
+	sbi_printf("Boot HART Features  : %s\n", str);
+	sbi_printf("Boot HART PMP Count : %d\n", sbi_hart_pmp_count(scratch));
+	sbi_printf("Boot HART MHPM Count: %d\n", sbi_hart_mhpm_count(scratch));
+	sbi_hart_delegation_dump(scratch, "Boot HART ", "   ");
 }
 
 static spinlock_t coldboot_lock = SPIN_LOCK_INITIALIZER;
Signature · 1 line
2.25.1
author
#31 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

In reply to Anup Patel

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105 +++++++++++++++++++++++++++++++++++++++++
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)
 create mode 100644 docs/domain_support.md
README.md +3 -0
index bd41ba3..03c02fb 100644
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of OpenSBI can be found under the
 * [Platform Documentation]: Documentation of the platforms currently supported.
 * [Firmware Documentation]: Documentation for the different types of firmware
   examples build supported by OpenSBI.
+* [Domain Support]: Documentation for the OpenSBI domain support which helps
+  users achieve system-level partitioning using OpenSBI.
 
 OpenSBI source code is also well documented. For source level documentation,
 doxygen style is used. Please refer to the [Doxygen manual] for details on this
@@ -278,6 +280,7 @@ make I=<install_directory> install_docs
 [Platform Support Guide]: docs/platform_guide.md
 [Platform Documentation]: docs/platform/platform.md
 [Firmware Documentation]: docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
 [Doxygen manual]: http://www.doxygen.nl/manual/index.html
 [Kendryte standalone SDK]: https://github.com/kendryte/kendryte-standalone-sdk
 [third party notices]: ThirdPartyNotices.md
docs/domain_support.md new file +105 -0

106 lines not on this page · raw message

docs/doxygen.cfg +1 -0
index dbf8ef9..82f31a7 100644
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md \
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
Signature · 1 line
2.25.1
#32 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 39 lines · Anup Patel

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105 +++++++++++++++++++++++++++++++++++++++++
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)
 create mode 100644 docs/domain_support.md
README.md +3 -0
index bd41ba3..03c02fb 100644
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of OpenSBI can be found under the
 * [Platform Documentation]: Documentation of the platforms currently supported.
 * [Firmware Documentation]: Documentation for the different types of firmware
   examples build supported by OpenSBI.
+* [Domain Support]: Documentation for the OpenSBI domain support which helps
+  users achieve system-level partitioning using OpenSBI.
 
 OpenSBI source code is also well documented. For source level documentation,
 doxygen style is used. Please refer to the [Doxygen manual] for details on this
@@ -278,6 +280,7 @@ make I=<install_directory> install_docs
 [Platform Support Guide]: docs/platform_guide.md
 [Platform Documentation]: docs/platform/platform.md
 [Firmware Documentation]: docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
 [Doxygen manual]: http://www.doxygen.nl/manual/index.html
 [Kendryte standalone SDK]: https://github.com/kendryte/kendryte-standalone-sdk
 [third party notices]: ThirdPartyNotices.md
docs/domain_support.md new file +0 -0
new file mode 100644
index 0000000..fad4f67

docs/domain_support.md line 8

Quoted 8 lines
docs/domain_support.md new file +8 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of underlying hardware
+having it's own memory regions (RAM and MMIO devices) and HARTs. The OpenSBI
+will try to achieve secure isolation between domains using RISC-V platform
+features such as PMP, IOPMP, etc.
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Quoted 36 lines
docs/domain_support.md new file +36 -0
+Important entities which help implement OpenSBI domain support are:
+
+* **struct sbi_domain_memregion** - Representation of a domain memory region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain assigned to it.
+The OpenSBI platform support is responsible for populating domains and
+providing HART id to domain mapping. The OpenSBI domain support will by
+default assign **the ROOT domain** to all HARTs of a RISC-V platform so
+it is not mandatory for the OpenSBI platform support to populate domains.
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct sbi_domain_memregion** in
+OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where **order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type (i.e.
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE, etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in OpenSBI and
+has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory region

docs/domain_support.md line 54

Quoted 10 lines
docs/domain_support.md new file +10 -0
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The domain
+  boot HART will be started at boot-time if boot HART is a possible and
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for this domain
+* **system_reset_allowed** - Is domain allowed to reset the system?
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 62

Quoted 8 lines
docs/domain_support.md new file +8 -0
+The memory regions represented by **regions** in **struct sbi_domain** have
+following additional constraints to align with RISC-V PMP requirements:
+
+* A memory region to protect OpenSBI firmware from S-mode and U-mode
+  should always be present
+* For two overlapping memory regions, one should be sub-region of another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Quoted 33 lines
docs/domain_support.md new file +33 -0
+* Memory access checks on overlapping address should prefer smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is assigned by
+default to all HARTs of a RISC-V platform. The OpenSBI domain support
+will hand-craft **the ROOT domain** very early at boot-time in the
+following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V platform
+  are assigned the ROOT domain which changes later based on OpenSBI
+  platform support
+* **possible_harts** - All valid HARTs of a RISC-V platform are possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode and U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing S-mode and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT domain
+* **next_addr** - Next booting stage address in coldboot HART scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART scratch space
+  is the next arg1 for the ROOT domain
+* **next_mode** - Next booting stage mode in coldboot HART scratch space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset the system
+
+Domain Effects
+--------------

docs/domain_support.md line 105

Quoted 10 lines
docs/domain_support.md new file +10 -0
+
+Few noteworthy effects of a system partitioned into domains are as follows:
+
+* At any point in time, a HART is running in exactly one OpenSBI domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory based on the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

Quoted 21 lines

\ No newline at end of file

docs/doxygen.cfg +1 -1
index dbf8ef9..82f31a7 100644
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md \
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
author
#33 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 55 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 8 lines · Anup Patel

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)
 create mode 100644 docs/domain_support.md
README.md +0 -0
index bd41ba3..03c02fb 100644
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms currently

supported.

Quoted 1 line

* [Firmware Documentation]: Documentation for the different types of

firmware

Quoted 10 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
 
 OpenSBI source code is also well documented. For source level

documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file mode

100644

Quoted 15 lines

docs/domain_support.md line 8

Quoted 12 lines

index 0000000..fad4f67

docs/domain_support.md new file +8 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of underlying
+hardware having it's own memory regions (RAM and MMIO devices) and
+HARTs. The OpenSBI will try to achieve secure isolation between
+domains using RISC-V platform features such as PMP, IOPMP, etc.
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 64 lines

docs/domain_support.md

Quoted 4 lines
docs/domain_support.md new file +3 -0
+Important entities which help implement OpenSBI domain support are:
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 5 lines
docs/domain_support.md new file +5 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain assigned

to it.

docs/domain_support.md line 20

Quoted 4 lines
docs/domain_support.md new file +4 -0
+The OpenSBI platform support is responsible for populating domains
+and providing HART id to domain mapping. The OpenSBI domain support
+will by default assign **the ROOT domain** to all HARTs of a RISC-V
+platform so it is not mandatory for the OpenSBI platform support to

populate domains.

Quoted 29 lines
docs/domain_support.md new file +29 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type (i.e.
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE, etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in OpenSBI
+and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain

docs/domain_support.md line 54

Quoted 10 lines
docs/domain_support.md new file +10 -0
+  boot HART will be started at boot-time if boot HART is a possible
+and
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for this
+domain
+* **system_reset_allowed** - Is domain allowed to reset the system?
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 61

Quoted 2 lines
docs/domain_support.md new file +2 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with RISC-V

PMP requirements:

docs/domain_support.md line 63

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 45 lines
Quoted 18 lines
docs/domain_support.md new file +17 -0
+* Memory access checks on overlapping address should prefer smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is assigned
+by default to all HARTs of a RISC-V platform. The OpenSBI domain
+support will hand-craft **the ROOT domain** very early at boot-time
+in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on OpenSBI
+  platform support

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode and
+U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing S-mode
+and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 5 lines
docs/domain_support.md new file +5 -0
+* **next_addr** - Next booting stage address in coldboot HART scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART scratch
+space
+  is the next arg1 for the ROOT domain

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **next_mode** - Next booting stage mode in coldboot HART scratch
+space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset the
+system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are as

follows:

Quoted 14 lines

docs/domain_support.md

Quoted 12 lines
docs/domain_support.md new file +12 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory based on
+the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory regions of domain assigned to the HART.

Is there anything missing ?

Quoted 18 lines
Quoted 18 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +1 -1
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md \
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

Quoted 10 lines
Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Regards,
Anup

#34 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 207 lines · Anup Patel
Quoted 56 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 8 lines · Anup Patel

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)
 create mode 100644 docs/domain_support.md
README.md +0 -0
index bd41ba3..03c02fb 100644
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms currently

supported.

Quoted 1 line

* [Firmware Documentation]: Documentation for the different types of

firmware

Quoted 10 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
 
 OpenSBI source code is also well documented. For source level

documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file mode

100644

Quoted 15 lines

docs/domain_support.md line 8

Quoted 12 lines

index 0000000..fad4f67

docs/domain_support.md new file +8 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of underlying
+hardware having it's own memory regions (RAM and MMIO devices) and
+HARTs. The OpenSBI will try to achieve secure isolation between
+domains using RISC-V platform features such as PMP, IOPMP, etc.
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 64 lines

docs/domain_support.md

Quoted 4 lines
docs/domain_support.md new file +3 -0
+Important entities which help implement OpenSBI domain support are:
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 5 lines
docs/domain_support.md new file +5 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain assigned

to it.

docs/domain_support.md line 20

Quoted 4 lines
docs/domain_support.md new file +4 -0
+The OpenSBI platform support is responsible for populating domains
+and providing HART id to domain mapping. The OpenSBI domain support
+will by default assign **the ROOT domain** to all HARTs of a RISC-V
+platform so it is not mandatory for the OpenSBI platform support to

populate domains.

Quoted 29 lines
docs/domain_support.md new file +29 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type (i.e.
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE, etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in OpenSBI
+and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain

docs/domain_support.md line 54

Quoted 10 lines
docs/domain_support.md new file +10 -0
+  boot HART will be started at boot-time if boot HART is a possible
+and
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for this
+domain
+* **system_reset_allowed** - Is domain allowed to reset the system?
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 61

Quoted 2 lines
docs/domain_support.md new file +2 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with RISC-V

PMP requirements:

docs/domain_support.md line 63

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 45 lines
Quoted 18 lines
docs/domain_support.md new file +17 -0
+* Memory access checks on overlapping address should prefer smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is assigned
+by default to all HARTs of a RISC-V platform. The OpenSBI domain
+support will hand-craft **the ROOT domain** very early at boot-time
+in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on OpenSBI
+  platform support

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode and
+U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing S-mode
+and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 5 lines
docs/domain_support.md new file +5 -0
+* **next_addr** - Next booting stage address in coldboot HART scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART scratch
+space
+  is the next arg1 for the ROOT domain

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **next_mode** - Next booting stage mode in coldboot HART scratch
+space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset the
+system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are as

follows:

Quoted 14 lines

docs/domain_support.md

Quoted 12 lines
docs/domain_support.md new file +12 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory based on
+the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

Quoted 6 lines

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory regions of domain assigned to the HART.

Is there anything missing ?

I meant to ask about the general effects of domain on PMP. I was not asking to follow up the previous sentence. Let me rephrase.

A child domain can run M-mode only payload where next_mode will be M-mode. Correct ?
In that case, can it modify PMP ? or what happens if a child domain next stage running in M-mode tries to modify some pmp configurations ?

Quoted 31 lines
Quoted 18 lines
Quoted 18 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +1 -1
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md \
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

Quoted 10 lines
Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Regards,
Anup

Signature · 2 lines
Regards,
Atish
author
#35 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 247 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 11:42
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 14 lines · Anup Patel
Quoted 14 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 29 lines · wrote:
Quoted 29 lines
Quoted 8 lines

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)  create mode 100644

docs/domain_support.md

README.md +0 -0
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms
currently

supported.

Quoted 2 lines

* [Firmware Documentation]: Documentation for the different types
of

firmware

Quoted 5 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

OpenSBI source code is also well documented. For source level
documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines
Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file

mode

Quoted 10 lines
Quoted 10 lines

100644

docs/domain_support.md line 5

Quoted 9 lines

index 0000000..fad4f67

docs/domain_support.md new file +5 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of
+underlying hardware having it's own memory regions (RAM and

MMIO

Quoted 2 lines
Quoted 2 lines

docs/domain_support.md line 7

Quoted 2 lines
docs/domain_support.md new file +2 -0
+devices) and HARTs. The OpenSBI will try to achieve secure
+isolation between domains using RISC-V platform features such as

PMP, IOPMP, etc.

Quoted 9 lines
Quoted 4 lines

docs/domain_support.md line 8

Quoted 1 line
docs/domain_support.md new file +1 -0
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 2 lines
Quoted 2 lines
docs/domain_support.md new file +1 -0
+Important entities which help implement OpenSBI domain support

are:

Quoted 16 lines
Quoted 16 lines

docs/domain_support.md line 11

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 6 lines
docs/domain_support.md new file +6 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain
+assigned

to it.

docs/domain_support.md line 22

Quoted 5 lines
docs/domain_support.md new file +5 -0
+The OpenSBI platform support is responsible for populating
+domains and providing HART id to domain mapping. The OpenSBI
+domain support will by default assign **the ROOT domain** to all
+HARTs of a RISC-V platform so it is not mandatory for the OpenSBI
+platform support to

populate domains.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
docs/domain_support.md new file +5 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
Quoted 8 lines

docs/domain_support.md

Quoted 8 lines
docs/domain_support.md new file +8 -0
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type

(i.e.

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
docs/domain_support.md new file +20 -0
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE,
+etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in
+OpenSBI and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory
+region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain
+  boot HART will be started at boot-time if boot HART is a
+possible and
Quoted 8 lines
Quoted 8 lines

docs/domain_support.md line 63

Quoted 8 lines
docs/domain_support.md new file +8 -0
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this
+domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for
+this domain
+* **system_reset_allowed** - Is domain allowed to reset the

system?

Quoted 69 lines
Quoted 11 lines

docs/domain_support.md line 64

Quoted 1 line
docs/domain_support.md new file +1 -0
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 67

Quoted 3 lines
docs/domain_support.md new file +3 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with
+RISC-V

PMP requirements:

docs/domain_support.md line 69

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 49 lines
Quoted 20 lines
docs/domain_support.md new file +19 -0
+* Memory access checks on overlapping address should prefer
+smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is
+assigned by default to all HARTs of a RISC-V platform. The
+OpenSBI domain support will hand-craft **the ROOT domain** very
+early at boot-time in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on
+OpenSBI
+  platform support

docs/domain_support.md line 103

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode
+and U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing
+S-mode and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 7 lines
docs/domain_support.md new file +7 -0
+* **next_addr** - Next booting stage address in coldboot HART
+scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART
+scratch space
+  is the next arg1 for the ROOT domain
+* **next_mode** - Next booting stage mode in coldboot HART

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+scratch space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset
+the system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are
+as

follows:

Quoted 10 lines
Quoted 10 lines

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory

based

Quoted 7 lines
Quoted 4 lines

docs/domain_support.md

Quoted 2 lines
docs/domain_support.md new file +2 -0
+on the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory

regions

Quoted 4 lines

of domain assigned to the HART.

Is there anything missing ?

Quoted 7 lines

I meant to ask about the general effects of domain on PMP. I was not asking to follow up the previous sentence. Let me rephrase.

A child domain can run M-mode only payload where next_mode will be M- mode. Correct ?
In that case, can it modify PMP ? or what happens if a child domain next stage running in M-mode tries to modify some pmp configurations ?

There is no parent child relationship between domains. It's much simpler now compared to initial ideas.

We just have a ROOT domain and all other domains are non-ROOT domains.

The M-mode software (OpenSBI) will ensure that PMP regions are configured for a HART based on the assigned domain.

Quoted 31 lines
Quoted 8 lines
Quoted 8 lines

docs/doxygen.cfg INPUT = @@SRC_DIR@@/README.md

Quoted 8 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +0 -0
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md

\

Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
docs/doxygen.cfg +1 -1
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

Quoted 4 lines

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

--
Regards,
Atish

Quoted 8 lines
Quoted 2 lines

Regards,
Anup

--
Regards,
Atish

Regards,
Anup

author
#36 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

In reply to Atish Patra
Quoted 247 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 11:42
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 14 lines · Anup Patel
Quoted 14 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 29 lines · wrote:
Quoted 29 lines
Quoted 8 lines

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)  create mode 100644

docs/domain_support.md

README.md +0 -0
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms
currently

supported.

Quoted 2 lines

* [Firmware Documentation]: Documentation for the different types
of

firmware

Quoted 5 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

OpenSBI source code is also well documented. For source level
documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines
Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file

mode

Quoted 10 lines
Quoted 10 lines

100644

docs/domain_support.md line 5

Quoted 9 lines

index 0000000..fad4f67

docs/domain_support.md new file +5 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of
+underlying hardware having it's own memory regions (RAM and

MMIO

Quoted 2 lines
Quoted 2 lines

docs/domain_support.md line 7

Quoted 2 lines
docs/domain_support.md new file +2 -0
+devices) and HARTs. The OpenSBI will try to achieve secure
+isolation between domains using RISC-V platform features such as

PMP, IOPMP, etc.

Quoted 9 lines
Quoted 4 lines

docs/domain_support.md line 8

Quoted 1 line
docs/domain_support.md new file +1 -0
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 2 lines
Quoted 2 lines
docs/domain_support.md new file +1 -0
+Important entities which help implement OpenSBI domain support

are:

Quoted 16 lines
Quoted 16 lines

docs/domain_support.md line 11

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 6 lines
docs/domain_support.md new file +6 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain
+assigned

to it.

docs/domain_support.md line 22

Quoted 5 lines
docs/domain_support.md new file +5 -0
+The OpenSBI platform support is responsible for populating
+domains and providing HART id to domain mapping. The OpenSBI
+domain support will by default assign **the ROOT domain** to all
+HARTs of a RISC-V platform so it is not mandatory for the OpenSBI
+platform support to

populate domains.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
docs/domain_support.md new file +5 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
Quoted 8 lines

docs/domain_support.md

Quoted 8 lines
docs/domain_support.md new file +8 -0
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type

(i.e.

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
docs/domain_support.md new file +20 -0
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE,
+etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in
+OpenSBI and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory
+region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain
+  boot HART will be started at boot-time if boot HART is a
+possible and
Quoted 8 lines
Quoted 8 lines

docs/domain_support.md line 63

Quoted 8 lines
docs/domain_support.md new file +8 -0
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this
+domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for
+this domain
+* **system_reset_allowed** - Is domain allowed to reset the

system?

Quoted 69 lines
Quoted 11 lines

docs/domain_support.md line 64

Quoted 1 line
docs/domain_support.md new file +1 -0
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 67

Quoted 3 lines
docs/domain_support.md new file +3 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with
+RISC-V

PMP requirements:

docs/domain_support.md line 69

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 49 lines
Quoted 20 lines
docs/domain_support.md new file +19 -0
+* Memory access checks on overlapping address should prefer
+smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is
+assigned by default to all HARTs of a RISC-V platform. The
+OpenSBI domain support will hand-craft **the ROOT domain** very
+early at boot-time in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on
+OpenSBI
+  platform support

docs/domain_support.md line 103

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode
+and U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing
+S-mode and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 7 lines
docs/domain_support.md new file +7 -0
+* **next_addr** - Next booting stage address in coldboot HART
+scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART
+scratch space
+  is the next arg1 for the ROOT domain
+* **next_mode** - Next booting stage mode in coldboot HART

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+scratch space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset
+the system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are
+as

follows:

Quoted 10 lines
Quoted 10 lines

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory

based

Quoted 7 lines
Quoted 4 lines

docs/domain_support.md

Quoted 2 lines
docs/domain_support.md new file +2 -0
+on the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory

regions

Quoted 4 lines

of domain assigned to the HART.

Is there anything missing ?

Quoted 7 lines

I meant to ask about the general effects of domain on PMP. I was not asking to follow up the previous sentence. Let me rephrase.

A child domain can run M-mode only payload where next_mode will be M- mode. Correct ?
In that case, can it modify PMP ? or what happens if a child domain next stage running in M-mode tries to modify some pmp configurations ?

I forgot to mention. Next stage running in M-mode can always overwrite any CSR programmed by OpenSBI so we cannot enforce restrictions on next stage when it runs in M-mode.

Regards,
Anup

Quoted 39 lines
Quoted 8 lines
Quoted 8 lines

docs/doxygen.cfg INPUT = @@SRC_DIR@@/README.md

Quoted 8 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +0 -0
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md

\

Quoted 18 lines
Quoted 18 lines
Quoted 15 lines
docs/doxygen.cfg +1 -1
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Quoted 6 lines
Quoted 3 lines

--
Regards,
Atish

Regards,
Anup

--
Regards,
Atish

#37 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 251 lines · Anup Patel
Quoted 251 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 11:42
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 14 lines · Anup Patel
Quoted 14 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 29 lines · wrote:
Quoted 29 lines
Quoted 8 lines

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)  create mode 100644

docs/domain_support.md

README.md +0 -0
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms
currently

supported.

Quoted 2 lines

* [Firmware Documentation]: Documentation for the different types
of

firmware

Quoted 5 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

OpenSBI source code is also well documented. For source level
documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines
Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file

mode

Quoted 10 lines
Quoted 10 lines

100644

docs/domain_support.md line 5

Quoted 9 lines

index 0000000..fad4f67

docs/domain_support.md new file +5 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of
+underlying hardware having it's own memory regions (RAM and

MMIO

Quoted 2 lines
Quoted 2 lines

docs/domain_support.md line 7

Quoted 2 lines
docs/domain_support.md new file +2 -0
+devices) and HARTs. The OpenSBI will try to achieve secure
+isolation between domains using RISC-V platform features such as

PMP, IOPMP, etc.

Quoted 9 lines
Quoted 4 lines

docs/domain_support.md line 8

Quoted 1 line
docs/domain_support.md new file +1 -0
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 2 lines
Quoted 2 lines
docs/domain_support.md new file +1 -0
+Important entities which help implement OpenSBI domain support

are:

Quoted 16 lines
Quoted 16 lines

docs/domain_support.md line 11

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 6 lines
docs/domain_support.md new file +6 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain
+assigned

to it.

docs/domain_support.md line 22

Quoted 5 lines
docs/domain_support.md new file +5 -0
+The OpenSBI platform support is responsible for populating
+domains and providing HART id to domain mapping. The OpenSBI
+domain support will by default assign **the ROOT domain** to all
+HARTs of a RISC-V platform so it is not mandatory for the OpenSBI
+platform support to

populate domains.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
docs/domain_support.md new file +5 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
Quoted 8 lines

docs/domain_support.md

Quoted 8 lines
docs/domain_support.md new file +8 -0
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type

(i.e.

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
docs/domain_support.md new file +20 -0
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE,
+etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in
+OpenSBI and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory
+region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain
+  boot HART will be started at boot-time if boot HART is a
+possible and
Quoted 8 lines
Quoted 8 lines

docs/domain_support.md line 63

Quoted 8 lines
docs/domain_support.md new file +8 -0
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this
+domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for
+this domain
+* **system_reset_allowed** - Is domain allowed to reset the

system?

Quoted 69 lines
Quoted 11 lines

docs/domain_support.md line 64

Quoted 1 line
docs/domain_support.md new file +1 -0
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 67

Quoted 3 lines
docs/domain_support.md new file +3 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with
+RISC-V

PMP requirements:

docs/domain_support.md line 69

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 49 lines
Quoted 20 lines
docs/domain_support.md new file +19 -0
+* Memory access checks on overlapping address should prefer
+smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is
+assigned by default to all HARTs of a RISC-V platform. The
+OpenSBI domain support will hand-craft **the ROOT domain** very
+early at boot-time in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on
+OpenSBI
+  platform support

docs/domain_support.md line 103

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode
+and U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing
+S-mode and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 7 lines
docs/domain_support.md new file +7 -0
+* **next_addr** - Next booting stage address in coldboot HART
+scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART
+scratch space
+  is the next arg1 for the ROOT domain
+* **next_mode** - Next booting stage mode in coldboot HART

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+scratch space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset
+the system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are
+as

follows:

Quoted 10 lines
Quoted 10 lines

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory

based

Quoted 7 lines
Quoted 4 lines

docs/domain_support.md

Quoted 2 lines
docs/domain_support.md new file +2 -0
+on the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory

regions

Quoted 4 lines

of domain assigned to the HART.

Is there anything missing ?

I meant to ask about the general effects of domain on PMP. I was not asking to follow up the previous sentence. Let me rephrase.

Quoted 9 lines
Quoted 4 lines

A child domain can run M-mode only payload where next_mode will be M- mode. Correct ?
In that case, can it modify PMP ? or what happens if a child domain next stage running in M-mode tries to modify some pmp configurations ?

I forgot to mention. Next stage running in M-mode can always overwrite any CSR programmed by OpenSBI so we cannot enforce restrictions on next stage when it runs in M-mode.

I think that's a security flaw that a "secure" domain running in M-mode can corrupt anything in OpenSBI. Is there a use case where you want to run your secure OS
on top of OpenSBI in M-mode ?

Otherwise, we can just disable this feature or at least enable it as a separate compile time option so that
production systems can't be compromised.

Quoted 42 lines

Regards,
Anup

Quoted 39 lines
Quoted 8 lines
Quoted 8 lines

docs/doxygen.cfg INPUT = @@SRC_DIR@@/README.md

Quoted 8 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +0 -0
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md

\

Quoted 18 lines
Quoted 18 lines
Quoted 15 lines
docs/doxygen.cfg +1 -1
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Quoted 6 lines
Quoted 3 lines

--
Regards,
Atish

Regards,
Anup

--
Regards,
Atish

Signature · 2 lines
Regards,
Atish
#38 lore raw

PATCH 16/16 docs: Add initial documentation for domain support

Quoted 263 lines · Atish Patra
Quoted 251 lines · Anup Patel
Quoted 251 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 11:42
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

Quoted 14 lines · Anup Patel
Quoted 14 lines

-----Original Message-----
From: Atish Patra <[email protected]>
Sent: 02 October 2020 04:32
To: Anup Patel <[email protected]>
Cc: Atish Patra <[email protected]>; Alistair Francis <[email protected]>; Anup Patel <[email protected]>; OpenSBI <[email protected]>
Subject: Re: [PATCH 16/16] docs: Add initial documentation for domain support

On Fri, Sep 25, 2020 at 4:30 AM Anup Patel <[email protected]>

Quoted 29 lines · wrote:
Quoted 29 lines
Quoted 8 lines

We add initial documentation for OpenSBI domain support to help RISC-V platform vendors achieve system-level partitioning.

---
 README.md              |   3 ++
 docs/domain_support.md | 105

+++++++++++++++++++++++++++++++++++++++++

Quoted 11 lines
 docs/doxygen.cfg       |   1 +
 3 files changed, 109 insertions(+)  create mode 100644

docs/domain_support.md

README.md +0 -0
@@ -225,6 +225,8 @@ Detailed documentation of various aspects of

OpenSBI can be found under the
* [Platform Documentation]: Documentation of the platforms
currently

supported.

Quoted 2 lines

* [Firmware Documentation]: Documentation for the different types
of

firmware

Quoted 5 lines

examples build supported by OpenSBI.

README.md +3 -0
+* [Domain Support]: Documentation for the OpenSBI domain support
+which helps
+  users achieve system-level partitioning using OpenSBI.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

OpenSBI source code is also well documented. For source level
documentation, doxygen style is used. Please refer to the [Doxygen manual] for details on this @@ -278,6 +280,7 @@ make I=<install_directory> install_docs [Platform Support Guide]:
docs/platform_guide.md [Platform Documentation]:

Quoted 8 lines
Quoted 8 lines

docs/platform/platform.md [Firmware Documentation]:
docs/firmware/fw.md
+[Domain Support]: docs/domain_support.md
[Doxygen manual]: http://www.doxygen.nl/manual/index.html
[Kendryte standalone SDK]:
https://github.com/kendryte/kendryte-standalone-sdk
[third party notices]: ThirdPartyNotices.md diff --git
a/docs/domain_support.md b/docs/domain_support.md new file

mode

Quoted 10 lines
Quoted 10 lines

100644

docs/domain_support.md line 5

Quoted 9 lines

index 0000000..fad4f67

docs/domain_support.md new file +5 -0
@@ -0,0 +1,105 @@
+OpenSBI Domain Support
+======================
+
+An OpenSBI domain is a system-level partition (subset) of
+underlying hardware having it's own memory regions (RAM and

MMIO

Quoted 2 lines
Quoted 2 lines

docs/domain_support.md line 7

Quoted 2 lines
docs/domain_support.md new file +2 -0
+devices) and HARTs. The OpenSBI will try to achieve secure
+isolation between domains using RISC-V platform features such as

PMP, IOPMP, etc.

Quoted 9 lines
Quoted 4 lines

docs/domain_support.md line 8

Quoted 1 line
docs/domain_support.md new file +1 -0
+

May be specify a few other options such as ePMP, SiFive shield as well as you mentioned in the patch commit.

Okay, will add.

Quoted 2 lines
Quoted 2 lines
docs/domain_support.md new file +1 -0
+Important entities which help implement OpenSBI domain support

are:

Quoted 16 lines
Quoted 16 lines

docs/domain_support.md line 11

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* **struct sbi_domain_memregion** - Representation of a domain

memory

docs/domain_support.md

Quoted 6 lines
docs/domain_support.md new file +6 -0
+region
+* **struct sbi_hartmask** - Representation of domain HART set
+* **struct sbi_domain** - Representation of a domain instance
+
+Each HART of a RISC-V platform must have an OpenSBI domain
+assigned

to it.

docs/domain_support.md line 22

Quoted 5 lines
docs/domain_support.md new file +5 -0
+The OpenSBI platform support is responsible for populating
+domains and providing HART id to domain mapping. The OpenSBI
+domain support will by default assign **the ROOT domain** to all
+HARTs of a RISC-V platform so it is not mandatory for the OpenSBI
+platform support to

populate domains.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
docs/domain_support.md new file +5 -0
+
+Domain Memory Region
+--------------------
+
+A domain memory region is represented by **struct
Quoted 8 lines

docs/domain_support.md

Quoted 8 lines
docs/domain_support.md new file +8 -0
+sbi_domain_memregion** in OpenSBI and has following details:
+
+* **order** - The size of a memory region is **2 ^ order** where
+**order**
+  must be **3 <= order <= __riscv_xlen**
+* **base** - The base address of a memory region is **2 ^ order**
+  aligned start address
+* **flags** - The flags of a memory region represent memory type

(i.e.

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
docs/domain_support.md new file +20 -0
+  RAM or MMIO) and allowed accesses (i.e. READ, WRITE, EXECUTE,
+etc)
+
+Domain Instance
+---------------
+
+A domain instance is represented by **struct sbi_domain** in
+OpenSBI and has following details:
+
+* **index** - Logical index of this domain
+* **name** - Name of this domain
+* **assigned_harts** - HARTs assigned to this domain
+* **possible_harts** - HARTs possible in this domain
+* **regions** - Array of memory regions terminated by a memory
+region
+  with order zero
+* **boot_hartid** - HART id of the HART booting this domain. The
+domain
+  boot HART will be started at boot-time if boot HART is a
+possible and
Quoted 8 lines
Quoted 8 lines

docs/domain_support.md line 63

Quoted 8 lines
docs/domain_support.md new file +8 -0
+  assigned for this domain.
+* **next_addr** - Address of the next booting stage for this
+domain
+* **next_arg1** - Arg1 (or 'a1' register) of the next booting stage for
+   this domain
+* **next_mode** - Priviledge mode of the next booting stage for
+this domain
+* **system_reset_allowed** - Is domain allowed to reset the

system?

Quoted 69 lines
Quoted 11 lines

docs/domain_support.md line 64

Quoted 1 line
docs/domain_support.md new file +1 -0
+

Will it be better to keep a bitwise flags for such features ? Does a domain allow modification of PMP regions ?

docs/domain_support.md line 67

Quoted 3 lines
docs/domain_support.md new file +3 -0
+The memory regions represented by **regions** in **struct
+sbi_domain** have following additional constraints to align with
+RISC-V

PMP requirements:

docs/domain_support.md line 69

Quoted 2 lines
docs/domain_support.md new file +2 -0
+
+* A memory region to protect OpenSBI firmware from S-mode and U-

mode

Quoted 6 lines

docs/domain_support.md line 62

Quoted 5 lines
docs/domain_support.md new file +5 -0
+  should always be present
+* For two overlapping memory regions, one should be sub-region of
+another
+* Two overlapping memory regions should not be of same size
+* Two overlapping memory regions canot have same flags

/canot/cannot/gc

Okay, will update.

Quoted 49 lines
Quoted 20 lines
docs/domain_support.md new file +19 -0
+* Memory access checks on overlapping address should prefer
+smallest
+  overlapping memory region flags.
+
+ROOT Domain
+-----------
+
+**The ROOT domain** is the default OpenSBI domain which is
+assigned by default to all HARTs of a RISC-V platform. The
+OpenSBI domain support will hand-craft **the ROOT domain** very
+early at boot-time in the following manner:
+
+* **index** - Logical index of the ROOT domain is always zero
+* **name** - Name of the ROOT domain is "root"
+* **assigned_harts** - At boot-time all valid HARTs of a RISC-V
+platform
+  are assigned the ROOT domain which changes later based on
+OpenSBI
+  platform support

docs/domain_support.md line 103

Quoted 10 lines
docs/domain_support.md new file +10 -0
+* **possible_harts** - All valid HARTs of a RISC-V platform are
+possible
+  HARTs of the ROOT domain
+* **regions** - Two memory regions available to the ROOT domain:
+  **A)** A memory region to protect OpenSBI firmware from S-mode
+and U-mode
+  **B)** A memory region of **order=__riscv_xlen** allowing
+S-mode and
+  U-mode access to full memory address space
+* **boot_hartid** - Coldboot HART is the HART booting the ROOT

domain

Quoted 7 lines
docs/domain_support.md new file +7 -0
+* **next_addr** - Next booting stage address in coldboot HART
+scratch
+  space is the next address for the ROOT domain
+* **next_arg1** - Next booting stage arg1 in coldboot HART
+scratch space
+  is the next arg1 for the ROOT domain
+* **next_mode** - Next booting stage mode in coldboot HART

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+scratch space
+  is the next mode for the ROOT domain
+* **system_reset_allowed** - The ROOT domain is allowed to reset
+the system
+
+Domain Effects
+--------------
+
+Few noteworthy effects of a system partitioned into domains are
+as

follows:

Quoted 10 lines
Quoted 10 lines

docs/domain_support.md

Quoted 10 lines
docs/domain_support.md new file +10 -0
+
+* At any point in time, a HART is running in exactly one OpenSBI
+domain context
+* The SBI IPI and RFENCE calls from HART A are restricted to the
+HARTs in
+  domain assigned to HART A
+* The SBI HSM calls which try to change/read state of HART B from
+HART A will
+  only work if both HART A and HART B are assigned same domain
+* A HART running in S-mode or U-mode can only access memory

based

Quoted 7 lines
Quoted 4 lines

docs/domain_support.md

Quoted 2 lines
docs/domain_support.md new file +2 -0
+on the
+  memory regions of the domain assigned to the HART

How about effects on PMP ?

PMP is only accessible to M-mode so S-mode and U-mode can only access memory based on PMP configuration which is based on the memory

regions

Quoted 4 lines

of domain assigned to the HART.

Is there anything missing ?

I meant to ask about the general effects of domain on PMP. I was not asking to follow up the previous sentence. Let me rephrase.

Quoted 9 lines
Quoted 4 lines

A child domain can run M-mode only payload where next_mode will be M- mode. Correct ?
In that case, can it modify PMP ? or what happens if a child domain next stage running in M-mode tries to modify some pmp configurations ?

I forgot to mention. Next stage running in M-mode can always overwrite any CSR programmed by OpenSBI so we cannot enforce restrictions on next stage when it runs in M-mode.

Quoted 4 lines

I think that's a security flaw that a "secure" domain running in M-mode can corrupt anything in OpenSBI. Is there a use case where you want to run your secure OS
on top of OpenSBI in M-mode ?

Let's remove the support for the next stage starting in M-mode. This means next stage either starts in S-mode or U-mode. On platforms having Ecore (i.e. core having M-mode and U-mode), we can start the next stage directly in U-mode.

Quoted 4 lines

Otherwise, we can just disable this feature or at least enable it as a separate compile time option so that
production systems can't be compromised.

We can extend the OpenSBI domains in future, if we figure-out a clean way to start the next stage in M-mode. Until then let's not take any risk.

Quoted 46 lines
Quoted 43 lines

Regards,
Anup

Quoted 39 lines
Quoted 8 lines
Quoted 8 lines

docs/doxygen.cfg INPUT = @@SRC_DIR@@/README.md

Quoted 8 lines

\ No newline at end of file

docs/doxygen.cfg index renamed from docs/doxygen.cfg +0 -0

dbf8ef9..82f31a7 100644

docs/doxygen.cfg +0 -0
@@ -795,6 +795,7 @@ INPUT                  = @@SRC_DIR@@/README.md

\

Quoted 18 lines
Quoted 18 lines
Quoted 15 lines
docs/doxygen.cfg +1 -1
                          @@SRC_DIR@@/docs/platform_guide.md \
                          @@SRC_DIR@@/docs/platform_requirements.md \
                          @@SRC_DIR@@/docs/library_usage.md \
+                         @@SRC_DIR@@/docs/domain_support.md \
                          @@SRC_DIR@@/docs/firmware \
                          @@SRC_DIR@@/docs/platform \
                          @@SRC_DIR@@/include \
--

2.25.1

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Quoted 6 lines
Quoted 3 lines

--
Regards,
Atish

Regards,
Anup

--
Regards,
Atish

Quoted 3 lines

--
Regards,
Atish

Regards,
Anup