Initial ESWIN/EIC7700 support
v2 cover Initial ESWIN/EIC7700 support
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree as FW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.
The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to lib/ and firmware/ to make it possible:
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- Introduce FIRMWARE_PACKED_RXRW for disabling power-of-2 RW split
The defconfig of EIC770X is separated out, because it has to require CONFIG_FIRMWARE_PACKED_RXRW to not run out of PMP entries. It'll change the layout of firmware sections to not power-of-2 align the RW sections, so do not by default enable this for other platforms.
---
Changes in v2:
- Major enhancement of PMP consolidation logic. Also fixed a Linux Panic bug due to the mismatch between PMP settings and reserved memory regions passed to Linux via FDT.
- Also protects the OpenSBI firmware in uncached memory portion of address space.
- More detailed documentation on EIC770X/P550
---
Bo Gan (5):
lib: sbi: allow platform to override PMP configuration
lib: sbi: Add __pmp_set_tor for setting TOR regions
firmware: add CONFIG_FIRMWARE_PACKED_RXRW
include: sbi: Add helpers for sbi_domain_memregion
platform: generic: eswin: add EIC7700
firmware/Kconfig | 11 + firmware/fw_base.ldS | 14 +- include/sbi/riscv_asm.h | 7 + include/sbi/sbi_domain.h | 23 ++ include/sbi/sbi_hart.h | 9 + include/sbi/sbi_platform.h | 53 +++ lib/sbi/riscv_asm.c | 75 ++++- lib/sbi/sbi_domain.c | 11 + lib/sbi/sbi_domain_context.c | 11 +- lib/sbi/sbi_hart.c | 97 ++++-- platform/generic/Kconfig | 6 + platform/generic/configs/eic770x_defconfig | 28 ++ platform/generic/eswin/Kconfig | 29 ++ platform/generic/eswin/eic770x.c | 361 +++++++++++++++++++++ platform/generic/eswin/objects.mk | 11 + platform/generic/include/eswin/eic770x.h | 66 ++++ 16 files changed, 752 insertions(+), 60 deletions(-) create mode 100644 platform/generic/configs/eic770x_defconfig create mode 100644 platform/generic/eswin/Kconfig create mode 100644 platform/generic/eswin/eic770x.c create mode 100644 platform/generic/eswin/objects.mk create mode 100644 platform/generic/include/eswin/eic770x.h
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
PATCH v2 1/5 lib: sbi: allow platform to override PMP (un)configuration
Platform sometimes wants to override the entire PMP (un)config phase, not just performing additional work for each individual PMP entry. This allows platforms to insert SoC/core specific PMP entries in a way that works together with the existing ones set by lib/ code, not conflicting. platform can also choose to merge or skip memory regions in a reasonable way in case there's a shortage of PMP entries.
In addition, logic in sbi_hart_oldpmp_configure is abstracted out as sbi_hart_oldpmp_set, and made public, so platform with traditional PMP can leverage it in its own pmp_configure
--- include/sbi/sbi_hart.h | 9 ++++ include/sbi/sbi_platform.h | 53 ++++++++++++++++++++ lib/sbi/sbi_domain_context.c | 11 +--- lib/sbi/sbi_hart.c | 97 ++++++++++++++++++++++++------------ 4 files changed, 129 insertions(+), 41 deletions(-)
include/sbi/sbi_hart.h +9 -0
@@ -133,6 +133,8 @@ struct sbi_hart_features { };struct sbi_scratch ; +struct sbi_domain ; +struct sbi_domain_memregion ;int sbi_hart_reinit (struct sbi_scratch *scratch);int sbi_hart_init (struct sbi_scratch *scratch,bool cold_boot);
@@ -147,7 +149,14 @@ unsigned int sbi_hart_pmp_log2gran(struct sbi_scratch *scratch);unsigned int sbi_hart_pmp_addrbits (struct sbi_scratch *scratch);unsigned int sbi_hart_mhpm_bits (struct sbi_scratch *scratch);bool sbi_hart_smepmp_is_fw_region (unsigned int pmp_idx); +void sbi_hart_pmp_unconfigure (struct sbi_scratch *scratch);int sbi_hart_pmp_configure (struct sbi_scratch *scratch); +void sbi_hart_oldpmp_set (struct sbi_scratch *scratch, +struct sbi_domain *dom, +struct sbi_domain_memregion *reg, +unsigned int pmp_idx, +unsigned int pmp_log2gran, +unsigned long pmp_addr_max);int sbi_hart_map_saddr (unsigned long base,unsigned long size);int sbi_hart_unmap_saddr (void );int sbi_hart_priv_version (struct sbi_scratch *scratch);
include/sbi/sbi_platform.h +53 -0
65 lines not on this page · raw message
lib/sbi/sbi_domain_context.c +1 -10
@@ -102,7 +102,6 @@ static int switch_to_next_domain_context(struct hart_context *ctx,struct sbi_trap_context *trap_ctx;struct sbi_domain *current_dom, *target_dom;struct sbi_scratch *scratch =sbi_scratch_thishart_ptr (); -unsigned int pmp_count =sbi_hart_pmp_count (scratch);if (!ctx || !dom_ctx || ctx == dom_ctx)return SBI_EINVAL ;
@@ -120,15 +119,7 @@ static int switch_to_next_domain_context(struct hart_context *ctx,sbi_hartmask_set_hartindex (hartindex, &target_dom->assigned_harts);spin_unlock (&target_dom->assigned_harts_lock); -/* Reconfigure PMP settings for the new domain */ -for (int i =0 ; i < pmp_count; i++) { -/* Don't revoke firmware access permissions */ -if (sbi_hart_smepmp_is_fw_region (i)) -continue ; - -sbi_platform_pmp_disable (sbi_platform_thishart_ptr (), i); -pmp_disable (i); - } +sbi_hart_pmp_unconfigure (scratch);sbi_hart_pmp_configure (scratch);/* Save current CSR context and restore target domain's CSR context */
lib/sbi/sbi_hart.c +66 -31
132 lines not on this page · raw message
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
PATCH v2 2/5 lib: sbi: Add __pmp_set_tor for setting TOR regions
In reply to Bo GanTOR can be utilized to cover memory regions that are not aligned with their sizes. Given that the address matching is formed bt 2 consecutive pmpaddr, i.e., pmpaddr(i-1) and pmpaddr(i), TOR should not be used generically to avoid pmpaddr conflict with other NA4/NAPOT regions.
Only use them in platform PMP (un)configure override code where the caller can ensure the index and order of every pmp region especially when there's a mixture of TOR/NA4/NAPOT. Don't use TOR in lib/ code. For lib/ code, maintain the 1:1 PMP entry <-> memory region correspondence.
--- include/sbi/riscv_asm.h | 7 ++++ lib/sbi/riscv_asm.c | 75 +++++++++++++++++++++++++++++++---------- 2 files changed, 65 insertions(+), 17 deletions(-)
include/sbi/riscv_asm.h +7 -0
@@ -215,6 +215,13 @@ int pmp_disable(unsigned int n);/* Check if the matching field is set */ int is_pmp_entry_mapped (unsigned long entry); +/** +* Top of range (TOR) matching mode. pmpaddr(n-1) will also be changed. +* Use it *very* carefully. +*/ +int __pmp_set_tor (unsigned int n,unsigned long prot,unsigned long addr, +unsigned long size); +int pmp_set (unsigned int n,unsigned long prot,unsigned long addr,unsigned long log2len);
lib/sbi/riscv_asm.c +58 -17
109 lines not on this page · raw message
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
PATCH v2 3/5 firmware: add CONFIG_FIRMWARE_PACKED_RXRW
In reply to Bo GanBy default we power-of-2 align the rw sections of firmware. Provide an option to disable this behavior. Platforms w/o Smepmp won't be able to enforce RX/RW in M mode, so this 2^n align only perhaps saves some memory by covering OpenSBI with 2 finer grained PMP entries, instead of 1. For platforms that are really in short of PMP, consider enabling this knob.
--- firmware/Kconfig | 11 +++++++++++ firmware/fw_base.ldS | 14 ++++++++++++-- lib/sbi/sbi_domain.c | 11 +++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-)
firmware/Kconfig +11 -0
@@ -25,4 +25,15 @@ config STACK_PROTECTOR_ALL Turn on the "stack-protector" with "-fstack-protector-all" option. Like -fstack-protector except that all functions are protected. +config FIRMWARE_PACKED_RXRW + bool "Do not power-of-2 align the RW portion" + default n + help + By default we align the beginning of read-write data sections to + 2^n. This facilitates the setting of NAPOT PMP entries to cover + text, rodata, data... sections with proper permissions. For those + platforms that're in short of PMP entries, and not using Smepmp, + they may choose to disable this alignment and the entire firmware + can be covered by a single PMP entry. + endmenu
firmware/fw_base.ldS +12 -2
@@ -56,12 +56,22 @@ /* End of the read-only data sections */ +#ifdef CONFIG_FIRMWARE_PACKED_RXRW + . = ALIGN(0x1000); /* Ensure next section is page aligned */ +#else /* - * PMP regions must be to be power-of-2. RX/RW will have separate - * regions, so ensure that the split is power-of-2. + * Align the start of RW sections to power-of-2, so READ/EXEC + * and READ/WRITE sections can be covered by different PMP + * entries. Platforms with Smepmp can utilize it to enfore RX/RW + * permission on different portions of the firmware. For platforms + * using traditional PMPs, there's no enforcement of permissions + * in M mode, so NAPOT aligning the RW sections has no security + * benefits other than perhaps saving some memory for SU mode to + * use. It does require 2 PMP entries to cover OpenSBI, not 1. */ . = ALIGN(1 << LOG2CEIL((SIZEOF(.rodata) + SIZEOF(.text) + SIZEOF(.dynsym) + SIZEOF(.rela.dyn)))); +#endif PROVIDE(_fw_rw_start = .);
lib/sbi/sbi_domain.c +11 -0
@@ -865,6 +865,7 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)SBI_INIT_LIST_HEAD (&domain_list); +#ifndef CONFIG_FIRMWARE_PACKED_RXRW if (scratch->fw_rw_offset ==0 || (scratch->fw_rw_offset & (scratch->fw_rw_offset -1 )) !=0 ) { sbi_printf("%s: fw_rw_offset is not a power of 2 (0x%lx)\n",
@@ -877,6 +878,7 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid) __func__);return SBI_EINVAL ; } +#endif domain_hart_ptr_offset = sbi_scratch_alloc_type_offset(void *);if (!domain_hart_ptr_offset)
@@ -904,6 +906,14 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid) root.possible_harts = root_hmask;/* Root domain firmware memory region */ +#ifdef CONFIG_FIRMWARE_PACKED_RXRW +sbi_domain_memregion_init (scratch->fw_start, scratch->fw_size, + (SBI_DOMAIN_MEMREGION_M_READABLE | +SBI_DOMAIN_MEMREGION_M_WRITABLE | +SBI_DOMAIN_MEMREGION_M_EXECUTABLE | +SBI_DOMAIN_MEMREGION_FW ), + &root_memregs[root_memregs_count++]); +#else sbi_domain_memregion_init(scratch->fw_start, scratch->fw_rw_offset, (SBI_DOMAIN_MEMREGION_M_READABLE | SBI_DOMAIN_MEMREGION_M_EXECUTABLE |
@@ -916,6 +926,7 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)SBI_DOMAIN_MEMREGION_M_WRITABLE | SBI_DOMAIN_MEMREGION_FW), &root_memregs[root_memregs_count++]); +#endif root.fw_region_inited = true;
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
PATCH v2 4/5 include: sbi: Add helpers for sbi_domain_memregion
In reply to Bo GanNew helpers:
- sbi_domain_memregion_disjoint:
Check if two memregion are disjoint
- sbi_domain_for_each_memregion_idx:
Iterate domain memregion with index
--- include/sbi/sbi_domain.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
include/sbi/sbi_domain.h +23 -0
@@ -164,6 +164,26 @@ struct sbi_domain_memregion {unsigned long flags; }; +/** +* Check if two memregion A/B are disjoint. +* @param a memregion A +* @param b memregion B +* @return true if disjoint +*/ +static inline bool sbi_domain_memregion_disjoint ( +struct sbi_domain_memregion * a, +struct sbi_domain_memregion * b) +{ +/* Bail out early to avoid overflow */ +if (a->order >= __riscv_xlen || b->order >= __riscv_xlen) +return false; + +if (a->base < b->base) +return b->base - a->base >= (1UL << a->order); +else +return a->base - b->base >= (1UL << b->order); +} +/** Representation of OpenSBI domain */ struct sbi_domain {/** Node in linked list of domains */
@@ -222,6 +242,9 @@ extern struct sbi_dlist domain_list;#define sbi_domain_for_each_memregion (__d, __r) \ for ((__r) = (__d)->regions; (__r)->order; (__r)++) +#define sbi_domain_for_each_memregion_idx (__d, __r, __i) \ + for ((__r) = (__d)->regions, (__i) = 0; (__r)->order; (__r)++, (__i)++) +/** * Check whether given HART is assigned to specified domain * @param dom pointer to domain
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
PATCH v2 5/5 platform: generic: eswin: add EIC7700
In reply to Bo GanInitial platform support for ESWIN Computing EIC7700 based on public SoC datasheet[1] and tested on HiFive Premier P550. Vendor U-boot/Linux boots fine, and I've tested Geekbench 6.5.0 Preview and got scores on par with the vendor OpenSBI. System shutdown is not implemented due to the lack of public doc regarding the UART communication protocol between the SoC and the onboard BMC, which controls ATX power on HiFive P550. [2]
The files and functions are intentionally named as eic770x in many places for future enhancements to support the 2 die version of the same SoC, namely EIC7702, seen on DC-ROMA AI PC FML13V03 [3]. This patch set only deals with the single die version, and the assumption is we can be either die with id=0 or id=1, but there's only a single die in the system, or we are only using a single die out of 2. However, the way the SoC handles 2- die greatly affects how we configure it in a 1-die setup. EIC770X address map has die 0/1 memory regions interleaved (see comments in eic770x.c). If only 1 die is connected or active, it creates holes in the address map for those regions corresponding to the remote die. Although not mentioned in the datasheet, experiment shows speculative or HW prefetch can trigger bus errors when touching those "holes". We need to use PMP to block the access to those "holes", necessitating very tricky PMP configuration.
To make matters worse, EIC770X doesn't have cache coherent DMA, and due to the fact that the P550 core lacks Svpbmt, the SoC maps main memory twice as different regions in address map, so it can bypass cache and fetch data directly from memory. In address space, we have two memory regions, one as cached, the other as uncached. Thus, we also need an extra PMP entry to protect OpenSBI blob from the uncached window. To do this, we require FIRMWARE_PACKED_RXRW, otherwise, we'll run out of PMP entries. Currently we barely made it to cover everything with the total 8 avilable PMP:
For root domain harts:
PMP[0] NAPOT: Protect OpenSBI in cached memory window PMP[1] NAPOT: Protect OpenSBI in uncached memory window PMP[2] NAPOT: Protect CLINT
PMP[3] NAPOT: Enable cached memory (die 1 only) PMP[4] NAPOT: Block P550 internal on remote die PMP[5-6] TOR: Block holes in memory port on remote die PMP[7] NAPOT: Allow everything
EIC770X also have several feature disable/enable CSRs accessible in M mode. By default many core features such as speculation and HW prefetch are disabled, and M mode software is responsible of enabling. Hence, introduce 4 new build time tunable parameters to Kconfig, which reflects the values get updated to those CSRs:
- ESWIN_EIC770X_FEAT0_CFG
- ESWIN_EIC770X_FEAT1_CFG
- ESWIN_EIC770X_L1_HWPF_CFG
- ESWIN_EIC770X_L2_HWPF_CFG
The default values are somewhat optimal for generic workloads. They are dumped when running SiFive's vendor OpenSBI, and in addition, with my own tuning to address the perf regression reported by drmpeg [4]
To build the firmware+u-boot blob, Use the following, and docs [5] for testing it with UART boot without flashing:
make PLATFORM_DEFCONFIG=eic770x_defconfig \
FW_TEXT_START=0x80000000 \
FW_PAYLOAD_OFFSET=0x200000 \
FW_PAYLOAD_PATH=u-boot-nodtb.bin \
FW_PAYLOAD_FDT_ADDR=0xf8000000 \
FW_FDT_PATH=u-boot.dtb[1] https://github.com/eswincomputing/EIC7700X-SoC-Technical-Reference-Manual
[2] https://www.sifive.com/boards/hifive-premier-p550#documentation
[3] https://github.com/geerlingguy/sbc-reviews/issues/82
[4] https://forums.sifive.com/t/low-1-core-stream-bandwidth/7274/15
[5] https://github.com/ganboing/EIC770x-Docs/blob/main/p550/bootchain/UART-Boot.md
--- platform/generic/Kconfig | 6 + platform/generic/configs/eic770x_defconfig | 28 ++ platform/generic/eswin/Kconfig | 29 ++ platform/generic/eswin/eic770x.c | 361 +++++++++++++++++++++ platform/generic/eswin/objects.mk | 11 + platform/generic/include/eswin/eic770x.h | 66 ++++ 6 files changed, 501 insertions(+) create mode 100644 platform/generic/configs/eic770x_defconfig create mode 100644 platform/generic/eswin/Kconfig create mode 100644 platform/generic/eswin/eic770x.c create mode 100644 platform/generic/eswin/objects.mk create mode 100644 platform/generic/include/eswin/eic770x.h
platform/generic/Kconfig +6 -0
@@ -88,7 +88,13 @@ config PLATFORM_SPACEMIT_K1 select FDT_HSM_SPACEMIT default n +config PLATFORM_ESWIN_EIC770X + bool "ESWIN EIC770X support" + depends on FIRMWARE_PACKED_RXRW + default n + source "$(OPENSBI_SRC_DIR)/platform/generic/andes/Kconfig" +source "$(OPENSBI_SRC_DIR)/platform/generic/eswin/Kconfig" source "$(OPENSBI_SRC_DIR)/platform/generic/thead/Kconfig" endif
platform/generic/configs/eic770x_defconfig new file +28 -0
@@ -0,0 +1,28 @@ +CONFIG_PLATFORM_ESWIN_EIC770X=y +CONFIG_FIRMWARE_PACKED_RXRW=y +CONFIG_FDT_CACHE=y +CONFIG_FDT_CACHE_SIFIVE_CCACHE=y +CONFIG_FDT_CPPC=y +CONFIG_FDT_CPPC_RPMI=y +CONFIG_FDT_HSM=y +CONFIG_FDT_HSM_RPMI=y +CONFIG_FDT_IPI=y +CONFIG_FDT_IPI_MSWI=y +CONFIG_FDT_IRQCHIP=y +CONFIG_FDT_IRQCHIP_PLIC=y +CONFIG_FDT_MAILBOX=y +CONFIG_RPMI_MAILBOX=y +CONFIG_FDT_MAILBOX_RPMI_SHMEM=y +CONFIG_FDT_SERIAL=y +CONFIG_FDT_SERIAL_UART8250=y +CONFIG_FDT_SUSPEND=y +CONFIG_FDT_SUSPEND_RPMI=y +CONFIG_FDT_TIMER=y +CONFIG_FDT_TIMER_MTIMER=y +CONFIG_FDT_MPXY=y +CONFIG_FDT_MPXY_RPMI_MBOX=y +CONFIG_FDT_MPXY_RPMI_CLOCK=y +CONFIG_FDT_MPXY_RPMI_VOLTAGE=y +CONFIG_FDT_MPXY_RPMI_DEVICE_POWER=y +CONFIG_FDT_MPXY_RPMI_PERFORMANCE=y +CONFIG_FDT_MPXY_RPMI_SYSMSI=y
platform/generic/eswin/Kconfig new file +29 -0
@@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-2-Clause + +config ESWIN_EIC770X_FEAT0_CFG + int "ESWIN EIC7700X Feature Disable 0 CSR Configuration" + default 0x4000 + help + CSR Value to initialize EIC770X_FEAT0 (0x7c1) with. + Refer to EIC770X SoC TRM for recommendations. + +config ESWIN_EIC770X_FEAT1_CFG + int "ESWIN EIC7700X Feature Disable 1 CSR Configuration" + default 0x80 + help + CSR Value to initialize EIC770X_FEAT1 (0x7c2) with. + Refer to EIC770X SoC TRM for recommendations. + +config ESWIN_EIC770X_L1_HWPF_CFG + int "ESWIN EIC7700X L1 HW Prefetcher CSR Configuration" + default 0x1005c1be649 + help + CSR Value to initialize EIC770X_L1_HWPF (0x7c3) with. + Refer to EIC770X SoC TRM for recommendations. + +config ESWIN_EIC770X_L2_HWPF_CFG + int "ESWIN EIC7700X L2 HW Prefetcher CSR Configuration" + default 0x929f + help + CSR Value to initialize EIC770X_L2_HWPF (0x7c4) with. + Refer to EIC770X SoC TRM for recommendations.
platform/generic/eswin/eic770x.c new file +361 -0
361 lines not on this page · raw message
platform/generic/eswin/objects.mk new file +11 -0
@@ -0,0 +1,11 @@ +# +# SPDX-License-Identifier: BSD-2-Clause +# +# Copyright (C) 2025 Bo Gan <[email protected]> +# + +carray-platform_override_modules-$(CONFIG_PLATFORM_ESWIN_EIC770X) += eswin_eic7700 +platform-objs-$(CONFIG_PLATFORM_ESWIN_EIC770X) += eswin/eic770x.o + +FW_PAYLOAD=y +FW_PAYLOAD_OFFSET=0x200000
platform/generic/include/eswin/eic770x.h new file +66 -0
66 lines not on this page · raw message
Signature · 7 lines
2.34.1 -- opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Quoted 24 lines · Bo Gan
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree asFW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to
lib/andfirmware/to make it possible:
Quoted 4 lines
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- IntroduceFIRMWARE_PACKED_RXRWfor disabling power-of-2 RW split
PMP TOR entries are not going to fly. Also, adding separate config option FIRMWARE_PACKED_RXRW and separate defconfig for EIC770X means distros can't use the same fw_dyanmic.bin on EIC770X and other platforms.
If PMP TOR is being used only to save one PMP entry then there are better ways to achieve this.
For example, you can do the following ...
include/sbi/sbi_platform.h +14 -0
@@ -76,6 +76,9 @@ struct sbi_platform_operations {/* Check if specified HART is allowed to do cold boot */ bool (*cold_boot_allowed)(u32 hartid); +/* Check if platform requires single firmware region */ +bool (*single_fw_region)(void ); +/* Platform nascent initialization */ int (*nascent_init)(void );
@@ -347,6 +350,23 @@ static inline bool sbi_platform_cold_boot_allowed(return true; } +/** +* Check whether platform requires single firmware region +* +* Note: Single firmware region only works with legacy PMP because with +* Smepmp M-mode only regions can't have RWX permissions. +* +* @param plat pointer to struct sbi_platform +* +* @return true if single firmware region required and false otherwise +*/ +static inline bool sbi_platform_single_fw_region(const struct
sbi_platform *plat)
+{
+ if (plat && sbi_platform_ops(plat)->single_fw_region)
+ return sbi_platform_ops(plat)->single_fw_region();
+ return false;
+}
+
/**
* Nascent (very early) initialization for current HART *
lib/sbi/sbi_domain.c +0 -0
@@ -904,18 +904,27 @@ int sbi_domain_init(struct sbi_scratch *scratch,
u32 cold_hartid)
root.possible_harts = root_hmask;
/* Root domain firmware memory region */- sbi_domain_memregion_init(scratch->fw_start, scratch->fw_rw_offset,
- (SBI_DOMAIN_MEMREGION_M_READABLE |
- SBI_DOMAIN_MEMREGION_M_EXECUTABLE |
- SBI_DOMAIN_MEMREGION_FW),
- &root_memregs[root_memregs_count++]);
-
- sbi_domain_memregion_init((scratch->fw_start + scratch->fw_rw_offset),
- (scratch->fw_size - scratch->fw_rw_offset),
- (SBI_DOMAIN_MEMREGION_M_READABLE |
- SBI_DOMAIN_MEMREGION_M_WRITABLE |
- SBI_DOMAIN_MEMREGION_FW),
- &root_memregs[root_memregs_count++]);
+ if (sbi_platform_single_fw_region(sbi_platform_ptr(scratch))) {
+ sbi_domain_memregion_init(scratch->fw_start, scratch->fw_size,
+ (SBI_DOMAIN_MEMREGION_M_READABLE |
+ SBI_DOMAIN_MEMREGION_M_WRITABLE | + SBI_DOMAIN_MEMREGION_M_EXECUTABLE | + SBI_DOMAIN_MEMREGION_FW),
+ &root_memregs[root_memregs_count++]);
+ } else{
+ sbi_domain_memregion_init(scratch->fw_start, scratch->fw_rw_offset,
+ (SBI_DOMAIN_MEMREGION_M_READABLE |
+ SBI_DOMAIN_MEMREGION_M_EXECUTABLE |
+ SBI_DOMAIN_MEMREGION_FW),
+ &root_memregs[root_memregs_count++]);
+
+ sbi_domain_memregion_init((scratch->fw_start + scratch->fw_rw_offset),
+ (scratch->fw_size - scratch->fw_rw_offset),
+ (SBI_DOMAIN_MEMREGION_M_READABLE |
+ SBI_DOMAIN_MEMREGION_M_WRITABLE |
+ SBI_DOMAIN_MEMREGION_FW),
+ &root_memregs[root_memregs_count++]);
+ }
root.fw_region_inited = true;
Regards,
Anup
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Hi Anup,
Thanks for the prompt reply. See inline.
Quoted 30 lines · Anup Patel
Quoted 24 lines · Bo Gan
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree asFW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to
lib/andfirmware/to make it possible:Quoted 4 lines
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- IntroduceFIRMWARE_PACKED_RXRWfor disabling power-of-2 RW split
Quoted 8 lines
PMP TOR entries are not going to fly. Also, adding separate config option
FIRMWARE_PACKED_RXRWand separate defconfig for EIC770X means distros can't use the same fw_dyanmic.bin on EIC770X and other platforms.If PMP TOR is being used only to save one PMP entry then there are better ways to achieve this.
Okay. I think I've found a way to use NAPOT to achieve the same effect.
Memory Port (die 0)
├─ 0x00_8000_0000 - 0x10_8000_0000 die 0 memory (cached) └─ 0x10_8000_0000 - 0x80_0000_0000 (hole)
Memory Port (die 1)
├─ 0x00_8000_0000 - 0x20_0000_0000 (hole) ├─ 0x20_0000_0000 - 0x30_0000_0000 die 1 memory (cached) └─ 0x30_0000_0000 - 0x80_0000_0000 (hole)
By default, for both die 0/1:
NAPOT[0] ...... MRWX,SU() ____ OpenSBI in cached mem NAPOT[1] ...... MRWX,SU() ____ OpenSBI in uncached mem NAPOT[2-5] <free>
NAPOT[6] [0x0, 0x80_0000_0000) L___
NAPOT[7] ...... <reserved>
For die 0 in root domain:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached mem
NAPOT[2] [0x200_0000, 0x201_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x2000_0000, 0x4000_0000) L___ for die 1 p550
NAPOT[4] [0x0, 0x10_0000_0000) _RWX M() SU(RWX)
NAPOT[5] [0x10_0000_0000, 0x10_8000_0000) _RWX M() SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000) L___
NAPOT[7] [0x0, 0xffffffffffffffff) _RWX M() SU(RWX)
Similarly, for die 1, use:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached mem
NAPOT[2] [0x400_0000, 0x401_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x0, 0x2000_0000) L___ for die 0 p550
NAPOT[4] [0x0, 0x8000_0000) _RWX M() SU(RWX)
NAPOT[5] [0x20_0000_0000, 0x30_8000_0000) _RWX M() SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000) L___
NAPOT[7] [0x0, 0xffffffffffffffff) _RWX M() SU(RWX)
It'll work also for non-root domain harts, and actually better if it doesn't contain a "catch all" region (order == __riscv_xlen). However, compared to the TOR approach, it consumes 1 more PMP on die 0 for root domain or any domain with a "catch all" region, and we've used up all entries. There's no room for more, so we can't do something like a trusted/untrusted-domain scenario as documented in domain_support.md, because we can't block the access of tdomain memory region in udomain. I'm not sure if this is an issue at this point. Perhaps it could be when someone tries out TEE? I don't have the answer today.
include/sbi/sbi_platform.h struct sbi_platform_operations { · line 81
Quoted 13 lines
For example, you can do the following ...
include/sbi/sbi_platform.h +3
-0@@ -76,6 +76,9 @@ struct sbi_platform_operations {/* Check if specified HART is allowed to do cold boot */ bool (*cold_boot_allowed)(u32 hartid); +/* Check if platform requires single firmware region */ +bool (*single_fw_region)(void ); +
Sounds good. I'll make the change in next version.
Bo
Quoted 75 lines
include/sbi/sbi_platform.h +17
-0/* Platform nascent initialization */ int (*nascent_init)(void );@@ -347,6 +350,23 @@ static inline bool sbi_platform_cold_boot_allowed(return true; } +/** +* Check whether platform requires single firmware region +* +* Note: Single firmware region only works with legacy PMP because with +* Smepmp M-mode only regions can't have RWX permissions. +* +* @param plat pointer to struct sbi_platform +* +* @return true if single firmware region required and false otherwise +*/ +static inline bool sbi_platform_single_fw_region (const struct sbi_platform *plat) +{ +if (plat &&sbi_platform_ops (plat)->single_fw_region) +return sbi_platform_ops (plat)->single_fw_region (); +return false; +} +/** * Nascent (very early) initialization for current HART *lib/sbi/sbi_domain.c +21
-12@@ -904,18 +904,27 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid) root.possible_harts = root_hmask;/* Root domain firmware memory region */ -sbi_domain_memregion_init (scratch->fw_start, scratch->fw_rw_offset, - (SBI_DOMAIN_MEMREGION_M_READABLE | -SBI_DOMAIN_MEMREGION_M_EXECUTABLE | -SBI_DOMAIN_MEMREGION_FW ), - &root_memregs[root_memregs_count++]); - -sbi_domain_memregion_init ((scratch->fw_start + scratch->fw_rw_offset), - (scratch->fw_size - scratch->fw_rw_offset), - (SBI_DOMAIN_MEMREGION_M_READABLE | -SBI_DOMAIN_MEMREGION_M_WRITABLE | -SBI_DOMAIN_MEMREGION_FW ), - &root_memregs[root_memregs_count++]); +if (sbi_platform_single_fw_region (sbi_platform_ptr (scratch))) { +sbi_domain_memregion_init (scratch->fw_start, scratch->fw_size, + (SBI_DOMAIN_MEMREGION_M_READABLE | +SBI_DOMAIN_MEMREGION_M_WRITABLE | +SBI_DOMAIN_MEMREGION_M_EXECUTABLE | +SBI_DOMAIN_MEMREGION_FW ), + &root_memregs[root_memregs_count++]); + }else { +sbi_domain_memregion_init (scratch->fw_start, scratch->fw_rw_offset, + (SBI_DOMAIN_MEMREGION_M_READABLE | +SBI_DOMAIN_MEMREGION_M_EXECUTABLE | +SBI_DOMAIN_MEMREGION_FW ), + &root_memregs[root_memregs_count++]); + +sbi_domain_memregion_init ((scratch->fw_start + scratch->fw_rw_offset), + (scratch->fw_size - scratch->fw_rw_offset), + (SBI_DOMAIN_MEMREGION_M_READABLE | +SBI_DOMAIN_MEMREGION_M_WRITABLE | +SBI_DOMAIN_MEMREGION_FW ), + &root_memregs[root_memregs_count++]); + } root.fw_region_inited = true;Regards,
Anup
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Hi Bo Gan,
Quoted 87 lines · Bo Gan
Hi Anup,
Thanks for the prompt reply. See inline.
Quoted 30 lines · Anup Patel
Quoted 24 lines · Bo Gan
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree asFW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to
lib/andfirmware/to make it possible:Quoted 4 lines
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- IntroduceFIRMWARE_PACKED_RXRWfor disabling power-of-2 RW splitQuoted 8 lines
PMP TOR entries are not going to fly. Also, adding separate config option
FIRMWARE_PACKED_RXRWand separate defconfig for EIC770X means distros can't use the same fw_dyanmic.bin on EIC770X and other platforms.If PMP TOR is being used only to save one PMP entry then there are better ways to achieve this.
Okay. I think I've found a way to use NAPOT to achieve the same effect.
Memory Port (die 0)
├─ 0x00_8000_0000 - 0x10_8000_0000 die 0 memory (cached) └─ 0x10_8000_0000 - 0x80_0000_0000 (hole)Memory Port (die 1)
├─ 0x00_8000_0000 - 0x20_0000_0000 (hole) ├─ 0x20_0000_0000 - 0x30_0000_0000 die 1 memory (cached) └─ 0x30_0000_0000 - 0x80_0000_0000 (hole)By default, for both die 0/1:
NAPOT[0] ...... MRWX,SU() ____ OpenSBI in cached mem NAPOT[1] ...... MRWX,SU() ____ OpenSBI in uncached mem NAPOT[2-5] <free>NAPOT[6] [
0x0, 0x80_0000_0000)L___
NAPOT[7] ...... <reserved>For die 0 in root domain:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached memNAPOT[2] [0x200_0000, 0x201_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x2000_0000, 0x4000_0000)L___for die 1 p550
NAPOT[4] [0x0, 0x10_0000_0000)_RWXM()SU(RWX)
NAPOT[5] [0x10_0000_0000, 0x10_8000_0000)_RWXM()SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000)L___
NAPOT[7] [0x0,0xffffffffffffffff)_RWXM()SU(RWX)Similarly, for die 1, use:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached memNAPOT[2] [0x400_0000, 0x401_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x0, 0x2000_0000)L___for die 0 p550
NAPOT[4] [0x0, 0x8000_0000)_RWXM()SU(RWX)
NAPOT[5] [0x20_0000_0000, 0x30_8000_0000)_RWXM()SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000)L___
NAPOT[7] [0x0,0xffffffffffffffff)_RWXM()SU(RWX)
Quoted 9 lines
It'll work also for non-root domain harts, and actually better if it doesn't contain a "catch all" region (order ==
__riscv_xlen). However, compared to the TOR approach, it consumes 1 more PMP on die 0 for root domain or any domain with a "catch all" region, and we've used up all entries. There's no room for more, so we can't do something like a trusted/untrusted-domain scenario as documented in domain_support.md, because we can't block the access of tdomain memory region in udomain. I'm not sure if this is an issue at this point. Perhaps it could be when someone tries out TEE? I don't have the answer today.
There is one more approach which can save PMP entries for you where you can create a large region in platform early init covering multiple regions added by OpenSBI drivers.
For example, you can create a single region for all M-mode only MMIO devices if platform address space allows you.
Regards,
Anup
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Hi Anup,
Quoted 100 lines · Anup Patel
Hi Bo Gan,
Quoted 87 lines · Bo Gan
Hi Anup,
Thanks for the prompt reply. See inline.
Quoted 30 lines · Anup Patel
Quoted 24 lines · Bo Gan
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree asFW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to
lib/andfirmware/to make it possible:Quoted 4 lines
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- IntroduceFIRMWARE_PACKED_RXRWfor disabling power-of-2 RW splitQuoted 8 lines
PMP TOR entries are not going to fly. Also, adding separate config option
FIRMWARE_PACKED_RXRWand separate defconfig for EIC770X means distros can't use the same fw_dyanmic.bin on EIC770X and other platforms.If PMP TOR is being used only to save one PMP entry then there are better ways to achieve this.
Okay. I think I've found a way to use NAPOT to achieve the same effect.
Memory Port (die 0)
├─ 0x00_8000_0000 - 0x10_8000_0000 die 0 memory (cached) └─ 0x10_8000_0000 - 0x80_0000_0000 (hole)Memory Port (die 1)
├─ 0x00_8000_0000 - 0x20_0000_0000 (hole) ├─ 0x20_0000_0000 - 0x30_0000_0000 die 1 memory (cached) └─ 0x30_0000_0000 - 0x80_0000_0000 (hole)By default, for both die 0/1:
NAPOT[0] ...... MRWX,SU() ____ OpenSBI in cached mem NAPOT[1] ...... MRWX,SU() ____ OpenSBI in uncached mem NAPOT[2-5] <free> NAPOT[6] [0x0, 0x80_0000_0000) L___ NAPOT[7] ...... <reserved>For die 0 in root domain:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached mem NAPOT[2] [0x200_0000, 0x201_0000) ____ M(RW)SU() CLINT NAPOT[3] [0x2000_0000, 0x4000_0000) L___ for die 1 p550 NAPOT[4] [0x0, 0x10_0000_0000) _RWX M() SU(RWX) NAPOT[5] [0x10_0000_0000, 0x10_8000_0000) _RWX M() SU(RWX) NAPOT[6] [0x0, 0x80_0000_0000) L___ NAPOT[7] [0x0, 0xffffffffffffffff) _RWX M() SU(RWX)Similarly, for die 1, use:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached memNAPOT[2] [0x400_0000, 0x401_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x0, 0x2000_0000)L___for die 0 p550
NAPOT[4] [0x0, 0x8000_0000)_RWXM()SU(RWX)
NAPOT[5] [0x20_0000_0000, 0x30_8000_0000)_RWXM()SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000)L___
NAPOT[7] [0x0,0xffffffffffffffff)_RWXM()SU(RWX)Quoted 9 lines
It'll work also for non-root domain harts, and actually better if it doesn't contain a "catch all" region (order ==
__riscv_xlen). However, compared to the TOR approach, it consumes 1 more PMP on die 0 for root domain or any domain with a "catch all" region, and we've used up all entries. There's no room for more, so we can't do something like a trusted/untrusted-domain scenario as documented in domain_support.md, because we can't block the access of tdomain memory region in udomain. I'm not sure if this is an issue at this point. Perhaps it could be when someone tries out TEE? I don't have the answer today.
Quoted 7 lines
There is one more approach which can save PMP entries for you where you can create a large region in platform early init covering multiple regions added by OpenSBI drivers.
For example, you can create a single region for all M-mode only MMIO devices if platform address space allows you.
Unfortunately, the address space layout makes it impossible. we need to block CLINT from S/U, but allow the Trace components, PLIC and L3 Cache Controller from S/U.
Die 0 P550 internal region (quoting from datasheet):
R:Read, W:Write, X:Exec, I:Inst Cacheable, D:Data Cacheable, A:Atomics
0x00000000 - 0x00000FFF Debug
0x00001000 - 0x00003FFF RWX A Error Device 0x00004000 - 0x00004FFF RW A Test Indicator 0x00018000 - 0x00018FFF RWX A Trace Funnel 0x00100000 - 0x00100FFF RWX A Trace Encoder 0 0x00101000 - 0x00101FFF RWX A Trace Encoder 1 0x00102000 - 0x00102FFF RWX A Trace Encoder 2 0x00103000 - 0x00103FFF RWX A Trace Encoder 3 0x00104000 - 0x00107FFF RWX A Hart 0 Private L2 Cache 0x00108000 - 0x0010BFFF RWX A Hart 1 Private L2 Cache 0x0010C000 - 0x0010FFFF RWX A Hart 2 Private L2 Cache 0x00110000 - 0x00113FFF RWX A Hart 3 Private L2 Cache 0x00200000 - 0x00200FFF RW A Bus blocker TL64D2D_Out 0x00202000 - 0x00202FFF RW A Bus blocker TL256D2D_Out 0x00204000 - 0x00204FFF RW A Bus blocker TL256D2D_In 0x01700000 - 0x01700FFF RWX A Hart 0 Bus-Error Unit 0x01701000 - 0x01701FFF RWX A Hart 1 Bus-Error Unit 0x01702000 - 0x01702FFF RWX A Hart 2 Bus-Error Unit 0x01703000 - 0x01703FFF RWX A Hart 3 Bus-Error Unit 0x02000000 - 0x0200FFFF RWX A CLINT 0x02010000 - 0x02013FFF RWX A L3 Cache Controller 0x08000000 - 0x083FFFFF RWX A L3 LIM 0x0C000000 - 0x0FFFFFFF RWX A PLIC 0x10010000 - 0x10010FFF RWX A Burst Bundler on Front Port 0x10030000 - 0x10033FFF RWX A TL-UH to TL-C Adapter
0x1A000000 - 0x1A3FFFFF RWXIDA L3 Zero Device
I re-read the EIC7700 datasheet, and I finally got a theory on the reason behind bus errors. There are 3 regions having data cacheable attribute in the chip hard-coded PMA:
- [0x00_1a00_0000, 0x00_1a40_0000) -- L3 zero device for die 0
- [0x00_3a00_0000, 0x00_3a40_0000) -- L3 zero device for die 1
- [0x00_8000_0000, 0x80_0000_0000) -- entire memory port
They can be speculatively accessed or HW prefetched, so if there's only 1 die and there're holes in these regions, you'll get bus error from time to time if the core happens to touch it. Thus, whatever belongs to the other (remote) die needs to be blocked. Hopefully this only applies for data cacheable regions, otherwise, I'd say it's horribly broken. I already dislike the fact that we have to use PMP to do it, but it's what it is. In the PMP configuration above, I chose to block die 1 L3 zero device from die 0 by disallowing the entire internal port for die 1. Or I can do it with finer granularity. Either way, it can't be omitted.
What I can do is configure the PMPs with NAPOTs as you suggested. It'll be like the detailed list above. This should cover 99% of use cases. For users with a udomain-tdomain like setup, they can still try it, but put a note in the comment and say due to HW limitation, it's experimental only and you can't properly protect tdomain from udomain.
Does it sound good?
Bo
Quoted 2 lines
Regards,
Anup
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Quoted 154 lines · Bo Gan
Hi Anup,
Quoted 100 lines · Anup Patel
Hi Bo Gan,
Quoted 87 lines · Bo Gan
Hi Anup,
Thanks for the prompt reply. See inline.
Quoted 30 lines · Anup Patel
Quoted 24 lines · Bo Gan
EIC7700 is the SoC used in HiFive P550 and Milk-V Megrez. This SoC is currently one of the only off-the-shelf board/chips that support H extension, although it's v0.6.1. It also supports pre-ratified N-trace. Add support for it so people can benefit from latest OpenSBI features.
The device-tree of HiFive P550 has been upstreamed to Linux:
https://lore.kernel.org/all/[email protected]/
However U-boot is not, and there are bugs in vendor U-boot device-tree, and also inconsistencies between the two. Thus, this patch is coded with the upstreamed device-tree as the reference, but tested with the patched vendor U-boot device tree asFW_FDT_PATH. The patched vendor U-boot is hosted here: https://github.com/ganboing/u-boot-eic7x/tree/eic7x-dt-fix
Refer to PATCH 5/5 for the instructions on building the firmware blob and launch it through UART boot.The major complication of this chip is that it requires certain memory regions to be blocked with PMP entries to prevent speculative execution or HW prefetcher from touching them to avoid bus errors. Also due to the fact that this SoC handles cache incoherent DMA by mapping memory twice, one as cached, and the other as uncached, we also need an extra PMP to protect the OpenSBI in the uncached portion in address space. Following changes are made to
lib/andfirmware/to make it possible:Quoted 4 lines
- Allow platform to override pmp_(un)configure
- Add helper function for settings PMP TOR
- Add helper function to check if memregions are disjoint
- IntroduceFIRMWARE_PACKED_RXRWfor disabling power-of-2 RW splitQuoted 8 lines
PMP TOR entries are not going to fly. Also, adding separate config option
FIRMWARE_PACKED_RXRWand separate defconfig for EIC770X means distros can't use the same fw_dyanmic.bin on EIC770X and other platforms.If PMP TOR is being used only to save one PMP entry then there are better ways to achieve this.
Okay. I think I've found a way to use NAPOT to achieve the same effect.
Memory Port (die 0)
├─ 0x00_8000_0000 - 0x10_8000_0000 die 0 memory (cached) └─ 0x10_8000_0000 - 0x80_0000_0000 (hole)Memory Port (die 1)
├─ 0x00_8000_0000 - 0x20_0000_0000 (hole) ├─ 0x20_0000_0000 - 0x30_0000_0000 die 1 memory (cached) └─ 0x30_0000_0000 - 0x80_0000_0000 (hole)By default, for both die 0/1:
NAPOT[0] ...... MRWX,SU() ____ OpenSBI in cached mem NAPOT[1] ...... MRWX,SU() ____ OpenSBI in uncached mem NAPOT[2-5] <free> NAPOT[6] [0x0, 0x80_0000_0000) L___ NAPOT[7] ...... <reserved>For die 0 in root domain:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached mem NAPOT[2] [0x200_0000, 0x201_0000) ____ M(RW)SU() CLINT NAPOT[3] [0x2000_0000, 0x4000_0000) L___ for die 1 p550 NAPOT[4] [0x0, 0x10_0000_0000) _RWX M() SU(RWX) NAPOT[5] [0x10_0000_0000, 0x10_8000_0000) _RWX M() SU(RWX) NAPOT[6] [0x0, 0x80_0000_0000) L___ NAPOT[7] [0x0, 0xffffffffffffffff) _RWX M() SU(RWX)Similarly, for die 1, use:
NAPOT[0] ...... M(RWX)SU() ____ OpenSBI in cached mem NAPOT[1] ...... M(RWX)SU() ____ OpenSBI in uncached memNAPOT[2] [0x400_0000, 0x401_0000) ____ M(RW)SU() CLINT
NAPOT[3] [0x0, 0x2000_0000)L___for die 0 p550
NAPOT[4] [0x0, 0x8000_0000)_RWXM()SU(RWX)
NAPOT[5] [0x20_0000_0000, 0x30_8000_0000)_RWXM()SU(RWX)
NAPOT[6] [0x0, 0x80_0000_0000)L___
NAPOT[7] [0x0,0xffffffffffffffff)_RWXM()SU(RWX)Quoted 9 lines
It'll work also for non-root domain harts, and actually better if it doesn't contain a "catch all" region (order ==
__riscv_xlen). However, compared to the TOR approach, it consumes 1 more PMP on die 0 for root domain or any domain with a "catch all" region, and we've used up all entries. There's no room for more, so we can't do something like a trusted/untrusted-domain scenario as documented in domain_support.md, because we can't block the access of tdomain memory region in udomain. I'm not sure if this is an issue at this point. Perhaps it could be when someone tries out TEE? I don't have the answer today.Quoted 7 lines
There is one more approach which can save PMP entries for you where you can create a large region in platform early init covering multiple regions added by OpenSBI drivers.
For example, you can create a single region for all M-mode only MMIO devices if platform address space allows you.
Unfortunately, the address space layout makes it impossible. we need to block CLINT from S/U, but allow the Trace components, PLIC and L3 Cache Controller from S/U.
Die 0 P550 internal region (quoting from datasheet):
R:Read, W:Write, X:Exec, I:Inst Cacheable, D:Data Cacheable, A:Atomics
0x00000000-0x00000FFFDebug0x00001000 - 0x00003FFF RWX A Error Device 0x00004000 - 0x00004FFF RW A Test Indicator 0x00018000 - 0x00018FFF RWX A Trace Funnel 0x00100000 - 0x00100FFF RWX A Trace Encoder 0 0x00101000 - 0x00101FFF RWX A Trace Encoder 1 0x00102000 - 0x00102FFF RWX A Trace Encoder 2 0x00103000 - 0x00103FFF RWX A Trace Encoder 3 0x00104000 - 0x00107FFF RWX A Hart 0 Private L2 Cache 0x00108000 - 0x0010BFFF RWX A Hart 1 Private L2 Cache 0x0010C000 - 0x0010FFFF RWX A Hart 2 Private L2 Cache 0x00110000 - 0x00113FFF RWX A Hart 3 Private L2 Cache 0x00200000 - 0x00200FFF RW A Bus blocker TL64D2D_Out 0x00202000 - 0x00202FFF RW A Bus blocker TL256D2D_Out 0x00204000 - 0x00204FFF RW A Bus blocker TL256D2D_In 0x01700000 - 0x01700FFF RWX A Hart 0 Bus-Error Unit 0x01701000 - 0x01701FFF RWX A Hart 1 Bus-Error Unit 0x01702000 - 0x01702FFF RWX A Hart 2 Bus-Error Unit 0x01703000 - 0x01703FFF RWX A Hart 3 Bus-Error Unit 0x02000000 - 0x0200FFFF RWX A CLINT 0x02010000 - 0x02013FFF RWX A L3 Cache Controller 0x08000000 - 0x083FFFFF RWX A L3 LIM 0x0C000000 - 0x0FFFFFFF RWX A PLIC 0x10010000 - 0x10010FFF RWX A Burst Bundler on Front Port 0x10030000 - 0x10033FFF RWX A TL-UH to TL-C Adapter
0x1A000000-0x1A3FFFFFRWXIDA L3 Zero DeviceI re-read the EIC7700 datasheet, and I finally got a theory on the reason behind bus errors. There are 3 regions having data cacheable attribute in the chip hard-coded PMA:
- [0x00_1a00_0000, 0x00_1a40_0000) -- L3 zero device for die 0
- [0x00_3a00_0000, 0x00_3a40_0000) -- L3 zero device for die 1
- [0x00_8000_0000, 0x80_0000_0000) -- entire memory port
Quoted 9 lines
They can be speculatively accessed or HW prefetched, so if there's only 1 die and there're holes in these regions, you'll get bus error from time to time if the core happens to touch it. Thus, whatever belongs to the other (remote) die needs to be blocked. Hopefully this only applies for data cacheable regions, otherwise, I'd say it's horribly broken. I already dislike the fact that we have to use PMP to do it, but it's what it is. In the PMP configuration above, I chose to block die 1 L3 zero device from die 0 by disallowing the entire internal port for die 1. Or I can do it with finer granularity. Either way, it can't be omitted.
PMP is a scarce resource across platforms and using it for work-arounds is an expensive affair.
Quoted 9 lines
What I can do is configure the PMPs with NAPOTs as you suggested. It'll be like the detailed list above. This should cover 99% of use cases. For users with a udomain-tdomain like setup, they can still try it, but put a note in the comment and say due to HW limitation, it's experimental only and you can't properly protect tdomain from udomain.
Does it sound good?
Yes, that should be fine.
Regards,
Anup
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi
Hi Anup,
Actually I managed to do a tdomain/udomain like setup on die 0. There's a free PMP entry there. I've also simplified the PMP consolidation logic and refactored the comments to make it more readable. Please see v3.
Thanks!
Bo
Signature · 3 lines
opensbi mailing list [email protected] http://lists.infradead.org/mailman/listinfo/opensbi