xen/arm: Add Armv8-R64 MPU support to Xen - Part#1
PATCH v6 2/11 xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Wei Chendomain_build use ioremap_wc to map a new non-cacheable virtual address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.
So in this patch, we add an iounmap to the end of domain_build, after Xen loaded initrd to guest memory.
--- xen/arch/arm/domain_build.c | 2 ++ 1 file changed, 2 insertions(+)
xen/arch/arm/domain_build.c +2 -0
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo) initrd, len);if ( res !=0 )panic ("Unable to copy the initrd in the hwdom memory\n" ); + +iounmap (initrd); }/*
Signature · 1 line
2.25.1
v6 cover xen/arm: Add Armv8-R64 MPU support to Xen - Part#1
The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)
Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.
We will split these patches to several parts, this series is the part#1, the full PoC can be found in [3]. More software for Armv8-R64 can be found in [4];
[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/
Penny Zheng (3):
xen/arm64: create boot-time MPU protection regions xen/arm64: introduce helpers for MPU enable/disable xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
Wei Chen (8):
xen/arm: remove xen_phys_start and xenheap_phys_end from config.h xen/arm: add iounmap after initrd has been loaded in domain_build xen/arm: disable EFI boot services for MPU systems xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA xen/arm: define Xen start address for FVP BaseR platform xen/arm: split MMU and MPU config files from config.h xen/arm: implement FIXMAP_ADDR for MPU systems xen/arm64: move MMU related code from head.S to head_mmu.S
xen/arch/arm/Kconfig | 15 +- xen/arch/arm/arm64/Makefile | 5 + xen/arch/arm/arm64/head.S | 429 ++---------------- xen/arch/arm/arm64/head_mmu.S | 364 +++++++++++++++ xen/arch/arm/arm64/head_mpu.S | 154 +++++++ xen/arch/arm/domain_build.c | 2 + xen/arch/arm/include/asm/arm64/flushtlb.h | 25 + xen/arch/arm/include/asm/arm64/macros.h | 52 ++- xen/arch/arm/include/asm/arm64/mpu.h | 13 + xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++ xen/arch/arm/include/asm/config.h | 99 +--- xen/arch/arm/include/asm/config_mmu.h | 119 +++++ xen/arch/arm/include/asm/config_mpu.h | 29 ++ xen/arch/arm/include/asm/fixmap.h | 25 + xen/arch/arm/include/asm/flushtlb.h | 22 + .../arm/include/asm/platforms/fvp_baser.h | 18 + xen/arch/arm/platforms/Kconfig | 16 +- 17 files changed, 976 insertions(+), 500 deletions(-) create mode 100644 xen/arch/arm/arm64/head_mmu.S create mode 100644 xen/arch/arm/arm64/head_mpu.S create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h create mode 100644 xen/arch/arm/include/asm/config_mmu.h create mode 100644 xen/arch/arm/include/asm/config_mpu.h create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
Signature · 1 line
2.25.1
PATCH v6 1/11 xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
These two variables are stale variables, they only have declarations in config.h, they don't have any definition and no any code is using these two variables. So in this patch, we remove them from config.h.
--- xen/arch/arm/include/asm/config.h | 2 -- 1 file changed, 2 deletions(-)
xen/arch/arm/include/asm/config.h +0 -2
@@ -172,8 +172,6 @@#define STACK_SIZE (PAGE_SIZE << STACK_ORDER)#ifndef __ASSEMBLY__ -extern unsigned long xen_phys_start; -extern unsigned long xenheap_phys_end;extern unsigned long frametable_virt_end;#endif
Signature · 1 line
2.25.1
PATCH v6 3/11 xen/arm: disable EFI boot services for MPU systems
In reply to Wei ChenCurrent EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support. But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.
--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/arm64/head.S | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-)
xen/arch/arm/Kconfig +1 -1
@@ -7,7 +7,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT - select ARM_EFI + select ARM_EFI if !HAS_MPU select HAS_FAST_MULTIPLY config ARM
xen/arch/arm/arm64/head.S +6 -2
@@ -172,8 +172,10 @@ efi_head: .byte 0x52 .byte 0x4d .byte 0x64 - .long pe_header - efi_head /* Offset to the PE header. */ - +#ifndef CONFIG_ARM_EFI + .long 0 /* 0 means no PE header. */ +#else + .long pe_header - efi_head /* Offset to the PE header. */ /* * Add the PE/COFF header to the file. The address of this header * is at offset 0x3c in the file, and is part of Linux "Image"
@@ -279,6 +281,8 @@ section_table: .short 0 /* NumberOfLineNumbers (0 for executables) */ .long 0xe0500020 /* Characteristics (section flags) */ .align 5 +#endif /* CONFIG_ARM_EFI */ + real_start: /* BSS should be zeroed when booting without EFI */ mov x26, #0 /* x26 := skip_zero_bss */
Signature · 1 line
2.25.1
PATCH v6 4/11 xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
In reply to Wei ChenFrom Arm ARM Supplement of Armv8-R AArch64 (DDI 0600A) [1], section D1.6.2 TLB maintenance instructions. We know that Armv8-R AArch64 permits an implementation to cache stage 1 VMSAv8-64 and stage 2 PMSAv8-64 attributes as a common entry for the Secure EL1&0 translation regime. But for Xen itself, it's running with stage 1 PMSAv8-64 on Armv8-R AArch64. The EL2 MPU updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we don't need any TLB invalidation for Xen itself in EL2.
So in this patch, we use empty macros to stub Xen TLB helpers for MPU system (PMSA), but still keep the Guest TLB helpers. Because when a guest running in EL1 with VMSAv8-64 (MMU), guest TLB invalidation is still needed. But we need some policy to distinguish MPU and MMU guest, this will be done in guest support of Armv8-R64 later.
[1] https://developer.arm.com/documentation/ddi0600/ac
--- xen/arch/arm/include/asm/arm64/flushtlb.h | 25 +++++++++++++++++++++++ xen/arch/arm/include/asm/flushtlb.h | 22 ++++++++++++++++++++ 2 files changed, 47 insertions(+)
xen/arch/arm/include/asm/arm64/flushtlb.h +25 -0
@@ -51,6 +51,8 @@ TLB_HELPER(flush_all_guests_tlb_local, alle1);/* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */ TLB_HELPER (flush_all_guests_tlb, alle1is); +#ifndef CONFIG_HAS_MPU +/* Flush all hypervisor mappings from the TLB of the local processor. */ TLB_HELPER (flush_xen_tlb_local, alle2);
@@ -66,6 +68,29 @@ static inline void __flush_xen_tlb_one(vaddr_t va) asmvolatile ("tlbi vae2is, %0;" : :"r" (va>>PAGE_SHIFT ) :"memory" ); } +#else + +/* +* When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU +* updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we +* don't need any TLB invalidation for Xen itself in EL2. See Arm ARM +* Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance +* instructions for more details. +*/ +static inline void flush_xen_tlb_local (void ) +{ +} + +static inline void __flush_xen_tlb_one_local (vaddr_t va) +{ +} + +static inline void __flush_xen_tlb_one (vaddr_t va) +{ +} + +#endif /* CONFIG_HAS_MPU */ +#endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */ /* * Local variables:
xen/arch/arm/include/asm/flushtlb.h +22 -0
@@ -28,6 +28,7 @@ static inline void page_set_tlbflush_timestamp(struct page_info *page)/* Flush specified CPUs' TLBs */ void arch_flush_tlb_mask (const cpumask_t *mask); +#ifndef CONFIG_HAS_MPU /* * Flush a range of VA's hypervisor mappings from the TLB of the local * processor.
@@ -66,6 +67,27 @@ static inline void flush_xen_tlb_range_va(vaddr_t va,isb (); } +#else + +/* +* When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU +* updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we +* don't need any TLB invalidation for Xen itself in EL2. See Arm ARM +* Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance +* instructions for more details. +*/ +static inline void flush_xen_tlb_range_va_local (vaddr_t va, +unsigned long size) +{ +} + +static inline void flush_xen_tlb_range_va (vaddr_t va, +unsigned long size) +{ +} + +#endif /* CONFIG_HAS_MPU */ +#endif /* __ASM_ARM_FLUSHTLB_H__ */ /* * Local variables:
Signature · 1 line
2.25.1
PATCH v6 5/11 xen/arm: define Xen start address for FVP BaseR platform
In reply to Wei ChenOn Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.
We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.
And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++ xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
xen/arch/arm/Kconfig +11 -0
@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allows guests to access real TEE via one of TEE mediators implemented in XEN. +config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform defined address" + default 0xFFFFFFFF + depends on HAS_MPU + help + This option allows to set the customized address at which Xen will be + linked on MPU systems. This address must be aligned to a page size. + Use 0xFFFFFFFF as the default value to indicate that user hasn't + customized this address, and Xen use use the default value that has + been defined in platform files. + source "arch/arm/tee/Kconfig" config STATIC_SHM
xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14 -0
@@ -0,0 +1,14 @@ +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__ +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__ + +/* +* 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS, +* we will use platform defined default address. +*/ +#if CONFIG_XEN_START_ADDRESS ==0xFFFFFFFF +#define XEN_START_ADDRESS 0x200000 +#else +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS +#endif + +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
xen/arch/arm/platforms/Kconfig +13 -3
@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8R ---help--- Choose which hardware platform to enable in Xen.
@@ -8,13 +9,14 @@ choice config ALL_PLAT bool "All Platforms" + depends on !ARM_V8R ---help--- Enable support for all available hardware platforms. It doesn't automatically select any of the related drivers. config QEMU bool "QEMU aarch virt machine support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select GICV3 select HAS_PL011 ---help---
@@ -23,7 +25,7 @@ config QEMU config RCAR3 bool "Renesas RCar3 support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_SCIF select IPMMU_VMSA ---help---
@@ -31,14 +33,22 @@ config RCAR3 config MPSOC bool "Xilinx Ultrascale+ MPSoC support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_CADENCE_UART select ARM_SMMU ---help--- Enable all the required drivers for Xilinx Ultrascale+ MPSoC +config FVP_BASER + bool "Fixed Virtual Platform BaseR support" + depends on ARM_V8R + help + Enable platform specific configurations for Fixed Virtual + Platform BaseR + config NO_PLAT bool "No Platforms" + depends on !ARM_V8R ---help--- Do not enable specific support for any platform.
Signature · 1 line
2.25.1
PATCH v6 6/11 xen/arm: split MMU and MPU config files from config.h
In reply to Wei ChenXen defines some global configuration macros for Arm in config.h. We still want to use it for Armv8-R systems, but there are some address related macros that are defined for MMU systems. These macros will not be used by MPU systems, Adding ifdefery with CONFIG_HAS_MPU to gate these macros will result in a messy and hard to read/maintain code.
So we keep some common definitions still in config.h, but move virtual address related definitions to a new file - config_mmu.h. And use a new file config_mpu.h to store definitions for MPU systems. To avoid spreading #ifdef everywhere, we keep the same definition names for MPU systems, like XEN_VIRT_START and HYPERVISOR_VIRT_START, but the definition contents are MPU specific.
--- xen/arch/arm/include/asm/config.h | 97 +-------------------- xen/arch/arm/include/asm/config_mmu.h | 119 ++++++++++++++++++++++++++ xen/arch/arm/include/asm/config_mpu.h | 27 ++++++ 3 files changed, 150 insertions(+), 93 deletions(-) create mode 100644 xen/arch/arm/include/asm/config_mmu.h create mode 100644 xen/arch/arm/include/asm/config_mpu.h
xen/arch/arm/include/asm/config.h +4 -93
@@ -71,99 +71,10 @@#include <xen/const.h> #include <xen/page-size.h> -/* -* Common ARM32 and ARM64 layout: -* 0 - 2M Unmapped -* 2M - 4M Xen text, data, bss -* 4M - 6M Fixmap: special-purpose 4K mapping slots -* 6M - 10M Early boot mapping of FDT -* 10M - 12M Livepatch vmap (if compiled in) -* -* ARM32 layout: -* 0 - 12M <COMMON> -* -* 32M - 128M Frametable: 24 bytes per page for 16GB of RAM -* 256M - 1G VMAP: ioremap and early_ioremap use this virtual address -* space -* -* 1G - 2G Xenheap: always-mapped memory -* 2G - 4G Domheap: on-demand-mapped -* -* ARM64 layout: -* 0x0000000000000000 - 0x0000007fffffffff (512GB, L0 slot [0]) -* 0 - 12M <COMMON> -* -* 1G - 2G VMAP: ioremap and early_ioremap -* -* 32G - 64G Frametable: 24 bytes per page for 5.3TB of RAM -* -* 0x0000008000000000 - 0x00007fffffffffff (127.5TB, L0 slots [1..255]) -* Unused -* -* 0x0000800000000000 - 0x000084ffffffffff (5TB, L0 slots [256..265]) -* 1:1 mapping of RAM -* -* 0x0000850000000000 - 0x0000ffffffffffff (123TB, L0 slots [266..511]) -* Unused -*/ - -#define XEN_VIRT_START _AT(vaddr_t,0x00200000) -#define FIXMAP_ADDR (n) (_AT(vaddr_t,0x00400000) + (n) * PAGE_SIZE) - -#define BOOT_FDT_VIRT_START _AT(vaddr_t,0x00600000) -#define BOOT_FDT_VIRT_SIZE _AT(vaddr_t, MB(4)) - -#ifdef CONFIG_LIVEPATCH -#define LIVEPATCH_VMAP_START _AT(vaddr_t,0x00a00000) -#define LIVEPATCH_VMAP_SIZE _AT(vaddr_t, MB(2)) -#endif - -#define HYPERVISOR_VIRT_START XEN_VIRT_START - -#ifdef CONFIG_ARM_32 - -#define CONFIG_SEPARATE_XENHEAP 1 - -#define FRAMETABLE_VIRT_START _AT(vaddr_t,0x02000000) -#define FRAMETABLE_SIZE MB(128-32) -#define FRAMETABLE_NR (FRAMETABLE_SIZE / sizeof(*frame_table)) -#define FRAMETABLE_VIRT_END (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1) - -#define VMAP_VIRT_START _AT(vaddr_t,0x10000000) -#define VMAP_VIRT_SIZE _AT(vaddr_t, GB(1) - MB(256)) - -#define XENHEAP_VIRT_START _AT(vaddr_t,0x40000000) -#define XENHEAP_VIRT_SIZE _AT(vaddr_t, GB(1)) - -#define DOMHEAP_VIRT_START _AT(vaddr_t,0x80000000) -#define DOMHEAP_VIRT_SIZE _AT(vaddr_t, GB(2)) - -#define DOMHEAP_ENTRIES 1024/* 1024 2MB mapping slots */ - -/* Number of domheap pagetable pages required at the second level (2MB mappings) */ -#define DOMHEAP_SECOND_PAGES (DOMHEAP_VIRT_SIZE >> FIRST_SHIFT) - -#else /* ARM_64 */ - -#define SLOT0_ENTRY_BITS 39 -#define SLOT0 (slot) (_AT(vaddr_t,slot) << SLOT0_ENTRY_BITS) -#define SLOT0_ENTRY_SIZE SLOT0(1) - -#define VMAP_VIRT_START GB(1) -#define VMAP_VIRT_SIZE GB(1) - -#define FRAMETABLE_VIRT_START GB(32) -#define FRAMETABLE_SIZE GB(32) -#define FRAMETABLE_NR (FRAMETABLE_SIZE / sizeof(*frame_table)) - -#define DIRECTMAP_VIRT_START SLOT0(256) -#define DIRECTMAP_SIZE (SLOT0_ENTRY_SIZE * (265-256)) -#define DIRECTMAP_VIRT_END (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1) - -#define XENHEAP_VIRT_START directmap_virt_start - -#define HYPERVISOR_VIRT_END DIRECTMAP_VIRT_END - +#ifdef CONFIG_HAS_MPU +#include <asm/config_mpu.h> +#else +#include <asm/config_mmu.h> #endif #define NR_hypercalls 64
xen/arch/arm/include/asm/config_mmu.h new file +119 -0
119 lines not on this page · raw message
xen/arch/arm/include/asm/config_mpu.h new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* config_mpu.h: A Linux-style configuration list for Arm MPU systems, +* only can be included by config.h +*/ + +#ifndef __ARM_CONFIG_MPU_H__ +#define __ARM_CONFIG_MPU_H__ + +#ifdef CONFIG_FVP_BASER +#include <asm/platforms/fvp_baser.h> +#endif + +/* +* All MPU platforms need to provide a XEN_START_ADDRESS for linker. This +* address indicates where Xen image will be loaded and run from. This +* address must be aligned to a PAGE_SIZE. +*/ +#if (XEN_START_ADDRESS %PAGE_SIZE ) !=0 +#error "XEN_START_ADDRESS must be aligned to PAGE_SIZE" +#endif + +#define XEN_VIRT_START _AT(paddr_t, XEN_START_ADDRESS) + +#define HYPERVISOR_VIRT_START XEN_VIRT_START + +#endif /* __ARM_CONFIG_MPU_H__ */
Signature · 1 line
2.25.1
PATCH v6 7/11 xen/arm: implement FIXMAP_ADDR for MPU systems
In reply to Wei ChenFIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.
But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using FIXMAP_ADDR to return the input physical address in MPU systems. So in MPU version, FIXMAP_ADDR will trim physical address to PAGE alignment. This will return an offset which is similar to MMU version FIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask with PAGE_MASK). The caller can return the input physical address directly.
As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.
--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/include/asm/config_mpu.h | 2 ++ xen/arch/arm/include/asm/fixmap.h | 25 +++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -1
@@ -16,7 +16,7 @@ config ARM select HAS_DEVICE_TREE select HAS_PASSTHROUGH select HAS_PDX - select HAS_PMAP + select HAS_PMAP if !HAS_MPU select IOMMU_FORCE_PT_SHARE config ARCH_DEFCONFIG
xen/arch/arm/include/asm/config_mpu.h +2 -0
@@ -24,4 +24,6 @@#define HYPERVISOR_VIRT_START XEN_VIRT_START +#define FIXMAP_ADDR (n) (_AT(paddr_t, n) & (PAGE_MASK)) +#endif /* __ARM_CONFIG_MPU_H__ */
xen/arch/arm/include/asm/fixmap.h +25 -0
@@ -7,6 +7,8 @@#include <xen/acpi.h> #include <xen/pmap.h> +#ifndef CONFIG_HAS_MPU +/* Fixmap slots */ #define FIXMAP_CONSOLE 0/* The primary UART */ #define FIXMAP_MISC 1/* Ephemeral mappings of hardware */
@@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)#endif /* __ASSEMBLY__ */ +#else + +/* +* FIXMAP_ADDR will trim physical address to PAGE alignment. +* This will return an offset which is similar to MMU version +* FIXMAP_ADDR. +* For example: +* EARLY_UART_VIRTUAL_ADDRESS is defined by: +* (FIXMAP_ADDR(FIXMAP_CONSOLE) + \ +* (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK)) +* With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions, +* EARLY_UART_VIRTUAL_ADDRESS can be restore to +* CONFIG_EARLY_UART_BASE_ADDRESS. +* In this case, we don't need to use #ifdef MPU in the code +* where are using FIXMAP_ADDR to make them to use physical +* address explicitily. +*/ +#ifdef CONFIG_EARLY_UART_BASE_ADDRESS +#define FIXMAP_CONSOLE CONFIG_EARLY_UART_BASE_ADDRESS +#endif + +#endif /* CONFIG_HAS_MPU */ +#endif /* __ASM_FIXMAP_H */
Signature · 1 line
2.25.1
PATCH v6 8/11 xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Wei ChenThere are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.
As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros: adr_l and load_paddr will be used by MMU and MPU later, so we move them to macros.h.
--- xen/arch/arm/arm64/Makefile | 3 + xen/arch/arm/arm64/head.S | 407 +----------------------- xen/arch/arm/arm64/head_mmu.S | 364 +++++++++++++++++++++ xen/arch/arm/include/asm/arm64/macros.h | 52 ++- 4 files changed, 432 insertions(+), 394 deletions(-) create mode 100644 xen/arch/arm/arm64/head_mmu.S
xen/arch/arm/arm64/Makefile +3 -0
@@ -8,6 +8,9 @@ obj-y += domctl.o obj-y += domain.o obj-y += entry.o obj-y += head.o +ifneq ($(CONFIG_HAS_MPU),y) +obj-y += head_mmu.o +endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += smc.o
xen/arch/arm/arm64/head.S +14 -393
448 lines not on this page · raw message
xen/arch/arm/arm64/head_mmu.S new file +364 -0
364 lines not on this page · raw message
xen/arch/arm/include/asm/arm64/macros.h +51 -1
61 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 11/11 xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
In reply to Wei ChenFrom: Penny Zheng <[email protected]>
setup_fixmap and remove_identity_mapping are two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow in head.S doesn't need to use #ifdef to gate MPU/MMU code.
In MMU system, setup_fixmap is used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but before setup_fixmap. This restriction is the same as MMU system.
For remove_identity_mapping, we just need an empty function to make head.S code flow happy.
--- xen/arch/arm/arm64/head_mpu.S | 49 +++++++++++++++++++ .../arm/include/asm/platforms/fvp_baser.h | 4 ++ 2 files changed, 53 insertions(+)
xen/arch/arm/arm64/head_mpu.S +49 -0
@@ -20,13 +20,20 @@ /* * In boot stage, we will use 1 MPU region: * Region#0: Normal memory for Xen text + data + bss (2MB) + * Region#1: Device memory for EARLY UART, size is defined + * by platform's EARLY_UART_SIZE */ #define BOOT_NORMAL_REGION_IDX 0x0 +#define BOOT_DEVICE_REGION_IDX 0x1 /* MPU normal memory attributes. */ #define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */ #define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */ +/* MPU device memory attributes. */ +#define PRBAR_DEVICE_MEM 0x20 /* SH=10 AP=00 XN=00 */ +#define PRLAR_DEVICE_MEM 0x09 /* NS=0 ATTR=100 EN=1 */ + .macro write_pr, sel, prbar, prlar msr PRSELR_EL2, \sel dsb sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings) ret ENDPROC(prepare_early_mappings) +/* + * In MMU system, setup_fixmap is used for Xen to map some essential data + * or devices in boot-time. In order to be consistent with MMU system, we + * inherit the function name for MPU system. + * setup_fixmap of MPU system will: + * - Map the early UART to MPU protection region when earlyprintk is + * enabled (The PRINT can't be used after turning on MPU but before + * setup_fixmap). + * + * Clobbers x0 - x3 + */ +ENTRY(setup_fixmap) +#ifdef CONFIG_EARLY_PRINTK + /* Map early uart to MPU device region for early printk. */ + mov x0, #BOOT_DEVICE_REGION_IDX + ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS + and x1, x1, #MPU_REGION_MASK + mov x3, #PRBAR_DEVICE_MEM + orr x1, x1, x3 + + ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS + ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1) + add x2, x2, x3 + and x2, x2, #MPU_REGION_MASK + mov x3, #PRLAR_DEVICE_MEM + orr x2, x2, x3 + + /* + * Write to MPU protection region: + * x0 for pr_sel, x1 for prbar x2 for prlar + */ + write_pr x0, x1, x2 +#endif + + ret +ENDPROC(setup_fixmap) + +/* Stub of remove_identity_mapping for MPU systems */ +ENTRY(remove_identity_mapping) + ret +ENDPROC(remove_identity_mapping) + /* * Enable EL2 MPU and data cache. Because we will disable cache * with MPU at the same time, in accordance with that, we have
xen/arch/arm/include/asm/platforms/fvp_baser.h +4 -0
@@ -11,4 +11,8 @@#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS#endif +#ifdef CONFIG_EARLY_PRINTK +#define EARLY_UART_SIZE 0x1000 +#endif +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
Signature · 1 line
2.25.1
PATCH v6 10/11 xen/arm64: introduce helpers for MPU enable/disable
In reply to Wei ChenFrom: Penny Zheng <[email protected]>
We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered. So in this situation, we need to disable the whole MPU to update the protection regions.
In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent with enable_mmu. For MMU systems, enable_mmu will turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.
In this patch, we also introduce a neutral name enable_mm for Xen to enable MMU/MPU. This can help us to keep one code flow in head.S
--- xen/arch/arm/arm64/head.S | 5 +++-- xen/arch/arm/arm64/head_mmu.S | 4 ++-- xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +3 -2
@@ -255,7 +255,8 @@ real_start_efi: * and protection regions for MPU systems. */ bl prepare_early_mappings - bl enable_mmu + /* Turn on MMU or MPU */ + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */ ldr x0, =primary_switched
@@ -313,7 +314,7 @@ GLOBAL(init_secondary) bl check_cpu_mode bl cpu_init bl prepare_early_mappings - bl enable_mmu + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */ ldr x0, =secondary_switched
xen/arch/arm/arm64/head_mmu.S +2 -2
@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings) * * Clobbers x0 - x3 */ -ENTRY(enable_mmu) +ENTRY(enable_mm) PRINT("- Turning on paging -\r\n") /*
@@ -239,7 +239,7 @@ ENTRY(enable_mmu) msr SCTLR_EL2, x0 /* now paging is enabled */ isb /* Now, flush the icache */ ret -ENDPROC(enable_mmu) +ENDPROC(enable_mm) /* * Remove the 1:1 map from the page-tables. It is not easy to keep track
xen/arch/arm/arm64/head_mpu.S +35 -0
@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings) ret ENDPROC(prepare_early_mappings) + +/* + * Enable EL2 MPU and data cache. Because we will disable cache + * with MPU at the same time, in accordance with that, we have + * to enable cache with MPU at the same time in this function. + * When MPU is disabled, the MPU background attributes will + * be used. On some platform, the background will treat all + * memory as IO memory. The access to IO memory will bypass + * the cache, even you have enabled the C bit in SCTLR. + * To avoid this implicit behavior, we disable cache with MPU + * explicitly to tell user that when MPU is disabled, the memory + * access is uncacheable. + */ +ENTRY(enable_mm) + mrs x0, SCTLR_EL2 + mov x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C) + /* Enable EL2 MPU and D-cache */ + orr x0, x0, x1 + dsb sy + msr SCTLR_EL2, x0 + isb + ret +ENDPROC(enable_mm) + +/* Disable MPU system, including data cache. */ +ENTRY(disable_mm) + mrs x0, SCTLR_EL2 + mov x1, #~(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C) + /* Disable EL2 MPU and D-cache */ + and x0, x0, x1 + dsb sy + msr SCTLR_EL2, x0 + isb + ret +ENDPROC(disable_mm)
Signature · 1 line
2.25.1
PATCH v6 9/11 xen/arm64: create boot-time MPU protection regions
In reply to Wei ChenFrom: Penny Zheng <[email protected]>
Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.
This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11. So we have to encode these Armv8-R MPU system registers in header file explicitly.
As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name prepare_early_mappings to replace create_page_tables for MMU and MPU.
--- xen/arch/arm/arm64/Makefile | 2 + xen/arch/arm/arm64/head.S | 13 ++-- xen/arch/arm/arm64/head_mmu.S | 4 +- xen/arch/arm/arm64/head_mpu.S | 70 +++++++++++++++++++ xen/arch/arm/include/asm/arm64/mpu.h | 13 ++++ xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++ 6 files changed, 185 insertions(+), 6 deletions(-) create mode 100644 xen/arch/arm/arm64/head_mpu.S create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
xen/arch/arm/arm64/Makefile +2 -0
@@ -10,6 +10,8 @@ obj-y += entry.o obj-y += head.o ifneq ($(CONFIG_HAS_MPU),y) obj-y += head_mmu.o +else +obj-y += head_mpu.o endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.o
xen/arch/arm/arm64/head.S +9 -4
@@ -79,12 +79,12 @@ * --------------------------- * * The requirements are: - * MMU = off, D-cache = off, I-cache = on or off, + * MMU/MPU = off, D-cache = off, I-cache = on or off, * x0 = physical address to the FDT blob. * * This must be the very first address in the loaded image. * It should be linked at XEN_VIRT_START, and loaded at any - * 4K-aligned address. All of text+data+bss must fit in 2MB, + * 4K-aligned address. All of text+data+bss must fit in 2MB, * or the initial pagetable code below will need adjustment. */
@@ -249,7 +249,12 @@ real_start_efi: bl check_cpu_mode bl cpu_init - bl create_page_tables + + /* + * Create boot memory management data, pagetable for MMU systems + * and protection regions for MPU systems. + */ + bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
@@ -307,7 +312,7 @@ GLOBAL(init_secondary) #endif bl check_cpu_mode bl cpu_init - bl create_page_tables + bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
xen/arch/arm/arm64/head_mmu.S +2 -2
@@ -123,7 +123,7 @@ * * Clobbers x0 - x4 */ -ENTRY(create_page_tables) +ENTRY(prepare_early_mappings) /* Prepare the page-tables for mapping Xen */ ldr x0, =XEN_VIRT_START create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
@@ -208,7 +208,7 @@ virtphys_clash: /* Identity map clashes with boot_third, which we cannot handle yet */ PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n") b fail -ENDPROC(create_page_tables) +ENDPROC(prepare_early_mappings) /* * Turn on the Data Cache and the MMU. The function will return on the 1:1
xen/arch/arm/arm64/head_mpu.S new file +70 -0
70 lines not on this page · raw message
xen/arch/arm/include/asm/arm64/mpu.h new file +13 -0
@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* mpu.h: Arm Memory Protection Unit definitions. +*/ + +#ifndef __ARM64_MPU_H__ +#define __ARM64_MPU_H__ + +#define MPU_REGION_SHIFT 6 +#define MPU_REGION_ALIGN (_AC(1, UL) << MPU_REGION_SHIFT) +#define MPU_REGION_MASK (~(MPU_REGION_ALIGN - 1)) + +#endif /* __ARM64_MPU_H__ */
xen/arch/arm/include/asm/arm64/sysregs.h +89 -0
95 lines not on this page · raw message
Signature · 1 line
2.25.1
Sorry for the wrong patch version. This is the first version of this series.
Cheers,
Wei Chen
Quoted 96 lines
-----Original Message-----
From: Wei Chen <[email protected]>
Sent: 2022年11月4日 18:08
To: [email protected]
Cc: nd <[email protected]>; Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)
Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.We will split these patches to several parts, this series is the part#1, the full PoC can be found in [3]. More software for Armv8-R64 can be found in [4];
[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-
05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-
/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/Penny Zheng (3):
xen/arm64: create boot-time MPU protection regions xen/arm64: introduce helpers for MPU enable/disable xen/arm64: addsetup_fixmapandremove_identity_mappingfor MPUWei Chen (8):
xen/arm: removexen_phys_startandxenheap_phys_endfromconfig.hxen/arm: add iounmap after initrd has been loaded indomain_buildxen/arm: disable EFI boot services for MPU systems xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA xen/arm: define Xen start address for FVP BaseR platform xen/arm: split MMU and MPU config files fromconfig.hxen/arm: implementFIXMAP_ADDRfor MPU systems xen/arm64: move MMU related code fromhead.Stohead_mmu.Sxen/arch/arm/Kconfig | 15 +- xen/arch/arm/arm64/Makefile | 5 + xen/arch/arm/arm64/head.S | 429 ++---------------- xen/arch/arm/arm64/head_mmu.S | 364 +++++++++++++++ xen/arch/arm/arm64/head_mpu.S | 154 +++++++ xen/arch/arm/domain_build.c | 2 + xen/arch/arm/include/asm/arm64/flushtlb.h | 25 + xen/arch/arm/include/asm/arm64/macros.h | 52 ++- xen/arch/arm/include/asm/arm64/mpu.h | 13 + xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++ xen/arch/arm/include/asm/config.h | 99 +--- xen/arch/arm/include/asm/config_mmu.h | 119 +++++ xen/arch/arm/include/asm/config_mpu.h | 29 ++ xen/arch/arm/include/asm/fixmap.h | 25 + xen/arch/arm/include/asm/flushtlb.h | 22 + .../arm/include/asm/platforms/fvp_baser.h | 18 + xen/arch/arm/platforms/Kconfig | 16 +- 17 files changed, 976 insertions(+), 500 deletions(-) create mode 100644 xen/arch/arm/arm64/head_mmu.S create mode 100644 xen/arch/arm/arm64/head_mpu.S create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h create mode 100644 xen/arch/arm/include/asm/config_mmu.h create mode 100644 xen/arch/arm/include/asm/config_mpu.h create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h--
2.25.1
Re: [PATCH v6 01/11] xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
In reply to Wei ChenHi,
Quoted 1 line · Wei Chen
These two variables are stale variables, they only have declarations
AFAICT, this has always been the case. I am guessing this was because the header was mostly likely copied from x86...
Quoted 4 lines
in
config.h, they don't have any definition and no any code is using these two variables. So in this patch, we remove them fromconfig.h.Signed-off-by Wei Chen <[email protected]>
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Wei Chen(+ Henry)
Hi,
Quoted 1 line · Wei Chen
domain_builduseioremap_wcto map a new non-cacheable virtual
s/use/uses/
Quoted 6 lines
address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.
So in this patch, we add an iounmap to the end of
domain_build, after Xen loaded initrd to guest memory.
Please a fixes tag. The issue was introduced by commit bb7e6d565d92.
Quoted 10 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/domain_build.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/domain_build.c +0
-0
xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422
Quoted 5 lines
xen/arch/arm/domain_build.c +2
-0@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo) initrd, len);if ( res !=0 )panic ("Unable to copy the initrd in the hwdom memory\n" ); + +iounmap (initrd);
This looks good to me. But I am wondering whether using ioremap_wc() is actually correct because we are reading the region. So it seems strang to map it with write-combine.
So I would consider to use ioremap_cache(). That said, this would be a separate patch.
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).
Cheers,
Signature · 1 line
Julien Grall
Hi Wei,
Quoted 24 lines · Wei Chen
The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)
Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
Quoted 6 lines
These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems
In reply to Wei ChenHi Wei,
Quoted 4 lines · Wei Chen
Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.
I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?
Quoted 15 lines
But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/arm64/head.S | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-)xen/arch/arm/Kconfig +0
-0
xen/arch/arm/Kconfig config ARM_64 · line 10
Quoted 5 lines
xen/arch/arm/Kconfig +1
-1@@ -7,7 +7,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT - select ARM_EFI + select ARM_EFI if !HAS_MPU
I think it would make sense to allow ARM_EFI to be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.
Quoted 21 lines
xen/arch/arm/Kconfig +0
-0select HAS_FAST_MULTIPLY config ARMxen/arch/arm/arm64/head.S +4
-2@@ -172,8 +172,10 @@ efi_head: .byte 0x52 .byte 0x4d .byte 0x64 - .long pe_header - efi_head /* Offset to the PE header. */ - +#ifndef CONFIG_ARM_EFI + .long 0 /* 0 means no PE header. */ +#else + .long pe_header - efi_head /* Offset to the PE header. */ /* * Add the PE/COFF header to the file. The address of this header * is at offset 0x3c in the file, and is part of Linux "Image"
xen/arch/arm/arm64/head.S section_table: · line 288
Quoted 8 lines
xen/arch/arm/arm64/head.S +2
-0@@ -279,6 +281,8 @@ section_table: .short 0 /* NumberOfLineNumbers (0 for executables) */ .long 0xe0500020 /* Characteristics (section flags) */ .align 5 +#endif /* CONFIG_ARM_EFI */ + real_start: /* BSS should be zeroed when booting without EFI */ mov x26, #0 /* x26 := skip_zero_bss */
Shouldn't the function efi_xen_start be stubbed as well?
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems
Quoted 56 lines · Julien Grall
Hi Wei,
Quoted 4 lines · Wei Chen
Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.
I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?
Quoted 15 lines
But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.
Signed-off-by Wei Chen <[email protected]>---
xen/arch/arm/Kconfig | 2 +-
xen/arch/arm/arm64/head.S| 8 ++++++--2 files changed, 7 insertions(+), 3 deletions(-)xen/arch/arm/Kconfig +0
-0@@ -7,7 +7,7 @@ config ARM_64Quoted 5 lines
def_booly
depends on !ARM_32
select 64BIT
- selectARM_EFI
+ selectARM_EFIif !HAS_MPUI think it would make sense to allow
ARM_EFIto be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.Quoted 23 lines
select
HAS_FAST_MULTIPLY
config ARMxen/arch/arm/arm64/head.S +0
-0@@ -172,8 +172,10 @@ efi_head:.byte
0x52
.byte0x4d
.byte0x64
- .longpe_header- efi_head /* Offset to the PE header. */xen/arch/arm/arm64/head.S +4
-1- +#ifndef CONFIG_ARM_EFI + .long 0 /* 0 means no PE header. */ +#else + .long pe_header - efi_head /* Offset to the PE header. *//*
* Add the PE/COFF header to the file. The address of this header
* is at offset0x3cin the file, and is part of Linux "Image"xen/arch/arm/arm64/head.S +0
-0@@ -279,6 +281,8 @@ section_table:.short 0 /* NumberOfLineNumbers (0 for executables) */ .long 0xe0500020 /* Characteristics (section flags) */
Quoted 8 lines
Quoted 6 lines
.align 5
xen/arch/arm/arm64/head.S +2
-0+#endif /* CONFIG_ARM_EFI */ +
real_start:
/* BSS should be zeroed when booting without EFI */ mov x26, #0 /* x26 :=skip_zero_bss*/Shouldn't the function
efi_xen_startbe stubbed as well?
Sorry, I mean protected rather than stubbed because there will be no user when !CONFIG_ARM_EFI.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Wei ChenQuoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...
Quoted 9 lines
xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hxen/arch/arm/Kconfig +0
-0
xen/arch/arm/Kconfig config TEE · line 143
Quoted 6 lines
xen/arch/arm/Kconfig +3
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allows guests to access real TEE via one of TEE mediators implemented in XEN. +config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform defined address" + default 0xFFFFFFFF
... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the Kconfig to select the proper address (see the config for selecting early printk address).
This will also avoids to use an invalid value here.
Quoted 36 lines
xen/arch/arm/Kconfig +8
-0+ depends on HAS_MPU + help + This option allows to set the customized address at which Xen will be + linked on MPU systems. This address must be aligned to a page size. + Use 0xFFFFFFFF as the default value to indicate that user hasn't + customized this address, and Xen use use the default value that has + been defined in platform files. + source "arch/arm/tee/Kconfig" config STATIC_SHMxen/arch/arm/include/asm/platforms/fvp_baser.h new file +14
-0@@ -0,0 +1,14 @@ +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__ +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__ + +/* +* 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS, +* we will use platform defined default address. +*/ +#if CONFIG_XEN_START_ADDRESS ==0xFFFFFFFF +#define XEN_START_ADDRESS 0x200000 +#else +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS +#endif + +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */ xen/arch/arm/platforms/Kconfig +0
-0
xen/arch/arm/platforms/Kconfig line 4
Quoted 4 lines
xen/arch/arm/platforms/Kconfig +1
-0@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8R
Is there any reason to create a new Kconfig rather than using MPU?
Quoted 42 lines
xen/arch/arm/platforms/Kconfig +7
-3---help--- Choose which hardware platform to enable in Xen.@@ -8,13 +9,14 @@ choice config ALL_PLAT bool "All Platforms" + depends on !ARM_V8R ---help--- Enable support for all available hardware platforms. It doesn't automatically select any of the related drivers. config QEMU bool "QEMU aarch virt machine support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select GICV3 select HAS_PL011 ---help---@@ -23,7 +25,7 @@ config QEMU config RCAR3 bool "Renesas RCar3 support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_SCIF select IPMMU_VMSA ---help---@@ -31,14 +33,22 @@ config RCAR3 config MPSOC bool "Xilinx Ultrascale+ MPSoC support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_CADENCE_UART select ARM_SMMU ---help--- Enable all the required drivers for Xilinx Ultrascale+ MPSoC +config FVP_BASER + bool "Fixed Virtual Platform BaseR support" + depends on ARM_V8R
xen/arch/arm/platforms/Kconfig config RCAR3 · line 54
Quoted 10 lines
xen/arch/arm/platforms/Kconfig +5
-0+ help + Enable platform specific configurations for Fixed Virtual + Platform BaseR + config NO_PLAT bool "No Platforms" + depends on !ARM_V8R ---help--- Do not enable specific support for any platform.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems
In reply to Wei ChenHi Wei,
Quoted 6 lines · Wei Chen
FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems.
FIXMAP_ADDRwill return a virtual address by index for special purpose phys-to-virt mapping usage. For example,FIXMAP_ADDR(FIXMAP_CONSOLE)for early console mapping andFIXMAP_ADDR(FIXMAP_MISC)forcopy_from_paddr.
To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.
Quoted 26 lines
But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using
FIXMAP_ADDRto return the input physical address in MPU systems. So in MPU version,FIXMAP_ADDRwill trim physical address to PAGE alignment. This will return an offset which is similar to MMU versionFIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask withPAGE_MASK). The caller can return the input physical address directly.As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/include/asm/config_mpu.h | 2 ++ xen/arch/arm/include/asm/fixmap.h | 25 +++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-)xen/arch/arm/Kconfig +0
-0
xen/arch/arm/Kconfig config ARM · line 19
Quoted 5 lines
xen/arch/arm/Kconfig +1
-1@@ -16,7 +16,7 @@ config ARM select HAS_DEVICE_TREE select HAS_PASSTHROUGH select HAS_PDX - select HAS_PMAP + select HAS_PMAP if !HAS_MPU
I can't find any change of mm.c in this series. So surely this will break the build?
Quoted 45 lines
xen/arch/arm/Kconfig +0
-0select IOMMU_FORCE_PT_SHARE config ARCH_DEFCONFIGxen/arch/arm/include/asm/config_mpu.h +2
-0@@ -24,4 +24,6 @@#define HYPERVISOR_VIRT_START XEN_VIRT_START +#define FIXMAP_ADDR (n) (_AT(paddr_t, n) & (PAGE_MASK)) +#endif /* __ARM_CONFIG_MPU_H__ */ xen/arch/arm/include/asm/fixmap.h +16
-0@@ -7,6 +7,8 @@#include <xen/acpi.h> #include <xen/pmap.h> +#ifndef CONFIG_HAS_MPU +/* Fixmap slots */ #define FIXMAP_CONSOLE 0/* The primary UART */ #define FIXMAP_MISC 1/* Ephemeral mappings of hardware */ @@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)#endif /* __ASSEMBLY__ */ +#else + +/* +* FIXMAP_ADDR will trim physical address to PAGE alignment. +* This will return an offset which is similar to MMU version +* FIXMAP_ADDR. +* For example: +* EARLY_UART_VIRTUAL_ADDRESS is defined by: +* (FIXMAP_ADDR(FIXMAP_CONSOLE) + \ +* (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK)) +* With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions, +* EARLY_UART_VIRTUAL_ADDRESS can be restore to +* CONFIG_EARLY_UART_BASE_ADDRESS. +* In this case, we don't need to use #ifdef MPU in the code
xen/arch/arm/include/asm/fixmap.h static inline unsigned int virt_to_fix(vaddr_t vaddr) · line 73
Quoted 10 lines
xen/arch/arm/include/asm/fixmap.h +9
-0+* where are using FIXMAP_ADDR to make them to use physical +* address explicitily. +*/ +#ifdef CONFIG_EARLY_UART_BASE_ADDRESS +#define FIXMAP_CONSOLE CONFIG_EARLY_UART_BASE_ADDRESS +#endif + +#endif /* CONFIG_HAS_MPU */ +#endif /* __ASM_FIXMAP_H */
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Wei ChenHi Wei,
Quoted 4 lines · Wei Chen
There are lots of MMU specific code in
head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code tohead_mmu.S, and keep common code still inhead.S.
I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.
This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.
Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.
Quoted 8 lines
As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros:adr_landload_paddrwill be used by MMU and MPU later, so we move them tomacros.h.Signed-off-by Henry Wang <[email protected]>Signed-off-by Wei Chen <[email protected]>
In general, the first signed-off should match the author. So who is who here?
Quoted 51 lines
--- xen/arch/arm/arm64/Makefile | 3 + xen/arch/arm/arm64/head.S | 407 +----------------------- xen/arch/arm/arm64/head_mmu.S | 364 +++++++++++++++++++++ xen/arch/arm/include/asm/arm64/macros.h | 52 ++- 4 files changed, 432 insertions(+), 394 deletions(-)create mode 100644
xen/arch/arm/arm64/head_mmu.Sxen/arch/arm/arm64/Makefile +2
-0@@ -8,6 +8,9 @@ obj-y += domctl.o obj-y += domain.o obj-y += entry.o obj-y += head.o +ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o +endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += smc.oxen/arch/arm/arm64/head.S +0
-15@@ -25,17 +25,6 @@ #include <efi/efierr.h> #include <asm/arm64/efibind.h> -#define PT_PT 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */ -#define PT_MEM 0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */ -#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */ -#define PT_DEV 0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */ -#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */ - -/* Convenience defines to get slot used by Xen mapping. */ -#define XEN_ZEROETH_SLOT zeroeth_table_offset(XEN_VIRT_START) -#define XEN_FIRST_SLOT first_table_offset(XEN_VIRT_START) -#define XEN_SECOND_SLOT second_table_offset(XEN_VIRT_START) - #define __HEAD_FLAG_PAGE_SIZE ((PAGE_SHIFT - 10) / 2) #define __HEAD_FLAG_PHYS_BASE 1@@ -82,73 +71,22 @@ * x30 - lr */ -#ifdef CONFIG_EARLY_PRINTK -/* - * Macro to print a string to the UART, if there is one. - *
xen/arch/arm/arm64/head.S line 75
Quoted 10 lines
xen/arch/arm/arm64/head.S +2
-8- * Clobbers x0 - x3 - */ -#define PRINT(_s) \ - mov x3, lr ; \ - adr x0, 98f ; \ - bl puts ; \ - mov lr, x3 ; \ - RODATA_STR(98, _s) +.section .text.header, "ax", %progbits +/*.aarch64*/
The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.
Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).
xen/arch/arm/arm64/head.S line 79
Quoted 5 lines
xen/arch/arm/arm64/head.S +2
-1/* - * Macro to print the value of register \xb + * Kernel startup entry point. + * ---------------------------
Same here about the indentation. I will not comment everywhere where the indentation was changed. So please look at it.
[...]
Quoted 32 lines
xen/arch/arm/arm64/head.S +0
-28-/* - * Map the UART in the fixmap (when earlyprintk is used) and hook the - * fixmap table in the page tables. - * - * The fixmap cannot be mapped in create_page_tables because it may - * clash with the 1:1 mapping. - * - * Inputs: - * x20: Physical offset - * x23: Early UART base physical address - * - * Clobbers x0 - x3 - */ -setup_fixmap: -#ifdef CONFIG_EARLY_PRINTK - /* Add UART to the fixmap table */ - ldr x0, =EARLY_UART_VIRTUAL_ADDRESS - create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3, type=PT_DEV_L3 -#endif - /* Map fixmap into boot_second */ - ldr x0, =FIXMAP_ADDR(0) - create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3 - /* Ensure any page table updates made above have occurred. */ - dsb nshst - - ret -ENDPROC(setup_fixmap) - /* * Setup the initial stack and jump to the C world *
xen/arch/arm/arm64/head.S launch: · line 461
Quoted 5 lines
xen/arch/arm/arm64/head.S +1
-1@@ -810,41 +458,14 @@ launch: ENDPROC(launch) /* Fail-stop */ -fail: PRINT("- Boot failed -\r\n") +ENTRY(fail)
This name is a bit too generic to be exposed. But it would be better to duplicate it.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions
In reply to Wei ChenHi Wei,
Quoted 11 lines · Wei Chen
From: Penny Zheng <[email protected]>
Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.
This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11. So we have to encode these Armv8-R MPU system registers in header file explicitly.
Quoted 7 lines
As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name
prepare_early_mappingsto replacecreate_page_tablesfor MMU and MPU.Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>
If Penny is the original author, then her signed-off-by should be first.
Quoted 30 lines
--- xen/arch/arm/arm64/Makefile | 2 + xen/arch/arm/arm64/head.S | 13 ++-- xen/arch/arm/arm64/head_mmu.S | 4 +- xen/arch/arm/arm64/head_mpu.S | 70 +++++++++++++++++++ xen/arch/arm/include/asm/arm64/mpu.h | 13 ++++ xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++ 6 files changed, 185 insertions(+), 6 deletions(-)create mode 100644
xen/arch/arm/arm64/head_mpu.Screate mode 100644xen/arch/arm/include/asm/arm64/mpu.hxen/arch/arm/arm64/Makefile +2
-0@@ -10,6 +10,8 @@ obj-y += entry.o obj-y += head.o ifneq ($(CONFIG_HAS_MPU),y) obj-y += head_mmu.o +else +obj-y += head_mpu.o endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.oxen/arch/arm/arm64/head.S +0
-0@@ -79,12 +79,12 @@ * ---------------------------
xen/arch/arm/arm64/head.S line 87
Quoted 10 lines
xen/arch/arm/arm64/head.S +2
-2* * The requirements are: - * MMU = off, D-cache = off, I-cache = on or off, + * MMU/MPU = off, D-cache = off, I-cache = on or off, * x0 = physical address to the FDT blob. * * This must be the very first address in the loaded image. * It should be linked at XEN_VIRT_START, and loaded at any - * 4K-aligned address. All of text+data+bss must fit in 2MB, + * 4K-aligned address. All of text+data+bss must fit in 2MB,
The double space after the final point was valid. This is fairly common to use it and this is a spurious change.
xen/arch/arm/arm64/head.S real_start_efi: · line 256
Quoted 13 lines
xen/arch/arm/arm64/head.S +5
-1* or the initial pagetable code below will need adjustment. */@@ -249,7 +249,12 @@ real_start_efi: bl check_cpu_mode bl cpu_init - bl create_page_tables + + /* + * Create boot memory management data, pagetable for MMU systems + * and protection regions for MPU systems. + */
head.S is now meant to be generic. So I would prefer if we keep comment as generic as possible. IOW, anything after the first comma should be dropped.
Quoted 41 lines
xen/arch/arm/arm64/head.S +2
-1+ bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */@@ -307,7 +312,7 @@ GLOBAL(init_secondary) #endif bl check_cpu_mode bl cpu_init - bl create_page_tables + bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */xen/arch/arm/arm64/head_mmu.S +2
-2@@ -123,7 +123,7 @@ * * Clobbers x0 - x4 */ -ENTRY(create_page_tables) +ENTRY(prepare_early_mappings) /* Prepare the page-tables for mapping Xen */ ldr x0, =XEN_VIRT_START create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3@@ -208,7 +208,7 @@ virtphys_clash: /* Identity map clashes with boot_third, which we cannot handle yet */ PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n") b fail -ENDPROC(create_page_tables) +ENDPROC(prepare_early_mappings) /* * Turn on the Data Cache and the MMU. The function will return on the 1:1xen/arch/arm/arm64/head_mpu.S new file +0
-0
xen/arch/arm/arm64/head_mpu.S line 1
Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1
-0@@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0-only
Coding style:
/* SPDX ... */
xen/arch/arm/arm64/head_mpu.S line 8
Quoted 7 lines
xen/arch/arm/arm64/head_mpu.S new file +7
-0+/* + * Start-of-day code for an Armv8-R MPU system. + */ + +#include <asm/arm64/mpu.h> +#include <asm/page.h> +#include <asm/early_printk.h>
Headers should be included in alphabetical order.
xen/arch/arm/arm64/head_mpu.S line 18
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10
-0+ +/* + * From the requirements of head.S we know that Xen image should + * be linked at XEN_START_ADDRESS, and all of text + data + bss + * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the + * address that Xen image should be loaded at. So for initial MPU + * regions setup, we use 2MB for Xen data memory to setup boot + * region, or the create boot regions code below will need adjustment. + */ +#define XEN_START_MEM_SIZE 0x200000
It sounds like something that should be defined in the header. Also, I think the size should be common between MPU and MMU.
In [1], I was going to name it XEN_VIRT_SIZE. I would be OK to remove "VIRT" in the name.
xen/arch/arm/arm64/head_mpu.S line 23
Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5
-0+ +/* + * In boot stage, we will use 1 MPU region: + * Region#0: Normal memory for Xen text + data + bss (2MB) + */
Are we only going to modify the MPU in head.S? If not, then I would define the layout in config_mpu.h so there are a single point where you can read how this works.
xen/arch/arm/arm64/head_mpu.S line 27
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+#define BOOT_NORMAL_REGION_IDX 0x0 + +/* MPU normal memory attributes. */ +#define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */
IIUC, this means that Xen will be mapped write-executable. Is this going to be forever? If not, when can't we already mapped Xen properly?
xen/arch/arm/arm64/head_mpu.S line 28
Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1
-0+#define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */
To me, it feels like this should be fined outside of head.S because this could be re-used by other part of Xen.
xen/arch/arm/arm64/head_mpu.S line 32
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+ +.macro write_pr, sel, prbar, prlar + msr PRSELR_EL2, \sel + dsb sy
Is it really necessary to use "sy" here? Also, it would be good to explain the logic. I.e. why do you need two dsb but only one isb?
In fact, I was expecting an "isb" here than "dsb" to wait for the completion of the instruction.
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+ msr PRBAR_EL2, \prbar + msr PRLAR_EL2, \prlar + dsb sy + isb
xen/arch/arm/arm64/head_mpu.S line 46
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10
-0+.endm + +.section .text.header, "ax", %progbits + +/* + * Static start-of-day EL2 MPU memory layout. + * + * It has a very simple structure, including: + * - 2MB normal memory mappings of xen at XEN_START_ADDRESS, which + * is the address where Xen was loaded by the bootloader.
Missing details on the clobberred registers.
xen/arch/arm/arm64/head_mpu.S line 54
Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8
-0+ */ +ENTRY(prepare_early_mappings) + /* Map Xen start memory to a normal memory region. */ + mov x0, #BOOT_NORMAL_REGION_IDX + ldr x1, =XEN_START_ADDRESS + and x1, x1, #MPU_REGION_MASK + mov x3, #PRBAR_NORMAL_MEM + orr x1, x1, x3
It looks like to me there are a potential for a macro to compute the register.
xen/arch/arm/arm64/head_mpu.S line 57
Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3
-0+ + ldr x2, =XEN_START_ADDRESS + mov x3, #(XEN_START_MEM_SIZE - 1)
XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.
xen/arch/arm/arm64/head_mpu.S line 65
Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8
-0+ add x2, x2, x3 + and x2, x2, #MPU_REGION_MASK + mov x3, #PRLAR_NORMAL_MEM + orr x2, x2, x3 + + /* + * Write to MPU protection region: + * x0 for pr_sel, x1 for prbar x2 for prlar
This is not a very useful comment because this can be inferred from the prototype of write_pr. What would be more interesting is to explain the logic within this function in the same way we do in head.S and head_mmu.S.
xen/arch/arm/include/asm/arm64/sysregs.h
Quoted 28 lines
xen/arch/arm/arm64/head_mpu.S new file +5
-0+ */ + write_pr x0, x1, x2 + + ret +ENDPROC(prepare_early_mappings)xen/arch/arm/include/asm/arm64/mpu.h new file +13
-0@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* mpu.h: Arm Memory Protection Unit definitions. +*/ + +#ifndef __ARM64_MPU_H__ +#define __ARM64_MPU_H__ + +#define MPU_REGION_SHIFT 6 +#define MPU_REGION_ALIGN (_AC(1, UL) << MPU_REGION_SHIFT) +#define MPU_REGION_MASK (~(MPU_REGION_ALIGN - 1)) + +#endif /* __ARM64_MPU_H__ */ xen/arch/arm/include/asm/arm64/sysregs.h +0
-0
In the context of this patch, it would be better to only define the registers you need. If you want to define all of them, then please define them in a separate patch before this one.
xen/arch/arm/include/asm/arm64/sysregs.h line 462
Quoted 6 lines
xen/arch/arm/include/asm/arm64/sysregs.h +2
-0@@ -458,6 +458,95 @@#define ZCR_ELx_LEN_SIZE 9#define ZCR_ELx_LEN_MASK 0x1ff +/* System registers for AArch64 with PMSA */ +#ifdef CONFIG_HAS_MPU
The #ifdef here seems unnecessary.
Quoted 80 lines
xen/arch/arm/include/asm/arm64/sysregs.h +80
-0+ +/* EL1 MPU Protection Region Base Address Register encode */ +#define PRBAR_EL1 S3_0_C6_C8_0 +#define PRBAR1_EL1 S3_0_C6_C8_4 +#define PRBAR2_EL1 S3_0_C6_C9_0 +#define PRBAR3_EL1 S3_0_C6_C9_4 +#define PRBAR4_EL1 S3_0_C6_C10_0 +#define PRBAR5_EL1 S3_0_C6_C10_4 +#define PRBAR6_EL1 S3_0_C6_C11_0 +#define PRBAR7_EL1 S3_0_C6_C11_4 +#define PRBAR8_EL1 S3_0_C6_C12_0 +#define PRBAR9_EL1 S3_0_C6_C12_4 +#define PRBAR10_EL1 S3_0_C6_C13_0 +#define PRBAR11_EL1 S3_0_C6_C13_4 +#define PRBAR12_EL1 S3_0_C6_C14_0 +#define PRBAR13_EL1 S3_0_C6_C14_4 +#define PRBAR14_EL1 S3_0_C6_C15_0 +#define PRBAR15_EL1 S3_0_C6_C15_4 + +/* EL1 MPU Protection Region Limit Address Register encode */ +#define PRLAR_EL1 S3_0_C6_C8_1 +#define PRLAR1_EL1 S3_0_C6_C8_5 +#define PRLAR2_EL1 S3_0_C6_C9_1 +#define PRLAR3_EL1 S3_0_C6_C9_5 +#define PRLAR4_EL1 S3_0_C6_C10_1 +#define PRLAR5_EL1 S3_0_C6_C10_5 +#define PRLAR6_EL1 S3_0_C6_C11_1 +#define PRLAR7_EL1 S3_0_C6_C11_5 +#define PRLAR8_EL1 S3_0_C6_C12_1 +#define PRLAR9_EL1 S3_0_C6_C12_5 +#define PRLAR10_EL1 S3_0_C6_C13_1 +#define PRLAR11_EL1 S3_0_C6_C13_5 +#define PRLAR12_EL1 S3_0_C6_C14_1 +#define PRLAR13_EL1 S3_0_C6_C14_5 +#define PRLAR14_EL1 S3_0_C6_C15_1 +#define PRLAR15_EL1 S3_0_C6_C15_5 + +/* EL2 MPU Protection Region Base Address Register encode */ +#define PRBAR_EL2 S3_4_C6_C8_0 +#define PRBAR1_EL2 S3_4_C6_C8_4 +#define PRBAR2_EL2 S3_4_C6_C9_0 +#define PRBAR3_EL2 S3_4_C6_C9_4 +#define PRBAR4_EL2 S3_4_C6_C10_0 +#define PRBAR5_EL2 S3_4_C6_C10_4 +#define PRBAR6_EL2 S3_4_C6_C11_0 +#define PRBAR7_EL2 S3_4_C6_C11_4 +#define PRBAR8_EL2 S3_4_C6_C12_0 +#define PRBAR9_EL2 S3_4_C6_C12_4 +#define PRBAR10_EL2 S3_4_C6_C13_0 +#define PRBAR11_EL2 S3_4_C6_C13_4 +#define PRBAR12_EL2 S3_4_C6_C14_0 +#define PRBAR13_EL2 S3_4_C6_C14_4 +#define PRBAR14_EL2 S3_4_C6_C15_0 +#define PRBAR15_EL2 S3_4_C6_C15_4 + +/* EL2 MPU Protection Region Limit Address Register encode */ +#define PRLAR_EL2 S3_4_C6_C8_1 +#define PRLAR1_EL2 S3_4_C6_C8_5 +#define PRLAR2_EL2 S3_4_C6_C9_1 +#define PRLAR3_EL2 S3_4_C6_C9_5 +#define PRLAR4_EL2 S3_4_C6_C10_1 +#define PRLAR5_EL2 S3_4_C6_C10_5 +#define PRLAR6_EL2 S3_4_C6_C11_1 +#define PRLAR7_EL2 S3_4_C6_C11_5 +#define PRLAR8_EL2 S3_4_C6_C12_1 +#define PRLAR9_EL2 S3_4_C6_C12_5 +#define PRLAR10_EL2 S3_4_C6_C13_1 +#define PRLAR11_EL2 S3_4_C6_C13_5 +#define PRLAR12_EL2 S3_4_C6_C14_1 +#define PRLAR13_EL2 S3_4_C6_C14_5 +#define PRLAR14_EL2 S3_4_C6_C15_1 +#define PRLAR15_EL2 S3_4_C6_C15_5 + +/* MPU Protection Region Enable Register encode */ +#define PRENR_EL1 S3_0_C6_C1_1 +#define PRENR_EL2 S3_4_C6_C1_1 + +/* MPU Protection Region Selection Register encode */ +#define PRSELR_EL1 S3_0_C6_C2_1 +#define PRSELR_EL2 S3_4_C6_C2_1
xen/arch/arm/include/asm/arm64/sysregs.h line 552
Quoted 10 lines
xen/arch/arm/include/asm/arm64/sysregs.h +7
-0+ +/* MPU Type registers encode */ +#define MPUIR_EL1 S3_0_C0_C0_4 +#define MPUIR_EL2 S3_4_C0_C0_4 + +#endif +/* Access to system registers */ #define WRITE_SYSREG64 (v, name) do { \
Cheers,
[1] https://lore.kernel.org/all/[email protected]/
Signature · 1 line
Julien Grall
Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable
In reply to Wei ChenHi Wei,
Quoted 12 lines · Wei Chen
From: Penny Zheng <[email protected]>
We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.
This raises the following question: Why can't we create the split regions right now?
In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).
Quoted 79 lines
So in this situation, we need to disable the whole MPU to update the protection regions.
In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent withenable_mmu. For MMU systems,enable_mmuwill turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.In this patch, we also introduce a neutral name
enable_mmfor Xen to enable MMU/MPU. This can help us to keep one code flow inhead.SSigned-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>--- xen/arch/arm/arm64/head.S | 5 +++-- xen/arch/arm/arm64/head_mmu.S | 4 ++-- xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +3
-2@@ -255,7 +255,8 @@ real_start_efi: * and protection regions for MPU systems. */ bl prepare_early_mappings - bl enable_mmu + /* Turn on MMU or MPU */ + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */ ldr x0, =primary_switched@@ -313,7 +314,7 @@ GLOBAL(init_secondary) bl check_cpu_mode bl cpu_init bl prepare_early_mappings - bl enable_mmu + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */ ldr x0, =secondary_switchedxen/arch/arm/arm64/head_mmu.S +2
-2@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings) * * Clobbers x0 - x3 */ -ENTRY(enable_mmu) +ENTRY(enable_mm) PRINT("- Turning on paging -\r\n") /*@@ -239,7 +239,7 @@ ENTRY(enable_mmu) msr SCTLR_EL2, x0 /* now paging is enabled */ isb /* Now, flush the icache */ ret -ENDPROC(enable_mmu) +ENDPROC(enable_mm) /* * Remove the 1:1 map from the page-tables. It is not easy to keep trackxen/arch/arm/arm64/head_mpu.S +0
-0@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 78
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8
-0ret ENDPROC(prepare_early_mappings) + +/* + * Enable EL2 MPU and data cache. Because we will disable cache + * with MPU at the same time, in accordance with that, we have + * to enable cache with MPU at the same time in this function. + * When MPU is disabled, the MPU background attributes will + * be used. On some platform, the background will treat all + * memory as IO memory.
I was under the impression that all access would be treated as Device Memory when the MMU is off. Isn't it the case for the MPU?
Also, I think the correct wording is "device memory" rather than "IO memory".
Quoted 1 line
The access to IO memory will bypass
Ditto.
Quoted 8 lines
+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */xen/arch/arm/arm64/head_mpu.S +3
-0+ENTRY(enable_mm) + mrs x0, SCTLR_EL2 + mov x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 96
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+ /* Enable EL2 MPU and D-cache */ + orr x0, x0, x1 + dsb sy + msr SCTLR_EL2, x0 + isb + ret +ENDPROC(enable_mm) + +/* Disable MPU system, including data cache. */ +ENTRY(disable_mm)
I would rather not introduce this function until there is a caller. This is because, I believe, there are some assumptions on the state of the cache before we can turn off the MMU. So I would like to see the caller in order to assess whether this function makes sense.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
In reply to Wei ChenHi,
Quoted 51 lines · Wei Chen
From: Penny Zheng <[email protected]>
setup_fixmapandremove_identity_mappingare two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow inhead.Sdoesn't need to use #ifdef to gate MPU/MMU code.In MMU system,
setup_fixmapis used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but beforesetup_fixmap. This restriction is the same as MMU system.For
remove_identity_mapping, we just need an empty function to makehead.Scode flow happy.Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>--- xen/arch/arm/arm64/head_mpu.S | 49 +++++++++++++++++++ .../arm/include/asm/platforms/fvp_baser.h | 4 ++ 2 files changed, 53 insertions(+)xen/arch/arm/arm64/head_mpu.S +7
-0@@ -20,13 +20,20 @@ /* * In boot stage, we will use 1 MPU region: * Region#0: Normal memory for Xen text + data + bss (2MB) + * Region#1: Device memory for EARLY UART, size is defined + * by platform's EARLY_UART_SIZE */ #define BOOT_NORMAL_REGION_IDX 0x0 +#define BOOT_DEVICE_REGION_IDX 0x1 /* MPU normal memory attributes. */ #define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */ #define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */ +/* MPU device memory attributes. */ +#define PRBAR_DEVICE_MEM 0x20 /* SH=10 AP=00 XN=00 */ +#define PRLAR_DEVICE_MEM 0x09 /* NS=0 ATTR=100 EN=1 */ + .macro write_pr, sel, prbar, prlar msr PRSELR_EL2, \sel dsb sy@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings) ret
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8
-0ENDPROC(prepare_early_mappings) +/* + * In MMU system, setup_fixmap is used for Xen to map some essential data + * or devices in boot-time. In order to be consistent with MMU system, we + * inherit the function name for MPU system. + * setup_fixmap of MPU system will: + * - Map the early UART to MPU protection region when earlyprintk is + * enabled (The PRINT can't be used after turning on MPU but before + * setup_fixmap).
For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.
In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.
Quoted 23 lines
xen/arch/arm/arm64/head_mpu.S +23
-0+ * + * Clobbers x0 - x3 + */ +ENTRY(setup_fixmap) +#ifdef CONFIG_EARLY_PRINTK + /* Map early uart to MPU device region for early printk. */ + mov x0, #BOOT_DEVICE_REGION_IDX + ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS + and x1, x1, #MPU_REGION_MASK + mov x3, #PRBAR_DEVICE_MEM + orr x1, x1, x3 + + ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS + ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1) + add x2, x2, x3 + and x2, x2, #MPU_REGION_MASK + mov x3, #PRLAR_DEVICE_MEM + orr x2, x2, x3 + + /* + * Write to MPU protection region: + * x0 for pr_sel, x1 for prbar x2 for prlar + */
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 119
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+ write_pr x0, x1, x2 +#endif + + ret +ENDPROC(setup_fixmap) + +/* Stub of remove_identity_mapping for MPU systems */ +ENTRY(remove_identity_mapping) + ret +ENDPROC(remove_identity_mapping)
This stub could be avoided if you move the call to remove_identity_mapping in enable_mmu() as I did for arm32. See [1].
[1] https://lore.kernel.org/all/[email protected]/
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Julien GrallHi Julien,
Quoted 8 lines
-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded indomain_build(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).
Sounds good to me, I am wondering if we need to include also patch #1 though.
If this patch wants to be in 4.17:
Kind regards,
Henry
Quoted 5 lines
Cheers,
--
Julien Grall
RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions
In reply to Julien GrallHi Julien
Quoted 179 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 4:47 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regionsHi Wei,
Quoted 11 lines · Wei Chen
From: Penny Zheng <[email protected]>
Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.
This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11.
So we have to encode these Armv8-R MPU system registers in header file explicitly.Quoted 7 lines
As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name
prepare_early_mappingsto replacecreate_page_tablesfor MMU and MPU.Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>If Penny is the original author, then her signed-off-by should be first.
Quoted 7 lines
--- xen/arch/arm/arm64/Makefile | 2 + xen/arch/arm/arm64/head.S | 13 ++-- xen/arch/arm/arm64/head_mmu.S | 4 +- xen/arch/arm/arm64/head_mpu.S | 70 +++++++++++++++++++ xen/arch/arm/include/asm/arm64/mpu.h | 13 ++++ xen/arch/arm/include/asm/arm64/sysregs.h | 89++++++++++++++++++++++++
Review
Quoted 5 lines
6 files changed, 185 insertions(+), 6 deletions(-)create mode 100644
xen/arch/arm/arm64/head_mpu.Screate mode 100644xen/arch/arm/include/asm/arm64/mpu.h(quoted diff) +0
-0b/xen/arch/arm/arm64/Makefile
Quoted 18 lines
index
22da2f54b5..438c9737ad100644xen/arch/arm/arm64/Makefile +2
-0@@ -10,6 +10,8 @@ obj-y += entry.o obj-y += head.o ifneq ($(CONFIG_HAS_MPU),y) obj-y += head_mmu.o +else +obj-y += head_mpu.o endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.o diff --git
a/xen/arch/arm/arm64/head.Sb/xen/arch/arm/arm64/head.Sindexd9a8da9120..6c1a5f74a1100644xen/arch/arm/arm64/head.S +0
-0@@ -79,12 +79,12 @@ * ---------------------------xen/arch/arm/arm64/head.S line 87
Quoted 10 lines
xen/arch/arm/arm64/head.S +2
-2* * The requirements are: - * MMU = off, D-cache = off, I-cache = on or off, + * MMU/MPU = off, D-cache = off, I-cache = on or off, * x0 = physical address to the FDT blob. * * This must be the very first address in the loaded image. * It should be linked at XEN_VIRT_START, and loaded at any - * 4K-aligned address. All of text+data+bss must fit in 2MB, + * 4K-aligned address. All of text+data+bss must fit in 2MB,The double space after the final point was valid. This is fairly common to use it and this is a spurious change.
xen/arch/arm/arm64/head.S real_start_efi: · line 254
Quoted 11 lines
xen/arch/arm/arm64/head.S +3
-1* or the initial pagetable code below will need adjustment. */@@ -249,7 +249,12 @@ real_start_efi: bl check_cpu_mode bl cpu_init - bl create_page_tables + + /* + * Create boot memory management data, pagetable for MMUsystems
xen/arch/arm/arm64/head.S real_start_efi: · line 256
Quoted 2 lines
xen/arch/arm/arm64/head.S +2
-0+ * and protection regions for MPU systems. + */
head.Sis now meant to be generic. So I would prefer if we keep comment as generic as possible. IOW, anything after the first comma should be dropped.xen/arch/arm/arm64/head.S real_start_efi:
Quoted 4 lines
xen/arch/arm/arm64/head.S +1
-0+ bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime VirtualAddress. */
xen/arch/arm/arm64/head.S GLOBAL(init_secondary) · line 318
Quoted 9 lines
xen/arch/arm/arm64/head.S +1
-1@@ -307,7 +312,7 @@ GLOBAL(init_secondary) #endif bl check_cpu_mode bl cpu_init - bl create_page_tables + bl prepare_early_mappings bl enable_mmu /* We are still in the 1:1 mapping. Jump to the runtime VirtualAddress. */
Review
Quoted 1 line
(quoted diff) +0
-0
b/xen/arch/arm/arm64/head_mmu.SQuoted 13 lines
index
1a3df81a38..fc64819a98100644xen/arch/arm/arm64/head_mmu.S +1
-1@@ -123,7 +123,7 @@ * * Clobbers x0 - x4 */ -ENTRY(create_page_tables) +ENTRY(prepare_early_mappings) /* Prepare the page-tables for mapping Xen */ ldr x0, =XEN_VIRT_START create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3xen/arch/arm/arm64/head_mmu.S virtphys_clash: · line 208
Quoted 1 line
xen/arch/arm/arm64/head_mmu.S +0
-0@@ -208,7 +208,7 @@ virtphys_clash: /* Identity map clashes with boot_third, which we cannot handle yet*/
xen/arch/arm/arm64/head_mmu.S virtphys_clash: · line 209
Quoted 1 line
xen/arch/arm/arm64/head_mmu.S +0
-0PRINT("- Unable to build boot page tables - virt and phys addressesclash. -\r\n")
xen/arch/arm/arm64/head_mmu.S virtphys_clash:
Quoted 6 lines
xen/arch/arm/arm64/head_mmu.S +1
-1b fail -ENDPROC(create_page_tables) +ENDPROC(prepare_early_mappings) /* * Turn on the Data Cache and the MMU. The function will return on the1:1
Review
Quoted 1 line
(quoted diff) +0
-0
b/xen/arch/arm/arm64/head_mpu.Sxen/arch/arm/arm64/head_mpu.S line 1
Quoted 6 lines
new file mode 100644
index 0000000000..f60611b556xen/arch/arm/arm64/head_mpu.S new file +1
-0@@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0-onlyCoding style:
/* SPDX ... */
xen/arch/arm/arm64/head_mpu.S line 8
Quoted 7 lines
xen/arch/arm/arm64/head_mpu.S new file +7
-0+/* + * Start-of-day code for an Armv8-R MPU system. + */ + +#include <asm/arm64/mpu.h> +#include <asm/page.h> +#include <asm/early_printk.h>Headers should be included in alphabetical order.
xen/arch/arm/arm64/head_mpu.S line 18
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10
-0+ +/* + * From the requirements of head.S we know that Xen image should + * be linked at XEN_START_ADDRESS, and all of text + data + bss + * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the + * address that Xen image should be loaded at. So for initial MPU + * regions setup, we use 2MB for Xen data memory to setup boot + * region, or the create boot regions code below will need adjustment. + */ +#define XEN_START_MEM_SIZE 0x200000It sounds like something that should be defined in the header. Also, I think the size should be common between MPU and MMU.
Quoted 3 lines
In [1], I was going to name it
XEN_VIRT_SIZE. I would be OK to remove "VIRT" in the name.
Thx and please, then I will replace it with XEN_SIZE
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S line 23
Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5
-0+ +/* + * In boot stage, we will use 1 MPU region: + * Region#0: Normal memory for Xen text + data + bss (2MB) + */Are we only going to modify the MPU in
head.S? If not, then I would define the layout inconfig_mpu.hso there are a single point where you can read how this works.
We will remap Xen in C codes in setup_mm().
The whole strategy is aligned with MMU: a very simple setup(map xen with the maximum size, 2M) at start-of-the-day, and a fit map in setup_mm.
All the following variables will be only used at head_mpu.S. It is not very generic, later, when introducing MPU memory region management in C codes, we will define different macros for various memory attributes.
Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S line 27
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+#define BOOT_NORMAL_REGION_IDX 0x0 + +/* MPU normal memory attributes. */ +#define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */IIUC, this means that Xen will be mapped write-executable. Is this going to be forever? If not, when can't we already mapped Xen properly?
No, this setup is the start-of-day setup, and will only last for a very short time.
To be aligned with MMU system, in which L3 memory attributes is as follows:
#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */ Xen shall be mapped read-only-executable here, and I will fix it.
Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S line 28
Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1
-0+#define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */To me, it feels like this should be fined outside of
head.Sbecause this could be re-used by other part of Xen.
It is like PT_MEM_L3 in head_mmu.S. It will be only used in compiler codes.
MMU is defining macro like PAGE_HYPERVISOR_RW, for memory attributes management in C codes, and we intend to follow the same strategy.
Quoted 41 lines
xen/arch/arm/arm64/head_mpu.S line 32
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+ +.macro write_pr, sel, prbar, prlar + msr PRSELR_EL2, \sel + dsb syIs it really necessary to use "sy" here? Also, it would be good to explain the logic. I.e. why do you need two dsb but only one isb?
In fact, I was expecting an "isb" here than "dsb" to wait for the completion of the instruction.
xen/arch/arm/arm64/head_mpu.S
Quoted 13 lines
xen/arch/arm/arm64/head_mpu.S new file +13
-0+ msr PRBAR_EL2, \prbar + msr PRLAR_EL2, \prlar + dsb sy + isb +.endm + +.section .text.header, "ax", %progbits + +/* + * Static start-of-day EL2 MPU memory layout. + * + * It has a very simple structure, including: + * - 2MB normal memory mappings of xen at XEN_START_ADDRESS,which
xen/arch/arm/arm64/head_mpu.S line 46
Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1
-0+ * is the address where Xen was loaded by the bootloader.Missing details on the clobberred registers.
xen/arch/arm/arm64/head_mpu.S line 54
Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8
-0+ */ +ENTRY(prepare_early_mappings) + /* Map Xen start memory to a normal memory region. */ + mov x0, #BOOT_NORMAL_REGION_IDX + ldr x1, =XEN_START_ADDRESS + and x1, x1, #MPU_REGION_MASK + mov x3, #PRBAR_NORMAL_MEM + orr x1, x1, x3It looks like to me there are a potential for a macro to compute the register.
Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S line 57
Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3
-0+ + ldr x2, =XEN_START_ADDRESS + mov x3, #(XEN_START_MEM_SIZE - 1)
XEN_START_MEM_SIZEis the maximum size of Xen. IOW, Xen may be smaller
and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.
Later, in setup_mm we will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.
IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.
Quoted 147 lines
xen/arch/arm/arm64/head_mpu.S line 65
Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S new file +8
-0+ add x2, x2, x + and x2, x2, #MPU_REGION_MASK + mov x3, #PRLAR_NORMAL_MEM + orr x2, x2, x3 + + /* + * Write to MPU protection region: + * x0 for pr_sel, x1 for prbar x2 for prlarThis is not a very useful comment because this can be inferred from the prototype of
write_pr. What would be more interesting is to explain the logic within this function in the same way we do inhead.Sandhead_mmu.S.Review
Quoted 6 lines
xen/arch/arm/arm64/head_mpu.S new file +5
-0+ */ + write_pr x0, x1, x2 + + ret +ENDPROC(prepare_early_mappings)(quoted diff) +0
-0
b/xen/arch/arm/include/asm/arm64/mpu.hReview
Quoted 19 lines
new file mode 100644
index 0000000000..d209eef6dbxen/arch/arm/include/asm/arm64/mpu.h new file +13
-0@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* mpu.h: Arm Memory Protection Unit definitions. +*/ + +#ifndef __ARM64_MPU_H__ +#define __ARM64_MPU_H__ + +#define MPU_REGION_SHIFT 6 +#define MPU_REGION_ALIGN (_AC(1, UL) << MPU_REGION_SHIFT) +#define MPU_REGION_MASK (~(MPU_REGION_ALIGN - 1)) + +#endif /* __ARM64_MPU_H__ */ (quoted diff) +0
-0
b/xen/arch/arm/include/asm/arm64/sysregs.hxen/arch/arm/include/asm/arm64/sysregs.h
Quoted 3 lines
index
54670084c3..a596042d6c100644xen/arch/arm/include/asm/arm64/sysregs.h +0
-0In the context of this patch, it would be better to only define the registers you need. If you want to define all of them, then please define them in a separate patch before this one.
xen/arch/arm/include/asm/arm64/sysregs.h line 462
Quoted 6 lines
xen/arch/arm/include/asm/arm64/sysregs.h +2
-0@@ -458,6 +458,95 @@#define ZCR_ELx_LEN_SIZE 9#define ZCR_ELx_LEN_MASK 0x1ff +/* System registers for AArch64 with PMSA */ +#ifdef CONFIG_HAS_MPU The #ifdef here seems unnecessary.
Quoted 89 lines
xen/arch/arm/include/asm/arm64/sysregs.h +87
-0+ +/* EL1 MPU Protection Region Base Address Register encode */ +#define PRBAR_EL1 S3_0_C6_C8_0 +#define PRBAR1_EL1 S3_0_C6_C8_4 +#define PRBAR2_EL1 S3_0_C6_C9_0 +#define PRBAR3_EL1 S3_0_C6_C9_4 +#define PRBAR4_EL1 S3_0_C6_C10_0 +#define PRBAR5_EL1 S3_0_C6_C10_4 +#define PRBAR6_EL1 S3_0_C6_C11_0 +#define PRBAR7_EL1 S3_0_C6_C11_4 +#define PRBAR8_EL1 S3_0_C6_C12_0 +#define PRBAR9_EL1 S3_0_C6_C12_4 +#define PRBAR10_EL1 S3_0_C6_C13_0 +#define PRBAR11_EL1 S3_0_C6_C13_4 +#define PRBAR12_EL1 S3_0_C6_C14_0 +#define PRBAR13_EL1 S3_0_C6_C14_4 +#define PRBAR14_EL1 S3_0_C6_C15_0 +#define PRBAR15_EL1 S3_0_C6_C15_4 + +/* EL1 MPU Protection Region Limit Address Register encode */ +#define PRLAR_EL1 S3_0_C6_C8_1 +#define PRLAR1_EL1 S3_0_C6_C8_5 +#define PRLAR2_EL1 S3_0_C6_C9_1 +#define PRLAR3_EL1 S3_0_C6_C9_5 +#define PRLAR4_EL1 S3_0_C6_C10_1 +#define PRLAR5_EL1 S3_0_C6_C10_5 +#define PRLAR6_EL1 S3_0_C6_C11_1 +#define PRLAR7_EL1 S3_0_C6_C11_5 +#define PRLAR8_EL1 S3_0_C6_C12_1 +#define PRLAR9_EL1 S3_0_C6_C12_5 +#define PRLAR10_EL1 S3_0_C6_C13_1 +#define PRLAR11_EL1 S3_0_C6_C13_5 +#define PRLAR12_EL1 S3_0_C6_C14_1 +#define PRLAR13_EL1 S3_0_C6_C14_5 +#define PRLAR14_EL1 S3_0_C6_C15_1 +#define PRLAR15_EL1 S3_0_C6_C15_5 + +/* EL2 MPU Protection Region Base Address Register encode */ +#define PRBAR_EL2 S3_4_C6_C8_0 +#define PRBAR1_EL2 S3_4_C6_C8_4 +#define PRBAR2_EL2 S3_4_C6_C9_0 +#define PRBAR3_EL2 S3_4_C6_C9_4 +#define PRBAR4_EL2 S3_4_C6_C10_0 +#define PRBAR5_EL2 S3_4_C6_C10_4 +#define PRBAR6_EL2 S3_4_C6_C11_0 +#define PRBAR7_EL2 S3_4_C6_C11_4 +#define PRBAR8_EL2 S3_4_C6_C12_0 +#define PRBAR9_EL2 S3_4_C6_C12_4 +#define PRBAR10_EL2 S3_4_C6_C13_0 +#define PRBAR11_EL2 S3_4_C6_C13_4 +#define PRBAR12_EL2 S3_4_C6_C14_0 +#define PRBAR13_EL2 S3_4_C6_C14_4 +#define PRBAR14_EL2 S3_4_C6_C15_0 +#define PRBAR15_EL2 S3_4_C6_C15_4 + +/* EL2 MPU Protection Region Limit Address Register encode */ +#define PRLAR_EL2 S3_4_C6_C8_1 +#define PRLAR1_EL2 S3_4_C6_C8_5 +#define PRLAR2_EL2 S3_4_C6_C9_1 +#define PRLAR3_EL2 S3_4_C6_C9_5 +#define PRLAR4_EL2 S3_4_C6_C10_1 +#define PRLAR5_EL2 S3_4_C6_C10_5 +#define PRLAR6_EL2 S3_4_C6_C11_1 +#define PRLAR7_EL2 S3_4_C6_C11_5 +#define PRLAR8_EL2 S3_4_C6_C12_1 +#define PRLAR9_EL2 S3_4_C6_C12_5 +#define PRLAR10_EL2 S3_4_C6_C13_1 +#define PRLAR11_EL2 S3_4_C6_C13_5 +#define PRLAR12_EL2 S3_4_C6_C14_1 +#define PRLAR13_EL2 S3_4_C6_C14_5 +#define PRLAR14_EL2 S3_4_C6_C15_1 +#define PRLAR15_EL2 S3_4_C6_C15_5 + +/* MPU Protection Region Enable Register encode */ +#define PRENR_EL1 S3_0_C6_C1_1 +#define PRENR_EL2 S3_4_C6_C1_1 + +/* MPU Protection Region Selection Register encode */ +#define PRSELR_EL1 S3_0_C6_C2_1 +#define PRSELR_EL2 S3_4_C6_C2_1 + +/* MPU Type registers encode */ +#define MPUIR_EL1 S3_0_C0_C0_4 +#define MPUIR_EL2 S3_4_C0_C0_4 + +#endif +/* Access to system registers */
Quoted 8 lines
xen/arch/arm/include/asm/arm64/sysregs.h line 552
Quoted 1 line
xen/arch/arm/include/asm/arm64/sysregs.h +0
-0#define WRITE_SYSREG64 (v, name) do { \Cheers,
[1] https://lore.kernel.org/all/[email protected]/
--
Julien Grall
Cheers,
--
Penny Zheng
RE: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
In reply to Julien GrallHi Julien
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:02 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 11/11] xen/arm64: addsetup_fixmapandremove_identity_mappingfor MPUHi,
Quoted 19 lines · Wei Chen
From: Penny Zheng <[email protected]>
setup_fixmapandremove_identity_mappingare two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow inhead.Sdoesn't need to use #ifdef to gate MPU/MMU code.In MMU system,
setup_fixmapis used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but beforesetup_fixmap. This restriction is the same as MMU system.For
remove_identity_mapping, we just need an empty function to makehead.Scode flow happy.Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>Quoted 7 lines
--- xen/arch/arm/arm64/head_mpu.S | 49 +++++++++++++++++++ .../arm/include/asm/platforms/fvp_baser.h | 4 ++ 2 files changed, 53 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/arm64/head_mpu.Sindex5a1b03e293..336c0a630f100644
Quoted 27 lines
xen/arch/arm/arm64/head_mpu.S +7
-0@@ -20,13 +20,20 @@ /* * In boot stage, we will use 1 MPU region: * Region#0: Normal memory for Xen text + data + bss (2MB) + * Region#1: Device memory for EARLY UART, size is defined + * by platform's EARLY_UART_SIZE */ #define BOOT_NORMAL_REGION_IDX 0x0 +#define BOOT_DEVICE_REGION_IDX 0x1 /* MPU normal memory attributes. */ #define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */ #define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */ +/* MPU device memory attributes. */ +#define PRBAR_DEVICE_MEM 0x20 /* SH=10 AP=00 XN=00 */ +#define PRLAR_DEVICE_MEM 0x09 /* NS=0 ATTR=100 EN=1 */ + .macro write_pr, sel, prbar, prlar msr PRSELR_EL2, \sel dsb sy@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings) ret ENDPROC(prepare_early_mappings)
Quoted 15 lines
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+/* + * In MMU system, setup_fixmap is used for Xen to map some essential +data + * or devices in boot-time. In order to be consistent with MMU +system, we + * inherit the function name for MPU system. + * setup_fixmap of MPU system will: + * - Map the early UART to MPU protection region when earlyprintk is + * enabled (The PRINT can't be used after turning on MPU but before + * setup_fixmap).For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.
Yes, both removing identity mapping and using fixmap virtual memory layout are not applied to the MPU system.
And in MMU system, the setup_fixmap function has a behavior to map the UART for early printk. And we are only trying to pertain this behavior on MPU system.
Quoted 2 lines
In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.
Hmm, I'm a bit confused here. Which restriction you are trying to remove? The whole fixmap virtual memory layout?
Quoted 38 lines
Quoted 24 lines
xen/arch/arm/arm64/head_mpu.S +23
-0+ * + * Clobbers x0 - x3 + */ +ENTRY(setup_fixmap) +#ifdef CONFIG_EARLY_PRINTK + /* Map early uart to MPU device region for early printk. */ + mov x0, #BOOT_DEVICE_REGION_IDX + ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS + and x1, x1, #MPU_REGION_MASK + mov x3, #PRBAR_DEVICE_MEM + orr x1, x1, x3 + + ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS + ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1) + add x2, x2, x3 + and x2, x2, #MPU_REGION_MASK + mov x3, #PRLAR_DEVICE_MEM + orr x2, x2, x3 + + /* + * Write to MPU protection region: + * x0 for pr_sel, x1 for prbar x2 for prlar + */xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 119
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+ write_pr x0, x1, x2 +#endif + + ret +ENDPROC(setup_fixmap) + +/* Stub of remove_identity_mapping for MPU systems */ +ENTRY(remove_identity_mapping) + ret +ENDPROC(remove_identity_mapping)This stub could be avoided if you move the call to
remove_identity_mappinginenable_mmu()as I did for arm32. See [1].
Quoted 2 lines
Thx! Understood, and I'll use the same logic for enable_mmu.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Henry WangQuoted 1 line · Henry Wang
Hi Julien,
Hi Henry,
Quoted 11 lines
Quoted 9 lines
-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded indomain_build(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).Sounds good to me, I am wondering if we need to include also patch #1 though.
If we were earlier in the release, I would have say yes. But now, it is a no as this just a cleanup (having extra definitions is harmless).
Quoted 4 lines
If this patch wants to be in 4.17:
Release-acked-by Henry Wang <[email protected]>
Thanks!
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
Hi Julien,
Quoted 8 lines
-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded indomain_buildQuoted 5 lines
Quoted 4 lines
(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).
Quoted 5 lines
Quoted 1 line
Sounds good to me, I am wondering if we need to include also patch #1
though.
If we were earlier in the release, I would have say yes. But now, it is a no as this just a cleanup (having extra definitions is harmless).
Thanks for your confirmation :) No problem.
Kind regards,
Henry
Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions
In reply to Penny ZhengQuoted 1 line · Penny Zheng
Hi Julien
Hi Penny,
Quoted 11 lines
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S line 23
Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5
-0+ +/* + * In boot stage, we will use 1 MPU region: + * Region#0: Normal memory for Xen text + data + bss (2MB) + */Are we only going to modify the MPU in
head.S? If not, then I would define the layout inconfig_mpu.hso there are a single point where you can read how this works.
Quoted 4 lines
We will remap Xen in C codes in
setup_mm().
The whole strategy is aligned with MMU: a very simplesetup(map xen with the maximum size, 2M)at start-of-the-day, and a fit map insetup_mm.
The strategy we are using for the MMU is completely broken (not compliant with the Arm Arm) and unnecessary. My long term goal is to actually remove the switch_ttbr() and most of setup_pagetables() for all setup but cache coloring. This means the concept of boot pages will not exist anymore.
For the MPU, we should aim to do better than what was done for the MMU. Right now, I see no argument for switching MPUs table. I am only seen argument against it because you need to disable the cache and is quite fragile.
[...]
Quoted 10 lines
Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S line 57
Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3
-0+ + ldr x2, =XEN_START_ADDRESS + mov x3, #(XEN_START_MEM_SIZE - 1)
XEN_START_MEM_SIZEis the maximum size of Xen. IOW, Xen may be smaller
and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.
Quoted 4 lines
Later, in
setup_mmwe will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.
But what prevents you to do this now?
Quoted 3 lines
IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.
Which is broken because we don't know what located after Xen binary. This could be reserved RAM, device which may requires non-caching attribute. Mapping those regions with caching attributes is going to break.
As I hinted above, there are a very long list of issues in the MMU boot code. So don't take that code for granted. Your MPU code should first be compliant with the Arm Arm. If it is close to the MMU code then that's a bonus. But bear in mind that the code may look very different soon (see [1]).
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
In reply to Penny ZhengHi Penny,
Quoted 87 lines · Penny Zheng
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:02 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 11/11] xen/arm64: addsetup_fixmapandremove_identity_mappingfor MPUHi,
Quoted 19 lines · Wei Chen
From: Penny Zheng <[email protected]>
setup_fixmapandremove_identity_mappingare two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow inhead.Sdoesn't need to use #ifdef to gate MPU/MMU code.In MMU system,
setup_fixmapis used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but beforesetup_fixmap. This restriction is the same as MMU system.For
remove_identity_mapping, we just need an empty function to makehead.Scode flow happy.Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>Quoted 7 lines
--- xen/arch/arm/arm64/head_mpu.S | 49 +++++++++++++++++++ .../arm/include/asm/platforms/fvp_baser.h | 4 ++ 2 files changed, 53 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/arm64/head_mpu.Sindex5a1b03e293..336c0a630f100644
Quoted 27 lines
xen/arch/arm/arm64/head_mpu.S +7
-0@@ -20,13 +20,20 @@ /* * In boot stage, we will use 1 MPU region: * Region#0: Normal memory for Xen text + data + bss (2MB) + * Region#1: Device memory for EARLY UART, size is defined + * by platform's EARLY_UART_SIZE */ #define BOOT_NORMAL_REGION_IDX 0x0 +#define BOOT_DEVICE_REGION_IDX 0x1 /* MPU normal memory attributes. */ #define PRBAR_NORMAL_MEM 0x30 /* SH=11 AP=00 XN=00 */ #define PRLAR_NORMAL_MEM 0x0f /* NS=0 ATTR=111 EN=1 */ +/* MPU device memory attributes. */ +#define PRBAR_DEVICE_MEM 0x20 /* SH=10 AP=00 XN=00 */ +#define PRLAR_DEVICE_MEM 0x09 /* NS=0 ATTR=100 EN=1 */ + .macro write_pr, sel, prbar, prlar msr PRSELR_EL2, \sel dsb sy@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings) ret ENDPROC(prepare_early_mappings)Quoted 15 lines
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+/* + * In MMU system, setup_fixmap is used for Xen to map some essential +data + * or devices in boot-time. In order to be consistent with MMU +system, we + * inherit the function name for MPU system. + * setup_fixmap of MPU system will: + * - Map the early UART to MPU protection region when earlyprintk is + * enabled (The PRINT can't be used after turning on MPU but before + * setup_fixmap).For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.
Yes, both removing identity mapping and using fixmap virtual memory layout are not applied to the MPU system.
Quoted 7 lines
And in MMU system, the
setup_fixmapfunction has a behavior to map the UART for early printk. And we are only trying to pertain this behavior on MPU system.Quoted 2 lines
In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.
Hmm, I'm a bit confused here. Which restriction you are trying to remove?
The fact that the fixmap entry is not created in create_page_tables() (now prepare_early_mappings()).
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Julien GrallQuoted 8 lines · Julien Grall
Hi Wei,
Quoted 4 lines · Wei Chen
There are lots of MMU specific code in
head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code tohead_mmu.S, and keep common code still inhead.S.
Quoted 5 lines
I am afraid that you can't simply move the MMU code out of
head.Sbecause this will break Xen when running using the identity map.This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before
_end_bootand checking at link time if this fits in 4KB.Now that you moved the MMU code outside of
head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.
Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 7 lines
Quoted 7 lines
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
Quoted 8 lines
Quoted 2 lines
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.I read this as I can build Xen and see it boots (not creating domain). However...
HAS_MPUis not defined and I was expectingmm.cto be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.
These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a relatively complete thing in this part#1 series.
We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the stage#1 will have about 30 patches?
Cheers,
Wei Chen
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable
In reply to Julien GrallHi Julien
Quoted 27 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disableHi Wei,
Quoted 11 lines · Wei Chen
From: Penny Zheng <[email protected]>
We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.
Quoted 3 lines
This raises the following question: Why can't we create the split regions right now?
The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes. Then we could reuse as much same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU all over the place.
Quoted 4 lines
In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).
Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?
So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/
Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.
One is in setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear. Otherwise it is hard and complex to maintain on runtime, especially during context switch.
Unchanging regions like xen text will not change during vcpu context switch. Flexible regions, like guest memory region, will display different contents during vcpu context switch.
Quoted 83 lines
Quoted 19 lines
So in this situation, we need to disable the whole MPU to update the protection regions.
In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent withenable_mmu. For MMU systems,enable_mmuwill turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.In this patch, we also introduce a neutral name
enable_mmfor Xen to enable MMU/MPU. This can help us to keep one code flow inhead.SQuoted 6 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Penny Zheng <[email protected]>--- xen/arch/arm/arm64/head.S | 5 +++-- xen/arch/arm/arm64/head_mmu.S | 4 ++-- xen/arch/arm/arm64/head_mpu.S | 35+++++++++++++++++++++++++++++++++++
Quoted 7 lines
3 files changed, 40 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +0
-0xen/arch/arm/arm64/head.S real_start_efi: · line 261
Quoted 8 lines
xen/arch/arm/arm64/head.S +2
-1@@ -255,7 +255,8 @@ real_start_efi: * and protection regions for MPU systems. */ bl prepare_early_mappings - bl enable_mmu + /* Turn on MMU or MPU */ + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime VirtualAddress. */
xen/arch/arm/arm64/head.S GLOBAL(init_secondary) · line 319
Quoted 9 lines
xen/arch/arm/arm64/head.S +1
-1ldr x0, =primary_switched@@ -313,7 +314,7 @@ GLOBAL(init_secondary) bl check_cpu_mode bl cpu_init bl prepare_early_mappings - bl enable_mmu + bl enable_mm /* We are still in the 1:1 mapping. Jump to the runtime VirtualAddress. */
Quoted 3 lines
xen/arch/arm/arm64/head.S +0
-0ldr x0, =secondary_switched(quoted diff) +0
-0
b/xen/arch/arm/arm64/head_mmu.Sindexfc64819a98..b542755bd2100644
Quoted 20 lines
xen/arch/arm/arm64/head_mmu.S +2
-2@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings) * * Clobbers x0 - x3 */ -ENTRY(enable_mmu) +ENTRY(enable_mm) PRINT("- Turning on paging -\r\n") /*@@ -239,7 +239,7 @@ ENTRY(enable_mmu) msr SCTLR_EL2, x0 /* now paging is enabled */ isb /* Now, flush the icache */ ret -ENDPROC(enable_mmu) +ENDPROC(enable_mm) /* * Remove the 1:1 map from the page-tables. It is not easy to keepQuoted 2 lines
track diff --git
a/xen/arch/arm/arm64/head_mpu.Sb/xen/arch/arm/arm64/head_mpu.Sindexf60611b556..5a1b03e293100644
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S +0
-0@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
Quoted 14 lines
xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings)
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8
-0ret ENDPROC(prepare_early_mappings) + +/* + * Enable EL2 MPU and data cache. Because we will disable cache + * with MPU at the same time, in accordance with that, we have + * to enable cache with MPU at the same time in this function. + * When MPU is disabled, the MPU background attributes will + * be used. On some platform, the background will treat all + * memory as IO memory.I was under the impression that all access would be treated as Device Memory when the MMU is off. Isn't it the case for the MPU?
Yes, without background region. Right now, background region is disabled here.
But if the Background region is enabled, then the MPU uses the default memory map as the Background region for generating the memory attributes when MPU is disabled. And it is also IMPLENMENTATION DEFINED~
Quoted 35 lines
Also, I think the correct wording is "device memory" rather than "IO memory".
Quoted 1 line
The access to IO memory will bypass
Ditto.
Quoted 8 lines
+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */xen/arch/arm/arm64/head_mpu.S +3
-0+ENTRY(enable_mm) + mrs x0, SCTLR_EL2 + mov x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 96
Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10
-0+ /* Enable EL2 MPU and D-cache */ + orr x0, x0, x1 + dsb sy + msr SCTLR_EL2, x0 + isb + ret +ENDPROC(enable_mm) + +/* Disable MPU system, including data cache. */ +ENTRY(disable_mm)I would rather not introduce this function until there is a caller. This is because, I believe, there are some assumptions on the state of the cache before we can turn off the MMU. So I would like to see the caller in order to assess whether this function makes sense.
Cheers,
--
Julien Grall
Quoted 1 line · Wei Chen
Hi Julien,
Hi,
Quoted 23 lines
Quoted 8 lines
Quoted 8 lines
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
Quoted 8 lines
Quoted 2 lines
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.I read this as I can build Xen and see it boots (not creating domain). However...
HAS_MPUis not defined and I was expectingmm.cto be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a relatively complete thing in this part#1 series.
Quoted 3 lines
We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the stage#1 will have about 30 patches?
30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions
In reply to Julien GrallHi Julien
Quoted 61 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:30 PM
To: Penny Zheng <[email protected]>; Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regionsQuoted 1 line · Penny Zheng
Hi Julien
Hi Penny,
Quoted 10 lines
Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S
Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4
-0+ +/* + * In boot stage, we will use 1 MPU region: + * Region#0: Normal memory for Xen text + data + bss (2MB) */Are we only going to modify the MPU in
head.S? If not, then I would define the layout inconfig_mpu.hso there are a single point where you can read how this works.Quoted 4 lines
We will remap Xen in C codes in
setup_mm().
The whole strategy is aligned with MMU: a very simplesetup(map xen with the maximum size, 2M)at start-of-the-day, and a fit map insetup_mm.The strategy we are using for the MMU is completely broken (not compliant with the Arm Arm) and unnecessary. My long term goal is to actually remove the
switch_ttbr()and most ofsetup_pagetables()for all setup but cache coloring. This means the concept of boot pages will not exist anymore.For the MPU, we should aim to do better than what was done for the MMU. Right now, I see no argument for switching MPUs table. I am only seen argument against it because you need to disable the cache and is quite fragile.
[...]
Quoted 10 lines
Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S line 57
Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3
-0+ + ldr x2, =XEN_START_ADDRESS + mov x3, #(XEN_START_MEM_SIZE - 1)
XEN_START_MEM_SIZEis the maximum size of Xen. IOW, Xen may be smaller and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.Quoted 4 lines
Later, in
setup_mmwe will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.But what prevents you to do this now?
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.
Which is broken because we don't know what located after Xen binary. This could be reserved RAM, device which may requires non-caching attribute. Mapping those regions with caching attributes is going to break.
Understood!
Then I will map Xen components by components in the compile time, then later the first MPU/Cache disabled will be eliminated, I guess.
Quoted 7 lines
Cheers,
--
Julien Grall
Hi Julien,
Quoted 43 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1Quoted 1 line · Wei Chen
Hi Julien,
Hi,
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
Quoted 8 lines
Quoted 2 lines
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.I read this as I can build Xen and see it boots (not creating domain). However...
HAS_MPUis not defined and I was expectingmm.cto be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.Quoted 4 lines
These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a
relatively
Quoted 4 lines
complete thing in this part#1 series.
We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the
Quoted 8 lines
stage#1
Quoted 1 line
will have about 30 patches?
30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).
I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.
For Armv8-R model, you can download Armv8-R AEM FVP model from link[1]. It's license free.
[1] https://developer.arm.com/downloads/-/arm-ecosystem-models
Cheers,
Wei Chen
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable
In reply to Penny ZhengQuoted 1 line · Penny Zheng
Hi Julien
Hi Penny,
Quoted 28 lines
Quoted 28 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disableHi Wei,
Quoted 11 lines · Wei Chen
From: Penny Zheng <[email protected]>
We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.
Quoted 6 lines
Quoted 3 lines
This raises the following question: Why can't we create the split regions right now?
The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes.
The MMU code is going to change pretty soon (see [1] for some ground work). The runtime page-tables for CPU0 will be created in assembly code and never switched after (aside when using cache coloring).
Although, I don't think I will apply the proper permissions in assembly (this is a bit trickier than with the MPU).
Quoted 3 lines
Then we could reuse as much
same interfaces as we could, in order to not leave #ifdefCONFIG_HAS_MPUall over the place.
Do you have a list of those interfaces that would require #ifdef?
Quoted 9 lines
Quoted 5 lines
In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).
Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?
No. I am more concerned about...
Quoted 2 lines
So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/
... this. We don't only need to flush before they are accessed but also after if they are modified.
It is possible to do it correctly, but it requires to be very careful. So if we can avoid disabling the cache/MPU then it will be a lot better.
Quoted 8 lines
Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.
One is in
setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear.
You should not need any reorg if you map the boot-only section towards in the higher slot index (or just after the fixed ones).
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Julien GrallQuoted 9 lines · Julien Grall
(+ Henry)
Hi,
Quoted 1 line · Wei Chen
domain_builduseioremap_wcto map a new non-cacheable virtuals/use/uses/
Quoted 8 lines
Quoted 6 lines
address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.
So in this patch, we add an iounmap to the end of
domain_build, after Xen loaded initrd to guest memory.Please a fixes tag. The issue was introduced by commit
bb7e6d565d92.
Well I forgot to add it on commit :/. I will try to remember to backport it.
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
In reply to Julien GrallHi Julien,
Quoted 46 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: 2022年11月7日 2:55
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded indomain_build(+ Henry)
Hi,
Quoted 1 line · Wei Chen
domain_builduseioremap_wcto map a new non-cacheable virtuals/use/uses/
Quoted 6 lines
address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.
So in this patch, we add an iounmap to the end of
domain_build, after Xen loaded initrd to guest memory.Please a fixes tag. The issue was introduced by commit
bb7e6d565d92.xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info
Quoted 10 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/domain_build.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/domain_build.c +0
-0@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info*kinfo)
xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422
Quoted 5 lines
xen/arch/arm/domain_build.c +2
-0initrd, len);if ( res !=0 )panic ("Unable to copy the initrd in the hwdom memory\n" ); + +iounmap (initrd);
Quoted 7 lines
This looks good to me. But I am wondering whether using
ioremap_wc()is actually correct because we are reading the region. So it seems strang to map it with write-combine.So I would consider to use
ioremap_cache(). That said, this would be a separate patch.
Ok, we will try to use ioremap_cache and test it. If everything works well we will introduce a separate patch in next version.
Cheers,
Wei Chen
Quoted 8 lines
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).
Cheers,
--
Julien Grall
RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build
Quoted 19 lines
Quoted 7 lines
xen/arch/arm/domain_build.c static void __init initrd_load(struct
Quoted 7 lines
1 file changed, 2 insertions(+)xen/arch/arm/domain_build.c +0
-0@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct
kernel_infoQuoted 11 lines
*kinfo)
xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422
Quoted 5 lines
xen/arch/arm/domain_build.c +2
-0initrd, len);if ( res !=0 )panic ("Unable to copy the initrd in the hwdom memory\n" ); + +iounmap (initrd);This looks good to me. But I am wondering whether using
ioremap_wc()is actually correct because we are reading the region. So it seems strang to map it with write-combine.
Quoted 7 lines
Quoted 3 lines
So I would consider to use
ioremap_cache(). That said, this would be a separate patch.Ok, we will try to use
ioremap_cacheand test it. If everything works well we will introduce a separate patch in next version.
Or is it better to send a separate patch for this? Because I think we might need something to address the v1 comments.
Cheers,
Wei Chen
Quoted 12 lines
Cheers,
Wei ChenQuoted 8 lines
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).
Cheers,
--
Julien Grall
RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable
In reply to Julien GrallQuoted 15 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 6:38 PM
To: Penny Zheng <[email protected]>; Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable
Quoted 4 lines
Quoted 1 line · Penny Zheng
Hi Julien
Hi Penny,
Hi Julien
Quoted 75 lines
Quoted 19 lines
Quoted 19 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disableHi Wei,
Quoted 2 lines · Wei Chen
From: Penny Zheng <[email protected]>
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will
be triggered.
Quoted 7 lines
Quoted 4 lines
This raises the following question: Why can't we create the split regions right now?
The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes.
The MMU code is going to change pretty soon (see [1] for some ground work). The runtime page-tables for CPU0 will be created in assembly code and never switched after (aside when using cache coloring).
Although, I don't think I will apply the proper permissions in assembly (this is a bit trickier than with the MPU).
Quoted 3 lines
Then we could reuse as much
same interfaces as we could, in order to not leave #ifdefCONFIG_HAS_MPUall over the place.Do you have a list of those interfaces that would require #ifdef?
Quoted 9 lines
Quoted 5 lines
In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).
Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?
No. I am more concerned about...
Quoted 3 lines
So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/
... this. We don't only need to flush before they are accessed but also after if they are modified.
It is possible to do it correctly, but it requires to be very careful. So if we can avoid disabling the cache/MPU then it will be a lot better.
Quoted 4 lines
Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.
Quoted 9 lines
Quoted 4 lines
One is in
setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchangingregions in the front and flexible ones in the rear.
You should not need any reorg if you map the boot-only section towards in the higher slot index (or just after the fixed ones).
"in the higher slot index" is really shining a light in my mind ;) And I'll try to enable it in v2.
Quoted 6 lines
Cheers,
--
Julien Grall
RE: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems
In reply to Julien GrallHi Julien,
Quoted 32 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:12
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systemsHi Wei,
Quoted 4 lines · Wei Chen
Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.
I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?
Quoted 10 lines
But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/arm64/head.S | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-)
Quoted 15 lines
xen/arch/arm/Kconfig config ARM_64 · line 10
Quoted 10 lines
xen/arch/arm/Kconfig +1
-1@@ -7,7 +7,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT - select ARM_EFI + select ARM_EFI if !HAS_MPUI think it would make sense to allow
ARM_EFIto be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.
Ok, how about address this comment in NUMA patch set#3 (Arm implementation), because about making ARM_EFI invisible to users, we had some discussions for NUMA RFC and V1.
Quoted 25 lines
xen/arch/arm/arm64/head.S efi_head: · line 175
Quoted 12 lines
xen/arch/arm/Kconfig +0
-0select HAS_FAST_MULTIPLY config ARMxen/arch/arm/arm64/head.S +0
-1@@ -172,8 +172,10 @@ efi_head: .byte 0x52 .byte 0x4d .byte 0x64 - .long pe_header - efi_head /* Offset to the PE header.*/
xen/arch/arm/arm64/head.S efi_head:
Quoted 7 lines
xen/arch/arm/arm64/head.S +4
-1- +#ifndef CONFIG_ARM_EFI + .long 0 /* 0 means no PE header. */ +#else + .long pe_header - efi_head /* Offset to the PE header. */ /* * Add the PE/COFF header to the file. The address of thisheader
xen/arch/arm/arm64/head.S section_table: · line 281
Quoted 3 lines
xen/arch/arm/arm64/head.S +0
-0* is at offset 0x3c in the file, and is part of Linux "Image"@@ -279,6 +281,8 @@ section_table: .short 0 /* NumberOfLineNumbers (0 forexecutables) */
Quoted 10 lines
xen/arch/arm/arm64/head.S section_table: · line 288
Quoted 7 lines
xen/arch/arm/arm64/head.S +2
-0.long 0xe0500020 /* Characteristics (section flags) */ .align 5 +#endif /* CONFIG_ARM_EFI */ + real_start: /* BSS should be zeroed when booting without EFI */ mov x26, #0 /* x26 := skip_zero_bss */Shouldn't the function
efi_xen_startbe stubbed as well?
Reply for your next email:
Yes, this is a good point, efi_xen_start should be protected. I will fix it.
Cheers,
Wei Chen
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Julien GrallHi Julien,
Quoted 65 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platformQuoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++I looked at the content of
fvp_baser.hafter this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...xen/arch/arm/Kconfig config TEE · line 138
Quoted 10 lines
xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hxen/arch/arm/Kconfig +0
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allowsguests
xen/arch/arm/Kconfig config TEE
Quoted 4 lines
xen/arch/arm/Kconfig +2
-0to access real TEE via one of TEE mediators implemented in XEN. +config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform definedaddress"
Quoted 9 lines
xen/arch/arm/Kconfig config TEE · line 143
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ default 0xFFFFFFFF... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the
Kconfigto select the proper address (see the config for selecting early printk address).This will also avoids to use an invalid value here.
We had considered to use Kconfig to define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value, Kconfig option for customized address).
We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.
So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.
[1] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[2] https://gitlab.com/xen-project/fusa/xen-integration/-/blob/integration/mpu/xen/arch/arm/Kconfig.debug
Quoted 36 lines
xen/arch/arm/Kconfig config TEE
Quoted 3 lines
xen/arch/arm/Kconfig +3
-0+ depends on HAS_MPU + help + This option allows to set the customized address at which Xen willbe
xen/arch/arm/Kconfig config TEE
Quoted 3 lines
xen/arch/arm/Kconfig +3
-0+ linked on MPU systems. This address must be aligned to a page size. + Use 0xFFFFFFFF as the default value to indicate that user hasn't + customized this address, and Xen use use the default value thathas
Review
Quoted 6 lines
xen/arch/arm/Kconfig +2
-0+ been defined in platform files. + source "arch/arm/tee/Kconfig" config STATIC_SHM(quoted diff) +0
-0
b/xen/arch/arm/include/asm/platforms/fvp_baser.hReview
Quoted 20 lines
new file mode 100644
index 0000000000..9450a411a9xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14
-0@@ -0,0 +1,14 @@ +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__ +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__ + +/* +* 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS, +* we will use platform defined default address. +*/ +#if CONFIG_XEN_START_ADDRESS ==0xFFFFFFFF +#define XEN_START_ADDRESS 0x200000 +#else +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS +#endif + +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */ (quoted diff) +0
-0b/xen/arch/arm/platforms/Kconfig
Quoted 11 lines
xen/arch/arm/platforms/Kconfig line 4
Quoted 8 lines
index
c93a6b2756..0904793a0b100644xen/arch/arm/platforms/Kconfig +1
-0@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8RIs there any reason to create a new
Kconfigrather than using MPU?
Did you mean FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.
If you meant Armv8R, that's because Armv8R does not equal to MPU. We only allow Armv8R code to detect MPU features. MPU is configurable In EL2.
Cheers,
Wei Chen
Quoted 57 lines
Quoted 42 lines
xen/arch/arm/platforms/Kconfig +7
-3---help--- Choose which hardware platform to enable in Xen.@@ -8,13 +9,14 @@ choice config ALL_PLAT bool "All Platforms" + depends on !ARM_V8R ---help--- Enable support for all available hardware platforms. It doesn't automatically select any of the related drivers. config QEMU bool "QEMU aarch virt machine support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select GICV3 select HAS_PL011 ---help---@@ -23,7 +25,7 @@ config QEMU config RCAR3 bool "Renesas RCar3 support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_SCIF select IPMMU_VMSA ---help---@@ -31,14 +33,22 @@ config RCAR3 config MPSOC bool "Xilinx Ultrascale+ MPSoC support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_CADENCE_UART select ARM_SMMU ---help--- Enable all the required drivers for Xilinx Ultrascale+ MPSoC +config FVP_BASER + bool "Fixed Virtual Platform BaseR support" + depends on ARM_V8Rxen/arch/arm/platforms/Kconfig config RCAR3 · line 54
Quoted 10 lines
xen/arch/arm/platforms/Kconfig +5
-0+ help + Enable platform specific configurations for Fixed Virtual + Platform BaseR + config NO_PLAT bool "No Platforms" + depends on !ARM_V8R ---help--- Do not enable specific support for any platform.Cheers,
--
Julien Grall
RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems
In reply to Julien GrallHi Julien,
Quoted 20 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsHi Wei,
Quoted 6 lines · Wei Chen
FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems.
FIXMAP_ADDRwill return a virtual address by index for special purpose phys-to-virt mapping usage. For example,FIXMAP_ADDR(FIXMAP_CONSOLE)for early console mapping andFIXMAP_ADDR(FIXMAP_MISC)forcopy_from_paddr.
Quoted 5 lines
To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining
FIXMAP_ADDR()& co for MPU.
I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic code. I also prefer this way. Frankly, I really think FIXMAP is awkward in MPU System.
Quoted 21 lines
Quoted 21 lines
But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using
FIXMAP_ADDRto return the input physical address in MPU systems. So in MPU version,FIXMAP_ADDRwill trim physical address to PAGE alignment. This will return an offset which is similar to MMU versionFIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask withPAGE_MASK). The caller can return the input physical address directly.As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/include/asm/config_mpu.h | 2 ++ xen/arch/arm/include/asm/fixmap.h | 25 +++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-)
Quoted 13 lines
xen/arch/arm/Kconfig config ARM · line 19
Quoted 10 lines
xen/arch/arm/Kconfig +1
-1@@ -16,7 +16,7 @@ config ARM select HAS_DEVICE_TREE select HAS_PASSTHROUGH select HAS_PDX - select HAS_PMAP + select HAS_PMAP if !HAS_MPUI can't find any change of
mm.cin this series. So surely this will break the build?
Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.
Quoted 63 lines
Review
Quoted 5 lines
xen/arch/arm/Kconfig +0
-0select IOMMU_FORCE_PT_SHARE config ARCH_DEFCONFIG(quoted diff) +0
-0
b/xen/arch/arm/include/asm/config_mpu.hReview
Quoted 11 lines
index
530abb8302..eee60dcffc100644xen/arch/arm/include/asm/config_mpu.h +2
-0@@ -24,4 +24,6 @@#define HYPERVISOR_VIRT_START XEN_VIRT_START +#define FIXMAP_ADDR (n) (_AT(paddr_t, n) & (PAGE_MASK)) +#endif /* __ARM_CONFIG_MPU_H__ */ (quoted diff) +0
-0
b/xen/arch/arm/include/asm/fixmap.hQuoted 30 lines
index
d0c9a52c8c..1e338759e9100644xen/arch/arm/include/asm/fixmap.h +16
-0@@ -7,6 +7,8 @@#include <xen/acpi.h> #include <xen/pmap.h> +#ifndef CONFIG_HAS_MPU +/* Fixmap slots */ #define FIXMAP_CONSOLE 0/* The primary UART */ #define FIXMAP_MISC 1/* Ephemeral mappings of hardware */ @@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)#endif /* __ASSEMBLY__ */ +#else + +/* +* FIXMAP_ADDR will trim physical address to PAGE alignment. +* This will return an offset which is similar to MMU version +* FIXMAP_ADDR. +* For example: +* EARLY_UART_VIRTUAL_ADDRESS is defined by: +* (FIXMAP_ADDR(FIXMAP_CONSOLE) + \ +* (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK)) +* With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions, +* EARLY_UART_VIRTUAL_ADDRESS can be restore to +* CONFIG_EARLY_UART_BASE_ADDRESS. +* In this case, we don't need to use #ifdef MPU in the code xen/arch/arm/include/asm/fixmap.h static inline unsigned int virt_to_fix(vaddr_t vaddr) · line 73
Quoted 10 lines
xen/arch/arm/include/asm/fixmap.h +9
-0+* where are using FIXMAP_ADDR to make them to use physical +* address explicitily. +*/ +#ifdef CONFIG_EARLY_UART_BASE_ADDRESS +#define FIXMAP_CONSOLE CONFIG_EARLY_UART_BASE_ADDRESS +#endif + +#endif /* CONFIG_HAS_MPU */ +#endif /* __ASM_FIXMAP_H */ Cheers,
--
Julien Grall
RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Julien GrallHi Julien,
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code fromhead.Stohead_mmu.SHi Wei,
Quoted 4 lines · Wei Chen
There are lots of MMU specific code in
head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code tohead_mmu.S, and keep common code still inhead.S.I am afraid that you can't simply move the MMU code out of
head.Sbecause this will break Xen when running using the identity map.
Quoted 9 lines
This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before
_end_bootand checking at link time if this fits in 4KB.Now that you moved the MMU code outside of
head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.
Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.
Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"
OK. We will create a patch to check the size, and place it in the head of the series.
Quoted 6 lines
Quoted 6 lines
As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros:adr_landload_paddrwill be used by MMU and MPU later, so we move them tomacros.h.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Henry Wang <[email protected]>Signed-off-by Wei Chen <[email protected]>In general, the first signed-off should match the author. So who is who here?
I will adjust this order.
Quoted 61 lines
Quoted 26 lines
--- xen/arch/arm/arm64/Makefile | 3 + xen/arch/arm/arm64/head.S | 407 +----------------------- xen/arch/arm/arm64/head_mmu.S | 364 +++++++++++++++++++++ xen/arch/arm/include/asm/arm64/macros.h | 52 ++- 4 files changed, 432 insertions(+), 394 deletions(-)create mode 100644
xen/arch/arm/arm64/head_mmu.Sxen/arch/arm/arm64/Makefile +2
-0@@ -8,6 +8,9 @@ obj-y += domctl.o obj-y += domain.o obj-y += entry.o obj-y += head.o +ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o +endif obj-y += insn.o obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += smc.oxen/arch/arm/arm64/head.S +0
-0xen/arch/arm/arm64/head.S line 28
Quoted 4 lines
xen/arch/arm/arm64/head.S +0
-1@@ -25,17 +25,6 @@ #include <efi/efierr.h> #include <asm/arm64/efibind.h> -#define PT_PT 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1*/
xen/arch/arm/arm64/head.S line 29
Quoted 1 line
xen/arch/arm/arm64/head.S +0
-1-#define PT_MEM 0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1*/
xen/arch/arm/arm64/head.S line 30
Quoted 1 line
xen/arch/arm/arm64/head.S +0
-1-#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1*/
xen/arch/arm/arm64/head.S line 31
Quoted 1 line
xen/arch/arm/arm64/head.S +0
-1-#define PT_DEV 0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1*/
xen/arch/arm/arm64/head.S line 32
Quoted 1 line
xen/arch/arm/arm64/head.S +0
-1-#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1*/
Quoted 22 lines
xen/arch/arm/arm64/head.S +0
-15- -/* Convenience defines to get slot used by Xen mapping. */ -#define XEN_ZEROETH_SLOT zeroeth_table_offset(XEN_VIRT_START) -#define XEN_FIRST_SLOT first_table_offset(XEN_VIRT_START) -#define XEN_SECOND_SLOT second_table_offset(XEN_VIRT_START) - #define __HEAD_FLAG_PAGE_SIZE ((PAGE_SHIFT - 10) / 2) #define __HEAD_FLAG_PHYS_BASE 1@@ -82,73 +71,22 @@ * x30 - lr */ -#ifdef CONFIG_EARLY_PRINTK -/* - * Macro to print a string to the UART, if there is one. - * - * Clobbers x0 - x3 - */ -#define PRINT(_s) \ - mov x3, lr ; \ - adr x0, 98f ; \
Quoted 9 lines
xen/arch/arm/arm64/head.S line 75
Quoted 5 lines
xen/arch/arm/arm64/head.S +2
-3- bl puts ; \ - mov lr, x3 ; \ - RODATA_STR(98, _s) +.section .text.header, "ax", %progbits +/*.aarch64*/The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.
Ok.
Quoted 4 lines
Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).
Did you mean to create a separate patch for moving the functions in macro.h? Or in header section?
Quoted 9 lines
xen/arch/arm/arm64/head.S line 79
Quoted 5 lines
xen/arch/arm/arm64/head.S +2
-1/* - * Macro to print the value of register \xb + * Kernel startup entry point. + * ---------------------------Same here about the indentation. I will not comment everywhere where the indentation was changed. So please look at it.
OK. we will use a separate patch to clean them.
Quoted 37 lines
[...]
Quoted 8 lines
xen/arch/arm/arm64/head.S +0
-8-/* - * Map the UART in the fixmap (when earlyprintk is used) and hook the - * fixmap table in the page tables. - * - * The fixmap cannot be mapped in create_page_tables because it may - * clash with the 1:1 mapping. - * - * Inputs:xen/arch/arm/arm64/head.S cpu_init:
Quoted 10 lines
xen/arch/arm/arm64/head.S +0
-10- * x20: Physical offset - * x23: Early UART base physical address - * - * Clobbers x0 - x3 - */ -setup_fixmap: -#ifdef CONFIG_EARLY_PRINTK - /* Add UART to the fixmap table */ - ldr x0, =EARLY_UART_VIRTUAL_ADDRESS - create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3,type=PT_DEV_L3
Quoted 16 lines
xen/arch/arm/arm64/head.S +0
-10-#endif - /* Map fixmap into boot_second */ - ldr x0, =FIXMAP_ADDR(0) - create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3 - /* Ensure any page table updates made above have occurred. */ - dsb nshst - - ret -ENDPROC(setup_fixmap) - /* * Setup the initial stack and jump to the C world *@@ -810,41 +458,14 @@ launch: ENDPROC(launch)
Quoted 6 lines
xen/arch/arm/arm64/head.S launch: · line 461
Quoted 3 lines
xen/arch/arm/arm64/head.S +1
-1/* Fail-stop */ -fail: PRINT("- Boot failed -\r\n") +ENTRY(fail)This name is a bit too generic to be exposed. But it would be better to duplicate it.
Yes, duplicate it might be better. We will do it in next version.
Cheers,
Wei Chen
Quoted 5 lines
Cheers,
--
Julien Grall
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 73 lines
Quoted 73 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platformQuoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++I looked at the content of
fvp_baser.hafter this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...xen/arch/arm/Kconfig config TEE · line 138
Quoted 10 lines
xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hxen/arch/arm/Kconfig +0
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allowsguests
xen/arch/arm/Kconfig config TEE
Quoted 4 lines
xen/arch/arm/Kconfig +2
-0to access real TEE via one of TEE mediators implemented in XEN. +config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform definedaddress"
xen/arch/arm/Kconfig config TEE · line 143
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ default 0xFFFFFFFF... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the
Kconfigto select the proper address (see the config for selecting early printk address).
Quoted 7 lines
Quoted 2 lines
This will also avoids to use an invalid value here.
We had considered to use
Kconfigto define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value,Kconfigoption for customized address).
Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?
Quoted 4 lines
We also had thought to use
Kconfigto define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.
Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).
Quoted 4 lines
So if we don’t use header files, just use the
Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.
I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.
I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.
Quoted 14 lines
Quoted 13 lines
Review
Quoted 1 line
(quoted diff) +0
-0b/xen/arch/arm/platforms/Kconfig
xen/arch/arm/platforms/Kconfig line 4
Quoted 8 lines
index
c93a6b2756..0904793a0b100644xen/arch/arm/platforms/Kconfig +1
-0@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8RIs there any reason to create a new
Kconfigrather than using MPU?
Quoted 3 lines
Did you mean
FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.
See above for this.
Quoted 2 lines
If you meant Armv8R, that's because Armv8R does not equal to MPU.
I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 21 lines
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsHi Wei,
Quoted 6 lines · Wei Chen
FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems.
FIXMAP_ADDRwill return a virtual address by index for special purpose phys-to-virt mapping usage. For example,FIXMAP_ADDR(FIXMAP_CONSOLE)for early console mapping andFIXMAP_ADDR(FIXMAP_MISC)forcopy_from_paddr.
Quoted 7 lines
Quoted 5 lines
To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining
FIXMAP_ADDR()& co for MPU.I agree, if we don't mind to add some
CONFIG_HAS_MPUin some generic code.
FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for HAS_MPU would do the work to map the region in the MPU.
[...]
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
xen/arch/arm/Kconfig | 2 +- xen/arch/arm/include/asm/config_mpu.h | 2 ++ xen/arch/arm/include/asm/fixmap.h | 25 +++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-)Quoted 13 lines
xen/arch/arm/Kconfig config ARM · line 19
Quoted 10 lines
xen/arch/arm/Kconfig +1
-1@@ -16,7 +16,7 @@ config ARM select HAS_DEVICE_TREE select HAS_PASSTHROUGH select HAS_PDX - select HAS_PMAP + select HAS_PMAP if !HAS_MPUI can't find any change of
mm.cin this series. So surely this will break the build?
Quoted 2 lines
Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.
Do you mean you added some stubs for PMAP? If so, I would not expect any caller for the pmap() to be used for the MPU. Therefore, why would they be necessary?
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 4 lines
Quoted 3 lines
The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.
Quoted 8 lines
Ok.
Quoted 4 lines
Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).
Did you mean to create a separate patch for moving the functions in
macro.h?
I mean the macros in macro.h. Sorry, I misquoted it.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Julien GrallQuoted 98 lines · Julien Grall
Quoted 73 lines
Quoted 73 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platformQuoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++I looked at the content of
fvp_baser.hafter this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...xen/arch/arm/Kconfig config TEE · line 138
Quoted 10 lines
xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hxen/arch/arm/Kconfig +0
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allowsguests
xen/arch/arm/Kconfig
Quoted 5 lines
xen/arch/arm/Kconfig +0
-0to access real TEE via one of TEE mediators implemented inXEN.
xen/arch/arm/Kconfig +2
-0+config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform definedaddress"
xen/arch/arm/Kconfig config TEE · line 143
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ default 0xFFFFFFFF... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the
Kconfigto select the proper address (see the config for selecting early printk address).Quoted 7 lines
Quoted 2 lines
This will also avoids to use an invalid value here.
We had considered to use
Kconfigto define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value,Kconfigoption for customized address).Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?
Quoted 4 lines
We also had thought to use
Kconfigto define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).
Quoted 4 lines
So if we don’t use header files, just use the
Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.
Quoted 7 lines
I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.
I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.
I agree with Julien, especially on this last point.
Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.
The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.
Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand the existing "Platform Support" kconfig menu.
If this was just XEN_START_ADDRESS, I would suggest to keep it in xen.git somewhere. But given that there are a few addresses and sizes to provide/calculate for Xen on MPU to work, using ImageBuilder could be a good idea.
Quoted 48 lines · Wei Chen
Hi Julien,
Quoted 46 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1Quoted 1 line · Wei Chen
Hi Julien,
Hi,
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
Quoted 8 lines
Quoted 2 lines
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.I read this as I can build Xen and see it boots (not creating domain). However...
HAS_MPUis not defined and I was expectingmm.cto be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.Quoted 4 lines
These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a
relatively
Quoted 4 lines
complete thing in this part#1 series.
We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the
stage#1
Quoted 1 line
will have about 30 patches?
Quoted 8 lines
Quoted 5 lines
30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).
I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.
One suggestion to make things easier to review and to commit is to organize the series in a way so that the first 10 patches can still be committed first independently, even if all 30 patches are sent together.
Or alternatively only send 10 patches but also add a link to a github/gitlab tree with all the 30+ patches so that maintainers can have a look how the whole work fit together.
I think we are all on the same page -- I just wanted to highlight that we don't have to finish the review of all 30 patches before we can start committing some of the initial patches in the series.
Cheers,
Stefano
RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems
In reply to Julien GrallHi Julien,
Quoted 45 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月10日 2:30
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 6 lines
Quoted 6 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 21 lines
Quoted 20 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsHi Wei,
Quoted 6 lines · Wei Chen
FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems.
FIXMAP_ADDRwill return a virtual address by index for special purpose phys-to-virt mapping usage. For example,FIXMAP_ADDR(FIXMAP_CONSOLE)for early console mapping andFIXMAP_ADDR(FIXMAP_MISC)forcopy_from_paddr.To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining
FIXMAP_ADDR()& co for MPU.
Quoted 7 lines
Quoted 1 line
I agree, if we don't mind to add some
CONFIG_HAS_MPUin some genericcode.
FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for
HAS_MPUwould do the work to map the region in the MPU.
Sorry, I am still confused about this comment, did you mean we can provider Some generic helpers like: early_map_console / eary_map_guest_memory. For non-MPU system, we still can call fixmap in these callers, but for MPU system, we have to map the region to MPU region?
Quoted 22 lines
[...]
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
xen/arch/arm/Kconfig | 2 +- xen/arch/arm/include/asm/config_mpu.h | 2 ++ xen/arch/arm/include/asm/fixmap.h | 25+++++++++++++++++++++++++
Quoted 16 lines
Quoted 15 lines
xen/arch/arm/Kconfig config ARM · line 19
Quoted 12 lines
3 files changed, 28 insertions(+), 1 deletion(-)xen/arch/arm/Kconfig +1
-1@@ -16,7 +16,7 @@ config ARM select HAS_DEVICE_TREE select HAS_PASSTHROUGH select HAS_PDX - select HAS_PMAP + select HAS_PMAP if !HAS_MPUI can't find any change of
mm.cin this series. So surely this will break the build?
Quoted 7 lines
Quoted 2 lines
Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.
Do you mean you added some stubs for PMAP? If so, I would not expect any caller for the
pmap()to be used for the MPU. Therefore, why would they be necessary?
No, I mean if we want to make pmap can be built successfully for MPU, we have to implement some stubs like: fix_to_virt, xen_fixmap and write_pte, to make compiling success. But just as you said, we would not expect MPU to use any PMAP function, so we have not implemented them for MPU. Instead we disable PMAP for MPU.
Cheer,
Wei Chen
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems
Quoted 55 lines · Wei Chen
Hi Julien,
Quoted 45 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月10日 2:30
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 6 lines
Quoted 6 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 21 lines
Quoted 20 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implementFIXMAP_ADDRfor MPU systemsHi Wei,
Quoted 6 lines · Wei Chen
FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems.
FIXMAP_ADDRwill return a virtual address by index for special purpose phys-to-virt mapping usage. For example,FIXMAP_ADDR(FIXMAP_CONSOLE)for early console mapping andFIXMAP_ADDR(FIXMAP_MISC)forcopy_from_paddr.To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining
FIXMAP_ADDR()& co for MPU.Quoted 7 lines
Quoted 1 line
I agree, if we don't mind to add some
CONFIG_HAS_MPUin some genericcode.
FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for
HAS_MPUwould do the work to map the region in the MPU.
Quoted 4 lines
Sorry, I am still confused about this comment, did you mean we can provider Some generic helpers like:
early_map_console/eary_map_guest_memory. For non-MPU system, we still can call fixmap in these callers, but for MPU system, we have to map the region to MPU region?
Yes.
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Stefano StabelliniHi Stefano, Julien,
Quoted 144 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2022年11月11日 6:13
To: Julien Grall <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platformQuoted 5 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address forFVP
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
BaseR platform
Quoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14++++++++++++++
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I looked at the content of
fvp_baser.hafter this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...Quoted 1 line
xen/arch/arm/platforms/Kconfig | 16+++++++++++++---
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
xen/arch/arm/Kconfig config TEE · line 138
Quoted 7 lines
xen/arch/arm/Kconfig +0
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allowsguests
Quoted 3 lines
xen/arch/arm/Kconfig +0
-0to access real TEE via one of TEE mediators implemented inXEN.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig
Quoted 2 lines
xen/arch/arm/Kconfig +2
-0+config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform definedaddress"
xen/arch/arm/Kconfig config TEE · line 143
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ default 0xFFFFFFFF... this default value will need to be tested everywhere. At least
for
Quoted 1 line
Quoted 1 line
Quoted 1 line
now, I think you can avoid the per platform header by using the
KconfigQuoted 1 line
Quoted 1 line
Quoted 1 line
to select the proper address (see the config for selecting early
printk
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
address).
This will also avoids to use an invalid value here.
We had considered to use
Kconfigto define the start addresses ofv8R64
Quoted 1 line
Quoted 1 line
platforms (prompt users to input the address). But we also want to
provide
Quoted 1 line
Quoted 1 line
a default start address for each platform (Discussed in [1], header
for
Quoted 2 lines
Quoted 1 line
default value,
Kconfigoption for customized address).Why do you want to provide a default value? And how it is guaranteed
that it
Quoted 8 lines
will work for most of the users?
Quoted 4 lines
We also had thought to use
Kconfigto define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.Most of the current Xen is board agnostic except the UART. We push back
the
Quoted 1 line
addition of new one because the address can be found in the firmware
table and
Quoted 1 line
I wanted to avoid increase the number of option (there are dozens of
platform
Quoted 8 lines
out...).
Quoted 4 lines
So if we don’t use header files, just use the
Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.I am not sure I see the problem to force the user to enter the start
address.
Quoted 1 line
My worry with per-platform default value is we end up to force each
vendor to
Quoted 3 lines
provide an header in order to boot Xen.
I think it would be better to provide a config tailored for that
platform
Quoted 1 line
(whether it is part of Xen can be debatable). This would allow a user to
try a
Quoted 1 line
release Xen on their platform with zero changes in the code.
I agree with Julien, especially on this last point.
Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.
Quoted 6 lines
The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.
Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand
Do you mean we can keep a config like armv8r_fvp_baser_config in xen/arch/arm/configs for users to generate a default config? If yes I think this method might be better for now. And about ImageBuilder solution we can do it after MPU support be merged?
Cheers,
Wei Chen
Quoted 6 lines
the existing "Platform Support" kconfig menu.
If this was just
XEN_START_ADDRESS, I would suggest to keep it in xen.git somewhere. But given that there are a few addresses and sizes to provide/calculate for Xen on MPU to work, using ImageBuilder could be a good idea.
Hi Stefano, Julien,
Quoted 76 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2022年11月11日 6:26
To: Wei Chen <[email protected]>
Cc: Julien Grall <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1Quoted 7 lines · Wei Chen
Hi Julien,
Quoted 5 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 3 lines
Quoted 3 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support toXen -
Quoted 13 lines
Quoted 13 lines
Part#1
Quoted 1 line · Wei Chen
Hi Julien,
Hi,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.I read this as I can build Xen and see it boots (not creating
domain).
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
However...
HAS_MPUis not defined and I was expectingmm.cto be modified to cater the MPU support. So I am a bit ensure what theseries
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
is actually doing.
These 11 patches are part#1 of stage#1, the full stage#1 has about
30
Quoted 1 line
Quoted 1 line
Quoted 1 line
patches. We have some concerns if we send so many patches at once,
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
review pressure of maintainers may be very high, so we only choose
about
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
10 to send as part of it. But this also means that we can't do a
relatively
Quoted 3 lines
complete thing in this part#1 series.
We want to hear some suggestions from you to make so many patches
can be
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Reviewed efficiently. Can we send the patches by stages, even the
stage#1
Quoted 1 line
will have about 30 patches?
30 patches in a go is no too bad. I would personally prefer that
because
Quoted 2 lines
Quoted 2 lines
at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-
R
Quoted 3 lines
Quoted 2 lines
model).
Quoted 8 lines
Quoted 2 lines
I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.
One suggestion to make things easier to review and to commit is to organize the series in a way so that the first 10 patches can still be committed first independently, even if all 30 patches are sent together.
I think this is foreseeable, and we have done in this way internally. Every patch can be built and will not broken other architectures.
Quoted 4 lines
Or alternatively only send 10 patches but also add a link to a github/gitlab tree with all the 30+ patches so that maintainers can have a look how the whole work fit together.
In this series we have linked the gitlab branch with the full patches.
Quoted 4 lines
I think we are all on the same page -- I just wanted to highlight that we don't have to finish the review of all 30 patches before we can start committing some of the initial patches in the series.
Yes, I agree with it. And above solutions are ok for us. There will not be much difference in efforts for these two ways for us, so if you guys think which method is the most efficient, I will follow it.
Cheers,
Wei Chen
Quoted 3 lines
Cheers,
Stefano
RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Wei ChenQuoted 150 lines · Wei Chen
Hi Stefano, Julien,
Quoted 148 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2022年11月11日 6:13
To: Julien Grall <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platformQuoted 5 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address forFVP
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
BaseR platform
Quoted 26 lines · Wei Chen
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14++++++++++++++
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I looked at the content of
fvp_baser.hafter this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...Quoted 1 line
xen/arch/arm/platforms/Kconfig | 16+++++++++++++---
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
xen/arch/arm/Kconfig config TEE · line 138
Quoted 7 lines
xen/arch/arm/Kconfig +0
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allowsguests
Quoted 3 lines
xen/arch/arm/Kconfig +0
-0to access real TEE via one of TEE mediators implemented inXEN.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig
Quoted 2 lines
xen/arch/arm/Kconfig +2
-0+config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform definedaddress"
xen/arch/arm/Kconfig config TEE · line 143
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ default 0xFFFFFFFF... this default value will need to be tested everywhere. At least
for
Quoted 1 line
Quoted 1 line
Quoted 1 line
now, I think you can avoid the per platform header by using the
KconfigQuoted 1 line
Quoted 1 line
Quoted 1 line
to select the proper address (see the config for selecting early
printk
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
address).
This will also avoids to use an invalid value here.
We had considered to use
Kconfigto define the start addresses ofv8R64
Quoted 1 line
Quoted 1 line
platforms (prompt users to input the address). But we also want to
provide
Quoted 1 line
Quoted 1 line
a default start address for each platform (Discussed in [1], header
for
Quoted 2 lines
Quoted 1 line
default value,
Kconfigoption for customized address).Why do you want to provide a default value? And how it is guaranteed
that it
Quoted 8 lines
will work for most of the users?
Quoted 4 lines
We also had thought to use
Kconfigto define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.Most of the current Xen is board agnostic except the UART. We push back
the
Quoted 1 line
addition of new one because the address can be found in the firmware
table and
Quoted 1 line
I wanted to avoid increase the number of option (there are dozens of
platform
Quoted 8 lines
out...).
Quoted 4 lines
So if we don’t use header files, just use the
Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.I am not sure I see the problem to force the user to enter the start
address.
Quoted 1 line
My worry with per-platform default value is we end up to force each
vendor to
Quoted 3 lines
provide an header in order to boot Xen.
I think it would be better to provide a config tailored for that
platform
Quoted 1 line
(whether it is part of Xen can be debatable). This would allow a user to
try a
Quoted 1 line
release Xen on their platform with zero changes in the code.
I agree with Julien, especially on this last point.
Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.
The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.
Quoted 5 lines
Quoted 2 lines
Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand
Do you mean we can keep a config like
armv8r_fvp_baser_configin xen/arch/arm/configs for users to generate a default config?
Yes
Quoted 2 lines
If yes I think this method might be better for now. And about ImageBuilder solution we can do it after MPU support be merged?
That's fine
Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 38 lines
Quoted 22 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code fromhead.Stohead_mmu.SHi Wei,
Quoted 4 lines · Wei Chen
There are lots of MMU specific code in
head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code tohead_mmu.S, and keep common code still inhead.S.I am afraid that you can't simply move the MMU code out of
head.Sbecause this will break Xen when running using the identity map.Quoted 9 lines
This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before
_end_bootand checking at link time if this fits in 4KB.Now that you moved the MMU code outside of
head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.
Quoted 6 lines
Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"OK. We will create a patch to check the size, and place it in the head of the series.
I thought a bit more about what I wrote. Xen binary should always start with the Image/Zimage header. At the moment, this is guaranteed because there is only one object using the section .text.header.
With the change introduced in this patch, there will be multiple objects using with the sections .text.header. This means we are relying on the compiler to always put the content of head.S first.
This is basically reverting to the behavior before commit 4267a33b19d ("xen/build: put image header into a separate section").
Therefore we do need a separate section to be used for head_*.S and maybe part of head.S. This new section could be called text.idmap so we know what the section is used for.
Cheers,
Signature · 1 line
Julien Grall
RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S
Hi Julien,
Quoted 65 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月14日 5:43
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code fromhead.Stohead_mmu.SQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 6 lines
Quoted 6 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;Bertrand
Quoted 3 lines
Quoted 3 lines
Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from
head.SQuoted 13 lines
Quoted 13 lines
to
head_mmu.SHi Wei,
Quoted 4 lines · Wei Chen
There are lots of MMU specific code in
head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code tohead_mmu.S, and keep common code still inhead.S.I am afraid that you can't simply move the MMU code out of
head.Sbecause this will break Xen when running using the identity map.Quoted 8 lines
Quoted 8 lines
This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before
_end_bootand checking at link time if this fits in 4KB.Now that you moved the MMU code outside of
head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identitymapped.
Quoted 8 lines
Quoted 1 line
Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.Quoted 6 lines
Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"OK. We will create a patch to check the size, and place it in the head of the series.
Quoted 8 lines
I thought a bit more about what I wrote. Xen binary should always start with the Image/Zimage header. At the moment, this is guaranteed because there is only one object using the section .text.header.
With the change introduced in this patch, there will be multiple objects using with the sections .text.header. This means we are relying on the compiler to always put the content of
head.Sfirst.
Yes, now it depends on the objects' order in Makefile. This should be a risk.
Quoted 7 lines
This is basically reverting to the behavior before commit
4267a33b19d("xen/build: put image header into a separate section").Therefore we do need a separate section to be used for head_*.S and maybe part of
head.S. This new section could be called text.idmap so we know what the section is used for.
Yes, we might place this new section after text.header.
Cheers,
Wei Chen
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Wei ChenHi Wei,
Quoted 85 lines · Wei Chen
CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.
So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And
XEN_START_ADDRESSis one kind of FVP BaseR platform's parameters. So we define defaultXEN_START_ADDRESSfor FVP BaseR in its platform file.We also introduce one
Kconfigoption for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For thisKconfigoption, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.
Signed-off-by Wei Chen <[email protected]>Signed-off-by Jiamei.Xie <[email protected]>--- xen/arch/arm/Kconfig | 11 +++++++++++ xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++ xen/arch/arm/platforms/Kconfig | 16 +++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)create mode 100644
xen/arch/arm/include/asm/platforms/fvp_baser.hxen/arch/arm/Kconfig +11
-0@@ -138,6 +138,17 @@ config TEE This option enables generic TEE mediators support. It allows guests to access real TEE via one of TEE mediators implemented in XEN. +config XEN_START_ADDRESS + hex "Xen start address: keep default to use platform defined address" + default 0xFFFFFFFF + depends on HAS_MPU + help + This option allows to set the customized address at which Xen will be + linked on MPU systems. This address must be aligned to a page size. + Use 0xFFFFFFFF as the default value to indicate that user hasn't + customized this address, and Xen use use the default value that has + been defined in platform files. + source "arch/arm/tee/Kconfig" config STATIC_SHMxen/arch/arm/include/asm/platforms/fvp_baser.h new file +14
-0@@ -0,0 +1,14 @@ +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__ +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__ + +/* +* 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS, +* we will use platform defined default address. +*/ +#if CONFIG_XEN_START_ADDRESS ==0xFFFFFFFF +#define XEN_START_ADDRESS 0x200000 +#else +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS +#endif + +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */ xen/arch/arm/platforms/Kconfig +0
-0
xen/arch/arm/platforms/Kconfig line 4
Quoted 4 lines
xen/arch/arm/platforms/Kconfig +1
-0@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8R
I could not spot the patch which introduced ARM_V8R.
Can you rename this to ARM64_V8R ? The reason being the underlying code is specific to R82 ie 64 bit V8R.
- Ayan
Quoted 56 lines
xen/arch/arm/platforms/Kconfig +12
-4---help--- Choose which hardware platform to enable in Xen.@@ -8,13 +9,14 @@ choice config ALL_PLAT bool "All Platforms" + depends on !ARM_V8R ---help--- Enable support for all available hardware platforms. It doesn't automatically select any of the related drivers. config QEMU bool "QEMU aarch virt machine support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select GICV3 select HAS_PL011 ---help---@@ -23,7 +25,7 @@ config QEMU config RCAR3 bool "Renesas RCar3 support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_SCIF select IPMMU_VMSA ---help---@@ -31,14 +33,22 @@ config RCAR3 config MPSOC bool "Xilinx Ultrascale+ MPSoC support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_CADENCE_UART select ARM_SMMU ---help--- Enable all the required drivers for Xilinx Ultrascale+ MPSoC +config FVP_BASER + bool "Fixed Virtual Platform BaseR support" + depends on ARM_V8R + help + Enable platform specific configurations for Fixed Virtual + Platform BaseR + config NO_PLAT bool "No Platforms" + depends on !ARM_V8R ---help--- Do not enable specific support for any platform. --2.25.1
RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
Hi Ayan,
Quoted 4 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */(quoted diff) +0
-0b/xen/arch/arm/platforms/Kconfig
Quoted 11 lines
xen/arch/arm/platforms/Kconfig line 4
Quoted 8 lines
index
c93a6b2756..0904793a0b100644xen/arch/arm/platforms/Kconfig +1
-0@@ -1,6 +1,7 @@ choice prompt "Platform Support" default ALL_PLAT + default FVP_BASER if ARM_V8RI could not spot the patch which introduced
ARM_V8R.
That patch is not in this part, it will be the last one of MPU support patch series. You can find it gitlab branch's full series.
Quoted 3 lines
Can you rename this to
ARM64_V8R? The reason being the underlying code is specific to R82 ie 64 bit V8R.
I renamed ARM64_V8R (in RFC patch) to ARM_V8R is because "Arm64 v8r" is not an official Arm architecture name. The Arm official name is Armv8-R AArch32/AArch64. And currently, MPU will only be selected by Arm64, so current MPU code can only work in AArch64 state. When you're trying to enable Armv8-R AArch32 like R52, you can remove this limitation, and use CONFIG_ARM64 or CONFIG_ARM32 to distinguish code between r82 and r52 code.
Cheers,
Wei Chen
Quoted 58 lines
- Ayan
Quoted 56 lines
xen/arch/arm/platforms/Kconfig +12
-4---help--- Choose which hardware platform to enable in Xen.@@ -8,13 +9,14 @@ choice config ALL_PLAT bool "All Platforms" + depends on !ARM_V8R ---help--- Enable support for all available hardware platforms. It doesn't automatically select any of the related drivers. config QEMU bool "QEMU aarch virt machine support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select GICV3 select HAS_PL011 ---help---@@ -23,7 +25,7 @@ config QEMU config RCAR3 bool "Renesas RCar3 support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_SCIF select IPMMU_VMSA ---help---@@ -31,14 +33,22 @@ config RCAR3 config MPSOC bool "Xilinx Ultrascale+ MPSoC support" - depends on ARM_64 + depends on ARM_64 && !ARM_V8R select HAS_CADENCE_UART select ARM_SMMU ---help--- Enable all the required drivers for Xilinx Ultrascale+ MPSoC +config FVP_BASER + bool "Fixed Virtual Platform BaseR support" + depends on ARM_V8R + help + Enable platform specific configurations for Fixed Virtual + Platform BaseR + config NO_PLAT bool "No Platforms" + depends on !ARM_V8R ---help--- Do not enable specific support for any platform. --2.25.1
Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems
In reply to Julien GrallHi Julien,
Quoted 10 lines · Julien Grall
Hi Wei,
Quoted 4 lines · Wei Chen
Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.
I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?
Sorry, I had missed this comment. Yes, you're right we had tried to boot Xen R82 Image from EFI loader, but it failed. UEFI loader will load Xen to a random address, which is not supported by Xen R82 now.
Cheers,
Wei Chen
Quoted 4 lines
Quoted 4 lines
But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
In reply to Julien GrallHi Julien,
I thought I had replied to your email, but when I am checking to see if I have addressed all your comments, I didn't find my reply on the mailing list. Maybe there is something wrong with my mail system, I didn't find it sooner. Sorry about it. Hope my reply is not too late.
Quoted 23 lines · Julien Grall
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 6 lines
Quoted 1 line
We had considered to use
Kconfigto define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value,Kconfigoption for customized address).Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?
Quoted 4 lines
We also had thought to use
Kconfigto define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
So if we don’t use header files, just use the
Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.
I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.
Above comments have been answered in my reply to you and Stefano.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Review
Quoted 1 line
(quoted diff) +0
-0b/xen/arch/arm/platforms/Kconfig
Quoted 8 lines
index
c93a6b2756..0904793a0b100644xen/arch/arm/platforms/Kconfig +0
-0@@ -1,6 +1,7 @@choice
prompt "Platform Support"
defaultALL_PLAT
+ defaultFVP_BASERifARM_V8RIs there any reason to create a new
Kconfigrather than using MPU?
Quoted 6 lines
Quoted 3 lines
Did you mean
FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.See above for this.
If we move board specific information to tailored config file, I think we don't need FVP_BASER.
Quoted 5 lines
Quoted 2 lines
If you meant Armv8R, that's because Armv8R does not equal to MPU.
I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?
No, I didn't mean that. I just think we can't use only one MPU or one ARM_V8R to cover all our changes in this series. For example, some changes like new system registers are brought by Armv8R not the MPU.
Cheers,
Wei Chen
Quoted 2 lines
Cheers,
Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform
Hi,
Quoted 26 lines · Wei Chen
On 2022/11/10 2:24, Julien Grall wrote:
(quoted diff) +0
-0Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
b/xen/arch/arm/platforms/Kconfig
Quoted 8 lines
index
c93a6b2756..0904793a0b100644xen/arch/arm/platforms/Kconfig +0
-0@@ -1,6 +1,7 @@choice
prompt "Platform Support"
defaultALL_PLAT
+ defaultFVP_BASERifARM_V8RIs there any reason to create a new
Kconfigrather than using MPU?Quoted 6 lines
Quoted 3 lines
Did you mean
FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.See above for this.
If we move board specific information to tailored config file, I think we don't need
FVP_BASER.Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
If you meant Armv8R, that's because Armv8R does not equal to MPU.
I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?
No, I didn't mean that. I just think we can't use only one MPU or one
ARM_V8Rto cover all our changes in this series. For example, some changes like new system registers are brought by Armv8R not the MPU.
I understand the theory. But in practice this needs to be a balance between finer grain and too much Kconfig.
From this series alone, it doesn't seem to be make sense to introduce the two. Furthermore, I am not entirely sure you will be able to make the MPU work without enable the ARMv8-R Kconfig.
Cheers,
Signature · 1 line
Julien Grall