Skip to contentSkip to conversation

xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

PATCH In review Wei Chen 6 participants 63 messages Nov 4, 2022 – Dec 5, 2022

lore
Share & tools
lore

Wei Chen

author
#0 lore raw

v6 cover xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.

We will split these patches to several parts, this series is the part#1, the full PoC can be found in [3]. More software for Armv8-R64 can be found in [4];

[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/

Penny Zheng (3):
xen/arm64: create boot-time MPU protection regions xen/arm64: introduce helpers for MPU enable/disable xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Wei Chen (8):
xen/arm: remove xen_phys_start and xenheap_phys_end from config.h xen/arm: add iounmap after initrd has been loaded in domain_build xen/arm: disable EFI boot services for MPU systems xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA xen/arm: define Xen start address for FVP BaseR platform xen/arm: split MMU and MPU config files from config.h xen/arm: implement FIXMAP_ADDR for MPU systems xen/arm64: move MMU related code from head.S to head_mmu.S

 xen/arch/arm/Kconfig                          |  15 +-
 xen/arch/arm/arm64/Makefile                   |   5 +
 xen/arch/arm/arm64/head.S                     | 429 ++----------------
 xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
 xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
 xen/arch/arm/domain_build.c                   |   2 +
 xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
 xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
 xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
 xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
 xen/arch/arm/include/asm/config.h             |  99 +---
 xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
 xen/arch/arm/include/asm/config_mpu.h         |  29 ++
 xen/arch/arm/include/asm/fixmap.h             |  25 +
 xen/arch/arm/include/asm/flushtlb.h           |  22 +
 .../arm/include/asm/platforms/fvp_baser.h     |  18 +
 xen/arch/arm/platforms/Kconfig                |  16 +-
 17 files changed, 976 insertions(+), 500 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v6 1/11 xen/arm: remove xen_phys_start and xenheap_phys_end from config.h

These two variables are stale variables, they only have declarations in config.h, they don't have any definition and no any code is using these two variables. So in this patch, we remove them from config.h.

---
 xen/arch/arm/include/asm/config.h | 2 --
 1 file changed, 2 deletions(-)
xen/arch/arm/include/asm/config.h +0 -2
index 0fefed1b8a..25a625ff08 100644
@@ -172,8 +172,6 @@
 #define STACK_SIZE  (PAGE_SIZE << STACK_ORDER)
 
 #ifndef __ASSEMBLY__
-extern unsigned long xen_phys_start;
-extern unsigned long xenheap_phys_end;
 extern unsigned long frametable_virt_end;
 #endif
 
Signature · 1 line
2.25.1
Acked-by on v6 1/11
#2 lore raw

Re: [PATCH v6 01/11] xen/arm: remove xen_phys_start and xenheap_phys_end from config.h

Hi,

Quoted 1 line · Wei Chen

These two variables are stale variables, they only have declarations

AFAICT, this has always been the case. I am guessing this was because the header was mostly likely copied from x86...

Quoted 4 lines

in config.h, they don't have any definition and no any code is using these two variables. So in this patch, we remove them from config.h.

Cheers,

Signature · 1 line
Julien Grall
author
#3 lore raw

PATCH v6 2/11 xen/arm: add iounmap after initrd has been loaded in domain_build

In reply to Wei Chen

domain_build use ioremap_wc to map a new non-cacheable virtual address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.

So in this patch, we add an iounmap to the end of domain_build, after Xen loaded initrd to guest memory.

---
 xen/arch/arm/domain_build.c | 2 ++
 1 file changed, 2 insertions(+)
xen/arch/arm/domain_build.c +2 -0
index 4fb5c20b13..bd30d3798c 100644
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo)
                                           initrd, len);
     if ( res != 0 )
         panic("Unable to copy the initrd in the hwdom memory\n");
+
+    iounmap(initrd);
 }
 
 /*
Signature · 1 line
2.25.1
#4 lore raw

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

(+ Henry)

Hi,

Quoted 1 line · Wei Chen

domain_build use ioremap_wc to map a new non-cacheable virtual

s/use/uses/

Quoted 6 lines

address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.

So in this patch, we add an iounmap to the end of domain_build, after Xen loaded initrd to guest memory.

Please a fixes tag. The issue was introduced by commit bb7e6d565d92.

Quoted 10 lines
---
  xen/arch/arm/domain_build.c | 2 ++
  1 file changed, 2 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index 4fb5c20b13..bd30d3798c 100644

xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422

Quoted 5 lines
xen/arch/arm/domain_build.c +2 -0
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo)
                                            initrd, len);
      if ( res != 0 )
          panic("Unable to copy the initrd in the hwdom memory\n");
+
+    iounmap(initrd);

This looks good to me. But I am wondering whether using ioremap_wc() is actually correct because we are reading the region. So it seems strang to map it with write-combine.

So I would consider to use ioremap_cache(). That said, this would be a separate patch.

I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Cheers,

Signature · 1 line
Julien Grall
#5 lore raw

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

Hi Julien,

Quoted 8 lines

-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Sounds good to me, I am wondering if we need to include also patch #1 though.

If this patch wants to be in 4.17:

Kind regards,
Henry

Quoted 5 lines

Cheers,

--
Julien Grall

#6 lore raw

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

Quoted 1 line · Henry Wang

Hi Julien,

Hi Henry,

Quoted 11 lines
Quoted 9 lines

-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Sounds good to me, I am wondering if we need to include also patch #1 though.

If we were earlier in the release, I would have say yes. But now, it is a no as this just a cleanup (having extra definitions is harmless).

Quoted 4 lines

If this patch wants to be in 4.17:

Thanks!

Cheers,

Signature · 1 line
Julien Grall
#7 lore raw

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

Hi Julien,

Quoted 8 lines

-----Original Message-----
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

Quoted 5 lines
Quoted 4 lines

(+ Henry)
I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Quoted 5 lines
Quoted 1 line

Sounds good to me, I am wondering if we need to include also patch #1

though.

If we were earlier in the release, I would have say yes. But now, it is a no as this just a cleanup (having extra definitions is harmless).

Thanks for your confirmation :) No problem.

Kind regards,
Henry

#8 lore raw

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

In reply to Julien Grall
Quoted 9 lines · Julien Grall

(+ Henry)

Hi,

Quoted 1 line · Wei Chen

domain_build use ioremap_wc to map a new non-cacheable virtual

s/use/uses/

Quoted 8 lines
Quoted 6 lines

address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.

So in this patch, we add an iounmap to the end of domain_build, after Xen loaded initrd to guest memory.

Please a fixes tag. The issue was introduced by commit bb7e6d565d92.

Well I forgot to add it on commit :/. I will try to remember to backport it.

Cheers,

Signature · 1 line
Julien Grall
author
#9 lore raw

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

In reply to Julien Grall

Hi Julien,

Quoted 46 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: 2022年11月7日 2:55
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

(+ Henry)

Hi,

Quoted 1 line · Wei Chen

domain_build use ioremap_wc to map a new non-cacheable virtual

s/use/uses/

Quoted 6 lines

address for initrd. After Xen copy initrd from this address to guest, this new allocated virtual address has not been unmapped.

So in this patch, we add an iounmap to the end of domain_build, after Xen loaded initrd to guest memory.

Please a fixes tag. The issue was introduced by commit bb7e6d565d92.

xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info

Quoted 10 lines
---
  xen/arch/arm/domain_build.c | 2 ++
  1 file changed, 2 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index 4fb5c20b13..bd30d3798c 100644
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info

*kinfo)

xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422

Quoted 5 lines
xen/arch/arm/domain_build.c +2 -0
                                            initrd, len);
      if ( res != 0 )
          panic("Unable to copy the initrd in the hwdom memory\n");
+
+    iounmap(initrd);
Quoted 7 lines

This looks good to me. But I am wondering whether using ioremap_wc() is actually correct because we are reading the region. So it seems strang to map it with write-combine.

So I would consider to use ioremap_cache(). That said, this would be a separate patch.

Ok, we will try to use ioremap_cache and test it. If everything works well we will introduce a separate patch in next version.

Cheers,
Wei Chen

Quoted 8 lines

I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Cheers,

--
Julien Grall

author
#10 lore raw

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

Quoted 19 lines
Quoted 7 lines

xen/arch/arm/domain_build.c static void __init initrd_load(struct

Quoted 7 lines
  1 file changed, 2 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index 4fb5c20b13..bd30d3798c 100644
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct

kernel_info

Quoted 11 lines

*kinfo)

xen/arch/arm/domain_build.c static void __init initrd_load(struct kernel_info *kinfo) · line 3422

Quoted 5 lines
xen/arch/arm/domain_build.c +2 -0
                                            initrd, len);
      if ( res != 0 )
          panic("Unable to copy the initrd in the hwdom memory\n");
+
+    iounmap(initrd);

This looks good to me. But I am wondering whether using ioremap_wc() is actually correct because we are reading the region. So it seems strang to map it with write-combine.

Quoted 7 lines
Quoted 3 lines

So I would consider to use ioremap_cache(). That said, this would be a separate patch.

Ok, we will try to use ioremap_cache and test it. If everything works well we will introduce a separate patch in next version.

Or is it better to send a separate patch for this? Because I think we might need something to address the v1 comments.

Cheers,
Wei Chen

Quoted 12 lines

Cheers,
Wei Chen

Quoted 8 lines

I think this wants to be in 4.17. This will avoid Xen to have two mappings with different caching attribute (initrd is part of the RAM and therefore directmap).

Cheers,

--
Julien Grall

author
#11 lore raw

PATCH v6 3/11 xen/arm: disable EFI boot services for MPU systems

In reply to Wei Chen

Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support. But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.

---
 xen/arch/arm/Kconfig      | 2 +-
 xen/arch/arm/arm64/head.S | 8 ++++++--
 2 files changed, 7 insertions(+), 3 deletions(-)
xen/arch/arm/Kconfig +1 -1
index 1fe5faf847..ad592367bd 100644
@@ -7,7 +7,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
-	select ARM_EFI
+	select ARM_EFI if !HAS_MPU
 	select HAS_FAST_MULTIPLY
 
 config ARM
xen/arch/arm/arm64/head.S +6 -2
index ad014716db..ccedf20dc7 100644
@@ -172,8 +172,10 @@ efi_head:
         .byte   0x52
         .byte   0x4d
         .byte   0x64
-        .long   pe_header - efi_head        /* Offset to the PE header. */
-
+#ifndef CONFIG_ARM_EFI
+        .long   0                    /* 0 means no PE header. */
+#else
+        .long   pe_header - efi_head /* Offset to the PE header. */
         /*
          * Add the PE/COFF header to the file.  The address of this header
          * is at offset 0x3c in the file, and is part of Linux "Image"
@@ -279,6 +281,8 @@ section_table:
         .short  0                /* NumberOfLineNumbers  (0 for executables) */
         .long   0xe0500020       /* Characteristics (section flags) */
         .align  5
+#endif /* CONFIG_ARM_EFI */
+
 real_start:
         /* BSS should be zeroed when booting without EFI */
         mov   x26, #0                /* x26 := skip_zero_bss */
Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

Hi Wei,

Quoted 4 lines · Wei Chen

Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.

I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?

Quoted 15 lines

But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.

---
  xen/arch/arm/Kconfig      | 2 +-
  xen/arch/arm/arm64/head.S | 8 ++++++--
  2 files changed, 7 insertions(+), 3 deletions(-)
xen/arch/arm/Kconfig +0 -0
index 1fe5faf847..ad592367bd 100644

xen/arch/arm/Kconfig config ARM_64 · line 10

Quoted 5 lines
xen/arch/arm/Kconfig +1 -1
@@ -7,7 +7,7 @@ config ARM_64
  	def_bool y
  	depends on !ARM_32
  	select 64BIT
-	select ARM_EFI
+	select ARM_EFI if !HAS_MPU

I think it would make sense to allow ARM_EFI to be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.

Quoted 21 lines
xen/arch/arm/Kconfig +0 -0
  	select HAS_FAST_MULTIPLY
 
  config ARM
xen/arch/arm/arm64/head.S +4 -2
index ad014716db..ccedf20dc7 100644
@@ -172,8 +172,10 @@ efi_head:
          .byte   0x52
          .byte   0x4d
          .byte   0x64
-        .long   pe_header - efi_head        /* Offset to the PE header. */
-
+#ifndef CONFIG_ARM_EFI
+        .long   0                    /* 0 means no PE header. */
+#else
+        .long   pe_header - efi_head /* Offset to the PE header. */
          /*
           * Add the PE/COFF header to the file.  The address of this header
           * is at offset 0x3c in the file, and is part of Linux "Image"

xen/arch/arm/arm64/head.S section_table: · line 288

Quoted 8 lines
xen/arch/arm/arm64/head.S +2 -0
@@ -279,6 +281,8 @@ section_table:
          .short  0                /* NumberOfLineNumbers  (0 for executables) */
          .long   0xe0500020       /* Characteristics (section flags) */
          .align  5
+#endif /* CONFIG_ARM_EFI */
+
  real_start:
          /* BSS should be zeroed when booting without EFI */
          mov   x26, #0                /* x26 := skip_zero_bss */

Shouldn't the function efi_xen_start be stubbed as well?

Cheers,

Signature · 1 line
Julien Grall
#13 lore raw

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

Quoted 56 lines · Julien Grall

Hi Wei,

Quoted 4 lines · Wei Chen

Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.

I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?

Quoted 15 lines

But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.

---
  xen/arch/arm/Kconfig      | 2 +-
  xen/arch/arm/arm64/head.S | 8 ++++++--

  2 files changed, 7 insertions(+), 3 deletions(-)
xen/arch/arm/Kconfig +0 -0
index 1fe5faf847..ad592367bd 100644
@@ -7,7 +7,7 @@ config ARM_64
Quoted 5 lines

      def_bool y
      depends on !ARM_32
      select 64BIT
-    select ARM_EFI
+    select ARM_EFI if !HAS_MPU

I think it would make sense to allow ARM_EFI to be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.

Quoted 23 lines

      select HAS_FAST_MULTIPLY
  config ARM

xen/arch/arm/arm64/head.S +0 -0
index ad014716db..ccedf20dc7 100644
@@ -172,8 +172,10 @@ efi_head:

          .byte   0x52
          .byte   0x4d
          .byte   0x64
-        .long   pe_header - efi_head        /* Offset to the PE header. */

xen/arch/arm/arm64/head.S +4 -1
-
+#ifndef CONFIG_ARM_EFI
+        .long   0                    /* 0 means no PE header. */
+#else
+        .long   pe_header - efi_head /* Offset to the PE header. */

          /*
           * Add the PE/COFF header to the file.  The address of this header
           * is at offset 0x3c in the file, and is part of Linux "Image"

xen/arch/arm/arm64/head.S +0 -0
@@ -279,6 +281,8 @@ section_table:

          .short  0                /* NumberOfLineNumbers  (0 for executables) */ .long   0xe0500020       /* Characteristics (section flags) */

Quoted 8 lines
Quoted 6 lines

          .align  5

xen/arch/arm/arm64/head.S +2 -0
+#endif /* CONFIG_ARM_EFI */
+

  real_start:
          /* BSS should be zeroed when booting without EFI */ mov   x26, #0                /* x26 := skip_zero_bss */

Shouldn't the function efi_xen_start be stubbed as well?

Sorry, I mean protected rather than stubbed because there will be no user when !CONFIG_ARM_EFI.

Cheers,

Signature · 1 line
Julien Grall
author
#14 lore raw

RE: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

In reply to Julien Grall

Hi Julien,

Quoted 32 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:12
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

Hi Wei,

Quoted 4 lines · Wei Chen

Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.

I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?

Quoted 10 lines

But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.

---
  xen/arch/arm/Kconfig      | 2 +-
  xen/arch/arm/arm64/head.S | 8 ++++++--
  2 files changed, 7 insertions(+), 3 deletions(-)
Quoted 15 lines

xen/arch/arm/Kconfig config ARM_64 · line 10

Quoted 10 lines
xen/arch/arm/Kconfig +1 -1
index 1fe5faf847..ad592367bd 100644
@@ -7,7 +7,7 @@ config ARM_64
  	def_bool y
  	depends on !ARM_32
  	select 64BIT
-	select ARM_EFI
+	select ARM_EFI if !HAS_MPU

I think it would make sense to allow ARM_EFI to be disabled even without the MPU support. So this would remove nearly 3K lines (just using wc -l *.c in the efi directories) for someone that don't need to boot using EFI.

Ok, how about address this comment in NUMA patch set#3 (Arm implementation), because about making ARM_EFI invisible to users, we had some discussions for NUMA RFC and V1.

Quoted 25 lines

xen/arch/arm/arm64/head.S efi_head: · line 175

Quoted 12 lines
xen/arch/arm/Kconfig +0 -0
  	select HAS_FAST_MULTIPLY
 
  config ARM
xen/arch/arm/arm64/head.S +0 -1
index ad014716db..ccedf20dc7 100644
@@ -172,8 +172,10 @@ efi_head:
          .byte   0x52
          .byte   0x4d
          .byte   0x64
-        .long   pe_header - efi_head        /* Offset to the PE header.

*/

xen/arch/arm/arm64/head.S efi_head:

Quoted 7 lines
xen/arch/arm/arm64/head.S +4 -1
-
+#ifndef CONFIG_ARM_EFI
+        .long   0                    /* 0 means no PE header. */
+#else
+        .long   pe_header - efi_head /* Offset to the PE header. */
          /*
           * Add the PE/COFF header to the file.  The address of this

header

xen/arch/arm/arm64/head.S section_table: · line 281

Quoted 3 lines
xen/arch/arm/arm64/head.S +0 -0
           * is at offset 0x3c in the file, and is part of Linux "Image"
@@ -279,6 +281,8 @@ section_table:
          .short  0                /* NumberOfLineNumbers  (0 for

executables) */

Quoted 10 lines

xen/arch/arm/arm64/head.S section_table: · line 288

Quoted 7 lines
xen/arch/arm/arm64/head.S +2 -0
          .long   0xe0500020       /* Characteristics (section flags) */
          .align  5
+#endif /* CONFIG_ARM_EFI */
+
  real_start:
          /* BSS should be zeroed when booting without EFI */
          mov   x26, #0                /* x26 := skip_zero_bss */

Shouldn't the function efi_xen_start be stubbed as well?

Reply for your next email:
Yes, this is a good point, efi_xen_start should be protected. I will fix it.

Cheers,
Wei Chen

Quoted 4 lines

Cheers,

--
Julien Grall

author
#15 lore raw

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

In reply to Julien Grall

Hi Julien,

Quoted 10 lines · Julien Grall

Hi Wei,

Quoted 4 lines · Wei Chen

Current EFI boot services support of Arm64 could not work well for Armv8-R64 system that only has MPU in EL2. That is because EFI boot services may need some relocation support or partial PIE/PIC support.

I am a bit confused with argument. We have nothing in Xen today to deal with relocation/partial PIE/PIC support. So what is the exact problem? Is it because UEFI can load Xen anywwhere?

Sorry, I had missed this comment. Yes, you're right we had tried to boot Xen R82 Image from EFI loader, but it failed. UEFI loader will load Xen to a random address, which is not supported by Xen R82 now.

Cheers,
Wei Chen

Quoted 4 lines
Quoted 4 lines

But these will not be supported in the initial stage of porting Xen to MPU systems. So in this patch, we disable EFI boot services support for Arm MPU systems.

author
#16 lore raw

PATCH v6 4/11 xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA

In reply to Wei Chen

From Arm ARM Supplement of Armv8-R AArch64 (DDI 0600A) [1], section D1.6.2 TLB maintenance instructions. We know that Armv8-R AArch64 permits an implementation to cache stage 1 VMSAv8-64 and stage 2 PMSAv8-64 attributes as a common entry for the Secure EL1&0 translation regime. But for Xen itself, it's running with stage 1 PMSAv8-64 on Armv8-R AArch64. The EL2 MPU updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we don't need any TLB invalidation for Xen itself in EL2.

So in this patch, we use empty macros to stub Xen TLB helpers for MPU system (PMSA), but still keep the Guest TLB helpers. Because when a guest running in EL1 with VMSAv8-64 (MMU), guest TLB invalidation is still needed. But we need some policy to distinguish MPU and MMU guest, this will be done in guest support of Armv8-R64 later.

[1] https://developer.arm.com/documentation/ddi0600/ac

---
 xen/arch/arm/include/asm/arm64/flushtlb.h | 25 +++++++++++++++++++++++
 xen/arch/arm/include/asm/flushtlb.h       | 22 ++++++++++++++++++++
 2 files changed, 47 insertions(+)
xen/arch/arm/include/asm/arm64/flushtlb.h +25 -0
index 7c54315187..fe445f6831 100644
@@ -51,6 +51,8 @@ TLB_HELPER(flush_all_guests_tlb_local, alle1);
 /* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */
 TLB_HELPER(flush_all_guests_tlb, alle1is);
 
+#ifndef CONFIG_HAS_MPU
+
 /* Flush all hypervisor mappings from the TLB of the local processor. */
 TLB_HELPER(flush_xen_tlb_local, alle2);
 
@@ -66,6 +68,29 @@ static inline void __flush_xen_tlb_one(vaddr_t va)
     asm volatile("tlbi vae2is, %0;" : : "r" (va>>PAGE_SHIFT) : "memory");
 }
 
+#else
+
+/*
+ * When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU
+ * updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we
+ * don't need any TLB invalidation for Xen itself in EL2. See Arm ARM
+ * Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance
+ * instructions for more details.
+ */
+static inline void flush_xen_tlb_local(void)
+{
+}
+
+static inline void  __flush_xen_tlb_one_local(vaddr_t va)
+{
+}
+
+static inline void __flush_xen_tlb_one(vaddr_t va)
+{
+}
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */
 /*
  * Local variables:
xen/arch/arm/include/asm/flushtlb.h +22 -0
index 125a141975..4b8bf65281 100644
@@ -28,6 +28,7 @@ static inline void page_set_tlbflush_timestamp(struct page_info *page)
 /* Flush specified CPUs' TLBs */
 void arch_flush_tlb_mask(const cpumask_t *mask);
 
+#ifndef CONFIG_HAS_MPU
 /*
  * Flush a range of VA's hypervisor mappings from the TLB of the local
  * processor.
@@ -66,6 +67,27 @@ static inline void flush_xen_tlb_range_va(vaddr_t va,
     isb();
 }
 
+#else
+
+/*
+ * When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU
+ * updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we
+ * don't need any TLB invalidation for Xen itself in EL2. See Arm ARM
+ * Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance
+ * instructions for more details.
+ */
+static inline void flush_xen_tlb_range_va_local(vaddr_t va,
+                                                unsigned long size)
+{
+}
+
+static inline void flush_xen_tlb_range_va(vaddr_t va,
+                                          unsigned long size)
+{
+}
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_ARM_FLUSHTLB_H__ */
 /*
  * Local variables:
Signature · 1 line
2.25.1
author
#17 lore raw

PATCH v6 5/11 xen/arm: define Xen start address for FVP BaseR platform

In reply to Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

---
 xen/arch/arm/Kconfig                           | 11 +++++++++++
 xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
 xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
 3 files changed, 38 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
xen/arch/arm/Kconfig +11 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
 	  This option enables generic TEE mediators support. It allows guests
 	  to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined address"
+	default 0xFFFFFFFF
+	depends on HAS_MPU
+	help
+	  This option allows to set the customized address at which Xen will be
+	  linked on MPU systems. This address must be aligned to a page size.
+	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
+	  customized this address, and Xen use use the default value that has
+	  been defined in platform files.
+
 source "arch/arm/tee/Kconfig"
 
 config STATIC_SHM
xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14 -0
new file mode 100644
index 0000000000..9450a411a9
@@ -0,0 +1,14 @@
+#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
+#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
+
+/*
+ * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
+ * we will use platform defined default address.
+ */
+#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
+#define XEN_START_ADDRESS 0x200000
+#else
+#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
+#endif
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
xen/arch/arm/platforms/Kconfig +13 -3
index c93a6b2756..0904793a0b 100644
@@ -1,6 +1,7 @@
 choice
 	prompt "Platform Support"
 	default ALL_PLAT
+	default FVP_BASER if ARM_V8R
 	---help---
 	Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
 config ALL_PLAT
 	bool "All Platforms"
+	depends on !ARM_V8R
 	---help---
 	Enable support for all available hardware platforms. It doesn't
 	automatically select any of the related drivers.
 
 config QEMU
 	bool "QEMU aarch virt machine support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select GICV3
 	select HAS_PL011
 	---help---
@@ -23,7 +25,7 @@ config QEMU
 
 config RCAR3
 	bool "Renesas RCar3 support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select HAS_SCIF
 	select IPMMU_VMSA
 	---help---
@@ -31,14 +33,22 @@ config RCAR3
 
 config MPSOC
 	bool "Xilinx Ultrascale+ MPSoC support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select HAS_CADENCE_UART
 	select ARM_SMMU
 	---help---
 	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+	bool "Fixed Virtual Platform BaseR support"
+	depends on ARM_V8R
+	help
+	  Enable platform specific configurations for Fixed Virtual
+	  Platform BaseR
+
 config NO_PLAT
 	bool "No Platforms"
+	depends on !ARM_V8R
 	---help---
 	Do not enable specific support for any platform.
 
Signature · 1 line
2.25.1
#18 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
---
  xen/arch/arm/Kconfig                           | 11 +++++++++++
  xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

Quoted 9 lines
  xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
  3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644

xen/arch/arm/Kconfig config TEE · line 143

Quoted 6 lines
xen/arch/arm/Kconfig +3 -0
@@ -138,6 +138,17 @@ config TEE
  	  This option enables generic TEE mediators support. It allows guests
  	  to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined address"
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the Kconfig to select the proper address (see the config for selecting early printk address).

This will also avoids to use an invalid value here.

Quoted 36 lines
xen/arch/arm/Kconfig +8 -0
+	depends on HAS_MPU
+	help
+	  This option allows to set the customized address at which Xen will be
+	  linked on MPU systems. This address must be aligned to a page size.
+	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
+	  customized this address, and Xen use use the default value that has
+	  been defined in platform files.
+
  source "arch/arm/tee/Kconfig"
 
  config STATIC_SHM
xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14 -0
new file mode 100644
index 0000000000..9450a411a9
@@ -0,0 +1,14 @@
+#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
+#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
+
+/*
+ * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
+ * we will use platform defined default address.
+ */
+#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
+#define XEN_START_ADDRESS 0x200000
+#else
+#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
+#endif
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
xen/arch/arm/platforms/Kconfig +0 -0
index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig line 4

Quoted 4 lines
xen/arch/arm/platforms/Kconfig +1 -0
@@ -1,6 +1,7 @@
  choice
  	prompt "Platform Support"
  	default ALL_PLAT
+	default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

Quoted 42 lines
xen/arch/arm/platforms/Kconfig +7 -3
  	---help---
  	Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
  config ALL_PLAT
  	bool "All Platforms"
+	depends on !ARM_V8R
  	---help---
  	Enable support for all available hardware platforms. It doesn't
  	automatically select any of the related drivers.
 
  config QEMU
  	bool "QEMU aarch virt machine support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select GICV3
  	select HAS_PL011
  	---help---
@@ -23,7 +25,7 @@ config QEMU
 
  config RCAR3
  	bool "Renesas RCar3 support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select HAS_SCIF
  	select IPMMU_VMSA
  	---help---
@@ -31,14 +33,22 @@ config RCAR3
 
  config MPSOC
  	bool "Xilinx Ultrascale+ MPSoC support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select HAS_CADENCE_UART
  	select ARM_SMMU
  	---help---
  	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+	bool "Fixed Virtual Platform BaseR support"
+	depends on ARM_V8R

xen/arch/arm/platforms/Kconfig config RCAR3 · line 54

Quoted 10 lines
xen/arch/arm/platforms/Kconfig +5 -0
+	help
+	  Enable platform specific configurations for Fixed Virtual
+	  Platform BaseR
+
  config NO_PLAT
  	bool "No Platforms"
+	depends on !ARM_V8R
  	---help---
  	Do not enable specific support for any platform.
 

Cheers,

Signature · 1 line
Julien Grall
author
#19 lore raw

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Hi Julien,

Quoted 65 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
---
  xen/arch/arm/Kconfig                           | 11 +++++++++++
  xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

xen/arch/arm/Kconfig config TEE · line 138

Quoted 10 lines
  xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
  3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
  	  This option enables generic TEE mediators support. It allows

guests

xen/arch/arm/Kconfig config TEE

Quoted 4 lines
xen/arch/arm/Kconfig +2 -0
  	  to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined

address"

Quoted 9 lines

xen/arch/arm/Kconfig config TEE · line 143

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the Kconfig to select the proper address (see the config for selecting early printk address).

This will also avoids to use an invalid value here.

We had considered to use Kconfig to define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value, Kconfig option for customized address).

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

[1] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[2] https://gitlab.com/xen-project/fusa/xen-integration/-/blob/integration/mpu/xen/arch/arm/Kconfig.debug

Quoted 36 lines

xen/arch/arm/Kconfig config TEE

Quoted 3 lines
xen/arch/arm/Kconfig +3 -0
+	depends on HAS_MPU
+	help
+	  This option allows to set the customized address at which Xen will

be

xen/arch/arm/Kconfig config TEE

Quoted 3 lines
xen/arch/arm/Kconfig +3 -0
+	  linked on MPU systems. This address must be aligned to a page size.
+	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
+	  customized this address, and Xen use use the default value that

has

Review

Quoted 6 lines
xen/arch/arm/Kconfig +2 -0
+	  been defined in platform files.
+
  source "arch/arm/tee/Kconfig"
 
  config STATIC_SHM
(quoted diff) +0 -0

b/xen/arch/arm/include/asm/platforms/fvp_baser.h

Review

Quoted 20 lines

new file mode 100644
index 0000000000..9450a411a9

xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14 -0
@@ -0,0 +1,14 @@
+#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
+#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
+
+/*
+ * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
+ * we will use platform defined default address.
+ */
+#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
+#define XEN_START_ADDRESS 0x200000
+#else
+#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
+#endif
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
(quoted diff) +0 -0

b/xen/arch/arm/platforms/Kconfig

Quoted 11 lines

xen/arch/arm/platforms/Kconfig line 4

Quoted 8 lines

index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig +1 -0
@@ -1,6 +1,7 @@
  choice
  	prompt "Platform Support"
  	default ALL_PLAT
+	default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

Did you mean FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.

If you meant Armv8R, that's because Armv8R does not equal to MPU. We only allow Armv8R code to detect MPU features. MPU is configurable In EL2.

Cheers,
Wei Chen

Quoted 57 lines
Quoted 42 lines
xen/arch/arm/platforms/Kconfig +7 -3
  	---help---
  	Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
  config ALL_PLAT
  	bool "All Platforms"
+	depends on !ARM_V8R
  	---help---
  	Enable support for all available hardware platforms. It doesn't
  	automatically select any of the related drivers.
 
  config QEMU
  	bool "QEMU aarch virt machine support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select GICV3
  	select HAS_PL011
  	---help---
@@ -23,7 +25,7 @@ config QEMU
 
  config RCAR3
  	bool "Renesas RCar3 support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select HAS_SCIF
  	select IPMMU_VMSA
  	---help---
@@ -31,14 +33,22 @@ config RCAR3
 
  config MPSOC
  	bool "Xilinx Ultrascale+ MPSoC support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
  	select HAS_CADENCE_UART
  	select ARM_SMMU
  	---help---
  	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+	bool "Fixed Virtual Platform BaseR support"
+	depends on ARM_V8R

xen/arch/arm/platforms/Kconfig config RCAR3 · line 54

Quoted 10 lines
xen/arch/arm/platforms/Kconfig +5 -0
+	help
+	  Enable platform specific configurations for Fixed Virtual
+	  Platform BaseR
+
  config NO_PLAT
  	bool "No Platforms"
+	depends on !ARM_V8R
  	---help---
  	Do not enable specific support for any platform.
 

Cheers,

--
Julien Grall

#20 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 73 lines
Quoted 73 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
---
   xen/arch/arm/Kconfig                           | 11 +++++++++++
   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

xen/arch/arm/Kconfig config TEE · line 138

Quoted 10 lines
   xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
   3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
   	  This option enables generic TEE mediators support. It allows

guests

xen/arch/arm/Kconfig config TEE

Quoted 4 lines
xen/arch/arm/Kconfig +2 -0
   	  to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined

address"

xen/arch/arm/Kconfig config TEE · line 143

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the Kconfig to select the proper address (see the config for selecting early printk address).

Quoted 7 lines
Quoted 2 lines

This will also avoids to use an invalid value here.

We had considered to use Kconfig to define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value, Kconfig option for customized address).

Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?

Quoted 4 lines

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).

Quoted 4 lines

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.

Quoted 14 lines
Quoted 13 lines

Review

Quoted 1 line
(quoted diff) +0 -0

b/xen/arch/arm/platforms/Kconfig

xen/arch/arm/platforms/Kconfig line 4

Quoted 8 lines

index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig +1 -0
@@ -1,6 +1,7 @@
   choice
   	prompt "Platform Support"
   	default ALL_PLAT
+	default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

Quoted 3 lines

Did you mean FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.

See above for this.

Quoted 2 lines

If you meant Armv8R, that's because Armv8R does not equal to MPU.

I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?

Cheers,

Signature · 1 line
Julien Grall
#21 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 98 lines · Julien Grall
Quoted 73 lines
Quoted 73 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
---
   xen/arch/arm/Kconfig                           | 11 +++++++++++
   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

xen/arch/arm/Kconfig config TEE · line 138

Quoted 10 lines
   xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
   3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
   	  This option enables generic TEE mediators support. It allows

guests

xen/arch/arm/Kconfig

Quoted 5 lines
xen/arch/arm/Kconfig +0 -0
   	  to access real TEE via one of TEE mediators implemented in

XEN.

xen/arch/arm/Kconfig +2 -0
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined

address"

xen/arch/arm/Kconfig config TEE · line 143

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least for now, I think you can avoid the per platform header by using the Kconfig to select the proper address (see the config for selecting early printk address).

Quoted 7 lines
Quoted 2 lines

This will also avoids to use an invalid value here.

We had considered to use Kconfig to define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value, Kconfig option for customized address).

Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?

Quoted 4 lines

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).

Quoted 4 lines

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

Quoted 7 lines

I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.

I agree with Julien, especially on this last point.

Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.

The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.

Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand the existing "Platform Support" kconfig menu.

If this was just XEN_START_ADDRESS, I would suggest to keep it in xen.git somewhere. But given that there are a few addresses and sizes to provide/calculate for Xen on MPU to work, using ImageBuilder could be a good idea.

author
#22 lore raw

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Hi Stefano, Julien,

Quoted 144 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2022年11月11日 6:13
To: Julien Grall <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 5 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for

FVP

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines

BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
---
   xen/arch/arm/Kconfig                           | 11 +++++++++++
   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14

++++++++++++++

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

Quoted 1 line
   xen/arch/arm/platforms/Kconfig                 | 16

+++++++++++++---

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
   3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644

xen/arch/arm/include/asm/platforms/fvp_baser.h

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

xen/arch/arm/Kconfig config TEE · line 138

Quoted 7 lines
xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
   	  This option enables generic TEE mediators support. It allows

guests

Quoted 3 lines
xen/arch/arm/Kconfig +0 -0
   	  to access real TEE via one of TEE mediators implemented in

XEN.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/Kconfig

Quoted 2 lines
xen/arch/arm/Kconfig +2 -0
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined

address"

xen/arch/arm/Kconfig config TEE · line 143

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least

for

Quoted 1 line
Quoted 1 line
Quoted 1 line

now, I think you can avoid the per platform header by using the

Kconfig

Quoted 1 line
Quoted 1 line
Quoted 1 line

to select the proper address (see the config for selecting early

printk

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

address).

This will also avoids to use an invalid value here.

We had considered to use Kconfig to define the start addresses of

v8R64

Quoted 1 line
Quoted 1 line

platforms (prompt users to input the address). But we also want to

provide

Quoted 1 line
Quoted 1 line

a default start address for each platform (Discussed in [1], header

for

Quoted 2 lines
Quoted 1 line

default value, Kconfig option for customized address).

Why do you want to provide a default value? And how it is guaranteed

that it

Quoted 8 lines

will work for most of the users?

Quoted 4 lines

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back

the

Quoted 1 line

addition of new one because the address can be found in the firmware

table and

Quoted 1 line

I wanted to avoid increase the number of option (there are dozens of

platform

Quoted 8 lines

out...).

Quoted 4 lines

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

I am not sure I see the problem to force the user to enter the start

address.

Quoted 1 line

My worry with per-platform default value is we end up to force each

vendor to

Quoted 3 lines

provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that

platform

Quoted 1 line

(whether it is part of Xen can be debatable). This would allow a user to

try a

Quoted 1 line

release Xen on their platform with zero changes in the code.

I agree with Julien, especially on this last point.

Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.

Quoted 6 lines

The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.

Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand

Do you mean we can keep a config like armv8r_fvp_baser_config in xen/arch/arm/configs for users to generate a default config? If yes I think this method might be better for now. And about ImageBuilder solution we can do it after MPU support be merged?

Cheers,
Wei Chen

Quoted 6 lines

the existing "Platform Support" kconfig menu.

If this was just XEN_START_ADDRESS, I would suggest to keep it in xen.git somewhere. But given that there are a few addresses and sizes to provide/calculate for Xen on MPU to work, using ImageBuilder could be a good idea.

#23 lore raw

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 150 lines · Wei Chen

Hi Stefano, Julien,

Quoted 148 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2022年11月11日 6:13
To: Julien Grall <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Quoted 5 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:20
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Jiamei Xie <[email protected]>
Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for

FVP

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines

BaseR platform

Quoted 26 lines · Wei Chen

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
---
   xen/arch/arm/Kconfig                           | 11 +++++++++++
   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14

++++++++++++++

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I looked at the content of fvp_baser.h after this series is applied. There are a bit of boiler plate that I expect to be part for other platforms. In particular...

Quoted 1 line
   xen/arch/arm/platforms/Kconfig                 | 16

+++++++++++++---

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
   3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644

xen/arch/arm/include/asm/platforms/fvp_baser.h

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

xen/arch/arm/Kconfig config TEE · line 138

Quoted 7 lines
xen/arch/arm/Kconfig +0 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
   	  This option enables generic TEE mediators support. It allows

guests

Quoted 3 lines
xen/arch/arm/Kconfig +0 -0
   	  to access real TEE via one of TEE mediators implemented in

XEN.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/Kconfig

Quoted 2 lines
xen/arch/arm/Kconfig +2 -0
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined

address"

xen/arch/arm/Kconfig config TEE · line 143

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least

for

Quoted 1 line
Quoted 1 line
Quoted 1 line

now, I think you can avoid the per platform header by using the

Kconfig

Quoted 1 line
Quoted 1 line
Quoted 1 line

to select the proper address (see the config for selecting early

printk

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

address).

This will also avoids to use an invalid value here.

We had considered to use Kconfig to define the start addresses of

v8R64

Quoted 1 line
Quoted 1 line

platforms (prompt users to input the address). But we also want to

provide

Quoted 1 line
Quoted 1 line

a default start address for each platform (Discussed in [1], header

for

Quoted 2 lines
Quoted 1 line

default value, Kconfig option for customized address).

Why do you want to provide a default value? And how it is guaranteed

that it

Quoted 8 lines

will work for most of the users?

Quoted 4 lines

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back

the

Quoted 1 line

addition of new one because the address can be found in the firmware

table and

Quoted 1 line

I wanted to avoid increase the number of option (there are dozens of

platform

Quoted 8 lines

out...).

Quoted 4 lines

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

I am not sure I see the problem to force the user to enter the start

address.

Quoted 1 line

My worry with per-platform default value is we end up to force each

vendor to

Quoted 3 lines

provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that

platform

Quoted 1 line

(whether it is part of Xen can be debatable). This would allow a user to

try a

Quoted 1 line

release Xen on their platform with zero changes in the code.

I agree with Julien, especially on this last point.

Of course we need a default configuration for a given platform, we don't want every user of the same platform to have to go and look at the manual to find the right address to use.

The question is where to put the per-platform default value. The kconfig "default" keyword is not great for that and it is not realistic to have a single address that works everywhere.

Quoted 5 lines
Quoted 2 lines

Instead, we could have a prepopulated kconfig under xen/arch/arm/configs, or something under ImageBuilder, or maybe expand

Do you mean we can keep a config like armv8r_fvp_baser_config in xen/arch/arm/configs for users to generate a default config?

Yes

Quoted 2 lines

If yes I think this method might be better for now. And about ImageBuilder solution we can do it after MPU support be merged?

That's fine

author
#24 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

In reply to Julien Grall

Hi Julien,

I thought I had replied to your email, but when I am checking to see if I have addressed all your comments, I didn't find my reply on the mailing list. Maybe there is something wrong with my mail system, I didn't find it sooner. Sorry about it. Hope my reply is not too late.

Quoted 23 lines · Julien Grall
Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 6 lines
Quoted 1 line

We had considered to use Kconfig to define the start addresses of v8R64 platforms (prompt users to input the address). But we also want to provide a default start address for each platform (Discussed in [1], header for default value, Kconfig option for customized address).

Why do you want to provide a default value? And how it is guaranteed that it will work for most of the users?

Quoted 4 lines

We also had thought to use Kconfig to define a default start address for each platform like what we had done for early printk in RFC[2]. But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back the addition of new one because the address can be found in the firmware table and I wanted to avoid increase the number of option (there are dozens of platform out...).

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

So if we don’t use header files, just use the Kconfig, we can't provide a default start address for platforms, and have to force users to enter the start address.

I am not sure I see the problem to force the user to enter the start address. My worry with per-platform default value is we end up to force each vendor to provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that platform (whether it is part of Xen can be debatable). This would allow a user to try a release Xen on their platform with zero changes in the code.

Above comments have been answered in my reply to you and Stefano.

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

Review

Quoted 1 line
(quoted diff) +0 -0

b/xen/arch/arm/platforms/Kconfig

Quoted 8 lines

index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig +0 -0
@@ -1,6 +1,7 @@

   choice
       prompt "Platform Support"
       default ALL_PLAT
+    default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

Quoted 6 lines
Quoted 3 lines

Did you mean FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.

See above for this.

If we move board specific information to tailored config file, I think we don't need FVP_BASER.

Quoted 5 lines
Quoted 2 lines

If you meant Armv8R, that's because Armv8R does not equal to MPU.

I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?

No, I didn't mean that. I just think we can't use only one MPU or one ARM_V8R to cover all our changes in this series. For example, some changes like new system registers are brought by Armv8R not the MPU.

Cheers,
Wei Chen

Quoted 2 lines

Cheers,

#25 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Hi,

Quoted 26 lines · Wei Chen

On 2022/11/10 2:24, Julien Grall wrote:

(quoted diff) +0 -0
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

b/xen/arch/arm/platforms/Kconfig

Quoted 8 lines

index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig +0 -0
@@ -1,6 +1,7 @@

   choice
       prompt "Platform Support"
       default ALL_PLAT
+    default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

Quoted 6 lines
Quoted 3 lines

Did you mean FVP_BASER? If yes, we want to give each board a MACRO to indicate its specific configurations. In current series, this MACRO only be used for board specific start address.

See above for this.

If we move board specific information to tailored config file, I think we don't need FVP_BASER.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If you meant Armv8R, that's because Armv8R does not equal to MPU.

I am not entirely sure to understand. Are you saying that an existing Xen can boot on Armv8R?

No, I didn't mean that. I just think we can't use only one MPU or one ARM_V8R to cover all our changes in this series. For example, some changes like new system registers are brought by Armv8R not the MPU.

I understand the theory. But in practice this needs to be a balance between finer grain and too much Kconfig.

From this series alone, it doesn't seem to be make sense to introduce the two. Furthermore, I am not entirely sure you will be able to make the MPU work without enable the ARMv8-R Kconfig.

Cheers,

Signature · 1 line
Julien Grall
#26 lore raw

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

In reply to Wei Chen

Hi Wei,

Quoted 85 lines · Wei Chen

CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

On Armv8-A, Xen has a fixed virtual start address (link address too) for all Armv8-A platforms. In an MMU based system, Xen can map its loaded address to this virtual start address. So, on Armv8-A platforms, the Xen start address does not need to be configurable. But on Armv8-R platforms, there is no MMU to map loaded address to a fixed virtual address and different platforms will have very different address space layout. So Xen cannot use a fixed physical address on MPU based system and need to have it configurable.

So in this patch, we reuse the existing arm/platforms to store Armv8-R platforms' parameters. And XEN_START_ADDRESS is one kind of FVP BaseR platform's parameters. So we define default XEN_START_ADDRESS for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the default Xen start address of selected platform, if they think the default address doesn't suit their scenarios. For this Kconfig option, we use an unaligned address "0xffffffff" as the default value to indicate that users haven't used a customized Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the existed Arm64 platforms should not be listed in Armv8-R platform list, so we add !ARM_V8R dependency for these platforms.

---
  xen/arch/arm/Kconfig                           | 11 +++++++++++
  xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
  xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
  3 files changed, 38 insertions(+), 3 deletions(-)

create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

xen/arch/arm/Kconfig +11 -0
index ad592367bd..ac276307d6 100644
@@ -138,6 +138,17 @@ config TEE
           This option enables generic TEE mediators support. It allows guests
           to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+       hex "Xen start address: keep default to use platform defined address"
+       default 0xFFFFFFFF
+       depends on HAS_MPU
+       help
+         This option allows to set the customized address at which Xen will be
+         linked on MPU systems. This address must be aligned to a page size.
+         Use 0xFFFFFFFF as the default value to indicate that user hasn't
+         customized this address, and Xen use use the default value that has
+         been defined in platform files.
+
  source "arch/arm/tee/Kconfig"
 
  config STATIC_SHM
xen/arch/arm/include/asm/platforms/fvp_baser.h new file +14 -0
new file mode 100644
index 0000000000..9450a411a9
@@ -0,0 +1,14 @@
+#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
+#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
+
+/*
+ * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
+ * we will use platform defined default address.
+ */
+#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
+#define XEN_START_ADDRESS 0x200000
+#else
+#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
+#endif
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
xen/arch/arm/platforms/Kconfig +0 -0
index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig line 4

Quoted 4 lines
xen/arch/arm/platforms/Kconfig +1 -0
@@ -1,6 +1,7 @@
  choice
         prompt "Platform Support"
         default ALL_PLAT
+       default FVP_BASER if ARM_V8R

I could not spot the patch which introduced ARM_V8R.

Can you rename this to ARM64_V8R ? The reason being the underlying code is specific to R82 ie 64 bit V8R.

- Ayan

Quoted 56 lines
xen/arch/arm/platforms/Kconfig +12 -4
         ---help---
         Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
  config ALL_PLAT
         bool "All Platforms"
+       depends on !ARM_V8R
         ---help---
         Enable support for all available hardware platforms. It doesn't
         automatically select any of the related drivers.
 
  config QEMU
         bool "QEMU aarch virt machine support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select GICV3
         select HAS_PL011
         ---help---
@@ -23,7 +25,7 @@ config QEMU
 
  config RCAR3
         bool "Renesas RCar3 support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select HAS_SCIF
         select IPMMU_VMSA
         ---help---
@@ -31,14 +33,22 @@ config RCAR3
 
  config MPSOC
         bool "Xilinx Ultrascale+ MPSoC support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select HAS_CADENCE_UART
         select ARM_SMMU
         ---help---
         Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+       bool "Fixed Virtual Platform BaseR support"
+       depends on ARM_V8R
+       help
+         Enable platform specific configurations for Fixed Virtual
+         Platform BaseR
+
  config NO_PLAT
         bool "No Platforms"
+       depends on !ARM_V8R
         ---help---
         Do not enable specific support for any platform.
 
--

2.25.1

author
#27 lore raw

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

Hi Ayan,

Quoted 4 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
(quoted diff) +0 -0

b/xen/arch/arm/platforms/Kconfig

Quoted 11 lines

xen/arch/arm/platforms/Kconfig line 4

Quoted 8 lines

index c93a6b2756..0904793a0b 100644

xen/arch/arm/platforms/Kconfig +1 -0
@@ -1,6 +1,7 @@
  choice
         prompt "Platform Support"
         default ALL_PLAT
+       default FVP_BASER if ARM_V8R

I could not spot the patch which introduced ARM_V8R.

That patch is not in this part, it will be the last one of MPU support patch series. You can find it gitlab branch's full series.

Quoted 3 lines

Can you rename this to ARM64_V8R ? The reason being the underlying code is specific to R82 ie 64 bit V8R.

I renamed ARM64_V8R (in RFC patch) to ARM_V8R is because "Arm64 v8r" is not an official Arm architecture name. The Arm official name is Armv8-R AArch32/AArch64. And currently, MPU will only be selected by Arm64, so current MPU code can only work in AArch64 state. When you're trying to enable Armv8-R AArch32 like R52, you can remove this limitation, and use CONFIG_ARM64 or CONFIG_ARM32 to distinguish code between r82 and r52 code.

Cheers,
Wei Chen

Quoted 58 lines

- Ayan

Quoted 56 lines
xen/arch/arm/platforms/Kconfig +12 -4
         ---help---
         Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
  config ALL_PLAT
         bool "All Platforms"
+       depends on !ARM_V8R
         ---help---
         Enable support for all available hardware platforms. It doesn't
         automatically select any of the related drivers.
 
  config QEMU
         bool "QEMU aarch virt machine support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select GICV3
         select HAS_PL011
         ---help---
@@ -23,7 +25,7 @@ config QEMU
 
  config RCAR3
         bool "Renesas RCar3 support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select HAS_SCIF
         select IPMMU_VMSA
         ---help---
@@ -31,14 +33,22 @@ config RCAR3
 
  config MPSOC
         bool "Xilinx Ultrascale+ MPSoC support"
-       depends on ARM_64
+       depends on ARM_64 && !ARM_V8R
         select HAS_CADENCE_UART
         select ARM_SMMU
         ---help---
         Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+       bool "Fixed Virtual Platform BaseR support"
+       depends on ARM_V8R
+       help
+         Enable platform specific configurations for Fixed Virtual
+         Platform BaseR
+
  config NO_PLAT
         bool "No Platforms"
+       depends on !ARM_V8R
         ---help---
         Do not enable specific support for any platform.
 
--

2.25.1

author
#28 lore raw

PATCH v6 6/11 xen/arm: split MMU and MPU config files from config.h

In reply to Wei Chen

Xen defines some global configuration macros for Arm in config.h. We still want to use it for Armv8-R systems, but there are some address related macros that are defined for MMU systems. These macros will not be used by MPU systems, Adding ifdefery with CONFIG_HAS_MPU to gate these macros will result in a messy and hard to read/maintain code.

So we keep some common definitions still in config.h, but move virtual address related definitions to a new file - config_mmu.h. And use a new file config_mpu.h to store definitions for MPU systems. To avoid spreading #ifdef everywhere, we keep the same definition names for MPU systems, like XEN_VIRT_START and HYPERVISOR_VIRT_START, but the definition contents are MPU specific.

---
 xen/arch/arm/include/asm/config.h     |  97 +--------------------
 xen/arch/arm/include/asm/config_mmu.h | 119 ++++++++++++++++++++++++++
 xen/arch/arm/include/asm/config_mpu.h |  27 ++++++
 3 files changed, 150 insertions(+), 93 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
xen/arch/arm/include/asm/config.h +4 -93
index 25a625ff08..189e932605 100644
@@ -71,99 +71,10 @@
 #include <xen/const.h>
 #include <xen/page-size.h>
 
-/*
- * Common ARM32 and ARM64 layout:
- *   0  -   2M   Unmapped
- *   2M -   4M   Xen text, data, bss
- *   4M -   6M   Fixmap: special-purpose 4K mapping slots
- *   6M -  10M   Early boot mapping of FDT
- *   10M - 12M   Livepatch vmap (if compiled in)
- *
- * ARM32 layout:
- *   0  -  12M   <COMMON>
- *
- *  32M - 128M   Frametable: 24 bytes per page for 16GB of RAM
- * 256M -   1G   VMAP: ioremap and early_ioremap use this virtual address
- *                    space
- *
- *   1G -   2G   Xenheap: always-mapped memory
- *   2G -   4G   Domheap: on-demand-mapped
- *
- * ARM64 layout:
- * 0x0000000000000000 - 0x0000007fffffffff (512GB, L0 slot [0])
- *   0  -  12M   <COMMON>
- *
- *   1G -   2G   VMAP: ioremap and early_ioremap
- *
- *  32G -  64G   Frametable: 24 bytes per page for 5.3TB of RAM
- *
- * 0x0000008000000000 - 0x00007fffffffffff (127.5TB, L0 slots [1..255])
- *  Unused
- *
- * 0x0000800000000000 - 0x000084ffffffffff (5TB, L0 slots [256..265])
- *  1:1 mapping of RAM
- *
- * 0x0000850000000000 - 0x0000ffffffffffff (123TB, L0 slots [266..511])
- *  Unused
- */
-
-#define XEN_VIRT_START         _AT(vaddr_t,0x00200000)
-#define FIXMAP_ADDR(n)        (_AT(vaddr_t,0x00400000) + (n) * PAGE_SIZE)
-
-#define BOOT_FDT_VIRT_START    _AT(vaddr_t,0x00600000)
-#define BOOT_FDT_VIRT_SIZE     _AT(vaddr_t, MB(4))
-
-#ifdef CONFIG_LIVEPATCH
-#define LIVEPATCH_VMAP_START   _AT(vaddr_t,0x00a00000)
-#define LIVEPATCH_VMAP_SIZE    _AT(vaddr_t, MB(2))
-#endif
-
-#define HYPERVISOR_VIRT_START  XEN_VIRT_START
-
-#ifdef CONFIG_ARM_32
-
-#define CONFIG_SEPARATE_XENHEAP 1
-
-#define FRAMETABLE_VIRT_START  _AT(vaddr_t,0x02000000)
-#define FRAMETABLE_SIZE        MB(128-32)
-#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
-#define FRAMETABLE_VIRT_END    (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1)
-
-#define VMAP_VIRT_START        _AT(vaddr_t,0x10000000)
-#define VMAP_VIRT_SIZE         _AT(vaddr_t, GB(1) - MB(256))
-
-#define XENHEAP_VIRT_START     _AT(vaddr_t,0x40000000)
-#define XENHEAP_VIRT_SIZE      _AT(vaddr_t, GB(1))
-
-#define DOMHEAP_VIRT_START     _AT(vaddr_t,0x80000000)
-#define DOMHEAP_VIRT_SIZE      _AT(vaddr_t, GB(2))
-
-#define DOMHEAP_ENTRIES        1024  /* 1024 2MB mapping slots */
-
-/* Number of domheap pagetable pages required at the second level (2MB mappings) */
-#define DOMHEAP_SECOND_PAGES (DOMHEAP_VIRT_SIZE >> FIRST_SHIFT)
-
-#else /* ARM_64 */
-
-#define SLOT0_ENTRY_BITS  39
-#define SLOT0(slot) (_AT(vaddr_t,slot) << SLOT0_ENTRY_BITS)
-#define SLOT0_ENTRY_SIZE  SLOT0(1)
-
-#define VMAP_VIRT_START  GB(1)
-#define VMAP_VIRT_SIZE   GB(1)
-
-#define FRAMETABLE_VIRT_START  GB(32)
-#define FRAMETABLE_SIZE        GB(32)
-#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
-
-#define DIRECTMAP_VIRT_START   SLOT0(256)
-#define DIRECTMAP_SIZE         (SLOT0_ENTRY_SIZE * (265-256))
-#define DIRECTMAP_VIRT_END     (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1)
-
-#define XENHEAP_VIRT_START     directmap_virt_start
-
-#define HYPERVISOR_VIRT_END    DIRECTMAP_VIRT_END
-
+#ifdef CONFIG_HAS_MPU
+#include <asm/config_mpu.h>
+#else
+#include <asm/config_mmu.h>
 #endif
 
 #define NR_hypercalls 64
xen/arch/arm/include/asm/config_mmu.h new file +119 -0

119 lines not on this page · raw message

xen/arch/arm/include/asm/config_mpu.h new file +27 -0
new file mode 100644
index 0000000000..530abb8302
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * config_mpu.h: A Linux-style configuration list for Arm MPU systems,
+ *               only can be included by config.h
+ */
+
+#ifndef __ARM_CONFIG_MPU_H__
+#define __ARM_CONFIG_MPU_H__
+
+#ifdef CONFIG_FVP_BASER
+#include <asm/platforms/fvp_baser.h>
+#endif
+
+/*
+ * All MPU platforms need to provide a XEN_START_ADDRESS for linker. This
+ * address indicates where Xen image will be loaded and run from. This
+ * address must be aligned to a PAGE_SIZE.
+ */
+#if (XEN_START_ADDRESS % PAGE_SIZE) != 0
+#error "XEN_START_ADDRESS must be aligned to PAGE_SIZE"
+#endif
+
+#define XEN_VIRT_START         _AT(paddr_t, XEN_START_ADDRESS)
+
+#define HYPERVISOR_VIRT_START  XEN_VIRT_START
+
+#endif /* __ARM_CONFIG_MPU_H__ */
Signature · 1 line
2.25.1
author
#29 lore raw

PATCH v6 7/11 xen/arm: implement FIXMAP_ADDR for MPU systems

In reply to Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using FIXMAP_ADDR to return the input physical address in MPU systems. So in MPU version, FIXMAP_ADDR will trim physical address to PAGE alignment. This will return an offset which is similar to MMU version FIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask with PAGE_MASK). The caller can return the input physical address directly.

As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.

---
 xen/arch/arm/Kconfig                  |  2 +-
 xen/arch/arm/include/asm/config_mpu.h |  2 ++
 xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -1
index ac276307d6..1458ffa777 100644
@@ -16,7 +16,7 @@ config ARM
 	select HAS_DEVICE_TREE
 	select HAS_PASSTHROUGH
 	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU
 	select IOMMU_FORCE_PT_SHARE
 
 config ARCH_DEFCONFIG
xen/arch/arm/include/asm/config_mpu.h +2 -0
index 530abb8302..eee60dcffc 100644
@@ -24,4 +24,6 @@
 
 #define HYPERVISOR_VIRT_START  XEN_VIRT_START
 
+#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
+
 #endif /* __ARM_CONFIG_MPU_H__ */
xen/arch/arm/include/asm/fixmap.h +25 -0
index d0c9a52c8c..1e338759e9 100644
@@ -7,6 +7,8 @@
 #include <xen/acpi.h>
 #include <xen/pmap.h>
 
+#ifndef CONFIG_HAS_MPU
+
 /* Fixmap slots */
 #define FIXMAP_CONSOLE  0  /* The primary UART */
 #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
@@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
 
 #endif /* __ASSEMBLY__ */
 
+#else
+
+/*
+ * FIXMAP_ADDR will trim physical address to PAGE alignment.
+ * This will return an offset which is similar to MMU version
+ * FIXMAP_ADDR.
+ * For example:
+ * EARLY_UART_VIRTUAL_ADDRESS is defined by:
+ *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
+ *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
+ * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
+ * EARLY_UART_VIRTUAL_ADDRESS can be restore to
+ * CONFIG_EARLY_UART_BASE_ADDRESS.
+ * In this case, we don't need to use #ifdef MPU in the code
+ * where are using FIXMAP_ADDR to make them to use physical
+ * address explicitily.
+ */
+#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
+#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
+#endif
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_FIXMAP_H */
Signature · 1 line
2.25.1
#30 lore raw

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Wei,

Quoted 6 lines · Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.

Quoted 26 lines

But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using FIXMAP_ADDR to return the input physical address in MPU systems. So in MPU version, FIXMAP_ADDR will trim physical address to PAGE alignment. This will return an offset which is similar to MMU version FIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask with PAGE_MASK). The caller can return the input physical address directly.

As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.

---
  xen/arch/arm/Kconfig                  |  2 +-
  xen/arch/arm/include/asm/config_mpu.h |  2 ++
  xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
  3 files changed, 28 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +0 -0
index ac276307d6..1458ffa777 100644

xen/arch/arm/Kconfig config ARM · line 19

Quoted 5 lines
xen/arch/arm/Kconfig +1 -1
@@ -16,7 +16,7 @@ config ARM
  	select HAS_DEVICE_TREE
  	select HAS_PASSTHROUGH
  	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU

I can't find any change of mm.c in this series. So surely this will break the build?

Quoted 45 lines
xen/arch/arm/Kconfig +0 -0
  	select IOMMU_FORCE_PT_SHARE
 
  config ARCH_DEFCONFIG
xen/arch/arm/include/asm/config_mpu.h +2 -0
index 530abb8302..eee60dcffc 100644
@@ -24,4 +24,6 @@
 
  #define HYPERVISOR_VIRT_START  XEN_VIRT_START
 
+#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
+
  #endif /* __ARM_CONFIG_MPU_H__ */
xen/arch/arm/include/asm/fixmap.h +16 -0
index d0c9a52c8c..1e338759e9 100644
@@ -7,6 +7,8 @@
  #include <xen/acpi.h>
  #include <xen/pmap.h>
 
+#ifndef CONFIG_HAS_MPU
+
  /* Fixmap slots */
  #define FIXMAP_CONSOLE  0  /* The primary UART */
  #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
@@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
 
  #endif /* __ASSEMBLY__ */
 
+#else
+
+/*
+ * FIXMAP_ADDR will trim physical address to PAGE alignment.
+ * This will return an offset which is similar to MMU version
+ * FIXMAP_ADDR.
+ * For example:
+ * EARLY_UART_VIRTUAL_ADDRESS is defined by:
+ *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
+ *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
+ * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
+ * EARLY_UART_VIRTUAL_ADDRESS can be restore to
+ * CONFIG_EARLY_UART_BASE_ADDRESS.
+ * In this case, we don't need to use #ifdef MPU in the code

xen/arch/arm/include/asm/fixmap.h static inline unsigned int virt_to_fix(vaddr_t vaddr) · line 73

Quoted 10 lines
xen/arch/arm/include/asm/fixmap.h +9 -0
+ * where are using FIXMAP_ADDR to make them to use physical
+ * address explicitily.
+ */
+#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
+#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
+#endif
+
+#endif /* CONFIG_HAS_MPU */
+
  #endif /* __ASM_FIXMAP_H */

Cheers,

Signature · 1 line
Julien Grall
author
#31 lore raw

RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Julien,

Quoted 20 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Wei,

Quoted 6 lines · Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

Quoted 5 lines

To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.

I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic code. I also prefer this way. Frankly, I really think FIXMAP is awkward in MPU System.

Quoted 21 lines
Quoted 21 lines

But in MPU systems, we can't map physical address to any virtual address. So we want the code that is using FIXMAP_ADDR to return the input physical address in MPU systems. So in MPU version, FIXMAP_ADDR will trim physical address to PAGE alignment. This will return an offset which is similar to MMU version FIXMAP_ADDR. But it's a physical offset got from input physical address, plus to an offset inside page (which is also got from physical address mask with PAGE_MASK). The caller can return the input physical address directly.

As pmap depends on FIXAMP, so we disable pmap for Arm with MPU enabled systems.

---
  xen/arch/arm/Kconfig                  |  2 +-
  xen/arch/arm/include/asm/config_mpu.h |  2 ++
  xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
  3 files changed, 28 insertions(+), 1 deletion(-)
Quoted 13 lines

xen/arch/arm/Kconfig config ARM · line 19

Quoted 10 lines
xen/arch/arm/Kconfig +1 -1
index ac276307d6..1458ffa777 100644
@@ -16,7 +16,7 @@ config ARM
  	select HAS_DEVICE_TREE
  	select HAS_PASSTHROUGH
  	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU

I can't find any change of mm.c in this series. So surely this will break the build?

Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.

Quoted 63 lines

Review

Quoted 5 lines
xen/arch/arm/Kconfig +0 -0
  	select IOMMU_FORCE_PT_SHARE
 
  config ARCH_DEFCONFIG
(quoted diff) +0 -0

b/xen/arch/arm/include/asm/config_mpu.h

Review

Quoted 11 lines

index 530abb8302..eee60dcffc 100644

xen/arch/arm/include/asm/config_mpu.h +2 -0
@@ -24,4 +24,6 @@
 
  #define HYPERVISOR_VIRT_START  XEN_VIRT_START
 
+#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
+
  #endif /* __ARM_CONFIG_MPU_H__ */
(quoted diff) +0 -0

b/xen/arch/arm/include/asm/fixmap.h

Quoted 30 lines

index d0c9a52c8c..1e338759e9 100644

xen/arch/arm/include/asm/fixmap.h +16 -0
@@ -7,6 +7,8 @@
  #include <xen/acpi.h>
  #include <xen/pmap.h>
 
+#ifndef CONFIG_HAS_MPU
+
  /* Fixmap slots */
  #define FIXMAP_CONSOLE  0  /* The primary UART */
  #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
@@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
 
  #endif /* __ASSEMBLY__ */
 
+#else
+
+/*
+ * FIXMAP_ADDR will trim physical address to PAGE alignment.
+ * This will return an offset which is similar to MMU version
+ * FIXMAP_ADDR.
+ * For example:
+ * EARLY_UART_VIRTUAL_ADDRESS is defined by:
+ *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
+ *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
+ * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
+ * EARLY_UART_VIRTUAL_ADDRESS can be restore to
+ * CONFIG_EARLY_UART_BASE_ADDRESS.
+ * In this case, we don't need to use #ifdef MPU in the code

xen/arch/arm/include/asm/fixmap.h static inline unsigned int virt_to_fix(vaddr_t vaddr) · line 73

Quoted 10 lines
xen/arch/arm/include/asm/fixmap.h +9 -0
+ * where are using FIXMAP_ADDR to make them to use physical
+ * address explicitily.
+ */
+#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
+#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
+#endif
+
+#endif /* CONFIG_HAS_MPU */
+
  #endif /* __ASM_FIXMAP_H */

Cheers,

--
Julien Grall

#32 lore raw

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 21 lines
Quoted 21 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Wei,

Quoted 6 lines · Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

Quoted 7 lines
Quoted 5 lines

To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.

I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic code.

FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for HAS_MPU would do the work to map the region in the MPU.

[...]

Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
   xen/arch/arm/Kconfig                  |  2 +-
   xen/arch/arm/include/asm/config_mpu.h |  2 ++
   xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
   3 files changed, 28 insertions(+), 1 deletion(-)
Quoted 13 lines

xen/arch/arm/Kconfig config ARM · line 19

Quoted 10 lines
xen/arch/arm/Kconfig +1 -1
index ac276307d6..1458ffa777 100644
@@ -16,7 +16,7 @@ config ARM
   	select HAS_DEVICE_TREE
   	select HAS_PASSTHROUGH
   	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU

I can't find any change of mm.c in this series. So surely this will break the build?

Quoted 2 lines

Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.

Do you mean you added some stubs for PMAP? If so, I would not expect any caller for the pmap() to be used for the MPU. Therefore, why would they be necessary?

Cheers,

Signature · 1 line
Julien Grall
author
#33 lore raw

RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Julien,

Quoted 45 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月10日 2:30
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 21 lines
Quoted 20 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Wei,

Quoted 6 lines · Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.

Quoted 7 lines
Quoted 1 line

I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic

code.

FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for HAS_MPU would do the work to map the region in the MPU.

Sorry, I am still confused about this comment, did you mean we can provider Some generic helpers like: early_map_console / eary_map_guest_memory. For non-MPU system, we still can call fixmap in these callers, but for MPU system, we have to map the region to MPU region?

Quoted 22 lines

[...]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
   xen/arch/arm/Kconfig                  |  2 +-
   xen/arch/arm/include/asm/config_mpu.h |  2 ++
   xen/arch/arm/include/asm/fixmap.h     | 25

+++++++++++++++++++++++++

Quoted 16 lines
Quoted 15 lines

xen/arch/arm/Kconfig config ARM · line 19

Quoted 12 lines
   3 files changed, 28 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -1
index ac276307d6..1458ffa777 100644
@@ -16,7 +16,7 @@ config ARM
   	select HAS_DEVICE_TREE
   	select HAS_PASSTHROUGH
   	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU

I can't find any change of mm.c in this series. So surely this will break the build?

Quoted 7 lines
Quoted 2 lines

Yes, in our internal testing, open PMAP for MPU will cause building failed, except we add some new stubs for MPU system.

Do you mean you added some stubs for PMAP? If so, I would not expect any caller for the pmap() to be used for the MPU. Therefore, why would they be necessary?

No, I mean if we want to make pmap can be built successfully for MPU, we have to implement some stubs like: fix_to_virt, xen_fixmap and write_pte, to make compiling success. But just as you said, we would not expect MPU to use any PMAP function, so we have not implemented them for MPU. Instead we disable PMAP for MPU.

Cheer,
Wei Chen

Quoted 4 lines

Cheers,

--
Julien Grall

#34 lore raw

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Quoted 55 lines · Wei Chen

Hi Julien,

Quoted 45 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月10日 2:30
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 3:45
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 21 lines
Quoted 20 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

Hi Wei,

Quoted 6 lines · Wei Chen

FIXMAP is a special virtual address section for Xen to map some physical ram or device memory temporarily in initialization for MMU systems. FIXMAP_ADDR will return a virtual address by index for special purpose phys-to-virt mapping usage. For example, FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

To me, we are bending quite a bit the definition of the fixmap. There are not many use of the FIXMAP within the code and I think it would simply be better to abstract the use (or removing it when possible) and avoid defining FIXMAP_ADDR() & co for MPU.

Quoted 7 lines
Quoted 1 line

I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic

code.

FAOD, this is not what I had in mind. Instead, it was to provide helper which for !HAS_MPU would call fixmap and for HAS_MPU would do the work to map the region in the MPU.

Quoted 4 lines

Sorry, I am still confused about this comment, did you mean we can provider Some generic helpers like: early_map_console / eary_map_guest_memory. For non-MPU system, we still can call fixmap in these callers, but for MPU system, we have to map the region to MPU region?

Yes.

Cheers,

Signature · 1 line
Julien Grall
author
#35 lore raw

PATCH v6 8/11 xen/arm64: move MMU related code from head.S to head_mmu.S

In reply to Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros: adr_l and load_paddr will be used by MMU and MPU later, so we move them to macros.h.

---
 xen/arch/arm/arm64/Makefile             |   3 +
 xen/arch/arm/arm64/head.S               | 407 +-----------------------
 xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
 xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
 4 files changed, 432 insertions(+), 394 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
xen/arch/arm/arm64/Makefile +3 -0
index 6d507da0d4..22da2f54b5 100644
@@ -8,6 +8,9 @@ obj-y += domctl.o
 obj-y += domain.o
 obj-y += entry.o
 obj-y += head.o
+ifneq ($(CONFIG_HAS_MPU),y)
+obj-y += head_mmu.o
+endif
 obj-y += insn.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o
 obj-y += smc.o
xen/arch/arm/arm64/head.S +14 -393

448 lines not on this page · raw message

xen/arch/arm/arm64/head_mmu.S new file +364 -0

364 lines not on this page · raw message

xen/arch/arm/include/asm/arm64/macros.h +51 -1

61 lines not on this page · raw message

Signature · 1 line
2.25.1
#36 lore raw

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Hi Wei,

Quoted 4 lines · Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.

This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.

Quoted 8 lines

As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros: adr_l and load_paddr will be used by MMU and MPU later, so we move them to macros.h.

In general, the first signed-off should match the author. So who is who here?

Quoted 51 lines
---
  xen/arch/arm/arm64/Makefile             |   3 +
  xen/arch/arm/arm64/head.S               | 407 +-----------------------
  xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
  xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
  4 files changed, 432 insertions(+), 394 deletions(-)

create mode 100644 xen/arch/arm/arm64/head_mmu.S

xen/arch/arm/arm64/Makefile +2 -0
index 6d507da0d4..22da2f54b5 100644
@@ -8,6 +8,9 @@ obj-y += domctl.o
  obj-y += domain.o
  obj-y += entry.o
  obj-y += head.o
+ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o
+endif
  obj-y += insn.o
  obj-$(CONFIG_LIVEPATCH) += livepatch.o
  obj-y += smc.o
xen/arch/arm/arm64/head.S +0 -15
index ccedf20dc7..d9a8da9120 100644
@@ -25,17 +25,6 @@
  #include <efi/efierr.h>
  #include <asm/arm64/efibind.h>
 
-#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
-#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
-#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
-
-/* Convenience defines to get slot used by Xen mapping. */
-#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
-#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
-#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
-
  #define __HEAD_FLAG_PAGE_SIZE   ((PAGE_SHIFT - 10) / 2)
 
  #define __HEAD_FLAG_PHYS_BASE   1
@@ -82,73 +71,22 @@
   *  x30 - lr
   */
 
-#ifdef CONFIG_EARLY_PRINTK
-/*
- * Macro to print a string to the UART, if there is one.
- *

xen/arch/arm/arm64/head.S line 75

Quoted 10 lines
xen/arch/arm/arm64/head.S +2 -8
- * Clobbers x0 - x3
- */
-#define PRINT(_s)          \
-        mov   x3, lr ;     \
-        adr   x0, 98f ;    \
-        bl    puts    ;    \
-        mov   lr, x3 ;     \
-        RODATA_STR(98, _s)
+.section .text.header, "ax", %progbits
+/*.aarch64*/

The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.

Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).

xen/arch/arm/arm64/head.S line 79

Quoted 5 lines
xen/arch/arm/arm64/head.S +2 -1
  /*
- * Macro to print the value of register \xb
+ * Kernel startup entry point.
+ * ---------------------------

Same here about the indentation. I will not comment everywhere where the indentation was changed. So please look at it.

[...]

Quoted 32 lines
xen/arch/arm/arm64/head.S +0 -28
-/*
- * Map the UART in the fixmap (when earlyprintk is used) and hook the
- * fixmap table in the page tables.
- *
- * The fixmap cannot be mapped in create_page_tables because it may
- * clash with the 1:1 mapping.
- *
- * Inputs:
- *   x20: Physical offset
- *   x23: Early UART base physical address
- *
- * Clobbers x0 - x3
- */
-setup_fixmap:
-#ifdef CONFIG_EARLY_PRINTK
-        /* Add UART to the fixmap table */
-        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
-        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3, type=PT_DEV_L3
-#endif
-        /* Map fixmap into boot_second */
-        ldr   x0, =FIXMAP_ADDR(0)
-        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
-        /* Ensure any page table updates made above have occurred. */
-        dsb   nshst
-
-        ret
-ENDPROC(setup_fixmap)
-
  /*
   * Setup the initial stack and jump to the C world
   *

xen/arch/arm/arm64/head.S launch: · line 461

Quoted 5 lines
xen/arch/arm/arm64/head.S +1 -1
@@ -810,41 +458,14 @@ launch:
  ENDPROC(launch)
 
  /* Fail-stop */
-fail:   PRINT("- Boot failed -\r\n")
+ENTRY(fail)

This name is a bit too generic to be exposed. But it would be better to duplicate it.

Cheers,

Signature · 1 line
Julien Grall
#37 lore raw

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Quoted 8 lines · Julien Grall

Hi Wei,

Quoted 4 lines · Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

Quoted 5 lines

I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.

This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.

Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.

Ideally, checking the size should be done in a separate pre-patch so it is easier to review.

Cheers,

Signature · 1 line
Julien Grall
author
#38 lore raw

RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

In reply to Julien Grall

Hi Julien,

Quoted 21 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Hi Wei,

Quoted 4 lines · Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.

Quoted 9 lines

This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.

Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.

Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"

OK. We will create a patch to check the size, and place it in the head of the series.

Quoted 6 lines
Quoted 6 lines

As we need to access "fail" and "puts" functions out of assembly file, so we have to export them in this patch. And the assembly
macros: adr_l and load_paddr will be used by MMU and MPU later, so we move them to macros.h.

Quoted 6 lines
Quoted 2 lines

In general, the first signed-off should match the author. So who is who here?

I will adjust this order.

Quoted 61 lines
Quoted 26 lines
---
  xen/arch/arm/arm64/Makefile             |   3 +
  xen/arch/arm/arm64/head.S               | 407 +-----------------------
  xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
  xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
  4 files changed, 432 insertions(+), 394 deletions(-)

create mode 100644 xen/arch/arm/arm64/head_mmu.S

xen/arch/arm/arm64/Makefile +2 -0
index 6d507da0d4..22da2f54b5 100644
@@ -8,6 +8,9 @@ obj-y += domctl.o
  obj-y += domain.o
  obj-y += entry.o
  obj-y += head.o
+ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o
+endif
  obj-y += insn.o
  obj-$(CONFIG_LIVEPATCH) += livepatch.o
  obj-y += smc.o
xen/arch/arm/arm64/head.S +0 -0
index ccedf20dc7..d9a8da9120 100644

xen/arch/arm/arm64/head.S line 28

Quoted 4 lines
xen/arch/arm/arm64/head.S +0 -1
@@ -25,17 +25,6 @@
  #include <efi/efierr.h>
  #include <asm/arm64/efibind.h>
 
-#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1

*/

xen/arch/arm/arm64/head.S line 29

Quoted 1 line
xen/arch/arm/arm64/head.S +0 -1
-#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1

*/

xen/arch/arm/arm64/head.S line 30

Quoted 1 line
xen/arch/arm/arm64/head.S +0 -1
-#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1

*/

xen/arch/arm/arm64/head.S line 31

Quoted 1 line
xen/arch/arm/arm64/head.S +0 -1
-#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1

*/

xen/arch/arm/arm64/head.S line 32

Quoted 1 line
xen/arch/arm/arm64/head.S +0 -1
-#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1

*/

Quoted 22 lines
xen/arch/arm/arm64/head.S +0 -15
-
-/* Convenience defines to get slot used by Xen mapping. */
-#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
-#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
-#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
-
  #define __HEAD_FLAG_PAGE_SIZE   ((PAGE_SHIFT - 10) / 2)
 
  #define __HEAD_FLAG_PHYS_BASE   1
@@ -82,73 +71,22 @@
   *  x30 - lr
   */
 
-#ifdef CONFIG_EARLY_PRINTK
-/*
- * Macro to print a string to the UART, if there is one.
- *
- * Clobbers x0 - x3
- */
-#define PRINT(_s)          \
-        mov   x3, lr ;     \
-        adr   x0, 98f ;    \
Quoted 9 lines

xen/arch/arm/arm64/head.S line 75

Quoted 5 lines
xen/arch/arm/arm64/head.S +2 -3
-        bl    puts    ;    \
-        mov   lr, x3 ;     \
-        RODATA_STR(98, _s)
+.section .text.header, "ax", %progbits
+/*.aarch64*/

The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.

Ok.

Quoted 4 lines

Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).

Did you mean to create a separate patch for moving the functions in macro.h? Or in header section?

Quoted 9 lines

xen/arch/arm/arm64/head.S line 79

Quoted 5 lines
xen/arch/arm/arm64/head.S +2 -1
  /*
- * Macro to print the value of register \xb
+ * Kernel startup entry point.
+ * ---------------------------

Same here about the indentation. I will not comment everywhere where the indentation was changed. So please look at it.

OK. we will use a separate patch to clean them.

Quoted 37 lines

[...]

Quoted 8 lines
xen/arch/arm/arm64/head.S +0 -8
-/*
- * Map the UART in the fixmap (when earlyprintk is used) and hook the
- * fixmap table in the page tables.
- *
- * The fixmap cannot be mapped in create_page_tables because it may
- * clash with the 1:1 mapping.
- *
- * Inputs:

xen/arch/arm/arm64/head.S cpu_init:

Quoted 10 lines
xen/arch/arm/arm64/head.S +0 -10
- *   x20: Physical offset
- *   x23: Early UART base physical address
- *
- * Clobbers x0 - x3
- */
-setup_fixmap:
-#ifdef CONFIG_EARLY_PRINTK
-        /* Add UART to the fixmap table */
-        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
-        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3,

type=PT_DEV_L3

Quoted 16 lines
xen/arch/arm/arm64/head.S +0 -10
-#endif
-        /* Map fixmap into boot_second */
-        ldr   x0, =FIXMAP_ADDR(0)
-        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
-        /* Ensure any page table updates made above have occurred. */
-        dsb   nshst
-
-        ret
-ENDPROC(setup_fixmap)
-
  /*
   * Setup the initial stack and jump to the C world
   *
@@ -810,41 +458,14 @@ launch:
  ENDPROC(launch)
 
Quoted 6 lines

xen/arch/arm/arm64/head.S launch: · line 461

Quoted 3 lines
xen/arch/arm/arm64/head.S +1 -1
  /* Fail-stop */
-fail:   PRINT("- Boot failed -\r\n")
+ENTRY(fail)

This name is a bit too generic to be exposed. But it would be better to duplicate it.

Yes, duplicate it might be better. We will do it in next version.

Cheers,
Wei Chen

Quoted 5 lines

Cheers,

--
Julien Grall

#39 lore raw

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 4 lines
Quoted 3 lines

The patch is already quite difficult to read. So I would rather prefer if the indentation is changed separately.

Quoted 8 lines

Ok.

Quoted 4 lines

Furthermore, I think it would be best if the functions moved in the header are done separately to help checking (I would be able to diff the source with the destination more easily).

Did you mean to create a separate patch for moving the functions in macro.h?

I mean the macros in macro.h. Sorry, I misquoted it.

Cheers,

Signature · 1 line
Julien Grall
#40 lore raw

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

In reply to Wei Chen
Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 38 lines
Quoted 22 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Hi Wei,

Quoted 4 lines · Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.

Quoted 9 lines

This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity mapped.

Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.

Quoted 6 lines

Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"

OK. We will create a patch to check the size, and place it in the head of the series.

I thought a bit more about what I wrote. Xen binary should always start with the Image/Zimage header. At the moment, this is guaranteed because there is only one object using the section .text.header.

With the change introduced in this patch, there will be multiple objects using with the sections .text.header. This means we are relying on the compiler to always put the content of head.S first.

This is basically reverting to the behavior before commit 4267a33b19d ("xen/build: put image header into a separate section").

Therefore we do need a separate section to be used for head_*.S and maybe part of head.S. This new section could be called text.idmap so we know what the section is used for.

Cheers,

Signature · 1 line
Julien Grall
author
#41 lore raw

RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Hi Julien,

Quoted 65 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月14日 5:43
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 4:06
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 3 lines
Quoted 3 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>; Henry Wang <[email protected]>
Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from

head.S

Quoted 13 lines
Quoted 13 lines

to head_mmu.S

Hi Wei,

Quoted 4 lines · Wei Chen

There are lots of MMU specific code in head.S. This code will not be used in MPU systems. If we use #ifdef to gate them, the code will become messy and hard to maintain. So we move MMU related code to head_mmu.S, and keep common code still in head.S.

I am afraid that you can't simply move the MMU code out of head.S because this will break Xen when running using the identity map.

Quoted 8 lines
Quoted 8 lines

This is because we only map the first 4KB of Xen with PA == VA. At the moment, we guarantee it by having everything that needs to be used in the identity mapping before _end_boot and checking at link time if this fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a different way to guarantee it. One way to do it would be to create a section that would be used for everything that needs to be identity

mapped.

Quoted 8 lines
Quoted 1 line

Quote from next email
"
Looking at the code this morning, I noticed that we already have the section ".text.header". For now, that should do the job. So we just need to check the size of .text.header.

Quoted 6 lines

Ideally, checking the size should be done in a separate pre-patch so it is easier to review.
"

OK. We will create a patch to check the size, and place it in the head of the series.

Quoted 8 lines

I thought a bit more about what I wrote. Xen binary should always start with the Image/Zimage header. At the moment, this is guaranteed because there is only one object using the section .text.header.

With the change introduced in this patch, there will be multiple objects using with the sections .text.header. This means we are relying on the compiler to always put the content of head.S first.

Yes, now it depends on the objects' order in Makefile. This should be a risk.

Quoted 7 lines

This is basically reverting to the behavior before commit 4267a33b19d ("xen/build: put image header into a separate section").

Therefore we do need a separate section to be used for head_*.S and maybe part of head.S. This new section could be called text.idmap so we know what the section is used for.

Yes, we might place this new section after text.header.

Cheers,
Wei Chen

Quoted 4 lines

Cheers,

--
Julien Grall

author
#42 lore raw

PATCH v6 9/11 xen/arm64: create boot-time MPU protection regions

In reply to Wei Chen

From: Penny Zheng <[email protected]>

Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.

This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11. So we have to encode these Armv8-R MPU system registers in header file explicitly.

As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name prepare_early_mappings to replace create_page_tables for MMU and MPU.

---
 xen/arch/arm/arm64/Makefile              |  2 +
 xen/arch/arm/arm64/head.S                | 13 ++--
 xen/arch/arm/arm64/head_mmu.S            |  4 +-
 xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
 xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
 xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++
 6 files changed, 185 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
xen/arch/arm/arm64/Makefile +2 -0
index 22da2f54b5..438c9737ad 100644
@@ -10,6 +10,8 @@ obj-y += entry.o
 obj-y += head.o
 ifneq ($(CONFIG_HAS_MPU),y)
 obj-y += head_mmu.o
+else
+obj-y += head_mpu.o
 endif
 obj-y += insn.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o
xen/arch/arm/arm64/head.S +9 -4
index d9a8da9120..6c1a5f74a1 100644
@@ -79,12 +79,12 @@
  * ---------------------------
  *
  * The requirements are:
- *   MMU = off, D-cache = off, I-cache = on or off,
+ *   MMU/MPU = off, D-cache = off, I-cache = on or off,
  *   x0 = physical address to the FDT blob.
  *
  * This must be the very first address in the loaded image.
  * It should be linked at XEN_VIRT_START, and loaded at any
- * 4K-aligned address.  All of text+data+bss must fit in 2MB,
+ * 4K-aligned address. All of text+data+bss must fit in 2MB,
  * or the initial pagetable code below will need adjustment.
  */
 
@@ -249,7 +249,12 @@ real_start_efi:
 
         bl    check_cpu_mode
         bl    cpu_init
-        bl    create_page_tables
+
+        /*
+         * Create boot memory management data, pagetable for MMU systems
+         * and protection regions for MPU systems.
+         */
+        bl    prepare_early_mappings
         bl    enable_mmu
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
@@ -307,7 +312,7 @@ GLOBAL(init_secondary)
 #endif
         bl    check_cpu_mode
         bl    cpu_init
-        bl    create_page_tables
+        bl    prepare_early_mappings
         bl    enable_mmu
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
xen/arch/arm/arm64/head_mmu.S +2 -2
index 1a3df81a38..fc64819a98 100644
@@ -123,7 +123,7 @@
  *
  * Clobbers x0 - x4
  */
-ENTRY(create_page_tables)
+ENTRY(prepare_early_mappings)
         /* Prepare the page-tables for mapping Xen */
         ldr   x0, =XEN_VIRT_START
         create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
@@ -208,7 +208,7 @@ virtphys_clash:
         /* Identity map clashes with boot_third, which we cannot handle yet */
         PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
         b     fail
-ENDPROC(create_page_tables)
+ENDPROC(prepare_early_mappings)
 
 /*
  * Turn on the Data Cache and the MMU. The function will return on the 1:1
xen/arch/arm/arm64/head_mpu.S new file +70 -0

70 lines not on this page · raw message

xen/arch/arm/include/asm/arm64/mpu.h new file +13 -0
new file mode 100644
index 0000000000..d209eef6db
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * mpu.h: Arm Memory Protection Unit definitions.
+ */
+
+#ifndef __ARM64_MPU_H__
+#define __ARM64_MPU_H__
+
+#define MPU_REGION_SHIFT  6
+#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
+#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
+
+#endif /* __ARM64_MPU_H__ */
xen/arch/arm/include/asm/arm64/sysregs.h +89 -0

95 lines not on this page · raw message

Signature · 1 line
2.25.1
#43 lore raw

Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Hi Wei,

Quoted 11 lines · Wei Chen

From: Penny Zheng <[email protected]>

Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.

This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11. So we have to encode these Armv8-R MPU system registers in header file explicitly.

Quoted 7 lines

As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name prepare_early_mappings to replace create_page_tables for MMU and MPU.

If Penny is the original author, then her signed-off-by should be first.

Quoted 30 lines
---
  xen/arch/arm/arm64/Makefile              |  2 +
  xen/arch/arm/arm64/head.S                | 13 ++--
  xen/arch/arm/arm64/head_mmu.S            |  4 +-
  xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
  xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
  xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++
  6 files changed, 185 insertions(+), 6 deletions(-)

create mode 100644 xen/arch/arm/arm64/head_mpu.S create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h

xen/arch/arm/arm64/Makefile +2 -0
index 22da2f54b5..438c9737ad 100644
@@ -10,6 +10,8 @@ obj-y += entry.o
  obj-y += head.o
  ifneq ($(CONFIG_HAS_MPU),y)
  obj-y += head_mmu.o
+else
+obj-y += head_mpu.o
  endif
  obj-y += insn.o
  obj-$(CONFIG_LIVEPATCH) += livepatch.o
xen/arch/arm/arm64/head.S +0 -0
index d9a8da9120..6c1a5f74a1 100644
@@ -79,12 +79,12 @@
   * ---------------------------

xen/arch/arm/arm64/head.S line 87

Quoted 10 lines
xen/arch/arm/arm64/head.S +2 -2
   *
   * The requirements are:
- *   MMU = off, D-cache = off, I-cache = on or off,
+ *   MMU/MPU = off, D-cache = off, I-cache = on or off,
   *   x0 = physical address to the FDT blob.
   *
   * This must be the very first address in the loaded image.
   * It should be linked at XEN_VIRT_START, and loaded at any
- * 4K-aligned address.  All of text+data+bss must fit in 2MB,
+ * 4K-aligned address. All of text+data+bss must fit in 2MB,

The double space after the final point was valid. This is fairly common to use it and this is a spurious change.

xen/arch/arm/arm64/head.S real_start_efi: · line 256

Quoted 13 lines
xen/arch/arm/arm64/head.S +5 -1
   * or the initial pagetable code below will need adjustment.
   */
 
@@ -249,7 +249,12 @@ real_start_efi:
 
          bl    check_cpu_mode
          bl    cpu_init
-        bl    create_page_tables
+
+        /*
+         * Create boot memory management data, pagetable for MMU systems
+         * and protection regions for MPU systems.
+         */

head.S is now meant to be generic. So I would prefer if we keep comment as generic as possible. IOW, anything after the first comma should be dropped.

Quoted 41 lines
xen/arch/arm/arm64/head.S +2 -1
+        bl    prepare_early_mappings
          bl    enable_mmu
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
@@ -307,7 +312,7 @@ GLOBAL(init_secondary)
  #endif
          bl    check_cpu_mode
          bl    cpu_init
-        bl    create_page_tables
+        bl    prepare_early_mappings
          bl    enable_mmu
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
xen/arch/arm/arm64/head_mmu.S +2 -2
index 1a3df81a38..fc64819a98 100644
@@ -123,7 +123,7 @@
   *
   * Clobbers x0 - x4
   */
-ENTRY(create_page_tables)
+ENTRY(prepare_early_mappings)
          /* Prepare the page-tables for mapping Xen */
          ldr   x0, =XEN_VIRT_START
          create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
@@ -208,7 +208,7 @@ virtphys_clash:
          /* Identity map clashes with boot_third, which we cannot handle yet */
          PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
          b     fail
-ENDPROC(create_page_tables)
+ENDPROC(prepare_early_mappings)
 
  /*
   * Turn on the Data Cache and the MMU. The function will return on the 1:1
xen/arch/arm/arm64/head_mpu.S new file +0 -0
new file mode 100644
index 0000000000..f60611b556

xen/arch/arm/arm64/head_mpu.S line 1

Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1 -0
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-only

Coding style:

/* SPDX ... */

xen/arch/arm/arm64/head_mpu.S line 8

Quoted 7 lines
xen/arch/arm/arm64/head_mpu.S new file +7 -0
+/*
+ * Start-of-day code for an Armv8-R MPU system.
+ */
+
+#include <asm/arm64/mpu.h>
+#include <asm/page.h>
+#include <asm/early_printk.h>

Headers should be included in alphabetical order.

xen/arch/arm/arm64/head_mpu.S line 18

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10 -0
+
+/*
+ * From the requirements of head.S we know that Xen image should
+ * be linked at XEN_START_ADDRESS, and all of text + data + bss
+ * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the
+ * address that Xen image should be loaded at. So for initial MPU
+ * regions setup, we use 2MB for Xen data memory to setup boot
+ * region, or the create boot regions code below will need adjustment.
+ */
+#define XEN_START_MEM_SIZE      0x200000

It sounds like something that should be defined in the header. Also, I think the size should be common between MPU and MMU.

In [1], I was going to name it XEN_VIRT_SIZE. I would be OK to remove "VIRT" in the name.

xen/arch/arm/arm64/head_mpu.S line 23

Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5 -0
+
+/*
+ * In boot stage, we will use 1 MPU region:
+ * Region#0: Normal memory for Xen text + data + bss (2MB)
+ */

Are we only going to modify the MPU in head.S? If not, then I would define the layout in config_mpu.h so there are a single point where you can read how this works.

xen/arch/arm/arm64/head_mpu.S line 27

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+#define BOOT_NORMAL_REGION_IDX  0x0
+
+/* MPU normal memory attributes. */
+#define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */

IIUC, this means that Xen will be mapped write-executable. Is this going to be forever? If not, when can't we already mapped Xen properly?

xen/arch/arm/arm64/head_mpu.S line 28

Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1 -0
+#define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */

To me, it feels like this should be fined outside of head.S because this could be re-used by other part of Xen.

xen/arch/arm/arm64/head_mpu.S line 32

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+
+.macro write_pr, sel, prbar, prlar
+    msr   PRSELR_EL2, \sel
+    dsb   sy

Is it really necessary to use "sy" here? Also, it would be good to explain the logic. I.e. why do you need two dsb but only one isb?

In fact, I was expecting an "isb" here than "dsb" to wait for the completion of the instruction.

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+    msr   PRBAR_EL2, \prbar
+    msr   PRLAR_EL2, \prlar
+    dsb   sy
+    isb

xen/arch/arm/arm64/head_mpu.S line 46

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10 -0
+.endm
+
+.section .text.header, "ax", %progbits
+
+/*
+ * Static start-of-day EL2 MPU memory layout.
+ *
+ * It has a very simple structure, including:
+ *  - 2MB normal memory mappings of xen at XEN_START_ADDRESS, which
+ * is the address where Xen was loaded by the bootloader.

Missing details on the clobberred registers.

xen/arch/arm/arm64/head_mpu.S line 54

Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8 -0
+ */
+ENTRY(prepare_early_mappings)
+    /* Map Xen start memory to a normal memory region. */
+    mov x0, #BOOT_NORMAL_REGION_IDX
+    ldr x1, =XEN_START_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_NORMAL_MEM
+    orr x1, x1, x3

It looks like to me there are a potential for a macro to compute the register.

xen/arch/arm/arm64/head_mpu.S line 57

Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3 -0
+
+    ldr x2, =XEN_START_ADDRESS
+    mov x3, #(XEN_START_MEM_SIZE - 1)

XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.

xen/arch/arm/arm64/head_mpu.S line 65

Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8 -0
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_NORMAL_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar

This is not a very useful comment because this can be inferred from the prototype of write_pr. What would be more interesting is to explain the logic within this function in the same way we do in head.S and head_mmu.S.

xen/arch/arm/include/asm/arm64/sysregs.h

Quoted 28 lines
xen/arch/arm/arm64/head_mpu.S new file +5 -0
+     */
+    write_pr x0, x1, x2
+
+    ret
+ENDPROC(prepare_early_mappings)
xen/arch/arm/include/asm/arm64/mpu.h new file +13 -0
new file mode 100644
index 0000000000..d209eef6db
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * mpu.h: Arm Memory Protection Unit definitions.
+ */
+
+#ifndef __ARM64_MPU_H__
+#define __ARM64_MPU_H__
+
+#define MPU_REGION_SHIFT  6
+#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
+#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
+
+#endif /* __ARM64_MPU_H__ */
xen/arch/arm/include/asm/arm64/sysregs.h +0 -0
index 54670084c3..a596042d6c 100644

In the context of this patch, it would be better to only define the registers you need. If you want to define all of them, then please define them in a separate patch before this one.

xen/arch/arm/include/asm/arm64/sysregs.h line 462

Quoted 6 lines
xen/arch/arm/include/asm/arm64/sysregs.h +2 -0
@@ -458,6 +458,95 @@
  #define ZCR_ELx_LEN_SIZE             9
  #define ZCR_ELx_LEN_MASK             0x1ff
 
+/* System registers for AArch64 with PMSA */
+#ifdef CONFIG_HAS_MPU

The #ifdef here seems unnecessary.

Quoted 80 lines
xen/arch/arm/include/asm/arm64/sysregs.h +80 -0
+
+/* EL1 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL1   S3_0_C6_C8_0
+#define PRBAR1_EL1  S3_0_C6_C8_4
+#define PRBAR2_EL1  S3_0_C6_C9_0
+#define PRBAR3_EL1  S3_0_C6_C9_4
+#define PRBAR4_EL1  S3_0_C6_C10_0
+#define PRBAR5_EL1  S3_0_C6_C10_4
+#define PRBAR6_EL1  S3_0_C6_C11_0
+#define PRBAR7_EL1  S3_0_C6_C11_4
+#define PRBAR8_EL1  S3_0_C6_C12_0
+#define PRBAR9_EL1  S3_0_C6_C12_4
+#define PRBAR10_EL1 S3_0_C6_C13_0
+#define PRBAR11_EL1 S3_0_C6_C13_4
+#define PRBAR12_EL1 S3_0_C6_C14_0
+#define PRBAR13_EL1 S3_0_C6_C14_4
+#define PRBAR14_EL1 S3_0_C6_C15_0
+#define PRBAR15_EL1 S3_0_C6_C15_4
+
+/* EL1 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL1   S3_0_C6_C8_1
+#define PRLAR1_EL1  S3_0_C6_C8_5
+#define PRLAR2_EL1  S3_0_C6_C9_1
+#define PRLAR3_EL1  S3_0_C6_C9_5
+#define PRLAR4_EL1  S3_0_C6_C10_1
+#define PRLAR5_EL1  S3_0_C6_C10_5
+#define PRLAR6_EL1  S3_0_C6_C11_1
+#define PRLAR7_EL1  S3_0_C6_C11_5
+#define PRLAR8_EL1  S3_0_C6_C12_1
+#define PRLAR9_EL1  S3_0_C6_C12_5
+#define PRLAR10_EL1 S3_0_C6_C13_1
+#define PRLAR11_EL1 S3_0_C6_C13_5
+#define PRLAR12_EL1 S3_0_C6_C14_1
+#define PRLAR13_EL1 S3_0_C6_C14_5
+#define PRLAR14_EL1 S3_0_C6_C15_1
+#define PRLAR15_EL1 S3_0_C6_C15_5
+
+/* EL2 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL2   S3_4_C6_C8_0
+#define PRBAR1_EL2  S3_4_C6_C8_4
+#define PRBAR2_EL2  S3_4_C6_C9_0
+#define PRBAR3_EL2  S3_4_C6_C9_4
+#define PRBAR4_EL2  S3_4_C6_C10_0
+#define PRBAR5_EL2  S3_4_C6_C10_4
+#define PRBAR6_EL2  S3_4_C6_C11_0
+#define PRBAR7_EL2  S3_4_C6_C11_4
+#define PRBAR8_EL2  S3_4_C6_C12_0
+#define PRBAR9_EL2  S3_4_C6_C12_4
+#define PRBAR10_EL2 S3_4_C6_C13_0
+#define PRBAR11_EL2 S3_4_C6_C13_4
+#define PRBAR12_EL2 S3_4_C6_C14_0
+#define PRBAR13_EL2 S3_4_C6_C14_4
+#define PRBAR14_EL2 S3_4_C6_C15_0
+#define PRBAR15_EL2 S3_4_C6_C15_4
+
+/* EL2 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL2   S3_4_C6_C8_1
+#define PRLAR1_EL2  S3_4_C6_C8_5
+#define PRLAR2_EL2  S3_4_C6_C9_1
+#define PRLAR3_EL2  S3_4_C6_C9_5
+#define PRLAR4_EL2  S3_4_C6_C10_1
+#define PRLAR5_EL2  S3_4_C6_C10_5
+#define PRLAR6_EL2  S3_4_C6_C11_1
+#define PRLAR7_EL2  S3_4_C6_C11_5
+#define PRLAR8_EL2  S3_4_C6_C12_1
+#define PRLAR9_EL2  S3_4_C6_C12_5
+#define PRLAR10_EL2 S3_4_C6_C13_1
+#define PRLAR11_EL2 S3_4_C6_C13_5
+#define PRLAR12_EL2 S3_4_C6_C14_1
+#define PRLAR13_EL2 S3_4_C6_C14_5
+#define PRLAR14_EL2 S3_4_C6_C15_1
+#define PRLAR15_EL2 S3_4_C6_C15_5
+
+/* MPU Protection Region Enable Register encode */
+#define PRENR_EL1 S3_0_C6_C1_1
+#define PRENR_EL2 S3_4_C6_C1_1
+
+/* MPU Protection Region Selection Register encode */
+#define PRSELR_EL1 S3_0_C6_C2_1
+#define PRSELR_EL2 S3_4_C6_C2_1

xen/arch/arm/include/asm/arm64/sysregs.h line 552

Quoted 10 lines
xen/arch/arm/include/asm/arm64/sysregs.h +7 -0
+
+/* MPU Type registers encode */
+#define MPUIR_EL1 S3_0_C0_C0_4
+#define MPUIR_EL2 S3_4_C0_C0_4
+
+#endif
+
  /* Access to system registers */
 
  #define WRITE_SYSREG64(v, name) do {                    \

Cheers,

[1] https://lore.kernel.org/all/[email protected]/

Signature · 1 line
Julien Grall
#44 lore raw

RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Hi Julien

Quoted 179 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 4:47 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Hi Wei,

Quoted 11 lines · Wei Chen

From: Penny Zheng <[email protected]>

Like boot-time page table in MMU system, we need a boot-time MPU protection region configuration in MPU system so Xen can fetch code and data from normal memory.

This operation need to access Armv8-R MPU system registers, but these system registers are not supported in GCC version < 11.
So we have to encode these Armv8-R MPU system registers in header file explicitly.

Quoted 7 lines

As MMU system and MPU system have different functions to create the boot MMU/MPU data, this will introduce extra #ifdef in code flow, so we introduce a neutral name prepare_early_mappings to replace create_page_tables for MMU and MPU.

If Penny is the original author, then her signed-off-by should be first.

Quoted 7 lines
---
  xen/arch/arm/arm64/Makefile              |  2 +
  xen/arch/arm/arm64/head.S                | 13 ++--
  xen/arch/arm/arm64/head_mmu.S            |  4 +-
  xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
  xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
  xen/arch/arm/include/asm/arm64/sysregs.h | 89

++++++++++++++++++++++++

Review

Quoted 5 lines
  6 files changed, 185 insertions(+), 6 deletions(-)

create mode 100644 xen/arch/arm/arm64/head_mpu.S create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h

(quoted diff) +0 -0

b/xen/arch/arm/arm64/Makefile

Quoted 18 lines

index 22da2f54b5..438c9737ad 100644

xen/arch/arm/arm64/Makefile +2 -0
@@ -10,6 +10,8 @@ obj-y += entry.o
  obj-y += head.o
  ifneq ($(CONFIG_HAS_MPU),y)
  obj-y += head_mmu.o
+else
+obj-y += head_mpu.o
  endif
  obj-y += insn.o
  obj-$(CONFIG_LIVEPATCH) += livepatch.o diff --git

a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index d9a8da9120..6c1a5f74a1 100644

xen/arch/arm/arm64/head.S +0 -0
@@ -79,12 +79,12 @@
   * ---------------------------

xen/arch/arm/arm64/head.S line 87

Quoted 10 lines
xen/arch/arm/arm64/head.S +2 -2
   *
   * The requirements are:
- *   MMU = off, D-cache = off, I-cache = on or off,
+ *   MMU/MPU = off, D-cache = off, I-cache = on or off,
   *   x0 = physical address to the FDT blob.
   *
   * This must be the very first address in the loaded image.
   * It should be linked at XEN_VIRT_START, and loaded at any
- * 4K-aligned address.  All of text+data+bss must fit in 2MB,
+ * 4K-aligned address. All of text+data+bss must fit in 2MB,

The double space after the final point was valid. This is fairly common to use it and this is a spurious change.

xen/arch/arm/arm64/head.S real_start_efi: · line 254

Quoted 11 lines
xen/arch/arm/arm64/head.S +3 -1
   * or the initial pagetable code below will need adjustment.
   */
 
@@ -249,7 +249,12 @@ real_start_efi:
 
          bl    check_cpu_mode
          bl    cpu_init
-        bl    create_page_tables
+
+        /*
+         * Create boot memory management data, pagetable for MMU

systems

xen/arch/arm/arm64/head.S real_start_efi: · line 256

Quoted 2 lines
xen/arch/arm/arm64/head.S +2 -0
+         * and protection regions for MPU systems.
+         */

head.S is now meant to be generic. So I would prefer if we keep comment as generic as possible. IOW, anything after the first comma should be dropped.

xen/arch/arm/arm64/head.S real_start_efi:

Quoted 4 lines
xen/arch/arm/arm64/head.S +1 -0
+        bl    prepare_early_mappings
          bl    enable_mmu
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual

Address. */

xen/arch/arm/arm64/head.S GLOBAL(init_secondary) · line 318

Quoted 9 lines
xen/arch/arm/arm64/head.S +1 -1
@@ -307,7 +312,7 @@ GLOBAL(init_secondary)
  #endif
          bl    check_cpu_mode
          bl    cpu_init
-        bl    create_page_tables
+        bl    prepare_early_mappings
          bl    enable_mmu
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual

Address. */

Review

Quoted 1 line
(quoted diff) +0 -0

b/xen/arch/arm/arm64/head_mmu.S

Quoted 13 lines

index 1a3df81a38..fc64819a98 100644

xen/arch/arm/arm64/head_mmu.S +1 -1
@@ -123,7 +123,7 @@
   *
   * Clobbers x0 - x4
   */
-ENTRY(create_page_tables)
+ENTRY(prepare_early_mappings)
          /* Prepare the page-tables for mapping Xen */
          ldr   x0, =XEN_VIRT_START
          create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3

xen/arch/arm/arm64/head_mmu.S virtphys_clash: · line 208

Quoted 1 line
xen/arch/arm/arm64/head_mmu.S +0 -0
@@ -208,7 +208,7 @@ virtphys_clash:
          /* Identity map clashes with boot_third, which we cannot handle yet

*/

xen/arch/arm/arm64/head_mmu.S virtphys_clash: · line 209

Quoted 1 line
xen/arch/arm/arm64/head_mmu.S +0 -0
          PRINT("- Unable to build boot page tables - virt and phys addresses

clash. -\r\n")

xen/arch/arm/arm64/head_mmu.S virtphys_clash:

Quoted 6 lines
xen/arch/arm/arm64/head_mmu.S +1 -1
          b     fail
-ENDPROC(create_page_tables)
+ENDPROC(prepare_early_mappings)
 
  /*
   * Turn on the Data Cache and the MMU. The function will return on the

1:1

Review

Quoted 1 line
(quoted diff) +0 -0

b/xen/arch/arm/arm64/head_mpu.S

xen/arch/arm/arm64/head_mpu.S line 1

Quoted 6 lines

new file mode 100644
index 0000000000..f60611b556

xen/arch/arm/arm64/head_mpu.S new file +1 -0
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-only

Coding style:

/* SPDX ... */

xen/arch/arm/arm64/head_mpu.S line 8

Quoted 7 lines
xen/arch/arm/arm64/head_mpu.S new file +7 -0
+/*
+ * Start-of-day code for an Armv8-R MPU system.
+ */
+
+#include <asm/arm64/mpu.h>
+#include <asm/page.h>
+#include <asm/early_printk.h>

Headers should be included in alphabetical order.

xen/arch/arm/arm64/head_mpu.S line 18

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S new file +10 -0
+
+/*
+ * From the requirements of head.S we know that Xen image should
+ * be linked at XEN_START_ADDRESS, and all of text + data + bss
+ * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the
+ * address that Xen image should be loaded at. So for initial MPU
+ * regions setup, we use 2MB for Xen data memory to setup boot
+ * region, or the create boot regions code below will need adjustment.
+ */
+#define XEN_START_MEM_SIZE      0x200000

It sounds like something that should be defined in the header. Also, I think the size should be common between MPU and MMU.

Quoted 3 lines

In [1], I was going to name it XEN_VIRT_SIZE. I would be OK to remove "VIRT" in the name.

Thx and please, then I will replace it with XEN_SIZE

Quoted 10 lines

xen/arch/arm/arm64/head_mpu.S line 23

Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5 -0
+
+/*
+ * In boot stage, we will use 1 MPU region:
+ * Region#0: Normal memory for Xen text + data + bss (2MB)
+ */

Are we only going to modify the MPU in head.S? If not, then I would define the layout in config_mpu.h so there are a single point where you can read how this works.

We will remap Xen in C codes in setup_mm().
The whole strategy is aligned with MMU: a very simple setup(map xen with the maximum size, 2M) at start-of-the-day, and a fit map in setup_mm.

All the following variables will be only used at head_mpu.S. It is not very generic, later, when introducing MPU memory region management in C codes, we will define different macros for various memory attributes.

Quoted 8 lines

xen/arch/arm/arm64/head_mpu.S line 27

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+#define BOOT_NORMAL_REGION_IDX  0x0
+
+/* MPU normal memory attributes. */
+#define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */

IIUC, this means that Xen will be mapped write-executable. Is this going to be forever? If not, when can't we already mapped Xen properly?

No, this setup is the start-of-day setup, and will only last for a very short time.

To be aligned with MMU system, in which L3 memory attributes is as follows:
#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */ Xen shall be mapped read-only-executable here, and I will fix it.

Quoted 5 lines

xen/arch/arm/arm64/head_mpu.S line 28

Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1 -0
+#define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */

To me, it feels like this should be fined outside of head.S because this could be re-used by other part of Xen.

It is like PT_MEM_L3 in head_mmu.S. It will be only used in compiler codes.

MMU is defining macro like PAGE_HYPERVISOR_RW, for memory attributes management in C codes, and we intend to follow the same strategy.

Quoted 41 lines

xen/arch/arm/arm64/head_mpu.S line 32

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+
+.macro write_pr, sel, prbar, prlar
+    msr   PRSELR_EL2, \sel
+    dsb   sy

Is it really necessary to use "sy" here? Also, it would be good to explain the logic. I.e. why do you need two dsb but only one isb?

In fact, I was expecting an "isb" here than "dsb" to wait for the completion of the instruction.

xen/arch/arm/arm64/head_mpu.S

Quoted 13 lines
xen/arch/arm/arm64/head_mpu.S new file +13 -0
+    msr   PRBAR_EL2, \prbar
+    msr   PRLAR_EL2, \prlar
+    dsb   sy
+    isb
+.endm
+
+.section .text.header, "ax", %progbits
+
+/*
+ * Static start-of-day EL2 MPU memory layout.
+ *
+ * It has a very simple structure, including:
+ *  - 2MB normal memory mappings of xen at XEN_START_ADDRESS,

which

xen/arch/arm/arm64/head_mpu.S line 46

Quoted 1 line
xen/arch/arm/arm64/head_mpu.S new file +1 -0
+ * is the address where Xen was loaded by the bootloader.

Missing details on the clobberred registers.

xen/arch/arm/arm64/head_mpu.S line 54

Quoted 8 lines
xen/arch/arm/arm64/head_mpu.S new file +8 -0
+ */
+ENTRY(prepare_early_mappings)
+    /* Map Xen start memory to a normal memory region. */
+    mov x0, #BOOT_NORMAL_REGION_IDX
+    ldr x1, =XEN_START_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_NORMAL_MEM
+    orr x1, x1, x3

It looks like to me there are a potential for a macro to compute the register.

Quoted 9 lines

xen/arch/arm/arm64/head_mpu.S line 57

Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3 -0
+
+    ldr x2, =XEN_START_ADDRESS
+    mov x3, #(XEN_START_MEM_SIZE - 1)

XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller
and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.

Later, in setup_mm we will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.

IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.

Quoted 147 lines

xen/arch/arm/arm64/head_mpu.S line 65

Quoted 9 lines
xen/arch/arm/arm64/head_mpu.S new file +8 -0
+    add x2, x2, x
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_NORMAL_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar

This is not a very useful comment because this can be inferred from the prototype of write_pr. What would be more interesting is to explain the logic within this function in the same way we do in head.S and head_mmu.S.

Review

Quoted 6 lines
xen/arch/arm/arm64/head_mpu.S new file +5 -0
+     */
+    write_pr x0, x1, x2
+
+    ret
+ENDPROC(prepare_early_mappings)
(quoted diff) +0 -0

b/xen/arch/arm/include/asm/arm64/mpu.h

Review

Quoted 19 lines

new file mode 100644
index 0000000000..d209eef6db

xen/arch/arm/include/asm/arm64/mpu.h new file +13 -0
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * mpu.h: Arm Memory Protection Unit definitions.
+ */
+
+#ifndef __ARM64_MPU_H__
+#define __ARM64_MPU_H__
+
+#define MPU_REGION_SHIFT  6
+#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
+#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
+
+#endif /* __ARM64_MPU_H__ */
(quoted diff) +0 -0

b/xen/arch/arm/include/asm/arm64/sysregs.h

xen/arch/arm/include/asm/arm64/sysregs.h

Quoted 3 lines

index 54670084c3..a596042d6c 100644

xen/arch/arm/include/asm/arm64/sysregs.h +0 -0

In the context of this patch, it would be better to only define the registers you need. If you want to define all of them, then please define them in a separate patch before this one.

xen/arch/arm/include/asm/arm64/sysregs.h line 462

Quoted 6 lines
xen/arch/arm/include/asm/arm64/sysregs.h +2 -0
@@ -458,6 +458,95 @@
  #define ZCR_ELx_LEN_SIZE             9
  #define ZCR_ELx_LEN_MASK             0x1ff
 
+/* System registers for AArch64 with PMSA */
+#ifdef CONFIG_HAS_MPU

The #ifdef here seems unnecessary.

Quoted 89 lines
xen/arch/arm/include/asm/arm64/sysregs.h +87 -0
+
+/* EL1 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL1   S3_0_C6_C8_0
+#define PRBAR1_EL1  S3_0_C6_C8_4
+#define PRBAR2_EL1  S3_0_C6_C9_0
+#define PRBAR3_EL1  S3_0_C6_C9_4
+#define PRBAR4_EL1  S3_0_C6_C10_0
+#define PRBAR5_EL1  S3_0_C6_C10_4
+#define PRBAR6_EL1  S3_0_C6_C11_0
+#define PRBAR7_EL1  S3_0_C6_C11_4
+#define PRBAR8_EL1  S3_0_C6_C12_0
+#define PRBAR9_EL1  S3_0_C6_C12_4
+#define PRBAR10_EL1 S3_0_C6_C13_0
+#define PRBAR11_EL1 S3_0_C6_C13_4
+#define PRBAR12_EL1 S3_0_C6_C14_0
+#define PRBAR13_EL1 S3_0_C6_C14_4
+#define PRBAR14_EL1 S3_0_C6_C15_0
+#define PRBAR15_EL1 S3_0_C6_C15_4
+
+/* EL1 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL1   S3_0_C6_C8_1
+#define PRLAR1_EL1  S3_0_C6_C8_5
+#define PRLAR2_EL1  S3_0_C6_C9_1
+#define PRLAR3_EL1  S3_0_C6_C9_5
+#define PRLAR4_EL1  S3_0_C6_C10_1
+#define PRLAR5_EL1  S3_0_C6_C10_5
+#define PRLAR6_EL1  S3_0_C6_C11_1
+#define PRLAR7_EL1  S3_0_C6_C11_5
+#define PRLAR8_EL1  S3_0_C6_C12_1
+#define PRLAR9_EL1  S3_0_C6_C12_5
+#define PRLAR10_EL1 S3_0_C6_C13_1
+#define PRLAR11_EL1 S3_0_C6_C13_5
+#define PRLAR12_EL1 S3_0_C6_C14_1
+#define PRLAR13_EL1 S3_0_C6_C14_5
+#define PRLAR14_EL1 S3_0_C6_C15_1
+#define PRLAR15_EL1 S3_0_C6_C15_5
+
+/* EL2 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL2   S3_4_C6_C8_0
+#define PRBAR1_EL2  S3_4_C6_C8_4
+#define PRBAR2_EL2  S3_4_C6_C9_0
+#define PRBAR3_EL2  S3_4_C6_C9_4
+#define PRBAR4_EL2  S3_4_C6_C10_0
+#define PRBAR5_EL2  S3_4_C6_C10_4
+#define PRBAR6_EL2  S3_4_C6_C11_0
+#define PRBAR7_EL2  S3_4_C6_C11_4
+#define PRBAR8_EL2  S3_4_C6_C12_0
+#define PRBAR9_EL2  S3_4_C6_C12_4
+#define PRBAR10_EL2 S3_4_C6_C13_0
+#define PRBAR11_EL2 S3_4_C6_C13_4
+#define PRBAR12_EL2 S3_4_C6_C14_0
+#define PRBAR13_EL2 S3_4_C6_C14_4
+#define PRBAR14_EL2 S3_4_C6_C15_0
+#define PRBAR15_EL2 S3_4_C6_C15_4
+
+/* EL2 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL2   S3_4_C6_C8_1
+#define PRLAR1_EL2  S3_4_C6_C8_5
+#define PRLAR2_EL2  S3_4_C6_C9_1
+#define PRLAR3_EL2  S3_4_C6_C9_5
+#define PRLAR4_EL2  S3_4_C6_C10_1
+#define PRLAR5_EL2  S3_4_C6_C10_5
+#define PRLAR6_EL2  S3_4_C6_C11_1
+#define PRLAR7_EL2  S3_4_C6_C11_5
+#define PRLAR8_EL2  S3_4_C6_C12_1
+#define PRLAR9_EL2  S3_4_C6_C12_5
+#define PRLAR10_EL2 S3_4_C6_C13_1
+#define PRLAR11_EL2 S3_4_C6_C13_5
+#define PRLAR12_EL2 S3_4_C6_C14_1
+#define PRLAR13_EL2 S3_4_C6_C14_5
+#define PRLAR14_EL2 S3_4_C6_C15_1
+#define PRLAR15_EL2 S3_4_C6_C15_5
+
+/* MPU Protection Region Enable Register encode */
+#define PRENR_EL1 S3_0_C6_C1_1
+#define PRENR_EL2 S3_4_C6_C1_1
+
+/* MPU Protection Region Selection Register encode */
+#define PRSELR_EL1 S3_0_C6_C2_1
+#define PRSELR_EL2 S3_4_C6_C2_1
+
+/* MPU Type registers encode */
+#define MPUIR_EL1 S3_0_C0_C0_4
+#define MPUIR_EL2 S3_4_C0_C0_4
+
+#endif
+
  /* Access to system registers */
 
Quoted 8 lines

xen/arch/arm/include/asm/arm64/sysregs.h line 552

Quoted 1 line
xen/arch/arm/include/asm/arm64/sysregs.h +0 -0
  #define WRITE_SYSREG64(v, name) do {                    \

Cheers,

[1] https://lore.kernel.org/all/[email protected]/

--
Julien Grall

Cheers,

--
Penny Zheng

#45 lore raw

Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Quoted 1 line · Penny Zheng

Hi Julien

Hi Penny,

Quoted 11 lines
Quoted 10 lines

xen/arch/arm/arm64/head_mpu.S line 23

Quoted 5 lines
xen/arch/arm/arm64/head_mpu.S new file +5 -0
+
+/*
+ * In boot stage, we will use 1 MPU region:
+ * Region#0: Normal memory for Xen text + data + bss (2MB)
+ */

Are we only going to modify the MPU in head.S? If not, then I would define the layout in config_mpu.h so there are a single point where you can read how this works.

Quoted 4 lines

We will remap Xen in C codes in setup_mm().
The whole strategy is aligned with MMU: a very simple setup(map xen with the maximum size, 2M) at start-of-the-day, and a fit map in setup_mm.

The strategy we are using for the MMU is completely broken (not compliant with the Arm Arm) and unnecessary. My long term goal is to actually remove the switch_ttbr() and most of setup_pagetables() for all setup but cache coloring. This means the concept of boot pages will not exist anymore.

For the MPU, we should aim to do better than what was done for the MMU. Right now, I see no argument for switching MPUs table. I am only seen argument against it because you need to disable the cache and is quite fragile.

[...]

Quoted 10 lines
Quoted 9 lines

xen/arch/arm/arm64/head_mpu.S line 57

Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3 -0
+
+    ldr x2, =XEN_START_ADDRESS
+    mov x3, #(XEN_START_MEM_SIZE - 1)

XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller
and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.

Quoted 4 lines

Later, in setup_mm we will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.

But what prevents you to do this now?

Quoted 3 lines

IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.

Which is broken because we don't know what located after Xen binary. This could be reserved RAM, device which may requires non-caching attribute. Mapping those regions with caching attributes is going to break.

As I hinted above, there are a very long list of issues in the MMU boot code. So don't take that code for granted. Your MPU code should first be compliant with the Arm Arm. If it is close to the MMU code then that's a bonus. But bear in mind that the code may look very different soon (see [1]).

Cheers,

[1] [email protected]

Signature · 1 line
Julien Grall
#46 lore raw

RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Hi Julien

Quoted 61 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:30 PM
To: Penny Zheng <[email protected]>; Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

Quoted 1 line · Penny Zheng

Hi Julien

Hi Penny,

Quoted 10 lines
Quoted 9 lines

xen/arch/arm/arm64/head_mpu.S

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S new file +4 -0
+
+/*
+ * In boot stage, we will use 1 MPU region:
+ * Region#0: Normal memory for Xen text + data + bss (2MB)  */

Are we only going to modify the MPU in head.S? If not, then I would define the layout in config_mpu.h so there are a single point where you can read how this works.

Quoted 4 lines

We will remap Xen in C codes in setup_mm().
The whole strategy is aligned with MMU: a very simple setup(map xen with the maximum size, 2M) at start-of-the-day, and a fit map in setup_mm.

The strategy we are using for the MMU is completely broken (not compliant with the Arm Arm) and unnecessary. My long term goal is to actually remove the switch_ttbr() and most of setup_pagetables() for all setup but cache coloring. This means the concept of boot pages will not exist anymore.

For the MPU, we should aim to do better than what was done for the MMU. Right now, I see no argument for switching MPUs table. I am only seen argument against it because you need to disable the cache and is quite fragile.

[...]

Quoted 10 lines
Quoted 9 lines

xen/arch/arm/arm64/head_mpu.S line 57

Quoted 3 lines
xen/arch/arm/arm64/head_mpu.S new file +3 -0
+
+    ldr x2, =XEN_START_ADDRESS
+    mov x3, #(XEN_START_MEM_SIZE - 1)

XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller and you will map memory that may not be part of Xen. Therefore, you likely want to compute the real size to avoid mapping too much.

Quoted 4 lines

Later, in setup_mm we will map XEN components by components, such as, one MPU memory region for read-only-executable text section, one MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.

But what prevents you to do this now?

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

IMHO, the mapping in compiler with maximum size of Xen is also what MMU does.

Which is broken because we don't know what located after Xen binary. This could be reserved RAM, device which may requires non-caching attribute. Mapping those regions with caching attributes is going to break.

Understood!
Then I will map Xen components by components in the compile time, then later the first MPU/Cache disabled will be eliminated, I guess.

Quoted 7 lines

Cheers,

[1] [email protected]

--
Julien Grall

author
#47 lore raw

PATCH v6 10/11 xen/arm64: introduce helpers for MPU enable/disable

In reply to Wei Chen

From: Penny Zheng <[email protected]>

We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered. So in this situation, we need to disable the whole MPU to update the protection regions.

In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent with enable_mmu. For MMU systems, enable_mmu will turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.

In this patch, we also introduce a neutral name enable_mm for Xen to enable MMU/MPU. This can help us to keep one code flow in head.S

---
 xen/arch/arm/arm64/head.S     |  5 +++--
 xen/arch/arm/arm64/head_mmu.S |  4 ++--
 xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +3 -2
index 6c1a5f74a1..228f01db69 100644
@@ -255,7 +255,8 @@ real_start_efi:
          * and protection regions for MPU systems.
          */
         bl    prepare_early_mappings
-        bl    enable_mmu
+        /* Turn on MMU or MPU */
+        bl    enable_mm
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
         ldr   x0, =primary_switched
@@ -313,7 +314,7 @@ GLOBAL(init_secondary)
         bl    check_cpu_mode
         bl    cpu_init
         bl    prepare_early_mappings
-        bl    enable_mmu
+        bl    enable_mm
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
         ldr   x0, =secondary_switched
xen/arch/arm/arm64/head_mmu.S +2 -2
index fc64819a98..b542755bd2 100644
@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
  *
  * Clobbers x0 - x3
  */
-ENTRY(enable_mmu)
+ENTRY(enable_mm)
         PRINT("- Turning on paging -\r\n")
 
         /*
@@ -239,7 +239,7 @@ ENTRY(enable_mmu)
         msr   SCTLR_EL2, x0          /* now paging is enabled */
         isb                          /* Now, flush the icache */
         ret
-ENDPROC(enable_mmu)
+ENDPROC(enable_mm)
 
 /*
  * Remove the 1:1 map from the page-tables. It is not easy to keep track
xen/arch/arm/arm64/head_mpu.S +35 -0
index f60611b556..5a1b03e293 100644
@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
 
     ret
 ENDPROC(prepare_early_mappings)
+
+/*
+ * Enable EL2 MPU and data cache. Because we will disable cache
+ * with MPU at the same time, in accordance with that, we have
+ * to enable cache with MPU at the same time in this function.
+ * When MPU is disabled, the MPU background attributes will
+ * be used. On some platform, the background will treat all
+ * memory as IO memory. The access to IO memory will bypass
+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */
+ENTRY(enable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
+    /* Enable EL2 MPU and D-cache */
+    orr   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(enable_mm)
+
+/* Disable MPU system, including data cache. */
+ENTRY(disable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #~(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
+    /* Disable EL2 MPU and D-cache */
+    and   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(disable_mm)
Signature · 1 line
2.25.1
#48 lore raw

Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Hi Wei,

Quoted 12 lines · Wei Chen

From: Penny Zheng <[email protected]>

We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.

This raises the following question: Why can't we create the split regions right now?

In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).

Quoted 79 lines

So in this situation, we need to disable the whole MPU to update the protection regions.

In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent with enable_mmu. For MMU systems, enable_mmu will turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.

In this patch, we also introduce a neutral name enable_mm for Xen to enable MMU/MPU. This can help us to keep one code flow in head.S

---
  xen/arch/arm/arm64/head.S     |  5 +++--
  xen/arch/arm/arm64/head_mmu.S |  4 ++--
  xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++
  3 files changed, 40 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +3 -2
index 6c1a5f74a1..228f01db69 100644
@@ -255,7 +255,8 @@ real_start_efi:
           * and protection regions for MPU systems.
           */
          bl    prepare_early_mappings
-        bl    enable_mmu
+        /* Turn on MMU or MPU */
+        bl    enable_mm
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
          ldr   x0, =primary_switched
@@ -313,7 +314,7 @@ GLOBAL(init_secondary)
          bl    check_cpu_mode
          bl    cpu_init
          bl    prepare_early_mappings
-        bl    enable_mmu
+        bl    enable_mm
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
          ldr   x0, =secondary_switched
xen/arch/arm/arm64/head_mmu.S +2 -2
index fc64819a98..b542755bd2 100644
@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
   *
   * Clobbers x0 - x3
   */
-ENTRY(enable_mmu)
+ENTRY(enable_mm)
          PRINT("- Turning on paging -\r\n")
 
          /*
@@ -239,7 +239,7 @@ ENTRY(enable_mmu)
          msr   SCTLR_EL2, x0          /* now paging is enabled */
          isb                          /* Now, flush the icache */
          ret
-ENDPROC(enable_mmu)
+ENDPROC(enable_mm)
 
  /*
   * Remove the 1:1 map from the page-tables. It is not easy to keep track
xen/arch/arm/arm64/head_mpu.S +0 -0
index f60611b556..5a1b03e293 100644
@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
 

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 78

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8 -0
      ret
  ENDPROC(prepare_early_mappings)
+
+/*
+ * Enable EL2 MPU and data cache. Because we will disable cache
+ * with MPU at the same time, in accordance with that, we have
+ * to enable cache with MPU at the same time in this function.
+ * When MPU is disabled, the MPU background attributes will
+ * be used. On some platform, the background will treat all
+ * memory as IO memory.

I was under the impression that all access would be treated as Device Memory when the MMU is off. Isn't it the case for the MPU?

Also, I think the correct wording is "device memory" rather than "IO memory".

Quoted 1 line

The access to IO memory will bypass

Ditto.

Quoted 8 lines

+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */

xen/arch/arm/arm64/head_mpu.S +3 -0
+ENTRY(enable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 96

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+    /* Enable EL2 MPU and D-cache */
+    orr   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(enable_mm)
+
+/* Disable MPU system, including data cache. */
+ENTRY(disable_mm)

I would rather not introduce this function until there is a caller. This is because, I believe, there are some assumptions on the state of the cache before we can turn off the MMU. So I would like to see the caller in order to assess whether this function makes sense.

Cheers,

Signature · 1 line
Julien Grall
#49 lore raw

RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Hi Julien

Quoted 27 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Hi Wei,

Quoted 11 lines · Wei Chen

From: Penny Zheng <[email protected]>

We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.

Quoted 3 lines

This raises the following question: Why can't we create the split regions right now?

The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes. Then we could reuse as much same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU all over the place.

Quoted 4 lines

In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).

Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?
So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/

Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.

One is in setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear. Otherwise it is hard and complex to maintain on runtime, especially during context switch.

Unchanging regions like xen text will not change during vcpu context switch. Flexible regions, like guest memory region, will display different contents during vcpu context switch.

Quoted 83 lines
Quoted 19 lines

So in this situation, we need to disable the whole MPU to update the protection regions.

In these helper, enable/disable MPU will also enable/disable the D-cache. There are two reasons for it:
1. Make the function semantic be consistent with enable_mmu. For MMU systems, enable_mmu will turn MMU and D-Cache at the same time.
2. When MPU is disabled, the MPU background attributes will be used. On some platforms, the background will treat all memory as device memory. The access to device memory will bypass the cache, even if the C bit is enabled in SCTLR. To avoid this implicit behavior, we disable cache with MPU explicitly to tell user that when MPU is disabled, the memory access is uncacheable.

In this patch, we also introduce a neutral name enable_mm for Xen to enable MMU/MPU. This can help us to keep one code flow in head.S

Quoted 6 lines
---
  xen/arch/arm/arm64/head.S     |  5 +++--
  xen/arch/arm/arm64/head_mmu.S |  4 ++--
  xen/arch/arm/arm64/head_mpu.S | 35

+++++++++++++++++++++++++++++++++++

Quoted 7 lines
  3 files changed, 40 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +0 -0
index 6c1a5f74a1..228f01db69 100644

xen/arch/arm/arm64/head.S real_start_efi: · line 261

Quoted 8 lines
xen/arch/arm/arm64/head.S +2 -1
@@ -255,7 +255,8 @@ real_start_efi:
           * and protection regions for MPU systems.
           */
          bl    prepare_early_mappings
-        bl    enable_mmu
+        /* Turn on MMU or MPU */
+        bl    enable_mm
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual

Address. */

xen/arch/arm/arm64/head.S GLOBAL(init_secondary) · line 319

Quoted 9 lines
xen/arch/arm/arm64/head.S +1 -1
          ldr   x0, =primary_switched
@@ -313,7 +314,7 @@ GLOBAL(init_secondary)
          bl    check_cpu_mode
          bl    cpu_init
          bl    prepare_early_mappings
-        bl    enable_mmu
+        bl    enable_mm
 
          /* We are still in the 1:1 mapping. Jump to the runtime Virtual

Address. */

Quoted 3 lines
xen/arch/arm/arm64/head.S +0 -0
          ldr   x0, =secondary_switched
(quoted diff) +0 -0

b/xen/arch/arm/arm64/head_mmu.S index fc64819a98..b542755bd2

100644

Quoted 20 lines
xen/arch/arm/arm64/head_mmu.S +2 -2
@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
   *
   * Clobbers x0 - x3
   */
-ENTRY(enable_mmu)
+ENTRY(enable_mm)
          PRINT("- Turning on paging -\r\n")
 
          /*
@@ -239,7 +239,7 @@ ENTRY(enable_mmu)
          msr   SCTLR_EL2, x0          /* now paging is enabled */
          isb                          /* Now, flush the icache */
          ret
-ENDPROC(enable_mmu)
+ENDPROC(enable_mm)
 
  /*
   * Remove the 1:1 map from the page-tables. It is not easy to keep
Quoted 2 lines

track diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S index f60611b556..5a1b03e293

100644

Quoted 4 lines
xen/arch/arm/arm64/head_mpu.S +0 -0
@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
 
Quoted 14 lines

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings)

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8 -0
      ret
  ENDPROC(prepare_early_mappings)
+
+/*
+ * Enable EL2 MPU and data cache. Because we will disable cache
+ * with MPU at the same time, in accordance with that, we have
+ * to enable cache with MPU at the same time in this function.
+ * When MPU is disabled, the MPU background attributes will
+ * be used. On some platform, the background will treat all
+ * memory as IO memory.

I was under the impression that all access would be treated as Device Memory when the MMU is off. Isn't it the case for the MPU?

Yes, without background region. Right now, background region is disabled here.
But if the Background region is enabled, then the MPU uses the default memory map as the Background region for generating the memory attributes when MPU is disabled. And it is also IMPLENMENTATION DEFINED~

Quoted 35 lines

Also, I think the correct wording is "device memory" rather than "IO memory".

Quoted 1 line

The access to IO memory will bypass

Ditto.

Quoted 8 lines

+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */

xen/arch/arm/arm64/head_mpu.S +3 -0
+ENTRY(enable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 96

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+    /* Enable EL2 MPU and D-cache */
+    orr   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(enable_mm)
+
+/* Disable MPU system, including data cache. */
+ENTRY(disable_mm)

I would rather not introduce this function until there is a caller. This is because, I believe, there are some assumptions on the state of the cache before we can turn off the MMU. So I would like to see the caller in order to assess whether this function makes sense.

Cheers,

--
Julien Grall

#50 lore raw

Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Quoted 1 line · Penny Zheng

Hi Julien

Hi Penny,

Quoted 28 lines
Quoted 28 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Hi Wei,

Quoted 11 lines · Wei Chen

From: Penny Zheng <[email protected]>

We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will be triggered.

Quoted 6 lines
Quoted 3 lines

This raises the following question: Why can't we create the split regions right now?

The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes.

The MMU code is going to change pretty soon (see [1] for some ground work). The runtime page-tables for CPU0 will be created in assembly code and never switched after (aside when using cache coloring).

Although, I don't think I will apply the proper permissions in assembly (this is a bit trickier than with the MPU).

Quoted 3 lines

Then we could reuse as much
same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU all over the place.

Do you have a list of those interfaces that would require #ifdef?

Quoted 9 lines
Quoted 5 lines

In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).

Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?

No. I am more concerned about...

Quoted 2 lines

So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/

... this. We don't only need to flush before they are accessed but also after if they are modified.

It is possible to do it correctly, but it requires to be very careful. So if we can avoid disabling the cache/MPU then it will be a lot better.

Quoted 8 lines

Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.

One is in setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear.

You should not need any reorg if you map the boot-only section towards in the higher slot index (or just after the fixed ones).

Cheers,

[1] [email protected]

Signature · 1 line
Julien Grall
#51 lore raw

RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Quoted 15 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 6:38 PM
To: Penny Zheng <[email protected]>; Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Quoted 4 lines
Quoted 1 line · Penny Zheng

Hi Julien

Hi Penny,

Hi Julien

Quoted 75 lines
Quoted 19 lines
Quoted 19 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Julien Grall
Sent: Monday, November 7, 2022 4:56 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

Hi Wei,

Quoted 2 lines · Wei Chen

From: Penny Zheng <[email protected]>

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

We need some helpers for Xen to enable/disable MPU in boot-time and runtime. For MPU enable helper, we know that it's an essential requirement of MPU system. But for MPU disable, we need to use it for some special situations. For example, in the progress of tranferring from boot-time to runtime, we need to update the MPU protection regions configuration, but we can't modify an MPU protection region if there is some data accessed by Xen. But in boot-time all of Xen text, data and BSS are in one MPU protection region, if Xen want to update this protection region, above restriction will

be triggered.

Quoted 7 lines
Quoted 4 lines

This raises the following question: Why can't we create the split regions right now?

The reason why we are not creating the split regions right now is that we are trying to go the same path MMU goes.

The MMU code is going to change pretty soon (see [1] for some ground work). The runtime page-tables for CPU0 will be created in assembly code and never switched after (aside when using cache coloring).

Although, I don't think I will apply the proper permissions in assembly (this is a bit trickier than with the MPU).

Quoted 3 lines

Then we could reuse as much
same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU all over the place.

Do you have a list of those interfaces that would require #ifdef?

Quoted 9 lines
Quoted 5 lines

In particular, disabling the MMU/Cache is fairly risky because you need to ensure that anything in the cache you care about have been written back to the RAM).

Hope I could understand your concern totally, you are worrying about stale info left in the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could be different before and after?

No. I am more concerned about...

Quoted 3 lines

So it is never enough that we only flush the variables which we will use during the disabling time, it should be everything in the cache...:/

... this. We don't only need to flush before they are accessed but also after if they are modified.

It is possible to do it correctly, but it requires to be very careful. So if we can avoid disabling the cache/MPU then it will be a lot better.

Quoted 4 lines

Since in current design, there are two time points in boot time where we will disable MPU/Cache to configure MPU.

Quoted 9 lines
Quoted 4 lines

One is in setup_mm, here, we will map XEN components by components, each MPU memory region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout before going runtime, and we will keep unchanging

regions in the front and flexible ones in the rear.

You should not need any reorg if you map the boot-only section towards in the higher slot index (or just after the fixed ones).

"in the higher slot index" is really shining a light in my mind ;) And I'll try to enable it in v2.

Quoted 6 lines

Cheers,

[1] [email protected]

--
Julien Grall

author
#52 lore raw

PATCH v6 11/11 xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

In reply to Wei Chen

From: Penny Zheng <[email protected]>

setup_fixmap and remove_identity_mapping are two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow in head.S doesn't need to use #ifdef to gate MPU/MMU code.

In MMU system, setup_fixmap is used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but before setup_fixmap. This restriction is the same as MMU system.

For remove_identity_mapping, we just need an empty function to make head.S code flow happy.

---
 xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
 .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
 2 files changed, 53 insertions(+)
xen/arch/arm/arm64/head_mpu.S +49 -0
index 5a1b03e293..336c0a630f 100644
@@ -20,13 +20,20 @@
 /*
  * In boot stage, we will use 1 MPU region:
  * Region#0: Normal memory for Xen text + data + bss (2MB)
+ * Region#1: Device memory for EARLY UART, size is defined
+ *           by platform's EARLY_UART_SIZE
  */
 #define BOOT_NORMAL_REGION_IDX  0x0
+#define BOOT_DEVICE_REGION_IDX  0x1
 
 /* MPU normal memory attributes. */
 #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
 #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
 
+/* MPU device memory attributes. */
+#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
+#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
+
 .macro write_pr, sel, prbar, prlar
     msr   PRSELR_EL2, \sel
     dsb   sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
     ret
 ENDPROC(prepare_early_mappings)
 
+/*
+ * In MMU system, setup_fixmap is used for Xen to map some essential data
+ * or devices in boot-time. In order to be consistent with MMU system, we
+ * inherit the function name for MPU system.
+ * setup_fixmap of MPU system will:
+ * - Map the early UART to MPU protection region when earlyprintk is
+ *   enabled (The PRINT can't be used after turning on MPU but before
+ *   setup_fixmap).
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(setup_fixmap)
+#ifdef CONFIG_EARLY_PRINTK
+    /* Map early uart to MPU device region for early printk. */
+    mov x0, #BOOT_DEVICE_REGION_IDX
+    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_DEVICE_MEM
+    orr x1, x1, x3
+
+    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
+    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_DEVICE_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar
+     */
+    write_pr x0, x1, x2
+#endif
+
+    ret
+ENDPROC(setup_fixmap)
+
+/* Stub of remove_identity_mapping for MPU systems */
+ENTRY(remove_identity_mapping)
+    ret
+ENDPROC(remove_identity_mapping)
+
 /*
  * Enable EL2 MPU and data cache. Because we will disable cache
  * with MPU at the same time, in accordance with that, we have
xen/arch/arm/include/asm/platforms/fvp_baser.h +4 -0
index 9450a411a9..acde3541a1 100644
@@ -11,4 +11,8 @@
 #define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
 #endif
 
+#ifdef CONFIG_EARLY_PRINTK
+#define EARLY_UART_SIZE   0x1000
+#endif
+
 #endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
Signature · 1 line
2.25.1
#53 lore raw

Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Hi,

Quoted 51 lines · Wei Chen

From: Penny Zheng <[email protected]>

setup_fixmap and remove_identity_mapping are two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow in head.S doesn't need to use #ifdef to gate MPU/MMU code.

In MMU system, setup_fixmap is used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but before setup_fixmap. This restriction is the same as MMU system.

For remove_identity_mapping, we just need an empty function to make head.S code flow happy.

---
  xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
  .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
  2 files changed, 53 insertions(+)
xen/arch/arm/arm64/head_mpu.S +7 -0
index 5a1b03e293..336c0a630f 100644
@@ -20,13 +20,20 @@
  /*
   * In boot stage, we will use 1 MPU region:
   * Region#0: Normal memory for Xen text + data + bss (2MB)
+ * Region#1: Device memory for EARLY UART, size is defined
+ *           by platform's EARLY_UART_SIZE
   */
  #define BOOT_NORMAL_REGION_IDX  0x0
+#define BOOT_DEVICE_REGION_IDX  0x1
 
  /* MPU normal memory attributes. */
  #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
  #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
 
+/* MPU device memory attributes. */
+#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
+#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
+
  .macro write_pr, sel, prbar, prlar
      msr   PRSELR_EL2, \sel
      dsb   sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
      ret

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +8 -0
  ENDPROC(prepare_early_mappings)
 
+/*
+ * In MMU system, setup_fixmap is used for Xen to map some essential data
+ * or devices in boot-time. In order to be consistent with MMU system, we
+ * inherit the function name for MPU system.
+ * setup_fixmap of MPU system will:
+ * - Map the early UART to MPU protection region when earlyprintk is
+ *   enabled (The PRINT can't be used after turning on MPU but before
+ *   setup_fixmap).

For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.

In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.

Quoted 23 lines
xen/arch/arm/arm64/head_mpu.S +23 -0
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(setup_fixmap)
+#ifdef CONFIG_EARLY_PRINTK
+    /* Map early uart to MPU device region for early printk. */
+    mov x0, #BOOT_DEVICE_REGION_IDX
+    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_DEVICE_MEM
+    orr x1, x1, x3
+
+    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
+    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_DEVICE_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar
+     */

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 119

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+    write_pr x0, x1, x2
+#endif
+
+    ret
+ENDPROC(setup_fixmap)
+
+/* Stub of remove_identity_mapping for MPU systems */
+ENTRY(remove_identity_mapping)
+    ret
+ENDPROC(remove_identity_mapping)

This stub could be avoided if you move the call to remove_identity_mapping in enable_mmu() as I did for arm32. See [1].

[1] https://lore.kernel.org/all/[email protected]/

Cheers,

Signature · 1 line
Julien Grall
#54 lore raw

RE: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Hi Julien

Quoted 68 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:02 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Hi,

Quoted 19 lines · Wei Chen

From: Penny Zheng <[email protected]>

setup_fixmap and remove_identity_mapping are two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow in head.S doesn't need to use #ifdef to gate MPU/MMU code.

In MMU system, setup_fixmap is used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but before setup_fixmap. This restriction is the same as MMU system.

For remove_identity_mapping, we just need an empty function to make head.S code flow happy.

Quoted 7 lines
---
  xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
  .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
  2 files changed, 53 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/arm64/head_mpu.S index 5a1b03e293..336c0a630f

100644

Quoted 27 lines
xen/arch/arm/arm64/head_mpu.S +7 -0
@@ -20,13 +20,20 @@
  /*
   * In boot stage, we will use 1 MPU region:
   * Region#0: Normal memory for Xen text + data + bss (2MB)
+ * Region#1: Device memory for EARLY UART, size is defined
+ *           by platform's EARLY_UART_SIZE
   */
  #define BOOT_NORMAL_REGION_IDX  0x0
+#define BOOT_DEVICE_REGION_IDX  0x1
 
  /* MPU normal memory attributes. */
  #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
  #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
 
+/* MPU device memory attributes. */
+#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
+#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
+
  .macro write_pr, sel, prbar, prlar
      msr   PRSELR_EL2, \sel
      dsb   sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
      ret
  ENDPROC(prepare_early_mappings)
 
Quoted 15 lines

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+/*
+ * In MMU system, setup_fixmap is used for Xen to map some essential
+data
+ * or devices in boot-time. In order to be consistent with MMU
+system, we
+ * inherit the function name for MPU system.
+ * setup_fixmap of MPU system will:
+ * - Map the early UART to MPU protection region when earlyprintk is
+ *   enabled (The PRINT can't be used after turning on MPU but before
+ *   setup_fixmap).

For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.

Yes, both removing identity mapping and using fixmap virtual memory layout are not applied to the MPU system.

And in MMU system, the setup_fixmap function has a behavior to map the UART for early printk. And we are only trying to pertain this behavior on MPU system.

Quoted 2 lines

In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.

Hmm, I'm a bit confused here. Which restriction you are trying to remove? The whole fixmap virtual memory layout?

Quoted 38 lines
Quoted 24 lines
xen/arch/arm/arm64/head_mpu.S +23 -0
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(setup_fixmap)
+#ifdef CONFIG_EARLY_PRINTK
+    /* Map early uart to MPU device region for early printk. */
+    mov x0, #BOOT_DEVICE_REGION_IDX
+    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_DEVICE_MEM
+    orr x1, x1, x3
+
+    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
+    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_DEVICE_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar
+     */

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 119

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+    write_pr x0, x1, x2
+#endif
+
+    ret
+ENDPROC(setup_fixmap)
+
+/* Stub of remove_identity_mapping for MPU systems */
+ENTRY(remove_identity_mapping)
+    ret
+ENDPROC(remove_identity_mapping)

This stub could be avoided if you move the call to remove_identity_mapping in enable_mmu() as I did for arm32. See [1].

Quoted 2 lines

[1] https://lore.kernel.org/all/[email protected]/

Thx! Understood, and I'll use the same logic for enable_mmu.

Quoted 4 lines

Cheers,

--
Julien Grall

#55 lore raw

Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Hi Penny,

Quoted 87 lines · Penny Zheng
Quoted 68 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: Monday, November 7, 2022 5:02 AM
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Penny Zheng <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Hi,

Quoted 19 lines · Wei Chen

From: Penny Zheng <[email protected]>

setup_fixmap and remove_identity_mapping are two functions that are used in Xen boot-time code flow. We implement these two functions for MPU system, in this case, the code flow in head.S doesn't need to use #ifdef to gate MPU/MMU code.

In MMU system, setup_fixmap is used for Xen to map some essentail data or devices in boot-time. For MPU system, we still have this requirement, we map the early UART to MPU protection region when earlyprintk is enabled. This also means PRINT can't be used after turning on MPU but before setup_fixmap. This restriction is the same as MMU system.

For remove_identity_mapping, we just need an empty function to make head.S code flow happy.

Quoted 7 lines
---
   xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
   .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
   2 files changed, 53 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/arm64/head_mpu.S index 5a1b03e293..336c0a630f

100644

Quoted 27 lines
xen/arch/arm/arm64/head_mpu.S +7 -0
@@ -20,13 +20,20 @@
   /*
    * In boot stage, we will use 1 MPU region:
    * Region#0: Normal memory for Xen text + data + bss (2MB)
+ * Region#1: Device memory for EARLY UART, size is defined
+ *           by platform's EARLY_UART_SIZE
    */
   #define BOOT_NORMAL_REGION_IDX  0x0
+#define BOOT_DEVICE_REGION_IDX  0x1
 
   /* MPU normal memory attributes. */
   #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
   #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
 
+/* MPU device memory attributes. */
+#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
+#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
+
   .macro write_pr, sel, prbar, prlar
       msr   PRSELR_EL2, \sel
       dsb   sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
       ret
   ENDPROC(prepare_early_mappings)
 
Quoted 15 lines

xen/arch/arm/arm64/head_mpu.S ENTRY(prepare_early_mappings) · line 86

Quoted 10 lines
xen/arch/arm/arm64/head_mpu.S +10 -0
+/*
+ * In MMU system, setup_fixmap is used for Xen to map some essential
+data
+ * or devices in boot-time. In order to be consistent with MMU
+system, we
+ * inherit the function name for MPU system.
+ * setup_fixmap of MPU system will:
+ * - Map the early UART to MPU protection region when earlyprintk is
+ *   enabled (The PRINT can't be used after turning on MPU but before
+ *   setup_fixmap).

For the MMU, we have this restriction because the fixmap could clash with the identity mapping. I don't think there are such restrictions for the MPU and therefore it seems strange to pertain the same behavior.

Yes, both removing identity mapping and using fixmap virtual memory layout are not applied to the MPU system.

Quoted 7 lines

And in MMU system, the setup_fixmap function has a behavior to map the UART for early printk. And we are only trying to pertain this behavior on MPU system.

Quoted 2 lines

In fact, I have plan to get rid of this restriction even for the MMU. So better this restriction is not spread if we can.

Hmm, I'm a bit confused here. Which restriction you are trying to remove?

The fact that the fixmap entry is not created in create_page_tables() (now prepare_early_mappings()).

Cheers,

Signature · 1 line
Julien Grall
author
#56 lore raw
In reply to Wei Chen

Sorry for the wrong patch version. This is the first version of this series.

Cheers,
Wei Chen

Quoted 96 lines

-----Original Message-----
From: Wei Chen <[email protected]>
Sent: 2022年11月4日 18:08
To: [email protected]
Cc: nd <[email protected]>; Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.

We will split these patches to several parts, this series is the part#1, the full PoC can be found in [3]. More software for Armv8-R64 can be found in [4];

[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-
05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-
/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/

Penny Zheng (3):
xen/arm64: create boot-time MPU protection regions xen/arm64: introduce helpers for MPU enable/disable xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Wei Chen (8):
xen/arm: remove xen_phys_start and xenheap_phys_end from config.h xen/arm: add iounmap after initrd has been loaded in domain_build xen/arm: disable EFI boot services for MPU systems xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA xen/arm: define Xen start address for FVP BaseR platform xen/arm: split MMU and MPU config files from config.h xen/arm: implement FIXMAP_ADDR for MPU systems xen/arm64: move MMU related code from head.S to head_mmu.S

 xen/arch/arm/Kconfig                          |  15 +-
 xen/arch/arm/arm64/Makefile                   |   5 +
 xen/arch/arm/arm64/head.S                     | 429 ++----------------
 xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
 xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
 xen/arch/arm/domain_build.c                   |   2 +
 xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
 xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
 xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
 xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
 xen/arch/arm/include/asm/config.h             |  99 +---
 xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
 xen/arch/arm/include/asm/config_mpu.h         |  29 ++
 xen/arch/arm/include/asm/fixmap.h             |  25 +
 xen/arch/arm/include/asm/flushtlb.h           |  22 +
 .../arm/include/asm/platforms/fvp_baser.h     |  18 +
 xen/arch/arm/platforms/Kconfig                |  16 +-
 17 files changed, 976 insertions(+), 500 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

--
2.25.1

#57 lore raw
In reply to Wei Chen

Hi Wei,

Quoted 24 lines · Wei Chen

The Armv-R architecture profile was designed to support use cases that have a high sensitivity to deterministic execution. (e.g. Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm architecture targeted at the Real-time profile. It introduces virtualization at the highest security level while retaining the Protected Memory System Architecture (PMSA) based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex-R82, which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. The latest Armv8-R64 document can be found [1]. And the features of Armv8-R64 architecture:
- An exception model that is compatible with the Armv8-A model
- Virtualization with support for guest operating systems
- PMSA virtualization using MPUs In EL2.
- Adds support for the 64-bit A64 instruction set.
- Supports up to 48-bit physical addressing.
- Supports three Exception Levels (ELs)
- Secure EL2 - The Highest Privilege
- Secure EL1 - RichOS (MMU) or RTOS (MPU)
- Secure EL0 - Application Workloads
- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

Quoted 6 lines

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.

Cheers,

Signature · 1 line
Julien Grall
author
#58 lore raw

Hi Julien,

Quoted 7 lines
Quoted 7 lines

- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

Quoted 8 lines
Quoted 2 lines

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.

These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a relatively complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the stage#1 will have about 30 patches?

Cheers,
Wei Chen

Quoted 4 lines

Cheers,

--
Julien Grall

#59 lore raw
Quoted 1 line · Wei Chen

Hi Julien,

Hi,

Quoted 23 lines
Quoted 8 lines
Quoted 8 lines

- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

Quoted 8 lines
Quoted 2 lines

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.

These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a relatively complete thing in this part#1 series.

Quoted 3 lines

We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the stage#1 will have about 30 patches?

30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).

Cheers,

Signature · 1 line
Julien Grall
author
#60 lore raw

Hi Julien,

Quoted 43 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

Quoted 1 line · Wei Chen

Hi Julien,

Hi,

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines

- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

Quoted 8 lines
Quoted 2 lines

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.

Quoted 4 lines

These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a

relatively

Quoted 4 lines

complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the

Quoted 8 lines

stage#1

Quoted 1 line

will have about 30 patches?

30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).

I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.

For Armv8-R model, you can download Armv8-R AEM FVP model from link[1]. It's license free.

[1] https://developer.arm.com/downloads/-/arm-ecosystem-models

Cheers,
Wei Chen

Quoted 4 lines

Cheers,

--
Julien Grall

#61 lore raw
Quoted 48 lines · Wei Chen

Hi Julien,

Quoted 46 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

Quoted 1 line · Wei Chen

Hi Julien,

Hi,

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines

- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

Quoted 8 lines
Quoted 2 lines

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the series is actually doing.

Quoted 4 lines

These 11 patches are part#1 of stage#1, the full stage#1 has about 30 patches. We have some concerns if we send so many patches at once, the review pressure of maintainers may be very high, so we only choose about 10 to send as part of it. But this also means that we can't do a

relatively

Quoted 4 lines

complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches can be Reviewed efficiently. Can we send the patches by stages, even the

stage#1

Quoted 1 line

will have about 30 patches?

Quoted 8 lines
Quoted 5 lines

30 patches in a go is no too bad. I would personally prefer that because at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-R model).

I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.

One suggestion to make things easier to review and to commit is to organize the series in a way so that the first 10 patches can still be committed first independently, even if all 30 patches are sent together.

Or alternatively only send 10 patches but also add a link to a github/gitlab tree with all the 30+ patches so that maintainers can have a look how the whole work fit together.

I think we are all on the same page -- I just wanted to highlight that we don't have to finish the review of all 30 patches before we can start committing some of the initial patches in the series.

Cheers,

Stefano

author
#62 lore raw

Hi Stefano, Julien,

Quoted 76 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2022年11月11日 6:26
To: Wei Chen <[email protected]>
Cc: Julien Grall <[email protected]>; [email protected]; nd <[email protected]>; Stefano Stabellini <[email protected]>; Bertrand Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

Quoted 7 lines · Wei Chen

Hi Julien,

Quoted 5 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年11月7日 18:16
To: Wei Chen <[email protected]>; [email protected]
Cc: nd <[email protected]>; Stefano Stabellini <[email protected]>;

Bertrand

Quoted 3 lines
Quoted 3 lines

Marquis <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to

Xen -

Quoted 13 lines
Quoted 13 lines

Part#1

Quoted 1 line · Wei Chen

Hi Julien,

Hi,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

- Supports only a single Security state - Secure.
- MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

These patch series are implementing the Armv8-R64 MPU support for Xen, which are based on the discussion of "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating

domain).

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

However... HAS_MPU is not defined and I was expecting mm.c to be modified to cater the MPU support. So I am a bit ensure what the

series

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

is actually doing.

These 11 patches are part#1 of stage#1, the full stage#1 has about

30

Quoted 1 line
Quoted 1 line
Quoted 1 line

patches. We have some concerns if we send so many patches at once,

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

review pressure of maintainers may be very high, so we only choose

about

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

10 to send as part of it. But this also means that we can't do a

relatively

Quoted 3 lines

complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches

can be

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Reviewed efficiently. Can we send the patches by stages, even the

stage#1

Quoted 1 line

will have about 30 patches?

30 patches in a go is no too bad. I would personally prefer that

because

Quoted 2 lines
Quoted 2 lines

at least I have better idea of the shape of the code after stage#1 and also possibly test it (I need to check if I have access for the ARMv8-

R

Quoted 3 lines
Quoted 2 lines

model).

Quoted 8 lines
Quoted 2 lines

I also prefer to this way. After we have addressed the comments in this series, we will send the full stage#1 patches together in v2.

One suggestion to make things easier to review and to commit is to organize the series in a way so that the first 10 patches can still be committed first independently, even if all 30 patches are sent together.

I think this is foreseeable, and we have done in this way internally. Every patch can be built and will not broken other architectures.

Quoted 4 lines

Or alternatively only send 10 patches but also add a link to a github/gitlab tree with all the 30+ patches so that maintainers can have a look how the whole work fit together.

In this series we have linked the gitlab branch with the full patches.

Quoted 4 lines

I think we are all on the same page -- I just wanted to highlight that we don't have to finish the review of all 30 patches before we can start committing some of the initial patches in the series.

Yes, I agree with it. And above solutions are ok for us. There will not be much difference in efforts for these two ways for us, so if you guys think which method is the most efficient, I will follow it.

Cheers,
Wei Chen

Quoted 3 lines

Cheers,

Stefano