Skip to contentSkip to conversation

Fourth MPU Series

PATCH In review Harry Ramsey 4 participants 12 messages Jan 14 – Jan 15

lore
Share & tools
lore
#0 lore raw

v4 cover Fourth MPU Series

This series aims to further the ongoing work to introduce support for MPU systems in xen.

The patches in this series implement various memory functions and enable the hypervisor timer.

Luca Fancellu (3):
arm/mpu: Implement copy_from_paddr for MPU systems arm/mpu: Implement vmap functions for MPU arm/mpu: Introduce modify_after_init_mappings

Penny Zheng (3):
arm/mpu: Implement free_init_memory for MPU systems
arm: Use secure hypervisor timer in MPU system arm/mpu: Map domain page in AArch64 MPU systems

 xen/arch/arm/Kconfig                     |   1 +
 xen/arch/arm/include/asm/arm64/sysregs.h |  11 ++
 xen/arch/arm/include/asm/mpu/mm.h        |  10 ++
 xen/arch/arm/include/asm/setup.h         |   5 +
 xen/arch/arm/mmu/setup.c                 |  15 ++
 xen/arch/arm/mpu/Makefile                |   1 +
 xen/arch/arm/mpu/domain-page.c           |  46 +++++
 xen/arch/arm/mpu/mm.c                    | 204 ++++++++++++++++++-----
 xen/arch/arm/mpu/setup.c                 |  54 +++++-
 xen/arch/arm/mpu/vmap.c                  |  14 +-
 xen/arch/arm/setup.c                     |  15 +-
 11 files changed, 318 insertions(+), 58 deletions(-)
 create mode 100644 xen/arch/arm/mpu/domain-page.c

--
2.43.0

#1 lore raw

PATCH v4 1/6 arm/mpu: Implement copy_from_paddr for MPU systems

From: Luca Fancellu <[email protected]>

Implement the function copy_from_paddr variant for MPU systems, using the map_pages_to_xen/destroy_xen_mappings to temporarily map the memory range to be copied.

---
v4:
- No changes
v3:
- No changes
v2:
- add Michal R-by

---
 xen/arch/arm/mpu/setup.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
xen/arch/arm/mpu/setup.c +13 -1
index 163573b932..ec264f54f2 100644
@@ -91,7 +91,19 @@ void * __init early_fdt_map(paddr_t fdt_paddr)
  */
 void __init copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
 {
-    BUG_ON("unimplemented");
+    paddr_t start_pg = round_pgdown(paddr);
+    paddr_t end_pg   = round_pgup(paddr + len);
+    unsigned long nr_mfns = (end_pg - start_pg) >> PAGE_SHIFT;
+    mfn_t mfn = maddr_to_mfn(start_pg);
+
+    if ( map_pages_to_xen(start_pg, mfn, nr_mfns, PAGE_HYPERVISOR_WC) )
+        panic("Unable to map range for copy_from_paddr\n");
+
+    memcpy(dst, maddr_to_virt(paddr), len);
+    clean_dcache_va_range(dst, len);
+
+    if ( destroy_xen_mappings(start_pg, end_pg) )
+        panic("Unable to unmap range for copy_from_paddr\n");
 }
 
 void __init remove_early_mappings(void)
Signature · 1 line
2.43.0
#2 lore raw

PATCH v4 2/6 arm/mpu: Implement vmap functions for MPU

In reply to Harry Ramsey

From: Luca Fancellu <[email protected]>

HAS_VMAP is not enabled on MPU systems, but the vmap functions are used in places across common code. In order to keep the existing code and maintain correct functionality, implement the vmap_contig and vunmap functions for MPU systems.

Introduce a helper function destroy_xen_mapping_containing to aid with unmapping an entire region when only the start address is known.

---
v4:
- Code style fixes
v3:
- Add additional comments for clarity regarding MPUMAP_REGION checks
- Ensure context_sync_mpu occurs after destroy_entire_xen_mapping
- Fix deadlock if destroy_entire_xen_mapping is called with an address that does not belong to a region
v2:
- Rename destroy_entire_xen_mapping to destroy_xen_mapping_containing
- Improve code documentation
- Refactor nested code
- Fix ignored rc error code in destroy_xen_mapping_containing

---
 xen/arch/arm/include/asm/mpu/mm.h | 10 ++++
 xen/arch/arm/mpu/mm.c             | 83 ++++++++++++++++++++++++++-----
 xen/arch/arm/mpu/vmap.c           | 14 ++++--
 3 files changed, 92 insertions(+), 15 deletions(-)
xen/arch/arm/include/asm/mpu/mm.h +10 -0
index e1ded6521d..1b5ffa5b64 100644
@@ -111,6 +111,16 @@ pr_t pr_of_addr(paddr_t base, paddr_t limit, unsigned int flags);
 int mpumap_contains_region(pr_t *table, uint8_t nr_regions, paddr_t base,
                            paddr_t limit, uint8_t *index);
 
+
+/*
+ * Destroys and frees (if reference count is 0) an entire xen mapping on MPU
+ * systems where only the start address is known.
+ *
+ * @param s     Start address of memory region to be destroyed.
+ * @return:     0 on success, negative on error.
+ */
+int destroy_xen_mapping_containing(paddr_t s);
+
 #endif /* __ARM_MPU_MM_H__ */
 
 /*
xen/arch/arm/mpu/mm.c +71 -12

101 lines not on this page · raw message

xen/arch/arm/mpu/vmap.c +11 -3
index f977b79cd4..54d949e7ce 100644
@@ -1,19 +1,27 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
 #include <xen/bug.h>
+#include <xen/mm.h>
 #include <xen/mm-frame.h>
 #include <xen/types.h>
 #include <xen/vmap.h>
 
 void *vmap_contig(mfn_t mfn, unsigned int nr)
 {
-    BUG_ON("unimplemented");
-    return NULL;
+    paddr_t base = mfn_to_maddr(mfn);
+
+    if ( map_pages_to_xen(base, mfn, nr, PAGE_HYPERVISOR ) )
+        return NULL;
+
+    return maddr_to_virt(base);
 }
 
 void vunmap(const void *va)
 {
-    BUG_ON("unimplemented");
+    paddr_t base = virt_to_maddr(va);
+
+    if ( destroy_xen_mapping_containing(base) )
+        panic("Failed to vunmap region\n");
 }
 
 /*
Signature · 1 line
2.43.0
Reviewed-by on v4 2/6
#3 lore raw

Re: [PATCH v4 2/6] arm/mpu: Implement vmap functions for MPU

Quoted 7 lines · Harry Ramsey

From: Luca Fancellu <[email protected]>

HAS_VMAP is not enabled on MPU systems, but the vmap functions are used in places across common code. In order to keep the existing code and maintain correct functionality, implement the vmap_contig and vunmap functions for MPU systems.

Quoted 5 lines

Introduce a helper function destroy_xen_mapping_containing to aid with unmapping an entire region when only the start address is known.

~Michal

#4 lore raw

PATCH v4 3/6 arm/mpu: Implement free_init_memory for MPU systems

In reply to Harry Ramsey

From: Penny Zheng <[email protected]>

Implement the function free_init_memory for MPU systems. In order to support this, the function modify_xen_mappings is implemented.

On MPU systems, we map the init text and init data sections using separate MPU memory regions. Therefore these are removed separately in free_init_memory.

Additionally remove warning messages from is_mm_attr_match as some attributes can now be updated by xen_mpumap_update_entry.

---
v4:
- Refactor printk regarding memory modifications
- Add Michal R-By
v3:
- Refactor MPU_ATTR_* defines
v2:
- Refactor is_mm_attr_match to return logical values regarding the attribute mismatch.
- Improve code documentation.

---
 xen/arch/arm/include/asm/setup.h |   2 +
 xen/arch/arm/mpu/mm.c            | 119 +++++++++++++++++++++++--------
 2 files changed, 93 insertions(+), 28 deletions(-)
xen/arch/arm/include/asm/setup.h +2 -0
index 1eaf13bd66..005cf7be59 100644
@@ -65,6 +65,8 @@ int map_irq_to_domain(struct domain *d, unsigned int irq,
 int map_range_to_domain(const struct dt_device_node *dev,
                         uint64_t addr, uint64_t len, void *data);
 
+extern const char __init_data_begin[], __bss_start[], __bss_end[];
+
 struct init_info
 {
     /* Pointer to the stack, used by head.S when entering in C */
xen/arch/arm/mpu/mm.c +91 -28

157 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw

PATCH v4 4/6 arm/mpu: Introduce modify_after_init_mappings

In reply to Harry Ramsey

From: Luca Fancellu <[email protected]>

During init_done, Xen sets the permissions of all symbols marked with __ro_after_init to be read-only. This does not work on MPU systems at present because part-region modification is not supported.

Therefore introduce the function modify_after_init_mappings for MMU and MPU, to handle the divergent approaches to setting permissions of __ro_after_init symbols.

For MPU systems modify_xen_mappings will shrink the RW mapping on one side and extend the RO mapping on the other. This approach prevents wasting an additional region between RW and RO mappings.

As the new function is marked with __init, it needs to be called before free_init_memory.

---
v4:
- No changes
v3:
- Add Michal R-by
v2:
- No changes

---
 xen/arch/arm/include/asm/setup.h |  3 +++
 xen/arch/arm/mmu/setup.c         | 15 ++++++++++++
 xen/arch/arm/mpu/mm.c            |  2 +-
 xen/arch/arm/mpu/setup.c         | 40 ++++++++++++++++++++++++++++++++
 xen/arch/arm/setup.c             | 15 ++----------
 5 files changed, 61 insertions(+), 14 deletions(-)
xen/arch/arm/include/asm/setup.h +3 -0
index 005cf7be59..899e33925c 100644
@@ -78,6 +78,9 @@ struct init_info
 paddr_t consider_modules(paddr_t s, paddr_t e, uint32_t size, paddr_t align,
                          int first_mod);
 
+/* Modify some mappings after the init is done */
+void modify_after_init_mappings(void);
+
 #endif
 /*
  * Local variables:
xen/arch/arm/mmu/setup.c +15 -0
index 9b874f8ab2..d042f73597 100644
@@ -213,6 +213,21 @@ void __init remove_early_mappings(void)
     BUG_ON(rc);
 }
 
+void __init modify_after_init_mappings(void)
+{
+    /*
+     * We have finished booting. Mark the section .data.ro_after_init
+     * read-only.
+     */
+    int rc = modify_xen_mappings((unsigned long)&__ro_after_init_start,
+                                 (unsigned long)&__ro_after_init_end,
+                                 PAGE_HYPERVISOR_RO);
+
+    if ( rc )
+        panic("Unable to mark the .data.ro_after_init section read-only (rc = %d)\n",
+              rc);
+}
+
 /*
  * After boot, Xen page-tables should not contain mapping that are both
  * Writable and eXecutables.
xen/arch/arm/mpu/mm.c +1 -1
index cfd75abdb0..6b3b0b06e9 100644
@@ -35,7 +35,7 @@ DECLARE_BITMAP(xen_mpumap_mask, MAX_MPU_REGION_NR) \
 /* EL2 Xen MPU memory region mapping table. */
 pr_t __cacheline_aligned __section(".data") xen_mpumap[MAX_MPU_REGION_NR];
 
-static DEFINE_SPINLOCK(xen_mpumap_lock);
+DEFINE_SPINLOCK(xen_mpumap_lock);
 
 static void __init __maybe_unused build_assertions(void)
 {
xen/arch/arm/mpu/setup.c +40 -0
index ec264f54f2..55317ee318 100644
@@ -8,11 +8,14 @@
 #include <xen/pfn.h>
 #include <xen/types.h>
 #include <xen/sizes.h>
+#include <xen/spinlock.h>
 #include <asm/setup.h>
 
 static paddr_t __initdata mapped_fdt_base = INVALID_PADDR;
 static paddr_t __initdata mapped_fdt_limit = INVALID_PADDR;
 
+extern spinlock_t xen_mpumap_lock;
+
 void __init setup_pagetables(void) {}
 
 void * __init early_fdt_map(paddr_t fdt_paddr)
@@ -106,6 +109,43 @@ void __init copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
         panic("Unable to unmap range for copy_from_paddr\n");
 }
 
+void __init modify_after_init_mappings(void)
+{
+    int rc;
+    uint8_t idx_rodata;
+    uint8_t idx_rwdata;
+
+    spin_lock(&xen_mpumap_lock);
+
+    rc = mpumap_contains_region(xen_mpumap, max_mpu_regions,
+                                (unsigned long)_srodata,
+                                (unsigned long)_erodata,
+                                &idx_rodata);
+
+    if ( rc < MPUMAP_REGION_FOUND )
+        panic("Unable to find rodata section (rc = %d)\n", rc);
+
+    rc = mpumap_contains_region(xen_mpumap, max_mpu_regions,
+                                (unsigned long)__ro_after_init_start,
+                                (unsigned long)__init_begin,
+                                &idx_rwdata);
+
+    if ( rc < MPUMAP_REGION_FOUND )
+        panic("Unable to find rwdata section (rc = %d)\n", rc);
+
+    /* Shrink rwdata section to begin at __ro_after_init_end */
+    pr_set_base(&xen_mpumap[idx_rwdata], (unsigned long)__ro_after_init_end);
+
+    /* Extend rodata section to end at __ro_after_init_end */
+    pr_set_limit(&xen_mpumap[idx_rodata], (unsigned long)__ro_after_init_end);
+
+    write_protection_region(&xen_mpumap[idx_rwdata], idx_rwdata);
+    write_protection_region(&xen_mpumap[idx_rodata], idx_rodata);
+    context_sync_mpu();
+
+    spin_unlock(&xen_mpumap_lock);
+}
+
 void __init remove_early_mappings(void)
 {
     int rc = destroy_xen_mappings(round_pgdown(mapped_fdt_base),
xen/arch/arm/setup.c +2 -13
index 7ad870e382..6310a47d68 100644
@@ -66,23 +66,12 @@ domid_t __read_mostly max_init_domid;
 
 static __used void noreturn init_done(void)
 {
-    int rc;
-
     /* Must be done past setting system_state. */
     unregister_init_virtual_region();
 
-    free_init_memory();
+    modify_after_init_mappings();
 
-    /*
-     * We have finished booting. Mark the section .data.ro_after_init
-     * read-only.
-     */
-    rc = modify_xen_mappings((unsigned long)&__ro_after_init_start,
-                             (unsigned long)&__ro_after_init_end,
-                             PAGE_HYPERVISOR_RO);
-    if ( rc )
-        panic("Unable to mark the .data.ro_after_init section read-only (rc = %d)\n",
-              rc);
+    free_init_memory();
 
     startup_cpu_idle_loop();
 }
Signature · 1 line
2.43.0
#6 lore raw

PATCH v4 5/6 arm: Use secure hypervisor timer in MPU system

In reply to Harry Ramsey

From: Penny Zheng <[email protected]>

As MPU systems only have one secure state, we have to use secure EL2 hypervisor timer for Xen in secure EL2.

---
v4:
- No changes
v3:
- Add Ayan R-by
- Add Michal A-by
v2:
- Remove unncessary kconfig attribute.
- Remove unncessary hypervisor timer macro.

---
 xen/arch/arm/include/asm/arm64/sysregs.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
xen/arch/arm/include/asm/arm64/sysregs.h +11 -0
index 7dfd20414d..19d409d3eb 100644
@@ -462,6 +462,17 @@
 #define ZCR_ELx_LEN_SIZE             9
 #define ZCR_ELx_LEN_MASK             0x1ff
 
+#ifdef CONFIG_MPU
+/*
+ * The Armv8-R AArch64 architecture always executes code in Secure
+ * state with EL2 as the highest exception level.
+ *
+ * Hypervisor timer registers for Secure EL2.
+ */
+#define CNTHP_CTL_EL2   CNTHPS_CTL_EL2
+#define CNTHP_CVAL_EL2  CNTHPS_CVAL_EL2
+#endif
+
 #define REGION_TEXT_PRBAR       0x38    /* SH=11 AP=10 XN=00 */
 #define REGION_RO_PRBAR         0x3A    /* SH=11 AP=10 XN=10 */
 #define REGION_DATA_PRBAR       0x32    /* SH=11 AP=00 XN=10 */
Signature · 1 line
2.43.0
#7 lore raw

PATCH v4 6/6 arm/mpu: Map domain page in AArch64 MPU systems

In reply to Harry Ramsey

From: Penny Zheng <[email protected]>

In MPU systems, we implement map_domain_page()/unmap_domain_page() through mapping the domain page with a MPU region on demand.

---
v4:
- Remove duplicate code by having map_domain_page and unmap_domain_page use vmap_contig and vunmap
v3:
- No changes
v2:
- No changes

---
 xen/arch/arm/Kconfig           |  1 +
 xen/arch/arm/mpu/Makefile      |  1 +
 xen/arch/arm/mpu/domain-page.c | 46 ++++++++++++++++++++++++++++++++++
 3 files changed, 48 insertions(+)
 create mode 100644 xen/arch/arm/mpu/domain-page.c
xen/arch/arm/Kconfig +1 -0
index cf6af68299..baa6c4cf15 100644
@@ -91,6 +91,7 @@ config MMU
 
 config MPU
 	bool "MPU" if UNSUPPORTED
+	select ARCH_MAP_DOMAIN_PAGE if ARM_64
 	select STATIC_MEMORY
 	help
 	  Memory Protection Unit (MPU). Select if you plan to run Xen on ARMv8-R
xen/arch/arm/mpu/Makefile +1 -0
index 4963c8b550..940297af3f 100644
@@ -1,5 +1,6 @@
 obj-$(CONFIG_ARM_32) += arm32/
 obj-$(CONFIG_ARM_64) += arm64/
+obj-$(CONFIG_ARM_64) += domain-page.o
 obj-y += mm.o
 obj-y += p2m.o
 obj-y += setup.init.o
xen/arch/arm/mpu/domain-page.c new file +46 -0

46 lines not on this page · raw message

Signature · 1 line
2.43.0
#8 lore raw

Re: [PATCH v4 6/6] arm/mpu: Map domain page in AArch64 MPU systems

Hi Harry,

Can we expand this to cover Arm32 as well ? I did some test and both Arm32 and Arm64 get to the same point.

Quoted 29 lines · Harry Ramsey

From: Penny Zheng <[email protected]>

In MPU systems, we implement map_domain_page()/unmap_domain_page() through mapping the domain page with a MPU region on demand.

---
v4:
- Remove duplicate code by having map_domain_page and unmap_domain_page use vmap_contig and vunmap
v3:
- No changes
v2:
- No changes

---
  xen/arch/arm/Kconfig           |  1 +
  xen/arch/arm/mpu/Makefile      |  1 +
  xen/arch/arm/mpu/domain-page.c | 46 ++++++++++++++++++++++++++++++++++
  3 files changed, 48 insertions(+)

create mode 100644 xen/arch/arm/mpu/domain-page.c

xen/arch/arm/Kconfig +0 -0
index cf6af68299..baa6c4cf15 100644

xen/arch/arm/Kconfig config MMU · line 94

Quoted 4 lines
xen/arch/arm/Kconfig +1 -0
@@ -91,6 +91,7 @@ config MMU
 
  config MPU
  	bool "MPU" if UNSUPPORTED
+	select ARCH_MAP_DOMAIN_PAGE if ARM_64

Remove ARM_64

xen/arch/arm/mpu/Makefile line 3

Quoted 11 lines
xen/arch/arm/Kconfig +0 -0
  	select STATIC_MEMORY
  	help
  	  Memory Protection Unit (MPU). Select if you plan to run Xen on ARMv8-R
xen/arch/arm/mpu/Makefile +1 -0
index 4963c8b550..940297af3f 100644
@@ -1,5 +1,6 @@
  obj-$(CONFIG_ARM_32) += arm32/
  obj-$(CONFIG_ARM_64) += arm64/
+obj-$(CONFIG_ARM_64) += domain-page.o

obj-y += domain-page.o

Quoted 45 lines
xen/arch/arm/mpu/Makefile +0 -0
  obj-y += mm.o
  obj-y += p2m.o
  obj-y += setup.init.o
xen/arch/arm/mpu/domain-page.c new file +36 -0
new file mode 100644
index 0000000000..b79afc493b
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#include <xen/bug.h>
+#include <xen/domain_page.h>
+#include <xen/mm.h>
+#include <xen/mm-frame.h>
+#include <xen/types.h>
+#include <xen/vmap.h>
+
+void *map_domain_page_global(mfn_t mfn)
+{
+    BUG_ON("unimplemented");
+    return NULL;
+}
+
+/* Map a page of domheap memory */
+void *map_domain_page(mfn_t mfn)
+{
+    return vmap_contig(mfn, 1);
+}
+
+/* Release a mapping taken with map_domain_page() */
+void unmap_domain_page(const void *ptr)
+{
+    vunmap(ptr);
+}
+
+mfn_t domain_page_map_to_mfn(const void *ptr)
+{
+    BUG_ON("unimplemented");
+    return INVALID_MFN;
+}
+
+void unmap_domain_page_global(const void *va)
+{
+    BUG_ON("unimplemented");

xen/arch/arm/mpu/domain-page.c line 46

Quoted 10 lines
xen/arch/arm/mpu/domain-page.c new file +10 -0
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

The rest LGTM.

With this, one can see R52 booting to the same level as R82.

(XEN) load tracking window length 1073741824 ns
(XEN) Allocated console ring of 16 KiB.
(XEN) VFP implementer 0x41 architecture 3 part 0x40 variant 0x2 rev 0x3 (XEN) CPU0: Guest atomics will try 1 times before pausing the domain (XEN) Brought up 1 CPUs
(XEN) Xen BUG at arch/arm/arm32/mpu/p2m.c:9

- Ayan

#9 lore raw

Re: [PATCH v4 6/6] arm/mpu: Map domain page in AArch64 MPU systems

Quoted 7 lines · Ayan Kumar

Hi Harry,

Can we expand this to cover Arm32 as well ? I did some test and both Arm32 and Arm64 get to the same point.

The only problem is that we don’t have an Arm32 setup to test these, if the maintainer are ok we can do it, but then it should be you to test on arm32.

Also, in case, could it be done while committing? (provided that the maintainer give their ack)

Cheers,
Luca

#10 lore raw

Re: [PATCH v4 6/6] arm/mpu: Map domain page in AArch64 MPU systems

Hi Luca,

Quoted 8 lines
Quoted 6 lines · Ayan Kumar

Hi Harry,

Can we expand this to cover Arm32 as well ? I did some test and both Arm32 and Arm64 get to the same point.

The only problem is that we don’t have an Arm32 setup to test these, if the maintainer are ok we can do it, but then it should be you to test on arm32.

yes, I have the setup and can test the changes for arm32-v8r. You can test if it compiles or not.

It saves some time and effort to send another patch for removing Kconfig. :-)

- Ayan

Reviewed-by
#11 lore raw

Re: [PATCH v4 6/6] arm/mpu: Map domain page in AArch64 MPU systems

In reply to Luca Fancellu
Quoted 4 lines · Luca Fancellu
Quoted 4 lines · Ayan Kumar
Quoted 8 lines
Quoted 3 lines

Hi Harry,

Can we expand this to cover Arm32 as well ? I did some test and both Arm32 and Arm64 get to the same point.

The only problem is that we don’t have an Arm32 setup to test these, if the maintainer are ok we can do it, but then it should be you to test on arm32.

Also, in case, could it be done while committing? (provided that the maintainer give their ack)

I can do that later on when committing.

~Michal