Remove the directmap
v5 cover Remove the directmap
Hi,
I picked v4 of this series and run it through XenRT extensively, fixing crashes and errors as I hit them. Likewise, I've run it through Gitlab, fixing various CI failures. I listed all changes per patch. I fixed ARM to the extent that "Gitlab is happy when CONFIG_ONDEMAND_DIRECTMAP=y", but I didn't go much further than that.
I _THINK_ I've covered previously unaddressed feedback, but please speak up if I missed something. Otherwise, same old same old.
Cheers,
Alejandro
============== Original cover letter ==============
Hi all,
A few years ago, Wei Liu implemented a PoC to remove the directmap from Xen. The last version was sent by Hongyan Xia [1].
I will start with thanking both Wei and Hongyan for the initial work to upstream the feature. A lot of patches already went in and this is the last few patches missing to effectively enable the feature.
=== What is the directmap? ===
At the moment, on both arm64 and x86, most of the RAM is mapped in Xen address space. This means that domain memory is easily accessible in Xen.
=== Why do we want to remove the directmap? ===
(Summarizing my understanding of the previous discussion)
Speculation attacks (like Spectre SP1) rely on loading piece of memory in the cache. If the region is not mapped then it can't be loaded.
So removing reducing the amount of memory mapped in Xen will also reduce the surface attack.
=== What's the performance impact? ===
As the guest memory is not always mapped, then the cost of mapping will increase. I haven't done the numbers with this new version, but some measurement were provided in the previous version for x86.
=== Improvement possible ===
The known area to improve on x86 are:
* Mapcache: There was a patch sent by Hongyan:
https://lore.kernel.org/xen-devel/4058e92ce21627731c49b588a95809dc0affd83a.1581015491.git.hongyxia@amazon.com/
* EPT: At the moment an guest page-tabel walk requires about 20 map/unmap. This will have an very high impact on the performance. We need to decide whether keep the EPT always mapped is a problem
The original series didn't have support for Arm64. But as there were some interest, I have provided a PoC.
There are more extra work for Arm64:
* The mapcache is quite simple. We would investigate the performance
* The mapcache should be made compliant to the Arm Arm (this is now more critical).
* We will likely have the same problem as for the EPT.
* We have no support for merging table to a superpage, neither free empty page-tables. (See more below)
=== Implementation ===
The subject is probably a misnomer. The directmap is still present but the RAM is not mapped by default. Instead, the region will still be used to map pages allocate via alloc_xenheap_pages().
The advantage is the solution is simple (so IHMO good enough for been merged as a tech preview). The disadvantage is the page allocator is not trying to keep all the xenheap pages together. So we may end up to have an increase of page table usage.
In the longer term, we should consider to remove the direct map completely and switch to vmap(). The main problem with this approach is it is frequent to use mfn_to_virt() in the code. So we would need to cache the mapping (maybe in the struct page_info).
=== Why arm32 is not covered? ===
On Arm32, the domheap and xenheap is always separated. So by design the guest memory is not mapped by default.
At this stage, it seems unnecessary to have to map/unmap xenheap pages every time they are allocated.
=== Why not using a separate domheap and xenheap? ===
While a separate xenheap/domheap reduce the page-table usage (all xenheap pages are contiguous and could be always mapped), it is also currently less scalable because the split is fixed at boot time (XXX:
Can this be dynamic?).
=== Future of secret-free hypervisor ===
There are some information in an e-mail from Andrew a few years ago:
https://lore.kernel.org/xen-devel/[email protected]/
Cheers,
[1] https://lore.kernel.org/xen-devel/[email protected]/
Alejandro Vallejo (1):
xen/arm32: Hardwire zeroeth_table_offset to 0 on ARM_32
Hongyan Xia (8):
x86: Create per-domain mapping for guest_root_pt
x86/pv: Rewrite how building PV dom0 handles domheap mappings
x86: Add a boot option to enable and disable the direct map x86/domain_page: Remove the fast paths when mfn is not in the
directmap
xen/page_alloc: Add a path for xenheap when there is no direct map x86/setup: Leave early boot slightly earlier xen/page_alloc: vmap heap nodes when they are outside the direct map x86/setup: Do not create valid mappings when directmap=no
Julien Grall (4):
xen/x86: Add support for the PMAP
xen/arm64: mm: Use per-pCPU page-tables xen/arm64: Implement a mapcache for arm64 xen/arm64: Allow the admin to enable/disable the directmap
Wei Liu (2):
x86/pv: Use copy_domain_page() to manage domheap pages during initrd
relocation
x86: Initialize mapcache for PV, HVM, and idle domains
docs/misc/xen-command-line.pandoc | 11 ++++ xen/arch/arm/Kconfig | 3 +- xen/arch/arm/arm32/mmu/mm.c | 1 + xen/arch/arm/arm64/mmu/mm.c | 51 ++++++++++++++- xen/arch/arm/include/asm/arm32/mm.h | 8 --- xen/arch/arm/include/asm/arm64/mm.h | 7 +- xen/arch/arm/include/asm/domain_page.h | 13 ++++ xen/arch/arm/include/asm/lpae.h | 2 +- xen/arch/arm/include/asm/mm.h | 8 +++ xen/arch/arm/include/asm/mmu/layout.h | 13 +++- xen/arch/arm/include/asm/mmu/mm.h | 2 + xen/arch/arm/mmu/domain_page.c | 45 +++++++++++-- xen/arch/arm/mmu/pt.c | 12 ++-- xen/arch/arm/mmu/setup.c | 23 +++---- xen/arch/arm/mmu/smpboot.c | 16 +---- xen/arch/arm/setup.c | 3 + xen/arch/x86/Kconfig | 1 + xen/arch/x86/domain.c | 13 +++- xen/arch/x86/domain_page.c | 80 ++++++++++++++++------ xen/arch/x86/include/asm/config.h | 10 ++- xen/arch/x86/include/asm/domain.h | 15 +++-- xen/arch/x86/include/asm/fixmap.h | 31 +++++++++ xen/arch/x86/include/asm/mm.h | 6 ++ xen/arch/x86/include/asm/pmap.h | 35 ++++++++++ xen/arch/x86/mm.c | 13 ++++ xen/arch/x86/pv/dom0_build.c | 70 ++++++++++++++------ xen/arch/x86/pv/domain.c | 44 +++++++++++-- xen/arch/x86/setup.c | 91 +++++++++++++++++++++++--- xen/arch/x86/spec_ctrl.c | 7 ++ xen/arch/x86/x86_64/asm-offsets.c | 1 + xen/arch/x86/x86_64/entry.S | 9 ++- xen/common/Kconfig | 22 +++++++ xen/common/page_alloc.c | 68 ++++++++++++++++--- xen/include/xen/mm.h | 11 ++++ 34 files changed, 613 insertions(+), 132 deletions(-) create mode 100644 xen/arch/arm/include/asm/domain_page.h create mode 100644 xen/arch/x86/include/asm/pmap.h
Signature · 1 line
2.47.1
PATCH v5 1/15 x86: Create per-domain mapping for guest_root_pt
From: Hongyan Xia <[email protected]>
This patch introduces a per-domain mapping for the guest_root_pt in PV guests as part of the effort to remove the direct map in Xen.
For the time being, the root_pgt is not mapped or unmapped, as it remains
a Xenheap page. This will be addressed in subsequent patches.
---v4->v5:
* bugfix: s/FREE_XENHEAP_PAGE/XFREE/ on destroying root_pt_l1tab.
* Add a few extra comments to the declarations for sanity's sake.
* Refactor pv_root_pt macro to use L1_PAGETABLE_ENTRIES instead.
* Removed extra newline in pv_destory_root_pt_l1tab()
v3->v4:
* Fix over-allocation issue
* Update the mappings when switching from kernel to user-mode
v2->v3:
* Rename SHADOW_ROOT
* Haven't addressed the potentially over-allocation issue as I don't
get it
v1->v2:
* Rework the shadow perdomain mapping solution in the follow-up
patches
Changes since Hongyan's version:
* Remove the final dot in the commit title
--- xen/arch/x86/include/asm/config.h | 10 ++++++- xen/arch/x86/include/asm/domain.h | 3 +++ xen/arch/x86/mm.c | 13 +++++++++ xen/arch/x86/pv/domain.c | 44 ++++++++++++++++++++++++++++--- xen/arch/x86/x86_64/asm-offsets.c | 1 + xen/arch/x86/x86_64/entry.S | 9 ++++++- 6 files changed, 74 insertions(+), 6 deletions(-)
xen/arch/x86/include/asm/config.h +9 -1
@@ -168,7 +168,7 @@/* Slot 260: per-domain mappings (including map cache). */ #define PERDOMAIN_VIRT_START (PML4_ADDR(260))#define PERDOMAIN_SLOT_MBYTES (PML4_ENTRY_BYTES >> (20 + PAGETABLE_ORDER)) -#define PERDOMAIN_SLOTS 3 +#define PERDOMAIN_SLOTS 4#define PERDOMAIN_VIRT_SLOT (s) (PERDOMAIN_VIRT_START + (s) * \ (PERDOMAIN_SLOT_MBYTES << 20))/* Slot 4: mirror of per-domain mappings (for compat xlat area accesses). */
@@ -282,6 +282,14 @@ extern unsigned long xen_phys_start;#define ARG_XLAT_START (v) \ (ARG_XLAT_VIRT_START + ((v)->vcpu_id << ARG_XLAT_VA_SHIFT)) +/* pv_root_pt mapping area. The fourth per-domain-mapping sub-area */ +#define PV_ROOT_PT_MAPPING_VIRT_START PERDOMAIN_VIRT_SLOT(3) +#define PV_ROOT_PT_MAPPING_ENTRIES MAX_VIRT_CPUS + +/* The address of a particular VCPU's PV_ROOT_PT */ +#define PV_ROOT_PT_MAPPING_VCPU_VIRT_START (v) \ + (PV_ROOT_PT_MAPPING_VIRT_START + ((v)->vcpu_id * PAGE_SIZE)) +#define ELFSIZE 64#define ARCH_CRASH_SAVE_VMCOREINFO
xen/arch/x86/include/asm/domain.h +3 -0
@@ -273,6 +273,9 @@ struct pv_domain {l1_pgentry_t **gdt_ldt_l1tab; +/* Array of pointers to the l1 PTs holding PV root PTs of each vCPU */ +l1_pgentry_t **root_pt_l1tab; +atomic_t nr_l4_pages;/* Is a 32-bit PV guest? */
xen/arch/x86/mm.c +13 -0
@@ -527,6 +527,14 @@ void make_cr3(struct vcpu *v, mfn_t mfn) v->arch.cr3 |=get_pcid_bits (v, false); } +/* Index of the l1 PT that maps v's root PT */ +#define pv_root_pt_idx (v) ((v)->vcpu_id / L1_PAGETABLE_ENTRIES) + +/* Pointer to the PTE that maps v's root PT in the perdomain area */ +#define pv_root_pt_pte (v) \ + ((v)->domain->arch.pv.root_pt_l1tab[pv_root_pt_idx(v)] + \ + ((v)->vcpu_id & (L1_PAGETABLE_ENTRIES - 1))) +void write_ptbase (struct vcpu *v) {const struct domain *d = v->domain;
@@ -538,11 +546,16 @@ void write_ptbase(struct vcpu *v)if (is_pv_domain (d) && d->arch.pv.xpti ) { +mfn_t guest_root_pt =_mfn (MASK_EXTR (v->arch.cr3,X86_CR3_ADDR_MASK )); +l1_pgentry_t *pte =pv_root_pt_pte (v); + cpu_info->root_pgt_changed = true; cpu_info->pv_cr3 =__pa (this_cpu (root_pgt));if ( new_cr4 &X86_CR4_PCIDE ) cpu_info->pv_cr3 |=get_pcid_bits (v, true);switch_cr3_cr4 (v->arch.cr3, new_cr4); + +l1e_write (pte,l1e_from_mfn (guest_root_pt, __PAGE_HYPERVISOR_RO)); }else {
xen/arch/x86/pv/domain.c +40 -4
92 lines not on this page · raw message
xen/arch/x86/x86_64/asm-offsets.c +1 -0
@@ -80,6 +80,7 @@ void __dummy__(void)#undef OFFSET_EF + OFFSET(VCPU_id, struct vcpu, vcpu_id); OFFSET(VCPU_processor, struct vcpu, processor); OFFSET(VCPU_domain, struct vcpu, domain); OFFSET(VCPU_vcpu_info, struct vcpu, vcpu_info_area.map);
xen/arch/x86/x86_64/entry.S +8 -1
@@ -170,9 +170,16 @@ FUNC_LOCAL(restore_all_guest) movabs $PADDR_MASK & PAGE_MASK, %rsi movabs $DIRECTMAP_VIRT_START, %rcx and %rsi, %rdi - and %r9, %rsi add %rcx, %rdi + + /* + * The address in the vCPU cr3 is always mapped in the per-domain + * pv_root_pt virt area. + */ + imul $PAGE_SIZE, VCPU_id(%rbx), %esi + movabs $PV_ROOT_PT_MAPPING_VIRT_START, %rcx add %rcx, %rsi + mov $ROOT_PAGETABLE_FIRST_XEN_SLOT, %ecx mov root_table_offset(SH_LINEAR_PT_VIRT_START)*8(%rsi), %r8 mov %r8, root_table_offset(SH_LINEAR_PT_VIRT_START)*8(%rdi)
Signature · 1 line
2.47.1
PATCH v5 2/15 x86/pv: Use copy_domain_page() to manage domheap pages during initrd relocation
In reply to Alejandro VallejoFrom: Wei Liu <[email protected]>
Replace the manual copying logic with a call to copy_domain_page() while relocating intird which map and unmap pages accordingly.
---v4->v5:
* No changes
v3->v4:
* Use the count of pages instead of calculating from order for nr_pages initialization
v2->v3:
* Rename commit title
* Rework the for loop copying the pages
v1->v2:
* Get rid of mfn_to_virt
* Don't open code copy_domain_page()
Changes since Hongyan's version:
* Add missing newline after the variable declaration
--- xen/arch/x86/pv/dom0_build.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
xen/arch/x86/pv/dom0_build.c +8 -1
@@ -637,17 +637,24 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d)if ( d->arch.physaddr_bitsize && ((mfn + count -1 ) >> (d->arch.physaddr_bitsize -PAGE_SHIFT )) ) { +unsigned int nr_pages = count; + order =get_order_from_pages (count); page =alloc_domheap_pages (d, order, MEMF_no_scrub);if ( !page )panic ("Not enough RAM for domain 0 initrd\n" ); +for ( count = -count; order--; )if ( count & (1UL << order) ) {free_domheap_pages (page, order); page +=1UL << order; + nr_pages -=1UL << order; } -memcpy (page_to_virt (page),__va (initrd->start), initrd_len); + +for ( ; nr_pages-- ; page++, mfn++ ) +copy_domain_page (page_to_mfn (page),_mfn (mfn)); +/* * The initrd was copied but the initrd variable is reused in the * calculations below. As to not leak the memory used for the
Signature · 1 line
2.47.1
PATCH v5 3/15 x86/pv: Rewrite how building PV dom0 handles domheap mappings
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
Building a PV dom0 is allocating from the domheap but uses it like the xenheap. Use the pages as they should be.
---v4->v5:
* Bugfix: Don't map l4start with a mapcache and unmap with another. This is a revert to how it originally was in the series. i.e: UNMAP_DOMAIN_PAGE(l4start) before overriding current and then re-mapping on the idle PTs if needed.
* Simplify UNMAP_MAP_AND_ADVANCE removing the do-while(). It's not needed with the ({ x }) expression. Assignments return the assigned value, so the last line was not needed either.
v3->v4:
* Reduce the scope of l{1,2,4}start_mfn variables
* Make the macro UNMAP_MAP_AND_ADVANCE return the new virtual
address
v2->v3:
* Fold following patch 'x86/pv: Map L4 page table for shim domain'
v1->v2:
* Clarify the commit message
* Break the patch in two parts
Changes since Hongyan's version:
* Rebase
* Remove spurious newline
--- xen/arch/x86/pv/dom0_build.c | 61 +++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 19 deletions(-)
xen/arch/x86/pv/dom0_build.c +42 -19
@@ -384,6 +384,8 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d)l3_pgentry_t *l3tab =NULL , *l3start =NULL ;l2_pgentry_t *l2tab =NULL , *l2start =NULL ;l1_pgentry_t *l1tab =NULL , *l1start =NULL ; +mfn_t l3start_mfn =INVALID_MFN ; +mfn_t l4start_mfn =INVALID_MFN ;/* * This fully describes the memory layout of the initial domain. All
@@ -738,22 +740,30 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d) v->arch.pv.event_callback_cs =FLAT_COMPAT_KERNEL_CS ; } +#define UNMAP_MAP_AND_ADVANCE (mfn_var, virt_var, maddr) ({ \ + unmap_domain_page(virt_var); \ + mfn_var = maddr_to_mfn(maddr); \ + maddr += PAGE_SIZE; \ + virt_var = map_domain_page(mfn_var); \ +}) +if ( !compat ) {maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l4_page_table; - l4start = l4tab =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; + l4tab =UNMAP_MAP_AND_ADVANCE (l4start_mfn, l4start, mpt_alloc);clear_page (l4tab); -init_xen_l4_slots (l4tab,_mfn (virt_to_mfn (l4start)), - d,INVALID_MFN , true); - v->arch.guest_table =pagetable_from_paddr (__pa (l4start)); +init_xen_l4_slots (l4tab, l4start_mfn, d,INVALID_MFN , true); + v->arch.guest_table =pagetable_from_mfn (l4start_mfn); }else {/* Monitor table already created by switch_compat(). */ - l4start = l4tab =__va (pagetable_get_paddr (v->arch.guest_table)); + l4start_mfn =pagetable_get_mfn (v->arch.guest_table); + l4start = l4tab =map_domain_page (l4start_mfn);/* See public/xen.h on why the following is needed. */ maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l3_page_table; l3start =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; +UNMAP_MAP_AND_ADVANCE (l3start_mfn, l3start, mpt_alloc); } l4tab +=l4_table_offset (v_start);
@@ -762,15 +772,17 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d) {if ( !((unsigned long )l1tab & (PAGE_SIZE -1 )) ) { +mfn_t l1start_mfn;maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l1_page_table; - l1start = l1tab =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; + l1tab =UNMAP_MAP_AND_ADVANCE (l1start_mfn, l1start, mpt_alloc);clear_page (l1tab);if ( count ==0 ) l1tab +=l1_table_offset (v_start);if ( !((unsigned long )l2tab & (PAGE_SIZE -1 )) ) { +mfn_t l2start_mfn;maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l2_page_table; - l2start = l2tab =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; + l2tab =UNMAP_MAP_AND_ADVANCE (l2start_mfn, l2start, mpt_alloc);clear_page (l2tab);if ( count ==0 ) l2tab +=l2_table_offset (v_start);
@@ -780,19 +792,19 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d) {maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l3_page_table; - l3start =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; +UNMAP_MAP_AND_ADVANCE (l3start_mfn, l3start, mpt_alloc); } l3tab = l3start;clear_page (l3tab);if ( count ==0 ) l3tab +=l3_table_offset (v_start); - *l4tab =l4e_from_paddr (__pa (l3start),L4_PROT ); + *l4tab =l4e_from_mfn (l3start_mfn,L4_PROT ); l4tab++; } - *l3tab =l3e_from_paddr (__pa (l2start),L3_PROT ); + *l3tab =l3e_from_mfn (l2start_mfn,L3_PROT ); l3tab++; } - *l2tab =l2e_from_paddr (__pa (l1start),L2_PROT ); + *l2tab =l2e_from_mfn (l1start_mfn,L2_PROT ); l2tab++; }if ( count < initrd_pfn || count >= initrd_pfn +PFN_UP (initrd_len) )
@@ -811,30 +823,37 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d)if ( compat ) { -l2_pgentry_t *l2t; -/* Ensure the first four L3 entries are all populated. */ for ( i =0 , l3tab = l3start; i <4 ; ++i, ++l3tab ) {if ( !l3e_get_intpte (*l3tab) ) { +mfn_t l2start_mfn;maddr_to_page (mpt_alloc)->u.inuse.type_info = PGT_l2_page_table; - l2tab =__va (mpt_alloc); mpt_alloc +=PAGE_SIZE ; -clear_page (l2tab); - *l3tab =l3e_from_paddr (__pa (l2tab),L3_PROT ); +UNMAP_MAP_AND_ADVANCE (l2start_mfn, l2start, mpt_alloc); +clear_page (l2start); + *l3tab =l3e_from_mfn (l2start_mfn,L3_PROT ); }if ( i ==3 )l3e_get_page (*l3tab)->u.inuse.type_info |= PGT_pae_xen_l2; } - l2t =map_l2t_from_l3e (l3start[3 ]); -init_xen_pae_l2_slots (l2t, d); -unmap_domain_page (l2t); +UNMAP_DOMAIN_PAGE (l2start); + l2start =map_l2t_from_l3e (l3start[3 ]); +init_xen_pae_l2_slots (l2start, d); } +#undef UNMAP_MAP_AND_ADVANCE + +UNMAP_DOMAIN_PAGE (l1start); +UNMAP_DOMAIN_PAGE (l2start); +UNMAP_DOMAIN_PAGE (l3start); +/* Pages that are part of page tables must be read only. */ mark_pv_pt_pages_rdonly (d, l4start, vpt_start, nr_pt_pages, &flush_flags); +UNMAP_DOMAIN_PAGE (l4start); +/* Mask all upcalls... */ for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ )shared_info (d, vcpu_info[i].evtchn_upcall_mask) =1 ;
@@ -1003,8 +1022,12 @@ static int __init dom0_construct(struct boot_info *bi, struct domain *d)* !CONFIG_VIDEO case so the logic here can be simplified. */ if ( pv_shim ) + { + l4start =map_domain_page (l4start_mfn);pv_shim_setup_dom (d, l4start, v_start, vxenstore_start, vconsole_start, vphysmap_start, si); +UNMAP_DOMAIN_PAGE (l4start); + }#ifdef CONFIG_COMPAT if ( compat )
Signature · 1 line
2.47.1
PATCH v5 4/15 x86: Initialize mapcache for PV, HVM, and idle domains
In reply to Alejandro VallejoFrom: Wei Liu <[email protected]>
To support the transition away from the direct map, the mapcache will now be used by HVM and idle domains as well. This patch lifts the mapcache to the arch level and moves its initialization to arch_domain_create().
For the idle domain to utilize the mapcache, this patch also populates the mapcache page tables within the PERDOMAIN region and adjusts the initialization sequence in arch_idle_init(), as it's no longer covered by arch_domain_create()
With this change, mapcache initialization is now unified across all domain types—PV, HVM, and idle.
---v4->v5:
* Move mapcache initialization and cleanup back to arch-specific code and reword commit message to reflect it. Since v3, the idle domain gained its own arch-init function, so initialise the mapcache there instead. Panic on failure to initialise it for the idle domain, as there's no possible recovery.
v2->v4:
* Reword the commit message
* Rebase it on top of staging
* The logic for the creation of the domain has been reworked so introduced #ifdef CONFIG_X86 in the common code to initialise the mapcache
v1->v2:
* Free resources if mapcache initialisation fails
* Remove is_idle_domain() check from create_perdomain_mappings()
--- xen/arch/x86/domain.c | 13 ++++++++++--- xen/arch/x86/domain_page.c | 22 ++++++++++------------ xen/arch/x86/include/asm/domain.h | 12 ++++++------ 3 files changed, 26 insertions(+), 21 deletions(-)
xen/arch/x86/domain.c +10 -3
@@ -777,6 +777,12 @@ void __init arch_init_idle_domain(struct domain *d) }; d->arch.ctxt_switch = &idle_csw; + +BUG_ON (mapcache_domain_init (d)); + +/* Slot 260: Per-domain mappings. */ + idle_pg_table[l4_table_offset (PERDOMAIN_VIRT_START )] = +l4e_from_page (d->arch.perdomain_l3_pg, __PAGE_HYPERVISOR_RW); } int arch_domain_create(struct domain *d,
@@ -832,6 +838,10 @@ int arch_domain_create(struct domain *d,spec_ctrl_init_domain (d); + rc =mapcache_domain_init (d); +if ( rc ) + goto fail; +if ( (rc =paging_domain_init (d)) !=0 ) goto fail; paging_initialised = true;
@@ -870,9 +880,6 @@ int arch_domain_create(struct domain *d, }else if (is_pv_domain (d) ) { -if ( (rc =mapcache_domain_init (d)) !=0 ) - goto fail; -if ( (rc =pv_domain_initialise (d)) !=0 ) goto fail; }
xen/arch/x86/domain_page.c +10 -12
@@ -82,11 +82,11 @@ void *map_domain_page(mfn_t mfn)#endif v =mapcache_current_vcpu (); -if ( !v || !is_pv_vcpu (v) ) +if ( !v )return mfn_to_virt (mfn_x (mfn)); - dcache = &v->domain->arch.pv.mapcache; - vcache = &v->arch.pv.mapcache; + dcache = &v->domain->arch.mapcache; + vcache = &v->arch.mapcache;if ( !dcache->inuse )return mfn_to_virt (mfn_x (mfn));
@@ -187,14 +187,14 @@ void unmap_domain_page(const void *ptr)ASSERT (va >=MAPCACHE_VIRT_START && va <MAPCACHE_VIRT_END ); v =mapcache_current_vcpu (); -ASSERT (v &&is_pv_vcpu (v)); +ASSERT (v); - dcache = &v->domain->arch.pv.mapcache; + dcache = &v->domain->arch.mapcache;ASSERT (dcache->inuse); idx =PFN_DOWN (va -MAPCACHE_VIRT_START ); mfn =l1e_get_pfn (MAPCACHE_L1ENT (idx)); - hashent = &v->arch.pv.mapcache.hash[MAPHASH_HASHFN (mfn)]; + hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN (mfn)];local_irq_save (flags);
@@ -233,11 +233,9 @@ void unmap_domain_page(const void *ptr)int mapcache_domain_init (struct domain *d) { -struct mapcache_domain *dcache = &d->arch.pv.mapcache; +struct mapcache_domain *dcache = &d->arch.mapcache;unsigned int bitmap_pages; -ASSERT (is_pv_domain (d)); -#ifdef NDEBUG if ( !mem_hotplug && max_page <=PFN_DOWN (__pa (HYPERVISOR_VIRT_END -1 )) )return 0 ;
@@ -261,12 +259,12 @@ int mapcache_domain_init(struct domain *d)int mapcache_vcpu_init (struct vcpu *v) {struct domain *d = v->domain; -struct mapcache_domain *dcache = &d->arch.pv.mapcache; +struct mapcache_domain *dcache = &d->arch.mapcache;unsigned long i;unsigned int ents = d->max_vcpus *MAPCACHE_VCPU_ENTRIES ;unsigned int nr =PFN_UP (BITS_TO_LONGS (ents) *sizeof (long )); -if ( !is_pv_vcpu (v) || !dcache->inuse ) +if ( !dcache->inuse )return 0 ;if ( ents > dcache->entries )
@@ -293,7 +291,7 @@ int mapcache_vcpu_init(struct vcpu *v)BUILD_BUG_ON (MAPHASHENT_NOTINUSE <MAPCACHE_ENTRIES );for ( i =0 ; i <MAPHASH_ENTRIES ; i++ ) { -struct vcpu_maphash_entry *hashent = &v->arch.pv.mapcache.hash[i]; +struct vcpu_maphash_entry *hashent = &v->arch.mapcache.hash[i]; hashent->mfn = ~0UL ;/* never valid to map */ hashent->idx =MAPHASHENT_NOTINUSE ;
xen/arch/x86/include/asm/domain.h +6 -6
@@ -287,9 +287,6 @@ struct pv_domain/* Mitigate L1TF with shadow/crashing? */ bool check_l1tf; -/* map_domain_page() mapping cache. */ -struct mapcache_domain mapcache; -struct cpuidmasks *cpuidmasks; };
@@ -328,6 +325,9 @@ struct arch_domainuint8_t scf;/* See SCF_DOM_MASK */ +/* map_domain_page() mapping cache. */ +struct mapcache_domain mapcache; +union {struct pv_domain pv;struct hvm_domain hvm;
@@ -518,9 +518,6 @@ struct arch_domainstruct pv_vcpu { -/* map_domain_page() mapping cache. */ -struct mapcache_vcpu mapcache; -unsigned int vgc_flags;struct trap_info *trap_ctxt;
@@ -614,6 +611,9 @@ struct arch_vcpu#define async_exception_state (t) async_exception_state[(t)-1]uint8_t async_exception_mask; +/* map_domain_page() mapping cache. */ +struct mapcache_vcpu mapcache; +/* Virtual Machine Extensions */ union {struct pv_vcpu pv;
Signature · 1 line
2.47.1
PATCH v5 5/15 x86: Add a boot option to enable and disable the direct map
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
This is added as a Kconfig option as well as a boot command line option. While being generic, the Kconfig option is only usable for x86 at the moment.
Note that there remains some users of the directmap at this point. The option is introduced now as it will be needed in follow-up patches.
---
There's a major change compared with v4. directmap= turned into asi= for compatibility with Roger's ASI series. In particular, after everything is done we expect the commandline to have somewhat different effects per arch depending on the extent of ASI implemented. e.g: x86 might implement distinct toggles for PV and HVM, which are nonsensical for anything but x86.
With the intent of parsing the commandline differently I've modified this patch to parse it on arch-specific code from the get-go. I also adjusted the commandline text to make it a lot more similar to the final text expected on Roger's series.
v4->v5:
* Moved cmdline parsing to arch-specific code and turned directmap= to asi= (where asi == !directmap). This is for compatibility with Roger's work, which appends extra suboptions to asi= on x86.
* Moved the printk() highlighting the sparse directmap to spec-ctrl with the rest of speculative mitigations. Further additions to asi= will interact with XPTI, so it makes sense to have it all there.
* Moved CONFIG_ONDEMAND_DIRECTMAP to the speculation hardening section of Kconfig.
* Reworded the help message to align it with other rows in that section and make it more ameanable for end users.
* Simplified #ifdef hunk defining has_directmap()
* Used #ifdef CONFIG_ONDEMAND_DIRECTMAP rather than CONFIG_HAS_ONDEMAND_DIRECTMAP. The former selects the feature, while the latter signals support for it on the arch.
v3->v4:
* Rename the Kconfig options
* Set opt_directmap to true if CONFIG_HAS_ONDEMAND_DIRECTMAP is not
enabled
v2->v3:
* No changes.
v1->v2:
* Introduce a Kconfig option
* Reword the commit message
* Make opt_directmap and helper generic
Changes since Hongyan's version:
* Reword the commit message
* opt_directmap is only modified during boot so mark it as
__ro_after_init --- docs/misc/xen-command-line.pandoc | 11 +++++++++++ xen/arch/x86/Kconfig | 1 + xen/arch/x86/include/asm/mm.h | 6 ++++++ xen/arch/x86/spec_ctrl.c | 7 +++++++ xen/common/Kconfig | 21 +++++++++++++++++++++ xen/include/xen/mm.h | 11 +++++++++++ 6 files changed, 57 insertions(+)
docs/misc/xen-command-line.pandoc +11 -0
@@ -202,6 +202,17 @@ to appropriate auditing by Xen. Argo is disabled by default. This option is disabled by default, to protect domains from a DoS by a buggy or malicious other domain spamming the ring. +### asi (x86) +> `= <boolean>` + +> Default: `false` + +Offers control over whether the hypervisor will engage in Address Space +Isolation, by not having potentially sensitive information permanently mapped +in the directmap. Enabling this option populates the directmap sparsely on +demand, blocking exploits that leak secrets via speculative memory access in +the directmap. + ### asid (x86) > `= <boolean>`
xen/arch/x86/Kconfig +1 -0
@@ -23,6 +23,7 @@ config X86 select HAS_IOPORTS select HAS_KEXEC select HAS_NS16550 + select HAS_ONDEMAND_DIRECTMAP select HAS_PASSTHROUGH select HAS_PCI select HAS_PCI_MSI
xen/arch/x86/include/asm/mm.h +6 -0
@@ -643,11 +643,17 @@ void write_32bit_pse_identmap(uint32_t *l2);/* * x86 maps part of physical memory via the directmap region. * Return whether the range of MFN falls in the directmap region. +* +* Enabling ASI on the commandline (i.e: using the `asi=` option) causes the +* directmap to be mostly empty, so this always returns false in that case. */ static inline bool arch_mfns_in_directmap (unsigned long mfn,unsigned long nr) {unsigned long eva =min (DIRECTMAP_VIRT_END ,HYPERVISOR_VIRT_END ); +if ( !has_directmap () ) +return false; +return (mfn + nr) <= (virt_to_mfn (eva -1 ) +1 ); }
xen/arch/x86/spec_ctrl.c +7 -0
@@ -85,6 +85,11 @@ static int8_t __initdata opt_gds_mit = -1;static int8_t __initdata opt_div_scrub =-1 ;bool __ro_after_init opt_bp_spec_reduce = true; +#ifdef CONFIG_ONDEMAND_DIRECTMAP +bool __ro_after_init opt_ondemand_dmap; +boolean_param ("asi" , opt_ondemand_dmap); +#endif + static int __init cf_check parse_spec_ctrl(const char *s) {const char *ss;
@@ -633,6 +638,8 @@ static void __init print_details(enum ind_thunk thunk) cpu_has_bug_l1tf ?"" :" not" , l1d_maxphysaddr, paddr_bits, l1tf_safe_maddr); +printk (" ASI: %s" , !has_directmap () ?"enabled" :"disabled" ); +/* * Alternatives blocks for protecting against and/or virtualising * mitigation support for guests.
xen/common/Kconfig +21 -0
@@ -74,6 +74,9 @@ config HAS_KEXEC config HAS_LLC_COLORING bool +config HAS_ONDEMAND_DIRECTMAP + bool + config HAS_PIRQ bool
@@ -214,6 +217,24 @@ config SPECULATIVE_HARDEN_LOCK If unsure, say Y. +config ONDEMAND_DIRECTMAP + bool "On-Demand Directmap" + depends on HAS_ONDEMAND_DIRECTMAP + help + Contemporary processors may use speculative execution as a + performance optimisation, but this can potentially be abused by an + attacker to leak data via speculative sidechannels. + + When enabled, this option provides defense in depth by preventing + most RAM from being constantly mapped on the hypervisor, thereby + greatly reducing the scope of data leaks after a successful + speculative attack. + + This option is disabled by default at run time, and needs to be + enabled on the command line. + + If unsure, say N. + endmenu config DIT_DEFAULT
xen/include/xen/mm.h +11 -0
@@ -169,6 +169,17 @@ extern unsigned long max_page;extern unsigned long total_pages;extern paddr_t mem_hotplug; +#ifdef CONFIG_ONDEMAND_DIRECTMAP +extern bool opt_ondemand_dmap; + +static inline bool has_directmap (void ) +{ +return !opt_ondemand_dmap; +} +#else +#define has_directmap () true +#endif +/* * Extra fault info types which are used to further describe * the source of an access violation.
Signature · 1 line
2.47.1
PATCH v5 6/15 xen/x86: Add support for the PMAP
In reply to Alejandro VallejoFrom: Julien Grall <[email protected]>
PMAP will be used in a follow-up patch to bootstrap map domain page infrastructure -- we need some way to map pages to setup the mapcache without a direct map.
The functions pmap_{map, unmap} open code {set, clear}_fixmap to break the loop.
---v4->v5:
* No changes.
v3->v4:
* Select PMAP KConfig option iff ONDEMAND_DIRECTMAP is used
v2->v3:
* No changes.
v1->v2:
* Declare PMAP entries earlier in fixed_addresses
* Reword the commit message
The PMAP infrastructure was upstream separately for Arm since Hongyan sent the secret-free hypervisor series. So this is a new patch to plumb the feature on x86.
--- xen/arch/x86/include/asm/fixmap.h | 6 ++++++ xen/arch/x86/include/asm/pmap.h | 35 +++++++++++++++++++++++++++++++ xen/common/Kconfig | 1 + 3 files changed, 42 insertions(+) create mode 100644 xen/arch/x86/include/asm/pmap.h
xen/arch/x86/include/asm/fixmap.h +6 -0
@@ -21,6 +21,8 @@#include <xen/acpi.h> #include <xen/pfn.h> +#include <xen/pmap.h> +#include <asm/apicdef.h> #include <asm/msi.h> #include <acpi/apei.h>
@@ -53,6 +55,10 @@ enum fixed_addresses { FIX_PV_CONSOLE, FIX_XEN_SHARED_INFO,#endif /* CONFIG_XEN_GUEST */ +#ifdef CONFIG_HAS_PMAP +FIX_PMAP_BEGIN , +FIX_PMAP_END =FIX_PMAP_BEGIN +NUM_FIX_PMAP , +#endif/* Everything else should go further down. */ FIX_APIC_BASE , FIX_IO_APIC_BASE_0,
xen/arch/x86/include/asm/pmap.h new file +35 -0
@@ -0,0 +1,35 @@ +#ifndef __ASM_PMAP_H__ +#define __ASM_PMAP_H__ + +#include <asm/fixmap.h> + +static inline void arch_pmap_map (unsigned int slot,mfn_t mfn) +{ +unsigned long linear = (unsigned long )fix_to_virt (slot); +l1_pgentry_t *pl1e = &l1_fixmap[l1_table_offset (linear)]; + +BUILD_BUG_ON (FIX_APIC_BASE -1 >L1_PAGETABLE_ENTRIES -1 ); +ASSERT (!(l1e_get_flags (*pl1e) & _PAGE_PRESENT)); + +l1e_write (pl1e,l1e_from_mfn (mfn,PAGE_HYPERVISOR )); +} + +static inline void arch_pmap_unmap (unsigned int slot) +{ +unsigned long linear = (unsigned long )fix_to_virt (slot); +l1_pgentry_t *pl1e = &l1_fixmap[l1_table_offset (linear)]; + +l1e_write (pl1e,l1e_empty ()); +flush_tlb_one_local (linear); +} + +#endif /* __ASM_PMAP_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/
xen/common/Kconfig +1 -0
@@ -220,6 +220,7 @@ config SPECULATIVE_HARDEN_LOCK config ONDEMAND_DIRECTMAP bool "On-Demand Directmap" depends on HAS_ONDEMAND_DIRECTMAP + select HAS_PMAP help Contemporary processors may use speculative execution as a performance optimisation, but this can potentially be abused by an
Signature · 1 line
2.47.1
PATCH v5 7/15 x86/domain_page: Remove the fast paths when mfn is not in the directmap
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
When mfn is not in direct map, never use mfn_to_virt for any mappings.
We replace mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) with arch_mfns_in_direct_map(mfn, 1) because these two are equivalent. The extra comparison in arch_mfns_in_direct_map() looks different but becauseDIRECTMAP_VIRT_END is always higher, it does not make any difference.
Lastly, domain_page_map_to_mfn() needs to gain to a special case for the PMAP.
---v4->v5:
* s/BUILD_BUG_ON/ASSERT/, or it won't build with gcc11.
* Add CONFIG_HAS_PMAP guards as needed so the code builds without PMAP
* Fix typos on 2 arch_mfns_in_directmap() calls in release config.
v3->v4:
* Introduce helper functions virt_is_fixmap and virt_in_fixmap_range
Changes since Hongyan's version:
* arch_mfn_in_direct_map() was renamed to arch_mfns_in_directmap()
* add a special case for the PMAP in domain_page_map_to_mfn()
--- xen/arch/x86/domain_page.c | 60 +++++++++++++++++++++++++------ xen/arch/x86/include/asm/fixmap.h | 25 +++++++++++++ 2 files changed, 75 insertions(+), 10 deletions(-)
xen/arch/x86/domain_page.c +50 -10
@@ -14,8 +14,10 @@#include <xen/sched.h> #include <xen/vmap.h> #include <asm/current.h> +#include <asm/fixmap.h> #include <asm/flushtlb.h> #include <asm/hardirq.h> +#include <asm/pmap.h> #include <asm/setup.h> static DEFINE_PER_CPU(struct vcpu *, override);
@@ -24,6 +26,7 @@ static inline struct vcpu *mapcache_current_vcpu(void) {/* In the common case we use the mapcache of the running VCPU. */ struct vcpu *v =this_cpu (override) ?: current; +struct vcpu *idle_v = idle_vcpu[smp_processor_id ()];/* * When current isn't properly set up yet, this is equivalent to
@@ -35,10 +38,11 @@ static inline struct vcpu *mapcache_current_vcpu(void)/* * When using efi runtime page tables, we have the equivalent of the idle * domain's page tables but current may point at another domain's VCPU. -* Return NULL as though current is not properly set up yet. +* Return the idle domains's vcpu on that core because the efi per-domain +* region (where the mapcache is) is in-sync with the idle domain. */ if (efi_rs_using_pgtables () ) -return NULL ; +return idle_v;/* * If guest_table is NULL, and we are running a paravirtualised guest,
@@ -48,7 +52,7 @@ static inline struct vcpu *mapcache_current_vcpu(void)if (unlikely (pagetable_is_null (v->arch.guest_table)) &&is_pv_vcpu (v) ) {/* If we really are idling, perform lazy context switch now. */ -if ( (v = idle_vcpu[smp_processor_id ()]) == current ) +if ( (v = idle_v) == current )sync_local_execstate ();/* We must now be running on the idle page table. */ ASSERT (cr3_pa (read_cr3 ()) ==__pa (idle_pg_table));
@@ -77,18 +81,28 @@ void *map_domain_page(mfn_t mfn)struct vcpu_maphash_entry *hashent;#ifdef NDEBUG -if (mfn_x (mfn) <=PFN_DOWN (__pa (HYPERVISOR_VIRT_END -1 )) ) +if (arch_mfns_in_directmap (mfn_x (mfn),1 ) )return mfn_to_virt (mfn_x (mfn));#endif v =mapcache_current_vcpu (); -if ( !v ) -return mfn_to_virt (mfn_x (mfn)); +if ( !v || !v->domain->arch.mapcache.inuse ) + { +if (arch_mfns_in_directmap (mfn_x (mfn),1 ) ) +return mfn_to_virt (mfn_x (mfn)); +else + { +#ifdef CONFIG_HAS_PMAP +BUG_ON (system_state >= SYS_STATE_smp_boot); +return pmap_map (mfn); +#else +BUG (); +#endif + } + } dcache = &v->domain->arch.mapcache; vcache = &v->arch.mapcache; -if ( !dcache->inuse ) -return mfn_to_virt (mfn_x (mfn));perfc_incr (map_domain_page_count);
@@ -184,6 +198,14 @@ void unmap_domain_page(const void *ptr)if ( !va || va >=DIRECTMAP_VIRT_START )return ; +#ifdef CONFIG_HAS_PMAP +if (virt_is_fixmap (va) ) + { +pmap_unmap (ptr); +return ; + } +#endif +ASSERT (va >=MAPCACHE_VIRT_START && va <MAPCACHE_VIRT_END ); v =mapcache_current_vcpu ();
@@ -237,7 +259,7 @@ int mapcache_domain_init(struct domain *d)unsigned int bitmap_pages;#ifdef NDEBUG -if ( !mem_hotplug && max_page <=PFN_DOWN (__pa (HYPERVISOR_VIRT_END -1 )) ) +if ( !mem_hotplug &&arch_mfns_in_directmap (0 , max_page) )return 0 ;#endif
@@ -308,7 +330,7 @@ void *map_domain_page_global(mfn_t mfn) local_irq_is_enabled()));#ifdef NDEBUG -if (mfn_x (mfn) <=PFN_DOWN (__pa (HYPERVISOR_VIRT_END -1 )) ) +if (arch_mfns_in_directmap (mfn_x (mfn),1 ) )return mfn_to_virt (mfn_x (mfn));#endif
@@ -335,6 +357,24 @@ mfn_t domain_page_map_to_mfn(const void *ptr)if ( va >=DIRECTMAP_VIRT_START )return _mfn (virt_to_mfn (ptr)); +#ifdef CONFIG_HAS_PMAP +/* +* The fixmap is stealing the top-end of the VMAP. So the check for +* the PMAP *must* happen first. +* +* Also, the fixmap translate a slot to an address backwards. The +* logic will rely on it to avoid any complexity. So check at +* compile time this will always hold. +*/ +ASSERT (fix_to_virt (FIX_PMAP_BEGIN ) >fix_to_virt (FIX_PMAP_END )); + +if (virt_in_fixmap_range (va,FIX_PMAP_BEGIN ,FIX_PMAP_END ) ) + { +BUG_ON (system_state >= SYS_STATE_smp_boot); +return l1e_get_mfn (l1_fixmap[l1_table_offset (va)]); + } +#endif /* CONFIG_HAS_PMAP */ +if ( va >=VMAP_VIRT_START && va <VMAP_VIRT_END )return vmap_to_mfn (va);
xen/arch/x86/include/asm/fixmap.h +25 -0
@@ -101,6 +101,31 @@ static inline unsigned long virt_to_fix(const unsigned long vaddr)return __virt_to_fix (vaddr); } +static inline bool virt_is_fixmap (const unsigned long vaddr) +{ +return vaddr >=FIXADDR_START && vaddr <FIXADDR_TOP ; +} + +static inline bool virt_in_fixmap_range ( +const unsigned long vaddr, +const unsigned int start_idx, +const unsigned int end_idx +) +{ +unsigned long start_addr = (unsigned long )fix_to_virt (start_idx); +unsigned long end_addr = (unsigned long )fix_to_virt (end_idx); + +/* +* The check ensures that the virtual address (vaddr) is within the +* fixmap range. The addresses are allocated backwards, meaning the +* start address is higher than the end address. As a result, the +* check ensures that the virtual address is greater than or equal to +* the end address, and less than or equal to the start address, which +* may appear counterintuitive due to the reverse allocation order. +*/ +return ((vaddr &PAGE_MASK ) <= start_addr) && (vaddr >= end_addr); +} +enum fixed_addresses_x {/* Index 0 is reserved since fix_x_to_virt(0) == FIXADDR_X_TOP. */ FIX_X_RESERVED ,
Signature · 1 line
2.47.1
PATCH v5 8/15 xen/page_alloc: Add a path for xenheap when there is no direct map
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
When there is not an always-mapped direct map, xenheap allocations need to be mapped and unmapped on-demand.
---v4->v5:
* Remove stray comma in printk() after XENLOG_WARNING.
Elias @ v4:
I have left the call to map_pages_to_xen() and destroy_xen_mappings() in the split heap for now. I am not entirely convinced this is
necessary
because in that setup only the xenheap would be always mapped and this doesn't contain any guest memory (aside the grant-table). So map/unmapping for every allocation seems unnecessary.
v3->v4:
* Call printk instead of dprintk()
v1->v2:
* Fix remaining wrong indentation in alloc_xenheap_pages()
Changes since Hongyan's version:
* Rebase
* Fix indentation in alloc_xenheap_pages()
* Fix build for arm32
--- xen/common/page_alloc.c | 43 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
xen/common/page_alloc.c +41 -2
@@ -2435,6 +2435,7 @@ void init_xenheap_pages(paddr_t ps, paddr_t pe)void *alloc_xenheap_pages (unsigned int order,unsigned int memflags) {struct page_info *pg; +void *virt_addr;ASSERT_ALLOC_CONTEXT ();
@@ -2443,17 +2444,36 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)if (unlikely (pg ==NULL ) )return NULL ; -return page_to_virt (pg); + virt_addr =page_to_virt (pg); + +if ( !has_directmap () && +map_pages_to_xen ((unsigned long )virt_addr,page_to_mfn (pg),1UL << order, +PAGE_HYPERVISOR ) ) + { +/* Failed to map xenheap pages. */ +free_heap_pages (pg, order, false); +return NULL ; + } + +return virt_addr; }void free_xenheap_pages (void *v,unsigned int order) { +unsigned long va = (unsigned long )v &PAGE_MASK ; +ASSERT_ALLOC_CONTEXT ();if ( v ==NULL )return ; +if ( !has_directmap () && +destroy_xen_mappings (va, va + (PAGE_SIZE << order)) ) +printk (XENLOG_WARNING +"Error while destroying xenheap mappings at %p, order %u\n" , + v, order); +free_heap_pages (virt_to_page (v), order, false); }
@@ -2477,6 +2497,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) {struct page_info *pg;unsigned int i; +void *virt_addr;ASSERT_ALLOC_CONTEXT ();
@@ -2489,16 +2510,28 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)if (unlikely (pg ==NULL ) )return NULL ; + virt_addr =page_to_virt (pg); + +if ( !has_directmap () && +map_pages_to_xen ((unsigned long )virt_addr,page_to_mfn (pg),1UL << order, +PAGE_HYPERVISOR ) ) + { +/* Failed to map xenheap pages. */ +free_domheap_pages (pg, order); +return NULL ; + } +for ( i =0 ; i < (1u << order); i++ ) pg[i].count_info |= PGC_xen_heap; -return page_to_virt (pg); +return virt_addr; }void free_xenheap_pages (void *v,unsigned int order) {struct page_info *pg;unsigned int i; +unsigned long va = (unsigned long )v &PAGE_MASK ;ASSERT_ALLOC_CONTEXT ();
@@ -2510,6 +2543,12 @@ void free_xenheap_pages(void *v, unsigned int order)for ( i =0 ; i < (1u << order); i++ ) pg[i].count_info &= ~PGC_xen_heap; +if ( !has_directmap () && +destroy_xen_mappings (va, va + (PAGE_SIZE << order)) ) +printk (XENLOG_WARNING +"Error while destroying xenheap mappings at %p, order %u\n" , + v, order); +free_heap_pages (pg, order, true); }
Signature · 1 line
2.47.1
PATCH v5 9/15 x86/setup: Leave early boot slightly earlier
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
When we do not have a direct map, memory for metadata of heap nodes in init_node_heap() is allocated from xenheap, which needs to be mapped and unmapped on demand. However, we cannot just take memory from the boot allocator to create the PTEs while we are passing memory to the heap allocator.
To solve this race, we leave early boot slightly sooner so that Xen PTE pages are allocated from the heap instead of the boot allocator. We can do this because the metadata for the 1st node is statically allocated, and by the time we need memory to create mappings for the 2nd node, we already have enough memory in the heap allocator in the 1st node.
---v4->v5:
* No changes.
v3->v4:
* Fix indentation
* Refactor the code to reduce code duplication
--- xen/arch/x86/setup.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
xen/arch/x86/setup.c +16 -2
@@ -1831,6 +1831,22 @@ void asmlinkage __init noreturn __start_xen(void)numa_initmem_init (0 , raw_max_page); +/* +* When we do not have a direct map, memory for metadata of heap nodes in +* init_node_heap() is allocated from xenheap, which needs to be mapped and +* unmapped on demand. However, we cannot just take memory from the boot +* allocator to create the PTEs while we are passing memory to the heap +* allocator during end_boot_allocator(). +* +* To solve this race, we need to leave early boot before +* end_boot_allocator() so that Xen PTE pages are allocated from the heap +* instead of the boot allocator. We can do this because the metadata for +* the 1st node is statically allocated, and by the time we need memory to +* create mappings for the 2nd node, we already have enough memory in the +* heap allocator in the 1st node. +*/ + system_state = SYS_STATE_boot; +if ( max_page -1 >virt_to_mfn (HYPERVISOR_VIRT_END -1 ) ) {unsigned long lo =virt_to_mfn (HYPERVISOR_VIRT_END -1 );
@@ -1862,8 +1878,6 @@ void asmlinkage __init noreturn __start_xen(void)else end_boot_allocator (); - system_state = SYS_STATE_boot; - bsp_stack =cpu_alloc_stack (0 );if ( !bsp_stack )panic ("No memory for BSP stack\n" );
Signature · 1 line
2.47.1
PATCH v5 10/15 xen/page_alloc: vmap heap nodes when they are outside the direct map
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
When we do not have a direct map, archs_mfn_in_direct_map() will always return false, thus init_node_heap() will allocate xenheap pages from an existing node for the metadata of a new node. This means that the metadata of a new node is in a different node, slowing down heap allocation.
Since we now have early vmap, vmap the metadata locally in the new node.
---v4->v5:
* Fix bug introduced in v4 by which node metadata would be unconditionally mapped at the tail of the heap node.
* Remove extra space in conditional
v3->v4:
* Change type of the parameters to paddr_t
* Use clear_domain_page() instead of open-coding it
v1->v2:
* vmap_contig_pages() was renamed to vmap_contig()
* Fix indentation and coding style
Changes from Hongyan's version:
* arch_mfn_in_direct_map() was renamed to arch_mfns_in_direct_map()
* Use vmap_contig_pages() rather than __vmap(...).
* Add missing include (xen/vmap.h) so it compiles on Arm
--- xen/common/page_alloc.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
xen/common/page_alloc.c +17 -8
@@ -139,6 +139,7 @@#include <xen/softirq.h> #include <xen/spinlock.h> #include <xen/vm_event.h> +#include <xen/vmap.h> #include <xen/xvmalloc.h> #include <asm/flushtlb.h>
@@ -615,22 +616,30 @@ static unsigned long init_node_heap(int node, unsigned long mfn, needed =0 ; }else if ( *use_tail && nr >= needed && -arch_mfns_in_directmap (mfn + nr - needed, needed) && (!xenheap_bits || !((mfn + nr -1 ) >> (xenheap_bits -PAGE_SHIFT ))) ) { - _heap[node] =mfn_to_virt (mfn + nr - needed); - avail[node] =mfn_to_virt (mfn + nr -1 ) + -PAGE_SIZE -sizeof (**avail) *NR_ZONES ; +if (arch_mfns_in_directmap (mfn + nr - needed, needed) ) + _heap[node] =mfn_to_virt (mfn + nr - needed); +else + _heap[node] =vmap_contig (_mfn (mfn + nr - needed), needed); + +BUG_ON (!_heap[node]); + avail[node] = (void *)(_heap[node]) + (needed <<PAGE_SHIFT ) - +sizeof (**avail) *NR_ZONES ; }else if ( nr >= needed && -arch_mfns_in_directmap (mfn, needed) && (!xenheap_bits || !((mfn + needed -1 ) >> (xenheap_bits -PAGE_SHIFT ))) ) { - _heap[node] =mfn_to_virt (mfn); - avail[node] =mfn_to_virt (mfn + needed -1 ) + -PAGE_SIZE -sizeof (**avail) *NR_ZONES ; +if (arch_mfns_in_directmap (mfn, needed) ) + _heap[node] =mfn_to_virt (mfn); +else + _heap[node] =vmap_contig (_mfn (mfn), needed); + +BUG_ON (!_heap[node]); + avail[node] = (void *)(_heap[node]) + (needed <<PAGE_SHIFT ) - +sizeof (**avail) *NR_ZONES ; *use_tail = false; } else if ( get_order_from_bytes(sizeof(**_heap)) ==
Signature · 1 line
2.47.1
PATCH v5 11/15 x86/setup: Do not create valid mappings when directmap=no
In reply to Alejandro VallejoFrom: Hongyan Xia <[email protected]>
Create empty mappings in the second e820 pass. Also, destroy existing direct map mappings created in the first pass.
To make xenheap pages visible in guests, it is necessary to create empty L3 tables in the direct map even when directmap=no, since guest cr3s copy idle domain's L4 entries, which means they will share mappings in the direct map if we pre-populate idle domain's L4 entries and L3 tables. A helper is introduced for this.
Also, after the direct map is actually gone, we need to stop updating the direct map in update_xen_mappings().
---v4->v5:
* No changes.
--- xen/arch/x86/setup.c | 73 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 66 insertions(+), 7 deletions(-)
xen/arch/x86/setup.c +66 -7
@@ -1060,6 +1060,56 @@ static struct domain *__init create_dom0(struct boot_info *bi)return d; } +/* +* This either populates a valid direct map, or allocates empty L3 tables and +* creates the L4 entries for virtual address between [start, end) in the +* direct map depending on has_directmap(); +* +* When directmap=no, we still need to populate empty L3 tables in the +* direct map region. The reason is that on-demand xenheap mappings are +* created in the idle domain's page table but must be seen by +* everyone. Since all domains share the direct map L4 entries, they +* will share xenheap mappings if we pre-populate the L4 entries and L3 +* tables in the direct map region for all RAM. We also rely on the fact +* that L3 tables are never freed. +*/ +static void __init populate_directmap (paddr_t pstart,paddr_t pend, +unsigned int flags) +{ +unsigned long vstart = (unsigned long )__va (pstart); +unsigned long vend = (unsigned long )__va (pend); + +if ( pstart >= pend ) +return ; + +BUG_ON (vstart <DIRECTMAP_VIRT_START ); +BUG_ON (vend >DIRECTMAP_VIRT_END ); + +if (has_directmap () ) +/* Populate valid direct map. */ +BUG_ON (map_pages_to_xen (vstart,maddr_to_mfn (pstart), +PFN_DOWN (pend - pstart), flags)); +else + { +/* Create empty L3 tables. */ +unsigned long vaddr = vstart & ~((1UL <<L4_PAGETABLE_SHIFT ) -1 ); + +for (unsigned long idx =l4_table_offset (vaddr); + idx <=l4_table_offset (vend); idx++ ) + { +l4_pgentry_t *pl4e = &idle_pg_table[l4_table_offset (idx)]; + +if ( !(l4e_get_flags (*pl4e) & _PAGE_PRESENT) ) + { +mfn_t mfn =alloc_boot_pages (1 ,1 ); + +clear_domain_page (mfn); +l4e_write (pl4e,l4e_from_mfn (mfn, __PAGE_HYPERVISOR)); + } + } + } +} +/* How much of the directmap is prebuilt at compile time. */ #define PREBUILT_MAP_LIMIT (1 << L2_PAGETABLE_SHIFT)
@@ -1681,8 +1731,17 @@ void asmlinkage __init noreturn __start_xen(void) map_e =min_t (uint64_t, e,ARRAY_SIZE (l2_directmap) <<L2_PAGETABLE_SHIFT ); -/* Pass mapped memory to allocator /before/ creating new mappings. */ +/* +* Pass mapped memory to allocator /before/ creating new mappings. +* The direct map for the bottom 4GiB has been populated in the first +* e820 pass. In the second pass, we make sure those existing mappings +* are destroyed when directmap=no. +*/ init_boot_pages (s,min (map_s, e)); +if ( !has_directmap () ) +destroy_xen_mappings ((unsigned long )__va (s), + (unsigned long )__va (min (map_s, e))); + s = map_s;if ( s < map_e ) {
@@ -1690,6 +1749,9 @@ void asmlinkage __init noreturn __start_xen(void) map_s = (s + mask) & ~mask; map_e &= ~mask;init_boot_pages (map_s, map_e); +if ( !has_directmap () ) +destroy_xen_mappings ((unsigned long )__va (map_s), + (unsigned long )__va (map_e)); }if ( map_s > map_e )
@@ -1703,8 +1765,7 @@ void asmlinkage __init noreturn __start_xen(void)if ( map_e < end ) { -map_pages_to_xen ((unsigned long )__va (map_e),maddr_to_mfn (map_e), -PFN_DOWN (end - map_e),PAGE_HYPERVISOR ); +populate_directmap (map_e, end,PAGE_HYPERVISOR );init_boot_pages (map_e, end); map_e = end; }
@@ -1713,13 +1774,11 @@ void asmlinkage __init noreturn __start_xen(void) {/* This range must not be passed to the boot allocator and * must also not be mapped with _PAGE_GLOBAL. */ -map_pages_to_xen ((unsigned long )__va (map_e),maddr_to_mfn (map_e), -PFN_DOWN (e - map_e), __PAGE_HYPERVISOR_RW); +populate_directmap (map_e, e, __PAGE_HYPERVISOR_RW); }if ( s < map_s ) { -map_pages_to_xen ((unsigned long )__va (s),maddr_to_mfn (s), -PFN_DOWN (map_s - s),PAGE_HYPERVISOR ); +populate_directmap (s, map_s,PAGE_HYPERVISOR );init_boot_pages (s, map_s); } }
Signature · 1 line
2.47.1
Re: [PATCH v5 11/15] x86/setup: Do not create valid mappings when directmap=no
Quoted 61 lines · Alejandro Vallejo
From: Hongyan Xia <[email protected]>
Create empty mappings in the second e820 pass. Also, destroy existing direct map mappings created in the first pass.
To make xenheap pages visible in guests, it is necessary to create empty L3 tables in the direct map even when directmap=no, since guest cr3s copy idle domain's L4 entries, which means they will share mappings in the direct map if we pre-populate idle domain's L4 entries and L3 tables. A helper is introduced for this.
Also, after the direct map is actually gone, we need to stop updating the direct map in
update_xen_mappings().Signed-off-by Hongyan Xia <[email protected]>Signed-off-by Julien Grall <[email protected]>Signed-off-by Elias El Yandouzi <[email protected]>Signed-off-by Alejandro Vallejo <[email protected]>---
v4->v5:
* No changes.--- xen/arch/x86/setup.c | 73 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 66 insertions(+), 7 deletions(-)xen/arch/x86/setup.c +28
-0@@ -1060,6 +1060,56 @@ static struct domain *__init create_dom0(struct boot_info *bi)return d; } +/* +* This either populates a valid direct map, or allocates empty L3 tables and +* creates the L4 entries for virtual address between [start, end) in the +* direct map depending on has_directmap(); +* +* When directmap=no, we still need to populate empty L3 tables in the +* direct map region. The reason is that on-demand xenheap mappings are +* created in the idle domain's page table but must be seen by +* everyone. Since all domains share the direct map L4 entries, they +* will share xenheap mappings if we pre-populate the L4 entries and L3 +* tables in the direct map region for all RAM. We also rely on the fact +* that L3 tables are never freed. +*/ +static void __init populate_directmap (paddr_t pstart,paddr_t pend, +unsigned int flags) +{ +unsigned long vstart = (unsigned long )__va (pstart); +unsigned long vend = (unsigned long )__va (pend); + +if ( pstart >= pend ) +return ; + +BUG_ON (vstart <DIRECTMAP_VIRT_START ); +BUG_ON (vend >DIRECTMAP_VIRT_END ); + +if (has_directmap () ) +/* Populate valid direct map. */ + BUG_ON(map_pages_to_xen(vstart, maddr_to_mfn(pstart),
xen/arch/x86/setup.c static struct domain *__init create_dom0(struct boot_info *bi) · line 1100
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+ PFN_DOWN(pend - pstart), flags)); + else + { +/* Create empty L3 tables. */ +unsigned long vaddr = vstart & ~((1UL <<L4_PAGETABLE_SHIFT ) -1 ); + +for (unsigned long idx =l4_table_offset (vaddr); + idx <=l4_table_offset (vend); idx++ ) + { +l4_pgentry_t *pl4e = &idle_pg_table[l4_table_offset (idx)];
As we are attempting to integrate this series with the per-CPU mappings work, there's an issue here. l4_table_offset() call is duplicated, as idx is already the L4 table index:
for ( unsigned long idx = l4_table_offset(vaddr);
idx <= l4_table_offset(vend); idx++ )
{
l4_pgentry_t *pl4e = &idle_pg_table[idx];This probably went unnoticed in small systems that can fit all the directmap in a single L4 entry, but does explode on bigger ones.
Leaving a note here in case anyone else picks this up before a new version is sent.
Regards, Roger.
PATCH v5 12/15 xen/arm64: mm: Use per-pCPU page-tables
In reply to Alejandro VallejoFrom: Julien Grall <[email protected]>
At the moment, on Arm64, every pCPU is sharing the same page-tables.
In a follow-up patch, we will allow the possibility to remove the direct map and therefore it will be necessary to have a mapcache.
While we have plenty of spare virtual address space to reserve part for each pCPU, it means that temporary mappings (e.g. guest memory) could be accessible by every pCPU.
In order to increase our security posture, it would be better if those mappings are only accessible by the pCPU doing the temporary mapping.
In addition to that, a per-pCPU page-tables opens the way to have per-domain mapping area.
Arm32 is already using per-pCPU page-tables so most of the code can be re-used. Arm64 doesn't yet have support for the mapcache, so a stub is provided (moved to its own header asm/domain_page.h).
Take the opportunity to fix a typo in a comment that is modified.
---v4->v5:
* Added missing asm/domain_page.h header to arm32. Compilation fails otherwise.
* NOTE: I rebased this patch over the LLC coloring as best as I could and may have messed it up. Please do double check.
--- xen/arch/arm/arm32/mmu/mm.c | 1 + xen/arch/arm/arm64/mmu/mm.c | 3 ++- xen/arch/arm/include/asm/arm32/mm.h | 8 -------- xen/arch/arm/include/asm/domain_page.h | 13 +++++++++++++ xen/arch/arm/include/asm/mm.h | 3 +++ xen/arch/arm/include/asm/mmu/mm.h | 2 ++ xen/arch/arm/mmu/pt.c | 6 +++--- xen/arch/arm/mmu/setup.c | 23 ++++++++++------------- xen/arch/arm/mmu/smpboot.c | 16 +--------------- xen/arch/arm/setup.c | 1 + 10 files changed, 36 insertions(+), 40 deletions(-) create mode 100644 xen/arch/arm/include/asm/domain_page.h
xen/arch/arm/arm32/mmu/mm.c +1 -0
@@ -6,6 +6,7 @@#include <xen/mm.h> #include <xen/param.h> #include <xen/pfn.h> +#include <asm/domain_page.h> #include <asm/fixmap.h> #include <asm/setup.h> #include <asm/static-memory.h>
xen/arch/arm/arm64/mmu/mm.c +2 -1
@@ -77,6 +77,7 @@ static void __init prepare_runtime_identity_mapping(void)paddr_t id_addr =virt_to_maddr (_start);lpae_t pte;DECLARE_OFFSETS (id_offsets, id_addr); +lpae_t *root =this_cpu (xen_pgtable);if ( id_offsets[0 ] >=IDENTITY_MAPPING_AREA_NR_L0 ) panic("Cannot handle ID mapping above %uTB\n",
@@ -87,7 +88,7 @@ static void __init prepare_runtime_identity_mapping(void) pte.pt.table =1 ; pte.pt.xn =0 ; -write_pte (&xen_pgtable[id_offsets[0 ]], pte); +write_pte (&root[id_offsets[0 ]], pte);/* Link second ID table */ pte =pte_of_xenaddr ((vaddr_t )xen_second_id);
xen/arch/arm/include/asm/arm32/mm.h +0 -8
@@ -1,12 +1,6 @@#ifndef __ARM_ARM32_MM_H__#define __ARM_ARM32_MM_H__ -#include <xen/percpu.h> - -#include <asm/lpae.h> - -DECLARE_PER_CPU(lpae_t *, xen_pgtable); -/* * Only a limited amount of RAM, called xenheap, is always mapped on ARM32. * For convenience always return false.
@@ -16,8 +10,6 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)return false; } -bool init_domheap_mappings (unsigned int cpu); -static inline void arch_setup_page_tables (void ) { }
xen/arch/arm/include/asm/domain_page.h new file +13 -0
@@ -0,0 +1,13 @@ +#ifndef __ASM_ARM_DOMAIN_PAGE_H__ +#define __ASM_ARM_DOMAIN_PAGE_H__ + +#ifdef CONFIG_ARCH_MAP_DOMAIN_PAGE +bool init_domheap_mappings (unsigned int cpu); +#else +static inline bool init_domheap_mappings (unsigned int cpu) +{ +return true; +} +#endif + +#endif /* __ASM_ARM_DOMAIN_PAGE_H__ */
xen/arch/arm/include/asm/mm.h +3 -0
@@ -2,6 +2,9 @@#define __ARCH_ARM_MM__#include <xen/kernel.h> +#include <xen/percpu.h> + +#include <asm/lpae.h> #include <asm/page.h> #include <public/xen.h> #include <xen/pdx.h>
xen/arch/arm/include/asm/mmu/mm.h +2 -0
@@ -2,6 +2,8 @@#ifndef __ARM_MMU_MM_H__#define __ARM_MMU_MM_H__ +DECLARE_PER_CPU(lpae_t *, xen_pgtable); +/* Non-boot CPUs use this to find the correct pagetables. */ extern uint64_t init_ttbr;
xen/arch/arm/mmu/pt.c +3 -3
@@ -607,9 +607,9 @@ static int xen_pt_update(unsigned long virt,unsigned long left = nr_mfns;/* -* For arm32, page-tables are different on each CPUs. Yet, they share -* some common mappings. It is assumed that only common mappings -* will be modified with this function. +* Page-tables are different on each CPU. Yet, they share some common +* mappings. It is assumed that only common mappings will be modified +* with this function. * * XXX: Add a check. */
xen/arch/arm/mmu/setup.c +10 -13
@@ -34,17 +34,15 @@* PCPUs. */ -#ifdef CONFIG_ARM_64 -DEFINE_PAGE_TABLE (xen_pgtable); -static DEFINE_PAGE_TABLE (xen_first); -#define THIS_CPU_PGTABLE xen_pgtable -#else /* Per-CPU pagetable pages */ /* xen_pgtable == root of the trie (zeroeth level on 64-bit, first on 32-bit) */ DEFINE_PER_CPU(lpae_t *, xen_pgtable);#define THIS_CPU_PGTABLE this_cpu(xen_pgtable)/* Root of the trie for cpu0, other CPU's PTs are dynamically allocated */ static DEFINE_PAGE_TABLE (cpu0_pgtable); + +#ifdef CONFIG_ARM_64 +static DEFINE_PAGE_TABLE (xen_first);#endif /* Common pagetable leaves */
@@ -368,17 +366,20 @@ void __init setup_pagetables(void)if ( llc_coloring_enabled )create_llc_coloring_mappings (); + p = cpu0_pgtable; + +/* arch_setup_page_tables() may need to access the root page-tables. */ +per_cpu (xen_pgtable,0 ) = cpu0_pgtable; +arch_setup_page_tables ();#ifdef CONFIG_ARM_64 pte =pte_of_xenaddr ((uintptr_t )xen_first); pte.pt.table =1 ; pte.pt.xn =0 ; - xen_pgtable[zeroeth_table_offset (XEN_VIRT_START )] = pte; + p[zeroeth_table_offset (XEN_VIRT_START )] = pte; - p = (void *) xen_first; -#else - p = (void *) cpu0_pgtable; + p = xen_first;#endif /* Map xen second level page-table */
@@ -415,10 +416,6 @@ void __init setup_pagetables(void) pte.pt.table =1 ; xen_second[second_table_offset (FIXMAP_ADDR (0 ))] = pte; -#ifdef CONFIG_ARM_32 -per_cpu (xen_pgtable,0 ) = cpu0_pgtable; -#endif -if ( llc_coloring_enabled ) { ttbr =virt_to_maddr (virt_to_reloc_virt (THIS_CPU_PGTABLE ));
xen/arch/arm/mmu/smpboot.c +1 -15
@@ -7,6 +7,7 @@#include <xen/domain_page.h> +#include <asm/domain_page.h> #include <asm/setup.h> /* Override macros from asm/page.h to make them work with mfn_t */
@@ -93,20 +94,6 @@ static void set_init_ttbr(lpae_t *root)unmap_domain_page (ptr); } -#ifdef CONFIG_ARM_64 -int prepare_secondary_mm (int cpu) -{ -clear_boot_pagetables (); - -/* -* Set init_ttbr for this CPU coming up. All CPUs share a single setof -* pagetables, but rewrite it each time for consistency with 32 bit. -*/ -set_init_ttbr (xen_pgtable); - -return 0 ; -} -#else int prepare_secondary_mm (int cpu) {lpae_t *root =alloc_xenheap_page ();
@@ -136,7 +123,6 @@ int prepare_secondary_mm(int cpu)return 0 ; } -#endif /* * Local variables:
xen/arch/arm/setup.c +1 -0
@@ -44,6 +44,7 @@#include <asm/gic.h> #include <asm/cpuerrata.h> #include <asm/cpufeature.h> +#include <asm/domain_page.h> #include <asm/platform.h> #include <asm/procinfo.h> #include <asm/setup.h>
Signature · 1 line
2.47.1
PATCH v5 13/15 xen/arm32: Hardwire zeroeth_table_offset to 0 on ARM_32
In reply to Alejandro VallejoInclude arm32 in 7c72147baa22("xen/arm: Restrict zeroeth_table_offset for ARM_64"). Otherwise va overflows on shift in DECLARE_OFFSETS().
--- xen/arch/arm/include/asm/lpae.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/arm/include/asm/lpae.h +1 -1
@@ -259,7 +259,7 @@ lpae_t mfn_to_xen_entry(mfn_t mfn, unsigned int attr);#define first_table_offset (va) TABLE_OFFSET(first_linear_offset(va))#define second_table_offset (va) TABLE_OFFSET(second_linear_offset(va))#define third_table_offset (va) TABLE_OFFSET(third_linear_offset(va)) -#ifdef CONFIG_PHYS_ADDR_T_32 +#if defined(CONFIG_PHYS_ADDR_T_32 ) || defined(CONFIG_ARM_32 )#define zeroeth_table_offset (va) 0#else #define zeroeth_table_offset (va) TABLE_OFFSET(zeroeth_linear_offset(va))
Signature · 1 line
2.47.1
PATCH v5 14/15 xen/arm64: Implement a mapcache for arm64
In reply to Alejandro VallejoFrom: Julien Grall <[email protected]>
At the moment, on arm64, map_domain_page() is implemented using virt_to_mfn(). Therefore it is relying on the directmap.
In a follow-up patch, we will allow the admin to remove the directmap. Therefore we want to implement a mapcache.
Thanksfully there is already one for arm32. So select ARCH_ARM_DOMAIN_PAGE and add the necessary boiler plate to support 64-bit:
- The page-table start at level 0, so we need to allocate the level 1 page-table
- map_domain_page() should check if the page is in the directmap. If yes, then use virt_to_mfn() to limit the performance impact when the directmap is still enabled (this will be selectable on the command line).
Take the opportunity to replace first_table_offset(...) with offsets[...].
Note that, so far, arch_mfns_in_directmap() always return true on arm64. So the mapcache is not yet used. This will change in a follow-up patch.
---v4->v5:
* Add missing "select ARCH_MAP_DOMAIN_PAGE". It was weirdly dropped from v2.
* Bugfix: Unwrap mfn_t before passing it to mfn_to_virt() in
map_domain_page().
Elias @ v4:
There are a few TODOs:
- It is becoming more critical to fix the mapcache implementation (this is not compliant with the Arm Arm)
- Evaluate the performance
--- xen/arch/arm/Kconfig | 2 +- xen/arch/arm/arm64/mmu/mm.c | 9 ++++++ xen/arch/arm/include/asm/mm.h | 5 +++ xen/arch/arm/include/asm/mmu/layout.h | 13 +++++++- xen/arch/arm/mmu/domain_page.c | 45 ++++++++++++++++++++++++--- xen/arch/arm/mmu/pt.c | 6 ++-- 6 files changed, 71 insertions(+), 9 deletions(-)
xen/arch/arm/Kconfig +1 -1
@@ -1,7 +1,6 @@ config ARM_32 def_bool y depends on "$(ARCH)" = "arm32" - select ARCH_MAP_DOMAIN_PAGE config ARM_64 def_bool y
@@ -12,6 +11,7 @@ config ARM_64 config ARM def_bool y + select ARCH_MAP_DOMAIN_PAGE select FUNCTION_ALIGNMENT_4B select GENERIC_UART_INIT select HAS_ALTERNATIVE if HAS_VMAP
xen/arch/arm/arm64/mmu/mm.c +9 -0
@@ -5,6 +5,7 @@#include <xen/mm.h> #include <xen/pfn.h> +#include <asm/domain_page.h> #include <asm/setup.h> #include <asm/static-memory.h> #include <asm/static-shmem.h>
@@ -283,6 +284,14 @@ void __init setup_mm(void)setup_frametable_mappings (ram_start, ram_end); max_page =PFN_DOWN (ram_end); +/* +* The allocators may need to use map_domain_page() (such as for +* scrubbing pages). So we need to prepare the domheap area first. +*/ +if ( !init_domheap_mappings (smp_processor_id ()) ) +panic ("CPU%u: Unable to prepare the domheap page-tables\n" , +smp_processor_id ()); +init_staticmem_pages ();init_sharedmem_pages (); }
xen/arch/arm/include/asm/mm.h +5 -0
@@ -434,6 +434,11 @@ static inline void page_set_xenheap_gfn(struct page_info *p, gfn_t gfn) } while ( (y = cmpxchg(&p->u.inuse.type_info, x, nx)) != x ); } +/* Helpers to allocate, map and unmap a Xen page-table */ +int create_xen_table (lpae_t *entry); +lpae_t *xen_map_table (mfn_t mfn); +void xen_unmap_table (const lpae_t *table); +#endif /* __ARCH_ARM_MM__ */ /* * Local variables:
xen/arch/arm/include/asm/mmu/layout.h +12 -1
@@ -36,9 +36,13 @@* * 32G - 64G Frametable: 56 bytes per page for 2TB of RAM * -* 0x00000a8000000000 - 0x00007fffffffffff (512GB+117TB, L0 slots [21..255]) +* 0x00000a8000000000 - 0x00007f7fffffffff (117TB, L0 slots [21..254]) * Unused * +* 0x00007f8000000000 - 0x00007fffffffffff (512GB, L0 slot [255]) +* (Relative offsets) +* 0 - 2G Domheap: on-demand-mapped +* * 0x0000800000000000 - 0x000084ffffffffff (5TB, L0 slots [256..265]) * 1:1 mapping of RAM *
@@ -133,6 +137,13 @@#define FRAMETABLE_SIZE GB(32)#define FRAMETABLE_NR (FRAMETABLE_SIZE / sizeof(*frame_table)) +#define DOMHEAP_VIRT_START SLOT0(255) +#define DOMHEAP_VIRT_SIZE GB(2) + +#define DOMHEAP_ENTRIES 1024/* 1024 2MB mapping slots */ +/* Number of domheap pagetable pages required at the second level (2MB mappings) */ +#define DOMHEAP_SECOND_PAGES (DOMHEAP_VIRT_SIZE >> FIRST_SHIFT) +#define DIRECTMAP_VIRT_START SLOT0(256)#define DIRECTMAP_SIZE (SLOT0_ENTRY_SIZE * (266 - 256))#define DIRECTMAP_VIRT_END (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1)
xen/arch/arm/mmu/domain_page.c +41 -4
89 lines not on this page · raw message
xen/arch/arm/mmu/pt.c +3 -3
@@ -33,7 +33,7 @@ mm_printk(const char *fmt, ...) {}#define HYP_PT_ROOT_LEVEL 1#endif -static lpae_t *xen_map_table (mfn_t mfn) +lpae_t *xen_map_table (mfn_t mfn) {/* * During early boot, map_domain_page() may be unusable. Use the
@@ -45,7 +45,7 @@ static lpae_t *xen_map_table(mfn_t mfn)return map_domain_page (mfn); } -static void xen_unmap_table (const lpae_t *table) +void xen_unmap_table (const lpae_t *table) {/* * During early boot, xen_map_table() will not use map_domain_page()
@@ -228,7 +228,7 @@ void *ioremap(paddr_t pa, size_t len)return ioremap_attr (pa, len,PAGE_HYPERVISOR_NOCACHE ); } -static int create_xen_table (lpae_t *entry) +int create_xen_table (lpae_t *entry) {mfn_t mfn;void *p;
Signature · 1 line
2.47.1
PATCH v5 15/15 xen/arm64: Allow the admin to enable/disable the directmap
In reply to Alejandro VallejoFrom: Julien Grall <[email protected]>
Implement the same command line option as x86 to enable/disable the directmap. By default this is kept enabled.
Also modify setup_directmap_mappings() to populate the L0 entries related to the directmap area.
---v4->v5:
* Fixed typo in comment. s/fdirect/direct/
* Adjusted comment so 's/directmap=no/asi=true'
* Adjusted printk() so 's/on/full' and 's/off/on demand'
* s/HAS_SECRET_HIDING/HAS_ONDEMAND_DIRECTMAP. Otherwise CONFIG_ONDEMAND_DIRECTMAP won't appear on the menu for ARM (Note: I didn't test ARM because I have no boxes to do so)
v1->v2:
* Rely on the Kconfig option to enable Secret Hiding on Arm64
* Use generic helper instead of arch_has_directmap()
--- docs/misc/xen-command-line.pandoc | 2 +- xen/arch/arm/Kconfig | 1 + xen/arch/arm/arm64/mmu/mm.c | 39 +++++++++++++++++++++++++++-- xen/arch/arm/include/asm/arm64/mm.h | 7 ++---- xen/arch/arm/setup.c | 2 ++ 5 files changed, 43 insertions(+), 8 deletions(-)
docs/misc/xen-command-line.pandoc +1 -1
@@ -202,7 +202,7 @@ to appropriate auditing by Xen. Argo is disabled by default. This option is disabled by default, to protect domains from a DoS by a buggy or malicious other domain spamming the ring. -### asi (x86) +### asi (arm64, x86) > `= <boolean>` > Default: `false`
xen/arch/arm/Kconfig +1 -0
@@ -8,6 +8,7 @@ config ARM_64 select 64BIT select HAS_FAST_MULTIPLY select HAS_LLC_COLORING if !NUMA + select HAS_ONDEMAND_DIRECTMAP config ARM def_bool y
xen/arch/arm/arm64/mmu/mm.c +37 -2
@@ -3,6 +3,7 @@#include <xen/init.h> #include <xen/llc-coloring.h> #include <xen/mm.h> +#include <xen/param.h> #include <xen/pfn.h> #include <asm/domain_page.h>
@@ -14,6 +15,11 @@#undef virt_to_mfn#define virt_to_mfn (va) _mfn(__virt_to_mfn(va)) +#ifdef CONFIG_ONDEMAND_DIRECTMAP +bool __ro_after_init opt_ondemand_dmap; +boolean_param ("asi" , opt_ondemand_dmap); +#endif +static DEFINE_PAGE_TABLE (xen_first_id);static DEFINE_PAGE_TABLE (xen_second_id);static DEFINE_PAGE_TABLE (xen_third_id);
@@ -204,16 +210,27 @@ void __init switch_ttbr(uint64_t ttbr)update_identity_mapping (false); } -/* Map the region in the directmap area. */ +/* +* This either populate a valid directmap, or allocates empty L1 tables +* and creates the L0 entries for the given region in the direct map +* depending on has_directmap(). +* +* When asi=true, we still need to populate empty L1 tables in the +* directmap region. The reason is that the root page-table (i.e. L0) +* is per-CPU and secondary CPUs will initialize their root page-table +* based on the pCPU0 one. So L0 entries will be shared if they are +* pre-populated. We also rely on the fact that L1 tables are never +* freed. +*/ static void __init setup_directmap_mappings (unsigned long base_mfn,unsigned long nr_mfns) { +unsigned long mfn_gb = base_mfn & ~((FIRST_SIZE >>PAGE_SHIFT ) -1 );int rc;/* First call sets the directmap physical and virtual offset. */ if (mfn_eq (directmap_mfn_start,INVALID_MFN ) ) { -unsigned long mfn_gb = base_mfn & ~((FIRST_SIZE >>PAGE_SHIFT ) -1 ); directmap_mfn_start =_mfn (base_mfn); directmap_base_pdx =mfn_to_pdx (_mfn (base_mfn));
@@ -234,6 +251,24 @@ static void __init setup_directmap_mappings(unsigned long base_mfn,panic ("cannot add directmap mapping at %lx below heap start %lx\n" , base_mfn,mfn_x (directmap_mfn_start)); +if ( !has_directmap () ) + { +vaddr_t vaddr = (vaddr_t )__mfn_to_virt (base_mfn); +lpae_t *root =this_cpu (xen_pgtable); +unsigned int i, slot; + + slot =first_table_offset (vaddr); + nr_mfns += base_mfn - mfn_gb; +for ( i =0 ; i < nr_mfns; i +=BIT (XEN_PT_LEVEL_ORDER (0 ),UL ), slot++ ) + { +lpae_t *entry = &root[slot]; + +if ( !lpae_is_valid (*entry) && !create_xen_table (entry) ) +panic ("Unable to populate zeroeth slot %u\n" , slot); + } +return ; + } + rc =map_pages_to_xen ((vaddr_t )__mfn_to_virt (base_mfn),_mfn (base_mfn), nr_mfns,PAGE_HYPERVISOR_RW | _PAGE_BLOCK);
xen/arch/arm/include/asm/arm64/mm.h +2 -5
@@ -3,13 +3,10 @@extern DEFINE_PAGE_TABLE (xen_pgtable); -/* -* On ARM64, all the RAM is currently direct mapped in Xen. -* Hence return always true. -*/ +/* On Arm64, the user can chose whether all the RAM is directmap. */ static inline bool arch_mfns_in_directmap (unsigned long mfn,unsigned long nr) { -return true; +return has_directmap (); }void arch_setup_page_tables (void );
xen/arch/arm/setup.c +2 -0
@@ -346,6 +346,8 @@ void asmlinkage __init start_xen(unsigned long fdt_paddr) device_tree_flattened =early_fdt_map (fdt_paddr);setup_mm (); +printk ("Booting with directmap: %s\n" , +has_directmap () ?"full" :"on demand" );vm_init ();
Signature · 1 line
2.47.1
Hi,
Bah, I forgot to remove Wei from the CC list. Sorry, and beware the bounces when replying on patches 2 and 4...
Cheers,
Alejandro
Quoted 8 lines · Alejandro Vallejo
Hi,
I picked v4 of this series and run it through XenRT extensively, fixing crashes and errors as I hit them. Likewise, I've run it through Gitlab, fixing various CI failures. I listed all changes per patch. I fixed ARM to the extent that "Gitlab is happy when
CONFIG_ONDEMAND_DIRECTMAP=y", but I didn't go much further than that.
Quoted 5 lines
I _THINK_ I've covered previously unaddressed feedback, but please speak up if I missed something. Otherwise, same old same old.
Cheers,
Alejandro
Ping. Could I get some eyes on this? Or thoughts?
Cheers,
Alejandro
Quoted 9 lines · Alejandro Vallejo
Quoted 8 lines · Alejandro Vallejo
Hi,
I picked v4 of this series and run it through XenRT extensively, fixing crashes and errors as I hit them. Likewise, I've run it through Gitlab, fixing various CI failures. I listed all changes per patch. I fixed ARM to the extent that "Gitlab is happy when
CONFIG_ONDEMAND_DIRECTMAP=y", but I didn't go much further than that.
Quoted 7 lines
Quoted 5 lines
I _THINK_ I've covered previously unaddressed feedback, but please speak up if I missed something. Otherwise, same old same old.
Cheers,
AlejandroPing. Could I get some eyes on this? Or thoughts?
Sorry, it's very high on my list of review, but I need to finish something for the release before. I've certainly not forgotten about it.
Roger.