xen: Switch to virtual mapped linear p2m list
Subsystems (3)
Juergen Gross
maintainerv4 cover xen: Switch to virtual mapped linear p2m list
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
Changes in V4:
- carved out style corrections into new patch 1 as requested by David Vrabel
- minor modification in patch 5 (former patch 3) as suggested by David Vrabel
- carved out change of page allocation scheme from p2m.c into own patch as requested by Konrad Wilk, corrected for 32 bit as requested by Andrew Cooper
- added several comments
- simplified error handling in patch 4 (former patch 2)
Changes in V3:
- Carved out (new) patch 1 to make pure code movement more obvious as requested by David Vrabel
- New patch 6 introducing __pfn_to_mfn() (taken from patch 7) as requested by David Vrabel
- New patch 8 to speed up set_phys_to_machine() as suggested by David Vrabel
Changes in V2:
- splitted patch 2 in 4 smaller ones as requested by David Vrabel
- added highmem check when remapping kernel memory as requested by David Vrabel
Juergen Gross (10):
xen: fix some style issues in p2m.c
xen: Make functions static
xen: use common page allocation function in p2m.c
xen: Delay remapping memory of pv-domain
xen: Delay m2p_override initialization
xen: Delay invalidating extra memory
x86: Introduce function to get pmd entry pointer
xen: Hide get_phys_to_machine() to be able to tune common path
xen: switch to linear virtual mapped sparse p2m list
xen: Speed up set_phys_to_machine() by using read-only mappings
arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/include/asm/xen/page.h | 48 +- arch/x86/mm/pageattr.c | 20 + arch/x86/xen/mmu.c | 38 +- arch/x86/xen/p2m.c | 1320 ++++++++++++++-------------------- arch/x86/xen/setup.c | 443 ++++++------ arch/x86/xen/xen-ops.h | 6 +- 7 files changed, 838 insertions(+), 1038 deletions(-)
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 1/9 xen: fix some style issues in p2m.c
The source arch/x86/xen/p2m.c has some coding style issues. Fix them.
--- arch/x86/xen/p2m.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
arch/x86/xen/p2m.c +7 -8
@@ -922,7 +922,7 @@ int set_foreign_p2m_mapping(struct gnttab_map_grant_ref *map_ops,continue ;if (map_ops[i].flags & GNTMAP_contains_pte) { - pte = (pte_t *) (mfn_to_virt (PFN_DOWN (map_ops[i].host_addr)) + + pte = (pte_t *)(mfn_to_virt (PFN_DOWN (map_ops[i].host_addr)) + (map_ops[i].host_addr & ~PAGE_MASK )); mfn =pte_mfn (*pte); }else {
@@ -970,7 +970,7 @@ int m2p_add_override(unsigned long mfn, struct page *page, address = (unsigned long )__va (pfn <<PAGE_SHIFT ); ptep =lookup_address (address, &level);if (WARN (ptep ==NULL || level !=PG_LEVEL_4K , -"m2p_add_override: pfn %lx not mapped" , pfn)) +"m2p_add_override: pfn %lx not mapped" , pfn))return -EINVAL ; }
@@ -1072,7 +1072,7 @@ int m2p_remove_override(struct page *page, ptep =lookup_address (address, &level);if (WARN (ptep ==NULL || level !=PG_LEVEL_4K , -"m2p_remove_override: pfn %lx not mapped" , pfn)) +"m2p_remove_override: pfn %lx not mapped" , pfn))return -EINVAL ; }
@@ -1102,9 +1102,8 @@ int m2p_remove_override(struct page *page,* hypercall actually returned an error. */ if (kmap_op->handle == GNTST_general_error) { -printk (KERN_WARNING "m2p_remove_override: " -"pfn %lx mfn %lx, failed to modify kernel mappings" , - pfn, mfn); +pr_warn ("m2p_remove_override: pfn %lx mfn %lx, failed to modify kernel mappings" , + pfn, mfn);put_balloon_scratch_page ();return -1 ; }
@@ -1112,14 +1111,14 @@ int m2p_remove_override(struct page *page,xen_mc_batch (); mcs =__xen_mc_entry ( -sizeof (struct gnttab_unmap_and_replace )); +sizeof (struct gnttab_unmap_and_replace )); unmap_op = mcs.args; unmap_op->host_addr = kmap_op->host_addr; unmap_op->new_addr = scratch_page_address; unmap_op->handle = kmap_op->handle;MULTI_grant_table_op (mcs.mc, - GNTTABOP_unmap_and_replace, unmap_op,1 ); + GNTTABOP_unmap_and_replace, unmap_op,1 ); mcs =__xen_mc_entry (0 ); MULTI_update_va_mapping(mcs.mc, scratch_page_address,
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 2/9 xen: Make functions static
In reply to Juergen GrossSome functions in arch/x86/xen/p2m.c are used locally only. Make them static. Rearrange the functions in p2m.c to avoid forward declarations.
--- arch/x86/include/asm/xen/page.h | 6 - arch/x86/xen/p2m.c | 346 ++++++++++++++++++++-------------------- 2 files changed, 172 insertions(+), 180 deletions(-)
arch/x86/include/asm/xen/page.h +0 -6
@@ -52,15 +52,9 @@ extern unsigned long set_phys_range_identity(unsigned long pfn_s,extern int set_foreign_p2m_mapping (struct gnttab_map_grant_ref *map_ops,struct gnttab_map_grant_ref *kmap_ops,struct page **pages,unsigned int count); -extern int m2p_add_override (unsigned long mfn,struct page *page, -struct gnttab_map_grant_ref *kmap_op);extern int clear_foreign_p2m_mapping (struct gnttab_unmap_grant_ref *unmap_ops,struct gnttab_map_grant_ref *kmap_ops,struct page **pages,unsigned int count); -extern int m2p_remove_override (struct page *page, -struct gnttab_map_grant_ref *kmap_op, -unsigned long mfn); -extern struct page *m2p_find_override (unsigned long mfn);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn);static inline unsigned long pfn_to_mfn (unsigned long pfn)
arch/x86/xen/p2m.c +172 -174
364 lines not on this page · raw message
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 3/10 xen: use common page allocation function in p2m.c
In reply to Juergen GrossIn arch/x86/xen/p2m.c three different allocation functions for obtaining a memory page are used: extend_brk(), alloc_bootmem_align() or __get_free_page(). Which of those functions is used depends on the progress of the boot process of the system.
Introduce a common allocation routine selecting the to be called allocation routine dynamically based on the boot progress. This allows moving initialization steps without having to care about changing allocation calls.
--- arch/x86/xen/mmu.c | 2 ++ arch/x86/xen/p2m.c | 57 +++++++++++++++++++++++++++++++++--------------------- 2 files changed, 37 insertions(+), 22 deletions(-)
arch/x86/xen/mmu.c +2 -0
@@ -1219,6 +1219,8 @@ static void __init xen_pagetable_init(void)paging_init ();#ifdef CONFIG_X86_64 xen_pagetable_p2m_copy (); +#else +xen_revector_p2m_tree ();#endif /* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list ();
arch/x86/xen/p2m.c +35 -22
139 lines not on this page · raw message
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 4/10 xen: Delay remapping memory of pv-domain
In reply to Juergen GrossEarly in the boot process the memory layout of a pv-domain is changed to match the E820 map (either the host one for Dom0 or the Xen one) regarding placement of RAM and PCI holes. This requires removing memory pages initially located at positions not suitable for RAM and adding them later at higher addresses where no restrictions apply.
To be able to operate on the hypervisor supported p2m list until a virtual mapped linear p2m list can be constructed, remapping must be delayed until virtual memory management is initialized, as the initial p2m list can't be extended unlimited at physical memory initialization time due to it's fixed structure.
A further advantage is the reduction in complexity and code volume as we don't have to be careful regarding memory restrictions during p2m updates.
--- arch/x86/include/asm/xen/page.h | 1 - arch/x86/xen/mmu.c | 4 + arch/x86/xen/p2m.c | 94 ---------- arch/x86/xen/setup.c | 369 ++++++++++++++++++---------------------- arch/x86/xen/xen-ops.h | 1 + 5 files changed, 172 insertions(+), 297 deletions(-)
arch/x86/include/asm/xen/page.h +0 -1
@@ -44,7 +44,6 @@ extern unsigned long machine_to_phys_nr;extern unsigned long get_phys_to_machine (unsigned long pfn);extern bool set_phys_to_machine (unsigned long pfn,unsigned long mfn); -extern bool __init early_set_phys_to_machine (unsigned long pfn,unsigned long mfn);extern bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn);extern unsigned long set_phys_range_identity (unsigned long pfn_s,unsigned long pfn_e);
arch/x86/xen/mmu.c +4 -0
@@ -1225,6 +1225,10 @@ static void __init xen_pagetable_init(void)/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list (); +/* Remap memory freed due to conflicts with E820 map */ +if (!xen_feature (XENFEAT_auto_translated_physmap)) +xen_remap_memory (); +xen_setup_shared_info ();xen_post_allocator_init (); }
arch/x86/xen/p2m.c +0 -94
100 lines not on this page · raw message
arch/x86/xen/setup.c +167 -202
492 lines not on this page · raw message
arch/x86/xen/xen-ops.h +1 -0
@@ -35,6 +35,7 @@ void xen_mm_pin_all(void);void xen_mm_unpin_all (void );void xen_set_pat (u64 ); +void __init xen_remap_memory (void );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );void __init xen_arch_setup (void );
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 5/10 xen: Delay m2p_override initialization
In reply to Juergen GrossThe m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.
As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via extend_brk().
While at it make some m2p override functions static as they are only used internally.
--- arch/x86/xen/p2m.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
arch/x86/xen/p2m.c +12 -7
@@ -428,8 +428,6 @@ void __init xen_build_dynamic_phys_to_machine(void) } p2m_top[topidx][mididx] = &mfn_list[pfn]; } - -m2p_override_init (); }#ifdef CONFIG_X86_64 unsigned long __init xen_revector_p2m_tree (void )
@@ -500,13 +498,15 @@ unsigned long __init xen_revector_p2m_tree(void) }/* This should be the leafs allocated for identity from _brk. */ } -return (unsigned long )mfn_list; +m2p_override_init (); +return (unsigned long )mfn_list; }#else unsigned long __init xen_revector_p2m_tree (void ) { use_brk =0 ; +m2p_override_init ();return 0 ; }#endif
@@ -796,15 +796,16 @@ bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)#define M2P_OVERRIDE_HASH_SHIFT 10#define M2P_OVERRIDE_HASH (1 << M2P_OVERRIDE_HASH_SHIFT) -static RESERVE_BRK_ARRAY(struct list_head, m2p_overrides, M2P_OVERRIDE_HASH); +static struct list_head *m2p_overrides;static DEFINE_SPINLOCK (m2p_override_lock);static void __init m2p_override_init (void ) {unsigned i; - m2p_overrides =extend_brk (sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , -sizeof (unsigned long )); + m2p_overrides =alloc_bootmem_align ( +sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , +sizeof (unsigned long ));for (i =0 ; i <M2P_OVERRIDE_HASH ; i++)INIT_LIST_HEAD (&m2p_overrides[i]);
@@ -932,10 +933,14 @@ EXPORT_SYMBOL_GPL(set_foreign_p2m_mapping);static struct page *m2p_find_override (unsigned long mfn) {unsigned long flags; -struct list_head *bucket = &m2p_overrides[mfn_hash (mfn)]; +struct list_head *bucket;struct page *p, *ret; +if (unlikely (!m2p_overrides)) +return NULL ; + ret =NULL ; + bucket = &m2p_overrides[mfn_hash (mfn)];spin_lock_irqsave (&m2p_override_lock, flags);
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 6/10 xen: Delay invalidating extra memory
In reply to Juergen GrossWhen the physical memory configuration is initialized the p2m entries for not pouplated memory pages are set to "invalid". As those pages are beyond the hypervisor built p2m list the p2m tree has to be extended.
This patch delays processing the extra memory related p2m entries during the boot process until some more basic memory management functions are callable. This removes the need to create new p2m entries until virtual memory management is available.
--- arch/x86/include/asm/xen/page.h | 3 + arch/x86/xen/p2m.c | 128 ++++++++-------------------------------- arch/x86/xen/setup.c | 98 ++++++++++++++++++++++-------- arch/x86/xen/xen-ops.h | 3 +- 4 files changed, 103 insertions(+), 129 deletions(-)
arch/x86/include/asm/xen/page.h +3 -0
@@ -41,6 +41,9 @@ typedef struct xpaddr {extern unsigned long *machine_to_phys_mapping;extern unsigned long machine_to_phys_nr; +extern unsigned long *xen_p2m_addr; +extern unsigned long xen_p2m_size; +extern unsigned long xen_max_p2m_pfn;extern unsigned long get_phys_to_machine (unsigned long pfn);extern bool set_phys_to_machine (unsigned long pfn,unsigned long mfn);
arch/x86/xen/p2m.c +25 -103
193 lines not on this page · raw message
arch/x86/xen/setup.c +73 -25
170 lines not on this page · raw message
arch/x86/xen/xen-ops.h +2 -1
@@ -29,12 +29,13 @@ void xen_build_mfn_list_list(void);void xen_setup_machphys_mapping (void );void xen_setup_kernel_pagetable (pgd_t *pgd,unsigned long max_pfn);void xen_reserve_top (void ); -extern unsigned long xen_max_p2m_pfn;void xen_mm_pin_all (void );void xen_mm_unpin_all (void );void xen_set_pat (u64 ); +unsigned long __ref xen_chk_extra_mem (unsigned long pfn); +void __init xen_inv_extra_mem (void );void __init xen_remap_memory (void );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 7/10 x86: Introduce function to get pmd entry pointer
In reply to Juergen GrossIntroduces lookup_pmd_address() to get the address of the pmd entry related to a virtual address in the current address space. This function is needed for support of a virtual mapped sparse p2m list in xen pv domains, as we need the address of the pmd entry, not the one of the pte in that case.
--- arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/mm/pageattr.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+)
arch/x86/include/asm/pgtable_types.h +1 -0
@@ -396,6 +396,7 @@ static inline void update_page_count(int level, unsigned long pages) { }extern pte_t *lookup_address (unsigned long address,unsigned int *level);extern pte_t *lookup_address_in_pgd (pgd_t *pgd,unsigned long address,unsigned int *level); +extern pmd_t *lookup_pmd_address (unsigned long address);extern phys_addr_t slow_virt_to_phys (void *__address);extern int kernel_map_pages_in_pgd (pgd_t *pgd,u64 pfn,unsigned long address,unsigned numpages,unsigned long page_flags);
arch/x86/mm/pageattr.c +20 -0
@@ -384,6 +384,26 @@ static pte_t *_lookup_address_cpa(struct cpa_data *cpa, unsigned long address, }/* +* Lookup the PMD entry for a virtual address. Return a pointer to the entry +* or NULL if not present. +*/ +pmd_t *lookup_pmd_address (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; + + pgd =pgd_offset_k (address); +if (pgd_none (*pgd)) +return NULL ; + + pud =pud_offset (pgd, address); +if (pud_none (*pud) ||pud_large (*pud) || !pud_present (*pud)) +return NULL ; + +return pmd_offset (pud, address); +} + +/* * This is necessary because __pa() does not work on some * kinds of memory, like vmalloc() or the alloc_remap() * areas on 32-bit NUMA systems. The percpu areas can
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 8/10 xen: Hide get_phys_to_machine() to be able to tune common path
In reply to Juergen GrossToday get_phys_to_machine() is always called when the mfn for a pfn is to be obtained. Add a wrapper __pfn_to_mfn() as inline function to be able to avoid calling get_phys_to_machine() when possible as soon as the switch to a linear mapped p2m list has been done.
--- arch/x86/include/asm/xen/page.h | 26 ++++++++++++++++++++------ arch/x86/xen/mmu.c | 2 +- arch/x86/xen/p2m.c | 6 +++--- 3 files changed, 24 insertions(+), 10 deletions(-)
arch/x86/include/asm/xen/page.h +20 -6
@@ -59,6 +59,21 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, struct page **pages, unsigned int count);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn); +/* +* When to use pfn_to_mfn(), __pfn_to_mfn() or get_phys_to_machine(): +* - pfn_to_mfn() returns either INVALID_P2M_ENTRY or the mfn. No indicator +* bits (identity or foreign) are set. +* - __pfn_to_mfn() returns the found entry of the p2m table. A possibly set +* identity or foreign indicator will be still set. __pfn_to_mfn() is +* encapsulating get_phys_to_machine(). +* - get_phys_to_machine() is to be called by __pfn_to_mfn() only to allow +* for future optimizations. +*/ +static inline unsigned long __pfn_to_mfn (unsigned long pfn) +{ +return get_phys_to_machine (pfn); +} +static inline unsigned long pfn_to_mfn (unsigned long pfn) {unsigned long mfn;
@@ -66,7 +81,7 @@ static inline unsigned long pfn_to_mfn(unsigned long pfn)if (xen_feature (XENFEAT_auto_translated_physmap))return pfn; - mfn =get_phys_to_machine (pfn); + mfn =__pfn_to_mfn (pfn);if (mfn !=INVALID_P2M_ENTRY ) mfn &= ~(FOREIGN_FRAME_BIT |IDENTITY_FRAME_BIT );
@@ -79,7 +94,7 @@ static inline int phys_to_machine_mapping_valid(unsigned long pfn)if (xen_feature (XENFEAT_auto_translated_physmap))return 1 ; -return get_phys_to_machine (pfn) !=INVALID_P2M_ENTRY ; +return __pfn_to_mfn (pfn) !=INVALID_P2M_ENTRY ; }static inline unsigned long mfn_to_pfn_no_overrides (unsigned long mfn)
@@ -113,7 +128,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)return mfn; pfn =mfn_to_pfn_no_overrides (mfn); -if (get_phys_to_machine (pfn) != mfn) { +if (__pfn_to_mfn (pfn) != mfn) {/* * If this appears to be a foreign mfn (because the pfn * doesn't map back to the mfn), then check the local override
@@ -129,8 +144,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)* entry doesn't map back to the mfn and m2p_override doesn't have a * valid entry for it. */ -if (pfn == ~0 && -get_phys_to_machine (mfn) ==IDENTITY_FRAME (mfn)) +if (pfn == ~0 &&__pfn_to_mfn (mfn) ==IDENTITY_FRAME (mfn)) pfn = mfn;return pfn;
@@ -176,7 +190,7 @@ static inline unsigned long mfn_to_local_pfn(unsigned long mfn)return mfn; pfn =mfn_to_pfn (mfn); -if (get_phys_to_machine (pfn) != mfn) +if (__pfn_to_mfn (pfn) != mfn)return -1 ;/* force !pfn_valid() */ return pfn; }
arch/x86/xen/mmu.c +1 -1
@@ -387,7 +387,7 @@ static pteval_t pte_pfn_to_mfn(pteval_t val)unsigned long mfn;if (!xen_feature (XENFEAT_auto_translated_physmap)) - mfn =get_phys_to_machine (pfn); + mfn =__pfn_to_mfn (pfn);else mfn = pfn;/*
arch/x86/xen/p2m.c +3 -3
@@ -787,7 +787,7 @@ static int m2p_add_override(unsigned long mfn, struct page *page,* because mfn_to_pfn (that ends up being called by GUPF) will * return the backend pfn rather than the frontend pfn. */ pfn =mfn_to_pfn_no_overrides (mfn); -if (get_phys_to_machine (pfn) == mfn) +if (__pfn_to_mfn (pfn) == mfn)set_phys_to_machine (pfn,FOREIGN_FRAME (mfn));return 0 ;
@@ -967,7 +967,7 @@ static int m2p_remove_override(struct page *page,* pfn again. */ mfn &= ~FOREIGN_FRAME_BIT ; pfn =mfn_to_pfn_no_overrides (mfn); -if (get_phys_to_machine (pfn) ==FOREIGN_FRAME (mfn) && +if (__pfn_to_mfn (pfn) ==FOREIGN_FRAME (mfn) &&m2p_find_override (mfn) ==NULL )set_phys_to_machine (pfn, mfn);
@@ -992,7 +992,7 @@ int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, }for (i =0 ; i < count; i++) { -unsigned long mfn =get_phys_to_machine (page_to_pfn (pages[i])); +unsigned long mfn =__pfn_to_mfn (page_to_pfn (pages[i]));unsigned long pfn =page_to_pfn (pages[i]);if (mfn ==INVALID_P2M_ENTRY || !(mfn &FOREIGN_FRAME_BIT )) {
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 9/10 xen: switch to linear virtual mapped sparse p2m list
In reply to Juergen GrossAt start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.
Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.
Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:
Elapsed time: 32:50 -> 32:35 System: 18:07 -> 17:47 User: 104:00 -> 103:30
Tested with following configurations:
- 64 bit dom0, 8GB RAM
- 64 bit dom0, 128 GB RAM, PCI-area above 4 GB
- 32 bit domU, 512 MB, 8 GB, 43 GB (more wouldn't work even without the patch)
- 32 bit domU, ballooning up and down
- 32 bit domU, save and restore
- 32 bit domU with PCI passthrough
- 64 bit domU, 8 GB, 2049 MB, 5000 MB
- 64 bit domU, ballooning up and down
- 64 bit domU, save and restore
- 64 bit domU with PCI passthrough
--- arch/x86/include/asm/xen/page.h | 20 +- arch/x86/xen/mmu.c | 34 +- arch/x86/xen/p2m.c | 735 +++++++++++++++++----------------------- arch/x86/xen/xen-ops.h | 2 +- 4 files changed, 347 insertions(+), 444 deletions(-)
arch/x86/include/asm/xen/page.h +16 -4
@@ -65,13 +65,25 @@ extern unsigned long m2p_find_override_pfn(unsigned long mfn, unsigned long pfn)* bits (identity or foreign) are set. * - __pfn_to_mfn() returns the found entry of the p2m table. A possibly set * identity or foreign indicator will be still set. __pfn_to_mfn() is -* encapsulating get_phys_to_machine(). -* - get_phys_to_machine() is to be called by __pfn_to_mfn() only to allow -* for future optimizations. +* encapsulating get_phys_to_machine() which is called in special cases only. +* - get_phys_to_machine() is to be called by __pfn_to_mfn() only in special +* cases needing an extended handling. */ static inline unsigned long __pfn_to_mfn (unsigned long pfn) { -return get_phys_to_machine (pfn); +unsigned long mfn; + +if (pfn < xen_p2m_size) + mfn = xen_p2m_addr[pfn]; +else if (unlikely (pfn < xen_max_p2m_pfn)) +return get_phys_to_machine (pfn); +else +return IDENTITY_FRAME (pfn); + +if (unlikely (mfn ==INVALID_P2M_ENTRY )) +return get_phys_to_machine (pfn); + +return mfn; }static inline unsigned long pfn_to_mfn (unsigned long pfn)
arch/x86/xen/mmu.c +20 -14
@@ -1158,20 +1158,16 @@ static void __init xen_cleanhighmap(unsigned long vaddr,* instead of somewhere later and be confusing. */ xen_mc_flush (); } -static void __init xen_pagetable_p2m_copy (void ) + +static void __init xen_pagetable_p2m_free (void ) {unsigned long size;unsigned long addr; -unsigned long new_mfn_list; - -if (xen_feature (XENFEAT_auto_translated_physmap)) -return ; size =PAGE_ALIGN (xen_start_info->nr_pages *sizeof (unsigned long )); - new_mfn_list =xen_revector_p2m_tree ();/* No memory or already called. */ -if (!new_mfn_list || new_mfn_list == xen_start_info->mfn_list) +if ((unsigned long )xen_p2m_addr == xen_start_info->mfn_list)return ;/* using __ka address and sticking INVALID_P2M_ENTRY! */
@@ -1189,8 +1185,6 @@ static void __init xen_pagetable_p2m_copy(void) size =PAGE_ALIGN (xen_start_info->nr_pages *sizeof (unsigned long ));memblock_free (__pa (xen_start_info->mfn_list), size); -/* And revector! Bye bye old array */ - xen_start_info->mfn_list = new_mfn_list;/* At this stage, cleanup_highmap has already cleaned __ka space * from _brk_limit way up to the max_pfn_mapped (which is the end of
@@ -1214,14 +1208,26 @@ static void __init xen_pagetable_p2m_copy(void) }#endif -static void __init xen_pagetable_init (void ) +static void __init xen_pagetable_p2m_setup (void ) { -paging_init (); +if (xen_feature (XENFEAT_auto_translated_physmap)) +return ; + +xen_vmalloc_p2m_tree (); +#ifdef CONFIG_X86_64 -xen_pagetable_p2m_copy (); -#else -xen_revector_p2m_tree (); +xen_pagetable_p2m_free ();#endif +/* And revector! Bye bye old array */ + xen_start_info->mfn_list = (unsigned long )xen_p2m_addr; +} + +static void __init xen_pagetable_init (void ) +{ +paging_init (); + +xen_pagetable_p2m_setup (); +/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list ();
arch/x86/xen/p2m.c +310 -425
972 lines not on this page · raw message
arch/x86/xen/xen-ops.h +1 -1
@@ -49,7 +49,7 @@ void xen_hvm_init_shared_info(void);void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void ); -unsigned long __init xen_revector_p2m_tree (void ); +void __init xen_vmalloc_p2m_tree (void );void xen_init_irq_ops (void );void xen_setup_timer (int cpu);
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v4 10/10 xen: Speed up set_phys_to_machine() by using read-only mappings
In reply to Juergen GrossInstead of checking at each call of set_phys_to_machine() whether a new p2m page has to be allocated due to writing an entry in a large invalid or identity area, just map those areas read only and react to a page fault on write by allocating the new page.
This change will make the common path with no allocation much faster as it only requires a single write of the new mfn instead of walking the address translation tables and checking for the special cases.
--- arch/x86/xen/p2m.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/x86/xen/p2m.c +8 -6
@@ -70,6 +70,7 @@#include <asm/cache.h> #include <asm/setup.h> +#include <asm/uaccess.h> #include <asm/xen/page.h> #include <asm/xen/hypercall.h>
@@ -316,9 +317,9 @@ static void __init xen_rebuild_p2m_list(unsigned long *p2m)paravirt_alloc_pte (&init_mm,__pa (p2m_identity_pte) >>PAGE_SHIFT );for (i =0 ; i <PTRS_PER_PTE ; i++) {set_pte (p2m_missing_pte + i, -pfn_pte (PFN_DOWN (__pa (p2m_missing)),PAGE_KERNEL )); +pfn_pte (PFN_DOWN (__pa (p2m_missing)),PAGE_KERNEL_RO ));set_pte (p2m_identity_pte + i, -pfn_pte (PFN_DOWN (__pa (p2m_identity)),PAGE_KERNEL )); +pfn_pte (PFN_DOWN (__pa (p2m_identity)),PAGE_KERNEL_RO )); }for (pfn =0 ; pfn < xen_max_p2m_pfn; pfn += chunk) {
@@ -365,7 +366,7 @@ static void __init xen_rebuild_p2m_list(unsigned long *p2m) p2m_missing : p2m_identity; ptep =populate_extra_pte ((unsigned long )(p2m + pfn));set_pte (ptep, -pfn_pte (PFN_DOWN (__pa (mfns)),PAGE_KERNEL )); +pfn_pte (PFN_DOWN (__pa (mfns)),PAGE_KERNEL_RO ));continue ; }
@@ -624,6 +625,9 @@ bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)return true; } +if (likely (!__put_user (mfn, xen_p2m_addr + pfn))) +return true; + ptep =lookup_address ((unsigned long )(xen_p2m_addr + pfn), &level);BUG_ON (!ptep || level !=PG_LEVEL_4K );
@@ -633,9 +637,7 @@ bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)if (pte_pfn (*ptep) ==PFN_DOWN (__pa (p2m_identity)))return mfn ==IDENTITY_FRAME (pfn); - xen_p2m_addr[pfn] = mfn; - -return true; +return false; }bool set_phys_to_machine (unsigned long pfn,unsigned long mfn)
Signature · 1 line
2.1.2
Quoted 10 lines · Juergen Gross
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
What are the results of the testing I asked for?
If Konrad or Boris are happy with this series I think it can go into 3.19.
David
Juergen Gross
maintainerQuoted 6 lines · David Vrabel
Quoted 5 lines · Juergen Gross
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
Quoted 7 lines
Quoted 5 lines
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
What are the results of the testing I asked for?
You asked for those replying to the patch introducing the linear mapping. So I added the description of the tested scenarios in the patch description of that patch.
Quoted 1 line
If Konrad or Boris are happy with this series I think it can go into 3.19.
Great, thanks!
Juergen
Quoted 15 lines · Juergen Gross
Quoted 6 lines · David Vrabel
Quoted 5 lines · Juergen Gross
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
Quoted 7 lines
Quoted 5 lines
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
What are the results of the testing I asked for?
Quoted 7 lines
You asked for those replying to the patch introducing the linear mapping. So I added the description of the tested scenarios in the patch description of that patch.
Quoted 1 line
If Konrad or Boris are happy with this series I think it can go into 3.19.
Great, thanks!
I am happy.
Quoted 4 lines
Juergen
David Vrabel
AppliedQuoted 10 lines · Juergen Gross
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
Applied to devel/for-linus-3.19.
Thanks.
David