xen: Switch to virtual mapped linear p2m list
Subsystems (3)
Juergen Gross
maintainerv2 cover xen: Switch to virtual mapped linear p2m list
Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.
By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.
Changes in V2:
- splitted patch 2 in 4 smaller ones as requested by David Vrabel
- added highmem check when remapping kernel memory as requested by David Vrabel
Juergen Gross (5):
Xen: Delay remapping memory of pv-domain
xen: Delay m2p_override initialization
xen: Delay invalidating extra memory
x86: Introduce function to get pmd entry pointer
Xen: switch to linear virtual mapped sparse p2m list
arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/include/asm/xen/page.h | 37 +- arch/x86/mm/pageattr.c | 20 + arch/x86/xen/mmu.c | 38 +- arch/x86/xen/p2m.c | 1149 +++++++++++++--------------------- arch/x86/xen/setup.c | 460 +++++++------- arch/x86/xen/xen-ops.h | 6 +- 7 files changed, 760 insertions(+), 951 deletions(-)
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v2 1/5 Xen: Delay remapping memory of pv-domain
Early in the boot process the memory layout of a pv-domain is changed to match the E820 map (either the host one for Dom0 or the Xen one) regarding placement of RAM and PCI holes. This requires removing memory pages initially located at positions not suitable for RAM and adding them later at higher addresses where no restrictions apply.
To be able to operate on the hypervisor supported p2m list until a virtual mapped linear p2m list can be constructed, remapping must be delayed until virtual memory management is initialized, as the initial p2m list can't be extended unlimited at physical memory initialization time due to it's fixed structure.
A further advantage is the reduction in complexity and code volume as we don't have to be careful regarding memory restrictions during p2m updates.
--- arch/x86/include/asm/xen/page.h | 1 - arch/x86/xen/mmu.c | 4 + arch/x86/xen/p2m.c | 149 ++++------------ arch/x86/xen/setup.c | 385 +++++++++++++++++++--------------------- arch/x86/xen/xen-ops.h | 1 + 5 files changed, 223 insertions(+), 317 deletions(-)
arch/x86/include/asm/xen/page.h +0 -1
@@ -44,7 +44,6 @@ extern unsigned long machine_to_phys_nr;extern unsigned long get_phys_to_machine (unsigned long pfn);extern bool set_phys_to_machine (unsigned long pfn,unsigned long mfn); -extern bool __init early_set_phys_to_machine (unsigned long pfn,unsigned long mfn);extern bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn);extern unsigned long set_phys_range_identity (unsigned long pfn_s,unsigned long pfn_e);
arch/x86/xen/mmu.c +4 -0
@@ -1223,6 +1223,10 @@ static void __init xen_pagetable_init(void)/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list (); +/* Remap memory freed because of conflicts with E820 map */ +if (!xen_feature (XENFEAT_auto_translated_physmap)) +xen_remap_memory (); +xen_setup_shared_info ();xen_post_allocator_init (); }
arch/x86/xen/p2m.c +33 -116
237 lines not on this page · raw message
arch/x86/xen/setup.c +185 -200
507 lines not on this page · raw message
arch/x86/xen/xen-ops.h +1 -0
@@ -35,6 +35,7 @@ void xen_mm_pin_all(void);void xen_mm_unpin_all (void );void xen_set_pat (u64 ); +void __init xen_remap_memory (void );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );void __init xen_arch_setup (void );
Signature · 1 line
2.1.2
Re: [Xen-devel] [PATCH V2 1/5] Xen: Delay remapping memory of pv-domain
Quoted 12 lines · Juergen Gross
Early in the boot process the memory layout of a pv-domain is changed to match the E820 map (either the host one for Dom0 or the Xen one) regarding placement of RAM and PCI holes. This requires removing memory pages initially located at positions not suitable for RAM and adding them later at higher addresses where no restrictions apply.
To be able to operate on the hypervisor supported p2m list until a virtual mapped linear p2m list can be constructed, remapping must be delayed until virtual memory management is initialized, as the initial p2m list can't be extended unlimited at physical memory initialization time due to it's fixed structure.
Quoted 3 lines
A further advantage is the reduction in complexity and code volume as we don't have to be careful regarding memory restrictions during p2m updates.
But I would like a second reviewed-by.
David
Juergen Gross
maintainerPATCH v2 2/5 xen: Delay m2p_override initialization
In reply to Juergen GrossThe m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.
As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via extend_brk().
While at it make some m2p override functions static as they are only used internally.
--- arch/x86/include/asm/xen/page.h | 6 -- arch/x86/xen/p2m.c | 198 ++++++++++++++++++++-------------------- 2 files changed, 101 insertions(+), 103 deletions(-)
arch/x86/include/asm/xen/page.h +0 -6
@@ -51,15 +51,9 @@ extern unsigned long set_phys_range_identity(unsigned long pfn_s,extern int set_foreign_p2m_mapping (struct gnttab_map_grant_ref *map_ops,struct gnttab_map_grant_ref *kmap_ops,struct page **pages,unsigned int count); -extern int m2p_add_override (unsigned long mfn,struct page *page, -struct gnttab_map_grant_ref *kmap_op);extern int clear_foreign_p2m_mapping (struct gnttab_unmap_grant_ref *unmap_ops,struct gnttab_map_grant_ref *kmap_ops,struct page **pages,unsigned int count); -extern int m2p_remove_override (struct page *page, -struct gnttab_map_grant_ref *kmap_op, -unsigned long mfn); -extern struct page *m2p_find_override (unsigned long mfn);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn);static inline unsigned long pfn_to_mfn (unsigned long pfn)
arch/x86/xen/p2m.c +101 -97
280 lines not on this page · raw message
Signature · 1 line
2.1.2
Re: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization
Quoted 11 lines · Juergen Gross
The m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.
As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via
extend_brk().While at it make some m2p override functions static as they are only used internally.
[...]
Quoted 5 lines
arch/x86/xen/p2m.c +0
-1}/* This should be the leafs allocated for identity from _brk. */ } -return (unsigned long )mfn_list;
arch/x86/xen/p2m.c unsigned long __init xen_revector_p2m_tree(void) · line 509
Quoted 10 lines
arch/x86/xen/p2m.c +3
-0+m2p_override_init (); + return (unsigned long)mfn_list; }#else unsigned long __init xen_revector_p2m_tree (void ) { use_brk =0 ; +m2p_override_init ();return 0 ; }
This is mentioned in the description...
Quoted 8 lines
#endif
arch/x86/xen/p2m.c +1
-1@@ -794,15 +794,16 @@ bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)#define M2P_OVERRIDE_HASH_SHIFT 10#define M2P_OVERRIDE_HASH (1 << M2P_OVERRIDE_HASH_SHIFT) -static RESERVE_BRK_ARRAY(struct list_head, m2p_overrides, M2P_OVERRIDE_HASH); +static struct list_head *m2p_overrides;static DEFINE_SPINLOCK (m2p_override_lock);
arch/x86/xen/p2m.c bool set_phys_to_machine(unsigned long pfn, unsigned long mfn) · line 806
Quoted 10 lines
arch/x86/xen/p2m.c +3
-2static void __init m2p_override_init (void ) {unsigned i; - m2p_overrides =extend_brk (sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , -sizeof (unsigned long )); + m2p_overrides =alloc_bootmem_align ( +sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , +sizeof (unsigned long ));
...as is this.
Quoted 7 lines
arch/x86/xen/p2m.c +0
-7-int set_foreign_p2m_mapping (struct gnttab_map_grant_ref *map_ops, -struct gnttab_map_grant_ref *kmap_ops, -struct page **pages,unsigned int count) -{ -int i, ret =0 ; -bool lazy = false; -pte_t *pte;
arch/x86/xen/p2m.c static unsigned long mfn_hash(unsigned long mfn) · line 832
Quoted 10 lines
arch/x86/xen/p2m.c +0
-10- -if (xen_feature (XENFEAT_auto_translated_physmap)) -return 0 ; - -if (kmap_ops && - !in_interrupt () && -paravirt_get_lazy_mode () ==PARAVIRT_LAZY_NONE ) { -arch_enter_lazy_mmu_mode (); - lazy = true; - }
... but, what's going on here. What are the rest of these changes?
I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.
David
Juergen Gross
maintainerRe: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization
Quoted 72 lines · David Vrabel
Quoted 11 lines · Juergen Gross
The m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.
As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via
extend_brk().While at it make some m2p override functions static as they are only used internally.
[...]
Quoted 5 lines
} /* This should be the leafs allocated for identity from _brk. */ }arch/x86/xen/p2m.c +0
-1-return (unsigned long )mfn_list;arch/x86/xen/p2m.c unsigned long __init xen_revector_p2m_tree(void) · line 509
Quoted 10 lines
arch/x86/xen/p2m.c +3
-0+m2p_override_init (); + return (unsigned long)mfn_list; }#else unsigned long __init xen_revector_p2m_tree (void ) { use_brk =0 ; +m2p_override_init ();return 0 ; }This is mentioned in the description...
Quoted 8 lines
#endifarch/x86/xen/p2m.c +1
-1@@ -794,15 +794,16 @@ bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)#define M2P_OVERRIDE_HASH_SHIFT 10#define M2P_OVERRIDE_HASH (1 << M2P_OVERRIDE_HASH_SHIFT) -static RESERVE_BRK_ARRAY(struct list_head, m2p_overrides, M2P_OVERRIDE_HASH); +static struct list_head *m2p_overrides;static DEFINE_SPINLOCK (m2p_override_lock);arch/x86/xen/p2m.c bool set_phys_to_machine(unsigned long pfn, unsigned long mfn) · line 806
Quoted 10 lines
arch/x86/xen/p2m.c +3
-2static void __init m2p_override_init (void ) {unsigned i; - m2p_overrides =extend_brk (sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , -sizeof (unsigned long )); + m2p_overrides =alloc_bootmem_align ( +sizeof (*m2p_overrides) *M2P_OVERRIDE_HASH , +sizeof (unsigned long ));...as is this.
Quoted 7 lines
arch/x86/xen/p2m.c +0
-7-int set_foreign_p2m_mapping (struct gnttab_map_grant_ref *map_ops, -struct gnttab_map_grant_ref *kmap_ops, -struct page **pages,unsigned int count) -{ -int i, ret =0 ; -bool lazy = false; -pte_t *pte;arch/x86/xen/p2m.c static unsigned long mfn_hash(unsigned long mfn) · line 832
Quoted 10 lines
arch/x86/xen/p2m.c +0
-10- -if (xen_feature (XENFEAT_auto_translated_physmap)) -return 0 ; - -if (kmap_ops && - !in_interrupt () && -paravirt_get_lazy_mode () ==PARAVIRT_LAZY_NONE ) { -arch_enter_lazy_mmu_mode (); - lazy = true; - }... but, what's going on here. What are the rest of these changes?
Quoted 4 lines
I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.
Do you really prefer forward declarations instead of pure code movement? I can do as you request, but just want to make sure.
While I agree that the diff output is really ugly, comparing the old and new code by looking at it in two editor windows side by side should show the pure movement of functions easily.
Juergen
Re: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization
Quoted 9 lines · Juergen Gross
Quoted 5 lines · David Vrabel
I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.
Do you really prefer forward declarations instead of pure code movement? I can do as you request, but just want to make sure.
Normally I prefer not having forward declarations... How about a patch that does this and then a 2nd patch that does the pure code motion. Maybe that will produce a better diff?
You can also try the --patience option for git diff (etc.). This may produce more readable diffs.
David
Juergen Gross
maintainerPATCH v2 3/5 xen: Delay invalidating extra memory
In reply to Juergen GrossWhen the physical memory configuration is initialized the p2m entries for not pouplated memory pages are set to "invalid". As those pages are beyond the hypervisor built p2m list the p2m tree has to be extended.
This patch delays processing the extra memory related p2m entries during the boot process until some more basic memory management functions are callable. This removes the need to create new p2m entries until virtual memory management is available.
--- arch/x86/include/asm/xen/page.h | 3 + arch/x86/xen/p2m.c | 130 ++++++++-------------------------------- arch/x86/xen/setup.c | 103 ++++++++++++++++++++++--------- arch/x86/xen/xen-ops.h | 3 +- 4 files changed, 107 insertions(+), 132 deletions(-)
arch/x86/include/asm/xen/page.h +3 -0
@@ -41,6 +41,9 @@ typedef struct xpaddr {extern unsigned long *machine_to_phys_mapping;extern unsigned long machine_to_phys_nr; +extern unsigned long *xen_p2m_addr; +extern unsigned long xen_p2m_size; +extern unsigned long xen_max_p2m_pfn;extern unsigned long get_phys_to_machine (unsigned long pfn);extern bool set_phys_to_machine (unsigned long pfn,unsigned long mfn);
arch/x86/xen/p2m.c +26 -104
197 lines not on this page · raw message
arch/x86/xen/setup.c +76 -27
181 lines not on this page · raw message
arch/x86/xen/xen-ops.h +2 -1
@@ -29,12 +29,13 @@ void xen_build_mfn_list_list(void);void xen_setup_machphys_mapping (void );void xen_setup_kernel_pagetable (pgd_t *pgd,unsigned long max_pfn);void xen_reserve_top (void ); -extern unsigned long xen_max_p2m_pfn;void xen_mm_pin_all (void );void xen_mm_unpin_all (void );void xen_set_pat (u64 ); +unsigned long __ref xen_chk_extra_mem (unsigned long pfn); +void __init xen_inv_extra_mem (void );void __init xen_remap_memory (void );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );
Signature · 1 line
2.1.2
Re: [PATCH V2 3/5] xen: Delay invalidating extra memory
Quoted 9 lines · Juergen Gross
When the physical memory configuration is initialized the p2m entries for not pouplated memory pages are set to "invalid". As those pages are beyond the hypervisor built p2m list the p2m tree has to be extended.
This patch delays processing the extra memory related p2m entries during the boot process until some more basic memory management functions are callable. This removes the need to create new p2m entries until virtual memory management is available.
But I would like a second reviewed by for all the patches in this series since it's such a complex area.
David
Juergen Gross
maintainerPATCH v2 4/5 x86: Introduce function to get pmd entry pointer
In reply to Juergen GrossIntroduces lookup_pmd_address() to get the address of the pmd entry related to a virtual address in the current address space. This function is needed for support of a virtual mapped sparse p2m list in xen pv domains.
--- arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/mm/pageattr.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+)
arch/x86/include/asm/pgtable_types.h +1 -0
@@ -396,6 +396,7 @@ static inline void update_page_count(int level, unsigned long pages) { }extern pte_t *lookup_address (unsigned long address,unsigned int *level);extern pte_t *lookup_address_in_pgd (pgd_t *pgd,unsigned long address,unsigned int *level); +extern pmd_t *lookup_pmd_address (unsigned long address);extern phys_addr_t slow_virt_to_phys (void *__address);extern int kernel_map_pages_in_pgd (pgd_t *pgd,u64 pfn,unsigned long address,unsigned numpages,unsigned long page_flags);
arch/x86/mm/pageattr.c +20 -0
@@ -384,6 +384,26 @@ static pte_t *_lookup_address_cpa(struct cpa_data *cpa, unsigned long address, }/* +* Lookup the PMD entry for a virtual address. Return a pointer to the entry +* or NULL if not present. +*/ +pmd_t *lookup_pmd_address (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; + + pgd =pgd_offset_k (address); +if (pgd_none (*pgd)) +return NULL ; + + pud =pud_offset (pgd, address); +if (pud_none (*pud) ||pud_large (*pud) || !pud_present (*pud)) +return NULL ; + +return pmd_offset (pud, address); +} + +/* * This is necessary because __pa() does not work on some * kinds of memory, like vmalloc() or the alloc_remap() * areas on 32-bit NUMA systems. The percpu areas can
Signature · 1 line
2.1.2
Juergen Gross
maintainerPATCH v2 5/5 Xen: switch to linear virtual mapped sparse p2m list
In reply to Juergen GrossAt start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.
Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.
Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:
Elapsed time: 32:50 -> 32:35 System: 18:07 -> 17:47 User: 104:00 -> 103:30
Tested on 64 bit dom0 and 32 bit domU.
--- arch/x86/include/asm/xen/page.h | 27 +- arch/x86/xen/mmu.c | 34 +- arch/x86/xen/p2m.c | 736 +++++++++++++++++----------------------- arch/x86/xen/xen-ops.h | 2 +- 4 files changed, 354 insertions(+), 445 deletions(-)
arch/x86/include/asm/xen/page.h +22 -5
@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, struct page **pages, unsigned int count);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn); +static inline unsigned long __pfn_to_mfn (unsigned long pfn) +{ +unsigned long mfn; + +if (pfn < xen_p2m_size) + mfn = xen_p2m_addr[pfn]; +else if (unlikely (pfn < xen_max_p2m_pfn)) +return get_phys_to_machine (pfn); +else +return IDENTITY_FRAME (pfn); + +if (unlikely (mfn ==INVALID_P2M_ENTRY )) +return get_phys_to_machine (pfn); + +return mfn; +} +static inline unsigned long pfn_to_mfn (unsigned long pfn) {unsigned long mfn;
@@ -66,7 +83,7 @@ static inline unsigned long pfn_to_mfn(unsigned long pfn)if (xen_feature (XENFEAT_auto_translated_physmap))return pfn; - mfn =get_phys_to_machine (pfn); + mfn =__pfn_to_mfn (pfn);if (mfn !=INVALID_P2M_ENTRY ) mfn &= ~(FOREIGN_FRAME_BIT |IDENTITY_FRAME_BIT );
@@ -79,7 +96,7 @@ static inline int phys_to_machine_mapping_valid(unsigned long pfn)if (xen_feature (XENFEAT_auto_translated_physmap))return 1 ; -return get_phys_to_machine (pfn) !=INVALID_P2M_ENTRY ; +return __pfn_to_mfn (pfn) !=INVALID_P2M_ENTRY ; }static inline unsigned long mfn_to_pfn_no_overrides (unsigned long mfn)
@@ -113,7 +130,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)return mfn; pfn =mfn_to_pfn_no_overrides (mfn); -if (get_phys_to_machine (pfn) != mfn) { +if (__pfn_to_mfn (pfn) != mfn) {/* * If this appears to be a foreign mfn (because the pfn * doesn't map back to the mfn), then check the local override
@@ -130,7 +147,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)* valid entry for it. */ if (pfn == ~0 && -get_phys_to_machine (mfn) ==IDENTITY_FRAME (mfn)) +__pfn_to_mfn (mfn) ==IDENTITY_FRAME (mfn)) pfn = mfn;return pfn;
@@ -176,7 +193,7 @@ static inline unsigned long mfn_to_local_pfn(unsigned long mfn)return mfn; pfn =mfn_to_pfn (mfn); -if (get_phys_to_machine (pfn) != mfn) +if (__pfn_to_mfn (pfn) != mfn)return -1 ;/* force !pfn_valid() */ return pfn; }
arch/x86/xen/mmu.c +21 -13
@@ -387,7 +387,7 @@ static pteval_t pte_pfn_to_mfn(pteval_t val)unsigned long mfn;if (!xen_feature (XENFEAT_auto_translated_physmap)) - mfn =get_phys_to_machine (pfn); + mfn =__pfn_to_mfn (pfn);else mfn = pfn;/*
@@ -1158,20 +1158,16 @@ static void __init xen_cleanhighmap(unsigned long vaddr,* instead of somewhere later and be confusing. */ xen_mc_flush (); } -static void __init xen_pagetable_p2m_copy (void ) + +static void __init xen_pagetable_p2m_free (void ) {unsigned long size;unsigned long addr; -unsigned long new_mfn_list; - -if (xen_feature (XENFEAT_auto_translated_physmap)) -return ; size =PAGE_ALIGN (xen_start_info->nr_pages *sizeof (unsigned long )); - new_mfn_list =xen_revector_p2m_tree ();/* No memory or already called. */ -if (!new_mfn_list || new_mfn_list == xen_start_info->mfn_list) +if ((unsigned long )xen_p2m_addr == xen_start_info->mfn_list)return ;/* using __ka address and sticking INVALID_P2M_ENTRY! */
@@ -1189,8 +1185,6 @@ static void __init xen_pagetable_p2m_copy(void) size =PAGE_ALIGN (xen_start_info->nr_pages *sizeof (unsigned long ));memblock_free (__pa (xen_start_info->mfn_list), size); -/* And revector! Bye bye old array */ - xen_start_info->mfn_list = new_mfn_list;/* At this stage, cleanup_highmap has already cleaned __ka space * from _brk_limit way up to the max_pfn_mapped (which is the end of
@@ -1214,12 +1208,26 @@ static void __init xen_pagetable_p2m_copy(void) }#endif -static void __init xen_pagetable_init (void ) +static void __init xen_pagetable_p2m_setup (void ) { -paging_init (); +if (xen_feature (XENFEAT_auto_translated_physmap)) +return ; + +xen_vmalloc_p2m_tree (); +#ifdef CONFIG_X86_64 -xen_pagetable_p2m_copy (); +xen_pagetable_p2m_free ();#endif +/* And revector! Bye bye old array */ + xen_start_info->mfn_list = (unsigned long )xen_p2m_addr; +} + +static void __init xen_pagetable_init (void ) +{ +paging_init (); + +xen_pagetable_p2m_setup (); +/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list ();
arch/x86/xen/p2m.c +310 -426
985 lines not on this page · raw message
arch/x86/xen/xen-ops.h +1 -1
@@ -49,7 +49,7 @@ void xen_hvm_init_shared_info(void);void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void ); -unsigned long __init xen_revector_p2m_tree (void ); +void __init xen_vmalloc_p2m_tree (void );void xen_init_irq_ops (void );void xen_setup_timer (int cpu);
Signature · 1 line
2.1.2
Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list
Quoted 15 lines · Juergen Gross
At start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.
Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.
Quoted 6 lines
Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:
Elapsed time: 32:50 -> 32:35 System: 18:07 -> 17:47 User: 104:00 -> 103:30
After implementing my suggestions below, please provided updated figure.
They should be better.
Quoted 1 line
Tested on 64 bit dom0 and 32 bit domU.
[...]
arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 62
Quoted 7 lines
arch/x86/include/asm/xen/page.h +1
-0@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, struct page **pages, unsigned int count);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn); +static inline unsigned long __pfn_to_mfn (unsigned long pfn)
These variations of pfn_to_mfn() (__pfn_to_mfn() and get_phys_to_machine() and any others), need comments explaining their differences.
Can you add __pfn_to_mfn() and the docs in a separate patch?
arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 63
Quoted 1 line
arch/x86/include/asm/xen/page.h +1
-0+pr_notice ("p2m virtual area at %p, size is %lx\n" , vm.addr, vm.size);
pr_info().
Quoted 11 lines
arch/x86/include/asm/xen/page.h +2
-3@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)return IDENTITY_FRAME (pfn); } - topidx =p2m_top_index (pfn); - mididx =p2m_mid_index (pfn); - idx =p2m_index (pfn); + ptep =lookup_address ((unsigned long )(xen_p2m_addr + pfn), &level); +BUG_ON (!ptep || level !=PG_LEVEL_4K );/*
arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 425
Quoted 10 lines
arch/x86/include/asm/xen/page.h +2
-2* The INVALID_P2M_ENTRY is filled in both p2m_*identity * and in p2m_*missing, so returning the INVALID_P2M_ENTRY * would be wrong. */ -if (p2m_top[topidx][mididx] == p2m_identity) +if (pte_pfn (*ptep) ==PFN_DOWN (__pa (p2m_identity)))return IDENTITY_FRAME (pfn); -return p2m_top[topidx][mididx][idx]; +return xen_p2m_addr[pfn];
You should test xen_p2m_addr[pfn] == INVALID_P2M_ENTRY before checking if it's an identity entry. This should skip the more expensive lookup_address() in the common case.
arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 426
Quoted 1 line
arch/x86/include/asm/xen/page.h +0
-0bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn)
I think you should map p2m_missing and p2m_identity as read-only and do the new page allocation on a write fault.
set_phys_to_machine() is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:
xen_p2m_addr[pfn] = mfn;
David
Juergen Gross
maintainerRe: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list
Quoted 19 lines · David Vrabel
Quoted 18 lines · Juergen Gross
At start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.
Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.
Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:
Quoted 6 lines
Quoted 3 lines
Elapsed time: 32:50 -> 32:35 System: 18:07 -> 17:47 User: 104:00 -> 103:30After implementing my suggestions below, please provided updated figure.
They should be better.
In dom0? I don't think so.
Quoted 9 lines
Quoted 2 lines
Tested on 64 bit dom0 and 32 bit domU.
[...]
Quoted 6 lines
arch/x86/include/asm/xen/page.h +0
-0@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, struct page **pages, unsigned int count);extern unsigned long m2p_find_override_pfn (unsigned long mfn,unsigned long pfn);
Quoted 7 lines
arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 62
Quoted 1 line
arch/x86/include/asm/xen/page.h +1
-0+static inline unsigned long __pfn_to_mfn (unsigned long pfn)These variations of
pfn_to_mfn()(__pfn_to_mfn()andget_phys_to_machine()and any others), need comments explaining their differences.Can you add
__pfn_to_mfn()and the docs in a separate patch?
Okay.
Quoted 24 lines
arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 63
Quoted 2 lines
arch/x86/include/asm/xen/page.h +1
-0+pr_notice ("p2m virtual area at %p, size is %lx\n" , vm.addr, vm.size);
pr_info().Quoted 19 lines
arch/x86/include/asm/xen/page.h +3
-4@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)return IDENTITY_FRAME (pfn); } - topidx =p2m_top_index (pfn); - mididx =p2m_mid_index (pfn); - idx =p2m_index (pfn); + ptep =lookup_address ((unsigned long )(xen_p2m_addr + pfn), &level); +BUG_ON (!ptep || level !=PG_LEVEL_4K );/* * The INVALID_P2M_ENTRY is filled in both p2m_*identity * and in p2m_*missing, so returning the INVALID_P2M_ENTRY * would be wrong. */ -if (p2m_top[topidx][mididx] == p2m_identity) +if (pte_pfn (*ptep) ==PFN_DOWN (__pa (p2m_identity)))return IDENTITY_FRAME (pfn);
Quoted 6 lines
arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 425
Quoted 2 lines
arch/x86/include/asm/xen/page.h +1
-1-return p2m_top[topidx][mididx][idx]; +return xen_p2m_addr[pfn];You should test xen_p2m_addr[pfn] ==
INVALID_P2M_ENTRYbefore checking if it's an identity entry. This should skip the more expensivelookup_address()in the common case.
I do. The check is in __pfn_to_mfn(). get_phys_to_machine() is called in this case only.
Quoted 10 lines
arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 426
Quoted 2 lines
arch/x86/include/asm/xen/page.h +0
-0bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn)I think you should map
p2m_missingandp2m_identityas read-only and do the new page allocation on a write fault.
set_phys_to_machine()is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:xen_p2m_addr[pfn] = mfn;
Nice idea. I'll try it.
Juergen
Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list
Quoted 30 lines · Juergen Gross
Quoted 22 lines · David Vrabel
On 06/11/14 05:47, Juergen Gross wrote:
[...]Quoted 20 lines
arch/x86/xen/p2m.c +3
-4@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)return IDENTITY_FRAME (pfn); } - topidx =p2m_top_index (pfn); - mididx =p2m_mid_index (pfn); - idx =p2m_index (pfn); + ptep =lookup_address ((unsigned long )(xen_p2m_addr + pfn), &level); +BUG_ON (!ptep || level !=PG_LEVEL_4K );/* * The INVALID_P2M_ENTRY is filled in both p2m_*identity * and in p2m_*missing, so returning the INVALID_P2M_ENTRY * would be wrong. */ -if (p2m_top[topidx][mididx] == p2m_identity) +if (pte_pfn (*ptep) ==PFN_DOWN (__pa (p2m_identity)))return IDENTITY_FRAME (pfn);Quoted 6 lines
arch/x86/xen/p2m.c unsigned long get_phys_to_machine(unsigned long pfn) · line 425
Quoted 2 lines
arch/x86/xen/p2m.c +1
-1-return p2m_top[topidx][mididx][idx]; +return xen_p2m_addr[pfn];You should test xen_p2m_addr[pfn] ==
INVALID_P2M_ENTRYbefore checking if it's an identity entry. This should skip the more expensivelookup_address()in the common case.
Quoted 2 lines
I do. The check is in
__pfn_to_mfn().get_phys_to_machine()is called in this case only.
So you do. I missed that.
Quoted 11 lines
Quoted 9 lines
arch/x86/xen/p2m.c unsigned long __init set_phys_range_identity(unsigned long pfn_s, · line 610
Quoted 1 line
arch/x86/xen/p2m.c +0
-0bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn)I think you should map
p2m_missingandp2m_identityas read-only and do the new page allocation on a write fault.
set_phys_to_machine()is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:xen_p2m_addr[pfn] = mfn;Nice idea. I'll try it.
You probably want to try this with a custom exception fixup (or abuse put_user()).
David
Juergen Gross
maintainerRe: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list
Quoted 44 lines · David Vrabel
Quoted 30 lines · Juergen Gross
Quoted 22 lines · David Vrabel
On 06/11/14 05:47, Juergen Gross wrote:
[...]Quoted 20 lines
arch/x86/xen/p2m.c +3
-4@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)return IDENTITY_FRAME (pfn); } - topidx =p2m_top_index (pfn); - mididx =p2m_mid_index (pfn); - idx =p2m_index (pfn); + ptep =lookup_address ((unsigned long )(xen_p2m_addr + pfn), &level); +BUG_ON (!ptep || level !=PG_LEVEL_4K );/* * The INVALID_P2M_ENTRY is filled in both p2m_*identity * and in p2m_*missing, so returning the INVALID_P2M_ENTRY * would be wrong. */ -if (p2m_top[topidx][mididx] == p2m_identity) +if (pte_pfn (*ptep) ==PFN_DOWN (__pa (p2m_identity)))return IDENTITY_FRAME (pfn);Quoted 6 lines
arch/x86/xen/p2m.c unsigned long get_phys_to_machine(unsigned long pfn) · line 425
Quoted 2 lines
arch/x86/xen/p2m.c +1
-1-return p2m_top[topidx][mididx][idx]; +return xen_p2m_addr[pfn];You should test xen_p2m_addr[pfn] ==
INVALID_P2M_ENTRYbefore checking if it's an identity entry. This should skip the more expensivelookup_address()in the common case.Quoted 2 lines
I do. The check is in
__pfn_to_mfn().get_phys_to_machine()is called in this case only.So you do. I missed that.
Quoted 8 lines
Quoted 8 lines
arch/x86/xen/p2m.c unsigned long __init set_phys_range_identity(unsigned long pfn_s, · line 610
Quoted 1 line
arch/x86/xen/p2m.c +0
-0bool __set_phys_to_machine (unsigned long pfn,unsigned long mfn)I think you should map
p2m_missingandp2m_identityas read-only and do the new page allocation on a write fault.
set_phys_to_machine()is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
xen_p2m_addr[pfn] = mfn;Nice idea. I'll try it.
You probably want to try this with a custom exception fixup (or abuse
put_user()).
Yeah, that was my idea, too.
Juergen