Skip to contentSkip to conversation

xen: Switch to virtual mapped linear p2m list

PATCH Superseded Juergen Gross 2 participants 15 messages Nov 6, 2014 – Nov 7, 2014

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Juergen Gross

author maintainer
#0 lore raw

v2 cover xen: Switch to virtual mapped linear p2m list

Paravirtualized kernels running on Xen use a three level tree for translation of guest specific physical addresses to machine global addresses. This p2m tree is used for construction of page table entries, so the p2m tree walk is performance critical.

By using a linear virtual mapped p2m list accesses to p2m elements can be sped up while even simplifying code. To achieve this goal some p2m related initializations have to be performed later in the boot process, as the final p2m list can be set up only after basic memory management functions are available.

Changes in V2:
- splitted patch 2 in 4 smaller ones as requested by David Vrabel
- added highmem check when remapping kernel memory as requested by David Vrabel

Juergen Gross (5):
Xen: Delay remapping memory of pv-domain
xen: Delay m2p_override initialization
xen: Delay invalidating extra memory
x86: Introduce function to get pmd entry pointer
Xen: switch to linear virtual mapped sparse p2m list

 arch/x86/include/asm/pgtable_types.h |    1 +
 arch/x86/include/asm/xen/page.h      |   37 +-
 arch/x86/mm/pageattr.c               |   20 +
 arch/x86/xen/mmu.c                   |   38 +-
 arch/x86/xen/p2m.c                   | 1149 +++++++++++++---------------------
 arch/x86/xen/setup.c                 |  460 +++++++-------
 arch/x86/xen/xen-ops.h               |    6 +-
 7 files changed, 760 insertions(+), 951 deletions(-)
Signature · 1 line
2.1.2
author maintainer
#1 lore raw

PATCH v2 1/5 Xen: Delay remapping memory of pv-domain

Early in the boot process the memory layout of a pv-domain is changed to match the E820 map (either the host one for Dom0 or the Xen one) regarding placement of RAM and PCI holes. This requires removing memory pages initially located at positions not suitable for RAM and adding them later at higher addresses where no restrictions apply.

To be able to operate on the hypervisor supported p2m list until a virtual mapped linear p2m list can be constructed, remapping must be delayed until virtual memory management is initialized, as the initial p2m list can't be extended unlimited at physical memory initialization time due to it's fixed structure.

A further advantage is the reduction in complexity and code volume as we don't have to be careful regarding memory restrictions during p2m updates.

---
 arch/x86/include/asm/xen/page.h |   1 -
 arch/x86/xen/mmu.c              |   4 +
 arch/x86/xen/p2m.c              | 149 ++++------------
 arch/x86/xen/setup.c            | 385 +++++++++++++++++++---------------------
 arch/x86/xen/xen-ops.h          |   1 +
 5 files changed, 223 insertions(+), 317 deletions(-)
arch/x86/include/asm/xen/page.h +0 -1
index c949923..c29619c 100644
@@ -44,7 +44,6 @@ extern unsigned long  machine_to_phys_nr;
 
 extern unsigned long get_phys_to_machine(unsigned long pfn);
 extern bool set_phys_to_machine(unsigned long pfn, unsigned long mfn);
-extern bool __init early_set_phys_to_machine(unsigned long pfn, unsigned long mfn);
 extern bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn);
 extern unsigned long set_phys_range_identity(unsigned long pfn_s,
 					     unsigned long pfn_e);
arch/x86/xen/mmu.c +4 -0
index a8a1a3d..d3e492b 100644
@@ -1223,6 +1223,10 @@ static void __init xen_pagetable_init(void)
 	/* Allocate and initialize top and mid mfn levels for p2m structure */
 	xen_build_mfn_list_list();
 
+	/* Remap memory freed because of conflicts with E820 map */
+	if (!xen_feature(XENFEAT_auto_translated_physmap))
+		xen_remap_memory();
+
 	xen_setup_shared_info();
 	xen_post_allocator_init();
 }
arch/x86/xen/p2m.c +33 -116

237 lines not on this page · raw message

arch/x86/xen/setup.c +185 -200

507 lines not on this page · raw message

arch/x86/xen/xen-ops.h +1 -0
index 28c7e0b..5b72a06 100644
@@ -35,6 +35,7 @@ void xen_mm_pin_all(void);
 void xen_mm_unpin_all(void);
 void xen_set_pat(u64);
 
+void __init xen_remap_memory(void);
 char * __init xen_memory_setup(void);
 char * xen_auto_xlated_memory_setup(void);
 void __init xen_arch_setup(void);
Signature · 1 line
2.1.2
Reviewed-by on v2 1/5
#2 lore raw

Re: [Xen-devel] [PATCH V2 1/5] Xen: Delay remapping memory of pv-domain

Quoted 12 lines · Juergen Gross

Early in the boot process the memory layout of a pv-domain is changed to match the E820 map (either the host one for Dom0 or the Xen one) regarding placement of RAM and PCI holes. This requires removing memory pages initially located at positions not suitable for RAM and adding them later at higher addresses where no restrictions apply.

To be able to operate on the hypervisor supported p2m list until a virtual mapped linear p2m list can be constructed, remapping must be delayed until virtual memory management is initialized, as the initial p2m list can't be extended unlimited at physical memory initialization time due to it's fixed structure.

Quoted 3 lines

A further advantage is the reduction in complexity and code volume as we don't have to be careful regarding memory restrictions during p2m updates.

But I would like a second reviewed-by.

David

author maintainer
#3 lore raw

PATCH v2 2/5 xen: Delay m2p_override initialization

In reply to Juergen Gross

The m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.

As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via extend_brk().

While at it make some m2p override functions static as they are only used internally.

---
 arch/x86/include/asm/xen/page.h |   6 --
 arch/x86/xen/p2m.c              | 198 ++++++++++++++++++++--------------------
 2 files changed, 101 insertions(+), 103 deletions(-)
arch/x86/include/asm/xen/page.h +0 -6
index c29619c..b475297 100644
@@ -51,15 +51,9 @@ extern unsigned long set_phys_range_identity(unsigned long pfn_s,
 extern int set_foreign_p2m_mapping(struct gnttab_map_grant_ref *map_ops,
 				   struct gnttab_map_grant_ref *kmap_ops,
 				   struct page **pages, unsigned int count);
-extern int m2p_add_override(unsigned long mfn, struct page *page,
-			    struct gnttab_map_grant_ref *kmap_op);
 extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops,
 				     struct gnttab_map_grant_ref *kmap_ops,
 				     struct page **pages, unsigned int count);
-extern int m2p_remove_override(struct page *page,
-			       struct gnttab_map_grant_ref *kmap_op,
-			       unsigned long mfn);
-extern struct page *m2p_find_override(unsigned long mfn);
 extern unsigned long m2p_find_override_pfn(unsigned long mfn, unsigned long pfn);
 
 static inline unsigned long pfn_to_mfn(unsigned long pfn)
arch/x86/xen/p2m.c +101 -97

280 lines not on this page · raw message

Signature · 1 line
2.1.2
#4 lore raw

Re: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization

Quoted 11 lines · Juergen Gross

The m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.

As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via extend_brk().

While at it make some m2p override functions static as they are only used internally.

[...]

Quoted 5 lines
arch/x86/xen/p2m.c +0 -1
 		}
 		/* This should be the leafs allocated for identity from _brk. */
 	}
-	return (unsigned long)mfn_list;
 

arch/x86/xen/p2m.c unsigned long __init xen_revector_p2m_tree(void) · line 509

Quoted 10 lines
arch/x86/xen/p2m.c +3 -0
+	m2p_override_init();
+	return (unsigned long)mfn_list;
 }
 #else
 unsigned long __init xen_revector_p2m_tree(void)
 {
 	use_brk = 0;
+	m2p_override_init();
 	return 0;
 }

This is mentioned in the description...

Quoted 8 lines

#endif

arch/x86/xen/p2m.c +1 -1
@@ -794,15 +794,16 @@ bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)
 #define M2P_OVERRIDE_HASH_SHIFT	10
 #define M2P_OVERRIDE_HASH	(1 << M2P_OVERRIDE_HASH_SHIFT)
 
-static RESERVE_BRK_ARRAY(struct list_head, m2p_overrides, M2P_OVERRIDE_HASH);
+static struct list_head *m2p_overrides;
 static DEFINE_SPINLOCK(m2p_override_lock);

arch/x86/xen/p2m.c bool set_phys_to_machine(unsigned long pfn, unsigned long mfn) · line 806

Quoted 10 lines
arch/x86/xen/p2m.c +3 -2
 
 static void __init m2p_override_init(void)
 {
 	unsigned i;
 
-	m2p_overrides = extend_brk(sizeof(*m2p_overrides) * M2P_OVERRIDE_HASH,
-				   sizeof(unsigned long));
+	m2p_overrides = alloc_bootmem_align(
+				sizeof(*m2p_overrides) * M2P_OVERRIDE_HASH,
+				sizeof(unsigned long));

...as is this.

Quoted 7 lines
arch/x86/xen/p2m.c +0 -7
-int set_foreign_p2m_mapping(struct gnttab_map_grant_ref *map_ops,
-			    struct gnttab_map_grant_ref *kmap_ops,
-			    struct page **pages, unsigned int count)
-{
-	int i, ret = 0;
-	bool lazy = false;
-	pte_t *pte;

arch/x86/xen/p2m.c static unsigned long mfn_hash(unsigned long mfn) · line 832

Quoted 10 lines
arch/x86/xen/p2m.c +0 -10
-
-	if (xen_feature(XENFEAT_auto_translated_physmap))
-		return 0;
-
-	if (kmap_ops &&
-	    !in_interrupt() &&
-	    paravirt_get_lazy_mode() == PARAVIRT_LAZY_NONE) {
-		arch_enter_lazy_mmu_mode();
-		lazy = true;
-	}

... but, what's going on here. What are the rest of these changes?

I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.

David

author maintainer
#5 lore raw

Re: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization

Quoted 72 lines · David Vrabel
Quoted 11 lines · Juergen Gross

The m2p overrides are used to be able to find the local pfn for a foreign mfn mapped into the domain. They are used by driver backends having to access frontend data.

As this functionality isn't used in early boot it makes no sense to initialize the m2p override functions very early. It can be done later without doing any harm, removing the need for allocating memory via extend_brk().

While at it make some m2p override functions static as they are only used internally.

[...]

Quoted 5 lines
  		}
  		/* This should be the leafs allocated for identity from _brk. */
  	}
arch/x86/xen/p2m.c +0 -1
-	return (unsigned long)mfn_list;
 

arch/x86/xen/p2m.c unsigned long __init xen_revector_p2m_tree(void) · line 509

Quoted 10 lines
arch/x86/xen/p2m.c +3 -0
+	m2p_override_init();
+	return (unsigned long)mfn_list;
  }
  #else
  unsigned long __init xen_revector_p2m_tree(void)
  {
  	use_brk = 0;
+	m2p_override_init();
  	return 0;
  }

This is mentioned in the description...

Quoted 8 lines
  #endif
arch/x86/xen/p2m.c +1 -1
@@ -794,15 +794,16 @@ bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)
  #define M2P_OVERRIDE_HASH_SHIFT	10
  #define M2P_OVERRIDE_HASH	(1 << M2P_OVERRIDE_HASH_SHIFT)
 
-static RESERVE_BRK_ARRAY(struct list_head, m2p_overrides, M2P_OVERRIDE_HASH);
+static struct list_head *m2p_overrides;
  static DEFINE_SPINLOCK(m2p_override_lock);

arch/x86/xen/p2m.c bool set_phys_to_machine(unsigned long pfn, unsigned long mfn) · line 806

Quoted 10 lines
arch/x86/xen/p2m.c +3 -2
 
  static void __init m2p_override_init(void)
  {
  	unsigned i;
 
-	m2p_overrides = extend_brk(sizeof(*m2p_overrides) * M2P_OVERRIDE_HASH,
-				   sizeof(unsigned long));
+	m2p_overrides = alloc_bootmem_align(
+				sizeof(*m2p_overrides) * M2P_OVERRIDE_HASH,
+				sizeof(unsigned long));

...as is this.

Quoted 7 lines
arch/x86/xen/p2m.c +0 -7
-int set_foreign_p2m_mapping(struct gnttab_map_grant_ref *map_ops,
-			    struct gnttab_map_grant_ref *kmap_ops,
-			    struct page **pages, unsigned int count)
-{
-	int i, ret = 0;
-	bool lazy = false;
-	pte_t *pte;

arch/x86/xen/p2m.c static unsigned long mfn_hash(unsigned long mfn) · line 832

Quoted 10 lines
arch/x86/xen/p2m.c +0 -10
-
-	if (xen_feature(XENFEAT_auto_translated_physmap))
-		return 0;
-
-	if (kmap_ops &&
-	    !in_interrupt() &&
-	    paravirt_get_lazy_mode() == PARAVIRT_LAZY_NONE) {
-		arch_enter_lazy_mmu_mode();
-		lazy = true;
-	}

... but, what's going on here. What are the rest of these changes?

Quoted 4 lines

I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.

Do you really prefer forward declarations instead of pure code movement? I can do as you request, but just want to make sure.

While I agree that the diff output is really ugly, comparing the old and new code by looking at it in two editor windows side by side should show the pure movement of functions easily.

Juergen

#6 lore raw

Re: [Xen-devel] [PATCH V2 2/5] xen: Delay m2p_override initialization

Quoted 9 lines · Juergen Gross
Quoted 5 lines · David Vrabel

I suppose this is the "make some functions static" but it's an unreviewable mess. If you can't do this with some one line changes adding "static" and perhaps some forward declarations then please drop this bit.

Do you really prefer forward declarations instead of pure code movement? I can do as you request, but just want to make sure.

Normally I prefer not having forward declarations... How about a patch that does this and then a 2nd patch that does the pure code motion. Maybe that will produce a better diff?

You can also try the --patience option for git diff (etc.). This may produce more readable diffs.

David

author maintainer
#7 lore raw

PATCH v2 3/5 xen: Delay invalidating extra memory

In reply to Juergen Gross

When the physical memory configuration is initialized the p2m entries for not pouplated memory pages are set to "invalid". As those pages are beyond the hypervisor built p2m list the p2m tree has to be extended.

This patch delays processing the extra memory related p2m entries during the boot process until some more basic memory management functions are callable. This removes the need to create new p2m entries until virtual memory management is available.

---
 arch/x86/include/asm/xen/page.h |   3 +
 arch/x86/xen/p2m.c              | 130 ++++++++--------------------------------
 arch/x86/xen/setup.c            | 103 ++++++++++++++++++++++---------
 arch/x86/xen/xen-ops.h          |   3 +-
 4 files changed, 107 insertions(+), 132 deletions(-)
arch/x86/include/asm/xen/page.h +3 -0
index b475297..28fa795 100644
@@ -41,6 +41,9 @@ typedef struct xpaddr {
 
 extern unsigned long *machine_to_phys_mapping;
 extern unsigned long  machine_to_phys_nr;
+extern unsigned long *xen_p2m_addr;
+extern unsigned long  xen_p2m_size;
+extern unsigned long  xen_max_p2m_pfn;
 
 extern unsigned long get_phys_to_machine(unsigned long pfn);
 extern bool set_phys_to_machine(unsigned long pfn, unsigned long mfn);
arch/x86/xen/p2m.c +26 -104

197 lines not on this page · raw message

arch/x86/xen/setup.c +76 -27

181 lines not on this page · raw message

arch/x86/xen/xen-ops.h +2 -1
index 5b72a06..02b0b0f 100644
@@ -29,12 +29,13 @@ void xen_build_mfn_list_list(void);
 void xen_setup_machphys_mapping(void);
 void xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn);
 void xen_reserve_top(void);
-extern unsigned long xen_max_p2m_pfn;
 
 void xen_mm_pin_all(void);
 void xen_mm_unpin_all(void);
 void xen_set_pat(u64);
 
+unsigned long __ref xen_chk_extra_mem(unsigned long pfn);
+void __init xen_inv_extra_mem(void);
 void __init xen_remap_memory(void);
 char * __init xen_memory_setup(void);
 char * xen_auto_xlated_memory_setup(void);
Signature · 1 line
2.1.2
Reviewed-by on v2 3/5
#8 lore raw

Re: [PATCH V2 3/5] xen: Delay invalidating extra memory

Quoted 9 lines · Juergen Gross

When the physical memory configuration is initialized the p2m entries for not pouplated memory pages are set to "invalid". As those pages are beyond the hypervisor built p2m list the p2m tree has to be extended.

This patch delays processing the extra memory related p2m entries during the boot process until some more basic memory management functions are callable. This removes the need to create new p2m entries until virtual memory management is available.

But I would like a second reviewed by for all the patches in this series since it's such a complex area.

David

author maintainer
#9 lore raw

PATCH v2 4/5 x86: Introduce function to get pmd entry pointer

In reply to Juergen Gross

Introduces lookup_pmd_address() to get the address of the pmd entry related to a virtual address in the current address space. This function is needed for support of a virtual mapped sparse p2m list in xen pv domains.

---
 arch/x86/include/asm/pgtable_types.h |  1 +
 arch/x86/mm/pageattr.c               | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+)
arch/x86/include/asm/pgtable_types.h +1 -0
index 0778964..d83f5e7 100644
@@ -396,6 +396,7 @@ static inline void update_page_count(int level, unsigned long pages) { }
 extern pte_t *lookup_address(unsigned long address, unsigned int *level);
 extern pte_t *lookup_address_in_pgd(pgd_t *pgd, unsigned long address,
 				    unsigned int *level);
+extern pmd_t *lookup_pmd_address(unsigned long address);
 extern phys_addr_t slow_virt_to_phys(void *__address);
 extern int kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address,
 				   unsigned numpages, unsigned long page_flags);
arch/x86/mm/pageattr.c +20 -0
index 36de293..1298108 100644
@@ -384,6 +384,26 @@ static pte_t *_lookup_address_cpa(struct cpa_data *cpa, unsigned long address,
 }
 
 /*
+ * Lookup the PMD entry for a virtual address. Return a pointer to the entry
+ * or NULL if not present.
+ */
+pmd_t *lookup_pmd_address(unsigned long address)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+
+	pgd = pgd_offset_k(address);
+	if (pgd_none(*pgd))
+		return NULL;
+
+	pud = pud_offset(pgd, address);
+	if (pud_none(*pud) || pud_large(*pud) || !pud_present(*pud))
+		return NULL;
+
+	return pmd_offset(pud, address);
+}
+
+/*
  * This is necessary because __pa() does not work on some
  * kinds of memory, like vmalloc() or the alloc_remap()
  * areas on 32-bit NUMA systems.  The percpu areas can
Signature · 1 line
2.1.2
author maintainer
#10 lore raw

PATCH v2 5/5 Xen: switch to linear virtual mapped sparse p2m list

In reply to Juergen Gross

At start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.

Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.

Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:

Elapsed time: 32:50 ->  32:35
System:       18:07 ->  17:47
User:        104:00 -> 103:30

Tested on 64 bit dom0 and 32 bit domU.

---
 arch/x86/include/asm/xen/page.h |  27 +-
 arch/x86/xen/mmu.c              |  34 +-
 arch/x86/xen/p2m.c              | 736 +++++++++++++++++-----------------------
 arch/x86/xen/xen-ops.h          |   2 +-
 4 files changed, 354 insertions(+), 445 deletions(-)
arch/x86/include/asm/xen/page.h +22 -5
index 28fa795..dd65a0c 100644
@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops,
 				     struct page **pages, unsigned int count);
 extern unsigned long m2p_find_override_pfn(unsigned long mfn, unsigned long pfn);
 
+static inline unsigned long __pfn_to_mfn(unsigned long pfn)
+{
+	unsigned long mfn;
+
+	if (pfn < xen_p2m_size)
+		mfn = xen_p2m_addr[pfn];
+	else if (unlikely(pfn < xen_max_p2m_pfn))
+		return get_phys_to_machine(pfn);
+	else
+		return IDENTITY_FRAME(pfn);
+
+	if (unlikely(mfn == INVALID_P2M_ENTRY))
+		return get_phys_to_machine(pfn);
+
+	return mfn;
+}
+
 static inline unsigned long pfn_to_mfn(unsigned long pfn)
 {
 	unsigned long mfn;
@@ -66,7 +83,7 @@ static inline unsigned long pfn_to_mfn(unsigned long pfn)
 	if (xen_feature(XENFEAT_auto_translated_physmap))
 		return pfn;
 
-	mfn = get_phys_to_machine(pfn);
+	mfn = __pfn_to_mfn(pfn);
 
 	if (mfn != INVALID_P2M_ENTRY)
 		mfn &= ~(FOREIGN_FRAME_BIT | IDENTITY_FRAME_BIT);
@@ -79,7 +96,7 @@ static inline int phys_to_machine_mapping_valid(unsigned long pfn)
 	if (xen_feature(XENFEAT_auto_translated_physmap))
 		return 1;
 
-	return get_phys_to_machine(pfn) != INVALID_P2M_ENTRY;
+	return __pfn_to_mfn(pfn) != INVALID_P2M_ENTRY;
 }
 
 static inline unsigned long mfn_to_pfn_no_overrides(unsigned long mfn)
@@ -113,7 +130,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)
 		return mfn;
 
 	pfn = mfn_to_pfn_no_overrides(mfn);
-	if (get_phys_to_machine(pfn) != mfn) {
+	if (__pfn_to_mfn(pfn) != mfn) {
 		/*
 		 * If this appears to be a foreign mfn (because the pfn
 		 * doesn't map back to the mfn), then check the local override
@@ -130,7 +147,7 @@ static inline unsigned long mfn_to_pfn(unsigned long mfn)
 	 * valid entry for it.
 	 */
 	if (pfn == ~0 &&
-			get_phys_to_machine(mfn) == IDENTITY_FRAME(mfn))
+			__pfn_to_mfn(mfn) == IDENTITY_FRAME(mfn))
 		pfn = mfn;
 
 	return pfn;
@@ -176,7 +193,7 @@ static inline unsigned long mfn_to_local_pfn(unsigned long mfn)
 		return mfn;
 
 	pfn = mfn_to_pfn(mfn);
-	if (get_phys_to_machine(pfn) != mfn)
+	if (__pfn_to_mfn(pfn) != mfn)
 		return -1; /* force !pfn_valid() */
 	return pfn;
 }
arch/x86/xen/mmu.c +21 -13
index d3e492b..0b43c45 100644
@@ -387,7 +387,7 @@ static pteval_t pte_pfn_to_mfn(pteval_t val)
 		unsigned long mfn;
 
 		if (!xen_feature(XENFEAT_auto_translated_physmap))
-			mfn = get_phys_to_machine(pfn);
+			mfn = __pfn_to_mfn(pfn);
 		else
 			mfn = pfn;
 		/*
@@ -1158,20 +1158,16 @@ static void __init xen_cleanhighmap(unsigned long vaddr,
 	 * instead of somewhere later and be confusing. */
 	xen_mc_flush();
 }
-static void __init xen_pagetable_p2m_copy(void)
+
+static void __init xen_pagetable_p2m_free(void)
 {
 	unsigned long size;
 	unsigned long addr;
-	unsigned long new_mfn_list;
-
-	if (xen_feature(XENFEAT_auto_translated_physmap))
-		return;
 
 	size = PAGE_ALIGN(xen_start_info->nr_pages * sizeof(unsigned long));
 
-	new_mfn_list = xen_revector_p2m_tree();
 	/* No memory or already called. */
-	if (!new_mfn_list || new_mfn_list == xen_start_info->mfn_list)
+	if ((unsigned long)xen_p2m_addr == xen_start_info->mfn_list)
 		return;
 
 	/* using __ka address and sticking INVALID_P2M_ENTRY! */
@@ -1189,8 +1185,6 @@ static void __init xen_pagetable_p2m_copy(void)
 
 	size = PAGE_ALIGN(xen_start_info->nr_pages * sizeof(unsigned long));
 	memblock_free(__pa(xen_start_info->mfn_list), size);
-	/* And revector! Bye bye old array */
-	xen_start_info->mfn_list = new_mfn_list;
 
 	/* At this stage, cleanup_highmap has already cleaned __ka space
 	 * from _brk_limit way up to the max_pfn_mapped (which is the end of
@@ -1214,12 +1208,26 @@ static void __init xen_pagetable_p2m_copy(void)
 }
 #endif
 
-static void __init xen_pagetable_init(void)
+static void __init xen_pagetable_p2m_setup(void)
 {
-	paging_init();
+	if (xen_feature(XENFEAT_auto_translated_physmap))
+		return;
+
+	xen_vmalloc_p2m_tree();
+
 #ifdef CONFIG_X86_64
-	xen_pagetable_p2m_copy();
+	xen_pagetable_p2m_free();
 #endif
+	/* And revector! Bye bye old array */
+	xen_start_info->mfn_list = (unsigned long)xen_p2m_addr;
+}
+
+static void __init xen_pagetable_init(void)
+{
+	paging_init();
+
+	xen_pagetable_p2m_setup();
+
 	/* Allocate and initialize top and mid mfn levels for p2m structure */
 	xen_build_mfn_list_list();
 
arch/x86/xen/p2m.c +310 -426

985 lines not on this page · raw message

arch/x86/xen/xen-ops.h +1 -1
index 02b0b0f..f92921f 100644
@@ -49,7 +49,7 @@ void xen_hvm_init_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
-unsigned long __init xen_revector_p2m_tree(void);
+void __init xen_vmalloc_p2m_tree(void);
 
 void xen_init_irq_ops(void);
 void xen_setup_timer(int cpu);
Signature · 1 line
2.1.2
#11 lore raw

Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list

Quoted 15 lines · Juergen Gross

At start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.

Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.

Quoted 6 lines

Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:

Elapsed time: 32:50 ->  32:35
System:       18:07 ->  17:47
User:        104:00 -> 103:30

After implementing my suggestions below, please provided updated figure.
They should be better.

Quoted 1 line

Tested on 64 bit dom0 and 32 bit domU.

[...]

arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 62

Quoted 7 lines
arch/x86/include/asm/xen/page.h +1 -0
@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops,
 				     struct page **pages, unsigned int count);
 extern unsigned long m2p_find_override_pfn(unsigned long mfn, unsigned long pfn);
 
+static inline unsigned long __pfn_to_mfn(unsigned long pfn)

These variations of pfn_to_mfn() (__pfn_to_mfn() and get_phys_to_machine() and any others), need comments explaining their differences.

Can you add __pfn_to_mfn() and the docs in a separate patch?

arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 63

Quoted 1 line
arch/x86/include/asm/xen/page.h +1 -0
+	pr_notice("p2m virtual area at %p, size is %lx\n", vm.addr, vm.size);

pr_info().

Quoted 11 lines
arch/x86/include/asm/xen/page.h +2 -3
@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)
 		return IDENTITY_FRAME(pfn);
 	}
 
-	topidx = p2m_top_index(pfn);
-	mididx = p2m_mid_index(pfn);
-	idx = p2m_index(pfn);
+	ptep = lookup_address((unsigned long)(xen_p2m_addr + pfn), &level);
+	BUG_ON(!ptep || level != PG_LEVEL_4K);
 
 	/*

arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 425

Quoted 10 lines
arch/x86/include/asm/xen/page.h +2 -2
 	 * The INVALID_P2M_ENTRY is filled in both p2m_*identity
 	 * and in p2m_*missing, so returning the INVALID_P2M_ENTRY
 	 * would be wrong.
 	 */
-	if (p2m_top[topidx][mididx] == p2m_identity)
+	if (pte_pfn(*ptep) == PFN_DOWN(__pa(p2m_identity)))
 		return IDENTITY_FRAME(pfn);
 
-	return p2m_top[topidx][mididx][idx];
+	return xen_p2m_addr[pfn];

You should test xen_p2m_addr[pfn] == INVALID_P2M_ENTRY before checking if it's an identity entry. This should skip the more expensive lookup_address() in the common case.

arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 426

Quoted 1 line
arch/x86/include/asm/xen/page.h +0 -0
 bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)

I think you should map p2m_missing and p2m_identity as read-only and do the new page allocation on a write fault.

set_phys_to_machine() is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:

    xen_p2m_addr[pfn] = mfn;

David

author maintainer
#12 lore raw

Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list

Quoted 19 lines · David Vrabel
Quoted 18 lines · Juergen Gross

At start of the day the Xen hypervisor presents a contiguous mfn list to a pv-domain. In order to support sparse memory this mfn list is accessed via a three level p2m tree built early in the boot process. Whenever the system needs the mfn associated with a pfn this tree is used to find the mfn.

Instead of using a software walked tree for accessing a specific mfn list entry this patch is creating a virtual address area for the entire possible mfn list including memory holes. The holes are covered by mapping a pre-defined page consisting only of "invalid mfn" entries. Access to a mfn entry is possible by just using the virtual base address of the mfn list and the pfn as index into that list. This speeds up the (hot) path of determining the mfn of a pfn.

Kernel build on a Dell Latitude E6440 (2 cores, HT) in 64 bit Dom0 showed following improvements:

Quoted 6 lines
Quoted 3 lines
Elapsed time: 32:50 ->  32:35
System:       18:07 ->  17:47
User:        104:00 -> 103:30

After implementing my suggestions below, please provided updated figure.
They should be better.

In dom0? I don't think so.

Quoted 9 lines
Quoted 2 lines

Tested on 64 bit dom0 and 32 bit domU.

[...]

Quoted 6 lines
arch/x86/include/asm/xen/page.h +0 -0
@@ -59,6 +59,23 @@ extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops,
  				     struct page **pages, unsigned int count);
  extern unsigned long m2p_find_override_pfn(unsigned long mfn, unsigned long pfn);
 
Quoted 7 lines

arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 62

Quoted 1 line
arch/x86/include/asm/xen/page.h +1 -0
+static inline unsigned long __pfn_to_mfn(unsigned long pfn)

These variations of pfn_to_mfn() (__pfn_to_mfn() and get_phys_to_machine() and any others), need comments explaining their differences.

Can you add __pfn_to_mfn() and the docs in a separate patch?

Okay.

Quoted 24 lines

arch/x86/include/asm/xen/page.h extern int clear_foreign_p2m_mapping(struct gnttab_unmap_grant_ref *unmap_ops, · line 63

Quoted 2 lines
arch/x86/include/asm/xen/page.h +1 -0
+	pr_notice("p2m virtual area at %p, size is %lx\n", vm.addr, vm.size);

pr_info().

Quoted 19 lines
arch/x86/include/asm/xen/page.h +3 -4
@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long pfn)
  		return IDENTITY_FRAME(pfn);
  	}
 
-	topidx = p2m_top_index(pfn);
-	mididx = p2m_mid_index(pfn);
-	idx = p2m_index(pfn);
+	ptep = lookup_address((unsigned long)(xen_p2m_addr + pfn), &level);
+	BUG_ON(!ptep || level != PG_LEVEL_4K);
 
  	/*
  	 * The INVALID_P2M_ENTRY is filled in both p2m_*identity
  	 * and in p2m_*missing, so returning the INVALID_P2M_ENTRY
  	 * would be wrong.
  	 */
-	if (p2m_top[topidx][mididx] == p2m_identity)
+	if (pte_pfn(*ptep) == PFN_DOWN(__pa(p2m_identity)))
  		return IDENTITY_FRAME(pfn);
 
Quoted 6 lines

arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 425

Quoted 2 lines
arch/x86/include/asm/xen/page.h +1 -1
-	return p2m_top[topidx][mididx][idx];
+	return xen_p2m_addr[pfn];

You should test xen_p2m_addr[pfn] == INVALID_P2M_ENTRY before checking if it's an identity entry. This should skip the more expensive lookup_address() in the common case.

I do. The check is in __pfn_to_mfn(). get_phys_to_machine() is called in this case only.

Quoted 10 lines

arch/x86/include/asm/xen/page.h unsigned long get_phys_to_machine(unsigned long pfn) · line 426

Quoted 2 lines
arch/x86/include/asm/xen/page.h +0 -0
  bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)

I think you should map p2m_missing and p2m_identity as read-only and do the new page allocation on a write fault.

set_phys_to_machine() is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:

     xen_p2m_addr[pfn] = mfn;

Nice idea. I'll try it.

Juergen

#13 lore raw

Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list

Quoted 30 lines · Juergen Gross
Quoted 22 lines · David Vrabel

On 06/11/14 05:47, Juergen Gross wrote:
[...]

Quoted 20 lines
arch/x86/xen/p2m.c +3 -4
@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long
 pfn)
          return IDENTITY_FRAME(pfn);
      }
 
-    topidx = p2m_top_index(pfn);
-    mididx = p2m_mid_index(pfn);
-    idx = p2m_index(pfn);
+    ptep = lookup_address((unsigned long)(xen_p2m_addr + pfn), &level);
+    BUG_ON(!ptep || level != PG_LEVEL_4K);
 
      /*
       * The INVALID_P2M_ENTRY is filled in both p2m_*identity
       * and in p2m_*missing, so returning the INVALID_P2M_ENTRY
       * would be wrong.
       */
-    if (p2m_top[topidx][mididx] == p2m_identity)
+    if (pte_pfn(*ptep) == PFN_DOWN(__pa(p2m_identity)))
          return IDENTITY_FRAME(pfn);
 
Quoted 6 lines

arch/x86/xen/p2m.c unsigned long get_phys_to_machine(unsigned long pfn) · line 425

Quoted 2 lines
arch/x86/xen/p2m.c +1 -1
-    return p2m_top[topidx][mididx][idx];
+    return xen_p2m_addr[pfn];

You should test xen_p2m_addr[pfn] == INVALID_P2M_ENTRY before checking if it's an identity entry. This should skip the more expensive lookup_address() in the common case.

Quoted 2 lines

I do. The check is in __pfn_to_mfn(). get_phys_to_machine() is called in this case only.

So you do. I missed that.

Quoted 11 lines
Quoted 9 lines

arch/x86/xen/p2m.c unsigned long __init set_phys_range_identity(unsigned long pfn_s, · line 610

Quoted 1 line
arch/x86/xen/p2m.c +0 -0
  bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)

I think you should map p2m_missing and p2m_identity as read-only and do the new page allocation on a write fault.

set_phys_to_machine() is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:

     xen_p2m_addr[pfn] = mfn;

Nice idea. I'll try it.

You probably want to try this with a custom exception fixup (or abuse put_user()).

David

author maintainer
#14 lore raw

Re: [Xen-devel] [PATCH V2 5/5] Xen: switch to linear virtual mapped sparse p2m list

Quoted 44 lines · David Vrabel
Quoted 30 lines · Juergen Gross
Quoted 22 lines · David Vrabel

On 06/11/14 05:47, Juergen Gross wrote:
[...]

Quoted 20 lines
arch/x86/xen/p2m.c +3 -4
@@ -526,23 +411,83 @@ unsigned long get_phys_to_machine(unsigned long
 pfn)
           return IDENTITY_FRAME(pfn);
       }
 
-    topidx = p2m_top_index(pfn);
-    mididx = p2m_mid_index(pfn);
-    idx = p2m_index(pfn);
+    ptep = lookup_address((unsigned long)(xen_p2m_addr + pfn), &level);
+    BUG_ON(!ptep || level != PG_LEVEL_4K);
 
       /*
        * The INVALID_P2M_ENTRY is filled in both p2m_*identity
        * and in p2m_*missing, so returning the INVALID_P2M_ENTRY
        * would be wrong.
        */
-    if (p2m_top[topidx][mididx] == p2m_identity)
+    if (pte_pfn(*ptep) == PFN_DOWN(__pa(p2m_identity)))
           return IDENTITY_FRAME(pfn);
 
Quoted 6 lines

arch/x86/xen/p2m.c unsigned long get_phys_to_machine(unsigned long pfn) · line 425

Quoted 2 lines
arch/x86/xen/p2m.c +1 -1
-    return p2m_top[topidx][mididx][idx];
+    return xen_p2m_addr[pfn];

You should test xen_p2m_addr[pfn] == INVALID_P2M_ENTRY before checking if it's an identity entry. This should skip the more expensive lookup_address() in the common case.

Quoted 2 lines

I do. The check is in __pfn_to_mfn(). get_phys_to_machine() is called in this case only.

So you do. I missed that.

Quoted 8 lines
Quoted 8 lines

arch/x86/xen/p2m.c unsigned long __init set_phys_range_identity(unsigned long pfn_s, · line 610

Quoted 1 line
arch/x86/xen/p2m.c +0 -0
   bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn)

I think you should map p2m_missing and p2m_identity as read-only and do the new page allocation on a write fault.

set_phys_to_machine() is used every grant map and unmap and in the common case (already allocated array page) it must be a fast and simple:

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
      xen_p2m_addr[pfn] = mfn;

Nice idea. I'll try it.

You probably want to try this with a custom exception fixup (or abuse put_user()).

Yeah, that was my idea, too.

Juergen