Skip to contentSkip to conversation

xen/balloon: fix balloon driver initial target

PATCH In review Roger Pau Monne 2 participants 8 messages Jan 28

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover xen/balloon: fix balloon driver initial target

Hello,

The fist patch is a bugfix to revert to the previous way of setting the balloon memory target for PV domains. The new way is off by ~97 pages on domUs, because page at PFN 0 and the ISA range are ignored by Linux, but populated from Xen's perspective.

Second patch aims to improve the initial memory target used by dom0. With this new approach the target set by the balloon driver matches exactly the target Xen by the toolstack when late-initializing it.

Thanks, Roger.

Roger Pau Monne (2):
Partial revert "x86/xen: fix balloon target initialization for PVH

    dom0"

xen/balloon: improve accuracy of initial balloon target for dom0

 arch/x86/xen/enlighten.c        |  2 +-
 drivers/xen/balloon.c           | 26 ++++++++++++++++++++++----
 drivers/xen/unpopulated-alloc.c |  3 +++
 include/xen/xen.h               |  2 ++
 4 files changed, 28 insertions(+), 5 deletions(-)
Signature · 1 line
2.51.0
#1 lore raw

PATCH 1/2 Partial revert "x86/xen: fix balloon target initialization for PVH dom0"

This partially reverts commit 87af633689ce16ddb166c80f32b120e50b1295de so the current memory target for PV guests is still fetched from start_info->nr_pages, which matches exactly what the toolstack sets the initial memory target to.

Using get_num_physpages() is possible on PV also, but needs adjusting to take into account the ISA hole and the PFN at 0 not considered usable memory despite being populated, and hence would need extra adjustments. Instead of carrying those extra adjustments switch back to the previous code. That leaves Linux with a difference in how current memory target is obtained for HVM vs PV, but that's better than adding extra logic just for PV.

However if switching to start_info->nr_pages for PV domains we need to differentiate between released pages (freed back to the hypervisor) as opposed to pages in the physmap which are not populated to start with. Introduce a new xen_unpopulated_pages to account for papges that have never been populated, and hence in the PV case don't need subtracting.

Fixes 87af633689ce ("x86/xen: fix balloon target initialization for PVH dom0")
---
 arch/x86/xen/enlighten.c        |  2 +-
 drivers/xen/balloon.c           | 19 +++++++++++++++----
 drivers/xen/unpopulated-alloc.c |  3 +++
 include/xen/xen.h               |  2 ++
 4 files changed, 21 insertions(+), 5 deletions(-)
arch/x86/xen/enlighten.c +1 -1
index 53282dc7d5ac..23b91bf9b663 100644
@@ -470,7 +470,7 @@ int __init arch_xen_unpopulated_init(struct resource **res)
 		 * driver to know how much of the physmap is unpopulated and
 		 * set an accurate initial memory target.
 		 */
-		xen_released_pages += xen_extra_mem[i].n_pfns;
+		xen_unpopulated_pages += xen_extra_mem[i].n_pfns;
 		/* Zero so region is not also added to the balloon driver. */
 		xen_extra_mem[i].n_pfns = 0;
 	}
drivers/xen/balloon.c +15 -4
index 49c3f9926394..8c44a25a7d2b 100644
@@ -724,6 +724,7 @@ static int __init balloon_add_regions(void)
 static int __init balloon_init(void)
 {
 	struct task_struct *task;
+	unsigned long current_pages;
 	int rc;
 
 	if (!xen_domain())
@@ -731,12 +732,18 @@ static int __init balloon_init(void)
 
 	pr_info("Initialising balloon driver\n");
 
-	if (xen_released_pages >= get_num_physpages()) {
-		WARN(1, "Released pages underflow current target");
-		return -ERANGE;
+	if (xen_pv_domain()) {
+		if (xen_released_pages >= xen_start_info->nr_pages)
+			goto underflow;
+		current_pages = min(xen_start_info->nr_pages -
+		                    xen_released_pages, max_pfn);
+	} else {
+		if (xen_unpopulated_pages >= get_num_physpages())
+			goto underflow;
+		current_pages = get_num_physpages() - xen_unpopulated_pages;
 	}
 
-	balloon_stats.current_pages = get_num_physpages() - xen_released_pages;
+	balloon_stats.current_pages = current_pages;
 	balloon_stats.target_pages  = balloon_stats.current_pages;
 	balloon_stats.balloon_low   = 0;
 	balloon_stats.balloon_high  = 0;
@@ -767,6 +774,10 @@ static int __init balloon_init(void)
 	xen_balloon_init();
 
 	return 0;
+
+ underflow:
+	WARN(1, "Released pages underflow current target");
+	return -ERANGE;
 }
 subsys_initcall(balloon_init);
 
drivers/xen/unpopulated-alloc.c +3 -0
index d6fc2aefe264..1dc0b495c8e5 100644
@@ -18,6 +18,9 @@ static unsigned int list_count;
 
 static struct resource *target_resource;
 
+/* Pages to subtract from the memory count when setting balloon target. */
+unsigned long xen_unpopulated_pages __initdata;
+
 /*
  * If arch is not happy with system "iomem_resource" being used for
  * the region allocation it can provide it's own view by creating specific
include/xen/xen.h +2 -0
index 61854e3f2837..f280c5dcf923 100644
@@ -69,11 +69,13 @@ extern u64 xen_saved_max_mem_size;
 #endif
 
 #ifdef CONFIG_XEN_UNPOPULATED_ALLOC
+extern unsigned long xen_unpopulated_pages;
 int xen_alloc_unpopulated_pages(unsigned int nr_pages, struct page **pages);
 void xen_free_unpopulated_pages(unsigned int nr_pages, struct page **pages);
 #include <linux/ioport.h>
 int arch_xen_unpopulated_init(struct resource **res);
 #else
+#define xen_unpopulated_pages 0UL
 #include <xen/balloon.h>
 static inline int xen_alloc_unpopulated_pages(unsigned int nr_pages,
 		struct page **pages)
Signature · 1 line
2.51.0
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] Partial revert "x86/xen: fix balloon target initialization for PVH dom0"

Quoted 19 lines · Roger Pau Monne

This partially reverts commit 87af633689ce16ddb166c80f32b120e50b1295de so the current memory target for PV guests is still fetched from start_info->nr_pages, which matches exactly what the toolstack sets the initial memory target to.

Using get_num_physpages() is possible on PV also, but needs adjusting to take into account the ISA hole and the PFN at 0 not considered usable memory despite being populated, and hence would need extra adjustments. Instead of carrying those extra adjustments switch back to the previous code. That leaves Linux with a difference in how current memory target is obtained for HVM vs PV, but that's better than adding extra logic just for PV.

However if switching to start_info->nr_pages for PV domains we need to differentiate between released pages (freed back to the hypervisor) as opposed to pages in the physmap which are not populated to start with. Introduce a new xen_unpopulated_pages to account for papges that have never been populated, and hence in the PV case don't need subtracting.

Quoted 3 lines
Fixes 87af633689ce ("x86/xen: fix balloon target initialization for PVH dom0")

Juergen

#3 lore raw

PATCH 2/2 xen/balloon: improve accuracy of initial balloon target for dom0

In reply to Roger Pau Monne

The dom0 balloon target set by the toolstack is the value returned by XENMEM_current_reservation. Do the same in the kernel balloon driver and set the current allocation to the value returned by XENMEM_current_reservation. On my test system this causes the kernel balloon driver target to exactly match the value set by the toolstack in xenstore.

Note this approach can be used by both PV and PVH dom0s, as the toolstack always uses XENMEM_current_reservation to set the initial target regardless of the dom0 type.

---
 drivers/xen/balloon.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)
drivers/xen/balloon.c +17 -10
index 8c44a25a7d2b..9b6531eb28b6 100644
@@ -724,7 +724,8 @@ static int __init balloon_add_regions(void)
 static int __init balloon_init(void)
 {
 	struct task_struct *task;
-	unsigned long current_pages;
+	long current_pages = 0;
+	domid_t domid = DOMID_SELF;
 	int rc;
 
 	if (!xen_domain())
@@ -732,15 +733,21 @@ static int __init balloon_init(void)
 
 	pr_info("Initialising balloon driver\n");
 
-	if (xen_pv_domain()) {
-		if (xen_released_pages >= xen_start_info->nr_pages)
-			goto underflow;
-		current_pages = min(xen_start_info->nr_pages -
-		                    xen_released_pages, max_pfn);
-	} else {
-		if (xen_unpopulated_pages >= get_num_physpages())
-			goto underflow;
-		current_pages = get_num_physpages() - xen_unpopulated_pages;
+	if (xen_initial_domain())
+		current_pages = HYPERVISOR_memory_op(XENMEM_current_reservation,
+		                                     &domid);
+	if (current_pages <= 0) {
+		if (xen_pv_domain()) {
+			if (xen_released_pages >= xen_start_info->nr_pages)
+				goto underflow;
+			current_pages = min(xen_start_info->nr_pages -
+			                    xen_released_pages, max_pfn);
+		} else {
+			if (xen_unpopulated_pages >= get_num_physpages())
+				goto underflow;
+			current_pages = get_num_physpages() -
+			                xen_unpopulated_pages;
+		}
 	}
 
 	balloon_stats.current_pages = current_pages;
Signature · 1 line
2.51.0
maintainer
#4 lore raw

Re: [PATCH 2/2] xen/balloon: improve accuracy of initial balloon target for dom0

Quoted 36 lines · Roger Pau Monne

The dom0 balloon target set by the toolstack is the value returned by XENMEM_current_reservation. Do the same in the kernel balloon driver and set the current allocation to the value returned by XENMEM_current_reservation. On my test system this causes the kernel balloon driver target to exactly match the value set by the toolstack in xenstore.

Note this approach can be used by both PV and PVH dom0s, as the toolstack always uses XENMEM_current_reservation to set the initial target regardless of the dom0 type.

---
  drivers/xen/balloon.c | 27 +++++++++++++++++----------
  1 file changed, 17 insertions(+), 10 deletions(-)
drivers/xen/balloon.c +2 -3
index 8c44a25a7d2b..9b6531eb28b6 100644
@@ -724,7 +724,8 @@ static int __init balloon_add_regions(void)
  static int __init balloon_init(void)
  {
  	struct task_struct *task;
-	unsigned long current_pages;
+	long current_pages = 0;
+	domid_t domid = DOMID_SELF;
  	int rc;
 
  	if (!xen_domain())
@@ -732,15 +733,21 @@ static int __init balloon_init(void)
 
  	pr_info("Initialising balloon driver\n");
 
-	if (xen_pv_domain()) {
-		if (xen_released_pages >= xen_start_info->nr_pages)

drivers/xen/balloon.c static int __init balloon_init(void) · line 738

Quoted 10 lines
drivers/xen/balloon.c +3 -7
-			goto underflow;
-		current_pages = min(xen_start_info->nr_pages -
-		                    xen_released_pages, max_pfn);
-	} else {
-		if (xen_unpopulated_pages >= get_num_physpages())
-			goto underflow;
-		current_pages = get_num_physpages() - xen_unpopulated_pages;
+	if (xen_initial_domain())
+		current_pages = HYPERVISOR_memory_op(XENMEM_current_reservation,
+		                                     &domid);

Is there any specific reason why this should be limited to dom0?

I _think_ this should work for other domains, too.

Juergen

Quoted 15 lines
drivers/xen/balloon.c +12 -0
+	if (current_pages <= 0) {
+		if (xen_pv_domain()) {
+			if (xen_released_pages >= xen_start_info->nr_pages)
+				goto underflow;
+			current_pages = min(xen_start_info->nr_pages -
+			                    xen_released_pages, max_pfn);
+		} else {
+			if (xen_unpopulated_pages >= get_num_physpages())
+				goto underflow;
+			current_pages = get_num_physpages() -
+			                xen_unpopulated_pages;
+		}
  	}
 
  	balloon_stats.current_pages = current_pages;
#5 lore raw

Re: [PATCH 2/2] xen/balloon: improve accuracy of initial balloon target for dom0

Quoted 47 lines · Jürgen Groß
Quoted 33 lines · Roger Pau Monne

The dom0 balloon target set by the toolstack is the value returned by XENMEM_current_reservation. Do the same in the kernel balloon driver and set the current allocation to the value returned by XENMEM_current_reservation. On my test system this causes the kernel balloon driver target to exactly match the value set by the toolstack in xenstore.

Note this approach can be used by both PV and PVH dom0s, as the toolstack always uses XENMEM_current_reservation to set the initial target regardless of the dom0 type.

---
  drivers/xen/balloon.c | 27 +++++++++++++++++----------
  1 file changed, 17 insertions(+), 10 deletions(-)
drivers/xen/balloon.c +2 -3
index 8c44a25a7d2b..9b6531eb28b6 100644
@@ -724,7 +724,8 @@ static int __init balloon_add_regions(void)
  static int __init balloon_init(void)
  {
  	struct task_struct *task;
-	unsigned long current_pages;
+	long current_pages = 0;
+	domid_t domid = DOMID_SELF;
  	int rc;
  	if (!xen_domain())
@@ -732,15 +733,21 @@ static int __init balloon_init(void)
  	pr_info("Initialising balloon driver\n");
-	if (xen_pv_domain()) {
-		if (xen_released_pages >= xen_start_info->nr_pages)

drivers/xen/balloon.c static int __init balloon_init(void) · line 736

Quoted 10 lines
drivers/xen/balloon.c +3 -7
-			goto underflow;
-		current_pages = min(xen_start_info->nr_pages -
-		                    xen_released_pages, max_pfn);
-	} else {
-		if (xen_unpopulated_pages >= get_num_physpages())
-			goto underflow;
-		current_pages = get_num_physpages() - xen_unpopulated_pages;
+	if (xen_initial_domain())
+		current_pages = HYPERVISOR_memory_op(XENMEM_current_reservation,
+		                                     &domid);

Is there any specific reason why this should be limited to dom0?

Quoted 1 line

I _think_ this should work for other domains, too.

Sadly it doesn't, I've already tested. The value returned by XENMEM_current_reservation on PV guests is slightly different than what's in xen_start_info->nr_pages, which exactly what the toolstack writes in xenstore. I assume there's some other stuff that's accounted for in d->tot_pages, but don't really know what I'm afraid.

And in the HVM/PVH case using XENMEM_current_reservation for domUs would also take into account the Video memory, which skews the target.

This is the best I could do I'm afraid, at the expense of having so many different way to fetch the information.

Thanks, Roger.

maintainer
#6 lore raw

Re: [PATCH 2/2] xen/balloon: improve accuracy of initial balloon target for dom0

Quoted 56 lines · Roger Pau Monné
Quoted 47 lines · Jürgen Groß
Quoted 33 lines · Roger Pau Monne

The dom0 balloon target set by the toolstack is the value returned by XENMEM_current_reservation. Do the same in the kernel balloon driver and set the current allocation to the value returned by XENMEM_current_reservation. On my test system this causes the kernel balloon driver target to exactly match the value set by the toolstack in xenstore.

Note this approach can be used by both PV and PVH dom0s, as the toolstack always uses XENMEM_current_reservation to set the initial target regardless of the dom0 type.

---
   drivers/xen/balloon.c | 27 +++++++++++++++++----------
   1 file changed, 17 insertions(+), 10 deletions(-)
drivers/xen/balloon.c +2 -3
index 8c44a25a7d2b..9b6531eb28b6 100644
@@ -724,7 +724,8 @@ static int __init balloon_add_regions(void)
   static int __init balloon_init(void)
   {
   	struct task_struct *task;
-	unsigned long current_pages;
+	long current_pages = 0;
+	domid_t domid = DOMID_SELF;
   	int rc;
   	if (!xen_domain())
@@ -732,15 +733,21 @@ static int __init balloon_init(void)
   	pr_info("Initialising balloon driver\n");
-	if (xen_pv_domain()) {
-		if (xen_released_pages >= xen_start_info->nr_pages)

drivers/xen/balloon.c static int __init balloon_init(void) · line 736

Quoted 10 lines
drivers/xen/balloon.c +3 -7
-			goto underflow;
-		current_pages = min(xen_start_info->nr_pages -
-		                    xen_released_pages, max_pfn);
-	} else {
-		if (xen_unpopulated_pages >= get_num_physpages())
-			goto underflow;
-		current_pages = get_num_physpages() - xen_unpopulated_pages;
+	if (xen_initial_domain())
+		current_pages = HYPERVISOR_memory_op(XENMEM_current_reservation,
+		                                     &domid);

Is there any specific reason why this should be limited to dom0?

Quoted 1 line

I _think_ this should work for other domains, too.

Sadly it doesn't, I've already tested. The value returned by XENMEM_current_reservation on PV guests is slightly different than what's in xen_start_info->nr_pages, which exactly what the toolstack writes in xenstore. I assume there's some other stuff that's accounted for in d->tot_pages, but don't really know what I'm afraid.

Quoted 5 lines

And in the HVM/PVH case using XENMEM_current_reservation for domUs would also take into account the Video memory, which skews the target.

This is the best I could do I'm afraid, at the expense of having so many different way to fetch the information.

Meh, too bad.

For now I think we need to live with that. But for the future I'd like to suggest to add a new Xenstore node memory/reservation-diff containing the difference between XENMEM_current_reservation output and the tools' view of the domain's memory size.

The balloon driver could read that node in balloon_init() and use XENMEM_current_reservation in case the node is found.

Juergen

maintainer Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] xen/balloon: improve accuracy of initial balloon target for dom0

In reply to Roger Pau Monne
Quoted 7 lines · Roger Pau Monne

The dom0 balloon target set by the toolstack is the value returned by XENMEM_current_reservation. Do the same in the kernel balloon driver and set the current allocation to the value returned by XENMEM_current_reservation. On my test system this causes the kernel balloon driver target to exactly match the value set by the toolstack in xenstore.

Quoted 5 lines

Note this approach can be used by both PV and PVH dom0s, as the toolstack always uses XENMEM_current_reservation to set the initial target regardless of the dom0 type.

Juergen