x86/xen: fix init of balloon stats for PV guests with memory != maxmem
Subsystems (2)
PATCH 1/1 x86/xen: fix init of balloon stats for PV guests with memory != maxmem
The handling of extra memory regions done in balloon_add_regions() is not correct for PV guests, since the initial target is set to reflect the real memory the system has, not what's described on the memory map, which can be higher if memory != maxmem.
Introduce separate logic for PV vs HVM in balloon_add_regions() and handle the extra region correctly by adding them to the total amount of pages, instead of subtracting from the current and target pages amounts.
---
Cc: Yannick Martin <[email protected]>
Cc: "Thorsten Leemhuis" <[email protected]>
--- drivers/xen/balloon.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
drivers/xen/balloon.c +15 -8
@@ -703,19 +703,26 @@ static int __init balloon_add_regions(void)balloon_append (pfn_to_page (pfn));/* -* Extra regions are accounted for in the physmap, but need -* decreasing from current_pages and target_pages to balloon -* down the initial allocation, because they are already -* accounted for in total_pages. +* For HVM domains: extra regions are accounted for in the +* physmap, but need decreasing from current_pages and +* target_pages to balloon down the initial allocation, because +* they are already accounted for in total_pages. +* +* For PV domains: extra regions are not accounted for in the +* initial memory target, and hence need adding to the stats as +* additional unpopulated regions. */ pages = extra_pfn_end - start_pfn; -if (pages >= balloon_stats.current_pages || - pages >= balloon_stats.target_pages) { +if (xen_pv_domain ()) { + balloon_stats.total_pages += pages; + }else if (pages >= balloon_stats.current_pages || + pages >= balloon_stats.target_pages) {WARN (1 ,"Extra pages underflow current target" );return -ERANGE ; + }else { + balloon_stats.current_pages -= pages; + balloon_stats.target_pages -= pages; } - balloon_stats.current_pages -= pages; - balloon_stats.target_pages -= pages; }return 0 ;
Signature · 1 line
2.53.0
Quoted 11 lines · Roger Pau Monne
The handling of extra memory regions done in
balloon_add_regions()is not correct for PV guests, since the initial target is set to reflect the real memory the system has, not what's described on the memory map, which can be higher if memory != maxmem.Introduce separate logic for PV vs HVM in
balloon_add_regions()and handle the extra region correctly by adding them to the total amount of pages, instead of subtracting from the current and target pages amounts.Fixes 87af633689ce ("x86/xen: fix balloon target initialization for PVH dom0")Signed-off-by Roger Pau Monné <[email protected]>
Juergen
Hi Roger,
thanks for picking this up, and Juergen, thanks for the quick review. Two things I believe are still worth addressing; the Fixes: tag can of course also be fixed up on application.
I think the Fixes: tag should point to 0949c646d646 ("Partial revert \"x86/xen: fix balloon target initialization for PVH dom0\""). Commit 87af633689ce changed the initial-page calculation and the extra-region subtraction together, so those two operations were coherent: the PV initial count then came from get_num_physpages(), which includes the extra regions. 0949c646d646 restored the PV start_info->nr_pages calculation, which excludes the extra regions, but retained the subtraction. Its 6.12.y backport is also the reporter's identified regression, first seen in 6.12.75. Applying this patch in a tree that has 87af633689ce but not 0949c646d646 (for example a 6.17-based distro tree) would double-account the extra region. This likely also wants Cc: [email protected], since both 6.12.y and 6.18.y carry the 0949c646d646 regression.
Separately, and not something this patch introduces: PVH dom0 has the same shape of problem on mainline since b13cd24c15d7. A successful XENMEM_current_reservation supplies current_pages for both PV and PVH dom0, and that count excludes the unpopulated xen_extra_mem, so the xen_pv_domain()-only branch leaves PVH dom0 subtracting those pages again (-ERANGE, or a silently wrong target, when CONFIG_XEN_UNPOPULATED_ALLOC=n leaves the regions for the balloon driver). I am happy to pursue that as its own thread once this one lands.
Would it be safer to pass balloon_add_regions() an explicit indication of whether the chosen initial-page count includes the extra physmap regions? That would cover PV, PVH dom0, and the XENMEM_current_reservation fallback without deriving the accounting rule solely from the domain type. On hypercall failure PVH dom0 falls back to get_num_physpages(), which includes the extra regions, so keying the accounting on the source of the count keeps the fallback correct as well.
Thanks,
Matthias
Quoted 6 lines · Matthias Goergens
Hi Roger,
thanks for picking this up, and Juergen, thanks for the quick review. Two things I believe are still worth addressing; the Fixes: tag can of course also be fixed up on application.
Quoted 7 lines
I think the Fixes: tag should point to
0949c646d646("Partial revert \"x86/xen: fix balloon target initialization for PVH dom0\""). Commit87af633689cechanged the initial-page calculation and the extra-region subtraction together, so those two operations were coherent: the PV initial count then came fromget_num_physpages(), which includes the extra regions.0949c646d646restored the PVstart_info->nr_pagescalculation, which excludes the extra regions, but retained the subtraction.
I've got the same doubts about which commit to reference in the Fixes tag. Here is my reasoning for picking the original bogus commit, and not the subsequent attempt at fixing it:
Even if 87af633689ce was coherent in the usage of initial pages vs extra regions, it was still wrong, and that's why it was (partially) reverted. I assume that anyone who picks the change in this patch will also have picked 0949c646d646, otherwise they have a problem with how they do backports.
Quoted 5 lines
Its 6.12.y
backport is also the reporter's identified regression, first seen in 6.12.75. Applying this patch in a tree that has87af633689cebut not0949c646d646(for example a 6.17-based distro tree) would double-account the extra region.
Why would someone apply this fix but not the preceding one? It makes no sense, you either pick backports consistently, or need to be very careful at knowing what to pick (and assume that sometimes stuff will break).
Quoted 11 lines
This likely also wants Cc: [email protected], since both 6.12.y and 6.18.y carry the
0949c646d646regression.Separately, and not something this patch introduces: PVH dom0 has the same shape of problem on mainline since
b13cd24c15d7. A successfulXENMEM_current_reservationsuppliescurrent_pagesfor both PV and PVH dom0, and that count excludes the unpopulatedxen_extra_mem, so the xen_pv_domain()-only branch leaves PVH dom0 subtracting those pages again (-ERANGE, or a silently wrong target, whenCONFIG_XEN_UNPOPULATED_ALLOC=nleaves the regions for the balloon driver). I am happy to pursue that as its own thread once this one lands.
Hm, I see. Running a PVH dom0 without CONFIG_XEN_UNPOPULATED_ALLOC will be a very bad idea anyway, as the kernel would likely end up triggering an OOM as all pages would be ballooned out to create grant/foreign mappings.
Quoted 7 lines
Would it be safer to pass
balloon_add_regions()an explicit indication of whether the chosen initial-page count includes the extra physmap regions? That would cover PV, PVH dom0, and theXENMEM_current_reservationfallback without deriving the accounting rule solely from the domain type. On hypercall failure PVH dom0 falls back toget_num_physpages(), which includes the extra regions, so keying the accounting on the source of the count keeps the fallback correct as well.
Possibly, this has grown organically to accommodate for the lack of proper interface to do memory balloon accounting.
I will send v2 attempting to take care of the PVH corner case and the error fallback. It's IMO best if we can get all the related fixes here in a single patch to backport.
Thanks, Roger.
PATCH v2 1/1 x86/xen: fix init of balloon stats again
The handling of extra memory regions done in balloon_add_regions() is not correct for PV guests, since the initial target is set to reflect the real memory the system has, not what's described on the memory map, which can be higher if memory != maxmem.
Introduce separate logic for addition vs subtraction in balloon_add_regions() and handle extra regions correctly by adding them to the total amount of pages, instead of subtracting from the current and target pages amounts.
In the common case PV domU/dom0 and PVH dom0 will use the addition path, since the initial target reflects the real assigned memory. HVM and PVH domUs use the subtraction path, since the target is set based on the amount of memory reported in the memory map, without accounting for released regions.
---
Cc: Yannick Martin <[email protected]>
Cc: "Thorsten Leemhuis" <[email protected]>
Cc: Matthias Goergens <[email protected]>
---
Changes since v1:
- Also fix PVH dom0 without unpopulated pages support.
- Account for XENMEM_current_reservation possibly failing.
--- drivers/xen/balloon.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
drivers/xen/balloon.c +19 -10
@@ -679,7 +679,7 @@ void xen_free_ballooned_pages(unsigned int nr_pages, struct page **pages) }EXPORT_SYMBOL (xen_free_ballooned_pages); -static int __init balloon_add_regions (void ) +static int __init balloon_add_regions (bool append) {unsigned long start_pfn, pages;unsigned long pfn, extra_pfn_end;
@@ -703,19 +703,26 @@ static int __init balloon_add_regions(void)balloon_append (pfn_to_page (pfn));/* -* Extra regions are accounted for in the physmap, but need -* decreasing from current_pages and target_pages to balloon -* down the initial allocation, because they are already -* accounted for in total_pages. +* There are two different use-cases depending on how the +* initial memory target is fetched. For PVH dom0 and PV the +* target is usually set to reflect the domain assigned memory, +* and hence extra regions need adding. +* +* OTOH for HVM and PVH domU the target is set to the amount of +* RAM reported in the memory map, and hence extra regions need +* subtracting to reflect the real memory usage. */ pages = extra_pfn_end - start_pfn; -if (pages >= balloon_stats.current_pages || - pages >= balloon_stats.target_pages) { +if (append) { + balloon_stats.total_pages += pages; + }else if (pages >= balloon_stats.current_pages || + pages >= balloon_stats.target_pages) {WARN (1 ,"Extra pages underflow current target" );return -ERANGE ; + }else { + balloon_stats.current_pages -= pages; + balloon_stats.target_pages -= pages; } - balloon_stats.current_pages -= pages; - balloon_stats.target_pages -= pages; }return 0 ;
@@ -726,6 +733,7 @@ static int __init balloon_init(void)struct task_struct *task;long current_pages =0 ;domid_t domid =DOMID_SELF ; +bool append = true;int rc;if (!xen_domain ())
@@ -745,6 +753,7 @@ static int __init balloon_init(void) }else {if (xen_unpopulated_pages >=get_num_physpages ()) goto underflow; + append = false; current_pages =get_num_physpages () - xen_unpopulated_pages; }
@@ -767,7 +776,7 @@ static int __init balloon_init(void)register_sysctl_init ("xen/balloon" , balloon_table);#endif - rc =balloon_add_regions (); + rc =balloon_add_regions (append);if (rc)return rc;
Signature · 1 line
2.53.0
Re: [PATCH v2] x86/xen: fix init of balloon stats again
Hi Roger,
v2 looks good to me. Keying append on the source of the initial count covers every case I can construct: by inspection, a domU always enters the fallback branch (current_pages stays 0), so HVM/PVH domU and the dom0 hypercall-failure path share the subtraction branch, while the PV start_info path and a successful dom0 XENMEM_current_reservation append.
I also ran it on a nested-KVM Xen rig (Xen 4.23-unstable, Linux 11028ab62899e as dom0, static busybox initramfs):
- PV dom0, dom0_mem=2048M,max:4096M and 3072M,max:4096M, CONFIG_XEN_UNPOPULATED_ALLOC=n: no WARN, current_kb matches dom0_mem (2 and 3 GiB respectively). Same with =y.
- PVH dom0, same two memory configurations, =n: v1 WARNed in balloon_init and returned -ERANGE in exactly these cases; v2 completes cleanly and the balloon driver initialises.
(Scope note: PVH dom0 userspace stalls later in boot under nested KVM for an unrelated reason, so the PVH evidence is boot-time dmesg and the balloon sysfs state.)
Thanks,
Matthias
Re: [PATCH v2] x86/xen: fix init of balloon stats again
In reply to Roger Pau MonneQuoted 16 lines · Roger Pau Monne
The handling of extra memory regions done in
balloon_add_regions()is not correct for PV guests, since the initial target is set to reflect the real memory the system has, not what's described on the memory map, which can be higher if memory != maxmem.Introduce separate logic for addition vs subtraction in
balloon_add_regions()and handle extra regions correctly by adding them to the total amount of pages, instead of subtracting from the current and target pages amounts.In the common case PV domU/dom0 and PVH dom0 will use the addition path, since the initial target reflects the real assigned memory. HVM and PVH domUs use the subtraction path, since the target is set based on the amount of memory reported in the memory map, without accounting for released regions.
Quoted 3 lines
Fixes 87af633689ce ("x86/xen: fix balloon target initialization for PVH dom0")Fixes 0949c646d646 ("Partial revert "x86/xen: fix balloon target initialization for PVH dom0"")Signed-off-by Roger Pau Monné <[email protected]>
Juergen