xen/balloon: Enforce various limits on target
PATCH 1/1 xen/balloon: Enforce various limits on target
This patch enforces on target limit statically defined in Linux Kernel source and limit defined by hypervisor or host.
Particularly this patch fixes bug which led to flood of dom0 kernel log with messages similar to:
System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be addedxen_balloon: reserve_additional_memory: add_memory() failed: -17
It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
--- drivers/xen/balloon.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)
drivers/xen/balloon.c +46 -1
@@ -65,6 +65,7 @@#include <xen/balloon.h> #include <xen/features.h> #include <xen/page.h> +#include <xen/xenbus.h> /* * balloon_process() state:
@@ -490,11 +491,55 @@ static void balloon_process(struct work_struct *work)mutex_unlock (&balloon_mutex); } -/* Resets the Xen limit, sets new target, and kicks off processing. */ +/* Enforce limits, set new target and kick off processing. */ void balloon_set_new_target (unsigned long target) { +domid_t domid =DOMID_SELF ; +int rc; +unsigned long long host_limit; + +/* Enforce statically defined limit. */ + target =min (target,MAX_DOMAIN_PAGES ); + +if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */ +if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { +pr_info ("xen_balloon: %s: Initial domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + + host_limit = rc; + }else { + rc =xenbus_scanf (XBT_NIL ,"memory" ,"static-max" , +"%llu" , &host_limit); + +if (rc !=1 ) { +pr_info ("xen_balloon: %s: Guest domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + +/* +* The given memory target limit value is in KiB, so it needs +* converting to pages. PAGE_SHIFT converts bytes to pages, +* hence PAGE_SHIFT - 10. +*/ + host_limit >>= (PAGE_SHIFT -10 ); + } + +/* Enforce hypervisor/host defined limit. */ + target =min (target, (unsigned long )host_limit); + +no_host_limit:/* No need for lock. Not read-modify-write updates. */ balloon_stats.target_pages = target; +schedule_delayed_work (&balloon_worker,0 ); }EXPORT_SYMBOL_GPL (balloon_set_new_target);
Signature · 1 line
1.7.10.4
Quoted 6 lines · Daniel Kiper
This patch enforces on target limit statically defined in Linux Kernel source and limit defined by hypervisor or host.
Particularly this patch fixes bug which led to flood of dom0 kernel log with messages similar to:
Quoted 7 lines
System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be added
xen_balloon:reserve_additional_memory:add_memory()failed: -17It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
Signed-off-by Daniel Kiper <[email protected]>
One nitpick below; David, could you take a look just for a extra set of eyes.
Quoted 31 lines
--- drivers/xen/balloon.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)drivers/xen/balloon.c +9
-1@@ -65,6 +65,7 @@#include <xen/balloon.h> #include <xen/features.h> #include <xen/page.h> +#include <xen/xenbus.h> /* * balloon_process() state: @@ -490,11 +491,55 @@ static void balloon_process(struct work_struct *work)mutex_unlock (&balloon_mutex); } -/* Resets the Xen limit, sets new target, and kicks off processing. */ +/* Enforce limits, set new target and kick off processing. */ void balloon_set_new_target (unsigned long target) { +domid_t domid =DOMID_SELF ; +int rc; +unsigned long long host_limit; + +/* Enforce statically defined limit. */ + target =min (target,MAX_DOMAIN_PAGES ); +
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 513
Quoted 10 lines
drivers/xen/balloon.c +10
-0+if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */ +if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { +pr_info ("xen_balloon: %s: Initial domain target limit " +"could not be established: %i\n" , __func__, rc);
Probably pr_debug. As this means the user booted without dom0_mem_max argument. (B/c rc == UINT_MAX) I would say if you can check for that and it as part of if (rc == -EPERM) check.
Quoted 35 lines
drivers/xen/balloon.c +27
-0+ goto no_host_limit; + } + + host_limit = rc; + }else { + rc =xenbus_scanf (XBT_NIL ,"memory" ,"static-max" , +"%llu" , &host_limit); + +if (rc !=1 ) { +pr_info ("xen_balloon: %s: Guest domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + +/* +* The given memory target limit value is in KiB, so it needs +* converting to pages. PAGE_SHIFT converts bytes to pages, +* hence PAGE_SHIFT - 10. +*/ + host_limit >>= (PAGE_SHIFT -10 ); + } + +/* Enforce hypervisor/host defined limit. */ + target =min (target, (unsigned long )host_limit); + +no_host_limit:/* No need for lock. Not read-modify-write updates. */ balloon_stats.target_pages = target; +schedule_delayed_work (&balloon_worker,0 ); }EXPORT_SYMBOL_GPL (balloon_set_new_target);Signature · 1 line
1.7.10.4
Quoted 8 lines · Daniel Kiper
This patch enforces on target limit statically defined in Linux Kernel source and limit defined by hypervisor or host.
Particularly this patch fixes bug which led to flood of dom0 kernel log with messages similar to:
System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be added
xen_balloon:reserve_additional_memory:add_memory()failed: -17
I think this helps in some cases, but because reserve_additional_memory() places the hotplugged memory after max_pfn without checking if there is anything already there, there are still ways it can repeatedly fail.
e.g.,
1. If dom0 has had its maximum reservation limited initially (with the dom0_mem option) /and/ the max reservation and target is subsequently raised then the balloon driver will attempt to hotplug memory that overlaps with E820_UNUSABLE regions in the e820 map and the hotplug will fail.
2. If a domU has passed-through PCI devices max_pfn is before the PCI memory window then the balloon driver will attempt to hotplug memory over the PCI device BARs.
I think reserve_additional_memory() should check the current resource map and the e820 map to find a large enough unused region. This can be done as an additional patch at a later date.
Quoted 2 lines
It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
This sentence confuses me.
Quoted 36 lines
Signed-off-by Daniel Kiper <[email protected]>--- drivers/xen/balloon.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)drivers/xen/balloon.c +13
-1@@ -65,6 +65,7 @@#include <xen/balloon.h> #include <xen/features.h> #include <xen/page.h> +#include <xen/xenbus.h> /* * balloon_process() state: @@ -490,11 +491,55 @@ static void balloon_process(struct work_struct *work)mutex_unlock (&balloon_mutex); } -/* Resets the Xen limit, sets new target, and kicks off processing. */ +/* Enforce limits, set new target and kick off processing. */ void balloon_set_new_target (unsigned long target) { +domid_t domid =DOMID_SELF ; +int rc; +unsigned long long host_limit; + +/* Enforce statically defined limit. */ + target =min (target,MAX_DOMAIN_PAGES ); + +if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 517
Quoted 10 lines
drivers/xen/balloon.c +10
-0+if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { +pr_info ("xen_balloon: %s: Initial domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + + host_limit = rc;
I think you should use this method for both dom0 and domUs. No need to check static-max from xenstore.
Quoted 10 lines
drivers/xen/balloon.c +10
-0+ }else { + rc =xenbus_scanf (XBT_NIL ,"memory" ,"static-max" , +"%llu" , &host_limit); + +if (rc !=1 ) { +pr_info ("xen_balloon: %s: Guest domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } +
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 537
Quoted 10 lines
drivers/xen/balloon.c +10
-0+/* +* The given memory target limit value is in KiB, so it needs +* converting to pages. PAGE_SHIFT converts bytes to pages, +* hence PAGE_SHIFT - 10. +*/ + host_limit >>= (PAGE_SHIFT -10 ); + } + +/* Enforce hypervisor/host defined limit. */ + target =min (target, (unsigned long )host_limit);
With the change above, you can change host_limit to unsigned long and avoid the cast.
David
Quoted 23 lines · David Vrabel
Quoted 8 lines · Daniel Kiper
This patch enforces on target limit statically defined in Linux Kernel source and limit defined by hypervisor or host.
Particularly this patch fixes bug which led to flood of dom0 kernel log with messages similar to:
System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be added
xen_balloon:reserve_additional_memory:add_memory()failed: -17I think this helps in some cases, but because
reserve_additional_memory()places the hotplugged memory aftermax_pfnwithout checking if there is anything already there, there are still ways it can repeatedly fail.e.g.,
1. If dom0 has had its maximum reservation limited initially (with the
dom0_memoption) /and/ the max reservation and target is subsequently raised then the balloon driver will attempt to hotplug memory that overlaps withE820_UNUSABLEregions in the e820 map and the hotplug will fail.
Quoted 3 lines
2. If a domU has passed-through PCI devices
max_pfnis before the PCI memory window then the balloon driver will attempt to hotplug memory over the PCI device BARs.
You are right. During work on this patch I discovered that but decided to enforce limits because it is more generic. Please look below why. However, I stated that it should be fixed too. I added it to my todo list. It requires a bit more work because I think new algorithm should cover many different cases. Probably add_memory() (it requires changes in mm/memory_hotplug.c) should be modified to look for range having sufficient size and not conflicting with others.
Quoted 8 lines
I think
reserve_additional_memory()should check the current resource map and the e820 map to find a large enough unused region. This can be done as an additional patch at a later date.Quoted 2 lines
It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
This sentence confuses me.
For example:
- boot guest domain with memory = 2048 and maxmem = 2048,
- xl mem-set 1024,
- xl mem-max 1536,
- echo 2097152 > /sys/devices/system/xen_memory/xen_memory0/target_kb,
- ...and balloon driver without this patch attempts to increase reservation until limit is not set to at least 2048...
That is why this patch is more generic.
Quoted 41 lines
Quoted 41 lines
Signed-off-by Daniel Kiper <[email protected]>--- drivers/xen/balloon.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)drivers/xen/balloon.c +18
-1@@ -65,6 +65,7 @@#include <xen/balloon.h> #include <xen/features.h> #include <xen/page.h> +#include <xen/xenbus.h> /* * balloon_process() state: @@ -490,11 +491,55 @@ static void balloon_process(struct work_struct *work)mutex_unlock (&balloon_mutex); } -/* Resets the Xen limit, sets new target, and kicks off processing. */ +/* Enforce limits, set new target and kick off processing. */ void balloon_set_new_target (unsigned long target) { +domid_t domid =DOMID_SELF ; +int rc; +unsigned long long host_limit; + +/* Enforce statically defined limit. */ + target =min (target,MAX_DOMAIN_PAGES ); + +if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */ +if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { + pr_info("xen_balloon: %s: Initial domain target limit "
Quoted 8 lines
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 517
Quoted 5 lines
drivers/xen/balloon.c +5
-0+ "could not be established: %i\n", __func__, rc); + goto no_host_limit; + } + + host_limit = rc;I think you should use this method for both dom0 and domUs. No need to check static-max from xenstore.
Sadly XENMEM_maximum_reservation for domU returns value which is set by xl mem-set not by xl mem-max :-(((... That is why I get this value from xenstore.
Quoted 15 lines
Quoted 15 lines
drivers/xen/balloon.c +15
-0+ }else { + rc =xenbus_scanf (XBT_NIL ,"memory" ,"static-max" , +"%llu" , &host_limit); + +if (rc !=1 ) { +pr_info ("xen_balloon: %s: Guest domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + +/* +* The given memory target limit value is in KiB, so it needs +* converting to pages. PAGE_SHIFT converts bytes to pages, +* hence PAGE_SHIFT - 10. +*/
Quoted 8 lines
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 537
Quoted 5 lines
drivers/xen/balloon.c +5
-0+ host_limit >>= (PAGE_SHIFT -10 ); + } + +/* Enforce hypervisor/host defined limit. */ + target =min (target, (unsigned long )host_limit);With the change above, you can change
host_limitto unsigned long and avoid the cast.
It is not possible. Please look above why...
Daniel
Quoted 28 lines · Daniel Kiper
Quoted 23 lines · David Vrabel
Quoted 8 lines · Daniel Kiper
This patch enforces on target limit statically defined in Linux Kernel source and limit defined by hypervisor or host.
Particularly this patch fixes bug which led to flood of dom0 kernel log with messages similar to:
System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be added
xen_balloon:reserve_additional_memory:add_memory()failed: -17I think this helps in some cases, but because
reserve_additional_memory()places the hotplugged memory aftermax_pfnwithout checking if there is anything already there, there are still ways it can repeatedly fail.e.g.,
1. If dom0 has had its maximum reservation limited initially (with the
dom0_memoption) /and/ the max reservation and target is subsequently raised then the balloon driver will attempt to hotplug memory that overlaps withE820_UNUSABLEregions in the e820 map and the hotplug will fail.Quoted 3 lines
2. If a domU has passed-through PCI devices
max_pfnis before the PCI memory window then the balloon driver will attempt to hotplug memory over the PCI device BARs.
Quoted 7 lines
You are right. During work on this patch I discovered that but decided to enforce limits because it is more generic. Please look below why. However, I stated that it should be fixed too. I added it to my todo list. It requires a bit more work because I think new algorithm should cover many different cases. Probably
add_memory()(it requires changes inmm/memory_hotplug.c) should be modified to look for range having sufficient size and not conflicting with others.
Ok, so we're agreed, this patch doesn't fix everything and that's fine.
Quoted 8 lines
Quoted 8 lines
I think
reserve_additional_memory()should check the current resource map and the e820 map to find a large enough unused region. This can be done as an additional patch at a later date.Quoted 2 lines
It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
This sentence confuses me.
I'm just confused by the English. Perhaps it should say:
"The balloon driver will limit target to the maximum reservation as any attempt to populate pages above the maximum reservation will always fail."
?
Quoted 52 lines
That is why this patch is more generic.
Quoted 41 lines
Quoted 41 lines
Signed-off-by Daniel Kiper <[email protected]>--- drivers/xen/balloon.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)drivers/xen/balloon.c +18
-1@@ -65,6 +65,7 @@#include <xen/balloon.h> #include <xen/features.h> #include <xen/page.h> +#include <xen/xenbus.h> /* * balloon_process() state: @@ -490,11 +491,55 @@ static void balloon_process(struct work_struct *work)mutex_unlock (&balloon_mutex); } -/* Resets the Xen limit, sets new target, and kicks off processing. */ +/* Enforce limits, set new target and kick off processing. */ void balloon_set_new_target (unsigned long target) { +domid_t domid =DOMID_SELF ; +int rc; +unsigned long long host_limit; + +/* Enforce statically defined limit. */ + target =min (target,MAX_DOMAIN_PAGES ); + +if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */ +if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { + pr_info("xen_balloon: %s: Initial domain target limit "Quoted 8 lines
drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 517
Quoted 5 lines
drivers/xen/balloon.c +5
-0+ "could not be established: %i\n", __func__, rc); + goto no_host_limit; + } + + host_limit = rc;I think you should use this method for both dom0 and domUs. No need to check static-max from xenstore.
Quoted 2 lines
Sadly
XENMEM_maximum_reservationfor domU returns value which is set by xl mem-set not by xl mem-max :-(((... That is why I get this value from xenstore.
It gets d->max_pages which the limit for d->tot_pages. d->max_pages is set by xl mem-max (and xl mem-set as it uses the enforce option to libxl_set_memory_target()).
If you set the target above d->max_pages you won't be able to populate them.
So, using the maximum_reservation call seems like the right thing to me.
David
Quoted 1 line · David Vrabel
On 06/03/13 16:47, Daniel Kiper wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I think
reserve_additional_memory()should check the current resource map and the e820 map to find a large enough unused region. This can be done as an additional patch at a later date.Quoted 2 lines
It does not allow balloon driver to execute infinite loops when target exceeds limits in other cases too.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
This sentence confuses me.
I'm just confused by the English. Perhaps it should say:
"The balloon driver will limit target to the maximum reservation as any attempt to populate pages above the maximum reservation will always fail."
?
That is OK.
[...]
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 4 lines
drivers/xen/balloon.c +4
-0+if (xen_initial_domain ()) { + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &domid); + +/* Limit is not enforced by hypervisor. */ drivers/xen/balloon.c static void balloon_process(struct work_struct *work) · line 517
Quoted 10 lines
drivers/xen/balloon.c +10
-0+if (rc == -EPERM ) + goto no_host_limit; + +if (rc <=0 ) { +pr_info ("xen_balloon: %s: Initial domain target limit " +"could not be established: %i\n" , __func__, rc); + goto no_host_limit; + } + + host_limit = rc;I think you should use this method for both dom0 and domUs. No need to check static-max from xenstore.
Quoted 5 lines
Quoted 2 lines
Sadly
XENMEM_maximum_reservationfor domU returns value which is set by xl mem-set not by xl mem-max :-(((... That is why I get this value from xenstore.It gets
d->max_pageswhich the limit ford->tot_pages.d->max_pagesis set by xl mem-max (and xl mem-set as it uses the enforce option to
No, it was tested by me and d->max_pages is set by xl mem-set (it is current target). d->tot_pages has count of pages for a given moment. It is confusing for me but this is the reality.
Quoted 5 lines
libxl_set_memory_target()).If you set the target above
d->max_pagesyou won't be able to populate them.So, using the
maximum_reservationcall seems like the right thing to me.
Please look above. If you use this value you would not be able to increase reservation.
Daniel
Quoted 8 lines · Daniel Kiper
Quoted 3 lines · David Vrabel
If you set the target above
d->max_pagesyou won't be able to populate them.So, using the
maximum_reservationcall seems like the right thing to me.Please look above. If you use this value you would not be able to increase reservation.
I don't think I'm understanding the use case you're talking about.
Do you mean we should allow a target <= d->tot_pages even if this is above d->max_pages? I agree with this.
Something like this:
void balloon_set_new_target(unsigned long target)
{
domid_t domid = DOMID_SELF;
int rc;
unsigned long host_limit;
target = min(target, MAX_DOMAIN_PAGES);
/* Prevent target from attempting the expand the reservation
above the max enforced by the hypervisor. */
rc = HYPERVISOR_memory_op(XENMEM_maximum_reservation, &domid);
if (rc > 0) {
host_limit = rc;
target = min(target,
max(host_limit, balloon_stats.current_pages));
}
balloon_stats.target_pages = target;
schedule_delayed_work(&balloon_worker, 0);
}David
Quoted 29 lines · David Vrabel
Quoted 8 lines · Daniel Kiper
Quoted 3 lines · David Vrabel
If you set the target above
d->max_pagesyou won't be able to populate them.So, using the
maximum_reservationcall seems like the right thing to me.Please look above. If you use this value you would not be able to increase reservation.
I don't think I'm understanding the use case you're talking about.
Do you mean we should allow a target <=
d->tot_pageseven if this is aboved->max_pages? I agree with this.Something like this:
void
balloon_set_new_target(unsigned long target){ domid_t domid = DOMID_SELF; int rc; unsigned long host_limit; target = min(target, MAX_DOMAIN_PAGES); /* Prevent target from attempting the expand the reservation above the max enforced by the hypervisor. */ rc = HYPERVISOR_memory_op(XENMEM_maximum_reservation, &domid); if (rc > 0) {
Quoted 8 lines
host_limit = rc; target = min(target, max(host_limit, balloon_stats.current_pages)); } balloon_stats.target_pages = target; schedule_delayed_work(&balloon_worker, 0); }
I dug deeper into Xen source and found some strange things for me.
If you call xl mem-set it does two things:
- sets target in xenstore,
- sets d->max_pages (sic!).
Last thing is very confusing because at least command comment does mention nothing about this behavior. Comment for this command states: "Set the current memory usage for a domain". It does say nothing about setting memory usage limit. IMO it is the role of xl mem-max (another strange thing is that xl mem-max sets d->max_pages but does not touch static-max; I think it should be changed too; now it is not possible to increase limit above maxmem defined at startup which makes memory hotplug practically unusable). That is why xl mem-set behavior should be changed (IIRC xm sets only target in xenstore) or comment should be aligned to what xl code doas in real. Then we could back to discussion about new balloon_set_new_target().
Daniel