Make ballooning work with maxmem > mem (i386 version)
PATCH 1/1 Make ballooning work with maxmem > mem (i386 version)
Trying to start a guest with maxmem > mem and then balloon up to a value greater than mem is currently failing. This have been already discovered (patch sent some days ago) for x86_64. i386 suffers from the same problem.
This patch fixes it.
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
# HG changeset patch
# User Glauber de Oliveira Costa <[email protected]>
# Node ID 2e2e6a3f9b2d0ad21c666dfbdc075f918e0bc7be
# Parent 20204db0891b0b7c10959822e3283656c3600500
[LINUX] Extend physical mapping to maxmem instead of mem - i386 version
As currently physical mapping only reaches the initial reservation, we're unable to balloon up to more than mem (even when maxmem > mem) in any situation. This have already been found out to be broken and fixed in x86_64
linux-2.6-xen-sparse/include/asm-i386/mach-xen/setup_arch_post.h +6 -0
@@ -13,6 +13,7 @@ {int rc;struct xen_memory_map memmap; +unsigned long arg =DOMID_SELF ;/* * This is rather large for a stack variable but this early in * the boot process we know we have plenty slack space.
@@ -26,6 +27,11 @@if ( rc == -ENOSYS ) { memmap.nr_entries =1 ; map[0 ].addr =0ULL ; + rc =HYPERVISOR_memory_op (XENMEM_maximum_reservation, &arg); +if ( rc <0 ) + map[0 ].size =PFN_PHYS (xen_start_info->nr_pages); +else + map[0 ].size =PFN_PHYS (rc); map[0 ].size =PFN_PHYS (xen_start_info->nr_pages);/* 8MB slack (to balance backend allocations). */ map[0 ].size +=8ULL <<20 ;
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 7 lines · Keir Fraser
Quoted 5 lines · Glauber de Oliveira Costa
Trying to start a guest with maxmem > mem and then balloon up to a value greater than mem is currently failing. This have been already discovered (patch sent some days ago) for
x86_64. i386 suffers from the same problem.
Quoted 7 lines
Quoted 1 line
This patch fixes it.
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].
And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
IMHO,If we have to ever fallback into default assumptions, it seems wiser to extend the physicall map to maximum_reservation, not current_reservation.
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
Quoted 11 lines · Glauber de Oliveira Costa
Quoted 5 lines
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
IMHO,If we have to ever fallback into default assumptions, it seems wiser to extend the physicall map to
maximum_reservation, notcurrent_reservation.
Maxmem will in future be fixed to track tot_pages. That was its original
purpose: to cap what memory the guest is allowed *now*, not to tell it the max that it will ever be allowed. Also on e.g., dom0 the max is implemented as ~0UL (no limit). "Fortunately" this looks like a -ve return code from the max_mem query hypercall, but if it didn't (for example we masked off the high bit to make sure the return value was +ve; a fix I will probably implement) then dom0 boot will be broken as it will try to allocate a much-too-big p2m table.
We have a way to tell the guest its maximum-ever memory allowance, it's just not implemented on the tools and hypervisor side yet. That should be fixed.
-- Keir
Quoted 8 lines · Glauber de Oliveira Costa
Quoted 5 lines
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
Oh, and bear in mind you can always use 'memory=' as a boot parameter to force the p2m size and mem_map size.
-- Keir
Quoted 11 lines · Keir Fraser
Quoted 6 lines · Glauber de Oliveira Costa
Quoted 5 lines
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].
Quoted 5 lines
Quoted 2 lines
And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
Oh, and bear in mind you can always use 'memory=' as a boot parameter to force the p2m size and
mem_mapsize.
That's not particuarly helpful when booting a domain with pygrub because now everytime you change the memory settings in the config for the guest you also need to update the grub config to actually make it take effect. Or, if you're not using pygrub, then every time you change the 'mem' setting in the guest config, you also need to update the 'extra' setting for the command line.
Regards,
Dan.
Signature · 4 lines
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Quoted 23 lines · Keir Fraser
Quoted 11 lines · Glauber de Oliveira Costa
Quoted 5 lines
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
IMHO,If we have to ever fallback into default assumptions, it seems wiser to extend the physicall map to
maximum_reservation, notcurrent_reservation.Maxmem will in future be fixed to track
tot_pages. That was its original
purpose: to cap what memory the guest is allowed *now*, not to tell it the max that it will ever be allowed. Also on e.g., dom0 the max is implemented as ~0UL (no limit). "Fortunately" this looks like a -ve return code from themax_memquery hypercall, but if it didn't (for example we masked off the high bit to make sure the return value was +ve; a fix I will probably implement) then dom0 boot will be broken as it will try to allocate a much-too-big p2m table.
Quoted 2 lines
We have a way to tell the guest its maximum-ever memory allowance, it's just not implemented on the tools and hypervisor side yet. That should be fixed.
This is not going to be any help for guests running on existing 3.0.3 tree though. Glauber's patch makes the maxmem stuff work correctly for guest kernels without requiring an upgrade of the HV which is much nicer for compatability allowing both current & future guests to work on both current and future HVs. With the increasingly widespread deployment of Xen I think this kind of compatability is very important.
Regards,
Dan.
Signature · 4 lines
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Quoted 6 lines · Daniel P. Berrange
This is not going to be any help for guests running on existing 3.0.3 tree though. Glauber's patch makes the maxmem stuff work correctly for guest kernels without requiring an upgrade of the HV which is much nicer for compatability allowing both current & future guests to work on both current and future HVs. With the increasingly widespread deployment of Xen I think this kind of compatability is very important.
I'd rather have a scheme that worked with older guests than one which worked with older hypervisors. It also happens to be the more tasteful solution here.
-- Keir
Quoted 11 lines · Keir Fraser
Quoted 9 lines · Glauber de Oliveira Costa
Quoted 5 lines
I took both patches and then changed my mind and immediately reverted them. There is a better way: we should support the
XENMEM_memory_maphypercall. We should provide a hypercall (domctl) to set amemory_map_limitparameter and then Xen can use that to fake a memory map whenXENMEM_memory_mapis called. The tools can set that parameter from config['maxmem'].And what happens when the hypercall ever returns ENOSYS, like a kernel running in a bit old Hypervisor?
Quoted 6 lines
Quoted 2 lines
IMHO,If we have to ever fallback into default assumptions, it seems wiser to extend the physicall map to
maximum_reservation, notcurrent_reservation.Maxmem will in future be fixed to track
tot_pages. That was its original
purpose: to cap what memory the guest is allowed *now*, not to tell it the max that it will ever be allowed.
In this scenario, what's the purpose of current_reservation, as the only difference from it now, is that it returns tot_pages instead of max_pages ?
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
Quoted 7 lines · Glauber de Oliveira Costa
Quoted 3 lines
Maxmem will in future be fixed to track
tot_pages. That was its original
purpose: to cap what memory the guest is allowed *now*, not to tell it the max that it will ever be allowed.In this scenario, what's the purpose of
current_reservation, as the only difference from it now, is that it returnstot_pagesinstead ofmax_pages?
The idea is that when you ask a guest to balloon down you will change its balloon target, then you will give it 'reasonable time' to drop its memory allocation (reflected by tot_pages). If it fails to comply you take action (e.g., warn user, suspend or kill the guest, etc). If it complies you drop max_pages.
So you see that max_pages tracks behind tot_pages when you balloon down, and tracks ahead when you balloon up. It has a distinct purpose separate from that of tot_pages. That's the theory anyway. It's never been properly implemented in the python tools. But that may well change in future, so I'm not going to bake assumptions into guest kernels.
-- Keir
Quoted 3 lines
We have a way to tell the guest its maximum-ever memory allowance, it's just not implemented on the tools and hypervisor side yet. That should be fixed.
Do we have any specification on how should this look like, and what kind of behaviour it should provide?
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
Quoted 5 lines · Glauber de Oliveira Costa
Quoted 2 lines
We have a way to tell the guest its maximum-ever memory allowance, it's just not implemented on the tools and hypervisor side yet. That should be fixed.
Do we have any specification on how should this look like, and what kind of behaviour it should provide?
XENMEM_set_memory_map taking a struct { domain, range_list }. This can initialise a small array embedded in the domain structure.
Plumb through libxc and python xc.so. Probably make the call from LinuxImageHandler (or whatever it's called) in image.py.
Done.
Most of the pain is in the tools plumbing (in particular arg parsing in the python wrapper).
-- Keir