xenpaging changes for xen-4.1
cover xenpaging changes for xen-4.1
These changes make xenpaging work for me.
The first one is a fix for a bug I introduced shortly before leaving for vacation. Currently the guest will hang because the ring fills up when the balloon driver removes pages.
The second one avoids a crash during BIOS startup.
The third one is another version of the machine_to_phys_mapping[] handling. The fourth is only required when the balloon driver is used in the guest. The fifth implements a config option to allow an automated start of xenpaging for a given guest.
And the last one lists some action items for xenpaging.
As of now, xenpaging will either crash the hypervisor or the guest due to page-out of low memory. Perhaps the building of xenpaging could be disabled in the Makefile?
Olaf
PATCH 1/6 xenpaging: correct dropping pages to avoid full ring buffer
xenpaging uses the mem_event ring buffer, which expects request/response pairs to make progress. The previous patch, which tried to establish a one-way communication from Xen to xenpaging, stalled the guest once the buffer was filled up with requests. A simple fix is to take the slow path and let p2m_mem_paging_resume() consume the response from xenpaging. This makes room for yet another request/response pair and avoids hanging guests.
--- tools/xenpaging/xenpaging.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
tools/xenpaging/xenpaging.c +11 -11
@@ -653,19 +653,19 @@ int main(int argc, char *argv[])ERROR ("Error populating page" ); goto out; } + } -/* Prepare the response */ - rsp.gfn = req.gfn; - rsp.p2mt = req.p2mt; - rsp.vcpu_id = req.vcpu_id; - rsp.flags = req.flags; +/* Prepare the response */ + rsp.gfn = req.gfn; + rsp.p2mt = req.p2mt; + rsp.vcpu_id = req.vcpu_id; + rsp.flags = req.flags; - rc =xenpaging_resume_page (paging, &rsp,1 ); -if ( rc !=0 ) - { -ERROR ("Error resuming page" ); - goto out; - } + rc =xenpaging_resume_page (paging, &rsp,1 ); +if ( rc !=0 ) + { +ERROR ("Error resuming page" ); + goto out; }/* Evict a new page to replace the one we just paged in */
PATCH 2/6 xenpaging: prevent page-out of first 16MB
In reply to Olaf HeringThis is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages, crashes will occour due to lack of support of xenpaging.
A more complet change to prevent the early crashes is to use the newly added wait_queue feature in the gfn_to_mfn() variants.
---
tools/xenpaging/policy_default.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/xenpaging/policy_default.c +3 -2
@@ -73,8 +73,9 @@ int policy_init(xenpaging_t *paging)for ( i =0 ; i < mru_size; i++ ) mru[i] =INVALID_MFN ; -/* Don't page out page 0 */ -set_bit (0 , bitmap); +/* Don't page out first 16MB */ +for ( i =0 ; i < ((16 *1024 *1024 )/4096 ); i++ ) +set_bit (i, bitmap); out:return rc;
PATCH 3/6 xenpaging: update machine_to_phys_mapping[] during page deallocation
In reply to Olaf HeringThe machine_to_phys_mapping[] array needs updating during page deallocation. If that page is allocated again, a call to get_gpfn_from_mfn() will still return an old gfn from another guest. This will cause trouble because this gfn number has no or different meaning in the context of the current guest.
This happens when the entire guest ram is paged-out before xen_vga_populate_vram() runs. Then XENMEM_populate_physmap is called with gfn 0xff000. A new page is allocated with alloc_domheap_pages. This new page does not have a gfn yet. However, in guest_physmap_add_entry() the passed mfn maps still to an old gfn (perhaps from another old guest). This old gfn is in paged-out state in this guests context and has no mfn anymore. As a result, the ASSERT() triggers because p2m_is_ram() is true for p2m_ram_paging* types. If the machine_to_phys_mapping[] array is updated properly, both loops in guest_physmap_add_entry() turn into no-ops for the new page and the mfn/gfn mapping will be done at the end of the function.
If XENMEM_add_to_physmap is used with XENMAPSPACE_gmfn, get_gpfn_from_mfn() will return an appearently valid gfn. As a result, guest_physmap_remove_page() is called. The ASSERT in p2m_remove_page triggers because the passed mfn does not match the old mfn for the passed gfn.
--- xen/common/page_alloc.c | 9 +++++++++ 1 file changed, 9 insertions(+)
xen/common/page_alloc.c +9 -0
@@ -1200,6 +1200,7 @@ void free_domheap_pages(struct page_info {int i, drop_dom_ref;struct domain *d =page_get_owner (pg); +unsigned long mfn;ASSERT (!in_irq ());
@@ -1257,6 +1258,14 @@ void free_domheap_pages(struct page_info drop_dom_ref =0 ; } +/* this page is not a gfn anymore */ + mfn =page_to_mfn (pg); +for ( i =0 ; i < (1 << order); i++ ) + { +page_set_owner (&pg[i],NULL ); +set_gpfn_from_mfn (mfn + i,INVALID_M2P_ENTRY ); + } +if ( drop_dom_ref )put_domain (d); }
PATCH 4/6 xenpaging: handle HVMCOPY_gfn_paged_out in copy_from/to_user
In reply to Olaf Heringcopy_from_user_hvm can fail when __hvm_copy returns HVMCOPY_gfn_paged_out for a referenced gfn, for example during guests pagetable walk. This has to be handled in some way.
For the time being, return -EAGAIN for the most common case (xen_balloon driver crashing in guest) until the recently added waitqueues will be used.
--- xen/arch/x86/hvm/hvm.c | 4 ++++ xen/common/memory.c | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 38 insertions(+), 5 deletions(-)
xen/arch/x86/hvm/hvm.c +4 -0
@@ -2163,6 +2163,8 @@ unsigned long copy_to_user_hvm(void *to, rc =hvm_copy_to_guest_virt_nofault ((unsigned long )to, (void *)from, len,0 ); +if (unlikely (rc == HVMCOPY_gfn_paged_out) ) +return -EAGAIN ;return rc ? len :0 ;/* fake a copy_to_user() return code */ }
@@ -2180,6 +2182,8 @@ unsigned long copy_from_user_hvm(void *t#endif rc =hvm_copy_from_guest_virt_nofault (to, (unsigned long )from, len,0 ); +if (unlikely (rc == HVMCOPY_gfn_paged_out) ) +return -EAGAIN ;return rc ? len :0 ;/* fake a copy_from_user() return code */ }
xen/common/memory.c +34 -5
@@ -48,6 +48,7 @@ static void increase_reservation(struct {struct page_info *page;unsigned long i; +unsigned long ctg_ret;xen_pfn_t mfn;struct domain *d = a->domain;
@@ -81,8 +82,13 @@ static void increase_reservation(structif ( !guest_handle_is_null (a->extent_list) ) { mfn =page_to_mfn (page); -if (unlikely (__copy_to_guest_offset (a->extent_list, i, &mfn,1 )) ) + ctg_ret =__copy_to_guest_offset (a->extent_list, i, &mfn,1 ); +if (unlikely (ctg_ret) ) + { +if ( (long )ctg_ret == -EAGAIN ) + a->preempted =1 ; goto out; + } } }
@@ -94,6 +100,7 @@ static void populate_physmap(struct memo {struct page_info *page;unsigned long i, j; +unsigned long cftg_ret;xen_pfn_t gpfn, mfn;struct domain *d = a->domain;
@@ -112,8 +119,13 @@ static void populate_physmap(struct memo goto out; } -if (unlikely (__copy_from_guest_offset (&gpfn, a->extent_list, i,1 )) ) + cftg_ret =__copy_from_guest_offset (&gpfn, a->extent_list, i,1 ); +if (unlikely (cftg_ret) ) + { +if ( (long )cftg_ret == -EAGAIN ) + a->preempted =1 ; goto out; + }if ( a->memflags & MEMF_populate_on_demand ) {
@@ -143,8 +155,13 @@ static void populate_physmap(struct memoset_gpfn_from_mfn (mfn + j, gpfn + j);/* Inform the domain of the new page's machine address. */ -if (unlikely (__copy_to_guest_offset (a->extent_list, i, &mfn,1 )) ) + cftg_ret =__copy_to_guest_offset (a->extent_list, i, &mfn,1 ); +if (unlikely (cftg_ret) ) + { +if ( (long )cftg_ret == -EAGAIN ) + a->preempted =1 ; goto out; + } } } }
@@ -213,6 +230,7 @@ int guest_remove_page(struct domain *d, static void decrease_reservation(struct memop_args *a) { unsigned long i, j; + unsigned long cfg_ret; xen_pfn_t gmfn; if ( !guest_handle_subrange_okay(a->extent_list, a->nr_done,
@@ -227,8 +245,13 @@ static void decrease_reservation(struct goto out; } -if (unlikely (__copy_from_guest_offset (&gmfn, a->extent_list, i,1 )) ) + cfg_ret =__copy_from_guest_offset (&gmfn, a->extent_list, i,1 ); +if (unlikely (cfg_ret) ) + { +if ( (long )cfg_ret == -EAGAIN ) + a->preempted =1 ; goto out; + }if ( tb_init_done ) {
@@ -509,6 +532,7 @@ long do_memory_op(unsigned long cmd, XENint rc, op;unsigned int address_bits;unsigned long start_extent; +unsigned long cfg_ret;struct xen_memory_reservation reservation;struct memop_args args;domid_t domid;
@@ -522,8 +546,13 @@ long do_memory_op(unsigned long cmd, XENcase XENMEM_populate_physmap: start_extent = cmd >>MEMOP_EXTENT_SHIFT ; -if (copy_from_guest (&reservation, arg,1 ) ) + cfg_ret =copy_from_guest (&reservation, arg,1 ); +if (unlikely (cfg_ret) ) + { +if ( (long )cfg_ret == -EAGAIN ) +return hypercall_create_continuation (__HYPERVISOR_memory_op,"lh" , cmd, arg);return start_extent; + }/* Is size too large for us to encode a continuation? */ if ( reservation.nr_extents > (ULONG_MAX >>MEMOP_EXTENT_SHIFT ) )
PATCH 5/6 xenpaging: start xenpaging via config option
In reply to Olaf HeringStart xenpaging via config option.
TODO: add config option for different pagefile directory
TODO: add libxl support
TODO: parse config values like 42K, 42M, 42G, 42%
---
v3:
move debug for stopping xenpaging to destroyXenPaging
v2:
unlink logfile instead of truncating it. allows hardlinking for further inspection
tools/examples/xmexample.hvm | 3 + tools/python/README.XendConfig | 1 tools/python/README.sxpcfg | 1 tools/python/xen/xend/XendConfig.py | 3 + tools/python/xen/xend/XendDomainInfo.py | 5 + tools/python/xen/xend/image.py | 87 ++++++++++++++++++++++++++++++++ tools/python/xen/xm/create.py | 5 + tools/python/xen/xm/xenapi_create.py | 1 8 files changed, 106 insertions(+)
tools/examples/xmexample.hvm +3 -0
@@ -127,6 +127,9 @@ disk = [ 'file:/var/images/min-el3-i386. # Device Model to be used device_model = 'qemu-dm' +# xenpaging, number of pages, or -1 for entire guest memory range +xenpaging = 42 + #----------------------------------------------------------------------------- # boot on floppy (a), hard disk (c), Network (n) or CD-ROM (d) # default: hard disk, cd-rom, floppy
tools/python/README.XendConfig +1 -0
@@ -120,6 +120,7 @@ otherConfig image.vncdisplay image.vncunused image.hvm.device_model + image.hvm.xenpaging image.hvm.display image.hvm.xauthority image.hvm.vncconsole
tools/python/README.sxpcfg +1 -0
@@ -51,6 +51,7 @@ image - vncunused (HVM) - device_model + - xenpaging - display - xauthority - vncconsole
tools/python/xen/xend/XendConfig.py +3 -0
@@ -147,6 +147,7 @@ XENAPI_PLATFORM_CFG_TYPES = { 'apic': int, 'boot': str, 'device_model': str, + 'xenpaging': int, 'loader': str, 'display' : str, 'fda': str,
@@ -512,6 +513,8 @@ class XendConfig(dict): self['platform']['nomigrate'] = 0 if self.is_hvm(): + if 'xenpaging' not in self['platform']: + self['platform']['xenpaging'] = None if 'timer_mode' not in self['platform']: self['platform']['timer_mode'] = 1 if 'viridian' not in self['platform']:
tools/python/xen/xend/XendDomainInfo.py +5 -0
@@ -2390,6 +2390,7 @@ class XendDomainInfo: if self.image: self.image.createDeviceModel() + self.image.createXenPaging() #if have pass-through devs, need the virtual pci slots info from qemu self.pci_device_configure_boot()
@@ -2402,6 +2403,10 @@ class XendDomainInfo: self.image.destroyDeviceModel() except Exception, e: log.exception("Device model destroy failed %s" % str(e)) + try: + self.image.destroyXenPaging() + except Exception, e: + log.exception("stopping xenpaging failed %s" % str(e)) else: log.debug("No device model")
tools/python/xen/xend/image.py +87 -0
105 lines not on this page · raw message
tools/python/xen/xm/create.py +5 -0
@@ -491,6 +491,10 @@ gopts.var('nfs_root', val="PATH", fn=set_value, default=None, use="Set the path of the root NFS directory.") +gopts.var('xenpaging', val='NUM', + fn=set_int, default=None, + use="Number of pages to swap.") + gopts.var('device_model', val='FILE', fn=set_value, default=None, use="Path to device model program.")
@@ -1076,6 +1080,7 @@ def configure_hvm(config_image, vals): args = [ 'acpi', 'apic', 'boot', 'cpuid', 'cpuid_check', + 'xenpaging', 'device_model', 'display', 'fda', 'fdb', 'gfx_passthru', 'guest_os_type',
tools/python/xen/xm/xenapi_create.py +1 -0
@@ -1085,6 +1085,7 @@ class sxp2xml: 'acpi', 'apic', 'boot', + 'xenpaging', 'device_model', 'loader', 'fda',
PATCH 6/6 xenpaging: document outstanding features
In reply to Olaf Hering--- docs/misc/xenpaging.txt | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
docs/misc/xenpaging.txt +20 -2
@@ -31,7 +31,7 @@ crash because the paged-out memory is no After a reboot of a guest, its guest_id changes, the current xenpaging binary has no target anymore. To automate restarting of xenpaging after -guest reboot, specify the number if pages in the guest configuration +guest reboot, specify the number of pages in the guest configuration file /etc/xen/vm/<guest_name>: xenpaging=32768
@@ -41,8 +41,26 @@ changes. Todo: +- implement xl support - implement stopping of xenpaging -- implement/test live migration +- implement live migration +- implement config option for XENPAGING_DEBUG and XENPAGING_POLICY_MRU_SIZE +- implement config option for xenpaging_dir +- implement better starting gfn in xenpaging policy + an initial gfn number in the middle of the gfn range may avoid page-ins + during BIOS startup +- fix machine_to_phys_mapping[] array handling during page deallocation + the gfn of a released page must be maintained properly in the array + http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00824.html +- fix HVMCOPY_gfn_paged_out handling + some callers of __hvm_copy() do not handle HVMCOPY_gfn_paged_out, such + as hypercalls and the MMIO emulation + the recently added waitqueue feature in Xen 4.1 should be used +- remove all retry code from gfn_to_mfn() calls + use the waitqueue feature to hide page-in from the caller and cover + all cases where a retry is currently missing +- do not bounce p2mt to xenpaging + p2m_mem_paging_populate/p2m_mem_paging_resume dont make use of p2mt # vim: tw=72
Quoted 13 lines · Olaf Hering
These changes make xenpaging work for me.
The first one is a fix for a bug I introduced shortly before leaving for vacation. Currently the guest will hang because the ring fills up when the balloon driver removes pages.
The second one avoids a crash during BIOS startup.
The third one is another version of the machine_to_phys_mapping[] handling. The fourth is only required when the balloon driver is used in the guest. The fifth implements a config option to allow an automated start of xenpaging for a given guest.
And the last one lists some action items for xenpaging.
Quoted 3 lines
As of now, xenpaging will either crash the hypervisor or the guest due to page-out of low memory. Perhaps the building of xenpaging could be disabled in the
Makefile?
The hard truth is that this misses 4.1.0. We will check this stuff in when 4.2 development opens and then, when we have a fully working set of patches, we can consider backport to 4.1.x. Frankly I'm a little doubtful even of that, as various subsystems' interaction with the waitqueue feature needs a bunch more bugfixing, let alone the xenpaging patches that sit above it. That is, the required full set of patches to get xenpaging, plus all existing features, working correctly is likely to be quite large indeed.
-- Keir
Quoted 9 lines
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel