Skip to contentSkip to conversation

xenpaging changes for xen-unstable

PATCH In review Olaf Hering 5 participants 33 messages Nov 26, 2010 – Dec 17, 2010

lore
Share & tools
lore
#0 lore raw

cover xenpaging changes for xen-unstable

Here are some changes for xenpaging in xen-unstable.

Patches 1 to 14 are likely non-controversial and could be applied. All later patches need more review (15/16), and more work from my side (17+).

With a similar series of changes for 4.0.1 my testcase, which is a continuous reboot of a Linux guest, passes now. It ran for more than 1000 iterations and exercised the early bootcode.

There are still some issues like guests not starting up when running the reboot loop with 4x1G guests parallel on a 4GB box with all guest memory paged. This seems to happen when the guest initially needs the full 1G until xenpaging frees that memory again.

Other than that, xenpaging is now more mature.
I havent seen data corruption anymore.

Olaf

#1 lore raw

PATCH 1/21 xenpaging: break endless loop during inital page-out with large pagefiles

To allow the starting for xenpaging right after 'xm start XYZ', I specified a pagefile size equal to the guest memory size in the hope to catch more errors where the paged-out state of a p2mt is not checked.

While doing that, xenpaging got into an endless loop because some pages cant be paged out right away. Now the policy reports an error if the gfn number wraps.

---
 tools/xenpaging/policy_default.c |   35 ++++++++++++++++++++++++++++-------
 tools/xenpaging/xenpaging.c      |    7 +++++--
 2 files changed, 33 insertions(+), 9 deletions(-)
tools/xenpaging/policy_default.c +28 -7
@@ -30,8 +30,12 @@
 
 
 static unsigned long mru[MRU_SIZE];
-static unsigned int i_mru = 0;
+static unsigned int i_mru;
 static unsigned long *bitmap;
+static unsigned long *unconsumed;
+static unsigned long current_gfn;
+static unsigned long bitmap_size;
+static unsigned long max_pages;
 
 
 int policy_init(xenpaging_t *paging)
@@ -43,6 +47,14 @@ int policy_init(xenpaging_t *paging)
     rc = alloc_bitmap(&bitmap, paging->bitmap_size);
     if ( rc != 0 )
         goto out;
+    /* Allocate bitmap to track unusable pages */
+    rc = alloc_bitmap(&unconsumed, paging->bitmap_size);
+    if ( rc != 0 )
+        goto out;
+
+    /* record bitmap_size */
+    bitmap_size = paging->bitmap_size;
+    max_pages = paging->domain_info->max_pages;
 
     /* Initialise MRU list of paged in pages */
     for ( i = 0; i < MRU_SIZE; i++ )
@@ -51,8 +63,6 @@ int policy_init(xenpaging_t *paging)
     /* Don't page out page 0 */
     set_bit(0, bitmap);
 
-    rc = 0;
-
  out:
     return rc;
 }
@@ -61,17 +71,27 @@ int policy_choose_victim(xc_interface *x
                          xenpaging_t *paging, domid_t domain_id,
                          xenpaging_victim_t *victim)
 {
+    unsigned long wrap = current_gfn;
     ASSERT(victim != NULL);
 
     /* Domain to pick on */
     victim->domain_id = domain_id;
-    
+
     do
     {
-        /* Randomly choose a gfn to evict */
-        victim->gfn = rand() % paging->domain_info->max_pages;
+        current_gfn++;
+        if ( current_gfn >= max_pages )
+            current_gfn = 0;
+        if ( wrap == current_gfn )
+        {
+            victim->gfn = INVALID_MFN;
+            return -ENOSPC;
+        }
     }
-    while ( test_bit(victim->gfn, bitmap) );
+    while ( test_bit(current_gfn, bitmap) || test_bit(current_gfn, unconsumed) );
+
+    set_bit(current_gfn, unconsumed);
+    victim->gfn = current_gfn;
 
     return 0;
 }
@@ -79,6 +99,7 @@ int policy_choose_victim(xc_interface *x
 void policy_notify_paged_out(domid_t domain_id, unsigned long gfn)
 {
     set_bit(gfn, bitmap);
+    clear_bit(gfn, unconsumed);
 }
 
 void policy_notify_paged_in(domid_t domain_id, unsigned long gfn)
tools/xenpaging/xenpaging.c +5 -2
@@ -446,7 +446,8 @@ static int evict_victim(xc_interface *xc
         ret = policy_choose_victim(xch, paging, domain_id, victim);
         if ( ret != 0 )
         {
-            ERROR("Error choosing victim");
+            if ( ret != -ENOSPC )
+                ERROR("Error choosing victim");
             goto out;
         }
 
@@ -525,7 +526,9 @@ int main(int argc, char *argv[])
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
     {
-        evict_victim(xch, paging, domain_id, &victims[i], fd, i);
+        rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
+        if ( rc == -ENOSPC )
+            break;
         if ( i % 100 == 0 )
             DPRINTF("%d pages evicted\n", i);
     }
#2 lore raw

PATCH 2/21 xenpaging: Open paging file only if xenpaging_init() succeeds

In reply to Olaf Hering

Open paging file only if xenpaging_init() succeeds. It can fail if the host does not support the required virtualization features such as EPT or if xenpaging was already started for this domain_id.

---
 tools/xenpaging/xenpaging.c |   18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
tools/xenpaging/xenpaging.c +9 -9
@@ -502,15 +502,6 @@ int main(int argc, char *argv[])
 
     victims = calloc(num_pages, sizeof(xenpaging_victim_t));
 
-    /* Open file */
-    sprintf(filename, "page_cache_%d", domain_id);
-    fd = open(filename, open_flags, open_mode);
-    if ( fd < 0 )
-    {
-        perror("failed to open file");
-        return -1;
-    }
-
     /* Seed random-number generator */
     srand(time(NULL));
 
@@ -522,6 +513,15 @@ int main(int argc, char *argv[])
         goto out;
     }
 
+    /* Open file */
+    sprintf(filename, "page_cache_%d", domain_id);
+    fd = open(filename, open_flags, open_mode);
+    if ( fd < 0 )
+    {
+        perror("failed to open file");
+        return -1;
+    }
+
     /* Evict pages */
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
#3 lore raw

PATCH 3/21 xenpaging: allow only one xenpaging binary per guest

In reply to Olaf Hering

Make sure only one xenpaging binary is active per domain. Print info when the host lacks the required features for xenpaging.

---
v2: use perror for default case

 tools/xenpaging/xenpaging.c |   12 +++++++++++-
 xen/arch/x86/mm/mem_event.c |    7 +++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
tools/xenpaging/xenpaging.c +11 -1
@@ -123,7 +123,17 @@ xenpaging_t *xenpaging_init(xc_interface
                              paging->mem_event.ring_page);
     if ( rc != 0 )
     {
-        ERROR("Error initialising shared page");
+        switch ( errno ) {
+            case EBUSY:
+                ERROR("xenpaging is (or was) active on this domain");
+                break;
+            case ENODEV:
+                ERROR("EPT not supported for this guest");
+                break;
+            default:
+                perror("Error initialising shared page");
+                break;
+        }
         goto err;
     }
 
xen/arch/x86/mm/mem_event.c +7 -0
@@ -214,6 +214,13 @@ int mem_event_domctl(struct domain *d, x
             mfn_t ring_mfn;
             mfn_t shared_mfn;
 
+            /* Only one xenpaging at a time. If xenpaging crashed,
+             * the cache is in an undefined state and so is the guest
+             */
+            rc = -EBUSY;
+            if ( d->mem_event.ring_page )
+                break;
+
             /* Currently only EPT is supported */
             rc = -ENODEV;
             if ( !(hap_enabled(d) &&
#4 lore raw

PATCH 4/21 xenpaging: populate paged-out pages unconditionally

In reply to Olaf Hering

Populate a page unconditionally to avoid missing a page-in request. If the page is already in the process of being paged-in, the this vcpu will be stopped and later resumed once the page content is usable again.

This matches other p2m_mem_paging_populate usage in the source tree.

---
 xen/common/grant_table.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
xen/common/grant_table.c +3 -2
@@ -156,10 +156,11 @@ static int __get_paged_frame(unsigned lo
 
     if ( p2m_is_valid(p2mt) ) {
         *frame = mfn_x(mfn);
-        if ( p2m_is_paged(p2mt) )
-            p2m_mem_paging_populate(p2m, gfn);
         if ( p2m_is_paging(p2mt) )
+        {
+            p2m_mem_paging_populate(p2m, gfn);
             rc = GNTST_eagain;
+        }
     } else {
        *frame = INVALID_MFN;
        rc = GNTST_bad_page;
#5 lore raw

PATCH 5/21 xenpaging: add signal handling

In reply to Olaf Hering

Leave paging loop if xenpaging gets a signal.
Remove paging file on exit.

---
v2:
unlink pagefile in signal handler to avoid stale pagefiles if xenpaging is stuck in some loop

 tools/xenpaging/xenpaging.c |   42 +++++++++++++++++++++++++++++++++---------
 1 file changed, 33 insertions(+), 9 deletions(-)
tools/xenpaging/xenpaging.c +33 -9
@@ -25,6 +25,7 @@
 #include <stdlib.h>
 #include <stdarg.h>
 #include <time.h>
+#include <signal.h>
 #include <xc_private.h>
 
 #include <xen/mem_event.h>
@@ -43,6 +44,14 @@
 #define DPRINTF(...) ((void)0)
 #endif
 
+static char filename[80];
+static int interrupted;
+static void close_handler(int sig)
+{
+    interrupted = sig;
+    if ( filename[0] )
+        unlink(filename);
+}
 
 static void *init_page(void)
 {
@@ -248,7 +257,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error tearing down domain paging in xen");
-        goto err;
     }
 
     /* Unbind VIRQ */
@@ -256,7 +264,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error unbinding event port");
-        goto err;
     }
     paging->mem_event.port = -1;
 
@@ -265,7 +272,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing event channel");
-        goto err;
     }
     paging->mem_event.xce_handle = -1;
     
@@ -274,7 +280,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing connection to xen");
-        goto err;
     }
     paging->xc_handle = NULL;
 
@@ -380,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
     return ret;
 }
 
-int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
+static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
 {
     int ret;
 
@@ -461,6 +466,11 @@ static int evict_victim(xc_interface *xc
             goto out;
         }
 
+        if ( interrupted )
+        {
+            ret = -EINTR;
+            goto out;
+        }
         ret = xc_mem_paging_nominate(paging->xc_handle,
                                      paging->mem_event.domain_id, victim->gfn);
         if ( ret == 0 )
@@ -485,6 +495,7 @@ static int evict_victim(xc_interface *xc
 
 int main(int argc, char *argv[])
 {
+    struct sigaction act;
     domid_t domain_id;
     int num_pages;
     xenpaging_t *paging;
@@ -498,7 +509,6 @@ int main(int argc, char *argv[])
 
     int open_flags = O_CREAT | O_TRUNC | O_RDWR;
     mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
-    char filename[80];
     int fd;
 
     if ( argc != 3 )
@@ -520,7 +530,7 @@ int main(int argc, char *argv[])
     if ( paging == NULL )
     {
         ERROR("Error initialising paging");
-        goto out;
+        return 1;
     }
 
     /* Open file */
@@ -529,9 +539,18 @@ int main(int argc, char *argv[])
     if ( fd < 0 )
     {
         perror("failed to open file");
-        return -1;
+        return 2;
     }
 
+    /* ensure that if we get a signal, we'll do cleanup, then exit */
+    act.sa_handler = close_handler;
+    act.sa_flags = 0;
+    sigemptyset(&act.sa_mask);
+    sigaction(SIGHUP,  &act, NULL);
+    sigaction(SIGTERM, &act, NULL);
+    sigaction(SIGINT,  &act, NULL);
+    sigaction(SIGALRM, &act, NULL);
+
     /* Evict pages */
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
@@ -539,6 +558,8 @@ int main(int argc, char *argv[])
         rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
         if ( rc == -ENOSPC )
             break;
+        if ( rc == -EINTR )
+            break;
         if ( i % 100 == 0 )
             DPRINTF("%d pages evicted\n", i);
     }
@@ -546,7 +567,7 @@ int main(int argc, char *argv[])
     DPRINTF("pages evicted\n");
 
     /* Swap pages in and out */
-    while ( 1 )
+    while ( !interrupted )
     {
         /* Wait for Xen to signal that a page needs paged in */
         rc = xc_wait_for_event_or_timeout(xch, paging->mem_event.xce_handle, 100);
@@ -637,8 +658,10 @@ int main(int argc, char *argv[])
             }
         }
     }
+    DPRINTF("xenpaging got signal %d\n", interrupted);
 
  out:
+    close(fd);
     free(victims);
 
     /* Tear down domain paging */
@@ -651,6 +674,7 @@ int main(int argc, char *argv[])
 
     xc_interface_close(xch);
 
+    DPRINTF("xenpaging exit code %d\n", rc);
     return rc;
 }
#6 lore raw

Re: [PATCH 05/21] xenpaging: add signal handling

Olaf, I haven't been looking at these patches as we've been going along, but there seem to be two things happening in this patch not mentioned in the description:

* Making xenpaging_teardown() not skip when a tear-down item fails, but continue to try to tear down the rest
* Making return values for the program as a whole (1 for initializing the paging, 2 for a failed file open)

These kinds of things should at least be mentioned in the description; and I would personally probably pull them out and put them in a separate patch.

-George

Quoted 180 lines · Olaf Hering

Leave paging loop if xenpaging gets a signal.
Remove paging file on exit.

---
v2:
 unlink pagefile in signal handler to avoid stale pagefiles if xenpaging is stuck in some loop

 tools/xenpaging/xenpaging.c |   42 +++++++++++++++++++++++++++++++++---------

 1 file changed, 33 insertions(+), 9 deletions(-)
tools/xenpaging/xenpaging.c +31 -9
@@ -25,6 +25,7 @@
  #include <stdlib.h>
  #include <stdarg.h>
  #include <time.h>
+#include <signal.h>
  #include <xc_private.h>
 
  #include <xen/mem_event.h>
@@ -43,6 +44,14 @@
  #define DPRINTF(...) ((void)0)
  #endif
 
+static char filename[80];
+static int interrupted;
+static void close_handler(int sig)
+{
+    interrupted = sig;
+    if ( filename[0] )
+        unlink(filename);
+}
 
  static void *init_page(void)
  {
@@ -248,7 +257,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error tearing down domain paging in xen");
-        goto err;
     }
 
     /* Unbind VIRQ */
@@ -256,7 +264,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error unbinding event port");
-        goto err;
     }
     paging->mem_event.port = -1;
 
@@ -265,7 +272,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing event channel");
-        goto err;
     }
     paging->mem_event.xce_handle = -1;
 
@@ -274,7 +280,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing connection to xen");
-        goto err;
     }
     paging->xc_handle = NULL;
 
@@ -380,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
     return ret;
  }
 
-int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
+static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
  {
     int ret;
 
@@ -461,6 +466,11 @@ static int evict_victim(xc_interface *xc
             goto out;
         }
 
+        if ( interrupted )
+        {
+            ret = -EINTR;
+            goto out;
+        }
         ret = xc_mem_paging_nominate(paging->xc_handle,
                                      paging->mem_event.domain_id, victim->gfn);
         if ( ret == 0 )
@@ -485,6 +495,7 @@ static int evict_victim(xc_interface *xc
 
  int main(int argc, char *argv[])
  {
+    struct sigaction act;
     domid_t domain_id;
     int num_pages;
     xenpaging_t *paging;
@@ -498,7 +509,6 @@ int main(int argc, char *argv[])
 
     int open_flags = O_CREAT | O_TRUNC | O_RDWR;
     mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
-    char filename[80];
     int fd;
 
     if ( argc != 3 )
@@ -520,7 +530,7 @@ int main(int argc, char *argv[])
     if ( paging == NULL )
     {
         ERROR("Error initialising paging");
-        goto out;
+        return 1;
     }
 
     /* Open file */
@@ -529,9 +539,18 @@ int main(int argc, char *argv[])
     if ( fd < 0 )
     {
         perror("failed to open file");
-        return -1;
+        return 2;
     }
 
+    /* ensure that if we get a signal, we'll do cleanup, then exit */
+    act.sa_handler = close_handler;
+    act.sa_flags = 0;
+    sigemptyset(&act.sa_mask);
+    sigaction(SIGHUP,  &act, NULL);
+    sigaction(SIGTERM, &act, NULL);
+    sigaction(SIGINT,  &act, NULL);
+    sigaction(SIGALRM, &act, NULL);
+
     /* Evict pages */
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
@@ -539,6 +558,8 @@ int main(int argc, char *argv[])
         rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
         if ( rc == -ENOSPC )
             break;
+        if ( rc == -EINTR )
+            break;
         if ( i % 100 == 0 )
             DPRINTF("%d pages evicted\n", i);
     }
@@ -546,7 +567,7 @@ int main(int argc, char *argv[])
     DPRINTF("pages evicted\n");
 
     /* Swap pages in and out */
-    while ( 1 )
+    while ( !interrupted )
     {
         /* Wait for Xen to signal that a page needs paged in */
         rc = xc_wait_for_event_or_timeout(xch, paging->mem_event.xce_handle, 100);
@@ -637,8 +658,10 @@ int main(int argc, char *argv[])
             }
         }
     }
+    DPRINTF("xenpaging got signal %d\n", interrupted);
 

 out:
+    close(fd);
    free(victims);

    /* Tear down domain paging */

tools/xenpaging/xenpaging.c +1 -0
@@ -651,6 +674,7 @@ int main(int argc, char *argv[])
 
     xc_interface_close(xch);
 
+    DPRINTF("xenpaging exit code %d\n", rc);
     return rc;
  }
 

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#7 lore raw

Re: [PATCH 05/21] xenpaging: add signal handling

Quoted 3 lines · George Dunlap

Olaf, I haven't been looking at these patches as we've been going along, but there seem to be two things happening in this patch not mentioned in the description:

This was a "grown" patch.

Quoted 2 lines

* Making xenpaging_teardown() not skip when a tear-down item fails, but continue to try to tear down the rest

If a domain is shutting down, xc_mem_event_disable will always fail because d->is_dying is checked. Thats why I removed the bail_out part.

Quoted 2 lines

* Making return values for the program as a whole (1 for initializing the paging, 2 for a failed file open)

return codes are currently not perfect, sometimes -1 is leaked. I think xenpaging should either return 0 or 1.

Quoted 3 lines

These kinds of things should at least be mentioned in the description; and I would personally probably pull them out and put them in a separate patch.

Will do better next time.
Thanks for the comment.

Olaf

#8 lore raw

Re: [PATCH 05/21] xenpaging: add signal handling

Yeah, they certainly seem like reasonable changes. They just need to be pointed out, so that people have an idea the actual impact of the patch (both now when accepting it, and later when going back trying to figure out where something broke).

FYI, I normally do a bunch of work (involving a large number of changes) first, then do "hg diff > working.diff ; hg update -C" and then go through working.diff to see where the individual changes are best suited, whether in a new patch, or in a modification to an existing patch. (emacs diff-mode is really helpful here.) That helps me review my own code, and promotes good patch hygiene. :-)

Peace,
-George

Quoted 36 lines · Olaf Hering
Quoted 3 lines · George Dunlap

Olaf, I haven't been looking at these patches as we've been going along, but there seem to be two things happening in this patch not mentioned in the description:

This was a "grown" patch.

Quoted 2 lines

* Making xenpaging_teardown() not skip when a tear-down item fails, but continue to try to tear down the rest

If a domain is shutting down, xc_mem_event_disable will always fail because d->is_dying is checked. Thats why I removed the bail_out part.

Quoted 2 lines

* Making return values for the program as a whole (1 for initializing the paging, 2 for a failed file open)

return codes are currently not perfect, sometimes -1 is leaked. I think xenpaging should either return 0 or 1.

Quoted 3 lines

These kinds of things should at least be mentioned in the description; and I would personally probably pull them out and put them in a separate patch.

Will do better next time.
Thanks for the comment.

Olaf

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#9 lore raw

PATCH 6/21 xenpaging: print info when free request slots drop below 2

In reply to Olaf Hering

Add debugging aid to free request slots in the ring buffer.
It should not happen that the ring gets full, print info anyway if it happens.

---
 xen/arch/x86/mm/mem_event.c |    5 +++++
 1 file changed, 5 insertions(+)
xen/arch/x86/mm/mem_event.c +5 -0
@@ -152,6 +152,11 @@ int mem_event_check_ring(struct domain *
     mem_event_ring_lock(d);
 
     free_requests = RING_FREE_REQUESTS(&d->mem_event.front_ring);
+    if ( unlikely(free_requests < 2) )
+    {
+        gdprintk(XENLOG_INFO, "free request slots: %d\n", free_requests);
+        WARN_ON(free_requests == 0);
+    }
     ring_full = free_requests < MEM_EVENT_RING_THRESHOLD;
 
     if ( (curr->domain->domain_id == d->domain_id) && ring_full )
#10 lore raw

PATCH 7/21 xenpaging: print p2mt for already paged-in pages

In reply to Olaf Hering

Add more debug output, print p2mt for pages which were requested more than once.

---
 tools/xenpaging/xenpaging.c |    2 ++
 1 file changed, 2 insertions(+)
tools/xenpaging/xenpaging.c +2 -0
@@ -634,8 +634,10 @@ int main(int argc, char *argv[])
             else
             {
                 DPRINTF("page already populated (domain = %d; vcpu = %d;"
+                        " p2mt = %x;"
                         " gfn = %"PRIx64"; paused = %"PRId64")\n",
                         paging->mem_event.domain_id, req.vcpu_id,
+                        req.p2mt,
                         req.gfn, req.flags & MEM_EVENT_FLAG_VCPU_PAUSED);
 
                 /* Tell Xen to resume the vcpu */
#11 lore raw

PATCH 8/21 xenpaging: notify policy only on resume

In reply to Olaf Hering

If a page is requested more than once, the policy is also notified more than once about the page-in. However, a page-in happens only once. Any further resume will only unpause the other vcpu. The multiple notify will put the page into the mru list multiple times and it will unlock other already resumed pages too early. In the worst case, a page that was just resumed can be evicted right away, causing a deadlock in the guest.

---
 tools/xenpaging/xenpaging.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
tools/xenpaging/xenpaging.c +5 -4
@@ -385,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
     return ret;
 }
 
-static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
+static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp, int notify_policy)
 {
     int ret;
 
@@ -395,7 +395,8 @@ static int xenpaging_resume_page(xenpagi
         goto out;
 
     /* Notify policy of page being paged in */
-    policy_notify_paged_in(paging->mem_event.domain_id, rsp->gfn);
+    if ( notify_policy )
+        policy_notify_paged_in(paging->mem_event.domain_id, rsp->gfn);
 
     /* Tell Xen page is ready */
     ret = xc_mem_paging_resume(paging->xc_handle, paging->mem_event.domain_id,
@@ -621,7 +622,7 @@ int main(int argc, char *argv[])
                 rsp.vcpu_id = req.vcpu_id;
                 rsp.flags = req.flags;
 
-                rc = xenpaging_resume_page(paging, &rsp);
+                rc = xenpaging_resume_page(paging, &rsp, 1);
                 if ( rc != 0 )
                 {
                     ERROR("Error resuming page");
@@ -650,7 +651,7 @@ int main(int argc, char *argv[])
                     rsp.vcpu_id = req.vcpu_id;
                     rsp.flags = req.flags;
 
-                    rc = xenpaging_resume_page(paging, &rsp);
+                    rc = xenpaging_resume_page(paging, &rsp, 0);
                     if ( rc != 0 )
                     {
                         ERROR("Error resuming");
#12 lore raw

PATCH 9/21 xenpaging: allow negative num_pages and limit num_pages

In reply to Olaf Hering

Simplify paging size argument. If a negative number is specified, it means the entire guest memory should be paged out. This is useful for debugging. Also limit num_pages to the guests max_pages.

---
 tools/xenpaging/xenpaging.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
tools/xenpaging/xenpaging.c +7 -2
@@ -521,8 +521,6 @@ int main(int argc, char *argv[])
     domain_id = atoi(argv[1]);
     num_pages = atoi(argv[2]);
 
-    victims = calloc(num_pages, sizeof(xenpaging_victim_t));
-
     /* Seed random-number generator */
     srand(time(NULL));
 
@@ -543,6 +541,13 @@ int main(int argc, char *argv[])
         return 2;
     }
 
+    if ( num_pages < 0 || num_pages > paging->domain_info->max_pages )
+    {
+        num_pages = paging->domain_info->max_pages;
+        DPRINTF("setting num_pages to %d\n", num_pages);
+    }
+    victims = calloc(num_pages, sizeof(xenpaging_victim_t));
+
     /* ensure that if we get a signal, we'll do cleanup, then exit */
     act.sa_handler = close_handler;
     act.sa_flags = 0;
#13 lore raw

PATCH 10/21 xenpaging: when populating a page, check if populating is already in progress

In reply to Olaf Hering

p2m_mem_paging_populate can be called serveral times from different vcpus. If the page is already in state p2m_ram_paging_in and has a new valid mfn, invalidating this new mfn will cause trouble later if p2m_mem_paging_resume will set the new gfn/mfn pair back to state p2m_ram_rw.
Detect this situation and keep p2m state if the page is in the process of being still paged-out or already paged-in.
In fact, p2m state p2m_ram_paged is the only state where the mfn type can be invalidated.

---
 xen/arch/x86/mm/p2m.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/mm/p2m.c +1 -1
@@ -2767,7 +2767,7 @@ void p2m_mem_paging_populate(struct p2m_
     /* XXX: It seems inefficient to have this here, as it's only needed
      *      in one case (ept guest accessing paging out page) */
     gfn_to_mfn(p2m, gfn, &p2mt);
-    if ( p2mt != p2m_ram_paging_out )
+    if ( p2mt == p2m_ram_paged )
     {
         p2m_lock(p2m);
         set_p2m_entry(p2m, gfn, _mfn(PAGING_MFN), 0, p2m_ram_paging_in_start);
#14 lore raw

PATCH 11/21 xenpaging: optimize p2m_mem_paging_populate

In reply to Olaf Hering

p2m_mem_paging_populate will always put another request in the ring. To reduce pressure on the ring, place only required requests in the ring. If the gfn was already processed by another thread, and the current vcpu does not need to be paused, p2m_mem_paging_resume will do nothing with the request. And also xenpaging will drop the request if the vcpu does not need a wakeup.

---
 xen/arch/x86/mm/p2m.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
xen/arch/x86/mm/p2m.c +7 -2
@@ -2757,12 +2757,12 @@ void p2m_mem_paging_populate(struct p2m_
     p2m_type_t p2mt;
     struct domain *d = p2m->domain;
 
-    memset(&req, 0, sizeof(req));
-
     /* Check that there's space on the ring for this request */
     if ( mem_event_check_ring(d) )
         return;
 
+    memset(&req, 0, sizeof(req));
+
     /* Fix p2m mapping */
     /* XXX: It seems inefficient to have this here, as it's only needed
      *      in one case (ept guest accessing paging out page) */
@@ -2781,6 +2781,11 @@ void p2m_mem_paging_populate(struct p2m_
         vcpu_pause_nosync(v);
         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
     }
+    else if ( p2mt != p2m_ram_paging_out && p2mt != p2m_ram_paged )
+    {
+        /* gfn is already on its way back and vcpu is not paused */
+        return;
+    }
 
     /* Send request to pager */
     req.gfn = gfn;
#15 lore raw

PATCH 12/21 xenpaging: print xenpaging cmdline options

In reply to Olaf Hering

Print xenpaging arguments to simplify domain_id mapping from xenpaging logfile to other logfiles and Xen console output.

---
 tools/xenpaging/xenpaging.c |    2 ++
 1 file changed, 2 insertions(+)
tools/xenpaging/xenpaging.c +2 -0
@@ -532,6 +532,8 @@ int main(int argc, char *argv[])
         return 1;
     }
 
+    DPRINTF("starting %s %u %d\n", argv[0], domain_id, num_pages);
+
     /* Open file */
     sprintf(filename, "page_cache_%d", domain_id);
     fd = open(filename, open_flags, open_mode);
#16 lore raw

PATCH 13/21 xenpaging: handle temporary out-of-memory conditions during page-in

In reply to Olaf Hering

p2m_mem_paging_prep() should return -ENOMEM if a new page could not be allocated. This can be handled in xenpaging to retry the page-in. Right now such condition would stall the guest because the requested page will not come back, xenpaging simply exits. So xenpaging could very well retry the allocation forever to rescue the guest.

---
 tools/xenpaging/xenpaging.c |   27 ++++++++++++++++++++-------
 xen/arch/x86/mm/p2m.c       |    2 +-
 2 files changed, 21 insertions(+), 8 deletions(-)
tools/xenpaging/xenpaging.c +20 -7
@@ -26,6 +26,7 @@
 #include <stdarg.h>
 #include <time.h>
 #include <signal.h>
+#include <unistd.h>
 #include <xc_private.h>
 
 #include <xen/mem_event.h>
@@ -415,19 +416,31 @@ static int xenpaging_populate_page(
     unsigned long _gfn;
     void *page;
     int ret;
+    unsigned char oom = 0;
 
-    /* Tell Xen to allocate a page for the domain */
-    ret = xc_mem_paging_prep(paging->xc_handle, paging->mem_event.domain_id,
-                             *gfn);
-    if ( ret != 0 )
+    _gfn = *gfn;
+    do
     {
-        ERROR("Error preparing for page in");
-        goto out_map;
+        /* Tell Xen to allocate a page for the domain */
+        ret = xc_mem_paging_prep(paging->xc_handle, paging->mem_event.domain_id,
+                                 _gfn);
+        if ( ret != 0 )
+        {
+            if ( errno == ENOMEM )
+            {
+                if ( oom++ == 0 )
+                    DPRINTF("ENOMEM while preparing gfn %lx\n", _gfn);
+                sleep(1);
+                continue;
+            }
+            ERROR("Error preparing for page in");
+            goto out_map;
+        }
     }
+    while ( ret && !interrupted );
 
     /* Map page */
     ret = -EFAULT;
-    _gfn = *gfn;
     page = xc_map_foreign_pages(paging->xc_handle, paging->mem_event.domain_id,
                                 PROT_READ | PROT_WRITE, &_gfn, 1);
     *gfn = _gfn;
xen/arch/x86/mm/p2m.c +1 -1
@@ -2802,7 +2802,7 @@ int p2m_mem_paging_prep(struct p2m_domai
     /* Get a free page */
     page = alloc_domheap_page(p2m->domain, 0);
     if ( unlikely(page == NULL) )
-        return -EINVAL;
+        return -ENOMEM;
 
     /* Fix p2m mapping */
     p2m_lock(p2m);
#17 lore raw

PATCH 14/21 xenpaging: increase recently used pages from 4MB to 64MB

In reply to Olaf Hering

Increase recently used pages from 4MB to 64MB.
Keeping more pages in memory allows the guest to make more progress if the paging file spans the entire guest memory.

---
 tools/xenpaging/policy_default.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/xenpaging/policy_default.c +1 -1
@@ -26,7 +26,7 @@
 #include "policy.h"
 
 
-#define MRU_SIZE 1024
+#define MRU_SIZE (1024 * 16)
 
 
 static unsigned long mru[MRU_SIZE];
#18 lore raw

PATCH 15/21 xenpaging: update machine_to_phys_mapping during page-in and page deallocation

In reply to Olaf Hering

The machine_to_phys_mapping array needs updating during page-in, and during page deallocation. If a page is gone, a call to get_gpfn_from_mfn will still return the old gfn for an already paged-out page. This happens when the entire guest ram is paged-out before xen_vga_populate_vram() runs. Then XENMEM_populate_physmap is called with gfn 0xff000. A new page is allocated with alloc_domheap_pages. This new page does not have a gfn yet. However, in guest_physmap_add_entry() the passed mfn maps still to an old gfn (perhaps from another old guest). This old gfn is in paged-out state in this guests context and has no mfn anymore. As a result, the ASSERT() triggers because p2m_is_ram() is true for p2m_ram_paging* types.

If the machine_to_phys_mapping array is updated properly, both loops in guest_physmap_add_entry() turn into no-ops for the new page and the mfn/gfn mapping will be done at the end of the function.

The same thing needs to happen dring a page-in, the current gfn must be written for the page.

If XENMEM_add_to_physmap is used with XENMAPSPACE_gmfn, get_gpfn_from_mfn() will return an appearently valid gfn. As a result, guest_physmap_remove_page() is called. The ASSERT in p2m_remove_page triggers because the passed mfn does not match the old mfn for the passed gfn.

---
v3:
invalidate machine_to_phys_mapping[] during page deallocation
v2:
call set_gpfn_from_mfn only if mfn is valid

---
 xen/arch/x86/mm/p2m.c   |   15 +++++++++++----
 xen/common/page_alloc.c |    9 +++++++++
 2 files changed, 20 insertions(+), 4 deletions(-)
xen/arch/x86/mm/p2m.c +11 -4
@@ -2825,10 +2825,17 @@ void p2m_mem_paging_resume(struct p2m_do
 
     /* Fix p2m entry */
     mfn = gfn_to_mfn(p2m, rsp.gfn, &p2mt);
-    p2m_lock(p2m);
-    set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
-    audit_p2m(p2m, 1);
-    p2m_unlock(p2m);
+    if ( mfn_valid(mfn) )
+    {
+        p2m_lock(p2m);
+        set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
+        set_gpfn_from_mfn(mfn_x(mfn), rsp.gfn);
+        audit_p2m(p2m, 1);
+        p2m_unlock(p2m);
+    } else {
+        gdprintk(XENLOG_ERR, "invalid mfn %lx for gfn %lx p2mt %x flags %lx\n",
+            mfn_x(mfn), rsp.gfn, p2mt, (unsigned long)rsp.flags);
+    }
 
     /* Unpause domain */
     if ( rsp.flags & MEM_EVENT_FLAG_VCPU_PAUSED )
xen/common/page_alloc.c +9 -0
@@ -1199,9 +1199,18 @@ void free_domheap_pages(struct page_info
 {
     int            i, drop_dom_ref;
     struct domain *d = page_get_owner(pg);
+    unsigned long mfn;
 
     ASSERT(!in_irq());
 
+    /* this page is not a gfn anymore */
+    mfn = page_to_mfn(pg);
+    if ( mfn_valid(mfn) )
+    {
+        for ( i = 0; i < (1 << order); i++ )
+            set_gpfn_from_mfn(mfn + i, INVALID_M2P_ENTRY);
+    }
+
     if ( unlikely(is_xen_heap_page(pg)) )
     {
         /* NB. May recursively lock from relinquish_memory(). */
#19 lore raw

PATCH 16/21 xenpaging: drop paged pages in guest_remove_page

In reply to Olaf Hering

Simply drop paged-pages in guest_remove_page(), and notify xenpaging to drop its reference to the gfn.

---
v2:
resume dropped page to unpause vcpus

 tools/xenpaging/xenpaging.c    |   17 +++++++---
 xen/arch/x86/mm/p2m.c          |   68 +++++++++++++++++++++++++++++++----------
 xen/common/memory.c            |    6 +++
 xen/include/asm-x86/p2m.h      |    4 ++
 xen/include/public/mem_event.h |    1 
 5 files changed, 75 insertions(+), 21 deletions(-)
tools/xenpaging/xenpaging.c +12 -5
@@ -628,12 +628,19 @@ int main(int argc, char *argv[])
                     goto out;
                 }
                 
-                /* Populate the page */
-                rc = xenpaging_populate_page(xch, paging, &req.gfn, fd, i);
-                if ( rc != 0 )
+                if ( req.flags & MEM_EVENT_FLAG_DROP_PAGE )
                 {
-                    ERROR("Error populating page");
-                    goto out;
+                    DPRINTF("Dropping page %"PRIx64" p2mt %x\n", req.gfn, req.p2mt);
+                }
+                else
+                {
+                    /* Populate the page */
+                    rc = xenpaging_populate_page(xch, paging, &req.gfn, fd, i);
+                    if ( rc != 0 )
+                    {
+                        ERROR("Error populating page");
+                        goto out;
+                    }
                 }
 
                 /* Prepare the response */
xen/arch/x86/mm/p2m.c +52 -16

88 lines not on this page · raw message

xen/common/memory.c +6 -0
@@ -162,6 +162,12 @@ int guest_remove_page(struct domain *d,
 
 #ifdef CONFIG_X86
     mfn = mfn_x(gfn_to_mfn(p2m_get_hostp2m(d), gmfn, &p2mt)); 
+    if ( unlikely(p2m_is_paging(p2mt)) )
+    {
+        guest_physmap_remove_page(d, gmfn, mfn, 0);
+        p2m_mem_paging_drop_page(p2m_get_hostp2m(d), gmfn);
+        return 1;
+    }
 #else
     mfn = gmfn_to_mfn(d, gmfn);
 #endif
xen/include/asm-x86/p2m.h +4 -0
@@ -471,6 +471,8 @@ int set_shared_p2m_entry(struct p2m_doma
 int p2m_mem_paging_nominate(struct p2m_domain *p2m, unsigned long gfn);
 /* Evict a frame */
 int p2m_mem_paging_evict(struct p2m_domain *p2m, unsigned long gfn);
+/* Tell xenpaging to drop a paged out frame */
+void p2m_mem_paging_drop_page(struct p2m_domain *p2m, unsigned long gfn);
 /* Start populating a paged out frame */
 void p2m_mem_paging_populate(struct p2m_domain *p2m, unsigned long gfn);
 /* Prepare the p2m for paging a frame in */
@@ -478,6 +480,8 @@ int p2m_mem_paging_prep(struct p2m_domai
 /* Resume normal operation (in case a domain was paused) */
 void p2m_mem_paging_resume(struct p2m_domain *p2m);
 #else
+static inline void p2m_mem_paging_drop_page(struct p2m_domain *p2m, unsigned long gfn)
+{ }
 static inline void p2m_mem_paging_populate(struct p2m_domain *p2m, unsigned long gfn)
 { }
 #endif
xen/include/public/mem_event.h +1 -0
@@ -37,6 +37,7 @@
 #define MEM_EVENT_FLAG_VCPU_PAUSED  (1 << 0)
 #define MEM_EVENT_FLAG_DOM_PAUSED   (1 << 1)
 #define MEM_EVENT_FLAG_OUT_OF_MEM   (1 << 2)
+#define MEM_EVENT_FLAG_DROP_PAGE    (1 << 3)
 
 
 typedef struct mem_event_shared_page {
#20 lore raw

PATCH 17/21 xenpaging: handle HVMCOPY_gfn_paged_out in copy_from/to_user

In reply to Olaf Hering

This is a quick fix to keep my guests running. The proper fix will use the recently added waitqueue API.

copy_from_user_hvm can fail when __hvm_copy returns HVMCOPY_gfn_paged_out for a referenced gfn, for example during guests pagetable walk. This has to be handled in some way. One hypercall that failed was do_memory_op/XENMEM_decrease_reservation which lead to a BUG_ON balloon.c. Since do_memory_op already has restart support for the hypercall, copy_from_guest uses this existing retry code. In addition, cleanup on error was added to increase_reservation and populate_physmap.

---
 xen/arch/x86/hvm/hvm.c |    4 ++++
 xen/common/memory.c    |   44 +++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 43 insertions(+), 5 deletions(-)
xen/arch/x86/hvm/hvm.c +4 -0
@@ -2066,6 +2066,8 @@ unsigned long copy_to_user_hvm(void *to,
 
     rc = hvm_copy_to_guest_virt_nofault((unsigned long)to, (void *)from,
                                         len, 0);
+    if ( rc == HVMCOPY_gfn_paged_out )
+        return -EAGAIN;
     return rc ? len : 0; /* fake a copy_to_user() return code */
 }
 
@@ -2083,6 +2085,8 @@ unsigned long copy_from_user_hvm(void *t
 #endif
 
     rc = hvm_copy_from_guest_virt_nofault(to, (unsigned long)from, len, 0);
+    if ( rc == HVMCOPY_gfn_paged_out )
+        return -EAGAIN;
     return rc ? len : 0; /* fake a copy_from_user() return code */
 }
 
xen/common/memory.c +39 -5

103 lines not on this page · raw message

#21 lore raw

PATCH 18/21 xenpaging: prevent page-out of first 16MB

In reply to Olaf Hering

This is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages, crashes will occour due to lack of support of xenpaging.

---
 tools/xenpaging/policy_default.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/xenpaging/policy_default.c +3 -2
@@ -60,8 +60,9 @@ int policy_init(xenpaging_t *paging)
     for ( i = 0; i < MRU_SIZE; i++ )
         mru[i] = INVALID_MFN;
 
-    /* Don't page out page 0 */
-    set_bit(0, bitmap);
+    /* Don't page out first 16MB */
+    for ( i = 0; i < ((16*1024*1024)/4096); i++ )
+        set_bit(i, bitmap);
 
  out:
     return rc;
#22 lore raw

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

Quoted 4 lines · Olaf Hering

This is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages, crashes will occour due to lack of support of xenpaging.

While looking at this again, I came up with this change. Any idea whats at 512K during BIOS startup?

Subject: xenpaging: prevent page-out of gfn 0x80

Add a workaround for missing handling of paged-out pages during BIOS startup. For some reason, only gfn 0x80 is affected.

(XEN) HVM3: HVM Loader
(XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000 (XEN) HVM3: Detected Xen v4.1
(XEN) HVM3: HVMLoader bug at util.c:604
(XEN) hvm.c:1085:d3 Triple fault on VCPU0 - invoking HVM system reset.

(XEN) HVM4: HVM Loader
(XEN) traps.c:649:d4 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000 (XEN) io.c:194:d4 MMIO emulation failed @ 0018:9ffff: 00 6a 10 80 c4 82 (XEN) hvm.c:1085:d4 Triple fault on VCPU0 - invoking HVM system reset.

---
 tools/xenpaging/policy_default.c |    3 +++
 1 file changed, 3 insertions(+)
tools/xenpaging/policy_default.c +3 -0
@@ -76,6 +76,9 @@ int policy_init(xenpaging_t *paging)
     /* Don't page out page 0 */
     set_bit(0, bitmap);
 
+    /* this is the trap page, disabled for the time being */
+    set_bit(0x80, bitmap);
+
  out:
     return rc;
 }
#23 lore raw

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

Quoted 12 lines
Quoted 1 line
Quoted 1 line · Olaf Hering
Quoted 4 lines · Olaf Hering

This is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages, crashes will occour due to lack of support of xenpaging.

While looking at this again, I came up with this change. Any idea whats at 512K during BIOS startup?

Quoted 8 lines

Subject: xenpaging: prevent page-out of gfn 0x80

Add a workaround for missing handling of paged-out pages during BIOS startup.
For some reason, only gfn 0x80 is affected.

(XEN) HVM3: HVM Loader
(XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000

This clearly is another place where page-in needs to be triggered. Isn't it the case that *any* gmfn_to_mfn() needs this special treatment? Couldn't therefore the translation function itself deal with requesting the page-in (and waiting for its completion, now that this can be done)?

Jan

#24 lore raw

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

Quoted 14 lines · Jan Beulich
Quoted 14 lines
Quoted 1 line
Quoted 1 line · Olaf Hering
Quoted 4 lines · Olaf Hering

This is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages, crashes will occour due to lack of support of xenpaging.

While looking at this again, I came up with this change. Any idea whats at 512K during BIOS startup?

Subject: xenpaging: prevent page-out of gfn 0x80

Quoted 8 lines
Quoted 6 lines

Add a workaround for missing handling of paged-out pages during BIOS startup.
For some reason, only gfn 0x80 is affected.

(XEN) HVM3: HVM Loader
(XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000

This clearly is another place where page-in needs to be triggered.

Yes, thats true.
I had a printk in gfn_to_mfn for some months, and it did not trigger during my testing. But then, I also had the patch which keeps the first 16MB in memory.

So as it stands, gfn_to_mfn() should call p2m_mem_paging_populate() at least, even if that does not fix this crash.
I think the few gfn_to_mfn* variants should not return the paging types anymore, and just sleep until the page is back. Thats something for next year, as time runs out for me.

And: I think the various redefines of those functions/macros can go as well.

Olaf

#25 lore raw

PATCH 19/21 xenpaging: start xenpaging via config option

In reply to Olaf Hering

Start xenpaging via config option.

TODO: add config option for different pagefile directory
TODO: add libxl support
TODO: parse config values like 42K, 42M, 42G, 42%

---
v2:
unlink logfile instead of truncating it.
allows hardlinking for further inspection

 tools/examples/xmexample.hvm            |    3 +
 tools/python/README.XendConfig          |    1 
 tools/python/README.sxpcfg              |    1 
 tools/python/xen/xend/XendConfig.py     |    3 +
 tools/python/xen/xend/XendDomainInfo.py |    6 ++
 tools/python/xen/xend/image.py          |   91 ++++++++++++++++++++++++++++++++
 tools/python/xen/xm/create.py           |    5 +
 tools/python/xen/xm/xenapi_create.py    |    1 
 8 files changed, 111 insertions(+)
tools/examples/xmexample.hvm +3 -0
@@ -127,6 +127,9 @@ disk = [ 'file:/var/images/min-el3-i386.
 # Device Model to be used
 device_model = 'qemu-dm'
 
+# xenpaging, number of pages
+xenpaging = 42
+
 #-----------------------------------------------------------------------------
 # boot on floppy (a), hard disk (c), Network (n) or CD-ROM (d) 
 # default: hard disk, cd-rom, floppy
tools/python/README.XendConfig +1 -0
@@ -120,6 +120,7 @@ otherConfig
                                 image.vncdisplay
                                 image.vncunused
                                 image.hvm.device_model
+                                image.hvm.xenpaging
                                 image.hvm.display
                                 image.hvm.xauthority
                                 image.hvm.vncconsole
tools/python/README.sxpcfg +1 -0
@@ -51,6 +51,7 @@ image
   - vncunused
   (HVM)
   - device_model
+  - xenpaging
   - display
   - xauthority
   - vncconsole
tools/python/xen/xend/XendConfig.py +3 -0
@@ -147,6 +147,7 @@ XENAPI_PLATFORM_CFG_TYPES = {
     'apic': int,
     'boot': str,
     'device_model': str,
+    'xenpaging': int,
     'loader': str,
     'display' : str,
     'fda': str,
@@ -508,6 +509,8 @@ class XendConfig(dict):
             self['platform']['nomigrate'] = 0
 
         if self.is_hvm():
+            if 'xenpaging' not in self['platform']:
+                self['platform']['xenpaging'] = None
             if 'timer_mode' not in self['platform']:
                 self['platform']['timer_mode'] = 1
             if 'viridian' not in self['platform']:
tools/python/xen/xend/XendDomainInfo.py +6 -0
@@ -2390,6 +2390,7 @@ class XendDomainInfo:
 
         if self.image:
             self.image.createDeviceModel()
+            self.image.createXenPaging()
 
         #if have pass-through devs, need the virtual pci slots info from qemu
         self.pci_device_configure_boot()
@@ -2402,6 +2403,11 @@ class XendDomainInfo:
                 self.image.destroyDeviceModel()
             except Exception, e:
                 log.exception("Device model destroy failed %s" % str(e))
+            try:
+                log.debug("stopping xenpaging")
+                self.image.destroyXenPaging()
+            except Exception, e:
+                log.exception("stopping xenpaging failed %s" % str(e))
         else:
             log.debug("No device model")
 
tools/python/xen/xend/image.py +91 -0

109 lines not on this page · raw message

tools/python/xen/xm/create.py +5 -0
@@ -491,6 +491,10 @@ gopts.var('nfs_root', val="PATH",
           fn=set_value, default=None,
           use="Set the path of the root NFS directory.")
 
+gopts.var('xenpaging', val='NUM',
+          fn=set_int, default=None,
+          use="Number of pages to swap.")
+
 gopts.var('device_model', val='FILE',
           fn=set_value, default=None,
           use="Path to device model program.")
@@ -1076,6 +1080,7 @@ def configure_hvm(config_image, vals):
     args = [ 'acpi', 'apic',
              'boot',
              'cpuid', 'cpuid_check',
+             'xenpaging',
              'device_model', 'display',
              'fda', 'fdb',
              'gfx_passthru', 'guest_os_type',
tools/python/xen/xm/xenapi_create.py +1 -0
@@ -1085,6 +1085,7 @@ class sxp2xml:
             'acpi',
             'apic',
             'boot',
+            'xenpaging',
             'device_model',
             'loader',
             'fda',
#26 lore raw

PATCH 20/21 xenpaging: add dynamic startup delay for xenpaging

In reply to Olaf Hering

This is a debug helper. Since the xenpaging support is still fragile, run xenpaging at different stages in the bootprocess. Different delays will trigger more bugs. This implementation starts without delay for 5 reboots, then increments the delay by 0.1 seconds It uses xenstore for presistant storage of delay values

TODO: find the correct place to remove the xenstore directory when the guest is shutdown or crashed

---
 tools/python/xen/xend/image.py |   32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
tools/python/xen/xend/image.py +32 -0
@@ -123,6 +123,18 @@ class ImageHandler:
 
         self.device_model = vmConfig['platform'].get('device_model')
         self.xenpaging = vmConfig['platform'].get('xenpaging')
+        self.xenpaging_delay = xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay" % self.vm.info['name_label'])
+        if self.xenpaging_delay == None:
+            log.warn("XXX creating /local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
+            xstransact.Mkdir("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay', '0.0'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_inc', '0.1'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_use', '5'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_used', '0'))
+        self.xenpaging_delay = float(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay" % self.vm.info['name_label']))
+        self.xenpaging_delay_inc = float(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_inc" % self.vm.info['name_label']))
+        self.xenpaging_delay_use = int(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_use" % self.vm.info['name_label']))
+        self.xenpaging_delay_used = int(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_used" % self.vm.info['name_label']))
 
         self.display = vmConfig['platform'].get('display')
         self.xauthority = vmConfig['platform'].get('xauthority')
@@ -401,6 +413,17 @@ class ImageHandler:
             return
         if self.xenpaging_pid:
             return
+        if self.xenpaging_delay_used < self.xenpaging_delay_use:
+            self.xenpaging_delay_used += 1
+        else:
+            self.xenpaging_delay_used = 0
+            self.xenpaging_delay += self.xenpaging_delay_inc
+        log.info("delay_used %s" % self.xenpaging_delay_used)
+        log.info("delay_use %s" % self.xenpaging_delay_use)
+        log.info("delay %s" % self.xenpaging_delay)
+        log.info("delay_inc %s" % self.xenpaging_delay_inc)
+        xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay', self.xenpaging_delay))
+        xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_used', self.xenpaging_delay_used))
         xenpaging_bin = auxbin.pathTo("xenpaging")
         args = [xenpaging_bin]
         args = args + ([ "%d" % self.vm.getDomid()])
@@ -434,6 +457,9 @@ class ImageHandler:
                 except:
                     log.warn("chdir %s failed" % xenpaging_dir)
                 try:
+                    if self.xenpaging_delay != 0.0:
+                        log.info("delaying xenpaging startup %s seconds ..." % self.xenpaging_delay)
+                        time.sleep(self.xenpaging_delay)
                     log.info("starting %s" % args)
                     os.execve(xenpaging_bin, args, env)
                 except Exception, e:
@@ -449,10 +475,16 @@ class ImageHandler:
             self.xenpaging_pid = xenpaging_pid
             os.close(null)
             os.close(logfd)
+            if self.xenpaging_delay == 0.0:
+                log.warn("waiting for xenpaging ...")
+                time.sleep(22)
+                log.warn("waiting for xenpaging done.")
 
     def destroyXenPaging(self):
         if self.xenpaging is None:
             return
+        # FIXME find correct place for guest shutdown or crash
+        #xstransact.Remove("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
         if self.xenpaging_pid:
             try:
                 os.kill(self.xenpaging_pid, signal.SIGHUP)
#27 lore raw

PATCH 21/21 xenpaging: (sparse) documenation

In reply to Olaf Hering

Write up some sparse documentation about xenpaging usage.

---
 docs/misc/xenpaging.txt |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
docs/misc/xenpaging.txt new file +48 -0
@@ -0,0 +1,48 @@
+Warning:
+
+The xenpaging code is new and not fully debugged.
+Usage of xenpaging can crash Xen or cause severe data corruption in the
+guest memory and its filesystems!
+
+Description:
+
+xenpaging writes memory pages of a given guest to a file and moves the
+pages back to the pool of available memory.  Once the guests wants to
+access the paged-out memory, the page is read from disk and placed into
+memory.  This allows the sum of all running guests to use more memory
+than physically available on the host.
+
+Usage:
+
+Once the guest is running, run xenpaging with the guest_id and the
+number of pages to page-out:
+
+  chdir /var/lib/xen/xenpaging
+  xenpaging <guest_id>  <number_of_pages>
+
+To obtain the guest_id, run 'xm list'.
+xenpaging will write the pagefile to the current directory.
+Example with 128MB pagefile on guest 1:
+
+  xenpaging 1 32768
+
+Caution: stopping xenpaging manually will cause the guest to stall or
+crash because the paged-out memory is not written back into the guest!
+
+After a reboot of a guest, its guest_id changes, the current xenpaging
+binary has no target anymore. To automate restarting of xenpaging after
+guest reboot, specify the number if pages in the guest configuration
+file /etc/xen/vm/<guest_name>:
+
+xenpaging=32768
+
+Redo the guest with 'xm create /etc/xen/vm/<guest_name>' to activate the
+changes.
+
+
+Todo:
+- implement stopping of xenpaging
+- implement/test live migration
+
+
+# vim: tw=72
Acked-by on v1 21/21
#28 lore raw

Re: [PATCH 21/21] xenpaging: (sparse) documenation

Olaf Hering writes ("[Xen-devel] [PATCH 21/21] xenpaging: (sparse) documenation"):

Quoted 3 lines

Write up some sparse documentation about xenpaging usage.

Marvellous, thanks. I think this should be applied right away.

Ian.

#29 lore raw
In reply to Olaf Hering

Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):

Quoted 3 lines

Here are some changes for xenpaging in xen-unstable.

Patches 1 to 14 are likely non-controversial and could be applied.

Having eyeballed them (but not in depth) I'd like to give notice that I intend to apply the tools patches in 1-14 unless someone objects.

Thanks,
Ian.

#30 lore raw
Quoted 8 lines · Ian Jackson

Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):

Quoted 3 lines

Here are some changes for xenpaging in xen-unstable.

Patches 1 to 14 are likely non-controversial and could be applied.

Having eyeballed them (but not in depth) I'd like to give notice that I intend to apply the tools patches in 1-14 unless someone objects.

I think you are looking at a stale patch series. Olaf's latest runs 00-17, with patches 1-7 only considered by him suitable for immediate application.

K.

Quoted 7 lines

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#31 lore raw
Quoted 6 lines · Keir Fraser
Quoted 4 lines · Ian Jackson

Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):

Quoted 2 lines

Here are some changes for xenpaging in xen-unstable.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Patches 1 to 14 are likely non-controversial and could be applied.

Having eyeballed them (but not in depth) I'd like to give notice that I intend to apply the tools patches in 1-14 unless someone objects.

I think you are looking at a stale patch series. Olaf's latest runs 00-17, with patches 1-7 only considered by him suitable for immediate application.

All of those patches are to tools/xenpaging, so I'll flush them from my queue and leave them to you.

-- Keir

Quoted 11 lines

K.

Quoted 7 lines

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#32 lore raw
In reply to Keir Fraser

Keir Fraser writes ("Re: [Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):

Quoted 2 lines

I think you are looking at a stale patch series. Olaf's latest runs 00-17, with patches 1-7 only considered by him suitable for immediate application.

Oh, sorry. I hadn't seen the second series and I'm rather behind on the list.

Ian.