Skip to contentSkip to conversation

stubdom-dm: Support live migration of HVM domain with stubdom-dm

PATCH In review Yosuke Iwamatsu 3 participants 9 messages Dec 16, 2008 – Dec 17, 2008

lore
Share & tools
lore

Yosuke Iwamatsu

author
#0 lore raw

cover stubdom-dm: Support live migration of HVM domain with stubdom-dm

Hi,

This series of patches is an attempt to support live migration of hvm domains with stubdom-dm.

For hvm domains to live-migrate, we should create a log-dirty shared memory segment between xc_save and qemu-dm. for a normal qemu-dm, memory sharing is done by inter process communication interface (shmget/shmat/shmctl), but it obviously doesn't work for a stubdom-dm because it is not a dom0 process but a domain itself.

The idea here is to allocate log-dirty the shared memory segment in the stubdom-dm and to map the memory region into the xc_save process by using grant table interface.

[PATCH 1/4] mini-os: Make utility function get_self_id() public [PATCH 2/4] Log-dirty shared memory between stubdom-dm and xc_save [PATCH 3/4] Log-dirty shared memory between stubdom-dm and xc_save
(ioemu-remote part)
[PATCH 4/4] xend: Give stubdom flag to xc_save

Thanks,
-----------------------
Yosuke Iwamatsu
NEC Corporation

#1 lore raw
Quoted 2 lines · Yosuke Iwamatsu, le Tue 16 Dec 2008 18:22:20 +0900, a écrit :

This series of patches is an attempt to support live migration of hvm domains with stubdom-dm.

Err, I'm a bit surprised. I haven't tested it since then, but I'm sure live migration was working with stubdoms a few months ago, without the need for a logdirty shared buffer. This works thanks to the few lines at the end of the cpu_physical_memory_rw() function that tell the hypervisor about modified memory. Actually, the plan was even to use that bit of code in all cases (not only non-stubdom), so as to get rid of the logdirty shared buffer completely.

Samuel

#2 lore raw
Quoted 3 lines · Samuel Thibault, le Tue 16 Dec 2008 11:02:31 +0100, a écrit :

Actually, the plan was even to use that bit of code in all cases (not only non-stubdom), so as to get rid of the logdirty shared buffer completely.

(That even _helps_ the shadow code, which can thus know that a pagetable was replaced by data from the disk, for instance.)

Samuel

#3 lore raw
In reply to Samuel Thibault

Samuel Thibault writes ("[Xen-devel] Re: [PATCH 0/4] stubdom-dm: Support live migration of HVM domain with stubdom-dm"):

Quoted 11 lines
Quoted 2 lines · Yosuke Iwamatsu, le Tue 16 Dec 2008 18:22:20 +0900, a écrit :

This series of patches is an attempt to support live migration of hvm domains with stubdom-dm.

Err, I'm a bit surprised. I haven't tested it since then, but I'm sure live migration was working with stubdoms a few months ago, without the need for a logdirty shared buffer. This works thanks to the few lines at the end of the cpu_physical_memory_rw() function that tell the hypervisor about modified memory. Actually, the plan was even to use that bit of code in all cases (not only non-stubdom), so as to get rid of the logdirty shared buffer completely.

I take it you're referring to this:

#ifdef CONFIG_STUBDOM
if (logdirty_bitmap != NULL)
xc_hvm_modified_memory(xc_handle, domid, _addr >> TARGET_PAGE_BITS,
((_addr + _len + TARGET_PAGE_SIZE - 1) >> TARGET_PAGE_BITS)
- (_addr >> TARGET_PAGE_BITS));
#endif

Yosuke Iwamatsu: is this not working for you then ? It's possible that I've broken it somehow.

Ian.

#4 lore raw
Quoted 7 lines · Ian Jackson, le Tue 16 Dec 2008 14:04:53 +0000, a écrit :

Samuel Thibault writes ("[Xen-devel] Re: [PATCH 0/4] stubdom-dm: Support live migration of HVM domain with stubdom-dm"):

Quoted 5 lines

This works thanks to the few lines at the end of the cpu_physical_memory_rw() function that tell the hypervisor about modified memory. Actually, the plan was even to use that bit of code in all cases (not only non-stubdom), so as to get rid of the logdirty shared buffer completely.

Quoted 8 lines

I take it you're referring to this:

#ifdef CONFIG_STUBDOM
if (logdirty_bitmap != NULL)
xc_hvm_modified_memory(xc_handle, domid, _addr >> TARGET_PAGE_BITS,
((_addr + _len + TARGET_PAGE_SIZE - 1) >> TARGET_PAGE_BITS)
- (_addr >> TARGET_PAGE_BITS));
#endif

Yes, precisely.

Samuel

author
#5 lore raw
In reply to Ian Jackson
Quoted 12 lines · Ian Jackson

Samuel Thibault writes ("[Xen-devel] Re: [PATCH 0/4] stubdom-dm: Support live migration of HVM domain with stubdom-dm"):

Quoted 10 lines
Quoted 2 lines · Yosuke Iwamatsu, le Tue 16 Dec 2008 18:22:20 +0900, a écrit :

This series of patches is an attempt to support live migration of hvm domains with stubdom-dm.

Err, I'm a bit surprised. I haven't tested it since then, but I'm sure live migration was working with stubdoms a few months ago, without the need for a logdirty shared buffer. This works thanks to the few lines at the end of the cpu_physical_memory_rw() function that tell the hypervisor about modified memory. Actually, the plan was even to use that bit of code in all cases (not only non-stubdom), so as to get rid of the logdirty shared buffer completely.

Quoted 8 lines

I take it you're referring to this:

#ifdef CONFIG_STUBDOM
if (logdirty_bitmap != NULL)
xc_hvm_modified_memory(xc_handle, domid, _addr >> TARGET_PAGE_BITS,
((_addr + _len + TARGET_PAGE_SIZE - 1) >> TARGET_PAGE_BITS)
- (_addr >> TARGET_PAGE_BITS));
#endif

Ah, I didn't notice this.

Quoted 2 lines

Yosuke Iwamatsu: is this not working for you then ? It's possible that I've broken it somehow.

Live-migration fails because stubdom-dm doesn't respond to the request from xc_save for the logdirty bitmap segment. The attached patch simply generated from old ioemu tree will fix the issue, then.

Derived from old xen internal tools/ioemu tree.

Signed-off-by Yosuke Iwamatsu <[email protected]>

Thanks,
-- Yosuke

stubdom: respond to logdirty event to re-enable live-migration.

Derived from old xen internal tools/ioemu tree.

Signed-off-by Yosuke Iwamatsu <[email protected]>
xenstore.c +5 -5
index d064df1..8129348 100644
@@ -596,10 +596,6 @@ extern int vga_ram_size, bios_size;
 
 void xenstore_process_logdirty_event(void)
 {
-#ifdef CONFIG_STUBDOM
-    /* XXX we just can't use shm. */
-    return;
-#else
     char *act;
     static char *active_path = NULL;
     static char *next_active_path = NULL;

@@ -627,6 +623,10 @@ void xenstore_process_logdirty_event(void)
             /* No key yet: wait for the next watch */
             return;
 
+#ifdef CONFIG_STUBDOM
+        /* We pass the writes to hypervisor */
+        seg = (void*)1;
+#else
         strncpy(key_terminated, key_ascii, 16);
         free(key_ascii);
         key = (key_t) strtoull(key_terminated, NULL, 16);

@@ -666,6 +666,7 @@ void xenstore_process_logdirty_event(void)
             seg = NULL;
             return;
         }
+#endif
 
         /* Remember the paths for the next-active and active entries */
         if (pasprintf(&active_path, 

@@ -702,7 +703,6 @@ void xenstore_process_logdirty_event(void)
     /* Ack that we've switched */
     xs_write(xsh, XBT_NULL, active_path, act, len);
     free(act);
-#endif
 }

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#7 lore raw
In reply to Yosuke Iwamatsu

Yosuke Iwamatsu writes ("Re: [Xen-devel] Re: [PATCH 0/4] stubdom-dm: Support live migration of HVM domain with stubdom-dm"):

Quoted 3 lines

Live-migration fails because stubdom-dm doesn't respond to the request from xc_save for the logdirty bitmap segment. The attached patch simply generated from old ioemu tree will fix the issue, then.

Thanks, applied. I take it this should be backported to 3.3 ?

Ian.

#8 lore raw
Quoted 6 lines · Ian Jackson, le Wed 17 Dec 2008 11:39:46 +0000, a écrit :

Yosuke Iwamatsu writes ("Re: [Xen-devel] Re: [PATCH 0/4] stubdom-dm: Support live migration of HVM domain with stubdom-dm"):

Quoted 3 lines

Live-migration fails because stubdom-dm doesn't respond to the request from xc_save for the logdirty bitmap segment. The attached patch simply generated from old ioemu tree will fix the issue, then.

Thanks, applied. I take it this should be backported to 3.3 ?

Yes (though in 3.3 the in-tree qemu is used by default).

Samuel