kvm host suspend/resume

I've committed kvm support for host suspend-to-disk (and hopefully resume too). This means you can suspend your machine with running vms, and resume it later will all vms returning to the saved state. This should please laptop owners.

The big ironers should also be happy as support for host cpu hotplug was also added as a side effect.

Please test this feature! A few testing cycles on one of my machines saw both Linux and Windows guests survive a suspend/resume cycle.

This is available from subversion trunk, or if you don't use subversion, from http://people.qumranet.com/avi/kvm-r4341.tar.gz.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

I pulled r4347 from SVN and built it...

I have a Dell M1210 (Core 2 Duo intel), and tried a few things, all with a debian Etch AMD64 guest:

1) chmod'ing /dev/kvm and starting guest as local user
2) starting guest as root
3) starting guest, closing it, then suspend/resuming without unloading the modules.
... then suspending/resuming the laptop.

Previously, I could suspend/resume with the modules loaded, but the laptop would hardlock as soon as I tried to start up a guest.

This time, in all cases the machine did not come back from suspend, even in case #3 where it used to come back fine. I just get a black screen and no network activity, so I don't have any interesting troubleshooting info. Anything I can do to be of more help?

Thanks,
-Dormando

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

This is expected: the cpu came up with vmx disabled.

Sprinkle printk()s in vmx_vcpu_decache(), vmx_hardware_enable(), vmx_hardware_disable(), kvm_cpu_hotplug(), kvm_suspend(), and kvm_resume(). Please include raw_smp_processor_id() in all outputs, then send the log (unfortunately you'll have to copy it by hand:
netconsole doesn't play well with suspend).

Please start with case #3 (perhaps even case #4 where you load the modules but don't start a guest).

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend and resume with the modules loaded. Starting a guest after resume gets the following oops in the host; guest is in the grub menu at that point.

kvm: msrs: 1
invalid opcode: 0000 [#1]
SMP
last sysfs file: /class/net/eth0/carrier

Modules linked in: snd_hda_intel snd_hda_codec button kvm_intel kvm 

rfcomm hidp l2cap i915 drm ipt_MASQUERADE iptable_nat nf_nat ipv6 autofs4 sunrpc bridge dummy nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables cpufreq_ondemand video sbs i2c_ec dock battery asus_acpi backlight ac parport_pc lp parport sr_mod cdrom snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss hci_usb fw_ohci snd_pcm e100 mii bluetooth sdhci ata_piix fw_core ipw3945 mmc_core sg serio_raw i2c_i801 iTCO_wdt iTCO_vendor_support snd_timer ieee80211 ieee80211_crypt ata_generic i2c_core snd soundcore snd_page_alloc joydev dm_snapshot dm_zero dm_mirror dm_mod ahci libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd

CPU:    1
EIP:    0060:[<f8d3bc6a>]    Not tainted VLI
EFLAGS: 00010207   (2.6.19-1.2913.fc7 #1)

EIP is at vmx_vcpu_load+0x49/0xc1 [kvm_intel]
eax: db452cbc ebx: f2f5847c ecx: 018e2100 edx: c075eb2c
esi: 00000001 edi: db452dc0 ebp: db452cd4 esp: db452cac
ds: 007b es: 007b ss: 0068
Process qemu (pid: 21176, ti=db452000 task=e2020030 task.ti=db452000)

Stack: 22222222 22222222 22222222 22222222 144d9000 00000000 db452dc0 
db452cd4
        f2f58484 f2f5847c db452ce4 f8e61ad6 f2f5847c f2f5847c db452cf0 
f8e6322f
        00000001 db452d4c f8d3cc69 c0603728 00000000 00000002 c0603751 
f2f58484
Call Trace:
  [<c04051c9>] show_trace_log_lvl+0x1a/0x2f
  [<c0405279>] show_stack_log_lvl+0x9b/0xa3
  [<c0405415>] show_registers+0x194/0x26a
  [<c0405618>] die+0x12d/0x242
  [<c0605167>] do_trap+0x79/0x91
  [<c0405bbb>] do_invalid_op+0x97/0xa1
  [<c0604f3c>] error_code+0x7c/0x84
  [<f8e61ad6>] vcpu_load+0x3b/0x3f [kvm]
  [<f8e6322f>] kvm_resched+0x20/0x23 [kvm]
  [<f8d3cc69>] kvm_vmx_return+0x1d0/0x1df [kvm_intel]
  [<f8e62139>] kvm_dev_ioctl+0x284/0x117a [kvm]
  [<c0482c36>] do_ioctl+0x22/0x67
  [<c0482ecd>] vfs_ioctl+0x252/0x265
  [<c0482f29>] sys_ioctl+0x49/0x64
  [<c0404058>] syscall_call+0x7/0xb
  =======================
Code: 00 40 39 73 48 89 45 e8 74 07 89 d8 e8 5d ff ff ff 8b 0c b5 80 b3 

71 c0 ba 2c eb 75 c0 8b 43 04 39 04 0a 74 31 89 04 0a 8d 45 e8 <0f> c7 30 0f 96 c0 84 c0 74 21 8b 45 e8 8b 55 ec c7 04 24 a3 d4
EIP: [<f8d3bc6a>] vmx_vcpu_load+0x49/0xc1 [kvm_intel] SS:ESP 0068:db452cac

Jeremy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

2.6.20-rc6 can't suspend my test machine (regardless of kvm), so this possible fix is completely untested.

error compiling committee.c: too many arguments to function

Index: linux/drivers/kvm/kvm_main.c
===================================================================
@@ -2113,6 +2113,16 @@
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_disable,
 					 NULL, 0, 1);
 		break;
+	}
+	return NOTIFY_OK;
+}
+
+static int kvm_cpu_hotplug2(struct notifier_block *notifier, unsigned long val,
+			   void *v)
+{
+	int cpu = (long)v;
+
+	switch (val) {
 	case CPU_UP_PREPARE:
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_enable,
 					 NULL, 0, 1);

@@ -2126,6 +2136,11 @@
 	.priority = 20, /* must be > scheduler priority */
 };
 
+static struct notifier_block kvm_cpu_notifier2 = {
+	.notifier_call = kvm_cpu_hotplug2,
+	.priority = 2, /* must be < scheduler priority */
+};
+
 static __init void kvm_init_debug(void)
 {
 	struct kvm_stats_debugfs_item *p;

@@ -2197,6 +2212,7 @@
 
 	on_each_cpu(kvm_arch_ops->hardware_enable, 0, 0, 1);
 	r = register_cpu_notifier(&kvm_cpu_notifier);
+	register_cpu_notifier(&kvm_cpu_notifier2);
 	if (r)
 		goto out_free_1;
 	register_reboot_notifier(&kvm_reboot_notifier);

@@ -2226,6 +2242,7 @@
 out_free_2:
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
+	unregister_cpu_notifier(&kvm_cpu_notifier2);
 out_free_1:
 	on_each_cpu(kvm_arch_ops->hardware_disable, 0, 0, 1);
 	kvm_arch_ops->hardware_unsetup();

@@ -2239,6 +2256,7 @@
 	sysdev_class_unregister(&kvm_sysdev_class);
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
+	unregister_cpu_notifier(&kvm_cpu_notifier2);
 	on_each_cpu(kvm_arch_ops->hardware_disable, 0, 0, 1);
 	kvm_arch_ops->hardware_unsetup();
 	kvm_arch_ops = NULL;

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Failure looks basically the same. Will try to do some more investigation later today.

kvm: msrs: 1
invalid opcode: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:1c.0/0000:02:00.0/cmd

Modules linked in: snd_hda_intel snd_hda_codec button kvm_intel kvm tun

i915 drm ipt_MASQUERADE iptable_nat nf_nat autofs4 hidp rfcomm l2cap sunrpc bridge dummy nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables cpufreq_ondemand video sbs i2c_ec dock battery asus_acpi backlight ac ipv6 parport_pc lp parport snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device hci_usb snd_pcm_oss sr_mod cdrom bluetooth snd_mixer_oss fw_ohci e100 mii i2c_i801 snd_pcm iTCO_wdt iTCO_vendor_support fw_core i2c_core ipw3945 ieee80211 ieee80211_crypt sdhci snd_timer serio_raw ata_piix snd soundcore snd_page_alloc mmc_core ata_generic joydev sg dm_snapshot dm_zero dm_mirror dm_mod ahci libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd

CPU:    1
EIP:    0060:[<f8e2ac6a>]    Not tainted VLI
EFLAGS: 00210207   (2.6.19-1.2913.fc7 #1)

EIP is at vmx_vcpu_load+0x49/0xc1 [kvm_intel]
eax: c3c63d24 ebx: e71d047c ecx: 018e2100 edx: c075eb48
esi: 00000001 edi: c3c63dc0 ebp: c3c63d3c esp: c3c63d14
ds: 007b es: 007b ss: 0068
Process qemu (pid: 12413, ti=c3c63000 task=e656a030 task.ti=c3c63000)

Stack: 22222222 22222222 22222222 22222222 22311000 00000000 c3c63dc0
c3c63d3c 
       e71d0484 e71d047c c3c63d4c f8e6dae0 f8e6debf fffffffe c3c63f44
f8e6e0c7 
       00000200 bfca8a68 e71d0000 f7fffa80 f7fffa80 f4ac9138 c3c63d98
c0473860 
Call Trace:
[<c04051c9>] show_trace_log_lvl+0x1a/0x2f
[<c0405279>] show_stack_log_lvl+0x9b/0xa3
[<c0405415>] show_registers+0x194/0x26a
[<c0405618>] die+0x12d/0x242
[<c0605167>] do_trap+0x79/0x91
[<c0405bbb>] do_invalid_op+0x97/0xa1
[<c0604f3c>] error_code+0x7c/0x84
[<f8e6dae0>] vcpu_load+0x3b/0x3f [kvm]
[<f8e6e0c7>] kvm_dev_ioctl+0x208/0x117a [kvm]
[<c0482c36>] do_ioctl+0x22/0x67
[<c0482ecd>] vfs_ioctl+0x252/0x265
[<c0482f29>] sys_ioctl+0x49/0x64
[<c0404058>] syscall_call+0x7/0xb
=======================
Code: 00 40 39 73 48 89 45 e8 74 07 89 d8 e8 5d ff ff ff 8b 0c b5 80 b3

71 c0 ba 48 eb 75 c0 8b 43 04 39 04 0a 74 31 89 04 0a 8d 45 e8 <0f> c7 30 0f 96 c0 84 c0 74 21 8b 45 e8 8b 55 ec c7 04 24 a3 c4
EIP: [<f8e2ac6a>] vmx_vcpu_load+0x49/0xc1 [kvm_intel] SS:ESP 0068:c3c63d14

Jeremy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Looks like ->hardware_enable() was not called for some reason. Sprinkling printks in the cpu hotplug and resume callbacks will help.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Multiple things failing here together.

1) With the versioning of Fedora kernels, things were falling back to the compat version of smp_call_function_single which then doesn't seem to work properly (->hardware_disable() and ->hardware_enable() never get called AFAICT)

2) Switching to use the real smp_call_function_single, I get an oops on suspend since we're trying to run code on the dead cpu. Attached patch makes it so that we do ->hardware_enable() and ->hardware_disable() when the cpu is in a functioning state. With it, I can suspend/resume successfully with no guests running. With a guest running, it's a bit of a toss-up. But one step at a time :)

Jeremy

Index: kvm_main.c
===================================================================
@@ -2107,13 +2107,15 @@
 	int cpu = (long)v;
 
 	switch (val) {
-	case CPU_DEAD:
+	case CPU_DOWN_PREPARE:
 	case CPU_UP_CANCELED:
+		printk(KERN_INFO, "kvm: disabling virtualization on CPU%d\n", cpu);
 		decache_vcpus_on_cpu(cpu);
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_disable,
 					 NULL, 0, 1);
 		break;
-	case CPU_UP_PREPARE:
+	case CPU_ONLINE:
+		printk(KERN_INFO, "kvm: enabling virtualization on CPU%d\n", cpu);
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_enable,
 					 NULL, 0, 1);
 		break;

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Hmm. It worked for me. I can't really see anything wrong with it.

I think the problem here is the race with the scheduler resuming up ahead of kvm. My previous patch tries to work around it by having one callback for suspend and one for resume, each with a different priority relative to the scheduler. Maybe an updated version will help.

Yes, so I'll apply this :). Thanks.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642