Suspending with kvm and kvm_loaded

Richard Hughes

author

2007-06-15 12:56 UTC #0 lore raw

Hi,

I'm getting lots of bug reports about installing kvm and then suspend breaking. Some background: when the fedora package kvm is installed, then the kvm and kvm_xxx modules get auto loaded at boot. To get the machine to suspend again, the modules have to be rmmod'd on suspend and then modprobe'd on resume.

We can automate the rmmodding and modprobeing adding a simple script in pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just bodging the issue I wanted to ask you guys first.

* Is it safe to rmmod kvm and kvm_x before suspend?
* Is there any work on adding suspend and resume hooks into the kvm driver to support suspend?
* Do you guys know why the kvm module might be stopping suspend?

Thanks for any pointers,

Richard Hughes.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Jeremy Katz

2007-06-15 13:27 UTC #1 lore raw

Quoted 9 lines · Richard Hughes

I'm getting lots of bug reports about installing kvm and then suspend breaking. Some background: when the fedora package kvm is installed, then the kvm and kvm_xxx modules get auto loaded at boot. To get the machine to suspend again, the modules have to be rmmod'd on suspend and then modprobe'd on resume.
We can automate the rmmodding and modprobeing adding a simple script in pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just bodging the issue I wanted to ask you guys first.

Also could go into the kvm package instead. Probably shouldn't be done generally in pm-utils (the mantra of fix the real problem, don't add more bodges around it ;-)

Quoted 1 line

* Is it safe to rmmod kvm and kvm_x before suspend?

Relatively. Although I vaguely remember having it cause an interesting problem once or twice, but not reproducibly enough to actually figure out what was going on.

Quoted 2 lines

* Is there any work on adding suspend and resume hooks into the kvm driver to support suspend?

The bigger thing is there's a need to add some hooks so that kvm can do things at the right point in the suspend sequence. Avi has patches, but they missed the .22 merge window. I need to sit down (maybe today) and refresh the Fedora kvm patch; I was thinking of pushing those changes in at the same time so that this can go away. And if that pans out for a little bit in rawhide, it could easily go out as an F7 update.

Jeremy

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-15 13:36 UTC #3 lore raw

Quoted 7 lines · Jeremy Katz

Quoted 6 lines · Richard Hughes
I'm getting lots of bug reports about installing kvm and then suspend breaking. Some background: when the fedora package kvm is installed, then the kvm and kvm_xxx modules get auto loaded at boot. To get the machine to suspend again, the modules have to be rmmod'd on suspend and then modprobe'd on resume.

Quoted 5 lines

Quoted 3 lines
We can automate the rmmodding and modprobeing adding a simple script in pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just bodging the issue I wanted to ask you guys first.
Also could go into the kvm package instead.

Ahh yes, that's what I meant, sorry if I was unclear.

Quoted 5 lines

Quoted 1 line
* Is it safe to rmmod kvm and kvm_x before suspend?
Relatively. Although I vaguely remember having it cause an interesting problem once or twice, but not reproducibly enough to actually figure out what was going on.

Okay, cool.

Quoted 9 lines

Quoted 2 lines
* Is there any work on adding suspend and resume hooks into the kvm driver to support suspend?
The bigger thing is there's a need to add some hooks so that kvm can do things at the right point in the suspend sequence. Avi has patches, but they missed the .22 merge window. I need to sit down (maybe today) and refresh the Fedora kvm patch; I was thinking of pushing those changes in at the same time so that this can go away. And if that pans out for a little bit in rawhide, it could easily go out as an F7 update.

Superb. When you push the new patch into rawhide, yell, and I'll get an army of testers to try and break suspend in new, exotic ways. :-)

Thanks for the super-quick response.

Richard

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-06-16 07:53 UTC #4 lore raw

In reply to Richard Hughes

Quoted 8 lines · Richard Hughes

Hi,
I'm getting lots of bug reports about installing kvm and then suspend breaking. Some background: when the fedora package kvm is installed, then the kvm and kvm_xxx modules get auto loaded at boot. To get the machine to suspend again, the modules have to be rmmod'd on suspend and then modprobe'd on resume.

Quoted 6 lines

We can automate the rmmodding and modprobeing adding a simple script in pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just bodging the issue I wanted to ask you guys first.
* Is it safe to rmmod kvm and kvm_x before suspend?

Yes, although you can't do that if you have any virtual machines running.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-16 10:29 UTC #6 lore raw

Quoted 9 lines · Avi Kivity

Quoted 8 lines · Richard Hughes
Hi,
I'm getting lots of bug reports about installing kvm and then suspend breaking. Some background: when the fedora package kvm is installed, then the kvm and kvm_xxx modules get auto loaded at boot. To get the machine to suspend again, the modules have to be rmmod'd on suspend and then modprobe'd on resume.

Quoted 8 lines

Quoted 6 lines
We can automate the rmmodding and modprobeing adding a simple script in pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just bodging the issue I wanted to ask you guys first.
* Is it safe to rmmod kvm and kvm_x before suspend?
Yes, although you can't do that if you have any virtual machines running.

Is there any easy way we can detect if virtual machines are running?

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Muli Ben-Yehuda

2007-06-16 12:30 UTC #7 lore raw

Quoted 1 line · Richard Hughes

Is there any easy way we can detect if virtual machines are running?

The module use counts will be elevanted and they'll refuse to unload (or there's a nasty bug somewhere).

Cheers,
Muli

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-16 17:20 UTC #9 lore raw

Quoted 5 lines · Muli Ben-Yehuda

Quoted 2 lines
Is there any easy way we can detect if virtual machines are running?
The module use counts will be elevanted and they'll refuse to unload (or there's a nasty bug somewhere).

Nahh, that's too brutal. Is there a way to (hypothetically) read:

# cat /sys/module/kvm/parameters/number_of_running_vms
2

Then if the user tries to suspend we can throw up a bubble saying

"Are you sure you want to suspend your computer with a running virtual machine?"

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Luca

2007-06-16 17:36 UTC #10 lore raw

Quoted 11 lines · Richard Hughes

Quoted 5 lines · Muli Ben-Yehuda
Quoted 2 lines
Is there any easy way we can detect if virtual machines are running?
The module use counts will be elevanted and they'll refuse to unload (or there's a nasty bug somewhere).
Nahh, that's too brutal. Is there a way to (hypothetically) read:
# cat /sys/module/kvm/parameters/number_of_running_vms
2

cat /sys/module/kvm_intel/refcnt (or kvm_amd) will tell you how many VMs are running.

Luca

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Muli Ben-Yehuda

2007-06-16 20:00 UTC #12 lore raw

Quoted 8 lines · Luca

Quoted 7 lines · Richard Hughes
Quoted 5 lines · Muli Ben-Yehuda
Quoted 2 lines
Is there any easy way we can detect if virtual machines are running?
The module use counts will be elevanted and they'll refuse to unload (or there's a nasty bug somewhere).

Quoted 7 lines

Quoted 4 lines
Nahh, that's too brutal. Is there a way to (hypothetically) read:
# cat /sys/module/kvm/parameters/number_of_running_vms
2
cat /sys/module/kvm_intel/refcnt (or kvm_amd) will tell you how many VMs are running.

... were running at the moment when you queried the kernel. Once you get back the answer, it's irrelevant --- a new VM could've been launched inbetween reading the refcnt and acting upon it. Really the only safe way to do this sort of thing is to try and rmmod the module.

Cheers,
Muli

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-06-17 07:02 UTC #13 lore raw

In reply to Richard Hughes

Quoted 10 lines · Richard Hughes

Quoted 5 lines · Muli Ben-Yehuda
Quoted 2 lines
Is there any easy way we can detect if virtual machines are running?
The module use counts will be elevanted and they'll refuse to unload (or there's a nasty bug somewhere).
Nahh, that's too brutal. Is there a way to (hypothetically) read:

Quoted 8 lines

# cat /sys/module/kvm/parameters/number_of_running_vms
2
Then if the user tries to suspend we can throw up a bubble saying
"Are you sure you want to suspend your computer with a running virtual machine?"

Why invest in workarounds when a fix is available?

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-17 08:34 UTC #15 lore raw

Quoted 6 lines · Avi Kivity

Quoted 1 line
"Are you sure you want to suspend your computer with a running
virtual
Quoted 2 lines
machine?"
Why invest in workarounds when a fix is available?

Totally agree. I was really looking for a proper fix, e.g. use libvirt to suspend the virtual machines before the machine is suspended (if any are running).

Does anybod know why libvirt does not detect my running qemu VM (that uses KVM acceleration)? I get:

Failed to connect to hypervisor

Or is livirt XEN specific (!)?

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-06-17 08:10 UTC #16 lore raw

Quoted 9 lines · Richard Hughes

Quoted 7 lines · Avi Kivity
Quoted 2 lines
"Are you sure you want to suspend your computer with a running
virtual
Quoted 3 lines
machine?"

Quoted 7 lines

Quoted 2 lines
Why invest in workarounds when a fix is available?
Totally agree. I was really looking for a proper fix, e.g. use libvirt to suspend the virtual machines before the machine is suspended (if any are running).

No no no. The proper fix will have suspend "just working", whether or not a virtual machine is running or not. When you resume, the virtual machines continue running as if nothing had happened.

I posted a patchset which does just that, when the F kernel integrates it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.

Quoted 6 lines

Does anybod know why libvirt does not detect my running qemu VM (that uses KVM acceleration)? I get:
Failed to connect to hypervisor

If you start a virtual machine outside libvirt's knowledge, it won't pick it up. You also need to enable the libvirtd service (or however it is called).

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-17 09:19 UTC #18 lore raw

Quoted 3 lines · Avi Kivity

No no no. The proper fix will have suspend "just working", whether or not a virtual machine is running or not. When you resume, the virtual machines continue running as if nothing had happened.

Ahh gotcha. In which case I apologize, I really don't follow kvm development very closely and didn't know this was even possible or planned.

Quoted 3 lines

I posted a patchset which does just that, when the F kernel integrates it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.

I think the latter is most likely, although if you have a link to the current patch I can push it in the way of davej and see what he says.

Quoted 6 lines

Quoted 3 lines
Does anybod know why libvirt does not detect my running qemu VM (that uses KVM acceleration)? I get:
If you start a virtual machine outside libvirt's knowledge, it won't pick it up. You also need to enable the libvirtd service (or however it is called).

Ohh, I see. In which case I can ignore libvirt. Thanks!

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Daniel Veillard

2007-06-17 08:41 UTC #19 lore raw

Quoted 9 lines · Richard Hughes

Quoted 6 lines · Avi Kivity
Quoted 3 lines
Does anybod know why libvirt does not detect my running qemu VM (that uses KVM acceleration)? I get:
If you start a virtual machine outside libvirt's knowledge, it won't pick it up. You also need to enable the libvirtd service (or however it is called).
Ohh, I see. In which case I can ignore libvirt. Thanks!

Well if there were a way to discover running KVM instances and
communicate with them, then we wouldn't need to keep a daemon as a parent process for libvirt access and control. It's a point I would love to see solved at the QEmu level, but it's not really urgent :-)

Daniel

Signature · 10 lines

Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
[email protected]  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-06-17 08:59 UTC #21 lore raw

Quoted 5 lines · Daniel Veillard

Well if there were a way to discover running KVM instances and
communicate with them, then we wouldn't need to keep a daemon as a parent process for libvirt access and control. It's a point I would love to see solved at the QEmu level, but it's not really urgent :-)

You can set the qemu monitor socket to a unix domain socket in some well-known directory, and enumerate virtual machines by readdiring that directory and trying to connect. Of course that only works for qemu processes started by libvirt, not random qemu processes.

(and I wish there was an SO_UNLINKONCLOSE option that could racelessly garbage collect those dead sockets)

I still think having a daemon is a good idea; it can collect events generated by qemu and possibly act on them.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-17 18:55 UTC #22 lore raw

In reply to a message not in the archive

Quoted 5 lines · Avi Kivity

I posted a patchset which does just that, when the F kernel integrates
it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.

I've tried the current kvm kernel git tree and that failed to suspend with a running VM. Is there a different branch I should try?

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-06-19 08:05 UTC #23 lore raw

Quoted 12 lines · Richard Hughes

Quoted 5 lines · Avi Kivity
I posted a patchset which does just that, when the F kernel integrates
it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.
I've tried the current kvm kernel git tree and that failed to suspend with a running VM. Is there a different branch I should try?

I've rebased the 'hotplug' branch. Please try it out.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Jeremy Katz

2007-06-19 12:33 UTC #25 lore raw

Quoted 11 lines · Avi Kivity

Quoted 8 lines · Richard Hughes
Quoted 5 lines · Avi Kivity
I posted a patchset which does just that, when the F kernel integrates
it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.
I've tried the current kvm kernel git tree and that failed to suspend with a running VM. Is there a different branch I should try?
I've rebased the 'hotplug' branch. Please try it out.

I actually extracted the patch set yesterday for the Fedora kernel and it's in today's rawhide kernel. Should be a good way to get some extra testing on it. In my quick trial run yesterday, it worked quite nicely!

Jeremy

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Richard Hughes

author

2007-06-19 12:41 UTC #27 lore raw

Quoted 8 lines · Jeremy Katz

Quoted 7 lines · Avi Kivity
Quoted 6 lines · Richard Hughes
Quoted 5 lines · Avi Kivity
I posted a patchset which does just that, when the F kernel integrates
it (through inclusion of 2.6.23 or by patching the current kernel), Fedora will be able to suspend/resume just fine.

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
I've tried the current kvm kernel git tree and that failed to suspend with a running VM. Is there a different branch I should try?
I've rebased the 'hotplug' branch. Please try it out.
I actually extracted the patch set yesterday for the Fedora kernel and it's in today's rawhide kernel. Should be a good way to get some extra testing on it. In my quick trial run yesterday, it worked quite nicely!

Yes, it works very well for me too. Thanks.

Richard.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Patrick C. F. Ernzer

2007-06-21 16:52 UTC #28 lore raw

Hiho,

Quoted 1 line · --On Tuesday, June 19, 2007 13:41:07 +0100 Richard Hughes…

On Tue, 2007-06-19 at 08:33 -0400, Jeremy Katz wrote:

[...]

Quoted 5 lines

Quoted 3 lines
I actually extracted the patch set yesterday for the Fedora kernel and it's in today's rawhide kernel. Should be a good way to get some extra testing on it. In my quick trial run yesterday, it worked quite nicely!
Yes, it works very well for me too. Thanks.

Can confirm that in a one-off test this worked here too. Thanks Jeremy.

If needed I can run more tests, for me this is enough as I actually use kvm very seldom on the affected machine.

RU

PCFE

Signature · 10 lines

Reg. Adresse:      Red Hat GmbH, Hauptstaetter Str 58, 70178 Stuttgart
Handelsregister:   Amtsgericht Stuttgart HRB 153243
Geschaeftsfuehrer: Alex Pinchev, Charlie Peters, David Owens,
                  Michael Cunningham, Werner Knoblich
------------------------ http://www.redhat.com/ ----------------------
    Patrick C. F. Ernzer          |        Senior Consultant
    Red Hat                       |   Global Professional Services
-----------------------------------+----------------------------------
Do not meddle in the internals of kernels,
for they are subtle and quick to panic.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel