A few KVM security questions
Hello,
I have the following questions regarding the KVM architecture. I looked at the slides available at linux-kvm.org, but didn't find definitive answers. I'm also interested to learn if given feature is or is not planned for the near future.
The questions follow:
1) Do you have any support for para-virtualized VMs? In particular, is it possible to move the qemu from the host to one of the VMs? Perhaps to have a separate copy of qemu for each VM? (ala Xen's stub-domains)
2) Is it possible to have driver domains in KVM? E.g. I would like to assign my NIC to one VM (a "network domain") and then I would like other domains to use this network domain for networking. In case of Xen, this is done by moving the network backend (which is not qemu BTW) into the network domain, and configuring the network frontends in other VMs to talk to this network domain's backend, rather then to Dom0's backend (in fact you can get rid of all the networking in Dom0).
3) Do you have any support for TXT-based trusted boot? I guess you indirectly have via tboot. However, how do you deal with VT-d protections? The tboot.gz should normally DMA-protect memory before handing execution over to Linux kernel. But then you need to allow your drivers to work. Do you unprotect all the memory for DMA, or do you have some support for selectively unprotect only those regions of memory which are needed by (some) drivers? If the latter, how do you determine which memory should be DMA-unprotected?
4) Do you have some method of excluding particular PCI devices from being initialized by your host Linux? E.g. those devices that are later to be assigned to some VMs (via VT-d passthrough)?
Thanks, I would appreciate any answers. Please note I'm not subscribed to the list, so won't get your response if sent only to the list.
Regards,
joanna.
Signature · 4 lines
Joanna Rutkowska Founder/CEO Invisible Things Lab http://invisiblethingslab.com/
Quoted 10 lines · Joanna Rutkowska
Hello,
I have the following questions regarding the KVM architecture. I looked at the slides available at linux-kvm.org, but didn't find definitive answers. I'm also interested to learn if given feature is or is not planned for the near future.
The questions follow:
1) Do you have any support for para-virtualized VMs?
Yes, for example, we support paravirtualized timers and mmu for Linux. These are fairly minimal compared to Xen's pv domains.
Quoted 4 lines
In particular, is
it possible to move the qemu from the host to one of the VMs? Perhaps to have a separate copy of qemu for each VM? (ala Xen's stub-domains)
It should be fairly easy to place qemu in a guest. You would leave a simple program on the host to communicate with kvm and pass any data written by the guest to qemu running in another guest, and feed any replies back to the guest.
It should also be possible to constrain qemu using SECCOMP.
None of this has been attempted to my knowledge.
Quoted 8 lines
2) Is it possible to have driver domains in KVM? E.g. I would like to assign my NIC to one VM (a "network domain") and then I would like other domains to use this network domain for networking. In case of Xen, this is done by moving the network backend (which is not qemu BTW) into the network domain, and configuring the network frontends in other VMs to talk to this network domain's backend, rather then to Dom0's backend (in fact you can get rid of all the networking in Dom0).
Should be doable by assigning the NIC to a driver domain and bridging it to a virtio driver; then have the driver domain's virtio device talk to the ordinary guests.
Quoted 9 lines
3) Do you have any support for TXT-based trusted boot? I guess you indirectly have via tboot. However, how do you deal with VT-d protections? The tboot.gz should normally DMA-protect memory before handing execution over to Linux kernel. But then you need to allow your drivers to work. Do you unprotect all the memory for DMA, or do you have some support for selectively unprotect only those regions of memory which are needed by (some) drivers? If the latter, how do you determine which memory should be DMA-unprotected?
I know nothing about tboot.
Quoted 5 lines
4) Do you have some method of excluding particular PCI devices from being initialized by your host Linux? E.g. those devices that are later to be assigned to some VMs (via VT-d passthrough)?
Yes, there is a stub driver that does this.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 1 line
1) Do you have any support for para-virtualized VMs?
Yes, for example, we support paravirtualized timers and mmu for Linux. These are fairly minimal compared to Xen's pv domains.
Can I run a regular Linux as PV-guest? Specifically, can I get rid of qemu totally, assuming I have only PV guests?
E.g. do you have PV network and disk frontends (PV drivers), that I could use on guests and that do not use qemu at all?
Quoted 9 lines
Quoted 8 lines
2) Is it possible to have driver domains in KVM? E.g. I would like to assign my NIC to one VM (a "network domain") and then I would like other domains to use this network domain for networking. In case of Xen, this is done by moving the network backend (which is not qemu BTW) into the network domain, and configuring the network frontends in other VMs to talk to this network domain's backend, rather then to Dom0's backend (in fact you can get rid of all the networking in Dom0).
Quoted 3 lines
Should be doable by assigning the NIC to a driver domain and bridging it to a virtio driver; then have the driver domain's virtio device talk to the ordinary guests.
But bridging would still require to have some networking support (+net backends) on the host (sure, without any real NIC driver, but still), correct?
Quoted 6 lines
Quoted 3 lines
4) Do you have some method of excluding particular PCI devices from being initialized by your host Linux? E.g. those devices that are later to be assigned to some VMs (via VT-d passthrough)?
Yes, there is a stub driver that does this.
Does this stub driver sets DMA protections, so that the device in question cannot access any host memory?
That is important, because once you assigned a device to some VM, we should assume the VM might have somehow compromised the device, e.g. reflashed the firmware of the NIC, perhaps. So, it's important to be able to protect the hypervisor from such devices.
Thanks,
joanna.
Quoted 13 lines · Joanna Rutkowska
Quoted 6 lines · Avi Kivity
Quoted 2 lines
1) Do you have any support for para-virtualized VMs?
Yes, for example, we support paravirtualized timers and mmu for Linux. These are fairly minimal compared to Xen's pv domains.
Can I run a regular Linux as PV-guest? Specifically, can I get rid of qemu totally, assuming I have only PV guests?
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
Quoted 3 lines
E.g. do you have PV network and disk frontends (PV drivers), that I could use on guests and that do not use qemu at all?
We do have PV network and disk frontends, but the backends (devices) are still in qemu.
Quoted 8 lines
Quoted 4 lines
Should be doable by assigning the NIC to a driver domain and bridging it to a virtio driver; then have the driver domain's virtio device talk to the ordinary guests.
But bridging would still require to have some networking support (+net backends) on the host (sure, without any real NIC driver, but still), correct?
If you were willing to hack a bit, you can use any IPC to pass the packets instead of the networking stack (for example, shared memory + eventfd for signalling).
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
4) Do you have some method of excluding particular PCI devices from being initialized by your host Linux? E.g. those devices that are later to be assigned to some VMs (via VT-d passthrough)?
Yes, there is a stub driver that does this.
Quoted 8 lines
Does this stub driver sets DMA protections, so that the device in question cannot access any host memory?
That is important, because once you assigned a device to some VM, we should assume the VM might have somehow compromised the device, e.g. reflashed the firmware of the NIC, perhaps. So, it's important to be able to protect the hypervisor from such devices.
kvm places assigned devices in an iommu protection domain so it cannot attack the host. Once the guest stops using the device, we reset it. If the guest is able to upload a malicious, persistent payload to the device, then when the device is reused whoever uses it will be vulnerable (whether a new guest or the host).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 9 lines · Joanna Rutkowska
Quoted 6 lines · Avi Kivity
Quoted 2 lines
1) Do you have any support for para-virtualized VMs?
Yes, for example, we support paravirtualized timers and mmu for Linux. These are fairly minimal compared to Xen's pv domains.
Quoted 8 lines
Quoted 4 lines
Can I run a regular Linux as PV-guest? Specifically, can I get rid of qemu totally, assuming I have only PV guests?
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
Can you point to a document/source file that would list all the possible interfaces between VM and the host? I.e. all the VMX handlers, and all the hypercalls (PV interfaces).
joanna.
Quoted 5 lines · Joanna Rutkowska
Can you point to a document/source file that would list all the possible interfaces between VM and the host? I.e. all the VMX handlers, and all the hypercalls (PV interfaces).
arch/x86/kvm/vmx.c is the entry point for all interaction, but it quickly diverges. arch/x86/include/asm/kvm_para.h is the pv interface.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
NB, unlike Xen, we can (and do) run qemu as non-root. Things like RHEV-H and oVirt constrain the qemu process with SELinux.
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
Regards,
Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
NB, unlike Xen, we can (and do) run qemu as non-root. Things like RHEV-H and oVirt constrain the qemu process with SELinux.
On Xen you can get rid of the qemu entirely, if you run only PV domains.
Quoted 8 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world program. The process <-> kernel interface seem to be just too fat. See e.g. the recent Linux kernel overflows by Spender.
Also, SELinux seems to me like a step into the wrong direction. It not only adds complexity to the already-too-complex kernel, but requires complex configuration. See e.g. this paper[1] for a nice example of how to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.
When some people tried to add SELinux-like-thing to Xen hypervisor, it only resulted in an exploitable heap overflow in Xen [2].
[1] http://invisiblethingslab.com/resources/misc08/xenfb-adventures-10.pdf
[2] http://invisiblethingslab.com/resources/bh08/part2-full.pdf
joanna.
Quoted 5 lines · Joanna Rutkowska
Quoted 5 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
Quoted 10 lines
Quoted 5 lines
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
Quoted 5 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world program. The process<-> kernel interface seem to be just too fat. See e.g. the recent Linux kernel overflows by Spender.
What about seccomp? You can easily simplify qemu to just a bunch of calculations served over a pipe.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 17 lines · Avi Kivity
Quoted 5 lines · Joanna Rutkowska
Quoted 5 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
Quoted 10 lines
Quoted 5 lines
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Quoted 4 lines
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
That's not true if you use VT-d.
Quoted 9 lines
Quoted 5 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world program. The process<-> kernel interface seem to be just too fat. See e.g. the recent Linux kernel overflows by Spender.
What about seccomp? You can easily simplify qemu to just a bunch of calculations served over a pipe.
But the qemu must somehow communicate with the external world too, no? You said you provide e.g. net backend via the qemu process...
joanna.
Quoted 7 lines · Joanna Rutkowska
Quoted 7 lines
Quoted 7 lines
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Quoted 7 lines
Quoted 5 lines
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
That's not true if you use VT-d.
AFAIK VT-d is only supported in Xen for fully virtualized guests. Maybe it changed while I wasn't watching, though.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world
program. The process<-> kernel interface seem to be just too fat. See
e.g. the recent Linux kernel overflows by Spender.
Quoted 7 lines
Quoted 4 lines
What about seccomp? You can easily simplify qemu to just a bunch of calculations served over a pipe.
But the qemu must somehow communicate with the external world too, no? You said you provide e.g. net backend via the qemu process...
It can use read() and write() (and shared memory) to communicate, just like Xen stub domains.
It's a lot of surgery, but it can be done.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 13 lines · Avi Kivity
Quoted 12 lines · Joanna Rutkowska
Quoted 12 lines
Quoted 7 lines
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
Quoted 6 lines
Quoted 2 lines
That's not true if you use VT-d.
AFAIK VT-d is only supported in Xen for fully virtualized guests. Maybe it changed while I wasn't watching, though.
Negative. VT-d can be used to contain PV DomUs as well. We actually verified it.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world
program. The process<-> kernel interface seem to be just too fat. See
e.g. the recent Linux kernel overflows by Spender.What about seccomp? You can easily simplify qemu to just a bunch of calculations served over a pipe.
Quoted 7 lines
Quoted 3 lines
But the qemu must somehow communicate with the external world too, no? You said you provide e.g. net backend via the qemu process...
It can use
read()andwrite()(and shared memory) to communicate, just like Xen stub domains.
Well, but the read() and write() syscalls, on a system like Linux, it's a gate to *lots* of code. These are very powerful system calls.
Quoted 2 lines
It's a lot of surgery, but it can be done.
And then you have the code with whom this qemu communicates (e.g. the network stack). You said we could somehow use IPC to delegate it to some VM (that would have VT-d assigned NIC). But then this VM would need to use qemu again (of course this time not for net emulation). Looks non-trivial.
joanna.
Quoted 9 lines · Joanna Rutkowska
Quoted 5 lines
AFAIK VT-d is only supported in Xen for fully virtualized guests. Maybe it changed while I wasn't watching, though.
Negative. VT-d can be used to contain PV DomUs as well. We actually verified it.
Ah, good for them.
Quoted 7 lines
Quoted 4 lines
It can use
read()andwrite()(and shared memory) to communicate, just like Xen stub domains.Well, but the
read()andwrite()syscalls, on a system like Linux, it's a gate to *lots* of code. These are very powerful system calls.
But you control all the file descriptors. A minimal system would just consist of a pair of eventfd fds for signalling and shared memory (the Xen equivalent is event channels and grant tables).
Quoted 9 lines
Quoted 3 lines
It's a lot of surgery, but it can be done.
And then you have the code with whom this qemu communicates (e.g. the network stack). You said we could somehow use IPC to delegate it to some VM (that would have VT-d assigned NIC). But then this VM would need to use qemu again (of course this time not for net emulation). Looks non-trivial.
It doesn't really need to be a VM. Once the seccomp constrained qemu processes the guest actions, the result is a fairly simple event stream.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 17 lines · Joanna Rutkowska
Quoted 16 lines · Avi Kivity
Quoted 14 lines · Joanna Rutkowska
Quoted 7 lines
Quoted 7 lines
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Quoted 5 lines
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
That's not true if you use VT-d.
Quoted 6 lines
Quoted 3 lines
AFAIK VT-d is only supported in Xen for fully virtualized guests. Maybe it changed while I wasn't watching, though.
Negative. VT-d can be used to contain PV DomUs as well. We actually verified it.
Oh, I didn't know that. This is a bit OT, but could you paste or point to a VT-d configuration with a PV guest?
-- Pasi
Quoted 20 lines · Joanna Rutkowska
Quoted 18 lines · Avi Kivity
Quoted 16 lines · Joanna Rutkowska
Quoted 10 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Quoted 8 lines
Quoted 5 lines
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
That's not true if you use VT-d.
I'm skeptical that VT-d in its current form provides protection against a malicious guest. The first problem is interrupt delivery. I don't think any hypervisor has really put much thought into mitigating interrupt storms as a DoS. I think there are a number of nasty things that can be done here.
Even if you assume that there aren't flaws in VT-d wrt malicious guests, we have generations of hardware that have not been designed to be robust against malicious operating systems. There are almost certainly untold numbers of exploitable hardware bugs that can be used to do all sorts of terrible things to the physical system.
VT-d protects against DMA access, but there's still plenty of things a malicious PCI device can do to harm the physical system. I'm sure you could easily program a PCI device to flood the bus which effectively mounts a DoS against other domains. There is no mechanism to arbitrate this today. It's really a dramatically different model from a security perspective.
Regards,
Anthony Liguori
Quoted 27 lines · Anthony Liguori
Quoted 26 lines · Joanna Rutkowska
Quoted 23 lines · Avi Kivity
Quoted 6 lines · Joanna Rutkowska
Quoted 6 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M
xenpv). The effect is that you are moving that privileged code from the
kernel (netback/blkback) to userspace (qemu -M xenpv).Quoted 10 lines
Quoted 5 lines
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
They're not really unprivileged, one can easily program the dma controller of their assigned pci card to read and write arbitrary host memory.
Quoted 9 lines
Quoted 2 lines
That's not true if you use VT-d.
I'm skeptical that VT-d in its current form provides protection against a malicious guest. The first problem is interrupt delivery. I don't think any hypervisor has really put much thought into mitigating interrupt storms as a DoS. I think there are a number of nasty things that can be done here.
Intel VT-d v1 doesn't support interrupt remapping, so I'm sure you're right here. But DoS attack is a different thing then a system subversion (think malware) attack. Of course which one you fear more would depend on your threat model.
Quoted 6 lines
Even if you assume that there aren't flaws in VT-d wrt malicious guests, we have generations of hardware that have not been designed to be robust against malicious operating systems. There are almost certainly untold numbers of exploitable hardware bugs that can be used to do all sorts of terrible things to the physical system.
Perhaps, although so far nobody presented a software-only VT-d escape attack. I think it's reasonable to assume some maniacs would discover a one or two in the coming years. Still, probably order of magnitude less likely than a Linux kernel overflow.
Quoted 7 lines
VT-d protects against DMA access, but there's still plenty of things a malicious PCI device can do to harm the physical system. I'm sure you could easily program a PCI device to flood the bus which effectively mounts a DoS against other domains. There is no mechanism to arbitrate this today. It's really a dramatically different model from a security perspective.
Agree, there are lots of DoS possibilities. It's just that for me, personally, they are not in the threat model.
joanna.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Muli Ben-Yehuda
Quoted 6 lines · Anthony Liguori
I'm skeptical that VT-d in its current form provides protection against a malicious guest. The first problem is interrupt delivery. I don't think any hypervisor has really put much thought into mitigating interrupt storms as a DoS. I think there are a number of nasty things that can be done here.
Seems to me that detecting an interrupt storm and shutting the offending domain and device off is fairly easy for MSI and MSI-X interrupts, and not-interesting for legacy INTx interrupts. I don't know that any hypervisor actually implements it, though.
Quoted 9 lines
Quoted 6 lines
Even if you assume that there aren't flaws in VT-d wrt malicious guests, we have generations of hardware that have not been designed to be robust against malicious operating systems. There are almost certainly untold numbers of exploitable hardware bugs that can be used to do all sorts of terrible things to the physical system.
To the device? Undoubtedly. To the host? I'm not so sure.
But in the context of SR-IOV, impacting the device may result in disrupting (and potentially exploiting) other domains.
And I'm waiting for the "malicious guest sets server on fire" CVE :-) I'm convinced there will be at least one.
Quoted 10 lines
Quoted 5 lines
VT-d protects against DMA access, but there's still plenty of things a malicious PCI device can do to harm the physical system. I'm sure you could easily program a PCI device to flood the bus which effectively mounts a DoS against other domains.
But is there any way the device could do this and also evade detection of evade being taken off-line by the host, after first killing its controlling VM?
Thing is, the bus is shared by the host too. So if the guest is able to bring all IO devices on the system to a halt, an administrator certainly couldn't connect remotely to take corrective action.
I think all of this could potentially be detected and handled but I assume there's years of research here before that's a reality.
Quoted 9 lines
Quoted 3 lines
There is no mechanism to arbitrate this today. It's really a dramatically different model from a security perspective.
I think we need to differentiate between assigning full (legacy) devices, and assigning an SRIOV VF. In the latter---more interesting---case, the host remains in control of the overall device, so shutting off a mis-behaving VF should be simple.
SR-IOV is worse IMHO because now there are multiple guests that can be impacted by a hardware exploit.
Regards,
Anthony Liguori
Quoted 25 lines · Joanna Rutkowska
Quoted 9 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
NB, unlike Xen, we can (and do) run qemu as non-root. Things like RHEV-H and oVirt constrain the qemu process with SELinux.
On Xen you can get rid of the qemu entirely, if you run only PV domains.
Quoted 9 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
Quoted 4 lines
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Right, in KVM, Linux == hypervisor. A process is our "unprivileged domain". Putting an unprivileged domain within an unprivileged domain is probably not helpful from a security perspective since the exposure surface is identical.
Quoted 5 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world program. The process <-> kernel interface seem to be just too fat. See e.g. the recent Linux kernel overflows by Spender.
That's the point of mandatory access control. Of course, you need the right policy and Spender highlighted an issue with the standard RHEL SELinux policy, but that should be addressed now upstream.
Quoted 8 lines
Also, SELinux seems to me like a step into the wrong direction. It not only adds complexity to the already-too-complex kernel, but requires complex configuration. See e.g. this paper[1] for a nice example of how to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.
When some people tried to add SELinux-like-thing to Xen hypervisor, it only resulted in an exploitable heap overflow in Xen [2].
It's certainly fair to argue the merits of SELinux as a mandatory access control mechanism.
Again though, that's the point of MLS. Our first line of defense is qemu. Our second line of defense is traditional Posix direct access control. Our third line of defense is namespace isolation (ala lxc). Our fourth line of defense is mandatory access control (ala SELinux and AppArmor).
If you take a somewhat standard deployment like RHEV-H, an awful lot of things have to go wrong before you can successfully exploit the system. And 5.4 doesn't even implement all of what's possible. If you're really looking to harden, you can be much more aggressive about privileges and namespace isolation.
Regards,
Anthony Liguori
Quoted 65 lines · Anthony Liguori
Quoted 25 lines · Joanna Rutkowska
Quoted 9 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
No. Paravirtualization just augments the standard hardware interface, it doesn't replace it as in Xen.
NB, unlike Xen, we can (and do) run qemu as non-root. Things like RHEV-H and oVirt constrain the qemu process with SELinux.
On Xen you can get rid of the qemu entirely, if you run only PV domains.
Quoted 9 lines
Also, you can use qemu to provide the backends to a Xen PV guest (see -M xenpv). The effect is that you are moving that privileged code from the kernel (netback/blkback) to userspace (qemu -M xenpv).
In general, KVM tends to keep code in userspace unless absolutely necessary. That's a fundamental difference from Xen which tends to do the opposite.
Quoted 4 lines
But the difference is that in case of Xen one can *easily* move the backends to small unprivileged VMs. In that case it doesn't matter the code is in kernel mode, it's still only in an unprivileged domain.
Right, in KVM, Linux == hypervisor. A process is our "unprivileged domain". Putting an unprivileged domain within an unprivileged domain is probably not helpful from a security perspective since the exposure surface is identical.
Quoted 5 lines
Sandboxing a process in a monolithic OS, like Linux, is generally considered unfeasible, for anything more complex than a hello world program. The process <-> kernel interface seem to be just too fat. See e.g. the recent Linux kernel overflows by Spender.
That's the point of mandatory access control. Of course, you need the right policy and Spender highlighted an issue with the standard RHEL SELinux policy, but that should be addressed now upstream.
Quoted 9 lines
Also, SELinux seems to me like a step into the wrong direction. It not only adds complexity to the already-too-complex kernel, but requires complex configuration. See e.g. this paper[1] for a nice example of how to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.
When some people tried to add SELinux-like-thing to Xen hypervisor, it only resulted in an exploitable heap overflow in Xen [2].
It's certainly fair to argue the merits of SELinux as a mandatory access control mechanism.
Again though, that's the point of MLS. Our first line of defense is qemu. Our second line of defense is traditional Posix direct access control. Our third line of defense is namespace isolation (ala lxc). Our fourth line of defense is mandatory access control (ala SELinux and AppArmor).
Quoted 6 lines
If you take a somewhat standard deployment like RHEV-H, an awful lot of things have to go wrong before you can successfully exploit the system. And 5.4 doesn't even implement all of what's possible. If you're really looking to harden, you can be much more aggressive about privileges and namespace isolation.
I think this ultimately comes down to the question: is the built-from-scratch minimal PV interface (as in Xen) more secure than the Linux's fat-but-sandboxed interface?
joanna.
Quoted 5 lines · Joanna Rutkowska
Also, SELinux seems to me like a step into the wrong direction. It not only adds complexity to the already-too-complex kernel, but requires complex configuration. See e.g. this paper[1] for a nice example of how to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.
Things have changed alot since the time the that Xen SELinux policy was written. The Xen policy was always a tradeoff between usability & security sine the XenD managment tools were playing no part in the configuration, leaving it upto the administrator. With KVM & SELinx, the management tools play an active part in configuration, removing this burden from the adminsitrator. Each VM runs under a SELinux context with a dedicated MLS category, and the resources the VM is assigned have their labelling set to match. The guest policy only allows it access to resources with a matching MLS level, so it not gain access to anything the administrator has not explicitly granted in the VM's configuration. This is actually simpler for administrators, since they no longer need to manage labelling themselves, while offering greater protection between VMs which was also not possible with the old Xen policy
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 11 lines · Avi Kivity
Quoted 4 lines · Joanna Rutkowska
In particular, is
it possible to move the qemu from the host to one of the VMs? Perhaps to have a separate copy of qemu for each VM? (ala Xen's stub-domains)It should be fairly easy to place qemu in a guest. You would leave a simple program on the host to communicate with kvm and pass any data written by the guest to qemu running in another guest, and feed any replies back to the guest.
But then you would need to have another qemu (on the host) to support running this "qemu-VM", where we want to put the qemu, right?
joanna.
Quoted 10 lines · Joanna Rutkowska
Quoted 7 lines
It should be fairly easy to place qemu in a guest. You would leave a simple program on the host to communicate with kvm and pass any data written by the guest to qemu running in another guest, and feed any replies back to the guest.
But then you would need to have another qemu (on the host) to support running this "qemu-VM", where we want to put the qemu, right?
Right, but to exploit this, you'd have to exploit the internal qemu, exploit the kernel, and exploit the external qemu. Well, if the exploit was in some central thing I guess there isn't much value in the nesting.
You could alternatively use Xenner to run Xen guests for your qemu. That emulates a lot less. But AFAIK xenner is moving towards qemu as well.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Joanna Rutkowska
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Joanna Rutkowska
In particular, is
it possible to move the qemu from the host to one of the VMs? Perhaps to have a separate copy of qemu for each VM? (ala Xen's stub-domains)Quoted 6 lines
It should be fairly easy to place qemu in a guest. You would leave a simple program on the host to communicate with kvm and pass any data written by the guest to qemu running in another guest, and feed any replies back to the guest.
Quoted 3 lines
But then you would need to have another qemu (on the host) to support running this "qemu-VM", where we want to put the qemu, right?
It really offers no advantage. The security assumption should be that a guest can break into qemu. If a guest can break out of qemu, putting it in another qemu means that we still need to assume it can break out of that qemu. The host should treat the qemu process as hostile and constrain it by using things like -runas, -chroot, SELinux, and containers. This is what most production systems do today. libvirt certainly takes this approach.
That's not to say that we know for sure that a guest can break into qemu, but designing around that assumption gives us MLS.
Regards,
Anthony Liguori
Quoted 3 lines
joanna.