Timing out virtio-pci config space access
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.
https://lkml.org/lkml/2021/7/6/219
Would that work for PCI transport also?
Thanks
vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 10 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register.
Did you mean the timeout is done by the hypervisor and the hypervisor can guarantee that the status register will not be blocked by the untrusted device?
Quoted 2 lines
Will that
allow Linux guest driver to gracefully fail its probe?
Note that the config read is not necessarily done in the path of
probe: For pci, we may need to read ISR in the INTX interrupt handler:
static irqreturn_t vp_interrupt(int irq, void *opaque)
{
struct virtio_pci_device *vp_dev = opaque;
u8 isr;
/* reading the ISR has the effect of also clearing it so it's very
* important to save off the value. */
isr = ioread8(vp_dev->isr);
}And there are more places that device could DOS a driver:
1) config space read and write after probe
2) control vq commands (we're busy waiting)
Quoted 4 lines
I don't see where Linux
handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
We can probably make virtio_cread() to be able to fail by checking DEVICE_NEEDS_RESET after the config space access. And the control vq commands, it can be done by having a timeout in the driver.
An alternative is to use or introduce a transport that can fail explificty in config space access.
Quoted 7 lines
I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.
https://lkml.org/lkml/2021/7/6/219
Would that work for PCI transport also?
For config space read, I think yes:
Hypervisor can maintain the config space and make sure the config read is completed fast. Need a protocol (for VDUSE it's the VDUSE_SET_CONFIG) for the device to set a new config space value to the hypervisor.
For config space writing, it needs more thoughts. (VDUSE only support read-only config space)
Thanks
Quoted 4 lines
Thanks
vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 13 lines · Jason Wang
Quoted 10 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register.
Quoted 3 lines
Did you mean the timeout is done by the hypervisor and the hypervisor can guarantee that the status register will not be blocked by the untrusted device?
Yes
Quoted 6 lines
Quoted 3 lines
Will that
allow Linux guest driver to gracefully fail its probe?Note that the config read is not necessarily done in the path of
probe: For pci, we may need to read ISR in the INTX interrupt handler:
Yes I was aware - we are dealing with MSI which I think does not read ISR in its interrut handler?
Quoted 16 lines
static
irqreturn_tvp_interrupt(int irq, void *opaque){ struct virtio_pci_device *vp_dev = opaque; u8 isr; /* reading the ISR has the effect of also clearing it so it's very * important to save off the value. */ isr = ioread8(vp_dev->isr); }And there are more places that device could DOS a driver:
1) config space read and write after probe
2) control vq commands (we're busy waiting)
Quoted 7 lines
Quoted 4 lines
I don't see where Linux
handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).We can probably make
virtio_cread()to be able to fail by checkingDEVICE_NEEDS_RESETafter the config space access.
I think there are other paths during probe where ioread() are issued - like virtio_pci_find_capability() for example. How do we handle stalls there (or rather ioreads returning -1 to indicate something amiss for example)?
Quoted 16 lines
And the control vq
commands, it can be done by having a timeout in the driver.An alternative is to use or introduce a transport that can fail explificty in config space access.
Quoted 7 lines
I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.
https://lkml.org/lkml/2021/7/6/219
Would that work for PCI transport also?
For config space read, I think yes:
Quoted 7 lines
Hypervisor can maintain the config space and make sure the config read is completed fast. Need a protocol (for VDUSE it's the
VDUSE_SET_CONFIG) for the device to set a new config space value to the hypervisor.For config space writing, it needs more thoughts. (VDUSE only support read-only config space)
Can you shed some light on what complexity handling config-space writes will require us to consider? I think its mostly the notification register that's written at runtime (beyond probe)? For notification writes, hypervisor can inject an event into backend and unblock the guest vcpu immediately.
Thanks
vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 4 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
Quoted 9 lines
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
DEVICE_NEEDS_RESET isn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.
Once thing you can do that will work a bit better is implementing surprise-removal in this case. So hypervisor detects a timeout (presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest. Note you will have to find a way to block device from poking at guest memory, implementing it in the hypervisor. We likely have some bugs around surprise-removal but generally are interested in fixing them.
Quoted 7 lines
I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.
https://lkml.org/lkml/2021/7/6/219
Would that work for PCI transport also?
Quoted 7 lines
Thanks
vatsa---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 15 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
Quoted 9 lines
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
Quoted 7 lines
DEVICE_NEEDS_RESETisn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.Once thing you can do that will work a bit better is implementing surprise-removal in this case.
My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?
Quoted 3 lines
So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.
Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?
Thanks
vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 22 lines · Srivatsa Vaddagiri
Quoted 20 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
Quoted 9 lines
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
DEVICE_NEEDS_RESETisn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.
Quoted 7 lines
Quoted 2 lines
Once thing you can do that will work a bit better is implementing surprise-removal in this case.
My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?
I think so, yes.
Quoted 10 lines
Quoted 3 lines
So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?
Thanks
vatsa
Removing guest access to device so access attempts end up in QEMU.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Srivatsa Vaddagiri
Quoted 22 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri
We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.
Quoted 9 lines
One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting
DEVICE_NEEDS_RESETin status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handlesDEVICE_NEEDS_RESETcurrently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).
DEVICE_NEEDS_RESETisn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.Once thing you can do that will work a bit better is implementing surprise-removal in this case.
Quoted 6 lines
Quoted 4 lines
My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?
I think so, yes.
Will that work during device probe also?
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.
Quoted 8 lines
Quoted 6 lines
Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?
Thanks
vatsaRemoving guest access to device so access attempts end up in QEMU.
Would QEMU end up terminating the guest or inject some bus error that guest VM can gracefully handle? We prefer not to bring down the VM because of this - rather have the driver probe fail gracefully.
Thanks
vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]