Skip to contentSkip to conversation

Timing out virtio-pci config space access

virtio-dev Srivatsa Vaddagiri 3 participants 7 messages Nov 4, 2021 – Nov 5, 2021

lore
Share & tools
lore
#0 lore raw

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.

https://lkml.org/lkml/2021/7/6/219

Would that work for PCI transport also?

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#1 lore raw
Quoted 10 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register.

Did you mean the timeout is done by the hypervisor and the hypervisor can guarantee that the status register will not be blocked by the untrusted device?

Quoted 2 lines

Will that
allow Linux guest driver to gracefully fail its probe?

Note that the config read is not necessarily done in the path of
probe: For pci, we may need to read ISR in the INTX interrupt handler:

static irqreturn_t vp_interrupt(int irq, void *opaque)

{
        struct virtio_pci_device *vp_dev = opaque;
        u8 isr;

        /* reading the ISR has the effect of also clearing it so it's very
         * important to save off the value. */
        isr = ioread8(vp_dev->isr);
}

And there are more places that device could DOS a driver:

1) config space read and write after probe
2) control vq commands (we're busy waiting)

Quoted 4 lines

I don't see where Linux
handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

We can probably make virtio_cread() to be able to fail by checking DEVICE_NEEDS_RESET after the config space access. And the control vq commands, it can be done by having a timeout in the driver.

An alternative is to use or introduce a transport that can fail explificty in config space access.

Quoted 7 lines

I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.

https://lkml.org/lkml/2021/7/6/219

Would that work for PCI transport also?

For config space read, I think yes:

Hypervisor can maintain the config space and make sure the config read is completed fast. Need a protocol (for VDUSE it's the VDUSE_SET_CONFIG) for the device to set a new config space value to the hypervisor.

For config space writing, it needs more thoughts. (VDUSE only support read-only config space)

Thanks

Quoted 4 lines

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#2 lore raw
Quoted 13 lines · Jason Wang
Quoted 10 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register.

Quoted 3 lines

Did you mean the timeout is done by the hypervisor and the hypervisor can guarantee that the status register will not be blocked by the untrusted device?

Yes

Quoted 6 lines
Quoted 3 lines

Will that
allow Linux guest driver to gracefully fail its probe?

Note that the config read is not necessarily done in the path of
probe: For pci, we may need to read ISR in the INTX interrupt handler:

Yes I was aware - we are dealing with MSI which I think does not read ISR in its interrut handler?

Quoted 16 lines

static irqreturn_t vp_interrupt(int irq, void *opaque)

{
        struct virtio_pci_device *vp_dev = opaque;
        u8 isr;

        /* reading the ISR has the effect of also clearing it so it's very
         * important to save off the value. */
        isr = ioread8(vp_dev->isr);
}

And there are more places that device could DOS a driver:

1) config space read and write after probe
2) control vq commands (we're busy waiting)

Quoted 7 lines
Quoted 4 lines

I don't see where Linux
handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

We can probably make virtio_cread() to be able to fail by checking DEVICE_NEEDS_RESET after the config space access.

I think there are other paths during probe where ioread() are issued - like virtio_pci_find_capability() for example. How do we handle stalls there (or rather ioreads returning -1 to indicate something amiss for example)?

Quoted 16 lines

And the control vq
commands, it can be done by having a timeout in the driver.

An alternative is to use or introduce a transport that can fail explificty in config space access.

Quoted 7 lines

I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.

https://lkml.org/lkml/2021/7/6/219

Would that work for PCI transport also?

For config space read, I think yes:

Quoted 7 lines

Hypervisor can maintain the config space and make sure the config read is completed fast. Need a protocol (for VDUSE it's the VDUSE_SET_CONFIG) for the device to set a new config space value to the hypervisor.

For config space writing, it needs more thoughts. (VDUSE only support read-only config space)

Can you shed some light on what complexity handling config-space writes will require us to consider? I think its mostly the notification register that's written at runtime (beyond probe)? For notification writes, hypervisor can inject an event into backend and unblock the guest vcpu immediately.

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#3 lore raw
In reply to Srivatsa Vaddagiri
Quoted 4 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

Quoted 9 lines

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

DEVICE_NEEDS_RESET isn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.

Once thing you can do that will work a bit better is implementing surprise-removal in this case. So hypervisor detects a timeout (presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest. Note you will have to find a way to block device from poking at guest memory, implementing it in the hypervisor. We likely have some bugs around surprise-removal but generally are interested in fixing them.

Quoted 7 lines

I saw some discussions in this regard for vDPA where similar solution seem to have been discussed.

https://lkml.org/lkml/2021/7/6/219

Would that work for PCI transport also?

Quoted 7 lines

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#4 lore raw
Quoted 15 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

Quoted 9 lines

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

Quoted 7 lines

DEVICE_NEEDS_RESET isn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.

Once thing you can do that will work a bit better is implementing surprise-removal in this case.

My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?

Quoted 3 lines

So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.

Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#5 lore raw
Quoted 22 lines · Srivatsa Vaddagiri
Quoted 20 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

Quoted 9 lines

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

DEVICE_NEEDS_RESET isn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.

Quoted 7 lines
Quoted 2 lines

Once thing you can do that will work a bit better is implementing surprise-removal in this case.

My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?

I think so, yes.

Quoted 10 lines
Quoted 3 lines

So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.

Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?

Thanks
vatsa

Removing guest access to device so access attempts end up in QEMU.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#6 lore raw
Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Srivatsa Vaddagiri
Quoted 22 lines · Michael S. Tsirkin
Quoted 4 lines · Srivatsa Vaddagiri

We are working on a virtio-pci implementation on a Type-1 hypervisor where backend drivers are hosted in another VM and are considered untrusted. PCI is the virtio transport used in this case.

Quoted 9 lines

One issue that crops up is a read/write of config space can potentially block forever, as the backend is untrusted and could be causing a denial-of-service of sorts. This causes the vcpu to stall forever. I was wondering if we can timeout in such case and have the hypervisor break the stall by letting read return "error" (-1) along with setting DEVICE_NEEDS_RESET in status register. Will that allow Linux guest driver to gracefully fail its probe? I don't see where Linux handles DEVICE_NEEDS_RESET currently and also am not sure if returning -1 will lead to graceful failure of the driver alone (we don't want VM to come down or panic because of a mis-behaving device).

DEVICE_NEEDS_RESET isn't handled ATM. the point of it in any case is a recoverable error, with a malicious backend this is not the case.

Once thing you can do that will work a bit better is implementing surprise-removal in this case.

Quoted 6 lines
Quoted 4 lines

My layman understanding of surprise removal is that it requires the PCI controller to interrupt OS and convey which device is removed, so that the PCI subsystem can mark it "removed"? Is that possible for the generic controller ("pci-host-ecam-generic") that virtio pci devices use?

I think so, yes.

Will that work during device probe also?

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

So hypervisor detects a timeout
(presumably it knows what to expect of the device) and then pretends to guest device is gone, unmapping it completely from guest.

Quoted 8 lines
Quoted 6 lines

Can you elaborate on what unmapping means? I think the reads should return -1 and writes to be dropped in such case - beyond that what would unmap entail?

Thanks
vatsa

Removing guest access to device so access attempts end up in QEMU.

Would QEMU end up terminating the guest or inject some bus error that guest VM can gracefully handle? We prefer not to bring down the VM because of this - rather have the driver probe fail gracefully.

Thanks
vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]