virtio_pci: properly clean up MSI-X state when initialization fails
PATCH 1/1 virtio_pci: properly clean up MSI-X state when initialization fails
From: Anthony Liguori <[email protected]>
If MSI-X initialization fails after setting msix_enabled = 1, then the device is left in an inconsistent state. This would normally only happen if there was a bug in the device emulation but it still should be handled correctly.
--- drivers/virtio/virtio_pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/virtio/virtio_pci.c +6 -2
@@ -357,7 +357,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v =ioread16 (vp_dev->ioaddr +VIRTIO_MSI_CONFIG_VECTOR );if (v ==VIRTIO_MSI_NO_VECTOR ) { err = -EBUSY ; - goto error; + goto error_msix_used; }if (!per_vq_vectors) {
@@ -369,11 +369,15 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, vp_vring_interrupt,0 , vp_dev->msix_names[v], vp_dev);if (err) - goto error; + goto error_msix_used; ++vp_dev->msix_used_vectors; }return 0 ; +error_msix_used: + v = --vp_dev->msix_used_vectors; +free_irq (vp_dev->msix_entries[v].vector, vp_dev); error: + vp_dev->msix_enabled =0 ;vp_free_vectors (vdev);return err; }
Signature · 1 line
1.7.9.5
Quoted 11 lines · Anthony Liguori
From: Anthony Liguori <[email protected]>
If MSI-X initialization fails after setting
msix_enabled= 1, then the device is left in an inconsistent state. This would normally only happen if there was a bug in the device emulation but it still should be handled correctly.Cc Matt Wilson <[email protected]>Cc Rusty Russell <[email protected]>Cc Michael Tsirkin <[email protected]>Signed-off-by Anthony Liguori <[email protected]>
--msw
Quoted 34 lines
--- drivers/virtio/virtio_pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)drivers/virtio/virtio_pci.c +6
-2@@ -357,7 +357,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v =ioread16 (vp_dev->ioaddr +VIRTIO_MSI_CONFIG_VECTOR );if (v ==VIRTIO_MSI_NO_VECTOR ) { err = -EBUSY ; - goto error; + goto error_msix_used; }if (!per_vq_vectors) {@@ -369,11 +369,15 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, vp_vring_interrupt,0 , vp_dev->msix_names[v], vp_dev);if (err) - goto error; + goto error_msix_used; ++vp_dev->msix_used_vectors; }return 0 ; +error_msix_used: + v = --vp_dev->msix_used_vectors; +free_irq (vp_dev->msix_entries[v].vector, vp_dev); error: + vp_dev->msix_enabled =0 ;vp_free_vectors (vdev);return err; }
Michael S. Tsirkin
maintainerQuoted 6 lines · Anthony Liguori
From: Anthony Liguori <[email protected]>
If MSI-X initialization fails after setting
msix_enabled= 1, then the device is left in an inconsistent state. This would normally only happen if there was a bug in the device emulation but it still should be handled correctly.
This might happen if host runs out of resources when trying to map VQs to vectors, so doesn't have to be a bug.
But I don't see what the problem is:msix_used_vectors reflects the number of used vectors and msix_enabled is set, thus vp_free_vectors will free all IRQs and then disable MSIX.
Where is the inconsistency you speak about?
Quoted 26 lines
Cc Matt Wilson <[email protected]>Cc Rusty Russell <[email protected]>Cc Michael Tsirkin <[email protected]>Signed-off-by Anthony Liguori <[email protected]>--- drivers/virtio/virtio_pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)drivers/virtio/virtio_pci.c +1
-1@@ -357,7 +357,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v =ioread16 (vp_dev->ioaddr +VIRTIO_MSI_CONFIG_VECTOR );if (v ==VIRTIO_MSI_NO_VECTOR ) { err = -EBUSY ; - goto error; + goto error_msix_used; }if (!per_vq_vectors) {@@ -369,11 +369,15 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, vp_vring_interrupt,0 , vp_dev->msix_names[v], vp_dev);if (err)
drivers/virtio/virtio_pci.c static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, · line 380
Quoted 10 lines
drivers/virtio/virtio_pci.c +5
-1- goto error; + goto error_msix_used; ++vp_dev->msix_used_vectors; }return 0 ; +error_msix_used: + v = --vp_dev->msix_used_vectors; +free_irq (vp_dev->msix_entries[v].vector, vp_dev); error: + vp_dev->msix_enabled =0 ;
As far as I can see, if you do this, guest will not call pci_disable_msix thus leaving the device with MSIX enabled. I'm not sure this won't break drivers if they then try to use the device without MSIX, and it definitely seems less elegant than reverting the device to the original state.
Quoted 5 lines
drivers/virtio/virtio_pci.c +0
-0vp_free_vectors (vdev);return err; }Signature · 1 line
1.7.9.5
Hi Michael,
Quoted 11 lines · Michael S. Tsirkin
Quoted 6 lines · Anthony Liguori
From: Anthony Liguori <[email protected]>
If MSI-X initialization fails after setting
msix_enabled= 1, then the device is left in an inconsistent state. This would normally only happen if there was a bug in the device emulation but it still should be handled correctly.This might happen if host runs out of resources when trying to map VQs to vectors, so doesn't have to be a bug.
Quoted 6 lines
But I don't see what the problem is:
msix_used_vectorsreflects the number of used vectors andmsix_enabledis set, thusvp_free_vectorswill free all IRQs and then disable MSIX.Where is the inconsistency you speak about?
I missed the fact that vp_free_vectors() conditionally sets msix_enabled=0. It seems a bit cludgy especially since it is called both before and after setting msix_enabled=1.
I ran into a number of weird problems due to read/write reordering that was causing this code path to fail. The impact was non-deterministic. I'll go back and try to better understand what code path is causing it.
Quoted 30 lines
Quoted 30 lines
Cc Matt Wilson <[email protected]>Cc Rusty Russell <[email protected]>Cc Michael Tsirkin <[email protected]>Signed-off-by Anthony Liguori <[email protected]>--- drivers/virtio/virtio_pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)drivers/virtio/virtio_pci.c +2
-2@@ -357,7 +357,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v =ioread16 (vp_dev->ioaddr +VIRTIO_MSI_CONFIG_VECTOR );if (v ==VIRTIO_MSI_NO_VECTOR ) { err = -EBUSY ; - goto error; + goto error_msix_used; }if (!per_vq_vectors) {@@ -369,11 +369,15 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, vp_vring_interrupt,0 , vp_dev->msix_names[v], vp_dev);if (err) - goto error; + goto error_msix_used; ++vp_dev->msix_used_vectors; }return 0 ;
Quoted 8 lines
drivers/virtio/virtio_pci.c static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, · line 380
Quoted 5 lines
drivers/virtio/virtio_pci.c +4
-0+error_msix_used: + v = --vp_dev->msix_used_vectors; +free_irq (vp_dev->msix_entries[v].vector, vp_dev); error: + vp_dev->msix_enabled =0 ;As far as I can see, if you do this, guest will not call
pci_disable_msixthus leaving the device with MSIX enabled.
I don't understand this comment. How is the work done in this path any different from what's done in vp_free_vectors()?
Regards,
Anthony Liguori
Quoted 11 lines
I'm not sure this won't break drivers if they then try to use the device without MSIX, and it definitely seems less elegant than reverting the device to the original state.
Quoted 5 lines
drivers/virtio/virtio_pci.c +0
-1vp_free_vectors (vdev);return err; } --1.7.9.5
Michael S. Tsirkin
maintainerQuoted 19 lines · Anthony Liguori
Hi Michael,
Quoted 16 lines · Michael S. Tsirkin
Quoted 6 lines · Anthony Liguori
From: Anthony Liguori <[email protected]>
If MSI-X initialization fails after setting
msix_enabled= 1, then the device is left in an inconsistent state. This would normally only happen if there was a bug in the device emulation but it still should be handled correctly.This might happen if host runs out of resources when trying to map VQs to vectors, so doesn't have to be a bug.
But I don't see what the problem is:
msix_used_vectorsreflects the number of used vectors andmsix_enabledis set, thusvp_free_vectorswill free all IRQs and then disable MSIX.
Quoted 5 lines
Quoted 1 line
Where is the inconsistency you speak about?
I missed the fact that
vp_free_vectors()conditionally sets msix_enabled=0. It seems a bit cludgy especially since it is called both before and after setting msix_enabled=1.
It's the style of initialization that records the current initialization stage, and then uses that to do all cleanup in a single place (as compared to detailed separate goto labels for each initialization stage).
I don't mind either keeping this style or changing to another style, but if we change it we should change it everywhere I think.
Quoted 62 lines
I ran into a number of weird problems due to read/write reordering that was causing this code path to fail. The impact was non-deterministic. I'll go back and try to better understand what code path is causing it.
Quoted 30 lines
Quoted 30 lines
Cc Matt Wilson <[email protected]>Cc Rusty Russell <[email protected]>Cc Michael Tsirkin <[email protected]>Signed-off-by Anthony Liguori <[email protected]>--- drivers/virtio/virtio_pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)drivers/virtio/virtio_pci.c +2
-2@@ -357,7 +357,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, v =ioread16 (vp_dev->ioaddr +VIRTIO_MSI_CONFIG_VECTOR );if (v ==VIRTIO_MSI_NO_VECTOR ) { err = -EBUSY ; - goto error; + goto error_msix_used; }if (!per_vq_vectors) {@@ -369,11 +369,15 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, vp_vring_interrupt,0 , vp_dev->msix_names[v], vp_dev);if (err) - goto error; + goto error_msix_used; ++vp_dev->msix_used_vectors; }return 0 ;Quoted 8 lines
drivers/virtio/virtio_pci.c static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, · line 380
Quoted 5 lines
drivers/virtio/virtio_pci.c +4
-0+error_msix_used: + v = --vp_dev->msix_used_vectors; +free_irq (vp_dev->msix_entries[v].vector, vp_dev); error: + vp_dev->msix_enabled =0 ;As far as I can see, if you do this, guest will not call
pci_disable_msixthus leaving the device with MSIX enabled.I don't understand this comment. How is the work done in this path any different from what's done in
vp_free_vectors()?Regards,
Anthony Liguori
Quoted 11 lines
I'm not sure this won't break drivers if they then try to use the device without MSIX, and it definitely seems less elegant than reverting the device to the original state.
Quoted 5 lines
drivers/virtio/virtio_pci.c +0
-1vp_free_vectors (vdev);return err; } --1.7.9.5