virtio-pci: enable bus master for old guests
PATCH 1/1 virtio-pci: enable bus master for old guests
commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices
Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.
Note: we should either back out the original patch from stable or apply this one on top.
--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.
Quoted 6 lines
Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>
I didn't signed off as I didn't write this patch ;). But you can add my reviewed-by if you like.
Jan
Quoted 19 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Michael S. Tsirkin
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Quoted 8 lines
Quoted 3 lines
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>I didn't signed off as I didn't write this patch ;). But you can add my reviewed-by if you like.
Jan
Should have been Cc :)
Quoted 23 lines
Quoted 19 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux
Quoting Michael S. Tsirkin (2014-09-08 11:05:02)
Quoted 10 lines
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
I've already backed out the other patch, and it looks we'll have to stick with that for now with the release being tomorrow. Will queue these back up for 2.1.2 though.
Quoted 24 lines
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Signature · 1 line
MST
Quoted 6 lines · Michael Roth
Quoting Michael S. Tsirkin (2014-09-08 11:05:02)
Quoted 5 lines
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices
Quoted 8 lines
Quoted 5 lines
Old guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
I've already backed out the other patch, and it looks we'll have to stick with that for now with the release being tomorrow.
That's fine.
Quoted 26 lines
Will queue these back
up for 2.1.2 though.Quoted 24 lines
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Signature · 1 line
MST
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0
hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :
Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:
hw/virtio/virtio-pci.c +7 -2
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }
If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Quoted 22 lines · Greg Kurz
Quoted 19 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 15 lines
hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 13 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Cc'ing Nikunj - he is SLOF expert.
I cannot comment - I have no idea what VIRTIO_PCI_FLAG_BUS_MASTER_BUG does.
Quoted 37 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND==0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issueVIRTIO_CONFIG_S_DRIVER_OKis not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:hw/virtio/virtio-pci.c +7
-2@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 1 line
Alexey
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).
Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.
Quoted 39 lines
The guest then continues to boot and hangs... It appears that waiting for the guest to issue
VIRTIO_CONFIG_S_DRIVER_OKis not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:hw/virtio/virtio-pci.c +7
-2@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Quoted 49 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).
Quoted 7 lines
Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.
BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
Quoted 39 lines
Quoted 39 lines
The guest then continues to boot and hangs... It appears that waiting for the guest to issue
VIRTIO_CONFIG_S_DRIVER_OKis not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:hw/virtio/virtio-pci.c +7
-2@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Quoted 53 lines · On Wed, 10 Sep 2014 12:32:30 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 52 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Quoted 8 lines
Quoted 4 lines
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64 prom_init() call).
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
The guest then continues to boot and hangs... It appears that waiting for the guest to issue
VIRTIO_CONFIG_S_DRIVER_OKis not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:hw/virtio/virtio-pci.c +7
-2@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Quoted 12 lines · Greg Kurz
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.
Quoted 7 lines
Quoted 3 lines
BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64
prom_init()call).
Is it before quiesce call?
Regards
Nikunj
Quoted 61 lines · Greg Kurz
Quoted 58 lines · On Wed, 10 Sep 2014 12:32:30 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 56 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.
Quoted 7 lines
Quoted 3 lines
BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64
prom_init()call).
OK I'm not sure I have all the details but does the patch I sent help you?
Signature · 1 line
MST
Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Quoted 7 lines
Quoted 4 lines
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
In SLOF, we do enable PCI MASTER during device scanning and then later disable it.
Regards
Nikunj
Quoted 60 lines · Nikunj A Dadhania
Quoted 58 lines · Michael S. Tsirkin
Quoted 56 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.
Quoted 8 lines
Quoted 2 lines
BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.
In SLOF, we do enable PCI MASTER during device scanning and then later disable it.
Regards
Nikunj
But device is then reset, right Greg?
You get as far as reset?
If yes I doubt something that happens before reset matters, unless we are leaking some state across reset which would be a bug in itself.
Quoted 45 lines · On Wed, 10 Sep 2014 13:44:49 +0530 Nikunj A Dadhania…
Quoted 42 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...
Quoted 7 lines
Quoted 3 lines
I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (
PCI_COMMAND==0x3).Getting confused, above you are talking about virtio-net and here it is virtio-blk.
I tried running rhel6.5 (old kernel that doesn't enable bus mastering on virtio PCI devices), with a virtio-blk based disk and a virtio-net based NIC for both x86_64 and ppc64. Results are as follow:
- x86_64: boots well but fails to activate network
- ppc64: does not boot because the virtio-blk notification doesn't reach the guest
Quoted 5 lines
Anyways, the routines still remains same for both of them. From SLOF during init we set
DRIVER_OK, and after using the device during the quiesce, called from linux kernelVIRTIO_CONFIG_S_FAILEDis set and then aVIRTIO_DEVICE_RESETis done.
I chose to debug by attaching gdb to qemu-system-ppc64 itself. It appears that SLOF seems to be enabling bus master during init but at some point bus master gets disabled... unfortunately my SLOF knowledge is limited and I don't know how exactly what's happening in the guest between SLOF and the kernel.
Quoted 39 lines
Quoted 39 lines
The guest then continues to boot and hangs... It appears that waiting for the guest to issue
VIRTIO_CONFIG_S_DRIVER_OKis not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:hw/virtio/virtio-pci.c +7
-2@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) {VirtIOPCIProxy *proxy =to_virtio_pci_proxy_fast (d); -if (msix_enabled (&proxy->pci_dev)) +if (msix_enabled (&proxy->pci_dev)) { +if (!(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { + proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }msix_notify (&proxy->pci_dev, vector); -else { + }else {VirtIODevice *vdev =virtio_bus_get_device (&proxy->bus);pci_set_irq (&proxy->pci_dev, vdev->isr &1 ); }If this is acceptable, I'll make it a helper and squash it into your patch.
Thoughts ?
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Signature · 7 lines
Gregory Kurz [email protected] [email protected] Software Engineer @ IBM/Meiosys http://www.ibm.com Tel +33 (0)562 165 496 "Anarchy is about taking complete responsibility for yourself." Alan Moore.
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
commit
cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devicesOld guests forgot to enable bus mastering, enable it automatically on
DRIVER_OK.Note: we should either back out the original patch from stable or apply this one on top.
Reported-by Greg Kurz <[email protected]>Signed-off-by Jan Kiszka <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- hw/virtio/virtio-pci.c | 2 ++ 1 file changed, 2 insertions(+)hw/virtio/virtio-pci.c +0
-0hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327
Quoted 8 lines
hw/virtio/virtio-pci.c +2
-0@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)if ((val &VIRTIO_CONFIG_S_DRIVER_OK ) && !(proxy->pci_dev.config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { proxy->flags |=VIRTIO_PCI_FLAG_BUS_MASTER_BUG ; +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); }break ;case VIRTIO_MSI_CONFIG_VECTOR :Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.
Quoted 8 lines
Michael,
This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND==0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issueVIRTIO_CONFIG_S_DRIVER_OKis not enough.
Got it. Writing to PCI_COMMAND disabled bus mastering again. This should do it, on top - can you confirm please?
hw/virtio/virtio-pci.c +8 -4
@@ -475,10 +475,14 @@ static void virtio_write_config(PCIDevice *pci_dev, uint32_t address,pci_default_write_config (pci_dev, address, val, len);if (range_covers_byte (address, len,PCI_COMMAND ) && - !(pci_dev->config[PCI_COMMAND ] &PCI_COMMAND_MASTER ) && - !(proxy->flags &VIRTIO_PCI_FLAG_BUS_MASTER_BUG )) { -virtio_pci_stop_ioeventfd (proxy); -virtio_set_status (vdev, vdev->status & ~VIRTIO_CONFIG_S_DRIVER_OK ); + !(pci_dev->config[PCI_COMMAND ] &PCI_COMMAND_MASTER )) { +if (proxy->flags &VIRTIO_PCI_FLAG_BUS_MASTER_BUG ) { +memory_region_set_enabled (&proxy->pci_dev.bus_master_enable_region, + true); + }else { +virtio_pci_stop_ioeventfd (proxy); +virtio_set_status (vdev, vdev->status & ~VIRTIO_CONFIG_S_DRIVER_OK ); + } } }