Skip to contentSkip to conversation

virtio-pci: enable bus master for old guests

PATCH Superseded Michael S. Tsirkin 6 participants 16 messages Sep 8, 2014 – Sep 10, 2014

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio-pci: enable bus master for old guests

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:
Signature · 1 line
MST
#1 lore raw
Quoted 8 lines · Michael S. Tsirkin

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Quoted 6 lines

Note: we should either back out the original patch from stable or apply this one on top.

I didn't signed off as I didn't write this patch ;). But you can add my reviewed-by if you like.

Jan

Quoted 19 lines
---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
#2 lore raw
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Michael S. Tsirkin

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

Quoted 8 lines
Quoted 3 lines

I didn't signed off as I didn't write this patch ;). But you can add my reviewed-by if you like.

Jan

Should have been Cc :)

Quoted 23 lines
Quoted 19 lines
---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
#3 lore raw
In reply to Michael S. Tsirkin

Quoting Michael S. Tsirkin (2014-09-08 11:05:02)

Quoted 10 lines

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

I've already backed out the other patch, and it looks we'll have to stick with that for now with the release being tomorrow. Will queue these back up for 2.1.2 though.

Quoted 24 lines
---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:
Signature · 1 line
MST
#4 lore raw
Quoted 6 lines · Michael Roth

Quoting Michael S. Tsirkin (2014-09-08 11:05:02)

Quoted 5 lines

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Quoted 8 lines
Quoted 5 lines

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

I've already backed out the other patch, and it looks we'll have to stick with that for now with the release being tomorrow.

That's fine.

Quoted 26 lines

Will queue these back
up for 2.1.2 though.

Quoted 24 lines
---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:
Signature · 1 line
MST
#5 lore raw
In reply to Michael S. Tsirkin
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
#6 lore raw
Quoted 22 lines · Greg Kurz
Quoted 19 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
Quoted 15 lines

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 13 lines
hw/virtio/virtio-pci.c +2 -0
index ddb5da1..af937d2 100644
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Cc'ing Nikunj - he is SLOF expert.
I cannot comment - I have no idea what VIRTIO_PCI_FLAG_BUS_MASTER_BUG does.

Quoted 37 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 1 line
Alexey
#7 lore raw
In reply to Greg Kurz
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

Quoted 39 lines

The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
#8 lore raw
Quoted 49 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Quoted 7 lines

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

Quoted 39 lines
Quoted 39 lines

The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
#9 lore raw
Quoted 53 lines · On Wed, 10 Sep 2014 12:32:30 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 52 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Quoted 8 lines
Quoted 4 lines

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64 prom_init() call).

Quoted 40 lines
Quoted 39 lines
Quoted 39 lines

The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
#10 lore raw
Quoted 12 lines · Greg Kurz
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

Quoted 7 lines
Quoted 3 lines

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64 prom_init() call).

Is it before quiesce call?

Regards
Nikunj

#11 lore raw
In reply to Greg Kurz
Quoted 61 lines · Greg Kurz
Quoted 58 lines · On Wed, 10 Sep 2014 12:32:30 +0300 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 56 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

Quoted 7 lines
Quoted 3 lines

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64 prom_init() call).

OK I'm not sure I have all the details but does the patch I sent help you?

Signature · 1 line
MST
#12 lore raw
In reply to Michael S. Tsirkin
Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Quoted 7 lines
Quoted 4 lines

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

In SLOF, we do enable PCI MASTER during device scanning and then later disable it.

Regards
Nikunj

#13 lore raw
Quoted 60 lines · Nikunj A Dadhania
Quoted 58 lines · Michael S. Tsirkin
Quoted 56 lines · Nikunj A Dadhania
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

Quoted 8 lines
Quoted 2 lines

BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests.

In SLOF, we do enable PCI MASTER during device scanning and then later disable it.

Regards
Nikunj

But device is then reset, right Greg?
You get as far as reset?

If yes I doubt something that happens before reset matters, unless we are leaking some state across reset which would be a bug in itself.

#14 lore raw
In reply to Nikunj A Dadhania
Quoted 45 lines · On Wed, 10 Sep 2014 13:44:49 +0530 Nikunj A Dadhania…
Quoted 42 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called...

Quoted 7 lines
Quoted 3 lines

I did some debugging: it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3).

Getting confused, above you are talking about virtio-net and here it is virtio-blk.

I tried running rhel6.5 (old kernel that doesn't enable bus mastering on virtio PCI devices), with a virtio-blk based disk and a virtio-net based NIC for both x86_64 and ppc64. Results are as follow:
- x86_64: boots well but fails to activate network
- ppc64: does not boot because the virtio-blk notification doesn't reach the guest

Quoted 5 lines

Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done.

I chose to debug by attaching gdb to qemu-system-ppc64 itself. It appears that SLOF seems to be enabling bus master during init but at some point bus master gets disabled... unfortunately my SLOF knowledge is limited and I don't know how exactly what's happening in the guest between SLOF and the kernel.

Quoted 39 lines
Quoted 39 lines

The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for MSI to work, I tried the following and it fixes the issue:

hw/virtio/virtio-pci.c +7 -2
index af937d2..3d72aa8 100644
@@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d);
 
-    if (msix_enabled(&proxy->pci_dev))
+    if (msix_enabled(&proxy->pci_dev)) {
+        if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+            proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        }
         msix_notify(&proxy->pci_dev, vector);
-    else {
+    } else {
         VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
         pci_set_irq(&proxy->pci_dev, vdev->isr & 1);
     }

If this is acceptable, I'll make it a helper and squash it into your patch.

Thoughts ?

Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
Signature · 7 lines
Gregory Kurz                                     [email protected]
                                                 [email protected]
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.
#15 lore raw
In reply to Greg Kurz
Quoted 38 lines · Greg Kurz
Quoted 24 lines · On Mon, 8 Sep 2014 19:05:02 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

commit cc943c36faa192cd4b32af8fe5edb31894017d35
pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't enable bus mastering for virtio PCI devices

Old guests forgot to enable bus mastering, enable it automatically on DRIVER_OK.

Note: we should either back out the original patch from stable or apply this one on top.

---
 hw/virtio/virtio-pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index ddb5da1..af937d2 100644

hw/virtio/virtio-pci.c static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) · line 327

Quoted 8 lines
hw/virtio/virtio-pci.c +2 -0
@@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
         if ((val & VIRTIO_CONFIG_S_DRIVER_OK) &&
             !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
             proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG;
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
         }
         break;
     case VIRTIO_MSI_CONFIG_VECTOR:

Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise.

Quoted 8 lines

Michael,

This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, this fails for rhel6.5 ppc64 because it is never called... I did some debugging:
it looks like the guest kernel calls the OF quisece call to flush pending DMA and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3). The guest then continues to boot and hangs... It appears that waiting for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough.

Got it. Writing to PCI_COMMAND disabled bus mastering again. This should do it, on top - can you confirm please?

hw/virtio/virtio-pci.c +8 -4
index af937d2..6b7ac39 100644
@@ -475,10 +475,14 @@ static void virtio_write_config(PCIDevice *pci_dev, uint32_t address,
     pci_default_write_config(pci_dev, address, val, len);
 
     if (range_covers_byte(address, len, PCI_COMMAND) &&
-        !(pci_dev->config[PCI_COMMAND] & PCI_COMMAND_MASTER) &&
-        !(proxy->flags & VIRTIO_PCI_FLAG_BUS_MASTER_BUG)) {
-        virtio_pci_stop_ioeventfd(proxy);
-        virtio_set_status(vdev, vdev->status & ~VIRTIO_CONFIG_S_DRIVER_OK);
+        !(pci_dev->config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
+        if (proxy->flags & VIRTIO_PCI_FLAG_BUS_MASTER_BUG) {
+            memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region,
+                                      true);
+        } else {
+            virtio_pci_stop_ioeventfd(proxy);
+            virtio_set_status(vdev, vdev->status & ~VIRTIO_CONFIG_S_DRIVER_OK);
+        }
     }
 }