Skip to contentSkip to conversation

target-arm: Add some omitted dma cache coherency flags

PATCH Applied Alexander Graf 6 participants 13 messages Feb 9, 2017 – Feb 10, 2017 Arm/versatile express platform

lore
Share & tools
lore
#0 lore raw

v2 cover target-arm: Add some omitted dma cache coherency flags

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

 hw/arm/vexpress.c        | 1 +
 hw/arm/virt-acpi-build.c | 2 ++
 hw/arm/virt.c            | 2 ++
 3 files changed, 5 insertions(+)
Signature · 1 line
2.10.0
#1 lore raw

PATCH v2 1/4 target-arm: Declare virtio-mmio as dma-coherent in dt

QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.

Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.

This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.

---
 hw/arm/vexpress.c | 1 +
 hw/arm/virt.c     | 1 +
 2 files changed, 2 insertions(+)
hw/arm/vexpress.c +1 -0
index 58760f4..e057568 100644
@@ -452,6 +452,7 @@ static int add_virtio_mmio_node(void *fdt, uint32_t acells, uint32_t scells,
                                        acells, addr, scells, size);
     qemu_fdt_setprop_cells(fdt, nodename, "interrupt-parent", intc);
     qemu_fdt_setprop_cells(fdt, nodename, "interrupts", 0, irq, 1);
+    qemu_fdt_setprop(fdt, nodename, "dma-coherent", NULL, 0);
     g_free(nodename);
     if (rc) {
         return -1;
hw/arm/virt.c +1 -0
index 1f216cf..14881fa 100644
@@ -797,6 +797,7 @@ static void create_virtio_devices(const VirtMachineState *vms, qemu_irq *pic)
         qemu_fdt_setprop_cells(vms->fdt, nodename, "interrupts",
                                GIC_FDT_IRQ_TYPE_SPI, irq,
                                GIC_FDT_IRQ_FLAGS_EDGE_LO_HI);
+        qemu_fdt_setprop(vms->fdt, nodename, "dma-coherent", NULL, 0);
         g_free(nodename);
     }
 }
Signature · 1 line
2.10.0
#2 lore raw

PATCH v2 2/4 hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI

In reply to Alexander Graf

Virtio-mmio devices can directly access guest memory and do so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.

---
 hw/arm/virt-acpi-build.c | 1 +
 1 file changed, 1 insertion(+)
hw/arm/virt-acpi-build.c +1 -0
index 07a10ac..8955a9d 100644
@@ -135,6 +135,7 @@ static void acpi_dsdt_add_virtio(Aml *scope,
         Aml *dev = aml_device("VR%02u", i);
         aml_append(dev, aml_name_decl("_HID", aml_string("LNRO0005")));
         aml_append(dev, aml_name_decl("_UID", aml_int(i)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
 
         Aml *crs = aml_resource_template();
         aml_append(crs, aml_memory32_fixed(base, size, AML_READ_WRITE));
Signature · 1 line
2.10.0
Reviewed-by on v2 2/4
#3 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/4] hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI

Quoted 8 lines · Alexander Graf

Virtio-mmio devices can directly access guest memory and do so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.

---
 hw/arm/virt-acpi-build.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 13 lines
hw/arm/virt-acpi-build.c +1 -0
index 07a10ac..8955a9d 100644
@@ -135,6 +135,7 @@ static void acpi_dsdt_add_virtio(Aml *scope,
         Aml *dev = aml_device("VR%02u", i);
         aml_append(dev, aml_name_decl("_HID", aml_string("LNRO0005")));
         aml_append(dev, aml_name_decl("_UID", aml_int(i)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
 
         Aml *crs = aml_resource_template();
         aml_append(crs, aml_memory32_fixed(base, size, AML_READ_WRITE));
Signature · 1 line
Shannon
#4 lore raw

PATCH v2 3/4 hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI

In reply to Alexander Graf

Fw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.

---
 hw/arm/virt-acpi-build.c | 1 +
 1 file changed, 1 insertion(+)
hw/arm/virt-acpi-build.c +1 -0
index 8955a9d..0835e59 100644
@@ -90,6 +90,7 @@ static void acpi_dsdt_add_fw_cfg(Aml *scope, const MemMapEntry *fw_cfg_memmap)
     aml_append(dev, aml_name_decl("_HID", aml_string("QEMU0002")));
     /* device present, functioning, decoding, not shown in UI */
     aml_append(dev, aml_name_decl("_STA", aml_int(0xB)));
+    aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
 
     Aml *crs = aml_resource_template();
     aml_append(crs, aml_memory32_fixed(fw_cfg_memmap->base,
Signature · 1 line
2.10.0
Reviewed-by on v2 3/4
#5 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 3/4] hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI

Quoted 8 lines · Alexander Graf

Fw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.

---
 hw/arm/virt-acpi-build.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 13 lines
hw/arm/virt-acpi-build.c +1 -0
index 8955a9d..0835e59 100644
@@ -90,6 +90,7 @@ static void acpi_dsdt_add_fw_cfg(Aml *scope, const MemMapEntry *fw_cfg_memmap)
     aml_append(dev, aml_name_decl("_HID", aml_string("QEMU0002")));
     /* device present, functioning, decoding, not shown in UI */
     aml_append(dev, aml_name_decl("_STA", aml_int(0xB)));
+    aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
 
     Aml *crs = aml_resource_template();
     aml_append(crs, aml_memory32_fixed(fw_cfg_memmap->base,
Signature · 1 line
Shannon
#6 lore raw

PATCH v2 4/4 hw/arm/virt: Declare fwcfg as dma cache coherent in dt

In reply to Alexander Graf

Fw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using DT.

---
 hw/arm/virt.c | 1 +
 1 file changed, 1 insertion(+)
hw/arm/virt.c +1 -0
index 14881fa..da9622f 100644
@@ -929,6 +929,7 @@ static FWCfgState *create_fw_cfg(const VirtMachineState *vms, AddressSpace *as)
                             "compatible", "qemu,fw-cfg-mmio");
     qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg",
                                  2, base, 2, size);
+    qemu_fdt_setprop(vms->fdt, nodename, "dma-coherent", NULL, 0);
     g_free(nodename);
     return fw_cfg;
 }
Signature · 1 line
2.10.0
Reviewed-by on the v2 series
#7 lore raw
In reply to Alexander Graf
Quoted 32 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

Quoted 5 lines
 hw/arm/vexpress.c        | 1 +
 hw/arm/virt-acpi-build.c | 2 ++
 hw/arm/virt.c            | 2 ++
 3 files changed, 5 insertions(+)

Famous last words:
series

Should we replicate patch #3 to QEMU0002 / FWCF in "hw/i386/acpi-build.c" too? Or is it that we couldn't care less about _CCA on x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)

Thanks
Laszlo

#8 lore raw
Quoted 43 lines · Laszlo Ersek
Quoted 32 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

Quoted 5 lines
 hw/arm/vexpress.c        | 1 +
 hw/arm/virt-acpi-build.c | 2 ++
 hw/arm/virt.c            | 2 ++
 3 files changed, 5 insertions(+)

Famous last words:
series

Quoted 7 lines

Should we replicate patch #3 to QEMU0002 / FWCF in "hw/i386/acpi-build.c" too? Or is it that we couldn't care less about _CCA on x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)

Thanks
Laszlo

ACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate, _CCA enables the OS to adapt to the differences

So I think we don't need it on x86.

Signature · 1 line
MST
#9 lore raw
Quoted 53 lines

Am 09.02.2017 um 19:13 schrieb Michael S. Tsirkin <[email protected]>:

Quoted 49 lines · Laszlo Ersek
Quoted 33 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

Quoted 5 lines
hw/arm/vexpress.c        | 1 +
hw/arm/virt-acpi-build.c | 2 ++
hw/arm/virt.c            | 2 ++
3 files changed, 5 insertions(+)

Famous last words:
series

Should we replicate patch #3 to QEMU0002 / FWCF in "hw/i386/acpi-build.c" too? Or is it that we couldn't care less about _CCA on x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)

Quoted 8 lines
Quoted 2 lines

Thanks
Laszlo

ACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate, _CCA enables the OS to adapt to the differences

So I think we don't need it on x86.

According to acpi 6.1, x86 explicitly defaults to dma coherent if _CCA is omitted. It's only illegal for ARM.

Alex

#10 lore raw
Quoted 56 lines · Alexander Graf
Quoted 56 lines

Am 09.02.2017 um 19:13 schrieb Michael S. Tsirkin <[email protected]>:

Quoted 51 lines · Laszlo Ersek
Quoted 33 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

Quoted 5 lines
hw/arm/vexpress.c        | 1 +
hw/arm/virt-acpi-build.c | 2 ++
hw/arm/virt.c            | 2 ++
3 files changed, 5 insertions(+)

Famous last words:
series

Should we replicate patch #3 to QEMU0002 / FWCF in "hw/i386/acpi-build.c" too? Or is it that we couldn't care less about _CCA on x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)

Thanks
Laszlo

Quoted 7 lines
Quoted 5 lines

ACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate, _CCA enables the OS to adapt to the differences

So I think we don't need it on x86.

According to acpi 6.1, x86 explicitly defaults to dma coherent if _CCA is omitted. It's only illegal for ARM.

Incredible; a finding that, for a change, does not create more work.

Thanks.
Laszlo

Reviewed-by on the v2 series
#11 lore raw
In reply to Alexander Graf
Quoted 26 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Quoted 6 lines

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

This looks correct to me.

#12 lore raw
In reply to Alexander Graf
Quoted 32 lines · Alexander Graf

ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.

That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.

That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.

We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.

This patch set adds the respective cache coherency flags for them in both DT and ACPI.

Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".

Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt

Quoted 4 lines
 hw/arm/vexpress.c        | 1 +
 hw/arm/virt-acpi-build.c | 2 ++
 hw/arm/virt.c            | 2 ++
 3 files changed, 5 insertions(+)

The patches in this series have more lines of Reviewed-by: tags than they do actual code changes :-)

Thanks, applied to target-arm.next.

-- PMM