target-arm: Add some omitted dma cache coherency flags
v2 cover target-arm: Add some omitted dma cache coherency flags
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (5d636e21 "hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build: _CCA attribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.
This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)
Signature · 1 line
2.10.0
PATCH v2 1/4 target-arm: Declare virtio-mmio as dma-coherent in dt
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
--- hw/arm/vexpress.c | 1 + hw/arm/virt.c | 1 + 2 files changed, 2 insertions(+)
hw/arm/vexpress.c +1 -0
@@ -452,6 +452,7 @@ static int add_virtio_mmio_node(void *fdt, uint32_t acells, uint32_t scells, acells, addr, scells, size);qemu_fdt_setprop_cells (fdt, nodename,"interrupt-parent" , intc);qemu_fdt_setprop_cells (fdt, nodename,"interrupts" ,0 , irq,1 ); +qemu_fdt_setprop (fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename);if (rc) {return -1 ;
hw/arm/virt.c +1 -0
@@ -797,6 +797,7 @@ static void create_virtio_devices(const VirtMachineState *vms, qemu_irq *pic)qemu_fdt_setprop_cells (vms->fdt, nodename,"interrupts" ,GIC_FDT_IRQ_TYPE_SPI , irq,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI ); +qemu_fdt_setprop (vms->fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename); } }
Signature · 1 line
2.10.0
PATCH v2 2/4 hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI
In reply to Alexander GrafVirtio-mmio devices can directly access guest memory and do so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.
--- hw/arm/virt-acpi-build.c | 1 + 1 file changed, 1 insertion(+)
hw/arm/virt-acpi-build.c +1 -0
@@ -135,6 +135,7 @@ static void acpi_dsdt_add_virtio(Aml *scope,Aml *dev =aml_device ("VR%02u" , i);aml_append (dev,aml_name_decl ("_HID" ,aml_string ("LNRO0005" )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (i))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 )));Aml *crs =aml_resource_template ();aml_append (crs,aml_memory32_fixed (base, size,AML_READ_WRITE ));
Signature · 1 line
2.10.0
Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/4] hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI
Quoted 8 lines · Alexander Graf
Virtio-mmio devices can directly access guest memory and do so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.
Signed-off-by Alexander Graf <[email protected]>--- hw/arm/virt-acpi-build.c | 1 + 1 file changed, 1 insertion(+)
Quoted 13 lines
hw/arm/virt-acpi-build.c +1
-0@@ -135,6 +135,7 @@ static void acpi_dsdt_add_virtio(Aml *scope,Aml *dev =aml_device ("VR%02u" , i);aml_append (dev,aml_name_decl ("_HID" ,aml_string ("LNRO0005" )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (i))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 )));Aml *crs =aml_resource_template ();aml_append (crs,aml_memory32_fixed (base, size,AML_READ_WRITE ));
Signature · 1 line
Shannon
PATCH v2 3/4 hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI
In reply to Alexander GrafFw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.
--- hw/arm/virt-acpi-build.c | 1 + 1 file changed, 1 insertion(+)
hw/arm/virt-acpi-build.c +1 -0
@@ -90,6 +90,7 @@ static void acpi_dsdt_add_fw_cfg(Aml *scope, const MemMapEntry *fw_cfg_memmap)aml_append (dev,aml_name_decl ("_HID" ,aml_string ("QEMU0002" )));/* device present, functioning, decoding, not shown in UI */ aml_append (dev,aml_name_decl ("_STA" ,aml_int (0xB ))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 )));Aml *crs =aml_resource_template (); aml_append(crs, aml_memory32_fixed(fw_cfg_memmap->base,
Signature · 1 line
2.10.0
Re: [Qemu-arm] [Qemu-devel] [PATCH v2 3/4] hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI
Quoted 8 lines · Alexander Graf
Fw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using ACPI.
Signed-off-by Alexander Graf <[email protected]>--- hw/arm/virt-acpi-build.c | 1 + 1 file changed, 1 insertion(+)
Quoted 13 lines
hw/arm/virt-acpi-build.c +1
-0@@ -90,6 +90,7 @@ static void acpi_dsdt_add_fw_cfg(Aml *scope, const MemMapEntry *fw_cfg_memmap)aml_append (dev,aml_name_decl ("_HID" ,aml_string ("QEMU0002" )));/* device present, functioning, decoding, not shown in UI */ aml_append (dev,aml_name_decl ("_STA" ,aml_int (0xB ))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 )));Aml *crs =aml_resource_template (); aml_append(crs, aml_memory32_fixed(fw_cfg_memmap->base,
Signature · 1 line
Shannon
PATCH v2 4/4 hw/arm/virt: Declare fwcfg as dma cache coherent in dt
In reply to Alexander GrafFw-cfg recently learned how to directly access guest memory and does so in cache coherent fashion. Tell the guest about that fact when it's using DT.
--- hw/arm/virt.c | 1 + 1 file changed, 1 insertion(+)
hw/arm/virt.c +1 -0
@@ -929,6 +929,7 @@ static FWCfgState *create_fw_cfg(const VirtMachineState *vms, AddressSpace *as) "compatible", "qemu,fw-cfg-mmio");qemu_fdt_setprop_sized_cells (vms->fdt, nodename,"reg" ,2 , base,2 , size); +qemu_fdt_setprop (vms->fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename);return fw_cfg; }
Signature · 1 line
2.10.0
Quoted 32 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt
Quoted 5 lines
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)
Famous last words:
series
Should we replicate patch #3 to QEMU0002 / FWCF in "hw/i386/acpi-build.c" too? Or is it that we couldn't care less about _CCA on x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)
Thanks
Laszlo
Quoted 43 lines · Laszlo Ersek
Quoted 32 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dtQuoted 5 lines
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)Famous last words:
seriesReviewed-by Laszlo Ersek <[email protected]>
Quoted 7 lines
Should we replicate patch #3 to QEMU0002 / FWCF in "
hw/i386/acpi-build.c" too? Or is it that we couldn't care less about_CCAon x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)Thanks
Laszlo
ACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate, _CCA enables the OS to adapt to the differences
So I think we don't need it on x86.
Signature · 1 line
MST
Quoted 53 lines
Am 09.02.2017 um 19:13 schrieb Michael S. Tsirkin <[email protected]>:
Quoted 49 lines · Laszlo Ersek
Quoted 33 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dtQuoted 5 lines
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)Famous last words:
seriesReviewed-by Laszlo Ersek <[email protected]>Should we replicate patch #3 to QEMU0002 / FWCF in "
hw/i386/acpi-build.c" too? Or is it that we couldn't care less about_CCAon x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)
Quoted 8 lines
Quoted 2 lines
Thanks
LaszloACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate,_CCAenables the OS to adapt to the differencesSo I think we don't need it on x86.
According to acpi 6.1, x86 explicitly defaults to dma coherent if _CCA is omitted. It's only illegal for ARM.
Alex
Quoted 56 lines · Alexander Graf
Quoted 56 lines
Am 09.02.2017 um 19:13 schrieb Michael S. Tsirkin <[email protected]>:
Quoted 51 lines · Laszlo Ersek
Quoted 33 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dtQuoted 5 lines
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)Famous last words:
seriesReviewed-by Laszlo Ersek <[email protected]>Should we replicate patch #3 to QEMU0002 / FWCF in "
hw/i386/acpi-build.c" too? Or is it that we couldn't care less about_CCAon x86? :) (Can't really muster the energy right now to look it up in the ACPI spec, sorry!)Thanks
Laszlo
Quoted 7 lines
Quoted 5 lines
ACPI spec says:
On platforms for which existing default cache-coherency behavior of the OS is not adequate,_CCAenables the OS to adapt to the differencesSo I think we don't need it on x86.
According to acpi 6.1, x86 explicitly defaults to dma coherent if
_CCAis omitted. It's only illegal for ARM.
Incredible; a finding that, for a change, does not create more work.
Thanks.
Laszlo
Quoted 26 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Quoted 6 lines
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt
This looks correct to me.
Peter Maydell
AppliedQuoted 32 lines · Alexander Graf
ARM is amazing when it comes to cache coherency and VMs. While any sane architecture allows the host to override the guest's caching attributes, that's very hard to do on ARM.
That means that the guest may directly access guest memory bypassing the cache while QEMU happily writes to / reads from cache. The end result is very nasty, because both sides see very different views of the world.
That means that we need to be very cautious to tell guests that devices that QEMU emulates are going to use data in the cache rather than directly on memory.
We added this to PCI a while back for DT (
5d636e21"hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT") and ACPI (bc64b96 "hw/arm/virt-
acpi-build:_CCAattribute is compulsory") but never updated virtio-mmio or fw-cfg in DT or ACPI tables.This patch set adds the respective cache coherency flags for them in both DT and ACPI.
Fortunately, no guests except for Linux 4.9.7 and 4.9.8 are broken because of this. Upstream realized quickly enough that every user of virtio-mmio out there describes its cache coherency incorrectly and reverted the patch that would require said dma coherency flag. But we should be safe for the future and "do the right thing".
Alexander Graf (4):
target-arm: Declare virtio-mmio as dma-coherent in dt hw/arm/virt: Declare virtio-mmio as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in ACPI hw/arm/virt: Declare fwcfg as dma cache coherent in dt
Quoted 4 lines
hw/arm/vexpress.c | 1 + hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ 3 files changed, 5 insertions(+)
The patches in this series have more lines of Reviewed-by: tags than they do actual code changes :-)
Thanks, applied to target-arm.next.
-- PMM