target-arm: Declare virtio-mmio as dma-coherent in dt
PATCH 1/1 target-arm: Declare virtio-mmio as dma-coherent in dt
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
--- hw/arm/vexpress.c | 1 + hw/arm/virt.c | 1 + 2 files changed, 2 insertions(+)
hw/arm/vexpress.c +1 -0
@@ -452,6 +452,7 @@ static int add_virtio_mmio_node(void *fdt, uint32_t acells, uint32_t scells, acells, addr, scells, size);qemu_fdt_setprop_cells (fdt, nodename,"interrupt-parent" , intc);qemu_fdt_setprop_cells (fdt, nodename,"interrupts" ,0 , irq,1 ); +qemu_fdt_setprop (fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename);if (rc) {return -1 ;
hw/arm/virt.c +1 -0
@@ -797,6 +797,7 @@ static void create_virtio_devices(const VirtMachineState *vms, qemu_irq *pic)qemu_fdt_setprop_cells (vms->fdt, nodename,"interrupts" ,GIC_FDT_IRQ_TYPE_SPI , irq,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI ); +qemu_fdt_setprop (vms->fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename); } }
Signature · 1 line
2.10.0
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit 5d636e21c44ecf982a22a7bc4ca89186079ac283
Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100
hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when doing DMA as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM. Since this essentially means that emulated PCI devices are DMA coherent, set the 'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI description, which already marks the PCI bridge as cache coherent (see commit bc64b96c984abf).
Message-id: [email protected]
Quoted 8 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
As noted above, commit bc64b96c984a ("hw/arm/virt-acpi-build: _CCA attribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.
Thus, do we need _CCA in the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device in acpi_dsdt_add_virtio().
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding _CCA in acpi_dsdt_add_virtio().
Ard, Shannon, any comments?
Thanks,
Laszlo
Quoted 32 lines
Signed-off-by Alexander Graf <[email protected]>--- hw/arm/vexpress.c | 1 + hw/arm/virt.c | 1 + 2 files changed, 2 insertions(+)hw/arm/vexpress.c +1
-0@@ -452,6 +452,7 @@ static int add_virtio_mmio_node(void *fdt, uint32_t acells, uint32_t scells, acells, addr, scells, size);qemu_fdt_setprop_cells (fdt, nodename,"interrupt-parent" , intc);qemu_fdt_setprop_cells (fdt, nodename,"interrupts" ,0 , irq,1 ); +qemu_fdt_setprop (fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename);if (rc) {return -1 ;hw/arm/virt.c +1
-0@@ -797,6 +797,7 @@ static void create_virtio_devices(const VirtMachineState *vms, qemu_irq *pic)qemu_fdt_setprop_cells (vms->fdt, nodename,"interrupts" ,GIC_FDT_IRQ_TYPE_SPI , irq,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI ); +qemu_fdt_setprop (vms->fdt, nodename,"dma-coherent" ,NULL ,0 );g_free (nodename); } }
Quoted 34 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when doing DMA as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM. Since this essentially means that emulated PCI devices are DMA coherent, set the 'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI description, which already marks the PCI bridge as cache coherent (see commit
bc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id: [email protected]
Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 5 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
Quoted 5 lines
Quoted 2 lines
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().
Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 1 line
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().
I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Alex
Quoted 51 lines · Alexander Graf
Quoted 38 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 7 lines
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Thank you!
Laszlo
Quoted 47 lines · Laszlo Ersek
Quoted 45 lines · Alexander Graf
Quoted 43 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 4 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
The _CCA property is mandatory on arm64, and there is no default. So this is a bug nonetheless, even if the de facto default matches the platform. So yes, let's get this fixed please.
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Thank you!
Laszlo
Quoted 1 line · Ard Biesheuvel
The
_CCAproperty is mandatory on arm64, and there is no default.
Is there a tool that can check this kind of requirement and complain about issues in the ACPI tables (and ditto, device tree)? It's really easy to produce a dt or ACPI table that works with current kernels and then turns out to have a problem six or twelve months down the line :-(
thanks
-- PMM
Quoted 9 lines · Peter Maydell
Quoted 1 line · Ard Biesheuvel
The
_CCAproperty is mandatory on arm64, and there is no default.Is there a tool that can check this kind of requirement and complain about issues in the ACPI tables (and ditto, device tree)? It's really easy to produce a dt or ACPI table that works with current kernels and then turns out to have a problem six or twelve months down the line :-(
I think the "bios bits" project or the FWTS (firmware test suite) project might do some ACPI sanity checks. They could be run in guests.
https://biosbits.org/
https://wiki.ubuntu.com/FirmwareTestSuite
However, I'm unsure if they support aarch64. Also... it's not like those projects are stationary. Even the ACPI spec is a moving target (and spec-level regressions exist).
I believe Heyi Guo @ Linaro used to work with FWTS on aarch64. CC'd.
No clue about DT conformance testing. Is there an industry standard or a working group behind DT?
Thanks
Laszlo
Hi all,
Just to confirm that FWTS is supporting aarch64.
Alex is the maintainer of FWTS and may provide more information.
Regards,
Gary (Heyi Guo)
Quoted 27 lines · Laszlo Ersek
Quoted 8 lines · Peter Maydell
Quoted 1 line · Ard Biesheuvel
The
_CCAproperty is mandatory on arm64, and there is no default.Is there a tool that can check this kind of requirement and complain about issues in the ACPI tables (and ditto, device tree)? It's really easy to produce a dt or ACPI table that works with current kernels and then turns out to have a problem six or twelve months down the line :-(
I think the "bios bits" project or the FWTS (firmware test suite) project might do some ACPI sanity checks. They could be run in guests.
https://biosbits.org/
https://wiki.ubuntu.com/FirmwareTestSuiteHowever, I'm unsure if they support aarch64. Also... it's not like those projects are stationary. Even the ACPI spec is a moving target (and spec-level regressions exist).
I believe Heyi Guo @ Linaro used to work with FWTS on aarch64. CC'd.
No clue about DT conformance testing. Is there an industry standard or a working group behind DT?
Thanks
Laszlo
Hi,
FWTS supports multiple architectures:
- amd64
- arm64
- armhf
- i386
- ppc64el
- s390x
Package details are available @
https://launchpad.net/~firmware-testing-team/+archive/ubuntu/ppa-fwts-stable/+packages
FWTS also supports device tree to some extent, thanks to contributions from Jeremy Kerr & Deb McLemore.
Cheers,
Alex Hung
Quoted 46 lines · Heyi Guo
Hi all,
Just to confirm that FWTS is supporting aarch64.
Alex is the maintainer of FWTS and may provide more information.
Regards,
Gary (Heyi Guo)
Quoted 33 lines · Laszlo Ersek
Quoted 4 lines · Peter Maydell
Quoted 1 line · Ard Biesheuvel
Quoted 8 lines
Quoted 1 line
The
_CCAproperty is mandatory on arm64, and there is no default.Is there a tool that can check this kind of requirement and complain about issues in the ACPI tables (and ditto, device tree)? It's really easy to produce a dt or ACPI table that works with current kernels and then turns out to have a problem six or twelve months down the line :-(
I think the "bios bits" project or the FWTS (firmware test suite) project might do some ACPI sanity checks. They could be run in guests.
https://biosbits.org/
https://wiki.ubuntu.com/FirmwareTestSuiteHowever, I'm unsure if they support aarch64. Also... it's not like those projects are stationary. Even the ACPI spec is a moving target (and spec-level regressions exist).
I believe Heyi Guo @ Linaro used to work with FWTS on aarch64. CC'd.
No clue about DT conformance testing. Is there an industry standard or a working group behind DT?
Thanks
Laszlo
Signature · 2 lines
Cheers, Alex Hung
(+Leif, Graeme)
Quoted 10 lines · Peter Maydell
Quoted 1 line · Ard Biesheuvel
The
_CCAproperty is mandatory on arm64, and there is no default.Is there a tool that can check this kind of requirement and complain about issues in the ACPI tables (and ditto, device tree)? It's really easy to produce a dt or ACPI table that works with current kernels and then turns out to have a problem six or twelve months down the line :-(
Perhaps Graeme or Leif could shed some light on this? I know this is on our roadmap, but I am not up to date with the latest status.
Quoted 56 lines · Laszlo Ersek
Quoted 55 lines · Alexander Graf
Quoted 38 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().
Quoted 5 lines
Quoted 3 lines
I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Thank you!
Is ACPI boot supposed to work at all with 4.9?
Booting Linux on physical CPU 0x0
Linux version 4.9.6-00018-g13e39d5 (agraf@achrid) (gcc version 6.1.1 20160711 (Linaro GCC 6.1-2016.08) ) #68 SMP PREEMPT Wed Feb 8 13:25:23 CET 2017 Boot CPU: AArch64 Processor [500f0001]
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
cma: Reserved 16 MiB at 0x00000000bf000000
ACPI: Early table checksum verification disabled
fACPI: Failed to init ACPI tables
On node 0 totalpages: 524288
DMA zone: 8192 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 524288 pages, LIFO batch:31
psci: is not implemented in ACPI.
ACPI: APIC not present
fmissing boot CPU MPIDR, not enabling secondaries
percpu: Embedded 21 pages/cpu @ffff80007efdd000 s47896 r8192 d29928 u86016
pcpu-alloc: s47896 r8192 d29928 u86016 alloc=21*4096
pcpu-alloc: [0] 0
Detected PIPT I-cache on CPU0
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 516096
Kernel command line: acpi=force console=ttyAMA0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 2030472K/2097152K available (8380K kernel code, 858K rwdata, 3632K rodata, 1024K init, 280K bss, 50296K reserved, 16384K cma-reserved) Virtual kernel memory layout:
modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB)
vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB)
.text : 0xffff000008080000 - 0xffff0000088b0000 ( 8384 KB)
.rodata : 0xffff0000088b0000 - 0xffff000008c40000 ( 3648 KB)
.init : 0xffff000008c40000 - 0xffff000008d40000 ( 1024 KB)
.data : 0xffff000008d40000 - 0xffff000008e16a00 ( 859 KB)
.bss : 0xffff000008e16a00 - 0xffff000008e5cc3c ( 281 KB)
fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB)
PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB)
vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum)
0xffff7e0000000000 - 0xffff7e0002000000 ( 32 MB actual)
memory : 0xffff800000000000 - 0xffff800080000000 ( 2048 MB)SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Preemptible hierarchical RCU implementation.
Build-time adjustment of leaf fanout to 64. RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1 NR_IRQS:64 nr_irqs:64 0
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not present
Kernel panic - not syncing: No interrupt controller found. CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.6-00018-g13e39d5 #68 Hardware name: linux,dummy-virt (DT) Call trace: [<ffff000008088500>] dump_backtrace+0x0/0x1a0 [<ffff0000080886b4>] show_stack+0x14/0x20 [<ffff000008374e54>] dump_stack+0x94/0xb8 [<ffff0000081658fc>] panic+0x110/0x278 [<ffff000008c424a8>] init_IRQ+0x24/0x2c [<ffff000008c409f0>] start_kernel+0x230/0x38c [<ffff000008c401d8>] __primary_switched+0x5c/0x64 ---[ end Kernel panic - not syncing: No interrupt controller found.
linux,dummy-virt (DT)
Alex
Quoted 74 lines · Alexander Graf
Quoted 71 lines · Laszlo Ersek
Quoted 67 lines · Alexander Graf
Quoted 44 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 4 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the
DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest
drivers
should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Quoted 5 lines
Quoted 1 line
Thank you!
Is ACPI boot supposed to work at all with 4.9?
Yes, but only if you boot via UEFI
Quoted 68 lines
Booting Linux on physical CPU
0x0
Linux version 4.9.6-00018-g13e39d5 (agraf@achrid) (gcc version 6.1.1 20160711 (Linaro GCC 6.1-2016.08) ) #68 SMP PREEMPT Wed Feb 8 13:25:23 CET 2017
Boot CPU: AArch64 Processor [500f0001]
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
cma: Reserved 16 MiB at0x00000000bf000000
ACPI: Early table checksum verification disabled
fACPI: Failed to init ACPI tables
On node 0 totalpages: 524288
DMA zone: 8192 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 524288 pages, LIFO batch:31
psci: is not implemented in ACPI.
ACPI: APIC not present
fmissing boot CPU MPIDR, not enabling secondaries
percpu: Embedded 21 pages/cpu @ffff80007efdd000 s47896 r8192 d29928 u86016
pcpu-alloc: s47896 r8192 d29928 u86016 alloc=21*4096
pcpu-alloc: [0] 0
Detected PIPT I-cache on CPU0
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 516096 Kernel command line: acpi=force console=ttyAMA0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 2030472K/2097152K available (8380K kernel code, 858K rwdata, 3632K rodata, 1024K init, 280K bss, 50296K reserved, 16384K cma-reserved) Virtual kernel memory layout:modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB) vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB) .text : 0xffff000008080000 - 0xffff0000088b0000 ( 8384 KB) .rodata : 0xffff0000088b0000 - 0xffff000008c40000 ( 3648 KB) .init : 0xffff000008c40000 - 0xffff000008d40000 ( 1024 KB) .data : 0xffff000008d40000 - 0xffff000008e16a00 ( 859 KB) .bss : 0xffff000008e16a00 - 0xffff000008e5cc3c ( 281 KB) fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB) PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB) vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum) 0xffff7e0000000000 - 0xffff7e0002000000 ( 32 MB actual) memory : 0xffff800000000000 - 0xffff800080000000 ( 2048 MB)SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Preemptible hierarchical RCU implementation.
Build-time adjustment of leaf fanout to 64. RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1 NR_IRQS:64 nr_irqs:64 0
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not present
ACPI: APIC not presentKernel panic - not syncing: No interrupt controller found. CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.6-00018-g13e39d5 #68 Hardware name: linux,dummy-virt (DT) Call trace: [<ffff000008088500>] dump_backtrace+0x0/0x1a0 [<ffff0000080886b4>] show_stack+0x14/0x20 [<ffff000008374e54>] dump_stack+0x94/0xb8 [<ffff0000081658fc>] panic+0x110/0x278 [<ffff000008c424a8>] init_IRQ+0x24/0x2c [<ffff000008c409f0>] start_kernel+0x230/0x38c [<ffff000008c401d8>] __primary_switched+0x5c/0x64 ---[ end Kernel panic - not syncing: No interrupt controller found.linux,dummy-virt (DT)
Alex
Quoted 75 lines · Ard Biesheuvel
Quoted 74 lines · Alexander Graf
Quoted 71 lines · Laszlo Ersek
Quoted 67 lines · Alexander Graf
Quoted 44 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 4 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the
DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest
drivers
should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Thank you!
Is ACPI boot supposed to work at all with 4.9?
Yes, but only if you boot via UEFI
I see, thanks. I guess there's no boot protocol to fetch ACPI tables without UEFI.
Interestingly enough, omitting the _CCA property didn't actually make too much of a difference. The guest still seems to boot fine without it, so it seems to treat virtio-mmio as cache coherent. Only if I explicitly set _CCA as 0 it's broken the same as dt.
If that's the case, it's scary if we have different defaults for dt and acpi descriptions, no?
Also, we should probably explicitly warn in the kernel every time we see an ACPI device try to do DMA operations without _CCA set in its descriptor.
Alex
Quoted 59 lines · Alexander Graf
Quoted 56 lines · Laszlo Ersek
Quoted 55 lines · Alexander Graf
Quoted 38 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Thank you!
Is ACPI boot supposed to work at all with 4.9?
It does work. Although you may need to specific console=ttyAMA0. I think there was/is an issue with getting the default console out of the SPCR.
Quoted 8 lines
Booting Linux on physical CPU
0x0
Linux version 4.9.6-00018-g13e39d5 (agraf@achrid) (gcc version 6.1.1 20160711 (Linaro GCC 6.1-2016.08) ) #68 SMP PREEMPT Wed Feb 8 13:25:23 CET 2017
Boot CPU: AArch64 Processor [500f0001]
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
Hmm, I'm not sure ACPI boot should work without AAVMF.
Thanks,
drew
Quoted 56 lines · Laszlo Ersek
Quoted 55 lines · Alexander Graf
Quoted 38 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 3 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest drivers should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().
Quoted 5 lines
Quoted 3 lines
I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Thank you!
In fact, don't some other devices also suffer from this? Fw-cfg is now DMA capable IIUC, so that should also get a _CCA attribute for example.
Alex
Quoted 74 lines · Alexander Graf
Quoted 71 lines · Laszlo Ersek
Quoted 67 lines · Alexander Graf
Quoted 44 lines · Laszlo Ersek
CC'ing Ard and Shannon (I recall this property from earlier):
Quoted 4 lines · Alexander Graf
QEMU emulated hardware is always dma coherent with its guest. We do annotate that correctly on the PCI host controller, but left out virtio-mmio.
I recommend to reference the following commit here:
commit
5d636e21c44ecf982a22a7bc4ca89186079ac283Author: Ard Biesheuvel <[email protected]> Date: Mon Jul 4 13:06:36 2016 +0100hw/arm/virt: mark the PCIe host controller as DMA coherent in the
DT
Since QEMU performs cacheable accesses to guest memory when
doing DMA
as part of the implementation of emulated PCI devices, guest
drivers
should use cacheable accesses as well when running under KVM.
Since this
essentially means that emulated PCI devices are DMA coherent, set
the
'dma-coherent' DT property on the PCIe host controller DT node.
This brings the DT description into line with the ACPI
description,
which already marks the PCI bridge as cache coherent (see commitbc64b96c984abf).Signed-off-by Ard Biesheuvel <[email protected]>Message-id:
[email protected]Reviewed-by Peter Maydell <[email protected]>Signed-off-by Peter Maydell <[email protected]>Quoted 7 lines
Recent kernels have started to interpret that flag rather than take dma coherency as granted with virtio-mmio. While that is considered a kernel bug, as it breaks previously working systems, it showed that our dt description is incomplete.
This patch adds the respective marker that allows guest OSs to evaluate that our virtio-mmio devices are indeed cache coherent.
Quoted 7 lines
As noted above, commit
bc64b96c984a("hw/arm/virt-acpi-build:_CCAattribute is compulsory", 2015-11-03) had done the same in the ACPI description of the PCIe host controller.Thus, do we need
_CCAin the ACPI description of the virtio-mmio transports, to parallel the DT change? See the LNRO0005 device inacpi_dsdt_add_virtio().Yes, we should also annotate it correctly in the DSDT. Today it's not a deal breaker as Linux always assumes virtio-mmio to be dma coherent, but it would make our platform description more accurate.
Quoted 3 lines
If that's the case, then I propose that either the patch please fix both DT and ACPI, or that at least we file a bug "somewhere", for adding
_CCAinacpi_dsdt_add_virtio().I agree that it should happen in the same patch (set). While I don't care a lot about ACPI right now (since dt is preferred on upstream kernels), I can take a look.
Quoted 6 lines
Quoted 1 line
Thank you!
In fact, don't some other devices also suffer from this? Fw-cfg is now DMA capable IIUC, so that should also get a
_CCAattribute for example.
Correct. Any described DMA capable device should have a _CCA attribute