Skip to contentSkip to conversation

hw/arm/virt: Avoid unexpected warning from Linux guest on host with Fujitsu CPUs

PATCH Applied Zhenyu Zhang 7 participants 11 messages Jun 12, 2024 – Jun 21, 2024

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 hw/arm/virt: Avoid unexpected warning from Linux guest on host with Fujitsu CPUs

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

---
v3: Add comments explaining why we add 'dma-coherent' property (Peter)

---
 hw/arm/virt.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
hw/arm/virt.c +11 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
     qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
     qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);
+
     /* /chosen must exist for load_dtb to fill in necessary properties later */
     qemu_fdt_add_subnode(fdt, "/chosen");
     if (vms->dtb_randomness) {
Signature · 1 line
2.43.0
Reviewed-by on v3 1/1
#1 lore raw
Quoted 61 lines · Zhenyu Zhang

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

Quoted 7 lines

---
v3: Add comments explaining why we add 'dma-coherent' property (Peter)

---
  hw/arm/virt.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
Reviewed-by on v3 1/1
#2 lore raw
In reply to Zhenyu Zhang
Quoted 80 lines · Zhenyu Zhang

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

---
v3: Add comments explaining why we add 'dma-coherent' property (Peter)

---
  hw/arm/virt.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
hw/arm/virt.c +4 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
      qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
      qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 287

Quoted 10 lines
hw/arm/virt.c +7 -0
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);
+
      /* /chosen must exist for load_dtb to fill in necessary properties later */
      qemu_fdt_add_subnode(fdt, "/chosen");
      if (vms->dtb_randomness) {

+1 to Peter's suggested comment, otherwise, unless privy to this thread, one would wonder how/why.

Reviewed-by Jonathan Cameron on v3 1/1
#3 lore raw
In reply to Zhenyu Zhang
Quoted 58 lines · On Tue, 11 Jun 2024 22:05:06 -0400 Zhenyu Zhang <[email protected]> wrote:

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Quoted 4 lines

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

Aligns with earlier discussion so...

Quoted 28 lines

---
v3: Add comments explaining why we add 'dma-coherent' property (Peter)

---
 hw/arm/virt.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
hw/arm/virt.c +11 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
     qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
     qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);
+
     /* /chosen must exist for load_dtb to fill in necessary properties later */
     qemu_fdt_add_subnode(fdt, "/chosen");
     if (vms->dtb_randomness) {
#4 lore raw
In reply to Zhenyu Zhang

Hi Zhenyu,

Quoted 76 lines · Zhenyu Zhang

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

---
v3: Add comments explaining why we add 'dma-coherent' property (Peter)

---
  hw/arm/virt.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
hw/arm/virt.c +0 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
      qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
      qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 283

Quoted 10 lines
hw/arm/virt.c +10 -0
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

Quoted 3 lines
hw/arm/virt.c +0 -0
      /* /chosen must exist for load_dtb to fill in necessary properties later */
      qemu_fdt_add_subnode(fdt, "/chosen");
      if (vms->dtb_randomness) {
#5 lore raw
Quoted 12 lines · Philippe Mathieu-Daudé

Hi Zhenyu,

Quoted 8 lines · Zhenyu Zhang
hw/arm/virt.c +0 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
      qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
      qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
Quoted 14 lines

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 283

Quoted 10 lines
hw/arm/virt.c +10 -0
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

create_device_tree() creates an empty device tree, not one with stuff in it. It seems reasonable to me for this property on the root to be set in the same place we set other properties of the root node.

thanks
-- PMM

#6 lore raw
Quoted 28 lines · Peter Maydell
Quoted 12 lines · Philippe Mathieu-Daudé

Hi Zhenyu,

Quoted 8 lines · Zhenyu Zhang
hw/arm/virt.c +0 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
       qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
       qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
Quoted 14 lines

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 283

Quoted 10 lines
hw/arm/virt.c +10 -0
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root node
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

Quoted 4 lines

create_device_tree() creates an empty device tree, not one with stuff in it. It seems reasonable to me for this property on the root to be set in the same place we set other properties of the root node.

OK. Still the question about other virt machines remains unanswered :)

#7 lore raw
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 29 lines · Peter Maydell
Quoted 27 lines · Philippe Mathieu-Daudé

Hi Zhenyu,

Quoted 11 lines · Zhenyu Zhang
hw/arm/virt.c +2 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
        qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
        qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root 

node

Quoted 8 lines

+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA +     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about +     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

Quoted 7 lines
Quoted 4 lines

create_device_tree() creates an empty device tree, not one with stuff in it. It seems reasonable to me for this property on the root to be set in the same place we set other properties of the root node.

OK. Still the question about other virt machines remains unanswered :)

From the DT consumer point of view, the interpretation and assumptions around coherency *are* generally architecture- or platform-specific. For instance on RISC-V, many platforms want to assume coherency by default (and potentially use "dma-noncoherent" to mark individual devices that aren't), while others may still want to do the opposite and use "dma-coherent" in the same manner as Arm and AArch64. Neither property existed back in ePAPR, so typical PowerPC systems wouldn't even be looking and will just make their own assumptions by other means.

Thanks,
Robin.

#8 lore raw
Quoted 8 lines · Robin Murphy
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Peter Maydell
Quoted 3 lines · Philippe Mathieu-Daudé

Hi Zhenyu,

Hello Philippe,

Quoted 33 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines · Zhenyu Zhang
Quoted 11 lines
hw/arm/virt.c +2 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
       qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
       qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root

node

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 283

Quoted 8 lines
hw/arm/virt.c +8 -0
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

Quoted 7 lines
Quoted 4 lines

create_device_tree() creates an empty device tree, not one with stuff in it. It seems reasonable to me for this property on the root to be set in the same place we set other properties of the root node.

OK. Still the question about other virt machines remains unanswered :)

Quoted 9 lines

From the DT consumer point of view, the interpretation and assumptions
around coherency *are* generally architecture- or platform-specific. For instance on RISC-V, many platforms want to assume coherency by default (and potentially use "dma-noncoherent" to mark individual devices that aren't), while others may still want to do the opposite and use "dma-coherent" in the same manner as Arm and AArch64. Neither property existed back in ePAPR, so typical PowerPC systems wouldn't even be looking and will just make their own assumptions by other means.

As Robin's comment says, each platform wants to assume coherency by default may be different. Adding it to all virt machines may introduce new risks. Currently, the issue is only valid on Fujitsu CPUs where the cache line size is 256 bytes, meaning the combination of kvm+virt-platform is needed to trigger the warning. So I'd be inclined to add this "dma-coherent" property for the "virt" platform first and advertise the property to other platforms if we hit the issue on those platforms.

Thanks,
Zhenyu

#9 lore raw
Quoted 52 lines · Zhenyu Zhang
Quoted 8 lines · Robin Murphy
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 5 lines · Peter Maydell
Quoted 3 lines · Philippe Mathieu-Daudé

Hi Zhenyu,

Hello Philippe,

Quoted 41 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines · Zhenyu Zhang
Quoted 11 lines
hw/arm/virt.c +2 -0
index 3c93c0c0a6..3cefac6d43 100644
@@ -271,6 +271,17 @@ static void create_fdt(VirtMachineState *vms)
        qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2);
        qemu_fdt_setprop_string(fdt, "/", "model", "linux,dummy-virt");
 
+    /*
+     * For QEMU, all DMA is coherent. Advertising this in the root

node

hw/arm/virt.c static void create_fdt(VirtMachineState *vms) · line 283

Quoted 8 lines
hw/arm/virt.c +8 -0
+     * has two benefits:
+     *
+     * - It avoids potential bugs where we forget to mark a DMA
+     *   capable device as being dma-coherent
+     * - It avoids spurious warnings from the Linux kernel about
+     *   devices which can't do DMA at all
+     */
+    qemu_fdt_setprop(fdt, "/", "dma-coherent", NULL, 0);

OK, but why restrict that to the Aarch64 virt machine? Shouldn't advertise this generically in create_device_tree()? Or otherwise at least in the other virt machines?

Quoted 7 lines
Quoted 4 lines

create_device_tree() creates an empty device tree, not one with stuff in it. It seems reasonable to me for this property on the root to be set in the same place we set other properties of the root node.

OK. Still the question about other virt machines remains unanswered :)

From the DT consumer point of view, the interpretation and assumptions
around coherency *are* generally architecture- or platform-specific. For instance on RISC-V, many platforms want to assume coherency by default (and potentially use "dma-noncoherent" to mark individual devices that aren't), while others may still want to do the opposite and use "dma-coherent" in the same manner as Arm and AArch64. Neither property existed back in ePAPR, so typical PowerPC systems wouldn't even be looking and will just make their own assumptions by other means.

TIL :)

Quoted 8 lines

As Robin's comment says, each platform wants to assume coherency by default may be different. Adding it to all virt machines may introduce new risks. Currently, the issue is only valid on Fujitsu CPUs where the cache line size is 256 bytes, meaning the combination of kvm+virt-platform is needed to trigger the warning. So I'd be inclined to add this "dma-coherent" property for the "virt" platform first and advertise the property to other platforms if we hit the issue on those platforms.

Thanks Robin & Zhenyu,

Phil.

#10 lore raw
In reply to Zhenyu Zhang
Quoted 59 lines · Zhenyu Zhang

Multiple warning messages and corresponding backtraces are observed when Linux guest is booted on the host with Fujitsu CPUs. One of them is shown as below.

[    0.032443] ------------[ cut here ]------------
[    0.032446] uart-pl011 9000000.pl011: ARCH_DMA_MINALIGN smaller than
CTR_EL0.CWG (128 < 256)
[    0.032454] WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:54
arch_setup_dma_ops+0xbc/0xcc
[    0.032470] Modules linked in:
[    0.032475] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-452.el9.aarch64
[    0.032481] Hardware name: linux,dummy-virt (DT)
[    0.032484] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.032490] pc : arch_setup_dma_ops+0xbc/0xcc
[    0.032496] lr : arch_setup_dma_ops+0xbc/0xcc
[    0.032501] sp : ffff80008003b860
[    0.032503] x29: ffff80008003b860 x28: 0000000000000000 x27: ffffaae4b949049c
[    0.032510] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[    0.032517] x23: 0000000000000100 x22: 0000000000000000 x21: 0000000000000000
[    0.032523] x20: 0000000100000000 x19: ffff2f06c02ea400 x18: ffffffffffffffff
[    0.032529] x17: 00000000208a5f76 x16: 000000006589dbcb x15: ffffaae4ba071c89
[    0.032535] x14: 0000000000000000 x13: ffffaae4ba071c84 x12: 455f525443206e61
[    0.032541] x11: 68742072656c6c61 x10: 0000000000000029 x9 : ffffaae4b7d21da4
[    0.032547] x8 : 0000000000000029 x7 : 4c414e494d5f414d x6 : 0000000000000029
[    0.032553] x5 : 000000000000000f x4 : ffffaae4b9617a00 x3 : 0000000000000001
[    0.032558] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff2f06c029be40
[    0.032564] Call trace:
[    0.032566]  arch_setup_dma_ops+0xbc/0xcc
[    0.032572]  of_dma_configure_id+0x138/0x300
[    0.032591]  amba_dma_configure+0x34/0xc0
[    0.032600]  really_probe+0x78/0x3dc
[    0.032614]  __driver_probe_device+0x108/0x160
[    0.032619]  driver_probe_device+0x44/0x114
[    0.032624]  __device_attach_driver+0xb8/0x14c
[    0.032629]  bus_for_each_drv+0x88/0xe4
[    0.032634]  __device_attach+0xb0/0x1e0
[    0.032638]  device_initial_probe+0x18/0x20
[    0.032643]  bus_probe_device+0xa8/0xb0
[    0.032648]  device_add+0x4b4/0x6c0
[    0.032652]  amba_device_try_add.part.0+0x48/0x360
[    0.032657]  amba_device_add+0x104/0x144
[    0.032662]  of_amba_device_create.isra.0+0x100/0x1c4
[    0.032666]  of_platform_bus_create+0x294/0x35c
[    0.032669]  of_platform_populate+0x5c/0x150
[    0.032672]  of_platform_default_populate_init+0xd0/0xec
[    0.032697]  do_one_initcall+0x4c/0x2e0
[    0.032701]  do_initcalls+0x100/0x13c
[    0.032707]  kernel_init_freeable+0x1c8/0x21c
[    0.032712]  kernel_init+0x28/0x140
[    0.032731]  ret_from_fork+0x10/0x20
[    0.032735] ---[ end trace 0000000000000000 ]---

In Linux, a check is applied to every device which is exposed through device-tree node. The warning message is raised when the device isn't DMA coherent and the cache line size is larger than ARCH_DMA_MINALIGN (128 bytes). The cache line is sorted from CTR_EL0[CWG], which corresponds to 256 bytes on the guest CPUs. The DMA coherent capability is claimed through 'dma-coherent' in their device-tree nodes or parent nodes.

Quoted 4 lines

Fix the issue by adding 'dma-coherent' property to the device-tree root node, meaning all devices are capable of DMA coherent by default.

Applied to target-arm.next, thanks.

-- PMM