arm64: cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)

Back in 97303480753e ("arm64: Increase the max granular size"), ARCH_DMA_MINALIGN was effectively increased to 128 bytes thanks to an increase in L1_CACHE_BYTES due to an unsubstantiated performance claim on the now obsolete ThunderX-1. Although this was reverted in d93277b9839b, ARCH_DMA_MINALIGN was kept at 128 bytes by ebc7e21e0fa2 ("arm64: Increase ARCH_DMA_MINALIGN to 128").

During discussion of the original patch, it was reported that the change also prevented a warning during boot on (again, now obsolete) Qualcomm server hardware where the cache writeback granule was larger than 64 bytes. The reason for this warning was because non-coherent DMA could lead to data corruption due to unexpected writeback from the CPU where a cacheline is shared with other allocations.

Since then, systems have appeared with larger cachelines still, and so commit 8f5c9037a55b ("arm64/mm: Correct the cache line size warning with non coherent device") reworked the warning so that it only appears on systems where non-coherent DMA is actually required and taints the kernel with TAINT_CPU_OUT_OF_SPEC. We are not aware of any systems, even including the aforementioned obsolete machines, which have a CWG larger than 64 bytes and require non-coherent DMA.

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

Cc Catalin Marinas <[email protected]>

Cc Mark Rutland <[email protected]>

Cc Ard Biesheuvel <[email protected]>

Cc Arnd Bergmann <[email protected]>

Cc Vincent Whitchurch <[email protected]>

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/linux-arm-kernel/CAOZdJXUiRMAguDV+HEJqPg57MyBNqEcTyaH+ya=U93NHb-pdJA@mail.gmail.com/ open here

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Will Deacon <[email protected]>

Back in 97303480753e ("arm64: Increase the max granular size"), ARCH_DMA_MINALIGN was effectively increased to 128 bytes thanks to an increase in L1_CACHE_BYTES due to an unsubstantiated performance claim on the now obsolete ThunderX-1. Although this was reverted in d93277b9839b, ARCH_DMA_MINALIGN was kept at 128 bytes by ebc7e21e0fa2 ("arm64: Increase ARCH_DMA_MINALIGN to 128").

During discussion of the original patch, it was reported that the change also prevented a warning during boot on (again, now obsolete) Qualcomm server hardware where the cache writeback granule was larger than 64 bytes. The reason for this warning was because non-coherent DMA could lead to data corruption due to unexpected writeback from the CPU where a cacheline is shared with other allocations.

Since then, systems have appeared with larger cachelines still, and so commit 8f5c9037a55b ("arm64/mm: Correct the cache line size warning with non coherent device") reworked the warning so that it only appears on systems where non-coherent DMA is actually required and taints the kernel with TAINT_CPU_OUT_OF_SPEC. We are not aware of any systems, even including the aforementioned obsolete machines, which have a CWG larger than 64 bytes and require non-coherent DMA.

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

Cc Catalin Marinas <[email protected]>

Cc Mark Rutland <[email protected]>

Cc Ard Biesheuvel <[email protected]>

Cc Arnd Bergmann <[email protected]>

Cc Vincent Whitchurch <[email protected]>

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/linux-arm-kernel/CAOZdJXUiRMAguDV+HEJqPg57MyBNqEcTyaH+ya=U93NHb-pdJA@mail.gmail.com/ open here

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Will Deacon <[email protected]>

---
 arch/arm64/include/asm/cache.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/include/asm/cache.h +1 -1

index a074459f8f2f..a9c0716e7440 100644
@@ -47,7 +47,7 @@
  * cache before the transfer is done, causing old data to be seen by
  * the CPU.
  */
-#define ARCH_DMA_MINALIGN	(128)
+#define ARCH_DMA_MINALIGN	L1_CACHE_BYTES
 
 #ifdef CONFIG_KASAN_SW_TAGS
 #define ARCH_SLAB_MINALIGN	(1ULL << KASAN_SHADOW_SCALE_SHIFT)

Signature · 1 line

2.31.1.818.g46aad6cb9e-goog

Quoted 6 lines

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

The rationale above makes sense to me, so:

Acked-by Mark Rutland <[email protected]>

Quoted 7 lines · Mark Rutland

Quoted 6 lines

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

Quoted 3 lines

The rationale above makes sense to me, so:

Acked-by Mark Rutland <[email protected]>

I think it would make sense to go even further than this in the future, and allow
setting a smaller minimum alignment depending what hardware is detected at boot time.

Back in 97303480753e ("arm64: Increase the max granular size"), ARCH_DMA_MINALIGN was effectively increased to 128 bytes thanks to an increase in L1_CACHE_BYTES due to an unsubstantiated performance claim on the now obsolete ThunderX-1. Although this was reverted in d93277b9839b, ARCH_DMA_MINALIGN was kept at 128 bytes by ebc7e21e0fa2 ("arm64: Increase ARCH_DMA_MINALIGN to 128").

During discussion of the original patch, it was reported that the change also prevented a warning during boot on (again, now obsolete) Qualcomm server hardware where the cache writeback granule was larger than 64 bytes. The reason for this warning was because non-coherent DMA could lead to data corruption due to unexpected writeback from the CPU where a cacheline is shared with other allocations.

Since then, systems have appeared with larger cachelines still, and so commit 8f5c9037a55b ("arm64/mm: Correct the cache line size warning with non coherent device") reworked the warning so that it only appears on systems where non-coherent DMA is actually required and taints the kernel with TAINT_CPU_OUT_OF_SPEC. We are not aware of any systems, even including the aforementioned obsolete machines, which have a CWG larger than 64 bytes and require non-coherent DMA.

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

Cc Catalin Marinas <[email protected]>

Cc Mark Rutland <[email protected]>

Cc Ard Biesheuvel <[email protected]>

Cc Arnd Bergmann <[email protected]>

Cc Vincent Whitchurch <[email protected]>

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/linux-arm-kernel/CAOZdJXUiRMAguDV+HEJqPg57MyBNqEcTyaH+ya=U93NHb-pdJA@mail.gmail.com/ open here

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Will Deacon <[email protected]>

---
 arch/arm64/include/asm/cache.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/include/asm/cache.h +1 -2

index a074459f8f2f..a9c0716e7440 100644
@@ -47,7 +47,7 @@
  * cache before the transfer is done, causing old data to be seen by
  * the CPU.
  */
-#define ARCH_DMA_MINALIGN      (128)
+#define ARCH_DMA_MINALIGN      L1_CACHE_BYTES
 
 #ifdef CONFIG_KASAN_SW_TAGS
 #define ARCH_SLAB_MINALIGN     (1ULL << KASAN_SHADOW_SCALE_SHIFT)
--

2.31.1.818.g46aad6cb9e-goog

Back in 97303480753e ("arm64: Increase the max granular size"), ARCH_DMA_MINALIGN was effectively increased to 128 bytes thanks to an increase in L1_CACHE_BYTES due to an unsubstantiated performance claim on the now obsolete ThunderX-1. Although this was reverted in d93277b9839b, ARCH_DMA_MINALIGN was kept at 128 bytes by ebc7e21e0fa2 ("arm64: Increase ARCH_DMA_MINALIGN to 128").

[...]

Back in 97303480753e ("arm64: Increase the max granular size"), ARCH_DMA_MINALIGN was effectively increased to 128 bytes thanks to an increase in L1_CACHE_BYTES due to an unsubstantiated performance claim on the now obsolete ThunderX-1. Although this was reverted in d93277b9839b, ARCH_DMA_MINALIGN was kept at 128 bytes by ebc7e21e0fa2 ("arm64: Increase ARCH_DMA_MINALIGN to 128").

During discussion of the original patch, it was reported that the change also prevented a warning during boot on (again, now obsolete) Qualcomm server hardware where the cache writeback granule was larger than 64 bytes. The reason for this warning was because non-coherent DMA could lead to data corruption due to unexpected writeback from the CPU where a cacheline is shared with other allocations.

Since then, systems have appeared with larger cachelines still, and so commit 8f5c9037a55b ("arm64/mm: Correct the cache line size warning with non coherent device") reworked the warning so that it only appears on systems where non-coherent DMA is actually required and taints the kernel with TAINT_CPU_OUT_OF_SPEC. We are not aware of any systems, even including the aforementioned obsolete machines, which have a CWG larger than 64 bytes and require non-coherent DMA.

More recently, it has been reported that a ARCH_DMA_MINALIGN of 128 bytes wastes considerable memory (~6% immediately after boot on one system).

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

Cc Catalin Marinas <[email protected]>

Cc Mark Rutland <[email protected]>

Cc Ard Biesheuvel <[email protected]>

Cc Arnd Bergmann <[email protected]>

Cc Vincent Whitchurch <[email protected]>

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/linux-arm-kernel/CAOZdJXUiRMAguDV+HEJqPg57MyBNqEcTyaH+ya=U93NHb-pdJA@mail.gmail.com/ open here

Link https://lore.kernel.org/linux-arm-kernel/[email protected]/ open here

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Will Deacon <[email protected]>

---

  arch/arm64/include/asm/cache.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/include/asm/cache.h +0 -0

index a074459f8f2f..a9c0716e7440 100644

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Quoted 14 lines

  arch/arm64/include/asm/cache.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/include/asm/cache.h +1 -1

index a074459f8f2f..a9c0716e7440 100644
@@ -47,7 +47,7 @@
   * cache before the transfer is done, causing old data to be seen by
   * the CPU.
   */
-#define ARCH_DMA_MINALIGN	(128)
+#define ARCH_DMA_MINALIGN	L1_CACHE_BYTES
 

arch/arm64/include/asm/cache.h line 53

Quoted 2 lines

arch/arm64/include/asm/cache.h +0 -0

  #ifdef CONFIG_KASAN_SW_TAGS
  #define ARCH_SLAB_MINALIGN	(1ULL << KASAN_SHADOW_SCALE_SHIFT)

Best regards

Signature · 2 lines

Marek Szyprowski, PhD
Samsung R&D Institute Poland

Hi Marek,

Quoted 6 lines · Marek Szyprowski

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Ouch!

I have a 3b in a drawer that I might be able to reproduce the issue with; can you tell me how you're booting that kernel? e.g. which FW and DT you're using?

On 02.06.2021 15:51, Mark Rutland wrote:

[   13.938691] debugfs: Directory '3f902000.hdmi' with parent 'vc4-hdmi'
already present!
[   13.983716] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops)
[   13.991816] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops)
[   14.000945] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops)
[   14.009644] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops)
[   14.017547] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops)
[   14.025780] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops)
[   14.033995] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops)
[   14.042196] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops)

[....] Waiting for /dev to be fully populated...[ 14.112812] [drm]
Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0 done.

Quoted 11 lines · Marek Szyprowski

On 02.06.2021 15:51, Mark Rutland wrote:

[   13.938691] debugfs: Directory '3f902000.hdmi' with parent 'vc4-hdmi'
already present!
[   13.983716] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops)
[   13.991816] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops)
[   14.000945] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops)
[   14.009644] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops)
[   14.017547] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops)
[   14.025780] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops)
[   14.033995] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops)
[   14.042196] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops)

Quoted 3 lines

[....] Waiting for /dev to be fully populated...[ 14.112812] [drm]
Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0 done.

I wonder if the vc4's cache is involved here rather than the CPU cache. If it's that, then removing the vc4 drivers should make it boot reliably, and it might be possible to fix it by manually padding any allocations that get passed to the GPU.

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · Marek Szyprowski

On 02.06.2021 15:51, Mark Rutland wrote:

[   13.938691] debugfs: Directory '3f902000.hdmi' with parent 'vc4-hdmi'
already present!
[   13.983716] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops)
[   13.991816] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops)
[   14.000945] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops)
[   14.009644] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops)
[   14.017547] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops)
[   14.025780] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops)
[   14.033995] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops)
[   14.042196] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops)

[....] Waiting for /dev to be fully populated...[ 14.112812] [drm]
Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0 done.

Quoted 4 lines

I wonder if the vc4's cache is involved here rather than the CPU cache. If it's that, then removing the vc4 drivers should make it boot reliably, and it might be possible to fix it by manually padding any allocations that get passed to the GPU.

Indeed, after disabling DRM_VC4 in the .config, system boots fine and seems to be working correctly.

On Wed, Jun 2, 2021 at 4:28 PM Marek Szyprowski <[email protected]> wrote:
I suppose most can be easily ruled out because they are not shared, or because they are larger than 64 bytes. I'll have a quick look if I find any smoking guns.

Quoted 7 lines · Mark Rutland

Quoted 6 lines · Marek Szyprowski

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Quoted 5 lines

Ouch!

I have a 3b in a drawer that I might be able to reproduce the issue with; can you tell me how you're booting that kernel? e.g. which FW and DT you're using?

I'm booting the kernel with the mainline dtb (arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b.dtb) from the u-boot, which downloads it via TFTP. I don't remember which firmware version is there, but without raspberry specific tools (which I don't have deployed there) it is hard to check that now.

The rootfs is on SD card, the system is some older Debian release.

Here is the last part of the boot
log if it helps:

[    7.906329] Freeing unused kernel memory: 8512K
[    7.911793] Run /sbin/init as init process
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
ERROR: could not open /proc/stat: No such file or directory [....] Starting the hotplug events dispatcher: systemd-udevdstarting version 236
. ok
[....] Synthesizing the initial hotplug events...[   13.641287] bcm2835-rng 3f104000.rng: hwrng registered
[   13.896575] i2c-bcm2835 3f805000.i2c: Could not read clock-frequency property
done.
[   13.938691] debugfs: Directory '3f902000.hdmi' with parent 'vc4-hdmi' already present!
[   13.983716] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops) [   13.991816] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops) [   14.000945] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops) [   14.009644] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops) [   14.017547] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops) [   14.025780] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops) [   14.033995] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops) [   14.042196] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops) [....] Waiting for /dev to be fully populated...[   14.112812] [drm] Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0
done.
[ ok ] Activating swap...done.
[....] Checking file systems...fsck from util-linux 2.29.2
done.
[ ok ] Cleaning up temporary files... /tmp.
[ ok ] Mounting local filesystems...done.
[ ok ] Activating swapfile swap...done.
[ ok ] Cleaning up temporary files....
[ ok ] Setting kernel variables...done.
[ ok ] Configuring network interfaces...done.
[ ok ] Starting RPC port mapper daemon: rpcbind.
[ ok ] Cleaning up temporary files....
[ ok ] Setting up ALSA...done.
[ ok ] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.

Best regards

Signature · 2 lines

Marek Szyprowski, PhD
Samsung R&D Institute Poland

Quoted 10 lines · Marek Szyprowski

Quoted 9 lines · Mark Rutland

Quoted 6 lines · Marek Szyprowski

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Ouch!

Quoted 8 lines

Quoted 3 lines

I have a 3b in a drawer that I might be able to reproduce the issue with; can you tell me how you're booting that kernel? e.g. which FW and DT you're using?

I'm booting the kernel with the mainline dtb (arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b.dtb) from the u-boot, which downloads it via TFTP. I don't remember which firmware version is there, but without raspberry specific tools (which I don't have deployed there) it is hard to check that now.

Thanks, this was enough info to get started.

For comparison, I have a 3Bv1.2 board.

I grabbed the latest RPI firmware, built myself a v2021.07-rc3 rpi_3_defconfig u-boot, and got a kernel booting (off the SD card rather than over the network).

For the kernel I'm testing commit 65688d2a05de; defconfig with DRM and VC4 built-in, since passing modules around is painful in my setup.

Quoted 1 line

The rootfs is on SD card, the system is some older Debian release.

For comparison, I built myself a buildroot 2021.02.2 filesystem.

So far, booting up an running I'm not seeeing issues (and no complaints from KASAN or similar), but I don't have a good way to stress the VC4 GPU, so I might not be triggering whatever's going wrong.

Quoted 1 line

From the log below I see the last message is about X sockets -- is the

lockup happening when the display manager starts?

Hi Mark,

I've just checked with the latest firmware from
https://github.com/raspberrypi/firmware (master branch, just copied everything to /boot) and the issue is still there.

If you start from arm64/defconfig without modules, please make sure you have enabled all RPi drivers, otherwise VC4 DRM won't come up. I've managed to reproduce the issue without the modules with the following changes to arm64's defconfig:

./scripts/config -e DRM -e DRM_VC4 -e CONFIG_CLK_RASPBERRYPI -e CONFIG_SENSORS_RASPBERRYPI_HWMON -e CONFIG_I2C_BCM2835 -e CONFIG_ARM_RASPBERRYPI_CPUFREQ

Quoted 6 lines · Marek Szyprowski

Hi Mark,

I've just checked with the latest firmware from
https://github.com/raspberrypi/firmware (master branch, just copied everything to /boot) and the issue is still there.

Quoted 8 lines

If you start from arm64/defconfig without modules, please make sure you have enabled all RPi drivers, otherwise VC4 DRM won't come up. I've managed to reproduce the issue without the modules with the following changes to arm64's defconfig:

./scripts/config -e DRM -e DRM_VC4 -e CONFIG_CLK_RASPBERRYPI -e CONFIG_SENSORS_RASPBERRYPI_HWMON -e CONFIG_I2C_BCM2835 -e CONFIG_ARM_RASPBERRYPI_CPUFREQ

Thanks for this!

With that config on commit 65688d2a05deb9f0 I also see a hang at the end of boot, but before reaching userspace, with the last messages in dmesg as below.

I'll go check that the ARCH_DMA_MINALIGN affects this, then I'll go play with debug options.

Quoted 14 lines · Mark Rutland

Quoted 4 lines · Marek Szyprowski

I've just checked with the latest firmware from
https://github.com/raspberrypi/firmware (master branch, just copied everything to /boot) and the issue is still there.

Quoted 8 lines

If you start from arm64/defconfig without modules, please make sure you have enabled all RPi drivers, otherwise VC4 DRM won't come up. I've managed to reproduce the issue without the modules with the following changes to arm64's defconfig:

./scripts/config -e DRM -e DRM_VC4 -e CONFIG_CLK_RASPBERRYPI -e CONFIG_SENSORS_RASPBERRYPI_HWMON -e CONFIG_I2C_BCM2835 -e CONFIG_ARM_RASPBERRYPI_CPUFREQ

Quoted 8 lines

Thanks for this!

With that config on commit 65688d2a05deb9f0 I also see a hang at the end of boot, but before reaching userspace, with the last messages in dmesg as below.

I'll go check that the ARCH_DMA_MINALIGN affects this, then I'll go play with debug options.

I can confirm that with the ARCH_DMA_MINALIGN change reverted, the hang goes away. Running with that reverted andwith KASAN, I get the slab-out-of-bounds splat below, which occurs at the time the hang would otherwise occur, and is possibly the problem:

[    3.609515] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops)
[    3.621451] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops)
[    3.628344] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops)
[    3.635904] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops)
[    3.643351] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops)
[    3.651238] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops)
[    3.659167] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops)
[    3.666499] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops)
[    3.688560] [drm] Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0
[    3.728010] ==================================================================
[    3.728042] BUG: KASAN: slab-out-of-bounds in vc4_atomic_commit_tail+0x1cc/0x910
[    3.728123] Read of size 8 at addr ffff000007360440 by task kworker/u8:0/7
[    3.728153]
[    3.728169] CPU: 2 PID: 7 Comm: kworker/u8:0 Not tainted 5.13.0-rc3-00009-g694c523e7267 #3
[    3.728203] Hardware name: Raspberry Pi 3 Model B (DT)
[    3.728225] Workqueue: events_unbound deferred_probe_work_func
[    3.728290] Call trace:
[    3.728301]  dump_backtrace+0x0/0x2b4
[    3.728358]  show_stack+0x1c/0x30
[    3.728407]  dump_stack+0xfc/0x168
[    3.728445]  print_address_description.constprop.0+0x2c/0x2c0
[    3.728495]  kasan_report+0x1dc/0x240
[    3.728529]  __asan_load8+0x98/0xd4
[    3.728565]  vc4_atomic_commit_tail+0x1cc/0x910
[    3.728621]  commit_tail+0x100/0x210
[    3.728675]  drm_atomic_helper_commit+0x1c4/0x3dc
[    3.728730]  drm_atomic_commit+0x80/0x94
[    3.728768]  drm_client_modeset_commit_atomic+0x2f4/0x3a0
[    3.728821]  drm_client_modeset_commit_locked+0x8c/0x230
[    3.728872]  drm_fb_helper_pan_display+0x164/0x3a0
[    3.728924]  fb_pan_display+0x12c/0x1fc
[    3.728963]  bit_update_start+0x34/0xa0
[    3.729013]  fbcon_switch+0x678/0x920
[    3.729058]  redraw_screen+0x17c/0x35c
[    3.729095]  fbcon_prepare_logo+0x484/0x5bc
[    3.729143]  fbcon_init+0x77c/0x970
[    3.729187]  visual_init+0x14c/0x1e4
[    3.729239]  do_bind_con_driver.isra.0+0x2c4/0x530
[    3.729279]  do_take_over_console+0x200/0x2e0
[    3.729317]  do_fbcon_takeover+0x90/0x120
[    3.729363]  fbcon_fb_registered+0x14c/0x164
[    3.729412]  register_framebuffer+0x308/0x4e0
[    3.729451]  __drm_fb_helper_initial_config_and_unlock+0x538/0x7d0
[    3.729506]  drm_fbdev_client_hotplug+0x204/0x374
[    3.729556]  drm_fbdev_generic_setup+0xf4/0x24c
[    3.729604]  vc4_drm_bind+0x1d4/0x1f0
[    3.729654]  try_to_bring_up_master+0x254/0x2dc
[    3.729709]  __component_add+0x10c/0x240
[    3.729759]  component_add+0x18/0x24
[    3.729807]  vc4_v3d_dev_probe+0x20/0x30
[    3.729854]  platform_probe+0x90/0x110
[    3.729907]  really_probe+0x148/0x744
[    3.729952]  driver_probe_device+0x8c/0xfc
[    3.729998]  __device_attach_driver+0x120/0x180
[    3.730048]  bus_for_each_drv+0xf4/0x15c
[    3.730091]  __device_attach+0x168/0x250
[    3.730137]  device_initial_probe+0x18/0x24
[    3.730186]  bus_probe_device+0xec/0x100
[    3.730230]  deferred_probe_work_func+0xe8/0x130
[    3.730279]  process_one_work+0x3b8/0x650
[    3.730319]  worker_thread+0x3cc/0x72c
[    3.730356]  kthread+0x21c/0x224
[    3.730402]  ret_from_fork+0x10/0x38
[    3.730442]
[    3.730453] Allocated by task 7:
[    3.730470]  kasan_save_stack+0x2c/0x60
[    3.730526]  __kasan_kmalloc+0x90/0xb4
[    3.730577]  vc4_hvs_channels_duplicate_state+0x60/0x1a0
[    3.730637]  drm_atomic_get_private_obj_state+0x144/0x230
[    3.730680]  vc4_atomic_check+0x40/0x73c
[    3.730732]  drm_atomic_check_only+0x998/0xe60
[    3.730769]  drm_atomic_commit+0x34/0x94
[    3.730804]  drm_client_modeset_commit_atomic+0x2f4/0x3a0
[    3.730854]  drm_client_modeset_commit_locked+0x8c/0x230
[    3.730904]  drm_client_modeset_commit+0x38/0x60
[    3.730951]  drm_fb_helper_set_par+0x104/0x17c
[    3.730998]  fbcon_init+0x43c/0x970
[    3.731041]  visual_init+0x14c/0x1e4
[    3.731090]  do_bind_con_driver.isra.0+0x2c4/0x530
[    3.731128]  do_take_over_console+0x200/0x2e0
[    3.731165]  do_fbcon_takeover+0x90/0x120
[    3.731210]  fbcon_fb_registered+0x14c/0x164
[    3.731258]  register_framebuffer+0x308/0x4e0
[    3.731296]  __drm_fb_helper_initial_config_and_unlock+0x538/0x7d0
[    3.731349]  drm_fbdev_client_hotplug+0x204/0x374
[    3.731398]  drm_fbdev_generic_setup+0xf4/0x24c
[    3.731446]  vc4_drm_bind+0x1d4/0x1f0
[    3.731493]  try_to_bring_up_master+0x254/0x2dc
[    3.731546]  __component_add+0x10c/0x240
[    3.731594]  component_add+0x18/0x24
[    3.731642]  vc4_v3d_dev_probe+0x20/0x30
[    3.731686]  platform_probe+0x90/0x110
[    3.731737]  really_probe+0x148/0x744
[    3.731781]  driver_probe_device+0x8c/0xfc
[    3.731827]  __device_attach_driver+0x120/0x180
[    3.731875]  bus_for_each_drv+0xf4/0x15c
[    3.731916]  __device_attach+0x168/0x250
[    3.731962]  device_initial_probe+0x18/0x24
[    3.732009]  bus_probe_device+0xec/0x100
[    3.732052]  deferred_probe_work_func+0xe8/0x130
[    3.732100]  process_one_work+0x3b8/0x650
[    3.732137]  worker_thread+0x3cc/0x72c
[    3.732172]  kthread+0x21c/0x224
[    3.732215]  ret_from_fork+0x10/0x38
[    3.732253]
[    3.732262] The buggy address belongs to the object at ffff000007360400
[    3.732262]  which belongs to the cache kmalloc-128 of size 128
[    3.732293] The buggy address is located 64 bytes inside of
[    3.732293]  128-byte region [ffff000007360400, ffff000007360480)
[    3.732329] The buggy address belongs to the page:
[    3.732344] page:(____ptrval____) refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7360
[    3.732380] flags: 0x3fffc0000000200(slab|node=0|zone=0|lastcpupid=0xffff)
[    3.732442] raw: 03fffc0000000200 dead000000000100 dead000000000122 ffff000004c02300
[    3.732478] raw: 0000000000000000 0000000000100010 00000001ffffffff 0000000000000000
[    3.732501] page dumped because: kasan: bad access detected
[    3.732518]
[    3.732527] Memory state around the buggy address:
[    3.732549]  ffff000007360300: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[    3.732579]  ffff000007360380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[    3.732608] >ffff000007360400: 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc fc
[    3.732629]                                            ^
[    3.732652]  ffff000007360480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[    3.732682]  ffff000007360500: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[    3.732703] ==================================================================
[    3.732718] Disabling lock debugging due to kernel taint
[    3.769129] Console: switching to colour frame buffer device 90x30
[    5.148699] vc4-drm soc:gpu: [drm] fb0: vc4drmfb frame buffer device

[Adding VC4 folks -- please see the KASAN splat below!]

Background here is that reducing ARCH_DMA_MINALIGN to 64 on arm64 (queued in -next) is causing vc4 to hang on Rpi3b due to a probable driver bug.

Quoted 5 lines · Mark Rutland

Quoted 1 line · Marek Szyprowski

[    3.728042] BUG: KASAN: slab-out-of-bounds in vc4_atomic_commit_tail+0x1cc/0x910
[    3.728123] Read of size 8 at addr ffff000007360440 by task kworker/u8:0/7

Quoted 3 lines

[    3.728495]  kasan_report+0x1dc/0x240
[    3.728529]  __asan_load8+0x98/0xd4
[    3.728565]  vc4_atomic_commit_tail+0x1cc/0x910

Quoted 5 lines · Will Deacon

[Adding VC4 folks -- please see the KASAN splat below!]

Background here is that reducing ARCH_DMA_MINALIGN to 64 on arm64 (queued in -next) is causing vc4 to hang on Rpi3b due to a probable driver bug.

The great news for the patch that caused it is that this has nothing to do with DMA alignment.

Quoted 6 lines · Mark Rutland

Quoted 5 lines · Mark Rutland

Quoted 1 line · Marek Szyprowski

[    3.728042] BUG: KASAN: slab-out-of-bounds in vc4_atomic_commit_tail+0x1cc/0x910
[    3.728123] Read of size 8 at addr ffff000007360440 by task kworker/u8:0/7

This is offset 0x40 into struct vc4_hvs_state, which is the 'pending_commit' pointer
for the array index 4, i.e. one after the end of the structure.

Quoted 3 lines

Quoted 3 lines

[    3.728495]  kasan_report+0x1dc/0x240
[    3.728529]  __asan_load8+0x98/0xd4
[    3.728565]  vc4_atomic_commit_tail+0x1cc/0x910

It seems to be this loop:

        for_each_old_crtc_in_state(state, crtc, old_crtc_state, i) {
                struct vc4_crtc_state *vc4_crtc_state =
                        to_vc4_crtc_state(old_crtc_state);
                unsigned int channel = vc4_crtc_state->assigned_channel;
                int ret;

                if (channel == VC4_HVS_CHANNEL_DISABLED)
                        continue;

                if (!old_hvs_state->fifo_state[channel].in_use)
                        continue;

                ret =
drm_crtc_commit_wait(old_hvs_state->fifo_state[i].pending_commit);
                if (ret)
                        drm_err(dev, "Timed out waiting for commit\n");
        }

I notice that it checks index 'fifos_state[channel].in_use', but then uses a different index 'i' for looking at the 'pending_commit' field beyond the end of the array.

This code was introduced by Maxime Ripard in commit 9ec03d7f1ed3
("drm/vc4: kms: Wait on previous FIFO users before a commit").

Quoted 8 lines · Arnd Bergmann

Quoted 3 lines · Will Deacon

Quoted 2 lines · Mark Rutland

Quoted 1 line · Mark Rutland

On Mon, Jun 07, 2021 at 11:58:32AM +0200, Marek Szyprowski wrote:

I notice that it checks index 'fifos_state[channel].in_use', but then uses a different index 'i' for looking at the 'pending_commit' field beyond the end of the array.

Quoted 2 lines

This code was introduced by Maxime Ripard in commit 9ec03d7f1ed3
("drm/vc4: kms: Wait on previous FIFO users before a commit").

Awesome, I tried to find out that bug a few weeks ago but couldn't reproduce the KASAN spat. You're right, it should be channel here instead of i. Since you did the whole work, do you want to send the patch?

Quoted 9 lines · Maxime Ripard

Quoted 8 lines · Arnd Bergmann

Quoted 3 lines · Will Deacon

Quoted 2 lines · Mark Rutland

Quoted 1 line · Mark Rutland

On Mon, Jun 07, 2021 at 11:58:32AM +0200, Marek Szyprowski wrote:

I notice that it checks index 'fifos_state[channel].in_use', but then uses a different index 'i' for looking at the 'pending_commit' field beyond the end of the array.

Quoted 7 lines

Quoted 2 lines

This code was introduced by Maxime Ripard in commit 9ec03d7f1ed3
("drm/vc4: kms: Wait on previous FIFO users before a commit").

Awesome, I tried to find out that bug a few weeks ago but couldn't reproduce the KASAN spat. You're right, it should be channel here instead of i. Since you did the whole work, do you want to send the patch?

Marek and Mark did most of the work finding the problem, I just looked in the right place a few times (and a bit in the wrong place). I'd suggest you send that patch with the corresponding Reported-by/Analyzed-by/ Tested-by tags.

Quoted 5 lines · Will Deacon

[Adding VC4 folks -- please see the KASAN splat below!]

Background here is that reducing ARCH_DMA_MINALIGN to 64 on arm64 (queued in -next) is causing vc4 to hang on Rpi3b due to a probable driver bug.

The great news for the patch that caused it is that this has nothing to do with DMA alignment.

Quoted 6 lines · Mark Rutland

Quoted 5 lines · Mark Rutland

Quoted 1 line · Marek Szyprowski

[    3.728042] BUG: KASAN: slab-out-of-bounds in vc4_atomic_commit_tail+0x1cc/0x910
[    3.728123] Read of size 8 at addr ffff000007360440 by task kworker/u8:0/7

This is offset 0x40 into struct vc4_hvs_state, which is the 'pending_commit' pointer
for the array index 4, i.e. one after the end of the structure.

Quoted 3 lines

Quoted 3 lines

[    3.728495]  kasan_report+0x1dc/0x240
[    3.728529]  __asan_load8+0x98/0xd4
[    3.728565]  vc4_atomic_commit_tail+0x1cc/0x910

It seems to be this loop:

        for_each_old_crtc_in_state(state, crtc, old_crtc_state, i) {
                struct vc4_crtc_state *vc4_crtc_state =
                        to_vc4_crtc_state(old_crtc_state);
                unsigned int channel = vc4_crtc_state->assigned_channel;
                int ret;

                if (channel == VC4_HVS_CHANNEL_DISABLED)
                        continue;

                if (!old_hvs_state->fifo_state[channel].in_use)
                        continue;

                ret =
drm_crtc_commit_wait(old_hvs_state->fifo_state[i].pending_commit);
                if (ret)
                        drm_err(dev, "Timed out waiting for commit\n");
        }

I notice that it checks index 'fifos_state[channel].in_use', but then uses a different index 'i' for looking at the 'pending_commit' field beyond the end of the array.

This code was introduced by Maxime Ripard in commit 9ec03d7f1ed3
("drm/vc4: kms: Wait on previous FIFO users before a commit").

Arnd

[Adding VC4 folks -- please see the KASAN splat below!]

Background here is that reducing ARCH_DMA_MINALIGN to 64 on arm64 (queued in -next) is causing vc4 to hang on Rpi3b due to a probable driver bug.

Will

Quoted 41 lines · Mark Rutland

Quoted 14 lines · Mark Rutland

Quoted 4 lines · Marek Szyprowski

I've just checked with the latest firmware from
https://github.com/raspberrypi/firmware (master branch, just copied everything to /boot) and the issue is still there.

Quoted 8 lines

If you start from arm64/defconfig without modules, please make sure you have enabled all RPi drivers, otherwise VC4 DRM won't come up. I've managed to reproduce the issue without the modules with the following changes to arm64's defconfig:

./scripts/config -e DRM -e DRM_VC4 -e CONFIG_CLK_RASPBERRYPI -e CONFIG_SENSORS_RASPBERRYPI_HWMON -e CONFIG_I2C_BCM2835 -e CONFIG_ARM_RASPBERRYPI_CPUFREQ

Quoted 8 lines

Thanks for this!

With that config on commit 65688d2a05deb9f0 I also see a hang at the end of boot, but before reaching userspace, with the last messages in dmesg as below.

I'll go check that the ARCH_DMA_MINALIGN affects this, then I'll go play with debug options.

I can confirm that with the ARCH_DMA_MINALIGN change reverted, the hang goes away. Running with that reverted andwith KASAN, I get the slab-out-of-bounds splat below, which occurs at the time the hang would otherwise occur, and is possibly the problem:

[    3.609515] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops)
[    3.621451] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops)
[    3.628344] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops)
[    3.635904] vc4-drm soc:gpu: bound 3f004000.txp (ops vc4_txp_ops)
[    3.643351] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops)
[    3.651238] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops)
[    3.659167] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops)
[    3.666499] vc4-drm soc:gpu: bound 3fc00000.v3d (ops vc4_v3d_ops)
[    3.688560] [drm] Initialized vc4 0.0.0 20140616 for soc:gpu on minor 0
[    3.728010] ==================================================================
[    3.728042] BUG: KASAN: slab-out-of-bounds in vc4_atomic_commit_tail+0x1cc/0x910
[    3.728123] Read of size 8 at addr ffff000007360440 by task kworker/u8:0/7

Quoted 107 lines

[    3.728153]
[    3.728169] CPU: 2 PID: 7 Comm: kworker/u8:0 Not tainted 5.13.0-rc3-00009-g694c523e7267 #3
[    3.728203] Hardware name: Raspberry Pi 3 Model B (DT)
[    3.728225] Workqueue: events_unbound deferred_probe_work_func
[    3.728290] Call trace:
[    3.728301]  dump_backtrace+0x0/0x2b4
[    3.728358]  show_stack+0x1c/0x30
[    3.728407]  dump_stack+0xfc/0x168
[    3.728445]  print_address_description.constprop.0+0x2c/0x2c0
[    3.728495]  kasan_report+0x1dc/0x240
[    3.728529]  __asan_load8+0x98/0xd4
[    3.728565]  vc4_atomic_commit_tail+0x1cc/0x910
[    3.728621]  commit_tail+0x100/0x210
[    3.728675]  drm_atomic_helper_commit+0x1c4/0x3dc
[    3.728730]  drm_atomic_commit+0x80/0x94
[    3.728768]  drm_client_modeset_commit_atomic+0x2f4/0x3a0
[    3.728821]  drm_client_modeset_commit_locked+0x8c/0x230
[    3.728872]  drm_fb_helper_pan_display+0x164/0x3a0
[    3.728924]  fb_pan_display+0x12c/0x1fc
[    3.728963]  bit_update_start+0x34/0xa0
[    3.729013]  fbcon_switch+0x678/0x920
[    3.729058]  redraw_screen+0x17c/0x35c
[    3.729095]  fbcon_prepare_logo+0x484/0x5bc
[    3.729143]  fbcon_init+0x77c/0x970
[    3.729187]  visual_init+0x14c/0x1e4
[    3.729239]  do_bind_con_driver.isra.0+0x2c4/0x530
[    3.729279]  do_take_over_console+0x200/0x2e0
[    3.729317]  do_fbcon_takeover+0x90/0x120
[    3.729363]  fbcon_fb_registered+0x14c/0x164
[    3.729412]  register_framebuffer+0x308/0x4e0
[    3.729451]  __drm_fb_helper_initial_config_and_unlock+0x538/0x7d0
[    3.729506]  drm_fbdev_client_hotplug+0x204/0x374
[    3.729556]  drm_fbdev_generic_setup+0xf4/0x24c
[    3.729604]  vc4_drm_bind+0x1d4/0x1f0
[    3.729654]  try_to_bring_up_master+0x254/0x2dc
[    3.729709]  __component_add+0x10c/0x240
[    3.729759]  component_add+0x18/0x24
[    3.729807]  vc4_v3d_dev_probe+0x20/0x30
[    3.729854]  platform_probe+0x90/0x110
[    3.729907]  really_probe+0x148/0x744
[    3.729952]  driver_probe_device+0x8c/0xfc
[    3.729998]  __device_attach_driver+0x120/0x180
[    3.730048]  bus_for_each_drv+0xf4/0x15c
[    3.730091]  __device_attach+0x168/0x250
[    3.730137]  device_initial_probe+0x18/0x24
[    3.730186]  bus_probe_device+0xec/0x100
[    3.730230]  deferred_probe_work_func+0xe8/0x130
[    3.730279]  process_one_work+0x3b8/0x650
[    3.730319]  worker_thread+0x3cc/0x72c
[    3.730356]  kthread+0x21c/0x224
[    3.730402]  ret_from_fork+0x10/0x38
[    3.730442]
[    3.730453] Allocated by task 7:
[    3.730470]  kasan_save_stack+0x2c/0x60
[    3.730526]  __kasan_kmalloc+0x90/0xb4
[    3.730577]  vc4_hvs_channels_duplicate_state+0x60/0x1a0
[    3.730637]  drm_atomic_get_private_obj_state+0x144/0x230
[    3.730680]  vc4_atomic_check+0x40/0x73c
[    3.730732]  drm_atomic_check_only+0x998/0xe60
[    3.730769]  drm_atomic_commit+0x34/0x94
[    3.730804]  drm_client_modeset_commit_atomic+0x2f4/0x3a0
[    3.730854]  drm_client_modeset_commit_locked+0x8c/0x230
[    3.730904]  drm_client_modeset_commit+0x38/0x60
[    3.730951]  drm_fb_helper_set_par+0x104/0x17c
[    3.730998]  fbcon_init+0x43c/0x970
[    3.731041]  visual_init+0x14c/0x1e4
[    3.731090]  do_bind_con_driver.isra.0+0x2c4/0x530
[    3.731128]  do_take_over_console+0x200/0x2e0
[    3.731165]  do_fbcon_takeover+0x90/0x120
[    3.731210]  fbcon_fb_registered+0x14c/0x164
[    3.731258]  register_framebuffer+0x308/0x4e0
[    3.731296]  __drm_fb_helper_initial_config_and_unlock+0x538/0x7d0
[    3.731349]  drm_fbdev_client_hotplug+0x204/0x374
[    3.731398]  drm_fbdev_generic_setup+0xf4/0x24c
[    3.731446]  vc4_drm_bind+0x1d4/0x1f0
[    3.731493]  try_to_bring_up_master+0x254/0x2dc
[    3.731546]  __component_add+0x10c/0x240
[    3.731594]  component_add+0x18/0x24
[    3.731642]  vc4_v3d_dev_probe+0x20/0x30
[    3.731686]  platform_probe+0x90/0x110
[    3.731737]  really_probe+0x148/0x744
[    3.731781]  driver_probe_device+0x8c/0xfc
[    3.731827]  __device_attach_driver+0x120/0x180
[    3.731875]  bus_for_each_drv+0xf4/0x15c
[    3.731916]  __device_attach+0x168/0x250
[    3.731962]  device_initial_probe+0x18/0x24
[    3.732009]  bus_probe_device+0xec/0x100
[    3.732052]  deferred_probe_work_func+0xe8/0x130
[    3.732100]  process_one_work+0x3b8/0x650
[    3.732137]  worker_thread+0x3cc/0x72c
[    3.732172]  kthread+0x21c/0x224
[    3.732215]  ret_from_fork+0x10/0x38
[    3.732253]
[    3.732262] The buggy address belongs to the object at ffff000007360400
[    3.732262]  which belongs to the cache kmalloc-128 of size 128
[    3.732293] The buggy address is located 64 bytes inside of
[    3.732293]  128-byte region [ffff000007360400, ffff000007360480)
[    3.732329] The buggy address belongs to the page:
[    3.732344] page:(____ptrval____) refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7360
[    3.732380] flags: 0x3fffc0000000200(slab|node=0|zone=0|lastcpupid=0xffff)
[    3.732442] raw: 03fffc0000000200 dead000000000100 dead000000000122 ffff000004c02300
[    3.732478] raw: 0000000000000000 0000000000100010 00000001ffffffff 0000000000000000
[    3.732501] page dumped because: kasan: bad access detected
[    3.732518]
[    3.732527] Memory state around the buggy address:
[    3.732549]  ffff000007360300: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[    3.732579]  ffff000007360380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc

Quoted 8 lines

[    3.732608] >ffff000007360400: 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc fc
[    3.732629]                                            ^
[    3.732652]  ffff000007360480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[    3.732682]  ffff000007360500: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[    3.732703] ==================================================================
[    3.732718] Disabling lock debugging due to kernel taint
[    3.769129] Console: switching to colour frame buffer device 90x30
[    5.148699] vc4-drm soc:gpu: [drm] fb0: vc4drmfb frame buffer device

Quoted 6 lines · Marek Szyprowski

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Ouch!

I have a 3b in a drawer that I might be able to reproduce the issue with; can you tell me how you're booting that kernel? e.g. which FW and DT you're using?

Is your filesystem on the SD card, or some USB storage? I'm guessing we'll need to stress DMA over one of those.

Quoted 13 lines · Mark Rutland

Quoted 6 lines · Marek Szyprowski

On 27.05.2021 14:43, Will Deacon wrote:
This patch landed in todays linux-next as commit 65688d2a05de ("arm64:
cache: Lower ARCH_DMA_MINALIGN to 64 (L1_CACHE_BYTES)"). It causes an issue on Raspberry Pi 3b board. System boots to userspace fine, but then it hangs somewhere during the init scripts after loading the modules. I didn't manage to track where it hangs yet though.

Ouch!

I have a 3b in a drawer that I might be able to reproduce the issue with; can you tell me how you're booting that kernel? e.g. which FW and DT you're using?

Quoted 2 lines

Is your filesystem on the SD card, or some USB storage? I'm guessing we'll need to stress DMA over one of those.

It could be something other than DMA in this case, possibly an out-of-bounds access into a dynamically allocated structure that happens to work when there is extra data at the end of it. Running that kernel with KASAN should help rule this out.

Running the same kernel on a Pi 4, or on a virtual machine could also help figure out if there is something hardware specific.

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

In-Reply-To: <[email protected]>

Quoted 2 lines

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

The warning is being triggered on Qualcomm MSM8996, as well as the out-of-spec taint:

------------[ cut here ]------------
rtc-pm8xxx 400f000.qcom,spmi:pmic@0:rtc@6000: ARCH_DMA_MINALIGN smaller than CTR_EL0.CWG (64 < 128)
WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:45 arch_setup_dma_ops+0xf8/0x10c

This warning is triggered with nearly every driver probe, not only rtc-pm8xxx.

Quoted 5 lines · Yassine Oudjana

In-Reply-To: <[email protected]>

Quoted 2 lines

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

The warning is being triggered on Qualcomm MSM8996, as well as the out-of-spec taint:

Is this booting with ACPI or DT?

Quoted 3 lines

------------[ cut here ]------------
rtc-pm8xxx 400f000.qcom,spmi:pmic@0:rtc@6000: ARCH_DMA_MINALIGN smaller than CTR_EL0.CWG (64 < 128)
WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:45 arch_setup_dma_ops+0xf8/0x10c

[...]

Quoted 1 line

This warning is triggered with nearly every driver probe, not only rtc-pm8xxx.

I have a suspicion none of the reported devices actually do any DMA, so in practice it should be safe but we need to figure out why arch_setup_dma_ops() gets called.

Quoted 12 lines · Catalin Marinas

Quoted 5 lines · Yassine Oudjana

In-Reply-To: <[email protected]>

Quoted 2 lines

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

The warning is being triggered on Qualcomm MSM8996, as well as the out-of-spec taint:

Is this booting with ACPI or DT?

Quoted 3 lines

------------[ cut here ]------------
rtc-pm8xxx 400f000.qcom,spmi:pmic@0:rtc@6000: ARCH_DMA_MINALIGN smaller than CTR_EL0.CWG (64 < 128)
WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:45 arch_setup_dma_ops+0xf8/0x10c

Quoted 6 lines

[...]

Quoted 1 line

This warning is triggered with nearly every driver probe, not only rtc-pm8xxx.

I have a suspicion none of the reported devices actually do any DMA, so in practice it should be safe but we need to figure out why arch_setup_dma_ops() gets called.

It gets called because there's no straightforward way to know that a platform device *isn't* DMA-capable, so we have to assume they are.

I would also assume that in a Qcom SoC there really are at least some things doing non-coherent DMA :(

Quoted 23 lines · Robin Murphy

Quoted 12 lines · Catalin Marinas

Quoted 5 lines · Yassine Oudjana

In-Reply-To: <[email protected]>

Quoted 2 lines

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the warning/taint to indicate if there are machines that unknowingly rely on this.

The warning is being triggered on Qualcomm MSM8996, as well as the out-of-spec taint:

Is this booting with ACPI or DT?

Quoted 3 lines

------------[ cut here ]------------
rtc-pm8xxx 400f000.qcom,spmi:pmic@0:rtc@6000: ARCH_DMA_MINALIGN smaller than CTR_EL0.CWG (64 < 128)
WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:45 arch_setup_dma_ops+0xf8/0x10c

Quoted 6 lines

[...]

Quoted 1 line

This warning is triggered with nearly every driver probe, not only rtc-pm8xxx.

I have a suspicion none of the reported devices actually do any DMA, so in practice it should be safe but we need to figure out why arch_setup_dma_ops() gets called.

It gets called because there's no straightforward way to know that a platform device *isn't* DMA-capable, so we have to assume they are.

Quoted 2 lines

I would also assume that in a Qcom SoC there really are at least some things doing non-coherent DMA :(

Agreed, unless this is a CPU erratum and the line size is being reported for a cache beyond the PoC, then I think we're going to have to revert the patch reducing ARCH_DMA_MINALIGN after all.

I can't find much information about the original Kryo core at all...

Quoted 37 lines · Will Deacon

Quoted 23 lines · Robin Murphy

Quoted 3 lines · Catalin Marinas

Quoted 2 lines · Yassine Oudjana

In-Reply-To: <[email protected]>

Quoted 1 line

Reduce ARCH_DMA_MINALIGN to 64 bytes and allow the

warning/taint to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

indicate if there are machines that unknowingly rely on this.

The warning is being triggered on Qualcomm MSM8996, as well as

the out-of-spec taint:

Quoted 5 lines

Is this booting with ACPI or DT?

Quoted 2 lines

------------[ cut here ]------------
rtc-pm8xxx 400f000.qcom,spmi:pmic@0:rtc@6000: ARCH_DMA_MINALIGN 

smaller than CTR_EL0.CWG (64 < 128)

Quoted 1 line

Quoted 1 line

WARNING: CPU: 0 PID: 1 at arch/arm64/mm/dma-mapping.c:45 

arch_setup_dma_ops+0xf8/0x10c

Quoted 2 lines

[...]

Quoted 1 line

This warning is triggered with nearly every driver probe, not

only rtc-pm8xxx.

Quoted 2 lines

I have a suspicion none of the reported devices actually do any

DMA, so

Quoted 8 lines

Quoted 2 lines

in practice it should be safe but we need to figure out why arch_setup_dma_ops() gets called.

It gets called because there's no straightforward way to know that a platform device *isn't* DMA-capable, so we have to assume they are.

I would also assume that in a Qcom SoC there really are at least some things doing non-coherent DMA :(

Agreed, unless this is a CPU erratum and the line size is being reported for a cache beyond the PoC, then I think we're going to have to revert the patch reducing ARCH_DMA_MINALIGN after all.

Quoted 1 line

I can't find much information about the original Kryo core at all...

I have similar issues with my QDF2400. The UART, RTC and DMA controllers are all screaming at me. I'm confident that the UART doesn't do any DMA (it is handled by the SBSA driver), but the DMA controllers are probably doing what it says on the tin.

Do we know whether Falkor and Kryo share any part of their design?

Quoted 3 lines · Will Deacon

On Tue, Jul 06, 2021 at 02:29:07PM +0100, Robin Murphy wrote:

I can't find much information about the original Kryo core at all...

I have similar issues with my QDF2400. The UART, RTC and DMA controllers are all screaming at me. I'm confident that the UART doesn't do any DMA (it is handled by the SBSA driver), but the DMA controllers are probably doing what it says on the tin.

Do we know whether Falkor and Kryo share any part of their design?

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Will Deacon

On Tue, Jul 06, 2021 at 02:29:07PM +0100, Robin Murphy wrote:

I can't find much information about the original Kryo core at all...

Quoted 4 lines

I have similar issues with my QDF2400. The UART, RTC and DMA controllers are all screaming at me. I'm confident that the UART doesn't do any DMA (it is handled by the SBSA driver), but the DMA controllers are probably doing what it says on the tin.

But that's a server chip, surely the DMA controller is fully cache coherent as required by SBSA? (please?)

Maybe just a misannotation on the device node?

Quoted 1 line

Do we know whether Falkor and Kryo share any part of their design?

I'm fairly sure the Snapdragon 821 / msm8996 is not cache coherent.

I can only speculate on how much got reused between the two, but as Falkor was released only after they had already given up on the full-custom Kryo core, it's plausible that it incorporates bits from that one. In particular the cache controller is probably easy to reuse even if the rest of it was a new design.

Quoted 5 lines · Arnd Bergmann

I can only speculate on how much got reused between the two, but as Falkor was released only after they had already given up on the full-custom Kryo core, it's plausible that it incorporates bits from that one. In particular the cache controller is probably easy to reuse even if the rest of it was a new design.

I guess we'll never find out, and I'm probably one of the few still having some access to this HW (not even sure for how long anyway).

I won't cry if we decide to pull the plug on it.

Quoted 11 lines · On Tue, Jul 6, 2021 at 4:46 PM Marc Zyngier [email protected] wrote:

Quoted 5 lines · On Tue, 06 Jul 2021 15:30:34 +0100, Arnd Bergmann [email protected] wrote:

I can only speculate on how much got reused between the two, but as Falkor was released only after they had already given up on the full-custom Kryo core, it's plausible that it incorporates bits from that one. In particular the cache controller is probably easy to reuse even if the rest of it was a new design.

I guess we'll never find out, and I'm probably one of the few still having some access to this HW (not even sure for how long anyway).

I won't cry if we decide to pull the plug on it.

Sure, but the Snapdragon 820E is one we do need to worry about. While the internet pretty much agrees on Falkor having 128 bytes L1 cache line, it might be good to rule out that Kryo just misreports it before we revert the patch.

Yassine, could you run the 'line' and 'cache' helper from lmbench to determine what the cache topology appears to be and if that matches the CTR_EL0 contents?

Something like

numactl -C 0 line -M 1M
numactl -C 3 line -M 1M
numactl -C 0 cache
numactl -C 3 cache

(the numactl command helps run this both on the 'big' and 'little' cores without running into migration)

Arnd

Quoted 4 lines

(the numactl command helps run this both on the 'big' and 'little' cores without running into migration)

Arnd

Here are the results:

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128
$ numactl -C 0 cache
L1 cache: 512 bytes 1.37 nanoseconds 64 linesize -1.00 parallelism

L2 cache: 24576 bytes 2.75 nanoseconds 64 linesize 5.06 parallelism
L3 cache: 131072 bytes 7.89 nanoseconds 64 linesize 3.85 parallelism
L4 cache: 524288 bytes 15.86 nanoseconds 128 linesize 3.48 parallelism
Memory latency: 145.93 nanoseconds 4.88 parallelism
$ numactl -C 3 cache
L1 cache: 24576 bytes 1.29 nanoseconds 64 linesize 5.00 parallelism
L2 cache: 1048576 bytes 8.60 nanoseconds 128 linesize 3.07 parallelism
Memory latency: 143.29 nanoseconds 5.37 parallelism

Quoted 6 lines · Yassine Oudjana

Quoted 4 lines

(the numactl command helps run this both on the 'big' and 'little' cores without running into migration)

Arnd

Here are the results:

Thanks, that was quick

Quoted 6 lines

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128
$ numactl -C 0 cache
L1 cache: 512 bytes 1.37 nanoseconds 64 linesize -1.00 parallelism

Quoted 8 lines

L2 cache: 24576 bytes 2.75 nanoseconds 64 linesize 5.06 parallelism
L3 cache: 131072 bytes 7.89 nanoseconds 64 linesize 3.85 parallelism
L4 cache: 524288 bytes 15.86 nanoseconds 128 linesize 3.48 parallelism
Memory latency: 145.93 nanoseconds 4.88 parallelism
$ numactl -C 3 cache
L1 cache: 24576 bytes 1.29 nanoseconds 64 linesize 5.00 parallelism
L2 cache: 1048576 bytes 8.60 nanoseconds 128 linesize 3.07 parallelism
Memory latency: 143.29 nanoseconds 5.37 parallelism

This is still somewhat inconclusive, but it does give some hope. The data that I found on random web sites was

- 32KB L1, 2MB/1MB L2 [1][2]
- 16KB L1, 1.5MB L2 [3]
- 32KB L1, 1MB/512KB L2 [4]

so none of the sizes really line up. My best guess is that the actual hierarchy

1MB per-core L2 cache on the two big CPU, 512KB per-core L2 cache on the two little ones, but no shared L2 or L3. The older Krait had a 4KB L0 cache, which could explain the 512-byte L1 output.

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

Quoted 38 lines · Arnd Bergmann

Quoted 6 lines · Yassine Oudjana

Quoted 4 lines

(the numactl command helps run this both on the 'big' and 'little' cores without running into migration)

Arnd

Here are the results:

Thanks, that was quick

Quoted 6 lines

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128
$ numactl -C 0 cache
L1 cache: 512 bytes 1.37 nanoseconds 64 linesize -1.00 parallelism

Quoted 8 lines

L2 cache: 24576 bytes 2.75 nanoseconds 64 linesize 5.06 parallelism
L3 cache: 131072 bytes 7.89 nanoseconds 64 linesize 3.85 parallelism
L4 cache: 524288 bytes 15.86 nanoseconds 128 linesize 3.48 parallelism
Memory latency: 145.93 nanoseconds 4.88 parallelism
$ numactl -C 3 cache
L1 cache: 24576 bytes 1.29 nanoseconds 64 linesize 5.00 parallelism
L2 cache: 1048576 bytes 8.60 nanoseconds 128 linesize 3.07 parallelism
Memory latency: 143.29 nanoseconds 5.37 parallelism

This is still somewhat inconclusive, but it does give some hope. The data that I found on random web sites was

- 32KB L1, 2MB/1MB L2 [1][2]
- 16KB L1, 1.5MB L2 [3]
- 32KB L1, 1MB/512KB L2 [4]

so none of the sizes really line up. My best guess is that the actual hierarchy

1MB per-core L2 cache on the two big CPU, 512KB per-core L2 cache on the two little ones, but no shared L2 or L3. The older Krait had a 4KB L0 cache, which could explain the 512-byte L1 output.

Quoted 2 lines

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

I can confirm that MSM8996, and a few derivatives, has 128 byte cache lines.

Quoted 1 line · On Tue, Jul 6, 2021 at 7:15 PM Yassine Oudjana [email protected] wrote:

...

This is still somewhat inconclusive, but it does give some hope. The data that

I found on random web sites was

(quoted diff) +0 -3

-   32KB L1, 2MB/1MB L2 [1][2]
-   16KB L1, 1.5MB L2 [3]
-   32KB L1, 1MB/512KB L2 [4]
 
    so none of the sizes really line up. My best guess is that the actual hierarchy
    1MB per-core L2 cache on the two big CPU, 512KB per-core L2 cache on
    the two little ones, but no shared L2 or L3. The older Krait had a 4KB L0
    cache, which could explain the 512-byte L1 output.

I think the million dollar question is whether the 128-byte cache-lines live in a cache above the PoC or not. If it's just a system level cache through which all DMA is "coherent", then it doesn't matter.

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Tue, Jul 6, 2021 at 7:15 PM Yassine Oudjana [email protected] wrote:

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128

Quoted 2 lines

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

$ numactl -C 0 line -M 128K
64
$ numactl -C 3 line -M 128K
64

I can confirm that MSM8996, and a few derivatives, has 128 byte cache lines.

Quoted 4 lines · Will Deacon

I think the million dollar question is whether the 128-byte cache-lines live in a cache above the PoC or not. If it's just a system level cache through which all DMA is "coherent", then it doesn't matter.

Quoted 9 lines · Yassine Oudjana

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Tue, Jul 6, 2021 at 7:15 PM Yassine Oudjana [email protected] wrote:

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128

Quoted 7 lines

Quoted 2 lines

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

$ numactl -C 0 line -M 128K
64
$ numactl -C 3 line -M 128K
64

Ok, so this seems to confirm that the L1 uses 64 byte lines, while the L2 (or possibly L3) uses 128 byte lines.

Quoted 2 lines · Bjorn Andersson

I can confirm that MSM8996, and a few derivatives, has 128 byte cache lines.

Ok, thanks. Assuming this is an outer cache and the L1 indeed has a smaller line size, can you also confirm that this 128 byte lines are north of the point of coherency?

In other words, does the CTR_EL0.DminLine field also show 128 bytes (in which case
it seems we already lost)? And if not, does a CPU store to the second half of a 128 byte L2 line cause DMA data in the first half to be clobbered?

Quoted 29 lines · Arnd Bergmann

Quoted 4 lines · Will Deacon

I think the million dollar question is whether the 128-byte cache-lines live in a cache above the PoC or not. If it's just a system level cache through which all DMA is "coherent", then it doesn't matter.

Quoted 9 lines · Yassine Oudjana

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Tue, Jul 6, 2021 at 7:15 PM Yassine Oudjana [email protected] wrote:

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128

Quoted 7 lines

Quoted 2 lines

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

$ numactl -C 0 line -M 128K
64
$ numactl -C 3 line -M 128K
64

Ok, so this seems to confirm that the L1 uses 64 byte lines, while the L2 (or possibly L3) uses 128 byte lines.

Quoted 8 lines

Quoted 2 lines · Bjorn Andersson

I can confirm that MSM8996, and a few derivatives, has 128 byte cache lines.

Ok, thanks. Assuming this is an outer cache and the L1 indeed has a smaller line size, can you also confirm that this 128 byte lines are north of the point of coherency?

Finding this old documentation has been painful :)

L0 I 64 byte cacheline
L1 I 64
L1 D 64
L2 unified 128 (shared between the CPUs of a duplex)

I believe L2 is within the POC, but I'm trying to dig up the old documentation to confirm.

Quoted 4 lines

In other words, does the CTR_EL0.DminLine field also show 128 bytes (in which case
it seems we already lost)? And if not, does a CPU store to the second half of a 128 byte L2 line cause DMA data in the first half to be clobbered?

Per the documentation I'm seeing, CTR_EL0.DminLine should show 128 bytes. I don't have hardware handy to confirm.

Quoted 47 lines · Jeffrey Hugo

Quoted 29 lines · Arnd Bergmann

Quoted 4 lines · Will Deacon

I think the million dollar question is whether the 128-byte cache-lines live in a cache above the PoC or not. If it's just a system level cache through which all DMA is "coherent", then it doesn't matter.

Quoted 9 lines · Yassine Oudjana

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Tue, Jul 6, 2021 at 7:15 PM Yassine Oudjana [email protected] wrote:

$ numactl -C 0 line -M 1M
128
$ numactl -C 3 line -M 1M
128

Quoted 7 lines

Quoted 2 lines

Can you rerun the the 'line' test with '-M 128K' to see if that confirms the 64 byte L1 line size that the 'cache' test reported?

$ numactl -C 0 line -M 128K
64
$ numactl -C 3 line -M 128K
64

Ok, so this seems to confirm that the L1 uses 64 byte lines, while the L2 (or possibly L3) uses 128 byte lines.

Quoted 8 lines

Quoted 2 lines · Bjorn Andersson

I can confirm that MSM8996, and a few derivatives, has 128 byte cache lines.

Ok, thanks. Assuming this is an outer cache and the L1 indeed has a smaller line size, can you also confirm that this 128 byte lines are north of the point of coherency?

Finding this old documentation has been painful :)

L0 I 64 byte cacheline
L1 I 64
L1 D 64
L2 unified 128 (shared between the CPUs of a duplex)

Quoted 2 lines

I believe L2 is within the POC, but I'm trying to dig up the old documentation to confirm.

Was able to track down a friendly hardware designer. The POC lies between L2 and L3. Hope this helps.

Quoted 6 lines · Jeffrey Hugo

Quoted 5 lines · Jeffrey Hugo

L0 I 64 byte cacheline
L1 I 64
L1 D 64
L2 unified 128 (shared between the CPUs of a duplex)

Quoted 5 lines

Quoted 2 lines

I believe L2 is within the POC, but I'm trying to dig up the old documentation to confirm.

Was able to track down a friendly hardware designer. The POC lies between L2 and L3. Hope this helps.

Damn, yes, it's bad news but thanks for chasing it up. I'll revert the patch at -rc1 and add a comment about MSM8996.

Quoted 10 lines · Marc Zyngier

Quoted 3 lines · Will Deacon

On Tue, Jul 06, 2021 at 02:29:07PM +0100, Robin Murphy wrote:

I can't find much information about the original Kryo core at all...

I have similar issues with my QDF2400. The UART, RTC and DMA controllers are all screaming at me. I'm confident that the UART doesn't do any DMA (it is handled by the SBSA driver), but the DMA controllers are probably doing what it says on the tin.

But that's a server chip, surely the DMA controller is fully cache coherent as required by SBSA? (please?)

Maybe just a misannotation on the device node?

Quoted 1 line

Do we know whether Falkor and Kryo share any part of their design?

I'm fairly sure the Snapdragon 821 / msm8996 is not cache coherent.

I can only speculate on how much got reused between the two, but as Falkor was released only after they had already given up on the full-custom Kryo core, it's plausible that it incorporates bits from that one. In particular the cache controller is probably easy to reuse even if the rest of it was a new design.