Skip to contentSkip to conversation

arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency

PATCH Applied Johan Hovold 5 participants 7 messages Nov 24, 2022 – Dec 2, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.

Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):

	could not locate request for tag 0x0
	nvme nvme0: invalid id 0 completed on queue 0

or corruption of the admin or I/O completion queues (malformed completion):

	could not locate request for tag 0x45f
	nvme nvme0: invalid id 25695 completed on queue 25965

presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

Fixes 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")
---
 arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10 -0
index 27f5c2f82338..7748cd29276d 100644
@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 {
 				 <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <6>;
 			num-lanes = <1>;
 
@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 {
 				 <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <5>;
 			num-lanes = <2>;
 
@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 {
 				 <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <4>;
 			num-lanes = <4>;
 
@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 {
 				 <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <3>;
 			num-lanes = <2>;
 
@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 {
 				 <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <2>;
 			num-lanes = <4>;
 
Signature · 1 line
2.37.4
Reviewed-by on v1 1/1
#1 lore raw
Quoted 34 lines · Johan Hovold

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.

Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):

	could not locate request for tag 0x0
	nvme nvme0: invalid id 0 completed on queue 0

or corruption of the admin or I/O completion queues (malformed completion):

	could not locate request for tag 0x45f
	nvme nvme0: invalid id 25695 completed on queue 25965

presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

Quoted 3 lines
Fixes 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")

---

Looks like 8450 should also be like this, good catch!

Konrad

Quoted 52 lines
 arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10 -0
index 27f5c2f82338..7748cd29276d 100644
@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 {
 				 <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <6>;
 			num-lanes = <1>;
 
@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 {
 				 <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <5>;
 			num-lanes = <2>;
 
@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 {
 				 <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <4>;
 			num-lanes = <4>;
 
@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 {
 				 <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <3>;
 			num-lanes = <2>;
 
@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 {
 				 <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <2>;
 			num-lanes = <4>;
 
Reviewed-by on v1 1/1
#2 lore raw
In reply to Johan Hovold
Quoted 8 lines · Johan Hovold

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.

Quoted 17 lines

Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.

Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):

	could not locate request for tag 0x0
	nvme nvme0: invalid id 0 completed on queue 0

or corruption of the admin or I/O completion queues (malformed completion):

Quoted 9 lines
	could not locate request for tag 0x45f
	nvme nvme0: invalid id 25695 completed on queue 25965

presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.

Quoted 2 lines
Fixes 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")

Anyway, this is a really good find!

Thanks,
Mani

Quoted 56 lines
---
 arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10 -0
index 27f5c2f82338..7748cd29276d 100644
@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 {
 				 <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <6>;
 			num-lanes = <1>;
 
@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 {
 				 <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <5>;
 			num-lanes = <2>;
 
@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 {
 				 <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <4>;
 			num-lanes = <4>;
 
@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 {
 				 <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <3>;
 			num-lanes = <2>;
 
@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 {
 				 <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>;
 			bus-range = <0x00 0xff>;
 
+			dma-coherent;
+
 			linux,pci-domain = <2>;
 			num-lanes = <4>;
 
Signature · 1 line
2.37.4
Signature · 1 line
மணிவண்ணன் சதாசிவம்
#3 lore raw
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · Johan Hovold

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

Quoted 3 lines

Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.

I mentioned the above as an example, but clearly it can affect also the other direction (e.g. as described below).

Quoted 27 lines
Quoted 17 lines

Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.

Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):

	could not locate request for tag 0x0
	nvme nvme0: invalid id 0 completed on queue 0

or corruption of the admin or I/O completion queues (malformed completion):

Quoted 9 lines
	could not locate request for tag 0x45f
	nvme nvme0: invalid id 25695 completed on queue 25965

presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

Quoted 3 lines

I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.

I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).

Johan

#4 lore raw
Quoted 49 lines · Johan Hovold
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · Johan Hovold

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

Quoted 3 lines

Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.

I mentioned the above as an example, but clearly it can affect also the other direction (e.g. as described below).

Quoted 27 lines
Quoted 17 lines

Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.

Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):

	could not locate request for tag 0x0
	nvme nvme0: invalid id 0 completed on queue 0

or corruption of the admin or I/O completion queues (malformed completion):

Quoted 9 lines
	could not locate request for tag 0x45f
	nvme nvme0: invalid id 25695 completed on queue 25965

presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

Quoted 3 lines

I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.

Quoted 7 lines

I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).

Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

The "go away" part sounded like a claim to me and hence I added the statement. But no worries :)

Thanks,
Mani

Quoted 1 line

Johan

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#5 lore raw
Quoted 8 lines · Manivannan Sadhasivam
Quoted 6 lines · Johan Hovold
Quoted 1 line · Manivannan Sadhasivam

On Thu, Nov 24, 2022 at 03:25:01PM +0100, Johan Hovold wrote:

I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).

Quoted 5 lines
Quoted 2 lines

Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.

The "go away" part sounded like a claim to me and hence I added the statement. But no worries :)

Hopefully it's clear enough if you also read the preceding sentence (with emphasis added):

presumably as these queues are small enough to not be allocated using CMA which in turn make them *more likely to be cached* (e.g. due to accesses to nearby pages through the cacheable linear map).

Johan

maintainer Applied
#6 lore raw
In reply to Johan Hovold
Quoted 9 lines · Johan Hovold

The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.

A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.

[...]

Applied, thanks!

[1/1] arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency

      commit: 0922df8f52b88d5c718d0cfe10794ac44b95ac78

Best regards,

Signature · 1 line
Bjorn Andersson <[email protected]>