arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency
PATCH 1/1 arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.
Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):
could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0
or corruption of the admin or I/O completion queues (malformed completion):
could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965
presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
--- arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10 -0
@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 { <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <6>; num-lanes = <1>;
@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 { <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <5>; num-lanes = <2>;
@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 { <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <4>; num-lanes = <4>;
@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 { <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <3>; num-lanes = <2>;
@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 { <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <2>; num-lanes = <4>;
Signature · 1 line
2.37.4
Quoted 34 lines · Johan Hovold
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
Note that this is much more likely to happen since commit
c44094eee32f("arm64: dma: Drop cache invalidation fromarch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):
could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0or corruption of the admin or I/O completion queues (malformed completion):
could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
Quoted 3 lines
Fixes 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")Signed-off-by Johan Hovold <[email protected]>---
Looks like 8450 should also be like this, good catch!
Konrad
Quoted 52 lines
arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10
-0@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 { <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <6>; num-lanes = <1>;@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 { <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <5>; num-lanes = <2>;@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 { <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <4>; num-lanes = <4>;@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 { <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <3>; num-lanes = <2>;@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 { <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <2>; num-lanes = <4>;
Quoted 8 lines · Johan Hovold
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.
Quoted 17 lines
Note that this is much more likely to happen since commit
c44094eee32f("arm64: dma: Drop cache invalidation fromarch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):
could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0or corruption of the admin or I/O completion queues (malformed completion):
Quoted 9 lines
could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.
Quoted 2 lines
Fixes 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")Signed-off-by Johan Hovold <[email protected]>
Anyway, this is a really good find!
Thanks,
Mani
Quoted 56 lines
--- arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/boot/dts/qcom/sc8280xp.dtsi +10
-0@@ -854,6 +854,8 @@ pcie4: pcie@1c00000 { <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <6>; num-lanes = <1>;@@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 { <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <5>; num-lanes = <2>;@@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 { <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <4>; num-lanes = <4>;@@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 { <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <3>; num-lanes = <2>;@@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 { <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <2>; num-lanes = <4>;Signature · 1 line
2.37.4
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · Johan Hovold
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
Quoted 3 lines
Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.
I mentioned the above as an example, but clearly it can affect also the other direction (e.g. as described below).
Quoted 27 lines
Quoted 17 lines
Note that this is much more likely to happen since commit
c44094eee32f("arm64: dma: Drop cache invalidation fromarch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):
could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0or corruption of the admin or I/O completion queues (malformed completion):
Quoted 9 lines
could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
Quoted 3 lines
I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.
I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).
Johan
Quoted 49 lines · Johan Hovold
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · Johan Hovold
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
Quoted 3 lines
Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss.
I mentioned the above as an example, but clearly it can affect also the other direction (e.g. as described below).
Quoted 27 lines
Quoted 17 lines
Note that this is much more likely to happen since commit
c44094eee32f("arm64: dma: Drop cache invalidation fromarch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping.Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion):
could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0or corruption of the admin or I/O completion queues (malformed completion):
Quoted 9 lines
could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
Quoted 3 lines
I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute.
Quoted 7 lines
I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).
Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
The "go away" part sounded like a claim to me and hence I added the statement. But no worries :)
Thanks,
Mani
Quoted 1 line
Johan
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 8 lines · Manivannan Sadhasivam
Quoted 6 lines · Johan Hovold
Quoted 1 line · Manivannan Sadhasivam
On Thu, Nov 24, 2022 at 03:25:01PM +0100, Johan Hovold wrote:
I never claimed it would fix the problem, I explicitly wrote that it made it less likely to occur (to the point where my reproducer no longer triggers).
Quoted 5 lines
Quoted 2 lines
Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away.
The "go away" part sounded like a claim to me and hence I added the statement. But no worries :)
Hopefully it's clear enough if you also read the preceding sentence (with emphasis added):
presumably as these queues are small enough to not be allocated using CMA which in turn make them *more likely to be cached* (e.g. due to accesses to nearby pages through the cacheable linear map).
Johan
Quoted 9 lines · Johan Hovold
The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption.
A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices.
[...]
Applied, thanks!
[1/1] arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency
commit: 0922df8f52b88d5c718d0cfe10794ac44b95ac78
Best regards,
Signature · 1 line
Bjorn Andersson <[email protected]>