arm64: errata: work around NVIDIA Olympus device store/load ordering
Subsystems (2)
PATCH v5 1/1 arm64: errata: work around NVIDIA Olympus device store/load ordering
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB or full DMB between the store and the load.
- Specific microarchitectural and timing conditions occur.
Insert a DMB OSH immediately before each raw MMIO load on affected CPUs. As a full barrier, DMB OSH orders the older Device store before the younger Device load and prevents the erroneous observation.
Add the barrier directly to the __raw_read*() helpers, independently of the existing device-load-acquire alternative. On affected CPUs this adds one DMB OSH per raw MMIO load, including each load used by memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.
---
Changes since v4:
- Reworked the workaround following Will Deacon's review: leave the raw MMIO write helpers unchanged and insert a DMB OSH before raw MMIO loads.
- Use DMB OSH after hardware confirmation that it fixes T410-OLY-1027.
- Dropped the separate memcpy_fromio() optimization patch because the benchmark showed no noticeable benefit over the per-load workaround.
- Updated the cpucap, Kconfig help text, and commit messages for the load-side workaround.
Changes since v3:
- Split the workaround into two patches: the erratum fix (1/2) and the
arm64 memset_io()/memcpy_toio() block writers (2/2).
- Reworked the raw MMIO write helpers to use a direct base-register str*/stlr* alternative sequence instead of a per-write static branch.
- Covered the write-combining __iowrite{32,64}_copy() path by patching dgh() to dmb osh on affected CPUs, keeping the contiguous STR groups and the ordering barrier outside the copy loop; the single-element case now uses a plain str* as well.
- Added arm64 memset_io()/memcpy_toio() so the byte/word block writers take one trailing dmb osh instead of a per-store store-release.
- Updated the commit messages to describe the offset-addressing trade-off.
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata
Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.
--- Documentation/arch/arm64/silicon-errata.rst | 2 ++ arch/arm64/Kconfig | 22 +++++++++++++++++++++ arch/arm64/include/asm/io.h | 16 ++++++++++++---- arch/arm64/kernel/cpu_errata.c | 8 ++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 45 insertions(+), 4 deletions(-)
Documentation/arch/arm64/silicon-errata.rst +2 -0
@@ -298,6 +298,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | Olympus core | T410-OLY-1027 | NVIDIA_OLYMPUS_1027_ERRATUM | ++----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Olympus core | T410-OLY-1029 | ARM64_ERRATUM_4118414 | +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A |
arch/arm64/Kconfig +22 -0
@@ -1363,6 +1363,28 @@ config NVIDIA_CARMEL_CNP_ERRATUM If unsure, say Y. +config NVIDIA_OLYMPUS_1027_ERRATUM + bool "NVIDIA Olympus: device store/load ordering erratum" + default y + help + This option adds an alternative code sequence to work around an + NVIDIA Olympus core erratum where a Device-nGnR* store can be + observed by a peripheral after a younger Device-nGnR* load to the + same peripheral. This breaks the program order that drivers rely + on for MMIO and can leave a device in an incorrect state. + + The workaround inserts a DMB OSH immediately before raw MMIO loads. + The erratum cannot occur when a DMB that orders loads appears + between the store and load, preventing the younger load from being + observed before the older store. + + The alternatives framework patches in DMB OSH only when an affected + CPU is detected. Other CPUs execute a NOP in its place. Disabling + this option leaves the original MMIO read instruction stream + unchanged. + + If unsure, say Y. + config ROCKCHIP_ERRATUM_3568002 bool "Rockchip 3568002: GIC600 can not access physical addresses higher than 4GB" default y
arch/arm64/include/asm/io.h +12 -4
@@ -54,7 +54,9 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr) static __always_inline u8 __raw_readb(const volatile void __iomem *addr) {u8 val; - asm volatile(ALTERNATIVE("ldrb %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldrb %w0, [%1]", "ldarb %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));
@@ -66,7 +68,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr) {u16 val; - asm volatile(ALTERNATIVE("ldrh %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldrh %w0, [%1]", "ldarh %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));
@@ -77,7 +81,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr) static __always_inline u32 __raw_readl(const volatile void __iomem *addr) {u32 val; - asm volatile(ALTERNATIVE("ldr %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldr %w0, [%1]", "ldar %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));
@@ -88,7 +94,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr) static __always_inline u64 __raw_readq(const volatile void __iomem *addr) {u64 val; - asm volatile(ALTERNATIVE("ldr %0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldr %0, [%1]", "ldar %0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));
arch/arm64/kernel/cpu_errata.c +8 -0
@@ -839,6 +839,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = { ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL), },#endif +#ifdef CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM + { + /* NVIDIA Olympus core */ + .desc = "NVIDIA Olympus device store/load ordering erratum", + .capability = ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027, + ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS), + }, +#endif #ifdef CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE { /*
arch/arm64/tools/cpucaps +1 -0
@@ -121,6 +121,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM WORKAROUND_CLEAN_CACHE WORKAROUND_DEVICE_LOAD_ACQUIRE WORKAROUND_NVIDIA_CARMEL_CNP +WORKAROUND_NVIDIA_OLYMPUS_1027 WORKAROUND_PMUV3_IMPDEF_TRAPS WORKAROUND_QCOM_FALKOR_E1003 WORKAROUND_QCOM_ORYON_CNTVOFF
Signature · 1 line
2.54.0.windows.1
Quoted 191 lines · Shanker Donthineni
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB or full DMB between the store and the load.
- Specific microarchitectural and timing conditions occur.Insert a DMB OSH immediately before each raw MMIO load on affected CPUs. As a full barrier, DMB OSH orders the older Device store before the younger Device load and prevents the erroneous observation.
Add the barrier directly to the __raw_read*() helpers, independently of the existing device-load-acquire alternative. On affected CPUs this adds one DMB OSH per raw MMIO load, including each load used by
memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.Co-developed-by Vikram Sethi <[email protected]>Signed-off-by Vikram Sethi <[email protected]>Signed-off-by Shanker Donthineni <[email protected]>---
Changes since v4:
- Reworked the workaround following Will Deacon's review: leave the raw MMIO write helpers unchanged and insert a DMB OSH before raw MMIO loads.
- Use DMB OSH after hardware confirmation that it fixes T410-OLY-1027.
- Dropped the separatememcpy_fromio()optimization patch because the benchmark showed no noticeable benefit over the per-load workaround.
- Updated the cpucap,Kconfighelp text, and commit messages for the load-side workaround.Changes since v3:
- Split the workaround into two patches: the erratum fix (1/2) and thearm64 memset_io()/memcpy_toio() block writers (2/2).- Reworked the raw MMIO write helpers to use a direct base-register str*/stlr* alternative sequence instead of a per-write static branch.
- Covered the write-combining __iowrite{32,64}_copy() path by patchingdgh()to dmb osh on affected CPUs, keeping the contiguous STR groups and the ordering barrier outside the copy loop; the single-element case now uses a plain str* as well.
- Added arm64 memset_io()/memcpy_toio() so the byte/word block writers take one trailing dmb osh instead of a per-store store-release.
- Updated the commit messages to describe the offset-addressing trade-off.Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errataChanges since v1:
- Updated the commit message based on feedback from Vladimir Murzin.--- Documentation/arch/arm64/silicon-errata.rst | 2 ++ arch/arm64/Kconfig | 22 +++++++++++++++++++++ arch/arm64/include/asm/io.h | 16 ++++++++++++---- arch/arm64/kernel/cpu_errata.c | 8 ++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 45 insertions(+), 4 deletions(-)Documentation/arch/arm64/silicon-errata.rst +2
-0@@ -298,6 +298,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | Olympus core | T410-OLY-1027 | NVIDIA_OLYMPUS_1027_ERRATUM | ++----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Olympus core | T410-OLY-1029 | ARM64_ERRATUM_4118414 | +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A |arch/arm64/Kconfig +22
-0@@ -1363,6 +1363,28 @@ config NVIDIA_CARMEL_CNP_ERRATUM If unsure, say Y. +config NVIDIA_OLYMPUS_1027_ERRATUM + bool "NVIDIA Olympus: device store/load ordering erratum" + default y + help + This option adds an alternative code sequence to work around an + NVIDIA Olympus core erratum where a Device-nGnR* store can be + observed by a peripheral after a younger Device-nGnR* load to the + same peripheral. This breaks the program order that drivers rely + on for MMIO and can leave a device in an incorrect state. + + The workaround inserts a DMB OSH immediately before raw MMIO loads. + The erratum cannot occur when a DMB that orders loads appears + between the store and load, preventing the younger load from being + observed before the older store. + + The alternatives framework patches in DMB OSH only when an affected + CPU is detected. Other CPUs execute a NOP in its place. Disabling + this option leaves the original MMIO read instruction stream + unchanged. + + If unsure, say Y. + config ROCKCHIP_ERRATUM_3568002 bool "Rockchip 3568002: GIC600 can not access physical addresses higher than 4GB" default yarch/arm64/include/asm/io.h +12
-4@@ -54,7 +54,9 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr) static __always_inline u8 __raw_readb(const volatile void __iomem *addr) {u8 val; - asm volatile(ALTERNATIVE("ldrb %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldrb %w0, [%1]", "ldarb %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));@@ -66,7 +68,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr) {u16 val; - asm volatile(ALTERNATIVE("ldrh %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldrh %w0, [%1]", "ldarh %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));@@ -77,7 +81,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr) static __always_inline u32 __raw_readl(const volatile void __iomem *addr) {u32 val; - asm volatile(ALTERNATIVE("ldr %w0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldr %w0, [%1]", "ldar %w0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));@@ -88,7 +94,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr) static __always_inline u64 __raw_readq(const volatile void __iomem *addr) {u64 val; - asm volatile(ALTERNATIVE("ldr %0, [%1]", + asm volatile(ALTERNATIVE("nop", "dmb osh", + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027) + ALTERNATIVE("ldr %0, [%1]", "ldar %0, [%1]", ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) : "=r" (val) : "r" (addr));arch/arm64/kernel/cpu_errata.c +8
-0@@ -839,6 +839,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = { ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL), },#endif +#ifdef CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM + { + /* NVIDIA Olympus core */ + .desc = "NVIDIA Olympus device store/load ordering erratum", + .capability = ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027, + ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS), + }, +#endif #ifdef CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE { /*arch/arm64/tools/cpucaps +0
-0
arch/arm64/tools/cpucaps WORKAROUND_CAVIUM_TX2_219_TVM · line 123
Quoted 3 lines
arch/arm64/tools/cpucaps +0
-0@@ -121,6 +121,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM WORKAROUND_CLEAN_CACHE WORKAROUND_DEVICE_LOAD_ACQUIRE WORKAROUND_NVIDIA_CARMEL_CNP
Nitpick: this ^ line has changed recently, so patch might not apply
arch/arm64/tools/cpucaps WORKAROUND_CAVIUM_TX2_219_TVM
Quoted 6 lines
arch/arm64/tools/cpucaps +1
-1+WORKAROUND_NVIDIA_OLYMPUS_1027 WORKAROUND_PMUV3_IMPDEF_TRAPS WORKAROUND_QCOM_FALKOR_E1003 WORKAROUND_QCOM_ORYON_CNTVOFF -- 2.54.0.windows.1
FWIW,
Thanks
Vladimir
Quoted 9 lines · Shanker Donthineni
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
[...]
Applied to arm64 (for-next/errata), thanks!
[1/1] arm64: errata: work around NVIDIA Olympus device store/load ordering
https://git.kernel.org/arm64/c/12aab25ca56e
Cheers,
Signature · 5 lines
Will https://fixes.arm64.dev https://next.arm64.dev https://will.arm64.dev