Skip to contentSkip to conversation

arm64: errata: Workaround NVIDIA Olympus device store/load ordering erratum

PATCH Superseded Shanker Donthineni 4 participants 10 messages Jun 10 – Jun 19 1 message not in the archive

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 arm64: errata: Workaround NVIDIA Olympus device store/load ordering erratum

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

 Documentation/arch/arm64/silicon-errata.rst |  2 ++
 arch/arm64/Kconfig                          | 23 ++++++++++++++++
 arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
 arch/arm64/kernel/cpu_errata.c              |  8 ++++++
 arch/arm64/tools/cpucaps                    |  1 +
 5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +2 -0
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
 +----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | T241 GICv3/4.x  | T241-FABRIC-4   | N/A                         |
arch/arm64/Kconfig +23 -0
index c65cef81be86a..d633eb70de1ac 100644
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
 	  If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+	bool "NVIDIA Olympus: device store/load ordering erratum"
+	default y
+	help
+	  This option adds an alternative code sequence to work around an
+	  NVIDIA Olympus core erratum where a Device-nGnR* store can be
+	  observed by a peripheral after a younger Device-nGnR* load to the
+	  same peripheral. This breaks the program order that drivers rely
+	  on for MMIO and can leave a device in an incorrect state.
+
+	  The workaround promotes the raw MMIO store helpers
+	  (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+	  required ordering. Because writel() and writel_relaxed() are built
+	  on __raw_writel(), both are covered without changes to the higher
+	  layers.
+
+	  The fix is applied through the alternatives framework, so enabling
+	  this option does not by itself activate the workaround: it is
+	  patched in only when an affected CPU is detected, and is a no-op on
+	  unaffected CPUs.
+
+	  If unsure, say Y.
+
 config ARM64_ERRATUM_834220
 	bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
 	depends on KVM
arch/arm64/include/asm/io.h +30 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
 /*
  * Generic IO read/write.  These perform native-endian accesses.
  */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+	return alternative_has_cap_unlikely(
+				ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
 #define __raw_writeb __raw_writeb
 static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
 {
 	volatile u8 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
+		asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
 }
 
@@ -33,6 +45,12 @@ static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
 static __always_inline void __raw_writew(u16 val, volatile void __iomem *addr)
 {
 	volatile u16 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
+		asm volatile("stlrh %w0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("strh %w0, %1" : : "rZ" (val), "Qo" (*ptr));
 }
 
@@ -40,6 +58,12 @@ static __always_inline void __raw_writew(u16 val, volatile void __iomem *addr)
 static __always_inline void __raw_writel(u32 val, volatile void __iomem *addr)
 {
 	volatile u32 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
+		asm volatile("stlr %w0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("str %w0, %1" : : "rZ" (val), "Qo" (*ptr));
 }
 
@@ -47,6 +71,12 @@ static __always_inline void __raw_writel(u32 val, volatile void __iomem *addr)
 static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr)
 {
 	volatile u64 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
+		asm volatile("stlr %x0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("str %x0, %1" : : "rZ" (val), "Qo" (*ptr));
 }
 
arch/arm64/kernel/cpu_errata.c +8 -0
index d597896b0f7f3..b096d9acca578 100644
@@ -838,6 +838,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL),
 	},
 #endif
+#ifdef CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM
+	{
+		/* NVIDIA Olympus core */
+		.desc = "NVIDIA Olympus device load/store ordering erratum",
+		.capability = ARM64_WORKAROUND_DEVICE_STORE_RELEASE,
+		ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS),
+	},
+#endif
 #ifdef CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE
 	{
 		/*
arch/arm64/tools/cpucaps +1 -0
index 811c2479e82d6..d367257bf7703 100644
@@ -120,6 +120,7 @@ WORKAROUND_CAVIUM_TX2_219_PRFM
 WORKAROUND_CAVIUM_TX2_219_TVM
 WORKAROUND_CLEAN_CACHE
 WORKAROUND_DEVICE_LOAD_ACQUIRE
+WORKAROUND_DEVICE_STORE_RELEASE
 WORKAROUND_NVIDIA_CARMEL_CNP
 WORKAROUND_PMUV3_IMPDEF_TRAPS
 WORKAROUND_QCOM_FALKOR_E1003
Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 131 lines · Shanker Donthineni

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

 Documentation/arch/arm64/silicon-errata.rst |  2 ++
 arch/arm64/Kconfig                          | 23 ++++++++++++++++
 arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
 arch/arm64/kernel/cpu_errata.c              |  8 ++++++
 arch/arm64/tools/cpucaps                    |  1 +
 5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +2 -0
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
 +----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | T241 GICv3/4.x  | T241-FABRIC-4   | N/A                         |
arch/arm64/Kconfig +23 -0
index c65cef81be86a..d633eb70de1ac 100644
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
 	  If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+	bool "NVIDIA Olympus: device store/load ordering erratum"
+	default y
+	help
+	  This option adds an alternative code sequence to work around an
+	  NVIDIA Olympus core erratum where a Device-nGnR* store can be
+	  observed by a peripheral after a younger Device-nGnR* load to the
+	  same peripheral. This breaks the program order that drivers rely
+	  on for MMIO and can leave a device in an incorrect state.
+
+	  The workaround promotes the raw MMIO store helpers
+	  (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+	  required ordering. Because writel() and writel_relaxed() are built
+	  on __raw_writel(), both are covered without changes to the higher
+	  layers.
+
+	  The fix is applied through the alternatives framework, so enabling
+	  this option does not by itself activate the workaround: it is
+	  patched in only when an affected CPU is detected, and is a no-op on
+	  unaffected CPUs.
+
+	  If unsure, say Y.
+
 config ARM64_ERRATUM_834220
 	bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
 	depends on KVM
arch/arm64/include/asm/io.h +6 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
 /*
  * Generic IO read/write.  These perform native-endian accesses.
  */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+	return alternative_has_cap_unlikely(
+				ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
 #define __raw_writeb __raw_writeb
 static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)

arch/arm64/include/asm/io.h line 42

Quoted 10 lines
arch/arm64/include/asm/io.h +6 -0
 {
 	volatile u8 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
+		asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
 }

Use an 'else' clause instead of the early return? (similarly for the other changes).

I still reckon you should do something with the memcpy-to-io routines. A simple option could be to make dgh() a dmb on parts with the erratum? That at least moves the barrier out of the loop.

Will

#2 lore raw

Hi Will,

Quoted 139 lines · Will Deacon

External email: Use caution opening links or attachments

Quoted 135 lines · Shanker Donthineni

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

  Documentation/arch/arm64/silicon-errata.rst |  2 ++
  arch/arm64/Kconfig                          | 23 ++++++++++++++++
  arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
  arch/arm64/kernel/cpu_errata.c              |  8 ++++++
  arch/arm64/tools/cpucaps                    |  1 +
  5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +2 -0
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
  +----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
  +----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
  +----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | T241 GICv3/4.x  | T241-FABRIC-4   | N/A                         |
arch/arm64/Kconfig +23 -0
index c65cef81be86a..d633eb70de1ac 100644
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
         If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+     bool "NVIDIA Olympus: device store/load ordering erratum"
+     default y
+     help
+       This option adds an alternative code sequence to work around an
+       NVIDIA Olympus core erratum where a Device-nGnR* store can be
+       observed by a peripheral after a younger Device-nGnR* load to the
+       same peripheral. This breaks the program order that drivers rely
+       on for MMIO and can leave a device in an incorrect state.
+
+       The workaround promotes the raw MMIO store helpers
+       (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+       required ordering. Because writel() and writel_relaxed() are built
+       on __raw_writel(), both are covered without changes to the higher
+       layers.
+
+       The fix is applied through the alternatives framework, so enabling
+       this option does not by itself activate the workaround: it is
+       patched in only when an affected CPU is detected, and is a no-op on
+       unaffected CPUs.
+
+       If unsure, say Y.
+
  config ARM64_ERRATUM_834220
       bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
       depends on KVM
arch/arm64/include/asm/io.h +8 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
  /*
   * Generic IO read/write.  These perform native-endian accesses.
   */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+     return alternative_has_cap_unlikely(
+                             ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
  #define __raw_writeb __raw_writeb
  static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
  {
       volatile u8 __iomem *ptr = addr;
+
+     if (arm64_needs_device_store_release()) {
Quoted 8 lines

arch/arm64/include/asm/io.h line 42

Quoted 6 lines
arch/arm64/include/asm/io.h +4 -0
+             asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+             return;
+     }
+
       asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
  }

Use an 'else' clause instead of the early return? (similarly for the other changes).

I agree. I’ll rework the raw write helpers to use an explicit if/else form instead of returning early from the STLR path.

Quoted 4 lines

I still reckon you should do something with the memcpy-to-io routines. A simple option could be to make dgh() a dmb on parts with the erratum? That at least moves the barrier out of the loop.

For the memcpy-to-IO routines, would it be acceptable to address the erratum by patching dgh() to a DMB OSH on affected CPUs, as shown below? I’ll also sync with the Olympus CPU hardware team to confirm this approach for the v4 patch.

#define dgh()		asm volatile(ALTERNATIVE("hint #6", "dmb osh",	\
					ARM64_WORKAROUND_DEVICE_STORE_RELEASE) \
				     : : : "memory")

This keeps the existing memcpy-to-IO store sequences unchanged while placing the ordering barrier outside the copy loop as you suggested.

-Shanker

#3 lore raw
In reply to Will Deacon

Hi,

Quoted 136 lines · Will Deacon
Quoted 135 lines · Shanker Donthineni

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

 Documentation/arch/arm64/silicon-errata.rst |  2 ++
 arch/arm64/Kconfig                          | 23 ++++++++++++++++
 arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
 arch/arm64/kernel/cpu_errata.c              |  8 ++++++
 arch/arm64/tools/cpucaps                    |  1 +
 5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +2 -0
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
 +----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
 | NVIDIA         | T241 GICv3/4.x  | T241-FABRIC-4   | N/A                         |
arch/arm64/Kconfig +23 -0
index c65cef81be86a..d633eb70de1ac 100644
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
 	  If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+	bool "NVIDIA Olympus: device store/load ordering erratum"
+	default y
+	help
+	  This option adds an alternative code sequence to work around an
+	  NVIDIA Olympus core erratum where a Device-nGnR* store can be
+	  observed by a peripheral after a younger Device-nGnR* load to the
+	  same peripheral. This breaks the program order that drivers rely
+	  on for MMIO and can leave a device in an incorrect state.
+
+	  The workaround promotes the raw MMIO store helpers
+	  (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+	  required ordering. Because writel() and writel_relaxed() are built
+	  on __raw_writel(), both are covered without changes to the higher
+	  layers.
+
+	  The fix is applied through the alternatives framework, so enabling
+	  this option does not by itself activate the workaround: it is
+	  patched in only when an affected CPU is detected, and is a no-op on
+	  unaffected CPUs.
+
+	  If unsure, say Y.
+
 config ARM64_ERRATUM_834220
 	bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
 	depends on KVM
arch/arm64/include/asm/io.h +8 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
 /*
  * Generic IO read/write.  These perform native-endian accesses.
  */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+	return alternative_has_cap_unlikely(
+				ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
 #define __raw_writeb __raw_writeb
 static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
 {
 	volatile u8 __iomem *ptr = addr;
+
+	if (arm64_needs_device_store_release()) {
Quoted 8 lines

arch/arm64/include/asm/io.h line 42

Quoted 6 lines
arch/arm64/include/asm/io.h +4 -0
+		asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+		return;
+	}
+
 	asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
 }

Use an 'else' clause instead of the early return? (similarly for the other changes).

Perhaps I'm missing something, but it is not clear to me why all that complexity is required.

IIUC, benefits coming with d044d6ba6f02 ("arm64: io: permit offset addressing") are from better code generation, so we:
- save code
- open opportunity for write-combining

d044d6ba6f02 ("arm64: io: permit offset addressing") comes with simple benchmark to measure effect of code generation:

| void writeq_zero_8_times(void *ptr)
| {
|        writeq_relaxed(0, ptr + 8 * 0);
|        writeq_relaxed(0, ptr + 8 * 1);
|        writeq_relaxed(0, ptr + 8 * 2);
|        writeq_relaxed(0, ptr + 8 * 3);
|        writeq_relaxed(0, ptr + 8 * 4);
|        writeq_relaxed(0, ptr + 8 * 5);
|        writeq_relaxed(0, ptr + 8 * 6);
|        writeq_relaxed(0, ptr + 8 * 7);
| }

which compiles to

| <writeq_zero_8_times>:
|        str     xzr, [x0]
|        str     xzr, [x0, #8]
|        str     xzr, [x0, #16]
|        str     xzr, [x0, #24]
|        str     xzr, [x0, #32]
|        str     xzr, [x0, #40]
|        str     xzr, [x0, #48]
|        str     xzr, [x0, #56]

v1/v2 compiles to

| <writeq_zero_8_times>:
|        str     xzr, [x0]
|        add     x1, x0, #0x8
|        str     xzr, [x1]
|        add     x1, x0, #0x10
|        str     xzr, [x1]
|        add     x1, x0, #0x18
|        str     xzr, [x1]
|        add     x1, x0, #0x20
|        str     xzr, [x1]
|        add     x1, x0, #0x28
|        str     xzr, [x1]
|        add     x1, x0, #0x30
|        str     xzr, [x1]
|        add     x0, x0, #0x38
|        str     xzr, [x0]

were alternatives are swapping str with stlr. In other words, we are rolling back to the pre-d044d6ba6f02 implementation.

v3 compiles to:

| <writeq_zero_8_times>:
|        nop
|        str     xzr, [x0]
|        add     x1, x0, #0x8
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x10
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x18
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x20
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x28
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x30
|        nop
|        str     xzr, [x1]
|        add     x0, x0, #0x38
|        nop
|        str     xzr, [x0]
|        ret

where static branch swapping nop with branch to stlr and back to add.

So it looks to me that we're losing an opportunity for write combining, but in terms of code size, v1/v2 seems to be the lesser of two evils.

Cheers
Vladimir

Quoted 7 lines

I still reckon you should do something with the memcpy-to-io routines. A simple option could be to make dgh() a dmb on parts with the erratum? That at least moves the barrier out of the loop.

Will

#4 lore raw

Hi Vladimir,

Quoted 236 lines · Vladimir Murzin

External email: Use caution opening links or attachments

Hi,

Quoted 136 lines · Will Deacon
Quoted 135 lines · Shanker Donthineni

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

  Documentation/arch/arm64/silicon-errata.rst |  2 ++
  arch/arm64/Kconfig                          | 23 ++++++++++++++++
  arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
  arch/arm64/kernel/cpu_errata.c              |  8 ++++++
  arch/arm64/tools/cpucaps                    |  1 +
  5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +2 -0
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
  +----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
  +----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
  +----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | T241 GICv3/4.x  | T241-FABRIC-4   | N/A                         |
arch/arm64/Kconfig +23 -0
index c65cef81be86a..d633eb70de1ac 100644
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
        If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+    bool "NVIDIA Olympus: device store/load ordering erratum"
+    default y
+    help
+      This option adds an alternative code sequence to work around an
+      NVIDIA Olympus core erratum where a Device-nGnR* store can be
+      observed by a peripheral after a younger Device-nGnR* load to the
+      same peripheral. This breaks the program order that drivers rely
+      on for MMIO and can leave a device in an incorrect state.
+
+      The workaround promotes the raw MMIO store helpers
+      (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+      required ordering. Because writel() and writel_relaxed() are built
+      on __raw_writel(), both are covered without changes to the higher
+      layers.
+
+      The fix is applied through the alternatives framework, so enabling
+      this option does not by itself activate the workaround: it is
+      patched in only when an affected CPU is detected, and is a no-op on
+      unaffected CPUs.
+
+      If unsure, say Y.
+
  config ARM64_ERRATUM_834220
      bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
      depends on KVM
arch/arm64/include/asm/io.h +8 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
  /*
   * Generic IO read/write.  These perform native-endian accesses.
   */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+    return alternative_has_cap_unlikely(
+                            ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
  #define __raw_writeb __raw_writeb
  static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
  {
      volatile u8 __iomem *ptr = addr;
+
+    if (arm64_needs_device_store_release()) {
Quoted 8 lines

arch/arm64/include/asm/io.h line 42

Quoted 6 lines
arch/arm64/include/asm/io.h +4 -0
+            asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+            return;
+    }
+
      asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
  }

Use an 'else' clause instead of the early return? (similarly for the other changes).

Perhaps I'm missing something, but it is not clear to me why all that complexity is required.

IIUC, benefits coming with d044d6ba6f02 ("arm64: io: permit offset addressing") are from better code generation, so we:
- save code
- open opportunity for write-combining

d044d6ba6f02 ("arm64: io: permit offset addressing") comes with simple benchmark to measure effect of code generation:

| void writeq_zero_8_times(void *ptr)
| {
|        writeq_relaxed(0, ptr + 8 * 0);
|        writeq_relaxed(0, ptr + 8 * 1);
|        writeq_relaxed(0, ptr + 8 * 2);
|        writeq_relaxed(0, ptr + 8 * 3);
|        writeq_relaxed(0, ptr + 8 * 4);
|        writeq_relaxed(0, ptr + 8 * 5);
|        writeq_relaxed(0, ptr + 8 * 6);
|        writeq_relaxed(0, ptr + 8 * 7);
| }

which compiles to

| <writeq_zero_8_times>:
|        str     xzr, [x0]
|        str     xzr, [x0, #8]
|        str     xzr, [x0, #16]
|        str     xzr, [x0, #24]
|        str     xzr, [x0, #32]
|        str     xzr, [x0, #40]
|        str     xzr, [x0, #48]
|        str     xzr, [x0, #56]

v1/v2 compiles to

| <writeq_zero_8_times>:
|        str     xzr, [x0]
|        add     x1, x0, #0x8
|        str     xzr, [x1]
|        add     x1, x0, #0x10
|        str     xzr, [x1]
|        add     x1, x0, #0x18
|        str     xzr, [x1]
|        add     x1, x0, #0x20
|        str     xzr, [x1]
|        add     x1, x0, #0x28
|        str     xzr, [x1]
|        add     x1, x0, #0x30
|        str     xzr, [x1]
|        add     x0, x0, #0x38
|        str     xzr, [x0]

were alternatives are swapping str with stlr. In other words, we are rolling back to the pre-d044d6ba6f02 implementation.

v3 compiles to:

| <writeq_zero_8_times>:
|        nop
|        str     xzr, [x0]
|        add     x1, x0, #0x8
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x10
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x18
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x20
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x28
|        nop
|        str     xzr, [x1]
|        add     x1, x0, #0x30
|        nop
|        str     xzr, [x1]
|        add     x0, x0, #0x38
|        nop
|        str     xzr, [x0]
|        ret
Quoted 5 lines

where static branch swapping nop with branch to stlr and back to add.

So it looks to me that we're losing an opportunity for write combining, but in terms of code size, v1/v2 seems to be the lesser of two evils.

Thanks, that makes sense.

My intent with the v3 change was to keep the offset-addressed STR sequence on unaffected CPUs and use the base-register STLR sequence only on affected CPUs. However, as you point out, because STLR only supports base-register addressing, the affected path still forces the address to be materialized in a register, and the alternative_has_cap_unlikely() check adds another instruction at each write site. So the generated code no longer preserves the benefit from d044d6ba6f02 in practice.

Given that, I agree the extra complexity is not justified. I’ll simplify the raw MMIO write helpers back to the direct ALTERNATIVE() form from v1/v2, where both the STR and STLR paths use base-register addressing. That is still a regression from the offset-addressed STR sequence on unaffected CPUs, but it avoids the additional static-branch/nop overhead and is the smaller of the two options.

-Shanker

#5 lore raw
In reply to Will Deacon
Quoted 3 lines · Will Deacon

I still reckon you should do something with the memcpy-to-io routines. A simple option could be to make dgh() a dmb on parts with the erratum? That at least moves the barrier out of the loop.

AFAIK only callers that know they are using WC memory should be calling dgh() and in that case we know it is NORMAL-NC and we don't need a different barrier

Other random users calling memcpy_to_io functions on real IO don't have to do dgh(), and AFAIK it doesn't do anything on the Device memory types?

Jason

A message that is not in the archive (sent off-list or not yet received): IA1PR12MB6089049028A73A2078FC6831C71B2@IA1PR12MB6089.namprd12.prod.outlook.com

#7 lore raw

Hi Will,

Quoted 159 lines · On 6/11/2026 8:39 AM, [email protected] wrote:

-----Original Message-----
From: Will Deacon <[email protected]>
Sent: Thursday, June 11, 2026 8:34 AM
To: Shanker Donthineni <[email protected]>
Cc: Catalin Marinas <[email protected]>; Vladimir Murzin <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Mark Rutland <[email protected]>; [email protected]; [email protected]; Vikram Sethi <[email protected]>; Jason Sequeira <[email protected]>
Subject: Re: [PATCH v3] arm64: errata: Workaround NVIDIA Olympus device store/load ordering erratum

External email: Use caution opening links or attachments

Quoted 135 lines · Shanker Donthineni

On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state, as a load is observed before an earlier store takes effect.

The erratum can occur only when all of the following apply:

- A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads.
- There is no DSB, and no DMB that orders loads, between the store and the load.
- Specific micro-architectural and timing conditions occur.

Promote the raw MMIO store helpers (__raw_writeb/w/l/q) from plain str* to stlr* (Store-Release), which removes the "store is not a store-release" condition for every device write the kernel issues. Because writel() and writel_relaxed() are both built on __raw_writel() in asm-generic/io.h, patching the raw variants covers both the non-relaxed and relaxed APIs without touching the higher layers. Note that writel()'s own barrier sits before the store, so it does not order the store against a subsequent readl(); the store-release promotion is what provides that ordering.

Like ARM64_ERRATUM_832075 on the load side, the change is gated on a new ARM64_WORKAROUND_DEVICE_STORE_RELEASE capability and only activated on parts that match MIDR_NVIDIA_OLYMPUS, so unaffected CPUs continue to use the plain str* sequence.

Note: stlr* only supports base-register addressing, so affected CPUs use a base-register stlr* path. Unaffected CPUs keep the original offset-addressed str* sequence introduced by commit d044d6ba6f02 ("arm64: io: permit offset addressing").

The __const_memcpy_toio_aligned32() and __const_memcpy_toio_aligned64() helpers are left unchanged. These helpers are intended for write-combining mappings, which are Normal-NC on arm64. Replacing their contiguous str* groups would defeat the write-combining behavior used to improve store performance.

---
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the existing offset-addressed STR sequence, while affected CPUs use the base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:

https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h
=for-next/errata

Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.

  Documentation/arch/arm64/silicon-errata.rst |  2 ++
  arch/arm64/Kconfig                          | 23 ++++++++++++++++
  arch/arm64/include/asm/io.h                 | 30 +++++++++++++++++++++
  arch/arm64/kernel/cpu_errata.c              |  8 ++++++
  arch/arm64/tools/cpucaps                    |  1 +
  5 files changed, 64 insertions(+)
Documentation/arch/arm64/silicon-errata.rst +3 -1
index ad09bbb10da80..fc45125dc2f80 100644
@@ -298,6 +298,8 @@ stable kernels.
  +----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
 
+----------------+-----------------+-----------------+----------------
-------------+
+| NVIDIA         | Olympus core    | T410-OLY-1027   | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
  | NVIDIA         | Olympus core    | T410-OLY-1029   | ARM64_ERRATUM_4118414       |
  +----------------+-----------------+-----------------+-----------------------------+

| NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A |

arch/arm64/Kconfig index renamed from arch/arm64/Kconfig +0 -0

c65cef81be86a..d633eb70de1ac 100644

arch/arm64/Kconfig +23 -0
@@ -564,6 +564,29 @@ config ARM64_ERRATUM_832075
 
         If unsure, say Y.
 
+config NVIDIA_OLYMPUS_1027_ERRATUM
+     bool "NVIDIA Olympus: device store/load ordering erratum"
+     default y
+     help
+       This option adds an alternative code sequence to work around an
+       NVIDIA Olympus core erratum where a Device-nGnR* store can be
+       observed by a peripheral after a younger Device-nGnR* load to the
+       same peripheral. This breaks the program order that drivers rely
+       on for MMIO and can leave a device in an incorrect state.
+
+       The workaround promotes the raw MMIO store helpers
+       (__raw_writeb/w/l/q) to Store-Release (STLR), which restores the
+       required ordering. Because writel() and writel_relaxed() are built
+       on __raw_writel(), both are covered without changes to the higher
+       layers.
+
+       The fix is applied through the alternatives framework, so enabling
+       this option does not by itself activate the workaround: it is
+       patched in only when an affected CPU is detected, and is a no-op on
+       unaffected CPUs.
+
+       If unsure, say Y.
+
  config ARM64_ERRATUM_834220
       bool "Cortex-A57: 834220: Stage 2 translation fault might be incorrectly reported in presence of a Stage 1 fault (rare)"
       depends on KVM
arch/arm64/include/asm/io.h +6 -0
index 8cbd1e96fd50b..801223e754c90 100644
@@ -22,10 +22,22 @@
  /*
   * Generic IO read/write.  These perform native-endian accesses.
   */
+static __always_inline bool arm64_needs_device_store_release(void)
+{
+     return alternative_has_cap_unlikely(
+                             ARM64_WORKAROUND_DEVICE_STORE_RELEASE);
+}
+
  #define __raw_writeb __raw_writeb

arch/arm64/include/asm/io.h line 41

Quoted 10 lines
arch/arm64/include/asm/io.h +6 -0
  static __always_inline void __raw_writeb(u8 val, volatile void
 __iomem *addr)  {
       volatile u8 __iomem *ptr = addr;
+
+     if (arm64_needs_device_store_release()) {
+             asm volatile("stlrb %w0, [%1]" : : "rZ" (val), "r" (addr));
+             return;
+     }
+
       asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));  }

Use an 'else' clause instead of the early return? (similarly for the other changes).

Quoted 3 lines

I still reckon you should do something with the memcpy-to-io routines. A simple option could be to make dgh() a dmb on parts with the erratum? That at least moves the barrier out of the loop.

Thanks Will. I looked again at both the arm64 comments and the generic iomap_copy.c contract, and I’m not convinced that making dgh() a dmb is the right fit for this path. Based on the documented comments, callers should not assume ordering from these helpers; if ordering is required around a memcpy, the call site should already be providing the necessary barriers.

Related data point in generic lib/iomap_copy.c:

/**
* __iowrite32_copy - copy data to MMIO space, in 32-bit units
* @to: destination, in MMIO space (must be 32-bit aligned)
* @from: source (must be 32-bit aligned)
* @count: number of 32-bit quantities to copy
*
* Copy data from kernel space to MMIO space, in units of 32 bits at a
* time. Order of access is not guaranteed, nor is a memory barrier
* performed afterwards.
*/
#ifndef __iowrite32_copy
void __iowrite32_copy(void __iomem *to, const void *from, size_t count)

/**
* __iowrite64_copy - copy data to MMIO space, in 64-bit or 32-bit units
* @to: destination, in MMIO space (must be 64-bit aligned)
* @from: source (must be 64-bit aligned)
* @count: number of 64-bit quantities to copy
*
* Copy data from kernel space to MMIO space, in units of 32 or 64 bits at a
* time. Order of access is not guaranteed, nor is a memory barrier
* performed afterwards.
*/
#ifndef __iowrite64_copy
void __iowrite64_copy(void __iomem *to, const void *from, size_t count)

/**
 * __iowrite32_copy - copy data to MMIO space, in 32-bit units
 * @to: destination, in MMIO space (must be 32-bit aligned)
 * @from: source (must be 32-bit aligned)
 * @count: number of 32-bit quantities to copy
 *
 * Copy data from kernel space to MMIO space, in units of 32 bits at a
 * time.  Order of access is not guaranteed, nor is a memory barrier
 * performed afterwards.
 */
#ifndef __iowrite32_copy
void __iowrite32_copy(void __iomem *to, const void *from, size_t count)

The arm64 comment says in arch/arm64/asm/io.h:

/*
 * The ARM64 iowrite implementation is intended to support drivers that want to
 * use write combining. For instance PCI drivers using write combining with a 64
 * byte __iowrite64_copy() expect to get a 64 byte MemWr TLP on the PCIe bus. *
 * Newer ARM core have sensitive write combining buffers, it is important that
 * the stores be contiguous blocks of store instructions. Normal memcpy
 * approaches have a very low chance to generate write combining.
 *
 * Since this is the only API on ARM64 that should be used with write combining
 * it also integrates the DGH hint which is supposed to lower the latency to
 * emit the large TLP from the CPU.
 */

So my reading is that dgh() in the arm64 implementation is there for the write-combining/gathering behavior. Replacing it with dmb would make this path stronger than the generic API contract and could penalize performance of the WC use case.

For the scalar MMIO helpers, the workaround promotes the raw writes to store-release on affected CPUs as v1/v2 shown below. For the memcpy-toIO helpers, could you please clarify the specific reason for adding a dmb despite the documented no-ordering contract? Is the concern that some drivers may be relying on ordering across memcpy_toio_*() today even though the API does not guarantee it, and that we should cover those cases defensively?

Would prefer to avoid replacing DGH() with DMB unless there is a strong reason to do so. Please let me know if I can post the v4 patch with the change below, while keeping DGH() as-is in the memcpy-toIO path.

#define __raw_writeb __raw_writeb
static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr) {

-       volatile u8 __iomem *ptr = addr;
-       asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
+       asm volatile(ALTERNATIVE("strb %w0, [%1]",

+ "stlrb %w0, [%1]",
+ ARM64_WORKAROUND_DEVICE_STORE_RELEASE)
+ : : "rZ" (val), "r" (addr));
}

#define __raw_writew __raw_writew
static __always_inline void __raw_writew(u16 val, volatile void __iomem *addr) {

-       volatile u16 __iomem *ptr = addr;
-       asm volatile("strh %w0, %1" : : "rZ" (val), "Qo" (*ptr));
+       asm volatile(ALTERNATIVE("strh %w0, [%1]",

+ "stlrh %w0, [%1]",
+ ARM64_WORKAROUND_DEVICE_STORE_RELEASE)
+ : : "rZ" (val), "r" (addr));
}

#define __raw_writel __raw_writel
static __always_inline void __raw_writel(u32 val, volatile void __iomem *addr) {

-       volatile u32 __iomem *ptr = addr;
-       asm volatile("str %w0, %1" : : "rZ" (val), "Qo" (*ptr));
+       asm volatile(ALTERNATIVE("str %w0, [%1]",

+ "stlr %w0, [%1]",
+ ARM64_WORKAROUND_DEVICE_STORE_RELEASE)
+ : : "rZ" (val), "r" (addr));
}

#define __raw_writeq __raw_writeq
static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr) {

-       volatile u64 __iomem *ptr = addr;
-       asm volatile("str %x0, %1" : : "rZ" (val), "Qo" (*ptr));
+       asm volatile(ALTERNATIVE("str %x0, [%1]",

+ "stlr %x0, [%1]",
+ ARM64_WORKAROUND_DEVICE_STORE_RELEASE)
+ : : "rZ" (val), "r" (addr));
}

-Shanker

#8 lore raw
Quoted 6 lines · Shanker Donthineni

For the scalar MMIO helpers, the workaround promotes the raw writes to store-release on affected CPUs as v1/v2 shown below. For the memcpy-toIO helpers, could you please clarify the specific reason for adding a dmb despite the documented no-ordering contract? Is the concern that some drivers may be relying on ordering across memcpy_toio_*() today even though the API does not guarantee it, and that we should cover those cases defensively?

I think given how arm implements them today the iocopy's are actually the _relaxed variations.. I wonder if this matters to any user?

Jason

#9 lore raw

Hi Will,

Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Shanker Donthineni

For the scalar MMIO helpers, the workaround promotes the raw writes to store-release on affected CPUs as v1/v2 shown below. For the memcpy-toIO helpers, could you please clarify the specific reason for adding a dmb despite the documented no-ordering contract? Is the concern that some drivers may be relying on ordering across memcpy_toio_*() today even though the API does not guarantee it, and that we should cover those cases defensively?

I think given how arm implements them today the iocopy's are actually the _relaxed variations.. I wonder if this matters to any user?

Following Jason's observation that on arm64 the memcpy_toio() /__iowrite{32,64}_copy() helpers are effectively the relaxed (write-combining) variants, I'd like to settle one open point before posting v4: should the workaround also promote dgh() > dmb on affected CPUs (now Olympus core), or leave dgh() as a plain hint?

If you'd still prefer the dmb defensively, to cover drivers that may rely on ordering across memcpy_toio() today despite the relaxed contract, I'm happy to fold it into v4.

Please let me know how you'd like me to proceed.

-Shanker

maintainer
#10 lore raw
Quoted 18 lines · Shanker Donthineni
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Shanker Donthineni

For the scalar MMIO helpers, the workaround promotes the raw writes to store-release on affected CPUs as v1/v2 shown below. For the memcpy-toIO helpers, could you please clarify the specific reason for adding a dmb despite the documented no-ordering contract? Is the concern that some drivers may be relying on ordering across memcpy_toio_*() today even though the API does not guarantee it, and that we should cover those cases defensively?

I think given how arm implements them today the iocopy's are actually the _relaxed variations.. I wonder if this matters to any user?

Following Jason's observation that on arm64 the memcpy_toio() /__iowrite{32,64}_copy() helpers are effectively the relaxed (write-combining) variants, I'd like to settle one open point before posting
v4: should the workaround also promote dgh() > dmb on affected CPUs (now Olympus core), or leave dgh() as a plain hint?

Quoted 3 lines

If you'd still prefer the dmb defensively, to cover drivers that may
rely on ordering across memcpy_toio() today despite the relaxed contract, I'm happy to fold it into v4.

The point is, you're going to have different behaviour to every other arm64 system out there. You may be able to find vague comments in the code that imply that you don't need to provide ordering, but at the end of the day it's a pretty cavalier attitude imo and if a driver ever shows up that relies on it then you're in trouble.

Quoted 1 line

Please let me know how you'd like me to proceed.

It's up to you. It's your broken CPU, not mine. You also haven't actually provided any performance data for others to assess the trade-off.

If it was up to _me_, I'd upgrade dgh() on these CPUs to that I don't need to worry about this again.

Will