Skip to contentSkip to conversation

iommu/arm-smmu-v3: Tegra264 invalidation workaround

PATCH Superseded Ashish Mhetre 5 participants 24 messages Jun 1 – Jun 9

Subsystems (3)

v3 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v3 cover iommu/arm-smmu-v3: Tegra264 invalidation workaround

Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

ATC_INV is not affected and must not be doubled.

This series implements the workaround by hooking the duplication into arm_smmu_cmdq_issue_cmdlist(), the single chokepoint that every synchronous CMDQ submission flows through.

Patch 1 is a preparatory refactor that factors the existing batch force-sync conditions out of arm_smmu_cmdq_batch_add_cmd_p() into a new arm_smmu_cmdq_batch_force_sync() helper. No functional change.

Patch 2 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string, exposes the condition via a new ARM_SMMU_OPT_TLBI_TWICE option bit, and adds a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that both the in-tree CMDQ path and the iommufd VSMMU path can share a single predicate.

Patch 3 wires the workaround in. arm_smmu_cmdq_issue_cmdlist() becomes a thin wrapper that re-issues a synced cmdlist a second time when the first command needs doubling. The Tegra264 condition is added to arm_smmu_cmdq_batch_force_sync() so a full batch carrying CFGI/TLBI commands flushes with sync=true and is then doubled. The iommufd VSMMU path (arm_vsmmu_cache_invalidate()) is also taught to split the user-supplied batch at every "needs doubling" / "doesn't need doubling" transition via a new arm_vsmmu_can_batch_cmd() predicate, since that path can otherwise mix CFGI/TLBI with ATC_INV in a single submission.

The series is based on Jason Gunthorpe's "Remove SMMUv3 struct arm_smmu_cmdq_ent" series [1], specifically commit 13428b0bf794 ("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.

[1] https://lore.kernel.org/all/[email protected]/

Changes since v2:
- Add new prep patch 1/3 that factors the existing force-sync conditions into arm_smmu_cmdq_batch_force_sync() (from Nicolin).
- Move arm_smmu_cmd_needs_tlbi_twice() to arm-smmu-v3.h as static inline taking (smmu, cmd*) and folding in the option check.
- Plug the Tegra264 condition into arm_smmu_cmdq_batch_force_sync() instead of carrying a separate need_sync in batch_add_cmd_p().
- Fix iommufd batching: arm_vsmmu_cache_invalidate() can mix CFGI/TLBI with ATC_INV in one batch. Split at the boundary via a new arm_vsmmu_can_batch_cmd() predicate.
- Patch 2 wording: "next patch wires" -> "a subsequent change will wire".

v2: https://lore.kernel.org/all/[email protected]/

Nicolin Chen (1):
iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

Ashish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 23 ++++++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 66 +++++++++++++++----
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   | 40 +++++++++++
 3 files changed, 117 insertions(+), 12 deletions(-)

base-commit: 13428b0bf7947098daf9a1db14a74d33eb1b5079

Signature · 1 line
2.50.1
#1 lore raw

PATCH v3 1/3 iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

From: Nicolin Chen <[email protected]>

arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:

- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.

Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++-----
 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +20 -6
index 9be589d14a3b..4d29bd343460 100644
@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu,
 	cmds->cmdq = arm_smmu_get_cmdq(smmu, cmd);
 }
 
+static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
+					   struct arm_smmu_cmdq_batch *cmds,
+					   struct arm_smmu_cmd *cmd)
+{
+	if (!cmds->num)
+		return false;
+
+	/* The batch's pre-assigned cmdq doesn't support the new command */
+	if (!arm_smmu_cmdq_supports_cmd(cmds->cmdq, cmd))
+		return true;
+
+	/* Arm erratum 2812531 */
+	if (cmds->num == CMDQ_BATCH_ENTRIES - 1 &&
+	    (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))
+		return true;
+
+	return false;
+}
+
 static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
 					  struct arm_smmu_cmdq_batch *cmds,
 					  struct arm_smmu_cmd *cmd)
 {
-	bool force_sync = (cmds->num == CMDQ_BATCH_ENTRIES - 1) &&
-			  (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC);
-	bool unsupported_cmd;
-
-	unsupported_cmd = !arm_smmu_cmdq_supports_cmd(cmds->cmdq, cmd);
-	if (force_sync || unsupported_cmd) {
+	if (arm_smmu_cmdq_batch_force_sync(smmu, cmds, cmd)) {
 		arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
 					    cmds->num, true);
 		arm_smmu_cmdq_batch_init_cmd(smmu, cmds, cmd);
Signature · 1 line
2.50.1
maintainer
#2 lore raw

Re: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

Quoted 25 lines · Ashish Mhetre

From: Nicolin Chen <[email protected]>

arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:

- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.

Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++-----
 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 9be589d14a3b..4d29bd343460 100644

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855

Quoted 9 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6 -0
@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu,
 	cmds->cmdq = arm_smmu_get_cmdq(smmu, cmd);
 }
 
+static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
+					   struct arm_smmu_cmdq_batch *cmds,
+					   struct arm_smmu_cmd *cmd)
+{
+	if (!cmds->num)
+		return false;

This check seems new?

Will

#3 lore raw

Re: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

Quoted 29 lines · Will Deacon
Quoted 28 lines · Ashish Mhetre

From: Nicolin Chen <[email protected]>

arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:

- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.

Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++-----
 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 9be589d14a3b..4d29bd343460 100644
@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu,
 	cmds->cmdq = arm_smmu_get_cmdq(smmu, cmd);
 }
 
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855

Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6 -0
+static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
+					   struct arm_smmu_cmdq_batch *cmds,
+					   struct arm_smmu_cmd *cmd)
+{
+	if (!cmds->num)
+		return false;

This check seems new?

You are right. Maybe the commit message should have mentioned that there is a slight behavior change to the unsupported_cmd path:

- Before: if cmds->num = 0 && unsupported_cmd, it would issue an empty batch (one CMD_SYNC)
- After : if cmds->num = 0, no issue on the empty batch

With that, I think it's good to have?

Thanks
Nicolin

maintainer
#4 lore raw

Re: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

Quoted 39 lines · Nicolin Chen
Quoted 29 lines · Will Deacon
Quoted 28 lines · Ashish Mhetre

From: Nicolin Chen <[email protected]>

arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:

- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.

Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++-----
 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -0
index 9be589d14a3b..4d29bd343460 100644
@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu,
 	cmds->cmdq = arm_smmu_get_cmdq(smmu, cmd);
 }
 
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855

Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6 -0
+static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
+					   struct arm_smmu_cmdq_batch *cmds,
+					   struct arm_smmu_cmd *cmd)
+{
+	if (!cmds->num)
+		return false;

This check seems new?

Quoted 8 lines

You are right. Maybe the commit message should have mentioned that there is a slight behavior change to the unsupported_cmd path:

- Before: if cmds->num = 0 && unsupported_cmd, it would issue an empty batch (one CMD_SYNC)
- After : if cmds->num = 0, no issue on the empty batch

With that, I think it's good to have?

It just shouldn't be part of the refactoring patch with "no functional change". Do it in the patch that needs it and explain why it's needed.

Will

#5 lore raw

PATCH v3 2/3 iommu/arm-smmu-v3: Detect Tegra264 erratum

In reply to Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.

No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  4 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++
 2 files changed, 43 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
index 4d29bd343460..08684bd40a6d 100644
@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	if (of_dma_is_coherent(dev->of_node))
 		smmu->features |= ARM_SMMU_FEAT_COHERENCY;
 
-	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu"))
+	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu")) {
 		tegra_cmdqv_dt_probe(dev->of_node, smmu);
+		smmu->options |= ARM_SMMU_OPT_TLBI_TWICE;
+	}
 
 	return ret;
 }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +40 -0
index 16353596e08a..106034c348a1 100644
@@ -928,6 +928,14 @@ struct arm_smmu_device {
 #define ARM_SMMU_OPT_MSIPOLL		(1 << 2)
 #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC	(1 << 3)
 #define ARM_SMMU_OPT_TEGRA241_CMDQV	(1 << 4)
+/*
+ * Tegra264 erratum: a TLB entry can survive an invalidation that races
+ * with concurrent traffic targeting the same entry. The software
+ * workaround is to issue every CFGI/TLBI command twice, each followed
+ * by CMD_SYNC. The second issue is guaranteed to evict the entry.
+ * ATC_INV commands are not affected and must not be doubled.
+ */
+#define ARM_SMMU_OPT_TLBI_TWICE		(1 << 5)
 	u32				options;
 
 	struct arm_smmu_cmdq		cmdq;
@@ -1211,6 +1219,38 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 				struct arm_smmu_cmd *cmds, int n,
 				bool sync);
 
+/*
+ * Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered
+ * by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected
+ * SMMU instance.
+ */
+static inline bool arm_smmu_cmd_needs_tlbi_twice(struct arm_smmu_device *smmu,
+						 struct arm_smmu_cmd *cmd)
+{
+	if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+		return false;
+
+	switch (FIELD_GET(CMDQ_0_OP, cmd->data[0])) {
+	case CMDQ_OP_CFGI_STE:
+	case CMDQ_OP_CFGI_ALL:
+	case CMDQ_OP_CFGI_CD:
+	case CMDQ_OP_CFGI_CD_ALL:
+	case CMDQ_OP_TLBI_NH_ALL:
+	case CMDQ_OP_TLBI_NH_ASID:
+	case CMDQ_OP_TLBI_NH_VA:
+	case CMDQ_OP_TLBI_NH_VAA:
+	case CMDQ_OP_TLBI_EL2_ALL:
+	case CMDQ_OP_TLBI_EL2_ASID:
+	case CMDQ_OP_TLBI_EL2_VA:
+	case CMDQ_OP_TLBI_S12_VMALL:
+	case CMDQ_OP_TLBI_S2_IPA:
+	case CMDQ_OP_TLBI_NSNH_ALL:
+		return true;
+	default:
+		return false;
+	}
+}
+
 #ifdef CONFIG_ARM_SMMU_V3_SVA
 bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
 void arm_smmu_sva_notifier_synchronize(void);
Signature · 1 line
2.50.1
maintainer
#6 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.

Quoted 39 lines

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.

No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  4 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++
 2 files changed, 43 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
index 4d29bd343460..08684bd40a6d 100644
@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	if (of_dma_is_coherent(dev->of_node))
 		smmu->features |= ARM_SMMU_FEAT_COHERENCY;
 
-	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu"))
+	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu")) {
 		tegra_cmdqv_dt_probe(dev->of_node, smmu);
+		smmu->options |= ARM_SMMU_OPT_TLBI_TWICE;
+	}
 
 	return ret;
 }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -0
index 16353596e08a..106034c348a1 100644
@@ -928,6 +928,14 @@ struct arm_smmu_device {
 #define ARM_SMMU_OPT_MSIPOLL		(1 << 2)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_device { · line 938

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
 #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC	(1 << 3)
 #define ARM_SMMU_OPT_TEGRA241_CMDQV	(1 << 4)
+/*
+ * Tegra264 erratum: a TLB entry can survive an invalidation that races
+ * with concurrent traffic targeting the same entry. The software
+ * workaround is to issue every CFGI/TLBI command twice, each followed
+ * by CMD_SYNC. The second issue is guaranteed to evict the entry.
+ * ATC_INV commands are not affected and must not be doubled.
+ */
+#define ARM_SMMU_OPT_TLBI_TWICE		(1 << 5)

nit: I think this should be named slightly differently as it covers CFGI as well. Maybe ARM_SMMU_OPT_REPEAT_TLBI_CFGI ?

The comment can be simpler too and avoid being specific to Tegra264. The main things to say are that it repeats {CFGI,TLBI}; SYNC sequences and does not apply to ATC_INV.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1226

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5 -0
+/*
+ * Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered
+ * by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected
+ * SMMU instance.
+ */

(remove the comment)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1228

Quoted 2 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +2 -0
+static inline bool arm_smmu_cmd_needs_tlbi_twice(struct arm_smmu_device *smmu,
+						 struct arm_smmu_cmd *cmd)

Rename the function to something like arm_smmu_erratum_cmd_needs_repeating()?

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231

Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -0
+{
+	if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+		return false;

Maybe we should make this a static key?

Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6 -0
+	switch (FIELD_GET(CMDQ_0_OP, cmd->data[0])) {
+	case CMDQ_OP_CFGI_STE:
+	case CMDQ_OP_CFGI_ALL:
+	case CMDQ_OP_CFGI_CD:
+	case CMDQ_OP_CFGI_CD_ALL:
+	case CMDQ_OP_TLBI_NH_ALL:

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1248

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10 -0
+	case CMDQ_OP_TLBI_NH_ASID:
+	case CMDQ_OP_TLBI_NH_VA:
+	case CMDQ_OP_TLBI_NH_VAA:
+	case CMDQ_OP_TLBI_EL2_ALL:
+	case CMDQ_OP_TLBI_EL2_ASID:
+	case CMDQ_OP_TLBI_EL2_VA:
+	case CMDQ_OP_TLBI_S12_VMALL:
+	case CMDQ_OP_TLBI_S2_IPA:
+	case CMDQ_OP_TLBI_NSNH_ALL:
+		return true;

Isn't this just everything < ATC_INV || >= CFGI_STE? Seems better than enumerating everything.

Will

#7 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Quoted 5 lines

That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.

That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.

What would you like us to do here for the consistency?

Thanks
Nicolin

maintainer
#8 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 20 lines · Nicolin Chen
Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Quoted 5 lines

That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.

Quoted 6 lines

That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.

What would you like us to do here for the consistency?

Gah, I now realise I've mixed up Tegra264 and Tegra241 because tegra_cmdqv_dt_probe() is only called if the compatible string matches "nvidia,tegra264-smmu" yet all the cmdqv stuff talks only about Tegra241. So I was surprised that the ACPI probing code for Tegra241 wasn't enabling the workaround.

But if you're saying that:

1. Tegra264 never uses ACPI
2. Tegra241 doesn't have the invalidation erratum

Then I'm less worried (even though it's a shame that we can't detect the erratum from the hardware itself).

Will

#9 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 26 lines · Will Deacon
Quoted 25 lines · Nicolin Chen
Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Quoted 5 lines

That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.

That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.

Quoted 7 lines
Quoted 1 line

What would you like us to do here for the consistency?

Gah, I now realise I've mixed up Tegra264 and Tegra241 because tegra_cmdqv_dt_probe() is only called if the compatible string matches "nvidia,tegra264-smmu" yet all the cmdqv stuff talks only about Tegra241. So I was surprised that the ACPI probing code for Tegra241 wasn't enabling the workaround.

I see. Numbers could be indeed confusing :)

Quoted 4 lines

But if you're saying that:

1. Tegra264 never uses ACPI
2. Tegra241 doesn't have the invalidation erratum

Yes.

Quoted 2 lines

Then I'm less worried (even though it's a shame that we can't detect the erratum from the hardware itself).

Agreed, though I think the compatible string is already a life saver as we wouldn't need to go through the whole ACPI route..

Nicolin

#10 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

In reply to Will Deacon
Quoted 73 lines · Will Deacon

External email: Use caution opening links or attachments

Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.

Quoted 39 lines

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.

No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.

---
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  4 ++-
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++
  2 files changed, 43 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
index 4d29bd343460..08684bd40a6d 100644
@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
       if (of_dma_is_coherent(dev->of_node))
               smmu->features |= ARM_SMMU_FEAT_COHERENCY;
 
-     if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu"))
+     if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu")) {
               tegra_cmdqv_dt_probe(dev->of_node, smmu);
+             smmu->options |= ARM_SMMU_OPT_TLBI_TWICE;
+     }
 
       return ret;
  }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -0
index 16353596e08a..106034c348a1 100644
@@ -928,6 +928,14 @@ struct arm_smmu_device {
  #define ARM_SMMU_OPT_MSIPOLL         (1 << 2)

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_device { · line 938

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
  #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)
  #define ARM_SMMU_OPT_TEGRA241_CMDQV  (1 << 4)
+/*
+ * Tegra264 erratum: a TLB entry can survive an invalidation that races
+ * with concurrent traffic targeting the same entry. The software
+ * workaround is to issue every CFGI/TLBI command twice, each followed
+ * by CMD_SYNC. The second issue is guaranteed to evict the entry.
+ * ATC_INV commands are not affected and must not be doubled.
+ */
+#define ARM_SMMU_OPT_TLBI_TWICE              (1 << 5)

nit: I think this should be named slightly differently as it covers CFGI as well. Maybe ARM_SMMU_OPT_REPEAT_TLBI_CFGI ?

Quoted 3 lines

The comment can be simpler too and avoid being specific to Tegra264. The main things to say are that it repeats {CFGI,TLBI}; SYNC sequences and does not apply to ATC_INV.

Ack, I will fix this in V4.

Quoted 6 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1226

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5 -0
+/*
+ * Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered
+ * by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected
+ * SMMU instance.
+ */

(remove the comment)

Ack.

Quoted 3 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1228

Quoted 2 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +2 -0
+static inline bool arm_smmu_cmd_needs_tlbi_twice(struct arm_smmu_device *smmu,
+                                              struct arm_smmu_cmd *cmd)

Rename the function to something like arm_smmu_erratum_cmd_needs_repeating()?

Ack.

Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231

Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -0
+{
+     if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+             return false;

Maybe we should make this a static key?

Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that static_branch_unlikely will precede the option bit check?

Quoted 16 lines
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6 -0
+     switch (FIELD_GET(CMDQ_0_OP, cmd->data[0])) {
+     case CMDQ_OP_CFGI_STE:
+     case CMDQ_OP_CFGI_ALL:
+     case CMDQ_OP_CFGI_CD:
+     case CMDQ_OP_CFGI_CD_ALL:
+     case CMDQ_OP_TLBI_NH_ALL:

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1248

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10 -0
+     case CMDQ_OP_TLBI_NH_ASID:
+     case CMDQ_OP_TLBI_NH_VA:
+     case CMDQ_OP_TLBI_NH_VAA:
+     case CMDQ_OP_TLBI_EL2_ALL:
+     case CMDQ_OP_TLBI_EL2_ASID:
+     case CMDQ_OP_TLBI_EL2_VA:
+     case CMDQ_OP_TLBI_S12_VMALL:
+     case CMDQ_OP_TLBI_S2_IPA:
+     case CMDQ_OP_TLBI_NSNH_ALL:
+             return true;
Quoted 2 lines

Isn't this just everything < ATC_INV || >= CFGI_STE? Seems better than enumerating everything.

Ack.

Quoted 1 line

Will

Thanks,
Ashish Mhetre

#11 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 8 lines · Ashish Mhetre
Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231

Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -0
+{
+     if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+             return false;

Maybe we should make this a static key?

Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that static_branch_unlikely will precede the option bit check?

You'd have the static key and the options. Keep it simple, enable the static key once if any driver probes to set TWICE. Check the key before options to get the best code gen

But IDK if it is really worth it, there are already lots of branches on the performance tlbi flow, and we didn't do this for other tlbi affecting errata..

IDK if we really care about branches we should also be doing things like disabling the range/non-range paths and ATC based on what is actually in use..

Jason

#12 lore raw

Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 7 lines · Jason Gunthorpe

External email: Use caution opening links or attachments

Quoted 3 lines · Ashish Mhetre
Quoted 3 lines
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -0
+{
+     if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+             return false;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Maybe we should make this a static key?

Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that static_branch_unlikely will precede the option bit check?

You'd have the static key and the options. Keep it simple, enable the static key once if any driver probes to set TWICE. Check the key before options to get the best code gen

Okay, I'll incorporate this in V4 and send.

Quoted 9 lines

But IDK if it is really worth it, there are already lots of branches on the performance tlbi flow, and we didn't do this for other tlbi affecting errata..

IDK if we really care about branches we should also be doing things like disabling the range/non-range paths and ATC based on what is actually in use..

Jason

#13 lore raw

PATCH v3 3/3 iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Ashish Mhetre

Apply the workaround for Tegra264 erratum ARM_SMMU_OPT_TLBI_TWICE by issuing every CFGI/TLBI cmdlist twice on affected SMMU instances, with CMD_SYNC after each. The erratum requires this exact sequencing:

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

Rename the existing arm_smmu_cmdq_issue_cmdlist() to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time when arm_smmu_cmd_needs_tlbi_twice() is true.

For the in-tree batching path, register the Tegra264 condition with arm_smmu_cmdq_batch_force_sync() so that a full batch carrying CFGI/TLBI commands flushes with sync=true.

For iommufd VSMMU path add an arm_vsmmu_can_batch_cmd() predicate that splits the iommufd batch at cmd which doesn't need doubling.

---
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 23 +++++++++++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 36 ++++++++++++++++---
 2 files changed, 54 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +22 -1
index 1e9f7d2de344..78c96a2b652b 100644
@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
 	return 0;
 }
 
+/*
+ * On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI
+ * submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is
+ * taken once per cmdlist based on the first command, so a single
+ * batch must not mix commands that need doubling with commands that
+ * do not. Split the iommufd batch whenever the next user command
+ * crosses that boundary.
+ */
+static bool arm_vsmmu_can_batch_cmd(struct arm_smmu_device *smmu,
+				    struct arm_vsmmu_invalidation_cmd *last,
+				    struct arm_vsmmu_invalidation_cmd *next)
+{
+	struct arm_smmu_cmd next_cmd = {
+		.data[0] = le64_to_cpu(next->ucmd.cmd[0]),
+	};
+
+	return arm_smmu_cmd_needs_tlbi_twice(smmu, &last->cmd) ==
+	       arm_smmu_cmd_needs_tlbi_twice(smmu, &next_cmd);
+}
+
 int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 			       struct iommu_user_data_array *array)
 {
@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
 		/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
 		cur++;
-		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+		    arm_vsmmu_can_batch_cmd(smmu, last, cur))
 			continue;
 
 		/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +32 -4
index 08684bd40a6d..f38c21b56f28 100644
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,
  *   insert their own list of commands then all of the commands from one
  *   CPU will appear before any of the commands from the other CPU.
  */
-int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
-				struct arm_smmu_cmdq *cmdq,
-				struct arm_smmu_cmd *cmds, int n,
-				bool sync)
+static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+					 struct arm_smmu_cmdq *cmdq,
+					 struct arm_smmu_cmd *cmds, int n,
+					 bool sync)
 {
 	struct arm_smmu_cmd cmd_sync;
 	u32 prod;
@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 	return ret;
 }
 
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+				struct arm_smmu_cmdq *cmdq,
+				struct arm_smmu_cmd *cmds, int n,
+				bool sync)
+{
+	int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	/*
+	 * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+	 * cmdlist with another CMD_SYNC to satisfy the erratum.
+	 * Callers must ensure the batch carries a uniform opcode class
+	 * so that checking the first command is enough; the iommufd
+	 * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+	 */
+	if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	return ret;
+}
+
 static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
 				     struct arm_smmu_cmd *cmd, bool sync)
 {
@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
 	    (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))
 		return true;
 
+	/*
+	 * Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds
+	 * a uniform opcode class, so checking the first command is enough.
+	 */
+	if (cmds->num == CMDQ_BATCH_ENTRIES &&
+	    arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds->cmds[0]))
+		return true;
+
 	return false;
 }
 
Signature · 1 line
2.50.1
Reviewed-by on v3 3/3
#14 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 15 lines · Ashish Mhetre

Apply the workaround for Tegra264 erratum ARM_SMMU_OPT_TLBI_TWICE by issuing every CFGI/TLBI cmdlist twice on affected SMMU instances, with CMD_SYNC after each. The erratum requires this exact sequencing:

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

Rename the existing arm_smmu_cmdq_issue_cmdlist() to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time when arm_smmu_cmd_needs_tlbi_twice() is true.

For the in-tree batching path, register the Tegra264 condition with arm_smmu_cmdq_batch_force_sync() so that a full batch carrying CFGI/TLBI commands flushes with sync=true.

Quoted 5 lines

For iommufd VSMMU path add an arm_vsmmu_can_batch_cmd() predicate that splits the iommufd batch at cmd which doesn't need doubling.

maintainer
#15 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Ashish Mhetre
Quoted 6 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0 -0
index 1e9f7d2de344..78c96a2b652b 100644
@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
 	return 0;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 360

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8 -0
 }
 
+/*
+ * On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI
+ * submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is
+ * taken once per cmdlist based on the first command, so a single
+ * batch must not mix commands that need doubling with commands that
+ * do not. Split the iommufd batch whenever the next user command
+ * crosses that boundary.
+ */

Again, I wouldn't bother with this comment. You probably _should_ update Documentation/arch/arm64/silicon-errata.rst, however.

Quoted 16 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +12 -0
+static bool arm_vsmmu_can_batch_cmd(struct arm_smmu_device *smmu,
+				    struct arm_vsmmu_invalidation_cmd *last,
+				    struct arm_vsmmu_invalidation_cmd *next)
+{
+	struct arm_smmu_cmd next_cmd = {
+		.data[0] = le64_to_cpu(next->ucmd.cmd[0]),
+	};
+
+	return arm_smmu_cmd_needs_tlbi_twice(smmu, &last->cmd) ==
+	       arm_smmu_cmd_needs_tlbi_twice(smmu, &next_cmd);
+}
+
 int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 			       struct iommu_user_data_array *array)
 {

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2 -1
@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
 		/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
 		cur++;
-		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+		    arm_vsmmu_can_batch_cmd(smmu, last, cur))
 			continue;

FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:

https://sashiko.dev/#/patchset/[email protected]

Quoted 29 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0 -0
 		/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +9 -4
index 08684bd40a6d..f38c21b56f28 100644
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,
  *   insert their own list of commands then all of the commands from one
  *   CPU will appear before any of the commands from the other CPU.
  */
-int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
-				struct arm_smmu_cmdq *cmdq,
-				struct arm_smmu_cmd *cmds, int n,
-				bool sync)
+static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+					 struct arm_smmu_cmdq *cmdq,
+					 struct arm_smmu_cmd *cmds, int n,
+					 bool sync)
 {
 	struct arm_smmu_cmd cmd_sync;
 	u32 prod;
@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 	return ret;
 }
 
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+				struct arm_smmu_cmdq *cmdq,
+				struct arm_smmu_cmd *cmds, int n,
+				bool sync)
+{

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837

Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10 -0
+	int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	/*
+	 * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+	 * cmdlist with another CMD_SYNC to satisfy the erratum.
+	 * Callers must ensure the batch carries a uniform opcode class
+	 * so that checking the first command is enough; the iommufd
+	 * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+	 */
+	if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))

Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838

Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?

Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+
+	return ret;
+}
+
 static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
 				     struct arm_smmu_cmd *cmd, bool sync)
 {

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 889

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
 	    (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))
 		return true;
 
+	/*
+	 * Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds
+	 * a uniform opcode class, so checking the first command is enough.
+	 */

Again, please drop the Tegra264 mention and just refer to the option.

Will

#16 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 13 lines · Will Deacon

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407

Quoted 8 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2 -1
@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
 		/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
 		cur++;
-		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+		    arm_vsmmu_can_batch_cmd(smmu, last, cur))
 			continue;

FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:

Quoted 1 line

https://sashiko.dev/#/patchset/[email protected]

Ack. I will submit a separate patch to address this.

Nicolin

#17 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Will Deacon
Quoted 5 lines · Will Deacon

be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?

There was an earlier version trying to do this and it was pretty ugly, there are alot of places because this errata touches the CD/STE invalidation flows too.

If it was just invalidation it could have been solved by manipulating the invs list and be almost free on the fast path..

Jason

#18 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Will Deacon
Quoted 13 lines · Will Deacon

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837

Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -0
+	/*
+	 * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+	 * cmdlist with another CMD_SYNC to satisfy the erratum.
+	 * Callers must ensure the batch carries a uniform opcode class
+	 * so that checking the first command is enough; the iommufd
+	 * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+	 */
+	if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))

Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.

Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838

Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.

That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..

Jason

maintainer
#19 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 14 lines · Jason Gunthorpe
Quoted 13 lines · Will Deacon

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837

Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -0
+	/*
+	 * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+	 * cmdlist with another CMD_SYNC to satisfy the erratum.
+	 * Callers must ensure the batch carries a uniform opcode class
+	 * so that checking the first command is enough; the iommufd
+	 * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+	 */
+	if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))

Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.

Quoted 8 lines
Quoted 4 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838

Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.

That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..

I was wondering about the case where the first command in a batch is unsupported. Can we issue a bare sync in that case?

Will

#20 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 17 lines · Will Deacon

External email: Use caution opening links or attachments

Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Will Deacon

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837

Quoted 8 lines

+ /*

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7 -0
+  * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+  * cmdlist with another CMD_SYNC to satisfy the erratum.
+  * Callers must ensure the batch carries a uniform opcode class
+  * so that checking the first command is enough; the iommufd
+  * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+  */
+ if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))

Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

+ ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.

That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..

I was wondering about the case where the first command in a batch is unsupported. Can we issue a bare sync in that case?

Seems a valid case. But considering the static branch approach, will
it be fine to add "n != 0" check before arm_smmu_cmd_needs_tlbi_twice() ? Something like: int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync); if(!n) return ret; if (arm_smmu_erratum_cmd_needs_repeating(smmu, &cmds[0]) && !ret && sync) ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Quoted 1 line

Will

Thanks,
Ashish Mhetre

#21 lore raw

Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Will Deacon
Quoted 12 lines · Will Deacon

External email: Use caution opening links or attachments

Quoted 8 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0 -0
index 1e9f7d2de344..78c96a2b652b 100644
@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
       return 0;
  }
 
Quoted 10 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 360

Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8 -0
+/*
+ * On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI
+ * submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is
+ * taken once per cmdlist based on the first command, so a single
+ * batch must not mix commands that need doubling with commands that
+ * do not. Split the iommufd batch whenever the next user command
+ * crosses that boundary.
+ */

Again, I wouldn't bother with this comment. You probably _should_ update Documentation/arch/arm64/silicon-errata.rst, however.

Ack, I'll remove this comment and add in Documentation.

Quoted 62 lines
Quoted 16 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +12 -0
+static bool arm_vsmmu_can_batch_cmd(struct arm_smmu_device *smmu,
+                                 struct arm_vsmmu_invalidation_cmd *last,
+                                 struct arm_vsmmu_invalidation_cmd *next)
+{
+     struct arm_smmu_cmd next_cmd = {
+             .data[0] = le64_to_cpu(next->ucmd.cmd[0]),
+     };
+
+     return arm_smmu_cmd_needs_tlbi_twice(smmu, &last->cmd) ==
+            arm_smmu_cmd_needs_tlbi_twice(smmu, &next_cmd);
+}
+
  int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
                              struct iommu_user_data_array *array)
  {

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407

Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2 -1
@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
               /* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
               cur++;
-             if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+             if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+                 arm_vsmmu_can_batch_cmd(smmu, last, cur))
                       continue;

FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:

https://sashiko.dev/#/patchset/[email protected]

Quoted 34 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0 -0
               /* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +14 -4
index 08684bd40a6d..f38c21b56f28 100644
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,
   *   insert their own list of commands then all of the commands from one
   *   CPU will appear before any of the commands from the other CPU.
   */
-int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
-                             struct arm_smmu_cmdq *cmdq,
-                             struct arm_smmu_cmd *cmds, int n,
-                             bool sync)
+static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+                                      struct arm_smmu_cmdq *cmdq,
+                                      struct arm_smmu_cmd *cmds, int n,
+                                      bool sync)
  {
       struct arm_smmu_cmd cmd_sync;
       u32 prod;
@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
       return ret;
  }
 
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+                             struct arm_smmu_cmdq *cmdq,
+                             struct arm_smmu_cmd *cmds, int n,
+                             bool sync)
+{
+     int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+     /*
+      * On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same
+      * cmdlist with another CMD_SYNC to satisfy the erratum.
Quoted 8 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +5 -0
+      * Callers must ensure the batch carries a uniform opcode class
+      * so that checking the first command is enough; the iommufd
+      * VSMMU path enforces this with arm_vsmmu_can_batch_cmd().
+      */
+     if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))

Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.

Okay, I'll fix this in V4.

Quoted 19 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838

Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
+             ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?

Quoted 11 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+
+     return ret;
+}
+
  static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
                                    struct arm_smmu_cmd *cmd, bool sync)
  {
@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
           (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))
               return true;
 
Quoted 5 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 889

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -0
+     /*
+      * Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds
+      * a uniform opcode class, so checking the first command is enough.
+      */

Again, please drop the Tegra264 mention and just refer to the option.

Ack.

Quoted 1 line

Will

#22 lore raw
In reply to Ashish Mhetre
Quoted 6 lines · Ashish Mhetre

Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:

This seems quite intrusive, will the TLB entry survive if you push a full invalidation instead?

Thanks,
Mostafa

Quoted 73 lines

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

ATC_INV is not affected and must not be doubled.

This series implements the workaround by hooking the duplication into arm_smmu_cmdq_issue_cmdlist(), the single chokepoint that every synchronous CMDQ submission flows through.

Patch 1 is a preparatory refactor that factors the existing batch force-sync conditions out of arm_smmu_cmdq_batch_add_cmd_p() into a new arm_smmu_cmdq_batch_force_sync() helper. No functional change.

Patch 2 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string, exposes the condition via a new ARM_SMMU_OPT_TLBI_TWICE option bit, and adds a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that both the in-tree CMDQ path and the iommufd VSMMU path can share a single predicate.

Patch 3 wires the workaround in. arm_smmu_cmdq_issue_cmdlist() becomes a thin wrapper that re-issues a synced cmdlist a second time when the first command needs doubling. The Tegra264 condition is added to arm_smmu_cmdq_batch_force_sync() so a full batch carrying CFGI/TLBI commands flushes with sync=true and is then doubled. The iommufd VSMMU path (arm_vsmmu_cache_invalidate()) is also taught to split the user-supplied batch at every "needs doubling" / "doesn't need doubling" transition via a new arm_vsmmu_can_batch_cmd() predicate, since that path can otherwise mix CFGI/TLBI with ATC_INV in a single submission.

The series is based on Jason Gunthorpe's "Remove SMMUv3 struct arm_smmu_cmdq_ent" series [1], specifically commit 13428b0bf794 ("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.

[1] https://lore.kernel.org/all/[email protected]/

Changes since v2:
- Add new prep patch 1/3 that factors the existing force-sync conditions into arm_smmu_cmdq_batch_force_sync() (from Nicolin).
- Move arm_smmu_cmd_needs_tlbi_twice() to arm-smmu-v3.h as static inline taking (smmu, cmd*) and folding in the option check.
- Plug the Tegra264 condition into arm_smmu_cmdq_batch_force_sync() instead of carrying a separate need_sync in batch_add_cmd_p().
- Fix iommufd batching: arm_vsmmu_cache_invalidate() can mix CFGI/TLBI with ATC_INV in one batch. Split at the boundary via a new arm_vsmmu_can_batch_cmd() predicate.
- Patch 2 wording: "next patch wires" -> "a subsequent change will wire".

v2: https://lore.kernel.org/all/[email protected]/

Nicolin Chen (1):
iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions

Ashish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 23 ++++++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 66 +++++++++++++++----
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   | 40 +++++++++++
 3 files changed, 117 insertions(+), 12 deletions(-)

base-commit: 13428b0bf7947098daf9a1db14a74d33eb1b5079

Signature · 1 line
2.50.1
#23 lore raw
Quoted 10 lines · Mostafa Saleh
Quoted 6 lines · Ashish Mhetre

Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:

This seems quite intrusive, will the TLB entry survive if you push a full invalidation instead?

It's 36 lines and completely contained to the insides of the command sumbissions code??

Stuff like this is why I was guiding you to use the more code exactly as is for pkvm. Historically there have been many invalidation related errata, and invalidate twice seems to be a theme in fixing many of them.

Jason