Skip to contentSkip to conversation

iommu/arm-smmu-v3: Tegra264 invalidation workaround

PATCH Superseded Ashish Mhetre 2 participants 7 messages May 29 – Jun 1

Subsystems (3)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover iommu/arm-smmu-v3: Tegra264 invalidation workaround

Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

ATC_INV is not affected and must not be doubled.

This series implements the workaround by hooking the duplication into the single chokepoint that every synchronous submission flows through, arm_smmu_cmdq_issue_cmdlist().

Patch 1 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string and exposes the condition via a new ARM_SMMU_OPT_TLBI_TWICE option bit.

Patch 2 wires the option into the CMDQ submission path: when @sync is true and the cmdlist carries a CFGI/TLBI opcode, the same cmdlist is re-issued a second time. The batch capacity-rollover path is also adjusted to force a SYNC on chunks that carry CFGI/TLBI commands so each flushed chunk is correctly doubled.

The series is based on Jason Gunthorpe's "Remove SMMUv3 struct arm_smmu_cmdq_ent" series [1], specifically commit 13428b0bf794 ("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.

[1] https://lore.kernel.org/all/[email protected]/

Changes since v1:
- Patch 1: Add IIDR/IDR/ACPI rationale to the commit message, explaining why the erratum is detected from the device tree compatible string rather than from a HW register or ACPI/IORT.
- Rebased onto the publicly accessible base 13428b0bf794 (final commit of Jason's series in linux-next) so that the base-commit is resolvable on lore.kernel.org.
- Patch 2: Picked up Reviewed-by: Jason Gunthorpe <[email protected]>.
- No code changes since v1.

Ashish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 70 +++++++++++++++++++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  8 +++
 2 files changed, 72 insertions(+), 6 deletions(-)

base-commit: 13428b0bf7947098daf9a1db14a74d33eb1b5079

Signature · 1 line
2.50.1
#1 lore raw

PATCH v2 1/2 iommu/arm-smmu-v3: Detect Tegra264 erratum

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 +++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 ++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
index 9be589d14a3b..88296c0a5337 100644
@@ -5229,8 +5229,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
 	if (of_dma_is_coherent(dev->of_node))
 		smmu->features |= ARM_SMMU_FEAT_COHERENCY;
 
-	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu"))
+	if (of_device_is_compatible(dev->of_node, "nvidia,tegra264-smmu")) {
 		tegra_cmdqv_dt_probe(dev->of_node, smmu);
+		smmu->options |= ARM_SMMU_OPT_TLBI_TWICE;
+	}
 
 	return ret;
 }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
index 16353596e08a..08d1abaf31ae 100644
@@ -928,6 +928,14 @@ struct arm_smmu_device {
 #define ARM_SMMU_OPT_MSIPOLL		(1 << 2)
 #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC	(1 << 3)
 #define ARM_SMMU_OPT_TEGRA241_CMDQV	(1 << 4)
+/*
+ * Tegra264 erratum: a TLB entry can survive an invalidation that races
+ * with concurrent traffic targeting the same entry. The software
+ * workaround is to issue every CFGI/TLBI command twice, each followed
+ * by CMD_SYNC. The second issue is guaranteed to evict the entry.
+ * ATC_INV commands are not affected and must not be doubled.
+ */
+#define ARM_SMMU_OPT_TLBI_TWICE		(1 << 5)
 	u32				options;
 
 	struct arm_smmu_cmdq		cmdq;
Signature · 1 line
2.50.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 7 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

Quoted 7 lines

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.

I was told to avoid "patch": once a patch is applied it becomes a commit. So, maybe "a subsequent change will wire".

Quoted 2 lines
#3 lore raw

Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Detect Tegra264 erratum

Quoted 12 lines · Nicolin Chen
Quoted 11 lines · Ashish Mhetre

Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.

The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.

Quoted 5 lines
Quoted 3 lines

Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.

I was told to avoid "patch": once a patch is applied it becomes a commit. So, maybe "a subsequent change will wire".

Sure, I'll update it in next version.

Quoted 2 lines
Quoted 1 line

Thanks Nicolin.

#4 lore raw

PATCH v2 2/2 iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

In reply to Ashish Mhetre

Apply the workaround for Tegra264 erratum by issuing every CFGI/TLBI command twice on affected SMMU instances, with CMD_SYNC after each. The erratum requires this exact sequencing:

TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC

To get this sequence with minimal surgery, hook the workaround into arm_smmu_cmdq_issue_cmdlist(). Rename the original function to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time.

A new arm_smmu_cmd_needs_tlbi_twice() helper classifies which opcodes need the doubling: CFGI_* and TLBI_*.

For batches that exceed CMDQ_BATCH_ENTRIES commands, arm_smmu_cmdq_batch_add_cmd_p() normally flushes the full buffer with sync=false, deferring the SYNC to the eventual batch_submit(). On affected SMMUs this would leave the first chunk's commands issued only once, since the WAR hook in arm_smmu_cmdq_issue_cmdlist() only fires on synced submissions. Force a SYNC on the capacity rollover when the buffer carries CFGI/TLBI commands so every flushed chunk is correctly doubled.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 +++++++++++++++++++--
 1 file changed, 61 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +61 -5
index 88296c0a5337..38d45f175a2c 100644
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,
  *   insert their own list of commands then all of the commands from one
  *   CPU will appear before any of the commands from the other CPU.
  */
-int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
-				struct arm_smmu_cmdq *cmdq,
-				struct arm_smmu_cmd *cmds, int n,
-				bool sync)
+static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+					 struct arm_smmu_cmdq *cmdq,
+					 struct arm_smmu_cmd *cmds, int n,
+					 bool sync)
 {
 	struct arm_smmu_cmd cmd_sync;
 	u32 prod;
@@ -820,6 +820,52 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 	return ret;
 }
 
+/*
+ * Returns true if @opcode is a CFGI_* or TLBI_* command, i.e. one of the
+ * invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE).
+ */
+static bool arm_smmu_cmd_needs_tlbi_twice(u8 opcode)
+{
+	switch (opcode) {
+	case CMDQ_OP_CFGI_STE:
+	case CMDQ_OP_CFGI_ALL:
+	case CMDQ_OP_CFGI_CD:
+	case CMDQ_OP_CFGI_CD_ALL:
+	case CMDQ_OP_TLBI_NH_ALL:
+	case CMDQ_OP_TLBI_NH_ASID:
+	case CMDQ_OP_TLBI_NH_VA:
+	case CMDQ_OP_TLBI_NH_VAA:
+	case CMDQ_OP_TLBI_EL2_ALL:
+	case CMDQ_OP_TLBI_EL2_ASID:
+	case CMDQ_OP_TLBI_EL2_VA:
+	case CMDQ_OP_TLBI_S12_VMALL:
+	case CMDQ_OP_TLBI_S2_IPA:
+	case CMDQ_OP_TLBI_NSNH_ALL:
+		return true;
+	default:
+		return false;
+	}
+}
+
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+				struct arm_smmu_cmdq *cmdq,
+				struct arm_smmu_cmd *cmds, int n,
+				bool sync)
+{
+	int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	/*
+	 * The driver's batch invariants keep a single submission's
+	 * opcode class uniform, so checking the first command is enough.
+	 */
+	if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+	    arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+						    cmds[0].data[0])))
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	return ret;
+}
+
 static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
 				     struct arm_smmu_cmd *cmd, bool sync)
 {
@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
 	}
 
 	if (cmds->num == CMDQ_BATCH_ENTRIES) {
+		/*
+		 * Force a SYNC only when the batch carries commands that
+		 * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE).
+		 * The batch holds a uniform opcode class, so checking
+		 * the first command is sufficient.
+		 */
+		bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+				 arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+									 cmds->cmds[0].data[0]));
+
 		arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
-					    cmds->num, false);
+					    cmds->num, need_sync);
 		arm_smmu_cmdq_batch_init_cmd(smmu, cmds, cmd);
 	}
 
Signature · 1 line
2.50.1
#5 lore raw

Re: [PATCH v2 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 5 lines · Ashish Mhetre
(quoted diff) +5 -0
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+				struct arm_smmu_cmdq *cmdq,
+				struct arm_smmu_cmd *cmds, int n,
+				bool sync)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+	int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	/*
+	 * The driver's batch invariants keep a single submission's
+	 * opcode class uniform, so checking the first command is enough.
+	 */
+	if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+	    arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+						    cmds[0].data[0])))
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

https://sashiko.dev/#/patchset/20260529140830.629738-1-amhetre%40nvidia.com
Sashiko pointed out that the iommufd path might mix commands when calling arm_smmu_cmdq_issue_cmdlist(), which is valid I think.

Quoted 7 lines

static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,

(quoted diff) +0 -0
 				     struct arm_smmu_cmd *cmd, bool sync)
 {
@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
 	}
 
 	if (cmds->num == CMDQ_BATCH_ENTRIES) {

Review static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, · line 921

Quoted 10 lines
(quoted diff) +10 -0
+		/*
+		 * Force a SYNC only when the batch carries commands that
+		 * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE).
+		 * The batch holds a uniform opcode class, so checking
+		 * the first command is sufficient.
+		 */
+		bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+				 arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+									 cmds->cmds[0].data[0]));
+

Also, given that this does "force a sync", I think it might be nicer to go to the force_sync path. One of my ongoing series also needs to add another force_sync condition, so I think it would be cleaner to have a helper function.

Maybe try the following diff. That arm_smmu_cmdq_batch_force_sync() might be added with a preparatory patch, but it's up to you.

--------------------------------------------------------------------

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +14 -1
index 1e9f7d2de3441..4c9ce974d31a8 100644
@@ -350,6 +350,18 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
 	return 0;
 }
 
+static bool arm_vsmmu_can_batch_cmd(struct arm_smmu_device *smmu,
+				    struct arm_vsmmu_invalidation_cmd *last,
+				    struct arm_vsmmu_invalidation_cmd *next)
+{
+	struct arm_smmu_cmd next_cmd = {
+		.data[0] = le64_to_cpu(next->ucmd.cmd[0]),
+	};
+
+	return arm_smmu_cmd_needs_tlbi_twice(smmu, &last->cmd) ==
+	       arm_smmu_cmd_needs_tlbi_twice(smmu, &next_cmd);
+}
+
 int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 			       struct iommu_user_data_array *array)
 {
@@ -382,7 +394,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
 		/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
 		cur++;
-		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+		    arm_vsmmu_can_batch_cmd(smmu, last, cur))
 			continue;
 
 		/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +31 -46

107 lines not on this page · raw message

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +31 -0
index 08d1abaf31ae2..e6afc832c0078 100644
@@ -1219,6 +1219,37 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 				struct arm_smmu_cmd *cmds, int n,
 				bool sync);
 
+/*
+ * Returns true if @cmd opcode is a CFGI_* or TLBI_* command, i.e. one of the
+ * invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE).
+ */
+static inline bool arm_smmu_cmd_needs_tlbi_twice(struct arm_smmu_device *smmu,
+						 struct arm_smmu_cmd *cmd)
+{
+	if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+		return false;
+
+	switch (FIELD_GET(CMDQ_0_OP, cmd->data[0])) {
+	case CMDQ_OP_CFGI_STE:
+	case CMDQ_OP_CFGI_ALL:
+	case CMDQ_OP_CFGI_CD:
+	case CMDQ_OP_CFGI_CD_ALL:
+	case CMDQ_OP_TLBI_NH_ALL:
+	case CMDQ_OP_TLBI_NH_ASID:
+	case CMDQ_OP_TLBI_NH_VA:
+	case CMDQ_OP_TLBI_NH_VAA:
+	case CMDQ_OP_TLBI_EL2_ALL:
+	case CMDQ_OP_TLBI_EL2_ASID:
+	case CMDQ_OP_TLBI_EL2_VA:
+	case CMDQ_OP_TLBI_S12_VMALL:
+	case CMDQ_OP_TLBI_S2_IPA:
+	case CMDQ_OP_TLBI_NSNH_ALL:
+		return true;
+	default:
+		return false;
+	}
+}
+
 #ifdef CONFIG_ARM_SMMU_V3_SVA
 bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
 void arm_smmu_sva_notifier_synchronize(void);

--------------------------------------------------------------------

Nicolinc

#6 lore raw

Re: [PATCH v2 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264

Quoted 11 lines · Nicolin Chen
Quoted 10 lines · Ashish Mhetre
(quoted diff) +10 -0
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+				struct arm_smmu_cmdq *cmdq,
+				struct arm_smmu_cmd *cmds, int n,
+				bool sync)
+{
+	int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+	/*
+	 * The driver's batch invariants keep a single submission's
+	 * opcode class uniform, so checking the first command is enough.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	 */
+	if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+	    arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+						    cmds[0].data[0])))
+		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);

https://sashiko.dev/#/patchset/20260529140830.629738-1-amhetre%40nvidia.com
Sashiko pointed out that the iommufd path might mix commands when calling arm_smmu_cmdq_issue_cmdlist(), which is valid I think.

Okay, I'll update the batching for iommufd path as suggested by you in next version.

Quoted 217 lines
Quoted 7 lines

static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
struct arm_smmu_cmd *cmd, bool sync)
{

(quoted diff) +0 -0
@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
  	}
 
  	if (cmds->num == CMDQ_BATCH_ENTRIES) {

Review static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, · line 921

Quoted 10 lines
(quoted diff) +10 -0
+		/*
+		 * Force a SYNC only when the batch carries commands that
+		 * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE).
+		 * The batch holds a uniform opcode class, so checking
+		 * the first command is sufficient.
+		 */
+		bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+				 arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+									 cmds->cmds[0].data[0]));
+

Also, given that this does "force a sync", I think it might be nicer to go to the force_sync path. One of my ongoing series also needs to add another force_sync condition, so I think it would be cleaner to have a helper function.

Maybe try the following diff. That arm_smmu_cmdq_batch_force_sync() might be added with a preparatory patch, but it's up to you.

--------------------------------------------------------------------

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +14 -1
index 1e9f7d2de3441..4c9ce974d31a8 100644
@@ -350,6 +350,18 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
  	return 0;
  }
 
+static bool arm_vsmmu_can_batch_cmd(struct arm_smmu_device *smmu,
+				    struct arm_vsmmu_invalidation_cmd *last,
+				    struct arm_vsmmu_invalidation_cmd *next)
+{
+	struct arm_smmu_cmd next_cmd = {
+		.data[0] = le64_to_cpu(next->ucmd.cmd[0]),
+	};
+
+	return arm_smmu_cmd_needs_tlbi_twice(smmu, &last->cmd) ==
+	       arm_smmu_cmd_needs_tlbi_twice(smmu, &next_cmd);
+}
+
  int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
  			       struct iommu_user_data_array *array)
  {
@@ -382,7 +394,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,
 
  		/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */
  		cur++;
-		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1)
+		if (cur != end && (cur - last) != CMDQ_BATCH_ENTRIES - 1 &&
+		    arm_vsmmu_can_batch_cmd(smmu, last, cur))
  			continue;
 
  		/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +31 -46
index a63155e9e7f28..9b150e3145034 100644
@@ -820,33 +820,6 @@ static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
  	return ret;
  }
 
-/*
- * Returns true if @opcode is a CFGI_* or TLBI_* command, i.e. one of the
- * invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE).
- */
-static bool arm_smmu_cmd_needs_tlbi_twice(u8 opcode)
-{
-	switch (opcode) {
-	case CMDQ_OP_CFGI_STE:
-	case CMDQ_OP_CFGI_ALL:
-	case CMDQ_OP_CFGI_CD:
-	case CMDQ_OP_CFGI_CD_ALL:
-	case CMDQ_OP_TLBI_NH_ALL:
-	case CMDQ_OP_TLBI_NH_ASID:
-	case CMDQ_OP_TLBI_NH_VA:
-	case CMDQ_OP_TLBI_NH_VAA:
-	case CMDQ_OP_TLBI_EL2_ALL:
-	case CMDQ_OP_TLBI_EL2_ASID:
-	case CMDQ_OP_TLBI_EL2_VA:
-	case CMDQ_OP_TLBI_S12_VMALL:
-	case CMDQ_OP_TLBI_S2_IPA:
-	case CMDQ_OP_TLBI_NSNH_ALL:
-		return true;
-	default:
-		return false;
-	}
-}
-
  int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
  				struct arm_smmu_cmdq *cmdq,
  				struct arm_smmu_cmd *cmds, int n,
@@ -858,9 +831,7 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
  	 * The driver's batch invariants keep a single submission's
  	 * opcode class uniform, so checking the first command is enough.
  	 */
-	if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
-	    arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
-						    cmds[0].data[0])))
+	if (!ret && sync && arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds[0]))
  		ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
 
  	return ret;
@@ -893,34 +864,48 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu,
  	cmds->cmdq = arm_smmu_get_cmdq(smmu, cmd);
  }
 
+static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu,
+					   struct arm_smmu_cmdq_batch *cmds,
+					   struct arm_smmu_cmd *cmd)
+{
+	if (!cmds->num)
+		return false;
+
+	/* The batch's pre-assigned cmdq doesn't support the new command */
+	if (!arm_smmu_cmdq_supports_cmd(cmds->cmdq, cmd))
+		return true;
+
+	/* Arm erratum 2812531 */
+	if (cmds->num == CMDQ_BATCH_ENTRIES - 1 &&
+	    (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))
+		return true;
+
+	/*
+	 * Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds a
+	 * uniform opcode class, so checking the first command is enough.
+	 */
+	if ((cmds->num == CMDQ_BATCH_ENTRIES) &&
+	    arm_smmu_cmd_needs_tlbi_twice(smmu, &cmds->cmds[0]))
+		return true;
+
+	return false;
+}
+
  static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
  					  struct arm_smmu_cmdq_batch *cmds,
  					  struct arm_smmu_cmd *cmd)
  {
-	bool force_sync = (cmds->num == CMDQ_BATCH_ENTRIES - 1) &&
-			  (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC);
-	bool unsupported_cmd;
+	bool force_sync = arm_smmu_cmdq_batch_force_sync(smmu, cmds, cmd);
 
-	unsupported_cmd = !arm_smmu_cmdq_supports_cmd(cmds->cmdq, cmd);
-	if (force_sync || unsupported_cmd) {
+	if (force_sync) {
  		arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
  					    cmds->num, true);
  		arm_smmu_cmdq_batch_init_cmd(smmu, cmds, cmd);
  	}
 
  	if (cmds->num == CMDQ_BATCH_ENTRIES) {
-		/*
-		 * Force a SYNC only when the batch carries commands that
-		 * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE).
-		 * The batch holds a uniform opcode class, so checking
-		 * the first command is sufficient.
-		 */
-		bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
-				 arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
-									 cmds->cmds[0].data[0]));
-
  		arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
-					    cmds->num, need_sync);
+					    cmds->num, false);
  		arm_smmu_cmdq_batch_init_cmd(smmu, cmds, cmd);
  	}
 
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +31 -1
index 08d1abaf31ae2..e6afc832c0078 100644
@@ -1219,6 +1219,37 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
  				struct arm_smmu_cmd *cmds, int n,
  				bool sync);
 
+/*
+ * Returns true if @cmd opcode is a CFGI_* or TLBI_* command, i.e. one of the
+ * invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE).
+ */
+static inline bool arm_smmu_cmd_needs_tlbi_twice(struct arm_smmu_device *smmu,
+						 struct arm_smmu_cmd *cmd)
+{
+	if (!(smmu->options & ARM_SMMU_OPT_TLBI_TWICE))
+		return false;
+
+	switch (FIELD_GET(CMDQ_0_OP, cmd->data[0])) {
+	case CMDQ_OP_CFGI_STE:
+	case CMDQ_OP_CFGI_ALL:
+	case CMDQ_OP_CFGI_CD:
+	case CMDQ_OP_CFGI_CD_ALL:
+	case CMDQ_OP_TLBI_NH_ALL:
+	case CMDQ_OP_TLBI_NH_ASID:
+	case CMDQ_OP_TLBI_NH_VA:
+	case CMDQ_OP_TLBI_NH_VAA:
+	case CMDQ_OP_TLBI_EL2_ALL:
+	case CMDQ_OP_TLBI_EL2_ASID:
+	case CMDQ_OP_TLBI_EL2_VA:
+	case CMDQ_OP_TLBI_S12_VMALL:
+	case CMDQ_OP_TLBI_S2_IPA:
+	case CMDQ_OP_TLBI_NSNH_ALL:
+		return true;
+	default:
+		return false;
+	}
+}
+
  #ifdef CONFIG_ARM_SMMU_V3_SVA
  bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
  void arm_smmu_sva_notifier_synchronize(void);
--------------------------------------------------------------------
Quoted 1 line

Nicolinc

Ack, I'll add force_sync patch by you as a preparatory patch and rebase my changes on top of it and send it in V3.

Thanks,
Ashish Mhetre