iommu/arm-smmu-v3: Tegra264 invalidation workaround
Subsystems (5)
v6 cover iommu/arm-smmu-v3: Tegra264 invalidation workaround
Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
ATC_INV is not affected and must not be doubled.
The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 is device-tree-only (no ACPI/IORT support), so detection is purely by compatible string.
This series is structured as a small refactor + infrastructure + enable sequence so that each step is reviewable in isolation:
1/3 Pure refactor (no functional change): lift the existing
force-sync conditions out of arm_smmu_cmdq_batch_add_cmd_p() into a new arm_smmu_cmdq_batch_force_sync() helper, so that adding another condition (in patch 2) is a one-line addition. Authored by Nicolin Chen.
2/3 Add the workaround infrastructure without enabling it. Defines
the file-local arm_smmu_erratum_repeat_tlbi_cfgi_key static key with an inline erratum description, the shared arm_smmu_erratum_cmd_needs_repeating() predicate, the arm_smmu_cmdq_issue_cmdlist() wrapper that can re-issue matching cmdlists, the batch-helper force-sync condition, and the iommufd batching split for mixed command classes.
3/3 Enable the workaround for the existing "nvidia,tegra264-smmu"
compatible and document the erratum in silicon-errata.rst.
The series applies cleanly on linux-next/master (base-commit below).
Changes since v5:
- Move arm_smmu_erratum_cmd_needs_repeating() into arm-smmu-v3.c and leave a declaration-only stub in arm-smmu-v3.h. Make arm_smmu_erratum_repeat_tlbi_cfgi_key file-local static and drop jump_label.h from the header.
- Add an inline erratum/workaround description at the static key, referenced from arm_smmu_cmdq_batch_force_sync().
- Drop the misleading !n comment above arm_smmu_cmdq_issue_cmdlist(); keep the defensive !n guard.
- Remove the unused smmu parameter from the predicate.
- Tweak 2/3 commit-message wording ("commit" vs "patch").
- Add Reviewed-by: Nicolin Chen on 3/3.
Changes since v4:
- Drop ARM_SMMU_OPT_REPEAT_TLBI_CFGI entirely: the option bit was set and read on the exact same "nvidia,tegra264-smmu" compatible as the static key, so it added no per-instance signal that the static key did not already carry. The predicate now gates purely on arm_smmu_erratum_repeat_tlbi_cfgi_key.
- Reorder the series so the compatible-string detection lands last, once all the infrastructure exists:
1/3 factor out force_sync helper (unchanged)
2/3 add static key + WAR functions (no functional change)
3/3 enable the key on nvidia,tegra264-smmu + silicon-errata Split the old v4 "Detect" and "Issue twice" patches accordingly.
- Update the /* See ARM_SMMU_OPT_REPEAT_TLBI_CFGI */ comment inside arm_smmu_cmdq_batch_force_sync() to reference the static key description instead.
Changes since v3:
- Drop the cmds->num == 0 early-return so the refactor is truly "no functional change".
- Rename ARM_SMMU_OPT_TLBI_TWICE -> ARM_SMMU_OPT_REPEAT_TLBI_CFGI and rephrase its kdoc to be hardware-agnostic.
- Rename arm_smmu_cmd_needs_tlbi_twice() -> arm_smmu_erratum_cmd_needs_repeating() and drop the kdoc above it.
- Replace the explicit opcode switch with a single range check opcode >= CMDQ_OP_CFGI_STE && opcode < CMDQ_OP_ATC_INV.
- Introduce arm_smmu_erratum_repeat_tlbi_cfgi_key static key:
the predicate gates on it first so unaffected kernels pay only a single static_branch_unlikely() check.
- Drop the verbose Tegra264-specific comments above arm_vsmmu_can_batch_cmd() and inside the batch helper.
- Document the erratum in Documentation/arch/arm64/silicon-errata.rst.
- Guard the repeat path in arm_smmu_cmdq_issue_cmdlist() with an n > 0 check so cmds[0] is never inspected on an empty cmdlist.
- Drop the carried Reviewed-by tags now that the patch shape has changed; re-review appreciated.
Changes since v2:
- Split into a 3-patch series (refactor / detect / apply) to keep each step small and bisectable.
- Move the classifier to arm-smmu-v3.h as static inline so the iommufd file can share it.
- Add arm_vsmmu_can_batch_cmd() to split iommufd batches at "needs repeating" transitions so the per-batch decision based on the first command stays correct under mixed user input.
- Spell out in the commit message why detection is via DT and not via IIDR/ACPI.
Changes since v1:
- Detect the erratum from the existing "nvidia,tegra264-smmu" compatible instead of adding a new property.
- Centralise the doubling at the CMDQ submission layer and only apply it to CFGI/TLBI (not ATC_INV).
- Drop the binding/dtsi patches accordingly.
Ashish Mhetre (2):
iommu/arm-smmu-v3: Introduce CFGI/TLBI-repeat workaround
infrastructure
iommu/arm-smmu-v3: Enable CFGI/TLBI-repeat workaround on Tegra264
Nicolin Chen (1):
iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
Documentation/arch/arm64/silicon-errata.rst | 2 + .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 15 +++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 75 ++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 4 files changed, 81 insertions(+), 12 deletions(-)
base-commit: bee763d5f341b99cf472afeb508d4988f62a6ca1
Signature · 1 line
2.50.1
PATCH v6 1/3 iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
From: Nicolin Chen <[email protected]>
arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:
- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.
Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +17 -6
@@ -847,16 +847,27 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); } +static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +/* The batch's pre-assigned cmdq doesn't support the new command */ +if (!arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd)) +return true; + +/* Arm erratum 2812531 */ +if (cmds->num ==CMDQ_BATCH_ENTRIES -1 && + (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC )) +return true; + +return false; +} +static void arm_smmu_cmdq_batch_add_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmdq_batch *cmds,struct arm_smmu_cmd *cmd) { -bool force_sync = (cmds->num ==CMDQ_BATCH_ENTRIES -1 ) && - (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC ); -bool unsupported_cmd; - - unsupported_cmd = !arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd); -if (force_sync || unsupported_cmd) { +if (arm_smmu_cmdq_batch_force_sync (smmu, cmds, cmd)) {arm_smmu_cmdq_issue_cmdlist (smmu, cmds->cmdq, cmds->cmds, cmds->num, true);arm_smmu_cmdq_batch_init_cmd (smmu, cmds, cmd);
Signature · 1 line
2.50.1
PATCH v6 2/3 iommu/arm-smmu-v3: Introduce CFGI/TLBI-repeat workaround infrastructure
In reply to Ashish MhetreTegra264 SMMU instances need every CFGI/TLBI command sequence issued twice, with the second issue executing only after the first issue's CMD_SYNC has completed:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
ATC_INV is not affected and must never be doubled.
Add arm_smmu_erratum_repeat_tlbi_cfgi_key and an arm_smmu_erratum_cmd_needs_repeating() helper that gates on the static key first and then range-checks the opcode (CFGI_STE .. ATC_INV), so subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.
Rename the existing arm_smmu_cmdq_issue_cmdlist() to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that re-issues the same cmdlist a second time when the predicate fires. Register the new condition with arm_smmu_cmdq_batch_force_sync() and add arm_vsmmu_can_batch_cmd() so iommufd batches split at every "needs repeating" transition.
No callers enable the static key yet, so there is no functional change. A subsequent change will enable the key on affected instances.
--- .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 15 +++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 53 +++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 3 files changed, 64 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +14 -1
@@ -350,6 +350,18 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ; } +static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_erratum_cmd_needs_repeating (&last->cmd) == +arm_smmu_erratum_cmd_needs_repeating (&next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {
@@ -382,7 +394,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +49 -4
77 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
@@ -1211,6 +1211,7 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,struct arm_smmu_cmdq *cmdq, struct arm_smmu_cmd *cmds, int n, bool sync); +bool arm_smmu_erratum_cmd_needs_repeating (struct arm_smmu_cmd *cmd);#ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported (struct arm_smmu_device *smmu);
Signature · 1 line
2.50.1
Re: [PATCH v6 2/3] iommu/arm-smmu-v3: Introduce CFGI/TLBI-repeat workaround infrastructure
Quoted 21 lines · Ashish Mhetre
Tegra264 SMMU instances need every CFGI/TLBI command sequence issued twice, with the second issue executing only after the first issue's
CMD_SYNChas completed:TLBI/CFGI ...
CMD_SYNCTLBI/CFGI ...CMD_SYNC
ATC_INVis not affected and must never be doubled.Add
arm_smmu_erratum_repeat_tlbi_cfgi_keyand anarm_smmu_erratum_cmd_needs_repeating()helper that gates on the static key first and then range-checks the opcode (CFGI_STE..ATC_INV), so subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.Rename the existing
arm_smmu_cmdq_issue_cmdlist()to__arm_smmu_cmdq_issue_cmdlist()and add a thin wrapper that re-issues the same cmdlist a second time when the predicate fires. Register the new condition witharm_smmu_cmdq_batch_force_sync()and addarm_vsmmu_can_batch_cmd()so iommufd batches split at every "needs repeating" transition.
Quoted 2 lines
No callers enable the static key yet, so there is no functional change. A subsequent change will enable the key on affected instances.
Maybe add a small note (better in patch-3).
Note: since guest-level VCMDQs issue commands directly to the HW, a guest kernel enabling the cmdqv feature on NVIDIA Tegra264 must apply this WAR.
Quoted 2 lines
Suggested-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>
Some small issues; please fix:
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 355
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +3
-0+static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next)
@smmu is unused here.
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0
-0
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c MODULE_PARM_DESC(disable_msipolling, · line 51
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7
-0@@ -42,6 +42,14 @@ MODULE_PARM_DESC(disable_msipolling,static const struct iommu_ops arm_smmu_ops;static struct iommu_dirty_ops arm_smmu_dirty_ops; +/* +* Repeat every {CFGI,TLBI};CMD_SYNC command sequence so that the second +* issue executes only after the first issue's CMD_SYNC has completed. +* Does not apply to ATC_INV. The key is global and is enabled from DT +* probe on affected hardware (currently Tegra264 only). +*/ +static DEFINE_STATIC_KEY_FALSE (arm_smmu_erratum_repeat_tlbi_cfgi_key);
Since we defined a static key, it would be better explicitly add:
#include <linux/jump_label.h>
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 906
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0@@ -860,6 +900,11 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))return true; +/* See the description at arm_smmu_erratum_repeat_tlbi_cfgi_key */ +if (cmds->num ==CMDQ_BATCH_ENTRIES && +arm_smmu_erratum_cmd_needs_repeating (&cmds->cmds[0 ])) +return true;
/* * See the description at arm_smmu_erratum_repeat_tlbi_cfgi_key. Batches * never mix CFGI/TLBI with others, so checking cmds[0] alone is enough. */
Nicolin
Re: [PATCH v6 2/3] iommu/arm-smmu-v3: Introduce CFGI/TLBI-repeat workaround infrastructure
Quoted 22 lines · Nicolin Chen
Quoted 21 lines · Ashish Mhetre
Tegra264 SMMU instances need every CFGI/TLBI command sequence issued twice, with the second issue executing only after the first issue's
CMD_SYNChas completed:TLBI/CFGI ...
CMD_SYNCTLBI/CFGI ...CMD_SYNC
ATC_INVis not affected and must never be doubled.Add
arm_smmu_erratum_repeat_tlbi_cfgi_keyand anarm_smmu_erratum_cmd_needs_repeating()helper that gates on the static key first and then range-checks the opcode (CFGI_STE..ATC_INV), so subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.Rename the existing
arm_smmu_cmdq_issue_cmdlist()to__arm_smmu_cmdq_issue_cmdlist()and add a thin wrapper that re-issues the same cmdlist a second time when the predicate fires. Register the new condition witharm_smmu_cmdq_batch_force_sync()and addarm_vsmmu_can_batch_cmd()so iommufd batches split at every "needs repeating" transition.
Quoted 6 lines
Quoted 2 lines
No callers enable the static key yet, so there is no functional change. A subsequent change will enable the key on affected instances.
Maybe add a small note (better in patch-3).
Note: since guest-level VCMDQs issue commands directly to the HW, a guest kernel enabling the cmdqv feature on NVIDIA Tegra264 must apply this WAR.
Ack.
Quoted 5 lines
Quoted 2 lines
Suggested-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>Reviewed-by Nicolin Chen <[email protected]>Some small issues; please fix:
Sure, will respin v7 fixing these.
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 355
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +3
-0+static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next)@smmu is unused here.
Ack.
Quoted 17 lines
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0
-0drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c MODULE_PARM_DESC(disable_msipolling, · line 51
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7
-0@@ -42,6 +42,14 @@ MODULE_PARM_DESC(disable_msipolling,static const struct iommu_ops arm_smmu_ops;static struct iommu_dirty_ops arm_smmu_dirty_ops; +/* +* Repeat every {CFGI,TLBI};CMD_SYNC command sequence so that the second +* issue executes only after the first issue's CMD_SYNC has completed. +* Does not apply to ATC_INV. The key is global and is enabled from DT +* probe on affected hardware (currently Tegra264 only). +*/ +static DEFINE_STATIC_KEY_FALSE (arm_smmu_erratum_repeat_tlbi_cfgi_key);Since we defined a static key, it would be better explicitly add:
Quoted 1 line
#include <
linux/jump_label.h>
Ack.
Quoted 13 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 906
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0@@ -860,6 +900,11 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))return true; +/* See the description at arm_smmu_erratum_repeat_tlbi_cfgi_key */ +if (cmds->num ==CMDQ_BATCH_ENTRIES && +arm_smmu_erratum_cmd_needs_repeating (&cmds->cmds[0 ])) +return true;/* * See the description at arm_smmu_erratum_repeat_tlbi_cfgi_key. Batches * never mix CFGI/TLBI with others, so checking cmds[0] alone is enough. */
Quoted 1 line
Nicolin
Ack.
Thanks,
Ashish Mhetre
PATCH v6 3/3 iommu/arm-smmu-v3: Enable CFGI/TLBI-repeat workaround on Tegra264
In reply to Ashish MhetreNvidia Tegra264 SMMU is affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice, and that infrastructure is already in place behind arm_smmu_erratum_repeat_tlbi_cfgi_key.
Neither IDR nor IIDR flags this Tegra264-specific bug, so hardware detection is not possible. Tegra264 is device-tree-only (no ACPI/IORT support) and already has a dedicated "nvidia,tegra264-smmu" compatible, so DT-probe is the only viable detection path.
Enable the workaround on instances matching the existing "nvidia,tegra264-smmu" compatible by calling static_branch_enable() on arm_smmu_erratum_repeat_tlbi_cfgi_key. Document the erratum in Documentation/arch/arm64/silicon-errata.rst.
--- Documentation/arch/arm64/silicon-errata.rst | 2 ++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-)
Documentation/arch/arm64/silicon-errata.rst +2 -0
@@ -312,6 +312,8 @@ stable kernels. | | | T241-MPAM-4, | | | | | T241-MPAM-6 | | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | T264 SMMU | T264-SMMU-3 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
@@ -5353,8 +5353,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,if (of_dma_is_coherent (dev->of_node)) smmu->features |=ARM_SMMU_FEAT_COHERENCY ; -if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) +if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) {tegra_cmdqv_dt_probe (dev->of_node, smmu); +static_branch_enable (&arm_smmu_erratum_repeat_tlbi_cfgi_key); + }return ret; }
Signature · 1 line
2.50.1