iommu/arm-smmu-v3: Tegra264 invalidation workaround
Subsystems (3)
v3 cover iommu/arm-smmu-v3: Tegra264 invalidation workaround
Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
ATC_INV is not affected and must not be doubled.
This series implements the workaround by hooking the duplication into arm_smmu_cmdq_issue_cmdlist(), the single chokepoint that every synchronous CMDQ submission flows through.
Patch 1 is a preparatory refactor that factors the existing batch force-sync conditions out of arm_smmu_cmdq_batch_add_cmd_p() into a new arm_smmu_cmdq_batch_force_sync() helper. No functional change.
Patch 2 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string, exposes the condition via a new ARM_SMMU_OPT_TLBI_TWICE option bit, and adds a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that both the in-tree CMDQ path and the iommufd VSMMU path can share a single predicate.
Patch 3 wires the workaround in. arm_smmu_cmdq_issue_cmdlist() becomes a thin wrapper that re-issues a synced cmdlist a second time when the first command needs doubling. The Tegra264 condition is added to arm_smmu_cmdq_batch_force_sync() so a full batch carrying CFGI/TLBI commands flushes with sync=true and is then doubled. The iommufd VSMMU path (arm_vsmmu_cache_invalidate()) is also taught to split the user-supplied batch at every "needs doubling" / "doesn't need doubling" transition via a new arm_vsmmu_can_batch_cmd() predicate, since that path can otherwise mix CFGI/TLBI with ATC_INV in a single submission.
The series is based on Jason Gunthorpe's "Remove SMMUv3 struct arm_smmu_cmdq_ent" series [1], specifically commit 13428b0bf794 ("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.
[1] https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Add new prep patch 1/3 that factors the existing force-sync conditions into arm_smmu_cmdq_batch_force_sync() (from Nicolin).
- Move arm_smmu_cmd_needs_tlbi_twice() to arm-smmu-v3.h as static inline taking (smmu, cmd*) and folding in the option check.
- Plug the Tegra264 condition into arm_smmu_cmdq_batch_force_sync() instead of carrying a separate need_sync in batch_add_cmd_p().
- Fix iommufd batching: arm_vsmmu_cache_invalidate() can mix CFGI/TLBI with ATC_INV in one batch. Split at the boundary via a new arm_vsmmu_can_batch_cmd() predicate.
- Patch 2 wording: "next patch wires" -> "a subsequent change will wire".
v2: https://lore.kernel.org/all/[email protected]/
Nicolin Chen (1):
iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
Ashish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
.../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 23 ++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 +++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++ 3 files changed, 117 insertions(+), 12 deletions(-)
base-commit: 13428b0bf7947098daf9a1db14a74d33eb1b5079
Signature · 1 line
2.50.1
PATCH v3 1/3 iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
From: Nicolin Chen <[email protected]>
arm_smmu_cmdq_batch_add_cmd_p() carries two distinct reasons for flushing the current batch with a CMD_SYNC before appending the new command:
- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.
Lift those checks into a new arm_smmu_cmdq_batch_force_sync() helper so that adding another force-sync condition becomes a one-line addition. No functional change.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +20 -6
@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); } +static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +if (!cmds->num) +return false; + +/* The batch's pre-assigned cmdq doesn't support the new command */ +if (!arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd)) +return true; + +/* Arm erratum 2812531 */ +if (cmds->num ==CMDQ_BATCH_ENTRIES -1 && + (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC )) +return true; + +return false; +} +static void arm_smmu_cmdq_batch_add_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmdq_batch *cmds,struct arm_smmu_cmd *cmd) { -bool force_sync = (cmds->num ==CMDQ_BATCH_ENTRIES -1 ) && - (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC ); -bool unsupported_cmd; - - unsupported_cmd = !arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd); -if (force_sync || unsupported_cmd) { +if (arm_smmu_cmdq_batch_force_sync (smmu, cmds, cmd)) {arm_smmu_cmdq_issue_cmdlist (smmu, cmds->cmdq, cmds->cmds, cmds->num, true);arm_smmu_cmdq_batch_init_cmd (smmu, cmds, cmd);
Signature · 1 line
2.50.1
Will Deacon
maintainerRe: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
Quoted 25 lines · Ashish Mhetre
From: Nicolin Chen <[email protected]>
arm_smmu_cmdq_batch_add_cmd_p()carries two distinct reasons for flushing the current batch with aCMD_SYNCbefore appending the new command:- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.Lift those checks into a new
arm_smmu_cmdq_batch_force_sync()helper so that adding another force-sync condition becomes a one-line addition. No functional change.Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0
-0
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855
Quoted 9 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6
-0@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); } +static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +if (!cmds->num) +return false;
This check seems new?
Will
Re: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
Quoted 29 lines · Will Deacon
Quoted 28 lines · Ashish Mhetre
From: Nicolin Chen <[email protected]>
arm_smmu_cmdq_batch_add_cmd_p()carries two distinct reasons for flushing the current batch with aCMD_SYNCbefore appending the new command:- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.Lift those checks into a new
arm_smmu_cmdq_batch_force_sync()helper so that adding another force-sync condition becomes a one-line addition. No functional change.Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0
-0@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); }
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6
-0+static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +if (!cmds->num) +return false;This check seems new?
You are right. Maybe the commit message should have mentioned that there is a slight behavior change to the unsupported_cmd path:
- Before: if cmds->num = 0 && unsupported_cmd, it would issue an empty batch (one CMD_SYNC)
- After : if cmds->num = 0, no issue on the empty batch
With that, I think it's good to have?
Thanks
Nicolin
Will Deacon
maintainerRe: [PATCH v3 1/3] iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditions
Quoted 39 lines · Nicolin Chen
Quoted 29 lines · Will Deacon
Quoted 28 lines · Ashish Mhetre
From: Nicolin Chen <[email protected]>
arm_smmu_cmdq_batch_add_cmd_p()carries two distinct reasons for flushing the current batch with aCMD_SYNCbefore appending the new command:- The batch's pre-assigned cmdq does not support the new command.
- The Arm erratum 2812531 workaround (ARM_SMMU_OPT_CMDQ_FORCE_SYNC) forces a SYNC at one entry before the batch is full.Lift those checks into a new
arm_smmu_cmdq_batch_force_sync()helper so that adding another force-sync condition becomes a one-line addition. No functional change.Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0
-0@@ -847,16 +847,30 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); }Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, · line 855
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6
-0+static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +if (!cmds->num) +return false;This check seems new?
Quoted 8 lines
You are right. Maybe the commit message should have mentioned that there is a slight behavior change to the
unsupported_cmdpath:- Before: if
cmds->num= 0 &&unsupported_cmd, it would issue an empty batch (oneCMD_SYNC)
- After : ifcmds->num= 0, no issue on the empty batchWith that, I think it's good to have?
It just shouldn't be part of the refactoring patch with "no functional change". Do it in the patch that needs it and explain why it's needed.
Will
PATCH v3 2/3 iommu/arm-smmu-v3: Detect Tegra264 erratum
In reply to Ashish MhetreTegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.
The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inline arm_smmu_cmd_needs_tlbi_twice() classifier in arm-smmu-v3.h so that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.
No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,if (of_dma_is_coherent (dev->of_node)) smmu->features |=ARM_SMMU_FEAT_COHERENCY ; -if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) +if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) {tegra_cmdqv_dt_probe (dev->of_node, smmu); + smmu->options |=ARM_SMMU_OPT_TLBI_TWICE ; + }return ret; }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +40 -0
@@ -928,6 +928,14 @@ struct arm_smmu_device {#define ARM_SMMU_OPT_MSIPOLL (1 << 2)#define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)#define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4) +/* +* Tegra264 erratum: a TLB entry can survive an invalidation that races +* with concurrent traffic targeting the same entry. The software +* workaround is to issue every CFGI/TLBI command twice, each followed +* by CMD_SYNC. The second issue is guaranteed to evict the entry. +* ATC_INV commands are not affected and must not be doubled. +*/ +#define ARM_SMMU_OPT_TLBI_TWICE (1 << 5)u32 options;struct arm_smmu_cmdq cmdq;
@@ -1211,6 +1219,38 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmd *cmds, int n, bool sync); +/* +* Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered +* by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected +* SMMU instance. +*/ +static inline bool arm_smmu_cmd_needs_tlbi_twice (struct arm_smmu_device *smmu, +struct arm_smmu_cmd *cmd) +{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false; + +switch (FIELD_GET (CMDQ_0_OP , cmd->data[0 ])) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL : +case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true; +default : +return false; + } +} +#ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported (struct arm_smmu_device *smmu);void arm_smmu_sva_notifier_synchronize (void );
Signature · 1 line
2.50.1
Will Deacon
maintainerRe: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.
Quoted 39 lines
Add the
ARM_SMMU_OPT_TLBI_TWICEoption and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inlinearm_smmu_cmd_needs_tlbi_twice()classifier inarm-smmu-v3.hso that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.
Signed-off-by Ashish Mhetre <[email protected]>Reviewed-by Nicolin Chen <[email protected]>--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3
-1@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,if (of_dma_is_coherent (dev->of_node)) smmu->features |=ARM_SMMU_FEAT_COHERENCY ; -if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) +if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) {tegra_cmdqv_dt_probe (dev->of_node, smmu); + smmu->options |=ARM_SMMU_OPT_TLBI_TWICE ; + }return ret; }drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0
-0@@ -928,6 +928,14 @@ struct arm_smmu_device {#define ARM_SMMU_OPT_MSIPOLL (1 << 2)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_device { · line 938
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8
-0#define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)#define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4) +/* +* Tegra264 erratum: a TLB entry can survive an invalidation that races +* with concurrent traffic targeting the same entry. The software +* workaround is to issue every CFGI/TLBI command twice, each followed +* by CMD_SYNC. The second issue is guaranteed to evict the entry. +* ATC_INV commands are not affected and must not be doubled. +*/ +#define ARM_SMMU_OPT_TLBI_TWICE (1 << 5)
nit: I think this should be named slightly differently as it covers CFGI as well. Maybe ARM_SMMU_OPT_REPEAT_TLBI_CFGI ?
The comment can be simpler too and avoid being specific to Tegra264. The main things to say are that it repeats {CFGI,TLBI}; SYNC sequences and does not apply to ATC_INV.
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1226
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5
-0+/* +* Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered +* by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected +* SMMU instance. +*/
(remove the comment)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1228
Quoted 2 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +2
-0+static inline bool arm_smmu_cmd_needs_tlbi_twice (struct arm_smmu_device *smmu, +struct arm_smmu_cmd *cmd)
Rename the function to something like arm_smmu_erratum_cmd_needs_repeating()?
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3
-0+{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false;
Maybe we should make this a static key?
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6
-0+switch (FIELD_GET (CMDQ_0_OP , cmd->data[0 ])) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL :
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1248
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10
-0+case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true;
Isn't this just everything < ATC_INV || >= CFGI_STE? Seems better than enumerating everything.
Will
Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Quoted 5 lines
That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.
That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.
What would you like us to do here for the consistency?
Thanks
Nicolin
Will Deacon
maintainerRe: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 20 lines · Nicolin Chen
Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Quoted 5 lines
That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.
Quoted 6 lines
That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.
What would you like us to do here for the consistency?
Gah, I now realise I've mixed up Tegra264 and Tegra241 because tegra_cmdqv_dt_probe() is only called if the compatible string matches "nvidia,tegra264-smmu" yet all the cmdqv stuff talks only about Tegra241. So I was surprised that the ACPI probing code for Tegra241 wasn't enabling the workaround.
But if you're saying that:
1. Tegra264 never uses ACPI
2. Tegra241 doesn't have the invalidation erratum
Then I'm less worried (even though it's a shame that we can't detect the erratum from the hardware itself).
Will
Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 26 lines · Will Deacon
Quoted 25 lines · Nicolin Chen
Quoted 13 lines · Will Deacon
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Quoted 5 lines
That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.
That's a good point. Yet, for ACPI to detect the erratum, we would need a new IORT model or flag, right? That would need to go through the entire ACPI protocol to update SMMU's IORT spec and the header accordingly, which we don't have a use case to do so or to test it.
Quoted 7 lines
Quoted 1 line
What would you like us to do here for the consistency?
Gah, I now realise I've mixed up Tegra264 and Tegra241 because
tegra_cmdqv_dt_probe()is only called if the compatible string matches "nvidia,tegra264-smmu" yet all the cmdqv stuff talks only about Tegra241. So I was surprised that the ACPI probing code for Tegra241 wasn't enabling the workaround.
I see. Numbers could be indeed confusing :)
Quoted 4 lines
But if you're saying that:
1. Tegra264 never uses ACPI
2. Tegra241 doesn't have the invalidation erratum
Yes.
Quoted 2 lines
Then I'm less worried (even though it's a shame that we can't detect the erratum from the hardware itself).
Agreed, though I think the compatible string is already a life saver as we wouldn't need to go through the whole ACPI route..
Nicolin
Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
In reply to Will DeaconQuoted 73 lines · Will Deacon
External email: Use caution opening links or attachments
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware ID. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
That seems odd to me -- whether the hardware has the erratum is completely unrelated to whether it probes using DT or ACPI, so I find it really weird to have the workaround enabled when booting with DT and not when booting with ACPI. We should have consistent behaviour between the two.
Quoted 39 lines
Add the
ARM_SMMU_OPT_TLBI_TWICEoption and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. Also add a static-inlinearm_smmu_cmd_needs_tlbi_twice()classifier inarm-smmu-v3.hso that subsequent changes wiring the workaround into the CMDQ submission and iommufd batching paths can share a single predicate.No callers consume the option yet; a subsequent change will wire the workaround into the CMDQ issue paths.
Signed-off-by Ashish Mhetre <[email protected]>Reviewed-by Nicolin Chen <[email protected]>--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3
-1@@ -5243,8 +5243,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,if (of_dma_is_coherent (dev->of_node)) smmu->features |=ARM_SMMU_FEAT_COHERENCY ; -if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) +if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) {tegra_cmdqv_dt_probe (dev->of_node, smmu); + smmu->options |=ARM_SMMU_OPT_TLBI_TWICE ; + }return ret; }drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0
-0@@ -928,6 +928,14 @@ struct arm_smmu_device {#define ARM_SMMU_OPT_MSIPOLL (1 << 2)drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_device { · line 938
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8
-0#define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)#define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4) +/* +* Tegra264 erratum: a TLB entry can survive an invalidation that races +* with concurrent traffic targeting the same entry. The software +* workaround is to issue every CFGI/TLBI command twice, each followed +* by CMD_SYNC. The second issue is guaranteed to evict the entry. +* ATC_INV commands are not affected and must not be doubled. +*/ +#define ARM_SMMU_OPT_TLBI_TWICE (1 << 5)nit: I think this should be named slightly differently as it covers CFGI as well. Maybe
ARM_SMMU_OPT_REPEAT_TLBI_CFGI?
Quoted 3 lines
The comment can be simpler too and avoid being specific to Tegra264. The main things to say are that it repeats {CFGI,TLBI}; SYNC sequences and does not apply to
ATC_INV.
Ack, I will fix this in V4.
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1226
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +5
-0+/* +* Returns true if @cmd is one of the CFGI_* or TLBI_* commands covered +* by the Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE) on an affected +* SMMU instance. +*/ (remove the comment)
Ack.
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1228
Quoted 2 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +2
-0+static inline bool arm_smmu_cmd_needs_tlbi_twice (struct arm_smmu_device *smmu, +struct arm_smmu_cmd *cmd)Rename the function to something like
arm_smmu_erratum_cmd_needs_repeating()?
Ack.
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3
-0+{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false;Maybe we should make this a static key?
Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that static_branch_unlikely will precede the option bit check?
Quoted 16 lines
Quoted 6 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6
-0+switch (FIELD_GET (CMDQ_0_OP , cmd->data[0 ])) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL :drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1248
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +10
-0+case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true;
Quoted 2 lines
Isn't this just everything <
ATC_INV|| >=CFGI_STE? Seems better than enumerating everything.
Ack.
Quoted 1 line
Will
Thanks,
Ashish Mhetre
Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 8 lines · Ashish Mhetre
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 1231
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3
-0+{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false;Maybe we should make this a static key?
Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that
static_branch_unlikelywill precede the option bit check?
You'd have the static key and the options. Keep it simple, enable the static key once if any driver probes to set TWICE. Check the key before options to get the best code gen
But IDK if it is really worth it, there are already lots of branches on the performance tlbi flow, and we didn't do this for other tlbi affecting errata..
IDK if we really care about branches we should also be doing things like disabling the range/non-range paths and ATC based on what is actually in use..
Jason
Re: [PATCH v3 2/3] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 7 lines · Jason Gunthorpe
External email: Use caution opening links or attachments
Quoted 3 lines · Ashish Mhetre
Quoted 3 lines
Quoted 3 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3
-0+{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Maybe we should make this a static key?
Okay. Shall I add just static key and remove option bit, or have static key alongside existing option bit such that
static_branch_unlikelywill precede the option bit check?You'd have the static key and the options. Keep it simple, enable the static key once if any driver probes to set TWICE. Check the key before options to get the best code gen
Okay, I'll incorporate this in V4 and send.
Quoted 9 lines
But IDK if it is really worth it, there are already lots of branches on the performance tlbi flow, and we didn't do this for other tlbi affecting errata..
IDK if we really care about branches we should also be doing things like disabling the range/non-range paths and ATC based on what is actually in use..
Jason
PATCH v3 3/3 iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Ashish MhetreApply the workaround for Tegra264 erratum ARM_SMMU_OPT_TLBI_TWICE by issuing every CFGI/TLBI cmdlist twice on affected SMMU instances, with CMD_SYNC after each. The erratum requires this exact sequencing:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
Rename the existing arm_smmu_cmdq_issue_cmdlist() to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time when arm_smmu_cmd_needs_tlbi_twice() is true.
For the in-tree batching path, register the Tegra264 condition with arm_smmu_cmdq_batch_force_sync() so that a full batch carrying CFGI/TLBI commands flushes with sync=true.
For iommufd VSMMU path add an arm_vsmmu_can_batch_cmd() predicate that splits the iommufd batch at cmd which doesn't need doubling.
--- .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 23 +++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 ++++++++++++++++--- 2 files changed, 54 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +22 -1
@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ; } +/* +* On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI +* submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is +* taken once per cmdlist based on the first command, so a single +* batch must not mix commands that need doubling with commands that +* do not. Split the iommufd batch whenever the next user command +* crosses that boundary. +*/ +static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_cmd_needs_tlbi_twice (smmu, &last->cmd) == +arm_smmu_cmd_needs_tlbi_twice (smmu, &next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {
@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +32 -4
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,* insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. */ -int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, -struct arm_smmu_cmdq *cmdq, -struct arm_smmu_cmd *cmds,int n, -bool sync) +static int __arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) {struct arm_smmu_cmd cmd_sync;u32 prod;
@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,return ret; } +int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) +{ +int ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +/* +* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum. +* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ])) + ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +return ret; +} +static int arm_smmu_cmdq_issue_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmd *cmd,bool sync) {
@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))return true; +/* +* Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds +* a uniform opcode class, so checking the first command is enough. +*/ +if (cmds->num ==CMDQ_BATCH_ENTRIES && +arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds->cmds[0 ])) +return true; +return false; }
Signature · 1 line
2.50.1
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 15 lines · Ashish Mhetre
Apply the workaround for Tegra264 erratum
ARM_SMMU_OPT_TLBI_TWICEby issuing every CFGI/TLBI cmdlist twice on affected SMMU instances, withCMD_SYNCafter each. The erratum requires this exact sequencing:TLBI/CFGI ...
CMD_SYNCTLBI/CFGI ...CMD_SYNCRename the existing
arm_smmu_cmdq_issue_cmdlist()to__arm_smmu_cmdq_issue_cmdlist()and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time whenarm_smmu_cmd_needs_tlbi_twice()is true.For the in-tree batching path, register the Tegra264 condition with
arm_smmu_cmdq_batch_force_sync()so that a full batch carrying CFGI/TLBI commands flushes with sync=true.
Quoted 5 lines
For iommufd VSMMU path add an
arm_vsmmu_can_batch_cmd()predicate that splits the iommufd batch at cmd which doesn't need doubling.Suggested-by Nicolin Chen <[email protected]>Signed-off-by Ashish Mhetre <[email protected]>
Will Deacon
maintainerRe: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Ashish MhetreQuoted 6 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0
-0@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 360
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8
-0} +/* +* On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI +* submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is +* taken once per cmdlist based on the first command, so a single +* batch must not mix commands that need doubling with commands that +* do not. Split the iommufd batch whenever the next user command +* crosses that boundary. +*/
Again, I wouldn't bother with this comment. You probably _should_ update Documentation/arch/arm64/silicon-errata.rst, however.
Quoted 16 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +12
-0+static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_cmd_needs_tlbi_twice (smmu, &last->cmd) == +arm_smmu_cmd_needs_tlbi_twice (smmu, &next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407
Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2
-1@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;
FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:
Quoted 29 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0
-0/* FIXME always uses the main cmdq rather than trying to group by type */ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +9
-4@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,* insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. */ -int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, -struct arm_smmu_cmdq *cmdq, -struct arm_smmu_cmd *cmds,int n, -bool sync) +static int __arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) {struct arm_smmu_cmd cmd_sync;u32 prod;@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,return ret; } +int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) +{
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +10
-0+int ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +/* +* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum. +* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ]))
Can you move the arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1
-0+ ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync);
Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0+ +return ret; +} +static int arm_smmu_cmdq_issue_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmd *cmd,bool sync) {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 889
Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))return true; +/* +* Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds +* a uniform opcode class, so checking the first command is enough. +*/
Again, please drop the Tegra264 mention and just refer to the option.
Will
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 13 lines · Will Deacon
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407
Quoted 8 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2
-1@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:
Quoted 1 line
Ack. I will submit a separate patch to address this.
Nicolin
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Will DeaconQuoted 5 lines · Will Deacon
be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?
There was an earlier version trying to do this and it was pretty ugly, there are alot of places because this errata touches the CD/STE invalidation flows too.
If it was just invalidation it could have been solved by manipulating the invs list and be almost free on the fast path..
Jason
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Will DeaconQuoted 13 lines · Will Deacon
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8
-0+/* +* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum. +* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ]))Can you move the
arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1
-0+ ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync);Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.
That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..
Jason
Will Deacon
maintainerRe: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 14 lines · Jason Gunthorpe
Quoted 13 lines · Will Deacon
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8
-0+/* +* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum. +* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ]))Can you move the
arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.
Quoted 8 lines
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1
-0+ ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync);Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.
That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..
I was wondering about the case where the first command in a batch is unsupported. Can we issue a bare sync in that case?
Will
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 17 lines · Will Deacon
External email: Use caution opening links or attachments
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Will Deacon
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837
Quoted 8 lines
+ /*
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +7
-0+* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum. +* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ]))Can you move the
arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
+ ret =
__arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case.
That seems like an AI mistake though, we shouldn't actually ever pass in 0 sized batches? 'zero streams' should fail probe, not get into this code, that might be missing but I wouldn't add checks here..
I was wondering about the case where the first command in a batch is unsupported. Can we issue a bare sync in that case?
Seems a valid case. But considering the static branch approach, will
it be fine to add "n != 0" check before arm_smmu_cmd_needs_tlbi_twice() ? Something like: int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync); if(!n) return ret; if (arm_smmu_erratum_cmd_needs_repeating(smmu, &cmds[0]) && !ret && sync) ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
Quoted 1 line
Will
Thanks,
Ashish Mhetre
Re: [PATCH v3 3/3] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Will DeaconQuoted 12 lines · Will Deacon
External email: Use caution opening links or attachments
Quoted 8 lines · Ashish Mhetre
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0
-0@@ -350,6 +350,26 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ; }
Quoted 10 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu, · line 360
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +8
-0+/* +* On Tegra264, arm_smmu_cmdq_issue_cmdlist() doubles every CFGI/TLBI +* submission (see ARM_SMMU_OPT_TLBI_TWICE). The doubling decision is +* taken once per cmdlist based on the first command, so a single +* batch must not mix commands that need doubling with commands that +* do not. Split the iommufd batch whenever the next user command +* crosses that boundary. +*/ Again, I wouldn't bother with this comment. You probably _should_ update
Documentation/arch/arm64/silicon-errata.rst, however.
Ack, I'll remove this comment and add in Documentation.
Quoted 62 lines
Quoted 16 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +12
-0+static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_cmd_needs_tlbi_twice (smmu, &last->cmd) == +arm_smmu_cmd_needs_tlbi_twice (smmu, &next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu, · line 407
Quoted 7 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +2
-1@@ -382,7 +402,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;FYI: Sashiko is unhappy with the existing code here, so somebody should check that out:
Quoted 34 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +0
-0/* FIXME always uses the main cmdq rather than trying to group by type */ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +14
-4@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,* insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. */ -int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, -struct arm_smmu_cmdq *cmdq, -struct arm_smmu_cmd *cmds,int n, -bool sync) +static int __arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) {struct arm_smmu_cmd cmd_sync;u32 prod;@@ -820,6 +820,26 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,return ret; } +int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) +{ +int ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +/* +* On Tegra264 (see ARM_SMMU_OPT_TLBI_TWICE) re-issue the same +* cmdlist with another CMD_SYNC to satisfy the erratum.
Quoted 8 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 837
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +5
-0+* Callers must ensure the batch carries a uniform opcode class +* so that checking the first command is enough; the iommufd +* VSMMU path enforces this with arm_vsmmu_can_batch_cmd(). +*/ +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ]))Can you move the
arm_smmu_cmd_... part to the start of the conjunction, please? If you make it a static key as I mentioned previously, then hopefully that should mean everything else is moved out of line.
Okay, I'll fix this in V4.
Quoted 19 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, · line 838
Quoted 1 line
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1
-0+ ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync);Sashiko is also unhappy here if n == 0 because we probably shouldn't be inspecting the command array in that case. Generally, it's a pity that we can't handle this all a bit further up in the stack when we know exactly what operationg we're trying to perform, but I suppose with all the different users of the invalidation commands that's hard to catch in one place?
Quoted 11 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0+ +return ret; +} +static int arm_smmu_cmdq_issue_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmd *cmd,bool sync) {@@ -863,6 +883,14 @@ static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, (smmu->options & ARM_SMMU_OPT_CMDQ_FORCE_SYNC))return true;
Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c static bool arm_smmu_cmdq_batch_force_sync(struct arm_smmu_device *smmu, · line 889
Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4
-0+/* +* Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds +* a uniform opcode class, so checking the first command is enough. +*/ Again, please drop the Tegra264 mention and just refer to the option.
Ack.
Quoted 1 line
Will
Quoted 6 lines · Ashish Mhetre
Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue must execute only after the first issue'sCMD_SYNChas completed, giving the sequence:
This seems quite intrusive, will the TLB entry survive if you push a full invalidation instead?
Thanks,
Mostafa
Quoted 73 lines
TLBI/CFGI ...
CMD_SYNCTLBI/CFGI ...CMD_SYNC
ATC_INVis not affected and must not be doubled.This series implements the workaround by hooking the duplication into
arm_smmu_cmdq_issue_cmdlist(), the single chokepoint that every synchronous CMDQ submission flows through.Patch 1 is a preparatory refactor that factors the existing batch force-sync conditions out of
arm_smmu_cmdq_batch_add_cmd_p()into a newarm_smmu_cmdq_batch_force_sync()helper. No functional change.Patch 2 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string, exposes the condition via a new
ARM_SMMU_OPT_TLBI_TWICEoption bit, and adds a static-inlinearm_smmu_cmd_needs_tlbi_twice()classifier inarm-smmu-v3.hso that both the in-tree CMDQ path and the iommufd VSMMU path can share a single predicate.Patch 3 wires the workaround in.
arm_smmu_cmdq_issue_cmdlist()becomes a thin wrapper that re-issues a synced cmdlist a second time when the first command needs doubling. The Tegra264 condition is added toarm_smmu_cmdq_batch_force_sync()so a full batch carrying CFGI/TLBI commands flushes with sync=true and is then doubled. The iommufd VSMMU path (arm_vsmmu_cache_invalidate()) is also taught to split the user-supplied batch at every "needs doubling" / "doesn't need doubling" transition via a newarm_vsmmu_can_batch_cmd()predicate, since that path can otherwise mix CFGI/TLBI withATC_INVin a single submission.The series is based on Jason Gunthorpe's "Remove SMMUv3 struct
arm_smmu_cmdq_ent" series [1], specifically commit13428b0bf794("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.[1] https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Add new prep patch 1/3 that factors the existing force-sync conditions intoarm_smmu_cmdq_batch_force_sync()(from Nicolin).
- Movearm_smmu_cmd_needs_tlbi_twice()toarm-smmu-v3.has static inline taking (smmu, cmd*) and folding in the option check.
- Plug the Tegra264 condition intoarm_smmu_cmdq_batch_force_sync()instead of carrying a separateneed_syncinbatch_add_cmd_p().
- Fix iommufd batching:arm_vsmmu_cache_invalidate()can mix CFGI/TLBI withATC_INVin one batch. Split at the boundary via a newarm_vsmmu_can_batch_cmd()predicate.
- Patch 2 wording: "next patch wires" -> "a subsequent change will wire".v2: https://lore.kernel.org/all/[email protected]/
Nicolin Chen (1):
iommu/arm-smmu-v3: Factor out CMDQ batch force-sync conditionsAshish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264.../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 23 ++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 +++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 40 +++++++++++ 3 files changed, 117 insertions(+), 12 deletions(-)base-commit:
13428b0bf7947098daf9a1db14a74d33eb1b5079Signature · 1 line
2.50.1
Quoted 10 lines · Mostafa Saleh
Quoted 6 lines · Ashish Mhetre
Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue must execute only after the first issue'sCMD_SYNChas completed, giving the sequence:This seems quite intrusive, will the TLB entry survive if you push a full invalidation instead?
It's 36 lines and completely contained to the insides of the command sumbissions code??
Stuff like this is why I was guiding you to use the more code exactly as is for pkvm. Historically there have been many invalidation related errata, and invalidate twice seems to be a theme in fixing many of them.
Jason