iommu/arm-smmu-v3: Tegra264 invalidation workaround
Subsystems (3)
v2 cover iommu/arm-smmu-v3: Tegra264 invalidation workaround
Nvidia Tegra264 SMMUs are affected by an erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue must execute only after the first issue's CMD_SYNC has completed, giving the sequence:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
ATC_INV is not affected and must not be doubled.
This series implements the workaround by hooking the duplication into the single chokepoint that every synchronous submission flows through, arm_smmu_cmdq_issue_cmdlist().
Patch 1 detects affected instances using the existing "nvidia,tegra264-smmu" compatible string and exposes the condition via a new ARM_SMMU_OPT_TLBI_TWICE option bit.
Patch 2 wires the option into the CMDQ submission path: when @sync is true and the cmdlist carries a CFGI/TLBI opcode, the same cmdlist is re-issued a second time. The batch capacity-rollover path is also adjusted to force a SYNC on chunks that carry CFGI/TLBI commands so each flushed chunk is correctly doubled.
The series is based on Jason Gunthorpe's "Remove SMMUv3 struct arm_smmu_cmdq_ent" series [1], specifically commit 13428b0bf794 ("iommu/arm-smmu-v3: Directly encode TLBI commands") which is the final patch of that series in linux-next.
[1] https://lore.kernel.org/all/[email protected]/
Changes since v1:
- Patch 1: Add IIDR/IDR/ACPI rationale to the commit message, explaining why the erratum is detected from the device tree compatible string rather than from a HW register or ACPI/IORT.
- Rebased onto the publicly accessible base 13428b0bf794 (final commit of Jason's series in linux-next) so that the base-commit is resolvable on lore.kernel.org.
- Patch 2: Picked up Reviewed-by: Jason Gunthorpe <[email protected]>.
- No code changes since v1.
Ashish Mhetre (2):
iommu/arm-smmu-v3: Detect Tegra264 erratum iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 70 +++++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++ 2 files changed, 72 insertions(+), 6 deletions(-)
base-commit: 13428b0bf7947098daf9a1db14a74d33eb1b5079
Signature · 1 line
2.50.1
PATCH v2 1/2 iommu/arm-smmu-v3: Detect Tegra264 erratum
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by CMD_SYNC) twice. The second issue is guaranteed to evict the entry. ATC_INV is not affected and must not be doubled.
The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Add the ARM_SMMU_OPT_TLBI_TWICE option and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 +++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -1
@@ -5229,8 +5229,10 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,if (of_dma_is_coherent (dev->of_node)) smmu->features |=ARM_SMMU_FEAT_COHERENCY ; -if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) +if (of_device_is_compatible (dev->of_node,"nvidia,tegra264-smmu" )) {tegra_cmdqv_dt_probe (dev->of_node, smmu); + smmu->options |=ARM_SMMU_OPT_TLBI_TWICE ; + }return ret; }
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +8 -0
@@ -928,6 +928,14 @@ struct arm_smmu_device {#define ARM_SMMU_OPT_MSIPOLL (1 << 2)#define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)#define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4) +/* +* Tegra264 erratum: a TLB entry can survive an invalidation that races +* with concurrent traffic targeting the same entry. The software +* workaround is to issue every CFGI/TLBI command twice, each followed +* by CMD_SYNC. The second issue is guaranteed to evict the entry. +* ATC_INV commands are not affected and must not be doubled. +*/ +#define ARM_SMMU_OPT_TLBI_TWICE (1 << 5)u32 options;struct arm_smmu_cmdq cmdq;
Signature · 1 line
2.50.1
Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 7 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.
Quoted 7 lines
The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Add the
ARM_SMMU_OPT_TLBI_TWICEoption and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.
I was told to avoid "patch": once a patch is applied it becomes a commit. So, maybe "a subsequent change will wire".
Quoted 2 lines
Signed-off-by Ashish Mhetre <[email protected]>
Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Detect Tegra264 erratum
Quoted 12 lines · Nicolin Chen
Quoted 11 lines · Ashish Mhetre
Tegra264 SMMU is affected by erratum where a TLB entry can survive an invalidation that races with concurrent traffic targeting the same entry. The hardware-recommended software workaround is to issue every CFGI/TLBI command (each followed by
CMD_SYNC) twice. The second issue is guaranteed to evict the entry.ATC_INVis not affected and must not be doubled.The erratum is not flagged by any SMMUv3 IDR/IIDR register, so it cannot be detected from hardware register. Tegra264 boots from device tree only and has no ACPI/IORT support, so detection is through device tree only.
Quoted 5 lines
Quoted 3 lines
Add the
ARM_SMMU_OPT_TLBI_TWICEoption and set it on instances matching the existing "nvidia,tegra264-smmu" compatible. No callers consume the option yet, next patch wires the workaround into the CMDQ issue paths.I was told to avoid "patch": once a patch is applied it becomes a commit. So, maybe "a subsequent change will wire".
Sure, I'll update it in next version.
Quoted 2 lines
Quoted 1 line
Signed-off-by Ashish Mhetre <[email protected]>Reviewed-by Nicolin Chen <[email protected]>
Thanks Nicolin.
PATCH v2 2/2 iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
In reply to Ashish MhetreApply the workaround for Tegra264 erratum by issuing every CFGI/TLBI command twice on affected SMMU instances, with CMD_SYNC after each. The erratum requires this exact sequencing:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
To get this sequence with minimal surgery, hook the workaround into arm_smmu_cmdq_issue_cmdlist(). Rename the original function to __arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on affected SMMUs and when @sync is true, re-issues the same cmdlist a second time.
A new arm_smmu_cmd_needs_tlbi_twice() helper classifies which opcodes need the doubling: CFGI_* and TLBI_*.
For batches that exceed CMDQ_BATCH_ENTRIES commands, arm_smmu_cmdq_batch_add_cmd_p() normally flushes the full buffer with sync=false, deferring the SYNC to the eventual batch_submit(). On affected SMMUs this would leave the first chunk's commands issued only once, since the WAR hook in arm_smmu_cmdq_issue_cmdlist() only fires on synced submissions. Force a SYNC on the capacity rollover when the buffer carries CFGI/TLBI commands so every flushed chunk is correctly doubled.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 +++++++++++++++++++-- 1 file changed, 61 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +61 -5
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,* insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. */ -int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, -struct arm_smmu_cmdq *cmdq, -struct arm_smmu_cmd *cmds,int n, -bool sync) +static int __arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) {struct arm_smmu_cmd cmd_sync;u32 prod;
@@ -820,6 +820,52 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,return ret; } +/* +* Returns true if @opcode is a CFGI_* or TLBI_* command, i.e. one of the +* invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). +*/ +static bool arm_smmu_cmd_needs_tlbi_twice (u8 opcode) +{ +switch (opcode) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL : +case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true; +default : +return false; + } +} + +int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq *cmdq, +struct arm_smmu_cmd *cmds,int n, +bool sync) +{ +int ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +/* +* The driver's batch invariants keep a single submission's +* opcode class uniform, so checking the first command is enough. +*/ +if (!ret && sync && (smmu->options &ARM_SMMU_OPT_TLBI_TWICE ) && +arm_smmu_cmd_needs_tlbi_twice (FIELD_GET (CMDQ_0_OP , + cmds[0 ].data[0 ]))) + ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync); + +return ret; +} +static int arm_smmu_cmdq_issue_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmd *cmd,bool sync) {
@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, }if (cmds->num ==CMDQ_BATCH_ENTRIES ) { +/* +* Force a SYNC only when the batch carries commands that +* have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE). +* The batch holds a uniform opcode class, so checking +* the first command is sufficient. +*/ +bool need_sync = (smmu->options &ARM_SMMU_OPT_TLBI_TWICE ) && +arm_smmu_cmd_needs_tlbi_twice (FIELD_GET (CMDQ_0_OP , + cmds->cmds[0 ].data[0 ])); +arm_smmu_cmdq_issue_cmdlist (smmu, cmds->cmdq, cmds->cmds, - cmds->num, false); + cmds->num, need_sync);arm_smmu_cmdq_batch_init_cmd (smmu, cmds, cmd); }
Signature · 1 line
2.50.1
Re: [PATCH v2 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 5 lines · Ashish Mhetre
(quoted diff) +5
-0+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, + struct arm_smmu_cmdq *cmdq, + struct arm_smmu_cmd *cmds, int n, + bool sync) +{
Review
Quoted 10 lines
(quoted diff) +10
-0+ int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync); + + /* + * The driver's batch invariants keep a single submission's + * opcode class uniform, so checking the first command is enough. + */ + if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) && + arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP, + cmds[0].data[0]))) + ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
https://sashiko.dev/#/patchset/20260529140830.629738-1-amhetre%40nvidia.com
Sashiko pointed out that the iommufd path might mix commands when calling arm_smmu_cmdq_issue_cmdlist(), which is valid I think.
Quoted 7 lines
static int arm_smmu_cmdq_issue_cmd_p(struct
arm_smmu_device*smmu,(quoted diff) +0
-0struct arm_smmu_cmd *cmd, bool sync) {@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, } if (cmds->num == CMDQ_BATCH_ENTRIES) {
Review static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, · line 921
Quoted 10 lines
(quoted diff) +10
-0+ /* + * Force a SYNC only when the batch carries commands that + * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE). + * The batch holds a uniform opcode class, so checking + * the first command is sufficient. + */ + bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) && + arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP, + cmds->cmds[0].data[0])); +
Also, given that this does "force a sync", I think it might be nicer to go to the force_sync path. One of my ongoing series also needs to add another force_sync condition, so I think it would be cleaner to have a helper function.
Maybe try the following diff. That arm_smmu_cmdq_batch_force_sync() might be added with a preparatory patch, but it's up to you.
--------------------------------------------------------------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +14 -1
@@ -350,6 +350,18 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ; } +static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_cmd_needs_tlbi_twice (smmu, &last->cmd) == +arm_smmu_cmd_needs_tlbi_twice (smmu, &next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {
@@ -382,7 +394,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;/* FIXME always uses the main cmdq rather than trying to group by type */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +31 -46
107 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +31 -0
@@ -1219,6 +1219,37 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmd *cmds, int n, bool sync); +/* +* Returns true if @cmd opcode is a CFGI_* or TLBI_* command, i.e. one of the +* invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). +*/ +static inline bool arm_smmu_cmd_needs_tlbi_twice (struct arm_smmu_device *smmu, +struct arm_smmu_cmd *cmd) +{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false; + +switch (FIELD_GET (CMDQ_0_OP , cmd->data[0 ])) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL : +case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true; +default : +return false; + } +} +#ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported (struct arm_smmu_device *smmu);void arm_smmu_sva_notifier_synchronize (void );
--------------------------------------------------------------------
Nicolinc
Re: [PATCH v2 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Quoted 11 lines · Nicolin Chen
Quoted 10 lines · Ashish Mhetre
(quoted diff) +10
-0+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, + struct arm_smmu_cmdq *cmdq, + struct arm_smmu_cmd *cmds, int n, + bool sync) +{ + int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync); + + /* + * The driver's batch invariants keep a single submission's + * opcode class uniform, so checking the first command is enough.
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ */ + if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) && + arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP, + cmds[0].data[0]))) + ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);https://sashiko.dev/#/patchset/20260529140830.629738-1-amhetre%40nvidia.com
Sashiko pointed out that the iommufd path might mix commands when callingarm_smmu_cmdq_issue_cmdlist(), which is valid I think.
Okay, I'll update the batching for iommufd path as suggested by you in next version.
Quoted 217 lines
Quoted 7 lines
static int arm_smmu_cmdq_issue_cmd_p(struct
arm_smmu_device*smmu,
structarm_smmu_cmd*cmd, bool sync)
{(quoted diff) +0
-0@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, } if (cmds->num == CMDQ_BATCH_ENTRIES) {Review static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu, · line 921
Quoted 10 lines
(quoted diff) +10
-0+ /* + * Force a SYNC only when the batch carries commands that + * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE). + * The batch holds a uniform opcode class, so checking + * the first command is sufficient. + */ + bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) && + arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP, + cmds->cmds[0].data[0])); +Also, given that this does "force a sync", I think it might be nicer to go to the
force_syncpath. One of my ongoing series also needs to add anotherforce_synccondition, so I think it would be cleaner to have a helper function.Maybe try the following diff. That
arm_smmu_cmdq_batch_force_sync()might be added with a preparatory patch, but it's up to you.--------------------------------------------------------------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +14
-1@@ -350,6 +350,18 @@ static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,return 0 ; } +static bool arm_vsmmu_can_batch_cmd (struct arm_smmu_device *smmu, +struct arm_vsmmu_invalidation_cmd *last, +struct arm_vsmmu_invalidation_cmd *next) +{ +struct arm_smmu_cmd next_cmd = { + .data[0 ] =le64_to_cpu (next->ucmd.cmd[0 ]), + }; + +return arm_smmu_cmd_needs_tlbi_twice (smmu, &last->cmd) == +arm_smmu_cmd_needs_tlbi_twice (smmu, &next_cmd); +} +int arm_vsmmu_cache_invalidate (struct iommufd_viommu *viommu,struct iommu_user_data_array *array) {@@ -382,7 +394,8 @@ int arm_vsmmu_cache_invalidate(struct iommufd_viommu *viommu,/* FIXME work in blocks of CMDQ_BATCH_ENTRIES and copy each block? */ cur++; -if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 ) +if (cur != end && (cur - last) !=CMDQ_BATCH_ENTRIES -1 && +arm_vsmmu_can_batch_cmd (smmu, last, cur))continue ;/* FIXME always uses the main cmdq rather than trying to group by type */ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +31
-46@@ -820,33 +820,6 @@ static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,return ret; } -/* -* Returns true if @opcode is a CFGI_* or TLBI_* command, i.e. one of the -* invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). -*/ -static bool arm_smmu_cmd_needs_tlbi_twice (u8 opcode) -{ -switch (opcode) { -case CMDQ_OP_CFGI_STE : -case CMDQ_OP_CFGI_ALL : -case CMDQ_OP_CFGI_CD : -case CMDQ_OP_CFGI_CD_ALL : -case CMDQ_OP_TLBI_NH_ALL : -case CMDQ_OP_TLBI_NH_ASID : -case CMDQ_OP_TLBI_NH_VA : -case CMDQ_OP_TLBI_NH_VAA : -case CMDQ_OP_TLBI_EL2_ALL : -case CMDQ_OP_TLBI_EL2_ASID : -case CMDQ_OP_TLBI_EL2_VA : -case CMDQ_OP_TLBI_S12_VMALL : -case CMDQ_OP_TLBI_S2_IPA : -case CMDQ_OP_TLBI_NSNH_ALL : -return true; -default : -return false; - } -} - int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq, struct arm_smmu_cmd *cmds, int n,@@ -858,9 +831,7 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,* The driver's batch invariants keep a single submission's * opcode class uniform, so checking the first command is enough. */ -if (!ret && sync && (smmu->options &ARM_SMMU_OPT_TLBI_TWICE ) && -arm_smmu_cmd_needs_tlbi_twice (FIELD_GET (CMDQ_0_OP , - cmds[0 ].data[0 ]))) +if (!ret && sync &&arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds[0 ])) ret =__arm_smmu_cmdq_issue_cmdlist (smmu, cmdq, cmds, n, sync);return ret;@@ -893,34 +864,48 @@ static void arm_smmu_cmdq_batch_init_cmd(struct arm_smmu_device *smmu, cmds->cmdq =arm_smmu_get_cmdq (smmu, cmd); } +static bool arm_smmu_cmdq_batch_force_sync (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_batch *cmds, +struct arm_smmu_cmd *cmd) +{ +if (!cmds->num) +return false; + +/* The batch's pre-assigned cmdq doesn't support the new command */ +if (!arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd)) +return true; + +/* Arm erratum 2812531 */ +if (cmds->num ==CMDQ_BATCH_ENTRIES -1 && + (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC )) +return true; + +/* +* Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). The batch holds a +* uniform opcode class, so checking the first command is enough. +*/ +if ((cmds->num ==CMDQ_BATCH_ENTRIES ) && +arm_smmu_cmd_needs_tlbi_twice (smmu, &cmds->cmds[0 ])) +return true; + +return false; +} +static void arm_smmu_cmdq_batch_add_cmd_p (struct arm_smmu_device *smmu,struct arm_smmu_cmdq_batch *cmds,struct arm_smmu_cmd *cmd) { -bool force_sync = (cmds->num ==CMDQ_BATCH_ENTRIES -1 ) && - (smmu->options &ARM_SMMU_OPT_CMDQ_FORCE_SYNC ); -bool unsupported_cmd; +bool force_sync =arm_smmu_cmdq_batch_force_sync (smmu, cmds, cmd); - unsupported_cmd = !arm_smmu_cmdq_supports_cmd (cmds->cmdq, cmd); -if (force_sync || unsupported_cmd) { +if (force_sync) {arm_smmu_cmdq_issue_cmdlist (smmu, cmds->cmdq, cmds->cmds, cmds->num, true);arm_smmu_cmdq_batch_init_cmd (smmu, cmds, cmd); }if (cmds->num ==CMDQ_BATCH_ENTRIES ) { -/* -* Force a SYNC only when the batch carries commands that -* have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE). -* The batch holds a uniform opcode class, so checking -* the first command is sufficient. -*/ -bool need_sync = (smmu->options &ARM_SMMU_OPT_TLBI_TWICE ) && -arm_smmu_cmd_needs_tlbi_twice (FIELD_GET (CMDQ_0_OP , - cmds->cmds[0 ].data[0 ])); -arm_smmu_cmdq_issue_cmdlist (smmu, cmds->cmdq, cmds->cmds, - cmds->num, need_sync); + cmds->num, false);arm_smmu_cmdq_batch_init_cmd (smmu, cmds, cmd); }drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +31
-1@@ -1219,6 +1219,37 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmd *cmds, int n, bool sync); +/* +* Returns true if @cmd opcode is a CFGI_* or TLBI_* command, i.e. one of the +* invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE). +*/ +static inline bool arm_smmu_cmd_needs_tlbi_twice (struct arm_smmu_device *smmu, +struct arm_smmu_cmd *cmd) +{ +if (!(smmu->options &ARM_SMMU_OPT_TLBI_TWICE )) +return false; + +switch (FIELD_GET (CMDQ_0_OP , cmd->data[0 ])) { +case CMDQ_OP_CFGI_STE : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_TLBI_NH_ALL : +case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NH_VA : +case CMDQ_OP_TLBI_NH_VAA : +case CMDQ_OP_TLBI_EL2_ALL : +case CMDQ_OP_TLBI_EL2_ASID : +case CMDQ_OP_TLBI_EL2_VA : +case CMDQ_OP_TLBI_S12_VMALL : +case CMDQ_OP_TLBI_S2_IPA : +case CMDQ_OP_TLBI_NSNH_ALL : +return true; +default : +return false; + } +} +#ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported (struct arm_smmu_device *smmu);void arm_smmu_sva_notifier_synchronize (void ); --------------------------------------------------------------------
Quoted 1 line
Nicolinc
Ack, I'll add force_sync patch by you as a preparatory patch and rebase my changes on top of it and send it in V3.
Thanks,
Ashish Mhetre