iommu/arm-smmu-v3: Add a user-configurable tlb_invalidate_threshold
Subsystems (2)
PATCH v2 1/1 iommu/arm-smmu-v3: Add a user-configurable tlb_invalidate_threshold
When receiving an __arm_smmu_tlb_inv_range() call with a large size, there could be a long latency at this function call: one part is coming from a large software overhead in the routine of building commands, and the other part is coming from CMDQ hardware consuming the large number of commands. This latency could be significantly large on an SMMU that does not support range invalidation commands, i.e. no ARM_SMMU_FEAT_RANGE_INV.
One way to optimize this is to replace a large number of VA invalidation commands with one single per-asid invalidation command, when the requested size reaches a threshold. This threshold can be configurable depending on the SMMU implementaion.
Add a tlb_invalidate_threshold threshold per SMMU device, and allow it to be configured via an SYSFS node. Set its default value next to the highest address SMMU translates, meaning by default there would be no replacement until a user sets the threshold. Also, add an ABI doc for this new node with a "%llu" format so the threshold can be 4GB size on a 32-bit kernel.
---
Changelog
v2:
* Renamed the sysfs node and changed it to represent size in bytes
* Set the upper/lower values of the threshold to be more related to SMMU
* Revised the ABI description accordingly and added upper/lower values
* Dropped the "arm-smmu-v3" subdirectory for the new sysfs node
* Some minor coding style and doc/comments updating
v1: https://lore.kernel.org/all/[email protected]/
.../ABI/testing/sysfs-class-iommu-arm-smmu-v3 | 13 +++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 54 ++++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 ++ 3 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-arm-smmu-v3
Documentation/ABI/testing/sysfs-class-iommu-arm-smmu-v3 new file +13 -0
@@ -0,0 +1,13 @@ +What: /sys/class/iommu/<iommu>/tlb_invalidate_threshold +Date: August 2023 +KernelVersion: 6.6 +Contact: Nicolin Chen <[email protected]> +Description: + Size threshold in bytes for SMMU to replace VA TLB-invalidation + commands with one single per-asid TLB-invalidation command. Its + default value is set next to the highest address that the SMMU + translates, so there will be no replacement of TLB-invalidation + commands by default. It is also the maximum value the threshold + can be set to. Its minimum value is the smallest page size SMMU + supports. + Format: %llu
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +53 -1
78 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3 -0
@@ -665,6 +665,9 @@ struct arm_smmu_device {unsigned long oas;/* PA */ unsigned long pgsize_bitmap; +/* Size threshold in bytes to convert VA range TLBI to asid TLBI */ +unsigned long long tlb_invalidate_threshold; +#define ARM_SMMU_MAX_ASIDS (1 << 16)unsigned int asid_bits;
Signature · 1 line
2.41.0
Quoted 11 lines · Nicolin Chen
When receiving an
__arm_smmu_tlb_inv_range()call with a large size, there could be a long latency at this function call: one part is coming from a large software overhead in the routine of building commands, and the other part is coming from CMDQ hardware consuming the large number of commands. This latency could be significantly large on an SMMU that does not support range invalidation commands, i.e. noARM_SMMU_FEAT_RANGE_INV.One way to optimize this is to replace a large number of VA invalidation commands with one single per-asid invalidation command, when the requested size reaches a threshold. This threshold can be configurable depending on the SMMU implementaion.
I'm rethinking about this size-based threshold, since what really affects the latency is the number of the invalidation commands in the request. So having an npages-based threshold might be optimal, though the idea and implementation would be similar.
Thanks
Nicolin
Will Deacon
maintainerQuoted 14 lines · Nicolin Chen
Quoted 11 lines · Nicolin Chen
When receiving an
__arm_smmu_tlb_inv_range()call with a large size, there could be a long latency at this function call: one part is coming from a large software overhead in the routine of building commands, and the other part is coming from CMDQ hardware consuming the large number of commands. This latency could be significantly large on an SMMU that does not support range invalidation commands, i.e. noARM_SMMU_FEAT_RANGE_INV.One way to optimize this is to replace a large number of VA invalidation commands with one single per-asid invalidation command, when the requested size reaches a threshold. This threshold can be configurable depending on the SMMU implementaion.
Quoted 4 lines
I'm rethinking about this size-based threshold, since what really affects the latency is the number of the invalidation commands in the request. So having an npages-based threshold might be optimal, though the idea and implementation would be similar.
On the CPU side, we just have:
#define MAX_TLBI_OPS PTRS_PER_PTE
in asm/tlbflush.h
Can we start off with something similar for the SMMU? I'm not massively keen on exposing this as a knob to userspace, because I don't think most people will have a clue about how to tune it.
Will
Quoted 22 lines · Will Deacon
Quoted 14 lines · Nicolin Chen
Quoted 11 lines · Nicolin Chen
When receiving an
__arm_smmu_tlb_inv_range()call with a large size, there could be a long latency at this function call: one part is coming from a large software overhead in the routine of building commands, and the other part is coming from CMDQ hardware consuming the large number of commands. This latency could be significantly large on an SMMU that does not support range invalidation commands, i.e. noARM_SMMU_FEAT_RANGE_INV.One way to optimize this is to replace a large number of VA invalidation commands with one single per-asid invalidation command, when the requested size reaches a threshold. This threshold can be configurable depending on the SMMU implementaion.
Quoted 4 lines
I'm rethinking about this size-based threshold, since what really affects the latency is the number of the invalidation commands in the request. So having an npages-based threshold might be optimal, though the idea and implementation would be similar.
On the CPU side, we just have:
Quoted 7 lines
#define
MAX_TLBI_OPSPTRS_PER_PTEin
asm/tlbflush.hCan we start off with something similar for the SMMU? I'm not massively keen on exposing this as a knob to userspace, because I don't think most people will have a clue about how to tune it.
Yes! I was hesitating about an arbitrary threshold setup that actually fits our situation better. Now it makes sense. Thanks for the input!
What I would do was to pick a number based on our test results. Yet, it seems that 1024 was chosen at the beginning to fix a softlock bug, and it changed to PTRS_PER_PTE for a better perf, IIUIC. Should we use the similar setup for SMMU? I found it is stored in data->bits_per_level, so perhaps pass it back to the driver via "struct io_pgtable_cfg".
Nicolin