SMMUv3.2 Range-based TLB Invalidation Support
v4 cover SMMUv3.2 Range-based TLB Invalidation Support
SMMU3.2 brings the support of range-based TLB invalidation and level hint. When this feature is supported, the SMMUv3 driver is allowed to send TLB invalidations for a range of IOVAs instead of using page based invalidation.
Implementing this feature in the virtual SMMUv3 device is mandated for DPDK on guest use case: DPDK uses hugepage buffers and guest sends invalidations for blocks. Without this feature, a guest invalidation of a block of 1GB for instance translates into a storm of page invalidations. Each of them is trapped by the VMM and cascaded downto the physical IOMMU. This completely stalls the execution. This integration issue was initially reported in [1].
Now SMMUv3.2 specifies additional parameters to NH_VA and NH_VAA stage 1 invalidation commands so we can support those extensions.
Supporting block mappings in the IOTLB look sensible in terms of TLB entry consumption. However looking at virtio/vhost device usage, without block mapping and without range invalidation (< 5.7 kernels it may be less performant. However for recent guest kernels supporting range invalidations [2], the performance should be similar.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu.git
branch: v5.1.0-rc1-smmuv3-ril-v4
References:
[1] [RFC v2 4/4] iommu/arm-smmu-v3: add CMD_TLBI_NH_VA_AM command for iova range invalidation
(https://lists.linuxfoundation.org/pipermail/iommu/2017-August/023679.html
[2] 5.7+ kernels featuring6a481a95d4c1 iommu/arm-smmu-v3: Add SMMUv3.2 range invalidation support
History:
v3 -> v4:
- use tg in hash function and directly compare key fields in the equal function
- don't claim 3.2 as we don't have BBML support yet
- fixed a bunch of indent issues
v2 -> v3:
- restore the Jenkins hash function and keep the key as a struct
- simplify the logic in smmu_hash_remove_by_asid_iova
- added HAD support
- expose AIDR (advertise v3.2 support) and fix IIDR offset
v1 -> v2:
- added "hw/arm/smmu: Introduce smmu_get_iotlb_key()"
- removed "[PATCH 5/9] hw/arm/smmuv3: Store the starting level in SMMUTransTableInfo"
- Collected Peter's R-b
- In this version the key still features TG/LVL.
- More details in individual history logs
Eric Auger (11):
hw/arm/smmu-common: Factorize some code in smmu_ptw_64() hw/arm/smmu-common: Add IOTLB helpers hw/arm/smmu: Introduce smmu_get_iotlb_key() hw/arm/smmu: Introduce SMMUTLBEntry for PTW and IOTLB value hw/arm/smmu-common: Manage IOTLB block entries hw/arm/smmuv3: Introduce smmuv3_s1_range_inval() helper hw/arm/smmuv3: Get prepared for range invalidation hw/arm/smmuv3: Fix IIDR offset
hw/arm/smmuv3: Let AIDR advertise SMMUv3.0 support hw/arm/smmuv3: Support HAD and advertise SMMUv3.1 support hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation
hw/arm/smmu-internal.h | 8 ++ hw/arm/smmuv3-internal.h | 10 +- include/hw/arm/smmu-common.h | 19 +++- include/hw/arm/smmuv3.h | 1 + hw/arm/smmu-common.c | 214 ++++++++++++++++++++++++----------- hw/arm/smmuv3.c | 142 +++++++++++------------ hw/arm/trace-events | 12 +- 7 files changed, 258 insertions(+), 148 deletions(-)
Signature · 1 line
2.21.3
PATCH v4 1/11 hw/arm/smmu-common: Factorize some code in smmu_ptw_64()
Page and block PTE decoding can share some code. Let's first handle table PTE and factorize some code shared by page and block PTEs.
---
v1 -> v2:
- restore goto error in case get_pte() fails
--- hw/arm/smmu-common.c | 48 ++++++++++++++++---------------------------- 1 file changed, 17 insertions(+), 31 deletions(-)
hw/arm/smmu-common.c +17 -31
@@ -186,7 +186,7 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,uint64_t subpage_size =1ULL <<level_shift (level, granule_sz);uint64_t mask = subpage_size -1 ;uint32_t offset =iova_level_offset (iova, inputsize, level, granule_sz); -uint64_t pte; +uint64_t pte, gpa;dma_addr_t pte_addr = baseaddr + offset *sizeof (pte);uint8_t ap;
@@ -199,56 +199,42 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,if (is_invalid_pte (pte) ||is_reserved_pte (pte, level)) {trace_smmu_ptw_invalid_pte (stage, level, baseaddr, pte_addr, offset, pte); - info->type =SMMU_PTW_ERR_TRANSLATION ; - goto error; +break ; } -if (is_page_pte (pte, level)) { -uint64_t gpa =get_page_pte_address (pte, granule_sz); +if (is_table_pte (pte, level)) { + ap =PTE_APTABLE (pte); - ap =PTE_AP (pte);if (is_permission_fault (ap, perm)) { info->type =SMMU_PTW_ERR_PERMISSION ; goto error; } - - tlbe->translated_addr = gpa + (iova & mask); - tlbe->perm =PTE_AP_TO_PERM (ap); + baseaddr =get_table_pte_address (pte, granule_sz); + level++; +continue ; + }else if (is_page_pte (pte, level)) { + gpa =get_page_pte_address (pte, granule_sz);trace_smmu_ptw_page_pte (stage, level, iova, baseaddr, pte_addr, pte, gpa); -return 0 ; - } -if (is_block_pte (pte, level)) { + }else {uint64_t block_size; -hwaddr gpa =get_block_pte_address (pte, level, granule_sz, - &block_size); - - ap =PTE_AP (pte); -if (is_permission_fault (ap, perm)) { - info->type =SMMU_PTW_ERR_PERMISSION ; - goto error; - } + gpa =get_block_pte_address (pte, level, granule_sz, + &block_size);trace_smmu_ptw_block_pte (stage, level, baseaddr, pte_addr, pte, iova, gpa, block_size >>20 ); - - tlbe->translated_addr = gpa + (iova & mask); - tlbe->perm =PTE_AP_TO_PERM (ap); -return 0 ; } - -/* table pte */ - ap =PTE_APTABLE (pte); - + ap =PTE_AP (pte);if (is_permission_fault (ap, perm)) { info->type =SMMU_PTW_ERR_PERMISSION ; goto error; } - baseaddr =get_table_pte_address (pte, granule_sz); - level++; - } + tlbe->translated_addr = gpa + (iova & mask); + tlbe->perm =PTE_AP_TO_PERM (ap); +return 0 ; + } info->type =SMMU_PTW_ERR_TRANSLATION ; error:
Signature · 1 line
2.21.3
PATCH v4 2/11 hw/arm/smmu-common: Add IOTLB helpers
In reply to Eric AugerAdd two helpers: one to lookup for a given IOTLB entry and one to insert a new entry. We also move the tracing there.
--- include/hw/arm/smmu-common.h | 2 ++ hw/arm/smmu-common.c | 36 ++++++++++++++++++++++++++++++++++++ hw/arm/smmuv3.c | 26 ++------------------------ hw/arm/trace-events | 5 +++-- 4 files changed, 43 insertions(+), 26 deletions(-)
include/hw/arm/smmu-common.h +2 -0
@@ -153,6 +153,8 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);#define SMMU_IOTLB_MAX_SIZE 256 +IOMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg,hwaddr iova); +void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,IOMMUTLBEntry *entry);void smmu_iotlb_inv_all (SMMUState *s);void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid);void smmu_iotlb_inv_iova (SMMUState *s,uint16_t asid,dma_addr_t iova);
hw/arm/smmu-common.c +36 -0
@@ -32,6 +32,42 @@/* IOTLB Management */ +IOMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg, +hwaddr iova) +{ +SMMUIOTLBKey key = {.asid = cfg->asid, .iova = iova}; +IOMMUTLBEntry *entry =g_hash_table_lookup (bs->iotlb, &key); + +if (entry) { + cfg->iotlb_hits++; +trace_smmu_iotlb_lookup_hit (cfg->asid, iova, + cfg->iotlb_hits, cfg->iotlb_misses, +100 * cfg->iotlb_hits / + (cfg->iotlb_hits + cfg->iotlb_misses)); + }else { + cfg->iotlb_misses++; +trace_smmu_iotlb_lookup_miss (cfg->asid, iova, + cfg->iotlb_hits, cfg->iotlb_misses, +100 * cfg->iotlb_hits / + (cfg->iotlb_hits + cfg->iotlb_misses)); + } +return entry; +} + +void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,IOMMUTLBEntry *entry) +{ +SMMUIOTLBKey *key =g_new0 (SMMUIOTLBKey,1 ); + +if (g_hash_table_size (bs->iotlb) >=SMMU_IOTLB_MAX_SIZE ) { +smmu_iotlb_inv_all (bs); + } + + key->asid = cfg->asid; + key->iova = entry->iova; +trace_smmu_iotlb_insert (cfg->asid, entry->iova); +g_hash_table_insert (bs->iotlb, key, entry); +} +inline void smmu_iotlb_inv_all (SMMUState *s) {trace_smmu_iotlb_inv_all ();
hw/arm/smmuv3.c +2 -24
@@ -636,7 +636,6 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, .addr_mask = ~(hwaddr)0, .perm = IOMMU_NONE, }; -SMMUIOTLBKey key, *new_key;qemu_mutex_lock (&s->mutex);
@@ -675,16 +674,8 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, page_mask = (1ULL << (tt->granule_sz)) -1 ; aligned_addr = addr & ~page_mask; - key.asid = cfg->asid; - key.iova = aligned_addr; - - cached_entry =g_hash_table_lookup (bs->iotlb, &key); + cached_entry =smmu_iotlb_lookup (bs, cfg, aligned_addr);if (cached_entry) { - cfg->iotlb_hits++; -trace_smmu_iotlb_cache_hit (cfg->asid, aligned_addr, - cfg->iotlb_hits, cfg->iotlb_misses, -100 * cfg->iotlb_hits / - (cfg->iotlb_hits + cfg->iotlb_misses));if ((flag &IOMMU_WO ) && !(cached_entry->perm &IOMMU_WO )) { status =SMMU_TRANS_ERROR ;if (event.record_trans_faults) {
@@ -698,16 +689,6 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, goto epilogue; } - cfg->iotlb_misses++; -trace_smmu_iotlb_cache_miss (cfg->asid, addr & ~page_mask, - cfg->iotlb_hits, cfg->iotlb_misses, -100 * cfg->iotlb_hits / - (cfg->iotlb_hits + cfg->iotlb_misses)); - -if (g_hash_table_size (bs->iotlb) >=SMMU_IOTLB_MAX_SIZE ) { -smmu_iotlb_inv_all (bs); - } - cached_entry =g_new0 (IOMMUTLBEntry,1 );if (smmu_ptw (cfg, aligned_addr, flag, cached_entry, &ptw_info)) {
@@ -753,10 +734,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, } status =SMMU_TRANS_ERROR ; }else { - new_key =g_new0 (SMMUIOTLBKey,1 ); - new_key->asid = cfg->asid; - new_key->iova = aligned_addr; -g_hash_table_insert (bs->iotlb, new_key, cached_entry); +smmu_iotlb_insert (bs, cfg, cached_entry); status =SMMU_TRANS_SUCCESS ; }
hw/arm/trace-events +3 -2
@@ -14,6 +14,9 @@ smmu_iotlb_inv_all(void) "IOTLB invalidate all" smmu_iotlb_inv_asid(uint16_t asid) "IOTLB invalidate asid=%d" smmu_iotlb_inv_iova(uint16_t asid, uint64_t addr) "IOTLB invalidate asid=%d addr=0x%"PRIx64 smmu_inv_notifiers_mr(const char *name) "iommu mr=%s" +smmu_iotlb_lookup_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" +smmu_iotlb_lookup_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" +smmu_iotlb_insert(uint16_t asid, uint64_t addr) "IOTLB ++ asid=%d addr=0x%"PRIx64 # smmuv3.c smmuv3_read_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
@@ -46,8 +49,6 @@ smmuv3_cmdq_tlbi_nh_va(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d a smmuv3_cmdq_tlbi_nh_vaa(int vmid, uint64_t addr) "vmid =%d addr=0x%"PRIx64 smmuv3_cmdq_tlbi_nh(void) "" smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d" -smmu_iotlb_cache_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" -smmu_iotlb_cache_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d" smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s" smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s"
Signature · 1 line
2.21.3
PATCH v4 3/11 hw/arm/smmu: Introduce smmu_get_iotlb_key()
In reply to Eric AugerIntroduce the smmu_get_iotlb_key() helper and the SMMU_IOTLB_ASID() macro. Also move smmu_get_iotlb_key and smmu_iotlb_key_hash in the IOTLB related code section.
---
v1 -> v2:
- creation
--- hw/arm/smmu-internal.h | 1 + include/hw/arm/smmu-common.h | 1 + hw/arm/smmu-common.c | 66 ++++++++++++++++++++---------------- 3 files changed, 38 insertions(+), 30 deletions(-)
hw/arm/smmu-internal.h +1 -0
@@ -96,4 +96,5 @@ uint64_t iova_level_offset(uint64_t iova, int inputsize, MAKE_64BIT_MASK(0, gsz - 3); } +#define SMMU_IOTLB_ASID (key) ((key).asid)#endif
include/hw/arm/smmu-common.h +1 -0
@@ -155,6 +155,7 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);IOMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg,hwaddr iova);void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,IOMMUTLBEntry *entry); +SMMUIOTLBKey smmu_get_iotlb_key (uint16_t asid,uint64_t iova);void smmu_iotlb_inv_all (SMMUState *s);void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid);void smmu_iotlb_inv_iova (SMMUState *s,uint16_t asid,dma_addr_t iova);
hw/arm/smmu-common.c +36 -30
97 lines not on this page · raw message
Signature · 1 line
2.21.3
PATCH v4 4/11 hw/arm/smmu: Introduce SMMUTLBEntry for PTW and IOTLB value
In reply to Eric AugerIntroduce a specialized SMMUTLBEntry to store the result of the PTW and cache in the IOTLB. This structure extends the generic IOMMUTLBEntry struct with the level of the entry and the granule size.
Those latter will be useful when implementing range invalidation.
---
v1 -> v2:
- fix 2 indents and typdef
- added Peter's R-b
--- include/hw/arm/smmu-common.h | 12 +++++++++--- hw/arm/smmu-common.c | 32 +++++++++++++++++--------------- hw/arm/smmuv3.c | 10 +++++----- 3 files changed, 31 insertions(+), 23 deletions(-)
include/hw/arm/smmu-common.h +9 -3
@@ -52,6 +52,12 @@ typedef struct SMMUTransTableInfo {uint8_t granule_sz;/* granule page shift */ } SMMUTransTableInfo; +typedef struct SMMUTLBEntry { +IOMMUTLBEntry entry; +uint8_t level; +uint8_t granule; +}SMMUTLBEntry ; +/* * Generic structure populated by derived SMMU devices * after decoding the configuration information and used as
@@ -140,7 +146,7 @@ static inline uint16_t smmu_get_sid(SMMUDevice *sdev)* pair, according to @cfg translation config */ int smmu_ptw (SMMUTransCfg *cfg,dma_addr_t iova,IOMMUAccessFlags perm, -IOMMUTLBEntry *tlbe,SMMUPTWEventInfo *info); +SMMUTLBEntry *tlbe,SMMUPTWEventInfo *info);/** * select_tt - compute which translation table shall be used according to
@@ -153,8 +159,8 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);#define SMMU_IOTLB_MAX_SIZE 256 -IOMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg,hwaddr iova); -void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,IOMMUTLBEntry *entry); +SMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg,hwaddr iova); +void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,SMMUTLBEntry *entry);SMMUIOTLBKey smmu_get_iotlb_key (uint16_t asid,uint64_t iova);void smmu_iotlb_inv_all (SMMUState *s);void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid);
hw/arm/smmu-common.c +17 -15
@@ -64,11 +64,11 @@ SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova)return key; } -IOMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg, -hwaddr iova) +SMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg, +hwaddr iova) {SMMUIOTLBKey key =smmu_get_iotlb_key (cfg->asid, iova); -IOMMUTLBEntry *entry =g_hash_table_lookup (bs->iotlb, &key); +SMMUTLBEntry *entry =g_hash_table_lookup (bs->iotlb, &key);if (entry) { cfg->iotlb_hits++;
@@ -86,7 +86,7 @@ IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,return entry; } -void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,IOMMUTLBEntry *entry) +void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,SMMUTLBEntry *new) {SMMUIOTLBKey *key =g_new0 (SMMUIOTLBKey,1 );
@@ -94,9 +94,9 @@ void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry)smmu_iotlb_inv_all (bs); } - *key =smmu_get_iotlb_key (cfg->asid, entry->iova); -trace_smmu_iotlb_insert (cfg->asid, entry->iova); -g_hash_table_insert (bs->iotlb, key, entry); + *key =smmu_get_iotlb_key (cfg->asid, new->entry.iova); +trace_smmu_iotlb_insert (cfg->asid, new->entry.iova); +g_hash_table_insert (bs->iotlb, key, new); }inline void smmu_iotlb_inv_all (SMMUState *s)
@@ -216,7 +216,7 @@ SMMUTransTableInfo *select_tt(SMMUTransCfg *cfg, dma_addr_t iova)* @cfg: translation config * @iova: iova to translate * @perm: access type -* @tlbe: IOMMUTLBEntry (out) +* @tlbe: SMMUTLBEntry (out) * @info: handle to an error info * * Return 0 on success, < 0 on error. In case of error, @info is filled
@@ -226,7 +226,7 @@ SMMUTransTableInfo *select_tt(SMMUTransCfg *cfg, dma_addr_t iova)*/ static int smmu_ptw_64 (SMMUTransCfg *cfg,dma_addr_t iova,IOMMUAccessFlags perm, -IOMMUTLBEntry *tlbe,SMMUPTWEventInfo *info) +SMMUTLBEntry *tlbe,SMMUPTWEventInfo *info) {dma_addr_t baseaddr, indexmask;int stage = cfg->stage;
@@ -246,8 +246,8 @@ static int smmu_ptw_64(SMMUTransCfg *cfg, baseaddr =extract64 (tt->ttb,0 ,48 ); baseaddr &= ~indexmask; - tlbe->iova = iova; - tlbe->addr_mask = (1 << granule_sz) -1 ; + tlbe->entry.iova = iova; + tlbe->entry.addr_mask = (1 << granule_sz) -1 ;while (level <=3 ) {uint64_t subpage_size =1ULL <<level_shift (level, granule_sz);
@@ -298,14 +298,16 @@ static int smmu_ptw_64(SMMUTransCfg *cfg, goto error; } - tlbe->translated_addr = gpa + (iova & mask); - tlbe->perm =PTE_AP_TO_PERM (ap); + tlbe->entry.translated_addr = gpa + (iova & mask); + tlbe->entry.perm =PTE_AP_TO_PERM (ap); + tlbe->level = level; + tlbe->granule = granule_sz;return 0 ; } info->type =SMMU_PTW_ERR_TRANSLATION ; error: - tlbe->perm =IOMMU_NONE ; + tlbe->entry.perm =IOMMU_NONE ;return -EINVAL ; }
@@ -321,7 +323,7 @@ error:* return 0 on success */ inline int smmu_ptw (SMMUTransCfg *cfg,dma_addr_t iova,IOMMUAccessFlags perm, -IOMMUTLBEntry *tlbe,SMMUPTWEventInfo *info) +SMMUTLBEntry *tlbe,SMMUPTWEventInfo *info) {if (!cfg->aa64) {/*
hw/arm/smmuv3.c +5 -5
@@ -626,7 +626,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,SMMUTranslationStatus status;SMMUState *bs =ARM_SMMU (s);uint64_t page_mask, aligned_addr; -IOMMUTLBEntry *cached_entry =NULL ; +SMMUTLBEntry *cached_entry =NULL ;SMMUTransTableInfo *tt;SMMUTransCfg *cfg =NULL ;IOMMUTLBEntry entry = {
@@ -676,7 +676,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, cached_entry =smmu_iotlb_lookup (bs, cfg, aligned_addr);if (cached_entry) { -if ((flag &IOMMU_WO ) && !(cached_entry->perm &IOMMU_WO )) { +if ((flag &IOMMU_WO ) && !(cached_entry->entry.perm &IOMMU_WO )) { status =SMMU_TRANS_ERROR ;if (event.record_trans_faults) { event.type =SMMU_EVT_F_PERMISSION ;
@@ -689,7 +689,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, goto epilogue; } - cached_entry =g_new0 (IOMMUTLBEntry,1 ); + cached_entry =g_new0 (SMMUTLBEntry,1 );if (smmu_ptw (cfg, aligned_addr, flag, cached_entry, &ptw_info)) {g_free (cached_entry);
@@ -743,9 +743,9 @@ epilogue:switch (status) {case SMMU_TRANS_SUCCESS : entry.perm = flag; - entry.translated_addr = cached_entry->translated_addr + + entry.translated_addr = cached_entry->entry.translated_addr + (addr & page_mask); - entry.addr_mask = cached_entry->addr_mask; + entry.addr_mask = cached_entry->entry.addr_mask;trace_smmuv3_translate_success (mr->parent_obj.name, sid, addr, entry.translated_addr, entry.perm);break ;
Signature · 1 line
2.21.3
PATCH v4 5/11 hw/arm/smmu-common: Manage IOTLB block entries
In reply to Eric AugerAt the moment each entry in the IOTLB corresponds to a page sized mapping (4K, 16K or 64K), even if the page belongs to a mapped block. In case of block mapping this unefficiently consumes IOTLB entries.
Change the value of the entry so that it reflects the actual mapping it belongs to (block or page start address and size).
Also the level/tg of the entry is encoded in the key. In subsequent patches we will enable range invalidation. This latter is able to provide the level/tg of the entry.
Encoding the level/tg directly in the key will allow to invalidate using g_hash_table_remove() when num_pages equals to 1.
---
v3 -> v4:
- also use the tg field when computing the hash
- in the key equal function, compare the fields instead of using memcmp()
- fixed a couple of indents
- added Peter's R-b
v2 -> v3:
- simplify the logic in smmu_hash_remove_by_asid_iova as suggested by Peter
- the key is a struct. We take into account the lvl in the jenkins hash function. Also the equal function is updated.
v1 -> v2:
- recompute starting_level
--- hw/arm/smmu-internal.h | 7 ++++ include/hw/arm/smmu-common.h | 10 ++++-- hw/arm/smmu-common.c | 67 ++++++++++++++++++++++++++---------- hw/arm/smmuv3.c | 6 ++-- hw/arm/trace-events | 2 +- 5 files changed, 67 insertions(+), 25 deletions(-)
hw/arm/smmu-internal.h +7 -0
@@ -97,4 +97,11 @@ uint64_t iova_level_offset(uint64_t iova, int inputsize, }#define SMMU_IOTLB_ASID (key) ((key).asid) + +typedef struct SMMUIOTLBPageInvInfo { +int asid; +uint64_t iova; +uint64_t mask; +}SMMUIOTLBPageInvInfo ; +#endif
include/hw/arm/smmu-common.h +7 -3
@@ -97,6 +97,8 @@ typedef struct SMMUPciBus {typedef struct SMMUIOTLBKey {uint64_t iova;uint16_t asid; +uint8_t tg; +uint8_t level; }SMMUIOTLBKey ; typedef struct SMMUState {
@@ -159,12 +161,14 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);#define SMMU_IOTLB_MAX_SIZE 256 -SMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg,hwaddr iova); +SMMUTLBEntry *smmu_iotlb_lookup (SMMUState *bs,SMMUTransCfg *cfg, +SMMUTransTableInfo *tt,hwaddr iova);void smmu_iotlb_insert (SMMUState *bs,SMMUTransCfg *cfg,SMMUTLBEntry *entry); -SMMUIOTLBKey smmu_get_iotlb_key (uint16_t asid,uint64_t iova); +SMMUIOTLBKey smmu_get_iotlb_key (uint16_t asid,uint64_t iova, +uint8_t tg,uint8_t level);void smmu_iotlb_inv_all (SMMUState *s);void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid); -void smmu_iotlb_inv_iova (SMMUState *s,uint16_t asid,dma_addr_t iova); +void smmu_iotlb_inv_iova (SMMUState *s,int asid,dma_addr_t iova);/* Unmap the range of all the notifiers registered to any IOMMU mr */ void smmu_inv_notifiers_all (SMMUState *s);
hw/arm/smmu-common.c +49 -18
121 lines not on this page · raw message
hw/arm/smmuv3.c +3 -3
@@ -674,7 +674,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr, page_mask = (1ULL << (tt->granule_sz)) -1 ; aligned_addr = addr & ~page_mask; - cached_entry =smmu_iotlb_lookup (bs, cfg, aligned_addr); + cached_entry =smmu_iotlb_lookup (bs, cfg, tt, aligned_addr);if (cached_entry) {if ((flag &IOMMU_WO ) && !(cached_entry->entry.perm &IOMMU_WO )) { status =SMMU_TRANS_ERROR ;
@@ -744,7 +744,7 @@ epilogue:case SMMU_TRANS_SUCCESS : entry.perm = flag; entry.translated_addr = cached_entry->entry.translated_addr + - (addr & page_mask); + (addr & cached_entry->entry.addr_mask); entry.addr_mask = cached_entry->entry.addr_mask;trace_smmuv3_translate_success (mr->parent_obj.name, sid, addr, entry.translated_addr, entry.perm);
@@ -972,7 +972,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)trace_smmuv3_cmdq_tlbi_nh_vaa (vmid, addr);smmuv3_inv_notifiers_iova (bs,-1 , addr); -smmu_iotlb_inv_all (bs); +smmu_iotlb_inv_iova (bs,-1 , addr);break ; }case SMMU_CMD_TLBI_NH_VA :
hw/arm/trace-events +1 -1
@@ -16,7 +16,7 @@ smmu_iotlb_inv_iova(uint16_t asid, uint64_t addr) "IOTLB invalidate asid=%d addr smmu_inv_notifiers_mr(const char *name) "iommu mr=%s" smmu_iotlb_lookup_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" smmu_iotlb_lookup_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d" -smmu_iotlb_insert(uint16_t asid, uint64_t addr) "IOTLB ++ asid=%d addr=0x%"PRIx64 +smmu_iotlb_insert(uint16_t asid, uint64_t addr, uint8_t tg, uint8_t level) "IOTLB ++ asid=%d addr=0x%"PRIx64" tg=%d level=%d" # smmuv3.c smmuv3_read_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
Signature · 1 line
2.21.3
Re: [PATCH for-5.2 v4 05/11] hw/arm/smmu-common: Manage IOTLB block entries
Quoted 18 lines · Eric Auger
At the moment each entry in the IOTLB corresponds to a page sized mapping (4K, 16K or 64K), even if the page belongs to a mapped block. In case of block mapping this unefficiently consumes IOTLB entries.
Change the value of the entry so that it reflects the actual mapping it belongs to (block or page start address and size).
Also the level/tg of the entry is encoded in the key. In subsequent patches we will enable range invalidation. This latter is able to provide the level/tg of the entry.
Encoding the level/tg directly in the key will allow to invalidate using
g_hash_table_remove()whennum_pagesequals to 1.Signed-off-by Eric Auger <[email protected]>
Quoted 7 lines
---
v3 -> v4:
- also use the tg field when computing the hash
- in the key equal function, compare the fields instead of usingmemcmp()
- fixed a couple of indents
- added Peter's R-b
You say that, but you didn't :-)
thanks
-- PMM
Re: [PATCH for-5.2 v4 05/11] hw/arm/smmu-common: Manage IOTLB block entries
Hi Peter,
Quoted 27 lines · Peter Maydell
Quoted 18 lines · Eric Auger
At the moment each entry in the IOTLB corresponds to a page sized mapping (4K, 16K or 64K), even if the page belongs to a mapped block. In case of block mapping this unefficiently consumes IOTLB entries.
Change the value of the entry so that it reflects the actual mapping it belongs to (block or page start address and size).
Also the level/tg of the entry is encoded in the key. In subsequent patches we will enable range invalidation. This latter is able to provide the level/tg of the entry.
Encoding the level/tg directly in the key will allow to invalidate using
g_hash_table_remove()whennum_pagesequals to 1.Signed-off-by Eric Auger <[email protected]>Quoted 7 lines
---
v3 -> v4:
- also use the tg field when computing the hash
- in the key equal function, compare the fields instead of usingmemcmp()
- fixed a couple of indents
- added Peter's R-b
Quoted 1 line
You say that, but you didn't :-)
Please forgive me for the oversight.
Thank you for the review.
Eric
Quoted 6 lines
Reviewed-by Peter Maydell <[email protected]>thanks
-- PMM
PATCH v4 6/11 hw/arm/smmuv3: Introduce smmuv3_s1_range_inval() helper
In reply to Eric AugerLet's introduce an helper for S1 IOVA range invalidation. This will be used for NH_VA and NH_VAA commands. It decodes the same fields, trace, calls the UNMAP notifiers and invalidate the corresponding IOTLB entries.
At the moment, we do not support 3.2 range invalidation yet. So it reduces to a single IOVA invalidation.
Note the leaf bit now is also decoded for the CMD_TLBI_NH_VAA command. At the moment it is only used for tracing.
---
v1 -> v2:
- added comment about leaf bit and added Peter's R-b
--- hw/arm/smmuv3.c | 36 +++++++++++++++++------------------- hw/arm/trace-events | 3 +-- 2 files changed, 18 insertions(+), 21 deletions(-)
hw/arm/smmuv3.c +17 -19
@@ -836,6 +836,22 @@ static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, dma_addr_t iova) } } +static void smmuv3_s1_range_inval (SMMUState *s,Cmd *cmd) +{ +dma_addr_t addr =CMD_ADDR (cmd); +uint8_t type =CMD_TYPE (cmd); +uint16_t vmid =CMD_VMID (cmd); +bool leaf =CMD_LEAF (cmd); +int asid =-1 ; + +if (type ==SMMU_CMD_TLBI_NH_VA ) { + asid =CMD_ASID (cmd); + } +trace_smmuv3_s1_range_inval (vmid, asid, addr, leaf); +smmuv3_inv_notifiers_iova (s, asid, addr); +smmu_iotlb_inv_iova (s, asid, addr); +} +static int smmuv3_cmdq_consume (SMMUv3State *s) {SMMUState *bs =ARM_SMMU (s);
@@ -966,27 +982,9 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)smmu_iotlb_inv_all (bs);break ;case SMMU_CMD_TLBI_NH_VAA : - { -dma_addr_t addr =CMD_ADDR (&cmd); -uint16_t vmid =CMD_VMID (&cmd); - -trace_smmuv3_cmdq_tlbi_nh_vaa (vmid, addr); -smmuv3_inv_notifiers_iova (bs,-1 , addr); -smmu_iotlb_inv_iova (bs,-1 , addr); -break ; - }case SMMU_CMD_TLBI_NH_VA : - { -uint16_t asid =CMD_ASID (&cmd); -uint16_t vmid =CMD_VMID (&cmd); -dma_addr_t addr =CMD_ADDR (&cmd); -bool leaf =CMD_LEAF (&cmd); - -trace_smmuv3_cmdq_tlbi_nh_va (vmid, asid, addr, leaf); -smmuv3_inv_notifiers_iova (bs, asid, addr); -smmu_iotlb_inv_iova (bs, asid, addr); +smmuv3_s1_range_inval (bs, &cmd);break ; - }case SMMU_CMD_TLBI_EL3_ALL :case SMMU_CMD_TLBI_EL3_VA :case SMMU_CMD_TLBI_EL2_ALL :
hw/arm/trace-events +1 -2
@@ -45,8 +45,7 @@ smmuv3_cmdq_cfgi_ste_range(int start, int end) "start=0x%d - end=0x%d" smmuv3_cmdq_cfgi_cd(uint32_t sid) "streamid = %d" smmuv3_config_cache_hit(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache HIT for sid %d (hits=%d, misses=%d, hit rate=%d)" smmuv3_config_cache_miss(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache MISS for sid %d (hits=%d, misses=%d, hit rate=%d)" -smmuv3_cmdq_tlbi_nh_va(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d" -smmuv3_cmdq_tlbi_nh_vaa(int vmid, uint64_t addr) "vmid =%d addr=0x%"PRIx64 +smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d" smmuv3_cmdq_tlbi_nh(void) "" smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d" smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d"
Signature · 1 line
2.21.3
PATCH v4 7/11 hw/arm/smmuv3: Get prepared for range invalidation
In reply to Eric AugerEnhance the smmu_iotlb_inv_iova() helper with range invalidation. This uses the new fields passed in the NH_VA and NH_VAA commands:
the size of the range, the level and the granule.
As NH_VA and NH_VAA both use those fields, their decoding and handling is factorized in a new smmuv3_s1_range_inval() helper.
---
v3 -> v4:
- Fixed over identation
- added Peter's R-b
--- hw/arm/smmuv3-internal.h | 4 +++ include/hw/arm/smmu-common.h | 3 +- hw/arm/smmu-common.c | 25 +++++++++++--- hw/arm/smmuv3.c | 64 +++++++++++++++++++++++------------- hw/arm/trace-events | 4 +-- 5 files changed, 69 insertions(+), 31 deletions(-)
hw/arm/smmuv3-internal.h +4 -0
@@ -298,6 +298,8 @@ enum { /* Command completion notification */ };#define CMD_TYPE (x) extract32((x)->word[0], 0 , 8) +#define CMD_NUM (x) extract32((x)->word[0], 12 , 5) +#define CMD_SCALE (x) extract32((x)->word[0], 20 , 5)#define CMD_SSEC (x) extract32((x)->word[0], 10, 1)#define CMD_SSV (x) extract32((x)->word[0], 11, 1)#define CMD_RESUME_AC (x) extract32((x)->word[0], 12, 1)
@@ -310,6 +312,8 @@ enum { /* Command completion notification */#define CMD_RESUME_STAG (x) extract32((x)->word[2], 0 , 16)#define CMD_RESP (x) extract32((x)->word[2], 11, 2)#define CMD_LEAF (x) extract32((x)->word[2], 0 , 1) +#define CMD_TTL (x) extract32((x)->word[2], 8 , 2) +#define CMD_TG (x) extract32((x)->word[2], 10, 2)#define CMD_STE_RANGE (x) extract32((x)->word[2], 0 , 5)#define CMD_ADDR (x) ({ \ uint64_t high = (uint64_t)(x)->word[3]; \
include/hw/arm/smmu-common.h +2 -1
@@ -168,7 +168,8 @@ SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova, uint8_t tg, uint8_t level);void smmu_iotlb_inv_all (SMMUState *s);void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid); -void smmu_iotlb_inv_iova (SMMUState *s,int asid,dma_addr_t iova); +void smmu_iotlb_inv_iova (SMMUState *s,int asid,dma_addr_t iova, +uint8_t tg,uint64_t num_pages,uint8_t ttl);/* Unmap the range of all the notifiers registered to any IOMMU mr */ void smmu_inv_notifiers_all (SMMUState *s);
hw/arm/smmu-common.c +20 -5
@@ -143,15 +143,30 @@ static gboolean smmu_hash_remove_by_asid_iova(gpointer key, gpointer value,if (info->asid >=0 && info->asid !=SMMU_IOTLB_ASID (iotlb_key)) {return false; } -return (info->iova & ~entry->addr_mask) == entry->iova; +return ((info->iova & ~entry->addr_mask) == entry->iova) || + ((entry->iova & ~info->mask) == info->iova); } -inline void smmu_iotlb_inv_iova (SMMUState *s,int asid,dma_addr_t iova) +inline void +smmu_iotlb_inv_iova (SMMUState *s,int asid,dma_addr_t iova, +uint8_t tg,uint64_t num_pages,uint8_t ttl) { -SMMUIOTLBPageInvInfo info = {.asid = asid, .iova = iova}; +if (ttl && (num_pages ==1 )) { +SMMUIOTLBKey key =smmu_get_iotlb_key (asid, iova, tg, ttl); -trace_smmu_iotlb_inv_iova (asid, iova); -g_hash_table_foreach_remove (s->iotlb, smmu_hash_remove_by_asid_iova, &info); +g_hash_table_remove (s->iotlb, &key); + }else { +/* if tg is not set we use 4KB range invalidation */ +uint8_t granule = tg ? tg *2 +10 :12 ; + +SMMUIOTLBPageInvInfo info = { + .asid = asid, .iova = iova, + .mask = (num_pages *1 << granule) -1 }; + +g_hash_table_foreach_remove (s->iotlb, + smmu_hash_remove_by_asid_iova, + &info); + } }inline void smmu_iotlb_inv_asid (SMMUState *s,uint16_t asid)
hw/arm/smmuv3.c +41 -23
111 lines not on this page · raw message
hw/arm/trace-events +2 -2
@@ -45,11 +45,11 @@ smmuv3_cmdq_cfgi_ste_range(int start, int end) "start=0x%d - end=0x%d" smmuv3_cmdq_cfgi_cd(uint32_t sid) "streamid = %d" smmuv3_config_cache_hit(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache HIT for sid %d (hits=%d, misses=%d, hit rate=%d)" smmuv3_config_cache_miss(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache MISS for sid %d (hits=%d, misses=%d, hit rate=%d)" -smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d" +smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, uint8_t tg, uint64_t num_pages, uint8_t ttl, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64" ttl=%d leaf=%d" smmuv3_cmdq_tlbi_nh(void) "" smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d" smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d" smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s" smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s" -smmuv3_inv_notifiers_iova(const char *name, uint16_t asid, uint64_t iova) "iommu mr=%s asid=%d iova=0x%"PRIx64 +smmuv3_inv_notifiers_iova(const char *name, uint16_t asid, uint64_t iova, uint8_t tg, uint64_t num_pages) "iommu mr=%s asid=%d iova=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64
Signature · 1 line
2.21.3
PATCH v4 8/11 hw/arm/smmuv3: Fix IIDR offset
In reply to Eric AugerThe SMMU IIDR register is at 0x018 offset.
--- hw/arm/smmuv3-internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/arm/smmuv3-internal.h +1 -1
@@ -63,7 +63,7 @@ REG32(IDR5, 0x14)#define SMMU_IDR5_OAS 4 -REG32(IIDR, 0x1c) +REG32(IIDR, 0x18) REG32(CR0, 0x20) FIELD(CR0, SMMU_ENABLE, 0, 1)FIELD (CR0 ,EVENTQEN ,2 ,1 )
Signature · 1 line
2.21.3
PATCH v4 9/11 hw/arm/smmuv3: Let AIDR advertise SMMUv3.0 support
In reply to Eric AugerAdd the support for AIDR register. It currently advertises SMMU V3.0 spec.
---
v3 -> v4:
- upper -> lower case
- added Peter's R-b
--- hw/arm/smmuv3-internal.h | 1 + include/hw/arm/smmuv3.h | 1 + hw/arm/smmuv3.c | 3 +++ 3 files changed, 5 insertions(+)
hw/arm/smmuv3-internal.h +1 -0
@@ -64,6 +64,7 @@ REG32(IDR5, 0x14)#define SMMU_IDR5_OAS 4 REG32(IIDR, 0x18) +REG32(AIDR, 0x1c) REG32(CR0, 0x20) FIELD(CR0, SMMU_ENABLE, 0, 1)FIELD (CR0 ,EVENTQEN ,2 ,1 )
include/hw/arm/smmuv3.h +1 -0
@@ -41,6 +41,7 @@ typedef struct SMMUv3State {uint32_t idr[6 ];uint32_t iidr; +uint32_t aidr;uint32_t cr[3 ];uint32_t cr0ack;uint32_t statusr;
hw/arm/smmuv3.c +3 -0
@@ -1251,6 +1251,9 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,case A_IIDR : *data = s->iidr;return MEMTX_OK ; +case A_AIDR : + *data = s->aidr; +return MEMTX_OK ;case A_CR0 : *data = s->cr[0 ];return MEMTX_OK ;
Signature · 1 line
2.21.3
PATCH v4 10/11 hw/arm/smmuv3: Support HAD and advertise SMMUv3.1 support
In reply to Eric AugerHAD is a mandatory features with SMMUv3.1 if S1P is set, which is our case. Other 3.1 mandatory features come with S2P which we don't have.
So let's support HAD and advertise SMMUv3.1 support in AIDR.
HAD support allows the CD to disable hierarchical attributes, ie. if the HAD0/1 bit is set, the APTable field of table descriptors walked through TTB0/1 is ignored.
--- hw/arm/smmuv3-internal.h | 2 ++ include/hw/arm/smmu-common.h | 1 + hw/arm/smmu-common.c | 2 +- hw/arm/smmuv3.c | 6 +++++- hw/arm/trace-events | 2 +- 5 files changed, 10 insertions(+), 3 deletions(-)
hw/arm/smmuv3-internal.h +2 -0
@@ -54,6 +54,7 @@ REG32(IDR1, 0x4) REG32(IDR2, 0x8) REG32(IDR3, 0xc) +FIELD (IDR3 ,HAD ,2 ,1 ); REG32(IDR4, 0x10) REG32(IDR5, 0x14)FIELD (IDR5 ,OAS ,0 ,3 );
@@ -578,6 +579,7 @@ static inline int pa_range(STE *ste) lo = (x)->word[(sel) *2 +2 ] & ~0xfULL ; \ hi | lo; \ }) +#define CD_HAD (x, sel) extract32((x)->word[(sel) * 2 + 2], 1, 1)#define CD_TSZ (x, sel) extract32((x)->word[0], (16 * (sel)) + 0, 6)#define CD_TG (x, sel) extract32((x)->word[0], (16 * (sel)) + 6, 2)
include/hw/arm/smmu-common.h +1 -0
@@ -50,6 +50,7 @@ typedef struct SMMUTransTableInfo {uint64_t ttb;/* TT base address */ uint8_t tsz;/* input range, ie. 2^(64 -tsz)*/ uint8_t granule_sz;/* granule page shift */ +bool had;/* hierarchical attribute disable */ } SMMUTransTableInfo; typedef struct SMMUTLBEntry {
hw/arm/smmu-common.c +1 -1
@@ -316,7 +316,7 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,if (is_table_pte (pte, level)) { ap =PTE_APTABLE (pte); -if (is_permission_fault (ap, perm)) { +if (is_permission_fault (ap, perm) && !tt->had) { info->type =SMMU_PTW_ERR_PERMISSION ; goto error; }
hw/arm/smmuv3.c +5 -1
@@ -254,6 +254,8 @@ static void smmuv3_init_regs(SMMUv3State *s) s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,EVENTQS ,SMMU_EVENTQS ); s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,SMMU_CMDQS ); + s->idr[3 ] =FIELD_DP32 (s->idr[3 ],IDR3 ,HAD ,1 ); +/* 4K and 64K granule support */ s->idr[5 ] =FIELD_DP32 (s->idr[5 ],IDR5 ,GRAN4K ,1 ); s->idr[5 ] =FIELD_DP32 (s->idr[5 ],IDR5 ,GRAN64K ,1 );
@@ -270,6 +272,7 @@ static void smmuv3_init_regs(SMMUv3State *s) s->features =0 ; s->sid_split =0 ; + s->aidr =0x1 ; } static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
@@ -506,7 +509,8 @@ static int decode_cd(SMMUTransCfg *cfg, CD *cd, SMMUEventInfo *event)if (tt->ttb & ~(MAKE_64BIT_MASK (0 , cfg->oas))) { goto bad_cd; } -trace_smmuv3_decode_cd_tt (i, tt->tsz, tt->ttb, tt->granule_sz); + tt->had =CD_HAD (cd, i); +trace_smmuv3_decode_cd_tt (i, tt->tsz, tt->ttb, tt->granule_sz, tt->had); } event->record_trans_faults =CD_R (cd);
hw/arm/trace-events +1 -1
@@ -39,7 +39,7 @@ smmuv3_translate_abort(const char *n, uint16_t sid, uint64_t addr, bool is_write smmuv3_translate_success(const char *n, uint16_t sid, uint64_t iova, uint64_t translated, int perm) "%s sid=%d iova=0x%"PRIx64" translated=0x%"PRIx64" perm=0x%x" smmuv3_get_cd(uint64_t addr) "CD addr: 0x%"PRIx64 smmuv3_decode_cd(uint32_t oas) "oas=%d" -smmuv3_decode_cd_tt(int i, uint32_t tsz, uint64_t ttb, uint32_t granule_sz) "TT[%d]:tsz:%d ttb:0x%"PRIx64" granule_sz:%d" +smmuv3_decode_cd_tt(int i, uint32_t tsz, uint64_t ttb, uint32_t granule_sz, bool had) "TT[%d]:tsz:%d ttb:0x%"PRIx64" granule_sz:%d had:%d" smmuv3_cmdq_cfgi_ste(int streamid) "streamid =%d" smmuv3_cmdq_cfgi_ste_range(int start, int end) "start=0x%d - end=0x%d" smmuv3_cmdq_cfgi_cd(uint32_t sid) "streamid = %d"
Signature · 1 line
2.21.3
PATCH v4 11/11 hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation
In reply to Eric AugerExpose the RIL bit so that the guest driver uses range invalidation. Although RIL is a 3.2 features, We let the AIDR advertise SMMUv3.1 support as v3.x implementation is allowed to implement features from v3.(x+1).
---
v2 -> v3:
- keep the AIDR equal to 2. BBML support, which is 3.2 mandatory feature will be addressed in a separate patch.
- added Peter's R-b (same version as in v1)
--- hw/arm/smmuv3-internal.h | 1 + hw/arm/smmuv3.c | 1 + 2 files changed, 2 insertions(+)
hw/arm/smmuv3-internal.h +1 -0
@@ -55,6 +55,7 @@ REG32(IDR1, 0x4) REG32(IDR2, 0x8) REG32(IDR3, 0xc)FIELD (IDR3 ,HAD ,2 ,1 ); +FIELD (IDR3 ,RIL ,10 ,1 ); REG32(IDR4, 0x10) REG32(IDR5, 0x14)FIELD (IDR5 ,OAS ,0 ,3 );
hw/arm/smmuv3.c +1 -0
@@ -254,6 +254,7 @@ static void smmuv3_init_regs(SMMUv3State *s) s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,EVENTQS ,SMMU_EVENTQS ); s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,SMMU_CMDQS ); + s->idr[3 ] =FIELD_DP32 (s->idr[3 ],IDR3 ,RIL ,1 ); s->idr[3 ] =FIELD_DP32 (s->idr[3 ],IDR3 ,HAD ,1 );/* 4K and 64K granule support */
Signature · 1 line
2.21.3
Quoted 15 lines · Eric Auger
SMMU3.2 brings the support of range-based TLB invalidation and level hint. When this feature is supported, the SMMUv3 driver is allowed to send TLB invalidations for a range of IOVAs instead of using page based invalidation.
Implementing this feature in the virtual SMMUv3 device is mandated for DPDK on guest use case: DPDK uses hugepage buffers and guest sends invalidations for blocks. Without this feature, a guest invalidation of a block of 1GB for instance translates into a storm of page invalidations. Each of them is trapped by the VMM and cascaded downto the physical IOMMU. This completely stalls the execution. This integration issue was initially reported in [1].
Quoted 8 lines
Now SMMUv3.2 specifies additional parameters to
NH_VAandNH_VAAstage 1 invalidation commands so we can support those extensions.Supporting block mappings in the IOTLB look sensible in terms of TLB entry consumption. However looking at virtio/vhost device usage, without block mapping and without range invalidation (< 5.7 kernels it may be less performant. However for recent guest kernels supporting range invalidations [2], the performance should be similar.
I think this is all reviewed now; I've put it on my list of series to apply to target-arm.next in once 5.1 is out and we reopen the trunk for 5.2.
thanks
-- PMM
Peter Maydell
AppliedQuoted 11 lines · Peter Maydell
Quoted 5 lines · Eric Auger
SMMU3.2 brings the support of range-based TLB invalidation and level hint. When this feature is supported, the SMMUv3 driver is allowed to send TLB invalidations for a range of IOVAs instead of using page based invalidation.
I think this is all reviewed now; I've put it on my list of series to apply to target-arm.next in once 5.1 is out and we reopen the trunk for 5.2.
Applied to target-arm.next for 5.2, thanks.
-- PMM