Skip to contentSkip to conversation

SMMUv3.2 Range-based TLB Invalidation Support

PATCH Superseded Eric Auger 3 participants 17 messages Jul 2, 2020 – Jul 7, 2020

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v2 cover SMMUv3.2 Range-based TLB Invalidation Support

SMMU3.2 brings the support of range-based TLB invalidation and level hint. When this feature is supported, the SMMUv3 driver is allowed to send TLB invalidations for a range of IOVAs instead of using page based invalidation.

Implementing this feature in the virtual SMMUv3 device is mandated for DPDK on guest use case: DPDK uses hugepage buffers and guest sends invalidations for blocks. Without this feature, a guest invalidation of a block of 1GB for instance translates into a storm of page invalidations. Each of them is trapped by the VMM and cascaded downto the physical IOMMU. This completely stalls the execution. This integration issue was initially reported in [1].

Now SMMUv3.2 specifies additional parameters to NH_VA and NH_VAA stage 1 invalidation commands so we can support those extensions.

patches [1, 5] are cleanup patches.
patches [6] changes the implementation of the VSMMUV3 IOTLB
This IOTLB is a minimalist IOTLB implementation that avoids to do the page table walk in case we have an entry in the TLB. Previously entries were page mappings only. Now they can be blocks.
patches [7, 9] bring support for range invalidation.

Supporting block mappings in the IOTLB look sensible in terms of TLB entry consumption. However looking at virtio/vhost device usage, without block mapping and without range invalidation (< 5.7 kernels it may be less performant. However for recent guest kernels supporting range invalidations [2], the performance should be similar.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu.git
branch: v5.0.0-smmuv3-ril-v2

References:
[1] [RFC v2 4/4] iommu/arm-smmu-v3: add CMD_TLBI_NH_VA_AM command for iova range invalidation
(https://lists.linuxfoundation.org/pipermail/iommu/2017-August/023679.html

[2] 5.7+ kernels featuring
6a481a95d4c1 iommu/arm-smmu-v3: Add SMMUv3.2 range invalidation support

History:
v1 -> v2:
- added "hw/arm/smmu: Introduce smmu_get_iotlb_key()"
- removed "[PATCH 5/9] hw/arm/smmuv3: Store the starting level in SMMUTransTableInfo"
- Collected Peter's R-b
- In this version the key still features TG/LVL.
- More details in individual history logs

Eric Auger (9):
hw/arm/smmu-common: Factorize some code in smmu_ptw_64() hw/arm/smmu-common: Add IOTLB helpers hw/arm/smmu: Introduce smmu_get_iotlb_key() hw/arm/smmu: Simplify the IOTLB key format hw/arm/smmu: Introduce SMMUTLBEntry for PTW and IOTLB value hw/arm/smmu-common: Manage IOTLB block entries hw/arm/smmuv3: Introduce smmuv3_s1_range_inval() helper hw/arm/smmuv3: Get prepared for range invalidation hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation

 hw/arm/smmu-internal.h       |  13 +++
 hw/arm/smmuv3-internal.h     |   5 +
 include/hw/arm/smmu-common.h |  21 ++--
 hw/arm/smmu-common.c         | 200 +++++++++++++++++++++++------------
 hw/arm/smmuv3.c              | 134 ++++++++++++-----------
 hw/arm/trace-events          |  10 +-
 6 files changed, 235 insertions(+), 148 deletions(-)
Signature · 1 line
2.21.3
author
#1 lore raw

PATCH v2 1/9 hw/arm/smmu-common: Factorize some code in smmu_ptw_64()

Page and block PTE decoding can share some code. Let's first handle table PTE and factorize some code shared by page and block PTEs.

---

v1 -> v2:
- restore goto error in case get_pte() fails

---
 hw/arm/smmu-common.c | 48 ++++++++++++++++----------------------------
 1 file changed, 17 insertions(+), 31 deletions(-)
hw/arm/smmu-common.c +17 -31
index e13a5f4a7c..a3f9e47398 100644
@@ -186,7 +186,7 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,
         uint64_t subpage_size = 1ULL << level_shift(level, granule_sz);
         uint64_t mask = subpage_size - 1;
         uint32_t offset = iova_level_offset(iova, inputsize, level, granule_sz);
-        uint64_t pte;
+        uint64_t pte, gpa;
         dma_addr_t pte_addr = baseaddr + offset * sizeof(pte);
         uint8_t ap;
 
@@ -199,56 +199,42 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,
         if (is_invalid_pte(pte) || is_reserved_pte(pte, level)) {
             trace_smmu_ptw_invalid_pte(stage, level, baseaddr,
                                        pte_addr, offset, pte);
-            info->type = SMMU_PTW_ERR_TRANSLATION;
-            goto error;
+            break;
         }
 
-        if (is_page_pte(pte, level)) {
-            uint64_t gpa = get_page_pte_address(pte, granule_sz);
+        if (is_table_pte(pte, level)) {
+            ap = PTE_APTABLE(pte);
 
-            ap = PTE_AP(pte);
             if (is_permission_fault(ap, perm)) {
                 info->type = SMMU_PTW_ERR_PERMISSION;
                 goto error;
             }
-
-            tlbe->translated_addr = gpa + (iova & mask);
-            tlbe->perm = PTE_AP_TO_PERM(ap);
+            baseaddr = get_table_pte_address(pte, granule_sz);
+            level++;
+            continue;
+        } else if (is_page_pte(pte, level)) {
+            gpa = get_page_pte_address(pte, granule_sz);
             trace_smmu_ptw_page_pte(stage, level, iova,
                                     baseaddr, pte_addr, pte, gpa);
-            return 0;
-        }
-        if (is_block_pte(pte, level)) {
+        } else {
             uint64_t block_size;
-            hwaddr gpa = get_block_pte_address(pte, level, granule_sz,
-                                               &block_size);
-
-            ap = PTE_AP(pte);
-            if (is_permission_fault(ap, perm)) {
-                info->type = SMMU_PTW_ERR_PERMISSION;
-                goto error;
-            }
 
+            gpa = get_block_pte_address(pte, level, granule_sz,
+                                        &block_size);
             trace_smmu_ptw_block_pte(stage, level, baseaddr,
                                      pte_addr, pte, iova, gpa,
                                      block_size >> 20);
-
-            tlbe->translated_addr = gpa + (iova & mask);
-            tlbe->perm = PTE_AP_TO_PERM(ap);
-            return 0;
         }
-
-        /* table pte */
-        ap = PTE_APTABLE(pte);
-
+        ap = PTE_AP(pte);
         if (is_permission_fault(ap, perm)) {
             info->type = SMMU_PTW_ERR_PERMISSION;
             goto error;
         }
-        baseaddr = get_table_pte_address(pte, granule_sz);
-        level++;
-    }
 
+        tlbe->translated_addr = gpa + (iova & mask);
+        tlbe->perm = PTE_AP_TO_PERM(ap);
+        return 0;
+    }
     info->type = SMMU_PTW_ERR_TRANSLATION;
 
 error:
Signature · 1 line
2.21.3
Reviewed-by on v2 1/9
#2 lore raw

Re: [PATCH v2 1/9] hw/arm/smmu-common: Factorize some code in smmu_ptw_64()

Quoted 5 lines · Eric Auger

Page and block PTE decoding can share some code. Let's first handle table PTE and factorize some code shared by page and block PTEs.

Quoted 7 lines

---

v1 -> v2:
- restore goto error in case get_pte() fails
---

thanks
-- PMM

author
#3 lore raw

PATCH v2 2/9 hw/arm/smmu-common: Add IOTLB helpers

In reply to Eric Auger

Add two helpers: one to lookup for a given IOTLB entry and one to insert a new entry. We also move the tracing there.

---
 include/hw/arm/smmu-common.h |  2 ++
 hw/arm/smmu-common.c         | 36 ++++++++++++++++++++++++++++++++++++
 hw/arm/smmuv3.c              | 26 ++------------------------
 hw/arm/trace-events          |  5 +++--
 4 files changed, 43 insertions(+), 26 deletions(-)
include/hw/arm/smmu-common.h +2 -0
index ca4a4b1ad1..1dceec5cb1 100644
@@ -153,6 +153,8 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);
 
 #define SMMU_IOTLB_MAX_SIZE 256
 
+IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg, hwaddr iova);
+void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry);
 void smmu_iotlb_inv_all(SMMUState *s);
 void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid);
 void smmu_iotlb_inv_iova(SMMUState *s, uint16_t asid, dma_addr_t iova);
hw/arm/smmu-common.c +36 -0
index a3f9e47398..f3aa581f80 100644
@@ -32,6 +32,42 @@
 
 /* IOTLB Management */
 
+IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
+                                 hwaddr iova)
+{
+    SMMUIOTLBKey key = {.asid = cfg->asid, .iova = iova};
+    IOMMUTLBEntry *entry = g_hash_table_lookup(bs->iotlb, &key);
+
+    if (entry) {
+        cfg->iotlb_hits++;
+        trace_smmu_iotlb_lookup_hit(cfg->asid, iova,
+                                    cfg->iotlb_hits, cfg->iotlb_misses,
+                                    100 * cfg->iotlb_hits /
+                                    (cfg->iotlb_hits + cfg->iotlb_misses));
+    } else {
+        cfg->iotlb_misses++;
+        trace_smmu_iotlb_lookup_miss(cfg->asid, iova,
+                                     cfg->iotlb_hits, cfg->iotlb_misses,
+                                     100 * cfg->iotlb_hits /
+                                     (cfg->iotlb_hits + cfg->iotlb_misses));
+    }
+    return entry;
+}
+
+void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry)
+{
+    SMMUIOTLBKey *key = g_new0(SMMUIOTLBKey, 1);
+
+    if (g_hash_table_size(bs->iotlb) >= SMMU_IOTLB_MAX_SIZE) {
+        smmu_iotlb_inv_all(bs);
+    }
+
+    key->asid = cfg->asid;
+    key->iova = entry->iova;
+    trace_smmu_iotlb_insert(cfg->asid, entry->iova);
+    g_hash_table_insert(bs->iotlb, key, entry);
+}
+
 inline void smmu_iotlb_inv_all(SMMUState *s)
 {
     trace_smmu_iotlb_inv_all();
hw/arm/smmuv3.c +2 -24
index 57a79df55b..cd2a2e7e14 100644
@@ -636,7 +636,6 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         .addr_mask = ~(hwaddr)0,
         .perm = IOMMU_NONE,
     };
-    SMMUIOTLBKey key, *new_key;
 
     qemu_mutex_lock(&s->mutex);
 
@@ -675,16 +674,8 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
     page_mask = (1ULL << (tt->granule_sz)) - 1;
     aligned_addr = addr & ~page_mask;
 
-    key.asid = cfg->asid;
-    key.iova = aligned_addr;
-
-    cached_entry = g_hash_table_lookup(bs->iotlb, &key);
+    cached_entry = smmu_iotlb_lookup(bs, cfg, aligned_addr);
     if (cached_entry) {
-        cfg->iotlb_hits++;
-        trace_smmu_iotlb_cache_hit(cfg->asid, aligned_addr,
-                                   cfg->iotlb_hits, cfg->iotlb_misses,
-                                   100 * cfg->iotlb_hits /
-                                   (cfg->iotlb_hits + cfg->iotlb_misses));
         if ((flag & IOMMU_WO) && !(cached_entry->perm & IOMMU_WO)) {
             status = SMMU_TRANS_ERROR;
             if (event.record_trans_faults) {
@@ -698,16 +689,6 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         goto epilogue;
     }
 
-    cfg->iotlb_misses++;
-    trace_smmu_iotlb_cache_miss(cfg->asid, addr & ~page_mask,
-                                cfg->iotlb_hits, cfg->iotlb_misses,
-                                100 * cfg->iotlb_hits /
-                                (cfg->iotlb_hits + cfg->iotlb_misses));
-
-    if (g_hash_table_size(bs->iotlb) >= SMMU_IOTLB_MAX_SIZE) {
-        smmu_iotlb_inv_all(bs);
-    }
-
     cached_entry = g_new0(IOMMUTLBEntry, 1);
 
     if (smmu_ptw(cfg, aligned_addr, flag, cached_entry, &ptw_info)) {
@@ -753,10 +734,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         }
         status = SMMU_TRANS_ERROR;
     } else {
-        new_key = g_new0(SMMUIOTLBKey, 1);
-        new_key->asid = cfg->asid;
-        new_key->iova = aligned_addr;
-        g_hash_table_insert(bs->iotlb, new_key, cached_entry);
+        smmu_iotlb_insert(bs, cfg, cached_entry);
         status = SMMU_TRANS_SUCCESS;
     }
 
hw/arm/trace-events +3 -2
index 0acedcedc6..b808a1bfc1 100644
@@ -14,6 +14,9 @@ smmu_iotlb_inv_all(void) "IOTLB invalidate all"
 smmu_iotlb_inv_asid(uint16_t asid) "IOTLB invalidate asid=%d"
 smmu_iotlb_inv_iova(uint16_t asid, uint64_t addr) "IOTLB invalidate asid=%d addr=0x%"PRIx64
 smmu_inv_notifiers_mr(const char *name) "iommu mr=%s"
+smmu_iotlb_lookup_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
+smmu_iotlb_lookup_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
+smmu_iotlb_insert(uint16_t asid, uint64_t addr) "IOTLB ++ asid=%d addr=0x%"PRIx64
 
 # smmuv3.c
 smmuv3_read_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
@@ -46,8 +49,6 @@ smmuv3_cmdq_tlbi_nh_va(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d a
 smmuv3_cmdq_tlbi_nh_vaa(int vmid, uint64_t addr) "vmid =%d addr=0x%"PRIx64
 smmuv3_cmdq_tlbi_nh(void) ""
 smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d"
-smmu_iotlb_cache_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
-smmu_iotlb_cache_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
 smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d"
 smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s"
 smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s"
Signature · 1 line
2.21.3
author
#4 lore raw

PATCH v2 3/9 hw/arm/smmu: Introduce smmu_get_iotlb_key()

In reply to Eric Auger

Introduce the smmu_get_iotlb_key() helper and the SMMU_IOTLB_ASID() macro. Also move smmu_get_iotlb_key and smmu_iotlb_key_hash in the IOTLB related code section.

---

v1 -> v2:
- creation

---
 hw/arm/smmu-internal.h       |  1 +
 include/hw/arm/smmu-common.h |  1 +
 hw/arm/smmu-common.c         | 66 ++++++++++++++++++++----------------
 3 files changed, 38 insertions(+), 30 deletions(-)
hw/arm/smmu-internal.h +1 -0
index 7794d6d394..3104f768cd 100644
@@ -96,4 +96,5 @@ uint64_t iova_level_offset(uint64_t iova, int inputsize,
             MAKE_64BIT_MASK(0, gsz - 3);
 }
 
+#define SMMU_IOTLB_ASID(key) ((key).asid)
 #endif
include/hw/arm/smmu-common.h +1 -0
index 1dceec5cb1..5f9f3535d2 100644
@@ -155,6 +155,7 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);
 
 IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg, hwaddr iova);
 void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry);
+SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova);
 void smmu_iotlb_inv_all(SMMUState *s);
 void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid);
 void smmu_iotlb_inv_iova(SMMUState *s, uint16_t asid, dma_addr_t iova);
hw/arm/smmu-common.c +36 -30

97 lines not on this page · raw message

Signature · 1 line
2.21.3
Reviewed-by on v2 3/9
#5 lore raw

Re: [PATCH v2 3/9] hw/arm/smmu: Introduce smmu_get_iotlb_key()

Quoted 5 lines · Eric Auger

Introduce the smmu_get_iotlb_key() helper and the SMMU_IOTLB_ASID() macro. Also move smmu_get_iotlb_key and smmu_iotlb_key_hash in the IOTLB related code section.

Quoted 7 lines

---

v1 -> v2:
- creation
---

thanks
-- PMM

author
#6 lore raw

PATCH v2 4/9 hw/arm/smmu: Simplify the IOTLB key format

In reply to Eric Auger

Instead of using a Jenkins hash function to generate the key let's just use a 64 bit unsigned integer that contains the asid and the 40 upper bits of the iova. A maximum of 52-bit IOVA is supported. This change in the key format also prepares for the addition of new fields in subsequent patches (granule and level).

---

v1 -> v2:
- split the inital patch into 2 patches and keep SMMUIOTLBKey type

---
 hw/arm/smmu-internal.h       |  5 ++++-
 include/hw/arm/smmu-common.h |  5 +----
 hw/arm/smmu-common.c         | 23 +++--------------------
 3 files changed, 8 insertions(+), 25 deletions(-)
hw/arm/smmu-internal.h +4 -1
index 3104f768cd..2ecb6f1dc6 100644
@@ -96,5 +96,8 @@ uint64_t iova_level_offset(uint64_t iova, int inputsize,
             MAKE_64BIT_MASK(0, gsz - 3);
 }
 
-#define SMMU_IOTLB_ASID(key) ((key).asid)
+#define SMMU_IOTLB_ASID_SHIFT  40
+
+#define SMMU_IOTLB_ASID(key) (((key) >> SMMU_IOTLB_ASID_SHIFT) & 0xFFFF)
+#define SMMU_IOTLB_IOVA(key) (((key) & MAKE_64BIT_MASK(0, 40)) << 12)
 #endif
include/hw/arm/smmu-common.h +1 -4
index 5f9f3535d2..455d7855b2 100644
@@ -88,10 +88,7 @@ typedef struct SMMUPciBus {
     SMMUDevice   *pbdev[]; /* Parent array is sparse, so dynamically alloc */
 } SMMUPciBus;
 
-typedef struct SMMUIOTLBKey {
-    uint64_t iova;
-    uint16_t asid;
-} SMMUIOTLBKey;
+ typedef uint64_t SMMUIOTLBKey;
 
 typedef struct SMMUState {
     /* <private> */
hw/arm/smmu-common.c +3 -20
index 7dc8541e8b..5e85e30bdf 100644
@@ -34,34 +34,17 @@
 
 static guint smmu_iotlb_key_hash(gconstpointer v)
 {
-    SMMUIOTLBKey *key = (SMMUIOTLBKey *)v;
-    uint32_t a, b, c;
-
-    /* Jenkins hash */
-    a = b = c = JHASH_INITVAL + sizeof(*key);
-    a += key->asid;
-    b += extract64(key->iova, 0, 32);
-    c += extract64(key->iova, 32, 32);
-
-    __jhash_mix(a, b, c);
-    __jhash_final(a, b, c);
-
-    return c;
+    return (guint)*(const uint64_t *)v;
 }
 
 static gboolean smmu_iotlb_key_equal(gconstpointer v1, gconstpointer v2)
 {
-    const SMMUIOTLBKey *k1 = v1;
-    const SMMUIOTLBKey *k2 = v2;
-
-    return (k1->asid == k2->asid) && (k1->iova == k2->iova);
+    return *((const uint64_t *)v1) == *((const uint64_t *)v2);
 }
 
 SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova)
 {
-    SMMUIOTLBKey key = {.asid = asid, .iova = iova};
-
-    return key;
+    return iova >> 12 | (uint64_t)(asid) << SMMU_IOTLB_ASID_SHIFT;
 }
 
 IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
Signature · 1 line
2.21.3
#7 lore raw

Re: [PATCH v2 4/9] hw/arm/smmu: Simplify the IOTLB key format

Quoted 23 lines · Eric Auger

Instead of using a Jenkins hash function to generate the key let's just use a 64 bit unsigned integer that contains the asid and the 40 upper bits of the iova. A maximum of 52-bit IOVA is supported. This change in the key format also prepares for the addition of new fields in subsequent patches (granule and level).

hw/arm/smmu-common.c +0 -5
index 7dc8541e8b..5e85e30bdf 100644
@@ -34,34 +34,17 @@
 
 static guint smmu_iotlb_key_hash(gconstpointer v)
 {
-    SMMUIOTLBKey *key = (SMMUIOTLBKey *)v;
-    uint32_t a, b, c;
-
-    /* Jenkins hash */
-    a = b = c = JHASH_INITVAL + sizeof(*key);

hw/arm/smmu-common.c line 38

Quoted 10 lines
hw/arm/smmu-common.c +1 -8
-    a += key->asid;
-    b += extract64(key->iova, 0, 32);
-    c += extract64(key->iova, 32, 32);
-
-    __jhash_mix(a, b, c);
-    __jhash_final(a, b, c);
-
-    return c;
+    return (guint)*(const uint64_t *)v;
 }

So the hash value is now going to be the lower 32 bits of the key, which is to say bits [40,12] of the IOVA, and won't include the ASID at all. Isn't that going to result in more hash collisions than would be ideal?

I was going to suggest using the glib builtin g_int64_hash() instead, but looking at the source that seems to be the identical implementation to this one. I guess that's intended for cases where an integer key is really a random integer, not one where it's got internal structure of different bit fields within it being for different purposes.

thanks
-- PMM

author
#8 lore raw

Re: [PATCH v2 4/9] hw/arm/smmu: Simplify the IOTLB key format

Hi Peter,

Quoted 40 lines · Peter Maydell
Quoted 23 lines · Eric Auger

Instead of using a Jenkins hash function to generate the key let's just use a 64 bit unsigned integer that contains the asid and the 40 upper bits of the iova. A maximum of 52-bit IOVA is supported. This change in the key format also prepares for the addition of new fields in subsequent patches (granule and level).

hw/arm/smmu-common.c +0 -5
index 7dc8541e8b..5e85e30bdf 100644
@@ -34,34 +34,17 @@
 
 static guint smmu_iotlb_key_hash(gconstpointer v)
 {
-    SMMUIOTLBKey *key = (SMMUIOTLBKey *)v;
-    uint32_t a, b, c;
-
-    /* Jenkins hash */
-    a = b = c = JHASH_INITVAL + sizeof(*key);

hw/arm/smmu-common.c line 38

Quoted 10 lines
hw/arm/smmu-common.c +1 -8
-    a += key->asid;
-    b += extract64(key->iova, 0, 32);
-    c += extract64(key->iova, 32, 32);
-
-    __jhash_mix(a, b, c);
-    __jhash_final(a, b, c);
-
-    return c;
+    return (guint)*(const uint64_t *)v;
 }

So the hash value is now going to be the lower 32 bits of the key, which is to say bits [40,12] of the IOVA, and won't include the ASID at all. Isn't that going to result in more hash collisions than would be ideal?

Quoted 7 lines

I was going to suggest using the glib builtin g_int64_hash() instead, but looking at the source that seems to be the identical implementation to this one. I guess that's intended for cases where an integer key is really a random integer, not one where it's got internal structure of different bit fields within it being for different purposes.

That's something I did not notice. Would you recommend to keep the Jenkins hash function then? Note the intel iommu also use the hash function which only covers the gfn.

Thanks

Eric

Quoted 4 lines

thanks
-- PMM

author
#9 lore raw

PATCH v2 5/9 hw/arm/smmu: Introduce SMMUTLBEntry for PTW and IOTLB value

In reply to Eric Auger

Introduce a specialized SMMUTLBEntry to store the result of the PTW and cache in the IOTLB. This structure extends the generic IOMMUTLBEntry struct with the level of the entry and the granule size.

Those latter will be useful when implementing range invalidation.

---

v1 -> v2:
- fix 2 indents and typdef
- added Peter's R-b

---
 include/hw/arm/smmu-common.h | 12 +++++++++---
 hw/arm/smmu-common.c         | 32 +++++++++++++++++---------------
 hw/arm/smmuv3.c              | 10 +++++-----
 3 files changed, 31 insertions(+), 23 deletions(-)
include/hw/arm/smmu-common.h +9 -3
index 455d7855b2..0db4403e94 100644
@@ -52,6 +52,12 @@ typedef struct SMMUTransTableInfo {
     uint8_t granule_sz;        /* granule page shift */
 } SMMUTransTableInfo;
 
+typedef struct SMMUTLBEntry {
+    IOMMUTLBEntry entry;
+    uint8_t level;
+    uint8_t granule;
+} SMMUTLBEntry;
+
 /*
  * Generic structure populated by derived SMMU devices
  * after decoding the configuration information and used as
@@ -137,7 +143,7 @@ static inline uint16_t smmu_get_sid(SMMUDevice *sdev)
  * pair, according to @cfg translation config
  */
 int smmu_ptw(SMMUTransCfg *cfg, dma_addr_t iova, IOMMUAccessFlags perm,
-             IOMMUTLBEntry *tlbe, SMMUPTWEventInfo *info);
+             SMMUTLBEntry *tlbe, SMMUPTWEventInfo *info);
 
 /**
  * select_tt - compute which translation table shall be used according to
@@ -150,8 +156,8 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);
 
 #define SMMU_IOTLB_MAX_SIZE 256
 
-IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg, hwaddr iova);
-void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry);
+SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg, hwaddr iova);
+void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *entry);
 SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova);
 void smmu_iotlb_inv_all(SMMUState *s);
 void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid);
hw/arm/smmu-common.c +17 -15
index 5e85e30bdf..a3301e56e7 100644
@@ -47,11 +47,11 @@ SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova)
     return iova >> 12 | (uint64_t)(asid) << SMMU_IOTLB_ASID_SHIFT;
 }
 
-IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
-                                 hwaddr iova)
+SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
+                                hwaddr iova)
 {
     SMMUIOTLBKey key = smmu_get_iotlb_key(cfg->asid, iova);
-    IOMMUTLBEntry *entry = g_hash_table_lookup(bs->iotlb, &key);
+    SMMUTLBEntry *entry = g_hash_table_lookup(bs->iotlb, &key);
 
     if (entry) {
         cfg->iotlb_hits++;
@@ -69,7 +69,7 @@ IOMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
     return entry;
 }
 
-void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry)
+void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *new)
 {
     SMMUIOTLBKey *key = g_new0(SMMUIOTLBKey, 1);
 
@@ -77,9 +77,9 @@ void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, IOMMUTLBEntry *entry)
         smmu_iotlb_inv_all(bs);
     }
 
-    *key = smmu_get_iotlb_key(cfg->asid, entry->iova);
-    trace_smmu_iotlb_insert(cfg->asid, entry->iova);
-    g_hash_table_insert(bs->iotlb, key, entry);
+    *key = smmu_get_iotlb_key(cfg->asid, new->entry.iova);
+    trace_smmu_iotlb_insert(cfg->asid, new->entry.iova);
+    g_hash_table_insert(bs->iotlb, key, new);
 }
 
 inline void smmu_iotlb_inv_all(SMMUState *s)
@@ -199,7 +199,7 @@ SMMUTransTableInfo *select_tt(SMMUTransCfg *cfg, dma_addr_t iova)
  * @cfg: translation config
  * @iova: iova to translate
  * @perm: access type
- * @tlbe: IOMMUTLBEntry (out)
+ * @tlbe: SMMUTLBEntry (out)
  * @info: handle to an error info
  *
  * Return 0 on success, < 0 on error. In case of error, @info is filled
@@ -209,7 +209,7 @@ SMMUTransTableInfo *select_tt(SMMUTransCfg *cfg, dma_addr_t iova)
  */
 static int smmu_ptw_64(SMMUTransCfg *cfg,
                        dma_addr_t iova, IOMMUAccessFlags perm,
-                       IOMMUTLBEntry *tlbe, SMMUPTWEventInfo *info)
+                       SMMUTLBEntry *tlbe, SMMUPTWEventInfo *info)
 {
     dma_addr_t baseaddr, indexmask;
     int stage = cfg->stage;
@@ -229,8 +229,8 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,
     baseaddr = extract64(tt->ttb, 0, 48);
     baseaddr &= ~indexmask;
 
-    tlbe->iova = iova;
-    tlbe->addr_mask = (1 << granule_sz) - 1;
+    tlbe->entry.iova = iova;
+    tlbe->entry.addr_mask = (1 << granule_sz) - 1;
 
     while (level <= 3) {
         uint64_t subpage_size = 1ULL << level_shift(level, granule_sz);
@@ -281,14 +281,16 @@ static int smmu_ptw_64(SMMUTransCfg *cfg,
             goto error;
         }
 
-        tlbe->translated_addr = gpa + (iova & mask);
-        tlbe->perm = PTE_AP_TO_PERM(ap);
+        tlbe->entry.translated_addr = gpa + (iova & mask);
+        tlbe->entry.perm = PTE_AP_TO_PERM(ap);
+        tlbe->level = level;
+        tlbe->granule = granule_sz;
         return 0;
     }
     info->type = SMMU_PTW_ERR_TRANSLATION;
 
 error:
-    tlbe->perm = IOMMU_NONE;
+    tlbe->entry.perm = IOMMU_NONE;
     return -EINVAL;
 }
 
@@ -304,7 +306,7 @@ error:
  * return 0 on success
  */
 inline int smmu_ptw(SMMUTransCfg *cfg, dma_addr_t iova, IOMMUAccessFlags perm,
-             IOMMUTLBEntry *tlbe, SMMUPTWEventInfo *info)
+                    SMMUTLBEntry *tlbe, SMMUPTWEventInfo *info)
 {
     if (!cfg->aa64) {
         /*
hw/arm/smmuv3.c +5 -5
index cd2a2e7e14..db74d27add 100644
@@ -626,7 +626,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
     SMMUTranslationStatus status;
     SMMUState *bs = ARM_SMMU(s);
     uint64_t page_mask, aligned_addr;
-    IOMMUTLBEntry *cached_entry = NULL;
+    SMMUTLBEntry *cached_entry = NULL;
     SMMUTransTableInfo *tt;
     SMMUTransCfg *cfg = NULL;
     IOMMUTLBEntry entry = {
@@ -676,7 +676,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
 
     cached_entry = smmu_iotlb_lookup(bs, cfg, aligned_addr);
     if (cached_entry) {
-        if ((flag & IOMMU_WO) && !(cached_entry->perm & IOMMU_WO)) {
+        if ((flag & IOMMU_WO) && !(cached_entry->entry.perm & IOMMU_WO)) {
             status = SMMU_TRANS_ERROR;
             if (event.record_trans_faults) {
                 event.type = SMMU_EVT_F_PERMISSION;
@@ -689,7 +689,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         goto epilogue;
     }
 
-    cached_entry = g_new0(IOMMUTLBEntry, 1);
+    cached_entry = g_new0(SMMUTLBEntry, 1);
 
     if (smmu_ptw(cfg, aligned_addr, flag, cached_entry, &ptw_info)) {
         g_free(cached_entry);
@@ -743,9 +743,9 @@ epilogue:
     switch (status) {
     case SMMU_TRANS_SUCCESS:
         entry.perm = flag;
-        entry.translated_addr = cached_entry->translated_addr +
+        entry.translated_addr = cached_entry->entry.translated_addr +
                                     (addr & page_mask);
-        entry.addr_mask = cached_entry->addr_mask;
+        entry.addr_mask = cached_entry->entry.addr_mask;
         trace_smmuv3_translate_success(mr->parent_obj.name, sid, addr,
                                        entry.translated_addr, entry.perm);
         break;
Signature · 1 line
2.21.3
author
#10 lore raw

PATCH v2 6/9 hw/arm/smmu-common: Manage IOTLB block entries

In reply to Eric Auger

At the moment each entry in the IOTLB corresponds to a page sized mapping (4K, 16K or 64K), even if the page belongs to a mapped block. In case of block mapping this unefficiently consumes IOTLB entries.

Change the value of the entry so that it reflects the actual mapping it belongs to (block or page start address and size).

Also the level/tg of the entry is encoded in the key. In subsequent patches we will enable range invalidation. This latter is able to provide the level/tg of the entry.

Encoding the level/tg directly in the key will allow to invalidate using g_hash_table_remove() when num_pages equals to 1.

---

v1 -> v2:
- recompute starting_level

---
 hw/arm/smmu-internal.h       |  9 ++++++
 include/hw/arm/smmu-common.h |  8 +++--
 hw/arm/smmu-common.c         | 63 ++++++++++++++++++++++++++++--------
 hw/arm/smmuv3.c              |  6 ++--
 hw/arm/trace-events          |  2 +-
 5 files changed, 67 insertions(+), 21 deletions(-)
hw/arm/smmu-internal.h +9 -0
index 2ecb6f1dc6..d57f5c05c2 100644
@@ -97,7 +97,16 @@ uint64_t iova_level_offset(uint64_t iova, int inputsize,
 }
 
 #define SMMU_IOTLB_ASID_SHIFT  40
+#define SMMU_IOTLB_LEVEL_SHIFT 56
+#define SMMU_IOTLB_TG_SHIFT    58
 
 #define SMMU_IOTLB_ASID(key) (((key) >> SMMU_IOTLB_ASID_SHIFT) & 0xFFFF)
 #define SMMU_IOTLB_IOVA(key) (((key) & MAKE_64BIT_MASK(0, 40)) << 12)
+
+typedef struct SMMUIOTLBPageInvInfo {
+    int asid;
+    uint64_t iova;
+    uint64_t mask;
+} SMMUIOTLBPageInvInfo;
+
 #endif
include/hw/arm/smmu-common.h +5 -3
index 0db4403e94..cc228b51b4 100644
@@ -156,12 +156,14 @@ IOMMUMemoryRegion *smmu_iommu_mr(SMMUState *s, uint32_t sid);
 
 #define SMMU_IOTLB_MAX_SIZE 256
 
-SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg, hwaddr iova);
+SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
+                                SMMUTransTableInfo *tt, hwaddr iova);
 void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *entry);
-SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova);
+SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova,
+                               uint8_t tg, uint8_t level);
 void smmu_iotlb_inv_all(SMMUState *s);
 void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid);
-void smmu_iotlb_inv_iova(SMMUState *s, uint16_t asid, dma_addr_t iova);
+void smmu_iotlb_inv_iova(SMMUState *s, int asid, dma_addr_t iova);
 
 /* Unmap the range of all the notifiers registered to any IOMMU mr */
 void smmu_inv_notifiers_all(SMMUState *s);
hw/arm/smmu-common.c +49 -14

106 lines not on this page · raw message

hw/arm/smmuv3.c +3 -3
index db74d27add..b717bde832 100644
@@ -674,7 +674,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
     page_mask = (1ULL << (tt->granule_sz)) - 1;
     aligned_addr = addr & ~page_mask;
 
-    cached_entry = smmu_iotlb_lookup(bs, cfg, aligned_addr);
+    cached_entry = smmu_iotlb_lookup(bs, cfg, tt, aligned_addr);
     if (cached_entry) {
         if ((flag & IOMMU_WO) && !(cached_entry->entry.perm & IOMMU_WO)) {
             status = SMMU_TRANS_ERROR;
@@ -744,7 +744,7 @@ epilogue:
     case SMMU_TRANS_SUCCESS:
         entry.perm = flag;
         entry.translated_addr = cached_entry->entry.translated_addr +
-                                    (addr & page_mask);
+                                    (addr & cached_entry->entry.addr_mask);
         entry.addr_mask = cached_entry->entry.addr_mask;
         trace_smmuv3_translate_success(mr->parent_obj.name, sid, addr,
                                        entry.translated_addr, entry.perm);
@@ -972,7 +972,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
 
             trace_smmuv3_cmdq_tlbi_nh_vaa(vmid, addr);
             smmuv3_inv_notifiers_iova(bs, -1, addr);
-            smmu_iotlb_inv_all(bs);
+            smmu_iotlb_inv_iova(bs, -1, addr);
             break;
         }
         case SMMU_CMD_TLBI_NH_VA:
hw/arm/trace-events +1 -1
index b808a1bfc1..f74d3e920f 100644
@@ -16,7 +16,7 @@ smmu_iotlb_inv_iova(uint16_t asid, uint64_t addr) "IOTLB invalidate asid=%d addr
 smmu_inv_notifiers_mr(const char *name) "iommu mr=%s"
 smmu_iotlb_lookup_hit(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache HIT asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
 smmu_iotlb_lookup_miss(uint16_t asid, uint64_t addr, uint32_t hit, uint32_t miss, uint32_t p) "IOTLB cache MISS asid=%d addr=0x%"PRIx64" hit=%d miss=%d hit rate=%d"
-smmu_iotlb_insert(uint16_t asid, uint64_t addr) "IOTLB ++ asid=%d addr=0x%"PRIx64
+smmu_iotlb_insert(uint16_t asid, uint64_t addr, uint8_t tg, uint8_t level) "IOTLB ++ asid=%d addr=0x%"PRIx64" tg=%d level=%d"
 
 # smmuv3.c
 smmuv3_read_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
Signature · 1 line
2.21.3
Reviewed-by on v2 6/9
#11 lore raw

Re: [PATCH v2 6/9] hw/arm/smmu-common: Manage IOTLB block entries

Quoted 32 lines · Eric Auger

At the moment each entry in the IOTLB corresponds to a page sized mapping (4K, 16K or 64K), even if the page belongs to a mapped block. In case of block mapping this unefficiently consumes IOTLB entries.

Change the value of the entry so that it reflects the actual mapping it belongs to (block or page start address and size).

Also the level/tg of the entry is encoded in the key. In subsequent patches we will enable range invalidation. This latter is able to provide the level/tg of the entry.

Encoding the level/tg directly in the key will allow to invalidate using g_hash_table_remove() when num_pages equals to 1.

SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,

hw/arm/smmu-common.c +9 -3
-                                hwaddr iova)
+                                SMMUTransTableInfo *tt, hwaddr iova)
 {
-    SMMUIOTLBKey key = smmu_get_iotlb_key(cfg->asid, iova);
-    SMMUTLBEntry *entry = g_hash_table_lookup(bs->iotlb, &key);
+    uint8_t tg = (tt->granule_sz - 10) / 2;
+    uint8_t inputsize = 64 - tt->tsz;
+    uint8_t stride = tt->granule_sz - 3;
+    uint8_t level = 4 - (inputsize - 4) / stride;
+    SMMUTLBEntry *entry = NULL;
+
+    while (level <= 3) {
+        uint64_t subpage_size = 1ULL << level_shift(level, tt->granule_sz);

hw/arm/smmu-common.c static gboolean smmu_iotlb_key_equal(gconstpointer v1, gconstpointer v2) · line 73

Quoted 10 lines
hw/arm/smmu-common.c +10 -0
+        uint64_t mask = subpage_size - 1;
+        SMMUIOTLBKey key;
+
+        key = smmu_get_iotlb_key(cfg->asid, iova & ~mask, tg, level);
+        entry = g_hash_table_lookup(bs->iotlb, &key);
+        if (entry) {
+            break;
+        }
+        level++;
+    }

So, this next bit is something of a digression:

Effectively what we're doing here is "look up in the hash table for each of the 3 possible page sizes this could be for this TG". I've had suggested to me a possible alternative data structure:
* hash on the asid (and vmid eventually?) to get...
* a sorted list of (start, length) ranges representing the TLB for that asid ...
* which we can binary-search to find the matching range (and the associated info)

The theoretical benefit is that we don't need to do three hash table lookups for each iotlb_lookup, and that it can support arbitrary-length ranges (so we could honour the Contiguous bit hint in the page tables, for instance). But to avoid 3 hash lookups it's probably not worth doing a complete rework of the data structures.

Quoted 6 lines
hw/arm/smmu-common.c +3 -2
-inline void smmu_iotlb_inv_iova(SMMUState *s, uint16_t asid, dma_addr_t iova)
+static gboolean smmu_hash_remove_by_asid_iova(gpointer key, gpointer value,
+                                         gpointer user_data)
 {
-    SMMUIOTLBKey key = smmu_get_iotlb_key(asid, iova);
+    SMMUTLBEntry *iter = (SMMUTLBEntry *)value;

hw/arm/smmu-common.c static gboolean smmu_hash_remove_by_asid(gpointer key, gpointer value, · line 133

Quoted 10 lines
hw/arm/smmu-common.c +10 -0
+    IOMMUTLBEntry *entry = &iter->entry;
+    SMMUIOTLBPageInvInfo *info = (SMMUIOTLBPageInvInfo *)user_data;
+    uint64_t *iotlb_key = (uint64_t *)key;
+
+    if (info->asid >= 0) {
+        return (info->asid == SMMU_IOTLB_ASID(*iotlb_key)) &&
+                ((info->iova & ~entry->addr_mask) == entry->iova);
+    } else {
+        return (info->iova & ~entry->addr_mask) == entry->iova;
+    }

Since the iova comparison logic is the same in both branches of this if(), you can write

    if (info->asid >= 0 && info->asid != SMMU_IOTLB_ASID(*iotlb_key)) {
        return false;
    }
    return (info->iova & ~entry->addr_mask) == entry->iova;

This seems particularly worthwhile given that a later patch makes the iova comparison logic more complicated.

Otherwise

thanks
-- PMM

author
#12 lore raw

PATCH v2 7/9 hw/arm/smmuv3: Introduce smmuv3_s1_range_inval() helper

In reply to Eric Auger

Let's introduce an helper for S1 IOVA range invalidation. This will be used for NH_VA and NH_VAA commands. It decodes the same fields, trace, calls the UNMAP notifiers and invalidate the corresponding IOTLB entries.

At the moment, we do not support 3.2 range invalidation yet. So it reduces to a single IOVA invalidation.

Note the leaf bit now is also decoded for the CMD_TLBI_NH_VAA command. At the moment it is only used for tracing.

---

v1 -> v2:
- added comment about leaf bit and added Peter's R-b

---
 hw/arm/smmuv3.c     | 36 +++++++++++++++++-------------------
 hw/arm/trace-events |  3 +--
 2 files changed, 18 insertions(+), 21 deletions(-)
hw/arm/smmuv3.c +17 -19
index b717bde832..e4a2cea7ad 100644
@@ -836,6 +836,22 @@ static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, dma_addr_t iova)
     }
 }
 
+static void smmuv3_s1_range_inval(SMMUState *s, Cmd *cmd)
+{
+    dma_addr_t addr = CMD_ADDR(cmd);
+    uint8_t type = CMD_TYPE(cmd);
+    uint16_t vmid = CMD_VMID(cmd);
+    bool leaf = CMD_LEAF(cmd);
+    int asid = -1;
+
+    if (type == SMMU_CMD_TLBI_NH_VA) {
+        asid = CMD_ASID(cmd);
+    }
+    trace_smmuv3_s1_range_inval(vmid, asid, addr, leaf);
+    smmuv3_inv_notifiers_iova(s, asid, addr);
+    smmu_iotlb_inv_iova(s, asid, addr);
+}
+
 static int smmuv3_cmdq_consume(SMMUv3State *s)
 {
     SMMUState *bs = ARM_SMMU(s);
@@ -966,27 +982,9 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
             smmu_iotlb_inv_all(bs);
             break;
         case SMMU_CMD_TLBI_NH_VAA:
-        {
-            dma_addr_t addr = CMD_ADDR(&cmd);
-            uint16_t vmid = CMD_VMID(&cmd);
-
-            trace_smmuv3_cmdq_tlbi_nh_vaa(vmid, addr);
-            smmuv3_inv_notifiers_iova(bs, -1, addr);
-            smmu_iotlb_inv_iova(bs, -1, addr);
-            break;
-        }
         case SMMU_CMD_TLBI_NH_VA:
-        {
-            uint16_t asid = CMD_ASID(&cmd);
-            uint16_t vmid = CMD_VMID(&cmd);
-            dma_addr_t addr = CMD_ADDR(&cmd);
-            bool leaf = CMD_LEAF(&cmd);
-
-            trace_smmuv3_cmdq_tlbi_nh_va(vmid, asid, addr, leaf);
-            smmuv3_inv_notifiers_iova(bs, asid, addr);
-            smmu_iotlb_inv_iova(bs, asid, addr);
+            smmuv3_s1_range_inval(bs, &cmd);
             break;
-        }
         case SMMU_CMD_TLBI_EL3_ALL:
         case SMMU_CMD_TLBI_EL3_VA:
         case SMMU_CMD_TLBI_EL2_ALL:
hw/arm/trace-events +1 -2
index f74d3e920f..c219fe9e82 100644
@@ -45,8 +45,7 @@ smmuv3_cmdq_cfgi_ste_range(int start, int end) "start=0x%d - end=0x%d"
 smmuv3_cmdq_cfgi_cd(uint32_t sid) "streamid = %d"
 smmuv3_config_cache_hit(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache HIT for sid %d (hits=%d, misses=%d, hit rate=%d)"
 smmuv3_config_cache_miss(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache MISS for sid %d (hits=%d, misses=%d, hit rate=%d)"
-smmuv3_cmdq_tlbi_nh_va(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d"
-smmuv3_cmdq_tlbi_nh_vaa(int vmid, uint64_t addr) "vmid =%d addr=0x%"PRIx64
+smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d"
 smmuv3_cmdq_tlbi_nh(void) ""
 smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d"
 smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d"
Signature · 1 line
2.21.3
author
#13 lore raw

PATCH v2 8/9 hw/arm/smmuv3: Get prepared for range invalidation

In reply to Eric Auger

Enhance the smmu_iotlb_inv_iova() helper with range invalidation. This uses the new fields passed in the NH_VA and NH_VAA commands:
the size of the range, the level and the granule.

As NH_VA and NH_VAA both use those fields, their decoding and handling is factorized in a new smmuv3_s1_range_inval() helper.

---
 hw/arm/smmuv3-internal.h     |  4 +++
 include/hw/arm/smmu-common.h |  3 +-
 hw/arm/smmu-common.c         | 28 ++++++++++++----
 hw/arm/smmuv3.c              | 64 +++++++++++++++++++++++-------------
 hw/arm/trace-events          |  4 +--
 5 files changed, 71 insertions(+), 32 deletions(-)
hw/arm/smmuv3-internal.h +4 -0
index 4112394129..5babf72f7d 100644
@@ -298,6 +298,8 @@ enum { /* Command completion notification */
 };
 
 #define CMD_TYPE(x)         extract32((x)->word[0], 0 , 8)
+#define CMD_NUM(x)          extract32((x)->word[0], 12 , 5)
+#define CMD_SCALE(x)        extract32((x)->word[0], 20 , 5)
 #define CMD_SSEC(x)         extract32((x)->word[0], 10, 1)
 #define CMD_SSV(x)          extract32((x)->word[0], 11, 1)
 #define CMD_RESUME_AC(x)    extract32((x)->word[0], 12, 1)
@@ -310,6 +312,8 @@ enum { /* Command completion notification */
 #define CMD_RESUME_STAG(x)  extract32((x)->word[2], 0 , 16)
 #define CMD_RESP(x)         extract32((x)->word[2], 11, 2)
 #define CMD_LEAF(x)         extract32((x)->word[2], 0 , 1)
+#define CMD_TTL(x)          extract32((x)->word[2], 8 , 2)
+#define CMD_TG(x)           extract32((x)->word[2], 10, 2)
 #define CMD_STE_RANGE(x)    extract32((x)->word[2], 0 , 5)
 #define CMD_ADDR(x) ({                                        \
             uint64_t high = (uint64_t)(x)->word[3];           \
include/hw/arm/smmu-common.h +2 -1
index cc228b51b4..82bd9a4e8c 100644
@@ -163,7 +163,8 @@ SMMUIOTLBKey smmu_get_iotlb_key(uint16_t asid, uint64_t iova,
                                uint8_t tg, uint8_t level);
 void smmu_iotlb_inv_all(SMMUState *s);
 void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid);
-void smmu_iotlb_inv_iova(SMMUState *s, int asid, dma_addr_t iova);
+void smmu_iotlb_inv_iova(SMMUState *s, int asid, dma_addr_t iova,
+                         uint8_t tg, uint64_t num_pages, uint8_t ttl);
 
 /* Unmap the range of all the notifiers registered to any IOMMU mr */
 void smmu_inv_notifiers_all(SMMUState *s);
hw/arm/smmu-common.c +22 -6
index 5c14127a24..198ed6a928 100644
@@ -127,18 +127,34 @@ static gboolean smmu_hash_remove_by_asid_iova(gpointer key, gpointer value,
 
     if (info->asid >= 0) {
         return (info->asid == SMMU_IOTLB_ASID(*iotlb_key)) &&
-                ((info->iova & ~entry->addr_mask) == entry->iova);
+               (((entry->iova & ~info->mask) == info->iova) ||
+               ((info->iova & ~entry->addr_mask) ==  entry->iova));
     } else {
-        return (info->iova & ~entry->addr_mask) == entry->iova;
+        return (((entry->iova & ~info->mask) == info->iova) ||
+               ((info->iova & ~entry->addr_mask) ==  entry->iova));
     }
 }
 
-inline void smmu_iotlb_inv_iova(SMMUState *s, int asid, dma_addr_t iova)
+inline void
+smmu_iotlb_inv_iova(SMMUState *s, int asid, dma_addr_t iova,
+                    uint8_t tg, uint64_t num_pages, uint8_t ttl)
 {
-    SMMUIOTLBPageInvInfo info = {.asid = asid, .iova = iova};
+    if (ttl && (num_pages == 1)) {
+        uint64_t key = smmu_get_iotlb_key(asid, iova, tg, ttl);
 
-    trace_smmu_iotlb_inv_iova(asid, iova);
-    g_hash_table_foreach_remove(s->iotlb, smmu_hash_remove_by_asid_iova, &info);
+        g_hash_table_remove(s->iotlb, &key);
+    } else {
+            /* if tg is not set we use 4KB range invalidation */
+            uint8_t granule = tg ? tg * 2 + 10 : 12;
+
+            SMMUIOTLBPageInvInfo info = {
+                 .asid = asid, .iova = iova,
+                 .mask = (num_pages * 1 << granule) - 1};
+
+            g_hash_table_foreach_remove(s->iotlb,
+                                        smmu_hash_remove_by_asid_iova,
+                                        &info);
+    }
 }
 
 inline void smmu_iotlb_inv_asid(SMMUState *s, uint16_t asid)
hw/arm/smmuv3.c +41 -23

111 lines not on this page · raw message

hw/arm/trace-events +2 -2
index c219fe9e82..3d905e0f7d 100644
@@ -45,11 +45,11 @@ smmuv3_cmdq_cfgi_ste_range(int start, int end) "start=0x%d - end=0x%d"
 smmuv3_cmdq_cfgi_cd(uint32_t sid) "streamid = %d"
 smmuv3_config_cache_hit(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache HIT for sid %d (hits=%d, misses=%d, hit rate=%d)"
 smmuv3_config_cache_miss(uint32_t sid, uint32_t hits, uint32_t misses, uint32_t perc) "Config cache MISS for sid %d (hits=%d, misses=%d, hit rate=%d)"
-smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" leaf=%d"
+smmuv3_s1_range_inval(int vmid, int asid, uint64_t addr, uint8_t tg, uint64_t num_pages, uint8_t ttl, bool leaf) "vmid =%d asid =%d addr=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64" ttl=%d leaf=%d"
 smmuv3_cmdq_tlbi_nh(void) ""
 smmuv3_cmdq_tlbi_nh_asid(uint16_t asid) "asid=%d"
 smmuv3_config_cache_inv(uint32_t sid) "Config cache INV for sid %d"
 smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s"
 smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s"
-smmuv3_inv_notifiers_iova(const char *name, uint16_t asid, uint64_t iova) "iommu mr=%s asid=%d iova=0x%"PRIx64
+smmuv3_inv_notifiers_iova(const char *name, uint16_t asid, uint64_t iova, uint8_t tg, uint64_t num_pages) "iommu mr=%s asid=%d iova=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64
 
Signature · 1 line
2.21.3
author
#14 lore raw

PATCH v2 9/9 hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation

In reply to Eric Auger

Expose the RIL bit so that the guest driver uses range invalidation.

---
 hw/arm/smmuv3-internal.h | 1 +
 hw/arm/smmuv3.c          | 2 ++
 2 files changed, 3 insertions(+)
hw/arm/smmuv3-internal.h +1 -0
index 5babf72f7d..4e7ec252ed 100644
@@ -54,6 +54,7 @@ REG32(IDR1,                0x4)
 
 REG32(IDR2,                0x8)
 REG32(IDR3,                0xc)
+    FIELD(IDR3, RIL,          10, 1);
 REG32(IDR4,                0x10)
 REG32(IDR5,                0x14)
      FIELD(IDR5, OAS,         0, 3);
hw/arm/smmuv3.c +2 -0
index 89ab11fc36..add4ba4543 100644
@@ -254,6 +254,8 @@ static void smmuv3_init_regs(SMMUv3State *s)
     s->idr[1] = FIELD_DP32(s->idr[1], IDR1, EVENTQS, SMMU_EVENTQS);
     s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS,   SMMU_CMDQS);
 
+    s->idr[3] = FIELD_DP32(s->idr[3], IDR3, RIL, 1);
+
    /* 4K and 64K granule support */
     s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN4K, 1);
     s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN64K, 1);
Signature · 1 line
2.21.3
#15 lore raw

Re: [PATCH v2 9/9] hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation

Quoted 2 lines · Eric Auger

Expose the RIL bit so that the guest driver uses range invalidation.

Hmm, this is a v3.2 feature... so strictly, in order to advertise it you would need to claim at least v3.1 in SMMU_AIDR and implement all the mandatory v3.1 behaviour ;)

Robin.

Quoted 33 lines
---
  hw/arm/smmuv3-internal.h | 1 +
  hw/arm/smmuv3.c          | 2 ++
  2 files changed, 3 insertions(+)
hw/arm/smmuv3-internal.h +1 -0
index 5babf72f7d..4e7ec252ed 100644
@@ -54,6 +54,7 @@ REG32(IDR1,                0x4)
 
  REG32(IDR2,                0x8)
  REG32(IDR3,                0xc)
+    FIELD(IDR3, RIL,          10, 1);
  REG32(IDR4,                0x10)
  REG32(IDR5,                0x14)
       FIELD(IDR5, OAS,         0, 3);
hw/arm/smmuv3.c +2 -0
index 89ab11fc36..add4ba4543 100644
@@ -254,6 +254,8 @@ static void smmuv3_init_regs(SMMUv3State *s)
      s->idr[1] = FIELD_DP32(s->idr[1], IDR1, EVENTQS, SMMU_EVENTQS);
      s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS,   SMMU_CMDQS);
 
+    s->idr[3] = FIELD_DP32(s->idr[3], IDR3, RIL, 1);
+
     /* 4K and 64K granule support */
      s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN4K, 1);
      s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN64K, 1);
author
#16 lore raw

Re: [PATCH v2 9/9] hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation

Hi Robin,

Quoted 7 lines · Robin Murphy
Quoted 2 lines · Eric Auger

Expose the RIL bit so that the guest driver uses range invalidation.

Hmm, this is a v3.2 feature... so strictly, in order to advertise it you would need to claim at least v3.1 in SMMU_AIDR and implement all the mandatory v3.1 behaviour ;)

AIDR is not modeled at the moment in this emulation code.
I do not see it used in the linux smmuv3 driver either. I can initialize it to 0x2 for the sake of completeness.

With respect to the 'mandatory features', could you please help me determining what are they. Most of the features that would impact this emulation code look optional to me (52bit support, PBHA, MPAM).

This emulation code is tested against the latest linux kernel.

Thanks

Eric

Quoted 34 lines

Robin.

Quoted 31 lines

---
  hw/arm/smmuv3-internal.h | 1 +
  hw/arm/smmuv3.c          | 2 ++

  2 files changed, 3 insertions(+)
hw/arm/smmuv3-internal.h +1 -0
index 5babf72f7d..4e7ec252ed 100644
@@ -54,6 +54,7 @@ REG32(IDR1,                0x4)
     REG32(IDR2,                0x8)
   REG32(IDR3,                0xc)
+    FIELD(IDR3, RIL,          10, 1);
   REG32(IDR4,                0x10)
   REG32(IDR5,                0x14)
        FIELD(IDR5, OAS,         0, 3);
hw/arm/smmuv3.c +1 -0
index 89ab11fc36..add4ba4543 100644
@@ -254,6 +254,8 @@ static void smmuv3_init_regs(SMMUv3State *s)
       s->idr[1] = FIELD_DP32(s->idr[1], IDR1, EVENTQS, SMMU_EVENTQS);
       s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS,   SMMU_CMDQS);
   +    s->idr[3] = FIELD_DP32(s->idr[3], IDR3, RIL, 1);
+
      /* 4K and 64K granule support */
       s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN4K, 1);
       s->idr[5] = FIELD_DP32(s->idr[5], IDR5, GRAN64K, 1);