Skip to contentSkip to conversation

ppc: misc fixes for Radix and Hash

PATCH Applied Cédric Le Goater 3 participants 14 messages Apr 11, 2019 – Apr 15, 2019

lore
Share & tools
lore
#0 lore raw

cover ppc: misc fixes for Radix and Hash

Hi,

Here is a series of patches from Ben providing MMU fixes for sPAPR and PowerNV. They seem of interest and deserve a closer look by people with good knowledge of the PPC MMU.

Thanks,

C.

Benjamin Herrenschmidt (6):
target/ppc: Don't check UPRT in radix mode when in HV real mode ppc/spapr: Use proper HPTE accessors for H_READ ppc/hash64: Rework R and C bit updates
ppc/hash32: Rework R and C bit updates
memory_ldst: Add atomic ops for PTE updates
ppc: Fix radix RC updates

 include/exec/memory_ldst.inc.h |  6 +++
 include/hw/ppc/spapr.h         |  2 +
 target/ppc/cpu.h               |  5 +-
 target/ppc/mmu-hash64.h        |  2 -
 hw/ppc/spapr.c                 | 41 ++++++++++++++--
 hw/ppc/spapr_hcall.c           | 24 +++++-----
 memory_ldst.inc.c              | 80 +++++++++++++++++++++++++++++++
 target/ppc/mmu-hash32.c        | 45 ++++++++++++------
 target/ppc/mmu-hash64.c        | 76 ++++++++++++++++++------------
 target/ppc/mmu-radix64.c       | 86 ++++++++++++++++++++++++++++------
 10 files changed, 288 insertions(+), 79 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/6 target/ppc: Don't check UPRT in radix mode when in HV real mode

From: Benjamin Herrenschmidt <[email protected]>

It appears that during kexec, we run for a while in hypervisor real mode with LPCR:HR set and LPCR:UPRT clear, which trips the assertion in ppc_radix64_handle_mmu_fault().

First this shouldn't be an assertion, it's a guest error.

Then we shouldn't be checking these things in hypervisor real mode (or in virtual hypervisor guest real mode which is similar) as the real HW won't use those LPCR bits in those cases anyway, so technically it's ok to have this discrepancy.

---
 target/ppc/mmu-radix64.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
target/ppc/mmu-radix64.c +13 -3
index ca1fb2673f93..cac076ee92d9 100644
@@ -228,10 +228,10 @@ int ppc_radix64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
     ppc_v3_pate_t pate;
 
     assert((rwx == 0) || (rwx == 1) || (rwx == 2));
-    assert(ppc64_use_proc_tbl(cpu));
 
-    /* Real Mode Access */
-    if (((rwx == 2) && (msr_ir == 0)) || ((rwx != 2) && (msr_dr == 0))) {
+    /* HV or virtual hypervisor Real Mode Access */
+    if ((msr_hv || cpu->vhyp) &&
+        (((rwx == 2) && (msr_ir == 0)) || ((rwx != 2) && (msr_dr == 0)))) {
         /* In real mode top 4 effective addr bits (mostly) ignored */
         raddr = eaddr & 0x0FFFFFFFFFFFFFFFULL;
 
@@ -241,6 +241,16 @@ int ppc_radix64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
         return 0;
     }
 
+    /*
+     * Check UPRT (we avoid the check in real mode to deal with
+     * transitional states during kexec.
+     */
+    if (!ppc64_use_proc_tbl(cpu)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "LPCR:UPRT not set in radix mode ! LPCR=%016lx\n",
+                      env->spr[SPR_LPCR]);
+    }
+
     /* Virtual Mode Access - get the fully qualified address */
     if (!ppc_radix64_get_fully_qualified_addr(env, eaddr, &lpid, &pid)) {
         ppc_radix64_raise_segi(cpu, rwx, eaddr);
Signature · 1 line
2.20.1
Applied
#2 lore raw

Re: [Qemu-devel] [PATCH 1/6] target/ppc: Don't check UPRT in radix mode when in HV real mode

Quoted 8 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

It appears that during kexec, we run for a while in hypervisor real mode with LPCR:HR set and LPCR:UPRT clear, which trips the assertion in ppc_radix64_handle_mmu_fault().

First this shouldn't be an assertion, it's a guest error.

Quoted 7 lines

Then we shouldn't be checking these things in hypervisor real mode (or in virtual hypervisor guest real mode which is similar) as the real HW won't use those LPCR bits in those cases anyway, so technically it's ok to have this discrepancy.

Applied to ppc-for-4.1, thanks.

Quoted 9 lines
---
 target/ppc/mmu-radix64.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
target/ppc/mmu-radix64.c +0 -0
index ca1fb2673f93..cac076ee92d9 100644

target/ppc/mmu-radix64.c int ppc_radix64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx, · line 234

Quoted 10 lines
target/ppc/mmu-radix64.c +3 -3
@@ -228,10 +228,10 @@ int ppc_radix64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
     ppc_v3_pate_t pate;
 
     assert((rwx == 0) || (rwx == 1) || (rwx == 2));
-    assert(ppc64_use_proc_tbl(cpu));
 
-    /* Real Mode Access */
-    if (((rwx == 2) && (msr_ir == 0)) || ((rwx != 2) && (msr_dr == 0))) {
+    /* HV or virtual hypervisor Real Mode Access */
+    if ((msr_hv || cpu->vhyp) &&
+        (((rwx == 2) && (msr_ir == 0)) || ((rwx != 2) && (msr_dr == 0)))) {

Hrm.. this looks like a different bug fix. Were we preivously incorrectly handling guest real mode when running powernv?

Quoted 20 lines
target/ppc/mmu-radix64.c +10 -0
         /* In real mode top 4 effective addr bits (mostly) ignored */
         raddr = eaddr & 0x0FFFFFFFFFFFFFFFULL;
 
@@ -241,6 +241,16 @@ int ppc_radix64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
         return 0;
     }
 
+    /*
+     * Check UPRT (we avoid the check in real mode to deal with
+     * transitional states during kexec.
+     */
+    if (!ppc64_use_proc_tbl(cpu)) {
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "LPCR:UPRT not set in radix mode ! LPCR=%016lx\n",
+                      env->spr[SPR_LPCR]);
+    }
+
     /* Virtual Mode Access - get the fully qualified address */
     if (!ppc_radix64_get_fully_qualified_addr(env, eaddr, &lpid, &pid)) {
         ppc_radix64_raise_segi(cpu, rwx, eaddr);
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#3 lore raw

PATCH 2/6 ppc/spapr: Use proper HPTE accessors for H_READ

In reply to Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

---
 hw/ppc/spapr_hcall.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
hw/ppc/spapr_hcall.c +5 -6
index be8044a92f2f..7761d4a841af 100644
@@ -304,8 +304,8 @@ static target_ulong h_read(PowerPCCPU *cpu, SpaprMachineState *spapr,
 {
     target_ulong flags = args[0];
     target_ulong ptex = args[1];
-    uint8_t *hpte;
     int i, ridx, n_entries = 1;
+    const ppc_hash_pte64_t *hptes;
 
     if (!valid_ptex(cpu, ptex)) {
         return H_PARAMETER;
@@ -317,13 +317,12 @@ static target_ulong h_read(PowerPCCPU *cpu, SpaprMachineState *spapr,
         n_entries = 4;
     }
 
-    hpte = spapr->htab + (ptex * HASH_PTE_SIZE_64);
-
+    hptes = ppc_hash64_map_hptes(cpu, ptex, n_entries);
     for (i = 0, ridx = 0; i < n_entries; i++) {
-        args[ridx++] = ldq_p(hpte);
-        args[ridx++] = ldq_p(hpte + (HASH_PTE_SIZE_64/2));
-        hpte += HASH_PTE_SIZE_64;
+        args[ridx++] = ppc_hash64_hpte0(cpu, hptes, i);
+        args[ridx++] = ppc_hash64_hpte1(cpu, hptes, i);
     }
+    ppc_hash64_unmap_hptes(cpu, hptes, ptex, n_entries);
 
     return H_SUCCESS;
 }
Signature · 1 line
2.20.1
Applied
#4 lore raw

Re: [Qemu-devel] [PATCH 2/6] ppc/spapr: Use proper HPTE accessors for H_READ

Quoted 4 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

Applied, thanks.

Quoted 36 lines
---
 hw/ppc/spapr_hcall.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
hw/ppc/spapr_hcall.c +5 -6
index be8044a92f2f..7761d4a841af 100644
@@ -304,8 +304,8 @@ static target_ulong h_read(PowerPCCPU *cpu, SpaprMachineState *spapr,
 {
     target_ulong flags = args[0];
     target_ulong ptex = args[1];
-    uint8_t *hpte;
     int i, ridx, n_entries = 1;
+    const ppc_hash_pte64_t *hptes;
 
     if (!valid_ptex(cpu, ptex)) {
         return H_PARAMETER;
@@ -317,13 +317,12 @@ static target_ulong h_read(PowerPCCPU *cpu, SpaprMachineState *spapr,
         n_entries = 4;
     }
 
-    hpte = spapr->htab + (ptex * HASH_PTE_SIZE_64);
-
+    hptes = ppc_hash64_map_hptes(cpu, ptex, n_entries);
     for (i = 0, ridx = 0; i < n_entries; i++) {
-        args[ridx++] = ldq_p(hpte);
-        args[ridx++] = ldq_p(hpte + (HASH_PTE_SIZE_64/2));
-        hpte += HASH_PTE_SIZE_64;
+        args[ridx++] = ppc_hash64_hpte0(cpu, hptes, i);
+        args[ridx++] = ppc_hash64_hpte1(cpu, hptes, i);
     }
+    ppc_hash64_unmap_hptes(cpu, hptes, ptex, n_entries);
 
     return H_SUCCESS;
 }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#5 lore raw

PATCH 3/6 ppc/hash64: Rework R and C bit updates

In reply to Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

With MT-TCG, we are now running translation in a racy way, thus we need to mimic hardware when it comes to updating the R and C bits, by doing byte stores.

The current "store_hpte" abstraction is ill suited for this, we replace it with two separate callbacks for setting R and C.

---
 include/hw/ppc/spapr.h  |  2 ++
 target/ppc/cpu.h        |  4 +--
 target/ppc/mmu-hash64.h |  2 --
 hw/ppc/spapr.c          | 41 +++++++++++++++++++---
 hw/ppc/spapr_hcall.c    | 13 +++----
 target/ppc/mmu-hash64.c | 76 ++++++++++++++++++++++++-----------------
 6 files changed, 93 insertions(+), 45 deletions(-)
include/hw/ppc/spapr.h +2 -0
index 9331f5e0b955..7e32f309c2be 100644
@@ -784,6 +784,8 @@ void spapr_reallocate_hpt(SpaprMachineState *spapr, int shift,
                           Error **errp);
 void spapr_clear_pending_events(SpaprMachineState *spapr);
 int spapr_max_server_number(SpaprMachineState *spapr);
+void spapr_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
+                      uint64_t pte0, uint64_t pte1);
 
 /* DRC callbacks. */
 void spapr_core_release(DeviceState *dev);
target/ppc/cpu.h +2 -2
index 07cea7268c0c..a4420636dde0 100644
@@ -1268,8 +1268,8 @@ struct PPCVirtualHypervisorClass {
     void (*unmap_hptes)(PPCVirtualHypervisor *vhyp,
                         const ppc_hash_pte64_t *hptes,
                         hwaddr ptex, int n);
-    void (*store_hpte)(PPCVirtualHypervisor *vhyp, hwaddr ptex,
-                       uint64_t pte0, uint64_t pte1);
+    void (*hpte_set_c)(PPCVirtualHypervisor *vhyp, hwaddr ptex, uint64_t pte1);
+    void (*hpte_set_r)(PPCVirtualHypervisor *vhyp, hwaddr ptex, uint64_t pte1);
     void (*get_pate)(PPCVirtualHypervisor *vhyp, ppc_v3_pate_t *entry);
     target_ulong (*encode_hpt_for_kvm_pr)(PPCVirtualHypervisor *vhyp);
 };
target/ppc/mmu-hash64.h +0 -2
index 6b555b72200f..bc362964122d 100644
@@ -10,8 +10,6 @@ int ppc_store_slb(PowerPCCPU *cpu, target_ulong slot,
 hwaddr ppc_hash64_get_phys_page_debug(PowerPCCPU *cpu, target_ulong addr);
 int ppc_hash64_handle_mmu_fault(PowerPCCPU *cpu, vaddr address, int rw,
                                 int mmu_idx);
-void ppc_hash64_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
-                           uint64_t pte0, uint64_t pte1);
 void ppc_hash64_tlb_flush_hpte(PowerPCCPU *cpu,
                                target_ulong pte_index,
                                target_ulong pte0, target_ulong pte1);
hw/ppc/spapr.c +37 -4

60 lines not on this page · raw message

hw/ppc/spapr_hcall.c +7 -6
index 7761d4a841af..a27c9a01aa97 100644
@@ -118,7 +118,7 @@ static target_ulong h_enter(PowerPCCPU *cpu, SpaprMachineState *spapr,
         ppc_hash64_unmap_hptes(cpu, hptes, ptex, 1);
     }
 
-    ppc_hash64_store_hpte(cpu, ptex + slot, pteh | HPTE64_V_HPTE_DIRTY, ptel);
+    spapr_store_hpte(cpu, ptex + slot, pteh | HPTE64_V_HPTE_DIRTY, ptel);
 
     args[0] = ptex + slot;
     return H_SUCCESS;
@@ -131,7 +131,8 @@ typedef enum {
     REMOVE_HW = 3,
 } RemoveResult;
 
-static RemoveResult remove_hpte(PowerPCCPU *cpu, target_ulong ptex,
+static RemoveResult remove_hpte(PowerPCCPU *cpu
+                                , target_ulong ptex,
                                 target_ulong avpn,
                                 target_ulong flags,
                                 target_ulong *vp, target_ulong *rp)
@@ -155,7 +156,7 @@ static RemoveResult remove_hpte(PowerPCCPU *cpu, target_ulong ptex,
     }
     *vp = v;
     *rp = r;
-    ppc_hash64_store_hpte(cpu, ptex, HPTE64_V_HPTE_DIRTY, 0);
+    spapr_store_hpte(cpu, ptex, HPTE64_V_HPTE_DIRTY, 0);
     ppc_hash64_tlb_flush_hpte(cpu, ptex, v, r);
     return REMOVE_SUCCESS;
 }
@@ -289,13 +290,13 @@ static target_ulong h_protect(PowerPCCPU *cpu, SpaprMachineState *spapr,
     r |= (flags << 55) & HPTE64_R_PP0;
     r |= (flags << 48) & HPTE64_R_KEY_HI;
     r |= flags & (HPTE64_R_PP | HPTE64_R_N | HPTE64_R_KEY_LO);
-    ppc_hash64_store_hpte(cpu, ptex,
-                          (v & ~HPTE64_V_VALID) | HPTE64_V_HPTE_DIRTY, 0);
+    spapr_store_hpte(cpu, ptex,
+                     (v & ~HPTE64_V_VALID) | HPTE64_V_HPTE_DIRTY, 0);
     ppc_hash64_tlb_flush_hpte(cpu, ptex, v, r);
     /* Flush the tlb */
     check_tlb_flush(env, true);
     /* Don't need a memory barrier, due to qemu's global lock */
-    ppc_hash64_store_hpte(cpu, ptex, v | HPTE64_V_HPTE_DIRTY, r);
+    spapr_store_hpte(cpu, ptex, v | HPTE64_V_HPTE_DIRTY, r);
     return H_SUCCESS;
 }
 
target/ppc/mmu-hash64.c +45 -31

101 lines not on this page · raw message

Signature · 1 line
2.20.1
Applied
#6 lore raw

Re: [Qemu-devel] [PATCH 3/6] ppc/hash64: Rework R and C bit updates

Quoted 11 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

With MT-TCG, we are now running translation in a racy way, thus we need to mimic hardware when it comes to updating the R and C bits, by doing byte stores.

The current "store_hpte" abstraction is ill suited for this, we replace it with two separate callbacks for setting R and C.

Applied, thanks.

Quoted 275 lines
---
 include/hw/ppc/spapr.h  |  2 ++
 target/ppc/cpu.h        |  4 +--
 target/ppc/mmu-hash64.h |  2 --
 hw/ppc/spapr.c          | 41 +++++++++++++++++++---
 hw/ppc/spapr_hcall.c    | 13 +++----
 target/ppc/mmu-hash64.c | 76 ++++++++++++++++++++++++-----------------
 6 files changed, 93 insertions(+), 45 deletions(-)
include/hw/ppc/spapr.h +2 -0
index 9331f5e0b955..7e32f309c2be 100644
@@ -784,6 +784,8 @@ void spapr_reallocate_hpt(SpaprMachineState *spapr, int shift,
                           Error **errp);
 void spapr_clear_pending_events(SpaprMachineState *spapr);
 int spapr_max_server_number(SpaprMachineState *spapr);
+void spapr_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
+                      uint64_t pte0, uint64_t pte1);
 
 /* DRC callbacks. */
 void spapr_core_release(DeviceState *dev);
target/ppc/cpu.h +2 -2
index 07cea7268c0c..a4420636dde0 100644
@@ -1268,8 +1268,8 @@ struct PPCVirtualHypervisorClass {
     void (*unmap_hptes)(PPCVirtualHypervisor *vhyp,
                         const ppc_hash_pte64_t *hptes,
                         hwaddr ptex, int n);
-    void (*store_hpte)(PPCVirtualHypervisor *vhyp, hwaddr ptex,
-                       uint64_t pte0, uint64_t pte1);
+    void (*hpte_set_c)(PPCVirtualHypervisor *vhyp, hwaddr ptex, uint64_t pte1);
+    void (*hpte_set_r)(PPCVirtualHypervisor *vhyp, hwaddr ptex, uint64_t pte1);
     void (*get_pate)(PPCVirtualHypervisor *vhyp, ppc_v3_pate_t *entry);
     target_ulong (*encode_hpt_for_kvm_pr)(PPCVirtualHypervisor *vhyp);
 };
target/ppc/mmu-hash64.h +0 -2
index 6b555b72200f..bc362964122d 100644
@@ -10,8 +10,6 @@ int ppc_store_slb(PowerPCCPU *cpu, target_ulong slot,
 hwaddr ppc_hash64_get_phys_page_debug(PowerPCCPU *cpu, target_ulong addr);
 int ppc_hash64_handle_mmu_fault(PowerPCCPU *cpu, vaddr address, int rw,
                                 int mmu_idx);
-void ppc_hash64_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
-                           uint64_t pte0, uint64_t pte1);
 void ppc_hash64_tlb_flush_hpte(PowerPCCPU *cpu,
                                target_ulong pte_index,
                                target_ulong pte0, target_ulong pte1);
hw/ppc/spapr.c +37 -4
index b81e237635cd..c56939a43b64 100644
@@ -1520,10 +1520,10 @@ static void spapr_unmap_hptes(PPCVirtualHypervisor *vhyp,
     /* Nothing to do for qemu managed HPT */
 }
 
-static void spapr_store_hpte(PPCVirtualHypervisor *vhyp, hwaddr ptex,
-                             uint64_t pte0, uint64_t pte1)
+void spapr_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
+                      uint64_t pte0, uint64_t pte1)
 {
-    SpaprMachineState *spapr = SPAPR_MACHINE(vhyp);
+    SpaprMachineState *spapr = SPAPR_MACHINE(cpu->vhyp);
     hwaddr offset = ptex * HASH_PTE_SIZE_64;
 
     if (!spapr->htab) {
@@ -1551,6 +1551,38 @@ static void spapr_store_hpte(PPCVirtualHypervisor *vhyp, hwaddr ptex,
     }
 }
 
+static void spapr_hpte_set_c(PPCVirtualHypervisor *vhyp, hwaddr ptex,
+                             uint64_t pte1)
+{
+    hwaddr offset = ptex * HASH_PTE_SIZE_64 + 15;
+    SpaprMachineState *spapr = SPAPR_MACHINE(vhyp);
+
+    if (!spapr->htab) {
+        /* There should always be a hash table when this is called */
+        error_report("spapr_hpte_set_c called with no hash table !");
+        return;
+    }
+
+    /* The HW performs a non-atomic byte update */
+    stb_p(spapr->htab + offset, (pte1 & 0xff) | 0x80);
+}
+
+static void spapr_hpte_set_r(PPCVirtualHypervisor *vhyp, hwaddr ptex,
+                             uint64_t pte1)
+{
+    hwaddr offset = ptex * HASH_PTE_SIZE_64 + 14;
+    SpaprMachineState *spapr = SPAPR_MACHINE(vhyp);
+
+    if (!spapr->htab) {
+        /* There should always be a hash table when this is called */
+        error_report("spapr_hpte_set_r called with no hash table !");
+        return;
+    }
+
+    /* The HW performs a non-atomic byte update */
+    stb_p(spapr->htab + offset, ((pte1 >> 8) & 0xff) | 0x01);
+}
+
 int spapr_hpt_shift_for_ramsize(uint64_t ramsize)
 {
     int shift;
@@ -4291,7 +4323,8 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     vhc->hpt_mask = spapr_hpt_mask;
     vhc->map_hptes = spapr_map_hptes;
     vhc->unmap_hptes = spapr_unmap_hptes;
-    vhc->store_hpte = spapr_store_hpte;
+    vhc->hpte_set_c = spapr_hpte_set_c;
+    vhc->hpte_set_r = spapr_hpte_set_r;
     vhc->get_pate = spapr_get_pate;
     vhc->encode_hpt_for_kvm_pr = spapr_encode_hpt_for_kvm_pr;
     xic->ics_get = spapr_ics_get;
hw/ppc/spapr_hcall.c +7 -6
index 7761d4a841af..a27c9a01aa97 100644
@@ -118,7 +118,7 @@ static target_ulong h_enter(PowerPCCPU *cpu, SpaprMachineState *spapr,
         ppc_hash64_unmap_hptes(cpu, hptes, ptex, 1);
     }
 
-    ppc_hash64_store_hpte(cpu, ptex + slot, pteh | HPTE64_V_HPTE_DIRTY, ptel);
+    spapr_store_hpte(cpu, ptex + slot, pteh | HPTE64_V_HPTE_DIRTY, ptel);
 
     args[0] = ptex + slot;
     return H_SUCCESS;
@@ -131,7 +131,8 @@ typedef enum {
     REMOVE_HW = 3,
 } RemoveResult;
 
-static RemoveResult remove_hpte(PowerPCCPU *cpu, target_ulong ptex,
+static RemoveResult remove_hpte(PowerPCCPU *cpu
+                                , target_ulong ptex,
                                 target_ulong avpn,
                                 target_ulong flags,
                                 target_ulong *vp, target_ulong *rp)
@@ -155,7 +156,7 @@ static RemoveResult remove_hpte(PowerPCCPU *cpu, target_ulong ptex,
     }
     *vp = v;
     *rp = r;
-    ppc_hash64_store_hpte(cpu, ptex, HPTE64_V_HPTE_DIRTY, 0);
+    spapr_store_hpte(cpu, ptex, HPTE64_V_HPTE_DIRTY, 0);
     ppc_hash64_tlb_flush_hpte(cpu, ptex, v, r);
     return REMOVE_SUCCESS;
 }
@@ -289,13 +290,13 @@ static target_ulong h_protect(PowerPCCPU *cpu, SpaprMachineState *spapr,
     r |= (flags << 55) & HPTE64_R_PP0;
     r |= (flags << 48) & HPTE64_R_KEY_HI;
     r |= flags & (HPTE64_R_PP | HPTE64_R_N | HPTE64_R_KEY_LO);
-    ppc_hash64_store_hpte(cpu, ptex,
-                          (v & ~HPTE64_V_VALID) | HPTE64_V_HPTE_DIRTY, 0);
+    spapr_store_hpte(cpu, ptex,
+                     (v & ~HPTE64_V_VALID) | HPTE64_V_HPTE_DIRTY, 0);
     ppc_hash64_tlb_flush_hpte(cpu, ptex, v, r);
     /* Flush the tlb */
     check_tlb_flush(env, true);
     /* Don't need a memory barrier, due to qemu's global lock */
-    ppc_hash64_store_hpte(cpu, ptex, v | HPTE64_V_HPTE_DIRTY, r);
+    spapr_store_hpte(cpu, ptex, v | HPTE64_V_HPTE_DIRTY, r);
     return H_SUCCESS;
 }
 
target/ppc/mmu-hash64.c +45 -31
index 8956ad55d96f..b4a27d46e382 100644
@@ -724,6 +724,39 @@ static void ppc_hash64_set_dsi(CPUState *cs, uint64_t dar, uint64_t dsisr)
 }
 
 
+static void ppc_hash64_set_r(PowerPCCPU *cpu, hwaddr ptex, uint64_t pte1)
+{
+    hwaddr base, offset = ptex * HASH_PTE_SIZE_64 + 16;
+
+    if (cpu->vhyp) {
+        PPCVirtualHypervisorClass *vhc =
+            PPC_VIRTUAL_HYPERVISOR_GET_CLASS(cpu->vhyp);
+        vhc->hpte_set_r(cpu->vhyp, ptex, pte1);
+        return;
+    }
+    base = ppc_hash64_hpt_base(cpu);
+
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, ((pte1 >> 8) & 0xff) | 0x01);
+}
+
+static void ppc_hash64_set_c(PowerPCCPU *cpu, hwaddr ptex, uint64_t pte1)
+{
+    hwaddr base, offset = ptex * HASH_PTE_SIZE_64 + 15;
+
+    if (cpu->vhyp) {
+        PPCVirtualHypervisorClass *vhc =
+            PPC_VIRTUAL_HYPERVISOR_GET_CLASS(cpu->vhyp);
+        vhc->hpte_set_c(cpu->vhyp, ptex, pte1);
+        return;
+    }
+    base = ppc_hash64_hpt_base(cpu);
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, (pte1 & 0xff) | 0x80);
+}
+
 int ppc_hash64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr,
                                 int rwx, int mmu_idx)
 {
@@ -734,7 +767,6 @@ int ppc_hash64_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr,
     hwaddr ptex;
     ppc_hash_pte64_t pte;
     int exec_prot, pp_prot, amr_prot, prot;
-    uint64_t new_pte1;
     const int need_prot[] = {PAGE_READ, PAGE_WRITE, PAGE_EXEC};
     hwaddr raddr;
 
@@ -882,19 +914,19 @@ skip_slb_search:
 
     /* 6. Update PTE referenced and changed bits if necessary */
 
-    new_pte1 = pte.pte1 | HPTE64_R_R; /* set referenced bit */
-    if (rwx == 1) {
-        new_pte1 |= HPTE64_R_C; /* set changed (dirty) bit */
-    } else {
-        /*
-         * Treat the page as read-only for now, so that a later write
-         * will pass through this function again to set the C bit
-         */
-        prot &= ~PAGE_WRITE;
+    if (!(pte.pte1 & HPTE64_R_R)) {
+        ppc_hash64_set_r(cpu, ptex, pte.pte1);
     }
-
-    if (new_pte1 != pte.pte1) {
-        ppc_hash64_store_hpte(cpu, ptex, pte.pte0, new_pte1);
+    if (!(pte.pte1 & HPTE64_R_C)) {
+        if (rwx == 1) {
+            ppc_hash64_set_c(cpu, ptex, pte.pte1);
+        } else {
+            /*
+             * Treat the page as read-only for now, so that a later write
+             * will pass through this function again to set the C bit
+             */
+            prot &= ~PAGE_WRITE;
+        }
     }
 
     /* 7. Determine the real address from the PTE */
@@ -953,24 +985,6 @@ hwaddr ppc_hash64_get_phys_page_debug(PowerPCCPU *cpu, target_ulong addr)
         & TARGET_PAGE_MASK;
 }
 
-void ppc_hash64_store_hpte(PowerPCCPU *cpu, hwaddr ptex,
-                           uint64_t pte0, uint64_t pte1)
-{
-    hwaddr base;
-    hwaddr offset = ptex * HASH_PTE_SIZE_64;
-
-    if (cpu->vhyp) {
-        PPCVirtualHypervisorClass *vhc =
-            PPC_VIRTUAL_HYPERVISOR_GET_CLASS(cpu->vhyp);
-        vhc->store_hpte(cpu->vhyp, ptex, pte0, pte1);
-        return;
-    }
-    base = ppc_hash64_hpt_base(cpu);
-
-    stq_phys(CPU(cpu)->as, base + offset, pte0);
-    stq_phys(CPU(cpu)->as, base + offset + HASH_PTE_SIZE_64 / 2, pte1);
-}
-
 void ppc_hash64_tlb_flush_hpte(PowerPCCPU *cpu, target_ulong ptex,
                                target_ulong pte0, target_ulong pte1)
 {
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#7 lore raw

PATCH 4/6 ppc/hash32: Rework R and C bit updates

In reply to Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

With MT-TCG, we are now running translation in a racy way, thus we need to mimic hardware when it comes to updating the R and C bits, by doing byte stores.

---
 target/ppc/mmu-hash32.c | 45 ++++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 14 deletions(-)
target/ppc/mmu-hash32.c +31 -14
index 3f4dee835e2d..55cf156a0b49 100644
@@ -345,6 +345,24 @@ static hwaddr ppc_hash32_pteg_search(PowerPCCPU *cpu, hwaddr pteg_off,
     return -1;
 }
 
+static void ppc_hash32_set_r(PowerPCCPU *cpu, hwaddr pte_offset, uint32_t pte1)
+{
+    target_ulong base = ppc_hash32_hpt_base(cpu);
+    hwaddr offset = pte_offset + 6;
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, ((pte1 >> 8) & 0xff) | 0x01);
+}
+
+static void ppc_hash32_set_c(PowerPCCPU *cpu, hwaddr pte_offset, uint64_t pte1)
+{
+    target_ulong base = ppc_hash32_hpt_base(cpu);
+    hwaddr offset = pte_offset + 7;
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, (pte1 & 0xff) | 0x80);
+}
+
 static hwaddr ppc_hash32_htab_lookup(PowerPCCPU *cpu,
                                      target_ulong sr, target_ulong eaddr,
                                      ppc_hash_pte32_t *pte)
@@ -403,7 +421,6 @@ int ppc_hash32_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
     hwaddr pte_offset;
     ppc_hash_pte32_t pte;
     int prot;
-    uint32_t new_pte1;
     const int need_prot[] = {PAGE_READ, PAGE_WRITE, PAGE_EXEC};
     hwaddr raddr;
 
@@ -519,20 +536,20 @@ int ppc_hash32_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
 
     /* 8. Update PTE referenced and changed bits if necessary */
 
-    new_pte1 = pte.pte1 | HPTE32_R_R; /* set referenced bit */
-    if (rwx == 1) {
-        new_pte1 |= HPTE32_R_C; /* set changed (dirty) bit */
-    } else {
-        /*
-         * Treat the page as read-only for now, so that a later write
-         * will pass through this function again to set the C bit
-         */
-        prot &= ~PAGE_WRITE;
-    }
-
-    if (new_pte1 != pte.pte1) {
-        ppc_hash32_store_hpte1(cpu, pte_offset, new_pte1);
+    if (!(pte.pte1 & HPTE32_R_R)) {
+        ppc_hash32_set_r(cpu, pte_offset, pte.pte1);
     }
+    if (!(pte.pte1 & HPTE32_R_C)) {
+        if (rwx == 1) {
+            ppc_hash32_set_c(cpu, pte_offset, pte.pte1);
+        } else {
+            /*
+             * Treat the page as read-only for now, so that a later write
+             * will pass through this function again to set the C bit
+             */
+            prot &= ~PAGE_WRITE;
+        }
+     }
 
     /* 9. Determine the real address from the PTE */
 
Signature · 1 line
2.20.1
Applied
#8 lore raw

Re: [Qemu-devel] [PATCH 4/6] ppc/hash32: Rework R and C bit updates

Quoted 8 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

With MT-TCG, we are now running translation in a racy way, thus we need to mimic hardware when it comes to updating the R and C bits, by doing byte stores.

Applied, thanks.

Quoted 75 lines
---
 target/ppc/mmu-hash32.c | 45 ++++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 14 deletions(-)
target/ppc/mmu-hash32.c +31 -14
index 3f4dee835e2d..55cf156a0b49 100644
@@ -345,6 +345,24 @@ static hwaddr ppc_hash32_pteg_search(PowerPCCPU *cpu, hwaddr pteg_off,
     return -1;
 }
 
+static void ppc_hash32_set_r(PowerPCCPU *cpu, hwaddr pte_offset, uint32_t pte1)
+{
+    target_ulong base = ppc_hash32_hpt_base(cpu);
+    hwaddr offset = pte_offset + 6;
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, ((pte1 >> 8) & 0xff) | 0x01);
+}
+
+static void ppc_hash32_set_c(PowerPCCPU *cpu, hwaddr pte_offset, uint64_t pte1)
+{
+    target_ulong base = ppc_hash32_hpt_base(cpu);
+    hwaddr offset = pte_offset + 7;
+
+    /* The HW performs a non-atomic byte update */
+    stb_phys(CPU(cpu)->as, base + offset, (pte1 & 0xff) | 0x80);
+}
+
 static hwaddr ppc_hash32_htab_lookup(PowerPCCPU *cpu,
                                      target_ulong sr, target_ulong eaddr,
                                      ppc_hash_pte32_t *pte)
@@ -403,7 +421,6 @@ int ppc_hash32_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
     hwaddr pte_offset;
     ppc_hash_pte32_t pte;
     int prot;
-    uint32_t new_pte1;
     const int need_prot[] = {PAGE_READ, PAGE_WRITE, PAGE_EXEC};
     hwaddr raddr;
 
@@ -519,20 +536,20 @@ int ppc_hash32_handle_mmu_fault(PowerPCCPU *cpu, vaddr eaddr, int rwx,
 
     /* 8. Update PTE referenced and changed bits if necessary */
 
-    new_pte1 = pte.pte1 | HPTE32_R_R; /* set referenced bit */
-    if (rwx == 1) {
-        new_pte1 |= HPTE32_R_C; /* set changed (dirty) bit */
-    } else {
-        /*
-         * Treat the page as read-only for now, so that a later write
-         * will pass through this function again to set the C bit
-         */
-        prot &= ~PAGE_WRITE;
-    }
-
-    if (new_pte1 != pte.pte1) {
-        ppc_hash32_store_hpte1(cpu, pte_offset, new_pte1);
+    if (!(pte.pte1 & HPTE32_R_R)) {
+        ppc_hash32_set_r(cpu, pte_offset, pte.pte1);
     }
+    if (!(pte.pte1 & HPTE32_R_C)) {
+        if (rwx == 1) {
+            ppc_hash32_set_c(cpu, pte_offset, pte.pte1);
+        } else {
+            /*
+             * Treat the page as read-only for now, so that a later write
+             * will pass through this function again to set the C bit
+             */
+            prot &= ~PAGE_WRITE;
+        }
+     }
 
     /* 9. Determine the real address from the PTE */
 
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#9 lore raw

PATCH 5/6 memory_ldst: Add atomic ops for PTE updates

In reply to Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

On some architectures, PTE updates for dirty and changed bits need to be performed atomically. This adds a couple of address_space_cmpxchg* helpers for that purpose.

---
 include/exec/memory_ldst.inc.h |  6 +++
 memory_ldst.inc.c              | 80 ++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/exec/memory_ldst.inc.h +6 -0
index 272c20f02eae..f3cfa7e9a622 100644
@@ -28,6 +28,12 @@ extern uint64_t glue(address_space_ldq, SUFFIX)(ARG1_DECL,
     hwaddr addr, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result);
 extern void glue(address_space_stw, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl, SUFFIX)(ARG1_DECL,
memory_ldst.inc.c +80 -0

86 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v1 5/6
#10 lore raw

Re: [Qemu-devel] [PATCH 5/6] memory_ldst: Add atomic ops for PTE updates

Quoted 8 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

On some architectures, PTE updates for dirty and changed bits need to be performed atomically. This adds a couple of address_space_cmpxchg* helpers for that purpose.

But I think this needs to go past Paolo for review as memory subsystem maintainer.

Quoted 113 lines
---
 include/exec/memory_ldst.inc.h |  6 +++
 memory_ldst.inc.c              | 80 ++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/exec/memory_ldst.inc.h +6 -0
index 272c20f02eae..f3cfa7e9a622 100644
@@ -28,6 +28,12 @@ extern uint64_t glue(address_space_ldq, SUFFIX)(ARG1_DECL,
     hwaddr addr, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result);
 extern void glue(address_space_stw, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl, SUFFIX)(ARG1_DECL,
memory_ldst.inc.c +80 -0
index acf865b900d7..1d58d2fea67d 100644
@@ -320,6 +320,86 @@ void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     RCU_READ_UNLOCK();
 }
 
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 4;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 4 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_ram_addr(mr) +
+                                                addr, 4, dirty_log_mask);
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+
+#ifdef CONFIG_ATOMIC64
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 8;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 8 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_ram_addr(mr) +
+                                                addr, 8, dirty_log_mask);
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+#endif /* CONFIG_ATOMIC64 */
+
 /* warning: addr must be aligned */
 static inline void glue(address_space_stl_internal, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs,
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#11 lore raw

Re: [Qemu-devel] [PATCH 5/6] memory_ldst: Add atomic ops for PTE updates

Quoted 8 lines · David Gibson
Quoted 7 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

On some architectures, PTE updates for dirty and changed bits need to be performed atomically. This adds a couple of address_space_cmpxchg*
helpers for that purpose.

Quoted 7 lines
Quoted 2 lines

But I think this needs to go past Paolo for review as memory subsystem maintainer.

This needs to go to rth and Peter. They were talking about a change in the abstraction to do these more cleanly... that said, it might be worthwhile merging the fixes first.

Cheers,
Ben.

Quoted 129 lines
Quoted 127 lines
---
 include/exec/memory_ldst.inc.h |  6 +++
 memory_ldst.inc.c              | 80
++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/exec/memory_ldst.inc.h +4 -0
index 272c20f02eae..f3cfa7e9a622 100644
@@ -28,6 +28,12 @@ extern uint64_t glue(address_space_ldq,
 SUFFIX)(ARG1_DECL,
     hwaddr addr, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult
 *result);
+extern uint32_t glue(address_space_cmpxchgl_notdirty,
 SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgq_notdirty,

SUFFIX)(ARG1_DECL,

include/exec/memory_ldst.inc.h +2 -0
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result);
 extern void glue(address_space_stw, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult

*result);
extern void glue(address_space_stl, SUFFIX)(ARG1_DECL,

memory_ldst.inc.c +29 -0
index acf865b900d7..1d58d2fea67d 100644
@@ -320,6 +320,86 @@ void glue(address_space_stl_notdirty,
 SUFFIX)(ARG1_DECL,
     RCU_READ_UNLOCK();
 }
 
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 4;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 4 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_

ram_addr(mr) +
+ addr, 4,
dirty_log_mask);

memory_ldst.inc.c +39 -0
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+
+#ifdef CONFIG_ATOMIC64
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 8;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 8 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_

ram_addr(mr) +
+ addr, 8,
dirty_log_mask);

memory_ldst.inc.c +10 -0
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+#endif /* CONFIG_ATOMIC64 */
+
 /* warning: addr must be aligned */
 static inline void glue(address_space_stl_internal,

SUFFIX)(ARG1_DECL,
hwaddr addr, uint32_t val, MemTxAttrs attrs,

#12 lore raw

Re: [Qemu-devel] [PATCH 5/6] memory_ldst: Add atomic ops for PTE updates

In reply to David Gibson
Quoted 7 lines · David Gibson
Quoted 6 lines · Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

On some architectures, PTE updates for dirty and changed bits need to be performed atomically. This adds a couple of address_space_cmpxchg* helpers for that purpose.

Quoted 6 lines
Quoted 2 lines

But I think this needs to go past Paolo for review as memory subsystem maintainer.

I will resend the last two patches in a new patchset and include a fix for Intel that Ben provided.

Thanks,

C.

Quoted 115 lines
Quoted 114 lines
---
 include/exec/memory_ldst.inc.h |  6 +++
 memory_ldst.inc.c              | 80 ++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/exec/memory_ldst.inc.h +6 -0
index 272c20f02eae..f3cfa7e9a622 100644
@@ -28,6 +28,12 @@ extern uint64_t glue(address_space_ldq, SUFFIX)(ARG1_DECL,
     hwaddr addr, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result);
+extern uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result);
 extern void glue(address_space_stw, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result);
 extern void glue(address_space_stl, SUFFIX)(ARG1_DECL,
memory_ldst.inc.c +80 -0
index acf865b900d7..1d58d2fea67d 100644
@@ -320,6 +320,86 @@ void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL,
     RCU_READ_UNLOCK();
 }
 
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgl_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint32_t old, uint32_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 4;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 4 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_ram_addr(mr) +
+                                                addr, 4, dirty_log_mask);
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+
+#ifdef CONFIG_ATOMIC64
+/* This is meant to be used for atomic PTE updates under MT-TCG */
+uint32_t glue(address_space_cmpxchgq_notdirty, SUFFIX)(ARG1_DECL,
+    hwaddr addr, uint64_t old, uint64_t new, MemTxAttrs attrs,
+    MemTxResult *result)
+{
+    uint8_t *ptr;
+    MemoryRegion *mr;
+    hwaddr l = 8;
+    hwaddr addr1;
+    MemTxResult r;
+    uint8_t dirty_log_mask;
+
+    /* Must test result */
+    assert(result);
+
+    RCU_READ_LOCK();
+    mr = TRANSLATE(addr, &addr1, &l, true, attrs);
+    if (l < 8 || !memory_access_is_direct(mr, true)) {
+        r = MEMTX_ERROR;
+    } else {
+        uint32_t orig = old;
+
+        ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
+        old = atomic_cmpxchg(ptr, orig, new);
+
+        if (old == orig) {
+            dirty_log_mask = memory_region_get_dirty_log_mask(mr);
+            dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE);
+            cpu_physical_memory_set_dirty_range(memory_region_get_ram_addr(mr) +
+                                                addr, 8, dirty_log_mask);
+        }
+        r = MEMTX_OK;
+    }
+    *result = r;
+    RCU_READ_UNLOCK();
+
+    return old;
+}
+#endif /* CONFIG_ATOMIC64 */
+
 /* warning: addr must be aligned */
 static inline void glue(address_space_stl_internal, SUFFIX)(ARG1_DECL,
     hwaddr addr, uint32_t val, MemTxAttrs attrs,
#13 lore raw

PATCH 6/6 ppc: Fix radix RC updates

In reply to Cédric Le Goater

From: Benjamin Herrenschmidt <[email protected]>

They should be atomic for MTTCG. Note: a real POWER9 core doesn't actually implement atomic PTE updates, it always fault for SW to handle it. Only the nest MMU (used by some accelerator devices and GPUs) implements those HW updates.

However, the architecture does allow the core to do it, and doing so in TCG is faster than letting the guest do it.

---
 target/ppc/cpu.h         |  1 +
 target/ppc/mmu-radix64.c | 70 +++++++++++++++++++++++++++++++++-------
 2 files changed, 60 insertions(+), 11 deletions(-)
target/ppc/cpu.h +1 -0
index a4420636dde0..5b4249ebdcbf 100644
@@ -503,6 +503,7 @@ typedef struct ppc_v3_pate_t {
 #define DSISR_AMR                0x00200000
 /* Unsupported Radix Tree Configuration */
 #define DSISR_R_BADCONFIG        0x00080000
+#define DSISR_ATOMIC_RC          0x00040000
 
 /* SRR1 error code fields */
 
target/ppc/mmu-radix64.c +59 -11

107 lines not on this page · raw message

Signature · 1 line
2.20.1