Skip to contentSkip to conversation

s390x/tcg: mem_helper: Fault-safe handling

PATCH Superseded David Hildenbrand 2 participants 68 messages Sep 6, 2019 – Sep 16, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover s390x/tcg: mem_helper: Fault-safe handling

This is the successor of

    "[PATCH v1 0/4] s390x/tcg: MOVE (MVC): Fault-safe handling"

----

This series fixes a bunch of issues related to some mem helpers and makes sure that they are fault-safe, meaning no system state is modified in case a fault is triggered.

I can spot tons of other issues with other mem helpers that will have to be fixed later. Also, fault-safe handling for some instructions (especially TR) might be harder to implement (you don't know what will actually be accessed upfront - we might need a buffer and go over inputs twice). Focusing on the MOVE instructions for now.

----

Newer versions of glibc use memcpy() in memmove() for forward moves. The implementation makese use of MVC. The TCG implementation of MVC is currently not able to handle faults reliably when crossing pages. MVC can cross with 256 bytes at most two pages.

In case we get a fault on the second page, we already moved data. When continuing after the fault we might try to move already overwritten data, which is very bad in case we have overlapping data on a forward move.

Triggered for now only by rpmbuild (crashes when checking the spec file) and rpm (database corruptions). This fixes installing Fedora rawhide (31) under TCG.

This was horrible to debug as it barely triggers and we fail at completely different places.

Cc Stefano Brivio <[email protected]>
Cc Florian Weimer <[email protected]>

v1 -> v2:
- Include many fixes
- Fix more instructions
- Use the new probe_access() function
- Include "tests/tcg: target/s390x: Test MVO"

David Hildenbrand (28):
s390x/tcg: Reset exception_index to -1 instead of 0 s390x/tcg: MVCL: Zero out unused bits of address s390x/tcg: MVCL: Detect destructive overlaps s390x/tcg: MVCL: Process max 2k bytes at a time s390x/tcg: MVC: Increment the length once s390x/tcg: MVC: Use is_destructive_overlap() s390x/tcg: MVPG: Check for specification exceptions s390x/tcg: MVPG: Properly wrap the addresses s390x/tcg: MVCLU/MVCLE: Process max 4k bytes at a time s390x/tcg: MVCS/MVCP: Check for special operation exceptions s390x/tcg: MVCS/MVCP: Properly wrap the length s390x/tcg: MVST: Check for specification exceptions s390x/tcg: MVST: Fix storing back the addresses to registers s390x/tcg: Always use MMU_USER_IDX for CONFIG_USER_ONLY s390x/tcg: Fault-safe memset
s390x/tcg: Fault-safe memmove
s390x/tcg: MVCS/MVCP: Use access_memmove_idx() s390x/tcg: MVC: Fault-safe handling on destructive overlaps s390x/tcg: MVCLU: Fault-safe handling
s390x/tcg: OC: Fault-safe handling
s390x/tcg: XC: Fault-safe handling
s390x/tcg: NC: Fault-safe handling
s390x/tcg: MVCIN: Fault-safe handling
s390x/tcg: MVN: Fault-safe handling
s390x/tcg: MVZ: Fault-safe handling
s390x/tcg: MVST: Fault-safe handling
s390x/tcg: MVO: Fault-safe handling
tests/tcg: target/s390x: Test MVO

 target/s390x/cpu.h              |   4 +
 target/s390x/helper.h           |   2 +-
 target/s390x/insn-data.def      |   2 +-
 target/s390x/mem_helper.c       | 672 ++++++++++++++++++++++----------
 target/s390x/translate.c        |  12 +-
 tests/tcg/s390x/Makefile.target |   1 +
 tests/tcg/s390x/mvo.c           |  25 ++
 7 files changed, 507 insertions(+), 211 deletions(-)
 create mode 100644 tests/tcg/s390x/mvo.c
Signature · 1 line
2.21.0
#1 lore raw

PATCH v2 1/28 s390x/tcg: Reset exception_index to -1 instead of 0

We use the marker "-1" for "no exception". s390_cpu_do_interrupt() might get confused by that.

---
 target/s390x/mem_helper.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
target/s390x/mem_helper.c +3 -3
index 29fcce426e..39ee9b3175 100644
@@ -1747,7 +1747,7 @@ uint32_t HELPER(tprot)(CPUS390XState *env, uint64_t a1, uint64_t a2)
 
     if (env->int_pgm_code == PGM_PROTECTION) {
         /* retry if reading is possible */
-        cs->exception_index = 0;
+        cs->exception_index = -1;
         if (!s390_cpu_virt_mem_check_read(cpu, a1, 0, 1)) {
             /* Fetching permitted; storing not permitted */
             return 1;
@@ -1757,7 +1757,7 @@ uint32_t HELPER(tprot)(CPUS390XState *env, uint64_t a1, uint64_t a2)
     switch (env->int_pgm_code) {
     case PGM_PROTECTION:
         /* Fetching not permitted; storing not permitted */
-        cs->exception_index = 0;
+        cs->exception_index = -1;
         return 2;
     case PGM_ADDRESSING:
     case PGM_TRANS_SPEC:
@@ -1767,7 +1767,7 @@ uint32_t HELPER(tprot)(CPUS390XState *env, uint64_t a1, uint64_t a2)
     }
 
     /* Translation not available */
-    cs->exception_index = 0;
+    cs->exception_index = -1;
     return 3;
 }
 
Signature · 1 line
2.21.0
#3 lore raw

PATCH v2 2/28 s390x/tcg: MVCL: Zero out unused bits of address

In reply to David Hildenbrand

We have to zero out unused bits in 24 and 31-bit addressing mode. Provide a new helper.

---
 target/s390x/mem_helper.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
target/s390x/mem_helper.c +22 -2
index 39ee9b3175..3152bdafe2 100644
@@ -469,6 +469,26 @@ static inline uint64_t get_address(CPUS390XState *env, int reg)
     return wrap_address(env, env->regs[reg]);
 }
 
+/*
+ * Store the address to the given register, zeroing out unused leftmost
+ * bits in bit positions 32-63 (24-bit and 31-bit mode only).
+ */
+static inline void set_address_zero(CPUS390XState *env, int reg,
+                                    uint64_t address)
+{
+    if (env->psw.mask & PSW_MASK_64) {
+        env->regs[reg] = address;
+    } else {
+        if (!(env->psw.mask & PSW_MASK_32)) {
+            address &= 0x00ffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        } else {
+            address &= 0x7fffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        }
+    }
+}
+
 static inline void set_address(CPUS390XState *env, int reg, uint64_t address)
 {
     if (env->psw.mask & PSW_MASK_64) {
@@ -772,8 +792,8 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
 
     env->regs[r1 + 1] = deposit64(env->regs[r1 + 1], 0, 24, destlen);
     env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
-    set_address(env, r1, dest);
-    set_address(env, r2, src);
+    set_address_zero(env, r1, dest);
+    set_address_zero(env, r2, src);
 
     return cc;
 }
Signature · 1 line
2.21.0
Reviewed-by on v2 2/28
#4 lore raw

Re: [Qemu-devel] [PATCH v2 02/28] s390x/tcg: MVCL: Zero out unused bits of address

Quoted 7 lines · David Hildenbrand

We have to zero out unused bits in 24 and 31-bit addressing mode. Provide a new helper.

---
 target/s390x/mem_helper.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Quoted 15 lines
target/s390x/mem_helper.c +7 -0
index 39ee9b3175..3152bdafe2 100644
@@ -469,6 +469,26 @@ static inline uint64_t get_address(CPUS390XState *env, int reg)
     return wrap_address(env, env->regs[reg]);
 }
 
+/*
+ * Store the address to the given register, zeroing out unused leftmost
+ * bits in bit positions 32-63 (24-bit and 31-bit mode only).
+ */
+static inline void set_address_zero(CPUS390XState *env, int reg,
+                                    uint64_t address)
+{

target/s390x/mem_helper.c static inline uint64_t get_address(CPUS390XState *env, int reg) · line 488

Quoted 10 lines
target/s390x/mem_helper.c +10 -0
+    if (env->psw.mask & PSW_MASK_64) {
+        env->regs[reg] = address;
+    } else {
+        if (!(env->psw.mask & PSW_MASK_32)) {
+            address &= 0x00ffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        } else {
+            address &= 0x7fffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        }

You could perhaps sink the deposit and store line into the outer else.

r~

#5 lore raw

Re: [Qemu-devel] [PATCH v2 02/28] s390x/tcg: MVCL: Zero out unused bits of address

Quoted 31 lines · Richard Henderson
Quoted 7 lines · David Hildenbrand

We have to zero out unused bits in 24 and 31-bit addressing mode. Provide a new helper.

---
 target/s390x/mem_helper.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Quoted 19 lines
target/s390x/mem_helper.c +11 -0
index 39ee9b3175..3152bdafe2 100644
@@ -469,6 +469,26 @@ static inline uint64_t get_address(CPUS390XState *env, int reg)
     return wrap_address(env, env->regs[reg]);
 }
 
+/*
+ * Store the address to the given register, zeroing out unused leftmost
+ * bits in bit positions 32-63 (24-bit and 31-bit mode only).
+ */
+static inline void set_address_zero(CPUS390XState *env, int reg,
+                                    uint64_t address)
+{
+    if (env->psw.mask & PSW_MASK_64) {
+        env->regs[reg] = address;
+    } else {
+        if (!(env->psw.mask & PSW_MASK_32)) {
Quoted 9 lines

target/s390x/mem_helper.c static inline uint64_t get_address(CPUS390XState *env, int reg) · line 488

Quoted 6 lines
target/s390x/mem_helper.c +6 -0
+            address &= 0x00ffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        } else {
+            address &= 0x7fffffff;
+            env->regs[reg] = deposit64(env->regs[reg], 0, 32, address);
+        }

You could perhaps sink the deposit and store line into the outer else.

Thanks, will do!

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw

PATCH v2 3/28 s390x/tcg: MVCL: Detect destructive overlaps

In reply to David Hildenbrand

We'll have to zero-out unused bit positions, so amke sure to write the addresses back.

---
 target/s390x/mem_helper.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
target/s390x/mem_helper.c +18 -1
index 3152bdafe2..2361ed6d54 100644
@@ -52,6 +52,19 @@ static inline bool psw_key_valid(CPUS390XState *env, uint8_t psw_key)
     return true;
 }
 
+static bool is_destructive_overlap(CPUS390XState *env, uint64_t dest,
+                                   uint64_t src, uint32_t len)
+{
+    if (!len || src == dest) {
+        return false;
+    }
+    /* Take care of wrapping at the end of address space. */
+    if (unlikely(wrap_address(env, src + len - 1) < src)) {
+        return dest > src || dest <= wrap_address(env, src + len - 1);
+    }
+    return dest > src && dest <= src + len - 1;
+}
+
 /* Reduce the length so that addr + len doesn't cross a page boundary.  */
 static inline uint32_t adj_len_to_page(uint32_t len, uint64_t addr)
 {
@@ -788,7 +801,11 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
     uint8_t pad = env->regs[r2 + 1] >> 24;
     uint32_t cc;
 
-    cc = do_mvcl(env, &dest, &destlen, &src, &srclen, pad, 1, ra);
+    if (is_destructive_overlap(env, dest, src, MIN(srclen, destlen))) {
+        cc = 3;
+    } else {
+        cc = do_mvcl(env, &dest, &destlen, &src, &srclen, pad, 1, ra);
+    }
 
     env->regs[r1 + 1] = deposit64(env->regs[r1 + 1], 0, 24, destlen);
     env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
Signature · 1 line
2.21.0
#8 lore raw

PATCH v2 4/28 s390x/tcg: MVCL: Process max 2k bytes at a time

In reply to David Hildenbrand

Process max 2k bytes at a time, writing back registers between the accesses. The instruction is interruptible.
"For operands longer than 2K bytes, access exceptions are not recognized for locations more than 2K bytes beyond the current location being processed."

MVCL handling is quite different than MVCLE/MVCLU handling, so split up the handlers.

We'll deal with fast_memmove() and fast_memset() not probing reads/writes properly later. Also, we'll defer interrupt handling, as that will require more thought, add a TODO for that.

---
 target/s390x/mem_helper.c | 44 +++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)
target/s390x/mem_helper.c +38 -6
index 2361ed6d54..2e22c183bd 100644
@@ -799,19 +799,51 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
     uint64_t srclen = env->regs[r2 + 1] & 0xffffff;
     uint64_t src = get_address(env, r2);
     uint8_t pad = env->regs[r2 + 1] >> 24;
-    uint32_t cc;
+    uint32_t cc, cur_len;
 
     if (is_destructive_overlap(env, dest, src, MIN(srclen, destlen))) {
         cc = 3;
+    } else if (srclen == destlen) {
+        cc = 0;
+    } else if (destlen < srclen) {
+        cc = 1;
     } else {
-        cc = do_mvcl(env, &dest, &destlen, &src, &srclen, pad, 1, ra);
+        cc = 2;
     }
 
-    env->regs[r1 + 1] = deposit64(env->regs[r1 + 1], 0, 24, destlen);
-    env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
-    set_address_zero(env, r1, dest);
-    set_address_zero(env, r2, src);
+    /* We might have to zero-out some bits even if there was no action. */
+    if (unlikely(!destlen || cc == 3)) {
+        set_address_zero(env, r2, src);
+        set_address_zero(env, r1, dest);
+        return cc;
+    } else if (!srclen) {
+        set_address_zero(env, r2, src);
+    }
 
+    /*
+     * Only perform one type of type of operation (move/pad) in one step.
+     * Process up to 2k bytes.
+     */
+    while (destlen) {
+        cur_len = MIN(destlen, 2048);
+        if (!srclen) {
+            fast_memset(env, dest, pad, cur_len, ra);
+        } else {
+            cur_len = MIN(cur_len, srclen);
+
+            fast_memmove(env, dest, src, cur_len, ra);
+            src = wrap_address(env, src + cur_len);
+            srclen -= cur_len;
+            env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
+            set_address_zero(env, r2, src);
+        }
+        dest = wrap_address(env, dest + cur_len);
+        destlen -= cur_len;
+        env->regs[r1 + 1] = deposit64(env->regs[r1 + 1], 0, 24, destlen);
+        set_address_zero(env, r1, dest);
+
+        /* TODO: Deliver interrupts. */
+    }
     return cc;
 }
 
Signature · 1 line
2.21.0
#9 lore raw

Re: [Qemu-devel] [PATCH v2 04/28] s390x/tcg: MVCL: Process max 2k bytes at a time

Quoted 49 lines · David Hildenbrand

Process max 2k bytes at a time, writing back registers between the accesses. The instruction is interruptible.
"For operands longer than 2K bytes, access exceptions are not recognized for locations more than 2K bytes beyond the current location being processed."

MVCL handling is quite different than MVCLE/MVCLU handling, so split up the handlers.

We'll deal with fast_memmove() and fast_memset() not probing reads/writes properly later. Also, we'll defer interrupt handling, as that will require more thought, add a TODO for that.

---
 target/s390x/mem_helper.c | 44 +++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)
target/s390x/mem_helper.c +11 -6
index 2361ed6d54..2e22c183bd 100644
@@ -799,19 +799,51 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
     uint64_t srclen = env->regs[r2 + 1] & 0xffffff;
     uint64_t src = get_address(env, r2);
     uint8_t pad = env->regs[r2 + 1] >> 24;
-    uint32_t cc;
+    uint32_t cc, cur_len;
 
     if (is_destructive_overlap(env, dest, src, MIN(srclen, destlen))) {
         cc = 3;
+    } else if (srclen == destlen) {
+        cc = 0;
+    } else if (destlen < srclen) {
+        cc = 1;
     } else {
-        cc = do_mvcl(env, &dest, &destlen, &src, &srclen, pad, 1, ra);
+        cc = 2;
     }
 
-    env->regs[r1 + 1] = deposit64(env->regs[r1 + 1], 0, 24, destlen);
-    env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
-    set_address_zero(env, r1, dest);
-    set_address_zero(env, r2, src);
+    /* We might have to zero-out some bits even if there was no action. */
+    if (unlikely(!destlen || cc == 3)) {
+        set_address_zero(env, r2, src);
+        set_address_zero(env, r1, dest);
+        return cc;

target/s390x/mem_helper.c uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2) · line 828

Quoted 10 lines
target/s390x/mem_helper.c +9 -0
+    } else if (!srclen) {
+        set_address_zero(env, r2, src);
+    }
 
+    /*
+     * Only perform one type of type of operation (move/pad) in one step.
+     * Process up to 2k bytes.
+     */
+    while (destlen) {
+        cur_len = MIN(destlen, 2048);

The language in the PoP is horribly written, and thus confusing. I can't believe it really means what it appears to say, about access exceptions not being recognized.

The code within Hercules breaks the action at every 2k address boundary -- for both src and dest. That's the only way that actually makes sense to me, as otherwise we end up allowing userspace to read/write into a page without permission. Which is a security hole.

r~

#10 lore raw

Re: [Qemu-devel] [PATCH v2 04/28] s390x/tcg: MVCL: Process max 2k bytes at a time

Quoted 4 lines · Richard Henderson

The code within Hercules breaks the action at every 2k address boundary -- for both src and dest. That's the only way that actually makes sense to me, as otherwise we end up allowing userspace to read/write into a page without permission. Which is a security hole.

Also, doesn't "2k" come from the old esa/360 page size?

Which means that we could break at 4k pages instead of 2k now and the program wouldn't really be able to tell the difference.

r~

#11 lore raw

Re: [Qemu-devel] [PATCH v2 04/28] s390x/tcg: MVCL: Process max 2k bytes at a time

Quoted 7 lines · Richard Henderson
Quoted 4 lines · Richard Henderson

The code within Hercules breaks the action at every 2k address boundary -- for both src and dest. That's the only way that actually makes sense to me, as otherwise we end up allowing userspace to read/write into a page without permission. Which is a security hole.

Also, doesn't "2k" come from the old esa/360 page size?

I have no idea, I was very confused with that.

Quoted 3 lines

Which means that we could break at 4k pages instead of 2k now and the program wouldn't really be able to tell the difference.

What I had in a previous iteration was to simply process until the end of the page(s), to not cross pages (there is one special case with 2k vs. 4k when crossing pages when wrapping and running into a low-address-protection). So essentially what you suggest. I can add that.

Signature · 3 lines
Thanks,

David / dhildenb
#12 lore raw

PATCH v2 5/28 s390x/tcg: MVC: Increment the length once

In reply to David Hildenbrand

Let's increment the length once.

While at it, cleanup the comment. The memset() example is given as a programming note in the PoP, so drop the description.

---
 target/s390x/mem_helper.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
target/s390x/mem_helper.c +12 -8
index 2e22c183bd..2bc2cd09c1 100644
@@ -320,16 +320,20 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
     HELPER_LOG("%s l %d dest %" PRIx64 " src %" PRIx64 "\n",
                __func__, l, dest, src);
 
-    /* mvc and memmove do not behave the same when areas overlap! */
-    /* mvc with source pointing to the byte after the destination is the
-       same as memset with the first source byte */
+    /* MVC always copies one more byte than specified - maximum is 256 */
+    l++;
+
+    /*
+     * "When the operands overlap, the result is obtained as if the operands
+     * were processed one byte at a time". Only non-destructive overlaps
+     * behave like memmove().
+     */
     if (dest == src + 1) {
-        fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l + 1, ra);
-    } else if (dest < src || src + l < dest) {
-        fast_memmove(env, dest, src, l + 1, ra);
+        fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
+    } else if (dest < src || src + l <= dest) {
+        fast_memmove(env, dest, src, l, ra);
     } else {
-        /* slow version with byte accesses which always work */
-        for (i = 0; i <= l; i++) {
+        for (i = 0; i < l; i++) {
             uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
             cpu_stb_data_ra(env, dest + i, x, ra);
         }
Signature · 1 line
2.21.0
#13 lore raw

PATCH v2 6/28 s390x/tcg: MVC: Use is_destructive_overlap()

In reply to David Hildenbrand

Let's use the new helper, that also detects destructive overlaps when wrapping.

---
 target/s390x/mem_helper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
target/s390x/mem_helper.c +1 -1
index 2bc2cd09c1..3c23c403cd 100644
@@ -330,7 +330,7 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
      */
     if (dest == src + 1) {
         fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
-    } else if (dest < src || src + l <= dest) {
+    } else if (!is_destructive_overlap(env, dest, src, l)) {
         fast_memmove(env, dest, src, l, ra);
     } else {
         for (i = 0; i < l; i++) {
Signature · 1 line
2.21.0
Reviewed-by on v2 6/28
#14 lore raw

Re: [Qemu-devel] [PATCH v2 06/28] s390x/tcg: MVC: Use is_destructive_overlap()

target/s390x/mem_helper.c static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest, · line 336

Quoted 7 lines · David Hildenbrand
     if (dest == src + 1) {
         fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
target/s390x/mem_helper.c +1 -1
-    } else if (dest < src || src + l <= dest) {
+    } else if (!is_destructive_overlap(env, dest, src, l)) {
         fast_memmove(env, dest, src, l, ra);
     } else {
         for (i = 0; i < l; i++) {

I suppose, though last time I checked fast_memmove didn't support wrapping.

r~

#15 lore raw

Re: [Qemu-devel] [PATCH v2 06/28] s390x/tcg: MVC: Use is_destructive_overlap()

Quoted 10 lines · Richard Henderson

target/s390x/mem_helper.c static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest, · line 336

Quoted 7 lines · David Hildenbrand
     if (dest == src + 1) {
         fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
target/s390x/mem_helper.c +1 -1
-    } else if (dest < src || src + l <= dest) {
+    } else if (!is_destructive_overlap(env, dest, src, l)) {
         fast_memmove(env, dest, src, l, ra);
     } else {
         for (i = 0; i < l; i++) {

I suppose, though last time I checked fast_memmove didn't support wrapping.

Yes, that's fixed by access_prepare(), access_memmove(). Wrapping is/was broken in most mem handlers ...

Quoted 5 lines

r~

Signature · 3 lines
Thanks,

David / dhildenb
#16 lore raw

PATCH v2 7/28 s390x/tcg: MVPG: Check for specification exceptions

In reply to David Hildenbrand

Perform the checks documented in the PoP.

---
 target/s390x/mem_helper.c | 7 +++++++
 1 file changed, 7 insertions(+)
target/s390x/mem_helper.c +7 -0
index 3c23c403cd..a763482ae0 100644
@@ -673,6 +673,13 @@ uint64_t HELPER(clst)(CPUS390XState *env, uint64_t c, uint64_t s1, uint64_t s2)
 /* move page */
 uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2)
 {
+    const bool f = extract64(r0, 11, 1);
+    const bool s = extract64(r0, 10, 1);
+
+    if ((f && s) || extract64(r0, 12, 4)) {
+        s390_program_interrupt(env, PGM_SPECIFICATION, ILEN_AUTO, GETPC());
+    }
+
     /* ??? missing r0 handling, which includes access keys, but more
        importantly optional suppression of the exception!  */
     fast_memmove(env, r1, r2, TARGET_PAGE_SIZE, GETPC());
Signature · 1 line
2.21.0
#18 lore raw

PATCH v2 8/28 s390x/tcg: MVPG: Properly wrap the addresses

In reply to David Hildenbrand

We have to mask of any unused bits. While at it, document what exactly is missing.

---
 target/s390x/mem_helper.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
target/s390x/mem_helper.c +9 -2
index a763482ae0..947a4277f0 100644
@@ -680,8 +680,15 @@ uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2)
         s390_program_interrupt(env, PGM_SPECIFICATION, ILEN_AUTO, GETPC());
     }
 
-    /* ??? missing r0 handling, which includes access keys, but more
-       importantly optional suppression of the exception!  */
+    r1 = wrap_address(env, r1 & TARGET_PAGE_MASK);
+    r2 = wrap_address(env, r2 & TARGET_PAGE_MASK);
+
+    /*
+     * TODO:
+     * - Access key handling
+     * - CC-option with surpression of page-translation exceptions
+     * - Store r1/r2 register identifiers at real location 162
+     */
     fast_memmove(env, r1, r2, TARGET_PAGE_SIZE, GETPC());
     return 0; /* data moved */
 }
Signature · 1 line
2.21.0
#20 lore raw

PATCH v2 9/28 s390x/tcg: MVCLU/MVCLE: Process max 4k bytes at a time

In reply to David Hildenbrand

... and indicate cc=3 in case there is work remaining. Keep unicode padding simple.

While reworking, properly wrap the addresses.

---
 target/s390x/mem_helper.c | 54 ++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 23 deletions(-)
target/s390x/mem_helper.c +31 -23
index 947a4277f0..6d8ebd18fa 100644
@@ -769,8 +769,8 @@ static inline uint32_t do_mvcl(CPUS390XState *env,
                                uint64_t *src, uint64_t *srclen,
                                uint16_t pad, int wordsize, uintptr_t ra)
 {
-    uint64_t len = MIN(*srclen, *destlen);
-    uint32_t cc;
+    int len = MIN(*destlen, TARGET_PAGE_SIZE);
+    int i, cc;
 
     if (*destlen == *srclen) {
         cc = 0;
@@ -780,32 +780,40 @@ static inline uint32_t do_mvcl(CPUS390XState *env,
         cc = 2;
     }
 
-    /* Copy the src array */
-    fast_memmove(env, *dest, *src, len, ra);
-    *src += len;
-    *srclen -= len;
-    *dest += len;
-    *destlen -= len;
+    if (!*destlen) {
+        return cc;
+    }
 
-    /* Pad the remaining area */
-    if (wordsize == 1) {
-        fast_memset(env, *dest, pad, *destlen, ra);
-        *dest += *destlen;
-        *destlen = 0;
+    /*
+     * Only perform one type of type of operation (move/pad) at a time.
+     * Process up to 4k bytes.
+     */
+    if (*srclen) {
+        /* Copy the src array */
+        len = MIN(len, *srclen);
+        *destlen -= len;
+        *srclen -= len;
+        fast_memmove(env, *dest, *src, len, ra);
+        *src = wrap_address(env, *src + len);
+        *dest = wrap_address(env, *dest + len);
+    } else if (wordsize == 1) {
+        /* Pad the remaining area */
+        *destlen -= len;
+        fast_memset(env, *dest, pad, len, ra);
+        *dest = wrap_address(env, *dest + len);
     } else {
-        /* If remaining length is odd, pad with odd byte first.  */
-        if (*destlen & 1) {
-            cpu_stb_data_ra(env, *dest, pad & 0xff, ra);
-            *dest += 1;
-            *destlen -= 1;
-        }
-        /* The remaining length is even, pad using words.  */
-        for (; *destlen; *dest += 2, *destlen -= 2) {
-            cpu_stw_data_ra(env, *dest, pad, ra);
+        /* The remaining length selects the padding byte. */
+        for (i = 0; i < len; (*destlen)--, i++) {
+            if (*destlen & 1) {
+                cpu_stb_data_ra(env, *dest, pad, ra);
+            } else {
+                cpu_stb_data_ra(env, *dest, pad >> 8, ra);
+            }
+            *dest = wrap_address(env, *dest + 1);
         }
     }
 
-    return cc;
+    return *destlen ? 3 : cc;
 }
 
 /* move long */
Signature · 1 line
2.21.0
#21 lore raw

Re: [Qemu-devel] [PATCH v2 09/28] s390x/tcg: MVCLU/MVCLE: Process max 4k bytes at a time

target/s390x/mem_helper.c static inline uint32_t do_mvcl(CPUS390XState *env, · line 803

Quoted 13 lines · David Hildenbrand
target/s390x/mem_helper.c +13 -0
+    if (*srclen) {
+        /* Copy the src array */
+        len = MIN(len, *srclen);
+        *destlen -= len;
+        *srclen -= len;
+        fast_memmove(env, *dest, *src, len, ra);
+        *src = wrap_address(env, *src + len);
+        *dest = wrap_address(env, *dest + len);
+    } else if (wordsize == 1) {
+        /* Pad the remaining area */
+        *destlen -= len;
+        fast_memset(env, *dest, pad, len, ra);
+        *dest = wrap_address(env, *dest + len);

Again, I'm not sure fast_memmove actually handles wrap, yet. Would it be easier to split at page boundaries rather than a fixed 4k length?

r~

#22 lore raw

Re: [Qemu-devel] [PATCH v2 09/28] s390x/tcg: MVCLU/MVCLE: Process max 4k bytes at a time

Quoted 15 lines · Richard Henderson

target/s390x/mem_helper.c static inline uint32_t do_mvcl(CPUS390XState *env, · line 803

Quoted 13 lines · David Hildenbrand
target/s390x/mem_helper.c +13 -0
+    if (*srclen) {
+        /* Copy the src array */
+        len = MIN(len, *srclen);
+        *destlen -= len;
+        *srclen -= len;
+        fast_memmove(env, *dest, *src, len, ra);
+        *src = wrap_address(env, *src + len);
+        *dest = wrap_address(env, *dest + len);
+    } else if (wordsize == 1) {
+        /* Pad the remaining area */
+        *destlen -= len;
+        fast_memset(env, *dest, pad, len, ra);
+        *dest = wrap_address(env, *dest + len);
Quoted 1 line

Again, I'm not sure fast_memmove actually handles wrap, yet.

Similarly, fixed by access_prepare(), access_memmove().

Quoted 2 lines

Would it be easier to split at page boundaries rather than a fixed 4k length?

Also had that already, can do.

Signature · 3 lines
Thanks,

David / dhildenb
#23 lore raw

PATCH v2 10/28 s390x/tcg: MVCS/MVCP: Check for special operation exceptions

In reply to David Hildenbrand

Let's perform the documented checks.

---
 target/s390x/mem_helper.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
target/s390x/mem_helper.c +12 -0
index 6d8ebd18fa..041d01d63d 100644
@@ -1961,12 +1961,18 @@ uint32_t HELPER(rrbe)(CPUS390XState *env, uint64_t r2)
 
 uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
 {
+    const uint8_t psw_as = (env->psw.mask & PSW_MASK_ASC) >> PSW_SHIFT_ASC;
     uintptr_t ra = GETPC();
     int cc = 0, i;
 
     HELPER_LOG("%s: %16" PRIx64 " %16" PRIx64 " %16" PRIx64 "\n",
                __func__, l, a1, a2);
 
+    if (!(env->psw.mask & PSW_MASK_DAT) || !(env->cregs[0] & CR0_SECONDARY) ||
+        psw_as == AS_HOME || psw_as == AS_ACCREG) {
+        s390_program_interrupt(env, PGM_SPECIAL_OP, ILEN_AUTO, ra);
+    }
+
     if (l > 256) {
         /* max 256 */
         l = 256;
@@ -1984,12 +1990,18 @@ uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
 
 uint32_t HELPER(mvcp)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
 {
+    const uint8_t psw_as = (env->psw.mask & PSW_MASK_ASC) >> PSW_SHIFT_ASC;
     uintptr_t ra = GETPC();
     int cc = 0, i;
 
     HELPER_LOG("%s: %16" PRIx64 " %16" PRIx64 " %16" PRIx64 "\n",
                __func__, l, a1, a2);
 
+    if (!(env->psw.mask & PSW_MASK_DAT) || !(env->cregs[0] & CR0_SECONDARY) ||
+        psw_as == AS_HOME || psw_as == AS_ACCREG) {
+        s390_program_interrupt(env, PGM_SPECIAL_OP, ILEN_AUTO, ra);
+    }
+
     if (l > 256) {
         /* max 256 */
         l = 256;
Signature · 1 line
2.21.0
#25 lore raw

PATCH v2 11/28 s390x/tcg: MVCS/MVCP: Properly wrap the length

In reply to David Hildenbrand

... and don't perform any move in case the length is zero.

---
 target/s390x/mem_helper.c | 6 ++++++
 1 file changed, 6 insertions(+)
target/s390x/mem_helper.c +6 -0
index 041d01d63d..de5e69b500 100644
@@ -1973,10 +1973,13 @@ uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
         s390_program_interrupt(env, PGM_SPECIAL_OP, ILEN_AUTO, ra);
     }
 
+    l = wrap_length(env, l);
     if (l > 256) {
         /* max 256 */
         l = 256;
         cc = 3;
+    } else if (!l) {
+        return cc;
     }
 
     /* XXX replace w/ memcpy */
@@ -2002,10 +2005,13 @@ uint32_t HELPER(mvcp)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
         s390_program_interrupt(env, PGM_SPECIAL_OP, ILEN_AUTO, ra);
     }
 
+    l = wrap_length(env, l);
     if (l > 256) {
         /* max 256 */
         l = 256;
         cc = 3;
+    } else if (!l) {
+        return cc;
     }
 
     /* XXX replace w/ memcpy */
Signature · 1 line
2.21.0
#26 lore raw

Re: [Qemu-devel] [PATCH v2 11/28] s390x/tcg: MVCS/MVCP: Properly wrap the length

target/s390x/mem_helper.c uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2) · line 1983

Quoted 8 lines · David Hildenbrand
target/s390x/mem_helper.c +3 -0
+    l = wrap_length(env, l);
     if (l > 256) {
         /* max 256 */
         l = 256;
         cc = 3;
+    } else if (!l) {
+        return cc;
     }

Um, wrap_length only takes 31 bits.
These insns take 32 bits in 24/31-bit modes.

r~

#27 lore raw

Re: [Qemu-devel] [PATCH v2 11/28] s390x/tcg: MVCS/MVCP: Properly wrap the length

Quoted 10 lines · Richard Henderson

target/s390x/mem_helper.c uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2) · line 1983

Quoted 8 lines · David Hildenbrand
target/s390x/mem_helper.c +3 -0
+    l = wrap_length(env, l);
     if (l > 256) {
         /* max 256 */
         l = 256;
         cc = 3;
+    } else if (!l) {
+        return cc;
     }
Quoted 2 lines

Um, wrap_length only takes 31 bits.
These insns take 32 bits in 24/31-bit modes.

Nice observation! Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#28 lore raw

PATCH v2 12/28 s390x/tcg: MVST: Check for specification exceptions

In reply to David Hildenbrand

Bit position 32-55 of general register 0 must be zero.

---
 target/s390x/mem_helper.c | 3 +++
 1 file changed, 3 insertions(+)
target/s390x/mem_helper.c +3 -0
index de5e69b500..afcd452a00 100644
@@ -699,6 +699,9 @@ uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
     uintptr_t ra = GETPC();
     uint32_t len;
 
+    if (c & 0xffffff00ull) {
+        s390_program_interrupt(env, PGM_SPECIFICATION, ILEN_AUTO, ra);
+    }
     c = c & 0xff;
     d = wrap_address(env, d);
     s = wrap_address(env, s);
Signature · 1 line
2.21.0
#30 lore raw

PATCH v2 13/28 s390x/tcg: MVST: Fix storing back the addresses to registers

In reply to David Hildenbrand

24 and 31-bit address space handling is wrong when it comes to storing back the addresses to the register.

---
 target/s390x/helper.h      |  2 +-
 target/s390x/insn-data.def |  2 +-
 target/s390x/mem_helper.c  | 20 ++++++++------------
 target/s390x/translate.c   |  8 ++++++--
 4 files changed, 16 insertions(+), 16 deletions(-)
target/s390x/helper.h +1 -1
index e9aff83b05..b32fce22ca 100644
@@ -20,7 +20,7 @@ DEF_HELPER_FLAGS_4(mvn, TCG_CALL_NO_WG, void, env, i32, i64, i64)
 DEF_HELPER_FLAGS_4(mvo, TCG_CALL_NO_WG, void, env, i32, i64, i64)
 DEF_HELPER_FLAGS_4(mvpg, TCG_CALL_NO_WG, i32, env, i64, i64, i64)
 DEF_HELPER_FLAGS_4(mvz, TCG_CALL_NO_WG, void, env, i32, i64, i64)
-DEF_HELPER_4(mvst, i64, env, i64, i64, i64)
+DEF_HELPER_4(mvst, i32, env, i64, i32, i32)
 DEF_HELPER_4(ex, void, env, i32, i64, i64)
 DEF_HELPER_FLAGS_4(stam, TCG_CALL_NO_WG, void, env, i32, i64, i32)
 DEF_HELPER_FLAGS_4(lam, TCG_CALL_NO_WG, void, env, i32, i64, i32)
target/s390x/insn-data.def +1 -1
index f421184fcd..449eee1662 100644
@@ -637,7 +637,7 @@
 /* MOVE PAGE */
     C(0xb254, MVPG,    RRE,   Z,   r1_o, r2_o, 0, 0, mvpg, 0)
 /* MOVE STRING */
-    C(0xb255, MVST,    RRE,   Z,   r1_o, r2_o, 0, 0, mvst, 0)
+    C(0xb255, MVST,    RRE,   Z,   0, 0, 0, 0, mvst, 0)
 /* MOVE WITH OPTIONAL SPECIFICATION */
     C(0xc800, MVCOS,   SSF,   MVCOS, la1, a2, 0, 0, mvcos, 0)
 /* MOVE WITH OFFSET */
target/s390x/mem_helper.c +8 -12
index afcd452a00..8dd58b3ab1 100644
@@ -694,8 +694,10 @@ uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2)
 }
 
 /* string copy (c is string terminator) */
-uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
+uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2)
 {
+    const uint64_t d = get_address(env, r1);
+    const uint64_t s = get_address(env, r2);
     uintptr_t ra = GETPC();
     uint32_t len;
 
@@ -703,8 +705,6 @@ uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
         s390_program_interrupt(env, PGM_SPECIFICATION, ILEN_AUTO, ra);
     }
     c = c & 0xff;
-    d = wrap_address(env, d);
-    s = wrap_address(env, s);
 
     /* Lest we fail to service interrupts in a timely manner, limit the
        amount of work we're willing to do.  For now, let's cap at 8k.  */
@@ -712,17 +712,13 @@ uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
         uint8_t v = cpu_ldub_data_ra(env, s + len, ra);
         cpu_stb_data_ra(env, d + len, v, ra);
         if (v == c) {
-            /* Complete.  Set CC=1 and advance R1.  */
-            env->cc_op = 1;
-            env->retxl = s;
-            return d + len;
+            set_address_zero(env, r1, d + len);
+            return 1;
         }
     }
-
-    /* Incomplete.  Set CC=3 and signal to advance R1 and R2.  */
-    env->cc_op = 3;
-    env->retxl = s + len;
-    return d + len;
+    set_address_zero(env, r1, d + len);
+    set_address_zero(env, r2, s + len);
+    return 3;
 }
 
 /* load access registers r1 to r3 from memory at a2 */
target/s390x/translate.c +6 -2
index 2927247c82..b76e10d832 100644
@@ -3488,9 +3488,13 @@ static DisasJumpType op_mvpg(DisasContext *s, DisasOps *o)
 
 static DisasJumpType op_mvst(DisasContext *s, DisasOps *o)
 {
-    gen_helper_mvst(o->in1, cpu_env, regs[0], o->in1, o->in2);
+    TCGv_i32 t1 = tcg_const_i32(get_field(s->fields, r1));
+    TCGv_i32 t2 = tcg_const_i32(get_field(s->fields, r2));
+
+    gen_helper_mvst(cc_op, cpu_env, regs[0], t1, t2);
+    tcg_temp_free_i32(t1);
+    tcg_temp_free_i32(t2);
     set_cc_static(s);
-    return_low128(o->in2);
     return DISAS_NEXT;
 }
 
Signature · 1 line
2.21.0
Reviewed-by on v2 13/28
#31 lore raw

Re: [Qemu-devel] [PATCH v2 13/28] s390x/tcg: MVST: Fix storing back the addresses to registers

Quoted 10 lines · David Hildenbrand

24 and 31-bit address space handling is wrong when it comes to storing back the addresses to the register.

---
 target/s390x/helper.h      |  2 +-
 target/s390x/insn-data.def |  2 +-
 target/s390x/mem_helper.c  | 20 ++++++++------------
 target/s390x/translate.c   |  8 ++++++--
 4 files changed, 16 insertions(+), 16 deletions(-)

target/s390x/mem_helper.c uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2) · line 700

Quoted 5 lines
target/s390x/mem_helper.c +3 -1
-uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
+uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2)
 {
+    const uint64_t d = get_address(env, r1);
+    const uint64_t s = get_address(env, r2);

While you're changing the signature, you could also read R0 implicitly from env instead of passing it as C.

r~

#32 lore raw

Re: [Qemu-devel] [PATCH v2 13/28] s390x/tcg: MVST: Fix storing back the addresses to registers

Quoted 15 lines · Richard Henderson
Quoted 10 lines · David Hildenbrand

24 and 31-bit address space handling is wrong when it comes to storing back the addresses to the register.

---
 target/s390x/helper.h      |  2 +-
 target/s390x/insn-data.def |  2 +-
 target/s390x/mem_helper.c  | 20 ++++++++------------
 target/s390x/translate.c   |  8 ++++++--
 4 files changed, 16 insertions(+), 16 deletions(-)
Quoted 9 lines

target/s390x/mem_helper.c uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2) · line 700

Quoted 5 lines
target/s390x/mem_helper.c +3 -1
-uint64_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint64_t d, uint64_t s)
+uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2)
 {
+    const uint64_t d = get_address(env, r1);
+    const uint64_t s = get_address(env, r2);

While you're changing the signature, you could also read R0 implicitly from env instead of passing it as C.

Had exactly the same in mind, will do!

Signature · 3 lines
Thanks,

David / dhildenb
#33 lore raw

PATCH v2 14/28 s390x/tcg: Always use MMU_USER_IDX for CONFIG_USER_ONLY

In reply to David Hildenbrand

Although we basically ignore the index all the time for CONFIG_USER_ONLY, let's simply skip all the checks and always return MMU_USER_IDX in cpu_mmu_index() and get_mem_index().

---
 target/s390x/cpu.h       | 4 ++++
 target/s390x/translate.c | 4 ++++
 2 files changed, 8 insertions(+)
target/s390x/cpu.h +4 -0
index 79202c0980..163dae13d7 100644
@@ -328,6 +328,9 @@ extern const VMStateDescription vmstate_s390_cpu;
 
 static inline int cpu_mmu_index(CPUS390XState *env, bool ifetch)
 {
+#ifdef CONFIG_USER_ONLY
+    return MMU_USER_IDX;
+#else
     if (!(env->psw.mask & PSW_MASK_DAT)) {
         return MMU_REAL_IDX;
     }
@@ -351,6 +354,7 @@ static inline int cpu_mmu_index(CPUS390XState *env, bool ifetch)
     default:
         abort();
     }
+#endif
 }
 
 static inline void cpu_get_tb_cpu_state(CPUS390XState* env, target_ulong *pc,
target/s390x/translate.c +4 -0
index b76e10d832..50124520c9 100644
@@ -318,6 +318,9 @@ static inline uint64_t ld_code4(CPUS390XState *env, uint64_t pc)
 
 static int get_mem_index(DisasContext *s)
 {
+#ifdef CONFIG_USER_ONLY
+    return MMU_USER_IDX;
+#else
     if (!(s->base.tb->flags & FLAG_MASK_DAT)) {
         return MMU_REAL_IDX;
     }
@@ -333,6 +336,7 @@ static int get_mem_index(DisasContext *s)
         tcg_abort();
         break;
     }
+#endif
 }
 
 static void gen_exception(int excp)
Signature · 1 line
2.21.0
Reviewed-by on v2 14/28
#34 lore raw

Re: [Qemu-devel] [PATCH v2 14/28] s390x/tcg: Always use MMU_USER_IDX for CONFIG_USER_ONLY

Quoted 9 lines · David Hildenbrand

Although we basically ignore the index all the time for CONFIG_USER_ONLY, let's simply skip all the checks and always return MMU_USER_IDX in cpu_mmu_index() and get_mem_index().

---
 target/s390x/cpu.h       | 4 ++++
 target/s390x/translate.c | 4 ++++
 2 files changed, 8 insertions(+)

r~

#35 lore raw

PATCH v2 15/28 s390x/tcg: Fault-safe memset

In reply to David Hildenbrand

Replace fast_memset() by access_memset(), that first tries to probe access to all affected pages (maximum is two). We'll use the same mechanism for other types of accesses soon.

Only in very rare cases (especially TLB_NOTDIRTY), we'll have to fallback to ld/st helpers.

---
 target/s390x/mem_helper.c | 99 ++++++++++++++++++++++++++++++---------
 1 file changed, 77 insertions(+), 22 deletions(-)
target/s390x/mem_helper.c +77 -22
index 8dd58b3ab1..8d654b24e7 100644
@@ -117,27 +117,82 @@ static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr,
     }
 }
 
-static void fast_memset(CPUS390XState *env, uint64_t dest, uint8_t byte,
-                        uint32_t l, uintptr_t ra)
-{
-    int mmu_idx = cpu_mmu_index(env, false);
+/*
+ * An access covers at most 4096 bytes and therefore at most two pages. If
+ * we can't access the host page directly, we'll have to do I/O access
+ * via ld/st helpers.
+ */
+typedef struct S390Access {
+    target_ulong vaddr1;
+    target_ulong vaddr2;
+    char *haddr1;
+    char *haddr2;
+    uint16_t size1;
+    uint16_t size2;
+} S390Access;
+
+static S390Access access_prepare_idx(CPUS390XState *env, vaddr vaddr, int size,
+                                     MMUAccessType access_type, int mmu_idx,
+                                     uintptr_t ra)
+{
+    S390Access access = {
+        .vaddr1 = vaddr,
+        .size1 = MIN(size, -(vaddr | TARGET_PAGE_MASK)),
+    };
+
+    g_assert(size > 0 && size <= 4096);
+    access.haddr1 = probe_access(env, access.vaddr1, access.size1, access_type,
+                                 mmu_idx, ra);
+
+    if (unlikely(access.size1 != size)) {
+        /* The access crosses page boundaries. */
+        access.vaddr2 = wrap_address(env, vaddr + access.size1);
+        access.size2 = size - access.size1;
+        access.haddr2 = probe_access(env, access.vaddr2, access.size2,
+                                     access_type, mmu_idx, ra);
+    }
+    return access;
+}
+
+static void access_memset_idx(CPUS390XState *env, vaddr vaddr, uint8_t byte,
+                              int size, int mmu_idx, uintptr_t ra)
+{
+    S390Access desta = access_prepare_idx(env, vaddr, size, MMU_DATA_STORE,
+                                          mmu_idx, ra);
+#ifdef CONFIG_USER_ONLY
+    g_assert(desta.haddr1 && (desta.haddr2 || !desta.size2));
+    memset(desta.haddr1, byte, desta.size1);
+    memset(desta.haddr2, byte, desta.size2);
+#else
+    TCGMemOpIdx oi = make_memop_idx(MO_UB, mmu_idx);
+    int i;
 
-    while (l > 0) {
-        void *p = tlb_vaddr_to_host(env, dest, MMU_DATA_STORE, mmu_idx);
-        if (p) {
-            /* Access to the whole page in write mode granted.  */
-            uint32_t l_adj = adj_len_to_page(l, dest);
-            memset(p, byte, l_adj);
-            dest += l_adj;
-            l -= l_adj;
-        } else {
-            /* We failed to get access to the whole page. The next write
-               access will likely fill the QEMU TLB for the next iteration.  */
-            cpu_stb_data_ra(env, dest, byte, ra);
-            dest++;
-            l--;
+    if (likely(desta.haddr1)) {
+        memset(desta.haddr1, byte, desta.size1);
+    } else {
+        for (i = 0; i < desta.size1; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr1 + i, byte, oi, ra);
+        }
+    }
+    if (likely(!desta.size2)) {
+        return;
+    }
+    if (likely(desta.haddr2)) {
+            memset(desta.haddr2, byte, desta.size2);
+    } else {
+        for (i = 0; i < desta.size2; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr2 + i, byte, oi, ra);
         }
     }
+#endif
+}
+
+static void access_memset(CPUS390XState *env, vaddr vaddr, uint8_t byte,
+                          int size, uintptr_t ra)
+{
+    int mmu_idx = cpu_mmu_index(env, false);
+
+    access_memset_idx(env, vaddr, byte, size, mmu_idx, ra);
 }
 
 #ifndef CONFIG_USER_ONLY
@@ -267,7 +322,7 @@ static uint32_t do_helper_xc(CPUS390XState *env, uint32_t l, uint64_t dest,
 
     /* xor with itself is the same as memset(0) */
     if (src == dest) {
-        fast_memset(env, dest, 0, l + 1, ra);
+        access_memset(env, dest, 0, l + 1, ra);
         return 0;
     }
 
@@ -329,7 +384,7 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
      * behave like memmove().
      */
     if (dest == src + 1) {
-        fast_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
+        access_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
     } else if (!is_destructive_overlap(env, dest, src, l)) {
         fast_memmove(env, dest, src, l, ra);
     } else {
@@ -798,7 +853,7 @@ static inline uint32_t do_mvcl(CPUS390XState *env,
     } else if (wordsize == 1) {
         /* Pad the remaining area */
         *destlen -= len;
-        fast_memset(env, *dest, pad, len, ra);
+        access_memset(env, *dest, pad, len, ra);
         *dest = wrap_address(env, *dest + len);
     } else {
         /* The remaining length selects the padding byte. */
@@ -852,7 +907,7 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
     while (destlen) {
         cur_len = MIN(destlen, 2048);
         if (!srclen) {
-            fast_memset(env, dest, pad, cur_len, ra);
+            access_memset(env, dest, pad, cur_len, ra);
         } else {
             cur_len = MIN(cur_len, srclen);
 
Signature · 1 line
2.21.0
Reviewed-by on v2 15/28
#36 lore raw

Re: [Qemu-devel] [PATCH v2 15/28] s390x/tcg: Fault-safe memset

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 175

Quoted 6 lines · David Hildenbrand
target/s390x/mem_helper.c +6 -0
+    if (likely(desta.haddr1)) {
+        memset(desta.haddr1, byte, desta.size1);
+    } else {
+        for (i = 0; i < desta.size1; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr1 + i, byte, oi, ra);
+        }

The only thing perhaps that we could do better here is to re-check tlb_vaddr_to_host after a singe access. This would handle NOTDIRTY with a single iotlb access and then fall back to memset, without re-checking host address for real mmio after every store.

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 181

Quoted 2 lines
target/s390x/mem_helper.c +2 -0
+    if (likely(desta.haddr2)) {
+            memset(desta.haddr2, byte, desta.size2);

Indentation.

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 185

Quoted 4 lines
target/s390x/mem_helper.c +3 -0
+    } else {
+        for (i = 0; i < desta.size2; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr2 + i, byte, oi, ra);
         }

Likewise wrt NOTDIRTY, which suggests a subroutine to handle a single page?

That said, what's here looks correct so

r~

#37 lore raw

Re: [Qemu-devel] [PATCH v2 15/28] s390x/tcg: Fault-safe memset

Quoted 8 lines · Richard Henderson

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 175

Quoted 6 lines · David Hildenbrand
target/s390x/mem_helper.c +6 -0
+    if (likely(desta.haddr1)) {
+        memset(desta.haddr1, byte, desta.size1);
+    } else {
+        for (i = 0; i < desta.size1; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr1 + i, byte, oi, ra);
+        }
Quoted 4 lines

The only thing perhaps that we could do better here is to re-check tlb_vaddr_to_host after a singe access. This would handle NOTDIRTY with a single iotlb access and then fall back to memset, without re-checking host address for real mmio after every store.

Yes - also thought about that :) - but I didn't find a clean way to do that without uglifying that code (and especially access_memmove() and friends).

Quoted 4 lines

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 181

Quoted 3 lines
target/s390x/mem_helper.c +2 -0
+    if (likely(desta.haddr2)) {
+            memset(desta.haddr2, byte, desta.size2);
Quoted 8 lines

Indentation.

target/s390x/mem_helper.c static inline void cpu_stsize_data_ra(CPUS390XState *env, uint64_t addr, · line 185

Quoted 4 lines
target/s390x/mem_helper.c +3 -0
+    } else {
+        for (i = 0; i < desta.size2; i++) {
+            helper_ret_stb_mmu(env, desta.vaddr2 + i, byte, oi, ra);
         }

Likewise wrt NOTDIRTY, which suggests a subroutine to handle a single page?

I'll try to see if this turns out okay-ish. Maybe I'll add via a separate patch on top.

Quoted 4 lines

That said, what's here looks correct so

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#38 lore raw

PATCH v2 16/28 s390x/tcg: Fault-safe memmove

In reply to David Hildenbrand

Replace fast_memmove() variants by access_memmove() variants, that first try to probe access to all affected pages (maximum is two pages).

In MVCOS, simply always call access_memmove_as() and drop the TODO about LAP. LAP is already handled in the MMU.

Get rid of adj_len_to_page(), which is now unused.

---
 target/s390x/mem_helper.c | 204 +++++++++++++++++++++-----------------
 1 file changed, 115 insertions(+), 89 deletions(-)
target/s390x/mem_helper.c +115 -89
index 8d654b24e7..db678ddf47 100644
@@ -65,17 +65,6 @@ static bool is_destructive_overlap(CPUS390XState *env, uint64_t dest,
     return dest > src && dest <= src + len - 1;
 }
 
-/* Reduce the length so that addr + len doesn't cross a page boundary.  */
-static inline uint32_t adj_len_to_page(uint32_t len, uint64_t addr)
-{
-#ifndef CONFIG_USER_ONLY
-    if ((addr & ~TARGET_PAGE_MASK) + len - 1 >= TARGET_PAGE_SIZE) {
-        return -(addr | TARGET_PAGE_MASK);
-    }
-#endif
-    return len;
-}
-
 /* Trigger a SPECIFICATION exception if an address or a length is not
    naturally aligned.  */
 static inline void check_alignment(CPUS390XState *env, uint64_t v,
@@ -129,6 +118,7 @@ typedef struct S390Access {
     char *haddr2;
     uint16_t size1;
     uint16_t size2;
+    int mmu_idx;
 } S390Access;
 
 static S390Access access_prepare_idx(CPUS390XState *env, vaddr vaddr, int size,
@@ -138,6 +128,7 @@ static S390Access access_prepare_idx(CPUS390XState *env, vaddr vaddr, int size,
     S390Access access = {
         .vaddr1 = vaddr,
         .size1 = MIN(size, -(vaddr | TARGET_PAGE_MASK)),
+        .mmu_idx = mmu_idx,
     };
 
     g_assert(size > 0 && size <= 4096);
@@ -195,42 +186,112 @@ static void access_memset(CPUS390XState *env, vaddr vaddr, uint8_t byte,
     access_memset_idx(env, vaddr, byte, size, mmu_idx, ra);
 }
 
-#ifndef CONFIG_USER_ONLY
-static void fast_memmove_idx(CPUS390XState *env, uint64_t dest, uint64_t src,
-                             uint32_t len, int dest_idx, int src_idx,
-                             uintptr_t ra)
-{
-    TCGMemOpIdx oi_dest = make_memop_idx(MO_UB, dest_idx);
-    TCGMemOpIdx oi_src = make_memop_idx(MO_UB, src_idx);
-    uint32_t len_adj;
-    void *src_p;
-    void *dest_p;
-    uint8_t x;
-
-    while (len > 0) {
-        src = wrap_address(env, src);
-        dest = wrap_address(env, dest);
-        src_p = tlb_vaddr_to_host(env, src, MMU_DATA_LOAD, src_idx);
-        dest_p = tlb_vaddr_to_host(env, dest, MMU_DATA_STORE, dest_idx);
-
-        if (src_p && dest_p) {
-            /* Access to both whole pages granted.  */
-            len_adj = adj_len_to_page(adj_len_to_page(len, src), dest);
-            memmove(dest_p, src_p, len_adj);
+static uint8_t access_get_byte(CPUS390XState *env, const S390Access *access,
+                               int offset, uintptr_t ra)
+{
+#ifdef CONFIG_USER_ONLY
+    if (offset < access->size1) {
+        return ldub_p(access->haddr1 + offset);
+    } else {
+        return ldub_p(access->haddr2 + offset - access->size1);
+    }
+#else
+    TCGMemOpIdx oi = make_memop_idx(MO_UB, access->mmu_idx);
+
+    if (offset < access->size1) {
+        if (likely(access->haddr1)) {
+            return ldub_p(access->haddr1 + offset);
+        }
+        return helper_ret_ldub_mmu(env, access->vaddr1 + offset, oi, ra);
+    }
+    if (access->haddr2) {
+        return ldub_p(access->haddr2 + offset - access->size1);
+    }
+    return helper_ret_ldub_mmu(env, access->vaddr2 + offset - access->size1, oi,
+                               ra);
+#endif
+}
+
+static void access_set_byte(CPUS390XState *env, const S390Access *access,
+                            int offset, uint8_t byte, uintptr_t ra)
+{
+#ifdef CONFIG_USER_ONLY
+    if (offset < access->size1) {
+        stb_p(access->haddr1 + offset, byte);
+    } else {
+        stb_p(access->haddr2 + offset - access->size1, byte);
+    }
+#else
+    TCGMemOpIdx oi = make_memop_idx(MO_UB, access->mmu_idx);
+
+    if (offset < access->size1) {
+        if (likely(access->haddr1)) {
+            stb_p(access->haddr1 + offset, byte);
         } else {
-            /* We failed to get access to one or both whole pages. The next
-               read or write access will likely fill the QEMU TLB for the
-               next iteration.  */
-            len_adj = 1;
-            x = helper_ret_ldub_mmu(env, src, oi_src, ra);
-            helper_ret_stb_mmu(env, dest, x, oi_dest, ra);
+            helper_ret_stb_mmu(env, access->vaddr1 + offset, byte, oi, ra);
+        }
+    } else {
+        if (likely(access->haddr2)) {
+            stb_p(access->haddr2 + offset - access->size1, byte);
+        } else {
+            helper_ret_stb_mmu(env, access->vaddr2 + offset - access->size1,
+                               byte, oi, ra);
+        }
+    }
+#endif
+}
+
+/*
+ * Move data with the same semantics as memmove() in case ranges don't overlap
+ * or src > dest. Undefined behavior on destructive overlaps.
+ */
+static void access_memmove_idx(CPUS390XState *env, vaddr dest, vaddr src,
+                               int size, int dest_idx, int src_idx,
+                               uintptr_t ra)
+{
+    S390Access srca = access_prepare_idx(env, src, size, MMU_DATA_LOAD, src_idx,
+                                         ra);
+    S390Access desta = access_prepare_idx(env, dest, size, MMU_DATA_STORE,
+                                          dest_idx, ra);
+    uint16_t diff;
+
+    /* Fallback to slow access in case we don't have access to all host pages */
+    if (unlikely(!desta.haddr1 || (desta.size2 && !desta.haddr2) ||
+                 !srca.haddr1 || (srca.size2 && !srca.haddr2))) {
+        uint8_t byte;
+        int i;
+
+        for (i = 0; i < desta.size1 + desta.size2; i++) {
+            byte = access_get_byte(env, &srca, i, ra);
+            access_set_byte(env, &desta, i, byte, ra);
         }
-        src += len_adj;
-        dest += len_adj;
-        len -= len_adj;
+        return;
+    }
+
+    if (srca.size1 == desta.size1) {
+        memmove(desta.haddr1, srca.haddr1, srca.size1);
+        memmove(desta.haddr2, srca.haddr2, srca.size2);
+    } else if (srca.size1 < desta.size1) {
+        diff = desta.size1 - srca.size1;
+        memmove(desta.haddr1, srca.haddr1, srca.size1);
+        memmove(desta.haddr1 + srca.size1, srca.haddr2, diff);
+        memmove(desta.haddr2, srca.haddr2 + diff, desta.size2);
+    } else {
+        diff = srca.size1 - desta.size1;
+        memmove(desta.haddr1, srca.haddr1, desta.size1);
+        memmove(desta.haddr2, srca.haddr1 + desta.size1, diff);
+        memmove(desta.haddr2 + diff, srca.haddr2, srca.size2);
     }
 }
 
+static void access_memmove(CPUS390XState *env, vaddr dest, vaddr src,
+                           int size, uintptr_t ra)
+{
+    int mmu_idx = cpu_mmu_index(env, false);
+
+    access_memmove_idx(env, dest, src, size, mmu_idx, mmu_idx, ra);
+}
+
 static int mmu_idx_from_as(uint8_t as)
 {
     switch (as) {
@@ -246,43 +307,14 @@ static int mmu_idx_from_as(uint8_t as)
     }
 }
 
-static void fast_memmove_as(CPUS390XState *env, uint64_t dest, uint64_t src,
-                            uint32_t len, uint8_t dest_as, uint8_t src_as,
-                            uintptr_t ra)
+static void access_memmove_as(CPUS390XState *env, uint64_t dest, uint64_t src,
+                              uint32_t len, uint8_t dest_as, uint8_t src_as,
+                              uintptr_t ra)
 {
     int src_idx = mmu_idx_from_as(src_as);
     int dest_idx = mmu_idx_from_as(dest_as);
 
-    fast_memmove_idx(env, dest, src, len, dest_idx, src_idx, ra);
-}
-#endif
-
-static void fast_memmove(CPUS390XState *env, uint64_t dest, uint64_t src,
-                         uint32_t l, uintptr_t ra)
-{
-    int mmu_idx = cpu_mmu_index(env, false);
-
-    while (l > 0) {
-        void *src_p = tlb_vaddr_to_host(env, src, MMU_DATA_LOAD, mmu_idx);
-        void *dest_p = tlb_vaddr_to_host(env, dest, MMU_DATA_STORE, mmu_idx);
-        if (src_p && dest_p) {
-            /* Access to both whole pages granted.  */
-            uint32_t l_adj = adj_len_to_page(l, src);
-            l_adj = adj_len_to_page(l_adj, dest);
-            memmove(dest_p, src_p, l_adj);
-            src += l_adj;
-            dest += l_adj;
-            l -= l_adj;
-        } else {
-            /* We failed to get access to one or both whole pages. The next
-               read or write access will likely fill the QEMU TLB for the
-               next iteration.  */
-            cpu_stb_data_ra(env, dest, cpu_ldub_data_ra(env, src, ra), ra);
-            src++;
-            dest++;
-            l--;
-        }
-    }
+    access_memmove_idx(env, dest, src, len, dest_idx, src_idx, ra);
 }
 
 /* and on array */
@@ -386,7 +418,7 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
     if (dest == src + 1) {
         access_memset(env, dest, cpu_ldub_data_ra(env, src, ra), l, ra);
     } else if (!is_destructive_overlap(env, dest, src, l)) {
-        fast_memmove(env, dest, src, l, ra);
+        access_memmove(env, dest, src, l, ra);
     } else {
         for (i = 0; i < l; i++) {
             uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
@@ -744,7 +776,7 @@ uint32_t HELPER(mvpg)(CPUS390XState *env, uint64_t r0, uint64_t r1, uint64_t r2)
      * - CC-option with surpression of page-translation exceptions
      * - Store r1/r2 register identifiers at real location 162
      */
-    fast_memmove(env, r1, r2, TARGET_PAGE_SIZE, GETPC());
+    access_memmove(env, r1, r2, TARGET_PAGE_SIZE, GETPC());
     return 0; /* data moved */
 }
 
@@ -847,7 +879,7 @@ static inline uint32_t do_mvcl(CPUS390XState *env,
         len = MIN(len, *srclen);
         *destlen -= len;
         *srclen -= len;
-        fast_memmove(env, *dest, *src, len, ra);
+        access_memmove(env, *dest, *src, len, ra);
         *src = wrap_address(env, *src + len);
         *dest = wrap_address(env, *dest + len);
     } else if (wordsize == 1) {
@@ -911,7 +943,7 @@ uint32_t HELPER(mvcl)(CPUS390XState *env, uint32_t r1, uint32_t r2)
         } else {
             cur_len = MIN(cur_len, srclen);
 
-            fast_memmove(env, dest, src, cur_len, ra);
+            access_memmove(env, dest, src, cur_len, ra);
             src = wrap_address(env, src + cur_len);
             srclen -= cur_len;
             env->regs[r2 + 1] = deposit64(env->regs[r2 + 1], 0, 24, srclen);
@@ -2453,16 +2485,10 @@ uint32_t HELPER(mvcos)(CPUS390XState *env, uint64_t dest, uint64_t src,
         s390_program_interrupt(env, PGM_ADDRESSING, 6, ra);
     }
 
-    /* FIXME: a) LAP
-     *        b) Access using correct keys
-     *        c) AR-mode
-     */
-#ifdef CONFIG_USER_ONLY
-    /* psw keys are never valid in user mode, we will never reach this */
-    g_assert_not_reached();
-#else
-    fast_memmove_as(env, dest, src, len, dest_as, src_as, ra);
-#endif
+    /* FIXME: Access using correct keys and AR-mode */
+    if (len) {
+        access_memmove_as(env, dest, src, len, dest_as, src_as, ra);
+    }
 
     return cc;
 }
Signature · 1 line
2.21.0
Reviewed-by on v2 16/28
#39 lore raw

Re: [Qemu-devel] [PATCH v2 16/28] s390x/tcg: Fault-safe memmove

Quoted 6 lines · David Hildenbrand

Replace fast_memmove() variants by access_memmove() variants, that first try to probe access to all affected pages (maximum is two pages).

In MVCOS, simply always call access_memmove_as() and drop the TODO about LAP. LAP is already handled in the MMU.

Quoted 6 lines

Get rid of adj_len_to_page(), which is now unused.

---
 target/s390x/mem_helper.c | 204 +++++++++++++++++++++-----------------
 1 file changed, 115 insertions(+), 89 deletions(-)

r~

#40 lore raw

Re: [Qemu-devel] [PATCH v2 16/28] s390x/tcg: Fault-safe memmove

In reply to David Hildenbrand

target/s390x/mem_helper.c static void access_memset(CPUS390XState *env, vaddr vaddr, uint8_t byte, · line 255

Quoted 8 lines · David Hildenbrand
target/s390x/mem_helper.c +8 -0
+static void access_memmove_idx(CPUS390XState *env, vaddr dest, vaddr src,
+                               int size, int dest_idx, int src_idx,
+                               uintptr_t ra)
+{
+    S390Access srca = access_prepare_idx(env, src, size, MMU_DATA_LOAD, src_idx,
+                                         ra);
+    S390Access desta = access_prepare_idx(env, dest, size, MMU_DATA_STORE,
+                                          dest_idx, ra);

I was just thinking that it might be worth passing in these Access structures.
It seems that usually we wind up computing them in several locations.

Hoisting it up it would make MVC look like

    midx = cpu_mmu_index(env);
    srca = access_prepare_idx(env, src, size, LOAD, midx, ra);
    dsta = access_prepare_idx(env, dst, size, STORE, midx, ra);

    if (dst == src + 1) {
        uint8_t x = access_get_byte(env, &srca, 0, ra);
        access_memset(env, &dsta, x, size, ra);
    } else if (!is_destructive_overlap(env, dst, src, size)) {
        access_memmove(env, &dsta, &srca, size, ra);
    } else {
        // byte by byte loop, but still need srca, dsta.
    }

which seems even More Correct, since the current access_memset path does not check for read watchpoints or faults on all of [src, src+size-1].

r~

#41 lore raw

Re: [Qemu-devel] [PATCH v2 16/28] s390x/tcg: Fault-safe memmove

Quoted 28 lines · Richard Henderson

target/s390x/mem_helper.c static void access_memset(CPUS390XState *env, vaddr vaddr, uint8_t byte, · line 255

Quoted 8 lines · David Hildenbrand
target/s390x/mem_helper.c +8 -0
+static void access_memmove_idx(CPUS390XState *env, vaddr dest, vaddr src,
+                               int size, int dest_idx, int src_idx,
+                               uintptr_t ra)
+{
+    S390Access srca = access_prepare_idx(env, src, size, MMU_DATA_LOAD, src_idx,
+                                         ra);
+    S390Access desta = access_prepare_idx(env, dest, size, MMU_DATA_STORE,
+                                          dest_idx, ra);

I was just thinking that it might be worth passing in these Access structures.
It seems that usually we wind up computing them in several locations.

Hoisting it up it would make MVC look like

    midx = cpu_mmu_index(env);
    srca = access_prepare_idx(env, src, size, LOAD, midx, ra);
    dsta = access_prepare_idx(env, dst, size, STORE, midx, ra);

    if (dst == src + 1) {
        uint8_t x = access_get_byte(env, &srca, 0, ra);
        access_memset(env, &dsta, x, size, ra);
    } else if (!is_destructive_overlap(env, dst, src, size)) {
        access_memmove(env, &dsta, &srca, size, ra);
    } else {
        // byte by byte loop, but still need srca, dsta.
    }
Quoted 3 lines

which seems even More Correct, since the current access_memset path does not check for read watchpoints or faults on all of [src, src+size-1].

I had precisely that in previous versions :) Can switch to that model.

Signature · 3 lines
Thanks,

David / dhildenb
#42 lore raw

PATCH v2 17/28 s390x/tcg: MVCS/MVCP: Use access_memmove_idx()

In reply to David Hildenbrand

As we are moving between address spaces, we can use access_memmove_idx() without checking for destructive overlaps (especially of real storage locations):
"Each storage operand is processed left to right. The storage-operand-consistency rules are the same as
for MOVE (MVC), except that when the operands
overlap in real storage, the use of the common real- storage locations is not necessarily recognized."

---
 target/s390x/mem_helper.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)
target/s390x/mem_helper.c +6 -14
index db678ddf47..2607dd1677 100644
@@ -2049,7 +2049,7 @@ uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
 {
     const uint8_t psw_as = (env->psw.mask & PSW_MASK_ASC) >> PSW_SHIFT_ASC;
     uintptr_t ra = GETPC();
-    int cc = 0, i;
+    int cc = 0;
 
     HELPER_LOG("%s: %16" PRIx64 " %16" PRIx64 " %16" PRIx64 "\n",
                __func__, l, a1, a2);
@@ -2068,12 +2068,8 @@ uint32_t HELPER(mvcs)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
         return cc;
     }
 
-    /* XXX replace w/ memcpy */
-    for (i = 0; i < l; i++) {
-        uint8_t x = cpu_ldub_primary_ra(env, a2 + i, ra);
-        cpu_stb_secondary_ra(env, a1 + i, x, ra);
-    }
-
+    /* TODO: Access key handling */
+    access_memmove_idx(env, a1, a2, l, MMU_SECONDARY_IDX, MMU_PRIMARY_IDX, ra);
     return cc;
 }
 
@@ -2081,7 +2077,7 @@ uint32_t HELPER(mvcp)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
 {
     const uint8_t psw_as = (env->psw.mask & PSW_MASK_ASC) >> PSW_SHIFT_ASC;
     uintptr_t ra = GETPC();
-    int cc = 0, i;
+    int cc = 0;
 
     HELPER_LOG("%s: %16" PRIx64 " %16" PRIx64 " %16" PRIx64 "\n",
                __func__, l, a1, a2);
@@ -2100,12 +2096,8 @@ uint32_t HELPER(mvcp)(CPUS390XState *env, uint64_t l, uint64_t a1, uint64_t a2)
         return cc;
     }
 
-    /* XXX replace w/ memcpy */
-    for (i = 0; i < l; i++) {
-        uint8_t x = cpu_ldub_secondary_ra(env, a2 + i, ra);
-        cpu_stb_primary_ra(env, a1 + i, x, ra);
-    }
-
+    /* TODO: Access key handling */
+    access_memmove_idx(env, a1, a2, l, MMU_PRIMARY_IDX, MMU_SECONDARY_IDX, ra);
     return cc;
 }
 
Signature · 1 line
2.21.0
Reviewed-by on v2 17/28
#43 lore raw

Re: [Qemu-devel] [PATCH v2 17/28] s390x/tcg: MVCS/MVCP: Use access_memmove_idx()

Quoted 9 lines · David Hildenbrand

As we are moving between address spaces, we can use access_memmove_idx() without checking for destructive overlaps (especially of real storage locations):
"Each storage operand is processed left to right. The storage-operand-consistency rules are the same as
for MOVE (MVC), except that when the operands
overlap in real storage, the use of the common real- storage locations is not necessarily recognized."

Quoted 4 lines
---
 target/s390x/mem_helper.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)

r~

#44 lore raw

PATCH v2 18/28 s390x/tcg: MVC: Fault-safe handling on destructive overlaps

In reply to David Hildenbrand

The last remaining bit for MVC is handling destructive overlaps in a fault-safe way.

---
 target/s390x/mem_helper.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
target/s390x/mem_helper.c +14 -2
index 2607dd1677..f636f3a011 100644
@@ -145,6 +145,14 @@ static S390Access access_prepare_idx(CPUS390XState *env, vaddr vaddr, int size,
     return access;
 }
 
+static S390Access access_prepare(CPUS390XState *env, vaddr vaddr, int size,
+                                 MMUAccessType access_type, uintptr_t ra)
+{
+    int mmu_idx = cpu_mmu_index(env, false);
+
+    return access_prepare_idx(env, vaddr, size, access_type, mmu_idx, ra);
+}
+
 static void access_memset_idx(CPUS390XState *env, vaddr vaddr, uint8_t byte,
                               int size, int mmu_idx, uintptr_t ra)
 {
@@ -420,9 +428,13 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
     } else if (!is_destructive_overlap(env, dest, src, l)) {
         access_memmove(env, dest, src, l, ra);
     } else {
+        S390Access srca = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+        S390Access desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+
         for (i = 0; i < l; i++) {
-            uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
-            cpu_stb_data_ra(env, dest + i, x, ra);
+            uint8_t byte = access_get_byte(env, &srca, i, ra);
+
+            access_set_byte(env, &desta, i, byte, ra);
         }
     }
 
Signature · 1 line
2.21.0
Reviewed-by on v2 18/28
#45 lore raw

Re: [Qemu-devel] [PATCH v2 18/28] s390x/tcg: MVC: Fault-safe handling on destructive overlaps

Quoted 12 lines · David Hildenbrand
target/s390x/mem_helper.c +8 -0
+static S390Access access_prepare(CPUS390XState *env, vaddr vaddr, int size,
+                                 MMUAccessType access_type, uintptr_t ra)
+{
+    int mmu_idx = cpu_mmu_index(env, false);
+
+    return access_prepare_idx(env, vaddr, size, access_type, mmu_idx, ra);
+}
+
 static void access_memset_idx(CPUS390XState *env, vaddr vaddr, uint8_t byte,
                               int size, int mmu_idx, uintptr_t ra)
 {

target/s390x/mem_helper.c static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest, · line 432

Quoted 5 lines
target/s390x/mem_helper.c +2 -0
@@ -420,9 +428,13 @@ static uint32_t do_helper_mvc(CPUS390XState *env, uint32_t l, uint64_t dest,
     } else if (!is_destructive_overlap(env, dest, src, l)) {
         access_memmove(env, dest, src, l, ra);
     } else {
+        S390Access srca = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+        S390Access desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);

I was just thinking it might be better to drop the non-idx functions:
access_prepare, access_memset, access_memmove. What this is leading to is computation of cpu_mmu_index multiple times, as here.

It could just as easily be hoisted to the top of do_helper_mvc and used in all of the sub-cases.

That said, the code here is correct so

r~

#46 lore raw

PATCH v2 19/28 s390x/tcg: MVCLU: Fault-safe handling

In reply to David Hildenbrand

The last remaining bit is padding with two bytes.

---
 target/s390x/mem_helper.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
target/s390x/mem_helper.c +5 -3
index f636f3a011..0366cbc753 100644
@@ -900,15 +900,17 @@ static inline uint32_t do_mvcl(CPUS390XState *env,
         access_memset(env, *dest, pad, len, ra);
         *dest = wrap_address(env, *dest + len);
     } else {
+        S390Access desta = access_prepare(env, *dest, len, MMU_DATA_STORE, ra);
+
         /* The remaining length selects the padding byte. */
         for (i = 0; i < len; (*destlen)--, i++) {
             if (*destlen & 1) {
-                cpu_stb_data_ra(env, *dest, pad, ra);
+                access_set_byte(env, &desta, i, pad, ra);
             } else {
-                cpu_stb_data_ra(env, *dest, pad >> 8, ra);
+                access_set_byte(env, &desta, i, pad >> 8, ra);
             }
-            *dest = wrap_address(env, *dest + 1);
         }
+        *dest = wrap_address(env, *dest + len);
     }
 
     return *destlen ? 3 : cc;
Signature · 1 line
2.21.0
#48 lore raw

PATCH v2 20/28 s390x/tcg: OC: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages.

---
 target/s390x/mem_helper.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
target/s390x/mem_helper.c +12 -4
index 0366cbc753..ff57fec8de 100644
@@ -385,17 +385,25 @@ uint32_t HELPER(xc)(CPUS390XState *env, uint32_t l, uint64_t dest,
 static uint32_t do_helper_oc(CPUS390XState *env, uint32_t l, uint64_t dest,
                              uint64_t src, uintptr_t ra)
 {
+    S390Access srca1, srca2, desta;
     uint32_t i;
     uint8_t c = 0;
 
     HELPER_LOG("%s l %d dest %" PRIx64 " src %" PRIx64 "\n",
                __func__, l, dest, src);
 
-    for (i = 0; i <= l; i++) {
-        uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
-        x |= cpu_ldub_data_ra(env, dest + i, ra);
+    /* OC always processes one more byte than specified - maximum is 256 */
+    l++;
+
+    srca1 = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = access_get_byte(env, &srca1, i, ra) |
+                          access_get_byte(env, &srca2, i, ra);
+
         c |= x;
-        cpu_stb_data_ra(env, dest + i, x, ra);
+        access_set_byte(env, &desta, i, x, ra);
     }
     return c != 0;
 }
Signature · 1 line
2.21.0
Reviewed-by on v2 20/28
#49 lore raw

Re: [Qemu-devel] [PATCH v2 20/28] s390x/tcg: OC: Fault-safe handling

target/s390x/mem_helper.c uint32_t HELPER(xc)(CPUS390XState *env, uint32_t l, uint64_t dest, · line 400

Quoted 2 lines · David Hildenbrand
target/s390x/mem_helper.c +2 -0
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);

We should find a way to perform this in one step.
RWM isn't uncommon...

That said,

r~

#50 lore raw

Re: [Qemu-devel] [PATCH v2 20/28] s390x/tcg: OC: Fault-safe handling

Quoted 6 lines · Richard Henderson

target/s390x/mem_helper.c uint32_t HELPER(xc)(CPUS390XState *env, uint32_t l, uint64_t dest, · line 400

Quoted 2 lines · David Hildenbrand
target/s390x/mem_helper.c +2 -0
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);

We should find a way to perform this in one step.
RWM isn't uncommon...

Yes, like converting MMU_* into flags we can then process in probe_access().

Quoted 6 lines

That said,

r~

Signature · 3 lines
Thanks,

David / dhildenb
#51 lore raw

PATCH v2 21/28 s390x/tcg: XC: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages. While at it, increment the length once.

---
 target/s390x/mem_helper.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)
target/s390x/mem_helper.c +13 -5
index ff57fec8de..88ff6c21ed 100644
@@ -354,23 +354,31 @@ uint32_t HELPER(nc)(CPUS390XState *env, uint32_t l, uint64_t dest,
 static uint32_t do_helper_xc(CPUS390XState *env, uint32_t l, uint64_t dest,
                              uint64_t src, uintptr_t ra)
 {
+    S390Access srca1, srca2, desta;
     uint32_t i;
     uint8_t c = 0;
 
     HELPER_LOG("%s l %d dest %" PRIx64 " src %" PRIx64 "\n",
                __func__, l, dest, src);
 
+    /* XC always processes one more byte than specified - maximum is 256 */
+    l++;
+
     /* xor with itself is the same as memset(0) */
     if (src == dest) {
-        access_memset(env, dest, 0, l + 1, ra);
+        access_memset(env, dest, 0, l, ra);
         return 0;
     }
 
-    for (i = 0; i <= l; i++) {
-        uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
-        x ^= cpu_ldub_data_ra(env, dest + i, ra);
+    srca1 = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = access_get_byte(env, &srca1, i, ra) ^
+                          access_get_byte(env, &srca2, i, ra);
+
         c |= x;
-        cpu_stb_data_ra(env, dest + i, x, ra);
+        access_set_byte(env, &desta, i, x, ra);
     }
     return c != 0;
 }
Signature · 1 line
2.21.0
#53 lore raw

PATCH v2 22/28 s390x/tcg: NC: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages.

---
 target/s390x/mem_helper.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
target/s390x/mem_helper.c +12 -4
index 88ff6c21ed..49b4879859 100644
@@ -329,17 +329,25 @@ static void access_memmove_as(CPUS390XState *env, uint64_t dest, uint64_t src,
 static uint32_t do_helper_nc(CPUS390XState *env, uint32_t l, uint64_t dest,
                              uint64_t src, uintptr_t ra)
 {
+    S390Access srca1, srca2, desta;
     uint32_t i;
     uint8_t c = 0;
 
     HELPER_LOG("%s l %d dest %" PRIx64 " src %" PRIx64 "\n",
                __func__, l, dest, src);
 
-    for (i = 0; i <= l; i++) {
-        uint8_t x = cpu_ldub_data_ra(env, src + i, ra);
-        x &= cpu_ldub_data_ra(env, dest + i, ra);
+    /* NC always processes one more byte than specified - maximum is 256 */
+    l++;
+
+    srca1 = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = access_get_byte(env, &srca1, i, ra) &
+                          access_get_byte(env, &srca2, i, ra);
+
         c |= x;
-        cpu_stb_data_ra(env, dest + i, x, ra);
+        access_set_byte(env, &desta, i, x, ra);
     }
     return c != 0;
 }
Signature · 1 line
2.21.0
#55 lore raw

PATCH v2 23/28 s390x/tcg: MVCIN: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages. Calculate the accessed range upfront - src is accessed right-to-left.

---
 target/s390x/mem_helper.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
target/s390x/mem_helper.c +11 -3
index 49b4879859..ba8a657e18 100644
@@ -473,12 +473,20 @@ void HELPER(mvc)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 /* move inverse  */
 void HELPER(mvcin)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 {
+    S390Access srca, desta;
     uintptr_t ra = GETPC();
     int i;
 
-    for (i = 0; i <= l; i++) {
-        uint8_t v = cpu_ldub_data_ra(env, src - i, ra);
-        cpu_stb_data_ra(env, dest + i, v, ra);
+    /* MVCIN always copies one more byte than specified - maximum is 256 */
+    l++;
+
+    src = wrap_address(env, src - l + 1);
+    srca = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = access_get_byte(env, &srca, l - i - 1, ra);
+
+        access_set_byte(env, &desta, i, x, ra);
     }
 }
 
Signature · 1 line
2.21.0
Reviewed-by on v2 23/28
#56 lore raw

Re: [Qemu-devel] [PATCH v2 23/28] s390x/tcg: MVCIN: Fault-safe handling

Quoted 7 lines · David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages. Calculate the accessed range upfront - src is accessed right-to-left.

---
 target/s390x/mem_helper.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

r~

#57 lore raw

PATCH v2 24/28 s390x/tcg: MVN: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages.

---
 target/s390x/mem_helper.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
target/s390x/mem_helper.c +12 -4
index ba8a657e18..5e38b2c4d8 100644
@@ -493,13 +493,21 @@ void HELPER(mvcin)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 /* move numerics  */
 void HELPER(mvn)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 {
+    S390Access srca1, srca2, desta;
     uintptr_t ra = GETPC();
     int i;
 
-    for (i = 0; i <= l; i++) {
-        uint8_t v = cpu_ldub_data_ra(env, dest + i, ra) & 0xf0;
-        v |= cpu_ldub_data_ra(env, src + i, ra) & 0x0f;
-        cpu_stb_data_ra(env, dest + i, v, ra);
+    /* MVN always copies one more byte than specified - maximum is 256 */
+    l++;
+
+    srca1 = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = (access_get_byte(env, &srca1, i, ra) & 0x0f) |
+                          (access_get_byte(env, &srca2, i, ra) & 0xf0);
+
+        access_set_byte(env, &desta, i, x, ra);
     }
 }
 
Signature · 1 line
2.21.0
#59 lore raw

PATCH v2 25/28 s390x/tcg: MVZ: Fault-safe handling

In reply to David Hildenbrand

We can process a maximum of 256 bytes, crossing two pages.

---
 target/s390x/mem_helper.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
target/s390x/mem_helper.c +12 -4
index 5e38b2c4d8..4c67c6f37e 100644
@@ -545,13 +545,21 @@ void HELPER(mvo)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 /* move zones  */
 void HELPER(mvz)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 {
+    S390Access srca1, srca2, desta;
     uintptr_t ra = GETPC();
     int i;
 
-    for (i = 0; i <= l; i++) {
-        uint8_t b = cpu_ldub_data_ra(env, dest + i, ra) & 0x0f;
-        b |= cpu_ldub_data_ra(env, src + i, ra) & 0xf0;
-        cpu_stb_data_ra(env, dest + i, b, ra);
+    /* MVZ always copies one more byte than specified - maximum is 256 */
+    l++;
+
+    srca1 = access_prepare(env, src, l, MMU_DATA_LOAD, ra);
+    srca2 = access_prepare(env, dest, l, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, l, MMU_DATA_STORE, ra);
+    for (i = 0; i < l; i++) {
+        const uint8_t x = (access_get_byte(env, &srca1, i, ra) & 0xf0) |
+                          (access_get_byte(env, &srca2, i, ra) & 0x0f);
+
+        access_set_byte(env, &desta, i, x, ra);
     }
 }
 
Signature · 1 line
2.21.0
#61 lore raw

PATCH v2 26/28 s390x/tcg: MVST: Fault-safe handling

In reply to David Hildenbrand

Access at most single pages and document why. Using the access helpers might over-indicate watchpoints within the same page, I guess we can live with that.

---
 target/s390x/mem_helper.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
target/s390x/mem_helper.c +16 -7
index 4c67c6f37e..73b00b582b 100644
@@ -845,21 +845,30 @@ uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2)
 {
     const uint64_t d = get_address(env, r1);
     const uint64_t s = get_address(env, r2);
+    const int len = MIN(-(d | TARGET_PAGE_MASK), -(s | TARGET_PAGE_MASK));
+    S390Access srca, desta;
     uintptr_t ra = GETPC();
-    uint32_t len;
+    int i;
 
     if (c & 0xffffff00ull) {
         s390_program_interrupt(env, PGM_SPECIFICATION, ILEN_AUTO, ra);
     }
     c = c & 0xff;
 
-    /* Lest we fail to service interrupts in a timely manner, limit the
-       amount of work we're willing to do.  For now, let's cap at 8k.  */
-    for (len = 0; len < 0x2000; ++len) {
-        uint8_t v = cpu_ldub_data_ra(env, s + len, ra);
-        cpu_stb_data_ra(env, d + len, v, ra);
+    /*
+     * Our access should not exceed single pages, as we must not report access
+     * exceptions exceeding the actually copied range (which we don't know at
+     * this point). We might over-indicate watchpoints within the pages
+     * (if we ever care, we have to limit processing to a single byte).
+     */
+    srca = access_prepare(env, s, len, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, d, len, MMU_DATA_STORE, ra);
+    for (i = 0; i < len; i++) {
+        const uint8_t v = access_get_byte(env, &srca, i, ra);
+
+        access_set_byte(env, &desta, i, v, ra);
         if (v == c) {
-            set_address_zero(env, r1, d + len);
+            set_address_zero(env, r1, d + i);
             return 1;
         }
     }
Signature · 1 line
2.21.0
Reviewed-by on v2 26/28
#62 lore raw

Re: [Qemu-devel] [PATCH v2 26/28] s390x/tcg: MVST: Fault-safe handling

Quoted 7 lines · David Hildenbrand
target/s390x/mem_helper.c +7 -0
+    /*
+     * Our access should not exceed single pages, as we must not report access
+     * exceptions exceeding the actually copied range (which we don't know at
+     * this point). We might over-indicate watchpoints within the pages
+     * (if we ever care, we have to limit processing to a single byte).
+     */
+    srca = access_prepare(env, s, len, MMU_DATA_LOAD, ra);

target/s390x/mem_helper.c uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2) · line 873

Quoted 10 lines
target/s390x/mem_helper.c +6 -1
+    desta = access_prepare(env, d, len, MMU_DATA_STORE, ra);
+    for (i = 0; i < len; i++) {
+        const uint8_t v = access_get_byte(env, &srca, i, ra);
+
+        access_set_byte(env, &desta, i, v, ra);
         if (v == c) {
-            set_address_zero(env, r1, d + len);
+            set_address_zero(env, r1, d + i);
             return 1;
         }

Worth using memchr + memmove w/ nondestructive overlap?

That said,

r~

#63 lore raw

Re: [Qemu-devel] [PATCH v2 26/28] s390x/tcg: MVST: Fault-safe handling

Quoted 19 lines · Richard Henderson
Quoted 7 lines · David Hildenbrand
target/s390x/mem_helper.c +7 -0
+    /*
+     * Our access should not exceed single pages, as we must not report access
+     * exceptions exceeding the actually copied range (which we don't know at
+     * this point). We might over-indicate watchpoints within the pages
+     * (if we ever care, we have to limit processing to a single byte).
+     */
+    srca = access_prepare(env, s, len, MMU_DATA_LOAD, ra);

target/s390x/mem_helper.c uint32_t HELPER(mvst)(CPUS390XState *env, uint64_t c, uint32_t r1, uint32_t r2) · line 873

Quoted 10 lines
target/s390x/mem_helper.c +6 -1
+    desta = access_prepare(env, d, len, MMU_DATA_STORE, ra);
+    for (i = 0; i < len; i++) {
+        const uint8_t v = access_get_byte(env, &srca, i, ra);
+
+        access_set_byte(env, &desta, i, v, ra);
         if (v == c) {
-            set_address_zero(env, r1, d + len);
+            set_address_zero(env, r1, d + i);
             return 1;
         }
Quoted 1 line

Worth using memchr + memmove w/ nondestructive overlap?

In theory yes, however, the issue is that we would have multiple accesses, which is not documented for this instruction. In case the memory is modified between memchr + memmove by another CPU, we could have an inconsistent instruction result. Unlikely but possible :)

Quoted 6 lines

That said,

r~

Signature · 3 lines
Thanks,

David / dhildenb
#64 lore raw

PATCH v2 27/28 s390x/tcg: MVO: Fault-safe handling

In reply to David Hildenbrand

Each operand can have a maximum length of 16. Make sure to prepare all reads/writes before writing.

---
 target/s390x/mem_helper.c | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)
target/s390x/mem_helper.c +14 -12
index 73b00b582b..7403124763 100644
@@ -514,31 +514,33 @@ void HELPER(mvn)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 /* move with offset  */
 void HELPER(mvo)(CPUS390XState *env, uint32_t l, uint64_t dest, uint64_t src)
 {
+    /* MVO always processes one more byte than specified - maximum is 16 */
+    const int len_dest = (l >> 4) + 1;
+    const int len_src = (l & 0xf) + 1;
     uintptr_t ra = GETPC();
-    int len_dest = l >> 4;
-    int len_src = l & 0xf;
     uint8_t byte_dest, byte_src;
-    int i;
+    S390Access srca, desta;
+    int i, j;
 
-    src += len_src;
-    dest += len_dest;
+    srca = access_prepare(env, src, len_src, MMU_DATA_LOAD, ra);
+    desta = access_prepare(env, dest, len_dest, MMU_DATA_STORE, ra);
 
     /* Handle rightmost byte */
-    byte_src = cpu_ldub_data_ra(env, src, ra);
-    byte_dest = cpu_ldub_data_ra(env, dest, ra);
+    byte_dest = cpu_ldub_data_ra(env, dest + len_dest - 1, ra);
+    byte_src = access_get_byte(env, &srca, len_src - 1, ra);
     byte_dest = (byte_dest & 0x0f) | (byte_src << 4);
-    cpu_stb_data_ra(env, dest, byte_dest, ra);
+    access_set_byte(env, &desta, len_dest - 1, byte_dest, ra);
 
     /* Process remaining bytes from right to left */
-    for (i = 1; i <= len_dest; i++) {
+    for (i = len_dest - 2, j = len_src - 2; i >= 0; i--, j--) {
         byte_dest = byte_src >> 4;
-        if (len_src - i >= 0) {
-            byte_src = cpu_ldub_data_ra(env, src - i, ra);
+        if (j >= 0) {
+            byte_src = access_get_byte(env, &srca, j, ra);
         } else {
             byte_src = 0;
         }
         byte_dest |= byte_src << 4;
-        cpu_stb_data_ra(env, dest - i, byte_dest, ra);
+        access_set_byte(env, &desta, i, byte_dest, ra);
     }
 }
 
Signature · 1 line
2.21.0
#66 lore raw

PATCH v2 28/28 tests/tcg: target/s390x: Test MVO

In reply to David Hildenbrand

Let's add the simple test based on the example from the PoP.

---
 tests/tcg/s390x/Makefile.target |  1 +
 tests/tcg/s390x/mvo.c           | 25 +++++++++++++++++++++++++
 2 files changed, 26 insertions(+)
 create mode 100644 tests/tcg/s390x/mvo.c
tests/tcg/s390x/Makefile.target +1 -0
index 151dc075aa..6a3bfa8b29 100644
@@ -6,3 +6,4 @@ TESTS+=ipm
 TESTS+=exrl-trt
 TESTS+=exrl-trtr
 TESTS+=pack
+TESTS+=mvo
tests/tcg/s390x/mvo.c new file +25 -0
new file mode 100644
index 0000000000..5546fe2a97
@@ -0,0 +1,25 @@
+#include <stdint.h>
+#include <stdio.h>
+
+int main(void)
+{
+    uint8_t dest[6] = {0xff, 0x77, 0x88, 0x99, 0x0c, 0xff};
+    uint8_t src[5] = {0xee, 0x12, 0x34, 0x56, 0xee};
+    uint8_t expected[6] = {0xff, 0x01, 0x23, 0x45, 0x6c, 0xff};
+    int i;
+
+    asm volatile (
+        "    mvo 0(4,%[dest]),0(3,%[src])\n"
+        :
+        : [dest] "d" (dest + 1),
+          [src] "d" (src + 1)
+        : "memory");
+
+    for (i = 0; i < sizeof(expected); i++) {
+        if (dest[i] != expected[i]) {
+            fprintf(stderr, "bad data\n");
+            return 1;
+        }
+    }
+    return 0;
+}
Signature · 1 line
2.21.0
#67 lore raw
In reply to David Hildenbrand
Quoted 72 lines · David Hildenbrand

This is the successor of

    "[PATCH v1 0/4] s390x/tcg: MOVE (MVC): Fault-safe handling"

----

This series fixes a bunch of issues related to some mem helpers and makes sure that they are fault-safe, meaning no system state is modified in case a fault is triggered.

I can spot tons of other issues with other mem helpers that will have to be fixed later. Also, fault-safe handling for some instructions (especially TR) might be harder to implement (you don't know what will actually be accessed upfront - we might need a buffer and go over inputs twice). Focusing on the MOVE instructions for now.

----

Newer versions of glibc use memcpy() in memmove() for forward moves. The implementation makese use of MVC. The TCG implementation of MVC is currently not able to handle faults reliably when crossing pages. MVC can cross with 256 bytes at most two pages.

In case we get a fault on the second page, we already moved data. When continuing after the fault we might try to move already overwritten data, which is very bad in case we have overlapping data on a forward move.

Triggered for now only by rpmbuild (crashes when checking the spec file) and rpm (database corruptions). This fixes installing Fedora rawhide (31) under TCG.

This was horrible to debug as it barely triggers and we fail at completely different places.

Cc Stefano Brivio <[email protected]>
Cc Florian Weimer <[email protected]>

v1 -> v2:
- Include many fixes
- Fix more instructions
- Use the new probe_access() function
- Include "tests/tcg: target/s390x: Test MVO"

David Hildenbrand (28):
s390x/tcg: Reset exception_index to -1 instead of 0 s390x/tcg: MVCL: Zero out unused bits of address s390x/tcg: MVCL: Detect destructive overlaps s390x/tcg: MVCL: Process max 2k bytes at a time s390x/tcg: MVC: Increment the length once s390x/tcg: MVC: Use is_destructive_overlap() s390x/tcg: MVPG: Check for specification exceptions s390x/tcg: MVPG: Properly wrap the addresses s390x/tcg: MVCLU/MVCLE: Process max 4k bytes at a time s390x/tcg: MVCS/MVCP: Check for special operation exceptions s390x/tcg: MVCS/MVCP: Properly wrap the length s390x/tcg: MVST: Check for specification exceptions s390x/tcg: MVST: Fix storing back the addresses to registers s390x/tcg: Always use MMU_USER_IDX for CONFIG_USER_ONLY s390x/tcg: Fault-safe memset
s390x/tcg: Fault-safe memmove
s390x/tcg: MVCS/MVCP: Use access_memmove_idx() s390x/tcg: MVC: Fault-safe handling on destructive overlaps s390x/tcg: MVCLU: Fault-safe handling
s390x/tcg: OC: Fault-safe handling
s390x/tcg: XC: Fault-safe handling
s390x/tcg: NC: Fault-safe handling
s390x/tcg: MVCIN: Fault-safe handling
s390x/tcg: MVN: Fault-safe handling
s390x/tcg: MVZ: Fault-safe handling
s390x/tcg: MVST: Fault-safe handling
s390x/tcg: MVO: Fault-safe handling
tests/tcg: target/s390x: Test MVO

Quoted 10 lines
 target/s390x/cpu.h              |   4 +
 target/s390x/helper.h           |   2 +-
 target/s390x/insn-data.def      |   2 +-
 target/s390x/mem_helper.c       | 672 ++++++++++++++++++++++----------
 target/s390x/translate.c        |  12 +-
 tests/tcg/s390x/Makefile.target |   1 +
 tests/tcg/s390x/mvo.c           |  25 ++
 7 files changed, 507 insertions(+), 211 deletions(-)
 create mode 100644 tests/tcg/s390x/mvo.c

I guess, for a minimal review besides the obvious fixes, looking at

"s390x/tcg: Fault-safe memset" and "s390x/tcg: Fault-safe memmove" and "s390x/tcg: MVC: Fault-safe handling on destructive overlaps" makes sense. They contain the real magic used within remaining patches.

Signature · 3 lines
Thanks,

David / dhildenb