Skip to contentSkip to conversation

s390x/tcg: STFLE fixes

PATCH In review David Hildenbrand 4 participants 12 messages May 31, 2019 – Jun 4, 2019

lore
Share & tools
lore
#0 lore raw

cover s390x/tcg: STFLE fixes

While testing vector instructions, I ran into various issues with user space binaries compiled with more recent compiler versions like

# gunzip /usr/share/man/man1/hexdump.1.gz
<dhildenb> *** stack smashing detected ***: <unknown> terminated

Turns out:
a) the STFLE instruction in semi-broken on the first invocation
b) the code expects a different STFLE behavior than documented in the PoP

Fix a) and make sure the code works by adjusting b).

David Hildenbrand (2):
s390x/tcg: Fix max_byte detection for stfle
s390x/tcg: Store only the necessary amount of doublewords for STFLE

 target/s390x/misc_helper.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/2 s390x/tcg: Fix max_byte detection for stfle

used_stfl_bytes() is 0, before initialized via prepare_stfl() on the first invocation. We have to move the calculation of max_bytes after prepare_stfl().

---
 target/s390x/misc_helper.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
target/s390x/misc_helper.c +2 -1
index ee67c1fa0c..34476134a4 100644
@@ -669,7 +669,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
 {
     const uintptr_t ra = GETPC();
     const int count_bytes = ((env->regs[0] & 0xff) + 1) * 8;
-    const int max_bytes = ROUND_UP(used_stfl_bytes, 8);
+    int max_bytes;
     int i;
 
     if (addr & 0x7) {
@@ -677,6 +677,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
     }
 
     prepare_stfl();
+    max_bytes = ROUND_UP(used_stfl_bytes, 8);
     for (i = 0; i < count_bytes; ++i) {
         cpu_stb_data_ra(env, addr + i, stfl_bytes[i], ra);
     }
Signature · 1 line
2.20.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH v1 1/2] s390x/tcg: Fix max_byte detection for stfle

Quoted 8 lines · David Hildenbrand

used_stfl_bytes() is 0, before initialized via prepare_stfl() on the first invocation. We have to move the calculation of max_bytes after prepare_stfl().

---
 target/s390x/misc_helper.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

r~

#3 lore raw

PATCH 2/2 s390x/tcg: Store only the necessary amount of doublewords for STFLE

In reply to David Hildenbrand

The PoP (z14, 7-382) says:
Doublewords to the right of the doubleword in which the highest-numbered facility bit is assigned for a model may or may not be stored.

However, stack protection in certain binaries can't deal with that. "gzip" example code:

f1b4:       a7 08 00 03             lhi     %r0,3
f1b8:       b2 b0 f0 a0             stfle   160(%r15)
f1bc:       e3 20 f0 b2 00 90       llgc    %r2,178(%r15)
f1c2:       c0 2b 00 00 00 01       nilf    %r2,1
f1c8:       b2 4f 00 10             ear     %r1,%a0
f1cc:       b9 14 00 22             lgfr    %r2,%r2
f1d0:       eb 11 00 20 00 0d       sllg    %r1,%r1,32
f1d6:       b2 4f 00 11             ear     %r1,%a1
f1da:       d5 07 f0 b8 10 28       clc     184(8,%r15),40(%r1)
f1e0:       a7 74 00 06             jne     f1ec <file_read@@Base+0x1bc>
f1e4:       eb ef f1 30 00 04       lmg     %r14,%r15,304(%r15)
f1ea:       07 fe                   br      %r14
f1ec:       c0 e5 ff ff 9d 6e       brasl   %r14,2cc8 <__stack_chk_fail@plt>

In QEMU, we currently have:

    max_bytes = 24

the code asks for (3 + 1) doublewords == 32 bytes.

If we write 32 bytes instead of only 24, and return "2 + 1" doublewords ("one less than the number of doulewords needed to contain all of the
facility bits"), the example code detects a stack corruption.

In my opinion, the code is wrong. However, it seems to work fine on real machines. So let's limit storing to the minimum of the requested and the maximum doublewords.

Cc Stefan Liebler <[email protected]>
Cc Andreas Krebbel <[email protected]>
---
 target/s390x/misc_helper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
target/s390x/misc_helper.c +1 -1
index 34476134a4..b561c5781b 100644
@@ -678,7 +678,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
 
     prepare_stfl();
     max_bytes = ROUND_UP(used_stfl_bytes, 8);
-    for (i = 0; i < count_bytes; ++i) {
+    for (i = 0; i < MIN(count_bytes, max_bytes); ++i) {
         cpu_stb_data_ra(env, addr + i, stfl_bytes[i], ra);
     }
 
Signature · 1 line
2.20.1
Reviewed-by on v1 2/2
#4 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

target/s390x/misc_helper.c uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr) · line 681

Quoted 6 lines · David Hildenbrand
target/s390x/misc_helper.c +1 -1
@@ -678,7 +678,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
 
     prepare_stfl();
     max_bytes = ROUND_UP(used_stfl_bytes, 8);
-    for (i = 0; i < count_bytes; ++i) {
+    for (i = 0; i < MIN(count_bytes, max_bytes); ++i) {

Before the loop, please put something like

/*
* The PoP says that doublewords beyond the highest-numbered
* facility bit may or may not be stored. However, existing
* hardware appears to not store the words, and existing
* software appears to depend on that.
*/

with that,

r~

#5 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

Quoted 17 lines · Richard Henderson

target/s390x/misc_helper.c uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr) · line 681

Quoted 6 lines · David Hildenbrand
target/s390x/misc_helper.c +1 -1
@@ -678,7 +678,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
 
     prepare_stfl();
     max_bytes = ROUND_UP(used_stfl_bytes, 8);
-    for (i = 0; i < count_bytes; ++i) {
+    for (i = 0; i < MIN(count_bytes, max_bytes); ++i) {

Before the loop, please put something like

/*
* The PoP says that doublewords beyond the highest-numbered
* facility bit may or may not be stored. However, existing
* hardware appears to not store the words, and existing
* software appears to depend on that.
*/

Quoted 3 lines

with that,

Makes sense, thanks for the ultra-fast review!

(have a great weekend!)

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

In reply to David Hildenbrand
Quoted 26 lines · David Hildenbrand

The PoP (z14, 7-382) says:
Doublewords to the right of the doubleword in which the highest-numbered facility bit is assigned for a model may or may not be stored.

However, stack protection in certain binaries can't deal with that. "gzip" example code:

f1b4:       a7 08 00 03             lhi     %r0,3
f1b8:       b2 b0 f0 a0             stfle   160(%r15)
f1bc:       e3 20 f0 b2 00 90       llgc    %r2,178(%r15)
f1c2:       c0 2b 00 00 00 01       nilf    %r2,1
f1c8:       b2 4f 00 10             ear     %r1,%a0
f1cc:       b9 14 00 22             lgfr    %r2,%r2
f1d0:       eb 11 00 20 00 0d       sllg    %r1,%r1,32
f1d6:       b2 4f 00 11             ear     %r1,%a1
f1da:       d5 07 f0 b8 10 28       clc     184(8,%r15),40(%r1)
f1e0:       a7 74 00 06             jne     f1ec <file_read@@Base+0x1bc>
f1e4:       eb ef f1 30 00 04       lmg     %r14,%r15,304(%r15)
f1ea:       07 fe                   br      %r14
f1ec:       c0 e5 ff ff 9d 6e       brasl   %r14,2cc8 <__stack_chk_fail@plt>

In QEMU, we currently have:

     max_bytes = 24

the code asks for (3 + 1) doublewords == 32 bytes.

Quoted 7 lines

If we write 32 bytes instead of only 24, and return "2 + 1" doublewords ("one less than the number of doulewords needed to contain all of the
facility bits"), the example code detects a stack corruption.

In my opinion, the code is wrong. However, it seems to work fine on real machines. So let's limit storing to the minimum of the requested and the maximum doublewords.

Hi David,

Thanks for catching this. I've reported the "gzip" example to Ilya and indeed, r0 is setup too large. He will fix it in gzip.

You've mentioned, that this is detected in certain binaries.
Can you please share those occurrences.
Perhaps on future machines with further facility bits, stfle will write beyond the provided doubleword-array on a real machine or in qemu.

Bye
Stefan

Quoted 22 lines
Cc Stefan Liebler <[email protected]>
Cc Andreas Krebbel <[email protected]>
---
  target/s390x/misc_helper.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
target/s390x/misc_helper.c +1 -1
index 34476134a4..b561c5781b 100644
@@ -678,7 +678,7 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
 
      prepare_stfl();
      max_bytes = ROUND_UP(used_stfl_bytes, 8);
-    for (i = 0; i < count_bytes; ++i) {
+    for (i = 0; i < MIN(count_bytes, max_bytes); ++i) {
          cpu_stb_data_ra(env, addr + i, stfl_bytes[i], ra);
      }
 
#7 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

Quoted 34 lines · Stefan Liebler
Quoted 26 lines · David Hildenbrand

The PoP (z14, 7-382) says:
Doublewords to the right of the doubleword in which the highest-numbered facility bit is assigned for a model may or may not be stored.

However, stack protection in certain binaries can't deal with that. "gzip" example code:

f1b4:       a7 08 00 03             lhi     %r0,3
f1b8:       b2 b0 f0 a0             stfle   160(%r15)
f1bc:       e3 20 f0 b2 00 90       llgc    %r2,178(%r15)
f1c2:       c0 2b 00 00 00 01       nilf    %r2,1
f1c8:       b2 4f 00 10             ear     %r1,%a0
f1cc:       b9 14 00 22             lgfr    %r2,%r2
f1d0:       eb 11 00 20 00 0d       sllg    %r1,%r1,32
f1d6:       b2 4f 00 11             ear     %r1,%a1
f1da:       d5 07 f0 b8 10 28       clc     184(8,%r15),40(%r1)
f1e0:       a7 74 00 06             jne     f1ec <file_read@@Base+0x1bc>
f1e4:       eb ef f1 30 00 04       lmg     %r14,%r15,304(%r15)
f1ea:       07 fe                   br      %r14
f1ec:       c0 e5 ff ff 9d 6e       brasl   %r14,2cc8 <__stack_chk_fail@plt>

In QEMU, we currently have:

     max_bytes = 24

the code asks for (3 + 1) doublewords == 32 bytes.

Quoted 7 lines

If we write 32 bytes instead of only 24, and return "2 + 1" doublewords ("one less than the number of doulewords needed to contain all of the
facility bits"), the example code detects a stack corruption.

In my opinion, the code is wrong. However, it seems to work fine on real machines. So let's limit storing to the minimum of the requested and the maximum doublewords.

Quoted 7 lines

Hi David,

Thanks for catching this. I've reported the "gzip" example to Ilya and indeed, r0 is setup too large. He will fix it in gzip.

You've mentioned, that this is detected in certain binaries. Can you please share those occurrences.

Hi Stafan,

thanks for your reply.

I didn't track all occurrences, it *could* be that it was only gzip in the background making other processes fail.

For example, the systemd "vitual console setup" unit failed, too, which was fixed by this change.

I also remember, seeing segfaults in rpmbuild, for example. They only "changed" with this fix - I m still chasing different errors. :)

You mentioned "He will fix it in gzip", so I assume this is a gzip issue and not a gcc/glibc/whatever toolchain issue?

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

Quoted 45 lines · David Hildenbrand
Quoted 34 lines · Stefan Liebler
Quoted 26 lines · David Hildenbrand

The PoP (z14, 7-382) says:
Doublewords to the right of the doubleword in which the highest-numbered facility bit is assigned for a model may or may not be stored.

However, stack protection in certain binaries can't deal with that. "gzip" example code:

f1b4:       a7 08 00 03             lhi     %r0,3
f1b8:       b2 b0 f0 a0             stfle   160(%r15)
f1bc:       e3 20 f0 b2 00 90       llgc    %r2,178(%r15)
f1c2:       c0 2b 00 00 00 01       nilf    %r2,1
f1c8:       b2 4f 00 10             ear     %r1,%a0
f1cc:       b9 14 00 22             lgfr    %r2,%r2
f1d0:       eb 11 00 20 00 0d       sllg    %r1,%r1,32
f1d6:       b2 4f 00 11             ear     %r1,%a1
f1da:       d5 07 f0 b8 10 28       clc     184(8,%r15),40(%r1)
f1e0:       a7 74 00 06             jne     f1ec <file_read@@Base+0x1bc>
f1e4:       eb ef f1 30 00 04       lmg     %r14,%r15,304(%r15)
f1ea:       07 fe                   br      %r14
f1ec:       c0 e5 ff ff 9d 6e       brasl   %r14,2cc8 <__stack_chk_fail@plt>

In QEMU, we currently have:

      max_bytes = 24

the code asks for (3 + 1) doublewords == 32 bytes.

Quoted 7 lines

If we write 32 bytes instead of only 24, and return "2 + 1" doublewords ("one less than the number of doulewords needed to contain all of the
facility bits"), the example code detects a stack corruption.

In my opinion, the code is wrong. However, it seems to work fine on real machines. So let's limit storing to the minimum of the requested and the maximum doublewords.

Quoted 7 lines

Hi David,

Thanks for catching this. I've reported the "gzip" example to Ilya and indeed, r0 is setup too large. He will fix it in gzip.

You've mentioned, that this is detected in certain binaries. Can you please share those occurrences.

Hi Stafan,

Quoted 7 lines

thanks for your reply.

I didn't track all occurrences, it *could* be that it was only gzip in the background making other processes fail.

For example, the systemd "vitual console setup" unit failed, too, which was fixed by this change.

At least "objdump -d /usr/lib/systemd/systemd-vconsole-setup" does not contain the stfle instruction, but "ldd /usr/lib/systemd/systemd-vconsole-setup" is showing libz.so which could also contain Ilya's patches with the stfle instruction (I assume there is the same bug as in gzip). But I have no idea if libz is really called.

Quoted 3 lines

I also remember, seeing segfaults in rpmbuild, for example. They only "changed" with this fix - I m still chasing different errors. :)

The same applies for rpmbuild.

Quoted 4 lines

You mentioned "He will fix it in gzip", so I assume this is a gzip issue and not a gcc/glibc/whatever toolchain issue?

Yes, this is a gzip bug. r0 was initialized with:
(sizeof(array-on-stack) / 8)
instead of:
(sizeof(array-on-stack) / 8) - 1

Ilya will fix it in gzip and zlib.
@Ilya: Please correct me if I'm wrong.

#9 lore raw

Re: [Qemu-devel] [PATCH v1 2/2] s390x/tcg: Store only the necessary amount of doublewords for STFLE

Quoted 66 lines · Stefan Liebler
Quoted 45 lines · David Hildenbrand
Quoted 34 lines · Stefan Liebler
Quoted 26 lines · David Hildenbrand

The PoP (z14, 7-382) says:
Doublewords to the right of the doubleword in which the highest-numbered facility bit is assigned for a model may or may not be stored.

However, stack protection in certain binaries can't deal with that. "gzip" example code:

f1b4:       a7 08 00 03             lhi     %r0,3
f1b8:       b2 b0 f0 a0             stfle   160(%r15)
f1bc:       e3 20 f0 b2 00 90       llgc    %r2,178(%r15)
f1c2:       c0 2b 00 00 00 01       nilf    %r2,1
f1c8:       b2 4f 00 10             ear     %r1,%a0
f1cc:       b9 14 00 22             lgfr    %r2,%r2
f1d0:       eb 11 00 20 00 0d       sllg    %r1,%r1,32
f1d6:       b2 4f 00 11             ear     %r1,%a1
f1da:       d5 07 f0 b8 10 28       clc     184(8,%r15),40(%r1)
f1e0:       a7 74 00 06             jne     f1ec <file_read@@Base+0x1bc>
f1e4:       eb ef f1 30 00 04       lmg     %r14,%r15,304(%r15)
f1ea:       07 fe                   br      %r14
f1ec:       c0 e5 ff ff 9d 6e       brasl   %r14,2cc8 <__stack_chk_fail@plt>

In QEMU, we currently have:

      max_bytes = 24

the code asks for (3 + 1) doublewords == 32 bytes.

Quoted 7 lines

If we write 32 bytes instead of only 24, and return "2 + 1" doublewords ("one less than the number of doulewords needed to contain all of the
facility bits"), the example code detects a stack corruption.

In my opinion, the code is wrong. However, it seems to work fine on real machines. So let's limit storing to the minimum of the requested and the maximum doublewords.

Quoted 7 lines

Hi David,

Thanks for catching this. I've reported the "gzip" example to Ilya and indeed, r0 is setup too large. He will fix it in gzip.

You've mentioned, that this is detected in certain binaries. Can you please share those occurrences.

Hi Stafan,

Quoted 7 lines

thanks for your reply.

I didn't track all occurrences, it *could* be that it was only gzip in the background making other processes fail.

For example, the systemd "vitual console setup" unit failed, too, which was fixed by this change.

At least "objdump -d /usr/lib/systemd/systemd-vconsole-setup" does not contain the stfle instruction, but "ldd /usr/lib/systemd/systemd-vconsole-setup" is showing libz.so which could also contain Ilya's patches with the stfle instruction (I assume there is the same bug as in gzip). But I have no idea if libz is really called.

Quoted 3 lines

I also remember, seeing segfaults in rpmbuild, for example. They only "changed" with this fix - I m still chasing different errors. :)

The same applies for rpmbuild.

Quoted 4 lines

You mentioned "He will fix it in gzip", so I assume this is a gzip issue and not a gcc/glibc/whatever toolchain issue?

Quoted 8 lines

Yes, this is a gzip bug. r0 was initialized with:
(sizeof(array-on-stack) / 8)
instead of:
(sizeof(array-on-stack) / 8) - 1

Ilya will fix it in gzip and zlib.
@Ilya: Please correct me if I'm wrong.

Makes sense, thanks for the explanation!

Signature · 3 lines
Thanks,

David / dhildenb
#10 lore raw
In reply to David Hildenbrand
Quoted 12 lines · On Fri, 31 May 2019 16:56:06 +0200 David Hildenbrand <[email protected]> wrote:

While testing vector instructions, I ran into various issues with user space binaries compiled with more recent compiler versions like

# gunzip /usr/share/man/man1/hexdump.1.gz
<dhildenb> *** stack smashing detected ***: <unknown> terminated

Turns out:
a) the STFLE instruction in semi-broken on the first invocation
b) the code expects a different STFLE behavior than documented in the PoP

Fix a) and make sure the code works by adjusting b).

So your problems actually did not have anything to do with vector instructions and were simply exposed by running binaries compiled with a more recent compiler version, right? Interesting :)

Quoted 7 lines

David Hildenbrand (2):
s390x/tcg: Fix max_byte detection for stfle
s390x/tcg: Store only the necessary amount of doublewords for STFLE

 target/s390x/misc_helper.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Do you want to include these in a pull request, or should I pick them up myself?

#11 lore raw
Quoted 13 lines · Cornelia Huck
Quoted 10 lines · On Fri, 31 May 2019 16:56:06 +0200 David Hildenbrand <[email protected]> wrote:

While testing vector instructions, I ran into various issues with user space binaries compiled with more recent compiler versions like

# gunzip /usr/share/man/man1/hexdump.1.gz
<dhildenb> *** stack smashing detected ***: <unknown> terminated

Turns out:
a) the STFLE instruction in semi-broken on the first invocation
b) the code expects a different STFLE behavior than documented in the PoP

Quoted 6 lines
Quoted 2 lines

Fix a) and make sure the code works by adjusting b).

So your problems actually did not have anything to do with vector instructions and were simply exposed by running binaries compiled with a more recent compiler version, right? Interesting :)

Yes, but it's not the complete story yet. There are still two BUGs somewhere in emulation code (again, could be !vector register related). These seem to be harder to track down (e.g., rpm database corruptions when installing a lot of packages at once).

Quoted 5 lines
Quoted 5 lines

David Hildenbrand (2):
s390x/tcg: Fix max_byte detection for stfle
s390x/tcg: Store only the necessary amount of doublewords for STFLE

Quoted 7 lines
Quoted 3 lines
 target/s390x/misc_helper.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Do you want to include these in a pull request, or should I pick them up myself?

I can include them.

Signature · 3 lines
Thanks,

David / dhildenb