Skip to contentSkip to conversation

accel/tcg: Fix unaligned stores to s390x low-address-protected lowcore

PATCH Superseded Ilya Leoshkevich 3 participants 11 messages Jul 11, 2022 – Jul 12, 2022 Arm/Amlogic Meson SoC support

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover accel/tcg: Fix unaligned stores to s390x low-address-protected lowcore

Hi,

I noticed that certain accesses to lowcore incorrectly trigger protection exceptions. I tracked it down to store_helper_unaligned() calling tlb_fill() with ranges like [0, 2000).

Patch 1 fixes the issue, patch 2 adds a new MMIO device that enables writing system tests for s390x, patch 3 adds a system test for this issue.

Best regards,
Ilya

Ilya Leoshkevich (3):
accel/tcg: Fix unaligned stores to s390x low-address-protected lowcore hw/misc: Add mmio-debug-exit device
tests/tcg/s390x: Test unaligned accesses to lowcore

 accel/tcg/cputlb.c                      |  8 ++-
 hw/misc/Kconfig                         |  3 +
 hw/misc/debugexit_mmio.c                | 80 +++++++++++++++++++++++++
 hw/misc/meson.build                     |  1 +
 hw/s390x/Kconfig                        |  1 +
 tests/tcg/s390x/Makefile.softmmu-target |  9 +++
 tests/tcg/s390x/unaligned-lowcore.S     | 24 ++++++++
 7 files changed, 123 insertions(+), 3 deletions(-)
 create mode 100644 hw/misc/debugexit_mmio.c
 create mode 100644 tests/tcg/s390x/Makefile.softmmu-target
 create mode 100644 tests/tcg/s390x/unaligned-lowcore.S
Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/3 accel/tcg: Fix unaligned stores to s390x low-address-protected lowcore

If low-address-protection is active, unaligned stores to non-protected parts of lowcore lead to protection exceptions. The reason is that in such cases tlb_fill() call in store_helper_unaligned() covers [0, addr + size) range, which contains the protected portion of lowcore. This range is too large.

The most straightforward fix would be to make sure we stay within the original [addr, addr + size) range. However, if an unaligned access affects a single page, we don't need to call tlb_fill() in store_helper_unaligned() at all, since it would be identical to the previous tlb_fill() call in store_helper(), and therefore a no-op. If an unaligned access covers multiple pages, this situation does not occur.

Therefore simply skip TLB handling in store_helper_unaligned() if we are dealing with a single page.

Fixes 2bcf018340cb ("s390x/tcg: low-address protection support")
---
 accel/tcg/cputlb.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
accel/tcg/cputlb.c +5 -3
index f90f4312ea..a46f3a654d 100644
@@ -2248,7 +2248,7 @@ store_helper_unaligned(CPUArchState *env, target_ulong addr, uint64_t val,
     const size_t tlb_off = offsetof(CPUTLBEntry, addr_write);
     uintptr_t index, index2;
     CPUTLBEntry *entry, *entry2;
-    target_ulong page2, tlb_addr, tlb_addr2;
+    target_ulong page1, page2, tlb_addr, tlb_addr2;
     MemOpIdx oi;
     size_t size2;
     int i;
@@ -2256,15 +2256,17 @@ store_helper_unaligned(CPUArchState *env, target_ulong addr, uint64_t val,
     /*
      * Ensure the second page is in the TLB.  Note that the first page
      * is already guaranteed to be filled, and that the second page
-     * cannot evict the first.
+     * cannot evict the first.  An exception to this rule is PAGE_WRITE_INV
+     * handling: the first page could have evicted itself.
      */
+    page1 = addr & TARGET_PAGE_MASK;
     page2 = (addr + size) & TARGET_PAGE_MASK;
     size2 = (addr + size) & ~TARGET_PAGE_MASK;
     index2 = tlb_index(env, mmu_idx, page2);
     entry2 = tlb_entry(env, mmu_idx, page2);
 
     tlb_addr2 = tlb_addr_write(entry2);
-    if (!tlb_hit_page(tlb_addr2, page2)) {
+    if (page1 != page2 && !tlb_hit_page(tlb_addr2, page2)) {
         if (!victim_tlb_hit(env, mmu_idx, index2, tlb_off, page2)) {
             tlb_fill(env_cpu(env), page2, size2, MMU_DATA_STORE,
                      mmu_idx, retaddr);
Signature · 1 line
2.35.3
Reviewed-by on v1 1/3
#2 lore raw
Quoted 14 lines · Ilya Leoshkevich

If low-address-protection is active, unaligned stores to non-protected parts of lowcore lead to protection exceptions. The reason is that in such cases tlb_fill() call in store_helper_unaligned() covers [0, addr + size) range, which contains the protected portion of lowcore. This range is too large.

The most straightforward fix would be to make sure we stay within the original [addr, addr + size) range. However, if an unaligned access affects a single page, we don't need to call tlb_fill() in store_helper_unaligned() at all, since it would be identical to the previous tlb_fill() call in store_helper(), and therefore a no-op. If an unaligned access covers multiple pages, this situation does not occur.

Quoted 5 lines

Therefore simply skip TLB handling in store_helper_unaligned() if we are dealing with a single page.

Fixes 2bcf018340cb ("s390x/tcg: low-address protection support")

Queueing to tcg-next. I'll let the test case go through s390x.

r~

Reviewed-by on v1 1/3
#3 lore raw
In reply to Ilya Leoshkevich
Quoted 46 lines · Ilya Leoshkevich

If low-address-protection is active, unaligned stores to non-protected parts of lowcore lead to protection exceptions. The reason is that in such cases tlb_fill() call in store_helper_unaligned() covers [0, addr + size) range, which contains the protected portion of lowcore. This range is too large.

The most straightforward fix would be to make sure we stay within the original [addr, addr + size) range. However, if an unaligned access affects a single page, we don't need to call tlb_fill() in store_helper_unaligned() at all, since it would be identical to the previous tlb_fill() call in store_helper(), and therefore a no-op. If an unaligned access covers multiple pages, this situation does not occur.

Therefore simply skip TLB handling in store_helper_unaligned() if we are dealing with a single page.

Fixes 2bcf018340cb ("s390x/tcg: low-address protection support")
---
 accel/tcg/cputlb.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
accel/tcg/cputlb.c +4 -2
index f90f4312ea..a46f3a654d 100644
@@ -2248,7 +2248,7 @@ store_helper_unaligned(CPUArchState *env, target_ulong addr, uint64_t val,
     const size_t tlb_off = offsetof(CPUTLBEntry, addr_write);
     uintptr_t index, index2;
     CPUTLBEntry *entry, *entry2;
-    target_ulong page2, tlb_addr, tlb_addr2;
+    target_ulong page1, page2, tlb_addr, tlb_addr2;
     MemOpIdx oi;
     size_t size2;
     int i;
@@ -2256,15 +2256,17 @@ store_helper_unaligned(CPUArchState *env, target_ulong addr, uint64_t val,
     /*
      * Ensure the second page is in the TLB.  Note that the first page
      * is already guaranteed to be filled, and that the second page
-     * cannot evict the first.
+     * cannot evict the first.  An exception to this rule is PAGE_WRITE_INV
+     * handling: the first page could have evicted itself.
      */
+    page1 = addr & TARGET_PAGE_MASK;
     page2 = (addr + size) & TARGET_PAGE_MASK;

accel/tcg/cputlb.c store_helper_unaligned(CPUArchState *env, target_ulong addr, uint64_t val, · line 2272

Quoted 10 lines
accel/tcg/cputlb.c +1 -1
     size2 = (addr + size) & ~TARGET_PAGE_MASK;
     index2 = tlb_index(env, mmu_idx, page2);
     entry2 = tlb_entry(env, mmu_idx, page2);
 
     tlb_addr2 = tlb_addr_write(entry2);
-    if (!tlb_hit_page(tlb_addr2, page2)) {
+    if (page1 != page2 && !tlb_hit_page(tlb_addr2, page2)) {
         if (!victim_tlb_hit(env, mmu_idx, index2, tlb_off, page2)) {
             tlb_fill(env_cpu(env), page2, size2, MMU_DATA_STORE,
                      mmu_idx, retaddr);
Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw

PATCH 2/3 hw/misc: Add mmio-debug-exit device

In reply to Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal success or failure to the test runner. Unfortunately it's not easily usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86. Also, it requires adding ISA bus, which an architecture might not otherwise need.

Introduce mmio-debug-exit device, which has the same semantics, but is triggered by writes to memory.

---
 hw/misc/Kconfig          |  3 ++
 hw/misc/debugexit_mmio.c | 80 ++++++++++++++++++++++++++++++++++++++++
 hw/misc/meson.build      |  1 +
 hw/s390x/Kconfig         |  1 +
 4 files changed, 85 insertions(+)
 create mode 100644 hw/misc/debugexit_mmio.c
hw/misc/Kconfig +3 -0
index cbabe9f78c..0f12735ef7 100644
@@ -15,6 +15,9 @@ config ISA_DEBUG
     bool
     depends on ISA_BUS
 
+config MMIO_DEBUGEXIT
+    bool
+
 config SGA
     bool
     depends on ISA_BUS
hw/misc/debugexit_mmio.c new file +80 -0

80 lines not on this page · raw message

hw/misc/meson.build +1 -0
index 95268eddc0..1d2a1067dc 100644
@@ -2,6 +2,7 @@ softmmu_ss.add(when: 'CONFIG_APPLESMC', if_true: files('applesmc.c'))
 softmmu_ss.add(when: 'CONFIG_EDU', if_true: files('edu.c'))
 softmmu_ss.add(when: 'CONFIG_FW_CFG_DMA', if_true: files('vmcoreinfo.c'))
 softmmu_ss.add(when: 'CONFIG_ISA_DEBUG', if_true: files('debugexit.c'))
+softmmu_ss.add(when: 'CONFIG_MMIO_DEBUGEXIT', if_true: files('debugexit_mmio.c'))
 softmmu_ss.add(when: 'CONFIG_ISA_TESTDEV', if_true: files('pc-testdev.c'))
 softmmu_ss.add(when: 'CONFIG_PCA9552', if_true: files('pca9552.c'))
 softmmu_ss.add(when: 'CONFIG_PCI_TESTDEV', if_true: files('pci-testdev.c'))
hw/s390x/Kconfig +1 -0
index 5e7d8a2bae..9223715dcc 100644
@@ -5,6 +5,7 @@ config S390_CCW_VIRTIO
     imply VFIO_AP
     imply VFIO_CCW
     imply WDT_DIAG288
+    imply MMIO_DEBUGEXIT
     select PCI
     select S390_FLIC
     select SCLPCONSOLE
Signature · 1 line
2.35.3
#5 lore raw

Re: [PATCH 2/3] hw/misc: Add mmio-debug-exit device

Quoted 11 lines · Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal success or failure to the test runner. Unfortunately it's not easily usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86. Also, it requires adding ISA bus, which an architecture might not otherwise need.

Introduce mmio-debug-exit device, which has the same semantics, but is triggered by writes to memory.

You shouldn't need this for s390x, as there are already (at least) two other paths to qemu_system_shutdown_request.

E.g. SIGP, which has a stop option.

r~

#6 lore raw

Re: [PATCH 2/3] hw/misc: Add mmio-debug-exit device

Quoted 15 lines · Richard Henderson
Quoted 8 lines · Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal success or failure to the test runner. Unfortunately it's not easily
usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86. Also, it requires adding ISA bus, which an architecture might not otherwise need.

Quoted 5 lines

Introduce mmio-debug-exit device, which has the same semantics, but is
triggered by writes to memory.

Quoted 9 lines

You shouldn't need this for s390x, as there are already (at least) two other paths to
qemu_system_shutdown_request.

E.g. SIGP, which has a stop option.

r~

I would normally use lpswe + disabled wait, but this always gives me exit status code 0, which doesn't allow easily distinguishing between success and failure.

Code-wise SIGP seems to do roughly the same thing, and a quick experiment with:

    lgfi %r4,-1
    lgfi %r5,-1
    larl %r6,_cpuaddr
    stap 0(%r6)
    lh %r6,0(%r6)
    nilh %r6,0
    sigp %r4,%r6,5

_cpuaddr: .short 0

confirmed that we get exit status code 0 as well.

Best regards,
Ilya

#7 lore raw

Re: [PATCH 2/3] hw/misc: Add mmio-debug-exit device

Quoted 40 lines · Ilya Leoshkevich
Quoted 15 lines · Richard Henderson
Quoted 8 lines · Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal success or failure to the test runner. Unfortunately it's not easily
usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86. Also, it requires adding ISA bus, which an architecture might not otherwise need.

Quoted 5 lines

Introduce mmio-debug-exit device, which has the same semantics, but is
triggered by writes to memory.

Quoted 9 lines

You shouldn't need this for s390x, as there are already (at least) two other paths to
qemu_system_shutdown_request.

E.g. SIGP, which has a stop option.

r~

I would normally use lpswe + disabled wait, but this always gives me exit status code 0, which doesn't allow easily distinguishing between success and failure.

Code-wise SIGP seems to do roughly the same thing, and a quick experiment with:

    lgfi %r4,-1
    lgfi %r5,-1
    larl %r6,_cpuaddr
    stap 0(%r6)
    lh %r6,0(%r6)
    nilh %r6,0
    sigp %r4,%r6,5
Quoted 3 lines

_cpuaddr: .short 0

confirmed that we get exit status code 0 as well.

disabled wait should trigger a qemu_system_guest_panicked().

But "panic_action == PANIC_ACTION_SHUTDOWN" seems to only make qemu_main_loop() return with main_loop_should_exit() == true.

main/qemu_main will always return 0.

We could return != 0 on guest panic, but not sure if that could break existing scripts. We'd need a new QEMU toggle for that most probably ...

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw

Re: [PATCH 2/3] hw/misc: Add mmio-debug-exit device

Quoted 54 lines · David Hildenbrand
Quoted 49 lines · Ilya Leoshkevich
Quoted 24 lines · Richard Henderson
Quoted 11 lines · Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal
success or failure to the test runner. Unfortunately it's not easily
usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86.
Also, it requires adding ISA bus, which an architecture might not
otherwise need.

Quoted 6 lines

Introduce mmio-debug-exit device, which has the same semantics, but
is
triggered by writes to memory.

You shouldn't need this for s390x, as there are already (at least)
two other paths to
qemu_system_shutdown_request.

Quoted 5 lines

E.g. SIGP, which has a stop option.

r~

I would normally use lpswe + disabled wait, but this always gives me
exit status code 0, which doesn't allow easily distinguishing between
success and failure.

Code-wise SIGP seems to do roughly the same thing, and a quick experiment with:

    lgfi %r4,-1
    lgfi %r5,-1
    larl %r6,_cpuaddr
    stap 0(%r6)
    lh %r6,0(%r6)
    nilh %r6,0
    sigp %r4,%r6,5
_cpuaddr: .short 0

Quoted 1 line

confirmed that we get exit status code 0 as well.

disabled wait should trigger a qemu_system_guest_panicked().

Quoted 9 lines

But "panic_action == PANIC_ACTION_SHUTDOWN" seems to only make qemu_main_loop() return with main_loop_should_exit() == true.

main/qemu_main will always return 0.

We could return != 0 on guest panic, but not sure if that could break existing scripts. We'd need a new QEMU toggle for that most probably
...

I wonder if a device is a cleaner way to solve this? It may be used on all architectures, so there is no need to invent per-architecture way to exit with a specific code. Maybe we can even replace Intel's debugexit with it.

#9 lore raw

Re: [PATCH 2/3] hw/misc: Add mmio-debug-exit device

In reply to Ilya Leoshkevich
Quoted 40 lines · Ilya Leoshkevich
Quoted 15 lines · Richard Henderson
Quoted 8 lines · Ilya Leoshkevich

System tests on x86 use isa-debug-exit device in order to signal success or failure to the test runner. Unfortunately it's not easily
usable on other architectures, since a guest needs to access address_space_io, which may not be as straightforward as on x86. Also, it requires adding ISA bus, which an architecture might not otherwise need.

Quoted 5 lines

Introduce mmio-debug-exit device, which has the same semantics, but is
triggered by writes to memory.

Quoted 9 lines

You shouldn't need this for s390x, as there are already (at least) two other paths to
qemu_system_shutdown_request.

E.g. SIGP, which has a stop option.

r~

I would normally use lpswe + disabled wait, but this always gives me exit status code 0, which doesn't allow easily distinguishing between success and failure.

Code-wise SIGP seems to do roughly the same thing, and a quick experiment with:

    lgfi %r4,-1
    lgfi %r5,-1
    larl %r6,_cpuaddr
    stap 0(%r6)
    lh %r6,0(%r6)
    nilh %r6,0
    sigp %r4,%r6,5
Quoted 3 lines

_cpuaddr: .short 0

confirmed that we get exit status code 0 as well.

disabled wait should trigger a qemu_system_guest_panicked().

But "panic_action == PANIC_ACTION_SHUTDOWN" seems to only make qemu_main_loop() return with main_loop_should_exit() == true.

main/qemu_main will always return 0.

We could return != 0 on guest panic, but not sure if that could break existing scripts. We'd need a new QEMU toggle for that most probably ...

Signature · 3 lines
Thanks,

David / dhildenb
#10 lore raw

PATCH 3/3 tests/tcg/s390x: Test unaligned accesses to lowcore

In reply to Ilya Leoshkevich

Add a small test to avoid regressions.

---
 tests/tcg/s390x/Makefile.softmmu-target |  9 +++++++++
 tests/tcg/s390x/unaligned-lowcore.S     | 24 ++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100644 tests/tcg/s390x/Makefile.softmmu-target
 create mode 100644 tests/tcg/s390x/unaligned-lowcore.S
tests/tcg/s390x/Makefile.softmmu-target new file +9 -0
new file mode 100644
index 0000000000..fdee4a7616
@@ -0,0 +1,9 @@
+S390X_SRC=$(SRC_PATH)/tests/tcg/s390x
+VPATH+=$(S390X_SRC)
+QEMU_OPTS=-device mmio-debug-exit,base=0x2000 -kernel
+
+%: %.S
+	$(CC) -march=z13 -m64 -nostartfiles -static -Wl,-Ttext=0 \
+		-Wl,--build-id=none $< -o $@
+
+TESTS += unaligned-lowcore
tests/tcg/s390x/unaligned-lowcore.S new file +24 -0
new file mode 100644
index 0000000000..2b34c3d5ef
@@ -0,0 +1,24 @@
+    .org 0x1D0                 /* Program new PSW */
+    .quad 0x180000000, _pgm    /* BA, EA */
+    .org 0x200                 /* lowcore padding */
+
+    .globl _start
+_start:
+    lctlg %c0,%c0,_c0
+    vst %v0,_unaligned
+    mviy _exit,0
+
+_pgm:
+    mviy _exit,1
+
+    .align 8
+_c0:
+    .quad 0x10060000           /* lowcore protection, AFP, VX */
+
+    .byte 0
+_unaligned:
+    .octa 0
+
+    .org 0x2000                /* mmio-debug-exit base */
+_exit:
+    .byte 0
Signature · 1 line
2.35.3