s390x: TOD refactoring + TCG CPU hotplug support

PATCH Applied David Hildenbrand 2 participants 11 messages Jun 27, 2018

Share & tools

David Hildenbrand

author

2018-06-27 13:44 UTC #0 lore raw

v4 cover s390x: TOD refactoring + TCG CPU hotplug support

The TOD in TCG is not handled correctly:
- each CPU has its own TOD based on CPU creation time vs. a system TOD
- TOD is not migrated
- TOD timer is not restarted during migration
- CKC interrupts/TOD timer is not cleared when resetting the CKC

This (and a cpu creation problem for single threaded TCG) currently made CPU hotplug under TCG not work. Now it's working

The third patch also refactors TOD handling for KVM (moved into a new TOD device). The second patch makes sure that we can use a const pointer for the new TOD clock struct.

v3 -> v4:
- "s390x/kvm: pass values instead of pointers to kvm_s390_set_clock_*()" -- Added a note in the patch description as reuested
- "[PATCH v3 5/9] s390x/tcg: properly implement the TOD"
-- tod->low < td->base.low -> td->base.low > tod->low
-- access td->base.low directly in sckc handler

v2 -> v3:
- added "s390x/tcg: avoid overflows in time2tod/tod2time"
- added "s390x/kvm: pass values instead of pointers to kvm_s390_set_clock_*()"
- "s390x/tod: factor out TOD into separate device"
-- Use "const" for parameters in function definitions
-- Use "s390-tod-qemu" and "s390-tod-kvm" as type names
-- Drop hotpluggable=false
- "s390x/tcg: properly implement the TOD"
-- s390_get_tod() -> s390_get_todstate()
-- use error_abort
- "s390x/tcg: SET CLOCK COMPARATOR can clear CKC interrupts"
-- Add a comment for the -1ULL check
- "s390x/tcg: implement SET CLOCK"
-- use error_abort
- "s390x/tcg: fix CPU hotplug with single-threaded TCG"
-- add a comment describing the run_on_cpu() difference

v1 -> v2:
- "s390x/tcg: properly implement the TOD" -- introduce "tcg_s390x.h" similar to "kvm_s390x.h" -- dropped one unnecessary include introduction
- "s390x/tcg: rearm the CKC timer during migration" -- introduce "tcg-stub.c" similar to "kvm-stub.c"

David Hildenbrand (9):
s390x/tcg: avoid overflows in time2tod/tod2time s390x/kvm: pass values instead of pointers to kvm_s390_set_clock_*() s390x/tod: factor out TOD into separate device
s390x/tcg: drop tod_basetime
s390x/tcg: properly implement the TOD
s390x/tcg: SET CLOCK COMPARATOR can clear CKC interrupts s390x/tcg: implement SET CLOCK
s390x/tcg: rearm the CKC timer during migration s390x/tcg: fix CPU hotplug with single-threaded TCG

 hw/s390x/Makefile.objs     |   3 +
 hw/s390x/s390-virtio-ccw.c |  57 +---------------
 hw/s390x/tod-kvm.c         |  64 ++++++++++++++++++
 hw/s390x/tod-qemu.c        |  87 +++++++++++++++++++++++++
 hw/s390x/tod.c             | 130 +++++++++++++++++++++++++++++++++++++
 include/hw/s390x/tod.h     |  65 +++++++++++++++++++
 target/s390x/Makefile.objs |   1 +
 target/s390x/cpu.c         |  57 ++++------------
 target/s390x/cpu.h         |   4 --
 target/s390x/helper.h      |   1 +
 target/s390x/insn-data.def |   3 +-
 target/s390x/internal.h    |  15 -----
 target/s390x/kvm-stub.c    |   4 +-
 target/s390x/kvm.c         |  12 ++--
 target/s390x/kvm_s390x.h   |   6 +-
 target/s390x/machine.c     |   6 ++
 target/s390x/misc_helper.c |  50 ++++++++++++--
 target/s390x/tcg-stub.c    |  20 ++++++
 target/s390x/tcg_s390x.h   |  18 +++++
 target/s390x/translate.c   |   9 +++
 20 files changed, 475 insertions(+), 137 deletions(-)
 create mode 100644 hw/s390x/tod-kvm.c
 create mode 100644 hw/s390x/tod-qemu.c
 create mode 100644 hw/s390x/tod.c
 create mode 100644 include/hw/s390x/tod.h
 create mode 100644 target/s390x/tcg-stub.c
 create mode 100644 target/s390x/tcg_s390x.h

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #1 lore raw

PATCH v4 1/9 s390x/tcg: avoid overflows in time2tod/tod2time

Big values for the TOD/ns clock can result in some overflows that can be avoided. Not all overflows can be handled however, as the conversion either multiplies by 4.096 or divided by 4.096.

Apply the trick used in the Linux kernel in arch/s390/include/asm/timex.h for tod_to_ns() and use the same trick also for the conversion in the other direction.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/internal.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

target/s390x/internal.h +3 -2

index e392a02d12..6cf63340bf 100644
@@ -243,13 +243,14 @@ enum cc_op {
 /* Converts ns to s390's clock format */
 static inline uint64_t time2tod(uint64_t ns)
 {
-    return (ns << 9) / 125;
+    return (ns << 9) / 125 + (((ns & 0xff10000000000000ull) / 125) << 9);
+
 }
 
 /* Converts s390's clock format to ns */
 static inline uint64_t tod2time(uint64_t t)
 {
-    return (t * 125) >> 9;
+    return ((t >> 9) * 125) + (((t & 0x1ff) * 125) >> 9);
 }
 
 static inline hwaddr decode_basedisp_s(CPUS390XState *env, uint32_t ipb,

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #2 lore raw

PATCH v4 2/9 s390x/kvm: pass values instead of pointers to kvm_s390_set_clock_*()

In reply to David Hildenbrand

We are going to factor out the TOD into a separate device and use const pointers for device class functions where possible. We are passing right now ordinary pointers that should never be touched when setting the TOD. Let's just pass the values directly.

Note that s390_set_clock() will be removed in a follow-on patch and therefore its calling convention is not changed.

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/cpu.c       |  4 ++--
 target/s390x/kvm-stub.c  |  4 ++--
 target/s390x/kvm.c       | 12 ++++++------
 target/s390x/kvm_s390x.h |  4 ++--
 4 files changed, 12 insertions(+), 12 deletions(-)

target/s390x/cpu.c +2 -2

index c268065887..68512e3e54 100644
@@ -413,9 +413,9 @@ int s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
     int r = 0;
 
     if (kvm_enabled()) {
-        r = kvm_s390_set_clock_ext(tod_high, tod_low);
+        r = kvm_s390_set_clock_ext(*tod_high, *tod_low);
         if (r == -ENXIO) {
-            return kvm_s390_set_clock(tod_high, tod_low);
+            return kvm_s390_set_clock(*tod_high, *tod_low);
         }
     }
     /* Fixme TCG */

target/s390x/kvm-stub.c +2 -2

index 29b10542cc..bf7795e47a 100644
@@ -60,12 +60,12 @@ int kvm_s390_get_clock_ext(uint8_t *tod_high, uint64_t *tod_low)
     return -ENOSYS;
 }
 
-int kvm_s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
+int kvm_s390_set_clock(uint8_t tod_high, uint64_t tod_low)
 {
     return -ENOSYS;
 }
 
-int kvm_s390_set_clock_ext(uint8_t *tod_high, uint64_t *tod_low)
+int kvm_s390_set_clock_ext(uint8_t tod_high, uint64_t tod_low)
 {
     return -ENOSYS;
 }

target/s390x/kvm.c +6 -6

index ac370da281..8bcd832123 100644
@@ -666,13 +666,13 @@ int kvm_s390_get_clock_ext(uint8_t *tod_high, uint64_t *tod_low)
     return r;
 }
 
-int kvm_s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
+int kvm_s390_set_clock(uint8_t tod_high, uint64_t tod_low)
 {
     int r;
     struct kvm_device_attr attr = {
         .group = KVM_S390_VM_TOD,
         .attr = KVM_S390_VM_TOD_LOW,
-        .addr = (uint64_t)tod_low,
+        .addr = (uint64_t)&tod_low,
     };
 
     r = kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attr);

@@ -681,15 +681,15 @@ int kvm_s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
     }
 
     attr.attr = KVM_S390_VM_TOD_HIGH;
-    attr.addr = (uint64_t)tod_high;
+    attr.addr = (uint64_t)&tod_high;
     return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attr);
 }
 
-int kvm_s390_set_clock_ext(uint8_t *tod_high, uint64_t *tod_low)
+int kvm_s390_set_clock_ext(uint8_t tod_high, uint64_t tod_low)
 {
     struct kvm_s390_vm_tod_clock gtod = {
-        .epoch_idx = *tod_high,
-        .tod  = *tod_low,
+        .epoch_idx = tod_high,
+        .tod  = tod_low,
     };
     struct kvm_device_attr attr = {
         .group = KVM_S390_VM_TOD,

target/s390x/kvm_s390x.h +2 -2

index c383bf4ee9..36eb34bf99 100644
@@ -25,8 +25,8 @@ int kvm_s390_get_ri(void);
 int kvm_s390_get_gs(void);
 int kvm_s390_get_clock(uint8_t *tod_high, uint64_t *tod_clock);
 int kvm_s390_get_clock_ext(uint8_t *tod_high, uint64_t *tod_clock);
-int kvm_s390_set_clock(uint8_t *tod_high, uint64_t *tod_clock);
-int kvm_s390_set_clock_ext(uint8_t *tod_high, uint64_t *tod_clock);
+int kvm_s390_set_clock(uint8_t tod_high, uint64_t tod_clock);
+int kvm_s390_set_clock_ext(uint8_t tod_high, uint64_t tod_clock);
 void kvm_s390_enable_css_support(S390CPU *cpu);
 int kvm_s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch,
                                     int vq, bool assign);

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #3 lore raw

PATCH v4 3/9 s390x/tod: factor out TOD into separate device

In reply to David Hildenbrand

Let's treat this like a separate device. TCG will have to store the actual state/time later on.

Include cpu-qom.h in kvm_s390x.h (due to S390CPU) to compile tod-kvm.c.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 hw/s390x/Makefile.objs     |   3 +
 hw/s390x/s390-virtio-ccw.c |  57 +-----------------
 hw/s390x/tod-kvm.c         |  64 ++++++++++++++++++++
 hw/s390x/tod-qemu.c        |  49 +++++++++++++++
 hw/s390x/tod.c             | 119 +++++++++++++++++++++++++++++++++++++
 include/hw/s390x/tod.h     |  46 ++++++++++++++
 target/s390x/cpu.c         |  32 ----------
 target/s390x/cpu.h         |   2 -
 target/s390x/kvm_s390x.h   |   2 +
 9 files changed, 286 insertions(+), 88 deletions(-)
 create mode 100644 hw/s390x/tod-kvm.c
 create mode 100644 hw/s390x/tod-qemu.c
 create mode 100644 hw/s390x/tod.c
 create mode 100644 include/hw/s390x/tod.h

hw/s390x/Makefile.objs +3 -0

index dc704b57d6..93282f7c59 100644
@@ -14,6 +14,9 @@ obj-$(CONFIG_PCI) += s390-pci-bus.o s390-pci-inst.o
 obj-$(call lnot,$(CONFIG_PCI)) += s390-pci-stub.o
 obj-y += s390-skeys.o
 obj-y += s390-stattrib.o
+obj-y += tod.o
+obj-$(CONFIG_KVM) += tod-kvm.o
+obj-$(CONFIG_TCG) += tod-qemu.o
 obj-$(CONFIG_KVM) += s390-skeys-kvm.o
 obj-$(CONFIG_KVM) += s390-stattrib-kvm.o
 obj-y += s390-ccw.o

hw/s390x/s390-virtio-ccw.c +3 ~~-54~~

75 lines not on this page · raw message

hw/s390x/tod-kvm.c new file +64 -0

64 lines not on this page · raw message

hw/s390x/tod-qemu.c new file +49 -0

49 lines not on this page · raw message

hw/s390x/tod.c new file +119 -0

119 lines not on this page · raw message

include/hw/s390x/tod.h new file +46 -0

46 lines not on this page · raw message

target/s390x/cpu.c +0 ~~-32~~

index 68512e3e54..03ea6eafa7 100644
@@ -390,38 +390,6 @@ unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu)
     return s390_count_running_cpus();
 }
 
-int s390_get_clock(uint8_t *tod_high, uint64_t *tod_low)
-{
-    int r = 0;
-
-    if (kvm_enabled()) {
-        r = kvm_s390_get_clock_ext(tod_high, tod_low);
-        if (r == -ENXIO) {
-            return kvm_s390_get_clock(tod_high, tod_low);
-        }
-    } else {
-        /* Fixme TCG */
-        *tod_high = 0;
-        *tod_low = 0;
-    }
-
-    return r;
-}
-
-int s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
-{
-    int r = 0;
-
-    if (kvm_enabled()) {
-        r = kvm_s390_set_clock_ext(*tod_high, *tod_low);
-        if (r == -ENXIO) {
-            return kvm_s390_set_clock(*tod_high, *tod_low);
-        }
-    }
-    /* Fixme TCG */
-    return r;
-}
-
 int s390_set_memory_limit(uint64_t new_limit, uint64_t *hw_limit)
 {
     if (kvm_enabled()) {

target/s390x/cpu.h +0 -2

index 6629a533f3..ac51c17fb4 100644
@@ -714,8 +714,6 @@ static inline void s390_do_cpu_load_normal(CPUState *cs, run_on_cpu_data arg)
 
 
 /* cpu.c */
-int s390_get_clock(uint8_t *tod_high, uint64_t *tod_low);
-int s390_set_clock(uint8_t *tod_high, uint64_t *tod_low);
 void s390_crypto_reset(void);
 bool s390_get_squash_mcss(void);
 int s390_set_memory_limit(uint64_t new_limit, uint64_t *hw_limit);

target/s390x/kvm_s390x.h +2 -0

index 36eb34bf99..6e52287da3 100644
@@ -10,6 +10,8 @@
 #ifndef KVM_S390X_H
 #define KVM_S390X_H
 
+#include "cpu-qom.h"
+
 struct kvm_s390_irq;
 
 void kvm_s390_floating_interrupt_legacy(struct kvm_s390_irq *irq);

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #4 lore raw

PATCH v4 4/9 s390x/tcg: drop tod_basetime

In reply to David Hildenbrand

Never set to anything but 0.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/cpu.c         | 1 -
 target/s390x/cpu.h         | 1 -
 target/s390x/misc_helper.c | 4 ++--
 3 files changed, 2 insertions(+), 4 deletions(-)

target/s390x/cpu.c +0 -1

index 03ea6eafa7..a41b3f34ab 100644
@@ -289,7 +289,6 @@ static void s390_cpu_initfn(Object *obj)
     qemu_get_timedate(&tm, 0);
     env->tod_offset = TOD_UNIX_EPOCH +
                       (time2tod(mktimegm(&tm)) * 1000000000ULL);
-    env->tod_basetime = 0;
     env->tod_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, s390x_tod_timer, cpu);
     env->cpu_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, s390x_cpu_timer, cpu);
     s390_cpu_set_state(S390_CPU_STATE_STOPPED, cpu);

target/s390x/cpu.h +0 -1

index ac51c17fb4..4abfe88a3d 100644
@@ -131,7 +131,6 @@ struct CPUS390XState {
 #endif
 
     uint64_t tod_offset;
-    uint64_t tod_basetime;
     QEMUTimer *tod_timer;
 
     QEMUTimer *cpu_timer;

target/s390x/misc_helper.c +2 -2

index de1ced2082..dd5273949b 100644
@@ -141,7 +141,7 @@ uint64_t HELPER(stck)(CPUS390XState *env)
     uint64_t time;
 
     time = env->tod_offset +
-        time2tod(qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) - env->tod_basetime);
+        time2tod(qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
 
     return time;
 }

@@ -161,7 +161,7 @@ void HELPER(sckc)(CPUS390XState *env, uint64_t time)
     /* nanoseconds */
     time = tod2time(time);
 
-    timer_mod(env->tod_timer, env->tod_basetime + time);
+    timer_mod(env->tod_timer, time);
 }
 
 /* Set Tod Programmable Field */

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #5 lore raw

PATCH v4 5/9 s390x/tcg: properly implement the TOD

In reply to David Hildenbrand

Right now, each CPU has its own TOD. Especially, the TOD will differ based on creation time of a CPU - e.g. when hotplugging a CPU the times will differ quite a lot, resulting in stall warnings in the guest.

Let's use a single TOD by implementing our new TOD device. Prepare it for TOD-clock epoch extension.

Most importantly, whenever we set the TOD, we have to update the CKC timer.

Introduce "tcg_s390x.h" just like "kvm_s390x.h" for tcg specific function declarations that should not go into cpu.h.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 hw/s390x/tod-qemu.c        | 46 ++++++++++++++++++++++++++++++++++----
 hw/s390x/tod.c             | 11 +++++++++
 include/hw/s390x/tod.h     | 19 ++++++++++++++++
 target/s390x/cpu.c         |  7 ------
 target/s390x/cpu.h         |  1 -
 target/s390x/internal.h    | 16 -------------
 target/s390x/misc_helper.c | 25 ++++++++++++++++-----
 target/s390x/tcg_s390x.h   | 18 +++++++++++++++
 8 files changed, 109 insertions(+), 34 deletions(-)
 create mode 100644 target/s390x/tcg_s390x.h

hw/s390x/tod-qemu.c +42 -4

index 03ea1ce4e8..59c015c69d 100644
@@ -11,19 +11,43 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/s390x/tod.h"
+#include "qemu/timer.h"
+#include "qemu/cutils.h"
+#include "cpu.h"
+#include "tcg_s390x.h"
 
 static void qemu_s390_tod_get(const S390TODState *td, S390TOD *tod,
                               Error **errp)
 {
-    /* FIXME */
-    tod->high = 0;
-    tod->low = 0;
+    *tod = td->base;
+
+    tod->low += time2tod(qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
+    if (tod->low < td->base.low) {
+        tod->high++;
+    }
 }
 
 static void qemu_s390_tod_set(S390TODState *td, const S390TOD *tod,
                               Error **errp)
 {
-    /* FIXME */
+    CPUState *cpu;
+
+    td->base = *tod;
+
+    td->base.low -= time2tod(qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
+    if (td->base.low > tod->low) {
+        td->base.high--;
+    }
+
+    /*
+     * The TOD has been changed and we have to recalculate the CKC values
+     * for all CPUs. We do this asynchronously, as "SET CLOCK should be
+     * issued only while all other activity on all CPUs .. has been
+     * suspended".
+     */
+    CPU_FOREACH(cpu) {
+        async_run_on_cpu(cpu, tcg_s390_tod_updated, RUN_ON_CPU_NULL);
+    }
 }
 
 static void qemu_s390_tod_class_init(ObjectClass *oc, void *data)

@@ -34,10 +58,24 @@ static void qemu_s390_tod_class_init(ObjectClass *oc, void *data)
     tdc->set = qemu_s390_tod_set;
 }
 
+static void qemu_s390_tod_init(Object *obj)
+{
+    S390TODState *td = S390_TOD(obj);
+    struct tm tm;
+
+    qemu_get_timedate(&tm, 0);
+    td->base.high = 0;
+    td->base.low = TOD_UNIX_EPOCH + (time2tod(mktimegm(&tm)) * 1000000000ULL);
+    if (td->base.low < TOD_UNIX_EPOCH) {
+        td->base.high += 1;
+    }
+}
+
 static TypeInfo qemu_s390_tod_info = {
     .name = TYPE_QEMU_S390_TOD,
     .parent = TYPE_S390_TOD,
     .instance_size = sizeof(S390TODState),
+    .instance_init = qemu_s390_tod_init,
     .class_init = qemu_s390_tod_class_init,
     .class_size = sizeof(S390TODClass),
 };

hw/s390x/tod.c +11 -0

index 0501affa7f..1c63f411e6 100644
@@ -30,6 +30,17 @@ void s390_init_tod(void)
     qdev_init_nofail(DEVICE(obj));
 }
 
+S390TODState *s390_get_todstate(void)
+{
+    static S390TODState *ts;
+
+    if (!ts) {
+        ts = S390_TOD(object_resolve_path_type("", TYPE_S390_TOD, NULL));
+    }
+
+    return ts;
+}
+
 #define S390_TOD_CLOCK_VALUE_MISSING    0x00
 #define S390_TOD_CLOCK_VALUE_PRESENT    0x01

include/hw/s390x/tod.h +19 -0

index 7096b574ae..413c0d7c02 100644
@@ -30,6 +30,9 @@ typedef struct S390TOD {
 typedef struct S390TODState {
     /* private */
     DeviceState parent_obj;
+
+    /* unused by KVM implementation */
+    S390TOD base;
 } S390TODState;
 
 typedef struct S390TODClass {

@@ -41,6 +44,22 @@ typedef struct S390TODClass {
     void (*set)(S390TODState *td, const S390TOD *tod, Error **errp);
 } S390TODClass;
 
+/* The value of the TOD clock for 1.1.1970. */
+#define TOD_UNIX_EPOCH 0x7d91048bca000000ULL
+
+/* Converts ns to s390's clock format */
+static inline uint64_t time2tod(uint64_t ns)
+{
+    return (ns << 9) / 125 + (((ns & 0xff10000000000000ull) / 125) << 9);
+}
+
+/* Converts s390's clock format to ns */
+static inline uint64_t tod2time(uint64_t t)
+{
+    return ((t >> 9) * 125) + (((t & 0x1ff) * 125) >> 9);
+}
+
 void s390_init_tod(void);
+S390TODState *s390_get_todstate(void);
 
 #endif

target/s390x/cpu.c +0 -7

index a41b3f34ab..40d6980229 100644
@@ -30,7 +30,6 @@
 #include "kvm_s390x.h"
 #include "sysemu/kvm.h"
 #include "qemu-common.h"
-#include "qemu/cutils.h"
 #include "qemu/timer.h"
 #include "qemu/error-report.h"
 #include "trace.h"

@@ -275,9 +274,6 @@ static void s390_cpu_initfn(Object *obj)
     CPUState *cs = CPU(obj);
     S390CPU *cpu = S390_CPU(obj);
     CPUS390XState *env = &cpu->env;
-#if !defined(CONFIG_USER_ONLY)
-    struct tm tm;
-#endif
 
     cs->env_ptr = env;
     cs->halted = 1;

@@ -286,9 +282,6 @@ static void s390_cpu_initfn(Object *obj)
                         s390_cpu_get_crash_info_qom, NULL, NULL, NULL, NULL);
     s390_cpu_model_register_props(obj);
 #if !defined(CONFIG_USER_ONLY)
-    qemu_get_timedate(&tm, 0);
-    env->tod_offset = TOD_UNIX_EPOCH +
-                      (time2tod(mktimegm(&tm)) * 1000000000ULL);
     env->tod_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, s390x_tod_timer, cpu);
     env->cpu_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, s390x_cpu_timer, cpu);
     s390_cpu_set_state(S390_CPU_STATE_STOPPED, cpu);

target/s390x/cpu.h +0 -1

index 4abfe88a3d..2c3dd2d189 100644
@@ -130,7 +130,6 @@ struct CPUS390XState {
     uint64_t cpuid;
 #endif
 
-    uint64_t tod_offset;
     QEMUTimer *tod_timer;
 
     QEMUTimer *cpu_timer;

target/s390x/internal.h +0 ~~-16~~

index 6cf63340bf..f2a771e2b4 100644
@@ -237,22 +237,6 @@ enum cc_op {
     CC_OP_MAX
 };
 
-/* The value of the TOD clock for 1.1.1970. */
-#define TOD_UNIX_EPOCH 0x7d91048bca000000ULL
-
-/* Converts ns to s390's clock format */
-static inline uint64_t time2tod(uint64_t ns)
-{
-    return (ns << 9) / 125 + (((ns & 0xff10000000000000ull) / 125) << 9);
-
-}
-
-/* Converts s390's clock format to ns */
-static inline uint64_t tod2time(uint64_t t)
-{
-    return ((t >> 9) * 125) + (((t & 0x1ff) * 125) >> 9);
-}
-
 static inline hwaddr decode_basedisp_s(CPUS390XState *env, uint32_t ipb,
                                        uint8_t *ar)
 {

target/s390x/misc_helper.c +19 -6

index dd5273949b..229f14d3da 100644
@@ -28,6 +28,8 @@
 #include "qemu/timer.h"
 #include "exec/exec-all.h"
 #include "exec/cpu_ldst.h"
+#include "qapi/error.h"
+#include "tcg_s390x.h"
 
 #if !defined(CONFIG_USER_ONLY)
 #include "sysemu/cpus.h"

@@ -39,6 +41,7 @@
 #include "hw/s390x/ioinst.h"
 #include "hw/s390x/s390-pci-inst.h"
 #include "hw/boards.h"
+#include "hw/s390x/tod.h"
 #endif
 
 /* #define DEBUG_HELPER */

@@ -138,17 +141,19 @@ void HELPER(spx)(CPUS390XState *env, uint64_t a1)
 /* Store Clock */
 uint64_t HELPER(stck)(CPUS390XState *env)
 {
-    uint64_t time;
+    S390TODState *td = s390_get_todstate();
+    S390TODClass *tdc = S390_TOD_GET_CLASS(td);
+    S390TOD tod;
 
-    time = env->tod_offset +
-        time2tod(qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
-
-    return time;
+    tdc->get(td, &tod, &error_abort);
+    return tod.low;
 }
 
 /* Set Clock Comparator */
 void HELPER(sckc)(CPUS390XState *env, uint64_t time)
 {
+    S390TODState *td = s390_get_todstate();
+
     if (time == -1ULL) {
         return;
     }

@@ -156,7 +161,7 @@ void HELPER(sckc)(CPUS390XState *env, uint64_t time)
     env->ckc = time;
 
     /* difference between origins */
-    time -= env->tod_offset;
+    time -= td->base.low;
 
     /* nanoseconds */
     time = tod2time(time);

@@ -164,6 +169,14 @@ void HELPER(sckc)(CPUS390XState *env, uint64_t time)
     timer_mod(env->tod_timer, time);
 }
 
+void tcg_s390_tod_updated(CPUState *cs, run_on_cpu_data opaque)
+{
+    S390CPU *cpu = S390_CPU(cs);
+    CPUS390XState *env = &cpu->env;
+
+    helper_sckc(env, env->ckc);
+}
+
 /* Set Tod Programmable Field */
 void HELPER(sckpf)(CPUS390XState *env, uint64_t r0)
 {

target/s390x/tcg_s390x.h new file +18 -0

new file mode 100644
index 0000000000..4e308aa0ce
@@ -0,0 +1,18 @@
+/*
+ * QEMU TCG support -- s390x specific functions.
+ *
+ * Copyright 2018 Red Hat, Inc.
+ *
+ * Authors:
+ *   David Hildenbrand <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef TCG_S390X_H
+#define TCG_S390X_H
+
+void tcg_s390_tod_updated(CPUState *cs, run_on_cpu_data opaque);
+
+#endif /* TCG_S390X_H */

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #6 lore raw

PATCH v4 6/9 s390x/tcg: SET CLOCK COMPARATOR can clear CKC interrupts

In reply to David Hildenbrand

Let's stop the timer and delete any pending CKC IRQ before doing anything else.

While at it, add a comment why the check for ckc == -1ULL is needed.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/misc_helper.c | 7 +++++++
 1 file changed, 7 insertions(+)

target/s390x/misc_helper.c +7 -0

index 229f14d3da..e4c1522140 100644
@@ -154,6 +154,13 @@ void HELPER(sckc)(CPUS390XState *env, uint64_t time)
 {
     S390TODState *td = s390_get_todstate();
 
+    /* stop the timer and remove pending CKC IRQs */
+    timer_del(env->tod_timer);
+    qemu_mutex_lock_iothread();
+    env->pending_int &= ~INTERRUPT_EXT_CLOCK_COMPARATOR;
+    qemu_mutex_unlock_iothread();
+
+    /* the tod has to exceed the ckc, this can never happen if ckc is all 1's */
     if (time == -1ULL) {
         return;
     }

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #7 lore raw

PATCH v4 7/9 s390x/tcg: implement SET CLOCK

In reply to David Hildenbrand

This allows a guest to change its TOD. We already take care of updating all CKC timers from within S390TODClass.

Use MO_ALIGN to load the operand manually - this will properly trigger a SPECIFICATION exception.

Acked-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/helper.h      |  1 +
 target/s390x/insn-data.def |  3 +--
 target/s390x/misc_helper.c | 16 ++++++++++++++++
 target/s390x/translate.c   |  9 +++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)

target/s390x/helper.h +1 -0

index 59cba86a27..97c60ca7bc 100644
@@ -127,6 +127,7 @@ DEF_HELPER_4(diag, void, env, i32, i32, i32)
 DEF_HELPER_3(load_psw, noreturn, env, i64, i64)
 DEF_HELPER_FLAGS_2(spx, TCG_CALL_NO_RWG, void, env, i64)
 DEF_HELPER_FLAGS_1(stck, TCG_CALL_NO_RWG_SE, i64, env)
+DEF_HELPER_FLAGS_2(sck, TCG_CALL_NO_RWG, i32, env, i64)
 DEF_HELPER_FLAGS_2(sckc, TCG_CALL_NO_RWG, void, env, i64)
 DEF_HELPER_FLAGS_2(sckpf, TCG_CALL_NO_RWG, void, env, i64)
 DEF_HELPER_FLAGS_1(stckc, TCG_CALL_NO_RWG, i64, env)

target/s390x/insn-data.def +1 -2

index 157619403d..5c6f33ed9c 100644
@@ -997,8 +997,7 @@
 /* SET ADDRESS SPACE CONTROL FAST */
     C(0xb279, SACF,    S,     Z,   0, a2, 0, 0, sacf, 0)
 /* SET CLOCK */
-    /* ??? Not implemented - is it necessary? */
-    C(0xb204, SCK,     S,     Z,   0, 0, 0, 0, 0, 0)
+    C(0xb204, SCK,     S,     Z,   la2, 0, 0, 0, sck, 0)
 /* SET CLOCK COMPARATOR */
     C(0xb206, SCKC,    S,     Z,   0, m2_64a, 0, 0, sckc, 0)
 /* SET CLOCK PROGRAMMABLE FIELD */

target/s390x/misc_helper.c +16 -0

index e4c1522140..7656a9b90a 100644
@@ -184,6 +184,22 @@ void tcg_s390_tod_updated(CPUState *cs, run_on_cpu_data opaque)
     helper_sckc(env, env->ckc);
 }
 
+/* Set Clock */
+uint32_t HELPER(sck)(CPUS390XState *env, uint64_t tod_low)
+{
+    S390TODState *td = s390_get_todstate();
+    S390TODClass *tdc = S390_TOD_GET_CLASS(td);
+    S390TOD tod = {
+        .high = 0,
+        .low = tod_low,
+    };
+
+    qemu_mutex_lock_iothread();
+    tdc->set(td, &tod, &error_abort);
+    qemu_mutex_unlock_iothread();
+    return 0;
+}
+
 /* Set Tod Programmable Field */
 void HELPER(sckpf)(CPUS390XState *env, uint64_t r0)
 {

target/s390x/translate.c +9 -0

index fdfec7feba..57c03cbf58 100644
@@ -4016,6 +4016,15 @@ static DisasJumpType op_stcke(DisasContext *s, DisasOps *o)
     return DISAS_NEXT;
 }
 
+static DisasJumpType op_sck(DisasContext *s, DisasOps *o)
+{
+    check_privileged(s);
+    tcg_gen_qemu_ld_i64(o->in1, o->addr1, get_mem_index(s), MO_TEQ | MO_ALIGN);
+    gen_helper_sck(cc_op, cpu_env, o->in1);
+    set_cc_static(s);
+    return DISAS_NEXT;
+}
+
 static DisasJumpType op_sckc(DisasContext *s, DisasOps *o)
 {
     check_privileged(s);

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #8 lore raw

PATCH v4 8/9 s390x/tcg: rearm the CKC timer during migration

In reply to David Hildenbrand

If the CPU data is migrated after the TOD clock, the CKC timer of a CPU is not rearmed. Let's rearm it when loading the CPU state.

Introduce tcg-stub.c just like kvm-stub.c for tcg specific stubs.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/Makefile.objs |  1 +
 target/s390x/machine.c     |  6 ++++++
 target/s390x/tcg-stub.c    | 20 ++++++++++++++++++++
 3 files changed, 27 insertions(+)
 create mode 100644 target/s390x/tcg-stub.c

target/s390x/Makefile.objs +1 -0

index 31932de9cf..22a9a9927a 100644
@@ -5,6 +5,7 @@ obj-$(CONFIG_SOFTMMU) += machine.o ioinst.o arch_dump.o mmu_helper.o diag.o
 obj-$(CONFIG_SOFTMMU) += sigp.o
 obj-$(CONFIG_KVM) += kvm.o
 obj-$(call lnot,$(CONFIG_KVM)) += kvm-stub.o
+obj-$(call lnot,$(CONFIG_TCG)) += tcg-stub.o
 
 # build and run feature list generator
 feat-src = $(SRC_PATH)/target/$(TARGET_BASE_ARCH)/

target/s390x/machine.c +6 -0

index 84b4928755..bd3230d027 100644
@@ -19,6 +19,7 @@
 #include "cpu.h"
 #include "internal.h"
 #include "kvm_s390x.h"
+#include "tcg_s390x.h"
 #include "sysemu/kvm.h"
 
 static int cpu_post_load(void *opaque, int version_id)

@@ -34,6 +35,11 @@ static int cpu_post_load(void *opaque, int version_id)
         return kvm_s390_vcpu_interrupt_post_load(cpu);
     }
 
+    if (tcg_enabled()) {
+        /* Rearm the CKC timer if necessary */
+        tcg_s390_tod_updated(CPU(cpu), RUN_ON_CPU_NULL);
+    }
+
     return 0;
 }

target/s390x/tcg-stub.c new file +20 -0

new file mode 100644
index 0000000000..c93501db0b
@@ -0,0 +1,20 @@
+/*
+ * QEMU TCG support -- s390x specific function stubs.
+ *
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *   David Hildenbrand <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "tcg_s390x.h"
+
+void tcg_s390_tod_updated(CPUState *cs, run_on_cpu_data opaque)
+{
+}

Signature · 1 line

2.17.1

David Hildenbrand

author

2018-06-27 13:44 UTC #9 lore raw

PATCH v4 9/9 s390x/tcg: fix CPU hotplug with single-threaded TCG

In reply to David Hildenbrand

run_on_cpu() doesn't seem to work reliably until the CPU has been fully created if the single-threaded TCG main loop is already running.

Therefore, hotplugging a CPU under single-threaded TCG does currently not work. We should use the direct call instead of going via run_on_cpu().

So let's use run_on_cpu() for KVM only - KVM requires it due to the initial CPU reset ioctl. As a nice side effect, we get rid of the ifdef.

Reviewed-by Thomas Huth <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 target/s390x/cpu.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

target/s390x/cpu.c +12 -5

index 40d6980229..271c5ce652 100644
@@ -218,11 +218,18 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
 #endif
     s390_cpu_gdb_init(cs);
     qemu_init_vcpu(cs);
-#if !defined(CONFIG_USER_ONLY)
-    run_on_cpu(cs, s390_do_cpu_full_reset, RUN_ON_CPU_NULL);
-#else
-    cpu_reset(cs);
-#endif
+
+    /*
+     * KVM requires the initial CPU reset ioctl to be executed on the target
+     * CPU thread. CPU hotplug under single-threaded TCG will not work with
+     * run_on_cpu(), as run_on_cpu() will not work properly if called while
+     * the main thread is already running but the CPU hasn't been realized.
+     */
+    if (kvm_enabled()) {
+        run_on_cpu(cs, s390_do_cpu_full_reset, RUN_ON_CPU_NULL);
+    } else {
+        cpu_reset(cs);
+    }
 
     scc->parent_realize(dev, &err);
 out:

Signature · 1 line

2.17.1

Cornelia Huck

Applied

2018-06-27 15:06 UTC #10 lore raw

In reply to David Hildenbrand

Quoted 6 lines · On Wed, 27 Jun 2018 15:44:01 +0200 David Hildenbrand <[email protected]> wrote:

The TOD in TCG is not handled correctly:
- each CPU has its own TOD based on CPU creation time vs. a system TOD
- TOD is not migrated
- TOD timer is not restarted during migration
- CKC interrupts/TOD timer is not cleared when resetting the CKC

Quoted 6 lines

This (and a cpu creation problem for single threaded TCG) currently made CPU hotplug under TCG not work. Now it's working
The third patch also refactors TOD handling for KVM (moved into a new TOD device). The second patch makes sure that we can use a const pointer for the new TOD clock struct.

Thanks, applied.