Skip to contentSkip to conversation

x86/hvm/rtc: preserved guest RTC offset during suspend/resume/migrate

PATCH In review Paul Durrant 3 participants 4 messages Dec 18, 2019 – Dec 23, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/hvm/rtc: preserved guest RTC offset during suspend/resume/migrate

The emulated RTC is synchronized with the PV wallclock; any write to the RTC will update struct domain's 'time_offset_seconds' field and call update_domain_wallclock().

However, the value of 'time_offset_seconds' is not preserved in any save record and indeed, when the RTC save record is loaded, the CMOS values will be updated based on an offset value which may or may not have been set by the toolstack [1]. This may result in making bogus values available to the guest and messing up any calculations done in the call to alarm_timer_update() at the end of rtc_load().

This patch extends the RTC save record to contain an offset value, which will be zero filled on load of an older record. The 'time_offset_secoonds' field in struct domain is also modified into a 'time_offset' struct, containing a 'seconds' field and a boolean 'set' field.

The code in rtc_load() then uses the new value in the save record to update the value of struct domain's 'time_offset.seconds' unless 'time_offset.set' is true, which will only be the case if the toolstack has already performed a XEN_DOMCTL_settimeoffset.

[1] There is currently no way for a toolstack to read the value of 'time_offset_seconds' from struct domain. In the past, any hope of preservation of the value across a guest life-cycle operation was based on relying on qemu-dm to write a value into xenstore whenever the RTC was updated, in response to an IOREQ with type IOREQ_TYPE_TIMEOFFSET being sent by Xen; see:

https://xenbits.xen.org/gitweb/?p=qemu-xen-traditional.git;a=blob;f=i386-dm/helper2.c#l457

but this behaviour was never forward-ported into upstream QEMU, which completely ignores that IOREQ type.
In either case, nothing in xl or libxl ever samples the value of RTC offset from xenstore so any offset adjustment to a non-zero value performed by the guest (which in the case of Windows is highly likely as it normally writes RTC in local time, whereas Xen maintains time in UTC) is completely lost with the de-facto toolstack, and always has been. Instead, PV drivers are relied upon to paper over this gaping hole.

---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

---
 xen/arch/arm/platform_hypercall.c      |  2 +-
 xen/arch/arm/time.c                    |  3 ++-
 xen/arch/arm/vtimer.c                  |  4 ++--
 xen/arch/x86/hvm/rtc.c                 | 12 ++++++++++--
 xen/arch/x86/time.c                    |  3 ++-
 xen/common/time.c                      |  6 +++---
 xen/include/public/arch-x86/hvm/save.h |  2 ++
 xen/include/xen/sched.h                |  5 ++++-
 8 files changed, 26 insertions(+), 11 deletions(-)
xen/arch/arm/platform_hypercall.c +1 -1
index 5aab856ce7..8efac7ee60 100644
@@ -53,7 +53,7 @@ long do_platform_op(XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op)
         if ( likely(!op->u.settime64.mbz) )
             do_settime(op->u.settime64.secs,
                        op->u.settime64.nsecs,
-                       op->u.settime64.system_time + SECONDS(d->time_offset_seconds));
+                       op->u.settime64.system_time + SECONDS(d->time_offset.seconds));
         else
             ret = -EINVAL;
         break;
xen/arch/arm/time.c +2 -1
index 739bcf186c..b0021c2c69 100644
@@ -353,7 +353,8 @@ void update_vcpu_system_time(struct vcpu *v)
 
 void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
 {
-    d->time_offset_seconds = time_offset_seconds;
+    d->time_offset.seconds = time_offset_seconds;
+    d->time_offset.set = true;
     /* XXX update guest visible wallclock time */
 }
 
xen/arch/arm/vtimer.c +2 -2
index e6aebdac9e..240a850b6e 100644
@@ -64,8 +64,8 @@ int domain_vtimer_init(struct domain *d, struct xen_arch_domainconfig *config)
 {
     d->arch.phys_timer_base.offset = NOW();
     d->arch.virt_timer_base.offset = READ_SYSREG64(CNTPCT_EL0);
-    d->time_offset_seconds = ticks_to_ns(d->arch.virt_timer_base.offset - boot_count);
-    do_div(d->time_offset_seconds, 1000000000);
+    d->time_offset.seconds = ticks_to_ns(d->arch.virt_timer_base.offset - boot_count);
+    do_div(d->time_offset.seconds, 1000000000);
 
     config->clock_frequency = timer_dt_clock_frequency;
 
xen/arch/x86/hvm/rtc.c +10 -2
index 42339682e8..bb41efe84a 100644
@@ -594,7 +594,7 @@ static void rtc_set_time(RTCState *s)
 
     /* We use the guest's setting of the RTC to define the local-time 
      * offset for this domain. */
-    d->time_offset_seconds += (after - before);
+    d->time_offset.seconds += (after - before);
     update_domain_wallclock_time(d);
     /* Also tell qemu-dm about it so it will be remembered for next boot. */
     send_timeoffset_req(after - before);
@@ -747,8 +747,10 @@ static int rtc_save(struct vcpu *v, hvm_domain_context_t *h)
         return 0;
 
     spin_lock(&s->lock);
+    s->hw.rtc_offset = d->time_offset.seconds;
     rc = hvm_save_entry(RTC, 0, h, &s->hw);
     spin_unlock(&s->lock);
+
     return rc;
 }
 
@@ -763,7 +765,7 @@ static int rtc_load(struct domain *d, hvm_domain_context_t *h)
     spin_lock(&s->lock);
 
     /* Restore the registers */
-    if ( hvm_load_entry(RTC, h, &s->hw) != 0 )
+    if ( hvm_load_entry_zeroextend(RTC, h, &s->hw) != 0 )
     {
         spin_unlock(&s->lock);
         return -EINVAL;
@@ -771,6 +773,12 @@ static int rtc_load(struct domain *d, hvm_domain_context_t *h)
 
     /* Reset the wall-clock time.  In normal running, this runs with host 
      * time, so let's keep doing that. */
+    if ( !d->time_offset.set )
+    {
+        d->time_offset.seconds = s->hw.rtc_offset;
+        update_domain_wallclock_time(d);
+    }
+
     s->current_tm = gmtime(get_localtime(d));
     rtc_copy_date(s);
 
xen/arch/x86/time.c +2 -1
index 216169a025..f2d03933e4 100644
@@ -1190,7 +1190,8 @@ static void update_domain_rtc(void)
 
 void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
 {
-    d->time_offset_seconds = time_offset_seconds;
+    d->time_offset.seconds = time_offset_seconds;
+    d->time_offset.set = true;
     if ( is_hvm_domain(d) )
         rtc_update_clock(d);
     update_domain_wallclock_time(d);
xen/common/time.c +3 -3
index a7caea99e0..82336e2d5a 100644
@@ -105,7 +105,7 @@ void update_domain_wallclock_time(struct domain *d)
     *wc_version = version_update_begin(*wc_version);
     smp_wmb();
 
-    sec = wc_sec + d->time_offset_seconds;
+    sec = wc_sec + d->time_offset.seconds;
     shared_info(d, wc_sec)    = sec;
     shared_info(d, wc_nsec)   = wc_nsec;
 #ifdef CONFIG_X86
@@ -148,13 +148,13 @@ void do_settime(u64 secs, unsigned int nsecs, u64 system_time_base)
 unsigned long get_localtime(struct domain *d)
 {
     return wc_sec + (wc_nsec + NOW()) / 1000000000ULL
-        + d->time_offset_seconds;
+        + d->time_offset.seconds;
 }
 
 /* Return microsecs after 00:00:00 localtime, 1 January, 1970. */
 uint64_t get_localtime_us(struct domain *d)
 {
-    return (SECONDS(wc_sec + d->time_offset_seconds) + wc_nsec + NOW())
+    return (SECONDS(wc_sec + d->time_offset.seconds) + wc_nsec + NOW())
            / 1000UL;
 }
 
xen/include/public/arch-x86/hvm/save.h +2 -0
index bb8fa7c12f..b2ad3fcd74 100644
@@ -500,6 +500,8 @@ struct hvm_hw_rtc {
     /* Index register for 2-part operations */
     uint8_t cmos_index;
     uint8_t pad0;
+    /* RTC offset from host time */
+    int64_t rtc_offset;
 };
 
 DECLARE_HVM_SAVE_TYPE(RTC, 11, struct hvm_hw_rtc);
xen/include/xen/sched.h +4 -1
index 9f7bc69293..94add37990 100644
@@ -406,7 +406,10 @@ struct domain
     /* Domain is paused by controller software? */
     int              controller_pause_count;
 
-    int64_t          time_offset_seconds;
+    struct {
+        int64_t seconds;
+        bool set;
+    } time_offset;
 
 #ifdef CONFIG_HAS_PCI
     struct list_head pdev_list;
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/1
#1 lore raw
Quoted 40 lines · Paul Durrant

The emulated RTC is synchronized with the PV wallclock; any write to the RTC will update struct domain's 'time_offset_seconds' field and call update_domain_wallclock().

However, the value of 'time_offset_seconds' is not preserved in any save record and indeed, when the RTC save record is loaded, the CMOS values will be updated based on an offset value which may or may not have been set by the toolstack [1]. This may result in making bogus values available to the guest and messing up any calculations done in the call to alarm_timer_update() at the end of rtc_load().

This patch extends the RTC save record to contain an offset value, which will be zero filled on load of an older record. The 'time_offset_secoonds' field in struct domain is also modified into a 'time_offset' struct, containing a 'seconds' field and a boolean 'set' field.

The code in rtc_load() then uses the new value in the save record to update the value of struct domain's 'time_offset.seconds' unless 'time_offset.set' is true, which will only be the case if the toolstack has already performed a XEN_DOMCTL_settimeoffset.

[1] There is currently no way for a toolstack to read the value of 'time_offset_seconds' from struct domain. In the past, any hope of preservation of the value across a guest life-cycle operation was based on relying on qemu-dm to write a value into xenstore whenever the RTC was updated, in response to an IOREQ with type IOREQ_TYPE_TIMEOFFSET being sent by Xen; see:

https://xenbits.xen.org/gitweb/?p=qemu-xen-traditional.git;a=blob;f=i386-dm/helper2.c#l457

but this behaviour was never forward-ported into upstream QEMU, which completely ignores that IOREQ type.
In either case, nothing in xl or libxl ever samples the value of RTC offset from xenstore so any offset adjustment to a non-zero value performed by the guest (which in the case of Windows is highly likely as it normally writes RTC in local time, whereas Xen maintains time in UTC) is completely lost with the de-facto toolstack, and always has been. Instead, PV drivers are relied upon to paper over this gaping hole.

Quoted 1 line

with one remark:

xen/arch/x86/hvm/rtc.c static int rtc_load(struct domain *d, hvm_domain_context_t *h) · line 780

Quoted 9 lines
xen/arch/x86/hvm/rtc.c +5 -0
@@ -771,6 +773,12 @@ static int rtc_load(struct domain *d, hvm_domain_context_t *h)
 
     /* Reset the wall-clock time.  In normal running, this runs with host 
      * time, so let's keep doing that. */
+    if ( !d->time_offset.set )
+    {
+        d->time_offset.seconds = s->hw.rtc_offset;
+        update_domain_wallclock_time(d);
+    }

It's not really clear to me which of the possible behaviors is the better one - overriding a tool stack set value with what the save record says, or (like you do) the other way around.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 60 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 19 December 2019 11:30
To: Durrant, Paul <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Volodymyr Babchuk <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH] x86/hvm/rtc: preserved guest RTC offset during suspend/resume/migrate

Quoted 8 lines · Paul Durrant

The emulated RTC is synchronized with the PV wallclock; any write to the RTC will update struct domain's 'time_offset_seconds' field and call update_domain_wallclock().

However, the value of 'time_offset_seconds' is not preserved in any save record and indeed, when the RTC save record is loaded, the CMOS values will be updated based on an offset value which may or may not have been set by the toolstack [1]. This may result in making bogus values

available

Quoted 5 lines

to the guest and messing up any calculations done in the call to alarm_timer_update() at the end of rtc_load().

This patch extends the RTC save record to contain an offset value, which will be zero filled on load of an older record. The

'time_offset_secoonds'

Quoted 6 lines

field in struct domain is also modified into a 'time_offset' struct, containing a 'seconds' field and a boolean 'set' field.

The code in rtc_load() then uses the new value in the save record to update the value of struct domain's 'time_offset.seconds' unless 'time_offset.set' is true, which will only be the case if the toolstack

has

Quoted 5 lines

already performed a XEN_DOMCTL_settimeoffset.

[1] There is currently no way for a toolstack to read the value of 'time_offset_seconds' from struct domain. In the past, any hope of preservation of the value across a guest life-cycle operation was

based

Quoted 1 line

on relying on qemu-dm to write a value into xenstore whenever the

RTC

Quoted 4 lines

was updated, in response to an IOREQ with type IOREQ_TYPE_TIMEOFFSET being sent by Xen; see:

https://xenbits.xen.org/gitweb/?p=qemu-xen-

traditional.git;a=blob;f=i386-dm/helper2.c#l457

Quoted 2 lines

but this behaviour was never forward-ported into upstream QEMU,

which

Quoted 3 lines

completely ignores that IOREQ type.
In either case, nothing in xl or libxl ever samples the value of RTC offset from xenstore so any offset adjustment to a non-zero

value

Quoted 1 line

performed by the guest (which in the case of Windows is highly

likely

Quoted 1 line

as it normally writes RTC in local time, whereas Xen maintains time

Quoted 8 lines

in

Quoted 5 lines

UTC) is completely lost with the de-facto toolstack, and always has been. Instead, PV drivers are relied upon to paper over this gaping hole.

Thanks.

Quoted 14 lines

with one remark:

Review static int rtc_load(struct domain *d,

Quoted 1 line
(quoted diff) +0 -0
@@ -771,6 +773,12 @@ static int rtc_load(struct domain *d,

hvm_domain_context_t *h)

Review static int rtc_load(struct domain *d, · line 773

Quoted 2 lines
(quoted diff) +0 -0
     /* Reset the wall-clock time.  In normal running, this runs with

host

xen/arch/x86/hvm/rtc.c static int rtc_load(struct domain *d, hvm_domain_context_t *h) · line 780

Quoted 6 lines
xen/arch/x86/hvm/rtc.c +5 -0
      * time, so let's keep doing that. */
+    if ( !d->time_offset.set )
+    {
+        d->time_offset.seconds = s->hw.rtc_offset;
+        update_domain_wallclock_time(d);
+    }
Quoted 3 lines

It's not really clear to me which of the possible behaviors is the better one - overriding a tool stack set value with what the save record says, or (like you do) the other way around.

It's not clear to me either, which is why I erred on the side of caution. I didn't want to break any out-of-tree toolstacks that might be overriding the offset early in the restore phase from a value acquired via QEMU hackery. I guess the boolean could be dispensed with if we retire the IOREQ and silently ignore the DOMCTL for HVM guests (so overrides from the aforementioned toolstacks simply have no effect).

Paul

Quoted 2 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by on v1 1/1
#3 lore raw
In reply to Paul Durrant

Hi Paul,

Quoted 52 lines · Paul Durrant

The emulated RTC is synchronized with the PV wallclock; any write to the RTC will update struct domain's 'time_offset_seconds' field and call update_domain_wallclock().

However, the value of 'time_offset_seconds' is not preserved in any save record and indeed, when the RTC save record is loaded, the CMOS values will be updated based on an offset value which may or may not have been set by the toolstack [1]. This may result in making bogus values available to the guest and messing up any calculations done in the call to alarm_timer_update() at the end of rtc_load().

This patch extends the RTC save record to contain an offset value, which will be zero filled on load of an older record. The 'time_offset_secoonds' field in struct domain is also modified into a 'time_offset' struct, containing a 'seconds' field and a boolean 'set' field.

The code in rtc_load() then uses the new value in the save record to update the value of struct domain's 'time_offset.seconds' unless 'time_offset.set' is true, which will only be the case if the toolstack has already performed a XEN_DOMCTL_settimeoffset.

[1] There is currently no way for a toolstack to read the value of 'time_offset_seconds' from struct domain. In the past, any hope of preservation of the value across a guest life-cycle operation was based on relying on qemu-dm to write a value into xenstore whenever the RTC was updated, in response to an IOREQ with type IOREQ_TYPE_TIMEOFFSET being sent by Xen; see:

https://xenbits.xen.org/gitweb/?p=qemu-xen-traditional.git;a=blob;f=i386-dm/helper2.c#l457

but this behaviour was never forward-ported into upstream QEMU, which completely ignores that IOREQ type.
In either case, nothing in xl or libxl ever samples the value of RTC offset from xenstore so any offset adjustment to a non-zero value performed by the guest (which in the case of Windows is highly likely as it normally writes RTC in local time, whereas Xen maintains time in UTC) is completely lost with the de-facto toolstack, and always has been. Instead, PV drivers are relied upon to paper over this gaping hole.

---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Quoted 4 lines
---
  xen/arch/arm/platform_hypercall.c      |  2 +-
  xen/arch/arm/time.c                    |  3 ++-
  xen/arch/arm/vtimer.c                  |  4 ++--

The Arm changes looks mostly mechanical:

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel