Skip to contentSkip to conversation

qemu-timer/mc146818rtc: fix timer issues related to clock changes

PATCH In review Paul Donohue 2 participants 8 messages Jun 12, 2015 – Jun 18, 2015

lore
Share & tools
lore
#0 lore raw

cover qemu-timer/mc146818rtc: fix timer issues related to clock changes

If the host clock jumps forward or backward without resetting the mc146818rtc's periodic timer, the periodic timer interrupts will either fire rapidly (consuming host CPU and blocking operation of the VM) until the timer catches up to the new host clock, or will stall until the host clock catches up with the prior date used by the periodic timer.

A "reset notifier" mechanism was introduced a while back that resets the mc146818rtc's periodic timer if the host clock jumps backward while the VM is running. However, the timer is not reset if the host clock jumps forward while the VM is running, or if the clock jumps either forward or backward when loading a snapshot or migration.

In one test case, when loading a 3 week old snapshot, the timer loop consumed CPU and blocked the VM for nearly 90 seconds.

The following patches fix this issue by resetting the timer in these cases.

Paul Donohue (2):
qemu-timer: Call clock reset notifiers on large forward jumps
mc146818rtc: Reset the periodic timer on resume from snapshot

 hw/timer/mc146818rtc.c | 6 ++++++
 include/qemu/timer.h   | 9 +++++++++
 qemu-timer.c           | 2 +-
 3 files changed, 16 insertions(+), 1 deletion(-)
#1 lore raw

PATCH 1/2 qemu-timer: Call clock reset notifiers on forward jumps

Commit 691a0c9c introduced a mechanism by which QEMU_CLOCK_HOST can notify other parts of the emulator when the host clock has jumped backward. This is used to avoid stalling timers that were scheduled based on the host clock.

However, if the host clock jumps forward, then timers that were scheduled based on the host clock may fire rapidly and cause other problems. For example, the mc146818rtc periodic timer will block execution of the VM and consume host CPU while firing every interrupt for the time period that was skipped by the host clock.

To correct that problem, this commit fires the reset notification if the host clock jumps forward by more than a hard-coded limit. The limit is currently set to a value of 60 seconds, which should be small enough to prevent excessive timer loops, but large enough to avoid frequent resets in idle VMs.

---
 include/qemu/timer.h | 9 +++++++++
 qemu-timer.c         | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
include/qemu/timer.h +8 -0
index e5bd494..0193036 100644
@@ -787,6 +787,14 @@ static inline int64_t get_ticks_per_sec(void)
     return 1000000000LL;
 }
 
+static inline int64_t get_max_clock_jump(void)
+{
+    // This should be small enough to prevent excessive interrupts from being
+    // generated by the RTC on clock jumps, but large enough to avoid frequent
+    // unnecessary resets in idle VMs.
+    return 60 * get_ticks_per_sec();
+}
+
 /*
 *   * Low level clock functions
 *     */
qemu-timer.c +1 -1
index 5741f0d..d27be3d 100644
@@ -573,7 +573,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
         now = get_clock_realtime();
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last || now > (last + get_max_clock_jump())) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
Signature · 1 line
2.1.4
#5 lore raw

PATCH 2/2 mc146818rtc: Reset the periodic timer on load

In reply to Paul Donohue

When loading a VM from a snapshot or migration, clock changes can cause the periodic timer to stall or loop rapidly.

qemu-timer has a reset notifier mechanism that is used to avoid timer stalls or loops if the host clock changes while the VM is running when using QEMU_CLOCK_HOST. However, when loading a snapshot or migration, qemu-timer is initialized and fires the reset notifier before mc146818rtc is initialized and has registered its reset handler. In addition, this mechanism isn't used when using QEMU_CLOCK_REALTIME, which might also change when loading a snapshot or migration.

To correct that problem, this commit resets the periodic timer after loading from a snapshot or migration if the clock has either jumped backward or has jumped forward by more than the clock jump limit that is used by the reset notifier code in qemu-timer.

---
 hw/timer/mc146818rtc.c | 6 ++++++
 1 file changed, 6 insertions(+)
hw/timer/mc146818rtc.c +6 -0
index f2b77fa..68cf1f0 100644
@@ -723,6 +723,12 @@ static int rtc_post_load(void *opaque, int version_id)
         check_update_timer(s);
     }
 
+    uint64_t now = qemu_clock_get_ns(rtc_clock);
+    if (now < (s->next_periodic_time - get_ticks_per_sec()) ||
+        now > (s->next_periodic_time + get_max_clock_jump())) {
+        periodic_timer_update(s, qemu_clock_get_ns(rtc_clock));
+    }
+
 #ifdef TARGET_I386
     if (version_id >= 2) {
         if (s->lost_tick_policy == LOST_TICK_POLICY_SLEW) {
Signature · 1 line
2.1.4
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] qemu-timer: Call clock reset notifiers on forward jumps

In reply to Paul Donohue

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 794

Quoted 5 lines · Paul Donohue
include/qemu/timer.h +5 -0
+static inline int64_t get_max_clock_jump(void)
+{
+    // This should be small enough to prevent excessive interrupts from being
+    // generated by the RTC on clock jumps, but large enough to avoid frequent
+    // unnecessary resets in idle VMs.

This is not how comments are layed out in QEMU...

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 800

Quoted 6 lines
include/qemu/timer.h +3 -0
+    return 60 * get_ticks_per_sec();
+}
+
 /*
 *   * Low level clock functions
 *     */

... and it also looks like your editor has mangled this patch. Nevertheless, I've applied it to my local branch and will send it out for inclusion after some more testing.

Paolo

Quoted 4 lines
qemu-timer.c +0 -0
index 5741f0d..d27be3d 100644
#6 lore raw

Re: [Qemu-devel] [PATCH 2/2] mc146818rtc: Reset the periodic timer on load

In reply to Paul Donohue
Quoted 15 lines · Paul Donohue

To correct that problem, this commit resets the periodic timer after loading from a snapshot or migration if the clock has either jumped backward or has jumped forward by more than the clock jump limit that is used by the reset notifier code in qemu-timer.

---
 hw/timer/mc146818rtc.c | 6 ++++++
 1 file changed, 6 insertions(+)
hw/timer/mc146818rtc.c +0 -0
index f2b77fa..68cf1f0 100644

hw/timer/mc146818rtc.c static int rtc_post_load(void *opaque, int version_id) · line 727

Quoted 5 lines
hw/timer/mc146818rtc.c +2 -0
@@ -723,6 +723,12 @@ static int rtc_post_load(void *opaque, int version_id)
         check_update_timer(s);
     }
 
+    uint64_t now = qemu_clock_get_ns(rtc_clock);
+    if (now < (s->next_periodic_time - get_ticks_per_sec()) ||

What is the reason for the "- get_ticks_per_sec()" adjustment? Can I just remove it?

Paolo

Quoted 1 line
hw/timer/mc146818rtc.c +1 -0
+        now > (s->next_periodic_time + get_max_clock_jump())) {
#3 lore raw

Re: [Qemu-devel] [PATCH 1/2] qemu-timer: Call clock reset notifiers on forward jumps

In reply to Paolo Bonzini
Quoted 7 lines · Paolo Bonzini

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 794

Quoted 5 lines · Paul Donohue
include/qemu/timer.h +5 -0
+static inline int64_t get_max_clock_jump(void)
+{
+    // This should be small enough to prevent excessive interrupts from being
+    // generated by the RTC on clock jumps, but large enough to avoid frequent
+    // unnecessary resets in idle VMs.

This is not how comments are layed out in QEMU...

Oops. Sorry, I'm not sure how I missed that. Force of habit from other projects, I guess...

Quoted 7 lines

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 800

Quoted 6 lines
include/qemu/timer.h +3 -0
+    return 60 * get_ticks_per_sec();
+}
+
 /*
 *   * Low level clock functions
 *     */

... and it also looks like your editor has mangled this patch.

Ugh, yes. I do remember noticing that, and I thought I fixed it, but apparently not.

I will correct and send an updated patch.

#4 lore raw

Re: [Qemu-devel] [PATCH 1/2] qemu-timer: Call clock reset notifiers on forward jumps

Quoted 18 lines · Paul Donohue
Quoted 7 lines · Paolo Bonzini

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 794

Quoted 5 lines · Paul Donohue
include/qemu/timer.h +5 -0
+static inline int64_t get_max_clock_jump(void)
+{
+    // This should be small enough to prevent excessive interrupts from being
+    // generated by the RTC on clock jumps, but large enough to avoid frequent
+    // unnecessary resets in idle VMs.

This is not how comments are layed out in QEMU...

Oops. Sorry, I'm not sure how I missed that. Force of habit from other projects, I guess...

Quoted 7 lines

include/qemu/timer.h static inline int64_t get_ticks_per_sec(void) · line 800

Quoted 6 lines
include/qemu/timer.h +3 -0
+    return 60 * get_ticks_per_sec();
+}
+
 /*
 *   * Low level clock functions
 *     */

... and it also looks like your editor has mangled this patch.

Quoted 4 lines

Ugh, yes. I do remember noticing that, and I thought I fixed it, but apparently not.

I will correct and send an updated patch.

No problem, I fixed this already.

Paolo

#7 lore raw

Re: [Qemu-devel] [PATCH 2/2] mc146818rtc: Reset the periodic timer on load

In reply to Paolo Bonzini
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Paul Donohue

To correct that problem, this commit resets the periodic timer after loading from a snapshot or migration if the clock has either jumped backward or has jumped forward by more than the clock jump limit that is used by the reset notifier code in qemu-timer.

hw/timer/mc146818rtc.c +0 -0
@@ -723,6 +723,12 @@ static int rtc_post_load(void *opaque, int version_id)
         check_update_timer(s);
     }
 
Quoted 5 lines

hw/timer/mc146818rtc.c static int rtc_post_load(void *opaque, int version_id) · line 727

Quoted 2 lines
hw/timer/mc146818rtc.c +2 -0
+    uint64_t now = qemu_clock_get_ns(rtc_clock);
+    if (now < (s->next_periodic_time - get_ticks_per_sec()) ||

What is the reason for the "- get_ticks_per_sec()" adjustment? Can I just remove it?

Short answer: Yes, you can remove it.

Long answer:
In the other (qemu-timer) patch, I thought it was a good idea to try to avoid calling the clock reset notifiers in cases where the clock "jumped" because the VM was idle, and err on the side of not calling the notifiers for real but small host clock changes. At that layer of the abstraction, it is not clear how any particular RTC implementation might use the reset notification, so I didn't want to assume it was safe to trigger the notification too frequently.

That thought process carried over when I patched mc146818rtc.c, and I was trying to avoid calling periodic_timer_update() in any cases where it wasn't strictly needed. At the time when a snapshot is taken, "now" should generally be less than "next_periodic_time" by up to one period worth of time. In the case where the snapshot is loaded when "now" is still less than "next_periodic_time", "now < s->next_periodic_time" would not necessarily indicate that the clock has jumped backward. Calculating the period so we could evaluate
"now < s->next_periodic_time - period" would involve duplicating a lot of the code from periodic_timer_update(), which I thought was a bad idea. So, I simply assumed that the period would always be less than 1 second, and used "- get_ticks_per_sec()" to err on the side of skipping the call to periodic_timer_update().

However, now that I'm looking at it again, calling periodic_timer_update() when "now" is less than "next_periodic_time" by less than one period would have no effect anyway, so the
"- get_ticks_per_sec()" adjustment is unnecessary, and it was silly for me to try to avoid a call to to periodic_timer_update() in that case.

Thanks!