Skip to contentSkip to conversation

clockevent: on resume program the next oneshot tick with the next actual event

PATCH In review Ian Campbell 3 participants 5 messages Mar 25, 2009 – Mar 26, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 clockevent: on resume program the next oneshot tick with the next actual event

When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with CONFIG_NO_HZ=y, the Xen clocksource has CLOCK_EVT_FEAT_ONESHOT but not CLOCK_EVT_FEAT_PERIODIC.

The issue is that on resume tick_resume_oneshot() tries to program an event for "now", e.g.

	tick_program_event(ktime_get(), 1);

However further down the call chain tick_dev_program_event() then compares that expiry time with a second call to ktime_get() and discards the event if the timeout is negative -- which it always will be since some time must have passed since tick_program_event was called.

Instead of asking for an immediate event on resume, instead ask for the next actual event.

With this fix I can successfully resume a Xen domain.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jeremy Fitzhardinge <[email protected]>
Cc Alex.Zeffertt <[email protected]>
---
 kernel/time/tick-oneshot.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/time/tick-oneshot.c +1 -1
index 2e8de67..8b2c8ed 100644
@@ -81,7 +81,7 @@ void tick_resume_oneshot(void)
 	struct clock_event_device *dev = td->evtdev;
 
 	clockevents_set_mode(dev, CLOCK_EVT_MODE_ONESHOT);
-	tick_program_event(ktime_get(), 1);
+	tick_program_event(dev->next_event, 1);
 }
 
 /**
Signature · 1 line
1.5.6.5
#1 lore raw
Quoted 10 lines · Ian Campbell

When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with CONFIG_NO_HZ=y, the Xen clocksource has CLOCK_EVT_FEAT_ONESHOT but not CLOCK_EVT_FEAT_PERIODIC.

The issue is that on resume tick_resume_oneshot() tries to program an event for "now", e.g.

	tick_program_event(ktime_get(), 1);

However further down the call chain tick_dev_program_event() then compares that

Where's that? Do you mean in clockevents_program_event():

	delta = ktime_to_ns(ktime_sub(expires, now));

	if (delta <= 0)
		return -ETIME;

?

Quoted 7 lines

expiry time with a second call to ktime_get() and discards the event if the timeout is negative -- which it always will be since some time must have passed since tick_program_event was called.

Instead of asking for an immediate event on resume, instead ask for the next actual event.

What if the next event is now anyway? What timebase is it in anyway?

Quoted 2 lines

With this fix I can successfully resume a Xen domain.

That's good, but I think there's more going on here.

J

#2 lore raw
Quoted 13 lines · Jeremy Fitzhardinge
Quoted 10 lines · Ian Campbell

When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with CONFIG_NO_HZ=y, the Xen clocksource has CLOCK_EVT_FEAT_ONESHOT but not CLOCK_EVT_FEAT_PERIODIC.

The issue is that on resume tick_resume_oneshot() tries to program an event for "now", e.g.

	tick_program_event(ktime_get(), 1);

However further down the call chain tick_dev_program_event() then compares that

Where's that? Do you mean in clockevents_program_event():

Quoted 7 lines
	delta = ktime_to_ns(ktime_sub(expires, now));

	if (delta <= 0)
		return -ETIME;

?

Yes that's the one.

Quoted 4 lines
Quoted 2 lines

With this fix I can successfully resume a Xen domain.

That's good, but I think there's more going on here.

Hmm, yes I think so too. I misread tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xen set_next_event hook get called which I thought wasn't getting called earlier.

Turns out the virtual timer IRQ isn't getting reinitialised before tick_oneshot_resume runs so we are just missing the interrupt, doh!

---

Subject: xen: resume interrupts before system devices.

otherwise the first timer interrupt after resume is missed and we never get another.

drivers/xen/manage.c +3 -3
index 0489ea2..5269bb4 100644
@@ -68,15 +68,15 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
-	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
 	if (!*cancelled) {
 		xen_irq_resume();
 		xen_console_resume();
 		xen_timer_resume();
 	}
 
+	sysdev_resume();
+	device_power_up(PMSG_RESUME);
+
 	return 0;
 }
 

Ian.

Quoted 8 lines

J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 7 lines · Ian Campbell

Hmm, yes I think so too. I misread tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xen set_next_event hook get called which I thought wasn't getting called earlier.

Turns out the virtual timer IRQ isn't getting reinitialised before tick_oneshot_resume runs so we are just missing the interrupt, doh!

While that ordering is a bug, I'm still not sure it completely explains what we're seeing here.

In drivers/xen/manage.c:do_suspend() we call clock_was_set(), which has the specific effect of causing all the timer events to get retriggered on all cpus. This is necessary because we don't unplug/replug all the cpus, and the normal sysdev_resume() timer resume only resumes the current cpu (which is cpu 0 in this case). It also deals with the clocksource timebase shifting, as it will over suspend/resume (esp suspend/reboot/resume, or suspend/migrate/resume). Your patch will only re-trigger the next cpu0 timer event, and leave the rest hanging without a next event.

So the question is why does your patch help?

I'm seeing much worse symptoms on my test machine: the resumed domain is just sitting there spinning dead with 100% cpu use. I don't know if this is related or something else.

J

Quoted 46 lines

Subject: xen: resume interrupts before system devices.

otherwise the first timer interrupt after resume is missed and we never get another.

drivers/xen/manage.c +3 -3
index 0489ea2..5269bb4 100644
@@ -68,15 +68,15 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
-	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
 	if (!*cancelled) {
 		xen_irq_resume();
 		xen_console_resume();
 		xen_timer_resume();
 	}
 
+	sysdev_resume();
+	device_power_up(PMSG_RESUME);
+
 	return 0;
 }
 

Ian.

Quoted 8 lines

J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 7 lines · Ian Campbell

Hmm, yes I think so too. I misread tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xen set_next_event hook get called which I thought wasn't getting called earlier.

Turns out the virtual timer IRQ isn't getting reinitialised before tick_oneshot_resume runs so we are just missing the interrupt, doh!

Quoted 6 lines

While that ordering is a bug, I'm still not sure it completely explains what we're seeing here.

In drivers/xen/manage.c:do_suspend() we call clock_was_set(), which has the specific effect of causing all the timer events to get retriggered on all cpus.

Not if CONFIG_HIGH_RES_TIMERS is not set, which I don't have. If I set it then things work as expected even without the patch.

When CONFIG_HIGH_RES_TIMERS is set though the call to clock_was_set ends up in hr_timer_force_reprogram, I'm not clear what the relationship between hrtimers and ticks is, they both seem to call down to the oneshot code eventually, so they must coexist somehow...

Ian.

Quoted 63 lines

This is necessary because we don't unplug/replug all the
cpus, and the normal sysdev_resume() timer resume only resumes the current cpu (which is cpu 0 in this case). It also deals with the clocksource timebase shifting, as it will over suspend/resume (esp suspend/reboot/resume, or suspend/migrate/resume). Your patch will only re-trigger the next cpu0 timer event, and leave the rest hanging without a next event.

So the question is why does your patch help?

I'm seeing much worse symptoms on my test machine: the resumed domain is just sitting there spinning dead with 100% cpu use. I don't know if this is related or something else.

J

Quoted 46 lines

Subject: xen: resume interrupts before system devices.

otherwise the first timer interrupt after resume is missed and we never get another.

drivers/xen/manage.c +3 -3
index 0489ea2..5269bb4 100644
@@ -68,15 +68,15 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
-	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
 	if (!*cancelled) {
 		xen_irq_resume();
 		xen_console_resume();
 		xen_timer_resume();
 	}
 
+	sysdev_resume();
+	device_power_up(PMSG_RESUME);
+
 	return 0;
 }
 

Ian.

Quoted 8 lines

J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/