clockevent: on resume program the next oneshot tick with the next actual event
Subsystems (2)
PATCH 1/1 clockevent: on resume program the next oneshot tick with the next actual event
When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with CONFIG_NO_HZ=y, the Xen clocksource has CLOCK_EVT_FEAT_ONESHOT but not CLOCK_EVT_FEAT_PERIODIC.
The issue is that on resume tick_resume_oneshot() tries to program an event for "now", e.g.
tick_program_event(ktime_get(), 1);
However further down the call chain tick_dev_program_event() then compares that expiry time with a second call to ktime_get() and discards the event if the timeout is negative -- which it always will be since some time must have passed since tick_program_event was called.
Instead of asking for an immediate event on resume, instead ask for the next actual event.
With this fix I can successfully resume a Xen domain.
--- kernel/time/tick-oneshot.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/time/tick-oneshot.c +1 -1
@@ -81,7 +81,7 @@ void tick_resume_oneshot(void)struct clock_event_device *dev = td->evtdev;clockevents_set_mode (dev,CLOCK_EVT_MODE_ONESHOT ); -tick_program_event (ktime_get (),1 ); +tick_program_event (dev->next_event,1 ); }/**
Signature · 1 line
1.5.6.5
Quoted 10 lines · Ian Campbell
When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with
CONFIG_NO_HZ=y, the Xen clocksource hasCLOCK_EVT_FEAT_ONESHOTbut notCLOCK_EVT_FEAT_PERIODIC.The issue is that on resume
tick_resume_oneshot()tries to program an event for "now", e.g.tick_program_event(ktime_get(), 1);However further down the call chain
tick_dev_program_event()then compares that
Where's that? Do you mean in clockevents_program_event():
delta = ktime_to_ns(ktime_sub(expires, now)); if (delta <= 0) return -ETIME;
?
Quoted 7 lines
expiry time with a second call to
ktime_get()and discards the event if the timeout is negative -- which it always will be since some time must have passed sincetick_program_eventwas called.Instead of asking for an immediate event on resume, instead ask for the next actual event.
What if the next event is now anyway? What timebase is it in anyway?
Quoted 2 lines
With this fix I can successfully resume a Xen domain.
That's good, but I think there's more going on here.
J
Quoted 13 lines · Jeremy Fitzhardinge
Quoted 10 lines · Ian Campbell
When resuming a Xen domU we were seeing an issue where the timer ticks never seemed to start up again. This was with
CONFIG_NO_HZ=y, the Xen clocksource hasCLOCK_EVT_FEAT_ONESHOTbut notCLOCK_EVT_FEAT_PERIODIC.The issue is that on resume
tick_resume_oneshot()tries to program an event for "now", e.g.tick_program_event(ktime_get(), 1);However further down the call chain
tick_dev_program_event()then compares thatWhere's that? Do you mean in
clockevents_program_event():
Quoted 7 lines
delta = ktime_to_ns(ktime_sub(expires, now)); if (delta <= 0) return -ETIME;?
Yes that's the one.
Quoted 4 lines
Quoted 2 lines
With this fix I can successfully resume a Xen domain.
That's good, but I think there's more going on here.
Hmm, yes I think so too. I misread tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xen set_next_event hook get called which I thought wasn't getting called earlier.
Turns out the virtual timer IRQ isn't getting reinitialised before tick_oneshot_resume runs so we are just missing the interrupt, doh!
---
Subject: xen: resume interrupts before system devices.
otherwise the first timer interrupt after resume is missed and we never get another.
drivers/xen/manage.c +3 -3
@@ -68,15 +68,15 @@ static int xen_suspend(void *data)gnttab_resume ();xen_mm_unpin_all (); -sysdev_resume (); -device_power_up (PMSG_RESUME ); -if (!*cancelled) {xen_irq_resume ();xen_console_resume ();xen_timer_resume (); } +sysdev_resume (); +device_power_up (PMSG_RESUME ); +return 0 ; }
Ian.
Quoted 8 lines
J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Ian Campbell
Hmm, yes I think so too. I misread
tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xenset_next_eventhook get called which I thought wasn't getting called earlier.Turns out the virtual timer IRQ isn't getting reinitialised before
tick_oneshot_resumeruns so we are just missing the interrupt, doh!
While that ordering is a bug, I'm still not sure it completely explains what we're seeing here.
In drivers/xen/manage.c:do_suspend() we call clock_was_set(), which has the specific effect of causing all the timer events to get retriggered on all cpus. This is necessary because we don't unplug/replug all the cpus, and the normal sysdev_resume() timer resume only resumes the current cpu (which is cpu 0 in this case). It also deals with the clocksource timebase shifting, as it will over suspend/resume (esp suspend/reboot/resume, or suspend/migrate/resume). Your patch will only re-trigger the next cpu0 timer event, and leave the rest hanging without a next event.
So the question is why does your patch help?
I'm seeing much worse symptoms on my test machine: the resumed domain is just sitting there spinning dead with 100% cpu use. I don't know if this is related or something else.
J
Quoted 46 lines
Subject: xen: resume interrupts before system devices.
otherwise the first timer interrupt after resume is missed and we never get another.
Signed-off-by Ian Campbell <[email protected]>drivers/xen/manage.c +3
-3@@ -68,15 +68,15 @@ static int xen_suspend(void *data)gnttab_resume ();xen_mm_unpin_all (); -sysdev_resume (); -device_power_up (PMSG_RESUME ); -if (!*cancelled) {xen_irq_resume ();xen_console_resume ();xen_timer_resume (); } +sysdev_resume (); +device_power_up (PMSG_RESUME ); +return 0 ; }Ian.
Quoted 8 lines
J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 7 lines · Ian Campbell
Hmm, yes I think so too. I misread
tick_dev_program_event(), it seems like it Does The Right Thing and I do see the Xenset_next_eventhook get called which I thought wasn't getting called earlier.Turns out the virtual timer IRQ isn't getting reinitialised before
tick_oneshot_resumeruns so we are just missing the interrupt, doh!
Quoted 6 lines
While that ordering is a bug, I'm still not sure it completely explains what we're seeing here.
In drivers/xen/manage.c:do_suspend() we call
clock_was_set(), which has the specific effect of causing all the timer events to get retriggered on all cpus.
Not if CONFIG_HIGH_RES_TIMERS is not set, which I don't have. If I set it then things work as expected even without the patch.
When CONFIG_HIGH_RES_TIMERS is set though the call to clock_was_set ends up in hr_timer_force_reprogram, I'm not clear what the relationship between hrtimers and ticks is, they both seem to call down to the oneshot code eventually, so they must coexist somehow...
Ian.
Quoted 63 lines
This is necessary because we don't unplug/replug all the
cpus, and the normalsysdev_resume()timer resume only resumes the current cpu (which is cpu 0 in this case). It also deals with the clocksource timebase shifting, as it will over suspend/resume (esp suspend/reboot/resume, or suspend/migrate/resume). Your patch will only re-trigger the next cpu0 timer event, and leave the rest hanging without a next event.So the question is why does your patch help?
I'm seeing much worse symptoms on my test machine: the resumed domain is just sitting there spinning dead with 100% cpu use. I don't know if this is related or something else.
J
Quoted 46 lines
Subject: xen: resume interrupts before system devices.
otherwise the first timer interrupt after resume is missed and we never get another.
Signed-off-by Ian Campbell <[email protected]>drivers/xen/manage.c +3
-3@@ -68,15 +68,15 @@ static int xen_suspend(void *data)gnttab_resume ();xen_mm_unpin_all (); -sysdev_resume (); -device_power_up (PMSG_RESUME ); -if (!*cancelled) {xen_irq_resume ();xen_console_resume ();xen_timer_resume (); } +sysdev_resume (); +device_power_up (PMSG_RESUME ); +return 0 ; }Ian.
Quoted 8 lines
J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/