Avoid triggering the softlockup BUG when offline for too long.
PATCH 1/1 Avoid triggering the softlockup BUG when offline for too long.
After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.
However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
# HG changeset patch
# User [email protected]
# Date 1164376767 18000
# Node ID 0f235d94eeabbca64c14ae6d5ae3708870522f60
# Parent 47fcd5f768fef50cba2fc6dbadc7b75de55e88a5
[LINUX] Avoid triggering the softlockup BUG when offline for too long.
After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.
However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.
linux-2.6-xen-sparse/arch/i386/kernel/time-xen.c +14 -1
@@ -129,6 +129,8 @@ static DEFINE_PER_CPU(u64, processed_sys/* How much CPU time was spent blocked and how much was 'stolen'? */ static DEFINE_PER_CPU(u64, processed_stolen_time); static DEFINE_PER_CPU(u64, processed_blocked_time); +/* How much time did we spend offline? */ +static DEFINE_PER_CPU(u64, offline_time);/* Current runstate of each CPU (updated automatically by the hypervisor). */ static DEFINE_PER_CPU(struct vcpu_runstate_info, runstate);
@@ -607,7 +609,7 @@ EXPORT_SYMBOL(profile_pc);irqreturn_t timer_interrupt (int irq,void *dev_id,struct pt_regs *regs) { -s64 delta, delta_cpu, stolen, blocked; +s64 delta, delta_cpu, stolen, blocked, offline;u64 sched_time;int i, cpu =smp_processor_id ();struct shadow_time_info *shadow = &per_cpu (shadow_time, cpu);
@@ -636,6 +638,8 @@ irqreturn_t timer_interrupt(int irq, voiper_cpu (processed_stolen_time, cpu); blocked = runstate->time[RUNSTATE_blocked] -per_cpu (processed_blocked_time, cpu); + offline = runstate->time[RUNSTATE_offline] - +per_cpu (offline_time, cpu);barrier (); }while (sched_time != runstate->state_entry_time); }while (!time_values_up_to_date (cpu));
@@ -710,6 +714,13 @@ irqreturn_t timer_interrupt(int irq, voi (cputime_t)delta_cpu); } +/* We know we were offline for too long, avoid triggering the +* softlockup_tick bug */ +if ((offline >10 *HZ )) { +touch_softlockup_watchdog (); +per_cpu (offline_time, cpu) += offline; + } +/* Local timer processing (see update_process_times()). */ run_local_timers ();if (rcu_pending (cpu))
@@ -734,6 +745,8 @@ static void init_missing_ticks_accountin runstate->time[RUNSTATE_blocked];per_cpu (processed_stolen_time, cpu) = runstate->time[RUNSTATE_runnable] + + runstate->time[RUNSTATE_offline]; +per_cpu (offline_time, cpu) = runstate->time[RUNSTATE_offline]; }
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 8 lines · Glauber de Oliveira Costa
After being offline for a long time, the softlockup watchdog triggers a
BUG()on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.However, by inspecting the contents of
RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.Signed-off-by Glauber de Oliveira Costa <[email protected]>
Would 'stolen' not be a good enough thing to test? Presumably this is really just dealing with xm pause/unpause (a single long offline) so this simpler fix would work just as well?
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 4 lines · Glauber de Oliveira Costa
After being offline for a long time, the softlockup watchdog triggers a
BUG()on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.
Quoted 8 lines
Quoted 4 lines
However, by inspecting the contents of
RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.Signed-off-by Glauber de Oliveira Costa <[email protected]>Would 'stolen' not be a good enough thing to test? Presumably this is really just dealing with xm pause/unpause (a single long offline) so this simpler fix would work just as well?
I thought about it, but I'm not 100 % sure. Reasons I had for not using stolen, were basically:
* Conceptually, (maybe not in practice) stolen could grow due to runnable time only.
* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of NS_PER_TICK chuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.
Waiting for your comments on this,
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"
Quoted 5 lines
* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of
NS_PER_TICKchuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.
FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.
Signature · 5 lines
Glauber de Oliveira Costa. "Free as in Freedom" Add your comments to GPLv3 at: http://gplv3.fsf.org/comments/gplv3-draft-2.html
Quoted 8 lines · Glauber de Oliveira Costa
Quoted 5 lines
* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of
NS_PER_TICKchuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.
That doesn't make sense. Processed_stolen_time should lag at most 1 jiffy behind actual stolen time. So you still need to accumulate at least 10*HZ-1 jiffies of stolen time in one go to end up touching the softlockup watchdog. As far as I can see, anyway. What workload did you run to test using stolen time?
-- Keir
Quoted 8 lines · Keir Fraser
Quoted 2 lines
FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.
That doesn't make sense.
Processed_stolen_timeshould lag at most 1 jiffy behind actual stolen time. So you still need to accumulate at least 10*HZ-1 jiffies of stolen time in one go to end up touching the softlockup watchdog. As far as I can see, anyway. What workload did you run to test using stolen time?
Thanks for pointing it, Keir.
After going back to it, I found it to be a small mistake of mine. I was calling the softlockup watchdog before accounting stolen ticks. Calling it after it does the trick.
I'll resend the patch soon.
Signature · 3 lines
Glauber de Oliveira Costa Red Hat Inc. "Free as in Freedom"