Skip to contentSkip to conversation

Avoid triggering the softlockup BUG when offline for too long.

PATCH In review Glauber de Oliveira Costa 3 participants 6 messages Nov 24, 2006 – Nov 29, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Avoid triggering the softlockup BUG when offline for too long.

After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.

However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.

Signature · 3 lines
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

# HG changeset patch
# User [email protected]
# Date 1164376767 18000
# Node ID 0f235d94eeabbca64c14ae6d5ae3708870522f60
# Parent 47fcd5f768fef50cba2fc6dbadc7b75de55e88a5
[LINUX] Avoid triggering the softlockup BUG when offline for too long.

After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.

However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.

linux-2.6-xen-sparse/arch/i386/kernel/time-xen.c +14 -1
diff -r 47fcd5f768fe -r 0f235d94eeab linux-2.6-xen-sparse/arch/i386/kernel/time-xen.c
@@ -129,6 +129,8 @@ static DEFINE_PER_CPU(u64, processed_sys
 /* How much CPU time was spent blocked and how much was 'stolen'? */
 static DEFINE_PER_CPU(u64, processed_stolen_time);
 static DEFINE_PER_CPU(u64, processed_blocked_time);
+/* How much time did we spend offline? */
+static DEFINE_PER_CPU(u64, offline_time);
 
 /* Current runstate of each CPU (updated automatically by the hypervisor). */
 static DEFINE_PER_CPU(struct vcpu_runstate_info, runstate);
@@ -607,7 +609,7 @@ EXPORT_SYMBOL(profile_pc);
 
 irqreturn_t timer_interrupt(int irq, void *dev_id, struct pt_regs *regs)
 {
-	s64 delta, delta_cpu, stolen, blocked;
+	s64 delta, delta_cpu, stolen, blocked, offline;
 	u64 sched_time;
 	int i, cpu = smp_processor_id();
 	struct shadow_time_info *shadow = &per_cpu(shadow_time, cpu);
@@ -636,6 +638,8 @@ irqreturn_t timer_interrupt(int irq, voi
 				per_cpu(processed_stolen_time, cpu);
 			blocked = runstate->time[RUNSTATE_blocked] -
 				per_cpu(processed_blocked_time, cpu);
+			offline = runstate->time[RUNSTATE_offline] -
+				per_cpu(offline_time, cpu);
 			barrier();
 		} while (sched_time != runstate->state_entry_time);
 	} while (!time_values_up_to_date(cpu));
@@ -710,6 +714,13 @@ irqreturn_t timer_interrupt(int irq, voi
 					    (cputime_t)delta_cpu);
 	}
 
+	/* We know we were offline for too long, avoid triggering the 
+	 * softlockup_tick bug */
+	if ((offline > 10*HZ)) {
+		touch_softlockup_watchdog();
+		per_cpu(offline_time, cpu) += offline;
+	}
+
 	/* Local timer processing (see update_process_times()). */
 	run_local_timers();
 	if (rcu_pending(cpu))
@@ -734,6 +745,8 @@ static void init_missing_ticks_accountin
 		runstate->time[RUNSTATE_blocked];
 	per_cpu(processed_stolen_time, cpu) =
 		runstate->time[RUNSTATE_runnable] +
+		runstate->time[RUNSTATE_offline];
+	per_cpu(offline_time, cpu) =
 		runstate->time[RUNSTATE_offline];
 }
 

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw
Quoted 8 lines · Glauber de Oliveira Costa

After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.

However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.

Would 'stolen' not be a good enough thing to test? Presumably this is really just dealing with xm pause/unpause (a single long offline) so this simpler fix would work just as well?

-- Keir

#2 lore raw
Quoted 9 lines · Keir Fraser
Quoted 4 lines · Glauber de Oliveira Costa

After being offline for a long time, the softlockup watchdog triggers a BUG() on our faces. This is expected, as in fact, we spent more than a fixed 10*HZ amount of time without touching the watchdog.

Quoted 8 lines
Quoted 4 lines

However, by inspecting the contents of RUNSTATE_offline, we can gain awareness of the fact, and do better than that. This patch fixes it.

Would 'stolen' not be a good enough thing to test? Presumably this is really just dealing with xm pause/unpause (a single long offline) so this simpler fix would work just as well?

I thought about it, but I'm not 100 % sure. Reasons I had for not using stolen, were basically:

* Conceptually, (maybe not in practice) stolen could grow due to runnable time only.
* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of NS_PER_TICK chuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.

Waiting for your comments on this,

Signature · 3 lines
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"
#3 lore raw
Quoted 5 lines

* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of NS_PER_TICK chuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.

FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.

Signature · 5 lines
Glauber de Oliveira Costa.
"Free as in Freedom"

Add your comments to GPLv3 at:
http://gplv3.fsf.org/comments/gplv3-draft-2.html
#4 lore raw
Quoted 8 lines · Glauber de Oliveira Costa
Quoted 5 lines

* stolen time, as well as blocked time, does not have it's corresponding per processor variable updated all in once, but in multiples of NS_PER_TICK chuncks. If we're out for too long, we could detect stolen being too great multiple times, leading to far more calls to the softlockup watchdog then we want too.

FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.

That doesn't make sense. Processed_stolen_time should lag at most 1 jiffy behind actual stolen time. So you still need to accumulate at least 10*HZ-1 jiffies of stolen time in one go to end up touching the softlockup watchdog. As far as I can see, anyway. What workload did you run to test using stolen time?

-- Keir

#5 lore raw
Quoted 8 lines · Keir Fraser
Quoted 2 lines

FYI, I just made a simple test checking for stolen time instead of offline, and it's in fact called way too oftenly.

That doesn't make sense. Processed_stolen_time should lag at most 1 jiffy behind actual stolen time. So you still need to accumulate at least 10*HZ-1 jiffies of stolen time in one go to end up touching the softlockup watchdog. As far as I can see, anyway. What workload did you run to test using stolen time?

Thanks for pointing it, Keir.

After going back to it, I found it to be a small mistake of mine. I was calling the softlockup watchdog before accounting stolen ticks. Calling it after it does the trick.

I'll resend the patch soon.

Signature · 3 lines
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"