Skip to contentSkip to conversation

Avoid PIT SMP lockups

RFC In review john stultz 8 participants 27 messages Oct 6, 2006 – Oct 20, 2006 virtio core

lore
Share & tools
lore
#0 lore raw

RFC 1/1 Avoid PIT SMP lockups

Hey Andi,
Mind testing this patch on the AMD SMP box you were using earlier w/
acpi=off? I have spent a bit of time trying to hunt down the cause of the reported SMP boxes hanging when they use the PIT for a clocksource, and have not been able to root cause it. Removing the first three PIT io instructions from pit_read() seemed to avoid the issue, but I can't see why.

My current theory is that we're livelocking somehow:

timer_interrupt:

	seq_write_lock_irqsave(xtime_lock)
	spin_lock_irqsave(i8253_lock)
	portio()
	spin_unlock_irqrestore(i8253_lock)
	seq_write_unlock_irqrestore(xtime_lock)

gettime:

	do {
		seq = read_seqbegin(xtime_lock)
		spin_lock_irqsave(i8253_lock)
		portio()
		spin_unlock_irqrestore(i8253_lock)
	} while (read_seqretry(&xtime_lock, seq))

Where maybe one cpu is running gettime, spinning like mad grabbing and releasing the i8253_lock, while another cpu is in the timer_interrupt thread already holding the xtime lock, trying to grab the i8253_lock.

Yea.. its a weak theory (and sysrq-t output doesn't support it)... Don't have a clue otherwise though. Your thoughts?

Anyway, since I can't figure it out, this patch should avoid the issue, by disabling the PIT on SMP boxes (and makes a minor change so we properly fall back to jiffies if the TSC is bad and there's nothing else).

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

thanks
-john

This patch avoids possible PIT livelock issues seen on SMP systems, by not allowing it as a clocksource on SMP boxes.

However, since the PIT may no longer be present, we have to properly handle the cases where SMP systems have TSC skew and fall back from the TSC. Since the PIT isn't there, it would "fall back" to the TSC again. So this changes the jiffies rating to 1, and the TSC-bad rating value to 0.

Thus you will get the following behavior priority on i386 systems:

tsc [if present & stable]
hpet [if present]
cyclone [if present]
acpi_pm [if present]
pit [if UP]
jiffies

Rather then the current more complicated:
tsc [if present & stable]
hpet [if present]
cyclone [if present]
acpi_pm [if present]
pit [if cpus < 4]
tsc [if present & unstable]
jiffies

arch/i386/kernel/i8253.c +1 -1
index 477b24d..9a0060b 100644
@@ -109,7 +109,7 @@ static struct clocksource clocksource_pi
 
 static int __init init_pit_clocksource(void)
 {
-	if (num_possible_cpus() > 4) /* PIT does not scale! */
+	if (num_possible_cpus() > 1) /* PIT does not scale! */
 		return 0;
 
 	clocksource_pit.mult = clocksource_hz2mult(CLOCK_TICK_RATE, 20);
arch/i386/kernel/tsc.c +3 -3
index b8fa0a8..fbc9582 100644
@@ -349,8 +349,8 @@ static int tsc_update_callback(void)
 	int change = 0;
 
 	/* check to see if we should switch to the safe clocksource: */
-	if (clocksource_tsc.rating != 50 && check_tsc_unstable()) {
-		clocksource_tsc.rating = 50;
+	if (clocksource_tsc.rating != 0 && check_tsc_unstable()) {
+		clocksource_tsc.rating = 0;
 		clocksource_reselect();
 		change = 1;
 	}
@@ -461,7 +461,7 @@ static int __init init_tsc_clocksource(v
 							clocksource_tsc.shift);
 		/* lower the rating if we already know its unstable: */
 		if (check_tsc_unstable())
-			clocksource_tsc.rating = 50;
+			clocksource_tsc.rating = 0;
 
 		init_timer(&verify_tsc_freq_timer);
 		verify_tsc_freq_timer.function = verify_tsc_freq;
kernel/time/jiffies.c +1 -1
index 126bb30..a99b2a6 100644
@@ -57,7 +57,7 @@ static cycle_t jiffies_read(void)
 
 struct clocksource clocksource_jiffies = {
 	.name		= "jiffies",
-	.rating		= 0, /* lowest rating*/
+	.rating		= 1, /* lowest valid rating*/
 	.read		= jiffies_read,
 	.mask		= 0xffffffff, /*32bits*/
 	.mult		= NSEC_PER_JIFFY << JIFFIES_SHIFT, /* details above */
#1 lore raw

07 Eki 2006 Cts 00:38 tarihinde, john stultz şunları yazmıştı:

Quoted 2 lines

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

Sorry for late response, ill try this with vmware tonight and will also send backtraces and logs to Andi.

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#2 lore raw
In reply to john stultz

07 Eki 2006 Cts 00:38 tarihinde, john stultz şunları yazmıştı:

Quoted 2 lines

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

Nothing changes inside the vmware, same panics occured as like before :(

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#3 lore raw
Quoted 5 lines · S.Çağlar Onur

07 Eki 2006 Cts 00:38 tarihinde, john stultz şunları yazmıştı:

Quoted 2 lines

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

Nothing changes inside the vmware, same panics occured as like before :(

Hmm.. Did you manage to grab the full log?

thanks
-john

#4 lore raw

10 Eki 2006 Sal 21:27 tarihinde, john stultz şunları yazmıştı:

Quoted 8 lines · S.Çağlar Onur
Quoted 5 lines

07 Eki 2006 Cts 00:38 tarihinde, john stultz şunları yazmıştı:

Quoted 2 lines

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

Nothing changes inside the vmware, same panics occured as like before :(

Hmm.. Did you manage to grab the full log?

Yep, [1] here is whole screen and used config, and as andi suggested i recompiled this kernel [pure vanilla 2.6.18] from scratch.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#5 lore raw
Quoted 6 lines · S.Çağlar Onur

10 Eki 2006 Sal 21:27 tarihinde, john stultz şunları yazmıştı:

Quoted 5 lines · S.Çağlar Onur
Quoted 4 lines

07 Eki 2006 Cts 00:38 tarihinde, john stultz şunları yazmıştı:

Quoted 2 lines

S.Çağlar: Could you give it a whirl to see if it changes your vmware issue?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Nothing changes inside the vmware, same panics occured as like before :(

Hmm.. Did you manage to grab the full log?

Yep, [1] here is whole screen and used config, and as andi suggested i recompiled this kernel [pure vanilla 2.6.18] from scratch.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/

Huh.. that's an odd trace. Looks like the alternative code is involved.

Mind booting w/ "noreplacement" to see if that avoids it?

thanks
-john

#6 lore raw

11 Eki 2006 Çar 20:59 tarihinde, john stultz şunları yazmıştı:

Quoted 8 lines
Quoted 4 lines

Yep, [1] here is whole screen and used config, and as andi suggested i recompiled this kernel [pure vanilla 2.6.18] from scratch.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/

Huh.. that's an odd trace. Looks like the alternative code is involved.

Mind booting w/ "noreplacement" to see if that avoids it?

Booting with "noreplacement" solved panics (i tried booting 10 times with both kernel) for both vanilla one and yours patch included one.

By the way i just realize, panic occurs between

Checking if this processor honours the WP bit even in supervisor mode... Ok.

and

Calibrating delay using timer specific routine.. xxxxx BogoMIPS (lpj=xxxxx)

lines, and system waits there about 5 sec. maybe more (no matter if it panics or continues to boot somehow)

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#7 lore raw
Quoted 4 lines · S.Çağlar Onur

11 Eki 2006 Çar 20:59 tarihinde, john stultz şunları yazmıştı:

Quoted 3 lines
Quoted 3 lines

Yep, [1] here is whole screen and used config, and as andi suggested i recompiled this kernel [pure vanilla 2.6.18] from scratch.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/

Huh.. that's an odd trace. Looks like the alternative code is involved.

Mind booting w/ "noreplacement" to see if that avoids it?

Booting with "noreplacement" solved panics (i tried booting 10 times with both kernel) for both vanilla one and yours patch included one.

Hey Gerd,
Looks like the smp replacements code in 2.6.18 is breaking with vmware.
I'm guessing we're taking an interrupt while apply_replacements is running. Any ideas?

Quoted 10 lines

By the way i just realize, panic occurs between

Checking if this processor honours the WP bit even in supervisor mode... Ok.

and

Calibrating delay using timer specific routine.. xxxxx BogoMIPS (lpj=xxxxx)

lines, and system waits there about 5 sec. maybe more (no matter if it panics or continues to boot somehow)

S.Çağlar: Didn't follow this bit at all. Could you explain a bit more?

thanks
-john

#8 lore raw

11 Eki 2006 Çar 21:43 tarihinde, john stultz şunları yazmıştı:

Quoted 1 line

S.Çağlar: Didn't follow this bit at all. Could you explain a bit more?

Of course, while system boots kernel waits ~5 seconds (maybe more) after printing "Checking if this processor honours the WP bit even in supervisor mode... Ok." line without any visual activity and after that waiting period kernel panics.

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#9 lore raw
Quoted 7 lines · S.Çağlar Onur

11 Eki 2006 Çar 21:43 tarihinde, john stultz şunları yazmıştı:

Quoted 1 line

S.Çağlar: Didn't follow this bit at all. Could you explain a bit more?

Of course, while system boots kernel waits ~5 seconds (maybe more) after printing "Checking if this processor honours the WP bit even in supervisor mode... Ok." line without any visual activity and after that waiting period kernel panics.

And this results in the same panic you linked to earlier?

thanks
-john

#10 lore raw

11 Eki 2006 Çar 22:26 tarihinde, john stultz şunları yazmıştı:

Quoted 10 lines · S.Çağlar Onur
Quoted 7 lines

11 Eki 2006 Çar 21:43 tarihinde, john stultz şunları yazmıştı:

Quoted 1 line

S.Çağlar: Didn't follow this bit at all. Could you explain a bit more?

Of course, while system boots kernel waits ~5 seconds (maybe more) after printing "Checking if this processor honours the WP bit even in supervisor mode... Ok." line without any visual activity and after that waiting period kernel panics.

And this results in the same panic you linked to earlier?

Yes, kernel only panics if it waits there, if somehow can pass here (as i wrote before it can boot normally for every ~1/10 reboot) no panic occurs.

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#11 lore raw
In reply to john stultz
Quoted 4 lines · john stultz

Hey Gerd,
Looks like the smp replacements code in 2.6.18 is breaking with vmware.
I'm guessing we're taking an interrupt while apply_replacements is running. Any ideas?

Try switching the vmware configuration to "other OS". This turns off os-specific binary patching. The alternatives code might have broken assumptions vmware does about the linux kernel code ...

cheers,

Gerd

Signature · 2 lines
Gerd Hoffmann <[email protected]>
http://www.suse.de/~kraxel/julika-dora.jpeg
#12 lore raw

12 Eki 2006 Per 10:28 tarihinde, Gerd Hoffmann şunları yazmıştı:

Quoted 3 lines

Try switching the vmware configuration to "other OS". This turns off os-specific binary patching. The alternatives code might have broken assumptions vmware does about the linux kernel code ...

I did before, i tried these combinations

* Guest Os: Linux, Version: Other Linux 2.6.x kernel
* Guest Os: Linux, Version: Other Linux
* Guest Os: Other, Version: Other

all of them ends up with panic.

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#13 lore raw
Quoted 7 lines · S.Çag(lar Onur

12 Eki 2006 Per 10:28 tarihinde, Gerd Hoffmann s,unlar? yazm?s,t?:

Quoted 4 lines

Try switching the vmware configuration to "other OS". This turns off os-specific binary patching. The alternatives code might have broken assumptions vmware does about the linux kernel code ...

Quoted 8 lines

I did before, i tried these combinations

* Guest Os: Linux, Version: Other Linux 2.6.x kernel
* Guest Os: Linux, Version: Other Linux
* Guest Os: Other, Version: Other

all of them ends up with panic.

We don't do any binary patching. In general, most of the guest OS types turn on various performance hints or activate fairly expensive workarounds to known OS bugs, but they don't make the system less correct by limiting the functionality available to the OS kernel.

It sounds to me like an interrupt is firing in the alternatives code. Perhaps some callee is doing a raw local_irq_enable from somewhere inside here, or nobody took care to disable interrupts in the first place. You need to have interrupts disabled here.

OMG! When did we start calling alternative_instructions from init with _after_ local_irq_enable()? This is totally wrong. Even on native hardware, you can be hit with a timer interrupt in this window where we are patching the kernel. It does not surprise me that you hit this repeatedly. It surprises me that other people don't! The kernel has managed to take enough time to get from the last timer IRQ (likely in lpj calibration, as you noted in the output) to the alternative patching point that 90% of the time, another timer IRQ is pending. This can happen on any machine, not just VMware, and I'm surprised it doesn't happen with other emulators as well.

It might only happen with SMP because the difficulty of getting good enough TSC / timer IRQ synchronization during boot increases exponentially with SMP configurations. And it might pass 10% of the time because you were lucky enough not to fire off another timer interrupt yet.

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Zach

arch/i386/kernel/alternative.c +3 -0
diff -r 2b8ef2e0e25f arch/i386/kernel/alternative.c
@@ -389,6 +389,7 @@ extern struct paravirt_patch *__start_pa
 
 void __init alternative_instructions(void)
 {
+	unsigned long flags;
 	if (no_replacement) {
 		printk(KERN_INFO "(SMP-)alternatives turned off\n");
 		free_init_pages("SMP alternatives",
@@ -396,6 +397,8 @@ void __init alternative_instructions(voi
 				(unsigned long)__smp_alt_end);
 		return;
 	}
+
+	local_irq_save(flags);
 	apply_alternatives(__alt_instructions, __alt_instructions_end);
 
 	/* switch to patch-once-at-boottime-only mode and free the
@@ -433,4 +436,5 @@ void __init alternative_instructions(voi
		alternatives_smp_switch(0);
	}

#endif
-}
+ local_irq_restore(flags);
+}

#14 lore raw

17 Eki 2006 Sal 01:17 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 3 lines

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Ok, I'll test and report back...

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#15 lore raw

17 Eki 2006 Sal 01:21 tarihinde, S.Çağlar Onur şunları yazmıştı:

Quoted 6 lines

17 Eki 2006 Sal 01:17 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 3 lines

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Ok, I'll test and report back...

Both 2.6.18 and 2.6.18.1 boots without any problem (and of course without noreplacement workarund) with that patch.

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#16 lore raw
Quoted 10 lines · S.Çağlar Onur

17 Eki 2006 Sal 01:21 tarihinde, S.Çağlar Onur şunları yazmıştı:

Quoted 6 lines

17 Eki 2006 Sal 01:17 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 3 lines

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Ok, I'll test and report back...

Both 2.6.18 and 2.6.18.1 boots without any problem (and of course without noreplacement workarund) with that patch.

Ok thanks.

I still think we need a solution for the NMIs though. I will think about it.

-Andi

#17 lore raw

PATCH 1/1 Fix potential interrupts during alternative patching [was Re: [RFC] Avoid PIT SMP lockups]

In reply to S.Çağlar Onur
Quoted 8 lines · S.Çağlar Onur

17 Eki 2006 Sal 01:21 tarihinde, S.Çağlar Onur şunları yazmıştı:

Quoted 6 lines

17 Eki 2006 Sal 01:17 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 4 lines

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Quoted 7 lines
Quoted 2 lines

Ok, I'll test and report back...

Both 2.6.18 and 2.6.18.1 boots without any problem (and of course without noreplacement workarund) with that patch.

Cheers

So this patch is an obvious bugfix - please apply, and to stable as well. I'm not sure when this broke, but taking interrupts in the middle of self modifying code is not a pretty sight.

Zach

Interrupts must be disabled during alternative instruction patching. On systems with high timer IRQ rates, or when running in an emulator, timing differences can result in random kernel panics because of running partially patched instructions. This doesn't yet fix NMIs, which requires extricating the patch code from the late bug checking and is logically separate (and also less likely to cause problems).

arch/i386/kernel/alternative.c +5 -1
diff -r 773ac0ebfeb4 arch/i386/kernel/alternative.c
@@ -344,6 +344,7 @@ void alternatives_smp_switch(int smp)
 
 void __init alternative_instructions(void)
 {
+	unsigned long flags;
 	if (no_replacement) {
 		printk(KERN_INFO "(SMP-)alternatives turned off\n");
 		free_init_pages("SMP alternatives",
@@ -351,6 +352,8 @@ void __init alternative_instructions(voi
 				(unsigned long)__smp_alt_end);
 		return;
 	}
+
+	local_irq_save(flags);
 	apply_alternatives(__alt_instructions, __alt_instructions_end);
 
 	/* switch to patch-once-at-boottime-only mode and free the
@@ -386,4 +389,5 @@ void __init alternative_instructions(voi
 		alternatives_smp_switch(0);
 	}
 #endif
-}
+	local_irq_restore(flags);
+}
#18 lore raw

Re: [PATCH] Fix potential interrupts during alternative patching [was Re: [RFC] Avoid PIT SMP lockups]

Quoted 1 line · Zachary Amsden

So this patch is an obvious bugfix - please apply, and to stable as well. I'm not sure when this broke, but taking interrupts in the middle of self modifying code is not a pretty sight.

I had actually seen this when I built the Xen paravirt kernel with SMP on, but I assumed it was something in the pv_ops tree rather than mainline...

J

#19 lore raw

Re: [PATCH] Fix potential interrupts during alternative patching [was Re: [RFC] Avoid PIT SMP lockups]

Quoted 4 lines · Jeremy Fitzhardinge
Quoted 1 line · Zachary Amsden

So this patch is an obvious bugfix - please apply, and to stable as well. I'm not sure when this broke, but taking interrupts in the middle of self modifying code is not a pretty sight.

I had actually seen this when I built the Xen paravirt kernel with SMP on, but I assumed it was something in the pv_ops tree rather than mainline...

Very likely to show up in qemu as well, if you use that.

#20 lore raw

Re: [PATCH] Fix potential interrupts during alternative patching [was Re: [RFC] Avoid PIT SMP lockups]

19 Eki 2006 Per 12:00 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 11 lines
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 3 lines · Zachary Amsden

So this patch is an obvious bugfix - please apply, and to stable as well. I'm not sure when this broke, but taking interrupts in the middle of self modifying code is not a pretty sight.

I had actually seen this when I built the Xen paravirt kernel with SMP on, but I assumed it was something in the pv_ops tree rather than mainline...

Very likely to show up in qemu as well, if you use that.

I can confirm qemu and virtual pc 2004 gaves same exception without that patch.

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#21 lore raw

Re: [PATCH] Fix potential interrupts during alternative patching [was Re: [RFC] Avoid PIT SMP lockups]

In reply to Zachary Amsden
Quoted 15 lines · Zachary Amsden
Quoted 14 lines · S.??a??lar Onur

17 Eki 2006 Sal 01:21 tarihinde, S.??a??lar Onur ??unlar?? yazm????t??:

Quoted 8 lines

17 Eki 2006 Sal 01:17 tarihinde, Zachary Amsden ??unlar?? yazm????t??:

Quoted 4 lines

My nasty quick patch might not apply - the only tree I've got is a very hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious enough.

Ok, I'll test and report back...

Both 2.6.18 and 2.6.18.1 boots without any problem (and of course without noreplacement workarund) with that patch.

Quoted 6 lines
Quoted 2 lines

Cheers

So this patch is an obvious bugfix - please apply, and to stable as well. I'm not sure when this broke, but taking interrupts in the middle of self modifying code is not a pretty sight.

Please send -stable patches to [email protected], not to me directly (we are a team and hand off ownership to each other, by sending it to the alias it makes sure that nothing gets lots in our individual mail boxes.)

Also, please let stable know when this is upstream, we don't want to apply it before then.

thanks,

greg k-h

#22 lore raw
In reply to Zachary Amsden
Quoted 5 lines

It might only happen with SMP because the difficulty of getting good enough TSC / timer IRQ synchronization during boot increases exponentially with SMP configurations. And it might pass 10% of the time because you were lucky enough not to fire off another timer interrupt yet.

We have the same problem with NMI watchdog events unfortunately. Need to call something in the nmi watchdog code to make sure it is not renewed and then reenabled.
Or maybe it's better to figure out a way that yields atomic patches.

I think the best way is to make sure all alternative() patches are always done before the code can be ever executed - this means doing it very early for the main kernel. The only exception would be the LOCK prefix patching, which should be atomic.

iirc there was some more patching except lock prefixes going on for SMP<->UP transisitions, but last time I checked they didn't look particularly useful and could be probably eliminated.

-Andi

#23 lore raw
Quoted 11 lines · Andi Kleen
Quoted 2 lines

It might only happen with SMP because the difficulty of getting good enough TSC / timer IRQ synchronization during boot increases exponentially with SMP configurations. And it might pass 10% of the time because you were lucky enough not to fire off another timer interrupt yet.

We have the same problem with NMI watchdog events unfortunately. Need to call something in the nmi watchdog code to make sure it is not renewed and then reenabled. Or maybe it's better to figure out a way that yields atomic patches.

I think the best way is to make sure all alternative() patches are always done before the code can be ever executed - this means doing it very early for the main kernel. The only exception would be the LOCK prefix patching, which should be atomic

Yes, this solves the problem in most cases. Lock patching is fine no matter when you do it. I think the problem with alternative patching in check_bugs() is that it happens way too late; the patching really has nothing at all to do with check_bugs(), and should be a separate step, probably part of setup_arch.

The paravirt-ops stuff also has some patching code. Fortunately, there, we can probably skirt the NMI issue by simply disallowing NMIs, but the issue pops up again in stop_machine_run - what happens if you take NMIs during stop_machine_run? Debug traps? Module unload is fine, but code patching done using stop_machine_run is not safe.

Zach

#24 lore raw
In reply to john stultz
Quoted 4 lines · john stultz

Hey Gerd,
Looks like the smp replacements code in 2.6.18 is breaking with vmware.
I'm guessing we're taking an interrupt while apply_replacements is running. Any ideas?

It's not the smp alternatives code, its the one for processor-specific instructions. The eip offset for alternative_instructions() in the trace suggests it is the first call to apply_replacements. The second one is the one for the smp alternatives (which doesn't do anything btw as we patch away the lock prefixes only).

cheers,

Gerd

Signature · 2 lines
Gerd Hoffmann <[email protected]>
http://www.suse.de/~kraxel/julika-dora.jpeg
#25 lore raw

Vmware problems was Re: [RFC] Avoid PIT SMP lockups

Quoted 11 lines · Gerd Hoffmann
Quoted 4 lines · john stultz

Hey Gerd,
Looks like the smp replacements code in 2.6.18 is breaking with vmware.
I'm guessing we're taking an interrupt while apply_replacements is running. Any ideas?

It's not the smp alternatives code, its the one for processor-specific instructions. The eip offset for alternative_instructions() in the trace suggests it is the first call to apply_replacements. The second one is the one for the smp alternatives (which doesn't do anything btw as we patch away the lock prefixes only).

I would have expected that they trap those writes and invalidate the cache. Even qemu and valgrind do that fine.

Perhaps Zach has some clue or can refer to someone who has.

-Andi

#26 lore raw

Re: Vmware problems was Re: [RFC] Avoid PIT SMP lockups

Quoted 16 lines · Andi Kleen
Quoted 7 lines · Gerd Hoffmann
Quoted 4 lines · john stultz

Hey Gerd,
Looks like the smp replacements code in 2.6.18 is breaking with vmware.
I'm guessing we're taking an interrupt while apply_replacements is running. Any ideas?

Quoted 5 lines

It's not the smp alternatives code, its the one for processor-specific instructions. The eip offset for alternative_instructions() in the trace suggests it is the first call to apply_replacements. The second one is the one for the smp alternatives (which doesn't do anything btw as we patch away the lock prefixes only).

Quoted 4 lines

I would have expected that they trap those writes and invalidate the cache. Even qemu and valgrind do that fine.

Perhaps Zach has some clue or can refer to someone who has.

Why do you think it has something to do with VMware's emulation - except timing? From what I see, there were bytes

0xF0 0x83 0x44 0x24 0x00 0x00

before apply_alternatives() was entered (binutils 2.17 would generate one byte shorter code, 0xF0 0x83 0x04 0x24 0x00, but that's another story - 2.16 and older treat 0(%esp) differently from (%esp)) . Now update alternatives comes in and starts overwritting alternative - note that it does that in two steps - first it memcpy()-ies alternative, then it memcpy()-ies nop padding. So after first memcpy() code looks like

0x0F 0xAE 0xE8 0x24 0x00 0x00

Now timer interrupt arrives, and these data are interpreted as

lfence; andb $0,%al; add %cl,0x465B9415(%ebx)

If it would not crash, once it would return 0x24 0x00 0x00 will get overwritten with 3 bytes NOP sequence and everybody will be happy.

AFAIT you do not see this because you have to use old binutils to repro this - I'm unable to reproduce this on Debian box with binutils 2.17, as then byte sequence is valid instruction even if partially overwritten - it just clears %al to zero, but nobody notices that...

So as far as I can tell, interrupts (and NMIs?) should be disabled when apply_alternatives() run if interrupt handlers are using alternatives - and as it was just proven, they do.

Reason you see this happening in VM often is that this first alternatives run invalidates lot of internal state, and it takes so long that next timer interrupt is for sure pending as soon as first "rep movsb" in alternatives finishes.

Petr