Skip to contentSkip to conversation

Invalid opcode

Opps S.Çağlar Onur 4 participants 16 messages Nov 5, 2006 – Nov 30, 2006

lore
Share & tools
lore
#0 lore raw

Hi;

2.6.18, 2.6.18.1 and 2.6.18.2 still panics randomly (it seems this is not related to smpreplacament bug solved in .2) in VmWare and Microsoft Virtual PC and in order to confirm this bug is not our distro specific i downloaded and tried latest OpenSuse also [1] and [2] are screens captured by vmware but exact same panic occurs in Virtual PC as reported to us in [3]. I CC'ed previous threads receivers also.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
[3] http://bugs.pardus.org.tr/show_bug.cgi?id=3804

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#1 lore raw
Quoted 4 lines · S.Çağlar Onur

Hi;

2.6.18, 2.6.18.1 and 2.6.18.2 still panics randomly (it seems this is not related to smpreplacament bug solved in .2)

How do you know this?

And does it still happen in 2.6.19-rc4?

Quoted 4 lines

in VmWare and Microsoft Virtual
PC and in order to confirm this bug is not our distro specific i downloaded and tried latest OpenSuse also [1] and [2] are screens captured by vmware but exact same panic occurs in Virtual PC as reported to us in [3].

Always the same BUG()?

Quoted 5 lines

I CC'ed
previous threads receivers also.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png

There is just some rolling Turkish text there.

-Andi

#2 lore raw

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching code done its job.

Quoted 1 line

And does it still happen in 2.6.19-rc4?

Will try

Quoted 7 lines
Quoted 5 lines

in VmWare and Microsoft Virtual
PC and in order to confirm this bug is not our distro specific i downloaded and tried latest OpenSuse also [1] and [2] are screens captured by vmware but exact same panic occurs in Virtual PC as reported to us in [3].

Always the same BUG()?

Yes, same bug

Quoted 1 line

There is just some rolling Turkish text there.

Ah im sorry here is the correct links :(

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_opensuse.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_pardus.png

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#3 lore raw
Quoted 5 lines

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching code done its job.

Possibly compiled a kernel with instructions your processor does not support? Come to think of cmov...

	-`J'
#4 lore raw

05 Kas 2006 Paz 19:25 tarihinde, Jan Engelhardt şunları yazmıştı:

Quoted 8 lines
Quoted 5 lines

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching code done its job.

Possibly compiled a kernel with instructions your processor does not support? Come to think of cmov...

That machine is a Intel(R) Pentium(R) 4 CPU 3.00GHz so if im not wrong cmov is supported on that processor

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#5 lore raw
In reply to S.Çağlar Onur
Quoted 5 lines · S.Çağlar Onur

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching code done its job.

Can you test with "noreplacement" to make sure?

Anyways I suspect we're just getting back some variant of the old CPU setup race.

Normally CPU booting in Linux follows a special "cpu hotplug" state machine, but for historical reasons i386 only implements one state of this. At one point we had a similar bug (but not in the callback on CPU #0, but in the timer on newly booted CPU). I don't see currently how it can happen (but i haven't thought very deeply about it yet)

Probably your timing is just unlucky on those simulators.

Previously we avoided converting i386 cpu bootup fully to the new state machine because it is very fragile, but it's possible that there is no other choice than to do it properly. Or maybe another kludge is possible.

-Andi

#6 lore raw

Hi;

05 Kas 2006 Paz 20:57 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

Can you test with "noreplacement" to make sure?

I sorry for not to mention that, i tried noreplacement before reporting which is also ends up with same panic.

Quoted 10 lines

Anyways I suspect we're just getting back some variant of the old CPU setup race.

Normally CPU booting in Linux follows a special "cpu hotplug" state machine, but for historical reasons i386 only implements one state of this. At one point we had a similar bug (but not in the callback on CPU #0, but in the timer on newly booted CPU). I don't see currently how it can happen (but i haven't thought very deeply about it yet)

Probably your timing is just unlucky on those simulators.

Hmm, Novell bugzilla seems has similiar issues,
https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones gaves same or similiar panic outputs.

Quoted 4 lines

Previously we avoided converting i386 cpu bootup fully to the new state machine because it is very fragile, but it's possible that there is no other choice than to do it properly. Or maybe another kludge is possible.

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#7 lore raw
Quoted 8 lines · S.Çağlar Onur

Hmm, Novell bugzilla seems has similiar issues, https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones gaves same or similiar panic outputs.

Quoted 5 lines

Previously we avoided converting i386 cpu bootup fully to the new state machine because it is very fragile, but it's possible that there is no other choice than to do it properly. Or maybe another kludge is possible.

Yes, this is some kind of softirq race during init.

#8 lore raw
Quoted 6 lines · Zachary Amsden
Quoted 5 lines · S.Çağlar Onur

Hmm, Novell bugzilla seems has similiar issues,
https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones gaves same or similiar panic outputs.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Previously we avoided converting i386 cpu bootup fully to the new state machine because it is very fragile, but it's possible that there is no other choice than to do it properly. Or maybe another kludge is possible.

Yes, this is some kind of softirq race during init.

Yes, the callbacks run at the wrong time. Unlike modern architectures i386 doesn't do callback cpu boot callback repeat, but boot all cpus then callback.

But the strange thing is that the BP hits it. Normally the new CPU hit it because it tried to run a timer interrupt before the callback ran and initialized all the per CPU state (this happened often when dual core CPUs were first introduced for some reason)

In this case it looks like the AP managed to queue a tasklet before the cpu up callback runs.

I suppose we'll either need to convert i386 really over to standard cpu_up() or add some additional spinlocks to stop the APs with interrupts off before the callbacks start to run.

-Andi

#9 lore raw

Fix for OpenSUSE kernel bug (was Re: [Opps] Invalid opcode)

In reply to S.Çağlar Onur
Quoted 29 lines · S.Çağlar Onur

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 2 lines

How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching code done its job.

Quoted 2 lines

And does it still happen in 2.6.19-rc4?

Will try

Quoted 8 lines
Quoted 6 lines

in VmWare and Microsoft Virtual
PC and in order to confirm this bug is not our distro specific i downloaded and tried latest OpenSuse also [1] and [2] are screens captured by vmware but exact same panic occurs in Virtual PC as reported to us in [3].

Always the same BUG()?

Yes, same bug

Quoted 2 lines

There is just some rolling Turkish text there.

Quoted 7 lines

Ah im sorry here is the correct links :(

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_opensuse.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_pardus.png

Cheers

I'm proposing this as a fix for your bug. Having tasklets scheduled before softirqd gets to run might be somewhat backwards, but there is nothing I can find wrong about it from a correctness point of view. Better to boot the kernel even when compiled with bug checking on, I think.

This bug started becoming apparent in 2.6.18 because of some rework with the CPU hotplug code, but in theory, it exists at least all the way back to 2.6.10, which is as far as I looked backwards in time.

Zach

It is possible to have tasklets get scheduled before softirqd has had a chance to spawn on all CPUs. This is totally harmless; after success during action CPU_UP_PREPARE, action CPU_ONLINE will be called, which immediately wakes softirqd on the appropriate CPU to process the already pending tasklets. So there is no danger of having a missed wakeup for any tasklets that were already pending.

In particular, i386 is affected by this during startup, and is visible when using a very large initrd; during the time it takes for the initrd to be decompressed, a timer IRQ can come in and schedule RCU callbacks. It is also possible that resending of a hardware IRQ via a softirq triggers the same bug.

Because of different timing conditions, this shows up in all emulators and virtual machines tested, including Xen, VMware, Virtual PC, and Qemu. It is also possible to trigger on native hardware with a large enough initrd, although I don't have a reliable case demonstrating that.

kernel/softirq.c +0 -2
Index: linux-2.6.18/kernel/softirq.c
===================================================================
@@ -574,8 +574,6 @@ static int __cpuinit cpu_callback(struct
 
 	switch (action) {
 	case CPU_UP_PREPARE:
-		BUG_ON(per_cpu(tasklet_vec, hotcpu).list);
-		BUG_ON(per_cpu(tasklet_hi_vec, hotcpu).list);
 		p = kthread_create(ksoftirqd, hcpu, "ksoftirqd/%d", hotcpu);
 		if (IS_ERR(p)) {
 			printk("ksoftirqd for %i failed\n", hotcpu);
#10 lore raw

Re: Fix for OpenSUSE kernel bug (was Re: [Opps] Invalid opcode)

Hi;

30 Kas 2006 Per 08:44 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 8 lines

I'm proposing this as a fix for your bug. Having tasklets scheduled before softirqd gets to run might be somewhat backwards, but there is nothing I can find wrong about it from a correctness point of view. Better to boot the kernel even when compiled with bug checking on, I think.

This bug started becoming apparent in 2.6.18 because of some rework with the CPU hotplug code, but in theory, it exists at least all the way back to 2.6.10, which is as far as I looked backwards in time.

I cannot reproduce that opps with 2.6.18.4 + your patch any longer, so at least works for me :), thanks

Cheers

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#11 lore raw
In reply to Andi Kleen

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

And does it still happen in 2.6.19-rc4?

Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!
#12 lore raw
Quoted 4 lines · S.Çağlar Onur

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 1 line

And does it still happen in 2.6.19-rc4?

Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

It's probably still there, just hopefully it won't be release critical for .19 then.

At some point it has to be fixed properly by converting i386 to the new hotplug architecture i suppose.

-Andi

#13 lore raw
In reply to S.Çağlar Onur
Quoted 7 lines · S.Çağlar Onur

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 2 lines

And does it still happen in 2.6.19-rc4?

Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

I would like to find the exact cause of the problem; I suspect, as does Andi, that it could just be dormant. You had problems still with 2.6.18.latest, correct? If I can find the cause, I would like to get a fix into 2.6.18-stable if possible. I think you already sent me the reproducing kernel config, but I seem to have misplaced it. Could you resend? I should have some time to look at this early this week.

Thanks,

Zach

#14 lore raw
Quoted 10 lines · Zachary Amsden
Quoted 7 lines · S.Çağlar Onur

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:

Quoted 2 lines

And does it still happen in 2.6.19-rc4?

Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

I would like to find the exact cause of the problem;

It's all related to i386's abuse of the cpu hotplug state machine. Eventually that needs to be fixed properly like it was on x86-64 (I didn't dare touch i386 back then because this code is so fragile on old hardware)

-Andi

#15 lore raw
In reply to Zachary Amsden

13 Kas 2006 Pts 06:20 tarihinde, Zachary Amsden şunları yazmıştı:

Quoted 6 lines

I would like to find the exact cause of the problem; I suspect, as does Andi, that it could just be dormant. You had problems still with 2.6.18.latest, correct? If I can find the cause, I would like to get a fix into 2.6.18-stable if possible. I think you already sent me the reproducing kernel config, but I seem to have misplaced it. Could you resend? I should have some time to look at this early this week.

Sorry for late reply, [1] is the kernel config i used and if you want i can provide ~5mb iso contains that kernel and its 26.19-rc5 version to test?

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/config.2.6.18

Signature · 4 lines
S.Çağlar Onur <[email protected]>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!