2.6.{26.2,27-rc} oops on virtualbox
Hi there,
Users of different Linux distros (includes Ubuntu, Mandriva, ArchLinux and
possibly Fedora) are reporting that kernel 2.6.26.2 is OOPSing in the Virtual Box emulator[1].
It is not clear if this is a kernel or Virtual Box bug, but as the kernel
is also OOPsing in QEMU (although with different behaivor) I have decided to post my debug results here in case someone is interested in debugging the kernel part further.
I have done a bisection by hand among kernel versions and found that
the commit which triggers the oops in _Virtual Box_ was introduced in 2.6.26-rc1 and the problem also happens with latest Linus tree.
By using git bisect I found that the commit is this:
"""
commit e587cadd8f47e202a30712e2906a65a0606d5865
Author: Mathieu Desnoyers <[email protected]> Date: Thu Mar 6 08:48:49 2008 -0500
x86: enhance DEBUG_RODATA support - alternatives
[...]
"""
By reverting this commit I don't get the OOPS anymore. I have
tested with 2.6.26-rc1 and latest Linus tree (2.6.27-rc3).
What puzzles me though is that a similar problem happens with
QEMU, but it also OOPSes with kernels before 2.6.26-rc1, reverting the patch above makes no difference and it works with current Linus tree.
Does this look like a kernel bug?
All my tests have been done with vanilla kernels, but I have
built .iso installation images with them and I'm not sure of what the build script does.
[1] http://en.wikipedia.org/wiki/Virtual_box
Thanks for reading this.
Signature · 1 line
Luiz Fernando N. Capitulino
H. Peter Anvin
reviewerQuoted 3 lines
Does this look like a kernel bug?
No, it looks like a very common virtualizer bug. Does the attached patch work for you?
-hpa
arch/x86/kernel/alternative.c +0 -2
@@ -157,8 +157,6 @@ static const struct nop { } noptypes[] = { { X86_FEATURE_K8, k8_nops }, { X86_FEATURE_K7, k7_nops }, - { X86_FEATURE_P4, p6_nops }, - { X86_FEATURE_P3, p6_nops }, { -1, NULL } };
H. Peter Anvin
reviewerQuoted 6 lines · H. Peter Anvin
Quoted 3 lines
Does this look like a kernel bug?
No, it looks like a very common virtualizer bug. Does the attached patch work for you?
Also, in addition to this, please try tip:master. There is a patch in tip:master which I hope should fix this problem, but the details are important.
-hpa
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines
Does this look like a kernel bug?
Quoted 7 lines
Quoted 3 lines
No, it looks like a very common virtualizer bug. Does the attached patch work for you?
Also, in addition to this, please try tip:master. There is a patch in tip:master which I hope should fix this problem, but the details are important.
access coordinates would be at:
http://people.redhat.com/mingo/tip.git/README
Ingo
Em Fri, 22 Aug 2008 08:50:12 +0200
Ingo Molnar <[email protected]> escreveu:
| | * H. Peter Anvin <[email protected]> wrote: | | > H. Peter Anvin wrote: | >>> | >>> Does this look like a kernel bug? | >>> | >> | >> No, it looks like a very common virtualizer bug. Does the attached | >> patch work for you? | >> | > | > Also, in addition to this, please try tip:master. There is a patch in | > tip:master which I hope should fix this problem, but the details are | > important. | | access coordinates would be at: | | http://people.redhat.com/mingo/tip.git/README
As I already have Linus tree downloaded I have cloned it in
the usual way.
Got the same results: OOPS in virtualbox but it works on QEMU.
The OOPS's output follows and I have attached the .config I'm using
to reproduce the problem.
"""
BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c01310f1>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in:
Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test24-tip #3)
EIP: 0060:[<c01310f1>] EFLAGS: 00010246 CPU: 0
EIP is at vprintk+0x181/0x440
EAX: 00000246 EBX: 00000000 ECX: c0130ca9 EDX: 0000dedd
ESI: c0474ae3 EDI: c04cf6bc EBP: c7435f24 ESP: c7435eb0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000)
Stack: 0000dedd c0130ca9 c7435f40 00000000 a026104f a026106c c7434000 c7435ee6
00000006 00000246 00000000 a0260cf3 0000001c c7434000 00000282 00000046
c11a85a0 c7435efc c0135c6f c7435f14 c0115fcb a0296e91 c0104c2c 00000000
Call Trace:
[<c0130ca9>] ? release_console_sem+0x199/0x1e0
[<c0135c6f>] ? irq_exit+0x3f/0x90
[<c0115fcb>] ? smp_apic_timer_interrupt+0x5b/0x90
[<c0104c2c>] ? apic_timer_interrupt+0x28/0x30
[<c0474ae3>] ? net_ns_init+0x0/0x1ad
[<c0474ae3>] ? net_ns_init+0x0/0x1ad
[<c0346ed9>] ? printk+0x18/0x1f
[<c0474b00>] ? net_ns_init+0x1d/0x1ad
[<c0474ae3>] ? net_ns_init+0x0/0x1ad
[<c0101116>] ? do_one_initcall+0x26/0x170
[<c0128f66>] ? try_to_wake_up+0xc6/0x240
[<c012910f>] ? wake_up_process+0xf/0x20
[<c014192d>] ? start_workqueue_thread+0x1d/0x20
[<c0141d4b>] ? __create_workqueue_key+0x1eb/0x240
[<c0141820>] ? worker_thread+0x0/0xf0
[<c044b387>] ? kernel_init+0x141/0x214
[<c044b246>] ? kernel_init+0x0/0x214
[<c0104dc7>] ? kernel_thread_helper+0x7/0x10
=======================
Code: c0 0f 84 0b 01 00 00 b8 50 f1 41 c0 c7 05 ec f1 41 c0 ff ff ff ff e8 cf 8b 21 00 e8 ea 04 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 ec f1 41 c0 e8
EIP: [<c01310f1>] vprintk+0x181/0x440 SS:ESP 0069:c7435eb0
---[ end trace 4eaa2a86a8e2da22 ]---"""
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 23 lines · * Luiz Fernando N. Capitulino ([email protected]) wrote:
Em Fri, 22 Aug 2008 08:50:12 +0200
Ingo Molnar <[email protected]> escreveu:| | * H. Peter Anvin <[email protected]> wrote: | | > H. Peter Anvin wrote: | >>> | >>> Does this look like a kernel bug? | >>> | >> | >> No, it looks like a very common virtualizer bug. Does the attached | >> patch work for you? | >> | > | > Also, in addition to this, please try tip:master. There is a patch in | > tip:master which I hope should fix this problem, but the details are | > important. | | access coordinates would be at: | | http://people.redhat.com/mingo/tip.git/README
Quoted 8 lines
As I already have Linus tree downloaded I have cloned it in
the usual way.Got the same results: OOPS in virtualbox but it works on QEMU.
The OOPS's output follows and I have attached the .config I'm using
to reproduce the problem.
Can you try booting with the kernel argument :
debug_alternative
The dmesg of the kernel bootup up to the oops would be helpful.
My guess is that there may be something wrong with irq disabling which protects text_poke_early in apply_alternatives().
Mathieu
Quoted 44 lines
"""
BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c01310f1>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in:Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test24-tip #3)
EIP: 0060:[<c01310f1>] EFLAGS: 00010246 CPU: 0
EIP is atvprintk+0x181/0x440
EAX: 00000246 EBX: 00000000 ECX:c0130ca9EDX:0000dedd
ESI:c0474ae3EDI:c04cf6bcEBP:c7435f24ESP:c7435eb0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000)Stack: 0000dedd c0130ca9 c7435f40 00000000 a026104f a026106c c7434000 c7435ee6 00000006 00000246 00000000 a0260cf3 0000001c c7434000 00000282 00000046 c11a85a0 c7435efc c0135c6f c7435f14 c0115fcb a0296e91 c0104c2c 00000000 Call Trace: [<c0130ca9>] ? release_console_sem+0x199/0x1e0 [<c0135c6f>] ? irq_exit+0x3f/0x90 [<c0115fcb>] ? smp_apic_timer_interrupt+0x5b/0x90 [<c0104c2c>] ? apic_timer_interrupt+0x28/0x30 [<c0474ae3>] ? net_ns_init+0x0/0x1ad [<c0474ae3>] ? net_ns_init+0x0/0x1ad [<c0346ed9>] ? printk+0x18/0x1f [<c0474b00>] ? net_ns_init+0x1d/0x1ad [<c0474ae3>] ? net_ns_init+0x0/0x1ad [<c0101116>] ? do_one_initcall+0x26/0x170 [<c0128f66>] ? try_to_wake_up+0xc6/0x240 [<c012910f>] ? wake_up_process+0xf/0x20 [<c014192d>] ? start_workqueue_thread+0x1d/0x20 [<c0141d4b>] ? __create_workqueue_key+0x1eb/0x240 [<c0141820>] ? worker_thread+0x0/0xf0 [<c044b387>] ? kernel_init+0x141/0x214 [<c044b246>] ? kernel_init+0x0/0x214 [<c0104dc7>] ? kernel_thread_helper+0x7/0x10 ======================= Code: c0 0f 84 0b 01 00 00 b8 50 f1 41 c0 c7 05 ec f1 41 c0 ff ff ff ff e8 cf 8b 21 00 e8 ea 04 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 ec f1 41 c0 e8 EIP: [<c01310f1>] vprintk+0x181/0x440 SS:ESP 0069:c7435eb0 ---[ end trace 4eaa2a86a8e2da22 ]---"""
Signature · 1 line
Luiz Fernando N. Capitulino
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Em Fri, 22 Aug 2008 11:34:52 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Fri, 22 Aug 2008 08:50:12 +0200 | > Ingo Molnar <[email protected]> escreveu: | > | > | | > | * H. Peter Anvin <[email protected]> wrote: | > | | > | > H. Peter Anvin wrote: | > | >>> | > | >>> Does this look like a kernel bug? | > | >>> | > | >> | > | >> No, it looks like a very common virtualizer bug. Does the attached | > | >> patch work for you? | > | >> | > | > | > | > Also, in addition to this, please try tip:master. There is a patch in | > | > tip:master which I hope should fix this problem, but the details are | > | > important. | > | | > | access coordinates would be at: | > | | > | http://people.redhat.com/mingo/tip.git/README | > | > As I already have Linus tree downloaded I have cloned it in | > the usual way. | > | > Got the same results: OOPS in virtualbox but it works on QEMU. | > | > The OOPS's output follows and I have attached the .config I'm using | > to reproduce the problem. | > | | Can you try booting with the kernel argument : | debug_alternative | | The dmesg of the kernel bootup up to the oops would be helpful. | | My guess is that there may be something wrong with irq disabling which | protects text_poke_early in apply_alternatives().
I have attached two files:
- normal.txt: normal boot with no debug options
- debug-alternative.txt ignore_loglevel and debug-alternative boot options
I had to pass ignore_loglevel otherwise it wouldn't print
anything.
Signature · 1 line
Luiz Fernando N. Capitulino
Linux version 2.6.27-rc4-test25 ([email protected]) (gcc version 4.3.1 20080626 (prerelease) (GCC) ) #89 SMP Fri Aug 22 12:47:34 BRT 2008 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable)
BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)last_pfn = 0x7ff0 max_arch_pfn = 0x100000
RAMDISK: 07b9b000 - 07fbf89d
DMI 2.5 present.
ACPI: RSDP 000E0000, 0024 (r2 VBOX )
ACPI: XSDT 07FF0030, 002C (r1 VBOX VBOXXSDT 1 ASL 61)
ACPI: FACP 07FF0060, 00F4 (r4 VBOX VBOXFACP 1 ASL 61)
ACPI: DSDT 07FF01A0, 1064 (r1 VBOX VBOXBIOS 2 INTL 20080213)
ACPI: FACS 07FF0160, 0040
0MB HIGHMEM available.
127MB LOWMEM available.
mapped low ram: 0 - 07ff0000
low ram: 00000000 - 07ff0000
bootmap 00002000 - 00003000
(9 early reservations) ==> bootmem [0000000000 - 0007ff0000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000]
#2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000]
#3 [0000100000 - 0000814b10] TEXT DATA BSS ==> [0000100000 - 0000814b10]
#4 [0007b9b000 - 0007fbf89d] RAMDISK ==> [0007b9b000 - 0007fbf89d]
#5 [0000815000 - 0000819000] INIT_PG_TABLE ==> [0000815000 - 0000819000]
#6 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000]
#7 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000]
#8 [0000002000 - 0000003000] BOOTMAP ==> [0000002000 - 0000003000]
Zone PFN ranges:
DMA 0x00000000 -> 0x00001000
Normal 0x00001000 -> 0x00007ff0
HighMem 0x00007ff0 -> 0x00007ff0
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000000 -> 0x0000009f
0: 0x00000100 -> 0x00007ff0ACPI: PM-Timer IO Port: 0x4008
SMP: Allowing 1 CPUs, 0 hotplug CPUs
Found and enabled local APIC!
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 10000000 (gap: 8000000:f7fc0000)
PERCPU: Allocating 40224 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32239
Kernel command line: initrd=alt0/all.rdz vga=788 splash=silent BOOT_IMAGE=alt0/vmlinuz vga=0 console=ttyS0,9600 console=tty0 Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 512 (order: 9, 2048 bytes)
TSC calibrated against PM_TIMER
Detected 2410.453 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 48
... MAX_LOCKDEP_KEYS: 8191
... CLASSHASH_SIZE: 4096
... MAX_LOCKDEP_ENTRIES: 8192 ... MAX_LOCKDEP_CHAINS: 16384 ... CHAINHASH_SIZE: 8192
memory used by lock dependency info: 2335 kB
per task-struct memory footprint: 1152 bytes
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 117088k/131008k available (2340k kernel code, 13364k reserved, 1027k data, 308k init, 0k highmem) virtual kernel memory layout:
fixmap : 0xffe18000 - 0xfffff000 (1948 kB)
pkmap : 0xff800000 - 0xffc00000 (4096 kB)
vmalloc : 0xc8800000 - 0xff7fe000 ( 879 MB)
lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB)
.init : 0xc0451000 - 0xc049e000 ( 308 kB)
.data : 0xc03493b8 - 0xc044a040 (1027 kB)
.text : 0xc0100000 - 0xc03493b8 (2340 kB)Checking if this processor honours the WP bit even in supervisor mode...Ok.
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 4820.90 BogoMIPS (lpj=2410453) Security Framework initialized
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20080609
ACPI: setting ELCR to 0200 (from 0c00)
weird, boot CPU (#0) not listedby the BIOS.
SMP motherboard not detected.
SMP disabled
Brought up 1 CPUs
Total of 1 processors activated (4820.90 BogoMIPS).
khelper used greatest stack depth: 7108 bytes leftnet_namespace: 384 bytes
Booting paravirtualized kernel on bare hardware
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfadb0, last bus=0
PCI: Using configuration type 1 for base access
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 9 *10 11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c012fc71>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in:
Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test25 #89)
EIP: 0060:[<c012fc71>] EFLAGS: 00010246 CPU: 0
EIP is at vprintk+0x181/0x440
EAX: 00000246 EBX: 00000000 ECX: c012f8a9 EDX: 00003b3a
ESI: 00000000 EDI: c04d76c1 EBP: c7435f20 ESP: c7435eac
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000)
Stack: 00003b3a c012f8a9 c7435f3c c02948b1 c7435f18 c02957f6 00000074 c7435ee2
00000006 00000246 00000000 00000000 00000021 00000000 00000001 00000000
a027c4ab a027c4c8 00000001 00000297 00000246 00000000 00000001 00000000
Call Trace:
[<c012f8a9>] ? release_console_sem+0x1c9/0x1e0
[<c02948b1>] ? put_device+0x11/0x20
[<c02957f6>] ? device_add+0x26/0x610
[<c0471c5c>] ? pnpacpi_init+0x0/0x89
[<c03450f4>] ? printk+0x18/0x1c
[<c0266f87>] ? register_acpi_bus_type+0x58/0x69
[<c0471ca5>] ? pnpacpi_init+0x49/0x89
[<c0101116>] ? do_one_initcall+0x26/0x170
[<c01e1d14>] ? create_proc_entry+0x54/0xa0
[<c016ef86>] ? register_irq_proc+0xb6/0xd0
[<c016efea>] ? init_irq_proc+0x4a/0x60
[<c045132d>] ? kernel_init+0x10f/0x166
[<c045121e>] ? kernel_init+0x0/0x166
[<c0104b67>] ? kernel_thread_helper+0x7/0x10
=======================
Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8
EIP: [<c012fc71>] vprintk+0x181/0x440 SS:ESP 0069:c7435eac
---[ end trace 4eaa2a86a8e2da22 ]---
Kernel panic - not syncing: Attempted to kill init!Linux version 2.6.27-rc4-test25 ([email protected]) (gcc version 4.3.1 20080626 (prerelease) (GCC) ) #89 SMP Fri Aug 22 12:47:34 BRT 2008 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable)
BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
debug: ignoring loglevel setting.last_pfn = 0x7ff0 max_arch_pfn = 0x100000
kernel direct mapping tables up to 7ff0000 @ 7000-d000
RAMDISK: 07b9b000 - 07fbf89d
DMI 2.5 present.
ACPI: RSDP 000E0000, 0024 (r2 VBOX )
ACPI: XSDT 07FF0030, 002C (r1 VBOX VBOXXSDT 1 ASL 61)
ACPI: FACP 07FF0060, 00F4 (r4 VBOX VBOXFACP 1 ASL 61)
ACPI: DSDT 07FF01A0, 1064 (r1 VBOX VBOXBIOS 2 INTL 20080213)
ACPI: FACS 07FF0160, 0040
0MB HIGHMEM available.
127MB LOWMEM available.
mapped low ram: 0 - 07ff0000
low ram: 00000000 - 07ff0000
bootmap 00002000 - 00003000
(9 early reservations) ==> bootmem [0000000000 - 0007ff0000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000]
#2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000]
#3 [0000100000 - 0000814b10] TEXT DATA BSS ==> [0000100000 - 0000814b10]
#4 [0007b9b000 - 0007fbf89d] RAMDISK ==> [0007b9b000 - 0007fbf89d]
#5 [0000815000 - 0000819000] INIT_PG_TABLE ==> [0000815000 - 0000819000]
#6 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000]
#7 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000]
#8 [0000002000 - 0000003000] BOOTMAP ==> [0000002000 - 0000003000]
Zone PFN ranges:
DMA 0x00000000 -> 0x00001000
Normal 0x00001000 -> 0x00007ff0
HighMem 0x00007ff0 -> 0x00007ff0
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000000 -> 0x0000009f
0: 0x00000100 -> 0x00007ff0On node 0 totalpages: 32655free_area_init_node: node 0, pgdat c041f600, node_mem_map c1000000
DMA zone: 3947 pages, LIFO batch:0
Normal zone: 28292 pages, LIFO batch:7
ACPI: PM-Timer IO Port: 0x4008
SMP: Allowing 1 CPUs, 0 hotplug CPUs
Found and enabled local APIC!
mapped APIC to ffffb000 (fee00000)
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 10000000 (gap: 8000000:f7fc0000)
PERCPU: Allocating 40224 bytes of per cpu dataNR_CPUS: 32, nr_cpu_ids: 1, nr_node_ids 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32239
Kernel command line: initrd=alt0/all.rdz vga=788 splash=silent BOOT_IMAGE=alt0/vmlinuz vga=0 console=ttyS0,9600 console=tty0 ignore_loglevel debug-alternative Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 512 (order: 9, 2048 bytes)
TSC calibrated against PM_TIMER
Detected 2410.976 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 48
... MAX_LOCKDEP_KEYS: 8191
... CLASSHASH_SIZE: 4096
... MAX_LOCKDEP_ENTRIES: 8192 ... MAX_LOCKDEP_CHAINS: 16384 ... CHAINHASH_SIZE: 8192
memory used by lock dependency info: 2335 kB
per task-struct memory footprint: 1152 bytes
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 117088k/131008k available (2340k kernel code, 13364k reserved, 1027k data, 308k init, 0k highmem) virtual kernel memory layout:
fixmap : 0xffe18000 - 0xfffff000 (1948 kB)
pkmap : 0xff800000 - 0xffc00000 (4096 kB)
vmalloc : 0xc8800000 - 0xff7fe000 ( 879 MB)
lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB)
.init : 0xc0451000 - 0xc049e000 ( 308 kB)
.data : 0xc03493b8 - 0xc044a040 (1027 kB)
.text : 0xc0100000 - 0xc03493b8 (2340 kB)Checking if this processor honours the WP bit even in supervisor mode...Ok.
CPA: page pool initialized 1 of 1 pages preallocated
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 4821.95 BogoMIPS (lpj=2410976) Security Framework initialized
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Checking 'hlt' instruction... OK.apply_alternatives: alt table c048afa4 -> c048f11f
SMP alternatives: switching to UP code
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20080609
ACPI: setting ELCR to 0200 (from 0c00)
weird, boot CPU (#0) not listedby the BIOS.
SMP motherboard not detected.
SMP disabled
Brought up 1 CPUs
Total of 1 processors activated (4821.95 BogoMIPS).
BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c012fc71>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in:
Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test25 #89)
EIP: 0060:[<c012fc71>] EFLAGS: 00010246 CPU: 0
EIP is at vprintk+0x181/0x440
EAX: 00000246 EBX: 00000000 ECX: c012f8a9 EDX: 00009695
ESI: 00000000 EDI: c04d76d7 EBP: c7435f98 ESP: c7435f24
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000)
Stack: 00009695 c012f8a9 c7435fb4 0000007b c012007b 000000d8 ffffff10 c7435f5a
00000006 00000246 00000000 c046902c 00000037 00000246 c03438ac c7435f7a
00000006 00000246 00000000 00000000 00000015 c7435f94 c045e900 00000030
Call Trace:
[<c012f8a9>] ? release_console_sem+0x1c9/0x1e0
[<c012007b>] ? resched_task+0x4b/0x70
[<c046902c>] ? relay_init+0xd/0x11
[<c03438ac>] ? end_local_APIC_setup+0xb9/0xf2
[<c045e900>] ? prefill_possible_map+0x7/0x8a
[<c03450f4>] ? printk+0x18/0x1c
[<c045eacd>] ? native_smp_cpus_done+0x93/0xe9
[<c04512f3>] ? kernel_init+0xd5/0x166
[<c045121e>] ? kernel_init+0x0/0x166
[<c0104b67>] ? kernel_thread_helper+0x7/0x10
=======================
Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8
EIP: [<c012fc71>] vprintk+0x181/0x440 SS:ESP 0069:c7435f24
---[ end trace 4eaa2a86a8e2da22 ]---
Kernel panic - not syncing: Attempted to kill init!Quoted 44 lines · * Luiz Fernando N. Capitulino ([email protected]) wrote:
Em Fri, 22 Aug 2008 11:34:52 -0400
Mathieu Desnoyers <[email protected]> escreveu:| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Fri, 22 Aug 2008 08:50:12 +0200 | > Ingo Molnar <[email protected]> escreveu: | > | > | | > | * H. Peter Anvin <[email protected]> wrote: | > | | > | > H. Peter Anvin wrote: | > | >>> | > | >>> Does this look like a kernel bug? | > | >>> | > | >> | > | >> No, it looks like a very common virtualizer bug. Does the attached | > | >> patch work for you? | > | >> | > | > | > | > Also, in addition to this, please try tip:master. There is a patch in | > | > tip:master which I hope should fix this problem, but the details are | > | > important. | > | | > | access coordinates would be at: | > | | > | http://people.redhat.com/mingo/tip.git/README | > | > As I already have Linus tree downloaded I have cloned it in | > the usual way. | > | > Got the same results: OOPS in virtualbox but it works on QEMU. | > | > The OOPS's output follows and I have attached the .config I'm using | > to reproduce the problem. | > | | Can you try booting with the kernel argument : | debug_alternative | | The dmesg of the kernel bootup up to the oops would be helpful. | | My guess is that there may be something wrong with irq disabling which | protects text_poke_early in apply_alternatives().
Quoted 9 lines
I have attached two files:
- normal.txt: normal boot with no debug options
- debug-alternative.txtignore_logleveland debug-alternative boot optionsI had to pass
ignore_loglevelotherwise it wouldn't print
anything.
Ok, now can you try booting with either of those args :
noreplace-paravirt
noreplace-smp
And see which one(s) works ?
Thanks,
Mathieu
Quoted 296 lines
Signature · 295 lines
Luiz Fernando N. Capitulino Linux version 2.6.27-rc4-test25 ([email protected]) (gcc version 4.3.1 20080626 (prerelease) (GCC) ) #89 SMP Fri Aug 22 12:47:34 BRT 2008 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable) BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data) BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) last_pfn = 0x7ff0 max_arch_pfn = 0x100000 RAMDISK: 07b9b000 - 07fbf89d DMI 2.5 present. ACPI: RSDP 000E0000, 0024 (r2 VBOX ) ACPI: XSDT 07FF0030, 002C (r1 VBOX VBOXXSDT 1 ASL 61) ACPI: FACP 07FF0060, 00F4 (r4 VBOX VBOXFACP 1 ASL 61) ACPI: DSDT 07FF01A0, 1064 (r1 VBOX VBOXBIOS 2 INTL 20080213) ACPI: FACS 07FF0160, 0040 0MB HIGHMEM available. 127MB LOWMEM available. mapped low ram: 0 - 07ff0000 low ram: 00000000 - 07ff0000 bootmap 00002000 - 00003000 (9 early reservations) ==> bootmem [0000000000 - 0007ff0000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000] #2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000] #3 [0000100000 - 0000814b10] TEXT DATA BSS ==> [0000100000 - 0000814b10] #4 [0007b9b000 - 0007fbf89d] RAMDISK ==> [0007b9b000 - 0007fbf89d] #5 [0000815000 - 0000819000] INIT_PG_TABLE ==> [0000815000 - 0000819000] #6 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000] #7 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000] #8 [0000002000 - 0000003000] BOOTMAP ==> [0000002000 - 0000003000] Zone PFN ranges: DMA 0x00000000 -> 0x00001000 Normal 0x00001000 -> 0x00007ff0 HighMem 0x00007ff0 -> 0x00007ff0 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000000 -> 0x0000009f 0: 0x00000100 -> 0x00007ff0 ACPI: PM-Timer IO Port: 0x4008 SMP: Allowing 1 CPUs, 0 hotplug CPUs Found and enabled local APIC! PM: Registered nosave memory: 000000000009f000 - 00000000000a0000 PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Allocating PCI resources starting at 10000000 (gap: 8000000:f7fc0000) PERCPU: Allocating 40224 bytes of per cpu data Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32239 Kernel command line: initrd=alt0/all.rdz vga=788 splash=silent BOOT_IMAGE=alt0/vmlinuz vga=0 console=ttyS0,9600 console=tty0 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 PID hash table entries: 512 (order: 9, 2048 bytes) TSC calibrated against PM_TIMER Detected 2410.453 MHz processor. Console: colour VGA+ 80x25 console [tty0] enabled console [ttyS0] enabled Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar ... MAX_LOCKDEP_SUBCLASSES: 8 ... MAX_LOCK_DEPTH: 48 ... MAX_LOCKDEP_KEYS: 8191 ... CLASSHASH_SIZE: 4096 ... MAX_LOCKDEP_ENTRIES: 8192 ... MAX_LOCKDEP_CHAINS: 16384 ... CHAINHASH_SIZE: 8192 memory used by lock dependency info: 2335 kB per task-struct memory footprint: 1152 bytes Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) Memory: 117088k/131008k available (2340k kernel code, 13364k reserved, 1027k data, 308k init, 0k highmem) virtual kernel memory layout: fixmap : 0xffe18000 - 0xfffff000 (1948 kB) pkmap : 0xff800000 - 0xffc00000 (4096 kB) vmalloc : 0xc8800000 - 0xff7fe000 ( 879 MB) lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB) .init : 0xc0451000 - 0xc049e000 ( 308 kB) .data : 0xc03493b8 - 0xc044a040 (1027 kB) .text : 0xc0100000 - 0xc03493b8 (2340 kB) Checking if this processor honours the WP bit even in supervisor mode...Ok. SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Calibrating delay loop (skipped), value calculated using timer frequency.. 4820.90 BogoMIPS (lpj=2410453) Security Framework initialized Mount-cache hash table entries: 512 CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. Checking 'hlt' instruction... OK. SMP alternatives: switching to UP code Freeing SMP alternatives: 10k freed ACPI: Core revision 20080609 ACPI: setting ELCR to 0200 (from 0c00) weird, boot CPU (#0) not listedby the BIOS. SMP motherboard not detected. SMP disabled Brought up 1 CPUs Total of 1 processors activated (4820.90 BogoMIPS). khelper used greatest stack depth: 7108 bytes left net_namespace: 384 bytes Booting paravirtualized kernel on bare hardware NET: Registered protocol family 16 ACPI: bus type pci registered PCI: PCI BIOS revision 2.10 entry at 0xfadb0, last bus=0 PCI: Using configuration type 1 for base access ACPI: Interpreter enabled ACPI: (supports S0 S5) ACPI: Using PIC for interrupt routing ACPI: PCI Root Bridge [PCI0] (0000:00) ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 10 *11) ACPI: PCI Interrupt Link [LNKD] (IRQs 5 9 *10 11) Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI init ACPI: bus type pnp registered BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c012fc71>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in: Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test25 #89) EIP: 0060:[<c012fc71>] EFLAGS: 00010246 CPU: 0 EIP is at vprintk+0x181/0x440 EAX: 00000246 EBX: 00000000 ECX: c012f8a9 EDX: 00003b3a ESI: 00000000 EDI: c04d76c1 EBP: c7435f20 ESP: c7435eac DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000) Stack: 00003b3a c012f8a9 c7435f3c c02948b1 c7435f18 c02957f6 00000074 c7435ee2 00000006 00000246 00000000 00000000 00000021 00000000 00000001 00000000 a027c4ab a027c4c8 00000001 00000297 00000246 00000000 00000001 00000000 Call Trace: [<c012f8a9>] ? release_console_sem+0x1c9/0x1e0 [<c02948b1>] ? put_device+0x11/0x20 [<c02957f6>] ? device_add+0x26/0x610 [<c0471c5c>] ? pnpacpi_init+0x0/0x89 [<c03450f4>] ? printk+0x18/0x1c [<c0266f87>] ? register_acpi_bus_type+0x58/0x69 [<c0471ca5>] ? pnpacpi_init+0x49/0x89 [<c0101116>] ? do_one_initcall+0x26/0x170 [<c01e1d14>] ? create_proc_entry+0x54/0xa0 [<c016ef86>] ? register_irq_proc+0xb6/0xd0 [<c016efea>] ? init_irq_proc+0x4a/0x60 [<c045132d>] ? kernel_init+0x10f/0x166 [<c045121e>] ? kernel_init+0x0/0x166 [<c0104b67>] ? kernel_thread_helper+0x7/0x10 ======================= Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8 EIP: [<c012fc71>] vprintk+0x181/0x440 SS:ESP 0069:c7435eac ---[ end trace 4eaa2a86a8e2da22 ]--- Kernel panic - not syncing: Attempted to kill init! Linux version 2.6.27-rc4-test25 ([email protected]) (gcc version 4.3.1 20080626 (prerelease) (GCC) ) #89 SMP Fri Aug 22 12:47:34 BRT 2008 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable) BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data) BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) debug: ignoring loglevel setting. last_pfn = 0x7ff0 max_arch_pfn = 0x100000 kernel direct mapping tables up to 7ff0000 @ 7000-d000 RAMDISK: 07b9b000 - 07fbf89d DMI 2.5 present. ACPI: RSDP 000E0000, 0024 (r2 VBOX ) ACPI: XSDT 07FF0030, 002C (r1 VBOX VBOXXSDT 1 ASL 61) ACPI: FACP 07FF0060, 00F4 (r4 VBOX VBOXFACP 1 ASL 61) ACPI: DSDT 07FF01A0, 1064 (r1 VBOX VBOXBIOS 2 INTL 20080213) ACPI: FACS 07FF0160, 0040 0MB HIGHMEM available. 127MB LOWMEM available. mapped low ram: 0 - 07ff0000 low ram: 00000000 - 07ff0000 bootmap 00002000 - 00003000 (9 early reservations) ==> bootmem [0000000000 - 0007ff0000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000] #2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000] #3 [0000100000 - 0000814b10] TEXT DATA BSS ==> [0000100000 - 0000814b10] #4 [0007b9b000 - 0007fbf89d] RAMDISK ==> [0007b9b000 - 0007fbf89d] #5 [0000815000 - 0000819000] INIT_PG_TABLE ==> [0000815000 - 0000819000] #6 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000] #7 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000] #8 [0000002000 - 0000003000] BOOTMAP ==> [0000002000 - 0000003000] Zone PFN ranges: DMA 0x00000000 -> 0x00001000 Normal 0x00001000 -> 0x00007ff0 HighMem 0x00007ff0 -> 0x00007ff0 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000000 -> 0x0000009f 0: 0x00000100 -> 0x00007ff0 On node 0 totalpages: 32655 free_area_init_node: node 0, pgdat c041f600, node_mem_map c1000000 DMA zone: 3947 pages, LIFO batch:0 Normal zone: 28292 pages, LIFO batch:7 ACPI: PM-Timer IO Port: 0x4008 SMP: Allowing 1 CPUs, 0 hotplug CPUs Found and enabled local APIC! mapped APIC to ffffb000 (fee00000) PM: Registered nosave memory: 000000000009f000 - 00000000000a0000 PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Allocating PCI resources starting at 10000000 (gap: 8000000:f7fc0000) PERCPU: Allocating 40224 bytes of per cpu data NR_CPUS: 32, nr_cpu_ids: 1, nr_node_ids 1 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32239 Kernel command line: initrd=alt0/all.rdz vga=788 splash=silent BOOT_IMAGE=alt0/vmlinuz vga=0 console=ttyS0,9600 console=tty0 ignore_loglevel debug-alternative Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 PID hash table entries: 512 (order: 9, 2048 bytes) TSC calibrated against PM_TIMER Detected 2410.976 MHz processor. Console: colour VGA+ 80x25 console [tty0] enabled console [ttyS0] enabled Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar ... MAX_LOCKDEP_SUBCLASSES: 8 ... MAX_LOCK_DEPTH: 48 ... MAX_LOCKDEP_KEYS: 8191 ... CLASSHASH_SIZE: 4096 ... MAX_LOCKDEP_ENTRIES: 8192 ... MAX_LOCKDEP_CHAINS: 16384 ... CHAINHASH_SIZE: 8192 memory used by lock dependency info: 2335 kB per task-struct memory footprint: 1152 bytes Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) Memory: 117088k/131008k available (2340k kernel code, 13364k reserved, 1027k data, 308k init, 0k highmem) virtual kernel memory layout: fixmap : 0xffe18000 - 0xfffff000 (1948 kB) pkmap : 0xff800000 - 0xffc00000 (4096 kB) vmalloc : 0xc8800000 - 0xff7fe000 ( 879 MB) lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB) .init : 0xc0451000 - 0xc049e000 ( 308 kB) .data : 0xc03493b8 - 0xc044a040 (1027 kB) .text : 0xc0100000 - 0xc03493b8 (2340 kB) Checking if this processor honours the WP bit even in supervisor mode...Ok. CPA: page pool initialized 1 of 1 pages preallocated SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Calibrating delay loop (skipped), value calculated using timer frequency.. 4821.95 BogoMIPS (lpj=2410976) Security Framework initialized Mount-cache hash table entries: 512 CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. Checking 'hlt' instruction... OK. apply_alternatives: alt table c048afa4 -> c048f11f SMP alternatives: switching to UP code Freeing SMP alternatives: 10k freed ACPI: Core revision 20080609 ACPI: setting ELCR to 0200 (from 0c00) weird, boot CPU (#0) not listedby the BIOS. SMP motherboard not detected. SMP disabled Brought up 1 CPUs Total of 1 processors activated (4821.95 BogoMIPS). BUG: unable to handle kernel NULL pointer dereference at 00000246 IP: [<c012fc71>] vprintk+0x181/0x440 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in: Pid: 1, comm: swapper Not tainted (2.6.27-rc4-test25 #89) EIP: 0060:[<c012fc71>] EFLAGS: 00010246 CPU: 0 EIP is at vprintk+0x181/0x440 EAX: 00000246 EBX: 00000000 ECX: c012f8a9 EDX: 00009695 ESI: 00000000 EDI: c04d76d7 EBP: c7435f98 ESP: c7435f24 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Process swapper (pid: 1, ti=c7434000 task=c7438000 task.ti=c7434000) Stack: 00009695 c012f8a9 c7435fb4 0000007b c012007b 000000d8 ffffff10 c7435f5a 00000006 00000246 00000000 c046902c 00000037 00000246 c03438ac c7435f7a 00000006 00000246 00000000 00000000 00000015 c7435f94 c045e900 00000030 Call Trace: [<c012f8a9>] ? release_console_sem+0x1c9/0x1e0 [<c012007b>] ? resched_task+0x4b/0x70 [<c046902c>] ? relay_init+0xd/0x11 [<c03438ac>] ? end_local_APIC_setup+0xb9/0xf2 [<c045e900>] ? prefill_possible_map+0x7/0x8a [<c03450f4>] ? printk+0x18/0x1c [<c045eacd>] ? native_smp_cpus_done+0x93/0xe9 [<c04512f3>] ? kernel_init+0xd5/0x166 [<c045121e>] ? kernel_init+0x0/0x166 [<c0104b67>] ? kernel_thread_helper+0x7/0x10 ======================= Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8 EIP: [<c012fc71>] vprintk+0x181/0x440 SS:ESP 0069:c7435f24 ---[ end trace 4eaa2a86a8e2da22 ]--- Kernel panic - not syncing: Attempted to kill init!
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Em Fri, 22 Aug 2008 12:35:20 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Fri, 22 Aug 2008 11:34:52 -0400 | > Mathieu Desnoyers <[email protected]> escreveu: | > | > | * Luiz Fernando N. Capitulino ([email protected]) wrote: | > | > Em Fri, 22 Aug 2008 08:50:12 +0200 | > | > Ingo Molnar <[email protected]> escreveu: | > | > | > | > | | > | > | * H. Peter Anvin <[email protected]> wrote: | > | > | | > | > | > H. Peter Anvin wrote: | > | > | >>> | > | > | >>> Does this look like a kernel bug? | > | > | >>> | > | > | >> | > | > | >> No, it looks like a very common virtualizer bug. Does the attached | > | > | >> patch work for you? | > | > | >> | > | > | > | > | > | > Also, in addition to this, please try tip:master. There is a patch in | > | > | > tip:master which I hope should fix this problem, but the details are | > | > | > important. | > | > | | > | > | access coordinates would be at: | > | > | | > | > | http://people.redhat.com/mingo/tip.git/README | > | > | > | > As I already have Linus tree downloaded I have cloned it in | > | > the usual way. | > | > | > | > Got the same results: OOPS in virtualbox but it works on QEMU. | > | > | > | > The OOPS's output follows and I have attached the .config I'm using | > | > to reproduce the problem. | > | > | > | | > | Can you try booting with the kernel argument : | > | debug_alternative | > | | > | The dmesg of the kernel bootup up to the oops would be helpful. | > | | > | My guess is that there may be something wrong with irq disabling which | > | protects text_poke_early in apply_alternatives(). | > | > I have attached two files: | > | > - normal.txt: normal boot with no debug options | > - debug-alternative.txt ignore_loglevel and debug-alternative boot | > options | > | > I had to pass ignore_loglevel otherwise it wouldn't print | > anything. | > | | Ok, now can you try booting with either of those args : | | noreplace-paravirt | noreplace-smp | | And see which one(s) works ?
noreplace-paravirt works, the other one causes no change.
You will find the full boot log (with debug-alternative enabled)
attached.
Signature · 1 line
Luiz Fernando N. Capitulino
Linux version 2.6.27-rc4-test25 ([email protected]) (gcc version 4.3.1 20080626 (prerelease) (GCC) ) #89 SMP Fri Aug 22 12:47:34 BRT 2008 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable)
BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
debug: ignoring loglevel setting.last_pfn = 0x7ff0 max_arch_pfn = 0x100000
kernel direct mapping tables up to 7ff0000 @ 7000-d000
RAMDISK: 07b9b000 - 07fbf89d
DMI 2.5 present.
ACPI: RSDP 000E0000, 0024 (r2 VBOX )
ACPI: XSDT 07FF0030, 002C (r1 VBOX VBOXXSDT 1 ASL 61)
ACPI: FACP 07FF0060, 00F4 (r4 VBOX VBOXFACP 1 ASL 61)
ACPI: DSDT 07FF01A0, 1064 (r1 VBOX VBOXBIOS 2 INTL 20080213)
ACPI: FACS 07FF0160, 0040
0MB HIGHMEM available.
127MB LOWMEM available.
mapped low ram: 0 - 07ff0000
low ram: 00000000 - 07ff0000
bootmap 00002000 - 00003000
(9 early reservations) ==> bootmem [0000000000 - 0007ff0000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000]
#2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000]
#3 [0000100000 - 0000814b10] TEXT DATA BSS ==> [0000100000 - 0000814b10]
#4 [0007b9b000 - 0007fbf89d] RAMDISK ==> [0007b9b000 - 0007fbf89d]
#5 [0000815000 - 0000819000] INIT_PG_TABLE ==> [0000815000 - 0000819000]
#6 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000]
#7 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000]
#8 [0000002000 - 0000003000] BOOTMAP ==> [0000002000 - 0000003000]
Zone PFN ranges:
DMA 0x00000000 -> 0x00001000
Normal 0x00001000 -> 0x00007ff0
HighMem 0x00007ff0 -> 0x00007ff0
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000000 -> 0x0000009f
0: 0x00000100 -> 0x00007ff0On node 0 totalpages: 32655free_area_init_node: node 0, pgdat c041f600, node_mem_map c1000000
DMA zone: 3947 pages, LIFO batch:0
Normal zone: 28292 pages, LIFO batch:7
ACPI: PM-Timer IO Port: 0x4008
SMP: Allowing 1 CPUs, 0 hotplug CPUs
Found and enabled local APIC!
mapped APIC to ffffb000 (fee00000)
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 10000000 (gap: 8000000:f7fc0000)
PERCPU: Allocating 40224 bytes of per cpu dataNR_CPUS: 32, nr_cpu_ids: 1, nr_node_ids 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32239
Kernel command line: initrd=alt0/all.rdz vga=788 splash=silent BOOT_IMAGE=alt0/vmlinuz vga=0 console=ttyS0,9600 console=tty ignore_loglevel debug-alternative noreplace-paravirt Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 512 (order: 9, 2048 bytes)
TSC calibrated against PM_TIMER
Detected 2410.580 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 48
... MAX_LOCKDEP_KEYS: 8191
... CLASSHASH_SIZE: 4096
... MAX_LOCKDEP_ENTRIES: 8192 ... MAX_LOCKDEP_CHAINS: 16384 ... CHAINHASH_SIZE: 8192
memory used by lock dependency info: 2335 kB
per task-struct memory footprint: 1152 bytes
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 117088k/131008k available (2340k kernel code, 13364k reserved, 1027k data, 308k init, 0k highmem) virtual kernel memory layout:
fixmap : 0xffe18000 - 0xfffff000 (1948 kB)
pkmap : 0xff800000 - 0xffc00000 (4096 kB)
vmalloc : 0xc8800000 - 0xff7fe000 ( 879 MB)
lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB)
.init : 0xc0451000 - 0xc049e000 ( 308 kB)
.data : 0xc03493b8 - 0xc044a040 (1027 kB)
.text : 0xc0100000 - 0xc03493b8 (2340 kB)Checking if this processor honours the WP bit even in supervisor mode...Ok.
CPA: page pool initialized 1 of 1 pages preallocated
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 4821.16 BogoMIPS (lpj=2410580) Security Framework initialized
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Checking 'hlt' instruction... OK.apply_alternatives: alt table c048afa4 -> c048f11f
SMP alternatives: switching to UP code
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20080609
ACPI: setting ELCR to 0200 (from 0c00)
weird, boot CPU (#0) not listedby the BIOS.
SMP motherboard not detected.
SMP disabled
Brought up 1 CPUs
Total of 1 processors activated (4821.16 BogoMIPS).
CPU0 attaching sched-domain:
domain 0: span 0 level CPU
groups: 0
khelper used greatest stack depth: 6824 bytes leftnet_namespace: 384 bytes
Booting paravirtualized kernel on bare hardware
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfadb0, last bus=0
PCI: Using configuration type 1 for base access
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
bus 00 -> node 0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 9 *10 11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 6 devices
ACPI: ACPI bus type pnp unregistered
PnPBIOS: Disabled by ACPI PNP
PCI: Using ACPI for IRQ routing
bus: 00 index 0 io port: [0, ffff]
bus: 00 index 1 mmio: [0, ffffffffffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
checking if image is initramfs...<7>Switched to high resolution mode on CPU 0
it is
Freeing initrd memory: 4242k freed
khelper used greatest stack depth: 6640 bytes left
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
apm: overridden by ACPI.
audit: initializing netlink socket (disabled)
type=2000 audit(1219414726.056:1): initialized
HugeTLB registered 4 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 237
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci 0000:00:02.0: Boot video device
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16450
brd: module loaded
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /class/input/input0
cpuidle: using governor ladder
cpuidle: using governor menu
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
TCP cubic registered
Using IPI No-Shortcut mode
registered taskstats version 1
Freeing unused kernel memory: 308k freed
H. Peter Anvin
reviewerHi Luiz, two more tests:
1. a small program to run in userspace and tell us what you get;
2. a patch against -linus for testing.
-hpa
#include <stdio.h>
int main(void)
{
unsigned int eax, edx;
asm(".byte 0x0f,0x1f,0x84,0x00,0x98,0x98,0x98,0x98,0x92,0x90,0x90,0x90"
: "=a" (eax), "=d" (edx)
: "a" (0x88776655), "d" (0x44332211)); printf("%08x:%08x\n", edx, eax);
return 0;
}
arch/x86/kernel/alternative.c +0 -2
@@ -157,8 +157,6 @@ static const struct nop { } noptypes[] = { { X86_FEATURE_K8, k8_nops }, { X86_FEATURE_K7, k7_nops }, - { X86_FEATURE_P4, p6_nops }, - { X86_FEATURE_P3, p6_nops }, { -1, NULL } };
Em Fri, 22 Aug 2008 11:11:25 -0700
"H. Peter Anvin" <[email protected]> escreveu:
| Hi Luiz, two more tests: | | 1. a small program to run in userspace and tell us what you get;
88776655:44332211
It is the same output in the virtualized system and the host
system.
| 2. a patch against -linus for testing.
I have tried this patch with Linus tree early today, should I try
it with Ingo's tree too?
Signature · 1 line
Luiz Fernando N. Capitulino
H. Peter Anvin
reviewerQuoted 6 lines · Luiz Fernando N. Capitulino
| 2. a patch against -linus for testing.
I have tried this patch with Linus tree early today, should I try
it with Ingo's tree too?
It doesn't apply to tip. This did not fix the problem?
-hpa
Em Fri, 22 Aug 2008 13:31:49 -0700
"H. Peter Anvin" <[email protected]> escreveu:
| Luiz Fernando N. Capitulino wrote: | > | > | 2. a patch against -linus for testing. | > | > I have tried this patch with Linus tree early today, should I try | > it with Ingo's tree too? | > | | It doesn't apply to tip. This did not fix the problem?
No, it did not. :(
Signature · 1 line
Luiz Fernando N. Capitulino
Em Fri, 22 Aug 2008 14:20:54 -0300
"Luiz Fernando N. Capitulino" <[email protected]> escreveu:
| Em Fri, 22 Aug 2008 12:35:20 -0400 | Mathieu Desnoyers <[email protected]> escreveu: | | | * Luiz Fernando N. Capitulino ([email protected]) wrote: | | > Em Fri, 22 Aug 2008 11:34:52 -0400 | | > Mathieu Desnoyers <[email protected]> escreveu: | | > | | > | * Luiz Fernando N. Capitulino ([email protected]) wrote: | | > | > Em Fri, 22 Aug 2008 08:50:12 +0200 | | > | > Ingo Molnar <[email protected]> escreveu: | | > | > | | > | > | | | > | > | * H. Peter Anvin <[email protected]> wrote: | | > | > | | | > | > | > H. Peter Anvin wrote: | | > | > | >>> | | > | > | >>> Does this look like a kernel bug? | | > | > | >>> | | > | > | >> | | > | > | >> No, it looks like a very common virtualizer bug. Does the attached | | > | > | >> patch work for you? | | > | > | >> | | > | > | > | | > | > | > Also, in addition to this, please try tip:master. There is a patch in | | > | > | > tip:master which I hope should fix this problem, but the details are | | > | > | > important. | | > | > | | | > | > | access coordinates would be at: | | > | > | | | > | > | http://people.redhat.com/mingo/tip.git/README | | > | > | | > | > As I already have Linus tree downloaded I have cloned it in | | > | > the usual way. | | > | > | | > | > Got the same results: OOPS in virtualbox but it works on QEMU. | | > | > | | > | > The OOPS's output follows and I have attached the .config I'm using | | > | > to reproduce the problem. | | > | > | | > | | | > | Can you try booting with the kernel argument : | | > | debug_alternative | | > | | | > | The dmesg of the kernel bootup up to the oops would be helpful. | | > | | | > | My guess is that there may be something wrong with irq disabling which | | > | protects text_poke_early in apply_alternatives(). | | > | | > I have attached two files: | | > | | > - normal.txt: normal boot with no debug options | | > - debug-alternative.txt ignore_loglevel and debug-alternative boot | | > options | | > | | > I had to pass ignore_loglevel otherwise it wouldn't print | | > anything. | | > | | | | Ok, now can you try booting with either of those args : | | | | noreplace-paravirt | | noreplace-smp | | | | And see which one(s) works ? | | noreplace-paravirt works, the other one causes no change.
I have asked Mandriva and Ubuntu users to test this and all of
them so far are saying that noreplace-paravirt works.
It makes the system slower, but it works.
Signature · 1 line
Luiz Fernando N. Capitulino
H. Peter Anvin
reviewerQuoted 6 lines · Luiz Fernando N. Capitulino
I have asked Mandriva and Ubuntu users to test this and all of
them so far are saying that noreplace-paravirt works.It makes the system slower, but it works.
Yes, the big issue is exactly what VirtualBox screws up in this matter, how to detect it, and how to work around it.
It's pretty clear it's a VirtualBox f*ckup at this point, but the failure mechanism isn't at all obvious and so far the workaround is elusive.
I'm strongly suspect this is a VirtualBox tcache management failure, but that doesn't help the situation without knowing how it happens.
-hpa
Quoted 8 lines · H. Peter Anvin
Quoted 6 lines · Luiz Fernando N. Capitulino
I have asked Mandriva and Ubuntu users to test this and all of
them so far are saying that noreplace-paravirt works.It makes the system slower, but it works.
Quoted 8 lines
Yes, the big issue is exactly what VirtualBox screws up in this matter, how to detect it, and how to work around it.
It's pretty clear it's a VirtualBox f*ckup at this point, but the failure mechanism isn't at all obvious and so far the workaround is elusive.
I'm strongly suspect this is a VirtualBox tcache management failure, but that doesn't help the situation without knowing how it happens.
On Archlinux we have the same problem. We have a bugreport here:
http://bugs.archlinux.org/task/11141
Myself test it with a LiveCD/Install-ISO which has 2.6.26 as install kernel. We have the guest oops on virtualbox-ose, virtualbox-sun and both on i686 or x86_64 hosts.
Some things i noticed:
- The system boots always when i either enable VT-x in guest settings or disable acpi and run the guest with acpi=off.
- The oops occurs always on (disk)-io, no matter which file system i use.
- When the oops has occured and the guest has to close and restart then, if i don't use VT-x or acpi=off, i always get an oops directly when initrd/kernel is starting. Last screen message before the oops then is "Freeing SMP alternatives".
Here is also an archive with guest dmesg and messages.log from such an oops when heavy disk io leads to the oops:
http://bugs.archlinux.org/task/11141?getfile=2445
Quoted 1 line
-hpa
Gerhard
Signature · 2 lines
Standards sind eine tolle Sache. Ich finde, jeder sollte einen haben.
Quoted 35 lines · * Gerhard Brauer ([email protected]) wrote:
Quoted 8 lines · H. Peter Anvin
Quoted 6 lines · Luiz Fernando N. Capitulino
I have asked Mandriva and Ubuntu users to test this and all of
them so far are saying that noreplace-paravirt works.It makes the system slower, but it works.
Quoted 8 lines
Yes, the big issue is exactly what VirtualBox screws up in this matter, how to detect it, and how to work around it.
It's pretty clear it's a VirtualBox f*ckup at this point, but the failure mechanism isn't at all obvious and so far the workaround is elusive.
I'm strongly suspect this is a VirtualBox tcache management failure, but that doesn't help the situation without knowing how it happens.
On Archlinux we have the same problem. We have a bugreport here:
http://bugs.archlinux.org/task/11141Myself test it with a LiveCD/Install-ISO which has 2.6.26 as install kernel. We have the guest oops on virtualbox-ose, virtualbox-sun and both on i686 or
x86_64hosts.Some things i noticed:
- The system boots always when i either enable VT-x in guest settings or disable acpi and run the guest with acpi=off.
- The oops occurs always on (disk)-io, no matter which file system i use.
- When the oops has occured and the guest has to close and restart then, if i don't use VT-x or acpi=off, i always get an oops directly when initrd/kernel is starting. Last screen message before the oops then is "Freeing SMP alternatives".
Quoted 4 lines
Here is also an archive with guest dmesg and messages.log from such an oops when heavy disk io leads to the oops:
http://bugs.archlinux.org/task/11141?getfile=2445
Hrm, can you try this ?
1 - Make sure you kernel is not CONFIG_DEBUG_RODATA
2 - Change the whole text_poke implementation inarch/x86/kernel/alternative.c to this :
void *__kprobes text_poke(void *addr, const void *opcode, size_t len) {
return text_poke_early(addr, opcode, len);
}
If this works, I suspect that the problem comes from a vmap/vunmap problem. If it still fails, the problem would likely come from a race with interrupt disabling probably due to missing data/instruction cache flush.
Then, after having tested (2), try this on top of it :
In arch/x86/kernel/alternative.c, alternatives_smp_switch()
Add unsigned long flags;
Changespin_lock -> spin_lock_irqsave(&smp_alt, flags); spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags);
This will help testing if there is a problem with interrupts coming shortly after the modification. If it fixes the problem, my guess is that we should flush the instruction cache (and maybe the data cache ?) in text_poke and text_poke early when interrupts are off.
Mathieu
Quoted 8 lines
Quoted 2 lines
-hpa
Gerhard
Signature · 2 lines
Standards sind eine tolle Sache. Ich finde, jeder sollte einen haben.
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Em Tue, 26 Aug 2008 10:53:38 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| * Gerhard Brauer ([email protected]) wrote: | > On Fri, Aug 22, 2008 at 02:08:13PM -0700, H. Peter Anvin wrote: | > > Luiz Fernando N. Capitulino wrote: | > >> | > >> I have asked Mandriva and Ubuntu users to test this and all of | > >> them so far are saying that noreplace-paravirt works. | > >> | > >> It makes the system slower, but it works. | > >> | > > | > > Yes, the big issue is exactly what VirtualBox screws up in this matter, | > > how to detect it, and how to work around it. | > > | > > It's pretty clear it's a VirtualBox f*ckup at this point, but the failure | > > mechanism isn't at all obvious and so far the workaround is elusive. | > > | > > I'm strongly suspect this is a VirtualBox tcache management failure, but | > > that doesn't help the situation without knowing how it happens. | > | > On Archlinux we have the same problem. We have a bugreport here: | > http://bugs.archlinux.org/task/11141 | > | > Myself test it with a LiveCD/Install-ISO which has 2.6.26 as install | > kernel. We have the guest oops on virtualbox-ose, virtualbox-sun and both on | > i686 or x86_64 hosts. | > | > Some things i noticed: | > - The system boots always when i either enable VT-x in guest settings or | > disable acpi and run the guest with acpi=off. | > - The oops occurs always on (disk)-io, no matter which file system i | > use. | > - When the oops has occured and the guest has to close and restart then, | > if i don't use VT-x or acpi=off, i always get an oops directly when | > initrd/kernel is starting. Last screen message before the oops then is | > "Freeing SMP alternatives". | > | > Here is also an archive with guest dmesg and messages.log from such an | > oops when heavy disk io leads to the oops: | > http://bugs.archlinux.org/task/11141?getfile=2445 | > | | Hrm, can you try this ? | | 1 - Make sure you kernel is not CONFIG_DEBUG_RODATA
"""
$ grep CONFIG_DEBUG_RODATA .config # CONFIG_DEBUG_RODATA is not set $ """
| 2 - Change the whole text_poke implementation in
| arch/x86/kernel/alternative.c to this :
|
| void *__kprobes text_poke(void *addr, const void *opcode, size_t len)
| {
| return text_poke_early(addr, opcode, len);
| }
|
| If this works, I suspect that the problem comes from a vmap/vunmap
| problem. If it still fails, the problem would likely come from a race
| with interrupt disabling probably due to missing data/instruction cache
| flush.I still get the oops with this change. :((
| Then, after having tested (2), try this on top of it : | | In arch/x86/kernel/alternative.c, alternatives_smp_switch() | | Add unsigned long flags; | Change | spin_lock -> spin_lock_irqsave(&smp_alt, flags); | spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags); | | This will help testing if there is a problem with interrupts coming | shortly after the modification. If it fixes the problem, my guess is | that we should flush the instruction cache (and maybe the data cache ?) | in text_poke and text_poke early when interrupts are off.
By 'on top of it' you mean I should make these changes with thetext_poke() version above right?
By the way, I have added a comment in the virtualbox's bugzilla
pointing out this thread but no feedback from them so far.
Signature · 1 line
Luiz Fernando N. Capitulino
Em Tue, 26 Aug 2008 10:53:38 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| Then, after having tested (2), try this on top of it : | | In arch/x86/kernel/alternative.c, alternatives_smp_switch() | | Add unsigned long flags; | Change | spin_lock -> spin_lock_irqsave(&smp_alt, flags); | spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags);
Hmm, I can't find spin_lock functions in alternatives_smp_switch()
looks like the current implementation is now using mutexes.
What tree are you referring to?
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 12 lines · * Luiz Fernando N. Capitulino ([email protected]) wrote:
Em Tue, 26 Aug 2008 10:53:38 -0400
Mathieu Desnoyers <[email protected]> escreveu:| Then, after having tested (2), try this on top of it : | | In arch/x86/kernel/alternative.c, alternatives_smp_switch() | | Add unsigned long flags; | Change | spin_lock -> spin_lock_irqsave(&smp_alt, flags); | spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags);
Quoted 3 lines
Hmm, I can't find
spin_lockfunctions inalternatives_smp_switch()
looks like the current implementation is now using mutexes.
Sorry, I was looking directly at the commit which caused the problem. Yes, these modif should go on top of the text_poke -> text_poke_early.
So in current mainline, change, in alternatives_smp_switch() :
mutex_lock(&smp_alt);
...
mutex_unlock(&smp_alt);
to
mutex_lock(&smp_alt);local_irq_save(flags);
...
local_irq_restore(flags);mutex_unlock(&smp_alt);
Thanks,
Mathieu
Quoted 4 lines
What tree are you referring to?
Signature · 1 line
Luiz Fernando N. Capitulino
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerI have been unable to replicate this on my own hardware mostly because my testing machine decided to blow its DVD drive in some very strange way, but I did pick apart the data from Luiz, and found it very interesting:
The code sequence before patching looks like:
c012fc69: 51 push %ecx c012fc6a: 52 push %edx c012fc6b: ff 15 40 b9 41 c0 call *0xc041b940 c012fc71: 5a pop %edx c012fc72: 59 pop %ecx
After patching:
50 9d 0f 1f 84 00 00 00 <00> 00
... which disassembles to (in Intel notation):
C012FC69 50 push eax C012FC6A 9D popfd C012FC6B 0F1F840000000000 nop dword [eax+eax+0x0]
We do, indeed have a return point that falls in the *middle* of a patched instruction, and if the patching happens in the middle of the instruction call, then, well, bad things happen.
Furthermore, why on Earth is %ecx/%edx pushed and popped in-line here? Surely it should be the responsibility of the PV call to present a no-clobber interface (using an assembly wrapper if necessary[*]), rather than bloating every callsite like this?
-hpa
[*] One can compile gcc code with -fcall-saved-* to use nonstandard register conventions. Unfortunately stock gcc only lets you do this with a file parameter, and doesn't support doing this with attributes.
Em Tue, 26 Aug 2008 13:18:22 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Tue, 26 Aug 2008 10:53:38 -0400 | > Mathieu Desnoyers <[email protected]> escreveu: | > | > | Then, after having tested (2), try this on top of it : | > | | > | In arch/x86/kernel/alternative.c, alternatives_smp_switch() | > | | > | Add unsigned long flags; | > | Change | > | spin_lock -> spin_lock_irqsave(&smp_alt, flags); | > | spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags); | > | > Hmm, I can't find spin_lock functions in alternatives_smp_switch() | > looks like the current implementation is now using mutexes. | > | | Sorry, I was looking directly at the commit which caused the problem. | Yes, these modif should go on top of the text_poke -> text_poke_early. | | So in current mainline, change, in alternatives_smp_switch() : | | mutex_lock(&smp_alt); | ... | | mutex_unlock(&smp_alt); | | to | | mutex_lock(&smp_alt); | local_irq_save(flags); | ... | | local_irq_restore(flags); | mutex_unlock(&smp_alt);
Did not help, same oops here.
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 39 lines · * Luiz Fernando N. Capitulino ([email protected]) wrote:
Em Tue, 26 Aug 2008 13:18:22 -0400
Mathieu Desnoyers <[email protected]> escreveu:| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Tue, 26 Aug 2008 10:53:38 -0400 | > Mathieu Desnoyers <[email protected]> escreveu: | > | > | Then, after having tested (2), try this on top of it : | > | | > | In arch/x86/kernel/alternative.c, alternatives_smp_switch() | > | | > | Add unsigned long flags; | > | Change | > | spin_lock -> spin_lock_irqsave(&smp_alt, flags); | > | spin_unlock(&smp_alt); -> spin_unlock_irqrestore(&smp_alt, flags); | > | > Hmm, I can't find spin_lock functions in alternatives_smp_switch() | > looks like the current implementation is now using mutexes. | > | | Sorry, I was looking directly at the commit which caused the problem. | Yes, these modif should go on top of the text_poke -> text_poke_early. | | So in current mainline, change, in alternatives_smp_switch() : | | mutex_lock(&smp_alt); | ... | | mutex_unlock(&smp_alt); | | to | | mutex_lock(&smp_alt); | local_irq_save(flags); | ... | | local_irq_restore(flags); | mutex_unlock(&smp_alt);
Quoted 2 lines
Did not help, same oops here.
Ok, it might still be caused by paravirt and alternatives instruction patching. What if you also do :
alternative_instructions()
+ unsigned long flags;
/* The patching is not fully atomic, so try to avoid local interruptions
that might execute the to be patched code.
Other CPUs are not running. */
stop_nmi();
#ifdef CONFIG_X86_MCE
stop_mce();#endif
+ local_irq_save(flags);
...
+ local_irq_restore(flags);
restart_nmi();
#ifdef CONFIG_X86_MCE
restart_mce();#endif
?
Hrm,
Since those local_irq_save/restore occur _before_ the paravirt patching is done, I wonder if there would be a race in the way cli/sti traps are handled by Virtualbox wrt incoming interrupt ?
Thanks,
Mathieu
Quoted 2 lines
Signature · 1 line
Luiz Fernando N. Capitulino
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerOne thing that I think really needs to be considered is that the current PV stubs are (a) large, and (b) non-atomic.
In the case at hand we have:
c012fc69: 51 push %ecx c012fc6a: 52 push %edx c012fc6b: ff 15 40 b9 41 c0 call *0xc041b940 c012fc71: 5a pop %edx c012fc72: 59 pop %ecx
Ten bytes replacing a two-byte native sequence.
If this was done as a call to an out-of-line stub, it would be only five bytes, which would reduce native icache overhead from 400% to 150%, but perhaps more importantly, it would not be subject to returns inside the sequence itself (since the out-of-line stub would still exist.) As an optional bonus, at least on 32 bits the indirect call could be replaced with a direct call in the out-of-line stub.
-hpa
Quoted 22 lines · Mathieu Desnoyers
Ok, it might still be caused by paravirt and alternatives instruction patching. What if you also do :
alternative_instructions()+ unsigned long flags;
/* The patching is not fully atomic, so try to avoid local interruptions
that might execute the to be patched code.
Other CPUs are not running. */
stop_nmi();#ifdef CONFIG_X86_MCE stop_mce();#endif
(quoted diff) +2
-0+ local_irq_save(flags); + local_irq_restore(flags); restart_nmi();#ifdef CONFIG_X86_MCE restart_mce();
Quoted 3 lines
#endif
?
Hej! This last changes (in addition to the others you mentioned) seems to be a good shot. I could reboot 8 times the guest, compile several packages (something which always leeds to the oops) and currently i build two big packages simultan. So this is heavy IO.
I will try tomorrow more heavy build tests (to gain the good feeling to the vbox+guest kernel again like it was with 2.6.25), but i think your changes goes in the right direction.
Here is the diff what i've changed on your hints:
,----[ arch/x86/kernel/alternative.c ]
| --- alternative.c.org 2008-07-13 23:51:29.000000000 +0200
| +++ alternative.c 2008-08-26 21:35:20.000000000 +0200
| @@ -343,6 +343,7 @@
| void alternatives_smp_switch(int smp)
| {
| struct smp_alt_module *mod;
| + unsigned long flags;
|
| #ifdef CONFIG_LOCKDEP
| /*
| @@ -359,7 +360,7 @@
| return;
| BUG_ON(!smp && (num_online_cpus() > 1));
|
| - spin_lock(&smp_alt);
| + spin_lock_irqsave(&smp_alt, flags);
|
| /*
| * Avoid unnecessary switches because it forces JIT based VMs to
| @@ -383,7 +384,7 @@
| mod->text, mod->text_end);
| }
| smp_mode = smp;
| - spin_unlock(&smp_alt);
| + spin_unlock_irqrestore(&smp_alt, flags);
| }
|
| #endif
| @@ -420,6 +421,7 @@
|
| void __init alternative_instructions(void)
| {
| + unsigned long flags;
| /* The patching is not fully atomic, so try to avoid local interruptions
| that might execute the to be patched code.
| Other CPUs are not running. */
| @@ -427,6 +429,7 @@
| #ifdef CONFIG_X86_MCE
| stop_mce();
| #endif
| + local_irq_save(flags);
|
| apply_alternatives(__alt_instructions, __alt_instructions_end);
|
| @@ -465,6 +468,7 @@
| (unsigned long)__smp_locks,
| (unsigned long)__smp_locks_end);
|
| + local_irq_restore(flags);
| restart_nmi();
| #ifdef CONFIG_X86_MCE
| restart_mce();
| @@ -508,33 +512,5 @@
| */
| void *__kprobes text_poke(void *addr, const void *opcode, size_t len)
| {
| - unsigned long flags;
| - char *vaddr;
| - int nr_pages = 2;
| - struct page *pages[2];
| - int i;
| -
| - if (!core_kernel_text((unsigned long)addr)) {
| - pages[0] = vmalloc_to_page(addr);
| - pages[1] = vmalloc_to_page(addr + PAGE_SIZE);
| - } else {
| - pages[0] = virt_to_page(addr);
| - WARN_ON(!PageReserved(pages[0]));
| - pages[1] = virt_to_page(addr + PAGE_SIZE);
| - }
| - BUG_ON(!pages[0]);
| - if (!pages[1])
| - nr_pages = 1;
| - vaddr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL);
| - BUG_ON(!vaddr);
| - local_irq_save(flags);
| - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len);
| - local_irq_restore(flags);
| - vunmap(vaddr);
| - sync_core();
| - /* Could also do a CLFLUSH here to speed up CPU recovery; but
| - that causes hangs on some VIA CPUs. */
| - for (i = 0; i < len; i++)
| - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]);
| - return addr;
| + return text_poke_early(addr, opcode, len);
| }`----
So if Luiz and others could also try all 3 mentioned changes, maybe we have a solution. I also will build tomorrow a new LiveCD/Install ISO with these patches to see if the error there is also gone.
Quoted 3 lines
Thanks,
Mathieu
Gerhard
Signature · 2 lines
Was wir wissen, ist ein Tropfen. Was wir nicht wissen, ein Ozean (Newton)
Quoted 27 lines · * Gerhard Brauer ([email protected]) wrote:
Quoted 22 lines · Mathieu Desnoyers
Ok, it might still be caused by paravirt and alternatives instruction patching. What if you also do :
alternative_instructions()+ unsigned long flags;
/* The patching is not fully atomic, so try to avoid local interruptions
that might execute the to be patched code.
Other CPUs are not running. */
stop_nmi();#ifdef CONFIG_X86_MCE stop_mce();#endif
(quoted diff) +2
-0+ local_irq_save(flags); + local_irq_restore(flags); restart_nmi();#ifdef CONFIG_X86_MCE restart_mce();Quoted 3 lines
#endif
?
Quoted 9 lines
Hej! This last changes (in addition to the others you mentioned) seems to be a good shot. I could reboot 8 times the guest, compile several packages (something which always leeds to the oops) and currently i build two big packages simultan. So this is heavy IO.
I will try tomorrow more heavy build tests (to gain the good feeling to the vbox+guest kernel again like it was with 2.6.25), but i think your changes goes in the right direction.
OK, so we have a problem with interrupts coming while we are doing the alternatives patching.
First thing, I wonder if Virtualbox expects the OS to patch all its paravirt instructions in one go ?
Also, could you then try to :
- to revert all those changes
- Do this to text_poke_early and text_poke :
- put the sync_core() within the irq off critical section (test)
- add a wbinvd(); just after the sync_core() in both functions (test).
Thanks,
Mathieu
Quoted 106 lines
Here is the diff what i've changed on your hints:
,----[
arch/x86/kernel/alternative.c]| --- alternative.c.org 2008-07-13 23:51:29.000000000 +0200 | +++ alternative.c 2008-08-26 21:35:20.000000000 +0200 | @@ -343,6 +343,7 @@ | void alternatives_smp_switch(int smp) | { | struct smp_alt_module *mod; | + unsigned long flags; | | #ifdef CONFIG_LOCKDEP | /* | @@ -359,7 +360,7 @@ | return; | BUG_ON(!smp && (num_online_cpus() > 1)); | | - spin_lock(&smp_alt); | + spin_lock_irqsave(&smp_alt, flags); | | /* | * Avoid unnecessary switches because it forces JIT based VMs to | @@ -383,7 +384,7 @@ | mod->text, mod->text_end); | } | smp_mode = smp; | - spin_unlock(&smp_alt); | + spin_unlock_irqrestore(&smp_alt, flags); | } | | #endif | @@ -420,6 +421,7 @@ | | void __init alternative_instructions(void) | { | + unsigned long flags; | /* The patching is not fully atomic, so try to avoid local interruptions | that might execute the to be patched code. | Other CPUs are not running. */ | @@ -427,6 +429,7 @@ | #ifdef CONFIG_X86_MCE | stop_mce(); | #endif | + local_irq_save(flags); | | apply_alternatives(__alt_instructions, __alt_instructions_end); | | @@ -465,6 +468,7 @@ | (unsigned long)__smp_locks, | (unsigned long)__smp_locks_end); | | + local_irq_restore(flags); | restart_nmi(); | #ifdef CONFIG_X86_MCE | restart_mce(); | @@ -508,33 +512,5 @@ | */ | void *__kprobes text_poke(void *addr, const void *opcode, size_t len) | { | - unsigned long flags; | - char *vaddr; | - int nr_pages = 2; | - struct page *pages[2]; | - int i; | - | - if (!core_kernel_text((unsigned long)addr)) { | - pages[0] = vmalloc_to_page(addr); | - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); | - } else { | - pages[0] = virt_to_page(addr); | - WARN_ON(!PageReserved(pages[0])); | - pages[1] = virt_to_page(addr + PAGE_SIZE); | - } | - BUG_ON(!pages[0]); | - if (!pages[1]) | - nr_pages = 1; | - vaddr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); | - BUG_ON(!vaddr); | - local_irq_save(flags); | - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len); | - local_irq_restore(flags); | - vunmap(vaddr); | - sync_core(); | - /* Could also do a CLFLUSH here to speed up CPU recovery; but | - that causes hangs on some VIA CPUs. */ | - for (i = 0; i < len; i++) | - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]); | - return addr; | + return text_poke_early(addr, opcode, len); | }`----
So if Luiz and others could also try all 3 mentioned changes, maybe we have a solution. I also will build tomorrow a new LiveCD/Install ISO with these patches to see if the error there is also gone.
Quoted 3 lines
Thanks,
Mathieu
Gerhard
Signature · 2 lines
Was wir wissen, ist ein Tropfen. Was wir nicht wissen, ein Ozean (Newton)
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 7 lines · Mathieu Desnoyers
OK, so we have a problem with interrupts coming while we are doing the alternatives patching.
First thing, I wonder if Virtualbox expects the OS to patch all its paravirt instructions in one go ?
Quoted 6 lines
Also, could you then try to :
- to revert all those changes
- Do this totext_poke_earlyandtext_poke:- put the
sync_core()within the irq off critical section (test)
Could you please explain more what to change? I don't see where to put sync_core(), i not found this section in both functions. (I'm not a developer)
Quoted 2 lines
- add a
wbinvd(); just after thesync_core()in both functions (test).
Also verbose please...
Quoted 3 lines
Thanks,
Mathieu
Thank you
Gerhard
Quoted 12 lines · * Gerhard Brauer ([email protected]) wrote:
Quoted 11 lines · Mathieu Desnoyers
OK, so we have a problem with interrupts coming while we are doing the alternatives patching.
First thing, I wonder if Virtualbox expects the OS to patch all its paravirt instructions in one go ?
Also, could you then try to :
- to revert all those changes
- Do this totext_poke_earlyandtext_poke:
Quoted 6 lines
Quoted 2 lines
- put the
sync_core()within the irq off critical section (test)Could you please explain more what to change? I don't see where to put
sync_core(), i not found this section in both functions. (I'm not a developer)
Sure,
First patch to test :
x86 alternative text_poke move sync_core
--- arch/x86/kernel/alternative.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/alternative.c +2 -2
@@ -488,8 +488,8 @@ void *text_poke_early(void *addr, constunsigned long flags;local_irq_save (flags);memcpy (addr, opcode, len); -local_irq_restore (flags);sync_core (); +local_irq_restore (flags);/* Could also do a CLFLUSH here to speed up CPU recovery; but that causes hangs on some VIA CPUs. */ return addr;
@@ -529,9 +529,9 @@ void *__kprobes text_poke(void *addr, coBUG_ON (!vaddr);local_irq_save (flags);memcpy (&vaddr[(unsigned long )addr & ~PAGE_MASK ], opcode, len); +sync_core ();local_irq_restore (flags);vunmap (vaddr); -sync_core ();/* Could also do a CLFLUSH here to speed up CPU recovery; but that causes hangs on some VIA CPUs. */ for (i =0 ; i < len; i++)
Quoted 5 lines
Quoted 2 lines
- add a
wbinvd(); just after thesync_core()in both functions (test).Also verbose please...
Second patch to apply on top of the first one :
x86 alternative text_poke add wbinvd
Add a cache flush instruction before reenabling interrupts in text_poke.
If this works, we could use clflush() (which is sadly buggy on some archs) which is faster since it only clear a cacheline instead of the entire cache.
--- arch/x86/kernel/alternative.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/alternative.c +2 -0
@@ -489,6 +489,7 @@ void *text_poke_early(void *addr, constlocal_irq_save (flags);memcpy (addr, opcode, len);sync_core (); +wbinvd ();local_irq_restore (flags);/* Could also do a CLFLUSH here to speed up CPU recovery; but that causes hangs on some VIA CPUs. */
@@ -530,6 +531,7 @@ void *__kprobes text_poke(void *addr, colocal_irq_save (flags);memcpy (&vaddr[(unsigned long )addr & ~PAGE_MASK ], opcode, len);sync_core (); +wbinvd ();local_irq_restore (flags);vunmap (vaddr);/* Could also do a CLFLUSH here to speed up CPU recovery; but
Thanks,
Mathieu
Quoted 7 lines
Quoted 3 lines
Thanks,
Mathieu
Thank you
Gerhard
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 8 lines · Mathieu Desnoyers
x86 alternative
text_pokeadd wbinvdAdd a cache flush instruction before reenabling interrupts in
text_poke.If this works, we could use
clflush()(which is sadly buggy on some archs) which is faster since it only clear a cacheline instead of the entire cache.
Well, in this case it's VirtualBox we're talking about, a virtual architecture. It's hard to know what it will do under *any* circumstances.
-hpa
Quoted 5 lines · Mathieu Desnoyers
* Gerhard Brauer ([email protected]) wrote:
First patch to test :
x86 alternative
text_pokemovesync_core
With this got the oops again when compiling in guest. Reboot afterwards leads to the early oops.
Quoted 3 lines
Second patch to apply on top of the first one :
x86 alternative
text_pokeadd wbinvd
With second patch i get the early oops after "freeing smp". Seems no way to get the guest bootet normaly (ony with replace-paravirt).
So the changes from the other mail took more effect IMHO. I could test more tomorrow if you have more ideas.
Quoted 1 line
Mathieu
Gerhard
Em Tue, 26 Aug 2008 22:34:49 +0200
Gerhard Brauer <[email protected]> escreveu:
| On Tue, Aug 26, 2008 at 02:15:58PM -0400, Mathieu Desnoyers wrote: | > | > Ok, it might still be caused by paravirt and alternatives instruction | > patching. What if you also do : | > | > alternative_instructions() | > | > + unsigned long flags; | > /* The patching is not fully atomic, so try to avoid local interruptions | > that might execute the to be patched code. | > Other CPUs are not running. */ | > stop_nmi(); | > #ifdef CONFIG_X86_MCE | > stop_mce(); | > #endif | > + local_irq_save(flags); | > | > | > ... | > + local_irq_restore(flags); | > restart_nmi(); | > #ifdef CONFIG_X86_MCE | > restart_mce(); | > #endif | > | > ? | | Hej! This last changes (in addition to the others you mentioned) seems | to be a good shot. I could reboot 8 times the guest, compile several | packages (something which always leeds to the oops) and currently i | build two big packages simultan. So this is heavy IO.
Yeah, it works for me too and it's good to know that you are doing
additional tests. I'm doing only boot tests... I was testing lots of kernels and doing additional tests would take a lot of time.
Now, what does this mean? Is VirtualBox issuing interrupts when it
shouldn't or should this section of the code be better protected?
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 35 lines · * Luiz Fernando N. Capitulino ([email protected]) wrote:
Em Tue, 26 Aug 2008 22:34:49 +0200
Gerhard Brauer <[email protected]> escreveu:| On Tue, Aug 26, 2008 at 02:15:58PM -0400, Mathieu Desnoyers wrote: | > | > Ok, it might still be caused by paravirt and alternatives instruction | > patching. What if you also do : | > | > alternative_instructions() | > | > + unsigned long flags; | > /* The patching is not fully atomic, so try to avoid local interruptions | > that might execute the to be patched code. | > Other CPUs are not running. */ | > stop_nmi(); | > #ifdef CONFIG_X86_MCE | > stop_mce(); | > #endif | > + local_irq_save(flags); | > | > | > ... | > + local_irq_restore(flags); | > restart_nmi(); | > #ifdef CONFIG_X86_MCE | > restart_mce(); | > #endif | > | > ? | | Hej! This last changes (in addition to the others you mentioned) seems | to be a good shot. I could reboot 8 times the guest, compile several | packages (something which always leeds to the oops) and currently i | build two big packages simultan. So this is heavy IO.
Quoted 7 lines
Yeah, it works for me too and it's good to know that you are doing
additional tests. I'm doing only boot tests... I was testing lots of kernels and doing additional tests would take a lot of time.Now, what does this mean? Is VirtualBox issuing interrupts when it
shouldn't or should this section of the code be better protected?
Since this problem appears while we are using a simple memcpy (the text_poke_early version), but disappears when we disable interrupts for a longer period of this, I suspect a problem with irq disabling in Virtualbox.
We could try to add some nsleep() or msleep() calls within text_poke and text_poke_early before and after the code modificatoin to see if the problem disappears. If it does, then that would somewhat confirm the racy irq disable thesis.
Mathieu
Quoted 2 lines
Signature · 1 line
Luiz Fernando N. Capitulino
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Em Wed, 27 Aug 2008 19:33:28 -0400
Mathieu Desnoyers <[email protected]> escreveu:
| * Luiz Fernando N. Capitulino ([email protected]) wrote: | > Em Tue, 26 Aug 2008 22:34:49 +0200 | > Gerhard Brauer <[email protected]> escreveu: | > | > | On Tue, Aug 26, 2008 at 02:15:58PM -0400, Mathieu Desnoyers wrote: | > | > | > | > Ok, it might still be caused by paravirt and alternatives instruction | > | > patching. What if you also do : | > | > | > | > alternative_instructions() | > | > | > | > + unsigned long flags; | > | > /* The patching is not fully atomic, so try to avoid local interruptions | > | > that might execute the to be patched code. | > | > Other CPUs are not running. */ | > | > stop_nmi(); | > | > #ifdef CONFIG_X86_MCE | > | > stop_mce(); | > | > #endif | > | > + local_irq_save(flags); | > | > | > | > | > | > ... | > | > + local_irq_restore(flags); | > | > restart_nmi(); | > | > #ifdef CONFIG_X86_MCE | > | > restart_mce(); | > | > #endif | > | > | > | > ? | > | | > | Hej! This last changes (in addition to the others you mentioned) seems | > | to be a good shot. I could reboot 8 times the guest, compile several | > | packages (something which always leeds to the oops) and currently i | > | build two big packages simultan. So this is heavy IO. | > | > Yeah, it works for me too and it's good to know that you are doing | > additional tests. I'm doing only boot tests... I was testing lots of | > kernels and doing additional tests would take a lot of time. | > | > Now, what does this mean? Is VirtualBox issuing interrupts when it | > shouldn't or should this section of the code be better protected? | > | | Since this problem appears while we are using a simple memcpy (the | text_poke_early version), but disappears when we disable interrupts for | a longer period of this, I suspect a problem with irq disabling in | Virtualbox. | | We could try to add some nsleep() or msleep() calls within text_poke and | text_poke_early before and after the code modificatoin to see if the | problem disappears. If it does, then that would somewhat confirm the | racy irq disable thesis.
Well, a Ubuntu kernel guy has reported in the virtualbox's ticket[1]
that the oops doesn't happen if he puts a printk() in the crash site.
The funny thing is that someone (who might be a virtualbox developer)
used the same race argument to say that this is a bug in the kernel.
What concerns me though is that how can virtualbox be worth using
in the Linux community if it's probably not working for various distros (currently Fedora, Ubuntu, Mandriva and ArchLinux).
Thanks for the effort, guys.
[1] http://www.virtualbox.org/ticket/1875
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 19 lines · Luiz Fernando N. Capitulino
Em Wed, 27 Aug 2008 19:33:28 -0400
Mathieu Desnoyers <[email protected]> escreveu:| | Since this problem appears while we are using a simple memcpy (the | text_poke_early version), but disappears when we disable interrupts for | a longer period of this, I suspect a problem with irq disabling in | Virtualbox. | | We could try to add some nsleep() or msleep() calls within text_poke and | text_poke_early before and after the code modificatoin to see if the | problem disappears. If it does, then that would somewhat confirm the | racy irq disable thesis.Well, a Ubuntu kernel guy has reported in the virtualbox's ticket[1]
that the oops doesn't happen if he puts aprintk()in the crash site.The funny thing is that someone (who might be a virtualbox developer)
used the same race argument to say that this is a bug in the kernel.
Quoted 7 lines
What concerns me though is that how can virtualbox be worth using
in the Linux community if it's probably not working for various distros (currently Fedora, Ubuntu, Mandriva and ArchLinux).Thanks for the effort, guys.
Ok, some news from archlinux side:
Our distribution kernel was upgraded from 2.6.26.2 to 2.6.26.3. With this upgrade to patchlevel .3 the "early oops"(freeing smp...) has gone. My virtual machines boots always fine with this, and i have one confirmation from a user about this.
Kernel upgrade does not solve the kernel panic during work with the VM, when there is heavy disk IO. I test and could reproduce this by untar 2 big files in seperate dirs: bsdtar -x -f VirtualBox-1.6.2-OSE.tar.bz2. Doing this simultan crashed the VM always.
SreenShot:
http://users.archlinux.de/~gerbra/tmp/2008-08-31-110449_724x456_scrot.png
This heavy IO oops does not occur under 2.6.26.2 when using the "3-changes-patch" against alternatives.c, which we have tested in the other mails. There must be something irq related which fix this 3-changes-patch, and what was not fixed in 2.6.26.3
On the other hand: I never have stressed a VM like this before researching for this problem. So it could also be that the heavy-IO problem way a total seperate problem from that we're talking about here. Doing my "normal" work now in VM (it's my devel VM for compiling and testing), until now i don't have had this IO oops.
We use a mostly unpatched kernel as distribution kernel.
So short summary from my side:
a) With "3-changes-patch" i got a rock solide VM
b) 2.6.26.2 have the early oops on boot and IO oops when sometimes bootet.
c) 2.6.26.3 have only the heavy-IO oops
I'll try a fresh VM, where i will test:
a) Using sata controller emulation as bus (now i have ide(piix3))
b) Using different filesystems (With 2.6.26.2 early oops and heavy-io oops could be reproduced with any filesystem).
Regards
Gerhard
Hi
On Sonntag, 31. August 2008, Gerhard Brauer wrote:
[...]
Quoted 5 lines
Ok, some news from archlinux side:
Our distribution kernel was upgraded from 2.6.26.2 to 2.6.26.3. With this upgrade to patchlevel .3 the "early oops"(freeing smp...) has gone. My virtual machines boots always fine with this, and i have one confirmation from a user about this.
Sorry, I can't confirm this here on Debian unstable (with virtualbox-ose 1.6.2 or 1.6.4), are you sure that other configuration options didn't change between the different kernel versions? Preemption and paravirt can influence the probability of the early boot panic seriously, without really avoiding it alltogether.
Actually I still get the same issues with implanting ftp://ftp5.gwdg.de/pub/linux/archlinux/core/os/i686/kernel26-2.6.26.3-1-i686.pkg.tar.gz into the test vm using virtualbox-ose 1.6.4.
Quoted 25 lines
Kernel upgrade does not solve the kernel panic during work with the VM, when there is heavy disk IO. I test and could reproduce this by untar 2 big files in seperate dirs: bsdtar -x -f VirtualBox-1.6.2-OSE.tar.bz2. Doing this simultan crashed the VM always.
SreenShot:
http://users.archlinux.de/~gerbra/tmp/2008-08-31-110449_724x456_scrot.pngThis heavy IO oops does not occur under 2.6.26.2 when using the "3-changes-patch" against
alternatives.c, which we have tested in the other mails. There must be something irq related which fix this 3-changes-patch, and what was not fixed in 2.6.26.3
On the other hand: I never have stressed a VM like this before researching for this problem. So it could also be that the heavy-IO problem way a total seperate problem from that we're talking about here. Doing my "normal" work now in VM (it's my devel VM for compiling and testing), until now i don't have had this IO oops.We use a mostly unpatched kernel as distribution kernel.
So short summary from my side:
a) With "3-changes-patch" i got a rock solide VM
b) 2.6.26.2 have the early oops on boot and IO oops when sometimes bootet.
c) 2.6.26.3 have only the heavy-IO oops
Quoted 8 lines
I'll try a fresh VM, where i will test:
a) Using sata controller emulation as bus (now i have ide(piix3))
b) Using different filesystems (With 2.6.26.2 early oops and heavy-io oops could be reproduced with any filesystem).Regards
Gerhard
Regards
Stefan Lippers-Hollmann
Quoted 10 lines · Stefan Lippers-Hollmann
Hi
On Sonntag, 31. August 2008, Gerhard Brauer wrote:
[...]Quoted 5 lines
Ok, some news from archlinux side:
Our distribution kernel was upgraded from 2.6.26.2 to 2.6.26.3. With this upgrade to patchlevel .3 the "early oops"(freeing smp...) has gone. My virtual machines boots always fine with this, and i have one confirmation from a user about this.
Quoted 5 lines
Sorry, I can't confirm this here on Debian unstable (with virtualbox-ose 1.6.2 or 1.6.4), are you sure that other configuration options didn't change between the different kernel versions? Preemption and paravirt can influence the probability of the early boot panic seriously, without really avoiding it alltogether.
Only changes between our 2.6.26.2-1 and 2.6.26.3-1 are some minor framebuffer changes in config. If i have a look at the different patchsets between the two versions i don't see something which could be the reason between work and not work.
Quoted 3 lines
Actually I still get the same issues with implanting ftp://ftp5.gwdg.de/pub/linux/archlinux/core/os/i686/kernel26-2.6.26.3-1-i686.pkg.tar.gz into the test vm using virtualbox-ose 1.6.4.
Hmm, one user also reports that he have no problem when using a vanilla 2.6.26 as guest kernel. But there must be some reasons when different distributions notice a major problem between 2.6.25 and 2.6.26 with their stock kernels. Although i don't even know if our few reports here are very representavive...
Quoted 2 lines
Regards
Stefan Lippers-Hollmann
Gerhard
Signature · 1 line
Heute ist das Morgen wovor du gestern Angst hattest...
Em Sun, 31 Aug 2008 11:29:23 +0200
Gerhard Brauer <[email protected]> escreveu:
| On Thu, Aug 28, 2008 at 10:30:13AM -0300, Luiz Fernando N. Capitulino wrote: | > Em Wed, 27 Aug 2008 19:33:28 -0400 | > Mathieu Desnoyers <[email protected]> escreveu: | > | | > | Since this problem appears while we are using a simple memcpy (the | > | text_poke_early version), but disappears when we disable interrupts for | > | a longer period of this, I suspect a problem with irq disabling in | > | Virtualbox. | > | | > | We could try to add some nsleep() or msleep() calls within text_poke and | > | text_poke_early before and after the code modificatoin to see if the | > | problem disappears. If it does, then that would somewhat confirm the | > | racy irq disable thesis. | > | > Well, a Ubuntu kernel guy has reported in the virtualbox's ticket[1] | > that the oops doesn't happen if he puts a printk() in the crash site. | > | > The funny thing is that someone (who might be a virtualbox developer) | > used the same race argument to say that this is a bug in the kernel. | > | > What concerns me though is that how can virtualbox be worth using | > in the Linux community if it's probably not working for various distros | > (currently Fedora, Ubuntu, Mandriva and ArchLinux). | > | > Thanks for the effort, guys. | > | > [1] http://www.virtualbox.org/ticket/1875 | | Ok, some news from archlinux side: | Our distribution kernel was upgraded from 2.6.26.2 to 2.6.26.3. With | this upgrade to patchlevel .3 the "early oops"(freeing smp...) has gone. | My virtual machines boots always fine with this, and i have one | confirmation from a user about this. | | Kernel upgrade does not solve the kernel panic during work with the VM, | when there is heavy disk IO. I test and could reproduce this by untar 2 | big files in seperate dirs: bsdtar -x -f VirtualBox-1.6.2-OSE.tar.bz2. | Doing this simultan crashed the VM always. | SreenShot: | http://users.archlinux.de/~gerbra/tmp/2008-08-31-110449_724x456_scrot.png | | This heavy IO oops does not occur under 2.6.26.2 when using the | "3-changes-patch" against alternatives.c, which we have tested in the | other mails. There must be something irq related which fix this | 3-changes-patch, and what was not fixed in 2.6.26.3 | On the other hand: I never have stressed a VM like this before | researching for this problem. So it could also be that the heavy-IO | problem way a total seperate problem from that we're talking about here. | Doing my "normal" work now in VM (it's my devel VM for compiling and | testing), until now i don't have had this IO oops.
Mandriva kernel was 2.6.26.3 based at the time I started testing
this and all my last tests have been done on 2.6.27-rc4. I think it's very unusual to have a change in a -stable kernel not present in the latest -rc.
Also note that CPU settings in the VM has a big influency in the
problem, so I'm pretty sure 2.6.26.3 doesn't fix the problem.
Signature · 1 line
Luiz Fernando N. Capitulino
Am Sonntag, den 31.08.2008, 11:09 -0300 schrieb Luiz Fernando N. Capitulino:
Quoted 4 lines
Mandriva kernel was 2.6.26.3 based at the time I started testing
this and all my last tests have been done on 2.6.27-rc4. I think it's very unusual to have a change in a -stable kernel not present in the latest -rc.
I was away the last days, but i notice that the virtualbox update to 2.0.2 solve all problems i mentioned. We also have same responses from other arch users.
I never saw the "early boot oops" (this have had gone still with our kernel update, mystical....). But also the "heavy IO oops" has gone. So it was a virtualbox problem, they fixed it in:
http://www.virtualbox.org/ticket/1875
So from my side this is solved.
Gerhard
Quoted 16 lines · Gerhard Brauer
Am Sonntag, den 31.08.2008, 11:09 -0300 schrieb Luiz Fernando N. Capitulino:
Quoted 4 lines
Mandriva kernel was 2.6.26.3 based at the time I started testing
this and all my last tests have been done on 2.6.27-rc4. I think it's very unusual to have a change in a -stable kernel not present in the latest -rc.I was away the last days, but i notice that the virtualbox update to 2.0.2 solve all problems i mentioned. We also have same responses from other arch users.
I never saw the "early boot oops" (this have had gone still with our kernel update, mystical....). But also the "heavy IO oops" has gone. So it was a virtualbox problem, they fixed it in:
http://www.virtualbox.org/ticket/1875
Quoted 1 line
So from my side this is solved.
great - the VirtualBox recompiler didnt notice the paravirt code modification sequence. Any Linux kernel side NOP issue (which caused that early oops) should be solved in 2.6.27-rc7 as well.
so the combo of 2.0.2 and later VirtualBox plus v2.6.27-rc7 and later should have no known bugs.
Ingo
Em Sun, 21 Sep 2008 15:41:39 +0200
Gerhard Brauer <[email protected]> escreveu:
| Am Sonntag, den 31.08.2008, 11:09 -0300 schrieb Luiz Fernando N. | Capitulino: | | > Mandriva kernel was 2.6.26.3 based at the time I started testing | > this and all my last tests have been done on 2.6.27-rc4. I think it's | > very unusual to have a change in a -stable kernel not present in the | > latest -rc. | | I was away the last days, but i notice that the virtualbox update to | 2.0.2 solve all problems i mentioned. We also have same responses from | other arch users. | I never saw the "early boot oops" (this have had gone still with our | kernel update, mystical....). But also the "heavy IO oops" has gone. So | it was a virtualbox problem, they fixed it in: | http://www.virtualbox.org/ticket/1875 | | So from my side this is solved.
Yeah, we have ran some tests here as well and it is solved.
Thanks a lot for the people involved in debugging this problem.
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 5 lines · Mathieu Desnoyers
We could try to add some
nsleep()ormsleep()calls withintext_pokeandtext_poke_earlybefore and after the code modificatoin to see if the problem disappears. If it does, then that would somewhat confirm the racy irq disable thesis.
nsleep isn't known here as a function, only references i found is maybe in posix-timers.c.
msleep() is known, but each time i add for ex.
msleep(100);
in any place in text_poke and/or text_poke_early it get a kernel panic on boot. Here's a screenie:
http://users.archlinux.de/~gerbra/tmp/2008-08-28-132337_724x456_scrot.png
I also tried to work with the isolated changes we have last made, but it seems that only the 3 changes together work.
Also i tried to went back to older versions of alternatives.c referenced in:
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.26.y.git;a=history;f=arch/x86/kernel/alternative.c;h=65c7857a90ddfc6ff084c6817baba045ced0ad71;hb=v2.6.26
But with my few knowledges i ran in too many errors.
So, have you any further ideas, code that i/we could test? Or - i'm naive - are the "3 changes" we made ready to go in the kernel without to harm something real important than virtualbox?
Quoted 1 line
Mathieu
Gerhard
Quoted 8 lines · Mathieu Desnoyers
Quoted 5 lines · * Gerhard Brauer ([email protected]) wrote:
Here is also an archive with guest dmesg and messages.log from such an oops when heavy disk io leads to the oops:
http://bugs.archlinux.org/task/11141?getfile=2445Hrm, can you try this ?
Sorry for the delay but i need to build a complete distribution kernel and my machine is not the fastest.
My host:
archlinux 2.6.26 P4 2Ghz
VirtualBox: Sun xVM 1.6.4
gcc 4.4.1-3
My guest:
archlinux 2.6.26
My "tests":
I could sometimes boot the guest with the "tricks" (VT-x enabled, acpi off,...). But i always get an oops if i compile something bigger on this guest (ex. virtualbox-modules where the tarball must be untarrt with bsdtar -> disk io)
If this happens the next reboot leads always to the early oops (Freeing smp....). Each reboot do this. Then i close virtualbox application, unload/reload vboxdrv from host and start vbox again. Then i could mostimes boot the guest again. But next heavy disk IO leads again to the oops.
If i could boot without oops, and reboot or halt the guest, then the next boots are clean.
Quoted 1 line
1 - Make sure you kernel is not
CONFIG_DEBUG_RODATA
Not set.
Quoted 2 lines
2 - Change the whole
text_pokeimplementation inarch/x86/kernel/alternative.cto this :
With this changes i also get the oops, in all above mentioned tests.
Quoted 8 lines
Then, after having tested (2), try this on top of it :
In
arch/x86/kernel/alternative.c,alternatives_smp_switch()Add unsigned long flags;
Changespin_lock->spin_lock_irqsave(&smp_alt, flags);spin_unlock(&smp_alt); ->spin_unlock_irqrestore(&smp_alt, flags);
With our distribution kernel i could change these spin_lock/unlock in alternatives.c. Fist thought was that there was a slightly better behavior (first boot goes on, i could compile something, but next package i build thee opps (heavy io opps) comes again. And then also after reboot the early oops (freeing smp...)
Here is a screenie from oops when building something:
http://users.archlinux.de/~gerbra/tmp/2008-08-26-210724_724x456_scrot.png
Sometimes (could not be reproduced) the virtualbox app also traps with an error dialog (Guru message), which offers a log from the VM and a scren shot. Maybe this could be helpfull. Log and screenie could be found here:
http://users.archlinux.de/~gerbra/tmp/vbox-guru/
Quoted 9 lines
This will help testing if there is a problem with interrupts coming shortly after the modification. If it fixes the problem, my guess is that we should flush the instruction cache (and maybe the data cache ?) in
text_pokeandtext_pokeearly when interrupts are off.From my side i would say: both changes would not solve the oops.
Mathieu
Regards
Gerhard
Em Tue, 26 Aug 2008 16:18:51 +0200
Gerhard Brauer <[email protected]> escreveu:
| On Fri, Aug 22, 2008 at 02:08:13PM -0700, H. Peter Anvin wrote: | > Luiz Fernando N. Capitulino wrote: | >> | >> I have asked Mandriva and Ubuntu users to test this and all of | >> them so far are saying that noreplace-paravirt works. | >> | >> It makes the system slower, but it works. | >> | > | > Yes, the big issue is exactly what VirtualBox screws up in this matter, | > how to detect it, and how to work around it. | > | > It's pretty clear it's a VirtualBox f*ckup at this point, but the failure | > mechanism isn't at all obvious and so far the workaround is elusive. | > | > I'm strongly suspect this is a VirtualBox tcache management failure, but | > that doesn't help the situation without knowing how it happens. | | On Archlinux we have the same problem. We have a bugreport here: | http://bugs.archlinux.org/task/11141 | | Myself test it with a LiveCD/Install-ISO which has 2.6.26 as install | kernel. We have the guest oops on virtualbox-ose, virtualbox-sun and both on | i686 or x86_64 hosts. | | Some things i noticed: | - The system boots always when i either enable VT-x in guest settings or | disable acpi and run the guest with acpi=off.
Yes, lots of ubuntu users have reported the same but another "lots"
of them have reported that the trick didn't work.
Thanks for joining!
Signature · 1 line
Luiz Fernando N. Capitulino
Quoted 9 lines · Luiz Fernando N. Capitulino
Em Tue, 26 Aug 2008 16:18:51 +0200
Gerhard Brauer <[email protected]> escreveu:| Some things i noticed: | - The system boots always when i either enable VT-x in guest settings or | disable acpi and run the guest with acpi=off.Yes, lots of ubuntu users have reported the same but another "lots"
of them have reported that the trick didn't work.
I must relativate above: i have two test enviroments, one is our LiveCD/Install-ISO with 2.6.26 which we made special for a linux conference last weekend (our official iso comes still with 2.6.25). With this iso the "trick" with VT-x or noacpi works.
But on an installed archlinux (with distribution kernel 2.6.26) this does'nt work. Sometimes it works when i restart the virtualbox application, but mostly not. So on this installed guest system the only working solution seems to add noreplace-paravirt as kernel parameter. But this makes the system terrible slow (mostly on udev things).
I try Mathieu's hints currently by building a new distribution kernel with the changes.
But i think the biggest problem to maybe solve this from the sight of kernel devs is that we all have different "test" enviroments (vbox versions, architectures, distribution kernels,...) where the oops (i think) not appears for all on the same place.
On the other hand, the more we we try such patches in different enviroments there is a better chance to get a real fix - from kernel dev side or from virtualbox/Sun side...
Quoted 2 lines
Thanks for joining!
Luiz Fernando N. Capitulino
Gerhard
Signature · 1 line
www,archlinux.de
H. Peter Anvin
reviewerWas looking at the code stream, and noticed this:
Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8 Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e8 7f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8
The EIP is in the *MIDDLE* of a NOPL instruction:
C012FC46 C00F84 ror byte [edi],0x84
C012FC49 0B01 or eax,[ecx]
C012FC4B 0000 add [eax],al
C012FC4D B8D0BF41C0 mov eax,0xc041bfd0
C012FC52 C7056CC041C0FFFF mov dword [dword 0xc041c06c],0xffffffff
-FFFF
C012FC5C E87F822100 call dword 0xc0347ee0
C012FC61 E81A030200 call dword 0xc014ff80
C012FC66 8B45B0 mov eax,[ebp-0x50]
C012FC69 50 push eax
C012FC6A 9D popfd
C012FC6B 0F1F840000000000 nop dword [eax+eax+0x0]
C012FC73 8B45BC mov eax,[ebp-0x44]
C012FC76 83C460 add esp,byte +0x60
C012FC79 5B pop ebx
C012FC7A 5E pop esi
C012FC7B 5F pop edi
C012FC7C 5D pop ebp
C012FC7D C3 ret
C012FC7E 6690 xchg ax,ax
C012FC80 A16CC041C0 mov eax,[0xc041c06c]There are two possibilities: VirtualBox mis-executes (not merely traps, which is what tip:master looks for) the NOPL instruction, or something is jumping into the middle of the sequence that is then replaced by the NOPL.
So, Luiz: the DEBUG_INFO version of vmlinux would be helpful. It would also help to know the exact version of VirtualBox you're running, what source you got it from, and what your host system looks like.
-hpa
Quoted 38 lines · * H. Peter Anvin ([email protected]) wrote:
Was looking at the code stream, and noticed this:
Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e87f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8
Code: c0 0f 84 0b 01 00 00 b8 d0 bf 41 c0 c7 05 6c c0 41 c0 ff ff ff ff e87f 82 21 00 e8 1a 03 02 00 8b 45 b0 50 9d 0f 1f 84 00 00 00 <00> 00 8b 45 bc 83 c4 60 5b 5e 5f 5d c3 66 90 a1 6c c0 41 c0 e8
The EIP is in the *MIDDLE* of a NOPL instruction:
C012FC46 C00F84 ror byte [edi],0x84 C012FC49 0B01 or eax,[ecx] C012FC4B 0000 add [eax],al C012FC4D B8D0BF41C0 mov eax,0xc041bfd0 C012FC52 C7056CC041C0FFFF mov dword [dword 0xc041c06c],0xffffffff -FFFF C012FC5C E87F822100 call dword 0xc0347ee0 C012FC61 E81A030200 call dword 0xc014ff80 C012FC66 8B45B0 mov eax,[ebp-0x50] C012FC69 50 push eax C012FC6A 9D popfd C012FC6B 0F1F840000000000 nop dword [eax+eax+0x0] C012FC73 8B45BC mov eax,[ebp-0x44] C012FC76 83C460 add esp,byte +0x60 C012FC79 5B pop ebx C012FC7A 5E pop esi C012FC7B 5F pop edi C012FC7C 5D pop ebp C012FC7D C3 ret C012FC7E 6690 xchg ax,ax C012FC80 A16CC041C0 mov eax,[0xc041c06c]There are two possibilities: VirtualBox mis-executes (not merely traps, which is what tip:master looks for) the NOPL instruction, or something is jumping into the middle of the sequence that is then replaced by the NOPL.
Quoted 5 lines
So, Luiz: the
DEBUG_INFOversion of vmlinux would be helpful. It would also help to know the exact version of VirtualBox you're running, what source you got it from, and what your host system looks like.-hpa
The patch which turns on this bug this this important change to the apply paravirt : it disables interrupts _near_ the code patching, _within_ the loop. Before, interrupts were disabled outside of the loop. It needs to disable interrupts within the loop to be able to use vmap in text_poke().
So I bet VirtualBox has a race in the way it handles interrupt disabling.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 10 lines · Mathieu Desnoyers
The patch which turns on this bug this this important change to the apply paravirt : it disables interrupts _near_ the code patching, _within_ the loop. Before, interrupts were disabled outside of the loop. It needs to disable interrupts within the loop to be able to use vmap in
text_poke().So I bet VirtualBox has a race in the way it handles interrupt disabling.
That seems a bit far-fetched. The fault is in an initcall, and there are no interrupts involved. Perhaps VirtualBox doesn't manage its tcache correctly, but I don't see this as being interrupt-related.
-hpa
Em Fri, 22 Aug 2008 10:16:07 -0700
"H. Peter Anvin" <[email protected]> escreveu:
| So, Luiz: the DEBUG_INFO version of vmlinux would be helpful. It would | also help to know the exact version of VirtualBox you're running, what | source you got it from, and what your host system looks like.
You will find vmlinux with DEBUG_INFO enabled at:
http://users.mandriva.com.br/~lcapitulino/virtualbox-oops/
I'm running Mandriva's VirtualBox 1.6.4 OSE, my host kernel is 2.6.26-3mnb
(patched).
I could try with upstream's VirtualBox just to be sure it's not
something else, but I don't think it is since there are reports for ArchLinux and Ubuntu as well:
https://bugs.launchpad.net/ubuntu/intrepid/+source/linux/+bug/246067
Signature · 1 line
Luiz Fernando N. Capitulino
H. Peter Anvin
reviewerQuoted 9 lines · Luiz Fernando N. Capitulino
Em Fri, 22 Aug 2008 10:16:07 -0700
"H. Peter Anvin" <[email protected]> escreveu:| So, Luiz: the
DEBUG_INFOversion of vmlinux would be helpful. It would | also help to know the exact version of VirtualBox you're running, what | source you got it from, and what your host system looks like.You will find vmlinux with
DEBUG_INFOenabled at:
Quoted 9 lines
I'm running Mandriva's VirtualBox 1.6.4 OSE, my host kernel is 2.6.26-3mnb
(patched).I could try with upstream's VirtualBox just to be sure it's not
something else, but I don't think it is since there are reports for ArchLinux and Ubuntu as well:https://bugs.launchpad.net/ubuntu/intrepid/+source/linux/+bug/246067
Not necessary, but I wanted to get the information so I can try to reproduce locally.
-hpa
H. Peter Anvin
reviewerQuoted 4 lines · Luiz Fernando N. Capitulino
I'm running Mandriva's VirtualBox 1.6.4 OSE, my host kernel is 2.6.26-3mnb
(patched).
What is your host *system* like -- CPU especially, and is your host kernel 32 or 64 bits?
-hpa
Em Fri, 22 Aug 2008 12:18:21 -0700
"H. Peter Anvin" <[email protected]> escreveu:
| Luiz Fernando N. Capitulino wrote: | > | > I'm running Mandriva's VirtualBox 1.6.4 OSE, my host kernel is 2.6.26-3mnb | > (patched). | > | | What is your host *system* like -- CPU especially, and is your host | kernel 32 or 64 bits?
32 bits, /proc/cpuinfo output:
"""
processor : 0vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
stepping : 1
cpu MHz : 2410.462
cache size : 1024 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe constant_tsc pebs bts pni monitor ds_cpl cid xtpr
bogomips : 4825.33
clflush size : 64
power management:
"""
I have 1G of RAM and a VIA mobo.
Signature · 1 line
Luiz Fernando N. Capitulino
Em Thu, 21 Aug 2008 14:34:07 -0700
"H. Peter Anvin" <[email protected]> escreveu:
| > | > Does this look like a kernel bug? | > | | No, it looks like a very common virtualizer bug. Does the attached | patch work for you?
Unfortunately it does not.
Signature · 1 line
Luiz Fernando N. Capitulino