Skip to contentSkip to conversation

2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

DISCUSSION Christopher S. Aker 3 participants 21 messages Jul 3, 2008 – Dec 31, 2008 virtio core

lore
Share & tools
lore
#0 lore raw

Xen: 3.2.1-rc5 64bit
Dom0: 2.6.18.8 (at cs 524) 32-pae
DomU: 2.6.26-rc8 32-pae

root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
Using config file "/linodes/xencaker/xen.conf".
Started domain xencaker
                        root@newark13:~#
root@newark13:~# xm dmesg
...
(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 
0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e019:[<00000000c0103b55>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
(XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 00000000deadbeef
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000c06c1000
(XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000002d351b000   cr2: 0000000000000000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c0667f08:
(XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
c065c580
(XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
00000000
(XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
00000000
(XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
1f898975
(XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
00000007
(XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
00000000
(XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
00040800
(XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000

-Chris

#1 lore raw
Quoted 7 lines · Christopher S. Aker

Xen: 3.2.1-rc5 64bit
Dom0: 2.6.18.8 (at cs 524) 32-pae
DomU: 2.6.26-rc8 32-pae

root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
Using config file "/linodes/xencaker/xen.conf".
Started domain xencaker

Is this consistent or sporadic? Does it depend on how much guest memory you give it? What was the last kernel version which worked?

Quoted 4 lines
                       root@newark13:~#
root@newark13:~# xm dmesg
...

Were there any other errors/warnings printed here?

Quoted 6 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

Quoted 19 lines

(XEN) RFLAGS: 0000000000000282 CONTEXT: guest

(XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 00000000deadbeef
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000c06c1000
(XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000002d351b000   cr2: 0000000000000000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c0667f08:
(XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 c065c580
(XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 00000000
(XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 00000000
(XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 1f898975
(XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 00000007
(XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 00000000
(XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 00040800
(XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000
Quoted 1 line

-Chris

Thanks,
J

#20 lore raw
In reply to Christopher S. Aker
Quoted 7 lines · Christopher S. Aker

Xen: 3.2.1-rc5 64bit
Dom0: 2.6.18.8 (at cs 524) 32-pae
DomU: 2.6.26-rc8 32-pae

root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
Using config file "/linodes/xencaker/xen.conf".
Started domain xencaker

Is this consistent or sporadic? Does it depend on how much guest memory you give it? What was the last kernel version which worked?

Quoted 4 lines
                       root@newark13:~#
root@newark13:~# xm dmesg
...

Were there any other errors/warnings printed here?

Quoted 7 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

Quoted 32 lines

(XEN) RFLAGS: 0000000000000282 CONTEXT: guest
(XEN) rax: 00000000ffffffea rbx: 00000000deadbeef rcx:
00000000deadbeef
(XEN) rdx: 00000000deadbeef rsi: 00000000deadbeef rdi:
00000000c06c1000
(XEN) rbp: 00000000c06c0000 rsp: 00000000c0667f08 r8:
0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11:
0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14:
0000000000000000
(XEN) r15: 0000000000000000 cr0: 000000008005003b cr4:
00000000000026b0
(XEN) cr3: 00000002d351b000 cr2: 0000000000000000
(XEN) ds: e021 es: e021 fs: e021 gs: e021 ss: e021 cs: e019
(XEN) Guest stack trace from esp=c0667f08:
(XEN) c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000
c065c580
(XEN) c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000
00000000
(XEN) 00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452
00000000
(XEN) 00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac
1f898975
(XEN) 40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000
00000007
(XEN) c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2
00000000
(XEN) c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201
00040800
(XEN) 00010676 f5800000 00000000 c0752000 00000000 00000000

Quoted 1 line

-Chris

Thanks,
J

#2 lore raw
In reply to Jeremy Fitzhardinge
Quoted 1 line · Jeremy Fitzhardinge

Is this consistent or sporadic?

Consistently fails to boot, across a few test boxes, slightly different Xen versions -- all around Xen 3.2.1, same config.

Quoted 1 line

Does it depend on how much guest memory you give it?

No. Tried with 128, 360, 5000. They all fail in the same way.

Quoted 1 line

What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Quoted 8 lines
Quoted 6 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks,
-Chris

#19 lore raw
In reply to Jeremy Fitzhardinge
Quoted 1 line · Jeremy Fitzhardinge

Is this consistent or sporadic?

Consistently fails to boot, across a few test boxes, slightly different Xen versions -- all around Xen 3.2.1, same config.

Quoted 1 line

Does it depend on how much guest memory you give it?

No. Tried with 128, 360, 5000. They all fail in the same way.

Quoted 1 line

What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Quoted 9 lines
Quoted 7 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks,
-Chris

#3 lore raw
In reply to Christopher S. Aker
Quoted 9 lines · Christopher S. Aker
Quoted 1 line · Jeremy Fitzhardinge

Is this consistent or sporadic?

Consistently fails to boot, across a few test boxes, slightly different Xen versions -- all around Xen 3.2.1, same config.

Quoted 1 line

Does it depend on how much guest memory you give it?

No. Tried with 128, 360, 5000. They all fail in the same way.

Quoted 7 lines
Quoted 1 line

What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Quoted 5 lines
Quoted 1 line

Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks, that's very interesting. I had a very similar crash in a current -rc kernel when running it under the Citrix XenServer product version of Xen, and I wasn't sure where to point fingers.

For reference:

$ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55						<- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:	ud2a   (gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:	push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:	mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:	push   %edi
...

Thanks,
J

#18 lore raw
In reply to Christopher S. Aker
Quoted 10 lines · Christopher S. Aker
Quoted 1 line · Jeremy Fitzhardinge

Is this consistent or sporadic?

Consistently fails to boot, across a few test boxes, slightly different Xen versions -- all around Xen 3.2.1, same config.

Quoted 1 line

Does it depend on how much guest memory you give it?

No. Tried with 128, 360, 5000. They all fail in the same way.

Quoted 7 lines
Quoted 1 line

What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e019:[<00000000c0103b55>]

Quoted 6 lines
Quoted 1 line

Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks, that's very interesting. I had a very similar crash in a current -rc kernel when running it under the Citrix XenServer product version of Xen, and I wasn't sure where to point fingers.

For reference:

$ gdb 2.6.26-rc8-linode11 
GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55						<- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:	ud2a   
(gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:	push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:	mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:	push   %edi
...

Thanks,
J

#4 lore raw
In reply to Jeremy Fitzhardinge
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 3 lines
Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create run. Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Quoted 16 lines

For reference:

$ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55                        <- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:    ud2a   (gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:    push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:    mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:    push   %edi

Cool, thanks :)

-Chris

#17 lore raw
In reply to Jeremy Fitzhardinge
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 3 lines
Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create run. Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Quoted 11 lines

For reference:

$ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
Quoted 8 lines
(gdb) x/i 0x00000000c0103b55                        <- note: need to add 
0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:    ud2a   (gdb) disassemble 
make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:    push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:    mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:    push   %edi

Cool, thanks :)

-Chris

#5 lore raw
In reply to Christopher S. Aker
Quoted 7 lines · Christopher S. Aker
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 3 lines
Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Quoted 7 lines

Unfortunately, I included the entire dmesg output from that xm create run. Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

J

#16 lore raw
In reply to Christopher S. Aker
Quoted 13 lines · Christopher S. Aker
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 3 lines
Quoted 1 line

Were there any other errors/warnings printed here?

Nope. Just some whitespace.

Are you sure? Could you send a full xm dmesg log output? The crash is from a BUG() caused by a failing hypercall. Since you have compiled Xen with debug enabled, it should have printed something about why it was failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create run. Here's more of the transcript in case it's useful:

Quoted 6 lines

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

J

#6 lore raw
In reply to Jeremy Fitzhardinge
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 3 lines

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

-Chris

#15 lore raw
In reply to Jeremy Fitzhardinge
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 4 lines

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

-Chris

#7 lore raw
In reply to Christopher S. Aker
Quoted 7 lines · Christopher S. Aker
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 5 lines

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Quoted 8 lines
Quoted 4 lines

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

J

#8 lore raw
In reply to Christopher S. Aker
Quoted 11 lines · Christopher S. Aker
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 4 lines

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

Good, that's what I was hoping to see. (Well, actually not that *particular* message, which is strange, but some explanatory message at least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

J

#9 lore raw
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:
(XEN) mm.c:1482:d31 Bad L1 flags 800000
(XEN) traps.c:437:d31 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 31 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-3.3.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: e033:[<ffffffff8020b3ed>]
(XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest

(XEN) rax: 00000000ffffffea   rbx: 000000000000077f   rcx: 0000000000000054
(XEN) rdx: 0000000000000000   rsi: 8000000331df5061   rdi: ffffffff8077f000
(XEN) rbp: ffffffff806e1f88   rsp: ffffffff806e1f28   r8:  0000000000000010
(XEN) r9:  0000000000000010   r10: ffffffff806e1e88   r11: ffffffff80827f60
(XEN) r12: ffffffff8077f000   r13: 8000000000000161   r14: 8000000331df5061
(XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000033c35c000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff806e1f28:
(XEN)    0000000000000054 ffffffff80827f60 ffffffff8020b3ed 000000010000e030
(XEN)    0000000000010082 ffffffff806e1f68 000000000000e02b ffffffff8020b3e9
(XEN)    0000000000000200 000000000000000c 000000000000000c 0000000000001800
(XEN)    ffffffff806e1ff8 ffffffff806f99cf 0000000000000000 0000000000001000
(XEN)    0000000000001800 ffffffff80e48000 0000000000020000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff80d45000
(XEN)    ffffffff80d46000 ffffffff80d47000 ffffffff80d48000 ffffffff80d49000
(XEN)    ffffffff80d4a000 ffffffff80d4b000 ffffffff80d4c000 ffffffff80d4d000
(XEN)    ffffffff80d4e000 ffffffff80d4f000 ffffffff80d50000 ffffffff80d51000
(XEN)    ffffffff80d52000 ffffffff80d53000 ffffffff80d54000 ffffffff80d55000
(XEN)    ffffffff80d56000 ffffffff80d57000 ffffffff80d58000 ffffffff80d59000
(XEN)    ffffffff80d5a000 ffffffff80d5b000 ffffffff80d5c000 ffffffff80d5d000
(XEN)    ffffffff80d5e000 ffffffff80d5f000 ffffffff80d60000 ffffffff80d61000
(XEN)    ffffffff80d62000 ffffffff80d63000 ffffffff80d64000 ffffffff80d65000
(XEN)    ffffffff80d66000 ffffffff80d67000 ffffffff80d68000 ffffffff80d69000
(XEN)    ffffffff80d6a000 ffffffff80d6b000 ffffffff80d6c000 ffffffff80d6d000
(XEN)    ffffffff80d6e000 ffffffff80d6f000 ffffffff80d70000 ffffffff80d71000
(XEN)    ffffffff80d72000 ffffffff80d73000 ffffffff80d74000 ffffffff80d75000
(XEN)    ffffffff80d76000 ffffffff80d77000 ffffffff80d78000 ffffffff80d79000
Signature · 4 lines
Guillaume Rousse
Service des Moyens Informatiques
INRIA Saclay - Ile de France
Tel: 01 69 35 69 62
#10 lore raw
Quoted 6 lines · Guillaume Rousse
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:

I don't know of any outstanding bugs which cause this kind of symptom. What's happening on the Linux side when the domain crashes this way?

J

Quoted 37 lines

(XEN) mm.c:1482:d31 Bad L1 flags 800000
(XEN) traps.c:437:d31 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 31 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-3.3.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: e033:[<ffffffff8020b3ed>]
(XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest

(XEN) rax: 00000000ffffffea   rbx: 000000000000077f   rcx: 0000000000000054
(XEN) rdx: 0000000000000000   rsi: 8000000331df5061   rdi: ffffffff8077f000
(XEN) rbp: ffffffff806e1f88   rsp: ffffffff806e1f28   r8:  0000000000000010
(XEN) r9:  0000000000000010   r10: ffffffff806e1e88   r11: ffffffff80827f60
(XEN) r12: ffffffff8077f000   r13: 8000000000000161   r14: 8000000331df5061
(XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000033c35c000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff806e1f28:
(XEN)    0000000000000054 ffffffff80827f60 ffffffff8020b3ed 000000010000e030
(XEN)    0000000000010082 ffffffff806e1f68 000000000000e02b ffffffff8020b3e9
(XEN)    0000000000000200 000000000000000c 000000000000000c 0000000000001800
(XEN)    ffffffff806e1ff8 ffffffff806f99cf 0000000000000000 0000000000001000
(XEN)    0000000000001800 ffffffff80e48000 0000000000020000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff80d45000
(XEN)    ffffffff80d46000 ffffffff80d47000 ffffffff80d48000 ffffffff80d49000
(XEN)    ffffffff80d4a000 ffffffff80d4b000 ffffffff80d4c000 ffffffff80d4d000
(XEN)    ffffffff80d4e000 ffffffff80d4f000 ffffffff80d50000 ffffffff80d51000
(XEN)    ffffffff80d52000 ffffffff80d53000 ffffffff80d54000 ffffffff80d55000
(XEN)    ffffffff80d56000 ffffffff80d57000 ffffffff80d58000 ffffffff80d59000
(XEN)    ffffffff80d5a000 ffffffff80d5b000 ffffffff80d5c000 ffffffff80d5d000
(XEN)    ffffffff80d5e000 ffffffff80d5f000 ffffffff80d60000 ffffffff80d61000
(XEN)    ffffffff80d62000 ffffffff80d63000 ffffffff80d64000 ffffffff80d65000
(XEN)    ffffffff80d66000 ffffffff80d67000 ffffffff80d68000 ffffffff80d69000
(XEN)    ffffffff80d6a000 ffffffff80d6b000 ffffffff80d6c000 ffffffff80d6d000
(XEN)    ffffffff80d6e000 ffffffff80d6f000 ffffffff80d70000 ffffffff80d71000
(XEN)    ffffffff80d72000 ffffffff80d73000 ffffffff80d74000 ffffffff80d75000
(XEN)    ffffffff80d76000 ffffffff80d77000 ffffffff80d78000 ffffffff80d79000
#11 lore raw
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 6 lines · Guillaume Rousse
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:

I don't know of any outstanding bugs which cause this kind of symptom. What's happening on the Linux side when the domain crashes this way?

Immediate crash, without further information.

I've opened a but report on mandriva bugzilla:
https://qa.mandriva.com/show_bug.cgi?id=46470

And I've yet to try with vanilla linus kernel.

Signature · 3 lines
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62
#12 lore raw
Quoted 6 lines · Guillaume Rousse
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 4 lines · Guillaume Rousse
Quoted 3 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:

I don't know of any outstanding bugs which cause this kind of symptom. What's happening on the Linux side when the domain crashes this way?

Immediate crash, without further information.

I've opened a but report on mandriva bugzilla:
https://qa.mandriva.com/show_bug.cgi?id=46470

Does earlyprintk=xen reveal anything more? What does the rip (ffffffff8020b3ed) map to in the kernel?

Quoted 1 line

And I've yet to try with vanilla linus kernel.

Please do. If possible, build a linus kernel with the mandriva config.

J

#13 lore raw
Quoted 10 lines · Jeremy Fitzhardinge
Quoted 9 lines · Guillaume Rousse
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 6 lines · Guillaume Rousse
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I don't know of any outstanding bugs which cause this kind of symptom. What's happening on the Linux side when the domain crashes this way?

Immediate crash, without further information.

I've opened a but report on mandriva bugzilla:
https://qa.mandriva.com/show_bug.cgi?id=46470

Does earlyprintk=xen reveal anything more? What does the rip (ffffffff8020b3ed) map to in the kernel?

I guess earlyprintk=xen is a dom0 option, because passing it to domU didn't change anything. The given adress map to set_page_prot, according to the System.map file.

BTW, it only happens with ISCSI-stored hosts, whereas it works OK with LVM-stored hosts (tough on slightly different hardware). On other hosts, the guest kernel boots OK, and then fails to mount underlying FS (but that's a different issue). I updated upstream report with details.

Signature · 4 lines
Guillaume Rousse
Service des Moyens Informatiques
INRIA Saclay - Ile de France
Tel: 01 69 35 69 62
#14 lore raw
Quoted 11 lines · Guillaume Rousse
Quoted 10 lines · Jeremy Fitzhardinge
Quoted 9 lines · Guillaume Rousse
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 6 lines · Guillaume Rousse
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 2 lines

FWIW, I just tried 2.6.26-rc1 and it booted fine. I can narrow it down further if that'd be useful to you.

Yes, please.

Was there any final conclusion there ? Because I'm currently trying a 2.6.27.5 with the same kind of issue:

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I don't know of any outstanding bugs which cause this kind of symptom. What's happening on the Linux side when the domain crashes this way?

Immediate crash, without further information.

I've opened a but report on mandriva bugzilla:
https://qa.mandriva.com/show_bug.cgi?id=46470

Does earlyprintk=xen reveal anything more? What does the rip (ffffffff8020b3ed) map to in the kernel?

I guess earlyprintk=xen is a dom0 option, because passing it to domU didn't change anything.

It will be most helpful if you're running a debug=y build of Xen, so the domU's early console output appears on the Xen console.

Quoted 1 line

The given adress map to set_page_prot, according to the System.map file.

That's interesting.

Quoted 1 line

BTW, it only happens with ISCSI-stored hosts, whereas it works OK with LVM-stored hosts (tough on slightly different hardware).

Uh, that's very mysterious. The underlying storage shouldn't have any effect on whether the kernel can boot - it's either loaded or not.

Quoted 1 line

On other hosts, the guest kernel boots OK, and then fails to mount underlying FS (but that's a different issue). I updated upstream report with details.

I'm on vacation at the moment, but I'll give this a closer look when I return.

J