kvm: guest can not startup due to triple fault

BUG Dave Young 5 participants 36 messages Oct 17, 2010 – Oct 19, 2010 KVM

lore

Share & tools

lore

Dave Young

author

2010-10-17 08:56 UTC #1 lore raw

Quoted 3 lines · Dave Young

linux guest can not startup normally, qemu keep rebooting.
host & guest both use linux-2.6.36-rc8+ kernel

BTW, qemu cmdline with -kernel will more likely trigger this bug

Quoted 20 lines

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:

[ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+ #62
[ 3924.503045] Call Trace:
[ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26 [kvm_intel]
[ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474 [kvm_intel]
[ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43 [kvm]
[ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
[ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
[ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
[ 3924.503117]  [<ffffffff810bee76>] ? fire_user_return_notifiers+0x2c/0x36
[ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
[ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
[ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
[ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
[ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
[ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Anyone can help on this?

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-17 09:34 UTC #2 lore raw

In reply to Dave Young

Quoted 21 lines · Dave Young

linux guest can not startup normally, qemu keep rebooting.

host& guest both use linux-2.6.36-rc8+ kernel

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:

[ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+ #62
[ 3924.503045] Call Trace:
[ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26 [kvm_intel]
[ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474 [kvm_intel]
[ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43 [kvm]
[ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
[ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
[ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
[ 3924.503117]  [<ffffffff810bee76>] ? fire_user_return_notifiers+0x2c/0x36
[ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
[ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
[ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
[ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
[ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
[ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Quoted 1 line

Anyone can help on this?

Please provide a trace (trace-cmd record -e kvm) leading to the failure.

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-17 10:14 UTC #3 lore raw

Quoted 28 lines · Avi Kivity

Quoted 27 lines · Dave Young

linux guest can not startup normally, qemu keep rebooting.

host& guest both use linux-2.6.36-rc8+ kernel

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:

[ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+
#62
[ 3924.503045] Call Trace:
[ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
[kvm_intel]
[ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
[kvm_intel]
[ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43
[kvm]
[ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
[ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
[ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
[ 3924.503117]  [<ffffffff810bee76>] ?
fire_user_return_notifiers+0x2c/0x36
[ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
[ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
[ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
[ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
[ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
[ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Quoted 4 lines

Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the failure.

Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat

Quoted 4 lines

--
error compiling committee.c: too many arguments to function

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-17 12:15 UTC #4 lore raw

Quoted 29 lines · Dave Young

Quoted 28 lines · Avi Kivity

Quoted 27 lines · Dave Young

linux guest can not startup normally, qemu keep rebooting.

host& guest both use linux-2.6.36-rc8+ kernel

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:

 [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+
 #62
 [ 3924.503045] Call Trace:
 [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
 [kvm_intel]
 [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
 [kvm_intel]
 [ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43
 [kvm]
 [ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
 [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
 [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
 [ 3924.503117]  [<ffffffff810bee76>] ?
 fire_user_return_notifiers+0x2c/0x36
 [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
 [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
 [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
 [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
 [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
 [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Quoted 8 lines

Quoted 4 lines
Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the failure.
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat

Was this done with an old version of trace-cmd? Please use the latest git. Make sure you have udis86 and udis86-devel installed when you build it.

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 04:43 UTC #5 lore raw

Quoted 41 lines · Avi Kivity

Quoted 40 lines · Dave Young

Quoted 37 lines · Avi Kivity

Quoted 31 lines · Dave Young

linux guest can not startup normally, qemu keep rebooting.

host& guest both use linux-2.6.36-rc8+ kernel

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:

 [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
2.6.36-rc8+
 #62
 [ 3924.503045] Call Trace:
 [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
 [kvm_intel]
 [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
 [kvm_intel]
 [ 3924.503080]  [<ffffffffa021f203>]
kvm_arch_vcpu_ioctl_run+0x95f/0xc43
 [kvm]
 [ 3924.503087]  [<ffffffff81503e54>] ?
__mutex_unlock_slowpath+0x11c/0x131
 [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
[kvm]
 [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
 [ 3924.503117]  [<ffffffff810bee76>] ?
 fire_user_return_notifiers+0x2c/0x36
 [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
 [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
 [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
 [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
 [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
 [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Quoted 1 line

Anyone can help on this?

Please provide a trace (trace-cmd record -e kvm) leading to the failure.

Quoted 6 lines

Quoted 3 lines
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat
Was this done with an old version of trace-cmd? Please use the latest git. Make sure you have udis86 and udis86-devel installed when you build it.

Hi, the trace-cmd was build from git version, but without udis86.

I updated trace.dat on the same url, is it a correct one?

Quoted 5 lines

--
error compiling committee.c: too many arguments to function

Signature · 2 lines

Regards
dave

Jan Kiszka

2010-10-18 07:14 UTC #6 lore raw

Quoted 46 lines · Dave Young

Quoted 45 lines · Avi Kivity
Quoted 43 lines · Dave Young
Quoted 32 lines · Avi Kivity
Quoted 31 lines · Dave Young
linux guest can not startup normally, qemu keep rebooting.
host& guest both use linux-2.6.36-rc8+ kernel
Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
 [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
2.6.36-rc8+
 #62
 [ 3924.503045] Call Trace:
 [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
 [kvm_intel]
 [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
 [kvm_intel]
 [ 3924.503080]  [<ffffffffa021f203>]
kvm_arch_vcpu_ioctl_run+0x95f/0xc43
 [kvm]
 [ 3924.503087]  [<ffffffff81503e54>] ?
__mutex_unlock_slowpath+0x11c/0x131
 [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
[kvm]
 [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
 [ 3924.503117]  [<ffffffff810bee76>] ?
 fire_user_return_notifiers+0x2c/0x36
 [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
 [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
 [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
 [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
 [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
 [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the
failure.
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat

Quoted 7 lines

Quoted 2 lines
Was this done with an old version of trace-cmd? Please use the latest git.
Make sure you have udis86 and udis86-devel installed when you build it.
Hi, the trace-cmd was build from git version, but without udis86.
I updated trace.dat on the same url, is it a correct one?

IIUC, plugin_kvm is only involved in reporting, not recording, so udis86 shouldn't matter.

I had a look out of curiosity (just managed to make my trace tools work smoothly), and the data is properly parsed by kernelshark here. There is a strange NMI storm going on right before the triple fault (on some MSR write) hits the fan.

Jan

Quoted 8 lines

Quoted 5 lines
--
error compiling committee.c: too many arguments to function

Avi Kivity

2010-10-18 08:55 UTC #7 lore raw

Quoted 55 lines · Jan Kiszka

Quoted 46 lines · Dave Young
Quoted 45 lines · Avi Kivity
Quoted 43 lines · Dave Young
Quoted 32 lines · Avi Kivity
    On 10/17/2010 10:54 AM, Dave Young wrote:
Quoted 31 lines
linux guest can not startup normally, qemu keep rebooting.
host& guest both use linux-2.6.36-rc8+ kernel
Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
 2.6.36-rc8+
  #62
  [ 3924.503045] Call Trace:
  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
  [kvm_intel]
  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
  [kvm_intel]
  [ 3924.503080]  [<ffffffffa021f203>]
 kvm_arch_vcpu_ioctl_run+0x95f/0xc43
  [kvm]
  [ 3924.503087]  [<ffffffff81503e54>] ?
 __mutex_unlock_slowpath+0x11c/0x131
  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
 [kvm]
  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
  [ 3924.503117]  [<ffffffff810bee76>] ?
  fire_user_return_notifiers+0x2c/0x36
  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the
failure.
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat
Quoted 7 lines
Quoted 2 lines
Was this done with an old version of trace-cmd? Please use the latest git.
Make sure you have udis86 and udis86-devel installed when you build it.
Hi, the trace-cmd was build from git version, but without udis86.
I updated trace.dat on the same url, is it a correct one?

Quoted 7 lines

IIUC, plugin_kvm is only involved in reporting, not recording, so udis86 shouldn't matter.
I had a look out of curiosity (just managed to make my trace tools work smoothly), and the data is properly parsed by kernelshark here. There is a strange NMI storm going on right before the triple fault (on some MSR write) hits the fan.

Doesn't parse here. I get a bus error.

Steven, is trace.dat meant to be transportable? What does it depend on?

Jan, can you upload your parsed data somewhere? I want to have a look as well.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

2010-10-18 08:58 UTC #8 lore raw

Quoted 73 lines · Avi Kivity

Quoted 64 lines · Jan Kiszka
Quoted 4 lines · Dave Young
Quoted 3 lines · Avi Kivity
Quoted 2 lines · Dave Young
On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Dave Young
Quoted 3 lines
linux guest can not startup normally, qemu keep rebooting.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
host& guest both use linux-2.6.36-rc8+ kernel
Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
 2.6.36-rc8+
  #62
  [ 3924.503045] Call Trace:
  [ 3924.503055]  [<ffffffffa0176047>] 
handle_triple_fault+0x11/0x26
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
[kvm_intel]
  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
  [kvm_intel]
  [ 3924.503080]  [<ffffffffa021f203>]
 kvm_arch_vcpu_ioctl_run+0x95f/0xc43
  [kvm]
  [ 3924.503087]  [<ffffffff81503e54>] ?
 __mutex_unlock_slowpath+0x11c/0x131
  [ 3924.503101]  [<ffffffffa021e1f9>] ? 
kvm_arch_vcpu_load+0x90/0xe6
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[kvm]
  [ 3924.503112]  [<ffffffffa0210faf>] 
kvm_vcpu_ioctl+0x109/0x54a [kvm]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
  [ 3924.503117]  [<ffffffff810bee76>] ?
  fire_user_return_notifiers+0x2c/0x36
  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
  [ 3924.503127]  [<ffffffff81002789>] ? 
do_notify_resume+0x717/0x73d
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
  [ 3924.503131]  [<ffffffff8103994e>] ? 
finish_task_switch+0x69/0xda
>>>>>   [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
>>>>>   [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
  [ 3924.503142]  [<ffffffff81002c02>] 
system_call_fastpath+0x16/0x1b
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the
failure.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat
Was this done with an old version of trace-cmd? Please use the
latest git.
Quoted 1 line
Quoted 1 line
Make sure you have udis86 and udis86-devel installed when you
build it.
Quoted 5 lines
Hi, the trace-cmd was build from git version, but without udis86.
I updated trace.dat on the same url, is it a correct one?
Quoted 7 lines
IIUC, plugin_kvm is only involved in reporting, not recording, so udis86 shouldn't matter.
I had a look out of curiosity (just managed to make my trace tools work smoothly), and the data is properly parsed by kernelshark here. There is a strange NMI storm going on right before the triple fault (on some MSR write) hits the fan.

Quoted 6 lines

Doesn't parse here. I get a bus error.
Steven, is trace.dat meant to be transportable? What does it depend on?
Jan, can you upload your parsed data somewhere? I want to have a look as well.

Never mind, user error here.

Signature · 1 line

error compiling committee.c: too many arguments to function

Steven Rostedt

2010-10-18 12:26 UTC #9 lore raw

Quoted 11 lines · Avi Kivity

Quoted 8 lines
Doesn't parse here. I get a bus error.
Steven, is trace.dat meant to be transportable? What does it depend on?
Jan, can you upload your parsed data somewhere? I want to have a look as well.
Never mind, user error here.

Everything all set then?

-- Steve

Avi Kivity

2010-10-18 12:27 UTC #10 lore raw

Quoted 7 lines · Steven Rostedt

Quoted 5 lines · Avi Kivity
Quoted 5 lines
Doesn't parse here. I get a bus error.
Steven, is trace.dat meant to be transportable? What does it depend on?

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
Jan, can you upload your parsed data somewhere? I want to have a look as well.
Never mind, user error here.
Everything all set then?

Yes, sorry for the noise.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

2010-10-18 09:15 UTC #11 lore raw

In reply to Jan Kiszka

Quoted 55 lines · Jan Kiszka

Quoted 46 lines · Dave Young
Quoted 45 lines · Avi Kivity
Quoted 43 lines · Dave Young
Quoted 32 lines · Avi Kivity
    On 10/17/2010 10:54 AM, Dave Young wrote:
Quoted 31 lines
linux guest can not startup normally, qemu keep rebooting.
host& guest both use linux-2.6.36-rc8+ kernel
Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
 2.6.36-rc8+
  #62
  [ 3924.503045] Call Trace:
  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
  [kvm_intel]
  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
  [kvm_intel]
  [ 3924.503080]  [<ffffffffa021f203>]
 kvm_arch_vcpu_ioctl_run+0x95f/0xc43
  [kvm]
  [ 3924.503087]  [<ffffffff81503e54>] ?
 __mutex_unlock_slowpath+0x11c/0x131
  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
 [kvm]
  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
  [ 3924.503117]  [<ffffffff810bee76>] ?
  fire_user_return_notifiers+0x2c/0x36
  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the
failure.
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat
Quoted 7 lines
Quoted 2 lines
Was this done with an old version of trace-cmd? Please use the latest git.
Make sure you have udis86 and udis86-devel installed when you build it.
Hi, the trace-cmd was build from git version, but without udis86.
I updated trace.dat on the same url, is it a correct one?

Quoted 8 lines

IIUC, plugin_kvm is only involved in reporting, not recording, so udis86 shouldn't matter.
I had a look out of curiosity (just managed to make my trace tools work smoothly), and the data is properly parsed by kernelshark here. There is a strange NMI storm going on right before the triple fault (on some MSR write) hits the fan.

It's secondary_startup_64 enabling efer.sce and efer.nx.

Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 12:57 UTC #12 lore raw

Quoted 65 lines · Avi Kivity

Quoted 55 lines · Jan Kiszka
Quoted 46 lines · Dave Young
Quoted 45 lines · Avi Kivity
Quoted 43 lines · Dave Young
Quoted 32 lines · Avi Kivity
    On 10/17/2010 10:54 AM, Dave Young wrote:
Quoted 31 lines
linux guest can not startup normally, qemu keep rebooting.
host& guest both use linux-2.6.36-rc8+ kernel
Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
 2.6.36-rc8+
  #62
  [ 3924.503045] Call Trace:
  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
  [kvm_intel]
  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
  [kvm_intel]
  [ 3924.503080]  [<ffffffffa021f203>]
 kvm_arch_vcpu_ioctl_run+0x95f/0xc43
  [kvm]
  [ 3924.503087]  [<ffffffff81503e54>] ?
 __mutex_unlock_slowpath+0x11c/0x131
  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
 [kvm]
  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
  [ 3924.503117]  [<ffffffff810bee76>] ?
  fire_user_return_notifiers+0x2c/0x36
  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 1 line
Anyone can help on this?
Please provide a trace (trace-cmd record -e kvm) leading to the
failure.
Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat
Quoted 7 lines
Quoted 2 lines
Was this done with an old version of trace-cmd? Please use the latest git.
Make sure you have udis86 and udis86-devel installed when you build it.
Hi, the trace-cmd was build from git version, but without udis86.
I updated trace.dat on the same url, is it a correct one?
Quoted 8 lines
IIUC, plugin_kvm is only involved in reporting, not recording, so udis86 shouldn't matter.
I had a look out of curiosity (just managed to make my trace tools work smoothly), and the data is properly parsed by kernelshark here. There is a strange NMI storm going on right before the triple fault (on some MSR write) hits the fan.

Quoted 4 lines

It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?

I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7250 @ 2.00GHz
stepping : 13
cpu MHz : 800.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips : 3991.42
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7250 @ 2.00GHz
stepping : 13
cpu MHz : 2001.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips : 3991.36
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

Avi Kivity

2010-10-18 13:26 UTC #13 lore raw

Quoted 7 lines · Dave Young

Quoted 5 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?
I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.

Strange.

Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 13:44 UTC #14 lore raw

Quoted 7 lines · Avi Kivity

Quoted 6 lines · Dave Young
Quoted 5 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?

Quoted 7 lines

Quoted 1 line
I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.
Strange.
Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.

I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:

[ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe
[ 3228.468328] efer = 0x100
[ 3228.468330] old_efer = 0x0

Avi Kivity

2010-10-18 13:46 UTC #15 lore raw

Quoted 10 lines · Dave Young

Quoted 9 lines · Avi Kivity
Quoted 7 lines · Dave Young
Quoted 5 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?
I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.

Quoted 8 lines

Quoted 5 lines
Strange.
Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.
I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:

Add it at the beginning please, otherwise we'll miss the important one due to an early return.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

2010-10-18 13:50 UTC #16 lore raw

Quoted 15 lines · Avi Kivity

Quoted 14 lines · Dave Young
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Dave Young
Quoted 4 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running
with NX
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
disabled?
I see nx bit in /proc/cpuinfo, Actually I don't know how to
disable it.
Quoted 3 lines
Strange.

Quoted 9 lines

Quoted 6 lines
Quoted 3 lines
Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.
I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:
Add it at the beginning please, otherwise we'll miss the important one due to an early return.

No need actually:

Quoted 1 line

[ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe

Bit 11 is reserved, so the guest can't enable nx.

From arch/x86/kvm/vmx.c:

Quoted 8 lines

static __init int hardware_setup(void)

{
    if (setup_vmcs_config(&vmcs_config) < 0)
        return -EIO;

    if (boot_cpu_has(X86_FEATURE_NX))
        kvm_enable_efer_bits(EFER_NX);

Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 13:57 UTC #17 lore raw

Quoted 41 lines · Avi Kivity

Quoted 19 lines · Avi Kivity
Quoted 18 lines · Dave Young
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Dave Young
Quoted 4 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host
running with NX
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
disabled?
I see nx bit in /proc/cpuinfo, Actually I don't know how to
disable it.
Quoted 6 lines
Strange.
Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.
Quoted 7 lines
Quoted 3 lines
I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:
Add it at the beginning please, otherwise we'll miss the important one due to an early return.
No need actually:
Quoted 1 line
[ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe
Bit 11 is reserved, so the guest can't enable nx.
From arch/x86/kvm/vmx.c:
Quoted 5 lines
static __init int hardware_setup(void)
{
if (setup_vmcs_config(&vmcs_config) < 0)
       return -EIO;

Quoted 6 lines

Quoted 3 lines
   if (boot_cpu_has(X86_FEATURE_NX))
       kvm_enable_efer_bits(EFER_NX);
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).

It's 0

Avi Kivity

2010-10-18 14:06 UTC #18 lore raw

Quoted 5 lines · Dave Young

Quoted 3 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0

Well, that's ridiculous. Looking back at your /proc/cpuinfo:

Quoted 5 lines

processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority

processor 0 doesn't have nx.

Ingo, Peter, any ideas? Where did the missing feature go?

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 14:11 UTC #19 lore raw

Quoted 7 lines · Avi Kivity

Quoted 6 lines · Dave Young
Quoted 4 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0

Please wait, I reply too quick, actually it boots two times fine then one time fail.

The printk only happen once. So to make sure the result I retested several times, can not get the print as well.

Quoted 25 lines

Well, that's ridiculous. Looking back at your /proc/cpuinfo:
Quoted 7 lines
processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
Quoted 6 lines
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor 0 doesn't have nx.
Ingo, Peter, any ideas? Where did the missing feature go?
--
error compiling committee.c: too many arguments to function

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-18 14:13 UTC #20 lore raw

Quoted 11 lines · Dave Young

Quoted 7 lines · Avi Kivity
Quoted 6 lines · Dave Young
Quoted 4 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0
Please wait, I reply too quick, actually it boots two times fine then one time fail.

What boots fine? The guest?

Quoted 2 lines

The printk only happen once. So to make sure the result I retested several times, can not get the print as well.

boot_cpu_has() is called only when the kvm module is inserted.

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 14:15 UTC #21 lore raw

Quoted 9 lines · Avi Kivity

Quoted 8 lines · Dave Young
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Dave Young
>
> Please patch hardware_setup() to show the value of > boot_cpu_has(X86_FEATURE_NX).

Quoted 6 lines

Quoted 4 lines
Quoted 1 line
Quoted 1 line
It's 0
Please wait, I reply too quick, actually it boots two times fine then one time fail.
What boots fine? The guest?

Yes the guest

Quoted 5 lines

Quoted 3 lines
The printk only happen once. So to make sure the result I retested several times, can not get the print as well.
boot_cpu_has() is called only when the kvm module is inserted.

Oh, then I need to test again.

Quoted 5 lines

--
error compiling committee.c: too many arguments to function

Signature · 2 lines

Regards
dave

Dave Young

author

2010-10-18 14:17 UTC #22 lore raw

Quoted 20 lines · Dave Young

Quoted 9 lines · Avi Kivity
Quoted 8 lines · Dave Young
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Dave Young
>
> Please patch hardware_setup() to show the value of > boot_cpu_has(X86_FEATURE_NX).
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
It's 0
Please wait, I reply too quick, actually it boots two times fine then one time fail.
What boots fine? The guest?
Yes the guest
Quoted 1 line

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
The printk only happen once. So to make sure the result I retested several times, can not get the print as well.
boot_cpu_has() is called only when the kvm module is inserted.
Oh, then I need to test again.

I'm lucky, this time I can make sure has_nx = 0

after insmod, guest boot failed at the first test

Quoted 13 lines

Quoted 6 lines
--
error compiling committee.c: too many arguments to function
--
Regards
dave

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-18 14:20 UTC #23 lore raw

In reply to Dave Young

Quoted 10 lines · Dave Young

Quoted 9 lines · Avi Kivity
Quoted 8 lines · Dave Young
Quoted 6 lines · Avi Kivity
     On 10/18/2010 03:58 PM, Dave Young wrote:
Quoted 5 lines
Quoted 4 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).

Quoted 9 lines

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
   It's 0
Please wait, I reply too quick, actually it boots two times fine then one time fail.
What boots fine? The guest?
Yes the guest

That makes some sort of wierd sense. First kvm samples X86_FEATURE_NX, sees that it has the feature, reports it to qemu, qemu runs the guest, NX gets disabled in between, the guest tries to enable NX, kvm kills it.

Second time round, kvm reports that NX is not available, qemu does not enable it, neither does the guest, and everyone is happy.

Please confirm that when the guest does boot, it boots without NX in /proc/cpuinfo.

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-18 14:25 UTC #24 lore raw

Quoted 26 lines · Avi Kivity

Quoted 13 lines · Dave Young
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Dave Young
On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<[email protected]> wrote:
> On 10/18/2010 03:58 PM, Dave Young wrote:
>>
>> >
>> > Please patch hardware_setup() to show the value of >> > boot_cpu_has(X86_FEATURE_NX).
>>
>> It's 0
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Please wait, I reply too quick, actually it boots two times fine then one time fail.
What boots fine? The guest?
Yes the guest
That makes some sort of wierd sense. First kvm samples X86_FEATURE_NX, sees that it has the feature, reports it to qemu, qemu runs the guest, NX gets disabled in between, the guest tries to enable NX, kvm kills it.

Quoted 6 lines

Second time round, kvm reports that NX is not available, qemu does not enable it, neither does the guest, and everyone is happy.
Please confirm that when the guest does boot, it boots without NX in /proc/cpuinfo.

Yes, there's no nx in guest cpuinfo

hardware_setup is run only when insmod so the nx is of host, thus the first printk result is right value?

Quoted 4 lines

--
error compiling committee.c: too many arguments to function

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-18 14:28 UTC #25 lore raw

Quoted 8 lines · Dave Young

Quoted 8 lines
That makes some sort of wierd sense. First kvm samples X86_FEATURE_NX, sees that it has the feature, reports it to qemu, qemu runs the guest, NX gets disabled in between, the guest tries to enable NX, kvm kills it.
Second time round, kvm reports that NX is not available, qemu does not enable it, neither does the guest, and everyone is happy.

Quoted 8 lines

Quoted 3 lines
Please confirm that when the guest does boot, it boots without NX in /proc/cpuinfo.
Yes, there's no nx in guest cpuinfo
hardware_setup is run only when insmod so the nx is of host, thus the first printk result is right value?

There's no right value since the two cpus have different values for nx.

Signature · 1 line

error compiling committee.c: too many arguments to function

H. Peter Anvin

2010-10-18 18:47 UTC #26 lore raw

In reply to Avi Kivity

Quoted 12 lines · Avi Kivity

Quoted 5 lines · Dave Young
Quoted 3 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0
Well, that's ridiculous. Looking back at your /proc/cpuinfo:
Quoted 3 lines
processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority

Quoted 7 lines

Quoted 2 lines
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor 0 doesn't have nx.
Ingo, Peter, any ideas? Where did the missing feature go?

I don't see the original report -- in particular I can't see the kernel version -- but I suspect this *might* be an instance of the same bug fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055 (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also needed; these both should be in -stable or on the way there.)

-hpa

Dave Young

author

2010-10-19 01:19 UTC #27 lore raw

Quoted 18 lines · H. Peter Anvin

Quoted 17 lines · Avi Kivity
Quoted 5 lines · Dave Young
Quoted 3 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0
Well, that's ridiculous. Looking back at your /proc/cpuinfo:
Quoted 5 lines
processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor 0 doesn't have nx.

Quoted 8 lines

Quoted 2 lines
Ingo, Peter, any ideas? Where did the missing feature go?
I don't see the original report -- in particular I can't see the kernel version -- but I suspect this *might* be an instance of the same bug fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055 (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also needed; these both should be in -stable or on the way there.)

The kernel is git version after 2.6.36-rc8, so I think the commits don't fix this problem.

Signature · 2 lines

Regards
dave

H. Peter Anvin

2010-10-19 02:59 UTC #28 lore raw

Quoted 22 lines · Dave Young

Quoted 21 lines · H. Peter Anvin
Quoted 19 lines · Avi Kivity
Quoted 5 lines · Dave Young
Quoted 3 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0
Well, that's ridiculous. Looking back at your /proc/cpuinfo:
Quoted 5 lines
processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor 0 doesn't have nx.
Ingo, Peter, any ideas? Where did the missing feature go?

Quoted 8 lines

Quoted 5 lines
I don't see the original report -- in particular I can't see the kernel version -- but I suspect this *might* be an instance of the same bug fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055 (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also needed; these both should be in -stable or on the way there.)
The kernel is git version after 2.6.36-rc8, so I think the commits don't fix this problem.

OK. Please install msr-tools on your system, and give us the outputs of:

rdmsr -p 0 -xc 0x1a0
rdmsr -p 1 -xc 0x1a0

-hpa

Signature · 2 lines

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Dave Young

author

2010-10-19 06:55 UTC #29 lore raw

Quoted 29 lines · H. Peter Anvin

Quoted 28 lines · Dave Young
Quoted 26 lines · H. Peter Anvin
Quoted 12 lines · Avi Kivity
Quoted 5 lines · Dave Young
Quoted 3 lines
Please patch hardware_setup() to show the value of boot_cpu_has(X86_FEATURE_NX).
It's 0
Well, that's ridiculous. Looking back at your /proc/cpuinfo:
Quoted 3 lines
processor : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
Quoted 7 lines
Quoted 2 lines
processor : 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
processor 0 doesn't have nx.
Ingo, Peter, any ideas? Where did the missing feature go?
I don't see the original report -- in particular I can't see the kernel version -- but I suspect this *might* be an instance of the same bug fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055 (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also needed; these both should be in -stable or on the way there.)

Quoted 6 lines

Quoted 2 lines
The kernel is git version after 2.6.36-rc8, so I think the commits don't fix this problem.
OK. Please install msr-tools on your system, and give us the outputs of:
rdmsr -p 0 -xc 0x1a0

0x1764952489

Quoted 1 line

rdmsr -p 1 -xc 0x1a0

0x1364952489

Signature · 2 lines

Regards
dave

Avi Kivity

2010-10-19 08:58 UTC #30 lore raw

Quoted 8 lines · Dave Young

Quoted 3 lines
OK. Please install msr-tools on your system, and give us the outputs of:
rdmsr -p 0 -xc 0x1a0
0x1764952489
Quoted 1 line
rdmsr -p 1 -xc 0x1a0
0x1364952489

That's MSR_IA32_MISC_ENABLE_XD_DISABLE. Looks like the kernel never plays with it.

Suggest:
- updating your BIOS
- playing with the NX feature in your BIOS setup screens

Signature · 1 line

error compiling committee.c: too many arguments to function

Dave Young

author

2010-10-19 11:51 UTC #31 lore raw

Quoted 4 lines · Avi Kivity

Quoted 3 lines · Dave Young
Quoted 3 lines
OK. Please install msr-tools on your system, and give us the outputs of:
rdmsr -p 0 -xc 0x1a0

Quoted 8 lines

Quoted 5 lines
0x1764952489
Quoted 1 line
rdmsr -p 1 -xc 0x1a0
0x1364952489
That's MSR_IA32_MISC_ENABLE_XD_DISABLE. Looks like the kernel never plays with it.

Hope someone can fix this. IMO we should not depends on bios setting

Quoted 4 lines

Suggest:
- updating your BIOS
- playing with the NX feature in your BIOS setup screens

Enable the XD in bios resolved the problem. Thanks you all for the help.

Quoted 5 lines

Signature · 2 lines

-- 
error compiling committee.c: too many arguments to function

H. Peter Anvin

2010-10-19 14:56 UTC #32 lore raw

In reply to Avi Kivity

Right... note in particular that the BIOS sets them up inconsistently. The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.

Quoted 21 lines · Avi Kivity

Quoted 8 lines · Dave Young
Quoted 3 lines
OK. Please install msr-tools on your system, and give us the outputs of:
rdmsr -p 0 -xc 0x1a0
0x1764952489
Quoted 1 line
rdmsr -p 1 -xc 0x1a0
0x1364952489
That's MSR_IA32_MISC_ENABLE_XD_DISABLE. Looks like the kernel never plays with it.
Suggest:
- updating your BIOS
- playing with the NX feature in your BIOS setup screens
Signature · 1 line
error compiling committee.c: too many arguments to function

Signature · 1 line

Sent from my mobile phone.  Please pardon any lack of formatting.

Avi Kivity

2010-10-19 15:04 UTC #33 lore raw

Quoted 1 line · H. Peter Anvin

Right... note in particular that the BIOS sets them up inconsistently. The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.

I guess kvm should be modified to mask GET_SUPPORTED_CPUID results with the boot_cpu features. This will make kvm take advantage of the core kernel doing the right thing.

H. Peter Anvin

2010-10-19 15:31 UTC #34 lore raw

That would probably be a good idea. The kernel also sets and clears bits for various errata or other special issues.

Quoted 7 lines · Avi Kivity

Quoted 1 line · H. Peter Anvin
Right... note in particular that the BIOS sets them up inconsistently. The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.
I guess kvm should be modified to mask GET_SUPPORTED_CPUID results with the boot_cpu features. This will make kvm take advantage of the core kernel doing the right thing.

Signature · 1 line

Sent from my mobile phone.  Please pardon any lack of formatting.

Dave Young

author

2010-10-18 13:50 UTC #35 lore raw

In reply to Avi Kivity

Quoted 17 lines · Avi Kivity

Quoted 16 lines · Dave Young
Quoted 14 lines · Avi Kivity
Quoted 7 lines · Dave Young
Quoted 5 lines
It's secondary_startup_64 enabling efer.sce and efer.nx.
Dave, please post your /proc/cpuinfo. Is your host running with NX disabled?
I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.
Strange.
Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), efer (an argument) and old_efer (a local), and post the results.

Quoted 5 lines

Quoted 2 lines
I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:
Add it at the beginning please, otherwise we'll miss the important one due to an early return.

here it is:

[ 3679.616500] efer_reserved_bits = 0xfffffffffffffafe
[ 3679.616504] efer = 0xd01
[ 3679.616506] old_efer = 0x500