rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: Running fedora xen on top of KVM?

Fedora-xen Konrad Rzeszutek Wilk 6 participants 11 messages Sep 17, 2015 – Sep 22, 2015 7 messages not in the archive KVM

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Konrad Rzeszutek Wilk

author

2015-09-17 20:10 UTC #7 lore raw

Quoted 81 lines · Cole Robinson

Quoted 18 lines · Konrad Rzeszutek Wilk
Quoted 17 lines · Cole Robinson
Quoted 11 lines · M A Young
Quoted 5 lines · Cole Robinson
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
<snip>
Quoted 10 lines
Quoted 9 lines
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff810032b0>]
Quoted 7 lines
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then
x/20i 0xffffffff810032b0
can help with that.
Updated to the latest kernel 4.1.6-201.fc22.x86_64. Trace is now:
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff810031f0>]
(XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest
(XEN) rax: 0000000000000015   rbx: ffffffff81c03e1c   rcx: 00000000c0010112
(XEN) rdx: 0000000000000001   rsi: ffffffff81c03e1c   rdi: 00000000c0010112
(XEN) rbp: ffffffff81c03df8   rsp: ffffffff81c03da0   r8:  ffffffff81c03e28
(XEN) r9:  ffffffff81c03e2c   r10: 0000000000000000   r11: 00000000ffffffff
(XEN) r12: ffffffff81d25a60   r13: 0000000004000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000406f0
(XEN) cr3: 0000000075c0b000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03da0:
(XEN)    00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810031f0
(XEN)    000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b
(XEN)    0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48
(XEN)    ffffffff8102a7a4 ffffffff81c03e48 ffffffff8102aa3b ffffffff81c03e48
(XEN)    cf1fa5f5e026f464 0000000001000000 ffffffff81c03ef8 0000000004000000
(XEN)    0000000000000000 ffffffff81c03e58 ffffffff81d5d142 ffffffff81c03ee8
(XEN)    ffffffff81d58b56 0000000000000000 0000000000000000 ffffffff81c03e88
(XEN)    ffffffff810f8a39 ffffffff81c03ee8 ffffffff81798b13 ffffffff00000010
(XEN)    ffffffff81c03ef8 ffffffff81c03eb8 cf1fa5f5e026f464 ffffffff81f1de9c
(XEN)    ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f28 ffffffff81d51c74 cf1fa5f5e026f464
(XEN)    0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f38 ffffffff81d51339 ffffffff81c03ff8
(XEN)    ffffffff81d548b1 0000000000000000 00600f1200000000 0000000100000800
(XEN)    0300000100000032 0000000000000005 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc

Quoted 7 lines

(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
gdb output:
(gdb) x/20i 0xffffffff810031f0
0xffffffff810031f0 <xen_read_msr_safe+16>: rdmsr

Fantastic! So we have some rdmsr that makes KVM inject an GP.

Looking at the stack you have some other values:
ffffffff81c03de0, ffffffff81c03e1c .. they should correspond
to other functions calling this one. If you do 'nm --defined vmlinux | grep ffffffff81c03e1' that should give an idea where they are. Or use 'gdb'.

That will give us an stack - and we can find what type of MSR this is. Oh wait, it is on the registers: 00000000c0010112

Ok, so where in the code is that MSR ah, that looks to be:
#define MSR_K8_TSEG_ADDR 0xc0010112

which is called at bsp_init_amd.

I think the problem here is that we are calling the 'safe' variant of MSR but we still get an injected #GP and don't expect that.

I am not really sure what the expected outcome should be here.

CC-ing xen-devel, KVM folks, and Andy who has been looking in mucking around in the _safe* pvops.

Quoted 12 lines

   0xffffffff810031f2 <xen_read_msr_safe+18>:	xor    %ecx,%ecx
   0xffffffff810031f4 <xen_read_msr_safe+20>:	shl    $0x20,%rdx
   0xffffffff810031f8 <xen_read_msr_safe+24>:	mov    %eax,%eax
   0xffffffff810031fa <xen_read_msr_safe+26>:	mov    %ecx,(%rsi)
   0xffffffff810031fc <xen_read_msr_safe+28>:	mov    %rdx,%rbx
   0xffffffff810031ff <xen_read_msr_safe+31>:	or     %rax,%rbx
   0xffffffff81003202 <xen_read_msr_safe+34>:	cmp    $0x1b,%edi
   0xffffffff81003205 <xen_read_msr_safe+37>:	
    jne    0xffffffff8100323a <xen_read_msr_safe+90>
   0xffffffff81003207 <xen_read_msr_safe+39>:	movl   $0x1,-0x18(%rbp)
   0xffffffff8100320e <xen_read_msr_safe+46>:	movl   $0x0,-0x10(%rbp)
   0xffffffff81003215 <xen_read_msr_safe+53>:	lea    -0x18(%rbp),%rdi

Quoted 10 lines

   0xffffffff81003219 <xen_read_msr_safe+57>:	lea    -0x14(%rbp),%rsi
   0xffffffff8100321d <xen_read_msr_safe+61>:	lea    -0x10(%rbp),%rdx
   0xffffffff81003221 <xen_read_msr_safe+65>:	lea    -0xc(%rbp),%rcx
   0xffffffff81003225 <xen_read_msr_safe+69>:	callq  *0xffffffff81c28618
   0xffffffff8100322c <xen_read_msr_safe+76>:	mov    %rbx,%rax
   0xffffffff8100322f <xen_read_msr_safe+79>:	and    $0xfb,%ah
   0xffffffff81003232 <xen_read_msr_safe+82>:	testb  $0x20,-0xe(%rbp)
   0xffffffff81003236 <xen_read_msr_safe+86>:	cmove  %rax,%rbx

--
xen mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/xen

Andy Lutomirski

2015-09-17 20:23 UTC #8 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 38 lines · Konrad Rzeszutek Wilk

Quoted 37 lines · Cole Robinson
Quoted 18 lines · Konrad Rzeszutek Wilk
Quoted 17 lines · Cole Robinson
Quoted 11 lines · M A Young
Quoted 5 lines · Cole Robinson
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
<snip>
Quoted 15 lines
Quoted 9 lines
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff810032b0>]
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then
x/20i 0xffffffff810032b0

Quoted 8 lines

Quoted 8 lines
Quoted 2 lines
can help with that.
Updated to the latest kernel 4.1.6-201.fc22.x86_64. Trace is now:
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]

What exactly does this mean?

Quoted 39 lines

(XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff810031f0>]
(XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest

(XEN) rax: 0000000000000015   rbx: ffffffff81c03e1c   rcx: 00000000c0010112
(XEN) rdx: 0000000000000001   rsi: ffffffff81c03e1c   rdi: 00000000c0010112
(XEN) rbp: ffffffff81c03df8   rsp: ffffffff81c03da0   r8:  ffffffff81c03e28
(XEN) r9:  ffffffff81c03e2c   r10: 0000000000000000   r11: 00000000ffffffff
(XEN) r12: ffffffff81d25a60   r13: 0000000004000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000406f0
(XEN) cr3: 0000000075c0b000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03da0:
(XEN)    00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810031f0
(XEN)    000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b
(XEN)    0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48
(XEN)    ffffffff8102a7a4 ffffffff81c03e48 ffffffff8102aa3b ffffffff81c03e48
(XEN)    cf1fa5f5e026f464 0000000001000000 ffffffff81c03ef8 0000000004000000
(XEN)    0000000000000000 ffffffff81c03e58 ffffffff81d5d142 ffffffff81c03ee8
(XEN)    ffffffff81d58b56 0000000000000000 0000000000000000 ffffffff81c03e88
(XEN)    ffffffff810f8a39 ffffffff81c03ee8 ffffffff81798b13 ffffffff00000010
(XEN)    ffffffff81c03ef8 ffffffff81c03eb8 cf1fa5f5e026f464 ffffffff81f1de9c
(XEN)    ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f28 ffffffff81d51c74 cf1fa5f5e026f464
(XEN)    0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f38 ffffffff81d51339 ffffffff81c03ff8
(XEN)    ffffffff81d548b1 0000000000000000 00600f1200000000 0000000100000800
(XEN)    0300000100000032 0000000000000005 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc

(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Quoted 7 lines

Quoted 4 lines
gdb output:
(gdb) x/20i 0xffffffff810031f0
0xffffffff810031f0 <xen_read_msr_safe+16>: rdmsr
Fantastic! So we have some rdmsr that makes KVM inject an GP.

What's the scenario? Is this Xen on KVM?

Why didn't the guest print anything?

Is the issue here that the guest died due to failure to handle an RDMSR failure or did the *hypervisor* die?

It looks like null_trap_bounce is returning true, which suggests that the failure is happening before the guest sets up exception handling.

Quoted 14 lines

Looking at the stack you have some other values:
ffffffff81c03de0, ffffffff81c03e1c .. they should correspond
to other functions calling this one. If you do 'nm --defined vmlinux | grep ffffffff81c03e1' that should give an idea where they are. Or use 'gdb'.
That will give us an stack - and we can find what type of MSR this is. Oh wait, it is on the registers: 00000000c0010112
Ok, so where in the code is that MSR ah, that looks to be:
#define MSR_K8_TSEG_ADDR 0xc0010112
which is called at bsp_init_amd.

Quoted 8 lines

I think the problem here is that we are calling the 'safe' variant of MSR but we still get an injected #GP and don't expect that.
I am not really sure what the expected outcome should be here.
CC-ing xen-devel, KVM folks, and Andy who has been looking in mucking around in the _safe* pvops.

It's too early of a failure, I think.

Cc: Borislav. Is TSEG guaranteed to exist? Can we defer that until we have exception handling working? Do we need to rig up exception handling so that it works earlier (e.g. in early_trap_init, which is presumably early enough)? Or is this just a KVM and/or Xen bug.

--Andy

Andrew Cooper

2015-09-17 21:29 UTC #9 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 36 lines · Andy Lutomirski

Quoted 34 lines · Konrad Rzeszutek Wilk
Quoted 33 lines · Cole Robinson
Quoted 16 lines · Konrad Rzeszutek Wilk
Quoted 15 lines · Cole Robinson
Quoted 10 lines · M A Young
Quoted 5 lines · Cole Robinson
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
<snip>
Quoted 14 lines
Quoted 9 lines
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff810032b0>]
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then
x/20i 0xffffffff810032b0

Quoted 8 lines

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
can help with that.
Updated to the latest kernel 4.1.6-201.fc22.x86_64. Trace is now:
about to get started...
(XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000]
What exactly does this mean?

This means that there was #GP fault originating from dom0 context, but dom0 has not yet registered a #GP handler with Xen. (I already have a patch pending to correct the wording of that error message.)

Would be a double fault on native.

Quoted 40 lines

(XEN) rax: 0000000000000015   rbx: ffffffff81c03e1c   rcx: 00000000c0010112
(XEN) rdx: 0000000000000001   rsi: ffffffff81c03e1c   rdi: 00000000c0010112
(XEN) rbp: ffffffff81c03df8   rsp: ffffffff81c03da0   r8:  ffffffff81c03e28
(XEN) r9:  ffffffff81c03e2c   r10: 0000000000000000   r11: 00000000ffffffff
(XEN) r12: ffffffff81d25a60   r13: 0000000004000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000406f0
(XEN) cr3: 0000000075c0b000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03da0:
(XEN)    00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810031f0
(XEN)    000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b
(XEN)    0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48
(XEN)    ffffffff8102a7a4 ffffffff81c03e48 ffffffff8102aa3b ffffffff81c03e48
(XEN)    cf1fa5f5e026f464 0000000001000000 ffffffff81c03ef8 0000000004000000
(XEN)    0000000000000000 ffffffff81c03e58 ffffffff81d5d142 ffffffff81c03ee8
(XEN)    ffffffff81d58b56 0000000000000000 0000000000000000 ffffffff81c03e88
(XEN)    ffffffff810f8a39 ffffffff81c03ee8 ffffffff81798b13 ffffffff00000010
(XEN)    ffffffff81c03ef8 ffffffff81c03eb8 cf1fa5f5e026f464 ffffffff81f1de9c
(XEN)    ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f28 ffffffff81d51c74 cf1fa5f5e026f464
(XEN)    0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f38 ffffffff81d51339 ffffffff81c03ff8
(XEN)    ffffffff81d548b1 0000000000000000 00600f1200000000 0000000100000800
(XEN)    0300000100000032 0000000000000005 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc

(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Quoted 7 lines

Quoted 6 lines
Quoted 4 lines
gdb output:
(gdb) x/20i 0xffffffff810031f0
0xffffffff810031f0 <xen_read_msr_safe+16>: rdmsr
Fantastic! So we have some rdmsr that makes KVM inject an GP.
What's the scenario? Is this Xen on KVM?

I believe from the thread that this is a Xen/dom0 combo running as a KVM guest.

Quoted 2 lines

Why didn't the guest print anything?

Lack of earlyprintk=xen on the dom0 command line. (IMO this really should be the default when a PVOPs detects that it is running under Xen)

Quoted 3 lines

Is the issue here that the guest died due to failure to handle an RDMSR failure or did the *hypervisor* die?

The guest suffered a GP fault which it couldn't handle. Therefore Xen crashed the domain.

When dom0 crashes, Xen goes down too.

Quoted 3 lines

It looks like null_trap_bounce is returning true, which suggests that the failure is happening before the guest sets up exception handling.

I concur.

Quoted 20 lines

Quoted 20 lines
Looking at the stack you have some other values:
ffffffff81c03de0, ffffffff81c03e1c .. they should correspond
to other functions calling this one. If you do 'nm --defined vmlinux | grep ffffffff81c03e1' that should give an idea where they are. Or use 'gdb'.
That will give us an stack - and we can find what type of MSR this is. Oh wait, it is on the registers: 00000000c0010112
Ok, so where in the code is that MSR ah, that looks to be:
#define MSR_K8_TSEG_ADDR 0xc0010112
which is called at bsp_init_amd.
I think the problem here is that we are calling the 'safe' variant of MSR but we still get an injected #GP and don't expect that.
I am not really sure what the expected outcome should be here.

Quoted 8 lines

Quoted 2 lines
CC-ing xen-devel, KVM folks, and Andy who has been looking in mucking around in the _safe* pvops.
It's too early of a failure, I think.
Cc: Borislav. Is TSEG guaranteed to exist? Can we defer that until we have exception handling working? Do we need to rig up exception handling so that it works earlier (e.g. in early_trap_init, which is presumably early enough)? Or is this just a KVM and/or Xen bug.

It would certainly help to move the exception setup as early as possible.

Quoted 1 line

From a Xen PV guests point of view, the kernel is already executing on

working pagetables and flat GDT when it starts. A set_trap_table hypercall (equivalent of lidt) ought to be the second action, following the stack switch.

This appears not to be the case, and the load_idt() is deferred until native cpu_init().

~Andrew

Borislav Petkov

2015-09-18 13:53 UTC #10 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

In reply to Andy Lutomirski

Quoted 1 line · Andy Lutomirski

Cc Borislav. Is TSEG guaranteed to exist? Can we defer that until

Why not? It is the tseg base address.

I think this is kvm injecting a #GP as it is not ignoring this MSR. Presumably modprobing kvm with "ignore_msrs=1" or so should hide the issue IIUC...

Signature · 5 lines

Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Andy Lutomirski

2015-09-18 15:20 UTC #11 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 8 lines · Borislav Petkov

Quoted 1 line · Andy Lutomirski
Cc Borislav. Is TSEG guaranteed to exist? Can we defer that until
Why not? It is the tseg base address.
I think this is kvm injecting a #GP as it is not ignoring this MSR. Presumably modprobing kvm with "ignore_msrs=1" or so should hide the issue IIUC...

But the issue should be fixed, no?

Does anyone know why this affects Xen on KVM but not native Linux on KVM? Or is there something weird about the particular KVM configuration?

In any event, Borislav, you must have typed rdmsr_safe for a reason :)
Either rdmsr_safe in that context is broken on Xen and not native
(which is entirely possible -- I didn't check carefully) or the rdmsr_safe isn't "safe" on any type of kernel. Persumably either the kernel or KVM (or both) should be fixed.

Given that we can handle fixups in the decompressor, surely it wouldn't be so hard to make early GPF fixups work in the main kernel.

--Andy

Borislav Petkov

2015-09-18 19:04 UTC #12 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 1 line · Andy Lutomirski

In any event, Borislav, you must have typed rdmsr_safe for a reason :)

Wasn't me:

6c62aa4a3c12 ("x86: make amd.c have 64bit support code")

I think the error handling of rdmsrl_safe() was needed to do the pfn games which are done in the if-clause.

Quoted 4 lines

Either rdmsr_safe in that context is broken on Xen and not native
(which is entirely possible -- I didn't check carefully) or the rdmsr_safe isn't "safe" on any type of kernel. Persumably either the kernel or KVM (or both) should be fixed.

What Paolo said.

Quoted 2 lines

Given that we can handle fixups in the decompressor, surely it wouldn't be so hard to make early GPF fixups work in the main kernel.

Frankly, I still am wondering what a sensible use case of running xen hypervisor/dom0 as a kvm guest would be.

Signature · 5 lines

Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Cole Robinson

2015-09-18 19:15 UTC #13 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 8 lines · Borislav Petkov

Quoted 2 lines · Andy Lutomirski
Given that we can handle fixups in the decompressor, surely it wouldn't be so hard to make early GPF fixups work in the main kernel.
Frankly, I still am wondering what a sensible use case of running xen hypervisor/dom0 as a kvm guest would be.

Testing libvirt + virt-manager xen support without requiring a physical machine running xen.

- Cole

Andy Lutomirski

2015-09-21 04:49 UTC #14 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

In reply to Borislav Petkov

Quoted 9 lines · Borislav Petkov

Quoted 1 line · Andy Lutomirski
In any event, Borislav, you must have typed rdmsr_safe for a reason :)
Wasn't me:
6c62aa4a3c12 ("x86: make amd.c have 64bit support code")
I think the error handling of rdmsrl_safe() was needed to do the pfn games which are done in the if-clause.

I just tried it. rdmsrl_safe and friends definitely work fine in that code. I think that Linux's Xen startup code is buggy and fails to set up early exception handling.

Try this (horribly whitespace damaged):

static void __init early_identify_cpu(struct cpuinfo_x86 *c) {
+ u64 tmp;

 #ifdef CONFIG_X86_64
        c->x86_clflush_size = 64;
        c->x86_phys_bits = 36;
@@ -752,6 +753,9 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
        c->cpu_index = 0;
        filter_cpuid_features(c, false);

+ pr_err("trying to crash\n");
+ rdmsrl_safe(0x12345678, &tmp);
+

It works fine. I bet it crashes on a Xen guest, though. I assume that Xen just works in most cases by luck.

--Andy

Konrad Rzeszutek Wilk

author

2015-09-22 18:23 UTC #15 lore raw

Quoted 31 lines · Andy Lutomirski

Quoted 9 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
In any event, Borislav, you must have typed rdmsr_safe for a reason :)
Wasn't me:
6c62aa4a3c12 ("x86: make amd.c have 64bit support code")
I think the error handling of rdmsrl_safe() was needed to do the pfn games which are done in the if-clause.
I just tried it. rdmsrl_safe and friends definitely work fine in that code. I think that Linux's Xen startup code is buggy and fails to set up early exception handling.
Try this (horribly whitespace damaged):
static void __init early_identify_cpu(struct cpuinfo_x86 *c) {
(quoted diff) +4 -0
+       u64 tmp;
 #ifdef CONFIG_X86_64
        c->x86_clflush_size = 64;
        c->x86_phys_bits = 36;
@@ -752,6 +753,9 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
        c->cpu_index = 0;
        filter_cpuid_features(c, false);
 
+       pr_err("trying to crash\n");
+       rdmsrl_safe(0x12345678, &tmp);
+
 

Quoted 2 lines

It works fine. I bet it crashes on a Xen guest, though. I assume that Xen just works in most cases by luck.

(d31) mapping kernel into physical memory
(d31) about to get started...
(XEN) traps.c:3151: GPF (0000): ffff82d0801a31ed -> ffff82d08023c77b
(XEN) traps.c:459:d31v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d080238213 create_bounce_frame+0x12b/0x13a (XEN) Domain 31 (vcpu#0) crashed on cpu#35:
(XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 35
(XEN) RIP: e033:[<ffffffff81041b64>]
(XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest

(XEN) rax: 0000000000000000   rbx: ffffffff81c03e64   rcx: 0000000012345678
(XEN) rdx: ffffffff81c03de8   rsi: ffffffff81c03dec   rdi: 0000000012345278
(XEN) rbp: ffffffff81c03e48   rsp: ffffffff81c03dd0   r8:  7420676e69797274
(XEN) r9:  6873617263206f74   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000012345678   r13: ffffffff81c03f00   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000001526f0
(XEN) cr3: 00000014e8c97000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03dd0:
(XEN)    0000000012345678 0000000000000000 0000000000000000 ffffffff81041b64
(XEN)    000000010000e030 0000000000010046 ffffffff81c03e18 000000000000e02b
(XEN)    ffffffff81041b5d ffffffff81c03e48 0000000000811809 0000000000000000
(XEN)    00000000000001a0 0000000001000000 ffffffff82009000 ffffffff81c03e68
(XEN)    ffffffff81d211ea 0000000000000000 0000000000000000 ffffffff81c03ed8
(XEN)    ffffffff81d1be59 ffffffff81c03ed8 ffffffff811892ab 0000000000000010
(XEN)    ffffffff81c03ee8 ffffffff81c03ea8 697a696c61697469 ffffffff81f15442
(XEN)    ffffffffffffffff ffffffff81db3900 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f28 ffffffff81d10f0a 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff81c03f38
(XEN)    ffffffff81d10603 ffffffff81c03ff8 ffffffff81d15f5c 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffd83a031f898b75 0000000022400800 0000000000000001
(XEN)    0000000000000000 0000000000000000 00010102464c457f 0000000000000000
(XEN)    00000001003e0003 0000000000000940 0000000000000040 00000000000012a0
(XEN)    0038004000000000 0011001200400004 0000000500000001 0000000000000000

[root@ovs107 ~]#

(gdb) x/20i 0xffffffff81041b64
0xffffffff81041b64: rdmsr

Quoted 2 lines

--Andy

--
xen mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/xen

Andy Lutomirski

2015-09-22 18:32 UTC #16 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

Quoted 74 lines · Konrad Rzeszutek Wilk

Quoted 31 lines · Andy Lutomirski
Quoted 9 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
In any event, Borislav, you must have typed rdmsr_safe for a reason :)
Wasn't me:
6c62aa4a3c12 ("x86: make amd.c have 64bit support code")
I think the error handling of rdmsrl_safe() was needed to do the pfn games which are done in the if-clause.
I just tried it. rdmsrl_safe and friends definitely work fine in that code. I think that Linux's Xen startup code is buggy and fails to set up early exception handling.
Try this (horribly whitespace damaged):
static void __init early_identify_cpu(struct cpuinfo_x86 *c) {
(quoted diff) +4 -0
+       u64 tmp;
 #ifdef CONFIG_X86_64
        c->x86_clflush_size = 64;
        c->x86_phys_bits = 36;
@@ -752,6 +753,9 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
        c->cpu_index = 0;
        filter_cpuid_features(c, false);
 
+       pr_err("trying to crash\n");
+       rdmsrl_safe(0x12345678, &tmp);
+
 
Quoted 2 lines
It works fine. I bet it crashes on a Xen guest, though. I assume that Xen just works in most cases by luck.
(d31) mapping kernel into physical memory
(d31) about to get started...
(XEN) traps.c:3151: GPF (0000): ffff82d0801a31ed -> ffff82d08023c77b
(XEN) traps.c:459:d31v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d080238213 create_bounce_frame+0x12b/0x13a (XEN) Domain 31 (vcpu#0) crashed on cpu#35:
(XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 35
(XEN) RIP: e033:[<ffffffff81041b64>]
(XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: ffffffff81c03e64   rcx: 0000000012345678
(XEN) rdx: ffffffff81c03de8   rsi: ffffffff81c03dec   rdi: 0000000012345278
(XEN) rbp: ffffffff81c03e48   rsp: ffffffff81c03dd0   r8:  7420676e69797274
(XEN) r9:  6873617263206f74   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000012345678   r13: ffffffff81c03f00   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000001526f0
(XEN) cr3: 00000014e8c97000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03dd0:
(XEN)    0000000012345678 0000000000000000 0000000000000000 ffffffff81041b64
(XEN)    000000010000e030 0000000000010046 ffffffff81c03e18 000000000000e02b
(XEN)    ffffffff81041b5d ffffffff81c03e48 0000000000811809 0000000000000000
(XEN)    00000000000001a0 0000000001000000 ffffffff82009000 ffffffff81c03e68
(XEN)    ffffffff81d211ea 0000000000000000 0000000000000000 ffffffff81c03ed8
(XEN)    ffffffff81d1be59 ffffffff81c03ed8 ffffffff811892ab 0000000000000010
(XEN)    ffffffff81c03ee8 ffffffff81c03ea8 697a696c61697469 ffffffff81f15442
(XEN)    ffffffffffffffff ffffffff81db3900 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff81c03f28 ffffffff81d10f0a 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff81c03f38
(XEN)    ffffffff81d10603 ffffffff81c03ff8 ffffffff81d15f5c 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffd83a031f898b75 0000000022400800 0000000000000001
(XEN)    0000000000000000 0000000000000000 00010102464c457f 0000000000000000
(XEN)    00000001003e0003 0000000000000940 0000000000000040 00000000000012a0
(XEN)    0038004000000000 0011001200400004 0000000500000001 0000000000000000

Quoted 5 lines

[root@ovs107 ~]#
(gdb) x/20i 0xffffffff81041b64
0xffffffff81041b64: rdmsr

Exactly. I think that Xen is missing code to wire up early_fixup_exception.

--Andy

Paolo Bonzini

maintainer

2015-09-18 15:27 UTC #17 lore raw

Re: rdmsr_safe in Linux PV (under Xen) gets an #GP:Re: [Fedora-xen] Running fedora xen on top of KVM?

In reply to Borislav Petkov

Quoted 7 lines · Borislav Petkov

Quoted 1 line · Andy Lutomirski
Quoted 1 line
Cc Borislav. Is TSEG guaranteed to exist? Can we defer that until
Why not? It is the tseg base address.
I think this is kvm injecting a #GP as it is not ignoring this MSR. Presumably modprobing kvm with "ignore_msrs=1" or so should hide the issue IIUC...

We should return 0 for the tseg MSRs. KVM always handles them in the northbridge emulation code, like Intel chipsets do. Will send a patch.

Paolo