lapic & npt
Hi,
I've just noticed that xenner doesn't work for 64bit xen guest kernels on my new shiny barcelona box. The VM crashes when trying to access the lapic. lapic setup is done before idt setup, register dump looks like init state, thus it most likely is a triple fault resetting the vcpu. Turning off npt fixes it.
xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000 (32bit guests). 32bit works fine even with npt enabled.
host kernel is fedora 9 with kvm-69 modules.
ideas anyone?
cheers,
Gerd
Signature · 1 line
http://kraxel.fedorapeople.org/xenner/
Quoted 8 lines · Gerd Hoffmann
Hi,
I've just noticed that xenner doesn't work for 64bit xen guest kernels on my new shiny barcelona box. The VM crashes when trying to access the lapic. lapic setup is done before idt setup, register dump looks like init state, thus it most likely is a triple fault resetting the vcpu. Turning off npt fixes it.
Quoted 8 lines
xenner maps the lapic to
ffff820000000000(64bit guests) orfe900000(32bit guests). 32bit works fine even with npt enabled.host kernel is fedora 9 with kvm-69 modules.
ideas anyone?
Turn on logging in lapic.c. See if something turns up.
If not, the page tables are probably set up incorrectly, but in a way that kvm doesn't notice.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Gerd Hoffmann
Hi,
I've just noticed that xenner doesn't work for 64bit xen guest kernels on my new shiny barcelona box. The VM crashes when trying to access the lapic. lapic setup is done before idt setup, register dump looks like init state, thus it most likely is a triple fault resetting the vcpu. Turning off npt fixes it.
xenner maps the lapic to
ffff820000000000(64bit guests) orfe900000(32bit guests). 32bit works fine even with npt enabled.
Quoted 8 lines
Quoted 3 lines
host kernel is fedora 9 with kvm-69 modules.
ideas anyone?
Turn on logging in
lapic.c. See if something turns up.If not, the page tables are probably set up incorrectly, but in a way that kvm doesn't notice.
Figured meanwhile the instruction accessing the lapic register somehow beames me into my gpf handler. With a quite strange rsp value. Hmm. To be analyzed further after kvm forum I guess, unless someone has a bright idea ...
cheers,
Gerd
Signature · 1 line
http://kraxel.fedorapeople.org/xenner/
Quoted 21 lines · Gerd Hoffmann
Quoted 19 lines · Avi Kivity
Quoted 8 lines · Gerd Hoffmann
Hi,
I've just noticed that xenner doesn't work for 64bit xen guest kernels on my new shiny barcelona box. The VM crashes when trying to access the lapic. lapic setup is done before idt setup, register dump looks like init state, thus it most likely is a triple fault resetting the vcpu. Turning off npt fixes it.
Quoted 7 lines
xenner maps the lapic to
ffff820000000000(64bit guests) orfe900000(32bit guests). 32bit works fine even with npt enabled.host kernel is fedora 9 with kvm-69 modules.
ideas anyone?
Turn on logging in
lapic.c. See if something turns up.
Quoted 9 lines
Quoted 3 lines
If not, the page tables are probably set up incorrectly, but in a way that kvm doesn't notice.
Figured meanwhile the instruction accessing the lapic register somehow beames me into my gpf handler. With a quite strange rsp value. Hmm. To be analyzed further after kvm forum I guess, unless someone has a bright idea ...
x86-64 will gpf if an address is not canonical. ffff820000000000 is canonical, but perhaps a type killed a zero somewhere?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Hi,
Quoted 7 lines
Quoted 4 lines
Figured meanwhile the instruction accessing the lapic register somehow beames me into my gpf handler. With a quite strange rsp value. Hmm. To be analyzed further after kvm forum I guess, unless someone has a bright idea ...
x86-64 will gpf if an address is not canonical.
ffff820000000000is canonical, but perhaps a type killed a zero somewhere?
Doesn't look like. Tried to analyze that. Compiled lastest kvm.git kernel (to get kvmtrace support), booted, and noticed "kvm-amd npt=0" doesn't work at all ...
cheers,
Gerd
Signature · 1 line
http://kraxel.fedorapeople.org/xenner/
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Gerd Hoffmann
Hi,
I've just noticed that xenner doesn't work for 64bit xen guest kernels on my new shiny barcelona box. The VM crashes when trying to access the lapic. lapic setup is done before idt setup, register dump looks like init state, thus it most likely is a triple fault resetting the vcpu. Turning off npt fixes it.
Quoted 8 lines
Quoted 6 lines
xenner maps the lapic to
ffff820000000000(64bit guests) orfe900000(32bit guests). 32bit works fine even with npt enabled.host kernel is fedora 9 with kvm-69 modules.
ideas anyone?
Turn on logging in
lapic.c. See if something turns up.
It doesn't come that far according to kvmtrace.
Quoted 2 lines
If not, the page tables are probably set up incorrectly, but in a way that kvm doesn't notice.
Dunno. Tried kvmtrace and got the results attached without and with NTP. Cutted down to the important sequence, starting with the apic MSR read. After the msr access there are some page faults (some page table pages are allocated and used to map the apic, thus likely triggering shadow pt updates). Then the apic access (id register).
With npt kvm doesn't see the apic access at all. Also the TDP_FAULT line looks fishy to me. The "virt = ... " isn't a (guest) virtual address.
btw: the iowrite to 0x00ea is console output.
ideas anyone?
Gerd
Signature · 1 line
http://kraxel.fedorapeople.org/xenner/
1673786946412 (+ 6068) VMEXIT vcpu = 0x00000000 pid = 0x0000102d [ exitcode = 0x0000007c, rip = 0xffff8300 0000a1d2 ]
0 (+ 0) MSR_READ vcpu = 0x00000000 pid = 0x0000102d [ MSR# = 0x0000001b, data = 0x00000000 fee00900 ]
1673786950748 (+ 4336) VMENTRY vcpu = 0x00000000 pid = 0x0000102d 1673786957068 (+ 6320) VMEXIT vcpu = 0x00000000 pid = 0x0000102d [ exitcode = 0x00000003, rip = 0xffff8300 00006ceb ] 0 (+ 0) CR_READ vcpu = 0x00000000 pid = 0x0000102d [ CR# = 3, value = 0x00000000 017a3000 ] 1673786966522 (+ 9454) VMENTRY vcpu = 0x00000000 pid = 0x0000102d 1673786973678 (+ 7156) VMEXIT vcpu = 0x00000000 pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 00004023 ] 0 (+ 0) PAGE_FAULT vcpu = 0x00000000 pid = 0x0000102d [ errorcode = 0x00000002, virt = 0xffff8300 00048000 ] 1673787002404 (+ 28726) VMENTRY vcpu = 0x00000000 pid = 0x0000102d 1673787034034 (+ 31630) VMEXIT vcpu = 0x00000000 pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 00004023 ] 0 (+ 0) PAGE_FAULT vcpu = 0x00000000 pid = 0x0000102d [ errorcode = 0x00000002, virt = 0xffff8300 00049000 ] 1673787049388 (+ 15354) VMENTRY vcpu = 0x00000000 pid = 0x0000102d 1673787080890 (+ 31502) VMEXIT vcpu = 0x00000000 pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 0000a1ef ] 0 (+ 0) PAGE_FAULT vcpu = 0x00000000 pid = 0x0000102d [ errorcode = 0x00000000, virt = 0xffff8200 00000020 ] 0 (+ 0) APIC_ACCESS vcpu = 0x00000000 pid = 0x0000102d [ offset = 0x00000020 ]
1673787096270 (+ 15380) VMENTRY vcpu = 0x00000000 pid = 0x0000102d
1834321779170 (+ 6312) VMEXIT vcpu = 0x00000000 pid = 0x00001080 [ exitcode = 0x0000007c, rip = 0xffff8300 0000a1d2 ]
0 (+ 0) MSR_READ vcpu = 0x00000000 pid = 0x00001080 [ MSR# = 0x0000001b, data = 0x00000000 fee00900 ]
1834321783872 (+ 4702) VMENTRY vcpu = 0x00000000 pid = 0x00001080 1834321793790 (+ 9918) VMEXIT vcpu = 0x00000000 pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 00004023 ] 0 (+ 0) TDP_FAULT vcpu = 0x00000000 pid = 0x00001080 [ errorcode = 0x00000006, virt = 0x00000000 00048000 ] 1834321818582 (+ 24792) VMENTRY vcpu = 0x00000000 pid = 0x00001080 1834321851480 (+ 32898) VMEXIT vcpu = 0x00000000 pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 00004023 ] 0 (+ 0) TDP_FAULT vcpu = 0x00000000 pid = 0x00001080 [ errorcode = 0x00000006, virt = 0x00000000 00049000 ] 1834321870858 (+ 19378) VMENTRY vcpu = 0x00000000 pid = 0x00001080 1834321906650 (+ 35792) VMEXIT vcpu = 0x00000000 pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 000053b1 ] 0 (+ 0) TDP_FAULT vcpu = 0x00000000 pid = 0x00001080 [ errorcode = 0x00000004, virt = 0x00000000 0000d000 ] 1834321912818 (+ 6168) VMENTRY vcpu = 0x00000000 pid = 0x00001080 1834321931382 (+ 18564) VMEXIT vcpu = 0x00000000 pid = 0x00001080 [ exitcode = 0x0000007b, rip = 0xffff8300 0000b256 ]
0 (+ 0) IO_WRITE vcpu = 0x00000000 pid = 0x00001080 [ port = 0x00ea, size = 1 ]
1834322090822 (+ 159440) VMENTRY vcpu = 0x00000000 pid = 0x00001080
Vringfd
In reply to Gerd HoffmannRusty/Anthony/Avi and all:
I did a read at Rusty's April 18 post of vringfd patch, though I
didn't find explicit code for how to call get_user_skb_frags, it seems to be clear the user only need to trigger a "flush_tx" command. All the rest will be automatically turned on, fetching the ring and send to bridge etc, which is perfect from performance point of view.
Now I am working on enabling VMDq which provide HW pre-sorting
mechanism base on MAC address, it can definetely take the advantage of vringfd with additional advantage of 0-copy in receive side, which makes VMDq enabled virtio-net fly with 0-copy for both transmit & receive. The only thing an VMDq agency (vringfd similar components in kernel) may bother user level is initialization and notification given that vringfd already solved the ring operations, since KVM virtual interrupt module is in kernel side, it is pretty easy to make user free of receiving packet at run time.
Any comments?
Thanks, eddie