Skip to contentSkip to conversation

current qemu-kvm doesn't work with vhost

DISCUSSION Applied Amos Kong 4 participants 9 messages Jun 10, 2011 – Jun 21, 2011 KVM

lore
Share & tools
lore

Amos Kong

author
#0 lore raw

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot -device virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

(qemu) [New Thread 0x7fffbe7dd700 (LWP 10410)]
[New Thread 0x7fffbdfdc700 (LWP 10411)]
qemu-kvm: /project/rh/qemu-kvm/hw/msix.c:616: msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.

Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff3233700 (LWP 10405)] 0x00007ffff6593ba5 in raise () from /lib/libc.so.6 (gdb) bt

#0  0x00007ffff6593ba5 in raise () from /lib/libc.so.6
#1  0x00007ffff65976b0 in abort () from /lib/libc.so.6
#2  0x00007ffff658ca71 in __assert_fail () from /lib/libc.so.6
#3  0x00000000005754eb in msix_unset_mask_notifier (dev=0x1e56c50) at /project/rh/qemu-kvm/hw/msix.c:616
#4  0x0000000000571ea4 in virtio_pci_set_guest_notifiers (opaque=0x1e56c50, assign=true) at /project/rh/qemu-kvm/hw/virtio-pci.c:674
#5  0x000000000042c279 in vhost_dev_start (hdev=0x11c0420, vdev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost.c:710
#6  0x0000000000429fe9 in vhost_net_start (net=0x11c0420, dev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost_net.c:142
#7  0x0000000000425644 in virtio_net_vhost_status (n=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:127
#8  0x00000000004256ec in virtio_net_set_status (vdev=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:144
#9  0x00000000005709e6 in virtio_set_status (vdev=0x1c40f10, val=7 '\a') at /project/rh/qemu-kvm/hw/virtio.h:136

#10 0x000000000057136f in virtio_ioport_write (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:338 #11 0x00000000005717db in virtio_pci_config_writeb (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:462 #12 0x000000000048409d in ioport_write (index=0, address=49202, data=7) at ioport.c:81
#13 0x0000000000484756 in cpu_outb (addr=49202, val=7 '\a') at ioport.c:266
#14 0x0000000000432dd2 in kvm_handle_io (port=49202, data=0x7ffff7ff3000, direction=1, size=1, count=1) at /project/rh/qemu-kvm/kvm-all.c:864 #15 0x0000000000433266 in kvm_cpu_exec (env=0x188e5b0) at /project/rh/qemu-kvm/kvm-all.c:1008
#16 0x00000000004091ef in qemu_kvm_cpu_thread_fn (arg=0x188e5b0) at /project/rh/qemu-kvm/cpus.c:808
#17 0x00007ffff79be971 in start_thread () from /lib/libpthread.so.0
#18 0x00007ffff664692d in clone () from /lib/libc.so.6
#19 0x0000000000000000 in ?? ()

#1 lore raw
Quoted 23 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot -device virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

(qemu) [New Thread 0x7fffbe7dd700 (LWP 10410)]
[New Thread 0x7fffbdfdc700 (LWP 10411)]
qemu-kvm: /project/rh/qemu-kvm/hw/msix.c:616: msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.

Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff3233700 (LWP 10405)] 0x00007ffff6593ba5 in raise () from /lib/libc.so.6 (gdb) bt

#0  0x00007ffff6593ba5 in raise () from /lib/libc.so.6
#1  0x00007ffff65976b0 in abort () from /lib/libc.so.6
#2  0x00007ffff658ca71 in __assert_fail () from /lib/libc.so.6
#3  0x00000000005754eb in msix_unset_mask_notifier (dev=0x1e56c50) at /project/rh/qemu-kvm/hw/msix.c:616
#4  0x0000000000571ea4 in virtio_pci_set_guest_notifiers (opaque=0x1e56c50, assign=true) at /project/rh/qemu-kvm/hw/virtio-pci.c:674
#5  0x000000000042c279 in vhost_dev_start (hdev=0x11c0420, vdev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost.c:710
#6  0x0000000000429fe9 in vhost_net_start (net=0x11c0420, dev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost_net.c:142
#7  0x0000000000425644 in virtio_net_vhost_status (n=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:127
#8  0x00000000004256ec in virtio_net_set_status (vdev=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:144
#9  0x00000000005709e6 in virtio_set_status (vdev=0x1c40f10, val=7 '\a') at /project/rh/qemu-kvm/hw/virtio.h:136
Quoted 10 lines

#10 0x000000000057136f in virtio_ioport_write (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:338 #11 0x00000000005717db in virtio_pci_config_writeb (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:462 #12 0x000000000048409d in ioport_write (index=0, address=49202, data=7) at ioport.c:81
#13 0x0000000000484756 in cpu_outb (addr=49202, val=7 '\a') at ioport.c:266
#14 0x0000000000432dd2 in kvm_handle_io (port=49202, data=0x7ffff7ff3000, direction=1, size=1, count=1) at /project/rh/qemu-kvm/kvm-all.c:864 #15 0x0000000000433266 in kvm_cpu_exec (env=0x188e5b0) at /project/rh/qemu-kvm/kvm-all.c:1008
#16 0x00000000004091ef in qemu_kvm_cpu_thread_fn (arg=0x188e5b0) at /project/rh/qemu-kvm/cpus.c:808
#17 0x00007ffff79be971 in start_thread () from /lib/libpthread.so.0
#18 0x00007ffff664692d in clone () from /lib/libc.so.6
#19 0x0000000000000000 in ?? ()

I already came across that symptom in a different context. Fixed by the patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies in qemu-kvm. Will post a fix.

Jan

-----8<-----

From: Jan Kiszka <[email protected]>
qemu-kvm: Fix error path of virtio_pci_set_guest_notifiers

The mask notifier is never installed on error, so this deinstallation will just trigger an assert in msix_unset_mask_notifier.

---
 hw/virtio-pci.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)
hw/virtio-pci.c +0 -4
index c4735c5..311c47b 100644
@@ -688,10 +688,6 @@ static int virtio_pci_set_guest_notifiers(void *opaque, bool assign)
 
 assign_error:
     /* We get here on assignment failure. Recover by undoing for VQs 0 .. n. */
-    if (assign) {
-        msix_unset_mask_notifier(&proxy->pci_dev);
-    }
-
     while (--n >= 0) {
         virtio_pci_set_guest_notifier(opaque, n, !assign);
     }
#2 lore raw
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot

-device

Quoted 11 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context. Fixed by the patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies in qemu-kvm. Will post a fix.

Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the guest:

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Greetz Georg

#3 lore raw
Quoted 22 lines · Georg Hopp
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot

-device

Quoted 11 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context. Fixed by the patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies in qemu-kvm. Will post a fix.

Jan

FYI

Quoted 8 lines

I encountered the same problem and applied the patch.

Well this results in the following error while starting the guest:

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Hmm, works fine for me. The vhost-net module is loaded (though I got a different message when I forgot to load it)?

Jan

#4 lore raw

Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:

Quoted 32 lines · Georg Hopp
Quoted 7 lines
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2

-snapshot

Quoted 5 lines

-device

Quoted 4 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context. Fixed

by the

Quoted 17 lines
Quoted 7 lines

patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies in qemu-kvm. Will post a fix.

Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the guest:

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

Quoted 7 lines
Quoted 1 line

and i have no network at all. I will disable vhost=on for now.

Hmm, works fine for me. The vhost-net module is loaded (though I got a different message when I forgot to load it)?

Jan

Generally it works for me until git revision b2146d8bd.

I have compiled vhost-net directly in my kernel so a have definetly not forgotten to load it... As i use gentoo i made an ebuild that installes exactly this revision.
If i find the time to do some debugging i will do so, but actually i am very busy with my job family and the use of kvm is just a sparetime thing. :D

If it would be of some help i can set a breakpoint just before the path and see what causes the message.

Greetz
Georg

#5 lore raw
Quoted 35 lines · Georg Hopp

Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:

Quoted 34 lines · Georg Hopp
Quoted 7 lines
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2

-snapshot

Quoted 5 lines

-device

Quoted 4 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context. Fixed by

the

Quoted 18 lines
Quoted 7 lines

patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies in qemu-kvm. Will post a fix.

Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the guest:

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Quoted 7 lines
Quoted 5 lines

Hmm, works fine for me. The vhost-net module is loaded (though I got a different message when I forgot to load it)?

Jan

Generally it works for me until git revision b2146d8bd.

You mean including that commit, right?

Quoted 11 lines

I have compiled vhost-net directly in my kernel so a have definetly not forgotten to load it...
As i use gentoo i made an ebuild that installes exactly this revision. If i find the time to do some debugging i will do so, but actually i am very busy with my job
family and the use of kvm is just a sparetime thing. :D

If it would be of some help i can set a breakpoint just before the path and see what
causes the message.

Let's start with double-checking that you are on ce5f0a588b, did a make clean && make, and then actually used that result. I suspect an inconsistent build as ce5f0a588b makes the difference between ENOSYS
(38) and working vhost support here.

Jan

#6 lore raw

Am 19.06.2011 10:58:20 schrieb(en) Jan Kiszka:

Quoted 66 lines · Georg Hopp
Quoted 15 lines

Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:

Quoted 14 lines · Georg Hopp
Quoted 7 lines
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2

-snapshot

Quoted 5 lines

-device

Quoted 4 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context.

Fixed by

Quoted 5 lines
Quoted 5 lines

the

Quoted 4 lines
Quoted 4 lines

patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies

in

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

qemu-kvm. Will post a fix.

Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the

guest:

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Hmm, works fine for me. The vhost-net module is loaded (though I

got a

Quoted 6 lines
Quoted 4 lines

different message when I forgot to load it)?

Jan

Generally it works for me until git revision b2146d8bd.

You mean including that commit, right?

Quoted 2 lines

I have compiled vhost-net directly in my kernel so a have definetly

not

Quoted 2 lines

forgotten to load it...
As i use gentoo i made an ebuild that installes exactly this

revision.

Quoted 1 line

If i find the time to do some debugging i will do so, but actually

i am

Quoted 4 lines

very busy with my job
family and the use of kvm is just a sparetime thing. :D

If it would be of some help i can set a breakpoint just before the

path

Quoted 2 lines

and see what
causes the message.

Quoted 8 lines

Let's start with double-checking that you are on ce5f0a588b, did a make clean && make, and then actually used that result. I suspect an inconsistent build as ce5f0a588b makes the difference between ENOSYS
(38) and working vhost support here.

Jan

Hi,

sorry for the long wait. Had a hard weekend. ;)

at revision: ce5f0a588b     - check
applied patch               - check
clean make                  - check

And now everything works as expected...well, at least i got no error at all. Thanks! Anything else i can do?

Georg

#7 lore raw
In reply to Jan Kiszka

Am 19.06.2011 10:58:20 schrieb(en) Jan Kiszka:

Quoted 66 lines · Georg Hopp
Quoted 15 lines

Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:

Quoted 14 lines · Georg Hopp
Quoted 7 lines
Quoted 6 lines · Jan Kiszka <jan.kiszka <at> web.de> writes:
Quoted 4 lines · Amos Kong

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2

-snapshot

Quoted 5 lines

-device

Quoted 4 lines

virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

Quoted 1 line

I already came across that symptom in a different context.

Fixed by

Quoted 5 lines
Quoted 5 lines

the

Quoted 4 lines
Quoted 4 lines

patch below.

However, the real issue is related to an upstream cleanup of the virtio-pci build. That reveals some unneeded build dependencies

in

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

qemu-kvm. Will post a fix.

Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the

guest:

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

qemu-system-x86_64: unable to start vhost net: 38:
falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Hmm, works fine for me. The vhost-net module is loaded (though I

got a

Quoted 6 lines
Quoted 4 lines

different message when I forgot to load it)?

Jan

Generally it works for me until git revision b2146d8bd.

You mean including that commit, right?

Quoted 2 lines

I have compiled vhost-net directly in my kernel so a have definetly

not

Quoted 2 lines

forgotten to load it...
As i use gentoo i made an ebuild that installes exactly this

revision.

Quoted 1 line

If i find the time to do some debugging i will do so, but actually

i am

Quoted 4 lines

very busy with my job
family and the use of kvm is just a sparetime thing. :D

If it would be of some help i can set a breakpoint just before the

path

Quoted 2 lines

and see what
causes the message.

Quoted 8 lines

Let's start with double-checking that you are on ce5f0a588b, did a make clean && make, and then actually used that result. I suspect an inconsistent build as ce5f0a588b makes the difference between ENOSYS
(38) and working vhost support here.

Jan

Hi again,

tried the current HEAD without any patches and vhost works again for me and with much better performance than before.

------------------------------------------------------------
Client connecting to host, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.100.4 port 48053 connected with 192.168.100.1 port 5001

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.63 GBytes  6.55 Gbits/sec

Thanks for the great work!

Greets
Georg

Applied
#8 lore raw
In reply to Jan Kiszka
Quoted 5 lines · Jan Kiszka

From: Jan Kiszka<[email protected]>
qemu-kvm: Fix error path of virtio_pci_set_guest_notifiers

The mask notifier is never installed on error, so this deinstallation will just trigger an assert in msix_unset_mask_notifier.

Applied, thanks.

Signature · 1 line
error compiling committee.c: too many arguments to function