Userspace hypercalls?

I've never really thought much about them until now. What's the case for supporting userspace hypercalls?

The current way the code works is a little scary. Hypercalls that aren't handled by kernelspace are deferred to userspace. Of course, kernelspace has no idea whether userspace is actually using a given hypercall so if kernelspace needs another one, the two may clash.

AFAICT, the primary reason to use hypercalls is performance. A vmcall is a few hundred cycles faster than a PIO exit. In the light-weight exit path, this may make a significant different. However, when going to userspace, it's not only a heavy-weight exit but it's also paying the cost of a ring transition. The few hundred cycle savings is small in comparison to the total cost so I don't think performance is a real benefit here.

The hypercall namespace is much smaller than the PIO namespace, and there's no "plug-and-play" like mechanism to resolve conflict. PIO/MMIO has this via PCI and it seems like any userspace device ought to be either a PCI device or use a static PIO port. Plus, paravirtual devices that use PCI/PIO/MMIO are much more likely to be reusable by other VMMs (Xen, QEMU, even VMware).

In the future, if we decide a certain hypercall could be done better in userspace, and we have guests using those hypercalls, it makes sense to plumb the hypercalls down.

My question is, should we support userspace hypercalls until that point?

This means a pvnet driver can be used either directly with a virtual interface on the host, or having some userspace processing in qemu.

Quoted 8 lines · Anthony Liguori

I've never really thought much about them until now. What's the case for supporting userspace hypercalls?

The current way the code works is a little scary. Hypercalls that aren't handled by kernelspace are deferred to userspace. Of course, kernelspace has no idea whether userspace is actually using a given hypercall so if kernelspace needs another one, the two may clash.

Quoted 8 lines

AFAICT, the primary reason to use hypercalls is performance. A vmcall is a few hundred cycles faster than a PIO exit. In the light-weight exit path, this may make a significant different. However, when going to userspace, it's not only a heavy-weight exit but it's also paying the cost of a ring transition. The few hundred cycle savings is small in comparison to the total cost so I don't think performance is a real benefit here.

Actually the heavyweight exit is much more expensive than the ring transition.

Quoted 7 lines

The hypercall namespace is much smaller than the PIO namespace, and there's no "plug-and-play" like mechanism to resolve conflict. PIO/MMIO has this via PCI and it seems like any userspace device ought to be either a PCI device or use a static PIO port. Plus, paravirtual devices that use PCI/PIO/MMIO are much more likely to be reusable by other VMMs (Xen, QEMU, even VMware).

Quoted 6 lines

In the future, if we decide a certain hypercall could be done better in userspace, and we have guests using those hypercalls, it makes sense to plumb the hypercalls down.

My question is, should we support userspace hypercalls until that point?

I've already mentioned this but I'll repeat it for google: allowing hypercalls to fallback to userspace gives you flexibility to have either a kernel implementation or a userspace implementation for the same functionality. This means a pvnet driver can be used either directly with a virtual interface on the host, or having some userspace processing in qemu. Similarly, pvblock can be processed in the kernel for real block devices, or in userspace for qcow format files, without the need to teach the kernel about the qcow format somehow.

Dor's initial pv devices are implemented in qemu with a view to having a faster implementation in the kernel, so userspace hypercalls are on the table now.

Thinking a little more about this, it isn't about handling hypercalls in userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The kernel handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

Quoted 9 lines · Avi Kivity

Thinking a little more about this, it isn't about handling hypercalls in userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The kernel handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

And thinking a tiny little bit more about this, we can have the kernel (optionally) fire an eventfd, so a separate userspace thread or process can be woken up to service the device, without a heavyweight exit.

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Avi Kivity

Thinking a little more about this, it isn't about handling hypercalls in userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The kernel handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

Quoted 4 lines

And thinking a tiny little bit more about this, we can have the kernel (optionally) fire an eventfd, so a separate userspace thread or process can be woken up to service the device, without a heavyweight exit.

Yes, I think this is much nicer. By "calls to ... a userspace handler" I presume you mean generating an exit to userspace with a new exit type similar to how hypercalls work today?

Quoted 19 lines · Anthony Liguori

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Avi Kivity

Thinking a little more about this, it isn't about handling hypercalls in userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The kernel handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

Quoted 4 lines

And thinking a tiny little bit more about this, we can have the kernel (optionally) fire an eventfd, so a separate userspace thread or process can be woken up to service the device, without a heavyweight exit.

Quoted 5 lines

Yes, I think this is much nicer. By "calls to ... a userspace handler" I presume you mean generating an exit to userspace with a new exit type similar to how hypercalls work today?

There are two options:
- hypercall handler sets some fields in vcpu->run and exits to userspace
- hypercall handler triggers an eventfd and returns to guest

Maybe we can unify the two by only allowing eventfd;

userspace can
attach a signal to the eventfd if it wants a synchronous exit (does eventfd allow fcntl(F_SETOWN)?)

Quoted 7 lines · Anthony Liguori

Quoted 3 lines · Avi Kivity

userspace can
attach a signal to the eventfd if it wants a synchronous exit (does eventfd allow fcntl(F_SETOWN)?)

Which would address the latency issue nicely. Looking at the fs code, it looks like eventfd shouldn't have to do anything special for it.

I'm not sure now. Which thread will be selected for accepting the signal?

if it isn't guaranteed to be the current thread, we're back with scheduler involvement, and possibly cacheline bouncing.

Quoted 8 lines · Avi Kivity

Quoted 6 lines · Anthony Liguori

Quoted 4 lines · Avi Kivity

userspace can
attach a signal to the eventfd if it wants a synchronous exit (does eventfd allow fcntl(F_SETOWN)?)

Quoted 6 lines

Quoted 3 lines

Which would address the latency issue nicely. Looking at the fs code, it looks like eventfd shouldn't have to do anything special for it.

I'm not sure now. Which thread will be selected for accepting the signal?

It's not specified.

Quoted 3 lines

if it isn't guaranteed to be the current thread, we're back with scheduler involvement, and possibly cacheline bouncing.

It's possible to use pthread_sigmask() to block the signal on all threads but one. But this would require changing the rest of the emulator; why not just select() the fd in a dedicated thread?

Quoted 8 lines · Luca

Quoted 3 lines

if it isn't guaranteed to be the current thread, we're back with scheduler involvement, and possibly cacheline bouncing.

It's possible to use pthread_sigmask() to block the signal on all threads but one. But this would require changing the rest of the emulator; why not just select() the fd in a dedicated thread?

When the guest issues that hypercall, it really wants the I/O to start. If it's a separate thread, the scheduler could choose to let the guest execute and keep the I/O thread waiting (CFS isn't likely to do that, but it's possible).

The scheduler could also choose to run the I/O thread on a different processor, and now all the data structures carefully loaded into cache by the guest need to be bounced to the processor running the I/O thread.

I think the best policy here is to cover our asses and allow userspace to choose which method it wants to use; and we should start with the guest exiting to userspace to avoid the costs I mentioned.

Quoted 1 line

Quoted 1 line

I've never really thought much about them until now. What's the

Quoted 7 lines

Quoted 7 lines

for supporting userspace hypercalls?

The current way the code works is a little scary. Hypercalls that aren't handled by kernelspace are deferred to userspace. Of course, kernelspace has no idea whether userspace is actually using a given hypercall so if kernelspace needs another one, the two may clash.

Quoted 1 line

Quoted 1 line

AFAICT, the primary reason to use hypercalls is performance. A

vmcall

Quoted 2 lines

Quoted 2 lines

is a few hundred cycles faster than a PIO exit. In the light-weight exit path, this may make a significant different. However, when

going

Quoted 1 line

Quoted 1 line

to userspace, it's not only a heavy-weight exit but it's also paying

the

Quoted 1 line

Quoted 1 line

cost of a ring transition. The few hundred cycle savings is small

Quoted 9 lines

Quoted 3 lines

comparison to the total cost so I don't think performance is a real benefit here.

Actually the heavyweight exit is much more expensive than the ring transition.

Quoted 2 lines

The hypercall namespace is much smaller than the PIO namespace, and there's no "plug-and-play" like mechanism to resolve conflict.

PIO/MMIO

Quoted 2 lines

Quoted 2 lines

has this via PCI and it seems like any userspace device ought to be either a PCI device or use a static PIO port. Plus, paravirtual

devices

Quoted 1 line

Quoted 1 line

that use PCI/PIO/MMIO are much more likely to be reusable by other

VMMs

Quoted 2 lines

Quoted 2 lines

(Xen, QEMU, even VMware).

Quoted 1 line

Quoted 1 line

In the future, if we decide a certain hypercall could be done better

in

Quoted 1 line

Quoted 1 line

userspace, and we have guests using those hypercalls, it makes sense

to

Quoted 3 lines

Quoted 3 lines

plumb the hypercalls down.

My question is, should we support userspace hypercalls until that

point?

Quoted 13 lines

Quoted 1 line

I've already mentioned this but I'll repeat it for google: allowing hypercalls to fallback to userspace gives you flexibility to have either a kernel implementation or a userspace implementation for the same functionality. This means a pvnet driver can be used either directly with a virtual interface on the host, or having some userspace processing in qemu. Similarly, pvblock can be processed in the kernel for real block devices, or in userspace for qcow format files, without the need to teach the kernel about the qcow format somehow.

Dor's initial pv devices are implemented in qemu with a view to

Quoted 3 lines

a faster implementation in the kernel, so userspace hypercalls are on the table now.

Thinking a little more about this, it isn't about handling hypercalls

userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The

handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

Quoted 3 lines

Thinking a little more about this, it isn't about handling hypercalls

in

Quoted 5 lines

userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's event channel, but assymetric) that has a channel parameter. The

kernel

Quoted 5 lines

handler for that hypercall dispatches calls to either a kernel handler or a userspace handler. That means we don't need a separate ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

Some points:
- These were none receive/send/block_send hypercalls on the first place. There were just register and notify hypercalls.

- The balloon code also uses hypercalls and let userspace handle them so higher layer will allow the guest inflate/deflate actions.