State of current pv_ops backend for KVM?

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

feel free to pick up the stuff i did - i'm somewhat "distracted" by the CFS scheduler at the moment :) You can find it at:

http://redhat.com/~mingo/kvm-paravirt-patches/kvm-patches.tar.gz

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

Quoted 6 lines · Anthony Liguori

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Quoted 3 lines

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

I think we should aim for a full direct paging implementation, where the cr3 cache is just the cherry on top that makes context switches proceed at native speed. Jun has a paper on it for OLS and I hope patches too.

A minor drawback with direct paging is that you are more or less forced to use pae on the guest, but I think that's acceptable.

Quoted 7 lines · Avi Kivity

Quoted 6 lines · Anthony Liguori

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Quoted 8 lines

Quoted 3 lines

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

I think we should aim for a full direct paging implementation, where the cr3 cache is just the cherry on top that makes context switches proceed at native speed. Jun has a paper on it for OLS and I hope patches too.

That's a bit more ambitious than I was thinking. I would be very happy with a pv timer, interrupt controller, and some basic CPU optimizations. I'm still not convinced the perf benefits of direct paging justify the annoyances that come with it (guests being aware of pfns).

This is especially true with NPT/EPT right around the corner.

Quoted 17 lines · Anthony Liguori

Quoted 7 lines · Avi Kivity

Quoted 6 lines · Anthony Liguori

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Quoted 8 lines

Quoted 3 lines

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

I think we should aim for a full direct paging implementation, where the cr3 cache is just the cherry on top that makes context switches proceed at native speed. Jun has a paper on it for OLS and I hope patches too.

Quoted 5 lines

That's a bit more ambitious than I was thinking. I would be very happy with a pv timer, interrupt controller, and some basic CPU optimizations. I'm still not convinced the perf benefits of direct paging justify the annoyances that come with it (guests being aware of pfns).

I thought that with paravirt_ops, all the awareness is limited to the paravirt implementation, and nothing in the core kernel is actually aware of host pfns.

pv timer and interrupt controller make a lot of sense, especially if steal timer functionality is included.

Quoted 1 line

This is especially true with NPT/EPT right around the corner.

It depends on the workload. It was advanced (by Ingo) that NPT/EPT increase the tlb miss latency, which is important for steady-state workloads, whereas shadow and direct paging do not. It will be interesting to measure this.

Quoted 10 lines

Quoted 6 lines

That's a bit more ambitious than I was thinking. I would be very happy with a pv timer, interrupt controller, and some basic CPU optimizations. I'm still not convinced the perf benefits of direct paging justify the annoyances that come with it (guests being aware of pfns).

I thought that with paravirt_ops, all the awareness is limited to the paravirt implementation, and nothing in the core kernel is actually aware of host pfns.

The pv_ops implementation still has to deal with guest pfn => host pfn translation. This means that a guest has to be put into a special mode for save/restore/migrate to work to ensure that no host pfn's are on the stack or in registers. Changing save/restore/migration to from guest-transparent to guest-interactive is a big loss to me.

Quoted 6 lines

Quoted 1 line

This is especially true with NPT/EPT right around the corner.

It depends on the workload. It was advanced (by Ingo) that NPT/EPT increase the tlb miss latency, which is important for steady-state workloads, whereas shadow and direct paging do not. It will be interesting to measure this.

There will most certainly be trade-offs, but I certainly don't expect direct paging to beat out NPT/EPT in certain areas.

Quoted 11 lines · Anthony Liguori

Quoted 4 lines · Avi Kivity

I think we should aim for a full direct paging implementation, where the cr3 cache is just the cherry on top that makes context switches proceed at native speed. Jun has a paper on it for OLS and I hope patches too.

That's a bit more ambitious than I was thinking. I would be very happy with a pv timer, interrupt controller, and some basic CPU optimizations. I'm still not convinced the perf benefits of direct paging justify the annoyances that come with it (guests being aware of pfns).

I thought that with paravirt_ops, all the awareness is limited to the paravirt implementation, and nothing in the core kernel is actually aware of host pfns.

Quoted 15 lines · Avi Kivity

Quoted 7 lines · Anthony Liguori

Quoted 5 lines · Avi Kivity

I think we should aim for a full direct paging implementation, where the cr3 cache is just the cherry on top that makes context switches proceed at native speed. Jun has a paper on it for OLS and I hope patches too.

Quoted 6 lines

That's a bit more ambitious than I was thinking. I would be very happy with a pv timer, interrupt controller, and some basic CPU optimizations. I'm still not convinced the perf benefits of direct paging justify the annoyances that come with it (guests being aware of pfns).

Quoted 4 lines

I thought that with paravirt_ops, all the awareness is limited to the paravirt implementation, and nothing in the core kernel is actually aware of host pfns.

It is, but now you've broken the abstraction you get other messes. You can no longer simply swap out pages behind the guest's back, copy on write or combine identical pages or other tricks without synchronizing with the guest.

It can be done,

but it's not clear to me that verifying PTEs is that
much faster than shadowing them if you're prepared to be that invasive in the guest anyway.

Quoted 4 lines · Rusty Russell

but it's not clear to me that verifying PTEs is that
much faster than shadowing them if you're prepared to be that invasive in the guest anyway.

This can probably be measured on Xen which can switch from direct to shadow mode on runtime.

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

Quoted 6 lines · Anthony Liguori

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Quoted 4 lines

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

"Me too". It's at least worth toying with the mmu batching stuff to see if that can shave some cycles there: it seems that cr3 caching is the main win for context switch so batching there won't help.

IIRC Ingo used a magic MSR to detect kvm, and there was a question mark over that.

VMI uses a different technique, which is probably worth
considering...

Cheers,
Rusty.

Quoted 8 lines · Rusty Russell

Quoted 6 lines · Anthony Liguori

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is? I know Ingo has an implementation that implements CR3 caching but I don't see any branches in Avi's git tree.

Quoted 9 lines

Quoted 4 lines

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as how the host infrastructure is there? I'd be willing to do some leg work here...

"Me too". It's at least worth toying with the mmu batching stuff to see if that can shave some cycles there: it seems that cr3 caching is the main win for context switch so batching there won't help.

I'm working on a simple patch that just adds the infrastructure.

Quoted 2 lines

IIRC Ingo used a magic MSR to detect kvm, and there was a question mark over that.

I am not too worried about using an MSR. There's really no fool proof method here. The only thing to do IMHO is move the kvm pv_ops backend into a separate file and add appropriate Kconfig stuff.

Quoted 3 lines

VMI uses a different technique, which is probably worth
considering...

VMI can determine that it's under VMware since VMware has to publish an option ROM. It's quite neat, but overkill for what we're doing (since we're not going to be using a ROM anyway).

Quoted 3 lines

VMI can determine that it's under VMware since VMware has to publish an option ROM. It's quite neat, but overkill for what we're doing (since we're not going to be using a ROM anyway).

The near to being blessed way to do this is to use a CPUID leaf (I believe it is 0x40000000) to identify the hypervisor. I don't have the exact details, but making CPUID 0x40000000 return 'KVMKVMKVMKVM' sounds like a pretty safe detection technique.

We're probably dropping the ROM probing in favor of this for 64-bit.

Zach