Shadow page table questions

Hello,

I was wondering if someone could point me to some documentation that explains the basic non-nested-paging shadow page table algorithm/strategy used by KVM. I understand that KVM caches shadow page tables across context switches and that there is a reverse mapping and page protection to help zap shadow page tables when the guest page tables change. However, I'm not entirely sure how the actual caching is done. At first I assumed that KVM would change the host CR3 on every guest context switch such that it would point to a cached shadow page table for the currently running guest user thread, however, as far as I can tell, the host CR3 does not change so I'm a little lost. If indeed it doesn't change the CR3, how does KVM solve the problem that arises when two processes in the guest OS share the same guest logical addresses?

I'm also interested in figuring out what KVM does when running with multiple virtual CPUs. Looking at the code, I can see that each VCPU has its own root pointer to a shadow page table graph, but I have yet to figure out if this graph has node's shared between VCPUs, or whether they are all private.

Quoted 3 lines · Marek Olszewski

Hello,

I was wondering if someone could point me to some documentation that explains the basic non-nested-paging shadow page table algorithm/strategy used by KVM. I understand that KVM caches shadow page tables across context switches and that there is a reverse mapping and page protection to help zap shadow page tables when the guest page tables change. However, I'm not entirely sure how the actual caching is done. At first I assumed that KVM would change the host CR3 on every guest context switch such that it would point to a cached shadow page table for the currently running guest user thread, however, as far as I can tell, the host CR3 does not change so I'm a little lost. If indeed it doesn't change the CR3, how does KVM solve the problem that arises when two processes in the guest OS share the same guest logical addresses?

The host cr3 does change, though not by using the 'mov cr3' instruction (that would cause the host to immediately switch to the guest address space, which would be bad).

See the calls to kvm_x86_ops->set_cr3().

Quoted 2 lines

I'm also interested in figuring out what KVM does when running with multiple virtual CPUs. Looking at the code, I can see that each VCPU has its own root pointer to a shadow page table graph, but I have yet to figure out if this graph has node's shared between VCPUs, or whether they are all private.

Everything is shared. If the guest is running with identical cr3s, kvm will load identical cr3s in guest mode.

An exception is when we use 32-bit pae mode. In that case, the guest cr3s will be different (but guest PDPTRs will be identical). Instead of dealing with the pae cr3, we deal with the four PDPTRs.

Thanks for the response. I've looked through the code some more and think I have figured it out now. I finally see that the root_hpa variable gets switched before entering the guest in mmu_alloc_roots, to correspond with the new cr3. Thanks again.

Perhaps you can help me with one more question. I was hoping to try out a certain change for a research project. I would like to "privatize" kvm_mmu_page's and their spe's for each guest thread running in certain designated guest processes. The goal is to give each thread its own shadow page table graphs that map the same guest logical addresses to guest physical addresses (with some changes to be introduced later). Are there any assumptions that KVM makes that will break if I do something like this? I understand that I will have to add some code throughout the mmu to make sure that these structures are synchronized when a guest thread makes a change, but I'm wondering if there is anything else. Does the reverse mapping data structure you have assume that there is only one shadow page per guest page?

It doesn't, and there are often multiple shadow pages per guest page, distinguished by their sp->role field.

Quoted 1 line

It doesn't, and there are often multiple shadow pages per guest page, distinguished by their sp->role field.

Oh, great! Does this mean that there is already a mechanism for synchronizing all shadow pages shadowing the same guest when such a guest page changes?

When using VMX without EPT, is it ever possible for a guest to receive a page fault without it first appearing (and being reinjected) in KVM?

I'm seeing some strange behavior where accesses to mprotected (but yet to be accessed) memory causes a fault in the guest OS, that I cannot see KVM intercepting.

Quoted 1 line · Marek Olszewski

When using VMX without EPT, is it ever possible for a guest to receive a page fault without it first appearing (and being reinjected) in KVM?

Yes. On Intel hosts only, and controlled by bypass_guest_pf.

Quoted 2 lines

I'm seeing some strange behavior where accesses to mprotected (but yet to be accessed) memory causes a fault in the guest OS, that I cannot see KVM intercepting.

Look for 'shadow_trap_nonpresent_pte' (which will trap into kvm) and 'shadow_notrap_nonpresent_pte' (which will not) in the code.

When a guest OS writes to a shadowed (and therefore page protected) guest page table, does the resulting page fault get handled in paging_tmpl.h:xxx_page_fault or does it call some rmap related code directly?

Also, what does the "direct" mmu page role mean?

Thanks for the fast response.

I'm trying to find the code that on a write to a guest page table entry, will iterate over all shadow page table entries that map that guest entry to update them. Can you point me to that code? I can't seem to find it myself :(

(re-adding list)

Quoted 3 lines · Marek Olszewski

Thanks for the fast response.

I'm trying to find the code that on a write to a guest page table entry, will iterate over all shadow page table entries that map that guest entry to update them. Can you point me to that code? I can't seem to find it myself :(

See kvm_mmu_pte_write().

Under VMX without EPT, I do not seeing any VM Exits due to task switches. Is there a way to enable these? I'm looking to intercept the guest whenever it does a iret.

I guess I only really care about intercepting ring 0 -> ring 3 transitions in the guest. Is there an easier way of intercepting these?

Marek

Quoted 5 lines · Avi Kivity

Quoted 1 line · Marek Olszewski

Under VMX without EPT, I do not seeing any VM Exits due to task switches. Is there a way to enable these? I'm looking to intercept the guest whenever it does a iret.

See EXIT_REASON_TASK_SWITCH. However, that won't fire on any iret, only irets that generate task switches. You can ask for exits on irets by setting CPU_BASED_VIRTUAL_NMI_PENDING and GUEST_INTR_STATE_NMI, and looking for EXIT_REASON_NMI_WINDOW.