RFC: out of sync shadow
cover RFC: out of sync shadow
Keep shadow pages temporarily out of sync, allowing more efficient guest PTE updates in comparison to trap-emulate + unprotect heuristics. Stolen from Xen :)
This version only allows leaf pagetables to go out of sync, for simplicity, but can be enhanced.
VMX "bypass_guest_pf" feature on prefetch_page breaks it (since new PTE writes need no TLB flush, I assume). Not sure if its worthwhile to convert notrap_nonpresent -> trap_nonpresent on unshadow or just go for unconditional nonpaging_prefetch_page.
* Kernel builds on 4-way 64-bit guest improve 10% (+ 3.7% for get_user_pages_fast).
* lmbench's "lat_proc fork" microbenchmark latency is 40% lower (a shadow worst scenario test).
* The RHEL3 highpte kscand hangs go from 5+ seconds to < 1 second.
* Windows 2003 Server, 32-bit PAE, DDK build (build -cPzM 3):
Windows 2003 Checked 64 Bit Build Environment, 256M RAM 1-vcpu:
vanilla + gup_fast: oos
0:04:37.375 0:03:28.047 (- 25%)
2-vcpus:
vanilla + gup_fast oos
0:02:32.000 0:01:56.031 (- 23%)
Windows 2003 Checked Build Environment, 1GB RAM
2-vcpus:
vanilla + fast_gup oos
0:02:26.078 0:01:50.110 (- 24%)
4-vcpus:
vanilla + fast_gup oos
0:01:59.266 0:01:29.625 (- 25%)
And I think other optimizations are possible now, for example the guest can be responsible for remote TLB flushing on kvm_mmu_pte_write().
Please review.
PATCH 1/13 x86/mm: get_user_pages_fast_atomic
From: Nick Piggin <[email protected]>
Provide a lockless pagetable walk function without fallback to mmap_sem on error.
arch/x86/mm/gup.c +40 -28
@@ -8,6 +8,7 @@#include <linux/mm.h> #include <linux/vmstat.h> #include <linux/highmem.h> +#include <linux/module.h> #include <asm/pgtable.h>
@@ -219,7 +220,7 @@ static int gup_pud_range(pgd_t pgd, unsireturn 1 ; } -int get_user_pages_fast (unsigned long start,int nr_pages,int write, +int get_user_pages_fast_atomic (unsigned long start,int nr_pages,int write,struct page **pages) {struct mm_struct *mm = current->mm;
@@ -234,7 +235,7 @@ int get_user_pages_fast(unsigned long st end = start + len;if (unlikely (!access_ok (write ?VERIFY_WRITE :VERIFY_READ , start, len))) - goto slow_irqon; +return -EFAULT ;/* * XXX: batch / limit 'nr', to avoid large irq off latency
@@ -261,38 +262,49 @@ int get_user_pages_fast(unsigned long st next =pgd_addr_end (addr, end);if (pgd_none (pgd)) - goto slow; + goto out_short;if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) - goto slow; + goto out_short; }while (pgdp++, addr = next, addr != end); -local_irq_enable ();VM_BUG_ON (nr != (end - start) >>PAGE_SHIFT ); +out_short: +local_irq_enable ();return nr; +} - { -int ret; - -slow: -local_irq_enable (); -slow_irqon: -/* Try to get the remaining pages with get_user_pages */ - start += nr <<PAGE_SHIFT ; - pages += nr; - -down_read (&mm->mmap_sem); - ret =get_user_pages (current, mm, start, - (end - start) >>PAGE_SHIFT , write,0 , pages,NULL ); -up_read (&mm->mmap_sem); - -/* Have to be a bit careful with return values */ -if (nr >0 ) { -if (ret <0 ) - ret = nr; -else - ret += nr; - } +int get_user_pages_fast (unsigned long start,int nr_pages,int write, +struct page **pages) +{ +struct mm_struct *mm = current->mm; +int nr =0 ; +int ret; -return ret; + nr =get_user_pages_fast_atomic (start, nr_pages, write, pages); +if (likely (nr == nr_pages)) +return nr; + +if (unlikely (nr <0 )) +return nr; + + start += nr <<PAGE_SHIFT ; + pages += nr; + nr_pages -= nr; + +down_read (&mm->mmap_sem); + ret =get_user_pages (current, mm, start, + nr_pages, write,0 , pages,NULL ); +up_read (&mm->mmap_sem); + +/* Have to be a bit careful with return values */ +if (nr >0 ) { +if (ret <0 ) + ret = nr; +else + ret += nr; } + +return ret; } + +EXPORT_SYMBOL_GPL (get_user_pages_fast_atomic);
include/asm-x86/uaccess.h +3 -0
@@ -443,6 +443,9 @@ extern struct movsl_mask {#define ARCH_HAS_NOCACHE_UACCESS 1 +int get_user_pages_fast_atomic (unsigned long start,int nr_pages,int write, +struct page **pages); +#ifdef CONFIG_X86_32 # include "uaccess_32.h" #else
Re: [patch 01/13] x86/mm: get_user_pages_fast_atomic
Quoted 5 lines · Marcelo Tosatti
From: Nick Piggin <[email protected]>
Provide a lockless pagetable walk function without fallback to
mmap_semon error.
I would like to avoid this if possible. Not only is this a change to the core (with backporting headaches), if we resync in atomic context this can mean a long time spent with preemption disabled.
We might get around the need by dropping the lock when we resync, fetch the gfns without the lock, and after reacquiring it check whether we can proceed or whether we need to abort and let the guest retry. We can probably proceed unless one of two things have happened: an mmu page was zapped, or out page was oos'ed while we were resyncing it.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 01/13] x86/mm: get_user_pages_fast_atomic
Quoted 9 lines · Avi Kivity
Quoted 5 lines · Marcelo Tosatti
From: Nick Piggin <[email protected]>
Provide a lockless pagetable walk function without fallback to
mmap_semon error.I would like to avoid this if possible. Not only is this a change to the core (with backporting headaches),
Chris mentioned that the backport could use down_read_trylock(mmap_sem), and zap the page on failure. Its a simple solution and it should be rare for mmap_sem to be acquired in write mode.
Quoted 2 lines
if we resync in atomic context
this can mean a long time spent with preemption disabled.
The resync time for a single page is comparable to prefetch_page (note that prefetch_page with direct access via gfn_to_page_atomic is about 50% faster than the current one) plus the gfn->pfn pagetable walks.
It could simply resched based on need_resched after each page synced. Would that cover your concern?
BTW, it might be interesting to spin_needbreak after resyncing a certain number of pages.
Quoted 5 lines
We might get around the need by dropping the lock when we resync, fetch the gfns without the lock, and after reacquiring it check whether we can proceed or whether we need to abort and let the guest retry. We can probably proceed unless one of two things have happened: an mmu page was zapped, or out page was oos'ed while we were resyncing it.
This sounds more complicated. First you have to grab the lock twice for each page synced. Secondly, the abort case due to oos'ed while resyncing means the page has to be zapped.
Re: [patch 01/13] x86/mm: get_user_pages_fast_atomic
Quoted 10 lines · Marcelo Tosatti
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Marcelo Tosatti
From: Nick Piggin <[email protected]>
Provide a lockless pagetable walk function without fallback to
mmap_semon error.
Quoted 7 lines
Quoted 2 lines
I would like to avoid this if possible. Not only is this a change to the core (with backporting headaches),
Chris mentioned that the backport could use
down_read_trylock(mmap_sem), and zap the page on failure. Its a simple solution and it should be rare formmap_semto be acquired in write mode.
Yes. Clever.
Quoted 7 lines
Quoted 2 lines
if we resync in atomic context this can mean a long time spent with preemption disabled.
The resync time for a single page is comparable to
prefetch_page(note thatprefetch_pagewith direct access viagfn_to_page_atomicis about 50% faster than the current one) plus thegfn->pfnpagetable walks.
These could be very expensive as the gfn->pfn mapping is essentially random and too big to be cached. 512 cache misses is a lot of time - perhaps upwards of 50 microseconds.
Snapshotting is a must here IMO -- and it avoids the need for a walk completely.
Quoted 4 lines
It could simply resched based on
need_reschedafter each page synced. Would that cover your concern?
I guess it's better than nothing.
Quoted 11 lines
BTW, it might be interesting to
spin_needbreakafter resyncing a certain number of pages.Quoted 2 lines
We might get around the need by dropping the lock when we resync, fetch the gfns without the lock, and after reacquiring it check whether we can proceed or whether we need to abort and let the guest retry. We can probably proceed unless one of two things have happened: an mmu page was zapped, or out page was oos'ed while we were resyncing it.
This sounds more complicated. First you have to grab the lock twice for each page synced. Secondly, the abort case due to oos'ed while resyncing means the page has to be zapped.
It is complicated, yes.
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH 2/13 KVM: MMU: switch to get_user_pages_fast
In reply to Marcelo TosattiAvoid mmap_sem / pt lock acquision if the pagetables are present. The improvement for hugepage backed guests is more significant, since pte walk + page grab for such mappings is serialized by mm->page_table_lock.
arch/x86/kvm/mmu.c +9 -14
@@ -405,16 +405,19 @@ static int host_largepage_backed(struct {struct vm_area_struct *vma;unsigned long addr; +int ret =0 ; addr =gfn_to_hva (kvm, gfn);if (kvm_is_error_hva (addr)) -return 0 ; +return ret; +down_read (¤t->mm->mmap_sem); vma =find_vma (current->mm, addr);if (vma &&is_vm_hugetlb_page (vma)) -return 1 ; + ret =1 ; +up_read (¤t->mm->mmap_sem); -return 0 ; +return ret; }static int is_largepage_backed (struct kvm_vcpu *vcpu,gfn_t large_gfn)
@@ -1136,9 +1139,7 @@ struct page *gva_to_page(struct kvm_vcpuif (gpa ==UNMAPPED_GVA )return NULL ; -down_read (¤t->mm->mmap_sem); page =gfn_to_page (vcpu->kvm, gpa >>PAGE_SHIFT ); -up_read (¤t->mm->mmap_sem);return page; }
@@ -1326,16 +1327,14 @@ static int nonpaging_map(struct kvm_vcpupfn_t pfn;unsigned long mmu_seq; -down_read (¤t->mm->mmap_sem);if (is_largepage_backed (vcpu, gfn & ~(KVM_PAGES_PER_HPAGE -1 ))) { gfn &= ~(KVM_PAGES_PER_HPAGE -1 ); largepage =1 ; } mmu_seq = vcpu->kvm->mmu_notifier_seq; -/* implicit mb(), we'll read before PT lock is unlocked */ +smp_rmb (); pfn =gfn_to_pfn (vcpu->kvm, gfn); -up_read (¤t->mm->mmap_sem);/* mmio */ if (is_error_pfn (pfn)) {
@@ -1484,15 +1483,13 @@ static int tdp_page_fault(struct kvm_vcpif (r)return r; -down_read (¤t->mm->mmap_sem);if (is_largepage_backed (vcpu, gfn & ~(KVM_PAGES_PER_HPAGE -1 ))) { gfn &= ~(KVM_PAGES_PER_HPAGE -1 ); largepage =1 ; } mmu_seq = vcpu->kvm->mmu_notifier_seq; -/* implicit mb(), we'll read before PT lock is unlocked */ +smp_rmb (); pfn =gfn_to_pfn (vcpu->kvm, gfn); -up_read (¤t->mm->mmap_sem);if (is_error_pfn (pfn)) {kvm_release_pfn_clean (pfn);return 1 ;
@@ -1805,15 +1802,13 @@ static void mmu_guess_page_from_pte_writreturn ; gfn = (gpte &PT64_BASE_ADDR_MASK ) >>PAGE_SHIFT ; -down_read (¤t->mm->mmap_sem);if (is_large_pte (gpte) &&is_largepage_backed (vcpu, gfn)) { gfn &= ~(KVM_PAGES_PER_HPAGE -1 ); vcpu->arch.update_pte.largepage =1 ; } vcpu->arch.update_pte.mmu_seq = vcpu->kvm->mmu_notifier_seq; -/* implicit mb(), we'll read before PT lock is unlocked */ +smp_rmb (); pfn =gfn_to_pfn (vcpu->kvm, gfn); -up_read (¤t->mm->mmap_sem);if (is_error_pfn (pfn)) {kvm_release_pfn_clean (pfn);
arch/x86/kvm/paging_tmpl.h +1 -7
@@ -102,14 +102,10 @@ static bool FNAME(cmpxchg_gpte)(struct kpt_element_t *table;struct page *page; -down_read (¤t->mm->mmap_sem); page =gfn_to_page (kvm, table_gfn); -up_read (¤t->mm->mmap_sem); table =kmap_atomic (page,KM_USER0 ); - ret =CMPXCHG (&table[index], orig_pte, new_pte); -kunmap_atomic (table,KM_USER0 );kvm_release_page_dirty (page);
@@ -418,7 +414,6 @@ static int FNAME(page_fault)(struct kvm_return 0 ; } -down_read (¤t->mm->mmap_sem);if (walker.level ==PT_DIRECTORY_LEVEL ) {gfn_t large_gfn; large_gfn = walker.gfn & ~(KVM_PAGES_PER_HPAGE -1 );
@@ -428,9 +423,8 @@ static int FNAME(page_fault)(struct kvm_ } } mmu_seq = vcpu->kvm->mmu_notifier_seq; -/* implicit mb(), we'll read before PT lock is unlocked */ +smp_rmb (); pfn =gfn_to_pfn (vcpu->kvm, walker.gfn); -up_read (¤t->mm->mmap_sem);/* mmio */ if (is_error_pfn (pfn)) {
arch/x86/kvm/vmx.c +0 -4
@@ -2010,9 +2010,7 @@ static int alloc_apic_access_page(structif (r) goto out; -down_read (¤t->mm->mmap_sem); kvm->arch.apic_access_page =gfn_to_page (kvm,0xfee00 ); -up_read (¤t->mm->mmap_sem); out:up_write (&kvm->slots_lock);return r;
@@ -2034,10 +2032,8 @@ static int alloc_identity_pagetable(struif (r) goto out; -down_read (¤t->mm->mmap_sem); kvm->arch.ept_identity_pagetable =gfn_to_page (kvm,VMX_EPT_IDENTITY_PAGETABLE_ADDR >>PAGE_SHIFT ); -up_read (¤t->mm->mmap_sem); out:up_write (&kvm->slots_lock);return r;
arch/x86/kvm/x86.c +0 -6
@@ -932,10 +932,8 @@ int kvm_set_msr_common(struct kvm_vcpu */* ...but clean it before doing the actual write */ vcpu->arch.time_offset = data & ~(PAGE_MASK |1 ); -down_read (¤t->mm->mmap_sem); vcpu->arch.time_page =gfn_to_page (vcpu->kvm, data >>PAGE_SHIFT ); -up_read (¤t->mm->mmap_sem);if (is_error_page (vcpu->arch.time_page)) {kvm_release_page_clean (vcpu->arch.time_page);
@@ -2305,9 +2303,7 @@ static int emulator_cmpxchg_emulated(uns val = *(u64 *)new; -down_read (¤t->mm->mmap_sem); page =gfn_to_page (vcpu->kvm, gpa >>PAGE_SHIFT ); -up_read (¤t->mm->mmap_sem); kaddr =kmap_atomic (page,KM_USER0 );set_64bit ((u64 *)(kaddr +offset_in_page (gpa)), val);
@@ -3077,9 +3073,7 @@ static void vapic_enter(struct kvm_vcpuif (!apic || !apic->vapic_addr)return ; -down_read (¤t->mm->mmap_sem); page =gfn_to_page (vcpu->kvm, apic->vapic_addr >>PAGE_SHIFT ); -up_read (¤t->mm->mmap_sem); vcpu->arch.apic->vapic_page = page; }
virt/kvm/kvm_main.c +17 -12
@@ -716,9 +716,6 @@ unsigned long gfn_to_hva(struct kvm *kvm }EXPORT_SYMBOL_GPL (gfn_to_hva); -/* -* Requires current->mm->mmap_sem to be held -*/ pfn_t gfn_to_pfn (struct kvm *kvm,gfn_t gfn) {struct page *page[1 ];
@@ -734,20 +731,23 @@ pfn_t gfn_to_pfn(struct kvm *kvm, gfn_treturn page_to_pfn (bad_page); } - npages =get_user_pages (current, current->mm, addr,1 ,1 ,0 , page, -NULL ); + npages =get_user_pages_fast (addr,1 ,1 , page);if (unlikely (npages !=1 )) {struct vm_area_struct *vma; +down_read (¤t->mm->mmap_sem); vma =find_vma (current->mm, addr); +if (vma ==NULL || addr < vma->vm_start || !(vma->vm_flags &VM_PFNMAP )) { +up_read (¤t->mm->mmap_sem);get_page (bad_page);return page_to_pfn (bad_page); } pfn = ((addr - vma->vm_start) >>PAGE_SHIFT ) + vma->vm_pgoff; +up_read (¤t->mm->mmap_sem);BUG_ON (!is_mmio_pfn (pfn)); }else pfn =page_to_pfn (page[0 ]);
@@ -1387,17 +1387,22 @@ out:static int kvm_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) { +struct page *page[1 ]; +unsigned long addr; +int npages; +gfn_t gfn = vmf->pgoff;struct kvm *kvm = vma->vm_file->private_data; -struct page *page; -if (!kvm_is_visible_gfn (kvm, vmf->pgoff)) + addr =gfn_to_hva (kvm, gfn); +if (kvm_is_error_hva (addr))return VM_FAULT_SIGBUS ; - page =gfn_to_page (kvm, vmf->pgoff); -if (is_error_page (page)) { -kvm_release_page_clean (page); + + npages =get_user_pages (current, current->mm, addr,1 ,1 ,0 , page, +NULL ); +if (unlikely (npages !=1 ))return VM_FAULT_SIGBUS ; - } - vmf->page = page; + + vmf->page = page[0 ];return 0 ; }
Re: [patch 02/13] KVM: MMU: switch to get_user_pages_fast
Quoted 7 lines · Marcelo Tosatti
Avoid
mmap_sem/ pt lock acquision if the pagetables are present. The improvement for hugepage backed guests is more significant, since pte walk + page grab for such mappings is serialized bymm->page_table_lock.CC Andrea Arcangeli <[email protected]>
I'd like to apply this. Andrea, can you verify that mmu notifiers interaction is okay?
Quoted 20 lines
static int
kvm_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf){virt/kvm/kvm_main.c +7
-5+struct page *page[1 ]; +unsigned long addr; +int npages; +gfn_t gfn = vmf->pgoff;struct kvm *kvm = vma->vm_file->private_data; -struct page *page; -if (!kvm_is_visible_gfn (kvm, vmf->pgoff)) + addr =gfn_to_hva (kvm, gfn); +if (kvm_is_error_hva (addr))return VM_FAULT_SIGBUS ; - page =gfn_to_page (kvm, vmf->pgoff); -if (is_error_page (page)) { -kvm_release_page_clean (page); +
virt/kvm/kvm_main.c out: · line 1407
Quoted 10 lines
virt/kvm/kvm_main.c +5
-2+ npages =get_user_pages (current, current->mm, addr,1 ,1 ,0 , page, +NULL ); +if (unlikely (npages !=1 ))return VM_FAULT_SIGBUS ; - } - vmf->page = page; + + vmf->page = page[0 ];return 0 ; }
Why this change?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 02/13] KVM: MMU: switch to get_user_pages_fast
Quoted 34 lines · Avi Kivity
Quoted 7 lines · Marcelo Tosatti
Avoid
mmap_sem/ pt lock acquision if the pagetables are present. The improvement for hugepage backed guests is more significant, since pte walk + page grab for such mappings is serialized bymm->page_table_lock.CC Andrea Arcangeli <[email protected]>I'd like to apply this. Andrea, can you verify that mmu notifiers interaction is okay?
Quoted 22 lines
static int kvm_vm_fault(struct
vm_area_struct*vma, structvm_fault
*vmf)
{virt/kvm/kvm_main.c +10
-5+struct page *page[1 ]; +unsigned long addr; +int npages; +gfn_t gfn = vmf->pgoff;struct kvm *kvm = vma->vm_file->private_data; -struct page *page; - if (!kvm_is_visible_gfn(kvm, vmf->pgoff)) + addr = gfn_to_hva(kvm, gfn); +if (kvm_is_error_hva (addr))return VM_FAULT_SIGBUS ; - page =gfn_to_page (kvm, vmf->pgoff); -if (is_error_page (page)) { -kvm_release_page_clean (page); + + npages =get_user_pages (current, current->mm, addr,1 ,1 ,0 , page, +NULL ); +if (unlikely (npages !=1 ))return VM_FAULT_SIGBUS ; - }
Quoted 8 lines
virt/kvm/kvm_main.c out: · line 1407
Quoted 6 lines
virt/kvm/kvm_main.c +2
-1- vmf->page = page; + + vmf->page = page[0 ];return 0 ; }Why this change?
Because get_user_pages_fast grabs mmap_sem if necessary, but ->vm_fault already holds it.
Deadlock:
CPU0 CPU1
down_read(mmap_sem)kvm_vm_fault
down_write(mmap_sem)gfn_to_pageget_user_pages_fastdown_read(mmap_sem)
Re: [patch 02/13] KVM: MMU: switch to get_user_pages_fast
Quoted 33 lines · Marcelo Tosatti
Quoted 23 lines
Quoted 23 lines
static int kvm_vm_fault(struct
vm_area_struct*vma, structvm_fault
*vmf)
{virt/kvm/kvm_main.c +10
-5+struct page *page[1 ]; +unsigned long addr; +int npages; +gfn_t gfn = vmf->pgoff;struct kvm *kvm = vma->vm_file->private_data; -struct page *page; - if (!kvm_is_visible_gfn(kvm, vmf->pgoff)) + addr = gfn_to_hva(kvm, gfn); +if (kvm_is_error_hva (addr))return VM_FAULT_SIGBUS ; - page =gfn_to_page (kvm, vmf->pgoff); -if (is_error_page (page)) { -kvm_release_page_clean (page); + + npages =get_user_pages (current, current->mm, addr,1 ,1 ,0 , page, +NULL ); +if (unlikely (npages !=1 ))return VM_FAULT_SIGBUS ; - }Quoted 9 lines
virt/kvm/kvm_main.c out: · line 1407
Quoted 7 lines
virt/kvm/kvm_main.c +2
-1- vmf->page = page; + + vmf->page = page[0 ];return 0 ; }Why this change?
Quoted 4 lines
Because
get_user_pages_fastgrabsmmap_semif necessary, but ->vm_fault already holds it.
Right. I think it merits its own patch (and changelog entry) since it differs in spirit from the rest.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 02/13] KVM: MMU: switch to get_user_pages_fast
In reply to Avi KivityQuoted 11 lines · Avi Kivity
Quoted 7 lines · Marcelo Tosatti
Avoid
mmap_sem/ pt lock acquision if the pagetables are present. The improvement for hugepage backed guests is more significant, since pte walk + page grab for such mappings is serialized bymm->page_table_lock.CC Andrea Arcangeli <[email protected]>I'd like to apply this. Andrea, can you verify that mmu notifiers interaction is okay?
Looks great to me. Thanks!
Re: [patch 02/13] KVM: MMU: switch to get_user_pages_fast
Quoted 8 lines · Andrea Arcangeli
Quoted 6 lines · Avi Kivity
Quoted 4 lines · Marcelo Tosatti
Avoid
mmap_sem/ pt lock acquision if the pagetables are present. The improvement for hugepage backed guests is more significant, since pte walk + page grab for such mappings is serialized bymm->page_table_lock.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
CC Andrea Arcangeli <[email protected]>I'd like to apply this. Andrea, can you verify that mmu notifiers interaction is okay?
Looks great to me. Thanks!
Thanks.
Marcelo, can you split the patch into two and repost? We can get this in quickly.
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH 3/13 KVM: MMU: gfn_to_page_atomic
In reply to Marcelo TosattiKVM wrapper around get_user_pages_fast_atomic.
include/linux/kvm_host.h +1 -0
@@ -177,6 +177,7 @@ int kvm_arch_set_memory_region(struct kvvoid kvm_arch_flush_shadow (struct kvm *kvm);gfn_t unalias_gfn (struct kvm *kvm,gfn_t gfn);struct page *gfn_to_page (struct kvm *kvm,gfn_t gfn); +struct page *gfn_to_page_atomic (struct kvm *kvm,gfn_t gfn);unsigned long gfn_to_hva (struct kvm *kvm,gfn_t gfn);void kvm_release_page_clean (struct page *page);void kvm_release_page_dirty (struct page *page);
virt/kvm/kvm_main.c +24 -0
@@ -773,6 +773,30 @@ struct page *gfn_to_page(struct kvm *kvmEXPORT_SYMBOL_GPL (gfn_to_page); +struct page *gfn_to_page_atomic (struct kvm *kvm,gfn_t gfn) +{ +struct page *page[1 ]; +unsigned long addr; +int npages; + + addr =gfn_to_hva (kvm, gfn); +if (kvm_is_error_hva (addr)) { +get_page (bad_page); +return bad_page; + } + + npages =get_user_pages_fast_atomic (addr,1 ,1 , page); + +if (unlikely (npages !=1 )) { +get_page (bad_page); +return bad_page; + } + +return page[0 ]; +} + +EXPORT_SYMBOL_GPL (gfn_to_page_atomic); +void kvm_release_page_clean (struct page *page) {kvm_release_pfn_clean (page_to_pfn (page));
PATCH 4/13 KVM: MMU: switch prefetch_page to gfn_to_page_atomic
In reply to Marcelo TosattiClose to 50% improvement for prefetch_page.
arch/x86/kvm/paging_tmpl.h +21 -15
@@ -477,9 +477,10 @@ static gpa_t FNAME(gva_to_gpa)(struct kvstatic void FNAME (prefetch_page )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) { -int i, j, offset, r; -pt_element_t pt[256 /sizeof (pt_element_t)]; -gpa_t pte_gpa; +int i; +struct page *page; +pt_element_t *pt; +void *gpte_kaddr;if (sp->role.metaphysical || (PTTYPE ==32 && sp->role.level >PT_PAGE_TABLE_LEVEL )) {
@@ -487,21 +488,26 @@ static void FNAME(prefetch_page)(structreturn ; } - pte_gpa =gfn_to_gpa (sp->gfn); -if (PTTYPE ==32 ) { - offset = sp->role.quadrant <<PT64_LEVEL_BITS ; - pte_gpa += offset *sizeof (pt_element_t); + page =gfn_to_page_atomic (vcpu->kvm, sp->gfn); +if (is_error_page (page)) { +nonpaging_prefetch_page (vcpu, sp); +kvm_release_page_clean (page); +return ; } + gpte_kaddr = pt =kmap_atomic (page,KM_USER0 ); + +if (PTTYPE ==32 ) + pt += sp->role.quadrant <<PT64_LEVEL_BITS ; -for (i =0 ; i <PT64_ENT_PER_PAGE ; i +=ARRAY_SIZE (pt)) { - r =kvm_read_guest_atomic (vcpu->kvm, pte_gpa, pt,sizeof pt); - pte_gpa +=ARRAY_SIZE (pt) *sizeof (pt_element_t); -for (j =0 ; j <ARRAY_SIZE (pt); ++j) -if (r ||is_present_pte (pt[j])) - sp->spt[i+j] = shadow_trap_nonpresent_pte; -else - sp->spt[i+j] = shadow_notrap_nonpresent_pte; +for (i =0 ; i <PT64_ENT_PER_PAGE ; i++) { +if (is_present_pte (*pt)) + sp->spt[i] = shadow_trap_nonpresent_pte; +else + sp->spt[i] = shadow_notrap_nonpresent_pte; + pt++; } +kunmap_atomic (gpte_kaddr,KM_USER0 ); +kvm_release_page_clean (page); }#undef pt_element_t
PATCH 5/13 KVM: MMU: do not write-protect large mappings
In reply to Marcelo TosattiThere is not much point in write protecting large mappings. This can only happen when a page is shadowed during the window between is_largepage_backed and mmu_lock acquision. Zap the entry instead, so the next pagefault will find a shadowed page via is_largepage_backed and fallback to 4k translations.
Simplifies out of sync shadow.
arch/x86/kvm/mmu.c +9 -2
@@ -1210,8 +1210,7 @@ static void mmu_set_spte(struct kvm_vcpu spte |=PT_WRITABLE_MASK ; shadow =kvm_mmu_lookup_page (vcpu->kvm, gfn); -if (shadow || - (largepage &&has_wrprotected_page (vcpu->kvm, gfn))) { +if (shadow) {pgprintk ("%s: found shadow page for %lx, marking ro\n" , __func__, gfn); pte_access &= ~ACC_WRITE_MASK ;
@@ -1222,6 +1221,14 @@ static void mmu_set_spte(struct kvm_vcpuif (write_fault) *ptwrite =1 ; } +/* +* Do not create write protected large translations. +*/ +if (largepage &&has_wrprotected_page (vcpu->kvm, gfn)) { + spte = shadow_trap_nonpresent_pte; + was_writeble =0 ; + *ptwrite =0 ; + } }if (pte_access &ACC_WRITE_MASK )
arch/x86/kvm/paging_tmpl.h +2 -3
@@ -307,11 +307,10 @@ static int FNAME(shadow_walk_entry)(strureturn 1 ; } -if (is_shadow_present_pte (*sptep) && !is_large_pte (*sptep)) +if (is_shadow_present_pte (*sptep))return 0 ; -if (is_large_pte (*sptep)) -rmap_remove (vcpu->kvm, sptep); +WARN_ON (is_large_pte (*sptep));if (level ==PT_DIRECTORY_LEVEL && gw->level ==PT_DIRECTORY_LEVEL ) { metaphysical =1 ;
Re: [patch 05/13] KVM: MMU: do not write-protect large mappings
Quoted 13 lines · Marcelo Tosatti
There is not much point in write protecting large mappings. This can only happen when a page is shadowed during the window between
is_largepage_backedandmmu_lockacquision. Zap the entry instead, so the next pagefault will find a shadowed page viais_largepage_backedand fallback to 4k translations.Simplifies out of sync shadow.
arch/x86/kvm/mmu.c +1
-0@@ -1222,6 +1221,14 @@ static void mmu_set_spte(struct kvm_vcpuif (write_fault) *ptwrite =1 ; } +/*
arch/x86/kvm/mmu.c static void mmu_set_spte(struct kvm_vcpu · line 1234
Quoted 10 lines
arch/x86/kvm/mmu.c +7
-0+* Do not create write protected large translations. +*/ +if (largepage &&has_wrprotected_page (vcpu->kvm, gfn)) { + spte = shadow_trap_nonpresent_pte; + was_writeble =0 ; + *ptwrite =0 ; + } }
Why are you clearing was_writable?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 05/13] KVM: MMU: do not write-protect large mappings
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Marcelo Tosatti
There is not much point in write protecting large mappings. This can only happen when a page is shadowed during the window between
is_largepage_backedandmmu_lockacquision. Zap the entry instead, so the next pagefault will find a shadowed page viais_largepage_backedand fallback to 4k translations.Simplifies out of sync shadow.
Quoted 16 lines
Quoted 4 lines
arch/x86/kvm/mmu.c +0
-0@@ -1222,6 +1221,14 @@ static void mmu_set_spte(struct kvm_vcpuif (write_fault) *ptwrite =1 ; }arch/x86/kvm/mmu.c static void mmu_set_spte(struct kvm_vcpu · line 1233
Quoted 10 lines
arch/x86/kvm/mmu.c +8
-0+/* +* Do not create write protected large translations. +*/ +if (largepage &&has_wrprotected_page (vcpu->kvm, gfn)) { + spte = shadow_trap_nonpresent_pte; + was_writeble =0 ; + *ptwrite =0 ; + } }Why are you clearing
was_writable?
No idea. Its wrong.
PATCH 6/13 KVM: MMU: global page keeping
In reply to Marcelo TosattiAccount pages that only have global entries. Such pages need to be synced on cr4 writes, but not cr3 writes.
arch/x86/kvm/mmu.c +8 -3
@@ -778,6 +778,7 @@ static struct kvm_mmu_page *kvm_mmu_alloASSERT (is_empty_shadow_page (sp->spt)); sp->slot_bitmap =0 ; sp->multimapped =0 ; + sp->flags = (1 << KVM_PG_global); sp->parent_pte = parent_pte; --vcpu->kvm->arch.n_free_mmu_pages;return sp;
@@ -1147,18 +1148,22 @@ struct page *gva_to_page(struct kvm_vcpustatic void mmu_set_spte (struct kvm_vcpu *vcpu,u64 *shadow_pte,unsigned pt_access,unsigned pte_access,int user_fault,int write_fault,int dirty, -int *ptwrite,int largepage,gfn_t gfn, -pfn_t pfn,bool speculative) +int global,int *ptwrite,int largepage, +gfn_t gfn,pfn_t pfn,bool speculative) {u64 spte;int was_rmapped =0 ;int was_writeble =is_writeble_pte (*shadow_pte); +struct kvm_mmu_page *sp =page_header (__pa (shadow_pte));pgprintk ("%s: spte %llx access %x write_fault %d" " user_fault %d gfn %lx\n" , __func__, *shadow_pte, pt_access, write_fault, user_fault, gfn); +if (!global) +kvm_clear_pg_global (sp); +if (is_rmap_pte (*shadow_pte)) {/* * If we overwrite a PTE page pointer with a 2MB PMD, unlink
@@ -1285,7 +1290,7 @@ static int direct_map_entry(struct kvm_sif (level ==PT_PAGE_TABLE_LEVEL || (walk->largepage && level ==PT_DIRECTORY_LEVEL )) {mmu_set_spte (vcpu, sptep,ACC_ALL ,ACC_ALL , -0 , walk->write,1 , &walk->pt_write, +0 , walk->write,1 ,0 , &walk->pt_write, walk->largepage, gfn, walk->pfn, false); ++vcpu->stat.pf_fixed;return 1 ;
include/asm-x86/kvm_host.h +17 -0
@@ -199,6 +199,7 @@ struct kvm_mmu_page {u64 *parent_pte;/* !multimapped */ struct hlist_head parent_ptes;/* multimapped, kvm_pte_chain */ }; +unsigned long flags; };struct kvm_pv_mmu_op_buffer {
@@ -757,4 +758,20 @@ asmlinkage void kvm_handle_fault_on_reboint kvm_unmap_hva (struct kvm *kvm,unsigned long hva);int kvm_age_hva (struct kvm *kvm,unsigned long hva); +enum kvm_page_flags { + KVM_PG_global, +}; + +#define KVMPGFLAG (name) \ +static inline int kvm_page_##name(struct kvm_mmu_page *sp) \ + { return test_bit(KVM_PG_##name, &sp->flags); } \ +static inline void kvm_set_pg_##name(struct kvm_mmu_page *sp) \ + { set_bit(KVM_PG_##name, &sp->flags); } \ +static inline void kvm_clear_pg_##name(struct kvm_mmu_page *sp) \ + { clear_bit(KVM_PG_##name, &sp->flags); } \ +static inline int kvm_test_clear_pg_##name(struct kvm_mmu_page *sp) \ + { return test_and_clear_bit(KVM_PG_##name, &sp->flags); } + +KVMPGFLAG (global); +#endif
arch/x86/kvm/paging_tmpl.h +3 -2
@@ -274,8 +274,8 @@ static void FNAME(update_pte)(struct kvmreturn ;kvm_get_pfn (pfn);mmu_set_spte (vcpu, spte, page->role.access, pte_access,0 ,0 , - gpte &PT_DIRTY_MASK ,NULL , largepage,gpte_to_gfn (gpte), - pfn, true); + gpte &PT_DIRTY_MASK , gpte &PT_GLOBAL_MASK ,NULL , + largepage,gpte_to_gfn (gpte), pfn, true); }/*
@@ -301,6 +301,7 @@ static int FNAME(shadow_walk_entry)(strummu_set_spte (vcpu, sptep, access, gw->pte_access & access, sw->user_fault, sw->write_fault, gw->ptes[gw->level-1 ] &PT_DIRTY_MASK , + gw->ptes[gw->level-1 ] &PT_GLOBAL_MASK , sw->ptwrite, sw->largepage, gw->gfn, sw->pfn, false); sw->sptep = sptep;
Re: [patch 06/13] KVM: MMU: global page keeping
arch/x86/kvm/mmu.c struct page *gva_to_page(struct kvm_vcpu · line 1153
Quoted 12 lines · Marcelo Tosatti
Account pages that only have global entries. Such pages need to be synced on cr4 writes, but not cr3 writes.
arch/x86/kvm/mmu.c +2
-2@@ -1147,18 +1148,22 @@ struct page *gva_to_page(struct kvm_vcpustatic void mmu_set_spte (struct kvm_vcpu *vcpu,u64 *shadow_pte,unsigned pt_access,unsigned pte_access,int user_fault,int write_fault,int dirty, -int *ptwrite,int largepage,gfn_t gfn, -pfn_t pfn,bool speculative) +int global,int *ptwrite,int largepage, +gfn_t gfn,pfn_t pfn,bool speculative)
bool global
One option to stop the ever-increasing m_s_s() parameter list is to put it into pte_access. Not sure if it's a good idea.
include/asm-x86/kvm_host.h struct kvm_mmu_page { · line 204
Quoted 10 lines
===================================================================
include/asm-x86/kvm_host.h +1
-0@@ -199,6 +199,7 @@ struct kvm_mmu_page {u64 *parent_pte;/* !multimapped */ struct hlist_head parent_ptes;/* multimapped, kvm_pte_chain */ }; +unsigned long flags; };
bool global;
(or: bool global : 1; if we run out of space)
Quoted 10 lines
include/asm-x86/kvm_host.h +0
-0int kvm_unmap_hva (struct kvm *kvm,unsigned long hva);int
kvm_age_hva(struct kvm *kvm, unsigned long hva);include/asm-x86/kvm_host.h +6
-0+enum kvm_page_flags { + KVM_PG_global, +}; + +#define KVMPGFLAG (name) \ +static inline int kvm_page_##name(struct kvm_mmu_page *sp) \
include/asm-x86/kvm_host.h asmlinkage void kvm_handle_fault_on_rebo · line 776
Quoted 10 lines
include/asm-x86/kvm_host.h +10
-0+ { return test_bit(KVM_PG_##name, &sp->flags); } \ +static inline void kvm_set_pg_##name(struct kvm_mmu_page *sp) \ + { set_bit(KVM_PG_##name, &sp->flags); } \ +static inline void kvm_clear_pg_##name(struct kvm_mmu_page *sp) \ + { clear_bit(KVM_PG_##name, &sp->flags); } \ +static inline int kvm_test_clear_pg_##name(struct kvm_mmu_page *sp) \ + { return test_and_clear_bit(KVM_PG_##name, &sp->flags); } + +KVMPGFLAG (global); +
Ugh. Note you're using locked operations even though the structure is protected by the mmu lock.
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH 7/13 KVM: MMU: mode specific sync_page
In reply to Marcelo TosattiExamine guest pagetable and bring the shadow back in sync. At the moment sync_page is simplistic and only cares about shadow present entries whose gfn remains unchanged.
It might be worthwhile to prepopulate the shadow in advance.
FIXME: the RW->RO transition needs a local TLB flush.
arch/x86/kvm/mmu.c +49 -0
@@ -868,6 +868,14 @@ static void nonpaging_prefetch_page(stru sp->spt[i] = shadow_trap_nonpresent_pte; } +static int nonpaging_sync_page (struct kvm_vcpu *vcpu, +struct kvm_mmu_page *sp) +{ +/* should never happen */ +WARN_ON (1 ); +return 1 ; +} +static struct kvm_mmu_page *kvm_mmu_lookup_page (struct kvm *kvm,gfn_t gfn) {unsigned index;
@@ -888,6 +896,43 @@ static struct kvm_mmu_page *kvm_mmu_lookreturn NULL ; } +static void kvm_sync_writeble (struct kvm_vcpu *vcpu,u64 *spte,int gpte_rw, +gfn_t gfn) +{ +if (is_writeble_pte (*spte) == gpte_rw) +return ; +if (is_writeble_pte (*spte)) + *spte &= ~PT_WRITABLE_MASK ; +else { +if (kvm_mmu_lookup_page (vcpu->kvm, gfn)) { +pgprintk ("%s: found shadow page for %lx, keeping ro\n" , + __func__, gfn); + }else + *spte |=PT_WRITABLE_MASK ; + } +return ; +} + +static void kvm_mmu_zap_page (struct kvm *kvm,struct kvm_mmu_page *sp); + +static int kvm_sync_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +int ret; + +rmap_write_protect (vcpu->kvm, sp->gfn); + ret = vcpu->arch.mmu.sync_page (vcpu, sp); +if (ret <=0 ) +/* possible optimization: unprotect all +* mappings (only originally writeble ones +* of course). +*/ +kvm_mmu_zap_page (vcpu->kvm, sp); +else +kvm_clear_pg_unsync (sp); + +return ret; +} + static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gaddr,
@@ -1536,6 +1581,7 @@ static int nonpaging_init_context(struct context->gva_to_gpa = nonpaging_gva_to_gpa; context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; + context->sync_page = nonpaging_sync_page; context->root_level =0 ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;
@@ -1583,6 +1629,7 @@ static int paging64_init_context_common( context->page_fault = paging64_page_fault; context->gva_to_gpa = paging64_gva_to_gpa; context->prefetch_page = paging64_prefetch_page; + context->sync_page = paging64_sync_page; context->free = paging_free; context->root_level = level; context->shadow_root_level = level;
@@ -1604,6 +1651,7 @@ static int paging32_init_context(struct context->gva_to_gpa = paging32_gva_to_gpa; context->free = paging_free; context->prefetch_page = paging32_prefetch_page; + context->sync_page = paging32_sync_page; context->root_level =PT32_ROOT_LEVEL ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;
@@ -1623,6 +1671,7 @@ static int init_kvm_tdp_mmu(struct kvm_v context->page_fault = tdp_page_fault; context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; + context->sync_page = nonpaging_sync_page; context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ;
arch/x86/kvm/paging_tmpl.h +79 -0
85 lines not on this page · raw message
include/asm-x86/kvm_host.h +4 -0
@@ -221,6 +221,8 @@ struct kvm_mmu {gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,gva_t gva);void (*prefetch_page)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page); +int (*sync_page)(struct kvm_vcpu *vcpu, +struct kvm_mmu_page *sp);hpa_t root_hpa;int root_level;int shadow_root_level;
@@ -760,6 +762,7 @@ int kvm_age_hva(struct kvm *kvm, unsigneenum kvm_page_flags { KVM_PG_global, + KVM_PG_unsync, };#define KVMPGFLAG (name) \
@@ -773,5 +776,6 @@ static inline int kvm_test_clear_pg_##na { return test_and_clear_bit(KVM_PG_##name, &sp->flags); }KVMPGFLAG (global); +KVMPGFLAG (unsync);#endif
Re: [patch 07/13] KVM: MMU: mode specific sync_page
Quoted 9 lines · Marcelo Tosatti
Examine guest pagetable and bring the shadow back in sync. At the moment
sync_pageis simplistic and only cares about shadow present entries whose gfn remains unchanged.It might be worthwhile to prepopulate the shadow in advance.
FIXME: the
RW->ROtransition needs a local TLB flush.
Yes!
Quoted 28 lines
static struct
kvm_mmu_page*kvm_mmu_get_page(structkvm_vcpu*vcpu,arch/x86/kvm/mmu.c +3
-0gfn_t gfn, gva_t gaddr,@@ -1536,6 +1581,7 @@ static int nonpaging_init_context(struct context->gva_to_gpa = nonpaging_gva_to_gpa; context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; + context->sync_page = nonpaging_sync_page; context->root_level =0 ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;@@ -1583,6 +1629,7 @@ static int paging64_init_context_common( context->page_fault = paging64_page_fault; context->gva_to_gpa = paging64_gva_to_gpa; context->prefetch_page = paging64_prefetch_page; + context->sync_page = paging64_sync_page; context->free = paging_free; context->root_level = level; context->shadow_root_level = level;@@ -1604,6 +1651,7 @@ static int paging32_init_context(struct context->gva_to_gpa = paging32_gva_to_gpa; context->free = paging_free; context->prefetch_page = paging32_prefetch_page; + context->sync_page = paging32_sync_page; context->root_level =PT32_ROOT_LEVEL ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;
arch/x86/kvm/mmu.c static int init_kvm_tdp_mmu(struct kvm_v · line 1677
Quoted 7 lines
arch/x86/kvm/mmu.c +1
-0@@ -1623,6 +1671,7 @@ static int init_kvm_tdp_mmu(struct kvm_v context->page_fault = tdp_page_fault; context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; + context->sync_page = nonpaging_sync_page; context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ;
Not sure this is right.
What if vcpu0 is in mode X, while vcpu1 is in mode Y. vcpu0 writes to some pagetable, causing both mode X and mode Y shadows to become unsynced, so on the next resync (either by vcpu0 or vcpu1) we need to sync both modes.
Same problem with kvm_mmu_pte_write(), which right now hacks around it.
Maybe we need a ->ops member.
Quoted 11 lines
arch/x86/kvm/paging_tmpl.h +8
-0+static int FNAME (sync_page )(struct kvm_vcpu *vcpu, +struct kvm_mmu_page *sp) +{ +int i, nr_present =0 ; +struct page *pt_page; +pt_element_t *pt; +void *gpte_kaddr; +
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 530
Quoted 10 lines
arch/x86/kvm/paging_tmpl.h +10
-0+ pt_page =gfn_to_page_atomic (vcpu->kvm, sp->gfn); +if (is_error_page (pt_page)) { +kvm_release_page_clean (pt_page); +return -EFAULT ; + } + + gpte_kaddr = pt =kmap_atomic (pt_page,KM_USER0 ); + +if (PTTYPE ==32 ) + pt += sp->role.quadrant <<PT64_LEVEL_BITS ;
Only works for level 1 pages (which is okay).
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 533
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h +3
-0+ +for (i =0 ; i <PT64_ENT_PER_PAGE ; i++) { +if (is_shadow_present_pte (sp->spt[i])) {
Helper function needed for contents of inner loop.
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 543
Quoted 11 lines
arch/x86/kvm/paging_tmpl.h +10
-0+struct page *page; +u64 spte; +unsigned pte_access; + +if (!is_present_pte (*pt)) { +rmap_remove (vcpu->kvm, &sp->spt[i]); + sp->spt[i] = shadow_notrap_nonpresent_pte; + pt++; +continue ; + }
Are we missing a tlb flush? Or will the caller take care of it?
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 560
Quoted 6 lines
arch/x86/kvm/paging_tmpl.h +5
-0+ + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, *pt); +/* user */ +if (pte_access &ACC_USER_MASK ) + spte |= shadow_user_mask;
There are some special cases involving cr0.wp=0 and the user mask. so spte.u is not correlated exactly with gpte.u.
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 574
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h +3
-0+/* guest->shadow accessed sync */ +if (!(*pt &PT_ACCESSED_MASK )) + spte &= ~PT_ACCESSED_MASK ;
spte shouldn't be accessible at all if gpte is not accessed, so we can set gpte.a on the next access (similar to spte not being writeable if gpte is not dirty).
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 577
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h +3
-0+/* shadow->guest accessed sync */ +if (spte &PT_ACCESSED_MASK ) +set_bit (PT_ACCESSED_SHIFT , (unsigned long *)pt);
host accessed and guest accessed are very different. We shouldn't set host accessed unless we're sure the guest will access the page very soon.
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 581
Quoted 2 lines
arch/x86/kvm/paging_tmpl.h +1
-0+set_shadow_pte (&sp->spt[i], spte);
What if permissions are reduced?
You can use PT_* instead of shadow_* as this will never be called when ept is active.
I'm worried about the duplication with kvm_mmu_set_pte(). Perhaps that can be refactored instead to be the inner loop.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 07/13] KVM: MMU: mode specific sync_page
Quoted 6 lines · Avi Kivity
What if vcpu0 is in mode X, while vcpu1 is in mode Y. vcpu0 writes to some pagetable, causing both mode X and mode Y shadows to become unsynced, so on the next resync (either by vcpu0 or vcpu1) we need to sync both modes.
From the oos core patch:
- hlist_for_each_entry(sp, node, bucket, hash_link)
- if (sp->gfn == gfn && sp->role.word == role.word) {
+ hlist_for_each_entry_safe(sp, node, tmp, bucket, hash_link)
+ if (sp->gfn == gfn) {
+ /*
+ * If a pagetable becomes referenced by more than one + * root, or has multiple roles, unsync it and disable + * oos. For higher level pgtables the entire tree + * has to be synced. + */
+ if (sp->root_gfn != root_gfn) {
+ kvm_set_pg_inuse(sp);
+ if (set_shared_mmu_page(vcpu, sp))
+ tmp = bucket->first;
+ kvm_clear_pg_inuse(sp);
+ unsyncable = 0;
+ }
So as soon as a pagetable is shadowed with different modes, its resynced and unsyncing is disabled.
Quoted 4 lines
Same problem with
kvm_mmu_pte_write(), which right now hacks around it.Maybe we need a ->ops member.
Quoted 9 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 543
Quoted 7 lines
arch/x86/kvm/paging_tmpl.h +6
-0+if (!is_present_pte (*pt)) { +rmap_remove (vcpu->kvm, &sp->spt[i]); + sp->spt[i] = shadow_notrap_nonpresent_pte; + pt++; +continue ; + }Are we missing a tlb flush? Or will the caller take care of it?
Yes, there's a local TLB flush missing, which can be collapsed into a single kvm_x86_ops->tlb_flush in the caller.
Quoted 9 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 560
Quoted 6 lines
arch/x86/kvm/paging_tmpl.h +5
-0+ + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, *pt); +/* user */ +if (pte_access &ACC_USER_MASK ) + spte |= shadow_user_mask;There are some special cases involving cr0.wp=0 and the user mask. so spte.u is not correlated exactly with gpte.u.
How come?
Quoted 8 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 574
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h +3
-0+/* guest->shadow accessed sync */ +if (!(*pt &PT_ACCESSED_MASK )) + spte &= ~PT_ACCESSED_MASK ;spte shouldn't be accessible at all if gpte is not accessed, so we can set gpte.a on the next access (similar to spte not being writeable if gpte is not dirty).
Right. Perhaps accessed bit synchronization to guest could be performed lazily somehow, so as to avoid a vmexit on every first page access.
Quoted 8 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 577
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h +3
-0+/* shadow->guest accessed sync */ +if (spte &PT_ACCESSED_MASK ) +set_bit (PT_ACCESSED_SHIFT , (unsigned long *)pt);host accessed and guest accessed are very different. We shouldn't set host accessed unless we're sure the guest will access the page very soon.
Quoted 4 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 581
Quoted 2 lines
arch/x86/kvm/paging_tmpl.h +1
-0+set_shadow_pte (&sp->spt[i], spte);What if permissions are reduced?
Then a local TLB flush is needed. Flushing the TLB's of remote vcpus should be done by the guest AFAICS.
Quoted 5 lines
You can use PT_* instead of shadow_* as this will never be called when ept is active.
I'm worried about the duplication with
kvm_mmu_set_pte(). Perhaps that can be refactored instead to be the inner loop.
Will look into that.
Re: [patch 07/13] KVM: MMU: mode specific sync_page
Quoted 25 lines · Marcelo Tosatti
Quoted 2 lines · Avi Kivity
What if vcpu0 is in mode X, while vcpu1 is in mode Y. vcpu0 writes to some pagetable, causing both mode X and mode Y shadows to become unsynced, so on the next resync (either by vcpu0 or vcpu1) we need to sync both modes.
From the oos core patch:
(quoted diff) +15
-2- hlist_for_each_entry(sp, node, bucket, hash_link) - if (sp->gfn == gfn && sp->role.word == role.word) { + hlist_for_each_entry_safe(sp, node, tmp, bucket, hash_link) + if (sp->gfn == gfn) { + /* + * If a pagetable becomes referenced by more than one + * root, or has multiple roles, unsync it and disable + * oos. For higher level pgtables the entire tree + * has to be synced. + */ + if (sp->root_gfn != root_gfn) { + kvm_set_pg_inuse(sp); + if (set_shared_mmu_page(vcpu, sp)) + tmp = bucket->first; + kvm_clear_pg_inuse(sp); + unsyncable = 0; + }
Quoted 3 lines
So as soon as a pagetable is shadowed with different modes, its resynced and unsyncing is disabled.
Okay. But the complexity here, esp. with rarely used cases like multiple mode shadows, is frightening.
Quoted 10 lines
Quoted 9 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 560
Quoted 7 lines
arch/x86/kvm/paging_tmpl.h +5
-0+ + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, *pt); +/* user */ +if (pte_access &ACC_USER_MASK ) + spte |= shadow_user_mask;There are some special cases involving cr0.wp=0 and the user mask. so spte.u is not correlated exactly with gpte.u.
Quoted 3 lines
How come?
When cr0.wp=0, the cpu ignores pte.w for cpl=0 accesses. kvm requires cr0.wp=1 (since we need to write protect pages, for many reasons, like emulating pte.dirty). This is how we handle pte.u=1 + pte.w=0:
- for cpl 0 accesses, we set spte.w=1 (to allow the write) and spte.u=0 (to forbid cpl>0 accesses)
- for cpl>0 accesses, we set spte.w=0 (to forbid userspace write accesses) and spte.u=1 (to allow cpl>0 read accesses)
this works well except if the accesses keep alternating between userspace and kernel.
Quoted 11 lines
Quoted 7 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 574
Quoted 5 lines
arch/x86/kvm/paging_tmpl.h +3
-0+/* guest->shadow accessed sync */ +if (!(*pt &PT_ACCESSED_MASK )) + spte &= ~PT_ACCESSED_MASK ;spte shouldn't be accessible at all if gpte is not accessed, so we can set gpte.a on the next access (similar to spte not being writeable if gpte is not dirty).
Right. Perhaps accessed bit synchronization to guest could be performed lazily somehow, so as to avoid a vmexit on every first page access.
I don't think this is doable (well you can do it if you make the guest page table not present, but then even reading the accessed bit faults).
Quoted 10 lines
Quoted 5 lines
arch/x86/kvm/paging_tmpl.h static void FNAME(prefetch_page)(struct · line 581
Quoted 3 lines
arch/x86/kvm/paging_tmpl.h +1
-0+set_shadow_pte (&sp->spt[i], spte);What if permissions are reduced?
Then a local TLB flush is needed. Flushing the TLB's of remote vcpus should be done by the guest AFAICS.
hm. It depends on why they are reduced. If the page became shadowed, then we are responsible.
Don't think this is the case here, so local flush is likely sufficient.
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH 8/13 KVM: MMU: record guest root level on struct guest_walker
In reply to Marcelo TosattiFor future usage.
arch/x86/kvm/mmu.c +5 -4
@@ -934,6 +934,7 @@ static int kvm_sync_page(struct kvm_vcpu } static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, + gfn_t root_gfn, gfn_t gfn, gva_t gaddr, unsigned level,
@@ -1343,8 +1344,8 @@ static int direct_map_entry(struct kvm_sif (*sptep == shadow_trap_nonpresent_pte) { pseudo_gfn = (addr &PT64_DIR_BASE_ADDR_MASK ) >>PAGE_SHIFT ; - sp =kvm_mmu_get_page (vcpu, pseudo_gfn, (gva_t )addr, level -1 , -1 ,ACC_ALL , sptep); + sp =kvm_mmu_get_page (vcpu,-1 , pseudo_gfn, (gva_t )addr, + level -1 ,1 ,ACC_ALL , sptep);if (!sp) {pgprintk ("nonpaging_map: ENOMEM\n" );kvm_release_pfn_clean (walk->pfn);
@@ -1466,7 +1467,7 @@ static void mmu_alloc_roots(struct kvm_vASSERT (!VALID_PAGE (root));if (tdp_enabled) metaphysical =1 ; - sp =kvm_mmu_get_page (vcpu, root_gfn,0 , + sp =kvm_mmu_get_page (vcpu,-1 , root_gfn,0 ,PT64_ROOT_LEVEL , metaphysical,ACC_ALL ,NULL ); root =__pa (sp->spt);
@@ -1489,7 +1490,7 @@ static void mmu_alloc_roots(struct kvm_v root_gfn = vcpu->arch.pdptrs[i] >>PAGE_SHIFT ; }else if (vcpu->arch.mmu.root_level ==0 ) root_gfn =0 ; - sp =kvm_mmu_get_page (vcpu, root_gfn, i <<30 , + sp =kvm_mmu_get_page (vcpu,-1 , root_gfn, i <<30 ,PT32_ROOT_LEVEL , metaphysical,ACC_ALL ,NULL ); root =__pa (sp->spt);
arch/x86/kvm/paging_tmpl.h +4 -1
@@ -64,6 +64,7 @@*/ struct guest_walker {int level; +int root_level;gfn_t table_gfn[PT_MAX_FULL_LEVELS ];pt_element_t ptes[PT_MAX_FULL_LEVELS ];gpa_t pte_gpa[PT_MAX_FULL_LEVELS ];
@@ -149,6 +150,7 @@ walk: --walker->level; }#endif + walker->root_level = walker->level;ASSERT ((!is_long_mode (vcpu) &&is_pae (vcpu)) || (vcpu->arch.cr3 &CR3_NONPAE_RESERVED_BITS ) ==0 );
@@ -322,7 +324,8 @@ static int FNAME(shadow_walk_entry)(stru metaphysical =0 ; table_gfn = gw->table_gfn[level -2 ]; } - shadow_page =kvm_mmu_get_page (vcpu, table_gfn, (gva_t )addr, level-1 , + shadow_page =kvm_mmu_get_page (vcpu, gw->table_gfn[gw->root_level -1 ], + table_gfn, (gva_t )addr, level-1 , metaphysical, access, sptep);if (!metaphysical) { r = kvm_read_guest_atomic(vcpu->kvm, gw->pte_gpa[level - 2],
PATCH 9/13 KVM: MMU: out of sync shadow core
In reply to Marcelo TosattiAllow global and single-root, single-role-per-gfn leaf shadowed pagetables to be unsynced.
Global unsync pages are saved into a per-vm array, synced on cr4/cr0 writes.
Non-global unsync pages are linked off their root shadow page, synced on cr3/cr4/cr0 writes.
Some of this logic is simplistic and could be smarter (page_multimapped and the full root sync on higher level pagetable sharing).
Also unsyncing of non-leaf nodes might be interesting (but more complicated).
arch/x86/kvm/mmu.c +241 -22
@@ -64,6 +64,17 @@ static void kvm_mmu_audit(struct kvm_vcp #define rmap_printk(x...) do { } while (0)#endif +#define OOS_DEBUG +#if defined (OOS_DEBUG ) +#define OOS_ASSERT (x) \ + if (!(x)) { \ + printk(KERN_WARNING "assertion failed %s:%d: %s\n", \ + __FILE__, __LINE__, #x); \ + dump_stack(); \ + } +#else +#define OOS_ASSERT (x) do { } while (0) +#endif #if defined(MMU_DEBUG ) || defined(AUDIT )static int dbg =0 ;
@@ -773,6 +784,8 @@ static struct kvm_mmu_page *kvm_mmu_allo sp =mmu_memory_cache_alloc (&vcpu->arch.mmu_page_header_cache,sizeof *sp); sp->spt =mmu_memory_cache_alloc (&vcpu->arch.mmu_page_cache,PAGE_SIZE ); sp->gfns =mmu_memory_cache_alloc (&vcpu->arch.mmu_page_cache,PAGE_SIZE ); +INIT_LIST_HEAD (&sp->oos_link); +INIT_LIST_HEAD (&sp->unsync_pages);set_page_private (virt_to_page (sp->spt), (unsigned long )sp);list_add (&sp->link, &vcpu->kvm->arch.active_mmu_pages);ASSERT (is_empty_shadow_page (sp->spt));
@@ -896,6 +909,31 @@ static struct kvm_mmu_page *kvm_mmu_lookreturn NULL ; } +static struct kvm_mmu_page *kvm_mmu_lookup_page_root (struct kvm_vcpu *vcpu, +gfn_t gfn) +{ +unsigned index; +struct hlist_head *bucket; +struct kvm_mmu_page *sp; +struct hlist_node *node; +struct kvm *kvm = vcpu->kvm; +int level = vcpu->arch.mmu.root_level; +if (!is_long_mode (vcpu) &&is_pae (vcpu)) + level--; + +pgprintk ("%s: looking for gfn %lx\n" , __func__, gfn); + index =kvm_page_table_hashfn (gfn); + bucket = &kvm->arch.mmu_page_hash[index]; +hlist_for_each_entry (sp, node, bucket, hash_link) +if (sp->gfn == gfn && !sp->role.metaphysical + && !sp->role.invalid && sp->role.level == level) { +pgprintk ("%s: found role %x\n" , + __func__, sp->role.word); +return sp; + } +return NULL ; +} +static void kvm_sync_writeble (struct kvm_vcpu *vcpu,u64 *spte,int gpte_rw,gfn_t gfn) {
@@ -913,12 +951,48 @@ static void kvm_sync_writeble(struct kvmreturn ; } -static void kvm_mmu_zap_page (struct kvm *kvm,struct kvm_mmu_page *sp); +static void kvm_unlink_unsync_page (struct kvm *kvm,struct kvm_mmu_page *sp) +{ +if (kvm_page_global (sp)) { +int i; +/* FIXME: save index into sp->flags */ +for (i =0 ; i <ARRAY_SIZE (kvm->arch.oos_global_pages); i++) +if (kvm->arch.oos_global_pages[i] == sp) { + kvm->arch.oos_global_pages[i] =NULL ; +break ; + } +OOS_ASSERT (i <ARRAY_SIZE (kvm->arch.oos_global_pages)); + --kvm->stat.mmu_unsync_global; + }else { +list_del (&sp->oos_link); + --kvm->stat.mmu_unsync; + } +} + +static void kvm_mmu_page_unlink_children (struct kvm *kvm, +struct kvm_mmu_page *sp); +static int kvm_mmu_zap_page (struct kvm *kvm,struct kvm_mmu_page *sp); + +static void kvm_mmu_zap_unsync_page (struct kvm *kvm,struct kvm_mmu_page *sp) +{ +if (unlikely (kvm_page_inuse (sp))) { +kvm_mmu_page_unlink_children (kvm, sp); +kvm_flush_remote_tlbs (kvm); +kvm_unlink_unsync_page (kvm, sp); +kvm_clear_pg_unsync (sp); + }else +kvm_mmu_zap_page (kvm, sp); +}static int kvm_sync_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) {int ret; +if (sp->role.glevels != vcpu->arch.mmu.root_level) { +kvm_mmu_zap_unsync_page (vcpu->kvm, sp); +return -EINVAL ; + } +rmap_write_protect (vcpu->kvm, sp->gfn); ret = vcpu->arch.mmu.sync_page (vcpu, sp);if (ret <=0 )
@@ -926,13 +1000,87 @@ static int kvm_sync_page(struct kvm_vcpu* mappings (only originally writeble ones * of course). */ -kvm_mmu_zap_page (vcpu->kvm, sp); - else +kvm_mmu_zap_unsync_page (vcpu->kvm, sp); + else { +kvm_unlink_unsync_page (vcpu->kvm, sp);kvm_clear_pg_unsync (sp); + }return ret; } +static int mmu_unsync_global_page (struct kvm_vcpu *vcpu, +struct kvm_mmu_page *sp) + +{ +struct kvm *kvm = vcpu->kvm; +unsigned idx = kvm->arch.oos_global_idx; + +if (kvm->arch.oos_global_pages[idx]) +kvm_sync_page (vcpu, kvm->arch.oos_global_pages[idx]); + +kvm_set_pg_unsync (sp); + kvm->arch.oos_global_pages[idx] = sp; + + kvm->arch.oos_global_idx++; +if (kvm->arch.oos_global_idx >=ARRAY_SIZE (kvm->arch.oos_global_pages)) + kvm->arch.oos_global_idx =0 ; + + ++kvm->stat.mmu_unsync_global; +return 0 ; +} + +static int page_multimapped (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +while (!sp->multimapped) { +if (!sp->parent_pte) +return 0 ; + sp =page_header (__pa (sp->parent_pte)); + } +return 1 ; +} + +static int mmu_unsync_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +struct kvm_mmu_page *root_sp; + +if (page_multimapped (vcpu, sp)) +return 1 ; + + root_sp =kvm_mmu_lookup_page_root (vcpu, sp->root_gfn); +if (!root_sp) +return 1 ; + +kvm_set_pg_unsync (sp); +list_add (&sp->oos_link, &root_sp->unsync_pages); + ++vcpu->kvm->stat.mmu_unsync; +return 0 ; +} + +static int kvm_unsync_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +if (kvm_page_global (sp)) +return mmu_unsync_global_page (vcpu, sp); +else +return mmu_unsync_page (vcpu, sp); +} + +static int set_shared_mmu_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) +{ +struct kvm_mmu_page *root_sp; +int ret =0 ; + +if (sp->role.level ==PT_PAGE_TABLE_LEVEL ) { +if (kvm_page_unsync (sp)) +kvm_sync_page (vcpu, sp); + }else if (sp->root_gfn !=-1 ) { + root_sp =kvm_mmu_lookup_page_root (vcpu, sp->root_gfn); + } + + sp->root_gfn =-1 ; +return ret; +} + static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, gfn_t root_gfn, gfn_t gfn,
@@ -947,7 +1095,8 @@ static struct kvm_mmu_page *kvm_mmu_get_unsigned quadrant;struct hlist_head *bucket;struct kvm_mmu_page *sp; -struct hlist_node *node; +struct hlist_node *node, *tmp; +int unsyncable =1 ; role.word =0 ; role.glevels = vcpu->arch.mmu.root_level;
@@ -963,8 +1112,24 @@ static struct kvm_mmu_page *kvm_mmu_get_ gfn, role.word); index =kvm_page_table_hashfn (gfn); bucket = &vcpu->kvm->arch.mmu_page_hash[index]; -hlist_for_each_entry (sp, node, bucket, hash_link) -if (sp->gfn == gfn && sp->role.word == role.word) { +hlist_for_each_entry_safe (sp, node, tmp, bucket, hash_link) +if (sp->gfn == gfn) { +/* +* If a pagetable becomes referenced by more than one +* root, or has multiple roles, unsync it and disable +* oos. For higher level pgtables the entire tree +* has to be synced. +*/ +if (sp->root_gfn != root_gfn) { +kvm_set_pg_inuse (sp); +if (set_shared_mmu_page (vcpu, sp)) + tmp = bucket->first; +kvm_clear_pg_inuse (sp); + unsyncable =0 ; + } +if (sp->role.word != role.word) +continue ; +mmu_page_add_parent_pte (vcpu, sp, parent_pte);pgprintk ("%s: found\n" , __func__);return sp;
@@ -975,6 +1140,9 @@ static struct kvm_mmu_page *kvm_mmu_get_return sp;pgprintk ("%s: adding gfn %lx role %x\n" , __func__, gfn, role.word); sp->gfn = gfn; +if (!unsyncable) + root_gfn =-1 ; + sp->root_gfn = root_gfn; sp->role = role;hlist_add_head (&sp->hash_link, bucket);if (!metaphysical)
@@ -1084,14 +1252,35 @@ static void kvm_mmu_unlink_parents(struc } } -static void kvm_mmu_zap_page (struct kvm *kvm,struct kvm_mmu_page *sp) +static void kvm_drop_unsync_children (struct kvm *kvm, +struct kvm_mmu_page *root_sp) +{ +struct kvm_mmu_page *sp, *n; + +list_for_each_entry_safe (sp, n, &root_sp->unsync_pages, oos_link) { +OOS_ASSERT (kvm_page_unsync (sp)); +OOS_ASSERT (!kvm_page_global (sp)); +OOS_ASSERT (sp->role.level ==PT_PAGE_TABLE_LEVEL ); +OOS_ASSERT (list_empty (&sp->unsync_pages)); +kvm_mmu_zap_page (kvm, sp); + } +} + +static int kvm_mmu_zap_page (struct kvm *kvm,struct kvm_mmu_page *sp) { +int ret =0 ; ++kvm->stat.mmu_shadow_zapped;kvm_mmu_page_unlink_children (kvm, sp);kvm_mmu_unlink_parents (kvm, sp);kvm_flush_remote_tlbs (kvm);if (!sp->role.invalid && !sp->role.metaphysical)unaccount_shadowed (kvm, sp->gfn); +if (kvm_test_clear_pg_unsync (sp)) +kvm_unlink_unsync_page (kvm, sp); +if (!list_empty (&sp->unsync_pages)) { +kvm_drop_unsync_children (kvm, sp); + ret =1 ; + }if (!sp->root_count) {hlist_del (&sp->hash_link);kvm_mmu_free_page (kvm, sp);
@@ -1101,6 +1290,7 @@ static void kvm_mmu_zap_page(struct kvmkvm_reload_remote_mmus (kvm); }kvm_mmu_reset_last_pte_updated (kvm); +return ret; }/*
@@ -1153,8 +1343,9 @@ static int kvm_mmu_unprotect_page(structif (sp->gfn == gfn && !sp->role.metaphysical) {pgprintk ("%s: gfn %lx role %x\n" , __func__, gfn, sp->role.word); -kvm_mmu_zap_page (kvm, sp); r =1 ; +if (kvm_mmu_zap_page (kvm, sp)) + n = bucket->first; }return r; }
@@ -1191,6 +1382,25 @@ struct page *gva_to_page(struct kvm_vcpureturn page; } +static int mmu_need_write_protect (struct kvm_vcpu *vcpu,gfn_t gfn, +int write_fault) +{ +struct kvm_mmu_page *shadow; + + shadow =kvm_mmu_lookup_page (vcpu->kvm, gfn); +if (shadow) { +if (!write_fault) +return 1 ; +if (shadow->role.level !=PT_PAGE_TABLE_LEVEL ) +return 1 ; +if (shadow->root_gfn ==-1 ) +return 1 ; +if (!kvm_page_unsync (shadow)) +return kvm_unsync_page (vcpu, shadow); + } +return 0 ; +} + static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *shadow_pte, unsigned pt_access, unsigned pte_access, int user_fault, int write_fault, int dirty,
@@ -1207,8 +1417,13 @@ static void mmu_set_spte(struct kvm_vcpu __func__, *shadow_pte, pt_access, write_fault, user_fault, gfn); -if (!global) +kvm_set_pg_inuse (sp); + +if (!global &&kvm_page_global (sp)) { +if (kvm_page_unsync (sp)) +kvm_sync_page (vcpu, sp);kvm_clear_pg_global (sp); + }if (is_rmap_pte (*shadow_pte)) {/*
@@ -1256,12 +1471,19 @@ static void mmu_set_spte(struct kvm_vcpuif ((pte_access &ACC_WRITE_MASK ) || (write_fault && !is_write_protection (vcpu) && !user_fault)) { -struct kvm_mmu_page *shadow; +/* +* Do not create write protected large translations. +*/ +if (largepage &&has_wrprotected_page (vcpu->kvm, gfn)) { + spte = shadow_trap_nonpresent_pte; + was_writeble =0 ; + *ptwrite =0 ; + goto set_shadow; + } spte |=PT_WRITABLE_MASK ; - shadow =kvm_mmu_lookup_page (vcpu->kvm, gfn); -if (shadow) { +if (mmu_need_write_protect (vcpu, gfn, write_fault)) {pgprintk ("%s: found shadow page for %lx, marking ro\n" , __func__, gfn); pte_access &= ~ACC_WRITE_MASK ;
@@ -1272,16 +1494,9 @@ static void mmu_set_spte(struct kvm_vcpuif (write_fault) *ptwrite =1 ; } -/* -* Do not create write protected large translations. -*/ -if (largepage &&has_wrprotected_page (vcpu->kvm, gfn)) { - spte = shadow_trap_nonpresent_pte; - was_writeble =0 ; - *ptwrite =0 ; - } } +set_shadow:if (pte_access &ACC_WRITE_MASK )mark_page_dirty (vcpu->kvm, gfn);
@@ -1309,6 +1524,7 @@ static void mmu_set_spte(struct kvm_vcpu vcpu->arch.last_pte_updated = shadow_pte; vcpu->arch.last_pte_gfn = gfn; } +kvm_clear_pg_inuse (sp); }static void nonpaging_new_cr3 (struct kvm_vcpu *vcpu)
@@ -1950,7 +2166,8 @@ void kvm_mmu_pte_write(struct kvm_vcpu **/ pgprintk ("misaligned: gpa %llx bytes %d role %x\n" , gpa, bytes, sp->role.word); -kvm_mmu_zap_page (vcpu->kvm, sp); +if (kvm_mmu_zap_page (vcpu->kvm, sp)) + n = bucket->first; ++vcpu->kvm->stat.mmu_flooded;continue ; }
@@ -2174,7 +2391,9 @@ void kvm_mmu_zap_all(struct kvm *kvm)spin_lock (&kvm->mmu_lock);list_for_each_entry_safe (sp, node, &kvm->arch.active_mmu_pages, link) -kvm_mmu_zap_page (kvm, sp); +if (kvm_mmu_zap_page (kvm, sp)) + node =container_of (kvm->arch.active_mmu_pages.next, + struct kvm_mmu_page, link);spin_unlock (&kvm->mmu_lock);kvm_flush_remote_tlbs (kvm);
include/asm-x86/kvm_host.h +10 -0
@@ -179,6 +179,10 @@ union kvm_mmu_page_role {struct kvm_mmu_page {struct list_head link;struct hlist_node hash_link; +/* FIXME: one list_head is enough */ +struct list_head unsync_pages; +struct list_head oos_link; +gfn_t root_gfn;/* root this pagetable belongs to, -1 if multimapped */ /* * The following two entries are used to key the shadow page in the
@@ -362,6 +366,8 @@ struct kvm_arch{unsigned int n_requested_mmu_pages;unsigned int n_alloc_mmu_pages;struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES ]; +struct kvm_mmu_page *oos_global_pages[7 ]; +unsigned oos_global_idx;/* * Hash table of struct kvm_mmu_page. */
@@ -390,6 +396,8 @@ struct kvm_vm_stat {u32 mmu_flooded;u32 mmu_recycled;u32 mmu_cache_miss; +u32 mmu_unsync; +u32 mmu_unsync_global;u32 remote_tlb_flush;u32 lpages; };
@@ -763,6 +771,7 @@ int kvm_age_hva(struct kvm *kvm, unsigneenum kvm_page_flags { KVM_PG_global, KVM_PG_unsync, + KVM_PG_inuse, };#define KVMPGFLAG (name) \
@@ -777,5 +786,6 @@ static inline int kvm_test_clear_pg_##naKVMPGFLAG (global);KVMPGFLAG (unsync); +KVMPGFLAG (inuse);#endif
arch/x86/kvm/x86.c +2 -0
@@ -100,6 +100,8 @@ struct kvm_stats_debugfs_item debugfs_en { "mmu_flooded", VM_STAT(mmu_flooded) }, { "mmu_recycled", VM_STAT(mmu_recycled) }, { "mmu_cache_miss", VM_STAT(mmu_cache_miss) }, + { "mmu_unsync", VM_STAT(mmu_unsync) }, + { "mmu_unsync_global", VM_STAT(mmu_unsync_global) }, { "remote_tlb_flush", VM_STAT(remote_tlb_flush) }, { "largepages", VM_STAT(lpages) }, { NULL }
arch/x86/kvm/paging_tmpl.h +10 -4
@@ -81,6 +81,7 @@ struct shadow_walker {int write_fault;int largepage;int *ptwrite; +int multiroot;pfn_t pfn;u64 *sptep; };
@@ -294,7 +295,7 @@ static int FNAME(shadow_walk_entry)(strustruct kvm_mmu_page *shadow_page;u64 spte;int metaphysical; -gfn_t table_gfn; +gfn_t table_gfn, root_gfn;int r;pt_element_t curr_pte;
@@ -324,9 +325,11 @@ static int FNAME(shadow_walk_entry)(stru metaphysical =0 ; table_gfn = gw->table_gfn[level -2 ]; } - shadow_page =kvm_mmu_get_page (vcpu, gw->table_gfn[gw->root_level -1 ], - table_gfn, (gva_t )addr, level-1 , - metaphysical, access, sptep); + root_gfn = gw->table_gfn[gw->root_level -1 ]; +if (sw->multiroot) + root_gfn =-1 ; + shadow_page =kvm_mmu_get_page (vcpu, root_gfn, table_gfn, (gva_t )addr, + level-1 , metaphysical, access, sptep);if (!metaphysical) { r =kvm_read_guest_atomic (vcpu->kvm, gw->pte_gpa[level -2 ], &curr_pte,sizeof (curr_pte));
@@ -336,6 +339,8 @@ static int FNAME(shadow_walk_entry)(strureturn 1 ; } } +if (shadow_page->root_gfn ==-1 ) + sw->multiroot =1 ; spte =__pa (shadow_page->spt) |PT_PRESENT_MASK |PT_ACCESSED_MASK |PT_WRITABLE_MASK |PT_USER_MASK ;
@@ -355,6 +360,7 @@ static u64 *FNAME(fetch)(struct kvm_vcpu .write_fault = write_fault, .largepage = largepage, .ptwrite = ptwrite, + .multiroot = 0, .pfn = pfn, };
Re: [patch 09/13] KVM: MMU: out of sync shadow core
Quoted 6 lines · Marcelo Tosatti
Allow global and single-root, single-role-per-gfn leaf shadowed pagetables to be unsynced.
Global unsync pages are saved into a per-vm array, synced on cr4/cr0 writes.
Why not a list?
Quoted 24 lines
Non-global unsync pages are linked off their root shadow page, synced on cr3/cr4/cr0 writes.
Some of this logic is simplistic and could be smarter (
page_multimappedand the full root sync on higher level pagetable sharing).Also unsyncing of non-leaf nodes might be interesting (but more complicated).
arch/x86/kvm/mmu.c +14
-0+static struct kvm_mmu_page *kvm_mmu_lookup_page_root (struct kvm_vcpu *vcpu, +gfn_t gfn) +{ +unsigned index; +struct hlist_head *bucket; +struct kvm_mmu_page *sp; +struct hlist_node *node; +struct kvm *kvm = vcpu->kvm; +int level = vcpu->arch.mmu.root_level; +if (!is_long_mode (vcpu) &&is_pae (vcpu)) + level--; + +pgprintk ("%s: looking for gfn %lx\n" , __func__, gfn); + index =kvm_page_table_hashfn (gfn);
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_look · line 935
Quoted 10 lines
arch/x86/kvm/mmu.c +10
-0+ bucket = &kvm->arch.mmu_page_hash[index]; +hlist_for_each_entry (sp, node, bucket, hash_link) +if (sp->gfn == gfn && !sp->role.metaphysical + && !sp->role.invalid && sp->role.level == level) { +pgprintk ("%s: found role %x\n" , + __func__, sp->role.word); +return sp; + } +return NULL ; +}
I'm worried about the complexity this (and the rest) introduces.
A possible alternative is:
- for non-leaf pages, including roots, add a 'unsync_children' flag.
- when marking a page unsync, set the flag recursively on all parents
- when switching cr3, recursively descend to locate unsynced leaves, clearing flags along the way
- to speed this up, put a bitmap with 1 bit per pte in the pages (512 bits = 64 bytes)
- the bitmap can be externally allocated to save space, or not
This means we no longer have to worry about multiple roots, when a page acquires another root while it is unsynced, etc.
Quoted 7 lines
arch/x86/kvm/mmu.c +1
-2@@ -963,8 +1112,24 @@ static struct kvm_mmu_page *kvm_mmu_get_ gfn, role.word); index =kvm_page_table_hashfn (gfn); bucket = &vcpu->kvm->arch.mmu_page_hash[index]; -hlist_for_each_entry (sp, node, bucket, hash_link) -if (sp->gfn == gfn && sp->role.word == role.word) { +hlist_for_each_entry_safe (sp, node, tmp, bucket, hash_link)
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_ · line 1125
Quoted 10 lines
arch/x86/kvm/mmu.c +9
-0+if (sp->gfn == gfn) { +/* +* If a pagetable becomes referenced by more than one +* root, or has multiple roles, unsync it and disable +* oos. For higher level pgtables the entire tree +* has to be synced. +*/ +if (sp->root_gfn != root_gfn) { +kvm_set_pg_inuse (sp);
What does inuse mean exactly?
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_
Quoted 4 lines
arch/x86/kvm/mmu.c +3
-0+if (set_shared_mmu_page (vcpu, sp)) + tmp = bucket->first; +kvm_clear_pg_inuse (sp);
Cleared here?
Quoted 11 lines
arch/x86/kvm/mmu.c +5
-0+ unsyncable =0 ; + } +if (sp->role.word != role.word) +continue ; +mmu_page_add_parent_pte (vcpu, sp, parent_pte);arch/x86/kvm/mmu.c +0
-0pgprintk ("%s: found\n" , __func__);return sp;
include/asm-x86/kvm_host.h union kvm_mmu_page_role { · line 185
Quoted 7 lines
include/asm-x86/kvm_host.h +3
-0@@ -179,6 +179,10 @@ union kvm_mmu_page_role {struct kvm_mmu_page {struct list_head link;struct hlist_node hash_link; +/* FIXME: one list_head is enough */ +struct list_head unsync_pages; +struct list_head oos_link;
That's okay, we may allow OOS roots one day.
include/asm-x86/kvm_host.h struct kvm_arch{ · line 371
Quoted 11 lines
include/asm-x86/kvm_host.h +1
-0+gfn_t root_gfn;/* root this pagetable belongs to, -1 if multimapped */ /* include/asm-x86/kvm_host.h +2
-0* The following two entries are used to key the shadow page in the @@ -362,6 +366,8 @@ struct kvm_arch{unsigned int n_requested_mmu_pages;unsigned int n_alloc_mmu_pages;struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES ]; +struct kvm_mmu_page *oos_global_pages[7 ]; +unsigned oos_global_idx;
What does the index mean? An lru pointer?
I became a little unsynced myself reading the patch. It's very complex.
We need to change something to keep it maintainable. Either a better data structure, or disallowing a parent to be zapped while any of its children are alive.
What happens when a sp->global changes its value while a page is oos?
Can we even detect a nonglobal->global change? Maybe instead of a flag, add a counter for active ptes and global ptes, and a page is global if the counters match. However most likely it doesn't matter at all.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 09/13] KVM: MMU: out of sync shadow core
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Marcelo Tosatti
Allow global and single-root, single-role-per-gfn leaf shadowed pagetables to be unsynced.
Global unsync pages are saved into a per-vm array, synced on cr4/cr0 writes.
Why not a list?
Could be.
Quoted 12 lines
I'm worried about the complexity this (and the rest) introduces.
A possible alternative is:
- for non-leaf pages, including roots, add a '
unsync_children' flag.
- when marking a page unsync, set the flag recursively on all parents
- when switching cr3, recursively descend to locate unsynced leaves, clearing flags along the way
- to speed this up, put a bitmap with 1 bit per pte in the pages (512 bits = 64 bytes)
- the bitmap can be externally allocated to save space, or not
Quoted 2 lines
This means we no longer have to worry about multiple roots, when a page acquires another root while it is unsynced, etc.
I thought about that when you first mentioned it, but it seems more complex than the current structure. Remember you have to clean the unsynced flag on resync, which means walking up the parents verifying if this is the last unsynced children.
Other than the bitmap space.
And see comments about multiple roles below.
Quoted 11 lines
Quoted 11 lines
arch/x86/kvm/mmu.c +5
-2@@ -963,8 +1112,24 @@ static struct kvm_mmu_page *kvm_mmu_get_ gfn, role.word); index =kvm_page_table_hashfn (gfn); bucket = &vcpu->kvm->arch.mmu_page_hash[index]; -hlist_for_each_entry (sp, node, bucket, hash_link) -if (sp->gfn == gfn && sp->role.word == role.word) { +hlist_for_each_entry_safe (sp, node, tmp, bucket, hash_link) +if (sp->gfn == gfn) { +/* +* If a pagetable becomes referenced by more than one +* root, or has multiple roles, unsync it and disable
Quoted 8 lines
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_
Quoted 6 lines
arch/x86/kvm/mmu.c +5
-0+* oos. For higher level pgtables the entire tree +* has to be synced. +*/ +if (sp->root_gfn != root_gfn) { +kvm_set_pg_inuse (sp);What does inuse mean exactly?
That we're going to access struct kvm_mmu_page, so kvm_sync_page won't free it (also used for global->nonglobal resync).
Quoted 6 lines
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_
Quoted 4 lines
arch/x86/kvm/mmu.c +3
-0+if (set_shared_mmu_page (vcpu, sp)) + tmp = bucket->first; +kvm_clear_pg_inuse (sp);Cleared here?
Can't be zapped after this point.
Quoted 22 lines
Quoted 12 lines
arch/x86/kvm/mmu.c +5
-0+ unsyncable =0 ; + } +if (sp->role.word != role.word) +continue ; +mmu_page_add_parent_pte (vcpu, sp, parent_pte);arch/x86/kvm/mmu.c +0
-0pgprintk ("%s: found\n" , __func__);return sp;include/asm-x86/kvm_host.h union kvm_mmu_page_role { · line 185
Quoted 7 lines
include/asm-x86/kvm_host.h +3
-0@@ -179,6 +179,10 @@ union kvm_mmu_page_role {struct kvm_mmu_page {struct list_head link;struct hlist_node hash_link; +/* FIXME: one list_head is enough */ +struct list_head unsync_pages; +struct list_head oos_link;That's okay, we may allow OOS roots one day.
Quoted 12 lines
include/asm-x86/kvm_host.h struct kvm_arch{ · line 371
Quoted 10 lines
include/asm-x86/kvm_host.h +3
-0+gfn_t root_gfn;/* root this pagetable belongs to, -1 if multimapped */ /* * The following two entries are used to key the shadow page in the @@ -362,6 +366,8 @@ struct kvm_arch{unsigned int n_requested_mmu_pages;unsigned int n_alloc_mmu_pages;struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES ]; +struct kvm_mmu_page *oos_global_pages[7 ]; +unsigned oos_global_idx;What does the index mean? An lru pointer?
Yes, last saved index into array.
Quoted 1 line
I became a little unsynced myself reading the patch. It's very complex.
Can you go into detail? Worrying about multiple roots is more about code change (passing root_gfn down to mmu_get_page etc) than structural complexity I think. It boils down to
if (sp->root_gfn != root_gfn) {
kvm_set_pg_inuse(sp);
if (set_shared_mmu_page(vcpu, sp))
tmp = bucket->first;
kvm_clear_pg_inuse(sp);
}And this also deals with the pagetable with shadows in different modes/roles case. You'd still have to deal with that by keeping unsync information all the way up to root.
Quoted 5 lines
We need to change something to keep it maintainable. Either a better data structure,
or disallowing a parent to be zapped while any of its children are alive.
What is the problem with that? And what the alternative would be, to zap all children first?
So more details please, what exactly is annoying you:
- Awareness of multiple roots in the current form ? I agree its not very elegant.
- The fact that hash table bucket and active_mmu_page for_each_entry_safe walks are unsafe because several list entries (the unsynced leafs) can be deleted ?
Quoted 1 line
What happens when a
sp->globalchanges its value while a page is oos?
Its resynced on global->nonglobal change.
Quoted 3 lines
Can we even detect a
nonglobal->globalchange? Maybe instead of a flag, add a counter for active ptes and global ptes, and a page is global if the counters match. However most likely it doesn't matter at all.
Yeah, it seems nonglobal->global change is quite uncommon.
Oh, and another argument in favour of atomic resync is that you can do it from mmu_get_page (for multiple role case), and from within mmu_set_spte (for global->nonglobal change).
Re: [patch 09/13] KVM: MMU: out of sync shadow core
Quoted 12 lines · Marcelo Tosatti
Quoted 12 lines
I'm worried about the complexity this (and the rest) introduces.
A possible alternative is:
- for non-leaf pages, including roots, add a '
unsync_children' flag.
- when marking a page unsync, set the flag recursively on all parents
- when switching cr3, recursively descend to locate unsynced leaves, clearing flags along the way
- to speed this up, put a bitmap with 1 bit per pte in the pages (512 bits = 64 bytes)
- the bitmap can be externally allocated to save space, or not
Quoted 8 lines
Quoted 2 lines
This means we no longer have to worry about multiple roots, when a page acquires another root while it is unsynced, etc.
I thought about that when you first mentioned it, but it seems more complex than the current structure. Remember you have to clean the unsynced flag on resync, which means walking up the parents verifying if this is the last unsynced children.
No, if you have a false positive you can simply ignore it.
Quoted 2 lines
Other than the bitmap space.
The bitmap space could be stored in a separate structure. Alternatively, put a few u16s with indexes into the page header. Would be faster to walk as well, though less general.
Quoted 24 lines
And see comments about multiple roles below.
Quoted 11 lines
Quoted 11 lines
arch/x86/kvm/mmu.c +5
-2@@ -963,8 +1112,24 @@ static struct kvm_mmu_page *kvm_mmu_get_ gfn, role.word); index =kvm_page_table_hashfn (gfn); bucket = &vcpu->kvm->arch.mmu_page_hash[index]; -hlist_for_each_entry (sp, node, bucket, hash_link) -if (sp->gfn == gfn && sp->role.word == role.word) { +hlist_for_each_entry_safe (sp, node, tmp, bucket, hash_link) +if (sp->gfn == gfn) { +/* +* If a pagetable becomes referenced by more than one +* root, or has multiple roles, unsync it and disable Quoted 9 lines
arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_
Quoted 7 lines
arch/x86/kvm/mmu.c +5
-0+* oos. For higher level pgtables the entire tree +* has to be synced. +*/ +if (sp->root_gfn != root_gfn) { +kvm_set_pg_inuse (sp);What does inuse mean exactly?
Quoted 4 lines
That we're going to access struct
kvm_mmu_page, sokvm_sync_pagewon't free it (also used forglobal->nonglobalresync).
Couldn't it be passed as a parameter?
Quoted 14 lines
Quoted 2 lines
I became a little unsynced myself reading the patch. It's very complex.
Can you go into detail? Worrying about multiple roots is more about code change (passing
root_gfndown tommu_get_pageetc) than structural complexity I think. It boils down toif (sp->root_gfn != root_gfn) { kvm_set_pg_inuse(sp); if (set_shared_mmu_page(vcpu, sp)) tmp = bucket->first; kvm_clear_pg_inuse(sp); }
Quoted 5 lines
And this also deals with the pagetable with shadows in different modes/roles case. You'd still have to deal with that by keeping unsync information all the way up to root.
I'm worried about the amount of state we add. Whether a page is single-root or multi-root, if it's in the same mode or multiple modes. The problems with the large amount of state is that the number of possible state transitions increases rapidly.
So far we treat each page completely independently of other pages (apart from the connectivity pointers), so we avoid the combinatorial explosion. The tree walk approach keeps that (at the expense of some efficiency, unfortunately).
Quoted 4 lines
Quoted 2 lines
or disallowing a parent to be zapped while any of its children are alive.
What is the problem with that?
It reduces the mmu flexibility. If we (say) introduce an lru algorithm, it is orthogonal to everything else in the mmu. If we have a root/child dependency, the lru has to know.
Quoted 2 lines
And what the alternative would be, to zap all children first?
That has the disadvantage of allowing very bad corner cases if we are forced to zap a root.
I'd really like to avoid bad worst cases.
Quoted 5 lines
So more details please, what exactly is annoying you:
- Awareness of multiple roots in the current form ? I agree its not very elegant.
Yes.
Quoted 5 lines
- The fact that hash table bucket and
active_mmu_pagefor_each_entry_safewalks are unsafe because several list entries (the unsynced leafs) can be deleted ?
Hadn't even considered that...
What worries me most is that everything is interconnected: multiple modes, cr3 switch, out-of-sync, zapping via the inuse flag. It's very difficult for me to understand, what about someone new?
We need to make this fit better. We need to morph some mmu infrastructure to something else, but we can't keep adding complexity.
Quoted 4 lines
Oh, and another argument in favour of atomic resync is that you can do it from
mmu_get_page(for multiple role case), and from withinmmu_set_spte(forglobal->nonglobalchange).
I'll be more comfortable with atomic resync if we have snapshots as a means not to require so many walks in an atomic context.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [patch 09/13] KVM: MMU: out of sync shadow core
Quoted 55 lines · Avi Kivity
Quoted 12 lines · Marcelo Tosatti
Quoted 12 lines
I'm worried about the complexity this (and the rest) introduces.
A possible alternative is:
- for non-leaf pages, including roots, add a '
unsync_children' flag.
- when marking a page unsync, set the flag recursively on all parents
- when switching cr3, recursively descend to locate unsynced leaves, clearing flags along the way
- to speed this up, put a bitmap with 1 bit per pte in the pages (512 bits = 64 bytes)
- the bitmap can be externally allocated to save space, or notQuoted 9 lines
Quoted 3 lines
This means we no longer have to worry about multiple roots, when a page acquires another root while it is unsynced, etc.
I thought about that when you first mentioned it, but it seems more complex than the current structure. Remember you have to clean the unsynced flag on resync, which means walking up the parents verifying if this is the last unsynced children.
No, if you have a false positive you can simply ignore it.
Quoted 2 lines
Other than the bitmap space.
The bitmap space could be stored in a separate structure. Alternatively, put a few u16s with indexes into the page header. Would be faster to walk as well, though less general.
Quoted 23 lines
And see comments about multiple roles below.
Quoted 19 lines
Quoted 7 lines
arch/x86/kvm/mmu.c +1
-2@@ -963,8 +1112,24 @@ static struct kvm_mmu_page *kvm_mmu_get_ gfn, role.word); index =kvm_page_table_hashfn (gfn); bucket = &vcpu->kvm->arch.mmu_page_hash[index]; -hlist_for_each_entry (sp, node, bucket, hash_link) -if (sp->gfn == gfn && sp->role.word == role.word) { +hlist_for_each_entry_safe (sp, node, tmp, bucket, hash_link)arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_get_ · line 1125
Quoted 10 lines
arch/x86/kvm/mmu.c +9
-0+if (sp->gfn == gfn) { +/* +* If a pagetable becomes referenced by more than one +* root, or has multiple roles, unsync it and disable +* oos. For higher level pgtables the entire tree +* has to be synced. +*/ +if (sp->root_gfn != root_gfn) { +kvm_set_pg_inuse (sp);What does inuse mean exactly?
Quoted 6 lines
Quoted 4 lines
That we're going to access struct
kvm_mmu_page, sokvm_sync_pagewon't free it (also used forglobal->nonglobalresync).Couldn't it be passed as a parameter?
Yes it could.
Quoted 14 lines
Quoted 14 lines
Quoted 2 lines
I became a little unsynced myself reading the patch. It's very complex.
Can you go into detail? Worrying about multiple roots is more about code change (passing
root_gfndown tommu_get_pageetc) than structural complexity I think. It boils down toif (sp->root_gfn != root_gfn) { kvm_set_pg_inuse(sp); if (set_shared_mmu_page(vcpu, sp)) tmp = bucket->first; kvm_clear_pg_inuse(sp); }
Quoted 8 lines
Quoted 5 lines
And this also deals with the pagetable with shadows in different modes/roles case. You'd still have to deal with that by keeping unsync information all the way up to root.
I'm worried about the amount of state we add. Whether a page is single-root or multi-root, if it's in the same mode or multiple modes.
You need similar complexity to handle pagetables with multiple roles under unsync-info-on-tree approach:
@@ -993,8 +1049,15 @@ static struct kvm_mmu_page *kvm_mmu_get_
gfn, role.word);
index = kvm_page_table_hashfn(gfn);
bucket = &vcpu->kvm->arch.mmu_page_hash[index];- hlist_for_each_entry(sp, node, bucket, hash_link)
- if (sp->gfn == gfn && sp->role.word == role.word) {
+ hlist_for_each_entry_safe(sp, node, tmp, bucket, hash_link)
+ if (sp->gfn == gfn) {
+ if (sp->role.word != role.word) {
+ if (kvm_page_unsync(sp))
+ mmu_invalidate_unsync_page(vcpu, sp);
+ unsyncable = 1;
+ continue;
+ }
+
mmu_page_add_parent_pte(vcpu, sp, parent_pte);
pgprintk("%s: found\n", __func__);
return sp;And such pagetables with multiple shadows will have to be marked as "unsyncable". Which is pretty similar to what the current patchset does, except that it accounts for multi-root other than multiple shadows.
The other option, which is to keep the multiple shadow unsync, and have them synced by their respective users, seems much more complex than this.
Quoted 40 lines
The problems with the large amount of state is that the number of possible state transitions increases rapidly.
So far we treat each page completely independently of other pages (apart from the connectivity pointers), so we avoid the combinatorial explosion. The tree walk approach keeps that (at the expense of some efficiency, unfortunately).
Quoted 5 lines
Quoted 3 lines
or disallowing a parent to be zapped while any of its children are alive.
What is the problem with that?
It reduces the mmu flexibility. If we (say) introduce an lru algorithm, it is orthogonal to everything else in the mmu. If we have a root/child dependency, the lru has to know.
Quoted 2 lines
And what the alternative would be, to zap all children first?
That has the disadvantage of allowing very bad corner cases if we are forced to zap a root.
I'd really like to avoid bad worst cases.
Quoted 5 lines
So more details please, what exactly is annoying you:
- Awareness of multiple roots in the current form ? I agree its not very elegant.
Yes.
Quoted 5 lines
- The fact that hash table bucket and
active_mmu_pagefor_each_entry_safewalks are unsafe because several list entries (the unsynced leafs) can be deleted ?
Quoted 8 lines
Hadn't even considered that...
What worries me most is that everything is interconnected: multiple modes, cr3 switch, out-of-sync, zapping via the inuse flag. It's very difficult for me to understand, what about someone new?
We need to make this fit better. We need to morph some mmu infrastructure to something else, but we can't keep adding complexity.
Another point that came to mind is that, with the unsync-info-on-tree approach, whenever a higher level pagetable is zapped all of its unsynced children need to be zapped.
Since at the moment resync on atomic context is not an option, the only option is to zap the children, potentially wasting a lot of shadowed entries. This is not an issue with the current approach since unsync children are tied to roots, not every parents all the way to root.
I'll measure how often that happens, but seems a bad side effect.
Re: [patch 09/13] KVM: MMU: out of sync shadow core
In reply to Avi KivityQuoted 58 lines · Avi Kivity
Quoted 6 lines · Marcelo Tosatti
Allow global and single-root, single-role-per-gfn leaf shadowed pagetables to be unsynced.
Global unsync pages are saved into a per-vm array, synced on cr4/cr0 writes.
Why not a list?
Quoted 25 lines
Non-global unsync pages are linked off their root shadow page, synced on cr3/cr4/cr0 writes.
Some of this logic is simplistic and could be smarter (
page_multimappedand the full root sync on higher level pagetable sharing).Also unsyncing of non-leaf nodes might be interesting (but more complicated).
+static struct
kvm_mmu_page*kvm_mmu_lookup_page_root(structkvm_vcpu
*vcpu,arch/x86/kvm/mmu.c +13
-0+ gfn_t gfn) +{ +unsigned index; +struct hlist_head *bucket; +struct kvm_mmu_page *sp; +struct hlist_node *node; +struct kvm *kvm = vcpu->kvm; +int level = vcpu->arch.mmu.root_level; +if (!is_long_mode (vcpu) &&is_pae (vcpu)) + level--; + +pgprintk ("%s: looking for gfn %lx\n" , __func__, gfn); + index =kvm_page_table_hashfn (gfn);arch/x86/kvm/mmu.c static struct kvm_mmu_page *kvm_mmu_look · line 935
Quoted 10 lines
arch/x86/kvm/mmu.c +10
-0+ bucket = &kvm->arch.mmu_page_hash[index]; +hlist_for_each_entry (sp, node, bucket, hash_link) +if (sp->gfn == gfn && !sp->role.metaphysical + && !sp->role.invalid && sp->role.level == level) { +pgprintk ("%s: found role %x\n" , + __func__, sp->role.word); +return sp; + } +return NULL ; +}I'm worried about the complexity this (and the rest) introduces.
A possible alternative is:
- for non-leaf pages, including roots, add a '
unsync_children' flag.
- when marking a page unsync, set the flag recursively on all parents
- when switching cr3, recursively descend to locate unsynced leaves, clearing flags along the way
- to speed this up, put a bitmap with 1 bit per pte in the pages (512 bits = 64 bytes)
- the bitmap can be externally allocated to save space, or not
Quoted 2 lines
This means we no longer have to worry about multiple roots, when a page acquires another root while it is unsynced, etc.
While trying to implement it, found a pretty nasty issue with this. A process can switch to a shadow root which has no present entries, whose unvisible (by cr3 switch time) children may be unsynced.
The Xen implementation gets away with that by syncing all oos pages on cr3 switch (which are a few).
The obvious way to fix it is to resync at kvm_mmu_get_page time, whenever grabbing an unsynced page. Which greatly reduces the out-of-sync time of a pagetable, breaking the ability to populate via pagefault without resync.
The singleroot scheme avoids such "invisible" unsynced pages, all out-of-sync pages in the root in question are reachable via sp->unsynced_pages.
Any ideas on how to get around that situation?
PATCH 10/13 KVM: MMU: sync roots on mmu reload
In reply to Marcelo TosattiBring the pages for the root being switched to back in sync.
arch/x86/kvm/mmu.c +48 -0
@@ -1065,6 +1065,20 @@ static int kvm_unsync_page(struct kvm_vcreturn mmu_unsync_page (vcpu, sp); } +static int mmu_sync_root (struct kvm_vcpu *vcpu,struct kvm_mmu_page *root_sp) +{ +struct kvm_mmu_page *sp, *n; +int ret =0 ; + +list_for_each_entry_safe (sp, n, &root_sp->unsync_pages, oos_link) { + ret =1 ; +kvm_sync_page (vcpu, sp); + } + +OOS_ASSERT (list_empty (&root_sp->unsync_pages)); +return ret; +} +static int set_shared_mmu_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) {struct kvm_mmu_page *root_sp;
@@ -1075,6 +1089,8 @@ static int set_shared_mmu_page(struct kvkvm_sync_page (vcpu, sp); }else if (sp->root_gfn !=-1 ) { root_sp =kvm_mmu_lookup_page_root (vcpu, sp->root_gfn); +if (root_sp) + ret =mmu_sync_root (vcpu, root_sp); } sp->root_gfn =-1 ;
@@ -1716,6 +1732,37 @@ static void mmu_alloc_roots(struct kvm_v vcpu->arch.mmu.root_hpa =__pa (vcpu->arch.mmu.pae_root); } +static void mmu_sync_roots (struct kvm_vcpu *vcpu) +{ +int i; +struct kvm_mmu_page *sp; + +if (!VALID_PAGE (vcpu->arch.mmu.root_hpa)) +return ; +if (vcpu->arch.mmu.shadow_root_level ==PT64_ROOT_LEVEL ) { +hpa_t root = vcpu->arch.mmu.root_hpa; + sp =page_header (root); +mmu_sync_root (vcpu, sp); +return ; + } +for (i =0 ; i <4 ; ++i) { +hpa_t root = vcpu->arch.mmu.pae_root[i]; + +if (root) { + root &=PT64_BASE_ADDR_MASK ; + sp =page_header (root); +mmu_sync_root (vcpu, sp); + } + } +} + +void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu) +{ +spin_lock (&vcpu->kvm->mmu_lock); +mmu_sync_roots (vcpu); +spin_unlock (&vcpu->kvm->mmu_lock); +} +static gpa_t nonpaging_gva_to_gpa (struct kvm_vcpu *vcpu,gva_t vaddr) {return vaddr;
@@ -1960,6 +2007,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)spin_lock (&vcpu->kvm->mmu_lock);kvm_mmu_free_some_pages (vcpu);mmu_alloc_roots (vcpu); +mmu_sync_roots (vcpu);spin_unlock (&vcpu->kvm->mmu_lock); kvm_x86_ops->set_cr3 (vcpu, vcpu->arch.mmu.root_hpa);kvm_mmu_flush_tlb (vcpu);
arch/x86/kvm/x86.c +1 -0
@@ -582,6 +582,7 @@ EXPORT_SYMBOL_GPL(kvm_set_cr4);void kvm_set_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3) {if (cr3 == vcpu->arch.cr3 && !pdptrs_changed (vcpu)) { +kvm_mmu_sync_roots (vcpu);kvm_mmu_flush_tlb (vcpu);return ; }
include/asm-x86/kvm_host.h +1 -0
@@ -610,6 +610,7 @@ int kvm_mmu_unprotect_page_virt(struct kvoid __kvm_mmu_free_some_pages (struct kvm_vcpu *vcpu);int kvm_mmu_load (struct kvm_vcpu *vcpu);void kvm_mmu_unload (struct kvm_vcpu *vcpu); +void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu);int kvm_emulate_hypercall (struct kvm_vcpu *vcpu);
PATCH 11/13 KVM: MMU: sync global pages on cr0/cr4 writes
In reply to Marcelo Tosattiarch/x86/kvm/mmu.c +14 -0
@@ -1097,6 +1097,20 @@ static int set_shared_mmu_page(struct kvreturn ret; } +void kvm_mmu_sync_global (struct kvm_vcpu *vcpu) +{ +int i; +struct kvm *kvm = vcpu->kvm; + +spin_lock (&kvm->mmu_lock); +for (i =0 ; i <ARRAY_SIZE (kvm->arch.oos_global_pages); i++) { +struct kvm_mmu_page *sp = kvm->arch.oos_global_pages[i]; +if (sp) +kvm_sync_page (vcpu, sp); + } +spin_unlock (&kvm->mmu_lock); +} + static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, gfn_t root_gfn, gfn_t gfn,
arch/x86/kvm/x86.c +2 -0
@@ -532,6 +532,7 @@ void kvm_set_cr0(struct kvm_vcpu *vcpu, kvm_x86_ops->set_cr0 (vcpu, cr0); vcpu->arch.cr0 = cr0; +kvm_mmu_sync_global (vcpu);kvm_mmu_reset_context (vcpu);return ; }
@@ -575,6 +576,7 @@ void kvm_set_cr4(struct kvm_vcpu *vcpu, } kvm_x86_ops->set_cr4 (vcpu, cr4); vcpu->arch.cr4 = cr4; +kvm_mmu_sync_global (vcpu);kvm_mmu_reset_context (vcpu); }EXPORT_SYMBOL_GPL (kvm_set_cr4);
include/asm-x86/kvm_host.h +1 -0
@@ -611,6 +611,7 @@ void __kvm_mmu_free_some_pages(struct kvint kvm_mmu_load (struct kvm_vcpu *vcpu);void kvm_mmu_unload (struct kvm_vcpu *vcpu);void kvm_mmu_sync_roots (struct kvm_vcpu *vcpu); +void kvm_mmu_sync_global (struct kvm_vcpu *vcpu);int kvm_emulate_hypercall (struct kvm_vcpu *vcpu);
PATCH 12/13 KVM: x86: trap invlpg
In reply to Marcelo TosattiWith pages out of sync invlpg needs to be trapped. For now simply nuke the entry.
The SVM code is untested and probably broken.
arch/x86/kvm/vmx.c +11 -0
@@ -1130,6 +1130,7 @@ static __init int setup_vmcs_config(struCPU_BASED_CR3_STORE_EXITING |CPU_BASED_USE_IO_BITMAPS |CPU_BASED_MOV_DR_EXITING | +CPU_BASED_INVLPG_EXITING |CPU_BASED_USE_TSC_OFFSETING ; opt =CPU_BASED_TPR_SHADOW | CPU_BASED_USE_MSR_BITMAPS |
@@ -2790,6 +2791,15 @@ static int handle_vmcall(struct kvm_vcpureturn 1 ; } +static int handle_invlpg (struct kvm_vcpu *vcpu,struct kvm_run *kvm_run) +{ +u64 exit_qualification =vmcs_read64 (EXIT_QUALIFICATION ); + +kvm_mmu_invlpg (vcpu, exit_qualification); +skip_emulated_instruction (vcpu); +return 1 ; +} +static int handle_wbinvd (struct kvm_vcpu *vcpu,struct kvm_run *kvm_run) {skip_emulated_instruction (vcpu);
@@ -2958,6 +2968,7 @@ static int (*kvm_vmx_exit_handlers[])(st [EXIT_REASON_MSR_WRITE] = handle_wrmsr, [EXIT_REASON_PENDING_INTERRUPT] = handle_interrupt_window, [EXIT_REASON_HLT] = handle_halt, + [EXIT_REASON_INVLPG] = handle_invlpg, [EXIT_REASON_VMCALL] = handle_vmcall, [EXIT_REASON_TPR_BELOW_THRESHOLD] = handle_tpr_below_threshold, [EXIT_REASON_APIC_ACCESS] = handle_apic_access,
arch/x86/kvm/mmu.c +18 -0
@@ -889,6 +889,12 @@ static int nonpaging_sync_page(struct kvreturn 1 ; } +static void nonpaging_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +{ +/* should never happen */ +WARN_ON (1 ); +} +static struct kvm_mmu_page *kvm_mmu_lookup_page (struct kvm *kvm,gfn_t gfn) {unsigned index;
@@ -1860,6 +1866,7 @@ static int nonpaging_init_context(struct context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; context->sync_page = nonpaging_sync_page; + context->invlpg = nonpaging_invlpg; context->root_level =0 ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;
@@ -1908,6 +1915,7 @@ static int paging64_init_context_common( context->gva_to_gpa = paging64_gva_to_gpa; context->prefetch_page = paging64_prefetch_page; context->sync_page = paging64_sync_page; + context->invlpg = paging64_invlpg; context->free = paging_free; context->root_level = level; context->shadow_root_level = level;
@@ -1930,6 +1938,7 @@ static int paging32_init_context(struct context->free = paging_free; context->prefetch_page = paging32_prefetch_page; context->sync_page = paging32_sync_page; + context->invlpg = paging32_invlpg; context->root_level =PT32_ROOT_LEVEL ; context->shadow_root_level =PT32E_ROOT_LEVEL ; context->root_hpa =INVALID_PAGE ;
@@ -1950,6 +1959,7 @@ static int init_kvm_tdp_mmu(struct kvm_v context->free = nonpaging_free; context->prefetch_page = nonpaging_prefetch_page; context->sync_page = nonpaging_sync_page; + context->invlpg = nonpaging_invlpg; context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ;
@@ -2343,6 +2353,14 @@ out: }EXPORT_SYMBOL_GPL (kvm_mmu_page_fault); +void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +{ +spin_lock (&vcpu->kvm->mmu_lock); + vcpu->arch.mmu.invlpg (vcpu, gva); +spin_unlock (&vcpu->kvm->mmu_lock); +} +EXPORT_SYMBOL_GPL (kvm_mmu_invlpg); +void kvm_enable_tdp (void ) { tdp_enabled = true;
arch/x86/kvm/paging_tmpl.h +26 -0
@@ -467,6 +467,32 @@ out_unlock:return 0 ; } + +static int FNAME (shadow_invlpg_entry )(struct kvm_shadow_walk *_sw, +struct kvm_vcpu *vcpu,u64 addr, +u64 *sptep,int level) +{ + +if (level ==PT_PAGE_TABLE_LEVEL ) { +if (is_shadow_present_pte (*sptep)) +rmap_remove (vcpu->kvm, sptep); +set_shadow_pte (sptep, shadow_trap_nonpresent_pte); +return 1 ; + } +if (!is_shadow_present_pte (*sptep)) +return 1 ; +return 0 ; +} + +static void FNAME (invlpg )(struct kvm_vcpu *vcpu,gva_t gva) +{ +struct shadow_walker walker = { + .walker = { .entry =FNAME (shadow_invlpg_entry), }, + }; + +walk_shadow (&walker.walker, vcpu, gva); +} +static gpa_t FNAME (gva_to_gpa )(struct kvm_vcpu *vcpu,gva_t vaddr) {struct guest_walker walker;
include/asm-x86/kvm_host.h +2 -0
@@ -227,6 +227,7 @@ struct kvm_mmu { struct kvm_mmu_page *page);int (*sync_page)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp); +void (*invlpg)(struct kvm_vcpu *vcpu,gva_t gva);hpa_t root_hpa;int root_level;int shadow_root_level;
@@ -618,6 +619,7 @@ int kvm_emulate_hypercall(struct kvm_vcpint kvm_fix_hypercall (struct kvm_vcpu *vcpu);int kvm_mmu_page_fault (struct kvm_vcpu *vcpu,gva_t gva,u32 error_code); +void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva);void kvm_enable_tdp (void );void kvm_disable_tdp (void );
arch/x86/kvm/svm.c +11 -1
@@ -525,6 +525,7 @@ static void init_vmcb(struct vcpu_svm *s (1ULL <<INTERCEPT_CPUID ) | (1ULL <<INTERCEPT_INVD ) | (1ULL <<INTERCEPT_HLT ) | + (1ULL <<INTERCEPT_INVLPG ) | (1ULL <<INTERCEPT_INVLPGA ) | (1ULL <<INTERCEPT_IOIO_PROT ) | (1ULL << INTERCEPT_MSR_PROT) |
@@ -1160,6 +1161,15 @@ static int cpuid_interception(struct vcpreturn 1 ; } +static int invlpg_interception (struct vcpu_svm *svm,struct kvm_run *kvm_run) +{ +/* FIXME: does this make any sense? */ +u64 vaddr = svm->vmcb->control.exit_info_1; +kvm_mmu_invlpg (&svm->vcpu, vaddr); +skip_emulated_instruction (&svm->vcpu); +return 1 ; +} +static int emulate_on_interception (struct vcpu_svm *svm,struct kvm_run *kvm_run) {
@@ -1413,7 +1423,7 @@ static int (*svm_exit_handlers[])(struct [SVM_EXIT_CPUID] = cpuid_interception, [SVM_EXIT_INVD] = emulate_on_interception, [SVM_EXIT_HLT] = halt_interception, - [SVM_EXIT_INVLPG] = emulate_on_interception, + [SVM_EXIT_INVLPG] = invlpg_interception, [SVM_EXIT_INVLPGA] = invalid_op_interception, [SVM_EXIT_IOIO] = io_interception, [SVM_EXIT_MSR] = msr_interception,
Re: [patch 12/13] KVM: x86: trap invlpg
Quoted 4 lines · Marcelo Tosatti
With pages out of sync invlpg needs to be trapped. For now simply nuke the entry.
We could kvm_mmu_pte_write() it, with some modifications. Probably worth it for the guest breaking cow.
arch/x86/kvm/vmx.c static int (*kvm_vmx_exit_handlers[])(st · line 2974
Quoted 7 lines
arch/x86/kvm/vmx.c +1
-0[EXIT_REASON_MSR_WRITE] = handle_wrmsr, [EXIT_REASON_PENDING_INTERRUPT] = handle_interrupt_window, [EXIT_REASON_HLT] = handle_halt, + [EXIT_REASON_INVLPG] = handle_invlpg, [EXIT_REASON_VMCALL] = handle_vmcall, [EXIT_REASON_TPR_BELOW_THRESHOLD] = handle_tpr_below_threshold, [EXIT_REASON_APIC_ACCESS] = handle_apic_access,
We shouldn't do that if ept or npt is enabled.
Quoted 5 lines
Index:
kvm/arch/x86/kvm/mmu.c
===================================================================
arch/x86/kvm/mmu.c static int nonpaging_sync_page(struct kv · line 897
Quoted 9 lines
arch/x86/kvm/mmu.c +5
-0@@ -889,6 +889,12 @@ static int nonpaging_sync_page(struct kvreturn 1 ; } +static void nonpaging_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +{ +/* should never happen */ +WARN_ON (1 ); +}
Nevertheless, invlpg is legal in real mode.
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH 13/13 KVM: MMU: ignore multiroot when unsyncing global pages
In reply to Marcelo TosattiSince global pages are not linked in terms of TLB flushing to any root.
arch/x86/kvm/mmu.c +1 -1
@@ -1429,7 +1429,7 @@ static int mmu_need_write_protect(structreturn 1 ;if (shadow->role.level !=PT_PAGE_TABLE_LEVEL )return 1 ; -if (shadow->root_gfn ==-1 ) +if (shadow->root_gfn ==-1 && !kvm_page_global (shadow))return 1 ;if (!kvm_page_unsync (shadow))return kvm_unsync_page (vcpu, shadow);
Quoted 4 lines · Marcelo Tosatti
Keep shadow pages temporarily out of sync, allowing more efficient guest PTE updates in comparison to trap-emulate + unprotect heuristics. Stolen from Xen :)
Quoted 8 lines
This version only allows leaf pagetables to go out of sync, for simplicity, but can be enhanced.
VMX "
bypass_guest_pf" feature onprefetch_pagebreaks it (since new
PTE writes need no TLB flush, I assume). Not sure if its worthwhile to
convertnotrap_nonpresent->trap_nonpresenton unshadow or just go for unconditionalnonpaging_prefetch_page.
Doesn't it kill bypass_guest_pf completely? As soon as we unsync a page, we can't have nontrapping nonpresent ptes in it.
We can try convertion on unsync, it does speed up demand paging.
Quoted 20 lines
* Kernel builds on 4-way 64-bit guest improve 10% (+ 3.7% for
get_user_pages_fast).* lmbench's "
lat_procfork" microbenchmark latency is 40% lower (a shadow worst scenario test).* The RHEL3 highpte kscand hangs go from 5+ seconds to < 1 second.
* Windows 2003 Server, 32-bit PAE, DDK build (build -cPzM 3):
Windows 2003 Checked 64 Bit Build Environment, 256M RAM 1-vcpu:
vanilla +gup_fast: oos
0:04:37.375 0:03:28.047 (- 25%)2-vcpus:
vanilla +gup_fastoos
0:02:32.000 0:01:56.031 (- 23%)
Quoted 10 lines
Windows 2003 Checked Build Environment, 1GB RAM
2-vcpus:
vanilla +fast_gupoos
0:02:26.078 0:01:50.110 (- 24%)4-vcpus:
vanilla +fast_gupoos
0:01:59.266 0:01:29.625 (- 25%)
Impressive results.
Quoted 3 lines
And I think other optimizations are possible now, for example the guest can be responsible for remote TLB flushing on
kvm_mmu_pte_write().
But kvm_mmu_pte_write() is no longer called, since we unsync?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines
VMX "
bypass_guest_pf" feature onprefetch_pagebreaks it (since new PTE writes need no TLB flush, I assume). Not sure if its worthwhile to convertnotrap_nonpresent->trap_nonpresenton unshadow or just go for unconditionalnonpaging_prefetch_page.Doesn't it kill
bypass_guest_pfcompletely? As soon as we unsync a page, we can't have nontrapping nonpresent ptes in it.
Yes, thats what I meant.
Quoted 7 lines
We can try convertion on unsync, it does speed up demand paging.
Quoted 3 lines
And I think other optimizations are possible now, for example the guest can be responsible for remote TLB flushing on
kvm_mmu_pte_write().But
kvm_mmu_pte_write()is no longer called, since we unsync?
On higher level pagetables it still is. The IPI functions show high up in profiling (probably more due to write protection, but still).
Quoted 8 lines · Marcelo Tosatti
Quoted 6 lines · Avi Kivity
Quoted 6 lines
VMX "
bypass_guest_pf" feature onprefetch_pagebreaks it (since new
PTE writes need no TLB flush, I assume). Not sure if its worthwhile to
convertnotrap_nonpresent->trap_nonpresenton unshadow or just go for unconditionalnonpaging_prefetch_page.
Quoted 6 lines
Quoted 2 lines
Doesn't it kill
bypass_guest_pfcompletely? As soon as we unsync a page, we can't have nontrapping nonpresent ptes in it.Yes, thats what I meant.
We can convert those ptes during unsync, and back during resync. That's a fairly cheap operation.
Signature · 1 line
error compiling committee.c: too many arguments to function
Hi Marcelo:
This patchset causes my RHEL3 guest to hang during boot at one of the early sym53c8xx messages:
sym53c8xx: at PCI bus 0, device 5, functions 0
Using ide instead of scsi the guest proceeds farther, but inevitably hangs as well. I've tried dropping the amount of ram to 1024 and varied the number of vcpus as well (including 1 vcpu).
When it hangs kvm on the host is spinning on one of the cpus, and kvm/qemu appears to be 1 thread short. For the kvm process I expect to see 2 + Nvcpus threads (ps -C kvm -L). With this patchset I see 2 + Nvcpus - 1. (e.g., I usually run with 4 vcpus, so there should be 6 threads. I see only 5).
I'm using kvm-git tip from a couple of days ago + this patch set. kvm userspace comes from kvm-75. Resetting to kvm-git and the guest starts up just fine.
david
Quoted 47 lines · Marcelo Tosatti
Keep shadow pages temporarily out of sync, allowing more efficient guest PTE updates in comparison to trap-emulate + unprotect heuristics. Stolen from Xen :)
This version only allows leaf pagetables to go out of sync, for simplicity, but can be enhanced.
VMX "
bypass_guest_pf" feature onprefetch_pagebreaks it (since new PTE writes need no TLB flush, I assume). Not sure if its worthwhile to convertnotrap_nonpresent->trap_nonpresenton unshadow or just go for unconditionalnonpaging_prefetch_page.* Kernel builds on 4-way 64-bit guest improve 10% (+ 3.7% for
get_user_pages_fast).* lmbench's "
lat_procfork" microbenchmark latency is 40% lower (a shadow worst scenario test).* The RHEL3 highpte kscand hangs go from 5+ seconds to < 1 second.
* Windows 2003 Server, 32-bit PAE, DDK build (build -cPzM 3):
Windows 2003 Checked 64 Bit Build Environment, 256M RAM 1-vcpu:
vanilla +gup_fast: oos
0:04:37.375 0:03:28.047 (- 25%)2-vcpus:
vanilla +gup_fastoos
0:02:32.000 0:01:56.031 (- 23%)Windows 2003 Checked Build Environment, 1GB RAM
2-vcpus:
vanilla +fast_gupoos
0:02:26.078 0:01:50.110 (- 24%)4-vcpus:
vanilla +fast_gupoos
0:01:59.266 0:01:29.625 (- 25%)And I think other optimizations are possible now, for example the guest can be responsible for remote TLB flushing on
kvm_mmu_pte_write().Please review.
Quoted 17 lines · David S. Ahern
Hi Marcelo:
This patchset causes my RHEL3 guest to hang during boot at one of the early sym53c8xx messages:
sym53c8xx: at PCI bus 0, device 5, functions 0
Using ide instead of scsi the guest proceeds farther, but inevitably hangs as well. I've tried dropping the amount of ram to 1024 and varied the number of vcpus as well (including 1 vcpu).
When it hangs kvm on the host is spinning on one of the cpus, and kvm/qemu appears to be 1 thread short. For the kvm process I expect to see 2 + Nvcpus threads (ps -C kvm -L). With this patchset I see 2 + Nvcpus - 1. (e.g., I usually run with 4 vcpus, so there should be 6 threads. I see only 5).
Quoted 3 lines
I'm using kvm-git tip from a couple of days ago + this patch set. kvm userspace comes from kvm-75. Resetting to kvm-git and the guest starts up just fine.
David,
Please reload the kvm-intel module with "bypass_guest_pf=0" option.
Quoted 7 lines · Marcelo Tosatti
On Thu, Sep 11, 2008 at 10:05:12PM -0600, David S. Ahern wrote:
David,
Please reload the kvm-intel module with "bypass_guest_pf=0" option.
DOH. You mentioned that in your description, and I forgot to disable it.
Works fine now.
Guest behavior is amazing. After 30 min of uptime, kscand shows only 17secs of cpu usage. Before, kscand was hitting over a minute after about 10 minutes of uptime.
david
Quoted 11 lines · David S. Ahern
Quoted 7 lines · Marcelo Tosatti
On Thu, Sep 11, 2008 at 10:05:12PM -0600, David S. Ahern wrote:
David,
Please reload the kvm-intel module with "bypass_guest_pf=0" option.
Quoted 6 lines
DOH. You mentioned that in your description, and I forgot to disable it.
Works fine now.Guest behavior is amazing. After 30 min of uptime, kscand shows only 17secs of cpu usage. Before, kscand was hitting over a minute after about 10 minutes of uptime.
Great. Note that its not fully optimized for the RHEL3 highpte kscand case, which calls invlpg for present pagetable entries.
The current patchset simply invalidates the shadow on invlpg, meaning that the next access will cause a pagefault exit.
It can instead read the guest pte and prefault, saving one exit per test-and-clear-accessed operation.
Quoted 17 lines · Marcelo Tosatti
Quoted 9 lines · David S. Ahern
Quoted 7 lines · Marcelo Tosatti
On Thu, Sep 11, 2008 at 10:05:12PM -0600, David S. Ahern wrote:
David,
Please reload the kvm-intel module with "bypass_guest_pf=0" option.
Quoted 6 lines
DOH. You mentioned that in your description, and I forgot to disable it.
Works fine now.Guest behavior is amazing. After 30 min of uptime, kscand shows only 17secs of cpu usage. Before, kscand was hitting over a minute after about 10 minutes of uptime.
Quoted 9 lines
Great. Note that its not fully optimized for the RHEL3 highpte kscand case, which calls invlpg for present pagetable entries.
The current patchset simply invalidates the shadow on invlpg, meaning that the next access will cause a pagefault exit.
It can instead read the guest pte and prefault, saving one exit per test-and-clear-accessed operation.
Meaning performance will get even better? Sweet.
david